fix(gfql): validate chain construction (#861)

lmeyerov · web-flow · commit edd6b16cb594 · 2025-12-03T18:45:44.000-08:00
* fix(gfql): validate chain construction

* docs(gfql): clarify validation defaults

* test(gfql): cover nested/from_json validation

* chore(mypy): let runner set python version; ignore pandas where(None) arg

* chore(mypy): silence pandas where(None) overload

* chore(kusto): type-safe null normalization
diff --git a/.gitignore b/.gitignore
@@ -74,6 +74,7 @@ demos/data/BIOGRID-IDENTIFIERS-3.3.123.tab.txt
 
 # Ignore local Python virtualenv folders (used by Jenkins when running tests, etc.)
 /.pyenv*/
+.venv/
 
 # Ignore IDE stuff
 .idea
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -8,6 +8,12 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 ## [Development]
 <!-- Do Not Erase This Section - Used for tracking unreleased changes -->
 
+### Fixed
+- **GFQL:** `Chain` now validates on construction (matching docs) and rejects invalid hops immediately; pass `validate=False` to defer validation when assembling advanced flows (fixes #860).
+
+### Docs
+- **GFQL validation:** Clarified `Chain` constructor validation defaults, `validate=False` defer option, validation phases, and guidance for large/nested ASTs to reduce redundant validation (issue #860).
+
 ## [0.46.0 - 2025-12-01]
 
 ### Added
diff --git a/demos/demos_databases_apis/memgraph/visualizing_iam_dataset.ipynb b/demos/demos_databases_apis/memgraph/visualizing_iam_dataset.ipynb
@@ -289,7 +289,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "![Screenshot](https://github.com/karmenrabar/pygraphistry_images/blob/main/memgraphlab.png?raw=true)"
+    "![Screenshot](https://raw.githubusercontent.com/karmenrabar/pygraphistry_images/refs/heads/main/memgraphlab.png)"
    ]
   },
   {
diff --git a/docs/source/gfql/spec/llm_guide.md b/docs/source/gfql/spec/llm_guide.md
@@ -25,7 +25,7 @@
 - [Graph Algorithms](#graph-algorithms) - PageRank, Louvain, UMAP, Hypergraph
 - [Visualization](#visualization) - Colors, icons, sizes
 - [Layouts](#layouts) - FA2 default, ring layouts
-- [Multi-Step (Let/Ref)](#multi-step-letref) - DAG composition
+- [Multi-Step (Let/Ref)](#let-multi-step) - DAG composition
 
 **Domain Guidance:**
 - [Icons & Palettes](#domain-guidance) - By vertical (Cyber, Fraud, Gov, Social, Supply Chain, Events)
diff --git a/docs/source/gfql/spec/python_embedding.md b/docs/source/gfql/spec/python_embedding.md
@@ -105,7 +105,7 @@ GFQL provides comprehensive validation to catch errors early:
 
 ### Syntax Validation
 
-Operations are automatically validated during construction:
+Chains validate on construction by default. Nodes, edges, predicates, refs, calls, and remote graphs are validated when a parent `Chain`/`Let` validates them or when you call `.validate()` directly. Schema validation is a separate, data-aware pass.
 
 ```python
 from graphistry.compute.chain import Chain
@@ -118,6 +118,28 @@ chain = Chain([
 ])
 ```
 
+For advanced flows (large/nested ASTs or staged assembly), you can defer structural validation and run it once after assembly:
+
+```python
+# Defer validation while building
+chain = Chain([
+    n({'type': 'person'}),
+    e_forward({'hops': -1})
+], validate=False)  # No validation yet
+
+# Later, validate once (or let g.gfql validate it)
+chain.validate()  # Raises GFQLTypeError: hops must be positive
+```
+
+Use deferred validation to avoid re-validating nested `Chain`/`Let` wrappers during assembly; keep the defaults for typical workflows so mistakes surface immediately.
+
+### Validation Phases
+
+- **Constructor defaults:** `Chain([...])` and `Let(...)` validate immediately; pass `validate=False` to defer.
+- **Parent-driven checks:** AST operations (`Node`, `Edge`, predicates, `Ref`, `Call`, `RemoteGraph`) validate when their parent validates, or via explicit `.validate()`.
+- **JSON defaults:** `to_json` / `from_json` default to `validate=True`, which runs structural validation during serialization/deserialization.
+- **Schema validation:** Use `validate_chain_schema(g, chain)` or `g.gfql(..., validate_schema=True)` to verify column/type compatibility before execution.
+
 ### Schema Validation
 
 You have two options for validating queries against your data schema:
@@ -409,4 +431,4 @@ result = g.gfql(let({
 ## See Also
 
 - {ref}`gfql-spec-language` - Core language specification
-- [GFQL Quick Reference](../quick.rst) - Python API examples
+- [GFQL Quick Reference](../quick.rst) - Python API examples
diff --git a/docs/source/graphistry.compute.gfql_validation.rst b/docs/source/graphistry.compute.gfql_validation.rst
@@ -1,27 +1,10 @@
 graphistry.compute.gfql\_validation package
 ===========================================
 
-Submodules
-----------
+.. note::
 
-graphistry.compute.gfql\_validation.exceptions module
------------------------------------------------------
-
-.. automodule:: graphistry.compute.gfql_validation.exceptions
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-graphistry.compute.gfql\_validation.validate module
----------------------------------------------------
-
-.. automodule:: graphistry.compute.gfql_validation.validate
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-Module contents
----------------
+   Deprecated. Use :mod:`graphistry.compute.gfql` instead. This module only re-exports
+   the GFQL validation APIs for backward compatibility.
 
 .. automodule:: graphistry.compute.gfql_validation
    :members:
diff --git a/docs/source/graphistry.compute.rst b/docs/source/graphistry.compute.rst
@@ -61,14 +61,6 @@ graphistry.compute.chain\_remote module
    :undoc-members:
    :show-inheritance:
 
-graphistry.compute.chain\_validate module
------------------------------------------
-
-.. automodule:: graphistry.compute.chain_validate
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
 graphistry.compute.cluster module
 ---------------------------------
 
diff --git a/graphistry/compute/ast.py b/graphistry/compute/ast.py
@@ -662,22 +662,26 @@ def from_json(cls, d: dict, validate: bool = True) -> 'ASTEdge':
 
 class ASTLet(ASTObject):
     """Let-bindings for named graph operations in a DAG.
-    
-    Allows defining reusable graph operations that can reference each other,
+
+    Lets you define reusable graph operations that can reference each other,
     forming a directed acyclic graph (DAG) of computations.
-    
-    :param bindings: Dictionary mapping names to graph operations
-    :type bindings: Dict[str, Union[ASTObject, Chain, Plottable]]
-    
-    :raises GFQLTypeError: If bindings is not a dict or contains invalid keys/values
-    
-    **Example::**
 
-        # Matchers now supported directly (operate on root graph)
-        dag = ASTLet({
-            'persons': n({'type': 'person'}),
-            'friends': ASTRef('persons', [e_forward({'rel': 'friend'})])
-        })
+    Parameters
+    ----------
+    bindings : Dict[str, Union[ASTObject, Chain, Plottable]]
+        Mapping from binding names to graph operations (AST objects or Plottables).
+
+    Raises
+    ------
+    GFQLTypeError
+        If ``bindings`` is not a dict or contains invalid keys/values.
+
+    Example
+    -------
+    >>> dag = ASTLet({
+    ...     "persons": n({"type": "person"}),
+    ...     "friends": ASTRef("persons", [e_forward({"rel": "friend"})])
+    ... })
     """
     bindings: Dict[str, Union['ASTObject', 'Chain', Plottable]]
     
diff --git a/graphistry/compute/chain.py b/graphistry/compute/chain.py
@@ -25,8 +25,11 @@
 
 class Chain(ASTSerializable):
 
-    def __init__(self, chain: List[ASTObject]) -> None:
+    def __init__(self, chain: List[ASTObject], validate: bool = True) -> None:
         self.chain = chain
+        if validate:
+            # Fail fast on invalid chains; matches documented automatic validation behavior
+            self.validate(collect_all=False)
 
     def validate(self, collect_all: bool = False) -> Optional[List['GFQLValidationError']]:
         """Override to collect all chain validation errors."""
@@ -116,9 +119,7 @@ def from_json(cls, d: Dict[str, JSONVal], validate: bool = True) -> 'Chain':
                 f"Chain field must be a list, got {type(d['chain']).__name__}"
             )
         
-        out = cls([ASTObject_from_json(op, validate=validate) for op in d['chain']])
-        if validate:
-            out.validate()
+        out = cls([ASTObject_from_json(op, validate=validate) for op in d['chain']], validate=validate)
         return out
 
     def to_json(self, validate=True) -> Dict[str, JSONVal]:
diff --git a/graphistry/compute/gfql_validation/__init__.py b/graphistry/compute/gfql_validation/__init__.py
@@ -1,13 +1,12 @@
 """
-DEPRECATED: This module is deprecated and will be removed in a future version.
+DEPRECATED: Use ``graphistry.compute.gfql`` instead.
 
-All functionality has been moved to graphistry.compute.gfql.
-Please update your imports:
-  FROM: graphistry.compute.gfql_validation
-  TO:   graphistry.compute.gfql
+All functionality moved to ``graphistry.compute.gfql``. Update imports:
 
-This duplicate module was created accidentally during code extraction and
-provides no additional functionality.
+- From: ``graphistry.compute.gfql_validation``
+- To:   ``graphistry.compute.gfql``
+
+This duplicate module was created during code extraction and provides no additional functionality.
 """
 
 import warnings
diff --git a/graphistry/compute/predicates/str.py b/graphistry/compute/predicates/str.py
@@ -268,28 +268,20 @@ def startswith(
     na: Optional[bool] = None
 ) -> Startswith:
     """
-    Return whether a given pattern or tuple of patterns is at the
-    start of a string
+    Return whether a given pattern or tuple of patterns is at the start of a string.
 
-    Args:
-        pat: Pattern (str) or tuple of patterns to match at start of
-             string. When tuple, returns True if string starts with ANY
-             pattern (OR logic)
-        case: If True, case-sensitive matching (default: True)
-        na: Fill value for missing values (default: None)
+    :param pat: Pattern (str) or tuple of patterns to match at start of string. When tuple,
+        returns True if the string starts with ANY pattern (OR logic).
+    :param case: If True, case-sensitive matching (default: True).
+    :param na: Fill value for missing values (default: None).
+    :returns: Startswith predicate.
 
-    Returns:
-        Startswith predicate
-
-    Examples:
-        >>> # Single pattern, case-sensitive (default)
-        >>> n({"name": startswith("John")})
-        >>> # Single pattern, case-insensitive
-        >>> n({"name": startswith("john", case=False)})
-        >>> # Multiple patterns (OR logic)
-        >>> n({"filename": startswith(("test_", "demo_"))})
-        >>> # Multiple patterns, case-insensitive
-        >>> n({"filename": startswith(("TEST", "DEMO"), case=False)})
+    Examples
+    --------
+    >>> n({"name": startswith("John")})
+    >>> n({"name": startswith("john", case=False)})
+    >>> n({"filename": startswith(("test_", "demo_"))})
+    >>> n({"filename": startswith(("TEST", "DEMO"), case=False)})
     """
     return Startswith(pat, case, na)
 
@@ -446,28 +438,20 @@ def endswith(
     na: Optional[bool] = None
 ) -> Endswith:
     """
-    Return whether a given pattern or tuple of patterns is at the
-    end of a string
+    Return whether a given pattern or tuple of patterns is at the end of a string.
 
-    Args:
-        pat: Pattern (str) or tuple of patterns to match at end of
-             string. When tuple, returns True if string ends with ANY
-             pattern (OR logic)
-        case: If True, case-sensitive matching (default: True)
-        na: Fill value for missing values (default: None)
+    :param pat: Pattern (str) or tuple of patterns to match at end of string. When tuple,
+        returns True if the string ends with ANY pattern (OR logic).
+    :param case: If True, case-sensitive matching (default: True).
+    :param na: Fill value for missing values (default: None).
+    :returns: Endswith predicate.
 
-    Returns:
-        Endswith predicate
-
-    Examples:
-        >>> # Single pattern, case-sensitive (default)
-        >>> n({"email": endswith(".com")})
-        >>> # Single pattern, case-insensitive
-        >>> n({"email": endswith(".COM", case=False)})
-        >>> # Multiple patterns (OR logic)
-        >>> n({"filename": endswith((".txt", ".csv"))})
-        >>> # Multiple patterns, case-insensitive
-        >>> n({"filename": endswith((".TXT", ".CSV"), case=False)})
+    Examples
+    --------
+    >>> n({"email": endswith(".com")})
+    >>> n({"email": endswith(".COM", case=False)})
+    >>> n({"filename": endswith((".txt", ".csv"))})
+    >>> n({"filename": endswith((".TXT", ".CSV"), case=False)})
     """
     return Endswith(pat, case, na)
 
diff --git a/graphistry/plugins/kusto.py b/graphistry/plugins/kusto.py
@@ -456,7 +456,10 @@ def _unwrap_nested(result: "KustoQueryResult") -> pd.DataFrame:
         if df[col].isna().all():
             df = df.drop(columns=[col])
 
-    df = df.astype(object).where(pd.notna(df), None)
+    # Replace pandas/NA with Python None for consistency without type ignores
+    df = df.astype(object)
+    df = df.where(pd.notna(df))
+    df = df.fillna(value=None)
     return df.reset_index(drop=True)
 
 
diff --git a/graphistry/tests/compute/test_chain_validation.py b/graphistry/tests/compute/test_chain_validation.py
diff --git a/mypy.ini b/mypy.ini

Original file line number	Diff line number	Diff line change
`@@ -289,7 +289,7 @@`
`289`	`289`	`"cell_type": "markdown",`
`290`	`290`	`"metadata": {},`
`291`	`291`	`"source": [`
`292`		`- "![Screenshot](https://github.com/karmenrabar/pygraphistry_images/blob/main/memgraphlab.png?raw=true)"`
	`292`	`+ "![Screenshot](https://raw.githubusercontent.com/karmenrabar/pygraphistry_images/refs/heads/main/memgraphlab.png)"`
`293`	`293`	`]`
`294`	`294`	`},`
`295`	`295`	`{`