Skip to content

graflo.architecture.contract.bindings

Resource connectors and named binding collections.

Modules:

Name Description
column_time_filter

Shared datetime / timestamp column filter for resource connectors.

connectors

Resource connector types (file, SQL table, SPARQL, API, Kafka).

core

Named connectors and resource-to-connector wiring.

Attributes

Bindings = BindingsRegistry module-attribute

__all__ = ['APIConnector', 'ApiResponseStructure', 'Bindings', 'BindingsConfig', 'BindingsRegistry', 'BoundSourceKind', 'ColumnTimeFilter', 'ConnectorTemplate', 'ConnectorUpdate', 'FileConnector', 'KafkaConnector', 'PaginationConfig', 'PaginationRequestConfig', 'ParentRequest', 'PushConnector', 'ResourceConnector', 'ResourceConnectorBinding', 'SparqlConnector', 'TableConnector', 'TemplatedConnector', 'WatermarkConfig'] module-attribute

Classes

APIConnector

Bases: TemplatedConnector

Connector for REST API endpoints.

Declares the non-secret access pattern (path, method, pagination). Runtime base_url and credentials are supplied via connector_connection -> conn_proxy -> :class:~graflo.connections.sources.RestApiConnConfig.

path and params are address templates and body a query text (see :class:TemplatedConnector); their variables come from values and from parent.fields.

Attributes:

Name Type Description
path str

Relative endpoint path (e.g. /api/users).

method str

HTTP method (default GET).

params dict[str, Any]

Static query parameters.

body dict[str, Any] | str | None

Request body: a mapping is sent as JSON, a string as is. Not allowed with GET.

pagination PaginationConfig | None

Pagination strategy and response path configuration.

row_annotations dict[str, Any]

Constant fields merged into every fetched document (doc wins).

headers dict[str, str]

Non-secret HTTP headers.

timeout float | None

Request timeout in seconds.

retries int

Number of retry attempts. 0 fails on the first error.

retry_backoff_factor float

Backoff factor for retries.

retry_status_forcelist list[int]

HTTP status codes to retry on. A Retry-After header on the response sets the wait.

verify bool

Verify SSL certificates.

watermark WatermarkConfig | None

Where a run left off; see :class:WatermarkConfig.

parent ParentRequest | None

Make one request per record of another API connector, with {variable} in path, params and body filled from that record; see :class:ParentRequest. Without its own conn_proxy mapping the child uses the parent's connection.

values dict[str, list[TemplateValue]] | None

One request per value of each variable; see :class:TemplatedConnector.

Source code in graflo/architecture/contract/bindings/connectors.py
class APIConnector(TemplatedConnector):
    """Connector for REST API endpoints.

    Declares the non-secret access pattern (path, method, pagination). Runtime
    ``base_url`` and credentials are supplied via ``connector_connection`` ->
    ``conn_proxy`` -> :class:`~graflo.connections.sources.RestApiConnConfig`.

    ``path`` and ``params`` are address templates and ``body`` a query text
    (see :class:`TemplatedConnector`); their variables come from ``values``
    and from ``parent.fields``.

    Attributes:
        path: Relative endpoint path (e.g. ``/api/users``).
        method: HTTP method (default ``GET``).
        params: Static query parameters.
        body: Request body: a mapping is sent as JSON, a string as is. Not
            allowed with ``GET``.
        pagination: Pagination strategy and response path configuration.
        row_annotations: Constant fields merged into every fetched document (doc wins).
        headers: Non-secret HTTP headers.
        timeout: Request timeout in seconds.
        retries: Number of retry attempts. ``0`` fails on the first error.
        retry_backoff_factor: Backoff factor for retries.
        retry_status_forcelist: HTTP status codes to retry on. A ``Retry-After``
            header on the response sets the wait.
        verify: Verify SSL certificates.
        watermark: Where a run left off; see :class:`WatermarkConfig`.
        parent: Make one request per record of another API connector, with
            ``{variable}`` in ``path``, ``params`` and ``body`` filled from that
            record; see :class:`ParentRequest`. Without its own ``conn_proxy``
            mapping the child uses the parent's connection.
        values: One request per value of each variable; see
            :class:`TemplatedConnector`.
    """

    path: str = Field(..., description="Relative API endpoint path")
    method: str = "GET"
    params: dict[str, Any] = Field(default_factory=dict)
    body: dict[str, Any] | str | None = None
    pagination: PaginationConfig | None = None
    headers: dict[str, str] = Field(default_factory=dict)
    timeout: float | None = None
    retries: int = 0
    retry_backoff_factor: float = 0.1
    retry_status_forcelist: list[int] = Field(
        default_factory=lambda: [429, 500, 502, 503, 504]
    )
    verify: bool = True
    watermark: WatermarkConfig | None = None
    parent: ParentRequest | None = None

    @model_validator(mode="after")
    def _check_body(self) -> Self:
        if self.body is not None and self.method.upper() == "GET":
            raise ValueError(
                f"API connector '{self._label()}' has a body; a GET request "
                "carries none"
            )
        return self

    def _address_templates(self) -> list[Any]:
        return [self.path, self.params]

    def _text_templates(self) -> list[Any]:
        return [self.body]

    def _bound_variables(self) -> set[str]:
        return set(self.parent.fields) if self.parent is not None else set()

    def _label(self) -> str:
        return self.name or self.path

    def rendered(self, values: Mapping[str, Any]) -> Self:
        return self.model_copy(
            update={
                "path": render_text(self.path, values, quote_values=True),
                "params": render_param(self.params, values),
                "body": None if self.body is None else render_body(self.body, values),
                "values": None,
            }
        )

    def _hash_payload(self) -> dict[str, Any]:
        payload = super()._hash_payload()
        for optional in ("body", "watermark", "parent"):
            if payload.get(optional) is None:
                # Unset, it must not move the hash of connectors declared before it.
                payload.pop(optional, None)
        return payload

    @staticmethod
    def _join_url(base_url: str, path: str) -> str:
        return f"{base_url.rstrip('/')}/{path.lstrip('/')}"

    def matches(self, resource_identifier: str) -> bool:
        """Match resource name, connector name, or path tail."""
        if self.name is not None and resource_identifier == self.name:
            return True
        if self.resource_name is not None and resource_identifier == self.resource_name:
            return True
        path_tail = self.path.rstrip("/").rsplit("/", 1)[-1]
        return resource_identifier in {self.path, path_tail}

    def bound_source_kind(self) -> BoundSourceKind:
        return BoundSourceKind.API

    def build_api_config(
        self,
        *,
        base_url: str,
        auth: ApiAuth | None = None,
        default_headers: dict[str, str] | None = None,
        page_size_override: int | None = None,
    ) -> APIConfig:
        """Merge contract fields with runtime connection config into ``APIConfig``."""
        from graflo.data_source.api import APIConfig

        headers = dict(default_headers or {})
        headers.update(self.headers)

        pagination = self.pagination
        if pagination is not None and page_size_override is not None:
            pagination = pagination.model_copy(
                update={
                    "request": pagination.request.model_copy(
                        update={"page_size": page_size_override}
                    )
                }
            )

        return APIConfig(
            url=self._join_url(base_url, self.path),
            method=self.method,
            headers=headers,
            auth=auth,
            params=dict(self.params),
            body=self.body,
            timeout=self.timeout,
            retries=self.retries,
            retry_backoff_factor=self.retry_backoff_factor,
            retry_status_forcelist=list(self.retry_status_forcelist),
            verify=self.verify,
            pagination=pagination,
            row_annotations=dict(self.row_annotations),
            watermark=self.watermark,
        )

Attributes

body = None class-attribute instance-attribute
headers = Field(default_factory=dict) class-attribute instance-attribute
method = 'GET' class-attribute instance-attribute
pagination = None class-attribute instance-attribute
params = Field(default_factory=dict) class-attribute instance-attribute
parent = None class-attribute instance-attribute
path = Field(..., description='Relative API endpoint path') class-attribute instance-attribute
retries = 0 class-attribute instance-attribute
retry_backoff_factor = 0.1 class-attribute instance-attribute
retry_status_forcelist = Field(default_factory=lambda: [429, 500, 502, 503, 504]) class-attribute instance-attribute
timeout = None class-attribute instance-attribute
verify = True class-attribute instance-attribute
watermark = None class-attribute instance-attribute

Methods:

bound_source_kind()
Source code in graflo/architecture/contract/bindings/connectors.py
def bound_source_kind(self) -> BoundSourceKind:
    return BoundSourceKind.API
build_api_config(*, base_url, auth=None, default_headers=None, page_size_override=None)

Merge contract fields with runtime connection config into APIConfig.

Source code in graflo/architecture/contract/bindings/connectors.py
def build_api_config(
    self,
    *,
    base_url: str,
    auth: ApiAuth | None = None,
    default_headers: dict[str, str] | None = None,
    page_size_override: int | None = None,
) -> APIConfig:
    """Merge contract fields with runtime connection config into ``APIConfig``."""
    from graflo.data_source.api import APIConfig

    headers = dict(default_headers or {})
    headers.update(self.headers)

    pagination = self.pagination
    if pagination is not None and page_size_override is not None:
        pagination = pagination.model_copy(
            update={
                "request": pagination.request.model_copy(
                    update={"page_size": page_size_override}
                )
            }
        )

    return APIConfig(
        url=self._join_url(base_url, self.path),
        method=self.method,
        headers=headers,
        auth=auth,
        params=dict(self.params),
        body=self.body,
        timeout=self.timeout,
        retries=self.retries,
        retry_backoff_factor=self.retry_backoff_factor,
        retry_status_forcelist=list(self.retry_status_forcelist),
        verify=self.verify,
        pagination=pagination,
        row_annotations=dict(self.row_annotations),
        watermark=self.watermark,
    )
matches(resource_identifier)

Match resource name, connector name, or path tail.

Source code in graflo/architecture/contract/bindings/connectors.py
def matches(self, resource_identifier: str) -> bool:
    """Match resource name, connector name, or path tail."""
    if self.name is not None and resource_identifier == self.name:
        return True
    if self.resource_name is not None and resource_identifier == self.resource_name:
        return True
    path_tail = self.path.rstrip("/").rsplit("/", 1)[-1]
    return resource_identifier in {self.path, path_tail}
rendered(values)
Source code in graflo/architecture/contract/bindings/connectors.py
def rendered(self, values: Mapping[str, Any]) -> Self:
    return self.model_copy(
        update={
            "path": render_text(self.path, values, quote_values=True),
            "params": render_param(self.params, values),
            "body": None if self.body is None else render_body(self.body, values),
            "values": None,
        }
    )

ApiResponseStructure

Bases: ConfigBaseModel

Maps JSON response envelope fields to extraction and pagination signals.

Source code in graflo/architecture/contract/bindings/connectors.py
class ApiResponseStructure(ConfigBaseModel):
    """Maps JSON response envelope fields to extraction and pagination signals."""

    records_path: str | None = Field(
        default=None,
        description="Dot path to the record list (e.g. ``results``).",
    )
    total_count_path: str | None = Field(
        default=None,
        description="Dot path to total item count across all pages (e.g. ``count``).",
    )
    offset_path: str | None = Field(
        default=None,
        description="Dot path to echoed page start index (e.g. ``offset``).",
    )
    next_offset_path: str | None = Field(
        default=None,
        description=(
            "Dot path to server-provided next offset for the following request "
            "(e.g. ``next_offset``)."
        ),
    )
    has_more_path: str | None = Field(
        default=None,
        description="Dot path to a boolean more-pages flag (e.g. ``has_more``).",
    )
    cursor_path: str | None = Field(
        default=None,
        description="Dot path to the next opaque cursor token.",
    )
    ok_path: str | None = Field(
        default=None,
        description=(
            "Dot path to a success flag in every response body (e.g. ``ok``). "
            "A body where it is not ``true`` raises, even with HTTP status 200."
        ),
    )
    error_path: str | None = Field(
        default=None,
        description=(
            "Dot path to an error code in the response body (e.g. ``error``). "
            "Named in the raised error; a non-empty value raises on its own."
        ),
    )
    batch_metadata_paths: dict[str, str] = Field(
        default_factory=dict,
        description=(
            "Map row annotation keys to response dot paths "
            "(e.g. ``_batch_id: result_id``)."
        ),
    )
    auto_detect: bool = Field(
        default=False,
        description=(
            "When true, infer unset response paths from the first response body."
        ),
    )

Attributes

auto_detect = Field(default=False, description='When true, infer unset response paths from the first response body.') class-attribute instance-attribute
batch_metadata_paths = Field(default_factory=dict, description='Map row annotation keys to response dot paths (e.g. ``_batch_id: result_id``).') class-attribute instance-attribute
cursor_path = Field(default=None, description='Dot path to the next opaque cursor token.') class-attribute instance-attribute
error_path = Field(default=None, description='Dot path to an error code in the response body (e.g. ``error``). Named in the raised error; a non-empty value raises on its own.') class-attribute instance-attribute
has_more_path = Field(default=None, description='Dot path to a boolean more-pages flag (e.g. ``has_more``).') class-attribute instance-attribute
next_offset_path = Field(default=None, description='Dot path to server-provided next offset for the following request (e.g. ``next_offset``).') class-attribute instance-attribute
offset_path = Field(default=None, description='Dot path to echoed page start index (e.g. ``offset``).') class-attribute instance-attribute
ok_path = Field(default=None, description='Dot path to a success flag in every response body (e.g. ``ok``). A body where it is not ``true`` raises, even with HTTP status 200.') class-attribute instance-attribute
records_path = Field(default=None, description='Dot path to the record list (e.g. ``results``).') class-attribute instance-attribute
total_count_path = Field(default=None, description='Dot path to total item count across all pages (e.g. ``count``).') class-attribute instance-attribute

BindingsConfig

Bases: ConfigBaseModel

Declarative bindings contract (connectors and resource wiring).

Source code in graflo/architecture/contract/bindings/core.py
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
class BindingsConfig(ConfigBaseModel):
    """Declarative bindings contract (connectors and resource wiring)."""

    connector_templates: list[ConnectorTemplate] = Field(
        default_factory=list,
        description=(
            "Named connector defaults for ``base:`` expansion on connector entries."
        ),
    )
    conn_proxy: str | None = Field(
        default=None,
        description=(
            "Default runtime connection proxy for connectors without an explicit "
            "``connector_connection`` entry; file and push connectors take one "
            "only explicitly."
        ),
    )
    connectors: list[AnyConnector] = Field(default_factory=list)
    # Accept dict entries at init-time (see validators below).
    # Internally and at runtime, Graflo uses typed lists derived from these.
    resource_connector: list[ResourceConnectorBinding | dict[str, str]] = Field(
        default_factory=list
    )
    # Connector -> runtime endpoint config indirection (proxy by name).
    connector_connection: list[ConnectorConnectionBinding | dict[str, str]] = Field(
        default_factory=list
    )
    _resource_connector_typed: list[ResourceConnectorBinding] = PrivateAttr(
        default_factory=list
    )
    _connector_connection_typed: list[ConnectorConnectionBinding] = PrivateAttr(
        default_factory=list
    )
    _connectors_index: dict[str, ResourceConnector] = PrivateAttr(default_factory=dict)
    _connectors_name_index: dict[str, str] = PrivateAttr(default_factory=dict)
    _resource_to_connector_hashes: dict[str, list[str]] = PrivateAttr(
        default_factory=dict
    )
    _connector_to_conn_proxy: dict[str, str] = PrivateAttr(default_factory=dict)

    @property
    def connector_connection_bindings(
        self,
    ) -> list[ConnectorConnectionBinding]:
        # Expose typed entries for downstream components (type-checker friendly).
        return self._connector_connection_typed

    @model_validator(mode="before")
    @classmethod
    def _expand_connector_templates(cls, data: Any) -> Any:
        if not isinstance(data, dict):
            return data
        return _expand_connectors_from_templates(dict(data))

    @field_validator("connector_templates", mode="before")
    @classmethod
    def _coerce_connector_templates(cls, v: Any) -> list[ConnectorTemplate]:
        if v is None:
            return []
        if not isinstance(v, list):
            raise ValueError("connector_templates must be a list")
        coerced: list[ConnectorTemplate] = []
        for i, item in enumerate(v):
            if isinstance(item, ConnectorTemplate):
                coerced.append(item)
                continue
            if isinstance(item, dict):
                try:
                    coerced.append(ConnectorTemplate.model_validate(item))
                except Exception as e:
                    raise ValueError(
                        f"Invalid connector_templates entry at index {i}: {item!r}."
                    ) from e
                continue
            raise ValueError(
                f"Invalid connector_templates entry at index {i}: expected dict or "
                f"ConnectorTemplate, got {type(item).__name__}."
            )
        return coerced

    def _rebuild_indexes(self) -> None:
        self._connectors_index = {}
        self._connectors_name_index = {}
        for connector in self.connectors:
            existing = self._connectors_index.get(connector.hash)
            if existing is not None:
                raise ValueError(
                    "Connector hash collision detected for connectors "
                    f"'{type(existing).__name__}' and '{type(connector).__name__}' "
                    f"(hash='{connector.hash}')."
                )
            self._connectors_index[connector.hash] = connector

            if connector.name:
                existing_hash = self._connectors_name_index.get(connector.name)
                if existing_hash is not None and existing_hash != connector.hash:
                    raise ValueError(
                        "Connector names must be unique when provided. "
                        f"Duplicate connector name '{connector.name}'."
                    )
                self._connectors_name_index[connector.name] = connector.hash

    def _append_resource_connector_hash(
        self, resource_name: str, connector_hash: str
    ) -> None:
        """Append *connector_hash* for *resource_name* if not already present (order kept)."""
        bucket = self._resource_to_connector_hashes.setdefault(resource_name, [])
        if connector_hash not in bucket:
            bucket.append(connector_hash)

    @field_validator("resource_connector", mode="before")
    @classmethod
    def _coerce_resource_connector_entries(
        cls, v: Any
    ) -> list[ResourceConnectorBinding]:
        if v is None:
            return []
        if not isinstance(v, list):
            raise ValueError(
                "resource_connector must be a list of {resource, connector} entries"
            )

        coerced: list[ResourceConnectorBinding] = []
        for i, item in enumerate(v):
            if isinstance(item, ResourceConnectorBinding):
                coerced.append(item)
                continue

            if isinstance(item, dict):
                missing = [k for k in ("resource", "connector") if k not in item]
                if missing:
                    raise ValueError(
                        f"Invalid resource_connector entry at index {i}: missing required keys {missing}. "
                        "Expected keys: ['resource', 'connector']."
                    )

                try:
                    coerced.append(ResourceConnectorBinding.model_validate(item))
                except Exception as e:
                    # Keep the message concise and contextual; nested pydantic
                    # errors can be noisy for config authors.
                    raise ValueError(
                        f"Invalid resource_connector entry at index {i}: {item!r}."
                    ) from e
                continue

            raise ValueError(
                f"Invalid resource_connector entry at index {i}: expected dict or "
                f"ResourceConnectorBinding, got {type(item).__name__}."
            )

        return coerced

    @field_validator("connector_connection", mode="before")
    @classmethod
    def _coerce_connector_connection_entries(
        cls, v: Any
    ) -> list[ConnectorConnectionBinding]:
        if v is None:
            return []
        if not isinstance(v, list):
            raise ValueError(
                "connector_connection must be a list of {connector, conn_proxy} entries"
            )

        coerced: list[ConnectorConnectionBinding] = []
        for i, item in enumerate(v):
            if isinstance(item, ConnectorConnectionBinding):
                coerced.append(item)
                continue

            if isinstance(item, dict):
                missing = [k for k in ("connector", "conn_proxy") if k not in item]
                if missing:
                    raise ValueError(
                        f"Invalid connector_connection entry at index {i}: missing required keys {missing}. "
                        "Expected keys: ['connector', 'conn_proxy']."
                    )
                try:
                    coerced.append(ConnectorConnectionBinding.model_validate(item))
                except Exception as e:
                    raise ValueError(
                        f"Invalid connector_connection entry at index {i}: {item!r}."
                    ) from e
                continue

            raise ValueError(
                f"Invalid connector_connection entry at index {i}: expected dict or "
                f"ConnectorConnectionBinding, got {type(item).__name__}."
            )

        return coerced

    @staticmethod
    def default_connector_name(connector: ResourceConnector) -> str:
        if connector.name:
            return connector.name
        if isinstance(connector, FileConnector):
            return connector.regex or str(connector.sub_path)
        if isinstance(connector, TableConnector):
            return connector.table_name
        if isinstance(connector, SparqlConnector):
            return connector.rdf_class
        if isinstance(connector, APIConnector):
            return connector.path
        if isinstance(connector, KafkaConnector):
            return connector.name or ",".join(connector.topics)
        if isinstance(connector, PushConnector):
            return connector.name
        raise TypeError(f"Unsupported connector type: {type(connector)!r}")

    @model_validator(mode="after")
    def _populate_resource_connector(self) -> Self:
        self._rebuild_indexes()
        self._resource_to_connector_hashes = {}

        # Create typed views so internal code never has to handle dicts.
        self._resource_connector_typed = [
            ResourceConnectorBinding.model_validate(m) if isinstance(m, dict) else m
            for m in self.resource_connector
        ]
        self._connector_connection_typed = [
            ConnectorConnectionBinding.model_validate(m) if isinstance(m, dict) else m
            for m in self.connector_connection
        ]

        for connector in self.connectors:
            if connector.resource_name is None:
                continue
            self._append_resource_connector_hash(
                connector.resource_name, connector.hash
            )

        for mapping in self._resource_connector_typed:
            connector_hash = self._connectors_name_index.get(mapping.connector)
            if connector_hash is None:
                if mapping.connector in self._connectors_index:
                    connector_hash = mapping.connector
                else:
                    raise ValueError(
                        f"resource_connector references unknown connector '{mapping.connector}' "
                        f"for resource '{mapping.resource}'."
                    )
            self._append_resource_connector_hash(mapping.resource, connector_hash)
        self._rebuild_connector_to_conn_proxy()
        self._check_parents()
        return self

    def get_parent_connector(self, connector: APIConnector) -> APIConnector | None:
        """Return the API connector whose records *connector* makes one request per."""
        if connector.parent is None:
            return None
        parent_hash = self._connectors_name_index.get(connector.parent.connector)
        parent = self._connectors_index.get(parent_hash) if parent_hash else None
        return parent if isinstance(parent, APIConnector) else None

    def _check_parents(self) -> None:
        """Refuse a parent that is unknown, not an API connector, or in a cycle."""
        for connector in self.connectors:
            if not isinstance(connector, APIConnector) or connector.parent is None:
                continue
            seen = {connector.hash}
            current: APIConnector = connector
            while current.parent is not None:
                ref = current.parent.connector
                parent_hash = self._connectors_name_index.get(ref)
                if parent_hash is None:
                    raise ValueError(
                        f"API connector '{current.name or current.path}' names an "
                        f"unknown parent connector '{ref}'"
                    )
                parent = self._connectors_index[parent_hash]
                if not isinstance(parent, APIConnector):
                    raise ValueError(
                        f"Parent connector '{ref}' of '{current.name or current.path}' "
                        "is not an API connector"
                    )
                if parent.hash in seen:
                    raise ValueError(
                        f"API connector parents form a cycle through '{ref}'"
                    )
                seen.add(parent.hash)
                current = parent

    def _resolve_connector_ref_to_hash(self, connector_ref: str) -> str:
        """Resolve a connector reference to its canonical connector hash.

        Allowed references:
        - ``connector.hash`` (canonical internal id), or
        - ``connector.name`` (when a name is provided / auto-filled).

        Ingestion resource names are not valid connector references (a resource
        may map to multiple connectors).
        """
        if connector_ref in self._connectors_index:
            return connector_ref
        resolved_hash = self._connectors_name_index.get(connector_ref)
        if resolved_hash is None:
            raise ValueError(f"Unknown connector reference '{connector_ref}'")
        return resolved_hash

    def _rebuild_connector_to_conn_proxy(self) -> None:
        self._connector_to_conn_proxy = {}
        for mapping in self._connector_connection_typed:
            connector_hash = self._resolve_connector_ref_to_hash(mapping.connector)
            existing = self._connector_to_conn_proxy.get(connector_hash)
            if existing is not None and existing != mapping.conn_proxy:
                raise ValueError(
                    "Conflicting conn_proxy mapping for connector "
                    f"'{connector_hash}' (existing='{existing}', new='{mapping.conn_proxy}')."
                )
            self._connector_to_conn_proxy[connector_hash] = mapping.conn_proxy

        if self.conn_proxy is not None:
            for connector in self.connectors:
                if (
                    connector.takes_default_conn_proxy
                    and connector.hash not in self._connector_to_conn_proxy
                ):
                    self._connector_to_conn_proxy[connector.hash] = self.conn_proxy

    def get_conn_proxy_for_connector(self, connector: AnyConnector) -> str | None:
        """Return the mapped runtime proxy name for a given connector."""
        return self._connector_to_conn_proxy.get(connector.hash)

    def conn_proxies(self) -> dict[str, list[AnyConnector]]:
        """Every ``conn_proxy`` label in use, with the connectors that name it.

        Labels come in sorted order; connectors in declaration order.
        """
        by_proxy: dict[str, list[AnyConnector]] = {}
        for connector in self.connectors:
            proxy = self._connector_to_conn_proxy.get(connector.hash)
            if proxy is not None:
                by_proxy.setdefault(proxy, []).append(connector)
        return dict(sorted(by_proxy.items()))

    def resolve_connector_refs_to_hashes(self, connector_refs: list[str]) -> set[str]:
        """Resolve connector names or hashes to canonical connector hashes."""
        return {self._resolve_connector_ref_to_hash(ref) for ref in connector_refs}

    def get_connectors_for_resource(self, resource_name: str) -> list[AnyConnector]:
        """Return connectors bound to *resource_name*, in binding order (unique by hash)."""
        result: list[AnyConnector] = []
        for h in self._resource_to_connector_hashes.get(resource_name, []):
            c = self._connectors_index.get(h)
            if isinstance(
                c,
                (
                    TableConnector,
                    FileConnector,
                    SparqlConnector,
                    APIConnector,
                    KafkaConnector,
                    PushConnector,
                ),
            ):
                result.append(c)
        return result

    @classmethod
    def from_dict(cls, data: dict[str, Any] | list[Any]) -> Self:
        if isinstance(data, list):
            raise ValueError(
                "Bindings.from_dict expects a mapping with 'connectors' and optional "
                "'resource_connector'. List-style connector payloads are not supported."
            )
        legacy_keys = {
            "postgres_connections",
            "table_connectors",
            "file_connectors",
            "sparql_connectors",
        }
        found_legacy = sorted(k for k in legacy_keys if k in data)
        if found_legacy:
            raise ValueError(
                "Legacy Bindings init keys are not supported. "
                f"Unsupported keys: {', '.join(found_legacy)}."
            )
        return cls.model_validate(data)

Attributes

conn_proxy = Field(default=None, description='Default runtime connection proxy for connectors without an explicit ``connector_connection`` entry; file and push connectors take one only explicitly.') class-attribute instance-attribute
connector_connection = Field(default_factory=list) class-attribute instance-attribute
connector_connection_bindings property
connector_templates = Field(default_factory=list, description='Named connector defaults for ``base:`` expansion on connector entries.') class-attribute instance-attribute
connectors = Field(default_factory=list) class-attribute instance-attribute
resource_connector = Field(default_factory=list) class-attribute instance-attribute

Methods:

conn_proxies()

Every conn_proxy label in use, with the connectors that name it.

Labels come in sorted order; connectors in declaration order.

Source code in graflo/architecture/contract/bindings/core.py
def conn_proxies(self) -> dict[str, list[AnyConnector]]:
    """Every ``conn_proxy`` label in use, with the connectors that name it.

    Labels come in sorted order; connectors in declaration order.
    """
    by_proxy: dict[str, list[AnyConnector]] = {}
    for connector in self.connectors:
        proxy = self._connector_to_conn_proxy.get(connector.hash)
        if proxy is not None:
            by_proxy.setdefault(proxy, []).append(connector)
    return dict(sorted(by_proxy.items()))
default_connector_name(connector) staticmethod
Source code in graflo/architecture/contract/bindings/core.py
@staticmethod
def default_connector_name(connector: ResourceConnector) -> str:
    if connector.name:
        return connector.name
    if isinstance(connector, FileConnector):
        return connector.regex or str(connector.sub_path)
    if isinstance(connector, TableConnector):
        return connector.table_name
    if isinstance(connector, SparqlConnector):
        return connector.rdf_class
    if isinstance(connector, APIConnector):
        return connector.path
    if isinstance(connector, KafkaConnector):
        return connector.name or ",".join(connector.topics)
    if isinstance(connector, PushConnector):
        return connector.name
    raise TypeError(f"Unsupported connector type: {type(connector)!r}")
from_dict(data) classmethod
Source code in graflo/architecture/contract/bindings/core.py
@classmethod
def from_dict(cls, data: dict[str, Any] | list[Any]) -> Self:
    if isinstance(data, list):
        raise ValueError(
            "Bindings.from_dict expects a mapping with 'connectors' and optional "
            "'resource_connector'. List-style connector payloads are not supported."
        )
    legacy_keys = {
        "postgres_connections",
        "table_connectors",
        "file_connectors",
        "sparql_connectors",
    }
    found_legacy = sorted(k for k in legacy_keys if k in data)
    if found_legacy:
        raise ValueError(
            "Legacy Bindings init keys are not supported. "
            f"Unsupported keys: {', '.join(found_legacy)}."
        )
    return cls.model_validate(data)
get_conn_proxy_for_connector(connector)

Return the mapped runtime proxy name for a given connector.

Source code in graflo/architecture/contract/bindings/core.py
def get_conn_proxy_for_connector(self, connector: AnyConnector) -> str | None:
    """Return the mapped runtime proxy name for a given connector."""
    return self._connector_to_conn_proxy.get(connector.hash)
get_connectors_for_resource(resource_name)

Return connectors bound to resource_name, in binding order (unique by hash).

Source code in graflo/architecture/contract/bindings/core.py
def get_connectors_for_resource(self, resource_name: str) -> list[AnyConnector]:
    """Return connectors bound to *resource_name*, in binding order (unique by hash)."""
    result: list[AnyConnector] = []
    for h in self._resource_to_connector_hashes.get(resource_name, []):
        c = self._connectors_index.get(h)
        if isinstance(
            c,
            (
                TableConnector,
                FileConnector,
                SparqlConnector,
                APIConnector,
                KafkaConnector,
                PushConnector,
            ),
        ):
            result.append(c)
    return result
get_parent_connector(connector)

Return the API connector whose records connector makes one request per.

Source code in graflo/architecture/contract/bindings/core.py
def get_parent_connector(self, connector: APIConnector) -> APIConnector | None:
    """Return the API connector whose records *connector* makes one request per."""
    if connector.parent is None:
        return None
    parent_hash = self._connectors_name_index.get(connector.parent.connector)
    parent = self._connectors_index.get(parent_hash) if parent_hash else None
    return parent if isinstance(parent, APIConnector) else None
resolve_connector_refs_to_hashes(connector_refs)

Resolve connector names or hashes to canonical connector hashes.

Source code in graflo/architecture/contract/bindings/core.py
def resolve_connector_refs_to_hashes(self, connector_refs: list[str]) -> set[str]:
    """Resolve connector names or hashes to canonical connector hashes."""
    return {self._resolve_connector_ref_to_hash(ref) for ref in connector_refs}

BindingsRegistry

Bases: BindingsConfig

Mutable bindings registry for programmatic connector updates.

Source code in graflo/architecture/contract/bindings/core.py
class BindingsRegistry(BindingsConfig):
    """Mutable bindings registry for programmatic connector updates."""

    def bind_connector_to_conn_proxy(
        self,
        connector: AnyConnector,
        conn_proxy: str,
    ) -> None:
        """Bind a connector to a non-secret runtime proxy name.

        Uses ``connector.name`` when available, falling back to ``connector.hash``.
        """
        # Ensure indexes include the connector and that a default name is set.
        if connector.hash not in self._connectors_index:
            self.add_connector(connector)
        # Pick a contract reference string that's stable and user-friendly.
        connector_ref = connector.name or connector.hash

        # Ensure uniqueness by connector.hash (not by ref-string).
        connector_hash = connector.hash
        existing_idx: int | None = None
        for i, m in enumerate(self._connector_connection_typed):
            try:
                if self._resolve_connector_ref_to_hash(m.connector) == connector_hash:
                    existing_idx = i
                    break
            except ValueError:
                continue

        if existing_idx is None:
            self._connector_connection_typed.append(
                ConnectorConnectionBinding(
                    connector=connector_ref, conn_proxy=conn_proxy
                )
            )
        else:
            self._connector_connection_typed[existing_idx] = ConnectorConnectionBinding(
                connector=connector_ref, conn_proxy=conn_proxy
            )
        # Keep the public contract field in sync for serialization / downstream.
        self.connector_connection = list(self._connector_connection_typed)

        self._rebuild_connector_to_conn_proxy()

    def apply_connector_update(self, update: ConnectorUpdate) -> None:
        """Patch a connector in-place in this binding (re-hashes and reindexes).

        Uses ``model_validate`` on merged data so connector validators (including
        hash recomputation) run; ``model_copy(update=...)`` would not re-run them.
        """
        connector_hash = self._resolve_connector_ref_to_hash(update.connector)
        old = self._connectors_index[connector_hash]
        patch = update.as_patch()
        if not patch:
            return
        merged = old.model_dump(mode="python")
        merged.update(patch)
        new = cast(
            AnyConnector,
            old.__class__.model_validate(merged),
        )
        self.replace_connector(old, new)

    def replace_connector(
        self,
        old: ResourceConnector | str,
        new: AnyConnector,
    ) -> None:
        """Swap *old* for *new*, preserving resource wiring and conn_proxy by hash."""
        old_hash = (
            old.hash
            if isinstance(old, ResourceConnector)
            else self._resolve_connector_ref_to_hash(old)
        )
        if old_hash not in self._connectors_index:
            raise KeyError(f"Connector not found for hash={old_hash!r}")

        old_connector = self._connectors_index[old_hash]
        if new.name is None and old_connector.name is not None:
            object.__setattr__(new, "name", old_connector.name)

        replaced = False
        for idx, c in enumerate(self.connectors):
            if c.hash == old_hash:
                self.connectors[idx] = new
                replaced = True
                break
        if not replaced:
            raise KeyError(
                f"Connector with hash={old_hash!r} not found in bindings.connectors list"
            )

        new_hash = new.hash
        for hashes in self._resource_to_connector_hashes.values():
            for i, h in enumerate(hashes):
                if h == old_hash:
                    hashes[i] = new_hash

        old_proxy = self._connector_to_conn_proxy.pop(old_hash, None)
        self._rebuild_indexes()
        if old_proxy is not None:
            self._connector_to_conn_proxy[new_hash] = old_proxy

    def add_connector(
        self,
        connector: AnyConnector,
    ) -> None:
        if connector.name is None:
            object.__setattr__(
                connector, "name", self.default_connector_name(connector)
            )
        existing_name_hash = None
        if connector.name:
            existing_name_hash = self._connectors_name_index.get(connector.name)
        if (
            connector.name
            and existing_name_hash is not None
            and existing_name_hash != connector.hash
        ):
            raise ValueError(
                "Connector names must be unique when provided. "
                f"Duplicate connector name '{connector.name}'."
            )

        if connector.hash in self._connectors_index:
            old_connector = self._connectors_index[connector.hash]
            for idx, existing in enumerate(self.connectors):
                if existing is old_connector:
                    self.connectors[idx] = connector
                    break
        else:
            self.connectors.append(connector)
        self._rebuild_indexes()
        if connector.resource_name is not None:
            self._append_resource_connector_hash(
                connector.resource_name, connector.hash
            )

    def bind_resource(
        self,
        resource_name: str,
        connector: AnyConnector,
    ) -> None:
        if connector.hash not in self._connectors_index:
            raise KeyError(f"Connector not found for hash='{connector.hash}'")
        self._append_resource_connector_hash(resource_name, connector.hash)
        connector_name = connector.name or self.default_connector_name(connector)
        self._resource_connector_typed.append(
            ResourceConnectorBinding(
                resource=resource_name,
                connector=connector_name,
            )
        )
        # Keep the public contract field in sync for serialization / downstream.
        self.resource_connector = list(self._resource_connector_typed)

Methods:

add_connector(connector)
Source code in graflo/architecture/contract/bindings/core.py
def add_connector(
    self,
    connector: AnyConnector,
) -> None:
    if connector.name is None:
        object.__setattr__(
            connector, "name", self.default_connector_name(connector)
        )
    existing_name_hash = None
    if connector.name:
        existing_name_hash = self._connectors_name_index.get(connector.name)
    if (
        connector.name
        and existing_name_hash is not None
        and existing_name_hash != connector.hash
    ):
        raise ValueError(
            "Connector names must be unique when provided. "
            f"Duplicate connector name '{connector.name}'."
        )

    if connector.hash in self._connectors_index:
        old_connector = self._connectors_index[connector.hash]
        for idx, existing in enumerate(self.connectors):
            if existing is old_connector:
                self.connectors[idx] = connector
                break
    else:
        self.connectors.append(connector)
    self._rebuild_indexes()
    if connector.resource_name is not None:
        self._append_resource_connector_hash(
            connector.resource_name, connector.hash
        )
apply_connector_update(update)

Patch a connector in-place in this binding (re-hashes and reindexes).

Uses model_validate on merged data so connector validators (including hash recomputation) run; model_copy(update=...) would not re-run them.

Source code in graflo/architecture/contract/bindings/core.py
def apply_connector_update(self, update: ConnectorUpdate) -> None:
    """Patch a connector in-place in this binding (re-hashes and reindexes).

    Uses ``model_validate`` on merged data so connector validators (including
    hash recomputation) run; ``model_copy(update=...)`` would not re-run them.
    """
    connector_hash = self._resolve_connector_ref_to_hash(update.connector)
    old = self._connectors_index[connector_hash]
    patch = update.as_patch()
    if not patch:
        return
    merged = old.model_dump(mode="python")
    merged.update(patch)
    new = cast(
        AnyConnector,
        old.__class__.model_validate(merged),
    )
    self.replace_connector(old, new)
bind_connector_to_conn_proxy(connector, conn_proxy)

Bind a connector to a non-secret runtime proxy name.

Uses connector.name when available, falling back to connector.hash.

Source code in graflo/architecture/contract/bindings/core.py
def bind_connector_to_conn_proxy(
    self,
    connector: AnyConnector,
    conn_proxy: str,
) -> None:
    """Bind a connector to a non-secret runtime proxy name.

    Uses ``connector.name`` when available, falling back to ``connector.hash``.
    """
    # Ensure indexes include the connector and that a default name is set.
    if connector.hash not in self._connectors_index:
        self.add_connector(connector)
    # Pick a contract reference string that's stable and user-friendly.
    connector_ref = connector.name or connector.hash

    # Ensure uniqueness by connector.hash (not by ref-string).
    connector_hash = connector.hash
    existing_idx: int | None = None
    for i, m in enumerate(self._connector_connection_typed):
        try:
            if self._resolve_connector_ref_to_hash(m.connector) == connector_hash:
                existing_idx = i
                break
        except ValueError:
            continue

    if existing_idx is None:
        self._connector_connection_typed.append(
            ConnectorConnectionBinding(
                connector=connector_ref, conn_proxy=conn_proxy
            )
        )
    else:
        self._connector_connection_typed[existing_idx] = ConnectorConnectionBinding(
            connector=connector_ref, conn_proxy=conn_proxy
        )
    # Keep the public contract field in sync for serialization / downstream.
    self.connector_connection = list(self._connector_connection_typed)

    self._rebuild_connector_to_conn_proxy()
bind_resource(resource_name, connector)
Source code in graflo/architecture/contract/bindings/core.py
def bind_resource(
    self,
    resource_name: str,
    connector: AnyConnector,
) -> None:
    if connector.hash not in self._connectors_index:
        raise KeyError(f"Connector not found for hash='{connector.hash}'")
    self._append_resource_connector_hash(resource_name, connector.hash)
    connector_name = connector.name or self.default_connector_name(connector)
    self._resource_connector_typed.append(
        ResourceConnectorBinding(
            resource=resource_name,
            connector=connector_name,
        )
    )
    # Keep the public contract field in sync for serialization / downstream.
    self.resource_connector = list(self._resource_connector_typed)
replace_connector(old, new)

Swap old for new, preserving resource wiring and conn_proxy by hash.

Source code in graflo/architecture/contract/bindings/core.py
def replace_connector(
    self,
    old: ResourceConnector | str,
    new: AnyConnector,
) -> None:
    """Swap *old* for *new*, preserving resource wiring and conn_proxy by hash."""
    old_hash = (
        old.hash
        if isinstance(old, ResourceConnector)
        else self._resolve_connector_ref_to_hash(old)
    )
    if old_hash not in self._connectors_index:
        raise KeyError(f"Connector not found for hash={old_hash!r}")

    old_connector = self._connectors_index[old_hash]
    if new.name is None and old_connector.name is not None:
        object.__setattr__(new, "name", old_connector.name)

    replaced = False
    for idx, c in enumerate(self.connectors):
        if c.hash == old_hash:
            self.connectors[idx] = new
            replaced = True
            break
    if not replaced:
        raise KeyError(
            f"Connector with hash={old_hash!r} not found in bindings.connectors list"
        )

    new_hash = new.hash
    for hashes in self._resource_to_connector_hashes.values():
        for i, h in enumerate(hashes):
            if h == old_hash:
                hashes[i] = new_hash

    old_proxy = self._connector_to_conn_proxy.pop(old_hash, None)
    self._rebuild_indexes()
    if old_proxy is not None:
        self._connector_to_conn_proxy[new_hash] = old_proxy

BoundSourceKind

Bases: BaseEnum

Physical source modality for a bound connector (how rows are retrieved).

This describes the connector-backed access pattern, not the abstract ingestion resource. File format (CSV, JSON, etc.) is chosen by the loader from file extensions.

Attributes:

Name Type Description
FILE

File-based connector (directory + pattern or paths).

SQL_TABLE

SQL table / database-backed connector.

SPARQL

SPARQL / RDF connector (endpoint or local RDF via rdflib).

API

REST API connector (path + pagination on a runtime base URL).

KAFKA

Kafka topic consumer (topics + group_id on a runtime bootstrap).

PUSH

Records handed to the resource by the caller; no address.

Source code in graflo/architecture/contract/bindings/connectors.py
class BoundSourceKind(BaseEnum):
    """Physical source modality for a bound connector (how rows are retrieved).

    This describes the connector-backed access pattern, not the abstract
    ingestion resource. File format (CSV, JSON, etc.) is chosen by the loader
    from file extensions.

    Attributes:
        FILE: File-based connector (directory + pattern or paths).
        SQL_TABLE: SQL table / database-backed connector.
        SPARQL: SPARQL / RDF connector (endpoint or local RDF via rdflib).
        API: REST API connector (path + pagination on a runtime base URL).
        KAFKA: Kafka topic consumer (topics + group_id on a runtime bootstrap).
        PUSH: Records handed to the resource by the caller; no address.
    """

    FILE = "file"
    SQL_TABLE = "sql_table"
    SPARQL = "sparql"
    API = "api"
    KAFKA = "kafka"
    PUSH = "push"

Attributes

API = 'api' class-attribute instance-attribute
FILE = 'file' class-attribute instance-attribute
KAFKA = 'kafka' class-attribute instance-attribute
PUSH = 'push' class-attribute instance-attribute
SPARQL = 'sparql' class-attribute instance-attribute
SQL_TABLE = 'sql_table' class-attribute instance-attribute

ColumnTimeFilter

Bases: ConfigBaseModel

Predicate on a single date/time column (SQL-friendly via :class:FilterExpression).

Source code in graflo/architecture/contract/bindings/column_time_filter.py
class ColumnTimeFilter(ConfigBaseModel):
    """Predicate on a single date/time column (SQL-friendly via :class:`FilterExpression`)."""

    column: str = Field(..., description="Column name for the time predicate.")
    start: str | None = Field(
        default=None,
        description="Lower bound (ISO date or datetime). Interpreted with start_inclusive.",
    )
    end: str | None = Field(
        default=None,
        description="Upper bound (ISO date or datetime). Interpreted with end_inclusive.",
    )
    interval: str | None = Field(
        default=None,
        description='Pandas timedelta string (e.g. "7D", "2H"); requires start; '
        "defines [start, start + interval). Mutually exclusive with end.",
    )
    not_equals: str | None = Field(
        default=None,
        description="If set, render column != value. Mutually exclusive with start/end/interval.",
    )
    start_inclusive: bool = Field(
        default=True,
        description="If True (default), lower bound uses >= when start is set.",
    )
    end_inclusive: bool = Field(
        default=False,
        description="If False (default), upper bound uses < when end is set.",
    )

    @model_validator(mode="after")
    def _check_shape(self) -> Self:
        has_pred = any(
            v is not None
            for v in (self.start, self.end, self.interval, self.not_equals)
        )
        if not has_pred:
            # Column-only hint (e.g. introspection sets datetime column without a WHERE).
            return self
        if self.not_equals is not None:
            if any(v is not None for v in (self.start, self.end, self.interval)):
                raise ValueError(
                    "not_equals cannot be combined with start, end, or interval"
                )
            return self
        if self.interval is not None:
            if self.start is None:
                raise ValueError("interval requires start")
            if self.end is not None:
                raise ValueError(
                    "interval cannot be combined with end; use start + interval"
                )
            self._validated_timedelta()
            return self
        if self.start is None and self.end is None:
            raise ValueError(
                "ColumnTimeFilter requires at least one of: start, end, interval, not_equals"
            )
        return self

    def _validated_timedelta(self) -> pd.Timedelta:
        assert self.interval is not None
        try:
            td = pd.Timedelta(self.interval)
        except (ValueError, TypeError) as e:
            raise ValueError(
                f"Invalid pandas timedelta string for interval: {self.interval!r}"
            ) from e
        if pd.isna(td):
            raise ValueError(
                f"Invalid pandas timedelta string for interval: {self.interval!r}"
            )
        return td

    def _interval_upper_literal(self) -> str:
        assert self.start is not None and self.interval is not None
        start_dt, date_only = parse_iso_date_or_datetime(self.start)
        delta = self._validated_timedelta().to_pytimedelta()
        end_dt = start_dt + delta
        upper_date_only = date_only and (end_dt.time() == time.min)
        return format_sql_literal(end_dt, upper_date_only)

    def _lower_literal(self) -> str:
        assert self.start is not None
        dt, date_only = parse_iso_date_or_datetime(self.start)
        return format_sql_literal(dt, date_only)

    def _upper_literal(self) -> str:
        assert self.end is not None
        dt, date_only = parse_iso_date_or_datetime(self.end)
        return format_sql_literal(dt, date_only)

    def as_filter_expression(self) -> FilterExpression | None:
        """Return a single composite AND of leaves, or None if misconfigured."""
        from graflo.filter.onto import (
            ComparisonOperator,
            FilterExpression,
            LogicalOperator,
        )

        if self.not_equals is not None:
            return FilterExpression(
                kind="leaf",
                field=self.column,
                cmp_operator=ComparisonOperator.NEQ,
                value=[self.not_equals],
            )

        leaves: list[FilterExpression] = []

        if self.interval is not None:
            assert self.start is not None
            # Half-open window [start, start + interval).
            leaves.append(
                FilterExpression(
                    kind="leaf",
                    field=self.column,
                    cmp_operator=ComparisonOperator.GE,
                    value=[self._lower_literal()],
                )
            )
            leaves.append(
                FilterExpression(
                    kind="leaf",
                    field=self.column,
                    cmp_operator=ComparisonOperator.LT,
                    value=[self._interval_upper_literal()],
                )
            )
        else:
            if self.start is not None:
                lower_op = (
                    ComparisonOperator.GE
                    if self.start_inclusive
                    else ComparisonOperator.GT
                )
                leaves.append(
                    FilterExpression(
                        kind="leaf",
                        field=self.column,
                        cmp_operator=lower_op,
                        value=[self._lower_literal()],
                    )
                )
            if self.end is not None:
                upper_op = (
                    ComparisonOperator.LE
                    if self.end_inclusive
                    else ComparisonOperator.LT
                )
                leaves.append(
                    FilterExpression(
                        kind="leaf",
                        field=self.column,
                        cmp_operator=upper_op,
                        value=[self._upper_literal()],
                    )
                )

        if not leaves:
            return None
        if len(leaves) == 1:
            return leaves[0]
        return FilterExpression(
            kind="composite",
            operator=LogicalOperator.AND,
            deps=leaves,
        )

Attributes

column = Field(..., description='Column name for the time predicate.') class-attribute instance-attribute
end = Field(default=None, description='Upper bound (ISO date or datetime). Interpreted with end_inclusive.') class-attribute instance-attribute
end_inclusive = Field(default=False, description='If False (default), upper bound uses < when end is set.') class-attribute instance-attribute
interval = Field(default=None, description='Pandas timedelta string (e.g. "7D", "2H"); requires start; defines [start, start + interval). Mutually exclusive with end.') class-attribute instance-attribute
not_equals = Field(default=None, description='If set, render column != value. Mutually exclusive with start/end/interval.') class-attribute instance-attribute
start = Field(default=None, description='Lower bound (ISO date or datetime). Interpreted with start_inclusive.') class-attribute instance-attribute
start_inclusive = Field(default=True, description='If True (default), lower bound uses >= when start is set.') class-attribute instance-attribute

Methods:

as_filter_expression()

Return a single composite AND of leaves, or None if misconfigured.

Source code in graflo/architecture/contract/bindings/column_time_filter.py
def as_filter_expression(self) -> FilterExpression | None:
    """Return a single composite AND of leaves, or None if misconfigured."""
    from graflo.filter.onto import (
        ComparisonOperator,
        FilterExpression,
        LogicalOperator,
    )

    if self.not_equals is not None:
        return FilterExpression(
            kind="leaf",
            field=self.column,
            cmp_operator=ComparisonOperator.NEQ,
            value=[self.not_equals],
        )

    leaves: list[FilterExpression] = []

    if self.interval is not None:
        assert self.start is not None
        # Half-open window [start, start + interval).
        leaves.append(
            FilterExpression(
                kind="leaf",
                field=self.column,
                cmp_operator=ComparisonOperator.GE,
                value=[self._lower_literal()],
            )
        )
        leaves.append(
            FilterExpression(
                kind="leaf",
                field=self.column,
                cmp_operator=ComparisonOperator.LT,
                value=[self._interval_upper_literal()],
            )
        )
    else:
        if self.start is not None:
            lower_op = (
                ComparisonOperator.GE
                if self.start_inclusive
                else ComparisonOperator.GT
            )
            leaves.append(
                FilterExpression(
                    kind="leaf",
                    field=self.column,
                    cmp_operator=lower_op,
                    value=[self._lower_literal()],
                )
            )
        if self.end is not None:
            upper_op = (
                ComparisonOperator.LE
                if self.end_inclusive
                else ComparisonOperator.LT
            )
            leaves.append(
                FilterExpression(
                    kind="leaf",
                    field=self.column,
                    cmp_operator=upper_op,
                    value=[self._upper_literal()],
                )
            )

    if not leaves:
        return None
    if len(leaves) == 1:
        return leaves[0]
    return FilterExpression(
        kind="composite",
        operator=LogicalOperator.AND,
        deps=leaves,
    )

ConnectorTemplate

Bases: ConfigBaseModel

Named connector defaults referenced by base on connector entries.

Template-only metadata (not copied onto connectors): name, conn_proxy. All other fields are merged into expanded connectors (dict fields such as params are deep-merged; scalars and blocks like pagination are replaced when the connector entry provides them explicitly).

Source code in graflo/architecture/contract/bindings/core.py
class ConnectorTemplate(ConfigBaseModel):
    """Named connector defaults referenced by ``base`` on connector entries.

    Template-only metadata (not copied onto connectors): ``name``, ``conn_proxy``.
    All other fields are merged into expanded connectors (dict fields such as
    ``params`` are deep-merged; scalars and blocks like ``pagination`` are replaced
    when the connector entry provides them explicitly).
    """

    model_config = ConfigDict(extra="allow")

    name: str = Field(
        ..., description="Template name referenced by connector ``base``."
    )
    conn_proxy: str | None = Field(
        default=None,
        description=(
            "Template metadata: auto-wires ``connector_connection`` for expanded "
            "connectors that declare ``name``."
        ),
    )

Attributes

conn_proxy = Field(default=None, description='Template metadata: auto-wires ``connector_connection`` for expanded connectors that declare ``name``.') class-attribute instance-attribute
model_config = ConfigDict(extra='allow') class-attribute instance-attribute
name = Field(..., description='Template name referenced by connector ``base``.') class-attribute instance-attribute

ConnectorUpdate

Bases: ConfigBaseModel

Patch payload for an existing connector (applied after manifest load).

Only connector is a fixed field; any other keys are captured as extras and merged into the resolved connector via model_validate (so validators, including hash recomputation, run). New connector types and fields do not require changes to this model. Not part of the stored GraphManifest; load from a separate file or build in code, then call Bindings.apply_connector_update.

Attributes:

Name Type Description
connector str

Connector name or hash (same resolution as bindings).

Source code in graflo/architecture/contract/bindings/connectors.py
class ConnectorUpdate(ConfigBaseModel):
    """Patch payload for an existing connector (applied after manifest load).

    Only ``connector`` is a fixed field; any other keys are captured as extras and
    merged into the resolved connector via ``model_validate`` (so validators,
    including hash recomputation, run). New connector types and fields do not
    require changes to this model. Not part of the stored ``GraphManifest``;
    load from a separate file or build in code, then call
    ``Bindings.apply_connector_update``.

    Attributes:
        connector: Connector ``name`` or ``hash`` (same resolution as bindings).
    """

    model_config = ConfigDict(extra="allow")

    connector: str = Field(
        ...,
        description="Connector reference: name or hash of the connector to patch.",
    )

    def as_patch(self) -> dict[str, Any]:
        """Return extra keys as a patch mapping (excludes ``connector``)."""
        return dict(self.model_extra or {})

Attributes

connector = Field(..., description='Connector reference: name or hash of the connector to patch.') class-attribute instance-attribute
model_config = ConfigDict(extra='allow') class-attribute instance-attribute

Methods:

as_patch()

Return extra keys as a patch mapping (excludes connector).

Source code in graflo/architecture/contract/bindings/connectors.py
def as_patch(self) -> dict[str, Any]:
    """Return extra keys as a patch mapping (excludes ``connector``)."""
    return dict(self.model_extra or {})

FileConnector

Bases: ResourceConnector

Connector for matching files.

Attributes:

Name Type Description
regex str | None

Regular expression pattern for matching filenames

sub_path Path

Directory to search for matching files (default: "./"). With a conn_proxy naming a files connection it is relative to that connection's root, a local directory or an S3 prefix; without one, a local path.

recursive bool

Also search the directories under sub_path. regex still matches the file's name, not its path.

file_type Literal['json', 'jsonl', 'table', 'parquet'] | None

Format of the matched files, for names that do not show it. Left unset, the format comes from the extension.

time_filter ColumnTimeFilter | None

Refused when set. A file is read whole, so a window declared here would not be applied; filter the rows in a pipeline step, or bind a :class:TableConnector.

Source code in graflo/architecture/contract/bindings/connectors.py
class FileConnector(ResourceConnector):
    """Connector for matching files.

    Attributes:
        regex: Regular expression pattern for matching filenames
        sub_path: Directory to search for matching files (default: "./"). With
            a ``conn_proxy`` naming a files connection it is relative to that
            connection's root, a local directory or an S3 prefix; without one,
            a local path.
        recursive: Also search the directories under ``sub_path``. ``regex``
            still matches the file's name, not its path.
        file_type: Format of the matched files, for names that do not show it.
            Left unset, the format comes from the extension.
        time_filter: Refused when set. A file is read whole, so a window
            declared here would not be applied; filter the rows in a pipeline
            step, or bind a :class:`TableConnector`.
    """

    takes_default_conn_proxy: ClassVar[bool] = False

    regex: str | None = None
    sub_path: pathlib.Path = Field(default_factory=lambda: pathlib.Path("./"))
    recursive: bool = False
    file_type: Literal["json", "jsonl", "table", "parquet"] | None = None
    time_filter: ColumnTimeFilter | None = None

    def _hash_payload(self) -> dict[str, Any]:
        payload = super()._hash_payload()
        # Left out at their defaults, so a connector that does not set them
        # keeps the hash it is referred to by.
        if not self.recursive:
            del payload["recursive"]
        if self.file_type is None:
            del payload["file_type"]
        return payload

    @model_validator(mode="after")
    def _validate_file_connector(self) -> Self:
        """Ensure sub_path is a Path."""
        if not isinstance(self.sub_path, pathlib.Path):
            object.__setattr__(self, "sub_path", pathlib.Path(self.sub_path))
        if self.row_annotations:
            raise ValueError("row_annotations is not implemented for FileConnector")
        if self.time_filter is not None:
            raise ValueError(
                "time_filter is not implemented for FileConnector: a file is read "
                "whole, so the window would not be applied"
            )
        return self

    def matches(self, resource_identifier: str) -> bool:
        """Check if connector matches a filename.

        Args:
            resource_identifier: Filename to match

        Returns:
            bool: True if connector matches
        """
        if self.regex is None:
            return False
        return bool(re.match(self.regex, resource_identifier))

    def bound_source_kind(self) -> BoundSourceKind:
        """File connector always uses ``BoundSourceKind.FILE``."""
        return BoundSourceKind.FILE

Attributes

file_type = None class-attribute instance-attribute
recursive = False class-attribute instance-attribute
regex = None class-attribute instance-attribute
sub_path = Field(default_factory=lambda: pathlib.Path('./')) class-attribute instance-attribute
takes_default_conn_proxy = False class-attribute
time_filter = None class-attribute instance-attribute

Methods:

bound_source_kind()

File connector always uses BoundSourceKind.FILE.

Source code in graflo/architecture/contract/bindings/connectors.py
def bound_source_kind(self) -> BoundSourceKind:
    """File connector always uses ``BoundSourceKind.FILE``."""
    return BoundSourceKind.FILE
matches(resource_identifier)

Check if connector matches a filename.

Parameters:

Name Type Description Default
resource_identifier str

Filename to match

required

Returns:

Name Type Description
bool bool

True if connector matches

Source code in graflo/architecture/contract/bindings/connectors.py
def matches(self, resource_identifier: str) -> bool:
    """Check if connector matches a filename.

    Args:
        resource_identifier: Filename to match

    Returns:
        bool: True if connector matches
    """
    if self.regex is None:
        return False
    return bool(re.match(self.regex, resource_identifier))

KafkaConnector

Bases: ResourceConnector

Connector for Kafka topic consumption.

Declares the non-secret access pattern (topics, consumer group, decode options). Runtime bootstrap servers and credentials are supplied via connector_connection -> conn_proxy -> :class:~graflo.connections.sources.KafkaConnConfig.

Attributes:

Name Type Description
topics list[str]

Topic names to subscribe to.

group_id str

Kafka consumer group id.

auto_offset_reset Literal['earliest', 'latest']

Where to start when no committed offset exists.

value_encoding Literal['json']

Payload decode mode (json only in v1).

include_headers bool

When True, attach decoded headers under _kafka_headers.

idle_ms int

Stop after this many milliseconds with no messages.

max_wait_ms int | None

Hard wall-clock cap for a single iter_batches run.

poll_timeout_ms int

Per-poll timeout passed to the Kafka client.

row_annotations dict[str, Any]

Constant fields merged into every decoded document (doc wins).

Source code in graflo/architecture/contract/bindings/connectors.py
class KafkaConnector(ResourceConnector):
    """Connector for Kafka topic consumption.

    Declares the non-secret access pattern (topics, consumer group, decode
    options). Runtime bootstrap servers and credentials are supplied via
    ``connector_connection`` -> ``conn_proxy`` ->
    :class:`~graflo.connections.sources.KafkaConnConfig`.

    Attributes:
        topics: Topic names to subscribe to.
        group_id: Kafka consumer group id.
        auto_offset_reset: Where to start when no committed offset exists.
        value_encoding: Payload decode mode (``json`` only in v1).
        include_headers: When True, attach decoded headers under ``_kafka_headers``.
        idle_ms: Stop after this many milliseconds with no messages.
        max_wait_ms: Hard wall-clock cap for a single ``iter_batches`` run.
        poll_timeout_ms: Per-poll timeout passed to the Kafka client.
        row_annotations: Constant fields merged into every decoded document (doc wins).
    """

    topics: list[str] = Field(..., min_length=1, description="Kafka topics to consume")
    group_id: str = Field(..., description="Consumer group id")
    auto_offset_reset: Literal["earliest", "latest"] = "earliest"
    value_encoding: Literal["json"] = "json"
    include_headers: bool = False
    idle_ms: int = Field(
        default=2000,
        ge=0,
        description="Stop after this many ms with no messages (0 disables idle stop).",
    )
    max_wait_ms: int | None = Field(
        default=None,
        ge=0,
        description="Optional wall-clock cap for one consume run.",
    )
    poll_timeout_ms: int = Field(
        default=500,
        ge=1,
        description="Timeout for each consumer poll call.",
    )

    @field_validator("topics")
    @classmethod
    def _non_empty_topic_names(cls, v: list[str]) -> list[str]:
        cleaned = [t.strip() for t in v if t and t.strip()]
        if not cleaned:
            raise ValueError("topics must contain at least one non-empty topic name")
        return cleaned

    def matches(self, resource_identifier: str) -> bool:
        """Match resource name, connector name, or a subscribed topic."""
        if self.name is not None and resource_identifier == self.name:
            return True
        if self.resource_name is not None and resource_identifier == self.resource_name:
            return True
        return resource_identifier in self.topics

    def bound_source_kind(self) -> BoundSourceKind:
        return BoundSourceKind.KAFKA

    def build_kafka_config(
        self,
        *,
        conn: KafkaConnConfig,
    ) -> KafkaConfig:
        """Merge contract fields with runtime connection config into ``KafkaConfig``."""
        from graflo.data_source.kafka import KafkaConfig

        return KafkaConfig(
            bootstrap_servers=conn.bootstrap_servers,
            security_protocol=conn.security_protocol,
            client_id=conn.client_id,
            sasl_mechanism=conn.sasl_mechanism,
            sasl_username=conn.sasl_username,
            sasl_password=conn.sasl_password,
            topics=list(self.topics),
            group_id=self.group_id,
            auto_offset_reset=self.auto_offset_reset,
            value_encoding=self.value_encoding,
            include_headers=self.include_headers,
            idle_ms=self.idle_ms,
            max_wait_ms=self.max_wait_ms,
            poll_timeout_ms=self.poll_timeout_ms,
            row_annotations=dict(self.row_annotations),
        )

Attributes

auto_offset_reset = 'earliest' class-attribute instance-attribute
group_id = Field(..., description='Consumer group id') class-attribute instance-attribute
idle_ms = Field(default=2000, ge=0, description='Stop after this many ms with no messages (0 disables idle stop).') class-attribute instance-attribute
include_headers = False class-attribute instance-attribute
max_wait_ms = Field(default=None, ge=0, description='Optional wall-clock cap for one consume run.') class-attribute instance-attribute
poll_timeout_ms = Field(default=500, ge=1, description='Timeout for each consumer poll call.') class-attribute instance-attribute
topics = Field(..., min_length=1, description='Kafka topics to consume') class-attribute instance-attribute
value_encoding = 'json' class-attribute instance-attribute

Methods:

bound_source_kind()
Source code in graflo/architecture/contract/bindings/connectors.py
def bound_source_kind(self) -> BoundSourceKind:
    return BoundSourceKind.KAFKA
build_kafka_config(*, conn)

Merge contract fields with runtime connection config into KafkaConfig.

Source code in graflo/architecture/contract/bindings/connectors.py
def build_kafka_config(
    self,
    *,
    conn: KafkaConnConfig,
) -> KafkaConfig:
    """Merge contract fields with runtime connection config into ``KafkaConfig``."""
    from graflo.data_source.kafka import KafkaConfig

    return KafkaConfig(
        bootstrap_servers=conn.bootstrap_servers,
        security_protocol=conn.security_protocol,
        client_id=conn.client_id,
        sasl_mechanism=conn.sasl_mechanism,
        sasl_username=conn.sasl_username,
        sasl_password=conn.sasl_password,
        topics=list(self.topics),
        group_id=self.group_id,
        auto_offset_reset=self.auto_offset_reset,
        value_encoding=self.value_encoding,
        include_headers=self.include_headers,
        idle_ms=self.idle_ms,
        max_wait_ms=self.max_wait_ms,
        poll_timeout_ms=self.poll_timeout_ms,
        row_annotations=dict(self.row_annotations),
    )
matches(resource_identifier)

Match resource name, connector name, or a subscribed topic.

Source code in graflo/architecture/contract/bindings/connectors.py
def matches(self, resource_identifier: str) -> bool:
    """Match resource name, connector name, or a subscribed topic."""
    if self.name is not None and resource_identifier == self.name:
        return True
    if self.resource_name is not None and resource_identifier == self.resource_name:
        return True
    return resource_identifier in self.topics

PaginationConfig

Bases: ConfigBaseModel

Configuration for API pagination (contract-level, secret-free).

Combines request construction (request) with response envelope parsing (response).

Source code in graflo/architecture/contract/bindings/connectors.py
class PaginationConfig(ConfigBaseModel):
    """Configuration for API pagination (contract-level, secret-free).

    Combines request construction (``request``) with response envelope parsing
    (``response``).
    """

    request: PaginationRequestConfig = Field(default_factory=PaginationRequestConfig)
    response: ApiResponseStructure = Field(default_factory=ApiResponseStructure)

Attributes

request = Field(default_factory=PaginationRequestConfig) class-attribute instance-attribute
response = Field(default_factory=ApiResponseStructure) class-attribute instance-attribute

PaginationRequestConfig

Bases: ConfigBaseModel

Configuration for building paginated HTTP requests.

Source code in graflo/architecture/contract/bindings/connectors.py
class PaginationRequestConfig(ConfigBaseModel):
    """Configuration for building paginated HTTP requests."""

    strategy: Literal["offset", "page", "cursor"] = "offset"
    offset_param: str = "offset"
    limit_param: str | None = Field(
        default="limit",
        description=(
            "Query parameter name for page size (offset strategy only). "
            "The value sent is ``page_size``, not a total item cap. "
            "When ``None``, no page-size query parameter is sent."
        ),
    )
    cursor_param: str = "cursor"
    page_param: str = "page"
    per_page_param: str = Field(
        default="per_page",
        description=(
            "Query parameter name for page size (page strategy only). "
            "The value sent is ``page_size``, not a total item cap."
        ),
    )
    initial_offset: int = 0
    initial_page: int = 1
    initial_cursor: str | None = None
    page_size: int = Field(
        default=100,
        description=(
            "Records requested per HTTP page. Sent as the value of "
            "``limit_param`` (offset) or ``per_page_param`` (page) when those "
            "param names are set."
        ),
    )
    carry_params: dict[str, str] = Field(
        default_factory=dict,
        description=(
            "Query param name -> response dot path. After each page, values are "
            "read and sent on later requests (e.g. ``results_id: '0.results_id'``). "
            "When empty, known session-token fields are auto-detected from the "
            "first response."
        ),
    )

Attributes

carry_params = Field(default_factory=dict, description="Query param name -> response dot path. After each page, values are read and sent on later requests (e.g. ``results_id: '0.results_id'``). When empty, known session-token fields are auto-detected from the first response.") class-attribute instance-attribute
cursor_param = 'cursor' class-attribute instance-attribute
initial_cursor = None class-attribute instance-attribute
initial_offset = 0 class-attribute instance-attribute
initial_page = 1 class-attribute instance-attribute
limit_param = Field(default='limit', description='Query parameter name for page size (offset strategy only). The value sent is ``page_size``, not a total item cap. When ``None``, no page-size query parameter is sent.') class-attribute instance-attribute
offset_param = 'offset' class-attribute instance-attribute
page_param = 'page' class-attribute instance-attribute
page_size = Field(default=100, description='Records requested per HTTP page. Sent as the value of ``limit_param`` (offset) or ``per_page_param`` (page) when those param names are set.') class-attribute instance-attribute
per_page_param = Field(default='per_page', description='Query parameter name for page size (page strategy only). The value sent is ``page_size``, not a total item cap.') class-attribute instance-attribute
strategy = 'offset' class-attribute instance-attribute

ParentRequest

Bases: ConfigBaseModel

Make one request per record of another API connector.

Attributes:

Name Type Description
connector str

Name of the parent API connector in the same bindings.

fields dict[str, str]

Template variable -> dot path into a parent record. Each variable fills {variable} in the child's path, params and body; a parent record missing any of them is skipped.

Source code in graflo/architecture/contract/bindings/connectors.py
class ParentRequest(ConfigBaseModel):
    """Make one request per record of another API connector.

    Attributes:
        connector: Name of the parent API connector in the same bindings.
        fields: Template variable -> dot path into a parent record. Each
            variable fills ``{variable}`` in the child's ``path``, ``params``
            and ``body``; a parent record missing any of them is skipped.
    """

    connector: str = Field(..., min_length=1)
    fields: dict[str, str] = Field(..., min_length=1)

Attributes

connector = Field(..., min_length=1) class-attribute instance-attribute
fields = Field(..., min_length=1) class-attribute instance-attribute

PushConnector

Bases: ResourceConnector

Connector for records pushed to a resource rather than fetched.

It has no address: the caller hands the records over at run time (RegistryBuilder.build(pushed=...)), keyed by the connector name. Declaring it keeps what a resource accepts versioned and reviewable like any other source.

Attributes:

Name Type Description
kind Literal['push']

Always push; required, so the connector is never read as another kind.

name str

Name the pushed records are keyed by.

records_path str | None

Dot path to the record list when a pushed item is an envelope rather than a record.

row_annotations dict[str, Any]

Constant fields merged into every record (record wins).

Source code in graflo/architecture/contract/bindings/connectors.py
class PushConnector(ResourceConnector):
    """Connector for records pushed to a resource rather than fetched.

    It has no address: the caller hands the records over at run time
    (``RegistryBuilder.build(pushed=...)``), keyed by the connector name. Declaring
    it keeps what a resource accepts versioned and reviewable like any other
    source.

    Attributes:
        kind: Always ``push``; required, so the connector is never read as
            another kind.
        name: Name the pushed records are keyed by.
        records_path: Dot path to the record list when a pushed item is an
            envelope rather than a record.
        row_annotations: Constant fields merged into every record (record wins).
    """

    takes_default_conn_proxy: ClassVar[bool] = False

    kind: Literal["push"]
    # Narrowed from the base's optional name; validate_assignment refuses None.
    name: str = Field(..., min_length=1)  # ty: ignore[invalid-mutable-override]
    records_path: str | None = None

    def _hash_payload(self) -> dict[str, Any]:
        payload = super()._hash_payload()
        # It has no address: the name is what records are pushed to, so two
        # push connectors with different names are two sources.
        payload["name"] = self.name
        return payload

    def matches(self, resource_identifier: str) -> bool:
        """Match connector name or resource name."""
        return resource_identifier == self.name or (
            self.resource_name is not None and resource_identifier == self.resource_name
        )

    def bound_source_kind(self) -> BoundSourceKind:
        return BoundSourceKind.PUSH

Attributes

kind instance-attribute
name = Field(..., min_length=1) class-attribute instance-attribute
records_path = None class-attribute instance-attribute
takes_default_conn_proxy = False class-attribute

Methods:

bound_source_kind()
Source code in graflo/architecture/contract/bindings/connectors.py
def bound_source_kind(self) -> BoundSourceKind:
    return BoundSourceKind.PUSH
matches(resource_identifier)

Match connector name or resource name.

Source code in graflo/architecture/contract/bindings/connectors.py
def matches(self, resource_identifier: str) -> bool:
    """Match connector name or resource name."""
    return resource_identifier == self.name or (
        self.resource_name is not None and resource_identifier == self.resource_name
    )

ResourceConnector

Bases: ConfigBaseModel, ABC

Abstract base class for resource connectors (files or tables).

Provides common API for connector matching and resource identification. All concrete connector types inherit from this class.

Connectors only describe source-side matching/query behavior. Resource-to- connector linkage is handled by Bindings.

Source code in graflo/architecture/contract/bindings/connectors.py
class ResourceConnector(ConfigBaseModel, abc.ABC):
    """Abstract base class for resource connectors (files or tables).

    Provides common API for connector matching and resource identification.
    All concrete connector types inherit from this class.

    Connectors only describe source-side matching/query behavior. Resource-to-
    connector linkage is handled by ``Bindings``.
    """

    name: str | None = Field(
        default=None,
        description="Optional connector name used by top-level resource_connector mapping.",
    )
    resource_name: str | None = Field(
        default=None,
        description="Optional direct resource binding declared on the connector itself.",
    )
    hash: str = Field(
        default="",
        exclude=True,
        description="Deterministic internal connector id derived from defining fields.",
    )
    #: Whether the bindings' default ``conn_proxy`` applies to this connector.
    #: A connector that can be read without a connection only uses one wired
    #: to it explicitly.
    takes_default_conn_proxy: ClassVar[bool] = True
    row_annotations: dict[str, Any] = Field(
        default_factory=dict,
        description=(
            "Constant fields merged into every fetched row as defaults (response "
            "fields take priority). Implemented for :class:`APIConnector` and "
            ":class:`KafkaConnector`; other connector types reject non-empty values."
        ),
    )

    def _hash_payload(self) -> dict[str, Any]:
        payload = self.model_dump(
            mode="json",
            by_alias=True,
            exclude={"hash", "name", "resource_name"},
        )
        payload["_connector_type"] = type(self).__name__
        return payload

    @model_validator(mode="after")
    def _compute_hash(self) -> Self:
        object.__setattr__(self, "hash", suthing.stable_hash(self._hash_payload()))
        return self

    @abc.abstractmethod
    def matches(self, resource_identifier: str) -> bool:
        """Check if connector matches a resource identifier.

        Args:
            resource_identifier: Identifier to match (filename or table name)

        Returns:
            bool: True if connector matches
        """

    @abc.abstractmethod
    def bound_source_kind(self) -> BoundSourceKind:
        """Return the physical source kind for this connector."""

    def row_constants(self) -> list[dict[str, Any]]:
        """The constant fields this connector puts on its rows, one mapping per request.

        A row field of the same name wins over a constant, so these are what
        the connector fixes, not what every row ends up holding.
        """
        return [dict(self.row_annotations)]

Attributes

hash = Field(default='', exclude=True, description='Deterministic internal connector id derived from defining fields.') class-attribute instance-attribute
name = Field(default=None, description='Optional connector name used by top-level resource_connector mapping.') class-attribute instance-attribute
resource_name = Field(default=None, description='Optional direct resource binding declared on the connector itself.') class-attribute instance-attribute
row_annotations = Field(default_factory=dict, description='Constant fields merged into every fetched row as defaults (response fields take priority). Implemented for :class:`APIConnector` and :class:`KafkaConnector`; other connector types reject non-empty values.') class-attribute instance-attribute
takes_default_conn_proxy = True class-attribute

Methods:

bound_source_kind() abstractmethod

Return the physical source kind for this connector.

Source code in graflo/architecture/contract/bindings/connectors.py
@abc.abstractmethod
def bound_source_kind(self) -> BoundSourceKind:
    """Return the physical source kind for this connector."""
matches(resource_identifier) abstractmethod

Check if connector matches a resource identifier.

Parameters:

Name Type Description Default
resource_identifier str

Identifier to match (filename or table name)

required

Returns:

Name Type Description
bool bool

True if connector matches

Source code in graflo/architecture/contract/bindings/connectors.py
@abc.abstractmethod
def matches(self, resource_identifier: str) -> bool:
    """Check if connector matches a resource identifier.

    Args:
        resource_identifier: Identifier to match (filename or table name)

    Returns:
        bool: True if connector matches
    """
row_constants()

The constant fields this connector puts on its rows, one mapping per request.

A row field of the same name wins over a constant, so these are what the connector fixes, not what every row ends up holding.

Source code in graflo/architecture/contract/bindings/connectors.py
def row_constants(self) -> list[dict[str, Any]]:
    """The constant fields this connector puts on its rows, one mapping per request.

    A row field of the same name wins over a constant, so these are what
    the connector fixes, not what every row ends up holding.
    """
    return [dict(self.row_annotations)]

ResourceConnectorBinding

Bases: ConfigBaseModel

Top-level resource -> connector-name mapping entry.

Source code in graflo/architecture/contract/bindings/core.py
class ResourceConnectorBinding(ConfigBaseModel):
    """Top-level resource -> connector-name mapping entry."""

    resource: str
    connector: str

Attributes

connector instance-attribute
resource instance-attribute

SparqlConnector

Bases: TemplatedConnector

Connector for matching SPARQL / RDF data sources.

Each request targets instances of a single rdf:Class; with values the connector makes one request per value (see :class:TemplatedConnector). rdf_class and graph_uri are address templates, sparql_query a query text. It can be backed either by a remote SPARQL endpoint (Fuseki, Blazegraph, ...) or by a local RDF file parsed with rdflib.

Attributes:

Name Type Description
rdf_class str

Full URI of the rdf:Class whose instances this connector fetches (e.g. "http://example.org/Person"), or a template of one ("http://example.org/{cls}").

endpoint_url str | None

SPARQL query endpoint URL. When set, instances are fetched via HTTP. When None the connector is for local file mode.

graph_uri str | None

Named-graph URI to restrict the query to (optional).

sparql_query str | None

Custom SPARQL SELECT query override. When provided the auto-generated per-class query is skipped.

rdf_file Path | None

Path to a local RDF file (.ttl, .rdf, .n3, .jsonld). Mutually exclusive with endpoint_url.

same_as Literal['collapse', 'keep']

collapse (default) reads IRIs joined by owl:sameAs as one record under the smallest IRI, the others listed in _same_as. keep reads the statements as a property.

Source code in graflo/architecture/contract/bindings/connectors.py
class SparqlConnector(TemplatedConnector):
    """Connector for matching SPARQL / RDF data sources.

    Each request targets instances of a single ``rdf:Class``; with ``values``
    the connector makes one request per value (see :class:`TemplatedConnector`).
    ``rdf_class`` and ``graph_uri`` are address templates, ``sparql_query`` a
    query text. It can be backed either by a remote SPARQL endpoint (Fuseki,
    Blazegraph, ...) or by a local RDF file parsed with *rdflib*.

    Attributes:
        rdf_class: Full URI of the ``rdf:Class`` whose instances this connector
            fetches (e.g. ``"http://example.org/Person"``), or a template of
            one (``"http://example.org/{cls}"``).
        endpoint_url: SPARQL query endpoint URL.  When set, instances are
            fetched via HTTP.  When ``None`` the connector is for local file mode.
        graph_uri: Named-graph URI to restrict the query to (optional).
        sparql_query: Custom SPARQL ``SELECT`` query override.  When provided
            the auto-generated per-class query is skipped.
        rdf_file: Path to a local RDF file (``.ttl``, ``.rdf``, ``.n3``,
            ``.jsonld``).  Mutually exclusive with *endpoint_url*.
        same_as: ``collapse`` (default) reads IRIs joined by ``owl:sameAs`` as
            one record under the smallest IRI, the others listed in
            ``_same_as``. ``keep`` reads the statements as a property.
    """

    rdf_class: str = Field(
        ..., description="URI of the rdf:Class to fetch instances of"
    )
    endpoint_url: str | None = Field(
        default=None, description="SPARQL query endpoint URL"
    )
    graph_uri: str | None = Field(
        default=None, description="Named graph URI (optional)"
    )
    sparql_query: str | None = Field(
        default=None, description="Custom SPARQL query override"
    )
    rdf_file: pathlib.Path | None = Field(
        default=None, description="Path to a local RDF file"
    )
    same_as: Literal["collapse", "keep"] = Field(
        default="collapse",
        description="How owl:sameAs statements between IRIs are read.",
    )
    typed_objects: list[str] = Field(
        default_factory=list,
        description=(
            "Properties (local names) whose objects are also read by class, as "
            "``<property>@<Class>``: for a property with several ranges."
        ),
    )

    def _hash_payload(self) -> dict[str, Any]:
        payload = super()._hash_payload()
        # Left out at their defaults, so a connector that does not set them
        # keeps the hash it is referred to by.
        if self.same_as == "collapse":
            del payload["same_as"]
        if not self.typed_objects:
            del payload["typed_objects"]
        return payload

    @model_validator(mode="after")
    def _reject_row_annotations(self) -> Self:
        # TODO: implement row_annotations for SparqlConnector row payloads.
        if self.row_annotations:
            raise ValueError("row_annotations is not implemented for SparqlConnector")
        return self

    def _address_templates(self) -> list[Any]:
        return [self.rdf_class, self.graph_uri]

    def _text_templates(self) -> list[Any]:
        return [self.sparql_query]

    def _label(self) -> str:
        return self.name or self.rdf_class

    def rendered(self, values: Mapping[str, Any]) -> Self:
        return self.model_copy(
            update={
                "rdf_class": render_text(self.rdf_class, values),
                "graph_uri": None
                if self.graph_uri is None
                else render_text(self.graph_uri, values),
                "sparql_query": None
                if self.sparql_query is None
                else render_text(self.sparql_query, values),
                "values": None,
            }
        )

    def matches(self, resource_identifier: str) -> bool:
        """Match by the local name (fragment) of the rdf:Class URI.

        Args:
            resource_identifier: Identifier to match against

        Returns:
            True when *resource_identifier* equals the class local name
        """
        local_name = self.rdf_class.rsplit("#", 1)[-1].rsplit("/", 1)[-1]
        return resource_identifier == local_name

    def bound_source_kind(self) -> BoundSourceKind:
        """Return ``BoundSourceKind.SPARQL``."""
        return BoundSourceKind.SPARQL

    def build_select_query(self) -> str:
        """Build a SPARQL SELECT query for instances of ``rdf_class``.

        If *sparql_query* is set it is returned as-is.  Otherwise a simple
        per-class query is generated::

            SELECT ?s ?p ?o WHERE {
              ?s a <rdf_class> .
              ?s ?p ?o .
            }

        Returns:
            SPARQL query string
        """
        if self.sparql_query:
            return self.sparql_query

        graph_open = f"GRAPH <{self.graph_uri}> {{" if self.graph_uri else ""
        graph_close = "}" if self.graph_uri else ""

        return (
            "SELECT ?s ?p ?o WHERE { "
            f"{graph_open} "
            f"?s a <{self.rdf_class}> . "
            f"?s ?p ?o . "
            f"{graph_close} "
            "}"
        )

Attributes

endpoint_url = Field(default=None, description='SPARQL query endpoint URL') class-attribute instance-attribute
graph_uri = Field(default=None, description='Named graph URI (optional)') class-attribute instance-attribute
rdf_class = Field(..., description='URI of the rdf:Class to fetch instances of') class-attribute instance-attribute
rdf_file = Field(default=None, description='Path to a local RDF file') class-attribute instance-attribute
same_as = Field(default='collapse', description='How owl:sameAs statements between IRIs are read.') class-attribute instance-attribute
sparql_query = Field(default=None, description='Custom SPARQL query override') class-attribute instance-attribute
typed_objects = Field(default_factory=list, description='Properties (local names) whose objects are also read by class, as ``<property>@<Class>``: for a property with several ranges.') class-attribute instance-attribute

Methods:

bound_source_kind()

Return BoundSourceKind.SPARQL.

Source code in graflo/architecture/contract/bindings/connectors.py
def bound_source_kind(self) -> BoundSourceKind:
    """Return ``BoundSourceKind.SPARQL``."""
    return BoundSourceKind.SPARQL
build_select_query()

Build a SPARQL SELECT query for instances of rdf_class.

If sparql_query is set it is returned as-is. Otherwise a simple per-class query is generated::

SELECT ?s ?p ?o WHERE {
  ?s a <rdf_class> .
  ?s ?p ?o .
}

Returns:

Type Description
str

SPARQL query string

Source code in graflo/architecture/contract/bindings/connectors.py
def build_select_query(self) -> str:
    """Build a SPARQL SELECT query for instances of ``rdf_class``.

    If *sparql_query* is set it is returned as-is.  Otherwise a simple
    per-class query is generated::

        SELECT ?s ?p ?o WHERE {
          ?s a <rdf_class> .
          ?s ?p ?o .
        }

    Returns:
        SPARQL query string
    """
    if self.sparql_query:
        return self.sparql_query

    graph_open = f"GRAPH <{self.graph_uri}> {{" if self.graph_uri else ""
    graph_close = "}" if self.graph_uri else ""

    return (
        "SELECT ?s ?p ?o WHERE { "
        f"{graph_open} "
        f"?s a <{self.rdf_class}> . "
        f"?s ?p ?o . "
        f"{graph_close} "
        "}"
    )
matches(resource_identifier)

Match by the local name (fragment) of the rdf:Class URI.

Parameters:

Name Type Description Default
resource_identifier str

Identifier to match against

required

Returns:

Type Description
bool

True when resource_identifier equals the class local name

Source code in graflo/architecture/contract/bindings/connectors.py
def matches(self, resource_identifier: str) -> bool:
    """Match by the local name (fragment) of the rdf:Class URI.

    Args:
        resource_identifier: Identifier to match against

    Returns:
        True when *resource_identifier* equals the class local name
    """
    local_name = self.rdf_class.rsplit("#", 1)[-1].rsplit("/", 1)[-1]
    return resource_identifier == local_name
rendered(values)
Source code in graflo/architecture/contract/bindings/connectors.py
def rendered(self, values: Mapping[str, Any]) -> Self:
    return self.model_copy(
        update={
            "rdf_class": render_text(self.rdf_class, values),
            "graph_uri": None
            if self.graph_uri is None
            else render_text(self.graph_uri, values),
            "sparql_query": None
            if self.sparql_query is None
            else render_text(self.sparql_query, values),
            "values": None,
        }
    )

TableConnector

Bases: ResourceConnector

Connector for matching database tables.

Supports simple single-table queries as well as multi-table JOINs and pushdown filters via FilterExpression.

Attributes:

Name Type Description
table_name str

Exact table name or regex pattern

schema_name str | None

Schema name (optional, defaults to public)

database str | None

Database name (optional)

time_filter ColumnTimeFilter | None

Optional structured filter on a date/time column, rendered via :class:~graflo.filter.onto.FilterExpression in SQL.

filters list[Any]

General-purpose pushdown filters rendered as SQL WHERE fragments.

joins list[JoinClause]

Multi-table JOIN specifications (auto-generated or explicit).

base_alias str

SQL alias for the base table when joins is non-empty.

select_columns list[str] | None

Explicit SELECT column list. None means * for the base table (plus aliased columns from joins).

Source code in graflo/architecture/contract/bindings/connectors.py
class TableConnector(ResourceConnector):
    """Connector for matching database tables.

    Supports simple single-table queries as well as multi-table JOINs and
    pushdown filters via ``FilterExpression``.

    Attributes:
        table_name: Exact table name or regex pattern
        schema_name: Schema name (optional, defaults to public)
        database: Database name (optional)
        time_filter: Optional structured filter on a date/time column, rendered
            via :class:`~graflo.filter.onto.FilterExpression` in SQL.
        filters: General-purpose pushdown filters rendered as SQL WHERE fragments.
        joins: Multi-table JOIN specifications (auto-generated or explicit).
        base_alias: SQL alias for the base table when ``joins`` is non-empty.
        select_columns: Explicit SELECT column list. None means ``*`` for the
            base table (plus aliased columns from joins).
    """

    table_name: str = Field(
        default="", validation_alias=AliasChoices("table_name", "table")
    )
    schema_name: str | None = Field(
        default=None, validation_alias=AliasChoices("schema_name", "schema")
    )
    database: str | None = None
    time_filter: ColumnTimeFilter | None = None
    filters: list[Any] = Field(
        default_factory=list,
        description="Pushdown FilterExpression filters (rendered to SQL WHERE).",
    )
    joins: list[JoinClause] = Field(
        default_factory=list,
        description="JOIN clauses for multi-table queries.",
    )
    base_alias: str = Field(
        default="base",
        description="SQL alias for the base table row when joins are present.",
    )
    select_columns: list[str] | None = Field(
        default=None,
        description="Explicit SELECT columns. None = SELECT * (plus join aliases).",
    )
    view: Any = Field(
        default=None,
        description="SelectSpec or dict for declarative view (alternative to table+joins+filters).",
    )

    @field_validator("filters", mode="before")
    @classmethod
    def _coerce_filters(cls, v: Any) -> list[Any]:
        from graflo.filter.onto import parse_filter_expression

        if v is None:
            return []
        if not isinstance(v, list):
            raise ValueError("filters must be a list")
        result: list[Any] = []
        for i, item in enumerate(v):
            try:
                result.append(parse_filter_expression(item))
            except (ValueError, ValidationError) as e:
                raise ValueError(f"filters[{i}]: {e}") from e
        return result

    @field_validator("view", mode="before")
    @classmethod
    def _coerce_view(cls, v: Any) -> Any:
        if v is None:
            return None
        if isinstance(v, dict):
            from graflo.filter.select import SelectSpec

            return SelectSpec.from_dict(v)
        return v

    @model_validator(mode="after")
    def _validate_table_connector(self) -> Self:
        """Validate table_name and join wiring."""
        if not self.table_name:
            raise ValueError("table_name is required for TableConnector")
        if not _BASE_TABLE_ALIAS_IDENT.match(self.base_alias):
            raise ValueError(
                "base_alias must be a simple SQL identifier "
                "(ASCII letter, digit, underscore)"
            )
        join_aliases = {jc.alias or jc.table for jc in self.joins}
        if self.base_alias in join_aliases:
            raise ValueError(
                f"base_alias {self.base_alias!r} conflicts with a join alias "
                f"{sorted(join_aliases)}"
            )
        if self.row_annotations:
            raise ValueError("row_annotations is not implemented for TableConnector")
        return self

    @property
    def date_field(self) -> str | None:
        """Column used for time filtering, if any (compat alias for ``time_filter.column``)."""
        return self.time_filter.column if self.time_filter else None

    def matches(self, resource_identifier: str) -> bool:
        """Check if connector matches a table name.

        Args:
            resource_identifier: Table name to match (format: schema.table or just table)

        Returns:
            bool: True if connector matches
        """
        if not self.table_name:
            return False

        # Compile regex expression
        if self.table_name.startswith("^") or self.table_name.endswith("$"):
            # Already a regex expression
            compiled_regex = re.compile(self.table_name)
        else:
            # Exact match expression
            compiled_regex = re.compile(f"^{re.escape(self.table_name)}$")

        # Check if resource_identifier matches
        if compiled_regex.match(resource_identifier):
            return True

        # If schema_name is specified, also check schema.table format
        if self.schema_name:
            full_name = f"{self.schema_name}.{resource_identifier}"
            if compiled_regex.match(full_name):
                return True

        return False

    def bound_source_kind(self) -> BoundSourceKind:
        return BoundSourceKind.SQL_TABLE

    def where_conditions(
        self,
        base_alias: str | None = None,
        extra_filters: Sequence[FilterExpression] = (),
    ) -> list[str]:
        """The connector's SQL conditions, each safe as one operand of ``AND``.

        In order: the time filter, the declared ``filters``, then
        *extra_filters* (conditions supplied for one run). With *base_alias*,
        unqualified columns are qualified with it.
        """
        from graflo.filter.onto import render_conjunct
        from graflo.onto import ExpressionFlavor

        raw: list[Any] = []
        if self.time_filter is not None:
            raw.append(self.time_filter.as_filter_expression())
        raw.extend(self.filters)
        raw.extend(extra_filters)

        conditions: list[str] = []
        for item in raw:
            filt_expr = self._coerce_filter_expression(item, base_alias)
            if filt_expr is None:
                continue
            rendered = render_conjunct(filt_expr, kind=ExpressionFlavor.SQL)
            if rendered:
                conditions.append(rendered)
        return conditions

    def build_where_clause(
        self,
        base_alias: str | None = None,
        extra_filters: Sequence[FilterExpression] = (),
    ) -> str:
        """Build SQL WHERE clause from time filter **and** general filters.

        Returns:
            WHERE clause string (without the WHERE keyword) or empty string if no filters
        """
        return " AND ".join(self.where_conditions(base_alias, extra_filters))

    def build_query(
        self,
        effective_schema: str | None = None,
        *,
        extra_filters: Sequence[FilterExpression] = (),
    ) -> str:
        """Build a complete SQL SELECT query.

        When ``view`` is set, delegates to ``view.build_sql()``. Otherwise
        incorporates the base table, any JoinClauses, explicit select_columns,
        time_filter, and FilterExpression filters.

        Args:
            effective_schema: Schema to use if ``self.schema_name`` is None.
            extra_filters: Conditions for this run only, such as a date range
                supplied at ingestion time. They are ANDed with the declared
                ones and qualified like them.

        Returns:
            Complete SQL query string.
        """
        schema = self.schema_name or effective_schema or "public"
        if self.view is not None:
            from graflo.filter.select import SelectSpec

            if isinstance(self.view, SelectSpec):
                return self.view.build_sql(
                    schema=schema,
                    base_table=self.table_name,
                    extra_conditions=self.where_conditions(
                        base_alias=self.view.effective_base_alias(),
                        extra_filters=extra_filters,
                    ),
                )
        base_alias = self.base_alias if self.joins else None
        base_ref = f'"{schema}"."{self.table_name}"'
        if base_alias:
            base_ref_aliased = f"{base_ref} {base_alias}"
        else:
            base_ref_aliased = base_ref

        # --- SELECT ---
        select_parts: list[str] = []
        if self.select_columns is not None:
            select_parts = list(self.select_columns)
        elif self.joins:
            select_parts.append(f"{base_alias}.*")
            for jc in self.joins:
                alias = jc.alias or jc.table
                jc_schema = jc.schema_name or schema
                if jc.select_fields is not None:
                    for col in jc.select_fields:
                        select_parts.append(f'{alias}."{col}" AS "{alias}__{col}"')
                else:
                    select_parts.append(f"{alias}.*")
        else:
            select_parts.append("*")

        select_clause = ", ".join(select_parts)

        # --- FROM + JOINs ---
        from_clause = base_ref_aliased
        for jc in self.joins:
            jc_schema = jc.schema_name or schema
            alias = jc.alias or jc.table
            join_ref = f'"{jc_schema}"."{jc.table}"'
            left_col = (
                f'{base_alias}."{jc.on_self}"' if base_alias else f'"{jc.on_self}"'
            )
            right_col = f'{alias}."{jc.on_other}"'
            from_clause += (
                f" {jc.join_type} JOIN {join_ref} {alias} ON {left_col} = {right_col}"
            )

        query = f"SELECT {select_clause} FROM {from_clause}"

        # --- WHERE ---
        where = self.build_where_clause(
            base_alias=base_alias, extra_filters=extra_filters
        )
        if where:
            query += f" WHERE {where}"

        return query

    def source_column(self, name: str) -> str | None:
        """The column the query's output *name* reads, as a filter field.

        A base column is returned bare (conditions qualify it with the base
        alias), a joined one as ``alias.column``. ``None`` when the query's
        output names are not known: raw ``select_columns``, or a join that
        selects all its columns.
        """
        if self.view is not None:
            from graflo.filter.select import SelectSpec

            return (
                self.view.source_column(name)
                if isinstance(self.view, SelectSpec)
                else None
            )
        if self.select_columns is not None:
            return None
        for jc in self.joins:
            if jc.select_fields is None:
                return None
            alias = jc.alias or jc.table
            for col in jc.select_fields:
                if name == f"{alias}__{col}":
                    return f"{alias}.{col}"
        return name

    def lacks_output(self, name: str) -> bool:
        """Whether the query is known not to output a column called *name*.

        Only a ``view`` declares all its output names; any other query reads
        every base column, so nothing is known to be missing.
        """
        from graflo.filter.select import SelectSpec

        return isinstance(self.view, SelectSpec) and self.view.lacks_output(name)

    @staticmethod
    def _qualified_column_ref(column: str, base_alias: str | None) -> str:
        if base_alias:
            return f'{base_alias}."{column}"'
        return f'"{column}"'

    @classmethod
    def _qualify_filter_payload(
        cls, payload: dict[str, Any], base_alias: str | None
    ) -> dict[str, Any]:
        qualified = dict(payload)
        if base_alias is None:
            return qualified
        if qualified.get("kind") == "leaf":
            field = qualified.get("field")
            if isinstance(field, str) and "." not in field:
                qualified["field"] = f"{base_alias}.{field}"
            return qualified
        deps = qualified.get("deps")
        if isinstance(deps, list):
            qualified["deps"] = [
                cls._qualify_filter_payload(dep, base_alias)
                if isinstance(dep, dict)
                else dep
                for dep in deps
            ]
        return qualified

    @classmethod
    def _coerce_filter_expression(
        cls, raw_filter: Any, base_alias: str | None
    ) -> FilterExpression | None:
        from graflo.filter.onto import parse_filter_expression

        if raw_filter is None:
            return None
        expr = parse_filter_expression(raw_filter)
        if base_alias is None:
            return expr
        payload = expr.model_dump(mode="python")
        return parse_filter_expression(cls._qualify_filter_payload(payload, base_alias))

Attributes

base_alias = Field(default='base', description='SQL alias for the base table row when joins are present.') class-attribute instance-attribute
database = None class-attribute instance-attribute
date_field property

Column used for time filtering, if any (compat alias for time_filter.column).

filters = Field(default_factory=list, description='Pushdown FilterExpression filters (rendered to SQL WHERE).') class-attribute instance-attribute
joins = Field(default_factory=list, description='JOIN clauses for multi-table queries.') class-attribute instance-attribute
schema_name = Field(default=None, validation_alias=AliasChoices('schema_name', 'schema')) class-attribute instance-attribute
select_columns = Field(default=None, description='Explicit SELECT columns. None = SELECT * (plus join aliases).') class-attribute instance-attribute
table_name = Field(default='', validation_alias=AliasChoices('table_name', 'table')) class-attribute instance-attribute
time_filter = None class-attribute instance-attribute
view = Field(default=None, description='SelectSpec or dict for declarative view (alternative to table+joins+filters).') class-attribute instance-attribute

Methods:

bound_source_kind()
Source code in graflo/architecture/contract/bindings/connectors.py
def bound_source_kind(self) -> BoundSourceKind:
    return BoundSourceKind.SQL_TABLE
build_query(effective_schema=None, *, extra_filters=())

Build a complete SQL SELECT query.

When view is set, delegates to view.build_sql(). Otherwise incorporates the base table, any JoinClauses, explicit select_columns, time_filter, and FilterExpression filters.

Parameters:

Name Type Description Default
effective_schema str | None

Schema to use if self.schema_name is None.

None
extra_filters Sequence[FilterExpression]

Conditions for this run only, such as a date range supplied at ingestion time. They are ANDed with the declared ones and qualified like them.

()

Returns:

Type Description
str

Complete SQL query string.

Source code in graflo/architecture/contract/bindings/connectors.py
def build_query(
    self,
    effective_schema: str | None = None,
    *,
    extra_filters: Sequence[FilterExpression] = (),
) -> str:
    """Build a complete SQL SELECT query.

    When ``view`` is set, delegates to ``view.build_sql()``. Otherwise
    incorporates the base table, any JoinClauses, explicit select_columns,
    time_filter, and FilterExpression filters.

    Args:
        effective_schema: Schema to use if ``self.schema_name`` is None.
        extra_filters: Conditions for this run only, such as a date range
            supplied at ingestion time. They are ANDed with the declared
            ones and qualified like them.

    Returns:
        Complete SQL query string.
    """
    schema = self.schema_name or effective_schema or "public"
    if self.view is not None:
        from graflo.filter.select import SelectSpec

        if isinstance(self.view, SelectSpec):
            return self.view.build_sql(
                schema=schema,
                base_table=self.table_name,
                extra_conditions=self.where_conditions(
                    base_alias=self.view.effective_base_alias(),
                    extra_filters=extra_filters,
                ),
            )
    base_alias = self.base_alias if self.joins else None
    base_ref = f'"{schema}"."{self.table_name}"'
    if base_alias:
        base_ref_aliased = f"{base_ref} {base_alias}"
    else:
        base_ref_aliased = base_ref

    # --- SELECT ---
    select_parts: list[str] = []
    if self.select_columns is not None:
        select_parts = list(self.select_columns)
    elif self.joins:
        select_parts.append(f"{base_alias}.*")
        for jc in self.joins:
            alias = jc.alias or jc.table
            jc_schema = jc.schema_name or schema
            if jc.select_fields is not None:
                for col in jc.select_fields:
                    select_parts.append(f'{alias}."{col}" AS "{alias}__{col}"')
            else:
                select_parts.append(f"{alias}.*")
    else:
        select_parts.append("*")

    select_clause = ", ".join(select_parts)

    # --- FROM + JOINs ---
    from_clause = base_ref_aliased
    for jc in self.joins:
        jc_schema = jc.schema_name or schema
        alias = jc.alias or jc.table
        join_ref = f'"{jc_schema}"."{jc.table}"'
        left_col = (
            f'{base_alias}."{jc.on_self}"' if base_alias else f'"{jc.on_self}"'
        )
        right_col = f'{alias}."{jc.on_other}"'
        from_clause += (
            f" {jc.join_type} JOIN {join_ref} {alias} ON {left_col} = {right_col}"
        )

    query = f"SELECT {select_clause} FROM {from_clause}"

    # --- WHERE ---
    where = self.build_where_clause(
        base_alias=base_alias, extra_filters=extra_filters
    )
    if where:
        query += f" WHERE {where}"

    return query
build_where_clause(base_alias=None, extra_filters=())

Build SQL WHERE clause from time filter and general filters.

Returns:

Type Description
str

WHERE clause string (without the WHERE keyword) or empty string if no filters

Source code in graflo/architecture/contract/bindings/connectors.py
def build_where_clause(
    self,
    base_alias: str | None = None,
    extra_filters: Sequence[FilterExpression] = (),
) -> str:
    """Build SQL WHERE clause from time filter **and** general filters.

    Returns:
        WHERE clause string (without the WHERE keyword) or empty string if no filters
    """
    return " AND ".join(self.where_conditions(base_alias, extra_filters))
lacks_output(name)

Whether the query is known not to output a column called name.

Only a view declares all its output names; any other query reads every base column, so nothing is known to be missing.

Source code in graflo/architecture/contract/bindings/connectors.py
def lacks_output(self, name: str) -> bool:
    """Whether the query is known not to output a column called *name*.

    Only a ``view`` declares all its output names; any other query reads
    every base column, so nothing is known to be missing.
    """
    from graflo.filter.select import SelectSpec

    return isinstance(self.view, SelectSpec) and self.view.lacks_output(name)
matches(resource_identifier)

Check if connector matches a table name.

Parameters:

Name Type Description Default
resource_identifier str

Table name to match (format: schema.table or just table)

required

Returns:

Name Type Description
bool bool

True if connector matches

Source code in graflo/architecture/contract/bindings/connectors.py
def matches(self, resource_identifier: str) -> bool:
    """Check if connector matches a table name.

    Args:
        resource_identifier: Table name to match (format: schema.table or just table)

    Returns:
        bool: True if connector matches
    """
    if not self.table_name:
        return False

    # Compile regex expression
    if self.table_name.startswith("^") or self.table_name.endswith("$"):
        # Already a regex expression
        compiled_regex = re.compile(self.table_name)
    else:
        # Exact match expression
        compiled_regex = re.compile(f"^{re.escape(self.table_name)}$")

    # Check if resource_identifier matches
    if compiled_regex.match(resource_identifier):
        return True

    # If schema_name is specified, also check schema.table format
    if self.schema_name:
        full_name = f"{self.schema_name}.{resource_identifier}"
        if compiled_regex.match(full_name):
            return True

    return False
source_column(name)

The column the query's output name reads, as a filter field.

A base column is returned bare (conditions qualify it with the base alias), a joined one as alias.column. None when the query's output names are not known: raw select_columns, or a join that selects all its columns.

Source code in graflo/architecture/contract/bindings/connectors.py
def source_column(self, name: str) -> str | None:
    """The column the query's output *name* reads, as a filter field.

    A base column is returned bare (conditions qualify it with the base
    alias), a joined one as ``alias.column``. ``None`` when the query's
    output names are not known: raw ``select_columns``, or a join that
    selects all its columns.
    """
    if self.view is not None:
        from graflo.filter.select import SelectSpec

        return (
            self.view.source_column(name)
            if isinstance(self.view, SelectSpec)
            else None
        )
    if self.select_columns is not None:
        return None
    for jc in self.joins:
        if jc.select_fields is None:
            return None
        alias = jc.alias or jc.table
        for col in jc.select_fields:
            if name == f"{alias}__{col}":
                return f"{alias}.{col}"
    return name
where_conditions(base_alias=None, extra_filters=())

The connector's SQL conditions, each safe as one operand of AND.

In order: the time filter, the declared filters, then extra_filters (conditions supplied for one run). With base_alias, unqualified columns are qualified with it.

Source code in graflo/architecture/contract/bindings/connectors.py
def where_conditions(
    self,
    base_alias: str | None = None,
    extra_filters: Sequence[FilterExpression] = (),
) -> list[str]:
    """The connector's SQL conditions, each safe as one operand of ``AND``.

    In order: the time filter, the declared ``filters``, then
    *extra_filters* (conditions supplied for one run). With *base_alias*,
    unqualified columns are qualified with it.
    """
    from graflo.filter.onto import render_conjunct
    from graflo.onto import ExpressionFlavor

    raw: list[Any] = []
    if self.time_filter is not None:
        raw.append(self.time_filter.as_filter_expression())
    raw.extend(self.filters)
    raw.extend(extra_filters)

    conditions: list[str] = []
    for item in raw:
        filt_expr = self._coerce_filter_expression(item, base_alias)
        if filt_expr is None:
            continue
        rendered = render_conjunct(filt_expr, kind=ExpressionFlavor.SQL)
        if rendered:
            conditions.append(rendered)
    return conditions

TemplatedConnector

Bases: ResourceConnector, ABC

A connector whose request fields may hold {name} variables.

values enumerates variables. The connector then makes one request per combination of their values -- in declaration order, the last variable varying fastest -- with the variables filled in, and every row carries the variables as fields unless it has a field of the same name. Which record becomes what is the resource's to say (a vertex_router on that field); the connector names no vertex.

Template fields are of two kinds. In an address (a URL path, a query parameter, an IRI) every {name} must be a declared variable. In a query text (a request body, a SPARQL query) braces are syntax, so only declared variables are filled and every other brace pair is literal.

Attributes:

Name Type Description
values dict[str, list[TemplateValue]] | None

Variable name -> the values it takes, in request order. Every variable must be used in a template field.

Source code in graflo/architecture/contract/bindings/connectors.py
class TemplatedConnector(ResourceConnector, abc.ABC):
    """A connector whose request fields may hold ``{name}`` variables.

    ``values`` enumerates variables. The connector then makes one request per
    combination of their values -- in declaration order, the last variable
    varying fastest -- with the variables filled in, and every row carries the
    variables as fields unless it has a field of the same name. Which record
    becomes what is the resource's to say (a ``vertex_router`` on that field);
    the connector names no vertex.

    Template fields are of two kinds. In an *address* (a URL path, a query
    parameter, an IRI) every ``{name}`` must be a declared variable. In a
    *query text* (a request body, a SPARQL query) braces are syntax, so only
    declared variables are filled and every other brace pair is literal.

    Attributes:
        values: Variable name -> the values it takes, in request order. Every
            variable must be used in a template field.
    """

    values: dict[str, list[TemplateValue]] | None = Field(
        default=None,
        description=(
            "Variable -> values; one request per combination, each row carrying "
            "the variables as fields."
        ),
    )

    @abc.abstractmethod
    def _address_templates(self) -> list[Any]:
        """Template fields in which every ``{name}`` must be declared."""

    @abc.abstractmethod
    def _text_templates(self) -> list[Any]:
        """Template fields in which only declared variables are filled."""

    def _bound_variables(self) -> set[str]:
        """Variables filled other than from ``values``."""
        return set()

    @abc.abstractmethod
    def _label(self) -> str:
        """How an error names this connector."""

    @abc.abstractmethod
    def rendered(self, values: Mapping[str, Any]) -> Self:
        """A copy for one request: template fields filled from *values*, no ``values``.

        The copy keeps this connector's name and hash, so its connection and
        bindings still resolve to it.
        """

    @field_validator("values")
    @classmethod
    def _check_values(
        cls, v: dict[str, list[TemplateValue]] | None
    ) -> dict[str, list[TemplateValue]] | None:
        if v is None:
            return v
        if not v:
            raise ValueError("values declares no variable; leave it unset")
        for name, items in v.items():
            if not _VARIABLE_NAME.match(name):
                raise ValueError(
                    f"values: {name!r} is not a variable name (a letter or "
                    "underscore, then letters, digits or underscores)"
                )
            if not items:
                raise ValueError(f"values: {name!r} lists no value")
            seen: list[tuple[type, TemplateValue]] = []
            for item in items:
                if (type(item), item) in seen:
                    raise ValueError(f"values: {name!r} lists {item!r} twice")
                seen.append((type(item), item))
        return v

    @model_validator(mode="after")
    def _check_variables(self) -> Self:
        enumerated = set(self.values or {})
        bound = self._bound_variables()
        if both := enumerated & bound:
            raise ValueError(
                f"connector '{self._label()}' declares {sorted(both)} both in "
                "values and as a parent field"
            )
        declared = enumerated | bound
        undeclared = template_variables(*self._address_templates()) - declared
        if undeclared:
            raise ValueError(
                f"connector '{self._label()}' uses template variables "
                f"{sorted(undeclared)} that neither values nor parent.fields declares"
            )
        used = template_variables(*self._address_templates(), *self._text_templates())
        if unused := enumerated - used:
            raise ValueError(
                f"connector '{self._label()}' declares values for {sorted(unused)} "
                "that no template field uses"
            )
        if clash := enumerated & set(self.row_annotations):
            raise ValueError(
                f"connector '{self._label()}' sets {sorted(clash)} both in values "
                "and in row_annotations"
            )
        return self

    def value_combinations(self) -> list[dict[str, Any]]:
        """One mapping per request, in request order; ``[{}]`` without ``values``."""
        if not self.values:
            return [{}]
        names = list(self.values)
        return [
            dict(zip(names, combination, strict=True))
            for combination in itertools.product(*(self.values[n] for n in names))
        ]

    def row_constants(self) -> list[dict[str, Any]]:
        return [
            {**self.row_annotations, **combination}
            for combination in self.value_combinations()
        ]

    def _hash_payload(self) -> dict[str, Any]:
        payload = super()._hash_payload()
        if self.values is None:
            # Unset, it must not move the hash of connectors declared before it.
            payload.pop("values", None)
        return payload

Attributes

values = Field(default=None, description='Variable -> values; one request per combination, each row carrying the variables as fields.') class-attribute instance-attribute

Methods:

rendered(values) abstractmethod

A copy for one request: template fields filled from values, no values.

The copy keeps this connector's name and hash, so its connection and bindings still resolve to it.

Source code in graflo/architecture/contract/bindings/connectors.py
@abc.abstractmethod
def rendered(self, values: Mapping[str, Any]) -> Self:
    """A copy for one request: template fields filled from *values*, no ``values``.

    The copy keeps this connector's name and hash, so its connection and
    bindings still resolve to it.
    """
row_constants()
Source code in graflo/architecture/contract/bindings/connectors.py
def row_constants(self) -> list[dict[str, Any]]:
    return [
        {**self.row_annotations, **combination}
        for combination in self.value_combinations()
    ]
value_combinations()

One mapping per request, in request order; [{}] without values.

Source code in graflo/architecture/contract/bindings/connectors.py
def value_combinations(self) -> list[dict[str, Any]]:
    """One mapping per request, in request order; ``[{}]`` without ``values``."""
    if not self.values:
        return [{}]
    names = list(self.values)
    return [
        dict(zip(names, combination, strict=True))
        for combination in itertools.product(*(self.values[n] for n in names))
    ]

WatermarkConfig

Bases: ConfigBaseModel

Where an API source left off, and how to ask it to resume from there.

The source tracks the largest value of field over the rows it yields. Values compare as numbers when both parse as numbers, and as text otherwise, so timestamps must be written in one sortable form (ISO 8601 in one zone). The caller stores that value and renders it into the next run's query parameters; GraFlo keeps no run state itself.

Attributes:

Name Type Description
field str

Dot path of the value on each row.

param str

Query parameter that carries the value on the next run.

template str | None

Parameter text with a {value} placeholder; None sends the bare value.

format str | None

strftime pattern the value is rendered with, in UTC, before the template, for an API whose query language takes another time form than its records carry ("%Y-%m-%d %H:%M"). The stored value is read as ISO 8601 (Z, +00:00 or +0000 offsets, fractional seconds; no offset means UTC), or as epoch seconds when it is a number or a string of one. None renders it unchanged.

Source code in graflo/architecture/contract/bindings/connectors.py
class WatermarkConfig(ConfigBaseModel):
    """Where an API source left off, and how to ask it to resume from there.

    The source tracks the largest value of ``field`` over the rows it yields.
    Values compare as numbers when both parse as numbers, and as text
    otherwise, so timestamps must be written in one sortable form (ISO 8601 in
    one zone). The caller stores that value and renders it into the next run's
    query parameters; GraFlo keeps no run state itself.

    Attributes:
        field: Dot path of the value on each row.
        param: Query parameter that carries the value on the next run.
        template: Parameter text with a ``{value}`` placeholder; ``None`` sends
            the bare value.
        format: strftime pattern the value is rendered with, in UTC, before
            the template, for an API whose query language takes another time
            form than its records carry (``"%Y-%m-%d %H:%M"``). The stored
            value is read as ISO 8601 (``Z``, ``+00:00`` or ``+0000`` offsets,
            fractional seconds; no offset means UTC), or as epoch seconds when
            it is a number or a string of one.
            ``None`` renders it unchanged.
    """

    field: str = Field(..., min_length=1)
    param: str = Field(..., min_length=1)
    template: str | None = None
    format: str | None = None

    @field_validator("format")
    @classmethod
    def _check_format(cls, value: str | None) -> str | None:
        if value is None:
            return value
        directives = _STRFTIME_DIRECTIVE.findall(value)
        unknown = sorted({d for d in directives if d[1:] not in _STRFTIME_CODES})
        if unknown:
            raise ValueError(f"watermark format has unknown directives: {unknown}")
        if not [d for d in directives if d != "%%"]:
            raise ValueError(
                f"watermark format {value!r} has no strftime directive, such as %Y"
            )
        return value

    def format_value(self, value: Any) -> Any:
        """*value* as the API takes it: rendered by ``format``, else unchanged.

        Raises:
            ValueError: With ``format`` set, when *value* is neither ISO 8601
                nor epoch seconds.
        """
        if self.format is None:
            return value
        return _as_utc(value).strftime(self.format)

    def render(self, value: Any) -> dict[str, Any]:
        """Return the query parameters that resume from *value*."""
        value = self.format_value(value)
        if self.template is None:
            return {self.param: value}
        return {self.param: self.template.replace("{value}", str(value))}

Attributes

field = Field(..., min_length=1) class-attribute instance-attribute
format = None class-attribute instance-attribute
param = Field(..., min_length=1) class-attribute instance-attribute
template = None class-attribute instance-attribute

Methods:

format_value(value)

value as the API takes it: rendered by format, else unchanged.

Raises:

Type Description
ValueError

With format set, when value is neither ISO 8601 nor epoch seconds.

Source code in graflo/architecture/contract/bindings/connectors.py
def format_value(self, value: Any) -> Any:
    """*value* as the API takes it: rendered by ``format``, else unchanged.

    Raises:
        ValueError: With ``format`` set, when *value* is neither ISO 8601
            nor epoch seconds.
    """
    if self.format is None:
        return value
    return _as_utc(value).strftime(self.format)
render(value)

Return the query parameters that resume from value.

Source code in graflo/architecture/contract/bindings/connectors.py
def render(self, value: Any) -> dict[str, Any]:
    """Return the query parameters that resume from *value*."""
    value = self.format_value(value)
    if self.template is None:
        return {self.param: value}
    return {self.param: self.template.replace("{value}", str(value))}