How do I keep database credentials out of the manifest?

Your manifest says which PostgreSQL tables feed which parts of the graph. You want to keep it in version control and share it, but the database address, user and password must not be in it, and they differ between your laptop and production.

The manifest therefore names the database by a label, here shop_db. When the program runs, it supplies the connection settings for that label. The same manifest then works against any database that has the tables, and it never contains a password.

flowchart LR
    subgraph manifest["manifest_shop.yaml"]
        connector["connector: purchases<br/>table_name: purchases"] --> label["conn_proxy: shop_db"]
    end
    subgraph program["ingest.py, at run time"]
        settings["PostgresConfig<br/>host, user, password"]
    end
    label -. resolved to .-> settings

What you need

• GraFlo installed (pip install graflo).
• A running PostgreSQL and a running ArangoDB. The repository ships containers for both; see docker/README.md.
• The shop data of example 09. The script loads ../09-infer-from-postgres/data/shop.sql into PostgreSQL, which drops and recreates the tables users, products, purchases and follows.

The data

The shop of example 09: 4 users, 4 products, 6 purchases (user to product) and 5 follows (user to user). manifest_shop.yaml describes the same graph that example 09 infers, written by hand.

Steps

1. Name each table's database by a label

In the bindings block, each connector names its table; connector_connection gives every connector the label shop_db:

bindings:
    connectors:
    -   name: users
        table_name: users
        resource_name: users
    # ... products, purchases, follows
    connector_connection:
    -   connector: users
        conn_proxy: shop_db
    # ... the same for products, purchases, follows

connector refers to a connector by its name, not to a resource. Nothing in the manifest says where shop_db is.

2. Supply the settings for the label

ingest.py reads the PostgreSQL settings from the container's configuration and registers them for shop_db:

postgres_conf = PostgresConfig.from_docker_env()

provider = InMemoryConnectionProvider()
provider.bind_single_config_for_bindings(
    bindings=manifest.require_bindings(),
    conn_proxy="shop_db",
    config=PostgresGeneralizedConnConfig(config=postgres_conf),
)

The connection provider maps each connector with the label shop_db to these settings. Outside this example, build PostgresConfig from your own secret store, or with PostgresConfig.from_env() from environment variables that start with POSTGRES_.

3. Ingest with the provider

engine.define_and_ingest(
    manifest=manifest,
    target_db_config=conn_conf,
    ingestion_params=IngestionParams(clear_data=True),
    recreate_schema=True,
    connection_provider=provider,
)

4. Run it

cd examples/11-connection-proxy
uv run python ingest.py

What you should see

The same graph as example 09:

	Count
users vertices	4
products vertices	4
purchases edges, users to products	6
follows edges, users to users	5

What goes wrong

The label has no settings. If define_and_ingest gets no provider, or one without shop_db, GraFlo cannot reach the tables. The run stops before it reads anything:

ValueError: Registry build failed in strict mode:
- Failed to register SQL source for resource 'users' (connector 'users'): no PostgreSQL connection configuration for table 'users'
- Failed to register SQL source for resource 'products' (connector 'products'): no PostgreSQL connection configuration for table 'products'
- Failed to register SQL source for resource 'purchases' (connector 'purchases'): no PostgreSQL connection configuration for table 'purchases'
- Failed to register SQL source for resource 'follows' (connector 'follows'): no PostgreSQL connection configuration for table 'follows'

Also possible

• Several databases: give their connectors different labels, register each label with provider.register_generalized_config(conn_proxy=..., config=...), then call provider.bind_from_bindings(bindings=...) once.
• The same pattern holds for SPARQL endpoints, REST APIs and Kafka topics. Example 12 reads the settings of each label from environment variables.

What to read next

• API sources from environment variables.
• Glossary: connection proxy and connection provider.

Files

The example lives in examples/11-connection-proxy.

manifest_shop.yaml

schema:
    metadata:
        name: shop
    graph:
        vertex_config:
            vertices:
            -   name: users
                properties: [id, name, email, created_at]
                identity: [id]
            -   name: products
                properties: [id, name, price, description, created_at]
                identity: [id]
        edge_config:
            edges:
            -   source: users
                target: products
                relation: purchases
                properties: [purchase_date, quantity, total_amount]
            -   source: users
                target: users
                relation: follows
                properties: [created_at]
ingestion_model:
    resources:
    -   name: users
        pipeline:
        -   vertex: users
    -   name: products
        pipeline:
        -   vertex: products
    -   name: purchases
        pipeline:
        -   vertex: users
            from:
                id: user_id
        -   vertex: products
            from:
                id: product_id
    -   name: follows
        pipeline:
        -   vertex: users
            from:
                id: follower_id
        -   vertex: users
            from:
                id: followed_id
bindings:
    connectors:
    -   name: users
        table_name: users
        resource_name: users
    -   name: products
        table_name: products
        resource_name: products
    -   name: purchases
        table_name: purchases
        resource_name: purchases
    -   name: follows
        table_name: follows
        resource_name: follows
    connector_connection:
    -   connector: users
        conn_proxy: shop_db
    -   connector: products
        conn_proxy: shop_db
    -   connector: purchases
        conn_proxy: shop_db
    -   connector: follows
        conn_proxy: shop_db

ingest.py

"""How do I keep database credentials out of the manifest?

Reads ``manifest_shop.yaml``, whose connectors name the shop database only by the
label ``shop_db``. Supplies the PostgreSQL connection settings for that label at
run time, loads the sample shop of example 09 into PostgreSQL and writes the
graph to ArangoDB. Run it from this directory:

    uv run python ingest.py
"""

from pathlib import Path

from suthing import FileHandle

from graflo import GraphManifest
from graflo.connections import (
    ArangoConfig,
    InMemoryConnectionProvider,
    PostgresConfig,
    PostgresGeneralizedConnConfig,
)
from graflo.db.postgres.util import load_schema_from_sql_file
from graflo.hq import GraphEngine, IngestionParams

SHOP_SQL = (
    Path(__file__).resolve().parents[1] / "09-infer-from-postgres" / "data" / "shop.sql"
)

manifest = GraphManifest.from_config(FileHandle.load("manifest_shop.yaml"))
manifest.finish_init()

# The credentials: read from the settings of the PostgreSQL container, not from
# the manifest.
postgres_conf = PostgresConfig.from_docker_env()
load_schema_from_sql_file(
    config=postgres_conf, schema_file=SHOP_SQL, continue_on_error=False
)

# Give the label "shop_db" its connection settings.
provider = InMemoryConnectionProvider()
provider.bind_single_config_for_bindings(
    bindings=manifest.require_bindings(),
    conn_proxy="shop_db",
    config=PostgresGeneralizedConnConfig(config=postgres_conf),
)

conn_conf = ArangoConfig.from_docker_env()
engine = GraphEngine(target_db_flavor=conn_conf.connection_type)
engine.define_and_ingest(
    manifest=manifest,
    target_db_config=conn_conf,
    ingestion_params=IngestionParams(clear_data=True),
    recreate_schema=True,
    connection_provider=provider,
)