Skip to content

API env wiring

You read records from REST APIs, and each API has an address and credentials that belong neither in the manifest nor in your code. This guide keeps them in environment variables: the manifest names each API by a label, and one call loads the address and credentials of every label. Afterwards you can run the same manifest against a test API and a production API by changing environment variables only.

What you need

  • A manifest whose bindings read from REST APIs; see API connector for the connector itself.
  • A shell or a process manager where you can set environment variables.

Steps

1. Name each API by a label

In bindings, give every API connector only a path, and map the connector to a label with connector_connection. The label is a conn_proxy, a name for a connection whose details are supplied at run time (see connection proxy):

bindings:
  connectors:
    - name: work_orders_api
      path: /api/work-orders
      resource_name: work_orders
    - name: readings_api
      path: /api/readings
      resource_name: readings
  connector_connection:
    - connector: work_orders_api
      conn_proxy: maintenance_api
    - connector: readings_api
      conn_proxy: sensor_feed

Here work orders come from the maintenance system's API and readings from the sensor feed's API. Neither address appears in the manifest.

2. Set the variables

Each label becomes a variable prefix: upper case, - replaced by _, and a trailing _. maintenance_api reads MAINTENANCE_API_*, and a label sensor-feed would read SENSOR_FEED_*.

Variable Required Meaning
{PREFIX}BASE_URL yes Base URL; the connector's path is appended to it
{PREFIX}AUTH_TYPE no, default bearer bearer, basic, digest or api_key
{PREFIX}TOKEN for bearer and api_key The token or key
{PREFIX}USERNAME, {PREFIX}PASSWORD for basic and digest The credentials
{PREFIX}HEADER_NAME no, default Authorization The header that carries the token or key
{PREFIX}PREFIX no, default Bearer Text put before a bearer token in the header

With none of TOKEN, USERNAME and PASSWORD set, the API is read without credentials and no authentication header is sent.

export MAINTENANCE_API_BASE_URL=https://maintenance.example.com
export MAINTENANCE_API_TOKEN=...
export SENSOR_FEED_BASE_URL=https://sensors.example.com
export SENSOR_FEED_AUTH_TYPE=api_key
export SENSOR_FEED_HEADER_NAME=X-API-Key
export SENSOR_FEED_TOKEN=...

The same rule covers every kind of source. The connectors that use a label decide which variables it reads:

Connectors using the label Variables Required
API the table above BASE_URL
Table (PostgreSQL) {PREFIX}URI, {PREFIX}USERNAME, {PREFIX}PASSWORD, {PREFIX}DATABASE, {PREFIX}SCHEMA_NAME URI
SPARQL {PREFIX}USERNAME, {PREFIX}PASSWORD (the endpoint is on the connector) none
Kafka {PREFIX}BOOTSTRAP_SERVERS, {PREFIX}SECURITY_PROTOCOL, {PREFIX}SASL_*, {PREFIX}CLIENT_ID BOOTSTRAP_SERVERS

A label used by connectors of two kinds is refused.

3. Load the variables and ingest

from suthing import FileHandle

from graflo import GraphEngine, GraphManifest
from graflo.connections import ArangoConfig, InMemoryConnectionProvider

manifest = GraphManifest.from_config(FileHandle.load("manifest.yaml"))
manifest.finish_init()

provider = InMemoryConnectionProvider.from_env(manifest.require_bindings())

conn_conf = ArangoConfig.from_env()
engine = GraphEngine(target_db_flavor=conn_conf.connection_type)
engine.define_and_ingest(
    manifest=manifest,
    target_db_config=conn_conf,
    connection_provider=provider,
    recreate_schema=True,
)

InMemoryConnectionProvider.from_env finds every label the bindings use and reads that label's variables. During the ingest each connector looks up its label in the bindings and takes the address and credentials from the provider.

4. Use other variable names, if you must

When the variables already exist under other names, map a label to its prefix:

provider = InMemoryConnectionProvider.from_env(
    manifest.require_bindings(),
    env_prefix_map={"maintenance_api": "WORK_ORDERS_"},
)

maintenance_api then reads WORK_ORDERS_BASE_URL and the rest; other labels keep the default rule.

What you should see

To check the wiring without contacting any API, print what each connector resolved to:

from graflo.connections import RestApiConnConfig, connector_config

bindings = manifest.require_bindings()
for connector in bindings.connectors:
    api = connector_config(connector, bindings, provider, RestApiConnConfig)
    print(connector.name, api.base_url + connector.path)
work_orders_api https://maintenance.example.com/api/work-orders
readings_api https://sensors.example.com/api/readings

Problems show up as one ValueError from from_env that lists every label at fault: a missing BASE_URL names the variable, an unknown AUTH_TYPE lists the valid ones, and a label shared by connectors of different kinds is named.