Skip to content

Graph export and migration

Since 1.8.6, GraFlo treats existing graph databases as sources: introspect schema and data from Neo4j or ArangoDB (or a file backend) and load into any supported target — another LPG, PostgreSQL, or disk — with GraphEngine.migrate_graph(). No manifest YAML is required for graph-to-graph moves.

Since 1.8.7, GraFloBackendConfig adds a chunked on-disk backend that is both a durable export format and an optional intermediate store for large or repeated migrations.

Overview

Direction Entry point Source backends Target backends
Graph → graph GraphEngine.migrate_graph() Neo4j, ArangoDB, file backend Any supported DBType (LPG, PostgreSQL, file backend)
Graph → file backend GraphEngine.migrate_graph() Neo4j, ArangoDB, file backend GraFloBackendConfig
File backend → graph GraphEngine.migrate_graph() GraFloBackendConfig Any supported LPG target
File backend → relational GraphEngine.migrate_graph() GraFloBackendConfig PostgreSQL (vertex + junction edge tables)
Tabular/RDF/API → graph GraphEngine.ingest() / define_and_ingest() CSV / JSON / SQL / API manifest resources Any supported target
Resources → file backend GraphEngine.ingest() / define_and_ingest() CSV / JSON / SQL / API manifest resources GraFloBackendConfig
Graph schema only GraphEngine.infer_schema_from_graph() Neo4j, ArangoDB, file backend — (returns Schema)
In-memory export GraphEngine.export_graph() Neo4j, ArangoDB, file backend — (returns GraFloOutput)

Manifest sources vs graph sources. PostgreSQL, CSV, RDF, and APIs enter through a GraphManifest and actor pipelines (Example 5). Existing graph databases enter through introspection — see the Graph DB migration guide.

Quick start — graph → graph

Direct migration in one call (no file backend intermediate):

from graflo import GraphEngine, DBType
from graflo.db import Neo4jConfig, ArangoConfig, TigergraphConfig

engine = GraphEngine(target_db_flavor=DBType.ARANGO)
engine.migrate_graph(
    Neo4jConfig.from_docker_env(),   # source
    ArangoConfig.from_docker_env(),  # target
    recreate_schema=True,
    sample_limit=100,
)

# Other targets — same API, different config:
GraphEngine(target_db_flavor=DBType.TIGERGRAPH).migrate_graph(
    Neo4jConfig.from_env(), TigergraphConfig.from_env(), recreate_schema=True,
)

The engine introspects the source once, applies target Sanitizer rules, defines DDL, and writes via DBWriter.

Quick start — via file backend

Use a chunked on-disk backend for large graphs, dry-runs, or replay to multiple targets.

File backend layout

A GraFlo file backend directory is self-describing:

artifacts/neo4j-backend/
├── INDEX.json              # manifest: version, schema hash, chunk inventory
├── schema.yaml             # Schema only (no data)
├── vertices/
│   ├── person.000.jsonl.gz
│   └── person.001.jsonl.gz
└── edges/
    └── person__knows__person.000.jsonl.gz
  • Chunks — gzip-compressed JSONL (one JSON document per line)
  • Edges — filenames use {source}__{relation}__{target} when names are safe; INDEX.json keys follow the same convention
  • Default chunk size — 50 000 records per file (configurable on GraFloBackendConfig)

PostgreSQL remains a manifest source for 3NF schema inference and SQL ingestion (Example 5). As a migration target, PostgreSQL stores the logical graph as relational tables rather than native LPG structures.

from pathlib import Path

from graflo import GraphEngine, DBType
from graflo.db import Neo4jConfig, ArangoConfig, PostgresConfig
from graflo.db.graflo_backend.config import GraFloBackendConfig

neo4j = Neo4jConfig.from_env()       # or Neo4jConfig.from_docker_env()
arango = ArangoConfig.from_env()
postgres = PostgresConfig.from_env()
backend = GraFloBackendConfig(output_dir=Path("artifacts/neo4j-backend"))

engine = GraphEngine(target_db_flavor=DBType.ARANGO)

# --- Neo4j → file backend (export to disk) ---
engine.migrate_graph(neo4j, backend, recreate_schema=True)

# --- File backend → Arango migration ---
engine.migrate_graph(backend, arango, recreate_schema=True)

# --- File backend → Postgres (relational vertex + junction edge tables) ---
pg_engine = GraphEngine(target_db_flavor=DBType.POSTGRES)
pg_engine.migrate_graph(backend, postgres, recreate_schema=True)

Config loading. All *Config classes support from_env(), from_docker_env() (reads docker/<backend>/.env), or direct constructor arguments.

Pre-sanitize for a future target. Set target_flavor_hint on the file-backend config so schema.yaml is sanitized before it is written:

backend = GraFloBackendConfig(
    output_dir=Path("artifacts/for-arango"),
    target_flavor_hint=DBType.ARANGO,
)
engine.migrate_graph(neo4j, backend, recreate_schema=True)

See Example 13: GraFlo file backend for a runnable script including ingest() to disk.

Ingest resources into a file backend

GraFloBackendConfig works as an ingest() target the same way as ArangoDB or Neo4j — only the config changes:

from pathlib import Path

from suthing import FileHandle

from graflo import GraphEngine, GraphManifest, DBType
from graflo.db.graflo_backend.config import GraFloBackendConfig
from graflo.hq.caster import IngestionParams

manifest = GraphManifest.from_config(FileHandle.load("manifest.yaml"))
manifest.finish_init()

backend = GraFloBackendConfig(output_dir=Path("artifacts/csv-backend"))
engine = GraphEngine(target_db_flavor=DBType.GRAFLO_BACKEND)

engine.define_and_ingest(
    manifest=manifest,
    target_db_config=backend,
    ingestion_params=IngestionParams(clear_data=True),
    recreate_schema=True,
)

Inspect the result without loading everything into memory:

from graflo.architecture.backend import GraFloBackendReader

reader = GraFloBackendReader(Path("artifacts/csv-backend"))
index = reader.read_index()
print(index.vertices)  # record counts and chunk paths per vertex type

GraFloBackendConfig and Connection API

Type Role
GraFloBackendConfig DBConfig subclass: output_dir, chunk_size, optional target_flavor_hint
GraFloBackendConnection Connection implementation registered in ConnectionManager
GraFloBackendWriter / GraFloBackendReader Low-level I/O primitives in graflo.architecture.backend
GraFloIndex Pydantic model for INDEX.json

Write path (target): init_db → upsert_docs_batch / insert_edges_batch → close() flushes INDEX.json.

Read path (source): introspect_graph_schema() reads schema.yaml; fetch_all_docs / fetch_all_edges stream from gzip JSONL chunks.

Use ConnectionManager.graph_export_flavors() to list backends with graph export support — includes DBType.GRAFLO_BACKEND alongside Neo4j and ArangoDB.

GraFloOutput (in-memory)

GraFloOutput pairs a full Schema with a GraphContainer. GraphEngine.export_graph() still returns it for small graphs or programmatic use:

from graflo.hq import GraphEngine
from graflo.db.connection import Neo4jConfig

output = engine.export_graph(Neo4jConfig(...))
assert output.core_schema is output.graph_schema.core_schema
assert output.data.vertices

For durable, large exports prefer migrate_graph(source, GraFloBackendConfig(...)) instead of holding the full graph in memory or a single YAML file.

GraphContainer edge keys in JSON

In Python, edge keys are tuples (source, target, relation). When serialized to JSON, each key becomes a compact JSON array string, for example ["person","department","works_in"].

Migrate graph → graph

This is the core 1.8.6 workflow — already shown in Quick start — graph → graph above. GraphEngine.migrate_graph() exports from the source in one connection pass, sanitizes the schema for the target flavor (unless the target is a file backend without target_flavor_hint), defines DDL, and loads data:

engine.migrate_graph(
    Neo4jConfig(...),          # source
    ArangoConfig(...),         # target
    recreate_schema=True,
    clear_data=False,
    sample_limit=100,
)

The engine reuses a single source connection for introspection and export, and applies target Sanitizer rules once before writing.

Migrate graph → PostgreSQL

PostgreSQL targets map each vertex type to a table and each edge type to a junction table named {source}_{target}_{relation}_edges with source_id, target_id, optional weight columns, and a surrogate primary key for parallel edges.

Capability guard

ConnectionManager.open_graph_connection() rejects backends without graph export support:

from graflo.db.manager import ConnectionManager

ConnectionManager.graph_export_flavors()
# e.g. [DBType.NEO4J, DBType.ARANGO, DBType.GRAFLO_BACKEND]

TigerGraph, FalkorDB, Memgraph, and NebulaGraph remain supported targets for manifest-driven ingestion and migrate_graph(); native graph-source introspection/export is not yet implemented for those live databases (use a file backend as an intermediate store).