Manifest evolution

GraFlo provides contract-level operations that transform a validated GraphManifest into a new manifest: logical vertices and edges, ingestion resources, optional bindings wiring, and the database profile are updated together. This is not an in-database migration of existing graph data; the intended workflow is to publish the new manifest and reingest from sources.

Identity and validation

• Stable hash: use manifest_hash from graflo.migrate.io (see graflo.migrate.io) to compare the composed schema, ingestion_model, and bindings blocks before and after an evolution.
• Validation: apply_evolution in graflo.architecture.evolution returns a deep copy and runs GraphManifest.finish_init() by default so the same cross-block checks apply as when loading YAML. API reference: graflo.architecture.contract.manifest.

Operations

Operation	Summary
Remove vertices	Drops named vertex types, removes incident edges, prunes ingestion resources that reference removed types (including vertex_router type_map / vertex_from_map via structured pipeline scan), trims merge_collections, filters resource_connector rows, and updates db_profile. Fails if ingestion would be left with no resources.
Merge vertices	Merges one or more source vertex types into a target name (into). If into already exists, sources are merged into it; otherwise a new vertex type is built from all sources. Endpoints on edges are rewritten and duplicate (source, target, relation) edge kinds are merged. Resource pipelines, infer_edge_only / infer_edge_except, and extra_weights are rewritten; db_profile logical keys follow the merge. Conflicting field types or default-value maps raise an error.
Rename vertices	Renames logical vertex type names across schema, edge endpoints, ingestion pipelines/selectors, and bindings resource references.
Rename relations	Renames logical edge relation values across schema, ingestion selectors/pipelines, and db_profile edge metadata.
Rename resources	Renames ingestion resource names and all bindings references (connectors[].resource_name, resource_connector[].resource).
Remove edges	Removes edge types by relation name from schema, db_profile.edge_specs, default_property_values.edges, and ingestion relation selectors.
Merge edges	Canonicalizes multiple relation names into one relation, then merges duplicate edge identities and deduplicates edge/profile defaults.
Rename vertex fields	Per-vertex {old_field: new_field} maps: updates schema field names, identities, db_profile index specs, and ingestion (vertex from, transform.rename targets) so documents still use the source column names where a reverse map is injected.
Remove vertex fields	Removes vertex properties, prunes vertex/edge index references, and rewrites ingestion references (from, keep_fields, vertex_weights).
Add vertex fields	Adds properties to existing vertices for schema enrichment and migration planning.
Rename edge fields	Per-relation edge property renames across schema edge properties/identities, db_profile edge indexes/defaults, and edge actor properties payloads.
Remove edge fields	Removes per-relation edge properties, prunes edge index/default references, and rewrites edge actor properties.
Add edge fields	Adds properties to existing relations for edge-schema enrichment.
Add inverse edges	For each directed forward relation R -> R_inv, appends inverse schema edges and mirrors ingestion (pipeline EdgeActor steps including dynamic endpoints, relation_field, redefined relation_map, nested descend), infer_edge_only / infer_edge_except, extra_weights, and db_profile. Skips directed: false, TigerGraph edge_specs[*].reverse_edge, and existing inverse triples.
Project manifest	Keeps a logical subgraph by vertex names and/or edge triples (source, target, relation). Prunes isolated vertex types from keep_vertices when they have no surviving edges (connectivity: induced_prune). Cascades to schema, db_profile, ingestion (pipeline steps, infer selectors, extra_weights), and bindings. Optional keep_resources filters ingestion resources. Inverse edges are not auto-kept. Fails if ingestion would be left empty.
Replace identity	Per-vertex identity policy swap covering both field-set and mode changes (natural / hash / assigned / blank). retire decides what becomes of the old field-set — demote (default) turns it into a secondary identity, keep leaves it as plain properties, drop removes it. endpoints decides whether edge steps follow the new identity (follow_new, default) or stay pinned to the demoted one (pin_to_retired). Drops db_profile indexes that encoded the retired identity. See Replacing a vertex identity.
Add / remove secondary identities	Declares or withdraws alternate lookup keys on existing vertices. Each field-set's non-unique index is derived by Schema.finish_init, so adding one needs no index authoring; removing one drops the derived index explicitly. Removal is rejected while an edge step still selects the field-set.
Replace edge identities	Replaces Edge.identities (uniqueness keys) per (source, target, relation). No retire policy — edge identities have no lookup plane. Non-endpoint tokens are merged into edge properties by Edge.finish_init.
Add vertices / add edges	Introduces new logical vertex types and edge relations unarily — the counterpart to what ComposeManifestsOp could previously only do binarily. Rejects existing names/triples and unknown endpoints.
Retarget edges	Changes which vertex types an edge connects, preserving its properties, identities, directed flag, and db_profile physical spec — all of which a remove-plus-add would lose. Rewrites the EdgeId in edge_config, edge_specs, and pipeline edge steps, keyed on the full triple so a different relation between the same types is untouched.
Change field types	Sets Field.type / item_type on vertex or edge properties. Validated against the profile's db_flavor via graflo.db.field_type_support, so an unsupported LIST target fails at op time rather than at define time. Refuses to make an identity field a LIST.
Add / remove vertex & edge indexes	Authors db_profile.vertex_indexes and edge_specs[].indexes directly. Indexes derived from secondary_identities cannot be removed this way — they would be re-registered by the next finish_init, so the op points at remove secondary identities instead.
Set edge directed	Sets Edge.directed on selected triples. Load-bearing for replay: directed decides what add inverse edges may duplicate.
Sanitize	Target-DBType policy: reserved-word-safe names on DatabaseProfile, reserved vertex field renames, and (for TigerGraph) consistent identity tuples per edge relation. This is the same work graflo.hq.sanitizer.Sanitizer applies by building a single SanitizeOp.
Compose manifests	Binary union of two full GraphManifests (schema and resources/bindings) via ComposeManifestsOp + compose_manifests(left, right, op). Consumes explicit equivalence maps only (no semantic inference): vertex→vertex, property alignment, optional derived identity, relation equivalences, resource renames / name_conflict. Distinct from unary MergeVerticesOp. Rejected by unary apply_evolution.

Compose two manifests

GraFlo stays deterministic: an external tool (or a human) may propose equivalences; core only applies them.

from graflo.architecture.evolution import (
    ComposeManifestsOp,
    PropertyEquivalence,
    RelationEquivalence,
    VertexEquivalence,
    compose_manifests,
)

composed = compose_manifests(
    left,
    right,
    ComposeManifestsOp(
        vertices=[
            VertexEquivalence(
                left="Client",
                right="Customer",
                into="Person",
                properties=[
                    PropertyEquivalence(
                        left="client_id", right="customer_id", into="id"
                    ),
                    PropertyEquivalence(
                        left="email", right="email_addr", into="email", identity=True
                    ),
                ],
                identity=["email"],  # optional explicit natural key; else merge + flags
            )
        ],
        relations=[RelationEquivalence(left="places", right="billed", into="activity")],
        resource_renames={},  # right resource name -> composed name
        name_conflict="error",  # or "prefix_right"
    ),
)

Empty vertices / relations yields a disjoint union (both resource sets and bindings retained), subject to collision policy.

API

from graflo.architecture.evolution import (
    AddInverseEdgesOp,
    ComposeManifestsOp,
    EdgeSelector,
    MergeEdgesOp,
    MergeVerticesOp,
    ProjectManifestOp,
    RenameRelationsOp,
    RemoveVerticesOp,
    SanitizeOp,
    apply_evolution,
    apply_sanitize,
    compose_manifests,
)
from graflo.migrate.io import manifest_hash
from graflo.onto import DBType

b = apply_evolution(
    a,
    [
        RemoveVerticesOp(op="remove_vertices", names=["legacy_vertex"]),
        MergeVerticesOp(op="merge_vertices", sources=["user", "person"], into="party"),
        RenameRelationsOp(op="rename_relations", relations={"works_at": "employed_by"}),
        MergeEdgesOp(op="merge_edges", sources=["employee_of"], into="employed_by"),
        AddInverseEdgesOp(
            op="add_inverse_edges",
            relations={"employed_by": "employs"},
        ),
    ],
    bump_version=True,  # default: increment schema metadata MINOR (see bump_semver_minor)
)

assert manifest_hash(a) != manifest_hash(b)

# Or sanitize an existing GraphManifest (same op `Sanitizer` uses internally):
apply_sanitize(manifest, SanitizeOp(db_flavor=DBType.TIGERGRAPH))

• bump_version: when True or "minor" (default), increments the numeric MAJOR.MINOR.PATCH prefix of schema.metadata.version if present (prerelease suffix preserved). Pass bump_version=False to leave the version string unchanged.
• Imports: graflo.architecture.evolution re-exports the ops and apply helpers; lower-level functions such as apply_remove_vertices, apply_merge_vertices, apply_rename_relations, apply_add_inverse_edges, apply_rename_vertex_properties, and apply_sanitize mutate a manifest in place (used mainly internally and by Sanitizer). Cross-manifest compose uses compose_manifests (not unary apply_evolution).

Tutorial: relation and property evolution

Use these recipes when converging ontologies or normalizing an existing manifest.

1) Rename relation labels (same semantics, new vocabulary)

from graflo.architecture.evolution import RenameRelationsOp, apply_evolution

renamed = apply_evolution(
    manifest,
    [RenameRelationsOp(relations={"works_at": "employed_by"})],
    bump_version=False,
)

2) Merge relation labels (canonicalization)

Use this when multiple labels represent the same concept: works_for, employee_of, employed_by -> employed_by.

from graflo.architecture.evolution import MergeEdgesOp, apply_evolution

canonical = apply_evolution(
    manifest,
    [
        MergeEdgesOp(
            sources=["works_for", "employee_of"],
            into="employed_by",
        )
    ],
    bump_version=False,
)

3) Evolve relation payload fields

from graflo.architecture.evolution import (
    AddEdgePropertiesOp,
    RemoveEdgePropertiesOp,
    RenameEdgePropertiesOp,
    apply_evolution,
)

updated = apply_evolution(
    manifest,
    [
        RenameEdgePropertiesOp(
            renames={"employed_by": {"since": "started_at"}},
        ),
        RemoveEdgePropertiesOp(
            removals={"employed_by": ["deprecated_score"]},
        ),
        AddEdgePropertiesOp(
            additions={"employed_by": ["confidence"]},
        ),
    ],
    bump_version=False,
)

4) Add new vertex fields for enrichment

from graflo.architecture.evolution import AddVertexPropertiesOp, apply_evolution

enriched = apply_evolution(
    manifest,
    [AddVertexPropertiesOp(additions={"person": ["canonical_id", "normalized_name"]})],
    bump_version=False,
)

5) Add inverse edge relations (bidirectional modeling)

Use this when a forward relation already exists in schema and ingestion (for example person --works_at--> company) and you want the reverse kind without hand-authoring every mirror (company --employs--> person).

from graflo.architecture.evolution import AddInverseEdgesOp, apply_evolution

bidirectional = apply_evolution(
    manifest,
    [
        AddInverseEdgesOp(
            relations={"works_at": "employs"},
        )
    ],
    bump_version=False,
)

For each directed schema edge whose relation is a key in the map, the op appends an inverse edge with swapped endpoints and the mapped relation name, copying properties, identities, and directed: true. The op does not run when:

• the forward edge has directed: false (use one undirected logical edge or TigerGraph UNDIRECTED EDGE instead), or
• the forward edge’s TigerGraph edge_specs[*].reverse_edge is already set (TigerGraph owns the paired reverse type via WITH REVERSE_EDGE).

What gets mirrored in ingestion

Location	Inverse behavior
Static pipeline edge step (from/to/relation)	Duplicate step with swapped endpoints and inverse relation
Dynamic edge step (source_role/target_role, mixed static+dynamic)	Duplicate step with swapped roles/static sides; match_source/match_target swapped
relation_field	Same field name on the inverse step
relation_map on the step	Redefined: same raw keys map to inverse canonical names (EMPLOYED_BY: employed_by forward → EMPLOYED_BY: employs after employed_by -> employs)
links	Each link item inverted independently
Nested descend pipelines	Recursively mirrored
infer_edge_only / infer_edge_except / extra_weights	Static triple specs appended when missing

Dynamic EdgeActor example (after AddInverseEdgesOp(relations={"employed_by": "employs"})):

Forward step:

- edge:
    source_role: source
    target_role: target
    relation_field: relation_type
    relation_map:
      EMPLOYED_BY: employed_by

Appended inverse step:

- edge:
    source_role: target
    target_role: source
    relation_field: relation_type
    relation_map:
      EMPLOYED_BY: employs

Choosing a bidirectional strategy (see also Core components — Edge):

Goal	Approach
Portable across DBs	Two logical directed edges + AddInverseEdgesOp
TigerGraph-native pair, single load path	One logical edge + db_profile.edge_specs[*].reverse_edge
Truly symmetric (friends, co-authors)	One logical edge with directed: false → UNDIRECTED EDGE on TigerGraph

6) Project to a subgraph slice

Use when you need a smaller manifest that retains only specific vertex types and edge triples (for example agent experiments or publishing a focused contract):

from graflo.architecture.evolution import (
    EdgeSelector,
    ProjectManifestOp,
    apply_evolution,
)

slice = apply_evolution(
    manifest,
    [
        ProjectManifestOp(
            keep_vertices=["person", "company"],
            keep_edges=[
                EdgeSelector(source="person", target="company", relation="works_at"),
            ],
        )
    ],
    bump_version=False,
)

With keep_vertices only, vertex types listed but not incident to any surviving edge are dropped (connectivity: induced_prune). List inverse edge triples explicitly in keep_edges when you need them; they are not inferred automatically.

Choosing RenameRelationsOp vs MergeEdgesOp

• Use RenameRelationsOp when there is a one-to-one label replacement.
• Use MergeEdgesOp when multiple relation labels should collapse into one canonical relation.
• Use AddInverseEdgesOp when forward and reverse relations should coexist with different labels (not a rename of the same edge kind).
• RenameRelationsOp and MergeEdgesOp propagate to schema, DatabaseProfile (edge_specs, defaults/indexes), and ingestion selectors/resources. AddInverseEdgesOp also propagates to db_profile and does not rename existing relations; it only adds missing inverse edges and ingestion mirrors.

Replacing a vertex identity

ReplaceIdentityOp is the one operation that touches how a vertex is keyed, so it carries an explicit policy for the identity being retired.

from graflo.architecture.evolution import ReplaceIdentityOp, apply_evolution

evolved = apply_evolution(
    manifest,
    [
        ReplaceIdentityOp(
            vertices={
                "party": {
                    "to": {"mode": "natural", "identity": ["party_uid"]},
                    "retire": "demote",  # default
                    "retire_as": "by_legacy",
                    "endpoints": "follow_new",  # default
                }
            }
        )
    ],
)

After this, party upserts on party_uid, and the old legacy_id key survives as a secondary identity named by_legacy — usable by any edge step that names it, and automatically indexed. Edge steps that were matching on the primary identity now match on party_uid; pass endpoints: "pin_to_retired" to keep them on by_legacy instead.

The to block reaches every identity mode:

to.mode	Required	Result
natural	identity: [...]	Named properties are the key
hash	hash_from: [...]	Deterministic synthetic id digested from those fields
assigned	—	Intentional UUID primary key
blank	—	Auto-generated placeholder ID

Things worth knowing before you reach for it:

• Demotion is downgraded to keep when the old identity was synthetic (hash, assigned, blank) or already equals the new one — demoting a generated id would create a lookup key no source carries. The op logs a warning when it does this.
• mode: blank cannot demote at all. A blank vertex may not declare secondary identities, so the op raises and points at keep / drop.
• New identity fields must already be declared. Vertex.set_identity would happily synthesise them as untyped fields, which makes an empty column the primary key; the op refuses and points at AddVertexPropertiesOp.
• A no-op replacement does not bump the schema version.

This is a contract-level change only. To see what it implies for a populated database, diff the schemas — a mode change or a non-widening key swap emits CHANGE_VERTEX_IDENTITY and REKEY_VERTEX, both CRITICAL and blocked by MigrationPlanner unless high risk is explicitly allowed.

Revision chains

Individual ops rewrite a manifest. A revision chain records sequences of them so a manifest's history can be stored, replayed and verified.

The model is a git log, not an Alembic script. Alembic's core abstraction is a reversible upgrade() / downgrade() pair, and GraFlo cannot honour that: merge_vertices discards which source each property came from, change_field_types discards the previous type, sanitize and project_manifest drop material outright. A downgrade that quietly produces a different manifest is worse than none. So revisions move forward, and going back means replaying from the base.

Deriving a change set

from graflo.architecture.evolution import diff_manifests_verified

ops, warnings = diff_manifests_verified(base, target)

diff_manifests is the only producer of ManifestOp values — the migrate-plane SchemaDiff emits description records that cannot be applied, and never looks at ingestion_model or bindings. The _verified variant additionally checks the

replay invariant: manifest_hash(apply_evolution(base, ops)) == manifest_hash(target)

and reports the residual when it does not hold, rather than letting a partial change set pass as complete.

Renames need hints. A dropped mail plus an added email is structurally identical to a rename, and guessing would turn a data-preserving rename into a destructive drop. Supply RenameHints when the intent is known:

ops, _ = diff_manifests(
    base, target, hints=RenameHints(vertex_properties={"party": {"mail": "email"}})
)

Recording and replaying

from graflo.architecture.evolution import build_revision, RevisionChain, apply_revisions

r1 = build_revision(base, ops, label="add email")
chain = RevisionChain(revisions=[r1])
restored = apply_revisions(base, chain)  # verifies every recorded hash

A Revision carries its ops, its parent (down_revision), and the manifest hash before and after it. build_revision applies the ops rather than trusting them, so both hashes describe a transition that actually happened. Revision ids are content-derived, so regenerating the same change set yields the same id instead of a duplicate under a new name.

RevisionChain validates that the links form one linear chain and that each revision's manifest_hash_before equals its predecessor's manifest_hash_after — a chain whose steps do not meet cannot describe one history. apply_revisions re-checks both hashes at every step, so a base that has drifted fails loudly instead of producing a plausible wrong answer.

Going back

downgrade_to(chain, target_revision, base=base)  # exact, always preferred
downgrade_to(chain, target_revision, current=head)  # inverses; may refuse

Replaying from the base is correct for every chain. Inversion is a fallback for when no base is available, and it is deliberately partial: invert_op returns None for the lossy ops, and a removal cannot be undone from the post-state because the data it would restore is gone. Both cases raise with the reason rather than returning an approximation.

Reversible	Irreversible
add ↔ remove: vertices, edges, vertex/edge properties, vertex/edge indexes	merge_vertices, merge_edges
rename: vertices, relations, resources, vertex/edge properties	change_field_types
set_edge_directed, add_inverse_edges, retarget_edges	sanitize, project_manifest
replace_identity (with retire: keep), add/remove secondary identities	compose_manifests (binary)

Storage and CLI

FileRevisionStore keeps one YAML file per revision under .graflo/revisions/, named <index>_<revision>_<slug>.yaml. Order comes from the parent links, not the filenames.

uv run revision new --from-manifest base.yaml --to-manifest target.yaml --label "add order"
uv run revision history
uv run revision verify --base base.yaml --against target.yaml
uv run revision apply --base base.yaml --upto <revision> --output-path out.yaml
uv run revision downgrade --base base.yaml --upto <revision>

Ops are serialized through graflo.architecture.evolution.codec, which verifies its own output: it emits the compact form, re-validates it, and falls back to the full form when the compact one would not load back. That is not defensive habit — IdentityTarget.mode has a default, so dropping defaults silently strips the discriminator a nested union needs. A change set that cannot be read back is the one failure this layer must not have.

Not in scope

Applying a revision chain to a live database. migrate remains the DB-facing plane, and extending it beyond additive DDL is tracked separately. Revisions describe the contract.

Scope notes

• Transforms: bodies of named transforms are not rewritten when vertex field names change during a merge; that remains an authoring concern. Use RenameVertexPropertiesOp / SanitizeOp when you need coordinated field rewrites at the manifest boundary.
• Identity ops are contract-level too: ReplaceIdentityOp rewrites the manifest, it does not re-key stored vertices. Propagating identity changes to a live database is not yet supported.
• Bindings: connector definitions are unchanged; only resource_connector rows pointing at dropped resources are removed after a remove operation.

See also

• Creating a Manifest — manifest structure
• Concepts overview — GraphManifest role in the pipeline