Version control¶

A manifest changes over time, often in more than one hand. This page shows how GraFlo records that history: how a manifest gets a content address, how change sets become commits, how you go back to an earlier version, and how two lines of change to the same manifest are reconciled. After reading it you can keep a manifest's history in a commit store and replay, verify and merge it from the shell or from Python.

Two operations here combine manifests, and they are different. A three-way merge reconciles two branches of one manifest's history (merge_three_way, graflo merge3, the version control example (22)); a union combines two unrelated manifests (merge_manifests, graflo merge, the manifest union example (20)), and is described on Merging manifests.

History moves forward, like a git log, not like a migration script with an upgrade() and a downgrade() for every step. A reversible pair per step is not possible here: merge_vertices discards which source each property came from, change_field_types discards the previous type, sanitize overwrites the storage names it replaces, and project_manifest drops material outright. A downgrade that produces a different manifest from the one you started with is worse than none. So going back means replaying from an earlier version.

Nothing here touches a database. A history records changes to the manifest; see Schema migration for changes to a database.

Content addressing¶

Two manifests that describe the same graph hash equal:

from graflo.architecture.evolution import manifest_hash

manifest_hash(a) == manifest_hash(b)  # same graph, however each was reached

The hash covers the schema, the ingestion model and the bindings. It leaves out the manifest's name, version and provenance. to_minimal_canonical_dict() already ignores defaults, None values, aliases and key order. What it keeps is list order, and most lists in a manifest are sets written in some order, so two identical schemas authored in different orders, or one authored and one replayed, would hash differently. canonical_payload adds that normalization.

Sorted or preserved¶

Which lists may be sorted is decided per field, in LIST_ORDER, which classifies every list field reachable from GraphManifest.

The two possible mistakes are not equally bad. Sorting a list whose order matters makes two different manifests hash equal, and nothing downstream can detect it. Preserving the order of a list whose order does not matter only misses a match, which is visible and harmless. So a doubtful field is preserved, and every sorted field states why its order does not matter.

Sorted	Preserved
vertices, edges, properties, secondary identities	resource pipelines (an ordered program)
resource and transform registries, bindings entries	`Vertex.identity`: a database addresses an endpoint through the first key field
index sets, edge specs	the columns of a compound index
semantic `exact_match` / `synonyms`	identity funnel branches (the first branch that fires wins)
selector and membership sets, weight `fields`	transform arguments, join and projection order, filter operands
identity digest inputs: `hash_identity_properties`, a branch's `fields` and `when_all_present` (the digest hashes a JSON object with sorted keys)

Sorting uses each element's canonical JSON rendering rather than a key per field, so it works for mixed element types, needs no tie-break rule, and cannot depend on the input order. Only the hash sees the sorted order; authored YAML keeps its own order.

A list field that is not classified raises UnclassifiedListField rather than being guessed. CANON_VERSION is part of the hashed bytes, so a change to these rules produces different hashes instead of reading old hashes under new rules.

Language version¶

metadata.gstl states the GSTL version a manifest is written in, as MAJOR.MINOR:

metadata:
    gstl: "1.0"

A manifest that states a major version graflo does not read is refused; any minor of a supported major is accepted, and an absent gstl means unstated. Every manifest the graflo CLI writes is stamped with the version it writes. Like all of metadata, gstl is outside the content hash.

Provenance¶

The content address and lineage can travel with the manifest, so a shipped file describes itself:

metadata:
    provenance:
        content_hash: "…64 hex…"
        canon: graflo/canon@5
        parents: [a3f9c21e4b70, 9e11d02c55aa]
        commit: c4d1e9a2b3f0
        recipe: "…"

Provenance is never part of the content hash. Content identity must not depend on the path taken, so that two routes to the same manifest agree that they arrived at the same place. A hash over the parents would make identity depend on history, and equal manifests could never be recognized as equal. The commit id is the hash that covers ancestry, as in git.

Stamping is explicit: stamp_provenance(...) or graflo stamp at a commit point, never a side effect of apply_evolution. Applying the same ops twice must not produce two files that disagree about their own lineage.

Commits¶

from graflo.architecture.evolution import History, build_commit, checkout

first = build_commit(base, ops, label="add commissioning date")
second = build_commit(after_first, more_ops, parents=[first.id], label="rekey")
history = History(commits=[first, second])

restored = checkout(base, history)  # replays, verifying every tree
as_of_first = checkout(base, history, first.id)

A Commit carries its ops, its parents (none for a root, one for an edit or a lift, two or more for a merge) and the content hash before and after it. Its kind says what produced it: root, edit, merge, merge3, revert or lift. build_commit applies the ops rather than trusting them, so both hashes describe a change that actually happened, and it refuses a change set that leaves the manifest unchanged, since such a commit would record nothing.

A commit id is derived from the ops and the order of the parents, so recording the same change set again yields the same id rather than a duplicate under a new name. graflo rehash recomputes every id when an op's serialization changes. A release that bumps CANON_VERSION moves every tree instead; graflo rehash --base <manifest> replays each lineage from its base and records the new trees, keeping every id (rehash_trees in Python). A history with several roots takes one --base ROOT=PATH per root.

Forks are recorded¶

Two commits may share a parent. History rejects what is broken (a duplicate id, a parent that does not exist, a cycle, a first-parent edge whose hashes do not line up) and records everything else, including several heads. Two people changing the same version is normal, so the history records both branches:

history.heads()  # more than one means it has forked
history.linearize()  # raises when there is no single path
history.topological()  # always available, deterministic on ties

Merge commits store a diff from the first parent¶

A merge commit's ops are the diff from its first parent to the merged result, not an interleaving of both sides. So replay and hash verification work the same way for edit and merge commits, and nothing downstream needs a special case. The record of how the merge was resolved is stored beside it as a recipe.

Undoing¶

History is append-only, so undoing a change moves forward: build_revert_commit (graflo revert) records a new commit that applies the inverses. Each inverse is computed against the manifest its op was applied to, passed as before=; checkout_parent(base, history, commit_id) rebuilds it from the base. The inverses are then applied to the head. Inversion is exact or it fails: a commit holding an op with no inverse is refused, and so is one whose inverses no longer apply to the head because a later commit changed what they touch. Checking out the parent commit is always exact.

Reversible	Irreversible
add and remove: vertices, edges, vertex and edge properties, indexes	`merge_vertices`, `merge_edges`
rename: vertices, relations, resources, properties; `canonicalize` that only renames	`change_field_types`, `canonicalize` that merges
`set_edge_directed`, `retarget_edges`, `add_inverse_edges`, `set_native_inverses`, `set_inverse_emission`; `declare_edge_inverses` and `retract_edge_inverses`	`sanitize`, `project_manifest`
`replace_identity` (with `retire: keep`), secondary identities, `replace_edge_identities`	`add_resource_transforms`, `ensure_extracted_fields`, `merge_manifests`
add, remove and replace resources; `set_bindings`, `set_db_profile`; grounding, superclasses, superproperties and descriptions

Whether an op can be undone depends on the op and on the manifest it met. invert_op applies its candidate inverse and offers it only when the result has the content hash of the state before, so an inverse is exact or absent:

a remove_vertices that also removed edges, profile entries or pipeline steps has no inverse, because adding the vertex back restores none of them. Remove the edges first, as diff_manifests does, and each step can be undone;
an op on a relation's properties has no inverse when the relation's edges disagreed about the property beforehand;
a property rename onto a name already taken folds two properties into one, and renaming back cannot make them two again;
a remove_resources of resources that bindings served has no inverse, because the bindings removed with them are not restored; nor has an add_resources or replace_resources that registered a transform the manifest did not hold, because no op withdraws one;
an op the manifest refuses has no inverse, because nothing was done.

A removed property is restored as the property it was, with its type, description and grounding.

Merging two branches¶

from graflo.architecture.evolution import find_merge_base, merge_three_way, take_left

base_id = find_merge_base(history, left_id, right_id)
merged, result = merge_three_way(ancestor, left, right)
if not result.clean:
    merged, result = merge_three_way(
        ancestor, left, right, resolutions=[take_left(result.conflicts[0])]
    )

A three-way merge is not a diff. Both sides descend from a common ancestor, so the question is not what differs between them but what each side changed, and whether those changes collide. When find_merge_base returns None, the two share no ancestor, and the operation you want is a union, not a three-way merge.

Slots¶

Changes are reconciled per slot: the place in the manifest an op touches, such as vertex/machine/field/serial_number. Changes to different slots merge on their own, the same change on both sides merges once, and different changes to one slot are a MergeConflict. A conflict carries both sides' ops and the ancestor's state, because what the slot looked like before either change is what a person resolving it needs to know, and a two-way diff cannot show it.

Three rules make the slot the right unit:

An ordered list is one slot. A resource pipeline is an ordered program, and merging half of each side's edits to it produces a program neither author wrote.
A rename occupies both names. Renaming machine to equipment while the other side adds a property to machine is a real collision, visible only if the rename counts as touching the old slot too.
An op touching several slots is atomic. If any of them is contested, the whole op is held back. Applying half an op is not a merge.

Slots nest, so vertex/machine contains vertex/machine/field/serial_number, and they are keyed on the canonical name: work_order and WorkOrder occupy the same slot and conflict, rather than merging into two unrelated types with the data split between them.

Edges nest under their relation: relation/feeds contains relation/feeds/edge/machine/line. Ops address edges two ways, by relation name (remove_edges, rename_relations, merge_edges) and by triple (set_edge_directed, retarget_edges, the index and identity ops), and nesting is what lets the two see each other: removing a relation on one side conflicts with flipping one of its edges on the other. A relation-wide property edit (relation/feeds/field) and a per-edge edit do not collide, and merge. An edge with no relation has its own root (edge/machine/line), since no op addressed by relation can reach it. A declared inverse sits under both of its relations (relation/feeds/inverse), so renaming or removing a relation conflicts with declaring or retracting its inverse on the other side; a symmetric declaration sits under its one relation. A native inverse is a slot of its relation (relation/feeds/native_inverse), as on TigerGraph, where the reverse type belongs to the relation's edge type.

What an op reads¶

A slot is what an op writes. That alone does not tell whether two ops are independent: add_edges writes an edge and depends on its endpoint types, while a remove_vertices on the other side, which also removes that type's edges, writes a different slot. Merged on written slots only, one order of the sides would drop the new edge without a word and the other would fail to apply. So an op also has a read set (op_reads):

Op	Reads
`add_edges`, `retarget_edges`, and every op addressed by edge triple	the endpoint types (old and new, for a retarget)
ops addressed by relation: edge properties, `remove_edges`, `rename_relations`, `merge_edges`, the inverse ops	the types that relation connects in the base, or, after a rename on the op's own side, the types the renamed relation connects; the op names only the relation
`replace_identity`, `add_secondary_identities`	the fields they key on, and those fields' types
`add_vertex_indexes`; edge index and identity ops	the fields they index or key on
`add_resources`, `replace_resources`	the types the pipeline names, the relations its edge steps write, and the properties it maps from a named column (a vertex step's `from`, a router's `vertex_from_map`); not a property a step fills under its own name

A read is disturbed by a write at or above it, never below: an edge onto line conflicts with removing or renaming line, and merges with a new property on it. Two ops reading the same thing are independent, like two edges onto one type. An op both sides made is agreement, not a dependency. The conflict is reported at the written slot with both ops attached, and is resolved like any other. ops_independent(a, b, base) is the test the merge and its tests share.

A change that no op expresses, such as one of a relation's edges gaining a property its siblings lack, cannot be merged at all: the merge is built from each side's ops, so the result would lack it. merge_three_way raises MergeError naming what is left over rather than return a result that looks clean and is incomplete.

Determinism¶

The same inputs produce the same merged manifest, the same conflicts in the same order, and the same content hash. A resolution takes the place of the ops it replaces rather than being appended, because op order is a precondition: diff_manifests emits an identity change before the secondary-identity addition that depends on it.

Union and three-way merge¶

	Three-way merge	Union
Inputs	two descendants of a common ancestor	two unrelated manifests
Names	expected to agree; a disagreement is a conflict	expected to differ; declared equivalences reconcile them
Function and command	`merge_three_way`, `graflo merge3`	`merge_manifests`, `graflo merge`
Order of the sides	significant: the commit's ops are the diff from the first parent	significant in six fields only; see below

Both produce commits with several parents. The stored commit kind and the CLI verb spell the three-way merge merge3 so that the two stay apart.

What depends on the order of the sides¶

A union of B onto A and of A onto B has the same content hash for everything the union assembles. Every list it concatenates (vertices, edges, resources, transforms, connectors, grounding, indexes) is sorted in the canonical form, so the order in which the two sides were walked disappears before anything is hashed. Metadata is not hashed at all, so the combined name (a+b), the joined description and the version do not change the content address either.

Five fields do depend on the order, all set when the two definitions of one type are combined. They are the fields the canonical form preserves, because their order carries meaning:

Vertex.identity: the column order of a composite key;
Vertex.filters;
IdentityFunnel.branches: branch order is the key's fallback order;
secondary identities named automatically (secondary_0, secondary_1), by position;
Vertex, Edge and Field descriptions, joined in side order.

Where a value is a claim rather than an order, the union refuses instead of choosing a side: two declared db_flavor values raise, a disputed iri is cleared, and conflicting storage names, field types and units raise.

A union is recorded too¶

A union joins two lines of history that share no ancestor, so both inputs must already be commits in the store: graflo commit --root starts the second line rather than extending the first. The commit stores the diff from its first parent, like any merge commit, so checkout and hash verification need no special case. Its recipe records the whole declaration (equivalences with their identities, canonical maps, and the resolved origins when the union names keys by origin) and no merge base, because there is none. A recipe whose declaration the op model no longer loads is refused as a CommitError that says so. Because the commit is a diff, every block the union changes needs an op; set_bindings and set_db_profile cover the bindings and the database profile.

Tracked merges and lifts¶

A merge, merge3 or lift commit carries a recipe (Commit.recipe): the declaration that produced it, with its content hash, which a replay reads. metadata.provenance.recipe points at it from a stamped manifest. A commit of another kind carries none, and a recipe of the wrong kind is refused.

A MergeRecipe records how a three-way merge was resolved. It is content-addressed, with its resolutions hashed in slot order:

from graflo.architecture.evolution import build_recipe, re_merge

recipe = build_recipe(ancestor, left, right, resolutions=resolutions)
merged, result = re_merge(recipe, ancestor, advanced_left, right)

When the left side moves on, re_merge replays the recorded decisions and reports only conflicts that are new. You keep an overlay on top of a manifest that changes, and decide each conflict once.

A recorded resolution whose slot no longer conflicts is reported as unused and never applied: applying an old decision to a slot nobody contests any more would undo someone's later change without anyone noticing.

A LiftRecipe records a lift: the aspects, the version of each, and the spec. plan_relift plans it again on a manifest that has changed since; see Recording a lift.

CLI¶

graflo commit --from-manifest base.yaml --to-manifest target.yaml -m "add work orders"
graflo log --graph
graflo verify --base base.yaml --against target.yaml
graflo checkout <commit> --base base.yaml --output-path out.yaml
graflo merge3 <left> <right> --base base.yaml --take left
graflo revert <commit> --base base.yaml
graflo stamp manifest.yaml --commit <commit>
graflo merge A.yaml B.yaml -o AB.yaml -m "join"   # a union, recorded as a two-parent commit
graflo lift A.yaml --spec lift.yaml -o lifted.yaml -m "lift"   # recorded as a lift commit

graflo commit derives the ops from two manifests with diff_manifests and refuses to store a change set that does not reproduce the target; --hints names renames the differ cannot infer. graflo merge3 stops at conflicts unless --take left or --take right settles them; --plot draws where the two branches met. When the history has several heads, graflo log names the verb that joins them: graflo merge3 for heads that share an ancestor, graflo merge for unrelated lineages. Commits live under .graflo/commits by default, one YAML file per commit (--store changes it). The store rebuilds the graph of commits from the recorded parent ids, not from the file names.

These commands record and replay changes to a manifest. graflo migrate-schema is different: it plans and applies changes to a database.

Rules and limits¶

A commit history changes manifests only. Applying it to a live database is not part of version control; graflo migrate-schema plans database changes and applies additive ones.
A change no op expresses, such as one of a relation's edges gaining a property its siblings lack, cannot be recorded from a diff or merged three ways; both refuse and name it.

What to read next¶

Version control example (22): a fork, a conflict, a resolution and a merge, end to end.
Manifest evolution: the ops a commit records.
Merging manifests: combining two unrelated manifests.