How do I turn a plain schema into one that tracks state and measurements over time?

Your manifest describes the machines on a plant floor and the production lines they are installed on. A machine has a status and an operating_temp, stored as properties of the machine. Each holds one value: when the status changes, the old value is gone, and nothing in the schema says when a value was true, where it came from, or in which unit a temperature is given. "What was this machine's status last March, and who reported it?" has no answer.

graflo lift rewrites the manifest so that it can answer. Values that change move to their own type with a validity interval, measurements get a type with a time and a unit, and two new types record where a fact came from and who reported it. You declare what the schema cannot say by itself; the lift writes the rest as a list of changes that you can review before you apply them.

flowchart LR
    subgraph before["manifest_in.yaml"]
        M1["Machine<br>machine_id, label,<br>status, operating_temp"]
    end
    subgraph after["artifacts/manifest_lifted.yaml"]
        M2["Machine<br>machine_id, label,<br>operating_temp in Cel"]
        S["MachineState<br>machine_id, status,<br>valid_from, valid_to"]
        O["MachineObservation<br>machine_id, observed_property,<br>result_value, result_unit, result_time"]
        E[Evidence]
        A[Agent]
        S -- specializationOf --> M2
        O -- hasFeatureOfInterest --> M2
        S -- wasDerivedFrom --> E
        O -- wasDerivedFrom --> E
        E -- wasAttributedTo --> A
    end
    M1 -.-> M2

What you need

• GraFlo installed (pip install graflo). No database is needed.

The data

There are no data files: the lift changes the manifest. The Machine type of manifest_in.yaml:

-   name: Machine
    description: A machine on the plant floor.
    properties:
    -   {name: machine_id, type: STRING}
    -   {name: label, type: STRING}
    -   {name: status, type: STRING}
    -   {name: operating_temp, type: FLOAT}
    identity: [machine_id]

ProductionLine has line_id, label and operating_mode. Two edges, installed_on (machine to line) and feeds (machine to machine), do not say whether they are directed.

Steps

1. Check the manifest against the profile

A conformance profile is a set of checks that GraFlo runs on a manifest. The world-model profile asks whether the manifest can say what its types mean, when a fact was true and where it came from:

Check	Passes when
grounded-types	every vertex and edge names the concept it denotes by an IRI (semantics.iri or semantics.exact_match)
declared-identity	every vertex declares how it is identified, instead of falling back to all its properties
declared-directionality	every edge states directed instead of taking the default
declared-units	every float property outside the key has a unit, or its type has a property that carries the unit in each row
temporal	at least one date-time property is grounded in a validity or observation-time vocabulary
provenance	one vertex type is grounded as an agent, and one edge in a derivation or attribution property

cd examples/23-state-core-lift
uv run graflo check --profile world-model manifest_in.yaml

profile world-model v0.1 -- manifest_in.yaml
  overall: FAIL

  [  FAIL] grounded-types: Types are grounded in an external vocabulary  (4 checked)
           - vertex:Machine: no semantics.iri or semantics.exact_match
           - vertex:ProductionLine: no semantics.iri or semantics.exact_match
           - edge:Machine-installed_on->ProductionLine: no semantics.iri or semantics.exact_match
           - edge:Machine-feeds->Machine: no semantics.iri or semantics.exact_match
  [  PASS] declared-identity: Every vertex declares an identity mode  (2 checked)
  [  FAIL] declared-directionality: Every edge declares its directionality  (2 checked)
           - edge:Machine-installed_on->ProductionLine: does not declare `directed`; it defaulted to True
           - edge:Machine-feeds->Machine: does not declare `directed`; it defaulted to True
  [  FAIL] declared-units: Every measured property carries a unit  (1 checked)
           - vertex:Machine.operating_temp: measured property carries no unit, and its type declares no unit-valued property to carry one per row
  [  FAIL] temporal: Temporal validity is declared or waived  (1 checked)
           - manifest: no property is grounded in a validity or observation-time vocabulary, so the model cannot say when a fact was true; declare one or waive this assertion with a reason
  [  FAIL] provenance: Provenance is expressible and attached  (4 checked)
           - manifest: no vertex is grounded as an agent, so a fact has nothing to be attributed to
           - manifest: no edge is grounded in a derivation or attribution property, so the model cannot record where a fact came from
           - manifest: manifest declares no ingestion model, so whether provenance is attached at ingest was not checked

Five of the six checks fail. Identity passes, because each type names its key. The command exits with code 1.

2. Declare what the schema cannot say

The lift can see structure: which edges leave directed unstated, which types have no IRI, which floats have no unit. It cannot see meaning, so lift.yaml declares four things:

Keys	What they say
grounding, edge_grounding	what each type and relation denotes, as IRIs from PROV-O and SOSA
stateful	which properties are facts that change over time
observed	which types are measured at a point in time
measured	the unit of an existing measurement, as a UCUM code (Cel is degrees Celsius)

grounding:
    Machine:
        iri: http://www.w3.org/ns/prov#Entity
        exact_match:
        -   http://www.w3.org/ns/prov#Entity
        -   http://www.w3.org/ns/sosa/FeatureOfInterest
# ProductionLine and the two edges are grounded the same way

stateful:
    Machine: [status]
    ProductionLine: [operating_mode]

observed: [Machine]

measured:
    Machine.operating_temp: Cel

3. Lift the manifest

uv run graflo lift manifest_in.yaml --spec lift.yaml \
    --emit-ops artifacts/ops.yaml -o artifacts/manifest_lifted.yaml

planned 10 operation(s):
  set_vertex_semantics
  set_edge_semantics
  set_edge_semantics
  set_edge_directed
  set_field_semantics
  add_vertices
  add_edges
  add_vertices
  add_edges
  remove_vertex_properties
written: artifacts/ops.yaml
profile world-model v0.1 -- artifacts/manifest_lifted.yaml
  overall: PASS
[... the report shown under "What you should see" ...]
written: artifacts/manifest_lifted.yaml

The first five operations add what lift.yaml declares and set directed: true on the two edges. Then the lift adds new types and their edges:

• MachineState and ProductionLineState hold the values that change, each over an interval valid_from to valid_to. A state is identified by its subject's key together with valid_from, because one machine has a different status in each interval. A new value closes the old interval instead of overwriting it.
• MachineObservation holds one measurement of a machine: which property, the value, its unit, and the time. The unit is a property of each row (result_unit), because one observation type holds temperatures and pressures alike.
• Evidence and Agent, with the edges wasDerivedFrom and wasAttributedTo, record where a state or an observation came from and who reported it.

The last operation removes status from Machine and operating_mode from ProductionLine: they now live on the state types. Everything before it only adds.

artifacts/ops.yaml is the plan. Applied to manifest_in.yaml it produces artifacts/manifest_lifted.yaml, and invert_ops turns it into the operations that undo the lift.

What you should see

The lifted manifest passes every check on its own:

uv run graflo check --profile world-model artifacts/manifest_lifted.yaml

profile world-model v0.1 -- artifacts/manifest_lifted.yaml
  overall: PASS

  [  PASS] grounded-types: Types are grounded in an external vocabulary  (16 checked)
  [  PASS] declared-identity: Every vertex declares an identity mode  (7 checked)
  [  PASS] declared-directionality: Every edge declares its directionality  (9 checked)
  [  PASS] declared-units: Every measured property carries a unit  (2 checked)
           - vertex:MachineObservation.result_value: unit carried per row by a unit-valued property
  [  PASS] temporal: Temporal validity is declared or waived  (1 checked)
           - manifest: temporal validity modelled by 6 property/properties
  [  PASS] provenance: Provenance is expressible and attached  (16 checked)
           - manifest: provenance is expressible: an agent type and 4 provenance relation(s)
           - manifest: manifest declares no ingestion model, so whether provenance is attached at ingest was not checked

The manifest has 7 vertex types and 9 edges. The last line is a limit of the lift: it changes the manifest, not how data is loaded. Nothing writes to the new types yet. In a manifest with an ingestion model, you add resources, or steps to existing ones, that fill them.

A larger model, for reference

reference.yaml is a model written by hand that passes the profile: six general types (Asset, State, Observation, Event, Agent, Evidence) and the relations between them, grounded in PROV-O, SOSA and QUDT. A lift does not produce it: a lift adds only what its input asks for, and names the new types after their subject (MachineState, not State). Use it as a pattern when you write such a model yourself; uv run graflo check reference.yaml reports overall: PASS.

Also possible

• retire: keep in lift.yaml leaves status on Machine as its current value next to the history; the plan then ends before the removal.
• provenance: false leaves out Evidence and Agent.
• graflo check --waivers FILE records a check that you decide not to meet, with a reason; the report shows it as waived, never as passed.

What to read next

• Conformance profiles: each check of the world-model profile, and waivers.
• Manifest evolution: the operations a lift plans, and how to apply and undo them.

Files

The example lives in examples/23-state-core-lift.

lift.yaml

# What the lift cannot work out for itself.
#
# A lift sees structure: which edges leave `directed` unstated, which types
# carry no grounding, where a float has no unit. It cannot see meaning. This
# file says four things the schema does not: what each type and relation
# denotes (grounding, edge_grounding), which properties change over time
# (stateful), which types are measured at a point in time (observed), and the
# unit of each existing measurement (measured).

grounding:
    Machine:
        iri: http://www.w3.org/ns/prov#Entity
        exact_match:
        -   http://www.w3.org/ns/prov#Entity
        -   http://www.w3.org/ns/sosa/FeatureOfInterest
    ProductionLine:
        iri: http://www.w3.org/ns/prov#Entity
        synonyms: [Line, AssemblyLine]

edge_grounding:
-   source: Machine
    target: ProductionLine
    relation: installed_on
    iri: http://www.w3.org/ns/prov#wasInfluencedBy
-   source: Machine
    target: Machine
    relation: feeds
    iri: http://www.w3.org/ns/prov#wasInfluencedBy

# Facts that change over time. They move off the entity onto `MachineState`
# (and `ProductionLineState`), which holds each value over an interval: a new
# value closes `valid_to` of the old one instead of overwriting it.
stateful:
    Machine: [status]
    ProductionLine: [operating_mode]

# Types that get measured at a time.
observed: [Machine]

# UCUM tokens. UCUM has no currency, so currency would use ISO-4217 (USD).
measured:
    Machine.operating_temp: Cel

manifest_in.yaml

# A plain plant-floor manifest: no grounding, no units, no validity intervals,
# no provenance. It fails five of the six checks of the `world-model` profile;
# only the identity check passes, because each type names its key.
metadata:
    name: plant-plain
    description: An ungrounded starting point, before the lift.
schema:
    metadata:
        name: plant-plain
        version: "1.0.0"
    graph:
        vertex_config:
            vertices:
            -   name: Machine
                description: A machine on the plant floor.
                properties:
                -   {name: machine_id, type: STRING}
                -   {name: label, type: STRING}
                -   {name: status, type: STRING}
                -   {name: operating_temp, type: FLOAT}
                identity: [machine_id]

            -   name: ProductionLine
                description: A line that turns out one product family.
                properties:
                -   {name: line_id, type: STRING}
                -   {name: label, type: STRING}
                -   {name: operating_mode, type: STRING}
                identity: [line_id]

        edge_config:
            edges:
            -   source: Machine
                target: ProductionLine
                relation: installed_on
            -   source: Machine
                target: Machine
                relation: feeds

reference.yaml

# A larger model, written by hand, that passes the `world-model` profile.
#
# Six general types -- Asset, State, Observation, Event, Agent, Evidence -- plus
# two Asset-to-Asset relations. Grounded in PROV-O, SOSA/SSN and QUDT.
#
# This is not what `graflo lift` produces: a lift adds only what its input
# asks for, and names the new types after their subject (`MachineState`, not
# `State`). The file shows the full shape, and that a hand-written model can
# pass every check of the profile.
metadata:
    name: state-core-reference
    description: >-
        Six abstract types -- Asset, State, Observation, Event, Agent and
        Evidence -- plus the Asset-to-Asset topology edges, grounded in PROV-O,
        SOSA/SSN and QUDT.
schema:
    metadata:
        name: state-core-reference
        version: "0.1.0"
        description: >-
            Abstract vocabulary for entities whose facts are qualified by when
            they held and where they came from. No ingestion, no bindings.
        semantics:
            exact_match: ["http://www.w3.org/ns/prov#"]
    graph:
        vertex_config:
            vertices:
            -   name: Asset
                description: >-
                    A thing the model is about: a machine, a site, a circuit, a
                    contract. Everything else in this model hangs off one.
                semantics:
                    iri: http://www.w3.org/ns/prov#Entity
                    exact_match:
                    -   http://www.w3.org/ns/prov#Entity
                    -   http://www.w3.org/ns/sosa/FeatureOfInterest
                properties:
                -   {name: asset_id, type: STRING}
                -   {name: label, type: STRING}
                -   {name: asset_type, type: STRING}
                identity: [asset_id]

            -   name: State
                description: >-
                    One property of one Asset holding one value over one
                    interval. Closing `valid_to` rather than overwriting is what
                    makes history queryable.
                semantics:
                    iri: http://www.w3.org/ns/prov#Entity
                    exact_match:
                    -   http://www.w3.org/ns/prov#Entity
                    -   http://www.w3.org/2006/time#ProperInterval
                properties:
                -   {name: asset_id, type: STRING}
                -   name: property
                    type: STRING
                    semantics:
                        exact_match: ["http://www.w3.org/ns/ssn/Property"]
                -   {name: value, type: STRING}
                -   name: valid_from
                    type: DATETIME
                    semantics:
                        iri: http://www.w3.org/ns/prov#generatedAtTime
                -   name: valid_to
                    type: DATETIME
                    semantics:
                        iri: http://www.w3.org/ns/prov#invalidatedAtTime
                # The interval start is part of the key: the same property of
                # the same asset holds many values over time, and they are
                # different facts rather than revisions of one.
                hash_identity_properties: [asset_id, property, valid_from]

            -   name: Observation
                description: >-
                    A measurement of an Asset at a time. The unit travels with
                    the row because this type is abstract -- an overlay
                    measuring temperature and one measuring pressure are the
                    same type here.
                semantics:
                    iri: http://www.w3.org/ns/sosa/Observation
                    exact_match: ["http://www.w3.org/ns/sosa/Observation"]
                properties:
                -   {name: asset_id, type: STRING}
                -   name: observed_property
                    type: STRING
                    semantics:
                        exact_match: ["http://www.w3.org/ns/sosa/observedProperty"]
                -   name: result_value
                    type: FLOAT
                    semantics:
                        exact_match: ["http://www.w3.org/ns/sosa/hasSimpleResult"]
                -   name: result_unit
                    type: STRING
                    semantics:
                        iri: http://qudt.org/schema/qudt/ucumCode
                -   name: result_time
                    type: DATETIME
                    semantics:
                        iri: http://www.w3.org/ns/sosa/resultTime
                hash_identity_properties: [asset_id, observed_property, result_time]

            -   name: Event
                description: >-
                    Something that happened to an Asset over an interval: a
                    maintenance job, an outage, a shipment.
                semantics:
                    iri: http://www.w3.org/ns/prov#Activity
                    exact_match: ["http://www.w3.org/ns/prov#Activity"]
                properties:
                -   {name: event_id, type: STRING}
                -   {name: event_type, type: STRING}
                -   name: started_at
                    type: DATETIME
                    semantics:
                        iri: http://www.w3.org/ns/prov#startedAtTime
                -   name: ended_at
                    type: DATETIME
                    semantics:
                        iri: http://www.w3.org/ns/prov#endedAtTime
                identity: [event_id]

            -   name: Agent
                description: >-
                    Who or what acted or measured: a person, a team, a sensor,
                    a system of record.
                semantics:
                    iri: http://www.w3.org/ns/prov#Agent
                    exact_match:
                    -   http://www.w3.org/ns/prov#Agent
                    -   http://www.w3.org/ns/sosa/Sensor
                properties:
                -   {name: agent_id, type: STRING}
                -   {name: label, type: STRING}
                -   {name: agent_kind, type: STRING}
                identity: [agent_id]

            -   name: Evidence
                description: >-
                    What a fact was read from: a document, an API response, a
                    table row. The attachment point for anything extracted from
                    unstructured sources.
                semantics:
                    iri: http://www.w3.org/ns/prov#Entity
                    exact_match: ["http://www.w3.org/ns/prov#Entity"]
                properties:
                -   {name: evidence_id, type: STRING}
                -   {name: source_uri, type: STRING}
                -   {name: media_type, type: STRING}
                -   name: retrieved_at
                    type: DATETIME
                    semantics:
                        iri: http://www.w3.org/ns/prov#generatedAtTime
                identity: [evidence_id]

        edge_config:
            edges:
            # -- Topology. Only the two relations every domain has; anything
            #    more specific belongs to the domain model built on this one.
            -   source: Asset
                target: Asset
                relation: partOf
                directed: true
                description: Containment. An asset is part of at most one whole at a time.
                semantics:
                    iri: http://purl.org/dc/terms/isPartOf
                    exact_match: ["http://purl.org/dc/terms/isPartOf"]

            -   source: Asset
                target: Asset
                relation: dependsOn
                directed: true
                description: >-
                    Functional dependence. Directed because "A depends on B" and
                    "B depends on A" are different facts, and reach is
                    computed by following it one way.
                semantics:
                    iri: http://www.w3.org/ns/prov#wasInfluencedBy
                    exact_match: ["http://www.w3.org/ns/prov#wasInfluencedBy"]

            # -- What a fact is about.
            -   source: State
                target: Asset
                relation: specializationOf
                directed: true
                semantics:
                    iri: http://www.w3.org/ns/prov#specializationOf
                    exact_match: ["http://www.w3.org/ns/prov#specializationOf"]

            -   source: Observation
                target: Asset
                relation: hasFeatureOfInterest
                directed: true
                semantics:
                    iri: http://www.w3.org/ns/sosa/hasFeatureOfInterest
                    exact_match: ["http://www.w3.org/ns/sosa/hasFeatureOfInterest"]

            -   source: Event
                target: Asset
                relation: used
                directed: true
                semantics:
                    iri: http://www.w3.org/ns/prov#used
                    exact_match: ["http://www.w3.org/ns/prov#used"]

            # -- Who did it.
            -   source: Observation
                target: Agent
                relation: madeBySensor
                directed: true
                semantics:
                    iri: http://www.w3.org/ns/sosa/madeBySensor
                    exact_match: ["http://www.w3.org/ns/sosa/madeBySensor"]

            -   source: Event
                target: Agent
                relation: wasAssociatedWith
                directed: true
                semantics:
                    iri: http://www.w3.org/ns/prov#wasAssociatedWith
                    exact_match: ["http://www.w3.org/ns/prov#wasAssociatedWith"]

            # -- Where it came from. Without these the model cannot say which
            #    source a fact came from, which is what the `provenance` check
            #    of the profile asks for.
            -   source: State
                target: Evidence
                relation: wasDerivedFrom
                directed: true
                semantics:
                    iri: http://www.w3.org/ns/prov#wasDerivedFrom
                    exact_match: ["http://www.w3.org/ns/prov#wasDerivedFrom"]

            -   source: Observation
                target: Evidence
                relation: wasDerivedFrom
                directed: true
                semantics:
                    iri: http://www.w3.org/ns/prov#wasDerivedFrom
                    exact_match: ["http://www.w3.org/ns/prov#wasDerivedFrom"]

            -   source: Evidence
                target: Agent
                relation: wasAttributedTo
                directed: true
                semantics:
                    iri: http://www.w3.org/ns/prov#wasAttributedTo
                    exact_match: ["http://www.w3.org/ns/prov#wasAttributedTo"]
    db_profile: {}