Ontologies and facts¶
OntoCast writes two kinds of graph: ontologies, which define the terms, and facts, which state what a document says in those terms. This page explains how each is identified, versioned and changed, and how to trace a fact back to the text it came from.
The catalog¶
The catalog is the set of ontologies OntoCast knows: the ones you load at
startup (--ontology-dir or
ONTOCAST_ONTOLOGY_DIRECTORY),
the ones you upload to /ontologies, and the ones runs create.
An ontology is identified by its IRI, the subject of its owl:Ontology
statement: an absolute http(s) IRI with no fragment, since its versions are
named <iri>#<key>. An empty fragment (http://www.w3.org/ns/prov#) is
dropped. Its namespace, where its terms live, is separate: <iri># for an
ontology OntoCast creates, and read from the file for one you load. Two
shorter names are labels of the entry: its ontology_id and the prefix its
author bound in Turtle. They can differ (an ontology with id
observation may use the prefix obs:), and they need not be unique: two
files whose IRIs end in /ontology both load with the id ontology. Wherever
OntoCast asks for an ontology, such as
ONTOLOGY_CONTEXT_FIXED_ONTOLOGY_ID,
the IRI always works, and a short name works when one ontology alone holds it;
a short name several hold is refused with their IRIs.
A loaded file's id is, in order: the id given with it, its
vann:preferredNamespacePrefix, the prefix the file binds to its own namespace
(a DBpedia file binding dbo: loads as dbo), else the last segment of its
IRI.
An ontology_id is a lowercase handle matching ^[a-z][a-z0-9_-]{1,31}$. The
names OntoCast uses itself (cd, doc, facts, oc, ontocast) are refused.
A standard prefix (sosa, prov, skos, schema, ...) names only the
ontology whose namespace is that prefix's standard one: the W3C SOSA file loads
as sosa and PROV-O as prov, while any other ontology, and every ontology
OntoCast creates, needs another id.
Prefixes¶
Every prompt, every parse of a model's answer and every export shows a namespace under one prefix, chosen in this order:
- a well-known namespace keeps its standard prefix (
unit:,sosa:,time:), also when a catalog file's terms live in it; - an ontology's own namespace takes the prefix its author bound in Turtle;
- else its
ontology_id, also when the author's prefix is a library name or the standard prefix of another namespace; - a namespace no catalog ontology owns takes the prefix the catalog binds to it most often.
The author's prefix is stored with each version, so it survives the triple
store. Any other prefix bound to a namespace, the ontology_id among them, is
an alias: an answer that writes it still parses, and a CURIE in a term's note or
a shape's sh:message written with it is shown under the chosen prefix.
The home ontology¶
Identity is minted by code, never written by the model. A run that can create
vocabulary (RENDER_MODE ontology or ontology_and_facts) resolves one
home ontology per document before its units are extracted, in one LLM
call. The catalog is listed under each ontology's shown prefix (below), which
unlike an id is unique, with its title and the start of its description: the
description is how the model tells that an ontology covers a document, so
give a seed ontology one. The model proposes an ontology_id, a title and a
description, scoped to the document's field rather than its subject, and is
asked to prefer a catalog ontology whose described scope the document falls
within. A shown prefix means the document extends that ontology; any
other id names a new one, whose IRI is
CURRENT_DOMAIN/<ontology_id>.
A new id is overridden when the document's text names at least five terms of
one catalog ontology (by label or local name, plurals included): the document
then extends that ontology, so documents of one domain share a home
(home_new_overridden).
Every unit of the document mints its new terms under that one ontology. The
model never writes an owl:Ontology header; one it writes anyway is dropped
and counted (ontology/model_header_triples_dropped).
Owned and external ontologies¶
An ontology whose IRI lies under CURRENT_DOMAIN is owned and is extended
in place. Any other, such as a published vocabulary loaded as a seed, is
external: OntoCast never writes to it. When a document's home would be an
external ontology, its terms go to the extension ontology
CURRENT_DOMAIN/<id>-ext-<hash>, created on first use, which owl:imports
the external one. <hash> is the first 10 hex digits of the SHA-256 of the
external IRI (without a trailing / or #), so every document and every run
that extends one external ontology shares its extension, and two externals
with the same id never do (home_extension). A triple whose subject is a term of an external
ontology is refused at the end of the document and counted
(external_subject_refused): the external terms are used by IRI, never
redeclared or given new axioms. To extend a seed ontology in place, serve it
under CURRENT_DOMAIN.
Every new term of a document lives under its home namespace; the render and
critic prompts name it, and a term newly declared anywhere else is dropped
(minted_outside_home_dropped). An ontology update declares classes,
properties and SKOS concepts (value sets) only: a person, place, organisation
or event named in the text is a fact, and is dropped from the update
(non_term_subjects_dropped). A value set has a class of its own under
skos:Concept, which is the range of the property that takes those values,
and each value is typed both skos:Concept and that class. Each statement is written to the ontology that
owns its subject; a restriction or a list travels whole with the term that
uses it, so a restriction on an external property is written beside the home
class, never into the external ontology.
A unit mints a class only at the level a second document in the same domain would reuse. A case the document reports, such as a named case or one measured value, is an individual or a property value in the facts. A distinction that varies from one instance to the next is a property with values, not a family of sibling subclasses, and an existing or well-known term is reused before a new one is minted. The per-unit checks enforce the parts of this a machine can see: hierarchy depth, value-set siblings, case-specific leaves and duplicates of catalog terms (see Validation layers).
An ontology you load from a file or upload to /ontologies keeps its own IRI.
The file declares exactly one named owl:Ontology that describes itself; a
header that is only typed owl:Ontology beside it is dropped, as W3C PROV-O
publishes one. Its namespace is the file's vann:preferredNamespaceUri when
terms live under it, else the namespace in which the file describes the most
terms: OM-2's is …/om-2/, PROV-O's is …/prov#. Its id is its
vann:preferredNamespacePrefix, else derived from the IRI; its title
dcterms:title, else rdfs:label, else the id; its version label
owl:versionInfo when that is a semantic version, else 1.0.0. It is stored
as a root version. An upload whose IRI is already in the catalog is refused;
delete the ontology first, or replace it with PUT /ontologies/{iri}.
Versions¶
An ontology is never edited in place. Each version is an immutable named graph
<iri>#<key>, its version IRI. For versions OntoCast writes, the key is the
content hash of the graph, so the same content is never stored twice and a run
that leaves an ontology unchanged writes no version. Inside that graph, the
ontology IRI carries the version header:
| Triple on the ontology IRI | Value |
|---|---|
owl:versionIRI |
The named graph, <iri>#<key> |
owl:priorVersion |
The version(s) this one was made from: none on a root, one for an update, several for a merge |
owl:versionInfo |
A semantic version label: 1.0.0 on a root, then bumped per ontology from the updates applied to it. Major when more than five triples are deleted and the updates declare or remove more than two classes or more than three properties; minor for any other deletion, or for five or more new classes or properties; patch otherwise |
vann:preferredNamespacePrefix, vann:preferredNamespaceUri |
The ontology_id and the namespace; the author's prefix is kept as an sh:declare |
dcterms:title, rdfs:label |
The title |
dcterms:description |
The description, when there is one |
dcterms:created |
When the root of the lineage was created, carried forward |
dcterms:modified |
When this version was created |
prov:wasDerivedFrom |
The source document's IRI, for versions a run wrote |
Author statements on the ontology IRI outside the header (owl:imports, a
licence) are kept. A removed version keeps its header and is marked
<graph> a oc:RemovedVersion (oc: is https://growgraph.dev/ontocast#),
so lineage through it never dangles.
The head of an ontology is a version that is not removed and from which no
other version that is not removed descends; descent passes through removed
versions, so removing the newest version makes its priors heads again. A
run reads one head per ontology, and older versions stay in the store, so you
can compare what a run changed with what was there before. An ontology with
several heads is diverged: GET /ontologies lists it and its versions can
be read, but no run extracts against it or extends it until a merge gives it
one head. Uploading the ontology with PUT /ontologies/{iri} is that merge: the
new version derives from every head. Ontology catalog describes how
the catalog picks heads safely when several processes share a store.
Changes are patches¶
The model never rewrites an ontology. It returns a patch: the triples to insert and the triples to delete. OntoCast checks the patch and applies it to the graph, deletions first. Two things follow:
- A render's output is proportional to the change, not to the size of the ontology, so a large ontology costs no more output tokens than a small one.
- The model cannot drop a triple by forgetting to repeat it; a removal is always an explicit delete.
The facts critic works the same way: its fixes are applied as a patch to the
unit's facts. The model writes patches as Turtle by default, or as compact
JSON-LD; see
LLM_GRAPH_FORMAT.
Either way, the patch that is applied is the same.
Facts and the two-namespace contract¶
Facts use two namespaces, and the rule between them is what keeps your ontology clean:
| Namespace | What goes there |
|---|---|
| Your ontology's namespaces | Classes, properties, and the individuals the ontology already declares, such as the entries of a controlled vocabulary. Facts use these terms but never add to them |
cd: |
Every entity found in the text, even when its class comes from your ontology |
cd: is https://growgraph.dev/facts/. It is fixed, not a setting.
The prompt names your ontology's namespaces as the ones its terms live in, under
the prefixes its term sheet shows. A standard vocabulary you load as an
ontology (PROV-O, SOSA) is one of them; the RDF languages (rdf, rdfs,
owl, xsd) never are.
The model is instructed to:
- name every new entity in
cd:, with alowercase_snake_caselocal name and anrdfs:labeltaken from the text; - reuse an ontology IRI for an individual only when the ontology declares that individual, never invent one in the ontology's namespace;
- treat a matching class as a type, not as an identity: two mentions of a
sample are new
cd:entities typed with the ontology'sSampleclass, not the class itself; - never type a
cd:entity as a class or a property; - extract statements: mint an entity only when a statement is about it (it is linked to another entity or holds a value), never for a bare mention;
- write a structured value (a quantity, a range, a bound) as a blank node, the object of the property that carries it, never as an entity of its own;
- name what every observation is about: an observation (of
sosa:Observation,sosa:Actuation,sosa:Samplingor a class of your catalog under one) is linked to its subject,sosa:hasFeatureOfInterestor your catalog's property for the same link, even when the sentence leaves the subject to its context. Declare your catalog's property as a sub-property ofsosa:hasFeatureOfInterest(or of its inversesosa:isFeatureOfInterestOf, or as an inverse of either) for it to count.
An observation or a value that names no subject is a statement_without_subject
finding in its part; the gap pass asks the model for the link
(FACTS_COMPLETION_PASSES), and facts_statements_without_subject counts what
is left.
The served graph holds statements only. After merging, a statement joining a
node to itself through a property not declared reflexive is dropped
(facts_self_loops_dropped), an entity linked to nothing and holding no value
is dropped, and so is a value no statement
carries (facts_unlinked_dropped). A value written as an entity of its own,
typed with a value class, and linked to nothing is dropped too. Any other
instance still linked to nothing is kept and listed in the run manifest
(facts_isolated).
A value class is one whose instances are values, never facts on their own:
qudt:QuantityValue, schema.org's StructuredValue, QuantitativeValue,
PropertyValue and MonetaryAmount, the value_class you set in
FACTS_QUANTITY_FALLBACK_VOCABULARY, and every class of your catalog that
subclasses one of them. To have your own value classes treated as values,
declare them as subclasses of one of these, or name one as value_class. Values are never merged with each other;
each is named under its carrier, <carrier>.<property> (then a number when a
carrier holds several values on one property), so it has a stable IRI.
You can add your own guidance on top of these rules with
facts_user_instruction; see Writing user
instructions.
Two vocabulary modes¶
FACTS_VOCABULARY says what the facts may be written in. Both modes use your
catalog first, for every class and property it has. Where it has none, the
model uses a well-known vocabulary, and then the namespaces you list in
FACTS_ADDITIONAL_STANDARD_NAMESPACES. The prompt names them all, with their
namespace IRIs:
| Prefix | Vocabulary | Terms checked |
|---|---|---|
rdf, rdfs, owl, xsd |
RDF, RDF Schema, OWL, XML Schema datatypes | yes |
skos |
SKOS | yes |
dcterms, dc |
DCMI Metadata Terms, Dublin Core elements | yes |
prov |
PROV-O | yes |
sosa, ssn |
SOSA, SSN | yes |
time |
OWL-Time | yes |
csvw |
CSVW | yes |
dcat |
DCAT | yes |
org |
the Organization Ontology | yes |
vcard |
vCard | yes |
foaf |
FOAF | yes |
schema |
schema.org (https:// and http://) |
yes |
qudt |
QUDT schema | yes |
unit |
QUDT units | yes |
quantitykind |
QUDT quantity kinds | no |
The model writes well-known terms from memory, so a term must exist in its
vocabulary: one it does not have, such as schema:vulnerableTo, is an
unknown_term finding with the vocabulary's near-misses as suggestions. A
vocabulary marked "no" is admitted by namespace, as are your own namespaces.
The QUDT schema list covers every term its 2.1 and 3.x releases publish; the
QUDT unit list covers every unit of its 3.1.4 release. A unit comes from the
ontology context first, then from QUDT; a unit neither has is kept as text on
the value node.
strict (the default) stops there. A thing that neither your catalog nor
these vocabularies can type or relate is not extracted. A stated quantity always
is: where no property carries it, it is written as an observation whose result
is the value and whose feature of interest is the thing measured. A class or property from any other
vocabulary is an unknown_term finding the renderer must rewrite or remove,
and at the gate non_catalog_vocabulary is an error for it. Both checks count
every term of your catalog, not only the unit's retrieved context. The model
sees the retrieved context, plus the definition of any catalog term the
conformance rules or a term's note name, so a property retrieval missed is
otherwise a statement made in a well-known vocabulary or not made at all: watch
non_catalog_vocabulary, facts_isolated and facts_unlinked_dropped, and
widen the context (ONTOLOGY_CONTEXT_MODE) when they grow.
open also admits any other established vocabulary the model knows, as a
last resort after the well-known ones. Such terms pass the unit validator and
are listed per document as non_catalog_vocabulary warnings, so the gaps in
your catalog are visible.
In both modes the carrier role of FACTS_QUANTITY_FALLBACK_VOCABULARY names
the property that carries a quantity when your catalog has none (unset, the
observation form above), and an entity
linked to nothing and holding no value is dropped at the gate.
Provenance¶
Every fact can be traced to the content unit it came from. OntoCast uses RDF 1.2 reification: each asserted triple gets a reifier that points to the unit, or to several units when more than one stated it.
_:r rdf:reifies <<( cd:sample_a ex:annealedAt cd:temperature_1 )>> ;
prov:wasDerivedFrom <unit_iri> .
Each unit is described once:
| Predicate on the unit | Value |
|---|---|
rdf:type |
prov:Entity, schema:Text |
schema:position |
The unit's position in the document |
schema:identifier |
A hash of the unit's text |
prov:generatedAtTime |
When it was extracted |
schema:articleSection |
Its section label, when it has one |
ontocast:sectionLabelSource |
Which classifier step decided the label |
ontocast:sectionLabelStrength |
strong or weak: whether the label could take part in dropping the unit |
ontocast: is https://growgraph.dev/ontocast#, outside the facts namespace,
so pipeline metadata is never mistaken for extracted facts. Provenance of
ontology changes is kept apart from the ontology, so the ontology you load
elsewhere carries none of it.
Provenance adds at least two triples for every fact. It is the chunk nodes and statement reifiers the pipeline mints, not the PROV namespace: facts you extract with PROV-O as your ontology are facts like any others and are kept. To leave provenance out of what you get back:
- over HTTP, pass
strip_provenance=trueto/process(provenance is kept by default); - with
ontocast process, the files are written without it by default; pass--keep-provenanceto keep it, together with<name>.units.jsonl, the text of each unit the provenance points at (see Reading run telemetry).
A graph without provenance has no link back to the text, so nothing in it can be checked against the source.
Provenance is written in the formats that carry RDF 1.2 triple terms: Turtle,
N-Triples, N-Quads and TriG. JSON-LD and RDF/XML cannot carry them, so asking
for either on a graph with provenance raises ProvenanceFormatError; strip the
provenance first (TripleStoreManager.strip_provenance) to write those.
In Python, an RDFGraph keeps its reifications beside its triples, in
graph.reifications. Copying, merging (+, +=) and a pydantic dump and load
carry them.
graph.hash() covers the statements, not the provenance.
Document identity¶
Each document gets an IRI built from
CURRENT_DOMAIN and a
hash of its content, and its units are linked to it. To state what the
document is, independent of its text, pass document_metadata: a JSON object
on /process, or --document-metadata '{...}' on ontocast process.
| Kind | Keys | Becomes |
|---|---|---|
| Bibliographic identifiers | doi, isbn, pmid, arxiv_id, handle |
dcterms:identifier literals |
| Other identifiers | identifiers: [{"scheme": "erp:doc", "value": "INV-001"}], or any key ending in an identifier word such as _id, _ref, _no, _code |
A dcterms:Identifier node with dcterms:type and rdf:value |
| Description | title, published or issued, source_system |
dcterms:title, dcterms:issued, prov:wasAttributedTo |
| Source | source_uri, source_url, stable_source_iri |
dcterms:source, or owl:sameAs for stable_source_iri |
| People and other entities | author, authors or creator; project; any other key |
A schema:Person (for authors) or prov:Entity, with an rdfs:label |
For example:
{"doi": "10.1234/example", "author": ["Jane Doe"], "project": {"name": "Coatings survey", "identifier": "PRJ-7"}}
becomes
<doc_iri> dcterms:identifier "10.1234/example" ;
dcterms:creator <doc_iri/janeDoe> ;
dcterms:relation <doc_iri/coatingsSurvey> .
<doc_iri/janeDoe> a schema:Person ; rdfs:label "Jane Doe" .
<doc_iri/coatingsSurvey> a prov:Entity ; rdfs:label "Coatings survey" ;
dcterms:identifier "PRJ-7" .
Keys match regardless of case and of snake_case, camelCase or kebab-case,
so DOI, doiId and doi_id all mean doi. To give an entity another class,
pass a dict with a type, such as {"name": "...", "type": "schema:Project"}.
Entities are minted per document; OntoCast does not merge the same author in
two documents into one. Cross-document identity
links entities that carry the same identifier without changing either.
Document metadata is always in the facts graph, even when the text never
mentions it, and it stays there when provenance is stripped. Without
document_metadata, ontocast process sets dcterms:title to the file name.
The facts are written to a named graph: by default the document namespace, so
two runs over the same converted document write one graph and the later one
replaces the earlier. A library caller that wants each run's facts kept apart
sets AgentState.facts_graph_iri (an absolute IRI or a URN) for the run; entity
IRIs stay in the document namespace either way, so the runs can be compared.
What to read next¶
- How OntoCast works: where each of these steps happens.
- Entity disambiguation: how
cd:entities from different units become one. - Cross-document identity: how one entity in several documents is linked.
- Triple stores: where ontologies and facts are stored.