Entity disambiguation¶
Each content unit is extracted on its own, so the same sample, company or person mentioned in three parts of a document arrives as three entities. This page explains how OntoCast decides which entities are the same, so you can predict when two of them will be merged and find out why two were not.
When two entities merge¶
Merging happens once per document, after every unit's facts are extracted (see How OntoCast works). Entities of different documents are never merged; Cross-document identity links them instead. For each pair of entities:
- Names alone prove nothing. Two units that both write
cd:temperature_valuemay mean two different measurements, so OntoCast keeps each unit's names apart and treats the pair like any other candidate (AGG_UNIT_SCOPED_FACT_IRIS). Terms of your ontology, such as an individual it declares, are the same wherever they appear. - A pair becomes a candidate when the embeddings of their labels are at
least
AGG_CANDIDATE_SIMILARITY_THRESHOLDsimilar, computed withAGG_EMBEDDING_MODEL. It is also a candidate, whatever its labels, when both entities state the same short value for a property that identifies things, such as a case number or a catalog code (AGG_NATURAL_KEY_MERGE). - Guards then check the candidate against what the two entities state, and any one of them can refuse it (see the table below). The threshold is set low on purpose: the guards, not the similarity score, decide.
- Refusals hold for the whole group. If A and C were refused, they stay apart even when A matches B and B matches C.
- The accepted groups are merged under one IRI. Entities that were not
merged but share a name keep it with a suffix:
cd:sample_aandcd:sample_a_1. - Validation can undo a merge. If the merged graph shows a sign of a wrong
merge, such as a single-valued property with two values, the offending
group is split and the document's facts are merged again, up to
FACTS_MERGE_REPAIR_PASSEStimes. See Validation and SHACL.
Why a merge was refused¶
retrieval_metrics.facts_rejected_merges counts the refused pairs. With
LOGGING_LEVEL=DEBUG,
the log names each pair and the reasons, which can be several:
| Reason | The pair was refused because | Setting |
|---|---|---|
same_unit |
One unit minted both: a unit names each thing once, so its two IRIs are two things | |
sibling |
Both are values of one subject, such as the lower and upper bound of a range, or two samples in a series | AGG_SIBLING_GUARD_SCOPE |
literal_conflict |
They state different values for the same property, such as two different temperatures | AGG_LITERAL_CONFLICT_GUARD |
functional_iri_conflict |
They point to different things through a property that allows one value, such as two different units | AGG_FUNCTIONAL_MIN_EMPIRICAL_SUPPORT |
initials_conflict |
Their labels differ only in an initial or a single letter: "Sample A" and "Sample B" | AGG_INITIALS_DISTINCT_GUARD |
role |
One is used as a property, the other as a thing | |
type |
Two entities found in the text: your ontologies declare their types, or the types' superclasses, disjoint (owl:disjointWith, owl:AllDisjointClasses), or their types share no class (superclasses included) and their names match only plausibly. Parts of a document type one thing differently, so differing types merge on an equal label, initials, a shared key or a coreference verdict. An entity and an individual of your ontology: they share no type |
AGG_TYPE_GUARD_UNTYPED decides whether a typed and an untyped entity may merge |
lexical |
Their labels are not close enough. Entities that carry values must match exactly; others may match loosely. Two labelled entities are compared by their labels alone; the local name of their IRI counts only when one has no label, or against a term of your ontology. Labels that carry different numbers ("June 2021", "29 June 2021") never match | AGG_LEXICAL_LABEL_JACCARD, AGG_LEXICAL_SEQUENCE_RATIO, AGG_LEXICAL_TOKEN_JACCARD |
cluster_veto |
The pair itself was fine, but merging it would have joined two entities another guard had refused |
A property counts as allowing one value when the ontology says so, or when
every subject in the document uses it once and at least
AGG_FUNCTIONAL_MIN_EMPIRICAL_SUPPORT subjects use it.
Entities that share a label are always compared, even when their embeddings place them apart; a label shared by more than eight entities is generic and brings nothing together.
Coreference the guards cannot settle¶
The parts of a document are extracted in parallel, so no part sees the entities another part named. After merging, one LLM call per document (of two parts or more) puts the pairs the rules above cannot settle to the model:
- merges made on a plausible match only, not on an equal label, initials or a shared key: "temperature" and "daily maximum temperature";
- candidates refused for their wording alone, with no guard saying they are distinct: "the first applicant" and "Jane Doe";
- two entities of one type from different parts that are the only two pointing to one node by one property: both "applicant in" one case.
The model sees each entity as its part stated it (labels, types, values,
relations, and the sentence that names it) and answers same, different or
unsure for each pair. "Same" merges the pair like a shared key, "different"
keeps it apart, and every guard above still applies: the model cannot merge
two things one part told apart. Its verdicts also hold when validation
re-merges the parts.
The step is on by default;
AGG_LLM_COREFERENCE=false
turns it off, and
AGG_LLM_COREFERENCE_MAX_PAIRS
bounds the pairs per call. facts_coreference_pairs,
facts_coreference_same and facts_coreference_different in
retrieval_metrics count them.
Tuning¶
- Too many duplicates (one thing, several entities): lower
AGG_CANDIDATE_SIMILARITY_THRESHOLD. More pairs reach the guards; the guards still refuse the ones whose facts disagree. - Wrong merges (two things, one entity): find the pair in the log. If no guard fired, raise the threshold or the lexical settings; if a guard is off, turn it on.
- A guard that is switched off shows its effect as a change in
facts_rejected_merges, which is how to measure what it does on your documents. - To see what merging does at all, run once with
AGG_ENABLED=false: every entity stays its own node, with its provenance, and no candidate is formed. It needsAGG_UNIT_SCOPED_FACT_IRIS=true, or same-named entities from different units fuse by name anyway. Validation can still rewrite the merged graph; setFACTS_MERGE_REPAIR_PASSES=0for a graph that only extraction shaped.
Comparing two graphs¶
The same matching compares an extracted graph with a reference one, for
example to evaluate a run against a hand-made graph. Three routes do this:
POST /match/entities aligns entities across graphs, POST
/match/derive-matches turns that into pairs, and POST /match/evaluate
scores precision, recall and F1 for triples, facts and entities. Entities are
the subjects and objects that are not vocabulary (classes, properties,
predicates). A score with nothing to divide by, such as precision for an empty
prediction, is null, so it does not count as zero in an average. The
match-graphs command runs the same comparison over two directories of Turtle
files. See the HTTP API.
The route, the command and the ontocast_align_entities agent tool use
AGG_SIMILARITY_THRESHOLD
and AGG_EMBEDDING_MODEL
unless the caller names a similarity_threshold or embedding_model. Neither
affects merging during a run, which uses only the candidate threshold above.
Facts scores count only relations between instances, such as a sample to its substrate; triple scores also count types and the class hierarchy. An ontology term that both graphs use is matched once and is not counted as an extra entity on either side.
What to read next¶
- Aggregation settings: every setting with its default.
- Validation and SHACL: the checks that run after merging.
- Performance tuning: sharing one embedding model between chunking, retrieval and merging.