Skip to content

Entity disambiguation

Each content unit is extracted on its own, so the same sample, company or person mentioned in three parts of a document arrives as three entities. This page explains how OntoCast decides which entities are the same, so you can predict when two of them will be merged and find out why two were not.

When two entities merge

Merging happens once per document, after every unit's facts are extracted (see How OntoCast works). Entities of different documents are never merged; Cross-document identity links them instead. For each pair of entities:

  1. Names alone prove nothing. Two units that both write cd:temperature_value may mean two different measurements, so OntoCast keeps each unit's names apart and treats the pair like any other candidate (AGG_UNIT_SCOPED_FACT_IRIS). Terms of your ontology, such as an individual it declares, are the same wherever they appear.
  2. A pair becomes a candidate when the embeddings of their labels are at least AGG_CANDIDATE_SIMILARITY_THRESHOLD similar, computed with AGG_EMBEDDING_MODEL. It is also a candidate, whatever its labels, when both entities state the same short value for a property that identifies things, such as a case number or a catalog code (AGG_NATURAL_KEY_MERGE).
  3. Guards then check the candidate against what the two entities state, and any one of them can refuse it (see the table below). The threshold is set low on purpose: the guards, not the similarity score, decide.
  4. Refusals hold for the whole group. If A and C were refused, they stay apart even when A matches B and B matches C.
  5. The accepted groups are merged under one IRI. Entities that were not merged but share a name keep it with a suffix: cd:sample_a and cd:sample_a_1.
  6. Validation can undo a merge. If the merged graph shows a sign of a wrong merge, such as a single-valued property with two values, the offending group is split and the document's facts are merged again, up to FACTS_MERGE_REPAIR_PASSES times. See Validation and SHACL.

Why a merge was refused

retrieval_metrics.facts_rejected_merges counts the refused pairs. With LOGGING_LEVEL=DEBUG, the log names each pair and the reasons, which can be several:

Reason The pair was refused because Setting
same_unit One unit minted both: a unit names each thing once, so its two IRIs are two things
sibling Both are values of one subject, such as the lower and upper bound of a range, or two samples in a series AGG_SIBLING_GUARD_SCOPE
literal_conflict They state different values for the same property, such as two different temperatures AGG_LITERAL_CONFLICT_GUARD
functional_iri_conflict They point to different things through a property that allows one value, such as two different units AGG_FUNCTIONAL_MIN_EMPIRICAL_SUPPORT
initials_conflict Their labels differ only in an initial or a single letter: "Sample A" and "Sample B" AGG_INITIALS_DISTINCT_GUARD
role One is used as a property, the other as a thing
type Two entities found in the text: your ontologies declare their types, or the types' superclasses, disjoint (owl:disjointWith, owl:AllDisjointClasses), or their types share no class (superclasses included) and their names match only plausibly. Parts of a document type one thing differently, so differing types merge on an equal label, initials, a shared key or a coreference verdict. An entity and an individual of your ontology: they share no type AGG_TYPE_GUARD_UNTYPED decides whether a typed and an untyped entity may merge
lexical Their labels are not close enough. Entities that carry values must match exactly; others may match loosely. Two labelled entities are compared by their labels alone; the local name of their IRI counts only when one has no label, or against a term of your ontology. Labels that carry different numbers ("June 2021", "29 June 2021") never match AGG_LEXICAL_LABEL_JACCARD, AGG_LEXICAL_SEQUENCE_RATIO, AGG_LEXICAL_TOKEN_JACCARD
cluster_veto The pair itself was fine, but merging it would have joined two entities another guard had refused

A property counts as allowing one value when the ontology says so, or when every subject in the document uses it once and at least AGG_FUNCTIONAL_MIN_EMPIRICAL_SUPPORT subjects use it.

Entities that share a label are always compared, even when their embeddings place them apart; a label shared by more than eight entities is generic and brings nothing together.

Coreference the guards cannot settle

The parts of a document are extracted in parallel, so no part sees the entities another part named. After merging, one LLM call per document (of two parts or more) puts the pairs the rules above cannot settle to the model:

  • merges made on a plausible match only, not on an equal label, initials or a shared key: "temperature" and "daily maximum temperature";
  • candidates refused for their wording alone, with no guard saying they are distinct: "the first applicant" and "Jane Doe";
  • two entities of one type from different parts that are the only two pointing to one node by one property: both "applicant in" one case.

The model sees each entity as its part stated it (labels, types, values, relations, and the sentence that names it) and answers same, different or unsure for each pair. "Same" merges the pair like a shared key, "different" keeps it apart, and every guard above still applies: the model cannot merge two things one part told apart. Its verdicts also hold when validation re-merges the parts. The step is on by default; AGG_LLM_COREFERENCE=false turns it off, and AGG_LLM_COREFERENCE_MAX_PAIRS bounds the pairs per call. facts_coreference_pairs, facts_coreference_same and facts_coreference_different in retrieval_metrics count them.

Tuning

  • Too many duplicates (one thing, several entities): lower AGG_CANDIDATE_SIMILARITY_THRESHOLD. More pairs reach the guards; the guards still refuse the ones whose facts disagree.
  • Wrong merges (two things, one entity): find the pair in the log. If no guard fired, raise the threshold or the lexical settings; if a guard is off, turn it on.
  • A guard that is switched off shows its effect as a change in facts_rejected_merges, which is how to measure what it does on your documents.
  • To see what merging does at all, run once with AGG_ENABLED=false: every entity stays its own node, with its provenance, and no candidate is formed. It needs AGG_UNIT_SCOPED_FACT_IRIS=true, or same-named entities from different units fuse by name anyway. Validation can still rewrite the merged graph; set FACTS_MERGE_REPAIR_PASSES=0 for a graph that only extraction shaped.

Comparing two graphs

The same matching compares an extracted graph with a reference one, for example to evaluate a run against a hand-made graph. Three routes do this: POST /match/entities aligns entities across graphs, POST /match/derive-matches turns that into pairs, and POST /match/evaluate scores precision, recall and F1 for triples, facts and entities. Entities are the subjects and objects that are not vocabulary (classes, properties, predicates). A score with nothing to divide by, such as precision for an empty prediction, is null, so it does not count as zero in an average. The match-graphs command runs the same comparison over two directories of Turtle files. See the HTTP API.

The route, the command and the ontocast_align_entities agent tool use AGG_SIMILARITY_THRESHOLD and AGG_EMBEDDING_MODEL unless the caller names a similarity_threshold or embedding_model. Neither affects merging during a run, which uses only the candidate threshold above.

Facts scores count only relations between instances, such as a sample to its substrate; triple scores also count types and the class hierarchy. An ontology term that both graphs use is matched once and is not counted as an extra entity on either side.