Skip to content

Reading run telemetry

Every run reports what it cost, where its time went, which parts of the document failed, what ontology retrieval found and whether the facts passed validation. This page shows where those reports are and which fields answer each question. Telemetry counters lists every key.

Where to find it

Where What you get
/process and /process_unit responses metadata: status, budget, retrieval_metrics, unit_context_sources, failed_units, facts_conformance, facts_validation_findings, facts_repairs, facts_gate_repairs, improvement_suggestions, ontology_versions
ontocast process Per document, beside <name>.facts.ttl: <name>.run.json, the run manifest, and <name>.facts.validation.json when validation has something to report. With --keep-provenance, also <name>.units.jsonl; with --dump-catalog, <name>.catalog.nq beside the ontology files
The log A summary of calls, tokens, durations and effective workers at INFO when a document finishes

ontocast process writes these files into --output-dir (or --facts-output-dir), and next to the input file when neither is given. For a .jsonl input with several records, each name carries the line number, such as <name>.L3.run.json.

What did it cost

budget counts calls and tokens:

Field Meaning
calls_count Calls sent to the provider
cache_hits Calls answered from the LLM response cache
input_tokens, output_tokens Tokens of the calls sent to the provider
cached_input_tokens, cached_output_tokens Tokens of cached answers: what those calls cost when they were first made. Not added to the provider totals
reasoning_tokens Reasoning tokens of the calls sent, included in output_tokens
cache_read_input_tokens Input tokens of the calls sent that the provider served from its own prompt cache, included in input_tokens
replayed_reasoning_tokens, replayed_cache_read_input_tokens The same for cached answers
prefix_cache_hit_rate Provider-cache reads over all input tokens, sent and cached
reasoning_share_of_output Reasoning tokens over all output tokens, sent and cached

Providers report tokens differently, and one that reports none leaves the token fields at 0 and both ratios at null. A null ratio means unmeasured, not zero. The two ratios say where to aim a cost change: a high reasoning_share_of_output points at the model's reasoning budget, a low prefix_cache_hit_rate at prompt reuse; see Performance tuning. OntoCast reports tokens, not money.

budget.tokens_by_phase splits the same counts by what each call was for: facts_render, facts_critic, facts_completion, facts_coreference, ontology_identity, ontology_plan, ontology_render, ontology_critic, ontology_consolidation, ontology_curation, ontology_selection, evidence_planning, unit_summary and section_classification. Each entry holds calls, replays, input, output, reasoning, cache_read, cache_write, replayed_input and replayed_output, in the same terms as a RunMeter. Across phases they add up to the totals above, so a run's cost can be attributed to its steps without reading prompt caches. An unlabeled entry means a call the pipeline does not name. A content unit that fails keeps its calls out of budget altogether; a RunMeter still counts them.

budget is complete once the run ends. To follow usage while the run is in progress, or to stop the run at a limit, bind a RunMeter; see Limit what a run spends.

Before reading cost or quality, check three counters in budget.counters:

  • llm/rate_limited and llm/timeouts: calls lost to throttling or to the request timeout. Each fails a render, so a run with many of them describes the provider, not the extraction.
  • llm/parse_abandoned: calls whose answer never parsed. Each is a content unit that contributed nothing. llm/parse_retry counts the re-sent renders before that point; How LLM responses are parsed explains both.

Where did the time go

budget.node_durations holds seconds per pipeline step. A plain step name, such as Chunk Text or Render Facts, is that step's wall clock. The two steps that process content units in parallel, Update Ontology and Render Facts, have more keys:

Key Meaning
<step>/unit_sum Time of every unit added up. Divided by the wall clock, it is the number of units that effectively ran at once
<step>/worker_wait Time units waited for a PARALLEL_WORKERS slot
<step>/loop_lag_total Time CPU work blocked every unit at once
llm/provider Time inside provider calls
llm/inflight_wait Time calls waited behind LLM_MAX_INFLIGHT

Measure your own runs explains how to act on these.

Which units failed

outcome.metadata.failed_units lists each content unit that produced nothing, with its unit_index, phase (ontology, facts or summarize), stage and reason. It is empty on a fully successful run. When no unit produced output, the document's result in a /process response has status_code 422 and error_code no_units_extracted, and ontocast process records the file as failed and exits non-zero.

A request the provider refuses as configured, such as a wrong key, a model your account cannot use, or an unsupported parameter, fails every call the same way. OntoCast stops instead of finishing an empty run: the call is counted under llm/calls_rejected, /process answers with an error, and ontocast process exits with code 78 and writes no output files for that document.

Did retrieval find anything

retrieval_metrics describes the ontology context each content unit was shown:

Key What to look for
ontology_context_mode The mode that chose the context
ontology_snapshot_triples How large the context was. Compare it with ONTOLOGY_CONTEXT_MAX_TRIPLES: at or above it, the context was condensed or passed through oversized
empty_snapshot_reason Why a unit's context was empty. Only the last unit's reason is kept
patch_retrieval In selected_vector_search_ontology mode, what retrieval matched: query count, terms kept (atoms_final), terms per ontology, and which whole small vocabularies were included or left out

unit_context_sources lists, per unit, the ontology IRIs its context was drawn from, by lane: ontology for the ontology stage, facts for the facts stage. Use it to check which ontologies each unit could see; the per-unit values in retrieval_metrics keep only the last unit's.

Set ONTOLOGY_PATCH_DUMP_ONTOLOGY_RANKS to add a per-query ranking of the ontologies to patch_retrieval. Retrieval explains what each number measures.

Did validation pass

metadata.facts_conformance summarizes validation of the returned facts: whether SHACL ran, whether the graph conforms, and counts by kind of finding. facts_validation_findings lists the findings that remain after every repair, and facts_gate_repairs and facts_repairs list the repairs OntoCast made without the LLM. If retrieval_metrics.validated_without_ontology_context is true, the facts were checked against no catalog vocabulary at all. The .facts.validation.json file holds the same information for ontocast process. Validation explains the findings.

The run manifest's critic block records what the facts critic did. Read calls before accepted: FACTS_CRITIC_PASSES is 0 by default, so calls: 0 is normal; above zero, the critic was turned on, and calls counts its reviews. accepted counts the reviews whose verdict let the unit leave the loop. The ontology_critic block is the same for the ontology loop, whose critic is off by default.

The run manifest

ontocast process writes one run manifest per document. It records the settings, the cost and the outcome, so you can compare two runs by diffing their manifests instead of rerunning them. The HTTP routes return the same budget and retrieval_metrics but write no manifest; GET /config returns the server's effective_config.

{
  "source": "paper.pdf",
  "ontocast_version": "<version>",
  "render_mode": "ontology_and_facts",
  "effective_config": {"AGG_ENABLED": true, "CHUNK_CHUNKING_MODE": "semantic",
                       "LLM_API_KEY": "<redacted>"},
  "chunking_mode": "semantic",
  "loops": {"max_visits": 1, "facts_critic_passes": 1, "ontology_critic_passes": 0},
  "llm": {"provider": "<provider>", "model_name": "<model>", "temperature": 0.0, "max_inflight": 16},
  "prompting": {"llm_graph_format": "turtle", "llm_output_layout": "compact",
                "ontology_chapter_format": "inherit",
                "ontology_context_scope": "unit", "fanout_warmup_units": 0,
                "parallel_workers": 16, "embedding_model_name": "<model>"},
  "budget": {"calls_count": 0, "input_tokens": 0, "output_tokens": 0,
             "node_durations": {"Render Facts": 0.0, "Render Facts/unit_sum": 0.0},
             "counters": {"llm/parse_retry": 0}},
  "selection": {"exclude_sections": ["references"], "bibliography_mode": "skip",
                "labeled_units": 0, "unlabeled_units": 0},
  "critic": {"calls": 0, "accepted": 0, "score_histogram": {}},
  "completion": {"calls": 0, "measurements_recovered": 0},
  "facts_triples": 0,
  "facts_triples_serialized": 0,
  "ontology_triples": 0,
  "retrieval_metrics": {"ontology_context_mode": "selected_single_ontology"}
}

The blocks you read most:

Block Answers
llm, loops, prompting Which model and settings produced the run, as actually applied
unit_context_sources Ontology IRIs behind each unit's context, by lane and unit index
effective_config Every setting, keyed by environment variable, credentials redacted. Diff two manifests' blocks to see every setting that differed, including those that make no LLM call
chunking_mode The mode the chunker ran in. naive under a semantic setting means the semantic-chunking extra is missing. Absent on the single-unit path, which does not chunk
budget Cost and time, as above
selection Which sections the run was given, how many units carried a section label, and how many were skipped as bibliography, too short or not content. route_histogram gives every route with its reason and size, which is where a section that went missing shows up. An exclusion list acts only on units with a strong label; label_strength_chars says how much of the kept text that is
critic, ontology_critic, completion What the review and completion passes did
validation_config The validation settings in effect, including the size of the loaded shapes
graph_metrics How connected the written facts graph is: components, largest component, isolated nodes

facts_triples is not the size of the .facts.ttl file

facts_triples counts the facts graph in memory, provenance included. facts_triples_serialized counts what the file holds after provenance is removed. Compare runs on facts_triples_serialized when the question is about extracted content.

Tracing a fact to its text, and the catalog across a batch

Two ontocast process options write files for checking a run against its input:

  • --keep-provenance keeps, in <name>.facts.ttl, the unit each statement came from (prov:wasDerivedFrom <doc_iri>/<hid>), and writes <name>.units.jsonl: one row per content unit with index, hid, iri, section_label, summarized and text. A statement's unit IRI is a row's iri. text is the unit as chunked, from the converted document, so markup the converter added is part of it; summarized is true when the model was shown a summary instead.
  • --dump-catalog writes <name>.catalog.nq after each document: every ontology in the catalog at that point, as the next document will see it, in sorted N-Quads with the ontology IRI as graph name. Comparing consecutive dumps shows how the catalog changed. Blank-node labels differ between dumps, so canonicalise them before comparing statements about blank nodes.

A batch run on the default in-memory store starts from the seed directory alone, so seeding a run with a saved catalog is how to run against it frozen (RENDER_MODE=facts). On Fuseki, an ontology already stored under an IRI is kept over a seed with the same IRI: use a fresh tenant or project for that.

External tracing

OntoCast is a LangGraph graph over LangChain chat models and has no tracing code of its own, so tools that trace LangChain show each pipeline step and each LLM call with its prompt, answer and token usage. LangSmith is enabled from the environment and works with ontocast serve and ontocast process:

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=...
export LANGSMITH_PROJECT=ontocast
ontocast process --input-path doc.pdf --output-dir ./out

Langfuse and OpenTelemetry instrumentation attach as LangChain callbacks, which OntoCast has no setting for. Attach them in your application when you embed OntoCast: make_ontocast_node passes your callbacks, tags and run metadata through to every step. See Embedding OntoCast in your agent.

Cached calls leave no LLM span

A call answered from the LLM response cache never reaches the provider, so it produces no LLM span, while budget still counts it. For a complete trace, set LLM_CACHE_ENABLED=false; every call is then sent to the provider.