Skip to content

Structured documents

Often only part of a document is worth extracting: the methods and results of a paper, the risk factors of an annual report. This page explains how OntoCast labels each part of a document with its section, how it decides which kind of document it is reading, how it decides which parts to drop, and how you choose the sections to extract.

Section labels

While chunking, OntoCast gives every content unit the label of the section it came from, such as methods, results or risk_factors. Units never span two labeled sections, so each has at most one label. A unit OntoCast cannot place confidently gets no label at all: an unlabeled unit is merely never selected by target_sections, while a wrongly labeled one would be selected by mistake. For the same reason, size never joins unlabeled text to a labeled unit, nor text with a weak label to text with a strong one; neighbouring unlabeled sections are joined with each other instead. A label alone never drops a unit: see how a unit is routed.

Labels are recorded in the facts' provenance, together with the step that decided each and its strength.

Section schemas

Labels only mean something for a kind of document. A section schema is the set of labels for one kind, with the headings that identify each label and a one-line description of what each label's sections do, which the model is shown. Every document is read against one schema:

Schema Documents Boilerplate Front and back matter Reference lists
academic (default) Research articles, conference papers, theses, reviews acknowledgements back_matter references
financial Annual and quarterly reports, filings references
clinical Trial protocols and study reports back_matter references
legal Contracts and other agreements between parties
patent Patents and patent applications citations
standard Specifications and standards for implementers acknowledgements normative_references, references
manual User guides, installation guides, runbooks
fiction Novels and short fiction acknowledgements
news News articles and press releases boilerplate, contact
general Anything else references

Each label has a route class in its schema file: content (every label not listed above), boilerplate, non_content (front and back matter: author blocks, ORCID, notes, availability and licence statements) or citations. The class is evidence for routing, never a decision on its own. Because the vocabulary belongs to the schema, Notes is back matter in a paper and content in a manual.

Use ontocast sections (below) to see a schema's labels on a real document.

Which schema a document gets

OntoCast never overrides what you say; it only fills the gap when you say nothing. In order:

  1. section_schema_id, when you pass it.
  2. document_type_hint, when it names a known kind of document as a whole word: 10-K and annual report mean financial, thesis means academic, RFC means standard.
  3. Detection from the document's own headings.
  4. academic, as a fallback: nothing chose it, so its boilerplate and front- and back-matter sections are routed as content.

A thesis whose headings are only Chapter 1, Chapter 2… is detected as fiction, whose novels have the same headings; pass document_type_hint=thesis for such documents.

Detection is set by CHUNK_SECTION_SCHEMA_DETECT:

Value How the schema is detected Cost
off Not at all; the default schema is used None
lexical From headings that only one schema recognizes None
headings (default) The above, then each heading votes for the schema whose labels it is closest to The chunker's local embedding model (semantic-chunking extra), used only when the first step cannot decide. Without the extra, this is lexical
auto The above, then the body text, for documents with almost no headings Same model, more text

Detection gives up rather than guess, and the document is then read as academic: a wrong schema would mislabel every section. Raising CHUNK_SECTION_SCHEMA_DETECT_MIN_MARGIN makes it detect less often and more surely. general is never detected; pass it explicitly.

How sections are recognized

CHUNK_SECTION_CLASSIFIER sets how far OntoCast goes to label a section. Each step adds its vote:

Value Steps LLM calls
off None. Units are not labeled, so section filters and boilerplate exclusions are off; reference lists and front matter are still recognized from their content None
heading The document's outline; heading patterns and keywords, so Results and Discussion and Experimental Section are recognized None
heuristic (default) The above, plus the nearest label of a heading no pattern or keyword names, and a vote from each part's content, which also labels regions without headings None
llm The above, plus one call in which the model labels the whole outline One per document, or one per window of CHUNK_SECTION_LLM_BATCH_SIZE regions for a long outline

The nearest-label step compares a section-like heading with each label's name and keywords, using the chunker's embedding model (semantic-chunking extra; without it the step is skipped). It recovers paraphrases and other languages (Closing Thoughts, Schlussfolgerung) and abstains on topical headings (Perovskite Stability), which are near no label. It names content sections only: a heading nearest a boilerplate, back-matter or reference-list label (a publisher's Access banner) stays unlabeled.

The model is shown each headed section once, by its heading and the start of its text, and each part of unheaded text on its own, with every label's description from the schema. It labels every region. Its vote is weak: it decides where no heading word or content spoke, overriding only the nearest-label step; where it disagrees with another step, its vote is kept in the unit's evidence. A long outline is sent in windows of CHUNK_SECTION_LLM_BATCH_SIZE regions. A window whose call fails costs only its votes: the run manifest's selection.outline_labels counts the windows sent and failed, by error class, and ontocast sections prints the same line.

A heading OntoCast does not recognize still ends the section before it, so a label never runs on into the next section. Where a document numbers its headings, the numbers say where each belongs: 2.1 sits inside 2 and takes its label unless it names its own, while a numbered top-level heading OntoCast does not recognize (4 Aging of encapsulated films) opens an unlabeled section of its own. Markdown heading levels place headings the same way in documents that use more than one level below the title. Elsewhere a descriptive heading (Spin coating of the precursor solution) belongs to the section before it. A reference list has no subsections either way. A heading with a subtitle is read by its head: Appendix A: Derivation of the rate opens the appendix. A keyword is read as the heading's head noun: Characterization Results is results, Summary of Results a summary, Results and Discussion results. References is recognized only as a whole part of a heading (Notes and References), never inside a phrase (Comparison with literature references).

Two settings refine this:

  • CHUNK_SECTION_DENSITY decides what content alone may label. conservative (default) recognizes only reference lists and acknowledgements (at least two thanks or funding statements), whose form is unmistakable, so these votes are strong. Such a label applies to that part only; it is not carried to the parts after it. aggressive also guesses methods, results and introduction, which content does not separate cleanly: these votes are weak, so they can select a unit (target_sections, summarize_sections) but never drop one.
  • CHUNK_SECTION_TEXT_HEADINGS finds headings in plain text, for documents whose conversion produced no heading structure.

Choosing the sections to extract

Pass these with the request, on /process (query, form or JSON body) or as flags of ontocast process (--target-sections and so on). They are not environment settings.

Parameter What it does
target_sections Keep only these sections
exclude_sections Drop these sections, where the label is strong (see below). Unset, the schema's boilerplate sections are dropped; an empty value excludes no section
section_schema_id The schema to read the document against
document_type_hint Free text describing the document, from which the schema is chosen
summarize_sections Summarize the units of these sections before extraction; * or empty for all units. It never changes what is extracted
summary_max_sentences The length of each summary

Lists are comma-separated (results,methods) or a JSON array. For example:

curl -X POST http://127.0.0.1:8999/process \
  -F "file=@paper.pdf" -F "target_sections=methods,results"
ontocast process --input-path ./reports --output-dir ./out \
  --section-schema-id financial --target-sections risk_factors,md_and_a

target_sections applies first, to labels of any strength, then routing. A label no schema knows is dropped with a warning; a list in which no label is known is rejected (HTTP 400, or a usage error from the CLI), so a typo cannot silently switch filtering off.

When a selection removes every unit, the run by default finishes with an empty graph, which looks the same as a document with nothing to extract. Set CHUNK_SECTION_FILTER_ON_EMPTY=error to fail the document instead: its result on /process has status 422, and ontocast process counts the file as failed and exits non-zero. This also covers a schema's boilerplate exclusions emptying a document on their own.

target_sections keeps only labeled units, so text whose label is missing is lost to it. The run manifest records the characters it removed and how many of them were unlabeled, and a warning is logged when it removes most of a document.

Summaries

With summarize_sections, each selected unit is summarized by the model just before it is extracted, and the render and critic read the summary instead of the full text. That costs one LLM call per unit and loses detail, so use it for long, discursive sections rather than for tables of measurements. A unit is summarized once, even when both the ontology and the facts block run.

How a unit is routed

Before extraction every unit takes one route: extracted, extracted as bibliographic records, kept but flagged as front or back matter, or dropped. Apart from the size floor and exclude_sections, a drop needs a strong label and content that agrees, or content that is conclusive on its own. A label is strong when it comes from an exact heading (References, Acknowledgements, Author Contributions), from a heading that its number or level places inside a section with a strong label (2.1 under 2 Methods), or from the content itself (a reference list, acknowledgements); a keyword inside a longer heading (Funding Mechanisms), a heading's nearest label, a label a heading takes from the section before it by its wording, or one set by the model is weak, and never drops a unit.

Every step that recognizes something adds its vote to the unit, and the strongest vote decides; between equally strong votes the heading's words win, then the content, then the model, then a heading's nearest label. So acknowledgements under a heading that only seemed to continue the results are labeled acknowledgements, while a Methods heading keeps its label even over text that reads like thanks.

The first row that matches a unit decides its route:

Unit Route
Shorter than CHUNK_MIN_UNIT_CHARS Dropped
A strong label exclude_sections names Dropped
A reference list, by its content (nearly every paragraph is one cited work), or under a strong reference-list label its content agrees with CHUNK_BIBLIOGRAPHY_MODE: skip (default) drops it; citations_only extracts bibliographic records (works, authors, venues), never domain facts; domain_facts treats it like any other unit
A strong boilerplate label on text that states no measurement, when exclude_sections is unset and the schema was not a fallback Dropped
A table of contents or a list of figures or tables, by its content (lines or table rows ending in ascending page numbers) CHUNK_NON_CONTENT_MODE: extract (default) keeps it, marked as non-content; skip drops it
Front or back matter that states no measurement: a strong non_content-class label, or a leading heading the schema names as front or back matter or boilerplate; or, under any schema or none, text made mostly of identifiers (emails, ORCIDs) or a short licence notice CHUNK_NON_CONTENT_MODE: extract (default) keeps it, marked as non-content; skip drops it
Anything else Extracted

A reference list at the start or end of a unit, such as an unheaded bibliography after a conclusion, is first cut into a unit of its own, so the prose around it is kept. A cited work opens with a marker ([12], 12., [RFC3986]) or its authors, and carries a year and a venue; an entry that a page break or line wrap split over several paragraphs counts as one. Table rows never count as cited works: a table that cites its sources holds data. Nor do captions, formulas and code, which a converted document marks as such, or an entry that states a measurement outside its links and DOIs, which is how a discussion of prior studies differs from a reference list. A table whose header names its columns is data too, never a table of contents.

Every unit records the reason for its route (route_reason), every route other than plain extraction is logged with the unit's size, label and the step that labeled it, and the run manifest's selection.route_histogram counts units and characters per route and reason. In citations_only mode the record vocabulary defaults to schema.org; set CHUNK_CITATION_VOCABULARY to use another, such as BIBO. The detectors work on English, Latin-script documents; elsewhere they find less, never more.

Preview before you extract

A wrong label changes what is extracted without showing up anywhere in the result. ontocast sections shows the decisions before you spend anything:

ontocast sections --input-path ./paper.pdf
ontocast sections --input-path ./paper.pdf --target-sections results --as-json

It prints the schema (with the step that chose it and the evidence for the runners-up), the document's outline with each heading's number and parent, and every unit's label with the step that decided it and its strength, its route with the reason, and how many converted items it holds, then every step's vote on each unit, so you see what would be dropped and why. It makes no LLM calls and needs no credentials, unless the section classifier is llm (--section-classifier llm, or CHUNK_SECTION_CLASSIFIER=llm).