Skip to content

Chunking

How text is cut into content units and which sections are kept.

CHUNK_MIN_SIZE

Default: 3000 · Type: int

Smallest chunk in characters. Shorter neighbouring pieces are merged until they reach it, without passing CHUNK_MAX_SIZE.

CHUNK_MAX_SIZE

Default: 6000 · Type: int

Largest chunk in characters. Raising it makes fewer, longer LLM calls, and the facts render then extracts fewer facts per character; lowering it extracts more at a proportional token cost.

CHUNK_EMBEDDING_MODEL

Default: sentence-transformers/paraphrase-multilingual-mpnet-base-v2 · Type: str

Sentence-transformers checkpoint for semantic chunking, embedding-based schema detection, and the embedding vote that labels headings no pattern or keyword names (classifier 'heuristic' or 'llm'). Shared process-wide with EMBEDDING_MODEL_NAME and AGG_EMBEDDING_MODEL when the names match, so aligning all three halves resident local-model memory. Changing it invalidates the on-disk chunk cache and shifts chunk boundaries.

CHUNK_CHUNKING_MODE

Default: semantic · Type: one of semantic, naive

How the chunker splits text: 'semantic' clusters sentence embeddings (needs the semantic-chunking extra, else it falls back to 'naive' with a warning); 'naive' splits by size alone. Part of the chunk cache key.

CHUNK_SECTION_CLASSIFIER

Default: heuristic · Type: one of llm, heuristic, heading, off

Section classification tiers, in increasing cost; each adds its vote and the strongest decides: 'off' = no section tagging (section filters and boilerplate exclusions are off; reference lists and front/back matter are still recognised from content); 'heading' = document outline plus heading pattern/keyword matching; 'heuristic' (default) = heading plus an embedding vote on section-like headings no pattern or keyword names (when the chunker's model is installed) and a content-density vote on every part, which also labels regions with no usable heading; 'llm' = heuristic plus one LLM call that labels the whole outline, as a weak vote. Only 'llm' makes LLM calls during chunking.

CHUNK_SECTION_TAG_MIN_CHARS

Default: 80 · Type: int

Min stripped length of a segment; smaller segments merge into neighbours before they are typed

CHUNK_SECTION_TEXT_HEADINGS

Default: true · Type: bool

Detect headings from plain-text layout (short, blank-line delimited, upper-case or numbered lines) in documents whose conversion produced no markdown heading structure at all.

CHUNK_SECTION_DENSITY

Default: conservative · Type: one of off, conservative, aggressive

Content-density section classification for regions with no usable heading. 'conservative' (default) recognises only reference lists and acknowledgements, whose surface form is near-unique. 'aggressive' also guesses methods/results/introduction from figure-reference, quantity and citation densities -- these do not separate those sections cleanly, and a wrong label is silently acted on by the section filters, so it is opt-in. Requires CHUNK_SECTION_CLASSIFIER=heuristic or llm.

CHUNK_SECTION_SCHEMA_DETECT

Default: headings · Type: one of off, lexical, headings, auto

How to infer the document-type schema when the request names none and its document_type_hint matches none: 'off' uses the manifest default; 'lexical' scores headings against each schema's vocabulary; 'headings' adds an embedding tier when the semantic extras are installed; 'auto' also allows a weaker content-based tier for documents with almost no headings. An explicit schema or a matching hint always wins, and detection falls back to the default rather than guess.

CHUNK_SECTION_SCHEMA_DETECT_MIN_SCORE

Default: 2.0 · Type: float

Minimum distinctive evidence (headings recognised by exactly one candidate schema) before a detection is accepted.

CHUNK_SECTION_SCHEMA_DETECT_MIN_MARGIN

Default: 1.8 · Type: float

Factor by which the winning schema's score must exceed the runner-up's; below it detection falls back to the default schema. Raise it to detect less often and more surely.

CHUNK_SECTION_SCHEMA_DETECT_CONTENT_MIN_MARGIN

Default: 4.0 · Type: float

Stricter margin for the content-based tier: body prose from one document type can resemble another's, so content separates schemas less sharply than headings do. Only used when CHUNK_SECTION_SCHEMA_DETECT=auto.

CHUNK_SECTION_LLM_BATCH_SIZE

Default: 40 · Type: int

Outline regions per LLM call when CHUNK_SECTION_CLASSIFIER=llm: a region is a headed section or a part of unheaded text. A longer outline is sent in consecutive windows of this size.

CHUNK_SECTION_FILTER_ON_EMPTY

Default: warn · Type: one of warn, error

What to do when a section selection removes every unit. 'warn' (default) logs and continues, which yields an empty facts graph indistinguishable from a document that genuinely had nothing to extract; 'error' fails the document instead (status 422 in its /process result; the batch run counts it as failed and exits non-zero). Covers the target_sections allowlist and the exclusions: exclude_sections, or the schema's boilerplate sections when it is unset.

CHUNK_BIBLIOGRAPHY_MODE

Default: skip · Type: one of domain_facts, citations_only, skip

Routing for units detected as reference lists: by their content alone, or by a strong citations-class label (such as an exact References heading) their content agrees with; a reference list at a unit's start or end is first cut into a unit of its own. 'skip' (default) drops those units before extraction, 'citations_only' extracts bibliographic metadata only, 'domain_facts' disables special handling.

CHUNK_MIN_UNIT_CHARS

Default: 0 · Type: int

Drop content units shorter than this many characters before extraction; 0 disables the floor. Unlike CHUNK_MIN_SIZE, which the chunker only aims at, this is enforced: a heading stub or caption fragment would otherwise cost a retrieval, a render and a critic call. Size it from the per-unit node durations in the budget summary.

CHUNK_NON_CONTENT_MODE

Default: extract · Type: one of extract, skip

What to do with units that state no domain facts: front or back matter (a strong non-content label, or a heading the document's schema names as front or back matter, on text with no number with a unit; under any schema, text made mostly of emails, URLs, ORCIDs and initials, or a short licence notice), and tables of contents or lists of figures or tables. 'extract' keeps such a unit and marks it is_non_content; 'skip' drops it before extraction and counts it in the run manifest.

CHUNK_MAX_MEASUREMENTS_PER_UNIT

Default: 0 · Type: int

Split a sized unit at the sentence boundary nearest its midpoint, recursively, while it states more unit-adjacent numbers than this; 0 (default) disables. A unit dense with measurements asks the model for many facts in one call; this splits only those units and leaves every other unit's size unchanged. Pieces never go below min_size: a dense unit shorter than twice min_size is left whole.

CHUNK_CITATION_VOCABULARY

Default: {"work_class": "schema:ScholarlyArticle", "fallback_class": "schema:CreativeWork", "tit... · Type: dict[str, str]

Terms the citation-metadata prompt uses in 'citations_only' mode, by role. Bibliographic entries are not domain facts, so unlike the rest of the pipeline these terms are not retrieved from the catalog -- they default to schema.org and are overridden here for catalogs that model citations with another vocabulary (e.g. bibo, FaBiO, DCMI). Keys are fixed roles; values are CURIEs or IRIs. Setting an empty mapping drops the vocabulary guidance.