Skip to content

ontocast.tool.chunk.prepare

Prepare content units: segment, label, select, size, trim and route.

Labelling and routing are separate. Every labelling tier adds evidence to a segment and one rule decides its label (:mod:ontocast.tool.chunk.assignment); the caller's target_sections selects among the labels; sizing builds units; citation runs at a unit's edge are trimmed into units of their own; and :func:ontocast.tool.chunk.routing.route_unit decides each unit's route from its label, the label's strength and its content. Every unit leaves with a route and a reason, kept or dropped.

Attributes

ALLOWLIST_WARN_SHARE = 0.5 module-attribute

NormalizedChunk = PreparedChunk module-attribute

logger = logging.getLogger(__name__) module-attribute

Classes

PrepareOptions dataclass

Options for the chunk preparation pipeline.

Source code in ontocast/tool/chunk/prepare.py
@dataclass
class PrepareOptions:
    """Options for the chunk preparation pipeline."""

    section_schema_id: str | None = None
    document_type_hint: str | None = None
    target_sections: list[str] | None = None
    summarize_sections: list[str] | None = None
    exclude_sections: list[str] | None = None

    def needs_section_prepare(self) -> bool:
        """True when a request option explicitly requires section labels.

        Section tagging itself is default-on (see ``CHUNK_SECTION_CLASSIFIER``);
        this only reports whether the request carries section-dependent options.
        """
        return (
            self.target_sections is not None
            or self.summarize_sections is not None
            or self.exclude_sections is not None
        )

    def filter_allowlist(self) -> list[str] | None:
        """The extraction allowlist: ``target_sections`` only.

        ``summarize_sections`` chooses which units are summarized and never
        what is extracted.
        """
        return self.target_sections

Attributes

document_type_hint = None class-attribute instance-attribute
exclude_sections = None class-attribute instance-attribute
section_schema_id = None class-attribute instance-attribute
summarize_sections = None class-attribute instance-attribute
target_sections = None class-attribute instance-attribute

Methods:

__init__(section_schema_id=None, document_type_hint=None, target_sections=None, summarize_sections=None, exclude_sections=None)
filter_allowlist()

The extraction allowlist: target_sections only.

summarize_sections chooses which units are summarized and never what is extracted.

Source code in ontocast/tool/chunk/prepare.py
def filter_allowlist(self) -> list[str] | None:
    """The extraction allowlist: ``target_sections`` only.

    ``summarize_sections`` chooses which units are summarized and never
    what is extracted.
    """
    return self.target_sections
needs_section_prepare()

True when a request option explicitly requires section labels.

Section tagging itself is default-on (see CHUNK_SECTION_CLASSIFIER); this only reports whether the request carries section-dependent options.

Source code in ontocast/tool/chunk/prepare.py
def needs_section_prepare(self) -> bool:
    """True when a request option explicitly requires section labels.

    Section tagging itself is default-on (see ``CHUNK_SECTION_CLASSIFIER``);
    this only reports whether the request carries section-dependent options.
    """
    return (
        self.target_sections is not None
        or self.summarize_sections is not None
        or self.exclude_sections is not None
    )

PreparedChunk dataclass

A prepared text chunk with optional structural metadata, label and route.

section_label_source and section_label_strength record which tier decided the label and whether it may take part in a drop; section_label_evidence holds every tier's vote. route and route_reason record what routing decided and why.

Source code in ontocast/tool/chunk/prepare.py
@dataclass(frozen=True)
class PreparedChunk:
    """A prepared text chunk with optional structural metadata, label and route.

    ``section_label_source`` and ``section_label_strength`` record which tier
    decided the label and whether it may take part in a drop;
    ``section_label_evidence`` holds every tier's vote. ``route`` and
    ``route_reason`` record what routing decided and why.
    """

    text: str
    headings: list[str] | None
    #: Docling items whose blocks the unit holds whole, set after trimming.
    doc_item_refs: tuple[str, ...] = ()
    section_label: str | None = None
    section_label_source: SectionLabelSource | None = None
    section_label_strength: LabelStrength | None = None
    section_label_evidence: tuple[LabelEvidence, ...] = ()
    route: UnitRoute = UnitRoute.EXTRACT
    route_reason: RouteReason = RouteReason.CONTENT

Attributes

doc_item_refs = () class-attribute instance-attribute
headings instance-attribute
route = UnitRoute.EXTRACT class-attribute instance-attribute
route_reason = RouteReason.CONTENT class-attribute instance-attribute
section_label = None class-attribute instance-attribute
section_label_evidence = () class-attribute instance-attribute
section_label_source = None class-attribute instance-attribute
section_label_strength = None class-attribute instance-attribute
text instance-attribute

Methods:

__init__(text, headings, doc_item_refs=(), section_label=None, section_label_source=None, section_label_strength=None, section_label_evidence=(), route=UnitRoute.EXTRACT, route_reason=RouteReason.CONTENT)

PreparedDocument dataclass

A document's prepared units, each with its route and reason, in order.

Attributes:

Name Type Description
routed list[PreparedChunk]

Every prepared unit in document order, kept or dropped.

schema_id str | None

The section schema the document was labelled against; None when section classification is off.

schema_source str | None

How the schema was chosen: explicit, hint, detected or default (a fallback, which applies no default exclusions); None when classification is off.

allowlist_excluded_chars int

Characters target_sections removed.

allowlist_excluded_unlabeled_chars int

The part of those that carried no label at all -- text a missing label, not the caller, kept out.

outline_labels OutlineLabelReport | None

What the model tier did; None when it did not run.

Source code in ontocast/tool/chunk/prepare.py
@dataclass(frozen=True)
class PreparedDocument:
    """A document's prepared units, each with its route and reason, in order.

    Attributes:
        routed: Every prepared unit in document order, kept or dropped.
        schema_id: The section schema the document was labelled against;
            ``None`` when section classification is off.
        schema_source: How the schema was chosen: ``explicit``, ``hint``,
            ``detected`` or ``default`` (a fallback, which applies no default
            exclusions); ``None`` when classification is off.
        allowlist_excluded_chars: Characters ``target_sections`` removed.
        allowlist_excluded_unlabeled_chars: The part of those that carried no
            label at all -- text a missing label, not the caller, kept out.
        outline_labels: What the model tier did; ``None`` when it did not run.
    """

    routed: list[PreparedChunk]
    schema_id: str | None = None
    schema_source: str | None = None
    allowlist_excluded_chars: int = 0
    allowlist_excluded_unlabeled_chars: int = 0
    outline_labels: OutlineLabelReport | None = None

    @property
    def units(self) -> list[PreparedChunk]:
        """Units kept for extraction (routed extract, citations or non-content)."""
        return [unit for unit in self.routed if unit.route is not UnitRoute.SKIP]

    @property
    def dropped(self) -> list[PreparedChunk]:
        """Units routed to skip, each with its reason."""
        return [unit for unit in self.routed if unit.route is UnitRoute.SKIP]

Attributes

allowlist_excluded_chars = 0 class-attribute instance-attribute
allowlist_excluded_unlabeled_chars = 0 class-attribute instance-attribute
dropped property

Units routed to skip, each with its reason.

outline_labels = None class-attribute instance-attribute
routed instance-attribute
schema_id = None class-attribute instance-attribute
schema_source = None class-attribute instance-attribute
units property

Units kept for extraction (routed extract, citations or non-content).

Methods:

__init__(routed, schema_id=None, schema_source=None, allowlist_excluded_chars=0, allowlist_excluded_unlabeled_chars=0, outline_labels=None)

SchemaDecision dataclass

Which label schema a document is prepared against, and why.

Recorded rather than recomputed: schema selection now depends on document text, and it used to be resolved independently in three places. If those disagreed, the deterministic tiers would tag against one schema while the LLM backfill validated against another, and normalise_llm_label drops labels absent from its schema -- silent label loss, not an error.

Source code in ontocast/tool/chunk/prepare.py
@dataclass(frozen=True)
class SchemaDecision:
    """Which label schema a document is prepared against, and why.

    Recorded rather than recomputed: schema selection now depends on document
    text, and it used to be resolved independently in three places. If those
    disagreed, the deterministic tiers would tag against one schema while the
    LLM backfill validated against another, and ``normalise_llm_label`` drops
    labels absent from its schema -- silent label loss, not an error.
    """

    schema: SectionLabelSchema
    source: str
    detection: SchemaDetection | None = None

    @property
    def schema_id(self) -> str:
        return self.schema.id

Attributes

detection = None class-attribute instance-attribute
schema instance-attribute
schema_id property
source instance-attribute

Methods:

__init__(schema, source, detection=None)

SectionSelectionEmptyError

Bases: ValueError

A section selection matched no segment in this document.

Distinct from a malformed parameter: target_sections=["reslts"] is syntactically fine and only turns out to be wrong once this document has been classified. Under the default CHUNK_SECTION_FILTER_ON_EMPTY=warn this is a log line and the run continues to an empty graph, which reads exactly like a document that genuinely had nothing to extract -- telling those two apart is the whole point of the error mode.

Subclasses :class:ValueError so existing parameter guards keep their shape. Deliberately not an api-layer error: nothing under tool/ imports ontocast.api, and the parameter here is well-formed.

Source code in ontocast/tool/chunk/prepare.py
class SectionSelectionEmptyError(ValueError):
    """A section selection matched no segment in this document.

    Distinct from a malformed parameter: ``target_sections=["reslts"]`` is
    syntactically fine and only turns out to be wrong once *this* document has
    been classified. Under the default ``CHUNK_SECTION_FILTER_ON_EMPTY=warn``
    this is a log line and the run continues to an empty graph, which reads
    exactly like a document that genuinely had nothing to extract -- telling
    those two apart is the whole point of the ``error`` mode.

    Subclasses :class:`ValueError` so existing parameter guards keep their
    shape. Deliberately *not* an ``api``-layer error: nothing under ``tool/``
    imports ``ontocast.api``, and the parameter here is well-formed.
    """

    def __init__(self, param: str, message: str) -> None:
        super().__init__(message)
        self.param = param

Attributes

param = param instance-attribute

Methods:

__init__(param, message)
Source code in ontocast/tool/chunk/prepare.py
def __init__(self, param: str, message: str) -> None:
    super().__init__(message)
    self.param = param

Functions:

heading_embedder(splitter, config)

The embedder for the embedding heading tier, or None when it is off.

The tier runs at the heuristic and llm levels; it is the chunker's own model, so it is absent wherever semantic chunking is.

Source code in ontocast/tool/chunk/prepare.py
def heading_embedder(splitter: ChunkerTool, config: ChunkConfig) -> TextEmbedder | None:
    """The embedder for the embedding heading tier, or ``None`` when it is off.

    The tier runs at the ``heuristic`` and ``llm`` levels; it is the
    chunker's own model, so it is absent wherever semantic chunking is.
    """
    if config.section_classifier not in ("heuristic", "llm"):
        return None
    return splitter.embed_texts

prepare_content_units(docling_doc, splitter, config, options, tools=None) async

Segment, tag, select, size, trim and route document text into units.

Section tagging is default-on: the sections-first flow runs unless CHUNK_SECTION_CLASSIFIER=off (which leaves only the content rules of routing; explicit section options are ignored with a warning in that case).

Parameters:

Name Type Description Default
docling_doc DoclingDocument

Converted source document.

required
splitter ChunkerTool

Chunker used to size oversized sections.

required
config ChunkConfig

Chunk configuration, including the classifier tier.

required
options PrepareOptions

Per-request section schema and filters.

required
tools 'ToolBox | None'

ToolBox providing the LLM. Required only when config.section_classifier == "llm"; the deterministic tiers need no LLM, so callers that only inspect sections may omit it.

None

Returns:

Type Description
PreparedDocument

The kept and dropped units, each with its route and reason, and how the

PreparedDocument

schema was chosen.

Raises:

Type Description
ValueError

section_classifier is "llm" but no ToolBox was supplied.

SectionSelectionEmptyError

target_sections or the exclusions removed every unit and CHUNK_SECTION_FILTER_ON_EMPTY=error.

Source code in ontocast/tool/chunk/prepare.py
async def prepare_content_units(
    docling_doc: DoclingDocument,
    splitter: ChunkerTool,
    config: ChunkConfig,
    options: PrepareOptions,
    tools: "ToolBox | None" = None,
) -> PreparedDocument:
    """Segment, tag, select, size, trim and route document text into units.

    Section tagging is default-on: the sections-first flow runs unless
    ``CHUNK_SECTION_CLASSIFIER=off`` (which leaves only the content rules of
    routing; explicit section options are ignored with a warning in that
    case).

    Args:
        docling_doc: Converted source document.
        splitter: Chunker used to size oversized sections.
        config: Chunk configuration, including the classifier tier.
        options: Per-request section schema and filters.
        tools: ToolBox providing the LLM. Required only when
            ``config.section_classifier == "llm"``; the deterministic tiers
            need no LLM, so callers that only inspect sections may omit it.

    Returns:
        The kept and dropped units, each with its route and reason, and how the
        schema was chosen.

    Raises:
        ValueError: ``section_classifier`` is ``"llm"`` but no ToolBox was
            supplied.
        SectionSelectionEmptyError: ``target_sections`` or the exclusions
            removed every unit and ``CHUNK_SECTION_FILTER_ON_EMPTY=error``.
    """
    if config.section_classifier == "llm" and tools is None:
        raise ValueError(
            "CHUNK_SECTION_CLASSIFIER=llm requires a ToolBox providing an LLM"
        )
    # Segmentation is CPU-bound and runs local embedding models, but this
    # coroutine is awaited on the event loop -- inline, it would freeze every
    # concurrent document's in-flight provider sockets for its whole duration.
    structure = await asyncio.to_thread(build_document_structure, docling_doc)
    document_text = structure.text
    block_kinds = structure.block_kinds

    if config.section_classifier == "off":
        if options.needs_section_prepare():
            logger.warning(
                "Section options requested but CHUNK_SECTION_CLASSIFIER=off; "
                "section filters are ignored"
            )
        chunks = await asyncio.to_thread(
            _simple_prepare, document_text, splitter, config
        )
        routed = _route_chunks(
            _trim_citation_runs(chunks, None, block_kinds),
            schema=None,
            schema_fallback=True,
            exclude_sections=None,
            config=config,
            block_kinds=block_kinds,
        )
        return PreparedDocument(routed=routed)

    decision = await asyncio.to_thread(
        resolve_prepare_schema, document_text, config, options, splitter
    )
    schema = decision.schema
    spans = await asyncio.to_thread(
        detect_section_spans,
        document_text,
        schema,
        include_text_headings=config.section_text_headings,
        embed=heading_embedder(splitter, config),
    )

    segments = await asyncio.to_thread(
        _semantic_section_segments, document_text, spans, splitter, config.max_size
    )
    if not segments:
        return PreparedDocument(
            routed=[], schema_id=schema.id, schema_source=decision.source
        )

    segments = await asyncio.to_thread(
        coalesce_small_segments_right,
        segments,
        config.section_tag_min_chars,
        schema,
    )
    if config.section_classifier in ("heuristic", "llm"):
        await asyncio.to_thread(
            _add_content_evidence, segments, schema, config, block_kinds
        )
    outline_labels: OutlineLabelReport | None = None
    if config.section_classifier == "llm" and tools is not None:
        outline_labels = await llm_label_outline(
            segments,
            tools,
            # The resolved schema is threaded, not re-derived: re-resolving from
            # the raw request would ignore a text-based detection and validate
            # LLM labels against a different schema, silently dropping them.
            schema,
            document_type_hint=options.document_type_hint,
            window=config.section_llm_batch_size,
        )
    segments = _attach_bare_headings(segments)

    unlabeled = sum(1 for s in segments if s.section_label is None)
    if unlabeled:
        # Normal for text with no headings (plain-text and JSON inputs); the
        # manifest's section_label_histogram is the record.
        logger.info(
            "%s segment(s) remain without section_label after classification",
            unlabeled,
        )

    on_empty = config.section_filter_on_empty

    excluded_chars = 0
    excluded_unlabeled_chars = 0
    allowlist = options.filter_allowlist()
    if allowlist is not None:
        before = len(segments)
        kept = _filter_segments(segments, allowlist)
        excluded_chars, excluded_unlabeled_chars = _allowlist_report(
            segments, kept, allowlist
        )
        segments = kept
        _guard_empty_selection(
            param="target_sections",
            selection=allowlist,
            before=before,
            after=len(segments),
            on_empty=on_empty,
        )

    chunks = await asyncio.to_thread(_size_segments, segments, splitter, config)
    chunks = _trim_citation_runs(chunks, schema, block_kinds)
    routed = _route_chunks(
        chunks,
        schema=schema,
        schema_fallback=decision.source == "default",
        exclude_sections=options.exclude_sections,
        config=config,
        block_kinds=block_kinds,
    )
    document = PreparedDocument(
        routed=routed,
        schema_id=schema.id,
        schema_source=decision.source,
        allowlist_excluded_chars=excluded_chars,
        allowlist_excluded_unlabeled_chars=excluded_unlabeled_chars,
        outline_labels=outline_labels,
    )
    _guard_excluded_everything(document, options, schema, on_empty)
    return document

resolve_prepare_schema(document_text, config, options, splitter=None)

Choose the section-label schema for one document.

Precedence: an explicit section_schema_id, then a document_type_hint that maps to a schema, then automatic detection, then the manifest default. Caller-supplied intent is never overridden -- detection only fills the gap where the request said nothing.

Parameters:

Name Type Description Default
document_text str

Markdown export used for heading detection.

required
config ChunkConfig

Chunk configuration, including the detection tier.

required
options PrepareOptions

Per-request schema id and document type hint.

required
splitter ChunkerTool | None

Chunker, used only to reach the embedding model already loaded for semantic chunking. None restricts detection to the lexical tier.

None

Returns:

Type Description
SchemaDecision

The chosen schema and how it was chosen.

Source code in ontocast/tool/chunk/prepare.py
def resolve_prepare_schema(
    document_text: str,
    config: ChunkConfig,
    options: PrepareOptions,
    splitter: ChunkerTool | None = None,
) -> SchemaDecision:
    """Choose the section-label schema for one document.

    Precedence: an explicit ``section_schema_id``, then a ``document_type_hint``
    that maps to a schema, then automatic detection, then the manifest default.
    Caller-supplied intent is never overridden -- detection only fills the gap
    where the request said nothing.

    Args:
        document_text: Markdown export used for heading detection.
        config: Chunk configuration, including the detection tier.
        options: Per-request schema id and document type hint.
        splitter: Chunker, used only to reach the embedding model already loaded
            for semantic chunking. ``None`` restricts detection to the lexical
            tier.

    Returns:
        The chosen schema and how it was chosen.
    """
    if options.section_schema_id and options.section_schema_id.strip():
        schema_id = resolve_section_schema_id(
            section_schema_id=options.section_schema_id
        )
        return SchemaDecision(load_section_label_schema(schema_id), "explicit")

    from_hint = schema_id_from_hint(options.document_type_hint)
    if from_hint is not None:
        return SchemaDecision(load_section_label_schema(from_hint), "hint")

    detection = _detect_schema(document_text, config, splitter)
    if detection is not None:
        logger.info(
            "Detected document schema %r via %s tier (score %.1f, margin %.1fx): %s",
            detection.schema_id,
            detection.tier,
            detection.score,
            detection.margin,
            ", ".join(detection.evidence[0].examples[:3]),
        )
        return SchemaDecision(
            load_section_label_schema(detection.schema_id), "detected", detection
        )

    default_id = resolve_section_schema_id()
    logger.info("No schema evidence; falling back to default schema %r", default_id)
    return SchemaDecision(load_section_label_schema(default_id), "default")