Skip to content

ontocast.tool.chunk.routing

Route each prepared unit before extraction: extract, citations, non-content or skip.

A section label is evidence, not a decision. A unit is dropped only when a strong label and the unit's content agree, or when the content alone is conclusive; anything less is extracted. Selecting sections (target_sections) is the caller's intent and happens earlier, on labels of any strength.

Each unit carries its label's strength, set where the label was assigned (:mod:ontocast.tool.chunk.assignment): strong for an anchored heading, a heading inside a strong section by its number or level, or the content itself; weak otherwise. A weak label may help selection, never a drop.

Rules, first match wins:

  1. a unit shorter than CHUNK_MIN_UNIT_CHARS is skipped;
  2. a strong label the request's exclude_sections names is skipped;
  3. unless CHUNK_BIBLIOGRAPHY_MODE=domain_facts, a reference list -- by content alone, or by a strong citations-class label the content agrees with -- takes the bibliography route;
  4. with the schema's default exclusions in force (no exclude_sections, a schema that was chosen rather than fallen back to), a strong boilerplate-class label on text that states no measurement is skipped;
  5. a table of contents or a list of figures or tables, recognised from the content alone, takes the non-content route;
  6. front/back matter -- a strong non-content-class label, or a heading the schema names as front/back matter, on text that states no measurement, or identifier and licence boilerplate recognised from the content alone -- takes the non-content route. A fallback schema lends no vocabulary;
  7. everything else is extracted, with a reason saying why a label that might have dropped it did not.

Attributes

__all__ = ['RouteReason', 'RoutingDecision', 'UnitRoute', 'route_unit'] module-attribute

Classes

RoutableUnit

Bases: Protocol

What routing reads from a unit.

Source code in ontocast/tool/chunk/routing.py
class RoutableUnit(Protocol):
    """What routing reads from a unit."""

    @property
    def text(self) -> str: ...

    @property
    def headings(self) -> list[str] | None: ...

    @property
    def section_label(self) -> str | None: ...

    @property
    def section_label_strength(self) -> LabelStrength | None: ...

Attributes

headings property
section_label property
section_label_strength property
text property

RouteReason

Bases: StrEnum

Why a unit took its route. Values are wire names (run manifest, logs).

Source code in ontocast/onto/enum.py
class RouteReason(StrEnum):
    """Why a unit took its route. Values are wire names (run manifest, logs)."""

    CONTENT = "content"
    #: A label that could have dropped the unit was weak.
    WEAK_LABEL = "weak_label"
    #: A strong citations or boilerplate label the content disagrees with.
    LABEL_CONTRADICTED = "label_contradicted"
    #: A boilerplate label under a schema nobody chose: no default exclusions.
    FALLBACK_SCHEMA = "fallback_schema"
    #: A boilerplate label the request's exclude_sections did not name.
    EXCLUSION_OVERRIDDEN = "exclusion_overridden"
    #: The content alone is a reference list.
    CITATION_LIST = "citation_list"
    #: A strong citations-class label and content that agrees.
    CITATION_LABEL = "citation_label"
    #: A strong boilerplate-class label on text that states no measurement.
    BOILERPLATE = "boilerplate"
    #: Front/back matter recognised from the content.
    NON_CONTENT = "non_content"
    #: The content alone is a table of contents or a list of figures or tables.
    CONTENTS_LIST = "contents_list"
    EXCLUDED_BY_REQUEST = "excluded_by_request"
    UNDERSIZED = "undersized"

Attributes

BOILERPLATE = 'boilerplate' class-attribute instance-attribute
CITATION_LABEL = 'citation_label' class-attribute instance-attribute
CITATION_LIST = 'citation_list' class-attribute instance-attribute
CONTENT = 'content' class-attribute instance-attribute
CONTENTS_LIST = 'contents_list' class-attribute instance-attribute
EXCLUDED_BY_REQUEST = 'excluded_by_request' class-attribute instance-attribute
EXCLUSION_OVERRIDDEN = 'exclusion_overridden' class-attribute instance-attribute
FALLBACK_SCHEMA = 'fallback_schema' class-attribute instance-attribute
LABEL_CONTRADICTED = 'label_contradicted' class-attribute instance-attribute
NON_CONTENT = 'non_content' class-attribute instance-attribute
UNDERSIZED = 'undersized' class-attribute instance-attribute
WEAK_LABEL = 'weak_label' class-attribute instance-attribute

RoutingDecision dataclass

A unit's route and the reason for it.

Source code in ontocast/tool/chunk/routing.py
@dataclass(frozen=True)
class RoutingDecision:
    """A unit's route and the reason for it."""

    route: UnitRoute
    reason: RouteReason

Attributes

reason instance-attribute
route instance-attribute

Methods:

__init__(route, reason)

UnitRoute

Bases: StrEnum

What happens to a unit before extraction. Values are wire names.

Source code in ontocast/onto/enum.py
class UnitRoute(StrEnum):
    """What happens to a unit before extraction. Values are wire names."""

    EXTRACT = "extract"
    #: Kept; the facts renderer extracts bibliographic metadata only.
    CITATIONS = "citations"
    #: Kept and flagged ``is_non_content``; checks that presume domain prose stand down.
    NON_CONTENT = "non_content"
    SKIP = "skip"

Attributes

CITATIONS = 'citations' class-attribute instance-attribute
EXTRACT = 'extract' class-attribute instance-attribute
NON_CONTENT = 'non_content' class-attribute instance-attribute
SKIP = 'skip' class-attribute instance-attribute

Functions:

route_unit(unit, *, schema, schema_fallback, exclude_sections, config, block_kinds=None)

Decide one unit's route from its label, the label's strength and its content.

Parameters:

Name Type Description Default
unit RoutableUnit

A prepared unit.

required
schema SectionLabelSchema | None

The schema the document was labelled against; None when section classification is off, which leaves the content rules only.

required
schema_fallback bool

The schema is the manifest default because nothing chose one; its default exclusions do not apply.

required
exclude_sections list[str] | None

The request's denylist: None keeps the schema's boilerplate exclusions, a list ([] included) replaces them.

required
config ChunkConfig

Chunk configuration (bibliography and non-content modes, the unit-size floor).

required
block_kinds BlockKinds | None

Docling block kinds of the document, when it was converted: a caption or formula is never a citation entry.

None

Returns:

Type Description
RoutingDecision

The route and its reason.

Source code in ontocast/tool/chunk/routing.py
def route_unit(
    unit: RoutableUnit,
    *,
    schema: SectionLabelSchema | None,
    schema_fallback: bool,
    exclude_sections: list[str] | None,
    config: ChunkConfig,
    block_kinds: BlockKinds | None = None,
) -> RoutingDecision:
    """Decide one unit's route from its label, the label's strength and its content.

    Args:
        unit: A prepared unit.
        schema: The schema the document was labelled against; ``None`` when
            section classification is off, which leaves the content rules only.
        schema_fallback: The schema is the manifest default because nothing
            chose one; its default exclusions do not apply.
        exclude_sections: The request's denylist: ``None`` keeps the schema's
            boilerplate exclusions, a list (``[]`` included) replaces them.
        config: Chunk configuration (bibliography and non-content modes, the
            unit-size floor).
        block_kinds: Docling block kinds of the document, when it was
            converted: a caption or formula is never a citation entry.

    Returns:
        The route and its reason.
    """
    text = unit.text
    if config.min_unit_chars and len(text) < config.min_unit_chars:
        return RoutingDecision(UnitRoute.SKIP, RouteReason.UNDERSIZED)

    label = unit.section_label.strip().lower() if unit.section_label else None
    strong = unit.section_label_strength is LabelStrength.STRONG
    route_class = (
        label_route_class(label, schema) if schema is not None else RouteClass.CONTENT
    )

    if exclude_sections is not None and label is not None:
        denied = {section.strip().lower() for section in exclude_sections}
        if label in denied and strong:
            return RoutingDecision(UnitRoute.SKIP, RouteReason.EXCLUDED_BY_REQUEST)

    if config.bibliography_mode != "domain_facts":
        reason: RouteReason | None = None
        if looks_like_bibliography(text, block_kinds):
            reason = RouteReason.CITATION_LIST
        elif (
            route_class is RouteClass.CITATIONS
            and strong
            and citation_share(text, block_kinds) >= LABEL_AGREEMENT_SHARE
        ):
            reason = RouteReason.CITATION_LABEL
        if reason is not None:
            route = (
                UnitRoute.SKIP
                if config.bibliography_mode == "skip"
                else UnitRoute.CITATIONS
            )
            return RoutingDecision(route, reason)

    if route_class is RouteClass.BOILERPLATE and strong:
        if (
            exclude_sections is None
            and not schema_fallback
            and not states_measurement(text)
        ):
            return RoutingDecision(UnitRoute.SKIP, RouteReason.BOILERPLATE)

    if looks_like_contents_list(text):
        route = (
            UnitRoute.SKIP
            if config.non_content_mode == "skip"
            else UnitRoute.NON_CONTENT
        )
        return RoutingDecision(route, RouteReason.CONTENTS_LIST)

    vocabulary = None if schema_fallback else schema
    strong_non_content = (
        route_class is RouteClass.NON_CONTENT
        and strong
        and not schema_fallback
        and not states_measurement(text)
    )
    if strong_non_content or is_non_content_unit(text, unit.headings, vocabulary):
        route = (
            UnitRoute.SKIP
            if config.non_content_mode == "skip"
            else UnitRoute.NON_CONTENT
        )
        return RoutingDecision(route, RouteReason.NON_CONTENT)

    return RoutingDecision(
        UnitRoute.EXTRACT,
        _extract_reason(
            route_class,
            strong=strong,
            schema_fallback=schema_fallback,
            exclusions_replaced=exclude_sections is not None,
            denied=label is not None
            and exclude_sections is not None
            and label in {s.strip().lower() for s in exclude_sections},
        ),
    )