Skip to content

ontocast.tool.chunk.sections

Document section spans for chunk prepare: outline spans plus front matter.

Attributes

ABSTRACT_FRONT_MATTER_MAX_CHARS = 6000 module-attribute

__all__ = ['ABSTRACT_FRONT_MATTER_MAX_CHARS', 'DocumentOutline', 'HeadingNode', 'SectionSpan', 'detect_section_spans', 'document_text_for_section_tagging', 'inject_front_matter_spans'] module-attribute

Classes

DocumentOutline

Bases: BasePydanticModel

Ordered headings detected in a document, with the document length.

Source code in ontocast/onto/section_models.py
class DocumentOutline(BasePydanticModel):
    """Ordered headings detected in a document, with the document length."""

    text_len: int
    nodes: list[HeadingNode] = Field(default_factory=list)

Attributes

nodes = Field(default_factory=list) class-attribute instance-attribute
text_len instance-attribute

HeadingNode

Bases: BasePydanticModel

One detected heading in the document outline.

Attributes:

Name Type Description
text str

Raw heading line as it appears in the document text.

normalised str

Heading text after decoration/numbering stripping.

start int

Character offset of the heading line itself.

body_start int

Character offset just past the heading line.

level int

Markdown heading depth (1 = top). Docling reports a flat level for PDF conversions, so levels place a heading only in documents whose levels vary.

number str | None

The heading's section number ("2.1", "II", "A.2"), if any.

parent int | None

Index in the outline of the heading this one sits under.

tree_placed bool

Whether numbering or markdown levels place this heading; parent is None then means top level. Unplaced headings are judged by genericity.

sectionlike bool

Whether the heading reads as a generic section name rather than a descriptive subsection title or a document title.

label str | None

Canonical section label, when resolved.

source SectionLabelSource

How label was decided.

strength LabelStrength

Whether label may take part in a drop decision.

Source code in ontocast/onto/section_models.py
class HeadingNode(BasePydanticModel):
    """One detected heading in the document outline.

    Attributes:
        text: Raw heading line as it appears in the document text.
        normalised: Heading text after decoration/numbering stripping.
        start: Character offset of the heading line itself.
        body_start: Character offset just past the heading line.
        level: Markdown heading depth (1 = top). Docling reports a flat level
            for PDF conversions, so levels place a heading only in documents
            whose levels vary.
        number: The heading's section number ("2.1", "II", "A.2"), if any.
        parent: Index in the outline of the heading this one sits under.
        tree_placed: Whether numbering or markdown levels place this heading;
            ``parent is None`` then means top level. Unplaced headings are
            judged by genericity.
        sectionlike: Whether the heading reads as a generic section name rather
            than a descriptive subsection title or a document title.
        label: Canonical section label, when resolved.
        source: How ``label`` was decided.
        strength: Whether ``label`` may take part in a drop decision.
    """

    text: str
    normalised: str
    start: int
    body_start: int
    level: int = 1
    number: str | None = None
    parent: int | None = None
    tree_placed: bool = False
    sectionlike: bool = True
    label: str | None = None
    source: SectionLabelSource = SectionLabelSource.OUTLINE_UNRESOLVED
    strength: LabelStrength = LabelStrength.WEAK

Attributes

body_start instance-attribute
label = None class-attribute instance-attribute
level = 1 class-attribute instance-attribute
normalised instance-attribute
number = None class-attribute instance-attribute
parent = None class-attribute instance-attribute
sectionlike = True class-attribute instance-attribute
source = SectionLabelSource.OUTLINE_UNRESOLVED class-attribute instance-attribute
start instance-attribute
strength = LabelStrength.WEAK class-attribute instance-attribute
text instance-attribute
tree_placed = False class-attribute instance-attribute

SectionSpan

Bases: BasePydanticModel

Character span of a document section with a normalised label.

label is None for a region whose section type is not (yet) known — for example an unrecognised but section-like heading. Such a span is explicitly unresolved rather than absent, which is what stops a neighbouring label from being smeared across it.

Source code in ontocast/onto/section_models.py
class SectionSpan(BasePydanticModel):
    """Character span of a document section with a normalised label.

    ``label`` is ``None`` for a region whose section type is not (yet) known —
    for example an unrecognised but section-like heading. Such a span is
    explicitly unresolved rather than absent, which is what stops a neighbouring
    label from being smeared across it.
    """

    label: str | None = None
    start: int
    end: int
    source: SectionLabelSource = SectionLabelSource.OUTLINE_UNRESOLVED
    strength: LabelStrength = LabelStrength.WEAK
    heading: str | None = Field(
        default=None,
        description="The heading line that opens the span; none for unheaded text.",
    )

Attributes

end instance-attribute
heading = Field(default=None, description='The heading line that opens the span; none for unheaded text.') class-attribute instance-attribute
label = None class-attribute instance-attribute
source = SectionLabelSource.OUTLINE_UNRESOLVED class-attribute instance-attribute
start instance-attribute
strength = LabelStrength.WEAK class-attribute instance-attribute

Functions:

detect_section_spans(text, schema=None, *, include_text_headings=False, embed=None)

Detect document sections and return a partition of the text into spans.

Every heading closes the preceding span, so an unrecognised heading yields an explicitly unresolved (label=None) span instead of letting the previous label run on to the next recognised heading.

Parameters:

Name Type Description Default
text str

Document text (the markdown export).

required
schema SectionLabelSchema | None

Section label schema; the manifest default when omitted.

None
include_text_headings bool

Enable the plain-text heading heuristic for documents with no markdown heading structure.

False
embed TextEmbedder | None

The chunker's embedder, for the embedding heading tier.

None

Returns:

Type Description
list[SectionSpan]

Section spans tiling text, ordered by start offset.

Source code in ontocast/tool/chunk/sections.py
def detect_section_spans(
    text: str,
    schema: SectionLabelSchema | None = None,
    *,
    include_text_headings: bool = False,
    embed: TextEmbedder | None = None,
) -> list[SectionSpan]:
    """Detect document sections and return a partition of the text into spans.

    Every heading closes the preceding span, so an unrecognised heading yields an
    explicitly unresolved (``label=None``) span instead of letting the previous
    label run on to the next recognised heading.

    Args:
        text: Document text (the markdown export).
        schema: Section label schema; the manifest default when omitted.
        include_text_headings: Enable the plain-text heading heuristic for
            documents with no markdown heading structure.
        embed: The chunker's embedder, for the embedding heading tier.

    Returns:
        Section spans tiling ``text``, ordered by start offset.
    """
    if not text:
        return []
    active = schema or get_default_section_schema()
    outline = build_document_outline(
        text, active, include_text_headings=include_text_headings, embed=embed
    )
    spans = outline_to_spans(outline)
    return inject_front_matter_spans(spans, text, active)

document_text_for_section_tagging(doc)

Export document text used for section heading detection.

Source code in ontocast/tool/chunk/sections.py
def document_text_for_section_tagging(doc: DoclingDocument) -> str:
    """Export document text used for section heading detection."""
    return doc.export_to_markdown()

inject_front_matter_spans(spans, text, schema, *, min_gap_chars=80, max_gap_chars=ABSTRACT_FRONT_MATTER_MAX_CHARS)

Label unheaded front matter before the first labeled section as abstract.

Leading unresolved spans (a title block, an unrecognised banner heading) are skipped when locating the first labeled section, so front matter is still recovered on documents whose first recognised section is not an IMRaD opener -- papers that jump straight to Results are common.

Source code in ontocast/tool/chunk/sections.py
def inject_front_matter_spans(
    spans: list[SectionSpan],
    text: str,
    schema: SectionLabelSchema,
    *,
    min_gap_chars: int = 80,
    max_gap_chars: int = ABSTRACT_FRONT_MATTER_MAX_CHARS,
) -> list[SectionSpan]:
    """Label unheaded front matter before the first labeled section as abstract.

    Leading unresolved spans (a title block, an unrecognised banner heading) are
    skipped when locating the first labeled section, so front matter is still
    recovered on documents whose first recognised section is not an IMRaD
    opener -- papers that jump straight to ``Results`` are common.
    """
    if "abstract" not in canonical_labels(schema):
        return spans
    if any(span.label == "abstract" for span in spans):
        return spans
    if not spans:
        return spans

    ordered = sorted(spans, key=lambda span: span.start)
    first_labeled = next(
        (span for span in ordered if span.label is not None),
        None,
    )
    if first_labeled is None or first_labeled.start <= 0:
        return spans

    gap = text[: first_labeled.start].strip()
    gap_len = len(gap)
    if gap_len < min_gap_chars or gap_len > max_gap_chars:
        return spans

    # The front matter may already be covered by unresolved spans; replace them
    # rather than overlapping, so the result stays a partition.
    tail = [span for span in ordered if span.start >= first_labeled.start]
    abstract_span = SectionSpan(
        label="abstract",
        start=0,
        end=first_labeled.start,
        source=SectionLabelSource.FRONT_MATTER,
        strength=LabelStrength.WEAK,
    )
    return [abstract_span, *tail]