Skip to content

ontocast.tool.chunk.structure

Document structure: the blocks of the markdown export and their kinds.

The markdown export is the document text everything else reads, so blocks are cut from it by markdown syntax: blank lines separate paragraphs, list bullets and citation markers start blocks, and heading lines and table rows are blocks of their own. A converted document adds what syntax cannot show: each docling item is serialized as the export serializes it, and a block whose text equals an item's text takes that item's kind and reference. So a caption, a footnote or a formula is known for what it is, and a unit can name the items it holds.

A block no item matches -- plain-text input, a paragraph a unit boundary cut through, text the converter joined differently -- keeps its syntax kind.

Attributes

BlockKinds = Mapping[str, ItemBlock] module-attribute

ENTRY_MARKER = re.compile('^\\s*(?:\\[\\d{1,4}\\]|\\(\\d{1,4}\\)|\\d{1,4}[.)])\\s+\\S') module-attribute

LIST_BULLET = re.compile('^\\s*[-*•·]\\s+') module-attribute

PLACEHOLDER = re.compile('<!--.*?-->', re.DOTALL) module-attribute

__all__ = ['ENTRY_MARKER', 'LIST_BULLET', 'PLACEHOLDER', 'Block', 'BlockKind', 'BlockKinds', 'DocumentStructure', 'ItemBlock', 'build_document_structure', 'docling_block_kinds', 'item_refs', 'split_blocks'] module-attribute

Classes

Block dataclass

One block of a text, with offsets into that text.

Source code in ontocast/tool/chunk/structure.py
@dataclass(frozen=True)
class Block:
    """One block of a text, with offsets into that text."""

    kind: BlockKind
    start: int
    end: int
    text: str
    ref: str | None = None

Attributes

end instance-attribute
kind instance-attribute
ref = None class-attribute instance-attribute
start instance-attribute
text instance-attribute

Methods:

__init__(kind, start, end, text, ref=None)

BlockKind

Bases: StrEnum

What a block of the export is.

Source code in ontocast/tool/chunk/structure.py
class BlockKind(StrEnum):
    """What a block of the export is."""

    TEXT = "text"
    HEADING = "heading"
    TABLE = "table"
    CAPTION = "caption"
    FOOTNOTE = "footnote"
    FORMULA = "formula"
    CODE = "code"
    PICTURE = "picture"

Attributes

CAPTION = 'caption' class-attribute instance-attribute
CODE = 'code' class-attribute instance-attribute
FOOTNOTE = 'footnote' class-attribute instance-attribute
FORMULA = 'formula' class-attribute instance-attribute
HEADING = 'heading' class-attribute instance-attribute
PICTURE = 'picture' class-attribute instance-attribute
TABLE = 'table' class-attribute instance-attribute
TEXT = 'text' class-attribute instance-attribute

DocumentStructure dataclass

The document text and the docling kinds of its blocks.

Source code in ontocast/tool/chunk/structure.py
@dataclass(frozen=True)
class DocumentStructure:
    """The document text and the docling kinds of its blocks."""

    text: str
    block_kinds: BlockKinds = field(default_factory=dict)

Attributes

block_kinds = field(default_factory=dict) class-attribute instance-attribute
text instance-attribute

Methods:

__init__(text, block_kinds=dict())

ItemBlock

Bases: NamedTuple

The kind and docling reference of an item, keyed by its block text.

Source code in ontocast/tool/chunk/structure.py
class ItemBlock(NamedTuple):
    """The kind and docling reference of an item, keyed by its block text."""

    kind: BlockKind
    ref: str

Attributes

kind instance-attribute
ref instance-attribute

Functions:

build_document_structure(doc)

The markdown export of doc and the docling kinds of its blocks.

Source code in ontocast/tool/chunk/structure.py
def build_document_structure(doc: DoclingDocument) -> DocumentStructure:
    """The markdown export of ``doc`` and the docling kinds of its blocks."""
    return DocumentStructure(
        text=doc.export_to_markdown(), block_kinds=docling_block_kinds(doc)
    )

docling_block_kinds(doc)

Kind and reference of each body item, keyed by its normalised block text.

Items are serialized one by one with the serializer the markdown export uses, so their paragraphs are the export's blocks. An inline group is one item (its parts serialize differently on their own); a picture or table contributes its captions. The first item wins when two share a text.

Source code in ontocast/tool/chunk/structure.py
def docling_block_kinds(doc: DoclingDocument) -> dict[str, ItemBlock]:
    """Kind and reference of each body item, keyed by its normalised block text.

    Items are serialized one by one with the serializer the markdown export
    uses, so their paragraphs are the export's blocks. An inline group is one
    item (its parts serialize differently on their own); a picture or table
    contributes its captions. The first item wins when two share a text.
    """
    doc_types = require("docling_core.types.doc", feature="Document structure")
    markdown = require(
        "docling_core.transforms.serializer.markdown", feature="Document structure"
    )
    serializer = markdown.MarkdownDocSerializer(doc=doc)
    kinds: dict[str, ItemBlock] = {}

    def register(text: str, kind: BlockKind, ref: str) -> None:
        for paragraph in _PARAGRAPH_BREAK.split(text):
            key = _normalise(paragraph)
            if key:
                kinds.setdefault(key, ItemBlock(kind, ref))

    for item, _depth in doc.iterate_items(with_groups=True):
        if _inside_inline_group(item, doc):
            continue
        if isinstance(item, doc_types.GroupItem):
            if item.label != doc_types.GroupLabel.INLINE:
                continue
            kind = BlockKind.TEXT
        elif isinstance(item, doc_types.FloatingItem) and not isinstance(
            item, doc_types.TextItem
        ):
            for caption in item.captions:
                resolved = caption.resolve(doc)
                register(resolved.text, BlockKind.CAPTION, resolved.self_ref)
            continue
        else:
            kind = _kind_of_label(str(item.label))
        register(serializer.serialize(item=item).text, kind, item.self_ref)
    return kinds

item_refs(text, kinds)

References of the docling items whose blocks text holds whole, in order.

Source code in ontocast/tool/chunk/structure.py
def item_refs(text: str, kinds: BlockKinds | None) -> tuple[str, ...]:
    """References of the docling items whose blocks ``text`` holds whole, in order."""
    refs: dict[str, None] = {}
    for block in split_blocks(text, kinds):
        if block.ref is not None:
            refs.setdefault(block.ref)
    return tuple(refs)

split_blocks(text, kinds=None)

Split a text into blocks, typed by docling items where they match.

Blank lines separate paragraphs. Inside a paragraph, a line that opens with a list bullet or a citation marker starts a new block, and lines without one continue the block before them, so a reference wrapped over two lines stays one entry. A paragraph that opens mid-sentence (lower case, a digit, a bracket, a DOI or URL) right after a block that opened with a marker continues that block too: converters split a wrapped entry that way. Heading lines and table rows are blocks of their own; a paragraph of placeholders only is a picture.

Parameters:

Name Type Description Default
text str

Markdown or plain text.

required
kinds BlockKinds | None

Docling kinds by block text (:func:docling_block_kinds). A matching text or picture block takes the item's kind; headings and table rows keep theirs. Every match records the item reference.

None

Returns:

Type Description
list[Block]

Blocks in order, with offsets into text.

Source code in ontocast/tool/chunk/structure.py
def split_blocks(text: str, kinds: BlockKinds | None = None) -> list[Block]:
    """Split a text into blocks, typed by docling items where they match.

    Blank lines separate paragraphs. Inside a paragraph, a line that opens with
    a list bullet or a citation marker starts a new block, and lines without one
    continue the block before them, so a reference wrapped over two lines stays
    one entry. A paragraph that opens mid-sentence (lower case, a digit, a
    bracket, a DOI or URL) right after a block that opened with a marker
    continues that block too: converters split a wrapped entry that way.
    Heading lines and table rows are blocks of their own; a paragraph of
    placeholders only is a picture.

    Args:
        text: Markdown or plain text.
        kinds: Docling kinds by block text (:func:`docling_block_kinds`). A
            matching text or picture block takes the item's kind; headings and
            table rows keep theirs. Every match records the item reference.

    Returns:
        Blocks in order, with offsets into ``text``.
    """
    blocks = _syntax_blocks(text)
    if not kinds:
        return blocks
    typed: list[Block] = []
    for block in blocks:
        item = kinds.get(_normalise(block.text))
        if item is None:
            typed.append(block)
            continue
        kind = (
            item.kind
            if block.kind in (BlockKind.TEXT, BlockKind.PICTURE)
            else block.kind
        )
        typed.append(Block(kind, block.start, block.end, block.text, item.ref))
    return typed