Skip to content

ontocast.tool.chunk.bibliography

Reference-list detection at block level, and trimming of citation runs.

A unit is split into blocks (:func:ontocast.tool.chunk.structure.split_blocks) and each text or footnote block is tested as one cited work, or as part of one when an entry runs over several blocks; captions, formulas, code, pictures and table rows never are. Units are judged by the share of their blocks that are citation entries, not by counting citation tokens over the whole unit, so a table that cites its sources or a discussion that enumerates prior studies is not a reference list.

A reference list does not sit inside prose, so only a run of entries at a unit's start or end is trimmed into a unit of its own; a run in the middle is left alone.

Subject-domain-agnostic but not language- or script-agnostic: markers, et al., venue abbreviations and the author shapes are Western conventions, and author-year styles that put the year last rely on the venue cue alone. The failure mode is a miss -- a reference list extracted as content -- never a dropped section.

Attributes

CONTENT_ONLY_SHARE = 0.8 module-attribute

LABEL_AGREEMENT_SHARE = 0.3 module-attribute

MIN_LIST_ENTRIES = 3 module-attribute

MIN_TRIM_ENTRIES = 5 module-attribute

__all__ = ['CONTENT_ONLY_SHARE', 'LABEL_AGREEMENT_SHARE', 'MIN_LIST_ENTRIES', 'MIN_TRIM_ENTRIES', 'citation_counts', 'citation_share', 'is_citation_entry', 'looks_like_bibliography', 'trim_citation_runs'] module-attribute

Classes

Functions:

citation_counts(text, kinds=None)

(entries, counted_blocks): citation entries among the unit's text blocks.

Headings, table rows and pictures are not counted, so a table never counts; captions, formulas and code count but are never entries.

Parameters:

Name Type Description Default
text str

Unit text.

required
kinds BlockKinds | None

Docling block kinds of the document, when it was converted.

None
Source code in ontocast/tool/chunk/bibliography.py
def citation_counts(text: str, kinds: BlockKinds | None = None) -> tuple[int, int]:
    """``(entries, counted_blocks)``: citation entries among the unit's text blocks.

    Headings, table rows and pictures are not counted, so a table never counts;
    captions, formulas and code count but are never entries.

    Args:
        text: Unit text.
        kinds: Docling block kinds of the document, when it was converted.
    """
    blocks = split_blocks(text, kinds)
    flags = _entry_flags(blocks)
    counted = sum(1 for block in blocks if block.kind in _COUNTED_KINDS)
    return sum(flags), counted

citation_share(text, kinds=None)

Citation entries divided by counted blocks, in [0, 1]; 0 with none.

Source code in ontocast/tool/chunk/bibliography.py
def citation_share(text: str, kinds: BlockKinds | None = None) -> float:
    """Citation entries divided by counted blocks, in ``[0, 1]``; 0 with none."""
    entries, counted = citation_counts(text, kinds)
    return entries / counted if counted else 0.0

is_citation_entry(text)

Whether one block reads as a single cited work.

It opens with a citation marker or an author shape, carries a publication year and a venue cue (DOI, venue token or phrase, volume and pages, or an abbreviated venue), has the length of one entry, and states no measurement outside its links and DOIs. The last test separates "Smith et al. (2019) stored films at 85 °C ..." from a reference.

Parameters:

Name Type Description Default
text str

One block.

required

Returns:

Type Description
bool

True when the block is a citation entry.

Source code in ontocast/tool/chunk/bibliography.py
def is_citation_entry(text: str) -> bool:
    """Whether one block reads as a single cited work.

    It opens with a citation marker or an author shape, carries a publication
    year and a venue cue (DOI, venue token or phrase, volume and pages, or an
    abbreviated venue), has the length of one entry, and states no measurement
    outside its links and DOIs. The last test separates "Smith et al. (2019)
    stored films at 85 °C ..." from a reference.

    Args:
        text: One block.

    Returns:
        True when the block is a citation entry.
    """
    body = _body(text)
    if not _MIN_ENTRY_CHARS <= len(body) <= _MAX_ENTRY_CHARS:
        return False
    if not _opens_entry(body) or not _YEAR.search(body):
        return False
    has_venue = bool(
        _DOI.search(body)
        or _VENUE_TOKEN.search(body)
        or _VENUE_PHRASE.search(body)
        or _VOLUME_PAGES.search(body)
        or _YEAR_VOLUME.search(body)
        or _ABBREVIATED_VENUE.search(body)
    )
    return has_venue and not states_measurement(_ADDRESS.sub(" ", body))

looks_like_bibliography(text, kinds=None)

Whether the content alone makes a unit a reference list.

Parameters:

Name Type Description Default
text str

Unit text.

required
kinds BlockKinds | None

Docling block kinds of the document, when it was converted.

None

Returns:

Type Description
bool

True when at least :data:MIN_LIST_ENTRIES entries make up at least

bool
Source code in ontocast/tool/chunk/bibliography.py
def looks_like_bibliography(text: str, kinds: BlockKinds | None = None) -> bool:
    """Whether the content alone makes a unit a reference list.

    Args:
        text: Unit text.
        kinds: Docling block kinds of the document, when it was converted.

    Returns:
        True when at least :data:`MIN_LIST_ENTRIES` entries make up at least
        :data:`CONTENT_ONLY_SHARE` of the counted blocks.
    """
    entries, counted = citation_counts(text, kinds)
    return (
        entries >= MIN_LIST_ENTRIES
        and counted > 0
        and entries / counted >= CONTENT_ONLY_SHARE
    )

trim_citation_runs(text, kinds=None)

Cut citation runs at a unit's start or end into pieces of their own.

Parameters:

Name Type Description Default
text str

Unit text.

required
kinds BlockKinds | None

Docling block kinds of the document, when it was converted.

None

Returns:

Type Description
list[tuple[str, bool]]

(piece, is_citation_run) pairs in document order. A unit with

list[tuple[str, bool]]

no run at either edge comes back whole as [(text, False)]; a unit

list[tuple[str, bool]]

that is all one run comes back as [(text, True)].

Source code in ontocast/tool/chunk/bibliography.py
def trim_citation_runs(
    text: str, kinds: BlockKinds | None = None
) -> list[tuple[str, bool]]:
    """Cut citation runs at a unit's start or end into pieces of their own.

    Args:
        text: Unit text.
        kinds: Docling block kinds of the document, when it was converted.

    Returns:
        ``(piece, is_citation_run)`` pairs in document order. A unit with
        no run at either edge comes back whole as ``[(text, False)]``; a unit
        that is all one run comes back as ``[(text, True)]``.
    """
    blocks = split_blocks(text, kinds)
    if not blocks:
        return [(text, False)]
    flags = _entry_flags(blocks)
    leading, trailing = _edge_runs(blocks, flags)
    if leading + trailing >= len(blocks) or leading == len(blocks):
        return [(text, True)] if (leading or trailing) else [(text, False)]
    if not leading and not trailing:
        return [(text, False)]

    pieces: list[tuple[str, bool]] = []
    body_start = blocks[leading].start if leading else 0
    body_end = blocks[len(blocks) - trailing - 1].end if trailing else len(text)
    remainder = PLACEHOLDER.sub("", text[body_start:body_end])
    if len(" ".join(remainder.split())) < _MIN_ENTRY_CHARS:
        # Nothing but placeholders or a stray line beside the run: not prose.
        return [(text, True)]
    if leading:
        pieces.append((text[: blocks[leading - 1].end].strip(), True))
    pieces.append((text[body_start:body_end].strip(), False))
    if trailing:
        pieces.append((text[blocks[len(blocks) - trailing].start :].strip(), True))
    return [(piece, is_run) for piece, is_run in pieces if piece]