ontocast.tool.chunk.prepare¶
Prepare content units: segment, label, select, size, trim and route.
Labelling and routing are separate. Every labelling tier adds evidence to a
segment and one rule decides its label (:mod:ontocast.tool.chunk.assignment);
the caller's target_sections selects among the labels; sizing builds units;
citation runs at a unit's edge are trimmed into units of their own; and
:func:ontocast.tool.chunk.routing.route_unit decides each unit's route from
its label, the label's strength and its content. Every unit leaves with a
route and a reason, kept or dropped.
Attributes¶
ALLOWLIST_WARN_SHARE = 0.5
module-attribute
¶
NormalizedChunk = PreparedChunk
module-attribute
¶
logger = logging.getLogger(__name__)
module-attribute
¶
Classes¶
PrepareOptions
dataclass
¶
Options for the chunk preparation pipeline.
Source code in ontocast/tool/chunk/prepare.py
Attributes¶
document_type_hint = None
class-attribute
instance-attribute
¶
exclude_sections = None
class-attribute
instance-attribute
¶
section_schema_id = None
class-attribute
instance-attribute
¶
summarize_sections = None
class-attribute
instance-attribute
¶
target_sections = None
class-attribute
instance-attribute
¶
Methods:¶
__init__(section_schema_id=None, document_type_hint=None, target_sections=None, summarize_sections=None, exclude_sections=None)
¶
filter_allowlist()
¶
The extraction allowlist: target_sections only.
summarize_sections chooses which units are summarized and never
what is extracted.
needs_section_prepare()
¶
True when a request option explicitly requires section labels.
Section tagging itself is default-on (see CHUNK_SECTION_CLASSIFIER);
this only reports whether the request carries section-dependent options.
Source code in ontocast/tool/chunk/prepare.py
PreparedChunk
dataclass
¶
A prepared text chunk with optional structural metadata, label and route.
section_label_source and section_label_strength record which tier
decided the label and whether it may take part in a drop;
section_label_evidence holds every tier's vote. route and
route_reason record what routing decided and why.
Source code in ontocast/tool/chunk/prepare.py
Attributes¶
doc_item_refs = ()
class-attribute
instance-attribute
¶
headings
instance-attribute
¶
route = UnitRoute.EXTRACT
class-attribute
instance-attribute
¶
route_reason = RouteReason.CONTENT
class-attribute
instance-attribute
¶
section_label = None
class-attribute
instance-attribute
¶
section_label_evidence = ()
class-attribute
instance-attribute
¶
section_label_source = None
class-attribute
instance-attribute
¶
section_label_strength = None
class-attribute
instance-attribute
¶
text
instance-attribute
¶
Methods:¶
__init__(text, headings, doc_item_refs=(), section_label=None, section_label_source=None, section_label_strength=None, section_label_evidence=(), route=UnitRoute.EXTRACT, route_reason=RouteReason.CONTENT)
¶
PreparedDocument
dataclass
¶
A document's prepared units, each with its route and reason, in order.
Attributes:
| Name | Type | Description |
|---|---|---|
routed |
list[PreparedChunk]
|
Every prepared unit in document order, kept or dropped. |
schema_id |
str | None
|
The section schema the document was labelled against;
|
schema_source |
str | None
|
How the schema was chosen: |
allowlist_excluded_chars |
int
|
Characters |
allowlist_excluded_unlabeled_chars |
int
|
The part of those that carried no label at all -- text a missing label, not the caller, kept out. |
outline_labels |
OutlineLabelReport | None
|
What the model tier did; |
Source code in ontocast/tool/chunk/prepare.py
Attributes¶
allowlist_excluded_chars = 0
class-attribute
instance-attribute
¶
allowlist_excluded_unlabeled_chars = 0
class-attribute
instance-attribute
¶
dropped
property
¶
Units routed to skip, each with its reason.
outline_labels = None
class-attribute
instance-attribute
¶
routed
instance-attribute
¶
schema_id = None
class-attribute
instance-attribute
¶
schema_source = None
class-attribute
instance-attribute
¶
units
property
¶
Units kept for extraction (routed extract, citations or non-content).
Methods:¶
__init__(routed, schema_id=None, schema_source=None, allowlist_excluded_chars=0, allowlist_excluded_unlabeled_chars=0, outline_labels=None)
¶
SchemaDecision
dataclass
¶
Which label schema a document is prepared against, and why.
Recorded rather than recomputed: schema selection now depends on document
text, and it used to be resolved independently in three places. If those
disagreed, the deterministic tiers would tag against one schema while the
LLM backfill validated against another, and normalise_llm_label drops
labels absent from its schema -- silent label loss, not an error.
Source code in ontocast/tool/chunk/prepare.py
SectionSelectionEmptyError
¶
Bases: ValueError
A section selection matched no segment in this document.
Distinct from a malformed parameter: target_sections=["reslts"] is
syntactically fine and only turns out to be wrong once this document has
been classified. Under the default CHUNK_SECTION_FILTER_ON_EMPTY=warn
this is a log line and the run continues to an empty graph, which reads
exactly like a document that genuinely had nothing to extract -- telling
those two apart is the whole point of the error mode.
Subclasses :class:ValueError so existing parameter guards keep their
shape. Deliberately not an api-layer error: nothing under tool/
imports ontocast.api, and the parameter here is well-formed.
Source code in ontocast/tool/chunk/prepare.py
Functions:¶
heading_embedder(splitter, config)
¶
The embedder for the embedding heading tier, or None when it is off.
The tier runs at the heuristic and llm levels; it is the
chunker's own model, so it is absent wherever semantic chunking is.
Source code in ontocast/tool/chunk/prepare.py
prepare_content_units(docling_doc, splitter, config, options, tools=None)
async
¶
Segment, tag, select, size, trim and route document text into units.
Section tagging is default-on: the sections-first flow runs unless
CHUNK_SECTION_CLASSIFIER=off (which leaves only the content rules of
routing; explicit section options are ignored with a warning in that
case).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
docling_doc
|
DoclingDocument
|
Converted source document. |
required |
splitter
|
ChunkerTool
|
Chunker used to size oversized sections. |
required |
config
|
ChunkConfig
|
Chunk configuration, including the classifier tier. |
required |
options
|
PrepareOptions
|
Per-request section schema and filters. |
required |
tools
|
'ToolBox | None'
|
ToolBox providing the LLM. Required only when
|
None
|
Returns:
| Type | Description |
|---|---|
PreparedDocument
|
The kept and dropped units, each with its route and reason, and how the |
PreparedDocument
|
schema was chosen. |
Raises:
| Type | Description |
|---|---|
ValueError
|
|
SectionSelectionEmptyError
|
|
Source code in ontocast/tool/chunk/prepare.py
859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 | |
resolve_prepare_schema(document_text, config, options, splitter=None)
¶
Choose the section-label schema for one document.
Precedence: an explicit section_schema_id, then a document_type_hint
that maps to a schema, then automatic detection, then the manifest default.
Caller-supplied intent is never overridden -- detection only fills the gap
where the request said nothing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
document_text
|
str
|
Markdown export used for heading detection. |
required |
config
|
ChunkConfig
|
Chunk configuration, including the detection tier. |
required |
options
|
PrepareOptions
|
Per-request schema id and document type hint. |
required |
splitter
|
ChunkerTool | None
|
Chunker, used only to reach the embedding model already loaded
for semantic chunking. |
None
|
Returns:
| Type | Description |
|---|---|
SchemaDecision
|
The chosen schema and how it was chosen. |