ontocast.tool.chunk¶
Document chunking tools for OntoCast.
Modules:
| Name | Description |
|---|---|
assignment |
Section label assignment: every tier adds evidence, one rule decides. |
bibliography |
Reference-list detection at block level, and trimming of citation runs. |
chunker |
|
contents |
Tables of contents and lists of figures or tables, recognised from their content. |
density |
Content-based section classification for regions with no usable heading. |
embedding |
Embedding helpers shared by the heading tiers: the embedder type and cosine. |
heading_embedding |
Embedding heading tier: a heading no pattern or keyword named votes for its nearest label. |
non_content |
Non-content unit detection: front and back matter that carries no domain facts. |
outline |
Document outline detection for chunk preparation. |
prepare |
Prepare content units: segment, label, select, size, trim and route. |
proposition |
Lightweight text splitting helpers (no ML dependencies). |
routing |
Route each prepared unit before extraction: extract, citations, non-content or skip. |
schema_detect |
Detect which document-type schema a document belongs to. |
section_llm |
LLM section tier: one call labels the document's outline, as weak votes. |
sections |
Document section spans for chunk prepare: outline spans plus front matter. |
segment |
Mutable text segments during chunk preparation (before sizing). |
sizing |
Shared text sizing helpers for OntoCast chunking. |
structure |
Document structure: the blocks of the markdown export and their kinds. |
util |
|
Attributes¶
NormalizedChunk = PreparedChunk
module-attribute
¶
__all__ = ['ChunkerTool', 'NormalizedChunk', 'PreparedChunk', 'PrepareOptions', 'SectionSelectionEmptyError', 'prepare_content_units', 'size_bounded_text']
module-attribute
¶
Classes¶
ChunkerTool
¶
Bases: Tool
Tool for semantic chunking of documents.
Falls back to naive chunking if sentence-transformers is not available. Includes caching to avoid re-chunking the same text with the same parameters.
Source code in ontocast/tool/chunk/chunker.py
56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 | |
Attributes¶
cache = Field(default=None, exclude=True)
class-attribute
instance-attribute
¶
chunking_mode = self.config.chunking_mode
class-attribute
instance-attribute
¶
config = Field(default_factory=ChunkConfig, description='Chunking configuration parameters')
class-attribute
instance-attribute
¶
Methods:¶
__call__(doc)
¶
Chunk a document into semantic segments.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
doc
|
str
|
The document text to chunk. |
required |
Returns:
| Type | Description |
|---|---|
list[str]
|
List of text chunks. |
Source code in ontocast/tool/chunk/chunker.py
__init__(chunk_config=None, cache=None, **kwargs)
¶
Initialize the ChunkerTool.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
chunk_config
|
ChunkConfig | None
|
Chunking configuration. If None, uses default ChunkConfig. |
None
|
cache
|
Cacher | None
|
Optional shared Cacher instance. If None, creates a new one. |
None
|
**kwargs
|
Any
|
Additional keyword arguments passed to the parent class. |
{}
|
Source code in ontocast/tool/chunk/chunker.py
embed_texts(texts)
¶
Embed short texts with the chunker's model, or None if unavailable.
Exposed so document-type detection can reuse the model already loaded
for semantic chunking instead of constructing a second one. Returns
None -- rather than raising -- when the semantic extras are absent,
so callers degrade to their deterministic tiers exactly as chunking
itself degrades to naive.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
texts
|
list[str]
|
Short strings to embed (headings or sampled paragraphs). |
required |
Returns:
| Type | Description |
|---|---|
list[list[float]] | None
|
One embedding per input, or |
Source code in ontocast/tool/chunk/chunker.py
embeddings()
¶
Embeddings over the process-shared encoder, or None if unavailable.
The encoder is shared with retrieval and entity clustering when their model names match, so this loads no weights of its own in that case, and its inference is serialised against theirs.
Source code in ontocast/tool/chunk/chunker.py
naive_split(doc)
¶
Split text by paragraph/sentence boundaries up to max_size.
Unlike :meth:_naive_chunk, does not enforce min_size filtering.
Source code in ontocast/tool/chunk/chunker.py
size_text(doc)
¶
Split doc to respect min_size / max_size using naive boundaries.
PrepareOptions
dataclass
¶
Options for the chunk preparation pipeline.
Source code in ontocast/tool/chunk/prepare.py
Attributes¶
document_type_hint = None
class-attribute
instance-attribute
¶
exclude_sections = None
class-attribute
instance-attribute
¶
section_schema_id = None
class-attribute
instance-attribute
¶
summarize_sections = None
class-attribute
instance-attribute
¶
target_sections = None
class-attribute
instance-attribute
¶
Methods:¶
__init__(section_schema_id=None, document_type_hint=None, target_sections=None, summarize_sections=None, exclude_sections=None)
¶
filter_allowlist()
¶
The extraction allowlist: target_sections only.
summarize_sections chooses which units are summarized and never
what is extracted.
needs_section_prepare()
¶
True when a request option explicitly requires section labels.
Section tagging itself is default-on (see CHUNK_SECTION_CLASSIFIER);
this only reports whether the request carries section-dependent options.
Source code in ontocast/tool/chunk/prepare.py
PreparedChunk
dataclass
¶
A prepared text chunk with optional structural metadata, label and route.
section_label_source and section_label_strength record which tier
decided the label and whether it may take part in a drop;
section_label_evidence holds every tier's vote. route and
route_reason record what routing decided and why.
Source code in ontocast/tool/chunk/prepare.py
Attributes¶
doc_item_refs = ()
class-attribute
instance-attribute
¶
headings
instance-attribute
¶
route = UnitRoute.EXTRACT
class-attribute
instance-attribute
¶
route_reason = RouteReason.CONTENT
class-attribute
instance-attribute
¶
section_label = None
class-attribute
instance-attribute
¶
section_label_evidence = ()
class-attribute
instance-attribute
¶
section_label_source = None
class-attribute
instance-attribute
¶
section_label_strength = None
class-attribute
instance-attribute
¶
text
instance-attribute
¶
Methods:¶
__init__(text, headings, doc_item_refs=(), section_label=None, section_label_source=None, section_label_strength=None, section_label_evidence=(), route=UnitRoute.EXTRACT, route_reason=RouteReason.CONTENT)
¶
SectionSelectionEmptyError
¶
Bases: ValueError
A section selection matched no segment in this document.
Distinct from a malformed parameter: target_sections=["reslts"] is
syntactically fine and only turns out to be wrong once this document has
been classified. Under the default CHUNK_SECTION_FILTER_ON_EMPTY=warn
this is a log line and the run continues to an empty graph, which reads
exactly like a document that genuinely had nothing to extract -- telling
those two apart is the whole point of the error mode.
Subclasses :class:ValueError so existing parameter guards keep their
shape. Deliberately not an api-layer error: nothing under tool/
imports ontocast.api, and the parameter here is well-formed.
Source code in ontocast/tool/chunk/prepare.py
Functions:¶
prepare_content_units(docling_doc, splitter, config, options, tools=None)
async
¶
Segment, tag, select, size, trim and route document text into units.
Section tagging is default-on: the sections-first flow runs unless
CHUNK_SECTION_CLASSIFIER=off (which leaves only the content rules of
routing; explicit section options are ignored with a warning in that
case).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
docling_doc
|
DoclingDocument
|
Converted source document. |
required |
splitter
|
ChunkerTool
|
Chunker used to size oversized sections. |
required |
config
|
ChunkConfig
|
Chunk configuration, including the classifier tier. |
required |
options
|
PrepareOptions
|
Per-request section schema and filters. |
required |
tools
|
'ToolBox | None'
|
ToolBox providing the LLM. Required only when
|
None
|
Returns:
| Type | Description |
|---|---|
PreparedDocument
|
The kept and dropped units, each with its route and reason, and how the |
PreparedDocument
|
schema was chosen. |
Raises:
| Type | Description |
|---|---|
ValueError
|
|
SectionSelectionEmptyError
|
|
Source code in ontocast/tool/chunk/prepare.py
859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 | |
size_bounded_text(text, config, split_fn, *, separator=DEFAULT_PART_SEPARATOR)
¶
Split text when needed, then enforce OntoCast chunk size bounds.