ontocast.api.process_helpers¶
Shared helpers for local batch processing and HTTP response assembly.
Attributes¶
GRAPH_RECURSION_LIMIT = 1000
module-attribute
¶
logger = logging.getLogger(__name__)
module-attribute
¶
Classes¶
Functions:¶
calculate_recursion_limit(head_chunks, server_config, *, max_visits_per_node=None)
¶
Recursion limit for one document run.
Arguments are accepted and ignored: the graph's depth is a property of its
topology, not of the document or the visit budget. See
:data:GRAPH_RECURSION_LIMIT.
Source code in ontocast/api/process_helpers.py
dump_catalog_nquads(tools, file_path, *, line_number=None, output_dir=None)
async
¶
Write the whole ontology catalog as it stands after this document.
Every ontology's current version, read back from the triple store -- the catalog the next document is extracted against -- as N-Quads with the ontology IRI as graph name, provenance stripped and lines sorted, so two consecutive dumps diff statement by statement. Blank-node labels are assigned per dump, so statements about blank nodes do not line up across dumps without canonicalising them.
Returns:
| Type | Description |
|---|---|
Path
|
The path written. |
Source code in ontocast/api/process_helpers.py
dump_facts_ttl(state, file_path, *, line_number=None, output_dir=None, strip_provenance=True)
¶
Write the facts Turtle when facts exist.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
state
|
AgentState
|
Document state carrying |
required |
file_path
|
Path
|
Source file the facts were extracted from. |
required |
line_number
|
int | None
|
Record number for JSONL inputs. |
None
|
output_dir
|
Path | None
|
Destination directory; defaults to the source's directory. |
None
|
strip_provenance
|
bool
|
Drop chunk-level provenance from the dump. Keeping it
is what lets a statement be traced back to its source span and
re-verified against the document; stripping it stays the default so
existing outputs are unchanged. Same meaning as the HTTP
|
True
|
Returns:
| Type | Description |
|---|---|
Path | None
|
The path written, or None when there are no facts. |
Source code in ontocast/api/process_helpers.py
dump_ontology_ttls(state, file_path, *, line_number=None, output_dir=None, ontology_manager=None)
¶
Write provenance-stripped ontology Turtle dumps when artifacts exist.
With ontology_manager, an ontology the document wrote a version of is
dumped as that stored version, header included.
Source code in ontocast/api/process_helpers.py
dump_run_manifest(state, file_path, *, config, line_number=None, output_dir=None, shapes_triples=None, shapes_prompt_selection=None, fanout_settings_apply=True, chunking_mode=None)
¶
Write the run's cost and configuration beside the facts TTL.
BudgetTracker is returned over HTTP and logged at INFO, then discarded,
so a batch run left no record of the model, the settings, or the tokens
behind its own output -- and no way to compare two dumps except by rerunning
them. One small JSON per document closes that.
chunking_mode is the chunker's mode after any fallback, which the
configured CHUNK_CHUNKING_MODE alone does not show.
Source code in ontocast/api/process_helpers.py
399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 | |
dump_units_jsonl(state, file_path, *, line_number=None, output_dir=None)
¶
Write the document's content units beside its facts, one JSON per line.
Kept provenance names each fact's source unit by IRI (<doc_iri>/<hid>)
only; this file holds the text behind each IRI, so a statement can be
traced to the passage it came from. text is the unit as chunked;
summarized is true when the model was shown a summary of it instead.
Returns:
| Type | Description |
|---|---|
Path | None
|
The path written, or None when the document has no units. |
Source code in ontocast/api/process_helpers.py
dump_validation_report(state, file_path, *, line_number=None, output_dir=None)
¶
Write the conformance summary and residual findings beside the facts TTL.
A batch run otherwise leaves no record of why a graph is non-conformant: the findings live on the state and are logged, and every downstream reader ends up re-running a validator to rebuild what the gate already computed.
unit_repairs and unit_failures are the per-unit counterparts of
gate_repairs: what the deterministic passes rewrote in each render
before aggregation, and which units produced nothing. A predicate the
machine substituted is otherwise indistinguishable in the TTL from one the
model asserted, and a unit that failed from one that found nothing.
Source code in ontocast/api/process_helpers.py
expand_input_to_states(file_path, *, config, head_chunks, ontology_context_mode_value, tenant, project, target_sections=None, exclude_sections=None, summarize_sections=None, summary_max_sentences=5, document_type_hint=None, section_schema_id=None, max_visits=None, document_metadata=None, facts_user_instruction='')
¶
Expand a local input file into one AgentState per logical record.
Source code in ontocast/api/process_helpers.py
686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 | |
facts_ttl_output_path(file_path, *, line_number=None, output_dir=None)
¶
Return the .facts.ttl path for a processed input file.
Source code in ontocast/api/process_helpers.py
finish_unit_pipeline_facts(state, ontology_graph, tools)
async
¶
The validation gate, for the single-unit path.
The document graph reaches it at VALIDATE_FACTS; the unit pipeline does
not run the graph, so both single-unit callers -- the CLI
--use-unit-pipeline batch path and the /process_unit route -- call
this after aggregation. Without it they would ship facts with no
functional-violation, coreference or SHACL check at all.
Source code in ontocast/api/process_helpers.py
flush_triple_configured_scope(tools)
async
¶
Match POST /flush without tenant/project: triple store only, current scope.
get_batch_input_extensions(tools)
¶
Return the suffixes ontocast process reads: conversion's plus JSONL.
A JSONL file is fanned out into one document per line before conversion, which only the batch path does.
Source code in ontocast/api/process_helpers.py
get_supported_input_extensions(tools)
¶
Return the file suffixes document conversion accepts (one document per file).
Source code in ontocast/api/process_helpers.py
ontology_ttl_output_path(file_path, *, line_number=None, output_dir=None, ontology_id=None)
¶
Return the .ontology.ttl path for a processed input file.
Source code in ontocast/api/process_helpers.py
persist_unit_pipeline_outputs(state, onto_result, facts_result, tools)
async
¶
Serialize unit-pipeline outputs using the standard document serializer.
Source code in ontocast/api/process_helpers.py
process_files_input(files, *, config, head_chunks, use_unit_pipeline, tools, workflow, ontology_context_mode_value, tenant, project, target_sections=None, exclude_sections=None, summarize_sections=None, summary_max_sentences=5, document_type_hint=None, section_schema_id=None, max_visits=None, document_metadata=None, facts_user_instruction='', output_dir=None, facts_output_dir=None, ontology_output_dir=None, strip_provenance=True, dump_catalog=False)
async
¶
Process each input file, isolating per-file failures.
Each document runs under a meter capped by MAX_INPUT_TOKENS_PER_DOCUMENT
/ MAX_OUTPUT_TOKENS_PER_DOCUMENT; a document stopped at a cap is a
failed file and gets no dumps. When the caller has bound its own
:class:~ontocast.onto.run_meter.RunMeter, that meter spans the batch and
:class:~ontocast.onto.run_meter.BudgetExceededError aborts it.
Returns:
| Type | Description |
|---|---|
list[Path]
|
The files that failed, in input order. Empty on full success. Callers |
list[Path]
|
use this to set a non-zero exit code -- previously every failure was |
list[Path]
|
logged and swallowed, so |
list[Path]
|
file produced any output. |
Source code in ontocast/api/process_helpers.py
877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 | |
resolve_batch_output_dirs(output_dir, facts_output_dir, ontology_output_dir)
¶
Resolve facts/ontology dump dirs from shared and override flags.
Returns:
| Type | Description |
|---|---|
tuple[Path | None, Path | None]
|
|
Source code in ontocast/api/process_helpers.py
safe_ontology_filename_id(ontology)
¶
Return a filesystem-safe ontology id fragment, or None if unavailable.
Source code in ontocast/api/process_helpers.py
select_unit_facts_ontology_graph(onto_result, facts_result)
¶
Return ontology graph for facts post-processing in unit pipeline flows.
Source code in ontocast/api/process_helpers.py
turtle_from_graph(graph, *, strip_provenance)
¶
Serialize graph to Turtle, optionally stripping reification/provenance.
Source code in ontocast/api/process_helpers.py
validate_unit_pipeline_facts(state, ontology_graph, tools)
¶
Run the post-aggregation invariant gate for the single-unit path.
merge_repair=False: un-merging re-aggregates retained units against
each other, which has no meaning for a single unit. Everything else is the
document path verbatim, so batch dumps stay comparable across the two entry
paths.