Skip to content

LLM Caching

OntoCast includes automatic LLM response caching to improve performance, reduce API costs, and enable offline testing capabilities.

Overview

The LLM caching system automatically caches responses from language model providers, ensuring that identical queries return cached results instead of making new API calls. This provides several benefits:

  • Performance: Cached responses return instantly
  • Cost Reduction: Avoids duplicate API calls
  • Offline Testing: Tests can run without API access
  • Transparency: Enabled by default; optional env vars control read-only mode, concurrency, and observability

Configuration

Setting Env Default Description
cache_enabled LLM_CACHE_ENABLED true Read/write disk cache
cache_read_only LLM_CACHE_READ_ONLY false Use cache without writing new entries
llm_max_inflight LLM_MAX_INFLIGHT 16 Max concurrent provider requests (all documents)

Cache size is bounded automatically — see Cache size and eviction:

Setting Env Default Description
cache_max_bytes ONTOCAST_CACHE_MAX_BYTES 1GB Ceiling for the whole cache directory; 0 disables eviction
cache_ttl_days ONTOCAST_CACHE_TTL_DAYS unset Drop entries unused for this many days
cache_prune_every ONTOCAST_CACHE_PRUNE_EVERY 256 Writes between size checks

Server-wide process concurrency (separate from LLM in-flight limit):

Setting Env Description
max_concurrent_processes MAX_CONCURRENT_PROCESSES Cap simultaneous /process and /process_unit handlers

Cache statistics are exposed on GET /info under llm_cache. Processing budget summaries include cache_hits when responses are served from disk (see Budget Tracking).

Concurrency layers

Three independent knobs affect parallelism:

Layer Setting What it limits
Unit workers PARALLEL_WORKERS Concurrent ontology/facts loops per document
Provider calls LLM_MAX_INFLIGHT Concurrent LLM HTTP requests across all units and documents
HTTP handlers MAX_CONCURRENT_PROCESSES Simultaneous /process and /process_unit pipelines

LLM_MAX_INFLIGHT prevents rate-limit storms when PARALLEL_WORKERS is high. MAX_CONCURRENT_PROCESSES is optional; when set, extra clients wait for a handler slot rather than starting unbounded full pipelines.

Shared Caching Architecture

OntoCast uses a shared caching architecture where:

  • Single Cacher Instance: One Cacher object manages all caching for all tools
  • Tool-Specific Subdirectories: Each tool gets its own subdirectory within the shared cache
  • Dependency Injection: Tools receive the shared Cacher instance through their constructors
  • Organized Storage: Cache files are organized by tool type (llm/, converter/, chunker/)

Benefits

  1. Memory Efficiency: Single cache instance instead of multiple
  2. Consistent Configuration: All tools use the same cache directory settings
  3. Centralized Management: Easy to clear, monitor, and manage all caches
  4. Better Organization: Clear separation of cache files by tool type

How It Works

Shared Caching

OntoCast uses a shared caching system where all tools share a single Cacher instance:

from ontocast.tool.llm import LLMTool
from ontocast.config import LLMConfig
from ontocast.tool.cache import Cacher

# Create shared cache instance
shared_cache = Cacher()

# Create LLM tool with shared cache
llm_config = LLMConfig(
    provider="openai", model_name="gpt-4o-mini", api_key="your-api-key"
)

llm_tool = LLMTool.create(config=llm_config, cache=shared_cache)

# The tool is async — await it (e.g. inside `asyncio.run(...)`).
# First call - hits API and caches response
response1 = await llm_tool("What is the capital of France?")

# Second call - returns cached response instantly
response2 = await llm_tool("What is the capital of France?")

Cache keys hash normalized prompt text (LangChain prompt values use to_string()) together with every setting that changes the provider's answer:

  • provider, model name, temperature, base URL
  • the Ollama generation knobs think, num_predict, num_ctx — these bound reasoning and output length, so the same prompt under a different num_ctx is a different response
  • the output schema name, for structured extract calls
  • a cache_format_version constant, bumped whenever the entry shape or the set of key inputs changes

Different configurations never share an entry. Binary inputs (PDFs and other documents fed to the converter) are hashed as raw bytes rather than decoded to text first.

Version 2 invalidates existing entries

The key gained the Ollama knobs and the format version, so entries written by earlier releases will not be hit. The first run after upgrading re-pays for every call; the stale entries age out under the size ceiling, or can be cleared immediately with ontocast cache clear.

Cache size and eviction

The cache bounds itself. Entries are regenerable, so once the directory exceeds ONTOCAST_CACHE_MAX_BYTES (1 GB by default) the least-recently-used entries are deleted until the total fits. Recency comes from each file's access time, so an entry that is written once and read constantly outlives one that was written recently and never touched again.

Pruning runs at process start and then after every ONTOCAST_CACHE_PRUNE_EVERY writes, which keeps a long-lived ontocast serve bounded between restarts. Set ONTOCAST_CACHE_MAX_BYTES=0 to disable eviction entirely.

The ontocast cache commands

ontocast cache stats                    # size per tool; flags orphaned subdirectories
ontocast cache prune                    # force a trim now
ontocast cache prune --max-bytes 500000000 --ttl-days 30
ontocast cache prune --orphaned         # drop subdirectories no current tool writes to
ontocast cache clear --subdir llm       # delete entries outright

--orphaned stays manual because "no live tool claims this directory" is an inference, unlike the size pass, which only ever discards entries the current code would regenerate. Cache versioning used to be done by renaming the subdirectory, so caches carried over from before this release may hold stray converter_v2/ and converter_v3/ directories; --orphaned clears them.

Cache Locations

Default Locations

The system automatically selects appropriate cache directories:

  • Tests: .test_cache/llm/ in the current working directory
  • Windows: %USERPROFILE%\AppData\Local\ontocast\llm\
  • Unix/Linux: ~/.cache/ontocast/llm/ (or $XDG_CACHE_HOME/ontocast/llm/)

Environment Variables

Set the cache directory via environment variables:

# OntoCast cache directory (recommended)
export ONTOCAST_CACHE_DIR=/path/to/custom/cache

# Or use XDG cache home (affects all XDG-compliant applications)
export XDG_CACHE_HOME=/path/to/custom/cache

Per-Invocation Override

There is no CLI flag for the cache directory — set the environment variable for a single invocation:

ONTOCAST_CACHE_DIR=/custom/cache/path ontocast serve

Cache Management

Cache Structure

The cache directory holds one flat subdirectory per tool. Provider and model are part of the key hash, not the path:

cache_dir/
├── llm/                    # LLM responses
│   ├── <sha256>.json
│   └── <sha256>.json
├── converter/              # Document conversion
└── chunker/                # Text chunking

Cache Files

Each cached response is stored as a JSON file containing: - Original prompt and parameters - Response content - Provider response metadata, replayed on a cache hit so cached and fresh calls behave identically - Cache key hash

Writes go to a temporary file and are renamed into place, so a reader never sees a half-written entry even with PARALLEL_WORKERS units in flight.

Testing with Caching

Offline Testing

Cached responses enable offline testing:

# First run - with API access
pytest test_llm_functionality.py

# Subsequent runs - offline (uses cached responses)
pytest test_llm_functionality.py

Test Isolation

Each test run uses a separate cache directory (.test_cache/llm/) to avoid interference between tests.

Performance Benefits

Speed Improvements

  • First Call: Normal API response time
  • Cached Calls: Near-instant response (< 1ms)
  • Batch Processing: Significant speedup for repeated operations

Cost Savings

  • Development: Avoid repeated API calls during development
  • Testing: Run tests without API costs
  • Production: Reduce API usage for common queries

Best Practices

Development

  1. Use Default Locations: Let the system choose appropriate cache directories
  2. Version Control: Add cache directories to .gitignore
  3. Cleanup: Automatic — inspect with ontocast cache stats if you want to see what is stored

Production

  1. Persistent Storage: Use persistent cache directories
  2. Monitoring: Monitor cache hit rates
  3. Sizing: Set ONTOCAST_CACHE_MAX_BYTES to whatever the deployment can spare; eviction handles the rest

Testing

  1. Isolated Caches: Each test run gets its own cache
  2. Deterministic: Cached responses ensure consistent test results
  3. Offline Capability: Tests can run without API access

Troubleshooting

Common Issues

  1. Cache Not Working: Check directory permissions
  2. Stale Responses: Clear cache directory
  3. Disk Space: Monitor cache directory size

Debug Cache

from ontocast.tool.llm import LLMTool

# Check cache directory
llm_tool = LLMTool.create(config=llm_config)
cache_dir = llm_tool.cache.shared_cacher.cache_dir / "llm"
print(f"Cache directory: {cache_dir}")

# List cached files
cache_files = list(cache_dir.glob("*.json"))
print(f"Cached responses: {len(cache_files)}")
print(f"Stats: {llm_tool.get_cache_stats()}")

Clear Cache

import shutil
from pathlib import Path

# Clear entire cache
cache_dir = Path.home() / ".cache" / "ontocast" / "llm"
if cache_dir.exists():
    shutil.rmtree(cache_dir)
    print("Cache cleared!")

Advanced Usage

Custom Cache Implementation

For advanced use cases, you can implement custom caching by extending the Cacher class:

from ontocast.tool.llm import LLMTool
from ontocast.tool.cache import Cacher
from pathlib import Path


class CustomLLMTool(LLMTool):
    def __init__(self, config, **kwargs):
        super().__init__(config, **kwargs)
        # Override with custom cache
        self.cache = Cacher(subdirectory="llm", cache_dir=Path("/custom/cache"))

Cache Statistics

from ontocast.tool.llm import LLMTool

llm_tool = LLMTool.create(config=llm_config)
stats = llm_tool.get_cache_stats()
# {"cache_hits": 12, "cache_misses": 3, "disk": {"total_files": 42, ...}}
print(f"Cache stats: {stats}")

On a running server, the same counters are available from GET /info (llm_cache field).

Integration with Other Tools

ToolBox Integration

Caching works seamlessly with the ToolBox through a shared Cacher instance:

from ontocast.toolbox import ToolBox
from ontocast.config import Config

# ToolBox automatically creates and uses a shared Cacher
config = Config()
tools = ToolBox(config)

# All tools (LLM, Converter, Chunker) share the same cache instance
result = tools.llm("Process this document")
converted = tools.converter(document_file)
chunks = tools.chunker(text)

Server Integration

The server automatically uses caching for all LLM operations:

# Start server with automatic caching (paths come from the environment)
ONTOCAST_CACHE_DIR=/data/cache ontocast serve

Security Considerations

Sensitive Data

  • Cache files may contain sensitive prompt data
  • Ensure proper file permissions on cache directories
  • Consider encryption for sensitive deployments

Access Control

  • Restrict access to cache directories
  • Use appropriate file system permissions
  • Consider network security for shared cache directories

Converter and Chunker Caching

In addition to LLM response caching, OntoCast also includes caching for document conversion and text chunking operations. This helps avoid redundant processing when the same documents or text are processed multiple times.

Converter Caching

The ConverterTool automatically caches document conversion results based on the input file content. This means:

  • PDF files: If the same PDF is processed multiple times, the conversion to markdown is cached
  • Other documents: PowerPoint, Word documents, etc. are also cached after conversion
  • Plain text: Text input is not cached as it doesn't require conversion

Chunker Caching

The ChunkerTool caches chunking results based on: - Input text content: The exact text being chunked - Chunking configuration: All chunking parameters (max_size, min_size, model, etc.) - Chunking mode: Whether semantic or naive chunking is used

This ensures that identical text with identical chunking parameters will return cached results.

Cache Organization

Caching is organized in subdirectories:

~/.cache/ontocast/
├── llm/            # LLM response cache
├── converter/      # Document conversion cache
└── chunker/        # Text chunking cache

Converter entries carry a format version inside the cache key, so a shape change no longer requires a new numbered directory. Directories left over from the older scheme are reported as orphaned by ontocast cache stats.

Cache Benefits

  1. Faster Processing: Repeated operations return instantly from cache
  2. Cost Reduction: Avoids redundant LLM API calls and processing
  3. Consistency: Identical inputs always produce identical outputs
  4. Offline Capability: Cached operations work without API access

Cache Management

You can access cache statistics and management through the tool instances:

from ontocast.tool.converter import ConverterTool
from ontocast.tool.chunk.chunker import ChunkerTool

# Get cache statistics
converter = ConverterTool()
stats = converter.cache.get_cache_stats()
print(
    f"Converter cache: {stats['total_files']} files, {stats['total_size_bytes']} bytes"
)

# Clear cache if needed
converter.cache.clear()

# Chunker cache management
chunker = ChunkerTool()
chunker.cache.clear()  # Clear chunker cache

Custom Cache Directories

You can specify custom cache directories in several ways:

1. Environment Variables

# OntoCast cache directory (recommended)
export ONTOCAST_CACHE_DIR=/custom/cache/path

# Or use XDG cache home (affects all XDG-compliant applications)
export XDG_CACHE_HOME=/custom/cache/path

2. Per-Invocation Environment

ONTOCAST_CACHE_DIR=/custom/cache/path ontocast process --input-path ./doc.pdf

3. Programmatic Configuration

from ontocast.toolbox import ToolBox
from ontocast.config import Config
from pathlib import Path

# Create config and set cache directory
config = Config()
config.tool_config.path_config.cache_dir = Path("/custom/cache/path")

# Create ToolBox with config (cache directory is automatically used)
tools = ToolBox(config)

# All tools will use the same custom cache directory
result = tools.llm("Process this document")
converted = tools.converter(document_file)
chunks = tools.chunker(text)

OpenAI Batch API (offline pre-fill)

For a large first pass over a corpus, you can pre-fill the disk cache using the provider Batch API — substantially cheaper per token, at hours of latency. See ontocast.tool.llm_batch:

from pathlib import Path
from ontocast.config import LLMConfig
from ontocast.tool.cache import Cacher
from ontocast.tool.llm_batch import (
    import_openai_batch_output_jsonl,
    write_openai_chat_batch_jsonl,
)

# 1. Build requests (custom_id -> prompt text mapping for import)
write_openai_chat_batch_jsonl(requests, Path("batch_input.jsonl"))
# 2. Submit batch_input.jsonl via OpenAI dashboard or API; download output JSONL
# 3. Import into the same cache directory the server will use
import_openai_batch_output_jsonl(
    Path("batch_output.jsonl"),
    shared_cache=Cacher(cache_dir="/path/to/cache"),
    llm_config=LLMConfig(),
    custom_id_to_cache_key={"req-1": "full prompt text used as cache key"},
)

Best Practices

  1. Let caching work automatically: No configuration needed for basic usage
  2. Monitor cache size: Check cache statistics periodically
  3. Clear cache when needed: If you change tool configurations significantly
  4. Use custom directories: For testing or specific deployment scenarios
  5. Cache persistence: Caches persist between runs for maximum benefit