Skip to content

LLM Caching

OntoCast includes automatic LLM response caching to improve performance, reduce API costs, and enable offline testing capabilities.

Overview

The LLM caching system automatically caches responses from language model providers, ensuring that identical queries return cached results instead of making new API calls. This provides several benefits:

  • Performance: Cached responses return instantly
  • Cost Reduction: Avoids duplicate API calls
  • Offline Testing: Tests can run without API access
  • Transparency: Enabled by default; optional env vars control read-only mode, concurrency, and observability

Configuration

Setting Env Default Description
cache_enabled LLM_CACHE_ENABLED true Read/write disk cache
cache_read_only LLM_CACHE_READ_ONLY false Use cache without writing new entries
llm_max_inflight LLM_MAX_INFLIGHT 16 Max concurrent provider requests (all documents)

Server-wide process concurrency (separate from LLM in-flight limit):

Setting Env Description
max_concurrent_processes MAX_CONCURRENT_PROCESSES Cap simultaneous /process and /process_unit handlers

Cache statistics are exposed on GET /info under llm_cache. Processing budget summaries include cache_hits when responses are served from disk (see Budget Tracking).

Concurrency layers

Three independent knobs affect parallelism:

Layer Setting What it limits
Unit workers PARALLEL_WORKERS Concurrent ontology/facts loops per document
Provider calls LLM_MAX_INFLIGHT Concurrent LLM HTTP requests across all units and documents
HTTP handlers MAX_CONCURRENT_PROCESSES Simultaneous /process and /process_unit pipelines

LLM_MAX_INFLIGHT prevents rate-limit storms when PARALLEL_WORKERS is high. MAX_CONCURRENT_PROCESSES is optional; when set, extra clients wait for a handler slot rather than starting unbounded full pipelines.

Shared Caching Architecture

OntoCast uses a shared caching architecture where:

  • Single Cacher Instance: One Cacher object manages all caching for all tools
  • Tool-Specific Subdirectories: Each tool gets its own subdirectory within the shared cache
  • Dependency Injection: Tools receive the shared Cacher instance through their constructors
  • Organized Storage: Cache files are organized by tool type (llm/, converter/, chunker/)

Benefits

  1. Memory Efficiency: Single cache instance instead of multiple
  2. Consistent Configuration: All tools use the same cache directory settings
  3. Centralized Management: Easy to clear, monitor, and manage all caches
  4. Better Organization: Clear separation of cache files by tool type

How It Works

Shared Caching

OntoCast uses a shared caching system where all tools share a single Cacher instance:

from ontocast.tool.llm import LLMTool
from ontocast.config import LLMConfig
from ontocast.tool.cache import Cacher

# Create shared cache instance
shared_cache = Cacher()

# Create LLM tool with shared cache
llm_config = LLMConfig(
    provider="openai",
    model_name="gpt-4o-mini",
    api_key="your-api-key"
)

llm_tool = LLMTool.create(config=llm_config, cache=shared_cache)

# First call - hits API and caches response
response1 = llm_tool("What is the capital of France?")

# Second call - returns cached response instantly
response2 = llm_tool("What is the capital of France?")

Cache keys hash normalized prompt text (LangChain prompt values use to_string()) together with provider, model, temperature, base URL, and schema-specific fields (for structured extract calls). Different configurations never share an entry.

Cache Locations

Default Locations

The system automatically selects appropriate cache directories:

  • Tests: .test_cache/llm/ in the current working directory
  • Windows: %USERPROFILE%\AppData\Local\ontocast\llm\
  • Unix/Linux: ~/.cache/ontocast/llm/ (or $XDG_CACHE_HOME/ontocast/llm/)

Environment Variables

Set the cache directory via environment variables:

# OntoCast cache directory (recommended)
export ONTOCAST_CACHE_DIR=/path/to/custom/cache

# Or use XDG cache home (affects all XDG-compliant applications)
export XDG_CACHE_HOME=/path/to/custom/cache

CLI Parameter

Specify cache directory via command line:

ontocast --env-path .env --working-directory ./work --cache-dir /custom/cache/path

Cache Management

Cache Structure

The cache directory contains organized subdirectories:

cache_dir/
├── openai/
│   ├── gpt-4o-mini/
│   │   ├── prompt_hash_1.json
│   │   └── prompt_hash_2.json
│   └── gpt-4/
│       └── prompt_hash_3.json
└── ollama/
    └── llama2/
        └── prompt_hash_4.json

Cache Files

Each cached response is stored as a JSON file containing: - Original prompt and parameters - Response content - Metadata (timestamp, model info) - Cache key hash

Testing with Caching

Offline Testing

Cached responses enable offline testing:

# First run - with API access
pytest test_llm_functionality.py

# Subsequent runs - offline (uses cached responses)
pytest test_llm_functionality.py

Test Isolation

Each test run uses a separate cache directory (.test_cache/llm/) to avoid interference between tests.

Performance Benefits

Speed Improvements

  • First Call: Normal API response time
  • Cached Calls: Near-instant response (< 1ms)
  • Batch Processing: Significant speedup for repeated operations

Cost Savings

  • Development: Avoid repeated API calls during development
  • Testing: Run tests without API costs
  • Production: Reduce API usage for common queries

Best Practices

Development

  1. Use Default Locations: Let the system choose appropriate cache directories
  2. Version Control: Add cache directories to .gitignore
  3. Cleanup: Periodically clean old cache files

Production

  1. Persistent Storage: Use persistent cache directories
  2. Monitoring: Monitor cache hit rates
  3. Maintenance: Implement cache cleanup strategies

Testing

  1. Isolated Caches: Each test run gets its own cache
  2. Deterministic: Cached responses ensure consistent test results
  3. Offline Capability: Tests can run without API access

Troubleshooting

Common Issues

  1. Cache Not Working: Check directory permissions
  2. Stale Responses: Clear cache directory
  3. Disk Space: Monitor cache directory size

Debug Cache

from ontocast.tool.llm import LLMTool

# Check cache directory
llm_tool = LLMTool.create(config=llm_config)
cache_dir = llm_tool.cache.shared_cacher.cache_dir / "llm"
print(f"Cache directory: {cache_dir}")

# List cached files
cache_files = list(cache_dir.glob("*.json"))
print(f"Cached responses: {len(cache_files)}")
print(f"Stats: {llm_tool.get_cache_stats()}")

Clear Cache

import shutil
from pathlib import Path

# Clear entire cache
cache_dir = Path.home() / ".cache" / "ontocast" / "llm"
if cache_dir.exists():
    shutil.rmtree(cache_dir)
    print("Cache cleared!")

Advanced Usage

Custom Cache Implementation

For advanced use cases, you can implement custom caching by extending the Cacher class:

from ontocast.tool.llm import LLMTool
from ontocast.tool.cache import Cacher
from pathlib import Path

class CustomLLMTool(LLMTool):
    def __init__(self, config, **kwargs):
        super().__init__(config, **kwargs)
        # Override with custom cache
        self.cache = Cacher(subdirectory="llm", cache_dir=Path("/custom/cache"))

Cache Statistics

from ontocast.tool.llm import LLMTool

llm_tool = LLMTool.create(config=llm_config)
stats = llm_tool.get_cache_stats()
# {"cache_hits": 12, "cache_misses": 3, "disk": {"total_files": 42, ...}}
print(f"Cache stats: {stats}")

On a running server, the same counters are available from GET /info (llm_cache field).

Integration with Other Tools

ToolBox Integration

Caching works seamlessly with the ToolBox through a shared Cacher instance:

from ontocast.toolbox import ToolBox
from ontocast.config import Config

# ToolBox automatically creates and uses a shared Cacher
config = Config()
tools = ToolBox(config)

# All tools (LLM, Converter, Chunker) share the same cache instance
result = tools.llm("Process this document")
converted = tools.converter(document_file)
chunks = tools.chunker(text)

Server Integration

The server automatically uses caching for all LLM operations:

# Start server with automatic caching
ontocast --env-path .env --working-directory /data/working

Security Considerations

Sensitive Data

  • Cache files may contain sensitive prompt data
  • Ensure proper file permissions on cache directories
  • Consider encryption for sensitive deployments

Access Control

  • Restrict access to cache directories
  • Use appropriate file system permissions
  • Consider network security for shared cache directories

Converter and Chunker Caching

In addition to LLM response caching, OntoCast also includes caching for document conversion and text chunking operations. This helps avoid redundant processing when the same documents or text are processed multiple times.

Converter Caching

The ConverterTool automatically caches document conversion results based on the input file content. This means:

  • PDF files: If the same PDF is processed multiple times, the conversion to markdown is cached
  • Other documents: PowerPoint, Word documents, etc. are also cached after conversion
  • Plain text: Text input is not cached as it doesn't require conversion

Chunker Caching

The ChunkerTool caches chunking results based on: - Input text content: The exact text being chunked - Chunking configuration: All chunking parameters (max_size, min_size, model, etc.) - Chunking mode: Whether semantic or naive chunking is used

This ensures that identical text with identical chunking parameters will return cached results.

Cache Organization

Caching is organized in subdirectories:

~/.cache/ontocast/
├── llm/           # LLM response cache
├── converter/     # Document conversion cache
└── chunker/       # Text chunking cache

Cache Benefits

  1. Faster Processing: Repeated operations return instantly from cache
  2. Cost Reduction: Avoids redundant LLM API calls and processing
  3. Consistency: Identical inputs always produce identical outputs
  4. Offline Capability: Cached operations work without API access

Cache Management

You can access cache statistics and management through the tool instances:

from ontocast.tool.converter import ConverterTool
from ontocast.tool.chunk.chunker import ChunkerTool

# Get cache statistics
converter = ConverterTool()
stats = converter.cache.get_cache_stats()
print(f"Converter cache: {stats['total_files']} files, {stats['total_size_bytes']} bytes")

# Clear cache if needed
converter.cache.clear()

# Chunker cache management
chunker = ChunkerTool()
chunker.cache.clear()  # Clear chunker cache

Custom Cache Directories

You can specify custom cache directories in several ways:

1. Environment Variables

# OntoCast cache directory (recommended)
export ONTOCAST_CACHE_DIR=/custom/cache/path

# Or use XDG cache home (affects all XDG-compliant applications)
export XDG_CACHE_HOME=/custom/cache/path

2. CLI Parameter

ontocast --env-path .env --working-directory ./work --cache-dir /custom/cache/path

3. Programmatic Configuration

from ontocast.toolbox import ToolBox
from ontocast.config import Config
from pathlib import Path

# Create config and set cache directory
config = Config()
config.tool_config.path_config.cache_dir = Path("/custom/cache/path")

# Create ToolBox with config (cache directory is automatically used)
tools = ToolBox(config)

# All tools will use the same custom cache directory
result = tools.llm("Process this document")
converted = tools.converter(document_file)
chunks = tools.chunker(text)

OpenAI Batch API (offline benchmarks)

For large first-pass benchmark runs, you can pre-fill the disk cache using the provider Batch API (~50% lower cost, hours of latency). See ontocast.tool.llm_batch:

from pathlib import Path
from ontocast.config import LLMConfig
from ontocast.tool.cache import Cacher
from ontocast.tool.llm_batch import (
    import_openai_batch_output_jsonl,
    write_openai_chat_batch_jsonl,
)

# 1. Build requests (custom_id -> prompt text mapping for import)
write_openai_chat_batch_jsonl(requests, Path("batch_input.jsonl"))
# 2. Submit batch_input.jsonl via OpenAI dashboard or API; download output JSONL
# 3. Import into the same cache directory the server will use
import_openai_batch_output_jsonl(
    Path("batch_output.jsonl"),
    shared_cache=Cacher(cache_dir="/path/to/cache"),
    llm_config=LLMConfig(),
    custom_id_to_cache_key={"req-1": "full prompt text used as cache key"},
)

Best Practices

  1. Let caching work automatically: No configuration needed for basic usage
  2. Monitor cache size: Check cache statistics periodically
  3. Clear cache when needed: If you change tool configurations significantly
  4. Use custom directories: For testing or specific deployment scenarios
  5. Cache persistence: Caches persist between runs for maximum benefit