Semantic Cache To LLM Rescue

(Repo url)

LLM applications have an unusual relationship with repetition.

Users repeatedly ask:

How do I reset my password?

Then they ask:

I forgot my password. What should I do?

Followed by:

Can you help me regain access to my account?

To a conventional cache, these are three unrelated strings.

To a human, they are the same support ticket wearing different hats.

If every version travels through retrieval, prompt assembly, a hosted model, and post-processing, the application repeatedly pays for roughly the same answer

This is less a sophisticated AI architecture and more a subscription service for déjà vu

This is the problem addressed by Semacache, a Python library that combines exact-key caching with semantic similarity lookup for LLM applications.

Repository:

https://github.com/uncle-voh-max/semacache

Semacache places a dual-layer cache in the query path

1. Look for the exact same request.

2. If that misses, look for a semantically similar request.

3. Only invoke the expensive generator when both miss.

** The library supports in-memory and Redis-backed stores, configurable embeddings, tenant isolation, cache policies, request coalescing, validation, health checks, invalidation, and metrics.

Why caching belongs in the LLM query path

Caching is hardly new.

We cache database records, HTTP responses, rendered pages, DNS lookups, and occasionally the answer to “what did product actually ask us to build?”

Traditional caches normally depend on exact identity:

same key → same result

That works well for deterministic requests such as:

GET /customers/123

LLM requests are different because users express equivalent intent using different language.

Consider:

A: What is semantic caching?
B: Explain semantic caching.
C: How does a cache based on meaning work?

An exact cache can reuse the response for another occurrence of A.

A semantic cache may reuse the response across A, B, and C : provided their embeddings are similar enough and their surrounding metadata is compatible.

This gives us two complementary cache layers.

Exact cache

The exact cache uses a deterministic key, such as a SHA-256 hash derived from the query and relevant metadata.

It is:

• fast

• cheap

• predictable

• appropriate for byte-for-byte or normalised repetitions

Semantic cache

The semantic cache embeds the incoming query and compares it with cached query embeddings.

It is

• more flexible

• slightly more expensive than exact lookup

• much cheaper than a full LLM generation

• sensitive to similarity thresholds and scope

The architectural principle is straightforward:

Use the cheapest reliable answer source first.

Do not generate an embedding when an exact hash lookup will do. Do not call a large model when an existing semantically equivalent answer is safe to reuse.

The Semacache query path

Semacache organises its implementation into a client layer, an orchestration service, and pluggable protocol-backed components for exact stores, semantic stores, embeddings, validation, eviction, coalescing, and metrics.

The effective query path looks like this

The repository’s architecture follows this ordering: exact lookup first, then embedding and semantic lookup, followed by generator execution and storage after a miss

That order matters.

If semantic search happened first, every query, including exact duplicates would pay the embedding cost.

A good cache should not perform a small expensive operation to discover that a free operation would have worked

Getting started without constructing a cache empire

For local development, Semacache provides an in-memory constructor:

from semacache import CacheMetadata, CachePolicy, SemanticCache
 
async with SemanticCache.in_memory(
    namespace="support-bot",
    ttl_seconds=3600,
) as cache:
    result = await cache.get_or_generate(
        query="How do I reset my password?",
        generator=generate_answer,
        metadata=CacheMetadata(agent="support"),
        policy=CachePolicy.READ,
    )
 
    print(result.status, result.value)

The first request is a miss and invokes generate_answer.

A repeated query becomes an exact hit.

A sufficiently similar query may become a semantic hit

result = await cache.get_or_generate(
    query="I forgot my password. How can I regain access?",
    generator=generate_answer,
    metadata=CacheMetadata(agent="support"),
    policy=CachePolicy.READ,
)

The primary API returns a typed result containing the value, cache status, semantic similarity where relevant, and execution information.

This is a useful API shape because callers should know whether a response was:

EXACT_HIT
SEMANTIC_HIT
MISS
DISABLED

A cache that hides its decisions is easy to adopt and difficult to operate.

Sensible defaults: useful, conservative, and replaceable

Defaults are part of a library’s architecture.

A poor default pushes every caller towards accidental complexity. A reckless default quietly trades correctness for performance and waits for production to notice.

Semacache starts with few defaults:

TTL:                         3600 seconds
Maximum entries:             10,000
Estimated memory ceiling:    512 MB
Eviction:                    TTL + LRU
Semantic caching:            enabled
Similarity threshold:        0.85
Similarity metric:           cosine
Embedding provider:          Ollama
Embedding model:             nomic-embed-text
Request coalescing:          enabled
Concurrent LLM calls:        5
Request timeout:             120 seconds
Reject empty responses:      true
Cache-read failure:          bypass

These defaults are published in the repository’s configuration reference.

Request coalescing

Suppose 100 identical requests arrive immediately after an entry expires.

Without coalescing:

100 cache misses → 100 LLM requests

With coalescing:

100 cache misses → 1 LLM request + 99 waiters

Semacache enables coalescing by default and supports memory-based or Redis-backed coordination.

This protects the system from a cache stampede, sometimes known in meetings as “unexpected organic growth.”

Validation before storage

Semacache rejects empty responses by default and provides a response-validation extension point.

This is an underrated feature.

Without validation, a transient model failure can be cached and served repeatedly:

First request:
LLM returns empty string
 
Next 10,000 requests:
Cache efficiently returns empty string

Caching amplifies both good and bad results.

Validate before storing:

• response is not empty

• expected schema is valid

• required citations exist

• safety checks passed

• result is not an error page

• result has an acceptable quality score

Configuring Semacache for production

A distributed configuration can use Redis for both cache layers and Ollama for embeddings:

from semacache import SemanticCache, SemanticCacheConfig
 
config = SemanticCacheConfig(
    namespace="customer-support",
    ttl_seconds=3600,
    exact_cache={
        "provider": "redis",
        "url": "redis://localhost:6379/0",
    },
    semantic_cache={
        "provider": "redis",
        "enabled": True,
        "similarity_threshold": 0.90,
    },
    embeddings={
        "provider": "ollama",
        "model": "nomic-embed-text",
    },
    coalescing={
        "enabled": True,
        "mode": "redis",
        "lock_timeout_seconds": 30,
    },
    max_concurrent_llm_requests=5,
    request_timeout_seconds=120,
)
 
cache = SemanticCache(config=config)

The same configuration can be loaded from YAML:

namespace: customer-support
ttl_seconds: 3600
 
exact_cache:
  provider: redis
  url: redis://localhost:6379/0
 
semantic_cache:
  provider: redis
  enabled: true
  similarity_threshold: 0.90
 
embeddings:
  provider: ollama
  model: nomic-embed-text
 
coalescing:
  enabled: true
  mode: redis
  lock_timeout_seconds: 30

Or through SEMACACHE_* environment variables. The repository supports all three configuration styles.

Senior-engineering tip: Put operational values in configuration, but keep policy ownership explicit. A threshold living in YAML is configurable, a threshold nobody owns is merely abandoned

Scope is part of the cache key

Semantic equivalence is not enough.

The same question may require different answers for:

• different tenants

• different users

• different agents

• different models

• different permissions

• different geographic regions

Semacache includes metadata fields for tenant, user scope, model, agent, and custom values:

metadata = CacheMetadata(
    tenant="acme-corp",
    user_scope="user-123",
    model="gpt-4.1-mini",
    agent="support-bot",
    extra={
        "locale": "en-AU",
        "knowledge_version": "2026-08",
    },
)

The repository builds multi-tenant and user-level isolation into its metadata model.

This is critical.

The following queries may be semantically identical

What is my current balance?
What amount do I have available?

They must never share an answer across users.

Similarity tells us that two questions mean the same thing.

It does not tell us that two callers are authorised to receive the same data.

Cache policies for different operational needs

Semacache exposes four policies.

READ

Use the cache normally. Read existing entries, call the generator on a miss, and store the result.

policy = CachePolicy.READ

This is the normal production path.

BYPASS

Call the generator directly without reading or writing the cache.

policy = CachePolicy.BYPASS

Useful for debugging, experiments, and queries that must always be fresh.

REFRESH

Generate a new response and update the cache.

policy = CachePolicy.REFRESH

Useful after content updates or when a caller explicitly requests freshness.

DISABLED

Skip cache behaviour entirely.

policy = CachePolicy.DISABLED

The repository documents these policies as part of its public API.

A common pattern is to select policy from request context

policy = (
    CachePolicy.REFRESH
    if request.force_refresh
    else CachePolicy.READ
)

Avoid scattering cache bypass conditions through business code. Centralise the decision so that nobody has to reverse-engineer why one endpoint has twelve mysterious if not cache branches.

What should not be semantically cached?

Semantic caching is most valuable when answers are reusable.

Strong candidates include:

FAQs
Documentation assistance
Product explanations
Policy summaries
Static code examples
General onboarding
Repeated classification requests

Poor candidates include:

Personalised account data
Rapidly changing prices
Current inventory
One-time security decisions
Nondeterministic creative output
Requests containing unique tool results
Prompts whose system instructions differ materially

You should also be cautious when query wording carries subtle meaning

Consider:

Can I cancel the transfer?
How can I cancel the transfer?
Why can’t I cancel the transfer?

These are nearby in embedding space, but not necessarily interchangeable.

The first asks about eligibility.

The second asks for procedure.

The third may indicate a failed or irreversible state.

A lower threshold may collapse all three into one cheerful but incorrect answer.

Evaluating the cache before celebrating the savings

A cache hit is not automatically a good outcome.

An exact hit is usually easy to reason about.

A semantic hit must be evaluated along two dimensions:

Efficiency: Did it avoid expensive work?
Correctness: Was the cached answer genuinely reusable?

The evaluation flow should look like this:

The minimum useful metrics are:

Hit rate

(exact hits + semantic hits) / total requests

Semantic precision

correct semantic hits / all semantic hits

LLM avoidance rate

requests served without generation / total requests

Cost reduction

baseline generation cost − cached-path cost

Latency reduction

Compare p50, p95, and p99 latency before and after enabling the cache.

Stale or incorrect response rate

Track semantic hits later rejected by users, validators, or downstream checks.

Do not optimise only for hit rate.

A cache with a 95% hit rate and a 20% incorrect-reuse rate is not efficient. It is confidently distributing leftovers.

Estimating expected cost reduction

Consider an illustrative workload of 100,000 monthly queries.

Assume:

Average LLM request cost:       $0.012
Average embedding cost:         $0.0001
Exact-cache hit rate:           20%
Semantic-cache hit rate:        35%
Miss rate:                      45%

Without caching:

100,000 × $0.012 = $1,200

With dual-layer caching:

Exact hits:
20,000 × approximately $0 = approximately $0
 
Semantic lookup candidates:
80,000 × $0.0001 = $8
 
LLM generations:
45,000 × $0.012 = $540
 
Approximate total:
$548

Estimated reduction:

$1,200 − $548 = $652 per month

That is roughly a 54% reduction in this illustrative example.

The real result will depend on traffic repetition, prompt size, model price, embedding cost, TTL, threshold, and correctness requirements.

The expected evaluation shape is:

More exact hits
    → almost pure savings
 
More correct semantic hits
    → substantial savings with small embedding overhead
 
More false semantic hits
    → impressive dashboard, unhappy users
 
Higher threshold
    → fewer but safer semantic hits
 
Lower threshold
    → more reuse, greater correctness risk

Failure policies should protect the primary path

Semacache allows cache-read failures to bypass the cache, write failures to be logged or raised, and embedding failures to raise or return no result.

A sensible production philosophy is

Cache unavailable:
Prefer degraded performance over total outage
 
LLM unavailable:
Return a deliberate error or approved stale response
 
Cache write fails:
Serve the valid generated answer and record the failure

The cache exists to improve the system.

It should not become a more fragile dependency than the expensive service it protects.

Whether to serve stale data should depend on the domain. The repository defaults serve_stale to false, which is the safer general-purpose choice.

Observe outcomes, not merely cache size

Semacache exposes statistics, health checks, execution information, and a metrics-recorder extension point. Its metrics protocol includes cache hits, misses, generator latency, embedding requests, and errors.

Track:

Exact_hit_rate
Semantic_hit_rate
Semantic_similarity_distribution
Cache_miss_rate
Generator_latency
Embedding_latency
Cache_read_failures
Cache_write_failures
Coalesced_request_count
Response_validation_failures
Estimated_cost_avoided

Break these down by:

• tenant

• model

• route

• cache namespace

• application version

A global 50% hit rate can hide one tenant receiving 90% incorrect reuse while another gets none.

Aggregates are useful. Segmentation is where the bodies are found.

Closing thought

Traditional caching asks:

Have I seen this key before?

Semantic caching asks:

Have I answered this meaning before?

That second question can dramatically reduce LLM latency and cost, but it introduces judgement into the cache path.

Semacache handles the mechanics thoughtfully:

• exact lookup before embeddings

• semantic lookup before generation

• explicit policies

• response validation

• request coalescing

• tenant-aware metadata

• pluggable stores and embedding providers

• observable cache outcomes

The remaining responsibility belongs to the integrating team:

• define what may be reused

• evaluate semantic precision

• tune thresholds against real queries

• invalidate when knowledge changes

• measure both cost savings and incorrect hits

The senior-engineering lesson is simple:

A semantic cache is not merely a performance optimisation. It is a controlled decision to reuse a previous answer.

Make that decision cheap.

Make it visible.

And most importantly, make sure it is correct before proudly announcing that the system has saved 54% by answering the wrong question faster.