llm-semantic-caching
verified0b15ba04-30c3-48aa-a881-3c3fc8cbab3c
Cache semantically-similar LLM requests with embeddings to cut cost and latency ā exact vs semantic layers, thresholds, and validation.
Metadata
Skill file
# LLM Semantic Caching
Use when users repeatedly ask the *same intent in different words* ā FAQ bots,
support copilots, dashboards, agent loops ā and you want to serve many of those
requests from cache instead of paying for identical inference each time.
Classic exact-match caching only catches byte-identical queries, which is rare
with free-text input. Semantic caching embeds the query and returns a stored
response when a *similar* past query is found above a similarity threshold,
cutting typical response latency from hundreds of ms to single-digit ms and
removing token cost entirely on the hit path.
## Two-layer architecture (recommended)
Check **exact** first (zero correctness risk, instant), then **semantic** on a
miss, then call the LLM only if both miss, and store the fresh response in both
layers. The exact layer catches repeat requests payload-fast; the semantic layer
catches paraphrases; the LLM call stays the last resort.
## Choosing a store
- **Redis + a vector index** (RediSearch `ON HASH` with `VECTOR` fields): robust,
production-ready, easy to co-locate with your app.
- **GPTCache** (Python library, Zilliz): fast to stand up, pluggable backend
(SQLite/Redis), but you inherit its abstraction choices.
- **Any vector DB you already run** (Qdrant/Milvus/pgvector): fine if you don't
want a new dependency ā you just need cosine search over query embeddings.
## Tuning the similarity threshold
This is the one knob that defines the risk/reward curve.
- **Too low** ā false hits: a cached answer that is only loosely related to the
new question is returned. This is a correctness bug.
- **Too high** ā few hits, little savings.
Start around **0.92ā0.95 cosine similarity** for a "safe but useful" default and
tune down only with validation. The right number depends on your embedding model
and how clustered your real queries are.
Hash the *question + prompt context* (not just the question) so two different
tasks don't collide. Never cache responses that contain user-specific data you'd
then leak to another user ā scope cache keys by user/tenant where relevant.
## Validation is mandatory
Semantic cache can silently return a wrong-but-plausible answer. Before
increasing hit rate:
- Build a set of near-duplicate and distinct queries for your domain.
- Confirm near-duplicates hit and distinct queries miss at your threshold.
- **Grade a 1ā5% sample of real hits** with an LLM-as-judge (or human) against
what the LLM *would* have returned fresh. If the false-positive rate climbs
above your tolerance, raise the threshold.
Reported positive-hit rates in the literature are typically 60ā70% with >95%
precision, but *your* corpus determines your numbers ā measure, don't copy.
## Pitfalls
- Caching responses to mutable-state queries (inventory, prices, anything that
changes) ā either skip caching them or use a short TTL.
- Basing the cache key only on the question when the answer depends on context
(system prompt, data source) ā collide across contexts and you leak answers.
- Zero TTL on a cache that should reflect freshness requirements.
- Not distinguishing cache provenance in telemetry, so you can't tell hits from
misses or audit wrong-fresh answers.
- Using a cheap embedding model with poor semantic sensitivity, then seeing
garbage hits because unrelated text scores high.
## Verify
- Time the same request twice: second should be single-digit ms.
- Confirm near-duplicate queries return the cached response and unrelated
queries do not, at your chosen threshold.
- Report you false-positive rate from a graded hit sample and show it's within
tolerance.