llm-prompt-caching
verifiedd01c69ed-5f6f-4ed5-bc82-e3833b639783
Cut LLM API cost and latency with prompt caching ā provider cache_control, prefix design, TTL, and hit-rate monitoring.
Metadata
Skill file
# LLM Prompt Caching
Use when your app sends overlapping prompt prefixes repeatedly ā a long system
prompt, a few-shot few examples, a tool/function schema block, or a fixed
instructions preamble ā and you want cheaper, faster requests without changing
your model.
## How it works
Providers cache the *prefix* of a request. When a later request shares that
exact prefix, the reused tokens are billed at a heavy discount (and prefilled
more quickly) because the provider doesn't re-run attention over them. The
longest matching prefix wins; anything after the divergence is billed and
processed normally. This is why **static content must go at the top and dynamic
content at the bottom.**
## Provider differences (verify current pricing before shipping)
- **OpenAI**: automatic, zero code changes. Cached input tokens bill at roughly
**50% of base input price**; reads are best-effort (a ~50% hit rate target,
not guaranteed). A minimum prefix length applies (typically 1,024 tokens).
Because it's automatic you must *measure* to know whether it's helping.
- **Anthropic**: *manual* ā you add `cache_control: {"type": "ephemeral"}`
breakpoints to messages. Cached reads bill at ~**90% off** base input and hits
are effectively guaranteed when the prefix matches, but you only pay the
reduced rate up to the last breakpoint. A write cost applies per breakpoint.
TTL is **5 minutes** by default (1-hour option on some models).
- **Google** and others: auto/manual flags differ; check the provider's current
docs ā don't assume a discount or a minimum from a year-old blog.
Anthropic's TTL is the sharpest gotcha: if your traffic is spaced more than the
TTL apart (e.g. nightly batch jobs), the cache goes cold and you pay the write
cost with no read benefit.
## Structure the prompt for hits
- Put the system prompt, tool/function schemas, few-shot examples first ā the
long part that rarely changes.
- Append user/variable content last so it doesn't break the shared prefix.
- Use clear delimiters between the static and dynamic blocks. A timestamp,
request id, or random spacer anywhere in the prefix destroys overlap.
- For Anthropic, place a `cache_control` breakpoint after each long stable block
(system prompt, tool block) ā don't put one on the tiny tail that changes.
- Keep all requests from one logical app sharing the *same* prefix layout so
they the hit each other's cache instead of fragmenting.
## Monitor, don't assume
Track the provider-reported counters, e.g. Anthropic
`cache_read_input_tokens` vs `cache_creation_input_tokens`; OpenAI exposes
cached_tokens equivalents. Ask for your provider's actual field names.
A **hit rate below ~70%** on a prefix you think is static usually means
something is inadvertently varying it (hidden timestamp, nondeterministic
ordering in a serialized list, per-user metadata injected into the shared
block). Log the counters per request and graph them.
## Pitfalls
- Premature caching of a prefix that's already cheap (small system prompt) ā
the write cost can exceed the read savings.
- Caching dynamic blocks: cache only what's stable, otherwise you pay write
costs that never repay.
- Ignoring TTL for batch/off-peak workloads that won't keep the cache warm.
- Multi-provider or per-request prefixes that fragment the cache into many
near-misses.
## Verify
- Run a representative request twice; confirm the second reports cached tokens.
- Measure real $ saved = (non-cached input tokens ā cached input tokens Ć
discounted rate) over a day of traffic.
- Confirm hit rate stays above your target across distinct prefix variants before
calling it done.