prompt-compression
verified4b3255f4-523d-4dd9-a649-8196fc682a70
Cut LLM prompt cost and latency in RAG and long-context apps with prompt compression — LLMLingua/LongLLMLingua selective-context token pruning, compression-ratio tuning, what to compress, and when it wins.
Metadata
Skill file
# Prompt Compression
Use when your prompts carry a lot of bulk — especially **retrieval-augmented
(RAG) context** — and you're paying for tokens and waiting on latency for content
the model mostly ignores. Prompt compression prunes low-information tokens before
they reach the model, cutting cost and latency, and in RAG it can even *raise*
accuracy by removing distracting/noise context (e.g. mitigating "lost in the
middle").
## Where it wins most
- **RAG retrieval blobs are the highest-yield target.** Retrieved chunks are
reliably redundant, compression ratios are highest, and quality impact is
lowest because you remove context the model wasn't using.
- **Long-context tasks** with big system prompts, histories, or multi-doc
contexts.
- Anywhere prompt tokens dominate spend (input-priced APIs) — prompt tokens are
often where the bill lives.
## The techniques (highest to lowest leverage)
1. **Manual restructuring (do this first, free).** Tighten verbose system
prompts, cut repeated instructions, deduplicate static blocks. Typical
20–40% reduction with zero quality loss. Cheap, safe, no new dependencies.
2. **Selective-context-style filtering (LLMLingua).** Use a small model to score
tokens/phrases by information content and keep only the informative ones.
LLMLingua reports up to ~10–20x compression on benchmarks with under ~2%
degradation; Selective Context reports ~50% reduction with minimal BERTScore
loss.
3. **Question-aware compression (LongLLMLingua) — best for RAG.** Compresses the
retrieved context *conditioned on the user's question*: chunks/tokens
relevant to the query survive, irrelevant ones go. Also reorders documents to
fight position bias. On NaturalQuestions it reports up to ~21% performance
gain at ~4x fewer tokens, and large cost reductions on long-context tasks.
4. **Retrieval-based compression** — instead of stuffing a cap of the best chunks,
retrieve and then compress the least-relevant chunks out (a form of
top-k tightening combined with LLMLingua).
## Doing it in practice
- Compress the **static/context part** (retrieved chunks, history), never the
user's actual instruction or key facts — those must survive.
- Keep the *question* outside the compression so the compressor can be
question-aware (that's what separates LongLLMLingua from plain LLMLingua).
- **Tune the ratio, don't max it.** Performance plateaus then drops quickly —
going above ~10–20x generally hurts. Start at 2–4x for RAG and measure.
- Choose the compressor's speed/size tradeoff: LLMLingua-2 is a fast token-level
selector with nearly constant latency; plain LLMLingua is iterative (latency
grows with retained tokens). For low-latency serving prefer the constant-cost
variants (LLMLingua-2 class).
- **Measure quality before/after** with your real eval set. Compression should
preserve (or improve) faithfulness — if it drops, the ratio is too high or
you're pruning the wrong parts (signature/final-answer content).
## Cost example (why it matters)
A 2,000-token prompt at $3/1M input tokens over 2,000 req/mo is ~$12/mo raw; at
10x compression (200 tokens) the same traffic is ~$1.2/mo — a 10x cut in the
prompt-token line, and proportional latency reduction on input-bound requests.
(Latency is dominated by *generation* tokens, so the saving is on input tokens +
prefill, largest in RAG/agent workloads that send big contexts.)
## Pitfalls
- **Compressing your instruction/task text** — never compress the operative parts
of the prompt.
- **Over-compressing** — ratios above ~10x (sometimes 20x) degrade quickly; tune,
don't guess.
- **Skipping measurement** — shipping compression without a before/after eval is
how you silently lose quality then chase ghosts.
- **Using an expensive/slow compressor whose cost negates the savings.**
- **Ignoring latency of compression itself** — the compressor runs synchronously;
for user-facing low-latency RAG prefer a fast/constant-latency selector.
- **Remembering generation dominates latency** — compression helps prefill/input,
so expect the biggest wins in input-heavy (RAG, agent context) workloads.
## Verify
- Compress a sample of prompts and confirm the surviving text still contains all
facts needed to answer (or that answer quality on your eval set is preserved).
- Compare cost and prefill latency before/after on identical traffic.
- Confirm faithfulness metric (see `rag-evaluation-ragas`) does not regress at
your chosen ratio — a win is stable-or-better quality at lower cost.