rag-query-rewriting
verifiedd316810e-0a60-41cd-91ea-1e447cb1e8cd
Fix the most common RAG failure — bad retrieval because the user's raw question doesn't match any chunk — with HyDE, multi-query, step-back prompting, and query decomposition.
Metadata
Skill file
# RAG Query Rewriting
Use when your RAG system isn't finding relevant chunks — the user asks "how do I set
up the payment processor for EU VAT?" and retrieval returns generic payment docs with
no VAT signal. The raw user question and the stored chunk language live in different
semantic spaces. Query transformation rewrites, expands, or abstracts the query to
close that gap, and is often the single highest-ROI fix for retrieval quality.
## Three transformation strategies (stackable)
**1. Multi-query.** Generate 3-5 alternative phrasings of the same question — change
structure, synonyms, specificity — then retrieve against each variant in parallel,
deduplicate (Reciprocal Rank Fusion is standard), and pass the fused result to
generation. Best when: the domain has multiple correct ways to express the same
thing, or users phrase questions inconsistently.
**2. HyDE (Hypothetical Document Embeddings).** Instead of embedding the query,
prompt the LLM to write a *hypothetical answer* to the question, then embed that
hypothetical document and use it as the search vector. A made-up answer is
semantically closer to a real answer than the bare question is, so retrieval finds
better chunks. Gao et al. (2023) introduced this; it is the go-to when your
documents are dense/factual and queries are short.
**3. Step-back prompting.** When the user asks a specific question that no single
chunk directly answers, first abstract to the *general principle* ("how do payment
processors handle regional tax?" instead of "EU VAT for Stripe?"), retrieve broad
context, then answer the specific from the general. Zheng et al. (2024) showed this
is especially effective for questions requiring inference from general rules. Best
when: the answer depends on principles that are spread across multiple chunks.
**Plus: Query decomposition.** For complex multi-hop questions ("which vendor has
the fastest delivery and also supports carbon offsets?"), decompose into two
sub-queries, retrieve for each independently, then synthesize. This is a separate
technique — you can decompose *and* rewrite each sub-query with HyDE.
## Reciprocal Rank Fusion (RRF) for multi-query dedup
```
score(chunk) = sum over retrieved-lists: 1 / (k + rank_in_that_list)
```
A chunk appearing in the top 3 of two different query-variant result sets gets a
strong fused rank. k is typically 60. Faster and simpler than training a scorer and
works well in practice.
## When *not* to rewrite
- The user's query exactly matches your chunk language — rewriting adds noise.
- Latency-sensitive streaming: each rewrite is an extra LLM call, which can add
200-500ms. For low-latency paths, test whether rewriting helps enough to justify it.
- Weak embedding model: garbage in, garbage out — no rewrite fixes a model that
can't distinguish relevance at all.
## Pitfalls
- HyDE hallucinating a wrong hypothetical answer that pulls entirely wrong chunks —
the hypothetical document guides retrieval, so if it's wrong, everything is wrong.
- Multi-query + retrieval for every variant without dedup → duplicated chunks and
bloated context.
- Step-back generalizing too far — "how do payment processors work?" is too generic.
- Not measuring: always A/B test raw-query retrieval vs rewritten retrieval on a
labeled set — intuition about which transform helps is often wrong.
## Verify
- Pick 30-50 real user questions with known correct chunks (annotation). Measure
hit-rate@k (does at least one correct chunk appear in top-k?) before and after
each transformation.
- Try HyDE first (it's the cheapest in terms of engineering); if hit-rate improves,
add multi-query on hard cases.
- Check that the rewritten queries actually produce *different* retrieved sets — if
the results look identical, the transform isn't doing work and you're burning
tokens for nothing.Attached files
No attached files.