rag-evaluation-ragas
verifiedccdc5129-e19f-4505-8501-1c621919def3
Evaluate a RAG pipeline the way the RAGAS framework does — faithfulness, answer relevance, context precision/recall — and read each metric to know which component (retriever vs generator) is broken.
Metadata
Skill file
# Evaluating RAG with RAGAS-style Metrics
Use when you need to know *where* your RAG system is failing — retriever or
generator — and to turn "it's kind of wrong sometimes" into a number you can
watch in CI. The RAGAS framework (Es et al., 2023) defines reference-free metrics
that separate retrieval quality from generation quality. You don't need the RAGAS
package to use the ideas, but it's the fastest way to get going.
## The four core metrics
RAGAS decomposes a RAG answer into two stages — *what the retriever brought back*
(context) and *what the generator said* (answer) — and scores each:
1. **Faithfulness** — is the *answer* fully supported by the *retrieved context*?
For each claim in the answer, count how many can be inferred from the context.
`supported_claims / total_claims`. This is your **hallucination gate**: a low
score means the model is saying things the retrieved chunks don't back up.
(Estimated with an LLM judge that extracts claims and checks each against the
context.)
2. **Answer relevance** — does the answer actually address the *question*
(as opposed to being faithful but evasive or padded with irrelevant text)?
Generated by asking the LLM to reconstruct plausible questions from the answer
and measuring embedding similarity to the real question. Low score => the
model answered something adjacent or padded.
3. **Context precision** — of the retrieved chunks, *how much is actually useful*
to answer the question (does it penalize noise/irrelevant retrieved text)?
A low score with faithful answers means the retriever is pulling junk the
generator then dutifully ignores.
4. **Context recall** — does the retrieved context *contain everything needed* to
answer? Measured against an ideal/reference answer: can each sentence of the
reference be attributed to the retrieved context? A low score is the classic
"the answer is too short / I can only answer part of the question" retriever
failure (missing chunks, bad top-k, chunking that splits needed info).
## Reading the metrics: which component is broken
This mapping is the whole point — don't just chase one number:
- **Faithfulness low** -> generator problem: prompt lets the model free-associate,
temperature too high, not enough grounding instruction, or retrieved context is
itself contradictory/irrelevant so the model fills gaps. Fix the prompt and
grounding, not the retriever.
- **Answer relevance low** -> generator/prompt problem: model pads or paraphrases
the question instead of answering it.
- **Context precision low** -> retriever problem: too many irrelevant chunks in
the top-k (poor embedding, bad hybrid weighting, top-k too large). Shrink top-k,
improve retrieval.
- **Context recall low** -> retriever/index problem: the answer's needed facts
aren't in what was retrieved at all — missing chunks, bad chunking (semantic
boundaries split mid-answer), embedding mismatch. Fix index/chunking/retrieval.
In practice your prod dashboards lean on **faithfulness, context recall, and
answer relevance** — they give the best diagnostic separation.
## Grounding metrics in the RAGAS package
```python
from ragas import EvaluationDataset, SingleTurnSample, evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
sample = SingleTurnSample(
user_input="What was the Q3 revenue?",
retrieved_contexts=["...chunk 1...", "...chunk 2..."],
response="Q3 revenue was $4.2B, up 12%.",
reference="Q3 revenue was $4.2B driven by advertising.",
)
dataset = EvaluationDataset(samples=[sample])
result = evaluate(dataset=dataset, metrics=[faithfulness, answer_relevancy, context_recall, context_precision])
```
- Pass `retrieved_contexts` = exactly what your retriever returned, and
`reference` = the ideal/expected answer, so context recall is measurable.
- RAGAS uses embeddings for the similarity parts and an LLM-as-judge for claim
verification, so it needs an LLM and an embedding model configured — those cost
API tokens per eval run, keep your eval set modest (50–200 samples) and
cached.
## Building the eval set
- **Write 50–200 realistic questions** an actual user would ask, plus a
`reference` ideal answer for each. Quality of the set matters more than size.
- Include hard cases: questions that need *multiple* chunks, questions with
no answer in the corpus (test refusal, not hallucination), paraphrases, and
timestamp-sensitive questions.
- Keep the set **versioned** in your repo so CI diffs are meaningful — a
"golden set" you re-run on every change.
## Operationalizing it
- Run it in CI on every retrieval/prompt change and **fail the build if
faithfulness or context recall regresses** past a threshold you set by
measuring current performance first.
- Track per-metric trends over time; a creeping faithfulness drop is an early
hallucination warning before users complain.
- Use it to make *decisions*: e.g. switching chunking, bumping top-k, adding
hybrid search — each change earns a before/after metric read.
## Pitfalls
- **Looking at only one metric.** Answer relevance can look great while context
recall is silently starving the answer.
- **Forgetting `reference`**: without it, context recall can't be computed and
you blind yourself to the retriever gap.
- **Judge bias on one model.** Faithfulness estimated by the same model that
generates the answer can be self-favoring; score claims on a sample with a
second model or by hand until you trust the judge.
- **Tiny/noisy eval sets:** 5 questions will not move ~reliably; version a real
set and sample consistently.
- **Chasing the mean** — check the distribution: a 0.9 average faithfulness can
hide a hard slice that scores 0.4 (multi-hop or numeric questions).
## Verify
- Compute all four metrics on a small eval set and confirm each responds to a
deliberately broken component: degrade the retriever (fewer chunks) and see
context recall drop; loosen the prompt grounding and see faithfulness drop.
- Confirm the numbers are stable run-to-run on a fixed sample (judge variance
should be small if you lock seeds/models).
- Set a regression threshold in CI and prove it blocks a real regression.