llm-eval-harness-rag
verified0b7d4993-741a-4a19-a0bf-c417b1c828f3
Evaluate RAG quality ā build question-answer evals, compute retrieval + generation metrics (recall@k, faithfulness, answer relevance), and iterate.
Metadata
Skill file
# Building an Eval Harness for RAG
Use when you need evidence that your RAG/LLM app is *actually* good, not just
plausible ā to catch regressions, compare chunking/retrieval changes, and stop
shipping on vibes.
## Two halves: retrieval and generation
| side | measures | metric |
|------|----------|--------|
| retrieval | did we fetch the right docs? | recall@k, precision@k, MRR |
| generation | is the answer right and grounded? | faithfulness, answer relevance, exact/LLM-judged |
## Build ground-truth eval sets
- 50ā200 Q/A pairs where each question has known-relevant document ids.
- Cover easy and adversarial cases: paraphrase, cross-doc, out-of-scope (should
answer "don't know").
- Keep the set versioned; re-run on every retrieval/chunking change.
## Compute metrics
- **recall@k** = fraction of relevant docs present in top-k.
- **faithfulness** = every claim in the answer is supported by retrieved context
(judge with a second LLM or entailment model).
- **answer relevance** = the answer addresses the question (not just grounded).
`lm-eval-harness`-style tooling and the `weights-and-biases` logger let you
track these per experiment; at minimum, print a scored table per run.
## Iterate the levers
Chunk size, overlap, embedding model, hybrid search, reranker, prompt template,
model ā change one at a time and re-run the harness.
## Pitfalls
- Judging only "did it answer" while retrieval silently returns nothing useful.
- Eval set too small / non-adversarial ā easy to overfit.
- Not holding out: tuning on your eval set leaks and misleads.
- Ignoring the "don't know" case that prevents hallucinated wrong answers.
## Verify
- Re-run the harness after every pipeline change; diff metric deltas.
- Ship the eval set + a single command to rerun it with the harness.