# RAG Evaluation Metrics Reference

## Retrieval metrics

| Metric | Formula | Meaning | Good for |
|--------|---------|---------|----------|
| recall@k | `|retrieved ∩ relevant| / |relevant|` | Did we find the right docs? | High-recall use cases |
| precision@k | `|retrieved ∩ relevant| / k` | Are the top docs right? | High-precision use cases |
| MRR | `mean(1/rank_of_first_relevant)` | How early is the first hit? | Single-answer questions |
| nDCG@k | `DCG / IDCG` | Ranked quality with graded relevance | Graded relevance sets |
| hit@k | `1 if any relevant in top-k` | Did we get at least one? | Coarse filter |

## Generation metrics

| Metric | Measures | How to compute |
|--------|----------|----------------|
| Faithfulness | Every claim supported by context | LLM judge or NLI entailment model |
| Answer relevance | Answer addresses the question | LLM judge scoring |
| Exact match | Answer == ground truth | String comparison (strict) |
| ROUGE / BLEU | n-gram overlap with reference | Token matching (fuzzy) |

## Building a golden eval set

### Composition guidelines

- **50–200 Q/A pairs** minimum for meaningful signal
- Each question has **known relevant document IDs** (not just an answer string)
- Cover **all categories**:
  - `easy`: direct lookup
  - `paraphrase`: different wording, same meaning
  - `cross-doc`: requires joining 2+ documents
  - `out-of-scope`: no relevant docs (should answer "don't know")
- **Version the set** — re-run on every pipeline change

### Anti-patterns to avoid

- ❌ Only easy queries (overfit, no adversarial signal)
- ❌ No out-of-scope cases (can't detect hallucination)
- ❌ Tuning chunk size on the eval set (leaks and misleads)
- ❌ Judging "did it answer" without checking retrieval quality

## Iteration loop

```
Change ONE lever → re-run harness → diff metric deltas
Levers: chunk size, overlap, embedding model, hybrid search,
        reranker, prompt template, generation model
```