## RAG Evaluation Set Design

### What to include (50-200 questions minimum)

**Coverage dimensions:**
- Single-chunk answers (easy)
- Multi-chunk answers (hard -- tests recall across chunks)
- Paraphrased queries (tests embedding quality)
- Questions with NO answer in corpus (tests refusal)
- Numeric / date-specific questions (tests fact extraction)
- Ambiguous questions (tests clarification vs guessing)

### Golden reference format (JSONL)
```json
{"question": "What was Q3 revenue?",
 "answer": "Q3 revenue was $4.2B, up 12% YoY.",
 "contexts": ["chunk_1 text...", "chunk_2 text..."],
 "reference": "Q3 revenue was $4.2B driven by advertising growth."}
```

### Metric thresholds to set in CI
| Metric | Red (<fail) | Yellow (warn) | Green |
|--------|-------------|---------------|-------|
| Faithfulness | < 0.70 | 0.70-0.85 | >= 0.85 |
| Answer Relevance | < 0.65 | 0.65-0.80 | >= 0.80 |
| Context Recall | < 0.60 | 0.60-0.75 | >= 0.75 |
| Context Precision | < 0.60 | 0.60-0.75 | >= 0.75 |

### Cost management
- RAGAS uses LLM-as-judge: each sample costs API tokens
- Keep eval set 50-200 samples; cache results
- Run on PR changes to retrieval/prompt; not every commit
- Use a cheaper model for the judge (gpt-4o-mini) unless precision critical

### Stability
- Lock seeds + model versions; re-run on same sample set
- Judge variance < 0.02 is acceptable; > 0.05 is suspect
- Check distribution, not just mean: a 0.9 mean can hide 0.4 on hard slices