prompt-evaluation-regression
verified9050a28a-366f-4c89-ab23-13cee044b631
Treat prompts like code — regression-test them with promptfoo, version them, and A/B test against a golden dataset so prompt changes never silently regress quality.
Metadata
Skill file
# Prompt Evaluation and Regression Testing
Use when your LLM behavior depends on prompt text that people keep tweaking — and you
want to stop the "someone edited the prompt and now output is worse" failure mode.
Prompts are the most-changed, least-tested artifact in most LLM apps. Evaluating them
like code — with a golden dataset, assertions, and CI gates — converts prompt changes
from a gamble into a measured, reversible action.
## Why this matters
Prompt quality is *distributional*: a prompt that works on your 3 test examples can
fail on the 300th. LLM outputs are non-deterministic, so a single manual spot-check
is meaningless. The only reliable signal is: run the prompt over a diverse golden set
and compare aggregate scores to a stored baseline. If you don't have this, every
prompt edit is a silent regression risk.
## Build a golden dataset first
The eval is only as good as the dataset. Curate 50-200 examples spanning:
- **Real user inputs** (logged from production), not hand-written toy cases.
- **Edge cases and failure modes** you've hit (negations, ambiguity, unusual formats).
- **Difficulty spread** — easy, typical, and adversarial.
Annotate each with expected output or ground-truth for scoring. This is the expensive
part and the part that compounds: the same golden set serves eval, prompt A/B testing,
and future model-migration testing.
## promptfoo (the standard open-source tool)
promptfoo (open-source) defines evals as YAML. You specify providers (the models/APIs),
prompts (with `{{variables}}`), tests (input vars + assertions), and it runs the matrix
and scores pass rates:
```yaml
providers:
- id: openai:gpt-4o-mini
prompts:
- "Summarize the following: {{document}}"
tests:
- vars: { document: "..." }
assert:
- type: contains
value: "VAT"
- type: llm-rubric
value: "The summary mentions the key point"
```
Assertions can be deterministic (`contains`, `equals`, regex) or LLM-graded
(`llm-rubric`, `model-graded`) — use deterministic checks where possible (they're
cheap and stable), reserve LLM-as-judge for open-ended quality where a rubric is the
only practical scoring.
## The workflow
1. **Baseline**: run the current prompt over the golden set; store the pass rate and
per-test results.
2. **Change**: edit the prompt (or model, or temperature).
3. **Diff**: re-run and compare against baseline — which tests went green→red?
4. **Gate**: fail CI if aggregate score drops below a threshold, or if any
must-pass test regresses.
5. **Promote**: keep the change only if it's a net improvement, and record the new
baseline.
Wire step 4 into your CI (promptfoo has a `--junit`/CI mode) so prompt edits in a PR
must pass eval before merge.
## A/B testing in production
The golden-set eval is your *offline* gate. For changes that affect user-facing
quality in ways you can't label ahead of time (tone, helpfulness), ship both prompts
behind a flag, split traffic, and compare an online metric (thumbs up/down, task
completion, deflection rate) over enough traffic to be significant. Offline eval
catches regressions; online A/B catches preference differences.
## Pitfalls
- Testing on 3 hand-picked examples — you'll overfit and miss real regressions.
- Over-relying on LLM-as-judge for things a `contains`/regex can check — judges add
noise and cost.
- No baseline stored before the change — you can't diff against history.
- Not version-controlling prompts — a prompt string in code with no history is
un-revertable.
- Golden-set drift: your examples rot as your product changes; refresh them periodically.
## Verify
- Run promptfoo over your golden set; confirm you get a stable pass rate (re-run
3× to check variance — high variance means your set is too small or scoring too
weak).
- Make a deliberately worse prompt edit and confirm the CI gate catches it (test your
test).
- Confirm the golden set includes real production inputs, not just synthetic ones.Attached files
No attached files.