llm-as-judge-evaluation
verifieddd4e31e9-452d-4d9f-86af-f2c0b712a797
Use an LLM as an automated evaluator for outputs — build the rubric, control position/length/self-preference bias, and validate against humans.
Metadata
Skill file
# LLM-as-a-Judge Evaluation
Use when you need to grade model outputs at scale — answer correctness,
faithfulness to a source, style, or agent steps — where hand-labeling is too
slow and you don't have a ground-truth reference to diff against. A judge LLM
scores or chooses between outputs using a prompt you design.
It is a *reference-free* metric (no golden answer required), which is what makes
it powerful and also what makes it prone to silent bias.
## When to use it (and when not to)
- Great for: relative grading (A vs B), rubric scoring of qualitative axes
(helpfulness, clarity, faithfulness), screening large eval sets where humans
spot-check.
- Avoid as the sole oracle for: fact precision (prefer reference-grounded checks
or exact-match on factual subsets), anything with legal/medical downstream
consequences where a biased judge is unacceptable.
- Always validate a judge against a human-labeled sample before trusting it.
Pick a judge model notably **capable and preferably different** from the one
being evaluated — using the same model to judge its own output invites
self-preference bias.
## Build a rubric, not vibes
- Give the judge a small number of concrete, mutually exclusive scoring levels
(e.g. 1–5 with a written descriptor **per level**).
- Define what "good" and "bad" look like with examples in the prompt.
- Score one criterion per judge call rather than one "overall" number that
mixes axes.
- Use a structured-output/JSON format so results are machine-parseable, and pin
a low temperature (0) for reproducibility.
## Known biases and how to counter them
- **Position bias**: judges tend to prefer the first (or sometimes last)
presented answer. Mitigate by running the comparison twice with the two
answers swapped and keeping only agreeing verdicts, or grading each candidate
independently on the rubric instead of head-to-head.
- **Length bias**: judges systematically prefer longer, verboser responses —
it's a known artifact even when length has nothing to do with quality. Reward
*conciseness* explicitly in the rubric; watch for the judge defaulting to the
bloated answer.
- **Self-preference**: a judge favors text that looks like its own style.
Mitigate with a different judge family, multiple judges, or grading on rubrics
divorced from style.
- **Verbosity / sycophancy in answers**: judge can reward hedging or
agreeableness. Anchor the rubric on facts and content coverage.
## G-Eval style scoring
The G-Eval approach asks the LLM to emit a chain-of-thought why-pass under a
rubric, then a bounded score (e.g. 0–6). It correlates better with humans than
plain numeric prompts, at the cost of more tokens and occasional rubric drift —
paragraph-length scoring rationales need their own validation.
## Validate before production
- Label 100–300 samples by hand; compute judge-vs-human agreement (Cohen's
kappa / accuracy).
- If agreement is poor, iterate on the rubric, not just the judge model.
- Periodically re-check: judges drift as the judge model is updated or the
population of outputs shifts.
## Pitfalls
- Trusting a judge never validated against humans.
- A single "overall" score collapsing several distinct failure modes.
- Caching judge outputs as if they were ground truth.
- Ignoring the judge model's own cost/latency when scoring large batches.
## Verify
- Run your judge on a held-out human-labeled set and report kappa/accuracy.
- Demonstrate that the judge catches an injected, obviously-wrong answer
(sanity positive control) and passes a clearly-correct one (negative control).
- Confirm swapping answer order changes <X% of verdicts.