knowledge-distillation-llm
verifiedfb7b883e-c265-4ab3-b8ab-7425a6e25ffe
Distill a large teacher model into a smaller, cheaper student — logit distillation, synthetic-data distillation, and the scaling laws that decide when it's worth it.
Metadata
Skill file
# Knowledge Distillation for LLMs
Use when you want the quality of a frontier model at a fraction of the cost/latency —
distilling a large *teacher* into a small *student* that you self-host. Distillation
is the standard path to a cheap, fast, task-specialized model, and it is how many
open "mini" models (e.g. DeepSeek-R1-Distill variants) are produced.
## The two main families
**Logit (response) distillation.** Train the student to match the teacher's output
probability distribution — its *soft labels* — not just the correct answer. Soft
labels carry the teacher's uncertainty and class-similarity structure, so the student
learns richer signal than one-hot labels. Key technique: a temperature parameter T
softens the distribution; the student is trained with a KL-divergence loss against
the teacher's soft targets (optionally blended with the hard-label cross-entropy
loss). For LLMs this typically means training on the teacher's full next-token
distribution, which requires the teacher to expose logits (so it works with
self-hosted teachers; most closed APIs only expose sampled text).
**Synthetic-data (sequence) distillation.** Prompt the teacher to generate
high-quality (input → output) pairs — often with reasoning traces or rationales —
then supervised-fine-tune the student on those pairs. This is the workhorse for
closed teachers (GPT-4, o1, Claude) where you only get text back. Classic examples:
Alpaca and Vicuna (instruction data distilled from larger models). Rationale
distillation — training the student on the teacher's chain-of-thought — is the main
way to transfer *reasoning* ability.
## Choosing the recipe
- Teacher exposes logits + you have compute: logit distillation is the most
information-dense option.
- Teacher is a closed API: synthetic-data distillation; generate diverse prompts,
collect (question, reasoning, answer) triples, fine-tune the student.
- Transferring reasoning: distill the *reasoning trace*, not just the final answer.
## Scaling laws: distillation isn't always the answer
A 2025 study (Busbridge et al., "Distillation Scaling Laws", Apple/Oxford) showed the
compute-optimal choice between distilling from a big teacher vs training a small model
from scratch depends on teacher size, student size, and compute budget — in some
regimes training a smaller model directly is *more* compute-efficient. Distillation
pays off most when: data is limited, the teacher is much stronger than the student,
or you're replicating general-purpose knowledge. Always check whether a
better-tuned small model or more data beats distillation before committing.
## Data and eval hygiene
- **Diversity in prompts matters** — narrow synthetic data produces narrow students.
Sample broadly across task types and difficulty.
- **Guard against model collapse**: recursively training on synthetic data can
degrade quality over generations; keep some human/real data in the mix and don't
distill from a distillate indefinitely.
- **Evaluate out-of-distribution**: distilled students often overfit the teacher's
distribution. Test on held-out tasks/domains and check calibration, not just
benchmark scores.
- **Licensing/terms**: verify the teacher's license and ToS allow distillation and
derivative-model publication.
## Pitfalls
- Expecting the student to match the teacher on everything — pick a clear target
capability and accept the tradeoff.
- Distilling answers without reasoning traces when you need the student to *reason*.
- Low-prompt-diversity synthetic data → a brittle student that memorizes.
- Skipping OOD eval and discovering the student only works on training-like inputs.
## Verify
- Fine-tune a small student (e.g. 1-8B) on your task and compare to the teacher on a
held-out eval set: accuracy, latency, and cost/token.
- If logit-distilling, verify the teacher's logits are accessible and your loss
(KL + temperature) is actually decreasing.
- Check OOD performance and calibration (confidence vs correctness) on a domain the
synthetic data didn't cover.Attached files
No attached files.