speculative-decoding
verified7c55d5a5-8582-4d39-bac6-2d3905544225
Speed up LLM inference 2-3x with speculative decoding — draft+verify, acceptance rates, EAGLE/Medusa draft heads, and when it pays off vs not.
Metadata
Skill file
# Speculative Decoding for Faster LLM Inference
Use when your autoregressive LLM is latency-bound and you want faster tokens per
second WITHOUT changing the output distribution. Speculative decoding
accelerates generation while producing (statistically) the exact same tokens a
plain greedy/sampled decode would, so it is a safe drop-in optimization — no
quality tradeoff.
## The core idea: draft then verify
Autoregressive decoding generates one token at a time and each token needs a
full forward pass of the model, which is why decoding dominates cost. Speculative
decoding exploits that the big model can verify K candidate tokens in a SINGLE
parallel forward pass, at roughly the cost of one step.
1. **Draft phase** — a small, fast draft model proposes K candidate tokens (e.g. K=4-8)
for the next K positions.
2. **Verify phase** — the large target model, in one batched forward pass, computes
the probability of each proposed token at each position.
3. **Accept/reject (rejection sampling)** — a token is accepted when it matches what
the target would sample; at the first rejection, resample that position from the
target's corrected distribution and discard everything after it.
If the draft is "good enough", you emit up to K+1 tokens per expensive forward
pass instead of 1. Realistic speedups are **2-3x**; best case is K+1 tokens. The
output is statistically identical to non-speculative decoding — this is a
property of the rejection-sampling step, not an approximation.
## Key levers
- **K (number of speculative tokens)** — too large and you waste compute verifying
tokens that get rejected; too small and you don't amortize. Typical start: 4-8.
- **Draft model quality** — speedup is driven by acceptance rate. A draft too unlike
the target gives low acceptance and no win. Draft should be 10-100x cheaper but
same-family, so its predictions correlate with the target.
- **Hardware** — drafting and verification can run on separate GPUs to overlap
(draft the next batch while the target verifies the current one), at the cost of
more GPUs. On a single GPU the draft steals compute.
## Common implementations
- **vLLM**: `--speculative-model <draft>` with `--num-speculative-tokens K`. Supports
a separate HF draft model, or an **EAGLE** draft head. Set a metric like
`--spec-decoding-acceptance-method` (vLLM versions vary) and watch throughput.
- **SGLang**: enable speculative decoding with an EAGLE-3 draft head for notable
gains on single-stream and short-context workloads; also supports Medusa-style
heads.
- **EAGLE / EAGLE-2 / EAGLE-3** — draft heads trained on the target's own hidden
states, giving much higher acceptance than a separate smaller model.
- **Medusa** — adds parallel decoding heads on top of the target model, no separate
draft model needed.
## When it pays off (and when it doesn't)
- **Wins:** single-stream / low-concurrency interactive workloads, long generations,
batched-but-idle-GPU scenarios, when you have spare GPU capacity.
- **Doesn't help (or hurts):** already at 100% GPU util on a single GPU where the
draft competes for the same compute; very short generations where one verify pass
barely amortizes; KV-cache-memory-bound serving where you can't fit draft+target.
Scheduled-generation and speculative approaches assume idle SM capacity.
## Pitfalls
- Expecting K+1 **every** step — real acceptance is 40-80%, so budget for 2-3x not 5x.
- Not tuning K and the draft together; they are coupled.
- Measuring only time-to-first-token (TTFT) — speculative decoding mostly helps
**tokens-per-second (inter-token latency / time-per-output-token)**, not TTFT.
- Ignoring memory: draft model + target KV cache must both fit; on memory-starved
GPUs the win collapses.
- Assuming output quality changes — it shouldn't, but verify determinism/sampling
matches your baseline to catch bugs (a "wrong" spec decoder can silently change
outputs).
## Verify
- Benchmark tokens/sec and TTFT with and without spec decoding on the SAME request
mix (use your real prompt-length distribution, not a toy prompt).
- Confirm acceptance rate is healthy (`--report-speculative-metrics` in vLLM or the
equivalent): if acceptance < ~50%, tune K or switch draft.
- Assert output distribution parity: run greedy and sampling decodes with and
without speculation over an eval set and confirm identical/statistically
equivalent outputs.