## Speculative Decoding Reference

### Implementation flags (per framework)

**vLLM:**
```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --speculative-model meta-llama/Llama-3.2-1B-Instruct \
  --num-speculative-tokens 5 \
  --spec-decoding-acceptance-method "rejection_sampler" \
  --report-speculative-metrics
```

**SGLang:**
```bash
python -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 5 \
  --speculative-eagle-topk 8
```

### Draft model options
| Type | Pros | Cons |
|------|------|------|
| Separate small model | Simple, flexible | Lower acceptance, extra VRAM |
| EAGLE head | High acceptance (~70%+) | Needs trained head per model |
| Medusa heads | No separate model | Multiple heads = more VRAM |
| n-gram / lookup | Zero model cost | Low acceptance on free text |

### Parameter coupling
- `K` (num speculative tokens) and draft quality are coupled
- Start K=4-8; if acceptance < 50%, lower K or switch draft
- `efSearch`-style "acceptance method" varies by framework version

### When it helps
- Single-stream, low-concurrency interactive workloads
- Long generations (amortizes verification)
- Idle GPU capacity available

### When it hurts
- 100% GPU util on a single GPU (draft steals compute)
- Very short generations (one verify pass barely amortizes)
- KV-cache-memory-bound (draft + target must both fit)

### Measurement checklist
- [ ] Tokens/sec (inter-token latency) — NOT TTFT
- [ ] Acceptance rate reported and healthy (>50%)
- [ ] Output parity: greedy + sampling identical to baseline
- [ ] Same request mix (real prompt-length distribution)