# GRPO Hyperparameter Guide

## Quick reference

| Parameter | Typical range | Start here | Notes |
|-----------|--------------|------------|-------|
| `num_generations` (G) | 4–64 | 8 | More = better baseline, more cost |
| `learning_rate` | 1e-7 – 5e-6 | 1e-6 | ~10-100× lower than SFT lr |
| `beta` (KL coef) | 0.01–0.1 | 0.04 | Higher = stay closer to ref model |
| `max_completion_length` | 256–2048 | 512 | Must cover CoT + final answer |
| `lora_alpha` | 16–64 | 32 | 2× r is common |
| `epochs` | 1–3 | 1 | RL converges fast on small datasets |

## Tuning KL divergence

Watch `kl_divergence` during training:
```
kl_divergence explodes  → reward is gameable, increase beta
kl_divergence near zero → no learning, decrease beta or increase lr
kl_divergence stable    → healthy training
```

## Reward component monitoring

Log each reward component separately:

| Signal | Meaning |
|--------|---------|
| Format reward ↑, accuracy flat | Model learned phrasing, not reasoning |
| Both ↑ | Healthy training |
| Format reward flat, accuracy ↑ | Rare — model getting smarter within loose format |
| Both flat | KL too tight, lr too low, or reward too sparse |

## Hardware estimates

| Model size | Method | GPUs needed | VRAM per GPU |
|-----------|--------|-------------|-------------|
| 0.5B | GRPO + LoRA | 1× A100-40GB | ~20 GB |
| 7B | QLoRA + GRPO | 1× A100-80GB | ~60 GB |
| 7B | LoRA + GRPO | 2× A100-80GB | ~80 GB |
| 70B | QLoRA + GRPO | 2× A100-80GB | ~140 GB total |

## When to stop training

- Reward stops improving for 20% of total planned steps
- KL divergence exceeds 0.05 and keeps climbing
- Generated outputs become repetitive (mode collapse)
- Held-out eval set accuracy stops improving (early stopping signal)