## vLLM Serving Reference

### Essential flags
| Flag | Purpose | Typical |
|------|---------|---------|
| `--model` | HuggingFace repo or local path | required |
| `--tensor-parallel-size` | Shard across GPUs | 1-8 |
| `--max-num-seqs` | Max concurrent sequences | 128-512 |
| `--max-model-len` | Context window cap | model default |
| `--max-num-batched-tokens` | Tokens per decode step | 4096-16384 |
| `--gpu-memory-utilization` | VRAM fraction for KV cache | 0.9 |
| `--served-model-name` | Name exposed on /v1/models | custom |
| `--quantization` | AWQ/GPTQ/FP8 | optional |

### Endpoints (OpenAI-compatible)
```
GET  /v1/models
POST /v1/chat/completions
POST /v1/completions
GET  /health
```

### VRAM sizing formula
```
VRAM_needed = model_weights + KV_cache + overhead
KV_cache ≈ 2 * num_layers * num_kv_heads * head_dim * dtype_bytes
             * max_num_seqs * max_model_len
```
If weights alone exceed VRAM, you get OOMs — check before tuning batch.

### Throughput vs latency tuning
| Goal | Favor |
|------|-------|
| Raw throughput | High `--max-num-batched-tokens` |
| Interactive latency | High `--max-num-seqs` headroom, moderate batching |
| Both | Profile TTFT + tokens/sec + queue latency separately |

### Quantization options
```bash
# AWQ/GPTQ (4-bit)
--quantization awq --model TheBloke/Llama-2-7B-AWQ

# FP8 (Hopper GPUs)
--quantization fp8
```

### Monitoring
- Watch `nvidia-smi` for actual VRAM usage vs `--gpu-memory-utilization`
- Track TTFT, tokens/sec, and queued-request latency separately
- Verify no OOM under load; lower `max-num-seqs` if OOM

### Pitfalls
- Don't copy H100 flags to a single 24GB card
- Set `--max-model-len` or KV cache pool has no room for batch depth
- Always enable client-side streaming for chat UX