# LLM Gateway Routing Strategies

## Strategy comparison

| Strategy | How it works | Best for |
|----------|-------------|----------|
| **Priority / fallback** | Ordered list: try A, then B, then C | Resilience; every setup needs this |
| **Cost-based** | Cheapest deployment by $/token | Optimizing spend |
| **Latency-based** | Fastest responding deployment | Interactive / streaming apps |
| **Least-busy** | Route to deployment with fewest active requests | High-throughput, even load |
| **Capability routing** | Simple Q → cheap model; hard Q → frontier | Maximizing quality-per-dollar |
| **Usage-based routing v2** | LiteLLM's built-in: weighted by cost + latency | General-purpose optimization |

## Cost-based routing with guardrails

```yaml
# LiteLLM: use cheapest, but cap spend
model_list:
  - model_name: tiered
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: ${OPENAI_API_KEY}
    model_info:
      max_budget: 500  # USD per month
  - model_name: tiered
    litellm_params:
      model: openai/gpt-4o
      api_key: ${OPENAI_API_KEY}
    model_info:
      max_budget: 200
```

## Capability routing pattern

```python
# Route by task difficulty (classifier or scoring heuristic)
def route_request(prompt: str) -> str:
    if is_simple(prompt):     # e.g., len < 100, no complex reasoning
        return "cheap-first"
    return "gpt-4o"           # Fall through to best model

# is_simple() can be:
# - Keyword heuristic: "summarize", "translate" → simple
# - Small classifier model (e.g., RouteLLM)
# - Token-count threshold
```

## Monitoring checklist

- [ ] **Cost per model per day** — alert on spikes
- [ ] **Fallback activation rate** — >5% means a provider problem
- [ ] **Per-request spend** — detect anomalous expensive calls
- [ ] **Virtual key budgets** — cut off individual teams without touching provider keys
- [ ] **Latency per model** — routing to a degraded backend?

## Provider key hygiene

- Keys live ONLY in gateway config (`${ENV_VAR}` references)
- Never in app code, never in client-side code
- Rotate quarterly; use scoped keys (not broad admin tokens)