# Observability Metrics & Dashboards

## Must-track metrics

### Per-model
| Metric | Why |
|--------|-----|
| `latency_p50` / `latency_p99` | Degradation early warning |
| `tokens_per_second` (output) | Throughput; models vary 10× |
| `cost_per_1k_tokens` | Spend attribution; detect billing changes |
| `error_rate` | Provider outages; quota limits |
| `ttft` (time-to-first-token) | Perceived latency for streaming users |

### Per-system
| Metric | Why |
|--------|-----|
| `total_tokens` / `total_cost` | Daily / weekly spend |
| `fallback_rate` | How often primary model fails |
| `retry_rate` | LLM flakiness; prompts needing repair |
| `unique_models_used` | Model proliferation |
| `session_count` / `reqs_per_session` | Usage patterns |

## Dashboard layout (Grafana example)

```
+----------------------------------------------------------+
|  LLM Observability Dashboard                  [24h ▼]      |
+----------------------------------------------------------+
| Total Cost: $42.18  | Total Tokens: 1.2M  | Errors: 0.3% |
+----------------------------------------------------------+
|                                                            |
|  [Cost per model — stacked bar]   [Latency p95 — line]    |
|                                                            |
|  [Error rate per provider — heatmap]                      |
|                                                            |
|  [Model usage distribution — pie]                          |
|                                                            |
|  [Recent slow traces — table: trace_id, model, latency]   |
+----------------------------------------------------------+
```

## Sampling strategy

Tracing every call is expensive (>50% overhead at scale). Sample:

- **Always:** errors, calls > $0.50, calls > 5s
- **Rate-based:** 10% of remaining calls (configurable)
- **Head-based:** sample at ingress so the full downstream trace is coherent

## Key design decisions

1. **Span hierarchy mirrors your pipeline** — agent.run → llm.call → embedding → tool.call → llm.call
2. **Don't log prompt bodies in traces** — they're huge and leak user data. Store separately.
3. **GenAI semantic conventions** — OpenTelemetry's `gen_ai.*` attributes. Use them; your observability tooling (Datadog, Grafana, Arize) auto-detects them.
4. **Cost attributes must be post-hoc** — you can't know tokens until the response arrives. Set them in the span after `end()` via `set_attribute`.

## Backend options

| Backend | Setup difficulty | Best for |
|---------|-----------------|----------|
| **LiteLLM built-in** | Zero (if already using LiteLLM) | Teams already on LiteLLM gateway |
| **OTel + Grafana Tempo** | Medium | Self-hosted, open-source trace store |
| **LangSmith / LangFuse** | Low | Prompt-centric debugging |
| **Weights & Biases** | Low | ML teams already tracking experiments |
| **Datadog / New Relic** | Low | Teams with existing APM contracts |