llm-observability-tracing
verified546cc0f8-fc94-4b2a-88a8-47e473fef4f7
Instrument LLM apps and agents for production — OpenTelemetry GenAI semantic conventions, tracing spans for calls/tools/agents, token/cost metrics, and evals.
Metadata
Skill file
# LLM Observability & Tracing
Use when your LLM/agent app is in (or near) production and you can't tell *which
model did what, at what cost, and when a trace breaks*. General APM shows you
latency/errors; LLM observability adds the model-specific context: prompts, token
usage, tool calls, agent reasoning, and evaluation signals.
## Use OpenTelemetry GenAI semantic conventions
The single most valuable move is to emit telemetry that follows the **OpenTelemetry
GenAI semantic conventions** — a CNCF-standard schema of `gen_ai.*` span/metric
attributes for GenAI operations. Because it's an open standard, any compliant backend
(Langfuse, MLflow, Datadog, Greptime, Tempo/Jaeger, etc.) can ingest it, and you can
**switch vendors without re-instrumenting**. This decouples your instrumentation from
any single observability product.
Common attributes the convention standardizes:
| concern | attribute(s) |
|---------|--------------|
| operation type | `gen_ai.operation.name` (e.g. `chat`, `embeddings`) |
| model + provider | `gen_ai.request.model`, `gen_ai.provider.name` |
| token usage | `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` |
| input/output | `gen_ai.input.messages`, `gen_ai.output.messages` (opt-in content) |
| span kind | a `chat` span for each LLM call, `execute_tool` for each tool invocation |
If you use an SDK-backed tool (OpenAI, Anthropic, LangChain, etc.), instrumentation
can be near-free — e.g. `OpenAIInstrumentor().instrument()` produces
semconv-compliant spans with no manual span creation.
## Trace what matters
Build a span tree that mirrors the agent's actual work, not just the final call:
- **Top-level span** = the agent/request (`invoke_agent`).
- **Child `chat` spans** = each LLM call, with model, provider, and token counts.
- **Child `execute_tool` spans** = each tool/function/MCP invocation, with tool name
and arguments.
- For MCP, the agent-side and MCP-server-side traces were historically disconnected —
the MCP semantic conventions aim to fix that so a single trace spans both sides.
Use them or explicitly correlate the two sides yourself.
This is what makes an outage diagnosable in minutes: "which step failed, which model,
which token burn, where the context window got blown."
## Metrics and evals in the loop
- **Cost per request** derived from token counts and the model's price table —
essential for catching runaway spend.
- **Latency by model/provider** and **error/fallback rates**.
- **Embed evaluations** as first-class telemetry: prompt + response + a judge score
(see the LLM-judge skill) attached to the span/trace so quality trends line up with
the raw calls that produced them.
## Practical wiring (pseudo-code)
```python
from opentelemetry import trace
tracer = trace.get_tracer("my-agent")
with tracer.start_as_current_span("invoke_agent") as root:
root.set_attribute("gen_ai.operation.name", "agent")
# ... call the model; inside it a child span sets:
# gen_ai.request.model, gen_ai.provider.name
# gen_ai.usage.input_tokens / output_tokens
# in the tool step, a child span sets gen_ai.operation.name="execute_tool"
```
Ship them via OTLP to a backend that speaks the convention (Langfuse's `/api/public/otel`
endpoint, MLflow, Greptime, Datadog v1.37+, etc.). A robust stack keeps all three
signal types (logs, metrics, traces) in storage you can query and correlate.
## Pitfalls
- **Not following semconv** → vendor lock-in; migrating to another backend means
re-instrumenting everything. Standardize on `gen_ai.*` from day one.
- **Recording full prompts/customer content by default** → PII/security exposure.
Make content capture **opt-in** and redact secrets from tool payloads.
- **Token counts missing** → cost stays a mystery. Always record usage per call.
- **One opaque span per request** → useless for agent debugging; you cannot tell which
step broke. Create child spans per LLM call and per tool call.
- **Ignoring eval telemetry** → you see cost/latency but not quality degradation until
users complain. Attach judge scores to traces.
## Verify
- A failed run produces a trace where you can pinpoint the broken step, its model,
and its token/cost impact in < 5 minutes.
- Cost-per-request and cost-by-model dashboards reflect reality from the token
counters.
- Switching to a second compliant backend requires *no* code changes (instrumentation
is portable).
- Content/PII capture is confirmed off (or redacted) before production.