observability-logs-metrics-traces
verified61147ac5-39ab-49a1-b2f2-ee30b3357213
Instrument production services ā structured logs, RED/USE metrics, and traces so outages are diagnosable in minutes.
Metadata
Skill file
# Observability: Logs + Metrics + Traces
Use when you run something in production and want to answer "what just broke,
for whom, and why" in minutes instead of digging through unstructured text.
## Three pillars, each has a job
| pillar | answers | tooling |
|--------|---------|---------|
| logs | what happened, in detail | structured JSON logs |
| metrics | is it healthy, is it getting worse | counters/gauges (Prometheus) |
| traces | which path is slow, end-to-end | OpenTelemetry spans |
## Log structured, searchable lines
- One JSON object per line: `ts`, `level`, `service`, `trace_id`, `msg`, fields.
- Never log secrets/tokens/keys. Redact PII at the source.
- Correlate with a `request_id`/`trace_id` so a request spans services.
## Pick metrics by what you act on
- **RED** (request rate, errors, duration) for services.
- **USE** (utilization, saturation, errors) for resources.
Use error *budgets*: alert on SLO breach, not on every blip.
## Trace the slow path
- Instrument your HTTP/DB/LLM calls with spans; capture duration + status.
- Look for spans that dominate p95 latency ā that's the actual slowness.
## Pitfalls
- Logging *everything* and drowning (no structured levels / sampling).
- Alert fatigue: alerting on every spike instead of SLOs.
- Metrics that nobody can act on, or dashboards with no owner.
- No correlation ID, so a single user error can't be traced across services.
## Verify
- Inject a test error/tear a dependency and confirm logs+metrics+traces all
show it within minutes.
- Confirm a new deploy is rollback-decidable from dashboards alone.