ci-failure-triage
verified7dcf703e-c6b5-4e53-861c-65764ffc0161
Use when CI is red — read the right job/step/first-failing-line, classify env vs test vs build failures, reproduce locally, and avoid the blind re-run trap.
Metadata
Skill file
# CI Failure Triage
**Use when** CI is red and you need to find and fix the cause fast. The trap: blind re-runs and reading only the tail of the log. Follow the read order.
## The Read Order
```text
1. WHICH job failed? (lint? test? build? deploy?)
2. WHICH step failed? (within the job)
3. What's the FIRST failing line? (not the last)
4. CLASSIFY: env / test / build / lint / flaky
```
### Finding the first failure (not the tail)
```text
In a log, the LAST error is often a cascade. The FIRST error is the cause.
GitHub Actions: scroll to the first red "✖" or "Error:" line.
Bash pipelines: the first non-zero exit is the culprit.
```
## Classification
| Type | Signature | Fix approach |
|------|-----------|--------------|
| **Env failure** | "command not found", missing secret, wrong Python version | Recreate env locally |
| **Test failure** | `assert` / traceback / `FAILED` | Reproduce locally, fix code |
| **Build failure** | compile error, missing dep, linker error | Fix build config |
| **Lint failure** | `ruff`/`eslint`/`mypy` violations | Run linter locally |
| **Flaky** | Passes on re-run, random failure | flaky-test-triage skill |
## Reproduce Locally (The "Works on My Machine" Checklist)
```bash
# 1. Match the CI Python/node version exactly
python --version # vs CI's version
node --version
# 2. Match the dependency versions
pip freeze # vs CI's lockfile
# Better: use the lockfile
pip install -r requirements.lock
# 3. Run the EXACT command from CI
# (find it in the workflow yml, e.g. .github/workflows/ci.yml)
pytest tests/ -x --cov=src/ --cov-fail-under=80
# 4. Run the linter with the SAME config
ruff check src/ --select ALL
mypy src/ --strict
```
## Common Causes
### Missing secret / env var
```text
CI error: "Missing required environment variable: STRIPE_API_KEY"
Fix: add the secret in repo settings, or use a test mode that doesn't need it.
```
```yaml
# .github/workflows/ci.yml
env:
STRIPE_API_KEY: ${{ secrets.STRIPE_API_KEY }} # must exist in settings
```
### Version pin drift
```text
CI passes locally because your local deps differ.
Fix: pin exact versions in requirements.txt / package-lock.json,
and run CI in a fresh container (act or docker).
```
```bash
# Run GitHub Actions locally
# https://github.com/nektos/act
act pull_request # simulates the CI workflow locally
```
### Ordering / flakiness
```text
A test that passes alone but fails in the full suite = ordering issue.
Run the full suite locally in the SAME order as CI:
pytest tests/ -x # full run, not just the failing test
```
### Lint config mismatch
```text
Local ruff passes but CI ruff fails.
Fix: ensure the SAME ruff version and config (pyproject.toml) are used.
```
```bash
ruff --version # match CI's pinned version
ruff check src/ # same command as CI
```
## The Blind Re-Run Trap
```text
Re-running CI without investigating is the #1 anti-pattern.
- If it's a flaky test -> re-run MASKS a real bug
- If it's a real failure -> re-run wastes 10 minutes
- If it's an env issue -> re-run does nothing
ALWAYS read the log and classify BEFORE re-running.
```
## GitHub Actions Specifics
```bash
# Re-run a failed job
gh run rerun <run-id> --failed
# View the failing step's log
gh run view <run-id> --log-failed
# List recent failed runs
gh run list --status=failure
```
```yaml
# .github/workflows/ci.yml — add useful logging to future-proof
- name: Run tests
run: pytest tests/ -x -v --tb=short
# -v verbose + --tb=short gives a compact but complete traceback
```
## Guardrails
- **Never blind re-run as a fix.** Classify first, then decide if re-run is appropriate.
- **Read the FIRST error, not the last.** Cascades hide the root cause.
- **Match the CI environment exactly.** Version drift is the #1 "works on my machine" cause.
- **Don't comment out failing tests to "make CI green".** That's deleting the alarm, not fixing the fire.
- **Log the exact CI command** in the workflow so local repro is one copy-paste.
## Pitfalls
| Pitfall | Fix |
|---------|-----|
| Blind re-run as the "fix" | Read the log and classify BEFORE any re-run |
| Reading only the log tail | Scroll to the FIRST "Error:" or "✖" |
| Version mismatch (works locally) | Pin versions; use `act` to simulate CI |
| Commenting out failing tests | Fix the test or the code, never silence CI |
| Not checking secrets/env | Verify all `${{ secrets.* }}` exist in settings |
## Verify / Checklist
- [ ] Failing job, step, and first-failing-line identified
- [ ] Failure classified (env/test/build/lint/flaky)
- [ ] Reproduced locally with matching version + command
- [ ] Root cause identified (not just "re-run passed")
- [ ] Fix applied and CI green after push
- [ ] No tests commented out or disabledAttached files
No attached files.