llm-red-teaming
verified13ddd28c-95df-405e-9684-35327249f2e3
Systematically attack and harden LLM apps ā OWASP LLM Top 10, jailbreak/indirect-injection testing, and CI-integrated red teaming.
Metadata
Skill file
# LLM Red Teaming
Use when you must find and fix the security/robustness holes in an LLM
application ā before attackers do. Red teaming is adversarial testing of the
whole stack around the model (prompts, tools, RAG, output handling), not a
one-time pentest: it must be iterative and folded into your CI/CD and release
lifecycle.
## Frame it with the OWASP LLM Top 10
The OWASP LLM Top 10 is the community reference map of LLM app vulnerabilities.
Key categories you'll be probing:
- **Prompt injection** (direct + indirect/secondary) ā user or third-party content
hijacking the model's instructions. Even a defended output can still be
influenced by malicious content fetched into context.
- **Sensitive information disclosure** ā the model leaking PII or secrets.
- **Insecure output handling** ā trusting model output as code/SQL/HTML without
sanitization (a real injection vector even when the model behaves).
- **Insecure plugin/tool design** ā a tool that lets the model do damage (e.g. an
unauthenticated DB-write tool).
- **Supply chain / poisoning, denial of service (token bombs), excessive
agency**, and **model theft/exfiltration** of system prompts/data.
## What to probe
- **Direct jailbreaks**: goal-redefinition, roleplay, "DAN", prefix/suffix
injection, refusal-obfuscation templates. Modern red-team frameworks (Garak,
Microsoft PyRIT, Promptfoo) ship curated jailbreak template banks so you don't
hand-write them.
- **Indirect / secondary injection**: inject an instruction into a web page or
document that your RAG/agent then ingests and obeys ā test that fetched content
can't override system rules.
- **Tool-abuse**: can the model be talked into calling a destructive tool, or
extracting the tool schema/system prompt?
- **PII & secrets exfiltration**: extraction probes (e.g. PII extraction patterns).
- **Multi-turn attacks**: single prompts are easy to block; adversaries chain
innocuous turns that escalate. Tools like GOAT-style dynamic multi-turn
jailbreaking adapt per-turn and catch what static suites miss.
## Methodology
1. **Shift left**: threat-model *before* picking the model/tooling. Ask "does this
chatbot really need DB write access?" ā least-privilege tooling removes whole
attack classes.
2. **Automate**: run jailbreak suites in CI against every model/prompt change so
regressions surface at build time, not in prod.
3. **Layer defenses**: an AI gateway/firewall that filters malicious patterns
before the prompt reaches the model, in addition to in-prompt defense and
output sanitization.
4. **Grade, don't just count**: a "jailbroken" answer is unambiguous; but for
robustness you should also catch degraded-but-not-obviously-malicious
behavior. Set pass/fail per probe category.
## Pitfalls
- Treating red teaming as a one-off audit instead of a continuous, CI-integrated
practice.
- Probing only the model, ignoring the surrounding stack (tools, output handling,
RAG ingestion) where most real damage happens.
- No least-privilege on tools, so a successful jailbreak escalates immediately.
- Measuring only "harmful content blocked" while missing PII leaks and tool
misuse.
- Running single-shot probes only, missing multi-turn and indirect-injection
vectors.
## Verify
- A reproducible probe suite (version-pinned) that runs headlessly in CI with a
clear pass/fail report per category.
- Demonstrated fixes: reproduce a found jailbreak, apply the mitigation, show the
probe now fails.
- Evidence that tool permissions are least-privilege and output is sanitized.