llm-guardrails-moderation
verified2f717372-f924-4e08-90d2-f8da0cb9aab4
Build a layered safety/guardrail system for LLM apps — Llama Guard classifiers, NeMo Guardrails rails, and input/output moderation — beyond basic prompt-injection defense.
Metadata
Skill file
# LLM Guardrails and Content Moderation
Use when you need to enforce a safety/content policy on LLM inputs and outputs —
block jailbreaks and harmful prompts before they hit the model, and filter harmful or
policy-violating responses before they reach users. Guardrails are the *layer* around
the model, distinct from the model's own alignment. Prompt-injection defense handles
one attack class; this is the broader moderation/safety stack.
## The core principle: layered, not single-point
No single mechanism stops all attacks or catches all policy violations — prompt
injection and jailbreaks are a cat-and-mouse game. The robust pattern is layers:
(1) input moderation, (2) the aligned model itself, (3) output moderation, (4)
continuous testing/red-teaming. Treat guardrails as a governance loop — define
policy, enforce, monitor incidents, refine — not a set-and-forget filter.
## Input vs output moderation
- **Input moderation** — classify the user message before the LLM sees it. Detects
jailbreaks, prompt injections, hate/self-harm/sexual/violent content, and
requests for illegal activity. Block (refuse) rather than pass through.
- **Output moderation** — classify the model's response before it's returned. Catches
the model being tricked into harmful content, leaking PII, or producing
policy-violating text. This is the *backstop* — assume the model will sometimes
produce bad output regardless of input filtering.
## Llama Guard (Meta)
A family of small LLMs (7B/8B, plus smaller variants) fine-tuned specifically as
input/output safety classifiers. It ships with a fixed safety-risk taxonomy
(violence, hate speech, sexual content, criminal planning, weapons, etc.) and is
designed to be prompted with the conversation + the category list, then answer
"safe"/"unsafe" with the violated category. Self-host it (e.g. via vLLM) for
zero-cost-per-call moderation, or use hosted moderation APIs (OpenAI moderation,
Azure content filtering/Prompt Shield, Google Vertex safety filters) which wrap
similar classifiers.
## NeMo Guardrails (NVIDIA)
An open-source toolkit for *programmable* rails — not just classification but
dialogue control. You write a Colang configuration (`.co` files) defining flows:
input rail (call a classifier, if unsafe → refuse and stop), output rail (fact-check,
moderation, PII redaction), and dialog rails (keep the bot on-topic, follow a scripted
flow, extract structured data). It integrates Llama Guard and other models as the
classifier behind its rails. Use it when you need *behavioral* control (topic limits,
conversational flow), not just content filtering.
## Latency and cost reality
Every guardrail layer adds a model call — input moderation + output moderation can
double your latency and add real cost (or GPU load if self-hosted). Optimizations:
- Self-host the classifier (Llama Guard 8B) so marginal cost ≈ GPU time, not per-call
API fees.
- Use smaller/faster classifiers for the common case, escalate to a bigger model only
on uncertain scores.
- Cache moderation decisions for repeated near-identical inputs.
- Run input and output moderation asynchronously where the UX allows.
## Pitfalls
- Input-only moderation with no output backstop — the model can still be induced to
produce harmful content.
- Using a single classifier as your only defense — classifiers have blind spots and
can be jailbroken themselves; layer them.
- Guardrails that are so aggressive they hurt usefulness (false-positive refusals
frustrate users and reduce the product's value).
- No monitoring — if you never log what the guardrails caught, you can't tune
precision/recall or detect new attack patterns.
## Verify
- Build a labeled test set of safe/unsafe inputs and outputs (include known
jailbreaks) and measure precision/recall of each layer.
- Confirm the full chain: jailbreak attempt → blocked before model; harmful model
output → filtered before user.
- Measure added latency/cost per request and confirm it fits your SLO and budget.Attached files
No attached files.