Metadata

Skill ID

c8a06c12-6d4e-497c-9cd5-f9f4f37897b7

Version

1

Owner

387274b7-2891-478b-81b8-e11d5adb9319

Tags

constrained-decodingstructured-outputxgrammaroutlinesgrammarjson-schemallminferencegbnf

Signature

verified

Integrity

OK

Content hash

7d46036c7472187b4e36cc89884e2cc4b769346c95903bae1bf9c1ba2f803802

Created

2026-08-15T03:21:29Z

Skill file

# Constrained Decoding: Grammar-Enforced Structured Output

Use when prompting alone ("output valid JSON") is not reliable enough — you need
*guaranteed* structural correctness at the token level, without retries or repair.
Constrained decoding modifies the sampling process to mask tokens that would violate
a grammar, so the model *cannot* produce structurally invalid output. The guarantee
is format-only (not semantic correctness), but format reliability often determines
whether an agent pipeline fails silently or succeeds.

This is distinct from prompt-level "JSON mode" (which only forces valid JSON syntax
but not a specific schema) and from structured-output prompting (which instructs for
a schema but doesn't enforce it). Constrained decoding *masks logits* so invalid
tokens cannot be sampled.

## The three approaches

**1. FSM / regex (Outlines, Guidance).** Compile a JSON schema or regex into a
finite-state machine, then at each decoding step, mask tokens whose continuation
would leave the FSM. Simple, reliable, but the FSM construction cost grows with
schema complexity. Best for static schemas known ahead of time — avoid for
dynamically-generated schemas where compilation latency adds up.

**2. CFG grammars (XGrammar, GBNF, llguidance).** Compile schemas to context-free
grammars rather than regex. XGrammar (from MLC, Nov 2024) splits the grammar into a
precomputed "context-independent" mask and a lightweight context-dependent check
overlapping with GPU execution. Reports per-token overhead under 40µs, up to 14×
faster than earlier approaches on JSON-schema generation and up to 80× on CFG-guided
generation. GBNF is llama.cpp's grammar format, widely supported for local models.

**3. Provider strict APIs.** OpenAI "structured outputs", Anthropic structured
outputs, Gemini response_schema, and AWS Bedrock structured outputs all compile your
schema to a grammar server-side and guarantee conformance. These are the simplest
correct option when calling a hosted API — use them unless you have a specific reason
not to.

## The backends for self-hosted inference

- **vLLM**: `guided_json`, `guided_grammar`, `guided_regex` parameters. Auto-selects
  between XGrammar (default) and Outlines backends depending on constraint type.
- **SGLang**: XGrammar (default), Outlines, or llguidance as grammar backend.
- **llama.cpp**: GBNF grammars with automatic JSON Schema → GBNF conversion.

## Format guarantee ≠ semantic guarantee

This is where teams develop false confidence. A constrained-decoded response is
structurally valid — it will parse. It is not guaranteed to be *correct*, consistent,
or follow business rules. The JSON field `"age": -1` is perfectly valid JSON.
Always validate semantics upstream of the constraint (with tests, a checker, or
LLM-as-judge) — the grammar ensures you can *run* the validator without parse
errors.

## Performance reality

Grammar compilation overhead matters only when schemas change per-request. For a
finite set of output schemas (e.g. 5 API response shapes), precompile once. XGrammar
handles dynamic schemas efficiently; Outlines and llama.cpp GBNF are better for
static schemas. The runtime per-token masking cost is typically negligible — in
some configurations, a grammar can *reduce* token count and improve throughput by
eliminating the model's need to "think about format."

## Pitfalls

- Treating valid output as correct output; structural validity is not task accuracy.
- Dynamically-generated schemas with Outlines (slow FSM compilation) — use XGrammar
  or Guidance instead.
- Over-constraining: making the schema too rigid so the model can't express
  reasonable answers.
- Using constrained decoding when the model is too small to follow the task at all —
  no grammar fixes a model that can't do the work.

## Verify

- With your schema, sample 50-100 outputs and confirm 100% structural parse rate
  (no unparseable outputs).
- Confirm semantic correctness on a separate held-out set — format is 100%, but what
  fraction of outputs are *right*?
- Measure per-token overhead: compare tokens/sec with and without the constraint.

Attached files

No attached files.