semantic-router-intent
verified7e92019c-09d4-4210-a462-3d1adf9757a6
Build a semantic router — a fast embedding-based decision layer that classifies user intent and routes to the right tool, pipeline, model, or guardrail before any slow LLM call.
Metadata
Skill file
# Semantic Routing for LLMs and Agents
Use when you want to decide the path for an incoming request (which tool, which
sub-agent, which model, which guardrail to run) in ~100ms with deterministic,
explainable logic — instead of spending 500-2000ms and a token-cost on an LLM call
to make the same routing decision.
Semantic routing compares the *meaning* of a query against a set of example
utterances using vector embeddings and cosine similarity, then picks the best
route. It is a decision layer: classifier first, expensive generation after.
## When to use a semantic router
- **Tool/endpoint selection** for agents — hand the agent only the tool subset
relevant to the intent (fewer tools in schema = fewer mistakes).
- **Routing to different models/pipelines** — e.g. a cheap small model for greetings
and chit-chat, a RAG pipeline for factual questions, an expensive strong model for
complex reasoning.
- **Safety/guardrail gates** — flag jailbreak or unsafe intents (a common documented
use is a route whose utterances are known jailbreak phrasings) and run a
deterministic block *before* invoking the LLM.
- **Pre-classification** for downstream workflows (ticket routing, intent tags).
The win: cost and latency, plus determinism — the routing result is reproducible in
a way raw LLM "decide" is not.
## How to build one
The open-source `semantic-router` library (Aurelio Labs) is the common starting
point. Core pieces:
1. **Routes** — each `Route(name, utterances=[...])` has a name and a handful of
example phrases that express its intent. Good utterances are the single biggest
quality lever.
2. **Encoder** — a small, fast embedding model: `text-embedding-3-small` (OpenAI) or
self-hosted `bge-small-en`. Local encoders need the `[local]` extra; many require
an API key env var.
3. **Index** — vector store for route vectors: local in-memory for small route sets,
Postgres (pgvector) under ~1M route vectors, Qdrant/Pinecone beyond.
4. **Router** — embed the incoming query, find the nearest route vector, and if the
similarity clears the threshold, return that route (with score); otherwise return
a fallback/likely-default.
Rolling your own is fine for a handful of routes: embed each utterance once, then
store per-route max similarity or centroid, and cosine-scan at query time.
## Key levers and gotchas
- **Threshold tuning** — a too-low threshold misroutes; too-high over-misses.
Tune per route on a labeled dev set, and design a sensible default/fallback route.
- **Threshold per route**, not global — some intents are tightly scoped (high
threshold), others broad (low).
- **Latency realism** — a single index query ≈ 100ms; don't over-embed.
- **Route collapse** — too few, too similar utterances make routes indistinguishable.
Add more diverse phrasings, or split.
- **Encoder consistency** — use the same encoder for utterances and queries; mixing
model families breaks similarity.
- **Hybrid (dense+sparse)** — `semantic-router[hybrid]` adds sparse vectors; helps
for exact-phrase intents (emails, IDs, product names).
## Pitfalls
- Using an LLM call *and* paying its latency to make a decision a 100ms vector
lookup can make.
- Believing the router is judgment — it's a classifier; keep a fallback path for
low-confidence and out-of-distribution queries.
- No labeled dev set, so you can't measure precision/recall of routing decisions.
- Forgetting key-only routes or connectors (some integrations need API keys).
## Verify
- Build a small labeled set of queries per route; report precision/recall at your
chosen thresholds.
- Measure end-to-end latency with vs without the router (expect ~100ms vs an LLM call).
- Assert adversarial/jailbreak phrasings hit their safety route with high recall.
- Confirm fallback behavior is graceful when no route clears threshold.