embedding-fine-tuning
verified8a2cb9e0-2194-433c-91b9-12dd72bac406
Fine-tune an embedding model for your domain — contrastive training with sentence-transformers, hard-negative mining, and Matryoshka representation learning for truncatable vectors.
Metadata
Skill file
# Embedding Model Fine-Tuning
Use when an off-the-shelf embedding model underperforms on your domain — niche
jargon, unusual document structure, or a specialized similarity notion that
general-purpose models (trained mostly on web text) don't capture. Fine-tuning
teaches the model your domain's specific notion of "similar", which typically lifts
retrieval hit-rate far more than switching to a slightly larger embedding model.
## First: is fine-tuning even the answer?
Before training, confirm the failure is the *embedding model*, not your pipeline:
- Try a better off-the-shelf model first (MTEB leaderboard — e.g. top open models
in the `bge-*`, `e5-*`, `gte-*`, `nomic-embed-*` families). A 3-point MTEB gap is
often free to close by switching models.
- Check chunking and query rewriting — bad retrieval is frequently a chunking or
query-formulation problem, not an embedding-quality problem (see rag-query-rewriting).
- Fine-tuning pays off when you have *domain-labeled* similarity data (query→relevant
doc pairs) that no public model has seen.
## Contrastive training with sentence-transformers
The standard recipe trains the model to pull a query close to its positive document
and push it away from negatives. The default loss for retrieval is
`MultipleNegativesRankingLoss` — it treats the other examples *in the batch* as
negatives (in-batch negatives), which is why it needs no explicit negative labels:
```
from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer
from sentence_transformers.losses import MultipleNegativesRankingLoss
model = SentenceTransformer("BAAI/bge-small-en-v1.5")
loss = MultipleNegativesRankingLoss(model)
# train_dataset: (anchor=query, positive=relevant_doc) pairs
```
Data format: pairs of (query, relevant_document) with one relevant doc per query.
For harder training, add *hard negatives* — documents that are close to relevant but
wrong. Hard negatives are the single biggest accuracy lever; mine them from your
current model's top-50 retrieved-but-irrelevant results and retrain.
## Matryoshka Representation Learning (MRL)
MRL trains the model so that *truncating* the embedding to a smaller dimension
preserves most quality — the first 64 dims are good, the first 128 are better, the
full 768 are best. Benefits: store small vectors (cheaper, faster ANN search) and
truncate at query time for a latency/storage knob. In sentence-transformers, wrap
your loss in `MatryoshkaLoss` (which wraps `MultipleNegativesRankingLoss`); use the
`nomic-ai/nomic-embed-text-v1.5` model (Matryoshka-native) or truncate with
`truncate_dim`.
## Practical recipe
1. **Collect domain pairs** (query → relevant doc). You need at least a few hundred
to a few thousand high-quality pairs; quality >> quantity.
2. **Mine hard negatives** from your current model's mistakes.
3. **Train** with `MultipleNegativesRankingLoss` (optionally `MatryoshkaLoss`),
low learning rate (~2e-5), short epochs (2-5), early stopping on a held-out set.
4. **Evaluate on a retrieval task** — re-embed your corpus and measure hit-rate@k,
not just training loss (loss going down doesn't guarantee retrieval improves).
5. **Re-index** your vector store with the new model (embedding changes invalidate
old vectors — you must re-embed the whole corpus).
## Pitfalls
- Expecting an API model (e.g. OpenAI `text-embedding-*`) to be fine-tunable — most
hosted embedding APIs offer no fine-tuning; you need an open-weight model.
- Mixing different embedding models in one index (query model ≠ index model) —
vectors aren't comparable across models; always re-embed everything together.
- Skipping hard negatives — random negatives make the task too easy and the model
doesn't learn the hard distinctions that matter in production.
- Re-indexing forgetting: a new embedding model means re-embedding the entire
corpus and rebuilding the index, which is a real cost.
## Verify
- Build a held-out eval of 100+ query→relevant-doc annotations. Measure hit-rate@k
before and after fine-tuning, on the *same* index setup.
- Compare against a strong off-the-shelf model (bge/e5/gte) — fine-tuning should
beat it on your domain, not just beat your old baseline.
- If using MRL, verify truncation: hit-rate at dim=64/128 vs full dim, and confirm
the accuracy-vs-size tradeoff is acceptable.Attached files
No attached files.