fine-tuning-data-curation
verified1f118ba3-6a38-4520-9fc6-83add5660d62
Build a high-quality fine-tuning dataset — near-deduplication (MinHash/semantic), quality filtering, benchmark decontamination, and data mixing — so your model learns signal, not memorized noise.
Metadata
Skill file
# Fine-Tuning Data Curation
Use when you're about to fine-tune a model and want the training data to actually
help. Data quality dominates all other fine-tuning decisions — the single most
reproducible result in LLM post-training is that a smaller, cleaner dataset beats a
larger, noisier one. Curating data means deduplication, quality filtering,
decontamination, and deliberate mixing. (For *generating* synthetic data, see the
synthetic-data skill; this is about *cleaning and selecting*.)
## 1. Deduplication
Duplicate examples waste compute and, worse, teach the model to repeat itself and
overfit a narrow distribution. Two levels:
- **Exact/near-exact dedup** — hash-based. MinHash (via `datasketch`) estimates
Jaccard similarity cheaply at scale; Locality-Sensitive Hashing (LSH) buckets
near-identical shingles together. For text, shingle on n-grams (n=5-13 words).
This catches verbatim and lightly-edited duplicates.
- **Semantic dedup** — embeddings + clustering (e.g. `semdedup`) to find
*paraphrase* duplicates that MinHash misses. More expensive; use only when exact
dedup isn't enough and you have embedding capacity.
Rule of thumb: exact dedup first (it's cheap and high-recall), then decide if
semantic dedup is worth the cost for your scale.
## 2. Quality filtering
Remove low-quality examples that would teach bad behavior. Common filters:
- **Perplexity/likelihood** — the model's own (or a reference model's) probability
of the text; very-low-perplexity text is often boilerplate/spam, very-high is
gibberish. Filter both tails.
- **Heuristics** — length floors/ceilings, language detection, ratio of
punctuation/repetition (repetitive text is a red flag), presence of required
structure.
- **Classifier / LLM-as-judge** — score a sample for relevance-to-task and quality;
drop the bottom band. Expensive but precise; sample a subset to tune your cheaper
filters, then apply cheap filters at scale.
## 3. Benchmark decontamination
The most embarrassing failure: your held-out eval examples leak into training data,
inflating your scores. Decontaminate by removing training examples that overlap your
benchmarks:
- **n-gram overlap** — for each training example, check for long n-gram matches
(e.g. 13-gram) with any benchmark example; remove or scrub matches.
- **Embedding near-neighbor search** — catch paraphrased contamination that n-gram
matching misses.
Do this *before* training, and re-run after any data refresh. Decontamination is
non-negotiable if you intend to publish trustworthy benchmark numbers.
## 4. Data mixing (ratio engineering)
Capabilities come from *diversity and proportion*, not just volume. Deliberately
control:
- **Task mix** — over-representing one task (e.g. only code) can degrade others.
Balance your target task against a general/diversity buffer so the model doesn't
forget its base abilities.
- **Quality tiering** — the now-standard recipe ("less is more", LIMA-style): a small
amount of very-high-quality data + a larger amount of adequate data. Don't dilute
the high-quality signal with noise.
- **Difficulty** — include examples across difficulty; models trained only on easy
or only on hard examples generalize poorly.
## Pitfalls
- Trusting raw scraped data — web corpora are full of duplicates, spam, and
benchmark leakage by construction.
- Skipping decontamination, then publishing inflated eval numbers that don't hold up
externally.
- Over-deduplicating — aggressive semantic dedup can remove legitimate diverse
examples and reduce coverage.
- Treating "more data" as always better — after a point, more noisy data *hurts*.
## Verify
- After dedup, measure what fraction was removed and spot-check that real duplicates
are gone and distinct examples survived.
- Run your decontamination check on the final training set against your held-out
benchmark — confirm near-zero overlap.
- Fine-tune a small model on the curated data vs the raw corpus; confirm the curated
version performs better on your held-out eval (data quality should show up here).Attached files
No attached files.