model-merging-slerp-ties
verified8eed94c5-045f-4a1e-81f9-f7f1f6b8fafd
Combine fine-tuned LLMs into one model with merging ā SLERP, task arithmetic, TIES, and DARE, when merging is a cheap alternative to joint training, and how to avoid task interference with mergekit.
Metadata
Skill file
# Merging LLMs (SLERP / TIES / DARE)
Use when you have two or more fine-tuned models (say, one good at math, one good
at code) and you want ONE model that can do both ā without retraining on combined
data. Model merging combines weights in parameter space to reuse experimentation.
It's far cheaper than joint training and often preserves the strengths of each
parent. The canonical tool is **mergekit**.
## When (and when not) to merge
- **Good fit:** merging community/fine-tuned checkpoints on the *same base model*
and vocabulary to combine capabilities; recovering a base you've fine-tuned in
several directions; quick experimentation.
- **Important caveat:** merging is not guaranteed to keep all safety/alignment
properties ā one model's alignment can be washed out by an unaligned parent, so
always *evaluate* the merge, never assume quality transfers.
- **Not a substitute for** training on genuinely novel data the parents never saw.
## Key concept: magnitudes matter
Fine-tuned weights can be written as *base* + *delta* (each model's "task
vector"). Merging decides how to combine these deltas. Naive averaging mixes
everything, which causes **task interference** ā the deltas cancel or corrupt each
other. The advanced methods exist to reduce that interference.
## The four main methods (in increasing sophistication)
1. **Model Soup / averaging** ā simple weight average. Cheap, but nothing against
interference; only safe when parents are similar.
2. **SLERP (Spherical Linear Interpolation)** ā interpolates along the *great
circle* between two parameter points instead of the straight line, so weights
stay on a constant-magnitude (spherical) manifold. The go-to for merging exactly
two models; better-behaved than linear interpolation when deltas are unequal.
For more than two models, SLERP is applied pairwise (bake into a tree).
3. **Task Arithmetic** ā merge the sum of task vectors added to the base
(`base + Ī»Ī£(delta_i)`), instead of averaging model weights. Works by treating
each fine-tune as a direction; Ī» controls the strength.
4. **TIES-Merging** ā handles interference explicitly in three steps: **trim** the
small/conflicting deltas, **elect** a sign per parameter (majority vote on the
direction of change), then merge only deltas that agree with the elected sign.
5. **DARE (Drop And REscale)** ā randomly drops a large fraction of the delta
parameters (setting them to base) and rescales the rest to preserve expected
value, then merges. Built on the observation that most delta values are
redundant. Often combined with TIES as **DARE-TIES**.
## Using mergekit
mergekit consumes a YAML config describing the merge and runs on CPU for most
sizes (you typically *don't* need a big GPU ā it's arithmetic on weights):
```yaml
# mergekit config: slerp two 7B models
slices:
- sources:
- model: myorg/code-7b
layer_range: [0, 32]
- model: myorg/math-7b
layer_range: [0, 32]
merge_method: slerp
base_model: myorg/base-7b
parameters:
t:
- filter: self_attn
value: [0.5, 0.5, 0.5, 0.5, 0.5]
- filter: mlp
value: [0.5, 0.5, 0.5, 0.5, 0.5]
tokenizer: myorg/base-7b
dtype: bfloat16
```
Then run:
```bash
mergekit-yaml merge.yml ./merged-model --cuda # --cuda only if you have VRAM
```
For TIES/DARE you set `merge_method: ties` or `dare_ties` (or `dare_linear`) and
tune `parameters: density` (e.g. 0.5-0.8) and a `lambda`/scaling term. The
`trust_remote_code` flag may be required for custom architectures. After merging,
push to the Hub (`huggingface-cli upload` or the mergekit one-liner) and run your
evals.
## Pitfalls
- Merging models with **different tokenizers/vocabularies or different base
models** ā deltas don't line up and you corrupt weights.
- Blindly averaging and then being surprised by catastrophic interference or
alignment loss.
- Using too high a `density` in DARE (keeps conflicting params) or too low (loses
real signal); start ~0.5-0.8 and tune.
- Only testing one or two cherry-picked prompts instead of a real eval set.
- Not configuring a `base_model` for task-vector methods (TIES/DARE need a base to
compute deltas from).
- Assuming a good merge ā always benchmark parents AND the merge on the same tasks.
## Verify
- Reconstruct the merged model and load it with the same tokenizer/base config.
- Eval the merged model on the parents' separate strengths (e.g. math bench AND
code bench) and confirm both are retained, not one dropping.
- Compare against linear averaging as a baseline to justify using SLERP/TIES.
- Check alignment/safety holds (run your red-team/PII checks on the merge).