14. Transfer Learning

Almost nobody trains a model from scratch any more. The interesting question stopped being how to adapt a pretrained model and became whether to adapt it at all — because the alternatives got very good.

So this chapter is organized around a decision, not a technique. First: should you change the weights? Usually not. Then, if you should: which weights, how many, and with what objective — where the honest answers are "the low-rank ones", "about 0.5% of them", and "it depends on whether you have labels, preferences or a verifier".

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. The three simulators and the four diagrams of this page run inside the slides. Open the slides in a new tab

Before you fine-tune: don't

Fine-tuning is the expensive answer to a question that four cheaper things often answer better. The ordering below is the one to work through, top to bottom, and stopping early is the normal outcome.

There is one economic case that reliably justifies fine-tuning: you have a prompt that works on a frontier model, and you need it to run 100× cheaper. Generate data with the big model, fine-tune a small one, and check the gap. That is distillation, and it is how most small production models are actually built now.

The classic picture, and why it changed

The original transfer-learning story¹ is about layers: early layers of a vision model learn edges and textures that transfer everywhere, late layers learn ImageNet's categories, so you freeze the front and retrain the back⁹. It is correct, and for a small task it is still exactly what to do.

"Freeze" means requires_grad = False: the weights still compute, but receive no gradient and no optimizer state. That gives three classic regimes, from cheap to expensive, priced here on the model of the lab (DistilBERT, 66.4 M parameters, with a 768 → 2 head for sentiment):

A frozen encoder has a second advantage that is easy to miss: its features do not change, so they can be computed once and cached, and training the head takes seconds.

The same regimes on the lab's model, with LoRA (the subject of the next sections) beside them: pick one and see what trains, how far down the backward pass has to go and what that costs. The band under the blocks is the classic picture: early layers general, late layers specific to the pretraining task.

The classic picture stopped being enough for one reason. In a language model, the behaviour you want to change is not in a last layer you can swap: it is spread across every block. "Retrain the back" becomes "retrain everything", and for an 8B-parameter model that means an optimizer state that does not fit.

Switch the method to full fine-tuning and read the split. The weights are not the problem. AdamW keeps an fp32 master copy plus two moment estimates — 12 bytes per trainable parameter, six times the size of the bf16 weights themselves. (bf16 and fp32 store a number in 2 and 4 bytes. The moments are running averages of the gradient and of its square; the master copy keeps the weights in fp32, so that small steps are not lost to bf16 rounding.) Add gradients and you need roughly 16 bytes per parameter before a single activation is stored. For the 8B model: 16 GB of weights, 16 of gradients and 96 of optimizer state.

That number is the whole motivation for parameter-efficient fine-tuning (PEFT): gradients and optimizer state are sized by trainable parameters, not by model size. Make 0.5% of the parameters trainable and the panel's bill for the 8B model drops from 129 GB to 18 GB, almost all of it the frozen weights themselves.

LoRA: the update is low-rank

LoRA's hypothesis² is a claim about adaptation, not about compression: the change a task requires has a much lower rank than the weight matrix it modifies. Freeze \(W_0\) and learn the update as a product of two thin matrices:

\(W_0\) is the pretrained \(d \times k\) matrix (\(d\) outputs, \(k\) inputs), \(r\) the rank of the update (the number of independent directions it can have) and \(\alpha\) a fixed scale. \(A\) is initialized randomly and \(B\) at zero, so \(\Delta W = 0\) at step one and training starts from exactly the pretrained model. (Not both at zero: the gradient of \(B\) is proportional to \(A\) and that of \(A\) to \(B\), so two zero factors would never move.) The scaling \(\alpha/r\) exists so that changing \(r\) does not force you to retune the learning rate.

The lab's LoRALinear, at rank 1 on a 2 × 3 matrix, line by line: the forward pass, the two gradients (\(\partial L/\partial A\) is exactly zero at the first step, because it goes through \(B\)), one SGD step, a second one and the merge. One step takes the loss from 0.29 to 0.1856; with \(A\) and \(B\) both at zero it never moves, and at a tenth of the learning rate it barely does (0.2785).

The parameter count follows from the shapes. A full update of a \(d \times k\) matrix has \(dk\) numbers; LoRA trains \(r(d+k)\):

The same shapes, to scale. Pick a matrix and a rank; the purple slabs are all that trains.

On DistilBERT, LoRA on all six linear layers of each of its six blocks adds \(6 \times 8 \times (4 \cdot 1536 + 2 \cdot 3840) = 663{,}552\) parameters at \(r = 8\), 1% of the model. The lab checks that number against peft.

Whether the hypothesis holds depends entirely on the task, and that is what the panel below is for.

The third panel — what the chosen rank cannot express — is the one to watch. A style or format change is genuinely rank-2 or so, and LoRA matches full fine-tuning. A domain shift wants 16 to 32. Teaching a new language is not low-rank at all, and no affordable \(r\) makes that panel empty.

The three spectra in the panel are made up to show the mechanism. The measured version is Biderman et al.⁸: on continued pretraining for code and mathematics (more next-token training on a large unlabelled corpus of the new domain), the weight change that full fine-tuning learns has a rank 10 to 100 times higher than typical LoRA settings, and LoRA falls behind. On a narrow task, the opposite: in the lab, sentiment classification stays within about 2 points from \(r = 1\) to \(r = 32\), with no gain past \(r = 4\).

QLoRA³ adds the other half: it shrinks the one item LoRA leaves alone, the frozen weights. The base is stored in 4-bit NF4 (NormalFloat: 16 levels placed at quantiles of a normal distribution, dense near zero, where most weights are), in blocks of 64 weights that share one scale; each block is dequantized to bf16 when it is multiplied, and the LoRA weights stay in bf16. Since the base is frozen, its quantization error is a fixed bias the adapter simply trains around. This is the technique that put 65B fine-tuning on a single 48 GB GPU. One block, step by step:

DoRA⁵ decomposes each weight into a magnitude and a direction and applies LoRA to the direction only. In its authors' benchmarks it beats LoRA consistently at about the same training cost, and it merges just the same, so inference costs nothing extra. It is the recent PEFT variant to try first when LoRA underperforms.

The rest of the PEFT zoo, honestly

Post-training: what you optimize, not which weights

PEFT answers "which weights move". The other half of the question is toward what, and the modern pipeline has three stages that do genuinely different things.

Step through the stages below. Each shows the data it consumes, the signal that trains it and what it changes in the model, and one prompt is followed from the base model to the deployed one.

Not every model goes through every stage: stage 3 needs answers that can be checked, and a distilled model often stops after SFT.

1. Supervised fine-tuning (SFT). Train on (prompt, good answer) pairs — the same next-token loss, on curated demonstrations. This is what turns a base model into something that answers rather than continues. Data quality dominates quantity here by a wide margin: a few thousand carefully written examples routinely beat hundreds of thousands of scraped ones.

2. Preference optimization. SFT teaches one good answer; it cannot teach that answer A is better than answer B. The original solution, RLHF⁷ (reinforcement learning from human feedback), trains a reward model on human comparisons and then optimizes the policy (the model being trained) with PPO, a reinforcement-learning algorithm — powerful, and a genuine engineering burden: four models in memory (the policy, a frozen reference, the reward model and a value network that predicts the expected reward) and a famously unstable loop.

DPO⁴ removed most of that. The insight is that the RLHF objective has a closed-form optimal policy, so the reward model can be substituted away and preferences optimized directly with a classification-style loss on pairs:

Here \(x\) is the prompt, \(y_w\) and \(y_l\) the preferred and the rejected answer, \(\pi_\theta\) the model in training, \(\pi_{\text{ref}}\) the frozen SFT model, \(\beta\) how far \(\pi_\theta\) may drift from \(\pi_{\text{ref}}\) and \(\sigma\) the sigmoid. Two models instead of four, no sampling loop, and it trains like supervised learning. For most practitioners it replaced PPO outright.

3. Reinforcement learning from a verifier (RLVR). When correctness can be checked — a unit test passes, a proof compiles, an arithmetic answer matches — you do not need a learned reward model at all. Sample many attempts, score them with the checker, and push probability toward the ones that worked. GRPO¹⁰ does exactly this and drops the value network too, normalizing rewards within each group of samples instead:

\(G\) attempts are sampled for the same prompt, \(r_i\) is the checker's score of attempt \(i\) and its advantage \(\hat{A}_i\) is positive for the attempts above the group mean. This is the training signal behind the reasoning models of the last two years⁶, and it is treated properly in chapter 15. The point to carry here: it is fine-tuning, and it needs a verifier rather than a label.

When to use what

Lab: fine-tuning, priced

Every number on this page that has DistilBERT in it is measured in a Colab notebook: DistilBERT (66 M parameters) adapted to SST-2, the sentiment of movie-review sentences, with 2,000 training examples. It needs a GPU, and Colab's free T4 is enough (Runtime → Change runtime type → T4 GPU); there is nothing to install. Eleven sections, each ending in a checkpoint with a check value:

1. the parameters of a Transformer encoder, counted by hand;
2. a linear probe: features computed once, 1,538 numbers trained;
3. full fine-tuning, with its optimizer state measured in bytes;
4. LoRA written by hand: the zero initialization, which factor gets a gradient first, the merge;
5. LoRA with peft on every linear layer, its \(r(d+k)\) checked against print_trainable_parameters, and the learning rate it needs;
6. a rank sweep from \(r = 1\) to \(r = 32\);
7. the merged model, checked against the unmerged one;
8. the rank of the update that full fine-tuning found on its own;
9. what each method forgot, measured on a second task;
10. (optional, CUDA only) the 4-bit base of QLoRA;
11. your turn: below how many labelled examples is the probe enough?

Open the lab in Colab — save a copy to your Drive (File → Save a copy in Drive) to run and edit it.

Key takeaways

1. The first question is whether to fine-tune at all. Prompting, retrieval, tools and a bigger model solve most of what people reach for fine-tuning to fix.
2. Fine-tune for behaviour, retrieve for knowledge. Facts in weights cannot be cited, updated or deleted.
3. Full fine-tuning is expensive because of optimizer state, not weights: AdamW costs 12 bytes per trainable parameter.
4. LoRA works when the required update is low-rank — which is true for style and format, roughly true for domain shifts, and false for new knowledge.
5. Apply LoRA to all linear layers, use a learning rate ~10× higher than full fine-tuning, and merge for deployment so inference costs nothing extra.
6. QLoRA quantizes the frozen base; the adapter trains around the quantization error. DoRA is the variant to try when LoRA falls short.
7. Post-training is SFT → preference optimization → RL from a verifier. DPO replaced PPO for most practitioners; GRPO with a checker is what produced the reasoning models.
8. Distillation from a stronger model is how most small production models are built.
9. Fine-tuning causes forgetting. Mix in general data, prefer LoRA, and evaluate off-task.