Artificial Neural Networks and Deep Learning · Insper

Transfer Learning & Fine-Tuning

Whether to change the weights, and which ones

The question changed

Your model answers correctly but ignores the output format you need. Do you fine-tune it?

  • Not first. Almost nobody trains from scratch any more, and adapting the weights is now the expensive option.
  • The question is no longer how to adapt a pretrained model, but whether to adapt it at all.
  • And if so: which weights, how many and toward what objective.

Before you fine-tune: don't

Try this first It fixes It cannot fix
A better prompt, with examples format, tone, task framing missing knowledge; cost per call
Retrieval (RAG) facts it lacks, facts that change, citations behaviour and style
Tools / function calls arithmetic, current data, actions anything without an API
A larger or newer model almost everything, at once your latency and unit-cost targets
Fine-tuning consistent behaviour at low cost; a small model matching a big one on your task facts that change

Work through it top to bottom. Stopping early is the normal outcome.

Fine-tune for behaviour, retrieve for knowledge

A fine-tune smears new facts over billions of weights: no citation, no way to update or delete one of them. An index does the same job with a PUT.

The company documentation changes every week. Someone proposes fine-tuning on it. What do you say?

  • Retrieval. Fine-tune for behaviour; retrieve for knowledge.
  • The one economic case that reliably justifies fine-tuning: a prompt that works on a frontier model must run 100× cheaper → generate data with the big model and fine-tune a small one. That is distillation.

The classic picture: layers

  • Early layers learn edges and textures: they transfer everywhere.
  • Late layers learn the pretraining task's categories.
  • So: freeze the front, retrain the back (Yosinski et al., 2014).

Freeze = requires_grad = False. The weights still compute, but get no gradient and no optimizer state.

Correct, and for a small task still exactly what to do. The lab uses DistilBERT (66.4 M parameters) with a new 768 → 2 head for sentiment: 768 is the size of the vector the encoder hands over per sentence; 2, the classes (negative, positive).

Three regimes, priced on DistilBERT

Regime What trains Trainable What it cannot do
Linear probe the new head only 768 · 2 + 2 = 1,538 change the features
Partial fine-tuning head + last blocks ~7.1 M per block fix the frozen early blocks
Full fine-tuning everything, small learning rate (lr) 66.4 M stay cheap

One block: 4 attention projections 768 × 768 + MLP 768 → 3072 → 768 + 2 LayerNorms =
4·(768² + 768) + (768·3072 + 3072) + (3072·768 + 768) + 2·2·768 = 7,087,872 (every Linear has a bias).
The model: embeddings 23,835,648 + 6 blocks 42,527,232 + head 1,538 = 66,364,418.

  • A frozen encoder has a hidden bonus: its features never change, so compute them once and cache them. The probe then trains in about a second.

What trains, and where the gradient goes

DistilBERT with a new head, drawn as a row: the embeddings, six Transformer blocks, the mean over tokens, the head and the loss. A button picks the regime. Linear probe: only the head trains, the backward pass stops at the head and the encoder runs once per sentence. Partial fine-tuning: block 6 and the head train and the backward pass stops at block 6. LoRA: the pretrained weights are frozen, adapters sit inside all six blocks and the backward pass crosses the whole encoder to reach them. Full fine-tuning: everything trains and the backward pass reaches the embeddings.

Why the classic picture stopped being enough

Full fine-tuning of an 8B model with AdamW in bf16. How many GB before a single activation?

  • bf16 stores a number in 2 bytes; fp32, in 4. Weights: 8·10⁹ × 2 bytes = 16 GB. Gradients, also in bf16: 16 GB.
  • AdamW: fp32 master copy (4) + two fp32 moments (4 + 4) = 12 bytes/param → 96 GB.
  • 2 + 2 + 12 = 16 bytes per parameter: 128 GB. The optimizer state (12) is six times the weights (2).
  • And in a language model the behaviour is spread over every block: "retrain the back" becomes "retrain everything".

Moments: the running averages of the gradient and of its square. Master copy: the weights in fp32, where each small step is added without being lost to bf16 rounding.

Simulator · the memory bill

The bill is sized by what trains

Gradients and optimizer state exist only for trainable parameters — not for the model.

8B model, AdamW full: 8·10⁹ trainable LoRA: 0.5% = 4·10⁷ trainable
weights (bf16, 2 bytes) 8·10⁹ × 2 = 16 GB 16 GB — the whole base
gradients (2 bytes) 8·10⁹ × 2 = 16 GB 4·10⁷ × 2 = 0.08 GB
optimizer state (12 bytes) 8·10⁹ × 12 = 96 GB 4·10⁷ × 12 = 0.48 GB
activations (checkpointing) 1.4 GB 1.4 GB
total 129 GB 18 GB

Activations: the simulator's estimate, with checkpointing (keep few activations, recompute the rest in the backward pass).
Measured in the lab, on DistilBERT in fp32 (no master copy, only the two 4-byte moments): 66,364,418 × 8 = 530.9 MB for full fine-tuning; 665,090 × 8 = 5.3 MB for LoRA r = 8.

LoRA: the update is low-rank

Hypothesis (Hu et al., 2022): the change a task requires has a much lower rank than the matrix it modifies.

  • is the pretrained matrix, ( outputs, inputs): frozen. Only and train.
  • Rank = how many independent directions a matrix has. has at most .
  • It is a claim about adaptation, not about compression: itself is full rank.
  • is a fixed scale (a hyperparameter); dividing by means changing does not force retuning the learning rate.

Initialization: one factor at zero

starts random, starts at zero → : training starts exactly at the pretrained model.

Why not both at zero?

With and (the gradient arriving at the layer's output):

  • Both at zero → both gradients zero → neither ever moves.
  • With only: at the first step gets zero gradient and does not. One step later, both train.

Lab, checkpoint 4 (the first block's lin1 with r = 8): ‖∂L/∂A‖ = 0.0 exactly; ‖∂L/∂B‖ = 1163.42.

Simulator · one LoRA layer, line by line

W₀ (frozen), x and A₀ · k = 3 columns
y and B₀ · d = 2 rows

The arithmetic: instead of

matrix full update LoRA, : ratio
4096 × 4096 projection (8B-class) 4096 · 4096 = 16,777,216 8 · (4096 + 4096) = 65,536 256×
DistilBERT q_lin, 768 × 768 768 · 768 = 589,824 8 · (768 + 768) = 12,288 48×
DistilBERT lin1, 768 → 3072 768 · 3072 = 2,359,296 8 · (768 + 3072) = 30,720 77×

LoRA, to scale

r =
A LoRA layer drawn to scale: the frozen matrix W0 of d rows and k columns, plus alpha over r times B, a slab of d rows and r columns, times A, a slab of r rows and k columns. Their product BA has the shape of W0 and rank at most r. Buttons pick the matrix (DistilBERT's q_lin, its lin1, or a 4096 by 4096 projection) and the rank, and the text below counts the numbers: d times k frozen against r times d plus k trainable.

Exercise · count the adapters

LoRA with on the six linear layers of each of DistilBERT's six blocks, plus the new head. How many trainable parameters? Fill it in before turning the slide.

LoRA on a matrix: . The head is new: it trains in full (weights + bias).

layer shape trainable
q_lin, k_lin, v_lin, out_lin (attention) 768 × 768 each ?
lin1 (MLP) 768 → 3072 ?
lin2 (MLP) 3072 → 768 ?
one block ?
six blocks ?
new head Linear(768, 2) 768 → 2 ?
total ?

Count the adapters · answer

layer shape trainable where it comes from
q_lin, k_lin, v_lin, out_lin 768 × 768 each 49,152 4 · 8 · (768 + 768) = 4 · 12,288
lin1 768 → 3072 30,720 8 · (768 + 3072)
lin2 3072 → 768 30,720 8 · (3072 + 768)
one block 110,592 8 · (4 · 1536 + 2 · 3840) = 8 · 13,824
six blocks 663,552 6 · 110,592
new head 768 → 2 1,538 768 · 2 + 2, trains in full
total 665,090 663,552 + 1,538

Exactly what peft's print_trainable_parameters printed in the lab: 665,090 of 67,027,970 = 0.99%. The total includes the adapters: 66,364,418 + 663,552.

Simulator · how much rank does the task need?

The third panel is the part you chose not to learn

The panel shows what of the update falls outside the chosen rank .

  • Style or format: genuinely rank 2 or so. LoRA matches full fine-tuning.
  • A domain shift: wants = 16 to 32.
  • A new language or new knowledge: not low-rank at any affordable .

The panel's spectra are made up. The measurement: on continued pretraining for code and maths, full fine-tuning learns an update of rank 10–100× higher than typical LoRA settings, and LoRA falls behind (Biderman et al., 2024).

  • A loss that plateaus high under LoRA is the constraint binding, not a bad hyperparameter.

Practical settings

  • , (scale ) to start; raise only if the training loss plateaus high.
  • All linear layers (attention q,k,v,o, MLP gate,up,down, in Llama's names), not just as in the original paper: that is what it takes to match full fine-tuning — and the MLP holds two-thirds of a block (in DistilBERT, 2 · 2.36 M of 7.09 M).
  • Learning rate ~10× that of full fine-tuning (in the lab, 5e-4 against 5e-5).
  • Merge for deployment: is one matrix again → inference costs nothing extra.
  • Keep it unmerged to serve many adapters over one shared base.

Measured in the lab: three regimes

DistilBERT, 2,000 SST-2 sentences, 3 epochs, test on the 872 of the validation split.

method trainable test acc optimizer state
random head — 0.520 —
linear probe 1,538 0.787 0.01 MB
full fine-tuning, lr 5e-5 66,364,418 0.849 530.9 MB
LoRA r = 8, lr 5e-4 665,090 0.853 5.3 MB
  • LoRA matches full fine-tuning with 1% of the trainable parameters (665,090 / 66,364,418) and of the optimizer state (5.3 / 530.9 MB).
  • But it was only ~15% faster (71 s against 85 s): the backward pass still crosses the whole encoder. LoRA saves memory, not compute.

One test sentence is 1/872 ≈ 0.11 point: differences of a point are noise. Random head ≈ chance (two classes).

Measured in the lab: rank and learning rate

trainable = 6 · 13,824 · + 1,538 test acc
1 84,482 0.839
2 167,426 0.848
4 333,314 0.858
8 665,090 0.853
16 1,328,642 0.860
32 2,655,746 0.846
  • About 2 points (0.839 to 0.860) across a 32× range of parameters, no trend past : sentiment is a low-rank update.
  • Same LoRA at the full fine-tuning lr (5e-5): 0.822. The update starts at zero; it needs the ~10× higher lr.
  • Full fine-tuning, left free: in the last block's q_lin, the top 8 directions hold 65% of 's energy (random: 4%).

Energy = sum of the squared singular values σᵢ (the size of ΔW along each of its independent directions): 65% = (σ₁² + … + σ₈²) / (σ₁² + … + σ₇₆₈²).

Merging, checked

After merge_and_unload(), how many parameters? How far are the logits from the unmerged model's?

  • 66,364,418 — exactly the model before LoRA.
  • Max logit difference: 6.2 × 10⁻⁶ (float32 rounding).
  • Same test accuracy. No extra layer, no extra multiply.

LoRA costs memory during training only. Merged, it is the base architecture.

QLoRA and DoRA

QLoRA (Dettmers et al., 2023)

  • Store the frozen base in 4-bit NF4 (0.5 byte per weight); train 16-bit adapters on top.
  • The base is frozen, so its quantization error is a fixed perturbation the adapter trains around.
  • Put a 65B fine-tune on one 48 GB GPU: 65·10⁹ × 0.5 byte = 32.5 GB of base (33.5 GB with the block scales, next slide).

DoRA (Liu et al., 2024)

  • Split each weight into magnitude and direction; LoRA on the direction only.
  • In its authors' benchmarks, consistently above LoRA at about the same cost.
  • Merges the same way.

Weights are the one item LoRA does not shrink. QLoRA shrinks them: 70B with LoRA ≈ 140 GB of weights (70·10⁹ × 2) + 9 GB of gradients, optimizer and activations = 149 GB; with QLoRA, 35 GB of weights (70·10⁹ × 0.5) + the same 9 = 44 GB, one 80 GB card.

QLoRA: stored in 4 bits, computed in 16

QLoRA's storage, step by step, on one block of 64 frozen weights: the weights on a number line; divided by the block's largest absolute value, so they lie between minus one and one; rounded to the nearest of the 16 NF4 levels, dense near zero, and stored as 4-bit indices; the scale constants quantized again; the forward pass, which dequantizes to bf16 and adds the bf16 LoRA path, with the gradient reaching A, B and the input but not the stored codes; and the bill for a 65B model: 130 GB in bf16 against 33.5 GB in NF4.

The rest of the PEFT zoo, honestly

Technique Status in 2026
LoRA the default; everything else is measured against it
QLoRA the default when memory is the constraint
DoRA try it when LoRA is not enough and full fine-tuning is not affordable
Adapter layers (Houlsby) historical: they add inference latency, cannot be merged
Prefix / prompt tuning historical: hard to optimize, weaker, eat the context window
BitFit (biases only), IA³ (one scale vector per activation) curiosities: useful to see what suffices, not to ship

Post-training: what you optimize

PEFT answers which weights move. The other half: toward what.

  1. SFT (supervised fine-tuning) — (prompt, good answer) pairs, the same next-token loss. Turns a base model into one that answers. Quality beats quantity.
  2. Preference optimization — "A is better than B", which SFT cannot teach. RLHF (reinforcement learning from human feedback: reward model + PPO), now mostly DPO.
  3. RL from a verifier — answers a program can check: GRPO.

Not every model goes through all three. Distillation writes the demonstrations of stage 1.

Post-training, one prompt through the stages

DPO: the reward model, substituted away

x: the prompt · yw, yl: the preferred and the rejected answer · πθ: the model in training (the policy) · πref: the SFT model, frozen · β: how far πθ may drift from πref · σ: sigmoid · PPO: the reinforcement-learning algorithm of the original RLHF · value network: a model that predicts the expected reward, PPO's baseline

  • RLHF with PPO: four models in memory (policy, reference, reward model, value network) and an unstable loop.
  • The RLHF objective has a closed-form optimal policy → the reward model can be written in terms of the policy.
  • DPO: two models (πθ + πref), no sampling loop, trains like supervised learning. Still needs SFT first and preference data.

GRPO: fine-tuning with a verifier instead of a label

When correctness can be checked (tests pass, a proof compiles, the answer matches), no learned reward model is needed.

  • Sample a group of attempts at the same prompt; is the checker's score for attempt ; (the advantage) pushes toward the ones above the group mean.
  • Example: , → mean 0.5, std 0.5 → .
  • No reward model and no value network (Shao et al., 2024). The training signal behind the reasoning models (DeepSeek-R1) — see the LLMs class.

Catastrophic forgetting is not hypothetical

Fine-tune hard on a narrow task and the model gets worse at everything else.

encoder AG News probe (news topics)
original 0.856
after full fine-tuning on SST-2 0.840 (−1.6 points)
after LoRA r = 8 on SST-2 0.843 (−1.3 points)

Measured in the lab: 3 epochs on 2,000 sentiment sentences already cost the news task 1.3–1.6 points. Probe = only a linear head trained on the frozen encoder. The 0.3-point gap between the two is noise at 1,000 test examples; at scale, LoRA forgets less (Biderman et al., 2024).

Mitigations: LoRA (changes less, forgets less), a low learning rate, ~10% general data in the mix, and evaluate off-task, before and after: the only way to find out.

When to use what

Situation Do this
Wrong format or tone Prompt engineering
Missing or changing facts RAG
Frontier quality at 1% of the cost Distil: SFT a small model on the big one's outputs
< 1k labelled examples Few-shot prompting, or frozen encoder + linear head
1k–100k examples, one GPU LoRA, all linear layers,
Same, memory-bound QLoRA
Many customers, one base LoRA adapters served unmerged
"Prefer this answer to that one" SFT → DPO
Answers a program can check SFT → GRPO with a verifier

Lab · fine-tuning, priced

Colab notebook, ~2 h, DistilBERT on SST-2, T4 GPU. Each section ends in a checkpoint with a check value.

  1. the model and the data: counting parameters
  2. linear probe: features once, head only
  3. full fine-tuning: the optimizer state, measured
  4. LoRA written by hand: init, gradients, merge
  5. LoRA with peft: against the printout
  1. the rank sweep
  2. merge and serve
  3. is the full update low-rank?
  4. what each method forgot
  5. (optional) QLoRA, CUDA only
  6. your turn: when is the probe enough?

open in Colab →

Takeaways

  1. First ask whether to fine-tune at all. Prompting, retrieval, tools and a bigger model fix most of it.
  2. Fine-tune for behaviour, retrieve for knowledge.
  3. Full fine-tuning is expensive because of optimizer state: AdamW costs 12 bytes per trainable parameter.
  4. LoRA trains instead of , starts at the pretrained model, merges to zero extra cost — and works when the update is low-rank.
  5. QLoRA quantizes the frozen base; DoRA when LoRA falls short.
  6. Post-training: SFT → DPO → GRPO with a verifier.
  7. Fine-tuning causes forgetting: prefer LoRA, mix in general data, evaluate off-task.

Outline: (1) before fine-tuning: the cheaper alternatives; (2) the classic regimes and the memory bill; (3) LoRA: the hypothesis, the arithmetic, the rank; (4) what the lab measured; (5) QLoRA, DoRA and the PEFT zoo; (6) post-training: SFT, DPO, GRPO; forgetting. Closes with the Colab lab.

Let them answer. Most will say "fine-tune". The answer for format is a better prompt with examples.

Lab, checkpoint 1: the embeddings (30,522 tokens and 512 positions, each a vector of 768, plus a LayerNorm) are more than a third of the model.

The previous table, drawn. Click the four buttons in order. Probe: the encoder runs once, the backward pass stops at the head. Partial: it stops at block 6. LoRA: everything frozen, yet the backward pass goes down to block 1 to reach its adapters: that is why LoRA saved memory and not time in the lab (71 s against 85 s). Full: down to the embeddings, 530.9 MB of AdamW state. Hover a block for its count. The band is the classic picture: early general, late specific.

Start at 8B, full fine-tuning: 129 GB, 74% optimizer state. Switch to LoRA: 18 GB, fits a 24 GB card. Then 70B with LoRA (149 GB) and QLoRA (44 GB): that is the one-GPU difference.

The previous slide's formulas with numbers: W0 is 2 × 3, r = 1, α = 2. Step to line 5: B.grad = [-1, -0.4], A.grad exactly 0, the lab's checkpoint 4 in miniature. Line 6: only B moved. Line 7: the loss 0.29 → 0.1856 and now A gets a gradient. Line 8: the merged W gives the same h. Then the cases: "A = 0 and B = 0" never moves; "the other way" also works (B random, A = 0), so the rule is one factor at zero, not which one; "full fine-tuning's learning rate" barely moves (0.2785).

The table above, drawn to scale. Start on q_lin, r = 8: the two slabs are 8 of 768 wide, 12,288 numbers against 589,824. Switch to lin1 (77x) and to the 4096 x 4096 projection (256x; at this scale the slabs are thinner than a pixel). Then r = 64 on q_lin: still 6x fewer. BA is never stored in training: the layer computes B(Ax).

Style: r=1 leaves a sliver, r=2 nothing. Domain: watch r=8 (84%) and r=16 (98%). Language: every extra rank buys the same 2%; at r=32 a third of the update is still missing.

Notebook sections 2, 3 and 5. The seconds are from an Apple GPU; on a T4 they will differ.

Six steps on one block of 64 frozen weights. Absmax scaling puts them in [-1, 1]; each is rounded to the nearest of the 16 NF4 levels (dense near zero) and stored as a 4-bit index: 4.5 bits per weight with the fp32 scale, 4.127 with double quantization. Step 5 is the point: the base is dequantized to bf16 for each multiply, A and B are bf16, and no gradient reaches the codes. Step 6: 65B in 33.5 GB, one 48 GB GPU.

Five stages, one prompt (how many r in strawberry). Base: it continues the text. SFT: the format, but wrong. DPO: the preferred answer gains probability against the rejected one, relative to the frozen SFT model. GRPO: a group of four, rewards (1, 0, 1, 1), mean 0.75, std 0.43, advantages +0.58 and -1.73. Deployed: frozen weights.