Skip to content

14. Transfer Learning

Nobody trains a model from scratch. The interesting question stopped being how to adapt a pretrained model and became whether to adapt it at all β€” because the alternatives got very good.

So this chapter is organized around a decision, not a technique. First: should you change the weights? Usually not. Then, if you should: which weights, how many, and with what objective β€” where the honest answers are "the low-rank ones", "about 0.5% of them", and "it depends on whether you have labels, preferences or a verifier".

Before you fine-tune: don't

Fine-tuning is the expensive answer to a question that four cheaper things often answer better. The ordering below is the one to work through, top to bottom, and stopping early is the normal outcome.

Try this first It fixes It cannot fix
A better prompt, with a few examples Format, tone, task framing, most "the model doesn't follow instructions" complaints Missing knowledge; latency; cost per call
Retrieval (RAG) Facts the model does not have, facts that change, per-customer data, citations Behaviour and style; reasoning ability
Tools and function calls Arithmetic, current data, actions in a system, anything with a right answer a program can compute Anything without an API
A larger or newer model Almost everything, immediately, at higher cost per token Your latency and unit-cost targets
Fine-tuning Consistent format and behaviour at low cost; a small model that matches a big one on your task; domain vocabulary and style Facts that change β€” those belong in retrieval

Fine-tuning is the wrong tool for teaching facts

This is the single most common and most expensive mistake in applied LLM work. A fine-tune smears new facts across billions of weights, non-verifiably, with no citation and no way to update or delete one of them. A retrieval index does the same job with a PUT. Fine-tune for behaviour; retrieve for knowledge. When somebody proposes fine-tuning on the company documentation, this is the sentence to say out loud.

There is one economic case that reliably justifies fine-tuning: you have a prompt that works on a frontier model, and you need it to run 100Γ— cheaper. Generate data with the big model, fine-tune a small one, and check the gap. That is distillation, and it is how most small production models are actually built now.

The classic picture, and why it changed

The original transfer-learning story is about layers: early layers of a vision model learn edges and textures that transfer everywhere, late layers learn ImageNet's categories, so you freeze the front and retrain the back1. It is correct, and for a small vision task it is still exactly what to do.

It stopped scaling for one reason: "retrain the back" of an 8B-parameter model still means an optimizer state that does not fit.

Switch the method to full fine-tuning and read the split. The weights are not the problem. AdamW keeps an fp32 master copy plus two moment estimates β€” 12 bytes per trainable parameter, six times the size of the bf16 weights themselves. Add gradients and you need roughly 16 bytes per parameter before a single activation is stored.

That number is the whole motivation for parameter-efficient fine-tuning: gradients and optimizer state are sized by trainable parameters, not by model size. Make 0.5% of the parameters trainable and 90% of the memory disappears.

LoRA: the update is low-rank

LoRA's hypothesis2 is a claim about adaptation, not about compression: the change a task requires has a much lower rank than the weight matrix it modifies. Freeze \(W_0\) and learn the update as a product of two thin matrices:

\[ W = W_0 + \Delta W = W_0 + \frac{\alpha}{r} BA, \qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r \ll \min(d,k) \]

\(A\) is initialized randomly and \(B\) at zero, so \(\Delta W = 0\) at step one and training starts from exactly the pretrained model. The scaling \(\alpha/r\) exists so that changing \(r\) does not force you to retune the learning rate.

Whether the hypothesis holds depends entirely on the task, and that is what the panel below is for.

The third panel β€” what the chosen rank cannot express β€” is the one to watch. A style or format change is genuinely rank-2 or so, and LoRA matches full fine-tuning. A domain shift wants 16 to 32. Teaching a new language is not low-rank at all, and no affordable \(r\) makes that panel empty.

Practical LoRA settings, and one piece of outdated advice to unlearn

The original paper applied LoRA to \(W_q\) and \(W_v\) only, and that recommendation is still repeated everywhere. Later work is clear that applying it to every linear layer β€” including the MLP β€” is better at equal parameter budget, and the MLP is where two-thirds of the parameters live. Sensible defaults:

  • \(r = 16\), \(\alpha = 32\) as a starting point; raise \(r\) only if the training loss plateaus high.
  • Target all linear projections: q,k,v,o,gate,up,down.
  • Learning rate 10Γ— higher than full fine-tuning β€” around 1e-4, not 1e-5.
  • Merge for deployment: \(W_0 + BA\) is one matrix again, so inference costs nothing extra. Keep it unmerged when you want to serve many adapters over one shared base.

QLoRA3 adds the other half: quantize the frozen base to 4-bit NF4 and keep the LoRA weights in bf16. Since the base is frozen, its quantization error is a fixed bias the adapter simply trains around. This is the technique that put 65B fine-tuning on a single 48 GB GPU.

DoRA5 decomposes the update into magnitude and direction and applies LoRA to the direction only, which closes most of the remaining gap to full fine-tuning at effectively the same cost. It is the one recent PEFT variant worth defaulting to when LoRA underperforms.

The rest of the PEFT zoo, honestly

Technique Status in 2026
LoRA The default. Everything else is measured against it.
QLoRA The default when memory is the constraint.
DoRA Worth trying when LoRA is not enough and full fine-tuning is not affordable.
Adapter layers (Houlsby) Historical. They add inference latency because they cannot be merged.
Prefix / prompt tuning Historical. Hard to optimize, weaker, and they consume context window.
BitFit, IAΒ³ Curiosities. Useful for understanding what is sufficient, not for shipping.

Post-training: what you optimize, not which weights

PEFT answers "which weights move". The other half of the question is toward what, and the modern pipeline has three stages that do genuinely different things.

flowchart LR
    A[Base model<br/>next-token prediction] --> B[SFT<br/>demonstrations]
    B --> C[Preference optimization<br/>DPO / GRPO]
    C --> D[Deployed model]
    B -.-> E[Distillation<br/>from a stronger model]
    E --> C

1. Supervised fine-tuning (SFT). Train on (prompt, good answer) pairs β€” the same next-token loss, on curated demonstrations. This is what turns a base model into something that answers rather than continues. Data quality dominates quantity here by a wide margin: a few thousand carefully written examples routinely beat hundreds of thousands of scraped ones.

2. Preference optimization. SFT teaches one good answer; it cannot teach that answer A is better than answer B. The original solution, RLHF7, trains a reward model on human comparisons and then optimizes the policy with PPO β€” powerful, and a genuine engineering burden: four models in memory and a famously unstable loop.

DPO4 removed most of that. The insight is that the RLHF objective has a closed-form optimal policy, so the reward model can be substituted away and preferences optimized directly with a classification-style loss on pairs:

\[ \mathcal{L}_{\text{DPO}} = -\log \sigma\!\left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \right) \]

Two models instead of four, no sampling loop, and it trains like supervised learning. For most practitioners it replaced PPO outright.

3. Reinforcement learning from a verifier (RLVR). When correctness can be checked β€” a unit test passes, a proof compiles, an arithmetic answer matches β€” you do not need a learned reward model at all. Sample many attempts, score them with the checker, and push probability toward the ones that worked. GRPO6 does exactly this and drops the value network too, normalizing rewards within each group of samples instead:

\[ \hat{A}_i = \frac{r_i - \text{mean}(r_{1..G})}{\text{std}(r_{1..G})} \]

This is the training signal behind the reasoning models of the last two years, and it is treated properly in chapter 15. The point to carry here: it is fine-tuning, and it needs a verifier rather than a label.

Catastrophic forgetting is not hypothetical

Fine-tune hard on a narrow task and the model gets worse at everything else β€” including instruction-following and safety behaviour it was aligned for. Mitigations, in order of how often they are the right answer: use LoRA (it changes less, so it forgets less), keep the learning rate low, mix ~10% general data into your fine-tuning set, and evaluate on tasks you are not training on before and after. That last one is the only way to find out.

When to use what

Situation Do this
Wrong format or tone Prompt engineering. Do not fine-tune.
Missing or changing facts RAG. Do not fine-tune.
Needs current data or actions Tools / function calling
Frontier-model quality at 1% of the cost Distil: generate data with the big model, SFT a small one
< 1k labelled examples Few-shot prompting, or a frozen encoder + a linear head for vision
1k–100k examples, one GPU LoRA, all linear layers, \(r=16\)
Same, but memory-bound QLoRA
> 100k examples, a real cluster, and the task is far from pretraining Full fine-tuning or continued pretraining
Many customers, one base model LoRA adapters served unmerged over a shared base
"Prefer this answer to that one" SFT β†’ DPO
Answers a program can check SFT β†’ GRPO with a verifier
A whole new domain corpus, unlabelled Continued pretraining, then SFT

Key takeaways

  1. The first question is whether to fine-tune at all. Prompting, retrieval, tools and a bigger model solve most of what people reach for fine-tuning to fix.
  2. Fine-tune for behaviour, retrieve for knowledge. Facts in weights cannot be cited, updated or deleted.
  3. Full fine-tuning is expensive because of optimizer state, not weights: AdamW costs 12 bytes per trainable parameter.
  4. LoRA works when the required update is low-rank β€” which is true for style and format, roughly true for domain shifts, and false for new knowledge.
  5. Apply LoRA to all linear layers, use a learning rate ~10Γ— higher than full fine-tuning, and merge for deployment so inference costs nothing extra.
  6. QLoRA quantizes the frozen base; the adapter trains around the quantization error. DoRA is the variant to try when LoRA falls short.
  7. Post-training is SFT β†’ preference optimization β†’ RL from a verifier. DPO replaced PPO for most practitioners; GRPO with a checker is what produced the reasoning models.
  8. Distillation from a stronger model is how most small production models are built.
  9. Fine-tuning causes forgetting. Mix in general data, prefer LoRA, and evaluate off-task.

  1. Pan, S. J., & Yang, Q. (2010). A Survey on Transfer Learning β€” IEEE TKDE. The framing that predates deep learning and still organizes the field. β†©

  2. Hu, E., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models β€” ICLR. β†©

  3. Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs β€” NeurIPS. 4-bit NF4, double quantization, paged optimizers. β†©

  4. Rafailov, R., et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model β€” NeurIPS. The derivation that removes the reward model. β†©

  5. Liu, S.-Y., et al. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation β€” ICML. β†©

  6. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. GRPO, and reasoning learned from a verifier rather than from demonstrations. β†©

  7. Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback β€” NeurIPS. InstructGPT; the SFT β†’ reward model β†’ PPO pipeline in full. β†©

  8. Biderman, D., et al. (2024). LoRA Learns Less and Forgets Less β€” TMLR. A careful measurement of both halves of the trade. β†©