The Frontier

The last stop of the arc that began with least squares in 1805. Its centerpiece is a map of the field: every technique of the previous lessons, grouped by the job it does, next to the techniques that lie just beyond this course, each with the lesson where it was taught and where to read more. Around the map: how transfer learning and foundation models changed the economics of machine learning, where the classical toolbox you built still wins, and the problems the field has not solved yet.

The lesson has no dataset of its own. Its two labs are guided exercises on this page: you click the map, route requests to the right tool, and check with a few lines of Python three claims the text makes — that the retrieval half of a RAG system is a k-NN, that the reward model of RLHF is a logistic regression, and that LoRA trains a tiny fraction of a model. Every number quoted comes from the code blocks on this page or from the papers cited.

How this lesson is organized

The lesson has two approaches. Each has slides, text and a lab to do in class; the quiz at the end covers the fundamental one. Start with the fundamental approach: the advanced one builds on it.

This lesson has no notebook. The code of each step runs in any Python with scikit-learn: open a new Colab notebook (nothing to install) and copy the blocks from this page, in order. Steps 1 and 4 need no code.

Fundamental approach

One question drives this part: where does everything you learned sit, and when is a foundation model the right tool? The answer starts from the map, goes through the two ideas that changed the economics of the field (reuse a pre-trained model; adapt it with a prompt instead of a training run), finds the pieces of this course inside the new stack, and ends with a routing rule: which kind of problem goes to which kind of model.

This part follows one path in four stages, drawn below: 1. map the field; 2. reuse, don’t restart; 3. find the classical inside; 4. route the problem. Each stage links to its section, and in the slides a strip at the bottom shows which stage you are in.

Fundamental approach slides

Click the slides and use the ← → arrows to navigate; F opens full screen and Esc shows every slide. Open the slides in a new tab

1. A map of the field

Stage 1. Techniques are grouped by the question they answer: prepare the data, find structure without labels, turn text into numbers, predict with lines, neighbors, trees or networks, measure honestly, explain, automate, operate. Two more groups hold what this course did not reach: other problem shapes (sequences, recommendations, rewards, graphs) and the frontier itself. Solid chips were taught here, and the code is the lesson; dashed chips lie beyond. Click a technique to see what it is, what it builds on, what builds on it, and where to read more.

Reading the map

The links between chips are the point of the map. Four walks through it:

• Retrieval-augmented generation builds on k-NN and dense embeddings. A 1951 idea (vote among the nearest points) and a 2013 idea (words as vectors) are the retrieval half of a 2020 system. Step 2 of the lab builds that half.
• Alignment (RLHF) trains a reward model on pairs of answers: the probability that answer A is preferred to answer B is modeled as \(\sigma(r_A - r_B)\), the sigmoid of a score difference. That is a logistic regression, fitted with the same cross-entropy you minimized in the logistic regression lesson. Step 3 of the lab fits one.
• Evaluating LLM systems builds on classification metrics and cross-validation & leakage. A benchmark whose questions leaked into the training data is the same bug as a scaler fitted before the split, at internet scale.
• Tabular foundation models are the one place where the frontier attacks the stronghold of gradient boosting head on. They are discussed in the advanced approach.

The same map as a list

Every node of the map, grouped the same way. This list is the map's source: it works without JavaScript and in print. marks the lesson where a technique was taught, marks what lies beyond this course, marks where to read (a classic paper first, then a modern resource).

2. Transfer learning: stop starting from zero

Stage 2. Classical supervised learning trains each model from scratch on its own labeled data. Transfer learning reuses knowledge: pre-train a large model on a huge generic corpus, then adapt it to your task. There are two ways to adapt:

• feature extraction: freeze the pre-trained model and use its representations as features for a classical head. You did this: sentence embeddings fed to a logistic regression or to a clustering step;
• fine-tuning: continue training some or all of the weights on your labeled data, typically 10²–10⁴ examples instead of 10⁶. LoRA (Hu et al., 2022) trains only a low-rank correction \(W + BA\) of each weight matrix, with \(B\) and \(A\) thin; for a \(4096 \times 4096\) matrix and rank 8, that is \(2 \times 4096 \times 8 = 65{,}536\) trainable numbers instead of \(16{,}777{,}216\), 0.39% of them.

The figure draws that count: the frozen matrix and the two thin matrices that LoRA trains (figure script). The advanced approach repeats the count for a whole 7-billion-parameter model.

This inverted the data economics of the field. ULMFiT (Howard & Ruder, 2018) and BERT (Devlin et al., 2019) showed that a language model pre-trained on unlabeled text, then fine-tuned, beats models trained from scratch on text classification with a fraction of the labels. Tasks that once demanded massive labeled datasets became feasible with hundreds of examples.

3. Foundation models

Scale the recipe: transformer architectures, self-supervised pre-training (predict masked or next tokens: the labels are free, so any text becomes training data), billions of parameters. Something qualitatively new appears. A foundation model (Bommasani et al., 2021) is one very large pre-trained model adapted to many downstream tasks: the GPT family, Claude, Gemini and Llama for text; CLIP and its successors for vision and language; Whisper for speech.

Why scale? Kaplan et al. (2020) measured that the test loss of language models falls as a smooth power law in the number of parameters, the size of the data and the compute, over several orders of magnitude. That regularity made it rational to spend on ever larger training runs. Sutton's "Bitter Lesson" (2019) reads the same history: over 70 years, general methods that exploit more computation have beaten hand-built knowledge.

With large language models, adaptation gets lighter still:

"Classify this support ticket as billing, technical or other": in 2018, a labeling project and a trained classifier; today, often a single prompt (Brown et al., 2020, showed GPT-3 doing new tasks from a few examples in the prompt). Prompting became the new fit() for a broad class of language tasks. What did not change: you still need a labeled test set to know whether the prompt works.

4. The classical toolbox in an LLM world

Stage 3. Look inside the modern stack and this course is everywhere:

• the output layer of an LLM is a softmax regression over the vocabulary; training is mini-batch gradient descent with weight decay (an L2 penalty, as in Ridge) on cross-entropy;
• RAG systems are k-NN over embeddings; a vector database is an approximate nearest-neighbor engine;
• the reward model of RLHF is a logistic regression on pairs of answers;
• evaluating an LLM system is precision/recall thinking plus honest held-out design; test questions that leak into training data are the field's current leakage problem;
• BERTopic already composed foundation-model embeddings with UMAP, HDBSCAN and TF-IDF.

Where classical ML still wins

Stage 4. Reach for the classical stack, not a foundation model, when:

The first row has a challenger: TabPFN (Hollmann et al., 2023, 2025), a transformer pre-trained on millions of synthetic tables, which predicts a new small table in a single forward pass, with no training and no tuning. On classification tables of up to 10,000 rows it is worth testing against boosted trees, the way you tested every model in this course; the advanced approach shows how it works and how to test it.

Routing: what to use when

The frontier practitioner's skill is routing: perception and language go to foundation models, structured prediction goes to the classical stack, and real systems use both. An LLM reads the free-text complaint, a gradient-boosted model scores the churn risk, and MLOps monitors both. In practice:

• free text, images or sound, few labels: a foundation model. Start with a prompt (zero- or few-shot); with a few hundred labels, embeddings plus a classical head (feature extraction); fine-tune (LoRA) when neither is good enough;
• the answer must come from your own documents: RAG, a k-NN over the embeddings of those documents feeding the prompt;
• a table of numbers and categories: the classical stack. Gradient boosting as the default; a logistic regression when the decision must be audited or scored in microseconds;
• both kinds of input: both models. The foundation model turns the text into features or labels; the classical model takes them as columns and decides;
• whatever the route: a labeled test set the system never saw, and the same honest validation as in the cross-validation lesson.

5. Summary: the frontier in five lines

1. Every technique has a place on the map, and most of the frontier builds on chips this course taught: k-NN, embeddings, logistic regression, gradient descent, cross-validation.
2. Transfer learning reuses a pre-trained model: frozen, as a feature extractor for a classical head, or fine-tuned on your labels (LoRA trains a thin slice of each matrix).
3. Foundation models are pre-trained on self-supervised tasks at scale; an LLM can be adapted with a prompt, a few examples in the prompt, retrieved passages (RAG) or fine-tuning.
4. Inside the LLM stack sits this course: a softmax output layer, mini-batch gradient descent, k-NN retrieval, a logistic reward model, and leakage as the evaluation problem.
5. Route the problem: language and perception to foundation models, tables to the classical stack, real systems to both, and always a labeled test set.

Fundamental lab — route the problem

The problem. A company has hired you to decide, for each new request that reaches the data team, which part of the toolbox gets it. Before routing anything, you check with your own code two claims of this lesson that the routing depends on: that the retrieval half of RAG is a k-NN, and that the reward model of RLHF is a logistic regression. The checks use tiny artificial data, built so that each one shows a single mechanism; they are not benchmarks.

Step 1 — find your course on the map

Predict. Of all the chips on the map, which one do the most other chips build on: a model, a representation or a way of validating? Make a guess before clicking.

Use the map: the green ring shows what builds on the chip you click, and the panel lists it under leads to. Click k-nearest neighbors, then Logistic & softmax regression, then Perceptron, MLP & backpropagation, Dense embeddings and Cross-validation & data leakage, and count the green rings of each.

Step 2 — the retrieval half of RAG is a k-NN

Predict. The question below shares the words choose and k-NN with one of the six passages and no word with most of the others. Which passage comes back first? How similar do you expect the second one to be?

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.neighbors import NearestNeighbors

passages = [
    "k-NN predicts from the k closest training rows; choose k by cross-validation.",
    "Gradient boosting adds small trees one at a time, each fitted to the errors of the previous ones.",
    "Fit the scaler on the training split only, or information leaks from the test set.",
    "Logistic regression turns a score into a probability with the sigmoid.",
    "A random forest averages many deep trees, each grown on a bootstrap sample.",
    "PCA rotates the data onto the directions of largest variance.",
]
vec = TfidfVectorizer()
P = vec.fit_transform(passages)                      # one sparse vector per passage
index = NearestNeighbors(n_neighbors=2, metric='cosine').fit(P)

def retrieve(question):
    dist, idx = index.kneighbors(vec.transform([question]))
    for d, i in zip(dist[0], idx[0]):
        print(f"{1 - d:.2f}  {passages[i]}")

retrieve("How do I choose k in k-NN?")

0.46  k-NN predicts from the k closest training rows; choose k by cross-validation.
0.00  Gradient boosting adds small trees one at a time, each fitted to the errors of the previous ones.

The question becomes a vector, and kneighbors returns the passages whose vectors are closest by cosine: the NearestNeighbors of the k-NN lesson, over the TF-IDF vectors of the text representation lesson. The k-NN passage comes back with a cosine similarity of 0.46; the runner-up has 0.00, because it shares no word with the question, and in TF-IDF two texts without a common word are orthogonal. A RAG system does exactly this and then pastes the retrieved passages into the LLM's prompt, so the answer can quote your documents. Real systems use dense embeddings and an approximate index instead of TF-IDF and brute force; the simulation shows why.

Simulation: the same question, in other words. Ask the same thing without the words of the passage: "How many neighbors should vote?" Which passage comes back now, and why?

retrieve("How many neighbors should vote?")

0.32  A random forest averages many deep trees, each grown on a bootstrap sample.
0.00  Gradient boosting adds small trees one at a time, each fitted to the errors of the previous ones.

Step 3 — the reward model is a logistic regression

Predict. In the data below, people's votes depend on whether an answer is correct and, a little, on whether it is polite, but not on its length. A logistic regression without intercept is fitted on the differences between the features of the two answers of each pair. Which weights do you expect it to find? Why can it do without an intercept?

from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(0)
n_pairs = 2000


def answers(n):
    # each answer described by three features
    return np.column_stack([rng.integers(0, 2, n),      # correct: 0 or 1
                            rng.random(n),              # polite: from 0 to 1
                            rng.random(n) * 5])         # length: from 0 to 5 hundred words


A, B = answers(n_pairs), answers(n_pairs)
w_people = np.array([3.0, 1.0, 0.0])                    # what people value: correctness, a bit of politeness, not length
p_A = 1 / (1 + np.exp(-(A - B) @ w_people))             # P(A preferred to B) = sigmoid(r_A - r_B)
prefers_A = (rng.random(n_pairs) < p_A).astype(int)     # one human vote per pair
print(f"{n_pairs} compared pairs, A preferred in {prefers_A.mean():.0%} of them")

reward = LogisticRegression(fit_intercept=False, C=np.inf).fit(A - B, prefers_A)
print("weights of the reward model:", reward.coef_[0].round(2))

best, worst = np.array([[1, 1.0, 1.0]]), np.array([[0, 0.0, 1.0]])
print(f"P(correct and polite beats wrong and rude) = {reward.predict_proba(best - worst)[0, 1]:.2f}")

2000 compared pairs, A preferred in 50% of them
weights of the reward model: [2.98 1.12 0.03]
P(correct and polite beats wrong and rude) = 0.98

The reward of an answer is a weighted sum of its features, \(r = w \cdot x\), and the probability that people prefer A to B is \(\sigma(r_A - r_B) = \sigma(w \cdot (x_A - x_B))\): a logistic regression on the difference of the two answers, fitted with cross-entropy (C=np.inf turns off the penalty, as in the logistic regression lesson). From 2,000 votes it recovers the weights people used, 2.98, 1.12 and 0.03 against 3, 1 and 0. No intercept is needed, and none should be there: swapping A and B must turn \(p\) into \(1 - p\), which \(\sigma(-z) = 1 - \sigma(z)\) guarantees only when there is no constant term (an intercept would favor whichever answer is shown first). In RLHF the features are not three columns but the internal representation of an LLM, and the reward model is a network with this same logistic output; the LLM is then tuned to produce answers the reward model scores high.

Simulation: voters who like long answers. Repeat the votes with the same pairs, but now with people who also like long answers: weights [3.0, 1.0, 0.8]. Refit the reward model. How likely is a short correct answer (50 words) to beat a long wrong one (500 words, equally polite), according to the fair reward model and according to the new one?

rng = np.random.default_rng(0)                          # same seed: the same pairs and the same random draws
A, B = answers(n_pairs), answers(n_pairs)
w_biased = np.array([3.0, 1.0, 0.8])
p_A = 1 / (1 + np.exp(-(A - B) @ w_biased))
prefers_A = (rng.random(n_pairs) < p_A).astype(int)
biased = LogisticRegression(fit_intercept=False, C=np.inf).fit(A - B, prefers_A)
print("weights, length-biased votes:", biased.coef_[0].round(2))
short_right, long_wrong = np.array([[1, 1.0, 0.5]]), np.array([[0, 1.0, 5.0]])
print(f"P(short and correct beats long and wrong): fair {reward.predict_proba(short_right - long_wrong)[0, 1]:.2f}"
      f" · biased {biased.predict_proba(short_right - long_wrong)[0, 1]:.2f}")

weights, length-biased votes: [3.02 1.03 0.76]
P(short and correct beats long and wrong): fair 0.95 · biased 0.40

Step 4 — route six requests

Predict. For each request below, decide with your partner: a foundation model, the classical stack, or both? Write one sentence of why, and how you would test the choice, before opening the solution.

To close the fundamental part. In one sentence each: (1) name two pieces of this course that sit inside an LLM system, and where; (2) what separates feature extraction, fine-tuning and prompting? (3) Why does a prompt still need a labeled test set?

Your turn (fundamental)

At home:

1. Three dashed chips. Pick three dashed chips on the map. For each, write what it builds on and in which lesson of this course you learned that.
2. Dense retrieval. Redo step 2 with the sentence embeddings of the sentence embeddings lesson instead of TF-IDF (NearestNeighbors(metric='cosine') works the same on dense vectors). Does "How many neighbors should vote?" now find the k-NN passage?
3. An intercept for the first answer. In step 3, make people slightly prefer whichever answer is shown first (add 0.5 to the score difference in p_A) and fit the reward model with fit_intercept=True. What does the intercept learn?

Advanced approach

The fundamental part placed the frontier on the map and routed problems to the right tool. This part opens three of the frontier's boxes: LoRA, counted for a whole model, and why a thin slice of each matrix can be enough; TabPFN, the challenger of boosted trees on small tables, and how to test it; and the open problems. The code blocks are independent of those of the fundamental part.

This part follows one path in three stages, drawn below: 1. fine-tune a thin slice; 2. challenge the trees; 3. face the open problems. Each stage links to its section, and in the slides a strip at the bottom shows which stage you are in.

Advanced approach slides

Click the slides and use the ← → arrows to navigate; F opens full screen and Esc shows every slide. Open the slides in a new tab

6. LoRA: fine-tune a thin slice

Stage 1. A layer of a network computes \(h = Wx\), with \(W\) a matrix of \(d_{out} \times d_{in}\) pre-trained weights. Full fine-tuning learns a new matrix \(W + \Delta W\), with \(\Delta W\) free: \(d_{out} \cdot d_{in}\) numbers to train, for every matrix of the model. LoRA (low-rank adaptation; Hu et al., 2022) keeps \(W\) frozen and forces the update to be a product of two thin matrices:

Only \(B\) and \(A\) are trained: \(r\,(d_{in} + d_{out})\) numbers instead of \(d_{in}\, d_{out}\). \(A\) starts random and \(B\) starts at zero, so \(BA = 0\) and training starts exactly at the pre-trained model; Hu et al. also multiply \(BA\) by a constant \(\alpha / r\), so that changing \(r\) does not change the size of the update. After training, \(BA\) can be added into \(W\) once, and the adapted model costs nothing extra to run.

Counting a whole model

Section 2 counted one \(4096 \times 4096\) matrix. A whole model has hundreds of matrices of two shapes. The code counts them for a model with the published shape of Llama 2 7B (Touvron et al., 2023): 32 layers, each with four \(4096 \times 4096\) attention matrices (\(q\), \(k\), \(v\), \(o\)) and three MLP matrices between 4,096 and 11,008 units, plus the embeddings of a 32,000-token vocabulary:

d, d_ff, n_layers, vocab = 4096, 11008, 32, 32000      # the published shape of Llama 2 7B

# every weight matrix of one transformer layer, as (rows, columns)
layer = {'q': (d, d), 'k': (d, d), 'v': (d, d), 'o': (d, d),               # attention
         'gate': (d, d_ff), 'up': (d, d_ff), 'down': (d_ff, d)}             # MLP
per_layer = sum(r * c for r, c in layer.values()) + 2 * d                  # + two normalization vectors
total = n_layers * per_layer + 2 * vocab * d + d                           # + embeddings, output head, final norm
print(f"one layer {per_layer:,} · whole model {total:,}")


def lora(adapted, r):
    # LoRA trains B (rows x r) and A (r x columns) for each adapted matrix of each layer
    return n_layers * sum(r * (layer[m][0] + layer[m][1]) for m in adapted)


for adapted in (['q', 'v'], ['q', 'k', 'v', 'o'], list(layer)):
    n = lora(adapted, r=8)
    print(f"LoRA r=8 on {'+'.join(adapted):22s} {n:>11,} trained · {n / total:.3%} of the model")

one layer 202,383,360 · whole model 6,738,415,616
LoRA r=8 on q+v                      4,194,304 trained · 0.062% of the model
LoRA r=8 on q+k+v+o                  8,388,608 trained · 0.124% of the model
LoRA r=8 on q+k+v+o+gate+up+down    19,988,480 trained · 0.297% of the model

The model has 6,738,415,616 weights, the "7B" of its name; one layer holds 202,383,360 of them, two thirds in the MLP. Hu et al. adapted only \(q\) and \(v\), and found that enough on their tasks: at rank 8 that is 4,194,304 trained numbers, 0.062% of the model. Adapting every matrix of every layer still trains less than 0.3%.

The simulator walks through the same count, one piece of the model per step: one matrix, the attention of one layer, the whole layer, 32 layers, the embeddings, and finally the memory that training needs (the subject of lab step 5). Before stepping, predict: as the pieces add up, does the share that LoRA trains grow, shrink or stay the same? Change the rank and the adapted matrices and watch every count follow.

Why a thin slice can be enough

The product \(BA\) has rank at most \(r\): whatever training does, LoRA can only make updates of rank \(\le r\). It works when the update that fine-tuning needs is close to low rank. Aghajanyan et al. (2021) found that fine-tuning a pre-trained language model has a low intrinsic dimension: on some tasks, training a few hundred numbers, projected into the full weight space, reaches 90% of the performance of full fine-tuning, and Hu et al. (2022) built LoRA on the hypothesis that the weight updates have a low intrinsic rank, which their experiments supported: ranks as small as 1 to 8 were competitive with full fine-tuning on their benchmarks. It is an empirical fact about pre-trained models, not a law: lab step 6 shows what happens when the update is not low rank.

7. Tabular foundation models: TabPFN

Stage 2. The routing rule sends tables to the classical stack. This is the one place where a foundation model challenges that rule head on.

Why trees still lead on tables

Grinsztajn, Oyallon & Varoquaux (2022) benchmarked 45 tabular datasets of around 10,000 rows and found tree ensembles ahead of deep networks. They traced the gap to three properties of tables: the target functions are irregular, and networks are biased toward smooth functions; tables carry many uninformative columns, which hurt networks more than trees; and the columns have meanings of their own (age, income), so mixing them, as the first layer of a network does, throws away structure that a tree's one-column questions keep.

A forward pass instead of a fit

TabPFN (Hollmann et al., 2023, 2025) takes a different route. It is a prior-data fitted network (Müller et al., 2022): a transformer pre-trained, once and offline, on millions of synthetic datasets drawn from a broad family of random data-generating processes. Each pre-training example is a whole small table: the training rows with their labels and some test rows without them, and the target is the labels of the test rows. To use it on your table, you give it your training rows and your new rows as input, and it outputs the predictions in one forward pass: no gradient step on your data, no hyperparameter search. It is in-context learning, as in few-shot prompting, with a table in place of the prompt; Müller et al. showed that such a network approximates Bayesian prediction under the prior it was trained on.

On classification datasets of up to 10,000 rows, the 2025 version beat, in 2.8 seconds, an ensemble of the strongest baselines (boosted trees among them) tuned for 4 hours. The price is paid at prediction time: the whole training table is part of the input of every prediction, so the model is large and slow to run next to a fitted tree, and it was designed for small tables (the 2025 version targets up to 10,000 rows and 500 features). Above that size, and whenever latency matters, boosted trees remain the default.

Testing the challenger

Test the challenger the way you tested every model in this course: same folds, same metric, paired comparison, and at the size of your table. Who wins depends on the size: lab step 7 shows a regularized linear model beating gradient boosting on a 300-row sample of California housing and losing on every fold from 1,000 rows on. A benchmark result ("best up to 10,000 rows") is a claim about a range of sizes, measured on other people's tables; your validation is the only test that counts.

8. Open problems — where this field is going

Stage 3. The map has dashed chips because the field is not finished. These problems are open:

• Hallucination and reliability: fluent falsehoods. Calibration at frontier scale: a model should know when it does not know.
• Evaluation: benchmarks saturate within a year or two and leak into training data; honest measurement is an arms race (Validation, scaled up).
• Alignment and safety: making systems pursue the goals we intend; RLHF and its successors are partial answers. The simulation of lab step 3 showed a reward model faithfully learning a bias of its voters.
• Efficiency: distillation, quantization, small specialized models against giant generalists.
• Data provenance, bias and governance: the ethics questions of the ML landscape lesson, now industrialized and regulated (EU AI Act; in Brazil, the LGPD and the AI bill PL 2338/2023, approved by the Senate in December 2024 and sent to the Chamber of Deputies).
• Causality: prediction answers "what will happen"; decisions need "what happens if we act". Causal ML joins the two.
• Agents: models that plan, call tools and act. How to evaluate them and keep them safe is largely unsolved.

9. Summary: inside the frontier's tools

1. LoRA trains \(B\) and \(A\), \(r\,(d_{in} + d_{out})\) numbers per adapted matrix, with \(W\) frozen: 0.062% of a 7-billion-parameter model at rank 8 on \(q\) and \(v\).
2. It saves memory and storage more than compute: the frozen copy still has to be held, but each task is a small adapter.
3. It works because fine-tuning updates are close to low rank in pre-trained models; an update that is not low rank cannot be captured by a thin slice.
4. TabPFN predicts a small table in one forward pass of a transformer pre-trained on synthetic tables; boosted trees remain the default for larger tables and tight latency.
5. A challenger is tested like any model: your table, your size, the same folds, fold by fold.

Advanced lab — count, compress, test

The problem. Your team wants to fine-tune a 7-billion-parameter model for a customer-service task, and someone proposes replacing the boosted trees of a small tabular model with a tabular foundation model. Before the meeting, you check three things with code: what fine-tuning costs in memory, when a low-rank update is enough, and how much the winner of a comparison depends on the size of the table.

Step 5 — what fine-tuning costs in memory

Predict. LoRA trains 0.062% of the numbers of the model. Does it need 0.062% of the memory of full fine-tuning? Much more? Why?

GB = 1e9
trained_bytes = 16          # mixed-precision Adam: 16-bit weight and gradient, 32-bit master copy and two moments
frozen_bytes = 2            # a frozen weight is only read: one 16-bit copy

full = total * trained_bytes
n_lora = lora(['q', 'v'], r=8)
with_lora = total * frozen_bytes + n_lora * trained_bytes
print(f"full fine-tuning: {full / GB:6.1f} GB · LoRA r=8 on q+v: {with_lora / GB:5.1f} GB")
print(f"saved per task:   {total * 2 / GB:6.1f} GB (a full copy) · {n_lora * 2 / 1e6:.1f} MB (an adapter)")

full fine-tuning:  107.8 GB · LoRA r=8 on q+v:  13.5 GB
saved per task:     13.5 GB (a full copy) · 8.4 MB (an adapter)

Training with the Adam optimizer in mixed precision keeps 16 bytes per trained number (the accounting of Rajbhandari et al., 2020; activations come on top and depend on the batch). Full fine-tuning needs 107.8 GB, more than a single 80 GB accelerator holds. With LoRA, 13.5 GB: 8 times less, not 1,600 times, because the frozen 16-bit copy of the model still has to be in memory, and now it is almost all of it. The other saving is storage: each new task is an adapter of 8.4 MB instead of a 13.5 GB copy of the model, so one base model can serve hundreds of tasks.

Simulation: shrink the frozen part. QLoRA (Dettmers et al., 2023) stores the frozen weights in 4 bits (half a byte) and trains LoRA on top. How much memory does that need? And how much does the rank \(r\) change the memory, from 1 to 64?

four_bit = total * 0.5 + n_lora * trained_bytes
print(f"QLoRA (frozen weights in 4 bits): {four_bit / GB:.1f} GB")
for r in (1, 4, 8, 16, 64):
    print(f"r={r:<3d} {lora(['q', 'v'], r):>11,} trained · {(total * frozen_bytes + lora(['q', 'v'], r) * trained_bytes) / GB:.2f} GB")

QLoRA (frozen weights in 4 bits): 3.4 GB
r=1       524,288 trained · 13.49 GB
r=4     2,097,152 trained · 13.51 GB
r=8     4,194,304 trained · 13.54 GB
r=16    8,388,608 trained · 13.61 GB
r=64   33,554,432 trained · 14.01 GB

Step 6 — when is a thin slice enough?

Predict. The update delta below has rank 4, plus a little noise. The function measures the best that any \(B \cdot A\) of rank \(r\) can do (the truncated SVD, by the Eckart–Young theorem), as the size of what is left over relative to the size of the update. How does that error change from \(r = 1\) to \(r = 32\)?

import numpy as np

rng = np.random.default_rng(0)
d = 512
# an update that really has rank 4, plus a little noise
delta = rng.normal(size=(d, 4)) @ rng.normal(size=(4, d)) + 0.1 * rng.normal(size=(d, d))


def best_rank_r_error(M, r):
    # the best any B (d x r) times A (r x d) can do: keep the r largest singular values (Eckart-Young)
    U, s, Vt = np.linalg.svd(M)
    approx = (U[:, :r] * s[:r]) @ Vt[:r]
    return np.linalg.norm(M - approx) / np.linalg.norm(M)


print(f"full update: {d * d:,} numbers")
for r in (1, 2, 4, 8, 32):
    print(f"r={r:<3d} {2 * d * r:>7,} numbers · error {best_rank_r_error(delta, r):.3f}")

full update: 262,144 numbers
r=1     1,024 numbers · error 0.841
r=2     2,048 numbers · error 0.667
r=4     4,096 numbers · error 0.050
r=8     8,192 numbers · error 0.049
r=32   32,768 numbers · error 0.045

The error falls slowly at \(r = 1\) and 2, then drops to 0.050 at \(r = 4\), the true rank, and stops there: what is left (0.049, 0.045) is the noise, which no low-rank slice can capture and none needs to. With 4,096 numbers instead of 262,144 (1.6%), the slice reproduces the update up to the noise. A larger \(r\) only buys more numbers. This is the situation LoRA bets on.

Simulation: an update with no structure. Replace delta by a matrix of pure noise, rng.normal(size=(d, d)), which has full rank. What is the error now at \(r = 1, 4, 8, 32\) and \(128\)?

noise = rng.normal(size=(d, d))
for r in (1, 4, 8, 32, 128):
    print(f"r={r:<3d} {2 * d * r:>7,} numbers · error {best_rank_r_error(noise, r):.3f}")

r=1     1,024 numbers · error 0.996
r=4     4,096 numbers · error 0.985
r=8     8,192 numbers · error 0.970
r=32   32,768 numbers · error 0.888
r=128 131,072 numbers · error 0.613

Step 7 — who wins depends on the size of the table

Predict. On samples of 100 to 10,000 rows of California housing, a regularized linear model (RidgeCV, with imputation and scaling) is compared with gradient boosting on the same five folds. At 100 rows, which do you expect to win? And at 10,000?

import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.impute import SimpleImputer
from sklearn.linear_model import RidgeCV
from sklearn.model_selection import KFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

housing = pd.read_csv('https://raw.githubusercontent.com/hsandmann/biblio/refs/heads/main/ml/aula02/housing.csv')
X_all = housing.drop(columns=['median_house_value', 'ocean_proximity'])
y_all = housing['median_house_value']
print(X_all.shape)

linear = make_pipeline(SimpleImputer(strategy='median'), StandardScaler(), RidgeCV())
boosting = HistGradientBoostingRegressor(random_state=0)
cv = KFold(n_splits=5, shuffle=True, random_state=0)

for n in (100, 300, 1000, 3000, 10000):
    sample = X_all.sample(n, random_state=0)
    X_n, y_n = sample, y_all.loc[sample.index]
    rmse = {}
    for name, model in (('linear', linear), ('boosting', boosting)):
        rmse[name] = -cross_val_score(model, X_n, y_n, cv=cv, scoring='neg_root_mean_squared_error')
    wins = (rmse['boosting'] < rmse['linear']).sum()
    print(f"n={n:>6,}  linear {rmse['linear'].mean():>7,.0f}  boosting {rmse['boosting'].mean():>7,.0f}"
          f"  boosting wins {wins} of 5 folds")

(20640, 8)
n=   100  linear  60,842  boosting  71,432  boosting wins 1 of 5 folds
n=   300  linear  64,830  boosting  71,952  boosting wins 0 of 5 folds
n= 1,000  linear  69,439  boosting  58,184  boosting wins 5 of 5 folds
n= 3,000  linear  69,139  boosting  53,752  boosting wins 5 of 5 folds
n=10,000  linear  70,025  boosting  49,547  boosting wins 5 of 5 folds

The winner flips between 300 and 1,000 rows. At 100 and 300 rows the linear model has the lower RMSE (in dollars of median house value) and boosting wins at most one fold: with so few rows, a flexible model has more variance than signal to fit. From 1,000 rows on, boosting wins every fold, and its error keeps falling (58,184 → 49,547) while the linear model's stays near 70,000, the limit of a straight line. The bias–variance trade-off decides, and the size of the table moves it. This is why the claim of a tabular foundation model is a claim about a range of sizes, and why the only test that settles it is the one on your table, at your size, on the same folds, fold by fold.

To close the lesson. In one sentence each: (1) why does LoRA cut the memory of fine-tuning 8 times and not 1,600 times? (2) When would a rank-8 update fail to adapt a model? (3) How would you decide between TabPFN and boosted trees for a 5,000-row table?

Your turn (advanced)

At home, with more time:

1. LoRA everywhere. With the code of section 6 and the memory accounting of step 5, how many numbers does LoRA train at \(r = 16\) on all seven matrices of each layer, and how much memory does it need, with the frozen weights in 16 bits and in 4 bits?
2. A third contender. In step 7, add the whole table (n = 20640) and a RandomForestRegressor(n_estimators=200, random_state=0, n_jobs=-1). At which sizes does each model win, fold by fold?
3. Read the challenger. Read Hollmann et al. (2025) in the references and list the kinds of tables TabPFN was not designed for. For each, say which model of this course you would use instead.

Where to go from here

• Deep learning, in depth: the Insper course Artificial Neural Networks and Deep Learning (CNNs, transformers, generative models); Prince, Understanding Deep Learning (2023, free online).
• The probabilistic view: Murphy, Probabilistic Machine Learning (2022, free online), for the Bayesian models, Gaussian processes and uncertainty this course only touched.
• LLMs, hands on: the Hugging Face courses (tokenizers, fine-tuning, RAG); Karpathy's Neural Networks: Zero to Hero, which builds a GPT from scratch.
• Reading papers: pick a dashed chip on the map, read its classic paper, then find a recent paper that cites it. The map's links are the starting points.

References

The map carries one to three references per technique. These are the ones the text of this lesson relies on.

• Sutton, R. "The Bitter Lesson." Essay (2019). Link
• Vaswani, A. et al. "Attention Is All You Need." NeurIPS (2017). arXiv:1706.03762
• Howard, J.; Ruder, S. "Universal Language Model Fine-tuning for Text Classification." ACL (2018). arXiv:1801.06146
• Devlin, J.; Chang, M.-W.; Lee, K.; Toutanova, K. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." NAACL-HLT (2019). arXiv:1810.04805
• Kaplan, J. et al. "Scaling Laws for Neural Language Models." arXiv (2020). arXiv:2001.08361
• Brown, T. B. et al. "Language Models are Few-Shot Learners." NeurIPS (2020) — GPT-3. arXiv:2005.14165
• Lewis, P. et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS (2020). arXiv:2005.11401
• Rajbhandari, S.; Rasley, J.; Ruwase, O.; He, Y. "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models." SC20 (2020) — the 16 bytes per trained parameter of mixed-precision Adam. arXiv:1910.02054
• Aghajanyan, A.; Gupta, S.; Zettlemoyer, L. "Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning." ACL (2021). arXiv:2012.13255
• Bommasani, R. et al. "On the Opportunities and Risks of Foundation Models." arXiv (2021). arXiv:2108.07258
• Hu, E. J. et al. "LoRA: Low-Rank Adaptation of Large Language Models." ICLR (2022). arXiv:2106.09685
• Müller, S.; Hollmann, N.; Pineda Arango, S.; Grabocka, J.; Hutter, F. "Transformers Can Do Bayesian Inference." ICLR (2022) — prior-data fitted networks. arXiv:2112.10510
• Ouyang, L. et al. "Training Language Models to Follow Instructions with Human Feedback." NeurIPS (2022) — RLHF. arXiv:2203.02155
• Grinsztajn, L.; Oyallon, E.; Varoquaux, G. "Why Do Tree-Based Models Still Outperform Deep Learning on Typical Tabular Data?" NeurIPS Datasets and Benchmarks (2022). arXiv:2207.08815
• Hollmann, N.; Müller, S.; Eggensperger, K.; Hutter, F. "TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second." ICLR (2023). arXiv:2207.01848
• Touvron, H. et al. "Llama 2: Open Foundation and Fine-Tuned Chat Models." arXiv (2023) — the shape counted in section 6. arXiv:2307.09288
• Dettmers, T.; Pagnoni, A.; Holtzman, A.; Zettlemoyer, L. "QLoRA: Efficient Finetuning of Quantized LLMs." NeurIPS (2023). arXiv:2305.14314
• Sainz, O. et al. "NLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for each Benchmark." Findings of EMNLP (2023). arXiv:2310.18018
• Hollmann, N. et al. "Accurate Predictions on Small Data with a Tabular Foundation Model." Nature 637 (2025): 319–326. DOI
• Regulation (EU) 2024/1689 (Artificial Intelligence Act). Official Journal of the European Union (2024). EUR-Lex

Quiz

The quiz covers the fundamental approach.