Machine Learning · Insper

The Frontier

Advanced approach

Inside the frontier's tools: LoRA, TabPFN and the problems still open

The path of this part

Fine-tune a thin sliceLoRA counted layer by layer; why low rank is enough
Challenge the treeswhy trees lead on tables; TabPFN; a fair test
Face the open problemsreliability, evaluation, alignment, efficiency, governance

Each stage answers one question: how few numbers can adapt a 7-billion-parameter model? → can a foundation model beat trees on tables? → what is still unsolved?

Stage 1 · how few numbers can adapt a 7-billion-parameter model?

LoRA: freeze \(W\), train \(B\) and \(A\)

The frozen matrix W plus the thin matrices B and A that LoRA trains

A layer computes \(h = Wx\): input \(x\), output \(h\), pre-trained weights \(W\) of \(d_{out} \times d_{in}\).

\[ h = Wx + B\,(Ax) \]

\(B\): \(d_{out} \times r\) · \(A\): \(r \times d_{in}\) · rank \(r \ll d\)

trained: \(r\,(d_{in} + d_{out})\) instead of \(d_{in}\, d_{out}\)

LoRA = low-rank adaptation (Hu et al., 2022). \(B\) starts at zero: training starts exactly at the pre-trained model. After training, \(BA\) is added into \(W\) once: no extra cost to run.

LoRA, counted layer by layer

Predict. A model with the shape of Llama 2 7B. As the pieces add up (one matrix, a layer, 32 layers, the embeddings), does the share LoRA trains grow, shrink or stay the same? And the memory?

Why a thin slice can be enough

\(BA\) has rank at most \(r\): LoRA can only learn updates of rank \(\le r\). Predict: for an update of true rank 4, what happens to the error past \(r = 4\)? And for pure noise?

Error of the best rank-r slice: an update of rank 4 drops to 0.050 at r = 4 and stays flat; pure noise stays near 1 until large r

Fine-tuning updates of pre-trained models are close to low rank (Aghajanyan et al., 2021; Hu et al., 2022): an empirical fact, not a law.

Stage 2 · can a foundation model beat trees on tables?

Why trees still lead on tables

Grinsztajn, Oyallon & Varoquaux (2022): 45 tabular datasets of around 10,000 rows, tree ensembles ahead of deep networks. Three reasons:

irregular targets

tables have jumps and thresholds; networks are biased toward smooth functions

useless columns

many uninformative columns hurt networks more than trees

columns mean something

a network's first layer mixes age with income; a tree asks about one column at a time

TabPFN: a forward pass instead of a fit

boosted trees

on your table: grow hundreds of trees, tune the hyperparameters with cross-validation, then predict

TabPFN (Hollmann et al., 2023, 2025)

a transformer pre-trained once on millions of synthetic tables. Input: your training rows and your new rows. Output: the predictions, in one forward pass

PFN = prior-data fitted network (Müller et al., 2022): in-context learning, as in few-shot prompting, with a table in place of the prompt.
Classification up to 10,000 rows: the 2025 version beat, in 2.8 seconds, an ensemble of the strongest baselines tuned for 4 hours.
Larger tables, or tight latency: boosted trees remain the default.

Testing the challenger: who wins depends on the size

Predict. California housing samples: a regularized linear model against gradient boosting, same five folds. Who wins at 100 rows? At 10,000?

rows	RMSE linear	RMSE boosting	boosting wins
100	60,842	71,432	1 of 5 folds
300	64,830	71,952	0 of 5
1,000	69,439	58,184	5 of 5
3,000	69,139	53,752	5 of 5
10,000	70,025	49,547	5 of 5

RMSE = root mean squared error, in dollars. The winner flips between 300 and 1,000 rows. Test any challenger on your table, at your size, on the same folds, fold by fold.

Stage 3 · what is still unsolved?

Open problems: where the field is going

Hallucination and reliability: calibration at frontier scale; a model should know when it does not know.
Evaluation: benchmarks saturate and leak into training data.
Alignment and safety: RLHF (reinforcement learning from human feedback) and its successors are partial answers; a reward model learns the biases of its voters.
Efficiency: distillation, quantization, small specialized models.

Data provenance, bias and governance: the ethics questions, now regulated (EU AI Act; in Brazil the LGPD, the data protection law, and the AI bill PL 2338/2023).
Causality: decisions need "what happens if we act".
Agents: models that plan, call tools and act; evaluating them and keeping them safe is largely unsolved.

Summary

LoRA trains \(r\,(d_{in} + d_{out})\) numbers per adapted matrix: 0.062% of a 7B model at \(r = 8\) on \(q\) and \(v\).
It saves memory and storage: 107.8 GB → 13.5 GB to train; an 8.4 MB adapter per task. The frozen copy now dominates.
It works because fine-tuning updates are close to low rank; a full-rank update does not fit a thin slice.
TabPFN predicts a small table in one forward pass; boosted trees remain the default for large tables and tight latency.
A challenger is tested like any model: your table, your size, the same folds.

Where to go from here

Deep learning, in depth: the Insper course Artificial Neural Networks and Deep Learning; Prince, Understanding Deep Learning (2023, free online).
The probabilistic view: Murphy, Probabilistic Machine Learning (2022, free online).
LLMs, hands on: the Hugging Face courses; Karpathy's Neural Networks: Zero to Hero.
Reading papers: pick a dashed chip on the map, read its classic paper, then a recent paper that cites it.

The course thesis, one last time: every "new" idea is a composition of things you now understand. The frontier moves; the foundations compound. Learn the parts, and no whole will be a black box.

Advanced lab · about 30 min, in pairs

Count, compress and test, steps 5 to 7

Same Colab notebook (or a new one): step 5 uses total and lora from the counting block of section 6; run it first.
Step 5: what fine-tuning costs in memory · step 6: when is a thin slice enough? · step 7: who wins depends on the size of the table.
Simulations: shrink the frozen part (step 5, QLoRA in 4 bits) and an update with no structure (step 6).
Step 7 fits 50 models: about two minutes.

Pace: step 5 and its simulation in 10 min; step 6 and its simulation in 10 min; step 7 in 10 min.