Machine Learning · Insper
The Frontier
The last stop of an arc that began with least squares in 1805
A map of the field, transfer learning and foundation models, where the classical toolbox still wins
The path of this lesson
- Map the fieldevery technique by the job it does; four walks
- Reuse, don’t restarttransfer learning, fine-tuning, foundation models, prompting
- Where classical winsthe toolbox inside the new stack, tables, routing
- What comes nextopen problems, where to go from here
Each stage answers one question: where does each technique you learned sit? → why start from zero when a model already knows so much? → when does the classical toolbox still win? → what is unsolved, and where do you go next?
This lesson has no lab.
Stage 1 · where does each technique you learned sit?
A map of the field
Solid chips were taught in the course, dashed ones lie beyond. Click a technique: what it is, what it builds on, what builds on it, where to read more. The map scrolls.
Reading the map: four walks
retrieval-augmented generation
builds on k-NN and dense embeddings: a 1951 idea and a 2013 idea are the retrieval half of a 2020 system
alignment (RLHF)
the reward model: \(P(\text{A preferred to B}) = \sigma(r_A - r_B)\), a logistic regression fitted with cross-entropy
evaluating LLM systems
classification metrics and leakage: a benchmark whose questions leaked into training is a scaler fitted before the split, at internet scale
tabular foundation models
the one place where the frontier attacks gradient boosting head on
Stage 2 · why start from zero when a model already knows so much?
Transfer learning: stop starting from zero
Pre-train a large model on a huge generic corpus, then adapt it to your task.
feature extraction
freeze the model, use its representations as features for a classical head. You did this: sentence embeddings fed to a logistic regression or to clustering.
fine-tuning
continue training some or all weights on your labeled data: typically 10²–10⁴ examples instead of 10⁶.
ULMFiT (2018) and BERT (2019): pre-trained on unlabeled text, then fine-tuned, they beat models trained from scratch with a fraction of the labels.
LoRA: fine-tune a thin slice
Train only a low-rank correction of each weight matrix: \(W + BA\), with \(B\) and \(A\) thin (Hu et al., 2022).
Predict. A \(4096 \times 4096\) weight matrix and rank 8. How many numbers does LoRA train?
LoRA, rank 8
65,536
\(2 \times 4096 \times 8\): 0.4% of them
Foundation models
- Transformers, self-supervised pre-training (predict masked or next tokens: the labels are free), billions of parameters.
- One very large model adapted to many tasks (Bommasani et al., 2021): GPT, Claude, Gemini, Llama for text; CLIP for vision and language; Whisper for speech.
- Why scale? Test loss falls as a smooth power law in parameters, data and compute (Kaplan et al., 2020).
Sutton's "Bitter Lesson" (2019): over 70 years, general methods that exploit more computation have beaten hand-built knowledge.
With LLMs, adaptation gets lighter still
| adaptation | labeled data needed | what happens |
|---|
| zero-shot | none | describe the task in the prompt |
| few-shot / in-context | a handful | examples in the prompt; no weights change |
| fine-tuning (full / LoRA) | hundreds+ | update the weights, or a low-rank slice |
| RAG | none (needs documents) | retrieve passages by nearest-neighbor search over embeddings, put them in the context |
"Classify this ticket as billing, technical or other": in 2018 a labeling project; today often one prompt. Prompting became the new fit(), and you still need a labeled test set to know whether it works.
Stage 3 · when does the classical toolbox still win?
The classical toolbox, inside the LLM stack
- The output layer is a softmax regression over the vocabulary; training is mini-batch gradient descent with weight decay (an L2 penalty) on cross-entropy.
- RAG systems are k-NN over embeddings; a vector database is an approximate nearest-neighbor engine.
- The reward model of RLHF is a logistic regression on pairs of answers.
- Evaluating an LLM system is precision/recall thinking plus honest held-out design; leaked test questions are the field's current leakage problem.
- BERTopic already composed foundation-model embeddings with UMAP, HDBSCAN and TF-IDF.
Where classical ML still wins
Predict. A table of churn data, a credit decision in milliseconds, an auditor who must see why. Foundation model or the classical stack?
| situation | why classical wins |
|---|
| tabular data | gradient boosting still leads on medium-sized tables (Grinsztajn et al., 2022); LLMs are poor at tables of numbers |
| latency, cost, scale | a logistic regression scores a row in microseconds |
| regulated decisions | odds ratios and SHAP values can be shown to an auditor; a prompt cannot |
| small, structured problems | 2,000 rows need bias–variance control, not a trillion parameters |
| determinism | fixed model + fixed input = fixed output; LLMs sample |
The challenger on tables: TabPFN
- 45 tabular datasets of around 10,000 rows: tree ensembles ahead of deep networks. Tables have irregular target functions and many uninformative columns.
- TabPFN (Hollmann et al., 2023, 2025): a transformer pre-trained on millions of synthetic tables, which predicts a new table in one forward pass, with no training and no tuning.
up to 10,000 rows
the 2025 version beat, in 2.8 seconds, an ensemble of the strongest baselines tuned for 4 hours
above that, or when latency matters
boosted trees remain the default
Test the challenger the way you tested every model: same folds, same metric, paired comparison.
The frontier practitioner's skill: routing
perception and language
→ foundation models
structured prediction
→ the classical stack
An LLM reads the free-text complaint, a gradient-boosted model scores the churn risk, and MLOps monitors both.
Stage 4 · what is unsolved, and where do you go next?
Open problems: where the field is going
- Hallucination and reliability: calibration at frontier scale; a model should know when it does not know.
- Evaluation: benchmarks saturate and leak into training data.
- Alignment and safety: RLHF and its successors are partial answers.
- Efficiency: distillation, quantization, small specialized models.
- Data provenance, bias and governance: the ethics questions, now regulated (EU AI Act; LGPD and PL 2338/2023 in Brazil).
- Causality: decisions need "what happens if we act".
- Agents: models that plan, call tools and act; evaluating them and keeping them safe is largely unsolved.
Where to go from here
- Deep learning, in depth: the Insper course Artificial Neural Networks and Deep Learning; Prince, Understanding Deep Learning (2023, free online).
- The probabilistic view: Murphy, Probabilistic Machine Learning (2022, free online).
- LLMs, hands on: the Hugging Face courses; Karpathy's Neural Networks: Zero to Hero.
- Reading papers: pick a dashed chip on the map, read its classic paper, then a recent paper that cites it.
The course thesis, one last time
Every "new" idea you will meet is a composition of things you now understand: losses, gradients, regularization, embeddings, neighbors, ensembles, honest validation.
The frontier moves; the foundations compound. Learn the parts, and no whole will be a black box.