The Frontier

The last stop of the arc that began with least squares in 1805. This lesson has no lab. Its centerpiece is a map of the field: every technique of the previous 26 lessons, grouped by the job it does, next to the techniques that lie just beyond this course, each with the lesson where it was taught and where to read more. Around the map: how transfer learning and foundation models changed the economics of machine learning, where the classical toolbox you built still wins, and the problems the field has not solved yet.

The lesson goes in four stages, drawn below: 1. map the field, every technique of the course in one picture, next to what lies beyond it; 2. reuse, don’t restart, transfer learning and foundation models; 3. where classical wins, the toolbox inside the new stack and the problems it still owns; 4. what comes next, the open problems and where to go from here. Each stage links to its section, and in the slides a strip at the bottom shows which stage you are in.

Map the fieldevery technique by the job it does; four walks
Reuse, don’t restarttransfer learning, fine-tuning, foundation models, prompting
Where classical winsthe toolbox inside the new stack, tables, routing
What comes nextopen problems, where to go from here

Slides

Click the slides and use the ← → arrows to navigate; F goes full screen and Esc shows every slide. Open the slides in a new tab

A map of the field

Stage 1. Techniques are grouped by the question they answer: prepare the data, find structure without labels, turn text into numbers, predict with lines, neighbors, trees or networks, measure honestly, explain, automate, operate. Two more groups hold what this course did not reach: other problem shapes (sequences, recommendations, rewards, graphs) and the frontier itself. Solid chips were taught here, and the number is the lesson; dashed chips lie beyond. Click a technique to see what it is, what it builds on, what builds on it, and where to read more.

Reading the map

The links between chips are the point of the map. Four walks through it:

Retrieval-augmented generation builds on k-NN and dense embeddings. A 1951 idea (vote among the nearest points) and a 2013 idea (words as vectors) are the retrieval half of a 2020 system.
Alignment (RLHF) trains a reward model on pairs of answers: the probability that answer A is preferred to answer B is modeled as \(\sigma(r_A - r_B)\), the sigmoid of a score difference. That is a logistic regression, fitted with the same cross-entropy you minimized in the logistic regression lesson.
Evaluating LLM systems builds on classification metrics and cross-validation & leakage. A benchmark whose questions leaked into the training data is the same bug as a scaler fitted before the split, at internet scale.
Tabular foundation models are the one place where the frontier attacks the stronghold of gradient boosting head on. They are discussed below.

The same map as a list

Every node of the map, grouped the same way. This list is the map's source: it works without JavaScript and in print. marks the lesson where a technique was taught, marks what lies beyond this course, marks where to read (a classic paper first, then a modern resource).

Foundations — what learning from data means

A history of learning machines — From least squares (1805) to the perceptron, two AI winters, deep learning and LLMs: each wave recombines older ideas with more data and compute.
1. Introduction & History · Turing (1950) · Sutton, "The Bitter Lesson" (2019)
Learning paradigms — Supervised (learn from labels), unsupervised (find structure), reinforcement (learn from rewards): the paradigm follows from the feedback the data carries.
2. The ML Landscape · Hastie, Tibshirani & Friedman (2009) · James et al. (2023)
Generalization & no free lunch — What matters is the error on data the model has not seen, and no algorithm wins on every problem: the assumptions must match the data. Builds on learning paradigms.
2. The ML Landscape · Wolpert (1996) · Domingos (2012)

Data — from raw records to a model-ready matrix

Exploratory data analysis — Look before modeling: distributions, missing values, outliers, associations. Split first, then explore only the training part.
3. Exploratory Data Analysis · Anscombe (1973) · Matejka & Fitzmaurice (2017)
Preprocessing — Scale numeric features, encode categories, transform skewed columns and impute missing values, always fitted on the training data only. Builds on EDA.
4. Data Preprocessing · Little & Rubin (2019) · scikit-learn: preprocessing
Pipelines — Chain the preprocessing and the model into one estimator, so that every fit happens inside the right split. Builds on preprocessing.
5. Pipelines · Buitinck et al. (2013) · scikit-learn: pipelines
Data-centric AI & label quality — Better data often beats a better model: find label errors, remove duplicates, document how the dataset was collected. Builds on EDA.
beyond this course · Northcutt, Athalye & Mueller (2021) · Gebru et al. (2021)

Unsupervised learning — structure without labels

PCA — Rotate the data onto the directions of largest variance and keep the first few, to compress, denoise or plot. Builds on preprocessing.
6. Dimensionality Reduction · Pearson (1901) · Jolliffe & Cadima (2016)
t-SNE & UMAP — Non-linear maps that keep each point's neighbors close, for looking at high-dimensional data; the distances between clusters mean little. Builds on PCA, k-NN.
6. Dimensionality Reduction · van der Maaten & Hinton (2008) · McInnes, Healy & Melville (2018)
k-means — Split the rows into k compact groups around centroids; choose k with inertia and silhouette. Builds on preprocessing.
7. Clustering · Lloyd (1982) · Arthur & Vassilvitskii (2007)
Hierarchical clustering — Merge the closest groups one step at a time and read the result as a dendrogram; cut it at the level you need. Builds on k-means.
8. Hierarchical Clustering · Ward (1963) · Sokal & Rohlf (1962)
DBSCAN & HDBSCAN — Clusters as dense regions of any shape; the points in sparse regions are labeled as noise. Builds on k-NN.
7. Clustering · Ester et al. (1996) · Campello, Moulavi & Sander (2013)
Anomaly detection — Flag the rare points that do not fit the rest: isolation forests, density scores, reconstruction errors. Builds on DBSCAN, random forest.
beyond this course · Liu, Ting & Zhou (2008) · scikit-learn: outlier detection
Self-supervised & contrastive learning — Invent a prediction task from the data itself (hide a word, match two crops of the same image) and learn representations without any human label. Builds on dense embeddings, neural networks.
beyond this course · Devlin et al. (2019) · Chen et al. (2020)

Text — turning words into vectors

Bag-of-words & TF-IDF — Count the tokens of each document and down-weight the ones every document uses: sparse, fast, interpretable.
9. Text Representation · Salton & Buckley (1988) · Manning, Raghavan & Schütze (2008)
Dense embeddings — Words and sentences as short dense vectors whose geometry encodes meaning: texts with similar meaning land close together. Builds on bag-of-words.
9. Text Representation · Mikolov et al. (2013) · Reimers & Gurevych (2019)
Topic models: LDA & BERTopic — Find the themes of a corpus: LDA as mixtures of words; BERTopic as clusters of embeddings, described by a class-based TF-IDF. Builds on dense embeddings, UMAP, HDBSCAN, TF-IDF.
10. Topic Modeling & BERTopic · Blei, Ng & Jordan (2003) · Grootendorst (2022)
Subword tokenization — How LLMs cut text: byte-pair encoding builds a vocabulary of frequent fragments, so no word is ever out of the vocabulary. Builds on bag-of-words.
beyond this course · Sennrich, Haddow & Birch (2016) · Hugging Face courses

Linear models — lines, planes and margins

Linear regression — Least squares: the closed-form fit, residuals, R² and the assumptions behind the inferences.
11. Linear Regression · Galton (1886) · James et al. (2023)
Gradient descent — Walk downhill on the loss, one batch or mini-batch at a time: the optimizer behind almost every modern model. Builds on linear regression.
12. Gradient Descent & Regularization · Ruder (2016) · Bottou, Curtis & Nocedal (2018)
Ridge & Lasso — Penalize large weights (L2) or push some of them to exactly zero (L1): a little bias bought for much less variance. Builds on linear regression.
12. Gradient Descent & Regularization · Hoerl & Kennard (1970) · Tibshirani (1996)
Logistic & softmax regression — Probabilities through the sigmoid or the softmax, trained on cross-entropy; the output layer of every classifier network. Builds on linear regression, gradient descent.
13. Logistic Regression · Berkson (1944) · Cox (1958)
Support vector machines & kernels — The boundary with the widest margin, bent into curves by a kernel that never builds the new features. Builds on logistic regression, Ridge.
18. Support Vector Machines · Cortes & Vapnik (1995) · Hsu, Chang & Lin (2003)
Probabilistic & Bayesian models — Distributions over parameters and predictions (naive Bayes, Bayesian regression, Gaussian processes): uncertainty by construction. Builds on linear regression, Ridge.
beyond this course · Rasmussen & Williams (2006) · Murphy (2022)

Instance-based learning — let the neighbors vote

k-nearest neighbors — Keep the training set and predict from the k closest rows; the scaling and the choice of k do all the work. Builds on preprocessing.
14. k-Nearest Neighbors · Fix & Hodges (1951) · Cover & Hart (1967)
Approximate nearest-neighbor search — Graph and quantization indexes (HNSW, FAISS) find the almost-nearest vectors among billions in milliseconds: the engine of vector databases. Builds on k-NN, dense embeddings.
beyond this course · Malkov & Yashunin (2020) · Johnson, Douze & Jégou (2019)

Trees & ensembles — the tabular workhorses

Decision trees — Recursive yes/no questions, each chosen to reduce impurity; easy to read, unstable on their own.
20. Decision Trees · Breiman et al. (1984) · Quinlan (1986)
Random forest — Average hundreds of de-correlated trees, each grown on a bootstrap sample; the out-of-bag error comes for free. Builds on decision trees.
21. Random Forest · Breiman (2001) · Louppe (2014)
Gradient boosting: XGBoost, LightGBM, CatBoost — Add small trees one at a time, each one fitted to the gradient of the loss: the usual winner on tabular data. Builds on decision trees, gradient descent.
22. Gradient Boosting · Friedman (2001) · Chen & Guestrin (2016) · Ke et al. (2017)
Stacking — Train a meta-model on the out-of-fold predictions of several different models. Builds on cross-validation, gradient boosting.
beyond this course · Wolpert (1992) · scikit-learn: ensembles

Neural networks — from the perceptron to deep learning

Perceptron, MLP & backpropagation — Stacked logistic units with non-linear activations, trained by the chain rule: the hidden layers learn their own features. Builds on logistic regression, gradient descent.
23. Neural Networks · Rosenblatt (1958) · Rumelhart, Hinton & Williams (1986)
Convolutional networks — Small filters shared across the whole image, stacked from edges to textures to objects. Builds on MLP.
beyond this course · LeCun et al. (1998) · Krizhevsky, Sutskever & Hinton (2017) · Insper ANN-DL
Recurrent networks & LSTMs — A state carried along a sequence, step by step; gates let an LSTM remember across long spans. Builds on MLP.
beyond this course · Hochreiter & Schmidhuber (1997) · Insper ANN-DL
Transformers & attention — Every token looks at every other one and takes a weighted average, with weights from learned similarities (a softmax). Builds on MLP, dense embeddings, softmax.
beyond this course · Vaswani et al. (2017) · Alammar (2018) · Insper ANN-DL
Generative models: VAEs, GANs, diffusion — Learn the distribution of the data itself and sample new images, sounds or molecules from it. Builds on MLP, convolutional networks.
beyond this course · Goodfellow et al. (2014) · Ho, Jain & Abbeel (2020) · Insper ANN-DL

Evaluation & model selection — honest numbers

Classification metrics — Confusion matrix, precision, recall, F1, MCC: pick the metric from the cost of each kind of error.
15. Classification Metrics · van Rijsbergen (1979) · Chicco & Jurman (2020)
ROC, PR & imbalanced data — Curves over every threshold; class weights, threshold moving and resampling when the positives are rare. Builds on classification metrics.
16. ROC, PR & Imbalanced Data · Fawcett (2006) · Saito & Rehmsmeier (2015)
Cross-validation & data leakage — Rotate the validation fold for a steadier estimate, keep every fitted step and every future fact away from the rows you score on, and compare models on the same folds (nested CV grades the choice itself). Builds on pipelines, generalization.
17. Cross-Validation & Data Leakage · Stone (1974) · Kapoor & Narayanan (2023)
Model selection & bias–variance — The bias–variance decomposition, validation and learning curves, regression to the mean: tell underfitting from overfitting and choose the complexity from validation scores, not from hope. Builds on cross-validation, Ridge & Lasso.
19. Model Selection · Geman, Bienenstock & Doursat (1992) · Belkin et al. (2019)
Calibration & conformal prediction — Probabilities that mean what they say, and prediction sets that contain the true answer with a guaranteed rate. Builds on classification metrics, cross-validation.
beyond this course · Guo et al. (2017) · Angelopoulos & Bates (2021)

Explainability & responsibility — why the model says what it says

Interpretable by design — Linear models and shallow trees, whose reasoning can be read directly; for high-stakes decisions they are often enough. Builds on logistic regression, decision trees.
24. Explainability · Rudin (2019) · Molnar, Interpretable Machine Learning
Permutation importance — Shuffle one feature on held-out data and measure how much the score drops. Builds on random forest, cross-validation.
24. Explainability · Fisher, Rudin & Dominici (2019) · scikit-learn: permutation importance
SHAP & LIME — Explain one prediction: each feature's contribution, from Shapley values (SHAP) or from a local linear surrogate (LIME). Builds on gradient boosting, permutation importance.
24. Explainability · Ribeiro, Singh & Guestrin (2016) · Lundberg & Lee (2017)
Ethics & responsibility — Models learn the bias in historical decisions; privacy, consent and accountability are part of the job, not an afterthought.
2. The ML Landscape · Obermeyer et al. (2019) · Bender et al. (2021)
Algorithmic fairness — Formal criteria (demographic parity, equalized odds), why they cannot all hold at once, and how to audit a model group by group. Builds on ethics, classification metrics.
beyond this course · Hardt, Price & Srebro (2016) · Barocas, Hardt & Narayanan (2023)
Mechanistic interpretability — Open the network itself and find the features and circuits its neurons compute. Builds on transformers, SHAP.
beyond this course · Olah et al. (2020) · Templeton et al. (2024)

AutoML — searching for the model

Random search & Bayesian optimization — Sample hyperparameters at random, or let a surrogate model decide where to try next (Optuna). Builds on model selection.
25. AutoML · Bergstra & Bengio (2012) · Snoek, Larochelle & Adams (2012)
Successive halving & Hyperband — Give every candidate a small budget, keep the best fraction, give the survivors more. Builds on random search.
25. AutoML · Jamieson & Talwalkar (2016) · Li et al. (2018)
Full-pipeline AutoML — Search over the preprocessing, the model family and its hyperparameters together, often ending in an ensemble. Builds on pipelines, Bayesian optimization, stacking.
25. AutoML · Feurer et al. (2015) · Erickson et al. (2020)
Neural architecture search — Search the network's architecture itself: layers, widths, connections. Builds on Bayesian optimization, MLP.
beyond this course · Zoph & Le (2017) · Elsken, Metzen & Hutter (2019)

MLOps — models in production

Serving, versioning & reproducibility — Persist the fitted pipeline, pin the data and library versions, serve predictions behind an API. Builds on pipelines.
26. MLOps · Sculley et al. (2015) · Breck et al. (2017)
Monitoring & drift — Watch the inputs, the predictions and the outcomes in production, and retrain when the world moves. Builds on classification metrics, serving.
26. MLOps · Gama et al. (2014) · Rabanser, Günnemann & Lipton (2019)
Model cards & documentation — A short standard sheet: what the model is for, how it was evaluated, on whom it fails. Builds on ethics, serving.
26. MLOps · Mitchell et al. (2019) · Hugging Face: model cards

Other problem shapes — beyond one row, one label

Time-series forecasting — Predict the next values of a sequence from its lags and seasonality, with validation that only ever looks backward. Builds on cross-validation, gradient boosting.
beyond this course · Hyndman & Athanasopoulos (2021) · Makridakis, Spiliotis & Assimakopoulos (2022)
Recommender systems — Predict who wants what from sparse interactions: matrix factorization, then embeddings of users and items searched by nearest neighbors. Builds on PCA, dense embeddings, approximate nearest neighbors.
beyond this course · Koren, Bell & Volinsky (2009) · Covington, Adams & Sargin (2016)
Reinforcement learning — An agent learns by acting and collecting rewards: bandits, Q-learning, policy gradients. The paradigm was introduced in the ML landscape lesson. Builds on learning paradigms, MLP.
beyond this course · Sutton & Barto (2018) · Mnih et al. (2015)
Graph neural networks — Learn on networks of connected entities (molecules, social networks, road maps) by passing messages between neighbors. Builds on MLP, k-NN.
beyond this course · Kipf & Welling (2017) · Sanchez-Lengeling et al. (2021)

The frontier — foundation models and open questions

Transfer learning & fine-tuning — Start from a pre-trained model: use its embeddings as features, or keep training its weights (all of them, or a low-rank LoRA slice) on your labels. Discussed below. Builds on dense embeddings, MLP.
beyond this course · Howard & Ruder (2018) · Hu et al. (2022)
Foundation models & LLMs — One very large model pre-trained on self-supervised tasks, then adapted to many downstream tasks; scale in data, parameters and compute keeps buying new abilities. Discussed below. Builds on transformers, self-supervised learning, transfer learning.
beyond this course · Kaplan et al. (2020) · Brown et al. (2020) · Bommasani et al. (2021)
Prompting & in-context learning — Describe the task and show a few examples in the prompt; no weight changes. Builds on foundation models.
beyond this course · Brown et al. (2020) · Wei et al. (2022)
Retrieval-augmented generation (RAG) — Embed the question, fetch the nearest passages from a document store, put them in the prompt: k-NN over embeddings feeding an LLM. Builds on k-NN, approximate nearest neighbors, dense embeddings, foundation models.
beyond this course · Lewis et al. (2020) · Karpukhin et al. (2020)
Alignment: RLHF and successors — Tune a model toward human preferences, through a reward model fitted on compared answer pairs (RLHF) or directly on those pairs (DPO). Builds on reinforcement learning, logistic regression, foundation models.
beyond this course · Christiano et al. (2017) · Ouyang et al. (2022) · Rafailov et al. (2023)
Evaluating LLM systems — Benchmarks saturate and leak into training data; held-out tasks, LLM-as-a-judge and human preference arenas are the new validation sets. Builds on classification metrics, cross-validation & leakage.
beyond this course · Liang et al. (2022) · Zheng et al. (2023) · Sainz et al. (2023)
Multimodal models — One embedding space for images and text (CLIP), then models that read, see and listen at once. Builds on self-supervised learning, transformers, convolutional networks.
beyond this course · Radford et al. (2021) · Dosovitskiy et al. (2021)
Agents & tool use — LLMs that plan, call tools (search, code, APIs) and act in a loop; how to evaluate them and keep them safe is largely open. Builds on prompting, RAG, reinforcement learning.
beyond this course · Yao et al. (2023) · Schick et al. (2023)
Tabular foundation models — A transformer pre-trained on millions of synthetic tables predicts a new small table in one forward pass (TabPFN), challenging boosted trees on small data. Discussed below. Builds on gradient boosting, transformers, model selection.
beyond this course · Grinsztajn, Oyallon & Varoquaux (2022) · Hollmann et al. (2023) · Hollmann et al. (2025)
Causal machine learning — From "what predicts Y?" to "what happens to Y if we act?": treatment effects, uplift, double machine learning. Builds on linear regression, cross-validation, gradient boosting.
beyond this course · Chernozhukov et al. (2018) · Hernán & Robins, Causal Inference: What If
Efficient models: distillation & quantization — Smaller, cheaper models that keep most of the quality: a student trained on a teacher's soft labels, weights stored in 4 or 8 bits. Builds on foundation models, transfer learning.
beyond this course · Hinton, Vinyals & Dean (2015) · Dettmers et al. (2023)
AI regulation & governance — Risk-based law and risk-management frameworks: the EU AI Act, NIST's AI RMF, and in Brazil the LGPD (article 20: the right to ask for review of automated decisions) and the AI bill PL 2338/2023. Builds on ethics, fairness, model cards, interpretable models.
beyond this course · EU AI Act (2024) · NIST AI RMF 1.0 (2023) · PL 2338/2023 (Senado)

Transfer learning: stop starting from zero

Stage 2. Classical supervised learning trains each model from scratch on its own labeled data. Transfer learning reuses knowledge: pre-train a large model on a huge generic corpus, then adapt it to your task. There are two ways to adapt:

feature extraction: freeze the pre-trained model and use its representations as features for a classical head. You did this: sentence embeddings fed to a logistic regression or to a clustering step;
fine-tuning: continue training some or all of the weights on your labeled data, typically 10²–10⁴ examples instead of 10⁶. LoRA (Hu et al., 2022) trains only a low-rank correction \(W + BA\) of each weight matrix, with \(B\) and \(A\) thin; for a \(4096 \times 4096\) matrix and rank 8, that is \(2 \times 4096 \times 8 = 65{,}536\) trainable numbers instead of \(16{,}777{,}216\), 0.4% of them.

This inverted the data economics of the field. ULMFiT (Howard & Ruder, 2018) and BERT (Devlin et al., 2019) showed that a language model pre-trained on unlabeled text, then fine-tuned, beats models trained from scratch on text classification with a fraction of the labels. Tasks that once demanded massive labeled datasets became feasible with hundreds of examples.

Foundation models

Scale the recipe: transformer architectures, self-supervised pre-training (predict masked or next tokens: the labels are free, so any text becomes training data), billions of parameters. Something qualitatively new appears. A foundation model (Bommasani et al., 2021) is one very large pre-trained model adapted to many downstream tasks: the GPT family, Claude, Gemini and Llama for text; CLIP and its successors for vision and language; Whisper for speech.

Why scale? Kaplan et al. (2020) measured that the test loss of language models falls as a smooth power law in the number of parameters, the size of the data and the compute, over several orders of magnitude. That regularity made it rational to spend on ever larger training runs. Sutton's "Bitter Lesson" (2019) reads the same history: over 70 years, general methods that exploit more computation have beaten hand-built knowledge.

With large language models, adaptation gets lighter still:

Adaptation	Labeled data needed	What happens
zero-shot	none	describe the task in the prompt
few-shot / in-context	a handful	show examples in the prompt; no weights change
fine-tuning (full / LoRA)	hundreds+	update the weights, or a low-rank slice of them
RAG	none (needs documents)	retrieve relevant passages by nearest-neighbor search over embeddings and put them in the context

"Classify this support ticket as billing, technical or other": in 2018, a labeling project and a trained classifier; today, often a single prompt (Brown et al., 2020, showed GPT-3 doing new tasks from a few examples in the prompt). Prompting became the new fit() for a broad class of language tasks. What did not change: you still need a labeled test set to know whether the prompt works.

The classical toolbox in an LLM world

Stage 3. Look inside the modern stack and this course is everywhere:

the output layer of an LLM is a softmax regression over the vocabulary; training is mini-batch gradient descent with weight decay (an L2 penalty, as in Ridge) on cross-entropy;
RAG systems are k-NN over embeddings; a vector database is an approximate nearest-neighbor engine;
the reward model of RLHF is a logistic regression on pairs of answers;
evaluating an LLM system is precision/recall thinking plus honest held-out design; test questions that leak into training data are the field's current leakage problem;
BERTopic already composed foundation-model embeddings with UMAP, HDBSCAN and TF-IDF.

Where classical ML still wins

Reach for the classical stack, not a foundation model, when:

Situation	Why classical wins
Tabular data (churn, credit, pricing, demand)	gradient boosting still leads the benchmarks on medium-sized tables (Grinsztajn et al., 2022); LLMs are poor at tables of numbers
Latency / cost / scale (decisions in milliseconds, billions of rows)	a logistic regression scores a row in microseconds, for almost nothing
Regulated decisions	odds ratios and SHAP values can be shown to an auditor; a prompt cannot
Small, well-structured problems	a 2,000-row dataset needs bias–variance control, not a trillion parameters
Determinism and stability	fixed model + fixed input = fixed output; LLMs sample

The first row has a challenger. Grinsztajn, Oyallon & Varoquaux (2022) benchmarked 45 tabular datasets of around 10,000 rows and found tree ensembles ahead of deep networks, because tables have irregular target functions and many uninformative columns, which trees handle and neural networks smooth over. TabPFN (Hollmann et al., 2023, 2025) takes a different route: a transformer pre-trained on millions of synthetic tables, which predicts a new table in a single forward pass, with no training and no tuning. On classification datasets of up to 10,000 rows, the 2025 version beat, in 2.8 seconds, an ensemble of the strongest baselines (boosted trees among them) tuned for 4 hours. Above that size, and whenever latency matters, boosted trees remain the default. Test the challenger the way you tested every model in this course: same folds, same metric, paired comparison.

The frontier practitioner's skill is routing: perception and language go to foundation models, structured prediction goes to the classical stack, and real systems use both. An LLM reads the free-text complaint, a gradient-boosted model scores the churn risk, and MLOps monitors both.

Open problems — where this field is going

Stage 4. The map has dashed chips because the field is not finished. These problems are open:

Hallucination and reliability: fluent falsehoods. Calibration at frontier scale: a model should know when it does not know.
Evaluation: benchmarks saturate within a year or two and leak into training data; honest measurement is an arms race (Validation, scaled up).
Alignment and safety: making systems pursue the goals we intend; RLHF and its successors are partial answers.
Efficiency: distillation, quantization, small specialized models against giant generalists.
Data provenance, bias and governance: the ethics questions of the ML landscape lesson, now industrialized and regulated (EU AI Act; in Brazil, the LGPD and the AI bill PL 2338/2023, approved by the Senate in December 2024 and sent to the Chamber of Deputies).
Causality: prediction answers "what will happen"; decisions need "what happens if we act". Causal ML joins the two.
Agents: models that plan, call tools and act. How to evaluate them and keep them safe is largely unsolved.

Where to go from here

Deep learning, in depth: the Insper course Artificial Neural Networks and Deep Learning (CNNs, transformers, generative models); Prince, Understanding Deep Learning (2023, free online).
The probabilistic view: Murphy, Probabilistic Machine Learning (2022, free online), for the Bayesian models, Gaussian processes and uncertainty this course only touched.
LLMs, hands on: the Hugging Face courses (tokenizers, fine-tuning, RAG); Karpathy's Neural Networks: Zero to Hero, which builds a GPT from scratch.
Reading papers: pick a dashed chip on the map, read its classic paper, then find a recent paper that cites it. The map's links are the starting points.

The course thesis, one last time

Every "new" idea you will meet is a composition of things you now understand: losses, gradients, regularization, embeddings, neighbors, ensembles, honest validation. The frontier moves; the foundations compound. Learn the parts, and no whole will be a black box.

References

The map carries one to three references per technique. These are the ones the text of this lesson relies on.

Sutton, R. "The Bitter Lesson." Essay (2019). Link
Vaswani, A. et al. "Attention Is All You Need." NeurIPS (2017). arXiv:1706.03762
Howard, J.; Ruder, S. "Universal Language Model Fine-tuning for Text Classification." ACL (2018). arXiv:1801.06146
Devlin, J.; Chang, M.-W.; Lee, K.; Toutanova, K. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." NAACL-HLT (2019). arXiv:1810.04805
Kaplan, J. et al. "Scaling Laws for Neural Language Models." arXiv (2020). arXiv:2001.08361
Brown, T. B. et al. "Language Models are Few-Shot Learners." NeurIPS (2020) — GPT-3. arXiv:2005.14165
Lewis, P. et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS (2020). arXiv:2005.11401
Bommasani, R. et al. "On the Opportunities and Risks of Foundation Models." arXiv (2021). arXiv:2108.07258
Hu, E. J. et al. "LoRA: Low-Rank Adaptation of Large Language Models." ICLR (2022). arXiv:2106.09685
Ouyang, L. et al. "Training Language Models to Follow Instructions with Human Feedback." NeurIPS (2022) — RLHF. arXiv:2203.02155
Grinsztajn, L.; Oyallon, E.; Varoquaux, G. "Why Do Tree-Based Models Still Outperform Deep Learning on Typical Tabular Data?" NeurIPS Datasets and Benchmarks (2022). arXiv:2207.08815
Hollmann, N.; Müller, S.; Eggensperger, K.; Hutter, F. "TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second." ICLR (2023). arXiv:2207.01848
Hollmann, N. et al. "Accurate Predictions on Small Data with a Tabular Foundation Model." Nature 637 (2025): 319–326. DOI
Sainz, O. et al. "NLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for each Benchmark." Findings of EMNLP (2023). arXiv:2310.18018
Regulation (EU) 2024/1689 (Artificial Intelligence Act). Official Journal of the European Union (2024). EUR-Lex