The Frontier
The last stop of the arc that began with least squares in 1805: where the field stands now, how foundation models changed the economics of ML, and — just as important for a practitioner — where the classical toolbox you spent this course building still wins.
Transfer learning: stop starting from zero
Classical supervised learning trains each model from scratch on its own labeled data. Transfer learning reuses knowledge: pre-train a large model on a huge generic corpus, then adapt it to your task:
- feature extraction — freeze the pre-trained model, use its representations as features for a classical head (you did this: sentence embeddings + logistic regression or clustering);
- fine-tuning — continue training some or all weights on your labeled data (typically 10²–10⁴ examples instead of 10⁶).
This inverted the data economics of the field: tasks that once demanded massive labeled datasets became feasible with hundreds of examples.
Foundation models
Scale the recipe — transformer architectures, self-supervised pre-training (predict masked/next tokens: the labels are free, so the entire internet is training data), billions of parameters — and something qualitatively new appears. A foundation model (Bommasani et al., 2021) is one giant pre-trained model adapted to many downstream tasks: the GPT family, Claude, Gemini, Llama for text; CLIP and its successors for vision-language; Whisper for speech.
With large language models, adaptation gets lighter still:
| Adaptation | Labeled data needed | What happens |
|---|---|---|
| zero-shot | none | describe the task in the prompt |
| few-shot / in-context | a handful | show examples in the prompt; no weights change |
| fine-tuning (full / LoRA) | hundreds+ | update (a low-rank slice of) the weights |
| RAG | none (needs documents) | retrieve relevant passages — via embedding nearest-neighbor search! — and stuff them into the context |
"Classify this support ticket as billing/technical/other" — in 2018, a labeling project and a trained classifier; today, often a single prompt. Prompting became the new fit() for a broad class of language tasks.
The classical toolbox in an LLM world
Look inside the modern stack and this course is everywhere:
- LLM output layers are softmax regression; training is mini-batch gradient descent with weight decay on cross-entropy;
- RAG systems are k-NN over embeddings (vector databases = approximate nearest-neighbor engines);
- evaluation of LLM systems is precision/recall thinking plus honest held-out design — contaminated test sets are the field's current leakage scandal;
- and BERTopic showed foundation-model embeddings composed with UMAP + HDBSCAN + TF-IDF.
Where classical ML still wins
Reach for Parts I–V, not a foundation model, when:
| Situation | Why classical wins |
|---|---|
| Tabular data (churn, credit, pricing, demand) | gradient boosting still tops benchmarks; LLMs are poor at tables of numbers |
| Latency / cost / scale (ms decisions, billions of rows) | logistic regression serves in microseconds for ~zero cost |
| Regulated decisions | odds ratios and SHAP satisfy auditors; a prompt does not |
| Small, well-structured problems | a 2,000-row dataset needs bias control, not a trillion parameters |
| Determinism and stability | fixed model + fixed input = fixed output; LLMs sample |
The frontier practitioner's skill is routing: perception and language → foundation models; structured prediction → the classical stack; systems → both (an LLM parses the free-text complaint; XGBoost scores the churn risk; MLOps monitors both).
Open problems — where this field is going
- Hallucination and reliability — fluent falsehoods; calibration (Metrics) at frontier scale;
- Evaluation — benchmarks saturate and leak into training data; honest measurement is an arms race (Validation, scaled up);
- Alignment and safety — making systems pursue intended goals; RLHF and successors;
- Efficiency — distillation, quantization, small specialized models vs giant generalists;
- Data provenance, bias, and governance — the ethics questions, now industrialized;
- Agents — models that plan, call tools, and act; evaluation and safety largely unsolved.
The course thesis, one last time
Every "new" idea you will meet is a composition of things you now understand: losses, gradients, regularization, embeddings, neighbors, ensembles, honest validation. The frontier moves; the foundations compound. Learn the parts, and no whole will be a black box.
Continue at: Artificial Neural Networks and Deep Learning — architectures, transformers, and generative models in full depth.