Machine Learning · Insper
Introduction & History
Programs that infer the rules from examples
What learning means, why now, and 200 years of history in one course
Two ways to get a computer to decide
write the rules
a person thinks of every case and writes it as code
learn the rules
we write a program that infers the rules from examples
Machine Learning (ML): the field that gives computers the ability to learn from data without being explicitly programmed for every situation.
What does "learn" mean? Mitchell (1997)
A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.
Every project in this course can be described in this vocabulary.
Example: a spam filter
Predict. What are the task, the experience and the performance measure of a spam filter?
task \(T\)
classify e-mails as spam or not spam
experience \(E\)
a corpus of e-mails already labeled by humans
performance \(P\)
the fraction classified correctly (or a more careful metric, in the lesson on classification metrics)
The path of this lesson
- Why learnwhen rules break down; the inverted flow
- Why nowdata, compute, software; 200 years of history
- A machine that learnsmatchboxes and beads learn Hexapawn
- The course aheadclassical ML end to end, up to the frontier
Each stage answers one question: why not just write the rules? → if the ideas are old, why only now? → what does learning look like with no computer at all? → what does this course cover?
Stage 1 · why not just write the rules?
Why learn from data? When rules break down
1. the rules are unknown
nobody can write the exact rules that tell a malignant tumor from a benign one in a raw image
2. the rules change
spammers adapt; a static filter decays, a learning system can be retrained
3. the rules are too many
handwritten digits with if/else: there are too many ways to write a "7"
4. personalization is needed
a recommender must behave differently for every user; learning from each history scales
Machine learning inverts the flow
What the inversion gives us
traditional
rules + data → answers
machine learning
data + answers → model
The model is an approximation of the rules, and we use it to answer new, unseen cases.
Stage 2 · if the ideas are old, why only now?
Why now?
Predict. Least squares is from 1805. If the core ideas are that old, why did ML become everywhere only in the last two decades?
data
the web, sensors and digitization produced datasets large enough to learn subtle patterns
compute
graphics processing units (GPUs) and cloud computing made it cheap to fit large models
algorithms & software
open-source libraries (scikit-learn, XGBoost, PyTorch): decades of research in a few lines
A 200-year conversation: statistics and computer science
| when | milestone |
|---|
| 1805 | least squares: the historical basis of regression |
| 1886 | regression to the mean: the word "regression" enters statistics |
| 1920s–1936 | statistical foundations: modern inference takes shape |
| 1950 | Turing's question: can machines think? |
| 1956–1959 | the birth of artificial intelligence (AI) and the perceptron |
| 1969–1980 | first AI winter: hype meets its limits |
… up to the current edge
| when | milestone |
|---|
| 1984–1986 | backpropagation and decision trees: learning revives on two fronts |
| 1995–1997 | statistical learning era: rigorous theory, powerful algorithms |
| 2001 | ensembles and the "two cultures": the workhorses of tabular ML |
| 2012–2016 | the deep learning breakthrough: neural networks return, at scale |
| 2017–2018 | transformers: attention is all you need |
| 2020–today | foundation models: one model, many tasks |
The full timeline, with sources, is on the lesson page.
Reading the arc: three lessons for this course
the classics never left
least squares (1805) is still the first regression model to try; logistic regression (1958) is a production workhorse
hype cycles are real
two AI winters (about 1974–1980 and 1987–1993); honest evaluation is the antidote
modern = classical, composed
BERTopic (2022) is embeddings + dimensionality reduction + clustering + TF-IDF (term frequency × inverse document frequency)
Knowing the parts lets you understand, and debug, the whole.
Stage 3 · what does learning look like with no computer at all?
A learning machine made of matchboxes
- Martin Gardner, Scientific American, 1962. No electronics at all.
- The mini-game Hexapawn: three pawns per side on a 3×3 board.
- One matchbox per possible board position, filled with beads, one color per legal move.
Predict. With only boxes and beads, how could this machine get better at the game after each match?
The rules of the matchboxes
- To move: open the box for the current position and draw a random bead. That is the move.
- If the machine loses: remove the bead that led to the losing move (punishment).
- If it wins: the beads stay, or extra copies are added (reward).
After a few dozen games the bad moves have literally been removed from the boxes: the machine plays perfectly.
Simulator: the matchbox machine
Predict. Train it for 50 games against a random opponent. Will the win rate climb smoothly, and will it ever stop losing?
What the matchboxes teach
- Reinforcement learning in its purest mechanical form. Donald Michie built the same idea for tic-tac-toe in 1961: MENACE.
Learning is nothing more than adjusting parameters (beads) based on feedback.
Stage 4 · what does this course cover?
Where this course fits
- Classical machine learning end to end: data, unsupervised learning, regression, classification, trees and ensembles.
- It finishes at the frontier: neural networks, explainability, AutoML, MLOps and foundation models.
- Deep learning gets one bridging lecture here; its full treatment is in the companion course Artificial Neural Networks and Deep Learning.
Summary
- ML: programs that infer the rules from examples.
- Mitchell: learning = performance \(P\) on task \(T\) improves with experience \(E\).
- It inverts the flow: data + answers → model.
- Why now: data, compute, open-source software.
- Matchboxes: learning is adjusting parameters from feedback.
Next: The ML Landscape, a map of the kinds of learning and of a real project.