Introduction & History

Machine Learning (ML) is the field of study that gives computers the ability to learn from data without being explicitly programmed for every situation. Instead of writing rules by hand, we write programs that infer the rules from examples.

The classic formal definition is due to Tom Mitchell:

Every project in this course can be described in that vocabulary. A spam filter, for instance:

• Task \(T\): classify e-mails as spam or not spam;
• Experience \(E\): a corpus of e-mails already labeled by humans;
• Performance \(P\): the fraction of e-mails classified correctly (or a more careful metric, as we will see in Classification Metrics).

The lesson tells one story in four stages, drawn below: 1. why learn, when hand-written rules break down and how learning inverts the flow; 2. why now, the three forces behind today's machine learning and 200 years of history; 3. a machine that learns, matchboxes that learn a game from feedback alone; 4. the course ahead, what this course covers and where it stops. Each stage links to its section, and in the slides a strip at the bottom shows which stage you are in.

Slides

Click the slides and use the ← → arrows to navigate; F goes full screen and Esc shows every slide. Open the slides in a new tab

Why learn from data?

Stage 1. Hand-written rules break down when:

1. The rules are unknown. Nobody can write down the exact rules that distinguish a malignant tumor from a benign one in a raw image.
2. The rules change. Spammers adapt; a static filter decays. A learning system can be retrained.
3. The rules are too many. Recognizing handwritten digits with if/else statements is hopeless — there are too many ways to write a "7".
4. Personalization is needed. A recommender must behave differently for every user; learning from each user's history scales, hand-tuning does not.

Machine learning inverts the traditional flow: instead of rules + data → answers, we feed data + answers to an algorithm and obtain a model — an approximation of the rules — which we then use to answer new, unseen cases.

Why now?

Stage 2. None of the core ideas are new — least squares is from 1805 — but three forces converged in the last two decades to make ML ubiquitous:

• Data: the web, sensors, and digitization produced datasets large enough to learn subtle patterns;
• Compute: GPUs and cloud computing made it cheap to fit large models;
• Algorithms & software: open-source libraries (scikit-learn, XGBoost, PyTorch) turned decades of research into a few lines of code.

A brief history of machine learning

The history of ML is a 200-year conversation between statistics and computer science. The timeline below marks the milestones this course will visit — from the historical basis to the current edge.

[timeline left alternate(./docs/2026.2/classes/introduction/timeline.json)]

Reading the arc

Three lessons from this history shape the design of this course:

1. The classics never left. Least squares (1805) is still the first model you should try on a regression problem. Logistic regression (1958) remains a production workhorse. Understanding them deeply is not archaeology — it is engineering.
2. Hype cycles are real. The field went through two "AI winters" (roughly 1974–1980 and 1987–1993) when promises outran results. Honest evaluation — the subject of the cross-validation lesson — is the antidote.
3. Modern methods are compositions of classical ideas. BERTopic (2022), which we will study in its own lesson, is literally a pipeline of embeddings + dimensionality reduction + clustering + TF-IDF — every ingredient is a classical technique. Gradient boosting is functional gradient descent on decision trees. Knowing the parts lets you understand — and debug — the whole.

A learning machine made of matchboxes

Stage 3. Long before GPUs, Martin Gardner's 1962 Scientific American column described a machine that learns to play a game using only matchboxes and colored beads — no electronics at all. For the mini-game Hexapawn (three pawns per side on a 3×3 board), build one matchbox per possible board position, and fill it with beads for each legal move:

1. to move, open the matchbox for the current position and draw a random bead — that's the move;
2. if the machine eventually loses the game, remove the bead that led to the losing move (punishment);
3. if it wins, the beads stay (or extra copies are added — reward).

After a few dozen games the bad moves have literally been removed from the boxes: the machine plays perfectly. This is reinforcement learning in its purest mechanical form (Donald Michie built the same idea for tic-tac-toe in 1961 and called it MENACE), and it makes Mitchell's definition tangible: the task \(T\) is playing Hexapawn, the experience \(E\) is games played, the performance \(P\) is the win rate — and learning is nothing more than adjusting parameters (beads) based on feedback.

Play against the machine below (you are blue and move first), then press train N games and watch the beads disappear from the boxes as the machine's win rate climbs. Keep an eye on the line under the simulator: at some point the machine can no longer be beaten.

Nothing in the boxes "understands" Hexapawn: the machine got better only because feedback removed the moves that lost.

Where this course fits

Stage 4. This course covers classical machine learning end to end and finishes at the frontier: neural networks, explainability, AutoML, MLOps, and foundation models. Deep learning gets one bridging lecture here; its full treatment lives in the companion course Artificial Neural Networks and Deep Learning.

References

• Turing, A. M. "Computing Machinery and Intelligence." Mind 59 (1950). DOI
• Samuel, A. L. "Some Studies in Machine Learning Using the Game of Checkers." IBM J. Res. Dev. 3 (1959). DOI
• Bishop, C. M. Pattern Recognition and Machine Learning. Springer, 2006.

Quiz