Introduction & History

Machine Learning (ML) is the field of study that gives computers the ability to learn from data without being explicitly programmed for every situation. Instead of writing rules by hand, we write programs that infer the rules from examples.

The classic formal definition is due to Tom Mitchell:

Mitchell (1997)

A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.

Every project in this course can be described in that vocabulary. A spam filter, for instance:

Task \(T\): classify e-mails as spam or not spam;
Experience \(E\): a corpus of e-mails already labeled by humans;
Performance \(P\): the fraction of e-mails classified correctly (or a more careful metric, as we will see in Classification & Metrics).

Why learn from data?

Hand-written rules break down when:

The rules are unknown. Nobody can write down the exact rules that distinguish a malignant tumor from a benign one in a raw image.
The rules change. Spammers adapt; a static filter decays. A learning system can be retrained.
The rules are too many. Recognizing handwritten digits with if/else statements is hopeless — there are too many ways to write a "7".
Personalization is needed. A recommender must behave differently for every user; learning from each user's history scales, hand-tuning does not.

flowchart LR
    subgraph Traditional programming
        A[Rules] --> C[Program]
        B[Data] --> C
        C --> D[Answers]
    end
    subgraph Machine learning
        E[Data] --> G[Learning algorithm]
        F[Answers / labels] --> G
        G --> H[Model ≈ rules]
    end

Machine learning inverts the traditional flow: instead of rules + data → answers, we feed data + answers to an algorithm and obtain a model — an approximation of the rules — which we then use to answer new, unseen cases.

Why now?

None of the core ideas are new — least squares is from 1805 — but three forces converged in the last two decades to make ML ubiquitous:

Data: the web, sensors, and digitization produced datasets large enough to learn subtle patterns;
Compute: GPUs and cloud computing made it cheap to fit large models;
Algorithms & software: open-source libraries (scikit-learn, XGBoost, PyTorch) turned decades of research into a few lines of code.

A brief history of machine learning

The history of ML is a 200-year conversation between statistics and computer science. The timeline below marks the milestones this course will visit — from the historical basis to the current edge.

Least Squares

1805

The historical basis of regression.
Adrien-Marie Legendre publishes the method of least squares for fitting orbits of comets; Carl Friedrich Gauss claims prior use and later gives it a probabilistic justification. Two centuries later it is still the first tool to try on a regression problem.

Legendre, A.-M. (1805). Nouvelles méthodes pour la détermination des orbites des comètes.

Regression to the Mean

1886

The word regression enters statistics.
Francis Galton, studying heredity, observes that children of unusually tall parents tend to be closer to average height — regression towards mediocrity. The phenomenon still traps analysts today, as we will see in Model Selection.

Galton, F. (1886). Regression towards mediocrity in hereditary stature.

Statistical Foundations

1920s–1936

Modern statistical inference takes shape.
Ronald Fisher develops maximum likelihood estimation, analysis of variance, and experimental design; in 1936 he introduces linear discriminant analysis with the iris dataset — still a canonical teaching dataset today.

Fisher, R. A. (1936). The use of multiple measurements in taxonomic problems.

Turing's Question

1950

Can machines think?
Alan Turing publishes Computing Machinery and Intelligence, proposing the imitation game (Turing Test) and, remarkably, suggesting that instead of programming an adult mind we should build a child machine that learns — anticipating machine learning itself.

Turing, A. M. (1950). Computing Machinery and Intelligence.

Birth of AI & the Perceptron

1956–1959

The field gets a name and a first learning machine.
The Dartmouth workshop (1956) coins artificial intelligence. Frank Rosenblatt (1958) builds the perceptron, a linear classifier that learns its weights from examples — the ancestor of today's neural networks. Arthur Samuel's checkers program (1959) popularizes the term machine learning.

Rosenblatt, F. (1958). The Perceptron: a probabilistic model for information storage and organization in the brain.

First AI Winter

1969–1980

Hype meets its limits.
Minsky and Papert prove that a single perceptron cannot solve non-linearly-separable problems such as XOR (1969). The Lighthill Report (1973) concludes AI failed to deliver; funding collapses across the US and UK.

Minsky, M., Papert, S. (1969). Perceptrons.

Backpropagation & Decision Trees

1984–1986

Learning revives on two fronts.
Breiman, Friedman, Olshen and Stone publish CART (1984); Quinlan releases ID3 (1986) — decision trees become practical. Rumelhart, Hinton and Williams popularize backpropagation (1986), making multi-layer neural networks trainable.

Rumelhart, D., Hinton, G., Williams, R. (1986). Learning representations by back-propagating errors.

Statistical Learning Era

1995–1997

Rigorous theory, powerful algorithms.
Cortes and Vapnik publish the support vector machine (1995), grounded in statistical learning theory. Freund and Schapire's AdaBoost (1997) shows weak learners can be combined into strong ones. ML becomes a discipline of margins, generalization bounds, and honest validation.

Cortes, C., Vapnik, V. (1995). Support-Vector Networks.

Ensembles & the Two Cultures

2001

The workhorses of tabular ML arrive.
Leo Breiman publishes Random Forests (2001) and, the same year, the essay Statistical Modeling: The Two Cultures, arguing for algorithmic prediction over pure data modeling. Friedman formalizes gradient boosting (2001).

Breiman, L. (2001). Random Forests. / Friedman, J. (2001). Greedy function approximation: a gradient boosting machine.

Deep Learning Breakthrough

2012–2016

Neural networks return, at scale.
AlexNet wins the ImageNet challenge by a huge margin using a deep convolutional network trained on GPUs (2012). Word2vec (2013) shows words can be embedded in vector spaces that capture meaning. XGBoost (2016) dominates tabular competitions while deep learning dominates perception.

Krizhevsky, A., Sutskever, I., Hinton, G. (2012). ImageNet classification with deep convolutional neural networks.

Transformers

2017–2018

Attention is all you need.
Vaswani et al. introduce the Transformer architecture (2017); BERT (2018) shows that pre-training on unlabeled text and fine-tuning transfers to nearly every language task. Sentence embeddings from these models will power BERTopic in Part II of this course.

Vaswani, A. et al. (2017). Attention Is All You Need.

Foundation Models — the Current Edge

2020–today

One model, many tasks.
GPT-3 (2020) demonstrates few-shot learning from a single giant pre-trained model; ChatGPT (2022) brings LLMs to the public; multimodal foundation models follow. Classical ML does not disappear — it powers the pipelines, evaluation, and tabular problems around these systems, and remains the right tool when data is small and structured.

Brown, T. et al. (2020). Language Models are Few-Shot Learners.

Reading the arc

Three lessons from this history shape the design of this course:

The classics never left. Least squares (1805) is still the first model you should try on a regression problem. Logistic regression (1958) remains a production workhorse. Understanding them deeply is not archaeology — it is engineering.
Hype cycles are real. The field went through two "AI winters" (roughly 1974–1980 and 1987–1993) when promises outran results. Honest evaluation — the subject of Part III — is the antidote.
Modern methods are compositions of classical ideas. BERTopic (2022), which we will study in Part II, is literally a pipeline of embeddings + dimensionality reduction + clustering + TF-IDF — every ingredient is a classical technique. Gradient boosting is functional gradient descent on decision trees. Knowing the parts lets you understand — and debug — the whole.

A learning machine made of matchboxes

Long before GPUs, Martin Gardner's 1962 Scientific American column described a machine that learns to play a game using only matchboxes and colored beads — no electronics at all. For the mini-game Hexapawn (three pawns per side on a 3×3 board), build one matchbox per possible board position, and fill it with beads for each legal move:

to move, open the matchbox for the current position and draw a random bead — that's the move;
if the machine eventually loses the game, remove the bead that led to the losing move (punishment);
if it wins, the beads stay (or extra copies are added — reward).

After a few dozen games the bad moves have literally been removed from the boxes: the machine plays perfectly. This is reinforcement learning in its purest mechanical form (Donald Michie built the same idea for tic-tac-toe in 1961 and called it MENACE), and it makes Mitchell's definition tangible: the task \(T\) is playing Hexapawn, the experience \(E\) is games played, the performance \(P\) is the win rate — and learning is nothing more than adjusting parameters (beads) based on feedback.

Read the original

Gardner, M. (1962). How to build a game-learning machine and then teach it to play and to win. Scientific American — PDF in the class folder. We play this game in the first class.

Where this course fits

This course covers classical machine learning end to end and finishes at the frontier: neural networks, explainability, AutoML, MLOps, and foundation models. Deep learning gets one bridging lecture here; its full treatment lives in the companion course Artificial Neural Networks and Deep Learning.

Class materials

Class slides (in Portuguese)

Aula 01 — Apresentação: open the slides
Hexapawn worksheets: training / test