How machine learning (ML) got here, and which questions the rest of the course will answer
| when | milestone |
|---|---|
| 1805 | least squares: the historical basis of regression |
| 1886 | regression to the mean: the word "regression" enters statistics |
| 1920s–1936 | statistical foundations: modern inference takes shape |
| 1950 | Turing's question: can machines think? |
| 1956–1959 | the birth of artificial intelligence (AI) and the perceptron |
| 1969–1980 | first AI winter: hype meets its limits |
| when | milestone |
|---|---|
| 1984–1986 | backpropagation and decision trees: learning revives on two fronts |
| 1995–1997 | statistical learning era: rigorous theory, powerful algorithms |
| 2001 | ensembles and the "two cultures": the workhorses of tabular ML |
| 2012–2016 | the deep learning breakthrough: neural networks return, at scale |
| 2017–2018 | transformers: attention is all you need |
| 2020–today | foundation models: one model, many tasks |
The full timeline, with sources, is on the lesson page.
Both learn from feedback on their mistakes: a loss removes a bead; an error nudges the perceptron's weights toward the right answer (Rosenblatt, 1958).
one box per position: a lookup table
what it learns in one box says nothing about any other; a position it never met has no box at all
a few weights shared by every input
each update changes its answer on examples it has never seen
| when | what happened |
|---|---|
| 1969–1973 | Minsky and Papert: one perceptron cannot learn XOR (exclusive or); the Lighthill Report: AI failed to deliver |
| 1974–1980 | first winter: funding collapses in the US and the UK |
| 1984–1986 | backpropagation trains multi-layer networks, which solve XOR; CART and ID3 make trees practical |
| 1987–1993 | second winter: the expert-systems boom, programs of hand-written rules, collapses |
| 1995–2001 | statistical learning: support vector machines, boosting, random forests; theory and honest validation |
| 2012– | deep learning: large labeled datasets, graphics processing units (GPUs) and open software |
least squares (1805) is still the first regression model to try; logistic regression (1958) is a production workhorse
two AI winters (about 1974–1980 and 1987–1993); honest evaluation is the antidote
BERTopic (2022) is embeddings + dimensionality reduction + clustering + TF-IDF (term frequency × inverse document frequency)
| question | where |
|---|---|
| kinds of learning; will it work on cases it never saw? | the ML landscape |
| what does the data look like, is it ready? | exploratory analysis, preprocessing, pipelines |
| structure with no labels at all? | dimensionality reduction, clustering |
| predict a number? | linear regression, gradient descent and regularization |
| predict a class? | logistic regression, k-nearest neighbors, SVM, trees, ensembles |
| how good is the model, really? | classification metrics, ROC and PR curves, cross-validation |
| which model, with which settings? | model selection |
| why did it decide that? how does it run in production? | explainability, MLOps |
SVM = support vector machine; ROC = receiver operating characteristic; PR = precision–recall.
Pace: step 4 and the simulation in 10 min; step 5 in 8 min; step 6 in 7 min.