Neural Networks

Part VI begins at the edge — and the edge is built from pieces you already own. A neural network is logistic regressions stacked and composed, trained by gradient descent, regularized with penalties you know from Ridge. This lesson is the bridge lecture; the full journey — CNNs, transformers, generative models — lives in the companion course ANN-DL.

From one neuron to a network

Rosenblatt's perceptron (1958) computes \(\hat{y} = \operatorname{step}(w^\top x + b)\) — a linear classifier. Its learning rule is beautifully simple: visit the data point by point, and on every mistake nudge the weights toward the misclassified point: \(w \leftarrow w + \eta\, y_i x_i\). Watch it converge:

Minsky & Papert (1969) proved a single such unit cannot solve XOR (no line separates it), triggering the first AI winter. The escape, made trainable by backpropagation (Rumelhart, Hinton & Williams, 1986): compose neurons in layers.

A multi-layer perceptron (MLP) with one hidden layer:

[ h = \sigma(W_1 x + b_1) \qquad\text{(hidden layer: learned features)} ] [ \hat{y} = \operatorname{softmax}(W_2\, h + b_2) \qquad\text{(a logistic/softmax layer on top)} ]

Read it in course vocabulary: the output layer is exactly multi-class logistic regression — but instead of running on hand-engineered features (the polynomials you built here), it runs on features \(h\) that the network learns for itself. That is the whole revolution: feature engineering becomes part of the optimization.

Activation functions: the essential nonlinearity

Without \(\sigma\), stacking layers collapses: \(W_2(W_1 x) = (W_2 W_1)x\) — still linear. The nonlinearity between layers is what buys expressive power. Modern default: ReLU, \(\max(0, z)\) — cheap and gradient-friendly. The universal approximation theorem (Cybenko, 1989; Hornik, 1991): one hidden layer with enough neurons can approximate any continuous function — existence guaranteed; learning it efficiently is what depth, data, and optimization tricks are for.

The single neuron draws its one line; sixteen hidden ReLU units learn a curved boundary — no polynomial features supplied, the hidden layer invented the representation.

Training: backpropagation

Training minimizes cross-entropy (or MSE) by mini-batch gradient descent. Backpropagation computes the gradients: it is the chain rule, applied layer by layer from the loss backwards, reusing intermediate results:

1. Forward pass — compute activations layer by layer, caching them;
2. Backward pass — propagate \(\partial L / \partial \text{activation}\) from output to input, obtaining every \(\partial L / \partial W_\ell\) in one sweep;
3. Update — step all weights: \(W_\ell \mathrel{-}= \eta\, \partial L / \partial W_\ell\).

New complication: the loss surface is non-convex — unlike logistic regression, no global-optimum guarantee. In practice, good local minima abound; momentum methods and Adam (adaptive learning rates, 2015) navigate reliably.

Regularization, translated: L2 penalty (called weight decay), early stopping (validation-loss version, as in boosting), and one genuinely new trick — dropout (randomly silence neurons during training), which trains an implicit ensemble of subnetworks.

from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

mlp = make_pipeline(StandardScaler(),               # gradient-trained ⇒ scale!
    MLPClassifier(hidden_layer_sizes=(64, 32), activation='relu',
                  alpha=1e-4,                        # L2 penalty
                  early_stopping=True, max_iter=500, random_state=0))
mlp.fit(X_train, y_train)

(scikit-learn's MLP is fine for tabular experiments; serious deep learning uses PyTorch/JAX — see ANN-DL.)

Why depth, and when

Deep networks stack many hidden layers, learning hierarchies of features (edges → textures → parts → objects, in vision). Depth pays off when raw inputs are perceptual — pixels, audio, text — where good features are unknown and data is plentiful. That is where deep learning crushed the field from 2012 on (AlexNet).

For tabular data, the honest current answer remains: gradient boosting usually wins, with less tuning and less data. Choose by data type, not by hype:

Quiz