Machine Learning · Insper
Neural Networks
Advanced approach
How far training can be trusted, and when a network is worth it
The path of this part
- Check the gradientfinite differences; one backward pass
- Train a non-convex modelstarting points, solvers, regularization
- Depth and datauniversal approximation; networks against boosting
Each stage answers one question: is backpropagation really the derivative, and why compute it that way? → what changes when the loss is not convex? → why go deep, and when is a network the right tool?
Stage 1 · is backpropagation really the derivative?
Check it the slow way: finite differences
The 2-2-1 network of the fundamental part: \(x = (0.5, 0.8)\), target \(y = 0\), squared loss \(L\). Put its 9 parameters in one vector \(p\); let \(e_i\) be the vector with 1 in position \(i\) and 0 elsewhere. Nudge one parameter at a time by \(\varepsilon = 10^{-6}\):
\[ \frac{\partial L}{\partial p_i} \approx \frac{L(p + \varepsilon e_i) - L(p - \varepsilon e_i)}{2\varepsilon} \]
backprop vs finite differences
1.3 × 10−10
largest difference over the 9 gradients: rounding error
one step, \(\eta = 0.5\)
0.5249 → 0.4405
the loss on this example, before and after updating all 9 parameters
Backpropagation is not an approximation: it is the exact derivative, computed efficiently.
Why backpropagation: the cost
finite differences
2 extra forward passes per parameter: 18 for the 9 parameters; 4,820 for the 2,410 parameters of a 64-32-10 digit network
backpropagation
one backward pass for all the gradients, about the cost of the forward pass
For modern networks, with billions of parameters, that ratio is what makes training possible at all.
With the cross-entropy loss instead of the squared loss, steps 1 and 2 collapse into \(\delta_y = \hat y - y\) (here 0.725): the same "prediction minus target" as in logistic regression. Steps 3 to 6 do not change.
Stage 2 · what changes when the loss is not convex?
Non-convex: the starting point matters
Predict. Logistic regression reaches the same minimum from any start. The same network with 2 hidden units, from 10 random starts: will they all end at the same accuracy? And with 8 units?
The noisy XOR, 10 random starts (random_state 0–9), training accuracy:
| hidden units | seeds 0–9 |
|---|
| 2 | 0.76 0.66 0.74 0.94 0.94 0.94 0.94 0.94 0.74 0.66 |
| 8 | 0.98 0.98 0.97 0.98 0.98 0.96 0.99 0.99 0.97 0.95 |
With 2 units, half the starts get stuck in a worse local minimum (0.66 to 0.76 against 0.94). Extra units give gradient descent more routes, one reason networks are wider than the minimum. Check conclusions over a few seeds.
Optimizers and regularization
adam
the default: mini-batches of 200 rows with momentum and a step size per parameter
sgd
plain mini-batch gradient descent, optional momentum
lbfgs
the whole training set at every step and an approximation of the curvature; best on small datasets
alpha: weight decay, the L2 penalty of Ridge on every weight of every layer.early_stopping=True: hold out 10% of the training rows, stop after 10 epochs without improvement on them.- dropout: switch off random units at each step; in deep-learning libraries, not in scikit-learn.
Stage 3 · why go deep, and when is a network the right tool?
Depth
- One hidden layer is already a universal approximator: with enough units it approximates any continuous function on a bounded region as closely as desired (Cybenko, 1989; Hornik, 1991).
- The theorem guarantees existence: not that training finds the weights, nor that the number of units is reasonable.
- Deep networks reach the same accuracy with far fewer units, because each layer builds on the previous one: in images, edges → textures and parts → objects.
That hierarchy is why deep learning took over perception (images, audio, text) from 2012 on, when data and GPUs (graphics processing units) became plentiful.
Tabular data: an honest comparison
Spambase, the same 5 folds, four models. Predict. Which one wins, and by how much?
| logistic | MLP | MLP early stop | boosting | boosting − MLP |
|---|
| fold 0 | 0.921 | 0.951 | 0.946 | 0.954 | 0.003 |
| fold 1 | 0.928 | 0.953 | 0.950 | 0.955 | 0.002 |
| fold 2 | 0.926 | 0.946 | 0.937 | 0.966 | 0.021 |
| fold 3 | 0.934 | 0.949 | 0.942 | 0.965 | 0.016 |
| fold 4 | 0.913 | 0.941 | 0.935 | 0.939 | −0.002 |
| mean | 0.924 | 0.948 | 0.942 | 0.956 | |
Boosting wins 4 folds of 5, by little: an edge, not a knockout. But it got there with its defaults and no preprocessing; the network needed a scaler, an architecture, alpha and a solver.
When is a network the first choice?
| data | first choice |
|---|
| tables of heterogeneous columns | gradient boosting or a random forest |
| images, audio, video | convolutional networks, vision transformers |
| text | transformers, or embeddings + a linear model |
| few rows | linear models, k-NN (k nearest neighbors), SVM (support vector machine) |
Large benchmarks agree (Shwartz-Ziv & Armon, 2022; Grinsztajn et al., 2022): trees split one feature at a time, so they ignore scale and useless columns and follow step-like relations; a network must learn all that from smooth functions.
Where to go next: the Deep Learning course
- This lesson stops at the multi-layer perceptron, in scikit-learn.
- Artificial Neural Networks and Deep Learning continues from exactly here: backpropagation in matrix form, then training deep networks (initialization, normalization, dropout).
- Then convolutional and recurrent networks, transformers and generative models, in PyTorch.
The lab ends on the reason convolutional networks exist: an MLP does not know which pixels are neighbors.
Advanced lab · about 30 min, in pairs
Early stopping, a fair comparison and the test, steps 6 to 8
- Same notebook, same session: the steps use
X_train, y_train, cv, cv_summary, models and grid.
- Read the loss curve and try early stopping; compare the MLP with logistic regression and the forest fold by fold; touch the test set once.
- Simulations: another start (step 6) and move the digits one pixel (step 8).
Pace: step 6 and the first simulation in 12 min; step 7 in 8 min; step 8 and the second simulation in 10 min.