What the gradient looks like with many features, why scaling decides whether the descent works, and why L1 zeroes where L2 shrinks
Model \(\hat y_i = w_0 + w_1 x_{i1} + \dots + w_d x_{id}\), residual \(e_i = y_i - \hat y_i\), loss \(J = \frac{1}{n}\sum_i e_i^2\). With the column of ones \(x_{i0} = 1\), the intercept is just one more weight, and \(\partial e_i / \partial w_j = -x_{ij}\) for every \(j\):
\[ \frac{\partial J}{\partial w_j} = -\frac{2}{n}\sum_{i=1}^{n} e_i\,x_{ij} \]| parameter | its column of \(X\) | partial derivative |
|---|---|---|
| \(w_0\) | the column of ones | \(-\frac{2}{n}\sum_i e_i \cdot 1\) |
| \(w_1\) | \(x_1\) | \(-\frac{2}{n}\sum_i e_i x_{i1}\) |
| \(w_j\) | \(x_j\) | \(-\frac{2}{n}\sum_i e_i x_{ij}\) |
The table of the previous slide, written compactly: one line of numpy for a million parameters.
set it to zero: \(X^\top(y - Xw) = 0\), the normal equations. Same statement, two routes: solve it at once, or walk there.
a residual-weighted sum: well-predicted points barely vote, badly missed points dominate the step.
Click on the left to drop a start and press Step. Then switch center x on and start again from a similar point.
a long narrow valley: tilting a line whose \(x\) values sit far from zero also raises or lowers it, so \(w_0\) and \(w_1\) are coupled. The gradient points across the valley; the path zigzags and drags.
a round bowl: the same steps go nearly straight to the answer, and a larger \(\eta\) is tolerated.
A degree-10 polynomial has 11 free parameters. Training error as root mean squared error (RMSE):
| training points | 5 | 8 | 10 | 11 | 15 | 30 |
|---|---|---|---|---|---|---|
| train RMSE | 3·10⁻¹⁵ | 1·10⁻¹¹ | 1·10⁻¹¹ | 0.40 | 0.77 | 1.53 |
Write the penalty as a budget: minimize the error subject to \(\sum |w_j| \le t\) (lasso, L1) or \(\sum w_j^2 \le t^2\) (ridge, L2). Every \(\alpha\) has a matching \(t\).
Drag the OLS optimum around and count how often each region is touched on an axis.
Eight features: three with real signal, three pure noise, two near-copies (\(c_1, c_2\), both truly 2). Predict. As \(\alpha\) grows, which coefficients does lasso drop first?
| α | lasso: non-zero | ridge: non-zero |
|---|---|---|
| 0.05 | 8 / 8 | 8 / 8 |
| 0.2 | 7 / 8 | 8 / 8 |
| 0.6 | 6 / 8 | 8 / 8 |
| 2.0 | 5 / 8 | 8 / 8 |
Lasso removes the noise features first; ridge keeps all eight, small.
The correlated pair at \(\alpha = 2\):
lasso (1.89, 0.98): starts to pick one, almost arbitrarily.
ridge (1.02, 0.97): splits the weight evenly.
condition number of \(X^\top X\) (largest eigenvalue over smallest): \(7.2\times10^{7}\)
OLS: \((1.000, -0.000)\), an arbitrary split
condition number: \(6\times10^{3}\)
answer: \((0.500, 0.500)\), the sum preserved
| decision | default | when to change it |
|---|---|---|
| closed form or gradient descent? | closed form for a few thousand features | many features, many rows or no closed form: stochastic gradient descent (SGDRegressor) in mini-batches |
| learning rate | try \(10^{-3}\), \(10^{-2}\), \(10^{-1}\) | the loss grows: smaller; barely moves: larger, after standardizing |
| scaling | standardize, inside a Pipeline | never skip it before gradient descent or a penalty |
| ridge, lasso or Elastic Net? | ridge: keeps every feature | lasso for a short list of features; Elastic Net for many correlated ones |
| \(\alpha\) | cross-validation (RidgeCV, LassoCV) | never the training error, which always prefers \(\alpha = 0\) |
x, y, gradient and train from steps 1 to 3.Pace: step 4 and the first simulation in 12 min; step 5 and the second in 15 min; step 6 in 8 min.