Gradient Descent & Regularization

The closed-form OLS solution is elegant, but it does not scale to huge feature counts, does not exist for most other models, and says nothing about controlling overfitting. This lesson adds the two tools that do: gradient descent — the optimization engine behind nearly all of modern ML — and regularization — the standard brake on model complexity.

Gradient descent

To minimize a differentiable loss \(J(w)\), repeatedly step against the gradient (the direction of steepest increase):

where \(\eta\) is the learning rate. For linear regression with mean squared error,

a convex bowl with a single global minimum — gradient descent is guaranteed to reach it for small enough \(\eta\).

The learning rate

• \(\eta\) too small → tiny steps, painfully slow convergence;
• \(\eta\) too large → steps overshoot the minimum; the loss oscillates or diverges;
• practical recipe: try \(\eta \in \{10^{-3}, 10^{-2}, 10^{-1}\}\), monitor the training-loss curve — it should fall smoothly.

Feel it yourself — step through the descent, then set \(\eta = 1.05\) and watch it explode:

Batch, stochastic, and mini-batch

from sklearn.linear_model import SGDRegressor
model = SGDRegressor(loss='squared_error', penalty='l2', alpha=1e-4,
                     learning_rate='invscaling', max_iter=1000)

The same loop — with different losses — trains logistic regression, SVMs, and neural networks. Learn it once, reuse it everywhere.

From lines to curves: polynomial features

Linear regression is linear in the parameters, not necessarily in the inputs. Expanding features to powers, \(x \mapsto (x, x^2, \dots, x^p)\), fits polynomials with the same OLS machinery:

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures

model = make_pipeline(PolynomialFeatures(degree=3), LinearRegression())

But flexibility cuts both ways:

Degree 1 underfits — too rigid to follow the sine. Degree 15 overfits — 16 parameters chase 25 noisy points, producing wild oscillations. The right panel keeps all 16 parameters but adds a Ridge penalty: the curve relaxes back to the signal. That is regularization at work.

Regularization

Instead of restricting the number of parameters, penalize their magnitude — add a complexity term to the loss:

Ridge (L2) — Tikhonov, 1943; Hoerl & Kennard, 1970

• shrinks all coefficients smoothly toward zero (never exactly zero);
• spreads weight across correlated features — the standard cure for multicollinearity;
• closed form still exists: \(\hat{w} = (X^\top X + \alpha I)^{-1} X^\top y\) — the \(\alpha I\) makes the matrix invertible even with collinear features.

Lasso (L1) — Tibshirani, 1996

• the absolute-value penalty has corners at zero: solutions land exactly at zero for weak features;
• performs automatic feature selection — the surviving nonzero coefficients name the features that matter;
• among a group of highly correlated features, it tends to keep one arbitrarily and zero the rest.

Elastic Net blends both penalties (l1_ratio) — a robust default when features are many and correlated.

from sklearn.linear_model import Ridge, Lasso, ElasticNet

Ridge(alpha=1.0)
Lasso(alpha=0.1)
ElasticNet(alpha=0.1, l1_ratio=0.5)

The knob α

\(\alpha\) trades data fidelity against coefficient size:

• \(\alpha \to 0\): plain OLS (no brake);
• \(\alpha \to \infty\): all coefficients crushed to ~0, model predicts the mean (full brake);
• the right \(\alpha\) is not known in advance — it is chosen by cross-validation (RidgeCV, LassoCV, or a grid search).

Class materials

Quiz