When the coefficients can be trusted, what breaks the exact formula, and what replaces it
The path of this part
Trust the coefficientsthe classical assumptions; multicollinearity
Limits of the closed formtime, memory and collinear columns
The gradienta sum weighted by the residuals
The landscapevalleys, bowls and why centering helps
Each stage answers one question: what does a conclusion about a coefficient depend on? → when does the exact formula stop working? → what replaces it? → why does the same algorithm sometimes crawl?
Stage 1 · what does a conclusion about a coefficient depend on?
Assumptions behind the inferences
OLS (ordinary least squares) predictions ask for very little. Interpreting coefficients, confidence intervals and p-values leans on:
assumption
its picture in the residual plot
linearity
no U or ∩ curve
independence of the residuals
no waves over time
homoscedasticity: constant spread
no funnel
normality (intervals and p-values only, not the fit)
a symmetric band
no severe multicollinearity
invisible there: look at the features
Predict. Two features correlated at \(r = 0.99\): are the predictions still good? And each coefficient?
Multicollinearity: the data cannot tell the coefficients apart
contours of the squared error over \((w_1, w_2)\): a long, thin valley
Truth \((w_1, w_2) = (1, 1)\). Along the valley the error barely changes: \((2, 0)\) fits within about 2%, \((3, -1)\) within about 8%.
A new sample can move each estimate by a whole unit, even flip a sign, while the sum stays near 2.
\(X^\top X\) is nearly singular. The predictions are fine; one coefficient alone means nothing.
Predict. Doubling the number of features \(d\): by how much does the time to solve \(X^\top X w = X^\top y\) grow?
Stage 2 · when does the exact formula stop working?
When the closed form struggles
time
solving a \(d \times d\) system costs \(O(d^3)\): double \(d\), about 8× the time
perfectly collinear columns: \(X^\top X\) is singular, no unique answer
features \(d\)
1,000
10,000
50,000
100,000
\(X^\top X\) alone
0.01 GB
0.8 GB
20 GB
80 GB
A text vectorized with TF-IDF (term frequency–inverse document frequency) passes 100,000 columns easily, and a neural network has no normal equations at all. Libraries never form the inverse: they solve by QR or SVD (singular value decomposition), as scipy.linalg.lstsq does.
Stage 3 · what replaces it?
From the exact answer to the gradient
The closed form: write \(J\), take its gradient, solve \(\nabla J = 0\). Gradient descent keeps the first two steps and walks downhill instead of solving.
\(\eta\): the step size (learning rate). Set both derivatives to zero: \(\sum e_i = 0\) and \(\sum e_i x_i = 0\), the identities of the fundamental part.
The gradient is a sum weighted by the residuals: move each parameter in the direction the mistakes point, as much as they are big. Still true for a 100-layer network.
One step, on the five points
start at \(w_0 = w_1 = 0\): the line \(\hat y = 0\); every residual is \(y\) itself
With \(\eta = 0.05\): \(w_0 = 0.42\), \(w_1 = 1.42\).
Both derivatives are negative: the line is too low, so the step raises both weights. \(w_1\) moves more because each residual enters multiplied by its \(x\). After 2000 such steps, the descent reaches \(\hat y = 1.8 + 0.8x\).
Stage 4 · why does the same algorithm sometimes crawl?
Simulator: walking down the loss surface
Predict. Run it, then switch center x on and run it again from the same kind of start. What changes in the path?
Why scaling comes before fitting
uncentered \(x\)
A long narrow valley. Changing the slope of a line whose \(x\) values are all far from zero also swings its height, so \(w_0\) and \(w_1\) are coupled. The gradient points across the valley: the path zigzags and creeps.
centered \(x - \bar x\)
A round bowl. The gradient points at the bottom, and the same steps go almost straight to the answer.
Schematic: contour lines of \(J(w_0, w_1)\), the optimum (green) and the steps of gradient descent (orange).
The closed form gives the same line either way. The iterative route cares enormously: standardizing reshapes the surface the optimizer walks.
Summary
Predictions ask little; interpreting coefficients needs linearity, independence, constant spread, normality and no severe multicollinearity.
With collinear features the sum of the coefficients is pinned, each one is not.
The closed form fails on time (\(O(d^3)\)), memory (\(d^2\) numbers) and collinearity, and does not exist for most models.
The gradient is a sum weighted by the residuals; descent walks to the same answer.
Centering turns a valley into a bowl: same line, far fewer steps.
Next lesson: Gradient Descent & Regularization, the step size, batches and penalties.
Advanced lab · about 40 min, in pairs
Where the closed form breaks, and a year of the Ibovespa: steps 5 to 10
Same notebook, same session: the steps use x, y, X and w of steps 1 to 4.
Measure the cost of solving; fit a year of daily closes of the Ibovespa, the São Paulo stock index; check independence, forecast the year end, then test the forecast on the past.