Skip to content

Linear Regression

Linear regression is the oldest algorithm in this course — Legendre published least squares in 1805 to fit comet orbits, Gauss claimed it earlier, and Galton's studies of heredity (1886) gave "regression" its name. Two centuries later, it remains the first model to try on any regression problem: fast, interpretable, and a surprisingly strong baseline.

The model

Predict a continuous target as a weighted sum of features:

\[ \hat{y} = w_0 + w_1 x_1 + w_2 x_2 + \dots + w_d x_d = w_0 + \sum_{j=1}^{d} w_j x_j \]
  • \(w_j\): the change in \(\hat{y}\) per unit change in \(x_j\), holding the other features fixed;
  • \(w_0\) (intercept/bias): the prediction when all features are zero.

In matrix form, with a leading column of 1s absorbed into \(X\): \(\hat{y} = Xw\).

Least squares

Choose \(w\) to minimize the sum of squared residuals — the squared vertical distances between data and fitted line:

\[ J(w) = \sum_{i=1}^{n} \big(y_i - \hat{y}_i\big)^2 = \lVert y - Xw \rVert^2 \]

Why squares? Squaring penalizes large errors heavily, yields a smooth (differentiable) objective, and — under Gaussian noise — coincides with maximum likelihood estimation.

The closed-form solution

Setting the gradient to zero, \(\nabla_w J = -2X^\top(y - Xw) = 0\), gives the normal equations:

\[ X^\top X \, w = X^\top y \qquad\Longrightarrow\qquad \hat{w} = (X^\top X)^{-1} X^\top y \]

For simple (one-feature) regression this reduces to the formulas worth memorizing:

\[ \hat{w}_1 = \frac{\sum_i (x_i - \bar{x})(y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2} = r_{xy}\frac{s_y}{s_x}, \qquad \hat{w}_0 = \bar{y} - \hat{w}_1 \bar{x} \]

— the slope is the correlation rescaled by the standard deviations, and the line always passes through \((\bar{x}, \bar{y})\).

OLS fit with residuals and residual plot

The residual plot (right) is the standard diagnostic: residuals should look like structureless noise around zero. Curvature suggests a missing nonlinear term; a funnel shape suggests non-constant variance; isolated extreme residuals point to outliers (recall Anscombe's quartet).

from sklearn.linear_model import LinearRegression

model = LinearRegression().fit(X_train, y_train)
model.coef_, model.intercept_
y_pred = model.predict(X_test)

Get a feel for the fit — drag points and watch the line chase the minimum SSE. Then drag one point far off and see how much a single outlier can pull the line (recall Anscombe's dataset 3):

When the closed form struggles

Inverting \(X^\top X\) costs \(O(d^3)\) and fails when features are perfectly collinear. For huge or ill-conditioned problems we switch to gradient descent and regularization — the next lesson.

Assumptions behind the inferences

OLS predictions need little; but interpreting coefficients and error bars leans on the classical assumptions:

  1. Linearity — the true relationship is (approximately) linear in the features;
  2. Independence — residuals are not correlated with each other (beware time series);
  3. Homoscedasticity — residual variance is constant across the range of \(\hat{y}\);
  4. Normality of residuals — needed for exact confidence intervals and p-values;
  5. No severe multicollinearity — highly correlated features make individual coefficients unstable (their sum may be well determined while each one swings wildly).

Measuring regression quality

With residuals \(e_i = y_i - \hat{y}_i\):

Metric Formula Reading
MAE \(\frac{1}{n}\sum \lvert e_i \rvert\) mean error in target units; robust to outliers
MSE \(\frac{1}{n}\sum e_i^2\) punishes large errors; the training objective
RMSE \(\sqrt{\text{MSE}}\) like MSE but back in target units
R² \(1 - \frac{\sum e_i^2}{\sum (y_i - \bar{y})^2}\) fraction of variance explained; 1 = perfect, 0 = no better than predicting \(\bar{y}\)

R² can be negative on test data — the model is then worse than the constant baseline \(\bar{y}\). Always compare against that baseline (DummyRegressor): it is embarrassing how often a fancy model barely beats it.

Evaluate on held-out data

All metrics above are only meaningful on data the model did not see — the subject of Validation & Data Leakage.

Class materials

Class notebook (in Portuguese)

Hands-on notebook used in class — Aula 09 — Regressão: open in Colab

Quiz