8.2. Regression

Regression predicts a number, so the error is also a number — and that is the whole difficulty. In classification an error is a fact: right or wrong. Here every prediction is a little bit wrong, and a metric is a decision about how to add up all that wrongness. Different ways of adding it up reward completely different models, so the metric is not a report at the end; it is a specification at the start.

Everything below is built on the residual:

and differs only in what it does with the sign, the magnitude and the outliers.

The choice that decides everything: absolute or squared

They look like two flavours of the same idea. They are not, and the difference has a precise statement: they are minimized by different predictions.

• Minimizing MSE gives you the mean of the target distribution¹.
• Minimizing MAE gives you the median.

Which is why the choice is not a matter of taste. If you train a delivery-time model on MSE, it predicts the average delivery time, and averages are dragged upward by the rare disaster. Train it on MAE and it predicts the time that half the deliveries beat — usually what the customer was asking about. Same data, same architecture, different question answered.

Below: a set of delivery times with a long right tail. The curves are MSE and MAE as functions of a single constant prediction \(c\); drag \(c\) and watch where each one bottoms out.

RMSE: the units come back

MSE is in squared units — squared minutes, squared reais — which nobody can interpret. Taking the root fixes that:

RMSE keeps MSE's preference for the mean and its sensitivity to large errors, while reading in the same units as the target. That makes it the default for reporting, and it comes with a free diagnostic:

with equality only when every error has exactly the same magnitude. So the ratio RMSE/MAE is a measure of how uneven your errors are. Plain Gaussian noise gives \(\sqrt{\pi/2} \approx 1.25\); noticeably above that means a few predictions are much worse than the rest, and you should go and look at those cases individually instead of tuning anything.

What one outlier does

Drag the red point. Nothing else changes: same data, same model family, same code.

Two things happen at once when you drag it, and only one of them is the metric's fault.

The metrics move: RMSE grows faster than MAE, and MedAE — the median of the absolute errors — barely notices, because a median does not care how far away the far point is. That is robustness measured rather than asserted.

The model moves too. The orange line is fitted by least squares, so it chases the outlier: its slope falls from 0.961 to 0.423 as you drag the point down. The green line, fitted to minimize absolute error, stays at 0.989 the whole time. The metric you optimize is the model you get — the same lesson as the mean and the median, now visible in the fit itself.

R²: a comparison, not a percentage

Look at what is in the denominator: the error of predicting the mean of \(y\) for everyone. So \(R^2\) does not measure quality in the abstract — it measures how much better than the dumbest reasonable baseline you are. That framing settles most of the confusion around it:

• \(R^2 = 1\) — perfect predictions.
• \(R^2 = 0\) — you did exactly as well as always guessing the mean. Your model added nothing.
• \(R^2 < 0\) — possible, and common on test data. You did worse than the constant mean. If you have never seen a negative \(R^2\), you have probably only ever computed it on the training set.

Percentage errors, and how they bite

MAPE is popular because "we're off by 8%" travels well in a meeting. It also has three sharp edges, and the first one alone disqualifies it from many problems:

1. It is undefined at \(y_i = 0\) and enormous near it. Demand forecasting has zeros. Any count-like target has zeros.
2. It is asymmetric. Predicting 50 when the truth is 100 gives 50%. Predicting 150 when the truth is 100 gives 50% too, but predicting 200 gives 100% — while the smallest possible error for under-prediction is capped at 100%. Optimizing MAPE therefore biases a model to predict low.
3. It weights small targets enormously. Being off by 1 unit on a true value of 2 costs 50%; the same absolute miss on a true value of 1000 costs 0.1%.

Predicting a range instead of a number

Sometimes a single number is the wrong output. Quantile loss (also called pinball loss) trains a model to predict the \(\tau\)-th quantile instead of the mean:

Read it as an asymmetric MAE: with \(\tau = 0.9\), under-predicting is penalized nine times more than over-predicting, so the model learns a value it will exceed only 10% of the time. Train three models at \(\tau = 0.1, 0.5, 0.9\) and you have a prediction interval rather than a point — often far more useful to whoever has to act on it. Note that \(\tau = 0.5\) is MAE, up to a factor of two⁴.

Which metric, and why

In code

import numpy as np
from sklearn.metrics import (mean_absolute_error, root_mean_squared_error,
                             median_absolute_error, r2_score)

mae  = mean_absolute_error(y_test, pred)
rmse = root_mean_squared_error(y_test, pred)          # sklearn >= 1.4
medae = median_absolute_error(y_test, pred)
r2   = r2_score(y_test, pred)

# the baselines — report these next to the model, always
mean_rmse   = root_mean_squared_error(y_test, np.full_like(y_test, y_train.mean()))
median_mae  = mean_absolute_error(y_test, np.full_like(y_test, np.median(y_train)))

print(f"RMSE {rmse:.3f} vs {mean_rmse:.3f} for predicting the mean")
print(f"MAE  {mae:.3f} vs {median_mae:.3f} for predicting the median")
print(f"RMSE/MAE {rmse/mae:.2f}  (≈1.25 for gaussian errors; higher means heavy tails)")
print(f"R² {r2:.3f}   ({'worse than the mean!' if r2 < 0 else 'better than the mean'})")

Key takeaways

1. The metric is not a report, it is a specification: minimizing MSE asks for the conditional mean, minimizing MAE asks for the median. Decide which one your user wants.
2. RMSE is the sane default for reporting — same units as the target — and RMSE/MAE tells you for free how uneven the errors are.
3. MedAE is the robust reading; use it when the tail is genuine and you refuse to chase it.
4. R² compares you to predicting the mean. It can be negative, and it is not comparable across datasets.
5. MAPE is undefined at zero, asymmetric, and biases models to predict low. Prefer MASE or WAPE.
6. Always report a baseline — mean, median, or last observation. A model that does not beat it is not a model.
7. If a decision needs a range, train quantiles and hand over the interval.

Additional Resources

1. Evaluating forecast accuracy — Hyndman, R. J., & Athanasopoulos, G., Forecasting: Principles and Practice (3rd ed.). The clearest treatment of scale-dependent versus scale-free errors, written by the people who proposed MASE. Free online, and the surrounding chapters are the reference on baselines.

2. Regression metrics — scikit-learn User Guide. Exact definitions⁵, including the sign conventions of the neg_* scorers that trip everyone up the first time they use cross_val_score.

3. Root mean square error (RMSE) or mean absolute error (MAE)? — Chai, T., & Draxler, R. R. (2014)⁶. A short, readable argument in a genuine disagreement between researchers — worth reading against Willmott & Matsuura's opposite case¹ to see that the choice really is a modelling decision and not a settled fact.

References

The works cited through the text, in order of appearance: