8.1. Classification

A model does not have "an accuracy". It has a whole family of behaviours, one for every threshold you could put on its output, and each metric summarizes that family from a different angle. Choosing the wrong summary is not a rounding error — it routinely means shipping a model that is worthless for the job it was built for.

So this page is organized around a single question: what decision are you making, and what does a mistake cost you? Every metric below is an answer to some version of that question, and the point is to know which one is asking yours.

Why accuracy is a trap

Start with a screening test for a disease that affects 1 in 100 people. Here is a model:

def predict(patient):
    return "healthy"          # always

On 10 000 patients it gets 9 900 right. 99% accuracy — better than most published models, and it would pass any dashboard that reports a single number. It also misses every single sick patient, so as a screening test it has no value whatsoever.

This is not a contrived example, it is the normal situation. Fraud is rare. Equipment failure is rare. Rare is exactly when detection matters, and accuracy is at its most misleading precisely there: the majority class drowns the number you are reading.

The confusion matrix: where every metric comes from

All of it — every metric in this chapter — is four numbers arranged in a square. Predict "sick" for someone who is sick and you have a true positive; predict "sick" for someone healthy and you have a false positive¹.

The two errors are not interchangeable, and their names in your domain tell you which metric to reach for:

Precision and recall exist because those two columns need to be read separately:

The formulas look alike and the difference is the entire subject, so read them as questions instead:

• Precision — of everything I flagged, how much was real? It divides by what the model said. It is the metric of the person acting on the alerts.
• Recall — of everything real, how much did I catch? It divides by what reality contains. It is the metric of the person who will suffer the misses.

Everything moves when you move the threshold

A classifier does not output a class. It outputs a score, and someone chose a cut-off — usually 0.5, usually without thinking. That choice is not part of the model; it is part of the decision, and it is where precision and recall trade against each other.

Below: two score distributions, negatives in blue and positives in orange. Drag the threshold and watch the confusion matrix, every metric, and the position on both curves move together.

Three experiments, in order:

1. Leave prevalence at 50% and sweep the threshold. Precision and recall move in opposite directions — always. Lower the threshold and you catch more positives (recall ↑) while flagging more innocents (precision ↓). There is no setting where both are maximal; there is only the setting that suits your costs.
2. Set prevalence to 1% and leave the threshold at 0.50. Accuracy reads about 0.505 while "always predict negative" scores 0.990. The simulator says so in red. Every metric that mixes the two classes into one number does this.
3. Keep the model fixed and change only the prevalence. ROC-AUC does not move — 0.921 at 50%, at 10%, at 1%. The PR-AUC collapses: 0.899 → 0.643 → 0.248. The model did not get worse; the task did, and only one of the two curves noticed.

That third experiment is the whole argument about which curve to use, so it is worth stating why it happens. ROC plots recall against the false positive rate, \(FP/(FP+TN)\) — and its denominator is the negatives, which is exactly the class that grows when positives become rare. Ten false alarms out of a million negatives is an invisible FPR. Precision divides those same ten false alarms by the handful of true positives, where they are catastrophic. When positives are rare and false alarms are expensive, ROC will flatter your model and PR will tell you the truth²³.

F1, and when the harmonic mean is the wrong average

The harmonic mean is deliberately unforgiving: it sits close to the smaller of the two. With \(P = 1.0\) and \(R = 0.01\) the arithmetic mean is a respectable 0.505, while \(F_1 = 0.0198\). That is the point — F1 refuses to let a model hide a catastrophic recall behind excellent precision.

But F1 encodes a specific opinion — that precision and recall matter equally — and that opinion is usually wrong. \(F_\beta\) lets you state yours:

\(\beta\) is how many times more you care about recall than precision. \(F_2\) (recall twice as important) is standard in medical screening; \(F_{0.5}\) (precision twice as important) in recommendation and search. If you can name the ratio of costs, you should not be using \(F_1\).

Which metric, and why

Multi-class: micro, macro, weighted

Beyond two classes, "the F1" no longer exists — you have to say how the per-class scores get averaged, and the choice changes the answer completely. Take a 1 000-sample problem with classes A (900), B (90) and C (10):

Those are the numbers in the matrix below. Move the recall of any class and watch which of the three averages notices:

Read those three numbers again. Micro says 0.938, weighted says 0.937 — and the model gets class C right 3 times out of 10. Macro is the only one that reports the failure, and the gap between macro and weighted is a direct measure of how much worse you do on the rare classes. If you report one number for an imbalanced multi-class problem, report macro.

Probabilities, not just labels

Ranking and thresholding both ignore a question that matters in production: when the model says 0.8, does the event happen 80% of the time? A model can rank perfectly and still be systematically overconfident — and if a human or a downstream system consumes those numbers as probabilities, the error is silent and expensive.

The threshold is a business decision, and it has a formula

"Use 0.5" is a default inherited from argmax, not a decision. If you can say what a miss costs relative to a false alarm, the optimal cut-off follows exactly. Predicting positive on a case whose true probability is \(p\) has expected cost \(C_{FP}(1-p)\); predicting negative costs \(C_{FN}\,p\). Flagging is worth it when the first is smaller⁸, which happens when

Read it: if a miss costs 9× a false alarm, then \(p^\star = 1/10\) — flag anything above 10%, not 50%. The 0.5 default is only correct when the two errors cost the same, which is almost never why it was chosen.

Note what the simulator does not need: a better model. Same weights, same scores, same everything — only the cut-off moves, and the expected cost changes by tens of percent. Threshold selection is usually the cheapest improvement available to a deployed model, and it is the one most often left at its default.

In code

Every metric above has one call in scikit-learn⁹, and the order of the calls is the argument of this page: rank first, then choose a threshold from costs, then report the matrix.

from sklearn.metrics import (classification_report, confusion_matrix,
                             roc_auc_score, average_precision_score,
                             precision_recall_curve, brier_score_loss)

probs = model.predict_proba(X_val)[:, 1]

# threshold-free: the two summaries, and they answer different questions
print("ROC-AUC:", roc_auc_score(y_val, probs))          # ranking quality
print("PR-AUC :", average_precision_score(y_val, probs)) # ranking quality where positives are rare

# pick the threshold from costs, on validation data — never on test
prec, rec, thr = precision_recall_curve(y_val, probs)
C_FP, C_FN = 1.0, 10.0
p_star = C_FP / (C_FP + C_FN)                            # = 0.0909...

pred = (probs >= p_star).astype(int)
print(confusion_matrix(y_val, pred))
print(classification_report(y_val, pred, digits=3))      # per-class P/R/F1 + macro & weighted

# is 0.8 really 80%?
print("Brier:", brier_score_loss(y_val, probs))

Key takeaways

1. Accuracy answers "how often am I right", which is the wrong question whenever the classes are imbalanced or the two errors cost different amounts. Always compare it against the always-predict-the-majority baseline.
2. Every metric here is four numbers in a square. When in doubt, look at the confusion matrix; it is the only report that cannot hide anything.
3. Precision divides by what you claimed, recall by what is true. Which one matters follows from which error hurts.
4. ROC-AUC measures ranking and is invariant to prevalence — which is a virtue when comparing models and a trap when positives are rare. PR-AUC is the honest one there.
5. F1 assumes precision and recall matter equally. If you can name the ratio, use \(F_\beta\); if negatives matter too, use MCC.
6. In multi-class, the averaging is the metric. Macro is the one that notices rare classes.
7. Ranking is not calibration. If a human or a system consumes the probability, measure it — log loss, Brier, a calibration curve — and fix it with temperature scaling.
8. The threshold is not part of the model. It comes from \(p^\star = C_{FP}/(C_{FP}+C_{FN})\), and moving it is usually the cheapest win available.

Additional Resources

1. ROC and AUC, Clearly Explained! — Starmer, J., StatQuest. Builds the ROC curve threshold by threshold, which is exactly what the first simulator animates. Watch it if the curve still feels like a formula rather than a picture:

2. Metrics and scoring: quantifying the quality of predictions — scikit-learn User Guide. The reference for what each function actually computes, including the averaging rules that decide what f1_score means in the multi-class case.

3. Classification: Accuracy, recall, precision — Google Machine Learning Crash Course. A second pass over the same trade-offs with different worked examples, and a good source of practice questions.

References

The works cited through the text, in order of appearance: