Classification & Metrics

Classification is supervised learning with a categorical target: spam/ham, default/repay, disease/healthy, or one of many classes (digit 0–9). Parts IV and V build a portfolio of classifiers; this lesson builds the tool you need before any of them — knowing how to measure whether a classifier is any good. Choosing the wrong metric is not a detail: it silently optimizes the wrong behavior.

The confusion matrix

For a binary problem, call one class positive (usually the rare/interesting one: fraud, disease) and the other negative. Every prediction lands in one of four cells:

from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
confusion_matrix(y_test, y_pred)          # rows = truth, cols = prediction
ConfusionMatrixDisplay.from_estimator(model, X_test, y_test)

The two error types usually have very different costs: a missed cancer (FN) is not the same as an unnecessary follow-up exam (FP); a blocked legitimate transaction (FP) is not the same as an approved fraud (FN). Metrics exist to weigh them explicitly.

Accuracy — and why it lies

Accuracy answers "what fraction of predictions were right?" — reasonable when classes are balanced and errors cost the same. But with imbalanced classes it degenerates. Fraud is 0.5% of transactions? The dumb rule "everything is legitimate" scores 99.5% accuracy while catching zero fraud.

Precision and recall

Both focus on the positive class, answering different questions:

• Precision — of the cases I flagged, how many were real? High precision = few false alarms;
• Recall (sensitivity) — of the real cases, how many did I catch? High recall = few misses.

They pull in opposite directions: flag more aggressively and recall rises while precision falls; flag conservatively and the reverse. Which to prioritize is a domain decision:

F1: one number when you must have one

The harmonic mean of precision and recall:

The harmonic mean punishes imbalance: precision 1.0 with recall 0.02 gives \(F_1 \approx 0.04\), not the arithmetic 0.51 — you cannot buy a good F1 by maxing one side. The general \(F_\beta\) weighs recall \(\beta\) times as heavily as precision (\(F_2\) for screening, \(F_{0.5}\) for spam).

from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred))

Multi-class

The confusion matrix generalizes to \(k \times k\) — off-diagonal cells reveal which classes get confused (useful diagnostics: is the model mixing 4s and 9s?). Per-class precision/recall/F1 are combined by:

• macro average — mean over classes, all classes equal (small classes count fully);
• weighted average — mean weighted by class frequency;
• micro average — compute from pooled counts (equals accuracy for single-label problems).

On imbalanced multi-class data, report macro-F1: it exposes failure on rare classes that weighted averages hide.

Scores, thresholds, and calibration

Most classifiers output a score or probability, and the label comes from a threshold (default 0.5):

proba = model.predict_proba(X_test)[:, 1]
y_pred = (proba >= 0.5).astype(int)       # the threshold is a choice!

Moving the threshold trades precision against recall — lower it to catch more positives (recall ↑, precision ↓), raise it for cleaner alarms. The threshold is a business decision applied after training, and evaluating a model across all thresholds is exactly what ROC and precision–recall curves do — the subject of ROC-AUC & Imbalanced Data.

When the predicted probabilities themselves matter (risk pricing, triage ordering), check calibration: among cases predicted "70%", do about 70% turn out positive? (sklearn.calibration.CalibrationDisplay).

Class materials

Quiz