Ir para o conteúdo

ROC-AUC & Imbalanced Data

Last lesson ended on a key insight: a classifier outputs scores, and the label depends on a threshold. This lesson evaluates models across all thresholds — the ROC and precision–recall curves — and then tackles the setting where all of this matters most: imbalanced datasets, where the interesting class is rare.

The ROC curve

Sweep the threshold from strict to permissive and plot, at each point:

\[ \text{TPR (recall)} = \frac{TP}{TP + FN} \qquad\text{vs.}\qquad \text{FPR} = \frac{FP}{FP + TN} \]
  • threshold ≈ 1: nothing flagged — bottom-left corner (0, 0);
  • threshold ≈ 0: everything flagged — top-right corner (1, 1);
  • in between: how much recall the model buys per unit of false alarm.

A random scorer moves along the diagonal (TPR = FPR); a perfect one hugs the top-left corner (100% recall with 0% false alarms).

AUC

The Area Under the ROC Curve compresses the curve into one threshold-free number with a beautiful probabilistic meaning:

\[ \text{AUC} = P\big(\text{score}(x^+) > \text{score}(x^-)\big) \]

— the probability that a random positive is ranked above a random negative. AUC 0.5 = coin flip, 1.0 = perfect ranking. It measures ranking quality: how well the model orders cases, independent of any threshold choice.

ROC curve and precision-recall curve on imbalanced data

from sklearn.metrics import roc_auc_score, RocCurveDisplay
roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])   # needs scores, not labels!

When ROC flatters: use precision–recall

FPR's denominator is the number of negatives. With 95% negatives, even a poor model keeps FPR small in absolute terms — the ROC curve looks great while the model's alarms are mostly false. The precision–recall curve (right panel) replaces FPR with precision, whose denominator is the model's own alarms, making it brutally honest on rare positives: its random-model baseline is the prevalence (here 0.05), not a diagonal.

Rule of thumb: balanced classes or costs on both classes → ROC-AUC; rare positive class and you care about the positives → PR curve and average precision (AP).

See the threshold trade-off live — slide it and watch every metric (and the ROC operating point) react at once:

Learning from imbalanced data

Metrics fixed, now the training side. With 1:1000 imbalance, most learners — which by default optimize accuracy-like objectives — converge to "predict the majority."

1. Get the evaluation right first

Stratified splits (Validation), PR-AUC / macro-F1 / recall at fixed precision — before touching the data. Many "imbalance problems" are actually metric problems.

2. Class weights: make minority errors expensive

Most scikit-learn classifiers accept class_weight, multiplying each class's contribution to the loss by \(w_c \propto n / (k \cdot n_c)\):

LogisticRegression(class_weight='balanced')
SVC(class_weight='balanced')
RandomForestClassifier(class_weight='balanced')

No data manipulation, no information loss — usually the first thing to try.

3. Resampling: change the data

  • Random undersampling — drop majority examples. Fast; discards information; viable when data is abundant;
  • Random oversampling — duplicate minority examples. Keeps all data; duplicates invite overfitting;
  • SMOTE (Chawla et al., 2002) — synthesize new minority points by interpolating between a minority example and its minority nearest neighbors: \(x_{\text{new}} = x_i + \lambda\,(x_{\text{nn}} - x_i)\), \(\lambda \sim U(0,1)\). Richer than duplication, but can manufacture points in noisy overlap regions — variants (Borderline-SMOTE, SMOTE-Tomek) mitigate.
# pip install imbalanced-learn
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import Pipeline as ImbPipeline

pipe = ImbPipeline([
    ('scaler', StandardScaler()),
    ('smote', SMOTE(random_state=0)),      # applied ONLY when fitting
    ('model', LogisticRegression()),
])

Resample inside the pipeline, after splitting

Oversampling before the train/test split copies (or interpolates) minority points into both sides — the model is tested on echoes of its own training data: leakage with inflated scores. imblearn's pipeline applies resampling only to training folds and never to validation/test data.

4. Move the threshold

Often the cheapest fix: keep the model, pick the threshold that meets the business constraint ("recall ≥ 90%", "precision ≥ 80%", or minimal expected cost) using the PR curve on validation data — never on test.

A sensible order of attack

flowchart LR
    A[Right metrics +<br>stratified CV] --> B[class_weight<br>= balanced] --> C[Threshold tuning<br>on validation] --> D[SMOTE / resampling<br>if still needed] --> E[Collect more<br>minority data]

Class materials

Class notebook (in Portuguese)

Hands-on notebook used in class — Aula 15 — Curva ROC-AUC e Datasets Desbalanceados: open in Colab

Quiz