Explainability

A good prediction nobody can read

• Random forests, gradient boosting and neural networks won the last lessons.
• A boosted ensemble of 200 trees predicts well, but "why this number?" has no answer you could write on a page.

Why explain a model?

The toolbox of this part

Model-specific methods read the internals; model-agnostic ones only call predict. LIME, a second local method, is in the advanced approach.

The path of this part

First: do you need a black box?

Three candidates on the same five folds, from fully readable to not readable at all.

Readability has a price, here

What the readable models say

Permutation importance: shuffle one column

1. Take held-out data and the model's score on it.
2. Shuffle one column: its values stay, its link with the target and with the rest of each row breaks.
3. Measure the drop in the score. Repeat for every column (Breiman, 2001).

for col in X_test.columns:
    X_perm = X_test.copy()
    X_perm[col] = rng.permutation(X_perm[col].to_numpy())
    drops[col] = base - r2_score(y_test, model.predict(X_perm))
  

Permutation importance, by hand

4,087 held-out districts, R² = 0.825 with nothing shuffled; each column shuffled 5 times. Predict. Can shuffling one column cost more than 0.825?

Two rankings of the same model

Held-out data, not training data

A forest with two noise columns: a continuous one and a coin flip.

Partial dependence: in which direction, with what shape?

Set feature \(j\) to a value \(v\) in every row, keep the rest, average the predictions (Friedman, 2001):

\(n\): number of rows (4,087 test districts); \(x_{i,k}\): feature \(k\) of row \(i\), unchanged except feature \(j\), replaced by \(v\); \(\hat f\): the model's prediction.

by_hand = [model.predict(X_test.assign(income=v)).mean() for v in grid]
  

The PDP of income, by hand

Grid: income 1.5 to 10 ($10,000). Each step gives every one of the 4,087 rows that income. Predict. Does the predicted value grow steadily with income, or does it saturate?

ICE: one curve per row

Income from $20,000 to $80,000: +$208,000 on average (the PDP), but from +$139,000 to +$269,000 district by district (5th to 95th percentile).

SHAP: why this prediction, for me?

Split one prediction into contributions, one per feature, that add up exactly:

• \(\phi_0\): the average prediction over a background dataset; \(\phi_j\): the share of feature \(j\) in the gap \(\hat f(x) - \phi_0\).
• Features are players; the prediction is the team's payout (Shapley, 1953; Lundberg & Lee, 2017).
• Value of a coalition \(S\), written \(v(S)\): set the features in \(S\) to the row's values, take the others from the background, average.
• \(\phi_j\): the average of \(v(S \cup \{j\}) - v(S)\) over the coalitions \(j\) can join, every order of arrival equally likely.

A toy model, small enough for a calculator

• Background: four districts, worth 120, 180, 160 and 100. So \(v(\varnothing) = 140\).
• We explain a rich, coastal, not big district: \(f(x) = 280\).

Follow the arrows where "rich" joins

\(v(\{\text{rich}\})\): set rich = 1 in the four districts: 200, 180, 280, 180, average 210.

Every number makes sense

80 + 65 − 5 = 140 = 280 − 140. The only attribution with these four properties:

efficiency (they add up) · symmetry · dummy (no effect → 0) · additivity (a sum of models → a sum of values)

shap.TreeExplainer on a tree that reproduces the formula returns [80, 65, −5] and base value 140.

Simulator: every coalition, every arrival order

Predict. Turn coastal off. Does the interaction still split in half? Does the waterfall still land on the prediction?

On the real model: a San Francisco district

• 8 features: \(2^8 = 256\) coalitions × 100 background districts = 25,600 predictions.
• Brute force and TreeExplainer agree to the sixth decimal (largest difference 2e-06).

Reading a waterfall

Global: local explanations, stacked

300 districts × 8 SHAP values. Mean |SHAP|, in $1,000: latitude 76.6, longitude 64.1, income 44.2, occupancy 22.5 ...

The beeswarm shows direction too: high incomes on the right, high occupancies and northern latitudes on the left.

Explanation ≠ causation

• "Low occupancy (1.98 people per household) adds $66,300 to the predicted price of the San Francisco district" (its SHAP value) is a true statement about the model.
• It does not mean that moving people out of a house makes it more expensive: crowded households are a marker of poorer neighborhoods.
• For auditing: if a credit model leans on the postcode, it may use it as a proxy for income or ethnicity.

Right for the wrong reasons

Hospital A: patients too sick to stand get the portable scanner. Hospital B: scanners assigned at random.

No validation on A could have warned you. The explanation did; only domain knowledge says the reliance will not survive.

Summary

1. Readable first: if an interpretable model is as good on the same folds, use it.
2. Permutation importance, on held-out data: how much the score needs each feature.
3. PDP for the average shape of an effect, ICE for the rows behind it; fanning curves mean interactions.
4. SHAP: contributions that add up exactly to prediction − base value; stacked, a global ranking with directions.
5. An explanation describes the model, not the world; when one feature dominates, ask when is this value recorded?

Explaining credit decisions, steps 1 to 7

• German credit: 1,000 loans, 20 attributes, 30% went bad.
• Scorecard or forest, on the same folds (ROC AUC, area under the receiver operating characteristic curve); then open the forest: MDI against permutation, PDP and ICE, SHAP globally and for the riskiest applicant.
• Download the lab notebook and open it in Jupyter or Colab (File → Upload notebook).
• At every step, answer the Predict before running.
• Simulations: permutation on the training rows (step 4) and a background of good loans only (step 7).

Pace: steps 1 to 3 in 10 min; step 4 and the first simulation in 10 min; steps 5 and 6 in 10 min; step 7 and the second simulation in 10 min.