Explainability

A good prediction nobody can read

• Random forests, gradient boosting and neural networks won the last lessons.
• A boosted ensemble of 200 trees predicts well, but "why this number?" has no answer you could write on a page.

Why explain a model?

The toolbox of this lesson

Model-specific methods read the internals; model-agnostic ones only call predict.

The path of this lesson

First: do you need a black box?

Three candidates on the same five folds, from fully readable to not readable at all.

Readability has a price, here

What the readable models say

Permutation importance: shuffle one column

1. Take held-out data and the model's score on it.
2. Shuffle one column: its values stay, its link with the target and with the rest of each row breaks.
3. Measure the drop in the score. Repeat for every column (Breiman, 2001).

for col in X_test.columns:
    X_perm = X_test.copy()
    X_perm[col] = rng.permutation(X_perm[col].to_numpy())
    drops[col] = base - r2_score(y_test, model.predict(X_perm))
  

Two rankings of the same model

Held-out data, not training data

A forest with two noise columns: a continuous one and a coin flip.

Correlated features (1): shared credit

Add an exact copy of income and refit the forest.

Correlated features (2): impossible rows

• Latitude and longitude: correlation −0.92, because California is a diagonal strip.
• Shuffle latitude alone: a San Francisco longitude gets a San Diego latitude, a point in the Pacific.

Partial dependence: in which direction, with what shape?

Set feature \(j\) to a value \(v\) in every row, keep the rest, average the predictions (Friedman, 2001):

by_hand = [model.predict(X_test.assign(income=v)).mean() for v in grid]
  

The PDP of income, by hand

partial_dependence(..., method='brute') returns the same eight numbers.

ICE: one curve per row

Income from $20,000 to $80,000: +$208,000 on average (the PDP), but from +$139,000 to +$269,000 district by district (5th to 95th percentile).

The extrapolation trap

SHAP: why this prediction, for me?

Split one prediction into contributions, one per feature, that add up exactly:

• Features are players; the prediction is the team's payout (Shapley, 1953; Lundberg & Lee, 2017).
• Value of a coalition \(S\): set the features in \(S\) to the row's values, take the others from a background dataset, average.
• \(\phi_j\): the average of \(v(S \cup \{j\}) - v(S)\) over the coalitions \(j\) can join, every order of arrival equally likely.

A toy model, small enough for a calculator

• Background: four districts, worth 120, 180, 160 and 100. So \(v(\varnothing) = 140\).
• We explain a rich, coastal, not big district: \(f(x) = 280\).

Follow the arrows where "rich" joins

\(v(\{\text{rich}\})\): set rich = 1 in the four districts: 200, 180, 280, 180, average 210.

Every number makes sense

80 + 65 − 5 = 140 = 280 − 140. The only attribution with these four properties:

efficiency (they add up) · symmetry · dummy (no effect → 0) · additivity (a sum of models → a sum of values)

shap.TreeExplainer on a tree that reproduces the formula returns [80, 65, −5] and base value 140.

Simulator: every coalition, every arrival order

Predict. Turn coastal off. Does the interaction still split in half? Does the waterfall still land on the prediction?

On the real model: a San Francisco district

• 8 features: \(2^8 = 256\) coalitions × 100 background districts = 25,600 predictions.
• Brute force and TreeExplainer agree to the sixth decimal (largest difference 2e-06).

Reading a waterfall

Global: local explanations, stacked

300 districts × 8 SHAP values. Mean |SHAP|, in $1,000: latitude 76.6, longitude 64.1, income 44.2, occupancy 22.5 ...

The beeswarm shows direction too: high incomes on the right, high occupancies and northern latitudes on the left.

Three rankings, three questions

LIME: a linear model around one row

In $1,000 per standard deviation (Ribeiro, Singh & Guestrin, 2016).

SHAP or LIME? They answer different questions

How local is "local"?

width	0.5	1.0	2.12	5.0	std, 5 seeds
income	54.7	54.8	42.1	38.2	1.5
house_age	−9.9	1.3	1.3	1.0	1.3
occupancy	−71.0	−67.4	−49.9	−45.0	0.9

Explanation ≠ causation

• "Around this district, one standard deviation more occupancy lowers the predicted price by about $50,000" is a true statement about the model.
• It does not mean that moving more people into a house makes it cheaper: crowded households are a marker of poorer neighborhoods.
• For auditing: if a credit model leans on the postcode, it may use it as a proxy for income or ethnicity.

Right for the wrong reasons

Hospital A: patients too sick to stand get the portable scanner. Hospital B: scanners assigned at random.

No validation on A could have warned you. The explanation did; only domain knowledge says the reliance will not survive.

A checklist for explaining a model

1. Try an interpretable model first; if it is as good on the same folds, use it.
2. Compute importances on held-out data.
3. Find correlated groups and permute or explain them together.
4. Look at ICE curves, not only the PDP; check that the plotted values are inside the data.
5. Use SHAP for local explanations; state the background you used.
6. When a feature dominates, ask when is this value recorded? (leakage).
7. Never read an explanation as a causal claim.

Explaining credit decisions

• German credit: 1,000 loans, 20 attributes, 30% went bad.
• Scorecard or forest? Logistic regression ROC AUC 0.771, random forest 0.785: a fold-by-fold difference of +0.014 ± 0.023, a tie.
• Open the forest: MDI against permutation, PDP and ICE, SHAP globally and for the riskiest applicant.
• Does the model need personal_status? Then plant a leak (collection letters, sent after a loan goes bad): CV ROC AUC 0.785 → 0.927, and the explanations find it.

Download the lab notebook and open it in Jupyter or Colab (File → Upload notebook).

Summary

1. Before explaining a black box, check whether a readable model is as good.
2. Permutation importance on held-out data; correlated features share credit and create impossible rows.
3. PDP for the average shape, ICE for the rows behind it; beware extrapolation.
4. SHAP: contributions that add up exactly to prediction − base value.
5. LIME: the local slope, sensitive to the kernel width.
6. An explanation describes the model, not the world.