Skip to content

Explainability

The models winning Parts V and VI β€” forests, boosters, networks β€” are black boxes: thousands of trees or millions of weights with no readable story. Yet the ethics discussion set a hard requirement: decisions affecting people must be explainable. Regulations (GDPR, Brazil's LGPD), model debugging, and leakage hunting all demand the same capability. Explainable ML (XAI) provides it.

Two complementary questions:

  • Global: how does the model behave overall β€” which features drive it?
  • Local: why did the model make this prediction for this case?

Interpretable by design

Before explaining a black box, ask if you need one. Linear/logistic regression (coefficients, odds ratios), small decision trees (readable rules), Naive Bayes (per-feature evidence) are transparent natively. When their accuracy suffices β€” often, on tabular problems β€” the simplest explanation is the model itself. When the accuracy gap justifies a black box, use post-hoc, model-agnostic methods:

Permutation importance (global)

Already met in Random Forest: shuffle one feature's column in held-out data and measure the score drop. Breaking the feature–target link destroys exactly the information the model extracted from that feature.

Permutation importance of the top 8 features

Repeats give a distribution (boxes), separating real signal from shuffle noise. One caveat: with strongly correlated features, shuffling one leaves its twin available β€” both look unimportant even when the pair is critical. Check the correlation matrix alongside.

SHAP (local + global)

SHAP (Lundberg & Lee, 2017) answers the local question with game-theoretic rigor. Treat features as players cooperating to produce the prediction; the Shapley value (Shapley, 1953) \(\phi_j\) is feature \(j\)'s fair share of the payout β€” its contribution averaged over all orders in which features could join:

\[ \hat{f}(x) = \underbrace{\mathbb{E}[\hat{f}]}_{\text{base value}} + \sum_{j=1}^{d} \phi_j(x) \]

The only attribution scheme satisfying fairness axioms (efficiency β€” contributions sum exactly to the prediction minus the average; symmetry; additivity). Exact computation is exponential, but TreeSHAP computes it efficiently for tree ensembles β€” a perfect match for XGBoost-family models.

# pip install shap
import shap

explainer = shap.TreeExplainer(model)          # for tree ensembles
shap_values = explainer(X_test)

shap.plots.waterfall(shap_values[0])   # LOCAL: this prediction, feature by feature
shap.plots.beeswarm(shap_values)       # GLOBAL: importance + direction of effects
shap.plots.scatter(shap_values[:, "age"])   # dependence: effect of age across data
  • Waterfall: from the base value, each feature pushes the prediction up (red) or down (blue) β€” the exact sentence a credit analyst needs: "denied mainly because: 3 overdue payments (+0.31), income below X (+0.12), tenure long (βˆ’0.05)";
  • Beeswarm: one dot per sample per feature β€” global importance with direction (high values of feature X push predictions up?).

LIME (local)

LIME (Ribeiro et al., 2016 β€” "Why Should I Trust You?") explains one prediction by fitting a simple model in the neighborhood: perturb the instance, get black-box predictions for the perturbed samples (weighted by proximity), and fit a small linear model locally. The local surrogate's coefficients are the explanation.

Intuitive and works for any model and data type (its image/text variants toggle superpixels/words). Weaknesses: explanations depend on the perturbation scheme and neighborhood width, and can be unstable β€” run twice, get different stories. SHAP has largely become the default for tabular work; LIME remains conceptually important and useful beyond tables.

Reading explanations responsibly

Explanation β‰  causation

SHAP/LIME describe what the model uses, not how the world works. "Zip code pushes the score down" is a fact about the model β€” and possibly evidence of proxy discrimination (zip code standing in for race/income), not a causal claim about zip codes. Use explanations to audit and debug; use causal inference to claim causes.

Explanations are also the everyday debugging instrument: an implausibly dominant feature in a SHAP beeswarm is the classic signature of data leakage; a nonsense dependence plot reveals bad encoding; drift in explanation patterns flags production trouble.

Quiz