Explainability

The models winning Parts V and VI — forests, boosters, networks — are black boxes: thousands of trees or millions of weights with no readable story. Yet the ethics discussion set a hard requirement: decisions affecting people must be explainable. Regulations (GDPR, Brazil's LGPD), model debugging, and leakage hunting all demand the same capability. Explainable ML (XAI) provides it.

Two complementary questions:

• Global: how does the model behave overall — which features drive it?
• Local: why did the model make this prediction for this case?

Interpretable by design

Before explaining a black box, ask if you need one. Linear/logistic regression (coefficients, odds ratios), small decision trees (readable rules), Naive Bayes (per-feature evidence) are transparent natively. When their accuracy suffices — often, on tabular problems — the simplest explanation is the model itself. When the accuracy gap justifies a black box, use post-hoc, model-agnostic methods:

Permutation importance (global)

Already met in Random Forest: shuffle one feature's column in held-out data and measure the score drop. Breaking the feature–target link destroys exactly the information the model extracted from that feature.

Repeats give a distribution (boxes), separating real signal from shuffle noise. One caveat: with strongly correlated features, shuffling one leaves its twin available — both look unimportant even when the pair is critical. Check the correlation matrix alongside.

SHAP (local + global)

SHAP (Lundberg & Lee, 2017) answers the local question with game-theoretic rigor. Treat features as players cooperating to produce the prediction; the Shapley value (Shapley, 1953) \(\phi_j\) is feature \(j\)'s fair share of the payout — its contribution averaged over all orders in which features could join:

The only attribution scheme satisfying fairness axioms (efficiency — contributions sum exactly to the prediction minus the average; symmetry; additivity). Exact computation is exponential, but TreeSHAP computes it efficiently for tree ensembles — a perfect match for XGBoost-family models.

# pip install shap
import shap

explainer = shap.TreeExplainer(model)          # for tree ensembles
shap_values = explainer(X_test)

shap.plots.waterfall(shap_values[0])   # LOCAL: this prediction, feature by feature
shap.plots.beeswarm(shap_values)       # GLOBAL: importance + direction of effects
shap.plots.scatter(shap_values[:, "age"])   # dependence: effect of age across data

• Waterfall: from the base value, each feature pushes the prediction up (red) or down (blue) — the exact sentence a credit analyst needs: "denied mainly because: 3 overdue payments (+0.31), income below X (+0.12), tenure long (−0.05)";
• Beeswarm: one dot per sample per feature — global importance with direction (high values of feature X push predictions up?).

LIME (local)

LIME (Ribeiro et al., 2016 — "Why Should I Trust You?") explains one prediction by fitting a simple model in the neighborhood: perturb the instance, get black-box predictions for the perturbed samples (weighted by proximity), and fit a small linear model locally. The local surrogate's coefficients are the explanation.

Intuitive and works for any model and data type (its image/text variants toggle superpixels/words). Weaknesses: explanations depend on the perturbation scheme and neighborhood width, and can be unstable — run twice, get different stories. SHAP has largely become the default for tabular work; LIME remains conceptually important and useful beyond tables.

Reading explanations responsibly

Explanations are also the everyday debugging instrument: an implausibly dominant feature in a SHAP beeswarm is the classic signature of data leakage; a nonsense dependence plot reveals bad encoding; drift in explanation patterns flags production trouble.

Quiz