Machine Learning · Insper

Decision Trees

Fundamental approach

How a tree chooses its questions, how it answers, and why it must be stopped

Twenty questions

A good player does not ask "is it my grandmother's teapot?"; they ask "is it alive?": either answer rules out about half the possibilities.

A tree plays the same game with data: each node asks feature ≤ threshold?
Each answer sends the example down a branch; each leaf holds a prediction.
Learning a tree is choosing the questions.

A good question is one whose answers leave groups that are purer — less mixed — than the group you started with.

The path of this part

Measure impuritytwenty questions; Gini and entropy
Find the best splitevery feature, every midpoint, scored
Grow and predictgreedy recursion; a leaf holds proportions
Stop in timeoverfitting and pre-pruning
Read the ruleseach path is a rule

Each stage answers one question: what makes a question good? → which question should a node ask? → how do splits become a model that answers? → when should the tree stop growing? → what does the tree tell us?

Stage 1 · what makes a question good?

12 customers, one question

	A	B	C	D	E	F	G	H	I	J	K	L
tenure (months)	2	3	5	6	8	11	14	19	24	30	38	50
bill (R$)	95	120	60	110	45	130	70	115	55	125	100	65
churn	1	1	0	1	0	1	0	0	0	1	0	0

Five of the 12 churned. Look for one question that separates the churners from the rest.

Every customer paying less than R$ 75 stayed. The rest of the lesson turns that hunch into a number, so that a computer can find the question without looking.

The parts of a tree

The full tree of the 12 customers with root, internal nodes, branches, leaves and depth

Why does the root ask about the bill, and why 82.5? Stage 2 answers it, step by step.

Impurity: how mixed is a node?

Gini (CART)

\[ G = 1 - \sum_c p_c^2 \]

chance that two examples drawn at random have different classes

Entropy (ID3, C4.5)

\[ H = -\sum_c p_c \log_2 p_c \]

bits needed to encode an example's class

Both are 0 at a pure node and largest at 50/50 ($G = 0.5$, $H = 1$). At the root, 7 stayed and 5 churned:

\[ G = 1 - \left(\tfrac{7}{12}\right)^2 - \left(\tfrac{5}{12}\right)^2 = 0.486 \qquad H = 0.980 \]

Stage 2 · which question should a node ask?

Choosing the root, step by step

Predict. Before stepping: which feature will the root ask about, and why? Then go through the 22 candidates and watch the formula change.

Stage 3 · how do splits become a model that answers?

Greedy and recursive


GROW(node):
  if the node is pure, at max depth
  or too small: make it a leaf
  for each feature j and
  each midpoint t:
    score x_j <= t by the decrease Δ
  apply the best split
  GROW(left); GROW(right)

The depth-2 tree as a partition of the plane and as a flowchart

Recursive: each child is the same problem, smaller. Greedy: the best split now, without looking ahead — the optimal tree is NP-complete.

Predicting is following a path; the leaf holds proportions

The depth-2 tree with three new customers: M (6 months, R$ 70) answers 70 ≤ 82.5 and lands in the leaf of 5 who stayed, P(stay) = 1; N (4 months, R$ 115) answers 115 > 82.5 and 4 ≤ 15 and lands in the leaf of 4 who churned, P(churn) = 1; O (25 months, R$ 110) answers 110 > 82.5 and 25 > 15 and lands in the leaf with H, J and K, 2 who stayed and 1 who churned, P(stay) = 2/3 = 0.67

predict_proba returns the leaf's proportions: O gets 0.67 because 2 of the 3 training customers there stayed. 3 leaves, only 3 possible probabilities.

Grown to purity

The tree growing from depth 0 to 3, with training accuracy 7/12, 10/12, 11/12 and 12/12

The third split exists because of one customer (J). 100% on training, nothing learned.

Simulator: growing a tree, one split at a time

Predict. Raise the depth from 0 to 8. Does the training impurity ever go up? What do the regions look like at depth 8?

Stage 4 · when should the tree stop growing?

On a real dataset: a fully grown tree memorizes

Spambase: 4,207 e-mails (without the 394 duplicated rows), 57 features.

limit	train	validation	leaves
none (fully grown)	1.000	0.906	181
`max_depth=3`	0.890	0.880	8
`max_depth=10`	0.967	0.915	86
`min_samples_leaf=50`	0.902	0.892	30
`max_leaf_nodes=40`	0.950	0.906	40

Pre-pruning: max_depth, min_samples_leaf, max_leaf_nodes. With 40 leaves, the same validation as the 181-leaf tree: the other 140 only memorized.

Stage 5 · what does the tree tell us?

A path is a rule

A depth-2 tree on Spambase (counts: not spam, spam):

$ rare and remove rare → not spam (1,770 against 356)
$ rare, but remove appears → spam (211 against 19)
$ frequent and hp rare → spam (687 against 50)
$ frequent, but hp appears → not spam (57 against 5): the data come from Hewlett-Packard

4 rules, 0.863 on the test set, against 0.923 for the full 221-leaf tree. The last rule is true in this dataset and useless anywhere else — and only reading tells you so.

Summary

A tree asks feature ≤ threshold questions, one at a time; predicting is following a path.
A good question leaves the children purer: weighted Gini or entropy.
The best split comes from brute force: every feature, every midpoint.
Growth is greedy and recursive.
Without a stopping rule the tree memorizes; cross-validation chooses how hard to hold it back.

symptom	what to do
training 1.000, validation well below	limit the depth or the leaf size
a small tree almost as good as the big one	keep the small one: it can be read

Fundamental lab · about 30 min, in pairs

A key for penguins, steps 1 to 5

Palmer penguins: 344 birds, three species, bill and flipper measurements, body mass, sex and island.
A field guide's dichotomous key is a decision tree written by hand: we will build one and read it.
Notebook on Colab (in Portuguese): nothing to download. Save a copy to your Drive.
At every step, answer the Predict before running.
Simulations: the root, chosen by you (step 4) and a tree memorizes even random labels (step 5).

Pace: steps 1 to 3 in 8 min; step 4 and the first simulation in 12 min; step 5 and the second in 10 min.