Machine Learning · Insper

Decision Trees

Fundamental approach

How a tree chooses its questions, how it answers, and why it must be stopped

Twenty questions

A good player does not ask "is it my grandmother's teapot?"; they ask "is it alive?": either answer rules out about half the possibilities.

A tree plays the same game with data: each node asks feature ≤ threshold?
Each answer sends the example down a branch; each leaf holds a prediction.
Learning a tree is choosing the questions.

A good question is one whose answers leave groups that are purer — less mixed — than the group you started with.

The path of this part

Measure impuritytwenty questions; Gini and entropy
Find the best splitevery feature, every midpoint, scored
Grow and predictgreedy recursion; a leaf holds proportions
Stop in timeoverfitting and pre-pruning
Read the ruleseach path is a rule

Each stage answers one question: what makes a question good? → which question should a node ask? → how do splits become a model that answers? → when should the tree stop growing? → what does the tree tell us?

Stage 1 · what makes a question good?

12 customers, one question

	A	B	C	D	E	F	G	H	I	J	K	L
tenure (months)	2	3	5	6	8	11	14	19	24	30	38	50
bill (R$)	95	120	60	110	45	130	70	115	55	125	100	65
churn	1	1	0	1	0	1	0	0	0	1	0	0

Five of the 12 churned. Look for one question that separates the churners from the rest.

Every customer paying less than R$ 75 stayed. The rest of the lesson turns that hunch into a number, so that a computer can find the question without looking.

The parts of a tree

The full tree of the 12 customers with root, internal nodes, branches, leaves and depth

Impurity: how mixed is a node?

Gini (CART)

\[ G = 1 - \sum_c p_c^2 \]

chance that two examples drawn at random have different classes

Entropy (ID3, C4.5)

\[ H = -\sum_c p_c \log_2 p_c \]

bits needed to encode an example's class

Both are 0 at a pure node and largest at 50/50 ($G = 0.5$, $H = 1$). At the root, 7 stayed and 5 churned:

\[ G = 1 - \left(\tfrac{7}{12}\right)^2 - \left(\tfrac{5}{12}\right)^2 = 0.486 \qquad H = 0.980 \]

Stage 2 · which question should a node ask?

The best question: every split

\[ \Delta = I(\text{parent}) - \tfrac{n_L}{n} I(L) - \tfrac{n_R}{n} I(R) \]

The winner, by hand: bill ≤ 82.5

Left: 5 stayed, 0 churned → Gini 0.
Right: 2 stayed, 5 churned → $G_R = 1 - (2/7)^2 - (5/7)^2 = 0.408$.

\[ \tfrac{5}{12}\cdot 0 + \tfrac{7}{12}\cdot 0.408 = 0.238 \qquad \Delta = 0.486 - 0.238 = 0.248 \]

No other bill threshold comes close; the best on tenure (≤ 4) scores 0.350.
Between two neighboring values every threshold gives the same split: CART only tries the midpoints.

DecisionTreeClassifier(max_depth=1) finds the same feature, the same threshold and the same three Gini values: 0.486, 0 and 0.408.

Stage 3 · how do splits become a model that answers?

Greedy and recursive


GROW(node):
  if the node is pure, at max depth
  or too small: make it a leaf
  for each feature j and
  each midpoint t:
    score x_j <= t by the decrease Δ
  apply the best split
  GROW(left); GROW(right)

The depth-2 tree as a partition of the plane and as a flowchart

Recursive: each child is the same problem, smaller. Greedy: the best split now, without looking ahead — the optimal tree is NP-complete.

Predicting is following a path; the leaf holds proportions

customer	tenure	bill	leaf	P(stay)	P(churn)
M	6	70	5 stayed · 0 churned	1.000	0.000
N	4	115	0 · 4	0.000	1.000
O	25	110	2 · 1	0.667	0.333

predict_proba returns the class proportions of the leaf's training examples. A tree with 3 leaves can only produce 3 distinct probabilities.

Grown to purity

The tree growing from depth 0 to 3, with training accuracy 7/12, 10/12, 11/12 and 12/12

The third split exists because of one customer (J). 100% on training, nothing learned.

Simulator: growing a tree, one split at a time

Predict. Raise the depth from 0 to 8. Does the training impurity ever go up? What do the regions look like at depth 8?

Stage 4 · when should the tree stop growing?

On a real dataset: a fully grown tree memorizes

Spambase: 4,207 e-mails (without the 394 duplicated rows), 57 features.

limit	train	validation	leaves
none (fully grown)	1.000	0.906	181
`max_depth=3`	0.890	0.880	8
`max_depth=10`	0.967	0.915	86
`min_samples_leaf=50`	0.902	0.892	30
`max_leaf_nodes=40`	0.950	0.906	40

Pre-pruning: max_depth, min_samples_leaf, max_leaf_nodes. With 40 leaves, the same validation as the 181-leaf tree: the other 140 only memorized.

Stage 5 · what does the tree tell us?

A path is a rule

A depth-2 tree on Spambase (counts: not spam, spam):

$ rare and remove rare → not spam (1,770 against 356)
$ rare, but remove appears → spam (211 against 19)
$ frequent and hp rare → spam (687 against 50)
$ frequent, but hp appears → not spam (57 against 5): the data come from Hewlett-Packard

4 rules, 0.863 on the test set, against 0.923 for the full 221-leaf tree. The last rule is true in this dataset and useless anywhere else — and only reading tells you so.

Summary

A tree asks feature ≤ threshold questions, one at a time; predicting is following a path.
A good question leaves the children purer: weighted Gini or entropy.
The best split comes from brute force: every feature, every midpoint.
Growth is greedy and recursive.
Without a stopping rule the tree memorizes; cross-validation chooses how hard to hold it back.

symptom	what to do
training 1.000, validation well below	limit the depth or the leaf size
a small tree almost as good as the big one	keep the small one: it can be read

Fundamental lab · about 30 min, in pairs

A key for penguins, steps 1 to 5

Palmer penguins: 344 birds, three species, bill and flipper measurements, body mass, sex and island.
A field guide's dichotomous key is a decision tree written by hand: we will build one and read it.
Notebook on Colab (in Portuguese): nothing to download. Save a copy to your Drive.
At every step, answer the Predict before running.
Simulations: the root, chosen by you (step 4) and a tree memorizes even random labels (step 5).

Pace: steps 1 to 3 in 8 min; step 4 and the first simulation in 12 min; step 5 and the second in 10 min.