The path of this part
- Spot redundancyfeatures that say the same thing; leaked columns
- Condition on a third variablecorrelation is not causation; Simpson's paradox
- Judge the odd valuesrules that disagree; error or rare but real
- Face the ceilingcapped values, censored targets
Each stage answers one question: do two columns say the same thing? → does the correlation survive a third variable? → is a flagged value an error? → what does a cap do to a model?
Stage 1 · do two columns say the same thing?
Redundant features
- The four counts (rooms, bedrooms, population, households) correlate at 0.86 to 0.98: all measure how big the district is.
- Iris: petal length and width, \(r = 0.963\).
- Not an error, but worth writing down: linear models get unstable coefficients; dimensionality reduction exploits it; for the counts, divide by the size.
A feature suspiciously good at predicting the target may be the target in disguise: a leaked column. If a model later turns out perfect, distrust it first.
Stage 2 · does the correlation survive a third variable?
Correlation is not causation
- Income and house value: \(r = 0.69\). Raising a district's income does not raise its prices, nor the reverse.
- Both respond to a third thing: the demand for living there (jobs, coast, schools).
- Ice cream sales and drownings correlate through summer.
Whenever you read a correlation, ask which third variable could be driving both.
Simpson's paradox: kidney stones
| success rate | treatment A | treatment B |
|---|---|---|
| small stones | 0.931 | 0.867 |
| large stones | 0.730 | 0.688 |
Predict. A wins for small stones and for large stones. Does A win overall?
Overall: A 0.780, B 0.826. A loses: \(273/350\) for A.
A was mostly given to large stones (263 of 350), which are harder to treat. The size of the stone is a confounder: it influences both the choice of treatment and the success.
The same trap in the housing data
Share of rooms that are bedrooms vs value:
overall \(r = -0.256\); within income bands Q1–Q4, +0.24 to +0.42.
Income drives both (\(r = -0.616\) with the bedroom share).
A correlation in pooled data is a hypothesis, not a finding: condition on the obvious third variables.
Stage 3 · is a flagged value an error?
On real data the rules disagree
population, IQR
1,196
population, \(|z| > 3\)
342
log(population), IQR
847
A long right tail puts many ordinary large districts beyond the fences at 1.5 IQR (interquartile range: \(Q_3 - Q_1\)).
The rules only flag candidates; they do not say what to do.
Error, or rare but real?
age = 190in a customer table: a typing error → fix it or set it to missing.- The owner's salary of 26: real → keep it; prefer robust statistics (median, quantiles, log).
- In between: values that are correct but mean something different.
Typical district: 2.82 people per household. One district: 7,460 people in 6 households, 19 rooms. No family does that; an institution (a prison, a base, a dormitory) does. The numbers are probably correct; their meaning changes.
Stage 4 · what does a cap do to a model?
Capped (censored) values
value at 500,001
965
4.7% of districts
age at 52
1,273
6.2%
income at 15.0001
49
0.2%
Top-coding: 500,001 means "at least 500,000"; the true value was lost. The capped districts are not random: mean income 7.83 against 3.87 overall.
A model trained on them underestimates the expensive districts. Keep and document, remove and restrict the model, or use a model for censored data. The indefensible option: not noticing.
How to spot a cap, and what it does
histogram
a spike at the very edge, taller than its neighbors (with enough bins)
scatter plot
a horizontal line: 965 districts at 500,001
the maximum
shared exactly by many rows: 1,273 districts at age 52
- A model trained on these values learns that no district is worth more than about 500,000.
- Relationships flatten at the top: income against value seems to level off after an income of about 8, partly because of the cap.
Predict. Will a model trained on this target make larger errors on cheap or on expensive districts?
Summary
| symptom | what is happening | what to do |
|---|---|---|
| two features with \(|r|\) above 0.9 | they measure the same thing | keep one, or make a ratio |
| one feature almost perfect at predicting the target | maybe the target in disguise | find out how it was produced |
| a correlation that changes sign within groups | a third variable drives both | report it within the groups |
| the IQR rule flags thousands of rows | a long tail, not errors | look at them; try a log or a ratio |
| many rows share the maximum exactly | the column is capped | document it; state the range the model covers |
Next: Data Preprocessing, acting on the problems the exploratory data analysis (EDA) found.
Advanced lab · about 30 min, in pairs
From the map to a data card, steps 8 to 11
- Same notebook, same session: continue from step 8.
- The map (latitude against value, band by band) → counts and ratios → outliers and odd districts → the written data card: facts, problems to fix, hypotheses to test.
- At every step, answer the Predict before running.
- Simulations: give location a direction (step 8) and clean every column with the IQR rule (step 10).
Pace: step 8 and the first simulation in 10 min; step 9 in 5 min; step 10 and the second simulation in 10 min; step 11 in 5 min.