Exploratory Data Analysis

Exploratory Data Analysis (EDA) is the disciplined practice of looking at your data before modeling it. The term was coined by John Tukey (1977), who argued that data analysis should begin with open-minded exploration — plots, summaries, anomalies — rather than jumping straight to hypothesis tests or models.

EDA is step 2 of the ML workflow, and skipping it is the most common source of silent failure: models trained on misunderstood data produce confident nonsense.

What to look for

A practical EDA checklist:

1. Shape and types — how many rows and columns? Which are numeric, categorical, dates, text, identifiers?
2. Missing values — how many, in which columns, and why are they missing?
3. Distributions — center, spread, skewness, multimodality of each feature;
4. Outliers — legitimate extremes or data-entry errors?
5. Relationships — correlations between features, and between features and the target;
6. Class balance — for classification, how frequent is each class?
7. Duplicates and leakage suspects — repeated rows, columns that "know the future" (see Validation & Data Leakage).

First contact with a dataset

import pandas as pd
from sklearn.datasets import load_iris

iris = load_iris(as_frame=True)
df = iris.frame

df.shape          # (150, 5) — rows, columns
df.head()         # first rows: eyeball the values
df.info()         # dtypes and non-null counts
df.describe()     # count, mean, std, min, quartiles, max
df.isna().sum()   # missing values per column
df.duplicated().sum()

Summary statistics

For a numeric feature \(x_1, \dots, x_n\):

• Mean: \(\bar{x} = \frac{1}{n}\sum_{i=1}^{n} x_i\) — sensitive to outliers;
• Median: the middle value — robust to outliers;
• Standard deviation: \(s = \sqrt{\frac{1}{n-1}\sum_{i=1}^{n}(x_i - \bar{x})^2}\) — typical distance from the mean;
• Quartiles / IQR: \( \text{IQR} = Q_3 - Q_1 \) — the range of the middle 50%, the basis of the boxplot.

A large gap between mean and median is a skewness alarm — think of income data, where a few large values drag the mean up.

Why statistics are not enough

The four datasets below — Anscombe's quartet — have identical means, variances, correlations (\(r \approx 0.816\)), and least-squares lines. Only a plot reveals how different they are:

Dataset 2 is nonlinear, dataset 3 has one outlier distorting the line, dataset 4 has a single point creating the entire correlation. Always plot.

Core visualizations

An example on the iris dataset — one distribution view and one correlation view:

The left panel already tells a modeling story: petal length alone almost separates the three species. The right panel warns that petal length and petal width are highly correlated (\(r = 0.96\)) — they carry nearly the same information, which will matter for dimensionality reduction and for interpreting linear model coefficients.

Correlation, carefully

The Pearson correlation between features \(x\) and \(y\):

Three standard warnings:

• \(r\) measures linear association only — Anscombe's dataset 2 has strong structure and misleading \(r\);
• correlation is not causation — ice cream sales and drownings correlate through a confounder (summer);
• correlation on aggregated data can invert at the individual level (Simpson's paradox).

Missing values: the why matters

Deleting all rows with any missing value is only harmless under MCAR — otherwise it biases the dataset. Imputation strategies are covered in Data Preprocessing.

Outliers

The classic boxplot rule flags points outside \([Q_1 - 1.5\,\text{IQR},\; Q_3 + 1.5\,\text{IQR}]\). But the rule only flags — the decision needs judgment:

• an age = 190 is a data-entry error → fix or remove;
• a purchase = R$ 500,000 may be your most important customer → keep, and choose robust models/metrics.

Beyond Pearson: measuring any association

Pearson's \(r\) only covers pairs of continuous variables. The full toolkit for bivariate analysis:

When reading feature–target associations, keep two class rules of thumb in mind:

• features strongly associated with each other → likely redundancy (temperature in °C and °F; price in R$ and US$) — consider dropping one;
• a feature suspiciously strongly associated with the target → it may be the target in disguise — a leaked column (see Data Leakage). If your model is suddenly perfect, distrust it first.

Don't snoop: split before you explore

One subtlety the class dataset drills in: perform the train/test split before the exploratory analysis, and run EDA on the training set only. Exploring the full dataset means "peeking" at the test set — data snooping — and what you learn (which features look promising, where the outliers are, which transformations to apply) silently influences decisions that the test set was supposed to judge from the outside.

Hands-on: California Housing

The class notebook works through a full EDA on the California Housing dataset (Géron's Hands-On ML, chapter 2): 20,640 districts × 10 columns — 9 features (location, housing age, rooms, income, ocean_proximity...) and the regression target median_house_value.

import pandas as pd

df = pd.read_csv('https://raw.githubusercontent.com/hsandmann/biblio/refs/heads/main/ml/aula02/housing.csv')

X = df.drop('median_house_value', axis=1).copy()   # features  (m × n)
y = df['median_house_value'].copy()                # target    (m,)

Guided exercises from the notebook:

1. How many examples and columns? What does each column mean — continuous or categorical?
2. Split train/test (80/20) first; explain the role of random_state;
3. Univariate analysis on the training set: descriptive statistics and plots per feature — hunt for anomalies (missing values in total_bedrooms, saturation at median_house_value = 500,001), skewed distributions, rare categories;
4. Bivariate analysis: which feature pairs are strongly associated? Which features associate most with the target?

Class materials

Quiz