Data Quality

Data Quality

Data quality is not tidiness. A dataset can be perfectly formatted, fully populated and entirely wrong; another can be full of holes and still answer the question you are asking. Quality is fitness for a decision — and it is only ever measurable against the decision the model is supposed to make.

That is the useful frame, because it turns a vague virtue into three checkable questions.

Everything on this page is one of those three going wrong. The last one gets its own chapter — see train/validation/test split and data leakage — so here we take the first two, and the one thing nobody checks: the labels.

1. Missing values

Every course teaches the same reflex: count the NaNs, fill them with the mean, move on. The reflex is wrong, and the reason is that not all missing data is missing for the same reason. What matters is not how much is gone — it is who is gone.

The names are unhelpful — "at random" means two different things — so read them as a single question: can I tell, from what I still have, who left?

• MCAR: nobody left in particular, so the sample that remains is still fair.
• MAR: somebody left, but another column tells you who, so the gap can be reconstructed.
• MNAR: somebody left, and the only thing that identified them is precisely what is missing. No amount of cleverness recovers it. This is the one that ends careers, and it is invisible in the data: MCAR and MNAR look identical in a df.isnull().sum().

The panel drops 30% of an income column three different ways and applies three standard repairs. The blue outline is the population; the orange bars are what you would end up analysing.

What imputation actually does

Imputation does not recover information. It invents a plausible value and then — this is the part that matters — removes the evidence that it was invented. Two consequences follow, and both are visible in the panel:

1. It cannot fix a bias. Mean imputation fills every hole with the mean of what remains, so the mean of the repaired column is exactly the mean of the observed rows. If dropping was biased, filling is biased by the same amount, only now it looks like a complete dataset.
2. It shrinks the spread. Every imputed row sits exactly at the centre, so the variance falls, correlations weaken and every confidence interval computed afterwards is too narrow. You become more confident precisely because you know less.

from sklearn.impute import SimpleImputer, KNNImputer
from sklearn.pipeline import Pipeline

# the imputer learns a statistic, so it is a fit step: inside the Pipeline it goes
pipe = Pipeline([
    ('impute', SimpleImputer(strategy='median')),   # median: robust to the outliers below
    ('model',  MLPClassifier()),
])

Missingness is itself a feature

If a value is MNAR, the fact that it is missing carries the information the value would have carried. Throwing that away is a second loss on top of the first:

# keep the fact, not just the filled value
X['income_missing'] = X['income'].isna().astype(int)
X['income'] = X['income'].fillna(X['income'].median())

SimpleImputer(add_indicator=True) does this for you. It is close to free, and under MNAR it is often the single most predictive column in the table — which should tell you something about how much the imputation threw away.

The missing values that do not look missing

df.isnull().sum() finds NaN. It does not find:

These are worse than NaN because every tool treats them as data. A sentinel of -999 in a height column does not just add an outlier — it drags the mean, inflates the standard deviation and, as the lab below shows, breaks the very test you would use to detect it.

2. Outliers

An outlier is a point far from the others. That is a statement about the distribution, and it is not a reason to delete anything. The question is always which of these it is:

• A genuine extreme. A salary of ten million in an income dataset. Real, rare, and often the most important row in the table — in fraud detection or equipment failure, the outliers are the task.
• An error. A height of 1750 cm because the decimal point slipped. There is nothing to learn here; the row is a typo with a number attached.

No statistical test can tell these apart, because the difference is about the world, not about the numbers. A test can only say this is unusual; deciding what it means is domain work.

Detection, and how it fails

# z-score: distance from the mean, in standard deviations
z = ((df - df.mean()) / df.std()).abs()
outliers = (z > 3).any(axis=1)

# IQR: distance from the quartiles — robust, because quartiles do not move
q1, q3 = df.quantile(0.25), df.quantile(0.75)
iqr = q3 - q1
outliers = ((df < q1 - 1.5 * iqr) | (df > q3 + 1.5 * iqr)).any(axis=1)

# modified z-score: distance from the median, in MADs — the most robust of the three
median = df.median()
mad = (df - median).abs().median()
outliers = (0.6745 * (df - median).abs() / mad > 3.5).any(axis=1)

The first one has a flaw that matters. It measures every point against the mean and the standard deviation — both of which the outliers themselves have already corrupted. One bad value inflates the standard deviation; ten inflate it enough that none of them looks unusual any more. They hide behind the damage they caused. The name for this is masking, and step 2 of the lab shows a z-score test going from finding all 50 planted values to finding zero out of 100 as the contamination grows.

The median and the MAD do not move when a minority of the data does. That is the whole reason to prefer them.

Handling

For a neural network specifically, the argument for capping or transforming is not statistical purity — it is that a single enormous input value produces an enormous gradient, and one such row can undo an epoch of learning.

3. Duplicates

Exact duplicates inflate whatever they duplicate: a row that appears three times counts three times in the loss and pulls the decision boundary toward itself.

print(df.duplicated().sum())
df = df.drop_duplicates()

Near-duplicates are the harder case and the more damaging one — two photographs a second apart, the same article scraped from two mirrors, a record re-inserted with a different id. They survive drop_duplicates() because nothing is byte-identical.

4. Label noise — the one nobody audits

Everything above is about the columns. The column nobody audits is the one the model is trying to predict.

Teams spend weeks on features and accept y as given. But y was produced by somebody — an annotator working through ten thousand images, a rule written in SQL three years ago, a clinical code entered under time pressure — and it is wrong more often than anyone budgets for. A study of ten of the most widely used benchmark test sets found label errors averaging 3.4%, and about 6% of the ImageNet validation set.¹ Those are the datasets whose numbers the whole field quotes to three decimal places.

For a neural network this is not a small problem, because of a specific and well-established fact: a network with enough capacity will fit any labels at all, including labels assigned entirely at random.² It does not resist your errors. It memorizes them, reports a beautiful training curve, and loses exactly as much held-out accuracy as you gave it wrong labels.

The panel makes this concrete. The dashed line is the rule the labels are meant to encode; the red-ringed points have been given the wrong answer. Set k = 1 and watch the model carve out an island around every single one:

Two things to take from it, and they generalize far past k-nearest-neighbours:

• A perfect training score is not evidence of anything. At k = 1 the training accuracy is 1.000 whether the labels are 0% or 40% wrong. The number that moves is the one measured against the truth.
• Fitting the training set less well can be the right answer. Raising k makes the model openly disagree with dozens of the labels it was given — and the honest score goes up.

What to do about it

In order of how much good they do:

1. Look at your labels. Sample 100 rows and label them yourself. Nothing else on this list substitutes for knowing your error rate.
2. Measure agreement. Two annotators on the same subset, then Cohen's \(\kappa\). If humans do not agree with each other, no model will agree with them.
3. Let the model find them. Rows with a persistently high loss, or which the model confidently contradicts, are the likeliest label errors. This idea, formalized, is confident learning — the method behind the benchmark audit cited above.¹
4. Stop early. Networks learn the signal before they memorize the noise, so the damage accumulates in the later epochs. In step 3 of the lab, early stopping turns 0.639 back into 0.825 at 40% noise.
5. Soften the targets. Label smoothing replaces a hard 1 with \(1-\varepsilon\), which caps how confident the network can become about any single example — including the wrong ones.
6. Average over it. An ensemble disagrees with itself precisely where the labels are unreliable.

The audit

Run this before anything else, on every new dataset, and read all of it:

def audit(df, target=None):
    print(f"shape             {df.shape}")
    print(f"exact duplicates  {df.duplicated().sum()}")

    print("\nmissing (%)")
    missing = (df.isna().mean() * 100).round(1)
    missing = missing[missing > 0].sort_values(ascending=False)
    print(missing.to_string() if len(missing) else "  none")

    print("\nsentinels posing as data")
    for col in df.select_dtypes('number'):
        if df[col].nunique() <= 2:                  # a flag or a binary target, not a measurement
            continue
        for sentinel in (-1, 0, -999, 9999):
            n = (df[col] == sentinel).sum()
            if n and n / len(df) > 0.01:
                print(f"  {col}: {sentinel} appears {n} times ({n/len(df):.1%})")

    print("\nconstant or near-constant columns")
    for col in df:
        top = df[col].value_counts(normalize=True, dropna=False).iloc[0]
        if top > 0.98:
            print(f"  {col}: one value covers {top:.1%} of rows")

    print("\ntails — a min or a max far outside the 1%/99% range is a suspect")
    print(df.describe(percentiles=[.01, .5, .99]).T[['1%', '50%', '99%', 'min', 'max']])

    if target:
        print(f"\nclass balance\n{df[target].value_counts(normalize=True)}")

Lab: three questions, measured

Three short experiments, reproducible with numpy and scikit-learn.

Step 1 — the mechanism decides, not the percentage

The same 30% of an income column goes missing three different ways. Income depends on age; age is always observed. bias_drop is what you get from dropping the incomplete rows, bias_model_impute from predicting the missing income out of the observed age, and std_ratio is what mean imputation does to the spread.

"""The mechanism decides whether a missing value costs you anything.

Income depends on age, and age is always observed. Thirty percent of the income
column is then hidden three different ways, and the same three repairs are
tried on each. What changes is not how much is missing — it is always 30% —
but whether the rows that vanished are the ones you needed.

Printed as a markdown table, in identifiers only, so one artifact serves both
the English and the Portuguese page.
"""

import numpy as np
from sklearn.linear_model import LinearRegression

N, FRACTION = 4000, 0.30

rng = np.random.default_rng(7)
age = rng.normal(size=N)                                    # always observed
income = np.exp(3.0 + 0.5 * age + 0.4 * rng.normal(size=N))  # and it depends on age
TRUE_MEAN, TRUE_STD = income.mean(), income.std()


def hide(mechanism):
    r = np.random.default_rng(0)
    if mechanism == "MCAR":                 # the server dropped rows, blindly
        return r.random(N) < FRACTION
    if mechanism == "MAR":                  # the young skip the question — and age is observed
        p = 1 / (1 + np.exp(2.2 * age))
        return r.random(N) < p * FRACTION / p.mean()
    return income >= np.quantile(income, 1 - FRACTION)      # MNAR: the rich do not answer


rows = []
for mechanism in ("MCAR", "MAR", "MNAR"):
    gone = hide(mechanism)
    seen = income[~gone]

    # 1. drop the rows — which is also exactly what mean imputation estimates
    drop = seen.mean()

    # 2. mean imputation: same centre, and a standard deviation that is now a lie
    filled = np.where(gone, drop, income)

    # 3. regression imputation: predict the missing income from the observed age
    fit = LinearRegression().fit(age[~gone, None], np.log(seen))
    residual = np.log(seen) - fit.predict(age[~gone, None])
    model = np.where(
        gone, np.exp(fit.predict(age[:, None]) + residual.var() / 2), income
    ).mean()

    rows.append((mechanism, gone.mean(), drop - TRUE_MEAN,
                 model - TRUE_MEAN, filled.std() / TRUE_STD))

print(f"| `mechanism` | `missing` | `bias_drop` | `bias_model_impute` | `std_ratio` |")
print("|---|---:|---:|---:|---:|")
for name, frac, bias_drop, bias_model, ratio in rows:
    print(f"| `{name}` | {frac:.0%} | **{bias_drop:+.2f}** | **{bias_model:+.2f}** "
          f"| {ratio:.2f} |")
print(f"| `truth` | 0% | mean = {TRUE_MEAN:.2f} | — | std = {TRUE_STD:.2f} |")

Read the rows against each other. Under MCAR, dropping is off by +0.05 on a mean of 24.14 — nothing. Under MAR, dropping is off by +3.18, a 13% error, and the model-based repair brings it back to −0.14: the bias was fixable because age told you who had left. Under MNAR dropping is off by −8.41 — the estimate is 35% too low — and the repair only gets to −5.69, because no column in the table knows what the rich would have answered.

The std_ratio column is the quieter failure. Mean-imputing 30% of a column leaves the spread at 0.86 of the truth even under MCAR, where nothing was biased. Under MNAR it falls to 0.30: the top third of the distribution has been replaced by a single value, and every downstream standard error is now fiction.

Step 2 — the outlier test that stops working

A height column in centimetres, with a legacy system writing -999 instead of NULL.

"""Why the z-score stops finding outliers exactly when there are enough to matter.

A height column, in centimetres, where some rows carry the sentinel -999 that a
legacy system writes instead of NULL. The z-score test measures each value
against the mean and the standard deviation — both of which the sentinels
themselves have already wrecked. Past a certain contamination the sentinels
have inflated the standard deviation so much that they no longer look extreme:
they hide behind the damage they caused. This is called masking.

The median and the MAD do not move when a minority of the values does, so the
modified z-score keeps working.

Printed as a markdown table, in identifiers only, so one artifact serves both
the English and the Portuguese page.
"""

import numpy as np

N, MEAN, STD, SENTINEL = 1000, 170.0, 10.0, -999.0

rng = np.random.default_rng(11)
clean = rng.normal(MEAN, STD, N)

print("| `sentinels` | `mean` | `std` | `planted` | `z > 3` | `mad_z > 3.5` |")
print("|---:|---:|---:|---:|---:|---:|")
for fraction in (0.00, 0.01, 0.05, 0.10, 0.15):
    height = clean.copy()
    hidden = rng.choice(N, int(fraction * N), replace=False)
    height[hidden] = SENTINEL
    k = len(hidden)

    # the textbook test: distance from the mean, in standard deviations
    z = np.abs(height - height.mean()) / height.std()

    # the robust twin: distance from the median, in MADs
    median = np.median(height)
    mad = np.median(np.abs(height - median))
    modified_z = 0.6745 * np.abs(height - median) / mad

    print(f"| {fraction:.0%} | {height.mean():.1f} | {height.std():.1f} | {k} "
          f"| **{int((z > 3).sum())}** | **{int((modified_z > 3.5).sum())}** |")

Follow the mean column first: 170.2 cm, then 158.5, then 111.6, then 53.3, then −5.2. That is the reported average height of a population, and no exception was raised at any point.

Now the two tests. At 1% and 5% contamination both find every sentinel. At 10%, the z-score finds 0 of 100 — the sentinels have pushed the standard deviation to 350.9, and against a spread that large nothing is three deviations from anything. The modified z-score finds all 100, and all 150 at 15%, because the median and the MAD never moved.

Step 3 — the network fits whatever you tell it

The inputs are untouched. Only a fraction of the training labels is flipped.

"""A network fits wrong labels as happily as right ones.

The task is linearly separable up to a little noise, so a clean network reaches
about 0.95 on held-out data. Then a fraction of the *training* labels is
flipped — the inputs are untouched, only the answers are corrupted — and the
same network is trained again.

The training accuracy stays at 1.000 no matter how much of the training set is
wrong. That is the whole point: a network with enough capacity memorizes the
errors, reports a perfect fit and tells you nothing. Only the held-out score
knows. The last column shows early stopping getting some of it back, because
memorizing noise takes longer than learning the signal.

Printed as a markdown table, in identifiers only, so one artifact serves both
the English and the Portuguese page.
"""

import numpy as np
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

N, DIM, SPLIT = 2000, 12, 1200

rng = np.random.default_rng(23)
X = rng.normal(size=(N, DIM))
direction = rng.normal(size=DIM)
y = ((X @ direction + 0.35 * rng.normal(size=N)) > 0).astype(int)
train, test = slice(0, SPLIT), slice(SPLIT, N)

net = lambda **kw: make_pipeline(
    StandardScaler(), MLPClassifier((256, 128), max_iter=4000, random_state=0, **kw)
)

print("| `flipped_labels` | `train_acc` | `test_acc` | `test_acc_early_stop` |")
print("|---:|---:|---:|---:|")
for rate in (0.00, 0.10, 0.20, 0.30, 0.40):
    noisy = y[train].copy()
    flip = np.random.default_rng(0).random(SPLIT) < rate
    noisy[flip] ^= 1                                   # only the answers are corrupted

    memorizer = net().fit(X[train], noisy)
    stopped = net(early_stopping=True, n_iter_no_change=10).fit(X[train], noisy)

    print(f"| {rate:.0%} | **{memorizer.score(X[train], noisy):.3f}** "
          f"| **{memorizer.score(X[test], y[test]):.3f}** "
          f"| {stopped.score(X[test], y[test]):.3f} |")

The train_acc column is 1.000 on every row. With 40% of its training labels wrong, the network fits all of them, perfectly, and its loss curve gives no sign whatever that anything is amiss. Meanwhile the honest score falls from 0.951 to 0.639 — very nearly point for point with the noise it was fed.

The last column is the consolation. Early stopping recovers 0.825 of that 0.639, because the signal is learned in the early epochs and the memorization comes later: stopping while the held-out score is still improving means stopping before the network gets around to the wrong labels. It is a mitigation, not a fix — 0.825 is still well below the 0.951 that clean labels would have bought.