Skip to content

Overview

Data in Machine Learning

A trained model is a function fitted to a sample. Everything it will ever know about the world arrives through the rows you hand it, and every decision taken before training — what to collect, what to keep, what to fix, what to normalize, what to hold back — is a decision about what the model is able to learn. None of those decisions involve a single weight.

"Garbage in, garbage out" is true and nearly useless, because it does not say which garbage, or how you would know. This chapter is the specific version: the handful of decisions that are made before any model exists, what each one can break, and how to tell whether you got it right.

The one structural idea

The data pipeline has a line drawn through it — the train/test split — and one rule governs everything on either side:

Nothing that learns a number from the data may be fitted before the split.

A mean, a median, a vocabulary, a category average, a set of principal components, a list of selected columns: each is a parameter estimated from rows. Estimate it using rows you will later score against, and the score stops measuring generalization. Everything else in this chapter is a consequence of that sentence, or a way of checking it.

The order of operations

flowchart LR
    A["collect"] --> B["explore"]
    B --> C["clean<br/><small>row-wise fixes</small>"]
    C --> D["SPLIT"]
    D --> E["fit transformers<br/><small>on train only</small>"]
    E --> F["train"]
    F --> G["evaluate<br/><small>once</small>"]

    style D fill:#f0883e,stroke:#f0883e,color:#0d1117
    style G fill:#7fad7f,stroke:#3fb950

The order is not a style preference. Each step is placed where it is because of what it learns:

Step Does it estimate anything from the data? Where it belongs
Reading a histogram, a describe(), a scatter plot no — but you learn before, and spend your curiosity carefully
Dropping exact duplicate rows no before
Fixing a unit error, a typo, a known sentinel no before
Filling a gap with the median yes after — inside the pipeline
Standardizing, min-max scaling yes after — inside the pipeline
Building a category vocabulary, target encoding yes after — inside the pipeline
Selecting the k best features yes after — inside the pipeline
Oversampling the minority class yes after — training fold only
PCA, t-SNE, any learned projection yes after — inside the pipeline

The left column is the test. If a step would produce a different number when handed a different set of rows, it has parameters, and parameters estimated from the test set are parameters the test set can no longer be used to judge.

The part that catches everyone

Getting this right once, for a single split, is easy. The moment you cross-validate, tune hyperparameters or bootstrap, a hand-written scaler.fit(X_train) is fitted once while the folds change five times — and every fold is contaminated again. The only version of the rule that survives resampling is a Pipeline. See data leakage.

What goes wrong, and where

Each topic in this chapter is one stage of that pipeline failing.

  • Feature Types

    What kind of thing is in each column, and why a model cannot be told. Get this wrong and the network silently assumes that blue > red.

    Feature types

  • Distributions

    What the columns look like before you touch them — shape, scale, skew, modes — and what each of those implies for the transformation you are about to choose.

    Distributions

  • Train / Val / Test Split

    The line in the diagram. What each of the three sets is for, why three and not two, and the one rule about the test set that everybody breaks.

    Splitting

  • Data Leakage

    What happens when information crosses that line. Four mechanisms, each measured: a filter run before the split turns pure noise into 0.870 accuracy.

    Leakage

  • Data Quality

    Missing values, outliers, duplicates — and the column nobody audits, the labels. A network fits 40% wrong labels to a perfect training score.

    Quality

  • Class Imbalance

    When one class is 1% of the rows, accuracy stops being a measurement. What to change: the sampling, the loss, the threshold, or the metric.

    Imbalance

  • Preprocessing

    Everything on the right of the line: scaling, encoding, transforming skew, and projecting down. Each one is fitted, so each one belongs in the pipeline.

    Preprocessing

  • Handouts

    Three full class walkthroughs — exploratory analysis, normalization and dimensionality reduction — as single-page documents, plus a Colab notebook that runs the path end to end.

    Handouts

Where to get data

Source Domain Format
UCI ML Repository general, small, classic CSV, ARFF
OpenML benchmarks with metadata and published results ARFF, Parquet
Kaggle Datasets everything, quality varies wildly CSV, JSON
Hugging Face Datasets NLP, vision, audio, at scale Arrow, Parquet
TensorFlow Datasets vision, NLP, audio, ready to stream TFRecord
Papers With Code research benchmarks, with leaderboards various
Google Dataset Search web-wide index various

Before you download anything, ask what it is a sample of

A dataset is a sample from some population, drawn by some process, at some moment. A model trained on it inherits every one of those. The question that matters is not "is this dataset clean?" but "is the population it was drawn from the population my model will be deployed on?" A benchmark that everybody uses is not exempt — see the label-error audit in data quality.