Overview
Data in Machine Learning
A trained model is a function fitted to a sample. Everything it will ever know about the world arrives through the rows you hand it, and every decision taken before training — what to collect, what to keep, what to fix, what to normalize, what to hold back — is a decision about what the model is able to learn. None of those decisions involve a single weight.
"Garbage in, garbage out" is true and nearly useless, because it does not say which garbage, or how you would know. This chapter is the specific version: the handful of decisions that are made before any model exists, what each one can break, and how to tell whether you got it right.
The one structural idea
The data pipeline has a line drawn through it — the train/test split — and one rule governs everything on either side:
Nothing that learns a number from the data may be fitted before the split.
A mean, a median, a vocabulary, a category average, a set of principal components, a list of selected columns: each is a parameter estimated from rows. Estimate it using rows you will later score against, and the score stops measuring generalization. Everything else in this chapter is a consequence of that sentence, or a way of checking it.
Simulator: NetLab · Train, validation and noise
This class has its own scenario in NetLab, the course's network lab: the graph comes built, with a note on what to try and a link back here. Train it, change it, break it. A graph you had in the simulator is not lost: it is one Ctrl+Z away.
The order of operations
The order is not a style preference. Each step is placed where it is because of what it learns:
| Step | Does it estimate anything from the data? | Where it belongs |
|---|---|---|
Reading a histogram, a describe(), a scatter plot | no — but you learn | before, and spend your curiosity carefully |
| Dropping exact duplicate rows | no | before |
| Fixing a unit error, a typo, a known sentinel | no | before |
| Filling a gap with the median | yes | after — inside the pipeline |
| Standardizing, min-max scaling | yes | after — inside the pipeline |
| Building a category vocabulary, target encoding | yes | after — inside the pipeline |
| Selecting the k best features | yes | after — inside the pipeline |
| Oversampling the minority class | yes | after — training fold only |
| PCA, t-SNE, any learned projection | yes | after — inside the pipeline |
The middle column is the test. If a step would produce a different number when handed a different set of rows, it has parameters, and parameters estimated from the test set are parameters the test set can no longer be used to judge.
The part that catches everyone
Getting this right once, for a single split, is easy. The moment you cross-validate, tune hyperparameters or bootstrap, a hand-written scaler.fit(X_train) is fitted once while the folds change five times — and every fold is contaminated again. The only version of the rule that survives resampling is a Pipeline. See data leakage.
What goes wrong, and where
Each topic in this chapter is one stage of that pipeline failing.
-
Feature Types
What kind of thing is in each column, and why a model cannot be told. Get this wrong and the network silently assumes that
blue > red. -
Distributions
What the columns look like before you touch them — shape, scale, skew, modes — and what each of those implies for the transformation you are about to choose.
-
Train / Val / Test Split
The line in the diagram. What each of the three sets is for, why three and not two, and the one rule about the test set that everybody breaks.
-
Data Leakage
What happens when information crosses that line. Four mechanisms, each measured: a filter run before the split turns pure noise into 0.870 accuracy.
-
Data Quality
Missing values, outliers, duplicates — and the column nobody audits, the labels. A network fits 40% wrong labels to a perfect training score.
-
Class Imbalance
When one class is 1% of the rows, accuracy stops being a measurement. What to change: the sampling, the loss, the threshold, or the metric.
-
Preprocessing
Everything on the right of the line: scaling, encoding, transforming skew, and projecting down. Each one is fitted, so each one belongs in the pipeline.
-
Handouts
Three full class walkthroughs — exploratory analysis, normalization and dimensionality reduction — as single-page documents, plus a Colab notebook that runs the path end to end.
Where to get data
| Source | Domain | Format |
|---|---|---|
| UCI ML Repository | general, small, classic | CSV, ARFF |
| OpenML | benchmarks with metadata and published results | ARFF, Parquet |
| Kaggle Datasets | everything, quality varies wildly | CSV, JSON |
| Hugging Face Datasets | NLP, vision, audio, at scale | Arrow, Parquet |
| TensorFlow Datasets | vision, NLP, audio, ready to stream | TFRecord |
| Google Dataset Search | web-wide index | various |
Before you download anything, ask what it is a sample of
A dataset is a sample from some population, drawn by some process, at some moment. A model trained on it inherits every one of those. The question that matters is not "is this dataset clean?" but "is the population it was drawn from the population my model will be deployed on?" A benchmark that everybody uses is not exempt — see the label-error audit in data quality.
Key takeaways
- A model only knows what the rows tell it. Every decision taken before training (what to collect, keep, fix, normalize, hold back) decides what it is able to learn.
- Nothing that learns a number from the data may be fitted before the split. A median, a scaler, a vocabulary, a choice of columns, a PCA: each is fitted on the training rows only, inside a
Pipeline, and thePipelineis what you cross-validate. - An encoding asserts which comparisons are meaningful. Integer codes invent an order and its distances; one-hot refuses any structure.
LabelEncoderon an input feature is never right. - Look at the distributions before transforming. Comparing mean and median catches most skew. A transformation does not add information: if the classes overlap on every feature, no model separates them, and the fix is a better feature.
- Validation is for every choice; the test set is for one number, read once. A single split is a lottery (worth 15 points on the splitting page), and any set you choose on stops being a set you can measure on.
- Leakage makes noise look like signal: a feature filter run before the split turned pure noise into 0.870 accuracy. The leak everyone warns about (standardizing early, +0.004 AUC) is the cheapest; selection, groups and resampling before the split cost the most. A surprisingly good score is an incident to investigate.
- Audit the data, labels included. What a missing value means depends on who left, the usual outlier test fails when the outliers are many, and a network fits 40% wrong labels to a perfect training score.
- With imbalance, change the report and the cut first: accuracy and ROC-AUC flatter a rare class, and the threshold is where the costs enter. Resample only inside the training fold.