AutoML

The question no lesson asked yet

• Every model comes with knobs; Model Selection explained what they do, Cross-Validation how to compare the results honestly.
• A grid with 5 hyperparameters and 5 values each: 3,125 cross-validations, most of them in regions that were obviously bad after the first dozen.

The path of this lesson

One call to the black box

\(\lambda\): a configuration. \(f(\lambda)\): its cross-validated score. Tuning = find the \(\lambda\) with the largest \(f(\lambda)\).

• kc1: 2,109 software modules of a NASA flight system, 21 code metrics; target: was a defect found later? (15.5% defective)
• One call = a 5-fold CV ROC AUC of a boosted model.

Expensive, noisy, and a black box

Designing the space (1): the scale

By hand: \(0.009 / 0.999 = 0.9\%\); on the log scale each of the 3 decades gets a third.

Designing the space (2): conditional knobs

• C belongs to the logistic regression, max_leaf_nodes to the booster, gamma to an RBF kernel.
• One grid over all of them would pair every C with every tree size: the same logistic regression, fitted again and again.

Grid or random? The argument

• A score that depends on one hyperparameter \(x_1\) (a bump of width 0.05); the other \(d - 1\) do nothing.
• A grid with \(k\) values per axis costs \(k^d\) evaluations, but tries \(x_1\) at only \(k\) values.
• Random search with budget \(B\) tries \(x_1\) at \(B\) values, whatever \(d\) is.

Grid or random? Measured on 4,000 problems

d	budget	grid	random
2	9	0.378	0.629
2	81	0.824	0.985
3	27	0.378	0.900
4	81	0.378	0.986
4	256	0.501	0.998

Mean best score. The grid pays for every irrelevant axis with a multiplication of its cost; random search pays nothing (Bergstra & Bengio, 2012).

Simulator: same budget, different searches

Predict. Budget 81: how many distinct values of \(x_1\) does the 9 × 9 grid try, and how many does random search try?

Bayesian optimization: a search that learns

Random search draws the 27th configuration exactly like the first. When each evaluation costs an hour, that is wasteful.

1. fit a cheap surrogate to the (configuration, score) pairs so far: it predicts the score anywhere, with an uncertainty;
2. choose the next configuration by maximizing an acquisition function: exploitation (high predicted score) against exploration (high uncertainty);
3. evaluate it, add it to the history, repeat.

A stand-in score with a known truth: a broad peak of 0.78 at 0.25 and a higher, narrower one of 0.82 at 0.72.

The surrogate: a Gaussian process

Three evaluations so far: 0.15 and 0.35 (0.72943), 0.95 (0.700). Best so far \(f^+ = 0.72943\).

Expected improvement, by hand

Four iterations

1: exploits 0.25 → 0.780. 2: its EI collapses, so it explores the gap → 0.72, 0.820. 3–4: refine at 0.685 and 0.775. Random search would need about 28 draws to land that close to 0.72.

The surrogate's assumptions matter

100 runs from random starting triples, 30 evaluations each. Share of runs that reached 0.81:

TPE: model \(x\) given the score

• Split the trials into the good ones (best fraction \(\gamma\)) and the rest.
• Density of the good values \(\ell(x)\), density of the others \(g(x)\): one Gaussian bump per trial.
• Sample where \(\ell(x)/g(x)\) is largest (Bergstra et al., 2011).

Optuna: TPE against random, measured

Six hyperparameters: one matters a lot, one a little, four not at all. Best possible: 0.870. 20 seeds.

Identical at 10: TPE's first 10 trials are random. After 40 it is above what random reaches after 60.

Define-by-run: the space is code

def build(trial):
    family = trial.suggest_categorical('family', ['logistic', 'boosting'])
    if family == 'logistic':
        C = trial.suggest_float('C', 1e-3, 1e2, log=True)
        return make_pipeline(FunctionTransformer(np.log1p), StandardScaler(),
                             LogisticRegression(C=C, max_iter=2000))
    return HistGradientBoostingClassifier(
        learning_rate=trial.suggest_float('learning_rate', 0.01, 0.5, log=True), ...)
  

Pruning: stop a bad trial early

• The objective reports its running mean after every fold (trial.report).
• The median pruner stops a trial worse than the median of earlier trials at the same fold.

Successive halving: a tournament

Start \(n = 27\) configurations on a small budget; keep the best third; give the survivors 3 × more; repeat.

What the tournament assumes

The ranking after a small budget predicts the ranking after the full one. Two tournaments: budget in trees, budget in rows.

winner with budget	full score	rank
trees	0.781	13 of 27
rows	0.787	5 of 27
best at full budget	0.791	1

With a small learning rate, 10 trees have barely started: the slow learners, best at 270 trees, all died in round 1.

Hyperband: several tournaments

How aggressive should the tournament be? Hyperband runs brackets of different aggressiveness, each costing about the same (Li et al., 2018).

Full-pipeline AutoML: the CASH problem

Combined Algorithm Selection and Hyperparameter optimization (Thornton et al., 2013).

It optimizes the leak you gave it

bug_reports: filed after release; the model must flag risky modules before release, when the count is zero for everyone.

Every automated check passes. The leak is in the data: only someone who knows when each column is recorded can catch it.

A longer search overfits the validation data

The rest of the job

And deployment: a stacked ensemble of 40 models may be the hardest to serve, explain and retrain.

A practical default

1. Lock a test set away and fix one cv.
2. Baselines first: a dummy, a linear model, a booster with defaults.
3. Random search or Optuna, 30–100 trials, log scales, the simple model in the space.
4. For large data or slow models: halving or pruning, with a budget unit that does not interact with the knobs.
5. Stop when the best score stops moving by more than its fold standard deviation.
6. Report the locked test score, or a nested CV of the whole search, never the search's own best score.

Four searches, one budget

phoneme: 5,404 vowels, oral or nasal. Grid, random, TPE and halving get 27 evaluations each.

Chosen (best CV): the halving winner, CV 0.9528, test 0.9455; the defaults drop as much (0.9468 → 0.9374).

Random search: best_score_ 0.9499; nested estimate 0.9472 ± 0.0067.

Run !pip install optuna first. Lab notebook (Jupyter or Colab, File → Upload notebook).

Summary

1. Tuning is an expensive, noisy black box: the budget is the number of calls.
2. Design the space: log scales and conditional knobs.
3. Random beats grid when only a few knobs matter.
4. Bayesian optimization and TPE learn where to look; their assumptions matter.
5. Halving and Hyperband stop bad configurations early, if the early ranking is right.
6. A search optimizes your leaks and its own luck: grade it on untouched data.