Machine Learning · Insper

AutoML

Advanced approach

Inside the smarter searches, and how far automation can go

The path of this part

Question the surrogateits length scale decides what it can find
Model the good trialsTPE densities; Optuna's define-by-run spaces
Prune and hedgestop a trial early; Hyperband's brackets
The whole pipelineCASH, AutoML systems, the rest of the job

Each stage answers one question: can the surrogate be trusted? → what if we model the good trials instead of the score? → can one trial stop early, and can the bet be hedged? → how far can automation go?

Stage 1 · can the surrogate be trusted?

The surrogate's assumptions matter

The four iterations of the fundamental part, rerun from 100 random starting triples, 30 evaluations each, with Bayesian optimization (BO) and with random search. Share of runs that reached 0.81:

	within 10	within 20	within 30
BO, length scale 0.15	40%	65%	65%
BO, length scale 0.10	61%	92%	92%
random search	37%	54%	65%

With 0.15 the surrogate is too sure the score is smooth: it predicts a flat, certain 0.70 over the narrow peak and stops looking. A wrong model makes wrong decisions confidently.

Stage 2 · what if we model the good trials instead of the score?

TPE: model \(x\) given the score

Tree-structured Parzen Estimator: split the trials into the good ones (best fraction \(\gamma\)) and the rest.
Density of the good values \(\ell(x)\), density of the others \(g(x)\): one Gaussian bump per trial.
Sample where \(\ell(x)/g(x)\) is largest (Bergstra et al., 2011).

\(x\)	0.1	0.25	0.5	0.7	0.9
\(\ell / g\)	0.03	1.24	0.45	5.73	0.09

12 random trials, 2 of the 3 good ones near 0.68: the next trial goes to 0.710, which scores 0.817. One pair of 1-D densities per knob: integers, categories and conditional knobs are easy.

Optuna: TPE against random, measured

Six hyperparameters: one matters a lot, one a little, four not at all. Best possible: 0.870. 20 seeds.

Predict. After 10 trials, who is ahead?

mean best after	10	20	40	60 trials
random	0.812	0.819	0.830	0.843
TPE	0.812	0.847	0.857	0.860

Identical at 10: TPE's first 10 trials are random. After 40 it is above what random reaches after 60.

Define-by-run: the space is code


def build(trial):
    family = trial.suggest_categorical('family', ['logistic', 'boosting'])
    if family == 'logistic':
        C = trial.suggest_float('C', 1e-3, 1e2, log=True)
        return make_pipeline(FunctionTransformer(np.log1p), StandardScaler(),
                             LogisticRegression(C=C, max_iter=2000))
    return HistGradientBoostingClassifier(
        learning_rate=trial.suggest_float('learning_rate', 0.01, 0.5, log=True), ...)

On kc1 the best is a logistic regression, C ≈ 0.03, 0.798, tied with the hand-picked booster; TPE spent 31 of its 40 trials on it. A space that leaves out the simple model can never tell you that you did not need the complicated one.

Stage 3 · can one trial stop early, and can the bet be hedged?

Pruning: stop a bad trial early

The objective reports its running mean after every fold (trial.report).
The median pruner stops a trial worse than the median of earlier trials at the same fold.

10 of 40

trials pruned

171 of 200

folds fitted: 15% of the budget saved

Pruning is a bet that a trial that starts badly ends badly: the bet successive halving makes for a whole tournament, here made one trial at a time.

Hyperband: several tournaments

How aggressive should the tournament of successive halving be? Hyperband runs brackets of different aggressiveness, each costing about the same (Li et al., 2018).

bracket	rounds (configurations × budget)	cost
s = 3	27 × 1 → 9 × 3 → 3 × 9 → 1 × 27	108
s = 2	12 × 3 → 4 × 9 → 1 × 27	99
s = 1	6 × 9 → 2 × 27	108
s = 0	4 × 27 (plain random search)	108

If the early ranking is reliable, bracket 3 finds the winner cheaply; if it misleads, bracket 0 still gives the slow learners a fair chance. 423 units for 49 configurations.

Stage 4 · how far can automation go?

Full-pipeline AutoML: the CASH problem

Combined Algorithm Selection and Hyperparameter optimization (Thornton et al., 2013).

system	how it searches	what it returns
auto-sklearn	Bayesian optimization, warm-started on similar data sets	an ensemble (ensemble selection)
TPOT	genetic programming over pipeline trees	the best pipeline, as scikit-learn code
AutoGluon-Tabular	little search: bagging and multi-layer stacking	a stacked ensemble
FLAML	cost-aware: cheap configurations first	the best single model in a time budget

Two recurring ideas: ensembles of good, different models, and warm starts. You still write the label, the metric, the time limit, and before all that, the train and test tables.

The rest of the job

framing the problem

what is predicted, for whom, when, and what an error costs

choosing the metric

84.5% of kc1 modules are clean: a search on accuracy approves a model that never flags

the validation design

groups, time order and leaks are exploited by the search, not corrected

data and features

removing one leaky column changed average precision by 0.18; tuning, by 0.007

And deployment: a stacked ensemble of 40 models may be the hardest to serve, explain and retrain.

A practical default

Lock a test set away and fix one cv.
Baselines first: a dummy, a linear model, a booster with defaults.
Random search or Optuna, 30–100 trials, log scales, the simple model in the space.
For large data or slow models: halving or pruning, with a budget unit that does not interact with the knobs.
Stop when the best score stops moving by more than its fold standard deviation.
Report the locked test score, or a nested CV of the whole search, never the search's own best score.

Advanced lab · about 35 min, in pairs

Four searches, one budget: steps 6 to 10

phoneme, continued: TPE with Optuna and successive halving join the grid and random search of steps 4 and 5, with the same budget.

Chosen by its cross-validation (CV) score: the halving winner, CV 0.9528, test area under the ROC curve (ROC AUC) 0.9455; the defaults drop as much (0.9468 → 0.9374).

Random search: best_score_ 0.9499; nested estimate 0.9472 ± 0.0067.

Same lab notebook, from step 6 (!pip install optuna). Simulations: trees as the budget (step 7) and the defaults on the same outer folds (step 10).

Pace: steps 6, 7 and the first simulation in 15 min; 8 and 9 in 10 min; step 10 and the second in 10 min.