AutoML

Grid search works, but it is brute force: cost multiplies per hyperparameter, most of the grid is wasted in bad regions, and someone still has to choose the grid. AutoML automates the search — over hyperparameters, and at the ambitious end, over the entire pipeline. The goal is not replacing the practitioner; it is spending your judgment on framing, data, and validation design while the machine grinds the search space.

Beyond grid: random search

Bergstra & Bengio (2012) made a simple, devastating observation: when only a few hyperparameters really matter (the usual case), a grid wastes evaluations — it retests the same few values of the important parameter while marching through irrelevant ones. Random sampling covers each dimension's range far better at equal budget:

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform, randint

param_dist = {
    'model__learning_rate': loguniform(1e-3, 0.3),   # continuous, log-scaled
    'model__max_depth': randint(2, 10),
    'model__subsample': (0.6, 0.8, 1.0),
}
search = RandomizedSearchCV(pipe, param_dist, n_iter=50, cv=5,
                            scoring='roc_auc', n_jobs=-1, random_state=0)

Random search is the honest baseline every smarter method must beat.

Bayesian optimization: search that learns

Each CV evaluation is expensive — why ignore what previous trials revealed? Bayesian optimization treats the score as an unknown function of the hyperparameters and loops:

1. fit a cheap surrogate model to (configuration → score) pairs seen so far;
2. choose the next configuration by an acquisition function balancing exploitation (near known good points) and exploration (uncertain regions);
3. evaluate, add to history, repeat.

Optuna (Akiba et al., 2019) is the ecosystem favorite (TPE surrogate by default), with a definitive API — you write the search space as a function:

import optuna
from sklearn.model_selection import cross_val_score

def objective(trial):
    params = {
        'learning_rate': trial.suggest_float('learning_rate', 1e-3, 0.3, log=True),
        'max_depth': trial.suggest_int('max_depth', 2, 10),
        'subsample': trial.suggest_float('subsample', 0.5, 1.0),
        'reg_lambda': trial.suggest_float('reg_lambda', 1e-3, 10, log=True),
    }
    model = xgb.XGBClassifier(**params, n_estimators=300)
    return cross_val_score(model, X_train, y_train, cv=5, scoring='roc_auc').mean()

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)
study.best_params

Because the space is defined in code, it can be conditional (tune gamma only if kernel == 'rbf') — impossible to express as a grid.

Successive halving: spend less on losers

Orthogonal trick — most configurations reveal their mediocrity early (few boosting rounds, small data subsets). Successive halving exploits this: start many configurations on a small budget, keep the best fraction, multiply their budget, repeat — a tournament where finalists get full training. Hyperband wraps it with multiple aggressiveness levels; Optuna's pruners do the same by killing bad trials mid-CV.

from sklearn.experimental import enable_halving_search_cv  # noqa
from sklearn.model_selection import HalvingRandomSearchCV

search = HalvingRandomSearchCV(pipe, param_dist, factor=3,
                               resource='model__n_estimators',
                               max_resources=1000, cv=5)

Full-pipeline AutoML

The widest scope: search over imputation × encoding × scaling × feature selection × model family × hyperparameters × ensembling jointly. Systems: auto-sklearn (Bayesian optimization + meta-learning warm starts + auto-ensembling), TPOT (evolves pipelines by genetic programming), FLAML (cost-aware, frugal), AutoGluon (multi-layer stacking of many models; frequently the strongest tabular results), plus commercial platforms.

from autogluon.tabular import TabularPredictor
predictor = TabularPredictor(label='churn', eval_metric='roc_auc') \
                .fit(train_df, presets='best_quality', time_limit=3600)

What AutoML cannot do

The search optimizes exactly what you tell it to. Garbage validation in, garbage model out — at scale. Still yours:

• problem framing and the choice of metric (optimize F1 vs recall@precision = different products);
• the validation design — leakage, group structure, time ordering: AutoML will happily exploit any leak harder than a human would;
• data quality and features — an hour of feature engineering routinely beats a day of hyperparameter search;
• fairness, explainability, and the decision to ship.

Quiz