MLOps

This lesson starts where the others ended

• Every lesson so far ended when the test score was printed.
• Value comes from predictions delivered reliably to the people who act on them.
• MLOps (machine-learning operations): the engineering habits that take a fitted pipeline into production and keep it honest there.

The model is the small box

Sculley et al. (2015), Hidden Technical Debt in Machine Learning Systems: the code that learns is a small box surrounded by infrastructure, and most failures happen in the infrastructure.

The rest load, check, package, serve, monitor and compare.

The loop, and what the monitor sends back

Click the buttons: what does a drift alert send back, and what does a data bug?

Every arrow can break without an error

The path of this lesson

Packaging: the artifact is the whole pipeline

German credit (1,000 past loans): 3 numeric and 5 categorical columns → scaler + one-hot → logistic regression.

joblib.dump(pipe, path)                  # the WHOLE pipeline, one file
loaded = joblib.load(path)
np.array_equal(proba, loaded.predict_proba(X_test)[:, 1])
  

Why not ship the model alone?

ValueError: X has 8 features, but LogisticRegression is expecting 30 features as input.
  

• The model saw the 30 columns of the column transformer: 3 standardized numbers + 27 one-hot indicators.
• Serving it alone means rebuilding those 30 columns by hand: same means, same category order, same unknowns. Any difference gives wrong predictions without an error.

A pickle is a program

class Innocent:
    def __reduce__(self):                 # tells pickle how to rebuild this object...
        return (print, ('this line ran while the file was being LOADED',))

obj = pickle.loads(pickle.dumps(Innocent()))   # ...and loading obeys
  

this line ran while the file was being LOADED
  

Pin the versions

scikit-learn stamps its version in every saved estimator. Fake a file from another version by changing one digit of the stamp:

InconsistentVersionWarning for: ['StandardScaler', 'OneHotEncoder',
  'ColumnTransformer', 'LogisticRegression', 'Pipeline']
  

Every estimator complains, and the file still loads. It may predict differently.

• store the versions (Python, scikit-learn, NumPy) next to the model, in a metadata file;
• pin them in the serving environment (requirements.txt, a container image);
• when upgrading the library, retrain and re-validate.

A safer format: skops

On loading, skops rebuilds only types from an allow-list; anything else must be trusted explicitly.

Outside Python, ONNX (skl2onnx) turns a fitted pipeline into a portable graph for C++, Java or JavaScript runtimes.

Serving: how the model is called

The pipeline accepts garbage

Only one. The other three get a probability that looks exactly as trustworthy as the good one.

A schema, and a function that checks it

Write down what the model saw in training (a range per number, a list per category) and check every request first:

The same function behind an HTTP endpoint

# app.py  (illustrative: run it with uvicorn, not in a notebook)
from fastapi import FastAPI
from service import predict_one           # the tested function

app = FastAPI()

@app.post('/predict')
def predict(payload: dict):
    return predict_one(payload)
  

Latency: pay per call, not per row

On the machine that built the page: 1.724 ms/row one at a time, 0.0006 ms/row in one batch, about 2,700×. A call costs about 1.7 ms whether it carries one row or a thousand: fixed work, not arithmetic.

Reproducibility: every ingredient has a version

Seeds: same score, different decisions

Two random forests that differ only in their seed:

Fingerprint the data

training data      68ad024c9661
one value changed  7eb64a1901f0
rows shuffled      a662547f0332
same data again    68ad024c9661
  

• One changed value out of 6,000 gives a completely different fingerprint; so does another row order.
• "Was it trained on this month's extract?" becomes a string comparison.

A hash only identifies the data; rebuilding needs the rows themselves (DVC, lakeFS, read-only snapshots).

An experiment log: one line per run, never edited

Monitoring: models decay silently

The service keeps answering 200 OK while its predictions get worse. Three layers: system, data quality, statistical drift.

Measuring a shift: PSI, by hand

What does PSI = 0.25 mean?

The alarm level is a move of half a standard deviation. With no shift at all, PSI is not exactly zero: two samples never have identical histograms.

Simulator: PSI, bin by bin

Predict. Turn on concept drift without moving the inputs. What happens to PSI, and to the accuracy?

Measuring a shift: the KS test

\[ D = \max_x \left| F_{\text{ref}}(x) - F_{\text{cur}}(x) \right| \]

By hand, 1–5 against 3–7: \(D = 0.4\), \(p = 0.873\): not surprising with five points per side.

The three shifts, measured

Labels arrive late: monitoring at two speeds

Degrade, detect, retrain

The population moves 0.3 sd per month. Static model vs a model retrained whenever its input PSI crosses 0.25.

month	static	monitored	best possible
0	0.801	0.801	0.802
2 (alert)	0.780	0.780	0.785
6	0.759	0.814	0.830
8	0.654	0.822	0.816
12	0.273	0.849	0.850

Training–serving skew, measured

A new client application sends a field in different units. The test set, three ways:

Shipping a new model safely

First an evaluation gate: beat the current model on recent labeled data neither saw, by more than the noise. Then roll out in stages:

Tests, documentation, and maturity

Shipping and monitoring a bank-marketing model

45,211 telemarketing calls, May 2008 to November 2010, in the order they were made. Deployed on 1 May 2009, followed for 19 months.

Build and validate (1–3), package and guard (4–6), monitor (7–9), retrain (10), write the model card (11).

The score PSI jumps above 0.8 from the second quarter: the cheapest and most useful single monitor.

Lab notebook (Jupyter or Colab, File → Upload notebook).

Summary

1. Ship the whole pipeline, check the round trip, pin the versions; a pickle is a program.
2. Put a schema in front of the model and answer with its version.
3. Version code, data (a hash), environment, seeds and results.
4. Monitor inputs and scores at once, outcomes later; PSI for magnitudes.
5. Drift is not damage; concept drift needs labels.
6. New models pass a gate and roll out in stages, with rollback ready.