Overview

BoxOffice has twelve engineers and merges about sixty pull requests a week. A PR waits, on average, most of a working day for its first human review; the on-sale of a stadium show is in three weeks. At the planning meeting someone says the obvious thing: "Let's put AI in the pipeline." Two weeks later the team has a bot that leaves fifteen comments on every PR (people stopped reading them on day three), an agent that "fixed" a red build by deleting the failing test, release notes announcing a refund feature nobody built, and an invoice nobody predicted.

None of that is a problem of the models. The same tools, designed well, cut review latency, find real bugs, write useful tests and shorten incidents. The difference is system design: where the AI step sits in the delivery lifecycle, what context it gets, what it is allowed to do, how its output is checked, what it costs, and how the team knows whether it helps. This class applies the course's toolbox — components, failure modes, numbers, trade-offs — to AI inside the software delivery pipeline.

Roadmap

1. This overview — AI across the delivery lifecycle, the autonomy spectrum, the contract of an AI step, DORA metrics and cost.
2. Code review — what AI review is good and bad at, context engineering, structured findings, noise control, gates, precision, architecture conformance.
3. Testing — generated tests and mutation testing, contract and property-based tests, load scripts from OpenAPI, flaky-test triage, test selection, synthetic data, the oracle problem.
4. Agents in CI — fix-the-build, dependency and migration agents; headless runs; sandboxing, permissions, budgets, idempotency and audit.
5. Release & operate — release notes, change risk scoring, canary analysis, incident response, postmortems, ChatOps: AI proposes, policies and humans dispose.
6. Security & governance — prompt injection in the pipeline, exfiltration, slopsquatting, SLSA, SBOM and signing, policy as code, licensing, privacy, cost governance and evaluation of AI tools.
7. Quiz — check your understanding.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab

Prerequisites from earlier classes

This class does not repeat two foundations; read them first if needed:

• Cloud-native — Delivery: CI, GitOps with Argo CD, deployment strategies, canary analysis with Argo Rollouts, and the DORA metrics. Every AI step here plugs into that pipeline.
• AI systems & agents: the LLM as a dependency (tokens, latency, cost), the LLM gateway, the agent loop, and safety and evals. Here the "users" of the model are not fans but engineers and pipelines.

AI across the delivery lifecycle

The software delivery lifecycle is a sequence of stages, each with its own inputs, outputs and failure modes. AI can help at every stage — but what "help" means, and what can go wrong, differs a lot:

flowchart LR
    plan["Plan<br/>AI drafts ADR"] --> code["Code<br/>assistant / agent"]
    code --> pr["Pull request"]
    pr --> review["AI review<br/>advisory comments"]
    pr --> tests["CI tests<br/>+ AI-generated tests<br/>judged by mutation score"]
    review --> human{"Human approval<br/>(branch protection)"}
    tests --> human
    tests -->|"red build"| agent["Fix agent<br/>draft PR"]
    agent --> human
    human --> build["Build · scan · SBOM · sign"]
    build --> release["Release notes<br/>grounded draft"]
    release --> canary["Canary<br/>policy verdict + AI explanation"]
    canary --> operate["Operate<br/>AI incident summary"]
    operate -->|"postmortem draft"| plan

Notice the shape: the AI steps branch off the main path and come back through a deterministic check or a human before anything changes in production. That is the pattern the rest of the class elaborates.

The autonomy spectrum

"AI in the pipeline" covers tools of very different autonomy. A useful scale, borrowed from how the industry talks about driving automation, is who decides the next action and who is accountable for it:

Moving up the scale buys throughput and costs control. The right level depends on the blast radius of a mistake and on how verifiable the output is: a typo fix in docs can be level 4; a change to seat locking in inventory stays at level 2–3 no matter how good the model is. Accountability never moves to the tool: the engineer who merges an AI-written change owns it.

The contract of an AI step

Every AI step in a pipeline — review, test generation, fix agent, notes, canary explanation — can be designed with the same five-part contract. Writing it down for each step is the fastest way to find what is missing:

Two properties follow from the contract. Least privilege: the permission of the action is the most the step can ever do, whatever the model says. Fail-safe direction: decide in advance whether an AI outage blocks the pipeline (fail closed) or skips the step (fail open). Advisory steps fail open; a security gate that you rely on must fail closed — and then it must be very reliable.

What happens to the DORA metrics

The DORA metrics — deployment frequency, lead time for changes, change failure rate and time to restore — are defined in Delivery. AI changes the inputs to all four, not always in the expected direction:

• Lead time can fall (faster first review, faster fixes of red builds) — or rise, if PRs get bigger because code is cheap to generate and slow to review.
• Deployment frequency follows lead time, but only if batches stay small.
• Change failure rate can rise if review attention drops ("the bot looked at it") or generated code is merged without understanding; it can fall if AI review and tests catch real defects.
• Time to restore can fall with good incident summaries and faster diagnosis; it can rise if a confident wrong explanation sends the on-call engineer down the wrong path.

The research so far is a warning against assuming the good direction. The 2024 DORA report estimated that a 25 % increase in AI adoption was associated with a 1.5 % decrease in delivery throughput and a 7.2 % decrease in delivery stability, even as individuals reported higher productivity and better code and documentation quality¹; its authors point to larger batch sizes and weaker fundamentals as plausible causes. The 2025 DORA research describes AI as an amplifier of whatever the organization already is — strong platforms, small batches and good tests get stronger; weak ones get weaker². And in a 2025 randomized controlled trial, experienced open-source developers took 19 % longer with AI tools while believing they were about 20 % faster³.

The engineering response is not to avoid AI but to measure it like any other change — with a baseline period and per-step metrics:

The numbers: what an AI step costs

An AI step has three costs: tokens (the provider's bill), runner minutes (CI time), and attention (every comment or proposal a human must read). The first two are easy to estimate; the third is usually the largest.

The table estimates the token cost of one review of a typical PR and the monthly bill for BoxOffice (12 engineers, ~250 reviews per month, counting re-reviews after pushes) and for a 500-engineer organization (~20,000 reviews per month). Assumptions, all illustrative: an 8,000-token stable prefix (instructions, conventions, ADRs), 6,000 tokens of diff and PR text, 1,500 output tokens (findings plus reasoning). Prices are the providers' list prices per million tokens when this page was written — check current pricing before using them.

What the numbers say:

• Tokens are cheap relative to people. Even with the largest model, a review costs cents; the 500-engineer organization pays in a month roughly what a few engineer-days cost. The expensive failure is not the bill, it is noise: a reviewer with 40 % precision that posts four comments on each of ~5,000 reviews a week produces about 12,000 useless comments a week — at one minute each, 200 hours — five engineers' full 40-hour weeks spent reading them.
• Output tokens are about half the bill for the large model, although they are a tenth of the tokens: each output token costs 5× an input token. Asking for "no praise, no restating the code" is also a cost optimization.
• Caching the stable prefix helps more as the prefix grows (big conventions files, many ADRs) — but only if the prefix is byte-identical across runs.
• Local models remove the token bill but not the cost: a CPU runner generating a few tokens per second makes the job minutes long, and lower recall means missed defects. Compare them on a golden set before choosing.
• Runner minutes: a review job of ~1.5 minutes on a hosted Linux runner costs on the order of one US cent at typical per-minute prices (check your CI provider) — comparable to the tokens. An agent that runs the full build three times costs far more in minutes than in tokens.

Agents behave differently: each step re-sends the growing context, so the cost grows roughly quadratically with the number of steps (AI systems — the numbers). The Agents in CI page computes it for a fix-the-build agent.

The LLM gateway is shared infrastructure

CI bots are clients of the model like the BoxOffice assistant is. At the scale of an organization they should go through the same LLM gateway: one place for keys, per-repository quotas and budgets, model routing ("small model for release notes, large model for security review"), fallbacks when a provider is down, and cost attribution per team. Without it, every repository holds its own API key, and a runaway agent loop in one repository can exhaust the organization's rate limit during the on-sale.

Try it: open the scenario below with the spike profile and imagine the spike is not fans but a mass dependency upgrade that triggers 300 PR reviews and 300 fix agents at once. The playground has one kind of client, so it shows what a shared quota does: over the run about 70% of the questions fail, against 37% at the normal load. Select the LLM Gateway and lower Provider quota from 10 to 5 calls/s: errors pass 90% — for everyone. That is why the gateway needs a quota per client (per repository, per team) with priority for interactive traffic: the bots should wait, the fans should not. The playground does not model per-client quotas; its Fallback model knob shows the other lever, capacity bought at a price (errors fall to about 3%, the cost per month doubles).

Open the full playground to change the components yourself. The gateway component is described in the component catalog.

The AI-assisted delivery concept map

Every AI step in the pipeline is a small AI system: an orchestrator around a model, tools with permissions, a policy layer, and telemetry that says whether it helps. The map links the palette components (green) that reappear here to the concepts of this class (blue):

flowchart LR
    ev["PR or pipeline event"] --> orch["orchestrator<br/>(reviewer, fix agent)"]:::comp
    orch --> llmgw["llmgw"]:::comp
    orch --> toolreg["toolreg<br/>tool allow-list"]:::comp
    orch --> safety["safety"]:::comp
    orch --> logs["logs<br/>audit trail"]:::comp
    obs["metrics · alerting"]:::comp
    orch -.- c1["autonomy levels,<br/>contract of an AI step"]:::tech
    llmgw -.- c2["per-repository quotas,<br/>routing, cost attribution"]:::tech
    toolreg -.- c3["least privilege, sandbox,<br/>draft PRs"]:::tech
    safety -.- c4["injection in PR text,<br/>slopsquatting, policy as code"]:::tech
    obs -.- c5["canary verdict by policy,<br/>AI explains"]:::tech
    obs -.- c6["DORA, reviewer precision,<br/>mutation score"]:::tech
    classDef comp fill:#1e9e5a22,stroke:#1e9e5a
    classDef tech fill:#2f6fed22,stroke:#2f6fed

Related

• Hands-on 13 — AI in CI/CD: build the reviewer, the test generator with PIT, the fix agent, release notes, the explained canary, the hardening and the measurement.
• Hands-on 11 — Delivery and Hands-on 12 — AI assistant: the pipeline and the LLM gateway this class builds on.
• Project: the K8s + CI/CD milestone may include one AI step, measured.
• Interview preparation: the question above is a common "design a developer tool" prompt.

Key takeaways

What breaks next

Every AI step of this class judges one change: one diff, one test class, one red build, one canary against its baseline at today's traffic. None of them answers the question the next on-sale asks of the whole system: with the edge, the caches, the waiting room, the breakers, Kubernetes and an assistant all in place, does BoxOffice keep p99 under 300 ms when 2 M fans arrive at 10:00, which component breaks first, and what does it cost to run? Fourteen chapters added boxes; none proved that they hold together. Proving it with load, profiles and a cost model — and fixing what the proof finds — is class 15 — Performance Engineering.