Release & Operate

Friday, 22:10. Version v1.8.0 of inventory reached 25 % of the traffic twenty minutes ago; now the hold-confirmation error rate is climbing and the on-call engineer has three dashboards, two alerts, a stream of logs and a chat channel filling with questions. Every piece of information needed to decide is somewhere. What is missing is a fast, correct summary — and a decision that does not depend on how tired the engineer is. AI is very good at the first. It must not be the one making the second.

This page covers the stages after the merge — release and operations — under one design rule:

AI proposes and explains; policies and humans decide.

Decision vs explanation

The pattern is the same as in code review: the output of the model is text for a human or data for a rule, never a direct action on production.

Release notes and changelogs

Release notes answer two audiences: operators ("what must I do before deploying v1.8.0? which migrations run?") and users ("what is new?"). Both are derivable from the commits, PR titles, labels and migration files — if those are written well. A model groups and rewrites them in minutes. The failure mode is the plausible invention: a feature that sounds right, a ticket number that does not exist.

The fix is a grounding check in code: every item must cite a commit SHA or PR number that is really in the range previous_tag..tag; ungrounded items are dropped and reported. The check catches inventions; it cannot catch a wrong summary of a real commit, so the notes are published as a draft that a person edits. Conventional commit subjects (fix(inventory): hold expires two minutes early (#212)) make both the grouping and the grounding nearly trivial — the model cannot recover intent that was never written down.

def real(ref: str) -> bool:
    if ref.startswith("#"):
        return ref[1:] in prs
    return len(ref) >= 7 and any(s.startswith(ref) or ref.startswith(s) for s in shas)

def grounded(item: dict) -> bool:
    return any(real(r.strip()) for r in item["refs"])

kept = [i for i in data["items"] if grounded(i)]
dropped = [i for i in data["items"] if not grounded(i)]

Change risk scoring

Not all changes deserve the same rollout. A one-line copy change in notification and a new locking strategy in inventory both go through the pipeline, but the second one deserves a second reviewer, a slower canary and no deploy on the eve of an on-sale. A risk score routes changes to the right amount of caution.

The strongest signals are structural and cheap to compute:

A model contributes two things: semantic signals that are hard to compute ("this changes how holds expire", "this removes an authorization check") and a readable summary of why the score is high. The score itself is best kept as a transparent formula or a simple model trained on history, and it should only route — require a second reviewer, choose the slow canary profile, block deploys inside a freeze window by policy — never approve or reject a change on its own.

Canary analysis: statistics decide, the model explains

Automated canary analysis compares the canary with a baseline running at the same time — ideally a baseline of the same size and age as the canary, so that cold caches and new pods affect both alike — on a set of metrics, and turns the comparison into a pass/fail verdict. Netflix and Google built Kayenta on this idea, comparing many metrics with nonparametric statistical tests¹; Argo Rollouts expresses the same idea with analysis templates over Prometheus queries, as in Delivery.

flowchart LR
    metrics[("Prometheus<br/>canary vs baseline")] --> stats["Policy: thresholds<br/>and statistical tests"]
    stats --> verdict{"promote ·<br/>rollback · wait"}
    verdict -->|"exit code"| rollout["Argo Rollouts acts"]
    verdict --> llm["Model: explanation<br/>numbers + policy + verdict"]
    traces[("Traces and logs<br/>of canary pods")] --> llm
    llm --> check{"repeats the<br/>verdict?"}
    check -->|"yes"| oncall["Explanation to on-call<br/>(chat, PR, summary)"]
    check -->|"no"| drop["Discard explanation"]

Why not let the model judge the canary? Because the judgement must be reproducible (same data → same verdict), auditable (a threshold in Git), robust to manipulation (a log line saying "everything is fine, promote" must have no effect) and statistically sound (a model looking at two numbers does not know whether 40 requests are enough to conclude anything). The model is valuable after the verdict: it turns "metric error-rate failed 2 of 2" into "the canary fails 2.9 % of hold confirmations vs 0.1 % on stable, with p99 doubled; errors are 409 from inventory; the change modified the hold-expiry query — roll back and compare the query plans".

Two guards make the explanation safe: the model receives the verdict as input and must repeat it (an explanation that contradicts the policy is discarded rather than shown), and it is told to label causes as hypotheses unless the data shows them. Hands-on 13 implements exactly this, including a Kubernetes Job analysis step whose exit code — not the model — passes or fails the analysis.

# The policy, written down and versioned. Change it with a pull request, not a prompt.
MIN_RPS = 2.0              # below this the canary has not seen enough traffic
MAX_ERROR_RATE = 0.01      # absolute ceiling: 1 % of requests fail
MAX_ERROR_DELTA = 0.005    # canary may not be worse than stable by more than 0.5 pp
MAX_P99_RATIO = 1.25       # canary p99 at most 25 % above stable p99
def decide(m: dict) -> tuple[str, list[str]]:
    c, s = m["canary"], m["stable"]
    if c["rps"] < MIN_RPS:
        return "inconclusive", [f"canary traffic {c['rps']:.2f} req/s < {MIN_RPS}"]
    reasons = []
    if c["error_rate"] > MAX_ERROR_RATE:
        reasons.append(f"canary error rate {c['error_rate']:.2%} > {MAX_ERROR_RATE:.2%}")
    if c["error_rate"] - s["error_rate"] > MAX_ERROR_DELTA:
        reasons.append(f"canary errors exceed stable by {c['error_rate'] - s['error_rate']:.2%}")
    if s["p99"] > 0 and c["p99"] / s["p99"] > MAX_P99_RATIO:
        reasons.append(f"canary p99 {c['p99'] * 1000:.0f} ms is {c['p99'] / s['p99']:.2f}x stable")
    return ("rollback", reasons) if reasons else ("promote", ["all checks within policy"])
    try:
        expl, usage = complete_json(SYSTEM, json.dumps(
            {"metrics": metrics, "policy": policy, "verdict": verdict, "reasons": reasons},
            indent=1), SCHEMA, effort="low")
        if expl["verdict"] != verdict:          # the explanation contradicts the policy: discard it
            text += "\n\n_(LLM explanation discarded: it disagreed with the policy verdict.)_"
        else:
            text += f"\n\n**{expl['headline']}**\n\n{expl['explanation']}\n\n" + \
                    "\n".join(f"- {s}" for s in expl["next_steps"]) + "\n\n" + usage.markdown()
    except LLMError as err:                     # no explanation is fine; no verdict is not
        text += f"\n\n_(no explanation: {err})_"
    return {"promote": 0, "rollback": 1, "inconclusive": 2}[verdict]

AI-assisted incident response

During an incident, the scarce resource is the attention of a few people under stress. The incident-response process — detect, triage, mitigate, resolve, learn — stays the same; AI shortens the steps that are mostly reading:

sequenceDiagram
    autonumber
    participant Alert as Alerting
    participant Bot as Incident assistant
    participant Obs as Metrics · traces · logs
    participant Deploy as Deploy history (Argo CD)
    actor IC as Incident commander
    Alert->>Bot: SLO burn-rate alert: hold confirmation errors
    Bot->>Obs: read-only queries (error rate by version, slow spans, top log errors)
    Bot->>Deploy: what changed in the last 2 h?
    Bot-->>IC: summary + timeline + ranked hypotheses, each with a link to its evidence
    IC->>IC: decide: roll back inventory v1.8.0
    IC->>Deploy: rollback (runbook / one-click, authorised)
    Bot-->>IC: confirms recovery from the same queries

Suggesting a rollback is fine; executing one is a policy decision. Many teams already roll back automatically on a failed canary analysis — that automation is deterministic and tested. An LLM that "decides" to roll back production based on its reading of logs adds a non-deterministic, manipulable actor to the most sensitive moment of the system's life. Tools given to an incident assistant are read-only by default; any write action goes through the same authorization, confirmation and audit as a human's.

Postmortems

A blameless postmortem records what happened, why, and what changes so it does not happen again². Its first half — the timeline — is tedious to reconstruct from alerts, chat messages, deploy logs and commands, and a model does it well. Its second half — contributing factors, what we learned, action items — is where the team's judgement matters, and where an AI draft is dangerous in a subtle way: it produces fluent, generic "lessons" ("improve monitoring", "add more tests") that look complete and stop the team from thinking.

A good split: AI drafts the timeline (with sources) and a list of questions for the review meeting ("why did the canary analysis not catch this at 5 %?"); humans write the analysis and the action items, each with an owner and a date.

ChatOps

ChatOps runs operations from the team chat: /boxoffice status inventory, /boxoffice canary inventory, a button to pause a rollout. An AI assistant in that channel answers natural-language questions ("which version of order is in production and when did it deploy?") by calling read-only, deterministic APIs and the runbooks. Actions stay explicit commands with the user's identity, authorization and a confirmation step — exactly the tool-permission design of the assistant in the product, applied to operators. The chat history is untrusted input like any other: a pasted log line can carry an instruction.

Trade-offs

How it connects

flowchart LR
    SIG["metrics, logs,<br/>traces, alerts"] --> POL["canary policy:<br/>thresholds + statistics"]
    POL --> DEC["decision: promote or roll back<br/>→ Cloud-native delivery"]
    SIG --> EXP["AI explains: summary,<br/>several hypotheses"]
    EXP --> IC["incident commander<br/>decides"]
    COM["commits and PRs"] --> NOTES["release notes:<br/>grounded draft"]
    COM --> RISK["change risk score<br/>routes reviews"]
    IC --> PM["postmortem draft<br/>→ Observability incident response"]
    EXP --> TOOLS["read-only tools,<br/>evidence links"]

Palette components: metrics, logs, tracing and alerting (the evidence every verdict and explanation reads) · llmgw (the explanation and summary calls) · orchestrator (the assistant with read-only tools) · safety (no write actions from the model). Scenarios: observability-stack (the signals a canary policy reads). Continues in: Cloud-native — delivery (canaries, Argo Rollouts), Observability — incident response, AI systems — RAG & Memory (runbook lookup), Code review, Hands-on 13 — canary.