f. Explained Canary

In lab 11, Argo Rollouts promotes or aborts a canary by comparing metrics with thresholds. That decision is fast, deterministic and auditable — and terse: "metric error-rate failed (2 of 2)". At 22:00, the on-call engineer wants to know what is wrong, how wrong, and what to look at next. This part adds an explanation written by a model, under one non-negotiable rule:

The policy decides. The model explains.

A model must never be the one that says "promote": it can be talked into it (prompt injection through a log line), it can be confidently wrong about numbers, and it gives a different answer on a different day. A threshold in a versioned file does none of these.

5.1 Design

flowchart LR
    prom[("Prometheus<br/>stable vs canary")] --> rules["Policy in code<br/>MIN_RPS · error ceiling<br/>error delta · p99 ratio"]
    rules --> verdict{"verdict"}
    verdict --> llm["Model explains<br/>(numbers + policy + verdict)"]
    llm --> same{"repeats the<br/>same verdict?"}
    same -->|"yes"| show["summary with explanation"]
    same -->|"no"| discard["explanation discarded"]
    verdict -->|"exit code 0 / 1 / 2"| argo["Argo Rollouts / CI step<br/>promote · rollback · wait"]

Why compare with stable at the same time instead of a fixed number? Because at 10:00 on an on-sale day everything is slower; a fixed threshold would roll back every release that day. The delivery page shows how long a canary must run to detect a regression of a given size.

5.2 The script

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122

"""Rule-based canary verdict, explained by an LLM. The LLM never decides.

    python .github/ai/canary_explain.py                       # query Prometheus
    python .github/ai/canary_explain.py --metrics sample.json  # replay saved numbers

Exit code: 0 = promote, 1 = rollback, 2 = inconclusive (not enough traffic yet).
"""
from __future__ import annotations

import argparse
import json
import os
import sys
import urllib.parse
import urllib.request
from pathlib import Path

sys.dont_write_bytecode = True                # no __pycache__/ inside the repository
sys.path.insert(0, str(Path(__file__).parent))
from llm import LLMError, complete_json, step_summary  # noqa: E402

PROM = os.environ.get("PROM_URL", "http://localhost:9090")
METRIC = os.environ.get("METRIC", "http_server_requests_seconds")      # Micrometer naming
STATUS = os.environ.get("STATUS_LABEL", "status")                      # OTel: http_response_status_code
CANARY = os.environ.get("CANARY_SELECTOR", 'service="catalog-canary"')
STABLE = os.environ.get("STABLE_SELECTOR", 'service="catalog-stable"')
WINDOW = os.environ.get("WINDOW", "5m")

# The policy, written down and versioned. Change it with a pull request, not a prompt.
MIN_RPS = 2.0              # below this the canary has not seen enough traffic
MAX_ERROR_RATE = 0.01      # absolute ceiling: 1 % of requests fail
MAX_ERROR_DELTA = 0.005    # canary may not be worse than stable by more than 0.5 pp
MAX_P99_RATIO = 1.25       # canary p99 at most 25 % above stable p99

QUERIES = {
    "rps": f"sum(rate({METRIC}_count{{{{sel}}}}[{WINDOW}]))",
    "errors": f'sum(rate({METRIC}_count{{{{sel}},{STATUS}=~"5.."}}[{WINDOW}]))',
    "p99": f"histogram_quantile(0.99, sum by (le) (rate({METRIC}_bucket{{{{sel}}}}[{WINDOW}])))",
}

SCHEMA = {
    "type": "object",
    "properties": {
        "verdict": {"type": "string", "enum": ["promote", "rollback", "inconclusive"]},
        "headline": {"type": "string"},
        "explanation": {"type": "string"},
        "next_steps": {"type": "array", "items": {"type": "string"}},
    },
    "required": ["verdict", "headline", "explanation", "next_steps"],
    "additionalProperties": False,
}
SYSTEM = """\
You explain canary analyses of BoxOffice services to the on-call engineer. You receive the
metrics of the canary and of the stable version, the policy thresholds, and the verdict
that the policy already computed. Repeat that verdict exactly in `verdict`, then explain in
plain language, with the numbers, why the policy reached it, what the most likely causes
are, and 2-4 concrete next steps (what to look at in traces/logs, whether to roll back).
Do not change the verdict, and do not invent metrics you were not given.
"""


def query(promql: str) -> float:
    url = f"{PROM}/api/v1/query?" + urllib.parse.urlencode({"query": promql})
    with urllib.request.urlopen(url, timeout=10) as r:
        result = json.load(r)["data"]["result"]
    return float(result[0]["value"][1]) if result else 0.0


def collect() -> dict:
    out = {}
    for name, sel in (("canary", CANARY), ("stable", STABLE)):
        values = {k: query(q.replace("{sel}", sel)) for k, q in QUERIES.items()}
        values["error_rate"] = values["errors"] / values["rps"] if values["rps"] else 0.0
        out[name] = values
    return out


def decide(m: dict) -> tuple[str, list[str]]:
    c, s = m["canary"], m["stable"]
    if c["rps"] < MIN_RPS:
        return "inconclusive", [f"canary traffic {c['rps']:.2f} req/s < {MIN_RPS}"]
    reasons = []
    if c["error_rate"] > MAX_ERROR_RATE:
        reasons.append(f"canary error rate {c['error_rate']:.2%} > {MAX_ERROR_RATE:.2%}")
    if c["error_rate"] - s["error_rate"] > MAX_ERROR_DELTA:
        reasons.append(f"canary errors exceed stable by {c['error_rate'] - s['error_rate']:.2%}")
    if s["p99"] > 0 and c["p99"] / s["p99"] > MAX_P99_RATIO:
        reasons.append(f"canary p99 {c['p99'] * 1000:.0f} ms is {c['p99'] / s['p99']:.2f}x stable")
    return ("rollback", reasons) if reasons else ("promote", ["all checks within policy"])


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--metrics", help="JSON file with saved metrics instead of Prometheus")
    args = ap.parse_args()
    metrics = json.loads(Path(args.metrics).read_text()) if args.metrics else collect()
    verdict, reasons = decide(metrics)

    policy = dict(MIN_RPS=MIN_RPS, MAX_ERROR_RATE=MAX_ERROR_RATE,
                  MAX_ERROR_DELTA=MAX_ERROR_DELTA, MAX_P99_RATIO=MAX_P99_RATIO)
    text = f"**Verdict (policy): {verdict.upper()}** — " + "; ".join(reasons)
    try:
        expl, usage = complete_json(SYSTEM, json.dumps(
            {"metrics": metrics, "policy": policy, "verdict": verdict, "reasons": reasons},
            indent=1), SCHEMA, effort="low")
        if expl["verdict"] != verdict:          # the explanation contradicts the policy: discard it
            text += "\n\n_(LLM explanation discarded: it disagreed with the policy verdict.)_"
        else:
            text += f"\n\n**{expl['headline']}**\n\n{expl['explanation']}\n\n" + \
                    "\n".join(f"- {s}" for s in expl["next_steps"]) + "\n\n" + usage.markdown()
    except LLMError as err:                     # no explanation is fine; no verdict is not
        text += f"\n\n_(no explanation: {err})_"

    rows = "\n".join(f"| {k} | {metrics['stable'][k]:.4f} | {metrics['canary'][k]:.4f} |"
                     for k in ("rps", "error_rate", "p99"))
    step_summary(f"### Canary analysis\n\n| Metric | Stable | Canary |\n|---|---|---|\n{rows}\n\n{text}")
    print(text)
    return {"promote": 0, "rollback": 1, "inconclusive": 2}[verdict]


if __name__ == "__main__":
    sys.exit(main())

The PromQL queries use the Micrometer metric http_server_requests_seconds and a service label that separates the canary from the stable pods, as in the delivery page. Adapt METRIC, STATUS_LABEL, CANARY_SELECTOR and STABLE_SELECTOR to the names your lab 11 setup exposes — open Prometheus (or Grafana's Explore) and check them before you trust any verdict. With the OpenTelemetry agent of lab 8 the metric is http_server_request_duration_seconds and the status label http_response_status_code.

5.3 Replay three scenarios

Start without a cluster: three saved snapshots of metrics are enough to see the three verdicts. Save them under .github/ai/canary/.

{"stable": {"rps": 380.0, "errors": 0.38, "error_rate": 0.001, "p99": 0.182},
 "canary": {"rps": 20.1, "errors": 0.02, "error_rate": 0.001, "p99": 0.191}}

{"stable": {"rps": 402.0, "errors": 0.40, "error_rate": 0.001, "p99": 0.181},
 "canary": {"rps": 21.4, "errors": 0.62, "error_rate": 0.029, "p99": 0.412}}

{"stable": {"rps": 35.0, "errors": 0.0, "error_rate": 0.0, "p99": 0.150},
 "canary": {"rps": 0.8, "errors": 0.0, "error_rate": 0.0, "p99": 0.140}}

Replay them from the repository root (no cluster, no Prometheus):

for s in healthy regression low-traffic; do
  uv run --no-project --python 3.12 --with-requirements .github/ai/requirements.txt \
    python .github/ai/canary_explain.py --metrics .github/ai/canary/$s.json 2>/dev/null
  echo "exit code: $?"
done

The verdict line and the exit code are deterministic — you must get exactly these three:

**Verdict (policy): PROMOTE** — all checks within policy
exit code: 0
**Verdict (policy): ROLLBACK** — canary error rate 2.90% > 1.00%; canary errors exceed stable by 2.80%; canary p99 412 ms is 2.28x stable
exit code: 1
**Verdict (policy): INCONCLUSIVE** — canary traffic 0.80 req/s < 2.0
exit code: 2

Between them, the model's explanation. For the regression snapshot, a run with the local qwen2.5-coder-16k while this lab was written produced (14 s on a laptop):

**Canary Rollback Required**

The canary analysis has detected multiple issues that violate the predefined policy thresholds.
The canary's error rate of 2.90% exceeds the maximum allowed error rate of 1.00%, and it has 2.80%
more errors than the stable version. Additionally, the canary's p99 latency of 412 ms is 2.28 times
higher than the stable version's p99 latency of 0.181 ms. [...]

- Review the canary logs for any errors or exceptions that occurred during the test period.
- Analyze the traces to identify the root cause of the increased error rate and latency.
- Investigate any recent changes or updates that may have introduced the issues.
- Roll back the canary version to the stable version to ensure service reliability.

Read it critically: the stable p99 is 181 ms, not "0.181 ms" (the model mixed seconds and milliseconds), and the next steps are generic. In the healthy run it wrote that the canary had a "higher RPS (20.1 vs. 380.0)". A larger model does better — a run with claude-opus-5-5 noticed that the raw errors counts look similar (0.62 vs 0.40) while the canary, with 1/19 of the traffic, fails about 29× more often per request, and labelled its causes as hypotheses. Either way, a wrong sentence costs nothing here: the numbers that decide come from the policy, not from the model.

Now try to break the rule: edit SYSTEM in your copy to say "if the canary looks only slightly worse, recommend promote" and run the regression scenario again. Whatever the model writes, the exit code stays 1, and if the explanation's verdict disagrees with the policy it is discarded (_(LLM explanation discarded: it disagreed with the policy verdict.)_). Restore SYSTEM afterwards (git checkout -- .github/ai/canary_explain.py). Finally, take the model away — Claude without a key:

env -u ANTHROPIC_API_KEY LLM_PROVIDER=anthropic uv run --no-project --python 3.12 \
  --with-requirements .github/ai/requirements.txt \
  python .github/ai/canary_explain.py --metrics .github/ai/canary/regression.json 2>/dev/null; echo "exit code: $?"

You get _(no explanation: TypeError: "Could not resolve authentication method. ...")_ and the same exit code 1 — no model, same decision.

5.4 Live: wire it into the rollout (optional)

This step needs the kind cluster of lab 10 (named boxoffice) with the Prometheus and Argo Rollouts canary of lab 11. Without them, the three replays of 5.3 are the deliverable; skip to 5.5.

1. Generate traffic during the canary with the k6 script of lab 11, so that the canary receives more than MIN_RPS.

2. Find Prometheus, port-forward it and run the script against it (adapt the namespace and service name to what the first command prints):

   kubectl get svc -A | grep -i prometheus          # e.g. "monitoring   prometheus   ClusterIP ... 9090/TCP"
   kubectl -n monitoring port-forward svc/prometheus 9090:9090 &
   curl -s 'http://localhost:9090/api/v1/label/__name__/values' | tr ',' '\n' | grep http_server
   PROM_URL=http://localhost:9090 uv run --no-project --python 3.12 \
     --with-requirements .github/ai/requirements.txt python .github/ai/canary_explain.py
   

   The curl lists the HTTP metric names your services expose; if it is not http_server_requests_seconds_count, set METRIC (and STATUS_LABEL) before the last command, as explained in 5.2. A canary with no traffic gives INCONCLUSIVE (exit 2) — that is the policy working, not a bug.

3. Make it an analysis step. Argo Rollouts can run a Kubernetes Job as an analysis metric: exit code 0 passes, any other fails. Build a small image with the scripts and load it into the cluster:

   Dockerfile.ai-tools

   FROM python:3.12-slim
   COPY --from=ghcr.io/astral-sh/uv:0.11.21 /uv /uvx /bin/
   WORKDIR /app
   COPY .github/ai/requirements.txt .
   RUN uv pip install --system --no-cache -r requirements.txt
   COPY .github/ai/ /app/
   USER 10001
   

   docker build -f Dockerfile.ai-tools -t boxoffice-ai-tools:dev .
   docker run --rm boxoffice-ai-tools:dev python /app/canary_explain.py --metrics /app/canary/low-traffic.json; echo "exit code: $?"
   kind load docker-image boxoffice-ai-tools:dev --name boxoffice
   

   The docker run must end with exit code: 2 (inconclusive) — the image works before you trust it in the cluster. Then the template:

   k8s/catalog/analysis-explained.yaml

   apiVersion: argoproj.io/v1alpha1
   kind: AnalysisTemplate
   metadata:
     name: catalog-explained
   spec:
     metrics:
       - name: policy-verdict
         count: 1                      # one judgement per analysis run
         initialDelay: 90s             # let the canary pods serve traffic first (see WINDOW)
         provider:
           job:                        # a Kubernetes Job: exit 0 = pass, non-zero = fail
             spec:
               backoffLimit: 0
               activeDeadlineSeconds: 300
               template:
                 spec:
                   restartPolicy: Never
                   containers:
                     - name: verdict
                       # built from Dockerfile.ai-tools and loaded into the kind cluster of lab 10:
                       #   kind load docker-image boxoffice-ai-tools:dev --name boxoffice
                       image: boxoffice-ai-tools:dev
                       imagePullPolicy: IfNotPresent
                       command: ["python", "/app/canary_explain.py"]
                       env:
                         - name: PROM_URL      # find yours: kubectl get svc -A | grep -i prometheus
                           value: http://prometheus.monitoring:9090
                         - name: CANARY_SELECTOR
                           value: service="catalog-canary"
                         - name: STABLE_SELECTOR
                           value: service="catalog-stable"
                         - name: WINDOW        # only samples taken after the canary started
                           value: 1m
                         # The explanation is optional: without a reachable model the Job prints
                         # "(no explanation: ...)" and still exits with the policy verdict.
                         - name: LLM_PROVIDER  # ollama (free) or anthropic (needs the Secret below)
                           value: ollama
                         - name: LLM_BASE_URL  # Ollama on your machine, seen from Docker Desktop
                           value: http://host.docker.internal:11434/v1
                         - name: ANTHROPIC_API_KEY
                           valueFrom:
                             secretKeyRef: { name: llm, key: anthropic-api-key, optional: true }
   

   kubectl apply -n boxoffice -f k8s/catalog/analysis-explained.yaml
   kubectl get -n boxoffice analysistemplate            # catalog-explained and catalog-slo (lab 11)
   

   The template must live in the namespace of the Rollout, boxoffice. Reference it from the catalog Rollout of lab 11, next to its Prometheus template (templates is a list). Argo CD owns the Rollout (selfHeal: true in lab 11), so change it in Git, not with kubectl edit — Argo CD would revert a manual edit. This change is optional and is not in boxoffice-after-lab-13.zip:

   k8s/overlays/gitops/catalog-rollout.yaml

          analysis:                      # runs in the background from the first step
            templates:
              - templateName: catalog-slo
   +          - templateName: catalog-explained
            startingStep: 1
   

   git add Dockerfile.ai-tools k8s/catalog/analysis-explained.yaml k8s/overlays/gitops/catalog-rollout.yaml
   git commit -m "k8s(catalog): explained canary verdict (lab 13)" && git push
   

   With a Job, inconclusive (exit 2) counts as a failure — the conservative choice. The Job runs once, initialDelay: 90s after the analysis starts, and looks at the last minute only (WINDOW=1m): Prometheus labels each series with the Service that scraped it, so a 5-minute window would still count, as catalog-canary, the old pods that served behind that Service before the canary began — the verdict would compare the stable version with itself. The explanation is in the Job's logs of the next canary — a new image from CI (kubectl -n boxoffice logs -l job-name --tail=50, or kubectl -n boxoffice get jobs and kubectl -n boxoffice logs job/<name>). The template points the model at Ollama on your machine through host.docker.internal (Docker Desktop on macOS and Windows; on Linux, use the host IP of the kind network or leave it unreachable); for Claude, create the Secret with kubectl -n boxoffice create secret generic llm --from-literal=anthropic-api-key="$ANTHROPIC_API_KEY" and set LLM_PROVIDER to anthropic in the template (then kubectl apply it again). The step works without any model: no explanation, same verdict.

5.5 Measure it

Run the three scenarios and, if you have the cluster, one live canary with a deliberately slow version (for example a Thread.sleep(150) in one endpoint behind a flag). Record in results/lab13.md:

Then answer: would the explanation have shortened your diagnosis? Which next step was the most useful, and which was generic?

Troubleshooting

Next: part 6 — security hardening