Skip to content

5. Canary with Argo Rollouts

A rolling update replaces every pod with the new version as soon as it is ready β€” but "ready" only means the process started. A version that answers 20% of the requests with 500 is ready too. A canary sends a small share of the traffic to the new version first, measures it against the service level objective (SLO), and continues only while the numbers are good; otherwise it aborts and the old version keeps serving. With Argo Rollouts the decision is automatic.

The pieces

Object File Role
Rollout catalog catalog-rollout.yaml replaces the Deployment's controller; workloadRef reuses the pod template of base/catalog.yaml
Service catalog-stable / catalog-canary catalog-services.yaml Argo Rollouts points each at its version's pods (it adds the pod-template hash to the selectors)
AnalysisTemplate catalog-slo catalog-analysis.yaml every 30 s, two PromQL queries on the canary: error ratio < 1%, p99 < 500 ms
Prometheus k8s/monitoring/prometheus.yaml scrapes the annotated Services and labels every series with service="<Service name>"
HorizontalPodAutoscaler (HPA) patched by the overlay now scales the Rollout

The Prometheus of this lab is deliberately minimal (one Deployment, one scrape job):

k8s/monitoring/prometheus.yaml
# A minimal in-cluster Prometheus for the canary analysis (lab 11).
# It discovers the Services of namespace boxoffice annotated with prometheus.io/scrape: "true"
# and labels every series with service="<Service name>" (catalog-stable, catalog-canary, ...).
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
  - apiGroups: [""]
    resources: ["services", "pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["discovery.k8s.io"]
    resources: ["endpointslices"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: monitoring
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
    scrape_configs:
      - job_name: boxoffice
        metrics_path: /actuator/prometheus
        kubernetes_sd_configs:
          - role: endpointslice
            namespaces:
              names: [boxoffice]
        relabel_configs:
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
            regex: "true"
            action: keep
          - source_labels: [__meta_kubernetes_endpointslice_port_name]
            regex: http
            action: keep
          - source_labels: [__meta_kubernetes_service_name]
            target_label: service
          - source_labels: [__meta_kubernetes_pod_name]
            target_label: pod
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus
      containers:
        - name: prometheus
          image: prom/prometheus:v3.13.4
          args:
            - --config.file=/etc/prometheus/prometheus.yml
            - --storage.tsdb.retention.time=2d
          ports:
            - name: http
              containerPort: 9090
          readinessProbe:
            httpGet: { path: /-/ready, port: http }
          resources:
            requests: { cpu: 100m, memory: 256Mi }
            limits: { cpu: "1", memory: 1Gi }
          volumeMounts:
            - name: config
              mountPath: /etc/prometheus
      volumes:
        - name: config
          configMap:
            name: prometheus-config
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring
spec:
  selector:
    app: prometheus
  ports:
    - name: http
      port: 9090
      targetPort: http

The metric is Spring Boot's own (Micrometer) http_server_requests_seconds, exposed at /actuator/prometheus by the dependency added in step 1 and the overlay's MANAGEMENT_ENDPOINTS_WEB_EXPOSURE_INCLUDE patch. Micrometer publishes only _count and _sum by default; the overlay's MANAGEMENT_METRICS_DISTRIBUTION_PERCENTILESHISTOGRAM_HTTP_SERVER_REQUESTS patch adds the _bucket series, without which the p99 query returns nothing and that metric would always pass. With the service label, the same query compares service="catalog-canary" with service="catalog-stable".

The steps of the Rollout (see catalog-rollout.yaml in step 4): 25% β†’ wait 2 min β†’ 50% β†’ wait 2 min β†’ 100%, with the analysis running in the background from the first step. Without a traffic router, Argo Rollouts reaches a weight by replica counts (with 2 replicas, "25%" is 1 canary pod next to the stable ones), so the real share is approximate; the delivery class shows exact weights with the Gateway API plugin.

flowchart LR
    K6["k6 / fans"] --> ING["Traefik Ingress"] --> SVC["Service catalog<br/>selects every catalog pod"]
    SVC --> S1["stable pod"]:::comp
    SVC --> S2["stable pod"]:::comp
    SVC --> C1["canary pod<br/>about a third of the requests"]
    CS["Service catalog-canary<br/>+ the pod-template hash"] -.-> C1
    PR["Prometheus<br/>scrapes every 15 s"]:::tech -->|"labels service=catalog-canary"| CS
    AN["AnalysisRun<br/>every 30 s"]:::tech -->|"PromQL"| PR

Play the life of one release β€” the two ways it can end:

stateDiagram-v2
    [*] --> SetWeight25 : Argo CD syncs a new image
    SetWeight25 --> Pause1 : the canary pod is ready
    Pause1 --> SetWeight50 : 2 min
    SetWeight50 --> Pause2 : stable scaled to 1 pod
    Pause2 --> Healthy : 2 min, the canary becomes stable
    Pause1 --> Degraded : second failed measurement
    Pause2 --> Degraded : second failed measurement
    Degraded --> [*] : canary scaled to 0
    Healthy --> [*]

Watch a rollout

Install the kubectl plugin of Argo Rollouts:

brew install argoproj/tap/kubectl-argo-rollouts
curl -fsSLo kubectl-argo-rollouts https://github.com/argoproj/argo-rollouts/releases/download/v1.10.1/kubectl-argo-rollouts-linux-amd64
chmod +x kubectl-argo-rollouts && sudo mv kubectl-argo-rollouts /usr/local/bin/

Keep load on the service while a new version rolls out β€” the analysis needs requests to measure β€” using the k6 script of lab 10 at a modest rate:

k6 run -e PEAK=40 load/catalog-hpa.js &          # about 11 minutes of traffic through the Ingress
kubectl argo rollouts -n boxoffice get rollout catalog --watch

Any commit to services/catalog on main produces a new image and a new rollout; a quick one is a comment in ChaosFilter.java. You should see the steps advance (SetWeight: 25 β†’ 50 β†’ 100), an AnalysisRun with βœ” measurements, and at the end the new ReplicaSet marked stable.

A bad release

Now ship a release that breaks 30% of the requests, through Git. In k8s/overlays/gitops/kustomization.yaml, add one more operation to the Deployment patch (after the MANAGEMENT_METRICS_DISTRIBUTION_PERCENTILESHISTOGRAM_HTTP_SERVER_REQUESTS one), with its - op: exactly under the other - op: lines of that patch. The Deploy commits have reformatted the file (4 spaces, as below); a different indentation breaks the patch, and Argo CD reports a ComparisonError instead of starting a canary:

    - op: add                      # lab 11 experiment: a bad release (remove it afterwards)
      path: /spec/template/spec/containers/0/env/-
      value: { name: CHAOS_ERROR_RATE, value: "0.3" }

CHAOS_ERROR_RATE is read by ChaosFilter, which exists only for this experiment:

services/catalog/src/main/java/boxoffice/catalog/ChaosFilter.java
package boxoffice.catalog;

import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.util.concurrent.ThreadLocalRandom;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;

/** Lab 11 only: CHAOS_ERROR_RATE=0.2 makes 20% of the API requests fail, to play a bad release. */
@Component
public class ChaosFilter extends OncePerRequestFilter {

    private final double errorRate;

    public ChaosFilter(@Value("${CHAOS_ERROR_RATE:0}") double errorRate) {
        this.errorRate = errorRate;
    }

    @Override
    protected boolean shouldNotFilter(HttpServletRequest request) {
        return errorRate <= 0 || request.getRequestURI().startsWith("/actuator");
    }

    @Override
    protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain chain)
            throws ServletException, IOException {
        if (ThreadLocalRandom.current().nextDouble() < errorRate) {
            response.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR, "injected failure");
            return;
        }
        chain.doFilter(request, response);
    }
}

Commit, push, and watch (keep the k6 load running):

git commit -am "experiment: bad release" && git pull --rebase && git push
kubectl argo rollouts -n boxoffice get rollout catalog --watch

You should see, about a minute after Argo CD syncs, the rollout aborted and the canary scaled down. From a test run:

Name:            catalog
Namespace:       boxoffice
Status:          βœ– Degraded
Message:         RolloutAborted: Rollout aborted update to revision 2: Background analysis phase error/failed: Metric "error-rate" assessed Failed due to failed (2) > failureLimit (1)
Strategy:        Canary
  Step:          0/4
  SetWeight:     0
  ActualWeight:  0
Images:          boxoffice/catalog:v1 (stable)

(That test run used a locally loaded boxoffice/catalog:v1; yours shows ghcr.io/<you>/boxoffice-catalog:sha-… as the stable image.)

and the measurements of the error-rate metric were [] (no canary traffic yet: counted as a success by len(result) == 0), 0.068 and 0.285 β€” two failures above failureLimit: 1. The fans who hit the canary saw errors for about a minute; everyone else never noticed.

Try it

The simulator replays the rollout from the moment the canary pod is ready, with the files of this lab. Like the real query, it counts the kubelet's probes and Prometheus's scrapes of the canary pod too, and a series needs two scrapes inside the 1-minute window.

  • As it opens β€” CHAOS_ERROR_RATE 0.3, k6 at 20 req/s β€” the measurements are [], 0.258 and 0.284, and the rollout aborts 60 s after the canary got traffic, at the third measurement, as in the test run ([], 0.068, 0.285; the model leaves out a new pod's first seconds). About 120 requests failed, 10% of those sent in that minute.
  • 0: every result is [] β€” with no errors there is no 500 series β€” and v2 becomes stable after the two pauses, at 240 s.
  • 0.02: the first errors take longer to show up as a series, and the rollout aborts at 90 s. 0.005: under the 1% threshold every time, so v2 is promoted and fails 0.5% of everyone's requests.
  • 0 req/s with 0.3: every result is [], and len(result) == 0 promotes the bad release. + Thread.sleep(400): a p99 of 0.409 s passes the canary's 0.5 s; only the performance gate of step 3 stops that regression.

Degraded is the signal to fix forward in Git: revert the experiment (git revert HEAD && git push), Argo CD syncs, and the next rollout starts from a clean state.

What a canary cannot catch

  • Database migrations run once, for both versions: they must be backward compatible (expand/contract), because the old version keeps running during β€” and after an aborted β€” canary.
  • Rare errors need many canary requests to be visible; at low traffic, lengthen the pauses.
  • Side effects (payments, e-mails) of canary requests are real.
  • No traffic, no evidence: with no canary requests every result is [], which len(result) == 0 counts as a pass β€” keep load on the service for the whole rollout.

Next: Measure it