5. Canary with Argo Rollouts

A rolling update replaces every pod with the new version as soon as it is ready — but "ready" only means the process started. A version that answers 20% of the requests with 500 is ready too. A canary sends a small share of the traffic to the new version first, measures it against the SLO, and continues only while the numbers are good; otherwise it aborts and the old version keeps serving. With Argo Rollouts the decision is automatic.

The pieces

The Prometheus of this lab is deliberately minimal (one Deployment, one scrape job):

# A minimal in-cluster Prometheus for the canary analysis (lab 11).
# It discovers the Services of namespace boxoffice annotated with prometheus.io/scrape: "true"
# and labels every series with service="<Service name>" (catalog-stable, catalog-canary, ...).
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
  - apiGroups: [""]
    resources: ["services", "pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["discovery.k8s.io"]
    resources: ["endpointslices"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: monitoring
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
    scrape_configs:
      - job_name: boxoffice
        metrics_path: /actuator/prometheus
        kubernetes_sd_configs:
          - role: endpointslice
            namespaces:
              names: [boxoffice]
        relabel_configs:
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
            regex: "true"
            action: keep
          - source_labels: [__meta_kubernetes_endpointslice_port_name]
            regex: http
            action: keep
          - source_labels: [__meta_kubernetes_service_name]
            target_label: service
          - source_labels: [__meta_kubernetes_pod_name]
            target_label: pod
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus
      containers:
        - name: prometheus
          image: prom/prometheus:v3.13.4
          args:
            - --config.file=/etc/prometheus/prometheus.yml
            - --storage.tsdb.retention.time=2d
          ports:
            - name: http
              containerPort: 9090
          readinessProbe:
            httpGet: { path: /-/ready, port: http }
          resources:
            requests: { cpu: 100m, memory: 256Mi }
            limits: { cpu: "1", memory: 1Gi }
          volumeMounts:
            - name: config
              mountPath: /etc/prometheus
      volumes:
        - name: config
          configMap:
            name: prometheus-config
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring
spec:
  selector:
    app: prometheus
  ports:
    - name: http
      port: 9090
      targetPort: http

The metric is Spring Boot's own (Micrometer) http_server_requests_seconds, exposed at /actuator/prometheus by the dependency added in step 1 and the overlay's MANAGEMENT_ENDPOINTS_WEB_EXPOSURE_INCLUDE patch. Micrometer publishes only _count and _sum by default; the overlay's MANAGEMENT_METRICS_DISTRIBUTION_PERCENTILESHISTOGRAM_HTTP_SERVER_REQUESTS patch adds the _bucket series, without which the p99 query returns nothing and that metric would always pass. With the service label, the same query compares service="catalog-canary" with service="catalog-stable".

The steps of the Rollout (see catalog-rollout.yaml in step 4): 25% → wait 2 min → 50% → wait 2 min → 100%, with the analysis running in the background from the first step. Without a traffic router, Argo Rollouts reaches a weight by replica counts (with 2 replicas, "25%" is 1 canary pod next to the stable ones), so the real share is approximate; the delivery class shows exact weights with the Gateway API plugin.

Watch a rollout

Install the kubectl plugin of Argo Rollouts:

brew install argoproj/tap/kubectl-argo-rollouts

curl -fsSLo kubectl-argo-rollouts https://github.com/argoproj/argo-rollouts/releases/download/v1.10.1/kubectl-argo-rollouts-linux-amd64
chmod +x kubectl-argo-rollouts && sudo mv kubectl-argo-rollouts /usr/local/bin/

Keep load on the service while a new version rolls out — the analysis needs requests to measure — using the k6 script of lab 10 at a modest rate:

k6 run -e PEAK=40 load/catalog-hpa.js &          # about 11 minutes of traffic through the Ingress
kubectl argo rollouts -n boxoffice get rollout catalog --watch

Any commit to services/catalog on main produces a new image and a new rollout; a quick one is a comment in ChaosFilter.java. You should see the steps advance (SetWeight: 25 → 50 → 100), an AnalysisRun with ✔ measurements, and at the end the new ReplicaSet marked stable.

A bad release

Now ship a release that breaks 30% of the requests, through Git. In k8s/overlays/gitops/kustomization.yaml, add one more operation to the Deployment patch (after the MANAGEMENT_METRICS_DISTRIBUTION_PERCENTILESHISTOGRAM_HTTP_SERVER_REQUESTS one), with its - op: exactly under the other - op: lines of that patch. The Deploy commits have reformatted the file (4 spaces, as below); a different indentation breaks the patch, and Argo CD reports a ComparisonError instead of starting a canary:

    - op: add                      # lab 11 experiment: a bad release (remove it afterwards)
      path: /spec/template/spec/containers/0/env/-
      value: { name: CHAOS_ERROR_RATE, value: "0.3" }

CHAOS_ERROR_RATE is read by ChaosFilter, which exists only for this experiment:

package boxoffice.catalog;

import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.util.concurrent.ThreadLocalRandom;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;

/** Lab 11 only: CHAOS_ERROR_RATE=0.2 makes 20% of the API requests fail, to play a bad release. */
@Component
public class ChaosFilter extends OncePerRequestFilter {

    private final double errorRate;

    public ChaosFilter(@Value("${CHAOS_ERROR_RATE:0}") double errorRate) {
        this.errorRate = errorRate;
    }

    @Override
    protected boolean shouldNotFilter(HttpServletRequest request) {
        return errorRate <= 0 || request.getRequestURI().startsWith("/actuator");
    }

    @Override
    protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain chain)
            throws ServletException, IOException {
        if (ThreadLocalRandom.current().nextDouble() < errorRate) {
            response.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR, "injected failure");
            return;
        }
        chain.doFilter(request, response);
    }
}

Commit, push, and watch (keep the k6 load running):

git commit -am "experiment: bad release" && git pull --rebase && git push
kubectl argo rollouts -n boxoffice get rollout catalog --watch

You should see, about a minute after Argo CD syncs, the rollout aborted and the canary scaled down. From a test run:

Name:            catalog
Namespace:       boxoffice
Status:          ✖ Degraded
Message:         RolloutAborted: Rollout aborted update to revision 2: Background analysis phase error/failed: Metric "error-rate" assessed Failed due to failed (2) > failureLimit (1)
Strategy:        Canary
  Step:          0/4
  SetWeight:     0
  ActualWeight:  0
Images:          boxoffice/catalog:v1 (stable)

(That test run used a locally loaded boxoffice/catalog:v1; yours shows ghcr.io/<you>/boxoffice-catalog:sha-… as the stable image.)

and the measurements of the error-rate metric were [] (no canary traffic yet: counted as a success by len(result) == 0), 0.068 and 0.285 — two failures above failureLimit: 1. The fans who hit the canary saw errors for about a minute; everyone else never noticed.

Degraded is the signal to fix forward in Git: revert the experiment (git revert HEAD && git push), Argo CD syncs, and the next rollout starts from a clean state.

Next: Measure it