11. Delivery
Goal
Turn git push into a safe, measured deployment. Add an integration test with Testcontainers and a GitHub Actions continuous integration pipeline (CI) that tests every service, enforces a k6 performance budget, builds the image, scans it (Trivy), produces a software bill of materials (SBOM) and signed provenance, and publishes it to the GitHub Container Registry (GHCR). Then deploy by GitOps: a Deploy workflow writes the new image tag to Git, Argo CD applies it to the cluster of lab 10, and Argo Rollouts releases it as a canary that aborts by itself when Prometheus shows errors.
The flash sale is next week and the team ships ten changes a day. Each of them could break checkout: a slow query, a vulnerable library, a version that fails 30% of the requests. Nobody can review every change by hand at that pace β the pipeline must answer "is this change safe to give to 2 million fans?" automatically, and limit the damage when the answer was wrong. This lab is the hands-on side of the delivery page of the cloud-native class: it implements continuous integration with a performance budget, GitOps and a canary judged by metrics.
The story so far
Chapter 11 of the story of the course, with the supply chain of chapter 12 and the performance gate of chapter 15. Lab 10 put BoxOffice on a cluster that heals and scales by itself: in its test run, a rolling update with readiness and preStop lost 0 of about 11,900 requests. But every release still starts with kubectl on a laptop and replaces every pod with an image nobody tested, scanned or measured: a release that fails 30% of its requests would reach all 2 million fans with nothing to roll it back (class 11 β Cloud-Native), and every image carries libraries nobody reviewed (class 12 β Network & Security). This lab builds the pipeline that answers "is this change safe?" by itself β CI with Testcontainers and a k6 performance budget (class 15 β Performance Engineering), a Trivy scan, an SBOM and signed provenance, GitOps with Argo CD and a canary judged by Prometheus β and measures where the pipeline spends its time, the lead time from git push to users, whether the budget catches a regression and how fast a bad canary aborts.
Slides
Click the slides and use β β to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab
Before you start
| Time | about 6 hours (two or three meetings); most of it is waiting for pipelines |
| Accounts | a GitHub account with your boxoffice repository (lab 0). GitHub Actions and GHCR are free for public repositories and include free minutes and storage for private ones; nothing in this lab needs a paid plan or a cloud account |
| Tools | Git, Docker, kubectl, kind, Helm and k6 (labs 0 and 10); GitHub CLI (command-line interface) gh (gh --version, gh auth login); Java Development Kit (JDK) 21 + Maven only to run the tests locally (optional) |
| Cluster | the kind cluster of lab 10 with Traefik and metrics-server (recreate it with lab 10, step 1 if needed); Docker with at least 8 GB |
Every command runs from the root of your boxoffice repository, in the state lab 10 left it:
boxoffice/
βββ compose.yaml # the Compose stack of labs 1-9
βββ k8s/ # lab 10: kind.yaml, base/, overlays/local/
βββ infra/ load/ scripts/
βββ services/ # auth, catalog, gateway, inventory, notification, order, payment
Catch up
Skipped or broke lab 10? Download boxoffice-after-lab-10.zip, unzip it, cd boxoffice, push it to your GitHub repository, and continue from here.
Every file of this lab is shown in full on these pages and published under https://insper.github.io/platform/2027.1/hands-on/11/code/<path>, except those under .github/ (the site skips folders whose name starts with a dot): copy those from the pages.
Architecture
flowchart LR
dev[git push / PR] --> test["CI: test (matrix, Testcontainers)"]
test --> perf["CI: performance budget (k6)"]
perf --> img["CI: image -> Trivy -> SBOM -> push -> attest"]
img --> ghcr[(ghcr.io)]
img -->|CI succeeded on main| dep["Deploy: kustomize edit set image + commit"]
dep --> repo[(k8s/overlays/gitops in Git)]
argo[Argo CD] -->|pull| repo
argo --> ro[Rollout catalog]
ro -->|"25% -> 50% -> 100%"| pods[catalog pods]
prom[Prometheus] -->|scrape| pods
ro -->|AnalysisTemplate| prom BoxOffice components touched
| Palette component | In this lab |
|---|---|
| App Server | catalog, built, scanned, signed and released as a canary |
| Metrics | an in-cluster Prometheus feeding the canary analysis |
| Ingress | Traefik from lab 10, in front of stable and canary pods |
What you will measure
| Question | Metric | Tool |
|---|---|---|
| Where does the pipeline spend its time? | duration of each job | GitHub Actions, gh run view |
How long from git push to users? | lead time per stage | timestamps |
| Does the gate catch a performance regression? | budget p95/p99 before and after | k6 in CI |
| Does the canary stop a bad release, and how fast? | sync β abort time, canary error ratio | Argo Rollouts, Prometheus |
Prerequisites
- Labs 8 (service level objectives, or SLOs, and metrics) and 10 (the cluster and
k8s/) done. - Read the delivery page and supply-chain security; for budgets, performance budgets.
Steps
- Tests and CI β a Testcontainers integration test and the
CIworkflow. - Image and supply chain β build, Trivy report and gate, SBOM, push, signed attestations, Dependabot.
- Performance gate β a k6 budget that fails the pull request; break it on purpose.
- GitOps with Argo CD β Argo CD, the GitOps overlay, the
Deployworkflow, drift and self-healing. - Canary with Argo Rollouts β Rollout + AnalysisTemplate on Prometheus; a bad release aborts by itself; replay the analysis in a simulator.
- Measure it β stage durations, lead time, the gate and the canary in numbers.
Try it in the playground
The boxoffice-canary preset is this lab's bad release: at t = 60 s, v2 of catalog β 30% of its answers fail β replaces all 4 pods at once, at 600 req/s, until the analysis rolls it back a minute later. Ship it as a canary instead: a node "catalog v1" with 3 pods next to v2 with 1 pod, so the ingress (which splits by capacity) gives v2 a quarter of the traffic. The bug is the same; its blast radius is a quarter: errors β v2's share Γ 30% Γ the time until the rollback.
The full simulator is in the playground. The analysis itself β the measurements every 30 s and failureLimit β has its own simulator in step 5.
Measure it
The Measure it step has the results table to keep in results/lab11.md.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
test fails with Could not find a valid Docker environment (locally) | Docker not running | start Docker; on CI this never happens on ubuntu-latest |
image fails with denied: permission_denied on push | the job lacks packages: write, or the package belongs to another repository | keep the permissions block; in the package settings, give your repository Write access |
image fails with repository name must be lowercase | the owner has uppercase letters | the workflow lowercases ${GITHUB_REPOSITORY_OWNER,,}: do not replace it by a literal |
| the Trivy gate fails | a CRITICAL vulnerability with a fix exists in the image | read the table in the log; update the base image (eclipse-temurin:21-jre gets patched regularly) or the library, then push |
Deploy fails with remote rejected β¦ protected branch | branch protection forbids the bot's push | allow GitHub Actions to bypass, or open a pull request instead (see step 4) |
Argo CD shows ComparisonError β¦ authentication required | the repository secret is missing or the token expired | recreate boxoffice-repo (step 4) |
pods in ImagePullBackOff with ghcr.io/... | private package and no pull secret, or wrong owner in the image | create ghcr-pull (step 4) or make the package public; check newName |
every error-rate measurement is [], and a release you expected to abort was promoted | no traffic reaches the canary, or Prometheus has no data: an empty result passes (len(result) == 0) | keep k6 running during the whole rollout; kubectl -n monitoring port-forward svc/prometheus 9090 and query http_server_requests_seconds_count |
Best practices in this lab
- No image unless tests and the performance budget pass β
needs: [test, performance]in.github/workflows/ci.yaml; why: continuous integration. - Supply chain under control: actions pinned by commit hash (SHA), a Trivy gate on fixable critical Common Vulnerabilities and Exposures (CVEs), an SBOM and provenance attestations, weekly Dependabot updates β
ci.yaml,.github/dependabot.yml; why: containers β supply chain. - Least privilege:
contents: readby default, write permissions only in the jobs that need them βci.yaml,deploy.yaml; why: GitOps β credentials. - CI never touches the cluster:
Deployonly commits a tag, Argo CD pulls it, one deploy at a time βdeploy.yaml,k8s/argocd/application.yaml; why: GitOps. - Releases judged by user-facing service level indicators (SLIs), with automatic abort β error ratio and p99 of the canary in
k8s/overlays/gitops/catalog-analysis.yaml; why: canary with metrics analysis.
Clean up
kubectl delete -f k8s/argocd/application.yaml # Argo CD stops managing (resources stay)
kind delete cluster --name boxoffice # frees all the memory of the cluster
Keep the workflows and the GitOps overlay: lab 13 adds AI steps to this pipeline. Disable the workflows (Actions β β¦ β Disable workflow) if you do not want CI minutes spent until then.
End state
boxoffice/
βββ .github/
β βββ workflows/ci.yaml # name: CI β test matrix, performance budget, image + supply chain
β βββ workflows/deploy.yaml # name: Deploy β writes the image tag of the GitOps overlay
β βββ ci/compose.perf.yaml # the performance-budget environment
β βββ dependabot.yml
βββ k8s/
β βββ argocd/application.yaml
β βββ monitoring/prometheus.yaml
β βββ overlays/gitops/ # kustomization, catalog-services, catalog-rollout, catalog-analysis
βββ load/budget.js
βββ services/catalog/
βββ pom.xml # + micrometer-registry-prometheus, test dependencies
βββ src/
βββ main/java/boxoffice/catalog/ChaosFilter.java
βββ test/java/boxoffice/catalog/CatalogApiTest.java
Commit and tag: git add -A && git commit -m "lab 11: delivery" && git tag lab11 && git push --tags.
Checklist
-
CIruns on every push and pull request: test matrix green, performance budget green, and onmainthe image job pushesghcr.io/<you>/boxoffice-catalog:sha-<commit>with provenance and SBOM attestations - the Trivy report is in Security β Code scanning; the gate blocks fixable CRITICAL vulnerabilities
- a pull request with the deliberate regression was blocked by the performance budget (screenshot)
- Argo CD deploys
k8s/overlays/gitopsfrom Git; a manual change was reverted by self-healing - a good release rolled out through the canary steps; the bad release was aborted automatically (output of
kubectl argo rollouts get rollout catalog) - results table filled in
results/lab11.md, with the answer to its question - self-check:
gh attestation verify oci://ghcr.io/<you>/boxoffice-catalog:main --owner <you>succeeds; pushed with the taglab11
Related
- Theory: Cloud-native platforms Β· Delivery: CI/CD, GitOps, deployment strategies Β· Zero trust and supply chain Β· Performance budgets
- Previous lab: 10. Kubernetes β the cluster and manifests deployed here.
- Project: team project β your pipeline is part of the milestone "K8s + CI/CD".
Next: lab 13 adds AI to this pipeline
Lab 13 builds directly on these workflows: an AI pull-request reviewer, test generation checked by mutation testing, an agent that proposes a fix when CI fails, AI release notes and an explained canary verdict β with the same discipline of permissions, budgets and measurements.
What breaks next
The pipeline now answers for every change: in the test run the budget held p95 around 7 ms against 300 ms and turned the pull request red with a 400 ms sleep, and the canary that failed 30% of its requests aborted by itself about a minute after Argo CD synced β only the fans who hit it saw errors. The platform can now carry the on-sale; what arrives next is a new kind of dependency. Fans want to ask "two seats under R$ 300 on Saturday?", and the answer would come from a model that takes seconds, is billed per token, can be wrong in fluent prose and reads the catalog and the seat map through tools β a box that no test, budget or canary of this lab knows how to judge. Lab 12 β The assistant puts it into BoxOffice behind a gateway, retrieval, read-only tools and guardrails: class 13 β AI Systems.