System Architecture · Insper

Hands-on 11 — Delivery

From git push to a canary that aborts by itself

The story so far

chapter 11 · lab 10a cluster that heals and scales
→
chapters 11, 12, 15 · lab 11tests, supply chain, budget, canary
→
next · lab 12the assistant

Lab 10: a rolling update lost 0 of ~11,900 requests — but it ships whatever image you built, untested, unscanned, unmeasured. Who checks a change before 2 M fans get it, and who rolls it back?

Ten changes a day, one flash sale

Each change could break checkout: a slow query, a vulnerable library, a version that fails 30% of
the requests. Who reviews all of them by hand?

  • Nobody can: the pipeline must answer "is this safe?" automatically.
  • And when it is wrong, the release must limit the damage.
  • Measure it: lead time and change failure rate (DORA: DevOps Research and Assessment).

The pipeline

CI: continuous integration, on every push and pull request
test (matrix)does it work? Testcontainers
performance budgetis it fast enough? k6
imagewhat is inside, is it vulnerable, who built it?
on main →
Deploywhich version should run? a Git commit
pull →
in the cluster
Argo CDapplies the overlay
Argo Rolloutsdeploys it, carefully

Step 1 · Test against the real thing

@SpringBootTest(webEnvironment = RANDOM_PORT)
@AutoConfigureTestRestTemplate
@Testcontainers
class CatalogApiTest {
    @Container @ServiceConnection
    static PostgreSQLContainer db = new PostgreSQLContainer("postgres:17");
    ...
}
  • Real migrations, SQL, Redis, JSON errors — Tests run: 3, Failures: 0.
  • name: CI, permissions: contents: read, one job per service, Maven cache.

Step 2 · Supply chain

buildload: true, not pushed yet
→
Trivyreport in Security; gate: fixable CRITICAL fails
→
SBOMSyft, SPDX: every package and version
→
pushghcr.io/…/boxoffice-catalog:sha-…
→
attestprovenance + SBOM, signed, bound to the digest
  • Scan before publishing: a vulnerable image never reaches the registry.
  • The software bill of materials (SBOM) answers "what is inside"; signed provenance answers "who built it, from which commit".

Every action pinned to a commit SHA (version in a comment): a tag can be moved by an attacker.

Step 3 · Performance budget

thresholds: {
  'http_req_duration{scenario:budget}': ['p(95)<300', 'p(99)<800'],
  'http_req_failed{scenario:budget}': ['rate<0.01'],
}
  • Fixed environment (.github/ci/compose.perf.yaml), 20 s warm-up not measured.
  • Normal run: p95 ≈ 7 ms. With Thread.sleep(400): ✗ → the pull request (PR) is red.
  • Margin on purpose: a noisy gate gets ignored.

Step 4 · GitOps

CIimage sha-…
→
Deploycommits the tag to k8s/overlays/gitops
→
Gitthe desired state
← polls every 3 min ←
Argo CDinside the cluster: apply, prune, self-heal
  • CI never touches the cluster: it only writes to Git.
  • Rollback = git revert; audit = git log; drift → reverted in seconds.

The cluster credentials never leave the cluster.

Step 5 · The canary

setWeight 251 canary pod + 2 stable: about a third
pause 2 min →
setWeight 501 canary + 1 stable
pause 2 min →
the canary becomes stable
AnalysisRun, every 30 s from the first stepcanary error ratio < 1% and p99 < 500 ms · failureLimit: 1 · empty result = pass
  • No traffic router: the Rollout reaches a weight by replica counts (workloadRef → the Deployment).
  • catalog-stable / catalog-canary Services point at each version; Prometheus labels every series with service.

A bad release, through Git

CHAOS_ERROR_RATE=0.3 committed to the overlay. What happens?

  • Argo CD syncs → the Rollout starts a canary pod.
  • Analysis: [] (no data yet), 0.068, 0.285 → failed twice > failureLimit: 1.
  • RolloutAborted → canary scaled down; stable keeps serving. Fix forward: git revert.

Without a cluster: replay the analysis

t = 0 s[] · no 500 series yet
→
t = 30 s0.258 ✖ (1)
→
t = 60 s0.284 ✖ (2) > failureLimit 1
→
RolloutAbortedabout 120 failed requests, 10% of that minute
  • The simulator on the canary step: a good release → every result [] → stable at 240 s.
  • 0 req/s with 0.3 → every result [] → the bad release is promoted: no traffic, no evidence.
  • Thread.sleep(400) → p99 0.409 s passes the canary; only the step 3 gate stops it.
  • Playground boxoffice-canary: the same bug as a canary costs a quarter of the errors.

What a canary cannot catch

  • Migrations run once for both versions → expand/contract only.
  • Rare errors need many canary requests → longer pauses at low traffic.
  • Side effects of canary requests (payments, e-mails) are real.
  • No traffic → no evidence: keep load during the analysis.

Step 6 · Measure it

# Experiment Record
A pipeline stages duration of each job
B lead time push → CI → deploy commit → synced → stable
C performance gate p95/p99/failed, normal vs regression
D canary sync → abort, error measurements, failed requests

Lead time is dominated by tests, image build, Argo CD polling (≤ 3 min) and the canary pauses — a deliberate price for safety.

What breaks next

lab 11 · safe changesbudget: p95 ~7 ms against 300 ms; the bad canary aborted in ~1 min
→
a new dependencya model: seconds per answer, billed per token, fluent mistakes, tools
→
lab 12 · assistantgateway, retrieval, read-only tools, guardrails
  • Fans ask: "two seats under R$ 300 on Saturday?"

Takeaways

  1. Test against real dependencies (Testcontainers); run CI on every PR and on main.
  2. Least privilege per job; pin every action to a commit SHA.
  3. Scan before publishing; SBOM + signed provenance answer "what" and "who".
  4. A performance budget turns the service level objective (SLO) into a build check — with margin.
  5. GitOps: Git is the source of truth; the cluster pulls.
  6. Canary + analysis stops bad releases automatically. Next: lab 13 adds AI to this pipeline.

Chapter 11 of the story of the course, with the supply chain of chapter 12 and the performance gate of chapter 15. Outline: (1) why automate safety; (2) CI with Testcontainers; (3) supply chain: Trivy, SBOM, provenance; (4) performance budget; (5) GitOps with Argo CD; (6) canary with Argo Rollouts; (7) measure: lead time.

The story of the course (2027.1 overview, "The story of the course"): this lab closes chapter 11 (roll back without a human) and builds the supply chain of chapter 12 and the performance gate of chapter 15.