System Architecture · Insper

Cloud-native platforms and delivery

Where the code runs, how it gets there, how it scales — and who pays

BoxOffice leaves the laptop

Nine services run fine with docker compose up. What changes on the morning a stadium show goes on sale?

  • 20× the traffic for 30 minutes, then back to normal — and nobody wants to pay for idle machines.
  • A fix to inventory at 09:55 must ship without downtime at 10:00.
  • Someone must know what it costs before the invoice arrives.
  • None of these is an application problem: they are platform problems.

What "cloud-native" means

Idea BoxOffice example
Immutable artifacts inventory:1.8.3 built once, promoted unchanged
Declarative desired state "5 replicas of catalog" — a controller makes it true
Elasticity catalog 4 → 40 replicas during the on-sale
Automation end to end merged pull request → pipeline → production

The app must cooperate: stateless processes, config from the environment, fast start and graceful shutdown, logs to stdout (twelve-factor).

Service models: who operates what

Infrastructure, containers, platform, functions, software as a service (IaaS, CaaS, PaaS, FaaS, SaaS); managed databases are DBaaS.

On-premises
IaaS
CaaS
PaaS
FaaS
SaaS
Data and access rules
you
you
you
you
you
you
Application code
you
you
you
you
you
provider
Runtime (JVM, Python)
you
you
you
provider
provider
provider
Orchestration and scaling
you
you
provider
provider
provider
provider
Operating system
you
you
shared
provider
provider
provider
Virtualization, hardware
you
provider
provider
provider
provider
provider
  • Your share shrinks one layer per model, except from PaaS to FaaS (same column: an app or a function).
  • Whatever the model, data, identities and access rules stay yours: a public bucket is a customer mistake.

Who acts when something happens?

Event Layer IaaS CaaS PaaS / FaaS
JDK flaw runtime you you (JDK in your image) provider
Kernel flaw OS you shared (nodes) provider
On-sale 20× orchestration and scaling you add VMs platform, within your limits platform, within your limits
Public bucket data and access you you you (and on SaaS too)

Simulator on the page: pick a model and an event, and see which layer it hits and who acts.

Who offers each model

Model AWS Google Cloud Microsoft Azure Other providers
IaaS EC2 Compute Engine Virtual Machines DigitalOcean, Oracle Cloud, Magalu Cloud
CaaS EKS, ECS on Fargate GKE, Cloud Run AKS, Container Apps DigitalOcean Kubernetes, OVHcloud
PaaS Elastic Beanstalk App Engine App Service Heroku, Railway, Render, Fly.io
FaaS Lambda Cloud Run functions Azure Functions Cloudflare Workers, Vercel
DBaaS RDS, Aurora, ElastiCache, MSK Cloud SQL, AlloyDB, Memorystore Azure Database, Managed Redis Neon, Supabase, Aiven
SaaS SES (e-mail) Workspace Microsoft 365, Dynamics 365 Stripe, Mercado Pago, Datadog, TOTVS, RD Station

Read by columns: AWS is on every row — a model describes a service, not a company.

BoxOffice, component by component

CaaS · managed Kubernetes
gateway · catalog · inventory · order · search
DBaaS · managed
PostgreSQL Multi-AZ
Redis · Kafka
FaaS
thumbnails · webhooks
SaaS
payments · e-mail

Why not run everything on FaaS?

  • Connection pools: thousands of short-lived functions would exhaust PostgreSQL.
  • Cold starts and concurrency limits hurt exactly at the 10:00 peak.
  • Steady-load cost: a service busy 24/7 is cheaper on always-on containers (FinOps break-even).

Platform engineering: golden paths

product teamorder, inventory…
uses →
developer portale.g. Backstage
→
service templateDockerfile · manifests · pipeline
self-service environments
built-in dashboards and alerts
→
platformKubernetes · Argo CD · Terraform modules · policies
  • An internal developer platform (IDP), built as a product by a platform team for the other teams.
  • A golden path is optional but attractive: a new service gets a hardened image, probes, a pipeline, a canary and dashboards on day one.

Treat the platform as a product with users — or it becomes the ticket queue again.

Containers are processes

Namespaces — what it sees

  • own PIDs, network, mounts, hostname
  • catalog is PID 1 in its own world

cgroups — what it may use

  • CPU, memory, I/O limits
  • above the memory limit → OOMKilled

vs. virtual machines

  • one shared kernel — start in ms, high density
  • weaker isolation → multi-tenant clouds add micro-VMs (Firecracker, gVisor)

The Java virtual machine (JVM) must size its heap from the container limit: -XX:MaxRAMPercentage=75.

A production image for catalog

FROM maven:3.9-eclipse-temurin-21 AS build
WORKDIR /src
COPY pom.xml .
RUN mvn -q dependency:go-offline          # cached until pom.xml changes
COPY src ./src
RUN mvn -q package -DskipTests && java -Djarmode=tools -jar target/catalog.jar \
      extract --layers --launcher --destination target/extracted

FROM gcr.io/distroless/java21-debian12:nonroot
COPY --from=build /src/target/extracted/dependencies/ ./
COPY --from=build /src/target/extracted/application/ ./
ENV JAVA_TOOL_OPTIONS="-XX:MaxRAMPercentage=75"
USER nonroot
ENTRYPOINT ["java", "org.springframework.boot.loader.launch.JarLauncher"]

Multi-stage · dependencies before source · distroless · non-root · exec form so SIGTERM reaches Java.

Graceful shutdown, step by step

pod Terminatingremoved from the endpoints
+
preStop: sleep 5 sproxies catch up
→
SIGTERM
→
stop accepting, finish in-flightup to 20 s
→
exit 0
still running after the 30 s grace period → SIGKILL
  • Spring Boot: server.shutdown: graceful and timeout-per-shutdown-phase: 20s — less than Kubernetes' 30 s.
  • Exec-form ENTRYPOINT: Java is PID 1 and receives the SIGTERM (shell form: sh swallows it).

Images and the supply chain

  • An Open Container Initiative (OCI) image = manifest + config + layers, all addressed by SHA-256 digest.
  • Tags move; digests do not — deploy by digest.
  • Scan (Trivy) · SBOM, software bill of materials (Syft, CycloneDX) · sign (Cosign) · provenance (SLSA: supply-chain levels for software artifacts).
  • Never bake secrets into layers: they stay in the history forever.

Small, layered images = faster scale-out: every new node must pull the image before the pod starts.

Kubernetes: the control loop

desired stateAPI server + etcd · "4 replicas"
watch →
controller comparesactual: 2 (a node died)
→
actcreate 2 pods
→
scheduler bindsa node with room
→
kubelet starts, probes passactual = desired
↺ forever, every few seconds: nothing to do while actual = desired
  • Self-healing, rolling updates and autoscaling are all the same loop.
  • The API is declarative and idempotent: applying the same manifest twice changes nothing.

The objects behind one service

Object Role for catalog
Deployment 4 identical pods; rolling update with maxUnavailable: 0
Service stable DNS name catalog:8080 in front of changing pods
HTTPRoute (Gateway API) /catalog from the internet → Service; weighted backends for canaries
ConfigMap / Secret database URL / credentials as environment variables
HPA (Horizontal Pod Autoscaler) 4 → 40 replicas, target 60 % CPU of the request
PodDisruptionBudget never fewer than 3 pods during node upgrades
topologySpreadConstraints replicas spread across availability zones

Probes answer three different questions

Probe Question On failure
Startup finished starting? (JVM warm-up) others disabled; restart after the threshold
Readiness should I get traffic now? removed from endpoints — no restart
Liveness stuck beyond repair? container restarted

Never put the database in liveness (restart storm) — and not in readiness either: one DB blip makes every pod unready at once.

Requests, limits, throttling

  • Request = what the scheduler reserves; the base of the HPA percentage.
  • Memory limit exceeded → OOMKilled (an error).
  • CPU limit exceeded → throttled (latency, not errors).

Average CPU 45 %, p99 spiky. Why?

  • Threads burn the 100 ms quota of the completely fair scheduler (CFS) early in each period and wait for the rest.
  • Check container_cpu_cfs_throttled_periods_total; keep the CPU request, consider no CPU limit.

Losing a node

node dies2 catalog pods
→
callers still send to themtens of seconds
→
NotReady, endpoints updated
→
pods evictedtolerations: 5 min by default
→
room: rescheduled
cluster full: Pending
  • Self-healing needs room (N + 1 nodes) and fast endpoint updates.
  • Playground k8s-node-failure (3 full nodes): errors ≈ 27% → 25% with room → ≈ 4% with fast endpoint updates.

Autoscaling at three levels — and its lag

  • HPA (pods): desired = ⌈current × metric / target⌉ → 4 pods at 90 % with target 60 % → 6.
  • Cluster Autoscaler / Karpenter (nodes): minutes — VM boot, join, image pull.
  • KEDA, Kubernetes Event-Driven Autoscaler (events): Kafka lag, queue length, cron; scale to zero.

A flash sale goes 5 % → 100 % in seconds; autoscaling needs minutes. Pre-scale at 09:45, keep spare nodes, and let a waiting room control admission.

The delivery pipeline

pull request
→
CIbuild · test · scan · SBOM · sign · push
→
config repodigest bumped
→
Argo CDsyncs the cluster to Git
→
canary5 → 25 → 50 %
→
healthy: 100 %
degraded: rollback
  • Continuous integration (CI) never touches production: it only writes to Git. Build once, promote by digest.
  • Infrastructure as code (IaC), Terraform / OpenTofu: virtual private cloud (VPC), cluster, PostgreSQL, buckets as reviewed code; remote, locked state; a scheduled plan detects drift.

GitOps: pull, don't push

Push (kubectl apply in CI) Pull (Argo CD, Flux)
Credentials CI is cluster-admin only the in-cluster agent
Drift unnoticed detected, reverted (selfHeal)
Audit CI logs git log
Rollback re-run a pipeline git revert

OpenGitOps: declarative · versioned and immutable · pulled automatically · continuously reconciled.

Deployment strategies

Strategy Downtime Extra capacity Rollback Initial blast radius
Recreate yes none slow 100 %
Rolling update no small gradual grows
Blue/green no 2× briefly instant switch 100 %
Canary no small weight → 0 1–5 %
Shadow no up to 2× n/a 0 % (side effects!)
Feature flags no none flip the flag chosen users

The database does not roll back: expand / contract so two versions can run side by side.

Expand / contract: the database does not roll back

v1 uses email
→
1 expandadd buyer_email, nullable
→
2 v2 writes bothreads new, falls back
→
3 backfill
→
4 contractdrop email, next release
rename in one release → v1 pods still running query a column that is gone
  • Old and new pods overlap during every rolling update or canary: each step must work with both.

Canary judged by metrics

  • Argo Rollouts: setWeight 5 → 25 → 50, an AnalysisTemplate queries Prometheus each minute.
  • Abort when canary error rate > 0.5 % or p99 > 300 ms twice — compared with the stable baseline, same traffic, same hour.
Regression (baseline 0.1 %) Requests needed Time at 5 % of 400 req/s
→ 1.0 % ≈ 835 < 1 min
→ 0.5 % ≈ 2 300 ≈ 2 min
→ 0.2 % ≈ 18 500 ≈ 15 min

Canary steps are minutes, not seconds: rare errors need samples.

DORA: speed and stability together

DevOps Research and Assessment (DORA): four metrics that move together.

Metric Measures
Deployment frequency throughput
Lead time for changes throughput
Change failure rate stability
Time to restore service stability
  • High performers are better on all four: small, frequent, automatically verified changes are faster and safer.
  • AI in the pipeline (pull-request review, generated tests, AI canary analysis) → class AI-Assisted Delivery.

From CapEx to OpEx

Capacity vs demand Two charts of resources over time with the same rising demand. Left, CapEx: capacity grows in large steps bought ahead; it sits far above demand for long periods (idle capacity you paid for) and falls behind before the next purchase arrives (shortage). Right, OpEx: capacity follows demand closely, with a small margin. CapEx: capacity bought in steps time → resources → OpEx: capacity follows demand time → resources → purchase purchase capacity demand idle (paid, unused) shortage (lost demand)

Capital expenditure (CapEx) buys capacity in steps months ahead: idle after each purchase, shortage before the next. Operational expenditure (OpEx) follows demand: BoxOffice runs 6 nodes all month and 24 more only for the 8 hours of on-sales.

FinOps: inform → optimize → operate

  • Inform: tags (service, team, env), Kubernetes cost by requests (OpenCost), showback.
  • Optimize: rightsizing, autoscaling, commitments for the floor, spot for interruptible work, ARM, less egress.
  • Operate: budgets, anomaly alerts, policies, monthly review next to the service level objective (SLO) review.

Surprises come from fixed hourly items (control plane, load balancer, NAT gateway), data transfer and observability — not from CPUs.

BoxOffice: where the money goes (model)

Line (approx., us-east-1) $/month
Content delivery network (CDN) egress, 20 TB ≈ 1 700
Worker nodes, 6 always on ≈ 840
Kafka, 3 brokers ≈ 610
PostgreSQL Multi-AZ + replica ≈ 560
Logs, 1 TB ingested ≈ 500
Redis, NAT, load balancer, control plane, e-mail… ≈ 790
24 extra nodes for 4 on-sales × 2 h ≈ 37

The peak is cheap when it is elastic: the same 24 nodes always on would cost ≈ $3 400/month.

Pay per request or per hour?

  • FaaS: no floor, higher cost per request. Containers: fixed floor, low marginal cost.
  • With the page's assumptions, break-even ≈ 10 req/s average.
  • Average decides FaaS cost; peak decides container size.
  • BoxOffice: thumbnails and webhooks on FaaS; the purchase path on containers.

Multi-AZ doubles the database bill. Waste?

  • No — it buys availability. FinOps is not "spend less", it is "know what each dollar buys".

The cloud-native concept map

Component Question it answers Key concepts
app · worker packaged to run the same everywhere? stateless, OCI image, digest, graceful shutdown
app · ingress · healthcheck who keeps N healthy replicas? control loop, Service, probes, requests/limits
app · worker · scheduler how does capacity follow the on-sale? HPA, node autoscaler, KEDA, pre-scaling
serverless pay per request or per hour? cold starts, concurrency, break-even
sql · cache · kafka · payment who operates each component? IaaS … SaaS, shared responsibility
metrics · ingress how does a change reach production? IaC, GitOps, canary judged on SLOs
cdn · nat · logs where does the money go? fixed vs variable, egress, unit economics

Cloud-native is not a box on the diagram: it is how every box is packaged, run, delivered and paid for.

Takeaways

  1. Cloud-native = immutable artifacts, declarative state, elasticity, automation — on stateless, disposable apps.
  2. Choose the service model per component — a model is a service, not a company; data, identity and config are always yours.
  3. Small, layered, signed images, deployed by digest; graceful shutdown on SIGTERM.
  4. Kubernetes is a control loop; get probes, requests/limits and PodDisruptionBudgets right; keep room for a lost node.
  5. Autoscaling lags: pre-scale for known peaks.
  6. IaC + CI + GitOps + canary gated by SLO metrics + expand/contract migrations.
  7. FinOps: OpEx follows demand but is not automatically cheaper; tag, model and optimize what dominates — usually data, egress and fixed items.

Outline: (1) cloud-native and service models; (2) containers; (3) Kubernetes; (4) IaC, CI, GitOps, deployment strategies and canary analysis; (5) FinOps and the BoxOffice cost model.

Let the class list the problems before revealing. Every item maps to a part of the class: elasticity (Kubernetes), safe delivery (GitOps + canary), cost (FinOps).

The page's table, as a matrix. The only shared cell is the OS on CaaS: with classic node groups the team rolls the nodes onto the provider's patched image; with Fargate or GKE Autopilot the provider does it all.

The simulator responsibility-stack sits under the stack table of the overview. The on-sale hits a real layer of this table (orchestration and scaling): on CaaS the HPA adds pods, but maxReplicas and the 09:45 pre-scale are the team's. On-premises the capacity had to be bought months before (CapEx, FinOps).

On AWS, BoxOffice is one column: EKS, RDS and ElastiCache, MSK, S3, Lambda. Nothing forces one company per system: payments go to Stripe or Mercado Pago, the code and CI to GitHub.

Multi-AZ: a standby in a second availability zone. Ticket PDFs and images go to the object store (S3), an IaaS-level storage service.

Endpoint removal and SIGTERM start at the same moment; without the preStop pause some proxies still send requests to a pod that no longer accepts them. Lab 10 measures it: 0 failed requests with the pause.

Ask: who restarts the pods when a node loses power? Answer: nobody — the ReplicaSet controller sees 2 instead of 4 and creates pods; the scheduler finds room.

The page's diagram plays the same steps. Remaining errors come from the survivors carrying the load of 6 pods with 4; 8 replicas bring the errors to about 1.7%.

Same steps as the diagram on the Delivery page. Never rename or drop a column in the release that stops using it.

OpEx removes the up-front bet but is not automatically cheaper: a workload at full load 24/7 for years can cost less on owned or committed capacity (the commitments of the pricing models). And a bill that follows usage also follows every mistake, within the month: that is what FinOps is for.