System Architecture · Insper

Hands-on 10 — Kubernetes

Declare the desired state; let control loops keep BoxOffice there

The story so far

chapter 10 · lab 9the patterns contain a slow provider
→
chapter 11 · lab 10kind, probes, autoscaling, Ingress
→
next · lab 11delivery: pipeline, GitOps, canary

Lab 9 brought GET /orders/{id} from 9 s at the median back to milliseconds — but every mitigation was a person at a terminal. Who restarts a dead replica, adds catalog pods at 09:45 or rolls a bad release back?

Nobody edits YAML at 10:00

Lab 3 scaled catalog by editing Compose. Lab 9 restarted a fallen service by hand. Who does that
during the flash sale?

  • Kubernetes: you declare desired state ("≥ 2 ready replicas, more above 60% central processing unit (CPU)").
  • Controllers compare desired vs actual every few seconds and act.
  • Self-healing, rollouts and autoscaling are all the same idea: reconciliation.

Step 1 · A cluster on the laptop

youlocalhost:80
→ :30080
control-plane
TraefikIngress controller · Helm · NodePort 30080
metrics-serverCPU for the autoscaler
worker
catalog
db-0 · redis
worker2
catalog
  • kind: each node is a Docker container. Horizontal Pod Autoscaler (HPA) needs metrics-server (--kubelet-insecure-tls in kind only).
  • Images: docker tag … boxoffice/catalog:dev + kind load docker-image — never latest.

ingress-nginx was retired in March 2026. Pick maintained, internet-facing components.

Step 2 · Kustomize: base + overlays

k8s/base/namespace, config, db, redis, catalog, hpa, ingress
+
k8s/overlays/local/image tag "dev" + lab secrets
→ kubectl apply -k →
namespace boxoffice11 objects, one desired state
  • secretGenerator → db-credentials-<hash>: new content, new name, rollout.
  • Lab 11 adds a GitOps overlay without touching the base.

Stateful vs stateless

db catalog
Kind StatefulSet Deployment
Identity stable: db-0 interchangeable
Storage PersistentVolumeClaim (PVC) per replica none
Service headless (clusterIP: None) ClusterIP, load-balanced

Production databases: managed service or an operator (backups, failover, upgrades).

Three probes, three questions

Probe Question On failure
startup has it finished starting? other probes wait; restart after the budget
liveness is it stuck beyond repair? restart
readiness should it get traffic now? removed from the Service

Never check a shared dependency in liveness: one database blip → every pod restarts at once.

Resources and graceful shutdown

  • requests (250m, 512Mi): what the scheduler reserves; HPA % is relative to it.
  • limits (1 CPU, 768Mi): CPU throttled, memory → OOMKilled; heap = 75% of the limit.
  • Rollout maxSurge: 1, maxUnavailable: 0: never below the desired ready count.
pod Terminating
→
endpoints removedTraefik updates a moment later
∥
preStop: sleep 5the Java virtual machine (JVM) still serves
→
SIGTERMserver.shutdown=graceful
→
exitwell within 30 s

Eventual convergence

catalog restarted twice right after kubectl apply. Bug?

  • No depends_on: catalog started before PostgreSQL → Flyway failed → exit → restart with back-off.
  • Components retry until dependencies exist; the system converges.
  • The first start also runs migrations → the startup budget must cover it (we raised it to 5 min).

Watch the controllers work

Self-healing
delete both pods
→
ReplicaSet: 0 of 2two new pods at once
→
2/2 readyJVM start: tens of seconds
Losing a node
docker stop worker2
→
NotReadyabout a minute
→
evictedafter the 5-min toleration
→
replaced on worker
Rolling update
+1 new podmaxSurge: 1
→ ready →
−1 old podmaxUnavailable: 0
→
repeat
  • Time each one: the self-healing time is the mean time to repair (MTTR) of a pod crash.
  • Detection dominates the lost node; JVM startup dominates the deleted pod.

Without a cluster: the simulator

  • The pages embed the lab's cluster as a simulator: the same nodes, manifests and commands.
  • Deploy: apply -k (catalog restarts twice), self-healing, a lost node.
  • Measure it: experiment B, with and without readiness + preStop.
  • Free mode: change how long Traefik takes to see an endpoint change.

Play it before the lab to know what to look for; your own cluster still gives the numbers you report.

Step 3 · The HPA

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 2
  maxReplicas: 8
  metrics: [{ type: Resource, resource: { name: cpu,
             target: { type: Utilization, averageUtilization: 60 } } }]
  behavior:
    scaleUp:   { stabilizationWindowSeconds: 0 }    # + at most 100% or 2 pods / 15 s
    scaleDown: { stabilizationWindowSeconds: 120 }  # - 1 pod / 30 s

desired = ⌈current × utilization / target⌉ → 2 pods at 150% → ⌈2 × 150 / 60⌉ = 5.

One turn of the loop: reactive scaling lags

metrics-serverCPU per pod
→
HPA, every 15 sdesired = ⌈current × CPU / 60%⌉
→
Deployment2 to 8 replicas
→
new podJVM start + probes: tens of s
→
readyaverage CPU falls
  • The HPA decides in seconds; capacity arrives only when a JVM is ready.
  • During the lag, existing pods absorb the step: p95 rises, then falls.
  • Known event (10:00 on-sale) → pre-scale; HPA for the unexpected.
  • More pods → more database connections: the bottleneck moves to the stateful tier.

Load it: warm pods first

A starting JVM burns CPU compiling code, and the HPA scales on that CPU too.

  • Right after a rollout, PEAK=250 took catalog to 8 replicas; liveness timed out, pods restarted, k6 saw 26% errors.
  • The same run with warm pods never left 2 replicas.
  • Warm up with lab 8's script, wait for 2 replicas, then k6 run -e PEAK=1000 load/catalog-hpa.js.
  • PEAK must push two warm pods above 60% of their request (≈ 600 req/s on the test laptop).

Step 4 · Ingress (and Gateway API)

curl localhost:80
→
node :30080kind.yaml
→
Traefikrule /catalog
→
Service catalog:8082
→
a ready pod
  • Ingress: host/path rules, ingressClassName: traefik.
  • Gateway API: Gateway (platform team) + HTTPRoute (service team); weights, headers, timeouts.

LoadBalancer Service = one address per service; Ingress = one door, many services; API gateway = auth, quotas, transformations.

Step 5 · Measure it

# Experiment Record
A fixed 2 replicas vs HPA, k6 ramp to PEAK p95, p99, errors, max replicas, time to scale
B rolling update under 100 req/s, with vs without readiness + preStop failed requests
C delete a pod / stop a node time until ready again

scripts/watch-hpa.sh logs CPU, desired, current and ready replicas every 5 s into a comma-separated values (CSV) file → plot it.

A · One run on a laptop (illustrative)

t CPU (% of request) desired ready
1:48 63 2 2
2:03 79 3 2
3:33 247 8 4
7:31 220 8 8
13:00 4 2 2
  • The eighth pod, asked at 3:33, was ready at 7:31 — after the hold ended.
  • HPA: p95 5–7 s, p99 ≈ 19 s, 0.3% errors. Two fixed warm pods: p95 ≈ 34 ms, no errors.
  • On a laptop the new JVMs share the CPUs that serve the load: scale-out cost more than it added. The lag is what pre-scaling removes.

B · A rollout under 100 req/s

Run Readiness probe + preStop Failed requests (test run)
with ✅ 0
without ❌ about 140 of 11,900 (1.2%, Traefik 502 and 504)
  • The startup probe still keeps a new pod out of the Service until the JVM answers.
  • Without the preStop pause, old pods stop accepting while still in Traefik's endpoint list.

Try it in the playground

  • scale-out with the stress profile: fixed replicas until p99 explodes, then more replicas.
  • Throughput per replica = the number behind your HPA target.
  • Kill a replica with and without a health check.

The playground predicts the shape; your CSV of replicas over time is the evidence.

What breaks next

lab 10 · self-healingHPA: a third pod 15 s after 60% CPU; rollout: 0 of ~11,900 lost
→
the limitkubectl from a laptop: any image goes to every pod, untested and unscanned
→
lab 11 · deliverytests, budget, scan, GitOps, a canary that aborts
  • A version that fails 30% of its requests would reach every fan.

Takeaways

  1. Kubernetes = desired state + control loops; self-healing, rollouts and autoscaling are reconciliation.
  2. StatefulSet + PVC for data; Deployments for interchangeable stateless pods.
  3. Startup, liveness, readiness answer different questions — no dependencies in liveness.
  4. Readiness + preStop + graceful shutdown = rollouts without errors.
  5. Detection dominates recovery: a lost node's pods are replaced only after NotReady + the toleration.
  6. The HPA scales on % of requests; it lags — pre-scale for known peaks.
  7. Ingress (or Gateway API) is the door; choose a maintained controller.

Chapter 11 of the story of the course. Outline: (1) why control loops; (2) kind + Traefik + metrics-server; (3) Kustomize base/overlay; (4) StatefulSet vs Deployment; (5) probes and graceful shutdown; (6) the controllers at work and the lab twin; (7) HPA; (8) Ingress and Gateway API; (9) measure.

The story of the course (2027.1 overview, "The story of the course"): chapter 11 opens with what Compose on one machine cannot do — heal, scale at 09:45, roll back.

The simulator models a laptop's timings (JVM start, PostgreSQL init) and Kubernetes' defaults (back-off, node detection, the 300 s toleration); the tips on the pages quote its numbers.