Kubernetes

Docker Compose starts BoxOffice on one machine. Production needs more: forty containers spread over several machines, a replacement started within seconds when one dies, a new version rolled out without downtime, and capacity added automatically when the on-sale begins. Doing that by hand — SSH into a node, docker run, update the load balancer — does not survive the first night on call. An orchestrator does it for you, and Kubernetes is the de facto standard⁵.

Kubernetes descends from Google's internal cluster managers, Borg and Omega; its designers summarized the lesson of a decade of running containers at scale as declarative state reconciled by independent control loops¹.

The control loop: desired state vs actual state

You never tell Kubernetes "start a container". You tell it "there should be five replicas of catalog, version 1.4.0, each with 1 CPU" — the desired state — and store it in the API server. Controllers then run forever:

loop:
    desired = read spec from the API server
    actual  = observe the cluster
    if actual != desired:
        act to reduce the difference (create, delete, update)

sequenceDiagram
    participant U as kubectl / Argo CD
    participant API as API server (+ etcd)
    participant RS as ReplicaSet controller
    participant S as Scheduler
    participant K as kubelet (node 2)
    U->>API: apply Deployment catalog, replicas = 5
    API->>RS: watch: desired 5, actual 4 (a node died)
    RS->>API: create Pod catalog-7f9c
    API->>S: watch: unscheduled pod
    S->>API: bind catalog-7f9c to node 2 (enough free CPU)
    API->>K: watch: new pod for this node
    K->>K: pull image, start container, run probes
    K->>API: status: Running, Ready
    Note over RS,API: actual = desired → nothing to do

This is why Kubernetes self-heals: a crashed process, a deleted pod or a dead node all appear as "actual ≠ desired", and the same loop fixes them. It is also why the API is declarative and idempotent: applying the same manifest twice changes nothing.

The pieces of a cluster

With a managed service (EKS, GKE, AKS) the provider runs and upgrades the control plane; you manage the nodes — or, with GKE Autopilot or EKS on Fargate, not even those.

The core objects

BoxOffice's catalog on Kubernetes

The manifests below show what a production catalog needs. Read them top to bottom; each block answers one operational question. Hands-on 10 writes a smaller version for a three-node cluster on a laptop (box after the manifests).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalog
  labels: { app: catalog }
spec:
  replicas: 4
  selector:
    matchLabels: { app: catalog }
  strategy:
    type: RollingUpdate
    rollingUpdate: { maxSurge: 25%, maxUnavailable: 0 }   # never below 4 ready pods
  template:
    metadata:
      labels: { app: catalog }
    spec:
      terminationGracePeriodSeconds: 30
      topologySpreadConstraints:                           # spread replicas across zones
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector: { matchLabels: { app: catalog } }
      containers:
        - name: catalog
          image: ghcr.io/boxoffice/catalog@sha256:9f2c0d7e1b4a   # digest, not a tag
          ports: [ { containerPort: 8080 } ]
          envFrom:
            - configMapRef: { name: catalog-config }
            - secretRef:    { name: catalog-db }
          resources:
            requests: { cpu: "500m", memory: "768Mi" }      # what the scheduler reserves
            limits:   { memory: "768Mi" }                   # hard ceiling (OOMKilled above)
          startupProbe:                                     # JVM warm-up: up to 60 s
            httpGet: { path: /actuator/health/liveness, port: 8080 }
            periodSeconds: 5
            failureThreshold: 12
          readinessProbe:                                   # receive traffic?
            httpGet: { path: /actuator/health/readiness, port: 8080 }
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:                                    # restart me?
            httpGet: { path: /actuator/health/liveness, port: 8080 }
            periodSeconds: 10
            failureThreshold: 3
          lifecycle:
            preStop: { sleep: { seconds: 5 } }              # let endpoints update before SIGTERM

apiVersion: v1
kind: Service
metadata: { name: catalog }
spec:
  selector: { app: catalog }
  ports: [ { port: 8080, targetPort: 8080 } ]
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: catalog }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: catalog }
  minReplicas: 4
  maxReplicas: 40
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 60 }   # % of the CPU request
  behavior:
    scaleUp:   { stabilizationWindowSeconds: 0,   policies: [ { type: Percent, value: 100, periodSeconds: 15 } ] }
    scaleDown: { stabilizationWindowSeconds: 300, policies: [ { type: Pods,    value: 2,   periodSeconds: 60 } ] }
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: catalog }
spec:
  minAvailable: 3
  selector: { matchLabels: { app: catalog } }

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: boxoffice-api }
spec:
  parentRefs: [ { name: public-gateway } ]
  hostnames: [ "api.boxoffice.example" ]
  rules:
    - matches: [ { path: { type: PathPrefix, value: /catalog } } ]
      backendRefs: [ { name: catalog, port: 8080 } ]
    - matches: [ { path: { type: PathPrefix, value: /inventory } } ]
      backendRefs: [ { name: inventory, port: 8080 } ]

The Gateway API (Gateway + HTTPRoute) is the successor of the older Ingress object: it separates the infrastructure owner (who runs the Gateway) from the application teams (who own routes), and supports weighted backends natively³ — which is how canaries split traffic (see Delivery). The Edge class covers the Ingress component.

env:
  - name: SERVER_SHUTDOWN
    value: graceful        # finish in-flight requests on SIGTERM
resources:
  requests:                # what the scheduler reserves; HPA percentages are relative to this
    cpu: 250m
    memory: 512Mi
  limits:
    cpu: "1"
    memory: 768Mi
startupProbe:              # the JVM needs time: no liveness kills before it has started
  httpGet: { path: /actuator/health/liveness, port: http }
  periodSeconds: 5
  failureThreshold: 60     # up to 5 min: the first start also runs the Flyway migrations
livenessProbe:             # "restart me": only for a process that is stuck
  httpGet: { path: /actuator/health/liveness, port: http }
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:            # "send me traffic": removed from the Service while not ready
  httpGet: { path: /actuator/health/readiness, port: http }
  periodSeconds: 5
  failureThreshold: 2
lifecycle:
  preStop:
    sleep:
      seconds: 5           # let the endpoints update before the JVM stops accepting

Probes: three different questions

Keep liveness local (is the JVM responsive?) and make readiness reflect only what this pod itself needs to serve. Spring Boot Actuator exposes /actuator/health/liveness and /actuator/health/readiness with exactly these semantics. The Observability class treats health checks as a palette component (Health Check).

Requests, limits and quality of service

• Request — what the scheduler reserves on a node. It decides where a pod fits and is the base of the HPA's utilization percentage.
• Limit — the ceiling. Above the memory limit the container is killed (OOMKilled); above the CPU limit it is throttled — it simply waits, which shows up as latency, not as errors.

Losing a node

The control loop replaces lost pods, but self-healing needs two things the loop does not provide. The replacement pods need room: the scheduler places a pod only on a node whose free capacity covers its requests, so in a cluster packed to 100% the pods of a dead node stay Pending. And callers must stop sending to the dead pods: until the node is declared NotReady (tens of seconds by default) and the Service endpoints are updated, part of the traffic still goes to pods that no longer exist. These are the two rules of redundancy and failover from the Reliability class, at the level of nodes: N + 1 capacity, and fast failure detection.

Try it: in k8s-node-failure, catalog and pricing run 6 pods each on 3 nodes × 4 pods — a full cluster — and at t = 60 s one node dies.

1. Run it as it is. For about 40 s, around 90% of the requests fail: callers still send to the dead pods. Errors then fall, and come back at 7–10% from t ≈ 150 s: the 2 lost pods of each service have nowhere to go, and 4 survivors carry the load of 6. Over the run: errors ≈ 27%, p99 ≈ 4.6 s.
2. Give the cluster room: set podsPerNode to 8 on the cluster frame. The late errors disappear (the lost pods are rescheduled), but the 40-s window stays: errors ≈ 25%.
3. Connect a Service Registry to catalog and pricing — the playground's stand-in for fast endpoint updates. Callers drop the dead pods in 3 s instead of 30–40 s: errors ≈ 4%.

Autoscaling at three levels

flowchart LR
    load["k6 / fans<br/>requests per second ↑"] --> hpa["HPA<br/>more pods"]
    hpa --> pend["Pods Pending<br/>(no room on nodes)"]
    pend --> ca["Cluster Autoscaler / Karpenter<br/>more nodes"]
    ca --> run["Pods scheduled<br/>image pulled, JVM warm"]
    run --> ready["Ready → receive traffic"]

1. Pods — Horizontal Pod Autoscaler (HPA). Every 15 s the HPA computes²

With 4 catalog pods at 90 % CPU (of request) and a 60 % target: ⌈4 × 90/60⌉ = 6 pods. Scaling on CPU works for CPU-bound services; for I/O-bound services scale on requests per second per pod or on queue lag through custom/external metrics.

2. Nodes — Cluster Autoscaler or Karpenter. When pods stay Pending because no node has room, a node autoscaler provisions new machines (Karpenter chooses instance types per workload and can use spot capacity). This takes minutes: VM boot, node join, image pull.

3. Events — KEDA. The Kubernetes Event-Driven Autoscaler scales on external sources (Kafka consumer lag, queue length, cron windows) and can scale to zero — natural for the notification workers⁴.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: catalog
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: catalog
  minReplicas: 2
  maxReplicas: 8
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60     # % of the CPU *request* (250m): scale when pods average > 150m
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0  # react at once to load
      selectPolicy: Max
      policies:
        - type: Percent
          value: 100                 # at most double...
          periodSeconds: 15
        - type: Pods
          value: 2                   # ...or +2 pods, whichever is more, every 15 s
          periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 120  # wait 2 min of lower load before removing pods
      policies:
        - type: Pods
          value: 1
          periodSeconds: 30

Try it: in the scenario below, run the spike profile with 2 replicas, then with 6. Watch p99 latency and errors while the replicas are busy; then enable autoscaling (if offered) and observe the lag between the start of the spike and the moment new replicas are healthy.

Where to run Kubernetes

Trade-offs

How it connects

flowchart LR
    ETCD["etcd: Raft<br/>→ Consensus (class 3)"] --> DS["desired state<br/>in the API server"]
    DS --> CL["control loop:<br/>actual → desired"]
    CL --> DEP["Deployment → pods"]
    DEP --> SVC["Service, Ingress, Gateway API<br/>→ Load balancing"]
    DEP --> PR["startup, readiness,<br/>liveness probes"]
    PR --> HC["health endpoints<br/>→ Reliability, Observability"]
    DEP --> RL["requests and limits,<br/>QoS, CPU throttling"]
    RL --> HPA["HPA, node autoscaler,<br/>KEDA on lag"]
    HPA --> PRE["pre-scale before 10:00<br/>→ Compute autoscaling"]
    RL --> COST["requests drive the bill<br/>→ FinOps"]
    DEP --> PDB["PDB, zone spread<br/>→ redundancy"]

Palette components: app · worker (KEDA scales notification on lag) · scheduler (CronJob releases expired holds) · lb (a Service is an L4 balancer) · ingress · healthcheck · metrics (what the HPA reads). Scenarios: scale-out, k8s-node-failure (N + 1 nodes), flash-sale (pre-scaling). Continues in: Traffic & edge — load balancing in Kubernetes, Distributed — consensus (etcd), Compute — autoscaling lag, Reliability — health checks, FinOps, Exercise — Kubernetes autoscaling.