Skip to content

2. Deploy BoxOffice

Compose describes containers to start. Kubernetes manifests describe a desired state β€” "two ready replicas of catalog, reachable as catalog:8082" β€” and controllers work continuously to make reality match it: a crashed pod is replaced, a dead node's pods are rescheduled, a new image is rolled out pod by pod. This step writes that desired state for the read path of BoxOffice: db, redis and catalog.

Layout: base + overlays

Manifests live in k8s/ of your repository, organized with Kustomize (built into kubectl): a base with everything that is the same in every environment, and one overlay per environment with only what differs (image tags, secrets, replica counts). Lab 11 adds an overlay for the GitOps deployment without touching the base.

k8s/
β”œβ”€β”€ kind.yaml                      # the cluster (step 1)
β”œβ”€β”€ base/
β”‚   β”œβ”€β”€ kustomization.yaml
β”‚   β”œβ”€β”€ namespace.yaml
β”‚   β”œβ”€β”€ config.yaml                # ConfigMap: DB_HOST, REDIS_HOST, ...
β”‚   β”œβ”€β”€ db.yaml                    # PostgreSQL StatefulSet + headless Service
β”‚   β”œβ”€β”€ redis.yaml
β”‚   β”œβ”€β”€ catalog.yaml               # Deployment + Service
β”‚   β”œβ”€β”€ catalog-hpa.yaml           # step 3
β”‚   └── ingress.yaml               # step 4
└── overlays/
    └── local/
        └── kustomization.yaml     # image tag "dev" + lab secrets
k8s/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: boxoffice
labels:
  - pairs:
      app.kubernetes.io/part-of: boxoffice
resources:
  - namespace.yaml
  - config.yaml
  - db.yaml
  - redis.yaml
  - catalog.yaml
  - catalog-hpa.yaml
  - ingress.yaml
k8s/overlays/local/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
images:
  - name: boxoffice/catalog          # the name used in base/catalog.yaml...
    newTag: dev                      # ...and the tag loaded into the local cluster
secretGenerator:
  - name: db-credentials             # lab only: real secrets never live in Git in plain text
    namespace: boxoffice
    literals:
      - DB_USER=boxoffice
      - DB_PASSWORD=boxoffice
k8s/base/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: boxoffice
k8s/base/config.yaml
# Non-secret settings shared by the services (the same variables as in docker-compose.yaml)
apiVersion: v1
kind: ConfigMap
metadata:
  name: boxoffice-config
data:
  DB_HOST: db
  DB_PORT: "5432"
  DB_NAME: boxoffice
  REDIS_HOST: redis
  REDIS_PORT: "6379"

The secretGenerator appends a hash of the content to the Secret's name (db-credentials-c67…) and rewrites every reference to it: changing a password produces a new Secret and therefore a rollout of the pods that use it β€” a stale credential cannot linger in a running pod.

Stateful and stateless

k8s/base/db.yaml
# PostgreSQL as a StatefulSet: a stable name (db-0) and its own persistent volume.
# Fine for a lab; in production use a managed database or an operator (e.g. CloudNativePG).
apiVersion: v1
kind: Service
metadata:
  name: db
spec:
  clusterIP: None            # headless: DNS "db" resolves to the pod itself
  selector:
    app: db
  ports:
    - name: postgres
      port: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db
  replicas: 1
  selector:
    matchLabels:
      app: db
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
        - name: postgres
          image: postgres:17
          ports:
            - name: postgres
              containerPort: 5432
          env:
            - name: POSTGRES_DB
              valueFrom:
                configMapKeyRef: { name: boxoffice-config, key: DB_NAME }
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef: { name: db-credentials, key: DB_USER }
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef: { name: db-credentials, key: DB_PASSWORD }
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          readinessProbe:
            exec:
              command: ["sh", "-c", "pg_isready -U \"$POSTGRES_USER\" -d \"$POSTGRES_DB\""]
            periodSeconds: 5
          resources:
            requests: { cpu: 250m, memory: 256Mi }
            limits: { cpu: "1", memory: 1Gi }
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 2Gi
k8s/base/redis.yaml
apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  selector:
    app: redis
  ports:
    - name: redis
      port: 6379
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
        - name: redis
          image: redis:8-alpine
          args: ["--maxmemory", "256mb", "--maxmemory-policy", "allkeys-lru"]
          ports:
            - name: redis
              containerPort: 6379
          readinessProbe:
            exec:
              command: ["redis-cli", "ping"]
            periodSeconds: 5
          resources:
            requests: { cpu: 100m, memory: 128Mi }
            limits: { cpu: 500m, memory: 320Mi }

A StatefulSet gives the database what a Deployment does not: a stable identity (db-0) and a PersistentVolumeClaim per replica that survives pod restarts. Deleting the pod keeps the data; deleting the PVC deletes it.

The catalog Deployment

k8s/base/catalog.yaml
apiVersion: v1
kind: Service
metadata:
  name: catalog
spec:
  selector:
    app: catalog
  ports:
    - name: http
      port: 8082
      targetPort: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalog
spec:
  replicas: 2                        # the HPA takes over from here
  selector:
    matchLabels:
      app: catalog
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1                    # one extra pod during a rollout...
      maxUnavailable: 0              # ...and never fewer ready pods than desired
  template:
    metadata:
      labels:
        app: catalog
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: catalog
          image: boxoffice/catalog:dev
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8082
          envFrom:
            - configMapRef:
                name: boxoffice-config
            - secretRef:
                name: db-credentials
          env:
            - name: SERVER_SHUTDOWN
              value: graceful        # finish in-flight requests on SIGTERM
          resources:
            requests:                # what the scheduler reserves; HPA percentages are relative to this
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: "1"
              memory: 768Mi
          startupProbe:              # the JVM needs time: no liveness kills before it has started
            httpGet: { path: /actuator/health/liveness, port: http }
            periodSeconds: 5
            failureThreshold: 60     # up to 5 min: the first start also runs the Flyway migrations
          livenessProbe:             # "restart me": only for a process that is stuck
            httpGet: { path: /actuator/health/liveness, port: http }
            periodSeconds: 10
            failureThreshold: 3
          readinessProbe:            # "send me traffic": removed from the Service while not ready
            httpGet: { path: /actuator/health/readiness, port: http }
            periodSeconds: 5
            failureThreshold: 2
          lifecycle:
            preStop:
              sleep:
                seconds: 5           # let the endpoints update before the JVM stops accepting

The three probes answer three different questions, and confusing them causes outages:

Probe Question On failure Spring Boot endpoint
startupProbe has the process finished starting? keeps the other probes off; restart after the budget /actuator/health/liveness
livenessProbe is the process stuck beyond repair? restart the container /actuator/health/liveness (internal state only)
readinessProbe should this pod receive traffic now? remove from the Service endpoints (no restart) /actuator/health/readiness

Never put a dependency in the liveness probe

If liveness checked the database, a database blip would make Kubernetes restart every catalog pod at once β€” turning a 10-second blip into minutes of outage while JVMs restart. Spring Boot's liveness group checks only the application's own state; that is why lab 1 enabled management.endpoint.health.probes.

requests are what the scheduler reserves on a node (and the base of HPA percentages); limits are the ceiling β€” above the memory limit the container is killed (OOMKilled), above the CPU limit it is throttled. Lab 1's Dockerfile sizes the heap with -XX:MaxRAMPercentage=75, so the JVM respects the 768 Mi limit.

Apply

kubectl apply -k k8s/overlays/localnamespace/boxoffice created
configmap/boxoffice-config created
secret/db-credentials-c677754gbf created
service/catalog created
service/db created
service/redis created
deployment.apps/catalog created
deployment.apps/redis created
statefulset.apps/db created
horizontalpodautoscaler.autoscaling/catalog created
ingress.networking.k8s.io/boxoffice created
kubectl -n boxoffice get pods -o wideNAME READY STATUS RESTARTS AGE NODE
catalog-8685666c5-5bv8t 1/1 Running 2 (45s ago) 3m31s boxoffice-worker
catalog-8685666c5-9z7n6 1/1 Running 2 (2m ago) 3m31s boxoffice-worker2
db-0 1/1 Running 0 3m31s boxoffice-worker
redis-6757b9b5ff-9klb4 1/1 Running 0 3m31s boxoffice-worker
curl -s -o /dev/null -w '%{http_code}\n' localhost/catalog/events/1200
Why did catalog restart twice?

Kubernetes has no depends_on: all pods start at once. catalog started before PostgreSQL accepted connections, Flyway failed, the JVM exited, and Kubernetes restarted it with an exponential back-off until the database was there (kubectl -n boxoffice logs deploy/catalog --previous shows the first failure). This is eventual convergence: instead of ordering the startup, every component retries until its dependencies exist. It also means the first start can be slow β€” the startup probe allows up to 5 minutes because the first successful start also runs the Flyway migrations of lab 1.

Watch the controllers work

kubectl -n boxoffice delete pod -l app=catalog --wait=false
kubectl -n boxoffice get pods -l app=catalog -w

The Deployment's ReplicaSet sees 0 of 2 pods and creates two new ones immediately. Time how long until both are 1/1 Running again: that is your MTTR (mean time to repair) for a pod crash β€” dominated by JVM startup.

docker stop boxoffice-worker2
kubectl get nodes -w            # NotReady after about a minute
kubectl -n boxoffice get pods -o wide -w
docker start boxoffice-worker2

Pods on the lost node are replaced on the remaining nodes β€” but only after the node is declared unhealthy and the pods' tolerations expire (about 5 minutes by default). Detection time dominates recovery, exactly as in the availability math.

kubectl -n boxoffice rollout restart deployment/catalog
kubectl -n boxoffice rollout status deployment/catalog

With maxSurge: 1 and maxUnavailable: 0, Kubernetes starts one new pod, waits until it is ready, then terminates one old pod, and repeats. In step 5 you will run k6 during a rollout, with and without the readiness probe and the preStop pause, and count the errors.

Graceful shutdown, step by step

When a pod is terminated: (1) it is removed from the Service endpoints and, in parallel, (2) the preStop hook runs (5 s sleep), then (3) SIGTERM reaches the JVM, which with server.shutdown=graceful stops accepting and finishes in-flight requests. Without the preStop pause, step 3 can happen before every node's proxy has stopped sending traffic to the pod β€” and those requests fail with connection refused.

Your turn: the other services

Deploy the rest of BoxOffice the same way β€” one Deployment + Service each, same probes, same envFrom. The table summarizes what changes:

Service Port Extra configuration Notes
auth 8081 JWT_ISSUER, ADMIN_EMAIL/ADMIN_PASSWORD (Secret) private key in a Secret
gateway 8080 AUTH_JWKS_URI, CATALOG_URL, ORDER_URL… (http://<service>:<port>) then point the Ingress at gateway instead of catalog
inventory 8083 DB_REPLICA_HOST a second StatefulSet for the replica, or skip the replica here
order, notification 8084, 8086 KAFKA_BOOTSTRAP_SERVERS Kafka on Kubernetes is usually run by an operator (e.g. Strimzi); optional in this lab

Next: Autoscaling