2. Deploy BoxOffice
Compose describes containers to start. Kubernetes manifests describe a desired state β "two ready replicas of catalog, reachable as catalog:8082" β and controllers work continuously to make reality match it: a crashed pod is replaced, a dead node's pods are rescheduled, a new image is rolled out pod by pod. This step writes that desired state for the read path of BoxOffice: db, redis and catalog.
Layout: base + overlays
Manifests live in k8s/ of your repository, organized with Kustomize (built into kubectl): a base with everything that is the same in every environment, and one overlay per environment with only what differs (image tags, secrets, replica counts). Lab 11 adds an overlay for the GitOps deployment without touching the base.
k8s/
βββ kind.yaml # the cluster (step 1)
βββ base/
β βββ kustomization.yaml
β βββ namespace.yaml
β βββ config.yaml # ConfigMap: DB_HOST, REDIS_HOST, ...
β βββ db.yaml # PostgreSQL StatefulSet + headless Service
β βββ redis.yaml
β βββ catalog.yaml # Deployment + Service
β βββ catalog-hpa.yaml # step 3
β βββ ingress.yaml # step 4
βββ overlays/
βββ local/
βββ kustomization.yaml # image tag "dev" + lab secrets
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
images:
- name: boxoffice/catalog # the name used in base/catalog.yaml...
newTag: dev # ...and the tag loaded into the local cluster
secretGenerator:
- name: db-credentials # lab only: real secrets never live in Git in plain text
namespace: boxoffice
literals:
- DB_USER=boxoffice
- DB_PASSWORD=boxoffice
The secretGenerator appends a hash of the content to the Secret's name (db-credentials-c67β¦) and rewrites every reference to it: changing a password produces a new Secret and therefore a rollout of the pods that use it β a stale credential cannot linger in a running pod.
Stateful and stateless
# PostgreSQL as a StatefulSet: a stable name (db-0) and its own persistent volume.
# Fine for a lab; in production use a managed database or an operator (e.g. CloudNativePG).
apiVersion: v1
kind: Service
metadata:
name: db
spec:
clusterIP: None # headless: DNS "db" resolves to the pod itself
selector:
app: db
ports:
- name: postgres
port: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: db
spec:
serviceName: db
replicas: 1
selector:
matchLabels:
app: db
template:
metadata:
labels:
app: db
spec:
containers:
- name: postgres
image: postgres:17
ports:
- name: postgres
containerPort: 5432
env:
- name: POSTGRES_DB
valueFrom:
configMapKeyRef: { name: boxoffice-config, key: DB_NAME }
- name: POSTGRES_USER
valueFrom:
secretKeyRef: { name: db-credentials, key: DB_USER }
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef: { name: db-credentials, key: DB_PASSWORD }
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
readinessProbe:
exec:
command: ["sh", "-c", "pg_isready -U \"$POSTGRES_USER\" -d \"$POSTGRES_DB\""]
periodSeconds: 5
resources:
requests: { cpu: 250m, memory: 256Mi }
limits: { cpu: "1", memory: 1Gi }
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 2Gi
apiVersion: v1
kind: Service
metadata:
name: redis
spec:
selector:
app: redis
ports:
- name: redis
port: 6379
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis
spec:
replicas: 1
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
spec:
containers:
- name: redis
image: redis:8-alpine
args: ["--maxmemory", "256mb", "--maxmemory-policy", "allkeys-lru"]
ports:
- name: redis
containerPort: 6379
readinessProbe:
exec:
command: ["redis-cli", "ping"]
periodSeconds: 5
resources:
requests: { cpu: 100m, memory: 128Mi }
limits: { cpu: 500m, memory: 320Mi }
A StatefulSet gives the database what a Deployment does not: a stable identity (db-0) and a PersistentVolumeClaim per replica that survives pod restarts. Deleting the pod keeps the data; deleting the PVC deletes it.
The catalog Deployment
apiVersion: v1
kind: Service
metadata:
name: catalog
spec:
selector:
app: catalog
ports:
- name: http
port: 8082
targetPort: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: catalog
spec:
replicas: 2 # the HPA takes over from here
selector:
matchLabels:
app: catalog
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1 # one extra pod during a rollout...
maxUnavailable: 0 # ...and never fewer ready pods than desired
template:
metadata:
labels:
app: catalog
spec:
terminationGracePeriodSeconds: 30
containers:
- name: catalog
image: boxoffice/catalog:dev
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 8082
envFrom:
- configMapRef:
name: boxoffice-config
- secretRef:
name: db-credentials
env:
- name: SERVER_SHUTDOWN
value: graceful # finish in-flight requests on SIGTERM
resources:
requests: # what the scheduler reserves; HPA percentages are relative to this
cpu: 250m
memory: 512Mi
limits:
cpu: "1"
memory: 768Mi
startupProbe: # the JVM needs time: no liveness kills before it has started
httpGet: { path: /actuator/health/liveness, port: http }
periodSeconds: 5
failureThreshold: 60 # up to 5 min: the first start also runs the Flyway migrations
livenessProbe: # "restart me": only for a process that is stuck
httpGet: { path: /actuator/health/liveness, port: http }
periodSeconds: 10
failureThreshold: 3
readinessProbe: # "send me traffic": removed from the Service while not ready
httpGet: { path: /actuator/health/readiness, port: http }
periodSeconds: 5
failureThreshold: 2
lifecycle:
preStop:
sleep:
seconds: 5 # let the endpoints update before the JVM stops accepting
The three probes answer three different questions, and confusing them causes outages:
| Probe | Question | On failure | Spring Boot endpoint |
|---|---|---|---|
startupProbe | has the process finished starting? | keeps the other probes off; restart after the budget | /actuator/health/liveness |
livenessProbe | is the process stuck beyond repair? | restart the container | /actuator/health/liveness (internal state only) |
readinessProbe | should this pod receive traffic now? | remove from the Service endpoints (no restart) | /actuator/health/readiness |
Never put a dependency in the liveness probe
If liveness checked the database, a database blip would make Kubernetes restart every catalog pod at once β turning a 10-second blip into minutes of outage while JVMs restart. Spring Boot's liveness group checks only the application's own state; that is why lab 1 enabled management.endpoint.health.probes.
requests are what the scheduler reserves on a node (and the base of HPA percentages); limits are the ceiling β above the memory limit the container is killed (OOMKilled), above the CPU limit it is throttled. Lab 1's Dockerfile sizes the heap with -XX:MaxRAMPercentage=75, so the JVM respects the 768 Mi limit.
Apply
configmap/boxoffice-config created
secret/db-credentials-c677754gbf created
service/catalog created
service/db created
service/redis created
deployment.apps/catalog created
deployment.apps/redis created
statefulset.apps/db created
horizontalpodautoscaler.autoscaling/catalog created
ingress.networking.k8s.io/boxoffice createdkubectl -n boxoffice get pods -o wideNAME READY STATUS RESTARTS AGE NODE
catalog-8685666c5-5bv8t 1/1 Running 2 (45s ago) 3m31s boxoffice-worker
catalog-8685666c5-9z7n6 1/1 Running 2 (2m ago) 3m31s boxoffice-worker2
db-0 1/1 Running 0 3m31s boxoffice-worker
redis-6757b9b5ff-9klb4 1/1 Running 0 3m31s boxoffice-workercurl -s -o /dev/null -w '%{http_code}\n' localhost/catalog/events/1200
Why did catalog restart twice?
Kubernetes has no depends_on: all pods start at once. catalog started before PostgreSQL accepted connections, Flyway failed, the JVM exited, and Kubernetes restarted it with an exponential back-off until the database was there (kubectl -n boxoffice logs deploy/catalog --previous shows the first failure). This is eventual convergence: instead of ordering the startup, every component retries until its dependencies exist. It also means the first start can be slow β the startup probe allows up to 5 minutes because the first successful start also runs the Flyway migrations of lab 1.
Watch the controllers work
kubectl -n boxoffice delete pod -l app=catalog --wait=false
kubectl -n boxoffice get pods -l app=catalog -w
The Deployment's ReplicaSet sees 0 of 2 pods and creates two new ones immediately. Time how long until both are 1/1 Running again: that is your MTTR (mean time to repair) for a pod crash β dominated by JVM startup.
docker stop boxoffice-worker2
kubectl get nodes -w # NotReady after about a minute
kubectl -n boxoffice get pods -o wide -w
docker start boxoffice-worker2
Pods on the lost node are replaced on the remaining nodes β but only after the node is declared unhealthy and the pods' tolerations expire (about 5 minutes by default). Detection time dominates recovery, exactly as in the availability math.
kubectl -n boxoffice rollout restart deployment/catalog
kubectl -n boxoffice rollout status deployment/catalog
With maxSurge: 1 and maxUnavailable: 0, Kubernetes starts one new pod, waits until it is ready, then terminates one old pod, and repeats. In step 5 you will run k6 during a rollout, with and without the readiness probe and the preStop pause, and count the errors.
Graceful shutdown, step by step
When a pod is terminated: (1) it is removed from the Service endpoints and, in parallel, (2) the preStop hook runs (5 s sleep), then (3) SIGTERM reaches the JVM, which with server.shutdown=graceful stops accepting and finishes in-flight requests. Without the preStop pause, step 3 can happen before every node's proxy has stopped sending traffic to the pod β and those requests fail with connection refused.
Your turn: the other services
Deploy the rest of BoxOffice the same way β one Deployment + Service each, same probes, same envFrom. The table summarizes what changes:
| Service | Port | Extra configuration | Notes |
|---|---|---|---|
auth | 8081 | JWT_ISSUER, ADMIN_EMAIL/ADMIN_PASSWORD (Secret) | private key in a Secret |
gateway | 8080 | AUTH_JWKS_URI, CATALOG_URL, ORDER_URL⦠(http://<service>:<port>) | then point the Ingress at gateway instead of catalog |
inventory | 8083 | DB_REPLICA_HOST | a second StatefulSet for the replica, or skip the replica here |
order, notification | 8084, 8086 | KAFKA_BOOTSTRAP_SERVERS | Kafka on Kubernetes is usually run by an operator (e.g. Strimzi); optional in this lab |