a. Overview

In lab 2 you found the knee of a single catalog instance. The simplest way to move it is horizontal scaling: more identical, stateless copies of the service behind a load balancer. The same box also solves a second problem — with one instance, a crash or a deploy takes BoxOffice down. With three, one can fail while the others keep serving. This lab builds the edge of the system: the single public entry point that every request crosses first.

Before you start

docker compose version        # Docker Compose v2.x (or newer)
java -version                 # 21 or newer
mvn -v                        # Apache Maven 3.9+
k6 version                    # k6 v1.0+
uv --version                  # uv (runs the Python helper scripts)

Start state — the end of lab 2:

📁 boxoffice/
├──  docker-compose.yaml        # db + catalog
├── 📁 load/  hello.js smoke.js load.js stress.js
├── 📁 results/  lab0.md lab1.md lab2.md lab2/ ...
├── 📁 scripts/  check-env.sh sweep.sh summarize.py watch-metrics.sh
└── 📁 services/catalog/

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab

Before you measure: the playground

Try it: the scale-out scenario has a load balancer in front of app replicas. Run it with the default replicas, then change the number of app servers (1 → 2 → 4) and compare throughput and p99. Finally, kill one replica while the load runs: how long until the balancer stops sending it traffic? Note your predictions for the real system — does 3× the replicas give 3× the throughput?

What you will build

flowchart LR
    k6([k6 / fans]) -->|":80"| edge["edge<br/>Nginx 1.30"]
    edge -->|round robin| c1["catalog-1<br/>1 CPU"]
    edge --> c2["catalog-2<br/>1 CPU"]
    edge --> c3["catalog-3<br/>1 CPU"]
    c1 --> db[("db<br/>PostgreSQL")]
    c2 --> db
    c3 --> db

Only the edge publishes a port (80). The replicas are reachable only inside the Compose network — the same shape as a public load balancer in front of private instances in a cloud VPC (class 12).

1. Make the replicas visible

When three replicas answer, you want to know which one answered. Add this filter to catalog; it stamps every response with the container's hostname:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

package boxoffice.catalog;

import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;

/** Adds "X-Instance: catalog-2" to every response: which replica answered? */
@Component
public class InstanceHeaderFilter extends OncePerRequestFilter {

    // Docker sets HOSTNAME to the container's hostname (catalog-1, catalog-2, ...)
    private final String instance = System.getenv().getOrDefault("HOSTNAME", "local");

    @Override
    protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response,
                                    FilterChain chain) throws ServletException, IOException {
        response.setHeader("X-Instance", instance);
        chain.doFilter(request, response);
    }
}

That is the only code change in this lab. Everything else is configuration — a good sign: a stateless service scales out without touching its code.

2. Three replicas in Compose

Replace your docker-compose.yaml. The x-catalog block is a YAML anchor: one definition, reused by catalog-1, catalog-2 and catalog-3; each only adds its own hostname.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70

name: boxoffice

# settings shared by the three catalog replicas (a YAML anchor)
x-catalog: &catalog
  build: ./services/catalog
  image: boxoffice-catalog
  environment:
    DB_HOST: db
    DB_PORT: 5432
    DB_NAME: ${DB_NAME:-boxoffice}
    DB_USER: ${DB_USER:-boxoffice}
    DB_PASSWORD: ${DB_PASSWORD:-boxoffice}
    DB_POOL_SIZE: ${DB_POOL_SIZE:-10}
  depends_on:
    db:
      condition: service_healthy
  deploy:
    resources:
      limits:
        cpus: "1.0"
        memory: 768M
  healthcheck:
    test: ["CMD-SHELL", "wget -qO- http://localhost:8082/actuator/health/readiness || exit 1"]
    interval: 10s
    timeout: 3s
    start_period: 40s
    retries: 3

services:

  db:
    image: postgres:17
    environment:
      POSTGRES_DB: ${DB_NAME:-boxoffice}
      POSTGRES_USER: ${DB_USER:-boxoffice}
      POSTGRES_PASSWORD: ${DB_PASSWORD:-boxoffice}
    ports:
      - "5432:5432"
    volumes:
      - db-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 5s
      timeout: 3s
      retries: 10

  # three identical replicas; no "ports:" — only the edge is reachable from outside
  catalog-1:
    <<: *catalog
    hostname: catalog-1
  catalog-2:
    <<: *catalog
    hostname: catalog-2
  catalog-3:
    <<: *catalog
    hostname: catalog-3

  edge:
    image: nginx:1.30-alpine
    ports:
      - "80:80"
    volumes:
      - ./infra/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - catalog-1
      - catalog-2
      - catalog-3

volumes:
  db-data:

3. The edge: nginx.conf

Nginx¹ reads one configuration file. Every directive below appears in it, in the same order.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68

# BoxOffice edge: reverse proxy + load balancer + rate limiter (lab 3)
user  nginx;
worker_processes  auto;

events {
    worker_connections  4096;
}

http {
    # one line per request: who served it, and how long the upstream took
    log_format  edge  '$remote_addr "$request" $status rt=$request_time '
                      'upstream=$upstream_addr ut=$upstream_response_time id=$request_id';
    access_log  /var/log/nginx/access.log  edge;

    # Docker's embedded DNS: lets nginx start even if a replica is down, and
    # notice replicas that come back (the "resolve" flag below, nginx >= 1.27.3)
    resolver  127.0.0.11  valid=10s  ipv6=off;

    upstream catalog {
        zone catalog 64k;                 # shared memory, required by "resolve"
        # least_conn;                     # try it: pick the replica with fewest active requests
        server catalog-1:8082 resolve max_fails=3 fail_timeout=10s;
        server catalog-2:8082 resolve max_fails=3 fail_timeout=10s;
        server catalog-3:8082 resolve max_fails=3 fail_timeout=10s;
        keepalive 64;                     # reuse TCP connections to the replicas
    }

    # rate limiting per API key: 20 req/s per key, bursts of up to 40.
    # Requests WITHOUT an X-Api-Key header have an empty key and are not limited
    # (in production you would also limit them per client IP).
    limit_req_zone  $http_x_api_key  zone=per_key:10m  rate=20r/s;
    limit_req_status  429;

    server {
        listen  80;

        # the edge's own liveness: does not touch any backend
        location = /healthz {
            access_log  off;
            return  200  "ok\n";
        }

        location /catalog/ {
            limit_req  zone=per_key  burst=40  nodelay;

            proxy_pass  http://catalog;
            proxy_http_version  1.1;
            proxy_set_header  Connection         "";      # keep upstream connections alive
            proxy_set_header  Host               $host;
            proxy_set_header  X-Real-IP          $remote_addr;
            proxy_set_header  X-Forwarded-For    $proxy_add_x_forwarded_for;
            proxy_set_header  X-Forwarded-Proto  $scheme;
            proxy_set_header  X-Request-Id       $request_id;
            add_header        X-Request-Id       $request_id always;

            # passive health checking: a failed attempt is retried on another replica
            proxy_connect_timeout  1s;
            proxy_read_timeout     5s;
            proxy_next_upstream    error timeout http_502 http_503 http_504;
            proxy_next_upstream_tries  2;
        }

        # nothing else is public (no /actuator from the internet)
        location / {
            return  404;
        }
    }
}

4. Run it and watch the distribution

docker compose up --build -d --remove-orphans   # removes the old single 'catalog' container of lab 1
docker compose ps                       # catalog-1..3 healthy, edge running
curl -s localhost/healthz               # the edge itself
for i in $(seq 1 9); do curl -s -o /dev/null -D - localhost/catalog/events/1 | grep -i x-instance; done | sort | uniq -c

Round robin: each replica gets one request in three. Compare rt (total time at the edge) with ut (time spent in the replica): the difference is the cost of the extra hop.

5. Failures: stop, freeze, recover

Run a steady load in one terminal and break things in another:

# terminal 1: 2 minutes at 150 req/s through the edge
BASE_URL=http://localhost k6 run -e RATE=150 -e DURATION=2m load/load.js

# terminal 2, while it runs:
docker compose stop catalog-2          # a crash: connections are refused immediately
sleep 20
docker compose start catalog-2         # it comes back (startup takes ~10 s)
sleep 30
docker compose pause catalog-3         # a freeze (GC pause, stuck disk): connections hang
sleep 20
docker compose unpause catalog-3

What to look for in the k6 summary and in docker compose logs edge:

You should see something like this at the end of the k6 run (a run for this lab: not a single error, but the pause shows up in the tail; p99 can be anywhere from ~1 s to ~5 s, depending on how many requests the freeze caught):

    http_req_duration..............: avg=48.31ms med=1.33ms p(95)=25.56ms p(99)=1s       max=5.05s
    http_req_failed................: 0.00%  0 out of 14990

The lesson: a dead replica is easy; a slow one is dangerous. Timeouts decide how much a slow replica costs you. Nginx open source only has passive checks² (it learns from failed real requests); active checks — probing /actuator/health/readiness every few seconds — exist in Nginx Plus, HAProxy, Envoy, cloud load balancers and Kubernetes readiness probes (lab 10).

6. Rate limiting

A partner integration that polls catalog 50 times per second can starve everybody else. The edge gives each API key 20 req/s with a burst of 40; above that, it answers 429 Too Many Requests without bothering the replicas.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

// Lab 3: two API clients share the edge. "greedy" sends 50 req/s, "polite" 10 req/s;
// the edge allows 20 req/s per X-Api-Key (burst 40). Who gets 429?
//   k6 run load/ratelimit.js
import http from 'k6/http';
import { check } from 'k6';

const BASE_URL = __ENV.BASE_URL || 'http://localhost';

export const options = {
  scenarios: {
    greedy: {
      executor: 'constant-arrival-rate', rate: 50, timeUnit: '1s', duration: '30s',
      preAllocatedVUs: 20, env: { API_KEY: 'partner-greedy' },
    },
    polite: {
      executor: 'constant-arrival-rate', rate: 10, timeUnit: '1s', duration: '30s',
      preAllocatedVUs: 10, env: { API_KEY: 'partner-polite' },
    },
  },
  thresholds: {
    // listed only so that k6 prints one line per scenario and status
    'http_reqs{scenario:greedy,status:200}': ['count>=0'],
    'http_reqs{scenario:greedy,status:429}': ['count>=0'],
    'http_reqs{scenario:polite,status:200}': ['count>=0'],
    'http_reqs{scenario:polite,status:429}': ['count>=0'],
  },
};

export default function () {
  const res = http.get(`${BASE_URL}/catalog/events/1`, { headers: { 'X-Api-Key': __ENV.API_KEY } });
  check(res, { 'answered 200 or 429': (r) => r.status === 200 || r.status === 429 });
}

k6 run load/ratelimit.js

You should see (from a run for this lab — 30 s at 50 and 10 req/s):

    http_reqs{scenario:greedy,status:200}
    ✓ 'count>=0' count=640
    http_reqs{scenario:greedy,status:429}
    ✓ 'count>=0' count=843
    http_reqs{scenario:polite,status:200}
    ✓ 'count>=0' count=299
    http_reqs{scenario:polite,status:429}
    ✓ 'count>=0' count=0

Expected shape (counts vary): the polite client (10 req/s) gets only 200s; the greedy one (50 req/s) gets about 20 200s per second and the rest 429. Nginx implements a leaky bucket: requests drain at rate; burst is the size of the bucket; nodelay serves the burst immediately instead of spacing it out.

7. Measure it: does 3× the replicas give 3× the throughput?

Use the sweep of lab 2, now through the edge, with two different workloads — the answer depends on which resource is the bottleneck:

• Details only (LIST_SHARE=0): primary-key lookups, cheap for PostgreSQL. The limit is the CPU of each replica (1 core).
• The lab 2 mix (LIST_SHARE=0.2): 20% listings, each scanning and sorting thousands of rows. The limit is PostgreSQL, which all replicas share.

First with one replica (stop the other two; Nginx removes them), then with three:

# 1 replica behind the edge
docker compose stop catalog-2 catalog-3 && sleep 15   # Nginx re-resolves the names every 10 s (resolver valid=10s)
LIST_SHARE=0 BASE_URL=http://localhost OUT=results/lab3-1replica-detail bash scripts/sweep.sh 300 600 900 1200 1500
BASE_URL=http://localhost OUT=results/lab3-1replica-mix bash scripts/sweep.sh 100 200 300 400 500

# 3 replicas behind the edge: start them and WARM THEM UP before measuring
docker compose start catalog-2 catalog-3 && sleep 30
BASE_URL=http://localhost k6 run -e RATE=300 -e DURATION=2m load/load.js > /dev/null
LIST_SHARE=0 BASE_URL=http://localhost OUT=results/lab3-3replicas-detail bash scripts/sweep.sh 600 1200 1800 2400 3000
BASE_URL=http://localhost OUT=results/lab3-3replicas-mix bash scripts/sweep.sh 100 200 300 400 500

for d in results/lab3-*; do echo "## $d"; uv run scripts/summarize.py $d/rate-*.json; done

Keep docker stats open during each run and note which container is at its limit: catalog-* near 100% (one core each) or db using several cores.

Record in results/lab3.md:

# Lab 3 — scale-out behind the edge

## Sweeps
<the four tables from summarize.py: 1/3 replicas × detail/mix>

## Capacity (highest rate with p99 < 300 ms and no errors)
| Workload | Replicas | Capacity (req/s) | Container at its limit (docker stats) |
|---|---:|---:|---|
| details only | 1 | | |
| details only | 3 | | |
| lab 2 mix | 1 | | |
| lab 2 mix | 3 | | |

Scaling efficiency = capacity(3) / (3 × capacity(1)):
- details only: ____ %
- lab 2 mix: ____ %
- Why are they so different? ____

## Edge overhead
- p50 direct to catalog (lab 2, same rate): ____ ms; p50 through the edge: ____ ms

## Failures (step 5)
- stop catalog-2: errors ____, p99 ____ ms
- pause catalog-3: errors ____, p99 ____ ms; how long until Nginx stopped sending it traffic? ____ s

## Rate limiting (step 6)
| Client | Sent | 200 | 429 |
|---|---:|---:|---:|
| greedy (50 req/s) | | | |
| polite (10 req/s) | | | |

Troubleshooting

Clean up

docker compose start catalog-2 catalog-3     # if you left them stopped after the sweep
docker compose unpause catalog-3 2>/dev/null || true
docker compose down                          # when you are done (the database volume is kept)

Keep everything: lab 4 puts the gateway behind this edge.

Deliverable

• InstanceHeaderFilter, the three-replica docker-compose.yaml and infra/nginx/nginx.conf in your repository.
• The distribution check (9 requests, 3 per replica) pasted in results/lab3.md.
• The failure experiment (stop and pause) described with numbers.
• The rate-limit table.
• Sweeps with 1 and 3 replicas for both workloads, capacities, scaling efficiency of each and the container at its limit.
• Repository tagged lab3; quiz answered.

End state

📁 boxoffice/
├──  docker-compose.yaml        # replaced: db, catalog-1..3 (no host ports), edge on port 80
├──  infra/nginx/nginx.conf     # new: reverse proxy, load balancer, rate limiter
├──  load/ratelimit.js          # new
├──  services/catalog/src/main/java/boxoffice/catalog/InstanceHeaderFilter.java   # new
└──  results/lab3.md  results/lab3-*/

From now on clients use http://localhost (the edge); the replicas are not reachable from your machine.

Next: Lab 4 — Gateway & identity.