Load Balancing

One inventory replica handles, say, 800 holds per second before its latency climbs (an illustrative number — in hands-on 2 you measure the real one for catalog). The stadium sale needs 5 000. The answer is obvious — run seven or eight replicas — but it immediately raises the questions this page answers: who decides which replica gets each request? What happens when one replica dies, or is slow, or is being redeployed? And what if the request needs to land on a specific replica?

A load balancer (LB) is the component that spreads work across a pool of interchangeable servers — the backends, targets or upstreams — so that the pool behaves like a single, larger and more reliable server¹. It is the lb component of the palette, and, in Kubernetes, the ingress component plays the same role for HTTP traffic entering the cluster.

Why load balance?

A load balancer gives you three things at once:

• Scalability: add replicas to handle more traffic (horizontal scaling, or scale-out), instead of buying a bigger machine (vertical scaling, or scale-up).
• Availability: when a replica fails its health checks, the LB stops sending it traffic; users do not notice.
• Operability: you can drain, replace and upgrade replicas one at a time (rolling deploys), with no downtime.

This only works if the replicas are interchangeable — any one of them can serve any request. That is why application servers should be stateless: session data, carts and seat holds live in a shared store (Redis, PostgreSQL), not in the memory of one replica. Statelessness is the precondition for everything on this page.

flowchart LR
    f([Fans]) --> lb{{Load balancer}}
    lb --> a1[inventory-1]
    lb --> a2[inventory-2]
    lb --> a3[inventory-3]
    lb -.->|"health check fails: skipped"| a4["inventory-4 ✗"]
    a1 --> db[(PostgreSQL)]
    a2 --> db
    a3 --> db

L4 versus L7

Load balancers differ in how much of the traffic they understand.

A layer-4 (L4) load balancer works with TCP connections or UDP datagrams. It sees IP addresses and ports, picks a backend when a connection opens, and forwards the bytes of that connection to it. It does not read HTTP: it cannot route by path, it cannot see that one connection carries 1 000 requests, and it does not need to decrypt TLS.

A layer-7 (L7) load balancer speaks HTTP. It terminates the client's connection, reads each request (method, path, headers, cookies), picks a backend per request, and opens (or reuses) its own connection to it. It is a full reverse proxy.

A common production layout uses both: an L4 load balancer (or anycast) at the front for raw capacity and DDoS resilience, spreading connections over a fleet of L7 proxies (Envoy, Nginx) that do TLS, routing and per-request balancing. This is how Google's and Cloudflare's front ends are built.

Balancing algorithms

Once the LB has a list of healthy backends, it needs a rule to pick one. The rule matters most when requests have different costs or backends have different speeds.

Two of these deserve more attention.

Power of two choices

Choosing the least-loaded backend among all of them sounds ideal, but with many load balancer instances (each with a slightly stale view) they all pick the same "least loaded" server at once and overload it — a herd. Picking a backend at random avoids the herd but leaves some servers much busier than others by chance. Mitzenmacher showed that a tiny compromise gets almost the best of both: sample two backends at random and pick the less loaded one². The maximum load drops exponentially compared with pure random choice, and there is no herd, because different LB instances sample different pairs. Envoy, Nginx (random two least_conn), HAProxy and the gRPC libraries implement it; it is the default choice of many modern proxies.

Consistent hashing

Sometimes you want the same key to go to the same backend: a cache server that holds a user's data, a WebSocket server that holds a chat room, a shard of a key-value store. The naive way is server = hash(key) mod N. It works until N changes: going from 5 to 6 servers changes the result of mod for most keys, so almost every key moves — every cache misses at once.

Consistent hashing (Karger et al., 1997)³ places both servers and keys on a circle of hash values (the ring). A key belongs to the first server found walking clockwise from its position.

flowchart LR
    A["node A<br/>position 10"] -->|clockwise| B["node B<br/>position 40"]
    B -->|clockwise| D["new node D<br/>position 55"]
    D -->|clockwise| C["node C<br/>position 75"]
    C -->|"clockwise (wraps to 0)"| A
    k1(["hold:42 → 30"]) -.->|owner| B
    k2(["hold:77 → 50"]) -.->|"owner: was C, now D"| D
    k3(["hold:91 → 90"]) -.->|"owner (wraps)"| A

In the diagram, positions go from 0 to 99: hold:42 hashes to 30 and belongs to B (the first node at or after 30); hold:91 hashes to 90 and wraps around to A. When node D joins at position 55, only the keys between 40 and 55 — such as hold:77 — move from C to D; nothing else changes. In general, when a server is added, it only takes over the keys between itself and its predecessor — about 1/(N+1) of them; when one is removed, only its keys move, to its successor.

With one point per server, the arcs of the ring have very different lengths, so some servers own much more than their share. The fix is virtual nodes⁵: each physical server is placed on the ring many times (100–200 points, for example), so the arcs average out. Virtual nodes also make it easy to give a bigger machine more points (weights) and, when a server leaves, spread its keys over all the others instead of dumping them on one neighbour.

The simulation below hashes 40 000 seat-hold keys onto 5 nodes. On the left, one point per node versus 200 virtual nodes per node; on the right, the fraction of keys that change owner when a 6th node joins:

Two lessons from the figure. With a single point per node, one node owns about half of all keys and another almost none — the ring is badly unbalanced. With 200 virtual nodes, every node is close to the ideal 20%. And when the 6th node arrives, hash mod N remaps about 83% of the keys (5/6 of them, as theory predicts), while the ring remaps about 17% — the 1/6 that the new node must own anyway.

Consistent hashing appears all over system design: Nginx hash $key consistent, Envoy's ring hash and Maglev balancers, memcached clients, Amazon's Dynamo and Cassandra for data partitioning (studied in Databases — partitioning), and Discord-style chat servers that pin a channel to a node.

Service discovery: how the LB knows the replicas

Replicas come and go — autoscaling adds them at 09:58, a deploy replaces them, a crash removes one — and their IP addresses change every time. Before a load balancer can pick a healthy replica, something must tell it which replicas exist right now. That is service discovery, and it has two shapes:

flowchart LR
    subgraph server ["Server-side discovery"]
        C1([gateway]) -->|"inventory.svc"| LB1["load balancer / kube-proxy"]
        LB1 --> R1[inventory-1]
        LB1 --> R2[inventory-2]
        REG1[("registry<br/>(K8s endpoints)")] -.->|"watch"| LB1
    end
    subgraph client ["Client-side discovery"]
        C2([gateway]) -->|"1 · who serves inventory?"| REG2[("registry<br/>(Consul, Eureka)")]
        REG2 -->|"2 · [10.0.1.7, 10.0.2.9]"| C2
        C2 -->|"3 · picks one itself"| R3[inventory-3]
    end

Registration happens either by the platform (Kubernetes adds a pod's IP to the Service's endpoints when its readiness probe passes) or by the instance itself (it registers on start and sends heartbeats; a missed heartbeat removes it). Either way, discovery and health checks are two halves of the same question — which replicas can serve now? — and the registry is a small, strongly consistent store (etcd, Consul's Raft) for the reasons in Distributed Systems — consensus.

Health checks

A load balancer is only as good as its knowledge of which backends are healthy. There are two complementary ways to find out:

Design the health endpoint carefully. Readiness ("can I serve traffic now?") should check what the instance needs to answer requests — but if it checks a shared dependency such as the database, a database blip marks every replica unhealthy at once and the LB has nothing left to route to. The usual compromise: readiness reflects the instance's own state (started, warmed up, not overloaded, not shutting down); dependency failures are handled by timeouts and circuit breakers (Reliability — patterns).

Connection draining

When a replica is removed — scaled in, or replaced by a deploy — cutting it off abruptly fails every request in flight. Connection draining (deregistration delay) fixes this: the LB stops sending new requests to the replica, waits for the in-flight ones to finish (up to, say, 30 s), and only then removes it. In Kubernetes, the pod becomes not ready, is removed from the Service endpoints, and receives SIGTERM; the application should finish its current requests before exiting (Spring Boot: server.shutdown=graceful).

Sticky sessions

Session affinity ("sticky sessions") makes the LB send every request from the same client to the same backend, using a cookie set by the LB or a hash of the client IP. It exists for applications that keep state in memory — a server-side session, an in-process cache, a WebSocket.

Prefer stateless replicas. Use affinity only where the state is inherently local and cheap to rebuild — a WebSocket connection, a local cache that only improves hit ratio — and prefer consistent hashing on a meaningful key (user id, room id) to cookie stickiness, because it survives LB restarts and scales smoothly.

The load balancer must not be the weak point

A single load balancer in front of ten replicas is a single point of failure. How the LB itself stays up depends on who runs it:

• Managed cloud load balancers (ALB/NLB, Google Cloud Load Balancing, Azure Load Balancer) are distributed systems run by the provider across several availability zones; you get a DNS name or an IP, and the provider scales and replaces the underlying machines.
• Self-hosted (Nginx, HAProxy, Envoy on VMs): run at least two, in an active-passive pair sharing a virtual IP (VIP) moved by VRRP (keepalived) when the active one fails, or active-active behind DNS or an L4 tier.
• Anycast at the edge: the same IP announced from many locations (DNS & CDN).

There is also a hierarchy of scopes: a global load balancer (anycast or DNS-based) chooses the region; a regional load balancer chooses the zone and instance; inside Kubernetes, the Service and ingress choose the pod.

flowchart TB
    u([Fans worldwide]) --> g["Global LB / DNS<br/>chooses the region"]
    g --> r1["Regional LB São Paulo<br/>across 3 zones"]
    g --> r2["Regional LB Virginia"]
    r1 --> i1["Ingress / Gateway<br/>in the cluster"]
    i1 --> p1[inventory pods]
    i1 --> p2[order pods]

Try it. Load the scale-out scenario and run the load profile with 2 replicas; note p99 latency and errors. Increase the replica count until p99 stabilises. Then kill one replica while the test runs (chaos medium, or stop it in the panel): watch the health checks remove it, the error spike that lasts until they do, and the remaining replicas absorb the load. Try a different balancing algorithm and compare p99.

Open the full playground

A load balancer in practice: Nginx

Hands-on 3 puts Nginx in front of three replicas of catalog. The core of the configuration is an upstream block — the pool of replicas and the algorithm: round robin by default, or one line such as least_conn;, random two least_conn; (power of two choices) or hash $arg_event consistent; — and a location that proxies to it.

upstream catalog {
    zone catalog 64k;                 # shared memory, required by "resolve"
    # least_conn;                     # try it: pick the replica with fewest active requests
    server catalog-1:8082 resolve max_fails=3 fail_timeout=10s;
    server catalog-2:8082 resolve max_fails=3 fail_timeout=10s;
    server catalog-3:8082 resolve max_fails=3 fail_timeout=10s;
    keepalive 64;                     # reuse TCP connections to the replicas
}
    location /catalog/ {
        limit_req  zone=per_key  burst=40  nodelay;

        proxy_pass  http://catalog;
        proxy_http_version  1.1;
        proxy_set_header  Connection         "";      # keep upstream connections alive
        proxy_set_header  Host               $host;
        proxy_set_header  X-Real-IP          $remote_addr;
        proxy_set_header  X-Forwarded-For    $proxy_add_x_forwarded_for;
        proxy_set_header  X-Forwarded-Proto  $scheme;
        proxy_set_header  X-Request-Id       $request_id;
        add_header        X-Request-Id       $request_id always;

        # passive health checking: a failed attempt is retried on another replica
        proxy_connect_timeout  1s;
        proxy_read_timeout     5s;
        proxy_next_upstream    error timeout http_502 http_503 http_504;
        proxy_next_upstream_tries  2;
    }

Note what is configured besides the algorithm: timeouts toward the backends, keep-alive connections to them, and proxy_next_upstream, which retries a failed request on another replica. Nginx does not retry non-idempotent requests (such as POST) by default — retrying a POST /holds on another replica could create two holds. The timeouts come from measurement, not taste: a read timeout shorter than the healthy p99 turns slow answers into retries, and the lab sets 5 s for a service whose listings take hundreds of milliseconds under load.

Load balancing in Kubernetes

In Kubernetes, three objects share the load-balancing work.

Service — a stable virtual IP and DNS name (inventory.boxoffice.svc.cluster.local) in front of the pods that match a label selector. Only ready pods are listed as endpoints. Traffic to the Service IP is spread across them at L4, by kube-proxy (iptables/IPVS rules) or by an eBPF data plane such as Cilium. A Service of type LoadBalancer also asks the cloud provider for an external L4 load balancer.

apiVersion: v1
kind: Service
metadata:
  name: inventory
spec:
  selector:
    app: inventory
  ports:
    - port: 80
      targetPort: 8080

Ingress — the original API for L7 HTTP routing into the cluster: rules by host and path, TLS termination, all implemented by an ingress controller (Traefik, HAProxy, cloud controllers; the community ingress-nginx controller was very popular but has been retired by the Kubernetes project). The Ingress API is simple but limited: anything beyond host/path routing (header matching, traffic splitting, timeouts) requires controller-specific annotations, which are not portable.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: boxoffice
spec:
  ingressClassName: traefik
  tls:
    - hosts: [api.boxoffice.com]
      secretName: boxoffice-tls
  rules:
    - host: api.boxoffice.com
      http:
        paths:
          - path: /api/holds
            pathType: Prefix
            backend:
              service:
                name: inventory
                port:
                  number: 80

Gateway API — the successor of Ingress, generally available since 2023⁴. It splits the configuration by role, so that each team owns its part:

Features that needed annotations in Ingress are part of the standard: weighted traffic splitting (canaries), header-based routing, request mirroring, timeouts and gRPC routing.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: public
spec:
  gatewayClassName: envoy
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: api.boxoffice.com
      tls:
        certificateRefs:
          - name: boxoffice-tls
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: inventory
spec:
  parentRefs:
    - name: public
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api/holds
      timeouts:
        request: 2s
      backendRefs:
        - name: inventory        # stable version: 95% of requests
          port: 80
          weight: 95
        - name: inventory-canary # new version: 5% of requests
          port: 80
          weight: 5

In hands-on 10 you deploy BoxOffice on a local cluster and expose it through an ingress; canary releases come back in the cloud-native class.

Trade-offs at a glance

How it connects

flowchart LR
    SD["service discovery"] --> LB["load balancer"]
    HC["health checks<br/>(readiness)"] --> LB
    LB --> ALG["algorithm: RR, least conn,<br/>P2C, consistent hashing"]
    ALG --> STATELESS["stateless replicas<br/>→ Compute"]
    ALG --> CH["consistent hashing<br/>→ sharding (class 6)"]
    LB --> HA["LB redundancy, anycast<br/>→ Reliability"]
    LB --> K8S["Service, Ingress, Gateway API<br/>→ Cloud-native"]
    STATELESS --> AS["autoscaling<br/>→ Compute"]
    HC --> OBS["health endpoints<br/>→ Observability"]

Palette components: lb · ingress · app (the replicas behind it) · healthcheck · mesh (client-side balancing in a sidecar). Scenarios: scale-out, chat (long-lived connections). Continues in: Compute — services, Databases — partitioning (consistent hashing for data), Cloud-native — Kubernetes, Hands-on 3.

Key takeaways