Overview

It is 09:59 on the day a stadium tour goes on sale on BoxOffice. Two million fans are refreshing the event page; 60 thousand seats exist. In the next ten minutes the platform must render the catalog for millions of page views, hold and sell seats without double booking, charge cards through an external payment provider, generate 60 thousand tickets and e-mails, and release seats whose holds expired. At 03:00 the same platform serves a few hundred requests per second.

The edge (class 4) decides where each request goes. This class is about what happens when it arrives: where the code runs, how it is split into services, how those services talk, what runs outside the request, how the fleet grows and shrinks on its own, and how to depend safely on systems you do not control — the payment provider, the e-mail provider, any third-party API.

Roadmap

1. This overview — the compute palette, stateless app servers, horizontal scaling and how many replicas you need.
2. Services — monolith, modular monolith and microservices; DDD bounded contexts; Conway's law; the architecture inside a service; auth, search, notifications and analytics as services.
3. API styles — REST, gRPC, GraphQL and asynchronous APIs for the same BoxOffice resource; contracts, OpenAPI and versioning.
4. Async work — workers and background jobs, job design, schedulers, distributed locks and leader election.
5. Serverless & autoscaling — functions as a service, cold starts and concurrency; autoscaling policies, HPA and why reactive scaling lags a flash sale.
6. External integrations — 3rd-party APIs, payment and e-mail providers: webhooks with signatures, idempotency keys, retries, reconciliation and provider outages.
7. Quiz — check your understanding.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab

The compute palette

In the System Design Playground, the COMPUTE category groups every component that runs your code. Each one is a different answer to the question "where does this piece of logic execute, and how does it scale?":

The rest of this page is about the most common of them — the app server — and the property that makes everything else possible: being stateless.

Stateless app servers

A server is stateless when it keeps nothing between requests that another replica would need: any replica can serve any request, because everything that must survive lives outside the process — in a database, a cache, an object store or the client itself. That one property is what lets a load balancer send each request anywhere, lets you add replicas in seconds, and lets you kill a replica without losing anything.

flowchart LR
    fan([Fan]) --> lb["Load balancer"]
    lb --> a1["catalog #1"]
    lb --> a2["catalog #2"]
    lb --> a3["catalog #3"]
    a1 --> redis[("Redis<br/>sessions, cache")]
    a2 --> redis
    a3 --> redis
    a1 --> pg[("PostgreSQL")]
    a2 --> pg
    a3 --> pg
    a3 --> s3[("Object store<br/>images, PDFs")]

The replicas are interchangeable boxes; the state is in the shared stores on the right. Every kind of state that tends to "stick" to a server has a standard new home:

This is factor VI of the Twelve-Factor App: "execute the app as one or more stateless processes"¹. It does not mean the system has no state — it means the state is concentrated in a few components built to hold it, which you then scale with different techniques (replicas, partitions, caches: classes 6 and 7).

/** Which routes are open, which need a valid token, which need the admin scope. */
@Bean
SecurityWebFilterChain security(ServerHttpSecurity http) {
    return http
            .csrf(ServerHttpSecurity.CsrfSpec::disable)        // stateless API, no cookies
            .authorizeExchange(ex -> ex
                    // open: getting a token, and the public keys to check one
                    .pathMatchers("/auth/register", "/auth/login", "/auth/.well-known/**").permitAll()
                    // open: browsing the catalog (HEAD is a GET without the body)
                    .pathMatchers(HttpMethod.GET, "/catalog/**").permitAll()
                    .pathMatchers(HttpMethod.HEAD, "/catalog/**").permitAll()
                    // admin only: changing the catalog (scope claim "user admin")
                    .pathMatchers("/catalog/**").hasAuthority("SCOPE_admin")
                    // health, metrics, routes: the edge never forwards /actuator from outside
                    .pathMatchers("/actuator/**").permitAll()
                    // everything else (and every future route): any valid token
                    .anyExchange().authenticated())
            .oauth2ResourceServer(oauth -> oauth.jwt(Customizer.withDefaults()))
            .build();
}

Scaling up or out

There are two ways to give a service more capacity:

App servers are scaled out almost by default, because statelessness makes it trivial. Databases are usually scaled up first, then with replicas and partitions — which is why the data tier, not the app tier, is where most bottlenecks end up.

How many replicas?

The question every design has to answer with a number. Start from one replica's capacity, measured with a load test (hands-on 2), and the peak load from your estimate (class 2):

Why a target utilization (typically 50–70%)? Because latency is not linear in load. As a server approaches 100% busy, requests start to queue, and waiting time explodes — the "knee" of the curve (queueing). The target leaves headroom for bursts and for losing a replica.

The chart below makes the knee visible. It models each replica as a 10-core server with 20 ms of work per request (an M/M/10 queue) and spreads 4,000 requests/s evenly over N replicas:

With 8 replicas the queue grows without limit (100% busy). With 9 replicas (89% busy) the mean latency is about 31 ms — 60% above the 20 ms of real work; from about 12 replicas on, adding more barely changes the mean. And the tail (p99) degrades much earlier and much faster than the mean (percentiles) — another reason to keep utilization moderate.

Try it in the playground

Open the scale-out scenario: a load balancer in front of N app replicas.

1. Run the load profile with 2 replicas and note p99 and the error rate.
2. Add replicas one at a time and watch p99 fall until it flattens — the point where queueing is gone.
3. Turn chaos to medium to kill replicas at random: health checks remove dead replicas from the pool, and the remaining ones absorb the load only if you left headroom.

Open the full simulator

A replica's life cycle

Scaling out means replicas are created and destroyed all the time — by deploys, by the autoscaler, by node failures. A replica that behaves well during its life cycle is as important as a stateless one:

stateDiagram-v2
    [*] --> Starting : scheduled on a node
    Starting --> Ready : startup + readiness probe pass
    Ready --> NotReady : readiness fails (e.g. DB unreachable)
    NotReady --> Ready : readiness passes again
    Ready --> Draining : SIGTERM (deploy, scale-in)
    Draining --> [*] : in-flight requests done
    Ready --> [*] : crash / liveness fails → restart

• Start fast, warm up before taking traffic. A JVM service may need several seconds to start; the readiness probe should pass only once connection pools are open and caches are warm. Slow starts make autoscaling slow (serverless & autoscaling).
• Shut down gracefully. On SIGTERM, stop accepting new requests, finish the in-flight ones, then exit. In Spring Boot, server.shutdown=graceful with spring.lifecycle.timeout-per-shutdown-phase=20s does this.
• Be disposable (Twelve-Factor IX): nothing important should depend on a replica living long. Health checks (liveness, readiness, startup) are covered in reliability.

Compute shapes

The same stateless service can run on very different substrates. The choice changes who manages what, how fast you scale and how you pay:

Class 11 (cloud-native) covers containers and Kubernetes in depth; this class covers the decisions that depend on the workload: serverless and autoscaling.

The compute concept map

Every compute component answers where a piece of logic runs and how it scales; the external components are the logic you call but do not run. The map links each palette component (green) to the concepts it relies on (blue):

flowchart LR
    app["app"]:::comp -.- c1["stateless, scale out,<br/>replica life cycle"]:::tech
    app -.- c2["REST, gRPC, GraphQL,<br/>async APIs"]:::tech
    app --> queue["queue"]:::comp
    queue --> worker["worker"]:::comp
    worker -.- c3["idempotent jobs,<br/>Little's law, DLQ"]:::tech
    scheduler["scheduler"]:::comp --> queue
    scheduler -.- c4["CronJob, locks,<br/>leader election"]:::tech
    serverless["serverless"]:::comp -.- c5["cold start, concurrency,<br/>autoscaling lag"]:::tech
    app --> shared["auth · search ·<br/>notifications · analytics"]:::comp
    shared -.- c6["bounded contexts,<br/>event-fed, own their data"]:::tech
    app --> ext["payment · email ·<br/>thirdparty"]:::comp
    ext -.- c7["idempotency keys, webhooks,<br/>reconciliation"]:::tech
    classDef comp fill:#1e9e5a22,stroke:#1e9e5a
    classDef tech fill:#2f6fed22,stroke:#2f6fed

Related

• Hands-on: hands-on 1 — first service (the catalog REST API and its OpenAPI contract), hands-on 3 — edge (three replicas behind Nginx, scale-out measurement), hands-on 4 — gateway & identity (one entry point for every service, the token checked once, no session in any replica), hands-on 7 — async (workers, outbox, idempotent consumer), hands-on 9 — resilience (order → payment with timeouts and circuit breaker), hands-on 10 — Kubernetes (HPA under a k6 load).
• Exercises: Kubernetes autoscaling, event-driven.
• Case studies: ticketing (BoxOffice flash sale), payment system, notification system.
• Component catalog: compute and external.

Key takeaways

What breaks next

Stateless replicas are cheap to add because they pushed every piece of state somewhere else, and for BoxOffice that somewhere is PostgreSQL: holds, orders and users now live in the one database that every replica shares. At 10:00 it receives about 33,000 hold attempts per second against the ~1,100 it can serve, and two fans who click seat A-12 in the same millisecond reach two different replicas, so no lock in a replica's memory can protect the seat. Choosing, using and scaling the store that must say "one seat, one hold" is class 6 — Databases.