System Architecture · Insper

Back-of-the-Envelope Estimation

Numbers, queues, percentiles and nines — the math behind every design decision

Can one PostgreSQL take the on-sale?

"No, we need a distributed database." — "Of course it can, Postgres is fast."
What do you need to know to settle this in five minutes?

  • Demand: how many hold attempts per second at 10:00?
  • Cost of one hold: how long does it keep the database busy?
  • Safe operating point: how full can the database run before latency explodes?
  • Experience: what do the unlucky fans see, and how often is it down?

Why estimate?

Decisions change by orders of magnitude — you need to know which side of a threshold you are on.

If the estimate says… …the decision is
40 GB of orders per year one primary is fine — don't shard
1,200 reads/s, 99% repeated cache them
100 k hold attempts/s on 60 k seats queue fans in a waiting room
8 Gbit/s of images serve from a content delivery network (CDN)
one sync dependency at 99.9% the service can't promise more — take it off the path

The method

  1. Name the decision — "do we need to shard?", not "how much data?".
  2. State assumptions out loud — round numbers.
  3. Powers of ten — keep one significant digit.
  4. Sanity-check against a number you know.
  5. Conclude — the design implication, and the most sensitive assumption.

1 day ≈ 10⁵ s · 1 M/day ≈ 12/s · 1 year ≈ 3 × 10⁷ s · 2¹⁰ ≈ 10³ · 1 GB/s = 8 Gbit/s

A complete capacity check

demand2 M fans / 60 s ≈ 33,000 holds/s (more with re-clicks)
capacity8 slots / 5 ms = 1,600/s · at ≤ 70% ≈ 1,100/s
→
verdicta 30× gap: no tuning closes it
→
designa waiting room admits ~1,000/s
→
result~30 k winning fans in ~30 s · hold 99th percentile (p99) ≈ 25 ms

One PostgreSQL primary is enough if demand is shaped; a "distributed database" would not remove the hot-row contention. The key number — 5 ms per hold — is an assumption: lab 2 measures it.

See it in the playground

monolith · breakpoint profilethroughput grows, then latency bends (the knee), errors come late. Capacity = the highest rate with p99 inside the target.
boxoffice-onsale2 M fans against one database of 8 connections × 5 ms: eighty inventory replicas change nothing. The hints build the waiting room that keeps it near 63%.

Latency numbers (2020s, orders of magnitude)

Operation Latency
Level-1 (L1) cache reference ~1 ns
Main memory reference ~100 ns
Random 4 KB read, NVMe solid-state drive (SSD) ~20–100 µs
Round trip inside a zone ~0.1–0.5 ms
Redis GET / primary-key query, same zone ~0.2–1 ms
Round trip between zones ~1–2 ms
Hard disk drive (HDD) seek ~5–10 ms
São Paulo ↔ United States east coast round trip ~110–130 ms

Light in fibre: ~5 µs per km → ~1 ms of round trip per 100 km. Sources: Dean (2009), Scott, Gregg.

What the table teaches

  • Random-access memory (RAM) ≈ 1,000× faster than SSD for random reads → caches.
  • Inside a data centre, the network costs more than the work → batch round trips (MGET, pipelining).
  • Across regions, physics wins → no sync cross-region calls on the critical path; serve from nearby.
  • Sequential beats random → logs, Kafka, batch processing.

Seat map with 60,000 seats: 60,000 queries ≈ 30 s; one query ≈ 20–40 ms; one cached blob ≈ 1–2 ms. Same data, four orders of magnitude.

Requests per second

queries per second (QPS) = daily active users × actions / 86,400 · peak = QPS × peak factor

Normal day

  • 2 M fans × 20 pages = 4 × 10⁷/day
  • ≈ 400 reads/s, peak ×3 ≈ 1,200
  • writes ≈ 5/s → read/write ≈ 100:1

On-sale spike

  • 2 M fans × 3 requests in 60 s
  • ≈ 100,000 req/s at the edge
  • ~80× the normal peak, for minutes

Size for the peak — and estimate spikes separately.

Storage and bandwidth

storage = items × size × retention × replication (× index overhead) · bandwidth = QPS × size

BoxOffice data Per year Where
Orders: 100 k/day × 1 KB ~37 GB Structured Query Language (SQL) database — no sharding needed
Ticket PDFs (Portable Document Format): 200 k/day × 100 KB ~7 TB Object Store (or generate on demand)
Images at peak: 1,200 views/s × 1 MB — ~10 Gbit/s → CDN

Bits vs bytes: a 10 Gbit/s link carries 1.25 GB/s.

The vocabulary of a queue

arrivalsλ requests/s · 1,000 holds/s
→
queuewaiting time Wq
→
c serversservice time S = 5 ms · c = 8 connections
→
completionsX requests/s (= λ when stable)
  • Utilization ρ = λ · S / c = 1,000 × 0.005 / 8 = 62.5%.
  • Response R = Wq + S; L = requests waiting or in service.
  • A "server" is anything that serves: a core, a thread, a connection, a payment application programming interface (API).

Little's Law

L = λ × W — in flight = arrival rate × time in the system

Form Question BoxOffice
L = λW how many in flight? 1,000 holds/s × 40 ms = 40 in flight
W = L/λ how long is the wait? 2 M in line ÷ 1,000/s = 33 min
λ = L/W what throughput? 8 connections ÷ 5 ms = 1,600 holds/s

Holds for any stable system, any distribution, any number of servers.

Little's Law at work

  • Cascading failure: 100 payments/s × 0.3 s = 30 threads waiting. Provider slows to 3 s → 300 threads → a 200-thread pool is exhausted, every route fails.
  • Load-test lie detector: closed loop, N = X × (R + Z). 100 virtual users (VUs), R = 50 ms → max 2,000 req/s; if R grows to 500 ms → 200 req/s. The test backs off exactly when the system struggles.

Pools: size them with L = λW plus headroom — a pool of 500 database connections adds no capacity, only contention.

Utilization and capacity

U = X × S · capacity X_max = c / S (c parallel servers, S service time)

  • 1,000 holds/s × 5 ms = 5 busy cores → 62.5% of 8.
  • The resource with the highest utilization is the bottleneck: it caps the whole system.

Why not run it at 95%?

The knee: R = S / (1 − ρ)

Utilization ρ Mean R p99 (M/M/1 ≈ 4.6 R) S = 5 ms
50% 2 S 9 S 10 ms
70% 3.3 S 15 S 17 ms
80% 5 S 23 S 25 ms
90% 10 S 46 S 50 ms (p99 230 ms)
95% 20 S 92 S 100 ms (p99 460 ms)
99% 100 S 460 S 500 ms (p99 2.3 s)

Random arrivals queue up in bursts; near saturation there are no quiet moments to catch up. The tail explodes first.

Play the knee

The simulator on the queueing page: holds wait in one queue for c servers, S = 5 ms, λ = ρ · c / S.

Experiment (60 s of model time) Mean R p99
1 server, ρ = 70%, random arrivals 18.2 ms 77 ms 🟢
1 server, ρ = 92% 68.3 ms 285 ms 🔴
1 server, 92%, steady arrivals (a waiting room) 31.7 ms —
1 server, 92%, bursty arrivals 98.8 ms —
8 servers, ρ = 70% — 24 ms 🟢

Utilization, variability and pooling: the three lessons of the curve, one knob each.

Where to operate

Component Target at peak Why
Latency-sensitive (API, DB primary) ~50–70% left of the knee; survive losing a replica
Batch / async workers 80–90%+ throughput and cost matter, not per-job latency
  • Headroom is failover: 4 replicas at 60% → lose one → 80%. At 80% → lose one → 107%.
  • Pooling (one queue, many servers) pushes the knee right.
  • Less variability (steady admission, no slow outliers) means less queueing — Kingman.

When more nodes stop helping

Amdahl: speedup ≤ 1/σ · Universal Scalability Law (USL): C(N) = N / (1 + σ(N−1) + κN(N−1))

  • σ (contention): serial work — one primary, a global lock. 5% serial → never more than 20×.
  • κ (coherency): nodes talking to each other — distributed locks, cache coherence. Throughput peaks at N* = √((1−σ)/κ), then falls.

Shared-nothing scales: BoxOffice partitions inventory by event, so holds for different events never contend.

Why averages lie

  • Latency distributions have a long tail: cache misses, garbage-collection (GC) pauses, locks, retries.
  • Example: mean 55 ms, p50 40 ms, p99 ~500 ms, p99.9 ~1.1 s.
  • Use p50 (typical), p99 / p99.9 (the unlucky — often your best customers).
  • A visit with 50 requests hits ≥ 1 above the p99 with probability 1 − 0.99⁵⁰ ≈ 39%.

Never average percentiles across servers or time — merge the histograms.

Fan-out amplifies the tail

P(user request slow) = 1 − (1 − q)ᴺ

N = 11% of user requests slow
→
N = 20 (BoxOffice search shards)18% slow · at q = 0.1%: 2%
→
N = 10063% slow · at q = 0.1%: 9.5%

q = the probability that one backend call is above its p99 (1%) or p99.9 (0.1%).

"The Tail at Scale": with fan-out, the backend's tail becomes the user's common case (its median from ~70 calls). BoxOffice search queries 20 shards in parallel: 18 % of searches see a shard's p99.

Taming the tail

  • Hedged requests: second copy after the p95 delay → ~5% extra load, much shorter tail (idempotent reads only).
  • Partial results after a deadline; less fan-out; precomputed aggregates.
  • Async everything that need not finish before the response; headroom.
  • Latency budget — checkout p99 1.5 s: edge 50 · order 50 · inventory 100 · payment 1,000 · write 50 · slack 250 ms.

Coordinated omission: a load generator that waits for slow responses stops sending — and the tail vanishes from the results.

Availability and the nines

A = MTTF / (MTTF + MTTR), mean time to failure and to recovery — 720 h / 2 h → 99.72%; auto-recover in 2 min → 99.995%

A per year per 30.44 days per 28 days
99.9% 8.77 h 43.8 min 40.3 min
99.95% 4.38 h 21.9 min 20.2 min
99.99% 52.6 min 4.38 min 4.03 min
99.999% 5.26 min 26.3 s 24.2 s

Each nine = 10× less downtime. Always state the window. 99.9% of 2,000 req/s = 2 failures every second.

Series and parallel

  • Series (all needed): ∏ aᵢ — worse than the weakest link. 0.999⁵ ≈ 99.5%.
  • Parallel (any one is enough): 1 − ∏(1 − aᵢ) — two at 99% → 99.99%.
  • Quorum 2-of-3 at 99%: a³ + 3a²(1 − a) ≈ 99.97%.
BoxOffice checkout Availability
naive: 7 sync links in series, incl. e-mail at 99.5% 🔴 99.19% (~5.8 h/month)
e-mail async 🟡 99.69%
+ second payment provider 🟡 99.79%
+ internal links at 99.99% ✅ 99.95%

Independence is an assumption

  • 3 replicas at 99.5% → "99.99999%"… until one bad release hits all three in 2 minutes.
  • Correlated failures: same zone, same release, same config, a shared dependency (the Domain Name System, DNS; a certificate).
  • Redundancy needs failover that works: detection + switch time × failures per year.

Rule of the extra nine: each critical dependency ~one nine better than the service. Turn hard dependencies into soft ones (cache, fallback, async).

The estimation concept map

Component Question it answers Key concepts
app worker how many instances, threads, connections? peak / (capacity × U) + spares, L = λW
sql can one primary take it? U = X × S, the knee, serial fraction (hot rows)
cache how much memory? how many misses? latency numbers, the hot set
cdn objstore how many Gbit/s and TB? bandwidth = QPS × size, retention × replication
queue how long is the line? W = L / λ, admission rate
search (shards) why is the user's p99 worse? 1 − (1 − q)ᴺ, hedging, latency budgets
lb healthcheck payment how available is the path? series ∏aᵢ, parallel 1 − ∏(1 − aᵢ), MTTR

Estimation adds no box to the palette — it puts a number on every box, and tells you which number to measure first.

Takeaways

  1. Estimate to pick a side of a threshold: decision → assumptions → powers of ten → check → conclusion.
  2. RAM ~100 ns, SSD ~100 µs, zone ~0.5 ms, continent ~100+ ms; 1 day ≈ 10⁵ s.
  3. Little's Law L = λW and U = X·S size pools, fleets and queues.
  4. Latency ~ S/(1 − ρ): run latency-critical parts at ~50–70%.
  5. Amdahl and USL: coordination caps scaling; partition to share nothing.
  6. Use percentiles; fan-out turns rare slowness into common slowness.
  7. Series multiply, parallel 1 − ∏(1 − a); failures are often correlated.
  8. Next: a 30× gap means many machines — and a network between them that loses, delays and duplicates (class 3).

Outline: (1) why estimate and the method; (2) numbers every engineer should know; (3) QPS, storage, bandwidth; (4) Little's Law, utilization and the knee; (5) Amdahl and USL; (6) percentiles and tail amplification; (7) availability math.

Each box uses a tool from later in the class: demand (numbers), capacity (queueing), the 25 ms p99 (M/M/c, queueing), and the availability of the chain (availability math).