System Architecture Β· Insper

Compute and services

Where the code runs, how it is split, how it scales β€” and how to depend on providers you do not control

09:59, on-sale day

2 million fans, 60 thousand seats, sale opens at 10:00. At 03:00 the same platform serves a few hundred requests per second.
What must the compute tier do in the next ten minutes?

  • Serve the catalog to millions of page views β€” scale out.
  • Hold and sell seats without double booking β€” state in the right place.
  • Charge cards through an external provider that may time out.
  • Issue 60 k tickets and e-mails β€” background work.
  • Release expired holds, open the sale on time β€” schedulers.
  • Grow from hundreds to tens of thousands of req/s β€” autoscaling.

The compute palette

Component Runs BoxOffice example
App Server request handlers behind a load balancer catalog, inventory, order
Worker jobs pulled from a queue ticket PDFs, e-mail sending
Serverless short functions started per event webhook receiver, image resize
Auth Service login, tokens auth issues JSON Web Tokens (JWTs)
Search full-text queries over an index "coldplay sΓ£o paulo"
Scheduler time-triggered work release holds every 30 s
Notifications fan-out to e-mail / push / SMS "your tickets are here"
Analytics event ingestion and aggregation sales per minute

Stateless app servers

Any replica can serve any request: state lives outside the process.

fans
β†’
load balancer
any β†’
interchangeable replicas
catalog #1
catalog #2
catalog #3
β†’
the state lives here
Redissessions, cache
PostgreSQLholds, orders
object storeimages, ticket PDFs

Seat holds in a HashMap: the next click hits another replica, a restart forgets every hold, two replicas sell the same seat.

Where each kind of state goes

State that sticks to a server Why it breaks scaling New home
HTTP session in memory next request, other replica: "logged out" JWT on the client, or Redis
Uploaded files on local disk other replicas cannot see them object store (S3, GCS, MinIO)
In-process cache each replica has its own stale copy Redis β€” or short times to live (TTLs)
@Scheduled job inside the app 10 replicas run it 10 times CronJob, lock or leader
Counters, rate limits in memory each replica counts only its share Redis INCR, API gateway
WebSocket connections a connection is state keep it; publish through pub/sub

Twelve-Factor VI: the system keeps state β€” concentrated in a few components built to hold it.

How many replicas?

N = ⌈ peak requests per second (RPS) / (RPS per replica Γ— target utilization) βŒ‰ + spare

  • catalog: 10 virtual CPUs (vCPUs) Γ— (1 / 20 ms) = 500 req/s per replica; peak 4,000 req/s.
  • At 100%: 8 replicas β€” zero headroom.
  • At a 65% target: 4,000 / 325 β‰ˆ 12.3 β†’ 13, + 1 spare = 14.
Replicas 9 10 12 14
Utilization 89% 80% 67% 57%
Mean latency (M/M/10) 31 ms 24 ms 21 ms 20.4 ms

Latency explodes near 100% busy (the knee) β€” and p99 degrades earlier than the mean.

Replicas are disposable

  • Start fast; readiness passes only after pools and caches are warm.
  • Graceful shutdown on SIGTERM: stop accepting, finish in-flight, exit.
  • Scale out the stateless tier; scale up (then replicate, partition) the stateful one.
Shape Scales in Pay for Typical use
Virtual machine (VM) minutes provisioned hours legacy, databases
Containers on Kubernetes seconds (pod) to minutes (node) nodes the default for services
Serverless containers seconds, to zero per use APIs with idle periods
Functions as a Service (FaaS) sub-second, to zero invocations + GB-s event glue, webhooks

Monolith β†’ modular monolith β†’ microservices

monolith
one appcatalog Β· inventory Β· order Β· payment
one database, every table shared
β†’
modular monolith
one app, one deploy
module catalog β†’ schema catalog
module inventory β†’ schema inventory
module order β†’ schema order
β†’
microservices
gateway
catalog β†’ catalog DB
inventory β†’ inventory DB
order β†’ order DB
  • Boundaries enforced by discipline β†’ by the build (modules, ArchUnit) β†’ by the network.
  • A modular monolith gets the boundaries and data ownership without paying for the network.

Three shapes compared

Monolith Modular monolith Microservices
Deployable units 1 1 many
Boundaries enforced by discipline the build (modules, ArchUnit) the network
Calls between parts in-process in-process network: can fail, ms
Transactions ACID (atomic, consistent, isolated, durable) across all ACID, tables owned per module sagas, eventual consistency
Scaling whole app whole app per service
Team autonomy πŸ”΄ low 🟑 medium 🟒 high
Operational cost 🟒 lowest 🟒 low πŸ”΄ high

Monolith first: moving a boundary inside one codebase is a refactoring; between services, a migration.

When splitting pays off

Force BoxOffice example
Different scaling profile catalog read-heavy vs inventory write-hot
Team autonomy search ships twice a day; payments monthly
Failure isolation assistant down must not stop ticket sales
Compliance scope only payment touches the provider: smaller Payment Card Industry (PCI DSS) scope
Different technology assistant in Python, core in Java

Distributed monolith: services deployed together, shared tables, long synchronous chains, a shared domain library.

Boundaries with Domain-Driven Design (DDD)

  • Subdomains: core (inventory, waiting room) β†’ build; supporting (catalog, notifications); generic (auth, payments, e-mail) β†’ buy.
  • Bounded context: where one model β€” and one meaning of each word β€” applies.

"Event" in three contexts

  • Catalog: artist, date, venue, images
  • Inventory: seat map, sections, holds
  • Order: event id, seat, price paid

Aggregate rules

  • reference other aggregates by id
  • one transaction changes one aggregate
  • the root enforces the invariants

Context maps and Conway's law

  • Open Host Service: catalog publishes a REST (representational state transfer) API + OpenAPI for any consumer.
  • Customer / Supplier: inventory plans its hold API with order.
  • Anti-Corruption Layer: payment translates the provider's model at the border.

"Organizations produce designs that copy their communication structure." β€” Conway, 1968

  • If boundaries and teams disagree, the teams win.
  • Inverse Conway maneuver: shape stream-aligned teams per capability, plus a platform team.
  • Database per service: others use the API or the events β€” never the tables.

Inside one service: ports and adapters

// core: a port in business terms
public interface PaymentGateway {
    ChargeResult charge(PaymentId id, Money amount,
                        PaymentMethodToken method, IdempotencyKey key);
}
// adapter: the only class that knows the provider's API
class ProviderPaymentGateway implements PaymentGateway { /* translate THEIR states */ }
  • SOLID (five class-design principles), especially Dependency inversion: business code depends on abstractions it owns.
  • Payoff: millisecond tests with a fake, a second provider is a new adapter.
  • Do not over-architect a create-read-update-delete (CRUD) service: layered is fine there.

One resource, four styles

REST + OpenAPI
GET /v1/events/evt_42?include=availability
cacheable, universal

GraphQL
event(id: "evt_42") { title availability { section seatsLeft } }
client picks the fields

gRPC + protobuf
rpc GetAvailability (Req) returns (Availability)
typed, binary, streaming, deadlines

Async + AsyncAPI
topic inventory.availability.v1
AvailabilityChanged {eventId, section, seatsLeft}

Same resource, different contracts: OpenAPI document, .proto, GraphQL schema, AsyncAPI.

Choosing a style

REST gRPC GraphQL Async
Payload JSON protobuf JSON JSON / Avro / protobuf
Browser-native βœ… yes ❌ no (gRPC-Web) βœ… yes push via WebSocket / server-sent events (SSE)
HTTP caching βœ… excellent ❌ none ⚠️ hard n/a
Coupling in time yes yes yes none
Best for public, partners, cacheable reads internal, many queries per second (QPS), streaming backend for frontend (BFF) composing services side effects, fan-out
  • BoxOffice: REST at the edge, GraphQL BFF for apps, gRPC order β†’ inventory, events after confirmation, webhooks in from the provider.
  • gRPC catch: long HTTP/2 connections defeat layer-4 (L4) load balancing. GraphQL catch: N+1 β†’ DataLoader.

Designing and evolving a contract

  • REST: correct methods and codes (201, 202, 409, 422, 429), Problem Details errors, cursor pagination, ETag + Cache-Control.
  • Unsafe writes (POST /holds, POST /orders) require an Idempotency-Key.
βœ… Compatible ❌ Breaking
add endpoint / optional field / response field remove or rename a field
new enum value in input change type or meaning (reais β†’ cents)
protobuf: new tag protobuf: reuse a tag (reserved!)
expandv2 next to v1
β†’
migrate consumers
β†’
Sunset headerannounces the date
β†’
v1 calls at zero?no: find who still calls
yes β†’
contractremove v1

Catch breaks before they ship with contract tests (Pact, Spring Cloud Contract).

Async work: off the request path

Does the user's answer depend on this step? No β†’ job on a queue β†’ worker.

fan
POST β†’
orderCONFIRMED + event, one transaction
β†’
"your order is confirmed"tens of ms
later
β†’
queue / Kafka
β†’
ticket workerPDF + QR β†’ object store
β†’
notification worker
β†’
e-mail providermay be slow or down
  • Gains: latency, availability (e-mail down β‰  sales down), load leveling, independent scaling.

100 orders/s, 0.8 s per ticket job, 40 workers. Backlog after 10 min of peak?

  • Capacity 40 / 0.8 = 50 jobs/s β†’ grows 50/s β†’ 30,000 jobs, last fan waits ~10 min.
  • Keeping up needs 100 Γ— 0.8 = 80 workers β€” if the e-mail provider's rate limit allows.

Jobs that survive the real world

Delivery is at least once β€” so:

  • Idempotent jobs (check before acting, dedupe keys); ack after the work.
  • Ids in messages, not data; a timeout per job.
  • Retry with backoff, then the dead-letter queue (DLQ) for poison messages; alert on DLQ growth.
  • Separate queues by priority: tickets β‰  marketing e-mails.
  • Respect downstream limits: more workers than the provider allows = 429s.
  • Scale workers on queue depth / lag, not CPU.

@Async is not a queue: work disappears on deploy. Spring Kafka: @RetryableTopic + @DltHandler.

Schedulers

@Scheduled on 10 replicas = the job runs 10 times.

Fix How
Kubernetes CronJob own pod; concurrencyPolicy: Forbid
Distributed lock (ShedLock) all try, the lock holder runs
Leader election Lease, etcd, ZooKeeper
Managed scheduler EventBridge Scheduler, Cloud Scheduler
  • Locks expire (a garbage-collection pause!) β†’ fencing tokens, or idempotent conditional writes.
  • Keep the scheduler off the critical path: lazy hold expiry + sweeper; onSaleAt checked in data.

Serverless

  • Platform runs your function per event, scales to zero, bills per invocation + GB-seconds.
  • Cold start: create sandbox + start runtime + init (Java virtual machine, JVM: seconds) β†’ provisioned concurrency, SnapStart, native image.
  • Concurrency = rate Γ— duration: 2,000 req/s Γ— 0.2 s = 400 environments.
  • Limits: account quota (Lambda default 1,000 per Region), 15-min max, no long connections.

Serverless scales; your database does not: 1,000 concurrent functions = 1,000 connections β†’ RDS Proxy / PgBouncer, reserved concurrency.

Cheapest for spiky, mostly idle work (~US$ 1 per million short invocations, illustrative); steady busy services favor containers.

Autoscaling

desiredReplicas = ⌈ currentReplicas Γ— currentMetric / target βŒ‰ β€” every 15 s (Horizontal Pod Autoscaler, HPA)

  • 6 replicas at 90% CPU, target 60% β†’ 9. Scale-down waits 5 min (stabilization).
  • Levels: HPA or Kubernetes Event-Driven Autoscaling (KEDA) for replicas, Vertical Pod Autoscaler (VPA) for size, Cluster Autoscaler / Karpenter for nodes.
  • Metric: CPU (CPU-bound), RPS or concurrency (I/O-bound), consumer lag (workers, KEDA).
  • Policies: target tracking, step, scheduled, predictive, queue-based.

maxReplicas from the database budget: 500 connections / pool of 20 = 25 replicas max.

Autoscaling lag

metrics15–60 s
β†’
HPA loop≀ 15 s
β†’
new node, if no room30–120 s
β†’
image pull5–60 s
β†’
JVM start + warm-up10–60 s
β†’
readiness5–10 s
=
~1–4 minuntil the first fan is served

Typical durations, illustrative. The HPA simulator on the Kubernetes page of class 11 plays the race with lab 10's settings.

A flash sale goes 1Γ— β†’ 20Γ— in seconds. For known peaks: scale on a schedule at 09:45 and pre-warm. For unknown ones: headroom, waiting room, rate limiting, load shedding.

External integrations: what you do not control

  • Latency with a heavy tail, their availability, rate limits, their API calendar, per-call cost.
  • Outcomes decided later: processing, 3-D Secure, webhook.
CREATED
accepted β†’
PENDING3-D Secure: REQUIRES_ACTION
webhook β†’
SUCCEEDED
FAILED
CREATED
timeout β†’
UNKNOWNit may have been charged
query / webhook β†’
PENDING

A timeout is not a failure: releasing the seats of a fan who was charged is a real bug. Nor is it a success.

  • One adapter per provider, with timeout, circuit breaker and bulkhead.

Idempotency keys

  • Inbound: the app sends Idempotency-Key per logical action (the Pay press), reused on every retry.
  • Outbound: BoxOffice sends a key derived from its own id (ord_9) to the provider.
create paymentkey ord_9
β†’
provider chargespay_77
answer lost βœ•
UNKNOWNretry with the SAME key
β†’
pay_77 returnedno second charge
Case Behavior
same key, same request, done replay the stored response
same key, in progress 409
same key, different request 422
failed before commit delete the key, allow retry

Implementation: table with primary key (client_id, key), INSERT … ON CONFLICT DO NOTHING, response stored in the same transaction, purge after 24 h.

Retries and unknown outcomes

What happened Could it have taken effect? What to do
connection refused, DNS or TLS failure ❌ no: nothing was sent retry with backoff
timeout after sending, 5xx ⚠️ unknown retry with the same key, or query first
429 Too Many Requests ❌ no wait for Retry-After
400, 402 declined, 422 ❌ no, and it never will do not retry

Simulator on the page: 1,000 fans press Pay with lab 9's client (1 s timeout, 3 attempts). 6.2% of the calls time out and, without a key, 63 fans pay twice or more; one key per order ("order-" + orderId): 0. On a slow day 133 checkouts end UNKNOWN, each charged once: the webhook or the reconciliation settles them.

Webhooks

Delivery property Receiver must…
anyone can call your URL verify the hash-based message authentication code (HMAC) over the raw body
captured requests can be replayed check the timestamp
at-least-once, retried for days deduplicate by event id
no ordering apply monotonic transitions
slow handler β†’ provider retries ack fast, process in a worker
deliveries can be lost reconcile

Standard Webhooks: webhook-id, webhook-timestamp, webhook-signature over id.timestamp.body; compare with hmac.compare_digest.

Reconciliation and provider outages

  • Reconcile at two speeds: a sweeper for UNKNOWN/PENDING payments (minutes) and a daily match against the provider's report.
  • Mismatch "provider succeeded, ours pending" β†’ confirm, or refund if the hold expired.
Provider down BoxOffice does
Payment breaker opens, fail fast, extend holds, sweeper resolves in-flight
E-mail queue, throttle to the limit, fail over to a second provider
Venue maps degrade: static image

The compute concept map

Component Question it answers Key concepts
app where does request logic run, how does it scale? stateless, 50–70% target, REST / gRPC / GraphQL
worker queue what runs outside the request? workers = rate Γ— time, at least once, DLQ
scheduler what runs at a time β€” once? CronJob, locks, leader election, fencing
serverless what scales per request, to zero? cold start, concurrency = rate Γ— duration
auth search notifications analytics which capabilities are shared? bounded contexts, event-fed, often bought
payment email thirdparty how to depend on what we don't control? idempotency keys, webhooks, reconciliation
(the whole tier) how many replicas, and when? HPA, KEDA, autoscaling lag, scheduled scaling

Compute scales easily because it is stateless β€” state lives in the stores of the next two classes.

Takeaways

  1. Stateless app servers scale out; size them at 50–70% utilization + spare.
  2. Start modular; split on a real force; boundaries = bounded contexts, shaped by teams.
  3. REST at the edge, gRPC inside, GraphQL for BFFs, events for side effects β€” evolve compatibly.
  4. Off the request path: idempotent jobs, retries, DLQ; schedulers that run once.
  5. Serverless: cold starts, concurrency = rate Γ— duration; autoscaling lags β€” pre-scale known peaks.
  6. Integrations: idempotency keys, verified webhooks, UNKNOWN state, reconciliation, an outage plan per provider.

Outline: (1) stateless app servers and replica sizing; (2) monolith, modular monolith, microservices, DDD and Conway; (3) API styles and contracts; (4) workers and schedulers; (5) serverless and autoscaling; (6) external integrations: idempotency, webhooks, reconciliation, outages.

Ask: why not 100 replicas? Cost, and 100 Γ— 20 connections = 2,000 connections to PostgreSQL: the bottleneck moves to the database. Show the scale-out scenario in the playground.