System Design & Scalability

Vertical scaling (scale up): adds more CPU, RAM or disk to the same machine. It is simple — no code changes required. But it has a physical and hardware limit (there is no machine with infinite RAM), creates a single point of failure, and cost grows exponentially with capacity.

Horizontal scaling (scale out): adds more machines. Theoretically unlimited and cheaper at scale. It requires stateless application servers, load balancing and coordination between nodes. Most large-scale web systems scale horizontally.

Practical strategy: start vertical (simpler, zero refactoring), but design for horizontal from the start. When vertical hits a cost or physical limit, move to horizontal. The mistake is not planning for this from the beginning.

Distributes requests across multiple backend servers, preventing any server from being overloaded while others sit idle. It also runs health checks and automatically removes failed servers from the pool.

Distribution algorithms:

• Round-robin: each server receives requests in turn. Simple, but ignores the current load.
• Weighted round-robin: more powerful servers receive a larger share of requests.
• Least connections: forwards to the server with the fewest active connections. Better for requests of variable duration.
• IP hash: shard = hash(client_IP) % N. Guarantees sticky sessions (same client → same server). Useful when the server holds state.

Implementation types: Hardware (F5, Cisco), software (Nginx, HAProxy, Envoy), cloud-managed (AWS ALB/NLB, GCP Cloud Load Balancing).

Watch out: the load balancer itself can become a single point of failure. Solution: active/passive pairs (primary + standby with automatic failover) or DNS-based load balancing (multiple IPs in DNS with a short TTL).

Denormalization: denormalizing a relational database means strategically duplicating data to avoid expensive JOINs. You trade consistency (updates must be made in several places) for read performance. It works well when reads far outnumber writes and JOINs are a proven bottleneck.

NoSQL — when and which to use:

• Key-Value (Redis, DynamoDB): O(1) access by key. Ideal for sessions, caching, leaderboards, rate limiting.
• Document (MongoDB, Couchbase): semi-structured data, flexible schema. Ideal for product catalogs, user profiles.
• Column-family (Cassandra, HBase): massive distributed writes, queries by partition key. Ideal for time series, IoT, logs.
• Graph (Neo4j, Amazon Neptune): complex relationships between entities. Ideal for social graphs, recommendation systems, fraud detection.

CAP Theorem: in a distributed system, you can guarantee at most 2 of the 3 properties: Consistency (all nodes see the same data), Availability (every request gets a response), Partition tolerance (the system keeps working despite network failures). In practice, P is unavoidable — you choose between CP (consistency) and AP (availability).

Splits the database across multiple nodes (shards), where each node stores a subset of the data. It lets you scale storage and query throughput horizontally beyond the limits of a single machine.

Sharding strategies:

• Vertical partitioning: different tables on different servers. Simple, but limits the growth of individual tables.
• Horizontal (range-based): users with id 1–1M on shard A, 1M–2M on B. Good for range queries, but can create hot spots if the distribution is uneven.
• Hash-based: shard = hash(user_id) % N. Uniform distribution, avoids hot spots. Makes range queries harder. Consistent hashing minimizes data remapping when nodes are added or removed.
• Directory-based: a lookup table maps key → shard. Flexible (lets you move data without rehashing), but the table becomes a bottleneck and a single point of failure.

Classic problem: queries that span shards (cross-shard joins) are extremely expensive. Choose the sharding key based on the most frequent access patterns. If 90% of queries are by user_id, use user_id as the sharding key.

Stores computed results or frequently accessed data in memory (much faster than disk) to avoid recomputation or database I/O. It is one of the highest-impact optimizations in read-heavy systems.

Eviction policies (when memory is full):

• LRU (Least Recently Used): removes the item accessed longest ago. It is the default — it works well for most workloads.
• LFU (Least Frequently Used): removes the least frequently accessed item. Better when data popularity is stable over time.
• TTL (Time-To-Live): each item has an expiration time. Essential for time-sensitive data (e.g. prices, sessions).

Write strategies:

• Write-through: writes to the cache and the database at the same time. Guaranteed consistency, but higher write latency.
• Write-back (write-behind): writes to the cache immediately and propagates to the database asynchronously. Higher throughput, but risk of data loss if the cache fails.
• Write-around: writes straight to the database, bypassing the cache. The cache is only populated on the next read. Useful when written data is rarely read right afterwards.

Classic problems: Cache stampede (many keys expiring at the same time cause a flood of database queries — use probabilistic expiration or a mutex lock), cold start (empty cache after a deploy — use warm-up scripts), cache poisoning (invalid data in the cache).

Instead of processing everything synchronously inside the user's request, you enqueue the heavy work for asynchronous execution. The user gets a fast response (an acknowledgment), and workers do the processing in the background.

Classic use cases:

• Sending email/SMS after sign-up
• Generating an image thumbnail after upload
• Payment processing with automatic retry
• Indexing content for a search engine
• Push notifications to mobile devices

Tools and tradeoffs:

• RabbitMQ: AMQP, delivery guarantees (at-least-once), complex routing via exchange/binding. Ideal for workloads that need delivery guarantees.
• Apache Kafka: distributed log, persistent message retention, replay capability, very high throughput (millions of msg/s). Ideal for event streaming, auditing, multiple independent consumers.
• Amazon SQS: managed, zero ops, SQS Standard (at-least-once, unordered) or FIFO (exactly-once, ordered). Ideal if you want simplicity on AWS.

The producer-consumer pattern decouples services, absorbs traffic spikes (the producer can be much faster than the consumer for short periods) and enables automatic retry when a worker fails.

Three fundamental metrics to understand and size the network capacity of a distributed system:

Components of latency:

• Propagation delay: speed of light × distance. São Paulo → New York ≈ 100ms minimum.
• Transmission delay: packet size ÷ bandwidth. A 1GB file over a 100Mbps link takes ~80 seconds.
• Processing delay: time routers take to process the header and decide the next hop.
• Queuing delay: time spent waiting in the buffers of congested routers. Variable and unpredictable.

Rule of thumb: optimizing throughput (batching, compression) can increase latency. Optimizing latency (smaller requests, persistent connections) can reduce throughput. Understand which one is more critical for your system before optimizing.

A programming paradigm for processing large datasets in parallel on a cluster. The programmer only defines the Map and Reduce functions; the framework takes care of distribution, parallelism, fault tolerance and the shuffle.

The flow of the two phases:

• Map: each worker reads a chunk of the input data and emits (key, value) pairs. E.g.: for word count, each line → emits (word, 1) for every word found.
• Shuffle & Sort: the framework automatically groups all values emitted for the same key and sends them to the same reducer.
• Reduce: the reducer receives (key, [v1, v2, v3, ...]) and produces the final result. E.g.: (word, [1,1,1]) → (word, 3).

The framework takes care of: distributing data among workers, parallel execution, automatic re-execution of failed tasks (fault tolerance), shuffle/sort between Map and Reduce, and monitoring.

Implementations: Hadoop MapReduce (batch, disk I/O — robust but slow), Apache Spark (in-memory — 10-100x faster for iterative algorithms such as ML, streaming via Spark Streaming).

When NOT to use it: real-time queries with low-latency requirements, ACID transactions that need strong consistency, caching systems or simple lookups. MapReduce is for batch processing of large volumes.

# Pseudocode — Word Count in MapReduce
def map(document_id, text):
    for word in text.split():
        emit(word, 1)           # emits (key, value)

def reduce(word, counts):       # receives (key, [v1, v2, ...])
    emit(word, sum(counts))     # aggregates and emits the final result

The problem: with shard = hash(key) % N, changing N (adding a node or losing one) remaps almost every key. In a distributed cache this amounts to emptying the whole cache at once — and all traffic falls on the database.

The solution: nodes and keys are placed on a hash ring (e.g. 0 … 2³²−1). Each key belongs to the first node found going clockwise. When a node is added or removed, only the keys in the neighboring range change owner — on average K/N keys.

• Virtual nodes: each physical node takes several positions on the ring, which evens out the distribution and lets you give more weight to bigger machines.
• Replication: the key is also stored on the next R−1 nodes of the ring (the Dynamo and Cassandra model).
• Where it shows up: Cassandra, DynamoDB, Riak, Memcached clients (ketama), Nginx (hash $request_uri consistent;).

Replicating data increases availability and read capacity, but forces you to choose what the client may see while the replicas converge.

Topologies:

• Leader-follower (primary-replica): writes go to the leader, reads scale out on the replicas. With asynchronous replication there is replication lag — a read on a replica may return stale data.
• Multi-leader: writes accepted in several regions. Requires conflict resolution (last-write-wins, CRDTs, merge in the application).
• Leaderless (quorum): N replicas; a write is acknowledged by W and a read queries R. If R + W > N, every read hits at least one replica with the latest acknowledged write. E.g.: N=3, W=2, R=2.

Consistency models (strongest to weakest): linearizability (strong) → read-your-writes → monotonic reads → eventual consistency.

Practical trick: to guarantee read-your-writes without global strong consistency, read from the leader for a few seconds after the user's own write (e.g. after editing their profile).

In the course: CAP Theorem · Consensus & Algorithms

A CDN (Content Delivery Network) is a network of edge servers that keep copies of content close to the user. It reduces latency (the RTT to the edge is much smaller than to the origin) and takes load off the origin.

• Pull CDN: the edge fetches from the origin on the first access (cache miss) and stores it according to Cache-Control/TTL. Simple, the default in most cases.
• Push CDN: you publish the content to the CDN explicitly. Useful for large, rarely changing files.
• Invalidation: prefer versioning the file name (app.3f9a1c.js) over invalidating the cache — invalidation is slow and expensive.
• Beyond static files: TLS termination, DDoS/WAF protection, edge compute and caching of dynamic responses with a short TTL.

Examples: Amazon CloudFront, Cloudflare, Akamai, Fastly.

Limits how many requests a client (user, API key, IP) can make within a time window. It protects against abuse and accidental spikes, ensures fair use and controls cost. The standard response is HTTP 429 Too Many Requests with a Retry-After header.

Algorithms:

• Token bucket: tokens are refilled at a fixed rate; each request consumes one. Allows bursts up to the bucket size. The most common choice for APIs.
• Leaky bucket: requests enter a queue that is drained at a constant rate. Smooths traffic (it is the model behind Nginx's limit_req).
• Fixed window counter: one counter per window (e.g. per minute). Simple, but allows up to 2× the limit at the window boundary.
• Sliding window (log or counter): more accurate than the fixed window, at a higher memory/computation cost.

Distributed: with several gateway replicas, the counter must be shared — typically Redis (INCR + EXPIRE, or a Lua script for atomicity). Also decide what happens if Redis goes down: fail-open (let traffic through) or fail-closed (block it).

In the course: Hands-on 2 — Gateway · Hands-on 8 — In-Memory Cache

API Gateway: the single entry point of the system. It concentrates cross-cutting concerns — routing, authentication (JWT validation), rate limiting, TLS, CORS, response aggregation — so internal services don't have to repeat them.

• Communication: synchronous (REST/gRPC) when the client needs the answer now; asynchronous (events on a queue) when the work can happen later.
• Database per service: each service owns its data. Consistency across services via Saga (orchestrated or choreographed) and the Outbox pattern to publish events reliably.
• Service discovery: the orchestrator's DNS (Kubernetes Services) or a registry (Eureka, Consul).

A tradeoff that impresses in interviews: knowing when not to use microservices. Small team and a still-unstable domain → start with a modular monolith and extract services when there is a concrete reason (independent scaling, independent deploys, separate teams).

In the course: Microservices — Architecture & Components · Hands-on 2 — Gateway · Hands-on 3 — Security

In distributed systems, partial failures are the rule. These patterns keep the failure of one dependency from bringing down the whole system (cascading failure).

• Timeout: every remote call needs one. Without a timeout, a slow dependency exhausts the threads of its callers.
• Retry with exponential backoff + jitter: only for transient errors and idempotent operations. Jitter keeps all clients from retrying at the same time (thundering herd).
• Circuit breaker: closed → open (fails fast, without calling the dependency) → half-open (tests whether it is back).
• Bulkhead: resource pools isolated per dependency, so that one cannot consume all the connections.
• Idempotency key: the client sends a unique identifier; the server deduplicates repetitions of the same operation.
• Fallback / graceful degradation: a degraded response (cache, default value) instead of an error.
• Load shedding: reject part of the traffic early when saturated, instead of degrading every request.

In the course: Resilience Patterns · Chaos Engineering

Designing the system is not enough: you need to know when and where it is failing. Bringing up observability at the end of the interview shows production maturity.

• Logs: what happened, event by event. Structured (JSON) and with a trace id.
• Metrics: how much and how often. The RED method for services (Rate, Errors, Duration) and USE for resources (Utilization, Saturation, Errors).
• Traces: where the time was spent in a request that crosses several services.

SLI / SLO / SLA: the SLI is the measurement (e.g. % of requests with latency < 300 ms); the SLO is the internal target (e.g. 99.9% over 28 days); the SLA is the contract with the customer, with a penalty. The error budget (1 − SLO) defines how much risk you can spend on deploys and experiments.

Use percentiles, not averages: p50, p95 and p99. The average hides the tail — and the tail is where the unhappy users are.

In the course: Observability · SLI / SLO / SLA · Hands-on 6 · Hands-on 10

• Choice of algorithm (token bucket vs. sliding window) and its justification.
• State shared across gateway replicas (Redis) and atomicity (Lua script).
• Rules per client/endpoint, a 429 response with Retry-After.
• Fail-open vs. fail-closed if the counter storage goes down.

• Fan-out on write (push to pre-computed timelines) vs. fan-out on read (build the timeline at read time).
• Hybrid model: push for regular users, pull for celebrities with millions of followers.
• Timeline in cache (a list of IDs in Redis), media in object storage + CDN.
• Cursor-based pagination, not offset-based.

• Persistent connections (WebSocket) and a service that knows which server each user is connected to.
• Per-conversation ordering (a sequence number per conversation_id) — global ordering is not needed.
• Delivery to offline users (store-and-forward) and delivery/read receipts.
• Partitioning the message storage by conversation (e.g. Cassandra).

• Multiple channels (push, e-mail, SMS) with separate queues and workers per channel.
• Retries with backoff and idempotency so nobody is notified twice.
• User preferences and opt-out, per-user rate limit, templates.
• Integration with external providers (APNs, FCM, SES, Twilio) and what to do when they fail.

• URL frontier: a prioritized queue partitioned by domain.
• Politeness: respect robots.txt and limit requests per domain.
• Deduplication: a Bloom filter for URLs already seen, a content hash for duplicate pages.
• DNS cache, distributed page storage, spider-trap detection.

• Frequent location updates (every few seconds) — a write-heavy system.
• Geospatial index: geohash, quadtree or H3 for "nearby drivers".
• Matching that guarantees a driver is never assigned to two rides (lock/strong consistency at that point).
• Surge pricing, ETA and partitioning by region/city.

• Partitioning with consistent hashing and virtual nodes.
• Replication with a configurable quorum (N, R, W) and the consistency × latency tradeoff.
• Failure detection (gossip), hinted handoff and anti-entropy (Merkle trees).
• Conflict resolution: vector clocks or last-write-wins.