Messaging

Messaging components put an asynchronous boundary between the part of the system that produces work and the part that does it. Instead of calling a service and waiting, a producer writes a message to a broker and moves on; consumers read it when they can. That single change decouples the two sides in time (the consumer may be down or slow), in rate (a spike becomes a backlog that drains at the consumers' pace) and in knowledge (the producer does not need to know who consumes). The price: the result is no longer immediate, messages can arrive twice or out of order, and you now have to watch backlogs and lag.

In BoxOffice, the waiting room of the flash sale is a queue; order publishes OrderConfirmed events to Kafka, and notification, analytics and search each consume them independently; e-mail sending sits behind a queue so a slow provider never slows checkout.

flowchart LR
    P1[order] -->|OrderConfirmed| K[(Kafka topic)]
    K --> N[notification]
    K --> A[analytics]
    K --> S[search]
    W[gateway] -->|admission tickets| Q[[Queue: waiting room]]
    Q --> I1[inventory worker]
    Q --> I2[inventory worker]

Message Queue

What it is. A message queue stores messages until a consumer takes them. Several consumers can read from the same queue — competing consumers — and each message goes to only one of them, so adding consumers increases throughput. A consumer acknowledges a message after processing it; if it crashes before the ack, the message becomes visible again (after a visibility timeout) and another consumer retries it. Messages that keep failing are moved to a dead-letter queue (DLQ) for inspection.

When to use.

• Background work the user does not need to wait for: sending e-mails, generating PDF tickets, transcoding videos, resizing posters.
• Absorbing spikes: producers enqueue at 10× the rate for a minute; consumers drain at a steady rate — the backlog is the buffer (scenario async-queue).
• Load leveling in front of a fragile resource: BoxOffice's waiting room admits fans into inventory at the rate the database can sustain.
• Retrying calls to unreliable external providers without blocking the request path.

When not to (or watch out).

• When the caller needs the result now (show the seat as held): keep that path synchronous.
• When several independent services need the same message: use pub/sub or a stream, not one queue per consumer glued by hand.
• Delivery is usually at-least-once: a message may be processed twice → consumers must be idempotent.
• Ordering is usually not guaranteed (or limited to FIFO queues with lower throughput).
• A growing backlog is a silent failure: monitor queue depth and the age of the oldest message, and alert on them.

Key numbers.

Failure modes.

• Poison message — one message always crashes the consumer and is retried forever → max receive count, then DLQ.
• Duplicate processing — consumer crashed after doing the work but before the ack → idempotent consumer (dedupe by message id).
• Unbounded backlog — consumers slower than producers for a long time → autoscale consumers on queue depth, backpressure, shed or reject at the edge.
• Visibility timeout too short — slow messages reappear and are processed in parallel by two consumers → timeout longer than p99 processing time, heartbeat extension.
• Consumer overload of the downstream — scaling consumers floods the database → cap consumer concurrency.

In the playground. queue is a messaging node and therefore an async boundary: the producer gets an ack after the broker's few milliseconds, the message enters the backlog, and downstream consumers (e.g. worker) drain it at their capacity, each message going to one consumer. Watch the metrics backlog, consumer lag (s) and end-to-end (async) latency; when the backlog exceeds its maximum or retention, messages are dropped and counted as errors. Scale the consumers' replicas or autoscale to drain faster. Scenarios: async-queue, flash-sale, notification-system, video-streaming.

Real products.

Learn more: Messaging › Queues & pub/sub · Messaging › Delivery semantics · Compute › Async work · Case study: notification system · Hands-on 7

Pub/Sub

What it is. In publish/subscribe, a producer publishes a message to a topic and the broker delivers a copy to every subscription. Publishers don't know who listens; adding a new subscriber requires no change to the publisher. Each subscription behaves like its own queue (with acks and retries). Classic pub/sub systems delete a message once each subscription has acknowledged it, and are optimized for low-latency fan-out rather than long retention or replay.

When to use.

• Fan-out of events to several independent services: EventCancelled → refunds, notifications, search index, cache invalidation.
• Real-time delivery across servers: in a chat, the server that receives a message publishes it on the conversation's channel and the servers holding the recipients' WebSockets deliver it (scenario chat).
• Cache invalidation across many app replicas.
• Notifications to many endpoints (mobile push, e-mail, webhooks) through per-channel subscriptions.

When not to (or watch out).

• When you need to replay history or add a consumer that must read past events: use an event stream (Kafka) with retention.
• When only one worker should process each message: that is a queue (or one subscription with competing consumers).
• Ordering across messages is usually not guaranteed; per-key ordering may be an option with lower throughput.
• In-memory pub/sub (Redis Pub/Sub) is fire-and-forget: subscribers that are offline lose the messages.
• Fan-out multiplies load: one publish × N subscribers × their downstream calls.

Key numbers.

Failure modes.

• Slow subscriber — one subscription accumulates a backlog while the others are fine → per-subscription monitoring, scale that consumer, DLQ.
• Lost messages with fire-and-forget pub/sub when a subscriber is down → durable subscriptions or a stream.
• Duplicates on redelivery → idempotent subscribers.
• Fan-out storm — a popular topic triggers thousands of downstream calls at once → batching, rate limits on subscribers.
• Accidental coupling — subscribers depend on undocumented fields of the message → schemas and versioning.

In the playground. pubsub is a messaging node (async boundary) that delivers a copy of every message to each consumer connected to it (fan-out), unlike queue, where consumers compete. Each consumer has its own backlog and lag. Watch how adding a consumer multiplies the total work but does not slow the producer. Scenarios: chat, event-fanout, notification-system.

Real products.

Learn more: Messaging › Queues & pub/sub · Messaging › Patterns · Edge › Protocols · Case study: chat · Case study: news feed

Event Stream

What it is. An event stream is an append-only, ordered log split into partitions and retained for a period (hours, days or forever), not deleted on consumption. Each consumer (or consumer group) keeps its own offset — its position in the log — so many independent consumers can read the same events at their own pace, and a new consumer can replay history from the beginning. Events with the same key go to the same partition, which gives ordering per key. This is the backbone of event-driven architectures, change data capture (CDC) and real-time analytics. Kafka is the best-known implementation and has its own entry below; this entry covers the concept and the managed alternatives.

When to use.

• Many consumers of the same facts, each at its own pace: OrderConfirmed read by notification (now), analytics (in batches) and a new fraud service (replaying last month).
• Event sourcing and CQRS: the log is the source of truth and read models are rebuilt from it.
• CDC: every change of the orders table published as an event (e.g. Debezium) to feed search, cache invalidation and the warehouse.
• High-throughput ingestion: clicks, location updates, metrics, logs.

When not to (or watch out).

• Simple background jobs with one consumer type: a queue is simpler (per-message acks, DLQ built in).
• Request/response interactions: a stream adds latency and complexity for no gain.
• Parallelism is bounded by the number of partitions; ordering only holds within a partition — pick the key carefully (e.g. order_id).
• A slow consumer does not slow the others, but its lag grows; if lag exceeds retention, it loses events.
• Schemas evolve: use a schema registry and compatible changes, or old consumers break.

Key numbers.

Failure modes.

• Consumer lag growing — consumers slower than producers → more consumers (up to the partition count), more partitions, batching, faster handlers.
• Hot partition — one key (the flash-sale event) gets most events → a finer key (seat_id instead of event_id), or accept the limit.
• Lag beyond retention — events lost for that consumer → alert on lag well before retention, longer retention.
• Reprocessing side effects on replay (e-mails sent again) → idempotent consumers, separate replay pipelines.
• Breaking schema change → schema registry with compatibility checks.

In the playground. stream is a messaging node with fan-out (every consumer gets every event) and a partitions parameter (default 4): the maximum parallelism of each consumer is min(partitions, consumer replicas). Adding consumer replicas beyond the partition count does not reduce lag — try it. Metrics: backlog, consumer lag (s), end-to-end latency. Scenarios: event-fanout, ride-sharing, news-feed.

Real products.

Learn more: Messaging › Queues & pub/sub · Messaging › Kafka · Messaging › Patterns · Case study: ride sharing

Kafka

What it is. Apache Kafka is a distributed event-streaming platform. Data is organized in topics, each split into partitions; each partition is an ordered, append-only log replicated across brokers (one leader, followers in the in-sync replica set). Producers choose the partition by key (same key → same partition → ordered). Consumers join consumer groups: within a group each partition is assigned to one consumer, and the group commits offsets to remember where it stopped. Messages are retained by time or size (or kept forever as the latest value per key with log compaction). Since Kafka 4.0, clusters run in KRaft mode (Raft-based metadata quorum, no ZooKeeper). Around the core: Kafka Connect (integrations, CDC with Debezium), Kafka Streams (stream processing) and schema registries.

When to use.

• The event backbone of a microservice system: BoxOffice order, payment and inventory publish domain events; others react (choreographed sagas, read models).
• Transactional outbox + CDC: Debezium reads the outbox table and publishes to Kafka, so database and events never diverge.
• High-volume pipelines: clickstreams, logs, metrics, location updates — with replay for reprocessing.
• When several teams need the same events with independent pace and retention.

When not to (or watch out).

• Small systems with a few background jobs: a managed queue is far simpler to operate.
• Per-message routing, priorities, delayed messages and per-message retries are not Kafka's strengths (queues do them better).
• Partition count is a capacity decision made up front: it caps consumer parallelism, and adding partitions later changes the key → partition mapping (ordering per key breaks across the change).
• "Exactly-once" exists for Kafka-to-Kafka processing (idempotent producer + transactions); side effects outside Kafka still need idempotent consumers.
• Operating Kafka (brokers, storage, rebalancing, upgrades) is real work → consider a managed service.

Key numbers.

Failure modes.

• Consumer lag — consumers can't keep up → scale consumers up to the partition count, add partitions (planned), optimize the handler, batch.
• Rebalance storms — consumers joining/leaving repeatedly pause consumption → cooperative rebalancing, stable group membership, longer session timeouts.
• Broker failure → leader election for its partitions from the in-sync replicas; with RF 3 and min.insync.replicas = 2, writes continue.
• Poison message blocks a partition (consumer keeps failing on the same offset) → retry topic + DLQ topic, skip after N attempts.
• Duplicates after a consumer crash between processing and offset commit → idempotent consumer (dedupe table keyed by event id).
• Hot partition from a skewed key → better key choice, or split hot keys.
• Disk full from retention too long or a stuck compaction → monitor broker disk, tiered storage.

In the playground. kafka is a messaging node like stream (fan-out to every consumer, retained backlog) with an explicit partitions parameter (default 12, on 3 brokers): each consumer's parallelism is min(partitions, consumer replicas). Try the classic mistake: 3 partitions, scale the consumer to 10 replicas, and watch the lag not move. Metrics: backlog, consumer lag (s), end-to-end latency; retention limits turn excess backlog into drops (errors). Scenarios: event-fanout, payment-system, flash-sale, observability-stack.

Real products.

Learn more: Messaging › Kafka · Messaging › Delivery semantics · Messaging › Patterns · Distributed › Consensus · Case study: payment system · Hands-on 7