Delivery Semantics

The morning after a big on-sale, two support tickets arrive. Ana received three identical confirmation e-mails. Bruno received none, although his order is confirmed and his tickets are in the app. Both problems have the same root: messages cross several machines and networks, any of them can fail between two steps, and each failure either loses a message or repeats it. This page is about deciding which of the two you can tolerate, and about writing consumers for which a repeat does no harm.

Where messages get lost or duplicated

Follow one OrderConfirmed from order to notification. It crosses three hops, and each hop has its own failure:

flowchart LR
    O["order<br/>(producer)"] -->|"1 · send"| B[("Kafka<br/>broker")]
    B -->|"2 · replicate"| R[("replicas")]
    B -->|"3 · deliver"| N["notification<br/>(consumer)"]
    N -->|"4 · side effect"| E["e-mail provider"]
    N -->|"5 · commit offset"| B

Ana's three e-mails are hop 3–5: notification sent the e-mail, then was killed during a deploy before committing, twice. Bruno's missing e-mail is hop 1: order saved his order and crashed before publishing.

The three delivery guarantees

A delivery guarantee says how many times each message is processed in the presence of failures¹:

Exactly-once, honestly

"Exactly-once delivery" in the strict sense is impossible over an unreliable network: the receiver can never be sure the sender saw its acknowledgement (the Two Generals problem; see Distributed systems). What systems actually offer is exactly-once processing within a boundary:

• Idempotent producer (enable.idempotence=true, default since Kafka 3.0). The broker gives each producer an id and tracks a sequence number per partition; a retried batch with a sequence it has already written is discarded. Removes hop-1 duplicates.
• Kafka transactions². A producer can write to several partitions and commit consumer offsets atomically. Consumers with isolation.level=read_committed only see committed transactions. This gives exactly-once for read from Kafka → process → write to Kafka pipelines (Kafka Streams: processing.guarantee=exactly_once_v2).
• The boundary. Kafka transactions cannot un-send an e-mail, un-charge a card or roll back a row in PostgreSQL. As soon as a side effect leaves Kafka, you are back to at-least-once.

So the practical recipe, used almost everywhere, is:

Effectively-once = at-least-once delivery + idempotent processing.

Idempotent consumers

An operation is idempotent if doing it twice has the same effect as doing it once³. An idempotent consumer makes duplicate deliveries harmless. There are three families of techniques:

The deduplication table is the general solution. It requires every message to carry a unique event id (a UUID generated once, by the producer, when the event is created — not on each retry):

CREATE TABLE processed_events (
    consumer      TEXT        NOT NULL,
    event_id      UUID        NOT NULL,
    processed_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (consumer, event_id)
);

@Component
@RequiredArgsConstructor
public class OrderConfirmedListener {

    private final JdbcTemplate jdbc;
    private final ConfirmationEmails emails;

    @KafkaListener(topics = "order-events", groupId = "notification")
    @Transactional   // dedup row and business change commit together, or not at all
    public void on(OrderConfirmed event) {
        int inserted = jdbc.update("""
            INSERT INTO processed_events (consumer, event_id)
            VALUES ('notification', ?)
            ON CONFLICT DO NOTHING
            """, event.eventId());
        if (inserted == 0) {
            return;                     // duplicate delivery: already handled
        }
        emails.schedule(event);         // INSERT INTO outgoing_emails ... (same transaction)
    }
}

Two details make this correct:

1. Same transaction. The dedup row and the business change commit together. If the service crashes before the commit, neither exists and the redelivered message is processed normally; after the commit, the redelivery is skipped. The offset is committed after the transaction.
2. External side effects are scheduled, not performed, inside the transaction. emails.schedule only inserts a row into outgoing_emails (with UNIQUE (order_id, kind)). A separate sender reads that table and calls the e-mail provider passing the row id as an idempotency key, so a retry of the call does not send twice. Providers of e-mail, SMS and payments commonly accept such keys (see Integrations).

Keep the dedup table bounded: delete rows older than the topic retention plus a margin (a redelivery older than the retention cannot happen).

/** @return true if an e-mail was sent, false if the event was a duplicate */
@Transactional
public boolean handle(OrderEvent e) {
    if (props.idempotent()) {
        int fresh = jdbc.sql("INSERT INTO processed_events (event_id) VALUES (?) ON CONFLICT DO NOTHING")
                .param(e.eventId())
                .update();
        if (fresh == 0) {
            log.info("duplicate event {} for order {} ignored", e.eventId(), e.orderId());
            return false;
        }
    }
    email.orderReceived(e); // throws -> the whole transaction rolls back, the dedupe row too
    jdbc.sql("""
            INSERT INTO notifications (event_id, order_id, email, kind, occurred_at)
            VALUES (?, ?, ?, ?, ?)
            """)
            .params(e.eventId(), e.orderId(), e.customerEmail(), e.eventType(),
                    OffsetDateTime.ofInstant(e.occurredAt(), ZoneOffset.UTC))
            .update();
    return true;
}

Ordering

Kafka guarantees order per partition, so per key. Several things can still put events out of order:

• Producer retries. If batch 1 fails and batch 2 succeeds, the retry of batch 1 lands after batch 2. The idempotent producer prevents this (it keeps order with up to 5 in-flight requests per connection).
• Different keys or topics. SeatsHeld on inventory-events and OrderCancelled on order-events have no relative order at all.
• Parallel processing inside a consumer. A thread pool processes records of the same key concurrently.
• Retries in the consumer. Moving a failed record to a retry topic lets later records of the same key overtake it (next section).

When order matters and cannot be guaranteed by the transport, make the consumer tolerant: carry a version (or sequence number) from the source of truth in each event, and ignore anything older than what you already applied:

-- search keeps "seats left" per event; inventory stamps each change with a version
UPDATE event_availability
   SET seats_left = :seatsLeft, version = :version
 WHERE event_id = :eventId
   AND version < :version;     -- a late, older update changes nothing

This makes the update idempotent and order-insensitive: applying versions 7, 9, 8 leaves version 9.

Retries and dead letter queues

A consumer will meet messages it cannot process. Two very different reasons:

• Transient failures: the database is restarting, the e-mail provider answers 503. Retrying later will work.
• Permanent failures: the payload is invalid JSON, it references an event id that does not exist, a bug throws on a specific field. Retrying will fail forever. Such a message is a poison message.

A poison message at the head of a partition blocks everything behind it: the consumer retries it forever and lag grows for every other order in that partition. The solution is a dead letter queue (DLQ, in Kafka usually a dead letter topic, DLT): after a bounded number of attempts, the message is moved aside, with the error, and the consumer moves on.

stateDiagram-v2
    [*] --> Delivered
    Delivered --> Processed : success
    Delivered --> RetryTopic : transient error (attempt < max)
    RetryTopic --> Delivered : after backoff (1 s, 4 s, 16 s)
    Delivered --> DeadLetter : permanent error, or max attempts reached
    DeadLetter --> Delivered : fixed and redriven by an operator
    Processed --> [*]

With Spring Kafka, non-blocking retries are declarative: failed records go to retry topics with increasing delays, while the main topic keeps flowing.

@RetryableTopic(
    attempts = "4",                                      // 1 try + 3 retries
    backOff = @BackOff(delay = 1000, multiplier = 4.0),  // 1 s, 4 s, 16 s
    exclude = { InvalidEventException.class },           // permanent: straight to the DLT
    dltTopicSuffix = "-dlt")
@KafkaListener(topics = "order-events", groupId = "notification")
public void on(OrderConfirmed event) {
    confirmations.send(event);
}

@DltHandler
public void onDeadLetter(OrderConfirmed event,
                         @Header(KafkaHeaders.DLT_EXCEPTION_MESSAGE) String error) {
    log.error("order {} moved to order-events-dlt: {}", event.orderId(), error);
}

Operating a DLQ:

• Alert on it. A DLQ with messages is an incident in slow motion: each message is a customer without an e-mail or an order not in the warehouse. Page or ticket when it is non-empty or growing.
• Keep context. Store the original topic, partition, offset, exception and attempt count as headers (Spring does this).
• Redrive. After fixing the bug, replay the DLQ into the main topic. This is only safe because consumers are idempotent.
• Mind the order. Non-blocking retries let later events of the same order overtake a retried one. If per-key order is essential (an OrderCancelled must not be applied before its OrderConfirmed), use blocking retries for transient errors (retry in place, with backoff, pausing the partition) or version-guarded consumers.

@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<String, String> template) {
    // after the retries, publish to <topic>-dlt, same partition, with the exception in headers
    DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template);

    ExponentialBackOffWithMaxRetries backOff = new ExponentialBackOffWithMaxRetries(3);
    backOff.setInitialInterval(500L);  // 0.5 s, 1 s, 2 s
    backOff.setMultiplier(2.0);
    backOff.setMaxInterval(5_000L);

    DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, backOff);
    // permanent failures: retrying is pointless, go straight to the DLT
    handler.addNotRetryableExceptions(JacksonException.class, IllegalArgumentException.class);
    return handler;
}

@KafkaListener(topics = "order-events-dlt", groupId = "notification-dlt")
public void onDeadLetter(ConsumerRecord<String, String> record) {
    // the listener's exception is wrapped: the CAUSE class says what really failed
    Header cause = record.headers().lastHeader(KafkaHeaders.DLT_EXCEPTION_CAUSE_FQCN);
    String why = cause == null ? "?" : new String(cause.value(), StandardCharsets.UTF_8);
    log.warn("dead letter at {}-{}@{}: {}", record.topic(), record.partition(), record.offset(), why);
    jdbc.sql("""
            INSERT INTO dead_letters (topic, partition, record_offset, record_key, payload, reason)
            VALUES (?, ?, ?, ?, ?, ?)
            """)
            .params(record.topic(), record.partition(), record.offset(), record.key(), record.value(), why)
            .update();
}

Backpressure in the consumer

A Kafka consumer pulls, so it is never flooded by the broker — but it can flood what it calls. The notification service must not exceed the e-mail provider's rate limit just because a backlog of 100,000 messages is waiting. Control the pace with max.poll.records and a client-side rate limiter, pause partitions while a downstream circuit breaker is open (see Resilience patterns), and let lag grow: a backlog that drains at the provider's pace is better than a ban from the provider.

Schema evolution

Events outlive the code that produced them: a topic with 7 days of retention contains messages written by last week's version of order, and the consumers that read them may be months older or newer. You cannot deploy producers and consumers atomically, so every schema change must be compatible with the versions still running⁴.

Never compatible: renaming a field, changing its type (total from number to string), changing its meaning (total from reais to cents), reusing a Protobuf field number.

{
  "type": "record", "name": "OrderConfirmed", "namespace": "boxoffice.events",
  "fields": [
    { "name": "eventId",    "type": "string" },
    { "name": "orderId",    "type": "string" },
    { "name": "totalCents", "type": "long" },
    { "name": "currency",   "type": "string", "default": "BRL" },
    { "name": "couponCode", "type": ["null", "string"], "default": null }
  ]
}

The tools:

• Serialization formats with schemas: Avro, Protobuf or JSON Schema. Plain JSON without a schema "works" until the first silent break.
• A schema registry (Confluent Schema Registry, Apicurio, AWS Glue Schema Registry) stores every version, checks compatibility when a producer registers a new one, and lets messages carry a small schema id instead of the full schema.
• An envelope with standard metadata. The CNCF CloudEvents specification defines common attributes so tools can route and trace events without parsing the payload:

{
  "specversion": "1.0",
  "id": "e-9f2c",
  "source": "/boxoffice/order",
  "type": "com.boxoffice.order.confirmed.v2",
  "subject": "o-7731",
  "time": "2027-03-14T10:00:07Z",
  "datacontenttype": "application/json",
  "data": { "orderId": "o-7731", "totalCents": 78000, "currency": "BRL" }
}

For a truly breaking change, publish the new version to a new topic (order-events.v2), have the producer write both for a transition period, migrate consumers one by one, then retire the old topic.

Putting it together

How it connects

flowchart LR
    H1["hop 1: producer → broker"] --> IP["idempotent producer"]
    H1 --> OB["transactional outbox<br/>→ Patterns"]
    H2["hop 2: broker"] --> DUR["acks=all, min ISR 2<br/>→ Kafka"]
    H3["hop 3: broker → consumer"] --> ALO["at-least-once:<br/>commit after processing"]
    TG["Two Generals: no exactly-once<br/>delivery → class 3"] --> EO
    ALO --> IDEM["idempotent consumer<br/>dedup table, unique keys"]
    IDEM --> EO["effectively-once"]
    ALO --> RETRY["retries with backoff<br/>→ Reliability"]
    RETRY --> DLQ["dead letter queue<br/>alert, redrive"]
    VER["version-guarded updates"] --> ORD["ordering across keys"]
    REG["schema registry<br/>compatibility rules"] --> EVO["schema evolution"]

Palette components: queue · kafka · worker · notifications · sql (processed_events in the same transaction) · search (version-guarded "seats left") · email (an external side effect: back to at-least-once) · alerting (a non-empty DLQ). Scenarios: notification-system, payment-system, retry-storm. Continues in: Distributed systems — idempotency, Reliability — retry with backoff and jitter, Compute — integrations (idempotency keys to providers), Observability — alerting, Hands-on 7 — idempotent consumer, Exercise — event-driven.