Async Work

When the payment for an order succeeds, BoxOffice has a lot to do: mark the order confirmed, turn the hold into sold seats, render a ticket PDF with a QR code (about 800 ms of CPU), e-mail it through the provider (300 ms, sometimes 5 s), send a push notification, update the sales dashboard, and re-index availability for search. If the checkout request does all of that before answering, the fan waits seconds, every slow provider slows checkout, and an e-mail outage fails ticket sales.

None of that work changes the answer the fan is waiting for — "your order is confirmed". It can happen a moment later, somewhere else. This page is about that "somewhere else": workers that process jobs from a queue, and schedulers that start work at a given time — and about making both correct when jobs are retried and replicas run in parallel.

What belongs in the request path?

Ask one question about every step: does the user's answer depend on it?

Everything below the first row becomes a job (a unit of work described by a small message) placed on a queue and executed by workers (processes that pull jobs and run them).

sequenceDiagram
    participant F as Fan
    participant O as order
    participant Q as Queue / Kafka
    participant W as ticket worker
    participant N as notification worker
    participant E as E-mail provider
    F->>O: POST /orders/ord_9/confirm
    O->>O: order CONFIRMED (one DB transaction)
    O->>Q: OrderConfirmed {orderId: ord_9}
    O-->>F: 200 Your order is confirmed
    Q->>W: render tickets for ord_9
    W->>W: PDF + QR → object store
    W->>Q: TicketsIssued {orderId: ord_9}
    Q->>N: TicketsIssued
    N->>E: send e-mail with link
    E-->>N: 202 queued

What you gain:

• Latency: checkout answers in tens of milliseconds instead of seconds.
• Availability: the e-mail provider can be down for 20 minutes; jobs wait in the queue and checkout keeps selling.
• Load leveling: the queue absorbs the on-sale spike; workers drain it at their own, steady pace.
• Independent scaling: ticket rendering is CPU-heavy, e-mail is I/O-bound — each worker pool scales on its own.

What you pay: the result is eventually there, not immediately; you need to show status ("your tickets are being issued"), handle failures without a user to show an error to, and make every job safe to run more than once.

@Transactional
public PlaceResult place(OrderRequest req) {
    int inserted = jdbc.sql("""
            INSERT INTO orders (id, show_id, hold_id, customer_email, amount, status)
            VALUES (?, ?, ?, ?, ?, 'PENDING_PAYMENT')
            ON CONFLICT (hold_id) DO NOTHING
            """)
            .params(UUID.randomUUID(), req.showId(), req.holdId(), req.customerEmail(), req.amount())
            .update();

    OrderView order = jdbc.sql(SELECT + " WHERE hold_id = ?")
            .param(req.holdId())
            .query(OrderView.class)
            .single();

    if (inserted == 0) {
        // same hold already ordered: idempotent replay, no second event
        return new PlaceResult(order, false);
    }

    OrderEvent event = OrderEvent.placed(order);
    if (props.notifyMode() == NotifyMode.SYNC) {
        notifications.send(event); // BEFORE: the fan waits for the e-mail
    } else {
        outbox.append(event.eventId(), "order", order.id(), event.eventType(), event);
    }
    return new PlaceResult(order, true);
}

How many workers?

Workers are sized like any server, with Little's law: the number of jobs in progress equals the arrival rate times the time each job takes.

When arrivals exceed capacity, the excess accumulates as backlog, and the backlog drains only at the difference between capacity and arrivals:

Two lessons from the plot. First, a queue converts a capacity problem into a delay: nothing failed with 40 workers, fans just waited. Second, the downstream sets the real limit: if the e-mail provider accepts at most, say, 100 messages per second on your plan, 500 notification workers do not send faster — they only collect 429 Too Many Requests. Size the worker pool for the slowest dependency and let the queue hold the rest.

Scale workers on queue depth or consumer lag, not CPU: a worker waiting on a slow provider uses no CPU but is not free. Serverless & autoscaling shows how (KEDA).

Try it in the playground

Open the async-queue scenario: producers write to a queue drained by a pool of workers.

1. Run the spike profile and watch backlog and consumer lag climb during the burst, then fall.
2. Double the worker replicas: the peak backlog drops and drains sooner — check it against the formula above.
3. Lower the downstream's rate limit: more workers stop helping, and errors appear instead.

Designing jobs that survive the real world

A queue is not magic: workers crash in the middle of jobs, brokers redeliver, providers time out. Most brokers give at-least-once delivery: a job is removed (acknowledged) only after the worker says it finished, so a crash before the ack means the job is delivered again. Every rule below follows from that.

A poison message is one that fails every time (a malformed payload, a bug). Without a retry limit it is retried forever and, on ordered partitions, blocks every message behind it. The DLQ parks it for a human (or a fixed version of the code) and lets the queue move on. Monitor the DLQ size: a growing DLQ is an incident.

With Spring for Apache Kafka, retries with backoff and a dead-letter topic are a few annotations³:

@Component
class TicketJobListener {

    private final TicketService tickets;

    TicketJobListener(TicketService tickets) { this.tickets = tickets; }

    @RetryableTopic(
        attempts = "4",                                    // 1 try + 3 retries
        backOff = @BackOff(delay = 1000, multiplier = 2.0), // 1 s, 2 s, 4 s (Spring Kafka 4)
        dltTopicSuffix = ".dlt")
    @KafkaListener(topics = "order.confirmed.v1", groupId = "ticket-worker")
    void onOrderConfirmed(OrderConfirmed event) {
        if (tickets.alreadyIssued(event.orderId())) {
            return;                                        // idempotent: duplicate delivery
        }
        tickets.renderAndStore(event.orderId());           // PDF + QR → object store
    }

    @DltHandler
    void onDeadLetter(OrderConfirmed event) {
        tickets.markFailed(event.orderId());               // visible to support; alert on rate
    }
}

Where jobs live

Durable execution engines deserve a mention: when a process has many steps and waits — "event cancelled → refund 60,000 orders → e-mail each fan → wait 7 days → close the accounting" — writing it as a workflow, whose progress the engine persists after every step, is simpler and safer than chaining queues by hand⁴.

Schedulers

Some work is triggered by time, not by a request or an event:

• every 30 s, release expired holds so their seats can be sold again;
• at 10:00:00, open the sale for an event;
• 24 h before each show, send a reminder to every ticket holder;
• every night, reconcile payments with the provider's report (integrations).

The duplicate-cron problem

The obvious implementation is a @Scheduled method in the service. It works on one replica. With 10 replicas, the job runs 10 times: 10 reminder e-mails per fan, 10 concurrent sweeps fighting over the same rows. The same happens with a cron entry copied to every VM. There are four standard fixes:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: release-expired-holds
spec:
  schedule: "* * * * *"              # every minute (cron's finest granularity)
  concurrencyPolicy: Forbid          # never two runs at the same time
  startingDeadlineSeconds: 30        # skip a run that could not start in time
  successfulJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 50      # kill a run that hangs
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: sweeper
              image: ghcr.io/boxoffice/inventory:1.8.2
              args: ["--job=release-expired-holds"]

@Component
class HoldSweeper {

    private static final Logger log = LoggerFactory.getLogger(HoldSweeper.class);
    private final HoldRepository holds;

    HoldSweeper(HoldRepository holds) { this.holds = holds; }

    @Scheduled(fixedRate = 30_000)
    @SchedulerLock(name = "releaseExpiredHolds",
                   lockAtMostFor = "PT25S",   // lock expires even if this replica dies
                   lockAtLeastFor = "PT5S")   // absorb small clock differences
    void releaseExpiredHolds() {
        int released = holds.releaseExpired(Instant.now(), 5_000);   // batch of 5,000
        log.info("released {} expired holds", released);
    }
}

Locks expire: fencing tokens

A distributed lock with an expiry has a subtle hole, described by Martin Kleppmann¹: a replica takes the lock, then pauses (a long garbage collection, a VM migration) beyond the lock's expiry. Another replica takes the lock and starts the job. The first one wakes up, still believing it holds the lock, and both write.

sequenceDiagram
    participant A as Replica A
    participant L as Lock store
    participant B as Replica B
    participant DB as Database
    A->>L: acquire (token 33), expires in 25 s
    Note over A: GC pause: 40 s
    B->>L: lock expired → acquire (token 34)
    B->>DB: release holds (token 34)
    A->>DB: release holds (token 33) — still thinks it owns the lock
    DB-->>A: rejected: token 33 < 34 already seen

The fix is a fencing token: the lock store hands out an increasing number with each acquisition, and the protected resource rejects writes carrying an older token. When the resource cannot check tokens, the job itself must be idempotent — which you need anyway. A lock is an optimization that avoids duplicated effort; correctness must come from idempotency or from the database (a conditional UPDATE ... WHERE status = 'ACTIVE' AND expires_at < now() releases each hold at most once, no matter how many sweepers run).

Delayed work: three ways to expire a hold

Releasing a hold 10 minutes after it was created is a delayed job. Three designs, often combined:

BoxOffice's rule: correctness never waits for the scheduler. A hold whose expires_at has passed cannot be confirmed, whether the sweeper has run or not (lazy expiry), and a sweeper returns its seats to sale. If the scheduler is down for 5 minutes, no seat is sold twice; the seats of expired holds only come back later. (A fully lazy design goes further: new holds may take the seats of an expired hold at once, without waiting for the sweeper.)

/** Payment succeeded (lab 7 calls this): seats become SOLD. */
@Transactional
public HoldOut confirm(String userId, UUID holdId) {
    int changed = jdbc.sql("""
            UPDATE inventory.holds SET status = 'CONFIRMED'
            WHERE id = :id AND user_id = :userId AND status = 'ACTIVE' AND expires_at > now()""")
            .param("id", holdId).param("userId", userId)
            .update();
    if (changed == 0) {
        throw new ResponseStatusException(HttpStatus.CONFLICT, "hold is not active (expired, released or unknown)");
    }
    jdbc.sql("UPDATE inventory.seats SET status = 'SOLD', version = version + 1 WHERE hold_id = :id")
            .param("id", holdId).update();
    return find(userId, holdId);
}
@Scheduled(fixedDelay = 5000)
@Transactional
public void expireHolds() {
    int freed = jdbc.sql("""
            WITH expired AS (
                UPDATE inventory.holds SET status = 'EXPIRED'
                WHERE id IN (SELECT id FROM inventory.holds
                             WHERE status = 'ACTIVE' AND expires_at < now()
                             ORDER BY expires_at
                             LIMIT 500
                             FOR UPDATE SKIP LOCKED)
                RETURNING id)
            UPDATE inventory.seats
            SET status = 'AVAILABLE', hold_id = NULL, version = version + 1
            WHERE status = 'HELD' AND hold_id IN (SELECT id FROM expired)""")
            .update();
    if (freed > 0) {
        log.info("expired holds freed {} seats", freed);
    }
}

The notification-system scenario combines a scheduler, a queue and the e-mail/push providers. Try raising the scheduler's job count to simulate the 24-hour reminder burst, and watch where the backlog builds up.

Open the full simulator

Trade-offs at a glance

How it connects

flowchart LR
    RP["request path: only what<br/>the answer depends on"] --> Q["jobs on a queue<br/>→ Messaging"]
    Q --> W["workers<br/>rate × time per job"]
    W --> JOB["idempotent jobs, ack after work,<br/>retries with backoff"]
    JOB --> DLQ["DLQ for poison messages"]
    W --> SCALE["scale on queue lag<br/>→ Autoscaling"]
    CRON["scheduled work<br/>on N replicas"] --> LOCK["CronJob, distributed lock,<br/>leader election"]
    LOCK --> FENCE["locks expire: fencing tokens<br/>→ Distributed consensus"]
    CRON --> LAZY["lazy expiry + sweeper<br/>seat holds → Databases"]

Palette components: queue (jobs and delayed messages) · worker (consumers sized by Little's law) · scheduler (cron and delayed work, once per cluster) · kafka (commit the offset after the work) · sql (ShedLock row, holds that expire lazily) · objstore (ticket PDFs) · notifications · email (the ticket e-mail, off the request path). Scenarios: async-queue, notification-system. Continues in: Serverless & Autoscaling — scaling on queue depth, Messaging — queues and pub/sub, Messaging — delivery semantics, Distributed — consensus (fencing), Hands-on 7.

Key takeaways