External Integrations

At 10:03 on sale day a fan clicks Pay. BoxOffice's payment service calls the payment provider, and the call times out after 5 seconds. Did the card get charged? The fan, seeing a spinner, clicks Pay again. Thirty seconds later the provider sends a notification that the first charge succeeded — and then another one for the second. Meanwhile, the e-mail provider is rate-limiting BoxOffice because 40 thousand confirmation e-mails were sent in 5 minutes, and a venue-map API used on the event page is down.

BoxOffice deliberately buys its generic subdomains (services) — card processing, e-mail delivery, maps. That is the right call, and it turns each of them into a dependency you do not control: its latency, its availability, its limits, its API changes and, most importantly, the fact that it decides some outcomes asynchronously. This page covers the techniques that make such integrations safe: idempotency keys, verified webhooks, retries that cannot duplicate, reconciliation, and a plan for provider outages. The palette's EXTERNAL category — 3rd Party API, Payment, Email — is what this page is about.

What you do not control

Each row maps to a design rule:

• Isolate the provider behind an adapter (an anti-corruption layer, services): the provider's model, status names and SDK stop at one class.
• Every call has a timeout, and retries are safe (idempotency keys) — sections below.
• Protect yourself from the provider with a circuit breaker and a bulkhead per provider (reliability), so a slow provider cannot exhaust your threads (the cascading-failure scenario).
• Respect their limits: throttle on your side, honor 429 and Retry-After.
• Assume outcomes arrive asynchronously: model states explicitly, accept webhooks, reconcile.
• Observe each provider separately: latency, error rate by error code, rate-limit hits, cost.

Anatomy of a payment

Card payments are the hardest integration BoxOffice has, so we use them as the guide. The card number never touches BoxOffice: the app collects it with the provider's hosted fields and receives a token, which keeps BoxOffice out of most of the PCI DSS scope⁷. The payment service then creates a payment with that token.

sequenceDiagram
    participant F as Fan (app)
    participant O as order
    participant P as payment
    participant X as Payment provider
    F->>X: card details in hosted fields → token tok_abc
    F->>O: POST /v1/orders {holdId, token} + Idempotency-Key k-1
    O->>P: charge order ord_9 (key ord_9)
    P->>X: create payment 450.00 BRL, token, Idempotency-Key ord_9
    X-->>P: status processing
    P-->>O: PENDING
    O-->>F: 202 Accepted — order ord_9 PENDING
    Note over X: bank authorizes (maybe after 3-D Secure)
    X->>P: webhook payment.succeeded (signed)
    P->>O: PaymentSucceeded ord_9
    O->>O: confirm hold → order CONFIRMED
    O-->>F: push: your tickets are being issued

Notice what is not in the picture: a synchronous "charge succeeded" answer on which checkout depends. The provider may answer processing, may ask the fan to authenticate with the bank (requires_action, 3-D Secure), or may simply time out. The definitive outcome often arrives later, by webhook. So BoxOffice models the payment as an explicit state machine:

stateDiagram-v2
    [*] --> CREATED
    CREATED --> PENDING : provider accepted the request
    CREATED --> UNKNOWN : timeout — outcome not known
    UNKNOWN --> PENDING : status query / webhook
    PENDING --> REQUIRES_ACTION : 3-D Secure needed
    REQUIRES_ACTION --> PENDING : fan authenticated
    PENDING --> SUCCEEDED : webhook payment.succeeded
    PENDING --> FAILED : webhook payment.failed
    SUCCEEDED --> REFUNDED : hold expired / event cancelled

The UNKNOWN state is the honest answer to "the call timed out": the request may or may not have reached the provider. Never treat a timeout as a failure (you would release the seats of a fan who was charged) nor as a success. Resolve it by querying the provider or waiting for the webhook — and make sure that retrying cannot charge twice.

Idempotency keys

An operation is idempotent if doing it twice has the same effect as doing it once. POST /payments is not — unless both sides agree on an idempotency key: a unique identifier of one logical operation, sent with every attempt of that operation. The server remembers the keys it has processed and, on a repeat, returns the stored result instead of executing again¹².

BoxOffice needs keys in both directions:

• Inbound — fans and apps retry: a double click, a mobile app retrying on a flaky connection, the gateway retrying a 503. BoxOffice's POST /v1/orders and POST /v1/events/{id}/holds require an Idempotency-Key header, generated by the client once per logical action (a UUID created when the fan presses Pay, reused on every retry of that press).
• Outbound — BoxOffice retries the provider. Every call that creates a charge or a refund carries a key derived from BoxOffice's own id (ord_9, or ord_9:refund:1), so a retry after a timeout reuses it and the provider returns the original payment instead of creating a second one.

sequenceDiagram
    participant P as payment
    participant X as Payment provider
    P->>X: create payment, Idempotency-Key ord_9
    Note over X: charge created (pay_77)
    X--xP: response lost (timeout)
    Note over P: state UNKNOWN → retry with the SAME key
    P->>X: create payment, Idempotency-Key ord_9
    X-->>P: pay_77 succeeded (stored result, no new charge)

Implementing it on your own API

The provider does this for you on its side; on BoxOffice's side you implement it. The core is a table whose primary key makes "first one wins" a database guarantee:

CREATE TABLE idempotency_keys (
    client_id     text        NOT NULL,              -- keys are scoped per caller
    key           text        NOT NULL,
    request_hash  text        NOT NULL,              -- SHA-256 of method + path + body
    status        text        NOT NULL,              -- IN_PROGRESS | COMPLETED
    response_code int,
    response_body jsonb,
    locked_at     timestamptz NOT NULL DEFAULT now(),
    created_at    timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (client_id, key)
);
CREATE INDEX idempotency_keys_created_at ON idempotency_keys (created_at);  -- purge after 24 h

@Component
class IdempotentExecutor {

    private final JdbcTemplate jdbc;
    private final TransactionTemplate tx;

    IdempotentExecutor(JdbcTemplate jdbc, TransactionTemplate tx) {
        this.jdbc = jdbc;
        this.tx = tx;
    }

    StoredResponse execute(String clientId, String key, String requestHash,
                           Supplier<StoredResponse> operation) {
        // 1. Claim the key. The primary key guarantees that only one request wins.
        int claimed = jdbc.update("""
                INSERT INTO idempotency_keys (client_id, key, request_hash, status)
                VALUES (?, ?, ?, 'IN_PROGRESS')
                ON CONFLICT (client_id, key) DO NOTHING
                """, clientId, key, requestHash);

        if (claimed == 0) {                                   // 2. Seen before.
            Map<String, Object> row = jdbc.queryForMap("""
                    SELECT request_hash, status, response_code, response_body::text AS body
                      FROM idempotency_keys WHERE client_id = ? AND key = ?
                    """, clientId, key);
            if (!requestHash.equals(row.get("request_hash")))
                throw new ResponseStatusException(HttpStatus.UNPROCESSABLE_ENTITY,
                        "Idempotency-Key reused with a different request");
            if ("IN_PROGRESS".equals(row.get("status")))
                throw new ResponseStatusException(HttpStatus.CONFLICT,
                        "a request with this Idempotency-Key is still being processed");
            return new StoredResponse((Integer) row.get("response_code"), (String) row.get("body"));
        }

        // 3. First time: run the operation and store its response in ONE transaction.
        try {
            return tx.execute(status -> {
                StoredResponse response = operation.get();
                jdbc.update("""
                        UPDATE idempotency_keys
                           SET status = 'COMPLETED', response_code = ?, response_body = ?::jsonb
                         WHERE client_id = ? AND key = ?
                        """, response.code(), response.body(), clientId, key);
                return response;
            });
        } catch (RuntimeException e) {                        // nothing was committed:
            jdbc.update("DELETE FROM idempotency_keys WHERE client_id = ? AND key = ?",
                        clientId, key);                       // let the client retry
            throw e;
        }
    }
}

The decisions hidden in those lines are the ones an interviewer will probe:

When the operation also calls the provider, step 3 is not purely local: the provider call cannot be rolled back with the transaction. That is why the outbound call carries its own idempotency key derived from the order id — if BoxOffice's transaction fails after the provider charged, the retry asks the provider again with the same key and gets the same payment back. For longer multi-step operations, Brandur Leach's "atomic phases and recovery points" design generalizes this³.

public PaymentResponse charge(UUID orderId, java.math.BigDecimal amount) {
    return http.post()
            .uri("/payments")
            .header("Idempotency-Key", "order-" + orderId) // same key on every retry
            .body(new PaymentRequest(orderId, amount))
            .retrieve()
            .body(PaymentResponse.class);
}

public Result charge(String idempotencyKey, ChargeRequest req) {
    Optional<PaymentOut> done = find(idempotencyKey);
    if (done.isPresent()) {
        return new Result(done.get(), false); // a retry: answer, do not charge again
    }
    // remote call OUTSIDE any database transaction: never hold a connection while waiting
    PspClient.Charge charge = psp.charge(idempotencyKey, req.amount(), "order-" + req.orderId());
    String status = "approved".equals(charge.status()) ? "AUTHORIZED" : "DECLINED";
    jdbc.sql("""
            INSERT INTO payments (id, idempotency_key, order_id, amount, status, provider_ref)
            VALUES (?, ?, ?, ?, ?, ?)
            ON CONFLICT (idempotency_key) DO NOTHING
            """)
            .params(UUID.randomUUID(), idempotencyKey, req.orderId(), req.amount(), status, charge.id())
            .update();
    return new Result(find(idempotencyKey).orElseThrow(), true);
}

Retries and unknown outcomes

Not all failures are equal. What you may do next depends on whether the request could have had an effect:

Retries follow the rules from reliability: exponential backoff with full jitter, a small maximum number of attempts, at one layer only, and inside the caller's time budget. For a user-facing checkout that budget is short — a few seconds; after that, the payment becomes UNKNOWN/PENDING and the fan sees "we are confirming your payment", while a background job resolves it.

Webhooks

A webhook is an HTTP request that the provider sends to your URL when something happens on its side — payment.succeeded, payment.failed, charge.refunded, email.bounced. It is how asynchronous outcomes reach you without polling. Treat each delivery as an untrusted, possibly duplicated, possibly late message:

sequenceDiagram
    participant X as Payment provider
    participant H as Webhook receiver
    participant DB as inbox table
    participant Q as Queue
    participant W as payment worker
    X->>H: POST /webhooks/payments (signed)
    H->>H: verify signature + timestamp
    H->>DB: INSERT event (unique event id)
    H->>Q: enqueue event id
    H-->>X: 200 OK (within milliseconds)
    X->>H: same event again (retry)
    H->>DB: INSERT → duplicate, ignored
    H-->>X: 200 OK
    Q->>W: process event
    W->>W: PENDING → SUCCEEDED (conditional update)

Verifying signatures

Providers sign each delivery with a secret shared only with you, usually an HMAC-SHA256 over the raw body plus a timestamp and a message id. The Standard Webhooks specification, adopted by several providers, defines the headers webhook-id, webhook-timestamp and webhook-signature, and signs the string id.timestamp.body⁴; other providers use the same idea with their own header names (Stripe's Stripe-Signature, for instance⁵). A receiver in Python with FastAPI:

import base64
import hashlib
import hmac
import json
import os
import time

from fastapi import FastAPI, Header, HTTPException, Request

from adapters import inbox, queue        # the service's own persistence and queue adapters

app = FastAPI()
SECRET = base64.b64decode(os.environ["WEBHOOK_SECRET"].removeprefix("whsec_"))
TOLERANCE_S = 300                        # reject deliveries older than 5 minutes


def verify(body: bytes, msg_id: str, timestamp: str, signatures: str) -> None:
    if abs(time.time() - int(timestamp)) > TOLERANCE_S:
        raise HTTPException(status_code=400, detail="timestamp outside tolerance")   # replay
    signed = f"{msg_id}.{timestamp}.".encode() + body
    expected = base64.b64encode(hmac.new(SECRET, signed, hashlib.sha256).digest()).decode()
    for candidate in signatures.split():                  # several during secret rotation
        version, _, value = candidate.partition(",")
        if version == "v1" and hmac.compare_digest(value, expected):                # constant time
            return
    raise HTTPException(status_code=401, detail="invalid signature")


@app.post("/webhooks/payments")
async def payment_webhook(
    request: Request,
    webhook_id: str = Header(...),
    webhook_timestamp: str = Header(...),
    webhook_signature: str = Header(...),
):
    body = await request.body()          # the RAW bytes: re-serialized JSON would not match
    verify(body, webhook_id, webhook_timestamp, webhook_signature)
    event = json.loads(body)
    if await inbox.insert_if_absent(event_id=webhook_id, payload=event):   # unique event id
        await queue.publish("payment.provider-events", {"eventId": webhook_id})
    return {"received": True}            # 2xx fast; a worker does the real processing

Details that matter: verify over the raw bytes (parsing and re-serializing JSON changes whitespace and key order); compare with a constant-time function (hmac.compare_digest) so the comparison time does not leak the signature; accept several signatures during secret rotation; keep the handler tiny so it always answers within the provider's timeout.

Out of order and stale events

The worker applies each event as a conditional state transition, so a late or duplicated event cannot move a payment backwards:

UPDATE payments
   SET status = 'SUCCEEDED',
       provider_updated_at = :occurred_at
 WHERE provider_payment_id = :provider_payment_id
   AND status IN ('CREATED', 'UNKNOWN', 'PENDING', 'REQUIRES_ACTION');
-- 0 rows updated = already final (duplicate or stale event): acknowledge and ignore

Many teams also treat the webhook as a notification only ("something changed on pay_77") and fetch the current object from the provider's API before acting. It costs one call, but the state you act on is always the latest and authoritative one.

Webhooks or polling?

Use both: webhooks as the fast path, and a sweeper that polls the provider for payments still UNKNOWN or PENDING after, say, 10 minutes.

Reconciliation

Idempotency and webhooks make each interaction safe. They do not guarantee that, at the end of the day, BoxOffice's records agree with the provider's: a webhook may be lost after the provider's retries ran out, a support agent may refund a payment in the provider's dashboard, a bug may skip a state update, and the provider deducts fees and chargebacks you only learn about from reports.

Reconciliation is the periodic job that compares the two sides and resolves differences — standard practice in any system that moves money⁶.

flowchart LR
    report["Provider report<br/>or list API since T"] --> load["Load into<br/>provider_payments"]
    ours[("payments<br/>(BoxOffice)")] --> match{"Match by<br/>provider id"}
    load --> match
    match -->|"equal"| ok["Matched ✓"]
    match -->|"theirs SUCCEEDED,<br/>ours PENDING"| fix["Auto-fix:<br/>confirm or refund"]
    match -->|"only in one side,<br/>amount differs"| human["Exception queue<br/>(finance on-call)"]

SELECT coalesce(p.id, r.reference)  AS payment,
       p.status                      AS ours,
       r.status                      AS theirs,
       p.amount_cents                AS our_amount,
       r.amount_cents                AS their_amount
  FROM payments p
  FULL OUTER JOIN provider_payments r
         ON r.provider_payment_id = p.provider_payment_id
 WHERE (p.created_at >= :day AND p.created_at < :day + interval '1 day'
        OR r.created_at >= :day AND r.created_at < :day + interval '1 day')
   AND (p.id IS NULL
        OR r.provider_payment_id IS NULL
        OR p.status <> r.status
        OR p.amount_cents <> r.amount_cents);

Run it at two speeds: a near-real-time sweeper (every few minutes, for payments stuck in UNKNOWN/PENDING, by querying the provider's API) and a daily reconciliation against the provider's settlement report, which also checks fees and payouts against the bank. Larger systems record every money movement in a double-entry ledger so that the books always balance and every difference has an explanation.

Provider outages

Providers fail like everything else — and their outage arrives on your busiest day, because your busiest day is their busiest day too. Decide in advance, per provider, what BoxOffice does:

flowchart LR
    n["notification worker"] --> cb{"circuit breaker<br/>primary ESP"}
    cb -->|"closed"| p["Primary e-mail provider"]
    cb -->|"open"| s["Secondary e-mail provider"]
    p -.->|"429 / 5xx"| retry["retry queue<br/>(backoff)"]
    s -.->|"429 / 5xx"| retry
    retry --> n

The BoxOffice decisions:

• Payment provider down → the breaker opens after a burst of failures; new checkouts fail fast with a clear message, and active holds are extended by the outage duration so fans do not lose their seats while waiting. Payments already in flight stay UNKNOWN and are resolved by the sweeper when the provider returns. (A second payment provider is on the roadmap, behind the same PaymentGateway port.)
• E-mail provider down or rate-limiting → messages wait in the queue; the notification worker throttles to the provider's limit; after a sustained outage, traffic fails over to a secondary provider. Ticket links remain available in the app, so a late e-mail is an inconvenience, not a lost sale.
• Venue-map API down → the event page renders without the interactive map (cached image fallback).

Try it in the playground

Open the payment-system scenario: checkout calls a payment service that calls an external provider, with retries and idempotency.

1. Raise the chaos level to high so the provider has outages and slow periods; watch checkout p99 and the error rate.
2. Set retries without backoff on the payment edge: the provider receives a retry storm and stays down longer. Switch to exponential backoff with jitter.
3. Add a timeout and a circuit breaker on the edge to the provider: checkout fails fast instead of piling up threads, and recovers when the provider does.

Open the full simulator

Security of integrations

• Secrets (API keys, webhook secrets) live in a secret manager, never in the repository; rotate them, and use restricted keys with only the permissions a service needs (network & security).
• Minimize PCI DSS scope: tokenize with the provider's hosted fields or payment page so that card numbers never cross BoxOffice's network.
• Egress control: calls to providers leave through a NAT gateway with fixed IPs (some providers allowlist them); deny other outbound traffic.
• Treat provider responses as input: validate them; a compromised or buggy provider must not be able to inject data into your system unchecked.

Trade-offs at a glance

How it connects

flowchart LR
    ADP["one adapter per provider<br/>ports and adapters"] --> RES["timeouts, circuit breaker,<br/>bulkhead → Reliability"]
    IK["idempotency keys<br/>inbound and outbound"] --> RETRY["safe retries"]
    TO["a timeout means UNKNOWN"] --> SM["payment state machine"]
    RETRY --> SM
    SM --> WH["webhooks: signature,<br/>dedup by event id"]
    SM --> REC["reconciliation with<br/>the provider's records"]
    WH --> ASYNC["ack fast, process async<br/>→ Messaging"]
    OUT["provider outages"] --> STRAT["fail fast, queue, fail over,<br/>degrade, extend holds"]
    STRAT --> RES
    IK -.-> DIST["idempotency<br/>→ Distributed systems"]

Palette components: payment (the provider behind the payment adapter) · email (rate limits, bounces, a secondary provider) · thirdparty (the venue-map API: degrade without it) · app (the adapter service) · sql (idempotency keys under a unique constraint, payment states) · queue (deliver later; process webhooks asynchronously) · scheduler (sweepers and reconciliation jobs) · serverless (a cheap webhook receiver). Scenarios: payment-system, cascading-failure, retry-storm, notification-system. Continues in: Reliability — circuit breaker, Distributed — idempotency, Distributed — transactions, Case study: payment system, Hands-on 9.

Key takeaways