Payment System

"Design the payment system of a ticketing platform. Fans pay by card or Pix through an external payment provider; promoters get paid after the show. Nobody may be charged twice, and every cent must be accounted for."

Payments are where "eventually consistent" stops being an acceptable answer. A fan charged twice calls the bank; a promoter paid twice is money you will never see again; a ticket issued without payment is a gift. Yet the system talks to an external provider over an unreliable network, which means timeouts with unknown outcomes, duplicated webhooks and money that arrives days later in a settlement file. The interview tests whether you can turn an at-least-once world into an exactly-once effect, and whether you know how money is recorded: a double-entry ledger, not a balance column.

This is BoxOffice's payment service — the adapter behind the order flow of the ticketing case. That page models the order as a state machine (HELD → PENDING_PAYMENT → CONFIRMED) and uses webhooks and reconciliation; this page goes one level deeper into what happens inside payment: idempotency, the ledger, the provider integration and the daily reconciliation.

1. Clarify the requirements

Payments have more hidden requirements than any other case. Ask (interview framework).

Functional requirements

1. Pay-in: charge the fan for an order (card, Pix, wallet) through an external PSP; support 3-D Secure challenges.
2. Refunds: full or partial, e.g. when a show is cancelled.
3. Ledger: record every money movement (charge, PSP fee, platform fee, promoter payable, refund, payout).
4. Payouts: pay promoters their share after the event (batch, out of the request path).
5. Reconciliation: every day, match the provider's settlement report against the ledger and flag differences.
6. Notify the order service of outcomes (confirm, fail) so tickets are issued.

Non-functional requirements

Out of scope (say it explicitly)

Building our own card processing (we are not an acquirer), fraud models (mention a fraud check hook), multi-currency FX, invoicing and taxes, chargeback disputes in detail.

2. Back-of-the-envelope estimates

# BoxOffice payments: assumptions (illustrative)
orders_day     = 300_000      # paid orders per day
peak_tps       = 300          # payments/s when several on-sales open at 10:00
avg_order      = 230.0        # currency units per order (tickets + fees)
psp_s          = 2.0          # provider latency for one payment call
entries_per    = 4            # ledger lines per captured payment
entry_b        = 200          # one ledger line, bytes
idem_b         = 1_000        # one idempotency record (key, hash, response)
webhooks_per   = 2            # e.g. authorized + captured
dup_rate       = 0.001        # share of payments that would be duplicated without idempotency

avg_tps        = orders_day / 86_400
inflight_peak  = peak_tps * psp_s                 # Little's Law: L = λ·W
ledger_lines   = orders_day * entries_per
ledger_gb_year = ledger_lines * entry_b * 365 / 1e9
idem_mb_day    = orders_day * idem_b / 1e6
webhooks_day   = orders_day * webhooks_per
gmv_day_m      = orders_day * avg_order / 1e6     # millions per day
dup_charges    = orders_day * dup_rate

rows = [
    ("avg_tps", avg_tps, "payments/s"), ("inflight_peak", inflight_peak, "calls"),
    ("ledger_lines", ledger_lines, "lines/day"), ("ledger_gb_year", ledger_gb_year, "GB/year"),
    ("idem_mb_day", idem_mb_day, "MB/day"), ("webhooks_day", webhooks_day, "events/day"),
    ("gmv_day_m", gmv_day_m, "M/day"), ("dup_charges", dup_charges, "charges/day"),
]
print("| quantity | value | unit |\n|---|---:|---|")
for name, value, unit in rows:
    v = f"{value:,.0f}" if abs(value) >= 100 else f"{value:,.2f}"
    print(f"| `{name}` | {v} | {unit} |")

What the numbers tell us:

• On average only ~3.5 payments/s; the peak of 300/s is set by on-sales. With a 2 s provider, ~600 calls are in flight at the peak (Little's Law). ⇒ size the adapter's connection pool and bulkhead for that, and remember the provider has its own rate limit.
• The ledger grows by 1.2 M lines/day, ~88 GB/year: a single, well-indexed PostgreSQL with a standby holds years of it. Payments are a correctness problem, not a scale problem.
• Idempotency records take ~300 MB/day; keep them for 24 h–7 days and they fit anywhere.
• Even a 0.1% duplicate rate would mean ~300 double charges per day — 300 angry fans and chargebacks. That is the number that justifies idempotency everywhere.
• ~69 M currency units/day flow through the system: reconciliation differences of a fraction of a percent are still real money.

3. API

POST /v1/payments                                   → 201 {paymentId, status, nextAction?}
     Idempotency-Key: order-7f3c…                    (one key per order attempt)
     {"orderId": "7f3c…", "amount": 23000, "currency": "BRL",
      "method": "card", "paymentToken": "tok_1Q…", "returnUrl": "https://…"}
     status: REQUIRES_ACTION (3-D Secure) | PROCESSING | SUCCEEDED | FAILED

GET  /v1/payments/{paymentId}                       → 200 {status, amount, refundedAmount, events[]}
POST /v1/payments/{paymentId}/refunds               → 201 {refundId, status}
     Idempotency-Key: refund-…
     {"amount": 10000, "reason": "event_cancelled"}

POST /v1/webhooks/psp                               ← provider → us, HMAC-signed
     PSP-Signature: t=1767261600,v1=5257a8…
     {"id": "evt_9a…", "type": "payment.captured", "data": {"pspPaymentId": "pi_3N…", …}}

• The fan's card data goes from the browser straight to the PSP (hosted fields or a redirect); we receive only a token (paymentToken). That is what keeps our servers out of most of the PCI DSS scope.
• nextAction tells the client to run the 3-D Secure challenge with the issuing bank; the result arrives later as a webhook.
• Every mutating call takes an Idempotency-Key; webhooks carry an event id that we deduplicate on.

4. Data model

CREATE TABLE payment (
    payment_id      UUID PRIMARY KEY,
    order_id        UUID        NOT NULL UNIQUE,      -- one payment per order attempt
    amount_cents    BIGINT      NOT NULL,
    currency        CHAR(3)     NOT NULL,
    status          TEXT        NOT NULL,             -- CREATED | REQUIRES_ACTION | PROCESSING
                                                      -- | UNKNOWN | SUCCEEDED | FAILED | REFUNDED
    psp             TEXT        NOT NULL,
    psp_payment_id  TEXT        UNIQUE,
    version         INT         NOT NULL DEFAULT 0,
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE idempotency_key (
    key             TEXT PRIMARY KEY,
    request_hash    TEXT        NOT NULL,             -- SHA-256 of method + path + body
    status          TEXT        NOT NULL,             -- IN_PROGRESS | DONE
    response_code   INT,
    response_body   JSONB,
    locked_until    TIMESTAMPTZ,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE ledger_entry (                           -- append-only: no UPDATE, no DELETE
    entry_id        BIGSERIAL PRIMARY KEY,
    txn_id          UUID        NOT NULL,             -- groups the lines of one transaction
    account         TEXT        NOT NULL,             -- 'psp_receivable', 'promoter_payable:88', …
    amount_cents    BIGINT      NOT NULL,             -- debit > 0, credit < 0
    currency        CHAR(3)     NOT NULL,
    payment_id      UUID,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE processed_webhook (event_id TEXT PRIMARY KEY, received_at TIMESTAMPTZ DEFAULT now());
CREATE TABLE outbox (id BIGSERIAL PRIMARY KEY, topic TEXT, payload JSONB, created_at TIMESTAMPTZ DEFAULT now());

5. High-level design

flowchart LR
    fan([Fan browser]) -->|"card data"| psp[["PSP<br/>hosted fields, 3-D Secure"]]
    fan --> gw["API gateway"]
    gw --> ord["order"]
    ord --> pay["payment service"]
    pay --> db[("PostgreSQL<br/>payment, idempotency,<br/>ledger, outbox")]
    pay -->|"charge(token, key)"| psp
    psp -. "signed webhook" .-> wh["webhook handler"]
    wh --> db
    db -. outbox relay .-> k[["Kafka<br/>payment events"]]
    k --> ord
    k --> notif["notification"]
    q[["Retry queue + DLQ"]] --> wk["workers<br/>retries, refunds, payouts"]
    pay --> q
    wk --> psp
    sched["scheduler<br/>reconciliation"] --> rec["reconciliation job"]
    psp -. "settlement file (daily)" .-> rec
    rec --> dwh[("Data warehouse")]
    rec --> db

A card payment with 3-D Secure, end to end:

sequenceDiagram
    autonumber
    participant F as Fan
    participant O as order
    participant P as payment
    participant D as PostgreSQL
    participant S as PSP
    F->>S: card data (hosted fields) → token
    F->>O: POST /orders/{id}/pay {token}
    O->>P: POST /payments (Idempotency-Key = order attempt)
    P->>D: insert idempotency key IN_PROGRESS + payment CREATED
    P->>S: create payment (idempotency key = paymentId)
    S-->>P: requires_action (3-D Secure)
    P->>D: status REQUIRES_ACTION, key DONE + stored response
    P-->>F: nextAction: 3-D Secure challenge
    F->>S: completes challenge with the bank
    S-->>P: webhook payment.captured (signed)
    P->>D: one transaction: status SUCCEEDED + 4 ledger lines + outbox
    D-)O: PaymentSucceeded (via Kafka)
    O-->>F: order CONFIRMED, tickets issued

6. Deep dives

6.1 Exactly-once effect: idempotency keys

There is no exactly-once delivery over a network: a request whose response was lost may or may not have been executed. What we can build is an exactly-once effect: the client retries until it gets an answer (at-least-once), and the server makes the operation idempotent — executing it twice has the same effect as once (delivery semantics).

The server-side algorithm for POST /v1/payments with Idempotency-Key: K:

flowchart TD
    r["Request with key K"] --> ins{"INSERT key K<br/>IN_PROGRESS<br/>(unique)"}
    ins -->|"inserted"| run["Execute: create payment,<br/>call PSP"]
    run --> save["Store response,<br/>status DONE"]
    save --> ok["Return response"]
    ins -->|"conflict"| chk{"Same request_hash?"}
    chk -->|"no"| e422["422: key reused<br/>with different body"]
    chk -->|"yes, DONE"| replay["Replay stored response"]
    chk -->|"yes, IN_PROGRESS"| e409["409: still processing,<br/>retry later"]

The same idea applies outward: our call to the PSP carries our idempotency key (the paymentId), so if the adapter times out and retries, the provider returns the existing payment instead of charging again. Stripe-style APIs accept an Idempotency-Key header; with providers that do not, use a unique merchant reference and look it up before creating a new charge.

# key choice matters
Idempotency-Key: order-7f3c-attempt-1   ✔ one key per (order, attempt): a retry of the same attempt is a no-op,
                                           a new card after a decline is a new attempt
Idempotency-Key: <random per HTTP call> ✘ every retry looks new → double charge
Idempotency-Key: user-42                ✘ blocks every future purchase of that user

public Result charge(String idempotencyKey, ChargeRequest req) {
    Optional<PaymentOut> done = find(idempotencyKey);
    if (done.isPresent()) {
        return new Result(done.get(), false); // a retry: answer, do not charge again
    }
    // remote call OUTSIDE any database transaction: never hold a connection while waiting
    PspClient.Charge charge = psp.charge(idempotencyKey, req.amount(), "order-" + req.orderId());
    String status = "approved".equals(charge.status()) ? "AUTHORIZED" : "DECLINED";
    jdbc.sql("""
            INSERT INTO payments (id, idempotency_key, order_id, amount, status, provider_ref)
            VALUES (?, ?, ?, ?, ?, ?)
            ON CONFLICT (idempotency_key) DO NOTHING
            """)
            .params(UUID.randomUUID(), idempotencyKey, req.orderId(), req.amount(), status, charge.id())
            .update();
    return new Result(find(idempotencyKey).orElseThrow(), true);
}

6.2 The unknown outcome

The adapter calls the PSP and gets a timeout. Was the card charged? You do not know — and this is the single most important sentence of the interview.

stateDiagram-v2
    [*] --> CREATED
    CREATED --> PROCESSING: call PSP
    PROCESSING --> REQUIRES_ACTION: 3-D Secure
    REQUIRES_ACTION --> PROCESSING: challenge done
    PROCESSING --> SUCCEEDED: PSP ok / webhook captured
    PROCESSING --> FAILED: declined
    PROCESSING --> UNKNOWN: timeout / 5xx
    UNKNOWN --> SUCCEEDED: status query or webhook says captured
    UNKNOWN --> FAILED: status query says not found / failed
    SUCCEEDED --> REFUNDED: refund
    SUCCEEDED --> [*]
    FAILED --> [*]

• Never map a timeout to FAILED: if the charge went through, the fan pays for nothing; if you retry with a new key, they pay twice.
• Keep the payment in UNKNOWN and resolve it: retry with the same idempotency key (safe), query the PSP by your reference, or wait for the webhook. A scheduler job sweeps payments stuck in UNKNOWN for more than a few minutes.
• Meanwhile the order stays PENDING_PAYMENT and the seat hold is extended (see the ticketing state machine); the fan sees "confirming your payment…".
• Timeouts on the PSP call are long enough for the provider's normal p99 (seconds), wrapped in a circuit breaker and a bulkhead so a slow provider cannot exhaust the payment service (reliability patterns).

6.3 The double-entry ledger

A balance column that you UPDATE loses history, cannot explain itself to an auditor and breaks silently under concurrent updates. Accounting solved this centuries ago: double-entry bookkeeping. Every transaction is a set of lines, each moving money into (debit) or out of (credit) an account, and the lines of every transaction sum to zero. Balances are derived by summing lines; nothing is ever updated or deleted — a mistake is fixed by a new, reversing transaction.

A fan buys a 200.00 ticket plus a 30.00 service fee; the PSP keeps a 2.5% fee; the promoter is paid after the show:

# Double-entry ledger for one BoxOffice ticket sale (amounts in cents; debit > 0, credit < 0)
from collections import defaultdict

transactions = {
    "T1 capture": [                       # fan pays 230.00 through the PSP
        ("psp_receivable",   +22_425),    # PSP owes us 230.00 minus its fee
        ("psp_fees",          +575),      # 2.5% of 230.00 is our expense
        ("promoter_payable", -20_000),    # we owe the promoter the ticket price
        ("platform_revenue", -3_000),     # the service fee is ours
    ],
    "T2 settlement": [                    # PSP pays out to our bank two days later
        ("bank",             +22_425),
        ("psp_receivable",   -22_425),
    ],
    "T3 payout": [                        # after the show we pay the promoter
        ("promoter_payable", +20_000),
        ("bank",             -20_000),
    ],
}

balances = defaultdict(int)
print("| transaction | lines | sum |\n|---|---:|---:|")
for name, lines in transactions.items():
    total = sum(amount for _, amount in lines)
    assert total == 0, f"{name} does not balance"
    for account, amount in lines:
        balances[account] += amount
    print(f"| {name} | {len(lines)} | {total} |")

print("\n| account | balance |\n|---|---:|")
for account, cents in sorted(balances.items()):
    print(f"| `{account}` | {cents / 100:,.2f} |")
print(f"\nSum of all balances: {sum(balances.values())}")

Read the result: after the three transactions the bank holds 24.25 — exactly the platform revenue (30.00) minus the PSP fee (5.75); the receivable from the PSP and the payable to the promoter are zero, as they should be once settled. If any of these balances is not what the business expects, the ledger tells you which transaction caused it.

Engineering rules for the ledger:

• Append-only at the database level (revoke UPDATE/DELETE for the application role). Refunds and corrections are new transactions.
• Write all lines of a transaction in one database transaction, together with the payment state change and the outbox row. Check the zero-sum invariant in code and with a periodic query (SELECT txn_id FROM ledger_entry GROUP BY txn_id HAVING sum(amount_cents) <> 0 must return nothing).
• Integer minor units (cents) and an explicit currency — never floating point for money.
• Account per counterparty where needed (promoter_payable:88) so the payout job knows what each promoter is owed.

6.4 Webhooks, outbox and reconciliation

Webhooks are how asynchronous methods (Pix, 3-D Secure, bank transfers) report outcomes. The handler must:

1. Verify the signature (HMAC over timestamp + body with a shared secret) and reject old timestamps — otherwise anyone can POST "payment succeeded".
2. Deduplicate on the provider's event id (INSERT INTO processed_webhook … ON CONFLICT DO NOTHING): providers deliver at least once.
3. Apply the change as a conditional transition (… WHERE status IN ('PROCESSING','UNKNOWN')), so out-of-order events (a failed after a captured) cannot move the state backwards.
4. Return 2xx quickly and do heavy work asynchronously; a slow handler makes the provider retry, multiplying load.

Outbox. The payment state, ledger lines and an outbox row are committed together; a relay publishes the outbox to Kafka. The order service never learns about a payment that was rolled back, and never misses one that was committed (messaging patterns).

Reconciliation is the safety net for everything above. Once a day the PSP publishes a settlement report (every captured payment, refund, fee and payout). A scheduled job loads it into the warehouse and matches it, line by line, against the ledger:

Retries toward the provider (refunds, payouts, status queries) run from a queue with exponential backoff and jitter; after N attempts the message goes to a dead-letter queue watched by humans. Money operations are never retried forever blindly.

7. Bottlenecks, failure modes and trade-offs

The trade-offs to say out loud: consistency over availability (if the payment database is down, stop taking payments rather than record them "later"), latency over certainty at the edge (answer the fan with "processing" and resolve asynchronously instead of holding the request until the truth is known), and outsourcing card handling to the PSP (less PCI scope and fraud work, in exchange for dependency and fees).

8. Try it in the playground

The payment-system scenario models the payment service, its database and an external payment provider with retries and idempotency.

Open the full playground

9. Follow-up questions an interviewer asks

10. Self-assessment rubric

Related: Integrations (webhooks, idempotency), Messaging patterns (outbox, saga), Ticketing case, Hands-on 7 (outbox), Hands-on 9 (resilience between order and payment).

How it connects

Which classes does this case test? Each branch is a class whose ideas the design relies on (blue), with the palette components where they show up (green):

flowchart LR
    CASE(["Payment system"]) --> DIS["Distributed<br/>exactly-once effect, unknown outcome"]:::tech
    CASE --> CMP["Compute<br/>PSP integration, webhooks, reconciliation"]:::tech
    CASE --> DB["Databases<br/>double-entry ledger, transactions"]:::tech
    CASE --> MSG["Messaging<br/>outbox, payment events"]:::tech
    CASE --> REL["Reliability<br/>backoff, retry storms, breakers"]:::tech
    CASE --> SEC["Network & Security<br/>PCI DSS scope, hosted fields"]:::tech
    DIS -.-> k1["sql<br/>(idempotency keys)"]:::comp
    CMP -.-> k2["payment · scheduler · dwh"]:::comp
    MSG -.-> k3["kafka · queue · worker"]:::comp
    SEC -.-> k4["apigw · app"]:::comp
    classDef comp fill:#1e9e5a22,stroke:#1e9e5a
    classDef tech fill:#2f6fed22,stroke:#2f6fed

Classes it tests: Distributed — idempotency · Distributed — transactions · Compute — integrations · Databases — transactions · Messaging — patterns · Reliability — patterns · Network & Security · Hands-on 7 and Hands-on 9. Palette components: apigw · app · sql · kafka · queue · worker · scheduler · payment · dwh. Scenarios: payment-system, retry-storm.

References

• XU, A.; LAM, S. System Design Interview — An Insider's Guide, vol. 2, ch. 11 "Payment System" and ch. 12 "Digital Wallet"¹.
• Idempotency keys as designed at Stripe².
• The PCI Data Security Standard and how tokenization reduces scope³.
• Accounting patterns for software (accounts, entries, transactions)⁴.
• KLEPPMANN, M.; RICCOMINI, C. Designing Data-Intensive Applications, 2nd ed. — exactly-once semantics, idempotence and end-to-end correctness⁵.