API Gateway & Rate Limiting

BoxOffice has nine services. Without a gateway, the mobile app would need to know nine addresses, every service would validate JWTs on its own (with nine slightly different bugs), every service would implement its own rate limit, and a fan's script hammering POST /api/holds 500 times per second would go straight to inventory. The API gateway is the single front door that solves all of this in one place — and it is also the place where many architectures quietly grow a bottleneck or a second monolith.

An API gateway is an L7 reverse proxy specialised for APIs: it is the single entry point for clients, routes each request to the right service, and applies cross-cutting policies — authentication, rate limits, quotas, transformations, observability — before the request reaches any business code¹. It is the apigw component of the palette.

What a gateway does

flowchart LR
    app([Mobile app]) --> gw
    web([Web app]) --> gw
    partner([Partner API client]) --> gw
    subgraph gw [API gateway]
        direction TB
        a["Authenticate: JWT / API key"] --> r["Rate limit and quota"]
        r --> t["Route + transform"]
        t --> o["Log, metrics, trace"]
    end
    gw --> catalog[catalog]
    gw --> inventory[inventory]
    gw --> order[order]
    gw --> assistant[assistant]

The key word is cross-cutting: concerns that are the same for every service. Anything that depends on the business — prices, seat allocation, refund rules — belongs in a service, not in gateway configuration.

cloud:
  gateway:
    server:
      webflux:
        httpclient:
          connect-timeout: 1000          # ms to open a connection to a service
          response-timeout: 5s           # never wait forever for a service
        default-filters:
          # retry idempotent GETs once on another instance when a connection fails
          - name: Retry
            args:
              retries: 1
              methods: GET
              series: SERVER_ERROR
              exceptions: java.io.IOException, java.util.concurrent.TimeoutException
        routes:
          - id: auth
            uri: ${AUTH_URL:http://auth:8081}
            predicates:
              - Path=/auth/**
          - id: catalog
            uri: lb://catalog              # "lb": pick one instance of "catalog" per request
            predicates:
              - Path=/catalog/**
  loadbalancer:
    configurations: health-check         # skip instances that are not ready
    health-check:
      interval: 5s
      path:
        default: /actuator/health/readiness   # this instance, not db/redis that every replica shares

Gateway, load balancer, ingress or service mesh?

These four components overlap, and products blur the lines (Envoy is the engine of many of them). What distinguishes them is which traffic they handle and what they know:

A typical BoxOffice request crosses a load balancer (managed, at the region entrance), the API gateway (policies), and, between services, the mesh. On Kubernetes the gateway is often deployed as the Gateway API implementation, merging the ingress and gateway roles. The mesh is studied in Network & Security.

Backend for Frontend (BFF)

The mobile app and the web app need different things: the app shows a compact event card and works on slow networks; the web page shows the full seat map and venue details. A single, generic API either over-fetches for mobile or forces the web app to make many calls.

The Backend for Frontend pattern gives each client type its own thin gateway layer, owned by the team that builds that frontend²:

flowchart LR
    m([Mobile app]) --> bm[Mobile BFF]
    w([Web app]) --> bw[Web BFF]
    p([Partners]) --> pg[Public API gateway]
    bm --> catalog[catalog]
    bm --> inventory[inventory]
    bw --> catalog
    bw --> inventory
    bw --> order[order]
    pg --> catalog

Each BFF aggregates and shapes the responses its client needs (one call for the "event screen" on mobile, with smaller images and fewer fields) and evolves at the speed of that frontend. GraphQL servers are often used as BFFs, because the client describes the exact shape it needs (see API styles).

Use a BFF when clients differ significantly and are owned by different teams; keep cross-cutting policies (auth, rate limits) in the shared gateway in front of the BFFs.

Rate limiting

Rate limiting caps how many requests a client may make in a period of time. It protects the backend from overload and abuse, keeps one noisy client from hurting the others (fairness), and enforces commercial plans. It answers with 429 Too Many Requests — "you, specifically, are sending too much" — unlike 503 Service Unavailable, which means "the server is overloaded for everyone".

Every rate limiter has to decide three things:

1. Who is being limited — the key: the user id (from the JWT), the API key, the client IP, or a combination such as user + route.
2. How much — the policy: 10 requests per second with bursts of 20; 1 000 per hour; 4 holds per event per user.
3. How to count — the algorithm, below.

Token bucket

The most widely used algorithm³. Each key has a bucket that holds at most capacity tokens (the burst size) and is refilled at a constant rate (the sustained limit). Each request takes a token; if the bucket is empty, the request is rejected.

flowchart LR
    req([Request from fan 7]) --> chk{"Token in<br/>fan 7's bucket?"}
    refill(["Refill: +10 tokens/s<br/>up to capacity 20"]) -.-> chk
    chk -->|"yes: take 1 token"| fwd[Forward to inventory]
    chk -->|"no: bucket empty"| rej["429 Too Many Requests<br/>Retry-After: 1"]

The implementation does not need a timer: store the token count and the time of the last update, and on each request add elapsed × rate tokens (capped at capacity) before checking. Two numbers per key, O(1) work per request.

The simulation below shows one fan's bucket (capacity 20, refill 10/s) under an illustrative traffic pattern: normal browsing at 4 req/s, a 2-second burst at 40 req/s (the fan mashes "refresh" at 10:00), and then 8 seconds at 14 req/s.

Read the two panels together:

• During the burst, the first second accepts about 29 requests — the 20 saved tokens plus the refill — and then the limiter settles at the refill rate of 10/s; the excess is rejected.
• During the sustained phase at 14 req/s, the bucket drains slowly (it loses 4 tokens per second) and accepts everything for about five seconds; after that, it accepts exactly 10/s.
• When traffic calms down (4 req/s, so a net refill of 6 tokens/s), the bucket is full again in about three seconds and the next burst will be absorbed again.

This is the behaviour you usually want: short bursts are tolerated, sustained excess is not. The two parameters map directly to product language — "10 requests per second, bursts of up to 20".

Leaky bucket

Requests enter a queue of fixed size and leave it at a constant rate, like water leaking from a bucket with a hole. If the queue is full, new requests are dropped. The output is perfectly smooth — the backend sees exactly the leak rate — at the cost of added latency for queued requests and no bursts reaching the backend. Nginx's limit_req is a leaky bucket; with burst=… nodelay it behaves much like a token bucket.

Fixed window counter

Count requests per key in fixed windows — "minute 10:00", "minute 10:01" — with a counter that resets at each boundary: INCR ratelimit:fan7:202610171000 with a 60 s expiry. Simple and cheap, but it has a boundary problem: with a limit of 100 per minute, a client can send 100 at 10:00:59 and another 100 at 10:01:00 — 200 requests in two seconds, twice the intended rate, all accepted.

Sliding window log

Store the timestamp of every request (for example in a Redis sorted set); on each request, drop timestamps older than the window and count the rest. Exact — there is no boundary problem — but memory grows with the limit: a limit of 1 000 per hour stores up to 1 000 timestamps per key.

Sliding window counter

A cheap approximation that fixes the boundary problem: keep the counters of the current and previous fixed windows and weight the previous one by how much of it still overlaps the sliding window:

Example: limit 100/min. The previous minute had 80 requests; 15 s into the current minute there have been 30. Estimate = 30 + 80 × (1 − 15/60) = 30 + 60 = 90 → the request is allowed. It assumes the previous window's requests were evenly spread, which is close enough in practice; Cloudflare reported that this approximation misjudged a tiny fraction of requests on real traffic⁴.

Comparing the algorithms

Distributed rate limiting

BoxOffice runs six gateway replicas. If each keeps its counters in memory, a fan whose requests are spread across them gets six times the limit. A rate limit that must hold across replicas needs a shared counter store, almost always Redis: fast (sub-millisecond in the same zone), with atomic operations and expiry.

The read-modify-write of a token bucket must be atomic — two gateway replicas reading "1 token left" at the same time would both accept. A Lua script runs atomically inside Redis:

-- KEYS[1] = bucket key (e.g. "rl:user:7:holds")
-- ARGV[1] = capacity, ARGV[2] = refill rate (tokens/s), ARGV[3] = cost of this request
local capacity = tonumber(ARGV[1])
local rate     = tonumber(ARGV[2])
local cost     = tonumber(ARGV[3])

local t   = redis.call("TIME")                        -- Redis clock: no skew between gateways
local now = tonumber(t[1]) * 1000 + math.floor(tonumber(t[2]) / 1000)

local b      = redis.call("HMGET", KEYS[1], "tokens", "ts")
local tokens = tonumber(b[1]) or capacity
local ts     = tonumber(b[2]) or now

tokens = math.min(capacity, tokens + (now - ts) / 1000 * rate)   -- refill since last call
local allowed = 0
if tokens >= cost then
  tokens = tokens - cost
  allowed = 1
end

redis.call("HSET", KEYS[1], "tokens", tokens, "ts", now)
redis.call("PEXPIRE", KEYS[1], math.ceil(capacity / rate * 1000) * 2)  -- idle keys disappear
return { allowed, tostring(tokens) }

Design points worth saying out loud:

• One round trip per request (a few hundred microseconds to a millisecond) — acceptable at the gateway, but it is on the critical path of every request.
• Time comes from Redis (TIME), not from each gateway: gateway clocks drift (see Time & clocks).
• Fail open or fail closed? If Redis is unreachable, should the gateway accept everything (protecting availability) or reject everything (protecting the backend)? Most APIs fail open with a local, per-replica fallback limit.
• Hot keys and scale: a Redis Cluster shards keys by hash; a single very hot key (a global limit) lands on one shard. Per-user keys spread naturally.
• Approximate alternatives: each replica enforces limit / replicas locally, or syncs counters asynchronously every 100 ms. Less exact, no extra round trip.

Where to rate limit

Use several layers: coarse and cheap at the edge, identity-aware at the gateway, business rules in the services. Rate limiting is also one of the resilience patterns of class 10 — there, the point of view is protecting a service from overload (load shedding).

Telling the client

A good limiter tells well-behaved clients how to behave:

HTTP/1.1 429 Too Many Requests
Retry-After: 2
RateLimit-Policy: "per-user";q=20;w=1
RateLimit: "per-user";r=0;t=2
Content-Type: application/problem+json

{"type": "https://boxoffice.example/problems/rate-limited", "title": "Too many requests", "status": 429}

Retry-After (seconds, or a date) is standard HTTP. Many APIs also send X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset; the IETF is standardising RateLimit and RateLimit-Policy fields for the same purpose⁵. Clients should respect Retry-After and back off with jitter — otherwise every rejected client retries at the same instant.

Configuration examples

spring:
  cloud:
    gateway:
      server:
        webflux:      # the property prefix since Spring Cloud 2025 (as in hands-on 4)
          routes:
            - id: inventory-holds
              uri: http://inventory:8080
              predicates:
                - Path=/api/holds/**
              filters:
                - name: RequestRateLimiter
                  args:
                    redis-rate-limiter.replenishRate: 10   # tokens per second
                    redis-rate-limiter.burstCapacity: 20   # bucket capacity
                    redis-rate-limiter.requestedTokens: 1  # cost of one request
                    key-resolver: "#{@userKeyResolver}"

@Configuration
public class RateLimitConfig {

    @Bean
    KeyResolver userKeyResolver() {   // the key: the authenticated user, or the IP
        return exchange -> exchange.getPrincipal()
            .map(Principal::getName)
            .switchIfEmpty(Mono.fromSupplier(() ->
                exchange.getRequest().getRemoteAddress().getAddress().getHostAddress()));
    }
}

# 10 MB of shared memory holds about 160 000 IP states
limit_req_zone $binary_remote_addr zone=per_ip:10m rate=10r/s;

server {
    location /api/ {
        limit_req zone=per_ip burst=20 nodelay;   # absorb bursts of 20, no queueing delay
        limit_req_status 429;
        proxy_pass http://gateway;
    }
}

# rate limiting per API key: 20 req/s per key, bursts of up to 40.
# Requests WITHOUT an X-Api-Key header have an empty key and are not limited
# (in production you would also limit them per client IP).
limit_req_zone  $http_x_api_key  zone=per_key:10m  rate=20r/s;
limit_req_status  429;
    location /catalog/ {
        limit_req  zone=per_key  burst=40  nodelay;

Try it. Load the rate-limiter scenario and run the spike profile without limits: the backend saturates and p99 explodes for everyone. Turn on the gateway rate limit and tune its rate and burst: watch the 429s absorb the excess while p99 for accepted requests stays flat. What happens if the limit is set higher than the backend's capacity?

Open the full playground

The full design of a distributed rate limiter, as an interview problem, is in the rate limiter case study.

Trade-offs at a glance

How it connects

flowchart LR
    GW["API gateway"] --> AUTH["JWT validation<br/>→ Network & Security"]
    GW --> RL["rate limiting"]
    RL --> TB["token bucket in Redis<br/>→ Caching (Redis)"]
    RL --> SHED["load shedding, 429<br/>→ Reliability"]
    RL --> RETRY["client retries with backoff<br/>→ retry storms"]
    GW --> BFF["BFF per client"]
    BFF --> STYLES["REST, GraphQL, gRPC<br/>→ Compute: API styles"]
    GW --> OBS["access logs, RED metrics<br/>→ Observability"]
    GW --> WR["waiting room<br/>→ Messaging (queue)"]

Palette components: apigw · auth (issues the tokens the gateway validates) · cache (shared rate-limit counters) · queue (the waiting room) · waf (coarse limits per IP, before the gateway). Scenarios: rate-limiter, retry-storm, flash-sale. Continues in: Case study: rate limiter, Reliability — patterns, Network & Security — identity, Hands-on 4.

Key takeaways