Serverless & Autoscaling

Look at a day of BoxOffice traffic. From midnight to 09:45 the API serves a few hundred requests per second. At 10:00, when a stadium tour goes on sale, it jumps to tens of thousands — in seconds. By 10:30 it is back to normal. The same shape repeats a few times a week, at times announced in advance, plus surprises (an artist posts the link on social media at 23:00).

Two bad options: size the fleet for the peak and pay for idle machines 99% of the time, or size it for the average and fall over at 10:00. This page covers the two families of tools that avoid both — serverless platforms, which scale per request and charge per use, and autoscaling, which grows and shrinks a fleet of replicas based on metrics or schedules — and the uncomfortable truth they share: scaling takes time, and a flash sale does not wait.

Serverless

Serverless means you deploy code and the platform provisions, scales and bills the machines underneath, including scaling to zero when there is no traffic. Two shapes dominate:

• Functions as a Service (FaaS) — AWS Lambda, Google Cloud Run functions, Azure Functions: you deploy a function, the platform calls it once per event (an HTTP request, a queue message, a file upload, a schedule).
• Serverless containers — Google Cloud Run, AWS Fargate (with ECS), Azure Container Apps: you deploy a container image that serves HTTP; the platform runs as many instances as the traffic needs, down to zero.

The idea has been described as "cloud programming simplified"¹: no servers to patch, no capacity to plan, pay only for what runs. The constraints are the price.

How a function runs

sequenceDiagram
    participant E as Event source
    participant P as Platform
    participant X as Execution environment
    E->>P: invoke (request 1)
    P->>X: no warm environment → create sandbox
    Note over X: INIT: download code, start runtime, run init code (cold start)
    X-->>P: ready
    P->>X: handle request 1
    X-->>E: response
    E->>P: invoke (request 2)
    P->>X: reuse the warm environment
    X-->>E: response (no cold start)
    Note over X: idle for a while → frozen, later destroyed

Each execution environment (a lightweight VM or sandbox) handles one request at a time in Lambda; a burst of 300 simultaneous requests needs 300 environments. Environments are reused while traffic continues and destroyed after some idle time². Everything follows from that model:

• Cold start: the first request on a new environment pays for creating it. For a small Python or Node.js function this is typically a few hundred milliseconds; for a Spring Boot application on a plain JVM it can be several seconds (illustrative ranges — measure your own).
• Concurrency = rate × duration (Little's law again): 2,000 requests/s × 0.2 s = 400 concurrent environments. Halving the duration halves the concurrency — and the bill.
• State dies with the environment: anything kept in memory or on the local disk may vanish between requests; it is the stateless rule from the overview, enforced.

Cold starts and how to reduce them

Cold starts matter on latency-critical, user-facing paths with spiky traffic. They do not matter for a queue consumer rendering ticket PDFs, where an extra second is invisible.

Concurrency limits protect — and throttle

Serverless scales fast, but not infinitely:

• Account and function limits: AWS Lambda has a default quota of 1,000 concurrent executions per Region shared by all functions of the account (it can be raised), and each function scales up by at most 1,000 environments every 10 seconds². Beyond the limit, invocations are throttled (429), and asynchronous events are retried later.
• Reserved concurrency caps one function so it cannot starve the others — and also caps the load it can put on a downstream.
• Instance concurrency: serverless containers such as Cloud Run let one instance serve several requests at once (80 by default)³, which means far fewer instances — and fewer cold starts — for I/O-bound services.

Cost: per use vs always on

FaaS bills per invocation plus GB-seconds (memory × duration). Containers bill for provisioned capacity whether used or not. Which is cheaper depends on how busy the service is. The chart uses illustrative list prices (AWS Lambda x86 in us-east-1 at the time of writing: US$ 0.20 per million requests and US$ 0.0000166667 per GB-second; a container replica at an assumed US$ 60/month, each handling up to 300 requests/s, minimum two for availability) — check current prices before using them in a real decision.

With these assumptions each invocation costs about US$ 1 per million, so functions are far cheaper below roughly a hundred million requests per month (a few dozen requests per second on average), and containers win for steady, busy services. Spiky, mostly idle workloads favor serverless; steady high-volume ones favor provisioned capacity. FinOps (cloud-native) builds on this calculation.

The landscape

Try it in the playground

Open the serverless-burst scenario: a serverless function behind an API receives a burst.

1. Run the spike profile and look at p99 during the first seconds: the cold-start penalty.
2. Lower the function's concurrency limit until throttled requests (errors) appear; then raise it and watch what happens to the database behind it.
3. Compare the live cost with the same load on a container tier (swap the node for an App Server with autoscaling).

Open the full simulator

Autoscaling

Autoscaling adjusts capacity automatically. It operates at three levels, which must work together:

The HPA adds pods, but if no node has room, the pods stay Pending until the cluster autoscaler adds a node — which takes much longer. Both loops matter for the response time.

Policies

The Kubernetes HPA

Every 15 seconds (by default) the HPA computes⁴:

Six catalog replicas at 90% CPU with a 60% target: ⌈6 × 90 / 60⌉ = 9 replicas. It ignores changes within a 10% tolerance, and uses stabilization windows so that it does not oscillate (scale-down waits 5 minutes by default).

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: catalog
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: catalog
  minReplicas: 4                 # never below N+1 across zones
  maxReplicas: 40                # bounded by what the database can take
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60   # % of the CPU *request* of each pod
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100               # at most double every 15 s
          periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 20                # remove at most 20% per minute
          periodSeconds: 60

CPU utilization is measured relative to the pod's CPU request, so the HPA only works if requests are set (and realistic). Hands-on 10 runs a laptop-sized version of this HPA under a k6 load (box below).

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: catalog
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: catalog
  minReplicas: 2
  maxReplicas: 8
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60     # % of the CPU *request* (250m): scale when pods average > 150m
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0  # react at once to load
      selectPolicy: Max
      policies:
        - type: Percent
          value: 100                 # at most double...
          periodSeconds: 15
        - type: Pods
          value: 2                   # ...or +2 pods, whichever is more, every 15 s
          periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 120  # wait 2 min of lower load before removing pods
      policies:
        - type: Pods
          value: 1
          periodSeconds: 30

Choosing the metric

Scaling workers on queue depth

KEDA (Kubernetes Event-Driven Autoscaling) feeds the HPA with metrics from event sources — Kafka lag, SQS depth, RabbitMQ queue length, Prometheus queries, cron — and can scale a Deployment to zero when there is nothing to do⁵:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: ticket-worker
spec:
  scaleTargetRef:
    name: ticket-worker            # the Deployment
  minReplicaCount: 1
  maxReplicaCount: 24              # = partitions of order.confirmed.v1
  cooldownPeriod: 120
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka:9092
        consumerGroup: ticket-worker
        topic: order.confirmed.v1
        lagThreshold: "200"        # target lag per replica

With a lag of 4,800 messages and a threshold of 200, KEDA asks for 24 replicas — the cap, because a Kafka consumer group cannot use more consumers than the topic has partitions (messaging).

Autoscaling lag

Reactive autoscaling has a built-in delay: the load must arrive, be measured, trigger a decision, and the new capacity must start and warm up. Each step is small; together they are minutes:

A flash sale goes from 1× to 20× in seconds (here, 1,000 to 20,000 req/s in 20 s). The simulation below compares a reactive autoscaler (target 60%, doubling at most every 15 s, new replicas ready after 90 s) with the same autoscaler plus a scheduled pre-scale at 09:45 to the forecast, 20,000 / (500 × 0.6) ≈ 67 replicas:

The red area is demand above capacity: requests that queue, time out or are rejected — in the first minutes of the sale, exactly when it matters most. Reactive scaling eventually catches up, but after the fans had a bad time.

Scheduled and predictive scaling

When you know the spike is coming, do not wait for metrics:

• Scheduled scaling: at 09:45 raise minReplicas (or the Auto Scaling group's minimum) to the forecast for 10:00, and bring it back down at 10:45. With KEDA, a cron trigger does exactly this.
• Pre-warm everything the replicas need: nodes (overprovisioning with low-priority "placeholder" pods that get evicted when real pods need the room), images pulled, caches filled, database connection pools opened.
• Predictive scaling (e.g. AWS predictive scaling) learns daily and weekly cycles from history and scales ahead of them. It helps with the regular evening peak, not with a one-off on-sale.

And when even that is not enough — two million fans for 60 thousand seats — capacity is not the answer at all: a waiting room admits fans at the rate the system can serve, rate limiting stops bots, and load shedding rejects excess early and cheaply (edge, reliability). The flash-sale scenario in the playground combines all of them.

Scaling in, safely

Scaling down causes incidents too:

• Flapping: scaling down as soon as load dips, then up again; stabilization windows and slow scale-down policies prevent it.
• Killing in-flight work: replicas must drain on SIGTERM (graceful shutdown, overview); workers must stop taking jobs and finish or release the current one.
• Scaling to zero saves money but brings back the cold start for the next request; keep a minimum on user-facing paths.

Containers or serverless?

How it connects

flowchart LR
    FAAS["functions per event"] --> COLD["cold starts,<br/>provisioned concurrency"]
    FAAS --> CONC["concurrency = rate × duration<br/>quotas, throttling"]
    CONC --> DOWN["the database does not scale with it<br/>→ Databases"]
    FAAS --> COST["per use vs always on<br/>→ FinOps"]
    AS["autoscaling"] --> HPA["HPA, KEDA, VPA, node autoscalers<br/>→ Kubernetes"]
    HPA --> MET["metric: CPU, RPS,<br/>concurrency, queue lag"]
    AS --> LAG["minutes of lag"]
    LAG --> SCHED["scheduled scaling + pre-warm<br/>for the 10:00 on-sale"]
    LAG --> PROT["waiting room, rate limiting,<br/>load shedding → Reliability"]
    HPA --> MAX["maxReplicas from<br/>the downstream budget"]
    MAX --> DOWN

Palette components: serverless (webhooks, image resizing, bursty glue) · app (the containers the HPA scales) · worker (KEDA on queue lag) · queue (the lag signal) · sql (the connection budget that caps maxReplicas) · apigw (rate limiting while capacity catches up) · metrics (what the autoscaler reads). Scenarios: serverless-burst, scale-out, async-queue, flash-sale. Continues in: Cloud-native — Kubernetes, Cloud-native — FinOps, Reliability — load shedding, Estimation — queueing, Exercise: Kubernetes autoscaling.

Key takeaways