SLIs, SLOs & Error Budgets

"Is BoxOffice reliable enough?" Nobody can answer that without numbers. The product manager wants to ship a new seat-map feature the week before a stadium on-sale; the on-call engineer says the purchase path had a bad month and wants to freeze deploys. Without a shared definition of "good enough", that discussion is opinion against opinion. Google's Site Reliability Engineering (SRE) practice introduced a vocabulary that turns it into arithmetic¹, and it is now standard across the industry:

flowchart LR
    tel["Telemetry<br/>(metrics, logs, traces)"] -->|"aggregated into"| sli["SLI<br/>good / valid events"]
    sli -->|"compared with"| slo["SLO<br/>internal target"]
    slo -->|"the gap is"| budget["Error budget<br/>licence to take risk"]
    slo -->|"looser version"| sla["SLA<br/>external contract"]
    budget -->|"drives"| decisions["Ship or freeze?<br/>Page or ticket?"]

Start from user journeys

An SLO is only useful if it measures what users care about. Start by listing BoxOffice's critical user journeys (CUJs) — the few interactions that must work — and then decide how to measure each one.

Not every journey needs the same target. Browsing can tolerate an occasional slow page; a failed payment loses a sale and a fan's trust.

SLI — measuring what users feel

An SLI is best written as a ratio of events, so it always lies between 0% and 100%²:

Both words matter. Good defines success from the user's point of view — "status is not 5xx and latency is below 500 ms". Valid excludes what should not count — 4xx responses caused by the client, health-check requests, synthetic load tests.

Where to measure

A common choice is to compute SLIs at the gateway (close to the user, but under your control) and complement them with synthetic probes of the purchase journey. With OpenTelemetry metrics exported to Prometheus (see Signals), the availability SLI of the seat-hold route over 5 minutes is:

sum(rate(http_server_request_duration_seconds_count{service_name="gateway", http_route="/holds", http_response_status_code!~"5.."}[5m]))
/
sum(rate(http_server_request_duration_seconds_count{service_name="gateway", http_route="/holds"}[5m]))

SLO — the target

An SLO is a target for an SLI over a window:

SLI ≥ target over a window

Use a rolling window (the last 28 days, moving every minute) rather than a calendar month: 28 days always contain four of each weekday, and the budget does not "reset" magically on the 1st.

How to choose the number:

• Measure first. Look at the SLI over the last 30–90 days, then set the SLO slightly below what you already achieve. A target you already miss every week is noise.
• Respect the dependency math. order cannot promise more than the product of what it calls synchronously (Reliability — composite availability). If the payment provider offers 99.9%, a purchase SLO of 99.99% is fiction.
• Ask what users notice. Going from 99.9% to 99.99% costs a lot (see the nines table); if fans would not notice, the money is better spent on features.
• Never 100%. It leaves no room for deploys, maintenance or the unavoidable, and every dependency you have is below it anyway.

Error budget

The error budget is the amount of unreliability the SLO allows — the team's licence to take risk:

Burn rate

The burn rate says how fast the budget is being spent relative to the window²:

With a 99.9% SLO, an error rate of 0.1% is a burn rate of 1 — the budget lasts exactly the 28 days. An error rate of 1.44% is a burn rate of 14.4, which exhausts the budget in \(28 / 14.4 \approx 1.9\) days.

The simulation below follows the purchase path's budget over one 28-day window with illustrative numbers: a quiet background of 0.01% errors, a bad deploy on day 9 (5% errors for 3 hours) and a slow degradation from day 16 (0.2% errors — a burn rate of only 2).

The two incidents teach different lessons:

• The bad deploy has a huge burn rate (50×) for a short time. It must page someone within minutes, and the budget shows what it cost: about a fifth of the month.
• The slow degradation never looks dramatic — 0.2% errors would not alarm anyone on a dashboard — but at 2× it exhausts the budget around day 26. Only a slow-burn alert over a long window catches it.

Alerting on burn rate

Alert on the budget, not on raw error rates or CPU. The SRE Workbook recommends multi-window, multi-burn-rate alerts²: a long window to measure significance, a short window to stop the alert quickly once the problem is fixed.

# Prometheus alerting rule: purchase path, SLO 99.9% (budget = 0.001)
- alert: PurchaseErrorBudgetFastBurn
  expr: |
    (
      sum(rate(http_server_request_duration_seconds_count{service_name="gateway", http_route=~"/holds|/orders", http_response_status_code=~"5.."}[1h]))
      / sum(rate(http_server_request_duration_seconds_count{service_name="gateway", http_route=~"/holds|/orders"}[1h]))
    ) > (14.4 * 0.001)
    and
    (
      sum(rate(http_server_request_duration_seconds_count{service_name="gateway", http_route=~"/holds|/orders", http_response_status_code=~"5.."}[5m]))
      / sum(rate(http_server_request_duration_seconds_count{service_name="gateway", http_route=~"/holds|/orders"}[5m]))
    ) > (14.4 * 0.001)
  labels:
    severity: page
  annotations:
    summary: "Purchase path burning the 28-day error budget 14.4x faster than allowed"

The details of alert design, routing and on-call are in Observability — alerting.

- uid: order-availability-fast-burn
  title: "order availability SLO: fast burn"
  condition: C
  data:
    - refId: A          # burn rate over the long window (1 h)
      relativeTimeRange: { from: 3600, to: 0 }
      datasourceUid: prometheus
      model:
        refId: A
        instant: true
        expr: |
          (
            sum(rate(http_server_request_duration_seconds_count{service_name="order", http_route!~"/actuator.*", http_response_status_code=~"5.."}[1h]))
            / sum(rate(http_server_request_duration_seconds_count{service_name="order", http_route!~"/actuator.*"}[1h]))
          ) / 0.005
    - refId: C
      datasourceUid: __expr__
      model:
        refId: C
        type: math
        expression: "$A > 14.4 && $B > 14.4"
  for: 1m
  noDataState: OK
  execErrState: Error
  labels:
    severity: page
    slo: order-availability
  annotations:
    summary: "order is burning its 28-day error budget 14.4x too fast"
    description: 'Burn rate {{ printf "%.1f" $values.A.Value }} (1 h) and {{ printf "%.1f" $values.B.Value }} (5 min): at this pace about 2% of the 28-day budget is gone in one hour.'

The error budget policy

An error budget is only a number on a dashboard unless product and engineering agree in advance what happens as it is spent. That agreement is the error budget policy:

SLA — the contract

An SLA is a formal agreement with customers that defines the expected service and the remedies (usually service credits) if it is not met. It is a legal document: breaching it costs money.

An SLA typically states: covered services and regions; the commitment (e.g. 99.9% monthly uptime); how it is measured; exclusions (scheduled maintenance, customer-caused problems, force majeure); and the credit schedule.

The commitments below come from the providers' public SLA pages³; conditions change over time, so always read the current page.

Two lessons hide in that table. First, the highest numbers apply only if you deploy redundantly — redundancy is the price of the extra nine. Second, the remedy is a credit on your bill, not compensation for your lost ticket sales: an SLA transfers a little financial risk; it does not make your system available.

Keep the SLO stricter than the SLA (e.g. SLO 99.95%, SLA 99.9%): the gap is a safety margin that warns you long before the contract is at risk.

How it connects

flowchart LR
    CUJ["critical user journeys"] --> SLI["SLI = good / valid events"]
    TEL["telemetry at the gateway<br/>→ Observability"] --> SLI
    SLI --> SLO["SLO over a<br/>28-day rolling window"]
    DEP["dependency math<br/>→ composite availability"] --> SLO
    SLO --> EB["error budget = 1 − SLO"]
    SLO --> SLA["SLA: a looser<br/>external contract"]
    EB --> BR["multi-window burn-rate alerts<br/>→ Observability — alerting"]
    EB --> POL["error budget policy:<br/>ship or freeze?"]
    POL --> DEL["canary, deploy freeze<br/>→ Cloud-native — delivery"]
    POL --> PM["postmortem over 20% of budget<br/>→ Incident response"]

Palette components: metrics (the SLI is a ratio of request counters) · alerting (multi-window burn-rate rules) · apigw (where BoxOffice measures the purchase SLI) · kafka (consumer lag, the early warning of the ticket-delivery freshness SLI) · payment (its 99.9% caps the purchase SLO). Scenarios: observability-stack, flash-sale (the event-scoped on-sale SLO). Continues in: Observability — alerting, Observability — incident response, Estimation — availability math, Cloud-native — delivery, Hands-on 8 — SLO and burn-rate alert.