3. SLO and Burn-Rate Alert
A dashboard needs someone looking at it. An alert looks for you — and a bad alert wakes you up at 3 a.m. for nothing, until you learn to ignore it. This step turns a user-facing promise into an alert that fires when, and only when, the promise is at risk. The theory is in the alerting class and the SLO class.
From promise to numbers
| Term | BoxOffice catalog |
|---|---|
| SLI (indicator) | share of catalog requests, health checks excluded, not answered with a 5xx |
| SLO (objective) | 99.5% over a rolling 28 days (the course standard window: four whole weeks) |
| Error budget | 0.5% of the requests may fail: at 100 req/s, 100 × 86 400 × 28 × 0.005 ≈ 1.21 million failed requests, or about 202 minutes of total outage per 28 days |
| Burn rate | how fast the budget is being spent, relative to "exactly on budget": error ratio / 0.005 |
A burn rate of 1 spends the whole budget in exactly 28 days. A burn rate of 14.4 spends about 2.1% of it in one hour (14.4 × 1 h / 672 h ≈ 2.1%) and all of it in less than two days. (The thresholds 14.4 and 6 come from the Google SRE workbook, which derived them for a 30-day window: 2% and 5% of the budget; with 28 days they mean 2.1% and 5.4% — close enough to keep the standard numbers.)
Why exclude /actuator/* from the SLI?
Docker, Kubernetes and the edge call the health endpoints every few seconds, and those calls almost never fail. Counting them inflates the success ratio with requests no user made — the SLI must measure what fans experience. The same reasoning excludes synthetic load you generate yourself from production SLIs, unless it is labeled and filtered.
Why "error rate > 1%" is a bad alert
| Alert | Problem |
|---|---|
errors > 1% for 5 min | fires on a 5-minute blip that cost about 0.01% of the budget (noise), and stays silent on a 0.8% error rate that burns the whole budget in 17.5 days (missed) |
errors > 0 | pages all the time |
p99 > 1 s on one instance | measures a machine, not users |
The Google SRE workbook's answer is the multiwindow, multi-burn-rate alert: page when the budget burns fast over a long window (it is a real problem, not a blip) and over a short window (it is still happening — the alert stops soon after recovery).
| Severity | Burn rate | Long window | Short window | Budget spent when it fires |
|---|---|---|---|---|
| page | 14.4 | 1 h | 5 min | ≈ 2.1% |
| page | 6 | 6 h | 30 min | ≈ 5.4% |
| ticket | 1 | 3 days | 6 h | ≈ 10.7% |
This lab provisions the first rule as a page and the second as a ticket (a slower, cheaper signal for a smaller team); the third is left as an exercise.
The rules, as code
Grafana loads alert rules from YAML files in its provisioning folder (mounted in step 1 from infra/grafana/alerting/). Each rule runs queries A and B — the burn rate over the two windows — and a math expression C that is the condition.
# Multiwindow, multi-burn-rate alerts for the catalog availability SLO (Google SRE workbook, ch. 5).
# SLI: share of catalog requests (health checks excluded) NOT answered with 5xx. SLO: 99.5% / 28 days.
apiVersion: 1
groups:
- orgId: 1
name: catalog-slo
folder: BoxOffice
interval: 30s
rules:
- uid: catalog-availability-fast-burn
title: "catalog availability SLO: fast burn"
condition: C
data:
- refId: A # burn rate over the long window (1 h)
relativeTimeRange: { from: 3600, to: 0 }
datasourceUid: prometheus
model:
refId: A
instant: true
expr: |
(
sum(rate(http_server_request_duration_seconds_count{service_name="catalog", http_route!~"/actuator.*", http_response_status_code=~"5.."}[1h]))
/ sum(rate(http_server_request_duration_seconds_count{service_name="catalog", http_route!~"/actuator.*"}[1h]))
) / 0.005
- refId: B # burn rate over the short window (5 min): stops paging soon after recovery
relativeTimeRange: { from: 300, to: 0 }
datasourceUid: prometheus
model:
refId: B
instant: true
expr: |
(
sum(rate(http_server_request_duration_seconds_count{service_name="catalog", http_route!~"/actuator.*", http_response_status_code=~"5.."}[5m]))
/ sum(rate(http_server_request_duration_seconds_count{service_name="catalog", http_route!~"/actuator.*"}[5m]))
) / 0.005
- refId: C
datasourceUid: __expr__
model:
refId: C
type: math
expression: "$A > 14.4 && $B > 14.4"
for: 1m
noDataState: OK
execErrState: Error
labels:
severity: page
slo: catalog-availability
annotations:
summary: "catalog is burning its 28-day error budget 14.4x too fast"
description: 'Burn rate {{ printf "%.1f" $values.A.Value }} (1 h) and {{ printf "%.1f" $values.B.Value }} (5 min): at this pace about 2% of the 28-day budget is gone in one hour.'
- uid: catalog-availability-slow-burn
title: "catalog availability SLO: slow burn"
condition: C
data:
- refId: A # 6 h window
relativeTimeRange: { from: 21600, to: 0 }
datasourceUid: prometheus
model:
refId: A
instant: true
expr: |
(
sum(rate(http_server_request_duration_seconds_count{service_name="catalog", http_route!~"/actuator.*", http_response_status_code=~"5.."}[6h]))
/ sum(rate(http_server_request_duration_seconds_count{service_name="catalog", http_route!~"/actuator.*"}[6h]))
) / 0.005
- refId: B # 30 min window
relativeTimeRange: { from: 1800, to: 0 }
datasourceUid: prometheus
model:
refId: B
instant: true
expr: |
(
sum(rate(http_server_request_duration_seconds_count{service_name="catalog", http_route!~"/actuator.*", http_response_status_code=~"5.."}[30m]))
/ sum(rate(http_server_request_duration_seconds_count{service_name="catalog", http_route!~"/actuator.*"}[30m]))
) / 0.005
- refId: C
datasourceUid: __expr__
model:
refId: C
type: math
expression: "$A > 6 && $B > 6"
for: 5m
noDataState: OK
execErrState: Error
labels:
severity: ticket
slo: catalog-availability
annotations:
summary: "catalog is burning its error budget 6x too fast"
description: 'Burn rate {{ printf "%.1f" $values.A.Value }} (6 h) and {{ printf "%.1f" $values.B.Value }} (30 min): about 5% of the 28-day budget in six hours.'
Restart otel-lgtm and open Alerting → Alert rules: both rules are listed in the folder BoxOffice as Normal.
Fire it
Produce steady traffic, then break the database that catalog depends on:
Start db once the rule fires, and leave k6 running: with traffic, the 5-minute burn rate falls below 14.4 within a minute or two and the page returns to Normal. Without traffic, the ratio is 0 / 0 (NaN) and Grafana keeps the alert in its last state.
Why the gateway probes /actuator/health/readiness
The SLI is measured by catalog itself, so catalog must keep receiving the requests while db is down. That holds because the gateway's balancer (lab 4) probes /actuator/health/readiness, not /actuator/health: the full health includes db and redis, so with db stopped it would mark every replica down after a few seconds and answer 503 itself — fans would get errors, catalog would see no requests, and the alert would never fire. Readiness says whether this instance can take traffic: removing every replica because of a dependency they all share helps nobody.
Predict: how long until the fast-burn rule fires?
During the outage every catalog request fails, so the short (5 min) burn rate jumps to about 1 / 0.005 = 200 within a minute. The long window is the slow one: the 1-hour error ratio must pass 14.4 × 0.5% = 7.2%. With steady traffic and an hour of healthy history, that takes 0.072 × 60 min ≈ 4.3 minutes of outage; add the for: 1m and up to 30 s of evaluation interval: ≈ 5–6 minutes to page. (If the stack has been up for less than an hour, the window holds less healthy history and the alert fires sooner; the stress test of step 2 less than an hour ago adds healthy history and delays it.)
That is the trade-off of burn-rate alerts: a total outage takes minutes to page; a persistent 4% error rate (burn rate 8) is caught by the 6-hour rule once the 6-hour ratio passes 6 × 0.5% = 3%, after about 6 h × 3/4 = 4.5 hours; and a 30-second blip never pages.
In a test run of this lab (with the stress test of step 2 still in the 1-hour window), the rule became Pending about 6.5 minutes and Firing about 440 s after db was stopped, with the description "Burn rate 17.0 (1 h) and 200.0 (5 min)". The slow-burn ticket fired too, about a minute later.
Record in your results the time you stopped db, the time the rule became Pending and Firing, and the burn rates shown in the alert's description.
Pitfalls
- No traffic, no SLI. Division by zero yields no data;
noDataState: OKkeeps a quiet night quiet, but a service that receives no traffic because the edge is down looks "healthy". Pair SLO alerts with a synthetic probe or an alert on traffic dropping to zero. - Alert on symptoms, investigate causes. CPU, heap and pool panels are for the dashboard; the page is about users. A full disk that does not hurt users yet is a ticket, not a page.
- Every page needs a runbook link (an
annotations.runbook_url) saying what to check first — see the incident response class.
Exercise: a latency SLO
Write a third rule for the promise "95% of GET /catalog/events/{id} complete in under 250 ms". Hint: the SLI is the ratio of the le="0.25" bucket rate to the _count rate for that route; the budget is 5%. Fire it by adding latency (in lab 9 you will use Toxiproxy for exactly this).