TopolearnThreshold alerts either page too often or miss slow burns; burn-rate alerting is the standard that balances both.
An SLO states a target such as "99.9 percent of requests succeed over 30 days". The allowed failure, 0.1 percent, is the error budget. The burn rate is how fast you are consuming it: a burn rate of 1 uses the whole budget in exactly 30 days; 14.4 uses it in about 2 days.
In PromQL the error ratio over a window is sum(rate(errors[w])) / sum(rate(requests[w])), and the burn rate is that ratio divided by (1 - SLO). Record the ratio for several windows so the alerts are cheap and consistent.
Alerting on a single window is a trade-off: a 5-minute window reacts fast but pages on blips; a 6-hour window is stable but slow and keeps firing long after recovery. The multi-window, multi-burn-rate approach from the Google SRE Workbook pairs a long window (detect a significant burn) with a short window one twelfth its size (confirm it is still happening, so the alert resolves quickly):
- Page: burn rate > 14.4 over 1h and 5m (2 percent of a 30-day budget in an hour).
- Page: burn rate > 6 over 6h and 30m (5 percent in 6 hours).
- Ticket: burn rate > 3 over 1d and 2h; burn rate > 1 over 3d and 6h.
Latency SLOs use the same shape with "slow requests" as the errors: rate(_count) - rate(_bucket{le="0.3"}), so the threshold must be a bucket boundary. Keep the SLO definition (metric, window, target) in one place; tools such as Sloth or Pyrra generate the recording rules and alerts from a short spec.
Remaining budget for a dashboard: 1 - (ratio over 30d) / (1 - SLO), where the 30-day ratio comes from increase() or a long recording-rule chain, not a 30-day rate() evaluated on every refresh.
Example
groups:
- name: api-slo
rules:
# error ratio per window (repeat for 30m, 1h, 2h, 6h, 1d, 3d)
- record: job:slo_errors_per_request:ratio_rate5m
expr: sum by (job) (rate(http_requests_total{job="api",status=~"5.."}[5m]))
/ sum by (job) (rate(http_requests_total{job="api"}[5m]))
- record: job:slo_errors_per_request:ratio_rate1h
expr: sum by (job) (rate(http_requests_total{job="api",status=~"5.."}[1h]))
/ sum by (job) (rate(http_requests_total{job="api"}[1h]))
# SLO 99.9% -> budget 0.001
- alert: ApiErrorBudgetBurn
expr: |
(job:slo_errors_per_request:ratio_rate1h{job="api"} > (14.4 * 0.001)
and job:slo_errors_per_request:ratio_rate5m{job="api"} > (14.4 * 0.001))
or
(job:slo_errors_per_request:ratio_rate6h{job="api"} > (6 * 0.001)
and job:slo_errors_per_request:ratio_rate30m{job="api"} > (6 * 0.001))
labels: {severity: page}
annotations:
summary: "api is burning its 30d error budget fast ({{ $value | humanizePercentage }} errors)"
runbook_url: https://runbooks.example.com/api/slo-burnRemember
- burn rate = error ratio / (1 - SLO); 1 = budget gone exactly at the SLO window's end
- Pair a long window with a 1/12 short window: fast detection, fast resolution
- 14.4x over 1h+5m and 6x over 6h+30m are the standard paging pair
Watch out for
- Evaluating rate(x[30d]) in a dashboard query instead of chaining recording rules, which is extremely expensive.