TopolearnAlerts are how monitoring reaches people; noisy or missing alerts are the main failure of monitoring setups.
An alerting rule is a PromQL expression: every series it returns is an active alert, carrying that series' labels.
for: 5m: the condition must hold for 5 minutes before the alert goes from pending to firing. This filters short blips; use it on almost every alert.keep_firing_for: 5m: keep firing for a while after the condition clears, to avoid flapping.labels: extra labels such asseverity: page, used by Alertmanager for routing.annotations: human text, templated with{{ $labels.instance }}and{{ $value }}, plus a runbook link.
Firing alerts are sent to Alertmanager, and appear in the UI's Alerts page and as the ALERTS series.
Good alerting practice:
- Alert on symptoms users feel: error ratio, latency, availability, rather than every possible cause like CPU usage.
- Every page should be actionable and link to a runbook.
- Use
absent()orabsent_over_time()to catch metrics that stopped arriving, since an expression over nothing returns nothing and never fires. - For SLOs, burn-rate alerts compare error rates over a short and a long window.
Test alert logic with promtool test rules, including the for duration and the expected labels and annotations.
Example
groups:
- name: api-alerts
rules:
- alert: HighErrorRatio
expr: job:http_error_ratio:rate5m{job="api"} > 0.05
for: 10m
labels:
severity: page
annotations:
summary: "{{ $labels.job }} error ratio is {{ $value | humanizePercentage }}"
runbook_url: https://wiki.example.com/runbooks/api-errors
- alert: TargetDown
expr: up == 0
for: 5m
labels: {severity: ticket}
annotations:
summary: "{{ $labels.instance }} of job {{ $labels.job }} is down"
- alert: ApiMetricsMissing
expr: absent(up{job="api"})
for: 10m
labels: {severity: page}Remember
- Each series returned by expr is an alert; for: delays firing
- Labels route alerts; templated annotations explain them
- Alert on user-facing symptoms and catch missing data with absent()
Watch out for
- Alerts without for: that page on every momentary spike.