TopolearnA rule is a small data pipeline; understanding each stage is what lets you debug "why did this fire?" and "why didn't it?".
A Grafana-managed alert rule is built in the rule editor as a chain of queries and expressions, each with a letter (A, B, C), and exactly one of them is marked as the alert condition.
- Queries run against any backend data source, over a relative window such as last 5 minutes. Each series returned becomes a separate alert instance, labelled with the series labels, so one rule covers every job, pod or disk.
- Reduce turns each time series into a number: function Last, Mean, Min, Max, Sum, Count; mode Strict (NaN if any point is non-numeric), Replace non-numbers, or Drop non-numbers.
- Math evaluates an expression over earlier results:
$B > 0.05,$A / $C * 100,abs($A - $B). Series are matched by labels. - Threshold compares with above, below, within range or outside range, and can define a custom recovery threshold (hysteresis) so an alert at 90% resolves only below 80%.
- Classic condition combines several queries into one instance, which loses the per-series labels; prefer the others.
The condition's result is boolean per instance: 1 fires. The simplified rule editor hides B and C behind "WHEN last() OF A IS ABOVE 0.05" and builds the same chain.
Then the rule gets a folder and evaluation group (which fixes the interval), a pending period such as 5m (the condition must stay true that long; 0 fires immediately) and in recent versions keep firing for, which holds a resolved alert for a while to avoid flapping notifications.
Labels (severity=page, team=api, templates allowed) drive routing. Annotations carry human text: summary, description, runbook_url, templated with {{ $labels.job }}, {{ $values.B }} or {{ humanizePercentage $values.B.Value }}. Link the rule to a dashboard and panel so notifications include a link, and use Preview to see instances before saving.
Example
# Query A (Prometheus, last 5m):
sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
/ sum by (job) (rate(http_requests_total[5m]))
# Expression B: Reduce input A, function Last, mode Strict
# Expression C: Threshold input B, IS ABOVE 0.05,
# custom recovery threshold: below 0.02 <- alert condition
# Rule settings
# Folder: Services Evaluation group: api-fast (every 1m)
# Pending period: 5m Keep firing for: 2m
# Labels: severity=page team=api service={{ $labels.job }}
# Annotations:
# summary = High error ratio for {{ $labels.job }}
# description = {{ humanizePercentage $values.B.Value }} of requests failed over the last 5m
# runbook_url = https://runbooks.example.com/api/error-ratio
# Dashboard: API overview Panel: Error ratioRemember
- Rule = queries + Reduce/Math/Threshold expressions; one is the alert condition
- Every returned series becomes its own alert instance with its labels
- Labels route, annotations explain; both can use {{ $labels }} and {{ $values }}
Watch out for
- Using Classic condition out of habit and losing the per-instance labels the notification policy needs.