TopolearnKnowing how an instance moves between states explains every "it fired late", "it never resolved" and "why did I get a DatasourceNoData page".
Rules live in evaluation groups inside a folder. The group sets the evaluation interval (every 1m, 5m, ...), and all rules in it are evaluated on that schedule; a rule's pending period should be a multiple of it. Grafana runs the queries, computes the condition per instance and updates each instance's state:
- Normal: condition false.
- Pending: condition true, but not yet for the whole pending period.
- Alerting (firing): condition true for at least the pending period; a notification is sent to Alertmanager routing.
- Recovering: condition became false but keep firing for has not elapsed yet.
- NoData: the query returned nothing.
- Error: the query or expression failed.
A transition into Alerting sends the alert; a transition back to Normal sends a resolved notification. Alerts are re-sent at the policy's repeat interval while firing.
The rule's No data and Error handling settings decide what those situations map to: Alerting, Normal (OK), NoData/Error as their own states (which raise special DatasourceNoData and DatasourceError alerts with the rule's labels), or Keep last state. Which one is right depends on the rule: for a "disk full" rule, no data probably means the exporter died and should page; for a rarely non-zero metric, no data is Normal. Instances whose series disappear are resolved after a configurable number of missed evaluations (missing series evaluations to resolve), so stale pods don't fire forever.
Rule health (ok, error, nodata) is separate from instance state and shown in the rule list. The state history view (Alerting, History, or on the rule) shows every transition with the values that caused it; it can be stored as annotations or in Loki. Pausing a rule stops evaluation without deleting it. Grafana also exports grafana_alerting_* metrics for monitoring the alerting system itself.
Example
# Timeline of one instance, evaluation every 1m, pending period 5m, keep firing 2m
# t=0 value 0.02 Normal
# t=1 value 0.08 Pending (condition true, 0m of 5m)
# t=5 value 0.07 Pending (4m)
# t=6 value 0.09 Alerting -> notification sent
# t=9 query fails Error -> depends on Error handling (Alerting / Error / OK / Keep last)
# t=10 value 0.01 Recovering (keep firing for 2m)
# t=12 value 0.01 Normal -> resolved notification
# Provisioned rule excerpt showing the handling settings
# for: 5m
# keep_firing_for: 2m
# noDataState: NoData # Alerting | NoData | OK | KeepLast
# execErrState: Error # Alerting | Error | OK | KeepLastRemember
- Normal, Pending, Alerting, Recovering, NoData, Error per instance; health per rule
- Notifications go out on entering Alerting and on resolve, then every repeat interval
- Choose No data and Error handling per rule; missing series resolve after a few evaluations
Watch out for
- Setting a 5m pending period on a group evaluated every 10m, so the alert needs two evaluations and fires after up to 20 minutes.