TopolearnDashboards only help when someone is looking; alert rules turn them into notifications.
Grafana Alerting has three main parts, found under Alerting:
- Alert rules: one or more queries, then expressions that reduce and compare them, for example query A, Reduce to the last value, Threshold above 0.05. Rules are evaluated by an evaluation group on an interval, and have a pending period: the condition must hold that long before the alert fires, just like Prometheus's
for. Rules can carry labels and annotations such as summary and runbook URL, and can be linked to a dashboard panel. - Contact points: where notifications go: email, Slack, PagerDuty, Microsoft Teams, webhooks and more, with templated messages.
- Notification policies: a tree that matches alert labels to contact points, with grouping, timing and nested policies, the same model as Prometheus Alertmanager. Silences and mute timings suppress notifications temporarily or on a schedule.
Grafana-managed rules can query any backend data source, including SQL databases, and combine several. For Prometheus or Mimir you can alternatively manage data source-managed rules, which are evaluated by the Prometheus ruler and routed by Alertmanager.
Each series returned by the query becomes its own alert instance, labelled by its labels, so one rule covers every instance. Configure No data and Error handling explicitly; the defaults decide whether a broken query pages you.
Example
# Grafana-managed alert rule (conceptual layout of the UI form)
# Query A (Prometheus):
sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
/ sum by (job) (rate(http_requests_total[5m]))
# Expression B: Reduce A -> Last
# Expression C: Threshold B -> Is above 0.05 (alert condition)
# Folder: Services Evaluation group: api (every 1m) Pending period: 5m
# Labels: severity=page team=api
# Annotations:
# summary = Error ratio for {{ $labels.job }} is {{ humanizePercentage $values.B.Value }}
# Notification policy: team=api -> contact point "api-oncall" (PagerDuty)
# No data: Keep last state Error: AlertingRemember
- Rules = queries + Reduce/Threshold expressions + pending period
- Contact points send; notification policies route by labels
- Each returned series becomes a separate alert instance
Watch out for
- Leaving No data handling at its default without thinking, so an exporter outage silently stops alerting or pages everyone.