TopolearnAlerts are only as good as who receives them and when; routing decides whether a page is useful or noise.
A contact point is a named destination with one or more integrations: email, Slack, Microsoft Teams, PagerDuty, Opsgenie, Discord, Telegram, webhook, OnCall, and others. Each has its own settings and a Test button that sends a sample alert. The message content comes from notification templates, Go templates you can customise, for example a Slack title that lists {{ .CommonLabels.alertname }} and a body that loops over .Alerts with their annotations and silence links.
The notification policy tree decides which contact point gets which alerts. The default policy at the root catches everything; child policies add label matchers (team = api, severity =~ "page|critical"). Evaluation walks the tree depth-first and stops at the first matching policy, unless that policy sets continue matching, which lets an alert hit several siblings (a Slack channel and PagerDuty). Each policy can override:
- Group by labels (default
grafana_folder,alertname): alerts with the same values are bundled into one notification. - Group wait (30s): how long to collect alerts before the first notification of a new group.
- Group interval (5m): minimum time between updates for a changed group.
- Repeat interval (4h): how often an unchanged, still-firing group is re-sent.
Mute timings are schedules (weekends, nightly maintenance) attached to a policy; during them notifications are held, not dropped. Silences are ad hoc: label matchers plus a duration, created from a firing alert with one click, and they only stop notifications, the rule keeps evaluating and the alert shows as suppressed.
An alert rule can also name a contact point directly (simplified routing), bypassing the tree, which is handy for small teams. Both contact points and policies can be exported as YAML or Terraform and provisioned.
Example
# Notification policy tree (Alerting > Notification policies), exported form
apiVersion: 1
policies:
- orgId: 1
receiver: default-email # root: catches everything else
group_by: [grafana_folder, alertname]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- receiver: api-pagerduty
object_matchers:
- [team, =, api]
- [severity, =, page]
continue: true # also try the next sibling
- receiver: api-slack
object_matchers:
- [team, =, api]
mute_time_intervals: [weekends]
repeat_interval: 24hRemember
- Contact points send; policies match labels to contact points, first match wins unless continue is set
- Group by, group wait, group interval and repeat interval control bundling and repetition
- Mute timings are scheduled, silences are ad hoc; both only suppress notifications
Watch out for
- A child policy with continue off placed above a more specific one, so the specific policy never receives anything.