TopolearnWithout Alertmanager configuration a single outage sends hundreds of notifications to the wrong people.
Alertmanager receives alerts from one or more Prometheus servers and decides who gets notified, how and when. Point Prometheus at it under alerting: alertmanagers:.
The configuration is a routing tree. The root route has a default receiver; child routes matchers on labels (for example severity="page" or team="payments") send alerts to other receivers. The first matching child wins unless continue: true.
Grouping batches related alerts into one notification: group_by: [alertname, cluster] means one message per alert name and cluster, even if 50 instances fail. Timers control batching:
group_wait(default 30s): wait before the first notification of a new group, to collect more alerts.group_interval(default 5m): wait before notifying about new alerts added to the group.repeat_interval(default 4h): resend a still-firing alert.
Inhibition mutes alerts while another is firing, such as all warnings in a cluster that is entirely down. Silences mute matching alerts for a time window, set in the UI, with amtool, or through the API.
Receivers support email, Slack, PagerDuty, Opsgenie, webhooks and more. Run Alertmanager as a cluster of two or three instances for high availability; they gossip to deduplicate notifications.
Example
# alertmanager.yml
global: {smtp_smarthost: "smtp.example.com:587", smtp_from: alerts@example.com}
route:
receiver: team-email
group_by: [alertname, cluster]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- matchers: ['severity="page"']
receiver: oncall-pager
receivers:
- name: team-email
email_configs: [{to: team@example.com}]
- name: oncall-pager
pagerduty_configs: [{routing_key: <secret>}]
inhibit_rules:
- source_matchers: ['alertname="ClusterDown"']
target_matchers: ['severity="warning"']
equal: [cluster]
# Send a test alert through the API:
# curl -X POST -H 'Content-Type: application/json' localhost:9093/api/v2/alerts \
# -d '[{"labels":{"alertname":"Test","severity":"page"}}]'Remember
- A routing tree matches alert labels to receivers
- group_by plus group_wait, group_interval and repeat_interval control notification volume
- Inhibition and silences mute alerts in known situations
Watch out for
- Grouping by instance, so a cluster-wide outage still sends one notification per machine.