TopolearnAn alert is only as good as the message the on-call engineer reads at 3 a.m.
Both labels and annotations in an alerting rule are Go templates evaluated per firing series. Available data:
$labels: the series' labels ({{ $labels.instance }}),$value: the sample value,$externalLabels,$externalURL.- Functions:
humanize(1234567 to 1.235M),humanize1024,humanizePercentage(0.042 to 4.2%),humanizeDuration,humanizeTimestamp,printf "%.2f",toUpper,reReplaceAll,title. query "expr"runs another PromQL query inside the template;first,label "name",valuepick from its result. Useful, but every firing alert runs it, so keep it cheap.
Use labels only for routing keys (severity, team): a templated label with $value would create a new alert identity on every change and never resolve. Everything human-readable goes in annotations.
Recommended annotations: summary (one line, used as the notification title), description (what, where, how bad, with values), runbook_url and dashboard_url. A runbook should say what the alert means, how to confirm it, the first three things to try and who to escalate to. If you can't write one, question whether the alert should page.
On the Alertmanager side, notification templates control the message per receiver. They see the whole group: .Status, .GroupLabels, .CommonLabels, .CommonAnnotations and range .Alerts with each alert's .Labels, .Annotations, .StartsAt and .GeneratorURL. Default templates for Slack and email already render summary, so most teams only override title and text. Templates live in files referenced by templates: and are tested by sending a synthetic alert through the API.
Example
groups:
- name: api
rules:
- alert: ApiHighLatency
expr: job_route:http_request_duration_seconds:p99_5m{job="api"} > 0.5
for: 10m
labels:
severity: page
team: checkout
annotations:
summary: "p99 latency high on {{ $labels.route }}"
description: >-
{{ $labels.job }} route {{ $labels.route }} has p99 of
{{ $value | humanizeDuration }} (threshold 500ms) for 10m.
Current rps: {{ with printf "job:http_requests:rate5m{job='%s'}" $labels.job | query }}
{{ . | first | value | humanize }}{{ end }}
runbook_url: https://runbooks.example.com/api/high-latency
dashboard_url: https://grafana.example.com/d/api?var-route={{ $labels.route }}
# Alertmanager Slack receiver using the group data:
# title: '[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }} ({{ .Alerts | len }})'
# text: '{{ range .Alerts }}- {{ .Annotations.summary }}\n{{ end }}'Remember
- $labels, $value and humanize* functions in annotations
- Routing keys in labels; never template $value into a label
- summary, description, runbook_url, dashboard_url on every page
- Alertmanager templates iterate .Alerts of a group
Watch out for
- Putting {{ $value }} in a label, so each value change becomes a new alert that never resolves.