TopolearnAlerting rules are code that pages people; a test file catches a bad threshold or a broken template before it deploys.
promtool can check syntax (promtool check rules file.yml, promtool check config prometheus.yml) and, more valuably, execute rules against fake data with promtool test rules tests.yml.
A test file names the rule_files to load, an evaluation_interval, and a list of tests. Each test has:
interval: spacing of the input samples.input_series: series written in a compact notation.'0+10x5'means start at 0 and add 10 five times (0 10 20 30 40 50);'1 1 0 0'lists values;_is a missing sample andstalea stale marker. Counters are just increasing sequences; a drop simulates a reset.alert_rule_test: ateval_time, which alerts ofalertnameare expected (exp_alerts) with theirexp_labelsandexp_annotations. An emptyexp_alertsasserts the alert is not firing, which is the test most people forget.promql_expr_test: evaluate any expression ateval_timeand compare withexp_samples; use it for recording rules and tricky expressions.
Time starts at 0 and eval_time is relative to it, so a for: 5m alert fires at eval_time: 5m only if the condition held since minute 0. Pending alerts are not reported.
exp_labels must list all labels of the alert, including those from the series and the rule, and annotations are compared after templating, which is how you test humanize output and runbook links.
Run the tests in CI alongside promtool check rules, and promtool check config with --syntax-only when the referenced files aren't present on the build machine.
Example
# tests/api.test.yml -> promtool test rules tests/api.test.yml
rule_files:
- ../rules/api.yml
evaluation_interval: 1m
tests:
- interval: 1m
input_series:
- series: 'http_requests_total{job="api",instance="a",status="200"}'
values: '0+100x20' # 100 req/min, steady
- series: 'http_requests_total{job="api",instance="a",status="500"}'
values: '0+10x20' # 10 errors/min = 9% error ratio
promql_expr_test:
- expr: job:http_error_ratio:rate5m
eval_time: 10m
exp_samples:
- labels: 'job:http_error_ratio:rate5m{job="api"}'
value: 0.09090909090909091
alert_rule_test:
- eval_time: 9m # for: 10m not satisfied yet
alertname: HighErrorRatio
exp_alerts: []
- eval_time: 16m
alertname: HighErrorRatio
exp_alerts:
- exp_labels: {job: api, severity: page}
exp_annotations:
summary: "api error ratio is 9.091%"
# Unit Testing: tests/api.test.yml
# SUCCESSRemember
- promtool test rules runs rules on synthetic input_series
- 'a+bxn' notation; _ gaps and stale markers
- Assert both firing and not-firing, with all labels and rendered annotations
Watch out for
- Forgetting that eval_time counts from 0, so a for: 10m alert tested at 5m 'mysteriously' doesn't fire.