TopolearnDashboards and alerts that query raw high-cardinality data get slow; recording rules make them fast and consistent.
A recording rule evaluates a PromQL expression every evaluation_interval and saves the result as a new time series. Dashboards and alerts then query the cheap precomputed series instead of crunching raw data on every refresh.
Rules live in YAML files listed under rule_files in prometheus.yml, organized into groups. Rules within a group run sequentially, so one rule can use the output of an earlier one in the same group; groups run in parallel.
Use recording rules for:
- Expensive aggregations shown on popular dashboards.
- Building blocks of SLO and alerting expressions, so all alerts use identical definitions.
- Reducing cardinality before federation or remote write: keep a per-job aggregate instead of every instance.
The naming convention is level:metric:operations: the aggregation level (labels kept), the original metric name, and the operations applied, for example job:http_requests:rate5m or job_route:http_request_duration_seconds:p95_5m.
Check rule files with promtool check rules, and unit-test them with promtool test rules, which feeds synthetic series into the rules and asserts on the output. The Rules page in the UI shows each rule's last evaluation time and errors.
Example
# rules/api.yml (listed under rule_files: in prometheus.yml)
groups:
- name: api-recording
interval: 30s # optional, overrides evaluation_interval
rules:
- record: job:http_requests:rate5m
expr: sum by (job) (rate(http_requests_total[5m]))
- record: job:http_errors:rate5m
expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
- record: job:http_error_ratio:rate5m # uses the two rules above
expr: job:http_errors:rate5m / job:http_requests:rate5m
# $ promtool check rules rules/api.yml
# SUCCESS: 3 rules foundRemember
- Recording rules store query results as new series
- Name them level:metric:operations
- Rules in a group run in order and can build on each other
Watch out for
- Recording every possible query; rules cost evaluation time and storage too.