TopolearnA small set of functions covers most dashboard and alerting needs; knowing which exist saves reinventing them.
Beyond rate() and aggregation, a few dozen functions do most of the work. The important ones, grouped:
Gauges over time: avg_over_time, max_over_time, min_over_time, last_over_time, quantile_over_time(0.9, g[1h]) take a range vector of a gauge and reduce each series' window to one number. delta(g[1h]) is the difference between first and last sample, deriv(g[1h]) a least-squares per-second slope, and predict_linear(g[1h], 4*3600) extrapolates that slope 4 hours ahead. The classic disk alert is predict_linear(node_filesystem_avail_bytes[6h], 24*3600) < 0.
Counters: resets(c[1h]) counts restarts, changes(g[1h]) counts value changes (handy on version gauges or process_start_time_seconds).
Missing data: an expression over no series returns nothing, so alerts can't fire. absent(up{job="api"}) returns a 1 when the selector is empty, and absent_over_time(x[1h]) when there were no samples at all in the window.
Labels: label_replace(v, "dst", "$1", "src", "(.*)-.*") writes a regex capture into a new or existing label; label_join concatenates several labels. Both work on the result, not on storage.
Values: clamp_min, clamp_max and clamp(v, lo, hi) bound values (for example a negative delta after a reset); round, abs, ceil, floor; time() gives the evaluation timestamp and timestamp(v) each sample's own.
Selection: sort, sort_desc order instant vectors for tables; topk/bottomk are aggregation operators.
Functions that take a range vector ([5m]) need a counter or gauge window; functions that take an instant vector work on the current values. Mixing them up is the most common syntax error.
Example
# Disk full within 24h at the current trend
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24*3600) < 0
# Pods that restarted more than twice in the last hour
resets(process_cpu_seconds_total[1h]) > 2
# Job has vanished entirely (returns {job="api"} 1 when nothing matches)
absent(up{job="api"})
# Derive a 'service' label from the pod name: checkout-7d9f-abc -> checkout
label_replace(rate(http_requests_total[5m]), "service", "$1", "pod", "([a-z]+)-.*")
# Peak queue length per hour, clamped to a sane maximum for the panel
clamp_max(max_over_time(queue_length[1h]), 1000)
# Seconds since the process started
time() - process_start_time_secondsRemember
- *_over_time, delta, deriv, predict_linear for gauges
- absent()/absent_over_time() make 'no data' alertable
- label_replace rewrites labels at query time; clamp_* bounds values
Watch out for
- Passing an instant vector to a function that needs a range (e.g. max_over_time(x)), or the reverse.