Subqueries

Prometheus · Advanced · 10 min

Questions like 'max request rate in the last day' need a range of a computed value, which only a subquery provides.

Range selectors (x[5m]) work only on raw series. A subquery evaluates an instant expression repeatedly over a window and hands the results to a function as a range vector:

max_over_time( sum(rate(http_requests_total[5m]))[1d:1m] )

The syntax is <expr>[<range>:<resolution>]. The resolution is optional and defaults to the global evaluation_interval; it controls how many evaluations happen. Here the rate is computed every minute across the day (1440 evaluations) and max_over_time picks the peak.

Typical uses:

  • Peaks and percentiles of rates: quantile_over_time(0.99, rate(x[5m])[1h:]).
  • Smoothing a derived ratio: avg_over_time((a / b)[30m:1m]).
  • Alerting on "the error ratio exceeded 5 percent at any point in the last hour".
  • Recording-rule-free aggregation over time in an ad-hoc investigation.

Costs and caveats:

  • A subquery runs the inner expression once per resolution step; [30d:15s] over a heavy expression is enormously expensive. Keep the resolution coarse and the inner query cheap.
  • Steps are aligned to multiples of the resolution, not to the query time, so results can shift slightly between evaluations. offset and @ apply to subqueries too.
  • For anything used on a dashboard or in an alert, a recording rule that stores the inner expression is cleaner and far cheaper; the subquery is then a plain range selector on the recorded series.

A subquery that simply wraps a selector, x[5m:], is pointless and should be the selector x[5m].

Example

# Peak request rate over the last 24 hours, sampled each minute
max_over_time(sum(rate(http_requests_total{job="api"}[5m]))[1d:1m])

# 99th percentile of the 5-minute error ratio across the last hour
quantile_over_time(0.99,
  (sum(rate(http_requests_total{status=~"5.."}[5m]))
   / sum(rate(http_requests_total[5m])))[1h:30s])

# Was the ratio above 5% at any point in the last hour?  (alert expr)
max_over_time(job:http_error_ratio:rate5m{job="api"}[1h:]) > 0.05

# Cheaper equivalent once a recording rule stores the rate:
#   record: job:http_requests:rate5m
max_over_time(job:http_requests:rate5m{job="api"}[1d])

Remember

  • expr[range:resolution] = range vector of a computed value
  • Default resolution is evaluation_interval; coarser is cheaper
  • Prefer recording rules for anything permanent

Watch out for

  • Writing rate(x[5m])[1d:] without a function around it; a bare subquery can't be graphed.

Check yourself

  1. How many times does the inner expression run in max_over_time(sum(rate(x[5m]))[6h:1m])?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: prometheus.io/docs/prometheus/latest/querying/basics/#subquery