offset and @ modifiers

Prometheus · Practical · 9 min

Comparing with yesterday or last week and freezing a value at a point in time are everyday dashboard needs.

Both modifiers attach to a selector (instant or range) or to a subquery, never to the whole expression.

offset shifts the evaluation time backwards by a duration: http_requests_total offset 1w gives the value as it was one week before the query time. With a range vector it moves the whole window: rate(http_requests_total[5m] offset 1d) is the rate at the same time yesterday. Negative offsets (offset -1h) look into the future relative to the evaluation time and are mainly useful inside range queries.

The @ modifier fixes the evaluation time to an absolute Unix timestamp: up @ 1760700000. In a range query every step would normally evaluate at a different time; with @ the selector returns the same value at every step, which is how you draw a constant "value at deploy time" line. Two special forms, @ start() and @ end(), refer to the start and end of the range query, so topk(5, rate(x[5m]) @ end()) ranks series by their latest value across the whole graph instead of re-ranking at every step and making lines appear and disappear.

The order is always selector [range] offset <d> @ <ts> or @ first, both are accepted.

Typical uses:

  • Week-over-week: rate(x[5m]) / rate(x[5m] offset 1w).
  • Stable top-N in dashboards with @ end().
  • Reproducing an incident: pin a query to the time of the page.

Remember that both modifiers only change when data is read; they don't fill gaps, so a comparison with a week when the service didn't exist yields nothing.

Example

# Value one day ago
node_memory_MemAvailable_bytes offset 1d

# Traffic now versus the same 5-minute window last week (ratio)
sum(rate(http_requests_total[5m]))
  / sum(rate(http_requests_total[5m] offset 1w))

# Constant line in a graph: requests/s at a fixed moment
sum(rate(http_requests_total[5m]) @ 1760700000)

# Stable top 5 across a dashboard panel
topk(5, sum by (route) (rate(http_requests_total[5m]) @ end()))

# Both together: the window ending one hour before the end of the graph
rate(http_requests_total[5m] offset 1h @ end())

Remember

  • offset <duration> shifts a selector into the past
  • @ <timestamp> pins it to one moment; start()/end() refer to the range query
  • Use @ end() for a stable topk in graphs

Watch out for

  • Writing rate(x[5m]) offset 1d instead of rate(x[5m] offset 1d); the modifier belongs to the selector.

Check yourself

  1. Why does topk(3, rate(x[5m])) make lines flicker in a Grafana graph, and what fixes it?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: prometheus.io/docs/prometheus/latest/querying/basics/#offset-modifier