TopolearnGraphs that change shape when you zoom, gaps, and spiky or empty rate() panels all come from interval maths.
For every panel Grafana decides how many points to request:
- Max data points defaults to roughly the panel's width in pixels; there's no point fetching more values than pixels.
- The interval is the time range divided by max data points, rounded to a friendly value and never below the min interval (the panel's setting, or the data source's scrape interval). For 24 hours on a 1,000-pixel panel that's 86 seconds, rounded to 1m.
$__intervalholds this value, and for Prometheus it also becomes the query step: Prometheus evaluates the expression once per step.
The trap: rate(x[$__interval]). When you zoom in, $__interval can drop below the scrape interval, the window then contains fewer than two samples, and rate() returns nothing. When zoomed out, a step larger than the range window means samples between steps are skipped, so spikes disappear.
$__rate_interval fixes this. It is max($__interval + scrape interval, 4 x scrape interval), using the scrape interval configured on the data source. The window always contains enough samples and always covers the gap between steps. Use it in every rate() and increase() on dashboards.
For long ranges, a coarser min interval or recording rules keep queries cheap. Relative time and time shift options let individual panels deviate from the dashboard range.
Example
# Data source scrape interval: 15s; panel 1000 px wide
# Range 1h -> $__interval ~ 5s (min interval 15s applies) -> step 15s
# $__rate_interval = max(15s + 15s, 60s) = 1m
# Range 24h -> $__interval 86s, rounded to 1m -> step 1m
# $__rate_interval = max(1m + 15s, 60s) = 1m15s
# Range 7d -> $__interval ~ 10m -> step 10m
# $__rate_interval = max(10m + 15s, 60s) = 10m15s
# Fragile: may be empty when zoomed in, and skips spikes when zoomed out
rate(http_requests_total[$__interval])
# Robust in every zoom level
rate(http_requests_total[$__rate_interval])Remember
- Max data points ~ panel width; interval = range / max data points, at least min interval
- $__interval is also the Prometheus step
- Use $__rate_interval for rate() and increase()
Watch out for
- Leaving the data source scrape interval at the default when Prometheus scrapes every 60s, making $__rate_interval too short.