TopolearnA Grafana that is slow is almost always a data source being asked too much, too often; the fixes are in the panel options and dashboard design.
Start with measurement. The Query inspector (panel menu, Inspect, Query) shows every request a panel made, its duration, the number of series and rows, and the exact query sent after variable substitution. The Stats tab gives total request time and data size; the JSON tab the panel model. Prometheus itself logs slow queries and exposes prometheus_engine_query_duration_seconds.
Levers inside a panel's Query options:
- Max data points (default panel width) and Min interval (
1mor more for long ranges) raise the step and shrink the response. Set a dashboard-wide minimum on the data source (timeInterval). - Relative time and Cache timeout; Query caching (Enterprise and Cloud) stores responses per data source for a TTL so ten viewers of a wall board hit Prometheus once.
- The Prometheus data source's incremental querying re-fetches only the new slice on refresh and merges it in the browser, with an overlap window for late samples.
Levers in the dashboard:
- Fewer panels, collapsed rows (their queries don't run), lazy loading (panels off-screen are not queried until scrolled into view), and a sane auto-refresh backed by
min_refresh_interval. - Recording rules in Prometheus for expensive aggregations, especially anything over
[1d]orhistogram_quantileacross many series. - Variables:
label_values(metric, label)with a metric selector is far cheaper thanlabel_values(label)on a busy server, and refresh on time range change re-runs every variable query on each zoom. - Avoid regex matchers with leading
.*and queries without a job filter.
Server side, [dataproxy] timeouts and max_idle_connections, a remote cache for sessions, and watching Grafana's own /metrics (grafana_datasource_request_duration_seconds) complete the picture. Rendering of images for reports and alert screenshots runs in a separate image renderer service that should be sized on its own.
Example
# Panel query options
# Max data points: 500 Min interval: 1m Relative time: (none)
# Cache timeout: 60 (query caching, Enterprise/Cloud)
# Expensive on a dashboard (every panel, every refresh):
histogram_quantile(0.99, sum by (le, route) (
rate(http_request_duration_seconds_bucket[$__rate_interval])))
# Cheap: a Prometheus recording rule evaluated once, used everywhere
# groups:
# - name: api
# rules:
# - record: route:http_request_duration_seconds:p99_5m
# expr: histogram_quantile(0.99, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))
route:http_request_duration_seconds:p99_5m
# Variable queries: narrow the metric instead of scanning all series
label_values(up{job=~"api.*"}, instance) # good
label_values(instance) # scans every seriesRemember
- Inspect first: the query inspector shows requests, timing and series counts
- Min interval, max data points, caching and incremental querying cut data source load
- Collapsed rows, lazy loading, recording rules and narrow variable queries fix slow dashboards
Watch out for
- Variables set to refresh on time range change combined with heavy label_values() queries, so every zoom fires a dozen metadata queries.