One templated dashboard replaces a copy per service, environment or instance.
Variables appear as drop-downs at the top of a dashboard and are referenced in queries as $name (or ${name}). Create them under Settings, Variables. Types include:
- Query: values from the data source. For Prometheus:
label_values(up, job),label_values(http_requests_total{job="$job"}, instance)(variables can depend on each other), orquery_result(...)for computed lists. - Custom: a fixed comma-separated list.
- Data source: switch between, say, several Prometheus servers.
- Interval, Constant and Text box.
With Multi-value or Include All enabled, a variable can hold several values. Grafana formats them for the data source; in PromQL use the regex matcher: {job=~"$job"}. A custom All value like .* avoids huge regexes when there are many values.
Built-in variables are always available: $__interval and $__rate_interval (based on the time range and panel width), $__range (the dashboard range, e.g. for increase(x[$__range])), and $__dashboard, $__user.
Repeat options create one panel, or one row, per selected value: choose the variable in the panel's Repeat options. Variable selections are stored in the URL (?var-job=api), so links share the exact view.
Example
# Variable "job" (Query, Prometheus)
label_values(up, job)
# Variable "instance" (Query, Multi-value, Include All, depends on $job)
label_values(up{job="$job"}, instance)
# Panel query using both; =~ because instance may hold several values
sum by (instance) (
rate(http_requests_total{job="$job", instance=~"$instance"}[$__rate_interval]))
# Requests during the whole selected time range (Stat panel)
sum(increase(http_requests_total{job="$job"}[$__range]))
# Shareable link with selections:
# /d/abc123/api?var-job=api&var-instance=All&from=now-6h&to=nowRemember
- label_values(metric, label) fills drop-downs from Prometheus
- Use =~ with multi-value and All variables
- $__rate_interval and $__range are built-in variables
Watch out for
- Using job="$job" with a multi-value variable; it needs =~ to match several values.