Dashboard variables and templating

Grafana · Practical · 12 min

One templated dashboard replaces a copy per service, environment or instance.

Variables appear as drop-downs at the top of a dashboard and are referenced in queries as $name (or ${name}). Create them under Settings, Variables. Types include:

  • Query: values from the data source. For Prometheus: label_values(up, job), label_values(http_requests_total{job="$job"}, instance) (variables can depend on each other), or query_result(...) for computed lists.
  • Custom: a fixed comma-separated list.
  • Data source: switch between, say, several Prometheus servers.
  • Interval, Constant and Text box.

With Multi-value or Include All enabled, a variable can hold several values. Grafana formats them for the data source; in PromQL use the regex matcher: {job=~"$job"}. A custom All value like .* avoids huge regexes when there are many values.

Built-in variables are always available: $__interval and $__rate_interval (based on the time range and panel width), $__range (the dashboard range, e.g. for increase(x[$__range])), and $__dashboard, $__user.

Repeat options create one panel, or one row, per selected value: choose the variable in the panel's Repeat options. Variable selections are stored in the URL (?var-job=api), so links share the exact view.

Example

# Variable "job"      (Query, Prometheus)
label_values(up, job)
# Variable "instance" (Query, Multi-value, Include All, depends on $job)
label_values(up{job="$job"}, instance)

# Panel query using both; =~ because instance may hold several values
sum by (instance) (
  rate(http_requests_total{job="$job", instance=~"$instance"}[$__rate_interval]))

# Requests during the whole selected time range (Stat panel)
sum(increase(http_requests_total{job="$job"}[$__range]))

# Shareable link with selections:
# /d/abc123/api?var-job=api&var-instance=All&from=now-6h&to=now

Remember

  • label_values(metric, label) fills drop-downs from Prometheus
  • Use =~ with multi-value and All variables
  • $__rate_interval and $__range are built-in variables

Watch out for

  • Using job="$job" with a multi-value variable; it needs =~ to match several values.

Check yourself

  1. A multi-value variable $env has prod and staging selected. Which matcher works?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: grafana.com/docs/grafana/latest/dashboards/variables/