Annotations, dashboard links and data links

Grafana · Practical · 10 min

Deploy markers and drill-down links are what turn a set of dashboards into a navigable system.

Annotations are vertical markers (or shaded regions) drawn on time-based panels. The built-in Annotations & Alerts query shows annotations stored in Grafana's own database: those you add by hand (Ctrl/Cmd-click a graph, or drag to mark a region, with text and tags) and state changes of alert rules linked to the dashboard. Under Settings, Annotations you add query annotations from any data source: a Prometheus expression such as a restart counter, a Loki query for lines containing "deploy", or Grafana's own annotations filtered by tag, so a deploy pipeline can POST {"tags": ["deploy","api"], "text": "v2.3.1"} to /api/annotations and every dashboard with that tag filter shows it. Annotation queries run on every refresh, so keep them cheap.

Dashboard links (Settings, Links) appear at the top: a link to a URL, to a specific dashboard, or a drop-down of all dashboards with certain tags. Tick Include current time range and Include current template variable values so the target opens on the same service and window. Panel links do the same in a panel's menu.

Data links are attached to values: clicking a point, bar, table cell or stat opens a URL built from the data. Variables available include ${__value.raw}, ${__value.time}, ${__field.labels.instance}, ${__series.name}, ${__data.fields.pod}, ${__url_time_range} and ${__from}/${__to}, plus dashboard variables. A link to another dashboard looks like /d/pods?var-pod=${__field.labels.pod}&${__url_time_range}; a link to Explore can carry a prepared query. Correlations (administration) define such links centrally, from one data source's results to a query in another.

Example

# Query annotation (Prometheus): process restarts, title/text from labels
changes(process_start_time_seconds{job="api"}[2m]) > 0
# Title: restart  Text: {{instance}}  Tags: restart

# Deploy markers posted by CI (shown by a Grafana annotation query filtered on tag "deploy")
# curl -X POST https://grafana.example.com/api/annotations \
#   -H "Authorization: Bearer $SA_TOKEN" -H 'Content-Type: application/json' \
#   -d '{"tags":["deploy","api"],"text":"api v2.3.1"}'

# Data link on a "requests by pod" panel -> pod dashboard, same time range
# Title: Open pod
# URL:   /d/pods/pod-detail?var-namespace=$namespace&var-pod=${__field.labels.pod}&${__url_time_range}

# Data link to Explore with Loki logs for the clicked series
# URL:   /explore?left={"datasource":"loki","queries":[{"expr":"{pod=\"${__field.labels.pod}\"}"}],"range":{"from":"${__from}","to":"${__to}"}}

Remember

  • Annotations come from Grafana's DB (manual, alerts, API) or from data source queries
  • Dashboard links can carry the time range and variable values to the target
  • Data links build URLs from the clicked value's labels and time

Watch out for

  • A data link to another dashboard without ${__url_time_range}, so it opens on the default range and the spike is gone.

Check yourself

  1. A CI pipeline should mark each deployment on every dashboard of the service. What is the simplest way?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: grafana.com/docs/grafana/latest/dashboards/build-dashboards/annotate-visualizations/