TopolearnDuring an incident you move from a spike in a graph to the log lines and the slow trace behind it; the correlation settings are what make that one click instead of copy-paste.
Metrics, logs and traces each answer a different question; correlations are the configured links that let you move between them in Explore and on dashboards without retyping identifiers. They all rely on shared labels (service, namespace, pod, cluster) being named the same across Prometheus, Loki and Tempo, and on applications writing the trace ID into their logs.
The main links, configured on the data sources:
- Exemplars (Prometheus data source, Exemplars section): Prometheus can store a trace ID alongside histogram samples when started with exemplar storage. Enable exemplars on a panel's query and they show as diamonds on latency graphs; the internal link maps the
trace_idlabel to the Tempo data source, so one click opens the exact slow request. - Derived fields (Loki data source): a regex such as
traceID=(\w+)or a structured-metadata field becomes a link next to each log line, pointing to Tempo with the matched ID. Clicking a line's Logs context shows surrounding lines. - Trace to logs (Tempo data source): a span's service and time range become a Loki query. Map tags (
service.nametoapp), add a time shift before and after the span, and optionally filter by trace or span ID. - Trace to metrics: query templates per span, for example the service's error ratio with
$__tagssubstituted from span attributes. Trace to profiles does the same with Pyroscope. - Correlations (Administration, Plugins and data): generic links from any result field of one data source to a query in another, with regex transformations.
The working pattern in Explore: open the RED graph, hover an exemplar, open the trace in split view, pick the slow span, jump to its logs. On dashboards, data links to Explore give the same path from a panel. When a hop fails, it is almost always a label mismatch or a missing trace ID in the logs.
Example
# Prometheus data source -> Exemplars
# Internal link: Tempo Label name: trace_id
# Panel query with exemplars enabled:
histogram_quantile(0.95, sum by (le) (
rate(http_request_duration_seconds_bucket{job="api"}[$__rate_interval])))
# Loki data source -> Derived fields
# Name: TraceID Regex: traceID=(\w+) Internal link: Tempo Query: ${__value.raw}
# Log line: level=error msg="timeout" traceID=4bf92f3577b34da6a3ce929d0e0e4736
# Tempo data source -> Trace to logs
# Data source: Loki Tags: service.name -> app Span start -1m / end +1m
# Generated for a span of service "api":
{app="api"} |= "4bf92f3577b34da6a3ce929d0e0e4736"
# Tempo -> Trace to metrics, query template using span tags
sum(rate(http_requests_total{$__tags, status=~"5.."}[5m]))Remember
- Exemplars link graph points to traces; derived fields link log lines to traces
- Trace to logs and trace to metrics turn a span into Loki and Prometheus queries by mapping tags
- Shared label names and trace IDs in logs are what make the jumps work
Watch out for
- Tempo's trace-to-logs tag mapping left at defaults, so the generated Loki query uses service.name as a label and returns nothing.