TopolearnWhen a request is slow across five services, a trace shows which hop took the time; metrics and logs can't.
A trace records one request's path through a system as a tree of spans: each span has a name, a start time and duration, a parent, and attributes such as http.status_code or db.statement, plus resource attributes identifying the service (service.name). Applications emit them with OpenTelemetry, and Tempo stores them cheaply in object storage, indexed by trace ID and, with its columnar format, searchable by attributes.
In Explore the Tempo data source offers:
- TraceQL queries.
{ resource.service.name = "api" && span.http.status_code >= 500 }selects traces containing a matching span;duration > 2sfilters by span duration,traceDurationby the whole trace. Pipelines aggregate:{ ... } | count() > 3for traces with more than three such spans,| avg(duration) > 1s. Structural operators find parent/child relations, for example{ span.http.route = "/checkout" } >> { resource.service.name = "payments" }. The Search tab builds these without typing. - TraceQL metrics (
| rate(),| quantile_over_time(duration, .95)by attributes) chart traces like metrics, when the Tempo metrics-generator is enabled. - Paste a trace ID to open it directly.
The trace view is a waterfall: spans nested by parent, bars scaled to duration, with a minimap to zoom, span details (attributes, events, logs) on click, and a critical-path highlight. Long bars with nothing under them mean time spent inside that service; many short sibling bars mean chatty calls that could be batched.
Tempo's metrics-generator can also derive span metrics and a service graph (nodes per service, edges with request rate, error ratio and latency), shown in the Node graph visualization, so RED dashboards exist even for services that have no metrics instrumentation.
Example
# Traces of the api service with a failing span
{ resource.service.name = "api" && span.http.status_code >= 500 }
# Slow checkout requests (any span over 2 seconds)
{ span.http.route = "/checkout" && duration > 2s }
# Traces where checkout calls payments (descendant relation)
{ span.http.route = "/checkout" } >> { resource.service.name = "payments" }
# More than 3 database spans in one trace (N+1 query smell)
{ span.db.system = "postgresql" } | count() > 3
# TraceQL metrics: request rate per route (needs metrics-generator)
{ resource.service.name = "api" } | rate() by (span.http.route)Remember
- Traces are trees of spans with attributes; resource.* identifies the service, span.* the operation
- TraceQL filters spans with { } and aggregates with | count(), | avg()
- The waterfall view shows where time goes; the service graph gives RED per service
Watch out for
- Searching without any service or attribute filter over a long range, which scans most of the trace store.