TopolearnLatency SLOs are defined in percentiles, and the wrong query silently produces meaningless numbers.
A classic histogram exposes cumulative _bucket counters labelled by le (less or equal). To get a percentile:
1. Take the rate of the buckets over a window. 2. Sum them across instances, keeping le (and any labels you want to break down by). 3. Apply histogram_quantile(φ, ...) with φ between 0 and 1.
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
The result is an estimate: Prometheus assumes observations are spread linearly inside a bucket. If buckets are 0.1 and 0.5 seconds and the p95 falls between them, you can only know it's somewhere in that range. Put bucket boundaries near the values you care about, especially your SLO threshold. If the quantile falls in the +Inf bucket, the result is the highest finite boundary.
The average is cheap and exact: rate(_sum[5m]) / rate(_count[5m]). For SLOs it's often better to ask "what fraction of requests were faster than 300 ms?": rate(bucket{le="0.3"}) / rate(_count) is exact if 0.3 is a bucket boundary.
Native histograms store high-resolution exponential buckets in a single series; histogram_quantile works on them directly without le, with far smaller error and no bucket planning. They must be enabled in both the client library and the Prometheus configuration.
Example
# p95 latency for the whole service
histogram_quantile(0.95,
sum by (le) (rate(http_request_duration_seconds_bucket{job="api"}[5m])))
# p99 per route: keep route as well as le
histogram_quantile(0.99,
sum by (route, le) (rate(http_request_duration_seconds_bucket[5m])))
# Exact average latency
sum(rate(http_request_duration_seconds_sum[5m]))
/ sum(rate(http_request_duration_seconds_count[5m]))
# Share of requests faster than 300 ms (0.3 must be a bucket boundary)
sum(rate(http_request_duration_seconds_bucket{le="0.3"}[5m]))
/ sum(rate(http_request_duration_seconds_count[5m]))Remember
- rate the buckets, sum by (le), then histogram_quantile
- Percentiles are estimates within bucket boundaries
- Ratios against a bucket boundary are exact and ideal for SLOs
Watch out for
- Dropping le in the aggregation (sum by (route) ...), which makes histogram_quantile return nothing.