Native histograms

They give accurate percentiles with one series per histogram and no bucket planning, at a fraction of the cost of classic histograms.

A classic histogram is a dozen or more series with hand-picked le boundaries, and quantiles are only as good as those boundaries. A native histogram stores a whole distribution in a single sample: buckets are exponential with a configurable schema (schema 3 means a growth factor of about 1.09 per bucket, 8 means 1.003), span sparse ranges, and only non-empty buckets are stored. Samples carry a count, sum, zero-bucket threshold and positive/negative bucket spans.

Enabling, in Prometheus 3.x:

Start the server with --enable-feature=native-histograms (still a feature flag in the 3.x line).
Clients must expose them over the protobuf exposition format; Prometheus negotiates it automatically when the flag is on. The text format can't carry them yet (OpenMetrics 2.0 is adding it).
In client_golang set NativeHistogramBucketFactor: 1.1 on a Histogram; the client picks the matching schema. NativeHistogramMaxBucketNumber plus NativeHistogramMinResetDuration cap memory by widening buckets or resetting after a time.
Per scrape job, always_scrape_classic_histograms: true keeps the classic series alongside for a migration period.

Querying: rate() and increase() work on the whole histogram, sum() and avg() merge histograms with identical schemas, and the dedicated functions take a histogram sample instead of bucket series:

histogram_quantile(0.95, rate(h[5m])) with no by (le).
histogram_count(), histogram_sum(), histogram_avg().
histogram_fraction(0, 0.3, rate(h[5m])): the share of observations between 0 and 0.3, the SLO query, now without needing 0.3 to be a boundary.
histogram_stddev, histogram_stdvar.

Remote write 2.0, Mimir, Thanos and Grafana support native histograms; Alertmanager doesn't care. Custom-bucket native histograms (convert_classic_histograms_to_nhcb) store a classic histogram's exact buckets in the compact form, an interim step for exporters you can't change.

Example

// client_golang: one histogram, native plus classic buckets for migration
dur := promauto.NewHistogramVec(prometheus.HistogramOpts{
	Name:                            "shop_http_request_duration_seconds",
	Help:                            "Request latency.",
	Buckets:                         []float64{0.05, 0.1, 0.25, 0.5, 1}, // classic (optional)
	NativeHistogramBucketFactor:     1.1,     // ~schema 3
	NativeHistogramMaxBucketNumber:  160,
	NativeHistogramMinResetDuration: time.Hour,
}, []string{"route"})
dur.WithLabelValues("/orders").Observe(0.123)

// prometheus --enable-feature=native-histograms
// PromQL on the native histogram (no le label anywhere):
//   histogram_quantile(0.99, sum by (route) (rate(shop_http_request_duration_seconds[5m])))
//   histogram_fraction(0, 0.3, sum(rate(shop_http_request_duration_seconds[5m])))  // share under 300ms
//   histogram_avg(rate(shop_http_request_duration_seconds[5m]))
//   histogram_count(rate(shop_http_request_duration_seconds[5m]))                   // req/s

Remember

One series per histogram, exponential sparse buckets, schema sets resolution
Feature flag plus protobuf scraping; clients opt in with a bucket factor
histogram_quantile without le; histogram_fraction for exact SLO ratios

Watch out for

Writing sum by (le) (rate(h[5m])) on a native histogram, which has no le label and returns nothing.

Check yourself

Which query gives the fraction of requests under 300 ms from a native histogram h?

Official docs: prometheus.io/docs/specs/native_histograms/

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks