TopolearnChoosing the right type decides which questions you can answer later and which PromQL functions apply.
Prometheus client libraries offer four metric types:
- Counter: a value that only goes up, reset to zero when the process restarts. Requests served, errors, bytes sent. Its raw value is rarely interesting; you query its rate of increase. Names end in
_total. - Gauge: a value that goes up and down. Memory in use, queue length, temperature, number of active connections. You look at its current value, average or max.
- Histogram: counts observations (like request durations) into configurable buckets. It exposes several series:
_bucket{le="0.1"}(cumulative count of observations less than or equal to 0.1),_sumand_count. Quantiles such as p95 are computed at query time, and histograms from many instances can be aggregated. - Summary: computes quantiles in the client and exposes them as
{quantile="0.95"}plus_sumand_count. Cheap to query, but quantiles from different instances cannot be meaningfully aggregated.
Rules of thumb: count events with counters, measure states with gauges, and use histograms for latencies and sizes. Prefer histograms to summaries unless you have a single instance and need exact quantiles.
Newer Prometheus versions also support native histograms, which use exponential buckets automatically and store a whole distribution as one series, removing the need to pick bucket boundaries.
Example
# TYPE jobs_processed_total counter
jobs_processed_total 5817 # only increases (resets on restart)
# TYPE queue_length gauge
queue_length 12 # goes up and down
# TYPE request_duration_seconds histogram
request_duration_seconds_bucket{le="0.1"} 940 # cumulative counts
request_duration_seconds_bucket{le="0.5"} 990
request_duration_seconds_bucket{le="1"} 998
request_duration_seconds_bucket{le="+Inf"} 1000
request_duration_seconds_sum 83.2
request_duration_seconds_count 1000
# TYPE rpc_latency_seconds summary
rpc_latency_seconds{quantile="0.95"} 0.31 # computed in the client
rpc_latency_seconds_sum 41.7
rpc_latency_seconds_count 400Remember
- Counter: only increases, query its rate
- Gauge: current value that goes up and down
- Histogram buckets are aggregatable; summary quantiles are not
Watch out for
- Averaging summary quantiles across instances; the result is not a real percentile.