TopolearnMost wrong dashboards come from treating a counter like a gauge or a gauge like a counter.
The counter/gauge decision shapes everything you can do later, so it's worth making it deliberately.
A counter answers "how many events happened?". Its stored value is the total since the process started, so two instances with different uptimes show unrelated numbers, and after a restart it resets to zero. Never compare or subtract raw counter values by hand: rate() and increase() understand resets and extrapolate across the window. Expose a counter even when you think you want "errors per minute"; the per-minute view is a query, not a metric.
A gauge answers "what is the value right now?": memory in use, temperature, items in a queue, number of goroutines. It can go up and down, so rate() on it is nonsense. Use avg_over_time, max_over_time, delta() or deriv() instead.
A few practical rules:
- If the natural operation is "add one", it's a counter. If it's "set to", it's a gauge.
- A gauge of "events in the last minute" computed by the app is an anti-pattern: Prometheus can't re-window it. Expose the counter.
- Timestamps belong in gauges:
last_backup_timestamp_secondsas Unix time;time() - last_backup_timestamp_secondsthen gives the age. - Counters sampled every 15 seconds miss nothing in total, but short spikes are smoothed by the rate window.
Client libraries forbid decrementing a counter. If you find yourself wanting to, it's a gauge.
Example
# Counter: raw values are not comparable, rate() is
jobs_processed_total{instance="a"} 182034 # up 3 days
jobs_processed_total{instance="b"} 417 # restarted 2 min ago (reset)
rate(jobs_processed_total[5m]) # ~3.2 and ~3.4 jobs/s: comparable
# Gauge: ask about the value itself
queue_length # now
max_over_time(queue_length[1h]) # worst in the last hour
delta(queue_length[30m]) # change, may be negative
# Timestamp gauge -> age
time() - last_backup_timestamp_seconds > 86400 # backups older than a dayRemember
- Counter = cumulative events; only its rate/increase is meaningful
- Gauge = current state; use *_over_time, delta, deriv
- Expose raw counters and timestamps, compute windows in PromQL
Watch out for
- Pre-computing a 'per minute' gauge in the application, which can't be re-aggregated or re-windowed.