TopolearnEvery query and every performance problem in Prometheus comes back to series and their labels.
Every time series is identified by a metric name and a set of labels (key-value pairs):
http_requests_total{method="GET", path="/api/users", status="200"}
Each distinct combination of label values is a separate series. A series is a list of samples: a float64 value and a millisecond timestamp. Prometheus adds two labels to every scraped series: job (from the scrape configuration) and instance (the target's host:port).
Labels make queries powerful: you can sum all requests, or break them down by method or status, with one metric. But they have a cost. The number of series is the product of the label values used. Adding user_id as a label creates a series per user, the classic cardinality explosion that makes Prometheus slow and memory-hungry. Keep label values bounded: HTTP method, status class, route template, never IDs, emails or full URLs.
Naming conventions from the Prometheus docs:
- snake_case, prefixed with the application or domain:
myapp_jobs_processed_total. - Include the base unit as a suffix:
_seconds,_bytes, not milliseconds or megabytes. - Counters end in
_total.
Label names starting with __ are reserved for internal use.
Example
# Exposition format: one line per series, # HELP and # TYPE describe the metric
# HELP http_requests_total Total HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 1027
http_requests_total{method="GET",status="404"} 3
http_requests_total{method="POST",status="200"} 112
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 2.4576e+07
# After scraping, each series also gets job and instance labels:
# http_requests_total{instance="10.0.0.5:8000",job="api",method="GET",status="200"}Remember
- A series = metric name + unique label set; each holds float samples
- job and instance are added at scrape time
- Keep label values bounded; base units and _total in names
Watch out for
- Putting user IDs, request IDs or raw URLs into labels, creating millions of series.