Label matchers and selectors in depth

Prometheus · Essentials · 9 min

Precise selectors are the difference between a query that returns the right series and one that silently returns too many or none.

A selector is a metric name plus any number of matchers in braces. All matchers must hold (they are ANDed), and you can repeat a label with different matchers: {status=~"5..", status!="503"}.

The metric name is just the special label __name__, so http_requests_total{job="api"} equals {__name__="http_requests_total", job="api"}. Matching on __name__ with a regex selects several metrics at once: {__name__=~"node_(cpu|memory).*"}. A selector must have at least one matcher that doesn't match the empty string, so {job=~".*"} alone is rejected.

Missing labels are treated as the empty string. up{env=""} finds targets without an env label and {env!=""} those with one. The regex .+ requires a non-empty value, while .* also matches absent labels.

Regexes use RE2 syntax and are fully anchored, as if wrapped in ^(?:...)$. path=~"/api" matches only exactly /api; use /api.* for a prefix. Alternation =~"GET|POST" is the usual way to pick a few values. Regex matchers are slower than equality on big series sets; prefer = where you can.

Since Prometheus 3.0 metric and label names may contain any UTF-8 characters, such as dots from OpenTelemetry. Such names are written quoted: {"http.server.request.count", "service.name"="api"}. Classic names need no quotes.

Example

# Several matchers on one label
http_requests_total{status=~"5..", status!="503"}

# Several metrics at once via __name__
{__name__=~"node_(cpu|memory).*", instance="web1:9100"}

# Series that lack a label / have the label
up{env=""}
up{env=~".+"}

# Anchored regex: prefix needs .*
http_requests_total{path=~"/api/v1/.*", method=~"GET|HEAD"}

# Prometheus 3.x quoted UTF-8 names (OpenTelemetry style)
{"http.server.request.count", "service.name"="checkout"}

Remember

  • Matchers are ANDed; a label may appear more than once
  • __name__ is a label: regex-select many metrics
  • Missing label == empty string; .+ demands a value
  • 3.x: quote names containing dots or other UTF-8

Watch out for

  • Writing path=~"/api" and matching nothing, because regexes are anchored.

Check yourself

  1. Which selector returns only series of up that carry a non-empty team label?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors