Service discovery and relabeling

In dynamic environments static target lists are useless; relabeling is the most powerful and most confusing part of Prometheus config.

Service discovery mechanisms find targets automatically: kubernetes_sd_configs (pods, services, endpoints, nodes), cloud ones (ec2_sd_configs, gce_sd_configs, azure_sd_configs), consul_sd_configs, dns_sd_configs and file_sd_configs (JSON or YAML files that another tool writes). Each discovered target arrives with meta labels such as __meta_kubernetes_pod_label_app or __meta_ec2_tag_Name.

relabel_configs run once per target before scraping, as an ordered list of steps. Each step reads source_labels (joined with ;), matches a fully anchored regex, and applies an action:

keep / drop: keep or discard targets whose value matches.
replace (default): write replacement (with $1 capture groups) to target_label.
labelmap: copy labels whose names match the regex, for example all pod labels.
hashmod: shard targets across several Prometheus servers.

Special labels control the scrape: __address__, __scheme__, __metrics_path__ and __param_<name>. After relabeling, all labels starting with __ are removed.

metric_relabel_configs use the same syntax but run on every scraped sample, after the scrape: the place to drop expensive metrics or labels before storage. Check the result on Status, Service discovery, which shows labels before and after relabeling.

Example

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # only pods annotated prometheus.io/scrape: "true"
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      # custom port from an annotation: 10.1.2.3:80 + 9102 -> 10.1.2.3:9102
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - action: labelmap                # copy all pod labels
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
    metric_relabel_configs:
      - source_labels: [__name__]       # drop a noisy metric before storage
        regex: go_gc_duration_seconds.*
        action: drop

Remember

Service discovery yields targets with __meta_* labels
relabel_configs shape targets before scraping; metric_relabel_configs filter samples after
Labels starting with __ vanish after relabeling

Watch out for

Using relabel_configs to drop metrics; they act on targets, so use metric_relabel_configs.

Check yourself

You want to stop storing one high-cardinality metric from an exporter. Where?

Official docs: prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks