Scrape configuration and staleness

Scrape settings decide data resolution, label conflicts and what happens when a target or series vanishes.

Each scrape_configs job accepts options that override global defaults.

scrape_interval (default 1m, 15s to 30s is common) and scrape_timeout (default 10s, must not exceed the interval). Use one interval everywhere if you can: mixed intervals make rate() windows hard to choose.
metrics_path, scheme, params for query strings, plus basic_auth, authorization and tls_config.
honor_labels: true: if a scraped sample already carries job or instance (Pushgateway, federation), keep the target's values instead of renaming them to exported_job/exported_instance.
honor_timestamps (default true): accept timestamps the target supplies; set false for misbehaving exporters.
Limits: sample_limit, label_limit, label_value_length_limit, body_size_limit. If exceeded the whole scrape fails and up is 0, protecting the server from a cardinality bomb.
scrape_protocols negotiates the format (OpenMetrics text, Prometheus text, protobuf). Since 3.0 a target that answers without a `Content-Type` header is rejected unless you set fallback_scrape_protocol: PrometheusText0.0.4.

Staleness: when a series present in one scrape is missing from the next, or a target disappears from discovery, Prometheus writes a stale marker and the series stops appearing in instant queries immediately. Without a marker (for example when the scrape simply fails) a query still returns the last sample for up to the lookback delta of 5 minutes (--query.lookback-delta), after which the series is considered gone. This is why up == 0 is a better down-detector than "metric is missing".

Every scrape also records scrape_duration_seconds, scrape_samples_scraped, scrape_samples_post_metric_relabeling and scrape_series_added, useful for spotting slow or exploding targets.

Example

global:
  scrape_interval: 15s
  scrape_timeout: 10s

scrape_configs:
  - job_name: pushgateway
    honor_labels: true             # keep job/instance pushed by batch jobs
    static_configs: [{targets: ["pushgateway:9091"]}]

  - job_name: legacy-exporter
    scrape_interval: 60s           # slow exporter, scrape less often
    scrape_timeout: 30s
    fallback_scrape_protocol: PrometheusText0.0.4   # it sends no Content-Type
    sample_limit: 20000            # fail the scrape rather than ingest a bomb
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: "legacy_debug_.*"
        action: drop
# Slow targets:  scrape_duration_seconds > 5
# Targets that nearly hit the limit:  scrape_samples_scraped > 18000

Remember

scrape_timeout <= scrape_interval; keep intervals uniform
honor_labels keeps pushed job/instance; limits fail dangerous scrapes
Stale markers drop series at once; otherwise the 5m lookback applies
3.x rejects responses without Content-Type unless a fallback protocol is set

Watch out for

Alerting on a metric 'being absent' and getting a 5-minute delay, where up == 0 would fire on the next scrape.

Check yourself

A target was scraped fine, then its process hangs and scrapes time out. What does an instant query of its metrics show right after?

Official docs: prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks