Running Prometheus and scraping targets

Everything else depends on targets being scraped correctly, and the configuration file is where that happens.

Prometheus is a single Go binary (or the prom/prometheus Docker image) configured by prometheus.yml. Its main sections:

global: defaults such as scrape_interval (the built-in default is 1m; 15s to 30s is common) and evaluation_interval for rules.
scrape_configs: a list of jobs. Each has a job_name, how to find targets (static_configs, or service discovery covered later) and options like metrics_path (default /metrics), scheme and scrape_timeout.
rule_files and alerting: covered later.

Start it with prometheus --config.file=prometheus.yml and open http://localhost:9090. Status, Target health shows every target with its last scrape and error. For each target Prometheus records synthetic series:

up: 1 if the last scrape succeeded, 0 if it failed. The simplest and most important alert is on up == 0.
scrape_duration_seconds and scrape_samples_scraped.

Validate configuration before applying it with promtool check config prometheus.yml. Prometheus reloads its configuration on SIGHUP or, if started with --web.enable-lifecycle, on POST /-/reload.

Storage is in ./data by default; set --storage.tsdb.path and --storage.tsdb.retention.time (default 15 days) for real deployments.

Example

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: prometheus             # Prometheus scrapes itself
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: api
    metrics_path: /metrics
    static_configs:
      - targets: ["10.0.0.5:8000", "10.0.0.6:8000"]
        labels:
          env: prod                  # extra label on every series
# $ promtool check config prometheus.yml   -> SUCCESS
# $ prometheus --config.file=prometheus.yml --storage.tsdb.retention.time=30d
# Query in the UI:  up{job="api"}   -> 1 per healthy target, 0 if down

Remember

scrape_configs lists jobs and their targets
up is 1 for a successful scrape and 0 for a failed one
Check config with promtool; reload with SIGHUP or /-/reload

Watch out for

Forgetting that the default scrape_interval is 1m, then wondering why graphs look coarse.

Check yourself

A target's process is down and the scrape fails. What does Prometheus record?

Official docs: prometheus.io/docs/prometheus/latest/getting_started/

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks