TopolearnUnderstanding the pull model and the components tells you where data comes from and where each problem can occur.
Prometheus is a monitoring system built around metrics: numbers that change over time, such as requests served, memory used or queue length. It was created at SoundCloud and is a graduated CNCF project.
The architecture:
- Targets expose their current metric values as plain text on an HTTP endpoint, usually
/metrics. Applications do this with a client library; for software you can't change, an exporter translates (for examplenode_exporterfor Linux host metrics). - The Prometheus server scrapes (pulls) every target on an interval, such as every 15 seconds, and stores the samples in its local time-series database (TSDB).
- PromQL, its query language, powers the built-in UI, Grafana dashboards and rules.
- Alerting rules evaluate PromQL continuously; firing alerts go to Alertmanager, which groups, silences and routes them to email, Slack or paging services.
The pull model means Prometheus knows when a target is down (the scrape fails and up becomes 0), and targets don't need to know where monitoring lives. Short-lived batch jobs that can't be scraped push to a Pushgateway instead.
Prometheus is designed for reliability, not perfect accuracy: each server is standalone, and it's not meant for per-request data like billing. Logs and traces are separate systems.
Example
# A target exposes metrics as text; Prometheus pulls this every scrape interval
curl -s http://localhost:9100/metrics | grep -m3 '^node_'
# node_boot_time_seconds 1.7603e+09
# node_cpu_seconds_total{cpu="0",mode="idle"} 81234.56
# node_cpu_seconds_total{cpu="0",mode="user"} 1520.33
# Prometheus server UI and API on port 9090
curl -s 'http://localhost:9090/api/v1/query?query=up'
# {"status":"success","data":{"resultType":"vector","result":[...]}}Remember
- Targets expose /metrics; Prometheus scrapes (pulls) them on an interval
- Data lives in a local TSDB and is queried with PromQL
- Alerting rules fire into Alertmanager, which routes notifications
Watch out for
- Using Prometheus for per-event data such as individual orders; it stores aggregated numeric samples.