Host and container metrics: node, cAdvisor

Infrastructure dashboards and capacity alerts almost always start from these two exporters.

node_exporter runs on every Linux host (port 9100) and reads /proc and /sys. Collectors are enabled with --collector.<name> flags; the defaults cover CPU, memory, filesystems, disk I/O, network and load. In Docker it needs --pid=host and the host's /, /proc and /sys mounted read-only, with --path.rootfs=/host. Key metrics and their idiomatic queries:

CPU: node_cpu_seconds_total{mode} is a counter per core and mode. Utilisation is 1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])).
Memory: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes (not MemFree, which ignores cache).
Disk: node_filesystem_avail_bytes and node_filesystem_size_bytes (filter fstype!~"tmpfs|overlay"), rate(node_disk_read_bytes_total[5m]).
Network: rate(node_network_receive_bytes_total{device!="lo"}[5m]).

The textfile collector reads *.prom files from a directory, letting cron jobs or scripts on the host expose their own metrics without a Pushgateway.

cAdvisor reports per-container resource usage from cgroups. On Kubernetes it is built into the kubelet (/metrics/cadvisor); elsewhere run the gcr.io/cadvisor/cadvisor image. Its metrics carry container, pod, namespace or name labels:

rate(container_cpu_usage_seconds_total[5m]) in cores; compare with the CPU limit.
container_memory_working_set_bytes is what the OOM killer looks at, not container_memory_usage_bytes.
container_network_receive_bytes_total, container_fs_writes_bytes_total.

Filter out the cgroup parents and pause containers (container!="", container!="POD") or totals double-count.

Example

# docker-compose.yml: host metrics from inside a container
services:
  node-exporter:
    image: prom/node-exporter:v1.9.1
    pid: host
    volumes: ["/:/host:ro,rslave"]
    command: ["--path.rootfs=/host", "--collector.textfile.directory=/host/var/lib/node_exporter"]
    ports: ["9100:9100"]
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.52.1
    volumes: ["/:/rootfs:ro", "/var/run:/var/run:ro", "/sys:/sys:ro", "/var/lib/docker:/var/lib/docker:ro"]
    ports: ["8080:8080"]

# PromQL
# host CPU busy %      100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])))
# host memory used %   100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)
# container CPU cores  sum by (name) (rate(container_cpu_usage_seconds_total{name!=""}[5m]))
# container RSS        container_memory_working_set_bytes{name!=""}

Remember

node_exporter: idle-CPU rate, MemAvailable, filesystem avail/size
cAdvisor: container_cpu_usage_seconds_total and memory_working_set_bytes
Mount the host into node_exporter; filter empty container labels in cAdvisor

Watch out for

Alerting on container_memory_usage_bytes, which includes reclaimable page cache and overstates pressure.

Check yourself

Which expression gives the percentage of CPU in use on each host?

Official docs: prometheus.io/docs/guides/node-exporter/

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks