Troubleshooting performance

When production is slow, a calm checklist beats guessing and restarting things.

Work through resources one at a time; for each, ask about utilization, saturation (queueing) and errors.

Load average (uptime, top): the average number of tasks running or waiting over 1, 5 and 15 minutes. On Linux it also counts tasks in uninterruptible I/O wait, so compare it with nproc and check whether CPU or I/O is the cause.
CPU: top or htop per process; mpstat -P ALL 1 per core. High us is your code, high sy the kernel, high wa waiting for I/O, and st (steal) means the hypervisor is giving your VM less than you think.
Memory: free -h (the available column), vmstat 1 with si/so for swapping. When memory runs out, the OOM killer terminates a process; dmesg -T | grep -i oom or journalctl -k show the victims.
Disk: iostat -xz 1 (sysstat package) shows utilization, queue size and latency (await) per device; iotop shows which processes do I/O; df -h rules out a full disk.
Per process: pidstat 1, /proc/<PID>/status, lsof -p for open files and sockets.
Pressure stall information in /proc/pressure/{cpu,memory,io} directly reports the share of time tasks were stalled.

vmstat 1 is the best single first command: its r, b, si/so, wa columns show at a glance which resource is under pressure.

Example

uptime                      # load average: 3.10, 2.05, 1.20  (1, 5, 15 min)
nproc                       # 4 -> a load above 4 means tasks are queuing
vmstat 1 5                  # r=runnable b=blocked si/so=swap wa=I/O wait st=steal
top -o %MEM                 # or htop; sort processes by memory
free -h                     # look at 'available'
cat /proc/pressure/memory   # some avg10=0.00 ... full avg10=0.00 ...
iostat -xz 1 3              # per-disk %util and await latency (sysstat)
sudo iotop -o               # processes doing I/O right now
pidstat -u 1 5              # per-process CPU over time
sudo dmesg -T | grep -iE 'out of memory|killed process'   # OOM killer victims
sudo lsof -p 1234 | wc -l   # how many files/sockets a process holds

Remember

Check CPU, memory, disk and network for utilization, saturation and errors
vmstat 1 is the best first command; load counts I/O waits too
dmesg / journalctl -k reveal OOM kills

Watch out for

Adding CPUs to a machine whose high load average is actually caused by tasks waiting on a slow disk.

Check yourself

A 4-core server has load average 12 but CPU is mostly idle and wa is high. What is the likely bottleneck?
A process disappeared and the kernel log says 'Out of memory: Killed process 4242'. What happened?

Official docs: man7.org/linux/man-pages/man8/vmstat.8.html

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks