/proc, /sys and sysctl

Linux · Advanced · 12 min

Tools like ps, top and free are just readers of /proc; going direct answers questions they cannot.

/proc is a virtual filesystem where the kernel publishes live information as text files. Nothing is stored on disk; each read is generated on demand.

System-wide files: /proc/cpuinfo, /proc/meminfo, /proc/loadavg, /proc/uptime, /proc/mounts, /proc/cmdline (kernel boot parameters) and /proc/pressure/* (pressure stall information: how long tasks waited for CPU, memory or I/O).

Per-process directories /proc/<PID>/ (and /proc/self for the reader):

  • cmdline (arguments separated by NUL bytes), environ, cwd and exe (symlinks),
  • status (state, memory such as VmRSS, threads, UIDs), limits (ulimits),
  • fd/ (one symlink per open file descriptor, including sockets), cgroup, ns/ (namespaces).

/sys (sysfs) exposes the device and driver model: network interfaces in /sys/class/net, block devices in /sys/block, CPU frequency, power settings and the cgroup tree in /sys/fs/cgroup.

Kernel tunables live in /proc/sys and are managed with sysctl: sysctl net.ipv4.ip_forward reads one, sysctl -w changes it until reboot, and files in /etc/sysctl.d/*.conf make settings permanent (sysctl --system applies them). Common examples: enabling IP forwarding for routers and containers, vm.swappiness, fs.inotify.max_user_watches for file watchers, and network buffer sizes. Change tunables deliberately and document why.

Example

grep -m1 'model name' /proc/cpuinfo
grep -E 'MemTotal|MemAvailable' /proc/meminfo
cat /proc/loadavg                       # 0.15 0.10 0.05 1/312 4567
tr '\0' ' ' < /proc/$$/cmdline; echo    # this shell's command line
readlink /proc/$$/cwd                   # its working directory
grep -E '^(State|VmRSS|Threads)' /proc/$$/status
ls -l /proc/$$/fd | tail -n +2 | head -n 3   # open file descriptors
cat /sys/class/net/lo/mtu               # 65536
cat /proc/sys/net/ipv4/ip_forward       # 0 or 1, same as: sysctl net.ipv4.ip_forward
# sudo sysctl -w vm.swappiness=10                                  # until reboot
# echo 'vm.swappiness = 10' | sudo tee /etc/sysctl.d/99-local.conf # permanent
# sudo sysctl --system                                             # apply all files

Remember

  • /proc/<PID>/ holds cmdline, status, fd, cwd and more for each process
  • /sys exposes devices, drivers and cgroups
  • sysctl -w for now, /etc/sysctl.d/*.conf for permanent

Watch out for

  • Changing a tunable with sysctl -w and losing it at the next reboot.

Check yourself

  1. Where can you see every file and socket a process with PID 812 has open?
  2. How do you make a sysctl setting survive reboots?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: man7.org/linux/man-pages/man5/proc.5.html