cgroups v2: limiting resources

Every memory limit in Docker or Kubernetes, and every systemd resource setting, is a cgroup setting underneath.

Control groups organise processes into a hierarchy and account for and limit their resource use. Modern distributions use the unified cgroup v2 hierarchy mounted at /sys/fs/cgroup. Each cgroup is a directory; its files are the interface:

cgroup.procs lists member PIDs (write a PID to move a process),
memory.max hard memory limit (the OOM killer acts inside the group when exceeded), memory.high a throttling threshold, memory.current usage,
cpu.max quota and period, for example 50000 100000 for half a CPU; cpu.weight a relative share,
io.max per-device bandwidth or IOPS limits, pids.max the number of tasks, which stops fork bombs,
memory.events, cpu.stat and *.pressure files for statistics.

Controllers must be enabled for children via cgroup.subtree_control, and in v2 processes live only in leaf cgroups.

systemd owns the tree: slices (system.slice, user.slice), services and scopes are cgroups. So the right way to limit a service is with unit properties such as MemoryMax=, CPUQuota=, TasksMax= and IOWeight= (in a drop-in), and systemd-run -p MemoryMax=... cmd for ad-hoc commands. systemd-cgls shows the tree and systemd-cgtop live usage per group.

Container runtimes create one cgroup per container: docker run --memory 512m simply writes memory.max. Inside a container, /proc/self/cgroup shows which group you are in, and applications should read limits from there rather than from total host memory.

Example

findmnt -t cgroup2                  # /sys/fs/cgroup  cgroup2  ...
cat /proc/self/cgroup               # 0::/user.slice/user-1000.slice/session-3.scope
systemd-cgls --no-pager | head      # tree of slices, services and scopes
systemd-cgtop                       # live CPU/memory per cgroup
cat /sys/fs/cgroup/system.slice/nginx.service/memory.current
# Limit a service (sudo systemctl edit myapp):
#   [Service]
#   MemoryMax=512M
#   CPUQuota=50%
#   TasksMax=200
systemd-run --user --scope -p MemoryMax=200M -p CPUQuota=25% ./batch-job.sh
# The raw interface that systemd and container runtimes use:
sudo mkdir /sys/fs/cgroup/demo
echo 100M | sudo tee /sys/fs/cgroup/demo/memory.max
echo "50000 100000" | sudo tee /sys/fs/cgroup/demo/cpu.max    # half a CPU
echo $$ | sudo tee /sys/fs/cgroup/demo/cgroup.procs          # move this shell in

Remember

cgroup v2 is a directory tree at /sys/fs/cgroup; files are the API
memory.max, cpu.max, io.max and pids.max set limits
Use systemd properties (MemoryMax=, CPUQuota=) rather than writing files by hand

Watch out for

Sizing a JVM or worker pool from host memory inside a container instead of the cgroup's memory.max.

Check yourself

What does writing `50000 100000` to cpu.max do?
A container hits its memory limit. What usually happens?

Official docs: docs.kernel.org/admin-guide/cgroup-v2.html

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks