Debugging containers

Docker · Advanced · 14 min

Containers fail in container-specific ways, and minimal images strip out the usual tools.

Work from the outside in.

Why did it stop? docker ps -a shows the exit code, and docker logs shows the last output. Common codes: 0 the process just finished (often a command that was meant to stay in the foreground); 1 an app error; 126 command not executable; 127 command not found (wrong CMD, or a script with Windows line endings or a missing interpreter); 137 SIGKILL (OOM or stop timeout); 143 SIGTERM. docker inspect gives State, Config.Cmd, Mounts and the effective Env.

What is Docker doing? docker events streams creates, starts, dies, OOM kills and health status changes.

A container that crashes on start can't be execed. Start the same image with a shell instead of its command: docker run --rm -it --entrypoint sh myimage, then run the real command by hand and watch it fail.

Images without a shell (distroless, scratch): attach a tools container to the target's namespaces. docker run --rm -it --network container:api --pid container:api nicolaka/netshoot shares its network and process view, so curl localhost:8000, ss -lntp and ps work against the target. Recent Docker Desktop versions offer docker debug api, which does this with a toolbox shell.

Files: docker cp api:/app/log.txt . copies out; docker diff api lists what changed in the writable layer.

Network checklist for "can't connect": same user-defined network? Using the service name, not localhost? Listening on 0.0.0.0? Port published if the caller is outside Docker? docker network inspect and nslookup from a netshoot container answer each one.

Build failures: rerun with --progress=plain --no-cache, or build up to the failing stage with --target and run it interactively.

Example

docker ps -a --filter name=api --format '{{.Status}}'   # Exited (127) 3s ago
docker logs --tail 50 api
docker inspect api --format '{{json .Config.Cmd}} {{.State.OOMKilled}}'
docker events --filter container=api --since 10m

# Crash on start: get a shell in the same image instead
docker run --rm -it --entrypoint sh acme/api:1.4.2

# No shell in the image: borrow its network + PID namespaces
docker run --rm -it --network container:api --pid container:api \
  nicolaka/netshoot
#   ss -lntp ; curl -v localhost:8000/healthz ; ps aux

docker diff api            # A /tmp/x  C /etc  D /app/old.conf
docker cp api:/app/logs/error.log ./error.log

Remember

  • Exit codes: 127 command not found, 137 SIGKILL/OOM, 143 SIGTERM
  • --entrypoint sh opens a shell in an image whose container crashes
  • --network/--pid container:<name> attaches a tools container to a shell-less one
  • docker events, inspect, diff and cp round out the toolkit

Watch out for

  • Trying to docker exec into a container that has already exited — start a fresh one with --entrypoint sh instead.

Check yourself

  1. A container exits immediately with code 127. What is the most likely cause?
  2. How can you run curl against a distroless container's localhost without changing its image?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: docs.docker.com/reference/cli/docker/debug/