TopolearnWhen logs say nothing, tracing system calls reveals the missing file, the hanging connection or the permission problem.
Programs cannot touch files, network or other processes directly: they ask the kernel through system calls such as openat, read, write, connect, execve, clone and mmap. Watching those calls shows what a program really does, regardless of language.
strace traces them:
strace cmdprints every call with arguments and return value; errors appear as= -1 ENOENT (No such file or directory),-ffollows child processes and threads,-p PIDattaches to a running process,-e trace=openat,connector classes like%file,%network,%processfilter,-Tshows time spent in each call,-tttimestamps,-ca summary table,-o filewrites to a file,-s 200shows longer strings.
Classic uses: finding which config file a program looks for (ENOENT on openat), seeing why it gets EACCES, discovering which host or port a hung program waits on (connect, poll), and following which commands a script executes (-e trace=execve -f).
strace relies on ptrace and slows the traced program considerably, so be careful in production. Related tools: ltrace for library calls, lsof -p PID for open files and sockets, /proc/PID/stack and wchan for where a blocked task waits, perf top / perf record for CPU profiling. For low-overhead tracing in production, eBPF tools such as bpftrace and the BCC collection (opensnoop, execsnoop, tcpconnect) observe the same events system-wide.
Example
strace -c ls / > /dev/null # summary: calls, errors and time per syscall
strace -e trace=openat,read cat /etc/hostname
# openat(AT_FDCWD, "/etc/hostname", O_RDONLY) = 3
# read(3, "web1\n", 131072) = 5
strace -f -e trace=execve bash -c 'ls | wc -l' 2>&1 | grep execve # every program run
strace -e trace=%file -o trace.txt ./app; grep ENOENT trace.txt # missing files
sudo strace -p 1234 -f -T -e trace=%network # attach: what is it waiting on?
sudo lsof -p 1234 -a -i # its network sockets
sudo cat /proc/1234/stack # kernel stack of a blocked task
sudo perf top # where CPU time goes, kernel and user space
sudo opensnoop-bpfcc # eBPF: every open() system-wide, low overheadRemember
- Everything a program does with the outside world is a system call
- strace -f -e trace=... -p PID shows what and why it fails or waits
- strace is heavy; prefer eBPF tools for production tracing
Watch out for
- Leaving strace attached to a busy production process, slowing it down dramatically.