TopolearnThe classic subprocess hang happens when parent and child both wait on a full pipe.
An OS pipe has a small buffer (often 64 KiB). If a child writes more than that to a PIPE that nobody is reading, the child blocks. If the parent is meanwhile blocked too, say in p.wait() or writing to the child's stdin, both wait forever: a deadlock.
Typical traps:
p.wait()withstdout=PIPEand lots of output,- reading
p.stdoutto the end while the child is stuck writing to a fullstderrpipe, - writing a large input to
p.stdinbefore reading output.
p.communicate(input=None, timeout=None) solves this: it writes input, reads stdout and stderr concurrently (with threads or select), and waits. run uses it internally. If it times out, the process is not killed; call p.kill() then p.communicate() again to collect the rest.
To build a real streaming pipeline, connect processes with OS pipes:
p1 = Popen(["cat", "big.log"], stdout=PIPE)
p2 = Popen(["grep", "ERROR"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close() # so p1 gets SIGPIPE if p2 exits early
out, _ = p2.communicate()Data flows directly between the children without passing through Python.
Example
import subprocess, sys
big = "import sys; sys.stdout.write('x' * 1_000_000); sys.stderr.write('done')"
p = subprocess.Popen([sys.executable, "-c", big],
stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
# p.wait() here could deadlock: 1 MB won't fit in the pipe buffer
out, err = p.communicate(timeout=10)
print(len(out), err, p.returncode) # 1000000 done 0
gen = subprocess.Popen(["seq", "1", "100000"], stdout=subprocess.PIPE)
grep = subprocess.Popen(["grep", "-c", "7"], stdin=gen.stdout,
stdout=subprocess.PIPE, text=True)
gen.stdout.close() # let gen get SIGPIPE if grep exits
print(grep.communicate()[0].strip()) # 40951 (lines containing a 7)
gen.wait()Remember
- A full, unread pipe blocks the child; waiting on it blocks you: deadlock
- communicate() reads all pipes concurrently and waits safely
- Connect Popen processes with stdin=p1.stdout for streaming pipelines
Watch out for
- Calling p.wait() on a process whose stdout or stderr is a PIPE you haven't read.