TopolearnTools used in pipelines like cat big.log | python tool.py | head must not crash or garble data.
A good filter reads stdin incrementally, writes stdout, and plays well with the rest of the pipeline.
Closed pipes. In python gen.py | head -n 3, head exits after three lines. The next write raises BrokenPipeError, and an ugly traceback appears. The recipe from the Python docs: catch it, point stdout at os.devnull so the final flush at shutdown doesn't fail again, and exit with code 1.
Encodings. Text streams use the locale encoding unless UTF-8 mode is active; Python 3.15 will make UTF-8 mode the default. Set PYTHONIOENCODING=utf-8, run with -X utf8, or call sys.stdout.reconfigure(encoding="utf-8", errors="replace") to be explicit.
Binary data. Images or compressed data must go through sys.stdin.buffer and sys.stdout.buffer, never through the text layer.
Many inputs. fileinput.input() iterates over the lines of every file named in sys.argv[1:], or stdin if there are none, mirroring tools like cat and grep.
Interrupts. Ctrl-C raises KeyboardInterrupt; catch it in main and exit with code 130 to avoid a traceback.
Example
import subprocess
import sys
gen = '''
import os, sys
try:
for i in range(100_000):
print(f"line {i}")
sys.stdout.flush()
except BrokenPipeError:
devnull = os.open(os.devnull, os.O_WRONLY)
os.dup2(devnull, sys.stdout.fileno()) # silence the shutdown flush
sys.exit(1)
'''
p1 = subprocess.Popen([sys.executable, "-c", gen], stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
first = p1.stdout.readline()
p1.stdout.close() # reader goes away, like `head`
_, err = p1.communicate()
print(first.decode().strip()) # line 0
print(p1.returncode, err == b"") # 1 TrueRemember
- Catch BrokenPipeError, redirect stdout to devnull, and exit non-zero
- Use
sys.stdin.buffer/sys.stdout.bufferfor bytes fileinput.input()reads named files or stdin, like classic Unix tools
Watch out for
- Decoding binary input through
sys.stdin; it raises UnicodeDecodeError or corrupts the data.