TopolearnPipelines that work on test data often break on filenames with spaces or on a machine with a different locale.
Filenames may contain spaces, quotes and even newlines; only / and the NUL byte are forbidden. A pipeline that passes names as lines therefore breaks on unusual names. The fix is to separate names with NUL:
find . -name '*.log' -print0writes NUL-terminated names;grep -lZ(or--null) does the same for matching file names; GNUsort -zandsed -zread them;xargs -0splits its input on NUL and runs a command with as many names per invocation as fit.
xargs turns lines of input into arguments. -n 1 passes one per command, -I{} substitutes each item into a template, and -P 4 runs four processes in parallel. Often find ... -exec cmd {} + is simpler and equally safe.
Locale changes what tools do. Under a UTF-8 locale [a-z] and sort order follow language rules, . matches a whole multi-byte character and case-insensitive matching handles accents. With LC_ALL=C:
- characters are bytes, so behaviour is identical on every machine;
- sorting is plain byte order, which
joinandcommexpect; - grep and sort can be noticeably faster on large ASCII data.
Use LC_ALL=C for machine-oriented processing, and keep the user's locale when the output is meant for people or the text isn't ASCII.
Example
mkdir -p demo && cd demo
touch 'plain.log' 'with space.log' 'other.txt'
echo 'ERROR x' > 'with space.log'
# breaks: the name is split into "with" and "space.log"
find . -name '*.log' | xargs grep -l ERROR 2>/dev/null
# safe: NUL-separated names end to end
find . -name '*.log' -print0 | xargs -0 grep -l ERROR # ./with space.log
grep -rlZ ERROR . | xargs -0 -I{} echo "found: {}" # found: ./with space.log
find . -name '*.log' -exec wc -l {} + # also safe
# byte-order sorting, identical everywhere
printf 'b\nB\na\n' | LC_ALL=C sort # B / a / bRemember
- Only NUL can't appear in a filename: use -print0, -Z and xargs -0
- find -exec cmd {} + is a safe alternative to xargs
- LC_ALL=C gives byte semantics: portable results and often faster
Watch out for
- Piping find output into a while read loop or xargs without -0 and breaking on names with spaces.