TopolearnPipelines let small tools combine into powerful one-liners for searching, counting and transforming data.
A pipeline a | b | c connects the stdout of each command to the stdin of the next. All the commands start at the same time and data streams through them, so pipelines work on files far bigger than memory and the first results appear quickly.
The Unix philosophy is to build pipelines from small tools: sort, uniq -c, head, tail, wc -l, cut, tr, grep, awk and sed. A typical pattern: filter lines with grep, extract a field with cut or awk, then sort | uniq -c | sort -rn | head for a top-10 list.
Only stdout goes into the pipe. To include errors, use 2>&1 | or the bash shorthand |&.
The exit status of a pipeline is the status of its last command, so failing-command | tee log.txt looks successful. Bash keeps every stage's status in the PIPESTATUS array, and set -o pipefail makes the pipeline fail if any stage fails; most scripts should turn it on.
When an early command stops reading, say head -n 1 exits, the writer gets a SIGPIPE and stops quietly. That is normal and why yes | head -n 3 terminates.
Example
printf '%s\n' banana apple cherry apple | sort | uniq -c
# 2 apple
# 1 banana
# 1 cherry
seq 1 100 | tail -n 3 | awk '{ s += $1 } END { print s }' # 297
cut -d: -f7 /etc/passwd | sort | uniq -c | sort -rn | head -n 3
ls /nope 2>&1 | wc -l # 1 (stderr sent into the pipe)
ls /nope |& wc -l # 1 (bash shorthand)
false | true; echo $? # 0 (status of the LAST command)
false | true; echo "${PIPESTATUS[@]}" # 1 0
yes | head -n 2 # y / y (yes stops on SIGPIPE)Remember
- a | b feeds a's stdout into b's stdin; stages run concurrently
- Use 2>&1 | or |& to pipe errors too
- A pipeline's status is the last command's unless pipefail is set
Watch out for
- Assuming
cmd | tee logfails when cmd fails: without pipefail it reports tee's success.