TopolearnKnowing which tool does which job is what lets you answer questions about logs and data files in one line.
Unix tools agree on one simple data format: text as a stream of lines. A filter reads lines from standard input (or from files named as arguments), transforms them and writes lines to standard output. Because every filter speaks the same format, you chain them with pipes and each one stays small.
The three classics each have a job:
grepselects lines that match a pattern and drops the rest.sededits lines as they stream past: substitute, delete, insert.awksplits each line into fields and runs a small program on them: filter by column, sum, count, reformat.
Around them sit helpers you'll use constantly: sort, uniq, cut, tr, head, tail, wc and xargs.
A good habit is to build a pipeline one stage at a time. Run the first command, look at the output (pipe into head while exploring), then add the next stage. Each stage is easy to test on its own, and when something goes wrong you can remove stages from the end until the output looks right again.
None of these tools modify their input file unless you explicitly ask (for example sed -i). They read, transform and print, so experimenting is safe.
Example
printf 'GET /home 200\nPOST /login 401\nGET /home 200\nGET /api 500\n' > access.log
grep 'GET' access.log # select: the three GET lines
sed 's/GET/get/' access.log # edit: lower-case the method on every line
awk '{ print $3 }' access.log # fields: 200 401 200 500
# combined: which paths returned 200, and how often?
grep ' 200$' access.log | awk '{ print $2 }' | sort | uniq -c
# 2 /homeRemember
- grep selects lines, sed edits lines, awk works with fields
- Filters read stdin or files and write stdout, so they chain with pipes
- Build pipelines stage by stage, checking the output each time
Watch out for
- Reaching for awk or a script when a single grep would do, or chaining five greps when one awk would be clearer.