TopolearnGroup-by summaries of logs and CSV exports are among the most useful one-liners you can write.
awk arrays are associative: the index is any string. That makes group-by trivial. count[$1]++ increments a counter for the value in column 1, creating the element on first use. sum[$2] += $3 keeps a running total per key.
To report, loop over the keys in END:
END { for (k in sum) print k, sum[k] }The order of for (k in arr) is unspecified. Pipe the output into sort (for example sort -k2,2nr to sort by the second column, numerically, descending), or in gawk set PROCINFO["sorted_in"] = "@ind_str_asc" before the loop.
Other array tools:
(key in arr)tests membership without creating the element;arr[key] == ""would create it.delete arr[key]removes one element;delete arrclears it (POSIX since 2024, long supported everywhere).- Composite keys:
arr[$1, $2]joins the parts withSUBSEP.
A classic idiom is !seen[$0]++, a pattern that is true the first time a line is seen: awk '!seen[$0]++' removes duplicate lines while keeping the original order, which sort -u can't do.
Example
printf 'GET /a 200\nGET /b 404\nPOST /a 200\nGET /a 500\n' > req.log
# requests per path
awk '{ n[$2]++ } END { for (p in n) print p, n[p] }' req.log | sort
# /a 3
# /b 1
# count per status, most frequent first
awk '{ c[$3]++ } END { for (s in c) print s, c[s] }' req.log | sort -k2,2nr
# 200 2 (then 404 1 and 500 1)
# de-duplicate while keeping first-seen order
printf 'b\na\nb\nc\na\n' | awk '!seen[$0]++' # b / a / cRemember
- arr[key]++ and arr[key] += x build counts and sums per key
- for (k in arr) has no guaranteed order: pipe to sort
- !seen[$0]++ removes duplicates and keeps order
Watch out for
- Testing arr[k] == "" to check membership, which silently creates the key; use (k in arr).