Topolearnjq normally loads each JSON value fully into memory; multi-gigabyte files and long log streams need different techniques.
By default jq parses each top-level JSON value completely before running the filter. For newline-delimited JSON that is fine: each line is small and processed independently, so jq streams naturally. Trouble starts with one huge value, such as a 5 GB array, or when you need state across many records.
inputs: with -n, the filter starts with null and calls input (next value) or inputs (all remaining values, lazily). jq -n 'reduce inputs as $r (0; . + $r.bytes)' access.log sums a field across millions of lines without slurping them into an array, unlike -s, which holds everything in memory.
--stream parses incrementally and emits events [path, leaf] for every scalar, plus closing markers [path]. Memory stays small no matter how big the document is.
fromstream(...)reassembles values from events;1 | truncate_stream(inputs)removes the first path element, sojq -cn --stream 'fromstream(1 | truncate_stream(inputs))' big.jsonemits each element of a giant top-level array one at a time.
Stopping early saves time: first(f), limit(n; f), until, and input exit after what they need. jq -n 'first(inputs | select(.level == "error"))' app.log stops at the first error.
Also useful: --seq for RFC 7464 record separators, input_filename to know which file a value came from, and $__loc__ for debugging.
Example
printf '{"bytes":10}\n{"bytes":32}\n{"bytes":8}\n' > access.log
jq -n 'reduce inputs as $r (0; . + $r.bytes)' access.log # 50 (constant memory)
jq -n '[limit(2; inputs | .bytes)]' -c access.log # [10,32]
jq -n 'first(inputs | select(.bytes > 20))' -c access.log # {"bytes":32}
echo '[{"id":1},{"id":2}]' > big.json
jq -c --stream . big.json
# [[0,"id"],1] [[0,"id"]] [[1,"id"],2] [[1,"id"]] [[1]]
jq -cn --stream 'fromstream(1 | truncate_stream(inputs))' big.json
# {"id":1}
# {"id":2}
jq -r 'input_filename' access.log big.json | uniq # access.log big.jsonRemember
- -n with inputs processes records lazily, unlike -s
- --stream emits [path, leaf] events for huge single documents
- fromstream(1 | truncate_stream(inputs)) splits a giant array
Watch out for
- Using -s on a multi-gigabyte log just to count or sum; reduce over inputs instead.