TopolearnLogs, exports and ML datasets are often too big to load as one JSON document.
json.load parses the whole document at once, so a 5 GB array needs memory for the text plus all the resulting Python objects, often several times the file size.
JSON Lines (.jsonl, also called NDJSON) solves this with one JSON value per line:
{"id": 1, "event": "login"}
{"id": 2, "event": "logout"}Each line is parsed independently with json.loads, so you can stream records with constant memory, append new records safely, and resume after a bad line. Write with json.dumps(record) plus "\n"; the default output never contains raw newlines because newlines inside strings are escaped. Don't use indent for JSON Lines.
Make readers tolerant: skip blank lines and report the line number of malformed ones instead of aborting a long job.
For streams of JSON values that aren't separated by newlines, JSONDecoder.raw_decode(s, idx) parses one value starting at idx and returns (value, end_index), so you can walk through the buffer value by value.
For genuinely huge single documents, such as one giant array, consider an incremental parser like ijson, or convert the data to JSON Lines or a columnar format like Parquet once.
Example
import json
records = [{"id": 1, "msg": "hi\nthere"}, {"id": 2, "msg": "bye"}]
with open("events.jsonl", "w", encoding="utf-8") as f:
for r in records:
f.write(json.dumps(r) + "\n") # escaped \n keeps one line
f.write("{broken\n")
def read_jsonl(path):
with open(path, encoding="utf-8") as f:
for n, line in enumerate(f, 1):
if not line.strip():
continue
try:
yield json.loads(line)
except json.JSONDecodeError as e:
print(f"line {n}: {e.msg}")
print([r["id"] for r in read_jsonl("events.jsonl")])
# line 3: Expecting property name enclosed in double quotes
# [1, 2]
dec, buf, i = json.JSONDecoder(), '{"a":1}{"b":2}', 0
while i < len(buf):
obj, i = dec.raw_decode(buf, i)
print(obj) # {'a': 1} then {'b': 2}Remember
- JSON Lines stores one JSON value per line: streamable and appendable
json.dumpsoutput never contains raw newlines, so it's safe for JSONLJSONDecoder.raw_decodeparses one value at a time from a buffer
Watch out for
- Writing JSONL records with
indent=2, which spreads each record across several lines.