TopolearnExports of millions of rows must not be loaded into memory, and web apps often build CSV without touching disk.
csv readers and writers work with any file-like object and process one row at a time, which gives you two powerful patterns.
Streaming pipelines: chain generators so only one row is in memory at a time.
def rows(path):
with open(path, newline="", encoding="utf-8") as f:
yield from csv.DictReader(f)
big = (r for r in rows("events.csv") if r["type"] == "error")
writer.writerows(big)itertools.islice(reader, 10) previews the first rows; sum(1 for _ in reader) counts records (not lines) without storing them. Group rows into fixed-size batches with itertools.batched(reader, 1000) (3.12+) for bulk database inserts.
In memory: io.StringIO is a file-like object backed by a string. Parse CSV that arrived as text, such as an HTTP response body or an upload: csv.reader(io.StringIO(text)). Build CSV for a download response by writing into a StringIO and calling getvalue(). For bytes streams, wrap them with io.TextIOWrapper(binary, encoding="utf-8", newline="").
Writing to sys.stdout works too, which makes command-line filters easy (pass lineterminator="\n" if you prefer Unix line endings for pipes).
Example
import csv, io, itertools
with open("events.csv", "w", newline="") as f:
w = csv.writer(f)
w.writerow(["id", "type"])
w.writerows([i, "error" if i % 3 == 0 else "ok"] for i in range(1, 100_001))
def read_rows(path):
with open(path, newline="", encoding="utf-8") as f:
yield from csv.DictReader(f)
errors = (r for r in read_rows("events.csv") if r["type"] == "error")
print([r["id"] for r in itertools.islice(errors, 3)]) # ['3', '6', '9']
batches = itertools.batched(read_rows("events.csv"), 40_000)
print([len(b) for b in batches]) # [40000, 40000, 20000]
raw = io.BytesIO("name\nZoë\n".encode("utf-8")) # e.g. an uploaded file
text = io.TextIOWrapper(raw, encoding="utf-8", newline="")
print(list(csv.reader(text))) # [['name'], ['Zoë']]Remember
- Readers and writers are row-at-a-time; chain generators for constant memory
- io.StringIO handles CSV text in memory; TextIOWrapper adapts byte streams
- itertools.islice and batched help preview and chunk large files
Watch out for
- Calling list(reader) on a multi-gigabyte file and running out of memory.