TopolearnTurning a 10-minute import into a 1-minute one usually takes a few simple changes.
The csv module's parser is written in C (_csv), so parsing itself is fast; most time goes into your per-row Python code. Typical speedups:
- `reader` over `DictReader`: DictReader builds a dict per row and is noticeably slower. On hot paths, read the header once, look up column indexes, and use plain lists.
- Avoid repeated work inside the loop: precompile regexes, hoist attribute lookups, convert only the columns you need.
- Batch I/O:
writer.writerows(generator)beats callingwriterowin a Python loop, and large file buffers (open(..., buffering=1 << 20)) cut syscalls. - Batch downstream work: insert into databases in batches (
executemany), not row by row. - Parallelism: CSV parsing is sequential, but CPU-heavy per-row work can be split across processes in chunks.
Measure first with time.perf_counter or cProfile.
For numerical analysis or files of many gigabytes, columnar tools are in another league: pandas.read_csv (C or pyarrow engine), polars.read_csv/scan_csv (multithreaded, lazy), and DuckDB querying CSV directly. Use the csv module for streaming transformations, small-to-medium files and zero-dependency scripts; reach for those when you need aggregation over hundreds of millions of rows.
Example
import csv, io, time
text = "id,name,qty\n" + "".join(f"{i},user{i},{i % 7}\n" for i in range(300_000))
def timed(label, fn):
t = time.perf_counter(); result = fn()
print(f"{label:12} {time.perf_counter() - t:.3f}s total={result}")
def with_dictreader():
return sum(int(r["qty"]) for r in csv.DictReader(io.StringIO(text)))
def with_reader():
rows = csv.reader(io.StringIO(text))
qty = next(rows).index("qty")
return sum(int(r[qty]) for r in rows)
timed("DictReader", with_dictreader) # e.g. DictReader 0.250s total=899997
timed("reader", with_reader) # e.g. reader 0.110s total=899997Remember
- Parsing is C-fast; per-row Python work dominates
- Plain reader with column indexes beats DictReader on hot paths
- Switch to pandas, polars or DuckDB for huge analytical workloads
Watch out for
- Optimising the parser when the real cost is one database insert per row.