TopolearnReal-world files are messy; a robust importer reports exactly which row failed and why instead of crashing or silently corrupting data.
The reader is lenient by default: an unbalanced quote may silently swallow the rest of the file into one field, and a stray quote in the middle of a field is kept as text. Make problems visible:
strict=Trueraisescsv.Erroron bad CSV input, such as a character after a closing quote.- A file ending inside a quoted field raises
csv.Error: unexpected end of datawhenstrict=Trueis on. reader.line_numgives the physical line where parsing stopped, so error messages can point at it.csv.field_size_limit()(default 131072 characters) guards against a runaway field; exceeding it raisescsv.Error: field larger than field limit. Raise it deliberately withcsv.field_size_limit(n)if you have big legitimate fields.
Ragged rows (wrong number of fields) aren't CSV syntax errors, so validate them yourself: compare len(row) with the header, or use DictReader's restkey (extra values) and restval (missing values) to detect them.
A robust importer separates concerns: parse with csv (syntax), validate row by row (shape and types), collect errors with line numbers instead of stopping at the first, and decide on a policy: reject the file, skip bad rows, or quarantine them to a separate file.
Example
import csv, io
data = 'id,name,qty\n1,Ada,3\n2,"Bob"x,5\n3,Cy\n4,Di,4,EXTRA\n'
errors, good = [], []
reader = csv.DictReader(io.StringIO(data), strict=True, restkey="_extra")
while True:
try:
row = next(reader)
except StopIteration:
break
except csv.Error as e:
errors.append((reader.line_num, f"syntax: {e}"))
continue
if row.get("_extra") or None in row.values():
errors.append((reader.line_num, "wrong number of fields"))
else:
good.append(row)
print([r["id"] for r in good]) # ['1']
for line, msg in errors:
print(line, msg)
# 2 syntax: ',' expected after '"' (line_num = last line fully read)
# 4 wrong number of fields
# 5 wrong number of fieldsRemember
- strict=True turns malformed quoting into csv.Error
- reader.line_num locates problems; field_size_limit caps huge fields
- Check field counts yourself; restkey/restval reveal ragged rows
Watch out for
- Running a lenient reader over a file with an unbalanced quote, merging thousands of rows into one field.