TopolearnInteroperability bugs come from small differences between what producers write and what parsers accept.
There is no single CSV standard. RFC 4180 (2005) documents the common format: records separated by CRLF, fields separated by commas, fields containing commas, quotes or line breaks enclosed in double quotes, and quotes inside quoted fields doubled. An optional header line has the same format.
Python's default excel dialect matches it closely, with deliberate leniencies:
- Line endings: the writer emits
\r\n; the reader accepts\r\n,\nor\r. Inside quoted fields, line breaks are kept exactly, which only works if the file object doesn't translate them, hencenewline="". - Whitespace is significant:
a, bgives' b'unlessskipinitialspace=True. RFC 4180 agrees: spaces are part of the field. - Empty lines are returned as empty lists by
readerand skipped byDictReader. - Quotes inside unquoted fields (
ab"c) are kept literally in non-strict mode; RFC 4180 doesn't allow them. - Final line may or may not end with a newline; both parse the same.
- RFC 4180 says nothing about encoding (it registers
text/csvwith an optionalcharset), types or nulls, which is whyQUOTE_NOTNULL/QUOTE_STRINGSare Python conventions, not portable semantics.
When exchanging data, document dialect, encoding, header and null conventions explicitly.
Example
import csv, io
cases = {
"mixed endings": "a,b\r\nc,d\ne,f\rg,h",
"spaces": "a, b ,c",
"blank lines": "a\n\nb\n",
"stray quote": 'ab"c,d',
"quoted break": '"line1\r\nline2",x',
}
for name, text in cases.items():
print(f"{name:14}", list(csv.reader(io.StringIO(text, newline=""))))
# mixed endings [['a', 'b'], ['c', 'd'], ['e', 'f'], ['g', 'h']]
# spaces [['a', ' b ', 'c']]
# blank lines [['a'], [], ['b']]
# stray quote [['ab"c', 'd']]
# quoted break [['line1\r\nline2', 'x']]
print(list(csv.DictReader(io.StringIO("h\n\n1\n")))) # [{'h': '1'}] (blank skipped)Remember
- Python's excel dialect closely follows RFC 4180, but its reader is more lenient
- Whitespace is data; blank lines give [] in reader and are skipped by DictReader
- Encoding, types and nulls are outside the spec: document them
Watch out for
- Assuming every producer writes RFC 4180; validate stray quotes and spaces from hand-made files.