TopolearnResults only matter once they reach a file, a colleague or another system in a usable, correctly typed form.
Every reader has a matching writer.
`df.to_csv(path, index=False)` is universal. index=False is important: otherwise the row index becomes an unnamed first column that reappears as Unnamed: 0 when you read it back. Other options: sep=";", float_format="%.2f", date_format="%Y-%m-%d", na_rep="", columns=[...] to choose and order columns. Compression is inferred from the extension: to_csv("out.csv.gz") writes gzip. CSV loses all type information, so dates, categories and leading zeros need care on the way back in.
`df.to_parquet(path)` (needs pyarrow) keeps dtypes, compresses well and reads back fast. It's the best choice between Python steps, for data lakes, and for anything big. partition_cols=["year"] writes a directory partitioned by a column, which query engines and read_parquet(filters=...) can skip through.
`df.to_excel(path, sheet_name="Summary", index=False)` (needs openpyxl) is for humans. To write several sheets into one workbook, use a writer:
with pd.ExcelWriter("report.xlsx") as xw:
summary.to_excel(xw, sheet_name="Summary", index=False)
detail.to_excel(xw, sheet_name="Detail", index=False)`df.to_json(orient="records", lines=True)` gives one JSON object per line, a common format for APIs and log pipelines; date_format="iso" makes dates readable.
Also available: to_sql for databases, to_feather for fast local interchange, to_markdown and to_html for reports, and to_dict(orient="records") to hand plain Python structures to other code.
Write to a temporary name and rename when the output feeds other jobs, so readers never see a half-written file.
Example
import pandas as pd
from pathlib import Path
out = Path("out"); out.mkdir(exist_ok=True)
df = pd.DataFrame({
"zip": ["02134", "10001"],
"day": pd.to_datetime(["2024-05-01", "2024-05-02"]),
"revenue": [1234.567, 98.1],
})
df.to_csv(out / "rev.csv.gz", index=False, float_format="%.2f", date_format="%Y-%m-%d")
back = pd.read_csv(out / "rev.csv.gz")
print(back.dtypes.astype(str).to_dict()) # zip -> int64, day -> text: CSV lost types
df.to_parquet(out / "rev.parquet", index=False)
print(pd.read_parquet(out / "rev.parquet").dtypes.astype(str).to_dict()) # types kept
with pd.ExcelWriter(out / "report.xlsx") as xw:
df.to_excel(xw, sheet_name="Detail", index=False)
df[["revenue"]].sum().to_frame("total").to_excel(xw, sheet_name="Summary")
print(df.to_json(orient="records", lines=True, date_format="iso").splitlines()[0])
# {"zip":"02134","day":"2024-05-01T00:00:00.000","revenue":1234.567}Remember
- Pass
index=Falsetoto_csvunless the index holds real data - Parquet keeps dtypes and is the default choice between pipeline steps
- Use
pd.ExcelWriterto put several sheets in one workbook - Compression is inferred from file extensions like
.gz
Watch out for
- Round-tripping through CSV and losing leading zeros, dates and categories without noticing.