TopolearnCombining monthly files, batches or feature blocks into one table is routine data plumbing.
`pd.concat(list_of_frames)` glues DataFrames (or Series) together along an axis. By default it stacks rows (axis=0).
Columns are aligned by name, not position. If one frame has a column the other lacks, the result contains it with NaN for the rows that didn't have it. join="inner" keeps only columns common to all frames instead.
The original index labels are kept, so stacking two frames with default indexes gives duplicate labels 0, 1, 2, 0, 1, 2. Pass `ignore_index=True` to renumber, which is what you usually want for row-wise stacking. Alternatively keys=["jan", "feb"] adds an outer index level recording where each row came from, which can later become a column via reset_index().
With axis=1, frames are placed side by side and rows are aligned on the index: rows with labels missing in one frame get NaN, much like an outer join on the index. That's convenient for combining several Series that share an index, such as per-day metrics, into one table.
The idiomatic pattern for reading many files:
frames = [pd.read_csv(p) for p in paths]
df = pd.concat(frames, ignore_index=True)Concatenate once at the end. Appending to a DataFrame inside a loop (the removed DataFrame.append, or pd.concat([df, row]) per iteration) copies all existing data each time, which is quadratic. Collect pieces in a list first.
concat is for stacking; when rows should be matched by a key column, use merge. Check dtypes afterwards too: concatenating an int column with a frame that lacks it produces float (because of the NaNs), and mismatched dtypes across frames may produce object.
Example
import pandas as pd
jan = pd.DataFrame({"sku": ["a", "b"], "units": [5, 3]})
feb = pd.DataFrame({"sku": ["a", "c"], "units": [7, 1], "promo": [True, False]})
both = pd.concat([jan, feb], ignore_index=True)
print(both)
# sku units promo
# 0 a 5 NaN
# 1 b 3 NaN
# 2 a 7 True
# 3 c 1 False
tagged = pd.concat([jan, feb], keys=["jan", "feb"], join="inner")
print(tagged.reset_index(level=0).rename(columns={"level_0": "month"}).values.tolist())
# [['jan', 'a', 5], ['jan', 'b', 3], ['feb', 'a', 7], ['feb', 'c', 1]]
views = pd.Series([100, 120], index=["mon", "tue"], name="views")
clicks = pd.Series([4, 9, 2], index=["mon", "tue", "wed"], name="clicks")
print(pd.concat([views, clicks], axis=1)) # wed row: views NaN
parts = [pd.DataFrame({"i": [k]}) for k in range(3)]
print(pd.concat(parts, ignore_index=True)["i"].tolist()) # [0, 1, 2]Remember
pd.concatstacks rows by default and aligns columns by name- Use
ignore_index=Trueto renumber, orkeys=to label sources axis=1places frames side by side, aligned on the index- Collect pieces in a list and concatenate once
Watch out for
- Concatenating inside a loop, which re-copies all data on every iteration.