TopolearnInternals explain performance cliffs: slow row access, fragmentation warnings and expensive non-unique lookups.
A DataFrame is not a list of Series. Its data lives in a BlockManager (reachable as the private df._mgr): a collection of blocks, each a 2-D NumPy array holding several columns of the same dtype, plus 1-D blocks for extension arrays such as Int64, category or Arrow types. Each block records which column positions it holds.
Consequences:
- Column operations are cheap, row operations aren't.
df["a"]slices one row of a block;df.iloc[5]must gather a value from every block and find a common dtype, oftenobject. This is whyiterrowsis slow and lossy. - Fragmentation: adding columns one at a time (
df[f"c{i}"] = ...in a loop) appends a new block each time. Past about 100 blocks, pandas emitsPerformanceWarning: DataFrame is highly fragmented. Build all new columns at once withpd.concat([df, new_cols], axis=1)or a dict, or calldf.copy()to consolidate. - Mixed-dtype `to_numpy()` must combine blocks into one array, copying and upcasting.
- Copy-on-write tracks references between blocks and the objects sharing them, so a write knows whether it must copy first.
The Index holds row labels and powers lookups through an index engine, a hash table built lazily on first lookup. Engines specialise by situation:
- Unique labels:
get_locis an O(1) hash lookup. - Monotonic (sorted) labels: slicing and
searchsorteduse binary search, and label slices don't need the hash table;is_monotonic_increasingis cached. - Non-unique, unsorted labels: lookups return masks or arrays of positions and are much slower; alignment between non-unique indexes produces cartesian products.
RangeIndex stores only start, stop and step, costing no memory. Keeping indexes unique and sorted, and resetting to a RangeIndex when labels carry no meaning, keeps both memory and lookups efficient.
Example
import warnings
import numpy as np
import pandas as pd
df = pd.DataFrame({"a": [1, 2], "b": [3, 4], "x": [0.5, 1.5], "s": ["p", "q"]})
print([blk.dtype for blk in df._mgr.blocks]) # e.g. [int64, float64, object/str] (private API)
print(df.iloc[0].dtype) # object: a row mixes dtypes
wide = pd.DataFrame({"k": np.arange(3)})
with warnings.catch_warnings(record=True) as w:
warnings.simplefilter("always")
for i in range(120):
wide[f"c{i}"] = i # one new block per column
print(any("fragmented" in str(x.message) for x in w)) # True
new = pd.DataFrame({f"c{i}": i for i in range(120)}, index=wide.index)
ok = pd.concat([wide[["k"]], new], axis=1) # built in one go
idx = pd.Index([30, 10, 20])
print(idx.is_unique, idx.is_monotonic_increasing) # True False
print(idx.get_loc(20)) # 2 (hash lookup)
dup = pd.Index(["a", "b", "a"])
print(dup.get_loc("a")) # [ True False True]
print(pd.RangeIndex(10**9).memory_usage()) # tiny, no array storedRemember
- Columns of one dtype share a 2-D block; extension types get their own blocks
- Column access is cheap; row access gathers across blocks and upcasts
- Adding columns one by one fragments the frame; add them together
- Unique, sorted indexes give fast hash and binary-search lookups
Watch out for
- Looping over
iloc[i]row access on a mixed-dtype frame, paying a cross-block gather and upcast for every row.