Adding and modifying columns

Deriving new columns from existing ones is the core of data preparation.

New columns come from vectorized expressions on existing ones:

df["total"] = df["price"] * df["qty"]

The right side can be a scalar (broadcast to every row), a Series (aligned by index), or an array of matching length. The same syntax overwrites an existing column.

`df.assign(total=..., margin=...)` does the same but returns a new DataFrame, which suits method chains. Values can be lambdas that receive the DataFrame so far, so later arguments can use earlier ones: df.assign(total=lambda d: d.price * d.qty).

For conditional values:

two cases: np.where(df["qty"] > 0, "in stock", "out")
several cases: np.select([cond1, cond2], [choice1, choice2], default=...)
a lookup: df["code"].map({"N": "North", "S": "South"}); unmatched keys become NaN
numeric bins: pd.cut(df["age"], bins=[0, 18, 65, 120], labels=["child", "adult", "senior"])

Other everyday column operations:

df.rename(columns={"old": "new"})
df.drop(columns=["a", "b"])
df["n"] = df["n"].astype("int32"); pd.to_numeric(s, errors="coerce") turns unparseable text into NaN
df.insert(0, "id", values) to place a column at a given position
df.columns = df.columns.str.lower().str.replace(" ", "_") to tidy names

All of these are vectorized: they run per column in compiled code. Don't loop over rows with iterrows to compute a new column; it's orders of magnitude slower and almost always expressible as one of the forms above. To change values only in some rows, use df.loc[mask, "col"] = value.

Example

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "Item": ["pen", "ink", "pad"],
    "Price": [1.5, 7.0, 3.25],
    "Qty": [10, 0, 4],
    "Region": ["N", "S", "N"],
})
df.columns = df.columns.str.lower()
df["total"] = df["price"] * df["qty"]
df["status"] = np.where(df["qty"] > 0, "in stock", "out")
df["region"] = df["region"].map({"N": "North", "S": "South"})

df = df.assign(
    vat=lambda d: d.total * 0.2,
    gross=lambda d: d.total + d.vat,      # uses vat from the line above
).drop(columns=["status"]).rename(columns={"qty": "units"})

df.loc[df["units"] == 0, "gross"] = np.nan
print(df)
#   item  price  units region  total  vat  gross
# 0  pen   1.50     10  North   15.0  3.0   18.0
# 1  ink   7.00      0  South    0.0  0.0    NaN
# 2  pad   3.25      4  North   13.0  2.6   15.6

Remember

df['new'] = expression creates or overwrites a column, vectorized
assign returns a new DataFrame and accepts lambdas for chaining
np.where, np.select, map and pd.cut cover conditional values
Avoid iterrows for computing columns

Watch out for

Using map with an incomplete dict and not noticing the NaNs it produces for unmatched keys.

Check yourself

`s.map({'a': 1, 'b': 2})` is applied to a Series containing 'a', 'b' and 'c'. What does 'c' become?
Why can a later argument of `df.assign(x=..., y=lambda d: d.x * 2)` use `x`?

Official docs: pandas.pydata.org/docs/reference/api/pandas.DataFrame.assign.html

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks