Working with text (.str)

Names, codes, addresses and free text need cleaning before you can group, join or filter on them reliably.

Text columns expose vectorized string operations through the `.str` accessor. Most mirror Python's str methods but apply to the whole column and pass missing values through as missing:

cleaning: str.strip(), str.lower(), str.title(), str.replace("-", "")
testing: str.contains("abc"), str.startswith("A"), str.isdigit(), str.len()
slicing: str[:3], str.slice(0, 3), str.zfill(5), str.pad
splitting: str.split(",") gives lists; str.split(",", expand=True) gives separate columns; str.get(0) picks an element
combining: s1.str.cat(s2, sep=" ") or simply s1 + " " + s2

Regular expressions: contains and match treat the pattern as a regex by default (pass regex=False for literal text, important if it contains . or (). replace is literal by default since pandas 2.0; pass regex=True for patterns. `str.extract(r"(?P<year>\d{4})-(?P<month>\d{2})")` pulls capture groups into named columns, a clean way to parse codes and IDs. str.extractall and str.findall return every match.

Missing values: tests like contains return missing for missing input, which breaks boolean indexing. Pass na=False to treat them as non-matches.

On dtypes: in pandas 2.x text columns are object dtype by default; the dedicated "string" dtype (astype("string")) makes intent explicit and uses pd.NA. pandas 3.0 makes a string dtype the default for text, backed by PyArrow when it's installed, which is much faster and more memory-efficient. The .str API is the same either way.

For a column with a few repeated values, consider category dtype: .str methods still work and are computed once per category.

Example

import pandas as pd

df = pd.DataFrame({
    "name": ["  Ana LOPEZ", "bo chen ", None, "Cy Diaz"],
    "code": ["ORD-2024-03-17", "ORD-2023-11-02", "ORD-2024-01-09", "BAD"],
})
df["name"] = df["name"].str.strip().str.title()
print(df["name"].tolist())             # ['Ana Lopez', 'Bo Chen', nan, 'Cy Diaz'] (None/nan)

parts = df["name"].str.split(" ", expand=True)
print(parts[1].tolist())               # ['Lopez', 'Chen', nan, 'Diaz']

dates = df["code"].str.extract(r"ORD-(?P<year>\d{4})-(?P<month>\d{2})")
print(dates.values.tolist())           # [['2024','03'], ['2023','11'], ['2024','01'], [nan, nan]]

print(df["name"].str.contains("chen", case=False, na=False).tolist())   # [False, True, False, False]
print(df["code"].str.replace("ORD-", "", regex=False).str.len().tolist())   # [10, 10, 10, 3]
print(df["code"].str.contains(".", regex=False).sum())                   # 0

Remember

.str applies string methods to a whole column, passing missing values through
contains/match use regex by default; replace is literal unless regex=True
str.extract with named groups parses text into columns
Use na=False in tests used for filtering

Watch out for

Filtering with df[s.str.contains('x')] when the column has missing values: the NA in the mask raises or misbehaves; add na=False.

Check yourself

`s.str.contains('a.b')` on a column containing 'axb'. What's the result for that row?
What does `s.str.split('-', expand=True)` return?

Official docs: pandas.pydata.org/docs/user_guide/text.html

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks