TopolearnNarrowing data down to the rows you care about is the most frequent pandas operation.
Filtering is NumPy boolean masking with labels. A comparison on a column returns a boolean Series aligned with the rows, and indexing with it keeps the True rows:
cheap = df[df["price"] < 10]Combine conditions with & (and), | (or) and ~ (not), wrapping each condition in parentheses: df[(df["price"] < 10) & (df["qty"] > 0)]. Python's and/or raise "The truth value of a Series is ambiguous".
Helpers for common conditions:
df["city"].isin(["Oslo", "Lima"])for membership in a set of valuesdf["temp"].between(10, 20)(inclusive on both ends by default)df["name"].str.contains("ann", case=False, na=False)for text;na=Falsetreats missing as no matchdf["x"].notna()to drop rows with a missing value in one column
`df.query("price < 10 and city == 'Oslo'")` expresses the same in a string. Inside query, and/or work, column names are bare, and @var refers to a Python variable. It reads nicely in method chains; backtick-quote column names with spaces.
The filtered result keeps the original index labels, so it has gaps. That's usually fine; call reset_index(drop=True) if you need 0..n-1 again.
To filter and select columns at once, use .loc: df.loc[df["qty"] > 0, ["city", "qty"]]. And if you'll modify the filtered rows as their own table, keep the filtered result as a new variable (taking .copy() was the old advice; with copy-on-write in recent pandas it's no longer needed to avoid warnings).
Example
import pandas as pd
df = pd.DataFrame({
"city": ["Oslo", "Lima", "Cairo", "Perth", "Quito"],
"temp": [3.5, 19.0, 28.2, 22.4, None],
"rain_mm": [49, 1, 0, 14, 80],
})
print(df[df["temp"] > 20]["city"].tolist()) # ['Cairo', 'Perth']
print(df[(df["temp"] > 10) & (df["rain_mm"] < 10)].city.tolist()) # ['Lima', 'Cairo']
print(df[~df["city"].isin(["Oslo", "Lima"])].shape) # (3, 3)
print(df[df["temp"].between(15, 25)].city.tolist()) # ['Lima', 'Perth']
print(df[df["temp"].isna()].city.tolist()) # ['Quito']
limit = 20
print(df.query("temp > @limit or rain_mm > 40").city.tolist()) # ['Oslo', 'Cairo', 'Perth', 'Quito']
wet = df.loc[df["rain_mm"] > 10, ["city", "rain_mm"]]
print(wet.index.tolist()) # [0, 3, 4] labels keptRemember
df[mask]keeps rows where the boolean Series is True- Combine conditions with
& | ~and parentheses isin,between,.str.containsandnotnacover common testsqueryoffers string syntax with@variables
Watch out for
- Comparing with NaN using
== np.nan, which is always False; useisna().