Filtering, sorting and selecting rows

Polars · Essentials · 10 min

Most questions about data start with 'which rows' and 'in what order'.

df.filter(expr) keeps the rows where a boolean expression is true. Combine conditions with & (and), | (or) and ~ (not), wrapping each comparison in parentheses because of Python operator precedence. Passing several expressions, df.filter(cond1, cond2), means all of them must hold. Rows where the condition is null are dropped, the same as SQL's WHERE.

Handy predicates:

  • pl.col("city").is_in(["Oslo", "Lima"])
  • pl.col("age").is_between(18, 65) (inclusive by default)
  • pl.col("name").str.starts_with("A")
  • pl.col("x").is_null() / .is_not_null()

Sorting: df.sort("age"), df.sort("age", descending=True), or several keys df.sort(["city", "age"], descending=[False, True]). Nulls go first by default; pass nulls_last=True to change that. For the n largest values use df.top_k(3, by="score") or df.bottom_k(...), which is faster than a full sort.

Positional slicing still exists for quick looks: df.head(5), df.slice(10, 5), df[2:4]. But because Polars has no index, real logic should select rows by values, not positions.

Removing duplicates is df.unique(subset=["email"], keep="first", maintain_order=True).

Example

import polars as pl

df = pl.DataFrame({
    "name": ["Ana", "Ben", "Cy", "Di"],
    "age": [34, 17, None, 52],
    "city": ["Oslo", "Lima", "Oslo", "Rome"],
})

adults_oslo_or_rome = df.filter(
    (pl.col("age") >= 18) & pl.col("city").is_in(["Oslo", "Rome"])
)
print(adults_oslo_or_rome["name"].to_list())     # ['Ana', 'Di']  (Cy's null age is dropped)

print(df.sort("age", descending=True, nulls_last=True)["name"].to_list())
# ['Di', 'Ana', 'Ben', 'Cy']
print(df.top_k(2, by="age")["name"].to_list())   # ['Di', 'Ana']

Remember

  • Combine conditions with &, |, ~ and parentheses
  • Rows whose condition is null are filtered out
  • top_k beats sort-then-head for the largest n

Watch out for

  • Writing cond1 and cond2: Python's and doesn't work on expressions; use &.

Check yourself

  1. Which filter keeps people aged 18 or over who do NOT live in Oslo?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: docs.pola.rs/user-guide/expressions/basic-operations/