Lists, arrays and structs

Polars · Advanced · 15 min

JSON data, group results and multi-output computations all produce nested columns, and handling them natively is far faster than Python loops.

Polars has three nested types:

  • List: a variable-length list per row, e.g. tags or the result of str.split. Methods live in .list: len(), get(0), contains(x), sum(), sort(), unique(), join(","), eval(expr).
  • Array: a fixed-size list (pl.Array(pl.Float32, 3)), good for embeddings or coordinates; methods in .arr.
  • Struct: named fields per row, like a small row inside a cell. Methods in .struct: field("x"), rename_fields, with_fields; df.unnest("col") spreads the fields into columns, and pl.struct("a", "b") packs columns into one.

Moving between shapes: explode("tags") gives one row per list element (empty lists now produce zero rows in Polars 2; pass empty_as_null=True for the old behaviour). The reverse is group_by(...).agg(pl.col("tags")) or .implode().

pl.element() refers to each element inside list.eval, so pl.col("scores").list.eval(pl.element().rank()) runs any expression per list in parallel. Prefer that to a Python lambda.

Structs are also how multi-output functions return several columns at once, like value_counts(), str.extract_groups() or struct.field on JSON decoded with str.json_decode().

Example

import polars as pl

df = pl.DataFrame({
    "user": ["ana", "ben"],
    "tags": ["py,sql", "rust"],
    "geo": [{"lat": 59.9, "lon": 10.7}, {"lat": -12.0, "lon": -77.0}],
})

out = df.with_columns(pl.col("tags").str.split(","))
print(out.select(n=pl.col("tags").list.len(),
                 has_py=pl.col("tags").list.contains("py")).rows())
# [(2, True), (1, False)]

print(out.explode("tags").select("user", "tags").rows())
# [('ana', 'py'), ('ana', 'sql'), ('ben', 'rust')]

print(out.unnest("geo").columns)          # ['user', 'tags', 'lat', 'lon']
print(df.select(pl.col("geo").struct.field("lat")).to_series().to_list())
# [59.9, -12.0]

Remember

  • List = variable length, Array = fixed size, Struct = named fields
  • explode and implode move between nested and flat rows
  • list.eval(pl.element()...) runs expressions inside each list

Watch out for

  • Looping over a list column in Python instead of using the list namespace or list.eval.

Check yourself

  1. What does explode('tags') do to a row whose tags list is empty, in Polars 2 by default?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: docs.pola.rs/user-guide/expressions/lists-and-arrays/