TopolearnColumn expressions are how you write nearly every Spark transformation.
Spark transformations are built from Column expressions. Refer to a column with F.col("price") or df.price, write literals with F.lit(5), and combine them with operators: F.col("price") * F.col("qty"), >, ==, &, |, ~. Import functions as from pyspark.sql import functions as F; the module has hundreds: F.upper, F.round, F.when, F.concat_ws, F.to_date, F.coalesce and more.
Core methods:
df.select("a", F.col("b") * 2): choose and compute columns.df.withColumn("total", expr): add or replace one column;withColumns({...})does several at once.df.withColumnRenamed("old", "new"),df.drop("x").df.filter(cond)or its aliasdf.where(cond): keep matching rows; rows where the condition is null are dropped.df.orderBy(F.desc("total")),df.limit(10),df.distinct(),df.dropDuplicates(["email"]).
Conditional logic uses F.when(cond, value).when(...).otherwise(default), like SQL CASE WHEN. F.expr("price * qty") and df.selectExpr("price * qty AS total") accept SQL snippets directly.
Calling withColumn in a long loop (hundreds of columns) creates huge plans that are slow to analyze. Build a list of expressions and use one select or withColumns instead.
Example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.master("local[2]").getOrCreate()
df = spark.createDataFrame(
[("tea", 3.0, 2), ("cake", 4.5, 1), ("jam", 2.0, 5)],
"item STRING, price DOUBLE, qty INT",
)
out = (
df.withColumn("revenue", F.col("price") * F.col("qty"))
.withColumn("size", F.when(F.col("qty") >= 5, "bulk").otherwise("single"))
.filter((F.col("revenue") > 4) & (F.col("item") != "cake"))
.select(F.upper("item").alias("item"), "revenue", "size")
.orderBy(F.desc("revenue"))
)
print([tuple(r) for r in out.collect()])
# [('JAM', 10.0, 'bulk'), ('TEA', 6.0, 'single')]
print(df.selectExpr("item", "price * qty AS rev").first())
# Row(item='tea', rev=6.0)Remember
- Build logic from
F.col,F.lit, operators andpyspark.sql.functions select,withColumn(s),filterandorderByare the core verbs- Use
&,|,~with parentheses for conditions
Watch out for
- Using Python
and/orbetween Column conditions, which raises an error.