TopolearnLazy evaluation explains when work happens, why errors show up late, and why repeated actions recompute everything.
Spark operations fall into two groups.
Transformations (select, filter, withColumn, join, groupBy().agg, orderBy) are lazy. They return a new DataFrame describing a computation but run nothing. Spark only builds up a logical plan.
Actions (count, show, collect, take, first, write...save, toPandas) need a result, so they make Spark optimize the plan and run it as a job, split into stages and tasks on the executors.
Consequences:
- A typo in a column name is caught when the plan is analyzed, which happens eagerly on most transformations. But a runtime problem, such as a bad cast under ANSI mode, appears only when an action runs, possibly far from the line that caused it.
- Every action recomputes the plan from the source unless you cache the DataFrame. Calling
count()and thenshow()reads the input twice. - Spark can optimize across your whole chain, for example pushing a filter written after a join down into the file scan.
Transformations are also classed as narrow (each output partition depends on one input partition: filter, select) or wide (data must move between partitions, a shuffle: groupBy, join, orderBy, distinct). Wide transformations create stage boundaries and are where most time goes.
Example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.master("local[2]").getOrCreate()
df = spark.range(5_000_000)
heavy = (df.withColumn("sq", F.col("id") * F.col("id"))
.filter(F.col("id") % 7 == 0)
.groupBy((F.col("id") % 3).alias("k")).count())
print(type(heavy).__name__) # DataFrame: only a plan so far, no job has run
rows = heavy.orderBy("k").collect() # action: the job runs now
print([tuple(r) for r in rows])
# [(0, 238096), (1, 238095), (2, 238095)]
print(heavy.count()) # 3 (recomputed from scratch)Remember
- Transformations are lazy and build a plan; actions run jobs
- Each action recomputes the plan unless cached
- Narrow transformations stay in a stage; wide ones shuffle
Watch out for
- Calling
count()'just to check' between steps, each time recomputing the whole pipeline.