TopolearnThese engine features explain why built-in expressions beat UDFs by large factors and how to read low-level plan details.
Once Catalyst has a physical plan, Tungsten, Spark's execution layer, makes it fast.
UnsafeRow binary format: rows are stored as compact byte arrays managed by Spark, not as JVM objects. That reduces garbage collection pressure and memory use, and lets sorts and hash aggregations work directly on bytes. Memory can be on-heap or off-heap (spark.memory.offHeap.enabled).
Whole-stage code generation: instead of calling each operator through virtual functions row by row (the Volcano model), Spark fuses a chain of operators, such as scan, filter, project and partial aggregate, into one generated Java function with tight loops, compiled at runtime by Janino. In explain() output, operators prefixed with *(n) belong to codegen stage n. df.explain("codegen") prints the generated code. Python UDFs, some complex expressions and very wide plans break codegen into pieces.
Vectorized columnar reading: Parquet and ORC are decoded in batches into column vectors rather than row by row (spark.sql.parquet.enableVectorizedReader).
Other runtime techniques: dynamic partition pruning uses join results to skip partitions of the other table, and runtime bloom filters reduce shuffle data.
Vendor engines such as Databricks Photon, and open-source native accelerators like Apache Gluten and DataFusion Comet, replace this JVM execution with native vectorized code while keeping the same Spark plans.
Example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.master("local[2]").getOrCreate()
spark.conf.set("spark.sql.adaptive.enabled", "false") # show the static plan
q = (spark.range(10_000_000)
.filter(F.col("id") % 3 == 0)
.select((F.col("id") * 2).alias("x"))
.agg(F.sum("x")))
print(q.first()[0]) # 33333336666666
plan = q._jdf.queryExecution().executedPlan().toString()
print("*(1) Filter" in plan) # True: operators fused into codegen stage 1
q.explain("codegen")
# Found 2 WholeStageCodegen subtrees.
# == Subtree 1 / 2 (maxMethodCodeSize:...) ==
# *(1) HashAggregate(keys=[], functions=[partial_sum(x#4L)], ...)
# +- *(1) Project [(id#0L * 2) AS x#4L]
# +- *(1) Filter ((id#0L % 3) = 0)
# +- *(1) Range (0, 10000000, step=1, splits=2)
# Generated code:
# /* 001 */ public Object generate(Object[] references) { ...Remember
- Tungsten stores rows as compact binary UnsafeRows to cut GC
- Whole-stage codegen fuses operators into one generated function (
*(n)in plans) - Parquet/ORC are read in vectorized column batches
Watch out for
- Inserting Python UDFs into hot paths, which breaks code generation and adds serialization.