TopolearnKnowing where Python actually runs explains PySpark's performance cliffs, memory errors and debugging quirks.
Spark itself runs on the JVM. PySpark is a thin layer around it.
On the driver, your Python process talks to a JVM through Py4J, a socket-based bridge. A DataFrame in Python is a wrapper around a JVM object (visible as df._jdf). Calls like df.filter(...) just forward to the JVM, which builds and optimizes the plan. Pure DataFrame code therefore runs at JVM speed regardless of Python.
On executors, Python runs only when needed: for Python UDFs, pandas UDFs, applyInPandas, mapInPandas and RDD operations with lambdas. The executor JVM launches Python worker processes (reused across tasks when spark.python.worker.reuse is on) and streams rows to them, pickled for classic UDFs or as Arrow batches for vectorized ones. That data crossing is the cost of Python UDFs, and Python worker memory sits outside the JVM heap, counted against the container's overhead memory.
Consequences: functions sent to executors are pickled with cloudpickle, so closures must be serializable and should not capture big objects (use broadcast variables or load once per partition). Python exceptions on executors come back wrapped in long JVM stack traces; look for the PythonException part.
Spark Connect replaces Py4J with a gRPC client. The Python client builds unresolved plans as protobuf messages and sends them to a remote Spark server. The client is lightweight (pyspark-client), can't touch _jdf or the SparkContext, and survives server-side upgrades.
Example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.master("local[2]").getOrCreate()
import os
df = spark.range(3)
print(type(df._jdf).__name__) # JavaObject (a Py4J proxy)
print(df._jdf.getClass().getSimpleName()) # Dataset (the JVM DataFrame)
driver_pid = os.getpid()
@F.udf("int")
def worker_pid(_):
import os
return os.getpid() # runs inside a Python worker on the executor
pids = {r[0] for r in df.select(worker_pid("id")).collect()}
print(driver_pid not in pids) # True: UDFs run in separate processes
plan = df.select(worker_pid("id"))._jdf.queryExecution().executedPlan().toString()
print("EvalPython" in plan) # True (BatchEvalPython or ArrowEvalPython: rows go to Python)Remember
- Driver: Python wraps JVM objects through Py4J; DataFrame code runs in the JVM
- Executors start Python workers only for UDFs and pandas APIs
- Spark Connect replaces Py4J with a gRPC client-server protocol
Watch out for
- Capturing a large Python object in a UDF closure, so it is pickled and shipped with every task.