TopolearnEverything in PySpark goes through a SparkSession, and the driver/executor split explains most of Spark's behaviour.
A Spark application has one driver and many executors. The driver runs your Python program, turns DataFrame code into an execution plan, and schedules tasks. Executors are JVM processes on cluster machines that run those tasks on partitions of the data in parallel and keep cached data. A cluster manager (Kubernetes, YARN or Spark's standalone manager) starts the executors.
Your entry point is the SparkSession:
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("etl").getOrCreate()getOrCreate() returns the existing session if there is one. On Databricks and in managed notebooks a session called spark already exists. For local development use .master("local[*]"), which runs the driver and executors in one process using all cores; local[2] uses two.
Settings go through .config("spark.sql.shuffle.partitions", "64") at build time, or spark.conf.set(...) later for SQL settings. spark.version shows the version, and the Spark UI (usually http://localhost:4040) shows jobs, stages and tasks.
With Spark Connect (SparkSession.builder.remote("sc://host:15002")), your Python client talks to a remote Spark server over gRPC instead of running a driver locally. The DataFrame API is the same. Install with pip install pyspark; Spark needs a Java 17 or 21 runtime.
Example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.master("local[2]").getOrCreate()
print(spark.version.startswith("4.")) # True
print(spark.sparkContext.master) # local[2]
print(spark.sparkContext.defaultParallelism) # 2 (one slot per core)
spark.conf.set("spark.sql.shuffle.partitions", "8")
print(spark.conf.get("spark.sql.shuffle.partitions")) # 8
df = spark.range(1_000_000) # a DataFrame with one column, id
print(df.count()) # 1000000 (run as tasks on the executors)
spark.stop()Remember
- The driver plans and schedules; executors run tasks on partitions
SparkSession.builder...getOrCreate()is the entry pointlocal[*]for development; Spark Connect for remote clients
Watch out for
- Running heavy pure-Python code on the driver and expecting Spark to parallelize it.