Topolearn

Learn PySpark

Python API for Apache Spark: distributed DataFrames, SQL and streaming

21 lessons · 5 hours · 4 levels · covers PySpark 4.x (Spark 4.2)

PySpark is the Python interface to Apache Spark, the engine for processing large datasets across a cluster with DataFrames, SQL, streaming and machine learning. It is the standard tool for big-data ETL on platforms like Databricks, EMR, Dataproc and Synapse. Learn it to transform terabytes with code that looks like pandas or SQL, and to understand how distributed query engines plan and run work.

Sign in free to track progressOfficial docs ↗

Before you start

PySpark builds on these; the tree links to the exact lessons you need.

PythonPractical levelSQLEssentials level

How deep do you want to go?

1. EssentialsRead and write simple code, do the everyday basics7 lessons · 1 hour in total
2. PracticalUse it productively at work, with the usual idioms14 lessons · 3 hours in total
3. AdvancedThe less common features, performance and trade-offs18 lessons · 4 hours in total
4. ExpertInternals and the edge cases experts know21 lessons · 5 hours in total

The PySpark tree

PySparkSparkSession and the cluster modelSparkSession and thecluster modelCreating DataFrames and schemasCreating DataFramesand schemasColumns, expressions and filtersColumns,expressions and filt…Transformations, actions and lazy evaluationTransformations,actions and lazy ev…Reading and writing data sourcesReading and writingdata sourcesGrouping and aggregationGrouping andaggregationData types, casting, nulls and ANSI modeData types, casting,nulls and ANSI modeSpark SQL and temporary viewsSpark SQL andtemporary viewsJoins in SparkJoins in SparkWindow functionsWindow functionsArrays, structs, maps, JSON and VARIANTArrays, structs,maps, JSON and V…Python UDFs and pandas UDFsPython UDFs andpandas UDFsPartitions, shuffles and repartitionPartitions, shufflesand repartitionCaching, persistence and checkpointsCaching, persistenceand checkpointsModules and importsModules and importsDictionariesDictionariesGROUP BY and HAVINGGROUP BY andHAVINGNULL and three-valued logicNULL andthree-valued logicSELECT: reading dataSELECT: reading dataInner joinsInner joinsWindow functionsWindow functionsDecoratorsDecoratorsThreads, processes and concurrent.futuresThreads, processesand concurrent.fut…
Essentials and Practical shown (14 of 21 lessons); the full list is below. Click a lesson to read it.

All 21 lessons, in order

SparkSession and DataFrames

  1. SparkSession and the cluster model · Essentials · 12 min

    Start a SparkSession, and understand the driver, executors, tasks and the local versus cluster modes.

  2. Creating DataFrames and schemas · Essentials · 12 min

    Build Spark DataFrames from Python data with explicit schemas, and inspect them with printSchema, show and columns.

  3. Data types, casting, nulls and ANSI mode · Essentials · 12 min

    Cast columns safely, handle nulls with na functions and coalesce, and understand Spark 4's ANSI mode errors.

Transformations and SQL

  1. Columns, expressions and filters · Essentials · 15 min

    Select, add and filter columns with col, lit, withColumn and the pyspark.sql.functions module.

  2. Transformations, actions and lazy evaluation · Essentials · 10 min

    Distinguish lazy transformations from actions that trigger jobs, and see how Spark builds a plan before running anything.

  3. Grouping and aggregation · Essentials · 12 min

    Summarize data with groupBy().agg() and functions like sum, avg, count and countDistinct, plus pivot.

  4. Spark SQL and temporary views · Practical · 10 min

    Register DataFrames as temp views, query them with spark.sql, and mix SQL with the DataFrame API.

  5. Joins in Spark · Practical · 15 min

    Join DataFrames with the right join type, avoid ambiguous columns, and use broadcast for small tables.

  6. Window functions · Practical · 15 min

    Compute rankings, running totals, lags and per-group shares with Window.partitionBy().orderBy() and frame specs.

  7. Arrays, structs, maps, JSON and VARIANT · Practical · 15 min

    Work with nested data: explode arrays, access struct fields, parse JSON with from_json, and use Spark 4's VARIANT type.

Data sources and streaming

  1. Reading and writing data sources · Essentials · 15 min

    Load CSV, JSON and Parquet with spark.read, supply schemas, and write results with modes and partitionBy.

  2. Structured Streaming · Advanced · 15 min

    Process unbounded data with readStream and writeStream: triggers, output modes, checkpoints and watermarks for late data.

UDFs and pandas

  1. Python UDFs and pandas UDFs · Practical · 15 min

    Write Python UDFs when built-ins fall short, understand their cost, and prefer Arrow-based and pandas UDFs.

  2. Arrow, toPandas and pandas function APIs · Advanced · 15 min

    Move data between Spark and pandas efficiently with Arrow, and run pandas code per group or partition with applyInPandas and mapInPandas.

Partitions and performance

  1. Partitions, shuffles and repartition · Practical · 15 min

    Understand how data is partitioned, what triggers a shuffle, and when to use repartition, coalesce and shuffle partition settings.

  2. Caching, persistence and checkpoints · Practical · 10 min

    Reuse expensive intermediate DataFrames with cache and persist, choose storage levels, and cut long lineage with checkpoints.

  3. Catalyst and reading explain() · Advanced · 15 min

    Follow a query through Catalyst's analyzed, optimized and physical plans, and read explain() output for pushdown, exchanges and join types.

  4. Adaptive execution and join strategies · Advanced · 15 min

    Know Spark's join strategies and how Adaptive Query Execution coalesces partitions, switches joins and splits skewed partitions at runtime.

Engine internals

  1. Spark UI, memory model and tuning · Expert · 20 min

    Diagnose jobs in the Spark UI (stages, tasks, skew, spill) and tune executor memory, cores and key SQL settings.

  2. How PySpark talks to the JVM · Expert · 15 min

    Understand Py4J on the driver, Python worker processes on executors, serialization paths, and how Spark Connect changes the architecture.

  3. Tungsten, codegen and columnar execution · Expert · 15 min

    See how Spark executes plans: Tungsten's binary row format, whole-stage code generation, vectorized Parquet reads and off-heap memory.