Arrays, structs, maps, JSON and VARIANT

Event logs and API data are nested JSON, and processing them natively is far faster than flattening in Python.

Spark columns can hold nested types:

array: F.array(...), F.split(s, ","), F.size, F.array_contains, F.element_at(arr, 1) (1-based), and higher-order functions such as F.transform(arr, lambda x: x * 2), F.filter, F.aggregate.
struct: named fields, accessed with F.col("geo.lat") or F.col("geo")["lat"]. select("geo.*") expands all fields into columns.
map: key/value pairs, F.col("attrs")["color"], F.map_keys, F.map_values.

F.explode("tags") turns each array element into its own row and drops rows with null or empty arrays; F.explode_outer keeps them with null. F.posexplode adds the position.

For JSON strings, F.from_json("payload", schema) parses into structs with a known schema, and F.to_json serializes back. F.get_json_object(col, "$.a.b") extracts single paths without a full schema.

Spark 4 adds the VARIANT type for semi-structured data whose schema varies: F.parse_json("payload") stores it in an efficient binary form, and F.variant_get(v, "$.price", "double") or F.try_variant_get extracts typed values. It is much faster than reparsing JSON strings and more flexible than a fixed struct schema.

Example

from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.master("local[2]").getOrCreate()

df = spark.createDataFrame(
    [("ana", "py,sql", '{"geo": {"lat": 59.9}, "price": 9.5}'),
     ("ben", "", '{"geo": {"lat": -12.0}}')],
    "user STRING, tags STRING, payload STRING",
)
tags = df.withColumn("tags", F.split("tags", ","))
print([r.n for r in tags.select(F.size(F.array_remove("tags", "")).alias("n")).collect()])
# [2, 0]
print([tuple(r) for r in tags.select("user", F.explode("tags").alias("tag"))
       .filter("tag != ''").collect()])
# [('ana', 'py'), ('ana', 'sql')]

parsed = df.select("user", F.from_json("payload", "geo STRUCT<lat: DOUBLE>, price DOUBLE").alias("p"))
print([tuple(r) for r in parsed.select("user", "p.geo.lat", "p.price").collect()])
# [('ana', 59.9, 9.5), ('ben', -12.0, None)]

v = df.select(F.parse_json("payload").alias("v"))
print([r.lat for r in v.select(F.variant_get("v", "$.geo.lat", "double").alias("lat")).collect()])
# [59.9, -12.0]

Remember

array, struct and map types with rich built-in functions
explode drops empty arrays; explode_outer keeps them
Use from_json with a schema, or Spark 4's VARIANT for flexible JSON

Watch out for

Parsing JSON with a Python UDF instead of from_json, get_json_object or parse_json.

Check yourself

F.split('', ',') on an empty string gives which array?

Official docs: spark.apache.org/docs/latest/api/python/user_guide/touroftypes.html

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks