TopolearnKnowing a DataFrame's schema and how to define one prevents most type surprises later.
A Spark DataFrame is a distributed table: rows split into partitions across executors, with a schema of named, typed columns. It is immutable; every transformation returns a new DataFrame.
Create one from local data with spark.createDataFrame(rows, schema). Rows can be tuples, dicts or Row objects, or a pandas DataFrame. The schema can be:
- a DDL string:
"name STRING, age INT, joined DATE"(concise and readable) - a
StructType([StructField("name", StringType(), True), ...])(programmatic) - omitted, letting Spark infer it from the data (fine for experiments, risky in production)
Inspect with:
df.printSchema(): a tree of columns, types and nullabilitydf.schema,df.columns,df.dtypesdf.show(n=20, truncate=True): prints rows as a table (an action)df.take(5)anddf.collect(): returnRowobjects to the driver;row.nameorrow["name"]reads fields
In real jobs, data comes from files and tables (next lessons) rather than Python lists. createDataFrame sends local data from the driver to executors, so it's for small inputs, tests and lookup tables only.
Example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.master("local[2]").getOrCreate()
from datetime import date
df = spark.createDataFrame(
[("Ana", 34, date(2024, 5, 1)), ("Ben", None, date(2025, 1, 9))],
schema="name STRING, age INT, joined DATE",
)
df.printSchema()
# root
# |-- name: string (nullable = true)
# |-- age: integer (nullable = true)
# |-- joined: date (nullable = true)
print(df.dtypes) # [('name', 'string'), ('age', 'int'), ('joined', 'date')]
row = df.first()
print(row.name, row["age"]) # Ana 34
print(df.count(), df.columns) # 2 ['name', 'age', 'joined']Remember
- A DataFrame is an immutable, partitioned table with a schema
- Define schemas explicitly, e.g. with a DDL string
collect()brings rows to the driver; use it only on small results
Watch out for
- Calling
collect()on a large DataFrame and crashing the driver with out-of-memory.