TopolearnEvery Spark job starts by reading and ends by writing, and the options you choose decide correctness and speed.
spark.read returns a DataFrameReader: spark.read.parquet(path), .csv(path, header=True, schema=...), .json(path), .orc(path), or the general spark.read.format("csv").option("sep", ";").load(path). Paths can be files, directories or globs, on local disk, HDFS or object storage (s3a://, gs://, abfss://).
For CSV and JSON, pass a schema. Schema inference (inferSchema=True) reads the data an extra time and can guess wrong. Useful CSV options: header, sep, nullValue, dateFormat, and mode (PERMISSIVE, DROPMALFORMED, FAILFAST).
Parquet (and table formats built on it, such as Delta Lake and Apache Iceberg) is the default choice: typed, columnar and compressed, with column pruning and filter pushdown.
Writing uses df.write:
.mode("overwrite" | "append" | "error" | "ignore"); the defaulterrorfails if the path exists..partitionBy("date")writesdate=2026-03-01/folders, so readers filtering on date skip others..parquet(path),.csv(path),.saveAsTable("db.table")for catalog tables.
Spark writes one file per partition of the DataFrame, into a directory, plus a _SUCCESS marker. Too many tiny files hurt later reads; coalesce(n) or repartition(...) before writing controls the count.
Example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.master("local[2]").getOrCreate()
import glob
with open("people.csv", "w") as f:
f.write("id;name;joined\n1;Ana;2026-01-05\n2;Ben;NA\nx;Bad;2026-02-01\n")
people = (spark.read
.schema("id INT, name STRING, joined DATE")
.options(header=True, sep=";", nullValue="NA", mode="DROPMALFORMED")
.csv("people.csv"))
print([tuple(r) for r in people.collect()])
# [(1, 'Ana', datetime.date(2026, 1, 5)), (2, 'Ben', None)]
(people.withColumn("year", F.year("joined"))
.write.mode("overwrite").partitionBy("year").parquet("people_pq"))
print(sorted(p.split("/")[-1] for p in glob.glob("people_pq/year=*")))
# ['year=2026', 'year=__HIVE_DEFAULT_PARTITION__']
print(spark.read.parquet("people_pq").filter("year = 2026").count()) # 1Remember
- Always give CSV and JSON readers an explicit schema
- Prefer Parquet (or Delta/Iceberg) for anything Spark reads again
partitionBycreates folders that later reads can skip
Watch out for
- Writing with the default mode into an existing path and getting 'path already exists' errors in a rerun.