Reading data into Dask DataFrames

Loading data the right way decides partition sizes and types, which drive everything that follows.

Dask DataFrames are usually created straight from files, so that each partition is loaded by a separate task:

dd.read_parquet("data/*.parquet", columns=[...], filters=[...]) is the best option. Parquet has types and statistics, and Dask reads only the requested columns and row groups.
dd.read_csv("logs/2026-*.csv", dtype={"zip": "string"}, blocksize="64MB") reads many files or splits big ones into blocks.
dd.from_pandas(pdf, npartitions=8) splits an in-memory DataFrame, mainly for tests and demos.
dd.from_delayed and dd.from_map build one partition per call of a function you write.

Paths can be local or remote (s3://, gs://, abfs://) with storage_options= for credentials.

Inspect cheaply: df.columns, df.dtypes and df.npartitions are known without reading data, because Dask keeps a small empty pandas frame called meta describing the result. df.head() reads only the first partition. len(df) and df.describe() scan everything, so they can take a while.

CSV pitfall: Dask infers dtypes from a sample at the start of the file. If later rows contain text in a "numeric" column, a task fails mid-computation. Pass explicit dtype= for such columns, or better, convert once to Parquet with df.to_parquet("out/").

Example

import dask.dataframe as dd
import pandas as pd

for i in range(3):   # three daily files
    pd.DataFrame({"day": [i] * 4, "zip": ["02134", "10001", "02134", "94105"],
                  "amount": [1.0, 2.0, 3.0, 4.0]}).to_csv(f"sales-{i}.csv", index=False)

df = dd.read_csv("sales-*.csv", dtype={"zip": "string"})
print(df.npartitions)                    # 3  (one per small file)
print(df.dtypes.astype(str).to_dict())
# {'day': 'int64', 'zip': 'string', 'amount': 'float64'}
print(df.head(2)["zip"].tolist())        # ['02134', '10001']

df.to_parquet("sales_pq/")               # convert once, read fast later
pq = dd.read_parquet("sales_pq/", columns=["zip", "amount"])
print(pq.amount.sum().compute())         # 30.0

Remember

Prefer read_parquet with columns= and filters=
meta gives columns and dtypes without reading data; head() reads one partition
Give CSV columns explicit dtypes, or convert to Parquet once

Watch out for

Relying on dtype inference for messy CSVs and getting failures halfway through a job.

Check yourself

Why does df.head() return quickly on a 500 GB Dask DataFrame while len(df) does not?

Official docs: docs.dask.org/en/stable/dataframe-create.html

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks