Topolearn

Learn PyArrow

Python bindings for Apache Arrow: columnar memory, Parquet and fast data exchange

15 lessons · 3 hours · 4 levels · covers PyArrow 25+

PyArrow is the Python library for Apache Arrow, the standard in-memory columnar format shared by pandas, Polars, DuckDB, Spark and many more. It gives you typed arrays and tables, fast Parquet, CSV and IPC readers and writers, compute functions, partitioned datasets and cloud filesystems. Learn it to move data between tools without copies and to read and write analytics files efficiently.

Sign in free to track progressOfficial docs ↗

Before you start

PyArrow builds on these; the tree links to the exact lessons you need.

PythonEssentials levelNumPyEssentials level

How deep do you want to go?

1. EssentialsRead and write simple code, do the everyday basics5 lessons · 58 min in total
2. PracticalUse it productively at work, with the usual idioms9 lessons · 2 hours in total
3. AdvancedThe less common features, performance and trade-offs12 lessons · 3 hours in total
4. ExpertInternals and the edge cases experts know15 lessons · 3 hours in total

The PyArrow tree

PyArrowArrays and data typesArrays and datatypesTables, schemas and record batchesTables, schemas andrecord batchesReading and writing ParquetReading and writingParquetFast CSV and JSON readingFast CSV and JSONreadingConverting to and from pandas and NumPyConverting to andfrom pandas and N…Compute functionsCompute functionsFiltering, sorting, grouping and joining tablesFiltering, sorting,grouping and joinin…Datasets and partitioned dataDatasets andpartitioned dataNested, dictionary and temporal typesNested, dictionaryand temporal typesListsListsModules and importsModules and importsDictionariesDictionariesReading and writing filesReading and writingfilesData types (dtypes)Data types (dtypes)Vectorized arithmeticVectorized arithmetic
Essentials and Practical shown (9 of 15 lessons); the full list is below. Click a lesson to read it.

All 15 lessons, in order

Arrays, types and tables

  1. Arrays and data types · Essentials · 12 min

    Create typed Arrow arrays with pa.array, read their type and null count, and meet ChunkedArray.

  2. Tables, schemas and record batches · Essentials · 12 min

    Build pa.Table objects from dicts or rows, inspect and enforce a schema, and select, rename and add columns.

  3. Converting to and from pandas and NumPy · Essentials · 12 min

    Move data between Arrow, pandas and NumPy with from_pandas, to_pandas and to_numpy, and know when it is zero-copy.

  4. Nested, dictionary and temporal types · Practical · 12 min

    Use list, struct and map types, dictionary encoding for repeated values, and timestamps with units and time zones.

Files and formats

  1. Reading and writing Parquet · Essentials · 12 min

    Write tables with pq.write_table and read them back with pq.read_table, choosing columns, filters and compression.

  2. Fast CSV and JSON reading · Essentials · 10 min

    Read CSV and line-delimited JSON with pyarrow.csv and pyarrow.json, setting column types, null markers and parse options.

  3. Parquet layout: row groups and statistics · Advanced · 15 min

    Understand Parquet's row groups, column chunks, pages, statistics and encodings, and tune writes for fast reads.

  4. IPC streams, Feather and memory mapping · Advanced · 12 min

    Serialize Arrow data with the IPC stream and file formats, use Feather v2, and read files zero-copy through memory maps.

  5. Filesystems: local, S3, GCS and more · Advanced · 12 min

    Read and write data on S3, GCS, Azure, HDFS or local disk through pyarrow.fs, and pass filesystems to datasets and Parquet.

Compute and datasets

  1. Compute functions · Practical · 15 min

    Use pyarrow.compute for vectorized arithmetic, comparisons, string functions, aggregations and filtering.

  2. Filtering, sorting, grouping and joining tables · Practical · 15 min

    Run table-level operations in PyArrow: filter with expressions, sort_by, group_by().aggregate() and join.

  3. Datasets and partitioned data · Practical · 15 min

    Treat many files as one dataset with pyarrow.dataset, use hive partitioning, filter and project lazily, and write partitioned output.

Memory and interop

  1. The Arrow columnar format · Expert · 15 min

    Look inside arrays: validity bitmaps, value buffers, offsets for strings and lists, alignment and slicing offsets.

  2. Memory pools, buffers and zero-copy · Expert · 12 min

    Track Arrow allocations with memory pools, choose an allocator, and reason about when operations copy or share buffers.

  3. C Data Interface and PyCapsules · Expert · 12 min

    Exchange Arrow data between libraries and languages with the C Data and Stream interfaces and the __arrow_c_stream__ PyCapsule protocol.