TopolearnProduction data lives in object storage, and PyArrow's filesystem layer is what many Python tools use to reach it.
pyarrow.fs gives one interface over many storage systems: LocalFileSystem, S3FileSystem, GcsFileSystem, AzureFileSystem, HadoopFileSystem, plus SubTreeFileSystem to root paths under a prefix. fsspec filesystems can be wrapped with PyFileSystem(FSSpecHandler(fs)).
fs.FileSystem.from_uri("s3://bucket/path") returns a filesystem and the path inside it. Credentials come from the usual environment and config files (AWS profile, GOOGLE_APPLICATION_CREDENTIALS and so on) or from constructor arguments such as region= and access_key=.
Main operations:
filesystem.get_file_info(path)orget_file_info(fs.FileSelector(dir, recursive=True))lists files with sizes and types.open_input_file(random access, needed by Parquet),open_input_stream,open_output_stream.create_dir,delete_dir_contents,move,copy_file.
Readers accept a filesystem= argument, and URIs work directly: pq.read_table("s3://bucket/sales/", filesystem=s3) or ds.dataset("gs://bucket/lake/", format="parquet").
Object stores are not real filesystems. Listing a huge prefix is slow, renames are copies, and each request has high latency. Reading fewer, larger files (hundreds of MB) and using column and row-group pruning matters even more remotely than locally.
Example
import pyarrow as pa
import pyarrow.parquet as pq
from pyarrow import fs
local, path = fs.FileSystem.from_uri("file:///tmp/arrow_demo")
print(type(local).__name__, path) # LocalFileSystem /tmp/arrow_demo
local.create_dir(path)
root = fs.SubTreeFileSystem(path, local) # every path is now relative to the demo dir
pq.write_table(pa.table({"x": [1, 2, 3]}), "x.parquet", filesystem=root)
for info in root.get_file_info(fs.FileSelector("", recursive=True)):
print(info.path, info.type == fs.FileType.File, info.size > 0) # x.parquet True True
print(pq.read_table("x.parquet", filesystem=root)["x"].to_pylist()) # [1, 2, 3]
# Same code against S3 (needs credentials):
# s3 = fs.S3FileSystem(region="eu-north-1")
# pq.read_table("my-bucket/lake/x.parquet", filesystem=s3)Remember
pyarrow.fs= one API over local, S3, GCS, Azure and HDFSFileSystem.from_uriturns a URI into (filesystem, path)- Remote storage rewards few large files and aggressive pruning
Watch out for
- Treating object-store renames and listings as cheap like a local disk.