TopolearnSaving intermediate results in the right format is faster, smaller and safer than ad hoc text dumps or pickles.
NumPy's own binary format is the fastest and most faithful way to store arrays:
np.save("x.npy", a)writes one array with its dtype and shape;np.load("x.npy")reads it back exactly.np.savez("data.npz", x=a, y=b)bundles several named arrays into one file;np.savez_compressedzips them.np.loadon an.npzreturns a lazy, dict-likeNpzFile; use it in awithblock so the file is closed.
For text data, np.loadtxt(path, delimiter=",", skiprows=1) reads clean numeric tables quickly, and np.genfromtxt tolerates missing values (filling them with nan). np.savetxt writes text, with fmt= to control number formatting. Text is portable and human-readable, but it's large, slow and can lose float precision if you format carelessly. For mixed-type tables with headers, pandas' read_csv is usually the better tool.
For arrays bigger than RAM, np.load("x.npy", mmap_mode="r") returns a memory-mapped array: data is read from disk on demand as you index it, so you can work on slices of a huge file. np.memmap does the same for raw binary files.
Security: object arrays are saved with pickle, and unpickling can execute arbitrary code. np.load therefore uses allow_pickle=False by default and refuses such files. Only enable it for files you trust.
Avoid a.tofile()/np.fromfile() for storage: they write raw bytes with no dtype, shape or byte-order information, so the reader must know all three.
Example
import numpy as np
from pathlib import Path
import tempfile
tmp = Path(tempfile.mkdtemp())
a = np.arange(6, dtype=np.float32).reshape(2, 3)
np.save(tmp / "a.npy", a)
b = np.load(tmp / "a.npy")
print(b.dtype, b.shape) # float32 (2, 3)
np.savez_compressed(tmp / "bundle.npz", features=a, labels=np.array([0, 1]))
with np.load(tmp / "bundle.npz") as f:
print(f.files) # ['features', 'labels']
print(f["labels"]) # [0 1]
np.savetxt(tmp / "a.csv", a, delimiter=",", fmt="%.3f", header="x,y,z")
t = np.loadtxt(tmp / "a.csv", delimiter=",") # '#' header skipped
print(t[1]) # [3. 4. 5.]
m = np.load(tmp / "a.npy", mmap_mode="r") # read lazily from disk
print(m[1, 2]) # 5.0Remember
.npystores one array exactly;.npzstores several, optionally compressedloadtxt/genfromtxtandsavetxthandle text; pandas is better for messy tablesmmap_mode='r'reads huge arrays lazily from diskallow_picklestays False unless you trust the file
Watch out for
- Storing arrays with
tofileand losing track of dtype and shape needed to read them back.