TopolearnScanning millions of files with naive pathlib code can be several times slower than necessary.
Path objects are convenient, but each one is a Python object that parses and stores its parts, and many operations create new objects. In typical scripts this cost is irrelevant compared to disk I/O. It starts to matter when you touch hundreds of thousands of paths in a loop.
Where the time goes:
- Repeated stat calls.
p.is_file(),p.is_dir()andp.stat()each issue a system call. Calling all three per file triples the work.os.scandirreturns DirEntry objects whose type is usually known from the directory listing itself, with no extra call;os.walkandPath.walkbuild on it. - Object creation.
root / namefor every entry allocates a Path and parses it; string joins are cheaper. - rglob over huge trees you'd prune anyway. Prefer
Path.walkand prune.
Recent releases (3.12 and 3.13) reworked glob internally for speed, so upgrading Python can itself help glob-heavy code.
Practical approach: write clear pathlib code first; profile; then, in the hot loop only, drop to os.scandir and strings, converting results to Path at the end. Caching stat results when you need several fields from the same file also helps.
Example
import os
import timeit
from pathlib import Path
d = Path("many")
d.mkdir()
for i in range(2000):
(d / f"f{i}.txt").touch()
def with_pathlib():
return sum(p.stat().st_size for p in d.iterdir() if p.is_file())
def with_scandir():
with os.scandir(d) as it:
return sum(e.stat().st_size for e in it if e.is_file())
print(with_pathlib() == with_scandir()) # True
t1 = timeit.timeit(with_pathlib, number=5)
t2 = timeit.timeit(with_scandir, number=5)
print(t2 < t1) # TrueRemember
- File system calls dominate: avoid redundant stat/is_file calls per entry
os.scandirreuses type info from the listing; walk APIs build on it- Write pathlib first, profile, and drop to os/scandir only in hot loops
Watch out for
- Calling
exists(),is_file()andstat()on the same path in sequence inside a large loop.