Fast listings with os.scandir

os · Practical · 9 min

On large directories scandir is several times faster than listdir plus a stat per file.

os.scandir(path) returns an iterator of DirEntry objects instead of plain names. Each entry carries:

  • entry.name and entry.path (the name joined with the directory you passed)
  • entry.is_file(), entry.is_dir(), entry.is_symlink()
  • entry.stat() for size, timestamps and permissions

The speed comes from the operating system: when it lists a directory it usually also reports each entry's type, so is_file() and is_dir() often need no extra system call. On Windows even stat() is free. With listdir you pay one stat per name. os.walk uses scandir internally for exactly this reason.

Use it as a context manager so the directory handle is closed even if you stop early:

with os.scandir(folder) as it:
    for entry in it:
        ...

Entries are a snapshot: if a file is deleted after the listing, entry.stat() raises FileNotFoundError. The type methods follow symlinks by default; pass follow_symlinks=False to look at the link itself.

A typical use is a recursive generator that yields file entries, giving you sizes without a separate os.stat call per file.

Example

import os

os.makedirs("logs/archive")
for name, size in [("app.log", 120), ("err.log", 30)]:
    with open(os.path.join("logs", name), "w") as f:
        f.write("x" * size)

def files_under(path):
    with os.scandir(path) as it:
        for entry in it:
            if entry.is_dir(follow_symlinks=False):
                yield from files_under(entry.path)
            elif entry.is_file():
                yield entry

total = sum(e.stat().st_size for e in files_under("logs"))
print(total)                                     # 150
print(sorted(e.name for e in files_under("logs")))  # ['app.log', 'err.log']

Remember

  • scandir yields DirEntry objects with name, path, type checks and stat()
  • Type checks usually avoid extra system calls, so it is much faster than listdir + stat
  • Use it in a with block to release the directory handle

Check yourself

  1. Why is os.scandir usually faster than os.listdir followed by os.path.isdir on each name?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: docs.python.org/3/library/os.html#os.scandir