Walking a directory tree

Searching, indexing, cleaning or summing over every file below a folder is a job for os.walk.

os.walk(top) is a generator that yields one tuple per directory in the tree:

for dirpath, dirnames, filenames in os.walk(top):
    ...

dirpath is the directory's path (starting with top), dirnames the names of its subdirectories and filenames the names of its other entries. Join dirpath with a name to get a usable path.

By default the walk is top-down: a directory is yielded before its children. That allows pruning: modify dirnames in place and os.walk will only descend into what remains:

dirnames[:] = [d for d in dirnames if not d.startswith(".")]

Rebinding (dirnames = [...]) does nothing, because os.walk keeps its own reference to the list.

With topdown=False the walk is bottom-up: children come before parents. That is the order you need when deleting or renaming directories as you go.

Other details:

Symbolic links to directories appear in dirnames but are not followed unless followlinks=True (which risks loops).
Errors from listing a directory are ignored by default; pass onerror=print (or a handler) to see them.
Order within a directory is arbitrary; sort dirnames in place for a deterministic walk.

Python 3.12 added pathlib.Path.walk(), which yields the same triples with Path objects.

Example

import os

for d in ["proj/src/pkg", "proj/.git/objects", "proj/docs"]:
    os.makedirs(d)
for f in ["proj/src/pkg/a.py", "proj/src/b.py", "proj/docs/x.md", "proj/.git/HEAD"]:
    open(f, "w").close()

py_files = []
for dirpath, dirnames, filenames in os.walk("proj"):
    dirnames[:] = sorted(d for d in dirnames if not d.startswith("."))
    for name in filenames:
        if name.endswith(".py"):
            py_files.append(os.path.join(dirpath, name))
print(sorted(py_files))  # ['proj/src/b.py', 'proj/src/pkg/a.py']

# bottom-up: children first, so empty dirs can be removed safely
order = [p for p, _, _ in os.walk("proj/src", topdown=False)]
print(order)             # ['proj/src/pkg', 'proj/src']

Remember

os.walk yields (dirpath, dirnames, filenames) for every directory
Prune by editing dirnames in place (dirnames[:] = ...) in top-down mode
Use topdown=False when deleting or renaming directories during the walk

Watch out for

Reassigning dirnames instead of mutating it, so the pruning silently has no effect.

Check yourself

Inside a top-down os.walk loop you write dirnames = [d for d in dirnames if d != 'node_modules']. What happens?

Official docs: docs.python.org/3/library/os.html#os.walk

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks