Fork safety, initializers and process lifecycles

Mixing threads, forks and long-lived pools causes the hardest-to-debug hangs in production Python.

fork() copies only the calling thread into the child. Every lock held by another thread at that instant is copied locked, with no owner to release it. If the child then touches that lock (logging, the import system, an HTTP client, malloc in a C library), it hangs forever.

Python 3.12 emits a DeprecationWarning when os.fork() (or the fork start method) runs in a process with more than one thread, and 3.14 switched Linux's default start method to forkserver. Guidelines:

Prefer spawn or forkserver contexts in any program that may have threads, including threads started by libraries.
If you must fork, do it early, before starting threads or opening network clients.
os.register_at_fork(before=..., after_in_parent=..., after_in_child=...) lets a library reset state (re-create locks, reconnect) after a fork.

Pool lifecycle tools:

initializer= / initargs= (in Pool and both executors) run once per worker: open connections, load models, set signal handlers.
max_tasks_per_child / maxtasksperchild recycles workers to bound memory leaks.
Always shut pools down (context manager); on Ctrl+C, call shutdown(cancel_futures=True).

Python 3.14 also adds InterpreterPoolExecutor, which runs tasks in subinterpreters: separate GILs inside one process.

Example

import multiprocessing as mp, os, threading, warnings
from concurrent.futures import ProcessPoolExecutor
_state = {}

def init_worker(prefix):
    _state["conn"] = f"{prefix}-conn-{os.getpid() % 1000}"   # e.g. open a DB connection
def task(n):
    return f"{_state['conn'][:9]} handled {n}"

os.register_at_fork(after_in_child=_state.clear)   # reset inherited state
if __name__ == "__main__":
    busy = threading.Event()
    threading.Thread(target=busy.wait).start()        # a background thread is alive...
    with warnings.catch_warnings(record=True) as w:
        warnings.simplefilter("always")
        pid = os.fork()                               # ...so forking is unsafe
        if pid == 0: os._exit(0)                      # child exits at once
        os.waitpid(pid, 0)
    busy.set()
    print([type(x.message).__name__ for x in w][:1])  # ['DeprecationWarning']

    ctx = mp.get_context("forkserver")
    with ProcessPoolExecutor(2, mp_context=ctx, initializer=init_worker,
                             initargs=("db",), max_tasks_per_child=10) as pool:
        print(list(pool.map(task, range(3))))         # ['db-conn-4 handled 0', ...] (pids vary)

Remember

fork copies only the calling thread; locks held elsewhere stay locked in the child
Prefer spawn or forkserver; Python 3.12+ warns on fork with threads
Use initializer and max_tasks_per_child to manage worker state and leaks

Watch out for

Creating a process pool with the fork method in a program whose libraries already started background threads.

Check yourself

Why can a child process hang right after fork() in a multithreaded parent?

Official docs: docs.python.org/3/library/os.html#os.register_at_fork

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks