TopolearnIt is the simplest way to make CPU-heavy Python code scale across cores on a standard build.
ProcessPoolExecutor has the same interface as the thread pool (map, submit, futures), but each worker is a separate process with its own interpreter and GIL. CPU-bound pure-Python work then runs truly in parallel.
The price is that data must cross process boundaries:
- Arguments and return values are pickled, sent through pipes and unpickled. Functions must be importable by name: module-level functions work; lambdas and nested functions don't.
- Large inputs are copied for every task; send compact inputs (a filename, a range) rather than huge objects. Use
chunksize=inmapto batch many small tasks and cut overhead. - Workers don't share memory with the parent; mutating a global in a worker doesn't change the parent's copy.
The `if __name__ == "__main__":` guard is mandatory with the spawn start method (the default on Windows and macOS), because each worker imports your main module; without the guard, it would try to start pools recursively. Use it everywhere for portability.
max_workers defaults to the number of CPUs (os.process_cpu_count() in 3.13, which respects CPU affinity). max_tasks_per_child (3.11) recycles workers to contain memory leaks. Startup costs tens of milliseconds, so it pays off for tasks that take noticeably longer.
Example
import time
from concurrent.futures import ProcessPoolExecutor
def count_primes(limit: int) -> int:
return sum(all(n % d for d in range(2, int(n ** 0.5) + 1))
for n in range(2, limit))
def run(executor_cls=None):
jobs = [60_000] * 8
t = time.perf_counter()
if executor_cls is None:
total = sum(map(count_primes, jobs))
else:
with executor_cls() as pool:
total = sum(pool.map(count_primes, jobs))
return total, round(time.perf_counter() - t, 2)
if __name__ == "__main__": # required for spawn-based workers
print("sequential", run()) # e.g. (48464, 1.6)
print("processes ", run(ProcessPoolExecutor)) # e.g. (48464, 0.45) on 4+ coresRemember
- Processes run CPU-bound Python in parallel, each with its own GIL
- Arguments and results are pickled: use module-level functions and compact data
- Always guard pool creation with if __name__ == '__main__':
Watch out for
- Submitting a lambda or local function to a process pool; it can't be pickled.