TopolearnUsing all your cores or overlapping slow I/O requires knowing which concurrency tool the GIL allows.
In the standard CPython build, the GIL (global interpreter lock) lets only one thread execute Python bytecode at a time. That leads to the core rule:
- I/O-bound work (network, disk, database waits): threads work well, because the GIL is released while a thread waits
- CPU-bound pure-Python work: use processes, each with its own interpreter and GIL, to use multiple cores
concurrent.futures gives both the same API. ThreadPoolExecutor(max_workers=n) and ProcessPoolExecutor() provide submit(fn, *args) which returns a Future, and map(fn, iterable) which yields results in input order. as_completed(futures) yields futures as they finish. Use the executor as a context manager so it waits for work and shuts down cleanly. Exceptions raised in a worker re-raise when you call future.result().
Process pools have extra requirements. Arguments and results are pickled to cross process boundaries, so they must be picklable (no lambdas, open files or locks), and large data is costly to send. On Windows and macOS new processes are started with spawn, which re-imports your main module — the pool code must sit under if __name__ == "__main__":.
Threads share memory, which is convenient and dangerous. The GIL does not make your code thread-safe: counter += 1 is several bytecode steps and can interleave. Protect shared mutable state with threading.Lock (used with with), or avoid sharing by passing messages through queue.Queue.
For lower-level control there are threading.Thread and multiprocessing.Process; for many concurrent network connections, asyncio is often lighter than threads.
Example
import math
import time
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor, as_completed
def fake_download(n):
time.sleep(0.2) # I/O wait releases the GIL
return n * 10
def count_primes(limit): # CPU-bound pure Python
return sum(1 for n in range(2, limit)
if all(n % d for d in range(2, math.isqrt(n) + 1)))
if __name__ == "__main__": # required for process pools under spawn
start = time.perf_counter()
with ThreadPoolExecutor(max_workers=8) as pool:
futures = [pool.submit(fake_download, i) for i in range(8)]
results = sorted(f.result() for f in as_completed(futures))
print(results, f"{time.perf_counter() - start:.1f}s") # ~0.2s, not 1.6s
with ProcessPoolExecutor() as pool: # one interpreter (and GIL) per worker
print(list(pool.map(count_primes, [20_000, 30_000, 40_000])))
# [2262, 3245, 4203]Remember
- Threads for I/O-bound work, processes for CPU-bound Python code
- Executors give
submit,mapand Futures for both - The GIL doesn't make code thread-safe; lock shared state
Watch out for
- Creating a ProcessPoolExecutor without the
__main__guard, which fails or recurses under spawn. - Passing lambdas or unpicklable objects to a process pool.