TopolearnExecutors are the simplest correct way to parallelise a loop of independent I/O calls.
concurrent.futures.ThreadPoolExecutor manages a fixed pool of worker threads and a queue of tasks. Use it as a context manager so the pool is shut down (and waits for pending work) at the end:
with ThreadPoolExecutor(max_workers=8) as pool:
pages = list(pool.map(fetch, urls))pool.map(fn, iterable) works like the built-in map, but calls run concurrently:
- results come back in input order,
- it returns a lazy iterator; wrap in
list()to wait for everything, - if a call raised, the exception is re-raised when you reach that result while iterating,
timeout=limits the total wait.
max_workers defaults to min(32, os.process_cpu_count() + 4) in 3.13. For I/O-bound tasks, a few dozen threads is usually fine; the limit is often the remote service, not your machine, so be polite with rate limits.
thread_name_prefix= makes thread names readable in logs and debuggers, and initializer= runs a setup function in each worker thread (for example to create a per-thread HTTP session).
For more control (handling each result as it completes, mixing different functions), use submit() and futures, the next lesson.
Example
import time
from concurrent.futures import ThreadPoolExecutor
def fetch(url: str) -> int:
time.sleep(0.3) # network latency
if "bad" in url:
raise ValueError(f"cannot fetch {url}")
return len(url)
urls = [f"https://example.com/page/{i}" for i in range(10)]
t = time.perf_counter()
with ThreadPoolExecutor(max_workers=10, thread_name_prefix="fetch") as pool:
sizes = list(pool.map(fetch, urls))
print(sizes[:3], f"{time.perf_counter() - t:.1f}s") # [26, 26, 26] 0.3s
with ThreadPoolExecutor() as pool:
try:
list(pool.map(fetch, ["ok", "bad-url"]))
except ValueError as e:
print("re-raised:", e) # re-raised: cannot fetch bad-urlRemember
- Use ThreadPoolExecutor as a context manager
- pool.map returns results in input order and re-raises errors
- Default max_workers is min(32, CPU count + 4)
Watch out for
- Creating a new executor inside a loop instead of reusing one pool for all tasks.