TopolearnParallel API calls give big speedups for I/O-bound AWS work, but sharing the wrong objects across threads causes subtle bugs.
boto3 calls are blocking network I/O, so threads parallelize them well despite the GIL. The thread-safety rules from the boto3 docs:
- Clients are generally thread-safe: create one and share it among threads. (Mutating a client's event handlers or config while in use isn't safe, which ordinary code doesn't do.)
- Sessions are not thread-safe: don't create clients from a shared session concurrently. Create clients up front in the main thread, or a session per thread.
- Resources are not thread-safe: create one per thread.
Each client has a connection pool of max_pool_connections (default 10). With 32 threads sharing a client, 22 wait for connections and urllib3 logs "Connection pool is full" warnings. Set Config(max_pool_connections=32) to match your thread count.
concurrent.futures.ThreadPoolExecutor is the simple pattern: submit one task per key, region or account, collect results with as_completed, and handle exceptions per task.
Mind API quotas: parallel calls hit throttling sooner. Use the standard or adaptive retry mode and cap concurrency per service.
For multiprocessing, create clients in each child process after the fork; clients hold sockets and must not be shared across processes. For asyncio applications, the third-party aiobotocore and aioboto3 packages provide async clients.
Example
from concurrent.futures import ThreadPoolExecutor, as_completed
import boto3
from botocore.config import Config
s3 = boto3.client("s3", region_name="us-east-1",
config=Config(max_pool_connections=16)) # match the pool to threads
s3.create_bucket(Bucket="demo-threads-2026")
keys = [f"in/{i:03}.txt" for i in range(40)]
for k in keys:
s3.put_object(Bucket="demo-threads-2026", Key=k, Body=k.encode())
def size_of(key): # shares the thread-safe client
return key, s3.head_object(Bucket="demo-threads-2026", Key=key)["ContentLength"]
sizes = {}
with ThreadPoolExecutor(max_workers=16) as pool:
futures = [pool.submit(size_of, k) for k in keys]
for fut in as_completed(futures):
key, size = fut.result() # re-raises errors per task
sizes[key] = size
print(len(sizes), sum(sizes.values())) # 40 400Remember
- Share clients across threads; not sessions or resources
- Match max_pool_connections to the number of threads
- ThreadPoolExecutor parallelizes I/O-bound AWS calls well
- Create clients per process after forking; use aioboto3 for asyncio
Watch out for
- Calling boto3.client() concurrently from many threads using the shared default session.