TopolearnHigh-throughput clients hit pool limits and threading pitfalls that are invisible at low volume.
Each HTTPAdapter owns a urllib3 PoolManager, which keeps one connection pool per host (scheme, host and port). Two parameters control it:
pool_connections: how many per-host pools to cache (default 10). Talking to more hosts than this evicts pools and loses keep-alive.pool_maxsize: how many connections each pool keeps for reuse (default 10). With more concurrent threads than this, extra connections are opened and then discarded after use, with the warning "Connection pool is full, discarding connection". Throughput drops because new TLS handshakes happen constantly.pool_block=Truemakes threads wait for a free connection instead of opening extra ones, capping concurrency per host.
So for a thread pool of 32 workers hitting one API, mount HTTPAdapter(pool_maxsize=32).
Thread safety: Requests does not formally guarantee that Session is thread-safe. In practice the connection pool is thread-safe and sharing one session for simple GETs works widely, but mutable session state is not protected: cookies being updated by concurrent responses, or code changing s.headers while other threads send, can interleave badly. Safer patterns are one session per thread (threading.local), or a shared session whose configuration is frozen after setup and that does not rely on cookies.
Connections are returned to the pool only when the response body has been fully read or closed. Forgetting to close streamed responses is the classic cause of pool exhaustion.
For thousands of concurrent requests, threads plus Requests become expensive; an async client is the better tool.
Example
import threading
from concurrent.futures import ThreadPoolExecutor
import requests
from requests.adapters import HTTPAdapter
local = threading.local()
def get_session():
if not hasattr(local, "s"): # one session per thread
s = requests.Session()
s.mount("https://", HTTPAdapter(pool_connections=4, pool_maxsize=8))
s.mount("http://", HTTPAdapter(pool_connections=4, pool_maxsize=8))
local.s = s
return local.s
def fetch(i):
r = get_session().get("https://httpbin.org/get", params={"i": i}, timeout=10)
return r.json()["args"]["i"]
with ThreadPoolExecutor(max_workers=8) as pool:
results = list(pool.map(fetch, range(20)))
print(results[:5], len(results)) # ['0', '1', '2', '3', '4'] 20Remember
- An HTTPAdapter's pool keeps pool_maxsize connections per host
- Size pool_maxsize to your thread count or connections are discarded
- Prefer one session per thread or treat a shared session as read-only
Watch out for
- Seeing 'Connection pool is full, discarding connection' and ignoring it.