TopolearnConcurrency is aiohttp's reason to exist: hundreds of requests in the time of a few.
One request at a time gains nothing from async. The power comes from running many requests concurrently on one session: while one waits on the network, others make progress.
Write a coroutine that fetches one URL, then run many of them together:
await asyncio.gather(*(fetch(session, u) for u in urls))returns results in input order. Withreturn_exceptions=True, failures come back as exception objects instead of cancelling the whole batch.async with asyncio.TaskGroup() as tg:(Python 3.11+) creates tasks withtg.create_task(...)and cancels the rest if one fails, which is cleaner for all-or-nothing work.asyncio.as_completed(...)processes results as they finish.
Always bound concurrency. Ten thousand simultaneous requests overload the server, hit rate limits and exhaust file descriptors. Two tools:
- an
asyncio.Semaphore(n)around each request, - the connector limits (
TCPConnector(limit=..., limit_per_host=...)), covered later. The defaultlimitis 100 connections in total, so extra requests already queue for a free connection.
The semaphore is still useful because it also bounds the work around the request (parsing, retries) and keeps tasks from waiting on the pool with timeouts running.
All tasks should share one session; pass it as an argument. Never use blocking calls such as requests.get or time.sleep inside these coroutines: they freeze the whole event loop.
Example
import asyncio
import time
import aiohttp
async def fetch(session, sem, n):
async with sem:
async with session.get("https://httpbin.org/delay/1", params={"n": n}) as resp:
return (await resp.json())["args"]["n"]
async def main():
sem = asyncio.Semaphore(10)
async with aiohttp.ClientSession() as session:
t = time.perf_counter()
results = await asyncio.gather(*(fetch(session, sem, i) for i in range(20)))
print(results[:5], len(results)) # ['0', '1', '2', '3', '4'] 20
print(f"{time.perf_counter() - t:.0f}s") # 2s (20 x 1s requests, 10 at a time)
asyncio.run(main())Remember
- gather or TaskGroup runs many request coroutines concurrently
- Share one session; bound concurrency with a Semaphore
- Blocking calls inside coroutines stall everything
Watch out for
- Calling time.sleep() or requests.get() inside a coroutine.