Connectors, pooling and DNS

At scale, the connector decides throughput, fairness across hosts and whether you get banned for hammering a server.

Every ClientSession uses a connector that manages the connection pool. The default is aiohttp.TCPConnector(); pass your own with ClientSession(connector=...).

Key TCPConnector options:

limit (default 100): maximum simultaneous connections in total. limit=0 means unlimited, which is rarely wise.
limit_per_host (default 0, unlimited): caps connections to one host, port and scheme. Set it when crawling many hosts so one host is not overwhelmed and others are not starved.
ttl_dns_cache (default 10 seconds) and use_dns_cache: DNS results are cached per connector. aiodns (installed with aiohttp[speedups]) makes resolution truly async.
keepalive_timeout: how long idle connections stay open for reuse; force_close=True disables keep-alive.
ssl=: False to skip verification (avoid), or an ssl.SSLContext for custom CAs and client certificates. Can also be passed per request.
family, local_addr, and happy_eyeballs_delay for IPv4/IPv6 control.

When all connections are in use, requests wait for one; that wait counts against the connect timeout.

A connector belongs to its session and is closed with it. To share one pool between several sessions (with different headers or cookies), pass connector_owner=False and close the connector yourself. For Unix sockets, use aiohttp.UnixConnector(path=...).

Proxies are given per request with proxy="http://proxy:3128", or read from environment variables with ClientSession(trust_env=True).

Example

import asyncio
import ssl
import aiohttp

async def main():
    connector = aiohttp.TCPConnector(
        limit=50,             # at most 50 connections overall
        limit_per_host=4,     # and 4 per host
        ttl_dns_cache=300,    # cache DNS answers for 5 minutes
        ssl=ssl.create_default_context(),   # or cafile="corp-ca.pem"
    )
    async with aiohttp.ClientSession(connector=connector) as session:
        async def one(i):
            async with session.get("https://httpbin.org/delay/1") as r:
                return r.status
        t = asyncio.get_running_loop().time()
        statuses = await asyncio.gather(*(one(i) for i in range(8)))
        print(statuses.count(200))                                 # 8
        print(round(asyncio.get_running_loop().time() - t))        # 2 (4 per host at a time)
    print(connector.closed)                                        # True

asyncio.run(main())

Remember

TCPConnector(limit=100 by default, limit_per_host=0 by default)
limit_per_host keeps crawlers polite and fair
Connectors also handle DNS caching, keep-alive and SSL contexts

Watch out for

Setting limit=0 (unlimited) and exhausting file descriptors or the remote server.

Check yourself

8 concurrent requests to one host with limit_per_host=4 each take 1 s. About how long overall?

Official docs: docs.aiohttp.org/en/stable/client_advanced.html#connectors

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks