Network I/O with streams

asyncio · Advanced · 14 min

Streams show how async I/O really works and are the base for custom protocols and tests.

asyncio's high-level networking API is streams: a StreamReader / StreamWriter pair.

Client: reader, writer = await asyncio.open_connection(host, port). Server: server = await asyncio.start_server(handle, host, port), where handle(reader, writer) is a coroutine called for each connection, each in its own task.

Reading:

  • await reader.readline(), await reader.readexactly(n), await reader.read(n) (up to n bytes; b"" means EOF).
  • readuntil(sep) for delimited protocols.

Writing is two steps: writer.write(data) buffers without waiting, then await writer.drain() waits until the buffer is below the high-water mark. Calling drain() is what gives you flow control; skipping it on big writes can grow memory without bound.

Closing: writer.close() then await writer.wait_closed(). Servers are async context managers; async with server: await server.serve_forever() runs until cancelled.

Under streams sit transports and protocols, a callback-based API used by high-performance libraries. start_tls() upgrades a connection, and open_connection(..., ssl=ctx) does TLS directly. For HTTP, use a library such as aiohttp or httpx instead of raw streams.

Example

import asyncio

async def handle(reader, writer):
    line = await reader.readline()                 # b"hello\n"
    writer.write(line.upper())
    await writer.drain()                           # flow control
    writer.close()
    await writer.wait_closed()

async def main():
    server = await asyncio.start_server(handle, "127.0.0.1", 0)
    port = server.sockets[0].getsockname()[1]
    async with server:
        reader, writer = await asyncio.open_connection("127.0.0.1", port)
        writer.write(b"hello\n")
        await writer.drain()
        print(await reader.readline())             # b'HELLO\n'
        writer.close()
        await writer.wait_closed()

asyncio.run(main())

Remember

  • open_connection / start_server give StreamReader and StreamWriter
  • write() buffers; await drain() applies back-pressure
  • Close with close() then await wait_closed()

Watch out for

  • Calling writer.write in a loop without awaiting drain(), letting the buffer grow unbounded.

Check yourself

  1. What does await reader.read(100) returning b'' mean?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: docs.python.org/3/library/asyncio-stream.html