TopolearnNetwork or disk handlers can stall request threads and the asyncio event loop; a queue decouples them.
Every handler's emit() runs synchronously in the thread that logged. A slow destination (SMTP, HTTP, a network file system, or even a busy disk) blocks the caller. In asyncio code, that means the event loop.
The fix is two-stage logging:
QueueHandler(q)is the only handler on your loggers. Itsemitjust prepares the record (formats the message, drops unpicklable args and exception objects in favour of text) and puts it on a queue: fast and non-blocking.QueueListener(q, *handlers)runs a background thread that pulls records off the queue and passes them to the real, slow handlers.
Start the listener at startup and call listener.stop() at shutdown to flush remaining records. Since 3.12, dictConfig can create both: a handler with "class": "logging.handlers.QueueHandler" and a "handlers" list builds a listener, available as handler.listener, which you start yourself.
With multiprocessing, a multiprocessing.Queue lets worker processes ship records to one listener in the main process, which avoids interleaved writes to a shared file. Record attributes must be picklable then.
Use respect_handler_level=True on the listener so each downstream handler's level still applies.
Example
import logging, logging.handlers, queue, sys, time
class SlowHandler(logging.StreamHandler):
def emit(self, record):
time.sleep(0.2) # pretend network I/O
super().emit(record)
q = queue.Queue(-1) # unbounded
listener = logging.handlers.QueueListener(
q, SlowHandler(sys.stdout), respect_handler_level=True)
log = logging.getLogger("api")
log.addHandler(logging.handlers.QueueHandler(q))
log.setLevel(logging.INFO)
listener.start()
t = time.perf_counter()
for i in range(3):
log.info("request %d", i)
print(f"logging took {time.perf_counter() - t:.3f}s") # ~0.000s
listener.stop() # flushes: request 0, 1, 2Remember
- Handlers run in the logging thread; slow ones block callers
- QueueHandler enqueues, QueueListener emits from a background thread
- Stop the listener at shutdown to flush pending records
Watch out for
- Forgetting listener.stop() at exit, losing the last log lines.