Scaling and observability in production

Production incidents in FastAPI services are usually pool exhaustion, a blocked loop or a slow upstream, and only telemetry tells you which.

Capacity is a chain of limits, and the smallest one wins:

Workers x replicas set how many event loops you have.
Each worker's 40-thread pool caps concurrent sync endpoints and sync dependencies.
Each worker's database pool (pool_size + max_overflow in SQLAlchemy) caps concurrent queries. Workers x replicas x pool must stay below the database's max_connections; beyond a few dozen app processes, put PgBouncer or a similar pooler in front.

Timeouts everywhere. An async server happily accepts thousands of requests that all wait on a slow upstream until memory or the pool runs out. Set client timeouts (httpx.AsyncClient(timeout=...)), database statement timeouts and a request deadline at the proxy, and shed load with 503s instead of queueing forever.

Graceful shutdown. On SIGTERM Uvicorn stops accepting connections, finishes in-flight requests and runs the lifespan shutdown. Use readiness checks so load balancers stop sending traffic first.

Observability:

Logs: structured JSON with a request id, set once in middleware and stored in a contextvars.ContextVar so every log line in the request can include it.
Metrics: rate, errors and duration per route template (/items/{item_id}, from request.scope["route"].path), never per raw path, which explodes metric cardinality.
Traces: OpenTelemetry spans across services. opentelemetry-instrumentation-fastapi works on any version, and FastAPI 0.142+ has native OpenTelemetry support, configured with FastAPI(telemetry={...}) and the standard OTEL_* variables, including spans for dependency solving and serialization.

When a worker hangs, py-spy dump --pid <pid> shows every thread's stack, and asyncio debug mode (PYTHONASYNCIODEBUG=1) logs callbacks that block the loop.

Example

import json, logging, time, uuid
from contextvars import ContextVar
from fastapi import FastAPI, Request
from fastapi.testclient import TestClient

request_id: ContextVar[str] = ContextVar("request_id", default="-")
logging.basicConfig(format="%(message)s")
log = logging.getLogger("api")
log.setLevel(logging.INFO)
app = FastAPI()

@app.middleware("http")
async def observe(request: Request, call_next):
    request_id.set(request.headers.get("x-request-id", uuid.uuid4().hex[:8]))
    start = time.perf_counter()
    response = await call_next(request)
    route = request.scope.get("route")        # set by the router after matching
    log.info(json.dumps({"rid": request_id.get(), "route": getattr(route, "path", "unmatched"),
                         "status": response.status_code,
                         "ms": round((time.perf_counter() - start) * 1000, 1)}))
    response.headers["x-request-id"] = request_id.get()
    return response

@app.get("/items/{item_id}")
async def read_item(item_id: int):
    return {"item_id": item_id}

TestClient(app).get("/items/42", headers={"x-request-id": "abc123"})
# {"rid": "abc123", "route": "/items/{item_id}", "status": 200, "ms": 1.2}

Remember

Throughput is capped by the smallest of workers, threads and DB connections
Set timeouts on every outbound call and shed load rather than queue forever
Label metrics by route template, not raw path
Correlate logs and traces with a request id in a ContextVar; use OpenTelemetry for traces

Watch out for

Scaling replicas until the database runs out of connections.

Check yourself

You scale from 4 to 20 replicas, each with 2 workers and a SQLAlchemy pool of 10 + 10 overflow. Postgres allows 200 connections. What's the risk?
Why label request metrics with /items/{item_id} instead of /items/42?

Official docs: fastapi.tiangolo.com/deployment/concepts/

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks