TopolearnDefault timeouts and retries rarely fit production; the wrong values cause hung requests or cascading throttling.
botocore.config.Config customizes a client: boto3.client("s3", config=Config(...)). Configs can be merged with cfg.merge(other).
Retries: botocore automatically retries throttling errors, 5xx responses and transient network errors with exponential backoff. The retries dict chooses the behavior:
mode="legacy": the long-standing default in boto3, with service-specific quirks.mode="standard": consistent rules across services, 3 total attempts by default, and a retry quota that stops retry storms. AWS recommends it for new code.mode="adaptive": standard plus client-side rate limiting that slows down when throttled; useful for single-process batch jobs hammering one API.
Mind the counting: in Config, total_max_attempts includes the first call, while the older max_attempts key counts retries only (so max_attempts=3 means up to 4 calls). The AWS_MAX_ATTEMPTS variable and the max_attempts profile setting count total attempts. The mode can also come from AWS_RETRY_MODE.
Timeouts: connect_timeout and read_timeout default to 60 seconds each, far too long for request-serving code: a stuck call ties up a worker for minutes once retries multiply it. Set them to what your latency budget allows.
Connection pool: max_pool_connections (default 10) limits concurrent connections per client; raise it when many threads share one client.
Endpoints: endpoint_url= on the client targets LocalStack, MinIO, VPC endpoints or FIPS endpoints. Other options include region_name, signature_version, s3={"addressing_style": "path"} and user_agent_extra.
Example
import boto3
from botocore.config import Config
fast_api = Config(
region_name="us-east-1",
retries={"mode": "standard", "total_max_attempts": 4}, # 1 call + up to 3 retries
connect_timeout=2,
read_timeout=5,
max_pool_connections=50,
)
ddb = boto3.client("dynamodb", config=fast_api)
print(ddb.meta.config.retries) # {'mode': 'standard', 'total_max_attempts': 4}
print(ddb.meta.config.read_timeout) # 5
batch = fast_api.merge(Config(retries={"mode": "adaptive", "total_max_attempts": 10}))
print(batch.retries["mode"], batch.connect_timeout) # adaptive 2
# Local S3-compatible storage (MinIO / LocalStack):
minio = boto3.client("s3", endpoint_url="http://localhost:9000",
config=Config(s3={"addressing_style": "path"}))
print(minio.meta.endpoint_url) # http://localhost:9000Remember
- Config(retries={'mode': 'standard', 'total_max_attempts': N}) controls retries
- In Config, total_max_attempts includes the first call; max_attempts counts retries only
- Default 60 s connect/read timeouts are too long for request paths
- endpoint_url targets LocalStack, MinIO or VPC endpoints
Watch out for
- Wrapping boto3 calls in your own retry loop on top of botocore's retries, multiplying attempts and latency.