TopolearnAt scale every AWS API pushes back, and naive retries turn a small throttle into a self-inflicted outage.
Every AWS API is rate limited, usually by token buckets per account, Region and operation (EC2 Describe* calls share buckets, for example). Throttles surface as ThrottlingException, TooManyRequestsException, RequestLimitExceeded, or HTTP 503 SlowDown from S3, which scales per key prefix (thousands of requests per second per prefix, growing gradually). Service Quotas are separate: limits on resources and rates, either adjustable (request an increase, ideally before launch, and alarm on usage metrics) or fixed.
Retry well:
- Exponential backoff with full jitter: sleep a random time between 0 and
min(cap, base * 2^attempt), so retrying clients spread out instead of synchronising. - Bound everything: few attempts, connect and read timeouts below your caller's timeout, and retry only at one layer; retries multiplied across layers amplify load.
- SDK standard mode adds a retry quota that stops retrying when most calls fail; adaptive mode adds client-side rate limiting.
Idempotency makes retries safe. Many create APIs accept a token (ClientToken on EC2 RunInstances, ClientRequestToken in CloudFormation and Secrets Manager): repeating a request with the same token returns the original result instead of creating a duplicate. Apply the same idea in your own APIs with idempotency keys.
Eventual consistency in control planes: a just-created resource may not be visible to Describe calls for a moment, and IAM changes take seconds to propagate. Use waiters, and retry NotFound briefly after creates.
Finally, keep control-plane calls (create, modify) out of recovery paths; rely on data-plane operations and pre-provisioned capacity.
Example
import random, time, uuid
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError
cfg = Config(retries={"mode": "adaptive", "max_attempts": 8}, connect_timeout=2, read_timeout=10)
ec2 = boto3.client("ec2", region_name="eu-west-1", config=cfg)
token = str(uuid.uuid4()) # ec2.run_instances(..., ClientToken=token): safe to retry
def full_jitter(op, attempts=5, base=0.1, cap=5.0, codes=("ThrottlingException",)):
for n in range(attempts):
try:
return op()
except ClientError as e:
if e.response["Error"]["Code"] not in codes or n == attempts - 1:
raise
time.sleep(random.uniform(0, min(cap, base * 2 ** n)))
throttle = ClientError({"Error": {"Code": "ThrottlingException"}}, "PutItem")
outcomes = iter([throttle, throttle, "ok"])
def flaky():
r = next(outcomes)
if isinstance(r, Exception):
raise r
return r
print(full_jitter(flaky)) # ok (after two throttled attempts)Remember
- Expect throttling; back off exponentially with full jitter
- Bound retries and timeouts, and retry at one layer only
- Use idempotency tokens so retried creates don't duplicate
- Control planes are eventually consistent: use waiters and keep them out of recovery paths
Watch out for
- Retrying in the SDK, the application and the load balancer at once, multiplying traffic during an incident.