TopolearnMoving millions of items through AWS is dominated by request count and latency, and batching changes the economics.
Each AWS call costs a network round trip and often a per-request fee, so throughput work is about fewer, fuller requests and more of them in flight.
Batch APIs:
- DynamoDB
table.batch_writer()buffersput_item/delete_itemcalls intoBatchWriteItemrequests of 25 items and automatically resends unprocessed items;overwrite_by_pkeys=[...]dedupes within a buffer. Reads usebatch_get_item(100 keys), where you must handleUnprocessedKeysyourself. - SQS
send_message_batchanddelete_message_batchtake up to 10 entries; check theFailedlist in the response, since partial failures don't raise. - S3
delete_objectsremoves up to 1000 keys per call.
Concurrency: share one client per service across a thread pool sized to the workload, with max_pool_connections matching. For asyncio services, aiobotocore/aioboto3 give awaitable clients with the same API.
Overheads to remove: create clients once (client creation parses models and can take tens of milliseconds), reuse connections (keep-alive is automatic), and avoid head_object before every get_object when the get alone tells you what you need.
Backpressure: watch for throttling (ProvisionedThroughputExceededException, SlowDown, ThrottlingException), use adaptive retries for batch jobs, and spread hot keys. Measure with CloudWatch request metrics, not assumptions.
Example
from decimal import Decimal
import boto3
ddb = boto3.resource("dynamodb", region_name="us-east-1")
table = ddb.create_table(TableName="events", BillingMode="PAY_PER_REQUEST",
KeySchema=[{"AttributeName": "id", "KeyType": "HASH"}],
AttributeDefinitions=[{"AttributeName": "id", "AttributeType": "S"}])
table.wait_until_exists()
with table.batch_writer(overwrite_by_pkeys=["id"]) as batch: # 25 items per request
for i in range(1000):
batch.put_item(Item={"id": f"evt-{i}", "value": Decimal(i)})
print(table.scan(Select="COUNT")["Count"]) # 1000, written with 40 requests
sqs = boto3.client("sqs", region_name="us-east-1")
url = sqs.create_queue(QueueName="jobs")["QueueUrl"]
resp = sqs.send_message_batch(QueueUrl=url, Entries=[
{"Id": str(i), "MessageBody": f"job {i}"} for i in range(10)])
print(len(resp["Successful"]), resp.get("Failed", [])) # 10 []Remember
- Batch APIs cut request count: DynamoDB 25 writes, SQS 10 messages, S3 1000 deletes
- batch_writer resends unprocessed items; batch_get_item and SQS batches need manual checks
- Share clients across a sized thread pool, or use aioboto3 for asyncio
- Create clients once; watch throttling metrics
Watch out for
- Ignoring the Failed list of send_message_batch and silently losing messages.