TopolearnEvery listing needs pagination, and knowing why Scan is expensive keeps it out of request paths.
Scan reads every item in a table or index, optionally dropping some with a FilterExpression. It consumes read capacity for everything it touches, so on a large table it is slow and costly. Use it for exports, migrations and admin jobs, not on a user's request path.
Both Query and Scan return at most 1 MB per call (or Limit items read). When more data remains, the response includes LastEvaluatedKey; pass it as ExclusiveStartKey in the next call. Keep going until the key is absent. Note that Limit caps items evaluated, before filtering, so a page can come back with fewer items than Limit, or even zero, while more pages remain.
For APIs, return LastEvaluatedKey (serialized, often base64 JSON) to the client as an opaque cursor. DynamoDB has no offset pagination: you cannot jump to page 7.
For big jobs, a parallel scan splits the table into TotalSegments and lets workers each scan one Segment concurrently. For full exports to analytics, the managed Export to S3 feature is better still: it reads from a point-in-time backup and consumes no table capacity.
The low-level client also offers paginators (client.get_paginator("query")) that loop for you.
Example
import boto3
table = boto3.resource("dynamodb").Table("Orders")
def scan_all(table, **kwargs):
"""Yield every item, following LastEvaluatedKey page by page."""
while True:
page = table.scan(**kwargs)
yield from page["Items"]
if "LastEvaluatedKey" not in page:
return
kwargs["ExclusiveStartKey"] = page["LastEvaluatedKey"]
with table.batch_writer() as bw:
for n in range(250):
bw.put_item(Item={"customer_id": f"c{n % 7}", "order_id": f"o{n:04}"})
print(sum(1 for _ in scan_all(table, Limit=40))) # 250 (7 pages of up to 40)Remember
- Scan reads and bills for every item, filters or not
- Pages are at most 1 MB; follow LastEvaluatedKey until it is missing
- Limit counts items evaluated before filtering
- Prefer Export to S3 or parallel scans for whole-table jobs
Watch out for
- Treating an empty page as the end of results while LastEvaluatedKey is still present.