TopolearnQueues and pub/sub absorb spikes, isolate failures and let services evolve independently.
SQS is a fully managed message queue. Producers SendMessage; consumers poll with ReceiveMessage (use long polling, WaitTimeSeconds up to 20, to cut empty responses), process, then DeleteMessage.
When a message is received it becomes invisible for the visibility timeout. If the consumer doesn't delete it in time (crash, slow processing), it reappears and is delivered again. So delivery is at least once and consumers must be idempotent. Set the timeout comfortably above processing time.
After a message has been received maxReceiveCount times, a redrive policy moves it to a dead-letter queue (DLQ), so a poison message doesn't block or loop forever. Alarm on DLQ depth and redrive messages back after fixing the bug. Messages are retained 4 days by default, up to 14.
Queue types:
- Standard: nearly unlimited throughput, best-effort ordering, occasional duplicates.
- FIFO: strict ordering within a message group ID, deduplication within a 5-minute window, lower throughput limits (high-throughput mode raises them).
SNS is pub/sub: publish once to a topic and SNS pushes to every subscription: SQS queues, Lambda, HTTPS endpoints, email, SMS. Fan-out (SNS topic to several SQS queues) gives each consumer its own buffered copy. Filter policies on subscriptions deliver only matching messages, based on message attributes or the body. Enable raw message delivery to SQS to skip the SNS envelope.
A queue subscribed to a topic needs a queue policy allowing sns.amazonaws.com to send, scoped with aws:SourceArn to the topic.
Example
import json, boto3
sqs = boto3.client("sqs", region_name="eu-west-1")
sns = boto3.client("sns", region_name="eu-west-1")
arn = lambda url: sqs.get_queue_attributes(QueueUrl=url, AttributeNames=["QueueArn"])["Attributes"]["QueueArn"]
dlq = sqs.create_queue(QueueName="orders-dlq")["QueueUrl"]
q = sqs.create_queue(QueueName="orders", Attributes={"VisibilityTimeout": "60",
"RedrivePolicy": json.dumps({"deadLetterTargetArn": arn(dlq), "maxReceiveCount": "5"})})["QueueUrl"]
topic = sns.create_topic(Name="order-events")["TopicArn"]
sns.subscribe(TopicArn=topic, Protocol="sqs", Endpoint=arn(q), Attributes={
"RawMessageDelivery": "true", "FilterPolicy": json.dumps({"type": ["created"]})})
# (in real AWS the queue also needs a policy allowing this topic to send)
for kind in ("created", "cancelled"):
sns.publish(TopicArn=topic, Message=json.dumps({"id": 7, "type": kind}),
MessageAttributes={"type": {"DataType": "String", "StringValue": kind}})
for m in sqs.receive_message(QueueUrl=q, MaxNumberOfMessages=10, WaitTimeSeconds=1).get("Messages", []):
print(m["Body"]) # {"id": 7, "type": "created"} (cancelled was filtered out)
sqs.delete_message(QueueUrl=q, ReceiptHandle=m["ReceiptHandle"])Remember
- Receive, process, delete; undeleted messages reappear after the visibility timeout
- At-least-once delivery means idempotent consumers
- DLQs catch poison messages; FIFO gives ordering per message group
- SNS fans out to many subscribers; filter policies route by attributes
Watch out for
- A visibility timeout shorter than processing time, causing the same message to be processed in parallel twice.