TopolearnFailed messages need somewhere to go other than an endless requeue loop or the void.
A queue can have a dead-letter exchange (DLX). A message is dead-lettered, republished to the DLX with its original or an overridden routing key, when:
- a consumer rejects or nacks it with
requeue=False, - it expires because of a TTL,
- the queue is over its length limit (
x-max-length) with the drop-head overflow behaviour, - it exceeds a quorum queue's delivery limit.
Dead-lettered messages carry an x-death header recording the queue, reason and count, useful for debugging and retry logic.
TTL comes in two forms: a per-queue x-message-ttl (milliseconds) and a per-message expiration property. Separately, x-expires deletes an unused queue.
A common delayed retry pattern combines both: a failing consumer rejects the message; it is dead-lettered into a retry queue that has no consumers, a TTL of, say, 30 seconds and a DLX pointing back to the work exchange. After the TTL it returns for another attempt. Check the x-death count to stop after N attempts and park the message in a final "dead" queue for humans.
Prefer configuring DLX, TTL and length limits through policies rather than queue arguments: policies can be changed later without redeclaring queues.
Example
import pika
ch = pika.BlockingConnection(pika.ConnectionParameters("localhost")).channel()
ch.exchange_declare("work", "direct")
ch.exchange_declare("retry", "direct")
ch.queue_declare("jobs", durable=True, arguments={
"x-dead-letter-exchange": "retry"}) # rejected -> retry
ch.queue_bind("jobs", "work", "job")
ch.queue_declare("jobs.retry", durable=True, arguments={
"x-message-ttl": 30000, # wait 30 s
"x-dead-letter-exchange": "work"}) # then back to work
ch.queue_bind("jobs.retry", "retry", "job")
def attempts(props):
deaths = (props.headers or {}).get("x-death", [])
return sum(d["count"] for d in deaths if d["queue"] == "jobs")
# in the consumer:
# if attempts(props) >= 3: publish to "jobs.dead" and ack
# else: channel.basic_reject(method.delivery_tag, requeue=False)Remember
- Rejected, expired, overflowing or over-delivered messages go to the DLX
- x-message-ttl and per-message expiration set TTLs
- TTL queue + DLX back to the work exchange = delayed retry
Watch out for
- Retrying forever without counting x-death, so permanently bad messages circulate endlessly.