TopolearnDistributing background jobs across workers is the most common RabbitMQ use, and prefetch decides how evenly it works.
When several consumers read the same queue, RabbitMQ delivers messages to them round robin: each message goes to exactly one worker. To process jobs faster, start more workers; that's horizontal scaling with no code change.
Round robin alone is naive: RabbitMQ pushes messages to consumers as fast as it can, so a worker can receive many messages while still busy with a slow one, and others sit idle. Prefetch fixes this. basic_qos(prefetch_count=N) limits each consumer to N unacknowledged messages; the broker sends the next one only after an ack. It requires manual acknowledgements (next lesson), since with auto-ack there is nothing to wait for.
Choosing N:
1: the fairest distribution, best for slow, uneven tasks, but the worker idles for a round trip between messages.10to300: much higher throughput for fast, small messages.- Unlimited (
0): risky; a single consumer may buffer the whole queue in memory.
Consumers in one queue also give you failover for free: if a worker dies, its unacknowledged messages go back to the queue and other workers pick them up. Keep task handlers idempotent, because that redelivery can happen after partial work.
Example
# worker.py -- run several copies of this process
import time, pika
conn = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
ch = conn.channel()
ch.queue_declare(queue="tasks", durable=True)
ch.basic_qos(prefetch_count=1) # at most 1 unacked task per worker
def handle(channel, method, properties, body):
print("working on", body.decode())
time.sleep(body.count(b".")) # simulate work: one second per dot
channel.basic_ack(delivery_tag=method.delivery_tag)
ch.basic_consume(queue="tasks", on_message_callback=handle)
ch.start_consuming()
# producer: ch.basic_publish("", "tasks", b"resize image.....")Remember
- Consumers on one queue share its messages round robin
- prefetch_count limits unacked messages per consumer
- Unacked messages of a dead worker are redelivered to others
Watch out for
- Leaving prefetch unlimited so one worker receives thousands of messages while others idle.