TopolearnWhen consumers fall behind, RabbitMQ protects itself by slowing or blocking publishers, which looks like a hung application.
RabbitMQ pushes back on publishers in two ways.
Credit-based flow control works continuously: each internal process grants limited credit to the one sending to it. A connection publishing faster than queues can absorb is throttled, and appears as flow in the management UI. This is normal under load.
Resource alarms are drastic. When a node's memory use crosses the high watermark (vm_memory_high_watermark.relative, default 0.6 of available RAM) or free disk drops below disk_free_limit, the alarm blocks all publishing connections across the cluster. Consumers keep working, so the broker can drain. Clients receive a connection.blocked notification; a publisher that ignores it just hangs.
Prevent alarms rather than react to them:
- Size consumers so queues stay short; long queues are the root cause. Classic queues (version 2 storage) and quorum queues keep few messages in memory, but backlogs still cost disk and recovery time.
- Use max-length policies with
reject-publishoverflow so publishers get a clear negative confirm instead of a whole-cluster block. - Separate publishing and consuming connections, so a blocked publisher connection doesn't stall consumers in the same process.
Monitor with the built-in rabbitmq_prometheus plugin (/metrics on port 15692) and the official Grafana dashboards: queue depth, publish versus deliver rates, unacked counts, memory, file descriptors, alarms and connection churn.
Example
# rabbitmq.conf
vm_memory_high_watermark.relative = 0.6
disk_free_limit.absolute = 5GB
consumer_timeout = 1800000 # 30 min in ms (the default)
# Is an alarm active?
rabbitmq-diagnostics alarms
# Node rabbit@rabbit1 reported no alarms, local or clustered
rabbitmq-diagnostics memory_breakdown --unit mb | head -5
# Cap a queue so publishers get nacks instead of a cluster-wide block
rabbitmqctl set_policy cap '^tasks$' \
'{"max-length":500000,"overflow":"reject-publish"}' --apply-to queues
# Prometheus metrics
curl -s localhost:15692/metrics | grep -m2 rabbitmq_queue_messages_readyRemember
- Credit flow throttles fast publishers; alarms block them entirely
- Memory watermark default 0.6; disk_free_limit protects the disk
- Keep queues short and monitor with the Prometheus plugin
Watch out for
- Publishing and consuming on the same connection; when it's blocked, consumers stall too and the backlog never drains.