TopolearnProduction brokers run as clusters, and misunderstanding what a cluster replicates leads to data loss or downtime.
A cluster is a group of nodes (usually three or five) that share metadata: users, vhosts, exchanges, bindings, queue definitions, policies. Nodes authenticate each other with a shared Erlang cookie and find each other through peer discovery (static list, DNS, Kubernetes, AWS, Consul). Clients can connect to any node; the cluster routes messages internally to wherever a queue's replicas live.
What's replicated depends on the queue type, not on clustering itself:
- Classic queues live on a single node. If that node is down, the queue is unavailable; if its disk is lost, so are its messages.
- Quorum queues and streams are replicated across nodes and survive node failures, as long as a majority of their replicas is up.
Metadata itself is stored in a database replicated to all nodes. Since RabbitMQ 4.2, new deployments use Khepri, a Raft-based metadata store, by default; it replaces the older Mnesia, which had weaker behaviour under network partitions and will be removed in a future version.
A network partition splits nodes into groups that can't see each other. With Raft-based components, only the side with a majority keeps operating for those queues; the minority stops accepting work and rejoins later. For Mnesia-based clusters the cluster_partition_handling strategy (pause_minority recommended) decides.
Practical rules: use an odd number of nodes (3 or 5), spread across availability zones with low latency, perform rolling upgrades one node at a time, and never put cluster nodes across a WAN; use federation or shovels to connect distant brokers.
Example
# Form a 3-node cluster (rabbit2 joins rabbit1; repeat for rabbit3)
rabbitmqctl -n rabbit@rabbit2 stop_app
rabbitmqctl -n rabbit@rabbit2 join_cluster rabbit@rabbit1
rabbitmqctl -n rabbit@rabbit2 start_app
rabbitmqctl cluster_status
# Disk Nodes: rabbit@rabbit1, rabbit@rabbit2, rabbit@rabbit3
# Running Nodes: ... Network Partitions: (none)
# Declarative alternative in rabbitmq.conf (every node):
# cluster_formation.peer_discovery_backend = classic_config
# cluster_formation.classic_config.nodes.1 = rabbit@rabbit1
# cluster_formation.classic_config.nodes.2 = rabbit@rabbit2
# cluster_formation.classic_config.nodes.3 = rabbit@rabbit3
# Before maintenance: move quorum queue leaders away, then stop the node
rabbitmq-upgrade drain
rabbitmq-diagnostics check_if_node_is_quorum_criticalRemember
- Clusters share metadata; data replication depends on the queue type
- Classic queues live on one node; quorum queues and streams are replicated
- Use 3 or 5 nodes in one region; federation or shovels across WANs
Watch out for
- Assuming a cluster protects classic queue messages; they still live on one node.