TopolearnRebalance storms cause lag spikes and duplicates in production; knowing the protocols lets you avoid them.
A rebalance reassigns partitions when group membership or subscriptions change. Consumers prove they are alive in two ways:
- heartbeats from a background thread; missing them for
session.timeout.ms(45 s default) removes the member - polling: if the application does not call
poll()withinmax.poll.interval.ms(5 minutes default), the client leaves the group, assuming processing is stuck
Slow processing per batch is the classic cause of rebalance loops: lower max.poll.records (Java) or process asynchronously while pausing partitions.
Assignment strategies:
- eager (range, round robin): every member gives up all partitions, then gets a new set; all consumption stops briefly
- cooperative sticky: only partitions that actually move are revoked, in incremental rounds; the others keep flowing
Static membership (group.instance.id set to a stable name per instance) lets a restarted instance reclaim its partitions within the session timeout without any rebalance, which is ideal for rolling deploys on Kubernetes.
Kafka 4.0 made the next-generation consumer group protocol (KIP-848) generally available: the broker computes assignments and members reconcile incrementally, removing the global synchronization barrier. Enable it on clients with group.protocol=consumer. Separately, share groups let many consumers process records from the same partition with per-record acknowledgement, like a queue.
Use rebalance callbacks (on_assign, on_revoke) to commit offsets or flush state before partitions move away.
Example
from confluent_kafka import Consumer
def on_revoke(consumer, partitions):
consumer.commit(asynchronous=False) # save progress before handing off
print("revoked", [p.partition for p in partitions])
def on_assign(consumer, partitions):
print("assigned", [p.partition for p in partitions])
consumer = Consumer({
"bootstrap.servers": "localhost:9092",
"group.id": "billing",
"group.instance.id": "billing-pod-0", # static membership
"partition.assignment.strategy": "cooperative-sticky",
"session.timeout.ms": 45000,
"max.poll.interval.ms": 300000,
"enable.auto.commit": False,
})
consumer.subscribe(["orders"], on_assign=on_assign, on_revoke=on_revoke)
# with group.protocol=consumer (KIP-848), assignment moves to the brokerRemember
- Heartbeats cover liveness; max.poll.interval.ms covers stuck processing
- Cooperative sticky moves only the partitions that must move
- Static membership avoids rebalances on quick restarts
- KIP-848 (group.protocol=consumer) moves assignment to the broker
Watch out for
- Doing minutes of work per poll batch and getting kicked out of the group by max.poll.interval.ms.