TopolearnThese mechanics define exactly when a write is safe and what can be lost in failures.
Followers replicate by fetching from the leader, like consumers. Each replica has a log end offset (LEO), the next offset it would write. A follower stays in the in-sync replica set (ISR) while it keeps up, specifically while it has caught up within replica.lag.time.max.ms (30 s default). Slow or dead followers are removed from the ISR and re-added once they catch up.
The leader advances the high watermark (HW) to the lowest LEO among ISR members. Records below the HW are committed: present on every in-sync replica. Consumers can only read up to the HW, so they never see data that might vanish in a failover. With acks=all, the producer gets its acknowledgement once the record is below the HW (and the ISR has at least min.insync.replicas members).
When a leader fails, the controller elects a new leader from the ISR, which by definition has every committed record. Leader epochs let replicas detect and truncate divergent uncommitted suffixes after failover.
If all ISR members are lost, unclean.leader.election.enable (false by default) decides between waiting for an ISR member to return (unavailability) and electing an out-of-sync replica (availability, but committed data is lost). Newer releases add Eligible Leader Replicas to keep more safe candidates available when the ISR shrinks.
Use rack awareness (broker.rack) so replicas spread across availability zones.
Remember
- ISR = replicas caught up within replica.lag.time.max.ms
- High watermark = smallest LEO in the ISR; consumers read up to it
- New leaders come from the ISR, so committed data survives
- Unclean election trades data loss for availability; off by default
Watch out for
- Enabling unclean leader election on topics holding financial data to avoid a short outage.