Replication and Sentinel

Redis · Expert · 14 min

High availability for Redis means understanding asynchronous replication and its failure modes.

A replica connects to a primary with REPLICAOF host port. It first performs a full sync (the primary forks to produce an RDB, then streams buffered writes), then receives the live command stream. After a short disconnect it resumes with a partial resync from the primary's replication backlog (repl-backlog-size); if the gap is larger, it needs a new full sync.

Replication is asynchronous: the primary acknowledges writes before replicas have them. If the primary dies, the last writes may be lost when a replica is promoted. WAIT numreplicas timeout blocks until that many replicas acknowledged previous writes, reducing (not eliminating) the window. min-replicas-to-write makes a primary refuse writes when too few replicas are connected, limiting damage in a network partition.

Replicas are read-only by default and can serve reads, with possible staleness.

Redis Sentinel provides automatic failover for non-cluster setups. Run at least three Sentinel processes on separate machines. They monitor the primary; when a quorum agrees it is down, one Sentinel is elected leader, promotes the best replica, reconfigures the others, and publishes the new address. Clients ask Sentinel for the current primary (SENTINEL get-master-addr-by-name mymaster) instead of hard-coding it.

Split brain is the classic hazard: an isolated old primary keeps accepting writes that are discarded when it rejoins as a replica.

Example

# on the replica
redis-cli -p 6380 REPLICAOF 127.0.0.1 6379          # OK
redis-cli -p 6380 INFO replication | grep -E 'role|master_link_status'
# role:slave
# master_link_status:up

# on the primary: wait until 1 replica has the previous writes (max 100 ms)
redis-cli SET balance:ana 100                        # OK
redis-cli WAIT 1 100                                 # (integer) 1

# sentinel.conf (one of three Sentinels)
# sentinel monitor mymaster 10.0.0.1 6379 2          # quorum 2
# sentinel down-after-milliseconds mymaster 5000
# sentinel failover-timeout mymaster 60000
redis-cli -p 26379 SENTINEL get-master-addr-by-name mymaster
# 1) "10.0.0.1"  2) "6379"

Remember

  • Replicas do a full sync, then follow the command stream; backlog enables partial resync
  • Replication is asynchronous: acknowledged writes can be lost on failover
  • WAIT and min-replicas-to-write narrow the loss window
  • Sentinel (3+ instances) detects failure by quorum and promotes a replica

Watch out for

  • Running a single Sentinel, which is itself a single point of failure and can't form a quorum.

Check yourself

  1. A client gets OK for a write, then the primary crashes before replicating it and a replica is promoted. What happens to the write?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: redis.io/docs/latest/operate/oss_and_stack/management/replication/