TopolearnScaling, replacing nodes and multi-DC topologies all depend on how the ring is laid out and how nodes coordinate.
Partitioner and tokens. The Murmur3Partitioner hashes each partition key to a 64-bit token. Each node owns ranges of the token ring; a partition's first replica is the node owning its token, and further replicas are found by walking the ring.
Virtual nodes. Rather than one big range, each node owns many small ranges (num_tokens, 16 by default since 4.0, with an allocation algorithm that keeps ownership balanced). Adding or removing a node then moves small slices from many peers, spreading streaming load.
Replica placement. NetworkTopologyStrategy places replicas per data center and, using the snitch, on different racks where possible, so a rack failure doesn't take out all copies. GossipingPropertyFileSnitch (each node declares its dc and rack in cassandra-rackdc.properties) is the standard choice.
Gossip. Every second, each node exchanges state (status, load, schema version, tokens) with a few peers, so cluster membership spreads epidemically without a master. The phi accrual failure detector turns gossip heartbeat timing into a suspicion level to mark nodes down.
Coordinators route requests to replicas, store hints for down ones, and use dynamic snitch scores to prefer fast replicas for reads.
Operations follow from this: nodetool status shows each node's state, load and ownership; bootstrapping a node streams its ranges; nodetool decommission streams them away; nodetool cleanup removes data a node no longer owns after others join.
Example
# $ nodetool status shop
# Datacenter: dc1
# Status=Up/Down State=Normal/Leaving/Joining/Moving
# -- Address Load Tokens Owns (effective) Host ID Rack
# UN 10.0.0.1 412 GiB 16 75.0% 8d5e... r1
# UN 10.0.0.2 398 GiB 16 75.0% 1f2a... r2
# UN 10.0.0.3 405 GiB 16 75.0% c3b7... r3
# DN 10.0.0.4 401 GiB 16 75.0% 9e04... r1
# cassandra.yaml
# num_tokens: 16
# endpoint_snitch: GossipingPropertyFileSnitch
# cassandra-rackdc.properties
# dc=dc1
# rack=r1
# $ nodetool getendpoints shop users 5b6962dd-3f90-4c93-8f61-eabfa4a803e2
# $ nodetool cleanup shop # after adding nodes, drop ranges no longer ownedRemember
- Murmur3 tokens map partitions onto a ring of node-owned ranges
- vnodes (num_tokens 16) spread ownership and streaming load
- Snitch + NetworkTopologyStrategy place replicas across racks and DCs
- Gossip and the phi accrual detector manage membership without a master
Watch out for
- Forgetting nodetool cleanup after adding nodes, so old nodes keep data they no longer own.