TopolearnTombstone-heavy reads and zombie data are classic Cassandra production failures, and both are preventable.
Deletes, TTL expiry and null writes leave tombstones. They must be kept for a while: if a replica was down during a delete and a tombstone were purged before that replica learned about it, the old value on that replica would come back during the next sync. This is zombie (resurrected) data.
So tombstones are kept for gc_grace_seconds (default 864000, 10 days), and the rule is: every replica must be repaired within gc_grace_seconds. Repair (anti-entropy) compares replicas with Merkle trees and streams differences. Run it routinely, e.g. nodetool repair -pr on each node in turn or via an automated tool, and prefer incremental repair which only processes unrepaired data. Hints and read repair help but don't replace it.
Tombstones also hurt reads: a query scanning a range must read every tombstone in it to know what's deleted. Cassandra logs a warning after tombstone_warn_threshold (1000) tombstones in one query and aborts it at tombstone_failure_threshold (100,000).
Avoiding tombstone trouble:
- don't use a table as a queue (insert then delete rows and scan from the start),
- don't write nulls; leave columns unset,
- prefer replacing frozen collections over large collection overwrites (overwriting a non-frozen collection writes a range tombstone),
- for time series, use TTL with TWCS and query only recent windows,
- lower
gc_grace_secondsonly if repairs reliably run more often.
Example
-- anti-pattern: a queue table; readers scan past ever more tombstones
CREATE TABLE shop.jobs (queue text, id timeuuid, payload text, PRIMARY KEY (queue, id));
DELETE FROM shop.jobs WHERE queue = 'email' AND id = 50554d6e-29bb-11ef-9f3c-0242ac120002;
SELECT * FROM shop.jobs WHERE queue = 'email' LIMIT 1; -- reads every tombstone first
-- WARN ... Read 1 live rows and 25000 tombstone cells for query ...
ALTER TABLE shop.jobs WITH gc_grace_seconds = 864000; -- 10 days (the default)
-- $ nodetool repair -pr shop # primary ranges of this node, run on every node
-- $ nodetool repair --full shop # full rather than incremental
-- $ nodetool tablestats shop.jobs # "Average tombstones per slice"Remember
- Tombstones live for gc_grace_seconds (10 days default)
- Repair every replica within gc_grace or deleted data can come back
- Reads must scan tombstones; queries warn at 1000 and fail at 100,000
- Avoid queue-like tables, null writes and needless collection overwrites
Watch out for
- Lowering gc_grace_seconds to shrink tombstones without making repairs run more often, resurrecting deleted data.