TopolearnWrite throughput, read latency and disk usage all follow from Cassandra's log-structured storage engine.
Cassandra uses a log-structured merge design.
Write path on each replica: the mutation is appended to the commit log (for durability; commitlog_sync is periodic by default, fsyncing every 10 seconds) and applied to an in-memory memtable per table. Nothing is read or updated in place, which is why writes are fast. When a memtable fills, it is flushed to disk as an immutable SSTable, and the corresponding commit log segments can be recycled.
Read path: a partition's data may be spread over the memtable and several SSTables, so the read merges them, picking the newest cell by timestamp and applying tombstones. To avoid touching every SSTable, each has a bloom filter (is this partition possibly here?), a partition index, and in-memory summaries; 5.0 adds the trie-based BTI SSTable format and trie memtables for lower memory use. Row and key caches are optional extras.
Compaction merges SSTables in the background, keeping the newest version of each cell and purging tombstones older than gc_grace_seconds. The strategy is set per table:
SizeTieredCompactionStrategy: write-optimised, needs spare disk for merges,LeveledCompactionStrategy: read-optimised, more I/O,TimeWindowCompactionStrategy: for TTL'd time series,UnifiedCompactionStrategy(5.0): configurable to behave like either tiered or leveled.
The number of SSTables a read touches (see nodetool tablehistograms) is a key health metric.
Example
-- per-table compaction choice
ALTER TABLE shop.readings_by_day
WITH compaction = {'class': 'TimeWindowCompactionStrategy',
'compaction_window_unit': 'DAYS', 'compaction_window_size': 1};
ALTER TABLE shop.users
WITH compaction = {'class': 'UnifiedCompactionStrategy'}; -- 5.0
-- $ nodetool flush shop users # memtable -> new SSTable
-- $ nodetool tablestats shop.users # SSTable count, bloom filter false ratio
-- $ nodetool tablehistograms shop users # SSTables touched per read (percentiles)
-- $ nodetool compactionstats # running and pending compactionsRemember
- Writes: commit log + memtable, flushed to immutable SSTables
- Reads merge memtable and SSTables, newest timestamp wins
- Bloom filters and indexes skip SSTables that can't contain the partition
- Compaction merges SSTables; strategy is chosen per table
Watch out for
- Picking LeveledCompactionStrategy for a write-heavy time-series table and drowning in compaction I/O.