TopolearnRetention decides storage cost and how far back consumers can replay; compaction turns a topic into a changelog of current state.
Each partition is stored as a series of segment files. Old data is removed a whole segment at a time according to the topic's cleanup.policy.
With cleanup.policy=delete (the default), segments are deleted when they are older than retention.ms (7 days by default) or when the partition exceeds retention.bytes (unlimited by default). Set retention.ms=-1 to keep data forever. Because only closed segments are deleted, data can live somewhat longer than the retention setting.
With cleanup.policy=compact, Kafka keeps at least the latest record for each key and removes older ones in the background. The topic becomes a changelog: replaying it from the start yields the current state of every key. Uses: user profiles, configuration, Kafka Streams state, CDC tables.
- A record with a key and a null value is a tombstone: it marks the key as deleted, and is itself removed after
delete.retention.ms(1 day by default). - Compaction never touches the active segment, and offsets never change; compacted topics simply have gaps.
compact,deletecombines both: latest value per key, but nothing older than the retention time.
Retention is set per topic with kafka-configs.sh --alter --entity-type topics, and changes apply without restarts.
Example
# keep 3 days of clickstream data
kafka-configs.sh --bootstrap-server localhost:9092 --alter \
--entity-type topics --entity-name clicks --add-config retention.ms=259200000
# a compacted topic holding the latest profile per user
kafka-topics.sh --bootstrap-server localhost:9092 --create --topic user-profiles \
--partitions 6 --replication-factor 3 --config cleanup.policy=compact
# records over time (key:value) after compaction
# u1:{"name":"Ann"} u2:{"name":"Bob"}
# u2:{"name":"Bob"} u1:{"name":"Ann K."}
# u1:{"name":"Ann K."} u3:null (tombstone, removed later)
# u3:{"name":"Cy"}
# u3:nullRemember
- delete policy: drop segments older than retention.ms or beyond retention.bytes
- compact policy: keep at least the latest value per key
- A null value is a tombstone that deletes a key
- Cleanup works on closed segments; offsets never change
Watch out for
- Expecting a compacted topic to contain only one record per key at all times; recent duplicates remain until compaction runs.