TopolearnUnderstanding storage explains Kafka's performance, retention behaviour and how to size disks and memory.
On disk, each partition replica is a directory of segments. A segment is a .log file holding record batches (as produced, often compressed) plus two sparse indexes: .index maps offsets to file positions and .timeindex maps timestamps to offsets. A new segment is rolled when the active one reaches segment.bytes (1 GiB default) or segment.ms age. Retention and compaction operate on closed segments only.
Why it is fast:
- sequential I/O: writes are appends and reads are mostly sequential scans, which disks and SSDs handle at full bandwidth
- the OS page cache does the caching: Kafka keeps little data on the JVM heap, so recently written data is usually served from memory, and consumers that keep up never touch the disk
- zero-copy: for plaintext connections, brokers send log data to the network with
sendfile, without copying through user space (TLS requires encryption in the JVM, which removes this benefit) - batching end to end: producer batches are stored and sent to consumers as-is, so compression happens once at the producer
Finding offset N means a binary search over segment file names (their base offsets), then the sparse index, then a short scan. Time-based lookups (offsetsForTimes) use the time index.
Tiered storage (production-ready since 3.9) moves closed segments to object storage such as S3, keeping only recent data on broker disks.
Example
# one partition replica on a broker's disk: <log.dirs>/orders-1/
ls -1 /var/lib/kafka/data/orders-1/
# 00000000000000000000.log segment starting at offset 0 (closed)
# 00000000000000000000.index sparse offset -> byte position
# 00000000000000000000.timeindex sparse timestamp -> offset
# 00000000000004718592.log active segment, base offset 4718592
# 00000000000004718592.index
# 00000000000004718592.timeindex
# leader-epoch-checkpoint
# partition.metadata
# inspect record batches in a segment
kafka-dump-log.sh --files /var/lib/kafka/data/orders-1/00000000000000000000.log | head -3Remember
- Partitions are directories of segments with sparse offset and time indexes
- Sequential I/O plus the OS page cache make reads and writes fast
- Zero-copy sendfile applies to plaintext; TLS loses it
- Tiered storage offloads closed segments to object storage
Watch out for
- Giving the broker JVM a huge heap and starving the page cache that Kafka relies on.