TopolearnPartitions are the unit of ordering, parallelism and scaling, so nearly every Kafka design decision involves them.
A topic is a named stream of records, such as orders or page-views. Each topic is split into one or more partitions. A partition is an append-only log: new records are added at the end and each gets a sequential offset (0, 1, 2, ...) that never changes.
A record has a key (optional), a value, a timestamp and optional headers. The key decides the partition: records with the same key always go to the same partition (as long as the partition count does not change). Records without a key are spread across partitions.
Ordering rules follow from this:
- within a partition, records are strictly ordered by offset
- across partitions of a topic, there is no global order
So if all events for order 17 must be processed in order, give them the key 17.
Partitions are also the unit of parallelism: different partitions can live on different brokers and be read by different consumers at the same time. More partitions allow more throughput but cost more resources; you can add partitions later but never remove them, and adding them changes which partition a key maps to.
Consumers track their position in each partition as an offset, which is what lets them stop, restart and replay.
Example
# topic "orders" with 3 partitions; records keyed by customer id
#
# partition 0: [0] c7:created [1] c7:paid [2] c2:created -> next offset 3
# partition 1: [0] c1:created [1] c1:paid [2] c1:shipped -> next offset 3
# partition 2: [0] c4:created -> next offset 1
#
# - every event for c1 is in partition 1, in order: created, paid, shipped
# - "c7:paid" and "c1:paid" have no defined order relative to each other
# - a consumer that has processed partition 1 up to offset 1 resumes at offset 2Remember
- Topics are split into partitions; each is an ordered append-only log
- Offsets identify records within a partition
- Same key, same partition, so per-key ordering is preserved
- No ordering across partitions; partitions can be added but not removed
Watch out for
- Expecting events in a multi-partition topic to be consumed in global production order.