TopolearnPartitioning decides ordering guarantees and load balance; getting it wrong causes subtle ordering bugs or overloaded partitions.
The producer's partitioner picks a partition for each record:
- with a key: a hash of the key modulo the partition count, so equal keys always go to the same partition
- without a key: modern clients use sticky partitioning, filling a batch for one partition before moving to another, which gives better batching than strict round robin
- you can also set the partition explicitly
The hash function differs between clients. The Java client uses murmur2; librdkafka (and thus confluent-kafka for Python, Go, .NET) defaults to consistent_random, based on CRC32. If producers in different languages write the same keys to one topic and ordering per key matters, set partitioner: murmur2_random in librdkafka clients to match Java.
Because the mapping is hash % partitions, adding partitions remaps keys: new records for a key may go to a different partition than old ones, breaking per-key ordering across the change. Choose the partition count with headroom up front.
Skewed keys create hot partitions: one huge customer or a null-like default key ("unknown") floods one partition and its consumer. Options are a more granular key (customer + order), salting a hot key across a few partitions when strict ordering is not needed, or a dedicated topic.
Example
from confluent_kafka import Producer
# match the Java client's murmur2 hashing so keys map to the same partitions
producer = Producer({
"bootstrap.servers": "localhost:9092",
"partitioner": "murmur2_random", # librdkafka default: consistent_random
})
placed = {}
def remember(err, msg):
placed.setdefault(msg.key().decode(), set()).add(msg.partition())
for i in range(30):
producer.produce("orders", key=f"c{i % 3}", value=str(i), on_delivery=remember)
producer.flush()
print(placed) # each key appears in exactly one partition, e.g.
# {'c0': {2}, 'c1': {0}, 'c2': {2}}Remember
- Keyed records: hash(key) % partitions; keyless: sticky batches
- Java uses murmur2; librdkafka defaults differ, set murmur2_random to match
- Adding partitions remaps keys and breaks per-key ordering across the change
- Skewed keys make hot partitions
Watch out for
- Mixing Java and Python producers on one keyed topic without aligning partitioners.