TopolearnKafka is fast by default, but the right settings often multiply throughput or cut latency for a given workload.
Kafka's throughput comes from batching: the producer groups records per partition and sends them together.
Producer settings:
linger.ms: how long to wait to fill a batch (5 ms by default in recent Java and librdkafka clients); raising it to 10 to 50 ms increases throughput at a small latency costbatch.size(Java) /batch.sizeandbatch.num.messages(librdkafka): maximum batch sizecompression.type:lz4orzstdcompress whole batches, reducing network and disk use, often by several times for JSON; brokers store batches compressed
Consumer settings: fetch.min.bytes and fetch.max.wait.ms trade latency for bigger fetches; max.poll.records (Java) bounds work per poll.
Partition count sets the ceiling on consumer parallelism. Estimate it from target throughput divided by what one consumer can process, with headroom. Very high partition counts increase metadata, open files and recovery time.
Monitor:
- consumer lag per group and partition (kafka-consumer-groups, Burrow, or exporter metrics); growing lag means consumers are too slow
- broker metrics: under-replicated partitions, request latency, network and disk usage, ISR shrink and expand rates
- producer metrics: record error rate, request latency, batch size average
Example
from confluent_kafka import Producer, Consumer
producer = Producer({
"bootstrap.servers": "localhost:9092",
"linger.ms": 20, # wait up to 20 ms to fill batches
"batch.size": 262144, # up to 256 KB per batch
"compression.type": "zstd", # compress whole batches
"enable.idempotence": True,
})
consumer = Consumer({
"bootstrap.servers": "localhost:9092",
"group.id": "analytics",
"fetch.min.bytes": 65536, # wait for at least 64 KB ...
"fetch.wait.max.ms": 100, # ... or 100 ms, whichever comes first
})
# lag of one partition = high watermark - committed offset
# low, high = consumer.get_watermark_offsets(tp); committed = consumer.committed([tp])Remember
- Batching (linger.ms, batch.size) and compression drive throughput
- zstd or lz4 compression often cuts bytes several times
- Partition count caps consumer parallelism; size with headroom
- Watch consumer lag and under-replicated partitions
Watch out for
- Sending each record synchronously and waiting for its acknowledgement, which removes batching entirely.