TopolearnThese settings decide whether a broker crash loses or duplicates your data.
acks sets when the leader acknowledges a write:
acks=0: no acknowledgement; fastest, may lose dataacks=1: the leader wrote it; lost if the leader dies before followers copy itacks=all(-1): all in-sync replicas have it; the default in modern Java and librdkafka clients
acks=all is only as strong as the ISR is large. The topic or broker setting min.insync.replicas sets the minimum ISR size for an acks=all write to succeed. The standard durable setup is replication factor 3, min.insync.replicas 2, acks=all: one broker can be down and writes continue, and every acknowledged record is on at least two brokers.
Producers retry transient failures automatically until delivery.timeout.ms (2 minutes by default) expires. Retries can create duplicates (the first attempt succeeded but the acknowledgement was lost). Idempotence (enable.idempotence=true) fixes this: the broker deduplicates using a producer id and per-partition sequence numbers, and ordering is preserved even with several requests in flight. It is on by default in the Java client since 3.0, but off by default in librdkafka, so enable it explicitly in confluent-kafka.
Even then, after delivery.timeout.ms the delivery callback reports an error; handle it (log, alert, persist) rather than ignoring it.
Example
from confluent_kafka import Producer
producer = Producer({
"bootstrap.servers": "localhost:9092",
"acks": "all", # wait for all in-sync replicas
"enable.idempotence": True, # no duplicates or reordering on retry
"delivery.timeout.ms": 120000, # give up (and report) after 2 minutes
})
failed = []
def on_delivery(err, msg):
if err is not None:
failed.append((msg.key(), err)) # do not ignore: alert or persist
producer.produce("payments", key="p-991", value='{"amount": 30}', on_delivery=on_delivery)
producer.flush()
print("failed:", failed) # failed: []
# topic side (kafka-configs.sh or at creation):
# replication.factor=3, min.insync.replicas=2Remember
- acks=all waits for every in-sync replica
- RF 3 + min.insync.replicas 2 + acks=all is the standard durable setup
- Idempotence removes retry duplicates; enable it explicitly in librdkafka
- Handle delivery errors after delivery.timeout.ms
Watch out for
- Using acks=all with min.insync.replicas=1, which acknowledges writes held by a single broker.