TopolearnProducing correctly, with keys and delivery checks, is the first step of every Kafka application.
The most widely used Python client is confluent-kafka (pip install confluent-kafka), a wrapper around the C library librdkafka. Java has the reference client; other languages mostly use librdkafka too, with the same configuration names.
A producer is created from a config dict; bootstrap.servers is the only required setting. produce(topic, key=..., value=...) does not send immediately: it appends the record to an in-memory buffer and returns. Background threads batch records per partition and send them to the leaders.
Because sending is asynchronous, you learn the outcome through a delivery callback (err, msg), called when the broker acknowledges (or the send ultimately fails). Callbacks are served when you call poll() or flush(), so long-running producers call producer.poll(0) regularly. Before exiting, call flush() to wait for everything in flight; otherwise buffered records are lost.
Keys and values are bytes; strings are encoded as UTF-8 for you. For structured data, serialize explicitly (JSON, Avro, Protobuf). If the local buffer fills up, produce raises BufferError; call poll to let it drain and retry.
Create one producer per process and share it; it is thread-safe and expensive to create.
Example
import json
from confluent_kafka import Producer
producer = Producer({"bootstrap.servers": "localhost:9092"})
def on_delivery(err, msg):
if err:
print("FAILED:", err)
else:
print(f"{msg.key().decode()} -> partition {msg.partition()} offset {msg.offset()}")
for order_id, customer in [(17, "c1"), (18, "c2"), (19, "c1")]:
event = {"order_id": order_id, "status": "created"}
producer.produce("orders", key=customer, value=json.dumps(event),
on_delivery=on_delivery)
producer.poll(0) # serve delivery callbacks
producer.flush(10) # wait up to 10 s for outstanding deliveries
# c1 -> partition 1 offset 0
# c2 -> partition 0 offset 0
# c1 -> partition 1 offset 1 (same key, same partition)Remember
produce()only enqueues; batching and sending happen in the background- Delivery callbacks report success or failure; serve them with poll()
- Always flush() before exit
- One shared producer per process
Watch out for
- Exiting a script right after produce() without flush(), silently losing the buffered records.