TopolearnTopics outlive the code that writes them, so event formats must evolve without breaking existing consumers.
Kafka stores bytes; the meaning of those bytes is a contract between producers and consumers. Common formats:
- JSON: readable and easy, but verbose, untyped and with nothing stopping a producer from changing a field's type
- Avro: compact binary with a schema; strong support for schema evolution, popular in data pipelines
- Protobuf: compact, typed, familiar to gRPC users
A schema registry (Confluent Schema Registry, Apicurio, AWS Glue Schema Registry) stores versioned schemas per topic. Serializers register or look up the schema and prefix each message with a small schema id; deserializers fetch the schema by id. The registry enforces a compatibility mode on new versions:
- BACKWARD (Confluent default): new consumers can read old data, e.g. adding a field with a default or deleting a field
- FORWARD: old consumers can read new data
- FULL: both
Upgrade order follows the mode: with BACKWARD, upgrade consumers first.
Design events as facts with a stable identity: include an event id, event type, timestamp and the entity key. Never repurpose a field. Keep the key format simple and stable, because it drives partitioning and compaction.
Example
import json
from dataclasses import dataclass, asdict
@dataclass
class OrderCreated:
event_id: str
order_id: int
customer: str
total_cents: int
schema_version: int = 1
def serialize(evt) -> bytes:
return json.dumps(asdict(evt), separators=(",", ":")).encode()
def deserialize(data: bytes) -> dict:
d = json.loads(data)
d.setdefault("currency", "USD") # tolerate older events (field added in v2)
return d
raw = serialize(OrderCreated("e-1", 17, "c1", 3590))
print(raw) # b'{"event_id":"e-1","order_id":17,"customer":"c1","total_cents":3590,"schema_version":1}'
print(deserialize(raw)["currency"]) # USDRemember
- Kafka stores bytes; the format is a producer/consumer contract
- Avro and Protobuf with a schema registry give typed, evolvable events
- Compatibility modes (BACKWARD, FORWARD, FULL) gate schema changes
- Include event id, type, timestamp and key in every event
Watch out for
- Changing a JSON field's type in the producer and breaking every consumer at once.