Consumer groups and scaling

Every consumer belongs to a consumer group named by group.id. Kafka assigns each partition of the subscribed topics to exactly one consumer in the group, so the group as a whole processes every record once, in parallel.

• Start a second instance of the same service with the same group.id, and partitions are redistributed between the two: a rebalance.
• If an instance dies, its partitions move to the survivors.
• A group can usefully have at most as many active consumers as partitions; extra consumers sit idle. Partition count therefore caps a group's parallelism.

Different groups are independent: the billing group and the analytics group each receive every record of orders, each with its own committed offsets. This is the publish/subscribe side of Kafka, and why adding a new consumer application is free for everyone else.

Committed offsets are stored per group and partition in the internal __consumer_offsets topic.

During a rebalance, consumption in the affected partitions pauses briefly. Frequent rebalances (instances crashing, or processing that takes longer than max.poll.interval.ms between polls) are a common source of duplicate processing and lag; later lessons cover how to avoid them.