Consumer Groups and Partition Assignment

A consumer group is not a pool of workers sharing a queue. Every partition belongs to exactly one member, the split is recomputed from scratch every time membership changes, and the algorithm that does the splitting runs inside one of your own consumer processes rather than on a broker. Three consumers reading three topics of five partitions each — fifteen partitions, plenty to go round — will end up holding six, six and three.

Nothing about that is a misconfiguration. It is what RangeAssignor, the assignor Kafka still selects by default, is documented to do: it divides each topic separately, and when the partitions of a topic do not divide evenly among the members, the first few members in sort order get the extra one. Do that across three topics and the same member is first three times.

The simulation below runs the real algorithms. Start by reading the default assignment — one member with double the work of another — then raise consumers in the group to 6 and watch a member end up with nothing at all, while fifteen partitions are in play and every other member holds three. Then switch to RoundRobinAssignor and see both problems disappear at once.

At six consumers the log says it plainly: each topic has five partitions and six members, so five members get one partition and the sixth gets a range of length zero — on every topic. There is no error, no warning and no metric for it. The member joins, heartbeats, is counted in kafka-consumer-groups --describe, and consumes nothing for as long as it runs.

The partition count is the ceiling, and it is set elsewhere

One partition, one member. That is the whole rule, and everything awkward about consumer groups follows from it. A partition is an ordered log, and Kafka's only ordering guarantee is per-partition, so two consumers reading the same partition concurrently would have no way to preserve it and no way to agree on a committed offset. So the broker never lets them.

Which makes the partition count your parallelism, and it is fixed by whoever created the topic. Raising it with kafka-topics --alter --partitions is possible and one-way: you can never go back down, and for a keyed topic you have just broken key affinity, because the producer's partitioner takes the key's hash modulo the partition count and every existing key now maps somewhere new. Records for the same key are suddenly in two partitions being read by two consumers in an unspecified order.

So the answer to "how many partitions?" is a capacity decision made before the topic exists, and the answer to "how many consumers?" is: never more than the partition count, and — with the default assignor — never more than the partition count of your smallest subscribed topic, or the surplus members do nothing.

There is one thing an idle member is good for. It is a warm standby. When another member dies its partitions are reassigned within session.timeout.ms, which the idle consumer is already inside the group to receive, so failover costs a rebalance rather than a pod start. That is a deliberate choice with a known cost, and it is not what most people who have scaled a Deployment to twelve replicas think they are buying.

Who computes the assignment

Under the classic protocol the broker does not decide anything about which member gets which partition. The sequence is:

1. FindCoordinator. The client hashes its group.id — Utils.abs(groupId.hashCode()) % offsets.topic.num.partitions, with that broker config defaulting to 50 — and the broker leading that __consumer_offsets partition is the group's coordinator. Type a different group.id into the simulation and the number changes; that is the whole of the coordinator election, and it is why two unrelated groups can land on the same overloaded broker.
2. JoinGroup. Every member sends its subscription and its list of supported assignors. The coordinator picks one that every member supports; if the lists do not intersect at all, the join fails with InconsistentGroupProtocolException. This is why setting partition.assignment.strategy on one consumer and restarting it changes nothing — the rest of the group still votes for the old one.
3. The leader assigns. The coordinator picks one member — the first to join — as group leader and sends it the full membership and subscription list. That client, your code's JVM, runs the assignor and returns a map. The coordinator relays it.
4. SyncGroup. Each member receives only its own slice, and starts fetching.

Two consequences worth keeping. The broker cannot tell you why an assignment looks the way it does, because it did not make it — there is no server-side log line to grep. And a bug in a custom assignor is a bug in a client that silently affects every other member of the group.

Kafka 4.x changes this, and it is worth knowing which world you are in. KIP-848 introduces a new consumer group protocol selected by group.protocol=consumer, in which the coordinator computes the assignment itself, incrementally, per member, with no group-wide barrier. In Kafka 4.1 the default for group.protocol is still classic, so everything above is what you are running unless you opted in. Under the new protocol partition.assignment.strategy is not read at all; the server-side assignor comes from the broker's group.consumer.assignors, which defaults to uniform,range, and a client can request one by name with group.remote.assignor. The new protocol is a much larger change than a swapped algorithm.

Choosing between the assignors

Set the group to 3 topics of 5 partitions and 3 consumers, and step through the three strategies.

• RangeAssignor gives 6, 6, 3. It divides each topic on its own, so the imbalance is multiplied by the number of topics. Its one virtue is co-partitioning: for two topics with the same partition count and the same subscribers, partition 3 of both lands on the same member, which is exactly what a join between two streams needs. That is why Kafka Streams cares about it and why it is still the default.
• RoundRobinAssignor gives 5, 5, 5. It pools every partition of every subscribed topic into one list and deals them out, so counts are within one of each other. It gives up co-partitioning to do it. Its documented weak spot is heterogeneous subscriptions: the javadoc's own example has three consumers subscribed to different topic sets and produces 1, 1 and 4.
• StickyAssignor is balanced like round-robin, and on top of that tries to leave each partition where it was. Set the event to worker-2 crashes and read "partitions that changed owner" under each strategy. Under RangeAssignor the departing member was holding 3 partitions and 6 change hands: the recomputation reshuffles three partitions that had no reason to move, and each of those is a consumer stopped and restarted from a committed offset for nothing. Under StickyAssignor the departing member was holding 5 and exactly 5 move — every survivor keeps everything it already had.

The default is a list, not a single value: [RangeAssignor, CooperativeStickyAssignor]. A group with that default runs RangeAssignor, because the coordinator picks the first strategy every member supports. The list exists so that you can migrate in two rolling restarts: the first deploys the pair to every member, the second removes RangeAssignor from the list, and at that point the only common strategy left is the cooperative one. Doing it in a single step means a period where half the group offers only the old strategy and half only the new, which is the InconsistentGroupProtocolException case.

Why restarts shuffle everything, and the fix

Set the event to every consumer is restarted, keep RangeAssignor, and leave group.instance.id unset. Most of the fifteen partitions change owner even though the membership is identical — same count, same processes, same subscription.

The assignors sort members by member.id, and a member.id is the client id with a UUID appended by the coordinator at join time. Consumers in the same group usually share a client id, so the sort is effectively a sort on random UUIDs. Restart them all and you get a fresh random order, and "the first few members get the extra partition" now means a different few. The RangeAssignor javadoc documents this exact scenario and calls the result "completely shuffled".

Now tick group.instance.id set on every consumer and repeat. Partitions moved: zero. A static member tells the coordinator who it is, the coordinator returns the same member.id it had before, the sort order is unchanged and the assignor is a pure function of its inputs, so its output is byte-identical. Static membership is worth reaching for on any consumer running under an orchestrator that restarts pods, though it has its own sharp edge: a genuinely dead static member is not removed until session.timeout.ms expires, so raising that timeout to survive restarts also lengthens your outage when a pod really does die.

Note what static membership does not fix. Try it with StickyAssignor: partitions still move. Stickiness is implemented by each member sending its previous assignment as userData in JoinGroup, and that data lives in the consumer's memory. A restarted process has none to send, so the assignor sees a group with no history and produces a fresh balanced assignment. Sticky protects you from other members' churn, not from your own restart; static membership protects you from your own restart. They solve different halves.

Reading it on a real group

One command answers almost everything:

kafka-consumer-groups --bootstrap-server localhost:9092 \
  --describe --group orders-processor

The output has one row per partition, with columns TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID. Read it three ways:

• Count the distinct CONSUMER-ID values. If it is fewer than the number of processes you are running, the difference is your idle members — and they will not appear in this output at all, because it lists partitions, not members. Use --describe --members for that; an idle member shows up there with #PARTITIONS of 0. That asymmetry is why idle consumers go unnoticed for months.
• Group the rows by CONSUMER-ID and count. Six, six and three is the RangeAssignor signature. If it is uneven and the counts are a multiple of your topic count, you are looking at per-topic division.
• Look at where the LAG is. Concentrated on the partitions of one CONSUMER-ID means assignment, not capacity; spread evenly across all of them means you are genuinely short of throughput. That distinction decides whether the fix is a config change or more partitions, and the lag metric alone cannot tell you which.

A row with a dash in CONSUMER-ID means the partition has no owner — the group is empty or mid-rebalance. To find the coordinator broker directly, --describe --state prints COORDINATOR (ID) alongside ASSIGNMENT-STRATEGY and the current STATE, which is the fastest way to confirm which assignor the group actually agreed on rather than which one you configured.

On the client side, the metric that catches imbalance without any CLI is kafka.consumer:type=consumer-coordinator-metrics,client-id=*'s assigned-partitions. Graph it per client id: a flat line at 0 on one pod is an idle member; a line at double the others is your busy one. It is a gauge that most dashboards never plot, and it answers in one glance what the CLI answers in three steps.

Next: what the group does when membership changes — the rebalance protocol and why it stops everyone — and the other thing partition ownership buys you, the right to commit an offset.