Kafka Partitions

A partition is two things at once, and they pull in opposite directions. It is the unit of parallelism — a consumer group hands each partition to exactly one member, so the partition count is a hard ceiling on how many consumers can do work. It is also the unit of ordering — Kafka guarantees the order records were written only inside one partition. Turning the first dial up is how you break the second guarantee, and Kafka will not let you turn it back down.

The reason is one line of the producer. BuiltInPartitioner.partitionForKey is Utils.toPositive(Utils.murmur2(serializedKey)) % numPartitions: hash the key's bytes, mask off the sign bit, take the remainder. Nothing in that expression remembers where the key went last week. Change numPartitions and you change the divisor, and most keys land somewhere else — while every record they have already written stays exactly where it was.

Below is a topic with 6 partitions, 240 distinct keys and a consumer group of 4. The hashing is real: the simulation runs a port of Kafka's murmur2, checked against Kafka's own test vectors, so the partition each key lands in is the partition your producer would pick. Tick apply the expansion to run kafka-topics.sh --alter --partitions 12 and watch what happens to the keys.

Tick the expansion box. Six partitions become twelve, and 53% of the keys move to a different partition — 127 of 240, on the gentlest expansion there is. Every one of those keys now has its old records in one partition and its new records in another, and the two are read by different consumers at different rates. Follow user-0: murmur2("user-0") is 637506763, which is 1 modulo 6 and 7 modulo 12. Its first three records are in partition 1, behind a 20,000-record backlog. Its next three are in partition 7, which was created empty a moment ago. The consumer on partition 7 reaches the head of its log immediately; the consumer on partition 1 needs ten seconds. For ten seconds this key's history is delivered backwards, and the trace panel lists it in the order it actually arrives.

The ceiling, and why extra consumers are silent about it

Untick apply the expansion first — the free preview left it on, and with it on the topic has 12 partitions rather than the count you are about to set, which is not the ceiling this section is about. Then set partitions now to 4 and drag consumers in the group from 1 to 16. Effective parallelism climbs to 4 and then stops dead. At 16 consumers, twelve of them own nothing.

This is not a tuning limit; it is what a consumer group is. Kafka's design document puts it in one sentence: a topic is "divided into a set of totally ordered partitions, each of which is consumed by exactly one consumer within each subscribing consumer group at any given time". One consumer per partition is what lets a group's entire progress be stored as one integer per partition rather than per-message acknowledgements. The parallelism ceiling is the price of that design, and the operations documentation states the consequence plainly: "the partition count impacts the maximum parallelism of your consumers".

What makes this expensive to discover is that the extra members do not fail. They join the group, they are assigned an empty set, they heartbeat happily, their liveness probes pass and their dashboards are green. A deployment scaled from 4 replicas to 16 will show 16 healthy pods and exactly the throughput of 4. The only place it shows is kafka-consumer-groups.sh --describe, where the extra members appear with no partitions beside them.

Switch the assignor between RangeAssignor and RoundRobinAssignor with 7 partitions and 4 consumers. Range hands out contiguous blocks and gives the first 7 mod 4 = 3 members an extra one, so c0 gets p0–p1, c1 p2–p3, c2 p4–p5, c3 p6. Round robin deals them one at a time, so c0 gets p0 and p4, c1 gets p1 and p5, c2 gets p2 and p6, c3 gets p3. Same counts, different partitions, and the per-consumer traffic shares differ because different keys land together. Neither invents a fifth worker. The assignor decides who gets what; it has no say in how many there are to give.

The consequence for capacity planning is that the partition count is a decision about your maximum future consumer fleet, taken at topic creation, when you know least. That is why teams over-provision — and why they cannot over-provision freely, because each partition is a directory of segment files on a broker, with its own index, its own leader election and its own entry in cluster metadata. The trade-off is real in both directions, which is the whole reason anyone ends up needing to expand a live topic.

The expansion is one-way

Set partitions now to 12 and --alter --partitions to 6 — below the current count — then tick the box. The controller refuses:

InvalidPartitionsException: The topic orders currently has 12 partition(s);
  6 would not be an increase.

That message comes from ReplicationControlManager.createPartitions in the KRaft controller — the component that owns cluster metadata since ZooKeeper was removed — and not from the command-line tool. There is no client library, no admin API and no flag that reaches past it. The Kafka documentation's own sentence is "Kafka does not currently support reducing the number of partitions for a topic."

The reason is that a partition is a durable log, not a routing rule. Deleting partition 11 would mean deleting or relocating every record in it, and every consumer group's committed offset for it, and every producer's idempotence state keyed to it. There is no correct place to put those records: appending them to another partition would put them after records written later, which is precisely the guarantee the partition exists to provide.

So the only way back is a new topic with the count you want, a copy job, and a cutover for every producer and consumer. Plan the expansion as a migration even when the command is one line, because reverting it is a migration whether you planned one or not.

What "same key, same partition" actually promises

Put the counts back to 6 and 12 and untick the expansion. The claim everyone builds on is that records with the same key go to the same partition, so a single consumer sees one key's events in order, so per-key state machines work. All of that is true, and every clause of it is conditional on numPartitions holding still.

The mapping is stateless. There is no table of key assignments anywhere in a broker, and no consistent hash ring with virtual nodes of the sort a sharded database uses to keep most keys in place when a shard is added. It is hash % n, recomputed on the producer for every record. Tick the expansion and read the hero number: at 6 to 12, 53% of keys move — 127 of the 240. That is not a bug in the choice of hash — a good hash is exactly what makes the remainder change when the divisor does.

There is an exact law here, and it is worth carrying around. A key keeps its partition only when h % p1 and h % p2 are the same number. That requires h to be congruent to some r < p1 modulo both counts, which by the Chinese remainder theorem pins h to one residue class modulo lcm(p1, p2) for each of the p1 possible values of r. So the fraction of keys that stay is gcd(p1, p2) / p2.

Test it in the panel. 6 to 12: gcd is 6, so 50% stay, and the readout says 53% moved. 6 to 18: 33% stay. 6 to 24: 25%. 6 to 7, which shares no factor: 1 in 7, and 89% of your keys move for one extra partition. Two things fall out of that formula. Doubling is the best expansion available — no other choice keeps more keys — and even doubling moves half of them. And an expansion to a coprime count, which is what you get by nudging 6 to 7 because a dashboard said the partitions were a bit hot, is close to the worst thing you can do.

Now follow what happens to one of the keys that moved. Its old records are in the partition it used to hash to. Kafka does not move them: the documentation says the broker "will not attempt to automatically redistribute existing data". Its new records are in the partition it hashes to now. Those are two independent logs, and Kafka's ordering guarantee says nothing at all about the relative order of two partitions.

In the steady state that is a small risk — two consumers, two slightly different latencies, a race measured in milliseconds. At the moment of the expansion it is not small, and the reordering window readout is why. The partitions created by an expansion are empty. Whatever backlog the existing partitions are working through, the new ones have none, so a consumer assigned to a new partition is at the head of its log instantaneously. Set the backlog to 20,000 records and the window is ten seconds; set it to 200,000, which is one bad afternoon on a busy topic, and it is 100 seconds. For that entire window, for every key that moved onto a new partition, later records are delivered before earlier ones.

Drag follow one key along and watch the trace panel switch between three verdicts: the key stayed put and is fine; the key moved to another old partition and happens to be fine; the key moved to a new partition and its updates arrive backwards. What a per-key state machine does with the third case depends entirely on what it is — a last-write-wins cache serves stale data and then corrects itself, an account balance built by applying deltas is simply wrong, and a deduplicating consumer that tracks "highest sequence seen" silently discards the older records forever.

Two smaller consequences show up in the same moment. The expansion changes the topic's metadata, which triggers a rebalance of the whole group — so the reordering window overlaps with a period in which partitions are being revoked and reassigned. And a brand-new partition has no committed offset for your group, so where consumption starts is decided by auto.offset.reset. With the common setting of latest, anything produced to the new partitions between their creation and your consumers noticing them is skipped. The Kafka documentation calls this out and now recommends auto.offset.reset=by_duration:<duration> instead, sized above metadata.max.age.ms — five minutes by default — for classic consumer groups.

Where more partitions does not buy anything

Skew. Drag key skew to 1.2 with 6 partitions and no expansion. The busiest partition holds 35.0% of the traffic where an even split would be 16.7%. Now expand: at 12 partitions it is 32.5%, at 24 it is 26.2%, and at 48 — the top of the slider, eight times the original count — it is 25.7%. It stops there. The floor is 25.6%, which is what the single hottest key carries by itself, and no partition count gets below it, not even one partition per key: that key hashes to exactly one partition and no number of partitions splits it. You octupled the count to buy the last nine percentage points and there is nothing further to buy. This is the same wall as a skewed shuffle in Spark, with one difference that matters — Spark can repartition the data with a different scheme on the next run, and a Kafka topic's history is already written.

The fix here is never the count. It is the key: a composite key like customer-7741#3 that spreads one customer over a few partitions, accepting that you have given up ordering for that customer on purpose rather than by accident. Or a custom partitioner.class, which is the one supported way to control the mapping — and the only way to make an expansion preserve affinity, by keeping a key's placement stable across the change.

A slow consumer. If one member is CPU-bound on your processing code, splitting its partition in two does not make it faster; it makes two partitions that will be assigned to the same member until you also add members. Watch the per-consumer share column while you raise the partition count without raising the consumer count: the distribution gets smoother and the total does not move.

Ordering you did not know you needed. The most expensive version of this is a topic where nobody set a key at all. With a null key the producer does not hash anything — partitionForKey is not called and the built-in partitioner picks a partition by its own sticky batching logic — so there is no per-key ordering to break and the expansion is harmless. Teams read that as "expansions are safe", and then apply the same reasoning to the topic next door that does use keys.

Internal topics. Never run --alter --partitions against __consumer_offsets, __transaction_state, __share_group_state or __cluster_metadata. The Kafka documentation's word is that doing so "can break coordinator mapping logic, cause state inconsistencies, and lead to data corruption or system failures", and the mechanism is exactly the one on this page: a group's coordinator is chosen by hashing the group id modulo the partition count of __consumer_offsets, so changing the count moves every group's coordinator away from the partition holding its committed offsets.

Doing it safely, and checking it afterwards

Before you expand, answer one question: does anything downstream depend on per-key ordering? Not "do we use keys" — plenty of topics set a key purely for log compaction — but does a consumer hold state per key, or write to a store keyed the same way, or deduplicate on a sequence number.

If the answer is no, expand and move on. If the answer is yes, the choices in decreasing order of safety:

• New topic, new count, dual write, cut over. Produce to both for a period, let consumers drain the old topic completely, then switch. This is the only option that never delivers a key's records out of order, because no key ever spans two topics that are both live for the same consumer.
• Expand during a genuine quiet period, with the backlog at zero. Set the simulation's backlog to 0 and read the reordering window: it goes to zero too. The risk that remains is the ordinary cross-partition race, which is milliseconds rather than minutes. Verify the backlog is actually zero rather than assuming it — that is what the lag check below is for.
• Stop the producers, drain to zero, expand, restart. The brute-force version of the same idea, and the one most teams can actually arrange at 3 a.m.
• A custom partitioner that preserves the old mapping for existing keys and uses the new count only for keys first seen after the change. Correct, and you now maintain a key-placement table forever.

Afterwards, three commands settle whether it went as intended.

kafka-topics.sh --bootstrap-server b:9092 --describe --topic orders lists every partition with its leader and ISR. The partitions added by an expansion are at the end of the list and their leaders are assigned by the controller, so check that they did not all land on the same broker — an expansion that puts four new partitions on one broker has moved your hot spot rather than removed it.

kafka-consumer-groups.sh --bootstrap-server b:9092 --describe --group g prints one row per partition with CURRENT-OFFSET, LOG-END-OFFSET, LAG and the CONSUMER-ID that owns it. Two things to read here. Any row with a dash in the consumer column is a partition nobody is consuming. Any member that appears with no rows at all is a replica you are paying for and not using — that is the parallelism ceiling, visible. And the per-partition LAG column is the honest version of consumer lag: a total across partitions hides the one partition that is not moving.

Then verify the mapping rather than trusting it. For any key you care about:

kafka-console-consumer.sh --bootstrap-server b:9092 --topic orders \
  --partition 8 --from-beginning \
  --property print.key=true --property print.partition=true

If a key you expected to find only in partition 2 also appears in partition 8, you are looking at a split history, and the timestamps on the two halves will tell you exactly when the expansion happened.

Next: the number everyone watches to decide whether any of this worked, and what it actually measures — consumer lag. Underneath both sits the mechanism that hands partitions to members in the first place, the consumer group.