Static Membership Trades a Short Group Pause for a Long Local Outage

Setting group.instance.id does not stop your consumer group rebalancing. It stops exactly one thing: a static consumer does not send LeaveGroup when it closes, so the coordinator keeps its slot and its exact partition assignment reserved until session.timeout.ms runs out. A restart that finishes inside that window costs no rebalance. It costs something else instead, and the something else is not free.

What it costs is this: nobody takes over those partitions while the consumer is away. A dynamic consumer that leaves hands its partitions to the survivors within a rebalance and they keep being read. A static consumer that leaves takes its partitions with it. They are reserved, unowned and unread for the whole restart.

Below is a six-consumer group on 24 partitions doing a rolling restart, one pod at a time, each pod down for 40 seconds. The group is static and session.timeout.ms is Kafka's default of 45 seconds, so every restart finishes inside the window and the rebalance count is zero. Switch membership to dynamic and watch the rebalance count go to 12 while the hero number goes down. Then drag restart time per consumer to 5 seconds and switch back and forth again.

The number that moved, and the number that did not

At the settings it loads with, the static group reports 961.2 partition-seconds with no consumer and 0 rebalances. The same rolling restart with group.instance.id unset reports 405.5 partition-seconds and 12 rebalances. Static membership eliminated every rebalance and more than doubled the amount of time partitions spent unread.

The arithmetic is simple once you see where the time goes. Each of the six consumers owns four partitions. When a static consumer closes, those four partitions are held for it and nobody reads them for the full 40 seconds: six consumers × four partitions × 40 seconds ≈ 960 partition-seconds, which is the whole number. When a dynamic consumer closes, its four partitions are handed to the other five within about three seconds and are read for the rest of the 40. What the dynamic group pays instead is twelve group-wide barriers — one when each consumer leaves and one when each comes back — each stopping all 24 partitions for the length of the slowest member's rejoin.

So the trade has a crossover, and you can find it with the slider. Drag restart time per consumer down and switch between the two settings at each step. Static membership wins at 16 seconds (385.2 against 405.5) and loses at 17 (409.2 against 395.7). Below about 17 seconds of restart time the partitions come back before the rebalances would have finished; above it, the reserved-and-unread window is the larger cost. At 5 seconds static membership is more than three times better — 121.2 against 395.7. At 40 seconds it is more than twice as bad.

One honest caveat about the dynamic figure: it bounces between roughly 367 and 406 as you move the restart-time slider, because whether a member's next heartbeat lands just before or just after a rebalance starts shifts that barrier by up to heartbeat.interval.ms. The model spreads heartbeat phases evenly so the picture is reproducible; on a real cluster the phases are arbitrary and that variation is noise. The trend is what matters: the dynamic cost barely depends on how long a pod takes to restart, and the static cost is directly proportional to it.

And one thing partition-seconds does not capture, which is usually the actual reason to turn static membership on. A rebalance under an eager assignor revokes and reassigns partitions, and if your consumer holds per-partition state — a Kafka Streams task with a RocksDB store, an open output file, a warmed cache — that state is destroyed and rebuilt. Twelve rebalances can mean twelve rounds of state restoration that cost far more than the 405 seconds of measured downtime. If that is your situation, static membership is the right call and this simulation is measuring the wrong thing for you. If your consumers are stateless, it very likely is not.

The one line that does all of it

All of static membership on the client side is AbstractCoordinator.shouldSendLeaveGroupRequest:

private boolean shouldSendLeaveGroupRequest(CloseOptions.GroupMembershipOperation membershipOperation) {
    if (!coordinatorUnknown() && state != MemberState.UNJOINED && generation.hasMemberId()) {
        return membershipOperation == LEAVE_GROUP || (isDynamicMember() && membershipOperation == DEFAULT);
    } else {
        return false;
    }
}

isDynamicMember() is true when group.instance.id is unset. That is the entire difference. A dynamic consumer closing normally sends LeaveGroup and the coordinator removes it that instant; a static consumer closing normally sends nothing at all, and the coordinator finds out only when the session timer fires.

Which means the coordinator cannot distinguish a restart from a permanent shutdown, because in both cases it receives exactly the same thing: silence. Everything else in this lesson follows from that.

On the broker side the corresponding piece is a map from group.instance.id to the current member id. When a static member comes back it presents its instance id, the coordinator looks up the old entry, swaps in the new member id, hands back the identical assignment and does not touch the group epoch. Under the classic protocol that is a JoinGroup that does not trigger a rebalance. Under the KIP-848 protocol the same idea is encoded in the wire format: ConsumerGroupHeartbeatRequest.MemberEpoch is documented as "0 to join the group; -1 to leave the group; -2 to indicate that the static member will rejoin", and the coordinator's handler for -2 logs Static member … temporarily left the consumer group and keeps the assignment, resetting the partition epochs to zero so the returning process can pick them straight back up.

A consequence people trip over: the instance id must be genuinely stable and genuinely unique. Two processes claiming the same group.instance.id is not a merge; the older one is thrown out with FENCED_INSTANCE_ID (org.apache.kafka.common.errors.FencedInstanceIdException), and under the new protocol a heartbeat that arrives while the id is still held gets UNRELEASED_INSTANCE_ID. This is exactly what a deploy that starts the new pod before the old one has exited produces, so a Kubernetes Deployment with the default RollingUpdate strategy and a maxSurge above zero will fence itself. Static membership wants a StatefulSet, or a Deployment with strategy.type: Recreate, and the instance id has to come from the pod's ordinal or hostname rather than being generated at startup.

Two events static membership does not help, and one it makes worse

A crash. Switch what happens at t=0 to one consumer is SIGKILLed. Static reports 232.4 partition-seconds and a worst single partition of 49.5 s. Now switch membership to dynamic. The number does not move: 232.4, identically. There is no close() in a SIGKILL and therefore no LeaveGroup either way, so both groups learn about the death the same way — the session timer fires and the coordinator removes the member. Static membership does nothing for the case people most want it to help with.

It does something worse than nothing, though, because of what you had to do to make the deploy work. Put membership back to static and what happens at t=0 back to the rolling restart, then set restart time per consumer to 90 seconds, leaving session.timeout.ms at 45. Rebalances jump from 0 to 12 and the rolling restart costs 1447.7 partition-seconds, because every pod now outstays its slot: the coordinator holds its partitions dark for 45 seconds, gives up, rebalances the whole group, and then rebalances again when the pod finally returns as a brand-new member. That is the worst of both designs.

The documented fix is to raise session.timeout.ms above the restart time, and the broker will let you: group.max.session.timeout.ms defaults to 1800000, thirty minutes. Drag it to 300 seconds and the rolling restart is back to zero rebalances.

Look at the hero number while you do it. It goes from 1447.7 to 2161.2 partition-seconds. Removing the rebalances made the deploy worse, because the rebalances were the only thing that was giving those partitions to anybody: at a 45-second timeout the group gave up on each pod after 45 seconds and the survivors read its partitions for the remaining 45, and now nobody reads them for the whole 90. Six consumers × four partitions × 90 seconds is 2160, which is the number. Then look at the other two figures.

Cost of one crash goes from 232.4 to 1252.4 partition-seconds. Switch what happens at t=0 to the SIGKILL to see it on the chart, and the worst single partition reads 304.5 s. Five minutes of a partition with nobody reading it, after a process that is already dead. The session.timeout.ms you raised to cover a slow deploy is the same number that decides how long a dead consumer's partitions stay unowned. There is no second knob. Kafka detects a departed member exactly one way, and static membership works by making that detection slower on purpose.

Cost of one scale-down goes from 207.6 to 1227.6. Switch the event to one consumer is shut down for good and read the log: the consumer closed cleanly, it sent no LeaveGroup because it is static, and the coordinator held its four partitions for the whole session timeout before reassigning them. A dynamic group does the same scale-down for 27.6 partition-seconds with a worst partition of 2.26 s. Removing a replica from a static consumer group is, by default, a multi-minute partial outage.

That one has a proper fix rather than a trade-off. Since KIP-1092 the consumer can be told to leave on the way out:

consumer.close(CloseOptions.groupMembershipOperation(
        CloseOptions.GroupMembershipOperation.LEAVE_GROUP));

The enum has three values — LEAVE_GROUP, REMAIN_IN_GROUP and DEFAULT — and DEFAULT is documented as applying the default behavior — "For static members: The consumer will remain in the group" and "For dynamic members: The consumer will leave the group" — which is the behaviour everything above describes. Wire LEAVE_GROUP to your scale-down path and nothing else, or do it from outside the process with Admin.removeMembersFromConsumerGroup, which takes the instance ids directly. What you must not do is use LEAVE_GROUP in your normal shutdown hook, because then every restart sends a LeaveGroup and you have turned static membership off while keeping all of its costs.

What the new protocol does to this decision

If you are on group.protocol=consumer, or planning to be, two things change and one of them will stop your application from starting.

You cannot set session.timeout.ms at all. It is in CONSUMER_PROTOCOL_UNSUPPORTED_CONFIGS alongside heartbeat.interval.ms and partition.assignment.strategy, and checkUnsupportedConfigsPostProcess does not warn, it throws:

throw new ConfigException(String.join(", ", invalidConfigs) +
        " cannot be set when " + GROUP_PROTOCOL_CONFIG + "=" + groupProtocol.name());

A service that has been carrying session.timeout.ms=300000 for three years to make static membership work will fail at construction with session.timeout.ms cannot be set when group.protocol=CONSUMER the moment somebody flips the protocol. Remove it first, then flip.

The timeout becomes the broker's, and it is much smaller. Under the new protocol the coordinator tells the member what session timeout to use: group.consumer.session.timeout.ms, default 45000, clamped between group.consumer.min.session.timeout.ms (45000) and group.consumer.max.session.timeout.ms (60000). The value itself is not stuck cluster-wide — since group configs arrived it can be set on one group, as consumer.session.timeout.ms via kafka-configs.sh --entity-type groups --entity-name orders, and GroupConfig.validateValues checks it against the same 45000 and 60000 bounds. What is cluster-wide is the ceiling: raising 60 seconds means raising group.consumer.max.session.timeout.ms on the brokers, for every group. Either way it is the operator's config and not yours. If your pods take ninety seconds to restart, static membership under KIP-848 will not cover them and there is nothing you can put in your own configuration to change that.

The good news is that this matters much less than it sounds, because the reason you wanted static membership has largely gone. Under the new protocol a consumer joining or leaving does not stop the group: the coordinator publishes a new target assignment and each member walks to it on its own heartbeat, so the twelve barriers in the dynamic run above become a handful of partition handovers that nobody else waits for. Run the numbers for your own group before you carry group.instance.id forward. If your consumers are stateless, the honest answer on the new protocol is usually to drop static membership and take the faster failure detection back.

Reading it on a real cluster

Confirm the group is actually static. Look for the GROUP-INSTANCE-ID column. It is not a column that is sometimes blank: ConsumerGroupCommand sets includeGroupInstanceId only if at least one member reports a non-empty instance id, and prints a narrower table without the column at all otherwise. Present means static; absent means your group.instance.id never reached the client.

$ kafka-consumer-groups.sh --bootstrap-server broker:9092 \
      --describe --group orders --members
GROUP   CONSUMER-ID                    GROUP-INSTANCE-ID  HOST       CLIENT-ID          #PARTITIONS
orders  orders-3-a1b2c3d4-...          orders-3           /10.0.1.7  consumer-orders-3  4

Do not try to read it off the consumer id instead. Under the classic protocol ClassicGroup.generateMemberId is groupInstanceId.map(s -> s + "-" + UUID.randomUUID()).orElseGet(() -> clientId + "-" + UUID.randomUUID()): a static member's id is its instance id plus a UUID, and a dynamic member's is its client id plus a UUID. Neither is a bare UUID, and if your client id and instance id are the same string — which they usually are, both being the pod name — the two are indistinguishable. The column is the answer; the consumer id is not.

When the column is missing, the usual cause is setting the config on the wrong client or letting an empty string through, which ConfigDef.NonEmptyString rejects on group.instance.id but a templating system happily produces.

Watch for the log line that says it worked. On the broker, a static restart writes Static member … with instance id … temporarily left the consumer group and, on return, a line recording that the instance id was remapped to a new member id. If you see neither of those on a deploy but you do see the group entering PreparingRebalance, the restart took longer than the session timeout and you are paying for static membership without receiving it.

Measure per-partition lag, not group lag. This is the diagnostic that makes the whole trade visible, and it is the reason the cost is usually invisible on a dashboard. During a static restart, four partitions out of 24 have no consumer and their lag climbs; the other twenty are fine. A group-level average or a total-lag graph shows a gentle bump. The per-partition <code>records-lag</code>, or the max across partitions, shows a 40-second cliff on exactly the partitions of the pod you restarted. Alert on the max, never the mean.

Set session.timeout.ms from a measurement, not a guess. The number you want is the p99 of your pod's full restart: SIGTERM to first successful poll, including image pull on a cold node, JVM start, and whatever your readiness gate does. Add margin, then accept that number as your crash-detection time, because that is what you have chosen. If the answer is uncomfortable, the fix is a faster startup, not a longer timeout.

Give scale-down its own path. Whatever removes a replica — a kubectl scale, a KEDA scaler, an autoscaler — must either call close with LEAVE_GROUP or follow up with Admin.removeMembersFromConsumerGroup. Without it, every scale-down is a session timeout of partial outage, and if something scales your group up and down repeatedly you have built a machine for generating them.

Next: the protocol change that attacks the same problem from the other end and makes most of this trade-off unnecessary — a rebalance that is not a barrier, so a consumer joining or leaving stops nobody.