Kafka Consumer Lag

Consumer lag is one subtraction, done once per partition: the log end offset minus the offset your group last committed, counted in records. That makes it a position. A position cannot tell you whether you are catching up, falling behind or holding steady, and those are the only three things you actually want to know. A lag of two million that is shrinking clears in four minutes. A lag of four hundred that is not shrinking never clears.

So the useful quantity is not the lag but its rate of change, and the rate of change is produce rate − consume rate per partition. When those two are equal the lag freezes at whatever value it happened to reach, and it will sit there until something changes. That value carries no information about your system except the size of the last incident.

Below is a topic with six partitions and a group of four consumers, running for ten minutes. Producers write at a rate you set, each consumer processes records at a fixed cost per record, and offsets are committed on a timer. Start by dragging producer rate up slowly and watch the two numbers that matter — lag trend and time to drain — do something completely different from the headline lag.

At the defaults the group is comfortable. 4,000 records a second arrive, four consumers at 0.60 ms per record can take 6,667, and the lag sits at zero except for the teeth in the chart, which peak at 20,000 and are the commit timer rather than any real backlog. Now drag producer rate to 6,800. In aggregate the group is short by only 133 records a second, and the lag reaches 725,440 in ten minutes with a slope of +1,209 a second.

Those two numbers disagree because the log says something the aggregate hides: capacity is not pooled. Six partitions across four consumers means an assignment of 2/2/1/1, so partitions p0–p3 are each served at 833 records a second while p4 and p5 get 1,667. The partition that decides whether anything drains is the tightest one, and at 6,800 it is p0, taking 1,156 a second against 833 of capacity. The readout labelled tightest partition's headroom reads −323/s. That, not the aggregate, is the number with the sign that matters.

A big lag is a schedule; a flat lag is an outage

Put the producer back to 4,000 and set backlog at t=0 to 2,000,000. Ten minutes later the headline lag reads 939,200, falling at 661 records a second, and time to drain says 27 minutes. Nothing is wrong. The correct action is to wait 27 minutes.

Now go back to the 6,800 case: headline lag 725,440. Two readings within 30% of each other. One is a queue that clears itself before the meeting ends and the other has no upper bound at all, and no threshold on the lag value can separate them. The sign of the slope separates them instantly: −661 against +1,209.

The reverse pairing is just as sharp. Set consumers to 6 and the producer to 10,000. Every consumer now owns exactly one partition at 1,667 records a second, and p0 takes 1,700 of the 10,000. Total lag after ten minutes: 20,000, which is a rounding error on any dashboard, twelve seconds of data, and one number in the whole panel says anything is wrong — the slope, at +33 a second. It will pass a million in eight hours and it will never come back down.

Those two states are indistinguishable on a lag gauge and opposite in meaning. The three quantities that separate them are all derived, and none of them is what the tooling gives you:

• Headroom = consume capacity − produce rate, in records per second. Positive means any lag is temporary. Zero or negative means the current lag is a floor, not a peak.
• Time to drain = lag ÷ headroom. This is the only number worth saying out loud during an incident, and it is the one nobody has on a dashboard.
• Record age = how long ago the record at your committed offset was written. This is what "how far behind are we" means to everyone who is not an engineer, and it is the number a service-level objective should be written against.

Record age is worth dwelling on, because it is the one that makes lag comparable across topics. Push the busiest partition's share to 60% with everything else at its default. Partition 0 now takes 2,400 records a second against the 833 its consumer can give it, so it falls behind by 1,567 a second, and after ten minutes it holds 940,000 records — 6.5 minutes of data. Partitions 1 through 5 are at exactly zero. The worst partition's share of lag readout says 100%. The group total is one number describing one partition that is six and a half minutes stale and five that are current, and averaging it across six partitions would make it look like a minute.

The number is against the committed offset, and there are three of them

Drag auto.commit.interval.ms from 5,000 down to 100 with everything else at its default. The peak of the chart falls from 20,000 records to 4,000, and the sawtooth disappears. No consumer got faster. No producer slowed down. You changed how often a number is written to __consumer_offsets, and the amplitude of the teeth is exactly produce rate × commit interval: 4,000 × 5 seconds, then 4,000 × 0.1.

This is worth being precise about, because "lag" names at least three different measurements and people quote them interchangeably.

• kafka-consumer-groups.sh --describe reports LOG-END-OFFSET − CURRENT-OFFSET, where CURRENT-OFFSET is what the group last committed. It therefore includes everything you have read but not yet committed, and its amplitude is roughly commit interval × consume rate even on a perfectly healthy group. It is also the only one of the three you can read without touching the consumer, which is why every exporter uses it.
• The consumer's own records-lag metric — and records-lag-max and records-lag-avg beside it — is computed from the client's read position against the high watermark, per assigned partition. It has no sawtooth, because committing is not part of it. Two numbers, both called lag, differing by however much you have read since your last commit.
• Record age or "time lag" is not produced by Kafka at all. Tools like Burrow and the various lag exporters synthesise it by looking up the timestamp of the record at your committed offset. It is the only one in the list denominated in a unit anyone outside the team understands.

There is a fourth metric in the same family that almost nobody watches and everybody should: records-lead-min. Lead is your position minus the partition's log start offset — how many records sit between you and the oldest record still retained. Lag tells you how far you are from the head. Lead tells you how close you are to the tail, and the tail is where retention deletes records you have not read yet. A lead approaching zero means you are about to lose data permanently, and no lag alert will fire while it happens.

Zero is not a safe reading

Tick the producer dies at 5:00. Lag drains to zero and stays there for five minutes. Every alert is green. Nothing is being delivered, because nothing is being produced, and offset lag has no way to express the difference between "caught up" and "there is nothing to catch up on".

This is the most common false negative in stream monitoring, and it is structural rather than accidental: lag is defined relative to the log end offset, so it is silent about whether the log end offset is moving. Two metrics catch it and both are cheap. Alert on the age of the newest record in the topic, which is upstream liveness. Or alert on records-consumed-rate falling to zero on a consumer that is supposed to be busy, which catches a stalled consumer and a stalled producer with the same rule.

There is a worse version of a zero reading, and it is the reason auto.offset.reset deserves a look before an incident rather than during one. If a group's committed offset for a partition is missing — a brand-new group, a partition added by an expansion, or an offset that has expired — the consumer has to choose where to start, and with the common setting of latest it starts at the end of the log. Lag goes to zero instantly and everything in between is skipped, silently and irreversibly.

Offsets expire on offsets.retention.minutes, which has defaulted to 10080 — seven days — since Kafka 2.0, up from a day before that. The expiry condition is narrower than people fear and still catches teams out: for a subscribed consumer group, the clock starts when the group loses all its members, so a running group's offsets never expire, and a group that is stopped over a long holiday shutdown loses them. That is exactly the moment a latest reset does the most damage.

Non-zero is not an unsafe reading either

Tick transactional producer. Total lag settles at exactly 6 — one per partition — with a slope of zero, and it refuses to reach zero however long the run goes on and however much capacity you add.

A transactional producer ends each transaction by writing a control record — a commit or abort marker. It is a real record in the log and it occupies a real offset, so the log end offset advances past it. Kafka's message format specification is explicit that "control records should not be returned to applications", so a consumer will never receive it and can never have a record to commit past it. If your loop commits only when poll() returned something, which is the usual shape, the committed offset stops one short and the LAG column reports 1 per partition forever.

Two related readings from the same feature. A read_committed consumer reads only up to the last stable offset, which the client documentation defines as "the one less than the offset of the first open transaction", and states plainly that such consumers "will not be able to read up to the high watermark when there are in flight transactions". So a long-running open transaction upstream produces lag that is entirely correct behaviour and entirely outside your control. And an aborted transaction's records occupy offsets that are filtered out on the way to your application, so the LAG column counts records you are guaranteed never to receive.

The practical rule that falls out: never alert on lag > 0. Alert on lag above a floor you have measured, or better, on record age and on the sign of the trend.

Why more consumers is usually the wrong first move

Set the producer to 9,000 and the consumers to 4. The lag reaches 1,607,200 and climbs at 2,679 a second. Drag consumers to 5: the slope halves to 1,357, but the tightest partition's headroom does not move at all, because p0 is still sharing its consumer with p1. At 6 every partition gets its own consumer, headroom flips to +137 a second and the lag is zero. At 7, 8, 9 and 10, nothing further happens — the topic has six partitions, a partition goes to exactly one member of a consumer group, and the extra consumers own nothing. The chart is byte-identical from 6 onwards.

That is the ceiling from the partitions lesson seen from the other side, and it is why "scale the consumer deployment" is a fix with a hard stop you cannot see from a lag dashboard. Past that point the lever is the processing cost per record, or partitions you cannot remove later.

Now the case where adding consumers makes lag worse. Put the producer back to 4,000, consumers to 4, and tick one consumer stops polling at 3:00. Total lag reaches 278,880 and rises at 664 a second, and the per-partition table puts 100% of it on p5 — the one partition that member owned. The other five are at zero and 0 seconds. Deploying more consumers changes nothing, because the coordinator will not take a partition away from a member that is still heartbeating: the reassignment waits for session.timeout.ms or max.poll.interval.ms, and in this run neither expires. Worse, each new member triggers a rebalance, and under the eager protocol every consumer in the group hands back every partition it owns and waits — so the five healthy partitions stop consuming too, and the total lag rises faster than before you intervened.

The shape of the lag chart tells these apart before you touch anything:

• A straight line rising on every partition — you are short of capacity. The slope is exactly your deficit in records per second.
• A straight line rising on one or two partitions, flat elsewhere — a stuck consumer or a hot key. The per-partition table separates them: compare the "in" rate against the "out" rate on the affected rows.
• A staircase — flat, jump, flat, jump — rebalances. Each riser is the group stopped; each tread is it running normally.
• A sawtooth with a flat floor — nothing at all. That is the commit timer.

What to run and what to graph

The one command, and the columns that matter in it:

kafka-consumer-groups.sh --bootstrap-server b:9092 --describe --group orders

TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG     CONSUMER-ID
events  0          48120993        49535871        1414878 consumer-orders-1-a1b2
events  1          51002144        51002145        1       consumer-orders-2-c3d4
events  2          49883201        49883202        1       -

Read it column by column rather than looking at the LAG total. Partition 0 is the real problem. Partitions 1 and 2 are at the transactional floor of 1 and are fine. Partition 2 has a dash in CONSUMER-ID, which means no member currently owns it — either the group is mid-rebalance or there are fewer consumers than partitions. Run the command twice, thirty seconds apart: the difference between the two LAG columns is the derivative, and it is worth more than either reading.

In Prometheus, with kafka_consumergroup_lag from any of the common exporters, the three queries to have saved:

• max by (topic, partition) (kafka_consumergroup_lag{group="orders"}) — the maximum, never the sum. A sum across forty partitions hides one dead one indefinitely.
• deriv(kafka_consumergroup_lag{group="orders"}[10m]) — the slope in records per second. Alert when this is positive and sustained, which fires on a group falling behind slowly and stays quiet through a large backlog that is draining.
• kafka_consumergroup_lag / on(topic, partition) rate(kafka_topic_partition_current_offset[5m]) — lag divided by produce rate, which is record age in seconds, and it is the one to put an objective on. "Ninety-nine per cent of the time, no partition is more than sixty seconds behind" is a statement a product owner can evaluate. "Lag is under 50,000" is not.

From the consumer itself, in the consumer-fetch-manager-metrics group: records-lag-max for the client's own view without the commit sawtooth, records-lead-min for how close you are to losing unread data to retention, and records-consumed-rate so that a consumer which has silently stopped is distinguishable from a topic which has silently gone quiet. Those three plus record age cover every case on this page.

Next: the number the whole measurement is subtracted from, and the ways it goes wrong on its own — offset commits. And the failure that produces the staircase shape, rebalancing.