Chunked Prefill

A GPU runs one forward pass at a time. If that pass is a 32,000-token prompt belonging to one new user, then for its whole duration every other user in the batch receives nothing — not a slow token, no token. Their streams stop for a second, and none of them sent a long prompt.

Two terms first. Prefill is the pass that reads a whole prompt and produces the first output token; it puts every prompt token through every matrix at once. Decode is a pass that produces one further token per sequence. A modern server re-forms its batch before every pass rather than per request — that is continuous batching — which means the question "what goes in this pass?" is answered thousands of times a second, and the answer decides who waits.

The scheduler's only real currency is a token budget: the maximum number of tokens it will put through one pass, spelled max_num_batched_tokens in vLLM. Thirty running generations contribute one token each. A pending 8,192-token prompt would like to contribute 8,192. Whether it is allowed to, or is made to arrive in pieces, is the entire subject here.

Below is a scheduler running iteration by iteration. A prompt arrives at time zero while some number of users are already generating. Watch the second lane — one existing user's token stream — as you change the policy.

At the defaults — 32 people generating, an 8,192-token prompt arriving — the unchunked policy costs one pass of 208 ms. During it the other 32 users get nothing, so the gap between their tokens goes from 9.8 ms to 217 ms: a 22× spike, delivered to people who were mid-sentence. Switch to chunked and that becomes five passes of 47 to 57 ms; the same users keep receiving tokens throughout — 160 of them — and the person who sent the prompt waits 217 ms for their first token instead of 208.

Why a chunk can be nearly free

The reason this works at all is that prefill and decode are short of different things. A decode pass reads the entire weight set — 15.3 GiB for this model — plus every resident sequence's cache, and then performs about two arithmetic operations per weight it read. It is bandwidth-bound by a wide margin: 24.3 GiB moved and 563 GFLOP performed, which is 9.8 ms of memory time against 0.8 ms of arithmetic, so the tensor cores are idle for 92% of it. A prefill chunk is the opposite. Its tokens go through the same weights the pass has already fetched, so it adds arithmetic and almost no traffic.

That gives a quantity the simulation computes for you and no configuration guide mentions: the free-ride capacity. It is the number of prompt tokens whose arithmetic still fits inside the time the pass was going to spend waiting on memory anyway.

free tokens ≈ ( bytes the decode pass moves ÷ bandwidth ) × peak FLOP/s ÷ ( 2 × parameters )

That formula gives 444 at the defaults. The readout says 378, and the difference is honest: the chunk also has to read the keys and values of everything already prefilled and do attention against them, which the simulation charges for and the back-of-envelope version does not. Either way it is a few hundred tokens, and it is the number the budget should be set near.

Watch the colour of the passes as you cross it. At a budget of 256 the chunks are 224 tokens, under the free ride, and the passes stay amber: 10.0 ms each against a 9.8 ms baseline, with 224 prompt tokens processed for 0.2 ms. At 512 the chunks are 480 tokens, just over, and the passes turn magenta at 14.4 ms. At 4,096 they are 4,064 tokens — ten times the free ride — and each pass costs 110 ms while all 32 other users wait. The colour change in that lane is the entire tuning problem, and it happens at a number you can compute rather than guess.

This is what Sarathi-Serve, the paper that introduced the technique, calls stall-free batching, and it named the failure it removes: a generation stall, which they define as what happens "when one or more prefills are scheduled in between consecutive decode iterations of a request". Their measured example is worth carrying around: on Falcon-180B across eight A100s, a single 4,000-token prompt takes about 1,150 ms as its own pass, against about 200 ms for a decode-only batch. Every user in that batch sees a one-second hole in their output because somebody else pasted a document.

The budget points in two directions

Now find the setting that is good for everyone. There isn't one, and the simulation will not let you pretend otherwise. Leave the policy on chunked and walk the token budget down from 16,384:

• 16,384 — one pass. First token at 208 ms, worst gap 208 ms, 21× the normal one, one token delivered to each running user in that window. Chunking is switched on and doing nothing, because the whole prompt fits in one budget.
• 2,048 — five passes. First token at 217 ms, worst gap 57 ms, 160 tokens delivered meanwhile.
• 512 — eighteen passes. First token at 230 ms, worst gap 14.4 ms, 576 tokens delivered.
• 256 — thirty-seven passes. Worst gap 10.4 ms, within 6% of the undisturbed rate. First token now at 379 ms — 82% worse — because the 15.3 GiB weight set has been read thirty-seven times to move one prompt through, 945 GiB of traffic in place of 39 GiB.

The interesting part of that list is that it is not a straight line. The stall above the undisturbed 9.8 ms baseline is 198 ms at a budget of 16,384 and 4.6 ms at 512: going there removes 98% of it and costs 11% of time to first token. Halving the budget again to 256 removes most of the remaining 2% and costs another 65% of time to first token. Almost all of the benefit is available almost free, and then the price rises steeply — which is the shape of every good tuning parameter, and the reason "smaller is safer" is wrong here.

The boundary where chunking stops being nearly free. Drag users already generating down to 1. The free-ride capacity only falls from 378 tokens to 264, because most of it is paid for by the weight read rather than by the caches — but there is now almost nothing riding along: at a budget of 256 the prompt takes 224 ms instead of 208, and the single running user receives 33 tokens during it instead of none. Chunking is not creating throughput; it is filling time that was already being spent. When the machine is empty there is nothing to fill, and the extra weight reads are pure loss. This is why the technique earns its keep on a loaded server and costs you a little on an idle one, and why an offline batch job that only does prefill should use the largest budget it can afford.

There is also a floor that is not a tradeoff but an error. Set the budget below the number of running users and the log turns magenta: with 32 users and a budget of 16, there is no room for the prompt after the decodes take their tokens, and nothing can ever finish. vLLM rejects the configuration at startup rather than deadlocking — max_num_batched_tokens must be greater than or equal to max_num_seqs.

Checking it on a real system

In vLLM V1, chunked prefill is on by default: "in V1, chunked prefill is enabled by default whenever possible", with the scheduler taking decodes first and filling the remaining budget with prefill. The number that matters is the budget, and its default depends on hardware and on how you launched:

--max-num-batched-tokens 8192     # OpenAI-compatible server on H100/H200
--max-num-batched-tokens 2048     # OpenAI-compatible server on A100 and below
--max-num-batched-tokens 16384    # offline LLM() class, H100/H200
--long-prefill-token-threshold N  # cap one request's share of a single pass

Those are read from vLLM's own defaults table, and the A100 case has a comment next to it in the source: large budgets measurably reduced throughput on that GPU. The lower ridge point of an A100 is the reason — fewer tokens are needed before a pass becomes compute-bound, so the free ride runs out sooner.

Measure two latencies separately, because a single average hides exactly the thing you are looking for. Time to first token is the prompt's problem. Time between tokens, sometimes inter-token latency, is everyone else's, and the statistic that matters is its p99, not its mean: a stall is by definition rare and enormous. In vLLM's Prometheus metrics those are vllm:time_to_first_token_seconds and vllm:time_per_output_token_seconds, both histograms. If the second one has a fat tail while the first looks fine, you are looking at prefills landing on top of generations, and the budget is the dial.

A useful diagnostic before you change anything: correlate the p99 of time per output token against the arrival of long prompts. If the spikes line up with your longest requests, chunking or a smaller budget will fix it. If they line up with Preemptions appearing in the steady-state log line instead, you have a KV cache capacity problem wearing the same symptom, and shrinking the token budget will make it worse by keeping more requests resident for longer.

Two interactions worth knowing before you turn the dial. Chunking composes badly with a cold prefix cache in one specific way that has a bug report attached: each chunk is a separate scheduling decision, and the first chunk can only reuse cached blocks that are aligned to the block boundary, so a chunked prefill can miss reuse that an unchunked one would have found. And a chunk's attention has to read the keys and values of every chunk before it, so the total cache traffic for one prompt grows with the number of chunks — Sarathi measures at most about 25% overhead at a chunk size of 512, which is the same effect the simulation charges you for in its traffic line.

Finally, the sanity check that costs nothing: divide your p95 prompt length by your token budget. That is how many passes a typical prompt will occupy, and therefore how many of your users' tokens it will delay. If the answer is 1, you have not enabled chunking, whatever the config says.

Next: what the passes are competing for in the first place, the KV cache; the scheduler this policy lives inside, continuous batching; the block allocator that lets a half-prefilled request hold only what it has produced, paged attention; and the cheapest prefill of all, the one you skip entirely with prefix caching.