Continuous Batching Is Not a Bigger Batch

A serving engine does not fill a batch and then run it. Before every single forward pass it decides again who is in the batch — retiring whoever just finished, admitting whoever fits, and sometimes throwing a running sequence out to make room. The batch is not a container you size. It is a decision the scheduler makes thousands of times a second, and the setting you think controls it is usually not the thing that binds.

Three terms, because everything below is built from them. A forward pass is one trip through the model's weights; the GPU runs exactly one at a time. Prefill is the pass that reads a whole prompt and produces a first output token. Decode is a pass that produces one further token for each sequence in it. Every sequence in a batch carries a KV cache — the keys and values, one pair per token per layer, that attention needs in order not to re-read the whole context — and that cache is what a sequence occupies while it is resident on the card.

Static batching is the mental model most people bring: collect n requests, run them together, return them together, collect the next n. It is what model.generate() on a padded tensor does. The problem is that requests in one batch do not finish together. In the workload below, a request that wants 50 tokens sits in the batch next to one that wants 781, and its slot is dead weight for 731 passes. Continuous batching — Orca's paper calls it iteration-level scheduling, and NVIDIA's TensorRT-LLM calls it in-flight batching — moves the unit of scheduling from the request to the pass, so that slot is refilled the moment it comes free.

Below is a scheduler serving 256 queued requests to exhaustion. It is already set the way you would run it: membership decided per pass, cache blocks allocated as they are needed, and --max-num-seqs at 256 so nothing is artificially held back. Read the second readout before you touch anything, then take --max-num-seqs down.

At the settings it loads with, the readout that matters is the second one. --max-num-seqs says 256. The largest batch the scheduler ever assembled was 83. The other 173 slots were never contended for, because the KV cache ran out first — and reaching for them cost 38 preemptions and 49,535 prompt tokens that had to be computed a second time.

Now drag --max-num-seqs down to 64. Throughput goes up, from 2,911 to 3,036 tokens per second, preemptions go to zero, and the bars turn from amber to teal. You made the configured batch smaller and the machine got faster. That is not a rounding error and it is not a quirk of this simulation: it is what happens when a scheduler is allowed to admit more sequences than its cache can carry.

Then put --max-num-seqs back at 64 and switch the unit of scheduling to the request, with allocation on reserve up front — static batching, the thing the mental model describes. 3,036 becomes 1,579. Same GPU, same model, same 256 requests, same slot cap. The lanes tell you where it went: requests that arrived at time zero are still sitting grey thirty seconds later, waiting for a batch they were not selected into to finish draining.

What the scheduler does on one step

vLLM's V1 scheduler makes the whole decision in one function, schedule(), and it runs once per forward pass. It has two phases and the order matters more than anything else in this lesson.

Phase one is the requests already running. It walks them in arrival order and asks each how many tokens it wants next. A sequence that is decoding wants one. A sequence still working through its prompt wants the rest of it, capped by the per-pass token budget — that is chunked prefill, and it is why there is no separate "prefill phase" in the scheduler at all. The source says so directly:

# NOTE(woosuk) on the scheduling algorithm:
# There's no "decoding phase" nor "prefill phase" in the scheduler.
# Each request just has the num_computed_tokens and num_tokens_with_spec.
# At each step, the scheduler tries to assign tokens to the requests
# so that each request's num_computed_tokens can catch up its
# num_tokens_with_spec.

For each request it then tries to allocate the cache blocks those tokens will need. If the allocation fails, it does not queue, wait or shrink the batch gracefully. It evicts somebody:

while True:
    new_blocks = self.kv_cache_manager.allocate_slots(request, num_new_tokens, ...)
    if new_blocks is not None:
        break                      # the request can be scheduled
    # The request cannot be scheduled.
    # Preempt the lowest-priority request.
    preempted_req = self.running.pop()

self.running.pop() takes the last element, and under the default first-come-first-served policy the running list is in arrival order. So the victim is the most recently admitted sequence — newest out first, which is what keeps the oldest requests from starving. The PagedAttention paper states the rule in words: the earliest arrival is served first and the latest preempted first.

What eviction costs is the part people meet as a surprise. The preempted request's blocks are freed, and then:

request.status = RequestStatus.PREEMPTED
request.num_computed_tokens = 0
...
self.waiting.prepend_request(request)

num_computed_tokens = 0. Not "resume from where you were" — every token of prefill this request had already done is discarded, and when it is readmitted it prefills from the beginning. That is where the "prompt tokens recomputed" readout comes from. At the default settings it is 49,535 tokens on a workload whose prompts total 350,309, so preemption added 14% more prefill work to a run that was already prefill-heavy. (It is partly refundable: if prefix caching is on and the freed blocks have not been evicted from the cache by the time the request comes back, the recompute is a cache hit instead. That is a race, not a guarantee.)

Phase two is the waiting queue, and it is guarded by one line that explains a symptom people find baffling:

if not preempted_reqs and self._pause_state == PauseState.UNPAUSED:
    while (self.waiting or self.skipped_waiting) and token_budget > 0:
        ...
        if num_running >= self.max_num_running_reqs:
            break

A pass that preempted anything admits nobody. So under memory pressure the server does not degrade smoothly; it alternates between steps that evict and steps that refill, and the queue behind it stops moving during the evicting ones. And max_num_running_reqs — which is exactly --max-num-seqs — is read in only one place in the whole scheduling decision: here, as a break in the admission loop. It is a ceiling on admission. It is not a target, not a reservation, and nothing in the engine tries to reach it.

Where the batch size actually comes from

The number of sequences that fit is arithmetic, and you can do it before you launch anything. Each token of context costs 2 × layers × kv_heads × head_dim × bytes of cache. For Qwen3-8B that is 2 × 36 × 8 × 128 × 2 = 147,456 bytes, or 144 KiB per token — the 8 rather than 32 is grouped-query attention already having cut it by four. A 16 GiB pool is therefore 116,496 tokens of cache, or 7,281 blocks of 16 tokens each. A request whose prompt is 1,368 tokens and which generates 256 ends up holding 102 blocks. Divide: about 71 sequences at their final size, and more than that early on when their caches are still short.

That is the whole story of the second readout. The scheduler reached 83 and then started evicting, because 83 sequences at their current lengths fit and 83 sequences at their eventual lengths do not. The cap is not a number you set. It is floor(pool ÷ bytes per resident sequence), and it moves during the run.

Where a bigger batch stops helping. The reason batching helps at all is arithmetic intensity: a decode pass reads the entire 15.26 GiB weight set whether one sequence is in it or two hundred, so each extra sequence rides along on a memory transfer that was already happening. That is a strong effect and it saturates. Walk --max-num-seqs up from 1 with the pool at 16 GiB:

• 1 — 158 tokens/s. One sequence pays for the whole weight read.
• 8 — 1,020 tokens/s. Six and a half times the work for eight times the sequences.
• 32 — 2,403 tokens/s. Still climbing, now clearly sublinear.
• 64 — 3,036 tokens/s, and every pass is at the slot cap. This is the best this pool can do.
• 128 — 2,911 tokens/s, batch peaking at 83, 38 preemptions. Worse.
• 256, 512 — 2,911 tokens/s. Identical, to the token. The scheduler never gets past 83, so the number in the config has stopped being read.

Two things are worth staring at there. The first is that the mean batch size at 128 is higher than at 64 — 47.5 against 44.3 — and throughput is lower. A bigger average batch bought less output, because the extra residents were paid for in re-prefilled prompts. The second is that 256 and 512 are not merely similar, they are bit-identical: past the memory wall the setting is inert, and any tuning experiment you run on it will report no change and no reason.

Now drag the KV cache pool instead, with --max-num-seqs at 256. 2 GiB gives 987 tokens/s and 68 preemptions; 8 GiB gives 2,300; 16 GiB gives 2,911; 32 GiB gives 3,465; 64 GiB gives 3,529 with zero preemptions and all 256 requests resident at once. Every one of those is a different answer to "what is my batch size?" and none of them involved changing the batch size setting.

The allocator is the other half, and it is invisible in the config

Two engines can both do iteration-level scheduling and still reach very different batch sizes, because they answer "how much cache does this request need?" differently at admission time.

Orca, the paper that introduced the technique, reserves. Its scheduling algorithm keeps a running count n_rsrv of reserved token slots and admits a new request only if n_rsrv + req.max_tokens ≤ n_slots, releasing the reservation when the request finishes. The consequence is stated as a feature, and it is one: a request that is admitted can never run out of memory, so Orca has no preemption path at all. The cost is that max_tokens is whatever the client sent, and clients send round numbers. The vLLM paper measured this on real systems and found that the fraction of KV cache memory actually holding a live token was between 20.4% and 38.2%.

Switch KV cache allocation to reserve up front and watch two readouts move together. At the default 4× declared max_tokens, the largest batch drops from 83 to 55, throughput from 2,911 to 2,521, and "held cache that holds a token" falls from 100% to 53%. Nearly half the pool is reserved against tokens that will never be generated. Now drag declared max_tokens:

• 1× — clients declare exactly what they use. 2,962 tokens/s, batch 71, 89% live. Reservation is nearly free, and slightly beats on-demand because it never preempts.
• 2× — 2,828 tokens/s, batch 65, 73% live.
• 4× — 2,521 tokens/s, batch 55, 53% live.
• 8× — 2,084 tokens/s, batch 42, 35% live. Two thirds of the cache is holding nothing.

Drag the same control with allocation on on demand and the throughput readout does not move at all: 2,911 at every setting. vLLM never looks at max_tokens when deciding whether to admit, which is why a client that asks for max_tokens: 4096 and stops at 200 costs you nothing there and costs you half your batch on an engine that reserves. That is the entire practical difference between the two designs, and neither engine's configuration file mentions it.

The trade is real in both directions. Reservation buys a guarantee — no preemption, no recompute, a predictable tail — at the price of a batch sized by what clients claim. On-demand buys the batch back and pays for it with a failure mode that only appears under load. Paging is what makes the second one tolerable: allocating in fixed 16-token blocks is what keeps the unused remainder of a sequence's last block down to a few tokens instead of a few thousand.

The boundary: when this is worth nothing

Continuous batching is not a throughput technique. It is a technique for removing idle slots, and if your slots are not idle it does nothing.

Set output-length spread to 1×, so all 256 requests generate exactly 256 tokens each, with --max-num-seqs at 64 and the pool at 16 GiB — settings where nothing preempts, so the only thing being measured is idle slots. Hold the allocator fixed and flip only the unit of scheduling. Reserving up front: 2,854 tokens/s per-request against 2,902 per-pass. On demand: 3,301 against 3,356. Both comparisons are a 1.7% improvement. The famous multiple has evaporated, and the residue is only the few passes it takes to fill the first batch.

Now put allocation back on reserve up front, leave the unit of scheduling on the request, and walk the spread up. At 1× it manages 2,854 tokens/s; at 16× the same total work takes 1,579. Static batching did not get slower because there was more to do — the output-token count is held constant — it got slower because the tail of the distribution decides when a batch is allowed to end. The gain from continuous batching is not a property of continuous batching. It is a property of your output-length distribution, and you can measure that without deploying anything.

This is the reason published speedups scatter so widely. Orca reports 36.9× over FasterTransformer, and that number is real and specific: a 175B model across multiple inter-layer partitions, compared at a matched median normalised latency of 190 ms, where FasterTransformer sustained 0.185 requests per second and Orca sustained 6.81. It is a chat workload with heavily varying lengths on a pipeline-parallel deployment where a stalled micro-batch idles several GPUs at once. Run a batch job where every request summarises to exactly 128 tokens and you will measure 1.02×, and both numbers are honest.

There is a second boundary in the simulation, and it is the one that costs people production incidents. Take the pool to 2 GiB and --max-num-seqs to 512. Throughput 987 tokens/s, 68 preemptions, 99,981 prompt tokens recomputed — against 350,309 tokens of prompt in the whole workload, so 29% of all prefill work was done twice. The lanes turn magenta. Nothing in that configuration is an error; every knob is within its documented range, and the engine will start and serve. It is simply admitting more sequences than it can carry, and paying for the difference in repeated work.

Checking it on a real system

vLLM tells you the memory-bound batch size at startup, in a line most people scroll past. It is emitted once, from update_kv_cache_capacity, and reads:

INFO ... GPU KV cache size: 434,657 tokens, Maximum concurrency for 8,192 tokens per request: 53.06x

Maximum concurrency is the answer to this entire lesson. It is the pool divided by max_model_len tokens of cache — the number of requests that fit if every one of them runs to the context limit. The “tokens per request” in that line is max_model_len, the context limit, not the per-pass token budget that happens to carry the same default on this card. If it says 53 and you are running with the default --max-num-seqs, you have 971 slots that the scheduler will never reach at full context length. Raising the setting cannot help; the two numbers you can actually move are --gpu-memory-utilization (0.92 by default, and worth pushing to 0.95 if you know what else is on the card) and --max-model-len, which divides straight into that concurrency figure.

Know what the default is before you decide it is too low. vLLM's get_batch_defaults() gives the OpenAI-compatible server max_num_seqs = 1024 on any GPU with at least 70 GiB that is not an A100, and 256 otherwise. On an H100 the shipped slot cap is already about twenty times the concurrency most deployments can afford. Almost nobody who "increases the batch size" is doing anything.

In the steady-state log line, one field is conditional and that is what makes it useful:

Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 812.4 tokens/s,
Running: 83 reqs, Waiting: 41 reqs, Preemptions: 38,
GPU KV cache usage: 99.8%, Prefix cache hit rate: 12.3%

Preemptions is printed only when the count is greater than zero, so its presence is the alert. The Prometheus counter behind it is vllm:num_preemptions, documented as "cumulative number of preemption from the engine"; alert on any non-zero rate, not on a threshold. Read the three fields together, because they diagnose different problems:

• Running well below --max-num-seqs with Waiting non-zero and KV cache usage near 100% — memory is your cap. Lower --max-model-len, raise --gpu-memory-utilization, quantize the cache, or shard it with tensor parallelism. Do not touch --max-num-seqs.
• Running pinned at --max-num-seqs with Waiting non-zero and KV cache usage well under 90% — the slot cap really is binding and raising it really will help. This is the rare case, and it is the only one where the obvious fix is the right one.
• Preemptions climbing at all — you are already past the wall. Every preemption is a discarded prefill, and the throughput you lose is roughly the preempted prompt tokens divided by your prefill rate.

One measurement to take before any of this, because it decides whether the subject is worth your time at all: pull the completion_tokens field off a day of responses and compute the median (p50) and the 99th percentile (p99). If p99 ÷ p50 is close to 1 — a classification service, a fixed-format extractor — iteration-level scheduling is buying you a couple of percent and your throughput problem is somewhere else entirely, most likely in how memory-bound your decode is. If it is 10 or more, which is what open-ended chat looks like, the scheduler is the largest single lever you have and the settings above are worth an afternoon.

A last structural note. This same shape — a scheduling unit so coarse that changing anything stops the whole assignment — is why Kafka spent years replacing its stop-the-world rebalance protocol; the fix there was likewise to make the unit smaller rather than to make the assignment bigger. If you have read consumer rebalancing, you have already met the argument in another accent.

Next: the memory this whole argument is about, the KV cache; the allocator that makes on-demand admission survivable, PagedAttention; what a prefill does to the decodes sharing its pass, chunked prefill; and the other way to get more tokens out of one forward pass, speculative decoding.