Prefix Caching Reuses a Prefix, Not Your System Prompt

Two requests arrive carrying the same 2,048-token system prompt. The server already computed that prompt's keys and values a second ago and still has them. Whether the second request gets to skip the work is not decided by comparing the text. It is decided by a chain of hashes, and the chain has a property nobody expects: one differing token anywhere near the front throws away every reusable thing behind it.

Four terms, because the rest is built from them. The KV cache is the per-token keys and values that attention needs, one pair per token per layer; that is what the KV cache lesson is about. A block is the fixed-size unit that cache is stored in — 16 tokens by default in vLLM, and the reason blocks exist at all is PagedAttention. Prefill is the forward pass that reads a whole prompt and produces its first output token; its duration is the time to first token, TTFT. And prefix caching is the engine keeping finished blocks around so that the next request whose prompt starts the same way can point at them instead of recomputing them.

Below is a serving engine handling 120 requests spread over a set of chat conversations. Every request carries the same system prompt and tool schema. Every request also carries an 11-token preamble that is unique to it — a timestamp, a request id, the user's display name, the kind of thing a chat template renders in without anyone thinking about it. Right now that preamble is at the end of the template, just before the user's question. Read the hero number, then move the preamble to the top. Change nothing else.

At the setting it loads with, the engine reuses 92.8% of the prompt tokens it is asked about, and a request reaches its first output token in 6.9 ms instead of the 78.4 ms it would take with no cache at all. That is the number every tuning guide promises you.

Now select first — above the system prompt. The hit rate is 0.0%. Not lower. Zero. Mean TTFT goes to 78.4 ms, the no-cache number, and the run recomputes 400,680 prompt tokens instead of 28,840. The system prompt did not change. The tool schema did not change. The conversation history did not change. Eleven tokens moved from one end of the template to the other.

The block picture is where this stops being surprising and starts being obvious. In the last request — 4,571 tokens, 285 full blocks — 284 of those 285 blocks contain exactly the same token ids as blocks that are sitting in the pool right now, and every one of them is recomputed. They are amber, not teal, because the cache was never asked whether it had them. The walk stopped at block 0 and never took another step.

Then try the middle option, immediately after the system prompt. The hit rate lands at 60.8% and TTFT at 32.2 ms. The engine keeps the 2,048 tokens of system prompt and loses every token of conversation history behind the preamble — 156 of the 157 recomputed blocks in that last request are, again, byte-identical to blocks in the pool. Three positions for the same eleven tokens: 92.8%, 60.8%, 0.0%.

What the engine actually computes when a request arrives

The rule is three lines of vLLM and it explains everything above. When a request's token ids are known, the engine walks them in fixed-size blocks and gives each full block a hash:

return BlockHash(
    hash_function((parent_block_hash, curr_block_token_ids_tuple, extra_keys))
)

parent_block_hash is the hash of the block before it. That one argument is the whole story. Block 7's key is not "these 16 token ids"; it is "these 16 token ids, arrived at through exactly those 112 tokens". Two blocks with identical contents at identical offsets get different keys if anything earlier differs, which is correct — the keys and values stored in a block were computed by attending over everything before it, so a block that followed a different prefix holds different numbers. The chain is not a hashing shortcut. It is the only thing making reuse sound.

The lookup is the matching half, and vLLM's own comment states the consequence better than a paragraph could:

# Phase 1: longest run of cached full blocks from the start. A missing
# block implies every later block misses too (chained hashes).
for block_hash in itertools.islice(full_block_hashes, max_length // block_size):
    cached_block = block_pool.get_cached_block(block_hash, kv_cache_group_ids)
    if not cached_block:
        break

break, not continue. There is no scan for a match further in. A cache hit is a contiguous run of whole blocks starting at token 0, and the run ends at the first block the pool does not hold. That is why the simulation's amber cells exist and why there is no setting that turns them teal.

One carve-out, and it is narrower than it sounds. Current vLLM has a phase 2 that runs only in fine-grained mode, where the granularity at which block hashes are computed (hash_block_size) is smaller than the physical block. It probes inside the first non-full block after the run, so it can extend a hit past a block boundary — but it still cannot jump over a miss, so nothing above changes. Reaching that mode at all takes more than a flag. resolve_kv_cache_block_sizes opens with if len(groups) <= 1: bs = cache_config.block_size * dcp; return bs, bs — one KV cache group means the hash granularity equals the physical block, and fine_grained, which is alignment_tokens < block_size, is false. A model has more than one KV cache group only when its layers do not all want the same block, which in practice means a hybrid model mixing attention types or a Mamba-attention stack. The knob that sets the granularity, prefix_match_unit — "the finest token boundary (in tokens) a prefix-cache hit can land on", default None — is read after that early return, so on a uniform full-attention model like the one in the simulation it has no effect at all. The block floor described below is what every non-hybrid model gets.

Two smaller rules ride along with it. Only full blocks are cached — the request hasher stops as soon as no further complete block exists, so a 24-token shared prefix at the default block size of 16 yields exactly one cacheable block. And the hit is deliberately capped one token short: vLLM sets max_cache_hit_length = request.num_tokens - 1 because a prompt that matched to its very last token would produce no logits to sample from. A prompt that is genuinely 100% cached still recomputes its final block.

extra_keys, the third element, is where the ids of any LoRA (low-rank adaptation) adapters, multi-modal input hashes and cache_salt go. Note where the salt is applied: [request.cache_salt] if (start_token_idx == 0 and request.cache_salt) else []. Only the first block carries it — and because the chain propagates, salting block 0 rekeys the entire request. That is exactly the behaviour you want for isolation and exactly the behaviour you do not want by accident.

Where real templates break the chain

The preamble in the simulation is not a contrivance. Here is what actually puts unique tokens in front of your shared text, roughly in order of how often it happens.

• A date or time in the system prompt. "Today is 20 August 2026, 14:31 UTC." Several widely used chat templates render the current date into the system message by default. At minute resolution your hit rate resets every minute; at second resolution it is zero.
• Per-user personalisation at the top. The user's name, plan tier, locale or account id inserted above the instructions. Every distinct user is a distinct prefix from token 0.
• Retrieved documents before the instructions. A retrieval-augmented prompt that puts context first and instructions after gives you no shared prefix at all, because the retrieved chunks differ per query. The same prompt with instructions first shares everything up to the first retrieved token.
• A request id or trace id in the template. Added for debugging, never removed.
• Tool schemas serialised from a dict. If the JSON serialisation order is not stable across processes, two replicas of your own service produce different token streams for the same tools — and so do two requests to the same replica if the dict was rebuilt.
• Anything before the system message at all. A beginning-of-sequence (BOS) token is fine because it is constant; a chat template that emits a per-request header is not.

The fix in every case is the same and it is free: order the prompt from most stable to least stable. Static instructions, then tool schema, then long-lived context, then conversation history, then per-request material, then the user's question. Anything that varies goes as late as it can go. The simulation gives you the price list for getting this wrong: at its default workload, moving eleven tokens from last to first costs 371,840 extra prompt tokens of compute across 120 requests, and mean TTFT goes from 6.9 ms to 78.4 ms.

One consequence people find genuinely surprising: because vLLM hashes request.all_token_ids and not just the prompt, the tokens the model generated are cached too. That is what makes turn n+1 of a conversation cheap — the previous answer is already in the pool by the time you send it back. Put the preamble position back to last, then set conversations in flight to 1, and watch the hit rate climb to 98.0% while the cold-start TTFT rises to 355.5 ms: a long chat is the best case prefix caching has, and it only works because the engine cached its own output. Leave it on immediately after the system prompt instead and the same single conversation reads 16.1%, because the preamble now breaks the chain at exactly block 128 — the end of the 2,048-token system prompt — on every turn after the first, no matter how long the history behind it has grown.

The block floor, and hit rates that buy nothing

A cache hit is a whole number of blocks. Everything below that granularity is lost, and the loss is not proportional — it is a floor.

Set preamble length to 0 — this section is about the floor, not about the preamble — then conversations in flight to 120 so every request is a fresh one-turn conversation, then system prompt + tools to 24 tokens. At --block-size 8 the hit rate is 33.1%. At the default 16 it is 22.0%. At 32 it is 0.0%, and it stays at zero for 64 and 128, because 24 tokens do not contain a single complete 32-token block. Nothing about the prompts changed. The reusable text is simply smaller than the unit of reuse. Now set the shared prefix to 120 tokens and sweep the block size again: 70.8%, 66.1%, 56.7%, 37.8%, 0.0%. This is the mechanism behind bug reports of the form "20% of my prompt is identical and my hit rate is under 0.1%" — the identical part was measured in characters, and the cache measures in blocks.

Now the more dangerous direction. Leave the preamble at 0 and conversations at 120, put the block size back to 16, and leave the shared prefix at 120 tokens. The hit rate reads 66.1%. The speedup reads 1.0x. Mean TTFT is 6.2 ms with the cache and 6.3 ms without it. Two thirds of every prompt is being reused and it is worth nothing at all, because the prompt was 168 tokens long and prefilling 168 tokens was never the expensive part.

The hit rate is a ratio of tokens, not of requests and not of time. vLLM computes it as hits / queries where queries is every prompt token asked about and hits is every prompt token served from cache, aggregated over the most recent 1,000 requests. That makes it an honest measure of how much prompt you are reusing and a poor measure of how much time you are saving. Walk the shared prefix up from 120 tokens with everything else fixed and watch the two numbers separate:

• 120 tokens shared: hit rate 66.1%, TTFT 1.0x faster.
• 256: hit rate 83.5%, 1.1x.
• 512: hit rate 90.7%, 2.0x.
• 2,048: hit rate 96.9%, 7.2x.
• 8,192: hit rate 98.6%, 25.7x.

Between the first and last rows the hit rate moves by 32 points and the actual saving moves from nothing to a factor of 26. If you are reporting prefix cache hit rate to anyone as a performance number, report TTFT next to it.

There is a floor under the saving too, and it is worth knowing why. The tokens you still have to compute must attend over every cached token, so a prefill with 4,336 tokens of context reused and 235 to compute is not free — it costs the projections and feed-forward work for 235 tokens plus attention for 235 queries against 4,571 keys. At the context lengths in this simulation that attention term is small. At 100k it is not, and the reason is the roofline: attention's cost grows with the product of new tokens and context, while the rest grows only with new tokens.

The boundary: a cache that costs more than it saves

Cached blocks live in the same pool as the blocks of running sequences. Every block held for a possible future hit is a block not available to admit the next request, and vLLM's allocator will take it back: a cached block whose reference count has dropped to zero goes to the tail of the free queue, and the head of that queue is what gets handed out next. That is least-recently-used eviction, and when the working set exceeds the pool it degenerates.

Return the controls to where they started — preamble 11 tokens, last, system prompt 2,048, block size 16 — because the numbers below are for that workload and the previous section left you somewhere else. Now set conversations in flight to 1 — the single best case, a long chat that reuses its own history — and take the GPU KV cache pool down to 8,192 tokens, which is 1.13 GiB. The hit rate falls to 11.2% and the engine performs 84,185 evictions across 120 requests. Mean TTFT is 321.6 ms against 355.5 ms with no cache at all: the cache is now doing roughly nothing while occupying every block it can get. One step up, at 16,384 tokens, you get 48.8%. At 32,768 you get 98.0% and 709 evictions. The curve is not gentle. Below the working set the cache thrashes; above it, it works.

This is the same failure mode as admitting more sequences than the pool can carry, described in continuous batching, and it has the same shape: a policy that looks free until the resource it quietly consumes runs out. The difference is that preemption is logged loudly and cache thrash is not — a thrashing prefix cache reports a low hit rate and nothing else.

The second boundary is isolation. Put the pool back to 262,144 tokens and conversations back to 8, then turn on send a per-conversation cache_salt, and the hit rate drops from 92.8% to 89.2%; the within-conversation reuse survives and the shared system prompt no longer crosses between conversations. Set conversations to 120 with the salt still on and the hit rate is 0.0%, because now every request is its own tenant and there is nothing left to share. The salt is the right tool when prompts carry data one tenant must not be able to probe for through timing, and it is worth understanding that its cost is precisely the cross-tenant sharing of the system prompt — which is usually the largest single win prefix caching has to offer.

A third boundary, and the reason a benchmark will lie to you: prefix caching only pays when requests that share a prefix arrive close enough together that the blocks survive. A synthetic load generator replays the same prompt set back to back and reports 99.8% where production reports 91%, and the difference is arrival pattern, not configuration.

Checking it on a real system

Prefix caching is on by default in current vLLM — enable_prefix_caching: bool = True in CacheConfig — so the question is almost never "is it enabled" and almost always "why is it not hitting". The steady-state log line carries the number:

Avg prompt throughput: 1904.3 tokens/s, Avg generation throughput: 812.4 tokens/s,
Running: 41 reqs, Waiting: 0 reqs,
GPU KV cache usage: 61.2%, Prefix cache hit rate: 4.1%

The fields and their order are what LoggingStatLogger.log() assembles; the values are an illustration of the case this section is about, not a capture from a real server.

That figure is hit_rate from CachingMetrics: the hit tokens divided by the queried tokens over a sliding window of the most recent 1,000 requests, defined by max_recent_requests. The two Prometheus counters underneath it are vllm:prefix_cache_queries ("Prefix cache queries, in terms of number of queried tokens") and vllm:prefix_cache_hits ("Prefix cache hits, in terms of number of cached tokens"). Because they are counters, the rate ratio over a window is the number you want on a dashboard:

rate(vllm:prefix_cache_hits_total[5m]) / rate(vllm:prefix_cache_queries_total[5m])

Now the diagnosis, which is a decision tree with three branches. Read the hit rate together with GPU KV cache usage:

• Hit rate near zero, cache usage low. Nothing is matching, and it is not a memory problem. Your prefixes differ at or near token 0. Go straight to the experiment below.
• Hit rate low, cache usage pinned near 100%, evictions implied by a hit rate that falls as load rises. The working set is larger than the pool. Raise --gpu-memory-utilization (0.92 by default), lower --max-model-len, or quantize the cache. Reordering the prompt will not help.
• Hit rate high, TTFT unchanged. The reused part of the prompt was never the expensive part. Nothing is broken; the metric is flattering you.

The experiment that settles branch one takes two minutes and needs no instrumentation beyond the log. Capture two consecutive real prompts from production — the actual strings your template produces, not the messages you passed in. Tokenize both and find the first index where the ids differ:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
a = tok(prompt_a).input_ids
b = tok(prompt_b).input_ids
i = next((i for i, (x, y) in enumerate(zip(a, b)) if x != y), min(len(a), len(b)))
print("diverge at token", i, "-> reusable blocks:", i // 16)

That printed block count is the ceiling on your hit rate, and it is computable before you change anything. If it says 0, no amount of pool tuning will move the metric and the fix is in your template. If it says 128 and your prompts are 4,000 tokens, you already know you are leaving 90% of the prompt on the floor and where the boundary is.

Reproduce the divergence point by rendering the template rather than by reading it. The most common cause found this way is a chat template inserting the current date; the second most common is a tool schema whose JSON key order is not stable. Both are invisible in the code and obvious in the token ids.

Two more places to look. vllm bench serve has a prefix_repetition dataset that will report a hit rate far above anything your traffic produces, so never use a benchmark hit rate as a baseline for a production alert. And if you serve multiple tenants from one engine, decide about cache_salt deliberately: it is a per-request field, so you can salt the tenant-specific prompts and leave a genuinely public system prompt unsalted, which keeps the largest shared block reusable while isolating the rest.

Next: the pool this all competes for, the KV cache; the allocator that makes block-level reuse possible at all, PagedAttention; what happens to the prefill that prefix caching leaves you when it has to share a forward pass with running decodes, chunked prefill; and the arithmetic that says why any of this moves TTFT and none of it moves tokens per second, arithmetic intensity.