Speculative Decoding Spends Arithmetic You May Not Have

Speculative decoding does not make the model do less work. It makes it do about twice as much arithmetic — 2.06× at the settings the simulation below starts on, and 4.53× if you push it to draft ten tokens ahead — and it wins anyway, because a decoding pass leaves most of the GPU's arithmetic units idle while it waits on memory, and this technique spends the idle part. That is a real and large win on an idle card. It is a loss on a busy one, and the setting that turns it into a loss is not any of the settings the feature exposes.

The mechanism in one paragraph. A small draft model generates the next γ tokens one at a time, guessing what the big target model would have said. The target model then evaluates all γ+1 positions in a single forward pass — it can, because it already knows the tokens and only has to score them. A rejection-sampling step walks the guesses in order, keeps each one with probability determined by the two models' disagreement, and stops at the first rejection, emitting a corrected token there. So one target pass yields somewhere between 1 and γ+1 tokens. The number it yields on average is the mean acceptance length, and the per-token probability that drives it is the acceptance rate, written α.

The part that surprises people first: the output distribution is unchanged. Not approximately, not usually — provably. Leviathan et al. show that sampling from the draft and correcting by rejection produces exactly the target model's distribution, so a request served speculatively is statistically indistinguishable from one that was not. There is no quality dial here. Everything that follows is about time.

Below is Qwen3-8B on one H100, generating 256 tokens for each of some number of concurrent users, with Qwen3-0.6B drafting for it. It loads at batch size 1 — one user, an otherwise empty GPU, which is the condition the published speedups were measured under. Read the speedup, then drag concurrent sequences to the right.

At batch size 1 the readout says 1.94×, and it is honest: 311 tokens per second against 160. That is the number the papers report and the number every blog post quotes. Look at the two readouts beside it. The mean acceptance length is 2.63, so each target pass is producing 2.63 tokens instead of one — but the speedup is only 1.94×, because 26% of the wall clock went on the draft model. And the arithmetic readout says 2.06×: the GPU did twice the floating-point work to produce the same text.

Now drag concurrent sequences to 64. The speedup is 0.88× — speculation is now 12% slower than not bothering. Nothing about the draft model changed, the acceptance length is still 2.61, and the arithmetic bill is still 2.07×. What changed is that there were 64 sequences to spread the weight read across, so the pass had already found something to do with the arithmetic that speculation wanted to borrow. Keep going to 1024 and it reaches 0.65×.

The ladder underneath shows the whole curve at once, and it is worth knowing where your own deployment sits on it before you turn anything on. Then switch where drafts come from to n-gram prompt lookup and watch the ladder go flat and teal: a drafter with no weights and no cache of its own keeps winning at every batch size, because it is not competing for the thing that ran out.

What one iteration actually does

Algorithm 1 of the paper is short enough to read in full, and reading it is the fastest way to stop thinking of this as an approximation.

Sample γ guesses x₁…x_γ from Mq autoregressively.
Run Mp in parallel:  p₁(x), …, p_γ₊₁(x) ← Mp(prefix), …, Mp(prefix + [x₁…x_γ])
Determine the number of accepted guesses n:
  r₁ … r_γ ~ U(0,1)
  n ← min({ i−1 | 1 ≤ i ≤ γ, rᵢ > pᵢ(x)/qᵢ(x) } ∪ {γ})
Adjust the distribution if needed:
  p′(x) ← p_{n+1}(x)
  if n < γ:  p′(x) ← norm(max(0, p_{n+1}(x) − q_{n+1}(x)))
Return prefix + [x₁ … x_n, t]  where t ~ p′(x)

Three things follow from those lines and they are all load-bearing.

The number of serial target passes can never increase. Even when every guess is rejected, the last line still emits one token sampled from the target's own corrected distribution. Worst case is exactly plain decoding plus the draft model's time. There is no configuration in which the token count gets worse — only the wall clock, and only because of the draft.

Rejection is prefix-truncating, not per-token. The min takes the first index that fails. If guesses 1, 2 and 4 would have been accepted and guess 3 is rejected, you keep two, not three. That is why the token row in the simulation greys out everything after the first magenta cell, and it is the whole reason the marginal value of draft position 5 is so much lower than position 1 even before acceptance starts falling.

The correction is what preserves the distribution. When guess n+1 is rejected, the replacement is not sampled from the target's distribution p but from norm(max(0, p − q)) — the part of p that q did not already over-represent. That residual is exactly the correction needed to make the composite sampler's output distribution equal p. Appendix A.1 of the paper proves it for any two distributions. Chen et al. at DeepMind derived the same scheme independently and reported 2× to 2.5× on Chinchilla 70B with the same guarantee.

So the acceptance rate is not a quality knob turned down for speed. α is a measurement of how often two models agree, and Corollary 3.6 pins it exactly: α = 1 − E(D_LK(p, q)), the expected total-variation-style distance between the two models' next-token distributions. You cannot raise it by accepting worse output. You can only raise it with a draft model that is a better imitation — which costs time, which is the subject of the next section.

The speedup is borrowed, and there is a lender

A decode pass for one sequence reads the entire 15.26 GiB weight set and then does about two floating-point operations per weight it read. That is catastrophically low arithmetic intensity: at batch 1 and 2,048 tokens of context the verify pass moves 15.6 GiB and performs 0.09 TFLOP, which is 6.23 ms of memory time against 0.12 ms of arithmetic. The readout says the tensor cores are idle 98% of that pass.

Speculative decoding is a way of putting tokens into that idle 98%. Checking five positions instead of one reads the same weights and the same cache — the cache read serves every query position — and adds only arithmetic. So the verify pass costs almost nothing extra and returns 2.63 tokens instead of 1.

Which means the technique is not competing with plain decoding. It is competing with every other use of the same idle arithmetic, and the biggest one by far is putting more sequences in the batch. Drag the concurrency slider and watch the idle readout fall: at 1 sequence the tensor cores are idle 98% of the verify pass, at 32 they are idle 62%, at 64 they are idle 45%, and at 1024 they are idle 5%. Continuous batching got there first and spent the same budget, and it spends it more efficiently — an extra sequence contributes a whole token per pass, while an extra draft position contributes α of one.

Leviathan et al. are explicit about the assumption this breaks. The walltime analysis in Section 3.3 says it "assume[s] that we have enough compute resources to support the increased concurrency" and that "we can run γ + 1 concurrent evaluations of M_p in parallel without increasing the walltime". Their measurements were taken "with a batch size of 1 on a single TPU-v4". Nothing in the paper is wrong. The assumption is simply false on a server with 64 people on it, and the assumption is where all the speedup lives.

The bill is quantified in the paper too. Theorem 3.11 gives the expected increase in total arithmetic as (1−α)(γĉ+γ+1)/(1−α^γ+1), where ĉ is the ratio of the draft's operations to the target's. At α = 0.75, γ = 4 and a negligible ĉ, that is 1.64×.

You can make the simulation reproduce that number by giving it the paper's assumptions. Set the drafter to n-gram, which is what makes ĉ zero, and acceptance to the same at every position, which is the i.i.d. assumption the derivation needs. The arithmetic readout then says 1.62×, computed from the simulation's own FLOP counts rather than from the formula. Turn both assumptions back off — a real 0.6B draft, acceptance falling with position — and it climbs to 2.06×, because ĉ is no longer negligible and because guesses that fail earlier waste more of what was drafted. Move the concurrency slider anywhere on the ladder and that number barely shifts: the arithmetic bill is a property of the draft and the acceptance, not of the batch. What the batch decides is whether you had the arithmetic to spare. You are buying latency with arithmetic at somewhere between three-to-two and two-to-one, and which end you land on is a property of your drafter, not of the technique.

Why a better draft model can be a worse deal

The obvious way to raise acceptance is a bigger draft model. Switch where drafts come from to Qwen3-1.7B and leave acceptance where it is: the speedup falls from 1.94× to 1.39×. That is expected — you have not given it credit for guessing better yet. So raise acceptance for it. At α = 0.95 — a draft that agrees with an 8B model nineteen times in twenty, which is a fantasy — the 1.7B draft reaches 1.86×, and it still loses to the 0.6B draft at α = 0.75.

This is not an artifact. It is the paper's own result, and its table says it plainly for English-to-German translation from T5-XXL at temperature 0:

• T5-small draft, α = 0.75, γ = 7 → 3.4×
• T5-base draft, α = 0.80, γ = 7 → 2.8×
• T5-large draft, α = 0.82, γ = 7 → 1.7×

Acceptance rises monotonically down that list and speedup falls by half. Whatever you are optimising, it is not the acceptance rate.

And the real cost is not the draft's weights. Look at what the simulation reports for the draft. Qwen3-0.6B holds 7% of the target's weights — but its KV cache costs 112 KiB per token against the target's 144 KiB, which is 78%. Qwen3-1.7B costs exactly the same 112 KiB, because cache size depends on layers × key-value heads × head dimension and those barely move between the three models: 28 layers and 8 key-value heads in the drafts, 36 and 8 in the target. Grouped-query attention shrank the target's cache by a factor of four and left the draft's alone.

So each of the γ draft passes re-reads a cache almost as large as the one the verify pass reads, γ times per iteration. Set the context slider to 8,192 with 64 concurrent sequences and the draft phase consumes 72% of the wall clock to deliver a 0.71× speedup. The draft model is not small where it matters.

That is the whole reason the field moved. An n-gram drafter has no weights and no cache; EAGLE and Medusa-style heads and the multi-token-prediction modules shipped with recent models run one extra layer on top of hidden states the target pass already computed, so their ĉ is a rounding error and their cache is either shared or absent. In the simulation, the n-gram option holds its speedup at every batch size on this context — it only loses at 1,024 sequences with 512 tokens of context each, where the verify pass finally goes compute-bound and there is genuinely nothing left to borrow.

Why γ has a ceiling

More speculation is not more speedup, and the simulation will not let you pretend otherwise. Hold everything else and walk num_speculative_tokens up with the 0.6B draft:

• At 1 concurrent sequence: γ=1 gives 1.61×, γ=3 gives 1.94×, γ=4 gives 1.94×, γ=6 gives 1.74×, γ=10 gives 1.42×.
• At 32 concurrent sequences: γ=1 gives 1.26×, γ=2 gives 1.27×, γ=4 gives 1.05×, γ=5 gives 0.94×, γ=10 gives 0.58×.

Two independent things pull the curve down. The first is truncation: because rejection cuts the prefix, position j only pays off if every position before it was accepted, so its expected contribution is α^j and the series converges. The closed form from the paper makes the ceiling obvious:

E(tokens per iteration) = (1 − α^γ+1) / (1 − α)  →  1/(1−α) as γ → ∞

At α = 0.75 that ceiling is 4 tokens per iteration no matter how large γ gets. Going from γ=4 to γ=10 buys you the difference between 3.05 and 3.83 expected tokens — a 26% improvement in tokens, in exchange for running the draft model six more times. You would need γ=17 to reach 3.98, and by then the seventeenth guess is surviving 0.75¹⁷ of the time — about once in every 133 iterations — while costing a draft pass in all of them.

The second is that α is not constant across positions, and the simulation's two acceptance models let you price that assumption. Switch acceptance across draft positions to the same at every position and the speedup at the defaults jumps from 1.94× to 2.29×, with the mean acceptance length going from 2.63 to 3.10. That 18% is the cost of the paper's i.i.d. simplification, and it is why the closed form flatters real deployments. In the falling model the per-position vector settles near 0.751, 0.477, 0.261, 0.123 — those are unconditional, so the fourth guess survives to be used in about one iteration in eight, and it costs a full draft pass every time.

The practical consequence: γ between 2 and 5 is almost always right, and the correct value depends on your concurrency rather than on your acceptance rate. Serving one user at a time, take γ=4. Serving 32, take γ=2 — or, at that concurrency with a draft model, take none.

Checking it on a real system

In vLLM, speculation is one config object rather than a set of flags:

--speculative-config '{"method": "ngram",
                       "num_speculative_tokens": 3,
                       "prompt_lookup_max": 4}'

--speculative-config '{"model": "Qwen/Qwen3-0.6B",
                       "num_speculative_tokens": 4}'   # method inferred: draft_model

method takes ngram, medusa, eagle, eagle3, draft_model and the growing family of model-specific multi-token-prediction heads; if you give model and leave method out, it is inferred. The draft model's blocks come out of the same paged KV cache as the target's, so enabling a separate draft model directly lowers the batch size the scheduler can reach — which, given everything above, is the wrong direction twice over.

The engine logs its own version of this lesson's readouts, from SpecDecodingLogging:

SpecDecoding metrics: Mean acceptance length: 2.61,
Accepted throughput: 2519.60 tokens/s, Drafted throughput: 6256.00 tokens/s,
Accepted: 10312 tokens, Drafted: 25600 tokens,
Per-position acceptance rate: 0.751, 0.477, 0.261, 0.123,
Avg Draft acceptance rate: 40.3%

The field names and their order are exactly what SpecDecodingLogging.log() emits. The values are this page's simulation at 64 concurrent sequences, not a capture from a real server, so read them as a shape rather than as a benchmark.

Read Per-position acceptance rate first and set γ from it directly. Each entry is counted over every draft, not over the drafts that reached that position, so it is the share of iterations in which that guess actually made it into the output — and it is therefore the marginal value of keeping that position. Cut γ where the entry falls below roughly 0.15: a position that survives one time in seven still costs a full draft pass every iteration. That vector is the most useful number the engine produces and almost nobody looks at it. In Prometheus the same information is assembled from counters, and vLLM documents the exact queries in the source:

rate(vllm:spec_decode_num_accepted_tokens_total[$interval]) /
rate(vllm:spec_decode_num_draft_tokens_total[$interval])          # acceptance rate

1 + ( rate(vllm:spec_decode_num_accepted_tokens_total[$interval]) /
      rate(vllm:spec_decode_num_drafts[$interval]) )              # acceptance length

vllm:spec_decode_num_accepted_tokens_per_pos[$interval] /
vllm:spec_decode_num_drafts[$interval]                            # per position

None of those tell you whether speculation is helping. They measure the draft, not the outcome, and it is entirely normal for acceptance to look excellent while throughput is down. The only honest test is an A/B under your real load: run the same benchmark twice, once with --speculative-config and once without, at your production concurrency, and compare output tokens per second. If you benchmark at concurrency 1 you will measure the paper's number and deploy a regression.

A decision rule that survives contact with production, in the order you should apply it:

• Is your steady-state Running count in single digits? Interactive single-user work, an internal tool, a batch of one. Speculation is a large latency win; use a draft model if you have a good one, γ=4.
• Is it in the tens or hundreds? Use a zero-cost drafter — n-gram, or a multi-token-prediction head if your model ships one — with γ=2 or 3, and A/B it. A separate draft model is very unlikely to pay.
• Is the workload copy-heavy? Code editing, retrieval-augmented answers that quote their sources, "rewrite this but change X". n-gram lookup is nearly free and acceptance on repeated spans is very high; this is the one case where the free drafter also has a good α. The paper found a trivial bigram model reached α ≈ 0.2 on translation and still gave 1.25× at γ=3.
• Is per-user latency the thing you are paid for, not total throughput? Then a speedup below 1.00× on the throughput readout may still be the right trade, because time to each token for the user in front of you improved. Say which one you are optimising before you read any benchmark.

Next: the idle arithmetic all of this is competing for, arithmetic intensity and the roofline; the other claimant on it, continuous batching; the cache the draft model doubles, the KV cache; and the way to make the weight read itself smaller, quantization.