Research note 3
Speculative decoding: tokens per verification and the cost of a round
Abstract
Speculative decoding promises several tokens for each forward pass of the model. On a small laboratory model we brought the cost down to 0.43 forward passes per token with a two-token draft, and found that this did not raise tokens per second until the whole round was compiled into one graph. At that scale a forward pass is mostly launch overhead, so removing passes removes little. We report the measurements, the reasoning that separates small models from large ones, and the pre-registered design of the run on an H100.
Background
In speculative decoding a cheap drafter proposes the next k tokens. The model scores all of them in one forward pass, keeps the longest prefix it accepts, and contributes one token of its own. The output distribution is that of the model alone [1, 2].
If draft position i is kept in a fraction pi of rounds, a round emits on average
tokens. The method pays when a round costs less than E plain steps. The general form and its dependence on acceptance are given in technical note 7.
Laboratory setup
The laboratory model is a byte-level transformer of 16.7 million parameters: six layers, width 512, eight heads. It stores a 64-number latent per token per layer in place of full keys and values, after the multi-head latent attention of [6], which makes its cache about sixteen times smaller. Weights are held in bfloat16 with 4-bit linear layers. Sampling is at temperature 0.8. Figures are medians of five warm runs of 200 tokens unless stated.
The model is small on purpose. A change can be made and measured in minutes. Its limits as evidence are set out in sections 7 and 9.
Three drafters
An n-gram lookup
A 4-gram table built from the training text proposes the next bytes. On an early checkpoint it kept about 32% of an eight-byte draft: 72% at the first position, 7% at the eighth. On a later checkpoint that generalizes better it kept 14 to 16%. A model that has memorized its corpus agrees with a corpus n-gram more often, so the better model is the one the n-gram serves worst.
A small transformer
A two-layer transformer drafter was accepted about as often and ran slower than plain decoding, because every draft byte was a full forward pass of a second network.
A draft head
The drafter we kept is a residual network of about 0.79 million parameters. It predicts the model’s next hidden state from the current state and the token embedding, and the model’s own frozen output layer reads a byte from it. A draft of k bytes is k very small steps. The approach is close to [3, 4]. On held-out text the head agrees with the model on 71% of next bytes.
One failure is worth recording. When the model’s checkpoint was replaced and the head was not retrained, agreement fell to 7%. The bench still showed a high first-position rate, because that column was counting the model’s own sample. A draft head is a dependency of the checkpoint it was trained against and must be versioned with it.
One verification pass per round
The first implementation verified the draft and then ran a second pass to commit the model’s own token. The second pass is unnecessary. One pass over the carried token and the drafts scores every draft, and its last row is the distribution of the token that follows if all are kept. On a rejection the replacement is sampled from the residual distribution and becomes the carried token of the next round.
| Mode | Passes per token | Draft positions kept | Tokens per second |
|---|---|---|---|
| Plain, eight tokens per call | 1.00 | 3,496 | |
| Speculative, k = 2 | 0.43 | 75%, 60% | 2,134 |
| Speculative, k = 4 | 0.30 | 85%, 68%, 43%, 35% | 2,593 |
| Speculative, k = 8 | 0.28 | 69% falling to 13% | 2,556 |
The passes fell by more than half and the speed fell with them. Fewer forward passes did not become more tokens per second.
Where the round went
| Draft the block | 0.57 ms |
|---|---|
| Forward pass of the model | 0.52 ms |
| Acceptance check, on the host | 0.03 ms |
| Trim the cache | 0.18 ms |
| Slice the hidden state | 0.02 ms |
The draft head occupies 1.6 MB. Reading it takes microseconds, so its 0.57 ms was the cost of launching work on the GPU and not of the work itself. The same holds for the forward pass, which matched a compiled one-token step. A round was a sequence of separate launches, and each launch cost about as much as the pass it was meant to save.
The round as one graph
We then compiled the whole round into a single graph: the forward pass over the carried token and the drafts, the rejection sampling, the cache trim, and the next draft. The host reads back the number of drafts kept and the emitted bytes once per round.
| Mode | Round | Tokens per round | Draft positions kept | Tokens per second |
|---|---|---|---|---|
| k = 2 | 0.77 ms | 2.34 | 75%, 59% | 3,114 |
| k = 3 | 0.84 ms | 2.57 | 69%, 50%, 37% | 3,024 |
| k = 4 | 0.95 ms | 2.93 | 71%, 51%, 38%, 32% | 3,011 |
| Plain, eight tokens per call | 2,960 |
The tokens per round agree with equation (1): for k = 2, 1 + 0.75 + 0.59 = 2.34. With the round fused, the two-token draft is ahead of plain decoding by about 5% in this pass. Longer drafts emit more tokens per round and cost more per round, and do not gain.
The lead is within the variation between sessions. The plain reference returned 3,496, 3,442, 3,325, 2,960 and 2,907 tokens per second on different occasions. Repeated paired runs with a three-token draft ended level with plain decoding. We therefore describe the laboratory result as parity reached by fusion, and not as a gain.
Why a large model differs
The laboratory model’s weights occupy about 33 MB, and reading them takes about 0.05 ms at the bench’s memory bandwidth. A step of 0.5 ms is therefore almost entirely launch overhead. In that regime a forward pass that scores three tokens costs the same as one that scores one, but so does every other launch, and the saving is spent on the extra launches a round requires.
A model of several billion parameters is in the other regime. One pass must read gigabytes of weights, which takes milliseconds, and a launch is a small fraction of it (technical note 8). A verification pass over k + 1 tokens reads the weights once. Tokens per verification then translate almost directly into speed, which is the result reported for large models [1, 4, 5].
Two effects limit the gain at scale and are the subject of the specified run. Acceptance depends on the task: code and mathematics are easier to draft than open conversation. And the gain falls as concurrency rises, because in a full batch the positions spent on verification displace other requests’ tokens.
The specified GPU run
| Hardware and model | 1 × H100 (PCIe). Qwen3-8B with FP8 weights, the revision used in research note 1. |
|---|---|
| Arms | No drafter, with the settings that won in research note 1. A block drafter on SGLang, with its own baseline on the same engine. A feature-level draft head on vLLM [4]. An n-gram lookup on vLLM. |
| Load | Closed loop at 1, 2, 4, 8, 16, 32 and 64 concurrent users. Open loop near the knee for the fleet figure. |
| Tasks | Chat prompts, GSM8K and HumanEval, reported separately. |
| Recorded | Tokens accepted per verification, each second, from the engine’s metrics. |
| Pass | The best drafter reaches twice the per-user speed of the no-drafter arm at four users or fewer. Greedy outputs are identical to that arm at one user. Accuracy lies within its 95% interval on GSM8K and 1,000 MMLU items. The concurrency at which the gain disappears is located. |
| Stop | Below 1.3 times. |
| Check | Above 5 times, the baseline is rerun before the result is accepted. |
Limitations
- The laboratory figures come from one small model on one machine. They show where a round spends its time. They are not evidence of speed on a production model.
- Session-to-session variation on the bench is as large as the differences between the fused modes.
- The specified run has not been reported. Its expected outcomes are not stated here, and only its bars are.
References
- Y. Leviathan, M. Kalman, Y. Matias. Fast Inference from Transformers via Speculative Decoding. ICML 2023.
- C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, J. Jumper. Accelerating Large Language Model Decoding with Speculative Sampling. 2023.
- T. Cai, Y. Li, Z. Geng, et al. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. 2024.
- Y. Li, F. Wei, C. Zhang, H. Zhang. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. 2025.
- F. Gloeckle, B. Y. Idrissi, B. Rozière, D. Lopez-Paz, G. Synnaeve. Better & Faster Large Language Models via Multi-token Prediction. 2024.
- DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. 2024.
Write to us about this note.