Technical notes
Ten short notes on the mechanisms behind the experiments. Each states one point, gives the arithmetic where there is any, and names the papers it rests on.
- Prefill and decode
- The bandwidth ceiling
- Batching shares the read
- Prefix caching and routing
- Goodput and service classes
- Open and closed loops
- Speculative decoding
- Launch overhead
- Precision
- Validity of a measurement
Prefill and decode are different workloads
A request passes through two phases with different limits. Prefill reads the whole prompt in one pass. Its work grows with the length of the prompt, and it is bound by compute. Decode then writes the reply one token per step. Each step touches every weight of the model and the request’s cached state, and it is bound by the rate at which memory can be read.
The phases interfere when they share a step. A long prefill admitted into a step delays the decode of every other request in it. Engines therefore limit the tokens processed per step and spread a long prompt across several steps, and some deployments run the two phases on separate machines.
Two latencies follow from the phases. The time to first token covers the queue, the prefill and the first decode step. The time per output token is the duration of one decode step, which every request in the batch shares.
See: Sarathi-Serve on chunked prefill; DistServe on separating the phases.
The memory-bandwidth ceiling
For a dense model a decode step reads all the weights once. If the weights occupy W bytes and the GPU reads memory at B bytes per second, one stream cannot exceed
tokens per second, whatever the software does.
| GPU | Memory bandwidth | One read of the weights | Ceiling, tokens per second |
|---|---|---|---|
| H100 (PCIe) | 2.0 TB/s | 4.1 ms | about 240 |
| H100 (SXM) | 3.35 TB/s | 2.4 ms | about 410 |
| H200 | 4.8 TB/s | 1.7 ms | about 585 |
Three things move the ceiling: fewer bytes per weight, which is quantization; fewer weights touched per token, as in mixture-of-experts models, where W counts only the active experts; and more tokens per read, which is speculative decoding. A well-written memory-bound kernel reaches about 90% of a GPU’s peak bandwidth, so the ceiling is approached and not met.
See: Getting Memory-bound Kernels to Speed-of-Light; Domain specific architectures for AI inference.
Batching shares the read
Requests batched into one step share the read of the weights. Each adds a read of its own cached state, K bytes on average. With b requests in the batch and a fixed overhead t0, a step takes about
Each user receives 1/T tokens per second and the fleet produces b/T. As the batch grows, the speed each user sees falls, and the fleet’s output rises toward B/K, a limit set by the cached state and not by the weights.
For Qwen3-8B the KV cache occupies about 144 KiB per token, so a conversation of 2,000 tokens carries about 280 MiB. This is why the precision of the KV cache and the number of sequences admitted were the two settings that mattered in research note 1, and why every experiment reports both the per-user speed and the fleet’s goodput. A change can raise one and lower the other.
Prefix caching and what a router can save
The KV cache holds what prefill computed. When a new prompt begins with text a worker has already processed, the worker reuses that part and computes only the rest. vLLM finds such prefixes by hashing the cache in blocks of 16 tokens, each hash chained to the one before.
The cache belongs to one worker. Suppose a share s of prompt tokens is common to all conversations, a system prompt for example, and the rest is particular to one conversation. With n workers behind a round-robin router, a turn returns to the worker holding its conversation one time in n. To first order the hit rate is
With four workers and half the tokens shared, this gives 0.63. Research note 2 measured 61 to 75% for round-robin. A cache-aware router raises h toward 1. It can recover only prefill that would otherwise be repeated, so the gain is larger for long histories and short replies, and grows with the number of workers.
See: Prompt Cache; Preble; Inside vLLM.
Goodput, attainment and service classes
A service class is a promise: a limit on the time to first token and a limit on the time per output token. Goodput is the rate of output tokens from requests that kept the promise. A request that broke it counts as zero. Attainment is the share of requests that kept it. Capacity is the highest goodput at any load at which attainment is at least 90%.
| Class | Time to first token | Time per output token | Intended traffic |
|---|---|---|---|
| Interactive | 300 ms | 10 ms | A person reading as the reply appears |
| Standard | 1 s | 50 ms | General chat |
| Agent | 1.5 s | 20 ms | Long histories, many calls in sequence |
A time per output token of 10 ms is 100 tokens per second for that user. The class matters as much as the number: a configuration that is best at the standard class can fail the interactive one.
See: research note 5; DistServe; Etalon.
Open-loop and closed-loop load
A closed-loop generator keeps a fixed number of users, each of whom waits for a reply before sending again. When the server slows, the users slow with it. A closed loop therefore cannot show overload: the offered load falls to whatever the server can bear.
An open-loop generator sends on a clock whether or not the server keeps up, as real traffic does. In calibration, one server held 3,277 tokens per second under a closed loop at 128 and at 256 users. Under an open loop beyond capacity its throughput fell to 2,426.
We take capacity from open-loop runs. Closed-loop runs are kept for effects that depend on the number of concurrent users, such as the decay of speculative decoding.
See: research note 5.
Speculative decoding: tokens per verification
A small drafter proposes k tokens. The model scores them in one pass, keeps the longest prefix it accepts, and adds one token of its own. If each draft token is accepted with probability α, a pass yields on average
tokens. If a draft step costs a fraction c of a model pass, the speedup over plain decoding is about
At α = 0.75 and k = 2, equation (4) gives 2.3 tokens per pass. Our laboratory runs measured 2.34.
Three conditions sit behind equation (5). A verification pass must cost about one plain step, which holds when the step is a read of the weights and fails when it is launch overhead (note 8). Acceptance depends on the task: code and mathematics accept more than open conversation. And the gain falls with concurrency, because in a full batch the positions used for verification displace other requests’ tokens.
See: research note 3; Leviathan, Kalman and Matias; EAGLE-3.
Launch overhead and the two regimes
A decode step has a cost that does not depend on the size of the model: launching work on the GPU and synchronizing with the host. Whether that cost matters depends on what it is compared with.
| 16.7M parameters, bfloat16 | 7B parameters, bfloat16 | |
|---|---|---|
| Weights | 33 MB | about 14 GB |
| One read of the weights | 0.05 ms | about 23 ms |
| One decode step | about 0.5 ms, mostly overhead | about 23 ms. A launch of 0.3 ms is about 1%. |
| What helps | Compiling the step, several steps per call, sampling on the device | Moving fewer bytes: quantization, a drafter, a smaller cache, batching |
| Worth | 6 to 10 times, measured | Compiling and unrolling: about 1.0 to 1.3 times |
A technique proved in one regime is not evidence in the other. Results on small models show where a step spends its time. They do not predict the speed of a production model.
See: research note 3; Making Deep Learning Go Brrrr From First Principles.
Precision: weights, KV cache, recurrent state
Three different things can be stored in fewer bits, and the risks differ.
Weights are read at every step and never written. Halving their bytes roughly halves the read, and the question is only what accuracy is lost once.
A KV cache entry is written once and read at every later step. It can be rounded once, at the time it is written. Its precision sets how many conversations fit in memory, which is why an FP8 cache was one of the two settings that produced the gain in research note 1.
A recurrent state, in models with linear attention, is rewritten at every token. If it is rounded at every token the errors compound. The LeapQuant paper reports the AIME score of Qwen3.5-9B falling from 87.9 to 72.1 under a bf16 state rounded per token. The remedy is to round less often: hold the low-precision state fixed for a window of tokens, keep the updates in higher precision, and fold once per window.
See: research note 4; LeapQuant; KIVI; MARLIN.
The validity of a measurement
An effect of 1.1 to 1.3 times is easily produced by the measurement itself. We found four faults in our own harness, each larger than that: a load generator that distorted the load, by 1.5 times; prompts that repeated, by 6 times; an output limit that was ignored, by 4 times; and latency tails that counted only the requests that finished.
The laboratory bench added two more. A compiled graph replayed the same random draws on every call, so that a stable figure was the figure for one frozen sequence. And single calls stalled for hundreds of milliseconds under memory pressure, enough to move a median of five.
Each has a guard that fails the run. The bar is written before the run, two configurations are compared back to back in one session, and a difference of a few percent is not accepted from a single bench.
See: research note 5.