Research note 1
Engine settings on a competent baseline: a pre-registered search
Abstract
We asked whether searching a serving engine’s settings raises goodput over its defaults by a factor of 1.5, a bar fixed before the run. Thirty of the eighty combinations of four settings were measured on one H100. The best raised goodput at 90% attainment from 3,494 to 3,978 output tokens per second, a factor of 1.14. Under the pre-registered rule the experiment stops. Two settings produced the whole gain and two had no measurable effect.
Question
A serving engine exposes settings that govern how requests are batched and how memory is used. Public tuning recipes suggest that the defaults leave capacity unused. We wished to know how much, on a baseline that is already sensible: a current engine, an FP8 model, and prefix caching enabled.
The outcome was judged on goodput, the output tokens per second from requests that met both latency targets [3], and not on raw throughput. The terms are defined in technical note 5.
Method
| Model | Qwen3-8B, FP8 weights |
|---|---|
| Hardware | 1 × NVIDIA H100 (PCIe) |
| Engine | vLLM 0.32.0, prefix caching enabled |
| Service class | Standard: time to first token under 1 s, time per output token under 50 ms |
| Load | Open loop and closed loop, three rounds |
| Search space | Four settings, eighty combinations, thirty measured |
| Rule fixed in advance | Pass at 1.5× the defaults or more. Stop below it. |
The four settings were --max-num-seqs, the number of requests in one step; --max-num-batched-tokens, the tokens processed per step, which bounds how much prefill a step admits [2]; --gpu-memory-utilization, the share of GPU memory the engine may claim; and --kv-cache-dtype, the number format of the KV cache [1].
Three arms were compared. Arm A is the engine’s defaults. Arm B is a public tuning recipe. Arm C is the best combination found by the search. The engine’s scheduler and KV manager, on which these settings act, are described in [5]. Capacity is the highest goodput at any load at which at least 90% of requests met both targets.
Results
| Arm | Configuration | Goodput | Ratio to A |
|---|---|---|---|
| A | Engine defaults | 3,494 | 1.00 |
| B | Public tuning recipe | 3,491 | 1.00 |
| C | Search winner: FP8 KV cache, 256 sequences | 3,978 | 1.14 |
The whole of the gain in arm C came from two changes: storing the KV cache in FP8, and admitting 256 sequences per step. The batched-token budget and the memory share had no measurable effect.
The ratio is bounded as well as measured. Arm A met the target at 14 requests per second and failed at 16. Arm C met it at 16 and failed at 20. No load between 16 and 20 was tested, so the true ratio lies between 1.00 and 1.43. Either bound is below the bar.
Discussion
The baseline was already competent
Arm A was not an untuned server. It ran an FP8 model on a current engine with prefix caching enabled. A search over four settings on top of that is a small lever. The larger gap is likely to lie earlier, between a stock deployment and this baseline, and that arm was not measured here.
Why the two settings mattered
Both changes act on the same constraint. An FP8 KV cache halves the bytes each cached token occupies, and a larger sequence limit lets the scheduler use the room. More conversations then share each read of the weights (technical note 3).
The recipe that did not transfer
Arm B’s batched-token budget of 16,000 failed half its requests on time to first token under closed-loop load. A step that large takes long to complete, and a request that arrives during one must wait for it before its own prefill begins. The standard class is loose enough to hide this under open-loop load. A tighter class should separate the settings further.
Limitations
- Every prompt was distinct, so prefix caching had nothing to reuse. Production traffic repeats itself.
- One model and one GPU.
- One service class. The interactive class was not measured.
- The load grid was coarse near the knee, as the bounds above show.
Further work
- A stock arm: the unquantized checkpoint, default limits and prefix caching disabled, to measure the gap to arm A.
- The interactive class, with time to first token under 300 ms and time per output token under 10 ms, on a load grid of one request per second near the knee.
- Speculative decoding, which this search did not include. See research note 3.
- Simulation of the engine, to narrow a search before GPU hours are spent on it [4].
References
- W. Kwon, Z. Li, S. Zhuang, et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.
- A. Agrawal, N. Kedia, A. Panwar, et al. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024.
- Y. Zhong, S. Liu, J. Chen, et al. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024.
- A. Agrawal, N. Kedia, J. Mohan, et al. Vidur: A Large-Scale Simulation Framework for LLM Inference. MLSys 2024.
- A. Gordić. Inside vLLM: Anatomy of a High-Throughput LLM Inference System. 2025.
Write to us about this note.