Research note 5

Measuring goodput: calibration and four guards

Date: 10 November 2026
Status: In use on every run.
Hardware: 1 × NVIDIA H100 (PCIe) for calibration
Model: Qwen3-8B, FP8 weights
Engine: vLLM 0.32.0

Abstract

A claim about speed rests on its measurement. We set out how our harness defines capacity, what calibration on one H100 showed about throughput and attainment, and four faults in our own measurement that were each larger than the effects we had set out to detect. Each fault now has a guard, and a run that fails a guard is discarded.

Definitions

A service class is a pair of limits: one on the time to first token and one on the time per output token. A request meets the class if it meets both. Goodput is the rate of output tokens from the requests that met the class [1]. If the requests that met it are the set M, request i produced n_i tokens, and the run lasted T seconds,

G = 1T ∑_{i ∈ M} n_i(1)

A request that misses either limit contributes nothing. Attainment is the share of requests in M. Capacity is the highest goodput at any load at which attainment is at least 90%. The classes we use are given in technical note 5, and the two latencies are defined as is usual [3]. A per-request criterion exposes stalls that an average conceals [2].

Calibration

Before the settings search of research note 1, the server was driven at several loads with the engine’s defaults.

**Table 1.** Calibration at the standard class: first token within 1 s, each later token within 50 ms.
Load	Throughput, tokens per second	Attainment
Open loop, 4 requests per second	1,005	100%
Closed loop, 128 users	3,277	76%
Closed loop, 256 users	3,277	0%
Open loop, beyond capacity	2,426	not recorded

Three things follow. Throughput reaches a plateau: doubling the users from 128 to 256 added nothing. Attainment does not plateau: over the same interval it fell from 76% to zero. Throughput calls the two loads equal, and by the criterion users experience the second delivered nothing. Thirdly, a closed loop flatters an overloaded server. Under open-loop load beyond capacity, throughput itself fell by a quarter, because requests continued to arrive whether or not earlier ones had finished (technical note 6).

Four faults and their guards

While building the harness we found four faults in our own measurement. Each was larger than the factor of 1.14 that the settings search eventually reported.

**Table 2.** Faults found in the harness, the size of the error each produced, and the check that now guards against it.
Fault	Error	Guard
The load generator distorted the load	1.5×	The load actually offered is checked against the plan at every level.
Prompts repeated within a run	6×	Prompts are made distinct at each level, so that a cache cannot answer for the model.
The limit on output length was ignored	4×	The tokens generated are checked against the tokens requested.
Latency tails counted only the requests that finished	not a single factor	Percentiles are taken over every request sent. A failure or a timeout counts as a miss.

All four guards must pass at every load level. A level that fails one is discarded and rerun.

Two further lessons from the laboratory bench

A compiled graph can freeze its random numbers

In the work of research note 3, a decode round compiled into one graph drew its random numbers from a global generator. The compiler captured the draws, and every call replayed the same ones. A bench that appeared stable was measuring one frozen sequence, and the figure depended on which sequence had been captured. The generator’s state is now passed into the graph and returned from it explicitly.

A median with a stall in it is not a measurement

Under memory pressure single calls stalled for 200 to 600 ms and the next call was normal. A median of five runs that contains one such stall says nothing about decoding. One configuration returned 3,496, 3,442, 3,325, 2,960 and 2,907 tokens per second in different sessions. We therefore compare two configurations back to back within a session, and repeat a bench before accepting a difference of a few percent.

Rules of procedure

The pass bar and the stop rule are written before the run.
A result is reported with one of three verdicts. Pass: the bar was met. Stop: it was not. Check: the gain was implausibly large, and the baseline is rerun before anything is concluded.
Each load level is run in three rounds, and intervals are taken by a block bootstrap.
Per-user speed and fleet goodput are both reported, since a change can raise one and lower the other (technical note 3).
A result that missed its bar is published with the same care as one that met it.

Limitations

Goodput depends on the class chosen. A result at the standard class says little about the interactive class.
Calibration was on one model and one GPU.
Cost per token follows from goodput and the price of the hardware, and depends strongly on utilization [4]. We do not report it in this note.

References

Y. Zhong, S. Liu, J. Chen, et al. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024.
A. Agrawal, A. Agarwal, N. Kedia, et al. Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems. 2024.
NVIDIA. NIM for LLMs Benchmarking Guide: Metrics.
C. Patil. Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation. 2026.

Write to us about this note.