FastPathLabs Research in LLM inference

A research and engineering laboratory for large language model inference.

FastPathLabs studies how a request becomes tokens: how it is routed, scheduled, executed and held in memory. Each mechanism is measured on real hardware against a bar fixed before the run, and the result is published whether or not it clears the bar. The work is directed toward an inference stack of our own.

Read the experiments Contact
Subject
Inference for large language models
Layers
Router, scheduler, engine, runtime
Method
Pre-registered experiments, judged on goodput at a latency target
In use
vLLM and SGLang, on NVIDIA H100, A100 and RTX A6000

Research program

Five lines of work. Each has a research note.

No.SubjectLayerStatus
1 Engine settings on a competent baseline Scheduler, engine Completed. Stopped at 1.14× against a bar of 1.5×.
2 Cache-aware routing for multi-turn chat Router Preliminary result. A larger run is specified.
3 Speculative decoding and the cost of a round Engine Laboratory result. The GPU run is specified.
4 Recurrent-state quantization for linear attention Runtime In progress. First kernel measured; the fold is being redesigned.
5 Measuring goodput: calibration and guards Method In use on every run.

The inference stack

Where the work sits. A request travels down seven layers and the tokens return the same way.

Applications

Chat, agents and batch jobs. Each sends a prompt and expects tokens within a latency target.

  • Chat
  • Agents
  • Batch jobs

Gateway

Authenticates the caller, applies limits, and holds the streaming connection.

  • Authentication
  • Rate limits
  • Streaming

Scope of our research

Router

Chooses the replica for each request. The right choice returns a conversation to the replica that already holds its cached state.

  • Replica choice
  • Cache-aware routing
  • Load balancing

Scheduler

Decides what runs in each step: which requests share a batch, how much prefill is admitted, and who waits.

  • Queue
  • Continuous batching
  • Chunked prefill

Engine

Executes the model for a batch. It reads each prompt once (prefill), then writes one token per step (decode).

  • PrefillDecodeSampling
  • KV cache
  • Speculation

Runtime

Turns a model step into GPU work: the compiled graph, the kernels, and the number format of weights and state.

  • Compiled graphs
  • Kernels
  • Quantization

Hardware

The GPU. Its memory capacity, memory bandwidth and compute bound every layer above.

  • Compute
  • Memory bandwidth
  • Interconnect
Figure 1. The layers of an inference deployment. The outlined region is the subject of our experiments. The layers outside it are held fixed and reported with each result.

Findings to date

Stated with their limits. Each links to the note that supports it.

  1. Engine defaults are a competent baseline

    Searching thirty combinations of four vLLM settings on one H100 raised goodput from 3,494 to 3,978 output tokens per second, a factor of 1.14 against a pre-registered bar of 1.5. Two settings produced the whole gain.

    Research note 1

  2. A router can recover only repeated prefill

    On four GPUs, cache-aware routing completed the same chats in 16 to 27% less time than round-robin, and 3 to 6% less than least-loaded routing, which already kept most turns on a warm cache. The saving is bounded by the prefill that would otherwise be repeated.

    Research note 2

  3. Fewer forward passes do not by themselves mean more tokens

    In laboratory runs a two-token draft brought the cost to 0.43 forward passes per token. It drew level with plain decoding only after verification, acceptance, cache trimming and the next draft were compiled into one graph.

    Research note 3

  4. A fold must fit a budget of about one millisecond

    Our first kernel for 8-bit recurrent state spends 84% of a step in the fold and is 7.2 times slower than SGLang’s window kernel at batch 512. The redesign is directed at the fold.

    Research note 4

  5. Throughput alone misleads

    One server produced 3,277 tokens per second with 128 concurrent users and with 256. In the first case 76% of requests met the latency target. In the second, none did.

    Research note 5

Technical notes

Ten short notes on the mechanisms behind the experiments.

  1. Prefill and decode are different workloads
  2. The memory-bandwidth ceiling
  3. Batching shares the read
  4. Prefix caching and what a router can save
  5. Goodput, attainment and service classes
  6. Open-loop and closed-loop load
  7. Speculative decoding: tokens per verification
  8. Launch overhead and the two regimes
  9. Precision: weights, KV cache, recurrent state
  10. The validity of a measurement

Papers

The experiments build on published work. The reading list gives the papers and articles we rely on, grouped by subject, with links to each. Our own working papers are listed with it.

Papers and reading list

Correspondence

We welcome correspondence from those who operate models at scale and from those who study the same questions.

Contact