Working papers

These are drafts in circulation. Write to us for a copy.

• A pre-registered search of engine settings on one H100

  FastPathLabs working paper, November 2026. Summarized in research note 1.

• KV-aware routing and prefix caching: a benchmark report

  FastPathLabs report, November 2026. Summarized in research note 2.

• From 300 to 3,000 tokens per second: a step-by-step account on a laboratory model

  FastPathLabs laboratory report, November 2026. Drawn on in research note 3.

• LeapQuant on our hardware: analysis and plan

  FastPathLabs engineering report, November 2026. Summarized in research note 4.

Reading list

Thirty-three papers and articles, grouped by the layer of the stack they concern.

Serving engines and scheduling

• Efficient Memory Management for Large Language Model Serving with PagedAttention

  W. Kwon, Z. Li, S. Zhuang, et al. SOSP 2023. arXiv:2309.06180

  The block-based KV cache that most engines now use, and the basis of prefix caching in vLLM.

• SGLang: Efficient Execution of Structured Language Model Programs

  L. Zheng, L. Yin, Z. Xie, et al. 2023. arXiv:2312.07104

  The engine whose linear-attention kernels are the baseline of research note 4.

• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

  A. Agrawal, N. Kedia, A. Panwar, et al. OSDI 2024.

  Chunked prefill and a token budget per step: how a long prompt is kept from stalling the decodes of other users.

• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

  Y. Zhong, S. Liu, J. Chen, et al. OSDI 2024.

  Defines goodput for LLM serving, and separates prefill from decode to raise it.

• Efficiently Scaling Transformer Inference

  R. Pope, S. Douglas, A. Chowdhery, et al. 2022. arXiv:2211.05102

  The arithmetic of batch size, KV cache and latency for transformer inference.

• Vidur: A Large-Scale Simulation Framework for LLM Inference

  A. Agrawal, N. Kedia, J. Mohan, et al. MLSys 2024. arXiv:2405.05465

  Simulation as a way to search configurations without spending GPU hours.

• Inside vLLM: Anatomy of a High-Throughput LLM Inference System

  A. Gordić. 2025.

  A walk through the scheduler, the KV manager and the decode path of one engine.

Caching and routing

• Prompt Cache: Modular Attention Reuse for Low-Latency Inference

  I. Gim, G. Chen, S. Lee, et al. 2023. arXiv:2311.04934

  Reuse of attention state across prompts that share text.

• Preble: Efficient Distributed Prompt Scheduling for LLM Serving

  V. Srivatsa, Z. He, R. Abhyankar, D. Li, Y. Zhang. 2024. arXiv:2407.00023

  Routing by prompt prefix across a cluster. The closest published work to research note 2.

• Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

  R. Qin, Z. Li, W. He, et al. 2024. arXiv:2407.00079

  A serving architecture organized around the KV cache as the scarce resource.

• Marconi: Prefix Caching for the Era of Hybrid LLMs

  R. Pan, Z. Wang, Z. Jia, et al. 2024. arXiv:2411.19379

  Prefix caching for models with recurrent state, which cannot be rolled back to an earlier position.

Speculative decoding

• Fast Inference from Transformers via Speculative Decoding

  Y. Leviathan, M. Kalman, Y. Matias. ICML 2023. arXiv:2211.17192

  The method, and the expression for tokens per verification used in technical note 7.

• Accelerating Large Language Model Decoding with Speculative Sampling

  C. Chen, S. Borgeaud, G. Irving, et al. 2023. arXiv:2302.01318

  The same idea reached independently, with the sampling rule that preserves the model’s distribution.

• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

  T. Cai, Y. Li, Z. Geng, et al. 2024. arXiv:2401.10774

  Extra decoding heads on the model in place of a separate draft model.

• EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test

  Y. Li, F. Wei, C. Zhang, H. Zhang. 2025. arXiv:2503.01840

  Drafting at the level of hidden features. The design our draft head follows most closely.

• Better & Faster Large Language Models via Multi-token Prediction

  F. Gloeckle, B. Y. Idrissi, B. Rozière, D. Lopez-Paz, G. Synnaeve. 2024. arXiv:2404.19737

  Models trained to predict several tokens ahead, which can then draft for themselves.

Quantization and recurrent state

• LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

  Y. Pan, H. Xi, K. Zhu, et al. 2026. arXiv:2609.38166

  An 8-bit recurrent state, by rounding once per window and compensating the largest errors. The subject of research note 4.

• STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

  B. Yao, H. Xu, H. Lin, et al. 2026. arXiv:2609.38169

  A different route to a low-bit recurrent state, with scales fitted over rows and columns.

• Gated Delta Networks: Improving Mamba2 with Delta Rule

  S. Yang, J. Kautz, A. Hatamizadeh. 2024. arXiv:2412.06464

  The linear-attention layer used in current hybrid models.

• KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

  Z. Liu, J. Yuan, H. Jin, et al. 2024. arXiv:2402.02750

  Quantization of the KV cache, the corresponding problem for standard attention.

• MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models

  E. Frantar, R. L. Castro, J. Chen, T. Hoefler, D. Alistarh. 2024. arXiv:2408.11743

  Four-bit weights at full speed under batching, and not only for a single stream.

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

  DeepSeek-AI. 2024. arXiv:2405.04434

  Multi-head latent attention, which stores a compressed latent in place of full keys and values.

Kernels and hardware

• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

  T. Dao, D. Y. Fu, S. Ermon, A. Rudra, C. Ré. NeurIPS 2022. arXiv:2205.14135

  Attention computed in tiles that fit in fast memory. The model for memory-aware kernels.

• ThunderKittens: Simple, Fast, and Adorable AI Kernels

  B. F. Spector, S. Arora, A. Singhal, D. Y. Fu, C. Ré. 2024. arXiv:2410.20399

  A small language for writing kernels in tiles that match hand-tuned code.

• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B

  Hazy Research, Stanford. 2025.

  A whole forward pass fused into one kernel, to remove the gaps between launches.

• Getting Memory-bound Kernels to Speed-of-Light

  Guo, Zadouri and Dao. 2025.

  How close a memory-bound kernel can come to a GPU’s peak bandwidth, and what that takes.

• Domain specific architectures for AI inference

  C. Fleetwood. 2025.

  Inference hardware reasoned from first principles: where the energy and the time go.

• Learning FlashAttention the Hard Way, Part 1

  D. Trifonov. 2026.

  The algebra that lets attention be computed in pieces and merged.

• Making Deep Learning Go Brrrr From First Principles

  H. He. 2022.

  Compute, memory bandwidth and overhead as the three places time can go.

• Publications of Rohan Yadav

  R. Yadav. 2022 to 2026.

  Work on scheduling and compiling GPU kernels, including pipelining schedules shown to be optimal.

Measurement and cost

• Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems

  A. Agrawal, A. Agarwal, N. Kedia, et al. 2024. First circulated as Metron. arXiv:2407.07000

  Metrics that expose stalls and unevenness which averages conceal.

• Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation

  C. Patil. 2026. arXiv:2606.11690

  Cost per token as a function of load, with a spread of up to 36 times on identical hardware.

• NIM for LLMs Benchmarking Guide: Metrics

  NVIDIA.

  Standard definitions of time to first token, inter-token latency and throughput.