vLLM Achieves 5,000 Tokens/GPU Throughput Serving Qwen3.8-2.4T on NVIDIA GB300
Summary
- • vLLM achieves 5,000 tokens/GPU throughput for Qwen3.8-2.4T on NVIDIA GB300 NVL72 cluster
- • 180 generated tokens/user achieved in low-latency mode on the same hardware config
- • Full pareto frontier published; srt-slurm recipes enable reproduction on any model
- • Qwen3.8-2.4T is a 2.4-trillion-parameter hybrid MoE model with 92 layers, 512 experts
Details
5,000 TPS/GPU in high-throughput mode
Peak total token throughput per GPU in high-throughput scenario on GB300 NVL72 with 8K/1K input/output workload configuration
180 gen tokens/user in low-latency mode
Generated tokens per user in low-latency scenario, optimizing for interactivity over raw throughput on the same hardware
Qwen3.8-2.4T model architecture
92-layer hybrid model: 69 GDN layers and 23 Full-Attn layers; MoE block in every layer with 512 experts per layer; 2.4 trillion total parameters
KV cache block sizing methodology
GDN state (4.125 MiB/request) dominates Full-Attn state (2 KiB/token) by 2108x; block size set to 4.125 MiB holding 2,112 tokens, determining concurrency limits
NVIDIA GB300 NVL72 benchmark platform
All benchmarks run on NVIDIA's GB300 NVL72 cluster; model weights are 91 GiB non-expert plus 1,242 GiB expert weights
Reproducible PD tuning methodology
Blog publishes step-by-step bottleneck identification process and srt-slurm recipes, designed to transfer to serving any frontier model beyond Qwen3.8-2.4T
Prior Qwen3.5 PD result: 25K TPS/GPU
Previous vLLM PD serving study on Qwen3.5 achieved 25K total TPS/GPU; the larger Qwen3.8-2.4T yields lower per-GPU throughput at this scale
PD = Prefill-Decode disaggregated serving; TPS/GPU = tokens per second per GPU; GDN = Gated Delta Network; MoE = Mixture of Experts
What This Means
vLLM's detailed PD serving benchmarks for the 2.4-trillion-parameter Qwen3.8-2.4T model push the frontier of practical large-scale model deployment. By publishing reproducible tuning recipes alongside raw benchmark numbers, vLLM lowers the barrier for ML teams trying to serve extremely large MoE models in production. The focus on both high-throughput and low-latency pareto points signals that frontier model serving is maturing beyond simple peak-performance benchmarks toward real-world usability. NVIDIA's GB300 NVL72 hardware emerges as the reference platform for this scale of inference.