vLLM Trains DSpark Speculative Decoder for Kimi K3, Achieving ~4× Speed Boost on Math Reasoning
Summary
- • vLLM's Speculators library extends DSpark to Kimi K3 (2.8T-param frontier model)
- • Single-stream math reasoning speed jumps from ~110 to ~435 tokens/sec per user
- • Up to 3.5× higher aggregate throughput under concurrent load vs. baseline
- • Draft model: 5B params, 5 layers, 8 tokens/step; standard Hugging Face-compatible format
Details
DSpark extends DFlash with Markov correction and confidence head
DSpark adds a Markov logit-bias head for sequential token coherence and a hardware-aware confidence head for prefix scheduling, fixing DFlash's suffix decay problem without extra transformer passes.
Single-stream speed boosted from ~110 to ~435 tok/s/user on math
The Kimi K3 DSpark speculator boosts single-stream math reasoning interactivity from ~110 to ~435 tokens per second per user on the GB300 NVL72 hardware configuration.
Up to 3.5× aggregate output throughput under concurrent load
At matched interactivity levels under concurrent user load, the speculator delivers up to ~3.5× higher aggregate output throughput compared to the no-speculator baseline.
4.11 macro-average accepted tokens per verification round across 9 domains
Performance peaks on structured tasks: 6.42 tokens for math reasoning, 4.96 for HumanEval, 4.65 for translation, with a 4.11 macro-average across all nine evaluation domains.
5B-parameter, 5-layer draft model proposes 8 tokens per step
The released Kimi K3 draft model uses 5 layers and 5 billion parameters, proposing 8 tokens per decoding step in a single non-causal backbone pass.
DSpark beats EAGLE-3 by 27–31% in accepted sequence length
Across Qwen3 target models, DSpark reports 16–18% longer accepted sequences than DFlash and 27–31% longer than EAGLE-3 per verification round. DeepSeek V4 production saw 60–85% per-user speed gains.
Validated across Qwen3, Gemma-4, GLM-5.2, and Kimi K3 frontier models
vLLM's Speculators library has been confirmed working on Qwen3.6-35B-A3B, Gemma-4-31B-it, GLM-5.2, and now Kimi K3, covering a broad range of current frontier models.
DSpark performance metrics and implementation details from vLLM's Speculators library training run for Kimi K3 on GB300 NVL72 hardware.
What This Means
vLLM's open-source Speculators library has made it practical for any developer to train and deploy DSpark speculative decoders for frontier-scale models, including the 2.8T-parameter Kimi K3. The near-4× single-stream speed boost and 3.5× throughput gain could substantially cut serving costs and latency for teams running Kimi K3 in production. By packaging speculators in standard Hugging Face format with a straightforward training workflow, vLLM democratizes a performance optimization technique that was previously limited to well-resourced production teams with specialized infrastructure expertise.