vLLM: Push-Based NIXL Connector Speeds Prefill-Decode Disaggregated Serving
Summary
- • vLLM added a push-based NIXL connector for prefill-decode disaggregated serving
- • Decode instance pre-registers target blocks, so KV transfer starts as soon as prefill output is ready
- • Reported gains versus pull mode: up to 3x better TTFT and up to 30% better TPOT
- • Built and validated on AWS Trainium with EFA, contributed to vLLM as accelerator-agnostic
Details
vLLM added a push-based mode to its NIXL connector for prefill-decode disaggregated serving
It is an alternative to the default pull-based mode. Disaggregated serving runs prefill and decode on separate instances, so KV cache must move between them before decoding.
Default pull mode runs sequential steps on the TTFT critical path after prefill finishes
The router relays block IDs, the decode instance allocates blocks, prepares the transfer and issues a NIXL READ, each step waiting on the previous.
Push mode has the decode instance register target blocks while prefill is still computing KV
Transfer starts as soon as blocks are ready, removing the round trip to discover block identifiers after prefill completes.
Transfer setup runs on the prefill side in a background thread
This moves NIXL descriptor preparation and submission off the decode instance's latency-critical token-generation path.
Up to 3x better TTFT and up to 30% better TPOT versus pull mode on Qwen3-32B
Measured on 1 prefill and 1 decode instance on Amazon EC2 P5en with EFA. Results depend on workload and configuration, so maximums should not be assumed everywhere.
Built and validated on AWS Trainium with EFA, then contributed to vLLM as accelerator-agnostic
The design is not tied to one chip vendor, though the reported benchmark ran on GPU instances.
Related to, but distinct from, the Oct 12 bi-directional KV transfer post
Both address KV transfer in disaggregated serving; this change concerns who initiates the transfer and when.
Product Launch = feature, Tech Info = mechanism, New Tech = capability, Stat = benchmark, Infrastructure = platform, Context = background
What This Means
In disaggregated LLM serving, the KV cache handoff sits directly on the time-to-first-token path. Letting the decode side prepare in advance and moving setup off its critical path targets that bottleneck. The up-to-3x gain comes from one small benchmark, so real results will vary; teams running disaggregated vLLM, especially on AWS EFA, may want to test push mode.