vLLM Introduces Hardware-Agnostic Layers to Preserve Cross-Hardware Compatibility
Summary
- • vLLM adds hardware-agnostic layers to maintain portability as frontier models require custom kernels
- • HW-agnostic layers achieve 96.6% of native H100 throughput while remaining torch.compile compatible
- • Frontier 'flat' models optimize for latest GPUs but are incompatible with torch.compile portability
- • Supports NVIDIA, AMD, Intel XPUs, Google TPUs, IBM Spyre, and Huawei Ascend hardware
Details
HW-agnostic layers achieve 96.6% of native H100 throughput across 3 recent models
On NVIDIA H100 GPUs, the hardware-agnostic layers achieve total token throughput within 3.4% of the native implementation, measured as a geometric mean across three recent frontier models.
Frontier 'flat' models use custom fusions incompatible with torch.compile
New frontier models added to vLLM now use hardware-specific custom fusions and kernel optimizations, making them fundamentally incompatible with torch.compile-based portability that underpins cross-hardware support.
HW-agnostic layers remain torch compilable and extensible for diverse hardware
Unlike flat models, HW-agnostic layers work with torch.compile, enabling portability across AMD GPUs, Intel XPUs, Google TPUs, IBM Spyre, Huawei Ascend, and older NVIDIA GPUs.
Two-track model: flat models for frontier performance, HW-agnostic for portability
vLLM now maintains hardware-specific flat models for maximum performance on latest GPUs (e.g., NVIDIA Blackwell, GB300 NVL72) alongside hardware-agnostic layers for users prioritizing portability and out-of-tree accelerators.
Coding agents (Claude Code, Codex) drive demand for hardware-specific kernel generation
Tools like Claude Code and OpenAI Codex are effective at generating hardware-specific optimizations but work best without cross-hardware constraints, driving the push toward per-model, per-hardware flat model definitions.
Frontier model architecture divergence is the root cause
DeepSeek V4 and Kimi K3 use completely different approaches for million-token context, exemplifying how frontier architectures are diverging in ways that strain any shared abstraction layer.
vLLM hardware abstraction architecture update: introducing HW-agnostic layers alongside hardware-specific flat models for frontier performance.
What This Means
vLLM's introduction of hardware-agnostic layers is a pragmatic response to a real fork in AI infrastructure: as frontier models increasingly require custom hardware-tuned kernels for peak performance, maintaining a single torch.compile-compatible codebase has become too costly. By running two parallel tracks — optimized flat models for the latest NVIDIA GPUs and portable HW-agnostic layers for everyone else — vLLM ensures the ecosystem doesn't fragment along hardware lines. At 96.6% native efficiency, the performance penalty for portability is now nearly negligible, which matters enormously to organizations running models on AMD, Intel, TPU, or older NVIDIA hardware. This architecture sets a template for how open-source inference frameworks can stay competitive at the frontier without abandoning their broad hardware communities.