DeepSeek V4 Flash (304B) Runs in Production on Single AMD MI300X at 168 tok/s
Summary
- • An engineer published a production-ready open-source configuration enabling DeepSeek V4 Flash (304B params) to run on a single AMD MI300X GPU without quantization or weight offloading.
- • Benchmarks: 168.6 tok/s single-stream decode, 542 tok/s aggregate at 8 concurrent streams, 830 tok/s at 64 streams — no out-of-memory errors — with 256K context validated.
- • The full model (156.67 GiB) fits in the MI300X's 192GB HBM3; AMD's MI300X offers 2.4× the HBM capacity of an H100 SXM5 at roughly half the list price.
- • Critical patches were required for AMD's non-standard FP8 format (fnuz vs. OCP), MoE routing bugs, and KV synchronization issues not covered in the official vLLM recipe.
Details
168.6 tok/s Single-Stream Decode on One MI300X
Measured median single-stream decode throughput for DeepSeek-V4-Flash-0731 running on a single AMD MI300X using the published vLLM ROCm configuration with DSpark speculative drafting.
830 tok/s Aggregate at 64 Concurrent Streams, No OOM
The system handles 64-stream burst workloads at 830 tokens per second aggregate with no out-of-memory errors, demonstrating production viability for multi-user inference scenarios on a single card.
304B Model Fits in 192GB HBM3 Without Quantization
The full FP8 checkpoint (156.67 GiB) fits within the MI300X's 192GB HBM3 memory, with room for a 20GB GPU KV cache and 96GB CPU KV offload tier. No weight quantization or layer offloading was needed.
MI300X: 2.4× H100 HBM Capacity at ~Half the List Price
The AMD MI300X has 2.4 times the HBM capacity of an H100 SXM5 and costs roughly half as much at list price, making it an attractive option for large open-weight model hosting where memory capacity is the primary bottleneck.
AMD FP8 Incompatibility Required Correctness Fixes
MI300X (CDNA3) uses the AMD/Graphcore fnuz FP8 variant rather than OCP-standard FP8. Kernels assuming OCP semantics on MI300X can produce results wrong by a factor of two in the scale domain. Correctness fixes were the first engineering priority before performance tuning.
Official vLLM Recipe Didn't Cover Single MI300X Production Config
The official vLLM configuration targets NVIDIA hardware and newer AMD GPUs (MI325X at 4K context, MI355X), leaving MI300X users without validated production guidance for the V4 Flash checkpoint. This open-source repo fills that gap with pinned versions, patches, and tuning tables.
DSpark Speculative Drafting + Hybrid KV Cache Strategy
The production configuration uses probabilistic DSpark drafting with static K=7 and block rejection, plus a hybrid KV strategy: 20GB GPU FP8 cache and 96GiB CPU offload with load-path fencing to handle cold prompts without stalling concurrent streams.
Source: GitHub repository published to Hacker News (August 2026) — original production configuration and benchmark results for DeepSeek V4 Flash on AMD MI300X.
What This Means
This demonstrates that a state-of-the-art 304B parameter model can be run in production on a single AMD GPU at competitive throughput — opening a cost-effective self-hosting path that doesn't require expensive NVIDIA hardware. The MI300X's 2.4× HBM advantage at roughly half the H100 list price makes it theoretically attractive, but the substantial engineering effort required to work around FP8 incompatibilities and vLLM gaps underscores AMD's ongoing software ecosystem challenge. The published open-source configuration makes this reproducible for the first time. As AI organizations weigh GPU procurement costs and seek alternatives to NVIDIA dependency, configurations like this accelerate the real-world viability of AMD-based inference deployments for large open-weight models.