vLLM: DeepSeek‑V4.1‑Flash on Vera Rubin NVL72 gets 58× GB300 throughput per GPU at 387 tokens/s per user
Corrected Wednesday, 18 November 2026: The 58× figure is throughput per GPU at 387 tokens/s per user, not the speed each user gets; added the benchmark, the P90 and batch size of the single‑user figure, that the HBM pool is experimental, and that the rack was early access hardware.
vLLM reports that on one early access NVIDIA Vera Rubin NVL72 rack, DeepSeek‑V4.1‑Flash serves 1.7M tokens/s per GPU while each user gets 387 tokens/s, 58× the throughput of GB300 NVL72 at that speed.
- On one NVIDIA Vera Rubin NVL72 rack, which NVIDIA provided in early access, vLLM serves DeepSeek‑V4.1‑Flash at 1.7M tokens per second per GPU while each user gets 387 tokens per second.
- At that per‑user speed, throughput per GPU is 58× that of GB300 NVL72, measured with the SemiAnalysis AgentX benchmark with P/D disaggregation and Wide‑EP; throughput counts cached input, uncached input and output tokens.
- The rack reaches 2.52M tokens per second at peak throughput, and 1,810 tokens per second (P90) for a single user at batch size 1.
- vLLM attributes these gains to kernels tuned for Rubin’s split memory and FP4 tensor cores, specialized decode kernels for low latency, and a rack‑wide KV cache and Engram tables shared over NVLink.
- With the Mooncake KV pool extended from DRAM to spare HBM (the HBM pool is still experimental) and Dynamo workload‑aware routing, three DEP2 prefill instances at concurrency 4,800 serve 68% more requests, cut TTFT P50 from 29.8 seconds to 4.2 seconds, and raise the prefix hit rate from 91.8% to 96.1%.
Who it affects Teams running or planning to run DeepSeek‑V4.1‑Flash with vLLM on NVIDIA Vera Rubin NVL72 racks, or using vLLM with Mooncake KV and Dynamo routing, can use these optimizations to increase agentic serving throughput and lower latency.
Source: vllm.ai ·