NVIDIA argues its full-stack inference software is cutting token costs on Blackwell, citing up to 5x improvements

NVIDIA is leaning into a cost-per-token argument as companies scale AI from experimentation to production, describing how its inference software stack is designed to deliver more useful output per dollar, per watt, and within latency targets. In a new post, the company says the move from predictable web workloads to agentic systems that coordinate many steps across models has shifted infrastructure decisions away from peak chip specifications toward system-level throughput.

The post frames modern AI inference as a stack problem. NVIDIA says that while earlier workloads could be handled by scaling out similar servers, agents can “reason, plan, call tools, spin up specialist subagents and manage massive context across multi-turn workflows.” That process can turn a single user request into distributed computing that spans hundreds of subagents, thousands of tasks, and multiple large language models across GPUs, CPUs, DPUs, and storage systems.

NVIDIA’s central claim is that its full-stack inference software helps avoid wasted capacity and reduces cost per token by turning individual optimizations into coordinated system performance. It describes its approach as connecting three layers: production operations and model runtimes, kernels, and communication libraries that manage hardware access. The company says this is how optimizations at different points in the pipeline can “compound” rather than remain isolated improvements.

On NVIDIA’s Blackwell platform, the company points to measured progress on DeepSeek V4. It says its inference software stack has already reduced token costs by up to 5x on the DeepSeek V4 model in just one month, based on performance improved across leading inference frameworks. NVIDIA also argues that this kind of short-cycle improvement is facilitated by the open-source development path it emphasizes for GPU-accelerated inference.

NVIDIA attributes additional throughput gains to specific technical building blocks. It cites disaggregated serving, large expert parallelism over NVIDIA NVLink interconnect technology, NVFP4 precision, and multi-token prediction as separate contributors that it says deliver meaningful gains on their own. Combined, the company says the approach increases throughput by up to 20x.

The company also uses open-source momentum as part of its explanation for how quickly improvements reach production. NVIDIA says many widely used open-source AI frameworks and inference projects are built natively on NVIDIA CUDA, enabling new research and software optimizations to run with leading performance on NVIDIA GPUs from day zero. As an example, it points to PyTorch, saying it launched in 2016 with native CUDA support and has coevolved with NVIDIA architectures through access to features including Tensor Cores, Transformer Engine, and NVFP4 through familiar framework APIs.

NVIDIA ties that framework ecosystem to specific performance advancements it says can be adopted on its hardware quickly. It mentions DFlash speculative decode, which it says can deliver up to 15x more throughput on existing hardware, and FastVideo, which it says can generate 1080p video in less than five seconds. In NVIDIA’s telling, when such breakthroughs land in CUDA-native paths within PyTorch, companies can convert research progress into lower token costs without needing to rewrite core systems from scratch.

In another example, NVIDIA says that when new frontier open models such as DeepSeek V4 are released, major inference frameworks like vLLM and SGLang have day-zero deployment recipes for the Blackwell architecture. NVIDIA then claims that DeepSeek V4 performance on Blackwell improved by up to 5x within about a month across vLLM and SGLang, cutting token costs to roughly one-fifth of previous levels.

NVIDIA’s post does not provide granular details on the operating conditions behind these results. It does not specify the exact hardware configurations, workload mix, batch sizes, concurrency levels, latency targets, or how “token cost” is defined in each case. It also does not attribute the 5x and one-fifth figures to an individual software feature, instead describing a coordinated approach across the inference stack and the deployment frameworks.

For operators and buyers, the practical takeaway is that scaling AI inference may be less about chasing raw chip peak performance and more about integrating runtime, kernels, communication, and serving strategies that maximize throughput under real latency and cost constraints. The immediate question to watch is whether broader model portfolios and more complex agent workflows continue to see similar compounding gains as software updates propagate through the CUDA-native open-source ecosystem on Blackwell.