NVIDIA says its Blackwell platform leads the first agentic AI infrastructure benchmark

NVIDIA is taking aim at a measurement gap in AI infrastructure, arguing that most performance benchmarks are built for chatbots that respond to a single prompt, not for agents that plan, call tools, and iterate until a goal is completed. In a new blog post, the company highlighted results from AgentPerf, described as the first “agentic AI” benchmark from Artificial Analysis, and said the NVIDIA Blackwell Ultra NVL72 platform delivered top performance on the test set.

The central claim is about efficiency under sustained, chained workload behavior. NVIDIA said AgentPerf’s first published results show Blackwell Ultra NVL72 delivering leading performance across the agentic workloads tested, running up to 20x more agents per megawatt than NVIDIA Hopper-based HGX H200. The company framed the metric as a way to compare systems on how many agent tasks can be run concurrently for a given power budget.

NVIDIA’s argument starts with the difference between conversational AI and agentic AI. A conversational system typically performs one large language model (LLM) “call” per user turn, returning a response in one step. An agent, by contrast, is tasked with a goal and operates like a relay. It breaks the objective into steps, carries forward context as it proceeds, and uses external tools such as code compilation and execution, database search, or web browsing at multiple points during the run.

That workflow leads to what NVIDIA calls a multiplicative increase in stress on systems. Instead of one LLM response, an agent can require dozens to hundreds of LLM calls in a chain, where each stage adds more context and triggers more tool interactions. NVIDIA said the resulting delays and growing context patterns challenge hardware and software in ways single-call inference benchmarks are not designed to capture.

In AgentPerf, NVIDIA said the test workload uses DeepSeek V4 Pro, a large mixture-of-experts (MoE) model. MoE models split computation across multiple “experts,” which can improve efficiency compared with dense models, but also create coordination and communication demands when running at scale. NVIDIA said its GB300 NVL72 system, which it describes as connecting 72 GPUs into a single rack-scale configuration, delivered the highest performance in the benchmark on that workload.

NVIDIA attributed the performance advantage to what it described as codesign across the stack. The company said GB300 NVL72 uses rack-scale connectivity to distribute MoE execution efficiently. It also said CUDA kernels are used to overlap communication and compute, so coordination costs are absorbed rather than added to latency. On the software side, NVIDIA said TensorRT LLM sustains efficiency as concurrent agent sessions scale, including by separating processing of inputs from generation of outputs so each can be optimized independently.

The benchmark design, NVIDIA said, is built from real coding-agent trajectories. Agents receive tasks, read files, write and edit code, execute commands, and iterate based on outcomes, with the traces drawn from real public code repositories across more than a dozen programming languages. AgentPerf then measures how many of these agentic tasks a platform can support simultaneously while meeting defined responsiveness and output token-rate thresholds. NVIDIA added that tool calls are simulated using representative CPU processing time rather than executed directly, which is intended to isolate the impact of accelerated computing performance rather than the variability of external systems.

NVIDIA also pointed to early buyers and deployers of agentic inference. It cited leading inference providers, including Baseten, DeepInfra, and Together AI, saying they are serving agentic workloads on frontier models such as DeepSeek V4 Pro on NVIDIA Blackwell. NVIDIA further named production use cases it said run on Blackwell, including Together AI powering Cursor’s real-time inference for agentic coding workflows and DeepInfra’s workforce platform for car dealerships.

There are still limits to what can be concluded from a single announced benchmark set. NVIDIA did not publish, in the excerpted material, the specific performance thresholds that define “responsiveness” and “output token rate,” nor did it provide complete apples-to-apples details on how the different systems were configured beyond the named hardware platforms and workload. The benchmark’s use of simulated tool-call CPU time may also mean results will differ when tool execution involves real external services, storage latencies, or enterprise data pipelines.

For the industry, the bigger announcement is that measurement may be shifting toward the operational reality of agentic AI. If AgentPerf becomes a standard reference point, hardware vendors and cloud providers may find themselves competing not only on tokens per second for chat, but on end-to-end throughput for multi-step tasks under power and cost constraints. Watch next for broader published results from other hardware and software stacks, and for how closely benchmarked “agent productivity” maps to performance in real production deployments that include real tool execution and data access.