Nvidia faces scrutiny over AI chip performance in “decode” stage for very large models

NVIDIA is drawing sharper attention from AI infrastructure buyers on a specific part of how large language models generate text: the decode step. In comments reported by Yahoo Finance, NVIDIA Compute CTO Jason Goodison suggested the company’s chips “really struggle” when tasked with decoding for extremely large models, even as the broader market has increasingly optimized for speed and throughput.

The decode stage is the part of inference in which a model produces new tokens one at a time (or in small groups). For customers deploying chat and assistant applications, decode performance can become a bottleneck when requests are long, models are large, or concurrency is high. The reported critique implies that, for some workloads, raw model size and generation settings may expose limits in current hardware-software stacks.

Goodison’s remarks also introduced a pricing and packaging concept tied to latency. He said that NVIDIA can sell “fast tokens” at a premium, with the reported figure ranging from 3 to 5 times the price of slower tokens. “Fast tokens” in this context refers to capacity or compute pathways that deliver lower generation latency for certain token production, which can be valuable for user experience and real-time interactions.

That premium claim matters because it suggests NVIDIA’s economics may hinge not only on overall performance, but on how performance is tiered across different token-generation speeds. If decoding for very large models constrains throughput, the ability to segment capacity into faster and slower streams could become central to monetization, particularly when customers demand consistent responsiveness during peak usage.

However, the market impact depends on how broadly the decode issue affects real deployments. NVIDIA did not, in the reported segment, lay out specific benchmarks, model sizes, latency targets, or deployment scenarios. The comments therefore raise questions rather than provide a full performance verdict for all workloads.

NVIDIA, which sells AI GPUs and related systems used by cloud providers and large enterprises, has built its recent positioning around accelerating training and inference for increasingly large AI models. Its public communications routinely emphasize end-to-end performance and the ability to run a wide range of model families. Still, the decode-focused critique points to an area where buyers may press for clearer comparative data, including how well a given configuration performs during sustained decoding for long outputs.

The reported pricing angle may also influence how customers negotiate. If fast token capacity is priced at a multiple, buyers will likely want to verify what portion of their workloads can realistically qualify for fast-token service and how much performance varies as prompts, context length, and output lengths change.

What to watch next is whether NVIDIA addresses the decode-performance question with additional specificity, such as controlled benchmark results across model sizes, memory and batching configurations, and measurable latency and throughput during decoding. Buyers will also likely monitor how “fast tokens” are defined in contracts and capacity plans, and whether the fast/slow segmentation becomes more prominent in public performance comparisons.