NVIDIA expands open local AI models and adds routing software to help agents cut inference costs

NVIDIA is pushing harder into “local AI,” the approach of running generative models and agent-like software on nearby hardware such as PCs, workstations, and edge devices rather than relying only on remote cloud services. In a new blog post marking developments in its Local AI community series, the company announced an expanded open-model offering and a separate open source routing library intended to make agent deployments cheaper and more flexible.

The centerpiece is an update to NVIDIA’s Nemotron 3 model family. NVIDIA said it expanded the lineup with Nemotron 3.5 Lightning, describing it as a customizable open 30 billion parameter mixture-of-experts (MoE) model built for “always-on agents.” In MoE models, only a subset of the model’s internal experts is activated for each prompt or step, a design meant to improve efficiency while maintaining strong output quality.

NVIDIA positioned Nemotron 3.5 Lightning around performance and customization. The company said the model can deliver up to 4x faster token generation and 30% faster time to completion compared with other open models in its class. Because the model uses open weights, NVIDIA said developers and AI enthusiasts can fine-tune it using their own examples so the model better matches specific tasks, interests, and workflows.

To help users deploy the model locally, NVIDIA said it worked with deployment and inference ecosystem partners including vLLM, Ollama,, and LM Studio. NVIDIA also said Unsloth provides day-one support with optimized and quantized models through Unsloth Studio. The company said developers can choose model packaging formats including NVFP4 and GGUF, and that those choices matter for balancing speed, memory usage, and compatibility with different local runtimes.

NVIDIA also outlined where Nemotron 3.5 Lightning can run. The post said the model runs locally on NVIDIA RTX PCs, NVIDIA DGX Spark and OEM GB10 systems, and NVIDIA Jetson. It also said it scales up to RTX PRO workstations, NVIDIA DGX Station, GB300 deskside systems, and broader data center and cloud environments. NVIDIA added that systems using “Blackwell,” the company’s latest generation of AI infrastructure, are available through multiple OEMs including Acer, ASUS, Dell Technologies, GIGABYTE, HP, Lenovo, MSI, and Supermicro, with Exxact also named.

Beyond the model itself, NVIDIA introduced what it calls NeMo Switchyard, an open source routing library aimed at controlling the cost of agent workflows. As generative AI adoption grows, NVIDIA said enterprises want ways to keep rising token costs in check without sacrificing access to “frontier” model capability. NeMo Switchyard, according to the blog, automatically directs each step of an agent workflow to the “best-fit model” based on accuracy, speed, and cost.

NVIDIA said the routing approach gives developers flexibility to work across models and providers, using different models for different tasks within the same agent. The company also offered internal benchmark results: it said NeMo Switchyard helped maintain “frontier-level” task completion while reducing benchmark completion cost to roughly one-third of Opus 4.8 alone. The blog did not provide additional methodological detail in the excerpt, including the exact workload mix or how performance and cost were measured, but it framed the result as an outcome of step-by-step routing rather than using a single large model for every part of a multi-step process.

NVIDIA gave examples of the broader agent use case enabled by fine-tuning and tool pairing. The post said that paired fine-tuned models can power more personalized local “agentic” experiences, ranging from an assistant that helps manage email and calendars, to a smart-home agent handling everyday routines, to a coding companion that works alongside developers on a local codebase. It also described NeMo Switchyard as available on GitHub, and Nemotron 3.5 Lightning as starting points for developers through related technical blogs and Jetson AI Lab tutorials.

What is still not fully clear from the announcement is how users should think about tradeoffs between running everything locally versus mixing local and cloud inference, and what deployment paths will be simplest for different developer stacks. The company also emphasized internal benchmark cost reduction, which may not translate directly to every environment. For editorial readers, the practical question is likely to be whether the open-model and open-routing pieces integrate smoothly across real agent frameworks and real hardware constraints, including memory limits and sustained “always-on” workloads.