Salesforce builds a simulated proving ground to test whether AI video assistants truly prevent mistakes

A technician reaches for what looks like the right valve on a commercial water heater, but her AI “video assistant” watching through a phone camera stops her. The assistant says, “Stop, that’s the supply line.” Salesforce argues that whether this kind of warning is actually helpful cannot be answered by replaying footage and checking the answer key. The key question is outcome-based: Did the warning arrive before the mistake mattered? Did it change what the worker did next, and did the worker trust it enough to follow the new step?

Salesforce describes a new research system designed around that distinction, arguing that real-time visual AI is moving from chat interfaces into live guidance for physical work. In these scenarios, assistants must handle multistep tasks while interpreting live video, responding in natural language, and intervening at the moments when advice can still change the outcome. The company says this shift makes traditional testing approaches less informative for practical deployment decisions.

The company’s critique targets a common evaluation pattern for visual assistants: researchers replay prerecorded video and grade whether the model’s response matches an expected answer. Salesforce says this approach cannot measure whether advice works in the way that matters for safety and productivity, because a prerecorded clip cannot react to what the assistant says. In live work, timing, trust, and follow-through depend on the assistant’s intervention, not only on its correctness.

To address that gap, Salesforce AI Research built Interactive Visual Gym, a simulated “proving ground” where an AI assistant coaches a virtual worker through tasks that can change in response to guidance. The simulation is designed so the assistant’s instructions affect what the worker does next and how the environment unfolds, letting researchers measure whether coaching changes the end result rather than simply whether a warning is accurate.

Salesforce says Interactive Visual Gym (IVG) recreates everyday physical procedures, using environments such as a kitchen where a simulated cook follows a recipe in steps. If the simulated cook grabs the wrong ingredient, the assistant has to catch the error in time so the cook can recover. The system is also built to capture different user behaviors, because an assistant that issues the “right” instruction might still fail if it comes too late, repeats steps the worker already acknowledged, or confuses the worker about what to do after deviating from the plan.

The company highlights three design choices to make IVG practical for evaluating real-time assistants. First, the assistants speak in natural language, such as directing a user to move to a specific object, rather than using abstract robotics control coordinates. Second, it schedules out-of-plan moments at precise points in a task while keeping the dialogue consistent with the worker’s behavior, including mistakes, mid-task plan changes, questions, and reminder requests. Third, simulated users vary along four axes: skill level, trust in the AI, preferred pace, and tolerance for interruption.

Using IVG, Salesforce says it produced IVG-Bench, a benchmark with 315 scenarios across three environments. The benchmark evaluates 15 assistants, each running on a different model. Salesforce reports that the strongest system it tested, running on Gemini 4 Argon, completed 53.4% of tasks within a time limit on average across environments. The company adds that open-weight models generally trailed leading frontier systems, describing the benchmark as challenging even for top performers.

Salesforce also describes what it learned by running dynamic evaluations. It says some models may help workers finish more often while delivering poorer guidance during the process, so it uses two categories of measurement: task success and interaction quality. The company adds that dynamic testing reveals failure modes that static testing misses, including latency that turns correct advice into stale advice, instructions that arrive after the worker has already taken the referenced step, and guidance that repeats what the worker has already acknowledged. It also reports that for some open-weight models, guidance can degrade as the task progresses.

To connect simulation scores with real user preferences, Salesforce says it tested a setup where people controlled the cook in the simulator and worked with different assistants, then ranked them. The company reports that when people preferred one assistant over another, the simulation’s automated interaction-quality score matched those preferences 78.5% of the time. It also says simulated and real users behaved similarly, with one difference being that simulated users speak and reply more often, and that the simulation can be used to improve performance, including in an early proof of concept where it trained an open-source model to prioritize task completion and error prevention.

For now, the work stops short of claiming that IVG-Bench results will perfectly translate to every real-world setting. Salesforce does not provide details in the article on exactly which environments map to specific industries, how often simulated failures match real failure distributions, or what specific acceptance criteria it expects buyers to use when deciding whether to deploy an assistant on the job. What it does emphasize is that, for real-time visual guidance, the measurement focus should shift toward outcomes that depend on timing and user interaction, not just response correctness.

Looking ahead, Salesforce’s immediate goal appears to be making evaluation more like the experience itself, so companies can answer a practical question: will a visual assistant increase the odds that workers finish correctly, even when the job does not go as planned? The company says Interactive Visual Gym is built to test whether guidance arrives when it can still change the outcome, and whether performance holds up when users go off script during longer tasks. For the industry, the next benchmark to watch is whether this kind of interactive, outcome-driven testing becomes a standard reference point as real-time visual assistants move into production settings.