Salesforce builds simulated gym to test real-time visual AI assistants

Salesforce AI Research introduced Interactive Visual Gym, a simulated proving ground for AI assistants that watch live video and coach people through physical tasks. Rather than grading answers against prerecorded clips, IVG pairs tasks such as cooking or adjusting phone settings with simulated users whose actions change in response to the assistant's spoken advice, so researchers can see whether guidance alters the outcome. The team also built IVG-Bench, 315 scenarios across three environments, and used it to evaluate 15 assistants each running on a different model. The strongest, on Gemini 4 Argon, completed 53.4% of tasks within the time limit. In an early proof of concept, training an open-source model inside the simulator improved its success rate.