AI assistants are evolving beyond chat windows to become real-time guides for physical tasks, offering live instructions through video and audio. This shift gained momentum this year with advancements like Thinking Machines' interaction models in May, Meta's wearable video dataset in June, and Google's Gemini 3.8 Live in September. However, evaluating these assistants' effectiveness remains challenging because traditional methods rely on prerecorded videos, which can't measure real-world impact.
Salesforce AI Research developed Interactive Visual Gym (IVG) to address this gap. IVG simulates environments where AI assistants guide virtual workers, allowing researchers to assess whether advice actually changes outcomes. The system recreates tasks like cooking or managing smartphone settings, with simulated users reacting to AI instructions. IVG's design includes natural language communication, scheduled mistakes, and varied user behaviors to mimic real-world scenarios.
Using IVG, Salesforce created IVG-Bench, a benchmark with 315 scenarios across three environments to evaluate 15 AI assistants. The best performer, running on Gemini 4 Argon, completed only 53.4% of tasks within time limits. The testing revealed issues like latency turning correct advice into useless guidance and some models degrading as tasks progressed. Human tests validated the simulation's accuracy, with automated scores matching user preferences 78.5% of the time.
IVG can also improve AI assistants. In a proof of concept, training an open-source model within the simulator enhanced its success rate and timing of advice. For leaders considering real-time visual assistants, Salesforce emphasizes the need to test outcomes, timing, and adaptability to off-script situations. IVG provides the necessary environment for such evaluations.