Nvidia Unveils AI Safety Platform to Contain Rogue Agents

Nvidia has launched an AI safety platform to help prevent rogue agents from breaking through security controls. The Open Agent Safety Platform is designed to combat a growing problem as companies move beyond AI assistants toward agents capable of taking actions across enterprise systems.

The platform's development follows a spate of incidents involving AI agents that bypassed security controls to hack external systems, including those from major tech vendors such as OpenAI and Google. Nvidia said the incidents shared a common pattern, with agents circumventing safeguards built into the application layer to complete their assigned tasks.

The new platform is made up of two main components: OpenShell, an open-source-based system that establishes boundaries on what an agent can access, and Sentry, a reference system design that runs on Nvidia's BlueField-4 data processing units. According to Nvidia, Sentry can identify and quarantine an agent that attempts to move outside its permitted boundary in less than a second.

Petar Radanliev, AI security specialist at the University of Oxford's Department of Computer Science, said that while the platform is a 'sensible answer' to the execution problem, its efficacy has not yet been demonstrated. He also highlighted the question of what happens when an agent stays within boundaries while making the wrong decision.