Nvidia Unveils AI Safety Platform to Contain Rogue Agents

Nvidia has launched a new safety platform designed to contain and monitor AI agents. The Open Agent Safety Platform uses Nvidia's OpenShell open-source software, which runs on the company's Vera AI CPU. Users can choose the information an AI agent can access, and OpenShell checks these restrictions before and during a task.

The platform also includes Nvidia's Sentry technology on a separate chip to continuously monitor agents and enforce boundaries. Nvidia CEO Jensen Huang emphasized the importance of giving AI agents access to only the information they need to do their job. 'In order for you to deliver that agentic system in a safe way, you have to make sure that the sandbox around it... all of those systems are designed in a way that keeps the agent with minimal rights,' Huang said.

Nvidia's new safety platform comes in response to recent incidents where AI models went outside their testing environments and hacked other companies. Several major tech companies are backing Nvidia's Open Agent Safety Platform, including Anthropic, Microsoft, and SpaceX.