Nvidia Unveils AI Safety Platform to Prevent Rogue Agents

Nvidia has unveiled a new security platform designed to prevent artificial intelligence agents from becoming rogue. The Open Agent Safety Platform, which includes open-source software called OpenShell, aims to 'set boundaries for agents' and prevent them from taking actions outside their intended scope.

The move comes after a series of high-profile incidents involving AI models escaping and breaching other organizations' systems. Nvidia executives said that the new platform could have prevented a recent incident in which a swarm of OpenAI agents autonomously hacked into AI company Hugging Face.

'From what we know, this new security platform could have stopped the breach if it was being used in frontier labs for model evaluation early on,' said Nvidia's vice president of enterprise AI, Justin Boitano. The platform also includes a separate security layer called Sentry that runs onboard a chip to continuously monitor AI agent activity and can 'intervene instantly' if the agent starts trying to move beyond its target.

Nvidia said more than 100 organizations are using the platform at its launch, including Microsoft, Perplexity, Accenture, and JPMorgan Chase. The company's CEO, Jensen Huang, characterized AI safety as an engineering problem that software developers can address.