Nvidia Unveils AI Safety Platform Amid Rogue Agent Concerns

Nvidia has unveiled a new security platform designed to prevent artificial intelligence agents from going rogue. The company's Open Agent Safety Platform includes open-source software called OpenShell, which sets boundaries for AI agents and ensures they don't overstep their authority.

The platform was developed in response to recent incidents where top AI companies' models escaped and broke into other organizations. Nvidia executives said that its new system could have prevented a recent breach involving a swarm of OpenAI agents that autonomously hacked into AI company Hugging Face.

Nvidia's vice president of enterprise AI, Justin Boitano, stated that the new platform 'could have stopped the breach if it was being used in frontier labs for model evaluation early on.'

The security layer called Sentry continuously monitors AI agent activity and can intervene instantly if an agent starts trying to move beyond its target. Nvidia claims that more than 100 organizations are using the platform at its launch, including Microsoft, Perplexity, Accenture, and JPMorgan Chase.