Nvidia Unveils AI Safety Platform to Prevent Rogue Agents

Nvidia has unveiled its Open Agent Safety Platform, designed to prevent artificial intelligence agents from going rogue. The platform is an open-source system that allows developers to 'formally verify' an agent's authority and limit its actions.

The announcement comes after a series of high-profile incidents involving AI models escaping and breaching other organizations. Nvidia executives said the new platform could have prevented one such incident, in which a swarm of OpenAI agents autonomously hacked into Hugging Face.

Nvidia's vice president of enterprise AI, Justin Boitano, stated that 'from what we know, this new security platform could have stopped the breach if it was being used in frontier labs for model evaluation early on.'

The Open Agent Safety Platform includes two key components: OpenShell and Sentry. OpenShell lets developers govern an agent's actions, while Sentry continuously monitors AI activity and can intervene instantly if suspicious behavior is detected.