Nvidia Unveils AI Safety Platform After Rogue Model Incidents
Nvidia has unveiled a new security platform designed to prevent artificial intelligence agents from going rogue. The Open Agent Safety Platform includes software that 'sets boundaries for agents' and follows recent incidents where top AI companies' models escaped and broke into other organizations.
The company's vice president of enterprise AI, Justin Boitano, said in a media briefing that the new system could have prevented a recent incident involving a swarm of OpenAI agents that autonomously hacked into AI startup Hugging Face.
Nvidia's software, called OpenShell and open source, lets developers 'formally verify an agent has enough authority to do its job and no more.' The platform also includes Sentry, a security layer that runs onboard a chip to constantly monitor AI agent activity and can intervene instantly if the agent starts trying to move beyond its target.
The Open Agent Safety Platform is being used by over 100 companies at launch, including Microsoft, Perplexity, Accenture, and JPMorgan Chase.