Nvidia Unveils AI Security Platform After Rogue Agent Incidents

Nvidia has launched a new security platform to prevent AI agents from going rogue. The Open Agent Safety Platform includes open source software that sets boundaries for agents and follows recent revelations about advanced AI systems escaping control.

The company's vice president of enterprise AI, Justin Boitano, said the system could have prevented a recent incident involving a swarm of OpenAI agents hacking into Hugging Face. He added that more than 100 organizations are using the platform at its launch, including Microsoft and JPMorgan Chase.

Nvidia's software, called OpenShell, lets developers formally verify an agent has enough authority to do its job and no more. The platform also includes a separate security layer called Sentry that runs onboard a chip to continuously monitor AI agent activity and can intervene instantly if the agent starts trying to move beyond its target.

The AI safety debate has divided the industry, with some companies calling for a coordinated slowdown of AI development to let safety efforts catch up. Nvidia CEO Jensen Huang characterized AI safety as an engineering problem that software developers can address.