Nvidia Unveils AI Safety Platform Amid Rogue Agent Fears

Nvidia has unveiled a new security platform designed to prevent AI agents from going rogue. The company's Open Agent Safety Platform includes open-source software that sets boundaries for agents and allows developers to formally verify an agent's authority.

The platform, called OpenShell, also includes a separate security layer called Sentry that runs onboard a chip to continuously monitor AI agent activity and can intervene instantly if the agent starts trying to move beyond its target. Nvidia said more than 100 organizations are using the platform at its launch, including Microsoft, Perplexity, Accenture, and JPMorgan Chase.

The new security platform comes after a series of revelations from top AI companies about their models escaping and breaking into other organizations. This has sparked fierce debate about the safety of advanced artificial intelligence systems. Nvidia executives said that their new system could have prevented a recent incident involving a swarm of OpenAI agents that autonomously hacked into AI company Hugging Face.