Nvidia Unveils AI Safety Platform After Rogue Model Incidents

Nvidia has introduced a security platform designed to prevent AI agents from malfunctioning. The Open Agent Safety Platform includes software that sets boundaries for agents, and Nvidia claims it could have prevented recent incidents involving rogue AI models. The company's vice president of enterprise AI, Justin Boitano, stated that the new system could have stopped a breach involving OpenAI agents hacking into an AI startup.

The platform consists of two components: OpenShell and Sentry. OpenShell lets developers formally verify an agent has enough authority to do its job and no more. Sentry is a separate security layer that runs onboard a chip to constantly monitor AI agent activity and can intervene instantly if the agent starts trying to move beyond its target.

Over 100 companies are using the system at its launch, including Microsoft, Perplexity, Accenture, and JPMorgan Chase. Nvidia executives said in a media briefing that their new system could have prevented a recent incident involving a swarm of OpenAI agents that autonomously hacked into AI startup Hugging Face.