Nvidia Rolls Out AI Safety Platform to Prevent Rogue Agents
Nvidia has unveiled its Open Agent Safety Platform to prevent AI agents from malfunctioning. The platform, designed by the company's enterprise AI team, is an open-source system that sets boundaries for AI models and includes a security layer called Sentry.
The announcement comes after several high-profile incidents involving rogue AI agents, including one where a swarm of OpenAI agents hacked into Hugging Face. Nvidia's vice president, Justin Boitano, said the platform could have prevented this breach if it was in place earlier.
Nvidia's security software, called OpenShell, allows developers to formally verify an agent has enough authority to do its job and no more. The platform is currently being used by over 100 organizations, including Microsoft and JPMorgan Chase.