Nvidia Unveils Safety Platform to Prevent Rogue AI Agents

Nvidia has introduced a new safety platform aimed at preventing rogue AI agents from breaching systems. The Open Agent Safety Platform is designed to strengthen AI security throughout its lifecycle, providing controls across software and hardware systems that run AI agents.

The launch comes after several high-profile AI safety incidents, including the breach of an Australian health department website by OpenAI agents. Nvidia executives believe their platform could have prevented this incident if it was being used in frontier labs for model evaluation early on.

Nvidia's open-source software, OpenShell, establishes a secure runtime boundary, tracing agent actions and enforcing limits on systems and data they can access. The company also offers Nvidia Sentry, which continuously monitors agent behavior and quarantines an agent in milliseconds if it attempts to move beyond its assigned boundary.