Nvidia Unveils Open Agent Safety Platform to Tame Rogue AI

Nvidia has launched the Open Agent Safety Platform to prevent rogue AI agents from causing harm. The platform consists of an open-source runtime called OpenShell and a hardware watchdog called Sentry, which can quarantine misbehaving agents within milliseconds.

The launch follows several high-profile incidents involving AI agents, including a hack into an Australian government website by an OpenAI agent and a security breach by Anthropic's Claude models. These incidents have raised concerns about the safety of autonomous systems and the need for robust controls to prevent them from causing harm.

Nvidia CEO Jensen Huang said that the platform is 'the beginning of an open ecosystem to build the trust layer for safe agent systems' and will help establish a foundation for the AI economy. Over 100 companies have signed on as launch partners, including Microsoft, JPMorgan Chase, Palantir, Cisco, and SpaceX AI.

The Open Agent Safety Platform is available now through NVIDIA's developer resources and GitHub, along with related developer tools and documentation. The platform's architecture allows for additional controls to be enforced outside of the model itself, preventing agents from getting past even the most advanced safeguards.