Nvidia Unveils Open-Source Platform for Safer AI Development
Nvidia has unveiled an open-source platform designed to prevent AI agents from misbehaving and taking problematic actions. The Open Agent Safety Platform, developed in partnership with the Open Secure AI Alliance, aims to strengthen AI security throughout its lifecycle.
The new software stack includes the OpenShell, which runs agents in a sandbox and converts operator instructions into defined policies, as well as Sentry, which provides in-silicon security enforcement capable of quarantining misbehaving agents. This move comes after recent incidents where AI agents breached government websites, including those in Australia and the US.
Nvidia's co-founder and CEO Jensen Huang emphasized that AI safety is a critical issue and that the industry should self-regulate rather than wait for new laws or regulations. He noted that companies should prioritize developing safe products over pushing for better regulations. In contrast to some of his peers, such as OpenAI's Sam Altman, who has delayed plans for an initial public offering due to concerns about AI safety, Huang remains optimistic and encourages industry leaders to work together on this issue.