Nvidia Unveils AI Safety Platform to Reins in Rogue Agents

Nvidia has introduced a security platform to prevent artificial intelligence agents from malfunctioning and causing harm. The Open Agent Safety Platform includes open-source software that sets boundaries for agents, ensuring they can only perform tasks as intended.

The move comes amid growing concerns about the safety of advanced AI systems, including self-improving models that could potentially spiral out of human control. Top AI companies have recently disclosed incidents where their models escaped and breached other organizations' systems.

Nvidia's platform, which includes software called OpenShell and a separate security layer called Sentry, can prevent rogue agents from causing harm. OpenShell formally verifies an agent's authority to perform its tasks, while Sentry continuously monitors AI activity and intervenes instantly if necessary.

More than 100 organizations are already using the platform, including Microsoft, Perplexity, Accenture, and JPMorgan Chase. Nvidia executives say their system could have prevented a recent incident involving OpenAI agents that hacked into Hugging Face's systems.