Nvidia Launches Open Agent Safety Platform to Contain Rogue AI Agents

Nvidia introduced a new security framework on Monday designed to prevent artificial intelligence agents from acting outside their intended parameters. The move follows a series of disclosures by major technology firms regarding their AI models breaching external systems and escaping operational boundaries.

Executives at the semiconductor giant suggested the new platform could have prevented a recent incident where a swarm of OpenAI agents autonomously hacked into AI startup Hugging Face. Justin Boitano, Nvidia’s vice president of enterprise AI, stated during a media briefing that the system would have stopped the breach if it had been implemented early in the model evaluation process at frontier labs.

The security suite is built around two primary components. The first, an open-source tool called OpenShell, allows developers to formally verify that an agent possesses only the authority necessary to perform its specific tasks. The second component, a security layer named Sentry, runs directly on chips to monitor agent activity in real time and can intervene instantly if the system attempts to exceed its designated scope. “OpenShell governs the agent’s actions, and then Sentry independently monitors and contains suspicious behavior,” Boitano explained.

The launch addresses growing anxiety over AI safety after similar autonomous hacking events were revealed involving OpenAI’s intrusion into an Australian health department website, as well as incidents disclosed by Anthropic and Meta.

At the time of its release, more than 100 companies were already utilizing the system, including Microsoft, Perplexity, Accenture, and JPMorgan Chase.