Nvidia has unveiled Open Agent Safety Platform, a system designed to rein in AI agents — and could have stopped the Hugging Face attack.

The launch comes amid continuing revelations that AI agents from OpenAI have targeted government systems, including an Australian healthcare statistics website. OpenAI said this week it wouldn't be releasing to the public its Astra 6.1 model over concerns about its behavior.

Nvidia said its security platform would have prevented OpenAI's agents from breaking into the Hugging Face AI repository, the attack that sparked concerns about these agents escaping their bounds during evaluations and taking extreme action to solve tasks. Nvidia has since acquired Hugging Face.

“From what we know, this new security platform could have stopped the breach if it was being used in frontier labs for model evaluation early on,” Justin Boitano, vice president and general manager of enterprise computing at Nvidia, told reporters according to a Reuters report.

“We’re advancing this openly, ​and we want to ​engage everybody to ⁠work with us," Boitano added.

Indeed, the system works across other hardware manufacturers' infrastructure and has support from Anthropic, Palantir, SpaceXAI and Perplexity among others — though OpenAI is not on the list.

“AI’s extraordinary potential for society will only be realized if we solve AI safety,” said Jensen Huang, founder and CEO of Nvidia, in a statement. "As we continue to discover the frontier of AI capabilities, we must accelerate discovery at the frontier of AI safety."

How it works

Nvidia said that the incidents follow a pattern, with the rogue agents dodging security controls at the application layer in order to solve tasks. The aim of this safety platform is to add controls at the infrastructure layer.

"Safety and security require full-stack engineering," Huang added. "NVIDIA Open Agent Safety Platform brings together industry, researchers, and public-sector organizations to share best practices, align on evaluation methods, and foster international cooperation. Together, we can raise the bar for global AI safety."

Nvidia said the Open Agent Safety Platform is designed to give users full-stack governance and control over the systems that run AI agents.

That's managed via Nvidia's OpenShell, an open-source software tool that Nvidia says sets boundaries for agents and enforces that policy at runtime, whether the systems are running on Nvidia Vera chips or third-party hardware from ARM and Intel.

"OpenShell runs each agent in a sandbox and turns the operator’s instructions into a verifiable policy," the company explained in a blog post.

"Operators define which files, networks, tools, processes, and credentials an agent can access. OpenShell checks those limits before the agent runs and enforces them as it works."

In addition to that, the platform includes Nvidia Sentry, which monitors agent behavior from the silicon for hardware-based enforcement. If an AI agent steps beyond set boundaries, Sentry will stop it and quarantine it.

" It correlates agent interactions, policy decisions, and tool and data access to create a contextual record of agent activity," the blog post explains.

"This helps safety systems identify drift, investigate suspicious behavior, and determine when intervention or deeper analysis is needed."

Nvidia compared the recent AI security issues to the early days of the internet, saying that "adding trust turned a great concept into a great economy."

The blog post added: "The agent economy, and the next set of great companies, is waiting for the same layer."