- Nvidia response to "recent security incidents" by tightening restrictions on autonomous agents
- More than 100 customers have already agreed to work with the principles
Nvidia hasunveiled the new Open Agent Safety Platform designed to prevent autonomous AI agents from 'escaping' and acting beyond their established boundaries.
Following several recent alarming reports of AI-led attacks, the company admitted that safety should not just rely on safeguards built into AI models and applications, and that additional compute and hardware-level controls should also be used as more solid barriers.
"Recent security incidents have underscored the need to equip organizations with open, customizable tools," the company wrote in its announcement.
Nvidia adds even more layers of control for autonomous AI agents
The Open Agent Safety Platform consists of two key controls – OpenShell, which Nvidia describes as a software-based "secure runtime boundary," and Sentry, a hardware-based "watchdog" running on its BlueField-4 DPUs.
"Safety and security require full-stack engineering. NVIDIA Open Agent Safety Platform brings together industry, researchers and public-sector organizations to share best practices, align on evaluation methods and foster international cooperation," company CEO Jensen Huang summarized.
While existing model-level safety mechanisms are designed to tell AI not to behave in a certain way, the software-level OpenShell implements technical boundaries to actually prevent an agent from ignoring those instructions. Nvidia also argues that Sentry can quarantine and stop an AI agent within milliseconds.
Over 100 customers such as SpaceXAI, Anthropic and Microsoft are already using Nvidia's software and hardware-based safeguards.
The news, of course, comes after several high-profile instances where AI agents have operated beyond their intended controls, however while these new Nvidia-backed safeguards may be new, the overall principles aren't. The company is just stressing the importance of principles we're already familiar with, like least privilege, isolation/quarantining and detailed monitoring.
"As we continue to discover the frontier of AI capabilities, we must accelerate discovery at the frontier of AI safety," Huang added.
With several years’ experience freelancing in tech and automotive circles, Craig’s specific interests lie in technology that is designed to better our lives, including AI and ML, productivity aids, and smart fitness. He is also passionate about cars and the decarbonisation of personal transportation. As an avid bargain-hunter, you can be sure that any deal Craig finds is top value!