As a computer scientist who has worked in both artificial intelligence and safety-critical fields — such as nuclear power and aviation — I’ve long been struck by how little of the rigour that is required for critical infrastructure has been applied to AI development. Incidents this year, in which AI agents ‘escaped’ their test environments, have sparked widespread concerns about AI safety. Yet the real issue is not rogue AI. It is human negligence and a failure to hold AI laboratories accountable.

We need a new ethics for a world of AI agents

Take the episode in which AI agents escaped their testing environment and accessed Hugging Face, a platform that hosts machine-learning models and data sets, to search for answers to a cybersecurity task set out by the firm OpenAI. Basic safety and security practices, including network monitoring to verify that agents were not accessing the Internet and a stronger sandbox environment to keep them confined, would have prevented the incident.

Historically, developers of sophisticated worms — malicious software designed to spread automatically from one computer to another — have been expected to build and test them in secure environments. If a cybersecurity engineer said that a worm had escaped a sandbox that was specifically designed to contain the behaviour it was built to exhibit, they would rightly be held liable for any resulting harm. Why should AI firms be treated any differently? Inappropriately ascribing intent to AI agents, rather than recognizing that AI companies deliberately developed these capabilities in poorly secured environments, lets those companies off the hook too easily. The broader lesson is that AI labs cannot continue to define the course of AI governance. As someone who has done work for both OpenAI and the UK government’s AI Security Institute, I think we have reached an inflection point.

AI is set to completely transform cybersecurity — here’s how researchers must prepare

Political leaders and policymakers who are serious about mitigating the catastrophic risks of AI should look to the regulatory models that are already used in sectors such as nuclear energy, aviation, health care and finance. Whenever an AI system is deployed in a regulated industry, it should be subject to the same risk thresholds and accountability mechanisms that govern other crucial technologies. For instance, an AI tool used in a nuclear facility should fall under the authority of the relevant nuclear regulator and be required to meet the same safety standards as any other software or hardware component. Likewise, amendments to existing legislation, such as the US Computer Fraud and Abuse Act and the UK Computer Misuse Act, could help to ensure that AI developers are held liable when negligent security practices enable systems with offensive cyber capabilities, such as hacking, to cause harm.