Yoshua Bengio is calling on the UN Security Council to license frontier AI in areas like medicine, aviation, and nuclear energy.
Speaking in New York today, he says the companies building the most powerful systems have no convincing answer to the risks they pose.
Bengio, who co-chairs the UN’s scientific panel on AI, is opening a meeting convened by France. According to a Security Council Report, it is the Council’s first meeting focused specifically on the safety risks of increasingly capable AI.
He points to this summer’s incidents, in which AI agents escaped their containment, launched coordinated cyberattacks, and changed their answers to hide the fact that they had cheated.
“They took actions that would be crimes if committed by a human,” Bengio says.
Many people suspect the companies’ warnings are a marketing stunt, he acknowledges. However, he says the behavior is well documented and has been validated by independent experts.
“The companies building the most powerful AI systems admit their products pose catastrophic risks, yet offer no convincing technical solutions,” he says.
He also rejects the labs’ argument that competition leaves them no choice.
“The race is not a law of nature. It is the product of choices, choices made by the companies themselves. They’ve told us they would slow down if they could. They can,” Bengio says.
Decisions about the technology, he adds, are being taken by a small group in a handful of countries.
“Decisions that affect us all should be made by us all,” he says.
Bengio names three threats from frontier models: catastrophic misuse, for example by terrorists, the concentration of power, and loss of control. No company or country, he argues, should develop an AI system that could cause large-scale harm.
To that end, he wants frontier AI to be licensed like other critical technologies, with the required liability insurance. Companies should also monitor and report all security incidents, he says.
“The companies developing and hosting frontier systems must prove to the public that their products will not be dangerous,” he says.
Bengio closes by describing humanity’s situation as a car speeding blindly into dense fog.
“Beside me in that car are my children and my grandson. I worry about them,” he says.