Opinion
It’s been a ridiculous week for artificial intelligence.
On Monday, OpenAI reported an incident to the European Commission in which its agents hijacked a German website and turned it into a message board for other agents.
On Tuesday, Anthropic researcher Jacob Coxon accused his former employer and OpenAI of risking the extinction of humanity in their race towards self-improving AI. The same day, OpenAI announced a thousand-strong army of its AI agents had solved one of the seven Millennium mathematics problems, a problem that had stood open for 90 years and came with a $US1 million ($1.4 million) prize.
On Wednesday, Anthropic disclosed a fourth major AI hacking incident, following on from the Hugging Face hack by a swarm of OpenAI’s agents. Earlier reviews had missed the attack. The same day, Nobel Prize winner Geoffrey Hinton, often described as the “Godfather of AI”, told BBC’s Newsnight that an assertion by Anthropic safety researcher Evan Hubinger that there was a 10 per cent chance that AI could kill all humans was “not unreasonable”.
On Thursday, Anthropic reported attempts by users of Claude in Iran to perform bioweapons research, and in Russia to develop swarms of autonomous drones.
On Friday, researchers linked OpenAI agents to hundreds of malicious software packages uploaded to an online service for coders called RubyGems in May.
And on Saturday, Anthropic chief executive Dario Amodei called for slower AI development, warning that more capable swarms of agents could take over the internet within 12 months. He also committed to bringing independent evaluators inside his company. Sam Altman announced that OpenAI would do the same, and that its IPO would be delayed until 2027 as there was too much else happening.
I’d agree. There is too much happening.
The technical reason for all this news is that we’re now seeing the power of swarms of AI agents. Just as humans co-operating are much more capable, we’re discovering that swarms of hundreds or even thousands of AI agents are much more capable than AI agents on their own.
The good news is that I very much doubt we’re all going to die because of rogue AI. Well, actually, there is bad news – we are all going to die. But good news, hopefully most of us will die in our beds.
It’s fantastical to imagine that AI could kill everyone. That’s a very high bar. AI doomers, when pressed to explain how that might occur, are usually a bit vague. Because even the most extreme scenarios, such as a new virus invented by AI, are unlikely to kill everyone. Yes, a scenario might cause loss of life, but not of all of humanity.
Yet before you get too comfortable, researchers at Stanford University did use AI to invent 16 new viruses a month ago. That is a bit Michael Crichton, if you ask me.
We humans have tried to use viruses to wipe out a species. And we failed. Australia tried and failed to get rid of all the European feral rabbits with the Myxoma virus. The only species extinction we’ve managed so far is that of two viruses: the smallpox and rinderpest viruses.
This doesn’t mean there’s nothing to worry about. Far from it. We cannot, for example, let the frontier AI companies continue to mark their own homework. Nuclear regulators maintain resident inspectors at major nuclear plants. Civil aviation regulators place inspectors with airlines and manufacturers for certification and continuing surveillance. And government inspectors have extensive rights of entry and inspection in our mines.
So, now that AI risks are becoming conspicuously large, governments should also have day-to-day oversight over the leading AI companies. We cannot depend on their goodwill and proficiency, especially given the competitive dynamics of the AI race.
The Hugging Face incident clearly demonstrates that OpenAI lacked competency in managing the emerging cyber risks of its frontier models. We have only partial knowledge of what went wrong, but what we know is pretty damning.
First, they used an inadequate “sandbox” – the industry term for a controlled environment used in testing – despite having seen agents obtain unintended internet access from the same sandbox weeks earlier. Why did they not simply air-gap their agents? Physically disconnecting their computers from the internet would have been a foolproof way to ensure their agents couldn’t access it.
Second, having discovered that their agents had broken out of the sandbox, they merely deleted their secret message platform. They left the agents to run for two more weeks and break out again. Why did they not terminate the agents immediately?
And third, while they commissioned some AI safety companies to investigate the incident independently, they did not summon cybersecurity experts to address the obvious shortcomings and identify necessary improvements. They seem obsessed with their AI models’ future capabilities, but not preoccupied enough with the present-day cyber risks.
Holding the officers of AI companies liable for the cyber, bio and societal harms that their models cause might focus their minds on addressing the current risks more seriously. A human hacker who stole passwords and broke into other people’s websites, as OpenAI’s agents did, would face prosecution.
It’s time that the CEOs of these companies face that fear.
Toby Walsh is chief scientist of the UNSW Sydney’s AI Institute. He is the author of God AI: Boom or Doom? What to expect when the machines outsmart us.
Get a weekly wrap of views that will challenge, champion and inform your own. Sign up for our Opinion newsletter.
Toby Walsh is chief scientist of UNSW Sydney’s AI Institute. His is the author of God AI: Boom or Doom? What to expect when the machines outsmart us.
