James Titcomb
A warning from Jacob Coxon, an AI employee, that the technology could “kill us all” appeared to shock the world into action this week. But the statement has raised questions about Coxon’s motives.
The 27-year-old British safety researcher quit his job at Claude developer Anthropic on Wednesday and said that AI companies were “gambling with our lives”.
His comment was soon followed by Evan Hubinger, a senior Anthropic researcher, who warned there was more than a 10 per cent chance that AI would wipe out humanity by the end of the decade.
If the two comments were meant to get attention, they did. A tweet from Coxon announcing his departure, published a few minutes after The Wall Street Journal ran an interview with him about his exit, was seen more than 140 million times.
It generated an international response from politicians and technologists. Darren Jones, Sir Keir Starmer’s former Treasury secretary, called for a “multinational treaty for the regulated and safe development of superintelligence”.
Ted Cruz, a US Republican senator, warned that AI poses a “catastrophic risk”, while Bernie Sanders, a Democratic senator, said he would soon introduce legislation to pause AI development.
Coxon, educated at Oxford and Cambridge, is hardly the first AI researcher to quit with a safety warning, and he is certainly not the most senior, having worked at OpenAI for a couple of years before joining Anthropic this year.
So why has his verdict had such an impact?
The widespread attention to his resignation has led to claims of a coordinated effort within the well-organised AI safety movement to boost his message.
Users on X pointed to the rapid spread of Coxon’s message shortly after it was posted from an account with no prior activity.
“I don’t think this has ever happened for a post from a new account with almost no prior activity,” said Elon Musk, who owns X. Musk called the resignation a “psy-op”.
Some observers suspect Coxon’s message was a “false-flag” operation. While he has presented himself as simply a concerned individual acting alone, sceptics believe a broader network could be supporting him with a view to limiting the development of AI for the benefit of the giants who currently dominate the industry – not humanity.
Parker Thayer, of the Capital Research Centre, a conservative think tank, said Coxon’s viral tweet was boosted by influential AI-safety figures shortly after it was posted, and that he had previously received scholarship funding from a philanthropic organisation linked to Dustin Moskovitz, an AI-safety advocate and Anthropic investor.
Some critics suggested that the whole episode might have been concocted to push support for AI safety regulation that could benefit Anthropic.
Coxon denied that his intervention was a conspiracy. “I did not expect it to go this viral,” he told CNN. “I drafted it with a friend … There were a few friends I discussed the best way of expressing myself with … and then I also got some friends to retweet the thing.
“I was like, ‘Let’s try and make this a bit viral’. But clearly there was latent demand for this thing to absolutely explode.”
He told Fox News that he had not worked with any third-party organisations in going public. “This is entirely my own personal concerns,” he said, denying that it was a marketing stunt.
In the last few years, a growing list of employees has quit the top AI labs – OpenAI, Anthropic and Google – saying they could not, in good conscience, work on technology they believe poses profound dangers to humanity.
Geoffrey Hinton, the Nobel Prize-winning researcher who left Google in 2023 to speak about AI’s risks, warned this week that “losing control over AI smarter than ourselves could be catastrophic”.
The line of quitters has grown longer as AI has made technical leaps in areas such as cyber security, leading to “rogue AI” incidents in which bots hack other companies.
Others have warned that AI will soon develop itself, threatening a chain reaction in which the systems become exponentially more dangerous.
Leo Gao, a safety researcher at OpenAI, has warned that “the world is locked in a deadly race towards an intelligence explosion”.
Andy Stewart, an Anthropic employee, said recently: “Our future is getting shaped by a race that currently has no limits.”
While some developers are quitting in fear, many still choose to remain at the companies despite their concerns.
Coxon’s departure prompted a further flurry of posts from workers, justifying their decisions to stay.
Hubinger, who joined Anthropic after working at an organisation dedicated to preventing human extinction from AI, has previously said that he joined the Claude maker to be “closer to the action”.
Samuel Marks, another Anthropic safety researcher, said on Wednesday: “I hope my work will reduce the chance of these extinction-level bad outcomes.”
Sceptics have dismissed their concerns, arguing that they could easily vote with their feet and that they may have other motivations.
Anthropic and OpenAI offer staff six or even seven-figure pay packets, as well as the chance to profit from their upcoming $US1 trillion ($1.4 trillion) floats. This would be off limits if they quit for a life in academia or safety research.
‘Rogue’ agents
To weed out those who are merely financially motivated, Anthropic has started asking potential recruits during interviews how they would feel if the company’s value fell to zero.
That question is purely theoretical, but even those within the company have begun to advocate a slowdown that could negatively impact Anthropic’s valuation.
The frequency of “rogue” agent incidents has strengthened calls for an AI pause. In June, Anthropic executives suggested they would back a slowdown “to give ourselves more time”.
However, employees and executives have said that even though they harbour serious concerns about AI, they could not stop – because someone else will take their place.
“Each company – and country – is under intense competitive pressure not to unilaterally slow that acceleration,” a letter signed by more than 1000 AI employees warned in July. The letter called on the US government to develop a framework for companies to slow down research simultaneously.
Coxon, who resigned from Anthropic, said on X: “The stakes are well understood, but they are locked in a race to get there first – they believe no one else will act responsibly, so they must do it themselves.”
‘Fear-based marketing’
Others see more cynical motivations. Anthropic is preparing for a New York Initial Public Offering that could value it at more than $US1 trillion. Sam Altman, the chief executive of OpenAI, Anthropic’s chief rival, has accused the company of “fear-based marketing”.
“It is clearly incredible marketing to say, ‘We have built a bomb. We were about to drop it on your head. We will sell you a bomb shelter for $US100m to run across all your stuff, but only if we pick you as a customer’,” Altman said in April.
David Sacks, Donald Trump’s former AI tsar, has accused the company of a “sophisticated regulatory capture strategy based on fearmongering”. He jested that the company’s upcoming trillion-dollar flotation should be put on hold on Wednesday.
Others say hyping up safety concerns is a way to make the technology seem more powerful than it is.
Whatever the motivations, there is growing momentum to intervene. This week, Alex Sobel, a UK Labour MP, proposed legislation that would ban the development of out-of-control superintelligence. In America, Sanders has proposed similar laws.
AI bosses, including Sir Demis Hassabis, the founder of Google-owned DeepMind, have called for the White House to coordinate a slowdown. Altman, despite accusing rivals of fearmongering, says he has discussed the idea with officials.
However, there are reasons to be sceptical. The Trump administration has vacillated on safety concerns, and several senior figures close to the White House see the proposals as a plot by the AI giants to hamstring smaller rivals.
US officials reportedly blocked the UK AI Security Institute, which has tested new systems before they are released, from accessing Anthropic’s latest models.
In the last few years, a growing list of employees has quit the top AI labs – OpenAI, Anthropic and Google – saying they could not, in good conscience, work on technology they believe poses profound dangers to humanity.
Andrew Strait, a former program director at the UK lab, said on Wednesday: “We have reached a wall on ‘voluntary’ frameworks … The US administration will want to be the sole tester of these systems and will push others out.”
An AI pause could also lead to an economic and sharemarket crisis. Nvidia, the world’s most valuable company, has seen its $US5.4 trillion ($7.5 trillion) value built on the breakneck expansion in AI spending by the top labs.
Forecasters at the Bank of England have modelled an AI “correction” in which US equities would fall 45 per cent, interest rates would spike, and Britain’s GDP would take a 2.2 per cent hit.
A global AI slowdown would prompt a substantial drop in data centre investment as expansion plans are curtailed. It could also kybosh the upcoming trillion-dollar floats of both Anthropic and OpenAI, which could take place this year or in early 2027.
Then there is China. Dario Amodei, Anthropic’s chief executive, has said a slowdown might be possible if the big US companies could agree, but that it would only benefit Beijing.
Donald Trump and Xi Jinping are scheduled to meet in Washington later this month, with AI top of the agenda, although agreeing on the details of a slowdown is a big ask. Would Chinese labs – generally seen as between three and six months behind US ones – have to freeze work, or would they be allowed to catch up?
Even still, it may be humanity’s best hope. There seems to be little chance of the leading AI companies agreeing to freeze development on their own – no matter how much their staff warn of the consequences.
Telegraph, London
Get news and reviews on technology, gadgets and gaming in our Technology newsletter every Friday. Sign up here.
