The Economist
When Anthropic, an artificial-intelligence lab, debuts on stock markets later this year, it is likely to be one of the biggest initial public offerings in history. Thatβs because Claude, the companyβs chatbot, is beloved of coders, who are willing to pay a lot for access.
Since Claude Code, its software-engineering agent, launched in February 2025, it has become indispensable for developers around the world. That includes Anthropicβs own: more than four-fifths of the code it published in May was written by Claude, the company says. Before Claude Code, the percentage was βlow single digitsβ.
The systems have improved in quality of output as well as quantity. An influential benchmark from METR, a think tank, shows that in early 2025 Anthropicβs models could complete tasks that took human engineers a little under an hour. The companyβs latest systems can complete tasks that would take more than a working day.
And so it may be easy to raise a cynical eyebrow when the company, at the top of its game and outclassing the competition, calls for the world to have βthe option to slow or temporarily pause frontier AI developmentβ, as it did on June 5. What market leader would not wish that its competition stop trying to catch up?
I, robot
Yet Anthropicβs leaders, who have for years worried about the prospect of out-of-control AI wreaking havoc, seem sincere. The latest generation of AI models are such competent coders, engineers and (soon) scientists that many worry they may be among the last ever made by humans. Jack Clark, an Anthropic co-founder, thinks there is a 60 per cent chance that, by the end of 2028, an AI system will be capable of creating its own successor with no human involvement at all.
That moment would mark the beginning of a process called βrecursive self-improvementβ (RSI), a closed loop. Version one of a model produces version two, which is faster and more capable; version two produces version three, which is more so again. The loop continues, and the improvements grow with each iteration. Build an AI system capable of this, and your human engineers never need to build another one again. βWhat can seem to many like a fanciful story may instead be a real trend,β says Clark.
AI doomers fear the superintelligence would be beyond human control.
Nobody knows for sure what the consequences of recursive self-improvement would be. Because AI can, unlike humans, work tirelessly and constantly, some think it would in short order lead to a superintelligent AI β a βfast take-offβ. (It has also been onomatopoeically dubbed βgoing foomβ, for the sound one might imagine an intelligence explosion making.)
AI doomers fear the superintelligence would be beyond human control, and that the start of RSI is the moment at which humanityβs fate is handed over to the machines.
Yet a self-improving AI would probably face speed limits, at least at first.
Building a model capable of RSI would require automating a range of specialist tasks currently carried out by humans.
At present, data scientists work on the theory of AI and coders put it into practice. Systems engineers build the foundations on which toy models can be raised to production scale. Other people seek out novel sources of training data, or experiment with ways to generate it fresh. Alignment and safety teams check that what comes out of the training process wonβt cause harm, intentional or otherwise.
The joy of repetition
Not all of those teams are equally amenable to AI assistance, and within each specialism some tasks are more automatable than others. It will not be too long until a human coder can do their job without ever writing a line of computer code themselves, but it may be some time until an AI is able to negotiate to acquire a previously undigitised collection of scientific papers.
It is not always obvious how the βjagged frontierβ will progress. Designing new algorithms seemed one of the safer jobs, until one of Google DeepMindβs models, AlphaEvolve, began doing it in May 2025.
It proposed a change to how Google spreads workloads across its data centres that saved 0.7 per cent of the companyβs worldwide computing power, and found better ways to perform matrix multiplication, which speeded up the training of Gemini, the companyβs flagship large language model (LLM), by 1 per cent.
Full RSI requires every task in this chain to become automated. The AI-powered acceleration of research and development (R&D) may be felt before then, however.
βAs the fraction of AI R&D performed by AI systems increases, the productivity boost over human-only R&Dβ could increase ten-fold, then a hundred-fold, then a thousand-fold, according to a report published in January by the Centre for Security and Emerging Technology (CSET), a think tank within Georgetown University. In that scenario, it warns that even if some aspects of AI R&D are initially difficult to automate, βthe accelerated rate of progress means those bottlenecks are soon overcomeβ.
Today, no AI model can build its own successor. But big AI models can build smaller models on their own. With human help they can build other big AI models, too. Earlier this year, Dr Andrej Karpathy, a then-independent researcher who now works for Anthropic, trained a chatbot about as capable as GPT-2, a large language model built by OpenAI in 2019.
Back then, the model took 168 hours of training to build on 32 state-of-the-art chips; Karpathy achieved the same result using a single computer with eight GPUs, the specialised chips used to build AI, in only three hours. With some more months of work he reduced the training time for his model, Nanochat, to just over two hours.
In March, he handed the work of speeding up the training process over to an AI agent called Autoresearch. In two days, the training time dropped to one hour and 48 minutes, and five days after that it fell to one hour and 39 minutes. βI didnβt touch anything,β Karpathy says.
The 18 per cent improvement on the human work is striking because Karpathy is a particularly talented human: he was a founding member of the research team at OpenAI and the head of AI at Tesla for five years.
The improvements themselves were prosaic. The AI agent picked better starting values for the training run, widened the scope of the LLMβs βattentionβ window and noticed that the modelβs focus was wandering. None of this is particularly novel, Karpathy says. But he had missed them. βThey stack up and actually improved Nanochat,β he says.
Speed-ups of this kind are inevitable as models become more capable. Much of the work of building terabyte-size frontier models is less glamorous than the AI industryβs enormous salaries and fancy offices suggest.
It involves plumbing together the layers of an infrastructure stack that are bought in from third parties, debugging hardware and software set-ups and tweaking βhyperparametersβ, the initial set-up of a training run, until the outcome looks solid. An AI system can do much of that today, with little supervision.
But even the more nuanced intellectual work is nearing automation, says Joe Spisak, a researcher at Reflection AI, a lab based in New York that is building frontier models that are open-weight (meaning their parameters are publicly released).
Give a frontier system a rough sketch of an idea for efficiency gains, and it is increasingly capable of designing an experiment, running tests on a toy model, seeing what works and responding with a plan that is ready to implement at scale.
AI models can carry out these sorts of tasks, which take humans hours, in around 30 minutes. Increasingly, humans play the role only of research director, steering the AI to run experiments, which the models code up, debug, optimise and monitor themselves.
The productivity boost is alluring, but also alarming. As the role that humans play in the production process shrinks, they may lose control. The end result could be models trained by models, to achieve goals set by models, whose safety is verified only by models.
Some fear a disaster. Professor Max Tegmark, a physicist and machine-learning researcher at the Massachusetts Institute of Technology, who has devoted much of the past decade to campaigning for AI safety, likens it to a driver flooring the accelerator on the motorway with their eyes closed. The result would be certain doom, he told The Economistβs βInside Techβ video show, as long as the driver refuses to open their eyes.
Powerful AI systems could outcompete humans as the decision-makers in government and commerce, says Tegmark, disempowering humanity; they could offer supreme power to whoever first builds them, ushering in global totalitarianism; or they could simply cease to care about humanity at all, and gradually squeeze people out to make room for more data centres and power generation.
Three years ago, Tegmark led a call for a pause in global AI development, arguing that the creation of the then-cutting-edge GPT-4 was tantamount to that blindfolded journey. This yearβs CSET report warned that the systems created by RSI βpose extreme risks. This warrants preparatory action now.β
Anthropic, it seems, is close to agreeing with that idea.
Hot chip
There are also several physical constraints that will, for now, impose limits on the speed at which models can improve themselves. The most important is access to compute. Despite efficiency gains, newer models continue to use more computing power to train than their predecessors, forcing progress to occur at the pace of data-centre development
Consumer use of AI may also slow down AI-powered research and development, says Helen Toner, interim executive director of CSET and a lead author of its recent report. The limited capacity in AI data centres needs to be carefully split between serving paying customers, training future models and carrying out open-ended R&D. The more demand there is in the first category, the less capacity, in the short term, there is for the other two.
Then there is the issue of training data. Much recent progress in AI has been in areas where models can teach themselves how to succeed thanks to βverifiable rewardsβ. A piece of software either runs or it does not; a mathematical proof is correct or it is not. In such cases, synthetic data, generated by models purely to train other models, can be checked for accuracy and added to the training data without risking the degeneracy that normally comes with training an AI on its own output.
It is trickier to make a model better at creative writing or legal judgment. If the models need to learn from the real world, that could also limit the reach of self-improvement.
βClosing the loopβ may be a step on the road to superintelligence and β depending on your disposition β utopia or doom. But it is not the only step required to produce exponential growth in AIβs capabilities.
The Economist
The Business Briefing newsletter delivers major stories, exclusive coverage and expert opinion. Sign up to get it every weekday morning.
