Updated ,first published
Australiaβs new AI Safety Institute has found powerful new artificial intelligence systems are capable of manipulating users, prompting federal minister Andrew Charlton to warn that building safeguards in early is the only way for the country to claim a share of the AI boom.
In a speech to the Australian AI Safety Forum on Tuesday, Charlton said frontier AI systems were βalready doing things their creators never intended – cheating, deceiving, going their own wayβ – and that the window to get ahead of that behaviour would not stay open for long.
The speech was the governmentβs most detailed public account of what the institute, announced in November and funded with $29.9 million, has done since it began operating this year. Charlton, the minister for science and technology, said the institute was testing frontier models with technical partners βin its first month of operationsβ, and named two research projects now under way.
Charlton used the speech to reject the idea that safety and economic opportunity pull against each other. βNo country will win the AI race with technology that its own citizens donβt trust,β he said, arguing that the nations that build safety in from the start would be the ones that come out ahead.
Public trust in AI was low, he said, and the greatest threat to Australiaβs AI ambitions was βnot a shortage of talent, capital or energy, but a shortage of trustβ.
The institute is led by Kate Conroy, a philosopher and Royal Australian Air Force reservist appointed its general manager in May. Charlton announced that Paul Salmon, whom he described as a leading international expert, would join this month as safety science research lead, alongside staff who have worked at the UK AI Security Institute and Google DeepMind.
To illustrate the risks, Charlton pointed to a series of laboratory cases. In a 2016 OpenAI experiment, a boat-racing AI model rewarded for points drove in endless circles to rack up a score rather than finish the race. A chess-playing model, facing defeat last year, hacked the game files to force its opponent to resign, reasoning that its job was to win βnot necessarily to win fairlyβ.
In a third case, a stress test published last year, an AI agent managing a fictional companyβs email learnt it was about to be shut down and, in 96 per cent of trials, βchose blackmailβ to stop it. Charlton stressed that the blackmail scenario was an engineered simulation and that no such behaviour had been seen in the real world.
βThese behaviours are being discovered in testing, before they can be discovered in the wild,β he said.
The two projects Charlton announced were work with the Gradient Institute on βmulti-agent riskβ, or how failures can cascade when large numbers of AI agents interact, much as traffic congestion emerges from individual drivers; and work with CSIRO on alignment, the problem of making systems do what their designers intend. Results are expected later this year.
Charlton linked that work to critical infrastructure. βWe do not let aircraft fly without airworthiness certification,β he said. βWe should not let unaligned AI systems into our critical social, democratic or economic infrastructure.β
The speech reflected the approach the government set out in its National AI Plan in December, which stepped back from European-style AI legislation in favour of applying existing laws sector by sector, backed by tougher enforcement where needed. Charlton described that as βfaster rules, applied by regulators who already understand their sectorsβ.
Charltonβs speech came as the government signals a more hands-on approach to AI, however. In a speech to the NSW Labor Conference in Sydney on Sunday, Prime Minister Anthony Albanese said that if Australia acted now it could βset the ground rules for AIβ and βshape the future, not let the future shape usβ. He said the world was βqueuing up to invest in Australiaβ, drawn by its skills, space, sunlight and natural resources, and that acting early would secure jobs and investment and let the country βbuild our sovereignty and our resilienceβ.
The speech followed a run of warnings from within the industry. In June, Anthropic, the maker of the Claude chatbot, which overtook OpenAI this year as the worldβs most valuable AI company, called for a global freeze on the development of the most powerful AI systems, arguing that humanity risked losing control of the technology. Charlton welcomed that intervention at the time. Late last month, the cybersecurity chiefs of the Five Eyes alliance, including Australiaβs, issued a rare joint statement warning that AI was reshaping cyber risk in months rather than years, and urged business and government leaders to act immediately.
The Business Briefing newsletter delivers major stories, exclusive coverage and expert opinion. Sign up to get it every weekday morning.
David Swan is the technology editor for The Age and The Sydney Morning Herald. He was previously technology editor for The Australian newspaper.Connect via X or email.