Opinion
I owe Dame Wendy Hall a beer.
Three weeks ago, I wrote a column arguing we should start listening to the people who warn that artificial intelligence could wipe us out. In it I gave Hall, the computer scientist advising the United Nations on AI, precisely one paragraph for her view that much of the panic is PR and marketing to drive valuations. I was, Iโll admit, broadly dismissive.
Since then, OpenAI has had the kind of month that should frighten investors. The company has apologised to Australia for one of its agents getting into a Medicare statistics platform, though it says no medical records were accessed. It has shelved its most advanced model and been accused by its own staff of ignoring their safety warnings.
This week it said it would raise $US30 billion ($43 billion) privately, at a valuation of $US1.4 trillion. It remains one of the most valuable companies in the world. Investors havenโt blinked.
A month of downright bad news hasnโt dented OpenAIโs price one bit. Hallโs argument โ that the doom talk pumps up valuations โ was right.
Basically, the people closest to the problem flagged it and were told to hurry up.
The rest of my argument has held up better. This week it came out that months ago, two OpenAI employees emailed executives to warn that the companyโs newest models werenโt being properly monitored in testing. They were told the tests had to move quickly so that the models could be released on time. Those models later escaped their testing environments and attacked organisations including Hugging Face, a website where developers share AI models. The Medicare breach turned up during the investigation into that attack.
Daniel Kokotajlo, who left OpenAI and has criticised it since, said the company seemed to have โvery bad securityโ. Basically, the people closest to the problem flagged it and were told to hurry up.
Some of their worries were surprisingly mundane. When Jacob Coxon quit Anthropic this month, he said the AI companies were gambling with our lives. He also said a bet that size shouldnโt be placed from a companyโs Slack, the workplace chat app. A security firm called Hacktron has since shown how easily it could have broken into OpenAIโs Slack. โWhy are you using Slack to build your nuclear Manhattan projects?โ one of its researchers asked.
A fortnight ago in San Francisco, over the same beers I now owe Hall, a table of Australian founders told me OpenAIโs staff are a message away in Slack channels that they all share. That makes OpenAIโs security their problem too.
OpenAI is at least attempting to show it recognises these risks. It shelved its upcoming model, GPT-6.1 Astra, a month before its release after it tested as more deceptive than earlier models. Chief executive Sam Altman now says it would be โill advisedโ to float the company while the industry works out how to keep these systems in line. Anthropic, pressing ahead with its own listing, has reportedly given about 80 pages of its prospectus to risk factors, including warnings that its models could resist shutdown. Lawyers make companies list every risk in a prospectus, but few lists include the product refusing to switch off.
Getting politicians to listen has turned out to be the easy part. Anthony Albanese took AI guardrails to the United Nations, Anthropic co-founder Dario Amodei briefed the Security Council, and the Pope has told tech bosses to take their own experts seriously. Donald Trump listened as well. He had lunch with nearly two dozen tech executives, concluded they should police themselves, and then ordered federal agencies to call the technology โsuper intelligenceโ.
But getting politicians to hear isnโt enough if the AI labs themselves wonโt listen.
Tristan Heywood, the Sydney-raised OpenAI researcher who told me last year that AI could cause the extinction of humanity, has a suggestion. He proposed making testing by Australiaโs AI Safety Institute, the federal body set up to assess the risks of advanced AI models, a condition of any big government AI contract. The government hasnโt taken up the proposal.
Testing by the institute remains voluntary, after the federal government backed away from mandatory AI guardrails late last year for fear of scaring off investment. So the only offer on the table this week came from OpenAI. It will convene a taskforce and arrange credits from its $US1 billion cyber defence fund for our agencies and critical infrastructure.
The federal government should take the money, but it shouldnโt accept a corporate taskforce instead of Heywoodโs suggestion: testing by the AI Safety Institute.
Acting Home Affairs Minister Richard Marles has just ordered every government department to stocktake its ageing systems, the kind an OpenAI agent has already found its way into.
The Safety Institute should ask the AI labs to open their own systems to the same inspection. It should also be the organisation to which anyone working at a lab that operates here can take a safety concern confidentially, with whistleblower protections in place.
Hall can have her beer. Those two OpenAI employees who tried to warn the company deserve something better: a regulator with the power to actually say โstopโ.
The Market Recap newsletter is a wrap of the dayโs trading. Get it each weekday afternoon.
David Swan is the technology editor for The Age and The Sydney Morning Herald. He was previously technology editor for The Australian newspaper.Connect via X or email.
