Nobody at OpenAI asked its agent to break into Medicare. It was given some questions about Australia during an internal test, went looking for answers, and found some of them in files the public was never meant to see.
โIn the course of that, our models took actions we did not intend,โ an OpenAI spokesperson said on Thursday. The company says its models reached โseveral Australian government websites and servicesโ, including aggregate health statistics and internal file names, but no patient records.
The agent got into the Medicare portal on June 18, hit repeated blocks and found ways around them. According to Services Australia, it also wrote files to an internal server. OpenAI detected the activity in August and told the government on September 10 by emailing a public Services Australia inbox.
Alastair MacGibbon, who was Australiaโs cybersecurity boss under Malcolm Turnbull, had already been briefed on the incident when I rang just after 7am. โThis was an agent that was not tasked with hacking,โ he told me. It โjust happened to use tools in its tool belt to go about achieving that objective, which involved, basically, hackingโ.
That doesnโt make it rogue AI. Luke Irwin, chief executive of Aegis Cybersecurity, says it is โcloser to a normal AI doing exactly what it has been asked to doโ.
Irwin describes agents as hyper-intelligent five-year-olds.
โThey are extremely capable and highly motivated to achieve the outcome you have asked for, but they do not inherently understand the boundaries that a human might consider obvious,โ he says. If it canโt do what you asked, it looks for another way. Anyone whoโs asked a small child to grab something off a high shelf knows how that ends.
The federal government has been betting that bringing these companies onshore can help Australia have some influence, and give it some say over how the AI models behave. OpenAI recently signed a $7 billion data centre deal with NextDC in western Sydney, and the government has signed a memorandum of understanding with Anthropic, which commits it to โjoint safety and security evaluationsโ, but is โnot intended to have legal effectโ.
Hosting the servers here wonโt count for much if the people who built the models couldnโt stop them wandering into a foreign governmentโs systems.
We still donโt know how it got past the blocks, and thatโs now a job for a taskforce Albanese announced on Thursday. Irwin warns against assuming anything was hacked in the movie sense. Agents donโt browse the way people do, and one can turn up a scrap of machine-readable data a human would never see. The question, he says, is whether the information โwas genuinely protected or whether it was technically accessible but simply difficult for a human to discoverโ.
Other AI labsโ confessions this year point to โhackingโ that is far less Hollywood than youโd think. Google said last week one of its Gemini models got into a companyโs systems by guessing passwords until one worked, while Anthropic said in July one of its models read passwords off an exposed debug page.
What it all means is weโre more exposed than ever before and our defences havenโt kept up. When My Health Record went opt-out in 2018, privacy advocates warned that putting every Australianโs medical history in one place would create a honeypot for hackers, and more than 2.5 million people opted out.
Cyberattacks that previously needed a skilled person with time on their hands can now be run by software that doesnโt get tired or bored.
โDonโt create an AI safety institute and fund it as if it was fixing a couple of country road black spots.โ
Alastair MacGibbon
Our laws and our basic concept of cybersecurity picture a person at the keyboard, possibly with a balaclava on their head. Albanese says the government will seek urgent advice on whether offences were committed and whether to refer the matter to the Australian Federal Police. Unauthorised access to restricted data is a crime under the Criminal Code, but the offence is written around humans, not bots.
As for how worried we should be about this specific breach, no personal information appears to have been taken, and OpenAI found and reported it itself. โWe shouldnโt shame these companies out of telling us,โ MacGibbon said. Irwin puts it at about five out of 10. โThere will be more, and they will get worse and more impactful.โ
So how safe is our personal data on government and other servers? Right now, itโs less safe than it was a year ago.
Two of the other agencies named on Thursday, the Australian Institute of Health and Welfare and the NSW Bureau of Crime Statistics and Research, mainly publish statistics. The bigger worry is everything else: the health records, bank details and tax file numbers sitting with thousands of organisations, which Australians were already losing at a record rate before agents came along.
The privacy commissioner received 1205 data breach notifications last year, the most since the scheme began in 2018, and there is now a report to the Australian Cyber Security Centre every six minutes on average. We should expect these numbers to accelerate.
What got under MacGibbonโs skin with the Medicare case is that nobody in government even noticed.
Security people have long said AI attacks are easy to catch because โtheyโre noisy and theyโre dumbโ, and the LinkedIn cartoons to prove it are everywhere. โImagine if you had a malicious human getting agents to do these tasks,โ he said. When OpenAI did own up, it emailed a public inbox โ the digital equivalent of a note under the windscreen wiper โ and it took another five days to reach the Australian Cyber Security Centre.
Five days after that email was sent, I watched Sam Altman on stage at Salesforceโs Dreamforce conference in San Francisco, calling the Hugging Face break-in โthe worst accident weโve seenโ.
โAccidents with any new technology are unavoidable, and we should have a great culture of transparent reporting about them,โ he said. He didnโt mention Australia.
Australia has set up a new body for these situations: the Australian AI Safety Institute began operating this year to test and advise on these exact systems. But itโs been given less than $30 million over four years and sits inside the industry department as an advisory body, not a regulator.
After this week, that looks inadequate.
โDonโt create an AI safety institute and fund it as if it was fixing a couple of country road black spots,โ is how MacGibbon put it.
Irwin runs his own research agents in a sandbox, walled off from client data, and treats them as โcompletely untrustedโ. The rest of us can borrow some of that caution: a passkey for myGov and your email so thereโs no password to guess, and a second thought before letting any AI assistant log in as you.
Governments could start with the same instinct. Make AI companies test agents walled off from the live internet, and make whoever deploys an agent responsible for where it goes, as Irwin suggests. Then require labs to report incidents within days, to someone other than a public inbox.
MacGibbon was still driving when he put the question that stuck with me. โWhat makes us think, with an increased threat, with the democratisation and increased automation, that weโre going to be doing better cybersecurity?โ
If this doesnโt force a decent answer then maybe nothing will.
The Market Recap newsletter is a wrap of the dayโs trading. Get it each weekday afternoon.
David Swan is the technology editor for The Age and The Sydney Morning Herald. He was previously technology editor for The Australian newspaper.Connect via X or email.