The only two Telstra engineers with deep expertise in the system that failed in July had been sent home on mandatory rest by the time the national mobile outage they had unwittingly triggered took hold, an external investigation has found.
The outage cut hundreds of people off from Triple Zero, halted trains in Victoria and NSW and brought down eftpos systems across the country. It has triggered a Senate inquiry and an investigation by the communications regulator that could see Telstra fined up to $30 million.
The pair had replaced a faulty power supply in a timing chassis in Melbourne in the early hours of July 8. When it came back online at 2.50am, a GPS card inside reset and began telling the mobile network the date was 2006. The card was missing a firmware update for a predictable fault the supplier had already flagged in bulletins, confirming this mastheadโs July reporting.
The report by Technology Audit Partners found alarms on the timing servers were checked only during business hours by a small number of staff, and did not appear in the monitoring tools used by round-the-clock support teams. When things went wrong, nobody on duty could find records of the servers, the work done that night or the people to call.
The outages have heaped fresh scrutiny on Australiaโs Triple Zero system and the reliability of the nationโs telecommunications networks more broadly.
Chief executive Vicki Brady said the company accepted every finding.
โAt its core, this came down to the network timing system not being given the priority that a critical network capability requires,โ she said on Wednesday. โThat is a miss on our side.โ
Brady also revealed Telstra had missed eight failed Triple Zero calls during the outage and never carried out the welfare checks required to find out whether the people who made them were safe. She blamed a manual processing fault and a monitoring fault.
The checks have since been done. One caller got through on another network, and another found a different way to connect. Five either needed no help or had misdialled. Police visited the home of the unidentified eighth caller.
โIt shouldnโt have happened. We should have picked those eight up,โ Brady said. โWe know how important Triple Zero is.โ
The eight are on top of the hundreds of unsuccessful Triple Zero calls Telstra has already disclosed.
The telco also revised the start of the outage to 2.50am on July 8, almost 50 minutes earlier than it had said. The previous time reflected when a maintenance ticket was closed rather than when the fault began.
Brady said the failure was โnot due to infrastructure failingโ, but to an undocumented design change made in October last year and a software update that was never applied to a GPS card.
Despite the auditors finding the team responsible had been constrained by budget priorities, Brady rejected suggestions of underinvestment, saying Telstra employed about 2800 network engineers and had spent more than $19 billion on capital works and spectrum over five years.
โAt its core, this came down to the network timing system not being given the priority that a critical network capability requires.โ
Telstra chief Vicki Brady
โItโs not about not enough resources. Itโs not about not enough money,โ she said. โItโs about the prioritisation.โ
The report also faulted a lack of curiosity among staff who saw warning signs and did not chase them, a criticism that Brady accepted.
โThat is a lesson weโve got to take,โ she said. โWe have to be always open to challenging the way weโre doing things.โ
Asked by this masthead whether anyone had lost their job over the outage, Brady said it was not the fault of an individual.
โClearly, for us to have an outage like this, our processes, our controls were not sufficient,โ she said. โItโs not about anyone losing their job.โ
The board has already docked her more than $600,000 in pay, with just under $2 million cut across her senior leadership team.
More than 30,000 customers have contacted Telstra for compensation, and it has paid just under $1 million in credits, an average of about $15 for consumers. Brady urged anyone still out of pocket to come forward.
โModern networks are complex, but complexity is not an excuse,โ she said.
The Business Briefing newsletter delivers major stories, exclusive coverage and expert opinion. Sign up to get it every weekday morning.
David Swan is the technology editor for The Age and The Sydney Morning Herald. He was previously technology editor for The Australian newspaper.Connect via X or email.