On July 8, 2026, the Australian telecom company Telstra experienced a significant outage related to an NTP issue. Telstra hired a consulting firm to do an incident review, and a couple of weeks ago, that firm released a public report: Findings Report on the July 8th Telstra Outage.
I’m using this blog post to write up my understanding of the failure mode. I found the report was often missing important details that I needed to make sense of what happened, so I tried to fill the gaps by looking at additional sources, and made my best guesses where I could. It was also more difficult for me because I don’t have domain expertise in any of the technologies involved here: I have some basic familiarity with NTP, and I have no familiarity at all with the various mobile telecom technologies involved.
Note: all block quotes in this post are from the public report unless explicitly indicated otherwise.
2010-2020 Telstra NTP system architecture
Telstra runs three NTP servers in their mobile network, geographically distributed at telecom exchanges across Australia.
These servers provide time synchronization to their clients, which could potentially be other NTP servers.
Note that some NTP clients are configured to point to multiple servers for redundancy. The diagram above shows one such client. Clients connected to multiple NTP servers will play an important role later in the incident.
According to the report, starting in 2010, Telstra ran two NTP servers at the Sydney and Melbourne exchanges. At each site, one server ran at stratum 2, and the other ran at stratum 3. The Perth exchange ran a single NTP server, at stratum 3. For their stratum 1 sources, the stratum 2 servers were clients of NTP servers hosted at Australia’s National Measurement Institute. It is the Melbourne site that will play the most significant role in this incident.
A note about arrows: NTP diagrams typically show arrows going from lower stratus to higher stratus (e.g., “stratum 1 → stratum 2″). However, I’ve drawn the arrows the other way because I’m used to the “client → server” convention of arrows directions.
I’m assuming that the NTP clients communicated only with the stratum 3 server, but there isn’t enough detail in the report to verify this was the case. I really wish they had drawn diagrams in the report!
For the purposes of this incident, the important detail here is that each stratum 3 server has a redundant stratum 2 source: all of the stratum 3 servers can use either the Sydney or Melbourne stratum 2 servers.
An upgrade leads to an old redundancy being lost, and two new redundancies added: all three changes will contribute to the incident
Six years before impact
In 2020, Telstra upgrades the timing hardware on their mobile network. With this upgrade, their NTP servers now run on Microchip SSU 2000 synchronization supply units, with the supporting vendor being a local company: Scientific Devices. This upgrade results in three significant changes to the architecture.
No more local NTP sources
Due to a limitation in the NTP timing chassis, a stratum 2 NTP server could not provide time to a
stratum 3 NTP server in the same chassis. – Findings Report on the July 8th Telstra Outage
With this upgrade, they are no longer able to have a stratum 3 NTP server running on the SSU 2000 use another NTP server on the same unit as its source. As a consequence, the Melbourne and Sydney stratum 3 NTP servers lose a level of redundancy: they now each have only one stratum 2 source.
Unfortunately, the writeup doesn’t explain why the limitation existed. I tried to figure this out by reading what SSU-2000 docs I could find. My best guess was that this was a security restriction, as described below (emphasis mine). For context: the PackeTime module is the thing that runs NTP on the SSU 2000, and NTP runs on the service port. “Independent mode” means that the two Ethernet ports of the module run independently (i.e., bonded), which I presume is how they had things set up so they could run two independent NTP servers.
In the Independent mode, the two ports are each assigned unique IP addresses and can each serve time to independent networks. From a security perspective, the important thing to note about the Independent mode is that the PackeTime Module does not support routing protocols between the two service ports or between the service ports and the maintenance port. This prevents a malicious attack on Port A (network A) to be used to send a malicious attack via Port B (network B) or vice-versa, or between Port A or B networks and the maintenance port’s network. – SSU-2000 Synchronization Supply Unit and the SDU-2000 Synchronization Distribution Unit: User Guide, Revision G – June 2008, Chapter 8, Output Module Reference Data, PackeTime Module, p274
GPS card for failover
However, while the upgrade cost them one form of redundancy, it granted them another form: each SSU 2000 unit had a GPS card in it. GPS gives you access to a high-accuracy clock, since it receives time information from multiple satellites which each have atomic clocks in them. The presence of a GPS card gave the NTP servers running inside of the SSU 2000 access to a stratum 0 source. The GPS card was therefore available as a failover time source if the stratum 3 NTP servers lost access to their (now sole) stratum 2 source.
But there is a problem lurking within this failover solution: a GPS rollover bug in the card.
Internally, GPS keeps track of the number of weeks since January 5, 1980, and it uses 10 bits to represent this count. This means that the count rolls over every 1024 weeks (roughly every 19.7 years). Some GPS devices were not originally designed to handle the rollover. The GPS card in the SSU 2000 is one of those devices. (One thing I learned here is that while the GPS rollover itself happened on April 6, 2019, devices that are vulnerable to rollover don’t necessarily roll over on the same day. For the SSU 2000, the rollover was September 18, 2022).
This means that the GPS card’s date will jump backwards in time after it rolls over. The rollover date is September 18, 2022. But just hitting that date isn’t enough: as long as the card is still running, it will behave properly. It’s only when it is powered off and on after that date that the rollover will happen.
Running the symmetric variant of NTP
There is also one additional architectural change that happens in the upgrade.
In the client/server variant, a persistent client sends packet mode 4 packets to a server, which returns packet mode 3 packets. Servers provide synchronization to one or more clients, but do not accept
synchronization from them…In the symmetric variant, a peer operates as both a server and client. – RFC 5905
The mobile core timing architecture was changed from client/server to peering, meaning that the
NTP server in the NTP timing chassis could acquire any NTP source that it could find and select to
use any of these sources that was providing the most stable and accurate time. This introduced
uncertainty in the time sources being used in mobile core timing hierarchy. – Findings Report on the July 8th Telstra Outage
Here’s where I struggle the most with the writeup, because I’m not familiar enough with the underlying technology. RFC 5905, which describes NTP, does not use the term “peering” anywhere, but it does talk about multiple variants of the protocol, including the client/server variant and the symmetric variant. I assume peering in the report is referring to the symmetric variant.
Here’s an example of behavior which I believe can only happen with the symmetric variant. Consider the following network of five NTP servers, labeled A-E.
Now, assume that server A goes down. This means that node B no longer has a source. When running in client/server mode, C is a client of B, which means that B cannot be a client of C. However, when running symmetric mode, B can become a client of C, like this:
This behavior will eventually contribute to increasing the severity of the incident by propagating bad time data further through the network. But we’re still six years away from that.
SSU 2000 GPS rollover day comes and goes with no issues
Four years before impact
On September 18, 2022, the rollover day for the SSU 2000, nothing happens. Because the GPS cards have been running uninterrupted, they do not roll over: that only happens after being reset. In addition, the cards are not even in use, since they are only used for failover, and failover has not occurred. The problem will only manifest when the GPU card is restarted, which will not happen for years.
Melbourne NTP server cannot reach Sydney, engineers fail over to GPS card
Nine months before impact
Several months before the outage, there’s a minor incident where the stratum 3 NTP server in the Melbourne exchange has been experiencing errors attempting to connect to its stratum 2 source in the Sydney exchange:
In October 2025, personnel were working on the Melbourne NTP server due to an incident report that the NTP stratum 3 clock was losing connectivity multiple times per day to its stratum 2 source clock at Sydney, resulting in NTP alarms.
The report notes that the server used peering to dynamically identify another NTP server, which happened to be a stratum 4 source, leading the Melbourne NTP server to become a stratum 5 source. However, they don’t know anything about this server:
The incident report also noted that the Melbourne NTP had become a stratum 5 source meaning that it was receiving time from a stratum 4 source in the mobile core timing architecture. Remember, Melbourne had only one stratum 2 source and when that source was lost, it used peering to find another source which happened to be a stratum 4 client of Sydney or Perth.
According to the report authors, this meant that timing loops were already present in the system (emphasis mine)
It is critical to have at least two independent reference sources. A timing loop is when sources that
appear independent are in fact validating one another – directly, or by tracing back to a common
reference. To avoid timing loops, you must clearly define the server hierarchy and document all
relationships between servers.…
looping was already occurring in mobile core timing when an intended NTP stratum 3 server was able to become a client of one (or more) of the clients and that a circular loop was established when the stratum 2 source was lost.
My understanding of the topology at the time of the October 2025 incident is captured in the following diagram, where the red arrow indicates a failed connection, and the blue arrows are network edges that were not in the previous diagram. Here I’m assuming the client in question was connected to the Perth stratum 3 server, but it may just as well have been the Sydney stratum 3 server.
While this evidence indicates that looping was already happening in the network in October 2025, the looping itself had not yet contributed to an incident: that will come later.
The network engineers mitigate the incident by failing over the Melbourne NTP server to use the GPS card as its source.
To resolve this issue the GPS card was activated and connected to the stratum 3 NTP server in Melbourne as a means of replacing the lost Sydney stratum 2 source. the incident appear to have chosen this approach as the best means to address the incident. The GPS card is known as a reference source (aka stratum 0) and so this connection made Melbourne a stratum 1 NTP source, effectively the same level as the NMI stratum 1 sources in the mobile core timing architecture. This impacted the clients of Melbourne, making them stratum 2. In peering mode, these stratum 2 clients can be time servers at the same priority as Sydney and Melbourne stratum 2 who were the intended stratum 2 sources.
A reminder from the SSU 2000 vendor about the firmware update
Seven months before impact
In January 2026, the vendor supporting the SSU 2000, Scientific Devices, sends a reminder to Telstra that there is a firmware update for the SSU 2000 related to the GPS card.
Here I’ll quote the CEO’s Senate testimony (emphasis mine):
It’s still subject to our investigation, but there was definitely a reminder in January of this year that
this update needed to be done. We believe it does go back several years that it was there. To provide some context, our investigation so far has shown that it’s not a case of the teams just ignoring that update. They did consider it. The way this was designed and the way we were using this particular server and this card, this update was affecting a feature that we were not using at that time.
…
We know that our teams did get the alert of a software update. As we said, they assessed that through our investigation, we can see, and made a choice. Again, this is not an excuse; it’s putting it in context. There are many software updates and changes getting made across the network. Any software update creates a risk, and we put it into testing. In this case, they thought, given the design that was in place, it was not relevant because we were not using this equipment in the way that would trigger this risk. – (Telstra CEO) Vicki Brady, Senate Environment and Communications References Committee
It sounds like the engineer who made the call was not aware that the GPS card was a failover source. So was the use of the GPS card an in-the-moment workaround using available resources, or was it an explicit part of the architecture that the engineer making the call about firmware updates was not aware of? Or was it something else? There simply isn’t enough information to tell. The public write-up implies that it was part of the architecture:
The architecture indicated that the GPS card was available to be used as an alternative failover source for time of day.
On the other hand, we have this Senate testimony from the Group Owner, End-to-End Service Performance and Resilience of Telstra
Based on the intentional design of that SSU 2000, the GPS module is not used as part of that design. Gerard Tracey, Senate Environment and Communications References Committee
I don’t know how to reconcile those two statements, beyond “different people in the system had different understandings of the architecture”. If you think you can eliminate these sorts of mismatches in understanding in your organization, good luck to you!
One last thing I want to point out here is the CEO’s comment that “any software update creates a risk”. So, not doing updates is dangerous, but doing updates is dangerous as well! It’s easy to judge the engineer in hindsight for this decision, but I have personally seen many incidents that were triggered by performing software updates.
Impact starts: A repair triggers a poisoned clock
On July 8, 2026, Telstra network engineers took the SSU 2000 offline in Melbourne in order to fix an issue with a redundant power supply:
When the maintenance team came to do the work on this server, they were required to power it down and take it offline because they needed to replace what we call the chassis, the case that it sits in, because its backup power supply was not working. So they put in a brand-new chassis. – (Telstra CEO) Vicki Brady, Senate Environment and Communications References Committee
The report doesn’t say how long the unit was offline, nor does it discuss what happened to the Sydney stratum 3 NTP server that had Melbourne as its sole source. One thing that’s clear is that there was no system impact while the unit was down.
It was when the unit came up again that the trouble started. At 2:50 AM, the unit is powered up and comes back online. When the GPS card comes back online, the rollback issue manifests, sending its clock back 19.7 years, from July 2026 to November 2006.
The poisoned clock time spreads across the network
Since Melbourne was operating at stratum 1 and peering was enabled and Melbourne stratum 2 had been turned off by the same chassis replacement, the incorrect date propagated throughout the mobile core timing network.
Because the Melbourne NTP servers had gone down for maintenance, Sydney lost its source for its stratum 3 server. While presumably it had a GPS card to fail over to, they were configured to only fail over manually. So, unlike Melbourne, Sydney does not switch to using GPS card as a source.
However, because the NTP servers are configured to support peering, the Sydney NTP server is able to use one of its clients a source. When Melbourne comes back up, an NTP client common to the two sites becomes a stratum 2, using Melbourne as the source. The Sydney NTP server uses this client as its time source, and now the poisoned time has propagated to the Sydney site, with its clients impacted as well.
(Note: The report does not say whether the Perth NTP deployment was impacted by this, so I’m going to assume it wasn’t).
Impact: mobile users can’t call or use data
70 minutes after impact start
Starting at around 4am July 8th, 2026, when Mobility Management Entity (MME) instances began to reset, end users of Telstra’s network experienced varying levels of impact in their ability to communicate on the network.
At this point of the write-up, I’m really out of my depth, because the details of the failure are telecom-specific. I don’t even know if an MME instance is a software thing or a hardware thing. Based on this case study on Ericsson’s website, I assume that Ericsson is the vendor, and according to this Ericsson page, it sounds like it could be either one, as it says: Our latest software release can be deployed as a physical network function (PNF), as well as a virtual network function (VNF).
What I do know is that it provides a control plane function that communicates between users’ phones and the mobility network. When these instances reset, it broke mobile functionality, including emergency service calls (Triple Zero in Australia).
These impacts precluded some users from being able to make/receive voice calls, including Emergency Calls (per the Emergency Call Service Determination:2019, also referred to as Triple Zero calls), send/receive messages, or access mobile data services. Those without data services were unable to do things such as access the web, stream content and most mobile applications were unusable as well.
During the worst of the incident, 45% of mobile users were impacted:
At its peak, this initial issue impacted approximately 45 per cent of all calls and data sessions on our mobile network. – (Telstra CEO) Vicki Brady, Senate Environment and Communications References Committee
Impact: valid IP addresses get blocked
Until the correct network date was restored, information for approximately 30,000 IP address prefixes were incrementally corrupted (written with the wrong date) by the Diameter Signalling Controller (DSC) in such a way that access attempts using those addresses were blocked from using the network regardless of available capacity.
My guess based on this is that a Diameter Signalling Controller (once again, I’m guessing they’re using an Ericsson product) is that the DSC keeps a table of session information about active IP addresses on the network. And because entries got written to the table with the rollback date, the system interpreted these as invalid and did not allow these addresses to access the network.
Note that this is a poisoned data scenario: network engineers had to fix these records before the users could regain access.
Telstra, with help from their equipment vendor, applied a workaround during the Outage and then at a later time performed repairs to remove the bad IP address prefix entries.
Omnipresent risks
A couple of weeks ago, I wrote a post titled omnipresent availability risks in cloud software. Here I’ll point out some of those risks that I see manifesting in this incident.
Problem area: saturation
This is a pretty clear example of an incident that involved saturation. The original MME instance resets don’t take out the entire mobile network, but they have the effect of reducing capacity enough that the remaining capacity is not sufficient to meet demand. From the report:
Initially users were impacted by a loss of network capacity (due to failure of MME instances) which caused some users to be disconnected from the network. The remaining capacity was insufficient to enable those users to all reconnect deterministically nor was it capable of servicing the nominal network traffic load. So essentially the network was overloaded.
Problem area: networking
Admittedly, this one is a gimme: every large-scale telecom-related outage is going to be networking related, since what telecoms provide is network access! That was no exception here.
Problem area: security
It’s unclear to me in this case whether security systems played a role at all in this incident. I hypothesized that the loss of redundancy was due to a security restriction in the SSU 2000 unit, but I don’t know for certain that’s the case. It’s also possible that the Diameter Signalling Controller blocking the IP addresses was related to a security function, but there’s not enough information.
Essential non-standard changes: mitigating an operational issue
This is a clear case where the act of mitigating an operational issue was itself a contributor to a new incident. The trigger for this incident was repairing the power supply of the SSU 2000 unit in the Melbourne exchange. The repair itself went fine: it was powering up the unit that exposed the latent bug in the GPS card.
Essential increase in essential complexity: reliability subsystem
The GPS card, which was the source of the poisoned time, was a reliability subsystem! It exists to provide failover functionality in the case where an NTP lost access to its source. And unexpected behavior of that reliability subsystem, when engaged, bit them hard.
Essential increase in essential complexity: migration
I think Telstra’s 2020 upgrade of their timing hardware qualifies as a migration. This resulted in architectural changes that created a system vulnerability, although this vulnerability would not manifest for six years.
I do appreciate how the CEO was sanguine about how the nature of these sorts of changes means that incidents are inevitable.
What I would say is that networks and environments like a mobile network are absolutely complex, and it’s evolving—any sort of complex technology environment—and certainly networks are not infallible. But our job is to make sure we are taking the actions and those steps that will mitigate as best we can. I would love to be able to sit here and say will be zero outages. The reality of a complex network environment with fast evolving technology is that no telco around the world could guarantee that. – (Telstra CEO) Vicki Brady, Senate Environment and Communications References Committee
Some parting observations
There is no NTP haiku
In my experience, NTP isn’t a technology that frequently shows up incidents, at least not compared to other networking protocols such as DNS or BGP. The authors of the public report takes Telstra to task for a lack of sufficient NTP expertise within the company, and I can imagine playing a more significant role in the telecom domain than in other cloud software domains. But I can’t help but wonder if the fact that NTP isn’t generally problematic contributes to organizations having less in-house NTP expertise.
Appalling incuriosity from the investigators on how the work gets done
the need for stronger competency and more curiosity to investigate when something looks wrong
As is always the case in these sorts of public reports, there’s a lot of ink spilled on the subject what engineers should have done in the run-up to the incident but didn’t. The bullet “Increase attention to the operation and management process discipline in all areas” pretty much says it all.
Whenever I see this sort of content in a report, my response is, “boy, the investigators really blew it.” And that’s because you can’t understand how the incident happens unless you understand how the work really gets done within the organization. The authors show an acute lack of curiosity about the circumstances of the individuals who were involved in making the decisions.
I’ll note that these authors managed to mess up the section numbers in their report: it goes 1,2,1,2 instead of 1,2,3,4. This leads them to reference a non-existent section 4.1. I would ask the authors: Do you think we should be taking advice on how to run a reliable telecom network from people who can’t even number sections correctly in a ten page document? Or do you think we should consider the circumstances that could lead to this sort of error?
Where are the other system vulnerabilities?
We observed instances where other areas and elements of the network and organizations were performing at better levels than NTP.
Finally, it’s easy to point out problems in a system that were exposed by a recent incident. The really impressive thing, to me, is to identify the problems that were not exposed here. I have high confidence that there are other architectural vulnerabilities in the system, just waiting to be turned into incidents because of future changes. And yet, somehow, investigators never manage to find new vulnerabilities that could lead to the next outage. Funny thing, that.
References
- Findings Report on the July 8th Telstra Outage By Technology Audit Partners, August 24, 2026.
- As it happened: Telstra execs grilled at Senate hearing into outage, David Swan and Elias Visontay, Sydney Morning Herald, July 17, 2026.
- Senate Environment and Communications References Committee, Triple Zero service outage, Committee Hansard, July 17, 2026
- Triple Zero service outage Submission 7 – Supplementary Submission: Environment and Communications References Committee inquiry into the: Triple Zero service outage, July 16, 2026
- NTP servers: S200 and S300 Series Internal Week Number Rollover event on September 18, 2022, community.ntppool.org, Steve Sommars, July 27, 2022
- The 18th September 2022 Rollover Event, Chronos Technology, Louise Davies, June 9, 2022
- RFC 5905: Network Time Protocol Version 4: Protocol and Algorithms Specification, D. Mills, J. Martin, J. Burbank, W. Kasch, June 2010.
- SSU-2000 Synchronization Supply Unit and the SDU-2000 Synchronization Distribution Unit: User Guide, Revision G – June 2008
Surfing Complexity 