A couple of weeks ago, I wrote a post on the eMudhra incident. The company has now published their Full Incident Report. Strangely, at least to me, they posted it as a comment on the Bugzilla ticket but not on their public page about the incident.

I saw quite a bit of chatter on Twitter about the topic of exporting private keys from hardware security modules (HSMs). I’m not going to comment on that because it’s outside my domain of expertise. What I did want to comment on were some of the details mentioned in the write-up that get at the constraints that the engineers at eMudhra faced, and how those constraints influenced decisions that the engineers made in the moment.

Technical background

Before we can get into it, I need to say a little about public key certificates. This was largely for my own benefit, because trying to explain the technology forces me to learn more about it.

Certificates and browsers

When your web browser makes a secure connection to a website, it does a check to verify that you are connected to the real website, and not some fake website that was set up by a nefarious actor who is trying to fool you. As part of the protocol, the website will return a server certificate that attests “this is the right website”. Your web browser will check that the certificate was properly signed by a trusted authority.

Let’s use everybody’s favorite example website: example.com. Your browser can show you information about the server certificate by clicking on the little icon next in the browser bar > Connection is secure > Certificate details. Here’s what you see in Google Chrome for example.com.

Certificate authorities

For the purposes of this post, the important bit is the “Issued By” section: That tells us which entity issued the certificate. An issuer of certificates is called a Certificate Authority (CA). For example.com, the CA’s common name is Cloudflare TLS Issuing ECC CA 3.

The browser needs the public key of the CA in order to check that the certificate’s signature is valid. It also needs to know whether it can trust the CA that issued the certificate. After all, anybody can set up a CA and issue whatever server certificates they want. The browser has to ensure that the certificate was issued by a trusted CA.

Browsers use the same mechanism for checking whether to trust a CA as they do for checking whether to trust a server: they check a certificate! Instead of a server certificate it’s a CA certificate, but it’s also issued by a CA. In Chrome’s certificate viewer, under “details”, you can see the chain of CAs involved:

Browsers need a way to trust some CAs without verifying that their CA certificates were signed by another CA, or else you have an infinite regress problem. And so there are a set of CAs that browsers automatically trust. You can view the Chrome Root Store, the set of CAs that are trusted in Chrome, by pointing your browser to chrome://certificate-manager/crscerts. It looks like this:

The website normally sends over the server certificate and intermediate certificates needed to build the chain up to one that is known to already be trusted by browsers. The browser then checks the chain to verify that the server certificate was signed by a trusted CA.

For example.com, here’s a diagram that depicts the various CAs that are involved in the chain.

Intermediate CAs

CAs that are already trusted by browsers are referred to as root CAs, and CAs below the root are called intermediate CAs or subordinate CAs. I’m calling out this terminology here because the CA involved in this incident was an intermediate CA. The important thing to remember is that an intermediate CA must have a CA certificate that’s signed by another CA. (Because root CAs are already trusted by browsers, their CA certificate doesn’t need to be signed by another trusted CA, and so they can be self-signed).

The ingredients of a CA

In order to operate an intermediate certificate authority, you need two things:

• a public-private keypair
• CA certificate

The private key is used to sign certificates. The public key is used by the browser to verify that the signature is valid. The CA certificate demonstrates that another CA trusts you.

A bit of detail that will be important later is that, in order to get that signed CA certificate, you need to already have generated your public-private keypair. If you haven’t generated a keypair yet, you can’t get that certificate.

Hardware security modules (HSMs)

If somebody gets a hold of your CA’s private key, then they can use it to forge certificates. The industry uses hardware security modules (HSMs), which are dedicated hardware devices you connect to a server to securely manage sensitive private keys.

If your server has an HSM installed, it can generate the keypair and keep the private key within the HSM, which means your server never needs to load the private key into memory in order to perform cryptographic functions like signing certificates. HSMs play an important role in this incident because the lack of HSMs ended up becoming a constraint, as we’ll see.

The constraints

With those background details out of the way, let’s dive into the constraints that eMudhra engineers faced as mentioned in the writeup.

CAs must be physically based in Europe and the U.S.

While eMudhra is based in India, their customers had data sovereignty requirements that there had to be CAs physically located in datacenters in the U.S. and Europe. This meant that they needed HSMs at these U.S. and European datacenters, since the private key for the CAs had to be managed by HSM.

The naming convention they used for the CAs at the three sites are:

• Series A – based in India
• Series B – based in Netherlands
• Series C – based in United States

According to the writeup, eMudhra ordered the HSMs for the U.S. site in April of 2025. (It doesn’t say when they ordered the HSMs for Europe). This timing will become important later on.

Supporting older clients

(This section is speculative because the rationale isn’t explicitly stated in the incident report).

eMudhra was setting up an intermediate CA. Recall from earlier that an intermediate CA needs to have a CA certificate signed by a different CA. The eMudhra intermediate CA’s certificate is signed by eMudhra’s own root CA, for which they use the brand name emSign.

Here’s a certificate associated with the American (C series) data center: EM DV TLS CA ‑ G3C. You can see the issuer was emSign Root TLS CA – G3.

If you are a CA and the client doesn’t trust your CA certificate, then you have a problem: the client won’t trust the certificates that you issue. However, you can get your CA certificate cross-signed by a CA that the client does trust, and then you’re good.

There are some older clients out there, ones that don’t have eMudhra’s in their trust stores, and eMudhra dealt with that by having their CA certificate cross-signed by an older CA. In this case, that CA was managed by a separate company: Sectigo (As an aside, here’s Sectigo’s statement on this incident).

Sectigo’s CAs go by the name “AAA Certificate Services”, and you can see the cross-signed certificate here. Below is a diagram of the various CA certificates involved here.

(Note that cross-signing technically creates separate certificates, which I’m not explicitly depicting)

To sum up, the need to support older clients (which didn’t trust their root CA) created a constraint that eMudhra had to get their CA certificate cross-signed by another CA; in this case, that CA was operated by Sectigo.

Client certificates and Chrome Root Store Policy Change Deadline

On the web, browsers use server certificates to verify the identity of websites, but it doesn’t work the other way: websites generally don’t use client certificates to verify user identity, even though browsers support it. Instead, they use things like username and password.

However, there are domains where public-key certificates are used by both servers and clients for verifying each other’s identities, and the use of both server and client certificates is called mutual TLS. One place I have personally seen this used was internally at Netflix. And some enterprises apparently do use client certificates in browsers. eMudhra apparently wanted their new intermediate CA to be able to issue both server certificates and client certificates. The incident writeup doesn’t say why eMudhra wanted this capability.

But there was an issue with this new intermediate CA being able to issue client certificates as well as server certificates. The intermediate CA would be chained to their root CA (emSign ECC Root CA – G3 in the diagram above), which was in the Chrome Root Store. Google had long required that CAs below a root CA in the Trust Store had to be included in the Common CA database (CCADB). But they were about to add a new constraint: intermediate CAs with a root CA in the Chrome Root Store would only be permitted to issue server certificates, not client certificates.

Back in February 2023, on a Chrome Root Program Policy page called Moving Forward, Together, in a section called Focusing on simplicity, they discussed how they value CAs that focus exclusively on web use cases (emphasis mine)

Though multipurpose root CAs have offered flexibility in addressing subscriber use cases over the last few decades, they are now faced with satisfying the demands of multiple sets of increasingly rigorous expectations and requirements that do not always align. We believe in the value of purpose-built, dedicated infrastructures and the increased opportunity for innovation that comes along with them. We also think there’s value in reducing the complexity of the Web PKI to increase the security and stability of the ecosystem. By promoting dedicated-use hierarchies, we can better align the related policies and processes that CAs must conform to with the expectations of subscribers, relying parties, and root program operators.

They noted that they may add a restriction that CAs in the program only support server authentication use cases [emphasis in the original]

In a future policy update or CA/Browser Forum Ballot Proposal, we may introduce:

• a requirement that would restrict extendedKeyUsage values in new subordinate CA and TLS server authentication subscriber certificates to only “id-kp-serverAuth.” Modifying the definition of a dedicated TLS PKI hierarchy to prohibit use of other extendedKeyUsage values (i.e., id-kp-clientAuth) in new CA certificates capable of issuing TLS server certificates more closely aligns CA practices with web browser use cases (i.e., securing connections to websites). Further, it eliminates the risk and unintended consequences of other use cases (e.g., IoT) from negatively affecting website authentication.

Google eventually committed to this policy, announcing that they would be enforcing this as of June 15, 2025. Here’s the relevant bit from Section 3.2.1 of the Chrome Root Program Policy doc [emphasis in the original]:

The Chrome Root Program will only accept [Common CA Database] Root Inclusion Requests from Applicant PKI hierarchies that are dedicated to TLS server authentication certificate issuance.

To qualify as a dedicated TLS server authentication PKI hierarchy under this policy:

1. All corresponding unexpired and unrevoked subordinate CA certificates operated beneath an applicant root CA MUST:
   • when disclosed to the CCADB…
     • prior to June 15, 2025, include the extendedKeyUsage extension and (1) only assert an extendedKeyUsage purpose of id-kp-serverAuth OR (2) only assert extendedKeyUsage purposes of id-kp-serverAuth and id-kp-clientAuth.
     • on or after June 15, 2025, include the extendedKeyUsage extension and only assert an extendedKeyUsage purpose of id-kp-serverAuth.
   • …

The writeup mentions this date, so I assume this is what they were referring to. The thing is, though, their root CA was already in the trust store, and for intermediate CAs with roots already in the key store, according to section 3.2.2 of that document, the deadline was a year later: June 15, 2026.

At any rate, I assume eMudhra believed that if they wanted their intermediate CA to be able to issue client certificates, they had to disclose it to the CCADB before June 15, 2025. That meant they needed it cross-signed by Sectigo before then. Which meant that they needed to generate the keypairs, which required the HSMs.

Supply chain delays

The HSMs had been ordered in April of 2025. But there was a problem: they were taking longer to arrive than expected. The incident report states that “delivery of the HSMs ordered for the US facility was delayed by global supply‑chain constraints.” and doesn’t go into more detail than that, nor does it mention anything about the European facility.

What is the clear, though, is that this added production pressure. If the HSMs didn’t make it to the site quickly enough for them to generate the keys and get the certificates cross-signed by Certigo then eMudhra could miss Google’s June deadline for disclosing to the CCADB. Alas, the business implications of missing the deadline aren’t spelled out in the report, but it was clearly enough to act as a significant constraint.

Requiring WebTrust certified environments

The report says “It was also eMudhra’s intention to only operate HSMs within WebTrust certified environments…” Unfortunately, they don’t go into detail about why that’s their intention.

The Server Cert working group of the CA/Browser Forum requires compliance audits for CAs (see section 8), using on approved auditing scheme, which explicitly mentions WebTrust as one of these approved schemes. But it doesn’t explicitly say that an audit is required before operating an HSM at the site. However, what’s clear is that eMudhra had an internal policy about this.

Presumably, the HSMs had to be in place first before the WebTrust certification audit could happen, and that would introduce additional delays. Even if the HSM made it before the June deadline, they couldn’t use the HSM to generate the keypair because that would violate their policy.

Moving keys across HSMs from different vendors

The HSMs that the eMudhra engineers used to generate the key pairs were made by a vendor named Utimaco, but the new HSMs they had purchased for U.S. and European sites were made by a different vendor: Crypto4A. According to the write-up, the fact that the HSMs were manufactured by different vendors constrained their ability to transfer keys from one to the other. I assume based on this that vendors provide better support for transferring keys between HSMs of the same vendor. But in this case, the eMudhra engineers had to write a custom Python script to support the transfer, and the resulting script they wrote had a bug in it that led to the unintentional truncation of the private key to a single byte.

The outcome: a workaround leads to an incident

Based on the constraints they were facing, the eMudhra engineers decided to generate the keypairs for the European and U.S. sites at the (WebTrust certified) India site, and then export the keys out of the Indian HSM into the U.S. and European ones when they were able to. This was a workaround and … things went wrong during a rehearsal of the transfer process, and as a result they ended up with a cryptographically weak private key. You can read about the specifics of the failure mode elsewhere, it’s not my focus here.

Instead, my focus is how constraints shape behavior. Some of the constraints were business-oriented constraints (wanting to support mutual TLS, support for older clients), some were technical (transferring keys across HSMs manufactured by different vendors) and some were external (supply chain induced delays), but others were explicitly designed to improve security:

• Data sovereignty requirements
• Chrome Root Store Policy change to increase simplicity
• Hardware security module requirements
• WebTrust certification requirements

These are all well-intentioned, reasonable requirements, that nevertheless conspired against the eMudhra engineers in this scenario. The goals of each of these is to increase security, and yet, in this particular scenario, they each contributed to a significant security incident.

It’s also a textbook example of the role of production pressure. Look at how these factors interacted to create a hard deadline, and how that deadline influenced the work. It’s tempting to fault the engineers for doing the wrong thing here. But if you really want to understand how incidents happen, you have to look at how constraints influence decisions and workarounds.

As you’ve no doubt heard, the India-based public-key infrastructure company eMudhra recently announced that eight of their global TLS intermedia certificate authorities were cryptographically weak: it was trivial to reconstruct the private key from the public key, which meant that attackers could easily forge digitally signed certificates. If you’re a terminally online person like I am, you’ve almost certainly seen the numerous jokes referencing the number four, which was a reference to one of these weak private keys:

pic.twitter.com/cqnjWJCZdq

— cultofnot (@cultofnot) October 22, 2026

I’m anxiously awaiting the report with the full story about what happened. eMudhra has contracted with KPMG to investigate the incident, and hopefully they’ll share some public form of the report once the investigation is complete. In the meantime, Lex Neva pointed me at this bugzilla ticket with some preliminary details about the incident. Here’s part of a comment from Naveen Kumar ML, which I am assuming from context is an eMudhra employee. I’ve made some small formatting adjustments below.

We have done some preliminary assessment, there appears to be a flaw in the KGC [Key Generation Ceremony] script that was used on that specific day to create these ECC [Elliptic Curve Cryptography] issuers.

As for these ECC keys, our review of the KGC, assessment of the code and view of HSM [Hardware Security Module] keys so far indicate that:

a) a set of original ECC keys were created onboard as expected.
b) an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance
c) a flaw in the code resulted in the second pair of keys being created onboard which were weaker
d) these second pair of weaker keys were used by mistake to create the csr [certificate signing request] (because the key labels were similar) and signed by our root
e) the CSR from these weaker keys were also sent over to Sectigo for cross signing (we verified the files sent over to Sectigo).

…

Some additional context – these issuers created were meant for export to our US and European data centers to solve data sovereignty issues for some of our customers (but never ended up operationally in production except for one).

We are still assessing the code to see except [sic] steps where this process went wrong.

And here’s some text from Q7 of the eMudhra_Response_to_Google PDF attachment on that ticket:

At Key Ceremony 1, the PKI Operations script generated each B and C key, then wrapped, unwrapped and re-imported it as a planned rehearsal. From that day there were two objects with the
same label for each CA, and the participants created them. What we did not understand then was that
the key objects contained different data, and the CA certificates had been issued over the re-imported
objects and not the originally generated ones.

This stuff is outside my personal area of expertise, but I’m going to do my best to describe in my words what happened at eMudhra back on April 30, 2025.

That day, eMudhra was creating a new certificate issuer. One of the steps involved in creating a new issuer is generating a public-private key pair, and the process for doing that is called a key generation ceremony. Nothing went wrong with the generation of this initial key pair: the private key was secure.

It’s not explicitly noted above, but my assumption is that the key pair was generated on Indian servers. Some of eMudhra’s customers have data sovereignty requirements: certain data has to be housed in a particular geographic region. eMudhra specifically calls out the need to house issuers in US and European data centers, so presumably these are American and European customers. As a consequence, eMudhra has to securely transmit the private key to servers based in U.S. and Europe.

One thing I learned from reading this is the process of encrypting a private key so it can be transmitted is called wrapping, and decrypting it is called unwrapping. During their key generation ceremony, they tested wrapping and unwrapping. And, somehow, (this appears to not yet be fully understand), a new (insecure) key was created rather than decrypting the original (secure) key.

What really jumped out at me here was that the insecure keys were introduced by a process change that was necessary due to data sovereignty customer requirements. Data sovereignty is itself a security concern; these customers imposed the requirement in order to increase security. And, yet, that requirement ultimately contributed to the creation of a security vulnerability. Even more ironically, this was described as a test of the wrap/unwrap process. Testing contributed to the introduction of a security vulnerability!

This is not to say “data sovereignty is bad” or “testing is bad”, even though both activities contributed here. Rather, it is yet another reminder that all of our actions involve risk and trade-offs, even if we are not explicitly aware of these. In particular, adding additional requirements, even security or reliability requirements, invariably means adding additional complexity to the system, which introduces new risks, whether we are aware of them in the moment or not. Never forget: all practitioner actions are gambles.

On Mastodon, Ole Peder Brandtzæg pointed me to (yet) another GitHub incident that happened back on Sept 23. The write-up is very short, so I’ll just excerpt the whole thing here (emphasis mine):

Starting at 07:57 UTC on September 23, GitHub experienced elevated 500 and 404 responses across several application pages. This caused failures when installing GitHub Apps, creating organizations, and making some organization membership changes. Customers also experienced delayed label updates and stale search results in Projects.

The infrastructure failure was isolated to our Azure Central US region. The API errors were mitigated by 10:58 UTC on September 23. Projects’ processing continued to recover while an accumulated backlog was drained, and full service was restored at 04:55 UTC on September 24.

The incident was caused by a failed planned maintenance operation on a primary database. Automated recovery initiated an emergency database failover, after which several replicas in the affected region were unable to resume replication correctly. This reduced available database capacity and caused the API errors and downstream Projects processing delays.

We have mitigated the immediate failure mode. We are also improving maintenance safety checks, database failover handling, post-failover replica validation, and downstream processing resilience to reduce the likelihood and impact of similar incidents.

There really isn’t much to go on here, but let’s make the best of it.

Unlike previous GitHub incidents, this one doesn’t sound like a “hit a tipping point due to increased load” sort of failure mode. Instead, the trigger here was planned operations work. They don’t discuss the rationale for the planned maintenance, and so it may very well be that the work was being done to increase capacity, in which case it does rhyme with the other “increased load on GitHub” incidents. Or perhaps the planned maintenance was intended to deal with some other issue. There just isn’t enough information here.

The incident was caused by a failed planned maintenance operation on a primary database.

All we really know from this write-up is that something went wrong with the maintenance operation, because it describes the operation as “failed”. From the write-up, it sounds like they were running a database cluster, where they had a primary and replicas.

The nice thing about this sort of setup is that, if the primary fails, then a replica can automatically take over as primary, which significantly reduces downtime, which is how this database cluster was configured.

Automated recovery initiated an emergency database failover, after which several replicas in the affected region were unable to resume replication correctly.

But, again, all we get from the write-up is something went wrong with this process. It sounds like a new replica was successfully promoted to primary, but the other replicas were somehow unable to connect to the new primary.

Another advantage of a primary-replica style database cluster is that you can divide up the read request traffic across the entire cluster, which gives you more capacity. Conversely, if you lose the replicas, then the primary has to service all of the requests. This increases the load on the primary.

This reduced available database capacity and caused the API errors and downstream Projects processing delays.

Based on the write-up, it sounds like the primary was not able to handle all of the traffic that resulted from losing the replicas. Note that even though this incident was not triggered by an increase in load, it still ended up being a saturation-related failure mode because of the loss of capacity due to the effective loss of the replica nodes.

Looking over our omnipresent availability risks, it might hit the following three:

• problem areas > saturation > databases (primary was saturated)
• essential non-standard changes > mitigating an operational issue (planned maintenance went wrong)
• essential increase in essential complexity > reliability subsystem (database failover went wrong)

Finally, even with the lack of detail here, we can still see hints of how multiple factors contributed to this incident:

• the original need for the planned maintenance
• whatever it was that went wrong during the maintenance itself
• whatever it was that prevented the replicas from reading from the new primary
• the primary node alone having insufficient capacity to handle all of the resulting traffic

Lex Neva (of SRE Weekly fame) pointed me to this recent GCP incident report, which describes networking issues they experienced in their us-central1 region. This incident is yet another reminder of why I don’t work in networking: I think that networking engineers are required to work within systems that are inherently unsafe by their very nature.

Now, I’m on the record for stating that human error is not a useful construct. However, I recognize that I’m in the minority here, and that most people believe that human error is a real and useful thing to talk about, so I’ll go with it here. I believe that for a system to be safe, it needs to be error-tolerant. An error-tolerant system is a system that does not fail hard even when a human makes an error. Achieving error-tolerance means having things like cross-checks, and the ability to recover quickly if an error does lead to a genuine problem. If you think about the way we typically roll out code to production, there are typically multiple checks involved (e.g., code review, unit tests, CI checks, tests done in a staging environment, canarying). In addition, we use techniques like progressive rollouts, blue-green deploys, and feature flags to reduce the harm if bad code makes it out to production.

I’m not a network engineer myself (I’ve never managed more than a single rack of servers, which involves a grand total of two switches), so I can’t speak from direct experience about the nature of the work. But every time I read an incident write-up that involves networking, it screams to me “this is not an error-tolerant system that network engineers have to work within”. Whenever you have to muck around with networking gear, very bad things can happen. From the GCP report, here’s the entire (sigh) “Root Cause” section (emphasis added):

The event was triggered by the inadvertent physical disconnection of network fiber-optic cables during routine hardware maintenance. A technician was performing a scheduled capacity upgrade on data center routers that support a fraction of capacity in the us-central1-b zone and a small fraction of capacity in the us-central1-f zone.

During the hands-on maintenance, optical fibers connecting the data center routers to the network fabric were unplugged and the optical transceivers were replaced with a transceiver that supports higher-density connectivity. This transceiver was not compatible with the transceivers still in place in the data center fabric, causing a loss of connectivity for each fiber.

The network is designed with redundant routers in separate data center rooms within the same building so that failure of one router does not interrupt service. The maintenance was intended to upgrade one router at a time over several days, with traffic diversions at each step and verification between steps that the network has returned to a fully-connected state.

However, a procedural error in the manually orchestrated upgrade process caused the complete list of transceiver replacements across all routers to be issued to the technician without instructions to sequence the work one router at a time. The maintenance workflow also did not include the expected human and software verification steps for detecting unintended disruption.

As a result, the technician sequentially unplugged all fiber paths across the affected devices within 13 minutes. The speed and nature of the error prevented warnings of incorrect action from reaching the engineer before connectivity was lost. Additionally, a standing procedure to halt if light is detected on any fiber-optic cable after unplug was not followed.

This resulted in compute capacity in the impacted zone being isolated from the network. Customers were unable to reach their virtual machines, and those virtual machines could not establish connections outside their zone.

The report attributes the incident to two errors:

1. correctly following an incorrect procedure
2. incorrectly following a correct procedure

Contrast the paucity of this analysis with the talk that Sean Klein gave at QCon SF 2025 about an Azure global networking outage that occurred back in January 2023. That also involved a networking engineer carrying out a change where the instructions contain a procedural error. However, Klein goes very deep into the history of how that came to be.

We’re also provided with no context about the nature of the work, so I have no idea how it came to be how the procedural error was introduced, or that the engineer didn’t follow the standing procedure. (Maybe they were distracted by some other aspect of the work that is part of the procedure?) All we have is the hindsight judgment that they didn’t do the right thing.

This line also jumped out at me:

 The maintenance workflow also did not include the expected human and software verification steps for detecting unintended disruption.

Our procedures are never going to be perfect. And that’s exactly why I wince every time I see “human didn’t follow the procedures” as a reason for an incident. Documented procedures are a tool, and the human consuming them has to use their judgment about how to apply the tool. If your system assumes humans perfectly following perfect procedures then, well, you’re in for some surprises.

Here are the follow-up action items (emphasis mine):

• Reinforce training in us-central1 region and globally around safe maintenance techniques, including the mandatory requirement to verify whether light is on every unplugged fiber.
• Finish the migration of network upgrade workflows to a fully automatically sequenced orchestration system. Most common regional network workflows were completed in 2025, and zonal workflows are in progress.
• Complete the deployment of work stop alerting for actions resulting in the unintended disconnection of live fiber cables in Google data centers.
• Complete the deployment of the automated system that will move regional traffic away from faulty zones, reducing the time from approximately 19 minutes in this instance to around 5 minutes.

Now, I’m confident that Google did a more extensive investigation than what they wrote up. But even so, attributing the outage to two human errors, and listing training as an action item (blame and train!) are big red flags to me that they don’t understand how work actually gets done in their own system.

I also note with interest how the the other actions involve deployment of new automation, which will likely introduce novel failure modes even as they eliminate the sorts of failure modes that occurred in this incident. I hope the Google SREs are ready for them.

In the meantime, I recommend they check out the book Behind Human Error.

A hallway in a tech company office. A and B are standing in the hallway. B is holding a multiple page document.

A: Wow, that incident from a couple of weeks ago was a real doozy. Do you know how it happened?

B brandishes the document.

B: As it happens, I just finished reading through the incident write-up. They did a pretty thorough investigation.

A: You printed it out? I didn’t know we even had printers in the office anymore.

B: Yeah, I find it easier to read on paper than a computer screen. Anyways, there’s a lot of great detail here, about the history behind the changes that contributed to the incident, about how confusing things were for the responders during the incident response, the tools that the responders used to diagnose what happened, the levers they pulled to actually mitigate the customer impact… I learned a ton from it. Would you like my copy?

A looks skeptically at the document.

A: Wow, that’s a long report.

B: Yeah, it’s about thirty pages. But that’s what it took the authors to explain everything. Like I said, lots of great detail.

A: I don’t know about you, but I’m pretty busy these days.

B: To be honest, I am too. The upcoming launch is keeping my whole team really busy. Not to mention all of these migrations that are going on, that we need to spend time on.

A: Tell me about it! I just don’t have the time to spend reading a doc like that. Can you just tell me the high points, so I can get the most out of it without reading the whole thing?

B: Well, the thing is, I’m pretty sure you’d learn different things from the write-up than I would. I’m sure there’s stuff in here that you already know, that I didn’t. And also, vice versa.

A: Hmmmm…

B: And even if I try to recount what I learned… well, it’s not the same as reading it. I’m not good at articulating what I learned. Just giving you a list of bullet points is not going to be as effective at just going through and reading the whole thing. And, from experience reading lists of “learnings” don’t stick in my head the way the same way as the stuff I learn reading a well-written incident write-up like this one.

A: I see what you’re saying, I really do. I’d like to be able to sit down and spend the time really reading through this whole document. But, given all of the current obligations that business has placed on me, I don’t have that time to invest. I want to learn from this incident, and I know I would learn more if I spent the time on reading the whole thing, but there’s a limit to how much time I can commit to that. So I need to have the important details summarized.

B: Just to make sure I understand: you’re saying that, even though you admit you’d learn more if you read the entire write-up, because of the various work pressures that you are facing, you’re making an explicit trade-off to reduce the amount of time you spend on this task, in order to have the time to spend on other tasks you have to work on?

A: Yes, exactly!

B: Well, now you know how the incident happened.

END OF SCENE

On July 8, 2026, the Australian telecom company Telstra experienced a significant outage related to an NTP issue. Telstra hired a consulting firm to do an incident review, and a couple of weeks ago, that firm released a public report: Findings Report on the July 8th Telstra Outage.

I’m using this blog post to write up my understanding of the failure mode. I found the report was often missing important details that I needed to make sense of what happened, so I tried to fill the gaps by looking at additional sources, and made my best guesses where I could. It was also more difficult for me because I don’t have domain expertise in any of the technologies involved here: I have some basic familiarity with NTP, and I have no familiarity at all with the various mobile telecom technologies involved.

Note: all block quotes in this post are from the public report unless explicitly indicated otherwise.

2010-2020 Telstra NTP system architecture

Telstra runs three NTP servers in their mobile network, geographically distributed at telecom exchanges across Australia.

These servers provide time synchronization to their clients, which could potentially be other NTP servers.

Note that some NTP clients are configured to point to multiple servers for redundancy. The diagram above shows one such client. Clients connected to multiple NTP servers will play an important role later in the incident.

According to the report, starting in 2010, Telstra ran two NTP servers at the Sydney and Melbourne exchanges. At each site, one server ran at stratum 2, and the other ran at stratum 3. The Perth exchange ran a single NTP server, at stratum 3. For their stratum 1 sources, the stratum 2 servers were clients of NTP servers hosted at Australia’s National Measurement Institute. It is the Melbourne site that will play the most significant role in this incident.

A note about arrows: NTP diagrams typically show arrows going from lower stratus to higher stratus (e.g., “stratum 1 → stratum 2″). However, I’ve drawn the arrows the other way because I’m used to the “client → server” convention of arrows directions.

I’m assuming that the NTP clients communicated only with the stratum 3 server, but there isn’t enough detail in the report to verify this was the case. I really wish they had drawn diagrams in the report!

For the purposes of this incident, the important detail here is that each stratum 3 server has a redundant stratum 2 source: all of the stratum 3 servers can use either the Sydney or Melbourne stratum 2 servers.

An upgrade leads to an old redundancy being lost, and two new redundancies added: all three changes will contribute to the incident

Six years before impact

In 2020, Telstra upgrades the timing hardware on their mobile network. With this upgrade, their NTP servers now run on Microchip SSU 2000 synchronization supply units, with the supporting vendor being a local company: Scientific Devices. This upgrade results in three significant changes to the architecture.

No more local NTP sources

Due to a limitation in the NTP timing chassis, a stratum 2 NTP server could not provide time to a
stratum 3 NTP server in the same chassis. – Findings Report on the July 8th Telstra Outage

With this upgrade, they are no longer able to have a stratum 3 NTP server running on the SSU 2000 use another NTP server on the same unit as its source. As a consequence, the Melbourne and Sydney stratum 3 NTP servers lose a level of redundancy: they now each have only one stratum 2 source.

Unfortunately, the writeup doesn’t explain why the limitation existed. I tried to figure this out by reading what SSU-2000 docs I could find. My best guess was that this was a security restriction, as described below (emphasis mine). For context: the PackeTime module is the thing that runs NTP on the SSU 2000, and NTP runs on the service port. “Independent mode” means that the two Ethernet ports of the module run independently (i.e., bonded), which I presume is how they had things set up so they could run two independent NTP servers.

In the Independent mode, the two ports are each assigned unique IP addresses and can each serve time to independent networks. From a security perspective, the important thing to note about the Independent mode is that the PackeTime Module does not support routing protocols between the two service ports or between the service ports and the maintenance port. This prevents a malicious attack on Port A (network A) to be used to send a malicious attack via Port B (network B) or vice-versa, or between Port A or B networks and the maintenance port’s network. – SSU-2000 Synchronization Supply Unit and the SDU-2000 Synchronization Distribution Unit: User Guide, Revision G – June 2008, Chapter 8, Output Module Reference Data, PackeTime Module, p274

GPS card for failover

However, while the upgrade cost them one form of redundancy, it granted them another form: each SSU 2000 unit had a GPS card in it. GPS gives you access to a high-accuracy clock, since it receives time information from multiple satellites which each have atomic clocks in them. The presence of a GPS card gave the NTP servers running inside of the SSU 2000 access to a stratum 0 source. The GPS card was therefore available as a failover time source if the stratum 3 NTP servers lost access to their (now sole) stratum 2 source.

But there is a problem lurking within this failover solution: a GPS rollover bug in the card.

Internally, GPS keeps track of the number of weeks since January 5, 1980, and it uses 10 bits to represent this count. This means that the count rolls over every 1024 weeks (roughly every 19.7 years). Some GPS devices were not originally designed to handle the rollover. The GPS card in the SSU 2000 is one of those devices. (One thing I learned here is that while the GPS rollover itself happened on April 6, 2019, devices that are vulnerable to rollover don’t necessarily roll over on the same day. For the SSU 2000, the rollover was September 18, 2022).

This means that the GPS card’s date will jump backwards in time after it rolls over. The rollover date is September 18, 2022. But just hitting that date isn’t enough: as long as the card is still running, it will behave properly. It’s only when it is powered off and on after that date that the rollover will happen.

Running the symmetric variant of NTP

There is also one additional architectural change that happens in the upgrade.

In the client/server variant, a persistent client sends packet mode 4 packets to a server, which returns packet mode 3 packets. Servers provide synchronization to one or more clients, but do not accept
synchronization from them…

In the symmetric variant, a peer operates as both a server and client. – RFC 5905

The mobile core timing architecture was changed from client/server to peering, meaning that the
NTP server in the NTP timing chassis could acquire any NTP source that it could find and select to
use any of these sources that was providing the most stable and accurate time. This introduced
uncertainty in the time sources being used in mobile core timing hierarchy. – Findings Report on the July 8th Telstra Outage

Here’s where I struggle the most with the writeup, because I’m not familiar enough with the underlying technology. RFC 5905, which describes NTP, does not use the term “peering” anywhere, but it does talk about multiple variants of the protocol, including the client/server variant and the symmetric variant. I assume peering in the report is referring to the symmetric variant.

Here’s an example of behavior which I believe can only happen with the symmetric variant. Consider the following network of five NTP servers, labeled A-E.

Now, assume that server A goes down. This means that node B no longer has a source. When running in client/server mode, C is a client of B, which means that B cannot be a client of C. However, when running symmetric mode, B can become a client of C, like this:

This behavior will eventually contribute to increasing the severity of the incident by propagating bad time data further through the network. But we’re still six years away from that.

SSU 2000 GPS rollover day comes and goes with no issues

Four years before impact

On September 18, 2022, the rollover day for the SSU 2000, nothing happens. Because the GPS cards have been running uninterrupted, they do not roll over: that only happens after being reset. In addition, the cards are not even in use, since they are only used for failover, and failover has not occurred. The problem will only manifest when the GPU card is restarted, which will not happen for years.

Melbourne NTP server cannot reach Sydney, engineers fail over to GPS card

Nine months before impact

Several months before the outage, there’s a minor incident where the stratum 3 NTP server in the Melbourne exchange has been experiencing errors attempting to connect to its stratum 2 source in the Sydney exchange:

In October 2025, personnel were working on the Melbourne NTP server due to an incident report that the NTP stratum 3 clock was losing connectivity multiple times per day to its stratum 2 source clock at Sydney, resulting in NTP alarms.

The report notes that the server used peering to dynamically identify another NTP server, which happened to be a stratum 4 source, leading the Melbourne NTP server to become a stratum 5 source. However, they don’t know anything about this server:

The incident report also noted that the Melbourne NTP had become a stratum 5 source meaning that it was receiving time from a stratum 4 source in the mobile core timing architecture. Remember, Melbourne had only one stratum 2 source and when that source was lost, it used peering to find another source which happened to be a stratum 4 client of Sydney or Perth.

According to the report authors, this meant that timing loops were already present in the system (emphasis mine)

It is critical to have at least two independent reference sources. A timing loop is when sources that
appear independent are in fact validating one another – directly, or by tracing back to a common
reference. To avoid timing loops, you must clearly define the server hierarchy and document all
relationships between servers.

…

looping was already occurring in mobile core timing when an intended NTP stratum 3 server was able to become a client of one (or more) of the clients and that a circular loop was established when the stratum 2 source was lost. 

My understanding of the topology at the time of the October 2025 incident is captured in the following diagram, where the red arrow indicates a failed connection, and the blue arrows are network edges that were not in the previous diagram. Here I’m assuming the client in question was connected to the Perth stratum 3 server, but it may just as well have been the Sydney stratum 3 server.

While this evidence indicates that looping was already happening in the network in October 2025, the looping itself had not yet contributed to an incident: that will come later.

The network engineers mitigate the incident by failing over the Melbourne NTP server to use the GPS card as its source.

To resolve this issue the GPS card was activated and connected to the stratum 3 NTP server in Melbourne as a means of replacing the lost Sydney stratum 2 source. the incident appear to have chosen this approach as the best means to address the incident. The GPS card is known as a reference source (aka stratum 0) and so this connection made Melbourne a stratum 1 NTP source, effectively the same level as the NMI stratum 1 sources in the mobile core timing architecture. This impacted the clients of Melbourne, making them stratum 2. In peering mode, these stratum 2 clients can be time servers at the same priority as Sydney and Melbourne stratum 2 who were the intended stratum 2 sources.

A reminder from the SSU 2000 vendor about the firmware update

Seven months before impact

In January 2026, the vendor supporting the SSU 2000, Scientific Devices, sends a reminder to Telstra that there is a firmware update for the SSU 2000 related to the GPS card.

Here I’ll quote the CEO’s Senate testimony (emphasis mine):

It’s still subject to our investigation, but there was definitely a reminder in January of this year that
this update needed to be done. We believe it does go back several years that it was there. To provide some context, our investigation so far has shown that it’s not a case of the teams just ignoring that update. They did consider it. The way this was designed and the way we were using this particular server and this card, this update was affecting a feature that we were not using at that time.
…
We know that our teams did get the alert of a software update. As we said, they assessed that through our investigation, we can see, and made a choice. Again, this is not an excuse; it’s putting it in context. There are many software updates and changes getting made across the network. Any software update creates a risk, and we put it into testing. In this case, they thought, given the design that was in place, it was not relevant because we were not using this equipment in the way that would trigger this risk. – (Telstra CEO) Vicki Brady, Senate Environment and Communications References Committee

It sounds like the engineer who made the call was not aware that the GPS card was a failover source. So was the use of the GPS card an in-the-moment workaround using available resources, or was it an explicit part of the architecture that the engineer making the call about firmware updates was not aware of? Or was it something else? There simply isn’t enough information to tell. The public write-up implies that it was part of the architecture:

The architecture indicated that the GPS card was available to be used as an alternative failover source for time of day.

On the other hand, we have this Senate testimony from the Group Owner, End-to-End Service Performance and Resilience of Telstra

Based on the intentional design of that SSU 2000, the GPS module is not used as part of that design. Gerard Tracey, Senate Environment and Communications References Committee

I don’t know how to reconcile those two statements, beyond “different people in the system had different understandings of the architecture”. If you think you can eliminate these sorts of mismatches in understanding in your organization, good luck to you!

One last thing I want to point out here is the CEO’s comment that “any software update creates a risk”. So, not doing updates is dangerous, but doing updates is dangerous as well! It’s easy to judge the engineer in hindsight for this decision, but I have personally seen many incidents that were triggered by performing software updates.

Impact starts: A repair triggers a poisoned clock

On July 8, 2026, Telstra network engineers took the SSU 2000 offline in Melbourne in order to fix an issue with a redundant power supply:

When the maintenance team came to do the work on this server, they were required to power it down and take it offline because they needed to replace what we call the chassis, the case that it sits in, because its backup power supply was not working. So they put in a brand-new chassis. – (Telstra CEO) Vicki Brady, Senate Environment and Communications References Committee

The report doesn’t say how long the unit was offline, nor does it discuss what happened to the Sydney stratum 3 NTP server that had Melbourne as its sole source. One thing that’s clear is that there was no system impact while the unit was down.

It was when the unit came up again that the trouble started. At 2:50 AM, the unit is powered up and comes back online. When the GPS card comes back online, the rollback issue manifests, sending its clock back 19.7 years, from July 2026 to November 2006.

The poisoned clock time spreads across the network

Since Melbourne was operating at stratum 1 and peering was enabled and Melbourne stratum 2 had been turned off by the same chassis replacement, the incorrect date propagated throughout the mobile core timing network.

Because the Melbourne NTP servers had gone down for maintenance, Sydney lost its source for its stratum 3 server. While presumably it had a GPS card to fail over to, they were configured to only fail over manually. So, unlike Melbourne, Sydney does not switch to using GPS card as a source.

However, because the NTP servers are configured to support peering, the Sydney NTP server is able to use one of its clients a source. When Melbourne comes back up, an NTP client common to the two sites becomes a stratum 2, using Melbourne as the source. The Sydney NTP server uses this client as its time source, and now the poisoned time has propagated to the Sydney site, with its clients impacted as well.

(Note: The report does not say whether the Perth NTP deployment was impacted by this, so I’m going to assume it wasn’t).

Impact: mobile users can’t call or use data

70 minutes after impact start

Starting at around 4am July 8th, 2026, when Mobility Management Entity (MME) instances began to reset, end users of Telstra’s network experienced varying levels of impact in their ability to communicate on the network.

At this point of the write-up, I’m really out of my depth, because the details of the failure are telecom-specific. I don’t even know if an MME instance is a software thing or a hardware thing. Based on this case study on Ericsson’s website, I assume that Ericsson is the vendor, and according to this Ericsson page, it sounds like it could be either one, as it says: Our latest software release can be deployed as a physical network function (PNF), as well as a virtual network function (VNF).

What I do know is that it provides a control plane function that communicates between users’ phones and the mobility network. When these instances reset, it broke mobile functionality, including emergency service calls (Triple Zero in Australia).

These impacts precluded some users from being able to make/receive voice calls, including Emergency Calls (per the Emergency Call Service Determination:2019, also referred to as Triple Zero calls), send/receive messages, or access mobile data services. Those without data services were unable to do things such as access the web, stream content and most mobile applications were unusable as well. 

During the worst of the incident, 45% of mobile users were impacted:

At its peak, this initial issue impacted approximately 45 per cent of all calls and data sessions on our mobile network. – (Telstra CEO) Vicki Brady, Senate Environment and Communications References Committee

Impact: valid IP addresses get blocked

Until the correct network date was restored, information for approximately 30,000 IP address prefixes were incrementally corrupted (written with the wrong date) by the Diameter Signalling Controller (DSC) in such a way that access attempts using those addresses were blocked from using the network regardless of available capacity.

My guess based on this is that a Diameter Signalling Controller (once again, I’m guessing they’re using an Ericsson product) is that the DSC keeps a table of session information about active IP addresses on the network. And because entries got written to the table with the rollback date, the system interpreted these as invalid and did not allow these addresses to access the network.

Note that this is a poisoned data scenario: network engineers had to fix these records before the users could regain access.

Telstra, with help from their equipment vendor, applied a workaround during the Outage and then at a later time performed repairs to remove the bad IP address prefix entries.

Omnipresent risks

A couple of weeks ago, I wrote a post titled omnipresent availability risks in cloud software. Here I’ll point out some of those risks that I see manifesting in this incident.

Problem area: saturation

This is a pretty clear example of an incident that involved saturation. The original MME instance resets don’t take out the entire mobile network, but they have the effect of reducing capacity enough that the remaining capacity is not sufficient to meet demand. From the report:

Initially users were impacted by a loss of network capacity (due to failure of MME instances) which caused some users to be disconnected from the network. The remaining capacity was insufficient to enable those users to all reconnect deterministically nor was it capable of servicing the nominal network traffic load. So essentially the network was overloaded.

Problem area: networking

Admittedly, this one is a gimme: every large-scale telecom-related outage is going to be networking related, since what telecoms provide is network access! That was no exception here.

Problem area: security

It’s unclear to me in this case whether security systems played a role at all in this incident. I hypothesized that the loss of redundancy was due to a security restriction in the SSU 2000 unit, but I don’t know for certain that’s the case. It’s also possible that the Diameter Signalling Controller blocking the IP addresses was related to a security function, but there’s not enough information.

Essential non-standard changes: mitigating an operational issue

This is a clear case where the act of mitigating an operational issue was itself a contributor to a new incident. The trigger for this incident was repairing the power supply of the SSU 2000 unit in the Melbourne exchange. The repair itself went fine: it was powering up the unit that exposed the latent bug in the GPS card.

Essential increase in essential complexity: reliability subsystem

The GPS card, which was the source of the poisoned time, was a reliability subsystem! It exists to provide failover functionality in the case where an NTP lost access to its source. And unexpected behavior of that reliability subsystem, when engaged, bit them hard.

Essential increase in essential complexity: migration

I think Telstra’s 2020 upgrade of their timing hardware qualifies as a migration. This resulted in architectural changes that created a system vulnerability, although this vulnerability would not manifest for six years.

I do appreciate how the CEO was sanguine about how the nature of these sorts of changes means that incidents are inevitable.

What I would say is that networks and environments like a mobile network are absolutely complex, and it’s evolving—any sort of complex technology environment—and certainly networks are not infallible. But our job is to make sure we are taking the actions and those steps that will mitigate as best we can. I would love to be able to sit here and say will be zero outages. The reality of a complex network environment with fast evolving technology is that no telco around the world could guarantee that. – (Telstra CEO) Vicki Brady, Senate Environment and Communications References Committee

Some parting observations

There is no NTP haiku

In my experience, NTP isn’t a technology that frequently shows up incidents, at least not compared to other networking protocols such as DNS or BGP. The authors of the public report takes Telstra to task for a lack of sufficient NTP expertise within the company, and I can imagine playing a more significant role in the telecom domain than in other cloud software domains. But I can’t help but wonder if the fact that NTP isn’t generally problematic contributes to organizations having less in-house NTP expertise.

Appalling incuriosity from the investigators on how the work gets done

the need for stronger competency and more curiosity to investigate when something looks wrong

As is always the case in these sorts of public reports, there’s a lot of ink spilled on the subject what engineers should have done in the run-up to the incident but didn’t. The bullet “Increase attention to the operation and management process discipline in all areas” pretty much says it all.

Whenever I see this sort of content in a report, my response is, “boy, the investigators really blew it.” And that’s because you can’t understand how the incident happens unless you understand how the work really gets done within the organization. The authors show an acute lack of curiosity about the circumstances of the individuals who were involved in making the decisions.

I’ll note that these authors managed to mess up the section numbers in their report: it goes 1,2,1,2 instead of 1,2,3,4. This leads them to reference a non-existent section 4.1. I would ask the authors: Do you think we should be taking advice on how to run a reliable telecom network from people who can’t even number sections correctly in a ten page document? Or do you think we should consider the circumstances that could lead to this sort of error?

Where are the other system vulnerabilities?

We observed instances where other areas and elements of the network and organizations were performing at better levels than NTP. 

Finally, it’s easy to point out problems in a system that were exposed by a recent incident. The really impressive thing, to me, is to identify the problems that were not exposed here. I have high confidence that there are other architectural vulnerabilities in the system, just waiting to be turned into incidents because of future changes. And yet, somehow, investigators never manage to find new vulnerabilities that could lead to the next outage. Funny thing, that.

References

• Findings Report on the July 8th Telstra Outage By Technology Audit Partners, August 24, 2026.
• As it happened: Telstra execs grilled at Senate hearing into outage, David Swan and Elias Visontay, Sydney Morning Herald, July 17, 2026.
• Senate Environment and Communications References Committee, Triple Zero service outage, Committee Hansard, July 17, 2026
• Triple Zero service outage Submission 7 – Supplementary Submission: Environment and Communications References Committee inquiry into the: Triple Zero service outage, July 16, 2026
• NTP servers: S200 and S300 Series Internal Week Number Rollover event on September 18, 2022, community.ntppool.org, Steve Sommars, July 27, 2022
• The 18th September 2022 Rollover Event, Chronos Technology, Louise Davies, June 9, 2022
• RFC 5905: Network Time Protocol Version 4: Protocol and Algorithms Specification, D. Mills, J. Martin, J. Burbank, W. Kasch, June 2010.
• SSU-2000 Synchronization Supply Unit and the SDU-2000 Synchronization Distribution Unit: User Guide, Revision G – June 2008

The other day, I wrote about a GitHub Actions incident. But GitHub Actions isn’t the only CI-as-a-service out there. Buildkite, which is another CI company, recently released a public write-up of an incident they experienced last week. It’s a great one: their incident report contains a lot more detail than GitHub’s. Here I’ll discuss my own observations.

The failure mode

The write-up describes what I found to be a fascinating failure mode. Here I’m going to do my best to explain it in my own words.

A bit about Kubernetes

First, some Kubernetes terminology to get us all on the same page. When you’re using Kubernetes to run your services and jobs, we say that you are running on a Kubernetes cluster. A cluster is made up of multiple nodes. If you’re running on EKS (Amazon’s Kubernetes offering), I believe those nodes are going to be EC2 instances.

Each node exists to run pods, which are the compute workloads that you run on Kubernetes. If you’ve been a service owner for a service that runs on Kubernetes, then you’re almost certainly familiar with the notion of pods, because that’s what’s exposed to you. But, unless you’ve worked in infrastructure, you’re likely not familiar with nodes and clusters, because those implementation details are deliberately hidden from the service owners.

Here’s a diagram that shows a four-node Kubernetes cluster. Each black rectangle depicts a pod that is running within that node. A pod is a set of containers that all get scheduled together (typically an application container and related sidecars), but that particular detail isn’t relevant here.

I’ve also labeled the blank part of one of the nodes as headroom. That’s not explicitly a Kubernetes concept. Rather, it’s a general term that here means “the amount of additional compute jobs that could still be run on this node.”

A few brief words on autoscaling

Let’s take the service owner’s perspective again. You own a service that runs on Kubernetes. That service is deployed as a group of pods, what Kubernetes calls a replicaset. Note that each pod is identical, so the only reason to have multiple pods is because you need additional compute resources. The amount of compute resources that your service needs can vary over time. For example, if you’re driven by requests, then the volume of requests can vary. Or, if your service consumes from a job queue, the number of pending jobs can vary at any one time.

Kubernetes deals with workloads whose resource needs vary with time by supporting autoscaling. The system I’m most familiar with is the HorizontalPodAutoscaler, which will increase or decrease the number of size of a replicaset depending on the load.

---

A brief sidebar about scaling terminology. There’s a distinction between horizontal scaling and vertical scaling. If you’re adding more servers, you’re horizontally scaling, whereas if you’re increasing the size of your servers, you’re vertically scaling. Technically, it’s more accurate to use the term scale out to refer to horizontal scaling, and scale up to refer to vertical scaling. and, indeed, the Buildkite write-up uses scale out. But I’m used to saying scale up for horizontal scaling as well, because that’s the kind of scaling I deal with more often. In this post, I use scale up and scale out interchangeably.

---

Here’s an example of a graph of a deployment that is undergoing autoscaling.

The red line indicates the desired replica count. That’s the number of pods that the autoscaler has determined should be running. At the beginning of this graph, the desired number of pods is 6. Later on, the autoscaler adjusts the desired replica count to 8.

The green indicates the number of pods that are in a running state. Note at the beginning of the graph that there are 6 pods in running state, and at the end there are 8 pods. Here’s how the Kubernetes docs describe the running state:

The Pod has been bound to a node, and all of the containers have been created. At least one container is still running, or is in the process of starting or restarting. – Kubernetes pod lifecycle docs

The gray is the interesting bit. That shows the pods that are in the pending state. Here’s how the Kubernetes docs describe the pending state:

The Pod has been accepted by the Kubernetes cluster, but one or more of the containers has not been set up and made ready to run. This includes time a Pod spends waiting to be scheduled as well as the time spent downloading container images over the network. – Kubernetes pod lifecycle docs

After the autoscaler increases the desired replica count to 8, another Kubernetes controller sees the difference between current replica count (6) and desired (8), and spins up two new pods. But those pods don’t immediately start running. Each new pod has to be scheduled onto a node in the cluster that has headroom. Until that happens, the pod is in the pending state.

A deployment consumes all of the headroom, workers try to scale up

Let’s turn back to the write-up:

At 22:44 UTC, an application deploy created a surge in application Pod volume… This surge consumed the available headroom on already-deployed Nodes.

When you deploy a new version of your software onto a Kubernetes cluster, the cluster spins up new pods with containers that are running the new version. In this case, these pods new consumed all of the available resources on all of the nodes of the cluster.

In Buildkite’s case, in addition to an application deployment eating up all the headroom, there was a replicaset of background workers that was being scaled up by the pod autoscaler at the same time. From the write-up:

At 22:44 UTC, an application deploy created a surge in application Pod volume… This surge consumed the available headroom on already-deployed Nodes, which limited applications’ capacity to autoscale promptly. Some background workers (including the aforementioned notification workers) were also attempting to scale out at this time. 

So, we have one application (background workers) trying to increase its replica count, at the same time as another application has eaten up all of the available capacity.

You can scale up your k8s cluster too, but you’ll need to wait

At 22:44 UTC, an application deploy created a surge in application Pod volume, which caused our EKS cluster to scale out. (emphasis added)

Just like a service owner can configure autoscaling for their service, a Kubernetes administrator can configure autoscaling for their cluster: when it starts to run out of headroom, the cluster autoscaler can automatically request new EC2 instances from AWS. Instead of using the horizontal pod autoscaler (which is for scaling pods), you use a cluster autoscaler, like Karpenter, which happens to have been written by AWS. But the general concept of autoscaling is the same here.

However, as we saw in the diagram above, autoscaling isn’t instantaneous: it takes time to provision new EC2 instances and add them to a Kubernetes cluster.

The headroom shortage and subsequent cluster autoscaling delayed provisioning of the compute requested by those services. Since there had been no change in the metrics that triggered the services to scale up, those services requested even more Pods.

Normally the impact of such runaway autoscaling would be limited by the services’ configured maximums. However, as mentioned previously the maximums for these services had been set higher than usual.

Here’s my best guess of the behavior they saw, based on the text.

The above diagram is a sort of schematic representation of my understanding of what happened. Some services running on the cluster scaled up, meaning the desired number of replicas increased for those services (that’s the red line going upwards). However, because the cluster was out of capacity, and the new nodes had not come online yet, the new pods were still in pending state waiting for the new capacity.

From the write-up, it sounds like the desired replica count continued to increase during this period. When configuring an autoscaling policy, you always configure a maximum number of replicas constant. This max replicas constant for the worker pool in question had previously been set to a high value when the pool was first created.

To ensure sufficient capacity for both pools, we initially configured each with the same high maxReplica count as the original shared pool, with the intention of reviewing and adjusting the limits for both pools downwards at a later date…

Normally the impact of such runaway autoscaling would be limited by the services’ configured maximums. However, as mentioned previously the maximums for these services had been set higher than usual.

Eventually, the new cluster nodes came online, and the new pods came online. This led to a large number of new pods coming online in the cluster.

(I can’t tell from the text whether the worker pool actually scaled all of the way up to max replicas or not, though).

Après autoscaling, le déluge

Buildkite services running in one of our production Kubernetes clusters depend on an internal DNS service CoreDNS to locate databases, queues, and other application components.

Once the new EKS nodes come online, the worker pool rapidly scales up. That’s not a problem for the worker pool itself. But it turns out that this rapid introduction of new compute resources into the cluster is a problem for a different service: CoreDNS.

The application deployment and runaway autoscaling combined to trigger an unusually high rate of change to applications, network endpoints, and cluster nodes.

From the write-up, it sounds like CoreDNS is the service discovery mechanism used by Buildkite. A service discovery system needs to keep track of the various resources in your system. The more churn you have in these dynamic resources, the greater the load is going to be on your service discovery system.

In this case, the step change in new cluster nodes and new application pods coming online put too much pressure on their CoreDNS service, and it fell over.

Once CoreDNS went unhealthy, the overall system was in real trouble:

CoreDNS unavailability caused failures across APIs, job dispatch, and notifications for all customers.

Like in the GitHub incident, the overloaded service wasn’t able to recover on its own. To remediate, the responders reduced the load by stopping new deploys, and then gave it more resources.

Retries and delayed work increased the load during recovery…

How we responded

We paused further application deployments, deployed more CoreDNS service replicas, raised the memory available to each CoreDNS replica, and expanded the node pool available to run them. The new set of CoreDNS Pods came into service by 23:12 UTC. After that, DNS errors fell rapidly. Customer-facing services processed the accumulated backlog and recovered fully by 23:16 UTC.

Actions taken for safety end up increasing risk

We often make operational decisions where we believe we are reducing risk. As mentioned earlier, it’s notable that the max replicas threshold was intentionally set high to reduce the risk of running out of capacity.

To ensure sufficient capacity for both pools, we initially configured each with the same high maxReplica count as the original shared pool, with the intention of reviewing and adjusting the limits for both pools downwards at a later date.

Pick a max replicas that’s too low, and your service can become overloaded: I’ve seen it many times. But picking a max replicas that’s too high created a risk that (I can only assume) nobody even conceived of in the moment. That’s the nature of the complex systems that we work with.

Control loops will drive you loopy

In one sense, this is a story of too much autoscaling, as it was a sudden, dramatic increase in scaling that led to CoreDNS being overwhlemed. But it’s also a story of not enough autoscaling: CoreDNS was not configured to autoscale at all.

The cluster’s CoreDNS service was running at a fixed size and did not automatically scale with the size or rate of change of the cluster.

An autoscaler is an example of a control loop in your system. In this particular incident, two control loops (the horizontal pod autoscaler and the cluster autoscaler) interacted in an unexpected way that led to an increase in CoreDNS, which was missing a control loop to ensure it had sufficient resources. In general, it’s difficult to reason about the behavior of interacting control loops. It’s these sorts of complex interactions that can make incidents hard to deal with.

If you’ve ever taken a course in control systems, you’ll know that stability is one of the primary concerns with building a control system, and that delays in your system can impact its stability. This was an interesting case of that where there were delays introduced because of the time required for the cluster to scale up. And, indeed, that led to that worker pool getting overscaled.

Rapid scale up as an increased load scenario

Because your application behaves differently on startup than when it’s fully warmed, if you’re in a situation when many pods are starting up, that’s a different mode of operation than your system normally deals with. Here we saw CoreDNS be the system that got overwhelmed. But any system that gets more interaction during startup than steady-state is at potential risk of being knocked over in a scenario like this.

Most load tests I’ve seen involved sending a significant amount of additional traffic to a system that’s in steady state. I’ve never seen anybody run a load test where they just dramatically scaled up a single service and watched what happened. But I bet you’d find some interesting failure modes that way that you wouldn’t find with traditional load testing.

Saturation, migration, networking, misbehaving reliability subsystems

The Buildkite incident checks off multiple boxes in my list of omnipresent availability risks in cloud software. Here, working involved in a migration (from ECS to EKS), and systems intended to improve reliability (autoscalers) resulted in saturation (CoreDNS), which impacted a networking system (once again, CoreDNS).

So many contributors

There’s a lot more detail in the write-up than I’ve covered in this post. I tried to collect the various contributors that they explicitly mention in the write-up. Here are the ones I spotted:

1. migrating from ECS to EKS
2. The EKS cluster was increasing in size during the course of the migration
3. An application deploy, which spiked the number of application pods
4. all remaining headroom in the EKS cluster being consumed by the application deploy
5. EKS cluster had to autoscale in order to make the new capacity available, this took time
6. background workers were attempting to scale out at the same time
7. metrics used for scaling background workers remaining unchanged while cluster is waiting for new capacity to come online
8. time-sensitive notification jobs had been moved from general-purpose pool to low-latency worker pool
9. high maxReplica count set on the low-latency worker pool
10. CoreDNS used for service discovery
11. large number of pods coming online led to high rate of change to apps, network endpoints, cluster nodes
12. CoreDNS runs at fixed size (does not autoscale)
13. defect in monitoring query which masked increases in CoreDNS query response times
14. Recent performance gains in application hid client-side latency increases
15. CoreDNS pods exceeded allowed memory limits, were restarted by kubernetes
16. continued demand for DNS kept load high on CoreDNS
17. DNS retries kept load high on CoreDNS

Finally, I commend Buildkite for providing so many technical details about the incident. It increases my esteem for their engineering organization.