The other day, I wrote about a GitHub Actions incident. But GitHub Actions isn’t the only CI-as-a-service out there. Buildkite, which is another CI company, recently released a public write-up of an incident they experienced last week. It’s a great one: their incident report contains a lot more detail than GitHub’s. Here I’ll discuss my own observations.

The failure mode

The write-up describes what I found to be a fascinating failure mode. Here I’m going to do my best to explain it in my own words.

A bit about Kubernetes

First, some Kubernetes terminology to get us all on the same page. When you’re using Kubernetes to run your services and jobs, we say that you are running on a Kubernetes cluster. A cluster is made up of multiple nodes. If you’re running on EKS (Amazon’s Kubernetes offering), I believe those nodes are going to be EC2 instances.

Each node exists to run pods, which are the compute workloads that you run on Kubernetes. If you’ve been a service owner for a service that runs on Kubernetes, then you’re almost certainly familiar with the notion of pods, because that’s what’s exposed to you. But, unless you’ve worked in infrastructure, you’re likely not familiar with nodes and clusters, because those implementation details are deliberately hidden from the service owners.

Here’s a diagram that shows a four-node Kubernetes cluster. Each black rectangle depicts a pod that is running within that node. A pod is a set of containers that all get scheduled together (typically an application container and related sidecars), but that particular detail isn’t relevant here.

I’ve also labeled the blank part of one of the nodes as headroom. That’s not explicitly a Kubernetes concept. Rather, it’s a general term that here means “the amount of additional compute jobs that could still be run on this node.”

A few brief words on autoscaling

Let’s take the service owner’s perspective again. You own a service that runs on Kubernetes. That service is deployed as a group of pods, what Kubernetes calls a replicaset. Note that each pod is identical, so the only reason to have multiple pods is because you need additional compute resources. The amount of compute resources that your service needs can vary over time. For example, if you’re driven by requests, then the volume of requests can vary. Or, if your service consumes from a job queue, the number of pending jobs can vary at any one time.

Kubernetes deals with workloads whose resource needs vary with time by supporting autoscaling. The system I’m most familiar with is the HorizontalPodAutoscaler, which will increase or decrease the number of size of a replicaset depending on the load.

---

A brief sidebar about scaling terminology. There’s a distinction between horizontal scaling and vertical scaling. If you’re adding more servers, you’re horizontally scaling, whereas if you’re increasing the size of your servers, you’re vertically scaling. Technically, it’s more accurate to use the term scale out to refer to horizontal scaling, and scale up to refer to vertical scaling. and, indeed, the Buildkite write-up uses scale out. But I’m used to saying scale up for horizontal scaling as well, because that’s the kind of scaling I deal with more often. In this post, I use scale up and scale out interchangeably.

---

Here’s an example of a graph of a deployment that is undergoing autoscaling.

The red line indicates the desired replica count. That’s the number of pods that the autoscaler has determined should be running. At the beginning of this graph, the desired number of pods is 6. Later on, the autoscaler adjusts the desired replica count to 8.

The green indicates the number of pods that are in a running state. Note at the beginning of the graph that there are 6 pods in running state, and at the end there are 8 pods. Here’s how the Kubernetes docs describe the running state:

The Pod has been bound to a node, and all of the containers have been created. At least one container is still running, or is in the process of starting or restarting. – Kubernetes pod lifecycle docs

The gray is the interesting bit. That shows the pods that are in the pending state. Here’s how the Kubernetes docs describe the pending state:

The Pod has been accepted by the Kubernetes cluster, but one or more of the containers has not been set up and made ready to run. This includes time a Pod spends waiting to be scheduled as well as the time spent downloading container images over the network. – Kubernetes pod lifecycle docs

After the autoscaler increases the desired replica count to 8, another Kubernetes controller sees the difference between current replica count (6) and desired (8), and spins up two new pods. But those pods don’t immediately start running. Each new pod has to be scheduled onto a node in the cluster that has headroom. Until that happens, the pod is in the pending state.

A deployment consumes all of the headroom, workers try to scale up

Let’s turn back to the write-up:

At 22:44 UTC, an application deploy created a surge in application Pod volume… This surge consumed the available headroom on already-deployed Nodes.

When you deploy a new version of your software onto a Kubernetes cluster, the cluster spins up new pods with containers that are running the new version. In this case, these pods new consumed all of the available resources on all of the nodes of the cluster.

In Buildkite’s case, in addition to an application deployment eating up all the headroom, there was a replicaset of background workers that was being scaled up by the pod autoscaler at the same time. From the write-up:

At 22:44 UTC, an application deploy created a surge in application Pod volume… This surge consumed the available headroom on already-deployed Nodes, which limited applications’ capacity to autoscale promptly. Some background workers (including the aforementioned notification workers) were also attempting to scale out at this time. 

So, we have one application (background workers) trying to increase its replica count, at the same time as another application has eaten up all of the available capacity.

You can scale up your k8s cluster too, but you’ll need to wait

At 22:44 UTC, an application deploy created a surge in application Pod volume, which caused our EKS cluster to scale out. (emphasis added)

Just like a service owner can configure autoscaling for their service, a Kubernetes administrator can configure autoscaling for their cluster: when it starts to run out of headroom, the cluster autoscaler can automatically request new EC2 instances from AWS. Instead of using the horizontal pod autoscaler (which is for scaling pods), you use a cluster autoscaler, like Karpenter, which happens to have been written by AWS. But the general concept of autoscaling is the same here.

However, as we saw in the diagram above, autoscaling isn’t instantaneous: it takes time to provision new EC2 instances and add them to a Kubernetes cluster.

The headroom shortage and subsequent cluster autoscaling delayed provisioning of the compute requested by those services. Since there had been no change in the metrics that triggered the services to scale up, those services requested even more Pods.

Normally the impact of such runaway autoscaling would be limited by the services’ configured maximums. However, as mentioned previously the maximums for these services had been set higher than usual.

Here’s my best guess of the behavior they saw, based on the text.

The above diagram is a sort of schematic representation of my understanding of what happened. Some services running on the cluster scaled up, meaning the desired number of replicas increased for those services (that’s the red line going upwards). However, because the cluster was out of capacity, and the new nodes had not come online yet, the new pods were still in pending state waiting for the new capacity.

From the write-up, it sounds like the desired replica count continued to increase during this period. When configuring an autoscaling policy, you always configure a maximum number of replicas constant. This max replicas constant for the worker pool in question had previously been set to a high value when the pool was first created.

To ensure sufficient capacity for both pools, we initially configured each with the same high maxReplica count as the original shared pool, with the intention of reviewing and adjusting the limits for both pools downwards at a later date…

Normally the impact of such runaway autoscaling would be limited by the services’ configured maximums. However, as mentioned previously the maximums for these services had been set higher than usual.

Eventually, the new cluster nodes came online, and the new pods came online. This led to a large number of new pods coming online in the cluster.

(I can’t tell from the text whether the worker pool actually scaled all of the way up to max replicas or not, though).

Après autoscaling, le déluge

Buildkite services running in one of our production Kubernetes clusters depend on an internal DNS service CoreDNS to locate databases, queues, and other application components.

Once the new EKS nodes come online, the worker pool rapidly scales up. That’s not a problem for the worker pool itself. But it turns out that this rapid introduction of new compute resources into the cluster is a problem for a different service: CoreDNS.

The application deployment and runaway autoscaling combined to trigger an unusually high rate of change to applications, network endpoints, and cluster nodes.

From the write-up, it sounds like CoreDNS is the service discovery mechanism used by Buildkite. A service discovery system needs to keep track of the various resources in your system. The more churn you have in these dynamic resources, the greater the load is going to be on your service discovery system.

In this case, the step change in new cluster nodes and new application pods coming online put too much pressure on their CoreDNS service, and it fell over.

Once CoreDNS went unhealthy, the overall system was in real trouble:

CoreDNS unavailability caused failures across APIs, job dispatch, and notifications for all customers.

Like in the GitHub incident, the overloaded service wasn’t able to recover on its own. To remediate, the responders reduced the load by stopping new deploys, and then gave it more resources.

Retries and delayed work increased the load during recovery…

How we responded

We paused further application deployments, deployed more CoreDNS service replicas, raised the memory available to each CoreDNS replica, and expanded the node pool available to run them. The new set of CoreDNS Pods came into service by 23:12 UTC. After that, DNS errors fell rapidly. Customer-facing services processed the accumulated backlog and recovered fully by 23:16 UTC.

Actions taken for safety end up increasing risk

We often make operational decisions where we believe we are reducing risk. As mentioned earlier, it’s notable that the max replicas threshold was intentionally set high to reduce the risk of running out of capacity.

To ensure sufficient capacity for both pools, we initially configured each with the same high maxReplica count as the original shared pool, with the intention of reviewing and adjusting the limits for both pools downwards at a later date.

Pick a max replicas that’s too low, and your service can become overloaded: I’ve seen it many times. But picking a max replicas that’s too high created a risk that (I can only assume) nobody even conceived of in the moment. That’s the nature of the complex systems that we work with.

Control loops will drive you loopy

In one sense, this is a story of too much autoscaling, as it was a sudden, dramatic increase in scaling that led to CoreDNS being overwhlemed. But it’s also a story of not enough autoscaling: CoreDNS was not configured to autoscale at all.

The cluster’s CoreDNS service was running at a fixed size and did not automatically scale with the size or rate of change of the cluster.

An autoscaler is an example of a control loop in your system. In this particular incident, two control loops (the horizontal pod autoscaler and the cluster autoscaler) interacted in an unexpected way that led to an increase in CoreDNS, which was missing a control loop to ensure it had sufficient resources. In general, it’s difficult to reason about the behavior of interacting control loops. It’s these sorts of complex interactions that can make incidents hard to deal with.

If you’ve ever taken a course in control systems, you’ll know that stability is one of the primary concerns with building a control system, and that delays in your system can impact its stability. This was an interesting case of that where there were delays introduced because of the time required for the cluster to scale up. And, indeed, that led to that worker pool getting overscaled.

Rapid scale up as an increased load scenario

Because your application behaves differently on startup than when it’s fully warmed, if you’re in a situation when many pods are starting up, that’s a different mode of operation than your system normally deals with. Here we saw CoreDNS be the system that got overwhelmed. But any system that gets more interaction during startup than steady-state is at potential risk of being knocked over in a scenario like this.

Most load tests I’ve seen involved sending a significant amount of additional traffic to a system that’s in steady state. I’ve never seen anybody run a load test where they just dramatically scaled up a single service and watched what happened. But I bet you’d find some interesting failure modes that way that you wouldn’t find with traditional load testing.

Saturation, migration, networking, misbehaving reliability subsystems

The Buildkite incident checks off multiple boxes in my list of omnipresent availability risks in cloud software. Here, working involved in a migration (from ECS to EKS), and systems intended to improve reliability (autoscalers) resulted in saturation (CoreDNS), which impacted a networking system (once again, CoreDNS).

So many contributors

There’s a lot more detail in the write-up than I’ve covered in this post. I tried to collect the various contributors that they explicitly mention in the write-up. Here are the ones I spotted:

1. migrating from ECS to EKS
2. The EKS cluster was increasing in size during the course of the migration
3. An application deploy, which spiked the number of application pods
4. all remaining headroom in the EKS cluster being consumed by the application deploy
5. EKS cluster had to autoscale in order to make the new capacity available, this took time
6. background workers were attempting to scale out at the same time
7. metrics used for scaling background workers remaining unchanged while cluster is waiting for new capacity to come online
8. time-sensitive notification jobs had been moved from general-purpose pool to low-latency worker pool
9. high maxReplica count set on the low-latency worker pool
10. CoreDNS used for service discovery
11. large number of pods coming online led to high rate of change to apps, network endpoints, cluster nodes
12. CoreDNS runs at fixed size (does not autoscale)
13. defect in monitoring query which masked increases in CoreDNS query response times
14. Recent performance gains in application hid client-side latency increases
15. CoreDNS pods exceeded allowed memory limits, were restarted by kubernetes
16. continued demand for DNS kept load high on CoreDNS
17. DNS retries kept load high on CoreDNS

Finally, I commend Buildkite for providing so many technical details about the incident. It increases my esteem for their engineering organization.

It wasn’t even a week ago when I wrote about a major GitHub incident. Yesterday, they had another big incident, which lasted almost eight hours. There’s a public write-up already posted, which is surprisingly quick. While I’m personally very impatient to read these, I also know that it takes time to collect and synthesize the information you need to do a good job with them. I wish they had posted this as a preliminary write-up and then done a more detailed write-up in a couple of weeks. That being said, let’s look at the write-up!

Saturation strikes yet again

The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic.

The failure mode is yet another example of saturation, a topic I’ve written about again and again on this blog. Heck, I even gave a talk on saturation a month ago.

Here’s the full paragraph on the failure mode:

The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers.

Based on this, it sounds like the failure cascade looked like:

increase in external traffic → istio sidecars saturate (concurrency limits) → HAProxy nodes saturate (flow limits) → authentication requests fail

An increase in load on the system saturated one of the components (istio sidecar), and that propagated to another component (HAProxy), whose saturation broke the auth flow.

I wish they had included an architectural diagram here, that showed the relationship between the load balancers, the service that whose Istio sidecar pod saturated, the gateway, and the services that handle auth requests. Also, the wording gives the impression that only a single sidecar pod that saturated (an Istio sidecar pod), which would be surprising, but I’m also not confident that this is what the authors intended.

Diagnostic details: missing in action

The write-up doesn’t talk about the diagnostic work of the incident responders at all, which is a shame. I can’t tell from this write-up how difficult it was for them to figure out what was happening. There were auth failures, but it doesn’t sound like there was an increase in auth traffic per se, nor was the problem caused by recent changes to the auth system, which is where I would think to look first.

As somebody who was watching the updates to the status page as the incident was happening, I was struck by how they updates alternated between “we have identified the problem” and “we are experiencing issues:

I can imagine how frustrating it must have been for the responders to think they had found and fixed the problem, only to continue to see impact.

Retries made things worse

Retries are one of the tools in our toolbox to improve availability. And, usually, retries do improve availability! But retry logic also adds complexity to a system, and adding complexity to a system can introduce new failure modes. In this incident, retries hurt rather than helped, by increasing the load on an overloaded system.

The problem was worsened by optimistic retry logic which overloaded internal load balancers.

…

Residual Copilot authentication failures continued because client retry behavior amplified load: a failed token operation could generate many extra requests and enter a retry loop.

This is a great example of unexpected behavior of a subsystem whose primary purpose was to improve reliability from my conjecture on why reliable systems fail.

The Copilot Token Service sees 10X traffic

Note that there were two independent retry behaviors mentioned in the previous section:

1. optimistic retry logic against the load balancers
2. client retry logic against the Copilot Token Service

It turns out that the client retry logic was due to a previously undiscovered bug in Visual Studio Code(!), which led to one particular service (Copilot Token Service) taking longer to recover:

Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.

…

Residual Copilot authentication failures continued because client retry behavior amplified load: a failed token operation could generate many extra requests and enter a retry loop. Copilot Token Service traffic increased from a normal 7–9K RPS to 70–100K RPS.

There’s no way you’re pushing out a VS Code bugfix to mitigate an incident! You’ve got to mitigate that on the server side, which is what the responders did, which brings us to the next section.

Mitigating the incident: multiple strategies

While the write-up doesn’t discuss diagnostic work, it does mention multiple mitigations that the responders undertook during the incident:

• shifting traffic from the Central US region to the Northern Virginia region
• paused HAProxy on the four saturated nodes
• changed gateway retry logic (via PR)
• blocked inbound Copilot Token Service token requests at the load balancer (returned 403s)
• gradually ramping up blocked traffic

As responders, we are always limited in our ability to intervene based on the tools that we have at our immediate disposal. It’s incredibly useful to be able to do things like selectively block traffic, or dynamically change or even disable a reliability-related subsystem. Think about how difficult it would be to block specific types of requests during an incident in your organization, and to ramp that traffic back up slowly after the system recovers. Note how the responders had to use a pull request to change the behavior of the gateway retry logic. I wonder if the failure mode made this more difficult to carry out, but the write-up doesn’t say.

“Never again” means never preparing for a novel incident

The writeup ends, as most writeups do, with some action items intended to prevent recurrence.

To prevent recurrence, our follow-up actions include:

• Correcting autoscaling policies to account for service-mesh sidecar concurrency and capacity.
• Auditing Istio request, concurrency, and scaling limits across affected services.
• Reviewing retry limits and backoff behavior across gateways and clients.
• Addressing the VS Code retry behavior that amplified Copilot token traffic.
• Improving load-balancer capacity monitoring and regional failover safeguards.

My eternal lament is that people spend too much of their focus on preventing the last incident from recurring. It’s not that I’m opposed to preventative work. It’s that I also want us to spend time on getting better at dealing with novel incidents. Engineering cycles are a finite resource, and every cycle spent on prevention is a cycle not spent on improving our ability to respond effectively to new incidents. And I promise you, you are going to face novel incidents in the future.

After all, I don’t think GitHub customers who experienced this outage take much solace in knowing that it was a different failure mode from the previous incident.

The folks at Microsoft Azure recently wrote up a post incident review for a networking issue in their West U.S region. From the included timeline, it looks like the impact was on the order of five hours. It’s a pretty short write-up, but let’s take a look at the contributors.

On 23 July 2026, a break-fix repair was initiated on an optical device to address a network reliability risk.

The first contributor mentioned in the write-up was work that was done to repair a device in their networking stack. Here I can’t help but think of the first bullet in my conjecture on why reliable systems fail. They made a change to the system in order to fix an ongoing problem, and due to a set of circumstances, things got worse rather than better.

A defect in our blast radius analysis system incorrectly expanded the scope of the repair event to include all optical devices egressing a specific datacenter. 

The second contributor mentioned was a (presumably) latent defect in their system. Note the irony of the failure mode here: I suspect this blast radius analysis system usually contributes to reliability, but in this case it hurt reliability by increasing the blast radius.

The safety validation step, which is designed to confirm that at least one of the two redundant datacenter paths remains available, ran but incorrectly concluded the operation was safe.

The third contributor mentioned was a safety check (good!) that passed even though the action was unsafe (bad!).

The checks validated each device individually rather than evaluating the aggregate effect of isolating all devices at once, a scenario that was not accounted for because the system was never designed to process a full datacenter’s worth of devices in a single request.

The reason it failed was due to an interaction with the second contributor: the blast radius being all of the optical devices egressing the datacenter. The designers never envisioned that the check would have to handle the sort of scenario that occurred as a result of the blast radius analysis system defect.

As a result, routes were withdrawn from multiple devices simultaneously, disrupting connectivity between the datacenter and the WAN – therefore impacting traffic entering or leaving the West US region.

It sounds like this change effectively disconnected the West US datacenter from the internet.

Once the route withdrawals took effect at 14:44 UTC, physical links and routing adjacencies continued to appear healthy, which initially masked the correlation between the break-fix activity and the connectivity disruption

Here we have our fourth contributor: the operators were receiving misleading signals from the system. The links and routes looked healthy, even though connectivity was broken.

The impact presented as a WAN routing anomaly, as third-party networks could not reach Azure in the region, rather than as a datacenter connectivity failure.

Our fifth contributor is another flavor of misleading signals. The symptoms presented as a routing issue between Azure and third-parties.

Although all physical work in the region was stopped, our engineers could not correlate to this recent change because the preparation activities in advance of the break-fix did not succeed, so the physical layer and traffic appeared healthy.

This is the sixth contributor mentioned in the writeup. The writing is a little oblique here, but I think what they are saying is that the repair event did not show up in their event log because the repair event didn’t actually complete. It sounds like the preparation activities were the ones that triggered the incident. But, because the repair event didn’t actually happen, the operators looking for events that correlate in time with the onset of the incident didn’t see the triggering event because it didn’t show up in the log of events. That’s my best guess, anyways.

Our automated recovery and rollback system detected the device failures, and attempted multiple retries to restore the affected devices. However, because that system depended on the same datacenter connectivity that had been disrupted, its automated rollback attempts were unsuccessful.

This is the seventh and final contributor mentioned. Azure has an automated recovery and rollback system (good!), but the failure mode in this case prevented automated rollback from succeeding (bad!).

As always, I’d love to know more about how the operators identified what the failure mode actually was, and how they traced it back to the optical device repair work.