Last week, GitHub Actions experienced another incident. As is typical of a GitHub public writeup, it was published very soon after the incident, but there also aren’t isn’t a lot of detail. But let’s see what we info we can glean from it.
Database saturation
The first thing I noticed is that this incident is, once again, a saturation-related failure mode. Specifically, it was a database that was saturated due to write traffic.
This impact was triggered by saturation of writes to the database primary used by the service processing triggers for Actions workflows.
As I mentioned in my last blog post, database-related saturation issues are particularly pernicious, because they can be very difficult to recover from.
Multiple contributors, but not much detail
The write-up mentions seven separate factors that contributed to the incident.
Growing peak daily load (this increased the writes to the database)
An upstream issue in GitHub’s event processing infrastructure (?), which further increased the load
Failing over from primary to replica did not lead to full recovery (?)
Existing throttles were set ~10% too high, so they provided insufficient overload protection
some jobs remained stuck in queued/waiting state after recovery (?)
another issue that left some jobs left in a waiting-for-runner state after recovery
a bug that left that some runs showing as queued even though they had already failed
I annotated contributors with (?) where I felt the report really didn’t provide any details at all. The mention of the upstream issue references a different GitHub incident, but there are no details on that other incident page at all. It does say “A detailed root cause analysis will be shared as soon as it is available”, so perhaps we’ll get more details on this other issue in the next few days.
What I’m most curious about, though, is what happened with the database failover. All we get in the write-up is this one line:
The primary was failed over, but the system did not fully recover.
What happened here??? Did the newly promoted primary get overwhelmed the same way that the previous one did? Did something else go wrong? I wish there they went into a lot more detail on the particular failure mode.
Slowly nursing an overloaded database back to health
On the plus side, the report does have some details on how they were able to mitigate. They throttled traffic to the database until it recovered, and then ramped the traffic back up slowly enough so that they didn’t knock it over again. Here’s the actual text:
At 15:45 UTC, throttling combined with service restarts recovered the service’s core health. Those throttles were gradually raised between 15:54 and 17:22 to restore full webhook processing for Actions runs. This ramp was deliberately slow to ensure we did not re-overwhelm the system given our original throttling was now known to be incorrectly set. The queue of webhook events was fully burned down at 17:40 UTC.
Two things I want to note about this. First of all, this sort of recovery approach is something you are going to need to do some day when your own database gets overloaded (and, believe me, it’s going to happen). If you’re prepared for this, you’ll have access to a throttle knob that the responders can manually control so they can cut the traffic and then increase it. It’s not something you want to have to build during an incident.
The second thing to note is that throttling means that you are deliberately cutting off access to the database for your users in order to bring it back up. This means that you will be temporarily increasing user pain in order to recover the system. This sucks, but it’s a decision you sometimes have to make during an incident: that you actually have to deliberately make the system behave worse from the user’s perspective in order to get it back into a healthy state. Now, if you have the ability of doing QoS-style throttling where you can selectively block the less important requests, then you might be able to reduce the amount of pain. But, once again, that’s something you need to have built into your system in advance.
Irony: fix was already in-flight when the incident struck
This line in the write-up broke my heart a little (emphasis mine):
Several changes to improve the general scalability of this part of Actions were already complete and deploying to production. Rollout of those changes will be complete within the next 24 hours.
They were already working on reducing the likelihood of an incident like this, but it bit them before they could finish rolling out the improvements. That’s really just bad luck.
Another GitHub incident, another limit hit
Finally, we continue to see GitHub hitting one limit after another as they experience continued growth. There are just so many different limits within a system like this. I won’t be surprised if I’m soon reading up on yet another saturation-related GitHub incident.
I’m using this post to gather together some common threads I’ve noticed after reading write-ups of major cloud software incidents. By cloud software, I’m referring to software-as-a-service (do people even say that anymore? in the cloud. This doesn’t just apply to cloud providers, although it does apply to them as well.
Here’s an outline of the topics in this post:
problem areas
saturation
example: databases
networking (traffic routing failure)
example: DNS
security (deny valid access)
example: SSL certificates
essential non-standard changes
mitigating an operational issue
migration
essential increase in essential complexity
reliability subsystem
migration
I think of all of these as omnipresent availability risks: I think these are fundamentally unavoidable, and will be contributing to software incidents until the end of time; or, at the very least, until the end of my own career in software.
There are three general areas that most major incidents seem to fall into: saturation, networking, and security. So, let’s start with those.
Saturation
Saturation is probably the topic I talk about most frequently, both on this blog and elsewhere (e.g.,: the saturation post I wrote for the Resilience in Software Foundation and my saturation talk at the Software Should Work conference). The system becomes saturated when it reaches a limit. That’s a pretty generic description, but there are many limits!
Databases
Many major incidents involve some system component becoming saturated in one form or another. I personally worry about database saturation the most. That’s because it’s difficult to recover from an overloaded production database. In addition, because database systems are such complex beasts, it can be quite difficult to even determine what the specific performance issue actually is. This is why having in-house database operational expertise is critical.
Saturation is an omnipresent risk because the finite nature of resources is a hard constraint in the world that we live in. Eventually, some resource in your system is going to run out.
While I prattle on endlessly about saturation, not every major incident involves saturation. You can encounter scenarios where all of your internal subsystems are reporting healthy, but from your customer’s point of view, your site is down: they can’t use it. One way this can happen is if your users can’t even reach your site, and that’s where the networking problem area comes in.
A networking problem can lead to packets being misrouted. These requests might be black-holed (i.e., silently dropped), or they might be incorrectly routed to a service that doesn’t have the capacity to respond to all of these requests, in which case you’ve got both a network routing issue and a saturation issue.
A visual depiction of an actual black hole. Image source: NASA
DNS
DNS issues are an example of this kind of network-related failure mode. There’s no way those packets are going to make it to their destination if the client can’t even determine which IP address to send them to. And when DNS breaks, that’s what happens.
I bring DNS up because it’s bitten folks enough times that there’s a famous haiku:
More generally, networking is an omnipresent risk because cloud software is inherently distributed, so networking is always a critical service. Now, I don’t work in networking, but from the outside, networking just feels like a dangerous domain to do operational stuff in. The blast radius of a networking issue can be very large. And, because network behavior is inherently distributed, reasoning about the behavior of operational changes is just inherently difficult. Honestly, that’s probably why I don’t work in networking.
And, so, I predict we’ll continue to see networking issues contribute to large-scale incidents.
There’s a fundamental tradeoff between availability and security: availability is about ensuring that the good people can access the system. Security is about ensuring that the bad people cannot access the system. This means that there’s always a risk that a security system designed to prevent bad actors from accessing the system can lead to good actors also being blocked. Consider this scenario: there’s an internal security subsystem that goes unhealthy (possibly due to saturation). Is your policy to fail closed or fail open in the event that this subsystem is erroring? Answering that requires making an availability-security tradeoff.
SSL certificate expiration
Another example of this failure mode, which keeps biting our industry again and again, is SSL certificate expiration. Here you have the behavior of a security system that is preventing legitimate access because the cert wasn’t renewed.
Even the mighty Google encounters SSL certificate expirations. This is from the Bazel incident
And so, my claim is availability incidents that involve security subsystems will continue to be a thing forever.
Your system is constantly undergoing change. Heck, if you stopped making changes, the system would eventually stop working properly. Now, there are some changes that your org does very frequently. Hopefully, you’re deploying often, flipping feature flags a lot, and so on. But there are other changes that your org has less experience with, because they happen much less often. That means that there hasn’t been as much investment in tooling to support these sorts of changes, and it means that the people making these changes don’t have the same level of expertise as they do with the more common changes. That makes these sorts of changes more dangerous: less mature tooling and less experienced humans.
Mitigating an operational issue
A few years ago, I wrote a post titled a conjecture on why reliable systems fail where I speculated on two common contributors to major incidents. One of those contributors was a manual intervention that was intended to mitigate a minor incident. Now, it may be that you frequently have to do manual interventions to mitigate system issues, in which case you’ll have a lot of experience with those sorts of interventions. But you’ll also be more motivated to put in the engineering effort to automate away those sorts of common issues.
It’s exactly the uncommon issues that require a human operator to intervene to mitigate that are dangerous, because they are uncommon. But they’re essential: there’s a problem in the system, and you need to fix it! But because all practitioner actions are gambles, the manual mitigation carries risk that you could make the problem even worse. And, eventually, this will happen to you.
If you’re at a tech company, unless it’s a start-up, you’ll be dealing with migrations, as old tech gets replaced by newer tech that is better suited to the problems that your org is currently facing. While migrations as a general category are extremely common, each migration is itself a snowflake. This means that the specific details of the migration work is an uncommon change. The work of migration involves making a kind of change to your system that you haven’t made before.
To make things worse, one of the dangers of migration is that, as you go along, you start to build confidence that your changes are safe, but there are actually hidden dangers lurking in the system for the next migration. The confidence in the safety of the work exceeds the actual safety. I mean, you made n-1 changes as part of the migration, and none of those changes had negative consequences. It’s natural to assume that the same outcome will occur with the nth change.
The late American computer scientist Fred Brooks wrote a famous software engineering essay titled No Silver Bullet where he drew a distinction between accidental complexity and essential complexity. The general idea was that there was some amount of complexity in a software system that didn’t need to be there (accidental complexity) and some amount that was just inherent to the nature of the problem space and solution space and so could not be removed (essential complexity).
Reliability subsystem
We’ve developed multiple techniques to improve the reliability of software systems, including retries, concurrency limiting, autoscaling, automated failover, circuit breakers, health checks, canaries, outlier detection, the list goes on and on. There’s one thing that all of these techniques have in common: they increase the complexity of the overall system! And they do this because they have to increase complexity in order to do their job. This is a consequence of Ashby’s Law, which states that if you want to build a control system that handles more scenarios, you have to increase the complexity of the controller itself.
This means that reliability subsystems result in a complexity trade-off. On the one hand, our system can now automatically recover from failure modes that previously required manual intervention. On the other hand, as we all know, increase in complexity is itself dangerous because it can introduce entirely new failure modes that weren’t there before.
Going back to my conjecture blog post, the second contributor I posited was: unexpected behavior of a subsystem whose primary purpose was to improve reliability. And this is exactly why. Adding reliability subsystems improves the robustness of our system, but it adds essential complexity to our system, which can lead to novel incidents.
Like all engineers, I’m a big fan of giving the answer “it depends” if somebody asks me a question about whether they should do X or Y. However, if someone came up to me and said, “Lorin, I’m preparing to do a migration at my company, and I’m trying to decide whether to do a big-bang migration or an incremental one”, then I would almost certainly say, “For the love of God, please do an incremental migration!”. Sometimes big-bang migrations are unavoidable, but when given a choice, I’m going to go for the incremental migration as the safer option.
However, when you do an incremental migration, it means that you need to simultaneously support the old system and the new system at the same time while you’re doing the migration. This means that even if the new system yields a net decrease in overall complexity over the old system, while the migration is happening, you’re going to see an increase in system complexity. And that means that you’ll see incidents arise as a byproduct of this increased complexity.
Incidents are inevitable, so you’d better be ready
To reiterate, I think all of the risks mentioned here are omnipresent: they are fundamental to the nature of cloud software. I don’t think that any of these risks can be eliminated. That’s why I believe so strongly in the value of getting better at incident response. Because, if you prepare, you can get better at dealing with problems that arise as a result of these risks.
Recently, two AI-related pieces of content caught my attention. The first was the blog post On-Call is Now Theatre by Boris Tane. He argues that AI agents are now capable of doing the majority of on-call work that is currently being done by humans, and that we should have AI agents act as first responders. Only when an AI agent isn’t capable of remediating the problem should a human actually be brought in, and the agent should be the one to page the human. As he puts it:
We need software that watches itself, triages its own alerts, investigates its own incidents, fixes what it can, and escalates to a human only when it hits something genuinely novel, with the evidence already assembled.
Put your AI agents in the worst on-call rotation imaginable, then give them a tool to page a human. Developers stop being the first responder, and step in only when an agent genuinely cannot figure something out.
Tane doesn’t think that companies will really start putting AI agents on-call (“Most teams won’t do this”), but he believes it should happen, and he’s started a company based on this premise.
I like to think of putting AI agents on-call as equivalent to using AI agents to implement control system automation. Because, after all, that’s what operations work is: it’s taking control actions to keep the system in a healthy state.
Now, AI agents are extremely complex software systems. I’d argue that they are the most complex software systems that we humans have ever built. That complexity is both good and bad. Ashby’s Law teaches us that the larger the set of system states that you want your control system to be able to handle, the more complex that it needs to be. It’s this complexity that makes it possible, in principle, to apply AI agents to solve a generic control problem like this.
On the other hand, the more complex a system becomes, the more difficult it is for a human to reason about the system’s behavior. That’s fine when the system is healthy, but if your now-even-more-complex system gets into a state that the automation can’t handle, that can make the problem even worse. Indeed, it’s precisely the unexpected behavior of complex control systems that contributes to the worst complex systems failures (see also: Air France 447, Boeing 737 MAX accidents).
The talk goes into detail about the surprising behavior of AI agents that resulted in security incidents at both OpenAI and Hugging Face. Honestly, this talk feels like something out of a movie about technology run amok; the sort of thing that still feels to me like absolute science fiction.
Because this was a talk at a security conference, the speakers focused on the lessons that apply to the security community. But as a reliability type, my biggest takeaway from this talk is that autonomous LLM agents can behave in ways that humans would have never expected. While agents today can perform complex cognitive tasks, they behave differently than a human would performing that task. We’re most familiar with this when they make a different kind of mistake than a human would make. In the OpenAI-HuggingFace incident, it wasn’t so much that they made a mistake, it’s that the agents pursued their goals in ways different than a human would do. If a teammate of yours used 0-day exploits to overcome internal security protocols in order to get their work done, you’d say they were acting unreasonably. And that’s exactly the risk here.
The inevitable improvement in frontier models does not mean that the agent behavior will be easier to reason about; I actually think it’s the opposite. The agent behavior will get even more complex with the more advanced models, but that doesn’t mean it will get more human-like. Humans are very complex, but we know how to reason about human behavior; at least, we do for the people we work with. After all, an employee whose behavior was unpredictable would not last long in the organization. As these agents become even more capable, they will be akin to alien minds: intelligence, but not as we know it.
Here’s how I think things will play out. I think that some teams will do what Tane proposes and will use AI agents as first-responders to deal with operational issues. And I think that for many cases, the agents will successfully remediate issues. Of course, for the agents to actually be able to remediate, they will need to have permissions to take operational actions without human intervention.
One day, though, there will be a complex incident which the agents will not be able to handle. Tane believes that the agents will defer to the humans in this case, by paging in a person. But that’s not the scenario I worry about. The one I worry about is that the agents attempt to remediate, and their attempt makes things worse. And it’s only after these failed remediation attempts that humans enter the loop. Maybe they eventually page in a human, or maybe a human notices that something is very wrong as the agents continue to try and fail in their remediation actions. But now the humans have to make sense of the combined software-AI-agent system behavior. The original problem was already so complex that the agents couldn’t handle it, and they have now made it worse by trying to remediate. I can even imagine the humans fighting the agents who keep trying to take actions to remediate that are failing.
This is the incident that’s coming. And it’s going to be very, very difficult to handle when it happens. And I have no idea how people will respond to the role of the AI agents in the wake of this incident.
In yesterday’s post about the recent GitHub outage, there was a detail in the writeup that I didn’t say anything about: the autoscaling policy on the service with the saturated Istio sidecar.
Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits.
I suspect readers of this blog are familiar with what autoscaling is and how it works, but here’s a brief summary in case you aren’t. The amount of compute and memory resources that a service requires depends on the load that’s placed on that service. The relevant source of load here is external requests against the service, also known as traffic. The volume of traffic varies over time. For example, for a company like GitHub, my guess is that they more traffic during working hours than evening and weekends.
Given that load changes dynamically, and that the compute and memory resources a service need is a function of load, there are two general strategies. One strategy is to provision your service for peak load. The other strategy is to dynamically adjust the resources allocated to your service, based on its current load; that’s called autoscaling.
If you want your service to use autoscaling, you need to define an autoscaling policy. In particular, you need to pick which metrics you want to use that represent load, and then you need to specify how resources should be added or removed based on how that metric changes.
CPU utilization is a common metric used for autoscaling. But note that a service can become saturated even if CPU is low. For example, imagine a scenario where you use thread-per-request with a threadpool, and the latency of your downstream requests increase, and all of the threads in the pool end up blocked. Here the service is saturated, and you’d benefit from spinning up new pods, but CPU is actually low, because the threads are blocked waiting on I/O (this happened to Slack back in 2021). Now, you can add additional rules to your autoscaling policy to handle such cases (which is what Slack did, where they rapidly scaled up based on number of threads). Or you can scale based on incoming request volume instead of CPU, if your service isn’t CPU-bound.
Based on the GitHub writeup, it sounds like the autoscaling policy for the impacted service used load metrics that only took into account load on the service itself, and not on the Istio sidecar.
In general, each service behaves differently under load, which means that every autoscaling policy is effectively bespoke. This means that a team that owns a service is not only responsible for the business logic, but also for an operational control system with custom parameters, that can really only be checked via load testing. (Are you doing load testing on all your services?) The service owners are also almost certainly not autoscaling experts. And so it’s not surprising to me that a misconfigured autoscaling policy was a contributor here.
But, while I think it’s worth discussing the particular defect with this policy, since it’s good for people to be aware of the risks of autoscaling, I also think it’s too easy to fixate on it to the exclusion of other factors involved in this incident. This is what David Woods refers to as the component substitution fallacy – the idea that the way to improve reliability is to focus efforts on identifying and fixing the defective components.
While, yes, you should identify and fix the defects uncovered by an incident, you should also recognize that:
despite the presence of all of these defects, your system is not constantly failing over
This means that component defects aren’t enough to take down your system, or your system would be down right now. Don’t just look at the individual components: treat the interactions as first-class. In the GitHub outage, we see discussion of interactions between factors such as: changing traffic patterns (including scrapers), autoscaling policy, the Istio sidecar saturation, retry logic, HAProxy node saturation, and authentication traffic.
There’s also a multitude of details we don’t have because this is a rapidly disseminated public writeup, and the good stuff can only be found in the internal writeup. I speculated in this post about the relationship between service owner and autoscaling policy, but I would love to know more about the history here (did this policy predate the use of Istio sidecars, for example?). I’d also love to know more about the problematic traffic. (What kinds of requests were they? Was it a sudden increase or a gradual ramp-up? Do we know why the traffic increased?).
You can’t get answers to these sorts of questions for public incident writeups, but you can for the internal ones at your own organization. It’s up to you to ask the questions.
It wasn’t even a week ago when I wrote about a major GitHub incident. Yesterday, they had another big incident, which lasted almost eight hours. There’s a public write-up already posted, which is surprisingly quick. While I’m personally very impatient to read these, I also know that it takes time to collect and synthesize the information you need to do a good job with them. I wish they had posted this as a preliminary write-up and then done a more detailed write-up in a couple of weeks. That being said, let’s look at the write-up!
Saturation strikes yet again
The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic.
The failure mode is yet another example of saturation, a topic I’ve written about again and again on this blog. Heck, I even gave a talk on saturation a month ago.
Here’s the full paragraph on the failure mode:
The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers.
Based on this, it sounds like the failure cascade looked like:
An increase in load on the system saturated one of the components (istio sidecar), and that propagated to another component (HAProxy), whose saturation broke the auth flow.
I wish they had included an architectural diagram here, that showed the relationship between the load balancers, the service that whose Istio sidecar pod saturated, the gateway, and the services that handle auth requests. Also, the wording gives the impression that only a single sidecar pod that saturated (an Istio sidecar pod), which would be surprising, but I’m also not confident that this is what the authors intended.
Diagnostic details: missing in action
The write-up doesn’t talk about the diagnostic work of the incident responders at all, which is a shame. I can’t tell from this write-up how difficult it was for them to figure out what was happening. There were auth failures, but it doesn’t sound like there was an increase in auth traffic per se, nor was the problem caused by recent changes to the auth system, which is where I would think to look first.
As somebody who was watching the updates to the status page as the incident was happening, I was struck by how they updates alternated between “we have identified the problem” and “we are experiencing issues:
Screen shot of some of the status page updates
I can imagine how frustrating it must have been for the responders to think they had found and fixed the problem, only to continue to see impact.
Retries made things worse
Retries are one of the tools in our toolbox to improve availability. And, usually, retries do improve availability! But retry logic also adds complexity to a system, and adding complexity to a system can introduce new failure modes. In this incident, retries hurt rather than helped, by increasing the load on an overloaded system.
The problem was worsened by optimistic retry logic which overloaded internal load balancers.
…
Residual Copilot authentication failures continued because client retry behavior amplified load: a failed token operation could generate many extra requests and enter a retry loop.
Note that there were two independent retry behaviors mentioned in the previous section:
optimistic retry logic against the load balancers
client retry logic against the Copilot Token Service
It turns out that the client retry logic was due to a previously undiscovered bug in Visual Studio Code(!), which led to one particular service (Copilot Token Service) taking longer to recover:
Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.
…
Residual Copilot authentication failures continued because client retry behavior amplified load: a failed token operation could generate many extra requests and enter a retry loop. Copilot Token Service traffic increased from a normal 7–9K RPS to 70–100K RPS.
There’s no way you’re pushing out a VS Code bugfix to mitigate an incident! You’ve got to mitigate that on the server side, which is what the responders did, which brings us to the next section.
Mitigating the incident: multiple strategies
While the write-up doesn’t discuss diagnostic work, it does mention multiple mitigations that the responders undertook during the incident:
shifting traffic from the Central US region to the Northern Virginia region
paused HAProxy on the four saturated nodes
changed gateway retry logic (via PR)
blocked inbound Copilot Token Service token requests at the load balancer (returned 403s)
gradually ramping up blocked traffic
As responders, we are always limited in our ability to intervene based on the tools that we have at our immediate disposal. It’s incredibly useful to be able to do things like selectively block traffic, or dynamically change or even disable a reliability-related subsystem. Think about how difficult it would be to block specific types of requests during an incident in your organization, and to ramp that traffic back up slowly after the system recovers. Note how the responders had to use a pull request to change the behavior of the gateway retry logic. I wonder if the failure mode made this more difficult to carry out, but the write-up doesn’t say.
“Never again” means never preparing for a novel incident
The writeup ends, as most writeups do, with some action items intended to prevent recurrence.
To prevent recurrence, our follow-up actions include:
Correcting autoscaling policies to account for service-mesh sidecar concurrency and capacity.
Auditing Istio request, concurrency, and scaling limits across affected services.
Reviewing retry limits and backoff behavior across gateways and clients.
Addressing the VS Code retry behavior that amplified Copilot token traffic.
Improving load-balancer capacity monitoring and regional failover safeguards.
My eternal lament is that people spend too much of their focus on preventing the last incident from recurring. It’s not that I’m opposed to preventative work. It’s that I also want us to spend time on getting better at dealing with novel incidents. Engineering cycles are a finite resource, and every cycle spent on prevention is a cycle not spent on improving our ability to respond effectively to new incidents. And I promise you, you are going to face novel incidents in the future.
After all, I don’t think GitHub customers who experienced this outage take much solace in knowing that it was a different failure mode from the previous incident.
The folks at Microsoft Azure recently wrote up a post incident review for a networking issue in their West U.S region. From the included timeline, it looks like the impact was on the order of five hours. It’s a pretty short write-up, but let’s take a look at the contributors.
On 23 July 2026, a break-fix repair was initiated on an optical device to address a network reliability risk.
The first contributor mentioned in the write-up was work that was done to repair a device in their networking stack. Here I can’t help but think of the first bullet in my conjecture on why reliable systems fail. They made a change to the system in order to fix an ongoing problem, and due to a set of circumstances, things got worse rather than better.
A defect in our blast radius analysis system incorrectly expanded the scope of the repair event to include all optical devices egressing a specific datacenter.
The second contributor mentioned was a (presumably) latent defect in their system. Note the irony of the failure mode here: I suspect this blast radius analysis system usually contributes to reliability, but in this case it hurt reliability by increasing the blast radius.
The safety validation step, which is designed to confirm that at least one of the two redundant datacenter paths remains available, ran but incorrectly concluded the operation was safe.
The third contributor mentioned was a safety check (good!) that passed even though the action was unsafe (bad!).
The checks validated each device individually rather than evaluating the aggregate effect of isolating all devices at once, a scenario that was not accounted for because the system was never designed to process a full datacenter’s worth of devices in a single request.
The reason it failed was due to an interaction with the second contributor: the blast radius being all of the optical devices egressing the datacenter. The designers never envisioned that the check would have to handle the sort of scenario that occurred as a result of the blast radius analysis system defect.
As a result, routes were withdrawn from multiple devices simultaneously, disrupting connectivity between the datacenter and the WAN – therefore impacting traffic entering or leaving the West US region.
It sounds like this change effectively disconnected the West US datacenter from the internet.
Once the route withdrawals took effect at 14:44 UTC, physical links and routing adjacencies continued to appear healthy, which initially masked the correlation between the break-fix activity and the connectivity disruption
Here we have our fourth contributor: the operators were receiving misleading signals from the system. The links and routes looked healthy, even though connectivity was broken.
The impact presented as a WAN routing anomaly, as third-party networks could not reach Azure in the region, rather than as a datacenter connectivity failure.
Our fifth contributor is another flavor of misleading signals. The symptoms presented as a routing issue between Azure and third-parties.
Although all physical work in the region was stopped, our engineers could not correlate to this recent change because the preparation activities in advance of the break-fix did not succeed, so the physical layer and traffic appeared healthy.
This is the sixth contributor mentioned in the writeup. The writing is a little oblique here, but I think what they are saying is that the repair event did not show up in their event log because the repair event didn’t actually complete. It sounds like the preparation activities were the ones that triggered the incident. But, because the repair event didn’t actually happen, the operators looking for events that correlate in time with the onset of the incident didn’t see the triggering event because it didn’t show up in the log of events. That’s my best guess, anyways.
Our automated recovery and rollback system detected the device failures, and attempted multiple retries to restore the affected devices. However, because that system depended on the same datacenter connectivity that had been disrupted, its automated rollback attempts were unsuccessful.
This is the seventh and final contributor mentioned. Azure has an automated recovery and rollback system (good!), but the failure mode in this case prevented automated rollback from succeeding (bad!).
As always, I’d love to know more about how the operators identified what the failure mode actually was, and how they traced it back to the optical device repair work.
On August 6, 2026, GitHub had a pretty rough incident: GitHub Actions was degraded for about nine hours. GitHub posted a public incident write-up. It’s only a few paragraphs long, but there are some interesting details in here.
This was yet another incident that involved saturation. In fact, the write-up even uses the word saturated when describing what happened.
The incident was triggered by a routine deployment to an internal Actions service responsible for processing events and generating Actions jobs. The deployment exposed an existing capacity and concurrency weakness. As pods were replaced during the deployment, remaining capacity became saturated, causing services to crash and triggering a cascading impact across multiple clusters and downstream services.
We often think of deployments as risky because we are changing the code that’s running in production, and the execution of that new code could trigger a behavior change in the system that could lead to an incident. But a deployment is itself also an operational change in the behavior of the system: our system behaves differently during a deployment than it does when nothing is being deployed. Ironically, this is one of the advantages of deploying more frequently: the more often we are deploying, the more that deployment becomes a normal part of the system behavior – we get more experience with the system in deploying state.
In this particular case, running in the deploying state reduces the number of pods available for doing work, as the older pods go online. In this scenario, it sounds like the system was running close enough to the margin that the reduction in capacity due to the deploy pushed the system over the edge, leading to a cascading failure. This is what the resilience folks call a brittle collapse, which is when the system fails in a non-graceful way when it reaches saturation.
As is common when recovering an overloaded the system, they got it back to healthy by shedding load (throttling) and by increasing capacity.
These services recovered at 17:00 after expanding capacity, throttling incoming webhook-triggered work to allow the system to recover, and increasing processing capacity for the backlog of affected events.
I wish the write-up had more details on what was involved in enabling throttling and getting that additional capacity to come online. In particular, I’m curious about whether this was easy to do or difficult. But, alas, you typically don’t get those kinds of details on public writeups.
And, of course, because every incident involves multiple contributing factors, there was a previously undiscovered bug that made things worse by consuming available capacity trying to run invalid jobs:
Due to a latent bug in one of the services responsible for job assignment, runners were getting assigned jobs that were no longer valid and then getting stuck retrying those jobs, preventing them from picking up valid work.
I would love to know more details about how the heck they figured out what was going on with these stuck jobs. I can just imagine being a responder to this incident, trying to get the backlog of work processed, and discovering that there are workers are blocked trying to execute invalid jobs! How did they figure out they were stuck? How did they figure out this was because of a bug?
They deployed a change to work around this problem, but I would love to know what kind of change that was. Was it a quick workaround to get things moving again? I bet it was, but we’ll never know…
This second stage of impact was mitigated by deploying changes to prevent runners from repeatedly attempting to acquire invalid jobs. These mitigations allowed the accumulated queues to drain and Actions to recover to normal operation.
The write-up ends with the typical “here’s what we’re doing to make sure this doesn’t happen again” text, but I am heartened by the last sentence (emphasis mine):
We are also making additional improvements to reduce the risk of cascading failures and accelerate recovery during large-scale Actions disruptions.
Too often, the focus of reliability work is entirely on prevention. I’m happy to see them also focus on preparing to recover more quickly. Because, as we all know, the next big incident is always just around the corner.
As a fan of resilience engineering, I often differ with people on where we should focus our scarce engineering cycles in order to improve reliability.
I thought it would be a useful exercise to brainstorm some of the differences in focus between what I’ll call the traditional view of reliability, and the resilience engineering view.
Traditional view focuses on
Resilience engineering view focuses on
accountability
coordination
prioritization
goal conflicts
risk mitigation
risk trade-offs
better processes and conformance thereof
more expertise
quantitative
qualitative
root cause
interaction of multiple factors
action items
insight
preventing future incidents, ensuring all incidents are novel
There’s a new software reliability conference that just spun up called Software Should Work, and I had a chance to give a talk there. I took the opportunity to speak about one of my favorite topics: saturation. Here’s a recording of it.
Over the years, mathematicians, logicians and computer scientists have developed various calculi. If you have a background in computer science, you’ve likely heard of the lambda calculus, a model of computation that was developed by Alonzo Church. If databases are more your thing, then you’ve been exposed to the relational calculus without even knowing it, since SQL is based on the relational calculus. If you are into formal methods, then you’ve worked with the predicate calculus, better known as first-order logic. Finally, if you enjoy reading academic papers on programming languages, you’ve almost certainly run into the sequent calculus. However, when someone says “calculus” without modification (e.g., “I’m taking calculus next semester”), there’s no ambiguity about which calculus they are referring to: it’s always one particular calculus. Or, rather, two calculi that happen to be deeply related to each other: differential calculus and integral calculus.
Visually, you can think of differential calculus as being about calculating the slope of a function at a given point. For example, consider this graph:
You might ask, “how quickly is this curve changing when x=6?” In other words, what is the slope of this function right in a neighborhood very close to x=6?
Differential calculus enables you to compute the slope of a function at a given point
Integral calculus, on the other hand, is about the area under the graph over a particular interval. For example, you might ask “what is the area under this curve between x=2 and x=7?
Integral calculus enables you to compute the area under a function over a given interval
If you study calculus, you’ll first be taught differential calculus (sometimes referred to “Calculus 1” or “Cal 1”) and then you’ll be taught integral calculus (“Cal 2”). When you study differential calculus, you learn the rules for calculating the derivative (slope-at-a-point) of a function. And it turns out that it’s quite straightforward to calculate a derivative, no matter what type of function it is. It’s just an algorithm, which means you can easily program a computer to compute derivatives if you wanted to. (As an aside, automatically computing derivatives is a fundamental element in the process of training LLMs. If you’re curious, look up automatic differentiation).
And then, you get to Cal 2, and you learn about how to compute an integral (area-under-a-curve). You will soon discover that, unlike in Cal 1, there is no algorithm for computing the integral of an arbitrary function. Instead, what you learn is a bag of tricks on how to compute integrals for different kinds of functions. You also learn that for some functions, there’s no closed-form solution at all for the integral! As an example, consider the Gaussian function, which shows up in the normal distribution. With zero mean and unit variance, it looks like this:
The infamous bell curve
Asking students to compute the derivative of this function would be a perfectly reasonable question on a Cal 1 final exam, the answer looks like this:
But asking students to compute the integral of this function on a Cal 2 final exam would be unfair, because it’s not possible to do with the techniques they learned in class (at least, I didn’t learn the technique you’d need until Cal 3). Because the integral doesn’t have a closed-form solution, you need to express the solution as an infinite series, like:
(Note: I asked AI for the integral of the Gaussian, I hope it got it right!)
It’s not obvious (at least, not to me) that differential calculus and integral calculus are related to each other. However, it turns out that these two calculi are opposite sides of the same coin, because integrals are anti-derivatives. That is, if f(x) is the derivative of F(x), then F(x) is the integral of f(x). This result is known as the Fundamental Theorem of Calculus.
This connection between differential and integral calculus raises an almost philosophical question: why is it so much easier to compute a derivative than it is to compute an integral? Back in 2011, somebody asked about this on the Mathematics Stack Exchange: Why is integration so much harder than differentiation? The top-voted answer was written by Qiaochu Yuan, and here’s the heart of it (emphasis mine):
Differentiation is a “local” operation: to compute the derivative of a function at a point you only have to know how it behaves in a neighborhood of that point. But integration is a “global” operation: to compute the definite integral of a function in an interval you have to know how it behaves on the entire interval (and to compute the indefinite integral you have to know how it behaves on all intervals). That is a lot of information to summarize. Generally, local things are much easier than global things.
In one sense, local things are easier than global things is a banal statement. Everybody knows that, for example, local optimization is much easier than global optimization. But it’s also a very deep one. And it gets at the title of this post, which is synthesis is harder than analysis.
I previously wrote about the difference between analysis and synthesis in the demon of the gaps. In analysis, we’re breaking a larger problem into smaller problems that separate out cleanly. These smaller problems are more localized, and hence easier to solve. This is why we advocate for principles like encapsulation and separation of concerns, to ensure our smaller problems are local.
The work of synthesis involves integrating(!) multiple things together. This pushes in the other direction: we are creating a problem that is less local. And global things are much harder than local things. The challenge we face is that some kinds of problems are just inherently synthesis problems. As I wrote in that previous post, incident response is one area where we are frequently confronted with synthesis problems: we have to understand how the pieces normally fit together in order to make sense of what is currently going wrong.
That’s why I think that this sort of synthesis work is important for SREs. Now, because synthesis is harder than analysis, and because SREs don’t have super-human cognitive abilities, it means that there is a limit to how deeply they will be able to understand any given component in the system. But the more they understand how the different components interact, the better positioned they are for helping resolve the tougher incidents.
Unfortunately, in our industry we haven’t recognized building up synthesis expertise as a first-class thing. That’s understandable because this work is very situated, it depends on the messy details of the particular system in the organization that an SRE works in. On the other hand, we can get better at learning how to learn about the operational details of a system. And that’s what I’d like to see more of.
Surfing Complexity 