It wasn’t even a week ago when I wrote about a major GitHub incident. Yesterday, they had another big incident, which lasted almost eight hours. There’s a public write-up already posted, which is surprisingly quick. While I’m personally very impatient to read these, I also know that it takes time to collect and synthesize the information you need to do a good job with them. I wish they had posted this as a preliminary write-up and then done a more detailed write-up in a couple of weeks. That being said, let’s look at the write-up!
Saturation strikes yet again
The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic.
The failure mode is yet another example of saturation, a topic I’ve written about again and again on this blog. Heck, I even gave a talk on saturation a month ago.
Here’s the full paragraph on the failure mode:
The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers.
Based on this, it sounds like the failure cascade looked like:
increase in external traffic → istio sidecars saturate (concurrency limits) → HAProxy nodes saturate (flow limits) → authentication requests fail
An increase in load on the system saturated one of the components (istio sidecar), and that propagated to another component (HAProxy), whose saturation broke the auth flow.
I wish they had included an architectural diagram here, that showed the relationship between the load balancers, the service that whose Istio sidecar pod saturated, the gateway, and the services that handle auth requests. Also, the wording gives the impression that only a single sidecar pod that saturated (an Istio sidecar pod), which would be surprising, but I’m also not confident that this is what the authors intended.
Diagnostic details: missing in action
The write-up doesn’t talk about the diagnostic work of the incident responders at all, which is a shame. I can’t tell from this write-up how difficult it was for them to figure out what was happening. There were auth failures, but it doesn’t sound like there was an increase in auth traffic per se, nor was the problem caused by recent changes to the auth system, which is where I would think to look first.
As somebody who was watching the updates to the status page as the incident was happening, I was struck by how they updates alternated between “we have identified the problem” and “we are experiencing issues:
I can imagine how frustrating it must have been for the responders to think they had found and fixed the problem, only to continue to see impact.
Retries made things worse
Retries are one of the tools in our toolbox to improve availability. And, usually, retries do improve availability! But retry logic also adds complexity to a system, and adding complexity to a system can introduce new failure modes. In this incident, retries hurt rather than helped, by increasing the load on an overloaded system.
The problem was worsened by optimistic retry logic which overloaded internal load balancers.
…
Residual Copilot authentication failures continued because client retry behavior amplified load: a failed token operation could generate many extra requests and enter a retry loop.
This is a great example of unexpected behavior of a subsystem whose primary purpose was to improve reliability from my conjecture on why reliable systems fail.
The Copilot Token Service sees 10X traffic
Note that there were two independent retry behaviors mentioned in the previous section:
- optimistic retry logic against the load balancers
- client retry logic against the Copilot Token Service
It turns out that the client retry logic was due to a previously undiscovered bug in Visual Studio Code(!), which led to one particular service (Copilot Token Service) taking longer to recover:
Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.
…
Residual Copilot authentication failures continued because client retry behavior amplified load: a failed token operation could generate many extra requests and enter a retry loop. Copilot Token Service traffic increased from a normal 7–9K RPS to 70–100K RPS.
There’s no way you’re pushing out a VS Code bugfix to mitigate an incident! You’ve got to mitigate that on the server side, which is what the responders did, which brings us to the next section.
Mitigating the incident: multiple strategies
While the write-up doesn’t discuss diagnostic work, it does mention multiple mitigations that the responders undertook during the incident:
- shifting traffic from the Central US region to the Northern Virginia region
- paused HAProxy on the four saturated nodes
- changed gateway retry logic (via PR)
- blocked inbound Copilot Token Service token requests at the load balancer (returned 403s)
- gradually ramping up blocked traffic
As responders, we are always limited in our ability to intervene based on the tools that we have at our immediate disposal. It’s incredibly useful to be able to do things like selectively block traffic, or dynamically change or even disable a reliability-related subsystem. Think about how difficult it would be to block specific types of requests during an incident in your organization, and to ramp that traffic back up slowly after the system recovers. Note how the responders had to use a pull request to change the behavior of the gateway retry logic. I wonder if the failure mode made this more difficult to carry out, but the write-up doesn’t say.
“Never again” means never preparing for a novel incident
The writeup ends, as most writeups do, with some action items intended to prevent recurrence.
To prevent recurrence, our follow-up actions include:
- Correcting autoscaling policies to account for service-mesh sidecar concurrency and capacity.
- Auditing Istio request, concurrency, and scaling limits across affected services.
- Reviewing retry limits and backoff behavior across gateways and clients.
- Addressing the VS Code retry behavior that amplified Copilot token traffic.
- Improving load-balancer capacity monitoring and regional failover safeguards.
My eternal lament is that people spend too much of their focus on preventing the last incident from recurring. It’s not that I’m opposed to preventative work. It’s that I also want us to spend time on getting better at dealing with novel incidents. Engineering cycles are a finite resource, and every cycle spent on prevention is a cycle not spent on improving our ability to respond effectively to new incidents. And I promise you, you are going to face novel incidents in the future.
After all, I don’t think GitHub customers who experienced this outage take much solace in knowing that it was a different failure mode from the previous incident.
Surfing Complexity 