Last week, GitHub Actions experienced another incident. As is typical of a GitHub public writeup, it was published very soon after the incident, but there also aren’t isn’t a lot of detail. But let’s see what we info we can glean from it.
Database saturation
The first thing I noticed is that this incident is, once again, a saturation-related failure mode. Specifically, it was a database that was saturated due to write traffic.
This impact was triggered by saturation of writes to the database primary used by the service processing triggers for Actions workflows.
As I mentioned in my last blog post, database-related saturation issues are particularly pernicious, because they can be very difficult to recover from.
Multiple contributors, but not much detail
The write-up mentions seven separate factors that contributed to the incident.
- Growing peak daily load (this increased the writes to the database)
- An upstream issue in GitHub’s event processing infrastructure (?), which further increased the load
- Failing over from primary to replica did not lead to full recovery (?)
- Existing throttles were set ~10% too high, so they provided insufficient overload protection
- some jobs remained stuck in queued/waiting state after recovery (?)
- another issue that left some jobs left in a waiting-for-runner state after recovery
- a bug that left that some runs showing as queued even though they had already failed
I annotated contributors with (?) where I felt the report really didn’t provide any details at all. The mention of the upstream issue references a different GitHub incident, but there are no details on that other incident page at all. It does say “A detailed root cause analysis will be shared as soon as it is available”, so perhaps we’ll get more details on this other issue in the next few days.
What I’m most curious about, though, is what happened with the database failover. All we get in the write-up is this one line:
The primary was failed over, but the system did not fully recover.
What happened here??? Did the newly promoted primary get overwhelmed the same way that the previous one did? Did something else go wrong? I wish there they went into a lot more detail on the particular failure mode.
Slowly nursing an overloaded database back to health
On the plus side, the report does have some details on how they were able to mitigate. They throttled traffic to the database until it recovered, and then ramped the traffic back up slowly enough so that they didn’t knock it over again. Here’s the actual text:
At 15:45 UTC, throttling combined with service restarts recovered the service’s core health. Those throttles were gradually raised between 15:54 and 17:22 to restore full webhook processing for Actions runs. This ramp was deliberately slow to ensure we did not re-overwhelm the system given our original throttling was now known to be incorrectly set. The queue of webhook events was fully burned down at 17:40 UTC.
Two things I want to note about this. First of all, this sort of recovery approach is something you are going to need to do some day when your own database gets overloaded (and, believe me, it’s going to happen). If you’re prepared for this, you’ll have access to a throttle knob that the responders can manually control so they can cut the traffic and then increase it. It’s not something you want to have to build during an incident.
The second thing to note is that throttling means that you are deliberately cutting off access to the database for your users in order to bring it back up. This means that you will be temporarily increasing user pain in order to recover the system. This sucks, but it’s a decision you sometimes have to make during an incident: that you actually have to deliberately make the system behave worse from the user’s perspective in order to get it back into a healthy state. Now, if you have the ability of doing QoS-style throttling where you can selectively block the less important requests, then you might be able to reduce the amount of pain. But, once again, that’s something you need to have built into your system in advance.
Irony: fix was already in-flight when the incident struck
This line in the write-up broke my heart a little (emphasis mine):
Several changes to improve the general scalability of this part of Actions were already complete and deploying to production. Rollout of those changes will be complete within the next 24 hours.
They were already working on reducing the likelihood of an incident like this, but it bit them before they could finish rolling out the improvements. That’s really just bad luck.
Another GitHub incident, another limit hit
Finally, we continue to see GitHub hitting one limit after another as they experience continued growth. There are just so many different limits within a system like this. I won’t be surprised if I’m soon reading up on yet another saturation-related GitHub incident.
Surfing Complexity 