When it comes to public incident write-ups, GitHub continues to be the gift that keeps on giving. They keep suffering collateral damage from the AI boom, as more developers using AI means more load on their system. Their developer focus also means that they provide some technical details about these incidents.

Today I’m writing about the incident that happened to them on September 13. The report has the vague title Incident with several GitHub Services, which was the title when the incident was declared. It’s a shame they don’t re-title their incidents later on.

The report is short enough that I’ll simply excerpt the three paragraphs that discuss the failure mode.

The cause was an internal data-cleanup job that began writing to a shared database cluster at 07:33 UTC. That cluster stores permission data read on nearly every authenticated request. The safeguard that was pacing the background job watched only one health signal — how far the database replicas were lagging — and that signal stayed low the whole time. It did not account for the load building on the primary itself, so the job kept writing while the primary quietly ran toward its limit.

When the primary ran out of available connections, requests that needed it could not complete. First, there was no quick timeout on these database calls, so request handlers waited on the stalled database instead of failing fast, and the shared request-handling capacity degraded into site-wide errors. Second, a retry loop around token creation kept re-sending the writes that were already failing, which held the database saturated rather than letting it recover.

Monitoring declared the incident at 08:50 UTC, but due to the broad impact and amplification from token creation, it took time to identify the source of the load. First responders mitigated by shedding internal load and pausing the job, and all services recovered by 10:44 UTC.

Here’s my attempt to depict this text with a diagram, and to rephrase the above in my own words.

Within GitHub, there is a database cluster in the typical SQL database style cluster configuration, with a primary and read replicas, where the primary is responsible for handling all of the writes. This cluster takes online traffic: meaning that activity from external users will generate queries against this cluster.

GitHub ran an offline job (i.e., not driven by user traffic) that made queries against the database to clean up unneeded data. The rate at which this cleanup job queried the database was governed by a safeguard mechanism that monitored the health of the database: the safeguard system would reduce the rate of queries if the database appeared to be going unhealthy. The safeguard monitored the health of the database using the replica lag (if they were using MySQL, presumably they were tracking Seconds_Behind_Source). An increase in this replica lag can be a symptom of high load on the replicas.

In this scenario, the replicas themselves were healthy, and the replica lag stayed below the threshold that the safeguard was monitoring. However, it was the primary node of the database cluster that was at risk of saturation. Specifically, the primary reached its maximum number of client connections (on MySQL, this is controlled by the max_connections server system variable). This means that any attempt to query the database over a new connection would fail with an error (on MySQL, this would be ER_CON_COUNT_ERROR, with an error message of Too many connections.

When some web servers attempted to make calls to this database, the requests did not succeed immediately, but neither did they fail immediately. Instead, these requests were stuck waiting. Unfortunately, the timeouts configured on these request handlers were long. This meant that these blocked requests accumulated in the impacted web servers. Each in-flight request consumes some amount of resources, and enough of these blocked requests accumulated in the web servers that they themselves became saturated, which led to sitewide impact.

To make matters work, there was a token creation process that kept trying to query the primary, failing, and then retrying. This placed a continual load on the primary that made it more difficult for the system to recover.

Aside: some unanswered technical questions

Here some questions I had that I couldn’t figure out from the text.

How did exhausted connections lead to timeouts on the web servers? Was it the case that the web servers were using connection pooling, not all of the connections in the pool were active, and the clients were stuck waiting for a new active connection that never came?

How did that background job lead to the primary exceeding the maximum number of connections? Did this job consume a surprisingly number of connections? How close was the database to the limit before the job started?

—

You can never truly know where the safety boundary is

When you have an offline job running against an online database, there’s always a risk that it can negatively impact the performance of the database, and thereby affect customer traffic. The nice thing about offline jobs is that they are, in principle, fully controllable by the organization. With user traffic, on the other hand, you are at the mercy of your users.

And, so, GitHub had an automated system (the safeguard) monitor the health of the database while the cleanup job was running, to ensure that the job was not applying too much load to the database. If the load was getting high, then the safeguard would slow down the rate at which the cleanup job was querying the database. The problem was that the metric being monitored by the safeguard did not give a complete picture of the health of the cluster. As far, as the safeguard was concerned, the database was still healthy, even though it was running out of connections. This is a great example of a gray failure: when your internal health monitoring systems register, incorrectly, that the system is healthy. In other word, the metrics are good, but the users are unhappy.

Putting things in resilience engineering terms, the system misjudged the location of the safety boundary. There was no signal that the system was getting close to being in an unsafe state until the boundary was crossed. Note that we never actually know where the safety boundary is until we actually cross it. But crossing the safety boundary is very, very bad. And so we always have to make an estimate about where the safety boundary is, and then we avoid getting too close. But there are so many limits in the system, that the chances of us not monitoring all of them is, tragically, quite high.

Offline jobs and online databases: sometimes you gotta run ’em

In an ideal world, we would want all of the query traffic for our online databases to be, well, online traffic. That’s why, for example, we don’t run our analytics queries against our online databases. However, there are scenarios where you have no choice: you have to run a job against an online database, because you want to effect some sort of change to the online data. This is a great example: here the job has to be run on the online database because the goal of the job is to make a change to that data.

The double-edged sword of cleanup

When it comes to the topic of “cleanup”, both for data and code, once you’ve seen enough incidents, you’ll notice a pattern: cleanup is a frequent risk. Sometimes incidents happen because a cleanup job was attempted, and the thing being cleaned up was still in use. And sometimes incidents happen because some entity that wasn’t even used anymore interacted in an unexpected way with a different part of the system. In other words: an absence of cleanup contributed to the incident. So, not cleaning up unused stuff is a risk: it can lead to incidents. But the act of cleanup is itself a change to the system that carries risk. It’s all risk tradeoffs.

The asymmetry of misconfigured timeouts

Timeouts are an essential tool in building reliable distributed systems. But there’s an asymmetry in the timeouts-are-too-short problem versus the timeouts-are-too-long problem. If your timeouts are too short, you’ll notice this during the normal operation of your system, by seeing excessive timeouts happening. And, so, when that happens, you adjust the timeouts to be longer.

However, timeouts that are too long are harder to detect. If you’re lucky, you might notice some sluggishness in your user interface and diagnose the problem as being related to timeouts being too long. But, in many cases, you won’t have any signal at all that timeouts are too long until they contribute to an incident. As we saw here, long timeouts create a saturation risk by increasing the number of in-flight requests in a service, thereby depleting its resources.

You probably have timeouts that are configured to be too long in your system, and nobody has noticed them yet.

Retries help, until they hurt

Retries are yet another essential tool in building reliable distributed systems. Because transient failures are common, retries can effectively mask these failures from your users: they’ll never know about that one bad pod in your deployment. But retries increase load, and in a saturation failure mode, you can get a pathological behavior where retries exacerbate the incident. That’s what happened in this case, where a particular service (token creation) kept retrying over and over again, leading to unending load on the database.

You need retries! But, like misconfigured timeouts, it can be hard to catch misconfigured retries until the incident happens. That’s exactly why our industry has a specific term for failure modes where retries made things worse: retry storms.

When you’re overloaded, you need to reduce the load

Let’s look at the last quoted paragraph again, since it has some details about the incident response (emphasis mine).

Monitoring declared the incident at 08:50 UTC, but due to the broad impact and amplification from token creation, it took time to identify the source of the load. First responders mitigated by shedding internal load and pausing the job, and all services recovered by 10:44 UTC.

Those of us who do incident response know that our first priority is always getting the system back to healthy. If you can quickly identify the source of the problem, that’s great! Go ahead and roll back that problematic deploy. But often we can’t tell what’s causing the problem. In a saturation failure mode, we can often identify which service has gotten overloaded (and, very often, it’s a database), but it isn’t obvious what the source of the problem actually is. In scenarios like this, you want the responders to be able to:

• quickly identify the sources of load (which queries, and who is making them?)
• manually shed load (block specific queries/sources)

To have that operational tooling at your fingertips during an incident, you need to have built it in advance. During the incident, it’s too late, you’re stuck with whatever tooling is available.

Your systems will eventually become overloaded, this I promise you. Your online databases at particular risk. The time to prepare for this is now.

The solution, as always, is to increase complexity

Here’s the last paragraph of the write-up:

To prevent recurrence, we are rate-limiting background jobs against shared, customer-serving databases by default, and adding automatic pausing and paging on primary-server load rather than replication lag alone. We are also surfacing running background work directly alongside database health signals so responders can see and pause it without leaving those dashboards, bounding retries in the token-issuing path, and adding request-level timeouts so one unhealthy database cannot consume shared web server capacity. In addition, we are breaking apart this database cluster to remove the single point of failure. We will be moving various service-specific data, including the authorization data, out of this shared cluster in the next two weeks.

The proposed new interventions include:

• adding rate-limits
• monitoring additional metrics (primary-server load)
• adding request-level timeouts
• sharding the database cluster

These all look good interventions to me. But note how these all increase complexity! They all involve adding new stuff to the system, as post-incident interventions always do. I am on record as stating that increasing complexity in order to improve reliability is inevitable. But we should be sanguine about this. By increasing complexity, we eliminate known failure modes at the cost of introducing novel failure modes. And introduce them we do.

Modern software systems contain within them a mind-boggling level of complexity. As software engineers, we make this complexity manageable through techniques like decomposition, information hiding, and abstraction. We endeavor to break our systems up into components that interact over well-defined interfaces. By doing this, the surface exposed to individual software engineers is dramatically reduced: no individual has to understand how the entire complex system works in order to contribute to their system. Instead, each software engineer needs to understand only the individual component that they work on, along with the interfaces of the other components that they interact with. Decomposition is synonymous with analysis, where you study a larger thing by breaking it up into smaller pieces that are more amenable to understanding.

You can see this strategy of complexity management in action in microservice architectures. An engineer needs to understand the service that their team owns, and the interfaces of the services that their team calls out to. This architecture effectively bounds the information that an engineer needs in order to work effectively. Microservice architectures aren’t there for scaling the software itself, they’re there for scaling the software organization.

Unfortunately, when the system breaks down, this complexity management strategy breaks down itself. Just as hurricanes don’t respect political boundaries, system failures don’t respect component boundaries. Yes, sometimes the problem in a software system is limited to the failure of a single component. Those are the easiest cases to diagnose and mitigate. However, the hairy incidents are the ones that arise due to unexpected interactions across components. Maybe you have several services that are throwing errors, or maybe none of the services are throwing errors but customers are still seeing incorrect behavior. There’s no obvious change that correlates with the start of impact, or maybe you don’t even know when the impact started because the customer impact isn’t reflected in your existing metrics.

When you’re in the throes of an incident that involves an unexpected interaction, this architecture that was built for managing complexity now works against you. Because you’ve built an analysis solution but you’re now faced with a synthesis problem. You need to understand how the pieces all normally fit together to function in order to determine what is going wrong with the system right now. You’ve optimized to avoid requiring anybody to understand how the whole thing works, but now the whole thing isn’t working, and no one person knows how the whole thing works.

The job of the incident responders is to collectively figure out how to do that synthesis. You’ve brought together a group of people who each understand the functions of different components of the system, and you need to work together to build enough of an understanding of how the system functions to debug what’s going wrong. As an ad hoc team, the incident responders have to move up and down the abstraction hierarchy to figure this out.

This sort of in-the-moment reconstruction of system function from component parts is an essential part of incident response for the most complex incidents, but it’s rarely treated as first-class work that’s worthy of study and support. The recent book Crisis Engineering by Marina Nitze, Matthew Weaver, and Mikey Dickerson is the exception that proves the rule: they do discuss the work of building a model of the system during a crisis to help figure out what’s gone wrong. But I struggle to recall any other guidance I’ve read about incident response that talks about how to prepare for doing this sort of work. It’s important work, and it’s difficult, and the ability to do it well can have a huge impact on the time it takes to mitigate the hardest incidents. This is stuff that even the best individual humans struggle with, because it involves a group of humans working together effectively, with each person having a partial model of the system. And if the best humans struggle with it, I don’t think AI SRE tools are going to save us here: if the best humans struggle, the AIs will too. We need to figure out how to get better at this collectively. Like so many things, it’s a coordination problem.

I’m a betting man; I just enjoy making bets, even when there are no stakes at all.

And when you talk about bets, you end up talking about odds.

It turns out that reliability is also about odds, even though we don’t use the language of odds in our domain. Consider how we talk about availability. We report system availability as a number of nines: for example, we might say “four nines of availability”, which means 99.99% of somethings are good over some time interval. The canonical example of those somethings are successful requests. In that case, if someone says a service has four nines of availability over the past three months, that means that 99.99% of requests succeeded over that time period. We could express the same information by saying that there is a one in ten-thousand chance that any given request failed in the past three months.

If your system has exhibited four nines of availability in the past three months, and you assume that the availability of your system in the near future will be like the availability of the past (a dangerous and unwarranted assumption, but let’s go with it for a moment), then we could also express this information using the language of odds, by stating that the odds of a request failing are ten-thousand to one.

But this isn’t a post about describing availability in the language of odds. Instead, what I want to talk about is how all reliability work is inherently about improving the odds, increasing the likelihood that the system stays up. Any time we build any sort of reliability mechanism, be it load shedding, autoscaling, canarying, staged deployments, automated rollbacks, or what have you, we are building automation into the system that either eliminates or reduces the impact a subset of potential problems. If you ask an engineer working on improving reliability, “will this prevent all future incidents”, they will tell you “no, of course not”.

However, we don’t explicitly think of reliability work in terms of improving the odds. Instead, we tend to think of it as deterministically addressing a specific class of problem. You’ll hear questions like, “how many historical incidents would this tech have prevented?” in trying to determine whether engineering should invest in a particular reliability solution. They are looking for an answer like, “this would have prevented 20% of our SEV1s and SEV0s”. This 20% isn’t interpreted as a likelihood, instead it’s used as an estimate of impact, as in “this will improve our availability by around 20%”. The idea is that this reliability work will deterministically eliminate or mitigate a certain fraction of incidents; we just don’t know exactly what that fraction is, so we estimate it from historical data.

What I would like to propose in this post is that we think about all of the various kinds of reliability work as improving the odds of our system being up longer, instead of assuming that reliability work will have a fixed effect, and try to estimate the effect size. I’ve got two motivations for taking this perspective of reliability work as odds improvement.

The first motivation is that I don’t think we can ever estimate the effect size without error bars that are so huge that the estimates are themselves meaningless. As I’ve written about previously, the variation in incidents is just too large relative to the amount of data we have available. And, to make the estimation problem from historical data even worse, our system is changing over time. Or, to put it in technical terms, I don’t believe that incidents can be modeled as a stationary process. (Heck, if they were stationary, then that means that reliability work could not have an impact, because then the process would change over time!). Note that I’ve never seen anybody try to validate the estimates, they’re always point-in-time estimates used to justify work, and then promptly forgotten about. In one sense, that’s fine, they served their purpose of convincing leadership that we should allocate cycles for a particular kind of reliability work. But we shouldn’t fool ourselves into believing that these estimates are meaningful: they’re for persuasion, not insight.

It’s my second motivation, though, that prompted me to write this blog post. And that’s because the idea of reliability work as improving the odds of effectively mitigating future incidents is a useful framework for thinking about work that improves resilience. I’m interested in improving the skills of the people who respond to incidents, putting them in a better position to deal with those future unforeseen, surprising scenarios. One way to do this is learning from how responders dealt with previous incidents, the different sorts of observability data they had access to and how, the different knobs that were able to turn, and so on. While the next incidents will be different, the set of tools that are available during incident response are generally the same. There’s no way I can give a quantitative of estimate how this sort of skill improvement work will impact reliability. And despite the enormous number of random factors, I am confident that it will improve our odds.