GitHub, autoscaling, and the component substitution fallacy

In yesterday’s post about the recent GitHub outage, there was a detail in the writeup that I didn’t say anything about: the autoscaling policy on the service with the saturated Istio sidecar.

Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits.

I suspect readers of this blog are familiar with what autoscaling is and how it works, but here’s a brief summary in case you aren’t. The amount of compute and memory resources that a service requires depends on the load that’s placed on that service. The relevant source of load here is external requests against the service, also known as traffic. The volume of traffic varies over time. For example, for a company like GitHub, my guess is that they more traffic during working hours than evening and weekends.

Given that load changes dynamically, and that the compute and memory resources a service need is a function of load, there are two general strategies. One strategy is to provision your service for peak load. The other strategy is to dynamically adjust the resources allocated to your service, based on its current load; that’s called autoscaling.

If you want your service to use autoscaling, you need to define an autoscaling policy. In particular, you need to pick which metrics you want to use that represent load, and then you need to specify how resources should be added or removed based on how that metric changes.

CPU utilization is a common metric used for autoscaling. But note that a service can become saturated even if CPU is low. For example, imagine a scenario where you use thread-per-request with a threadpool, and the latency of your downstream requests increase, and all of the threads in the pool end up blocked. Here the service is saturated, and you’d benefit from spinning up new pods, but CPU is actually low, because the threads are blocked waiting on I/O (this happened to Slack back in 2021). Now, you can add additional rules to your autoscaling policy to handle such cases (which is what Slack did, where they rapidly scaled up based on number of threads). Or you can scale based on incoming request volume instead of CPU, if your service isn’t CPU-bound.

Based on the GitHub writeup, it sounds like the autoscaling policy for the impacted service used load metrics that only took into account load on the service itself, and not on the Istio sidecar.

In general, each service behaves differently under load, which means that every autoscaling policy is effectively bespoke. This means that a team that owns a service is not only responsible for the business logic, but also for an operational control system with custom parameters, that can really only be checked via load testing. (Are you doing load testing on all your services?) The service owners are also almost certainly not autoscaling experts. And so it’s not surprising to me that a misconfigured autoscaling policy was a contributor here.

But, while I think it’s worth discussing the particular defect with this policy, since it’s good for people to be aware of the risks of autoscaling, I also think it’s too easy to fixate on it to the exclusion of other factors involved in this incident. This is what David Woods refers to as the component substitution fallacy – the idea that the way to improve reliability is to focus efforts on identifying and fixing the defective components.

While, yes, you should identify and fix the defects uncovered by an incident, you should also recognize that:

your system is, at this very moment, filled with latent component defects
despite the presence of all of these defects, your system is not constantly failing over

This means that component defects aren’t enough to take down your system, or your system would be down right now. Don’t just look at the individual components: treat the interactions as first-class. In the GitHub outage, we see discussion of interactions between factors such as: changing traffic patterns (including scrapers), autoscaling policy, the Istio sidecar saturation, retry logic, HAProxy node saturation, and authentication traffic.

There’s also a multitude of details we don’t have because this is a rapidly disseminated public writeup, and the good stuff can only be found in the internal writeup. I speculated in this post about the relationship between service owner and autoscaling policy, but I would love to know more about the history here (did this policy predate the use of Istio sidecars, for example?). I’d also love to know more about the problematic traffic. (What kinds of requests were they? Was it a sudden increase or a gradual ramp-up? Do we know why the traffic increased?).

You can’t get answers to these sorts of questions for public incident writeups, but you can for the internal ones at your own organization. It’s up to you to ask the questions.

GitHub has another tough day

On August 6, 2026, GitHub had a pretty rough incident: GitHub Actions was degraded for about nine hours. GitHub posted a public incident write-up. It’s only a few paragraphs long, but there are some interesting details in here.

This was yet another incident that involved saturation. In fact, the write-up even uses the word saturated when describing what happened.

The incident was triggered by a routine deployment to an internal Actions service responsible for processing events and generating Actions jobs. The deployment exposed an existing capacity and concurrency weakness. As pods were replaced during the deployment, remaining capacity became saturated, causing services to crash and triggering a cascading impact across multiple clusters and downstream services.

We often think of deployments as risky because we are changing the code that’s running in production, and the execution of that new code could trigger a behavior change in the system that could lead to an incident. But a deployment is itself also an operational change in the behavior of the system: our system behaves differently during a deployment than it does when nothing is being deployed. Ironically, this is one of the advantages of deploying more frequently: the more often we are deploying, the more that deployment becomes a normal part of the system behavior – we get more experience with the system in deploying state.

In this particular case, running in the deploying state reduces the number of pods available for doing work, as the older pods go online. In this scenario, it sounds like the system was running close enough to the margin that the reduction in capacity due to the deploy pushed the system over the edge, leading to a cascading failure. This is what the resilience folks call a brittle collapse, which is when the system fails in a non-graceful way when it reaches saturation.

As is common when recovering an overloaded the system, they got it back to healthy by shedding load (throttling) and by increasing capacity.

These services recovered at 17:00 after expanding capacity, throttling incoming webhook-triggered work to allow the system to recover, and increasing processing capacity for the backlog of affected events.

I wish the write-up had more details on what was involved in enabling throttling and getting that additional capacity to come online. In particular, I’m curious about whether this was easy to do or difficult. But, alas, you typically don’t get those kinds of details on public writeups.

And, of course, because every incident involves multiple contributing factors, there was a previously undiscovered bug that made things worse by consuming available capacity trying to run invalid jobs:

Due to a latent bug in one of the services responsible for job assignment, runners were getting assigned jobs that were no longer valid and then getting stuck retrying those jobs, preventing them from picking up valid work.

I would love to know more details about how the heck they figured out what was going on with these stuck jobs. I can just imagine being a responder to this incident, trying to get the backlog of work processed, and discovering that there are workers are blocked trying to execute invalid jobs! How did they figure out they were stuck? How did they figure out this was because of a bug?

They deployed a change to work around this problem, but I would love to know what kind of change that was. Was it a quick workaround to get things moving again? I bet it was, but we’ll never know…

This second stage of impact was mitigated by deploying changes to prevent runners from repeatedly attempting to acquire invalid jobs. These mitigations allowed the accumulated queues to drain and Actions to recover to normal operation.

The write-up ends with the typical “here’s what we’re doing to make sure this doesn’t happen again” text, but I am heartened by the last sentence (emphasis mine):

We are also making additional improvements to reduce the risk of cascading failures and accelerate recovery during large-scale Actions disruptions.

Too often, the focus of reliability work is entirely on prevention. I’m happy to see them also focus on preparing to recover more quickly. Because, as we all know, the next big incident is always just around the corner.

Traditional versus resilience engineering views

As a fan of resilience engineering, I often differ with people on where we should focus our scarce engineering cycles in order to improve reliability.

I thought it would be a useful exercise to brainstorm some of the differences in focus between what I’ll call the traditional view of reliability, and the resilience engineering view.

Traditional view focuses on	Resilience engineering view focuses on
accountability	coordination
prioritization	goal conflicts
risk mitigation	risk trade-offs
better processes and conformance thereof	more expertise
quantitative	qualitative
root cause	interaction of multiple factors
action items	insight
preventing future incidents, ensuring all incidents are novel	better handling of novel incidents
reducing complexity	navigating complexity
objectives	production pressure
robustness	resilience
human variability as liability	human variability as asset
building accurate system model	repairing inevitable model errors
rigor	improvisation
explicit knowledge	tacit knowledge
automation, benefits of	automation, risks introduced by

Dear researchers column

The Journal of System and Software publishes a regular column called Dear Researchers: The perspective of software practitioners. Each column is an open letter to the software engineering research community from someone who works in tech. It’s edited by Austin Henley and Olaf Zimmermann, both of whom have experience in the two worlds of academia and industry.

They invited me to submit a column, which I did. When it finally gets published, you’ll be able to find it here: Dear researchers: help me deal with incidents! The published version will eventually go behind the journal’s paywall, but here’s a preprint of the column that you can always read free of charge.

The demon of the gaps

Mephistopheles (a medieval demon from German folklore) flying over Wittenberg, in a lithograph by Eugène Delacroix.

Modern software systems contain within them a mind-boggling level of complexity. As software engineers, we make this complexity manageable through techniques like decomposition, information hiding, and abstraction. We endeavor to break our systems up into components that interact over well-defined interfaces. By doing this, the surface exposed to individual software engineers is dramatically reduced: no individual has to understand how the entire complex system works in order to contribute to their system. Instead, each software engineer needs to understand only the individual component that they work on, along with the interfaces of the other components that they interact with. Decomposition is synonymous with analysis, where you study a larger thing by breaking it up into smaller pieces that are more amenable to understanding.

You can see this strategy of complexity management in action in microservice architectures. An engineer needs to understand the service that their team owns, and the interfaces of the services that their team calls out to. This architecture effectively bounds the information that an engineer needs in order to work effectively. Microservice architectures aren’t there for scaling the software itself, they’re there for scaling the software organization.

Unfortunately, when the system breaks down, this complexity management strategy breaks down itself. Just as hurricanes don’t respect political boundaries, system failures don’t respect component boundaries. Yes, sometimes the problem in a software system is limited to the failure of a single component. Those are the easiest cases to diagnose and mitigate. However, the hairy incidents are the ones that arise due to unexpected interactions across components. Maybe you have several services that are throwing errors, or maybe none of the services are throwing errors but customers are still seeing incorrect behavior. There’s no obvious change that correlates with the start of impact, or maybe you don’t even know when the impact started because the customer impact isn’t reflected in your existing metrics.

When you’re in the throes of an incident that involves an unexpected interaction, this architecture that was built for managing complexity now works against you. Because you’ve built an analysis solution but you’re now faced with a synthesis problem. You need to understand how the pieces all normally fit together to function in order to determine what is going wrong with the system right now. You’ve optimized to avoid requiring anybody to understand how the whole thing works, but now the whole thing isn’t working, and no one person knows how the whole thing works.

The job of the incident responders is to collectively figure out how to do that synthesis. You’ve brought together a group of people who each understand the functions of different components of the system, and you need to work together to build enough of an understanding of how the system functions to debug what’s going wrong. As an ad hoc team, the incident responders have to move up and down the abstraction hierarchy to figure this out.

This sort of in-the-moment reconstruction of system function from component parts is an essential part of incident response for the most complex incidents, but it’s rarely treated as first-class work that’s worthy of study and support. The recent book Crisis Engineering by Marina Nitze, Matthew Weaver, and Mikey Dickerson is the exception that proves the rule: they do discuss the work of building a model of the system during a crisis to help figure out what’s gone wrong. But I struggle to recall any other guidance I’ve read about incident response that talks about how to prepare for doing this sort of work. It’s important work, and it’s difficult, and the ability to do it well can have a huge impact on the time it takes to mitigate the hardest incidents. This is stuff that even the best individual humans struggle with, because it involves a group of humans working together effectively, with each person having a partial model of the system. And if the best humans struggle with it, I don’t think AI SRE tools are going to save us here: if the best humans struggle, the AIs will too. We need to figure out how to get better at this collectively. Like so many things, it’s a coordination problem.

Reliability as a game of improving the odds

I’m a betting man; I just enjoy making bets, even when there are no stakes at all.

And when you talk about bets, you end up talking about odds.

It turns out that reliability is also about odds, even though we don’t use the language of odds in our domain. Consider how we talk about availability. We report system availability as a number of nines: for example, we might say “four nines of availability”, which means 99.99% of somethings are good over some time interval. The canonical example of those somethings are successful requests. In that case, if someone says a service has four nines of availability over the past three months, that means that 99.99% of requests succeeded over that time period. We could express the same information by saying that there is a one in ten-thousand chance that any given request failed in the past three months.

If your system has exhibited four nines of availability in the past three months, and you assume that the availability of your system in the near future will be like the availability of the past (a dangerous and unwarranted assumption, but let’s go with it for a moment), then we could also express this information using the language of odds, by stating that the odds of a request failing are ten-thousand to one.

But this isn’t a post about describing availability in the language of odds. Instead, what I want to talk about is how all reliability work is inherently about improving the odds, increasing the likelihood that the system stays up. Any time we build any sort of reliability mechanism, be it load shedding, autoscaling, canarying, staged deployments, automated rollbacks, or what have you, we are building automation into the system that either eliminates or reduces the impact a subset of potential problems. If you ask an engineer working on improving reliability, “will this prevent all future incidents”, they will tell you “no, of course not”.

However, we don’t explicitly think of reliability work in terms of improving the odds. Instead, we tend to think of it as deterministically addressing a specific class of problem. You’ll hear questions like, “how many historical incidents would this tech have prevented?” in trying to determine whether engineering should invest in a particular reliability solution. They are looking for an answer like, “this would have prevented 20% of our SEV1s and SEV0s”. This 20% isn’t interpreted as a likelihood, instead it’s used as an estimate of impact, as in “this will improve our availability by around 20%”. The idea is that this reliability work will deterministically eliminate or mitigate a certain fraction of incidents; we just don’t know exactly what that fraction is, so we estimate it from historical data.

What I would like to propose in this post is that we think about all of the various kinds of reliability work as improving the odds of our system being up longer, instead of assuming that reliability work will have a fixed effect, and try to estimate the effect size. I’ve got two motivations for taking this perspective of reliability work as odds improvement.

The first motivation is that I don’t think we can ever estimate the effect size without error bars that are so huge that the estimates are themselves meaningless. As I’ve written about previously, the variation in incidents is just too large relative to the amount of data we have available. And, to make the estimation problem from historical data even worse, our system is changing over time. Or, to put it in technical terms, I don’t believe that incidents can be modeled as a stationary process. (Heck, if they were stationary, then that means that reliability work could not have an impact, because then the process would change over time!). Note that I’ve never seen anybody try to validate the estimates, they’re always point-in-time estimates used to justify work, and then promptly forgotten about. In one sense, that’s fine, they served their purpose of convincing leadership that we should allocate cycles for a particular kind of reliability work. But we shouldn’t fool ourselves into believing that these estimates are meaningful: they’re for persuasion, not insight.

It’s my second motivation, though, that prompted me to write this blog post. And that’s because the idea of reliability work as improving the odds of effectively mitigating future incidents is a useful framework for thinking about work that improves resilience. I’m interested in improving the skills of the people who respond to incidents, putting them in a better position to deal with those future unforeseen, surprising scenarios. One way to do this is learning from how responders dealt with previous incidents, the different sorts of observability data they had access to and how, the different knobs that were able to turn, and so on. While the next incidents will be different, the set of tools that are available during incident response are generally the same. There’s no way I can give a quantitative of estimate how this sort of skill improvement work will impact reliability. And despite the enormous number of random factors, I am confident that it will improve our odds.

Flipping the bozo bit on flips the learning off

I’m too young to have seen Bozo the Clown myself, but I’m old enough to get the references

“Flipping the bozo bit” is an expression from the software world. Think about a time when you reached a point where you simply stopped respecting the opinion of a particular person, most likely a co-worker. From that point on, you disregarded what they said. This is what flipping the bozo bit is. This person isn’t worth listening to, they’re a bozo.

There’s a related phenomenon, where we hear an anecdote about some bad outcome that happened to someone else, and our conclusion is that this outcome occurred because, well, that person is a bozo. I’m writing, of course, about incidents. You’ve seen this happen, right? An incident happens, the details of the incident get passed around, and somebody makes a comment like, “how could they have [not] done X?” The subtext is “what a bunch of bozos!”

This is on my mind because of the latest AI-related incident that befell PocketOS. You can read about it in the Twitter post written by the PocketOS founder, Jer Crane. The post is titled An AI Agent Just Destroyed Our Production Data. It Confessed in Writing. Unsurprisingly, this post got a lot of online attention. I saw a lot of “wow, was this guy ever a bozo” reactions to this story. I want to talk about why this reaction is counter-productive. I also want to call out the technical term for this phenomenon, which is a cousin of flipping the bozo bit. It’s called distancing through differencing.

The term distancing through differencing was introduced by the American resilience engineering researchers Richard Cook and David Woods in their 2006 paper: Distancing Through Differencing: An Obstacle to Organizational Learning Following Accidents. Technically, it’s a book chapter, from Resilience Engineering: Concepts and Precepts. It’s very readable, and I recommend it. All of the quoted text below is from that paper.

By focusing on the differences, they see no lessons for their own operation and practices.

When people hear about an incident and respond by concluding “an incident like that would never happen to us; that happened to those workers over there because they are clearly not as careful as we are, that’s distancing through differencing in action.

Overall they decided the incident “couldn’t happen here”.

The Cook and Woods paper illustrates the phenomenon with a case study of a chemical fire that broke out at an American manufacturing plant. There had been a similar fire that had occurred previously at the same company, at an overseas plant. The American employees knew about the previous fire, but they had concluded that there was nothing to learn from that other fire, as that sort of accident couldn’t happen to them in the U.S. After all, those overseas workers were less skilled, less motivated, and less careful. In short, those overseas workers were perceived as different.

Ironically, after the chemical fire at the Ameircan plant, other workers at that very same plant also exhibited distancing through differencing.

Workers in the same plant, working in the same area in which the fire occurred but on a different shift, attributed the fire to lower skills of the workers on the other shift.

Cook and Woods note that our tendency to focus on differences between us and them when the incident happens to them leads us to miss aspects of the system that we actually have in common with them. By focusing on the differences, we miss the opportunity to learn from their experiences, because it seduces us into believing there’s nothing for us to learn here.

do not discard other events because they appear on the surface to be dissimilar. At some level of analysis, all events are unique; while at other levels of analysis, they reveal common patterns.

Now let’s circle back to the PocketOS AI-related incident. If we come to the conclusion that PocketOS employees were simply using AI irresponsibly, and that we are more responsible than that, we learn nothing from the experience. I was heartened to see that Railway, the vendor used by PocketOS that exposed the delete API, has made changes to the overall system to improve safety; see their post: Your AI wants to nuke your database. Guardrails fix that.

Stepping back, this isn’t the last AI-related incident we’re going to see in our industry, not by a long shot. The next time you read one of those, if your reaction is “they should have known not to do X”, then you’ve fallen into the distancing through differencing trap.

(As an aside, “they should have known…” is an incoherent sentence. It’s one thing if somebody deliberately took on excessive risk. But it’s another thing if they unknowingly took on excessive risk. How can you blame a person for not knowing something?)

When this process of learning moved past the obstacle of distancing through differencing in this case, the organizational response changed.

After all, there but for the grace of God go we all.

How incidents can teach us about what’s already working well

Here’s a famous optical illusion, which was developed by the American neuroscientist Edward H. Adelson.

Even though square A appears darker than square B, the two are, in fact, the exact same shade of gray. It’s such a powerful illusion that, even knowing the illusion doesn’t destroy its effect; you’ll still “see” the illusion after you know about it. It’s so powerful that you may not believe me over your lying eyes. If you’re on macOS, you can confirm the illusion by opening the Digital Color Meter app and hovering your mouse pointer over each square in turn. You’ll see that both squares have the same RGB value. In hex, the value is #646464.

I’m going to suggest two stylized reactions to witnessing this illusion. One reaction is to say, “Oh, no! This illusion clearly illustrates a flaw in the human visual system! We should work on developing a vision correction technology so that people don’t fall victim to problems that would arise from this failure mode in human visual processing.”

A very different reaction is to say, “Oh, wow! This illusion gives us a hint into how the human visual system functions! Our brain must contain a prior model about the relationship between light, shadow, and objects, and is imposing that model when processing the signals coming from our optic nerve. This illusion appears to be an example of a pathological case which violates the human brain’s model.”

The first reaction is, admittedly, a ridiculous strawman. These sorts of illusions are harmless, so there’s no motivation to try to “correct” from them. After all, it’s no coincidence that the illusion was developed by a researcher who studies human vision. Even though our visual system is failing us in this strange case, the value of an illusion like this is not to learn the circumstances in which our vision fails, but instead to use the failure to gain insight into how our vision works so effectively for the vast majority of the time.

Last week, I wrote a post about Safety-II, the idea that we will learn more about how to create reliability in our system by studying the (common) successful cases rather than the (rare) failure cases. But we can also use the failure cases to learn about how the system normally succeeds! Just as neuroscientists can use optical illusions (where the vision system fails) to learn how the visual system succeeds, we can use incidents (when our system fails) to learn about how our system succeeds.

To make this more concrete, imagine you’re in an incident review meeting, and one of the incident responders, someone who is a real expert at your company, is talking about how, in hindsight, they misdiagnosed the problem during the incident. The signals that they saw misled them until thinking that the system was in state A, when really the system was in state B. And that led to the incident taking much longer to resolve, because the responders went down the wrong path.

The typical sort of question to ask in a review meeting would be along the lines of “what can we do to make sure we don’t misdiagnose this type of problem in the future?” But, there’s a very different question that you ask. And that question is, “how did the responder come to the conclusion the system was in state A?” Asking this question will expose details about the responder’s mental model of how the system actually works. If the responder was an expert, and they were led astray by the signals, then it’s likely that this incident was a pathological case, an operational equivalent of the optical illusion we saw above. By asking the responder about how they made the diagnosis, you are giving the meeting attendees the opportunity to learn from the expert responder. Similarly, you can ask the responder, “how did you finally figure out that the system was in state B?”, which will give you another chance to retroactively witness the work of an expert in action.

Like optical illusions, incidents are pathological cases. But, unlike illusion, incidents aren’t harmless. This means that the natural reaction is, “what went wrong here, and how do we stop doing that?” But if our goal is improvement, we should recognize there’s a lot more leverage in maximizing the opportunity to learn about what’s working well today, from the experts who are doing that work well. After all, there’s a reason we called that responder an expert; their work had led to a lot more success than failure.

Life comes at you fast

Now, here, you see, it takes all the running you can do, to keep in the same place. – Lewis Carroll, Through the Looking-Glass, and What Alice Found There

LLM coding may be revolutionizing software development productivity, but it doesn’t seem to be generating the same sorts of gains in software reliability yet. Two events that caught my eye today, although only one is directly related to LLMs.

The first event was that Anthropic suffered from another incident today, which lasted about an hour and a half.

This brought Claude Code down to one nine over the past 60 days, although they’re at two nines if you look over 90 days. I know, I know, I shouldn’t even talk about the nines, but they do make for a great screenshot.

The second event, the one I really want to focus in here, was GitHub’s CTO Vlad Fedorov writing the blog post: An update on GitHub availability. It was only six weeks ago that he wrote Addressing GitHub’s recent availability issues, which is clearly a sign that GitHub is concerned about the impact of recent incidents on their brand.

I want talk about GitHub’s post in the context of David Woods’s Messy 9 collection of patterns about complex systems. I’ve mentioned them before, but to re-iterate, they are: congestion, cascades, conflicts, saturation, lag, friction, tempos, surprises, tangles.

Fedorov notes that AI is driving a lot more activity on the site: the counts of pull requests, commits, repos are growing like never before.

Source: An update on GitHub availability

This is a great example of an increase in tempo: the environment that GitHub exists within is changing faster than it has previously. Heck, it’s right there in the title of that graphic: “Record Acceleration”. In particular, the load on GitHub as a system has increased significantly, and GitHub is struggling to keep up with this load. It puts GitHub at risk of saturation.

This exponential growth does not stress one system at a time. A pull request can touch Git storage, mergeability checks, branch protection, GitHub Actions, search, notifications, permissions, webhooks, APIs, background jobs, caches, and databases. At high scale, small inefficiencies compound: queues deepen, cache misses become database load, indexes fall behind, retries amplify traffic, and one slow dependency can affect several product experiences.

GitHub has to make changes to its internal systems in order to handle this load. I don’t work at GitHub, so I don’t know the details, but I have high confidence that they can’t simply horizontally scale their way out of the problem. They will likely have to rearchitect parts of their system in order to handle the increased load. And that will take time, even in the age of AI. And this is where the lags come in. It takes time to actually implement long-term solutions that can handle the load, which increases the probability of short-term outages since the system is running too close to the margin, and those outages delay the long-term solution work because the short-term firefighting steals engineering cycles, and so on. It’s a dangerous place to be, and I don’t envy them.

(As an aside, one other aspect of Fedorov’s post that I found interesting was how the increasing popularity of monorepos is also putting additional stress on GitHub as a system. People are using them in ways that designers had not envisioned!)

I don’t know whether Anthropic will reveal any details about the nature of their most recent outage, but as I’ve written about previously, the author of Claude Code mentioned on Twitter that Anthropic’s availability issues are related to unexpectedly rapid increases in demand. They are victims of their own success.

One of the reasons I don’t expect AI to improve reliability is that I don’t think LLMs are well-suited to mitigate the risk of saturation. As GitHub demonstrates, LLMs are more likely to be on the supply side when it comes to risk of saturation.

The normal work of creating reliability

Here’s a recent comment on LinkedIn from John Allspaw, on a post by Gandhi Mathi Nathan Kumar about availability.

Allspaw’s comment is a succinct description of a safety model proposed by the Danish resilience engineering researcher Erik Hollnagel: Safety-II. Hollnagel has described Safety-II in his book Safety-I and Safety-II: The Past and Future of Safety Management, as well as in white papers aimed at aviation and medical audiences. The book and white papers are all quite approachable, and I recommend checking them out.

Hollnagel’s observation is simultaneously trite and surprising: most of the time our systems are succeeding; incidents are the exception, not the norm. After all, this is why we measure availability in nines. The traditional approach to safety, what Hollnagel calls Safety-I, is to try to reduce the bad stuff, the work that leads to incidents. Hollnagel asks us to think about things differently: what if, instead, we focused on cultivating the good stuff: the everyday work that is consistently preventing accidents? There’s a lot more good stuff happening than bad stuff! Or, as my former colleague Ryan Kitchens put it, instead of asking why do things go wrong, it’s more productive to ask how do things go right?

In Hollnagel’s Safety-II model, the normal work that people in your organization do everyday is actively creating safety. Or, as the American organizational psychologist Karl Weick put it in his 1987 paper Organizational culture as a source of high reliability, reliability is a dynamic non-event. That is, the work is explicitly positive, and by the nature of this work, people are constantly doing work that is preventing incidents from happening. However, this work isn’t able to prevent all incidents, which is why they still happen. But taking Safety-II seriously means trying to understand how it is that normal work prevented previous incidents, rather than just trying to understand how it failed to prevent the last one. In Hollnagel’s words, the purpose of an investigation is to understand how things usually go right as a basis for explaining how things occasionally go wrong.

Focusing on the scenarios where things go right is a radical reframing of the problem, so much so that it is a genuinely strange idea, something that violates our intuitions about how systems break. We operate under a baseline, unspoken assumption that reliability is a passive thing, that the default behavior of a system is to stay up, and that somebody needs to actively do something wrong in order to cause the system to break. In other words, we view the day-to-day work people in the system do as a potential threat to reliability. And then, when an incident happens, we try to identify the bad work that broke the system.

If we were to take Safety-II seriously, we’d have to focus on how people adapt their work. It means seeing that people change how they do their work based on the pressures that they are currently facing and the constraints that they are under. More importantly, it means that we have to acknowledge that these adaptations are usually successful. If you only look at these adaptation within the context of an incident, and try to improve reliability by preventing these adaptations, it’s like believing you can figure out how to win the lottery by examining the behaviors of lottery winners. Sure, you can identify patterns among the behavior of lottery winners. But there are even more folks who lose the lottery who exhibit those behaviors, you’re just not looking at those. Note, though, how much this goes against the way people think about how incidents happen.

Safety-II is also challenging to adopt because organizations are simply not used to studying the normal work that goes on in an organization in order to answer the question, “what work is going particularly well, and how can we do more of it?” The closest we probably get is shadowing that happens when new employees join. We do have developer experience surveys, but those focus specifically on problems with existing tooling. I don’t know of any reliability organization at any tech company out there that takes a Safety-II approach and spends time understanding what’s happening when it looks like there’s nothing happening. Perhaps they’re out there, but if they are, they aren’t writing about this work. The one exception to this is the resilience in software folks, but even with us, we’re generally focused on shifting the emphasis of post-incident examination of work, rather than examining work outside of the context of incidents.

Now, attention is a limited resource in an organization, and incidents win the attention of an organization because they are troubling by their nature. Because attention is limited, if all the indicators are currently green, that’s taken as a sign that we can safely spend our attention budget elsewhere. In the tech industry, we also don’t have great models for how to study normal work within an organization, because nobody seems to be doing it. Or, if they are, they aren’t writing about it. In his Safety-II book, Hollnagel recommends doing interviews and field observations. In tech, field observations are trickier because the majority of our work is effectively invisible; we do our work alone at a computer. We can observe interactions over channels like Slack and Zoom, but that’s only part of the story. I suspect that interviews are our best potential source of information here. And then we need to take what we’ve learned from the interviews and use those insights to improve reliability by amplifying what’s already working well. That’s not something we have experience with.

It’s no surprise, then, that Safety-II hasn’t caught on our field. It cuts against our intuitions about the nature of complex systems failure, and we don’t have good public examples to work from about this. We resilience in software folks are trying to push the industry in this direction with trying to get people to think differently about what we can get out of incident analysis, and that’s probably our best bet right now. But we have a long way to go.