Wild AI-related reliability incidents are coming

Recently, two AI-related pieces of content caught my attention. The first was the blog post On-Call is Now Theatre by Boris Tane. He argues that AI agents are now capable of doing the majority of on-call work that is currently being done by humans, and that we should have AI agents act as first responders. Only when an AI agent isn’t capable of remediating the problem should a human actually be brought in, and the agent should be the one to page the human. As he puts it:

We need software that watches itself, triages its own alerts, investigates its own incidents, fixes what it can, and escalates to a human only when it hits something genuinely novel, with the evidence already assembled.

Put your AI agents in the worst on-call rotation imaginable, then give them a tool to page a human. Developers stop being the first responder, and step in only when an agent genuinely cannot figure something out.

Tane doesn’t think that companies will really start putting AI agents on-call (“Most teams won’t do this”), but he believes it should happen, and he’s started a company based on this premise.

I like to think of putting AI agents on-call as equivalent to using AI agents to implement control system automation. Because, after all, that’s what operations work is: it’s taking control actions to keep the system in a healthy state.

Now, AI agents are extremely complex software systems. I’d argue that they are the most complex software systems that we humans have ever built. That complexity is both good and bad. Ashby’s Law teaches us that the larger the set of system states that you want your control system to be able to handle, the more complex that it needs to be. It’s this complexity that makes it possible, in principle, to apply AI agents to solve a generic control problem like this.

On the other hand, the more complex a system becomes, the more difficult it is for a human to reason about the system’s behavior. That’s fine when the system is healthy, but if your now-even-more-complex system gets into a state that the automation can’t handle, that can make the problem even worse. Indeed, it’s precisely the unexpected behavior of complex control systems that contributes to the worst complex systems failures (see also: Air France 447, Boeing 737 MAX accidents).

And that brings me to the other piece of content I saw recently: the OpenAI talk at BlackHat (h/t David Blank-Edelman). Yes, it’s a 37 minute talk, but I encourage you to watch it.

The talk goes into detail about the surprising behavior of AI agents that resulted in security incidents at both OpenAI and Hugging Face. Honestly, this talk feels like something out of a movie about technology run amok; the sort of thing that still feels to me like absolute science fiction.

Because this was a talk at a security conference, the speakers focused on the lessons that apply to the security community. But as a reliability type, my biggest takeaway from this talk is that autonomous LLM agents can behave in ways that humans would have never expected. While agents today can perform complex cognitive tasks, they behave differently than a human would performing that task. We’re most familiar with this when they make a different kind of mistake than a human would make. In the OpenAI-HuggingFace incident, it wasn’t so much that they made a mistake, it’s that the agents pursued their goals in ways different than a human would do. If a teammate of yours used 0-day exploits to overcome internal security protocols in order to get their work done, you’d say they were acting unreasonably. And that’s exactly the risk here.

The inevitable improvement in frontier models does not mean that the agent behavior will be easier to reason about; I actually think it’s the opposite. The agent behavior will get even more complex with the more advanced models, but that doesn’t mean it will get more human-like. Humans are very complex, but we know how to reason about human behavior; at least, we do for the people we work with. After all, an employee whose behavior was unpredictable would not last long in the organization. As these agents become even more capable, they will be akin to alien minds: intelligence, but not as we know it.

Here’s how I think things will play out. I think that some teams will do what Tane proposes and will use AI agents as first-responders to deal with operational issues. And I think that for many cases, the agents will successfully remediate issues. Of course, for the agents to actually be able to remediate, they will need to have permissions to take operational actions without human intervention.

One day, though, there will be a complex incident which the agents will not be able to handle. Tane believes that the agents will defer to the humans in this case, by paging in a person. But that’s not the scenario I worry about. The one I worry about is that the agents attempt to remediate, and their attempt makes things worse. And it’s only after these failed remediation attempts that humans enter the loop. Maybe they eventually page in a human, or maybe a human notices that something is very wrong as the agents continue to try and fail in their remediation actions. But now the humans have to make sense of the combined software-AI-agent system behavior. The original problem was already so complex that the agents couldn’t handle it, and they have now made it worse by trying to remediate. I can even imagine the humans fighting the agents who keep trying to take actions to remediate that are failing.

This is the incident that’s coming. And it’s going to be very, very difficult to handle when it happens. And I have no idea how people will respond to the role of the AI agents in the wake of this incident.

GitHub, autoscaling, and the component substitution fallacy

In yesterday’s post about the recent GitHub outage, there was a detail in the writeup that I didn’t say anything about: the autoscaling policy on the service with the saturated Istio sidecar.

Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits.

I suspect readers of this blog are familiar with what autoscaling is and how it works, but here’s a brief summary in case you aren’t. The amount of compute and memory resources that a service requires depends on the load that’s placed on that service. The relevant source of load here is external requests against the service, also known as traffic. The volume of traffic varies over time. For example, for a company like GitHub, my guess is that they more traffic during working hours than evening and weekends.

Given that load changes dynamically, and that the compute and memory resources a service need is a function of load, there are two general strategies. One strategy is to provision your service for peak load. The other strategy is to dynamically adjust the resources allocated to your service, based on its current load; that’s called autoscaling.

If you want your service to use autoscaling, you need to define an autoscaling policy. In particular, you need to pick which metrics you want to use that represent load, and then you need to specify how resources should be added or removed based on how that metric changes.

CPU utilization is a common metric used for autoscaling. But note that a service can become saturated even if CPU is low. For example, imagine a scenario where you use thread-per-request with a threadpool, and the latency of your downstream requests increase, and all of the threads in the pool end up blocked. Here the service is saturated, and you’d benefit from spinning up new pods, but CPU is actually low, because the threads are blocked waiting on I/O (this happened to Slack back in 2021). Now, you can add additional rules to your autoscaling policy to handle such cases (which is what Slack did, where they rapidly scaled up based on number of threads). Or you can scale based on incoming request volume instead of CPU, if your service isn’t CPU-bound.

Based on the GitHub writeup, it sounds like the autoscaling policy for the impacted service used load metrics that only took into account load on the service itself, and not on the Istio sidecar.

In general, each service behaves differently under load, which means that every autoscaling policy is effectively bespoke. This means that a team that owns a service is not only responsible for the business logic, but also for an operational control system with custom parameters, that can really only be checked via load testing. (Are you doing load testing on all your services?) The service owners are also almost certainly not autoscaling experts. And so it’s not surprising to me that a misconfigured autoscaling policy was a contributor here.

But, while I think it’s worth discussing the particular defect with this policy, since it’s good for people to be aware of the risks of autoscaling, I also think it’s too easy to fixate on it to the exclusion of other factors involved in this incident. This is what David Woods refers to as the component substitution fallacy – the idea that the way to improve reliability is to focus efforts on identifying and fixing the defective components.

While, yes, you should identify and fix the defects uncovered by an incident, you should also recognize that:

your system is, at this very moment, filled with latent component defects
despite the presence of all of these defects, your system is not constantly failing over

This means that component defects aren’t enough to take down your system, or your system would be down right now. Don’t just look at the individual components: treat the interactions as first-class. In the GitHub outage, we see discussion of interactions between factors such as: changing traffic patterns (including scrapers), autoscaling policy, the Istio sidecar saturation, retry logic, HAProxy node saturation, and authentication traffic.

There’s also a multitude of details we don’t have because this is a rapidly disseminated public writeup, and the good stuff can only be found in the internal writeup. I speculated in this post about the relationship between service owner and autoscaling policy, but I would love to know more about the history here (did this policy predate the use of Istio sidecars, for example?). I’d also love to know more about the problematic traffic. (What kinds of requests were they? Was it a sudden increase or a gradual ramp-up? Do we know why the traffic increased?).

You can’t get answers to these sorts of questions for public incident writeups, but you can for the internal ones at your own organization. It’s up to you to ask the questions.

Tough days at GitHub, a continuing series

It wasn’t even a week ago when I wrote about a major GitHub incident. Yesterday, they had another big incident, which lasted almost eight hours. There’s a public write-up already posted, which is surprisingly quick. While I’m personally very impatient to read these, I also know that it takes time to collect and synthesize the information you need to do a good job with them. I wish they had posted this as a preliminary write-up and then done a more detailed write-up in a couple of weeks. That being said, let’s look at the write-up!

Saturation strikes yet again

The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic.

The failure mode is yet another example of saturation, a topic I’ve written about again and again on this blog. Heck, I even gave a talk on saturation a month ago.

Here’s the full paragraph on the failure mode:

The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers.

Based on this, it sounds like the failure cascade looked like:

increase in external traffic → istio sidecars saturate (concurrency limits) → HAProxy nodes saturate (flow limits) → authentication requests fail

An increase in load on the system saturated one of the components (istio sidecar), and that propagated to another component (HAProxy), whose saturation broke the auth flow.

I wish they had included an architectural diagram here, that showed the relationship between the load balancers, the service that whose Istio sidecar pod saturated, the gateway, and the services that handle auth requests. Also, the wording gives the impression that only a single sidecar pod that saturated (an Istio sidecar pod), which would be surprising, but I’m also not confident that this is what the authors intended.

Diagnostic details: missing in action

The write-up doesn’t talk about the diagnostic work of the incident responders at all, which is a shame. I can’t tell from this write-up how difficult it was for them to figure out what was happening. There were auth failures, but it doesn’t sound like there was an increase in auth traffic per se, nor was the problem caused by recent changes to the auth system, which is where I would think to look first.

As somebody who was watching the updates to the status page as the incident was happening, I was struck by how they updates alternated between “we have identified the problem” and “we are experiencing issues:

Screen shot of some of the status page updates

I can imagine how frustrating it must have been for the responders to think they had found and fixed the problem, only to continue to see impact.

Retries made things worse

Retries are one of the tools in our toolbox to improve availability. And, usually, retries do improve availability! But retry logic also adds complexity to a system, and adding complexity to a system can introduce new failure modes. In this incident, retries hurt rather than helped, by increasing the load on an overloaded system.

The problem was worsened by optimistic retry logic which overloaded internal load balancers.

…

Residual Copilot authentication failures continued because client retry behavior amplified load: a failed token operation could generate many extra requests and enter a retry loop.

This is a great example of unexpected behavior of a subsystem whose primary purpose was to improve reliability from my conjecture on why reliable systems fail.

The Copilot Token Service sees 10X traffic

Note that there were two independent retry behaviors mentioned in the previous section:

optimistic retry logic against the load balancers
client retry logic against the Copilot Token Service

It turns out that the client retry logic was due to a previously undiscovered bug in Visual Studio Code(!), which led to one particular service (Copilot Token Service) taking longer to recover:

Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.

…

Residual Copilot authentication failures continued because client retry behavior amplified load: a failed token operation could generate many extra requests and enter a retry loop. Copilot Token Service traffic increased from a normal 7–9K RPS to 70–100K RPS.

There’s no way you’re pushing out a VS Code bugfix to mitigate an incident! You’ve got to mitigate that on the server side, which is what the responders did, which brings us to the next section.

Mitigating the incident: multiple strategies

While the write-up doesn’t discuss diagnostic work, it does mention multiple mitigations that the responders undertook during the incident:

shifting traffic from the Central US region to the Northern Virginia region
paused HAProxy on the four saturated nodes
changed gateway retry logic (via PR)
blocked inbound Copilot Token Service token requests at the load balancer (returned 403s)
gradually ramping up blocked traffic

As responders, we are always limited in our ability to intervene based on the tools that we have at our immediate disposal. It’s incredibly useful to be able to do things like selectively block traffic, or dynamically change or even disable a reliability-related subsystem. Think about how difficult it would be to block specific types of requests during an incident in your organization, and to ramp that traffic back up slowly after the system recovers. Note how the responders had to use a pull request to change the behavior of the gateway retry logic. I wonder if the failure mode made this more difficult to carry out, but the write-up doesn’t say.

“Never again” means never preparing for a novel incident

The writeup ends, as most writeups do, with some action items intended to prevent recurrence.

To prevent recurrence, our follow-up actions include:

Correcting autoscaling policies to account for service-mesh sidecar concurrency and capacity.

Auditing Istio request, concurrency, and scaling limits across affected services.

Reviewing retry limits and backoff behavior across gateways and clients.

Addressing the VS Code retry behavior that amplified Copilot token traffic.

Improving load-balancer capacity monitoring and regional failover safeguards.

My eternal lament is that people spend too much of their focus on preventing the last incident from recurring. It’s not that I’m opposed to preventative work. It’s that I also want us to spend time on getting better at dealing with novel incidents. Engineering cycles are a finite resource, and every cycle spent on prevention is a cycle not spent on improving our ability to respond effectively to new incidents. And I promise you, you are going to face novel incidents in the future.

After all, I don’t think GitHub customers who experienced this outage take much solace in knowing that it was a different failure mode from the previous incident.

Quick thoughts on Azure Regional Outage from July 23, ’26

The folks at Microsoft Azure recently wrote up a post incident review for a networking issue in their West U.S region. From the included timeline, it looks like the impact was on the order of five hours. It’s a pretty short write-up, but let’s take a look at the contributors.

On 23 July 2026, a break-fix repair was initiated on an optical device to address a network reliability risk.

The first contributor mentioned in the write-up was work that was done to repair a device in their networking stack. Here I can’t help but think of the first bullet in my conjecture on why reliable systems fail. They made a change to the system in order to fix an ongoing problem, and due to a set of circumstances, things got worse rather than better.

A defect in our blast radius analysis system incorrectly expanded the scope of the repair event to include all optical devices egressing a specific datacenter.

The second contributor mentioned was a (presumably) latent defect in their system. Note the irony of the failure mode here: I suspect this blast radius analysis system usually contributes to reliability, but in this case it hurt reliability by increasing the blast radius.

The safety validation step, which is designed to confirm that at least one of the two redundant datacenter paths remains available, ran but incorrectly concluded the operation was safe.

The third contributor mentioned was a safety check (good!) that passed even though the action was unsafe (bad!).

The checks validated each device individually rather than evaluating the aggregate effect of isolating all devices at once, a scenario that was not accounted for because the system was never designed to process a full datacenter’s worth of devices in a single request.

The reason it failed was due to an interaction with the second contributor: the blast radius being all of the optical devices egressing the datacenter. The designers never envisioned that the check would have to handle the sort of scenario that occurred as a result of the blast radius analysis system defect.

As a result, routes were withdrawn from multiple devices simultaneously, disrupting connectivity between the datacenter and the WAN – therefore impacting traffic entering or leaving the West US region.

It sounds like this change effectively disconnected the West US datacenter from the internet.

Once the route withdrawals took effect at 14:44 UTC, physical links and routing adjacencies continued to appear healthy, which initially masked the correlation between the break-fix activity and the connectivity disruption

Here we have our fourth contributor: the operators were receiving misleading signals from the system. The links and routes looked healthy, even though connectivity was broken.

The impact presented as a WAN routing anomaly, as third-party networks could not reach Azure in the region, rather than as a datacenter connectivity failure.

Our fifth contributor is another flavor of misleading signals. The symptoms presented as a routing issue between Azure and third-parties.

Although all physical work in the region was stopped, our engineers could not correlate to this recent change because the preparation activities in advance of the break-fix did not succeed, so the physical layer and traffic appeared healthy.

This is the sixth contributor mentioned in the writeup. The writing is a little oblique here, but I think what they are saying is that the repair event did not show up in their event log because the repair event didn’t actually complete. It sounds like the preparation activities were the ones that triggered the incident. But, because the repair event didn’t actually happen, the operators looking for events that correlate in time with the onset of the incident didn’t see the triggering event because it didn’t show up in the log of events. That’s my best guess, anyways.

Our automated recovery and rollback system detected the device failures, and attempted multiple retries to restore the affected devices. However, because that system depended on the same datacenter connectivity that had been disrupted, its automated rollback attempts were unsuccessful.

This is the seventh and final contributor mentioned. Azure has an automated recovery and rollback system (good!), but the failure mode in this case prevented automated rollback from succeeding (bad!).

As always, I’d love to know more about how the operators identified what the failure mode actually was, and how they traced it back to the optical device repair work.

GitHub has another tough day

On August 6, 2026, GitHub had a pretty rough incident: GitHub Actions was degraded for about nine hours. GitHub posted a public incident write-up. It’s only a few paragraphs long, but there are some interesting details in here.

This was yet another incident that involved saturation. In fact, the write-up even uses the word saturated when describing what happened.

The incident was triggered by a routine deployment to an internal Actions service responsible for processing events and generating Actions jobs. The deployment exposed an existing capacity and concurrency weakness. As pods were replaced during the deployment, remaining capacity became saturated, causing services to crash and triggering a cascading impact across multiple clusters and downstream services.

We often think of deployments as risky because we are changing the code that’s running in production, and the execution of that new code could trigger a behavior change in the system that could lead to an incident. But a deployment is itself also an operational change in the behavior of the system: our system behaves differently during a deployment than it does when nothing is being deployed. Ironically, this is one of the advantages of deploying more frequently: the more often we are deploying, the more that deployment becomes a normal part of the system behavior – we get more experience with the system in deploying state.

In this particular case, running in the deploying state reduces the number of pods available for doing work, as the older pods go online. In this scenario, it sounds like the system was running close enough to the margin that the reduction in capacity due to the deploy pushed the system over the edge, leading to a cascading failure. This is what the resilience folks call a brittle collapse, which is when the system fails in a non-graceful way when it reaches saturation.

As is common when recovering an overloaded the system, they got it back to healthy by shedding load (throttling) and by increasing capacity.

These services recovered at 17:00 after expanding capacity, throttling incoming webhook-triggered work to allow the system to recover, and increasing processing capacity for the backlog of affected events.

I wish the write-up had more details on what was involved in enabling throttling and getting that additional capacity to come online. In particular, I’m curious about whether this was easy to do or difficult. But, alas, you typically don’t get those kinds of details on public writeups.

And, of course, because every incident involves multiple contributing factors, there was a previously undiscovered bug that made things worse by consuming available capacity trying to run invalid jobs:

Due to a latent bug in one of the services responsible for job assignment, runners were getting assigned jobs that were no longer valid and then getting stuck retrying those jobs, preventing them from picking up valid work.

I would love to know more details about how the heck they figured out what was going on with these stuck jobs. I can just imagine being a responder to this incident, trying to get the backlog of work processed, and discovering that there are workers are blocked trying to execute invalid jobs! How did they figure out they were stuck? How did they figure out this was because of a bug?

They deployed a change to work around this problem, but I would love to know what kind of change that was. Was it a quick workaround to get things moving again? I bet it was, but we’ll never know…

This second stage of impact was mitigated by deploying changes to prevent runners from repeatedly attempting to acquire invalid jobs. These mitigations allowed the accumulated queues to drain and Actions to recover to normal operation.

The write-up ends with the typical “here’s what we’re doing to make sure this doesn’t happen again” text, but I am heartened by the last sentence (emphasis mine):

We are also making additional improvements to reduce the risk of cascading failures and accelerate recovery during large-scale Actions disruptions.

Too often, the focus of reliability work is entirely on prevention. I’m happy to see them also focus on preparing to recover more quickly. Because, as we all know, the next big incident is always just around the corner.

Synthesis is harder than analysis

Over the years, mathematicians, logicians and computer scientists have developed various calculi. If you have a background in computer science, you’ve likely heard of the lambda calculus, a model of computation that was developed by Alonzo Church. If databases are more your thing, then you’ve been exposed to the relational calculus without even knowing it, since SQL is based on the relational calculus. If you are into formal methods, then you’ve worked with the predicate calculus, better known as first-order logic. Finally, if you enjoy reading academic papers on programming languages, you’ve almost certainly run into the sequent calculus. However, when someone says “calculus” without modification (e.g., “I’m taking calculus next semester”), there’s no ambiguity about which calculus they are referring to: it’s always one particular calculus. Or, rather, two calculi that happen to be deeply related to each other: differential calculus and integral calculus.

Visually, you can think of differential calculus as being about calculating the slope of a function at a given point. For example, consider this graph:

You might ask, “how quickly is this curve changing when x=6?” In other words, what is the slope of this function right in a neighborhood very close to x=6?

Differential calculus enables you to compute the slope of a function at a given point

Integral calculus, on the other hand, is about the area under the graph over a particular interval. For example, you might ask “what is the area under this curve between x=2 and x=7?

Integral calculus enables you to compute the area under a function over a given interval

If you study calculus, you’ll first be taught differential calculus (sometimes referred to “Calculus 1” or “Cal 1”) and then you’ll be taught integral calculus (“Cal 2”). When you study differential calculus, you learn the rules for calculating the derivative (slope-at-a-point) of a function. And it turns out that it’s quite straightforward to calculate a derivative, no matter what type of function it is. It’s just an algorithm, which means you can easily program a computer to compute derivatives if you wanted to. (As an aside, automatically computing derivatives is a fundamental element in the process of training LLMs. If you’re curious, look up automatic differentiation).

And then, you get to Cal 2, and you learn about how to compute an integral (area-under-a-curve). You will soon discover that, unlike in Cal 1, there is no algorithm for computing the integral of an arbitrary function. Instead, what you learn is a bag of tricks on how to compute integrals for different kinds of functions. You also learn that for some functions, there’s no closed-form solution at all for the integral! As an example, consider the Gaussian function, which shows up in the normal distribution. With zero mean and unit variance, it looks like this:

\frac{1}{\sqrt{2\pi}} e^{-\frac{x^2}{2}}

Asking students to compute the derivative of this function would be a perfectly reasonable question on a Cal 1 final exam, the answer looks like this:

-\frac{x}{\sqrt{2\pi}}e^{-\frac{x^2}{2}}

But asking students to compute the integral of this function on a Cal 2 final exam would be unfair, because it’s not possible to do with the techniques they learned in class (at least, I didn’t learn the technique you’d need until Cal 3). Because the integral doesn’t have a closed-form solution, you need to express the solution as an infinite series, like:

\frac{1}{\sqrt{2\pi}} \sum_{n=0}^{\infty} \frac{(-1)^n }{2^n n!(2n+1)}x^{2n+1} = \frac{1}{\sqrt{2\pi}} \left( x-\frac{x^3}{6}+\frac{x^5}{40}-\frac{x^7}{336}+\frac{x^9}{3456}-\cdots \right)

(Note: I asked AI for the integral of the Gaussian, I hope it got it right!)

It’s not obvious (at least, not to me) that differential calculus and integral calculus are related to each other. However, it turns out that these two calculi are opposite sides of the same coin, because integrals are anti-derivatives. That is, if f(x) is the derivative of F(x), then F(x) is the integral of f(x). This result is known as the Fundamental Theorem of Calculus.

This connection between differential and integral calculus raises an almost philosophical question: why is it so much easier to compute a derivative than it is to compute an integral? Back in 2011, somebody asked about this on the Mathematics Stack Exchange: Why is integration so much harder than differentiation? The top-voted answer was written by Qiaochu Yuan, and here’s the heart of it (emphasis mine):

Differentiation is a “local” operation: to compute the derivative of a function at a point you only have to know how it behaves in a neighborhood of that point. But integration is a “global” operation: to compute the definite integral of a function in an interval you have to know how it behaves on the entire interval (and to compute the indefinite integral you have to know how it behaves on all intervals). That is a lot of information to summarize. Generally, local things are much easier than global things.

In one sense, local things are easier than global things is a banal statement. Everybody knows that, for example, local optimization is much easier than global optimization. But it’s also a very deep one. And it gets at the title of this post, which is synthesis is harder than analysis.

I previously wrote about the difference between analysis and synthesis in the demon of the gaps. In analysis, we’re breaking a larger problem into smaller problems that separate out cleanly. These smaller problems are more localized, and hence easier to solve. This is why we advocate for principles like encapsulation and separation of concerns, to ensure our smaller problems are local.

The work of synthesis involves integrating(!) multiple things together. This pushes in the other direction: we are creating a problem that is less local. And global things are much harder than local things. The challenge we face is that some kinds of problems are just inherently synthesis problems. As I wrote in that previous post, incident response is one area where we are frequently confronted with synthesis problems: we have to understand how the pieces normally fit together in order to make sense of what is currently going wrong.

That’s why I think that this sort of synthesis work is important for SREs. Now, because synthesis is harder than analysis, and because SREs don’t have super-human cognitive abilities, it means that there is a limit to how deeply they will be able to understand any given component in the system. But the more they understand how the different components interact, the better positioned they are for helping resolve the tougher incidents.

Unfortunately, in our industry we haven’t recognized building up synthesis expertise as a first-class thing. That’s understandable because this work is very situated, it depends on the messy details of the particular system in the organization that an SRE works in. On the other hand, we can get better at learning how to learn about the operational details of a system. And that’s what I’d like to see more of.

I am dreading our LLM-written incident report future

The other day, Reginald Braithwaite posted the following toot. For posterity, I’ve also included my own response to it:

Screenshot of a Mastodon toot by Reginald Braithwaite (@raganwald) and a response by Lorin Hochstein (@norootcause).

Reginald: Writing incident reports is a time-consuming process that produces a document nobody in the org has any incentive to read. Interested in solving this problem?

Join our incredibly journey building an AI Ops tool that writes incident reports for AI to read and act upon. And it will summarize the reports so that busy humans don't have to read about every minute detail.

This will be a game-changer. Find us on LinkedIn, Substack, and X.

Lorin: I hate you

Braithwaite’s post is dripping with sarcasm, but make no mistake, incident reports written entirely by LLMs is coming. And I am not looking forward to this future.

Before I dive in here, I want to note that there is a lot of toil you need to do in order to gather the data you need to write a good incident report, and LLMs can help significantly reduce that toil. I’ve got no issues there. But there’s a world of difference between using LLMs to help you assemble the ingredients involved in writing an incident report, and using an LLM to actually write the report itself.

Braithwaite’s post is horrifying to me precisely because of the seduction of the LLM as a tool for generating an incident report. After all, you can just ask it to write the report, and it’ll do it. And that’s exactly what scares me.

There’s a famous quote by the cartoonist Dick Guindon: “Writing is Nature’s way of showing you how sloppy your thinking is“. You might think you understand a concept, but it’s only when you put metaphorical pen to paper, when you actually try to explain the concept in written words to a potential reader, that you realize how fuzzy your understanding actually is. Writing in your own words forces you to confront how much you actually understand what it is that you’re writing about. Or, as Leslie Lamport put it, “If you’re thinking without writing, you only think you’re thinking.”

Having an LLM generate the text of an incident write-up bypasses this thinking step. Now there’s no human in the loop of the writing process that has to confront whether the explanation is actually consistent with the evidence that they’ve gathered. Instead, what you get is a plausible explanation of what happened to someone who is not intimately familiar with the details. They might read, nod along, and think, “yes, that makes sense.” But the LLM may have invented couplings between systems that aren’t there, and may miss critical interactions that were actually part of the incident, and because nobody did the hard work of actually synthesizing the data to do the write-up, nobody will notice. Because if you’re trying to reduce the writing effort, how much effort are you really going to put into checking the LLMs work.

In my view, LLM-generated incident write-ups are more dangerous than using LLM for coding or for AI SRE style tasks. For coding tasks, there’s always a testing step to check that the code exhibits the desired behavior, even if nobody looks at the code itself for meaningful details. For AI SRE tasks, either the LLM output helps you resolve the incident, or it doesn’t. In both cases, Nature is the ultimate arbiter of the LLM output.

But incident write-ups aren’t like that. The consequences of a poor report aren’t immediately apparent the way incorrect code or an incorrect operational diagnosis are in the moment. Instead, we get incident reports that have the superficially correct form, but are actually incorrect, with no obvious test for correctness.

And, because incident reports are time-consuming to write, the temptation to use AI tools to generate them will be overwhelming. But these LLMs will not go around talking to people that were involved in the incident. These reports will be simulacra; they will have the right form, but they will not provide readers with genuine insights into the nature of the system. The amount of learning will be significantly curtailed.

And, yes, people will probably use AI to summarize them as well.

It’s not a future I’m looking forward to.

Dear researchers column

The Journal of System and Software publishes a regular column called Dear Researchers: The perspective of software practitioners. Each column is an open letter to the software engineering research community from someone who works in tech. It’s edited by Austin Henley and Olaf Zimmermann, both of whom have experience in the two worlds of academia and industry.

They invited me to submit a column, which I did. When it finally gets published, you’ll be able to find it here: Dear researchers: help me deal with incidents! The published version will eventually go behind the journal’s paywall, but here’s a preprint of the column that you can always read free of charge.

The demon of the gaps

Mephistopheles (a medieval demon from German folklore) flying over Wittenberg, in a lithograph by Eugène Delacroix.

Modern software systems contain within them a mind-boggling level of complexity. As software engineers, we make this complexity manageable through techniques like decomposition, information hiding, and abstraction. We endeavor to break our systems up into components that interact over well-defined interfaces. By doing this, the surface exposed to individual software engineers is dramatically reduced: no individual has to understand how the entire complex system works in order to contribute to their system. Instead, each software engineer needs to understand only the individual component that they work on, along with the interfaces of the other components that they interact with. Decomposition is synonymous with analysis, where you study a larger thing by breaking it up into smaller pieces that are more amenable to understanding.

You can see this strategy of complexity management in action in microservice architectures. An engineer needs to understand the service that their team owns, and the interfaces of the services that their team calls out to. This architecture effectively bounds the information that an engineer needs in order to work effectively. Microservice architectures aren’t there for scaling the software itself, they’re there for scaling the software organization.

Unfortunately, when the system breaks down, this complexity management strategy breaks down itself. Just as hurricanes don’t respect political boundaries, system failures don’t respect component boundaries. Yes, sometimes the problem in a software system is limited to the failure of a single component. Those are the easiest cases to diagnose and mitigate. However, the hairy incidents are the ones that arise due to unexpected interactions across components. Maybe you have several services that are throwing errors, or maybe none of the services are throwing errors but customers are still seeing incorrect behavior. There’s no obvious change that correlates with the start of impact, or maybe you don’t even know when the impact started because the customer impact isn’t reflected in your existing metrics.

When you’re in the throes of an incident that involves an unexpected interaction, this architecture that was built for managing complexity now works against you. Because you’ve built an analysis solution but you’re now faced with a synthesis problem. You need to understand how the pieces all normally fit together to function in order to determine what is going wrong with the system right now. You’ve optimized to avoid requiring anybody to understand how the whole thing works, but now the whole thing isn’t working, and no one person knows how the whole thing works.

The job of the incident responders is to collectively figure out how to do that synthesis. You’ve brought together a group of people who each understand the functions of different components of the system, and you need to work together to build enough of an understanding of how the system functions to debug what’s going wrong. As an ad hoc team, the incident responders have to move up and down the abstraction hierarchy to figure this out.

This sort of in-the-moment reconstruction of system function from component parts is an essential part of incident response for the most complex incidents, but it’s rarely treated as first-class work that’s worthy of study and support. The recent book Crisis Engineering by Marina Nitze, Matthew Weaver, and Mikey Dickerson is the exception that proves the rule: they do discuss the work of building a model of the system during a crisis to help figure out what’s gone wrong. But I struggle to recall any other guidance I’ve read about incident response that talks about how to prepare for doing this sort of work. It’s important work, and it’s difficult, and the ability to do it well can have a huge impact on the time it takes to mitigate the hardest incidents. This is stuff that even the best individual humans struggle with, because it involves a group of humans working together effectively, with each person having a partial model of the system. And if the best humans struggle with it, I don’t think AI SRE tools are going to save us here: if the best humans struggle, the AIs will too. We need to figure out how to get better at this collectively. Like so many things, it’s a coordination problem.

Reliability as a game of improving the odds

I’m a betting man; I just enjoy making bets, even when there are no stakes at all.

And when you talk about bets, you end up talking about odds.

It turns out that reliability is also about odds, even though we don’t use the language of odds in our domain. Consider how we talk about availability. We report system availability as a number of nines: for example, we might say “four nines of availability”, which means 99.99% of somethings are good over some time interval. The canonical example of those somethings are successful requests. In that case, if someone says a service has four nines of availability over the past three months, that means that 99.99% of requests succeeded over that time period. We could express the same information by saying that there is a one in ten-thousand chance that any given request failed in the past three months.

If your system has exhibited four nines of availability in the past three months, and you assume that the availability of your system in the near future will be like the availability of the past (a dangerous and unwarranted assumption, but let’s go with it for a moment), then we could also express this information using the language of odds, by stating that the odds of a request failing are ten-thousand to one.

But this isn’t a post about describing availability in the language of odds. Instead, what I want to talk about is how all reliability work is inherently about improving the odds, increasing the likelihood that the system stays up. Any time we build any sort of reliability mechanism, be it load shedding, autoscaling, canarying, staged deployments, automated rollbacks, or what have you, we are building automation into the system that either eliminates or reduces the impact a subset of potential problems. If you ask an engineer working on improving reliability, “will this prevent all future incidents”, they will tell you “no, of course not”.

However, we don’t explicitly think of reliability work in terms of improving the odds. Instead, we tend to think of it as deterministically addressing a specific class of problem. You’ll hear questions like, “how many historical incidents would this tech have prevented?” in trying to determine whether engineering should invest in a particular reliability solution. They are looking for an answer like, “this would have prevented 20% of our SEV1s and SEV0s”. This 20% isn’t interpreted as a likelihood, instead it’s used as an estimate of impact, as in “this will improve our availability by around 20%”. The idea is that this reliability work will deterministically eliminate or mitigate a certain fraction of incidents; we just don’t know exactly what that fraction is, so we estimate it from historical data.

What I would like to propose in this post is that we think about all of the various kinds of reliability work as improving the odds of our system being up longer, instead of assuming that reliability work will have a fixed effect, and try to estimate the effect size. I’ve got two motivations for taking this perspective of reliability work as odds improvement.

The first motivation is that I don’t think we can ever estimate the effect size without error bars that are so huge that the estimates are themselves meaningless. As I’ve written about previously, the variation in incidents is just too large relative to the amount of data we have available. And, to make the estimation problem from historical data even worse, our system is changing over time. Or, to put it in technical terms, I don’t believe that incidents can be modeled as a stationary process. (Heck, if they were stationary, then that means that reliability work could not have an impact, because then the process would change over time!). Note that I’ve never seen anybody try to validate the estimates, they’re always point-in-time estimates used to justify work, and then promptly forgotten about. In one sense, that’s fine, they served their purpose of convincing leadership that we should allocate cycles for a particular kind of reliability work. But we shouldn’t fool ourselves into believing that these estimates are meaningful: they’re for persuasion, not insight.

It’s my second motivation, though, that prompted me to write this blog post. And that’s because the idea of reliability work as improving the odds of effectively mitigating future incidents is a useful framework for thinking about work that improves resilience. I’m interested in improving the skills of the people who respond to incidents, putting them in a better position to deal with those future unforeseen, surprising scenarios. One way to do this is learning from how responders dealt with previous incidents, the different sorts of observability data they had access to and how, the different knobs that were able to turn, and so on. While the next incidents will be different, the set of tools that are available during incident response are generally the same. There’s no way I can give a quantitative of estimate how this sort of skill improvement work will impact reliability. And despite the enormous number of random factors, I am confident that it will improve our odds.