Category: Uncategorized

Omnipresent availability risks in cloud software

I’m using this post to gather together some common threads I’ve noticed after reading write-ups of major cloud software incidents. By cloud software, I’m referring to software-as-a-service (do people even say that anymore? in the cloud. This doesn’t just apply to cloud providers, although it does apply to them as well.

Here’s an outline of the topics in this post:

  • problem areas
    • saturation
      • example: databases
    • networking (traffic routing failure)
      • example: DNS
    • security (deny valid access)
      • example: SSL certificates
  • essential non-standard changes
    • mitigating an operational issue
    • migration
  • essential increase in essential complexity
    • reliability subsystem
    • migration

I think of all of these as omnipresent availability risks: I think these are fundamentally unavoidable, and will be contributing to software incidents until the end of time; or, at the very least, until the end of my own career in software.

There are three general areas that most major incidents seem to fall into: saturation, networking, and security. So, let’s start with those.

Saturation

Saturation is probably the topic I talk about most frequently, both on this blog and elsewhere (e.g.,: the saturation post I wrote for the Resilience in Software Foundation and my saturation talk at the Software Should Work conference). The system becomes saturated when it reaches a limit. That’s a pretty generic description, but there are many limits!

Databases

Many major incidents involve some system component becoming saturated in one form or another. I personally worry about database saturation the most. That’s because it’s difficult to recover from an overloaded production database. In addition, because database systems are such complex beasts, it can be quite difficult to even determine what the specific performance issue actually is. This is why having in-house database operational expertise is critical.

Saturation is an omnipresent risk because the finite nature of resources is a hard constraint in the world that we live in. Eventually, some resource in your system is going to run out.

Example: GitHub Incident, Aug 26, 2026

Networking

While I prattle on endlessly about saturation, not every major incident involves saturation. You can encounter scenarios where all of your internal subsystems are reporting healthy, but from your customer’s point of view, your site is down: they can’t use it. One way this can happen is if your users can’t even reach your site, and that’s where the networking problem area comes in.

A networking problem can lead to packets being misrouted. These requests might be black-holed (i.e., silently dropped), or they might be incorrectly routed to a service that doesn’t have the capacity to respond to all of these requests, in which case you’ve got both a network routing issue and a saturation issue.

A visual depiction of an actual black hole. Image source: NASA

DNS

DNS issues are an example of this kind of network-related failure mode. There’s no way those packets are going to make it to their destination if the client can’t even determine which IP address to send them to. And when DNS breaks, that’s what happens.

I bring DNS up because it’s bitten folks enough times that there’s a famous haiku:

More generally, networking is an omnipresent risk because cloud software is inherently distributed, so networking is always a critical service. Now, I don’t work in networking, but from the outside, networking just feels like a dangerous domain to do operational stuff in. The blast radius of a networking issue can be very large. And, because network behavior is inherently distributed, reasoning about the behavior of operational changes is just inherently difficult. Honestly, that’s probably why I don’t work in networking.

And, so, I predict we’ll continue to see networking issues contribute to large-scale incidents.

Example: Buildkite incident, Aug 25, 2026

Security

There’s a fundamental tradeoff between availability and security: availability is about ensuring that the good people can access the system. Security is about ensuring that the bad people cannot access the system. This means that there’s always a risk that a security system designed to prevent bad actors from accessing the system can lead to good actors also being blocked. Consider this scenario: there’s an internal security subsystem that goes unhealthy (possibly due to saturation). Is your policy to fail closed or fail open in the event that this subsystem is erroring? Answering that requires making an availability-security tradeoff.

SSL certificate expiration

Another example of this failure mode, which keeps biting our industry again and again, is SSL certificate expiration. Here you have the behavior of a security system that is preventing legitimate access because the cert wasn’t renewed.

Bazel expired certificate
Even the mighty Google encounters SSL certificate expirations. This is from the Bazel incident

And so, my claim is availability incidents that involve security subsystems will continue to be a thing forever.

Example: Bazel incident, Sep 27, 2025

Essential uncommon changes

Your system is constantly undergoing change. Heck, if you stopped making changes, the system would eventually stop working properly. Now, there are some changes that your org does very frequently. Hopefully, you’re deploying often, flipping feature flags a lot, and so on. But there are other changes that your org has less experience with, because they happen much less often. That means that there hasn’t been as much investment in tooling to support these sorts of changes, and it means that the people making these changes don’t have the same level of expertise as they do with the more common changes. That makes these sorts of changes more dangerous: less mature tooling and less experienced humans.

Mitigating an operational issue

A few years ago, I wrote a post titled a conjecture on why reliable systems fail where I speculated on two common contributors to major incidents. One of those contributors was a manual intervention that was intended to mitigate a minor incident. Now, it may be that you frequently have to do manual interventions to mitigate system issues, in which case you’ll have a lot of experience with those sorts of interventions. But you’ll also be more motivated to put in the engineering effort to automate away those sorts of common issues.

It’s exactly the uncommon issues that require a human operator to intervene to mitigate that are dangerous, because they are uncommon. But they’re essential: there’s a problem in the system, and you need to fix it! But because all practitioner actions are gambles, the manual mitigation carries risk that you could make the problem even worse. And, eventually, this will happen to you.

Example: Azure Regional Outage, Jul 23, 2026

Migration

If you’re at a tech company, unless it’s a start-up, you’ll be dealing with migrations, as old tech gets replaced by newer tech that is better suited to the problems that your org is currently facing. While migrations as a general category are extremely common, each migration is itself a snowflake. This means that the specific details of the migration work is an uncommon change. The work of migration involves making a kind of change to your system that you haven’t made before.

To make things worse, one of the dangers of migration is that, as you go along, you start to build confidence that your changes are safe, but there are actually hidden dangers lurking in the system for the next migration. The confidence in the safety of the work exceeds the actual safety. I mean, you made n-1 changes as part of the migration, and none of those changes had negative consequences. It’s natural to assume that the same outcome will occur with the nth change.

Example: Rogers Network outage, Jul 8, 2022

Essential increase in essential complexity

The late American computer scientist Fred Brooks wrote a famous software engineering essay titled No Silver Bullet where he drew a distinction between accidental complexity and essential complexity. The general idea was that there was some amount of complexity in a software system that didn’t need to be there (accidental complexity) and some amount that was just inherent to the nature of the problem space and solution space and so could not be removed (essential complexity).

Reliability subsystem

We’ve developed multiple techniques to improve the reliability of software systems, including retries, concurrency limiting, autoscaling, automated failover, circuit breakers, health checks, canaries, outlier detection, the list goes on and on. There’s one thing that all of these techniques have in common: they increase the complexity of the overall system! And they do this because they have to increase complexity in order to do their job. This is a consequence of Ashby’s Law, which states that if you want to build a control system that handles more scenarios, you have to increase the complexity of the controller itself.

This means that reliability subsystems result in a complexity trade-off. On the one hand, our system can now automatically recover from failure modes that previously required manual intervention. On the other hand, as we all know, increase in complexity is itself dangerous because it can introduce entirely new failure modes that weren’t there before.

Going back to my conjecture blog post, the second contributor I posited was: unexpected behavior of a subsystem whose primary purpose was to improve reliability. And this is exactly why. Adding reliability subsystems improves the robustness of our system, but it adds essential complexity to our system, which can lead to novel incidents.

Example: OpenAI incident, Dec 11, 2024

Migration

Like all engineers, I’m a big fan of giving the answer “it depends” if somebody asks me a question about whether they should do X or Y. However, if someone came up to me and said, “Lorin, I’m preparing to do a migration at my company, and I’m trying to decide whether to do a big-bang migration or an incremental one”, then I would almost certainly say, “For the love of God, please do an incremental migration!”. Sometimes big-bang migrations are unavoidable, but when given a choice, I’m going to go for the incremental migration as the safer option.

However, when you do an incremental migration, it means that you need to simultaneously support the old system and the new system at the same time while you’re doing the migration. This means that even if the new system yields a net decrease in overall complexity over the old system, while the migration is happening, you’re going to see an increase in system complexity. And that means that you’ll see incidents arise as a byproduct of this increased complexity.

Example: Cloudflare incident, Jul 14, 2025

Incidents are inevitable, so you’d better be ready

To reiterate, I think all of the risks mentioned here are omnipresent: they are fundamental to the nature of cloud software. I don’t think that any of these risks can be eliminated. That’s why I believe so strongly in the value of getting better at incident response. Because, if you prepare, you can get better at dealing with problems that arise as a result of these risks.

My talk from the Software Should Work conference

There’s a new software reliability conference that just spun up called Software Should Work, and I had a chance to give a talk there. I took the opportunity to speak about one of my favorite topics: saturation. Here’s a recording of it.

Links from the talk

I can’t bear to read AI-generated prose

I can’t bring myself to read text if I believe it to be AI-generated.

Now, I ask LLMs questions all of the time, and I do read those answers. I frequently use tools like ChatGPT and Claude as replacements for Google for answering specific questions; that’s not what I’m talking about here. I’m also not talking about reading LLM-generated code. What I mean is, if I’m reading some sort of a document, if I suspect that the document was AI-generated, my motivation to read through it drops down to approximately zero. If I was browsing non-fiction books in a bookstore, and a book was marked as having been AI-generated, I wouldn’t pick it up.

Being honest with myself, I think this point of view is irrational. My personal primary goal for reading any sort of non-fiction document is to advance my understanding of a topic, or put new ideas into my head. In principle, it shouldn’t matter whether the words on the page were emerged from the thoughts of a human being or via an autoregressive stochastic process. In addition, I’m very far from being a perfect detector of AI-generated text, so there isn’t even a way for me to know whether a particular document I’m reading came from a human or a machine. Also, AI generation is a spectrum. No document is completely AI generated: they all start with a prompt was written by a human. Some texts will have been iteratively generated by a collaboration of human and AI. If I knew that an author had used AI as a copyeditor, or to tighten up some of their sentences, that wouldn’t bother me at all. There’s not some magical threshold in my head about how much AI assistance I would consider to be OK, nor would it ever be possible for me to know whether that threshold was exceeded unless the author explicitly told me.

And yet, despite knowing this, I’m just turned off by reading anything that strikes me as being AI-generated. If I’m asked to read a design document, and I suspect the doc was written by AI, I need to fight myself to actually get through it. I feel like a writer should always spend more time generating a document than a reader should spend consuming it, and asking me to spend more time on understanding something that someone else didn’t put the effort into writing feels like a violation of an implicit contract.

As I said, I think this is an irrational response. And I expect the quality of LLM writing to continue to improve over time, so that we stop referring to it as slop. But there’s just something, well, soulless about the idea of writing generated by a machine.

References from my SREcon talk on stories

This past week at SREcon 2026 Americas, I gave a plenary talk titled The Power of Stories. I referenced several books and papers in that talk, which are linked below.

Books

From Novice to Expert: Excellence and Power in Clinical Nursing Practice by Patricia Benner. Benner used narrative vignettes in her research to illustrate the different skills of expert nurses.

Visual Explanations: Images and Quantities, Evidence and Narrative by Edward Tufte. This book contains Tufte’s criticism of slides used by NASA engineers to support the Challenger launch decision.

The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA by Diane Vaughan. This book describes Vaughan’s findings from studying NASA’s engineering culture in the wake of Challenger disaster.

Storycraft: The Complete Guide to Writing Narrative Nonfiction by Jack Hart. Advice on writing journalism pieces in a narrative style.

Storyworthy by Matthew Dicks. Advice on how to improve your storytelling.

Papers

When Do Stories Work? Evidence and Illustration in the Social Sciences by Andrew Gelman and Thomas Basbøll. Gelman and Basbøll lay out a set of criteria for what makes a good story from a social science perspective.

Two Years Before the Mast: Learning How to Learn about Patient Safety by Richard Cook. Some observations about learning from safety accidents.

An Investigation of the Therac-25 Accidents by Nancy Leveson and Clark Turner. A detailed account of the Therac-25 radiation therapy overdose accidents, including software implementation details and operator interface details.

A Tale of Two Stories: Contrasting Views of Patient Safety by Richard Cook, David Woods, and Charlotte Miller. Report on a workshop about second stories in patient safety.

Report of the Presidential Comission on the Space Shuttle Challenger Accident, Volume 2: Appendix F – Personal Observations on Reliability of Shuttle by Richard Feynman. Feynman’s observations on the difference in risk estimates between NASA engineers and leadership.

On variability

I was listening to Todd Conklin’s Pre-Accident Investigation Podcast the other day, to the episode titled When Normal Variability Breaks: The ReDonda Story. The name ReDonda in the title refers to ReDonda Vaught, an American registered nurse. In 2017, she was working at the Vanderbilt University Medical Center in Nashville when she unintentionally administered the wrong drug to a patient under her care, a patient who later died. Vaught was fired, then convicted by the state of Tennessee for criminally negligent homicide and abuse of an impaired adult. It’s a terrifying story, really a modern tale of witch-burning, but it’s not what this post is about. Instead, I want to home in a term from the podcast title: normal variability.

In the context of the field of safety, the term variability refers to how human performance is, well, variable. We don’t always do the work the exact same way. This variation happens between humans, where different people will do work in different ways. And the variation also happens within humans, the same person will perform a task differently over time. The sources of variation in human performance are themselves varied: level of experience, external pressures being faced by the person, number of hours of sleep the night before, and so on.

In the old view of safety, there is an explicitly safe way to perform the work, as specified in documented procedures. Follow the procedures, and incidents won’t happen. In the software world, these procedures might be: write unit tests for new code, have the change reviewed by a peer, run end-to-end tests in staging, and so on. Under this view of the world, variability is necessarily a bad thing. Since variability means people do work differently, and since safety requires doing work the proscribed way, human variability is a source of incidents. Traditional automation doesn’t have this variability problem: it always does the work the same way. Hence you get the old joke:

The factory of the future will have only two employees: a man and a dog. The man will be there to feed the dog. The dog will be there to keep the man from touching the equipment.

In the new view of safety, normal variability is viewed as an asset rather than a liability. In this view, the documented procedures for doing the work are always inadequate, they can never capture all of the messy details of real work. It is the human ability to adapt, to change the way that they do the work based on circumstances, that creates safety. That’s why you’ll hear resilience engineering folks use the (positive) term adaptive capacity rather than the (more neutral) human variability, to emphasize that human variability is, quite literally, adaptive. This is why tech companies still staff on-call rotations even though they have complex automation that is supposed to keep things up and running. It’s because the automation can never handle all of the cases that the universe will throw at it. Even sophisticated automation always eventually proves too rigid to be able to handle some particular circumstance that was never foreseen by the designers. This is the perfect-storm, weird-edge-case stuff that post-incident write-ups are made of.

This, again, brings us back to AI.

My own field of software development is being roiled by the adoption of AI-based coding tools like Anthropic’s Claude Code, OpenAI’s Codex, and Google’s Gemini Code Assist. These AI tools are rapidly changing the way that software is being developed, and you can read many blog posts of early adopters who are describing their experiences using these new tools. Just this week, there was a big drop in the market value of multiple software companies; I’ve already seen references to the beginning of the SaaS-Pocalypse, the idea being that companies will write bespoke tools using AI rather than purchasing software from vendors. The field of software development has seen a lot of change in terms of tooling in my own career, but one thing that is genuinely different about these AI-based tools is that they are inherently non-deterministic. You interact with these tools by prompting them, but the same prompt yields different results.

Non-determinism in software development tools is seen as a bad thing. The classic example of non-determinism-as-bad is flaky tests. A flaky test is non-deterministic: the same input may lead to a pass or a fail. Nobody wants non-determinism like this in our test suite. On the build side of things, we hope that our compiler emits the same instructions given the same source file and arguments. There’s even a whole movement around reproducible builds, the goal of which is to stamp out all of the non-determinism in the process of producing binaries from the original source code, where the ideal is achieving bit-for-bit identical binaries. Unsurprisingly, then, the non-determinism of the current breed of AI coding tools is seen as a problem. Here’s a quote from a recent article in the Wall Street Journal by Chip Cutter and Sebastian Herrera: Here’s Where AI Is Tearing Through Corporate America:

Satheesh Ravala is chief technology officer of Candescent, which makes digital technology used by banks and credit unions. He has fielded questions from employees about what innovations like Anthropic’s new features mean for the company, and responded by telling them banks rely on the company for software that does exactly what it’s supposed to every time—something AI struggles with. 

“If I want to transfer $10,” he said, “it better be $10 not $9.99.”

I believe the AI coding tools are only going to improve with time, though I don’t feel confident in predicting whether future improvements will be orders-of-magnitude or merely incremental. What I do feel confident in predicting is that the non-determinism in these tools isn’t going away.

At their heart, these tools are sophisticated statistical models: they are prediction machines. When you’re chatting with one, it is predicting the next word to say, and then it feeds back the entire conversation so far, predicts the next word to say again, and so on. Because they are statistical models, there is some probability distribution of next word to predict. You could build the system to always choose the most likely word to say next. Statistical models aren’t just an AI thing, and many statistical models do use such a maximum likelihood approach. But that’s not what LLMs do in general. Instead, there’s some randomness that is intentionally injected into the system so that it doesn’t always just pick the most likely next word, but instead does a biased random selection of the next word, based on the statistical model of what’s most likely to come next, and based on a parameter called temperature, drawing an analogy to physics. If the temperature is zero, then the system always outputs the most likely next word. The higher the temperature, the more random the selection is.

What’s fascinating to me about this is the deliberate injection of randomness improved the output of the models, as judged qualitatively by humans. In other words, increasing the variability of the system improved outcomes.

Now, these LLMs haven’t achieved the level of adaptability that humans possess, though they can certainly perform some impressive cognitive tasks. I wouldn’t say they have adaptive capacity, and I firmly believe that humans will still need to be on-call for software system for the remainder of my career, despite the proliferation of AI SRE solutions. But what I am saying instead is that the ability of LLMs to perform cognitive tasks well depends upon them being able to leverage variability. And my prediction is that this dependence on variability isn’t going to go away. LLMs will get better, and they might even get much better, but I don’t think they’ll ever be deterministic. I think variability is an essential ingredient for a system to be able to perform these sorts of complex cognitive tasks.

The illegible nature of software development talent

Here’s another blog post on gathering some common threads from reading recent posts. Today’s topic is about the unassuming nature of talented software engineers.

The first thread was a tweet by Mitchell Hashimoto about how his best former colleagues are ones where you would have no signal about their skills based on their online activities or their working hours.

The second thread was a blog post written a week later by Nikunj Kothari titled The Quiet Ones: Working within the seams. In this post, Kothari wasn’t writing about a specific engineer per se, but rather a type of engineer, one whose contributions aren’t captured by the organization’s performance rubric (emphasis mine):

They don’t hit your L5 requirements because they’re doing L3 and L7 work simultaneously. Fixing the deploy pipeline while mentoring juniors. Answering customer emails while rebuilding core systems. They can’t be ranked because they do what nobody thought to measure.

The third thread was a LinkedIn post written yesterday by Gergly Orosz (emphasis mine).

One of the best staff-level engineers I worked with is on the market.
…
What you need to know about this person: every team he’s ever worked on, he did standout work, in every situation. He got stuff done with high quality, helped others, is not argumentative but is firm in holding up common sense and practicality, and is very curious and humble to top all of this off.
…
And still, from the outside, this engineer is near completely invisible.

He has no social media footprint. His LinkedIn lists his companies he worked at, and nothing else: no technologies, no projects, nothing. His GitHub is empty for the last 5 years, and has perhaps a dozen commits throughout the last 10.

That reason that Mitchell Hashimoto, NIkunj Kothari, and Gergly Orosz were able to identify these talented colleagues as because they worked directly with them. People making hiring decisions don’t have that luxury. For promotions, there are organizational constraints that push organizations to define a formal process with explicit criteria.

For both hiring and promotion, decision-makers have a legibility problem. This problem will inevitability lead to a focus on details that are easier to observe directly precisely because they are easier to observe directly. This is how fields like graphology and phrenology come about. But just because we can directly observe someone’s handwriting or the shapes of the bumps on their head doesn’t mean that those are effective techniques for learning something about that person’s personality.

I think it’s unlikely the industry will get much better at identifying and evaluating candidates anytime soon. And so I’m sure we’ll continue to see posts about the importance of your LinkedIn profile, or your GitHub, or your passion project. But you neglect at your peril the engineers who are working nine-to-five days at boring companies.

The trap of tech that’s great in the small but not in the large

There are software technologies that work really well in-the-small, but they don’t scale up well. The challenge here is that the problem size grows incrementally, and migrating off of them requires significant effort, and so locally it makes sense it to keep using them, but then you reach a point where you’re well into the size where they are a liability rather than an asset. Here are some examples.

Shell scripts

Shell scripts are fantastic in the small: throughout my career, I’ve written hundreds and hundreds of bash scripts that are twenty lines are less, typically closer than to ten, frequently less than even five lines. But, as soon as I need to write an if statement, that’s a sign to me that I should probably write it in something like Python instead. Fortunately, I’ve rarely encountered large shell scripts in the wild these days, with DevStack being a notable exception.

Makefiles

I love using makefiles as simple task runners. In fact, I regularly use just, which is like an even simpler version of make, and has similar syntax. And I’ve seen makefiles used to good effect for building simple Go programs.

But there’s a reason technologies like Maven, Gradle, and Bazel emerged, and it’s because large-scale makefiles are an absolute nightmare. Someone even wrote a paper called Recursive Make Considered Harmful.

YAML

I’m not a YAML hater, I actually like it for configuration files that are reasonably sized, where “reasonably sized” means something like “30 lines or fewer”. I appreciate support for things like comments and not having to quote strings.

However, given how much of software operations runs on YAML these days, I’ve been burned too many times by having to edit very large YAML files. What’s human-readable in the small isn’t human-readable is the large.

Spreadsheets

The business world runs on spreadsheets: they are the biggest end-user programming success story in human history. Unfortunately, spreadsheets sometimes evolve into being de facto databases, which is terrifying. The leap required to move from using a spreadsheet as your system of record to a database is huge, which explains why this happens so often.

Markdown

I’m a big fan of Markdown, but I’ve never tried to write an entire book with it. I’m going to outsource this example to Hillel Wayne, see his post Why I prefer rST to markdown: I will never stop dying on this hill.