In service of the narrative

The most important part of an operational surprise writeup is the narrative description. That section of the writeup tells the story of how the surprise unfolded over time, taking into account the perspectives of the different people who were involved. If you want your readers to learn about how work is done in your organization, you need to write effective narrative descriptions.

Narrative descriptions need to be engaging. The best ones are vivid and textured: they may be quite long, but they keep people reading until the end. A writeup with a boring narrative has no value, because nobody will read through it.

Writing engaging narrative descriptions is hard. Writing is a skill, and like all skills, the only way to get better is through practice. That being said, there are some strategies that I try to keep in mind to make my narrative descriptions more effective. In this blog post, I cover a few of them.

Goal is learning, not truth or completeness

At a high level, it’s important to keep in mind what you’re trying to achieve with your writeup. I’m interested in maximizing how much the reader will learn from the writeup. That goal should drive decisions you make on what to include and how to word things.

I’m not trying to get at the truth, because the truth is inaccessible. I’ll never know what really happened, and that’s ok, because my goal of learning doesn’t require perfect knowledge of the history of the world.

I’m also not trying to be complete; I don’t try to convey every single bit of data in the narrative that I’ve been able to capture in an investigation. For example, I don’t include every single exchange of a chat conversation in a narrative.

Because of my academic background, this is an instinct I have to fight: academics tend towards being as complete as possible in writing things up. However, including inappropriate level of detail makes the narrative harder to read.

I do include a “raw timeline” section in the appendix with lots of low level events that have been captured (chat transcripts, metrics data, times of when relevant production changes happened). These details don’t all make it into the narrative description, but they’re available if the reader wants to consult them.

Treat the people involved like people

Effective fiction authors create characters that you can empathize with. They convey what the characters see, what they feel, what they have experienced, what motivates them. If a character in a movie or a novel makes a decision that doesn’t seem to make sense to us, we get frustrated. We consider that lousy writing.

In a narrative description, you have to describe actions taken by people. These aren’t fictional characters, they are real people; they are the colleagues that you work alongside every day. However, like the characters in a good piece of fiction, your colleagues also make decisions based on what they see, what they feel, what they have experienced, and what motivates them.

The narrative must answer this question for the reader: How did it make sense for the people involved to come to their conclusions and take their actions? In order for your reader to learn this, you need to convey details such as what they were seeing, what they were thinking, what they knew and what they did not know. You want to try to tell the part of the narrative that describes their actions from their perspective.

One of the challenges is that you won’t have easy access to these details. That’s why an important precursor to doing a writeup is to talk with the people involved to try to get as much information as you can about how the world looked from their eyes as events were unfolding. Doing that well is too big a topic for this post.

Start with some background

I try never to start my narratives with “An alert fired for …”. There’s always a history behind the contributing factors that enabled the surprise. For the purposes of the writeup, that means starting the narrative further back in time, to tell the reader some of the relevant history.

You won’t be able to describe the historical information with the same level of vividness as the unfolding events, because it happened much further back in time, and the tempo of this part of the narrative is different from the part that describes the unfolding events. But that’s ok.

It’s also useful to provide additional context about how the overall system works, to help readers who may not be as familiar with the specific details of the systems involved. For example, you may have to explain what the various services involved actually do. Don’t be shy about adding this detail, since people who already know it will just skim this part. Adding these details also makes these writeups useful for new hires to learn how the system works.

Make explicit how details serve the narrative

If you provide details in your narrative description, it has to be obvious to the reader why you are telling them these details. For example, if you write an alert fired eight hours before the surprise, you need to make it obvious to the reader why this alert is relevant to the narrative. There may be very different reasons, for example:

• This alert had important information about the nature of the operational surprise. However, it was an email only alert, not a paging one. And it was one of many email alerts that had fired, and those alerts are typically not actionable. It was ignored, just like the other ones.
• The alert was a paging alert, and the on-call who engaged concluded that it was just noise. In fact, it was noise. However, when the real alert fired eight hours later, the symptom was the same, and the on-call assumed it was another example of noise.
• The alert was a paging alert. The particular alert was unrelated to the surprise that would happen later, but it woke the on-call up in the middle of the night. They were quite tired the next day, when the surprise happened.

If you just say, “an alert fired earlier” without more detail, the reader doesn’t know why they should care about this detail in the writeup, which makes the writing less engaging. See also: The Law of Conservation of Detail.

Write in the present tense

This is just a stylistic choice of mine, but I find that if I write narratives in the present tense (e.g., “When X looks at the Y dashboard, she notices that signal Z has dropped…”), it reinforces the idea that the narrative is about understanding events as they were unfolding.

Use retrospective knowledge for foreshadowing

Unbeknownst to the princess but knownst to us, danger lurks in the stars above…

Opening crawl from the movie “Spaceballs”

When you are writing up a narrative description, you know a lot more about what happened than the people who were directly involved in the operational surprise as it was happening.

You can use this knowledge to make the writing more compelling through foreshadowing. You know about the consequences of actions that the people in the narrative don’t.

To help prevent the reader falling into the trap of hindsight bias, in your writeup, make it as explicit as possible that the knowledge the reader had is not knowledge that the people involved had. For example:

At 11:39, X takes action Y. What X does not know is that, six months earlier, Z had deployed a change to service Q, which changes what happens when action Y is taken.

This type of foreshadowing is helpful for two reasons:

• It pushes against hindsight bias by calling out explicitly how it came to be that a person involved had a mental model that deviated from reality.
• It creates “what happened next?” tension in the reader, encouraging them to read on.

Conclusion

We all love stories. We learn best from our own direct experiences, but storytelling provides an opportunity for us to learn from the experiences of others. Writing effective narratives is a kind of superpower because it gives you the ability to convey enormous amounts of detail to a large number of people. It’s a skill worth developing.