Writing docs well: why should a software engineer care?

Recently I gave a guest lecture in a graduate level software engineering course on the value of technical writing for software engineers. This post is a sort of rough transcript of my talk.

I live-sketched my slides as I went.

I talked about three goals of doing doing technical writing.

The first one is about building shared understanding among stakeholders of a document. One of the hardest problems in software engineering is getting multiple people to have a sufficient understanding of some technical aspect, like the actual problem being solved, or a proposed solution. This is ostensibly the only real goal of technical writings.

Shared understanding is related to the idea of common ground that you’ll sometimes hear the safety folks talk about.

If you’re a programmer who works completely alone, then this is a problem you generally don’t have to solve, because there’s only one person involved in the software project.

But as soon as you are working in a team, then you have to address the problem of shared understanding.

When we work on something technical, like software, we develop a much deeper understanding because we’re immersed in it. This can make communication hard when we’re talking to someone who hasn’t been working in the same area and so doesn’t have the same level of technical understanding of that particular bit.

If you’re working only with a small, co-located group (e.g., in a co-located startup), then having a discussion in front of a whiteboard is a very effective mechanism for building shared understanding. In this type of environment, writing effective technical docs is much less important.

The problem with the discuss-in-front-of-the-whiteboard approach is that it doesn’t scale up, and it also doesn’t work for distributed environments.

And this is where technical documents come in.

I like to say that the hardest problem in software engineering is getting the appropriate information into the heads of the people who need to have that information in order to do their work effectively.

In large organizations, a lot of the work is interconnected, which means that some work that somebody else is doing can affect your work. If you’re not aware of that, you can end up working at cross-purposes.

The challenge is that there’s so much potential information that might be useful. Everyone could potentially spend all of their working hours reading docs, and still not read everything that might be relevant.

To write a doc well means to get people to gain sufficient understanding so that you can coordinate work effectively.

The second goal of writing I talked about was using writing to help with your own thinking.

The cartoonist Richard Guindon has a famous quote: “writing is nature’s way of letting you know how sloppy your thinking is.” You might have an impression that you understand something well, but that sense of clarity is often an illusion, and when you go to explicitly capture your understanding in a document, you discover that you didn’t understand things as well as you thought. There’s nowhere to hide in your own document.

When writing technical docs, I always try hard to work explicitly through examples to demonstrate the concepts. This is one of the biggest weaknesses I see in practice in technical docs, that the author has not described a scenario from start to finish. Conceptually, you want your doc to have something like a storyboard that’s used in the film industry, to tell the story. Writing out a complete example will force you to confront the gaps in your understanding.

The third goal is a bit subversive: it’s how to use effective technical writing to have influence in a larger organization when you’re at the bottom of the hierarchy.

If you want influence, you likely have some sort of vision of where you want the broader organization to go, and the challenge is to persuade people of influence to move things closer to your vision.

Because senior leadership, like everyone else in the organization, only has a finite amount of time and attention, their view of reality are shaped by the interactions they do have: which is largely through meetings and documents. Effective technical documents shape the view of reality that leadership has, but only if they’re written well.

If you frame things right, you can make it seem as if your view is reality rather than simply your opinion. But this requires skill.

Software engineers often struggle to write effective docs. And that’s understandable, because writing effective technical docs is very difficult.

Anyone who has set down at a computer to write a doc and has stared at the blinking cursor at an empty doc knows how difficult it can be to just get started.

Even the best-written technical docs aren’t necessarily easy to read.

Poorly written docs are hard to read. However, just because a doc is hard to read, doesn’t mean it’s poorly written!

This talk is about technical writing, but technical reading is also a skill. Often, we can’t understand a paragraph in a technical document without having a good grasp of the surrounding context. But we also can’t understand the context without reading the individual paragraphs, not only of this document, but of other documents as well!

This means we often can’t understand a technical document by reading from beginning to end. We need to move back and forth between working to understand the text itself and working to understand the wider context. This pattern is known as the hermeneutic circle, and it is used in Biblical studies.

Finally, some pieces of advice on how to improve your technical writing.

Know explicitly in advance what your goal is in doing the writing. Writing to improve your own understanding is different from writing to improve someone else’s understanding, or to persuade someone else.

Make sure your technical document has concrete examples. These are the hardest to write, but they are most likely to help achieve your goals in your document.

Get feedback on your drafts from people that you trust. Even the best writers in the world benefit from having good editors.

An old lesson about a fish

Back when I was in college [1], I was required to take several English courses. I still remember an English professor handing out an excerpt from the book ABC of Reading by Ezra Pound [2]:

No man is equipped for modern thinking until he has understood the anecdote of Agassiz and the fish:

A post-graduate student equipped with honours and diplomas went to Agassiz to receive the final and finishing touches. The great man offered him a small fish and told him to describe it.

Post-Graduate Student: ‘That’s only a sunfish.’

Agassiz: ‘I know that. Write a description of it.’

After a few minutes the student returned with the description of the Ichthus Heliodiplodokus, or whatever term is used to conceal the common sunfish from vulgar knowledge, family of Heliichtherinkus, etc., as found in textbooks of the subject.

Agassiz again told the student to describe the fish.

The student produced a four-page essay. Agassiz then told him to look at the fish. At the end of three weeks the fish was in an advanced state of decomposition, but the student knew something about it.

I remember my eighteen-year-old self hating this anecdote. It sounded like Agassiz just wasted the graduate student’s time, leaving him with nothing but a rotting fish for his troubles. As an eventual engineering major, I had no interest in the work of analyzing texts that was required in English courses. I thought such analysis was a waste of time.

It would take about two decades for the lesson of this anecdote to sink into my brain. The lesson I eventually took away from it is that there is real value in devoting significant effort to close study of an object. If you want to really understand something, a casual examination just won’t do.

To me, this is the primary message of the learning from incidents in software movement. Doing an incident investigation, like studying the fish, will take time. Completing an investigation may take weeks, even months. Keep in mind, though, that you aren’t really studying an incident at all: you’re studying your system through the lens of an incident. And, even though the organization will have long since moved on, once you’re done, you’ll know something about your system.

[1] Technically it was CEGEP, but nobody outside of Quebec knows what that is.

[2] Pound is likely retelling an anecdote originally told by either Nathaniel Shaler or Samuel Hubbard Scudder, both of whom were students of Agassiz.

Stories as a vehicle for learning from the experience of others

Senior software engineering positions command higher salaries than junior positions. The industry believes (correctly, I think) that engineers become more effective as they accumulate experience, and that perception is reflected in market salaries.

Learning from direct experience is powerful, but there’s a limit to the rate at which we can learn from our own experiences. Certainly, we learn more from some experiences than others; we joke about “ten years of experience” versus “one year of experience ten times over”, as well as using scars as a metaphor for these sometimes unpleasant but more impactful experiences. But there’s only so many hours in a day, and we may not always be…errr… lucky enough to be exposed to many high-value learning opportunities.

There’s another resource we can draw on besides our own direct experience, and that’s the experiences of peers in our organization. Learning from the experiences of others isn’t as effective as learning directly from our own experience. But, if the organization you work in is large enough, then high-value learning opportunities are probably happening around you all of the time.

Given these opportunities abound, the challenge is: how can we learn effectively from the experiences of others? One way that humans learn from others is through telling stories.

Storytelling enables a third person to experience events by proxy. When we tell a story well, we run a simulation of the events in the mind of the listener. This kind of experience is not as effective as the first-hand kind, but it still leaves an impression on the listener when done well. In addition, storytelling scales very well: we can write down stories, or record them, and then publish these across the organization.

A second challenge is: what stories should we tell? It turns out that incidents make great stories. You’ll often hear engineers tell tales of incidents to each other. We sometimes calling these war stories, horror stories (the term I prefer), or ghost stories.

Once we recognize the opportunity of using incidents as a mechanism for second-hand-experiential-learning-through-storytelling, this shifts our thinking about the role and structure of an incident writeup. We want to tell a story that captures the experiences of the people involved in the incident, so that the readers can imagine what is was like, in the moment, when the alerts were going off and confusion reigned.

When we want to use incidents for second-hand experiential learning, it shifts the focus of an incident investigation away from action items as being the primary outcome and towards the narrative, the story we want tell.

When we hire for senior positions, we don’t ask candidates to submit a list of action items for tasks that could improve our system. We believe the value of their experience lies in them being able to solve novel problems in the future. Similarly, I don’t think we should view incident investigations as being primarily about generating action items. If, instead, we view them as an opportunity to learn collectively from the experiences of individuals, then more of us will get better at solving novel problems in the future.

What Deming got wrong

One of my Father’s Day presents this year was The Essential Deming, an anthology of Deming’s shorter writings. I thoroughly enjoyed Deming’s Out of the Crisis and was looking to read more from him.

Reading this book, I was surprised to discover that Deming was opposed to workers training workers, which he considered a faulty practice. The most effective way to become an expert is through the apprenticeship model, where a novice works alongside an expert and directly observes how the expert does their work. That Deming would reject this model, and would believe that an outsider could more effectively train a worker than someone who actually does the day-to-day work is, frankly, bizarre.

Deming also asserted that the most effective teachers at a university were the professors that were the best researchers. This also seemed to me to be an extremely odd claim, and one I’m extremely skeptical of.

Deming had a deep understanding of systems thinking, and the importance of holistic, expert judgment (he used the term leadership) over chasing metrics, and that shines through in this book. However, while Deming seemed to recognize the value of expertise, he did not seem to have a good understanding of how people acquire it.

Postmodern engineering

When I was younger, I wanted to be a physicist. I ended up majoring in computer engineering, because I also wanted gainful employment, but my heart was always in physics, and computer engineering seemed like a good compromise between my love of physics and early interest in computers.

I didn’t think too deeply about the philosophy of science back then, but my beliefs were in line with the school of positivism. I believed there was a single underlying reality , the nature of this reality was potentially knowable, and science was an effective tool for understanding that reality. I was vaguely aware of the postmodernist movement, but mostly by reading about the Sokal hoax, where the physicist Alan Sokal had demonstrated that postmodernism was nonsense.

Around the same time, I also read To Engineer is Human: the Role of Failure in Successful Design by the civil engineering researcher Henry Petroski. The book is a case study on how civil engineering advanced through understanding structural failures. Success, on the other hand, teaches the engineer nothing.

Many years later, I find myself operationally a postmodernist (although constructivist might be a more accurate term). When I study how incidents happen, I no longer believe that there is a single, underlying reality of what really happened that we can access. Instead, I believe that the best we can do is construct narratives based on the perspectives of the different people that were involved in the incident. These narratives will inevitably be partial, and some of them may conflict. And there are things that we will never really know or understand. In addition, contra Petroski, I also believe that we can learn from studying successes as well as from studying failure.

I suspect that most engineers are steeped in the positivist tradition of thinking as well. This change in perspective is a big one: I’m not even sure how my own thinking evolved over time, and so I don’t know how to encourage this shift in others. But I do believe that if we want to learn as much as we can from incidents, we need to work on changing how our fellow engineers think about what is knowable. And that’s a tall order.

Not apprenticeship!

Mark Guzdial points to an article by Nicholas Lemann in the Chronicle of Higher Ed entitled The Soul of the Research University. It’s a good essay about the schizophrenic nature of the modern research university. But Lemann takes some shots at the notion of teaching skills in the university. Here’s some devil’s advocacy from the piece:

Why would you want to be taught by professors who devote a substantial part of their time to writing projects, instead of working professionals whose only role at the university is to teach? Why shouldn’t the curriculum be devoted to imparting the most up-to-the-minute skills, the ones that will have most value in the employment market? Embedded in those questions is a view that a high-quality apprenticeship under an attentive mentor would represent no loss, and possibly an improvement, over a university education.

Later on, Lemann refutes that perspective, that students are better off being taught at research universities by professors engaged in research. He seems to miss the irony that this apprenticeship model is precisely how these research universities train PhD students. For bonus irony, here was the banner ad I saw atop the article: skills-webinar

Software Analysis

At McGill University, the computer engineering program evolved out of the electrical engineering program, so it was very EE-oriented. I was required to take four separate courses that involved (analog) circuit analysis: fundamental of electrical engineering, circuit analysis, electronic circuits I, and electronic circuits II.

I’m struggling to think of what the equivalent of “circuit analysis” would be for software engineering. To keep the problem structure the same as circuit analysis, it would be something like: Given a (simplified model of?) a computer program, for a given input, what will the program output?

It’s hard to imagine even a single course in a software engineering program dedicated entirely to this type of manual “program analysis”. And yet, reading code is so important to what we do, and remains such a difficult task.

Training is a dirty word

Two posts caught my eye this week. The first was Anil Dash’s The Blue Collar Coder, and the second was Greg Wilson’s Dark Matter, Public Health, and Scientific Computing. Anil wrote about high school students and Greg spoke about scientists, but ultimately they’re both about teaching computer skills to people without a formal background in computing. In other words, training.

In the hierarchy of academia, training is pretty firmly at the bottom. Education at least gets some lip service, being the primary mission of the university and all. But training is a base, vulgar activity. And it’s a real shame, because the problems that Anil and Greg are trying to address are important ones that need solving. Help will need to come from somewhere else.