What’s allowed to count as a cause: ALERRT edition

The Advanced Law Enforcement Rapid Response Training (ALERRT) Center, based at Texas State University, trains law enforcement officers on how to deal with active shooter incidents. After the shooting at Uvalde, ALERRT produced an after-action report titled Robb Elementary School Attack Response Assessment and Recommendations.

The “Tactical Assessment” section of the report criticizes the action of the responding officers. It’s too long to excerpt in this post, but here are some examples:

A reasonable officer would have considered this an active situation and devised a plan to address the suspect. Even if the suspect was no longer firing his weapon, his presence and prior actions were preventing officers from accessing victims in the classroom to render medical aid (ALERRT & FBI, 2020, p. 2-17).

ALERRT report, p17

In a hostage/barricade, officers are taught to utilize the 5 Cs (Contain, Control, Communicate, Call SWAT, Create a Plan; ALERRT & FBI, 2020, pp. 2-17 to 2-19). In this instance, the suspect was contained in rooms 111 and 112. The officers established control in that they slowed down the assault. However, the officers did not establish communication with the suspect. The UCISD PD Chief did request SWAT/tactical teams. SWAT was called, but it takes time for the operators to arrive on scene. In the meantime, it is imperative that an immediate action plan is created. This plan is used if active violence occurs. It appears that the officers did not create an immediate action plan.

ALERRT report, p17

(Note: per the interim report, the officers did try to establish communication with the suspect, but the ALERRT authors weren’t aware of this at the time).

At 11:40:58, the suspect fired one shot. At 11:44:00, the suspect fired another shot, and finally, at 12:21:08, the suspect fired 4 more shots. During each of these instances, the situation had gone active, and the immediate action plan should have been triggered because it was reasonable to believe that people were being killed.

ALERRT report, p18

Additionally, we have noted in this report that it does not appear that effective incident command was established during this event. The lack of effective command likely impaired both the Stop the Killing and Stop the Dying parts of the response.

ALERT report, p19

The interim report also covers some of this territory in the subsection titled “ALERRT Standard for Active Shooter Training”, which starts on p17.

What struck me after reading the ALERRT report is that there is no mention of the fact that several of the responding police officers had received ALERRT training, including the chief of the Uvalde school district police, Pete Arredondo. From the interim report:

Before joining the Uvalde CISD Police Department, Chief Arredondo received active shooter training from the ALERRT Center, which the FBI has recognized as “the National Standard in Active Shooter Response Training.” Every school district peace officer in Texas must be trained on how to respond in active shooter scenarios. Not all of them get ALERRT training, but Chief Arredondo and other responders at Robb Elementary did.

Interim report, pp 17–18

The ALERRT report discusses how the actions of the officers is contrary to ALERRT training, and that is one potential explanation for why things went badly. But another potential explanation is that the ALERRT training wasn’t good enough to prepare the officers to deal with this situation. For example, perhaps the training doesn’t go into enough detail about the danger of fixation, where Chief Arredondo focused on trying to get a key for the door, when it wasn’t even clear whether the door was locked or not. (Does ALERRT train peace officers to diagnose fixation in other responders?)

The interim report gestures in the direction of ALERRT training being inadequate when it comes to checking the locks, although not in about the more general problem of fixation.

ALERRT has noted the failure to check the lock in its criticisms. See ALERRT, Robb Elementary School Attack Response Assessment and Recommendations at 18-19 (July 6, 2022). A representative of ALERRT testified before the Committee that the “first rule of breaching” is to check the lock. See Testimony of John Curnutt, ALERRT (July 11, 2022). Unfortunately, ALERRT apparently has neglected to include that “first rule of breaching” in its active- shooter training materials, which includes modules entitled “Closed and Locked Interior Doors” and “Entering Locked Buildings Quickly, Discreetly, and Safely.” See Federal Bureau of Investigation & ALERRT, Active Shooter Response – Level 1, at STU 3-8 – 3-10, 4-20 – 4-25.

Interim report, p64, footnote 206

Now, these criticisms are hindsight-laden, and my goal here isn’t to criticize ALERRT’s training: this isn’t my domain, and I don’t pretend to know how to train officers to deal with active shooter scenarios. Rather, my point is that the folks writing the ALERRT report were never going to consider that their own training is inadequate. After all, they’re the experts!

ALERRT was recognized as the national standard in active shooter response training by the FBI in 2013. ALERRT’s excellence in training was recognized in 2016 with a Congressional Achievement Award.

More than 200,000 state, local, and tribal first responders (over 140,000 law enforcement) from all 50 states, the District of Columbia, and U.S. territories have received ALERRT training over the last 20 years.

ALERRT training is research based. The ALERRT research team not only evaluates the efficacy of specific response tactics (Blair & Martaindale, 2014; Blair & Martaindale, 2017; Blair, Martaindale, & Nichols, 2014; Blair, Martaindale, & Sandel, 2019; Blair, Nichols, Burns, & Curnutt, 2013;) but also has a long, established history of evaluating the outcomes of active shooter events to inform training (Martaindale, 2015; Martaindale & Blair, 2017; Martaindale, Sandel, & Blair, 2017). Specifically, ALERRT has utilized case studies of active shooter events to develop improved curriculum to better prepare first responders to respond to similar situations (Martaindale & Blair, 2019).

For these reasons, ALERRT staff will draw on 20 years of experience training first responders and researching best practices to fulfill the Texas DPS request and objectively evaluate the law enforcement response to the May 24, 2022, attack at Robb Elementary School.

ALERRT report, p1

I think it’s literally inconceivable for the ALERRT staff to consider the inadequacy of their own training curriculum as being a contributor to the incident. It’s a great example of something that isn’t allowed to count as a cause.

I’ll end this blog post with some shade that the interim report threw on the ALERRT report.

The recent ALERRT report states that “[o]nce the officers retreated, they should have quickly made a plan to stop the attacker and gain access to the wounded,” noting “[t]here were several possible plans that could have been implemented.” “Perhaps the simplest plan,” according to ALERRT, “would have been to push the team back down the hallway and attempt to control the classrooms from the windows in the doors.” The report explains the purported simplicity of the plan by noting: “Any officer wearing rifle-rated body armor (e.g., plates) would have assumed the lead as they had an additional level of protection.” ALERRT, Robb Elementary School Attack Response Assessment and Recommendations (July 6, 2022). A problem with ALERRT’s depiction of its “simplest plan” is that no officer present was wearing “rifle-rated body armor (e.g., plates).” The Committee agrees the officers should have attempted to breach the classrooms even without armor, but it is inflammatory and misleading to release to the public a report describing “plans that could have been implemented” that assume the presence of protective equipment that the officers did not have.

Interim Report, pp51–52, footnote 158


Last week, the Investigative Committee on the Robb Elementary shooting (in Uvalde, Texas) released an interim report with their findings. I recommend reading it if you’re interested in incidents, especially section 5, “May 24 Incident & Law Enforcement Response“, which goes into detail on how the police responded.

I was pleasantly surprised to see terms like contributing factors and systemic failures in the report, and not a single reference to root cause. On the other hand, there’s way too much counterfactual reasoning in the report in my taste: there’s an entire subsection with the title “What Didn’t Happen in Those 73 Minutes?” It doesn’t get more counterfactual-y than that. There’s also normative language like egregious poor decision making. It’s disappointing, but not surprising, to see this type of language, given the nature of the incident.

Reading the report, I found there was too much I wanted to comment on to fit into one post, and so I’m going to try and write a series of posts instead. I’ve also created a GitHub repo with pointers to various artifacts related to the shooting (reports, images, videos).

Imagine there’s no human error…

When an incident happens, one of the causes is invariably identified as human error: somebody along the way made a mistake, did something they shouldn’t have done. For example: that engineer shouldn’t have done that clearly risky deployment and then walked away without babysitting it. Labeling an action as human error is an unfortunately effective way at ending an investigation (root cause: human error).

Some folks try to make progress on the current status quo by arguing that, since human error is inevitable (people make mistakes!), it should be the beginning of the investigation, rather than the end. I respect this approach, but I’m going to take a more extreme view here: we can gain insight into how incidents happen, even those that involve operator actions as contributing factors, without reference to human error at all.

Since we human beings are physical beings, you can think of us as machines. Specifically, we are machines that make decisions and take action based on those decisions. Now, imagine that every decision we make involves our brain trying to maximize a function: when provided with a set of options, it picks the one that has the largest value. Let’s call this function g, for goodness.

Pushing button A has a higher value of g than pushing button B.

(The neuroscientist Karl Friston has actually proposes something similar as a theory: organisms make decisions to minimize model surprise, a construct that Friston calls free energy).

In this (admittedly simplistic) model of human behavior, all decision making is based on an evaluation of g. Each person’s g will vary based on their personal history and based on their current context: what they currently see and hear, as well as other factors such as time pressure and conflicting goals. “History” here is very broad, as g will vary based not only on what you’ve learned in the past, but also on physiological factors like how much sleep you had last night and what you ate for breakfast.

Under this paradigm, if one of the contributing factors in an incident was the user pushing “A” instead of “B”, we ask “how did the operator’s g function score a higher value for pushing A over B”? There’s no concept of “error” in this model. Instead, we can explore the individual’s context and history to get a better understanding of how their g function valued A over B. We accomplish this by talking to them.

I think the model above is much more fruitful than the one where we identify errors or mistakes. In this model, we have to confront the context and the history that a person was exposed to, because those are the factors that determine how decisions get made.

The idea of human error is a hard thing to shake. But I think we’d be better off if we abandoned it entirely.

Some additional reading on the idea of human error:

We value possession of experience, but not its acquisition

Imagine you’re being interviewed for a software engineering position, and the interviewer asks you: “Can you provide me with a list of the work items that you would do if you were hired here?” This is how the action item approach to incident retrospectives feels to me.

We don’t hire people based on their ability to come up with a set of work items. We’re hiring them for their judgment, their ability to make good engineering decisions and tradeoffs based on the problems that they will encounter at the company. In the interview process, we try to assess their expertise, which we assume they have developed based on their previous work experience.

Incidents provide us with excellent learning opportunities because they confront us with surprises. If we examine an incident in detail, we can learn something about our system behavior that we didn’t know before.

Yet, while we recognize the value of experienced candidates when we do hiring, we don’t seem to recognize the value of increasing the experience of our current employees. Incidents are a visceral type of experience, and reflecting on these sorts of experiences is what increases our expertise. But you have to reflect on them to maximize the value, and you have to share this information out to the organization so that it isn’t just the incident responders that can benefit from the experience.

To me, learning from incidents is about increasing the expertise of an organization by reflecting on and sharing out the experiences of surprising operational events. Action items are a dime a dozen. What I care about is improving the organization’s ability to engineer software.

Software engineering in-the-large: the coordination challenge

Back when I was an engineering student, I wanted to know “How do the big companies develop software? How does it happen in the real world?”

Now that I work at a company that has to do large-scale software development, I understand better why it’s not something you can really teach effectively in a university setting. It’s not that companies doing large-scale software development are somehow better at writing software than companies that work on smaller-scale software projects. It’s that large-scale projects face challenges that small-scale projects don’t.

The biggest challenge at large-scale is coordination. My employer provides a single service, which means that, in theory, any project that anyone is working on inside of the company could potentially impact what anybody else is working on. In my specific case, I work on delivery tools, so we might be called upon to support some new delivery workflow.

You can take a top-down command-and-control style approach to the problem, by having the people at the top attempting to filter all of the information to just what they need, and them coordinating everyone hierarchically. However, this structure isn’t effective in dynamic environments: as the facts on the ground change, it takes too long for information to work its way up the hierarchy, adapt, and then change the orders downwards.

You can take a bottoms-up approach to the problem where you have a collection of teams that work autonomously. But the challenge there is getting them aligned. In theory, you hire people with good judgment, and provide them with the right context. But the problem is that there’s too much context! You can’t just firehose all of the available information to everyone, that doesn’t scale: everyone will spend all of their time reading docs. How do you get the information into the heads of the people that need it? becomes the grand challenge in this context.

It’s hard to convey the nature of this problem in a university classroom, if you’ve never worked in a setting like this before. The flurry of memos, planning documents, the misunderstandings, the sync meetings, the work towards alignment, the “One X” initiatives, these are all things that I had to experience viscerally, on a first-hand basis, to really get a sense of the nature of the problem.

OR, DevOps, and LFI

I’m currently reading Systems, Experts, and Computers: the Systems Approach in Management and Engineering, World War II and After. The book is really more of a collection of papers, each written by a different author. This post is about the second chapter, The Adoption of Operations Research in the United States During World War II, written by Erik Rau.

During World War II, the British and the Americans were actively investing in developing radar technology in support of the war effort, with applications such as radar-based air defense. It turned out that developing the technology itself wasn’t enough: the new tech had to be deployed and operated effectively in the field to actually serve its purpose. Operating these systems required coordination between machines, human operators, and the associated institutions.

The British sent out scientists and engineers into the field to study and improve how these new systems were used. To describe this type of work, the physicist Albert Rowe coined the term operational research (OR), to contrast it with developmental research.

After the surprise attack on Pearl Harbor, the U.S. Secretary of War tapped the British radar pioneer Robert Wattson-Watt to lead a study on the state of American air defense systems. In the report, Wattson-Watt describe U.S. air defense as “insufficient organization applied to technically inadequate equipment used in exceptionally difficult conditions“. The report suggested the adoption of the successful British technology and techniques, which included OR.

At this time, the organization responsible for developmental research into weapons systems was the Office of Scientific Research and Development (OSRD), headed by Vannevar Bush. While a research function like OR seemed like it should belong under OSRD, there was a problem: Bush didn’t want it there. He wanted to protect the scientific development work of his organization from political interference by the military, and so he sought to explicitly maintain a boundary between the scientists and engineers that were developing the technology, and the military that was operating it.

[The National Defense Research Committee] is concerned with the development of equipment for military use, whereas these [OR] groups are concerned with the analysis of its performance, and the two points of view do not, I believe, often mix to advantage.

Vannevar Bush, quoted on p69

In the end, the demand for OR was too great, and Bush relented, creating the Office of Field Service within the OSRD.

Two things struck me reading this chapter. The first was that operational research was a kind of proto-DevOps, a recognition of the need to create a cultural shift in how development and operations work related to each other, and the importance of feedback between the two groups. It was fascinating to see the resistance to it from Bush. He wasn’t opposed to OR itself, he was opposed to unwanted government influence, which drove his efforts to keep development and operations separate.

The second thing that struck me was this idea of OR being doing research on operations. I had always thought of OR as being about things like logistics, basically graph theory problems addressed by faculty who work in business schools instead of computer science departments. But, here, OR was sending researchers into the field to study how operations was done in order to help improve development and operations. This reminded me very much of the aims of the learning from incidents (LFI) movement.

Prussia meets Versailles: a review of Moral Mazes

Managers rarely speak of objective criteria for achieving success because once certain crucial points in one’s career are passed, success and failure seem to have little to do with one’s accomplishments.


Almost all management books are prescriptive: they’re self-help books for managers. Moral Mazes is a very different kind of management book. Where most management books are written by management gurus, this one is written by a sociologist. This book is the result of a sociological study that the author conducted at three U.S. companies in the 1980s: a large textile firm, a chemical company, and a large public relations agency. He was interested in understanding the ethical decision-making process of American managers. And the picture he paints is a bleak one.

American corporations are organized into what Jackall calls patrimonial bureaucracies. Like the Prussian state, a U.S. company is organized as a hierarchy, with a set of bureaucratic rules that binds all of the employees. However, like a monarchy, people are loyal to individuals rather than offices. Effectively, it is a system of patronage, where leadership doles out privileges. Like in the court of King Louis XIV, factions within the organization jockey to gain favor.

With the exception of the CEO, all of the managers are involved in both establishing the rules of the game, and are bound by the rules. But, because the personalities of leadership play a strong role, and because leadership often changes over time, the norms are always contingent. When the winds change, the standards of behavior can change as well.

Managers are also in a tough spot because they largely don’t have control over the outcomes on which they are supposed to be judged. They are typically held responsible for hitting their numbers, but luck and timing play an enormous role over whether they are able to actually meet their objectives. As a result, managers are in a constant state of anxiety, since they are forever subject to the whims of fate. Failure here is socially defined, and the worst outcome is to be in the wrong place at the wrong time, and to have your boss say “you failed”.

Managers therefore focus on what they can control, which is the image they project. They put in a lot of hours, so they appear to be working hard. They strive to be seen as someone who is a team player, who behaves predictably and makes other managers feel comfortable. To stand out from their peers, they have to have the right style: the ability to relate to other people, to sell ideas, appear in command. To succeed in this environment, a manager needs social capital as well as the ability to adapt quickly as the environment changes.

Managers commonly struggle with decision making. Because the norms of behavior are socially defined, and because these norms change over time, they are forever looking to their peers to identify what the current norms are. Compounding the problem is the tempo of management work: because a manager’s daily schedule is typically filled with meetings and interrupts, with only fragmented views of problems being presented, there is little opportunity to gain a full view of problems and reflect deeply on them.

Making decisions is dangerous, and managers will avoid it when possible, even if this costs the organization in the long run. Jackall tells an anecdote about a large, old battery at a plant. The managers did not want to be on the hook for the decision to replace it, and so problems with it were patched up. Eventually, it failed completely, and the resulting cost to replace it and to deal with costs related to EPA violations and lawsuits was over $100M in 1979 dollars. And yet, this was still rational decision-making on behalf of the managers, because it was a risk for them in the short-term to make the decision to replace the battery.

Ethical decision making is particularly fraught here. Leadership wants success without wanting to be bothered with the messy details of how that success is achieved: a successful middle manager shields leadership from the details. Managers don’t have a professional ethic in the way that, say, doctors or lawyers do. Ethical guidelines are situational, they vary based on changing relationships. Expediency is a virtue, and a good manager is one who is pragmatic about decision making.

All moral issues are transmuted into practical concerns. Arguing based on morality rather than pragmatism is frowned upon, because moral arguments compel managers to act, and they need to be able to take stock of the social environment in order to judge whether a decision would be appropriate. Effective managers use social cues to help make decisions. They conform to what Jackall calls institutional logic: the ever-changing set of rules and incentives that the culture creates and re-creates to keep people’s perspectives and behaviors consistent and predictable.

There comes a time in every engineer’s career when you ask yourself, “do I want to go into management?” I’ve flirted with the idea in the past, but ultimately came down on the “no” side. After reading Moral Mazes, I’m more confident than ever that I made the right decision.

Taking the hit

Here’s a scenario I frequently encounter: I’m working on writing up an incident or an OOPS. I’ve already interviewed key experts on the system, and based on those interviews, I understand the implementation details well enough to explain the failure mode in writing.

But, when I go to write down the explanation, I discover that I don’t actually have a good understanding of all of the relevant details. I could go back and ask clarifying questions, but I worry that I’ll have to do this multiple times, and I want to avoid taking up too much of other people’s time.

I’m now faced with a choice when describing the failure mode. I can either:

(a) Be intentionally vague about the parts that I don’t understand well.

(b) Make my best guess about the implementation details for the parts I’m not sure about.

Whenever I go with option (b), I always get some of the details incorrect. This becomes painfully clear to me when I show a draft to the key experts, and they tell me straight-out, “Lorin, this section is wrong.”

I call choosing option (b) taking the hit because, well, I hate the feeling of being wrong about something. However, I always try to go with this approach because this maximizes both my own learning and (hopefully) the learning of the readers. I take the hit. When you know that your work will be reviewed by an expert, it’s better to be clear and wrong than vague.

Aristotle’s revenge

Imagine you’re walking around a university campus. It’s a couple of weeks after the spring semester has ended, and so there aren’t many people about. You enter a building and walk into one of the rooms. It appears to be some kind of undergraduate lab, most likely either physics or engineering.

In the lab, you come across a table. On the table, someone has balanced a rectangular block on its smallest end.

You nudge the top of the block. As expected, it falls over with a muted plonk. You look around to see if you might have gotten in trouble, but nobody’s around.

You come across another table. This table has some sort of track on it. The table also has a block on it that’s almost identical to the one on the other table, except that the block has a pin in it that connects it to some sort of box that is mounted on the track.

Not being able to resist, you nudge the top of this block, and it starts to fall. Then, the little box on the track whirs to life, moving along the track in the same direction that you nudged the block in. Because of the motion of the box, the block stays upright.

The ancient philosopher Aristotle believed that there were four distinct types of causes that explained why things happened. One of these is what Aristotle called the efficient cause: “Y behaved the way it did because X acted on Y“. For example, the red billiard ball moved because it was struck by the white ball. This is the most common way we think about causality today, and it’s sometimes referred to as linear causality.

Efficient cause does a good job of explaining the behavior of the first rectangular block in the anecdote: it fell over because we nudged it with our finger. But it doesn’t do a good job of explaining the observed behavior of the second rectangular block: we nudged it, and it started to fall, but it righted itself, and ended up balanced again.

Another type of cause Aristotle talked about was what he called the final cause. This is a teleological view of cause, which explains the behavior of objects in terms of their purpose or goal.

Final cause sounds like an archaic, unscientific view of the world. And, yet, reasoning with final cause enables us to explain the behavior of the second block more effectively than efficient cause does. That’s because the second block is being controlled by a negative-feedback system that was designed to keep the block balanced. The system will act to compensate for disturbances that could lead to the block falling over (like somebody walking over and nudging it). Because the output, a sensor that reads the angle of the block, is fed back into the input of the control system, the relationship between external disturbance and system behavior isn’t linear. This is sometimes referred to as circular causality, because of the circular nature of feedback loops.

The systems that we deal with contain goals and feedback loops, just like the inverted pendulum control system that keeps the block balanced. If you try to use linear causality to understand a closed-loop system, you will be baffled by the resulting behavior. Only when you understand the goals that the system is trying to achieve, and the feedback loops that it uses to adjust its behavior to reach those goals, will the resulting behavior make sense.