I’ve been enjoying the ongoing MIT STAMP workshop. In particular, I’ve been enjoying listening to Nancy Leveson talk about system safety. Leveson is a giant in the safety research community (and, incidentally, an author of my favorite software engineering study). She’s also a critic of root cause and human error as explanations for accidents. Despite this, she has a different perspective on safety than many in the resilience engineering community. To sharpen my thinking, I’m going to capture my understanding of this difference in this post below.
From Leveson’s perspective, the engineering design should ensure that the system is safe. More specifically, the design should contain controls that eliminate or mitigate hazards. In this view, accidents are invariably attributable to design errors: a hazard in the system was not effectively controlled in the design.
By contrast, many in the resilience engineering community claim that design alone cannot ensure that the system is safe. The idea here is that the system design will always be incomplete, and the human operators must adapt their local work to make up for the gaps in the designed system. These adaptations usually contribute to safety, and sometimes contribute to incidents, and in post-incident investigations we often only notice the latter case.
These perspectives are quite different. Leveson believes that depending on human adaptation in the system is itself dangerous. If we’re depending on human adaptation to achieve system safety, then the design engineers have not done their jobs properly in controlling hazards. The resilience engineering folks believe that depending on human adaptation is inevitable, because of the messy nature of complex systems.
Surfing Complexity 
I find I’m agreeing with both Leveson’s and the resilience engineers’ perspectives, which makes me think they’re complementary—a shift in focus rather than a disagreement—at least in their softer forms.
That is, depending on human adaptation is both dangerous and inevitable, and we should strive to reduce the danger (by designing systems for safety) and to build expertise to deal with the inevitability (by analyzing work as done, for instance), aware that these efforts feed into each other.
If we make a stronger claim on either side—if we say that it is possible to design perfectly safe systems, or that because human adaptation is inevitable there is no point in trying to reduce it—then I see the disagreement, but I think these stronger claims are less useful or defensible than their weaker counterparts.
Here’s the rub, from the appendix of an unpublished paper written by Nancy Leveson: https://drive.google.com/file/d/1yvzVMrvzOXNdhhQ1i_MYLX3-2jRb5WeQ/view
“Engineered systems are designed, the design is known, and it is usually possible to predict the emergent behavior.” p109
I think the fundamental disagreement is the degree to which emergent behavior of engineered systems can be predicted.
Makes one wonder.
Human factors folks and general systems theory folks typically define “emergent” behavior as “new” behavior occurring at system interfaces (machine-machine, machine-human) and that those “emergent” behaviors are, by definition, unknown and often unknowable prior to complete & correct (validation) testing.
Agree! Wave-particle duality analog. All systems have human operators involved, even completely autonomous systems; those “operators” are called installation and maintenance personnel.