Naturalistic Decision Making

Caroline E. Zsambok, Gary Klein, Caroline E. Zsambok, Gary Klein

• Publisher: Psychology Press
• Year: 2014
• Print ISBN: 9780805818741
• eBook ISBN: 9781317779599
• Topic: Psychology
• Subtopic: History & Theory in Psychology
• Index: Psychology

If you aren't using the term naturalistic decision making, or NDM, you soon will be. Even as a very young field, NDM has already had far-reaching applications in areas as diverse as management, aviation, health care, nuclear power, military command and control, corporate teamwork, and manufacturing.

Put simply, NDM is the way people use their experience to make decisions in the context of a job or task. Of particular interest to NDM researchers are the effects of high-stake consequences, shifting goals, incomplete information, time pressure, uncertainty, and other conditions that are present in most of today's work places and that add to the complexity of decision making. Applications of NDM research findings target decision aids and training that help people in their decision-making processes.

This book reports the findings of top NDM researchers, as well as many of their current applications. In addition, the book offers a historical perspective on the emergence of this new paradigm, describes recent theoretical and methodological advancements, and points to future developments. It was written for people interested in decision making research and applications relative to a diverse array of work settings and products such as human-computer interfaces, decision support systems, individual and team training, product designs, and organizational development and planning.

Part I
About Naturalistic Decision Making

Chapter 1
Naturalistic Decision Making: Where Are We Now?

Caroline E. Zsambok
Klein Associates Inc.

The short answer to the question of where we are now is: We are in very exciting times. If you are engaged in research or applications concerning Naturalistic Decision Making (NDM), you are already aware of this. If you are not an NDM researcher, you may not know about this emerging movement and the interest it is generating. For a start, over the last decade, informal estimates of funding from the U.S. Department of Defense for NDM research range from $25 million to $35 million. A multitude of projects have been funded in hopes that an NDM research perspective can improve decision support systems installed in military command-and-control suites in various aircraft, naval vessels, and land-based platforms. The U.S. Marine Corps began applying NDM research findings to improve its command and control procedures. On the civilian side, several leading companies in the nuclear power industry are beginning to realign their procedures, training, and decision support systems through the use of NDM research methods such as cognitive task analysis, and through the use of NDM-derived training methods. The aviation industry is sponsoring programs on situation awareness to improve decision making. And this is just a sampling. We are, indeed, in exciting times with NDM research.

A longer answer to the question of where we are now in NDM research can be found in the pages of this book. The purpose of this chapter is to frame the longer answer by highlighting major themes in this volume and identifying the value of pursuing this research. First, however, some background about NDM and a definition are in order.

Background and Definition

The term NDM first appeared in 1989, when a conference was organized so that researchers who had stepped outside of the traditional decision research paradigms could discuss their findings, and then publish them in a single volume (Klein, Orasanu, Calderwood, & Zsambok, 1993). In the 1980s, these researchers had begun to study how experienced people actually make decisions in their natural environments or in simulations that preserve key aspects of their environments. In fact, the foregoing can serve as a short definition of NDM: NDM is the way people use their experience to make decisions in field settings.

A sampling of the types of decision makers studied by participants in that first NDM conference includes firefighters, pilots and cockpit crews, corporate executives, trouble shooters of electronic equipment, computer software designers, military commanders, and physicians.

What these researchers found is that the processes and strategies of "naturalistic" decision making differ from those revealed in traditional decision research. For example, it was discovered that in NDM, the focus in the decision event is more front-loaded, so that decision makers are more concerned about sizing up the situation and refreshing their situation awareness through feedback, rather than developing multiple options to compare to one another. In contrast, most traditional decision research has involved inexperienced people who are engaged in laboratory tasks where contextual or situational factors play a limited role. The traditional paradigm emphasizes understanding the back end of the decision event—choosing among options (Beach & Lipshitz, 1993).

Further, NDM researchers took issue with the fact that the traditional paradigm compares the quality of decisions against abstract, "rational" standards, such as multiattribute utility analysis. Rational standards and formal models of decision making may be appropriate for typical laboratory tasks, but they do not take into account effects of most contextual factors that accompany decision making in real-world settings, nor do they adequately model the adaptive characteristics of real-world behavior (Cohen, 1993). Indeed, many participants in the 1989 conference had embarked on NDM-like research because formal models lacked explanatory or predictive power in real-world settings they were studying, or because they discovered problems with the prescriptive advice from the traditional paradigm. As one of many examples, several researchers such as Jeff Grossman and Ed Salas had seen that training principles derived from formal models produced counterproductive behaviors in the military. A case in point is the military requirement of commanders to develop and compare multiple courses of action, even when time is precious and when a single option is an obvious workable choice, given the situation and context.

The identification of key contextual factors that affect the way real-world decision making occurs, in contrast to their counterparts in the traditional decision research paradigm, evolved as a major contribution of the 1989 NDM conference (Orasanu & Connolly, 1993). They are:

1. Ill-structured problems (not artificial, well-structured problems).
2. Uncertain, dynamic environments (not static, simulated situations).
3. Shifting, ill-defined, or competing goals (not clear and stable goals).
4. Action/feedback loops (not one-shot decisions).
5. Time stress (as opposed to ample time for tasks).
6. High stakes (not situations devoid of true consequences for the decision maker).
7. Multiple players (as opposed to individual decision making).
8. Organizational goals and norms (as opposed to decision making in a vacuum).

This array of task and setting factors is one of four defining markers for NDM research. The other three markers concern the research participants (experienced decision makers, not naive subjects); the purpose of the research (discovering how experienced people actually make decisions in context-rich environments, not how they ought to make decisions in approximation to a rational standard); and locus of interest within the decision episode (not just in the option selection process, but also in situation awareness).

Considering these four markers, a definition richer than the earlier short-hand version has evolved:

The study of NDM asks how experienced people, working as individuals or groups in dynamic, uncertain, and often fast-paced environments, identify and assess their situation, make decisions and take actions whose consequences are meaningful to them and to the larger organization in which they operate.

This definition is too long to remember. The shorter version will do. But, this definition is important for two reasons. First, it is more comprehensive and carries more meaning than the short version. Second, it is stated in positive terms, not oppositional terms relative to what it is not. A large portion of the 1989 NDM conference was spent defining NDM in opposition to traditional research. This was an important step in the beginning of the NDM research endeavor. We needed to define our work against the backdrop of the then current paradigm—to say what NDM research is not, as a point of departure for stating what NDM research is.

Now, there is a clearer understanding of what defines NDM in positive terms. The value added by this position was visible at the second NDM conference in 1994. Much less of our time was spent clarifying who we are and what we do compared to the traditional standard. The majority of our time was spent on discussions of related research lines and what we can learn from them, on reports of new research findings, on applications of NDM research to a variety of domains, on NDM theory development, and on methods that can be used in practical applications and for pushing forward our research.

The value added by a positive definition of NDM is also reflected in this book, because it grew from the 1994 conference. Before turning to major themes in the book, two points should be clarified about the foregoing discussion and definition of NDM. First, the comprehensive definition emphasizes complex, uncertain, and unstable situations where decision makers cannot rely on routine action or thinking. But, as Rouse and Valusek (1993) pointed out, a great deal of real-world decision making involves "activities that are quite routine, with actions following well-worn patterns and observations agreeing with expectations" (p. 274). To be sure, some NDM researchers explicitly attempt to model both simple and complex decision making. For example, Rasmussen (1993) distinguished two levels of routine performance—the less conscious and habitual skill-based performance and the more conscious rule-based performance. As another example, Klein's RPD model describes simple, or routine, NDM as different from complex, or nonroutine, NDM (Klein, 1993).

The second point to be clarified is that NDM research does not mandate field studies as the only methodology. To the extent that laboratory studies can replicate the factors present in real-world decision making such that subjects take the tasks almost as seriously as they do in real life, lab studies can be considered as included under the umbrella of NDM research. Hammond (1993) argued that what matters is not where the research takes place, but what is studied. His point is that naturalistic approaches must provide testable theories of the environment that describe its formal properties and their consequences for cognitive activity. These approaches cannot remove the complexity and ambiguity present in the environment, or they will amount to removal of the object of study. As stated earlier, lab studies generally remove that complexity, but they need not. An excellent example from past NDM research is Hammond and Grassia's (1985) application of laboratory studies on interpersonal conflict and interpersonal learning to a naturalistic test within the public political domain. Many more examples are to be found in the current NDM research activities described by contributors to this volume.

Themes in NDM Research

This section provides a framework for answering in a more specific way the question of where we are now with NDM research. Unlike the structure represented by the table of contents, this is a framework composed of six themes that criss-cross chapters in this book. As such, it offers an organizing tool to understand where we are now in NDM research.

The Big Picture: Models and Theories

Both the development and testing of models and theories have occurred since the 1989 conference, although not to the degree that some would like (Howell, chapter 4, this volume). Five examples of development and testing contained in this volume are as follows.

First, Cohen, Freeman, and Thompson (chapter 25, this volume) describe a new model called the Recognition/Metacognition (R/M) model. This model describes a set of metacognitive skills that supplement recognitional processes in decision events involving novel situations. Among these skills are the identification of key situational assessments, checking the stories one constructs for completeness and consistency relative to these assessments, and generating alternative stories when too much conflicting information is encountered. The model was developed through critical incident interviews in two military domains.

Training based on the model has been conducted with command staff in the Army and Navy, and the results of the training are consistent with hypotheses derived from the model. Training in these metacognitive skills led participants to consider more factors in their situation assessment evaluations compared to nontrained controls. It also led them to consider less of the neutral (irrelevant) information and more of the conflicting information than did controls. Last, trained participants developed a better understanding of the situation than did controls, as evaluated by blind subject matter experts.

As a second example, Klein (chapter 27, this volume) reports the addition of a component to an earlier version of the Recognition-Primed Decision (RPD) model: the diagnostic function. Decision makers engage in diagnosis either for the purpose of evaluating a situation assessment about which they are uncertain, or to compare alternative explanations of events. The diagnostic function was added because of its evidence in new studies conducted by Klein and his colleagues, and because of its evidence in story building by jurors and others who try to diagnose situations by developing plausible explanations or stories (Pennington & Hastie, 1993). Klein also reports model testing activities in a study of skilled chess players. Findings were consistent with the model on three counts: Skilled players produced a high quality move as the first one considered; time pressure did not cripple their performance; skilled players could adopt a course of action (i.e., a chess move) without comparing and contrasting it to others.

Third, Endsley (chapter 26, this volume) describes a model of situation awareness (SA). The National Transportation Safety Board discovered that 88% of aircraft accidents in a 4-year period were due to human error in situation assessment. Endsley's model represents an initial attempt to provide a general framework for understanding processes and factors that impact SA. The model describes three levels of SA: perception, comprehension, and prediction. It also contains mechanisms for goal selection, attention to critical cues and expectancies, and action. This model is similar to other models of human performance and decision making, but it emphasizes the role of SA within the decision event.

A fourth example of model/theory development and testing concerns the work of Serfaty, MacMillan, Entin, & Entin (chapter 23, this volume). They developed a mental-model theory and framework for studying expertise in battle-command decision making. Interestingly, this framework includes among other sources Duncker's (1945) often-ignored decision theory (see Hoffman, 1994), which is remarkably compatible with the NDM perspective yet predates the NDM movement by more than 50 years.

Serfaty et al. derived a number of hypotheses from their model; these hypotheses concern the difference between experts and novices in the quality of their mental models. Serfaty et al. found that participants with higher expertise levels generated more detailed courses of action, with more contingencies, than those with lower expertise levels. Similarly, high-expertise participants were more able to focus immediately on critical unknowns and to ask diagnostic questions. The hypothesis that experts build more complex mental models was supported by five behavioral measures, including their ability to "see" what could go wrong with their plans. This finding is consistent with other NDM models described earlier.

Finally, Lipshitz and Ben Shaul (chapter 28, this volume) made a significant contribution to model and theory development by critically reviewing three NDM models described in this volume (cf. Endsley; Klein; Serfaty et al.) and incorporating relevant concepts from Neisser (1976) and Rouse and Morris (1986). Lipshitz and Ben Shaul compared findings from studies with more and less experienced sea combat personnel to predictions expected from these models. To account for these and other expert/novice differences available in the literature, they concluded that a comprehensive theory of NDM must include constructs that reference both memorial structures ("schemata") and current representations ("mental models," also called "situation assessments" by others).

These authors have developed a comprehensive model of NDM called "recognition-primed decision making as schemata-driven mental modeling." Because the model specifies the nature of influence on one another from each of five elements in the decision episode, it is possible to generate hypotheses that can be tested. This is significant because lack of testable NDM theories is a concern that has been voiced by NDM researchers. The value added by theory/model development and testing is the same as in other lines of research—this is how we can advance and refine our understanding of phenomena under scrutiny. However, theory development is of particular importance in NDM research because so much of our work is funded for applications, or to solve specific problems, rather than for basic research or comprehensive theory development. At both the 1989 and the 1994 NDM conferences, researchers raised a concern that our theoretical progress is slower than we would like, but that it is taking place. There is a general agreement that we should try to draw on applied project funding to cover theoretical testing whenever possible. Examples can be found in many of the contributions to this volume.

Related Lines of Research

A second theme in NDM research that cuts across the chapters that follow is that we take an interdisciplinary approach to our work. Several related lines of research are discussed, most notably expertise.

Expertise. The study of expertise is one of the most closely allied resea...

Table of contents

• Cover Page
• Half Title Page
• Series Page
• Title Page
• Copyright Page
• Contents
• Series Editor's Preface
• Preface
• Conference Participants and Affiliations
• Part I About Naturalistic Decision Making
• Part II Applications of NDM Research: Perspectives from Panels of Applied Researchers
• Part III Research Reports
• Part IV Methodological and Theoretical Considerations
• Part V Naturalistic Decision Making: Where Are We Going?
• Author Index
• Subject Index