Automation and Human Performance

Theory and Applications

Raja Parasuraman, Mustapha Mouloua

• Publisher: CRC Press
• Year: 2018
• eBook ISBN:
  9781351465045
• Topic: Ciencia de la computación
• Subtopic: Ingeniería industrial

There is perhaps no facet of modern society where the influence of computer automation has not been felt. Flight management systems for pilots, diagnostic and surgical aids for physicians, navigational displays for drivers, and decision-aiding systems for air-traffic controllers, represent only a few of the numerous domains in which powerful new automation technologies have been introduced. The benefits that have been reaped from this technological revolution have been many. At the same time, automation has not always worked as planned by designers, and many problems have arisen--from minor inefficiencies of operation to large-scale, catastrophic accidents. Understanding how humans interact with automation is vital for the successful design of new automated systems that are both safe and efficient.

The influence of automation technology on human performance has often been investigated in a fragmentary, isolated manner, with investigators conducting disconnected studies in different domains. There has been little contact between these endeavors, although principles gleaned from one domain may have implications for another. Also, with a few exceptions, the research has tended to be empirical and only theory-driven. In recent years, however, various groups of investigators have begun to examine human performance in automated systems in general and to develop theories of human interaction with automation technology.

This book presents the current theories and assesses the impact of automation on different aspects of human performance. Both basic and applied research is presented to highlight the general principles of human-computer interaction in several domains where automation technologies are widely implemented. The major premise is that a broad-based, theory-driven approach will have significant implications for the effective design of both current and future automation technologies. This volume will be of considerable value to researchers in human

I

Theories and Major Concepts

1	Decomposing Automation: Apparent Simplicity, Real Complexity

David D. Woods

The Ohio State University

INTRODUCTION

We usually focus on the perceived benefits of new automated or computerized devices. Our fascination with the possibilities afforded by technology in general often obscures the fact that new computerized and automated devices also create new burdens and complexities for the individuals and teams of practitioners responsible for operating, troubleshooting, and managing high-consequence systems. First, the demands may involve new or changed tasks such as device setup and initialization, configuration control, or operating sequences. Second, cognitive demands change as well, creating new interface management tasks, new attentional demands, the need to track automated device state and performance, new communication or coordination tasks, and new knowledge requirements. Third, the role of people in the system changes as new technology is introduced. Practitioners may function more as supervisory controllers, monitoring and instructing lower-order automated systems. New forms of cooperation and coordination emerge when automated systems are capable of independent action. Fourth, new technology links together different parts that were formerly less connected. As more data flows into some parts of a system, the result is often data overload. Coupling a more extensive system more tightly together can produce new patterns of system failure. As technology change occurs we must not forget that the price of new benefits is often a significant increase in operational complexity. Fifth, the reverberations of technology change, especially the new burdens and complexities, are often underappredated by the advocates of technology change. But their consequences determine when, where, and how technology change will succeed.

My colleagues and I have been studying the impact of technology change on practitioners—those people who do cognitive work to monitor, diagnoses and manage complex systems—pilots, anesthesiologists, process plant operators, space flight controllers (e.g., Woods, Johanessen, Cook, & Sarter, 1994). In these investigations we have seen that technology change produces a complex set of effects. In other words, automation is a wrapped package—a package that consists of changes on many different dimensions bundled together as a hardware/software system. When new automated systems are introduced into a field of practice, change is precipitated along multiple dimensions. In this chapter I examine the reverberations that technology change produces along several different dimensions:

• Automation seen as more autonomous machine agents. Introducing automated and intelligent agents into a larger system in effect changes the team composition. It changes how human supervisors coordinate their activities with those of the machine agents. Miscommunications and poor feedback about the activities of automated subsystems have been part of accident scenarios in highly automated domains.

• Automation seen as an increase in flexibility. As system developers, we can provide users with high degrees of flexibility through multiple options and modes. We also have the ability to place multiple virtual devices on one physical platform so that a single device will be used in many contexts that can differ substantially. But do these flexibilities create new burdens on practitioners, burdens that can lead to predictable forms of error?

• Automation seen as more computerization. Technology change often means that people shift to multifunction computer-based interfaces as the means for acquiring information and utilizing new resources. Poor design of the computer interface can force users to devote cognitive resources to the interface itself, asking questions such as: Where is the data I want? What does the interface allow me to do? How do I navigate to that display? What set of instructions will get the computer to understand my intention? Successful computer interfaces (e.g., visualization, direct manipulation) help users focus on their task without cognitive resources (attention, knowledge, workload) being devoted to the interface per se.

• Automation seen as an increase in coupling across diverse parts and agents of a system. Tighter coupling between parts propagates effects throughout the system more rapidly. This can produce efficiency benefits by reducing transfer costs, but it also means that problems have greater and more complex effects, effects that can propagate quickly. But when automated partners are strong, silent, clumsy, and difficult to direct, then handling these demands becomes more difficult. The result is coordination failures and new forms of system failure.

• Much technology change is justified, at least in part, based on claims about the impact of technology on human performance—the new system will “reduce workload,” “help practitioners focus on the important part of the job,” “decrease errors,” and so on. But these claims often go unexamined. A number of studies have examined the impact of automation on the cognition and behavior of human practitioners. These studies, many of which are discussed in other chapters of this book, have shown repeatedly that systems introduced to aid practitioners in fact created new complexities and new types of error traps.

• The success of new technology depends on how it affects the people in the field of practice. The dimensions addressed earlier represent some of the ways that technology change can have surprising impacts on human and system performance. By closely examining the reverberations of technology change we can better steer the possibilities of new technology into fruitful directions.

HOW TO MAKE AUTOMATED SYSTEMS TEAM PLAYERS

Heuristic and algorithmic technologies expand the range of subtasks and cognitive activities that can be automated. Automated resources can, in principle, offload practitioner tasks. Computerized systems can be developed that assess or diagnose the situation at hand, alerting practitioners to various concerns and advising practitioners on possible responses.

Our image of these new machine capabilities is that of a machine alone, rapt in thought and action. But the reality is that automated subtasks exist in a larger context of interconnected tasks and multiple actors. Introducing automated and intelligent agents into a larger system changes the composition of the distributed system of monitors and managers and shifts the human’s role within that cooperative ensemble (Hutchins, 1994). In effect, these “intelligent” machines create joint cognitive systems that distribute cognitive work across multiple agents (Hutchins, 1990; Roth, Bennett, & Woods, 1987). It seems paradoxical, but studies of the impact of automation reveal that design of automated systems is really the design of a new human-machine cooperative system. The design of automated systems is really the design of a team and requires provisions for the coordination between machine agents and human practitioners (e.g., Layton, Smith, & McCoy, 1994).

However, research on human interaction with automation in many domains, including aviation and anesthesiology, has shown that automated systems often fail to function as team players (Billings, 1991; Malin et al., 1991; Sarter & Woods, 1994b). To summarize the data, automated systems that are strong, silent, clumsy, and difficult to direct are not team players. Automated systems are:

Strong when they can act autonomously.

Silent when they provide poor feedback about their activities and intentions.

Clumsy when they interrupt their human partners during high workload or high criticality periods, or when they add new mental burdens during these high-tempo periods.

Difficult to direct when it is costly for the human supervisor to instruct the automation about how to change as circumstances change.

Systems with these characteristics create new problems for their human partners and new forms of system failure.

“Strong” automation refers to two properties of machine agents. In simpler devices, each system activity was dependent on immediate operator input. As the power of automated systems increases, machine agents, once they are instructed and activated, are capable of carrying out long sequences of tasks without further user interventions. In other words, automated systems can differ in degree of autonomy (Woods, 1993). Automated systems also can differ in degree of authority. This means that the automated system is capable of taking over control of the monitored process from another agent if it decides that intervention is warranted based on its perception of the situation and its internal criteria (Sarter & Woods, 1994a).

Increasing autonomy and authority create new monitoring and coordination demands for humans in the system (Norman, 1990; Sarter & Woods, 1995; Wiener, 1989). Human supervisors have to keep track of the status and activities of their automated partners. For example, consider the diagnostic situation in a multi-agent environment, when one notices an anomaly in a process being monitored (Woods, 1994). Is the anomaly an indication of an underlying fault, or does the anomaly indicate some activity by another agent in the system, unexpected by this monitor? In fact, in a number of different settings, we observe that human practitioners respond to anomalies by first checking for what other agents have been or are doing to the process jointly managed (Johannesen, Cook, & Woods, 1994).

When machine agents have high autonomy, they will act in the absence of immediate user input. Human practitioners have to anticipate how the automated system will behave as circumstances change. Depending on the complexity of the system and the feedback about system activities, this may be difficult. As one commentator has put it, the most common questions people ask about their automated partners are: What is it doing? Why is it doing that? What will it do next? How in the world did we get into that mode? (Wiener, 1989). These questions are indications of coordination breakdowns—what has been termed “automation surprises.” Automation surprises are situations where automated systems act in some way outside of the expectations of their human supervisors. Data from studies of these surprises in aviation and medicine (Moll van Charante, Cook, Woods, Yue, & Howie, 1993; Norman, 1990; Sarter & Woods, 1994b) indicate that poor feedback about the activities of automated systems to their human partners is an important contributor to these problems.

Autonomy and authority are properties that convey an agentlike status on the system from the point of view of human observers. This raises an important point. Automated systems have two kinds of interpretations. Based on knowledge of underlying mechanisms, an automated system is deterministic and predictable. However, those who monitor or interact with the system in context may perceive the system very differently. For example, with the benefit of knowledge of outcome and no time pressure, one can retrospectively show how a system’s behavior was deterministic. But as system complexity increases, and depending on the feedback mechanisms available, predicting the system’s behavior in context may be much more difficult.

A user’s perception of the device depends on an interaction between its capabilities and the feedback mechanisms that influence what is observable about system behavior in relation to events in the environment. What feedback is available depends on the “image” the device presents to users (Norman, 1988). When a device is complex, has high autonomy and authority, and provides weak feedback about its activities (what has been termed “low observability”), it can create the image of an animate agent capable of independent perception and willful action. We refer to this as the perceived animacy of the automated system. In effect, the system, although determinate from one perspective, seems to behave as if it were an animate agent capable of activities independent of the operator (Sarter & Woods, 1994a).

Flightdeck automation on commercial transport jets illustrates how autonomy combined with low observability can create the perception of animacy (Sarter & Woods, 1994b). Pilots sometimes experience difficulties with tracking system behavior in situations that involve indirect mode transitions. In these situations, the system changes its behavior independent of any immediate pilot instructions. The system acts in response to reaching a preset target (e.g., leveling off at a target altitude) or because an envelope protection threshold is crossed. In other words, based on the programmed mechanisms, the system “realizes” the need for a mode change, carries it out without requesting pilot consent, and provides only weak feedback about the change or the implications of the change for future aircraft behavior. It is in this type of situation that pilots are known to ask questions such as: What is it doing? Why did it do this? What will it do next? (Wiener, 1989). These are questions one asks about another agent with an agenda of its own and an agent that does not communicate very well.

Much work in this area has noted that poor feedback on system status and behavior is at the heart of automation surprises. But what does it mean to say “poor feedback?” When we take a close look at the data provided to the operators of many advanced systems, it becomes quite clear that the amount of data available to the human is increasing. All of the necessary data to build a picture of their automated partner’s activities is present in general. But the effectiveness of this data depends on the cognitive work needed to turn it into a coherent interpretation in context.

Effective feedback depends on more than display formats; it is a relation among the system’s function, the image the system presents to outsiders, and the observer embedded in an evolving context (Woods, 1995). As a result, it is better to refer to interface and feedback issues in terms of observability. This term captures the fundamental relationship among thing observed, observer, and context of observation that is fundamental to effective feedback. Observability depends on the cognitive work needed to extract meaning from the data available. We, as researchers, need to make progress on better ways to measure this property of cognitive systems.

Because automated systems are deterministic, if one has complete knowledge of how a system works, complete recall of the past instructions given to the system, and total awareness of environmental conditions, then one can project accurately the behavior of the automated partner. However, as the system becomes more complex, projecting its behavior also becomes more cognitively challenging. One has to have an accurate model of how the system works, one has to call to mind the portions of this knowledge that are relevant for the current situation, one has to recall past instructions that may have occurred some time ago and may have been provided by someone else, one has to be aware of the current and projected state of various parameters that are inputs to the automation, one has to monitor the activities of the automated system, and one has to integrate all of this information and knowledge together in order to project how the automation will behave in the future. As a result, an automated system can look very different from the perspective of a user in context as compared to an analyst taking a bird’s-eye view with knowledge of outcome. The latter will see how the system’s behavior was a direct and natural result of previous instructions and current state; the former will see a system that appears to do surprising things on its own. This is the paradox of perceived animacy of automated systems that have high autonomy and authority but low observability (Fig. 1.1). This situation has strong implications for error analysis and incident reconstruction (Woods et al., 1994).

FIG. 1.1. A paradox associated with perceived animacy. Automated systems that have high autonomy and authority but low observability appear to behave as if they are animate agents capable of activities independent of the operator. However, such systems are deterministic, and their behavior is predictable if one has complete and available knowledge of how the system works, complete recall of the past instructions given to the system, and total awareness of the situation and environmental conditions.

The trend in automation seems to be for greater increases in system autonomy and authority, whereas feedback...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Table of Contents
• Series Foreword
• Foreword
• Preface
• Contributors
• PART I: THEORIES AND MAJOR CONCEPTS
• PART II: ASSESSMENT OF HUMAN PERFORMANCE IN AUTOMATED SYSTEMS
• PART III: APPLICATIONS
• PART IV: FUTURE TRENDS
• Author Index
• Subject Index