Handbook of Human Factors Testing and Evaluation

Samuel G. Charlton, Thomas G. O'Brien, Samuel G. Charlton, Thomas G. O'Brien

• Publisher: CRC Press
• Year: 2019
• Print ISBN: 9780805832914
• eBook ISBN: 9781000751574
• Edition: 2
• Topic: Technology & Engineering
• Subtopic: Software Development
• Index: Technology & Engineering

Like the first edition, the revision of this successful Handbook responds to the growing need for specific tools and methods for testing and evaluating human-system interfaces. Indications are that the market for information on these tools and applications will continue to grow in the 21st century. One of the goals of offering a second edition is to expand and emphasize the application chapters, providing contemporary examples of human factors test and evaluation (HFTE) enterprises across a range of systems and environments. Coverage of the standard tools and techniques used in HFTE have been updated as well. New features of the Handbook of Human Factors Testing and Evaluation include:
*new chapters covering human performance testing, manufacturing ergonomics, anthropometry, generative design methods, and usability testing;
*updated tools and techniques for modeling, simulation, embedded testing, training assessment, and psychophysiological measurement;
*new applications chapters presenting human factors testing examples in aviation and avionics, forestry, road safety, and software systems; and
*more examples, illustrations, graphics and tables have been added. The orientation of the current work has been toward breadth of coverage rather than in-depth treatment of a few issues or techniques. Experienced testers will find much that is familiar, as well as new tools, creative approaches, and a rekindled enthusiasm. Newcomers will discover the diversity of issues, methods, and creative approaches that make up the field. In addition, the book is written in such a way that individuals outside the profession should learn the intrinsic value and pleasure in ensuring safe, efficient, and effective operation, as well as increased user satisfaction through HFTE.

Part I

TOOLS AND TECHNIQUES

INTRODUCTION

We have devoted the first half of this second edition to the description of the methods and approaches that have been used in human factors test and evaluation (HFTE). Some of the chapters are updated versions from the first edition of the handbook, others are new treatments of subjects covered previously, and others are new additions to our coverage of HFTE tools and techniques. Undoubtedly, we will have missed some topics; that is an inevitability in any treatment of our diverse discipline. Although we may not be able to provide an exhaustive account of every HFTE technique, nor treat every topic in the depth deserved, we have sought to strike a balance between the scope of our coverage and the detail provided. In other words, while these chapters will not suffice in making someone an expert in HFTE, there will be enough information for a reader to grasp the essentials of the methods. It is hoped that by the end of this section, even the casual reader will become a more knowledgeable consumer of HFTE.

Our coverage of these methods begins with a brief historical overview of HFTE. As with many other technology driven enterprises, HFTE has a legacy to the machinery of armed conflict. Our first chapter describes these historical roots and how HFTE evolved from the somewhat informal practice of field testing weaponry to the development of more systematic time and motion studies in industrial settings. Our history goes on to describe how the practice of HFTE as a separate discipline was initiated by the establishment of government-directed laboratories and acquisition procedures, and it concludes by covering the recent emergence of modeling, simulation, and usability testing methods.

The current processes and procedures of HFTE are described in detail in the second chapter. We describe the formalized system development process used by government agencies, the corresponding practices used by private industries, and the newer practice of integrated product development. The position of HFTE has shifted from the end of the design and engineering chain to become more thoroughly integrated with the system development process; HFTE activities occur in the earliest stages of concept exploration through to production. This chapter discusses the activities and issues confronting human factors testers at each stage of the system development process.

Although it can rightly be said that there is no standard formula for planning and conducting a human factors test, the SITE methodology presented in the third chapter outlines one approach that has been used to introduce students and new human factors professionals to the testing discipline. The approach is essentially an organizational aid for selecting human factors test measures and placing test results in the context of product development and performance. The chapter provides several examples of how the approach has been implemented in both student projects and full system tests.

Chapter 4 covers test support documentation and builds on the planning and reporting theme, providing the reader with a guide through the host of human factors standards and HFTE reporting requirements. From the design, performance, and process standards used in the early stages of test planning through the test plans themselves, and ultimately to the myriad of test reporting formats, this chapter provides many descriptions and examples that will be helpful to both new and experienced testers.

The fifth chapter addresses one of the classic human factors test methods: human performance testing. This chapter begins by describing the theory and discussing 50 years of history associated with performance testing. The chapter provides the reader with detailed guidance for designing and assessing performance tests, with special attention given to attaining adequate test reliability. The final part of the chapter addresses how to administer performance tests, particularly in the critical area of establishing standardized testing conditions.

Another classic HFTE methodology is the measurement of cognitive states. Chapter 6 explores some of the techniques for measuring cognitive states, particularly cognitive workload and situation awareness. Here we examine the theoretical and definitional issues that underlie the measurement of these elusive but important determinants of human behavior. The chapter also outlines several criteria that can be used to compare the various measures of workload and situation awareness currently in use. Descriptions of the most frequently used workload and situation measures are provided, along with advice on how to collect and analyze data with each of them.

The subject of chapter 7 is psychophysiological measurement, another classic HFTE methodology. The author provides an in-depth look at an extensive range of these measures, organized according to the type of system application where they can best be employed. Because the use of psychophysiological measures is still relatively new to many areas of HFTE, the chapter contains thoughtful guidance on the types of equipment available, the practicalities involved in using the measures for best effect, and a brief bibliography for additional reading. A thorough review of the latest HFTE literature, as well as a look at what the future holds for psychophysiological methods, will bring the reader current with the latest developments in this challenging field.

In the chapter describing the purposes, processes, and products of measurement in manufacturing ergonomics, the author introduces us to a variety of ergonomics measurement and analysis techniques and provides an overview of the design principles of the ergonomics measurement tools themselves. For example, the issues of resolution and precision of measurement, decision thresholds, and interactions in complex stress-exposure measures, are all clearly described. Finally, the chapter covers the key intervention opportunities for ergonomists in manufacturing settings.

The increasing emphasis on performing HFTE as early as possible has meant testers have had to turn to methods such as mock-ups, models, and simulations to evaluate designs, even before the first article is produced. Chapter 9 provides the reader with a wealth of information on these tools, from the old standards, such as plywood mock-ups, to the latest in computer-based manikins and operator performance models. The authors also describe various evaluation support tools that can help the tester select the most appropriate tool for a given test. The chapter goes on to describe various embedded testing techniques and the latest trends and developments in HFTE tools.

The final two chapters in this section present two commonly used, but frequently unpopular, HFTE techniques: questionnaires and environmental measurement. These techniques feature to some degree in nearly every human factors test, but the ease with which they can be poorly performed, along with the perceived difficulties associated with doing them properly, means that these methods are rarely employed to their full potential. These two chapters are designed to acquaint the reader with the necessary basics of how to properly use these techniques. It is our hope that removing many of the misconceptions surrounding questionnaires and environmental measurement will lead to both greater care in their application and to a better appreciation of their potential benefits to HFTE.

Chapter 1

Human Factors Testing and Evaluation: An Historical Perspective

Thomas G. O’Brien

O’Brien & Associates

David Meister

Human Factors Consultant

History gives us an appreciation of where we are today and what it took to get us here. Like anything else, HFTE has a history, and it is important, as well as interesting, to understand and appreciate the early techniques that led to modern practices.

Where did the formal practice of HFTE begin? For argument’s sake, let’s say it began about the same time as the discipline of human factors engineering (HFE). Although there are arguably some differences in definitions, for the purpose of discussion, we consider engineering psychology and ergonomics synonymous with HFE, and later, we encompass all of these under the general term human factors, to include, in addition, specialties such as training, manpower, personnel, and safety. Human factors was begot from a number of other disciplines, as Meister points out in his history of human factors and ergonomics (Meister, 1999). For this reason, to assign a chronological beginning would be difficult, if not impossible. Indeed, the origin of human factors has been a matter of recent debate. Meister provides a broad historical explanation of the discipline, outlining the morphology of human factors and ergonomics in various cultures and providing a great deal of insight into its early days as the natural offspring of experimental psychology and the progenitor of what might be the next phase of evolution—usability engineering. But what of HFTE?

Like a two-sided coin, HFTE compliments the design aspect of human factors. Without it, we might never know if our system or component design meets our intended technical and operational objectives. Moreover, without HFTE, it would be difficult to impart measurable improvements to our design. And so, like all methodologists, we should define HFTE. Our simple definition is “Human factors testing and evaluation is a set of methodologies to characterize, measure, assess, and evaluate the technical merit and operational effectiveness and suitability of any human-system interface.” A more detailed definition would include training, manpower and personnel, health hazards, and perhaps others.

Measurement is as integral to human factors as it is to every scientific and engineering discipline. Indeed, it would be correct to say that HFTE preceded the development of the discipline because it is only when one can measure phenomena that one can organize these phenomena into a discipline. Ordinarily, we associate human factors with World War II; however, to get a more complete understanding of the underlying precepts and causes leading to our contemporary notion of HFTE, we must reach back to the days prior to World War II and even World War I. For example, in the American Civil War, organized efforts emerged to test how well soldiers and sailors performed with new weapons and accoutrements. Indeed, throughout written history, makers of weapons and apparel tested their wares for designs that best suited the owner in form, fit, and function. For example, a suit of armor was no good to the knight unless it fit him.

In the first edition of this book (O’Brien & Charlton, 1996), we described a general historical model of HFTE comprising premodern and modern periods. In this edition, we refine our argument to make the point that HFTE has roots, deep and penetrating, reaching well before the 19th century. The examples we cite are mostly military because, in the early days, the military aspect of technology was more important to the movers and shakers than it was for domestic use. As with the advancement of military applications, there is a corresponding, later concern for human factors, particularly in medicine. Today, the application of HFTE is as well-founded in the industrial environments as it is in the military.

THE PREMODERN ERA

Prior to the Second World War, the only test of human fit to the machine was an informal process of trial and error. The human either could or could not function with the machine. If the human could not, another human was selected until the right match was made. The operator was viewed as a convenient and dispensable element whose importance was limited to operating the machine. While this selection model (Brogan, Hedge, McElroy, & Katznelson, 1981) worked well with mass-produced weapons and equipment, we might argue that other, more sophisticated methods were engaged for individual components, for example, body armor.

Although scant written evidence survives to support the notion that HFTE was anything more than trial and error prior to World War II, there is evidence of the use of tools to measure bodily parts, for example, calipers to measure the cranium for fitting helmets. Also, chroniclers of the Middle Ages tell of inventors who look the time to record the effects of different equipment designs on the ability of humans to perform tasks. Recording the pattern of arrows striking a target set at measured distances from archers helped 13th century developers determine the optimal length and curvature for that weapon, considering an average man’s height, reach, and draw strength. Today, we would call this human performance testing. The results of such testing ultimately contributed to the invention of the crossbow, which was more powerful. Some crossbows required a windlass (i.e., a hand-crank mechanism for drawing back the crossbow string into firing pos...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Dedication
• Table of Contents
• Preface
• Contributors
• PART I. TOOLS AND TECHNIQUES
• PART II. APPLICATIONS
• Author Index
• Subject Index