Developing Organizational Simulations

A Guide for Practitioners, Students, and Researchers

George C. Thornton III, Rose A. Mueller-Hanson, Deborah E. Rupp

• Publisher: Routledge
• Year: 2017
• Print ISBN:
  9781138119291
• eBook ISBN:
  9781317312901
• Topic: Psychology
• Subtopic: History & Theory in Psychology
• Index: Psychology

This second edition of Developing Organizational Simulations provides a concise source of information on effective and practical methods for constructing simulation exercises for the assessment of psychological characteristics relevant to effectiveness in work organizations. Incorporating new additions such as the multiple ways technology can be used in the design, delivery, scoring, and evaluating of simulation exercises, as well as the delivery of feedback based on the results, this book is user-friendly with practical how-to guidance, including many graphics, boxes, and examples. This book is ideal for practitioners, consultants, HR specialists, students, and researchers in need of guidance developing organizational simulations for personnel selection, promotion, diagnosis, training, or research. It is also suited for courses, workshops, and training programs in testing and measurement, personnel selection, training and development, and research methodology.

Part I

Concepts and Principles

This section provides background, concepts, and information relevant to the development of many different types of simulation exercises. Chapter 1 defines simulations in comparison with related assessment techniques, explains the importance of simulations including ways simulations are used, and the importance of developing good simulations. Chapter 2 provides a model for simulation development and a set of specific steps for building simulations. Chapter 3 summarizes several conceptual frameworks and their practical implications for the development of effective simulations. Chapter 4 discusses issues relevant for developing simulations and offers many tips for writing simulation scenarios. Tips for preparing simulations for use in cross-cultural settings are provided. Chapter 5 describes several ways technology has been used to enhance the effectiveness and efficiency of simulations, including when technology is and is not feasible and useful.

1

Simulations in Context

This book is designed to help human resource managers, industrial-organizational psychologists, consultants, researchers, and students develop simulations to assess, develop, and evaluate various attributes relevant to effective performance in organizations. When developing simulations, the objective is to design an assessment process that:

• Presents participants with complex and realistic situations
• Engages and motivates participants to demonstrate “honest” responses
• Elicits behavior relevant to specific performance dimensions or task completion
• Allows reliable and valid observation and evaluation of behavior
• Helps managers and others make judgments about performance effectiveness for the purposes of assessment, diagnosis, and development of individuals and teams.

The first section of the book begins with some basic terminology, a model and specific steps for the simulation construction process, several theories and conceptual frameworks that provide practical implications for the development of simulations, a discussion of the ways technology makes simulations more effective and efficient, and numerous practical tips for building simulations, designing simulations for cross-cultural settings, and incorporating technology. This set of chapters is based on the premise that good simulations are grounded in systematic planning, solid theory, and basic research.

The second section, which comprises the core of the book, consists of chapters providing practical guidance for building several types of simulations. Each of these chapters provides background, describes steps in building the simulation content (including specific substantive material), and discusses special issues pertinent to the class of simulations being discussed. In addition, each chapter describes how the class of simulations can be used for selected purposes and how selected aspects of technology can be incorporated into those types of simulations.

The third section of the book includes chapters on how to train various staff required for the effective implementation of simulations (including administrators, observers/assessors, role players, feedback providers/coaches, and administrative/technical support staff); how to evaluate the effectiveness of simulations according to their intended purpose; and the importance of complying with various professional guidelines, standards, regulations, and laws related to the use of simulations as a form of behavioral assessment. A final chapter summarizes the key points made across the entire book. Sprinkled throughout the book are numerous boxed features offering tips, personal accounts, and anecdotes describing successes and failures in the development and use of simulations.

It is our strong belief, based on theory, research, and practice, that simulations provide many advantages over other techniques used for assessment, training, and research. As assessment techniques, simulations provide opportunities to present rich sets of complex stimuli to challenge participants, in comparison to the short descriptions of behavior used within traditional tests and interviews. Simulations provide the opportunity to observe and evaluate complex, overt behavioral responses in comparison to pre-provided choices common in other forms of assessment. Behavioral assessments provide means to assess and develop complex managerial, leadership, and interpersonal competencies, in comparison with the typical, narrow attributes measured by cognitive ability tests and personality questionnaires. As a result, behavioral assessment contributes to effective selection and promotional decisions and to thorough diagnosis of development needs. As used for training, simulations provide safe, experiential learning environments within which participants can make mistakes and try out new approaches to working and leading. Simulations provide opportunities for individuals to practice behavioral skills and receive developmental feedback, including specific behaviors for improvement. As a research tool, simulations allow scientists to observe and experiment with behavior in realistic yet controlled environments. They allow for a compromise between the unrealistic laboratory environments and the often undecipherable complexity of real-world settings.

What Are Simulations?

A simulation is an activity resembling an organizational situation in which participants are presented with complex stimuli and required to display complex overt behavior. The complex stimuli may be written case material, actions of a role player depicting another organizational member, other participants in the simulation, or an assessor who interacts with the participant. Participants may be presented with printed materials; written, visual, and/or interactive materials presented via computers; or other types of tools or equipment. Expected overt responses may include writing a report, leading a group discussion, communicating ideas, directing others in a complex team-based task, or working physically with equipment. Various methods are used to evaluate participants’ behavioral responses. Commonly, assessors observe participant behavior and rate its effectiveness.

An example of a simulation is a Leaderless Group Discussion (LGD). There are many forms of LGDs, which we describe in Chapters 8 and 9. Four to six participants are given a list of problems the organization is facing and asked to make recommendations for how core issues might be addressed. They are given a certain amount of time (e.g., 45 minutes) to individually present their ideas, discuss various options, and prepare a written report containing the group’s recommendations. The behavior of the participants is observed by assessors who evaluate each participant’s skills on relevant dimensions or competencies, such as oral communication, problem solving, and leadership. Assessors can also evaluate the quality of the recommended solutions.

Simulations are a type of situational test. Like any test, simulations consist of standardized procedures for eliciting a sample of behavior from the examinee and for providing a quantitative measure of some defined attribute(s). Simulations are different from traditional paper-and-pencil or online tests in several ways. In contrast to the low level of fidelity of typical tests to the job, simulations involve moderate to high fidelity to the performance domain of interest. The situations vary in their resemblance to the job, organizational context, and environment, from moderately similar to highly similar to a portion of a job in a specific organization in a specified culture. Exact replicas of the focal job are often called work samples. Thus, simulations have more fidelity than traditional paper-and-pencil and online tests, but less fidelity than a work sample.

A second difference between simulations and traditional tests is the level of response fidelity to the job. Standard tests may elicit complex thought processes required to solve a verbal or quantitative problem or to respond to a thought-provoking question about one’s personality, yet not require the display of overt, job-relevant behavior. With these methods, the only overt behavior shown is the simple behavior of selecting an alternative among multiple options or indicating agreement with a pre-provided statement. In contrast, behavioral simulations require the display of high-fidelity behaviors very similar to the complex, overt, and observable on-the behaviors required by the focal job (e.g., delivering a presentation or speaking with others).

The third difference between behavioral simulations and traditional forms of testing is that whereas traditional tests are usually objectively scored by counting the number of correct answers, or the strength of endorsement to items that have been determined to be indicative of a personality characteristic, most simulations require trained assessors to observe behavior, classify behavior into categories being assessed, and make judgments about the level of proficiency in each category assessed. In this context, the “category” may be a dimension (that is, some human attribute) or a task being completed in the simulation. A term related to dimension is “competency,” and we will use these words interchangeably throughout the book. In the human resource management literature, competency is used in variety of ways to mean a variety of concepts ranging from a human attribute to an organizational strength or strategic goal. In this book, competency refers to a cluster of behaviors encompassing a human attribute of some kind. Dimensions and tasks are different from traits: traits are commonly understood to be underlying, causal characteristics that are predictive of behavior, whereas constructs assessed in simulations are defined in terms of observable behaviors that reflect a combination of knowledge, skills, abilities, and other attributes. The complex social judgment required in the process of observing, classifying, and evaluating behaviors highlights the importance of the observer/assessor in this special form of psychological assessment, and demonstrates the need for the careful training of these individuals. Chapter 13 is devoted to the process of selecting and training assessors.

Simulations in Context

As mentioned previously, assessments vary in terms of their fidelity, ranging from samples of behaviors to signs of behavior (Wernimont & Campbell, 1968). This is depicted in Figure 1.1 (and discussed in more depth in Chapter 3). It should be noted that even within the category “Simulations,” there exists a range of fidelity (in terms of the stimulus materials, response modes, and material delivery).

Comparisons of Major Assessment Techniques

Table 1.1 provides comparisons between simulations and alternative testing and assessment methods. In making such comparisons, we are assuming a fairly high-fidelity simulation as our point of comparison. Comparisons may differ if lower-fidelity simulations are being compared to alternatives. As is shown, in comparison with cognitive ability tests (Kuncel & Beatty, 2013), a simulation can measure multiple competencies, likely with less adverse impact. In comparison with self-report personality questionnaires (Hough & Connelly, 2013), simulations are harder to fake and are seen as more fair by examinees. In comparison with personal history forms (Schmitt & Goluovich, 2013), simulations reveal here-and-now behavior. In comparison with multi-source feedback methods (Hoffman & Baldwin, 2011), simulations provide standardized input from trained assessors. In comparison with background interviews (Dipboye, Wooten, & Halverson, 2004), simulations provide observations of current behavioral skills that are difficult to fake.

Figure 1.1 A Continuum of Fidelity of Assessment Techniques

Table 1.1 A Comparison of Simulations and Other Assessment Methods

Alternative Assessment Techniques	Simulations
Cognitive ability tests — Measures reasoning abilities — Presents abstract problems — Abilities implied from responses — Often cause adverse impact	— Can measure many abilities — Presents work-related problems — Requires demonstration of overt behavior/ability — Causes less adverse impact
Personality inventories — Measures stable traits — Easy to fake — Is a self-description — Has low face validity	— Can measure both stable traits and developable skills — Harder to fake — Provides description by observers — Participants respond favorably
Personal history forms — Covers a wide range of experiences — Collects background information — Aspects of a person’s past are outside of his or her control	— Reveals here-and-now behavior — Shows “can do” abilities — Assesses behavior under the control of the candidate
Work samples — Directly measures ability to do job tasks — High levels of fidelity to job — Appropriate for entry-level jobs — Measures current skill	— Moderate/high leve...

Table of contents

• Cover
• Title
• Copyright
• Dedication
• Contents
• List of Illustrations
• Series Foreword
• Preface
• Acknowledgments
• Part I Concepts and Principles
• Part II Categories and Types of Simulations
• Part III Implementation and Evaluation
• Index