Performance Measurement

Current Perspectives and Future Challenges

Winston Bennett, Charles E. Lance, David J. Woehr

• Publisher: Psychology Press
• Year: 2014
• eBook ISBN: 9781317824534
• Topic: Psychology
• Subtopic: Industrial & Organizational Psychology
• Index: Psychology

Over the course of the past few years, teaching, research, and practice has underscored the importance of performance measurement and criterion development as topics of great interest, considerable debate, and some misunderstanding. It has also become clear that the field needs to address a compendium of research, applications, and issues.

Performance Measurement: Current Perspectives and Future Challenges brings together internationally recognized leaders in the field and each examines the subject matter in a way that has never been done--focusing on the dynamic nature of work and the tremendous demands being placed on assessment and measurement as core organizational activities. It also uniquely uses their expertise to provide critical pointers to not only the practical implications of work in the field, but also to the new and continuing issues to be addressed and research to be conducted.

The book will be useful to both scientists and practitioners.

1
Introduction

Winston Bennett, Jr.
U.S. Air Force Research Laboratory,
Human Effectiveness Directorate, Mesa, Arizona
Charles E. Lance
The University of Georgia
David J. Woehr
The University of Tennessee-Knoxville

The measurement of individual work performance is one of the cornerstones of organizational research and practice. Performance measurement and prediction play a vital role in nearly all personnel decisions as well as many other organizational decisions. The importance of individual performance measurement is certainly reflected in the literature. The "criterion problem"—the inherent difficulty associated with criterion development, measurement, and performance evaluation—continues to be a mainstay of both the popular and academic literature. Evident in this literature is the fact that the criterion problem continues to be one of the most vexing issues facing organizational researchers and practitioners today.

Our goal in this book is to summarize critical issues relating to criterion development and performance measurement with a focus on both existing issues and future challenges. We begin with a review of the problems, issues, and concerns surrounding the criterion problem. These issues and concerns are then examined from a variety of perspectives and contexts—models of work performance, alternate measurement approaches, measuring performance in teams, multisource systems, and the impact of cross-cultural contexts. Finally, directions for future research and practice are examined.

Overview of the Chapters

Each of the chapters has been written to accomplish three specific goals: first, to identify and elaborate a specific and critical issue related to criterion development and/or measurement; second, to review the pertinent literature and summarize what we do and do not know about the issue today; and third, to recommend a defensible and practical research agenda that will help to address the issue as identified. The chapters are grouped into four parts according to the themes addressed and the issues raised.

The first part, composed of three chapters, focuses on problems, prospects, and continuing issues in criterion development and performance measurement. Austin and Crespin (chap. 2) describe and discuss the multiple problems of criteria as opportunities rather than as threats. This chapter is a follow-up to Austin and Villanova (1992), who present a review of the literature from 1917 to 1992 organized in roughly 20-year intervals focusing on dimensions, research methods, and categorizing frameworks. Austin and Villanova conclude that the early dominance of inductive research, best illustrated in Viteles' (1932) treatise Industrial Psychology, was being replaced by a complementary balancing of data-driven and theory-grounded strategies (Alliger, 1992). They also suggest considering the construct validity of criteria per se is a fruitful research topic, but that criterion research and performance appraisal practice were drifting apart. Austin and Crespin use Austin and Villanova's set of dimensions, methods and analysis, and categorizing frameworks as a means of integrating and discussing research subsequent to 1992 from the perspective of progress and continuing problems. As such, this chapter provides a launching point for the remaining chapters in the book.

In chapter 3, Lance, Baxter, and Mahan examine rating source effects and their impact on performance measurement. They present and comparatively evaluate two competing theoretical interpretations of source effects in performance ratings, a traditional normative accuracy model that is rooted in classical test theory of true and error scores, and an alternative ecological perspective derived from social psychological literature on the accuracy of social judgments. They provide evidence that empirical support is stronger for the ecological perspective and conclude that this provides theoretical justification for multisource feedback systems that are currently quite popular.

Kane and Woehr (chap. 4) review the literature pertaining to performance ratings based on frequency estimation. They present results from a meta-analytic examination of the research on the accuracy of frequency estimation in the cognitive psychology literature and conclude that the evidence suggests people are able to generate highly accurate estimates of the frequency of occurrence of specific events. Kane and Woehr also provide a detailed examination of the extent to which frequency estimation may be subject to potential biases stemming from source, object, content, and context of assessments. With this in mind, a significant contribution of this chapter is the development of a taxonomy of the contingencies and a review of the evidence that has accumulated in each of the resulting categories. For each contingency, the authors point out ways to potentially enhance the accuracy of frequency estimates, possible susceptibilities to various biases and errors, and finally, where further research is needed.

The second part of the book consists of four chapters focusing on general models of job performance. In chapter 5, Knapp sets the stage by discussing the Joint Service Job Performance Measurement (JPM) project, an initiative that provided an important foundation for much of the current thinking with regard to general performance models. The JPM project involved development and administration of a wide array of performance measures to personnel in roughly 30 military enlisted jobs. Measurement methods included hands-on work sample tests, written job knowledge tests, and supervisor and peer ratings. The research offered important opportunities to examine the performance domain using a very broad lens. This chapter summarizes the Services' contributions to the JPM project, including the Army's Project A, which was part of this initiative, and in the process talks more generally about how performance can be defined and measured. Finally, Knapp comments on the enduring legacy of this large performance measurement project and discusses future research needs.

In chapter 6, Hanson and Borman review the theoretical and empirical literature on what some might consider the "softer" aspects of job performance, which have been referred to variously as "organizational citizenship behavior," "interpersonal performance," and "contextual performance." They also describe and contrast major constructs that have been proposed in this area and summarize research exploring the dimensionality, antecedents, and consequences of citizenship performance. The chapter concludes with the development of an organizing framework that includes the central role of motives and the importance of situational variables, both as causal factors and as moderators of relationships between other antecedents and citizenship performance.

Tubré, Arthur, and Bennett (chap. 7) describe recent efforts to develop and validate a measure of job performance based on a general model of job performance. The authors explore the potential of contextual dimensions (discussed in chap. 5) as opposed to job-specific task performance dimensions for assessing work performance. Because contextual performance behaviors are postulated to be similar across a wide range of settings, the value of completely tailoring performance appraisal systems to specific jobs or organizations may diminish. The chapter discusses a number of issues related to the determination of an "appropriate" level of specificity and its implications on general assessments of performance and on future appraisal systems and methods.

Williams and Mahan (chap. 8) identify and discuss a number of critical variables related to multiteam functioning. Multiteam systems represent very common organizational structures that exist due to (a) the constraints from organizational ecology, (b) the demands for uniquely tasked work groups, and (c) protocols that improve productivity vis-à-vis integrated work systems. However, there remains little empirical research that focuses on the constructs that define similarities to, or more important, differences from that of other complex work systems. The authors highlight both ecological and functional characteristics of multiteams that may help form a theoretical basis from which one can better understand multiteam functioning.

The third part of the book consists of four chapters, each dealing with a different applied challenge facing criterion developers and measurement systems for today and the future. Caligiuri (chap. 9) discusses the importance of performance measurement within the context of today's multinational personnel and appraisal systems. The chapter begins with a discussion of the fit between a firm's business strategy and the organization of a firm's performance measurement systems. The established conceptualization of transnational business strategies (based on the global integration vs. local responsiveness framework) is used as the framework to propose three approaches to organizing performance measurement: centralized, synergistic, or localized. The major challenges in developing criteria and measuring performance for employees with diverse cultural backgrounds are discussed with an emphasis on the importance of addressing the cross-cultural context with respect to the comparability of performance dimensions, the effect of culture on method of performance evaluation, and the effect of culture on the raters and ratings of performance. The last section of this chapter focuses on challenges in performance measurement and criteria development for individuals who work as global assignees.

Salas, Burke, and Fowlkes (chap. 10) examine performance measurement tools and techniques that can be used to assess teams in a variety of applied settings—as they term—"in the wild." The chapter begins with a brief review of the fundamental characteristics of teams, followed by an identification of the types of teams that are evolving within the present workforce. Next, the authors summarize current knowledge about team performance measurement systems, as well as identify some basic requirements of those systems. The authors conclude with a set of guidelines for practitioners for pursuing these difficult challenges.

Alliger (chap. 11) explores the implications for distributed management and work systems in terms of how best to deploy performance measurement and employee management in a distributed environment. The chapter focuses on the concept of distributed knowledge technology, and in particular the wide use of the computer networks, which potentially provide altogether new opportunities for the use of job performance data. The critical practical and research issue addressed in this chapter is how distributed work systems can increase the availability of job performance information and the increasing ability to transport and display that information, and design jobs to take advantage of that ability. The chapter concludes with a discussion of the potential benefits and dangers posed to employees by use of such technology.

Tannenbaum (chap. 12) takes a more applied view of the issue of measurement, and this chapter is the capstone for the part. Unlike other chapters in this book, it is not replete with research citations. It is based, in great part, on the author's experiences attempting to apply measurement in organizations. The primary purpose of the chapter is to clarify the dynamics and challenges of using measurement in applied settings, so that both scientists and practitioners can help improve its value and efficacy.

In the final chapter of the book. Fair and Jacobs provide an overview and commentary on the work presented in the preceding chapters with an eye toward identifying unifying perspectives and potential directions for the future. This final chapter is one of the most unique aspects of this book. Farr and Jacobs report an epiphany (based in part, we hope, on the first 12 chapters) that performance appraisal really belongs on the "O" side of I/O psychology at least as much as it belongs on the "I" side. The latter view is based on a more traditional view of performance appraisal describe using what they refer to as a test development metaphor. But Farr and Jacobs argue that performance appraisal in organizations is more completely described using what they describe as political and due process metaphors that reflect more contemporary thinking on performance appraisal as it really happens in organizations and performance appraisal issues that face performance appraisal researchers and practitioners now and others that they will face in the future. Farr and Jacobs focus on trust in organizations as one of the key ingredients for the development, implementation, and maintenance of a successful performance appraisal program and tie the previous 12 chapters together by linking their themes organizational trust and the three performance appraisal metaphors developed earlier in the chapter.

References

Alliger, G. A. (1992). The theory and structure of industrial/organizational psychology. In K. Kelley (Ed.), Issues, theory, and research in industrial/organizational psychology (pp. 3-27). Amsterdam: North-Holland.

Austin, J. T., & Villanova, P. (1992). The criterion problem: 1917-1992. Journal of Applied Psychology, 77, 836-874.

Viteles, M. S. (1932). Industrial psychology. New York: Norton.

I
Problems, Prospects, and Continuing Issues

2
From “Criterion Problem” to Problems of Criteria in Industrial and Organizational Psychology: Progress, Pitfalls, and Prospects

James T. Austin
The Ohio State University
Timothy R. Crespin
Primetrics, Inc., Columbus, Ohio

We describe in this chapter progress, pitfalls, and prospects pertaining to criteria of job performance in industrial and organizational psychology. Progress refers to the advances made over the last decade. Pitfalls are the unresolved research issues as well as forces that increase distance between research and the practice of performance measurement (cf. Dulebohn, Ferris, & Stodd, 1995: Dunnette, 1990). Prospects are trends and trajectories in the investigation and use of job performance measurement. Progress and pitfalls comprise most of the chapter, whereas the concluding section addresses prospects. As a stage for discussing problems of criteria (P. C. Smith, 1976), let us place the idea of criteria within a historical context following Austin and Villanova (1992).

Historical Context

The period spanning the 1900s saw invention-inspired ingenuity, consumer marketing, and mass production emerge as engines of economic development. This storm of change coincided with a "search for order" (Hale, 1980; Wiebe, 1967). From the perspective of organizations then assuming the paternalistic role of personnel management (Dulebohn et al., 1995; Nestor, 1986), the conversion from an agrarian to an agrarian-manufacturing economy required methods of sorting and ordering those seeking work, and then supervising those hired.

Scientific psychology was becoming institutionalized within the United States at this time (Evans, Sexton, & Cadwallader, 1992). Practitioners of industrial psychology emerged after 1900 (Fagan & VandenBos, 1993; Ferguson, 1961; Wren, 1994) and were followed by those of organizational psychology during the Great Depression (Flanagan, 1981; Leavitt, 1962). Pioneers in the industrial psychology domain presented practical contributions to society and industry based on a functionalist psychology (Austin & Davies, 2000; Katzell & Austin, 1992) and a utilitarian philosophy (Schönpflug, 1992). Psychologists provided ordering with a measurement-based approach to personnel, grounded in cause-and-effect theories of work performance. In the simplest sense, the criterion manifests the effect of the intervention or predictor. The evolution of the field can be partially understood thro...

Table of contents

• Cover
• Title
• Copyright
• Contents
• Series Foreword
• Preface
• About the Contributors
• 1 Introduction
• I: PROBLEMS, PROSPECTS, AND CONTINUING ISSUES
• II: MODELS OF PERFORMANCE
• III: CHALLENGES IN APPLIED MEASUREMENT
• IV: CONCLUSIONS AND RECOMMENDATIONS FOR RESEARCH AND PRACTICE
• Author Index
• Subject Index