Scale Construction and Psychometrics for Social and Personality Psychology

Mike Furr

• Publisher: SAGE Publications Ltd
• Year: 2011
• Print ISBN: 9780857024046
• eBook ISBN: 9781446244906
• Edition: 1
• Topic: Psychology
• Subtopic: Social Psychology
• Index: Psychology

Providing conceptual and practical foundations in scale construction and psychometrics for producers and consumers of social/personality research, this guide covers basic principles, practices, and processes in scale construction, scale evaluation, scale use, and interpretation of research results in the context of psychological measurement. It explains fundamental concepts and methods related to dimensionality, reliability, and validity. In addition, it provides relatively non-technical introductions to special topics and advanced psychometric perspectives such as Confirmatory Factor Analysis, Generalizability Theory, and Item Response Theory.

The SAGE Library in Social and Personality Psychology Methods provides students and researchers with an understanding of the methods and techniques essential to conducting cutting-edge research.

Each volume within the Library explains a specific topic and has been written by an active scholar (or scholars) with expertise in that particular methodological domain. Assuming no prior knowledge of the topic, the volumes are clear and accessible for all readers. In each volume, a topic is introduced, applications are discussed, and readers are led step by step through worked examples. In addition, advice about how to interpret and prepare results for publication are presented.

Introduction

 

Social and personality psychologists often use psychological inventories, tests, or questionnaires to measure important psychological phenomena. Indeed, such instruments may be the most widely-used method of measuring variables such as attitudes, traits, self-concept, self-evaluation, beliefs, abilities, motivation, goals, social perceptions, and so on. In both experimental and non-experimental research, social and personality psychologists often rely upon previously-developed scales, develop entirely new scales, or generate revised scales based upon those developed by other researchers when measuring one or more variables. These methods have produced important advances, revealing the psychological mechanisms underlying many important psychological phenomena.

The current volume provides a conceptual and practical foundation for producers and consumers of research based upon psychological scales. More specifically, it addresses issues in scale construction, scale use, scale evaluation, and interpretation of research results emerging from psychological scales. It covers basic principles, practices, and processes, and it introduces advanced techniques that expand one’s psychometric toolkit. In covering these issues, the volume highlights their fundamental importance for the analysis and interpretation of psychological research, and it provides relatively non-technical discussions intended to facilitate basic appreciation, understanding, and interest.

Attention to psychometrics and measurement seems somewhat compartmentalized. Some, perhaps many, researchers view psychometrics and measurement as relevant only for the study of “individual differences,” for “correlational research,” and/or for self-report scales.

Such perceptions are incorrect. In fact, psychometrics and measurement are important for all psychological research—experimental and non-experimental research, research-based self-reports and research based upon behavioral observations, physiological data, reaction times, and other form of measurement used in social/personality psychology. Regardless of the internal validity of one’s research, the importance of one’s research questions, or the apparent objectivity of one’s measurement strategy, psychometric issues such as dimensionality, reliability, and validity have important implications for one’s ability to draw meaningful conclusions from psychological research.

Importance of Well-grounded Scale Construction and Psychometric Understanding

---

Effective scale construction and adequate psychometric quality have important implications for the proper interpretation of psychological research and its psychological meaning. An important goal of this volume is to articulate several such implications—hopefully providing broader insight into the importance of strong measurement.

First, the quality of our measures affects the apparent size of effects obtained in our analyses. According to basic psychometric theory, the apparent association between any two variables is affected directly by the reliability of the measures of one or more of those variables. More specifically, imperfect reliability reduces, or attenuates, the effects actually observed in one’s research, as compared with the true psychological effects. This is true for experimental research as much as for non-experimental research. Whether a particular analysis involves manipulated independent variables and measured dependent variables or it involves several measured variables, the reliability of the measured variables directly affects the resulting magnitude of the differences or associations.

Second, by affecting the sizes of statistical effects, measurement quality indirectly affects the statistical significance of those effects. Of course, the size of an observed difference or the size of an observed correlation directly affects the likelihood that the difference or correlation will reach statistical significance. Thus, if poor measurement quality produces an attenuated effect for a given group difference, main effect, interaction effect, correlation, or regression slope, then that effect is relatively unlikely to reach statistical significance.

Third, the quality of one’s measures (and manipulations) affects the psychological meaning of one’s results. That is, the psychological meaning of a scale’s scores has important implications for the psychological inferences to be drawn from research using that scale. If the scores have clear meaning in terms of a psychological construct, then any research using the scale can be interpreted confidently with regard to that construct. However, if a scale’s scores have ambiguous or undemonstrated psychological meaning, then research using the scale cannot be interpreted confidently in terms of any particular psychological construct. More generally, if a measurement process lacks empirically-demonstrated validity evidence, then researchers cannot draw well-grounded inferences about its psychological implications. However, if a measurement process is constructed with attention to psychometric quality, then researchers—both producers and consumers of the research—can confidently interpret the size and statistical significance of the result, and they can make well-grounded psychological inferences.

This volume will examine these implications in depth, reviewing procedures that are valuable for producers and consumers of psychological research. For producers, this volume hopefully enhances motivation and ability to implement effective and well-understood measurement strategies. For producers and consumers of psychological research, it hopefully enhances motivation and ability to interpret research within the proper psychometric context—understanding the implications of specific measurement strategies, understanding how to evaluate the quality of those strategies, and understanding the ways in which measurement quality affects psychological conclusions.

Overview

---

After briefly highlighting basic principles, practices, and recommendations, this volume provides guidance and background helping social/personality psychologists construct, use, evaluate, and interpret psychological measures. The first section describes steps in the construction of psychometrically-sound scales, it introduces basic psychometric properties such as dimensionality, reliability, and validity, and it examines potential threats to psychometric quality. The second section introduces special topics and advanced psychometric perspectives, focussing on the use of difference scores, and on the logic and use of Confirmatory Factor Analysis, Generalizability Theory, and Item Response Theory as advanced psychometric perspectives. These advanced perspectives differ in important ways from the traditional psychometric perspective with which most readers might be familiar.

Core Principles, Best Practices, and an Overview of Scale Constructions

This chapter presents principles and practices that are among the broadest and most fundamental issues for scale construction, modification, use, evaluation, and interpretation. The points are rather straightforward but are vitally important in conducting informative research. Thus, this chapter provides nontechnical overviews of each point, to be complemented by greater exploration and depth later in this volume.

Several of these principles and practices strike me, as an editor, reviewer, and reader of social/personality research, as being somewhat under-appreciated. To retain focus on those issues, this chapter bypasses issues that, though fundamental to scale construction and psychometrics, seem generally well-known and well-implemented. Indeed, much social/personality research is based upon measurement that is well-conceived and appropriately-executed. This discussion is intended to raise awareness and understanding of issues that, if appreciated even more widely, will enhance the generally good conduct and interpretation of research. The issues are summarized in Table 2.1.

Most facets of the process and principles covered in this chapter apply to all forms of psychological measurement. For example, this chapter addresses the need to articulate the construct and context of a measurement strategy, the need to evaluate psychometric properties, and the need to revise the measurement strategy if necessary—all of which apply to measurement strategies such as “tests” of maximal performance, reaction time, behavioral observations, physiological, measures, choices and decisions, informant-reports, and so on.

In addition, this chapter outlines scale construction in terms of four steps (Figure 2.1). Reflecting contemporary social/personality psychology (John & Benet-Martinez, 2000), this chapter (and this volume more generally) blends several approaches to scale construction. It involves rationally-focussed item-writing, attention to scale dimensionality and internal coherence, and empirical examination of the scale’s psychological meaning.

Table 2.1 Under-appreciated principles and practices in scale construction, use, evaluation, and interpretation

Examination and Interpretation of Reliability and Validity

---

First and most broadly, the psychometric quality of the current data should be evaluated and considered when interpreting results. Reliability and validity are crucial in understanding statistical results and their psychological implications. Roughly stated, reliability is the precision of scores—the degree to which scores accurately reflect some psychological variable in a given sample. Validity, then, concerns the “some variable” reflected by those scores—specifically, validity is the degree to which scores can be interpreted in terms of a specific psychological construct. Note that scores can be reliable (i.e., they can be good indicators of something), but—at the same time—they can be interpreted invalidly (i.e., they can be interpreted in terms of a construct that they do not truly reflect).

Figure 2.1 The scale construction process

Thus, there are at least two issues that should be addressed in any psychological study. The first is the psychometric properties and qualities of the measures used in the study. Reliability and validity are fundamental facets of psychometric quality, and as such, researchers should provide evidence regarding the nature and strength of the reliability and validity of any scale, test, assessment, or dependent variable used in a study—is the scale performing well in the sample being studied, do the scale’s scores truly reflect the construct that researchers wish to measure? The second issue is the implications that scales’ reliability and validity have for analysis and psychological implications. Among their effects, reliability affects one’s statistical results, and validity affects one’s ability to interpret results in terms of specific psychological phenomena.

Without proper understanding of the psychometric properties of the measures in a given study, researchers and readers cannot be sure that those measures were used and interpreted appropriately. Despite this importance, fundamental psychometric information is sometimes omitted from research reports. Unfortunately, we cannot assume confidently that the reliability of a scale’s scores in one study or with one sample of participants generalizes to all studies or all samples. Thus, each time a scale is used, its scores should be evaluated in terms of psychometric quality. This volume and others (Nunnally & Bernstein, 1994; Furr & Bacharach, 2008) provide broad backgrounds for addressing psychometric quality.

Reliability can often be estimated quite easily for multi-item scales, and researchers usually assume that validity evidence generalizes across similar samples of participants. However, both of these practices are limited, as discussed later.

Dimensionality

---

A scale’s dimensionality, or factor structure, reflects the number and nature of variables assessed by its items. Some questionnaires, tests, and inventories are unidimensional, with items cohering and reflecting a single psychological variable. Other questionnaires are multidimensional, with sets of items reflecting different psychological variables.

Usually based upon factor analysis, an accurate understanding of the number and nature of a scale’s dimensionality directly affects its scoring, psychometric quality, and psychological meaning. Dimensionality dictates the number of meaningful scores that a scale produces for each participant. If a scale includes two independent dimensions, its items should be scored to reflect those dimensions. For example, the Positive Affect Negative Affect Schedule (PANAS; Watson et al., 1988) is a multidimensional questionnaire that produces one score for Positive Affect (PA) and another for Negative Affect (NA). Researchers typically do not combine items across the two dimensions, as it would produce scores reflecting no coherent psychological variable. By dictating the number of meaningful scores derived from a questionnaire, dimensionality also directs researchers’ evaluations of reliability and validity. That is, researchers must understand the psychometric quality of each score obtained from a questionnaire. For example, the PANAS has been developed and used with psychometric attention to each of its two “subscales.” Thus, researchers who develop and use psychological scales must understand the dimensionality of those scales. This is true even for short scales that might appear to reflect a single psychological variable. Inaccurate understanding of dimensionality can produce scores that are empirically and psychologically meaningless.

An important related point is that a scale’s dimensionality is not clearly reflected by the familiar Cronbach’s coefficient alpha. Based upon a scale’s internal consistency, alpha is an estimate of a scale’s reliability; however, it is not an index of unidimensionality. That is, a large alpha value cannot be interpreted as clear evidence of unidimensionality (see Chapter 4).

Finally, there are several recommendations that contradict many applications of factor analysis in evaluating the dimensionality of psychological scales. One is that the “eigenvalue greater than one” rule is a poor way to evaluate the number of dimensions underlying a scale’s items; other procedures, such as scree plots, are preferable. A second recommendation is that oblique rotations are preferable to orthogonal rotations. These recommendations are detailed in Chapter 4.

Ad Hoc Scales

---

Occasionally, researchers create scales to measure specific constructs for a study. Of course, scale-development is important for psychological research, and there are good reasons to create new scales (e.g., one might wish to measure a construct for which a scale has not yet been developed). However, there are two caveats related to ad hoc scales.

The first caveat is that previously-validated scales are generally preferable to ad hoc scales. That is, when well-validated scales exist for a given construct, researchers should strongly consider using those scales rather than a new scale. For example, there are many well-validated self-esteem scales differing in length, psychological breadth, and dimensionality. With such diversity and known psychometric quality within easy reach, there seems little reason to assemble a new ad hoc self-esteem scale.

The second caveat is that, if ad hoc scales are created, they require psychometric evaluation, including validity evidence that goes beyond face validity. Ideally, scales that are intended to measure important psychological variables are developed through a rigorous process emphasizing psychometric quality. However, ad hoc scales sometimes seem to be applied without much evaluation, with researchers apparently relying upon on face validity and assuming that items obviously reflect the intended construct. Not only should researchers examine the dimensionality and reliability of ad hoc scales, but they should strive to obtain and report independent evidence of validity. For example, researchers might recruit independent raters (e.g., colleagues or students) to read the items along with items intended to reflect other variables, ask the raters to rate the clarity with which each item reflects each of several v...

Table of contents

• Cover Page
• Title
• Copyright
• Contents
• 1 Introduction
• 2 Core Principles, Best Practices and an Overview of Scale Construction
• 3 Response Formats and Item Writing
• 4 Evaluating Psychometric Properties: Dimensionality and Reliability
• 5 Evaluating Psychometric Properties: Validity
• 6 Threats to Psychometric Quality
• 7 Difference Scores
• 8 Confirmatory Factor Analysis
• 9 Generalizability Theory
• 10 Item Response Theory
• References
• Index