How to Measure Survey Reliability and Validity

Mark S. Litwin

• Publisher: SAGE Publications, Inc
• Year: 1995
• Print ISBN: 9780803957046
• eBook ISBN: 9781506319810
• Edition: 1
• Topic: Social Sciences
• Subtopic: Social Science Research & Methodology
• Index: Social Sciences

UCLA School of Medicine, Los Angeles

Aimed at helping readers improve the accuracy of their survey, Litwin?s book guides in assessing and interpreting the quality of their survey data by thoroughly examining the survey instrument used. The book also explains how to code and pilot test new and established surveys. In addition, it covers such issues as how to measure reliability (e.g., test-retest, alternate-form, internal consistency, inter-observer, and intra-observer reliability), how to measure validity (including content, criterion, and construct validity), how to address cross-cultural issues in survey research, and how to scale and score a survey.

"I found this work to be of very high quality with respect to both content and writing. It is commendable in terms of communicating and facilitating understanding of sometimes difficult concepts. It will make an excellent text for my introductory course on survey research and, I imagine, for many similar or related courses in the social sciences or education. All the pedagogical features, including the exercises, are excellent, and the level of writing throughout not only is appropriate for an introductory volume, but also engaging and lively."

--Daniel U. Levine, Department of Education, University of Nebraska

1	Overview of Psychometrics

A successful data collection survey is not simply a set of well-designed questions that are written down and administered to a sample population. There are good surveys and bad ones. Bad surveys produce bad data, that is, data that are unreliable, irreproducible, or invalid or that waste resources. Conversely, good surveys yield critical information and provide important windows into the heart of the topic of interest.

Psychometrics is the branch of survey research that enables you to determine how good the survey is. It provides a way to quantify the precision of measurement of qualitative concepts, such as satisfaction. Example 1.1 presents two surveys of hotel guest satisfaction. It is not obvious whether one or the other is better, but they are clearly different. During the course of this book, you will learn how to evaluate their differences.

---

EXAMPLE 1.1
Two Surveys of Guest Satisfaction

Purpose of the Survey: To assess guest satisfaction at a downtown hotel with a three-item survey at the time of checkout

Survey 1: Circle one number for each item

1. Did you enjoy your stay?

Yes	1
No	2

2. How was the service at the hotel?

Good	1
Bad	2

3. Would you stay here again?

Yes	1
No.	2

Survey 2: Circle one number for each item

1. Overall, considering the service, food, and all other aspects of our hotel, how would you describe your stay here?

Very enjoyable	1
Somewhat enjoyable	2
Neither enjoyable nor unenjoyable	3
Somewhat unenjoyable	4
Very unenjoyable	5

2. How would you describe our service?

More efficient than other hotels I have stayed in	1
Equally efficient to other hotels I have stayed in	2
Less efficient than other hotels I have stayed in	3

3. How likely are you to stay here again?

Highly likely	1
Likely	2
Not sure	3
Unlikely	4
Highly unlikely	5

---

Which survey is better? Survey 2 appears as though it might produce more useful information because it provides more than just the yes/no type questions used in Survey 1, but you really don’t know quantitatively which survey is better. What exactly is meant by better! The better survey will more accurately measure guest satisfaction, producing more useful data from which to draw conclusions about the hotel’s performance. Strictly speaking, it is difficult to assess the quality of the data you collect. It is easier to assess the accuracy of the survey instrument used to collect that data. This assessment consists primarily of looking at the reliability and the validity of the survey instrument. Example 1.2 demonstrates that different tools used to measure electrical resistance may produce completely different results. The only way to determine which, if either, is correct is by looking directly at the accuracy of the measurement tools.

---

EXAMPLE 1.2
Resistance Meters

Two licensed electricians use different resistance meters to measure the ohms in four brand-new circuits during a new product analysis. Pat uses the old meter she has been using for the last 15 years. Jerry uses a new model that he just bought from HomeLine, Inc., a reputable mail order company. After the measurements are taken, Pat’s data are 6, 16, 38, and 119 ohms. Jerry’s data are 19, 28, 73, and 184 ohms. Because there is no way to determine which, if either, data set is correct, the accuracy of the resistance meters themselves must...

Table of contents

• Cover Page
• Title Page
• THE SURVEY KIT
• Contents
• How to Measure Survey Reliability and \klidity: Learning Objectives
• 1. Overview of Psychometrics
• 2. Reliability
• 3. Validity
• 4. Scaling and Scoring
• 5. Creating and Using a Codebook
• 6. Pilot Testing
• 7. Multicultural Issues
• Exercises
• Answers
• Suggested Readings
• Glossary
• About the Author