Designing and Constructing Instruments for Social Research and Evaluation

David Colton, Robert W. Covert

• Publisher: Jossey-Bass
• Year: 2015
• Print ISBN: 9780787987848
• eBook ISBN: 9781119177975
• Edition: 1
• Topic: Education
• Subtopic: Evaluation & Assessment in Education

Written in easy-to-understand language, this important text provides a systematic and commonsense approach to developing instruments for data collection and analysis. This book can be used by both those who are developing instruments for the first time and those who want to hone their skills, including students, agency personnel, program managers, and researchers.

This book provides a thorough presentation of instrument construction, from conception to development and pre-testing of items, formatting the instrument, administration, and, finally, data management and presentation of the findings. Throughout the book, the authors emphasize how to create an   instrument that will produce trustworthy and accurate data. To that end they have included guidelines for reviewing and revising the questionnaire to enhance validity and reliability. They also show how to work effectively with stakeholders such as instruments designers, decision-makers, agency personnel, clients, and raters or respondents.

PART ONE
CONCEPTS

CHAPTER ONE
INTRODUCTION

In this chapter we will

• Explain the purpose and function of a social science instrument.
• Describe nomenclature used to describe instruments.
• List and describe the components of an instrument.
• Outline the steps in the instrument construction process.

We are living in a time characterized as the information age, and we encounter data-gathering instruments in all facets of our lives. For example, we are familiar with polls that gather information about political preferences and voting behaviors. Surveys of potential voters try to predict who will be elected or what proposition will pass. Media commentators remind us of the margin of error associated with a survey or note that an election is still too close to call.

The proliferation of instruments to provide data and information for decision making is not unique to political polls. Survey questionnaires can be used to obtain factual information and to assess attitudes and beliefs across a variety of topics and groups. For example, surveys can assess consumer behaviors, client satisfaction with services, employee attitudes, and the general public’s values and beliefs. The federal government is perhaps the greatest consumer and user of survey questionnaires, as these instruments are used to collect data about such topics as criminal activity, educational needs, services to the mentally ill, and health care utilization, not to mention the data for the U.S. Census.

You may be familiar with other forms of measurement instruments as well. For example, many organizations conduct annual evaluations of employee work performance. Often these evaluation instruments use scales to rate job performance on a number of attributes, such as attendance, ability to work with others, or the ability to complete work tasks in a timely manner. Although the intended purpose of a checklist or rating instrument such as an employee evaluation is different from the purpose of a survey questionnaire, the same principles are used in constructing these instruments.

As you read this introduction, you are likely thinking of instruments that you have personally used or have been asked to complete, including instruments related to your work or study. And as a reader of this text, you are also interested in creating an instrument, perhaps in conjunction with a research project or job-related activity.

Social science instruments are tools for the collection and measurement of data, and the purpose of this book is to describe how good instruments are constructed. We use the adjective good because there are indeed standards and guidelines that can produce an efficient and effective instrument rather than a mediocre one. Our goal is to help you identify the components that make for a “good” instrument, one that provides trustworthy information. As you will see, no instrument is perfect, as there are many ways to pose a question. However, by being aware of the conditions that affect your results, you can create an instrument that effectively meets your need for information.

We believe that instrument construction is as much an art as it is a science. Research has demonstrated that some approaches to instrument development can increase the accuracy and dependability of responses. And statistical tests can be used to measure the consistency of people’s responses. Nevertheless, much of what you will do involves common sense, interpersonal skills, and to a degree even creativity. You should not shy away from developing your own instrument because you believe the process is too technical.

We compare instrument construction to the activities of a painter creating a work of art. The first stage involves conceptualizing the project, a purely mental process in which the artist begins to visualize the subject and what it should look like when completed. The designer of a questionnaire goes through a similar process as he or she defines the purpose of the study, obtains information about the subject to be studied, and contemplates the items that might be included. The next step is preparatory. The artist may develop a number of sketches to define the subject, conceptualize the composition of the painting, and experiment with the use of color. Similarly, through an iterative and interactive process, the creator of a questionnaire will draft the instrument, test the items, and revise and modify both individual items and the instrument format.

An artist may set the canvas aside from time to time to evaluate its progress. During such periods, the artist may experiment with subtle changes in composition, colors, or techniques for applying the paint. The questionnaire designer will also fine-tune the instrument, often in response to feedback from content experts or potential users. Items may be reworded, different item types may be tried, and the layout of items within the instrument may be reorganized. In the end both painter and questionnaire designer must reach a point where they are comfortable that their goals have been attained and the product is ready to be unveiled.

Instrumentation

An instrument is a mechanism for measuring phenomena, which is used to gather and record information for assessment, decision making, and ultimately understanding. An instrument such as a questionnaire is typically used to obtain factual information, support observations, or assess attitudes and opinions. For example, a survey may ask respondents to list the type of soap they purchase (factual information), recall how often they purchased the item in the past year (an observation), and consider the factors that influenced their purchase, such as smell, touch, or appearance (attitudes toward the product). The term subjective describes information that originates within an individual and is reflected by items that measure attitudes, feelings, opinions, values, and beliefs. Information that is objective attempts to be free of personal interpretation and is typified by data that are observable.¹

Some instruments consist of all objective items, like the medical history questionnaire at the end of this chapter. Respondents are asked to provide demographic information such as their weight, height, and age as well as information about their physical health, such as allergies and previous illnesses. Conversely, some instruments are designed to obtain primarily subjective responses, such as information about political preferences. Although political polls include objective demographic questions, the body of the instrument consists of items that require the respondent to express an opinion or attitu...

Table of contents

• COVER
• TABLE OF CONTENTS
• TITLE
• COPYRIGHT
• FIGURES, EXHIBITS, TABLES, AND INSTRUMENTS
• PREFACE: ASKING AND ANSWERING
• THE AUTHORS
• PART ONE: CONCEPTS
• PART TWO: APPLICATION
• PART THREE: ORGANIZATION AND ADMINISTRATION
• REFERENCES
• INDEX
• End User License Agreement