Systematic Data Collection

Susan C. Weller, A. Kimball Romney

• Publisher: SAGE Publications, Inc
• Year: 1988
• Print ISBN: 9780803930742
• eBook ISBN: 9781506349756
• Edition: 1
• Topic: Social Sciences
• Subtopic: Social Science Research & Methodology
• Index: Social Sciences

The process of collecting accurate data through interviewing, questionnaires, and other methods has not always been clear. However, data collection in field settings can be done in a structured, systematic and scientific way. These authors show us how. First, they focus on the importance of finding the right questions to ask. By providing a variety of formats - triadic comparisons and rating scales for data collection, both oral and written methods - and stressing cultural relativity, Weller and Romney suggest ways to improve not only the data collected, but also the interpretation and analysis of such data. Primarily addressed to qualitative social scientists, this volume is also appropriate for anyone who wants to study attitudes and beliefs. In particular, it is an ideal text for courses in anthropology, linguistics, qualitative research methods, health care, and survey research.

SYSTEMATIC DATA
COLLECTION

SUSAN C. WELLER
University of Pennsylvania

A. KIMBALL ROMNEY
University of California, Irvine

1. INTRODUCTION TO STRUCTURED INTERVIEWING

This book is designed to help students and researchers in the social sciences collect better interview or questionnaire data. We hope to do this by providing directions for using a variety of formats for the collection of such data. Our discussion is limited to systematic interviewing where each informant is asked the same set of questions. This excludes many kinds of very valuable interviewing procedures such as the genealogical method of Rivers (1910) and clinical interviewing in psychology and psychiatry. We also exclude the collection of response data elicited by open-ended questions that need to be coded, categorized, or reworked by the researcher.

There is a large literature on various forms of open-ended interviewing where subjects are asked to give long explanatory answers to a variety of questions. These forms of interviewing are very valuable in many kinds of research and in the beginning stages of more formal investigations. In this book, however, we restrict ourselves to questions and tasks that are asked in such a way that informants’ responses and informants’ categories are elicited and used directly. These procedures help avoid researcher bias resulting from imposing a priori categories that may not correspond to those of the people being studied.

The social and behavioral sciences, like all sciences, depend upon experimental and observational data as the “raw material” for increased understanding. Such understanding, the ultimate goal of all sciences, requires the careful analysis of observations to assess their relationship to currently accepted knowledge. Major advances in our understanding usually require systematic observation, classification, interpretation, and study over a period of time. It is our hope that the wider use of structured interviewing formats provided in this book would improve the quality of much of the “raw material” in the social sciences.

The gathering of data is only the first step in the quest for understanding; data must then be analyzed and interpreted with the aid of theory. The task of theory is to develop models for the processes that underlie phenomena. Theories facilitate the calculation of properties of the models. Observations or experimental results can then be compared to model properties. When such comparisons are favorable and the predictions accurate, understanding is advanced. The increased use of structured formats should aid in this overall task of understanding human behavior.

At the beginning of any social science study involving the collection of interview data the researcher is faced with the problem of what questions to ask and what is the best format for the phrasing of questions. Different data collection techniques are appropriate during different stages of research. In the beginning stages, informal exploratory interviewing is necessary in order to define the area of inquiry and obtain a general notion of what to expect. Later, formal methods are appropriate in order to measure accurately the phenomena under consideration.

There recently has been a good deal of discussion in the social sciences about the relative merits of data collected through interviewing and talking to informants as compared to observing subjects in natural or experimental settings. Some have gone so far as to suggest that people are not capable of providing good data through verbal reports. We believe that this issue is frequently distorted and misunderstood. Using the various methods for collecting data presented in this book, reliable and valid results can be produced routinely. The accuracy of the results are not generally appreciated. With reasonable care reliability figures above .95 are easily possible. Various studies of validity demonstrate that equally high results are reasonable to expect. Specific evidence will be presented in the final chapters of the book.

An important issue that needs to be discussed is how the choice of a data collection method depends upon the type of data desired. Data type is not necessarily a function of the data collection format; it is frequently determined by the type of question that is asked and how responses are tabulated. Therefore, we would like to present the data collection methods as somewhat independent of the kind of data produced. This book is organized by method and the different “types” of data that can be obtained from each method are discussed in each chapter. Data types that are referred to include: similarity data, ordered data, and test performance data. Similarity data consist of informants’ judged similarity estimates among study items. Ordered data consist of an ordinal ranking (or rating) of items on a single conceptual scale. Performance data refer to responses that can be graded as “correct” or “incorrect.”

Plan of the Book

In the following chapters we describe the use of various formats for the collection of data in structured interviewing and paper-and-pencil questionnaires. In each chapter we discuss the method in general and then give detailed step-by-step directions for its use. We also give examples of the use of each of the methods as well as summarize their strengths and weaknesses.

We begin, in Chapter 2, by describing the procedures for finding how respondents define the boundaries of a given subject matter or domain of interest. Chapters 3 and 4 discuss the use of the pile sort as a useful format for obtaining judged similarity among a large number of objects. The pile sort is a format in which the subject is asked to sort items into piles so that objects within piles are more similar to each other than to objects in the other piles. Chapter 5 presents the method of triads. In the triad task the informant is presented with three items at a time and typically picks the one most different from the other two. Triads may be used to collect either similarity or ordered data. Rating scales, one of the most varied and widely used methods in the social sciences, are discussed in Chapter 6. Rating scales may be used to collect ordered data or to collect similarity data on pairs of items. Rank order tasks are described in Chapter 7. Chapter 8 provides a complete description and explanation of balanced incomplete block designs that may be used to shorten and simplify triad, paired comparison, and ranking tasks. We devote Chapter 9 to a discussion of the use of sentence frames as a systematic interviewing technique that is especially useful where the items in a domain are characterized by a variety of features. In Chapter 10 we discuss a variety of other data collection formats. These include true-false questions, multiple-choice questions, fill-in-the-blank questions, matching designs, direct estimation, and “pick n” items from an array. In the last two chapters we discuss reliability and validity. We illustrate various ways in which confidence in the results from interviewing can be increased by using such techniques as replication and the convergence of results from different methods.

Although this book is addressed primarily to social scientists, it is appropriate for anyone who wants to study attitudes and beliefs. While a political scientist may want to use a technique to study how particular political candidates are perceived, a market researcher may use the same techniques to study preferences for different brands of cigarettes. Similarly, one sociologist may use structured interviewing techniques to study perceptions of occupational prestige, while another may study social networks. Anthropologists working with nonliterates may be particularly interested in the oral data collection methods. While we cannot review all the possible applications, we would like to note that the interviewing and data collection tasks contained in this volume are as appropriate for use in such exotic settings as the highlands of New Guinea as they are in the corporate offices on Wall Street. It is our expectation that the book will provide the researcher with a larger choice of data collection techniques than has been heretofore available in a single place.

2. DEFINING A DOMAIN AND FREE LISTING

The first step in any study is to obtain a clear understanding of the definition and boundaries of what is being studied. Since this book is about interviewing there is an implicit assumption that the researcher is interested in what the respondents think about “something.” For convenience we call the “something” a semantic or cultural domain. The semantic or cultural domain is simply the subject matter of interest, a set of related items. Examples of domains that have been studied include color terms, kinship terms, diseases, plant terms, animal terms, airplane piloting errors, kinds of pain, and characteristics of infant feeding methods. The concept of a domain is a very general one and may include almost any coherently defined subject matter.

A domain may be defined as an organized set of words, concepts, or sentences, all on the same level of contrast, that jointly refer to a single conceptual sphere. The items in a domain derive their meanings, in part, from their position in a mutually interdependent system reflecting the way in which a given language or culture classifies the relevant conceptual sphere. For example, the concept of “shape,” may have category members such as “round,” “square,” “rectangular,” and so on. Each of these is a kind of shape and each says something different about shape. We refer to “shape” as the generic name of the category and the words “round” and the like as items or objects in the domain.

The overall success of any study depends in part on giving careful attention to the definition of the domain as the first step of the research. Generally the domain should be defined by the informants, in their language, and not by the investigator. It is easy to assume falsely that the investigator knows the domain and can therefore define what items belong in the domain and what items do not belong in the domain. If researchers want to study beliefs about discipline in different ethnic groups, it is only after disciplinary actions and punishments are explicitly listed that it is appropriate to worry about the format and design of the interview instrument. The immediate problem is to specify which disciplinary actions and punishments should be included in the interview.

There are many ways to compile a list of items to define the domain of study items. On rare occasions there may be an absolute definition. The alphabet, the Morse code, the names of the months, the days of the week, presidents of the United States, and the states of the United States are all examples of a priori defined domains. Usually, however, the investigator does not know the boundaries of the domain and needs some sort of elicitation procedure to ensure that the investigator’s definition of the domain corresponds to that of the informants.

The most useful general techniqu...

Table of contents

• Cover page
• Title
• Copyright
• Contents
• Editors’ Introduction
• 1. Introduction to Structured Interviewing
• 2. Defining a Domain and Free Listing
• 3. Pile Sort I: Single Sorts
• 4. Pile Sort II: Successive Sorts and the Construction of Taxonomies and Trees
• 5. Triadic Comparisons
• 6. Rating Scales
• 7. Rank Order Methods: Complete and Partial Techniques
• 8. Balanced-Incomplete Block Designs
• 9. Sentence Frame Formats
• 10. Other Common Structured Formats: Dichotomous, Multiple Choice, Fill-in-the-Blank, Matching, Direct Estimation, and Pick N
• 11. Reliability, Consensus, and Sample Size
• 12. Validity and Replication with Variations
• Appendices
• References
• About the Authors