Survey Methodology

Robert M. Groves, Floyd J. Fowler, Mick P. Couper, James M. Lepkowski, Eleanor Singer, Roger Tourangeau

• Publisher: Wiley
• Year: 2011
• ISBN:
  9781118211342
• Edition: 2
• Topic: Mathematics
• Subtopic: Probability & Statistics
• Index: Mathematics

Praise for the First Edition:

"The book makes a valuable contribution by synthesizing current research and identifying areas for future investigation for each aspect of the survey process."
— Journal of the American Statistical Association

"Overall, the high quality of the text material is matched by the quality of writing..."
— Public Opinion Quarterly

"... it should find an audience everywhere surveys are being conducted."
— Technometrics

This new edition of Survey Methodology continues to provide a state-of-the-science presentation of essential survey methodology topics and techniques. The volume's six world-renowned authors have updated this Second Edition to present newly emerging approaches to survey research and provide more comprehensive coverage of the major considerations in designing and conducting a sample survey.

Key topics in survey methodology are clearly explained in the book's chapters, with coverage including sampling frame evaluation, sample design, development of questionnaires, evaluation of questions, alternative modes of data collection, interviewing, nonresponse, post-collection processing of survey data, and practices for maintaining scientific integrity. Acknowledging the growing advances in research and technology, the Second Edition features:

• Updated explanations of sampling frame issues for mobile telephone and web surveys

• New scientific insight on the relationship between nonresponse rates and nonresponse errors
  
  

• Restructured discussion of ethical issues in survey research, emphasizing the growing research results on privacy, informed consent, and confidentiality issues
  
  

• The latest research findings on effective questionnaire development techniques
  
  

• The addition of 50% more exercises at the end of each chapter, illustrating basic principles of survey design
  
  

• An expanded FAQ chapter that addresses the concerns that accompany newly established methods
  
  

Providing valuable and informative perspectives on the most modern methods in the field, Survey Methodology, Second Edition is an ideal book for survey research courses at the upper-undergraduate and graduate levels. It is also an indispensable reference for practicing survey methodologists and any professional who employs survey research methods.

CHAPTER ONE

AN INTRODUCTION TO SURVEY METHODOLOGY

A Note to the Reader

You are about to be exposed to a system of principles called “survey methodology” for collecting information about the social and economic world. We have written this book in an attempt to describe the excitement of designing, conducting, analyzing, and evaluating sample surveys. To appreciate this fully, use the devices we have placed in each chapter to enrich your memory of the material. Throughout the book, you will see boxes with illustrations and examples of key principles, terminology notes, and highlights of classic research studies in the field. In the outside margin of each page you will find key terms, at the point where they are defined. At the end of each chapter is a set of exercises that you can use to test your understanding of that chapter’s material. The best strategy is to read the text through, then, at the end of each chapter, go back, read the boxes, and review the key terms.

At 8:30 AM on the day before the first Friday of each month, a group of economists and statisticians enter a soundproof and windowless room in a building at 2 Massachusetts Avenue, NE, in Washington, DC, USA. Once those authorized are present, the room is sealed.

Those in the room are professional staff of the U.S. Bureau of Labor Statistics (BLS), and their task is to review and approve a statistical analysis of key economic data. Indeed, they have spent the week poring over sets of numbers, comparing them, examining indicators of their qualities, looking for anomalies, and writing drafts of a press release describing the numbers. They write the press release in simple language, understandable by those who have no technical knowledge about how the numbers were produced.

At 8:00 AM the next day, a group of journalists assemble in a monitored room in the nearby main Department of Labor building, removed from any contact with the outside world. The BLS staff enter the room and then reveal the results to the journalists. The journalists immediately prepare news stories based on the briefing. At exactly 8:30 AM, they simultaneously electronically transmit their stories to their news organizations and sometimes telephone editors and producers.

The statistics revealed are the unemployment rate of the prior month and the number of jobs created in the prior month. The elaborate protections and security used prior to their release stem from the enormous impact the numbers can have on society. Indeed, in months when the numbers signal important changes in the health of the U.S. economy, thousands of stock market investors around the world make immediate buy and sell decisions. Within 45 minutes of the announcement, trillions of dollars can move in and out of markets around the world based on the two numbers revealed at 8:30 AM.

Both the unemployment rate and the jobs count result from statistical surveys. A household survey produces the unemployment rate; an employer survey, the jobs count. The households and employers surveyed have been carefully selected so that their answers, when summarized, reflect the answers that would be obtained if the entire population were questioned. In the surveys, thousands of individual people answer carefully phrased questions about their own or their company’s attributes. In the household survey, professional interviewers ask the questions and enter the answers onto laptop computers. In the employer survey, the respondents complete a standardized questionnaire either on paper or electronically. Complex data processing steps follow the collection of the data, to assure internal integrity of the numbers.

These two numbers have such an impact because they address an important component of the health of the nation’s economy, and they are credible. Macroeconomic theory and decades of empirical results demonstrate their importance. However, only when decision makers believe the numbers do they gain value. This is a book about the process of generating such numbers through statistical surveys and how survey design can affect the quality of survey statistics. In a real sense, it addresses the question of when numbers from surveys are credible and when they are not.

1.1 INTRODUCTION

This chapter is an introduction to survey methodology as a field of knowledge, as a profession, and as a science. The initial sections of the chapter define the field so that the reader can place it among others. At the end of the chapter, readers will have a sense of what survey methodology is and what survey methodologists do.

A “survey” is a systematic method for gathering information from (a sample of) entities for the purposes of constructing quantitative descriptors of the attributes of the larger population of which the entities are members. The word “systematic” is deliberate and meaningfully distinguishes surveys from other ways of gathering information. The phrase “(a sample of)” appears in the definition because sometimes surveys attempt to measure everyone in a population and sometimes just a sample.

The quantitative descriptors are called “statistics.” Statistics are quantitative summaries of observations on a set of elements. Some are “descriptive statistics,” describing the size and distributions of various attributes in a population (e.g., the mean years of education of persons, the total number of persons in the hospital, the percentage of persons supporting the president). Others are “analytic statistics,” measuring how two or more variables are related (e.g., a regression coefficient describing how much increases in income are associated with increases in years of education; a correlation between education and number of books read in the last year). That goal sets surveys apart from other efforts to describe people or events. The statistics attempt to describe basic characteristics or experiences of large and small populations in our world.

Almost every country in the world uses surveys to estimate their rate of unemployment, basic prevalence of immunization against disease, opinions about the central government, intentions to vote in an upcoming election, and people’s satisfaction with services and products that they buy. Surveys are a key tool in tracking global economic trends, the rate of inflation in prices, and investments in new economic enterprises. Surveys are one of the most commonly used methods in the social sciences to understand the way societies work and to test theories of behavior. In a very real way, surveys are a crucial building block in a modern information-based society.

Although a variety of activities are called surveys, this book focuses on surveys that have the following characteristics:

1) Information is gathered primarily by asking people questions.

2) Information is collected either by having interviewers ask questions and record answers or by having people read or hear questions and record their own answers.

3) Information is collected from only a subset of the population to be described—a sample—rather than from all members.

Since “ology” is Greek for “the study of,” survey methodology is the study of survey methods. It is the study of sources of error in surveys and how to make the numbers produced by surveys as accurate as possible. Here the word “error” refers to deviations or departures from the desired outcome. In the case of surveys, “error” is used to describe deviations from the true values applicable to the population studied. Sometimes, the phrase “statistical error” is used to differentiate this meaning from a reference to simple mistakes.

The way each of the above steps is carried out—which questions are asked, how answers are collected, and which people answer the questions—can affect the quality (or error properties) of survey results. This book will describe how to conduct surveys in the real world and how to evaluate the quality of survey results. It will describe what is known, and not known, about how to minimize error in survey statistics. Most of all, this book will attempt to distill the results of 100 years of scientific studies that have defined the theories and principles, as well as practices, of high-quality survey research.

1.2 A BRIEF HISTORY OF SURVEY RESEARCH

Converse (1987) has produced an important account of the history of survey research in the United States, and we recount some of the highlights here. There are four perspectives on surveys that are worth describing: the purposes to which surveys were put, the development of question design, the development of sampling methods, and the development of data collection methods.

1.2.1 The Purposes of Surveys

Perhaps the earliest type of survey is the census, generally conducted by governments. Censuses are systematic efforts to count an entire population, often for purposes of taxation or political representation. In the United States, the Constitution stipulates that a census must be conducted every ten years, to reapportion the House of Representatives reflecting current population residence patterns. This gives the statistics from a census great political import. Because of this, they are often politically contentious (Anderson, 1990).

A prominent early reason for surveys was to gain understanding of a social problem. Some people trace the origins of modern survey research to Charles Booth, who produced a landmark study titled Life and Labour of the People of London (1889–1903) (http://booth.lse.ac.uk/). As Converse recounts it, Booth spent his own money to collect voluminous data on the poor in London and the reasons why they were poor. He wrote at least 17 volumes based on the data he collected. He did not use methods like the ones we use today—no well-defined sampling techniques, no standardized questions. Indeed, interviewer observation and inference produced much of the information. However, the Booth study used quantitative summaries from systematic measurements to understand a fundamental societal problem.

In contrast to studies of social problems, journalism and market research grew to use surveys to gain a systematic view of “the man on the street.” A particular interest was reactions to political leaders and preferences in upcoming elections. That interest led to the development of modern public opinion polling.

In a related way, market research sought knowledge about reactions of “real” people to existing and planned products or services. As early as the 1930s, there was serious research on what programs and messages delivered via the radio would be most popular. The researchers began to use surveys of broader samples to produce information more useful to commercial decision makers.

Over the early 20th century, public opinion polling and market research, sometimes done by the same companies, evolved to use mail surveys and telephone surveys. They often sampled from available lists, such as telephone, driver’s license, registered voter, or magazine subscriber listings. They collected their data primarily by asking a fixed set of questions; observations by interviewers and proxy reporting of other people’s situations were not part of what they needed. These features were directly tied to the most important difference between what they were doing and what those who had gone before had done; rather than collecting data about facts and objective characteristics of people, the polling and market research surveyors were interested in what people knew, felt, and thought.

Schuman (1997) on “Poll” Versus “Survey”

What is the difference between a poll and a survey? The word “poll” is most often used for private-sector opinion studies, which use many of the same design features as studies that would be called “surveys.” “Poll” is rarely used to describe studies conducted in government or scientific domains. There are, however, no clear distinctions between the meanings of the two terms. Schuman notes that the two terms have different roots: “’Poll’ is a four letter word, generally thought to be from an ancient Germanic term referring to ‘head,’ as in counting heads. The two-syllable word ‘survey,’ on the other hand, comes from the French survee, which in turn derives from Latin super (over) and videre (to look). The first is therefore an expression with appeal to a wider public, the intended consumers of results from Gallup, Harris, and other polls. The second fits the needs of academicians in university institutes who wish to emphasize the scientific or scholarly character of their work.” (page 7)

The measurement of attitudes and opinions is a key foundation of the modern management philosophies that place much weight on customer satisfaction. Customer satisfaction surveys measure expectations of purchasers about the quality of a product or service and how well their expectations were met in specific transactions. Such surveys are ubiquitous tools of management to improve the performance of their organizations.

Politicians and political strategists now believe that opinion polls are critical to good decisions on campaign strategy and messages to the public about important issues. Indeed, a common criticism of modern politicians is that they rely too heavily on polling data to shape their personal opinions, choosing to reflect the public’s views rather than provide leadership to the public about an issue.

1.2.2 The Development of Standardized Questioning

The interest in measuring subjective states (i.e., characteristics that cannot be observed, internalized within a person) also had the effect of focusing attention on question wording and data collection methods. When collecting factual information, researchers had not thought it important to carefully word questions. Often, interviewers were sent out with lists of objectives, such as age, occupation, and education, and the interviewers would decide on how the questions would be worded. Experienced researchers often did the interviewing, with great confidence that they knew how to phrase questions to obtain good answers.

However, the market research and polling organizations were doing large numbers of interviews, using newly hired people with no special background in the social sciences. Of necessity, researchers needed to specify more carefully the information sought by the survey. Further, researchers found that small changes in wording of an attitude question sometimes had unusually large effects on the answers.

Thus, early in the development of opinion surveys, attention began to be paid to giving interviewers carefully worded questions that they were to ask exactly the same way for each interview. Also, as interviewers were used more to ask questions, it was found that how they asked questions and recorded answers could affect the results. This led eventually to researchers training and supervising interviewers more formally than earlier.

Question wording also was influenced as the academics started to pay some attention to what the commercial researchers were doing. Psychometricians, psychologists who quantify psychological states, had been interested in how to put meaningful numbers on subjective states. Measuring intelligence was the first effort in this direction. However, people such as Thurstone also worked on how to assign numbers to attitudes, feelings, and ratings (e.g., Thurstone and Chave, 1929).

For the most part, their approaches were extremely cumbersome and were used primarily when they could get captive college student volunteers to fill out lengthy, highly redundant questionnaires. Such instruments were not going to be useful for most survey interviews with representative samples; they took too long to measure one or a few attitudes. Rensis Likert in his PhD dissertation (Likert, 1932), however, demonstrated that a single, streamlined question, with a scaled set of answers, could accomplish much the same thing as a lengthy series of paired comparisons. Likert applied the work to surveys (and later founded the University of Michigan Survey Research Center in 1946).

1.2.3 The Development of Sampling Methods

Early researchers, such as Booth, essentially tried to collect data on every element of a defined population. Such censuses avoided problems of errors arising from measuring just a subset of the population, but were clearly impractical for large populations. Indeed, the difficulty of analyzing complete census data led to early efforts to summarize a census by taking a sample of returns. Early efforts to sample would study a “typical” town, or they would purposively try to collect individuals to make the samples look like the population—for example, by interviewing about half men and half women, and trying to have them distributed geographically in a way that is similar to the population.

Although the theory of probability was established in the 18th century, its application to practical sample survey work was largely delayed until the 20th century. The first applications were the taking of a “1 in N” systematic selection from census returns. These were “probability samples”; that is, every record had a known nonzero chance of selection into the sample.

A big breakthrough in sampling came from people who did research on agriculture. In order to predict crop yields, statisticians had worked out a strategy they called “area probability sampling.” This is just what it sounds like: they would sample areas or plots of land and find out what farmers were doing with those plots in the spring (for example, if they were planting something on them and, if so, what) in order to project what the fall crops would look like. The same technique was developed to sample households. By drawing samples of geographic blocks in cities or tracts of land in rural areas, listing the housing units on the blocks or rural tracts, then sampling the housing units that were listed, samplers found a way to give all households and, by extension, the people living in them, a chance to be sampled. The attraction of this technique included the elimination of the need for a list of all persons or all households in the population prior to drawing the sample.

The Depression and World War II were major stimuli for survey research. One of the earliest modern probability samples was drawn for the Monthly Survey of Unemployment, starting in December, 1939, led by a 29-year-old statistician, Morris Hansen, who later became a major figure in the field (Hansen, Hurwitz, and Madow, 1953). During the war, the federal government became interested in conducting surveys to measure people’s attitudes and opinions, such as interest in buying war bonds, as well as factual information. Considerable resources were devoted to surveys during the war, and researchers who were recruited to work with the government during the war later came to play critical roles in the development of survey methods. When the war was over, methodologists understood that in order to produce good population-based statistics it was necessary to attend to three aspects of survey methodology: how questions were designed; how the data were collected, including the training of interviewers; and how s...