A Companion to Survey Research

Michael Ornstein

• Publisher: SAGE Publications Ltd
• Year: 2013
• Print ISBN: 9781446209097, 9781446209080
• eBook ISBN: 9781446291849
• Edition: 1
• Topic: Social Sciences
• Subtopic: Social Science Research & Methodology
• Index: Social Sciences

A Companion to Survey Research provides a critical overview and guide to survey methods. Rather than a set of formulas, survey design is understood as a craft where the translation of research questions into a questionnaire, sample design and data collection strategy is based on understanding how respondents answer questions and their willingness to complete a survey.

Following an account of the invention of survey research in the 1930s, a synthesis of research on question design is followed by a practical guide to designing a questionnaire. Chapters on sampling, which deal with the statistical basis of survey sampling and practical design issues, are followed by extensive discussions of survey pretesting and data collection. The book concludes with a discussion of the extent and implications of falling response rates.

This book is written for researchers, analysts and policy makers who want to understand the survey data they use, for researchers and students who want to conduct a survey, and for anyone who wants to understand contemporary survey research.

1

The Invention of Survey Research

Survey research has roots in centuries of census taking, intelligence and psychological testing beginning in the late nineteenth century, research on attitudes from the 1920s, and ‘social surveys’ of the conditions of the urban poor, pioneered in England by Charles Booth in the 1880s and Joseph Rowntree in the 1890s, and extended to many other countries in the first third of the twentieth century (Bulmer et al., 1991). Modern surveys did not evolve directly from these ancestors, however. Instead, aspiring to represent entire populations, to ask questions on almost any topic, to gather data in a timely manner, and to yield quantitative results, surveys developed out of the public opinion polls initiated in the mid-1930s by American market researchers, notably Archibald Crossley, Elmo Roper and George Gallup (Converse, 1987: 87ff.). Surveys’ successful arrival was signalled by their correct prediction of the 1936 US presidential election.¹ In Roper’s words, ‘advertising men … are to be credited with … the early development of the technique which has been evolved for sampling public opinion’ (1940: 325).

Academics were no strangers to the public opinion industry. In 1937, psychology professor Hadley Cantril of Princeton University and Frank Stanton, then research director and later president of Columbia Broadcasting, obtained a Rockefeller Foundation grant to study the psychological and cultural effects of radio. To direct the project they hired sociologist Paul Lazarsfeld, whose first American publication, ‘The art of asking why’, appeared in the 1935 inaugural issue of the National Marketing Review. The first description of longitudinal surveys, by Lazarsfeld and Fiske (1938), is in the second volume of Public Opinion Quarterly, established in 1937. During the Second World War, the two US government organizations responsible for surveys were led by Elmo Wilson, an associate of Roper, and academic psychologist Rensis Likert. The monumental surveys of the US Army – over half a million soldiers were surveyed with more than 200 questionnaires – were led by Harvard sociologist Samuel Stouffer (compiled as The American Soldier² by Stouffer et al., 1949). According to Lazarsfeld, the development of survey research

might be dated from the appearance of ‘The American Soldier’ after World War II. In this work, a large body of data was made coherent and meaningful by careful statistical analysis. ‘Survey analysis’ … became the language of empirical social research, possessing its own rules for forming basic concepts and combining them into meaningful propositions. (Cited in Rosenberg, 1968: vii)³

The next three sections of this chapter describe the development of survey sampling, questionnaire design and data collection until the early 1950s. This work established the conceptual core of modern survey research, but 60 years later no longer serves as a practical guide.

About Survey Sampling

The fundamental idea of applied survey sampling, which is that a properly selected random sample can accurately represent any population, no matter how large and diverse, dates to the late nineteenth century when Anders Kaier employed his ‘representative method’ to survey the entire Norwegian population. First, Kaier divided the country in two, separating urban and rural areas. In the urban ‘stratum’ he selected all of the five largest cities and eight smaller cities, to represent medium and small communities; then in each of the 13 selected cities he divided streets into groups according to size and selected a sample of streets; and finally he selected a fraction of the dwellings on each selected street. Counts from the Norwegian Census were used to calculate the appropriate number of selections in each community. In rural areas, a sample of municipalities was chosen on the basis of their main industry. Because Kaier calculated the probabilities at each stage of sampling to give every dwelling in Norway the same chance of selection, characteristics of the population could be estimated directly from the sample without weights (Bethlehem, 2009: 10ff.; Kuusella 2011: 91ff.). In modern parlance, the sample was ‘self-weighting’. In other samples selected by Kaier, parts of the population had different probabilities of selection and weights were used to produce unbiased estimates of population characteristics.^4,5

Kaier’s sample of the Norwegian population is like a modern multi-stage probability sample, except that the municipalities were not selected at random, but rather ‘purposively’, on the basis of his knowledge of their characteristics. The key idea is that a representative sample of a complex population can be obtained using two or more stages of selection – selecting communities, then streets within the communities, then dwellings on the streets – as long as the probability of selection at each stage is known. ‘Systematic selection’ is still used routinely, in place of strict random sampling and when the sample size is very small (for example, selecting a small number of communities) there is a good argument for selecting a ‘purposive’ sample based on a deep knowledge of the units, rather than a random sample. Kaier’s methods are close to the modern ideas of ‘balanced’ samples, discussed in Chapter 5.⁶

Skip forward more than 30 years and modern survey sampling begins with the publication of Neyman’s 1934 article, ‘On two different aspects of the representative method: the method of stratified sampling and the method of purposive selection’, which demonstrates conclusively the risk of bias when a sample is not selected at random.⁷ Also, he demonstrated the value of sample stratification, whereby the population is divided into two or more sectors or ‘strata’ and a separate sample is selected in each stratum, not necessarily with the same probability.

Neyman established the criterion for the precision of an estimate of a population characteristic: for a random sample, the confidence interval is the range of possible values of a population characteristic, with a specified probability. So, the conventional 95 per cent (or any other) confidence interval refers to the range of values, computed from the sample itself, that includes the true population value with 95 per cent probability. Important for researchers who need to design a sample before collecting any data, the expected confidence interval of an estimate can be computed from the sample design parameters – the structure and size of the sample – and (except for small samples with non-normal distributions) does not depend on the distribution of the variable of interest.

Neyman also showed how to compute confidence intervals for cluster samples, where the first stage involves the selection of a random sample of groups, and then individuals (‘elements’) are selected within the groups. The most important application of this idea was to ‘area probability’ samples where geographical areas, such as municipalities, rural districts or city blocks, are selected first, and then a random sample of households is selected in the areas. The huge advantage of area probability samples is that a list of the entire population is not required, only a list of all the geographical areas into which the population is found, taken from a census or government records. Neyman’s 1937 lectures at the Graduate School of the US Department of Agriculture (published as Neyman, 1938) led to the design of the Sample Survey of Unemployment, the first modern labour force survey, soon renamed the Current Population Survey⁸ (Frankel and Stock, 1942; Fienberg and Tanur, 1983: 136).

According to Smith, ‘The only major features of current survey design that he [Neyman] failed to introduce were multi-stage sampling and variable probability (p.p.s.) sampling, but these followed logically from his work’ (1976: 185). The development of area probability samples required two further steps. First, working at the US Bureau of the Census, Hansen and Hurwitz (1943) showed that the most precise estimates of population characteristics were obtained with ‘paired selection’ – at each stage of a sample where clusters are selected (for example, communities, census tracts or city blocks), two clusters should be selected at random, with probability proportional to their size. Second, in the mid-1960s Leslie Kish and his colleagues developed a method called ‘balanced repeated replication’ to estimate the precision of estimates of complex statistics, such as differences between means and regression coefficients, from multi-stage samples (Kish and Frankel, 1970).⁹

The first standard texts on survey sampling appeared in the 1950s (Smith, 1976: 186), but the practical methods for estimating errors in complex samples only came into view in the 1970s (Kish and Frankel, 1974) and they were not incorporated in standard survey analysis software until the mid-2000s.

For studies of political attitudes and market research, the adoption of probability samples was much slower. In 1944, Stock wrote:

A stratified random sample may be entirely selected in the central office, in which case the interviewer’s quota will consist of a specific list of names and addresses; or the stratification alone may be determined by the central office, in which case the interviewer’s quota will consist of a set number of interviews with each of the various types of people. With this method the individuals representing each type are selected ‘at random’ by the interviewer. The first method, widely used by government agencies, is more accurate but also more expensive. The second method is relatively inexpensive and accurate enough for most public opinion research. It is used by the vast majority of opinion research agencies today. (p. 142)

The ‘various types of people’ from which interviewers were to select specified numbers of survey respondents ‘at random’ were identified by their ‘colour’, age, sex and economic status. Bias could arise from mistakes in classifying people on sight, but also there was ‘reluctance of the typical middle-class interviewer to approach people in the lowest economic brackets’ (Rugg, 1944a: 149).

Berinsky describes the rationale for quota samples in the public opinion research of the 1930s and 1940s as follows:

Gallup and Roper did not trust that chance alone would ensure that their sample would accurately represent the sentiment of the nation. Through the selection of particular interviewing locales and the construction of detailed quotas for their employees conducting interviews in those locales, these researchers presumed that they could construct a representative sample. (2006: 502)

While this suggests a distrust of the fundamental principles of probability, the strategy has some merit in light of the cost constraints of the public opinion industry, the small sizes of the sample in each community, and the vagaries of survey fieldwork of the time, particularly the interviewers’ difficult-to-control avoidance of poorer dwellings and poorer-looking and less cooperative respondents. Without the guarantee of unbiased estimates that comes with probability samples, to a degree the success of non-probability methods relies on luck, which eventually fails.

Pre-election polls in US presidential elections first legitimized and then undermined quota sampling. After successfully predicting the winners of the four elections between 1920 and 1932, the Literary Digest magazine’s 1936 poll mistakenly projected the election of Landon over Roosevelt, based on the 25 per cent return of more than 10 million ‘ballots’ sent out to its readers and to names taken from car registrations and telephone books. The failure is attributed to the over-representation of the middle and upper class among the magazine’s subscribers, car owners and households with telephones, compounded by similar bias in the response rates of people who did receive a ballot. Also, changes in political support over the course of the campaign may not have been captured, because many ballots were returned early in the campaign (Squire, 1988; Cahalan, 1989), a problem that plagues election polling to this day. Surveys using quota samples by George Gallup’s American Institute of Public Opinion and by Archibald Crossley correctly predicted Roosevelt’s victory (Crossley, 1937), apparently vindicating their sampling method.

Then in 1948 polls by Crossley, Gallup and Roper all incorrectly predicted the victory of Dewey over Truman in the US presidential election. An investigative committee appointed by the Science Research Council and headed by statistician Frederick Mosteller did not fault quota sampling in principle, although it concluded that: ‘It is impossible to separate the error introduced by the quotas set from that arising in the process of selection by interviewers’ (Committee on Analysis of Pre-Election Polls and Forecasts of the Social Science Research Council, 1948: 608). The fault was seen to lie in the design of the particular samples – setting quotas that did not match the voting population – or the misapplication of quotas by interviewers. Nevertheless, this Report effectively ended the use of quota sampling for academic studies and led to its slow demise in market research.

About Survey Questionnaires

Long before modern surveys, censuses and the social surveys included extensive questions about individual demographic characteristics and the economic condition of households, and early twentieth-century ‘intelligence’ tests employed questionnaires of a kind. Modern surveys covered a much broader range of topics, beginning with attitude studies by psychologists, market research and election polling, then extending to a wide range of research on personal experience and perceptions of life.

The first book on question design and the culmination of this period was Payne’s The Art of Asking Questions; his

little book was not written by an expert in semantics, not even by a specialist in question wording. The author is just a general practitioner in research … the reader will be disappointed if he expects to find here a set of definite rules or explicit directions. The art of asking questions is not likely ever to be reduced to easy formulas. (1951: xi)

Modesty did not leave Payne short on concrete suggestions, including ‘a concise checklist of 100 considerations’ for question design, detailed consideration of question formats and an annotated checklist of 1000 common words.

It was recognized that questions on subjective topics were more ambiguous and prone to bias. Abstract concepts and greater detail could make a question more difficult to answer, increasing measurement error and non-response, and the use of response categories with vague boundaries (such as ‘agree’ versus ‘strongly agree’) was unavoidable. Also, the validity of answers to subjective questions could not be established by comparison to records or other concrete measures (Cantril and Fried, 1944: 23; Connelly, 1945).

Cantril and Fried’s list of the pitfalls of question design is perfectly contemporary. Questions could be ‘too vague to...

Table of contents

• Cover Page
• Title Page
• Copyright
• Contents
• About the Author
• Introduction
• 1 The Invention of Survey Research
• 2 Writing Survey Questions
• 3 Designing a Questionnaire
• 4 Fundamentals of Probability Sampling for Surveys
• 5 Applied Sample Design
• 6 Survey Pretesting
• 7 Survey Data Collection
• 8 The Future of Survey Research
• References
• Index