Designing and Doing Survey Research

Lesley Andres

• Editorial: SAGE Publications Ltd
• Año: 2012
• ISBN:
  9781446273098
• Edición: 1
• Categoría: Sciences sociales
• Categoría: Recherche scientifique et méthodologie

Designing and Doing Survey Research is an introduction to the processes and methods of planning and conducting survey research in the real world.

Taking a mixed method approach throughout, the book provides step-by-step guidance on:

• Designing your research

• Ethical issues

• Developing your survey questions

• Sampling

• Budgeting, scheduling and managing your time

• Administering your survey

• Preparing for data analysis

With a focus on the impact of new technologies, this book provides a cutting-edge look at how survey research is conducted today as well as the challenges survey researchers face. Packed full of international examples from various social science disciplines, the book is ideal for students and researchers new to survey research.

ONE

Survey research design – then and now

On almost a daily basis, individuals are confronted with survey research – either as potential participants or as recipients of results. Survey research projects are carried out or commissioned by companies, newspapers, school districts, and other organizations and are administered by individuals, governments, university researchers, polling organizations, and survey research firms. The results of survey research are reported regularly in vehicles such as scholarly research papers or government reports, or in the media ranging from the New York Times to Cosmopolitan magazine. Today, we read about the results of survey research with the purpose of informing our decisions about a range of activities such as buying a car or adopting a new fitness regime. Conversely, consumers of survey research may have been invited to participate in a survey research project because they had recently purchased a car or engaged in a particular fitness regime. According to Igo (2007: 5), today ‘the public is simultaneously object, participant, and audience’ of survey research.

In general, there are two major types of survey research. Large-scale surveys such as national censuses, opinion polls, or research projects are carried out by institutions equipped with vast resources in terms of money, staff, and access to databases that are not normally available to an individual researcher. Smaller-scale surveys are carried out by institutions such as schools, post-secondary institutions, hospitals, and other organizations and individuals with the goal of gathering facts about or learning more about the demographic characteristics, behaviours, and attitudes of their students, employees, patients, clients, or members.

These large-scale ‘sample’ surveys – that is, those based on a representative sample of a larger population – tend to employ standardized questionnaire formats with the goal of generating statistics in order to generalize to a larger population. Such surveys and their related designs are privileged in most survey methods books. Sometimes, other types of survey formats and sampling strategies are not addressed at all. Other times, a sort of schizophrenic approach is employed in survey methods books in that the topics of interviewing, non-probabilistic sampling, and open-ended questions are introduced or touched upon, but are not fully developed to the same extent as studies that are based on probabilistic designs (those employing random samples). For example, authors of many survey texts consider the group administration of a survey, for example within a classroom setting, as legitimate. However, the same authors either do not include or are critical of the convenience sample (again, students conveniently located within a classroom) as a legitimate sampling strategy. As such, these alternative approaches are usually relegated to an ‘inferior’ status or condemned – what Weisberg (2005: 237) calls ‘usual textbook injunctions against [non-probability sampling]’. Examples of survey research in many existing texts are based on large probability samples that are (1) not feasible for the small-scale researcher to carry out, and (2) not particularly instructive when trying to learn how to conduct survey research. The word ‘interview’ in many textbooks means highly structured interviews and does not embrace the full range of uses of interviews in survey methods.

Few researchers have the privilege, ability, or desire to collect data through some form of probability sample. Access to lists of potential respondents is often restricted; only large data collection centres have the resources and political clout to gain access to such lists and carry out large-scale projects. Although the survey instrument and sample are integral and interrelated dimensions of survey research, the former does not need to be limited exclusively to a set of standardized questions presented to a sample with the intention of generalizing to larger populations. Within both of these dimensions there are many options.

In reality, most real-world survey research is conducted on a much smaller scale, to specifically targeted audiences. This book is designed to help those – for example, senior undergraduate and graduate students, small business owners, institutional researchers – to design meaningful surveys through the skilful crafting of questions that are posed to the audience or audiences best suited to answering the questions.

Many texts on survey research design are grounded solidly in the positivistic paradigm and related notions of objectivity and parsimony. However, because the social science world that we investigate is full of subjectivities and objectivities, survey research need not be limited to a tight set of rules that limit our ability to capture life as experienced by our respondents. The phrase ‘survey research’ provides a rubric for many types of approaches, ranging from self-administered to interviewer-administered approaches. Because we can maintain rigour through strong design, data collection from different perspectives should be encouraged.

In addition, survey research can be carried out in a myriad of ways to ask questions ranging from open-ended interviews to those that are closed-ended and strictly standardized. Many types of approaches, ranging from pen-and-paper mail-out surveys to face-to-face interviews conducted via webcam can be employed. Surveys can be free-standing or can be embedded in larger research designs such as ethnographies, case studies, or experimental research.

Questionnaires, face-to-face interviews, and focus groups all belong to the rubric of survey research. In addition, either probabilistic (based on random samples), non-probabilistic techniques (purposive or not based on random samples), or combinations of both types of sampling designs can be employed. Statistics can be generated from surveys, text can be analysed qualitatively from interviews or open-ended survey questions, or these methods can be combined by, for example, quantifying interview data by reading codes and data stored in qualitative software programs into statistical software programs. The goal of survey research may be to generalize to larger populations or it may be intended to be transferable – that is, the ‘findings will be useful to others in similar situations, with similar research questions or questions of practice’ (Marshall and Rossman, 2006: 201). In other words, survey research can and should be conducted from both quantitative and qualitative perspectives.

This is not a new idea. In the past many approaches to data collection were included under the label ‘survey research’. The ‘paradigm wars’ that were prominent at the end of the twentieth century have not been resolved but rather have ‘agreed to a détente’ (Bergman, 2008b: 2). In terms of survey research, the quantitative–qualitative divide remains in that quantitative researchers continue to resist the idea that open-ended interviewing is a valid form of survey research (see Fowler, 2009) and qualitative researchers are surprised to learn that when they interview people to make statements that extend beyond the sample, they are actually conducting survey research! In many instances, by informing themselves of some of the key tenets of survey research design, qualitative researchers could strengthen their research projects which would, in turn, be more rigorous and hence more credible, and quantitative researchers could expand the results of their findings by enlivening them with the voices of respondents. In 1944, Paul Lazarsfeld, a leading figure in twentieth-century American sociology and survey research, addressed what he described as ‘two philosophies of research … one wedded to so-called in-depth interviewing, and the other content with more objective methods of research’ (Lazarsfeld, 1944) and concluded that a combination of methods (p. 60) would result in improving research design, analysis, and interpretation. In his 1962 presidential address to the American Sociological Association, Lazarsfeld said that early in his academic career he was assigned the task of analysing the occupational choices of young people. While it was relatively straightforward to conduct analyses to portray relationships among choices, social stratification, and age differences, problems arose in trying to interpret the reasons for choices. He noted that ‘reasons’ provided were contradictory and led to ambiguous responses that defied meaningful analysis. He concluded that ‘an investigator’s lack of skill in the art of asking “why” questions led to meaningless statistical results’ (1962: 758). ‘Ever since’, he asserted, ‘I have continued to search for sound ways for making empirical studies of action’ (p. 758).

Surveys that are more quantitatively oriented will include more closed-ended questions, and those that are more qualitatively oriented will include more open-ended questions. However, considerable advances in mixed methods research design invite us to consider including both types of question and to use different types of survey format, such as mail-out questionnaires and interviews. In fact, the variety of data collection formats and ways of posing questions to respondents makes survey research a naturally occurring mixed methods design. Survey research, in and of itself, is a large, heterogeneous family of methods which do not fit tidily into either a quantitative or qualitative box.

TEXT BOX 1.1

The movie Kitchen Stories, directed by Bent Hamer, provides a delightful example of research paradigm wars.

http://en.wikipedia.org/wiki/Kitchen_Stories

Trying to separate the two approaches and to downplay or dismiss the latter does indeed hamper (Bergman, 2008c) the creative ways that survey research alone, or in combination with other types of research, can produce rich findings. Often, the choice of survey method is based on the researcher’s training, too often in one or other of the methodological camps. As a result, studies are designed not according to the research questions at hand, but rather to conform to the researcher’s analytical skill set(s). As we shall see in Chapter 9, advances in analytical software programs have broken down the barriers between quantitative and qualitative analysis, rendering combinations of the various data collection methods no longer problematic. As such, ‘specific data collection and analysis [may] now be connected far more directly and explicitly to a research focus, research context, and research design’ (Bergman, 2008c: 18).

Historically, survey research embraced a wide variety of data collection methods. In the next section, I provide a brief overview of the history of survey research.

A short history of survey research

Gathering data on individuals has a long history, with various censuses dating back to antiquity. The first documented census occurred in China more than 4000 years ago (http://www.thecanadianencyclopedia.com/index.cfm?PgNm=TCE&Params=A1ARTA0020060). Several references to census-taking in relation to taxation occur in the Christian Bible. Originally, data on individuals were collected primarily for reasons of taxation and military service. The Domesday Book of 1086, commissioned by William the Conqueror, documented the land holdings and livestock of most of the English population for the purpose of levying taxes. The male census in Norway in the 1660s was conducted for a similar purpose. The oldest documented complete nominal census – that is, containing the names of the members of the population – was in Iceland in 1703, where the following data were collected: name, age, and position in the household, along with health information and whether the person was a pauper without abode (Garðarsdóttir and Guðmundsson, 2005; Tomasson, 1977).

TEXT BOX 1.2

A description of world censuses can be found in Wikipedia:

http://en.wikipedia.org/wiki/Censusnsus

Sweden stakes claim to the first comprehensive endeavour in population statistics, the Tablellverket in 1749 (Sköld, 2004). Today, Sweden remains a leader in gathering data on its population. Eventually censuses became more all-encompassing by first linking commercialism with economic growth and then extending to the physical and moral health of the population (Thorvaldson, 2007).

Nominal censuses began in Great Britain in 1801, Denmark in 1834, Sweden in 1860, and Norway in 1865. The first national census was undertaken in the USA in 1790, Canada in 1871, and Australia in 1881. Simultaneously, other non-state organizations in Britain, Germany, France, the USA and elsewhere began collecting ‘vital statistics’ data for purposes such as monitoring disease and creating insurance tables. As Igo (2007: 7) points out, ‘Western countries in the nineteenth century witnessed a wave of surveying by private citizens and philanthropists, producing a veritable “avalanche of numbers” in the service of industrial and social reform’. In addition, there is evidence that in the 1800s surveys were being conducted within academic circles. For example, Max Weber, a sociologist and political scientist most famous for his work The Protestant Ethic and the Spirit of Capitalism, took part in six data collection endeavours involving the administration of questionnaires to individuals. The first, carried out in 1890, included the study of workers’ attitudes (Lazarsfeld and Oberschall, 1965).

However, the notion of survey in these early studies meant something quite different to what it does today. This is best exemplified by the research of Charles Booth at the end of the nineteenth and beginning of the twentieth century in London. Booth has been called the ‘founding father’¹ (Converse, 1987: 1) of the empirical tradition in the social sciences, and it is worth taking a few minutes to examine his accomplishments.

Shortly after moving to London in 1875, the shipping magnate Charles Booth became passionately interested in the problems of poverty and unemployment plaguing the city that ‘politically and administratively … had scarcely advanced beyond the Middle Ages’ (Fried and Elman, 1968: xv). He engaged in debates and discussions with politicians, socialists, and social workers who were unable to adequately answer his key concern: ‘exactly how the poor lived, exactly how discontented they were, how concretely they might be helped’ (Fried and Elman, 1968: xvi). By creatively piecing together available information such as census data and drawing on a plurality of data gathering approaches, Booth set out to answer this question and focused his attention initially on the East End of London.

TEXT BOX 1.3

Visit the Charles Booth Online Archives at the London School of Economics: http://booth.lse.ac.uk/
Courtesy of Senate House Library, University of London, Goldsmiths Library, MS797/11/96/2.

He collaborated with London School Board visitors who were able to provide detailed accounts of all families with children of preschool age. In order to gather information about the poor who did not have children, Booth extended his data collection endeavours to include sources such as Poor Law statistics, registered lodging houses documented in police reports (Fried and Elman, 1968: xviii), and what we might today call ‘focus groups’ with colleagues of diverse political persuasions (Englander and O’Day, 1995). Through a detailed analysis of the data, Booth and his small team of colleagues and staff constructed an eight-category economic classification system and eventually created a definition of the ‘poverty line’. Booth and his team set out to conduct a massive endeavour interviewing members of households and lodging residences. The collected data were used to create colour-...