The Problem with Survey Research

George Beam

• Publisher: Taylor & Francis
• Year: 2017
• Print ISBN: 9781138516779
• eBook ISBN: 9781351476256
• Edition: 1
• Topic: Social Sciences
• Subtopic: Sociology
• Index: Social Sciences

The Problem with Survey Research makes a case against survey research as a primary source of reliable information. George Beam argues that all survey research instruments, all types of asking-including polls, face-to-face interviews, and focus groups-produce unreliable and potentially inaccurate results. Because those who rely on survey research only see answers to questions, it is impossible for them, or anyone else, to evaluate the results. They cannot know if the answers correspond to respondents' actual behaviors (objective phenomena) or to their true beliefs and opinions (subjective phenomena). Reliable information can only be acquired by observation, experimentation, multiple sources of data, formal model building and testing, document analysis, and comparison. In fifteen chapters divided into six parts-Ubiquity of Survey Research, The Problem, Asking Instruments, Asking Settings, Askers, and Proper Methods and Research Designs-The Problem with Survey Research demonstrates how asking instruments, settings in which asking and answering take place, and survey researchers themselves skew results and thereby make answers unreliable. The last two chapters and appendices examine observation, other methods of data collection and research designs that may produce accurate or correct information, and shows how reliance on survey research can be overcome, and must be.

Part One

Ubiquity of Survey Research

Ultimately, survey research would be omnivorous, taking everything it could get of an individual record—attitudes, opinions, facts, information, behaviors, beliefs, experiences, personality measures, and even physiological measures.

—Jean M. Converse, Survey Research in the United States, 1987, 2009

We are a nation of the interviewed.

—Arthur Miller, Echoes Down the Corridor, 2000

endless, compendious American surveys

—Rachel Bowlby, London Review of Books, 2009

a culture of surveying…. [a] poll-saturated culture

—Sarah E. Igo, The Averaged American, 2007

the prominence and pervasiveness of public opinion polling … throughout the world … continue[s] to grow

—Herbert Asher, Polling and the Public, 7^th ed., 2007

1

Everyone Asks

The ubiquity of asking and, thereby, the ubiquity of unreliable information, is evident, in part, by the number of organizations and individuals that ask¹. Organizations devoted exclusively to survey research I name, Asking Businesses; e.g., Gallup. Other organizations that ask are corporations, consulting firms, interest groups, governments, political parties, unions, churches, schools, volunteer associations, newspapers, TV stations, and most university research centers.

Individual askers include most social—and in word only—“science”² professors, as well as many teachers and researchers in other disciplines or fields, such as management and social work. Professors who ask about the phenomena they investigate are Asking Academics.

Individual and institutional askers number in the millions, if not billions. To say that everyone asks is only a slight overstatement and, maybe not, given that even Karl Marx used survey research; at least once when he constructed and administered his “Workers’ Questionnaire”³.

Asking Businesses

Many of the largest and best known asking businesses have been in operation for half a century; some even longer. Gallup, in addition to asking on its own and contracting to ask for others, publishes a weekly, the Tuesday Briefing, which summarizes poll results by other asking businesses. Gallup also makes available its grandly named, Gallup Brain. According to one of Gallup’s mailed advertisements (dated, 22 November 2002), the Gallup Brain “houses responses from more than 3.5 million people interviewed by the Gallup Poll since 1935”. Without modesty and, presumably, not intending irony, the ad states: “The Gallup Brain is your portal to the truth”. Actually, according to the Christian Fiction in which Jesus is the truth and the only way (portal) to it, this is neither modesty nor irony; it’s blasphemy.

More than a few institutional askers, such as Manpower Inc., maintain offices throughout the world, as does Oxford Research International (UK) which, in late 2003, administered in a land being made free by American bombs and bullets “the newest and biggest survey yet of Iraqi sentiments”⁴.

Other asking businesses (many using the Internet to ask, as does, for instance, Harris, via Harris Interactive) include: Real Measures, Inc. (Rmi); Market Probe; Luntz Research Companies; TNS; Fako and Associates; Spectrum Group; Hart-Teeter Research; Infoserv, Inc.;FGI Research; Market Shares, Corp; International Survey Research, specializing in employee and management opinion surveys for national and multination organizations⁵; and the Survey/Marketing Researche-Store, which advertises itself as a “one-stop” site “for survey and marketing research products and services”⁶. SurveySite—“a professional online market research company that specializes in web-based surveys, online focus groups and web site evaluation”—is an asking business⁷. Survey Sampling International, according to its Web page, “is the premier global supplier of survey solutions to the survey research industry in the U.S., Canada, Europe, Latin America, Australia, and Asia …. [and] offer[s] sampling products for reaching various populations by Internet, telephone, postal mail, and personal interview”⁸.

Also in the asking business are Greenberg Quinlan⁹ and TouchPoll Tulsa, as well as GSO Research, survey.com, Harvey Research, Inc., Michelson & Associates, Inc., CRA International, Survey Research Information Co. Ltd., Mediamark Research, Inc., International Communications Research, Inc., Knowledge Networks, and Avue Technologies Corporation which, according to a professional association newsletter, is “the world leader in workforce management technology solutions for [the] federal government”¹⁰.

Westat, claiming “experience in custom research and program evaluation studies across a broad range of subject areas”, asks¹¹. Also in the asking business are: Roper ASW, Monroe Mendelsohn Research, Inc., Jan Krukowski and Company, J. D. Power and Associates, Morpace International, Yankelovich Partners, Peter D. Hart Research Associates, and Zogby International.

Belden Russonello & Stewart develop and administer asking efforts for political campaigns and other clients, including focus groups for misnamed nonprofits¹²; such as Housing Illinois¹³. Alan Guttmacher Institute, Institute for Women’s Policy Research, and National Bureau of Economic Research ask, and all three claim nonprofit status.

PollingReport.com is part of the asking business. Instead of asking, this firm provides answers generated by others who asked. PollingReport “puts polling results from all over the U.S. on its Web site as the results become available” and, trying to hide inevitable biases that occur in every research endeavor, promotes itself as “an independent¹⁴, nonpartisan¹⁵ resource on trends in American public opinion”¹⁶.

Prominent UK asking businesses, in addition to the already-mentioned Oxford Research International, are Market and Opinion Research and YouGov. ZDF and Allensbach Institute are renown askers in Germany. Berlin-based Transparency International is a survey researcher. Ipsos-Sofinco (France-based) asks in Europe and Canada, and recently queried the French about their consumer spending plans¹⁷. The French are also asked by Sofres. Public Opinion Foundation conducts polls throughout Russia. The Beirut Centre for Research and Information, based in Lebanon, asks.

StoryCorps (calling itself, nonprofit) is a relatively recent addition to the ranks of asking businesses¹⁸, as is another organization collecting stories, Veterans History Project, that asks veterans for their stories and, like StoryCorps (which, by 2008, received more than 30,000 stories¹⁹), archives results in the American Folklife Center. Another story-collecting enterprise is The History Makers. This so-called nonprofit endeavor video records stories of African-Americans²⁰.

Other segments of the asking business are companies that provide software for administering Web-based polls and surveys. For example, Digivey sells survey software that, according to its web site, generates “interactive, multilingual surveys for data collection with tablet PCs, portable touch screen computers, or free standing kiosk survey stations”²¹. Ultimate Software Designs claims that users of its “Ultimate Poll ASP” can “create polls on [a] web site in under an hour”²². With SurveyMonkey you can “[c]reate smart, professional surveys with ease. No software to install…. Just open a browser and go”²³.

Generally, asking businesses make most of their money selling their expertise and products (such as constructing and administering polls and interviews, coding and statistically manipulating responses²⁴, developing survey software, and so on) to individual academic researchers, news organizations, university research centers, government agencies, interest groups, corporations, and so-called nonprofit organizations. For example, Harris was hired by the Chicago Council on Foreign Relations to conduct telephone interviews for the Council’s Worldviews 2002 survey.

Corporations, Businesses Large and Small, and Consulting Firms Ask

Corporations manufacturing automobiles, pharmaceuticals, and other products, as well as businesses large and small, ask when developing marketing plans and when hiring. Also, many ask their personnel and use the answers to develop programs for organizational development and change²⁵. Pfizer asks. Olay, a manufacturer of beauty products, is an asker. Durex, a condom making company, asked about sexual behaviors in the workplace and about satisfying one’s partner. IBM asks. Accountemps, a temporary staffing firm, asks, as do other staffing and recruiting firms, such as Spherion Corporation. AIRS, a recruitment and training firm, asks. Dotcoms, such as Expedia.com, are askers. Companies picking up and processing waste ask. Investment firms, such as Merrill Lynch, ask, as does The Hartford Financial Services Group. Vanguard, an institutional i...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Dedication
• Table of Contents
• Acknowledgements
• Abbreviations and Acronyms
• Preface
• Part One Ubiquity of Survey Research
• Part Two The Problem
• Part Three Asking Instruments
• Part Four Asking Settings
• Part Five Askers
• Part Six Proper Methods and Research Designs
• Appendixes
• References
• Index