The Art of Empirical Investigation

Julian Simon

• Publisher: Routledge
• Year: 2017
• eBook ISBN: 9781351305624
• Topic: Social Sciences
• Subtopic: Social Science Research & Methodology
• Index: Social Sciences

Julian Simon was known for his methodical, and often controversial, writings challenging conventional beliefs about overpopulation, pollution, disappearing farmland, and the scarcity of energy sources and raw materials. But throughout his works is a common theme: that responsible, unbiased research and examination of the data is indispensable to formulating a well-informed and accurate opinion. The Art of Empirical Investigation teaches student, professor, researcher, and those interested in ascertaining the truth about social issues just how to proceed.

The Art of Empirical Investigation is a textbook on the basics of social-scientific research. It discusses all the important empirical methods used in social science, and its examples, drawn from a wide variety of academic and applied fields, illustrate the use of each method in its most appropriate context. The actual decisions a researcher must make at every stage of a project are emphasized, as well as obstacles to knowledge--such as observer bias, deception, unreliability of data, and sampling costs--and how to overcome them. Presupposing nothing, the book introduces the reader to the foundations of empirical social-science research, regardless of a specific field. It also makes an important contribution to beginning researchers' understanding of an operational definition of causality, which cuts through philosophical obscurity and teaches the researcher how to decide whether or not a given relationship is causal.

1 introduction

1. Purpose of the Book
2. What Kinds of Research Are Called “Empirical”?
3. The Place of Statistics in the Study of Research Methods
4. Some General Remarks

1. Purpose of the Book

This book is primarily for students who have never before studied or done empirical social-scientific research. I hope that the book contains good advice that will help you get your first research project off the ground successfully and increase your efficiency in later work. As for those of you who will not do empirical research, the book may teach you to distinguish good research from poor research and help you to understand why empirical researchers do things as they do them.

People who have had some training or experience in empirical research may also gain from the elementary level of the discussion. Basic concepts often are bypassed as one rushes to learn the methods of particular fields. Coming back to fundamentals can widen the perspective of an advanced student and fill holes in his knowledge. If an advanced student is to gain something from this book, however, he must have the wisdom to realize that the apparent simplicity of the basic concepts is often deceiving. For example, everyone knows that ceteris paribus—holding “all other things equal”—is important. But the more research you do, the more you realize how complex is the ceteris paribus idea, how difficult it is to choose the right ceteris paribus conditions, and how often research is useless because other things really were not made sufficiently equal.

The book is intended for future producers of research, of course. But many people who study research methods will never produce research; rather, they will be research consumers, in their jobs and as citizens. For this important latter group of people, the aim of the book is to teach how to evaluate research done by others—to know which research is good and which is not; where the weak spots are in a piece of research and how important they are; and whether a research finding, whether presented in the professional literature, in an informal report, or in the popular press, is likely to be valid or not. For example, Volvo recently advertised that “90 percent of the Volvos sold in the last eleven years in the U.S. are still on the road.” A good course in research methods is likely to alter your impression of what that claim might mean.

I hope that you yourself carry out some empirical research—no matter how small in scope—as you read this book. It is not enough to study empirical research the way one studies astronomy, economics, psychology, and other academic subjects. Reading about research principles is certainly useful. But research is not entirely an academic subject. Rather, it is largely an art, a how-to-do-it subject like musical composition, writing advertisements, or swimming.¹ You never really know how to do research until you do it, any more than you can know how to swim after only pool-side instruction. You must jump into the water, thrash around, and gradually improve with practice. For the same reason, skill in empirical research requires experience —that is, doing research of your own and criticizing the research of others.² For example, when you do a piece of research you cannot fail to learn just how complicated even the simplest research really is.

Eventually you would learn much that is in this book by trial and error. Instruction can only hasten the process and make it less painful by showing what has and has not worked for others. But that is as far as the teaching can go. (Reading what others say about research can be enormously profitable, however, if you will benefit by the experience of others.)

Now a theme that will recur throughout the book: There is never a single, standard, correct method of carrying out a piece of research. Do not wait to start your research until you find out the proper approach, because there are always many ways to tackle a problem—some good, some bad, but probably several good ways. There is no single perfect design. A research method for a given problem is not like the solution to a problem in algebra. It is more like a recipe for beef stroganoff; there is no one best recipe.

For technical matters, too, there may be several satisfactory techniques, and there is no cut-and-dried answer. For example, if you want to do a questionnaire survey, should you interview by mail, by telephone, or in person? Chapter 13 discusses the pros and cons of each method, but eventually sound judgment is required for this technical decision; no handy rule book can make such decisions for you. Or, should you pay your subjects in an experiment or survey? Again there is no pat answer; instead, this book tries to give you some principles that will help you to make sound technical decisions about your research methods.

In this book the word “method” refers to empirical techniques and devices of various sorts. To the philosophers the term refers to “scientific method”— the whole process of getting knowledge, including the theoretical and the empirical steps. But even then it is surely true that there is no single “scientific method.” “. . . There is no one scientific method . . . there will be as many different scientific methods as there are fundamentally different kinds of problems” (Northrop, pp. ix, 19). “The scientific method, as far as it is a method, is nothing more than doing one’s damnedest with one’s mind, no holds barred” (Bridgman, p. 450).

2. What Kinds of Research Are Called “Empirical”?

This book deals with empirical³ research and not with scientific speculation. Much of scientific work consists of thinking up ideas about the nature of the world, generalizing from observed facts to scientific “laws,” and developing logical systems that are called “theories” or “models.” But simply as a division of labor the speculative part of science is not covered here. The subject of this book is “getting the facts.” Of course empirical work goes beyond “mere” observation and description, and it is inextricably intertwined with explaining nature and making predictions about it. But the process of thinking up explanations or hypotheses about nature and its laws is beyond our scope here.⁴

Nor is this book concerned with the logical process of building scientific theories. That is, it does not deal with the process of finding the logical relationships between various scientific statements or with the process of developing generalizations or scientific laws. Rather, its subject is the less glamorous craft of producing and examining factual and material evidence and sense data to develop the descriptions, measurements, comparisons, and tests of hypothesized relationships that are themselves part of the speculative side of scientific work.

Lest there be misunderstanding, I emphasize that a good idea is the keystone of an empirical study. Mere data collection and measurement are worthless unless the subject is important. Theory is often the fount of important ideas for empirical research, and sound theory is of inestimable value in any field. The relationship between theory and empirical research is explored in Chapter 5.

Finally, “empirical research” excludes knowledge obtained by consulting authorities, in books or in person.⁵ It includes only knowledge obtained from data resulting from first-hand observations, either by you or by someone else. Reexamination of data collected by others, such as U.S. Census data, is empirical research, of course.

Most of the examples in this book are drawn from “pure” research. But I also draw examples concerning policy decisions from the “applied” social sciences, and they often have a dollars-and-cents orientation. Applied research methods are sometimes more sophisticated than are methods used in pure research (Stouffer, 1950a, pp. 198-9), because it is possible to work up some calculation, even though crude, to compare the benefits expected from a research method against the cost of doing the research with that method. Such calculation leads to efficiency in research. (Pure research can be defined as research whose social or economic payoff is far in the future, whereas applied research is expected to have a quick payoff. But pure research is often done without any thought at all about payoff, just to satisfy the desire to understand.)

Here are two examples of how dollars-and-cents calculation in an applied problem helps to make sensible decisions about research: First, an advertiser can calculate whether comparing two advertisements in a split-run test⁶ is worth the cost of the test. One can also compare the costs and benefits of a split run against the costs and benefits of other types of advertising research. Second, a candy firm can sensibly calculate its sample size when comparing two new flavors of candy. It can reckon the relevant costs and dollar benefits. It is much harder to determine the sample size sensibly when I. Pavlov, for example, studies how the flow of saliva in dogs can be conditioned to the sound of a bell. The possible benefits of such pure research—some intangible gain in the quality of human life perhaps far in the future or merely the satisfaction of an urge to understand our world better —are less predictable and much more difficult to evaluate in money terms to balance against the money cost of the research. Yet the decisions must be made anyway.⁷

Our examples come from the social sciences, not only because the book is intended for students of the social sciences, but also because empirical scientific methods have been used with greater variety and greater subtlety in the social sciences than in the natural sciences (Chapter 33 defends this claim).

We shall turn frequently to a few such famous studies as Alfred Kinsey’s research on sexual behavior, employment and unemployment surveys, Herman Ebbinghaus’ learning experiments, Presidential election polls, television-audience ratings, Sigmund Freud’s case history of Anna O., the U.S. Surgeon General’s Report on Smoking and Health, and Ivan Pavlov’s work on conditioning reflexes. These studies have been chosen for several reasons: They are inherently interesting; most students have a general knowledge of them from survey courses or general reading; most of the studies have been repeated or scrutinized closely by outside experts (for example, the American Statistical Association appointed three top statisticians to report on Kinsey’s methods) ; and they show us a wide variety of methods and a broad representation from the social sciences.

I also refer frequently to my own work, despite its limitations, because I know exactly what went into the work—the difficulties, decisions, errors, corrections of errors, and the order in which things took place. One cannot have such intimate knowledge of anyone else’s work. Yet it is these details and decisions, seldom written about, that are hardest for the novice to understand and master and that one usually learns only by serving as an apprentice (that is, “graduate assistant”) or by trial and error in one’s own work.

The book emphasizes the design and plan of research, rather than the analysis of research data. Except for studies that reanalyze data collected by others, the most important and interesting decisions arise at the design stage, or at least they should arise then. If you postpone these decisions until after the data have been collected, you may suffer heartbreak and wasted expense.

Here are three brief examples of the importance of good design: First, two library scientists sought to determine the proportions of various-sized books in research libraries. So they measured the heights of each of the hundreds of thousands of books in a major library. With a little planning and sound design they would have needed only to measure a fraction of that number of books. And with sound design the results could also have been applied to libraries other than the one they studied.

Second, a family-planning group tested one birth-control propaganda campaign in Village A against another propaganda campaign in Village B, forgetting that subsequent differences in birth rates and contraception-acceptance rates might reflect basic differences between the two villages unconnected to the campaigns, rather than only the differences between the campaigns. Sound planning would complete the design by alternating both campaigns in the two villages or by other methods. The same sort of error has often been made by experimental psychologists and sociologists.

Third, our understanding of voting behavior has been greatly enhanced by the use of the panel method, in which the same people are quizzed about their voting behavior several times during the same election campaign. These repeated observations make it possible to understand the mechanism of voting and vote shifting in ways that are impossible without the panel design.

Mistakes at the design stage can be mended only at great extra cost, or not at all. By comparison, mistakes at the analysis stage can be remedied at slight or no cost as long as the mistakes have not gotten into print or been acted upon.

Think through the research design carefully in advance. Failing to consider all the necessary details at the design stage because of procrastination or mental laziness is one reason that many researchers get very little done. Of course not everything can be foreseen, especially in exploratory studies, but one should use as much foresight as possible. It is useful to talk to your friends about the design, and to prepare an outline of it; both processes reveal fuzziness in your design thinking.

The design of a piece of research must depend upon the particular purpose that the research is intended to serve; this is a message I shall repeat again and again. For example, the Internal Revenue Service publishes statistics on the amount of advertising done by groups of firms that sell various products, and these statistics include all firms’ advertising, because the government requires data on the entire economy. But the statistics gathered and published by industry trade associations cover only the leading firms, because the industry-collected statistics are designed to meet ...

Table of contents

• Cover Page
• Title Page
• Copyright Page
• Contents
• Introduction to the Transaction Edition
• Preface to Second Edition
• Acknowledgments ta Second Edition
• Acknowledgments to First Edition
• Prologue
• 1 Introduction
• PART ONE the Process of Social-Science Research
• PART TWO Research Decisions and Procedures
• PART THREE the Obstacles to Social-Science Knowledge and Wags to Overcome them
• PART FOUR Extracting the Meaning of Data
• PART FIVE Epilogue
• Bibliography
• Index