Understanding Political Science Research Methods

The Challenge of Inference

Maryann Barakso, Daniel M. Sabet, Brian Schaffner

• Publisher: Routledge
• Year: 2013
• Print ISBN: 9780415895200
• eBook ISBN: 9781136622380
• Topic: Politics & International Relations
• Subtopic: Politics
• Index: Politics & International Relations

This text starts by explaining the fundamental goal of good political science research—the ability to answer interesting and important questions by generating valid inferences about political phenomena. Before the text even discusses the process of developing a research question, the authors introduce the reader to what it means to make an inference and the different challenges that social scientists face when confronting this task. Only with this ultimate goal in mind will students be able to ask appropriate questions, conduct fruitful literature reviews, select and execute the proper research design, and critically evaluate the work of others.

The authors' primary goal is to teach students to critically evaluate their own research designs and others' and analyze the extent to which they overcome the classic challenges to making inference: internal and external validity concerns, omitted variable bias, endogeneity, measurement, sampling, and case selection errors, and poor research questions or theory. As such, students will not only be better able to conduct political science research, but they will also be more savvy consumers of the constant flow of causal assertions that they confront in scholarship, in the media, and in conversations with others.

Three themes run through Barakso, Sabet, and Schaffner's text: minimizing classic research problems to making valid inferences, effective presentation of research results, and the nonlinear nature of the research process. Throughout their academic years and later in their professional careers, students will need to effectively convey various bits of information. Presentation skills gleaned from this text will benefit students for a lifetime, whether they continue in academia or in a professional career.

Several distinctive features make this book noteworthy:

• A common set of examples threaded throughout the text give students a common ground across chapters and expose them to a broad range of subfields in the discipline.
• Box features throughout the book illustrate the nonlinear, "non-textbook" reality of research, demonstrate the often false inferences and poor social science in the way the popular press covers politics, and encourage students to think about ethical issues at various stages of the research process.

Section II
A Menu of Approaches

Chapter 4
The Challenge of Descriptive Inference

Content

Conceptualization

Different Types of Data

Operationalization and Measurement Error

Operationalization and Sampling Error

Making Descriptive Inferences and Presenting Data

Summing Up

Key Terms

In Chapter 1, we introduced the idea of descriptive inference. We noted that inference is “the process of using the facts we know to learn about facts we do not know,” and we divided inferences into two types: descriptive and causal.¹ As the term suggests, in making descriptive inferences, our goal is to describe. This might mean determining what something is, establishing how prevalent or common a phenomenon is, or resolving if it is increasing or decreasing over time. For example, we might want to know: What are the voter turnout rates across different U.S. states? What percent of Afghanis supported the NATO-led military intervention in their country in 2001? Is corruption in African countries increasing or decreasing? Descriptive inference can be contrasted with causal inference, which goes a step further and asks why something occurs. Why is voter turnout higher in Midwestern states than in Southern states? Why did some Afghanis support the NATO-led intervention and others did not? Why is corruption increasing in some countries and decreasing in others? As we noted, however, we cannot make causal inferences until we are confident in our descriptive inferences. For example, it would not make much sense to ask why corruption is increasing in Africa, if it is actually stable or even decreasing. As we can see, making descriptive inferences is an important research goal in its own right, and it is also an essential first step to making causal inferences. In this chapter, therefore, we will focus on description.

In Chapter 1 we introduced three broad challenges to making descriptive inferences: (1) conceptualization, (2) measurement, or operationalization, and (3) case selection, or sampling. In this chapter we pick up these threads and explore these themes in greater detail. The chapter first explores the challenges to defining the concepts that we are interested in measuring and studying. Before moving directly into a discussion on measurement, the chapter explores how the difficulties that we face vary based on the type of data that we are studying. For example, a comparative study of countries produces very different challenges and opportunities than a survey of the U.S. public. Then we explore the challenges of measurement and sampling using different types of data. Having examined the major challenges, the chapter turns towards some of the basic, practical tools available to students and researchers to draw descriptive inferences. These range from basic bar charts and measures of central tendency, used with quantitative data, to narratives and quotes, used with qualitative data.

Conceptualization

The first step in drawing a valid inference is to be clear about just what it is we are making inferences about. Defining our variables of interest is known as conceptualization. Perhaps not surprisingly, most of the variables that we are concerned with in political science are difficult to define. Consider for example the difficult task of defining democracy, human rights, globalization, corruption, and even war. We use these terms commonly in political science courses, and yet they mean very different things to different people. Many other variables are not just difficult, but controversial to define. Virtually any hot topic in U.S. politics involves a conflict over definition. Consider for example defining gun rights, religious freedom, or free speech. Individuals on one side of these issues typically favor very broad definitions while their opponents prefer narrow conceptualizations.

Let’s explore a concrete example: in the last few decades scholars have had an interest in trying to measure corruption. Understandably, people want to know just how pervasive corruption is in a given country, how that country compares with other countries, and whether the problem is increasing or decreasing over time. These are all great questions, but, before being able to answer them, scholars first have to define what is meant by “corruption.” The non-governmental organization Transparency International, one of the pioneers in the study of corruption, uses the simple definition of “the abuse of entrusted authority for private gain.”² On the one hand, this definition seems perfect because it is straightforward and because it appears to capture what most of us have in our head when we hear the term “corruption.” However, if we dig a little deeper, we can identify several problems with this definition. For example, who defines abuse? Is it defined by the specific laws of a country, by the culture of a country, or are there conceptions of abuse that can be applied universally across legal systems and across different cultural groups? Likewise, who defines authority? Should corruption be limited to public officials or should abuses of “authority” in a business, a non-profit organization, or even in a family be calculated into a country’s measure of corruption? What is private gain? Does private gain require a monetary exchange? If an elected official abuses her office for the benefit of her family, is that private gain? What if she commits abuses to benefit her friends or her political party?

In short, corruption could be defined as a universal concept or as something very specific to country and culture. It could be defined very narrowly, as specific behaviors that involve public officials and entail a monetary benefit, or it could be defined very broadly, as any abuses of authority for a wide variety of benefits. Our point is that the term “corruption,” a term that students of politics use extensively in daily life, is much more complicated than it first appears.

Here is one of the key points in research where we can clearly see the nonlinearity of the process. Ask yourself which definition will be easier to operationalize, or to measure: (1) a broad concept that varies based on the cultural context and involves a wide array of behaviors and actors or (2) a narrow set of specific behaviors involving specific actors that can be observed in any cultural context? From a measurement perspective, the latter is clearly preferable. With this in mind, when trying to arrive at a definition of their concept of study, many scholars have to already be thinking ahead to what they will actually be able to measure in the real world.

This is a major challenge for many students engaging in empirical research for the first time. In much university course work, students are asked to embrace complexity and nuance rather than to simplify and reduce. In fact, if your class had a group activity and tried to define corruption, we are fairly confident that you would arrive at a very broad definition rather than a narrow one. Making our definition of a concept dependent on our measurements can at times be frustrating. Consider for example the idea of “democracy.” If democracy is defined literally as “rule by the people,” then few countries would actually qualify as democracies. As a result, most political scientists have defined “democracy” as a form of representative democracy involving free and fair elections. Even this narrowing of the concept begs the question: What is meant by free and fair? Narrow definitions of free and fair elections would likely have to tolerate some abuses of civil liberties, press restrictions, abuses of power, nepotism, and clientelism, all of which do not necessarily match with the idea of “democracy” that we have in our heads. Some scholars attempt to recognize this tension by using the term operational definition, meaning a definition that can be measured, or operationalized.³

Another approach is to move up and down what Giovanni Sartori referred to as a “ladder of generality.”⁴ Rather than study “democracy,” we could move up the ladder of generality and study “regimes,” or we could move down the ladder of generality and study a subtype of democracy, such as “parliamentary democracy.”⁵ In like fashion, we could study a type of corruption, such as “petty corruption,” or relatively small bribes paid to public officials to perform or fail to perform their duties. Once we have arrived at an operational definition, we are now ready to think more specifically about measurement. The first step in this process is considering the enormous amount of variation in the type of measurements that we could develop.

Different Types of Data

It stands to reason that conceptualization and measurement challenges will vary considerably based on the type of information that we are interested in. A natural scientist interested in arsenic contamination in water is going to use a very different set of tools and face a very different set of challenges than a political scientist interested in corruption, the effect of negative campaign advertising, or governing common pool resources. Data can be divided into a number of different categories based on the answers to the following questions:

• What is the unit of analysis?
• What is the level of analysis?
• Do the data cover the entire population or are they based on a sample drawn from a larger population?
• Are the data cross-sectional or longitudinal?
• Are the data qualitative or quantitative?

These terms might not have much meaning to you yet, but we will explore each in turn. The unit of analysis is simply what is being studied or compared; in political science research, the units of analysis are typically political actors, political acts, or geographic areas. For example, one might study citizens, households, countries, U.S. states, U.S. or foreign cities, legislation introduced in a legislative body, roll call votes of legislators, laws, newspaper articles, court decisions, or words used in speeches of prominent politicians, to name just a few. Different units of analysis present different challenges. For example, even if a researcher compares all the countries in the world, he or she would still have a limited number of observations—just under 200 depending on how one defines a country. (Yes, just about everything in political science confronts a definitional problem.)⁶ Survey data from a survey of U.S. households, on the other hand, often entail far more observations (typically over 1,000 households) but confronts challenges in ensuring that those households studied are representative of the larger U.S. population.

Comparing a study of countries and a study of households illustrates two more distinctions in the types of data political scientists study. The first of these is the level of analysis. The level of analysis refers to the scale of the data, or whether or not they have been aggregated. For example a country is made up of millions of households and households are made up of several individuals. At the micro-level, an individual earns an income. At a slightly higher level of analysis, all the individual incomes in a household can be added to yield the household income. Scaling up yet another level, household income can be combined from throughout a country (along with the income from firms and a few other sources) to derive the Gross National Income.

Moving from the micro-level (e.g. individual) to the macro-level (e.g. country) is known as aggregation, and moving in the opposition direction is known as disaggregation. Often times we are more interested in aggregated data, particularly for making descriptive inferences. For example, if we conduct a survey of 1,000 Americans about whom they plan to vote for in an upcoming presidential election, it doesn’t really tell us much to know that respondent number 342 favors the Republican candidate. Instead, we would rather “aggregate” all the individuals’ responses to learn that 51 percent of surveyed Americans favor the Republican candidate. As we will see below, however, aggregated data generate their own challenges.

There is another important difference in a study of countries and a study of households. In a study of countries, it is possible (although often difficult) to collect data for all countries. In a study of households, doing so is extremely rare. Once every ten years the U.S. Census Bureau does attempt to conduct a census, or a survey of all U.S. households; however, political scientists do not have this luxury. Instead, political scientists interested in public opinion typically study a sample, or subset, of the larger population, or universe of subjects. How that sample is selected is an essential challenge to making descriptive inference. If inference is using the facts we know to generalize about the facts that we do not know, then it is essential that our sample be representative of the population that we wish to generalize about.

Data can vary in other ways as well. While some data capture a snapshot at one point in time, what we call cross-sectional data, other data include changes over time, known as longitudinal or time series data. For example, a cross-sectional study might measure the level of corruption across all countries for a given year and then compare a...

Table of contents

• Cover
• Title
• Copyright
• Dedication
• Contents
• Preface
• Acknowledgments
• Introduction
• SECTION I Establishing the Framework
• SECTION II A Menu of Approaches
• Index