Political Science Research Methods in Action

M. Bruter, M. Lodge, M. Bruter, M. Lodge

• Publisher: Palgrave Macmillan
• Year: 2013
• Print ISBN: 9780230367753
• eBook ISBN: 9781137318268
• Topic: Politics & International Relations
• Subtopic: Political Philosophy
• Index: Politics & International Relations

Through accounts from innovative research projects by world-leading political scientists, this volume offers a unique perspective on research methodology. It discusses the practical and intellectual dilemmas researchers face throughout the research process in a wide range of fields from implicit attitude testing to media analysis and interviews.

1

Introduction: Writer’s Block

Michael Bruter and Martin Lodge

A social scientist’s writer’s block is not the same thing as a novelist’s writer’s block. ‘I just can’t find an interesting enough idea for a book’; that is a novelist’s writer’s block. ‘I’ve got a hell of a good idea but how on earth am I going to transform it into half as good a research project?’; that, by contrast, is a social scientist’s writer’s block.

It is true that the work of a social scientist is highly creative, which is one of the reasons why we all find it so stimulating. It is also true that you cannot do research without, first and foremost, a good idea. Nevertheless, the truth of the matter is that most of our time is not spent in the luminous glory of the drawing room but back in the darkness of the engine room. There, good ideas produce research questions, and the existing literature is dissected with about as much consideration as that of the biologist for the frog, whose corpse he is studying. There, research questions generate hypotheses, and the data that already exist or could be gathered to test these hypotheses are weighted, evaluated, and compared with more precision than fantasy. That is our engine room: a place which may not be glamorous but is certainly essential and where, not unlike craftspeople, we continuously work on the machine, which, in turn, will enable us to produce research. Without good and hard work in the engine room, our great idea will never become good research, and nobody may even realize that there, once laid, a good idea. In fact, it is in the engine room, where we construct, shape, and reshape this ad hoc ‘research machine’, in fact, that great ideas become social science.

In its full complexity, the construction of that machine – which is, incidentally, also known as a research design – is never fully apparent to those who consume our research. Of course, any rigorous social science book or article provides numerous details as to the choices that were ultimately made – the questionnaires that we created, the way in which we conducted our interviews, or the manner in which we coded discourses and manifestos. But that is really only what the machine looks like at the end, and it does not say anything about the way it was built, what choices were made, and why. How a research design is put together to tackle a specific research question, how we decide, in the engine room, how it will be powered and how not, where to put the screws, and what it should be able to resist, as well as how to realize when things are not going well and what to do about it, is what this book is precisely concerned with.

The principal aim of this volume is to offer an insight into the engine room of select research projects. As such, therefore, it is in between a grandmother’s recipe book and the academic equivalent of Joseph Roth’s Confession of a Murderer (1936). It is a collection of accounts, by a number of social scientists with different methodological preferences and approaches, of how they set out to do research. In their different ways, all contributors explain how they set up their research design, and how they encountered and dealt with critical choices and dilemmas. Just like any craftsperson, the social scientist has to choose between competing paths to optimally make something ‘work’. Each chapter therefore discusses how solutions to particular problems were dealt with, and what the methodological, ethical, and analytical consequences are that should be borne in mind when evaluating the research that results from these choices. The picture that we deliver may neither be that of a high precision laboratory, nor be that of a workshop populated by dodgy cowboy builders, but it includes a series of honest descriptions, by people who are thoughtful, introspective, and passionate about the methodological choices they make as they try to be aware of their limits and potential pitfalls.

Understanding the object of the book: The Bible(s) and the Talmud (or the rules of methods vs the consequences of methods)

Theologists often contrast aspects within religions that emphasize canonical certainty on the one hand and continuous deliberation and debate on the other hand. One typical example distinguishes between the Bible (in any of its numerous Judeo-Christian versions) and the Talmud. The former is said to be defined by its certainty, its absence of room for doubt, and its focus on good and evil. The latter, by contrast, is a permanent, never-ending debate, whereby some religious authorities never reach any agreement on the correct interpretation of one or other aspect of religious rules. In its various forms, this tension between religious prescription and religious debate has split numerous religions from Judaism to Hinduism and Protestantism. The existing literature already offers a number of excellent ‘bibles’ on research design. They may be either general or dedicated to specific designs. Some of the most influential books written on research design include Przeworski and Teune (1982), King, Keohane, and Verba (1994), Penning et al. (2005), Coombs (1964), and Shively (2005). To a certain extent, all of these volumes have a prescriptive value, and tell researchers what they should and should not do when preparing to conduct a piece of social science research. In fact, this perceived prescriptive value is such that at least two of these volumes are regularly nicknamed ‘the bible’ by their respective supporters. And as any text associated with religious status, these books have similarly received their fair share of commentary (e.g., Brady and Collier, 2004).

By contrast, this volume has no claim to be yet another ‘bible’. It is, instead, very much a ‘Talmud’ – a perpetual discussion of the methodological, ethical, and analytical dilemmas that most social scientists come across when engineering their research designs in the face of a messy reality. It is a book which focuses more on the questions than on the answers. These few hundred pages may well disappoint those who seek categorical and universal simple solutions but will hopefully stimulate those who like to think about their own work, have ever experienced doubt about how to tackle an apparently tricky research question, and who chose the social sciences because of their complex human and social object of study and not in spite of it. Of course, the various methods and approaches presented in the following pages are all deemed by the authors to be useful solutions to difficult problems, but none is presented as a panacea or even as a problem-free option. They are not even described as ‘compromises’, because we are not ‘bargaining’ ethics, analysis, and methods. Instead, we simply think of them as dilemmas and choices. As such, they are discussed in the context of their positive and negative as well as their intended and unintended consequences, even though each of the scholars who defends them is ready to (and does) explain why they think that the particular positive–negative balance of their approach is overall preferable to its alternatives in specific contexts. This is also why this book has in many ways more to do with how to deal with problems in research design than in trying to come up with ideals of research design. This is a ‘plan B’, perhaps, but, we believe, an important one, because in a world where the science is in the method, how we think and talk about our methods is of fundamental value, not purely in a pedagogical sense but also in a social sense. The book tells the story of how learning takes place in so-called communities of practice with their ongoing exchanges about experiences in researching social phenomena, rather than through the repetitive chanting of prescriptive ideals.

Yet another way of describing the specificity of our book is by explaining that it is not so much concerned with the ‘rules’ of methods as it is interested in the ‘consequences’ of methods. The contributors are not authoritatively explaining how a method should be used; they are, instead, sharing their reflections on and experience of the consequences of the methodological choices they like to make and to avoid. They provide a first-hand account of the ‘secret’ considerations that have led them to make their own decisions in the context of complex research projects which, in their combination, represent a rather broad spectrum of the types of research question, approaches, and methods that social scientists focus on across the range of our disciplinary subfields.

Genesis of an engine-room project

Just as the craftsperson with their tools and the materials available to them, the social scientist is faced with numerous challenges in terms of moving from an interesting idea to a working product. In this section we discuss the genesis of a research project from an engine-room perspective. We are dealing with this experience at an abstract level because the following chapters will add their individual flavour to them. We should also stress that the ‘genesis’ approach discussed here does not suggest that every research project follows linear paths. Far more frequently the progress of research projects requires that we step back and reconsider some previous infelicitous choice only revealed by a later dead end. Nevertheless, the following stages describe what many of us consider to be the typical natural birth process of a research project.

As we have already suggested, a good project always starts with a good idea. This is the first and irreplaceable step in the genesis of a social science research project – genesis of a research project, ‘day one’. Good research starts in one’s head, and no geeky knowledge of the literature and no engine fiddling will ever replace some hard thinking about what we – social scientists and citizens – need to know. Social scientists face a permanent challenge when it comes to taking enough distance to understand what our discipline has still failed to address, what is the gap that has remained unnoticed between two important advancements of our collective knowledge, what are the apparent contradictions between studies which suggest that a question has still not been taken to its end. Sometimes, finding an idea might seem daunting. At other times one seems to have far too many ideas to ever stop working. But the coherence of this creative first step in the genesis of social science research comes from something that it is purely introspective. Nobody can find an idea for someone else, and without an idea, and preferably a good one at that, there can be no research.

With regard to this first stage, let us focus on the example of Sarah Harrison’s contribution (Chapter 3). She spotted that in between works assessing the respective impacts of institutions, the economy, other parties, voters’ demographics, and so on, on the success of extreme right parties, something was missing in the literature – the impact of the discursive and ideological choices of extreme right parties themselves. Looking at whether these extreme right parties’ choices matched the preferences of their specific voters, and whether this relative match influences the success of these parties, is something that, paradoxically, had never been touched upon by the literature before. The ‘paradoxically’ is not unimportant here because the idea makes intuitive sense without being the object of academic exploration before. In that and other senses, it seemed important enough an idea to generate research.

‘Day two’ of the genesis, however, is the transformation of an idea into a research question. This stage is no less indispensable than the existence of the good idea, which serves as its prerequisite. While the original idea for research may be a theme, or an intuition, the research question is a specifically constructed line of investigation, a workable question, which will be directly, fully, and specifically answered by the research work that will be created for it. A research question has to be set out transparently and explicitly, and any book, article, or thesis will be purely dedicated to answering it, fully and exclusively. It is at the end of this second stage of engine-room proceedings that the social science sketch becomes a recognizable project, and that research is truly on its way as a liveable enterprise. Suddenly, fascination gives way to hard analytical thinking, and creativity to creation.

When it comes to understanding the transformation of a research idea into a research question, let us consider Michael Bruter’s contribution (Chapter 2). His idea was to study the emergence of a mass European identity. However, because of the history of identity building and the importance of identities as markers of interaction between citizens and political institutions, this led him to focus on a far more specific research question: Can political institutions, via the generation of political system symbols, and the mass media, via the way they inform us, encourage or impede the emergence of a new mass European identity? The idea was an important theme that needed social science attention, but the research question was a specific line of investigation which could be answered by a research design and a book. In other words, within the framework of this particular phrasing of the research question, the author could formulate a set of hypotheses that would theoretically answer the question, and create empirical tests that would assess whether these hypotheses were correct.

On ‘day three’ of the genesis, the social scientist spends a month in the library trying to get a pretty-near-exhaustive knowledge of the relevant literature – in many ways, what architects would call ‘exploring the terrain’. One of the difficulties of this particular step is to understand what exactly constitutes the terrain. For instance, if a social scientist decides to study democratization in Russia, the terrain will certainly not be limited to the literature on Russia, as theories used in other contexts and countries are the most likely to help the social scientist to build a novel model adapted to his own case study. Similarly, a study of the motivations of anti-globalization protesters will imply a need to cover the literatures on political protest and political participation more generally, or an enquiry into the determinants of tax policy may require a knowledge and understanding of the literatures on neighbouring public policies as well as political economy and bureaucracy. Unfortunately, there is obviously a fine line between missing out on some essential and relevant literature and comparing ‘apples and oranges’. That is why the process of reviewing the existing literature is not just one of digesting existing accounts but also an interactive (if partly ‘virtual’) discussion between the researcher and his academic environment.

An excellent example is provided by the Mark Franklin and Maja Renko’s contribution (Chapter 5). The emergence of the Propensity to Vote (PTV) solution was only conceivable on the basis of a very comprehensive understanding of the electoral behaviour literature, looking at both the item phrasing in existing questionnaires and the apparent substantial limits or paradoxes of existing findings about voters’ choice. It is on that basis that Franklin and Renko could provide this new solution to an old but not fully understood problem affecting an entire body of literature.

On ‘day four’ of the genesis of a research project, the social scientist will normally develop a model, which can answer the proposed research question theoretically and will be enriched by the literature review that was conducted previously. Indeed, the literature review stage of research project engineering is useful not only to understand what we already know and to avoid replicating existing knowledge but also to use others’ insights to develop the theoretical and analytical model that we will want to test in a given project. A model is very much a simplified version of how we believe the world works. It usually takes the form of a series of testable and falsifiable hypotheses, which the research project will confront to determine what we can empirically find out about the real world. ‘Testability’ means that there exists empirical evidence which could be used to verify whether a hypothesis is indeed upheld in reality (as opposed to something that we could never check in practice); ‘falsifiability’ means that the hypothesis is phrased in such a way that if it is ‘wrong’ we should also be able to find that out (as opposed to a phrasing which could never be disproved because it is too vague or too general).

Here, a typical example is contributed by Robert Erikson, Aaron Strauss, and Michael Bruter (Chapter 11) who use pure modelling and simulation to (re)frame the problem of understanding how parties choose their election manifestos. The authors realize that traditional expectations – not least that of convergence under Downsian assumptions – are not actually verified in real life. They also note that existing research had not yet been able to come to terms with these paradoxes. They thus propose to model the choice of their manifestos by two political parties competing in a given party system. They consider the variables that are expected to influence this choice, such as a party’s relative preference for vote-seeking or policy-seeking priorities, what we know of an electorate’s true policy preference, which party is expected to win and by how much, and how the other party is framing its own manifesto. They then model the way in which these factors may interact for each of the two parties competing in the election. This leads them to a set of mathematical simulations which help us to understand in theoretically meaningful terms under what conditions we expect the two parties to find an equilibrium, and under which circumstances they will not, chasing each other continuously instead. In this case, the model is run purely mathematically without any empirical test, but, ultimately, all social science models hope for an adequate test to be designed and applied.

The design of an appropriate test is precisely what normally constitutes ‘day five’ of the research enterprise – it is the equivalent of choosing tools for the job in hand. Now that we know what theoretical propositions need to be assessed, how can we choose the most appropriate empirical tests to do so? Can our research question be answered equally well using a quantitative or a qualitative research design? What is the most appropriate empirical test for the model that we propose? What data are available or may realistically be collected, and are they relevant to the way in which the question and hypotheses are framed? For instance, a research question which has to do with a party’s ideology may rely on an analysis (quantitative, qualitative, or both) of party manifestos (Bruter and Harrison, 2011), or on interviews of party leaders, or again on evaluations of the party’s policy either by experts (de Lange, 2007) or by the public at large via mass surveys. However, these different types of possible empirical test will have various consequences for what specific questions can be answered and also, of course, the type of bias – or systematic error – that may contaminate the results. For instance, relying on public perceptions might mean that we could measure a population’s prejudice as much as (or conceivably more than) the reality of a party’s ideology. Conversely, relying on interviews of party leaders may mean that we capture a set of ideological and policy perceptions which may not be shared by the party members, only by their leaders. In this sense the choice of a methodological protocol is a crucial aspect of the craftsperson’s work in the engine room. Indeed, it is not only the stage when the social scientist will perform a background census of the methodological options that exist in theory (How could we an...

Table of contents

• Cover
• Title Page
• Copyright
• Contents
• List of Tables and Figures
• Notes on Contributors
• 1. Introduction: Writer’s Block
• Part I: Approaching and Measuring Social Science Objects
• Part II: Making Sense of Data
• Conclusion
• Index