Methodological Challenges and New Approaches to Research in International Development

L. Camfield

• Publisher: Palgrave Macmillan
• Year: 2014
• Print ISBN: 9781137293619
• eBook ISBN: 9781137293626
• Topic: Politics & International Relations
• Subtopic: International Relations
• Index: Politics & International Relations

Development researchers face many challenges in producing robust and persuasive analyses, often within a short time-frame. This edited volume tackles these challenges head-on, using examples from other fields to provide practical guidance to research producers and users.

1

Introduction

Laura Camfield

1.1 Background

When leading UK politicians commit to spending 0.7 per cent of Gross National Income on overseas aid from 2013 and the main distributor of this, the UK Department for International Development (DFID), states its commitment to the “three pillars” of research, evidence and evaluation, surely there has never been a better time to be a development researcher? Yet while in the UK and North America there is increasing demand for research presented in the form of “evidence” or “key messages” and pressure on researchers to demonstrate their policy “impact,” what is the quality of the research that underpins their claims? The chapters in this volume recognize that research quality may be compromised by donor interests, short timescales, and limited reflection (Humphrey, 2007; Behague and Storeng, this volume). They present conceptual frameworks and methodologies that represent new directions for development research, alongside examples of good practice in other fields such as sociology and anthropology (for example, qualitative secondary data analysis and “revisits,” Bornat and Crow this volume). By bringing together scholarship from the Global North and South, the contributors challenge the assumed separation between developed and developing countries (cf. Humble and Smith, 2007). These examples are valuable to development researchers since there is a growing expectation that data will be available for analysis/re-analysis and that development policy will be based on evidence of all types, appropriately assessed for quality.

The volume arises from a concern with the quality of data and its interpretation, which is framed around a discussion of the relationships involved in gathering data, analysing them, and addressing particular audiences. For example, the relationship of the development researcher to the people they research, the people they work with (peers, students, etc.), the people who fund them (donors, tax payers, etc.), and the people who use their analyses (policymakers, practitioners, activists, journalists, etc.). In thinking about what constitutes quality in terms of data and analysis, or how relationships shape research, we see three main challenges: maintaining research relationships and developing appropriate methodologies for studying the relational dimensions of poverty (Section I); time and changes over time (Section II); and analysis and representation (Section III). Section I looks primarily at research relationships, drawing on papers by Arvidson, Taylor et al. and Crow, but also at how the relational dimensions of poverty can be taken into account in resource-poor communities (Coulthard et al.). Section II addresses longitudinal studies at individual, household and community levels and the particular challenges posed by changes in the object of study as households disperse and reform (Nasirumbi et al., P. Davis), communities change in response to policy and intervention (Bevan) and children develop (Roelen). Finally, Section III focuses on analysis and representation, exploring the implications of the growth in data deposit, which enables secondary data analysis and re-analysis/replication of influential analyses, and the power of evidence derived from Randomised Controlled Trials.

1.2 Quality of research in international development

In the remainder of the introduction I describe problems particularly apparent in development research (aka the “global poverty research industry,” Green and Hulme, 2005). These include the absence of theorization (Harriss-White, 2007) and challenges in establishing the quality of research due to lack of methodological documentation and access to datasets (Camfield and Palmer-Jones, 2013). I then outline the contents of the chapters in more detail. I argue that the problems I describe arise from the funding and structure of development research and are common to both applied and basic research. They include neglect of the researcher’s positionality (Lewis and Opoku-Mensah, 2006; Bakewell, 2007), unreflective assumptions about the quality of large datasets (see Duvendack and Palmer-Jones, this volume) or the (in)completeness of interview transcripts (Jackson, 2012), failure to acknowledge the mediating effects of transcription and translation (Temple et al., 2006), and disconnects between data production, analysis and theorization. These problems could be treated as a social fact that needs to be located in a deeper understanding of how the development sector works (e.g. Behague and Storeng, this volume, ask whose interests are served by particular epistemological standpoints). However, it may also be possible to treat them as a practical problem that can be partially addressed using some of the approaches outlined in this volume. For example, Baillie-Smith and Jenkins (2012, p. 75) propose “considering the potential contribution that emotional methodologies may make to generating alternative knowledges of development” through “key concepts of responsibility and solidarity,” building on earlier work on the “boundaries and borders” that are negotiated in doing qualitative research on development (Smith, 2007). For this reason the volume has dual and potentially competing goals – both doing development research better and seeing the field of knowledge in development as telling us something about that world and the interests that define what is important or understandable in it.

My discussion of the background to these problems is informed by Bourdieu’s concept of “epistemic reflexivity” (Bourdieu and Wacquant, 1992), which extends from the individual researcher to encompass scientific practice and the organization of disciplines and fields. Bourdieu argues that a rigorous approach to social science requires us to “constantly scrutinize … the collective scientific unconscious embedded in the theories, problems, and … categories of scholarly judgment” (ibid., p. 40). This involves not only reflecting on our practice as development researchers, but also understanding the social and economic forces within and outside our field that shape categories of thought and action, including “the constructed and highly political nature of the boundaries, borders and identities on which the development project and idea has rested” (Smith, 2007, p. 3) (see also Duvendack and Palmer-Jones and Behague and Storeng, this volume). The relevance of these points will become apparent in the next section as I discuss the problems evident in development research.

The quality of research in international development, as in many applied fields, is extremely variable. There often seems to be an inverse relationship between the strength of the evidence base and the vehemence with which findings are asserted. This point is made respectively by Duvendack and Palmer-Jones in relation to micro-finance, G. Davis on governance, and Behague and Storeng on emergency obstetric care (this volume). These disconnects reflect the political economy of research in international development (Mohan and Wilson, 2005; Bakewell, 2007; Harriss-White, 2007) (for example, the particular nexus between DFID, the World Bank¹ and research in some UK universities). A similar critique is made of US poverty research in O’Connor (2001) who suggests that its professionalization in the latter half of the 20th century moved it further away from the experience of poverty and closer to the interests of national government and private foundations such as Rockefeller. This change was signalled by its increased dependence on human capital theory, econometric methods, large datasets, and experimental programme designs, which has resonances with contemporary development studies. Harriss-White (2007, p. 48) sees a similar rationale behind the increase in commissioned policy research due to the outsourcing of core policy functions by bureaucracies such as DFID. This creates “commodified policy,” “result[ing] both in a decline in the quality of evidence used in policymaking and in an increase in the difficulty with which the residual bureaucracy processes and interprets the outcome of this process.” The increase in funding for commissioned research, at a time when research councils and universities are reducing their expenditure on research, has supported an exponential expansion of consultancies and self-funding research institutes in developed and developing countries (Lewis and Opoku-Mensah, 2006, p. 669). Due to the competitiveness of the sector there is little coordination between funders or researchers, which can lead to multiple simultaneous evaluations of high profile programmes with no collaboration or data sharing ( Jones and Young, 2007). Harriss-White (2007, p. 48) argues that “what is most striking about the lists [of research topics generated in the plenary of the IDS 40th anniversary conference] is that these are themes in search of theories. Development studies is increasingly defining itself in relation to policy … and it is the impetus coming from policy that makes us define development problems as themes rather than as problems for theory” (see also Lewis and Opoku-Mensah, 2006, p. 671). In another paper she expresses concern that the emphasis on “relevance” by research funders constrains the types of research that can be done as what is considered relevant reflects “cyclical fashions” among policymakers and entrenched local interests (Harriss-White and Harriss, 2007, p. 20). These problems are known and structural, so this volume is intended as a constructive engagement with those involved in data collection and analysis.

As discussed above, there are strong incentives for development researchers to “present a smooth ex post account of research methods (often airbrushing dead ends and false starts)” (Prowse, 2010, p. 211), although this phenonmenon is not confined to development. Prowse suggests that these accounts do a disservice to policymakers as well as other researchers because fully describing the context of primary research (for example, the way the research team was structured) and its philosophical standpoint “can help to explain how research findings are generated, how robust findings are, and how findings can or cannot be extrapolated” (ibid., p. 211; see also Crow and Bornat, this volume). Sumner (2012) makes a similar point in his blog in relation to the depoliticization of mainstream poverty research which he says cannot be blamed on the “measurement obsession,” but relates instead to a reluctance to “embed poverty research within an analysis that includes distribution, social differentiation and the process of economic development – in short the political economy of poverty.” The examples he gives are lack of attention to inequality (for example, failing to ask “‘who are the rich?’ and ‘why are they rich?’”) and a tendency to treat poverty as a residual rather than a structural outcome.

So what is going wrong? Giri and Van Ufford (2004) suggest there is a clear distinction between the research priorities of what they call the “world of action,” where manageability is the central concern of those commissioning development research (e.g. the results-based management agenda, critiqued in Eyben, 2013), and the “world of reflection” (academia). They suggest this distinction is relatively recent as “in the past development sociologists and anthropologists had stressed the importance of their academic research for improving development interventions. But now critical scholars tend to focus on analysing the hegemonic relations which development entailed” (ibid., p. 8). The result of this is that “the domains of critical understanding and developmental action became increasingly separated from each other” and researchers engage in “sustained scientific criticism … not accompanied by a passion of reconstructive responsibility” (ibid., p. 3).

One example of this tension was the reception of David Mosse’s account of his experiences working with a rural development programme in India, which Eyben (2009) suggests illustrate some of the fundamental differences between academic research and consultancy. Ex-colleagues of Mosse felt they could ask for changes in the typescript because they saw the book as a form of project report, i.e. a consensus document that aimed to define “the truth” of what the project was about. The ensuing debate (see Mosse, 2006; Sridhar, 2005) supported Eyben’s (2009) conclusion that “development organisations may be particularly competent at exercising the privileges of power … because of their quasi-religious function whereby power is legitimated by reference to ‘the poor’ for whose sake the organisation exists” (pp. 93–4).

Although Giri and Van Ufford (2004) distinguish between applied and basic or “pure” research, Harriss-White and Harriss (2007) argue persuasively that no research is “pure” and well-funded research centres face many of the same pressures, in addition to challenges that relate to the way development research is structured. For example, as I discuss later, research centres frequently operate with a “data extraction” model where data are collected in the study countries and analysed in the UK. There is little support given to the professional development of junior researchers in country teams and limited space and resources for more senior researchers in country to analyse the data they have collected. Norton (2012) suggests in his blog that this relates to a lack of research capacity (or incentives to develop it) “with southern researchers pulled into consultancy for aid agencies or advising wealthy businesses how to make higher profits” (a point also made by Lewis and Opoku-Mensah (2006) in relation to NGO research). He argues that “serious capacity building [in the social sciences] will need significant support from development partners … a major initiative and not sticking plasters.”

But while social sciences departments in African universities undoubtedly receive little financial support, and the support they do receive is often tied to other people’s agendas and/or framings of problems, this does not let individual research programmes off the hook. The same separation of data production/ management and analysis can be seen within UK teams and UK-based projects, which maps to hierarchies within research teams and the social science more broadly. Mauthner and Doucet (2008) suggest that collaborative and team-based research “relies on a division of labour that creates divisions and hierarchies of knowledge, particularly between researchers who gather embodied and contextual knowledge ‘in the field’ and those who produce textual knowledge ‘in the office’” (p. 971). Textual knowledge is seen as more objective and thus has higher status than knowledge gained through participating in fieldwork and is rewarded accordingly. However, by focusing solely on the interview transcript and demoting the embodied knowledge of the fieldworker to “background information” much of the context that gives meaning to the transcript, and that the interviewer would be aware of, is lost. As Bourdieu et al. (1999 in Mauthner and Doucet, 2008, p. 976) suggest “everything that came up in the interview – which cannot be reduced to what is actually recorded on the tape recorder … tends to be stripped away by writing … everything that often gives the real meaning and the real interest.”

The dependency on survey and transcribed interview data is particularly common in development research where there is a stark separation between field workers (contract researchers and/or students in the study country who carry out the fieldwork), field researchers (typically UK-based contract researchers, who manage the fieldwork and the data), and academic researchers (UK-based grant holders, who play a more strategic role in the management of the research and often write the papers). These hierarchies are justified by a perception that fieldwork is “a technical activity that can be done by anyone, rather than an intellectual process in which meaning and knowledge are being shaped and created by subjective researchers” (Mauthner and Doucet, 2008, p. 979). Recognizing field staff as “intellectual partners … equally engaged in the production entails a radical shift in power relations and dynamics within research teams” (ibid.). While this would be a radical shift in development research practice, the debates around the viability of secondary analysis of qualitative data – see Bornat, this volume – suggest that the “cultural habitus” acquired through participating in fieldwork is essential to data analysis. Without it researchers can succumb to what Mauthner et al. (1998) call “naive realism” where transcripts or extracts of multiple transcripts coded in a qualitative data analysis programme become the data, divorced from both fieldworkers and respondents. There are clear limits to the interpretations that can be made from these data and these cannot be easily circumvented. Decontextualization is not inevitable as some UK-based studies also draw on detailed field notes and/or oral or written debriefings, although these require a different relationship with field workers, as described in Bevan, this volume. These data provide some of the specificity of the encounter between interviewer and respond...

Table of contents

• Cover
• Title
• Copyright
• Contents
• List of Tables, Figures, and Maps
• Acknowledgements
• Notes on Contributors
• 1 Introduction
• Section I Relationships
• Section II Time and Changes over Time
• Section III Analysis and Representation
• Index