World Yearbook of Education 2019

Comparative Methodology in the Era of Big Data and Global Networks

Radhika Gorur, Sam Sellar, Gita Steiner-Khamsi, Radhika Gorur, Sam Sellar, Gita Steiner-Khamsi

• Publisher: Routledge
• Year: 2018
• eBook ISBN: 9781351376853
• Edition: 1
• Topic: Education
• Subtopic: Education General

Digital methodologies, new forms of data visualization and computer-based learning and assessment are creating new challenges as well as opportunities for scholars in educational research. The World Yearbook of Education 2019 explores this highly relevant topic, opening a new discussion about the various conceptual and methodological challenges and opportunities in contemporary educational research.

This volume explores contemporary methods of inquiry, with chapters organized around four topics of enduring interest in this field: impacts, patterns, relations and contexts.

The World Yearbook of Education 2019 comprises contributions from internationally renowned scholars exploring novel concepts and methodologies in grappling with contemporary empirical phenomena in educational research. Vital questions such as how we understand the technological developments that are creating new possibilities for and demands on education, and how we make sense of complex cases that cut across multiple nations, are discussed.

This newest addition to the prestigious World Yearbook of Education series provides a fascinating read for scholars in the fields of education policy and comparative education. It is not only a useful resource for educational researchers and policy makers examining new trends and emerging issues, but would be of interest to graduate students exploring innovative methodologies, particularly in the study of education and education policy.

1 Big data and even bigger consequences

Radhika Gorur, Sam Sellar and Gita Steiner-Khamsi

Globalization has been a central theme and a salient backdrop for scholars in social research for several decades.¹ In educational research, the topic gained prominence among scholars associated with comparative policy studies. The period of globalization between 1989 and 2008 was welcomed by some as an opening up of possibilities and breaking down of boundaries, inducing calls to overcome “methodological nationalism” and explore new, dynamic methodologies that could cope with the movement of people, goods, ideas and money across various “scapes” (Appadurai, 1991; Sassen, 2007).

The World Conference on Education for All held in Jomtien in 1989 was a key event that set in motion the continuous institutionalization of a global script for education in developing countries (Chabbott, 2015). Culminating in a global agreement to push for universal access to education, it triggered the development of a series of national action plans, accompanied by processes for monitoring the implementation of these plans globally. This vision of Education for All as an urgent, global imperative was consolidated in 2000 at the World Education Forum in Dakar, where specific international education goals were set out in the Framework for Action. That same year, the Millennium Development Goals (MDGs) were endorsed. These events and global strategies did much to consolidate a sense that global strategies were needed to address a global education crisis.

A global script of a different kind emerged at the turn of the millennium. This time, global reform encompassed all countries, that is, developing countries as well as Organisation for Economic Cooperation and Development (OECD) countries, and propelled a shift from inputs to outcomes. The liberalization of inputs encouraged non-state actors, including businesses, to provide goods and services in education systems that are funded by the state. As a corollary, choice, vouchers and charter school policies mushroomed in different corners of the world. Rather than regulating the inputs and provision of education, governments were urged to focus on establishing learning standards and monitoring learning outcomes. Monitoring entails measurement. The measurement turn, or governance by numbers, produced an ever-expanding test industry in which individuals are assessed at critical stages of their school career and throughout their lifespan. Twenty years after the market reforms in education, education systems are drowning in data and their administrators have become champions in international comparison. Test-based accountability of educational systems (Verger & Parcerisa, 2017) has become the norm rather than the exception.

The measurement turn in educational research was accompanied by a vociferous rhetoric of global competencies, 21st-century skills and the urgency to prepare national populations to compete in a global workforce. The OECD has been instrumental in exerting “soft” pressure on national educational systems to comply with international “best practices” by way of reviewing, measuring, comparing and advising governments. In effect, governments authorized the OECD to symbolically accredit their own national educational system through Programme for International Student Assessment (PISA). The results on the test became a stamp of approval certifying that the system indeed prepares students for global or transnational skills needed in the 21st century. The power of OECD as a symbolic transnational accreditation agency in education is not to be underestimated. In an era of global interconnectedness, the growing influence of transnational authorities poses a great threat to public education systems that by definition operate nationally. National public education systems are at a disadvantage compared to multinational agencies, such as global education businesses, that are able to use English as a language of instruction, focus on global competencies, or evoke the imaginary of the mobile, cosmopolitan citizen, while at the same time downplaying national curricular content, which increasingly is seen as parochial and outdated.

The desire for an internationalized, cosmopolitan education has now grown in all segments of society (that is, beyond the middle class), and this is clearly apparent in the shift, in many low-income nations, to low-cost private schools, often teaching in English and with “international” or “global” in their names. Curricula are being revisited to include content about different countries and cultures, exchange programs are being developed and many universities are recruiting staff from around the world to “internationalize” the faculty. Overseas campuses of American and European universities have sprung up in Asia and the Middle East. Internationalization has become one of the parameters by which universities are ranked. These developments have brought in their wake a variety of international surveys and rankings, and with them the methodological challenges associated with mobility, commensuration and comparison.

For the field of comparative education, the sense of open borders and global imaginaries was associated with a growth of regional and global large-scale assessments (Sellar & Lingard, 2013), a convergence of policies (Steiner-Khamsi, 2006), a slew of global ranking exercises and the search for “best practices” (Grek, 2013). During the 1990s, the involvement of international organizations in education policy began to give coherence to a global field of education policy shaped by the discourses and data analyses of the OECD, World Bank and UNESCO (Lingard, Martino, Rezai-Rashti, & Sellar, 2016). At this time, the seeds were being sown for the emergence of neoliberal policy discourses that have been characterized as a Global Education Reform Movement (Sahlberg, 2011) or School Autonomy With Accountability (SAWA) (Verger & Parcerisa, 2017).

Then the global financial crisis in 2008 abruptly confronted us with the limits of capital (Wallerstein, Collins, Mann, Derluguian, & Collins, 2013), introducing new tensions within the European Union and elsewhere. The confident neoliberal theory of change premised upon marketized approaches, competition, devolution of responsibility, transparency and accountability failed in its bid to improve educational outcomes, and instead presented governments with a whole new set of problems, including rising inequities. Today, confronted with growing uncertainties, we are witnessing a reassertion of nationalism, isolationism and exceptionalism, to which unexpected events such as Brexit, and the election of Donald Trump and of far right and nationalist parties in several European nations, bear witness. The growing political fragmentation, an almost unprecedented humanitarian crisis, extreme inequities, the current brinkmanship and talk of trade wars and protectionism (in contrast to free trade and the idea of a global marketplace), as well as a dystopian sense of uncertainty that characterizes the world today, demand that any study of societies attends to multiple scales and contradictory national and global impulses.

Intertwined with these upheavals are extraordinary technological developments. Where once social media were celebrated for breaking down boundaries and connecting diverse groups, today they are also, simultaneously, implicated in creating ideological silos and echo chambers of “fake news.” Even as knowledge is increasingly being made public, and people are exhorted to become informed participants in governance and public life, it is becoming increasingly difficult to tell facts from fiction. Big data, digital methodologies, new forms of data visualization, and computer-based learning and assessment are on the rise, both as research tools and objects of investigation.

The MDGs have given way, in 2015, to the Sustainable Development Goals (SDGs) – a 15-year global framework for providing access to quality education for all. Different from the two earlier global scripts, Education For All (EFA) and the MDGs, the SDGs are not only addressed to developing countries but to each and every government in the world. In addition, inscribed in the new global script is the unabashed commitment to quality of education, and thus, to measuring and monitoring the quality of education. Without any doubt, the shift in focus from access to education in the MDGs to the quality of education and learning in the SDGs has given rise to new waves of regional and international comparative assessments, as well as a renewed set of global reform strategies, promoted by the large development agencies and international nongovernmental organizations (INGOs) acting in concert. The OECD has come up with a variation on its signature PISA in the form of PISA for Development. UNICEF is promoting the Southeast Asia Primary Learning Metrics (SEA-PLM) which, it says, “directly supports the regional standardisation of data around learning and the mobilization of technical expertise, institutional capacity and political will to translate the data into action”². This epitomizes the current global project in the pursuit of SDG4 (the education SDG), which involves a combination of standardization of data gathering, capacity building to enable and sustain this data project, and advocacy to influence governments.

The theory of change that is driving current global efforts to improve learning outcomes includes a strong preference for decentralization and the devolution of responsibility to the school or village level, accompanied by strong accountability practices to encourage fidelity in implementation. This has meant not only more testing of students, but the creation of extensive and highly descriptive databases and management information systems, supported by a variety of software packages.

For scholars in educational research, this historical moment, shaped by contradictory impulses of globalization and resurgent nationalisms, the rise of fundamentalism and far-right politics amid involuntary migration at an unprecedented scale, and widespread speculation that technological change will cause unprecedented disruption, presents a fresh set of conceptual and methodological challenges. The field of comparative education has always been methodologically diverse, inviting the attention of a range of scholars, including anthropologists, economists, historians, philosophers, political scientists, sociologists, as well as psychometricians and statisticians. They have responded in a variety of ways to the emerging challenges of recent decades with methodologies that both reflect and affect changing societal and educational contexts.

Comparison as a way of knowing

Although large-scale international statistical comparisons in education are now routine, they were once thought impossible to carry out, given the vast differences in social, cultural and financial contexts between countries, and the differences in the way education systems were organized (Gorur, 2015). It took an enormous global effort over several decades to develop the international standard classification of education (ISCED) and global education indicators that make such comparisons possible (Gorur, 2018). On the basis of these indicators and globally agreed norms for generating education data, such publications as the OECD’s Education at a Glance are now confidently able to present global comparative statistical data on education systems. And the publication of international comparative data on a regular basis as an aid to policy-making is not limited to education – the OECD also produces similar “at a glance” publications in a number of other fields, including Health at a Glance, Environment at a Glance, Entrepreneurship at a Glance, Government at a Glance, Society at a Glance and so on. Making the development of indicators possible were innovations in psychometrics and other measurement and survey technologies, as well as well-developed infrastructures for routinely collecting the required data. The Internet has added exponentially to the possibilities for data to be collected faster and in greater detail.

While these developments have led to an explosion in quantitative comparisons, and technical advances have produced more and more complex and sophisticated numbers, challenges and criticisms of such exercises remain. A serious practical challenge is that standardization and fidelity in data collection processes across different nations is difficult to ensure, especially with many more low-income nations, where the data-generating infrastructures are less well developed, coming into the orbit of such comparisons. And then there is the vexing issue of context. Statistical comparisons across diverse contexts, and policy lessons based on such comparisons, have been severely critiqued by qualitative researchers on the grounds that standardization ignores too much of the context that is relevant to understanding education systems. Regardless, such comparisons have gained enormous policy leverage and have become very influential in shaping policy globally.

Whereas international comparisons have become routine for colleagues who use statistical and quantitative methodologies, for sociologists using ethnographic or anthropological approaches, comparison presents some conundrums. Niewöhner and Scheffer (2010) point out that comparison was central to the development of the discipline of sociology – and they cite the modernist comparisons of Tocqueville, Martineau, Comte, Marx, Weber and Durkheim as cases in point. Indeed, they say that the volume of ethnographic and sociological work in comparison is ever on the increase. Despite this, they insist that ethnographers and anthropologists share an ambivalent relationship toward comparison, because, on the one hand, comparison is “criticised as mechanistic, technical, and naïve (vis-à-vis hegemonic concepts and categories)” but on the other hand, ethnographers have been able to play the role of “cultural translators” in the productive engagement with the high demand for cross-cultural comparisons. To address the “epistemic slippage” that is created by this ambivalence, they suggest that ethnographers should engage with “thick comparison” (after Geertz’s “thick description”) – a research practice where researchers not only perform comparison but also reflect on the act of producing objects of comparison. Thickness here is not only about describing field sites, but in exploring the “comparative enterprise itself” (p. 4).

Thus the challenges to comparison have been posed not only by an increasingly complex, globalizing world with new networks and new technologies, but by the epistemic complexities of comparison. Both quantitative and qualitative social scientists have responded to these challenges in a variety of ways. In the current moment, certain methodologies have gained traction – among them are randomized controlled trials, meta-analyses and large-scale standardized assessments, which privilege epistemological stances from the natural and statistical sciences; network methodologies, which draw from a relational ontology and spatial theories; and digital platforms, big data studies and digital methodologies, which are enabled by new and ever-changing technologies. More traditional methodologies such as case studies are also undergoing interesting adaptations, and these micro-sociologies are challenging macro-sociological and multi-sited approaches that characterized the field of comparative education. This volume brings together chapters that showcase some of these developments. At the same time, keeping the idea of “thick comparison” in mind, the authors have been asked to reflect on the methodologies they employ, focusing on their affordances and limitations.

Methodologies and their consequences

Theoretical and methodological commitments are often described using the metaphors of the “lens” or “toolkit.” But the implications of adopting these metaphors are seldom explored carefully. The “lens” metaphor presupposes an independent reality subject to observation, but which is distorted (or magnified, which is a form of distortion) in the act of observation. It also invokes things that may get left out of the field of vision. In this paradigm, then, different methodological “lenses” result in different partial glimpses of the truth. To get closer to the truth, researchers might employ multiple “lenses,” each with its own bias, each possibly counteracting the biases of the others. Researchers who subscribe to the “lens” metaphor of methodology might make a disclaimer at the outset to alert the reader/user of research to the partiality (in both senses of the word) of their methodologies, but in relation to a tacitly assumed totality beyond direct experience.

The toolkit metaphor could also signify “digging away” to get at the truth, an act that would require not just observing or studying through a “lens” but interacting in more consequential ways with the researched. But this sense of the toolkit shares similar presuppositions about an independent reality subject to discovery. The toolkit metaphor could also suggest that research is a process of construction rather than uncovering – a construction that requires a set of tools that can perform different functions within a complex field of research. This more pragmatic inflection would sidestep ontological questions in favor of questions concerning consequences. These metaphors clearly index a set of complex yet often unexamined philosophical issues that are nevertheless crucial to the way we construct our research design – and indeed to why and how we conduct our research.

Despite the highly consequential nature of methodological decisions, it is common to claim that we “choose” our tools according to the task at hand, based on the topics of research and the research questions under investigation. We are less likely to concede, in the “justification” parts of our publications and grant applications, that to someone with a hammer, everything looks like a nail. The methodologies that are part of our epistemic cultures (Knorr-Cetina, 1999) inevitably influence our areas of focus and determine the questions we deem to be of interest. Indeed, methodologies are so inherently constitutive of epistemic cultures that the determination of which methodologies to deploy may even be a foregone conclusion within our disciplines and paradigms.

Crucially, three points are to be noted with regard to methodology. The first critical feature, from a relational and constructivist perspect...

Table of contents

• Cover
• Title
• Copyright
• Contents
• List of figures
• List of tables
• List of contributors
• 1 Big data and even bigger consequences
• Part 1 Impacts: Randomized Controlled Trials: league leader in the hierarchy of evidence?
• Part 2 Patterns: In search of patterns: with enough data, do the numbers speak for themselves?
• Part 3 Relations: Global education policy in evolving network societies
• Part 4 Context: Contextualizing “context”
• World Yearbook of Education Series – full list of titles
• Index