![]()
Part 1:
Introduction
![]()
ONE
Introduction: themes of the book
The policy evidence cycle
A cosy idea for many of those working in an area of public policy such as education is that policy-making is evidence-informed. Policy-makers and their advisers come up with ideas for changes in policy, which may or may not be based on solid research evidence. Some of these ideas are implemented and can be evaluated in terms of their policy objectives. Policy-makers and their advisers then react to this newer evidence, and the improving cycle of policy continues.
In reality, of course, the cycle is nothing like this. In education, new policies and interventions are rarely based on good prior evidence of effectiveness and of their side effects. Many policy areas are evidence-resistant, and examples of these are discussed in this book. Evidence-resistant here means that the policies are proposed and implemented, even though the clear weight of evidence is against them. Of the rest, too many policies are still not evaluated robustly at all. This means we can have no good idea whether they work as intended and whether they have damaging side effects. Of the few that are robustly evaluated, as far as this is possible with live policy issues, many are then found to have been ineffective or even harmful. But their ineffectiveness does not lead to them being improved or cancelled. The policy cycle does not seem to permit policy-makers to backtrack like this. Rather, policies are seemingly just left to wither or until they are formally reversed by a change of government. Again, examples are described in the book. This leads to at least three kinds of damage to society.
First and most obviously there is the cost. Large amounts of public and other money are spent around the world on educational initiatives that have no basis in evidence and little chance of working, and are continued overlong once their ineffectiveness has been revealed. Second, there is the possibility of harm from untested interventions for those learners in every generation who have only their one chance to get it right. There is always a kind of opportunity cost, given that every unwarranted policy uses time, effort and resources that could have been used for a genuine improvement. And finally, this chaotic approach inhibits the search for good policy – of a kind that would justify the costs and lead to real improvements in the future.
The problems uncovered in this book include exaggerating supposed 'crises' such as high failure rates at schools, the increasing student sex gap and cultures of laddishness, the unfairness of university admissions processes, poor social mobility and problems in the supply of teachers and scientists in particular. There are also equally exaggerated accounts of the success and power of new types of schools such as academies, older types such as grammar schools, approaches such as learning styles or enhanced feedback, improved leadership, the use of technology and setting targets in education. Because these areas are so commonly misunderstood, even if a good policy were available it would probably be deployed to address the wrong problem (or wrong interpretation of the problem). This misdiagnosis or failure to specify more carefully the precise issue or pattern to be addressed by policy is one of the biggest causes of money and opportunities being wasted, endangering the progress of learners. The new political arithmetic approach (see Gorard, with Taylor 2004) utilised in this book shows a way to help overcome this.
Of course, evidence cannot and should not determine education policy. It should simply help to inform policy-making, so that where a policy-maker has a clear objective, the best evidence can help to achieve that objective safely and efficiently, but the evidence itself should be largely neutral about the objective. For example, the most secure evidence might suggest that a particular education policy would increase the average attainment of students but also increase the average difference between high and low attainers. So how this evidence should be used depends on the policy objectives. If raising average attainment is paramount, the policy could be deemed a success. But if making the system fairer is a priority, the policy may be a failure (although, as this book shows throughout, raising attainment overall and reducing attainment gaps are not necessarily in tension).
Evidence must be 'multiplied' by the values of any policy objectives to reach an appropriate decision. The values themselves are not evidence-based in the same way, and it is possible to argue against a policy both because it has what appear to be incorrect values or, accepting its objective, because it will not work in the way it has been implemented. This book is mostly about the latter issues.
Although my work has been concerned with improving the quality and provision of education of all kinds, where this leads to a tension with fairness (as in the hypothetical example above) I have tended to favour concern for fairness or equity over average outcomes. Partly this depends on a consideration of what education is for. Education prepares people for the world of work (see Chapters 11 and 12), or for more education (Chapter 10). It socialises them into society or gives them skills for life (Chapter 9). It provides them with a general knowledge about the world. And it keeps young children secure and engaged while their parents work. However, one of the main reasons that initial education is free, compulsory, universal and state-regulated in most countries is so that what, and how much, children learn is not just determined by their family circumstances. Education up to a certain level is one of the guaranteed rights for all children and young people in a civilised society.
These issues are revisited in Chapter 13 after consideration of the kind of evidence available in a wide range of education policy areas.
Structure of the book
The book is in three main parts. Following this introductory chapter, Chapter 2 illustrates what I have learned about the conduct of education research, some methodological innovations that I have proposed and used, and an outline of the methods used in the substantive Chapters 3 to 12. Chapter 3 then sets out the patterns of attainment and participation in education from early to later life – patterns based on geography, era, income, family background and personal characteristics such as age or ethnic origin. Chapters 4 to 12 examine possible explanations for, and solutions to, these patterns, starting with who one goes to school with and ending with the kind of occupation one is involved in. These chapters generally move towards consideration of older learners and later phases of formal education, and also towards more non-cognitive outcomes of education and learning beyond institutions such as schools and universities. Chapter 13 summarises what has been learned over 20 years, and what this might mean for researchers, research funders, policy-makers and educational systems worldwide.
![]()
TWO
The nature of the evidence assembled
This chapter is in two parts. The first describes my approach to the conduct of research, and some of the innovations I have adopted or invented over 20 years. The second describes the kinds of designs, data and methods of analysis used in the substantive Chapters 3 to 12. There is no space for full details of the methods of each study, but all are referenced for readers to follow as they wish. If readers find this chapter hard to start with, it can be skipped and picked up again at the end.
My approach to research
One strand of my research work over 20 years, since 1997 (Gorard, 1997a), has been writing about the conduct of research itself, based on my own experiences, observation of the work of others and reading copious research reports for reviews of evidence. Part of this writing has been about the role of funders and research organisations (Gorard, 2002a, 2004a; Gorard and Cook, 2007). Much has been capacity-building and development work intended to benefit new researchers (Gorard, 2017a, b), including on combining methods (Gorard, 2002b; Gorard, with Taylor 2004), perhaps through design experiments (Sloane and Gorard, 2003; Gorard et al, 2004a). In retrospect, it is clear that much of the writing has been about simplifying the process of research for others, including showing that supposed divisions, such as the widespread 'qualitative':'quantitative' schism, are unfounded in any way, including philosophically (Gorard, 2002b, 2004b; Gorard and Smith, 2006; Symonds and Gorard, 2010). In the end, there is only research (Gorard and Siddiqui, 2018a).
Using large-scale data at an aggregated level tends to emphasise the role of structure and even predictability in people's lifelong education trajectories. The same factors rarely appear in individuals' own in-depth accounts of their lives, which tend to emphasise choice, the role of others and serendipity. Using only one of these forms of data in research would be likely to impoverish and even bias the findings, and any practical conclusions drawn from them. Therefore, both have a role in studying any research area. I apply this simple holistic idea to teaching, research capacity-building, knowledge transfer and all of my research. It makes the work simpler, the communication of findings easier to wide audiences and reviews of evidence more secure and less biased.
The current push in the UK and elsewhere towards more 'quantitative' work and more complex methods of analysis is therefore misguided (Gorard, 2003a, 2007a; Gorard et al, 2004b). There is a special problem with the abuse of inferential statistics (Gorard, 2010a, 2014a, 2015a, 2017c; White and Gorard, 2017). Working with numbers, as a matter of course, as part of any research study does not have to involve significance tests and related concepts, and it can and should be simple (Gorard, 2001, 2006a, b, 2010b, 2017d). At present, we need greater emphasis on initial research design, independent of subsequent methods of data collection, where the design permits a simple and intuitive approach to analysis (Gorard, 2010c, 2013a).
The research process, as revealed by conducting the projects described in this book, seems to me to be both more sophisticated and less inhibiting than is commonly portrayed. There is a clear spiral of research within programmes, fields or topics of study, moving from consideration of what we already know (as in Phase 1 in Figure 2.1) to developing ideas and artefacts, then to providing robust evidence, and eventually monitoring the results as they are used in policy (Phase 7).
Different kinds of research questions would be suitably addressed in successive phases, from the descriptive such as 'What do we know about this issue?' in Phase 1 to the causal 'Does this approach work to improve this issue?' in Phase 6. Each question in each phase requires different research designs (Gorard, 2013a). A comparative design might be suitable for descriptive work, a longitudinal approach could be used to identify risk factors for an undesirable educational outcome, and a quasi-experimental approach to try and modify those risk factors. None of these designs is related to specific methods of data collection or analysis. A longitudinal study is longitudinal whether it involves collecting observations, survey ...
