Quality Assurance in Higher Education

Contemporary Debates

Kenneth A. Loparo, A. Amaral, Kenneth A. Loparo, A. Amaral, Maria João Rosa

• Publisher: Palgrave Macmillan
• Year: 2014
• Print ISBN: 9781137374622
• eBook ISBN: 9781137374639
• Topic: Education
• Subtopic: Business General

This book weighs up the consequences of introducing Quality Enhancement and Risk Management as new dimensions in Higher Education quality control on a global scale. The authors include Chief Executive Officers of Quality Agencies, policy analysts and leading scholars in Quality Evaluation and Comparative Higher Education policy analysis.

1

Introduction

Maria João Rosa and Alberto Amaral

The main objective of this collection of studies is to open up a critical debate on recent changes and trends in quality assessment systems that will be useful both for those responsible for quality agencies and those who periodically come under the scrutiny of quality agencies.

This book presents a critical analysis of contemporary developments in the quality assurance of higher education, their advantages, their benefits and their possible unintended consequences. Special emphasis is given to new instrumentalities such as the U-Map and U-Multirank transparency tools, an initiative supported by the European Commission and the Ministers of Education; the AHELO (Assessment of Higher Education Learning Outcomes) project led by the OECD (Organisation for Economic Co-operation and Development) for the measurement of learning outcomes; the risk management theoretical framework and the current state of its implementation in England; and the quality enhancement approach. By doing so we have highlighted the emerging and most recent trends in quality assurance, providing as well an opportunity to compare trends in Europe with those in the US and Latin America while weighing the views and accounts of different actors and interests: academics, students and quality agencies.

Discussing and analysing contemporary debates on this topic poses an obvious problem of up-to-dateness once the book finally gets out to its readers. The field of quality assurance has indeed been quite active since the emergence of the ‘Evaluative State’ in the late 1980s (Neave, 1988). From that moment on, the development of quality assurance in Europe was fast, as Schwarz and Westerheijden (2004) have reported: while at the beginning of the 1990s only about 50 per cent of European countries had initiated quality assessment activities, by 2003 all countries except Greece had entered some form of supra-institutional assessment. Furthermore, the Bologna Declaration (1999) and the emphasis it gave to the need for developing comparable criteria and methodologies for assuring quality in higher education also contributed to new developments in the field, the more prominent probably being the establishment of the European Association for Quality Assurance in Higher Education (ENQA); the adoption, at the 2005 Bergen Ministerial Conference, by the European Ministers Responsible for Higher Education of the European Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG) and the establishment of the European Quality Assurance Register for Higher Education (EQAR) endorsed by the Ministers at the 2007 London Conference.

In the aftermath of these developments new ones have come to the fore. These are analysed and discussed in this book with the aim of building some common understanding about what the future of this area may well be. The authors of the book examine the views of the main actors in the quality nexus: national agency executives, policy analysts, and academics and students currently and actively engaged in exploring and laying down the future tasks to be taken up by quality agencies as well as those in higher education institutions responsible for developing quality procedures. As Neave rightly states in his chapter in this book, we intend to step aside from ‘policy as action’ and instead provide an opportunity for the reader to spend some time examining what the main construct for operationalising and developing quality, efficiency and enterprise (Neave, 1988) – namely, the Evaluative State – has achieved, weighing up the moving frontiers of quality assurance in higher education.

Despite the fact that the recent developments in the area of quality presented in this book seem to pull quality assurance into different directions, some common elements have emerged from the different perspectives outlined in individual chapters. In the final chapter we review the main aspects of the contributions of the different authors in order to collect and further develop those emergent common elements. We call the reader’s attention to some of the most interesting ideas expressed during the 2012 A3ES and Cipes Conference held in Porto, while at the same time analysing some challenging questions that constitute interesting issues for the future development of quality assurance in higher education.

Brief summary of the chapters

This volume comprises four parts. The first two chapters (by Alberto Amaral and Guy Neave) provide a broad panorama of recent developments of quality assurance and of the emergence of the Evaluative State. The second part of the book (chapters by Don Westerheijden, Alberto Amaral and Diana Dias, Anthony McClaran, Colin Raban and Murray Saunders) focuses on new challenges and instrumentalities being developed – U-Map and U-Multirank, the AHELO project, risk management and quality enhancement. The third part (chapters by Bjorn Stensaker, Judith Eaton and Maria-José Lemaitre) analyses recent developments in three different regional settings (Europe, the US and Latin America). The fourth part (chapters by Maria João Rosa, Liliya Ivanova and Achim Hopbach) presents a view of quality assurance based on the perspectives of three different constituencies (academics, students and quality assurance agencies). A final chapter presents the main findings and conclusions.

In Part I, Alberto Amaral’s chapter, puts forward the framework on which the remaining chapters are based. The chapter describes the rise of quality concerns in higher education and the emergence of quality as a public issue. It then discusses the consequences of the loss of trust between higher education institutions and the State and society, which played an important role in determining the major characteristics of current quality assessment systems, namely the movement towards accreditation schemes. The increasing use of market-like mechanisms as instruments of public policy is also referred as a trend giving legitimacy to State intervention – under the guise of consumer protection – through the introduction of an increasing number of compliance mechanisms, including performance indicators and measures of academic quality, which are transforming quality assurance into a compliance tool. Finally the chapter introduces developments – U-Map and U-Multirank, AHELO, Risk Management and Quality Enhancement – that will be dealt with in the second part of the book, establishing the grounds for the debates currently taking place around quality assurance in higher education.

Guy Neave’s chapter discusses more thoroughly what quality enhancement and the advent of risk analysis may both bring as new and significant additions to the instrumentality of the Evaluative State. Using the Evaluative State as the basic analytical framework opens an alternative interpretation to the usual technical and metrical perspective that tends to predominate in the main literature of quality assurance and accreditation. By setting out the development of the Evaluative State in Europe in four broad chronological stages – origins, scope and purpose; refining procedures, re-defining ownership; quality enhancement: an evolutionary dimension; and higher education as a risky business – the author interprets both quality enhancement and risk factoring as successive stages that place new interpretations and open the way to new insights on the Evaluative State. However, national systems display considerable variations in timing, rationale and in the purpose they assign to quality assurance mechanisms (in this respect particular attention is paid to developments in UK, France and Portugal, which present significant variations). Finally the author briefly explores possible implications that may arise from moving forward on quality enhancement and risk management.

The second part of the book discusses the new challenges faced by quality assurance by introducing the new instrumentalities that are emerging to deal with them. Don Westerheijden’s chapter discusses the new ‘instruments of transparency’, U-Map and U-Multirank, currently under development in Europe as multiple tools for different users, packaged within single ‘engines’. To understand their difference from current rankings, the chapter investigates the basic concept of diversity of higher education before reflecting on some basics of process analysis. These conceptual considerations show that the activities and performance of higher education institutions are multidimensional, and that current rankings favour largely a single dimension. In effect, they do not take account of the ensuing institutional horizontal diversity. Finally, U-Map and U-Multirank are critically analysed and their contribution to a more inclusive conceptualisation of quality of higher education, is discussed.

The chapter by Alberto Amaral and Diana Dias describes this OECD-implemented project and discusses the results of its feasibility study, very recently disclosed. Douglas Bennett (2001) argues that a feasible strategy for measuring the quality of HE will consist of assessing outcomes, by evaluating the skills and capabilities students have acquired as they graduate (or shortly after) or the recognition they gain in further competition. This is the road the OECD has apparently been trying to follow and that is discussed by Amaral and Dias. For the OECD, the ‘Assessment of Higher Education Learning Outcomes (AHELO) is a ground-breaking initiative to assess learning outcomes on an international scale by creating measures that would be valid for all cultures and languages’ (OECD, 2009). OECD has launched a ‘feasibility study’ of AHELO that includes measuring learning outcomes in terms of generic skills and discipline-related skills (in engineering and economics). The final report of this exercise was recently published and a public debate of its conclusions took place in March 2013. In Amaral and Dias’s chapter the methodology of the feasibility study and its main conclusions are presented and critically analysed.

The next two chapters, by Anthony McClaran and Colin Raban, analyse emerging developments related to the use of risk management in quality assurance processes. Risk management is a technique imported from business. It identifies, assesses and assigns priorities to risks the better to minimise, if not to eliminate, the impact of untoward, threatening or negative factors in institutional development. The chapter by McClaran describes and analyses the implementation in England of a more risk-based approach to quality assurance, following the issue of the 2011 White Paper on Higher Education, Students at the Heart of the System. This government document was followed by another, the technical consultation, A Fit-for-Purpose Regulatory Framework for the Higher Education Sector, which examined what changes in procedures, powers and duties were required to implement the reforms proposed in the earlier White Paper, including risk-based quality assurance. The government response indicated a risk-based approach as the most desirable means of regulating higher education in England. Following this, the steps to implementation moved forward. The chapter analyses the implementation process and its timetable in detail, presenting the aspiration that these changes should provide a clearer understanding of the changing landscape of higher education provision in England and the wider UK, safeguarding its reputation and quality for the future. According to the new approach the risk each institution faces ought compulsorily to be externally assessed and the level of risk detected should determine the frequency of reviews by the Quality Assurance Agency: ‘low risk’ institutions face full institutional reviews less frequently than either new providers or institutions whose provision is deemed of lower quality.

How far it is feasible – and desirable – to combine a risk-based quality assurance system with a genuine quality enhancement approach, remains debatable. This is an issue addressed from different perspectives by many of the contributors to this book. Raban discusses the theme thoroughly in his chapter, arguing that in discharging their responsibilities for assuring the quality and standards of their provision, universities should employ the ideas of risk and risk management in ways that are very different from the proposals set out in the White Paper and in recent publications from the HEFCE (Higher Education Funding Council for England) and QAA (Quality Assurance Agency). A risk management approach, possibly any form of internal quality assurance, will not work unless there is a culture that accepts risk, encourages staff to disclose risks and is open to the frank exchange of information and ideas on the management of risk. It would be difficult to sustain such a culture if staff were to believe that frank disclosure would leave their institutions exposed in the face of external scrutiny. In this respect, recent developments in external review methods do not augur well.

The Scottish approach stands apart from its counterpart in England. In this case the predominant view in higher education is to associate risk management with quality enhancement on the grounds that enhancement is the outcome of change and innovation. Both change and innovation, however, involve elements of risk. Accordingly, in Scotland, the individual institution is expected to manage risk and thus provide reasonable safeguards against it in the interest of students. In his chapter Murray Saunders provides an overview of the Scottish experience with quality enhancement, based on an analysis of the policy intervention that set it up, which was done using an evaluative research approach. The Quality Enhancement Framework had its inception in 2003 and has been coordinated by the Scottish Funding Council with the participation of all Scottish universities. It aims to improve university learning, teaching and assessment, establishing an integrated approach that emphasises ‘enhancement’ rather than ‘assurance’. According to the author, the approach was understood as a positive departure from assurance-based engagement between the Scottish universities and the UK-based national framework. It is also considered to be owned by the higher education community, or at least by senior academic managers. Nevertheless the change process it entails was complex, involving several areas of tension and implementation challenges.

With the goal of offering the reader an opportunity to compare trends in Europe with those in the US and Latin America, Part III of the book provides a setting for the most recent developments taking place in these three regions of the world. Starting with Europe, the main argument of Bjorn Stensaker’s chapter is that European external quality assurance (EQA) is developing in a complex way, and that we may currently be at a crossroads concerning how this activity is to be organised in the future. Due to developments taking place at the national level, different scenarios can be developed as to the future of EQA. While some of the scenarios may be interpreted as quite gloomy – at least from an agency point of view – the situation should not be interpreted in a deterministic way. There are many possibilities for developing EQA beyond the trends and scenarios laid out in the chapter. However, alternative routes require joint actions in which national authorities, agencies and higher education institutions all have a role to play.

At the national level, authorities need to develop a more nuanced view on the use and purpose of EQA. The European Standards and Guidelines should not be seen as a hindrance for national policy-making, although one can suspect that this is the case in some countries. While standardisation indeed has brought European EQA forward in many respects, there is currently a need for some innovation to take EQA to the next level.

In the next chapter, Judith Eaton examines three dimensions of US accreditation: what it has been, recent trends affecting its operation and what its future is likely to be. She concludes by contrasting this likely future with a more desirable scenario. She urges both higher education and accreditation communities to work together in designing a future of the desirable rather than the likely. The probable future for accreditation, if current trends are not modified, is sobering. Accreditation is less likely to be the dominant method of judging academic quality, and rather just one among a number of voices judging quality – voices devoid of academic expertise and experience. Increasingly, accreditation will be controlled by government, with implications for the independence of not only accreditation, but also colleges and universities. Of paramount significance, the core values accompanying accreditation may themselves risk being reduced or transformed. This likely future need not prevail, however. If the accreditation and higher education communities work together, they may hold government influence and calls for accountability in balance by sustaining institutional and faculty academic leadership for accountability as framed by the academy. They can further engage innovation so that creative change in higher education is accompanied by a commitment to a vision of intellectual development of all students.

Finally María-José Lemaître introduces the case of Latin America. Quality assurance schemes have been in place in this region for two decades, and have developed around different models to respond to the needs of national higher education systems. The overall view of universities in those countries with longer experience and more consolidated quality assurance processes is that they have been effective, and have contributed significantly to the recognition and improvement of increasingly complex and diversified higher education systems. Yet it is clear that the growth and development of higher education, increases in enrolment, and institutional differentiation pose challenges that higher education must address and take into account in its revision of quality assurance processes. The study briefly reported in this chapter points to significant lessons, which can contribute to improved policy making at the national level; to changes in higher education institutions, both in terms of new managerial arrangements and in teaching and learning practices; and, most of all, to the need for updated and revised standards, procedures and practices of quality assurance agencies. Higher education is a dynamic system – it is not served well by quality assurance processes that are not prepared to learn (and to un-learn), to adapt and adjust to the changing needs of students, institutions and society.

The last part of the book offers different actors’ perspectives on recent trends, presenting the views of the academic and student’s constituencies, as well as the views of the agencies. The chapter by Maria João Rosa explores academics’ perspectives on higher education quality assessment based on the answers given by a sample of Portuguese academics to an online questionnaire designed to investigate their degree of support towards quality assessment goals and purposes. Overall the analysis performed reveals that Portuguese academics tend to support the majority of goals and purposes of quality assessment, although more support is given to quality assessment mechanisms that privilege improvement over control. Portuguese academics indeed seem to prefer a formative type of quality assessment that promotes self-reflection and knowledge, and the continuous improvement of teaching and learning. Additionally the results show that academics’ perspectives are not homogeneous, and depend to a certain extent on their own characteristics, such as gender, type of institution they belong to or scientific affiliation.

The results presented are especially relevant for those working in both higher education institutions and governmental agencies, since they may contribute to the design of quality assurance systems that academics are more likely to support and are therefore more likely to be successful; that is, more likely to contribute effectively to improvements in the quality of higher education institutions and systems.

Liliya Ivanova’s chapter offers the views of students. The Bologna Process holds students to be competent and constructive partners in the joint shaping of the higher education experience. The European Students’ Union is an active advocate of student participation in QA processes, and provides expertise in QA. In recent years, QA ...

Table of contents

• Cover
• Title
• Copyright
• Contents
• List of Figures and Tables
• Notes on Contributors
• 1 Introduction
• Part I The Frontier and Its Shifts
• Part II New Challenges, New Instrumentalities
• Part III Regional Setting
• Part IV Quality Assurance: The Actors’ Perspectives on Recent Trends
• Part V Conclusion
• Index