To relieve the discussion and understanding of the studies presented in this special issue, and to clarify their connections, similarities and commonalities, a methodological framework of impact evaluation of QM in HEIs is drawn in some detail. The framework consists of key guidance issues for impact evaluation and the characteristics of the before-after comparison approach. In addition, the so-called attribution problem is contextualized and the types of available data are described.
Key guidance issues for impact evaluation
A few years ago, Frans Leeuw and Jos Vaessen advised evaluation theorists as well as practitioners with nine methodological and managing key guidance issues for conceptualizing, designing, and implementing impact evaluations:
(1) âIdentify the type and scope of the intervention.â
(2) âAgree on what is valuated.â
(3) âCarefully articulate the theories linking interventions to outcomes.â
(4) âAddress the attribution problem.â
(5) âUse a mixed-method approach.â
(6) âBuild on existing knowledge relevant to the impact of interventions.â
(7) âDetermine if an impact evaluation is feasible and worth the cost.â
(8) âStart collecting data early.â
(9) âFront-end planning is important.â (Leeuw and Vaessen 2009, x).
In the five empirical case studies presented in this special issue, the above key guidance issues could be used as a guideline.
Issue (1) was fulfilled because the QA interventions were well-established QA procedures, such as programme accreditation, institutional accreditation and programme evaluation. Issue (2) was observed in the sense that it was clarified what the intended QA effects were. Issue (6) was also realized as much as possible, since the available knowledge about the QA procedures and their potential efficacy as well as knowledge about other potentially efficacious activities and interventions was used. Furthermore, issue (8) was met in the sense that data were timely collected in all case studies; particularly in the before-after comparison approaches, the time schedules of the baseline, midline and endline surveys were adjusted to the schedules of the QA procedures (Table 2).
Issue (9) was fulfilled since the empirical case studies followed ex-ante plans comprising internal meetings, public conferences, publications and workshops based on work plans with mile stones, division of work tasks and responsibilities. Independence, or impartiality, of the evaluating teams vis-Ă -vis the stakeholders with whom they were collaborating was also a sub-theme of issue (9): in the projects, a dialogue-based critical friends approach was utilized, i.e. the impact research groups were transparently discussing and deciding all methodological and managerial aspects of their projects independently, i.e. without interference from their home institutions, formal superiors, and national or regional governments. Furthermore, concerning independence it seems appropriate to state that the impact analyses presented in this special issue implemented participatory approaches because stakeholders â namely students from participating programmes, teaching staff, leadership and QA staff from participating universities, and QA agenciesâ staff â were involved in the determination of evaluation objectives and relevant indicators as well as in data analysis and publishing of results (Bejan et al. 2015;2018;Damian, Grifoll, and Rigbers 2015;ICP 2016;Jurvelin, Kajaste, and Malinen 2018;Kajaste, Prades, and Scheuthle 2015;Leiber, Moutafidou, and Welker 2018;Leiber, Prades, and Ălvarez 2018;Leiber, Stensaker, and Harvey 2015).
The attribution problem, issue (4), was also addressed, see analysis below. Issue (5), i.e. mixed methods in a strict sense, however, could only be applied in one of the five empirical case studies (Seyfried and Pohlenz 2018), while it was not applicable in the other four case studies, simply because quantitative data were not available, see discussion below.
The importance and meaning of issue (3) resides in the fact that it is an ingredient of plausible explanations for the changes observed, ideally identifying causal mechanisms connecting interventions with the consequences generated by the interventions. Therefore, it is assumed that the intervention design is based on a âtheoryâ. The required articulation of such âtheoriesâ, particularly when they are implicit or rather weakly developed,
Can use one or more pieces of evidence â ranging from the interventionâs existing logical framework, to insights and expectations of [âŠ] [well-informed] stakeholders [and experts] on the expected way target groups are affected, to theoretical and empirical research on processes of change or past experiences of similar interventions. (Leeuw and Vaessen 2009, xii)
Such assumptions on the effects of an intervention have to be tested, either âby carefully constructing the causal âstoryâ about the way the intervention has produced results (as by using âcausal contribution analysisâ [and process tracing]) or by formally testing the causal assumptions using appropriate methodsâ (Leeuw and Vaessen 2009, xii; see also Befani and Mayne 2014). In the four case studies based on before-after comparison (Bejan et al. 2018; Jurvelin, Kajaste, and Malinen 2018; Leiber, Moutafidou, and Welker 2018; Leiber, Prades, and Ălvarez 2018), issue (3) was fulfilled theoretically (see Leiber, Stensaker, and Harvey 2015, Table 1, Figure 2, Figure 3), while more detailed empirical hypotheses would be desirable but were not developed because of lack of resources. Due to a rather specific QA procedure, one of these case studies, however, achieved a little more in this regard by developing some structural mechanism hypotheses based on assumptions about intended aims of the QA procedure (Leiber, Moutafidou, and Welker 2018). The fifth case study approaches issue (3) by an ordinary least squares regression model to explain perceived QA effectiveness through structural variables and certain QA-related activities of quality managers (Seyfried and Pohlenz 201...
