The Evaluative Study of Action Research

Rigorous Findings on Process and Impact from Around the World

Eileen Piggot-Irvine, Lesley Ferkins, Wendy Rowe, Shankar Sankaran, Eileen Piggot-Irvine, Lesley Ferkins, Wendy Rowe, Shankar Sankaran

• Publisher: Routledge
• Year: 2021
• Print ISBN:
  9780367680947
• eBook ISBN:
  9781000336481
• Topic: Education
• Subtopic: Education General

The Evaluative Study of Action Research presents all eight published papers as part of the six-year, global, Evaluative Study of Action Research (ESAR) in one volume. The study sought to enhance the academic rigour of Action Research (AR) and provide greater evidence of its impact.

This research contained in this book shows, in a cohesive way, how the ESAR exemplifies original research incorporating new methodologies to create new knowledge. An Evaluative AR framework and indicators were created for initial qualitative data collection with six initial case studies using interviews, survey, documentary analysis, and Goal Attainment Scaling methods. The initial study was followed by a large-scale mixed method survey with 174 projects from across the globe. Almost all projects exhibited positive elements linked to AR precursors (focus clarification, stakeholder engagement, funding), processes (phased, planned yet flexible activity, data collection and analysis, ongoing collaboration and leadership), and outcomes/impacts (change, knowledge mobilisation and continuing action). The results of the ESAR, elaborated in this volume, offer important indications for how to create the sort of respectful engagement that is required for collective strength in solution based, innovative, change.

This book will be a valuable resource for: action researchers throughout the world; postgraduate research students, academics and libraries; evaluators; and anyone in communities who wishes to know how to create sustainable change.

1 Introduction

Eileen Piggot-Irvine, Lesley Ferkins, Wendy Rowe, and Shankar Sankaran

This book concludes the seven-year journey of the Evaluative Study of Action Research (ESAR). A summary of how we developed the research team and elements of the ESAR itself is provided in several of the chapters in this book. In this introductory chapter, we have used a series of questions to guide a summary of those elements. Such questions cover how the idea for the ESAR came about, who we are as a team, how we navigated our way of working together and how we developed our purpose, what the scope of the ESAR was, and what our overall journey in the research entailed. Finally, in this chapter we briefly overview each subsequent chapter: chapters covering journal publications.

How did the idea for the ESAR come about?

As early as 2008, Eileen Piggot-Irvine and Brendan Bartlett in an earlier book on evaluating action research (AR) (Piggot-Irvine and Bartlett 2008) had noted a comprehensive critique of AR could not prevail without clear evaluative data on the outcomes or impact. At that stage, there were a few publications looking at the results from several projects of a similar nature or focus, but no examples of evaluating and critiquing the principles and processes of AR across a diversity of context and scope.

In 2011, when Eileen was appointed to a position at Royal Roads University (RRU), Victoria, British Columbia, Canada, the space and timing was right to address the gap in evaluation and critique of AR. Consultation with AR colleagues supported the launch of an extensive evaluation study and we subsequently applied for, and won, a grant from the Canada-wide Social Science and Humanities Research Council of Canada, SSHRC (grant number: 611-2012-0274) to assemble an international team to establish such a study.

Who are the ESAR team?

In the summer of 2013, the initiating ESAR team met for two days on the campus of RRU. This location could not have been more conducive to a deep-thinking investigation of this nature – we were in the drawing room of a castle overlooking a lagoon and distant snow-capped mountains, nestled in a forest reserve.

Figure 1.1 The Evaluative Study of Action Research (ESAR) initiating team. From left: Phil Cady, Wendy Rowe (both Royal Roads University, Canada), Judith Kearney (Griffith University, Australia), Eileen Piggot-Irvine (RRU), Shankar Sankaran (University of Technology, Australia), Bernard Schissell (RRU), Lesley Ferkins (Auckland University of Technology, New Zealand). Missing from this photo is Deborah Zornes (RRU), who was at another meeting when the photo was taken.

Figure 1.1 shows this initiating team. Judith and Bernard were invaluable early advisers on design. Later members who contributed also need to be noted: Fiona Robinson (RRU) and Howard Youngs (New Zealand), who were considerable authors on the paper noted as Chapter 7. Maria Anderson (RRU) and Katie Dee (New Zealand) are deeply acknowledged for the data analysis and editing support.

How did we intend to work together, and was that realised?

The first thing the team members did at our initial meeting was to develop a set of protocols – a charter – for the way we would work together. We all agreed that we would strive to establish the ideals of a truly collaborative process that is lauded for AR generally. In sum, we would attempt to be a high trust AR team evaluating AR at a global level. Several of the chapters included in this book detail what such high trust involves because we were also evaluating that in the AR projects reviewed in the study. Essentially, in such collaboration, non-defensive, open dialogue is strived for. Openness, respect and subsequently trust are anticipated outcomes.

Did we achieve that collaborative intent? Chapter 3 explores this a little . . . with honesty. Each of us will also quickly comment as follows.

Eileen (ESAR lead): As a group of extremely busy professors it was not always easy to sustain such a continuing level of collaboration over seven years, but as lead of the core team (alphabetically) of Cady, Ferkins, Piggot-Irvine, Rowe, and Sankaran (and lesser extent Zornes), I have considerable pride in the way that we have survived the usual team ‘stage’ dynamics. I believe we have grown in our mutual relational respect and depth to become a highly effective team and that has been no mean feat for a research team over such an extended time frame. I defer to the rest of the team to provide probably a more accurate perspective, though.

Wendy (team member): Forming as a research team for such a large project that needed to operate across ideological and geographical barriers was not without some struggle and confusion in the beginning. Was the team to operate as a body of co-investigators or as a research team lead by a principle investigator? Grant requirements recognised only a principal investigator (who was the inspiration and initial organiser). With time and discussion, a collaborative team emerged with Eileen as the team lead. She set direction and pace for the study, with other members of the team contributing ideas, work task completion, constructive feedback, and comrad-ery as a cohesive group committed to the same goals. Trust and confidence in the team structure was established, which served us well as we progressed iteratively (in true AR manner) through different stages of the study.

Lesley (team member): In reflecting on my ESAR experience, my first thought is of the value of being able to encapsulate seven years of involvement. This extended period of time allows for such rich learning about how collaboration plays out during ‘thick and thin’. My second most vivid thought is how important it has been to have a ‘PI’ [principal investigator]. Eileen has indeed set the pace, maintained the momentum, facilitated the involvement, and fostered the collaboration among team members. Intriguingly, we explore such dynamics in Chapter 9, the leadership paper where we found a very similar phenomenon occurring across multiple AR projects. While leadership was collaborative, a single person leading was pivotal to the project. For me, our own ESAR project mirrors this very outcome. So, yes, I think we did achieve our collaborative intent because we had a single leader who knew how to facilitate sustained collaborative involvement, and because we had team members who knew how to be collaborative.

Shankar (team member): I joined the team slightly later but I could immediately see the energy in the team to do something critical to convince the world that AR was a reliable and effective methodology to conduct participative research to develop theory as well as advance practice. Having been accidentally introduced to AR during my PhD, as the university in which I enrolled for my doctorate insisted on using it as a methodology, despite growing up as a positivist due to my engineering education and experience, I soon fell in love with AR and its collaborative and reflective approach. It also helped with my transition to become an effective engineering manager as I now had to deal with people in addition to engineering techniques. I also had great respect for Eileen’s work due to my association with her through the Action Learning and Action Research Association in Australia (ALARA) and was enthusiastic about joining a team that she led. I quickly became part of the rest of the team and enjoyed our deep reflections in the majestic castle at Royal Roads and its serene surroundings. The discussions we had with a systemic approach towards AR also energised me to start an Action Research Special Integration Group at the International Society for the System Sciences (ISSS) that has grown rapidly. This prompted me to work on one of the papers in the ESAR network where I gathered action researchers of ALARA and systems thinkers at ISSS to develop a maturity profile for AR that contributes to a systemic appreciation of AR.

What were our goals at the initial stage, and did they change?

Following our initial team collaboration charter establishment at our ‘castle meeting’ in Victoria, we established our goals. Overall, in the ESAR, we aimed to address the earlier noted gap in evaluation and critique of AR by examining projects from diverse community, disciplinary, and organisational settings. Our overarching goal took some time to refine but essentially was:

Evaluative Study of Action Research (ESAR) is a mixed methodology study which seeks to address the under-exploration of AR outputs, outcomes, and impact via an evaluative study of multiple AR projects in several countries and varied contexts.

Put another way, we sought to answer the following question: in what ways can AR be validated as a contributor to meaningful individual, community, organisational, and societal change? The overarching goal and our research questions did not substantially change over the entire study.

Though already stated, a key point to remember is that the ESAR was not intended as an evaluation study of individual projects, but rather an evaluation of AR at a meta level drawn from multiple projects and perspectives from countries around the world. We gathered data from a wide variety of projects to explore, test and illustrate key principles of AR. From the exploration of precursors, processes and outcomes from a wide variety of projects, we intended to build: 1) a framework of AR indicators; 2) a discernment of key factors that contribute to process effectiveness, outputs and outcomes; and 3) a deeper understanding of AR change outcomes. We wanted to demonstrate the legitimacy, rigour, and relevancy of AR as both a research methodology and change strategy.

Following clarification of our goal, a detailed plan was established for developing the indicators we would use as a benchmark for evaluation, as well as the methodological and reporting approaches we would adopt. Again, detail of these elements is provided through the subsequent chapters.

What was the ESAR global range?

At our initial meeting and in the subsequent six months, we attempted an ambitious goal to locate and create a directory of multiple diverse AR projects from across the globe. In reality, although we scanned an extensive range of journals and located international publications and presentations and then contacted the authors, we found it difficult to get responses reflective of the entire globe. As any researcher has probably experienced, in these frantic communication times, getting response is far from easy. Within two years, however, we had responses from 195 action researchers and we developed an AR directory of their projects.

As noted in Chapter 7, our analysis of the project range indicated the highest representation from Australia (50%), Canada (25%), and the United Kingdom (10%). The AR projects took place in multiple sectors including public, private, government, community, healthcare, and not-for-profit, with the largest contingency taking place in with the education sector (60%). Generally, the primary target constituent groups for the AR projects included multiple people, such as teams (35.1%), organisations (27.1%), society/community (14.9%), or departments (11.7%). Individual projects represented 9% and larger, regional projects were only the focus of two projects (or 1.1%).

The question could be asked regarding whether the ESAR could be described as ‘global’, given that we did not achieve complete international spread. We are confident that we attempted to cover the world in which AR is conducted and we definitely attracted diverse projects, so we are content to describe our ‘reach’ as internationally diverse.

What happened in the ESAR journey?

In Chapter 4, we have outlined the research framework developed for our Evaluative AR study (the EvAR). The framework was established in the second year of the study and we substantially followed that EvAR as a guide to our data collection and associated events. Despite the fact that the framework is outlined later in this book, we think it is worth providing a little more background next to the events associated with our seven-year journey as a team.

We have already noted that we met first in the summer (July) of 2013 at the beautiful drawing room of the castle which is a key administration building of Royal Roads University. The choice of the site provided an essential backdrop for creative and collaborative thinking. Given that our intent was to replicate the collaborative principles of AR in our ESAR, the attention paid to that beginning was critical. As noted, we established our initial thinking on a charter for working together, our goals, and our aims and questions, as well as research indicators. We agreed on our intent to employ a mixed methodology design and to create an evaluative research framework to guide that. Such a framework had not been utilised in the evaluation arena to date. At our initial meeting, delegations were set for ‘fleshing out’ varied elements of the design.

In the following six months following the castle meeting, we fully utilised online collaboration to refine initial thinking about many elements of the ESAR. We deeply discussed then amended and updated our aim and research questions. We confirmed comprehensive evaluative indicators (as outlined in Chapter 2) which fell under headings of precursors, processes, outcomes, and impacts. Such indicators are commonly associated with evaluation research. We created the evaluation framework (EvAR) for the whole study (see Chapter 4). The EvAR is new to the AR field, but such a deeply collaborative, responsive AR approach has also not been utilised in the broader evaluation arena either. During this initial period, we employed a part-time graduate student (Maria Anderson) to help locate AR projects (195) for our international directory and to assist with later data analysis. We also created our data collection tools based on the indicators and gained ethics consent for the study.

In this early six months, we also determined that we would begin data collection with six case studies. The choice of cases was limited considerably by who responded to be involved at this depth (as opposed to just giving a name, etc., for a project to be listed in the directory). The six case studies were located in Australia, New Zealand, and Canada and were from highly diverse contexts linked to sport, disability, health, community development, and education. Data collection with those case studies was conducted in early 2015, with each core ESAR team member allocated a case. Chapter 8 provides the context and results for the case studies. We detail later why we did not publish those results earlier, given that it was an initial data collection phase, but in essence, the data from the cases was complex – more so because the six of us as team members each conducted one case. We also used the case studies as a way to pilot our tools for later data collection and, in our overload, we simply put the results on the back burner. A survey instrument (which we later called the ‘long survey’ based on feedback from our case participants about it being too long) was employed for the case studies. This sat alongside project lead and team member interviews with four cases, and a focus group with two cases. We also conducted comprehensive documentary analysis with material such as the case study meeting minutes and publications.

At the same time that the case studies were conducted, we continuously reflected on our own collaborative processes and our role as action researchers. This reflection is reported in Chapter 3 with our analysis of networks and collaboration, and in Chapter 5 in our discussion of a ‘maturity profile’ as a way of considering improvement to the ways action researchers manage their projects. In essence, we were ‘reflecting in action’ (Schön 1984) while we were deeply in the midst of conducting the ESAR.

In 2016, we decided to return to our case study participants subsequent to our initial data collection with them b...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Contents
• Chapter credits
• 1 Introduction
• 2 Conceptualising indicators for evaluating action research
• 3 Action research networks: role and purpose in the evaluation of research outcomes and impacts
• 4 Developing a framework for research evaluation in complex contexts such as action research
• 5 Developmental progress in conducting action research
• 6 Goal Attainment Scaling in action research: enhancing a systems thinking orientation
• 7 Struggling to achieve desired results from your AR projects? insights from the Evaluative Study of Action Research may help
• 8 Thumbs up for action research in case studies from the Evaluative Study of Action Research
• 9 Leadership within action research: surfacing the collective nature of leadership
• 10 Conclusion and reflections
• Index