Evaluating e-Learning

Guiding Research and Practice

Rob Phillips, Carmel McNaught, Gregor Kennedy

• Publisher: Routledge
• Year: 2012
• Print ISBN: 9780415881937
• eBook ISBN: 9781136699511
• Edition: 1
• Topic: Education
• Subtopic: Education General

How can novice e-learning researchers and postgraduate learners develop rigorous plans to study the effectiveness of technology-enhanced learning environments? How can practitioners gather and portray evidence of the impact of e-learning? How can the average educator who teaches online, without experience in evaluating emerging technologies, build on what is successful and modify what is not?

By unpacking the e-learning lifecycle and focusing on learning, not technology, Evaluating e-Learning attempts to resolve some of the complexity inherent in evaluating the effectiveness of e-learning. The book presents practical advice in the form of an evaluation framework and a scaffolded approach to an e-learning research study, using divide-and-conquer techniques to reduce complexity in both design and delivery. It adapts and builds on familiar research methodology to offer a robust and accessible approach that can ensure effective evaluation of a wide range of innovative initiatives, including those covered in other books in the Connecting with e-Learning series.

Readers will find this jargon-free guide is a must-have resource that provides the proper tools for evaluating e-learning practices with ease.

I Setting the Scene

1 e-Learning, Learning and Evaluation

DOI: 10.4324/9780203813362-1

1.1 Introduction

In this chapter we will provide a brief history of e-learning, emphasizing the notion that using technology in an educational setting involves conscious design decisions. These design decisions must involve considerations of how learners learn, and so we will establish some key characteristics of learning which, we believe, should be used to guide the design of technology-enhanced learning (or ‘e-learning’) environments. We will explore the shorthand term ‘e-learning’ that will be used throughout the book by introducing the concept of an e-learning life cycle. We will firmly establish our position, which is that the focus needs to be on learning, not technology, and that there are a number of ways in which a range of technologies can assist learners to learn.

Questions such as the following naturally arise when one designs any learning environment for learners: “How do we know that the overall learning environment or context – with its technology components, whatever they may be – supports learning? Is this the best way to organize the learning environment?” These are evaluation questions and the central thrust of the book is to provide pragmatic advice to teachers about how to answer such questions. Good evaluation is systematic and scholarly; it involves data collection and careful reflection on the implications of the data.

Overall, this book attempts to resolve some of the complexity of evaluation of the effectiveness of e-learning, by unpacking the e-learning life cycle and the types of evaluation and research that are appropriate at any given part of the life cycle. It also presents practical advice in the form of an evaluation framework and a scaffolded approach to the design of an e-learning research study.

1.2 What is e-Learning?

Computers have been used for education and training since the 1960s, and increasingly since the 1990s. Large and increasing amounts of money have been spent worldwide on the development of computer applications designed to assist people to learn. In the 1990s, learning software tended to be authored as multimedia-rich, monolithic applications, addressing learning needs across a period of time, often several weeks. Among many terms, such ‘courseware’ was called ‘interactive multimedia’, ‘computer-based learning’ (Phillips, 1997), ‘computer-facilitated learning’ (McNaught, Phillips, Rossiter & Winn, 2000) and ‘interactive learning systems’ (Reeves & Hedberg, 2003). The authoring software used at that time made it expensive to adapt material to other purposes and to update it for changed information.

With the advent of the World Wide Web in the mid-1990s, it became possible to develop courseware (hyperlinked web pages) which was much more adaptable, and less expensive to create, but which lacked the educational and multimedia richness of the earlier monolithic applications, primarily because of difficulties in delivering large media files over the Internet, which at that stage had very limited bandwidth. Shortly thereafter, the first learning- management systems appeared on the market. These applications delivered web pages to learners and provided online interactive tools that allowed learners to interact with course materials, other learners and their teachers, but primarily only using text.

Bandwidth increases in the early 2000s enabled effective interaction and multimedia capabilities to be provided over the web. At the same time, the ‘learning object’ movement started to gain momentum, driven largely by the American military's requirement to provide reusable and adaptable training content for its staff (Advanced Distributed Learning, 2003), independent of the computer system being used.

A large number of terms has been used to describe the range of these computer applications over the last two decades. A current favourite is technology- enhanced learning; however, while enhanced learning is the intention, it is not always the practice and, in a book about evaluation, we have chosen to use the more neutral term e-learning. While we will argue in Chapter 3 that there are semantic nuances surrounding the term e-learning which make it problematic, we will continue to use the term in this book, because it is a useful shorthand and widely accepted. We choose to use, as one among many, the definition of Littlejohn and Pegler (2007, p. 15): “the process of learning and teaching with computers and other associated technologies, particularly through use of the Internet”.

For several years, proponents of e-learning have used the term in a ‘one size fits all’ fashion, despite its use in a range of contexts. We agree with Friesen (2009, p. 4) that e-learning has “come to represent a useful shorthand for a range of different orientations to ... the use of technologies in education and learning. Often the precise way that the term e-learning is used is dependent on the author's particular purposes or specific research agenda”. Because the term is often used without explicating the underlying assumptions of the author, understanding e-learning is hampered, leading to confusion on the part of practitioners and policy-makers.

e-Learning is primarily a branch of the discipline of education. However, it also brings in influences from other fields, including computer engineering, information technology, design and media studies. “It is inter disciplinary in that it seeks to combine and explore the inter connections between new and different approaches from different fields and specializations; it is multi disciplinary in that it simultaneously tries to respect the multiplicity of differences that can separate one research approach from another” (Friesen, 2009, p. 12). Its multidisciplinary nature and its rapid evolution have led to individual researchers taking different approaches to evaluation and research, derived from their individual contexts and often with little reflection on the appropriateness of that approach to the task of evaluating e-learning.

An important factor underpinning the arguments presented in this book is that e-learning results from a design activity, where the outcome of the design activity is an e-learning artefact. We take a broad view of the interpretation of artefact to mean both tools developed using information and communication technologies (ICTs) and tasks designed through these tools (see section 1.5, p. 9 for further discussion on this). Research into designed artefacts has an extra element not present when researching natural phenomena – that is, whether the designed artefact functions as it was designed, or can be improved. This element of improvement is not present in traditional science research into natural phenomena, but is present in artificial sciences such as engineering, architecture, computer science, and e-learning. These ideas will be explored further in section 6.3, p. 87.

Indeed, in this book, we will emphasize evaluation of the effectiveness of a designed and constructed e-learning environment, purposefully including one or more artefacts. Such an environment might perhaps be used in a number of different contexts.

1.3 Key Characteristics of Learning

In recent years, a consensus has emerged about the type of learning environment that is likely to support learners to develop thinking skills such as critical thinking and problem solving in order to become knowledge workers. Such learning environments are the design outcome of a largely constructivist pedagogical philosophy, adoption of a deep approach to learning and a learner-centred approach to teaching, and outcomes-centred subject design (Phillips, 2005 b). It is not the intention of this section to provide a comprehensive summary of learning theories; what we would like to provide are some key principles about learning that will be used progressively in the book to anchor our comments about designing and evaluating effective learning experiences.

More specific advice is provided by a report from the National Research Council in the United States of America (USA) into how people learn (Bransford, Brown & Cocking, 1999, 2000). Among the key findings of this comprehensive review of research, several themes stand out as having most relevance to tertiary education.

An important goal of tertiary education is for novices to become experts in particular discipline areas. Bransford and colleagues distinguished a key difference between novices and experts, which is that experts have in-depth and organized content knowledge. This enables them to see relationships and patterns between pieces of information, and also to retrieve important parts of their knowledge relatively easily. In essence, an expert has a well-organized, personal knowledge network.

This finding has important implications for universities. For tertiary learners to become experts, they need to attain a deep, organized and contextualized understanding of their discipline – and the learning environment needs to support this. It is much easier to develop a knowledge network if knowledge is organized by fundamental principles, as this enables learners to categorize and interrelate areas of understanding in ways that make sense to them.

A further factor noted by Bransford and colleagues is that for learning to be effective, it needs to be transferable to other contexts. To know ‘about’ something is not sufficient; one needs to know how to use knowledge in new situations that arise. So, learners need to see knowledge not as a fixed commodity, but rather as something to be extended and utilized. In addition, learners are most successful when they see themselves as being active agents in the process of gaining and using knowledge.

These characteristics closely match those of ‘deep approaches to learning’ reported in the tertiary learning literature (Biggs, 1999; Gibbs, 1992; Ramsden, 1988, 1992). Learners who adopt a deep approach to learning are interested in the topic and actively try to understand key concepts. Once these key concepts are understood, an attempt is made to link together the concepts to make a coherent whole. Then new knowledge can be related to previous knowledge and to personal experiences, and thus integrated into a personal knowledge network.

The ‘approaches to learning’ research has identified several key principles about how teachers should design and teach to increase the likelihood that learners adopt a deep approach to learning. Three key ideas are as follows:

1. Transmission approaches lead to surface learning.
2. Depth of learning is determined by the nature of the learning tasks.
3. Surface and deep approaches are reactions to the teaching environment.

However, the situation is more complex and the motivation of individual learners is a critical factor. Many readers may remember boring, didactic learning environments where they were, nevertheless, engaged by the topic and adopted a deep approach to learning; they may also remember interactive learning activities that did not engage them, perhaps because of conflicting demands on their attention. Teasing out how best to design for learning is always complex and involves a degree of compromise (Goodyear, 2009). Perhaps the most fundamental principle is that the modern university teacher needs to design flexible learning experiences that support diverse groups of learners as they learn how to learn.

Well-designed learning tasks become an important precursor for effective learning to take place. A learning activity is “an interaction between a learner and an environment (optionally involving other learners, practitioners, resources, tools and services) to achieve a planned learning outcome. Under this definition, task, content and context are fundamentally inseparable” (Beetham, 2004). We will also distinguish between learning tasks and learning activities, according to Goodyear's (2009) insightful definition: “Tasks are what teachers set. Activities are what students actually do” (p. 13).

However, while the interrelatedness and holistic nature of this description of learning tasks and learning activities is important to bear in mind, some analysis is needed in order to be able to suggest principles and strategies to teachers about how to decide whether learning tasks are well designed or not in any particular learning context.

Learning tasks are usually planned in terms of outcomes or objectives. Allan contended that there are “fundamental conceptual differences between outcome-led design and the traditional university approach which emphasizes input and process” (Allan, 1996, p. 104). The modern university teacher needs to design effective learning tasks, and to “facilitate their productive use by the student” (Laurillard, 2002b, p. 24). This argument leads us to think of learning as having three components: the learning environment designed for learners to use; the learning processes used by students (how they learn); and the learning outcomes that students can demonstrate (what they learn).

These ideas are discussed further in Chapter 3, where a conceptual model is developed which characterizes learning in terms of the interactions between learners, teachers and the learning environment, learning processes and learning outcomes. These ideas are fundamental to the structure of this book.

1.4 Evaluating Learning and e-Learning

Evaluation is a term that is often used imprecisely, with different meanings in different contexts. This section seeks to unpack the various meanings of ‘evaluation’ in an educational context, and clarify the way that we interpret the term in this book, as well as setting the scope of what is covered in this book.

There are numerous ways that the term ‘evaluation’ is used in higher education. There is also a cultural element to definitions of ‘evaluation’. In education textbooks originating in the USA, ‘evaluation’ is used in the sense of making judgements about assessable learner work. In this book, we follow the United Kingdom (UK) convention, describing the process whereby teachers set specific tasks to judge the extent to which learners can demonstrate learning outco...

Table of contents

• Cover Page
• Half-Title Page
• Series Page
• Title Page
• Copyright Page
• Dedication
• Table Of Contents
• Figures
• Tables
• Series Editors' Foreword
• Foreword
• Preface
• Acknowledgements
• PART I Setting the Scene
• PART II Theory
• PART III Practical Aspects of Evaluation Research
• References
• Author Index
• Subject Index