Assessment in Open, Distance, and e-Learning

Lessons from Practice

Jessica Evans, Sally Jordan, Freda Wolfenden

• Publisher: Taylor & Francis
• Year: 2020
• Print ISBN: 9781138361034
• eBook ISBN: 9780429780233
• Edition: 1
• Topic: Education
• Subtopic: Education General

Universities across the globe are attempting to change assessment practices to address challenges in student engagement and achievement and to respond to a global employability agenda demanding evidence of a broader range of skills and competencies. In the UK this has acquired urgency given the shift of higher education over the last 20 years from the prerogative of an elite minority to mass participation in a highly diversified market system. Integral to this interrogation of objectives for assessment is the identified need to develop and improve academics' assessment practice. Strategies frequently focus on attendance at formal Continuous Professional Development events and/or implementation of institutional blueprints.

This book showcases how scholarship as part of academics' practice can be part of an academic toolkit for change that expands awareness and knowledge of the purposes and effects of the pedagogy of assessment. The case studies – ranging from assessment in Massive Open Online Courses (MOOCs), to assessment design for students whose first language is not English, to the effectiveness of peer learning to support academic integrity and programme-level assessment strategies – are framed by an introduction that explores a 'communities of practice' approach to the institution-wide improvement of assessment. It argues – through a case study from The Open University (OU) – that academics' professional expertise is best deepened through participation in authentic activities of teaching and scholarship. The discussion identifies what is involved in such an approach including the role of an enabling principles-based framework, the constraints on implementation, and the implications for leaders of teaching and learning.

This book was originally published as a special issue of the Open Learning journal.

Assessment worlds colliding? Negotiating between discourses of assessment on an online open course

Laura Hills and Jonathan Hughes

ABSTRACT

Using the badged open course, Taking your first steps into Higher Education, this case study examines how assessment on online open courses draws on concepts of assessment used within formal and informal learning. Our experience was that assessment used within open courses, such as massive open online courses, is primarily determined by the requirements of quality assurance processes to award a digital badge or statement of participation as well as what is technologically possible. However, this disregards much recent work in universities that use assessment in support of learning. We suggest that designers of online open courses should pay greater attention to the relationship of assessment and learning to improve participant course completion.

Introduction

The Open University, UK (OU) has existed as a provider of open distance education (ODE) accredited courses for over four decades. Over this period, these courses have been ‘open’ in the sense that no prior qualifications are required to enrol. Since 2006, the sense of openness has been extended with the development of open online courses that are free but not formally accredited. This heritage is evident in the creation of OpenLearn ( www.open.edu/openlearn ), which continues to be a free resource of materials which draw on OU materials originally developed for modules in qualifications. Then in 2013, the OU founded the MOOC platform FutureLearn ( www.futurelearn.com ), in which it was the leading partner, a worldwide consortium of universities both ODE and campus-based.

However, whilst FutureLearn is to date somewhat remote from the core teaching business of the OU, a more recent initiative seeks to strengthen the relationship between free learning and paid-for learning. This is the development of badged open courses (BOCs), the first tranche of which were launched in 2015. BOCs are housed on the OpenLearn platform and provide more structured pathways through OpenLearn materials (Law, 2015a ). Unlike MOOCs, there is no institutional instructor or defined start date. They differ from other OpenLearn courses in that they have assessment, successful completion of which results in the learner being able to claim a ‘badge’. Badges can be described as an ‘assessment and credentialing mechanism that is housed and managed online’ (MacArthur Foundation, 2015 ) and can be viewed as a virtual form of the type of badges typically associated with scouting. They can also be displayed on learners’ social media profiles using software such as Mozilla ( http://openbadges.org ), and therefore shared with friends and employers.

Each of these BOCs in volves 24 h of study spread over 8 weeks, with formative assessment (quizzes) each week and summative assessment at the mid- and end-points of the course. The format for both formative and summative quizzes is the same, including free text, drag and drop and multiple choice, with three attempts allowed for each question and an increasing amount of feedback being given after each attempt (see Appendix 1, Figures 1 – 4 ). However, only the summative quizzes contribute to the overall course mark. Passing the course results in a Mozilla-compatible OU badge.

A particular feature of the OU BOCs is that they require learners to engage for a significantly longer period of time and to do significantly more, in terms of activities and assessment, than is required in most badged courses, which are usually only a few hours long. They, therefore, aim to deliver a structured means to prepare learners who are considering or about to enrol for qualifications in online and distance education. As a result, the BOCs have the specific aim of developing skills and confidence to encourage what is conceptualised by the OU as a personal ‘journey from informal to formal learning’ (or JIFL); they offer a stepping stone into accredited education. Evidence suggests that 28% of BOC learners ‘click-through’ to make an enquiry to the OU (Hills, Gore, & Hughes, 2016 ). The BOCs include subject areas such as introductory mathematics, English language skills and learning to learn. They are therefore aimed at a wider group of learners than many existing MOOCs, which have tended to attract an already well-educated audience (Lane, 2012 ). They target learners who fit into ‘widening participation’ categories and in this respect BOCs are a development of the ‘traditional’ OU curriculum offer designed to encourage under-represented groups into higher education.

The focus of this paper is the Taking your first steps into higher education BOC ( www.open.edu/openlearn/ocw/course/view.php?id=1139), and the experience of the authors in developing an appropriate assessment strategy for the course. Working within the prescribed structure of the platform (based on Moodle), we endeavoured to develop an approach that provided learners both with evidence of achievement and feedback on their progress. In so doing, we looked at models of assessment from within formal education, in particular higher education, and also informal learning, including OERs, and traditional informal and lifelong learning. We identified two critical tensions between what we wanted from a pedagogical perspective versus what the technology allowed; and between the conceptualisation of the BOC as an instrument for purely informal learning and the institutional requirements associated with awarding an institutional branded badge. Such tensions are increasingly pertinent within the broader higher education sector as it diversifies and the distinction between formal and informal learning becomes less defined.

The BOC: informal or formal learning?

Attempting to differentiate between informal learning and formal learning presents a paradox. In some ways, this difference appears straightforward. The European Commission ( 2000 ) suggests that formal learning takes place in institutional settings and leads to recognised qualifications. Learners intentionally participate and are aware that learning is occurring. In contrast, informal learning is described as ‘the natural accompaniment to everyday life’ (European Commission, 2000, p. 8) and is held to be such an integral part of life that it is often not recognised as learning.

However, underneath these definitions lie many layers of complexity and a wide range of typologies and approaches. These include attempts to enlist metaphors, such as the comparison with food production made by Golding, Brown and Foley ( 2009 ) in which formal learning is compared to large-scale food production, whilst informal learning is seen to be ‘more organic and home-grown’ (Golding et al., 2009, p. 41). Such a metaphorical approach might provide a comfortingly easy distinction between formal and informal learning as if there is some ‘deterministic dichotomy between formality and informality’ (Cameron & Harrison, 2012, p. 277). Challenging this binary characterisation, Colley, Hodkinson and Malcolm ( 2003 ) indicate that key attributes of informal and formal learning will play out differently in specific learning situations. They note:

Attributes of in/formality are interrelated differently in different learning situations. Those attributes and their interrelationships influence the nature and effectiveness of learning. Changing the balance between formal and informal attributes changes the nature of the learning. (Colley et al., 2003, Executive Summary)

Such a contextual approach is particularly useful for considering the design, approval and enactment of assessment in open online courses.

Our BOC includes characteristics of both formal and informal learning. Teaching is provided by a university and is structured for the learner with clearly defined activities for each week of study and the learning is summatively assessed – all normal characteristics of formal learning. However, the BOC is permanently available on OpenLearn, meaning that anyone can enrol at any time and engage with all the materials over a period of time of their own choosing; and no direct interaction with a teacher is needed to study the course. Moreover, no credit is conferred for successful completion and in these senses it could be described as an instance of informal learning.

Even this fairly cursory look at Taking your first steps into higher education highlights how attributes of both formal and informal learning are apparent and the exact balance will be relational, depending on the motivation, future plans and context of each individual learner (Colley et al., 2003 ). For some, study on the BOC will indeed be a ‘step up’ from undirected browsing on websites like OpenLearn and a step towards the formality of accredited university modules. For others a BOC will be an instance of learning leading onto another learning episode comparable in its level of informality. This diversity has implications for how assessment in the course is perceived by different stakeholders.

Assessment and the BOC

As academics charged with writing the course, our primary concern was supporting students to develop academic learning habits and skills appropriate to study in higher education. Assessment needed to be appropriate to this purpose. Our challenges were multiple. Firstly, as an open online course enrolment is unrestricted and we have no prior knowledge of those enroling, their previous experiences of assessment or the way in which they will approach the assessment tasks.

Secondly, defining the nature of our engagement with quality assurance and adherence to regulatory frameworks (Stowell, Falahee, & Woolf, 2016 ), a key feature of assessment from HE institutions is problematic. The decision facing HE providers is the extent to which assessment on OERs, MOOCs and BOCs should reflect the standards applied to summative assessment on accredited courses. Adherence to these procedures was evident when it came to designing assessment for our BOC. Significantly, BOC assessment had to be approved by the university’s ‘examinations office’ in the same way as any other assessment. For the institutional branding of the badges meant that they could not just be ‘given away’ as this might incur reputational damage that could have implications for the more formal offerings of the university (Law, 2015a ) – the badge is driving the assessment and not broader notions of learning. Thus, for example, the requirement that learners on a BOC took (and passed with a minimum 50% score) two moodle quizzes was clearly shaped by an institutional view of the role of summative assessment for credit. Such concerns extended to determine user engagement with the weekly quizzes on the BOC. On the BOC course platform, it is technically possible to allow unlimited attempts at these quizzes. However, this would have opened up the possibility that ‘learners’ will just keep clicking until they get the answers right. To try to ensure that assessment on the BOC sat comfortably with prevailing understands of assessment used for the award of credit, this situation was deemed unacceptable and a limit of three attempts within a 24-h period was set. Furthermore, the form of such quizzes greatly influences the ‘skills, competencies or knowledge designers and learners prioritise, and how they will be achieved’ (Cross, Whitelock, & Galley, 2014 ).

It is now commonly accepted in the sector that a critical feature of contemporary assessment in formal higher education, including ODE, must be an increasing focus on assessment for learning (Nicol & Macfarlane-Dick, 2006 ) which reflects the notion that ‘the majority of students regard assessment as the most important aspect of their course and, consequently, use it to guide and frame their learning’ (Price, Carroll, O’Donovan, & Rust, 2011, p. 480). However, it is more complex to apply the findings of such research to a context where there is no direct communication between student and educator, and where there are practical as well as financial limitations on the assessment task used. Nonetheless, recent developments on computer-generated assessment and feedback in accredited learning do point to a role for the use of e-assessment in empowering students to self-regulate their learning (Jordan, 2014 ). But even though approaches to the assessment of informal learning, including within the rapidly growing world of OER, have been drawn primarily from theories of assessment in the formal learning sector (Farrell & Rushby, 20...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Contents
• Citation Information
• Notes on Contributors
• Editorial
• Introduction: Developing academics' assessment practices in open, distance and e-learning: an institutional change agenda
• 1 Assessment worlds colliding? Negotiating between discourses of assessment on an online open course
• 2 Paragogy and flipped assessment: experience of designing and running a MOOC on research methods
• 3 'I understood the words but I didn't know what they meant': Japanese online MBA students' experiences of British assessment practices
• 4 Case study: what supports students to improve their grades?
• 5 Using collaboration to foster academic integrity
• 6 The introduction and refinement of the assessment of digitally recorded audio presentations
• 7 Designing and developing a programme-focused assessment strategy: a case study
• Index