Learning Analytics in Higher Education

Current Innovations, Future Potential, and Practical Applications

Jaime Lester, Carrie Klein, Aditya Johri, Huzefa Rangwala

• Publisher: Routledge
• Year: 2018
• eBook ISBN: 9781351400527
• Topic: Education
• Subtopic: Education General

Learning Analytics in Higher Education provides a foundational understanding of how learning analytics is defined, what barriers and opportunities exist, and how it can be used to improve practice, including strategic planning, course development, teaching pedagogy, and student assessment. Well-known contributors provide empirical, theoretical, and practical perspectives on the current use and future potential of learning analytics for student learning and data-driven decision-making, ways to effectively evaluate and research learning analytics, integration of learning analytics into practice, organizational barriers and opportunities for harnessing Big Data to create and support use of these tools, and ethical considerations related to privacy and consent. Designed to give readers a practical and theoretical foundation in learning analytics and how data can support student success in higher education, this book is a valuable resource for scholars and administrators.

1

ABSORPTIVE CAPACITY AND ROUTINES

Understanding Barriers to Learning Analytics Adoption in Higher Education

Aditya Johri

Introduction

As I sit here writing this near the start of a new semester, it is hard to imagine that it is almost a decade since the term ‘Big Data’, in its current incarnation, “the mining and processing of petabytes’ worth of information to gain insights into customer behavior, supply chain efficiency and many other aspects of business performance” (Pearson & Wegener, 2013, p. 1), was first introduced in the mainstream media by The Economist (2010). Since then, the notion of data analytics has infused almost all thinking about how organizations go about their business; and data-driven organizations and organizations driving data-driven practices, including infrastructures such as cloud computing, have become the jewels of the business world (e.g., Amazon™, Google™, etc.). It was reported in a recent study of more than 400 large companies conducted by Bain & Company that early adopters of Big Data analytics had a significant lead over the rest of the corporate world (Pearson & Wegener, 2013). The companies that had adopted Big Data analytics, according to this report, were (1) twice as likely to be in the top quartile of financial performance within their industries, (2) five times as likely to make decisions faster than market peers, (3) three times as likely to execute decisions as intended, and (4) twice as likely to use data frequently when making decisions (Pearson & Wegener, 2013). Given reports like this, it is not surprising that many organizations, spanning various industries, are looking toward data analytics as way to propel themselves forward.

Higher education institutions are also cognizant of the potential value of analytics to improve organizational performance. As a result, at least two leading ideas and communities—educational data mining (EDM) and Learning Analytics (LA)—have emerged on the scene (Lester, Klein, Rangwala, & Johri, 2017). EDM, which has a more computational stance, is concerned largely with developing, researching, and applying computerized methods to detect patterns in large collections of educational data that would otherwise be hard or impossible to analyze due to the enormous volume of data within which they exist. EDM, as the name implies, is defined as “the application of data mining (DM) techniques to this specific type of dataset that come from educational environments to address important educational questions” (Romero & Ventura, 2013, p. 12). Overall, EDM researchers and practitioners analyze data generated by any type of information system that supports supporting learning or education, defined broadly—schools, colleges, or universities. These data are broad and include interactions of individual students within an educational system (e.g., navigation behavior, input in quizzes, and interactive exercises) but also administrative data (e.g., school, school district, teacher), demographic data (e.g., gender, age), and so forth. The other allied field, LA, is concerned more with learners directly and includes as its purview “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (Siemens et al., 2011, p. 4). Whereas LA is largely concerned with improving learner success (Gašević, Dawson & Siemens, 2015), the practitioners or LA differentiate academic analytics as “the improvement of organizational processes, workflows, resource allocation, and institutional measurement through the use of learner, academic, and institutional data. Academic analytics, akin to business analytics, are concerned with improving organizational effectiveness” (Siemens et al., 2011, p. 4). For the purposes of this chapter, I am going to use LA as a catchall for all the data and analysis techniques mentioned above.

We hear continuously about how LA has the potential to change higher education and how these data are making inroads, but the reality at the level of everyday work practices is different. This is not to say that higher education organizations are not leveraging data and analytics (Arroway, Morgan, O’Keefe, Yanosky, 2016), but the adoption lags behind what organizations in other sectors are doing or moving toward. According to Bichsel (2012), in 2012, between 55% and 65% of institutions reported engaging in data activity at the level of finance and human resources, but less than 20% of institutions reported data analytics activity in the functional areas of instructional management, centralized information technology (IT) infrastructure, student learning, strategic planning, alumni and advancement, research administration, library, cost to complete a degree, human resources, facilities, faculty promotion and tenure, faculty teaching performance, procurement, and faculty research performance. A lot of the fervor is still about the potential and not necessarily the actual implementation of LA. For instance, the most innovative work in terms of using machine learning and DM is limited largely to researchers in the educational research community (SOLAR, EDM, etc.). Companies that are providing products are, for the most part, using rudimentary techniques for data analysis and presentation—bar graphs, pie charts, and large Excel™ files. This is complicated by data access issues, including data ethics and privacy, and as many authors in this volume point out, that is not necessarily a bad thing. Yet, a lack of clarity around data use has limited the development of innovative applications. There is also limited understanding of the impact LA can have beyond the immediate concerns that most higher education institutions, especially publicly funded ones, are facing, such as student retention. Tuition fees are becoming an ever larger portion of the budget as public funding is declining. The other funding option is externally funded grants, which essentially means overhead, and, therefore, this is another area in which analytical efforts are targeted. Finally, LA is also prevalent in reporting, as accreditation concerns overwhelm institutions often at the expense of institutional effectiveness. At the infrastructure level—data warehousing, for instance—there has been significant uptake of information technology (IT) in most higher education institutions, and, therefore, there is a reasonable expectation that slowly LA will percolate to other aspects of higher education institutions.

In this chapter, my goal is to shed light on what prevents a greater adoption of LA within higher education. As opposed to other chapters in the volume, I first use a personal perspective on LA, based on my experiences as an instructor, an administrator, and a researcher, to shed light on what I believe are some essential issues that need to be addressed. After that, I look at some recent reports that shed light on what organizations—largely business enterprises—have to do in order to leverage data analytics successfully. I then move toward some theoretical explanations for the relative lack of data analytics application in higher education organizations and use these theoretical underpinnings to examine three case studies from my own experience working on an LA tool research project—cases, which are likely familiar to readers from their own experience. Finally, I end with practical considerations for overcoming barriers to the use of LA in higher education.

I have to start with a caveat—my personal characteristics and experiences shape my experience of using LA. I am a technology adopter and so are most of the people who worked on the project I refer to in this chapter. I sit in an engineering school and teach analytics. I have a research interest in this area. I know people I can reach out to when I need help with technology, and I know the resources to refer to when I hit a wall. This is probably not the case for most people on campus. Just as an example, there is a wide variation in just using a learning management system (LMS) across the institution. Some of this has to do with lack of technological expertise, but a lot of it has to do with a lack of understanding of what LMS can and does add to teaching. In many cases, it actually does little beyond acting as a repository of resources. Even in terms of using email, which is a standard practice now, there is a diversity of tools people use (a large part of the user base actually using Gmail™ to access the university mail services). Therefore, there will always be a vast variation in any kind of technology use on a campus, and this, itself, I will argue, is problematic.

Missed Opportunities for Learning Analytics

I start with reflecting on my work this week; a busy week as the new semester starts shortly. Here are the instances of data or analytics use I can come up with in my work. I looked at some data about my research expenditures—Excel™ charts sent to me to make sure expenditures were progressing as planned and charges were correct. I looked at it, saw some numbers in red, and went about taking action by emailing a few folks to ensure things were corrected. I had six such reports to go through, and most of my action was taken when things were way off or when things were off but I thought it was a temporary issue. The other data I looked at were the number of students enrolled in my classes to make sure that one of the classes had the minimum number of enrollments. Otherwise, I would have done some more advertising and publicity for the class. Then, I went into our LMS, Blackboard™, to set up course pages for the upcoming semester. I copied some stuff, I updated some other stuff, and I tweaked some settings. Now, my hope is that things will work smoothly when the semester starts. There doesn’t seem to be a lot of analytics going on here and minimal use of data. It is clear that as a faculty member, the use of data and analytics is not a part of my everyday practices or of what some organizational theorists will call my ‘routines’. The idea of routines or practices that have become a norm and are embedded across the organization is a critical one for my argument, and I will address it in detail later.

Are there instances, though, within my work practices where data and analytics would have been important for me or would have helped me in some ways? Can it be integrated into my routines—actions I have to take habitually? Certainly. For instance, I would much rather be able to do a dynamic review of my grant expenditures. Grant funding comes with a time stamp—it has to be spent within a specific amount of time, and the funds can be used only for certain activities and items as specified in the proposal. Yet, there is some leeway wherein spending can be different than what is exactly proposed. This means that it is important to monitor and make adjustments as the grant period progresses. The systems in place to monitor spending—many driven by federal regulations—make the monitoring problematic. There is always a delay between spending and when it is posted against the grant account, for instance. There is also a lag, as finding a student with the right expertise can take time. All this makes running the grant very fluid, and this problem is compounded with each additional grant. Hence, some way of continuous monitoring is essential. And yes, I know there are systems that allow me to do that to some extent, and these vary by the institution, but they are cumbersome to use and no single system provides me with all the information I need to take action. At the end of the day, it will take an email or a face-to-face visit with a fiscal person or a post-award administrator to resolve the issue, and often there is no single point of contact. The grants office is responsible for certain issues, while the home department of college is responsible for others.

When it comes to teaching, in addition to the number of students, it would be great if I could get some information about the students. Now, there is a way in which I can log in to another system, dig through a few screens, and get their photos and degree information, but, once again, it is cumbersome. What I really want to know is their prior knowledge, their achievements, and their interests. In my role as a teacher, my primary responsibility lies in ensuring students learn. A lack of knowledge of what students actually know, except for a few broad markers, is a real barrier to how I go about my work. Not that I will be able to take care of all the variance in prior knowledge, but at least I will have some idea of where the students are coming from. In an ideal world, their years of schooling and acceptance into a program should convey some of that, but the reality is quite different.

Finally, the LMS is a black box, where I put in content and effort but nothing much comes out. In one of my classes, which I teach online, everything is managed through our LMS, Blackboard™—the course readings and videos, the quizzes, the discussions, the reflection assignments submitted by students; all of this is online. Yet, I have very limited knowledge of what is actually going on in the class until I see a submission from a student and need to grade that. There is no overview dashboard that tells me who has looked at the content, who is on track to meet the deadline, how long is it taking students to read the content, and so on. The end-of-semester evaluations are not mandatory, and, therefore, it is hard to interpret and use that data to revise the course (although it is still used to evaluate faculty as that is the only data point available). The data go somewhere, somebody benefits—makes a lot of money off my effort—but nothing feeds back to my work. And yes, there are ways to better monitor the use of the system—you can turn on the option to store the number of views, for instance—but most of that information is summative. Formative analytics is unfortunately missing.

This is the crux of the issue—none of my work practices incentivize me to put effort into utilizing analytics more effectively. I still try my best to incorporate these data, because I want to work more efficiently. From research practices, to teaching practices, to advising, nothing is built to draw on or benefit from data and analytics, and, hence, nothing does. The incentive for grants is real—Who wants to go over budget? So, I pay some attention. And even if I want to change the way I teach or run grants using more data and analytics, it is hard to do unless the infrastructure is in place. To some extent, these issues are personal, and organizational members need to be invested and willing to make changes, but without the infrastructure in place, these jobs becomes much harder. Why, after all this effort, is this still the case? I try to address this issue in this chapter. I’m not technology averse or analytic averse; I even have multiple grants on this topic and write about it. So, why hasn’t it made it into my practice? Is this lack of integration of LA into my practice a problem, and, if so, what is the solution? Here are some practical ideas, and later, I will discuss why these often fail to make their way into higher education.

Practical Ideas for Success with Learning Analytics

Let’s take a normative look at what needs to be done if one wants to leverage analytics in a meaningful manner. Nothing is more normative than prescriptions by professional consulting firms such as McKinsey, so I am drawing on multiple reports and papers from them including the following: Arellano, DiLeonardo, and Felix (2017), Brown, Kanagasabai, Pant, and Pinto (2017), Chui, Henke, and London (2017), Kirkland and Wagner (2017).

Data Capture and Availability

One of the first issues that needs to be addressed for any form of analytics to be performed is the capture of data. Without data—useful data—there is no scope for any analysis to be performed. The proliferation of digitization across organizations means that it is possible to capture a wide variety of data and also to acquire large volumes of it. For instance, a retailer now has access not only to sales data and customer information through their credit cards but also their online customer profiles and even log data for every action that they perform on the retailer’s website. In higher education organizations, similarly, there is the opportunity to capture a variety of data about students such as their incoming Grade Point Average (GPA), high school performance, their interaction with an LMS, and even their swipe access data using their student identity card. Of course, these increased data bring with them numerous challenges for capture, storage, and analysis, especially in regard to whether useful data are being captured. For instance, if we take the mission of a higher education institution to be improving student learning, we need to then think about whether the data that are captured and can be analyzed assist us with this mission. As of now, we have very little that data speak to learning that helps us understand students’ cognitive process or misconceptions. We have grades and GPA information, which is more a signal of achievement rather than cognition. At best, it is an indirect marker of knowledge. To leverage useful LA data that assess learning requires the collection of disparate data, sources need to be monitored and stored—from student admission and enrollment data to their activities on the LMS.

Modeling and Analysis

The second important step in the analytic process is the availability and use of different mathematical models that can take useful data and turn them into something actionable—and provide insights that allows us to better understand an issue. There are dozens, if not more, models or techniques available for analyzing data, including those that draw on traditional statistics and social science such as statistical models, visualization, social network analysis, sentiment analysis, influence analytics, discourse analysis, concept analysis, and sense-making models, and those that draw on computational DM such as classification, clustering, Bayesian modeling, relationship mining, and discovery with models (Romero & Ventura, 2013). These techniques have been used for predicting student performance, providing feedback for supporting instructors, recommending problems or contents to students, creating alerts for stakeholders such as students to complete a task, domain modeling to describe the domain of instruction in terms of concepts, and for planning and scheduling future courses. These applications thoug...

Table of contents

• Cover
• Title Page
• Copyright Page
• Contents
• List of Tables
• List of Figures
• Preface
• Acknowledgements
• 1 Absorptive Capacity and Routines: Understanding Barriers to Learning Analytics Adoption in Higher Education
• 2 Analytics in the Field: Why Locally Grown Continuous Improvement Systems are Essential for Effective Data-Driven Decision-Making
• 3 Big Data, Small Data, and Data Shepherds
• 4 Evaluating Scholarly Teaching: A Model and Call for an Evidence-Based Approach
• 5 Discipline-Focused Learning Analytics Approaches with Users Instead of for Users
• 6 Student Consent in Learning Analytics: The Devil in the Details?
• 7 Using Learning Analytics to Improve Student Learning Outcomes Assessment: Benefits, Constraints, & Possibilities
• 8 Data, Data Everywhere: Implications and Considerations
• Editors
• Contributors
• Index