Innovative Learning Analytics for Evaluating Instruction

A Big Data Roadmap to Effective Online Learning

Theodore W. Frick, Rodney D. Myers, Cesur Dagli, Andrew F. Barrett

• Publisher: Routledge
• Year: 2021
• Print ISBN: 9781032000183
• eBook ISBN: 9781000454772
• Topic: Éducation
• Subtopic: Éducation générale

Innovative Learning Analytics for Evaluating Instruction covers the application of a forward-thinking research methodology that uses big data to evaluate the effectiveness of online instruction. Analysis of Patterns in Time (APT) is a practical analytic approach that finds meaningful patterns in massive data sets, capturing temporal maps of students' learning journeys by combining qualitative and quantitative methods. Offering conceptual and research overviews, design principles, historical examples, and more, this book demonstrates how APT can yield strong, easily generalizable empirical evidence through big data; help students succeed in their learning journeys; and document the extraordinary effectiveness of First Principles of Instruction. It is an ideal resource for faculty and professionals in instructional design, learning engineering, online learning, program evaluation, and research methods.

1 Learning Journeys in Education

DOI: 10.4324/9781003176343-01

Summary: We introduce the concept of a learning journey. We use the Oregon Trail as a metaphor to explain why traditional quantitative and qualitative research methods are inadequate for capturing learning journeys. On the other hand, temporal maps do capture learning journeys, and Analysis of Patterns in Time (APT) can be used to count occurrences of qualitative patterns in temporal maps. We discuss the fundamental limitations of traditional qualitative and quantitative research approaches for determining effectiveness of instructional methods.

Metaphor of a Journey: The Oregon Trail

Education is a temporal process. Teaching and learning occur through intervals of time. Education is analogous to a journey, such as following the Oregon Trail. The destination we often want to reach in education is successful student learning achievement. Teachers provide the guidance to help students reach the destination. From a practical perspective, we want to document what makes some trips more successful than others. If other educators who come along later follow routes that have previously been successful, then their students would be expected to benefit also.

The Oregon Trail was a major route that early U.S. settlers traveled in the mid-1800s from what is now Independence, Missouri, near Kansas City, to the Columbia River Valley, near what is now Portland, Oregon (see https://en.wikipedia.org/wiki/Oregon_Trail). Lewis and Clark were pioneers who successfully made the trip in the early 1800s largely by horseback, canoes and rafts, and on foot, mostly by following the Missouri and Columbia Rivers with help from native Americans along the route. Later, settlers traveled by covered wagons pulled by oxen, on horseback, with many walking. The winding 2,200-mile trip took many months and was perilous. Some travelers died on the trip; and others never made it to Oregon, settling in places along the route. Those who did make it often arrived with very little food and belongings. Many travelers lost weight or died due to malnutrition and disease during the trip. Diseases took more lives than anything else, including cholera, dysentery, typhoid fever, and diphtheria. In that sense, those who were fortunate succeeded, but survivors often were somewhat poorer and weighed less when they arrived at their destination.

Fast-forward to May 2020.

Many people now can get to Portland from Kansas City by driving an automobile and by taking interstate highways. According to Google Maps, the trip should take about 27 hours of driving time. If stopping overnight to rest, then the trip might take three driving days, travelling about 600 miles each day. A minivan could be rented to transport five passengers and their luggage for about $500, plus the additional cost of fuel, two motel rooms for two nights of lodging, food purchased for meals along the way, etc. A conservative estimate of the cost of the 1,800-mile trip would be approximately $300 per person, assuming expenses and motel rooms are shared.

Another way to make that trip is to fly on commercial passenger airlines. For example, Southwest Airlines offers one-way fares for about $400 for an adult that will typically take seven to ten hours, depending on stops and plane changes. Two bags, 50 pounds or less, can be taken as luggage with no extra charge.

Finally, for the more wealthy, private jets can be chartered. A web search in May 2020, indicated that the one of the least expensive flights would cost approximately $18,000 for three hours of flight time in a small jet for six passengers, or approximately $3,000 per person. Flights can often be scheduled to depart and arrive when convenient for passengers. These jets typically have more comfortable seating and more amenities.

The State-Trait Approach to Measurement: Quantitative Methods

In this approach to research, we measure states or properties of individual persons, things and conditions at various points in time. Using quantitative research methods, we might conduct an Oregon Trail Study by assessing personal wealth, a person’s weight at the start and end of the trip, duration of the trip, and overall cost. In effect, the above description characterized states and traits of those individuals at various points in time for different classes of people and different eras. Separate measures would be taken: cost per person in U.S. dollars, weights of each person and their luggage, trip duration in hours, whether or not the destination was reached, and the personal wealth of each individual when the trip begins and ends.

A quantitative study using a linear models approach (LMA) would most likely find an inverse linear relationship between cost of trip and its duration. Trips that take less time cost more. For wealthy individuals, a charter jet rental would not appreciably change their personal wealth. For someone with only $1,000 in personal savings, a charter flight would leave them several thousand dollars in debt. For most individuals in 2020, their weight would not appreciably change when measured at the start and end of the trip, while that would not be true for many early settlers in the 1800s, who arrived at their destination emaciated and starving.

Suppose we did take independent measures such as these on a sample of 500 individuals who took trips from Kansas City to Portland, Oregon in the mid-1800s and early 2020.

Here is what we would likely find. Richer people complete the trip in less time than poorer travelers. There would be a negative linear relationship between personal wealth and trip duration. Greater wealth would be associated with shorter trips. If we look at the relationship between individuals’ weight lost during the trips, early settlers would have lost more weight on average, whereas travelers in 2020 would not appreciably change their weight. Overall, there would be a positive linear relationship. Trips of longer duration would be associated with more weight lost. More deaths also occurred during the longer trips, also a positive linear relationship—statistically speaking.

Without information about what happened during the trip, it would be difficult to explain the weight loss. Moreover, why is it that richer people make the trip in less time? Without additional information about modes of travel, speed along the way, routes taken, relative costs, and causes of deaths and not finishing the trip (i.e., dropouts), it would be unclear why.

For educational research using quantitative methods, this approach is descriptive of thousands of studies from the 1960s to the present. Quantitative research studies typically use a state-trait method of measurement, and statistical linear models are used to relate those measures (Kirk, 1999, 2013; Tabachnick & Fidell, 2018). In effect, snapshots are taken at each point in time, but there is often lack of information about what happened between the snapshots.

Individual Episodic Stories: Qualitative Methods

When using qualitative research methods that are narratives, we essentially tell episodic stories (e.g., Creswell & Creswell, 2018; Creswell & Poth, 2018). For example, consider the trip made by Lewis and Clark in the early 1800s. Then compare it with a trip taken by Bill Gates in 2020, who is a multi-billionaire and could easily afford his own private jet. Telling unique stories about their contrasting trips might be quite interesting and illuminating, but it would be unwise to make any generalizations about all Oregon Trail travelers from this non-random sample of two trips.

In qualitative research, we may discover some interesting common patterns in the unique stories, but generalizability of findings is on shaky grounds. For example, Lewis, Clark, and Gates became famous men, who were willing to take risks when younger. Samples are typically very small. Sampling error and lack of generalizability to larger populations is a paramount criticism. Rich, detailed descriptions of a few cases may provide insight into what is happening in education, but it makes it difficult to generalize about what educational methods are more effective than others. It makes it difficult to predict educational outcomes in general—about what is likely to make a difference in successful learning.

Qualitative Temporal Mapping that is Quantifiable and Generalizable: A Third Alternative for Educational Research Methods

There is another way to approach this, referred to as Analysis of Patterns in Time (APT). The state-trait approach does not capture event-by-event temporal details, even so-called time-series analysis methods. The state-trait approach is analogous to taking still photographs. What happens between those snapshots is often unknown. As a further example, use of box scores to characterize baseball games exemplifies the state-trait approach.

On the other hand, APT temporal maps are analogous to “documentary movies” about what happens at various times to students and teachers during their educational journeys. APT temporal maps can describe learning journeys as well as other temporal processes. Temporal maps consist of coded episodic events, as are coded stories in qualitative methods. A temporal map of a baseball game describes what happened during the game, not just the total runs scored and which team won.

However, unlike qualitative methods, APT queries can subsequently identify patterns of events within temporal maps. This can help identify activities that are more or less successful in helping students reach their learning destinations. When temporal maps are representative of large populations, results of APT queries are generalizable to those populations (Frick, 1990). Analyzing temporal maps of professional baseball games can help identify patterns and strategies that lead to a team’s winning season, year in and year out.

APT has been around for several decades, and the benefits of such an approach have been demonstrated in a number of educational research studies (An, 2003; Barrett, 2015; Dagli, 2017; Frick, 1983, 1990, 1992; Frick et al., 2008; 2009; 2010; Howard et al., 2010; Koh, 2008; Koh & Frick, 2009; Lara, 2013; Luk, 1994; Myers, 2012; Myers & Frick, 2015; Plew, 1989; Yin, 1998). A major obstacle to adoption by educational researchers has been the time and effort required to create temporal maps, as well as lack of adequate software for subsequently querying collections of such maps (see https://aptfrick.sitehost.iu.edu/).

It’s relatively easier to measure a few things independently and apply linear models, or to observe a few cases and write detailed descriptions. Nonetheless, APT techniques have been used outside of education with well-documented success (e.g., Moneyball; Lewis, 2004), especially now with big data and large arrays of computers doing parallel processing. Perhaps the most successful application of APT is Google Analytics (2005: https://en.wikipedia.org/wiki/Google_Analytics, https://analytics.google.com/analytics/academy/course/6). It is further likely that APT is used in proprietary extant artificial intelligence systems which use pattern matching for making predictions (see Segaran, 2008; Segaran & Hammerbacher, 2009).

With increasing use of the World Wide Web by billions of people, websites which are used for research purposes can now provide rich data sets that consist of temporal maps. Google has been doing this tracking since 2005. Google Analytics is the current hardware/software platform that allows vast amounts of temporal data to be collected through large computer data centers which utilize state-of-the art parallel processing systems ( so-called “cloud computing”, e.g., https://en.wikipedia.org/wiki/Cloud_computing; https://computing.llnl.gov/tutorials/parallel_comp/). While Google typically sells these services to business clients whose goal is to make a profit, these Google services can also be used by educational researchers who know how to leverage them, even for free.

While APT can be done locally on laptop computers and tablets, or basically on any computing devices including smartphones, this is on a very miniscule scale when compared to Google cloud computing. The real computing power is in the cloud, and computers we touch with our hands are the clients, such as running web browsers, which access the cloud computers via the Internet.

The Larger Problem in Educational Research

It is therefore not surprising that educational research has largely failed to provide widely generalizable empirical results supporting educational methods that are more effective than others. Since the Coleman et al. (1966) study we have known, for example, that there is a positive relationship between socioeconomic status (SES) and student learning achievement as measured by standardized tests. Little else makes a significant difference in learning achievement after statistically controlling for SES, which accounts for a large proportion of the variance in student achievement, when taking a state-trait approach to measurement and using linear models for investigating relationships among variables. Linear models include multiple and logistical regression, factor analysis, canonical analysis, path analysis, discriminant analysis, ANOVA, MANOVA, ANCOVA, hierarchical linear models, time-series analysis, and the li...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Table of Contents
• Foreword
• Preface
• Chapter Summaries
• List of Tables, Figures, and Maps
• 1 Learning Journeys in Education
• 2 Overview of the Big Study
• 3 The Indiana University Plagiarism Tutorials and Tests
• 4 More Details of the Big Study
• 5 Analysis of Patterns in Time as a Research Methodology
• 6 Using Analysis of Patterns in Time for Formative Evaluation of a Learning Design
• 7 Analysis of Patterns in Time with Teaching and Learning Quality Surveys
• 8 Analysis of Patterns in Time as an Alternative to Traditional Approaches
• Epilogue
• Abbreviations and Symbols
• Index