Introduction

Student engagement—the focused, committed, energetic involvement in learning—is seen as an essential element to academic success (Sinatra et al. 2015). Research has shown student engagement to be significantly related to important educational outcomes, such as achievement, persistence, and completion (Finn and Owings 2006; Fredricks et al. 2004; Kuh et al. 2007; Reschly and Christenson 2012), making student engagement a valuable predictor of academic success.

While student engagement is important to any learning experience, it is particularly relevant to technology-mediated learning. Students taking online and blended courses is on the rise (Halverson et al. 2016; Watson e t al. 2014). As adoption increases, a growing concern is the high dropout rates for online instruction (Jordan 2014). Knowing what promotes engagement in technology-mediated learning is necessary to maintain technology-mediated learning as a viable learning option (Dixson 2010).

Necessary to developing a knowledge-base for what promotes student engagement is informative student engagement measures (Hollands and Bakir 2015; Oncu and Cakir 2011; Sinatra et al. 2015). Research is ongoing in developing measures that identify disengaging students or how an instructional intervention impacts student engagement. Self-report methods are common, though other methods are still being explored (Fredricks and McColskey 2012; Henrie et al. 2015). Measures of student engagement most useful to improving technology-mediated instruction will be minimally disruptive, scalable, and measure engagement at the activity level. The purpose of this article is to explore a source of data with potential to meet these measurement criteria: user activity log data produced from learning management systems (LMS).

Theoretical background

Defining student engagement

There are many unique conceptualizations of student engagement. While general conceptualizations exist, many theorists argue for a multidimensional definition of student engagement. The literature describes academic, behavioral, emotional, affective, psychological, social, cognitive, and agentic dimensions (Finn and Zimmer 2012; Fredricks et al. 2004; Henrie et al. 2015; Reeve and Tseng 2011; Reschly and Christenson 2012; Sinatra et al. 2015). Our work focuses on student’s cognitive and emotional engagement. While a variety of engagement factors exist, cognitive and emotional engagement have an established empirical research and theoretical base that supports the importance of these two constructs to learning (see Fredricks et al. 2004, Halverson and Graham 2015). Cognitive engagement is the mental energy students apply to learning, including concentration, curiosity and absorption in learning. We define emotional engagement as the positive emotional response students have to learning, such as enjoyment, excitement, and interest. Cognitive engagement implies an effort to learn, while emotional engagement indicates a willingness to learn. This synergy between cognitive and emotional engagement can lead to the greatest level of learning gains (see also Fredricks et al. 2011), making both of these factors an important consideration for measurement.

Activity-level

Student engagement has been studied at the level of learning within a single activity, focusing on what is happening in the moment, to the level of a student’s whole school experience. Skinner and Pitzer’s (2012) framework of levels for studying student engagement is useful for identifying the purpose and scope of various engagement measures, from factors specific to a single learning activity to broader institutional concerns. For example, the National Survey of Student Engagement (Kuh 2001) is best suited for studying institution-level engagement, with questions focused on learners’ general experience in school. Institution-level measures would be inadequate to identify insights as to how a specific learning activity affected learner engagement in a course (Janosz 2012; Lawson and Lawson 2013). Wang et al. (2014) stated, “Engagement should be measured at the same specificity level as the intervention and as other key variables” (p. 518). Our interest is on student learning, persistence through a course, and the impact of instructional design on student engagement. The most appropriate level of engagement for this focus would be the activity level, where measures focus on students’ engagement in specific learning activities.

Minimally disruptive

Russell et al. (2005) described student engagement as “energy in action,” a description also applied by other researchers (see Appleton et al. 2006; Halverson and Graham 2015; Skinner and Pitzer 2012). Some measures of student engagement are effective at capturing energy in action, while others must disrupt engagement or wait until the learning episode is over to obtain the measure. An example of a disruptive measure of student engagement is self-report, which requires that students’ turn their focus from learning to participating in a survey (see Gobert et al. 2015). Current work is exploring other approaches to measure the cognitive and emotional dimensions of student engagement, such as using physiological sensors (see D’Mello and Graesser 2012). This work is innovative, but can be complicated and costly to use. Future measurement development work should focus on providing valid measures of student engagement that minimize disruption to learning.

Scalable

The capacity for expansive use of a measure, particularly for quantitative analysis, is an important feature. Some measures are more scalable than others. Observational techniques, which employ human observers to record behavior, facial expression, and dialogue, are an example of a measure that is difficult to scale. It can be costly to train and use observers. It is also difficult to obtain good inter-rater reliability (Gobert et al. 2015). Greater issues arise for using observational techniques when learners are engaged in learning at a distance. Rather than using an observer for a group of students in one location, like a classroom or computer lab, observers would have to be scattered across the range of locations where learners may be learning. Surveys, on the other hand, are more scalable, with minimal cost and the capability for mass electronic distribution.

Potential of log data

One possible measure of student engagement that is minimally disruptive, scalable, and at the activity-level is log data from learning systems. Log data are a record of a user’s activity within a system, including click or page view counts, time spent on a given action, keyboard strokes, results of an activity (such as performance on a quiz), and counts of any other activity that may occur within a system. Log data are an activity-level measure, capturing real-time changes in user interactions with the system; minimally disruptive as data is automatically tracked behind the scenes; and scalable, where learning software and online programs can be designed to be used by millions of users. Despite these strengths, further work is needed to understand the value and meaning of learning system log data.

The use of learning management systems (LMS) to deliver learning content, facilitate instructor and student interaction, and track performance is becoming ubiquitous. Log data from these systems are often available for extraction from the LMS provider, making it a potential source of activity-level data to study student learning. Previous research has explored the value of LMS log data (Beer et al. 2010; Cocea and Weibelzahl 2011; Macfadyen and Dawson 2010; Morris et al. 2005). Much of that attention has focused on the relationship between log data and performance. Recent work, however, has investigated the relationship between log data and other outcomes, such as student satisfaction (Henrie et al. 2015) and learner affective states (Baker et al. 2012). What has not yet been done is an investigation of how the log data from the LMS relates to students’ cognitive and emotional engagement (Park 2015). This work is needed in order to verify that log data could be used as a meaningful measure of student engagement.

Students’ log data is an indicator of their involvement in learning, the energy put forth to accomplish learning tasks. Research has argued that student’s cognitive and emotional engagement precedes their behavior (D’Mello and Graesser 2011; Reschly and Christenson 2012). Other research argues that emotion does not always cause behavior, but that the two work together within a feedback system regulated by cognition (Baumeister et al. 2007). Either way, there is strong reason to expect that behavior as manifest through log data is related to student’s cognitive and emotional engagement.

The purpose of this exploratory project is to examine how log data from the LMS relates to student’s responses from a self-report survey of their cognitive and emotional engagement. While previous work has used human observers to tie affective states to log data, we believe self-report is an appropriate measure to use for students’ cognitive and emotional engagement. Appleton et al. (2006) argue that engagement as measured through observation “is highly inferential; therefore, obtaining the student perspective results in a more valid understanding of the student’s experience and meaning in the environment” (p. 431). A strong relationship between the two sources of data will indicate how well log data can be used as a proxy measure for what can be obtained from self-report instruments.

Method

Participants

This study analyzes the relationship of LMS log data and survey responses of undergraduate students from three undergraduate courses (8 sections) from a university in the western United States: one lower-division history course and two upper-level educational technology courses. All courses were offered on the Canvas learning management system in a blended format where seat time was reduced to allow for more online learning. Two hundred and twenty students volunteered to participate in the study during the Fall 2014 semester. While the majority of the students in each class chose to participate in the study, a small number did not, which may bias the results. Possible reasons for nonparticipation include survey emails not going through, a misunderstanding of what participation would entail, or a disinterest in taking surveys. Study participants were predominantly female (81%) between the ages of 21 and 23 (84%). Table 1 explains the number of participants from each course.

Table 1 Number of participants from each course
Full size table

Measures

Student engagement was measured using the 7-item activity-level student engagement survey (see Henrie et al. 2016; instrument can be located in the “Appendix” section). This survey assesses students’ perceptions of their engagement in a specific learning activity. The survey contained both cognitive and emotional engagement items. Responses to items were given using a 5-point.

Likert scale or 7-point semantic differential response scale. “Appendix” section contains the seven engagement items from the activity-level student engagement survey.

Log data from the LMS were collected for all 220 research participants. Log data records were obtained by submitting online requests to the Canvas application programming interface (API). Obtained records included the URLs of all course pages visited by participating students during the semester, time stamps of when each page was visited, the number of discussion posts or replies created, whether an assignment was turned in on time, and grades for each assignment. Two main variables were created using this data: page views and time spent on a page. Page view counts were created based on URLs. Time spent was calculated by taking the time stamp for a page and subtracting its value from the time stamp of the previous page.

It’s possible that an LMS page was open but students were not active on the page. While the product of this calculation is not a precise measure of actual time spent on a page, it does provide a meaningful starting point for capturing data on student engagement. Additionally, some time-spent estimates may be misleading as Canvas did not force user logouts after a given amount of inactivity. If a student never logged out of Canvas, it could be hours or even days between one page view and another. To account for this, time spent scores were capped at a half an hour, which cap is a general internet data analytics standard (Cooley et al. 2013; Drutsa and Serdyukov 2015). After reviewing the distribution of time spent on a page, we found that 89% of the data were less than half an hour in length. This approach may not represent true activity that was occurring on the LMS, but we believe this approach to be the most reasonable in representing time spent on a page.

Student engagement survey data and log data were collected at three points during the semester (about mid-course, three-quarters of the way through the semester, and end of the semester). Researchers emailed a link to a student engagement survey and directed students to respond to their experience in a specified learning activity. Various online activities were selected for each course, including discussion boards, quizzes, online videos, and class projects, all of which took place in the LMS. We chose these activities because they are common to many online courses and take place predominantly in the LMS, ensuring that most student engagement in the activity would be captured by the LMS activity log. The portions of log data that corresponded to the chosen learning activities were extracted from the LMS. Survey data and log data were then paired for analysis.

Analysis

We used a cross-sectional correlation analysis to study how log data correlated with self-reported student engagement scores. Log data and survey data were correlated within a given class for a specific time point. This was done because of the varied nature of each activity.

Log data can be analyzed using different data structuring methods. Structure would include both the amount of log data to include as well as the granularity of the log data. For the amount of log data to use, we chose to look at one week’s worth of log data. Henrie et al. (2015) found that most activity on the LMS occurred on the day that an assignment is due, though students who reported being more satisfied with a learning activity also tended to review an assignment page 24 h or more before an assignment was due. Using one week’s worth of log data would capture early work on an assignment, perhaps indicating high student engagement. Additionally, extending log data to a week’s worth would account for complex activities that could require more than one day to complete.

We also chose to examine three granularities of log data: 1. the general level, 2. the LMS interaction type level, and 3. specific LMS pages level. At the general level, we investigated the relationship between overall page views or time spent on the Canvas LMS during a given amount of time. We also looked at time spent and the number of page views on pages where students were learning as opposed to other types of LMS activity, such as navigating. While there could be any number of categories that could be created, we chose a framework based on the work of Borup et al. (2013), Hawkins et al. (2013) and Heinemann (2005). This framework has three types of LMS interactions: learning, procedural, and social interactions. These categories comprise a number of different types of LMS pages, organized by the purpose of the page view. Table 2 defines and describes the three different types of LMS interactions and the specific types of pages that are included within each category.

Table 2 Hierarchy of LMS activity types and page views
Full size table

Finally, we structured log data to look at target assignment page views, target assignment time spent, and target assignment previews. Students responded in the engagement survey to a specified learning activity. The variable target assignment page views is the number of page views of the Canvas page where the targeted learning activity took place, and target assignment time spent is the amount of time spent on those pages. Target assignment previews was the number of times a student visited the targeted learning activity assignment page at least 24 h before the assignment was submitted. We hypothesize that these three variables represent the most direct log data information about student engagement in the targeted learning activity. The latter was particularly included because of results from a study that suggested that students who were more satisfied with their learning experience previewed an assignment page well before it was due (Henrie et al. 2015).

All log data variables under a given structuring method were correlated to cognitive and emotional engagement scores from the self-report survey using bivariate Pearson correlation analysis, based on whether log data and engagement scores meet the statistical assumptions of data independence, normal distribution and linearity. Table 3 describes the different correlational analyses conducted for a given activity at a specific time point. Strength of relationships between specific log data variables and cognitive or emotional engagement were explored by considering the strength and statistical significance of correlations. This was done to understand how log data is uniquely related to cognitive or emotional engagement. Strengths of correlations were also considered to determine which log data granularity level was most useful for studying activity-level student engagement.

Table 3 Correlation analyses to be done for data from a given activity at a specific time point
Full size table

Previous research in comparing self-report student engagement data to other measurement methods have generally found low to moderate correlations (r = 0.15–0.43; see Fredricks et al. 2005; Fredricks and McColskey 2012; Skinner et al. 2009). These comparisons were between student self-report and teacher observations of student engagement. The more internal aspects of student engagement, like emotional engagement, tended to have lower correlations in these studies (0.15–0.20). This is likely due to the challenge of accurately identifying someone else’s emotions through observation (Skinner et al. 2009). In our review of the literature, we were unable to find reported correlations between self-reported engagement and other types of student engagement data. In order for log data to act as a reliable proxy for students’ self-reported cognitive and emotional engagement, a strong correlation between the log data and survey responses would need to exist. We hypothesized we would find stronger correlations between self-report and log data than what was found with teacher observations because we relied on data of students’ own behavior rather than the observations of others.

Results

Review of descriptive statistics

Of the 220 students who consented to participate in the study, only 155 completed at least one of the three student engagement surveys. An additional two students dropped the course before the end, which removed their log data from the system. This left data for 153 participants, for a participation rate of 70%. From the remaining 153 participants, the response rate varied across time points. Time point 1 had an 89% completion rate, time point 2 had a 71% completion rate, and time point 3 had a 76% completion rate. Maximum Likelihood Rubust (MLR) estimation was used to estimate missing data on the survey. The survey items showed good scale reliability within constructs and across time points (see Table 4).

Table 4 Scale reliability (Cronbach’s alpha) for cognitive and emotional engagement across time
Full size table

Checking statistical assumptions

We reviewed the distribution and presence of outliers for both the log data and the survey data to determine whether the data met the assumptions for the Pearson’s correlational analysis. About half of the variables had normal distributions, while others did not. For the variables that did not have a normal distribution, Spearman’s nonparametric correlational analysis was used instead of Pearson’s. Some variables also had significant outliers. These outliers were removed so as not to bias the results. We also investigated scatter plots for nonlinear relationships and found nothing of concern.

Correlation analysis

Students’ cognitive and emotional engagement survey scores were each correlated with the LMS log data variables. Because of the number of relationships being tested, the likelihood of committing a Type-1 error increases. To reduce the chance of this error, corrections can be made to the alpha value (α). One accepted approach for when a large number of tests is being conducted is the false discovery rate (FDR) correction (Benjamini and Hochberg 1995). Using this approach, the smallest p value has the most stringent alpha level, with increasingly less conservative alphas being applied to subsequent p values. P values are considered not statistically significant once the first adjusted alpha is exceeded. To obtain the alpha for the first smallest p value, FDR is calculated as follows: α*1/n tests conducted. The alpha for the second smallest p value is calculated as α*2/n tests conducted, and so forth. In this study, there are 180 correlational tests observed (2 engagement variables and 10 log data variables across 3 courses and 3 time points). Taking 0.05 alpha value as the base statistical significance criterion, the lowest observed p value must be less than 0.000278 to be considered statistically significant.

Overall, there were very few meaningful correlations between survey scores and log data. The highest correlation was in the educational technology 1 course between cognitive engagement and time spent on Learning Pages at time point 2 (r = 0.377). That correlation also had the smallest p value (0.00086), which did not meet the first FDR adjusted alpha value of 0.000278. All remaining correlations are considered not statistically significant. There were no meaningful correlations between the log data variables and student engagement survey scores for the history or educational technology 2 courses. Tables 5, 6, and 7 show correlation data for each of the three courses.

Table 5 Correlations between log data and associated student engagement survey scores from a history course
Full size table
Table 6 Correlations between log data and associated survey scores from educational technology course 1
Full size table
Table 7 Correlations between log data and associated survey scores from educational technology course 2
Full size table

Overall, the strength of the observed correlations indicates the absence of a simple linear relationship between the amount of activity and time spent on the LMS and cognitive and emotional engagement. Examining scatterplots between log data variables and survey scores revealed possible clusters based on unobserved variables that might better help explain the relationship between log data and survey scores (see Fig. 1). For example, students who are experiencing frustration may be spending more time trying to understand an assignment and experience a decrease in their emotional engagement.

Fig. 1
figure 1

Scatterplot of emotional engagement and total time spent at time point 1

Full size image

Discussion

The purpose of this study was to determine whether LMS log data could be used as a proxy for student engagement survey scores. Students’ cognitive and emotional engagement are essential for deep learning, persistence, and satisfaction. Furthermore, this internal energy ultimately takes a physical manifestation as learners engage in learning activities. We have used self-report data to measure students’ cognitive and emotional engagement. There is a significant cost in collecting engagement data using this method: namely that learning is disrupted in order to take the survey, and it requires the cooperation and contribution of students to obtain the data. Our hypothesis was that LMS log data would stand as a useful proxy of students’ emotional and cognitive engagement as measured through self-report. The results of our study to test this hypothesis were mixed.

Overall, we found no statistically significant correlations between the student engagement survey data and the LMS log data. There are several possible reasons for this. For example, our method of collecting the log data may have impacted the results observed in this study. We chose to use log data collected within 7 days from the assignment due date, with the assumption that most, if not all, learning activity was occurring during that time. Henrie et al. (2015) found that most activity occurred within 24 h of the due date. However, looking at descriptive statistics, it was possible that students in this study did their work well before 7 days before an assignment was due, or didn’t work on the assignment until after the due date. Depending on how much this occurred, this could affect our ability to detect a significant relationship between survey data and log data.

We chose to use due dates as the cut-off time as it was the more practical approach. Another approach would be to go off the date an assignment was submitted, particularly since one of the LMS pages in Canvas is a submission confirmation page. However, this would only work if the learning activity required something to be submitted to the LMS. This is not the case for discussion boards, or when viewing content pages or videos. One could go off the last date the page was visited, but it is possible that students revisit pages long after they actually engaged in the activity, such as when reviewing content before an exam. It would require significant manual work to review the log data and determine when engagement in the learning activity likely occurred. This approach would not scale well to large data sets. These are real limitations when the purpose is to study student engagement at the activity level using log data. Still, we are confident that our approach likely recorded much of the actual engagement in the learning activities we studied.

Another explanation for the results we observed is that students’ prior knowledge and abilities may have impacted the amount of time required to successfully complete the assignment. Some students may have needed less time than others to successfully complete the activity because of prior skill or experience. Students with less knowledge or skill may have needed substantially more time to understand and complete the assignment. This would prevent a strong linear relationship from existing between time spent or page views and survey data. Having more information about students’ prior abilities could better delineate the relationship between self-reported and observed engagement.

Comparing self-report to observational data may also have impacted the results we obtained. Studies that have compared self-report data to observational data have found mixed results. Prince et al. (2008) conducted a systematic review of 187 studies that compared observational data to self-report data in measuring physical activity in adults. They found that the correlation between the two sources of data ranged from −0.71 to 0.96, indicating that comparisons can be quite mixed. Elliot (2004) also found disparities when comparing survey data on students’ locus of control to data from interviews and observations. Using self-report data has several limitations, including the possibility that participants do not respond accurately out of shame, or because they do not understand the survey questions (see Elliot 2004; Gobert et al. 2015).

There are also limitations in treating Likert scale items, such as those used in the survey for this study, as interval data, where the amount of change between units is considered equivalent. Comparing data obtained through Likert scales, where the data is considered “ordinal at best” may not match well to data that is interval, such as time spent on or page views of an assignment (Fulmer and Frijters 2009). Significant effort was put in to develop a valid measure of student engagement. The measure we used was founded in existing engagement research and scale development (see Henrie et al. 2016). Our model of students’ cognitive and emotional engagement fit the data well (RMSEA = 0.050, CFI = 0.985, TLI = 0.971, SRMR = 0.033). If we assume that the self-report instrument is a valid measure of student’s cognitive and emotional engagement, then this study gives good evidence that LMS log data would not make a good proxy measure. Other types of log data may be useful to explore. For instance, it is possible to plug in external tools into some LMSs (such as Canvas) for mouse tracking. Baker et al. (2012) had better success when comparing log data obtained from an intelligent tutoring system to data obtained from human observers in detecting student engagement. D’Mello and Graesser (2012) had similar results in their work in comparing log data from an intelligent tutoring system to data collected from physiological sensors on students’ affective states. These methods hold promise, but assume that internal aspects of engagement, such as emotion, are being validly measured through observation or physiological detection. Future research needs to continue to address the challenges of comparing data obtained through different methods.

Interest in educational data collected from computer systems, big data, and learning analytics is growing (Bienkowski et al. 2012; Bodily et al. 2017; Ferguson 2012; Siemens 2013). This data has helped us explore learning in unimaginable ways. It is an abundant source of in-the-moment activity data with potential to help us better understand the phenomenon of learning with minimal interference to the student. Future exploration is needed, however, to understand the value of this data. In terms of using LMS log data to inform us about cognitive and emotional engagement, we came across significant limitations and challenges. In-the-moment behavior captured through log data may be more complex than we realize. Simply spending more time or having more activity on an assignment does not necessarily mean positive student engagement. Other factors need to be accounted for to better understand what it means to be effectively engaged in learning, such as previous knowledge and abilities, motivation to learn, or level of confusion or frustration. The emerging field of data analytics is a potential approach to studying these complex relationships. Further work with other methods for measuring student engagement, like mouse tracking, physiological instruments, or human observers may also yield valuable results.