Keywords

1 Introduction

For the past decade, the field of learning analytics has faced challenges in accurately aligning trace data, including unobtrusive data sources, with learning processes (Gašević et al., 2015). Learning is a complex phenomenon, and the retrospective analysis of behavioural trace data can provide only limited insights into students’ learning processes. Despite the increased adoption of technologies in education, very few studies can empirically demonstrate the impact of learning analytics on student learning (Dawson et al., 2019). Ultimately, the goal of education is to prepare and develop the necessary skills, knowledge, and capabilities of individuals for productive participation in society. This requires a solid foundation in knowledge and skills, as well as the development of personal and social competencies, such as critical thinking, creativity, and problem-solving. The skills for learning in uncertainty or building sensemaking capabilities are increasingly necessary for future education models. The chapters in this section unpack and highlight the constraints and priorities for future research and allude to new ways of using unobtrusive data to better inform teaching and learning practice. The following section first summarizes the commonalities among the presented works before challenging readers to reflect on missing topics and areas for future discussion. The commentary aims to bring forward perspectives on using unobtrusive data to improve teaching and learning practices.

2 Critical Overview of the Chapters

Collectively, the chapters demonstrate the opportunities afforded by analysing unobtrusive data sources. Chapter 2 by Zheng et al. explores an under-researched area in learning analytics by focusing on measuring emotion dynamics. The research literature demonstrates a clear association between emotion, motivation, and feedback. As such, the chapter delves more deeply into the earlier framing established by Winne. Zheng et al. explain how a learner’s emotional state changes over time and in response to the learning context and situation. In short, emotions are not static and fluctuate from moment to moment, from context to context. The authors first present a classification system of emotional dynamics characteristics, namely, emotional variability, instability, inertia, cross-lags, and patterns. In so doing, the authors identify some of the methods for detecting emotions in a non-intrusive way, such as emote aloud, facial and vocal expressions, language and discourse, and physiological sensors. At this point, unobtrusive observation data morph into what could be termed multi-modal data or multi-modal learning analytics.

The ability to effectively make sense of information and solve problems is essential for all learners. However, education has struggled to develop efficient and reliable measures of problem-solving skills, particularly in pedagogical models that involve social dynamics and complex processes. In Chap. 3, Wang et al. report on the use of log data analytics to study problem-solving processes in simulation-based learning environments. The authors examine how features extracted from log data can predict problem-solving outcomes and specific problem-solving practices. The results indicate that the deliberate use of pauses during problem-solving is a crucial feature associated with problem-solving competencies. In this context, the use of pauses as an intentional teacher practice can also be seen to promote metacognition. As framed by Flavell (1979), the concept of metacognition involves both metacognitive knowledge and metacognitive regulation, with the latter involving the ability to manage one’s thinking processes. The use of deliberate pauses, along with additional feedback and direction, can support metacognitive regulation and, therefore, the development of problem-solving skills.

In Chap. 4, the authors present a case study to measure leadership skills in a workplace learning context. Assessing complex capabilities or so-called “soft” skills is challenging and context-specific. Most studies to date reporting on the assessment of skills such as leadership tend to employ introspective methods such as self-reported questionnaires and inventories. The chapter presented here clearly details how unobtrusive measures such as learner trace data can offer more scalable and reliable assessments. Interestingly, the authors developed an automated machine learning classifier to extract measures from reflective artefacts incorporated within the learning tasks associated with the case study. This aligns with Winne’s call for more information-enriched data to better interpret the learning events for subsequent alignment with theory.

The increased adoption of education technologies has led to an expanse of research mining user interactions to predict learner outcomes, attrition or SRL skills. In Chap. 5, Choi et al. examine the opportunities and challenges in measuring motivational constructs using trace data. The authors draw on the COPES (Conditions, Operations, Products, Evaluations, and Standards) model and how trace data can inform how learners engage in multiple cycles of SRL events. Here, the authors note how clickstream data can be used to understand goal changes over time and identify the external conditions preceding a change in a learner’s motivational state. As flagged by Choi and Winne, there is a lack of prior work seeking to produce valid measures of motivation in learning analytics. The authors suggest using the Evidence-Centred Design (ECD) framework to identify a construct’s critical attributes and their operational definitions. The article provides an example of using the ECD design pattern to distinguish between performance-oriented and mastery-oriented goals.

Finally, in Chap. 6, Winne presents prior work on the COPES (Conditions, Operations, Products, Evaluations, and Standards) model of SRL to illustrate how underlying information can bring meaning to the learning events and operations students undertake. Winne argues that the inclusion of information-enriched data can better support the interpretation of specific learning events. In essence, Winne posits that understanding or supporting the development of SRL requires information or knowledge of the discrete tasks and standards presented to learners. While this is only a partial component of the overall story, it is integral for aligning trace data with SRL processes. In Winne’s terms: “Information-enriched data lend meaning to learning events beyond whether an event occurred”.

There are many similarities and alignments in the presented chapters. All chapters cover the relationship between user behaviour and learning intention, from identifying emotional states associated with learning activities to identifying problem-solving skills or complex capabilities. The use of unobtrusive observation data in education calls for greater interdisciplinary research. All chapters reflect this interdisciplinarity. The chapters also highlight the inadequacies, or at least the limitations, of current research methods.

3 What Is Currently Missing in the Modelling of Learning Processes?

There are many strengths to the presented chapters in this section, and the following is by no means intended as a critique of the presented works. More so, the commentary is a reflection on the areas that can be used to complement and extend the current suite of chapters.

As detailed in all the chapters presented in this section, the practice of education has undergone significant change over the past decade. The recent introduction of generative artificial intelligence into education signals the potential for further disruption. Despite changes in the delivery of education, technology adoption, or the need to assess complex capabilities, the importance of feedback remains a consistent theme for supporting student learning. Contemporary research has shifted conceptions of feedback from that of a product to a process (Winstone et al., 2017). While all chapters demonstrate the affordances of unobtrusive observations to measure and support student learning, how such data can also support student agency and feedback remains a challenge. The provision of supportive feedback should be seen as a dialogic process that can develop student feedback literacy.

Unobtrusive data are commonly used for developing student- and teacher-facing learning analytics dashboards to support the development of SRL. As Valle et al. (2021) demonstrated in their systematic review of Learning Analytics Dashboards (LADs), there remains a lack of alignment between stated evaluation measures and target outcomes. Similarly, Matcha et al. (2020) undertook a systematic literature review of learning analytics dashboards (LADs) to determine the impact on learning and teaching. The results indicated that existing LADs are not grounded in learning theory, do not support metacognition, do not provide information about effective learning tactics and strategies, and have limitations in their evaluation.

While there are clearly opportunities to bring unobtrusive data sources into line with feedback, there is much work on how the “pipeline” from course outcomes to design, learning activities, assessment and feedback collectively inter-relate. For instance, Zamecnik et al. (2022) developed a LAD to support collaborative learning and explore how student teams interact and engage with the provided information. The study showed significant diversity in how team members interact with the information depending on their allocated roles. For example, team leaders were noted to be more engaged with data that monitored team collaboration. In this regard, the actual LAD design reflects more event-level information for students and the gap between presented data and intended outcomes is very close. In contrast, many LADs present a significant gap between discrete engagement behaviours and understanding of individual learning progress. While LADs can help teams self-regulate, and instructors can monitor team behaviours, there is a need for further research to investigate student understanding of their learning data and how this can be used for developing feedback literacy. This challenging space was not extensively covered in the presented chapters and is one significant area for future work.

Unobtrusive observations have a rich history in the field of Intelligent Tutoring Systems (ITS). In short, ITS are computer-based systems that provide adaptive learning for students in specific knowledge domains. The goal of ITS is to support learning progress that is tailored to each student’s unique strengths and weaknesses. Shute’s (2011) concept of stealth assessment was spawned from work in ITS and involves using data generated from students’ interactions with digital learning environments to assess their knowledge, skills, and abilities. The concept of stealth here is analogous to unobtrusive observations. Importantly, as framed by Shute (2011) and in the preceding chapters, the goal in analysing these forms of naturally occurring learner data is to increase the frequency and opportunities for formative feedback.