Keywords

1 Introduction

The technology-enhanced learning (TEL) ecosystem is becoming increasingly complex, given the inclusion of new Information and Communication Technologies (ICTs). The COVID-19 global crisis has amplified this complexity, making it evident that ICTs will play a major role of the future of education at all levels. Indeed, all students will need at least some access to digital contents and tools from their homes and in the classroom. Thus, to address local and national restrictions and recommendations, hybrid learning spaces (Cohen et al., 2020) are and will be present due to the need for mixing teaching and learning modalities and spaces.

The affordances of ICTs are often powerful and presumably make teaching and learning more efficient and effective (Linn and Eylon, 2011), easing the life of the involved stakeholders. However, such complex TEL ecosystems will demand an extraordinary effort from the teachers as they will need to design appropriate learning scenarios, manage them under real-world conditions, and make decisions for the most effective pedagogical interventions. In other terms, teachers will face the challenge of carrying out the design and orchestration of the learning and teaching process in increasingly uncertain and complex TEL ecosystems (de Quincey et al., 2013; Goodyear, 2015).

Learning analytics (LA) has emerged in the last decade as a powerful means to support teachers and other stakeholders (e.g., researchers, instructional designers, technology developers, administrators, and students) to navigate the complexities of teaching and learning in TEL ecosystems. The LA field deals with the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs (Siemens, 2012). More concretely, LA may provide support for the complete cycle of Teacher Inquiry into Student Learning (TISL) (McPherson et al., 2016) and evidence-based decision-making. In spite of all its advances and contributions, LA has not yet delivered on its promised potential, since the main LA proposals have not been able to provide sufficient actionable insights to the teachers (Sergis and Sampson, 2017) in their role of designers and orchestrators of complex TEL ecosystems (Gasevic et al., 2017).

Human-Centered Learning Analytics (HCLA) (Buckingham Shum et al., 2019), a significant trend observed in recent literature, claims that a human-centered perspective in LA might overcome several obstacles toward actionable tools and practices (i.e., LA solutions). For example, some HCLA guidelines suggest bringing teachers in the loop through intensive inter-stakeholder communication (Prestigiacomo et al., 2020); carefully exploiting the connection between learning design, monitoring, and learning analytics (Rodríguez-Triana et al., 2015; Maldonado-Mahauad et al., 2018); following a balanced design of artificial intelligence and human agents (Goodyear and Dimitriadis, 2013); or embedding learning theory through the teachers’ technological pedagogical content knowledge (TPACK) (Wiley et al., 2020).

In this chapter, we investigate the role of LA solutions in supporting an evidence-based approach to teaching and the importance of inter-stakeholder collaboration for making design decisions in such complex TEL environments. Focusing on teachers as key LA stakeholders, designers, and orchestrators, we study how LA can be designed to position teachers as designers of effective pedagogical interventions and orchestration actions. To address this overall goal, we adopt a human-centered design (HCD) perspective of LA, taking advantage of existing knowledge in the design and human-computer interaction (HCI) communities while considering the specific characteristics of learning and teaching. With this perspective, we offer and illustrate HCD principles to guide the process of designing and orchestrating actionable LA solutions.

The rest of this chapter is structured as follows: Sect. 2 provides an extensive description of the most relevant concepts and research lines regarding learning design, orchestration, learning analytics, and HCD for LA. Our principles for the HCD process are described in Sect. 3. Section 4 describes two illustrating examples of how the HCD principles can be implemented. Finally, Sect. 5 discusses open issues, draws the main conclusions, and points at future research and development directions.

2 Background

2.1 Current Approaches for Designing for Learning, Analytics, and Orchestration

Teachers (supported by other stakeholders such as researchers, system developers, and instructional designers) need to design and orchestrate the increasingly complex TEL environments. As Goodyear (2015) suggest, one can design for the social architecture (the groupings of students that are most appropriate), the tasks to be performed (not the activities that depend on the learners’ actions and decisions), and the physical and digital environment (the tools that will be employed, the artifacts that will be created and evolved throughout the activities, and the resources that are available). The design outcomes should be effective and efficient processes for making configurations, monitoring learner performance and engagement, executing orchestration actions, and making and implementing decisions for redesign and interventions.

On the other hand, a decade of research in LA has produced significant outcomes, especially in mining patterns of student behavior based on trace data (Luckin et al., 2010), deriving predictive models regarding performance and dropout (Ranjeeth et al., 2020), and providing dashboards to make sense of the behavioral data (Kali et al., 2015). However, most research and development efforts have been centered on exploiting powerful data by applying well-known artificial intelligence (AI) and data science (DS) methods to new datasets of clickstreams, mainly serving administrators and researchers. More impact is being sought to enable the main stakeholders, i.e., students and teachers, to take advantage of actionable insights provided by meaningful indicators and LA tools in authentic contexts (Hunziker et al., 2011). Thus, there is an urgent need to study how LA solutions can be designed for effectively supporting pedagogical interventions and orchestration actions.

Yet, a critical question arises: Should the technology (e.g., AI) substitute teachers or mediate orchestration through tools that balance the orchestration load (Sharples, 2013)? For example, some tools may hold substantial agency by automatically intervening and regulating the learning activity, like it occurs with intelligent tutoring systems (ITS). By contrast, LA tools may mirror rather than directly orchestrate what occurs in the TEL ecosystem. Such tools can recommend orchestration and redesign actions or help teachers to monitor the learning activity and make informed decisions (Soller et al., 2005). However, finding the right balance between humans and digital tools with respect to the orchestration load and agency can be challenging (Goodyear and Dimitriadis, 2013). Eventually teacher augmentation might be pursued to bring such balance (An et al., (2020)), since scholar design knowledge can be embedded in tools and can complement both the tacit and explicit design knowledge of teachers, typically expressed through teachers’ TPACK (i.e., their joint knowledge on content, pedagogy, and technology) (Knight et al., 2020). Therefore, how can the different stakeholders form part of a design team, in which the different types of expertise can be fully considered? We argue that teachers (and learners) can serve as designers (Jørnø and Gynther, 2015; Gasevic et al., 2019) and, as such, they should participate in not only the design and orchestration of the teaching and learning processes but also the associated support tools.

In recent literature, several design principles and approaches toward effective LA practices and tools have been proposed. These principles and approaches consider the role of the involved stakeholders and take advantage of the relation between learning design, learning analytics, and learning environment. For example, Beer et al. (2019) suggested that educational theory and the characteristics of the learning task should provide guidance for design aspects in learning analytics including data selection, data analysis, and implementation. Wise and Vytasek (2017) proposed three design principles within their Learning Analytics Implementation Design (LAID) framework on how LA solutions might be designed and implemented in practice. The LAID principles are based on an assertion that LA and learning design are intimately intertwined (Rodríguez-Triana et al., 2015; Maldonado-Mahauad et al., 2018). On the one hand, LA may provide evidence that informs about the effectiveness of learning design and supports the Teacher Inquiry into Student Learning process, i.e., provide them actionable insights on how to orchestrate and redesign. On the other hand, learning design can frame what are the analytics to be generated, guide the way analytics may be meaningfully interpreted, and eventually inform and recommend teachers and students to take decisions.

Accordingly, Wise and Vytasek (2017) suggest coordinating (conceptually and logistically) the LA solution with respect to the overall learning design so that appropriate data and indicators are selected for generating analytics that can be understood by teachers. They also suggest, albeit with caution, comparing learner metrics against an absolute value set by the learning objectives or a relative tendency across courses or across different activities of the same learner. Furthermore, they suggest customizing the LA system to the needs and profiles of its users, through either an adaptive LA system (where AI agency becomes predominant) or a solution that can be configured based on the preferences of the users (where the engagement of the teacher/student is crucial in all phases of the design, development, and enactment phases).

As mentioned above, dominant LA solutions have been mostly built using knowledge from Data Science. Considering limitations of those LA solutions, Gašević et al. (2015) proposed a consolidated model in which learning analytics lies at the intersection of learning theory, design, and data science. These authors particularly emphasize the critical role of educational theories for designing actionable LA solutions that can be relevant to the learning task at hand and meaningful to teachers and students. In the same vein, Reimann (2016) suggests, more is needed than just data to discover meaningful relations and Echeverria et al. (2018) suggest, in the title of their paper, “Let’s not forget: Learning analytics are about learning.” On the other hand, design has not been as deeply explored as data science and theory, and the amalgamation of the three is far from being mature. But learning theory, design principles for the LA solution, or data science methods may not be sufficient if we do not define principles that govern the process for designing LA solutions that can be orchestrated and adopted in practice.

Addressing this, Prestigiacomo et al. (2020) argued for the need for a strong inter-stakeholder communication and provided instruments for expressing needs and knowledge. Their analysis of the obstacles of LA adoption from the orchestration lens led to the recommendation of using the OrLA (Orchestrating Learning Analytics) framework to guide the LA design process. Thus, effective orchestration support, including LA solutions, should enable teachers to design and configure the learning environment, monitor the learning activities, and become aware of what is going on. This suggests the need for participatory and co-design methods that could be used to imbue LA solutions with the needs and preferences of the stakeholders while taking into account all practical classroom constraints as well as the theories regarding learning and teaching.

2.2 Human-Centered Design for Learning Analytics

The term Human-Centered Learning Analytics (HCLA) has recently emerged in the LA community of research to refer to the adoption and adaptation of design practices, well-known in HCI, with the purpose of engaging educational stakeholders, such as teachers, students, and educational decision-makers, in the design process of data-intensive educational innovations (Buckingham Shum et al., 2019). The main paradigm shift proposed by design communities, such as participatory design (Schuler and Namioka, 1993) and co-design (Bannon and Ehn, 2012), is to move from designing for users to designing with people as equal partners in the design process (Sanders and Stappers, 2008). The aim is to make the most of the creativity of designers and people not formally trained in design, but that can have other relevant types of expertise, by letting them work together across the whole span of the design process.

Therefore, HCD approaches are relevant for creating LA interfaces aimed at effectively supporting teachers and students in making decisions in terms they can make sense of and use. However, work in this area is embryonic in LA, with a growing number of pioneering researchers advocating for rapid cycles of prototyping with teachers (e.g., Mangaroska and Giannakos, 2018) and conducting interviews with students to generate a deeper understanding of their perspectives on data analytics (e.g., Mavrikis et al., 2019). Goodyear and Dimitriadis (2013) were among the first researchers in adapting various generative (or ideation) tools and co-design techniques to identify teachers’ data needs and design prototypes of awareness and orchestration tools to be used with ITSs in the classroom.

Teachers have been the most commonly involved group of stakeholders in LA co-design studies thus far (Buckingham Shum et al., 2019). For example, Ahn et al. (2019) established partnerships with teachers to design an LA dashboard that meets the local needs of a particular educational context. Similarly, Dillenbourg et al. (2019) discussed how participatory semi-structured interviews can be organized to engage teachers in long-term LA projects. Martinez-Maldonado et al. (2019) organized participatory workshops with teachers as an entry level for them to learn to use authoring tools in the context of an ITS that provides automated feedback. Wise and Jung (2019) combined LA interface walkthroughs and transcript analysis to generate understanding of how teachers can effectively make sense of student data and, thus, designed a teacher dashboard accordingly. They proposed a process model of how instructors may use LA, in which they connect sense-making with pedagogical response, iteratively and bidirectionally, going from questions of interest to reading data and explaining patterns, taking action, waiting and seeing, or even reflecting on their pedagogy, before checking the impact of their actions. Similarly, Mor et al. (2015) proposed a method to run participatory workshops in order to elicit data needs from pre-service school teachers to understand what kinds of analytics can effectively support their evidence-based teaching practices.

Some examples of LA design projects that engage various stakeholders besides teachers have also started to emerge. For example, Prieto et al. (2019) developed a tool to facilitate design conversations between teachers and students, using a learner journey technique, to jointly identify the form and opportunities for providing automated feedback to students in the context of nursing education. The same authors developed a deck of design cards to facilitate co-design sessions by scaffolding the conversations and addressing potential power inequalities by ensuring all stakeholders have a voice in the design decisions (Alvarez et al., (2020)). This approach is similar to that of Vezzoli et al. (2020) who proposed using inspiration cards to engage teachers in early stages of the design process of an LA system. HCLA conceptual and empirical work particularly aimed at giving students an active voice in the LA design process are also starting to emerge (Prieto-Alvarez et al., 2018; Prieto et al., 2019).

In summary, these studies demonstrate the growing interest in bringing HCD approaches in LA. However, most of these papers have reported local projects and particular solutions that can certainly inspire other researchers to organize co-design sessions in their institutions.

The next two sections of this chapter conceptualize the process of designing and orchestrating actionable, human-centered LA solutions, through the proposal and discussion of principles, and their illustration using case studies in authentic contexts.

3 Principles for the Process of Human-Centered Design

After providing a brief view of what have been the main trends of LA research and based on the aforementioned literature survey and authors’ first-hand experience in co-designing LA innovations with teachers and other stakeholders (to be presented below), we can conceptually distil three basic HCD principles to govern the process of designing actionable LA solutions:

  1. 1.

    Agentic positioning of teachers and other stakeholders

  2. 2.

    Integration of the learning design cycle and the LA design process

  3. 3.

    Reliance on educational theories to guide the LA solution design and implementation.

The three principles reflect a human-centered perspective, since learning design and orchestration are typically carried out by teachers and instructional designers and educational theories are produced by researchers.

3.1 HCD Process Principle #1: Agentic Positioning of Stakeholders

The primary objective for the agentic positioning of relevant stakeholders during the design process is to facilitate the exchange of expertise and the development of a mutual understanding of each stakeholder’s priorities, values, and constraints. In other words, the voices and expertise of all relevant stakeholders should be considered and leveraged, respectively, in the LA design process. However, a major challenge in meeting this objective is facilitating this communication. In some cases, this challenge can be managed by careful planning to permit meetings in which all stakeholders can engage synchronously in time and/or space. In other cases, stakeholder meetings can occur asynchronously through communication media, whether digital or analog. The stakeholder forms described by Prestigiacomo et al. (2020) can support such inter-stakeholder communication, as they guide both the content of information exchange and the sequence of stakeholders’ responses. The work on human-centered design presented in Sect. 2.2 supports this principle, together with the literature review that motivates the OrLA framework (Prestigiacomo et al., 2020).

3.2 HCD Process Principle #2: Integration of the Learning Design Cycle and LA Design

Asensio-Pérez et al. (2017) describe the learning design cycle as a three-phase process consisting in rounds of creation, orchestration, and assessment (Fig. 1). The cycle begins with the creation of specific tasks, intended social structures, artifacts, and resources to facilitate the desired learning process. During the orchestration phase, the learners’ engagement with these elements is monitored, regulated, and scaffolded with the goal of supporting the desired learning. Learners’ artifacts are then assessed to determine how the learning design can be redesigned or reinstituted to achieve the desired learning.

Fig. 1
figure 1

The three phases of the learning design cycle: creation, orchestration, and assessment

Full size image

Integrating the process of LA development with the learning design cycle can enable LA solutions to effectively support Teacher Inquiry into Student Learning and evidence-based decision-making. To illustrate, after creating the learning design, specific elements of the design are identified as targets for the LA tool (Fig. 2, 1). During the orchestration phase, the LA tool is implemented. The selected targets feed data into the LA tool (Fig. 2, 2a), and the subsequent analysis by the LA tool supports the understanding of the learning taking place and informs the pedagogical interventions and orchestration actions needed to optimize that learning process (Fig. 2, 2b). The output from the LA tool can also support the assessment phase of the learning design cycle, by providing insight into the effectiveness of the targeted elements in facilitating the desired learning outcomes (Fig. 2, 2c). This principle was inspired by the related work described in Sect. 2.1 and especially by Rodríguez-Triana et al. (2015), Maldonado-Mahauad et al. (2018) and Wise and Vytasek (2017).

Fig. 2
figure 2

The integration of LA development into the learning design cycle. (1) LA design, learning design elements selected as targets for LA solution, (2) LA implementation: (a) data from LA targets is analyzed by the LA tool, and the resulting LA informs, (b) orchestration, (c) and assessment

Full size image

Achieving the alignment of these two processes can be complicated by the fact that typically no single stakeholder is responsible for all aspects. For example, a system developer may design an LA solution for a learning design that a researcher or instructional designer creates and a teacher orchestrates. However, the challenges associated with aligning the two processes can be mitigated by implementing HCD process principle #1, namely, increase the likelihood that the voices from all relevant stakeholders be considered in the LA design process, regardless of the configuration of stakeholder responsibilities.

3.3 HCD Process Principle #3: Educational Theory Guidance

For this principle, we assume that the learning design has been developed in accordance with an educational theory (i.e., a theory of learning or research-based professional standards). As such, the educational theory that guides LA design and implementation should be the same as that used for the learning design. During the LA design process, educational theory informs the selection of data and extracting metrics that can be associated with higher-order meaningful constructs relevant to the learning design at hand. Moreover, educational theory can inform how to use the LA to generate actionable insights and inform orchestration actions and help to identify the goal toward which learning and its environment are optimized (i.e., learning design redesigns). A potential challenge in meeting principle #3, particularly when viewed in light of principle #2, is when the learning design is created by stakeholders without intimate knowledge of educational theories. In such case, a knowledgeable stakeholder can retroactively apply an educational theory to the learning design to inform LA data selection and analysis. However, LA targets that do not align with the theory may need to be excluded from the candidate pool to realize the benefit of this principle. The work by Gašević et al. (2015) and Reimann (2016), presented in Sect. 2.1, has mainly motivated the proposal of this principle.

In the next section, we describe two studies that illustrate how to implement these HCD process principles during LA design.

4 Illustrative Studies

4.1 Study 1: A Performance Analysis Tool for an Online Middle School Science Unit

This study illustrates how the three HCD process principles for designing effective LA solutions can be implemented when a learning design is created by multiple stakeholders. Specifically, it is a design-based research (DBR) study, consisting of a 2-year partnership involving three researchers, three system developers, and five middle school science teachers. The study goal was to develop an activity-centered LA solution (Klerkx et al., 2017) for a Web-based Inquiry Science Environment (WISE) unit on global climate change. Given its call for a design process that is participatory and theory-grounded (Sandoval and Bell, 2004), DBR functioned as a scaffold for implementing HCD principles #1 and #3. To further implement principle #1, the study methods included inter-stakeholder dialogues (Prestigiacomo et al., 2020) for which the researchers served as liaisons between stakeholder groups. While the unit activities were created by the researchers, teachers designed and interleaved their own offline activities to complete the science instruction for their students. Thus, the complete learning design, the WISE unit plus the teacher-provided offline instruction, was co-designed. Therefore, inter-stakeholder dialogue (Prestigiacomo et al., 2020) was essential for developing an LA solution that incorporated the design knowledge of each stakeholder. These in-person, inter-stakeholder discussions were guided by the three LAID principles (i.e., coordination, comparison, and customization; Wise and Vytasek, 2017), which helped stakeholders attend to issues relevant for designing an LA solution that could be effectively implemented in classrooms.

The researcher-teacher meetings focused on issues related to all three LAID principles, such as presenting and explaining the unit’s learning design and underlying theory of learning, understanding teachers’ goals and priorities for assessing and supporting student learning, and discussing the impact and influence of the LA solution on teaching and learning. From these meetings, the stakeholders decided that the LA tool would provide teachers with data related to seven multiple-choice items that engaged students in distinguishing their ideas about how the sun warms the earth (Fig. 3). More specifically, the LA tool would provide teachers with aggregated and individualized data on students’ answer patterns for the seven multiple-choice items. These unit items were chosen because they both aligned with the focus of teachers’ offline activities and functioned as measures for the higher-order construct targeted by the learning design, namely, distinguishing ideas.

Fig. 3
figure 3

Example of a multiple-choice item from the WISE Global Climate Change unit that was selected as a target for the LA tool. Note: SR solar radiation

Full size image

The unit’s learning design was designed in accordance with the Knowledge Integration (KI) pedagogical framework (Koehler et al., 2013), which operationalizes the constructivism theory of learning. This theory holds that learners construct new knowledge by building on their prior ideas. In a KI-based learning design, student’s topic-related ideas are first elicited, after which students are provided with opportunities to discover new ideas, make distinctions among the ideas, and finally make relevant connection between ideas. Prior research identified the distinguishing ideas step as particularly challenging for students to engage in Vitale et al. (2016) and for teachers to support (Wiley et al., 2019).

Integrating the first cycle of LA design with the unit’s learning design cycle allowed the LA tool to serve as an evaluative tool for how well the unit’s learning design was supporting the desired learning (ref. Fig. 2, 2a-c), which for this study was integrated knowledge of concepts related to global climate change. The LA revealed that students who did not correctly answer the multiple-choice items also did not heed the feedback to review the related simulation where they could discover the relevant ideas. This information provided the researchers with the insight needed to restructure the unit. They did so by placing the assessment items, which supported students in distinguishing ideas, on the same page as the related simulations, which facilitated the discovery of new ideas.

The second cycle of LA development was integrated into the learning design cycle for the offline teacher-created activities. During this cycle the “reseacher-system developer” meetings functioned prominently. These meetings focused on issues related to the coordination and customization principles, such as the researchers understanding the WISE system capabilities, the system developers understanding the objectives and priorities of the researchers and teachers, and workflow management for developing the LA artifact. From these meetings, the stakeholders decided to create an LA report as the artifact. Teachers received an LA report for each assessment item after completion by a majority of students (Fig. 4). Drawing on the principles for data storytelling (Echeverria et al., 2019), the analytics in the report were contextualized by presenting them directly beneath the question prompt, learning objective, and aligned science standard. This contextualization was designed to orient and remind teachers of the unit’s researcher-created learning design. Additionally, the LA report included a researcher-created hypothesis, called Researcher Insight, to explain the students’ performance and to identify their potential learning needs. In this cycle of the DBR process, the Researcher Insight was generated manually by the researcher based on the analysis of student work and unit navigation patterns using clickstream data. In the following cycle, data was fed automatically from the analysis module to the LA dashboard.

Fig. 4
figure 4

This is a reconstruction of the emailed LA report that was sent to teachers after at least 50% of students completed the associated multiple-choice item

Full size image

Since the LA solution aligned with the learning design knowledge of both researchers and teachers (i.e., aligned to unit items that measured constructs targeted by both researcher- and teacher-created learning activities), it was able to support teachers in designing and redesigning their orchestration actions and pedagogical interventions. For example, in one researcher-teacher meeting, a teacher described his LA-supported actions as follows:

I review the most common incorrect answer and have table talks and then classroom discussions about why students might have that as a misconception, why it’s a misconception, and why the correct answer is correct. For a couple of the questions, I have supplemented the classroom discussions with various simulations and videos to try and change the students’ understanding of the misconception. (Wiley et al., 2019, p.576)

Informed by the analysis presented in an LA report, another teacher decided to redesign his classroom instruction to implement more pre-activities that help students understand their background knowledge. This redesign highlights how the LA solution captured the researcher and teachers design knowledge, as this teacher’s redesign aligned with the theory used to design the unit, namely, eliciting students’ prior ideas to make them available for further knowledge development.

The actions that teachers took in response to the LA solution, while consistent in many ways with the design knowledge of the researchers, also reflected their individual TPACK. The freedom that teachers had to reconfigure the learning environment and scaffold students in accordance with their TPACK without conflicting with the design knowledge of the researchers and system developers illustrates the value of the three HCD principles shown in Sect. 3: agentic positioning of key stakeholders, integration of the learning design cycle and LA development, and guidance by a theory of learning.

4.2 Study 2: A Multimodal Reflection Tool for Healthcare Simulation

This study illustrates how meeting the three HCD principles for creating effective LA solutions occurred in close partnership with relevant stakeholders with the purpose of creating an LA tool that explicitly reflected the learning intentions of the educator. This involved a long-standing 4-year partnership with two healthcare researchers, six LA researchers, two teaching support staff members, three nursing lecturers, and various nursing undergraduate students representing diverse and intense stakeholder involvement. The goal of the study was to develop a reflection tool to be used to support team debriefing in nursing simulation (Martinez-Maldonado et al., 2015). These simulations involve face-to-face classes of 25–30 students led by one educator. The classrooms are simulated hospital wardrooms with high-fidelity patient manikins located on 5–6 beds. The educator commonly starts the class with some explanations, followed by students breaking into smaller teams. After the teams complete their simulations, the educator leads a class debrief. In this context, educators often create their learning designs based on clinical theory and national healthcare guidelines for the purpose of accreditation and for students to develop the graduate attributes they need to become registered nurses. We focus on one of such designs in which students are required to provide basic life support (BLS) to a simulated patient after he lost consciousness.

An initial set of co-design sessions involved inter-stakeholder communication using OrLA forms (Prestigiacomo et al., 2020) asynchronously for the healthcare researchers, LA researchers, teachers, and system developers to identify data and orchestration needs and how these data could be feasibly captured. The stakeholders identified multimodal sources of evidence educators could use to provide feedback to students. As a result, the learning space was instrumented using a combination of sensors and an annotation console that could be orchestrated by the teaching support staff members or the LA researchers. Additional co-design sessions were organized with educators and students to identify particular characteristics of the LA tool including graphical interface and interaction design requirements and the medium to be used. Techniques such as focus groups, learner journey-mapping, and rapid prototyping were used in facilitated sessions (Prieto et al., 2019). A visualization was created to provide feedback on students’ performance by highlighting errors (e.g., critical actions missing or performed in the wrong order) and delays using logged actions and positioning traces of each nurse (Fig. 5).

Fig. 5
figure 5

Team timeline highlighting errors observed during phase 2 of the simulation (BLS support) for one team of nursing students. Errors are highlighted using visual elements such as (a) a prescriptive title, (b) text annotations, (c) shaded areas, and (d) color encoding (orange and blue for errors and correct actions, respectively)

Full size image

A mapping was performed from low-level data to clinical constructs that educators and students could understand. For example, the higher-order construct targeted in the exemplar simulation corresponds to the effective performance of BLS. According to clinical literature (Holstein et al., 2019) and national guidelines (ANZCOR, 2016), four subconstructs were selected by the educator to assess students’ performance, such as opening patient’s airway, and partly modeled based on the positioning data and logged actions.

The educators’ learning design served to configure the LA tool for the interface to be aligned with these four subconstructs as learning goals. A data storytelling approach (Echeverria et al., 2019) was followed for making the learning goals explicit in the LA interface. Each learning goal is assessed against learners’ data (using rule-based algorithms) to automatically generate visual and textual elements to enable educators and students to understand whether the learning goal was accomplished and receive feedback on areas of improvement. For example, Fig. 5 presents one of such data stories for a team of two nurses who performed chest compressions (subconstruct 3) slowly and shallowly (Dollinger et al., 2019). The visualization is enhanced with text explaining to students the errors they made.

In this illustrative study, the voices of various relevant stakeholders were considered, first, to understand the data and orchestration needs of teachers and how the hybrid learning space could be instrumented with sensing technology with integrity and considering practical aspects that may affect orchestration (HCD principle #1). Teachers, students, and healthcare researchers were further involved in the design process of the tool and the strategies to embed the tool into the current teaching and learning practice. The alignment between the LA solution and the learning design was made explicit in the LA tool itself, based on the data storytelling paradigm, in which each learning goal established by the teacher is co-configured in the learning design phase for the tool to provide feedback via a combination of text and visual enhancements: data stories pre-configured by the teacher (HCD principle #2). Although in the study this preconfiguration was performed by the LA researchers, based on the outputs from the co-design sessions with teachers, this configuration can eventually be automated or be part of the responsibilities of a stakeholder in charge of the learning design. Finally, this case also shows how theory can guide the design and implementation of the LA solution (HCD principle #3). Although the theory the teacher explicitly considered in this example comes from clinical literature instead of educational literature, similar simulation-based pedagogical approaches are used in other educational areas and levels, beyond the healthcare sector.

5 Discussion and Conclusions

Learning analytics solutions may contribute to more effective and efficient design for learning and orchestration, allowing for informed decision-making, pedagogical interventions, and orchestration actions. However, learning analytics has not delivered yet up to its potential through the provision of actionable insights to the main stakeholders, i.e., teachers and students. A human-centered design approach for learning analytics has emerged in recent years, although it is still a toddler, aiming to bring together all relevant stakeholders through participatory design, co-design, design-based research, and research-practice partnerships. In this chapter we focused on the role of teachers as designers and their connection with researchers, system developers, and other stakeholders in the process of designing and implementing learning analytics solutions, i.e., tools and practices. We called for strong inter-stakeholder communication, and we proposed three human-centered design principles for learning analytics, which were illustrated through two case studies in authentic contexts. In both studies, teachers became active agents in the design process of the LA solution (HCD principle #1). The studies demonstrated how the voices from multiple stakeholders are needed not only to consider teaching and learning aspects but also to connect these with technical and practical requirements that can impose limitations on what can be achieved with the resources available. The studies proposed two different ways to integrate the learning design cycle and the LA design process (HCD principle #2), by enabling teachers to assess their learning design based on the analytics (study 1) or by imbuing the analytics with the pedagogical intentions stated in the teacher’s learning design (study 2). Finally, we also illustrated the power of educational theory for designing meaningful LA solutions (HCD principle #3). Study 1 demonstrated how a well-known theory of learning drove critical design aspects of the LA solution through the Knowledge Integration (KI) pedagogical framework. By contrast, study 2 illustrated a more specific instance in which clinical theory was embedded into a simulation-based learning pedagogical approach to drive both the learning design and the design of the LA interface. In sum, the proposed principles ask for stronger involvement and agency of the teachers, so that all voices of involved stakeholders can be considered, integration of the learning design cycle and the LA design process, and reliance on educational theories to guide the LA solution design and implementation. This way, targets can be defined based on the learning design and pedagogically sound theories, reflecting both scholar and practitioner design knowledge, so that meaningful analytics can be determined and appropriate support for interventions, orchestration, and redesign can be provided.

However, it is still necessary for the research community to move forward and address multiple issues in relation to the design and implementation of learning analytics solutions for complex technology-enhanced learning ecosystems. For example, sustainable adoption of HCD approaches requires that researchers and teachers embrace design methods effectively, stakeholders should ideally be involved in the design at institutional levels, and there is a need to upskill the LA community in generative methods, design thinking, and co-design methodologies. A question that can immediately emerge as a response is: Is it worthy to deal with all the complexity and the resource-intensive process of human-centered design, i.e., co-design and participatory design, to create analytics aimed at supporting human decision-making? The short answer is yes. Although it may initially seem that collaborative design sessions may be time-consuming, in the long term, the benefits of co-creating effective tools that address authentic challenges can reduce costs and offer much more value than trying to force the integration of poorly designed analytics into current practices. Sanders and Stappers (2008) explained how design approaches solely based on observing how users work cannot address the scale or the complexity of the challenges we face today. HCD methods are thus expected to become increasingly critical for designing LA systems to be embedded in the increasingly complex technology-enhanced learning ecosystems we have today. HCD methods can also help researchers, practitioners, and designers in keeping a balance between technical aspects and human factors in LA. For example, co-designing with teachers can contribute to increasing teachers’ agency as designers by considering their beliefs, attitudes, preferences, and knowledge. It can also enhance the technology, pedagogy, and content knowledge of teachers toward better orchestration and redesign and ultimately balance the role of the artificial intelligence and the human agents, toward an eventual augmentation of teachers and students. Although more empirical research is still needed to provide maturity to human-centered approaches in LA, the two studies described in this chapter are aimed at providing confidence in the potential benefits of involving critical stakeholders in the design process of LA systems to improve teaching and learning.

Against the two approaches illustrated through the studies presented above, we envisage future empirical work will aim at understanding how we can move toward explainable learning analytics (e.g., using data storytelling principles from the human-computer interaction and data science fields), instead of asking for an enhanced data literacy of the users for them to be able to interact with learning analytics solutions (Verbert et al., 2020). More work is also needed to identify what needs to be the right balance between orchestration and learning design aspects being embedded into the LA tool (embedded analytics) versus creating orchestrable learning analytics that can more freely be used by teachers according to their design intentions. Finally, we do hope that the discussion in this chapter may contribute to some maturity of the human-centered design perspective for learning analytics solutions.