Keywords

1 Introduction

The field of learning analytics (LA), with its associated methods of online student data analysis, is able to provide novel and real-time approaches to assessing critical issues such as student progression and retention, thereby informing decisions related to teaching and learning. While LA has gained much attention and has been/is being adopted by many higher education institutions (HEIs) in Europe and other parts of the world, the maturity levels of HEIs in terms of being ‘student data informed’ are only in the early stages. Literature has identified that the adoption of LA in complex educational systems requires a systematic approach to bring about effective changes [2]. Moreover, some common challenges that beset the adoption at a wide scale need to be addressed by involving all relevant stakeholders [3]. Our research project sets out to tackle the identified problems by building a policy framework that is based on findings of various consultations with a diversity of stakeholders. The study aims to answer four questions: (1) what is the state of the art in terms of LA adoption in Europe, (2) what are the key challenges that impede institutional adoption of LA, (3) how do expectations of LA vary among different stakeholders, and (4) how can we address LA related actions and challenges through policies.

The goal of the study is to incorporate existing experiences of institutional adoption with key stakeholders’ perspectives regarding opportunities for LA and concerns about it, thereby developing a policy framework to support effective and responsible adoption at an institutional scale.

2 Methods

The policy framework is developed using mixed methods. Between 2016 and 2017, various datasets have been collected through online group concept mapping (GCM), interviews, surveys, and focus groups. With the online GCM, we have collected 99 statements from 29 LA experts across the world. With the interview method, we had in-depth conversations with 64 institutional leaders from 51 HEIs across Europe. With the survey method, we have reached out to institutional leaders from 46 European HEIs, 3,263 students from six European HEIs and 210 teaching staff from four European HEIs. With focus groups, we have carried out in-depth conversations with 74 students and 59 teaching staff from four European HEIs. The development of protocols for the above mentioned activities were driven by the research questions listed above. The methods used for analysis include cluster analysis, exploratory factor analysis, confirmatory factor analysis, and thematic content analysis. The development of the policy framework was inspired and guided by the Rapid Outcome Mapping Approach (ROMA) [1, 2, 5] model that begins by defining an overarching policy objective, followed by six steps designed to provide policy makers with context-based information: (1) map political context, (2) identify key stakeholders, (3) identify desired behaviour changes, (4) develop engagement strategy, (5) analyse internal capacity to effect change, and (6) establish monitoring and learning frameworks.

3 Results

3.1 Essential Features of a LA Policy

The group concept mapping activity received 99 statements in response to the prompt “an essential feature of a higher education institution’s LA policy should be...”. Six key themes emerged from these statements including (1) privacy & transparency, (2) roles & responsibilities (of all stakeholders), (3) objectives of LA (learner and teacher support), (4) risks & challenges, (5) data management, and (6) research & data analysis. The rating results of the these statements show an obvious drop of ratings for the ease of implementation level of these themes, compared to their importance level. One of the implications is that the six features could potentially be challenges to deal with in the policy development process. Moreover, issues around privacy and transparency were considered the most important elements, but also the easiest to include in a policy.

3.2 State of Adoption – Senior Managers’ Perspective

The interview data showed that 21 out of 51 institutions were already implementing centrally-supported LA projects, 9 of which had reached institution-wide level, 7 partial-level (including pilot projects), and 5 were at the data exploration and cleaning stage. Meanwhile, 18 institutions were in preparation to roll out institutional LA projects, and 12 did not have any concrete plans for an institutional LA project yet. The survey data revealed that 15 institutions had implemented LA, of which 2 had reached full implementation and 13 were in small scale testing phases. Sixteen institutions were in preparation for LA projects, and 15 were interested but had no concrete plans yet. One of the implications of the two data sets is that there was high interest in LA among HEIs in Europe, but the maturity of adoption was low.

From the survey, we identified that five top drivers for institutions to adopt LA were to improve student learning performance and satisfaction, teaching excellence, student retention, and to explore what LA can do for the institution/staff/students. These drivers were also repeatedly mentioned by the interview participants. In particular, for those who were driven by the fifth reason, their adoption was predominately experimental and exploratory. As a result, there was a sense of uncertainty about the return of investment in these institutions given that the contextual relevance and benefits of LA were still unclear.

3.3 Interests and Concern – Perspectives of Students and Staff

The result of the student survey that compared ideal and realistic expectations of LA identified two factors: ethical expectations and service expectations. Students held strong beliefs toward the university securely holding all collected data, whilst the belief that a university should seek consent before the collection, use, and analysis of educational data appeared to elicit the lowest average response for each sample of students. Moreover, students appeared to show strong interest in receiving regular updates on their learning, but low interest in receiving early interventions if LA services found them to be at-risk. The result suggests a student preference over a LA service that facilitates independent learning rather than one which would impede their self-direction.

Consultations with students and teaching staff through focus groups [4] revealed a strong interest in using LA to enable personalised support and provide an overview of learning progress, so as to improve pedagogical effectiveness and learning experience and success. Despite their interest in LA, both students and teaching staff expressed various concerns about adopting LA. Among these, ethical and privacy issues, such as access, security and anonymity, appeared to be the top concerns for students. As for teaching staff, time pressure and potential use of LA in judging teaching performance particularly concerned them.

4 Conclusion

This research project has reached out to nearly half of the European countries, and observed high interest in LA among HEIs. However, few HEIs have taken a systematic approach to LA with defined strategy and policy. Our preliminary findings have identified prominent challenges that need to be tackled through an overarching policy. Up to now, the research team has developed the first draft of a policy framework primarily based on the interview data. This policy framework maps out 51 HEIs’ experience to the six dimensions of the ROMA model and presents key actions to take towards systematic adoption of LA, key challenges to address in the adoption process, and key questions to answer when developing an institutional learning analytics policy (Fig. 1). The policy framework will be updated with findings from other datasets and connected to detailed case studies as a reference model and can then be used to guide the development of institutional policies and strategic planning for learning analytics.

Fig. 1.
figure 1

The policy framework structure

Full size image