Keywords

1 Introduction

Study success includes the successful completion of a first degree in higher education to the largest extent and the successful completion of individual learning tasks to the smallest extent (Sarrico, 2018). The essence here is to capture any positive learning satisfaction, improvement or experience during learning. As some of the more common and broader definitions of study success include terms such as retention, persistence and graduation rate, the opposing terms include withdrawal, dropout, noncompletion, attrition and failure (Mah, 2016).

Learning analytics (LA) show promise to enhance study success in higher education (Pistilli & Arnold, 2010). For example, students often enter higher education academically unprepared and with unrealistic perceptions and expectations of academic competencies for their studies. Both the inability to cope with academic requirements and unrealistic perceptions and expectations of university life, in particular with regard to academic competencies, are important factors for leaving the institution prior to degree completion (Mah, 2016). Yet Sclater and Mullan (2017) reported on the difficulty to isolate the influence of the use of LA, as often they are used in addition to wider initiatives to improve student retention and academic achievement.

However, the success of LA in improving higher education students’ learning has yet to be proven systematically and based on rigorous empirical findings. Only a few works have tried to address this but limited evidence is shown (Suchithra, Vaidhehi, & Iyer, 2015). This chapter aims to form a critical reflection on empirical evidence demonstrating how LA have been successful in facilitating study success in continuation and completion of students’ university courses.

2 Current Empirical Findings on Learning Analytics and Study Success

There have been a number of research efforts, some of which focussed on various LA tools and some focussed on practices and policies relating to learning analytics system adoption at school level, higher education and national level. Still, significant evidence on the successful usage of LA for improving students’ learning in higher education is lacking for large-scale adoption of LA (Buckingham Shum & McKay, 2018).

An extensive systematic literature review of empirical evidence on the benefits of LA as well as the related field of educational data mining (EDM) was conducted by Papamitsiou and Economides (2014). They classified the findings from case studies focussing on student behaviour modelling, prediction of performance, increase self-reflection and self-awareness, prediction of dropout as well as retention. Their findings suggest that large volumes of educational data are available and that pre-existing algorithmic methods are applied. Further, LA enable the development of precise learner models for guiding adaptive and personalised interventions. Additional strengths of LA include the identification of critical instances of learning, learning strategies, navigation behaviours and patterns of learning (Papamitsiou & Economides, 2014). Another related systematic review on LA was conducted by Kilis and Gülbahar (2016). They conclude from the reviewed studies that log data of student’s behaviour needs to be enriched with additional information (e.g. actual time spent for learning, semantic-rich information) for better supporting learning processes. Hence, LA for supporting study success requires rich data about students’ efforts and performance as well as detailed information about psychological, behavioural and emotional states.

As further research is conducted in the field of LA, the overriding research question of this chapter remains: Is it possible to identify a link between LA and related prevention and intervention measures to increase study success in international empirical studies?

2.1 Research Methodology

Our critical reflection on empirical evidence linking study success and LA was conducted in 2017. Literature review contributions to LA were first analysed, followed by individual experimental case studies containing research findings and empirical conclusions as well as evidence. Search terms included “learning analytics” in combination with “study success”, “retention”, “dropout”, “prevention”, “course completion” and “attrition”. We searched international databases including Google Scholar, ACM Digital Library, Web of Science, ScienceDirect, ERIC and DBLP. Additionally, we searched articles published in journals such as Journal of Learning Analytics, Computers in Human Behaviour, Computers & Education, Australasian Journal of Educational Technology and British Journal of Educational Technology. 6220 articles were located, and after duplicated papers were removed, 3163 were remaining. All of these abstracts of papers were screened and were included in our critical reflection on empirical evidence according to our inclusion criteria as follows: (a) were situated in the higher education context, (b) were published between 2013 and 2017, (c) were published in English, (d) presented either qualitative or quantitative analyses and findings and (e) were peer-reviewed. The number of key studies identified was 374 (in the first round) then limited to 46 (due to substantiality of empirical evidence); an elaboration of the identified empirical evidence from the limited studies will form our upcoming work. In this paper, we provide a general overview of the identified empirical evidence.

2.2 Results of the Critical Reflection

This section is divided into (1) positive evidence on the use of LA to support study success, (2) insufficient evidence on the use of LA to support study success and (3) link between LA and intervention measures to facilitate study success.

2.2.1 Positive Evidence on the Use of Learning Analytics to Support Study Success

Some of the positive empirical evidence presented by Sclater and Mullan (2017) include the following: At the University of Nebraska-Lincoln after LA was adopted, their 4-year graduation rate increased by 3.8% in 4 years. At Columbus State University College, Georgia, course completion rates rose 4.2%. Similarly, at the University of New England, South Wales, the dropout rate decreased from 18% to 12%. Control group studies yield the following results: there was a significant improvement in final grade (6%) at Marist College; at Strayer University, Virginia, the identified at-risk students were given intervention and resulted in 5% increase in attendance, 12% increase in passing and 8% decrease in dropout. At the University of South Australia, 549 of 730 at-risk students were contacted; 66% passed with average GPA of 4.29. Fifty-two percent of un-contacted at-risk students passed with average GPA of 3.14. At Purdue University, Indiana, it was found that using the university’s predictive analytics system (Course Signal), there were consistently higher levels of Bs and Cs grades obtained than Ds and Fs grades in two semesters of courses. A 15% increase in recruitment and a 15% increase in retention as a result was reported (Tickle, 2015).

We also identified positive evidence on the use of LA to support study success through the use of assessment data, engagement indicators, online platform data and the use of personalised feedback, as follows.

Predictive Analytics Using Assessment Data

It was found on average that there was a 95% probability if a student had not submitted their assignment and that they will not finish the course (Hlosta, Zdrahal, & Zendulka, 2017). Here, assessment description referred to (1) students’ demographic information (e.g. age, gender, etc.), (2) students’ interactions with the VLE system, (3) information about students’ date of registration and (4) a flag indicating student assignment submission. This information is used to extract learning patterns from the students where their progress of the course can be predicted. The assessment of the first assignment provides a critical indicator for the remainder of the course. The conducted experiments showed this method can successfully predict at-risk students.

Predictive Analytics Using Engagement Indicators

Information about students’ behaviour that is made available during the course can be used to predict the decrease of engagement indicators at the end of a learning sequence. Three main tasks that students conducted in a MOOC environment were able to yield good results in the prediction if there would be a decrease in engagement in the course as signalled by engagement indicators (Bote-Lorenzo & Gomez-Sanchez, 2017). The authors found that three engagement indicators derived from tasks being carried out in a MOOC were very successful in predicting study success—watching lectures, solving finger exercises and submitting assignments. It was suggested that their predictive method would be useful to detect disengaging students in digital learning environments.

Predictive Analytics Using Digital Platform Data

Self-report and digital learning system information (i.e. trace data) can be used to identify students at risk and in need of support as demonstrated by a study conducted by Manai, Yamada, and Thorn (2016). For example, some self-report survey items measure non-cognitive factors such as indicative predictors of student outcomes allowing one to inform actionable insights with only a few items’ data. Certain formulas were used in their study such as (1) if students showing higher levels of fixed mindset and to be at risk, a growth mindset is promoted to them by engaging them in growth mindset activities and also giving feedback to students that establishes high standards and assuring that the student is capable of meeting them; (2) if students showing higher levels of belonging uncertainty, group activities that facilitate building a learning community for all students in the classroom are provided; and (3) if students showing low levels of math conceptual knowledge, scaffolding for students is provided during the use of the online learning platform. Similarly, Robinson, Yeomans, Reich, Hulleman, and Gehlbach (2016) utilised natural language processing, and their experiment showed promising predictions from unstructured text which students would successfully complete an online course.

Personalised Feedback Leading to Learning Gains

Feedback can be tailored based on the student’s affective state in the intelligent support system. The affective state is derived from speech and interaction, which is then used to determine the type of appropriate feedback and its presentation (interruptive or non-interruptive) (Grawemeyer et al., 2016). Their results showed that students using the environment were less bored and less off-task showing that students had higher learning gains and there is a potential and positive impact affect-aware intelligent support.

2.2.2 Insufficient Evidence on the Use of LA to Support Study Success

The most recent review of learning analytics published in 2017 from Sclater and Mullan (2017) described the use of LA to be most concentrated in the United States, Australia and England; most institutional initiatives on LA are at an early stage and lacking sufficient time to find concrete empirical evidence of their effectiveness (Ifenthaler, 2017a). However, some of the most successful projects were in the US for-profit sector, and these findings are unpublished. In the review conducted by Ferguson et al. (2016), the state of the art in the implementation of LA for education and training in Europe, United States and Australia was presented which is still scarce. Specifically, it was noticed that there are relatively scarce information on whether LA improves teaching and learners’ support at universities, and problems with the evidence include lack of geographical spread, gaps in our knowledge (informal learning, workplace learning, ethical practice, lack of negative evidence), little evaluation of commercially available tools and lack of attention to the learning analytics cycle (Ferguson & Clow, 2017).

Threats deriving from LA include ethical issues, data privacy and danger of overanalysis, which do not bring any benefits and overconsumption of resources (Slade & Prinsloo, 2013). Accordingly, several principles for privacy and ethics in LA have been proposed. They highlight the active role of students in their learning process, the temporary character of data, the incompleteness of data on which learning analytics are executed, the transparency regarding data use as well as the purpose, analyses, access, control and ownership of the data (Ifenthaler & Schumacher, 2016; West, Huijser, & Heath, 2016). In order to overcome concerns over privacy issues while adopting LA, an eight-point checklist based on expert workshops has been developed that can be applied by teachers, researchers, policymakers and institutional managers to facilitate a trusted implementation of LA (Drachsler & Greller, 2016). The DELICATE checklist focusses on Determination, Explain, Legitimate, Involve, Consent, Anonymise, Technical aspects and External partners. However, empirical evidence towards student perceptions of privacy principles related to learning analytics is still in its infancy and requires further investigation and best practice examples (Ifenthaler & Tracey, 2016).

Ferguson et al. (2016) documented a number of tools that have been implemented for education and training and raised a number of important points—(a) most LA tools are provided on the supply side from education institutions and not on the demand side required by students and learners; (b) data visualisation tools are available, however do not provide much help in advising steps that learners should take in order to advance their studies/increase study success; and (c) especially evidence is lacking on formal validation and evaluation of LA tools of the impact and success, although national policies in some European countries such as Denmark, the Netherlands and Norway and universities such as Nottingham Trent University, Open University UK and Dublin City University have commenced to create an infrastructure to support and enable policies of utilisation of LA or implementation/incorporation of LA systems. Hence, the evidence on successful implementation and institution-wide practice is still limited (Buckingham Shum & McKay, 2018). Current policies for learning and teaching practices include developing LA that are supported through pedagogical models and accepted assessment and feedback practices. It is further suggested that policies for quality assessment and assurance practices include the development of robust quality assurance processes to ensure the validity and reliability of LA tools as well as developing evaluation benchmarks for LA tools (Ferguson et al., 2016).

2.2.3 Link between Learning Analytics and Intervention Measures to Facilitate Study Success

Different LA methods are used to predict student dropout such as predictive models and student engagement with the virtual learning environment (VLE) (more reliable indicator than gender, race and income) (Carvalho da Silva, Hobbs, & Graf, 2014; Ifenthaler & Widanapathirana, 2014). Some of the significant predictors of dropout used in these methods can be indicated and include the following: posting behaviour in forums, social network behaviour (Yang, Sinha, Adamson, & Rose, 2013), percentage of activities delivered, average grades, percentage of resources viewed and attendance (85% accuracy of at-risk student identification) (Carvalho da Silva et al., 2014). Similarly, different factors are used at Nottingham Trent University to signal student engagement: library use, card swipes into buildings, VLE use and electronic submission of coursework, analyses the progression and attainment in particular groups (Tickle, 2015). An example technique is as follows: if there is no student engagement for 2 weeks, tutors will get an automatic email notification, and they are encouraged to open up a dialogue with the at-risk student. Their LA system intends to help increase not only study retention but also to increase study performance. Prevention measures include pedagogical monitoring. The timeliness of the institution or university’s intervention is very important including noticing signs of trouble and responding immediately to these (Tickle, 2015). A question concerning ethics may be “do students want an algorithm applied to their data to show they are at risk of dropping out?” causing intervention from respective tutors to take place (West et al., 2016).

LA are often discussed and linked with regard to self-regulated learning. Self-regulated learning can be seen as a cyclical process, starting with a forethought phase including task analysis, goal setting, planning and motivational aspects (Ifenthaler, 2012). The actual learning occurs in the performance phase, i.e. focussing, applying task strategies, self-instruction and self-monitoring. The last phase contains self-reflection, as learners evaluate their outcomes versus their prior set goals. To close the loop, results from the third phase will influence future learning activities (Zimmerman, 2002). Current findings show that self-regulated learning capabilities, especially revision, coherence, concentration and goal setting, are related to students’ expected support of LA systems (Gašević, Dawson, & Siemens, 2015; Schumacher & Ifenthaler, 2018b). For example, LA facilitate students through adaptive and personalised recommendations to better plan their learning towards specific goals (McLoughlin & Lee, 2010; Schumacher & Ifenthaler, 2018a). Other findings show that many LA systems focus on visualisations and outline descriptive information, such as time spent online, the progress towards the completion of a course and comparisons with other students (Verbert, Manouselis, Drachsler, & Duval, 2012). Such LA features help in terms of monitoring. However, to plan upcoming learning activities or to adapt current strategies, further recommendations based on dispositions of students, previous learning behaviour, self-assessment results and learning goals are important (McLoughlin & Lee, 2010; Schumacher & Ifenthaler, 2018b). In sum, students may benefit from LA through personalised and adaptive support of their learning journey; however, further longitudinal and large-scale evidence is required to demonstrate the effectiveness of LA.

3 Conclusion

This critical reflection of current empirical findings indicates that a wider adoption of LA systems is needed as well as work towards standards for LA which can be integrated into any learning environment providing reliable at-risk student prediction as well as personalised prevention and intervention strategies for supporting study success. In particular, personalised learning environments are increasingly demanded and valued in higher education institutions to create a tailored learning package optimised for each individual learner based on their personal profile which could contain information such as their geo-social demographic backgrounds, their previous qualifications, how they engaged in the recruitment journey, their learning activities and strategies, affective states and individual dispositions, as well as tracking information on their searches and interactions with digital learning platforms (Ifenthaler, 2015). Still, more work on ethical and privacy guidelines supporting LA is required to support the implementation at higher education institutions (Ifenthaler & Tracey, 2016), and there are still many open questions how LA can support learning, teaching as well as the design of learning environments (Ifenthaler, 2017b; Ifenthaler, Gibson, & Dobozy, 2018). Another field requiring rigorous empirical research and precise theoretical foundations is the link between data analytics and assessment (Ifenthaler, Greiff, & Gibson, 2018). Further, as LA are of growing interest for higher education institutions, it is important to understand students’ expectations of LA features (Schumacher & Ifenthaler, 2018a) to be able to align them with learning theory and technical possibilities before implementing them (Marzouk et al., 2016). As higher education institutions are moving towards adoption of LA systems, change management strategies and questions of capabilities are key for successful implementations (Ifenthaler, 2017a). The preliminary findings obtained in this critical reflection suggest that there are a considerable number of sophisticated LA tools which utilise effective techniques in predicting study success and at-risk students of dropping out.

Limitations of this study include the difficulty in comparing results of different studies as various techniques and algorithms, research questions and aims were used. Although much empirical evidence is documented in these papers, many studies are still works-in-progress, experimental studies and at very small scale. The papers discuss how LA can work to predict study success, and the steps following this to the discussions with the students and the approaches that teachers can take to address to at-risk students are under-documented. The questions raised concerning this are, for example: (a) Will students be able to respond positively and proactively when informed that their learning progress is hindered or inactivated? (b) Will instructors be able to influence the at-risk students positively so that they will re-engage with the studies? (c) In addition, ethical dimensions regarding descriptive, predictive and prescriptive learning analytics need to be addressed with further empirical studies and linked to study success indicators.

However, evidence on a large scale to support the effectiveness of LA actually retaining students onto courses are still lacking, and we are currently examining the remainder of the key studies thoroughly to obtain a clearer and more exact picture of how much empirical evidence there is that LA can support study success. Methods and advice also can be used as a guide in helping students to stay on the course after they have been identified as at-risk students. One suggestion is to leverage existing learning theory by clearly designing studies with clear theoretical frameworks and connect LA research with decades of previous research in education. Further documented evidence on LA include that LA cannot be used as a one-size-fits-all approach, i.e. requiring personalisation, customisation and adaption (Gašević, Dawson, Rogers, & Gašević, 2016; Ifenthaler, 2015).

Our future work also includes locating learning theories onto LA (which is currently lacking)—there is missing literature on variables as key indicators of interaction and study success in digital learning environments. Hence, while the field of learning analytics produces ever more diverse perspectives, solutions and definitions, we expect analytics for learning to form a novel approach for guiding the implementation of data- and analytics-driven educational support systems based on thorough educational and psychological models of learning as well as producing rigorous empirical research with a specific focus on the processes of learning and the complex interactions and idiosyncrasies within learning environments.