Keywords

1 Motivation

The increase of e-learning in higher education in recent decades was further fueled by the Covid-19 pandemic. The majority of students worldwide were affected by measures such as lockdowns, social distancing, and university closures. Often face-to-face teaching was discontinued, and online teaching was offered instead [1]. The widespread use of online teaching creates vast amounts of academic data. The systematic evaluation of this data is called Learning Analytics (LA), generally defined as “the measurement, collection, analysis, and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” [2] (p. 32). Regarding teaching in times of the pandemic, LA offers, for example, the possibility of relieving students’ sense of isolation by offering comparisons with peers. It can also provide teachers with guidance on how to adapt teaching materials to students’ performance or interests without seeing them face-to-face. Nevertheless, before LA systems (LAS) are developed or refined, one should take a step back and consider what the educational stakeholders involved, namely students, teachers, and institutions [3], perceive as opportunities and threats. There has been an underrepresentation of student perspectives in research [4]. However, their inclusion is essential to the development and use of LAS and can increase satisfaction, motivation, and commitment [5]. This article aims to fill this gap by addressing the following research questions:

  1. (1)

    What are the opportunities and threats of LA from the students’ perspective?

  2. (2)

    How do students’ views relate to those in current LA literature?

To answer these questions, a qualitative survey with university students is conducted and groups of opportunities and threats are derived, which are then cross-referenced with findings from current literature. The paper contains six sections. Section two provides an overview of current LA research. Section three outlines the research design. Section four presents the study’s results, identifying the opportunities and threats. Section five discusses the results in relation to findings in current literature and highlights further needs for research. Finally, section six presents the conclusion and limitations.

2 Theoretical Foundation

2.1 Learning Analytics

The increase in online learning and developments in the field of data analytics has led to a growing number of universities considering how the data generated in learning management systems (LMS), for instance, can be meaningfully analyzed and used. The usage of educational data can transform learning and teaching practices and serve as a foundation for educational research [6]. Higher Education Institutions (HEIs) use multiple interactive e-learning environments nowadays, collecting vast amounts of data that contain information about the users themselves (e.g., academic performance), their interaction with systems (e.g., log ins, user pathways, download activity), communication with other students or teachers (e.g., e-mails, forum posts), as well as information about courses with their underlying curriculum and learning objectives [3, 7]. Through descriptive and predictive models, such data is processed in real-time or a time-lagged manner to derive meaningful insights that can assist educational stakeholders in decision support and can help to improve learning and teaching contents and environments [8]. Predictions can involve an entire group of learners or individual students, looking at overarching issues like dropout or failure rates, and more small-scale matters like boredom and short-term learning [6], on a course-level or departmental-level [2].

Most LA research focuses on the learning process, “analyzing the relationship between learner, content, institution and educator” [2] (p. 36). The usability, effectiveness, and validity of LAS are often examined [7, 9]. There have also been studies about developing specific tools and underlying design principles for LAS [8]. The main stakeholders of LA are students, teachers, and institutions [3], however, there is surprisingly little research on students’ perception of LA [7, 10]. Ferguson states that a focus on students’ perspective is crucial to the development of LAS, concentrating on their demands rather than the institution’s demands, to motivate and satisfy students to meet their career goals [5]. Although the remainder of this paper focuses solely on students’ perspective, LA also poses many benefits and challenges to teachers and the institution [2, 11, 12].

2.2 Opportunities of Learning Analytics

There are typical issues in online teaching, some of which have intensified during the Covid-19 pandemic. For example, students may feel isolated due to a lack of contact with peers or lose track of the many online courses, materials, and assignments [13]. Some also struggle with technical problems and lose their motivation to study [13]. At the same time, teachers of online classes cannot rely on the visual cues of face-to-face classroom interaction anymore, which usually signals them if students feel satisfied, overwhelmed, or bored with the course content [14]. Again, the use of LA can provide valuable assistance here. With the help of LA, teachers can assess the learning behavior of their students, e.g., which topics were hardly worked on or which tasks were repeatedly solved incorrectly, to adapt course content or teaching methods [8]. The students benefit from customized assignments and courses, leading to an overall improvement of the academic program, enrichment of the student experience, and promotion of better learning [12]. LA offers students insights into their learning habits, enabling them to adjust their learning behavior accordingly [2]. Based on engagement data of previous cohorts, models for successful behavior can be developed to provide learners with automated guidance on how to achieve improvement [15]. Automated academic advising systems are also used, for example, to provide students with recommendations for course selection based on their learning style and performance [16].

Based on LA data, assessment analytics allow learners to evaluate their attainment across time, in relation to their personal goals or against their peers [2, 11]. The information about their place in the cohort, e.g., in terms of final grades or specific learning outcomes, provides students with a sense of belonging, reducing the feeling of isolation and encouraging them to work harder [2, 11]. By using data about weaknesses or common mistakes of former cohorts, learners can be provided pre-submission feedback before assignments, enhancing their approach to the same task [11]. Furthermore, LA is used to detect students whose performance in individual courses or their general studies is comparatively weak [9]. The at-risk students then benefit, for example, from automated alerts or early interventions by instructors, aiming to reduce dropout and failure rates and enabling as many students as possible to graduate successfully [5, 8].

2.3 Threats of Learning Analytics

The increasing amount of personal data collected about students and their activities also brings risks. Many ethical objections have been voiced [17]. Vast amounts of LA data can be analyzed with different objectives, possibly invading students’ privacy [7]. Questions arise about student consent, data protection and security, the duration of data storage, and access to the data [5, 12]. In some cases, students are not offered an option to opt out of data collection, or the data is later used in a way that students did not actively agree to [18]. Security risks exist if student data is not stored in a secure location or if unauthorized individuals gain access to it [17, 19]. There are also country-specific differences as to whom the owner of personal data is, the individual or the collector [7], complicating, for example, students’ chances to opt out at a later time. It is also problematic to base educational decisions or predictions exclusively on data, reducing students to a metric [19, 20] and putting them into categories by stereotyping and generalizing [21]. The collection and exploitation of LA data can add to one’s overall sense of increasing surveillance in all areas of life [17, 22], interfering with students’ privacy and their academic freedom [11]. Ellis states that, e.g., automated course recommendation systems infantilize students, pushing them to take classes that are most likely better for their GPA but do not necessarily fit students’ interests or goals [11]. The same is described as a loss of autonomy or paternalism by other researchers. Students are supposed to choose an institutionally preferred action rather than acting out on their preferences [5, 20]. The underlying problem is that most predictive LA models are based on a behaviorist model, meaning that “individuals with approximately the same profile generally select and or prefer the same thing” [23] (p.191), compared to models used in other fields like the rational utility maximizer model or a habitual perspective of behavioral economics [20]. The misconception of students and teachers that LAS are value-neutral and provide objective aid is an opportunity threat should they decide to rely exclusively on the system [20].

LA also bears the threat of invalid predictions or false interpretations [24] due to inadequate or flawed data [17]. E-learning systems only capture a fraction of the learning, not providing a holistic view that considers all possible influences on students’ failure or success, like personal problems or financial difficulties [19]. There is also the risk of mistaking correlation for causation. For example, students’ engagement rates are often used to predict their success; however, one can argue that exceptional students need to engage less than weaker students to achieve good grades [17]. The use of LA can also demotivate students. Continual monitoring can cause conscious or unconscious behavior changes, making students feel pressured to constantly self-optimize, which leads to stress or non-participation [19]. Furthermore, classifying weaker students as at-risk students may influence their outlook on success and deflate their potential [21], leading to self-fulfilling prophecies [17]. Labeling of learners as, e.g., exceptional or at-risk, can also affect teacher’s or faculty’s perception, resulting in different or discriminative behavior [19]. Adjustments based on LA may focus on certain groups of students: For many institutions, the emphasis lies on minimizing student withdrawal or failure [5] or driving academic excellence [2]. Thus, at-risk and high achieving students get the biggest share of attention, ignoring others on the achievement spectrum [11].

3 Research Design

After section two has provided an overview of the opportunities and threats of LA for students in existing literature, the students’ views will now be examined with the help of a qualitative survey. The aim is to get detailed insights of students’ thoughts concerning the opportunities and threats of LA usage in HEIs. The study was designed as an online questionnaire and contained three open questions:

  • (Q1) Which opportunities do you see for Learning Analytics at universities?

  • (Q2) Which threats do you see for Learning Analytics at universities?

  • (Q3) Which outweighs the other for you personally, opportunities or threats?

The study was conducted in the participants’ native language, and the results were translated afterwards. The study participants were undergraduate business administration and information systems students enrolled in a Business Intelligence course, having basic data modeling and applied analytics knowledge.

The answers to Q1 and Q2 were analyzed using the qualitative content analysis according to Mayring, a systematic approach to the qualitative analysis of texts [25]. Mayring promotes a step model for inductive category development, as part of his summative approach. For the evaluation of the questionnaire in this paper each step has been performed independently by two researchers. First, the participants answers were paraphrased and generalized to a level of suitable abstraction into core sentences. Then, in a first step of reduction, contents that did not answer the questions were cut out. In a second step of reduction, the core sentences were combined with similar ones and thus classified into categories. After working through 30% of the answers a revision of categories was carried out, combining similar categories and thus reducing the number of categories further. The participants’ answers and the categories derived from them were checked for plausibility with a focus group of three other researchers. Finally, the results were interpreted, including quantitative steps like the calculation of frequencies.

4 Results

A total of 139 students participated in the study, with 136 questionnaires being valid. 64% of the participants were male and 36% female. Sections 4.1 and 4.2 provide an overview of the most commonly named opportunities, threats, and notable quotes. Regarding Q3, the result is as follows: 89% of the participants answered that the opportunities of LA outweigh the threats, 5% stated that the threats would outweigh the opportunities, and 6% said they consider opportunities and risks to be in balance.

4.1 Learning Analytics Opportunities from Students’ Perspective

Table 1 summarizes the five identified categories of opportunities, containing 20 subcategories, each with the absolute and percentage numbers of mentions. The three most frequently mentioned opportunities are displayed with a gray background.

Table 1. Learning analytics opportunities from students’ perspective
Full size table

Category one refers to university-wide opportunities. Ten percent of participants envision new or adapted classes within the curriculum, according to the students’ interests and popular classes, as shown by LA. Some (7%) also wish for an automated course recommender system that can help them navigate the many course offers and is based on their performance, following the curriculum, and presenting their interests. The second category focuses on LA opportunities for the course design. The adaption of contents and teaching materials was the second most named opportunity (36%). Teachers can identify students’ strengths and weaknesses, as well as their likes and dislikes and modify courses accordingly. Also, 27% of the participants mentioned that this would motivate them to engage and study more. The most common answer (45%) was that the instructor could provide additional course work, explanations, or videos that solely focus on the most challenging topics of the class, as identified by LA. The third category contains opportunities that affect the individual learning behavior. One-fifth of participants would adapt their learning behavior based on LA, believing that it can help them study more effectively. Some participants (12%) express that LA supports more targeted exam preparation. One participant wrote: “One has a better overview of the learning success, especially compared to peers. The time still available until the exam can be better planned and used, considering other modules taken during the semester. It can have a positive impact on a student’s time management.” The comparison to peers has been the most commonly named opportunity in this category (33%). Students can feel isolated or at loss of orientation in online classes. The constant knowledge of how they are doing (e.g., time-wise, knowledge-wise, grade-wise) compared to peers can be beneficial. One participant noted: “Students are more motivated by this. The learning progress is considered during the semester, preventing students from postponing studying until the end of the semester. Through LA, it feels like a virtual classroom is formed, which is familiar to many students from high school. Thus, the learning situation at the university is not so strange for the students, and they gain more control. For some, the freedom (e.g., no compulsory attendance) at university is not a great thing.” The fourth category is about transparency. Easily accessible, clear overviews of grades and the current GPA have been outlined as a benefit by 13% of the participants. Even more (26%) view the traceability of final grades as an advantage, as they are often made up of individual components, e.g., assignments, participation, and midterms. Opportunities concerning the communication or interaction with instructors (category 5) have only been mentioned by a few participants. These include the possibility of tailored, more efficient help that an instructor can give individual students based on their performance, engagement, and time spent on particular course contents (8%). Interventions for at-risk students have been mentioned by 7%. The potential of automated, anonymous feedback to teachers about the course was described by 4%: “Instructors can analyze whether the assignments are too complex or too easy. Often, students do not dare to ask questions or demand a further explanation in person”.

4.2 Learning Analytics Threats from Students’ Perspective

Respectively to Table 1, Table 2 shows four categories and 21 subcategories of threats mentioned by participants. Again, the most common threats are highlighted in gray.

Table 2. Learning analytics threats from students’ perspective
Full size table

Participants have most commonly voiced ethical concerns (category 1), especially the threat of insufficient data protection (43%). They fear that a LAS could be attacked and data leaked publicly or that unauthorized university personnel could access it for other purposes than they agreed on. Some participants believe that the collection of LA data violates their privacy (26%). However, they are less concerned about the academic data and more concerned about demographic data, e.g., gender, ethnicity, and income. A few participants worry (6%) that students will not be seen as individuals with their own preferences, feelings, personal challenges, and learning strategies anymore but rather be reduced to a metric defined by their academic success.

Regarding the threats of implementing a LAS (category 2), the risk of collecting inadequate data (10%) or that the system cannot represent offline learning (13%) are mentioned most commonly. One participant wrote: “Teachers might rely too much on data. The data on how long someone reads a book borrowed from the library at home cannot be answered by learning analytics. If a professor then makes videos available online that are supposed to illustrate the material, but these are not viewed, the misconception could arise that the students are doing little for the module when in fact, they only have a different learning style than the one served by the professor.” Participants also voiced the concern that the LAS could be manipulated by students (9%), e.g., by creating data that makes instructors believe the topics are too complex or the scope of materials is too high so that the instructor feels pressured to make the course easier.

Findings of the survey show that the usage of LA could affect students’ behavior in a negative way (category 3). Continuous monitoring, comparisons with peers, and the constant highlighting of own weaknesses can increase the pressure on students to self-optimize (26%). If they know that they are constantly performing worse than their peers, lower achieving students can feel demotivated by LA (15%). Students who prefer to study offline and do not feel represented by the LAS can also feel discouraged. One participant noted: “For students, there is a risk of evaluating their learning success solely on the quantitative metrics of learning success (compared to fellow students). Thus, students might disregard relevant topics in which they initially achieved good test results but which require continuous and intensive study. Also, students’ learning behaviors vary widely. Some will take advantage of and benefit from a time and place independent learning system. At the same time, other students lack the necessary self-discipline to work through the material. Also, computers are not suitable for everyone; e.g., some will experience greater fatigue due to screen time. I prefer to use pen and paper.” Students could also misinterpret the data (15%) or focus solely on fulfilling the given learning objective (5%) and not study voluntarily or do extracurricular activities.

Category four includes perceived threats associated with data analysis or interpretation by instructors. Some participants (6%) believe that teachers lack digital competencies and knowledge to properly evaluate LA data and results. Participants are most commonly concerned about invalid interpretations and predictions (32%), e.g., leading to the inadequate adaption of courses or teaching methods. The fear of discrimination against groups of learners (15%) or individual students (9%), e.g., based on bad academic performances, gender, or ethnicity, was also mentioned. Ten percent of the participants are concerned that teachers spend so much time becoming involved with LA and optimizing metrics that they have less time for core tasks, like good quality teaching and personal communication with students. Students fear that if teachers rely too much on data, they might think personal interaction and feedback from students is unnecessary (8%). Furthermore, the risk of misuse of LA data by instructors has been described by 13% of the participants. They are concerned that instructors may feel personally insulted if their videos are barely watched, or their assignments are not adequately completed. Teachers could then take “revenge” on students by focusing primarily on those topics in the exam that were little worked on or poorly understood.

5 Discussion

The participants of the study came up with numerous ideas, 449 opportunities that could be summarized in 20 categories, and 399 risks, categorized in 21 subcategories. Many of the opportunities and risks outlined by participants can also be found in current literature, albeit over- or underrepresented in some cases. On a promising note, most participants had a rather positive attitude toward LA. A positive stakeholder attitude provides a good foundation for the successful implementation of LA [7, 24].

The participants, who are students themselves, saw most of the benefits at the course level, especially in adapting course content and improving their learning behavior. University-wide effects of LA or faculty-student interaction played only a secondary role. The adaptation of course content based on LA data, which is supposed to improve teaching quality, is also commonly found in literature and illustrated in case studies, as is the comparison to fellow learners [10, 12, 17]. Especially online-only teaching, as it was common worldwide in times of the Covid-19 pandemic, can lead to a feeling of isolation [1]. Through comparisons with peers, learners develop a sense of belonging and can be motivated to get more involved [17]. Participants frequently mentioned this aspect, which is also often included in current research. There is a lot of literature concerned with automated course recommender systems [8, 16, 20] and interventions for at-risk students [11, 17, 21]. However, these opportunities have only been pointed out by five percent, respectively, seven percent, of the participants. A much higher percentage of participants viewed additional assignments and explanations for complex topics and the overview of the learning process as an opportunity of LA. One could argue that assignments and explanations are part of the adaption of course materials. However, most participants, 61 in total, pointed out that the focus on making content that is difficult to understand better comprehendible is one of the leading LA benefits. Therefore, it is presented individually in this study. This is sometimes mentioned in the literature but is usually not the focus of current research [12]. The same applies to the clear overview of the learning progress, often briefly referred to by researchers but not individually examined, e.g., by evaluating which visualization can provide the best overview using a LA dashboard [5, 9]. Four subcategories are interestingly hardly ever mentioned in LA research: The more targeted exam preparation, the better time management, transparency of the final course grade, and anonymous feedback through LA data. This might be because current literature views LA opportunities on a higher, more abstract level than the students who focus more on a micro-level, thinking about their everyday student life. This is partly because, as stated in the introduction of this article, there is very little research specifically addressing the demands of LAS students. When looking at the results concerning LA threats, it can be stated that the majority of the study results have been examined in previous research. In particular, the ethical concerns indicated by many participants have already been investigated several times, and guidelines for handling, e.g., the collected data, have already been developed [17, 20,21,22]. Some participants worry that the data collected by the LAS do not adequately reflect their learning behavior and experiences. New approaches such as the use of multimodal data [26] or the use of artificial intelligence [27] can provide a solution in this regard. Nevertheless, some risks identified in this study hardly receive any attention in current research. For example, one concern is that students adapt their learning behavior to LA so that they solely focus on optimizing LA metrics and reaching the given learning objectives. Their own interests in the course material and the efforts that go beyond the required targets are pushed into the background. Another concern is that instructors spend so much time and effort on LA that the quality of teaching decreases. Also, participants worry that LA leads to less in-person student-teacher communication because teachers could rely solely on LA data to review their courses and on automated feedback to students. The last topic that does not receive attention in current research is the participants’ fear that instructors take some metrics, like few video views, personally and then misuse the data to make assignments or exams harder. Thus, these aspects must be considered when introducing or adapting LAS and should be communicated.

The findings from this survey serve as a foundation for further research. To validate and extend the results, a larger group of students from different study programs and nations should be questioned. Results from interviews and focus groups can provide additional value. Potentially, the research could be expanded to the school context. To gain an overarching view, the perspectives of teachers and institutions should be included. The results could then be used to identify drivers and barriers of LA. Finally, for practitioners, an overview of stakeholder requirements for LAS can be derived.

6 Conclusion

The contribution of this paper is an in-depth view of students’ perception of LA, which can help identify further needs for research and provide practitioners with guidance on what they should consider when implementing or adapting a LAS at university. Students are the most crucial stakeholder group, but their views are still underrepresented in current research. Ultimately, the more students are involved in the process of a LAS development, the higher the acceptance rate of the system can be. Therefore, a qualitative survey of 136 university students was conducted. Multiple researchers classified the responses, arranged them in tables, and enriched them with further explanations and quotes. The discussion then addressed how these results fit in with current literature and identified further research possibilities.

It can be inferred that the majority of participants felt generally positive toward LA. Many of the opportunities identified in the survey can also be found in academic literature, but the priorities are distributed differently in some areas. The benefits most frequently cited by participants were the customization of course materials using LA insights, additional explanations and assignments for complex content, and the comparison to other students. Very apparent is the small-scale and short-sighted nature of many answers, that focus mainly on the course-level, e.g., more targeted exam preparation or the comprehensibility of final grades. Only a few participants mentioned opportunities on a departmental or university-wide level. The three most commonly mentioned risks were insufficient data protection, privacy violation, and invalid predictions or interpretations based on LA data. In addition, many participants voiced their concern about predictions partly based on data that could be inadequate or not represent their offline learning efforts. Interestingly, students were also concerned that teachers could feel offended by specific LA data and then misuse LA to make exams harder to pass.

The results of our study are subject to some limitations, which mostly relate to the selection of participants for the survey. Firstly, it should be noted that the participants come from a specific European country and depending on the country, laws, and cultural background, the benefits and risks from the students’ point of view may vary. Secondly, the participants were university students generally familiar with data analysis. Therefore, students’ views from other disciplines or institutions should be considered in further investigations to enhance the foundation laid in this manuscript. In conclusion, digital teaching during the Covid19-pandemic has allowed students worldwide to continue studying despite lockdowns and closures. LA can improve online learning and motivate students through customized courses and comparisons with peers. Even after the pandemic, LA will continue to play an essential role in teaching and learning. However, universities should take the concerns of students seriously. The analytical processes should be transparent and accessible. Students’ wishes and requirements must be considered so that LAS can develop their full potential.