Keywords

1 Introduction

The world has drastically changed since the emergence of the Covid-19 pandemic. Most in-person activities have been either diminished or modified to reduce contact between persons which includes all learning activities have been moved predominantly to an online setting. This transition has caused many difficulties [1,2,3] in the delivery and consumption of educational content.

Also new applications and tools have been adopted to facilitate online teaching and learning such as learning management systems, teleconferencing platforms and so on. This use of technologies has prompted an exponential creation of educational data which is readily available and more accessible than ever before. But there is glaring issue of how do we transform this large amounts of raw data to meaningful and actionable information which can be used by educators to improve their pedagogical approaches?

As such there is a need for learning analytics tools that have a versatile data visualization module to take advantage of complex information, by break it down to easy to understand graphical translations that convey meaningful insights to educators for enhanced decision-making.

The use of data visualization for better comprehension and decision- making is not something new. The graphical translation of raw information has its roots in early map-making and has grown into the current statistical charts, graphs, heatmaps and etc. [4]. Today, there are various definitions to data visualization such as visual representations of statistics and patterns to visual text, but the context of ‘data visualization’ is generally understood as graphical representations of data to facilitate understanding [5, 6] and sense-making. With the growth of technology, computers can now process large amounts of data for analysis and render vibrant and informative visualizations very quickly and efficiently for consumption by the user.

The well-known advantages to data visualization has been adopted in many domains of research such as meteorology [7], healthcare [8], and business intelligence [9], to significantly improve overall compression and decision-making.

The education domain has also embraced data visualization for decision-making. This is because the educational data generated is relatively large and complex, making it challenging to understand as it is [10]. Data visualization has the ability to expand human working memory by reducing cognitive load using visual aids, providing space efficient visualization to represent complex data, identify patterns through visually explicit representation and also making it easier to monitor larger datasets with aggregated views of it [8].

Data visualization which is used in the educational context, it is often associated with learning analytics. Learning analytics is defined as “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” [11]. Overall, the direction of data visualization is consistent with the growing movement of learning analytics research and practice, which is to optimize the teaching and learning process with the analysis of relevant data. Over the last decade online learning management systems have been enhanced with data visualization as part of learning analytics to track and visualize educational data to reveal actionable insights.

However, a concern within the emerging area of data visualization in learning analytics is how to determine when a visualization is communicating information successfully to its intended user and how that success can be measured [12]. The majority of research has focused on how to utilize data visualization in learning analytics and techniques to inform choices and action among stakeholders [10, 13], limited research has been conducted on data visualization’s impact on learning analytics when it is in use for online learning. There are various ways to determine the successful implementation of data visualization in learning analytics depending on its goal and context. Notably when it comes to teaching and learning, the success of an online learning system is measured based on students’ performance metrics and retention rate.

Research has suggested the importance of evaluating a variety of factors of an online learning system [14] in order to be effective at achieving its intended outcomes. Specifically more attention is needed with respect to the quality of information presented in the online learning system when data visualization is involved.

It is also suggested that not only the relevance and comprehensiveness of content determines the success of an online learning system but also the data visualization and its design is notably associated with successful outcomes too [15]. Although there is a need to ensure a comprehensive examination of data visualization as an element of information quality. To date there is a lack of research which systematically models and validates its actual impact in learning analytics for online learning. Therefore an investigation of data visualization as the determinant of a successful learning analytics system is vital. This study explores the impact of data visualization in learning analytics towards educators’ satisfaction and students’ academic performance. The Information System success model [16] and Technology Acceptance model [17] were integrated as a solution to this research problem.

2 Literature Review

Before a successful implementation of an online learning system, it is vital for us to examine theories to identify key elements and its attributes [18]. This is because the systems for learning is a complex process involving different stakeholders (i.e. students, educators, administrators), especially when more elements are introduced such as data visualization in learning analytics. Although a ‘one-size-fits-all’ format would be ideal but not all stakeholders are driven by the same goals and have the same interest in the same sets of data or have the same visualization [19]. For example, educators are interested with data visualization on their course materials to understand student behavior and engagement with the content, while administrators strive to utilize data visualization for retention and enrolment purposes. As such what is needed is a more comprehensive understanding of the role of data visualization in learning analytics that goes beyond merely presenting information in visual form instead it should be examined from the perspective of data visualization’s impact on the overall system success.

2.1 Data Visualization in Learning Analytics

Data visualization is defined as a graphical representation of collected data to aid understanding of complex information that is sometimes difficult to be conveyed in words [20] or to identify patterns of specific situational behaviors. It can be said that data visualization is a core aspect of learning analytics work in relation to the provision of data to support the teaching and learning environment [19, 21, 22]. It is not only critical in visualizing large datasets but also creating meaningful insights for actionable decisions.

The benefits of data visualization are straightforward, as mentioned above, learning is a complex process involving different stakeholders and large amounts of collected data. Hence effective data visualizations for comprehension of collected data in learning analytics research is crucial and extremely valuable. A recent study by Paiva et al. [23] indicated that data visualizations are indeed an effective component to make information extracted from learning analytics understandable among educators. A study by Knight et al. [13] examined the web-based writing analytics tool by including data visualization in the formative feedback to students. Results showed that the visualization aspects provided additional writing support to students within and outside the classrooms. A similar result was reported in a study examining the students’ outcomes and the use of e-portfolio where the inclusion of data visualization positively impacted their performance over time. The findings from Schaaf et al. [24] posited that data visualization enables students to interpret their development better with indicators for further improvement.

The varying positive value of data visualization to different stakeholders is shown when visual representations of data interact with human knowledge construction and inference-making, there is evidence for interpretative bias among different stakeholders and their visualization literacy [25]. But Yalcin et al.’s [26] comparative analysis on novice and skilled users in visual data exploration revealed that even novice users were able to gain data insights from visualization at the same level of skilled users. This two findings suggest even though there may be interpretative bias due to different levels of visualization literacy and stakeholders, there is value to be gained from visual representation of data no matter who the end user is.

A review of existing literature to date on data visualization for learning analytics in an online learning system validates data visualization as a pivotal determinant in an information system. For example, according to Roca, Chiu and Martinez [27], data visualization that is appealing and easy to understand significantly influences the system use and user satisfaction of an online learning system. This indicates that users of an online learning system require a high level of information quality where the visualized information presented will improve their productivity in performing tasks. Vazquez-Ingelmo et al. [28] observed that data visualization to be an indicator of information quality in learning analytics. This is because data visualization supports not only educators but also students in gaining insights about their teaching or learning. Another study by Park and Jo [29] indicated that although the learning analytics system used in the mentioned study did not significantly impact students’ academic performance, but when students viewed the data visualization of the analytics data they had an increased understanding of their subject matter and overall satisfaction of the learning analytics system. Another example that presents the power of data visualization is the Social Networks Adapting Pedagogical Practice (SNAPP) tool which incorporates the social network analysis in learning analytics [30]. The networking visualized students’ interaction in the learning management system which helped to identify patterns of their discussion activities effectively. Course Signal in Purdue University also utilized data visualization to give alarm signals that helped prevent students dropping-out and enhances their success [31]. Wise and colleagues [32] also provided a good example of how data visualization can be effectively conducted in learning analytics to promote quality of knowledge construction among students.

But there is an important gap that should be addressed, there is a need for data visualization in learning analytics to be grounded by theory rather than just being functionality and feature centric. Gasevic, Kovnovic & Joksimovic [15] mentioned previous studies rather focused on the functionality and features of data visualization in learning analytics, highlighting its potential impact on online learning and neglected to relate it to established theories in technology acceptance. Liu, Nersessian and Stasko [33] criticized that just building data visualization based on established principles is insufficient to contribute to body of research knowledge nor guide evaluation in the field of information system. Data visualization should be more than just attractive designs but instead we should incorporate theoretical considerations pertinent to the purpose a visualization aims to achieve such as support for computer aided learning [34]. In short, the crucial role of data visualization towards a successful online learning system has to be designed based on existing related theory, in order to validate its associations between the related system constructs.

It is worth noting here that many research work that focus on learning analytics are grounded by learning theories such as self-regulated learning [62], constructivism [63], collaborative learning [64] and others [65]. But only a few from the perspective of information systems [36] and with Technology Acceptance Model (TAM) [17] most frequently used [42]. However, TAM overlooks assessment of technology utilization [66]. DeLone and McLean’s Information Systems (IS) Success model on the other hand assesses the technology used by looking at the impact of overall quality (information, system and service). With the implementation of online learning system, it is important to explore the overall quality of the system. Due to its simplicity and predictive accuracy in the field of technology acceptance, IS Success model and TAM have been widely used in various domains, including education technology [36,37,38, 42,43,44, 47], as such this is believe to an important addition to the research model. Hence in view of this, this research puts forth an integrated IS Success Model with perceived ease of use and usefulness from TAM to explore the relationship of data visualization and learning analytics in its impact on academic performance.

2.2 Information Systems Success Model

DeLone and McLean’s, IS success model is regularly used in the field of technology to examine the factors affecting IS success and its impact. It is also consistently cited in various research works throughout the years. The original model includes six major variables that measure different aspects of IS success, which are information and system quality, and use, user satisfaction, individual impact and organizational impact [35].

Information quality and system quality was highlighted as the primary two antecedents for predicting system use and user satisfaction. This model differentiates from other models by focusing on users’ beliefs and perceptions as an indicator of system use. In the IS success model, system quality refers to the features or characteristics required for a good IS system such as ease of use, flexibility, reliability and functionalities, while information quality refers to the effectiveness of how an IS system captures input and generates output such as its timeliness, accuracy, completeness and usability.

The causal relationships in the model have been used and tested in numerous studies especially in the field of online learning [36,37,38]. These studies are mostly aimed to investigate the success factors responsible for the implementation of an online learning system which contributes to the performance of students, and the results obtained generally found that the hypothesized relationships to be significant.

However years later, DeLone and McLean [16] made changes to their original model by adding another determinant of system use and user satisfaction into the model, which is service quality (see Fig. 1). This new determinant focuses on quality of support in terms of ease of maintenance and end-user support to the system user. Hence in total there are three major dimensions of qualities that contributes to the system use and user satisfaction, but the strength of their significance depend on the context of the studies and the type of statistical analyses applied. In general, information and system quality are advocated to measure success of information system for individual level while service quality inclines towards organizational level.

Fig. 1.
figure 1

The updated IS success model.

Full size image

An ineffective development and implementation of IS systems that lack quality functionalities, inconsistent user interface design, and end-user support can be costly for higher education institutions as it requires a substantial amount of resources and monetary investment to remedy [40]. Thus DeLone & McLean IS success model’s system quality, information quality and service quality are vital in ensuring the successful implementation of technology for its users and the organizations as a whole.

2.3 Technology-Acceptance Model

TAM another accepted and consistently cited model was conceptualized based on Fishbein and Ajzen’s [41] Theory of Reasoned Action, which proposed two crucial factors for understanding behavioral intention – perceived usefulness and perceived ease of use of technology. Perceived usefulness is the degree to which an individual believes the technology will help his/her productivity, while perceived ease of use is the degree to which an individual believes that such technology can be used with minimum to little mental effort [17].

Due to its simplicity and predictive accuracy in the field of technology acceptance, TAM has been widely used in various domains, such as learning analytics [42,43,44], mobile learning [45], virtual laboratory [46], e-learning [47], e-procurement [48], healthcare systems [49], sustainable energy technology [50] and also ride-sharing application [51]. Many studies in the related domains have shown that the perceived usefulness is one of the stronger drivers influencing one’s behavioral intention in using the technology for practice.

In the following years, the original TAM was revised with a new proposed addition by Venkatesh and Davis [52] by including cognitive instrumental processes (output quality, result demonstrability, and job relevance), social influence elements (image, subjective norm, and voluntariness) and also experience to explain on the factor usage intention as well as perceived usefulness. With that the enhanced model is known as TAM2. Venkatesh [53] further investigated the significant role of perceived ease of use by including two classes of antecedents, which are anchors (self-efficacy, facilitating conditions, computer playfulness and anxiety) and adjustments (system-specific perceived enjoyment, system usability) (Fig. 2).

Fig. 2.
figure 2

Technology acceptance model (TAM) [67].

Full size image

In 2008, Venkatesh and Bala proposed another extended model known as TAM3 that integrated TAM2 and the extended perceived ease of use variable by reviewing available research works in relation to determinants of perceived ease of use and usefulness. And subsequently a new comprehensive integrated model encompassing both variables antecedents were posited and empirically analyzed. The results indicated that they were generally consistent with those obtained from earlier research and most importantly, none of the determinants of perceived ease of use had significant effect on perceived usefulness and vice versa. However, perceived usefulness still emerged as the stronger predictor of behavioral usage intention [54].

3 Conceptualizing of the Integrated Learning Analytics Model

This research intends to integrate the IS success model and TAM to determine the impact of data visualization of learning analytics on educator’s satisfaction and academic performance. The research model of the present study is illustrated in Fig. 3.

Fig. 3.
figure 3

Integrated learning analytics model.

Full size image

In this research model the following previous constructs from IS success model are renamed to create a more relevant label aligned to the present study: User Satisfaction is renamed to “Educator’s Satisfaction”, Intention to Use/Use to “Learning Analytics (LA) Use in System” and Net Benefits to “Academic Performance”. The rest of the constructs are maintained as they are, which includes System Quality and Information Quality. The constructs integrated from TAM are Perceived Usefulness and Perceived Ease of Use. In addition, two new constructs are added as antecedent to System Quality and Information Quality, which are LA Features and Data Visualization respectively.

These nine constructs proposed from literature review and the associated hypotheses are described below.

3.1 System Quality

This construct measures the system’s features and functionalities related to the online learning system. The empirical research by Aldholay et al. [38] showed that system quality has positive effect on the use of an online learning system. The hypotheses are as follows.

H1: System Quality is positively associated with the LA Use in online learning system.

H2: System Quality is positively associated with Educator’s Satisfaction on using online learning system.

3.2 Information Quality

This construct measures information generated by the online learning system specifically on students learning data. Previous studies showed that Information Quality affect both User Satisfaction and their Use of it [14, 55]. The hypotheses are as follows.

H3: Information Quality is positively associated with the LA Use in online learning system.

H4: Information Quality is positively associated with Educator’s Satisfaction on using online learning system.

3.3 Perceived Usefulness

This construct measures the degree of a user’s subjective belief that using the online learning system would promote and ease their task/problem within an organizational context. Theoretically, perceived usefulness affect both system use and user satisfaction of an online learning system [39]. Studies by Rienties et al. [44] and Al-Fraihat et al. [56] showed that perceived usefulness is a pivotal factor in determining user satisfaction and their intention to use educational technologies. The hypotheses are as follows.

H5: Perceived Usefulness is positively associated with the LA Use in online learning system.

H6: Perceived Usefulness is positively associated with Educator’s Satisfaction on using online learning system.

3.4 Perceived Ease of Use

This construct measures the degree of user’s subjective belief that using the online learning system would be free of cognitive effort. Studies have investigated the relationship between Perceived Ease of Use and System Use and User Satisfaction on learning management system [57]. The findings confirmed that Perceived Ease of Use is a strong predictor of User Satisfaction. Thus, the hypotheses are as follows.

H7: Perceived Ease of Use is positively associated with the LA Use in online learning system.

H8: Perceived Ease of Use is positively associated with Educator’s Satisfaction on using online learning system.

H9: Perceived Ease of Use is positively associated with Perceived Usefulness.

3.5 Learning Analytics Features

This construct measures the LA functionalities and features offered by the online learning system such as student content interaction analytics, quiz analytics, module analytics, single lesson analytics and social interaction analytics. Studies have shown the importance of System Quality in affecting the use of the system [14]. With the introduction of analytics into the online learning system, it will influence the system quality offered, which subsequently lead to higher user satisfaction as well as increase use of it. This leads to the following hypothesis.

H10: LA Features is positively associated with System Quality.

3.6 Data Visualization

This construct measures the desired visual content characteristics that are valued by users. As mentioned in Sect. 2.1, the type and design of data visualization provided will have an impact on the online learning system. Schaaf et al. [24] suggested that data visualization has a direct effect on the information one received. Learning analytics for online learning systems rely heavily on the information collected, hence it is believe data visualization is one of the factors which affect the information generated, having the characteristic quality of actionablity and explanability on the data presented. Thus, the hypothesis is as follows.

H11: Data Visualization is positively associated with Information Quality.

3.7 Learning Analytics Use in System

This construct measures the extent of which learning analytics is used in the online learning system such as the analytics and reporting elements are actually used/accessed. The LA Use in System is most useful in a setting where the usage of learning analytics is voluntary rather than mandatory [58]. A recent study by Koceska and Koceski [55] on factors affecting students satisfaction of online learning systems concluded that usage has a significant effect on one’s satisfaction and their performance, which is also in-line with previous work done [59]. The hypothesis is as follows.

H12: LA Use in System is positively associated with Educator’s Satisfaction on using online learning system.

3.8 Educator’s Satisfaction

This construct measures the expectation of educators on the adopted online learning system, as compared to the available information. It is considered to be one of the most important constructs in the IS success model as it will determine the overall success of a system. Many studies have investigated the mediating effect of user satisfaction on performance impact of online learning [36, 39]. DeLone and McLean [16] also stressed that user satisfaction affect the use of a learning system. If they are satisfied with the system, their adoption of it into their activities are higher. The hypothesis is as follows.

H13: Educator’s Satisfaction is positively associated with overall Academic Performance.

3.9 Academic Performance

This construct measures the net benefit or impact caused by the implementation of data visualization to the online learning system. The ultimate objective of the online learning systems use is to enhance learning pedagogy and improve academic performance. Hence student academic performance indicators (e.g. grades) is used as an assessment of individual students’ to measure the impact of the change introduced. It has been a focus of many studies in this field [55, 60].

4 Discussion and Limitations

A conceptual model is a very valuable guide to real-world implications of a system [61]. This section will discuss on the implication and limitations of the integrated Learning Analytics model for an online learning system.

This paper proposes that data visualization as a construct is a significant influence in producing quality information for decision-making in the integrated model. The nature of data visualization is to present large amounts of data in visually actionable information that can ease the decision-making process as well as justifying findings. This factor must be recognized in learning analytics systems as it will influence the information quality produced, leading to effective decision-making among stakeholders such as educators in improving their teaching approaches.

This research has expanded the IS success model [16] along with TAM [17] to be put into operation for a new targeted context. A range of online learning systems have been studied in respect to its adoption as such this research provides an insight into an integrated model that takes into account the value of LA Features and Data Visualization as an antecedents to System Quality and Information Quality of IS success model, with the aim to create a more comprehensive model for use in the online learning context.

Although this research provides new insight in theory, an experimental study with the implementation of the proposed model should be conducted to further justify its relevant. The model is presented as a theoretical means of addressing the role of data visualization in learning analytics systems. It is envisioned that upon analysis and further investigation of the model, it may support the development of effective and efficient learning analytics systems so that the potential of data visualization can be fully exploited to optimize the teaching and learning environment.

5 Conclusion

With data easily captured and available via online learning systems, data visualization is a fast growing field with new research work on techniques and tools to enhance the efficiency and effectiveness of online teaching and learning. Though the benefits of data visualization is evident, it must be grounded in proper theory and it is important that we focus on its impact in an overall system for an effective uptake instead of simply implementing it as an ornamental add-on. Therefore, a preliminary model which integrates the IS success model and TAM with an extension of LA Features and Data Visualization as antecedents are presented to investigate the role of data visualization and its impact. This preliminary integrated model aims to support researchers, practitioners and software designers who aspire to implement data visualization in an online learning setting. A prototype is currently being trialed and enhanced based on the proposed model and its result will be used to further develop, refine and validate the proposed model.