Keywords

1 Introduction

Over the years, technology has been evolving on a large scale, which has led many organizations to take advantage of this event and make efforts to implement learning management systems (LMS), which improves the quality and scope of learning processes [1]. The Education sector is one of the most interesting sectors to analyze regarding the use of this type of LMS tools [2]. In order for the LMS to fulfill its objective efficiently, certain aspects must be considered, such as motivation, interest, commitment, focus on the task, behavior and much more [3].

Sometimes, the efficiency of the LMS is affected by the lack of usability. This is because the user could invest more effort in trying to understand the software than in carrying out their activities. Therefore, there are two factors that must be considered for the success of the LMS: the quality of information and the way in which it is displayed to the user. According to Westfall, it was reported that 61% of people surveyed affirm that one of the inconveniences they had with LMS was learning how it works [4].

Therefore, usability is one of the key factors for the user to achieve satisfaction while using the LMS. This involves making the software easy to use and learn, with a user-friendly interface; displaying the content of a simple and orderly manner, thus preventing the user to focus on these issues [5]. Measuring usability is considered one of the great challenges, this is because there is no standard model which covers all the necessary usability attributes [6].

Heuristic evaluation is one of the most commonly used qualitative techniques to inspect software interfaces and find problems that affect usability [7]. On the other hand, quantitatively assessing the level of usability provides certain benefits, such as making objective comparisons between the software of the same type, which makes it easier for companies to make decisions to choose the best product [8]. With a numerical score, it favors the identification of the state of the level of usability of the software [9].

In this research, it is proposed to find those most important features about usability and electronic learning, as well as quantitative usability evaluations, taking into account heuristics and guidelines. To achieve this, a systematic literature review was carried out, following the Kitchenham methodology. Thus, the most important research on the subject was identified, which will serve as support to build a preliminary framework that allows quantifying the level of usability in those support tools for learning, in the near future.

2 A Systematic Literature Review

To identify those important key aspects that LMS must meet, a systematic review was conducted. The purpose of identifying those aspects was to analyze them, systematize them, and to propose in a future research, a framework to assess the usability and user experience based on the most important identified aspects.

The present systematic review was performed based on the parameters defined by Kitchenham and Charters [10]. In this case, the activities that were performed are: (1) definition of the research questions and search strategy, (2) selection of the primary studies, the extraction of the papers and (3) analysis of the results.

2.1 Research Questions

The main objective was to summarize some studies related to usability, user experience, learning management systems and usability evaluation methods. We used the PICOC table criteria in order to do this review according to the protocol established by Petticrew and Roberts [11]. In addition, we employed synonyms and related terms to find better results. These criteria are shown in Table 1.

Table 1. General concepts defined using the PICOC criteria.
Full size table

Based on the concepts established using PICOC, we defined the following research questions oriented to identify relevant aspects in the design of graphical interfaces for learning management systems:

  • RQ1: Which are the most relevant aspects of usability and user experience?

  • RQ2: Which are the most important characteristics in LMS?

  • RQ3: Which are the activities/characteristics of the usability evaluation models?

  • RQ4: Which are the guidelines considered in other domains to evaluate usability?

2.2 Search Strategy

We defined our search strategy based on the general concepts. Some synonymous were selected to achieve a more comprehensive search. The search process was performed by using four recognized databases to search for primary studies: ACM Digital Library, IEEExplore, SCOPUS and SpringerLink. Grey literature was excluded since it is not peer reviewed.

In this phase, the search chains or queries that were used in the search engines of each database were formulated. It is important to mention that the syntax of the queries could vary according to the database that is employed. After grouping a series of concepts using the connectors AND/OR, the resulting search string was the following:

(“usability” OR “user experience” OR “UX” OR “HCI”) AND (“heuristic” OR “heuristic evaluation” OR “usability evaluation” OR “model evaluation usability”) AND (“quantify” OR “quantitative” OR “approach quantitative” OR “comparative” OR “comparative analysis” OR “methodology to evaluate usability” OR “quantify usability” OR “measure usability” OR “method to evaluate usability”) AND (“interface” OR “software” OR “web” OR “system” OR “satisfaction” OR “characteristic” OR “guideline” OR “design” OR “methodology” OR “case study”) AND (“elearning” OR “e-learning” OR “learning management system” OR “LMS” OR “education” OR “higher education” OR “university” OR “e-learning system” OR “web platform to teaching” OR “web platform” OR “web system to support teaching”).

2.3 Search Process and Data Extraction

In order to determine if an article must be considered as relevant, we defined the following inclusion criteria: the study should present a methodology, framework or study case in which the usability and user experience are evaluated. In the same way, we defined the exclusion criteria: in the study, the specialists do not apply a usability or user experience evaluation in an intangible product as a software. The automated search for our systematic mapping review was performed on October 20th, 2018. Table 2 shows the search results that were found. In addition, Table 3 shows the selected studies from the four databases used in this research. These studies were selected by discarding the studies that do not meet the inclusion criteria and present any of the exclusion criteria.

Table 2. Search results for RQ1, RQ2, RQ3 and RQ4.
Full size table
Table 3. Selected primary studies.
Full size table

In addition, some secondary studies were identified applying the inclusion and exclusion criteria to the references established in the most relevant of the primary studies. These secondary papers are shown in Table 4.

Table 4. Selected secondary studies.
Full size table

Once these studies were obtained through the secondary review process, these were cataloged based on the relevant works. This information is shown in Table 5.

Table 5. Mapping of primary studies with secondary studies.
Full size table

3 Results of the Review

3.1 Relevant Aspects of Usability and User Experience

There were 18 papers that answer RQ1 (SS1, SS4, SS6, SS14, SS24, SS26, SS27, SS29, SS30, SS35, SS36, SS37, SS38, SS44, SS47, SS51, SS53, SS54). The usability/UX aspects that found in the systematic review are shown in Table 6.

Table 6. Usability and UX Aspects found by the Systematic Review Process (RQ1).
Full size table

3.2 Important Characteristics in Learning Management Systems

In the same way, to answer RQ2, 14 papers were selected (SS5, SS9, SS13, SS16, SS19, SS21, SS28, SS31, SS32, SS39, SS42, SS46, SS49, SS50). According to the authors, there are relevant design aspects that contribute to satisfy the learning objectives that are frequently requested by the educational institutions when they search for learning tools. We detail some of those design aspects in Table 7.

Table 7. LMS Characteristics found by the Systematic Review Process (RQ2).
Full size table

3.3 Activities and Characteristics of the Usability Evaluation Models

There were 11 papers that answer RQ3 (SS3, SS12, SS15, SS17, SS20, SS22, SS23, SS28, SS32, SS45, SS52). Some important strategies related to the process that must be followed in a usability/UX evaluation were obtained from these studies. These approaches were compared to decide how to establish the assessment framework. We show some of those strategies in Table 8.

Table 8. Characteristics/Activities of Usability Evaluations found by the Systematic Review Process (RQ3).
Full size table

3.4 Guidelines to Evaluate the Usability Attribute

Finally, the RQ4 is related with the principles used in the heuristic evaluation models which allow to get the usability level - quantitatively or qualitatively - in the platforms from other domains, that can be used as well for learning support platforms. There were 18 papers that were used to answer this research question (SS2, SS3, SS7, SS8, SS10, SS11, SS18, SS23, SS25, SS28, SS32, SS33, SS34, SS40, SS41, SS43, SS48, SS55). Some the guidelines found are shown in Table 9.

Table 9. Guidelines to Evaluate Usability found in the Systematic Review Process (RQ4).
Full size table

4 Conclusions and Future Works

The usability and UX have become critical aspects to be considered in the development of software products. Nowadays, these quality attributes represent the main concerns of the software industries, since ensuring a high level of ease of use and UX in the applications, leads to establish an environment of appropriate use for the interaction with the system. The new paradigm in the software development is not only about providing the users with a tool to achieve their goals but also to ensure that the user experience is quality enough to generate satisfaction on the end user.

In this study, we performed a systematic literature review following a recognized and widely used methodology. According to this protocol, we identified 105 studies, from which 23 were selected. Furthermore, we considered 32 secondary studies, that provided more information to 4 research questions. This work allowed to identify the most relevant aspects in both usability as learning. Additionally, this review allowed to find characteristics that help to quantify the level usability and UX and some heuristics/guidelines related to domain of study.

The objective of this systematic literature review was to obtain relevant information from previous research to build a preliminary framework that allows quantifying the level of usability and UX in learning support platforms, through a checklist. Therefore, a deeper analysis should be carried out to generate the results to the objectives for the construction of the framework. Also, some of the results must be complemented and validated by interviews and expert judgment, respectively.