Keywords

1 Introduction

Usability represents an essential quality attribute in the construction of software products [1]. Several evaluation methods have emerged as a consequence to ensure the design of interfaces that are usable, easy to use, understandable and attractive [2]. In the same way, there are also a number of methodological procedures that allow specialists to verify if the user interfaces meet the minimum standards of usability [3]. In this wide range of available tools to measure usability, the questionnaires stand out for being a straightforward method that does not demand the consumption of multiple resources. However, few are the proposals of questionnaires cover all the aspects that are relevant in a particular software category. Most of these questionnaires that exist in the literature to evaluate the usability of a software product, are focused on generic and global aspects. When developing applications of very specific categories such as the medical, mathematical or even electronic commerce domain, new features appear that must be considered to ensure the ease of use of the final product [4]. It is not the same for a specialist to analyze the usability of an app designed for mobile devices than a touchscreen-based system or an interface embedded in a technological device. Attention should be paid to new features since each type of system has its own peculiarities.

In e-commerce web applications, there are aspects related to the financial transaction, the purchase process, the feedback from users on products and services, the security of the sensible information (credits cards and personal information), etc., that affect the end-user experience [5]. These aspects, despite being important in the analysis of the user experience, are not covered by the current evaluation proposals, especially by the questionnaires. However, Bonastre and Granollers [6], through an exhaustive study of Nielsen's heuristics and a literature review in conjunction with collected information from two consulting firms, developed a new proposal that has proven to be effective in this area. This new set of guidelines has been validated in the assessment of twenty-one E-Commerce websites, demonstrating that it is possible to identify a greater number of problems when this new proposal is used compared to the current evaluation questionnaires [7]. In this study, we have tested the approach proposed by Bonastre and Granollers in the evaluation of the three leading E-Commerce websites in Peru. The intention of this research was to provide more experimental evidence on this evaluation instrument and determine if it is applicable to websites from a different cultural context. Likewise, it has been possible to determine whether the most important local companies in the country take this quality aspect into consideration, such as usability for the design and development of their websites.

This document is structured as follows. In Sect. 2, we detail the new assessment questionnaire that has been used to measure the level of usability of the Peruvian e-commerce websites. In Sect. 3, the design of the case study is described. In Sect. 4, we present the obtained results and a brief analysis of them. Finally, in Sect. 5, conclusions and future works are established.

2 A New Usability Evaluation Proposal

Within the wide range of usability evaluation methods that exist [8], questionnaires represent a more feasible instrument to use than procedures that require, for instance, preparation of material, training, and recruitment of participants, such as user tests. In the same way, questionnaires solve the problem of the qualitative approaches. Using a questionnaire, it is possible to quantify the level of usability of a software product [9]. Obtaining a numerical value on the level of usability offers the possibility of making comparisons. Many times, when the engineers employ parallel design, and there are multiple proposals, quantifying the usability can be a decisive factor in selecting the best option. Similarly, when a company intends to know how far it is from its main competitors, this value can be helpful for decision making [10]. Another advantage of questionnaires is that there is not much complexity to use them. The evaluators must interact previously with the software product and after that, they must answer about the degree of compliance with each of the items established in the questionnaire. One difference that has implications in the calculation of the final value is that some proposals establish YES/NO questions for the evaluation, while other approaches set up a Likert scale for the items.

In the usability questionnaires such as SUS [10], SUMI [11], CSUQ [12], ASQ [13], etc. that are widely recognized by the academia and industry, the items have the purpose of evaluating a specific usability construct or sub-attribute. Many specialists have made an effort to define the concept of usability and have concluded that this feature is composed of others. One of the most accepted approaches by the scientific community is Nielsen's proposal [14] that defines usability in five components: (1) learnability, (2) efficiency, (3) memorability, (4) errors, and (5) satisfaction. Similarly, another highly employed approach is the definition specified by the the ISO/IEC 9126 standard [15] that establishes that usability is composed of five sub-attributes: (1) understandability, (2) learnability, (3) operability, (4) attractiveness, and (5) usability compliance. The items included in the questionnaire are intended to evaluate the different aspects that comprise usability. However, the known and conventional proposals have been developed with an exclusive focus on usability and its sub-attributes, leaving aside aspects of the software product. Through previous studies, it has been demonstrated that to ensure a good user experience, it is necessary to focus on specific attributes of the software product category. For example, it is not possible to evaluate the user experience that a videogame provides, if aspects of gameplay and playability are not taken into consideration [16]. In the same way, it is not possible to carry out a proper inspection of the user experience that a banking system generates, if the perception of security conveyed by the system is not taken into account and analyzed [17]. Electronic commerce web applications are not exempt from this scenario, and in this specific area it is necessary to consider aspects such as: (1) the security of personal information and financial transactions, (2) the degree of available information that can be visualized about the products and services offered on the website, (3) user help and support functionalities, (4) the possibility of obtaining references and feedback from other users about the products and services, (5) the capacity of the system to be customized according to user preferences, among others [6].

Given the gap of the traditional proposals present, Granollers and Bonastre [6] developed a new questionnaire that addresses all aspects that are relevant and essential in a usability evaluation to an E-Commerce website. After an exhaustive review of the literature and an in-depth analysis of the most important aspects reported about e-commerce applications on the web, the authors formulated a questionnaire with 64 items grouped into 6 categories: (1) Need recognition and problem awareness, (2) information search, (3) purchase decision making, (4) transaction, (5) post-sales behavior, and (6) factors that affect the UX during the whole purchase process. Most of the questions have been designed to be answered on an agree-disagree Likert scale from 1 to 5, where: 1 represents “strongly disagree”, 2 “disagree”, 3 “neither agree nor disagree”, 4 “agree” and 5 “strongly agree”. Nevertheless, a smaller percentage has been formulated as YES/NO questions. In Table 1 details an example of the some of the questions formulated as part of the proposal.

Table 1. Example of a selection of questions formulated in the new usability evaluation proposal for E-Commerce websites [6].
Full size table

Unlike the current assessment instruments, the new Granollers and Bonastre proposal [6], is directed to be answered by specialists who have an extensive knowledge about the concept of usability and have experience in the area of Human-Computer Interaction. On the contrary, most of the proposals mentioned above as SUS [10], SUMI [11], CSUQ [12], ASQ [13], etc., aim to analyze the perception of end-users. The traditional proposals have an approach to collect the opinion of the participants regarding a software product. In the case of the new proposal, the questions are aimed at determining whether the system meets certain characteristics that would significantly improve the degree of user experience. Likewise, the proposal includes technical questions that would be hard to answer for people who are unfamiliar with the topic. The purpose of this novel contribution is for the technology specialist to be able to answer based on their technical knowledge and expertise, and thus determine both the degree of usability and the aspects of improvement.

To calculate the final numerical usability value of the system, the authors have established that a sum of all the values is necessary. However, those questions that are YES/NO would receive a value of 5 if the answer is affirmative and a value of 1 if the answer is negative. Considering that there are 64 items, the maximum value obtainable is 320 and the minimum value is 64. However, in an update of the proposal, Granollers establishes that when those answers that 64 points would represent a null level of usability of the system (because it is induced that all the answers are of type “Strongly disagree” or in their defect negative for the case of questions YES/NO) and 320 points would represent that the system fully covers all the aspects that are necessary to guarantee an appropriate and satisfactory level of user experience [7].

3 Design of the Case Study

In this study, we have used the usability evaluation proposal established by Granollers and Bonastre [6] for e-commerce websites. The research was intended to test this new approach in the assessment of the three main e-commerce websites in Peru. In this way, what is intended is to place more evidence that can be used afterward to generalize the use of this instrument in different contexts and settings. Likewise, it has been possible to determine the degree of importance that both usability and user experience have for local companies.

The websites to be evaluated were selected according to a previous research [18] in which an accessibility evaluation was carried out to determine the Peruvian companies that meet the guidelines indicated in the international WCAG 2.0 standard. From this study, three websites were selected:

Regarding the process, Granollers and Bonastre [6] determine that in each evaluation of a specific website, 3 to 5 evaluators must participate. In this experimental case study, three specialists in the field of Human-Computer Interaction were requested to participate. It can be established that all specialists that agreed to contribute with this study, have the same degree of expertise because they all have multiple projects carried out in this area as well as published research papers. In addition, all they have technological formation in careers related to Computer Science.

Once the evaluators kindly and voluntarily agreed to participate in our study, they received an Excel template that was previously prepared. This template detailed an informed consent protocol that the evaluators had to sign to be part of this case study. Likewise, instructions for the evaluation process were provided. The evaluators were requested to spend 30 to 45 min browsing each website. After the interaction, the specialists were asked to answer each of the items in the questionnaire. This process was repeated at different time intervals for the evaluation of each of the websites. However, it was the same evaluators who participated in the three evaluations of the three websites. Likewise, the template was configured with both the questions and the possible answers to each of the items, to avoid possible errors in the evaluation.

Once the evaluators finished conducting the evaluation, the templates were collected to be analyzed and processed by the authors of this research. The results were interpreted taking into consideration the percentage of compliance with all the aspects indicated in the proposal. Figure 1 summarizes the evaluation process.

Fig. 1.
figure 1

Experimental design of the case study

Full size image

4 Analysis of the Results

The usability evaluation was performed during the month of January 2021. Due to local restrictions on holding face-to-face meetings because of the Covid-19 scenario, the evaluation was carried out virtually. The templates were sent to the specialists via email and subsequently the information was collected in the same way. Since there is no automated system, the results have been analyzed and interpreted manually.

Table 2, 3 and 4 detail the usability scores obtained for each of the evaluated websites. We observe that the scores obtained are uniform, which is an indicatory that the three specialists have evaluated with the same orientation, and that the questionnaire could be reliable. It is possible to determine some rating are different in some categories, however this does not mean that one evaluator is more accurate than the others, simply indicates that to increase the reliability of this evaluation would need to introduce more specialists into the experiment. We can also appreciate that although usability levels are not are high, they are almost remarkably similar values.

In Table 5, the final score of usability for each website is presented. From these obtained results, it is possible to highlight that apparently there is a correlation between the relevance of the website and the level of usability it presents, given that this selected sample corresponds to the three most relevant Peruvian local websites chosen in order of importance. Likewise, it is possible to determine that despite being the most representative websites on the E-Commerce domain in Peru, the level of usability identified is quite low, evidencing that there is indeed little concern currently by companies about this quality attribute.

Table 2. Average of the scores obtained for each category in the usability evaluation conducted to the E-Commerce website of www.sagafalabella.com.pe.
Full size table
Table 3. Average of the scores obtained for each category in the usability evaluation conducted to the E-Commerce website of www.ripley.com.pe.
Full size table
Table 4. Average of the scores obtained for each category in the usability evaluation conducted to the E-Commerce website of www.linio.com.pe.
Full size table
Table 5. Final scores on the usability level of the evaluated websites.
Full size table

Through an analysis of the total scores obtained by all the evaluators: 181, 176, 208 for SagaFalabella, 171, 178, 172 for Ripley and 165, 149, 156 for Linio, it is possible to conclude that the level of usability of E-Commerce websites in Peru is low. Despite being frequently visited, widely promoted websites, these do not consider all the necessary aspects to guarantee a satisfying interaction experience.

5 Conclusions and Future Works

Usability is a quality attribute that must be considered in the development of software products to guarantee success in the market. This becomes much more important for companies dedicated to the field of electronic commerce since there are multiple options on the web and the competition is high. Given the relevance of usability and the need for development teams to ensure that products are easy to use, intuitive, understandable and attractive, methods have been developed that evaluate whether a software product meets the features and aspects that allow guaranteeing a satisfying interaction experience.

Questionnaires have emerged as one of the most used and widely accepted methods. In addition, they are easy to use and apply, as they do not require many resources like other existing methods. However, one of the problems with the current proposals is that they do not cover all the aspects that affect the experience of the end-users. To face with this problem, Granollers and Bonastre carried out a systematic review of the literature in search of identifying all the most important aspects that should be present in E-Commerce websites. Based on an analysis of several studies and the information obtained from two consulting companies, they developed a new questionnaire composed of 64 items that covers, in addition to usability aspects, the characteristics that every e-commerce website must have in order to be usable. Most of the items in the questionnaire have been constructed under a 5-point Likert scale, where the specialist must verify the degree of compliance of the item by answering from 1 to 5, where 1 is referred to “strongly disagree” and 5 is referred to “strongly agree”. However, some items have been established as YES/NO questions. Likewise, the proposal is accompanied by a mode of interpreting the results, whose score can range between 64 and 320, depending on the degree of compliance with the aspects.

The objective of this study was to use the new usability evaluation proposal in order to generate more evidence for a future generalization of this evaluation instrument, and also, to determine if the main companies in Peru dedicated to the electronic commerce sector are concerned that their websites are usable and allow the achievement of the user's goals. In this sense, the voluntary participation of three specialists from the Human-Computer Interaction area was requested, who kindly agreed to determine, based on the new evaluation questionnaire, the level of compliance of the three most important websites in Peru.

With reference to a previous study, the three most relevant websites were selected for evaluation. A virtual process was established because of the current scenario that makes it not possible to carry out the experimental case study in a laboratory. The results show that, despite being the most relevant platforms in the country, they must improve many aspects of usability. There is little concern on the part of companies in offering websites that are intuitive and that generate satisfaction to end-users. This coincides in a certain way with the previous results of an accessibility evaluation carried out to these same sites, in which it was evidenced that despite being the most visited sites, they were not complying with the minimum international accessibility standards, restricting the use of these technologies and undermining the right that all people with different abilities and capacities have to technological advances. Regarding the evaluation instrument, it was possible to determine that the results obtained by the different specialists, despite being low, are similar. It is possible to mention that the instrument is reliable. However, it is necessary to carry out more experiments and apply statistics to confirm this statement.

Finally, in relation to future works, it is expected to use this new proposal in other scenarios, possibly for the evaluation of E-Commerce websites from other countries and even to incorporate a greater number of specialists to analyze whether the degree of correlation varies. With a greater amount of data from future experiments, it would be possible to apply statistics that allow corroborating the internal and external validity of the instrument in order to generalize its use to any scenario. The intention of future research that arises from this work is to provide an instrument that can be used by academia and industry for the development of high-quality software products that are usable and that generate satisfaction to end-users.