Keywords

1 Introduction

The growing number of online stores have brought tension among many stores on the web as competition grows. More options for the consumer means that businesses are more prone to less traffic and a lower sale rate. Apart from typical electronic shops an important area of e-commerce is the sector of online auctions with several research fields. These include the analysis of behavior of users, the optimal design, the use of data and integration auctions with business models [1]. Other research topics are related to psychological aspects such as fear and distrust [2]. Online auctions are analyzed from the perspective of learning processed and acquiring knowledge about relations between bids [3]. Quantitative methods, like structural econometric, are used for price prediction [4] and identification of determinants of prices [5]. Analysis of decisions taken by online auction users takes place [6] as well as searching for factors affecting repurchase intensions [7]. Other than process and algorithmic characteristic the usability of online auction platforms is subject to the research and results that navigation and interactions have highest importance priority [8]. Other research emphasizes the role of auction websites quality in trust and continuous usage [9]. Dedicated approaches are used for assessment of auction website quality [10] and analysis of influence of quality of auction platforms on customer loyalty [11]. To receive better results business owners use analytic software [12], web mining techniques [13] or conversion maximization systems [14]. What is even more serious is realizing what single factors are affecting the performance of online platforms and customer loyalty [15]. Some major things to establish is the building of trust [16, 17], making sure the systems are top quality [18, 19], making sure there are levels of security and privacy [20], how accessible it is [21], development or international versions [22], fixing any critical problems [23] and pushing forward new features that help consumer satisfaction and the usability of the website [24]. To help observing the website quality, different types of methods based on key factors affecting websites assessment are used [25,26,27]. Because the evaluation of websites is a multi-criteria issue, attempts of using Multi-Criteria Decision Analysis (MCDA) methods for evaluating the websites are observed.

Presented research is an extended version of earlier work [28]. However, the goal of this article is to make an assessment model of the most popular auction websites while implementing the PEQUAL methodology. PEQUAL methodology is based on extension of classical EQUAL method with the use MCDA methodology. It has its justification as application of the MCDA method makes it possible to carry out a broad analysis and correction of, obtained in research, website rankings, and of user’s preferences. This problem has importance for various sectors and website quality evaluation methods used today will allow doing this kind of side by side analyses only to a limited extent. The article is broken down as follows: Sect. 2 includes literature review, Sect. 3 shows the methodological framework of a proposed approach, Sect. 4 presents the findings from the study with conclusions in Sect. 5.

2 Literature Review

2.1 Websites Usability and Quality

Quality and usability are concepts related to each other and they comprise a similar semantic range [29]. There are many definitions of usability. According to ISO 9241, usability “extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” [30]. Next, in the norm ISO 25010:2011 concerning the software programs, usability is defined as “the ability of software to be in intelligible, easy to learn and use as well as attractive to the user in specified circumstances” [31]. According to Nielsen, on the other hand, “usability is a complex concept consisting of many factors such as: learnability, efficiency, memorability, errors and satisfaction [29]. Nielsen specifies that usability is a quality attribute describing how a user interface is easy to use [32]. One can list many papers dealing with the research into website usability. Fernandez, among other things, divides usability research methods into five groups: user testing, inspection methods, inquiry methods, analytical modelling and simulation methods [33]. Usability testing methods can also be divided in a different way, for example, into expert methods and user inquiry methods [34].

Expert methods are obtaining research results from a group of experts or a single expert examining a website [35]. One can indicate such methods as heuristic evaluation, guideline review, Cognitive Walkthrough, Action Analysis, Analytical Modeling or Inspections [36, 37]. The methods are a collection of instructions, good practices or general rules on the basis of which experts and users evaluate a website and find out potential problems [33]. Often, the methods are based on heuristics (e.g. Nielsen’s heuristics [29]). In the case of Inspections, experts carry out inspections of features and functions offered by a website as well as conformity to standards (for example reaction time) [35].

The second group dealing with user inquiry methods is characterized by the fact that research results are obtained on the basis of activity of users in a website [33]. Here, one can list the following methods: Interviews, Focus group, Surveys, Questionnaires. In the Interviews method, the expert asks the user questions concerning the website [29]. The interview may be based on questions prepared in advance. However, the expert is also able to ask additional questions. In the Focus group method, a group consisting of several people, supervised by the expert, holds a discussion [35]. The expert moderates the discussion in order to obtain essential information on the users’ needs with regard to the website [36]. Thanks to Surveys one can collect users’ opinions about elements of a website, acceptability of solutions adopted in the website or possible errors [34]. In a questionnaire method, users answer prepared open-ended or closed-ended questions and respondents express their opinions about the website in a verbal form or by means of a questionnaire [29].

The last group of methods is based on tests involving users. One can distinguish here methods such as Thinking Aloud, Thinking-Aloud Protocol [33], Question-Asking Protocol, Performance Measurement, Log Analysis, Web traffic analysis, [34] or Field Observation) [35]. In the Thinking Aloud method [37], by means of testing scenarios, a user shows experts his or her way of perceiving a website. It leads to the identification of key interactions and problematic elements of the website. There are different variants of this method, for example, Constructive Interaction, Codiscovery Learning, Retrospective Testing, Coaching Method, Question-Asking Protocol [33, 35]. Another method, Performance Measurement, consists in collecting numerical data while a user is using a website [35]. Next, the data is processed and performance measurements of a solution are obtained. The measurements can be, for instance, a number of tasks or time necessary for a user to do a task [33]. In the Log Analysis method, an expert or software analyses data related to a user’s navigation on a website. Clicktracking or Eye Tracking can also be counted as the Log Analysis method [38, 39]. In the Field Observation method, an expert monitors a user’s interaction with a website in their natural working environment [29].

2.2 Website Evaluation Methods

Website evaluation methods use different models that look at quality and because of this they are different in what they use and in their structure [40]. To obtain users thoughts on websites, most of the time the sites use surveys and then grades are put on an n-degree Likert scale [41]. In Table 1 one can see described individual quality assessment methods for websites evaluation. For techniques utilizing surveys it is expected that the quantity of clients assessing a site ought to in any event add up to 30 [37].

Table 1. Characteristics of selected methods of website quality assessment
Full size table

The EQUAL method uses Quality Function Deployment which is a method that had the job of ensuring the means of identification and providing users’ thoughts on the quality of a material on different stages of it being made [42]. This method was able to look at e-commerce and government [43], websites successfully. Web Portal Site Quality appeared on the premise of a Technology Acceptance Model. The TAM is to clarify the impact of seeing, by the client, data framework qualities on his or her acknowledgment of the given framework. It depends on two quality measurements, that is, saw handiness and saw convenience [51]. The Model of Information Systems Success by DeLone and McLean incorporates data quality and framework quality [52]. The WPSQ strategy is utilized as a part of assessing entries conveying comprehensively characterized data and administrations [47].

The Ahn technique, comparatively to Web Portal Site Quality, was formulated with the utilization of Technology Acceptance Model (TAM) [53]. The primary adaptation of the Ahn method was to consider the impact of trust to bank sites on the acknowledgment by clients [54]. At the point when taking a shot at the strategy, the first TAM model was stretched out with ensuing components which were imperative from the viewpoint of the Internet: data quality, framework quality and administration quality. These components were acquired from a broadened Model of Information Systems Success of DeLone and McLean [55]. Likewise, quality attributes with respect to exchange: the nature of an item and its conveyance were included [44].

The SiteQual technique [45] appeared as a blend of the SERVQUAL and Data Quality [56] models. This model was developed on the premise of surveys concerning music online business sites [45]. While setting up the Website Evaluation Questionnaire technique, criteria utilized as a part of the Website User Satisfaction (WUS) model were utilized [57]. As in WUS, in each trademark there is one negative paradigm, which is utilized to check dependability assessment [46]. This technique appeared keeping in mind the end goal to analyse e-government sites, yet it can likewise be utilized to evaluate different sorts of websites [46]. The E-S-QUAL and E-RecS-Qual methods originate from the SERVQUAL technique utilized for contemplating and assessing administration quality [58]. They are a consequence of modifying the SERVQUAL scale to the requirements of administration quality evaluation on the Internet. Here, some assessment criteria in the SERVQUAL model were kept and new criteria basic for deciding e-benefit quality were presented. These techniques were utilized to assess quality on bank websites and also online business [49] websites. While setting up the Website Quality Model technique, Kano’s quality model was utilized, in which there are characterized three levels of clients’ desires with respect to the nature of an item or an administration: essential, execution, and energizing [48]. The WAES (Website Attribute Evaluation System) strategy is intended for surveying office and organization sites. It comprises of two gatherings of qualities portraying straightforwardness and intuitiveness of a site. A specialist’s assessment on a parallel scale is utilized in the technique [50].

The most appropriate technique, out of every single examined one, is by all accounts eQual, which is described by the most elevated formalization level. The eQual technique uses list of criteria (Table 2) as survey inquiries. While assessing, a Likert scale, which ranges from 1 to 7, is utilized. Weights of individual criteria are resolved similarly. Aside from criterial assessment, respondents likewise give general assessment of a website. On the premise of this appraisal, the unwavering quality of incomplete conclusions of each client is checked [43]. At the point when an accumulation of poll results has been assembled, an examination of the surveys is led concerning unwavering quality and inner attachment. To decide the dependability of aftereffects of a poll in the eQual technique, Cronbach’s alpha is utilized. It is accepted that the unwavering quality of results is fitting, if the estimation of coefficient alpha adds up to no less than 0.6 [43].

Table 2. Survey inquiries in eQual method
Full size table

The issue identified with a pragmatic utilization of the strategy is to pick up weights of criteria by method for surveys, in light of the fact that unequivocal revelation of clients’ inclination may produce mistakes in the exploration [59]. This is additionally affirmed by the creators’ examination, in which it was shown that weights of criteria got by method for surveys prompt to inaccurate choice arrangements [60].

2.3 Evaluation of Websites with the Use of MCDA Methods

Aside from talked about above, in the writing there are likewise endeavours at utilizing MCDA techniques for assessment of websites. It is justified since evaluation of websites is a multi-criteria issue, in which one needs to think about many dimensions and its measurements [61]. For example, Lee and Kozar [62] utilized the AHP method to assess business and travel websites. Chmielarz broadly utilizes scoring method to asses an extensive variety of business and e-managing oriented websites [63, 64]. Sun and Lin [65] assessed online business websites with use of TOPSIS technique. Del Vasto-Terrientes et al. [66] assessed traveller websites using ELECTRE-III-H method. Besides, in progress of Lin [67] and additionally Kong and Liu [68] AHP technique was utilized. Additionally a hybrids of different MCDA techniques are additionally utilized. In the paper by Bilsel et al. [69] determining the weights of criteria was directed by AHP method, while a positioning of healing facility websites was built with the utilization of the Promethee technique. Thus, Kaya [70] utilized the AHP technique to characterize weights of criteria, and used the TOPSIS method to build an internet business websites positioning. A mix of MCDA techniques: Simple Additive Weighting, Multiplicative Exponent Weighting, TOPSIS, concordance and conflict investigation techniques was utilized by Huang et al. [71].

The examination of use of MCDA methods in website assessment demonstrates that the vast majority of them utilized surveys to gather evaluations of websites. Concerning weights of criteria, pairwise comparison and the AHP technique are frequently utilized for this reason. Most of methods use a predetermined number of criteria. Just a couple papers utilized hypothetical bases recognizing the requirement for displaying both particular quality measures and criteria [66, 70]. Additionally, just in a few papers the robustness analysis was completed [69, 71]. Even though their usage is in early stage applying MCDA methods to assess websites has a more prominent potential than simply less formalized approaches.

3 Research Methodology

3.1 PEQUAL Methodological Framework

The authors’ methodology of website quality assessment named PEQUAL (Promethee – eQual) depends on the eQual method, which has its establishments in Quality Function Deployment. PEQUAL methodology is presented in details in [28]. To do observational research at initially, surveys were gathered from 32 clients. In the examination test, there were PC proficient clients who are knowledgeable about doing the shopping on the auction websites. Every one of them assessed 6 online auction websites: Allegro, Aukro, Ebay.com, Ebay.pl, Swistak and Trademe. The explanation behind selecting the above mentioned set of auction websites was the consequence of examination of legitimate rankings of top web based auction sites introduced, in addition to other things, in [72, 73]. Along these lines, 320 polls were gathered which then were checked as far as consistency unwavering quality and Cronbach’s alfa was resolved. Surveys evaluation was conducted with the use of the eQual method and the results of the questionnaires were also evaluated using the Promethee II method. Using Promethee II method and GAIA plane the broad analysis of the final websites ranking was carried out. Different research scenarios were taken into consideration in the process of aggregation of partial evaluations in a final ranking. In the first scenario, using Promethee II method, the aggregation of mean criterial evaluations into a overall evaluation with the use of pseudo criteria was conducted. In the next step the analysis of the obtained ranking was carried out with the use of the GAIA method, and two dimensions of analysis were taken into account: criteria and groups of criteria. In third step, apart from GAIA analysis, a sensitivity analysis of final ranking was also conducted. In real decision situations, expert evaluations obtained in the questionnaires can be characterized by some degree of uncertainty [25]. Therefore, in fourth step new, true criteria based model was constructed. Obtained results were compared with final ranking form first scenario (Step I). In the last step, obtained rankings were compared with the group ranking. Using PROMETHEE GDSS method GAIA analysis for particular decision – makers was performed. The presented practical approach is depicted in Fig. 1.

Fig. 1.
figure 1

Website evaluation process using PEQUAL methodology

Full size image

3.2 MCDA Foundations of PEQUAL Framework

Apart from eQual technique PEQUAL framework is methodologically based on Promethee II method. The method Promethee II is very popular MCDA method and it employs pairwise comparison and a outranking relation in order to select the best decision alternative. Also, the method uses positive and negative preference flows determining to what extent a given variant outranks other ones and to what extent it is outranked by other variants [74]. In the PROMETHEE II method, the decision-maker may choose from six preference functions: a usual criterion, a quasi-criterion with an indifference threshold, a criterion with linear preference and a preference threshold, a level-criterion with indifference and preference thresholds, a criterion with linear preference and an indifference area, or, finally, a Gaussian criterion [75, 76]. Main steps of PROMETHEE II procedure are: pairwise comparison of decision alternatives with regard to criteria, applying a preference function for each criterion; determining an alternative preference index, determining positive and negative preference flows for alternatives, and determining net preference flow [77]. A preference index of alternatives calculated according formula (1):

$$ \pi (a_{i} ,b_{j} ) = \tfrac{{\sum\limits_{k = 1}^{n} {w_{k}^{*} \varphi_{k} (a_{i} ,b_{j} )} }}{{\sum\limits_{k = 1}^{n} {w_{k} } }} $$
(1)

where \( \varphi_{k} \) means a concordance factor for a pair of alternatives compared with regard to a criterion k. Positive and negative preference flows are calculated according formulas (2) and (3).

$$ \phi^{ + } (a_{i} ) = \sum\limits_{j = 1}^{n} {\pi (a_{i} ,b_{j} )} $$
(2)
$$ \phi^{ - } (a_{i} ) = \sum\limits_{j = 1}^{n} {\pi (b_{j} ,a_{i} )} $$
(3)

Finally, a total order of decision variants (represented by a net preference flow) should be calculated (4):

$$ \phi (a_{i} ) = \phi^{ + } (a_{i} ) - \phi^{ - } (a_{i} ) $$
(4)

The Promethee GDSS method stems from and directly uses the Promethee II procedure. The Promethee GDSS procedure extends the Promethee II functionality with the concept of group decision making. The final aggregation of individual decision-makers’ evaluations takes place by means of the Promethee II method [74].

Apart from calculating a group ranking of Promethee II, in the Promethee GDSS method, the GAIA (ang. Geometrical Analysis for Interactive Assistance) analysis is also carried out. In the methodology of GAIA, information concerning a k-criterion decision problem presented in a k-dimensional space is projected on a plane, therefore, part of the information is lost. On the plane, among other things, a vector Л showing a compromise direction resulting from weights attributed to individual decision-makers (in a general case – to criteria) is presented [78]. Alternatives are represented by points and decision-makers’ preferences are symbolized by vectors. If decision-makers have similar preferences, the vectors are turned in the same direction, whereas contradictory preferences result in opposite senses of the vectors. If there is no connection between experts’ preferences, their vectors are turned perpendicularly to each other. The length of a vector denotes force of preferences represented by the vector. The closer the end of the vector to a given decision alternative, the more the vector supports the alternative in the ranking of results. When the analysis of GAIA points out that decision-makers’ preferences are in conflict with each other, the following is recommended: a change of weights attributed to decision-makers, a change of individual evaluations, a change of criteria, a change of alternatives or adding another decision-maker [77, 78].

4 Research Findings and Discussion

4.1 EQual Based Analysis

In the first step of research, reliability analysis of obtained surveys was performed based on Cronbach’s Alpha. Table 3 presents the results of these analysis. The minimum value of Cronbach’s alpha, which confirms the reliability of the survey is equal to 0.6. Therefore, it can be stated that the results of the survey are reliable.

Table 3. The results of reliability analysis
Full size table

Next, a overall value of eQual Index was determined. The scores of the total value for groups of criteria are presented in Table 4.

Table 4. Assessment results (eQual index) of auction websites according to eQual method
Full size table

The most preferred online auction websites, by the opinion of users, are the most popular in Poland are Allegro and Ebay.pl. This is probably since the respondents have the greatest experience in the use from these sites, and their habits related to the use of the auction platforms have their origin in these sites.

4.2 Promethee II Based Analysis of Solution

In the next step, Promethee II method was applied. The averaged data from survey results are the source for calculations of Promethee II. Table 5 presents performance table of considered alternatives.

Table 5. Performance table for Promethee II
Full size table

In the Promethee II method for each criterion a preference model with a linear preference function, with an indifference threshold q = 1 and a preference threshold p = 5 was used. A preference direction was maximized. The selected preference model was assumed in order to reflect, as accurately as possible, the model used in the eQual method. The final ranking of the decision varies, the preference values of input and output and net flows are presented in Table 6.

Table 6. The final ranking and performance of decision variants (linear preference function with thresholds)
Full size table

On the basis of comparison, Tables 4 and 6, we can note that the ranking of variants obtained using the Promethee II is identical with the ranking eQual. This indicates the correctness of the usage MCDA method in the website evaluation field. This also confirms correctness of usage with the Promethee II method as the alternative to eQual.

4.3 Graphical Analysis of Promethee II Solution

Additionally, the results obtained by the Promethee II were analysed using the GAIA methodology. Figure 2 shows the results of the individual criteria analysis carried out by GAIA plane.

Fig. 2.
figure 2

GAIA analysis for criteria

Full size image

The analysis of Fig. 2 demonstrates that all criteria support the four leading variants in varying degrees. The nearest alternatives to compromise the solution are Allegro, Ebay.pl, Trademe and Ebay.com. This sequence is in accordance with the ranking obtained using EQUAL method. The biggest impact on the final ranking have the criteria, where vectors in the plane GAIA are the longest, i.e., C2, C3, C19, C1, C13 and C17. In addition, a small conflict is observed between two sets of criteria, the first set is in the beginning of the first and the second in the second quarters of the system of coordinates. The first group include the criteria belonging to the group Empathy (C19–C21) and partially group Trust (C18, C22). The second set of criteria are assigned to the group Usability (C1–C4) and partly to the criteria Site Design (C5, C7). This observation was confirmed by the GAIA analysis carried out for groups of criteria, shown in Fig. 3, which is represented by a conflict between groups Usability and Empathy. This means that high ratings of an auction platform in terms of the criteria belonging to the group Usability are associated with low assessment of the same platform in terms of criteria from Empathy group. Since the criteria of the groups Usability and Site design conflict with the criteria of belonging to groups Empathy and Trust, a compromise is supported by the criteria belonging to the group Information quality. In addition, Fig. 3 expresses the similarity of evaluations about the Trust and Empathy criteria groups, because their vectors are located quite close to each other. In other words, if the auction website receives high ratings in terms of the criteria of Empathy group, it is usually also highly rated in terms of the criteria of a group Trust.

Fig. 3.
figure 3

GAIA analysis for the group of criteria

Full size image

4.4 Sensitivity Analysis of Solution

Apart from GAIA analysis, a sensitivity analysis was also conducted. The analysis results indicate that the obtained rankings are very stable, because only changes for weights criterion C21 can cause shift in the first position in the ranking. However, the change to the second position can occur only in the case of changing the weights of criteria: C8, C18–C20. The ranges of stability for the weights of individual criteria are included in Table 7.

Table 7. Sensitivity analysis - the ranges of stability for the weights of criteria
Full size table

4.5 Reliability Analysis of Solution

To verify the assurance of the resulting solution, the new ranking was determined, New model was based on true criterion preference model (without thresholds). As a result, it was observed that even a small advantage of one variant over another in terms of a specific criterion C1–C22 causes that variant is considered as globally preferred. The obtained ranking using the true criterion is presented in Table 8. This ranking coincide with rankings presented in Tables 4 and 6 for the first two positions. There was a shift of variants in positions 3–4 and 5–6. The result from Table 8 is not also the same as ranking obtained by eQual method. It should be assumed that the application of the true criterion causes that the resulting ranking, GAIA analysis and sensitivity analysis for such rankings do not reflect exactly the preferences of the users. This is important, because many MCDA methods use only true criterion function, e.g., Melchior, Regime, Qualiflex Electre I, Electre II. It’s worth to notice that the application of these methods should be done very carefully with focus on proper preference modelling and avoiding its oversimplifying.

Table 8. The final ranking and performance of decision variants (true criterion function)
Full size table

4.6 Comparison of Averaged Rankings with Group Ranking

In the last step of research the group ranking using PROMETHEE GDSS method was determined. Based on individual rankings for each user and with the use linear preference function and thresholds q = 1 and p = 5, the individual rankings were aggregated in the group ranking. The result of using the PROMETHEE GDSS method is shown in Table 9.

Table 9. PROMETHEE GDSS based ranking
Full size table

Analysis of the final ranking from PROMETHEE GDSS indicates that results are similar to the rankings contained in Table 6 and ranking of eQual method (Table 4). Positions of Ebay.com and Trademe are reversed, and it is associated with a change in the value ϕnet Trademe site. Values of ϕnet other sites are close to the values shown in Table 6. However, the difference in the rankings indicates that the ranking GDSS does not reflect the exact preferences of users. Therefore, on the basis of analysis may lead to erroneous conclusions regarding the ranking embodiment, however, the results PROMETHEE GDSS are sufficient for the analysis of conflict between users in carrying out embodiments using Gaia plane. This plane is shown in Fig. 4. It indicates the existence of small conflicts of opinion, i.e., decision makers DM1, DM3, DM5, DM7, DM9, and DM10, whose vectors have been placed in the fourth quarter of the system of coordinates, with the opinion of DM4, whose vector is located in quarter II. In addition, it must be noted that the most consistent with the final ranking of GDSS is assessment of user DM6.

Fig. 4.
figure 4

GAIA analysis for the group ranking

Full size image

5 Conclusions

Online auctions are one of the most explored e-commerce sector. Growing competition requires improvements in terms of pricing models, purchasing processes and overall website quality and usability. Dedicated methods are required to include various preferences and often conflicting interests. While several methods were used earlier they lack of multi-criteria approach and preferences aggregation.

In the proposed approach the multi-stage development of the model was acknowledged as to the criteria taken from the eQual technique with the utilization of the Promethee method (PEQUAL). It broadens prior methodologies by presenting MCDA based multi organize assessment and investigations. In the article, 6 well known auction sites were assessed. On the basis of the presented research, one can state that e-commerce websites most highly valued by users are: Allegro, Ebay and Trademe The conclusions were affirmed by stability study of obtained ranking, especially in the terms of its sensitivity and robustness analysis.

Besides, the utilization of the Promethee GDSS technique and the GAIA investigation, which is an essential part of the Promethee strategy, made it conceivable to show clients’ individual inclinations. Likewise, the GAIA examination permitted inspecting shared conditions between individual gatherings and bunches of criteria on the premise of realistic information. The understanding of the GAIA plane is less tedious and simpler than the investigation of number estimations of assessments, and the conclusions drawn on its premise are similarly basic.

The examination structure of websites exhibited in the article can be the reason for their assessment alongside the rightness check of acquired assessments and inclinations of the respondents. As it has been exhibited in the introduced inquire about, this arrangement is practically wealthier than established MCDA-based techniques for site assessment strategies which have been utilized as a part of the writing to date.