Keywords

1 Introduction

Usability is one of most important software quality attribute, as highlighted in various standards and usability models, e.g. ISO 9241-11 [1], 9126 [2], McCall [3] and Boehm [4] etc. It is associated with the “ease of use” of a given software system [S5]. ISO 9241-11 defines usability as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, satisfaction and efficiency in a specified context of use” [1]. “Context of use” means the description of actual conditions under which an interactive system is being evaluated. Context analysis [6, 7] is required for conducting research on software usability.

Usability evaluation is generally done through qualitative and quantitative assessments. Qualitative assessments are performed by evaluation team during evaluation stage. This stage is very crucial because it empowers developers to encompass expert feedback till admissible level of usability is achieved. Generally, research practitioners focus on qualitative assessments of usability. Quantitative assessments summarize the results in single metrics after evaluating various dimensions of software usability [8, 9]. Also, Quantitative assessments are not strong enough to compute overall usability of a software system [8, 10]. This motivates us to focus on qualitative assessments of usability. So, an attempt has been made to achieve qualitative results through Qualitative usability inspection (QUI).

Some models are defined for usability evaluation of interactive systems [11,12,13]. Usability evaluation has also been done on early prototypes with data-mining techniques [14, 15]. The context of use as a whole has been addressed but only homepage of academic websites are considered for usability evaluation. Further, there are several existing data-mining approaches for assessing and evaluating the usability [17,18,19,20]. Furthermore, usability evaluation has been done in different domains e.g. mobile [21,22,23,24] and website [25,26,27,28,29]. Usability evaluation has been significantly done for web-domain [26]. Hence, it also motivates us for usability evaluation in web-domain, mainly for academic websites.

Nowadays, one of the major challenges for usability evaluation is identifying the common usability related problems for a context of use as a whole. Recognising such problems can assist novice research practitioners in evaluating an advanced interface belonging to same context and to restrain errors when a new interactive system is being produced. To overcome with this challenge, QUI is employed that emphasizes on the “what” over the “how many” questions belonging to identification of usability related problems for a context of use as a whole.

QUI provides information about usability problems that occurs due to violations of heuristics belonging to a system. Identified usability problems belonging to a system, are prioritized. And, then a usability document is generated, containing a prioritized list of identified usability problems. QUI incorporates various divergent methods that emphasize on evaluating the interfaces of the systems [5, 8, 10]. These methods turn out to be complicated when substantial number of systems related to same context of use, are considered jointly to impart an extensive diagnosis. The principal cause for this is due to substantial quantity of information that should be conceptualized and handled simultaneously.

To handle substantial quantity of information, this paper proposes a novel approach called QUIc Footnote 1. This qualitative usability inspection is based upon data-mining techniques. Association rules and decision trees are used to find the common usability related problems for a context of use as a whole [30]. Each identified usability problem pattern belongs to a relevant usability characteristic of academic websites. Such usability problem patterns are then analysed through data-mining knowledge discovery process. Then, the usability document comprises the final output that is a list of prioritized usability problems.

To validate this proposed approach, an experiment is conducted in which academic websitesFootnote 2 are evaluated and common usability problems patterns related to these websites are identified. In other words, a general diagnosis for the context of use of academic websites is analysed by employing QUIc approach. The experiment involves processing of 2475 records that store qualitative information. The evaluation team uses heuristic evaluation and defines the 55-heuristic questions related to academic websites. Each heuristic question represents usability features of academic websites. Each feature is related to usability attribute of academic websites (shown in Table 1)Footnote 3. For usability evaluation of websites, data is collected manually that is time consuming and cumbersome task [15]. So, evaluation team decides to use automated tool that provides answers to maximum defined heuristic related questions within minimum resources of time and money. In this study, these answers are collected with help of automated tool named woorank [31].

Table 1. Correspondence between heuristic-related questions belonging to Mobile (8 questions), Social Media (6 questions) and Security (3 questions) Category with attributes name
Full size table

The remaining part of this paper is organised as follows: Sect. 2 presents related work. Section 3 gives an overview of major concepts of data-mining knowledge discovery from data repository, association rules and decision trees. Section 4 provides details of proposed approach that augment the conventional QUI process with woorank tool and data-mining knowledge discovery process. Such novel approach identifies common usability problems patterns for context of use as a whole, but mainly from a qualitative viewpoint. Section 5 reports experimental results that validate the proposed approach. Section 6 summarizes conclusion and future work. This paper is motivated by some preliminary research work done on usability evaluation of early prototypes through association rules [14], and usability testing through datamining techniques [15, 16].

2 Related Work

Various studies have discussed the usability inspection methods [42, 43]. Some articles have explained the state of art of methodologies and the qualitative usability evaluation [8, 44,45,46,47]. For usability evaluation of interactive systems, some models are defined [11,12,13]. Usability evaluation of early prototypes with data-mining techniques e.g. association rule, is explained in [14]. Few studies have integrated usability evaluation with KDD process (association rules and decision trees) for finding significant patterns belonging to academic websites [15, 16]. The context of use as a whole has been addressed but only homepage of websites are considered. [17] uses association rule mining for assessing the usability of system. [18] provides recommendation models that are explained as a set of association rules to enhance the usability of the system. Even, the decision trees are applied for improving the usability evaluation [19, 20]. But these techniques are not used to analyse usability of a context of use as a whole. Software usability evaluation has also been done using fuzzy approaches [48, 49] and model driven development approach [28]. Furthermore, usability evaluation has been done in different domains e.g. mobile [21,22,23,24] and website [25,26,27,28,29].

3 An Overview of Data-Mining Knowledge Discovery from Data Repository

Data-Mining Knowledge Discovery from data repository is an approach that extracts previously unknown and potentially constructive information from the available data. It is a process that normalizes the data from heterogeneous sources and stores it in data repository or database. From data repository, attributes are generally identified. This extraction process in data-mining knowledge discovery [32] is executed as follows:

1. Data pre-processing: It consists of a number of steps that are prerequisite to produce data for mining process, e.g. (a) Data cleaning: The aim of this step is to remove irrelevant data and fill in missing values etc. (b) Data integration: Data from heterogeneous sources may be combined into single repository or database. (c) Data Selection: Relevant data required for evaluation, are retrieved from the database (d) Data transformation: Data are transformed into appropriate forms for mining process.

2. Data-mining: After pre-processing the data, various data-mining techniques can be executed on data, to extract significant patterns from it. There are several data-mining techniques available in literature e.g. Association rules, Classification, Clustering, Prediction, Sequential Patterns, Decision trees, trend analysis, Bayesian data analysis, Regression, neural network models etc. [33]

3. Pattern Identification: After applying datamining algorithms, patterns are identified that represents undiscovered knowledge. These patterns are then evaluated by using different measures.

4. Graphical representation: Different visualization and presentation techniques are employed to represent mined knowledge or results to users.

A software platform for Data-Mining Knowledge Discovery process generally involves 3-components (See Fig. 1.): (1) Data Repository to store the data (2) a Data-Mining Knowledge Discovery engine for conducting and executing datamining operations. This engine offers various datamining algorithms and modules for pattern evaluation and visualization (3) a Query Interface that assists user to interact with engine by posing queries in datamining query language. It is important to note that datamining is most vital step in Data-Mining Knowledge Discovery process, since it extracts previously unknown information and identifies hidden patterns of information for evaluation of patterns.

Fig. 1.
figure 1

Major components of data-mining knowledge discovery process platform

Full size image

Data mining tasks demands the use of robust software platforms. As the number of existing platforms are persistently growing and therefore, choosing the suitable platform is becoming complicated task. Further, Different free-open source platforms have also been used for conducting data-mining knowledge discovery process [34]. e.g. RapidMiner [35], R [36], Orange [37], and Weka [38, 39]. Each platform has their own particular DMQL language that varies from basic command-line interpreters to sophisticated tools.

This paper is based upon some preliminary research work done by Gonzalez et al. [14], on usability evaluation of early prototypes through association rules and usability testing through data-mining techniques [15, 16]. For usability evaluation, Weka is generally used for applying data-mining techniques. Hence, the tool used in this study is Weka [39].

Furthermore, in this paper, we present an extended approach to the conventional qualitative usability inspection process by integrating 2-datamining techniques, e.g. association rules and decision trees. In next sub-section, we summarize main characteristics of these 2-techniques.

3.1 Association Rules

Association rules help in identifying significant relationships among attributes in a given data set [30]. The identification of these relationships is generally used in decision making processes for real world problems, e.g. market-basket analysis etc. In other words, association rules guide us to identify various patterns in datasets. For creating association rules, various algorithms have been used for mining knowledge from huge databases, e.g. Apriori, Predictive Apriori, Filtered Associator and FPGrowth [30]. The thresholds values are generally specified for confidence and support values so that computational complexities can be reduced.

3.2 Decision Trees

A decision tree is a flow-chart-like tree structure, where each node denotes a test on an attribute value, each branch represents an outcome of the test, and tree leaves represent classes or class distributions. In order to classify an unknown sample, the attribute values of the sample are tested against the decision tree [33, 38]. Therefore, the decision trees can be used in predicting the behavior of context of use under evaluation. Furthermore, decision trees can be generated from relatively small transactional databases to identify ‘‘target attribute’’. Various algorithms have been used for creating decision tree e.g. ID3, J48, FT and C4.5 [33]. During construction of decision trees, 66.6% data is used for training whereas remaining 33.3% is used for testing.

Fig. 2.
figure 2

Proposed approach for Qualitative Usability Inspection process (QUIc) for interactive systems.

Full size image

4 A Proposed Approach for Usability Evaluation

In this section, a novel approach of QUIc process is proposed that is based upon the integration of a conventional usability inspection [40, 41], an automated woorank tool [31] and Data-Mining Knowledge Discovery Process [32]. Figure 2. represents architecture of the proposed approach. The proposed approach of QUIc process is composed of the following steps:

I. Formulating QUI c process:

The QUIc process starts with the selection of team-members for evaluation team. The evaluation team chooses a sample of interactive systems, Wc = {W1…Wn} that is taken to present entire context C. Further, the team selects appropriate method for usability evaluation of system. The automated tools required for usability evaluation are also being explored. Furthermore, this step considers the selection of particular software that is required for implementation of Data-Mining Knowledge Discovery Process.

II. Defining the Heuristic related questions and gathering QUI c data:

Heuristic evaluation is a frequently used method for usability evaluation [42, 26]. The evaluation team defines heuristics-related questions. Each question belongs to various usability features or attributes of interactive systems that are to be evaluated. The correspondence between each heuristic question with attributes is shown in Table 1. The QUIc process is based upon qualitative data that is collected using automated tool. The collected data is stored in data repository e.g. database and spreadsheet etc.

III. Implementing Data-Mining Knowledge Discovery Process within QUI c process:

Once the qualitative data is stored, the next task is to implement Data-Mining Knowledge Discovery Process. The purpose of this process is to discover usability problem patterns among data-items by using data-mining techniques. Various data-mining techniques are implemented using Weka platform. Different queries can be posed to mine the data, to obtain association rules and decision trees. Association rules represent unknown and hidden relationships among data items. Decision tree presents usability problem patterns belonging to usability features of interactive system. As an output, a ranked list of association rules is generated that should be analysed by the evaluation team. The decision tree illustrates usability problem patterns each belonging to usability features. These patterns can also assist in evaluating interfaces of new system belonging to same context of use.

IV. Graphical representation and analysis of usability problem patterns:

This is one of the most important steps of proposed approach. As, it deals with graphical representation and analysis of usability problem patterns generated in above step. The evaluation team uses different visualization tools provided by Weka. These tools help the team in visualising different graphics belonging to generated usability problem patterns. For components in ranked lists of rules, results are shown as a sequence of association rules (as shown in Table 3.). The decision trees are very descriptive by themselves in their tree like representation (as shown in Fig. 3). Further, evaluation team analyses each obtained usability problem patterns to detect usability problems belonging to interactive system of same context of use. Every detected usability pattern is discussed on the basis of its significance (e.g. support and confidence). The relative rank of each association rule with respect to entire ranked list (as shown in Table 3) is related to prioritization of the usability problem patterns. For admissible decision trees, the evaluation team decides to consider the threshold values of 70% for classified instances. These admissible decision trees can be added in final usability document.

V. Paraphrasing and reporting QUI C conclusions in usability document:

The QUIc process generates the output as a set of general usability problems patterns. Each detected usability problem pattern should be paraphrased as a usability problem in standard format [5] as shown in Table 3. If needed, then different visualizations of the usability problem patterns should be attached to final output using this format.

5 Experimentation and Results

In this section, obtained results are provided after applying proposed approach on 50-academic websites.

Step 1: First step is to select the evaluation team-members. The evaluation team is formed of 2 members: 1-usability experts, 1-Associate professor in university, 1-PHD student. All members of team have knowledge about usability and are frequent users of academic websites. The Google chrome browser is used to visualize the academic websites. Ideally, QUIc process must consider comprehensive usability evaluation of all academic websites in context of use. But practically, it is not a feasible task because of huge associated resources in terms of time and cost. Therefore, a sample of academic websites Wc = {W1…Wn} is taken to present entire context C. The guidelines have been followed to impart a representative sample for academic websites [22]. As a sample, websites of top 50-academic universities of India listed in National Institutional Ranking FrameworkFootnote 4 are considered. Furthermore, the evaluation team searches for automated tool for usability evaluation of academic websites. And then, decides to use automated tool, named woorank [31]. This tool provides answers to maximum defined heuristic related questions within minimum time. The output spreadsheet contains answers of heuristic related questions of the website evaluation. Moreover, the planning step considers the selection of particular software that is required for implementation of Data-Mining Knowledge Discovery Process. So, Weka is used for applying data-mining techniques e.g. association rules and decision trees.

Step 2: The evaluation team has chosen heuristic evaluation as it is frequently used methods for usability evaluation [10, 42, 26]. The team defines 55-general heuristic-related questions. Each question belongs to various characteristic of academic websites that are to be evaluated. Range values for each possible answer are also defined. Each heuristic-related question is associated with an attribute name of website usability that woorank tool evaluatesFootnote 5. Table 1. represents correspondence between heuristic-related questions belonging to Mobile, Social Media and Security category with associated attribute name and possible answers. Due to the space constraint, only three categories that represent latest technological issues are defined and presented in Table 1.

Woorank tool is employed to evaluate an academic website as a wholeFootnote 6. During evaluation of each website, tool generates pdf file. This file contains answer for each heuristic question in terms of green, red and orange color. Green color represents that particular attribute or usability feature is available (or gets pass) on that website and we assign it as “yes” in output spreadsheet. Similarly, red color represents that particular attribute or usability feature is not available (or contains error) on that website and we assign it as “no” in output spreadsheet. Orange color represents that particular attribute or usability feature is available on that website but needs improvement and we assign it as “ni” in output spreadsheet. In this way, answers of maximum defined heuristic questions are evaluated and collected. These answers are stored in output spreadsheet that represents usability data. Now, each heuristic question is linked with an attribute in output spreadsheet. The attribute’s names appear as a column heading in output spreadsheet. Each row of output spreadsheet corresponds to complete usability evaluation of a specific academic website. Hence, 45 websites are evaluated and 55 heuristic questions are defined and their answers are stored into output spreadsheet, resulting into 2475 records.

Step 3: After collecting and storing qualitative data, the next task in QUIc process is to implement Data-Mining Knowledge Discovery Process. The aim of this step is to discover usability problem pattern among attributes by using data-mining techniques. Evaluation team uses weka software platform and query interface to pose queries for data-mining techniques. Association rules and decision tree are implemented on qualitative data. The association rules express unknown and hidden relationships among attribute representing usability data. For implementing association rules, apriori algorithm is used. As an output, ranked list of rules is obtained and is presented in Table 2. The evaluation team uses this list to identify the hidden relationship among attributes. The decision tree is also implemented that presents usability problem pattern belonging to usability features of academic websites. Further, these patterns can assist in evaluating new websites belonging to same context of use. For implementing decision tree, ID3 algorithm is used. As an output, decision tree is generated and presented in Fig. 3. The evaluation team considers decision tree representation as new knowledge that helps in identifying usability problems pattern concerning the context of use of academic websites as a whole.

Step 4: WEKA supports apriori algorithm that generates a ranked list of best 11-association rules (with support 70% and confidence 90%). Table 2 represents obtained ranked list L1 of association rules belonging to heuristic related questions of Mobile, Security and Social Category.

Table 2. Ranked list L1 of association rules belonging to heuristic-related questions (generated by WEKA)
Full size table

It is important to note that association rules from L1 list impart initial guidelines to identify problematic attributes in discovered patterns. These problematic attributes generally represent usability problems. Therefore, association rules with problematic attributes are only considered. For example, association rules #1, #2, #3, #4, and #5 in L1 show that there exist usability problem patterns related with absence of Mobile_Frameworks in academic websites. Another example, association rules #10 and #11 in L1 show that there exist usability problem patterns related with absence of Google+_Publisher in academic websites. The usability problem patterns also help in detecting significant relationships among attributes. By just looking at output spreadsheet, the evaluation team cannot easily detect these significant relationships that are found in association rules #1, #2, #3, #4, and #5 in L1. The discovered patterns can impart constructive information for evaluating the usability of academic websites that is not considered in National Institutional Ranking Framework but relates to same context of use. Further, these usability problem patterns emphasize on significant heuristic-related questions that should be carefully considered during usability evaluation. The usability problem patterns can be considered as guidelines for development of novel academic webpages.

Fig. 3.
figure 3

Decision tree by using ID3 algorithm (target attribute Mobile_friendliness with possible values {Yes, No, NI, NF}) (visualization provided by WEKA platform)

Full size image

The Evaluation team can pose some more queries to further mine the information form output spreadsheet. But, queries must have different support and confidence values. A query can be used to determine some other problematic attributes in L1 (mention in following example).

GetRules(Output_Spreadsheet) where

[Antecedent has {(Mobile_Frameworks = sometimes)}and support >0.8 and confidence >0.9]}

The evaluation team poses more queries to get decision trees. The decision trees generally help in predicting the behavior of context of use under evaluation. WEKA selects 66.66% of available data from the output spreadsheet as training set and remaining data as test set. For admissible decision trees, the threshold values are set to 70% of classified instances. Figure 3. provides textual representation of resulting decision tree for any target attribute e.g. Mobile_friendliness.

Every branch of decision tree can be read as “if-then” rule. Figure 3 represents rule as: if (Mobile_Viewport = No) and (Touchscreen_Readiness = NI) and (Mobile_Speed = No) and (Mobile_Compatibility = NI) then (Mobile_friendliness = No). It is important to notice that values stored in the attributes Mobile_Viewport, Touchscreen_Readiness and Mobile_Speed are significant to determine the value of attribute Mobile_friendliness (as they are adjacent to root of the decision tree). The value stored in the Mobile_Compatibility is not extremely relevant for predicting the value for Mobile_friendliness. This stored information in decision tree is previously unknown and hidden for evaluation team. Hence, evaluation team considers decision tree representation as new knowledge that helps in identifying usability problems concerning the context of use.

Step 5: The QUIc process generates the output as a set of usability problem patterns. It is recommended to paraphrase each pattern as a usability problem. Each usability problem should be reported and documented in a standard format [5] given in Table 3.

Table 3. Usability document reporting usability problem
Full size table

6 Conclusions and Future Work

QUI methods emphasize on evaluating the interfaces of the systems. These methods turn out to be complicated when substantial number of systems related to same context of use, are considered jointly. To overcome with this problem, a novel approach of QUIc process is proposed that is based upon the integration of a conventional usability inspection (HE), an automated woorank tool and Data-Mining Knowledge Discovery Process. The study provides usability problem patterns highlighting the latest technological issues in academic websites. Identifying such problem patterns and proposing solution to these problems can help the developers and usability professional in various aspects. The insight about these patterns can assist developers to restrain from these problems, when new academic website is being developed. Further, detecting such problems at early stages can reduce development efforts in time and costs. Various instincts that were informally expressed by evaluation team during QUIc process can now be significantly examined through data-mining knowledge discovery process. Therefore, for real world problems (i.e. usability problem patterns in Indian academic websites), the study can conclude that proposed integrated approach, can be executed successfully. The primary advantage of proposed approach is detection of unknown and hidden relationships among attributes belonging to academic websites. Another advantage is detection of usability problems patterns in academic websites. In comparison to previous related works, this study addresses certain differences as follows:

  1. 1.

    To the best of author’s knowledge, there is no alternative approach that extends traditional QUI process with automated woorank tool and data-mining knowledge discovery process, as presented in this paper.

  2. 2.

    Qualitative usability inspection like HE method performs usability evaluation that is closer to real user insights. And thereafter, generates more relevant results.

  3. 3.

    In proposed approach, an attempt has been made to consider latest technological issues related to academic websites (e.g. mobile, social media and security etc.). 55 Heuristics related questions are defined for evaluating the usability of academic websites. The study presents only 17 heuristic questions due to space constraint.

  4. 4.

    In this study, 7 categories are considered for defining heuristic questions related to academic websites. These categories are defined to identify the usability problem patterns belonging to latest technological issues in mobile domain etc. Whereas other study has defined only 4 categories [15] without considering these issues.

  5. 5.

    Other study [15] has evaluated only homepages of academic websites. In this study, the automated woorank tool is used to provide extensive evaluation of entire academic websites. Using this tool, the data is collected within limited time period as well.

  6. 6.

    The proposed approach highlights the fact that academic websites, generally, are not mobile optimised. To best of author’s knowledge, this issue is not addressed in any study.

  7. 7.

    Another issue is related with server signature feature that found to be enabled on some websites. These full server signatures can be exploited and attacked by hackers. Hence, academic websites are not fully secured.

  8. 8.

    Although, Google+ has reached 100 million users faster than Facebook and Twitter. But academic websites do not have official Google+ Page to socialize them on the social network. The proposed approach highlights another fact that academic websites do not have Google+ _Publisher for linking to Google+ _Page.

The future work can be done in employing different algorithms for association rules and decision trees. Such advanced algorithms can provide more extensive results. One can also emphasize on testing different ranking functions for association rules. Further, the generated data is compiled into databases that can be used for statistical analysis. Such analysis can help in quantitative assessment of usability for academic websites. In near future, research can be pursued in these directions.