Decision making for cloud service selection: a novel and hybrid MCDM approach

Abstract

Cloud computing has emerged as a promising Internet technology enabling cloud users to access computing resources on-demand via the Internet in a “pay-as-you-use” fashion. Many cloud service providers (CSPs) have arisen over the last few years with similar features at varying prices and performance levels. With the rising number of CSPs, cloud customers face the challenge of choosing the right CSP that satisfies their Quality of Service requirements. However, it poses a major challenge: “How to evaluate a suitable CSP with high accuracy and consistency.” To address this challenge, this paper proposes a hybrid Multi-Criteria Decision Making methodology to aid the decision-maker to evaluate different cloud services. Specifically, a novel approach is introduced based on a comprehensive assessment that combines subjective and objective aspects. The comprehensive assessment results are utilized to rank the eligible CSPs based on their prioritized list. The simulation results are validated through a real-life case study which further justifies that the proposed approach provides better satisfaction degree from the user’s perspective and is efficient in terms of accuracy and reliability. Finally, we perform a sensitivity analysis to show the robustness and stability of our approach.

1 Introduction

Cloud computing refers to an emerging paradigm where consumers can receive on-demand computing services (both hardware and software) over the Internet in self-service mode regardless of the system or the location [1]. The idea of cloud computing comes from many cutting-edge innovations and has some common aspects with other computing models. Cloud computing offers strong benefits, compared with conventional models in terms of price and performance [2]. Based on the user’s requirements, the conventional cloud computing model has the following three service levels. Software as a service (SaaS) is a way to offer software applications on-demand over the Internet. Platform as a Service (PaaS) provides a computing platform and environment for its user to build applications and services over the Internet. Infrastructure as a service (IaaS) delivers computing resources (servers, storage, and virtualization) over the Internet. Cloud computing has revolutionized small, medium, and large-scale enterprises’ business with reduced cost, on-demand service, scalability, and service elasticity. Realizing these advantages, several business companies outsource their business to cloud-based computing. As a result, cloud computing usage and development have grown exponentially [3].

As the demand for cloud computing grows, more cloud service providers (CSPs) such as Google, IBM, Microsoft, Amazon GoGrid, etc., have joined the cloud service business market which offers different cloud services with various price and performance choices. This makes it highly difficult for cloud customers to choose the right service that can satisfy their functional and non-functional requirements. Numerous factors impact the complexity of service selection. For an appropriate cloud service selection, a cloud consumer has to identify the service requirements in QoS (Quality of Service) [4]. For example, on one hand, criteria such as performance or reliability are essential to specify service characteristics. On the other hand, privacy and usability are of utmost importance for cloud customers as well. In such situation, a wide variety of divergent evaluation criteria that characterize several cloud services offered by many CSPs must be considered. Therefore, the primary concern is how to find an effective cloud service based on customers’ requirements with a high degree of reliability and precision which makes cloud service selection (CSS) a significant and interesting research problem. Due to the intrinsic relationship among the multiple QoS attributes, alternatives, and decision makers’ opinions, the CSS problem can therefore be formulated as a multi-criteria decision making (MCDM) problem [5] as shown in the Fig. 1.

Recently, MCDM has come up as one of the most efficient decision-making tools, showing its potential to solve real-world problems [5, 6]. There have also been numerous works using MCDM methods on CSS problems. However, most of the research works [7,8,9,10,11] are designed to discover the most appropriate cloud services concerning either subjective (defined in qualitative statements, e.g. strategy, management, etc.) or objective (defined in numerical terms, e.g. cost, time, speed, etc.) assessment but not both. Moreover, most of these methods have treated both objective and subjective evaluations equally which produces much noise in the selection approach. Both kinds of evaluations must be treated differently. This is because objective evaluation cannot be simple for ordinary cloud users who can be overwhelmed with multiple quantitative data. On the other hand, subjective evaluation is easy to comprehend, but it can contain favoritism and malicious evaluation, therefore, not wholly trusted. Nonetheless, the methods considering both objective and subjective assessment leave scope for further improvement in consistency. Thus, a suitable and efficient approach for decision-making is therefore highly desirable to resolve the following issues:

• The customer’s preferences must be taken into account and estimated accurately. Since a simple quality evaluation without customer’s expectations is insufficient.

• The selection method should be able to handle the false or artificial QoS criteria information which is usually published by the service provider.

• The selection method should be flexible enough to accommodate any number of criteria and cloud service alternatives.

In the view of above issues, we have presented a novel methodology for selecting suitable cloud services by aggregating both objective assessments using quantitative analysis and subjective assessments based on cloud customer’s opinions. Here, we took two ways to confirm the objectivity of the QoS data source and avoid the impact of artificial QoS data values: (1) First, instead of considering pure feedback rating, we derived QoS criteria information from a trustworthy third-party monitoring tool. (2) Second, we have used an entropy weighting method [12] to find out the objective weight for different QoS criteria rather than the traditional average weighting method for increasing the objectivity of evaluation results. As only objective assessments are not enough to scrutinize the cloud service performance, we have performed a subjective evaluation also to achieve high satisfaction for cloud clients. To assign the subjective weights of QoS criteria as per customer’s preferences, we have employed a systematic method called the Analytic Hierarchy Process (AHP) [13, 14]. To the best knowledge so far, we are the first to explore the benefits of combining AHP with entropy for the proposed weighting strategy, i.e., integrating the objective and subjective aspects, which is further utilized with Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) method to achieve the final rank.

In summary, the main contributions are as follows:

• 1. A novel service selection methodology based on the QoS parameters has been proposed.

• 2. We have resolved complicated decision-making issues involving a variety of objective and subjective constraints.

• 3. We have presented an organic combination of the subjective analysis and the objective analysis to facilitate a higher degree of satisfaction to the cloud customers.

• 4. We use quantitative QoS metrics which leads to more accurate results compared to qualitative QoS metrics that are more likely to be biased.

• 5. We check the consistency of proposed methodology with sensitivity analysis.

The rest of the paper is structured as follows: Section 2 briefs the related works. Section 3 discusses the preliminaries. Section 4 proposes our hybrid decision-making methodology. Section 5 describes the case study of proposed approach. A comprehensive analysis along with results validation is shown in Section 6. The concluding remarks are delineated in Section 7.

Fig. 1

An instance of Cloud Service Selection problem

2 Research gap

As the number of cloud service providers has grown over the past years, several research efforts have attempted to resolve cloud service selection problems based on QoS [8,9,10, 15,16,17,18,19]. In [20], authors have introduced the SMICLOUD framework to compare and assess three IaaS cloud services. The authors used the AHP method to measure the weight of the criteria and evaluates the three IaaS cloud services. This method was mainly based on three basic phases- decomposition of problem, priority evaluation and ranking the IaaS service providers. In the first phase, a hierarchical structure is constructed which links the selection goal, the QoS attributes and the service providers. A pairwise comparison matrix was utilized in the second phase to find the criteria weights. In the last phase, the rank of IaaS cloud service providers is calculated using the criteria weight. In this research, CSMIC presents several quantitative key performance indicators (KPIs) for QoS attributes, and different cloud providers use these KPIs to compare it. Also, the application of AHP enables the measurement of criterion weights based on user preference and the estimation of interdependencies between metrics. Based on cloud customer’s requirements and preferences, [21] used the AHP methodology to identify the right cloud database provider. In this paper, a hierarchy is used that includes three critical criteria and seven sub criteria. The AHP approach was used with the genetic algorithm in the [22] for an IaaS cloud assessment. This paper suggested a CloudGenius framework to assess and assess the best IaaS service provider with 15 QoS parameters.

Paper [15] proposes a mutual evaluation-based cloud service mapping (MECSM) framework which addresses the bidirectional evaluation of both the service providers and consumers. In this paper, a unidirectional approach is used where the mapping of service is based on the service provider’s evaluation in the context of QoS requirements of a consumer. Paper [23] proposes a novel framework called Optimal Service Selection and Ranking of Cloud Computing Services (CCS-OSSR), which allows cloud customers to compare available service choices based on QoS (Quality of Criteria) criteria. The CCS-OSSR utilizes a hybrid multi-criteria decision making approach. Best worst method is used to rank and prioritize the QoS criteria and Technique for Order Preference by Similarity to Ideal Solution approach is employed to obtain the final rank of cloud services. In [24], MAUT was utilized to develop a cloud service evaluation technique that evaluates the utility value of each cloud service alternatives based on cost and quality parameter. Cloud service architecture and a key cloud service selection algorithm was discussed in [25]. The cloud service selection procedure depends primarily on the agreement between maximized profit and minimized cost and it is determined using multi-attribute utility function. In [26], SAW method was discussed for measuring reusability by allocating various weight values to the QoS criteria. The reusability of cloud services was measured in this contribution by two different points of view: service provider’s point of view and customer’s point of view. A fuzzy MCDM cloud service assessment model was used in [27] with the use of Fuzzy SAW.

In [28], an improved DEA and SDEA approach has been implemented to select appropriate cloud services between different cloud services, based on customer requirements. In article [7], to evaluate and rank different cloud services using the Grey TOPSIS and AHP method, the author implemented a SELCLOUD model. Here, author employ AHP method to evaluate the QoS attribute weight and measure the rank of cloud alternatives using the Grey TOPSIS method. Actually, most of the work assume that QoS attributes are independent. Rahman et al. proposed a framework in [29, 30] to support the selection of IaaS cloud services. In this article, a status checker on a virtual machine monitors the performance of the IaaS cloud. Here, a centralized QoS repository is used to collect and manage cloud performance reports for service assessment and selection. Authors have suggested a time aware selection methodology based on the various MCDM methods AHP, ANP and TOPSIS. By scrutinizing the aforementioned studies, it is noticed that sufficient attention has not been paid to the classification of cloud evaluations prior to service selection. So, a method for quantifying both objective and subjective assessment is required to make the decision making results more reliable. The aforementioned issues not only complicate the decision-making process but also raise questions regarding the authenticity and reliability of the results. In summary, they are as follows. (1) An increase in the number of decision criteria intensifies implementation and computational complexity of these methods, which in turn reduces performance, especially in real world problems. (2) It is quite challenging to achieve highly consistent comparisons. (3) It is even more challenging to revisit comparisons in case of inconsistencies. (4) Unsuitable for decision making when both objective and subjective assessments are desired.

In these views, we address these issues and develop a novel cloud selection approach. This study provides an organic combination of the subjective analysis the objective analysis to achieve a better comprehensive assessment of the results.

3 Preliminaries

We first explain an example for motivation of our work and then discuss about evaluation criteria.

3.1 Motivation example

An example to inspire our work in practice is given in this subsection. Suppose ABC is a major company that provides its clients medical services.The business plans to draw more customers through improved service quality. For the following reasons, ABC move its business from in-house to the public cloud to deliver different services at high performance, high security and low maintenance costs:

• Better cost control ABC can reduce its staff cost, maintenance cost and technology costs considerably using cloud services.

• Save time & effort With cloud solutions, software upgrades, maintenance and data protection do not require technical expertise. Due to this reason, all employees can shift their attention freely from system to customer.

• Appealing to modern consumers Clients want to access various services with their mobile phones and tablets. Customers and staff are able to communicate with a cloud-based service at any time and anywhere.

• Keep Up With Industry Trends The cloud service allows ABC to keep up with real-time delivery in the on-line market industry. By introducing new features and services, cloud computing provides real opportunities for business growth.

Furthermore, ABC is a service company that provides its users with an extremely desirable, high-quality, and extensive sales service. ABC searches for a cloud service provider for the deployment of its services. A range of cloud service providers (CSPs) provides several services that vary based on QoS requirements in the current market. ABC compares every QoS criteria and analyses each cloud service’s efficiency to pick an adequate cloud service from the existing services. ABC will prefer a cloud service provider with more memory capacity, fast processing speed and a lower maintenance cost (e.g., availability and response time). The selection process will rely on an organization’s particular requirements, which may depend on different objectives.The company needs to clarify the needs and significance of each QoS criterion in terms of its goals. The highest performing cloud service provider is selected as the best service provider.

In this situation, company preferences and demands need to be acknowledged to achieve a high degree of ABC satisfaction. Therefore in choosing a cloud service the preference of ABC should be taken into consideration and correctly evaluated. In our approach, we took the customer preferences into account and evaluated each QoS criterion’s weight.

Table 1 QoS parameters and their definition

3.2 Evaluation criteria

An important problem with the selection of cloud services is that identification of key criteria which determine whether the cloud service provided meets the business and technical demands of cloud clients. A cloud customer has two types of requirements: functional and non-functional. In short, a functional requirement refers to a service’s specific behavior and performance of a service are denoted by non-functional requirement. Due to complexity of cloud service and the absence of standard metrics, it’s a very difficult task to choose appropriate cloud QoS criteria. In fact, many researchers made a large number of efforts in the cloud service selection. [31] proposed the cloud service measurement index (SMI) that groups cloud QoS criteria into seven key categories which are widely accepted as a criterion for cloud service selection. Some QoS parameters which are explained in [20] and shown in Table 1 is also used to accurately measure the cloud provider’s service level. An intuitive calculation is performed for each attribute and its results can be considered as an input for all approaches to cloud service selection.

Fig. 2

Flowchart of proposed service selection methodology

4 Cloud service selection methodology

Here, we present our proposed service selection methodology as highlighted with the help of a flow chart shown in Fig. 2.

We have also shown the framework in Fig. 3 which uses our proposed methodology to rate different cloud services according to user preferences and QoS information provided by the cloud service providers. It consists of the 4 key components, i.e., (1) Cloud Service Repository (2) Cloud Service Discovery Engine (3) Cloud Service Selection Engine (4) Cloud Service Prioritizing Engine and (5) Cloud Service Pool.

Fig. 3

Proposed cloud service selection framework

• Cloud Service Repository It contains the detailed cloud provider information like provider id, artifacts, functional and non-functional specification, service level agreement (SLA).

• Cloud Service Discovery Engine This service mainly interacts with cloud clients and discovers different cloud services based on their preferences. Two essential services are: (1) cloud service matchmaker and (2) cloud service filter. This component compares the various services and the end-user service requirements. Finally, this component provides a list of potential cloud service providers in the service selection process.

• Cloud Service Selection Engine This component is responsible for QoS value conversion and cloud service discovery.

• Cloud Service Prioritizing Engine This component collects information from all the above and provides the prospective cloud customer with a list of optimal cloud service providers to make a decision.

• Cloud Service Pool This component stores information about the cloud services and their features advertised by various cloud providers.

The detailed description of the proposed methodology is as follows:

Step 1: Construct a decision matrix Let us suppose m cloud service provider alternatives $P_i$ ($i=1,\ldots ,m$) to be evaluated against n QoS selection criteria $Q_j$ ($j=1,\ldots ,n$). The decision matrix $DM= (x_{ij})_{m*n}$ denotes the QoS information of alternative $P_i$ corresponding to criterion $Q_j$. The decision matrix can be expressed as:

$$\begin{aligned} DM = \begin{bmatrix} x_{11} &{} \ldots &{} x_{1n} \\ \vdots &{} \ddots &{} \vdots \\ x_{m1} &{} \ldots &{} x_{mn} \end{bmatrix} \end{aligned}$$

(1)

Step 2: Employ entropy weight method for objective assessment The approach relies on unbiased data, so it is capable of overcoming disturbances caused by man in order to obtain results with higher accuracy. Entropy theory is about thermodynamics and Shannon presents it in information theory first [32]. It is used to calculate the degree of disorder in terms of probability theory between the set of information. The existing facts say if there is a big difference between the QoS values of alternative, then data information is more beneficial to select different cloud service alternatives. Thus, entropy method prioritizes various QoS criteria by calculating their entropy weights. The steps are as follows:

Step 2.1: Determine the characteristics proportion of the cloud service alternatives For m cloud service alternatives, there are m different QoS values for a certain criterion. These values may be same or different in terms of scales and units. This indicates different levels of probability among different values for a particular criterion. Hence, we utilize characteristics proportion that is the probability ratio of criterion value with respect to the sum of other provider’s values. The characteristics proportion ($p_{ij}$) of the $j$th QoS criteria for $i$th cloud service is defined as follows:

$$\begin{aligned} p_{ij}= \frac{x_{ij}}{\sum _{i=1}^{m}{x_{ij}}} \qquad i=1,2,\ldots ,m; \qquad j=1,2,\ldots ,n \end{aligned}$$

(2)

Step 2.2: Compute the entropy value for each QoS criterion The probability theory indicates that larger the deviation in the values of all candidate cloud services for a certain QoS criterion, higher the importance (weighted) of this criterion is than other QoS criteria. For each QoS criterion, we can identify and calculate entropy as follows:

$$\begin{aligned}&E_j=-k\sum _{i=1}^{m}p_{ij} \ln p_{ij},\qquad i= 1,2,\ldots m; \qquad j= 1,2,\ldots n ;\nonumber \\&\quad k= \frac{1}{\ln m} \end{aligned}$$

(3)

where $E_j$ is the entropy value for $j$th QoS criterion of all the cloud service alternatives. The value of $E_j$ lies between 0 to 1.

Step 2.3: Finding the deviation degree for each QoS criterion Let $G_j$ is the deviation degree of jth criterion, then we calculate the value of $G_j$ as follows:

$$\begin{aligned} G_j= 1-E_j, \qquad j= 1,2,\ldots n \end{aligned}$$

(4)

Note that if the entropy value $E_j$ is small, then the deviation degree of $j$th criterion is large and provides larger amount of information. In other words, this $j$th criterion shows greater value and objectivity compared to other ones.

Step 2.4: Obtain the entropy weight for each QoS criterion For each criterion, we determine the entropy weight $ew_j$ as follows:

$$\begin{aligned} ew_j=\frac{G_j}{\sum _{j=1}^{n} (G_j)}, \qquad \sum _{j=1}^{n}ew_j=1 \end{aligned}$$

(5)

Step 3: Employ AHP method for subjective assessment This method measures the subjective weight of each QoS criterion. The steps are as follows:

Step 3.1: Breaks the decision problem into a hierarchy In this step, we break the cloud service selection problem into a hierarchy of interrelated decision elements (i.e., goal, evaluation criteria and service provider alternatives). The Fig. 4 has shown an example for 11 alternatives and 5 criteria.

Fig. 4

A decision hierarchy model in AHP

Step 3.2: Construct a pairwise comparison matrix and find criteria weights The pair-wise comparison matrix A is shown in Eq. 6 based on discrete scale preferences from 1 to 9. We have n QoS criteria and each element of matrix A, i.e., $a_{ij}$ $(i, j = 1, 2, \ldots , n)$ denotes preference of $i$th criterion over $j$th criterion.

$$\begin{aligned} A = \begin{bmatrix} 1 &{} a_{12} &{} \ldots &{} a_{1n} \\ a_{21} &{} 1 &{}\ldots &{} a_{2n} \\ \ldots &{}\ldots &{}\ldots &{}\ldots \\ a_{n1} &{} a_{n2} &{} \ldots &{} 1 \end{bmatrix} \end{aligned}$$

(6)

where, $a_{ij}$ = $\frac{1}{a_{ji}}, a_{ji} > 0$

Next, the weight vector $W= \{sw_{1},\ldots ,sw_{n} \}$, where $sw_{i}$ is subjective weight of $i$th criterion, is determined by solving the following characteristic equation.

$$\begin{aligned} AW=\lambda _{max}W \end{aligned}$$

(7)

where $\lambda _{max}$ is the maximum eigenvalue of A and equals to the sum of the elements of the column vector AW, i.e., $\lambda _{max}=\sum AW$. The W satisfies the following normalization condition.

$$\begin{aligned} \sum _{j=1}^{n}~sw_j=1, ~sw_j\ge 0, ~j\in \{1,...,n\} \end{aligned}$$

(8)

Step 3.3: Check Consistency The pair-wise comparison matrix should be consistent. To test this, with the help of Eqs. 9 and 10, we determine the consistency index (CI) and consistency ratio (CR).

$$\begin{aligned} CI= & {} \frac{\lambda _{max} - n}{n-1}\end{aligned}$$

(9)

$$\begin{aligned} CR= & {} \frac{CI}{RI} \end{aligned}$$

(10)

Here, RI is random index value as per Table 2. If CR $<0.10$, the consistency of matrix is acceptable; otherwise, it required to be revised in Step 3.2.

Table 2 Average random index values

Step 4: Combine objective and subjective weights for each QoS criterion The weights obtained from objective and subjective assessments are now combined ($cw_{j}$) using Eq. 11.

$$\begin{aligned} cw_j=ew_j*sw_j \qquad j= 1,2,\ldots n \end{aligned}$$

(11)

where, $ew_{j}$ denotes the objective weight (Step 2.4) and $sw_{j}$ denotes the subjective weight (Step 3.2). Further, we normalize the combined weight $cnw_{j}$ using Eq. 12.

$$\begin{aligned} cnw_j = (cw_1, cw_2, \ldots , cw_n)^T j= 1,2,\ldots n \end{aligned}$$

(12)

These normalized weights are next used in TOPSIS.

Step 5: Calculate the final rank using TOPSIS This approach is very beneficial in solving ranking issues where numerous alternatives can be evaluated using the same criteria. This method provides two ideal solutions, i.e., the positive ideal solution and the negative ideal solution. On the one hand, we minimize the cost criteria and simultaneously maximize the benefit criteria in the positive ideal solution. On the other hand, we maximize the cost criteria and minimizes the benefit criteria in the negative ideal solution. This approach allows us to choose the best alternatives with the shortest distance from the positive ideal solution and the farthest away from the negative ideal solution. Due to its computational simplicity and efficient distance measurement, the TOPSIS method is extensively used in many area [33, 34]. The detailed description is as follows:

Step 5.1: Formation of the weighted normalized decision matrix Here, we multiply the columns of the normalized decision matrix $nx_{ij}$ (in Eq. 13) with combined weights $cnw_{j}$ as shown in Eq. 14 to obtain a weighted decision matrix.

$$\begin{aligned}&nx_{ij}= \frac{x_{ij}}{\sqrt{\sum _{i=1}^{m}{x^2_{ij}}}} \qquad i= 1,2, \ldots , m; \nonumber \\&\quad j= 1,2, \ldots , n. \end{aligned}$$

(13)

$$\begin{aligned}&v_{ij}= cnw_{j} * nx_{ij} \end{aligned}$$

(14)

Step 5.2: Determination of the positive and negative ideal solutions Here, positive ideal solution is denoted by $v_{j}^{+}$ and negative ideal solution is denoted by $v_{j}^{-}$ and determined for each criterion as follows:

$v_{j}^{+}$ = max $\{v_{1j},$ ...$v_{mj}\}$ and $v_{j}^{-}$ = min $\{v_{1j},$ ...$v_{mj}\}$ for benefit criteria

$v_{j}^{+}$ = min $\{v_{1j},$ ...$v_{mj}\}$ and $v_{j}^{-}$ = max $\{v_{1j},$ ...$v_{mj}\}$ for cost criteria

Step 5.3: Calculation of the separation measure from each cloud service alternatives It is calculated by calculating the distance between each alternative and positive and negative ideal solution using the Euclidean distance as follows:

$$\begin{aligned}&D_{i}^{+}= \sqrt{\sum _{j=1}^{n}(v_{ij}-v_{j}^{+})^2} \end{aligned}$$

(15)

$$\begin{aligned}&D_{i}^{-}= \sqrt{\sum _{j=1}^{n}(v_{ij}- v_{j}^{-})^2} \end{aligned}$$

(16)

Step 5.4: Calculation of the relative closeness to the ideal solution It is measured as follows:

$$\begin{aligned} CC_{i}= \frac{D_{i}^-}{D_{i}^- + D_{i}^+} \end{aligned}$$

(17)

The cloud service alternatives are ranked according to the descending order of $CC_{i}$ value. The cloud service alternative having highest relative closeness (CC value) is chosen to be the best cloud service provider.

5 Applications: illustrative view

The section presents illustrative point of view for two different applications applied on two different datasets.

5.1 Application-I: evaluating the high CPU based cloud service

This application is designed to assist in the selection of cloud services with a high CPU configuration. High-CPU cloud services are best suited for high-performance computing tasks. The detailed explanation is as follows.

5.1.1 Data set and experimental setup

We used the cloudharmoney.com platform to collect data for several cloud service providers. QoS performance reports are generated in real time by this platform for a variety of cloud service providers. These reports are based on the aggregate experience of millions of Internet users connecting via various networks.

Amazon, Azure, Google, Rackspace, Digital Ocean, and IBM SoftLayer are some of the cloud service providers being explored for performance evaluation. A variety of high-CPU efficient cloud services are available from these providers. For performance evaluation, we consider six cloud service providers. Response time, Availability, Reliability, Throughput, and Cost are the QoS criteria used to evaluate cloud service providers. The collected dataset is given in Table 3.

Note that our testing dataset is restricted to particular QoS factor chosen by decision makers. Apart from the QoS factors studied in this study, there are various more parameters that influence cloud service provider selection. Suitability, Sustainability, Scalability, usability, and so on are examples. However, the criteria used to evaluate cloud service providers may differ from one scenario to the another scenario. The preferences of the cloud customer/decision maker have a large role in the selection of QoS criteria.

5.1.2 Performance evaluation

Here, the cloud service selection and rating procedure is carried out using our proposed approach. Figure 2 depicts a flow diagram of the implementation process. The following steps are taken to implement the same.

Table 3 Decision matrix DM

• Decision matrix formation A decision matrix is constructed using five cloud service alternatives, namely Amazon EC2, Digital Ocean, Google, Microsoft Azure, Rackspace and Softlayer. The QoS criteria are Response time ($Q_1$), Availability ($Q_2$), Reliability ($Q_3$), Throughput ($Q_4$) and Cost ($Q_5$). Here $Q_2$, $Q_3$, and $Q_4$ are considered as benefit criteria (it should be maximum). Remaining are considered as cost criteria (it should be minimum). Table 3 shows the decision matrix based on these six cloud alternatives and five QoS criteria.

• Calculate the objective weight of each criterion Using the characteristic proportion probability values (shown in Table 4) along with Eqs. 2–5, we compute the objective weight of each criterion. The results are shown in Table 5.

• Calculating the subjective weight of each criterion Using Eqs. 6–10, we determine the subjective weights. A pair-wise comparison matrix is shown in Table 6. We calculate the consistency ratio (CR), and subjective weight of each criterion as shown in Table 6. As CR (0.07) is within the tolerance limit having value less than 0.10, criteria weights are consistent and acceptable.

• Calculating combined weights for criteria Table 7 shows the combined weights (using Eqs. 11 and 12) which is further used in the TOPSIS for evaluating the performance of cloud service alternatives.

• Calculating final ranking using TOPSIS Next, cloud service alternatives are ranked using Eqs. 13–17. Table 8 shows the normalized decision matrix using Eq. 13. Next, Table 9 shows the the weighted normalized decisions matrix using Eq. 14. Finally, Tables 10 and 11 presents the results obtained using next steps of TOPSIS. It is clear by analyzing the closeness coefficient ($CC_i$) values in Table 11 that Amazon EC2 is ranked first as the best cloud services provider.

Table 4 Characteristic proportion for each criteria

Table 5 The entropy weight of each criteria

Table 6 Pair-wise comparison of calculated relative weights of QoS criteria

Table 7 Combined weight for each criteria

Table 8 Normalized decision matrix

Table 9 Weighted normalized decision matrix

Table 10 Positive ideal solution and negative ideal solution for QoS criteria

Table 11 Distance between each service and ideal solution, trustworthiness values and ranking

5.2 Application-II: IaaS selection

This application is concerned about selection of IaaS services in order to test its applicability. We do this by examining a dataset [7] that comprises cloud service providers i.e. City cloud, Amazon, Rackspace, CenturyLink, HP and Google. For IaaS service selection, we examined QoS factors such as CPU Performance (QoS1), Memory Performance (QoS2), Disk Performance (QoS3), Network Latency (QoS4), and Cost on Demand (QoS5). Table 12 summarizes the dataset. Here, QoS1, QoS2, and QoS3 are benefit criteria (maximum should be preferable), and the remaining QoS4 and QoS5 are cost criteria (minimum should be preferable). The step-wise process applied on the used dataset is same as in Sect. 5.1.2. The results are shown in Table 13. According to the findings, Google is the most suitable service for choosing, followed by Amazon, Rackspace, HP, City Cloud, and Century Cloud.

Table 12 Decision matrix for IaaS Selection

Table 13 Overall score and final ranks of IaaS service selection

Fig. 5

Comparison analysis for subjective, objective, and combined assessments

6 Comprehensive analysis

We conducted experiments in MATLAB R2015b on Windows 7, 64-bit operating system with Intel(R) Core(TM) i3-2330M CPU @ 2.20 GHz processor and 8 GB installed memory. The proposed evaluation methodology can be introduced into cloud markets since there are no proper quality attributes evaluation methods available. The proposed approach is validated through a case study (see Sect. 5) that utilizes the real QoS dataset. The goal is to judge whether the proposed strategy is feasible, effective, and beneficial.

Fig. 6

Rank comparison of Application-I

Fig. 7

Rank comparison of Application-II

6.1 Ranking analysis based on subjective, objective and combined assessments

The importance of combining subjective and objective weights can be seen from Fig. 5 where there is a variation between subjective and objective weights. For example, $Q_4$ was regarded as the most important criterion with a subjective weight of 0.3373, but it resulted in the least important one having value of 0.029 in terms of objective assessment. Therefore, comparatively the combined approach of weighting is more suitable for a well-rounded weighted outcome. Taking a step forward, the equivalent rankings for the six alternatives can be obtained as Amazon EC2 > Rackspace > Google > Digital Ocean > Microsoft Azure > Softlayer for the subjective assessment, and Amazon EC2 > Google > Rackspace > Digital Ocean > Softlayer > Microsoft Azure for the objective assessment. These rankings appear to be inconsistent compared to the sequence generated by combined assessment, implying that the weights of the criteria may influence the final ranking of the alternatives; thus, the combined weighting method would provide a more comprehensive ranking result.

6.2 Result validation

Here, we conduct a comparative study to determine the efficiency of the proposed methodology. The overall goal is to elucidate the degree of correctness on ranks computed using the proposed method by validating against other MCDM methods such as AHP, AHP-TOPSIS, and Entropy-TOPSIS [12, 20, 35]. For comparing, we use the same QWS dataset with the same set of criteria as presented in our case study. Considering this comparative analysis, we observe that the suggested scheme ranks almost similar to the other methods. The ranks of each cloud service alternatives obtained using proposed process, AHP, AHP-TOPSIS, and Entropy-TOPSIS are displayed in the Fig. 6. For example, Amazon EC2 is the best option in all the methods. The cloud service provider Google, Rackspace and Digital Ocean is considered to be on second, third, and fourth rank, respectively in most of the methods. If we observe Google and Rackspace closely (based on Table 8), our approach ranks Google on second position unlike AHP and AHP-TOPSIS. In a similar vein, for application-II (as depicted in Fig. 7), 100% (i.e. 4/4) of the tested method agree on the top two rankings. On the other hand, 75% (3/4) of the tested methods agree on the third and last rank. We can see from the findings that our proposed methodology outperforms existing MCDM techniques. Finally, we may deduce that the rank acquired using the given methodology is more accurate and produces more consistent results.

6.3 Rank correlation analysis

Correlation is a statistical measurement that measure the strength of association between two variables and makes inferences about their linear relationship.With this analysis, we try to determine the relationship between the obtained ranks of cloud services using the proposed methodology and other compared methodologies. Therefore, in order to measure rank correlation analysis, we performed two non-parametric correlation tests i.e. Spearman’s rho and Kendall’s tau

Fig. 8

Spearman’s rank correlation

6.3.1 Spearman’s rank correlation coefficient

In order to assess the similarity between two sets of ranking values, we calculate the spearman rank correlation coefficient. The spearman rank correlation coefficient, denoted by $\rho$, is calculated using the following equation:-

$$\begin{aligned} \rho =\frac{\sum _{i=1}^{n} \{(x_{i}-{\overline{x}})(y_{i}-{\overline{y}})\}}{\sqrt{\sum _{i=1}^{n}(x_{i}- {\overline{x}})^2}{\sqrt{\sum _{i=1}^{n}(y_{i}- {\overline{y}})^2}}}, \end{aligned}$$

(18)

where $x_{i}$and $y_{i}$ are the ranks of each compared method and ${\overline{x}}$ and ${\overline{y}}$ are the averages of rank values. Spearman’s rank correlation coefficient value varies between 1 and $+$ 1. Here, the value − 1 denotes a perfect negative correlation between two rank orderings, the value of $+$ 1 denotes a perfect positive correlation between two rank orderings and the value 0.00 represents a lack of correlation between two rank orderings. Figure 8 shows the test findings. Here, we observe that for application-I, there is a perfect degree of correlation between the ranks of proposed methodologies and Entropy-TOPSIS method. We also note a high degree of correlation against AHP, AHP-TOPSIS methods. Likewise, for application-II, a very high degree of correlation can be observed between the proposed methodology and all other considered methods.

Fig. 9

Kendall’s rank correlation

6.3.2 Kendall’s rank correlation

To determine the relationship between two measured quantities, the Kendall rank correlation coefficient, or Kendall’s Tau ($\tau$) coefficient, is used. It takes two ranks that contain the same elements and calculates the correlation between them. With the help of Kandall’s Tau, we try to find out the similarity between ranking using concordant and discordant pairs from same set of objects.

$$\begin{aligned} \tau _{X,Y}=\frac{n_c-n_d}{\sqrt{(n_0-n_X)(n_0-n_Y)}} \end{aligned}$$

(19)

where $n_c$ and $n_d$ are concordant and discordant pairs respectively. Kendall’s rank correlation coefficient value varies between 1 and $+$ 1. Here, the value − 1 denotes the discordance is perfect (the ranking of one of variables is reverse to the other) and the value of $+$ 1 indicates rankings are the same (the concordance between two variables is perfect). The value of zero stands for non relationship. Figure 9 shows that the ranks of proposed methodology and other compared method have a higher degree of positive correlation for application-I and application-II.

Finally, we may infer that the ranking performance of the proposed methodology is comparable to the existing MCDM approaches based on the findings of rank correlation analysis.

Fig. 10

Ranks of application-I services in the sensitivity analysis

Fig. 11

Ranks of application-II services in the sensitivity analysis

6.4 Sensitivity analysis

This subsection validates the robustness and efficiency of the suggested scheme using sensitivity analysis. To carry out the sensitivity analysis, we check how the cloud service provider’s ranking may change under different weight values. In this scenario, we execute the whole process to monitor the changes in various circumstances. By swapping weights of each QoS criterion one by one, we generated various scenarios. For example, Q1–Q2 implies that Q1 and Q2 criterion weights have been exchanged. The ranks of cloud service providers are determined for each case by evaluating the effect of changes in criterion weight. For this experiment, we have conducted a total of 10 experiments. For each experiment we calculate $CC_i$ value. If the ranks of cloud service alternatives are consistent in each scenario, the proposed methodology is said to be robust. From the results, it can be noted that the proposed approach is robust and able of recommending the best cloud service providers. Figure 10 indicates that AmazonEC2 is the best alternatives in all conditions. Similarly, as per the findings of the sensitivity analysis for application-II, Google is the optimal service (See Fig. 11). This is the same result as we got in Sect. 5.2. These findings indicate that the suggested approach is stable in nature and rarely sensitive to changes in criterion weights.

7 Concluding remarks

7.1 Discussion

Generally, QoS information and individual preferences of users or experts can be outlined as two main aspects during the MCDM process [36]. Apart from weighting methods, QoS information aggregation is also significant in service selection, which requires a proper aggregation technique. The primary need of any decision making process is how to aggregate QoS information on alternatives and individual customer preference information on criteria [37]. The greatest strength of our approach is to consider and efficiently integrate not only the QoS data but also the subjective as well as objective preferences. Therefore, the proposed approach makes the decision results more reliable and consistent. This illustrates the strength of suggested method when dealing with complex cloud service selection problems. However, we are not claiming that our proposed approach is the best method. Besides, our approach will also impact in the decision-making process at the information aggregation level. It is also applicable to such real-world decision problems which usually involve technical data and qualitative information. Specifically, situations where decision is made on both quantitative and qualitative criteria, highlight the validity and effectiveness of the proposed approach compared to existing approaches. Collection of accurate data on quantified decision alternatives is important and critical to accurate derivation of objective weights using the Entropy method. Moreover, adopting a team-based pairwise comparison for deriving subjective weights is equally important to come up with realistic and representative assessments.

7.2 Conclusion

This study proposes a novel approach to solve the cloud service selection problem. The proposed approach considers both objective as well as subjective aspects. Here, the objective weights are derived from the QoS criteria information provided by a reliable third party while the subjective weights are derived using cloud customers’ preferences with respect to different QoS criteria. Finally, the combined weights are integrated and used to evaluate the ranks of the service providers which in-turn gives the best cloud service provider. We validate the proposed approach through a real-life case study. The experimental results are presented to confirm the reliability of the proposed scheme. The results show that Amazon EC2 outperforms than other service providers. The robustness and consistency of the proposed methodology is also ensured with the use of sensitivity analysis. In our future work, we will evaluate the performance of the proposed algorithm by integrating it with new MCDM approaches using other real-life test cases.