Keywords

1 Introduction

With the fast development of cloud computing, numerous services with similar functions have been presented to cloud customers at different prices and performances, therefore it is a challenge for customers to select the most “worthy” cloud services. Often, there may be trade-off between cost and performance which makes it difficult to evaluate services from different cloud providers in an objective way.

Lots of work have been done for comparing different cloud services. Some researchers specified factors or attributes important for evaluating cloud computing models [5, 10], such as security, availability, performance, cost, etc. Based on these attributes, cloud services are compared and ranked for selection.

Some of existing works [8, 11, 15, 16] compared services only based on performances while the cost was not considered. Some researchers considered both. Brebner [3] used statistical graphs to compare cost of different cloud scenarios. Li [9] proposed Cloudcmp to compare public cloud providers on different aspects separately. The work in [13] analyzed the application of different multicriteria decision analysis (MCDA) methods to cloud services selection, including AHP, TOPSIS and other methods taking multiple QoS attributes and cost into account. Saurabh [6] introduced a framework named SMICloud using AHP method. It is parametric and requires customers to define the preference value for each factor first. However, sometimes we might only care about the minimum performance and the maximum cost, while the preference of QoS attributes is of little concern. In this case, the AHP method might not be appropriate, because different preference values will give different results.

DEA is a non-parametric method suitable for situation without user’s preference. In this paper, we introduce how to apply DEA to evaluate and compare a group of cloud services according to their prices and performances. According to features of DEA models and characteristic of cloud services, we combine results of \(\mathrm {C}^2\mathrm {R}\) [4] and \(\mathrm {BC}^2\) [2], and define several efficiency levels for cloud services based on their relative efficiency scores and slacks. Thereby customers can select services of higher efficiency level. Moreover, for less efficient services, we show how they should be improved in order to be more efficient.

2 Classifying Cloud Services with DEA Models

2.1 DEA Models

Here we introduce how to construct DEA models for calculating relative efficiency scores and slacks. The problem can be defined as following: given n cloud services, each having s QoS attributes and m cost items, finding out which service’s QoS value could be improved with the cost remaining the same. The cloud services, QoS attributes and cost items are modeled as DMUs, output variables and input variables respectively.

The \(\text {C}^2\text {R}\) and \(\text {BC}^2\) models are used to calculate efficiency scores. The \(\text {C}^2\text {R}\) model is designed with the assumption of constant returns to scale, meaning that if all inputs are doubled, the outputs are also expected to double. The \(\text {BC}^2\) model takes into account that the performance at the most efficient point may not be attainable and thus assumes variable returns to scale.

The \(\text {C}^2\text {R}\) model \(D^I_{C^2R}\) is given in Eq. 1, where \(x_{ij}\) and \(y_{rj}\) are constants representing the ith input and rth output for the jth service, and \(\theta \) and \(\lambda _j\) are variables of the LP problem. The \(\text {BC}^2\) model \(D^I_{BC_2}\) is similar to \(D^I_{C^2R}\) with one more constraint \(\sum _{j=1}^{n}\lambda _{j}=1\). The optimal values of the two models \(h^{I}_{C^2R}\) and \(h^{I}_{BC^2}\) are called technical and scale efficient score (TSE score) and technical efficient score (TE score) respectively. It can be inferred that if TSE score is 1, then TE score is 1, but not vice versa.

$$\begin{aligned} D^I_{C^2R} {\left\{ \begin{array}{ll} \min \theta = h^{I}_{C^2R} &{}\\ \mathop {\sum }\nolimits _{j=1}^{n}\lambda _{j}x_{ij}\le \theta x_{i0} &{} i=1,\dots ,m \\ \mathop {\sum }\nolimits _{j=1}^{n}\lambda _{j}y_{rj}\ge y_{r0} &{} r=1,\dots ,s\\ \lambda _{j}\ge 0 &{}j=1,\dots ,n \end{array}\right. } \end{aligned}$$
(1)

Because of the existence of weak efficient point which is on the efficient frontier but still possible to be improved, we apply the slack-based method [12] to calculate slacks. The model \(\bar{D}^I_{C^2R}\) expressed by Eq. 2 can be used to check if a service is weak technical and scale efficient (W-TSE). Accordingly, a similar model \(\bar{D}^I_{BC^2}\) with one more constraint (\(\sum _{j=1}^{n}\lambda _{j}=1\)) can check weak technical efficient (W-TE). Here, \(\lambda _j, s_i^-\) and \(s_r^+\) are variables of the equations. \(s_i^{*-}\) and \(s_r^{*+}\) are values of \(s_i^-\) and \(s_r^+\), the sum of which is the optimal value (called slack) of the model. If the slacks for the two models are greater than 0, the cloud service is called W-TSE and W-TE respectively.

$$\begin{aligned} \bar{D}^I_{C^2R} {\left\{ \begin{array}{ll} \max \sum _{i=1}^{m}s_i^- + \sum _{r=1}^{s}s_r^+= \sum _{i=1}^{m}s_i^{*-} + \sum _{r=1}^{s}s_r^{*+}\\ \sum _{j=1}^{n}\lambda _{j}x_{ij} + s_i^- = h^{I}_{C^2R}x_{i0} &{} i=1,\dots ,m \\ \sum _{j=1}^{n}\lambda _{j}y_{rj} - s_r^+ = y_{r0} &{} r=1,\dots ,s \\ \lambda _{j}\ge 0 &{} j=1,\dots ,n \\ s_i^- \ge 0 &{} i=1,\dots ,m \\ s_r^+ \ge 0 &{} r=1,\dots ,s \end{array}\right. } \end{aligned}$$
(2)
Table 1. Service efficiency types and levels
Full size table

2.2 Efficiency Levels

Table 1 shows different efficiency types and their levels, where level 1 is the most efficient and level 5 is the least. Consider the cases of the second and the third row, if a cloud service is W-TSE, it can be improved by the distance of the slack. However, a TE service might not be able to be improved, because its improvement requires moving along the frontier of the \(\text {C}^2\text {R}\) model and the efficient target might be an impossible situation for a cloud service. For example, the percent of SLA agreement is bounded to the range of [0,1] and it cannot be greater than 1 in order to be efficient. Therefore, if a cloud service is TE, we believe that its efficiency is better under current cloud services group. The order of efficiency level of 3, 4 and 5 is obvious. According to the above discussion, we can have the efficiency order defined in Table 1.

2.3 Method Implementation

Now we give detailed steps to evaluating cloud services’ efficiency with DEA.

The first step is preparing data for the model. One thing is to check non-numeric value and missing value. Non-numeric value should be transformed to numeric value according to its definition. For example, if a service’s security protection level is specified as poor, average or extensive, these values can be transformed to integer values of 1, 2 and 3, with larger value meaning better performance. Missing value can be filled with the average value of the attribute or predict its value with a precise assumption [7, 14]. The other thing is to check QoS attributes. DEA generally minimizes “inputs” and maximizes “outputs”. In other words, smaller levels of the former and larger levels of the latter represent better efficiency. Therefore, for attribute having smaller value for better performance, such as response time, we use the reciprocal of the variable instead.

The second step is to determine efficiency level of a cloud service based on its efficiency scores and slacks. First we calculate the TSE score with the model \(D^I_{C^2R}\). If it is equal to 1, the model \(\bar{D}^I_{C^2R}\) is used to obtain the slack. If the slack is 0, then the cloud service is of level 1, otherwise it is of level 3. If the TSE score is smaller than 1, we continue to solve the model \(D^I_{BC^2}\). If the TE score is smaller than 1, the service is inefficient and of level 5. Otherwise, the model \(\bar{D}^I_{BC^2}\) is applied to get the slack of \(\text {BC}^2\) model. The service is of level 2 if the slack is 0, otherwise it is of level 4.

The last step is to obtain the improving targets of less efficient cloud services by calculating their projections on frontier of either \(D^I_{C^2R}\) model or \(D^I_{BC^2}\) model. If a cloud service’s efficiency is of level 2 or 3, its projection can be obtained according to Eq. 3, where \(s_i^{*-}\) and \(s_r^{*+}\) are from the optimal value of model \(\bar{D}^I_{C^2R}\). If the efficiency level of the cloud service is 4 or 5, its projection can be obtained according to Eq. 4, where \(s_i^{*-}\) and \(s_r^{*+}\) are from the optimal value of model \(\bar{D}^I_{BC^2}\).

$$\begin{aligned} (\mathrm {P}-j_0) {\left\{ \begin{array}{ll} \hat{x}_{i0} = h^{I}_{C^2R} x_{i0} - s_i^{*-}, \text { for } i =1,\dots ,m \\ \hat{y}_{r0} = h^{I}_{C^2R} y_{r0} + s_r^{*+}, \text { for } r = 1,\dots ,s \end{array}\right. } \end{aligned}$$
(3)
$$\begin{aligned} (\mathrm {P}-j_0) {\left\{ \begin{array}{ll} \hat{x}_{i0} = h^{I}_{BC^2} x_{i0} - s_i^{*-}, \text { for } i =1,\dots ,m \\ \hat{y}_{r0} = h^{I}_{BC^2} y_{r0} + s_r^{*+}, \text { for } r = 1,\dots , s \end{array}\right. } \end{aligned}$$
(4)

3 Case Study

We illustrate our method by evaluating relative efficiency of a group of IaaS services from an online data set [1]. Due to limited pages, we select 15 services from three cloud providers. The input data and analysis results are listed in Table 2, where p1s1 corresponds to the first service from the first provider, and (0.12,1,2,3.75,410) represents values of price/hour, number of virtual cores, compute units, memory(GB) and disk(GB).

Table 2. Case study result
Full size table

It can be seen that services from the first and second providers are more efficient. Projections for services of level 1 are the same as their original values, while others are targets for improving services’ performance in order to become efficient. For example, the projection of service p2s2 is (0.12,4,4,4,80), whose input data is (0.12,2,4,4,80), thus its number of virtual cores should be changed from 2 to 4, and the distance between the two value (4\(-\)2=2) is slack for virtual cores. From all the slack values, we can see that services from the third provider need more improvement.

4 Conclusion and Future Work

In this work, non-parametric method based on DEA model is applied to evaluate the relative efficiency of cloud services, which is the ratio of performance and price. The method can evaluate and classify cloud services by analyzing their efficiency, and further show how to improve inefficient services.

Our work can classify a group of cloud services according to their efficiency levels, but services of the same level cannot be distinguished yet. Next, we will study how to distinguish services of the same efficiency level so as to offer more support for user’s decision.