A comparative analysis of multi-criteria decision-making methods

Abstract

In this work, we empirically compare the rankings produced by several multi-criteria decision-making methods. We analyzed multi-MOORA, TOPSIS and three different settings for VIKOR. Using decision matrices with different number of alternatives and criteria, we compared the rankings produced using the Spearman’s correlation coefficient index. Our results showed that VIKOR could fail to obtain a ranking due to the failure of certain calculations. The rankings produced by TOPSIS and multi-MOORA were very similar, while the rankings produced by the different VIKOR variants showed a great variability.

1 Introduction

Multi-criteria decision-making (MCDM) methods are mathematical models that help to take decisions in scenarios where the possible alternatives are evaluated over multiple conflicting criteria.

The application areas of these methods are huge [8]. Examples can be found in supplier selection [18], technical evaluation of tenderers [10], evaluation of service quality [9] or in renewable energy [1].

When facing a specific MCDM problem, there are no clear guidelines on which MCDM method should be used to solve it. This issue is controversial and it has been studied in the literature since many decades ago [15–17, 19]. It is true that depending on the MCDM method applied, the solution could be different, specially when the alternatives are very similar. Therefore, we seek to do a comparative analysis among some MCDM methods, in order to better understand their similarities and differences. The long-term goal is to have guidelines to support the decision-maker in the selection of which MCDM method to apply. We consider this work a step towards such goal.

There are many MCDM methods in the literature, as PROMETHEE [3, 4], AHP [14], ELECTRE [13], etc. In this work, we focus on multi-MOORA [2, 5], TOPSIS [7] and VIKOR [11, 12]. Multi-MOORA applies aggregation operators, while TOPSIS and VIKOR operates calculating distances to “ideal” or “reference” points. We selected these methods for comparison because they have the same input and all of them rely on a normalization procedure.

The comparison among methods is done over a set of randomly generated decision matrices, as in [19] and then the ranking “agreement” between pairs of methods is assessed through the Spearman’s correlation coefficient.

The remainder of this paper is organized as follows. Section 2 provides a brief overview on what is an MCDM problem, and describes the basic calculations of multi-MOORA, TOPSIS and VIKOR. Section 3 describes the experimental framework and the results of the experiments. Finally, Sect. 4 is devoted to conclusions.

2 Multi-criteria decision-making problem and methods

An MCDM problem [15] is composed by a finite set of alternatives represented as $A = \{A_{i} \, | \, i=1,2,\ldots ,m \}$, m being the number of the alternatives. The alternatives are evaluated according to certain criteria, denoted as $C = \{C_{j} \,|\, j=1,2,\ldots ,n \}$, where n is the number of the criteria. The criteria can have different domains, and may represent a cost (which is desirable to minimize) or a benefit (desirable to maximize). In addition, each criterion is assigned an importance weight, represented as $W = \{w_{j} \,|\, j=1,2,\ldots ,n \}$. These weights are normalized to add up to one, i.e., $\sum ^{n}_{j=1}w_{j}=1$.

This information is organized in a decision matrix ($M^{m \times n}$) as in Table 1, where each element $x_{ij}$ represents the value of the alternative $A_{i}$ with respect to the criterion $C_{j}$. The matrix M and the vector of weights $W = \{w_1, w_2, \ldots , w_{n}\}$ are the fundamental inputs for the MCDM methods that we will consider here.

Table 1 Decision matrix of an MCDM problem

2.1 TOPSIS method

The TOPSIS method [7] evaluates the alternatives in terms of their distance to the so-called “positive” and “negative” ideal solution. TOPSIS is composed by the following steps:

Step 1  Normalize the decision matrix replacing every $x_{ij}$ by $n_{ij}$ using the following formula:

$$\begin{aligned} {n}_{ij} = \frac{x_{ij}}{\sqrt{\sum _{j=1}^{m}(x_{ij})^2}}, \end{aligned}$$

(1)

where $i=1,2,\ldots ,m$ and $j =1,2,\ldots ,n$.

Step 2  Calculate the weighted normalized values as ${v}_{ij} = w_{j} * {n}_{ij}$, where $w_{j}$ correspond to the weight of the jth criterion, $i=1,2,\ldots ,m$ and $j =1,2,\ldots ,n$.

Step 3  Calculate the positive ideal solution (PIS), ${A}^+$, and the negative ideal solution (NIS), ${A}^-$, as follows:

$$\begin{aligned} \begin{array}{ccc} (\mathrm{PIS}) &{}=&{} {A}^+ = \{{v}_{1}^+, {v}_{2}^+, \ldots , {v}_{j}^+, \ldots , {v}_{n}^+ \},\\ (\mathrm{NIS}) &{}=&{} {A}^- = \{{v}_{1}^-, {v}_{2}^-, \ldots , {v}_{j}^-, \ldots ,{v}_{n}^-\}, \end{array} \end{aligned}$$

(2)

where ${v}_{j}^+ = \max _i({v}_{ij})$ and ${v}_{j}^- = \min _{i}({v}_{ij})$ if the $j^{th}$ criterion is benefit; and ${v}_{j}^+ = \min _{i}({v}_{ij})$ and ${v}_{j}^- = \max _i({v}_{ij})$ if the $j^{th}$ criterion is cost, $i=1,2,\ldots ,m$ and $j=1,2,\ldots ,n$.

Step 4  Calculate the distances from every alternative to the ideal solutions, $d_{i}^+$ being the distance to ${A}^+$, and $d_{i}^-$ the distance to ${A}^-$ as following:

$$\begin{aligned} d_{i}^+= & {} \left\{ \sum _{j=1}^{n}(v_{ij}-v_{j}^+)^2 \right\} ^{1/2},\nonumber \\ d_{i}^-= & {} \left\{ \sum _{j=1}^{n}(v_{ij}-v_{j}^-)^2 \right\} ^{1/2}, \end{aligned}$$

(3)

which correspond to the m-dimensional Euclidean distance and $i=1,2,\ldots ,m$.

Step 5  Calculate the relative closeness to both ideal solutions as following:

$$\begin{aligned} R_{i} = \frac{d_{i}^-}{d_{i}^++d_{i}^-}, \end{aligned}$$

(4)

where $i=1,2,\ldots ,m$. If $R_{i} = 0$, then $d_{i}^- = 0$ means that it is the worst possible case. On the other hand, if $R_{i}$ = 1, then $d_{i}^+$ = 0 means that it is the best possible case. In general, $0 \le R_{i} \le 1$.

Step 6  Rank the alternatives according to $R_{i}$ in descending order. The best alternative is the one with the highest $R_{i}$.

2.2 VIKOR method

The VIKOR method [11] is, as TOPSIS method, also based in the idea of the distances to “ideal solutions”. However, some differences exist between both methods, as stated in [12].

VIKOR method follows these steps:

Step 1  Determine the best ${f}_{j}^{*}$ and worst ${f}^-_{j}$ values of each criterion as ${f}_{j}^{*} = \max _{i}(x_{ij})$ and ${f}^-_{j} = \min _{i}(x_{ij})$, if the $j^{th}$ criterion is benefit, and as ${f}_{j}^{*} = \min _{i}(x_{ij})$ and ${f}^-_{j} = \max _{i}(x_{ij})$ if the jth criterion is cost, $i=1,2,\ldots ,m$ and $j=1,2,\ldots ,n$.

Step 2  Normalize the $x_{ij}$ values as follows:

$$\begin{aligned} n_{ij}=\frac{f^{*}_{j} - x_{ij}}{f^{*}_{j} - f^-_{j}}, \end{aligned}$$

(5)

where $i=1,2,\ldots ,m$ and $j=1,2,\ldots ,n$.

Step 3  Calculate the values $S_{i}$ and $R_{i}$, $i=1,2,\ldots ,m$ and $j=1,2,\ldots ,n$:

$$\begin{aligned} {S}_{i}= & {} \sum _{j=1}^{n} w_{j}*n_{ij}, \end{aligned}$$

(6)

$$\begin{aligned} {R}_{i}= & {} \max _{j} \left[ w_{j}*n_{ij}\right] \end{aligned}$$

(7)

Step 4  Calculate $Q_{i}$ as follows:

$$\begin{aligned} {Q}_{i} = v \frac{(S_{i} - S^{*})}{(S^- - S^{*})} + (1 - v)\frac{(R_{i} - R^{*})}{(R^- - R^{*})}, \end{aligned}$$

(8)

where $S^{*}= \min _{i}(S_{i})$, $S^-=\max _{i} (S_{i})$, $R^{*}= \min _{i}(R_{i})$, $R^-=\max _{i} (R_{i})$, and $v\in$[0,1]. Parameter v balances the relative importance of indexes S and R.

Step 5  Sort Q in increasing order. The best-ranked alternative is the one with the lowest value of Q.

Step 6  Compromise solution: the so-called compromise solution is the alternative $a'$ which is the best ranked according to Q (minimum) if the following two conditions are satisfied:

• Condition 1: Acceptable advantage. $Q(a'') - Q(a') \ge DQ$ where $a''$ is the second best alternative according to Q and $DQ = 1/(m-1)$ (m is the number of alternatives).

• Condition 2: Acceptable stability in decision-making. Alternative $a'$ must be also the best ranked according to S and/or R (the alternative with the lowest value).

If one of the conditions is not satisfied, then a set of compromise solutions is proposed, which consists of:

• The alternatives $a'$ and $a''$ if condition 1 is true and condition 2 is false, or

• The set of alternatives $a',a'',\ldots , a^{(p)}$ if condition 1 is false; p being the position in the ranking of the alternative $a^{(p)}$ verifying $Q(a^{(p)}) - Q(a') < DQ$.

The best alternative, ranked by Q, is the one with the minimum value of Q. The compromise ranking result is the compromise ranking list of the alternatives with the “advantage rate”.

2.3 Multi-MOORA method

Multi-MOORA constructs a ranking departing from three calculations: the “Ratio System”, the “Reference Point” and the “Full Multiplicative Form of Multiple Objectives” [5].

2.3.1 Ratio system

The first step is the normalization of the decision matrix. Normalization is done according to Eq. 1 (as in TOPSIS) and the values are denoted as $n_{ij}$. Then, the ratio $y_{i}^{*}$ of every alternative is calculated as follows:

$$\begin{aligned} {y}_{i}^{*}=\sum _{j=1}^{g} n_{ij}*w_{j} - \sum _{j=g+1}^{n} n_{ij}*w_{j}, \end{aligned}$$

(9)

where $i=1,2,\ldots ,m$, $j=1,2,\ldots ,g$ are the benefit criteria and $j=g+1,2,\ldots ,m$ are the cost criteria. A higher ratio $y_{i}^{*}$ implies a better ranking of the alternative.

2.3.2 Reference point

Initially, a reference point $r_{j}$ is calculated using the normalized values and the weights. It is defined as $r_{j}=\max _{j}(n_{ij}*w_{j})$ if $C_{j}$ is a benefit criteria, and as $r_{j}=\min _{j}(n_{ij}*w_{j})$ if $C_{j}$ is a cost criteria. Then, every alternative is assigned a value using the following metric:

$$\begin{aligned} \min _{i} (\max _{j} |{r}_{j} - n_{ij}*w_{j}|) \end{aligned}$$

(10)

The lower the value, the better the alternative is.

2.3.3 Full multiplicative form

An additional value $U_{i}$ is calculated for every alternative:

$$\begin{aligned} U_{i} = \frac{\prod _{j=1}^{g} n_{ij}^{w_{j}}}{\prod _{j=g+1}^{n} n_{ij}^{w_{j}}}, \end{aligned}$$

(11)

where $i=1,2,\ldots ,m$, $j=1,2,\ldots ,g$ are the benefit criteria and $j=g+1,2,\ldots ,m$ are the cost criteria. Finally, the best-ranked alternative according to the full multiplicative form is the one that has the highest value of U.

In order to construct the final ranking, multi-MOORA calculates a “summary of rankings” from the Ratio System, Reference Point and Full Multiplicative Form by applying the “Theory of Dominance” [6].

3 Experiments and results

3.1 Methodology

Our experiments departs from a set of randomly generated MCDM problems (i.e., decision matrices). These matrices were generated according to the procedure described in [19], where the following parameters were used:

• 1. Number of criteria: $n \in \{5, 10, 15, 20\}$.

• 2. Number of alternatives: $m \in \{ 3, 5, 7, 9\}$.

• 3. Values of the alternatives: $x_{ij}$: randomly generated from a uniform distribution in $[0.01,\ldots ,1]$.

• 4. Criteria Weights: all of them are considered equally important, thus $w_i=1/n$.

• 5. Number of replications: we have generated 100 matrices for each combination of m and n, thus producing $4 \times 4 \times 100 = 1600$ MCDM problems.

• 6. Methods: five methods are considered: TOPSIS, multi-MOORA (MM) and three different parametrization of VIKOR using $v = \{0, 0.5, 1\}$ (named as VIKOR$^0$, VIKOR$^{0.5}$, VIKOR$^1$). When using VIKOR, we just take the ranking generated with the Q index, without considering the conditions of the “Compromise solution”, since these are strong conditions (hard to meet) for the number of alternatives in our experiments.

Every MCDM problem is solved using every method, giving a total of 8000 rankings.

Then, we measure the agreement between every pair of rankings (one from each method) using the Spearman’s correlation coefficient ($\rho$). An index value closer to 1 indicates a high level of agreement (a “1” is obtained if the rankings are equal). If the index is close to 0, then there is no agreement between the rankings. Finally, if the index is close to $-$1, the rankings are almost inverted.

For every combination of number of alternatives and criteria, we analyze the results in terms of the average values of $\rho$ over the corresponding 100 MCDM problems.

The analysis of results is separated in three parts. The first part focuses on a problem affecting two variants of the VIKOR method (VIKOR$^0$ and VIKOR$^{0.5}$). The second part provides a comparison of the methods on those cases where VIKOR variants worked. Finally, in the third part, we compare multi-Moora, TOPSIS and VIKOR$^1$ over the whole set of problems.

3.2 A problem concerning the VIKOR method

We have observed that on some MCDM problems, VIKOR$^{0}$ and VIKOR$^{0.5}$ failed to produce a ranking. This failure does not appear with VIKOR$^1$.

The reason lies at the core of the R calculation performed in VIKOR (Eq. 7). In some problems, we observed that all the alternatives have the same $R_{i}$ value. As a consequence $R^{*} = R^-$ and the calculation of the Q (Eq. 8) index becomes indeterminate for any $v \ne 1$. Potentially, the same situation may occur with the $S_{i}$ values, but this never happened in our experiment.

Table 2 shows, for every combination of alternatives and criteria, the percentage of problems solved by VIKOR$^{0}$ and VIKOR$^{0.5}$. Increasing the number of criteria, led to a decrease in the percentage of problems solved (see the table by rows). Also, if the number of criteria is fixed (reading the table by columns), the problem becomes “more solvable” as the number of alternatives increases. For example, when $n=5$, the percentage of problems solved raised from 32 % when $m=3$ to 100 % when $m=7$ and $m=9$. Just in these last combinations ($n=5, m=\{7,9\}$), VIKOR is able to provide a ranking for all the MCDM problems available.

Table 2 Percentage of problems solved (out of 100) by VIKOR$^0$ and VIKOR$^{0.5}$

Figure 1 graphically shows the results of Table 2. It can be observed that the rate of problems solved according to the number of criteria varies in terms of the number of alternatives available. When having $n=20$ criteria, the decision problem should have much more than $m=9$ alternatives to produce results. However, when $n=5$, it is almost enough to have just $m=5$ alternatives to produce a ranking when the MCDM problems are generated as we did. In general, the percentage of problems solved increases with the number of the alternatives available.

Fig. 1

Percentage of problems solved (out of 100) by VIKOR$^0$ and VIKOR$^{0.5}$

3.3 First comparison: all methods over a subset of problems

Now, we will show an all-against-all comparison of the methods restricted to the cases where VIKOR$^0$ and VIKOR$^{0.5}$ solved more than 80 % of the problems. These six cases are $n=5, m = \{5,7,9\}$; $n=10, m = \{7,9\}$ and $n=15, m=9$. Recall that we had tested 100 problems in every case. The global results are shown in Table 3, where the average values of $\rho$ (over 600 MCDM problems) between every pair of methods is shown.

Table 3 Mean of $\rho$ over 600 MCDM problems (considering $n=5, m = \{5,7,9\}$; $n=10, m = \{7,9\}$ and $n=15, m=9$)

On average, the agreement of the rankings produced by TOPSIS and VIKOR$^1$, and TOPSIS with multi-MOORA is very high ($\rho \ge 0.87$). Less similar rankings are produced by VIKOR$^0$ and TOPSIS ($\rho =0.33$). Moreover, as the last four rows correspond to VIKOR$^0$, we can say that this method produced rankings that shows very little similarity with those produced by other methods. A similar observation can be done regarding VIKOR$^{0.5}$.

If we split the analysis in terms of the problem size, we obtain the results depicted in Figs. 2, 3 and 4. Figures show the mean of $\rho$ for every pair of methods. Figure 2 shows the case for 5 criteria and different number of alternatives (methods are sorted according with the results when $m=9$). Figure 3 considers 10 criteria and $m = \{7,9\}$ (methods are also sorted according with the results when $m=9$). Finally, Fig. 4 corresponds to $n=15, m=9$. The corresponding values for Fig. 2 are shown in Table 4.

Fig. 2

Mean of $\rho$ for 5 criteria and $m = \{5,7,9\}$. MM multi-MOORA

Fig. 3

Mean of $\rho$ for 10 criteria and $m = \{7,9\}$. MM multi-MOORA

Fig. 4

Mean of $\rho$ for 15 criteria and $m = 9$. MM multi-MOORA

Table 4 Mean of $\rho$ for MCDM problems with 5 criteria

The similarities among methods when considering the problem sizes are quite similar to those shown in the global analysis (Table 3). This fact can be checked seeing the order of the methods in the X axis of the plots in Figs. 2, 3 and 4. In addition, we can mention that the mean of $\rho$ is higher when the number of alternatives is increased.

A fact somehow “surprising” is the influence of the v value in the output of VIKOR method with respect to TOPSIS. When considering $v=1$ (VIKOR$^1$), the output of the methods are almost similar. However, when $v=0$ (VIKOR$^0$), the outputs produced are quite different. In other words, VIKOR shows a wide range of behaviors depending on a single parameter that makes it to behave or not like TOPSIS. To the best of our knowledge, there are no guidelines to set up such value when facing a new problem, so it is hard to imagine how such value should be defined.

It should also be highlighted that TOPSIS and multi-MOORA show a high mean value of $\rho$, stating that their outputs are quite similar.

3.4 Second comparison: a subset of methods over all the problems

In this subsection, we provide an all-against-all comparison of multi-MOORA, TOPSIS and VIKOR with $v=1$ (VIKOR$^1$) over all the test problems considered (1600 different decision matrices).

Table 5 shows the mean of $\rho$ for every pair of methods, and for all combinations of number of criteria and alternatives. The values are also shown as plots in Fig. 5.

Fig. 5

Mean of $\rho$ for all-against-all method’s comparison

Table 5 Mean of $\rho$ for all-against-all method’s comparison

Focusing first on the comparison of multi-MOORA vs. TOPSIS, we can observe a quite high level of agreement between the rankings. The less similar case achieved an average of $\rho = 0.81$ when $m=3, n=10$. As Fig. 5a shows, such similarity is independent of the number of alternatives and criteria.

The situation is different when multi-MOORA is compared against VIKOR$^1$. Now, the average $\rho$ ranges from 0.58 (when $m=3, n=10$) to 0.85 (when $m=9, n=5$). From Fig. 5b, it is clear that as the number of alternatives increased, the outputs of the methods became more similar.

The comparison of TOPSIS vs. VIKOR$^1$ led to a similar result. We should highlight the case where $m = 9$ and $n = 5$, with an average $\rho = 0.91$. This means that the outputs of both methods are almost the same over the 100 tested cases. Again, as Fig. 5c indicates, there is a clear tendency showing that a higher number of alternatives implies a higher rank similarity (independent of the number of criteria).

4 Conclusions

In this work, we performed a comparison of different MCDM methods over 1600 randomly generated decision problems to understand their similarities and differences in terms of the rankings they produced.

The MCDM methods compared were: multi-MOORA, TOPSIS and VIKOR, considering $v=0$, $v=0.5$ and $v=1$, i.e., considering the rankings produced by S, Q and R, respectively, in VIKOR.

From this comparison, two main points can be outlined: the first one is relative to VIKOR method in particular, while the second one is relative to multi-MOORA, TOPSIS and VIKOR with $v=1$.

The first observation we did concerns the VIKOR method in two aspects. Firstly, we point out that its output strongly depends on the parameter v. The rankings produced when using $v=\{0.5, 1\}$ are quite similar. However, the use of $v=0$ led to rankings that showed quite low similarities with those from the other methods considered.

Secondly, when using VIKOR with $v=\{0, 0.5\}$ we found that the method failed to produce a ranking for many decision problems. In the most extreme case, VIKOR could not solved any of the 100 problems with 3 alternatives and 20 criteria. We may assume that this is a quite unrealistic situation, but it also failed to provide a ranking in 44 problems (out of 100) with 9 alternatives and 20 criteria. The problem relies on the calculation of the Q index, which is a convex combination of two terms. One of them (R) produced a division by zero thus leading to a indetermination in the Q value. In such cases, VIKOR will fail for any $v \ne 1$. In other words, just one possible value for v is “safe”.

The second observation address the results of the comparison among multi-MOORA, TOPSIS and VIKOR$^1$. Multi-MOORA and TOPSIS obtained very similar results (their rankings are almost the same), being the lowest value of $\rho$ 0.81. A high level of similarity was obtained independently of the number of criteria/alternatives considered.

The similarity of multi-MOORA with VIKOR$^1$, and VIKOR$^1$ with TOPSIS is not as high, but it clearly increased as more alternatives are available. In both cases, the number of criteria showed a quite minor influence on the similarity values.

In short, our conclusions are in two lines: (1) if the problem will be solved using several methods, then the user should not choose simultaneously multi-MOORA and TOPSIS; and (2) VIKOR’s ranking is very sensitive to the parameter v, thus it should be carefully defined.