Keywords

1 Introduction

Multiple-Criteria Decision Analysis (MCDA) – also known as Multi-Criteria Decision-Making (MCDM) – is an efficient and effective operations research tool to evaluate and solve decision-making problems that involve conflicting criteria providing a final ranking of the considered alternatives “from best to worst”. MCDA has seen significant interest over the past years, and it has been applied to a number of different areas, including sustainable design and manufacturing. Several MCDA methods have been proposed, each of them with peculiar strengths and weaknesses. However, although applying more than one MCDA method to a given problem is beneficial (because it can increase the confidence in the final decision making), one significant drawback is that conflicting rankings can be obtained.

This work describes the first steps in developing a structured, systematic framework able to reconcile conflicting rankings from various MCDA methods and return to the decision maker a unique final solution. A comprehensive literature review is performed to gain an understanding of the existing approaches followed by a critical analysis of relevant advantages and disadvantages. Finally, a first attempt to develop a new reconciliation method is outlined and applied considering five MCDA methods: TOPSIS, VIKOR, ELECTRE, MULTIMOORA and WASPAS. Input data from four case studies related to environmental sustainability choices are collected and applied to the five MCDA methods. Finally, the proposed ranking reconciliation method is applied to illustrate its use and validate its effectiveness.

2 Conflicting MCDA Ranking Reconciliation Approaches

A notable limitation of applying multiple MCDA methods is that they may return dissimilar results (i.e. rankings) for the same problem, and it is often not simple to identify one MCDA method as the most suitable for a given decision problem. MCDA techniques have their restrictions, hypotheses, theories and perspectives, which may lead to distinctive outcomes when applied to the same problem. Additionally, the choice of specific MCDA methods can be subjectively related to the decision maker preferences and assumptions, making challenging the analysis, evaluation and comparison of different methods [1]. An overview of existing approaches to reconcile conflicting rankings obtained by different MCDA methods follows.

One possible technique to solve the described problem is using Spearman’s correlation coefficient to measure the agreement between rankings. Specifically, if conflicting rankings are detected, Kou et al. [2] calculated Spearman’s rank correlation coefficient \(\rho \), first, between pairs of k-th and i-th MCDA methods

$$\begin{aligned} \rho _{ki} = 1- \frac{6 \sum _{j=1}^n{d_j^2}}{n(n^2-1)} \end{aligned}$$
(1)

where n is the number of alternatives and \(d_j\) is the difference in ranking of each j-th alternative for the ki pair of MCDA methods considered. Then, a coefficient for each k-th MCDA method (of the q in total) is obtained

$$\begin{aligned} \rho _{k} = \frac{1}{q-1} \sum _{i=1,i\ne k}^q \rho _{ki} \qquad k = 1,2,\dots ,q \end{aligned}$$
(2)

and it is used (suitably normalized) as weight to combine MCDA scores to return the final, unique ranking [2].

Alternatively, Kendall’s tau-b correlation coefficient can be used in a similar way. For example, Zamani-Sabzi et al. [3] developed an approach that contrasts the results of different MCDA methods highlighting their similarities and inconsistencies. The authors selected 10 MCDA techniques (including SAW, TOPSIS, AHP, ELECTRE, VIKOR) and statistically evaluated their conflicting rankings using Kendall’s tau-b correlation coefficients \(\tau _B\) [3]. In general, for each pair k-th and i-th of MCDA methods, \(\tau _{B,ki}\) is calculated in the following way

$$\begin{aligned} \tau _{B,ki} = \frac{n_c-n_d}{\sqrt{(n_0-n_k)(n_0-n_i)}} \end{aligned}$$
(3)

where \(n_c\) and \(n_d\) are the number of concordant and discordant pairs between methods k and i. Terms \(n_0\), \(n_k\) and \(n_i\) are calculated as follows

$$\begin{aligned} n_0=\frac{n(n-1)}{2} \quad n_k=\sum _j \frac{t_j(t_j-1)}{2} \quad n_i=\sum _l \frac{u_l(u_l-1)}{2} \end{aligned}$$
(4)

where n is the number of alternatives, whereas \(n_k\) and \(n_i\) take into account of potential ties in the ranking of the respective k-th and i-th MCDA method. Specifically, \(n_k\) is constructed counting how many groups of ties are present in the ranking and, for each j-th group, measures the number of tied alternatives in that group \(t_j\). Similarly, \(n_i\) is calculated on the basis of the number of tied alternatives \(u_l\) for each l-th group of ties in the ranking of the i-th MCDA method. Then, \(\tau _B\) coefficients can be used by the decision maker to appreciate the magnitude of correlation between methods and select the simplest method that produces reasonably similar ranks to the others. Alternatively, it is possible to calculate a weight coefficient for each MCDA method substituting \(\tau _{B,k}\) to \(\rho _k\) into (2).

Mohammadi and Rezaei propose a method to minimize the distance between each ranking using half quadratic theory to approximate the overall aggregate ranking \(R^*\) [4]. The principle behind this method is to assign weights proportional to the difference of rankings and are, therefore, objectively set. Such weights are then used to calculate the final, output ranking [4]. The minimization function is

$$\begin{aligned} \min _{R^*} \frac{1}{2} \sum _{k=1}^q || R_k - R^* ||^2 \end{aligned}$$
(5)

with q the number of MCDA methods considered and \(R_k\) the ranking of the k-th MCDA method. The arithmetic mean of the rankings is the closed-form solution of the minimization problem of \(R^*\) that has been found [4]:

$$\begin{aligned} R^* = \frac{1}{q} \sum _{k=1}^q R_k \end{aligned}$$
(6)

More reliability is achieved introducing two indicators: the consensus index and trust level. The consensus index is an indicator that shows the agreements between MCDA methods, while the trust level reports the reliability of the resulting final ranking. If there is an agreement between the methods, the consensus index and trust level are equal to 1/q which means that their values are both high. If there is a disagreement between all the ranks, then their values are low [4].

A fourth method that can be considered is the Fusion approach [2]. In this case the weights from different MCDA methods are determined by seeking a compromising ranking that has a minimized solution between the overall rankings and the optimal result [2]. To calculate the relevant weights \(w_k\), the rankings of q MCDA methods \(R_k\) are combined to obtain the compromise ranking \(R^*\)

$$\begin{aligned} R^* = \sum _{k=1}^q w_k\,R_k \end{aligned}$$
(7)

Then, it is selected among the available rankings \(R_k\) the one with the minimum difference from the optimal \(R^*\) calculated [2].

A fifth method of consideration is using the Gini Index which is one of the primary metrics to measure statistical heterogeneity. Selmi et al. [5] propose a comparative study to identify the similarities and differences between MCDA methods to reconcile conflicting rankings. The Gini index D used to measure the ranking dispersion is defined as:

$$\begin{aligned} D = \frac{4}{(n-1)(q^2-|\sin {\frac{q\pi }{2}|})} \sum _{k=1}^{q-1} \sum _{i=k+1}^q |R_k-R_i| \end{aligned}$$
(8)

where n is for the number of alternatives, \(R_k\) and \(R_i\) are the position in the ranking according to methods k and i, whereas q is the number of MCDA methods. Equation (8) provides a normalised value between zero (same rankings) and one (ranking highly dependent on the MCDA method). D can be calculated for each alternative to assess the stability of the ranking across all the methods considered. However, such values are intended only to support the decision maker that is, then, left to decide a compromised ranking, without a structured procedure to follow [5]. Furthermore, D could be also evaluated between pairs of MCDA methods to appreciate similarities between them for the problem at hand [5].

A final method of consideration is at the basis of the SIMUS software, as proxy method [6]. The creator of SIMUS claims that there is always a risk of disagreement between various MCDA methods caused by their inherent procedural subjectivity linked (for example) to criteria weighting [7]. The proposed solution by SIMUS is based on linear programming that leads to rankings independent from the normalization method and removes subjectivity from the decision making process [7].

Considering pros and cons of the described methods (Table 1), there is no ideal solution to reconcile conflicting MCDA rankings in the literature. Therefore, a new approach is proposed and implemented in this work as a first step towards a complete method able to reconcile multiple, conflicting rankings with minimal shortcomings.

Table 1. Advantages and limitations of existing approaches to reconcile conflicting rankings by multiple MCDA methods found in the scientific literature
Full size table

3 A Novel Reconciliation Method

An approach based on the mode of the position in the rankings for each alternative is proposed and it is based on the principle that the preferable solution is the one with the widest agreement among the MCDA methods. Furthermore, this approach implies that no MCDA method is better than the others and allows to scale the approach to any desired number of MCDA methods automatically. One restriction to the proposed approach in the current form is that the number of MCDA methods to be considered must be odd to avoid ambiguities in the final ranking obtained.

The approach will be illustrated and tested with four case studies pertaining sustainability and extracted from the scientific literature.

4 Results and Discussion

To illustrate and test the effectiveness of the proposed approach, four case studies in the field of sustainability were collected from the scientific literature. More specifically, for each case study, data comprising the typical decision making matrix were extracted: i.e. alternatives, criteria and their measure of performance along with an indication of their beneficial or non-beneficial impact (i.e. a greater value is better or worse). Rankings were calculated according to five well-established MCDA methods: TOPSIS [8, 9], VIKOR [10], MULTIMOORA [11], WASPAS [12], and ELECTRE [13]. The cases were evaluated using the R computer language [14] package MCDM [15].

Case study 1 compares competing plans to build a sewer network. The criteria considered are the dynamic performance in terms of flow volume, the cost of construction to implement the plan, the cost of maintenance for inspection or repairing, the environmental impact and the potential future profit [16]. It can be seen that, in this case, TOPSIS appears to return a significantly different ranking in comparison to the other MCDA methods (Table 2). Consequently, in this case, TOPSIS has minimal influence on the final ranking.

Table 2. Case study 1 alternative rankings.
Full size table

Case study 2 compares ten different designs of support structures for offshore wind turbines. Ten criteria are considered: compliance or maximum displacement of the rotor, dynamic performance, design redundancy, cost of maintenance, cost of installation, environmental impact of the installation, operation and decommissioning, carbon footprint, certification, likely cost of the concept and depth compatibility [17]. In this case there seem to be a lack of agreement between methods with the mode of the position in the ranking showing low frequency (Table 3). In particular, a frequency of two is observed for the alternatives “Jacket” and “Barge”.

Table 3. Case study 2 alternative rankings.
Full size table

Case study 3 compares thirteen renewable energy investment projects considering the following criteria: power, investment ratio, useful life, cost, operating hours, implementation period and the amount of carbon dioxide emissions [18]. It can be observed (Table 4) that there is a good agreement between WASPAS and MULTIMOORA for most of the alternatives, unlike TOPSIS and VIKOR that return dissimilar rankings. Furthermore, this case shows that, when the number of alternatives increases, the agreement between the rankings of MCDA methods usually decreases.

Table 4. Case study 3 alternative rankings.
Full size table

Case study 4 is a material selection investigation to design car bodies with environmentally friendly processes [19]. Ten alternative materials were evaluated in terms of physical properties, durability, suitability to manufacturing processes (called “technical”) and environmental properties. In agreement with previous cases, MULTIMOORA and WASPAS return similar rankings influencing with their agreement the final output (Table 5).

Table 5. Case study 4 alternative rankings.
Full size table

As previously briefly mentioned, a factor that appears to cause a significant difference between rankings is the number of alternatives n. In case studies 3 and 4 (that are characterized by a relatively large n) there is a low frequency of agreed positions in the rankings among MCDA methods. ELECTRE, for example, is particularly sensitive to the value of n because of its estimation of the concordance and discordance matrices [13]. This consideration seems to be reinforced observing that cases with fewer alternatives, like case study 1, show good agreement among methods. However, a broader validation with more cases would be beneficial before drawing a final conclusion.

Since the approach proposed in this work is the result of multiple MCDAs, it provides the decision maker with more confidence in the results, increasing the reliability and robustness of consequent choices. Moreover, another reason that may make a multi-method ranking preferable, is the consideration that, in general, no MCDA method can be considered superior.

On the other hand, an obvious drawback to the adoption of the proposed approach is the additional time required. However, the calculation process of the output ranking has been designed to be quite simple to minimize additional time and effort asked to the decision maker. Another related consideration regards the specific MCDA methods to be included in the set of evaluations. For instance, some methods not presented in this work may require additional work or more resources from the decision makers. For example, the Analytic Hierarchy Process (AHP) requires demanding pairwise comparisons of criteria performed by experts that implies significant additional effort [20]. Furthermore, cases with low consensus among MCDA methods can be still valuable because they signal to the decision maker that any rankings for the specific case at hand do not have a solution that can be trusted.

5 Conclusion and Future Work

Different Multiple Criteria Decision Analysis (MCDA) methods are known, in some cases, to return conflicting rankings when applied to the same problem. Since each MCDA method is unique in its way with specific characteristics, the validity of their answers can be questioned by this potential inconsistency. Deep research and analysis of existing approaches to solve this issue were presented in this work and, subsequently, a novel reconciliation approach has been presented. The method is simple and based on the mode of the position in the rankings for each alternative. Four case studies in the field of sustainability were selected and analysed to illustrate and test the applicability of the proposed approach. Based on the results, the new ranking system appears promising and represents an important step into the definition of a framework to reconcile conflicting rankings of multiple MCDA methods.

Areas of further development lie in the handling of special or corner cases (e.g. with methods like AHP or PROMETHEE) and including sensitivity studies to corroborate the robustness of the final ranking. AHP or PROMETHEE introduce different type of thresholds and scaling that may affect the weights and the final ranking of alternatives in a different way compared to the methods considered in this work. Sensitivity can be measured, for example, to test the robustness of the approach against uncertainties in interpretative values or weights. Further validation with more case studies would be also valuable.