Multi-Criteria Decision Analysis Under Uncertainty: Two Approaches to Incorporating Data Uncertainty into Water, Sanitation and Hygiene Planning

Abstract

In the era of the Sustainable Development Goals, for which one of the aims is to provide universal access to safe water, sanitation and hygiene (WASH) services, it is crucial to target and prioritize those who remain unserved. Multi-criteria decision analysis (MCDA) models can play an important role in WASH planning by supporting priority-setting and policy-making. However, in order to avoid misleading assumptions and policy decisions, data uncertainty — intrinsic to the available collection methods — must be integrated into the decision analysis process. In this paper, we present two approaches to incorporating data uncertainty into MCDA models (MAUT and ELECTRE-III). We use WASH planning in rural Kenya as a case study to illustrate and compare the two approaches. The comparison focuses on the way these two models handle uncertainty in the available data. The analysis shows that, while both methods incorporate data uncertainty in a considerably different manner, they lead to similar prioritization settings.

1 Introduction

Achieving universal access to safe water, sanitation and hygiene (WASH) services by 2030 is a huge endeavour for countries worldwide (UN-Water 2018). Targets 6.1 and 6.2 of the Sustainable Development Goals challenge governments to tackle the ‘unfinished business’ of extending WASH services to those who remain unserved, as well as to progressively improve the level of services provided. The progressive realization of universal access to WASH and the reduction of inequalities in service levels is also consistent with the United Nations resolution on the human rights to water and sanitation (United Nations 2010). However, the commitment to ‘leave no one behind’ requires increased targeting and prioritization of those most in need of better WASH services. As the UN Special Rapporteur on the human right to safe drinking water and sanitation (2011) explains, governments must give “priority to realizing a basic level of service for everyone before improving service levels for those already served”.

Among other things, this requires WASH planning tools that target the neediest and support equity-oriented prioritization (Giné-Garriga et al. 2015). Evidence-based targeting and prioritization procedures not only allow the identification of the segments and sectors of population in which to focus policies, but also guide a more equitable allocation of resources. Yet, decision-making processes in the WASH sector often lack transparency and accountability, and can lead to discrimination against certain population groups (Ibid.). A step forward to support targeting and prioritization is thus the establishment of appropriate decision-making tools that assist policymakers and implementers in revealing which population groups are the most in need of further WASH services.

Multi-criteria decision analysis (MCDA) models can play an important role in informing WASH planning. MCDA is a quantitative decision analysis model that evaluates and compares alternative decision options (e.g. communities or administrative sub-units) in terms of their services on a set of criteria (e.g. service coverage, service levels, etc.). By ranking population groups against multiple planning criteria, MCDA models can provide insight on priority-setting and development of WASH interventions. A wide variety of MCDA models currently exist, which can be grouped into two main approaches (Ishizaka and Nemery 2013): (i) value measurement models (or ‘American school’), based on the construction of a numerical score for each alternative (e.g. Multi-Attribute Utility Theory, MAUT), and (ii) outranking models (or ‘European school’), based on the pairwise comparison between the alternatives (e.g. ELimination and Choice Expressing REality, ELECTRE).

The differences between the two MCDA families are substantial. First, there is no underlying utility function in outranking models: the output is a ranking of alternatives without any scores to indicate the extent to which one alternative is preferred over another. Second, the set of decision rules describing the aggregation procedure in outranking models are only partially compensatory, which limits the trade-offs between the different criteria (Stewart and Losa 2003). Yet, despite these considerable differences, only a few studies have attempted to compare these methods or their decision analysis procedures. Table 1 summarizes the most relevant studies in water resources management that address these comparisons.

Table 1 Summary of studies in water resources management comparing different MCDA models

What has not been done before is extending MCDA models to the context where the data feeding the analysis have a certain level of uncertainty. Data uncertainty — the degree to which data are inaccurate, imprecise or unknown — arises from various factors, such measurement errors, data staleness and repeated measurements (Tsang et al. 2011). In the WASH sector, data uncertainty is intrinsic to the available collection methods. Household surveys represent a crucial source of data and have developed into standardized sampling techniques and harmonized questionnaire designs to produce comparable estimates across countries and over time (WHO and UNICEF 2006). However, data from household surveys are not extent from uncertainty. All survey point estimates have a certain level of error, regardless of the size or design of the sample. This is particularly important in decentralized contexts with small populations (e.g. fewer than 500 households), where the high level of disaggregation makes it indispensable to balance the precision of survey data against survey costs (Pérez-Foguet and Giné-Garriga 2018). In using the household survey data for WASH planning, policymakers and implementers must consider the underlying uncertainty in order to avoid making decisions based on false or misleading assumptions (Giné-Garriga et al. 2013c).

Against this background, we present two MCDA approaches, based on MAUT and ELECTRE-III, for integrating data uncertainty into the decision analysis process. Our aim is guided by three main research questions:

• 1. How can we adapt MAUT and ELECTRE-III models for incorporating the uncertainty of the input data during preference modelling?

• 2. In which manner can we characterize the uncertainty of the input data and quantify its effect on the resulting model’s output?

• 3. How convergent or divergent are the results (i.e. rankings) of each model?

The contributions of this paper are twofold. First, to the best of our knowledge, this work is the first attempt to extend MAUT and ELECTRE-III models to tackle data uncertainty in water decision-making. Second, the paper addresses the growing need in WASH sector for improved targeting and prioritization instruments. Although our motivation comes from the WASH sector, the two approaches we present can also be applied in other areas of water management to address the issue of numerical inaccuracy in the data.

The remainder of the paper is structured as follows. We present an overview of the MAUT and ELECTRE-III mtehods in Section 2 and describe a case study of WASH planning in rural Kenya in Section 3. In Section 4, we present and discuss our proposed MCDA models for incorporating data uncertainty (4.1), characterizing uncertainty and treating propagation of uncertainties (4.2) and the comparison between rankings (4.3). Finally, we present our conclusions in Section 5.

2 MCDA Methods

Both MCDA methods begin by defining the decision problem, compromising:

• A set of m alternatives, A: A = {a₁,a₂,...,a_i,...,a_m}

• A set of n criteria, C: C = {c₁,c₂,...,c_j,...,c_n}

• A set of n weights coefficient for the criteria, W: W = {w₁,w₂,...,w_j,...,w_n}

• The evaluation matrix, G, with the performance values of each alternative a_i on criterion c_j in row i and column j: G[i,j] = g_j(a_i)

The first of the two models is derived from the multi-attribute utility theory. The model considers two steps:

• Aggregation: a utility function is defined to construct the global value of each alternative. Several possible functions (additive, multiplicative and multi-linear) can be used. For simplicity reasons, we restricted our attention here to the additive form: the utility value for each alternative, U(a_i), is calculated as the sum of the weighted performance values for each criterion.

• Exploitation: the utility values obtained in the first step are used to rank the alternatives.

The second model is based on ELECTRE-III (Roy et al. 1992). It also consists of two steps:

• Outranking relation: the method starts by building a preference relation, known as ‘outranking relation’ S(a₁,a₂), between each pair of alternatives. To do so, a series of pairwise comparisons of the alternatives are made using the concordance-discordance principles.

• Exploitation: the outranking relations obtained in the first step are used to build two pre-orders through descending and ascending distillations, Z₁ and Z₂. A final pre-order of the alternatives is suggested as the intersection of these two.

The construction of the concordance and discordance indexes for each pair of alternatives requires the definition of three discrimination thresholds for each criterion: indifference threshold (q_j), preference threshold (p_j) and veto threshold (v_j). However, choosing these thresholds values can be, however, challenging for decision-makers, as it involves a high degree of subjectivity (Ezbakhe and Pérez-Foguet 2018). Figure 1 illustrates a summary of the decision analysis procedures of both MCDA models.

Fig. 1

General decision analysis procedures of the MAUT (left) and ELECTRE-III (right) models

3 Case Study

In rural Kenya, a large proportion of the population lacks access to safe WASH services. According to published national official statistics, only half of the people living in rural areas used improved sources of drinking water and less than 20% have access to safe sanitation and hygiene facilities (Kenya National Bureau of Statistics 2010). In order to increase the access to appropriate WASH services, the Kenyan Government (in collaboration with UNICEF) launched an initiative in 2010 to target the most vulnerable rural populations. This case study focuses on these rural areas, located in 21 districts across the country (Fig. 2).

Fig. 2

Map of Kenya with WASH Program districts (adapted from Giné Garriga and Pérez Foguet 2013a)

The 2010 initiative included a survey that reached 4,925 households across the 21 targeted districts. In each household, service level was captured through a structured questionnaire covering multiple WASH-related issues. Issues included: (i) quality of the water source, (ii) type of main drinking water source, (iii) distance from dwelling to the water source, (iv) functionality of water supply in the household, (v) person responsible for dwelling water, (vi) domestic water consumption, (vii) type of sanitation facilities, (viii) sanitary inspection of water supplies, and (iv) point-of-use water treatment. The standards (i.e. the minimum levels to be attained in the provision of WASH services) are shown in Table 2.

Table 2 WASH issues considered in the case study

Each household was given a value of 0 or 1 depending on whether it met the standard (1) or not (0). This provided the number of households (x_i) meeting the required standard. The proportion of households that met the standards (p_i) was estimated for each district as x_i/n, with n the total number of households sampled in each district. The survey data are shown in Table 3. These data constituted the performance values for our MCDA models. The alternatives in the decision problem were the rural communities in the 21 districts in rural Kenya, using the criteria the ones shown in Table 2.

Table 3 Results of the household survey (with n sample size, x_i the number of households meeting the standard, and p_i the estimated proportion of population meeting the standard)

In the MAUT model, the weights of criteria were determined by principal analysis component (PCA) following the methodology developed by Nardo et al. (2005). This method has already been used in different WASH-related indices (Giné Garriga and Pérez Foguet 2010, 2013b; Pérez-Foguet and Giné-Garriga 2011). It is important to draw attention to the fact that, while weights in MAUT represent the relative importance of criteria, weights in ELECTRE-III express the decision-makers’ deliberate position regarding the ‘voting power’ of each criterion (Figueira et al. 2010). Consequently, a study involving the decision makers of the WASH sector in Kenya would be necessary to assess their positions on the different criteria. Without access to these decision makers, it was necessary to translate the weights obtained in the MAUT model into indices of importance for ELECTRE-III. In this case, we assigned the same set of weights for both models (Table 4).

Table 4 Criteria weights used in both MCDA models (obtained from PCA)

4 Results and Discussion

4.1 Incorporating Data Uncertainty into the MCDA Models

In order to integrate the uncertainty of the input data into the preference modelling process, we adapted the MAUT and ELECTRE-III methodological frameworks as follows.

Model U, based on the MAUT theory, starts by building the global utility value of each district U(a_i). To estimate the uncertainty associated to this utility value, the model combines the uncertainty components of the performance values for each criterion g_j(a_i) through an ‘uncertainty propagation’ method. This allows us to have the districts’ utility values together with their uncertainties (i.e. the probability distribution of the utility values). Finally, the model conducts a statistical hypothesis test (in this case, a Welch’s t-test) between each pair of districts to determine their statistical significance. If the null hypothesis of no differences in the utility value means is accepted, the districts are considered to occupy the same ranking position; otherwise, one district ranks higher than the other.

Model S, derived from ELECTRE-III, incorporates data uncertainty in a different manner. Uncertainty of input data is characterized and used to define the discrimination thresholds, according to Eqs. 1–3.

$$\begin{array}{@{}rcl@{}} &&q_{j} = \max{|g_{j}(a)_{U} - g_{j}(a)| , |g_{j}(b)_{L} - g_{j}(b)|} \end{array} $$

(1)

$$\begin{array}{@{}rcl@{}} &&{\kern5pt} p_{j} = |g_{j}(a)_{U} - g_{j}(a)| + |g_{j}(b)_{L} - g_{j}(b)| \end{array} $$

(2)

$$\begin{array}{@{}rcl@{}} &&{\kern68pt} v_{j} = k \cdot p_{j} \end{array} $$

(3)

with g_j(a) the performance values of alternative a on criterion j, g_j(a)_U and g_j(a)_L the upper and lower limits of its confidence interval, and k the veto/preference ratio (k = v_j/p_j). In this case, we adapted a ratio of k = 2.

The concept behind these equations is simple: if the performance values and their associated uncertainties overlap, it is reasonable to consider them indifferent (indifference threshold q_j). Otherwise, if there is no overlap, one alternative may be preferred over the other (preference threshold p_j). Once the discrimination thresholds are calculated, the model follows ELECTRE-III outranking procedure to obtain the final ranking of districts.

Figure 3 highlights the different ways models U and S integrate data uncertainty. Model S is more straightforward, as data uncertainty is directly included through the discrimination thresholds. In contrast, model U requires more steps to propagate uncertainty and conduct hypothesis testing before obtaining the final ranking.

Fig. 3

Incorporating data uncertainty into MCDA models. The model U is based on the MAUT theory, and model S, on ELECTRE-III.

4.2 Characterizing and Propagating Uncertainty

Uncertainty of input data can be characterized using various methods, both in terms of qualitative and quantitative parameters. In this case, our input data are proportions of populations in each district, estimated from the household surveys. Consequently, data can be considered to follow binomial probability distribution, with parameters n as the number of households surveyed and p, the proportion of households verifying the criteria (note that we assumed sample sizes n are much smaller than the population size N). To characterize the uncertainty in our data (populations estimates), we used confidence intervals. According to Clopper and Pearson (1934) ‘exact’ method, the lower and upper limits of the confidence intervals can be expressed as:

$$\begin{array}{@{}rcl@{}} p_{i_{L}} = \left[ 1+ \frac{n-x_{i}+ 1}{x_{i} \cdot F_{1-\alpha/2, \ 2x_{i}, \ 2(n-x_{i}+ 1)}} \right]^{-1} \end{array} $$

(4)

$$\begin{array}{@{}rcl@{}} p_{i_{U}} = \left[ 1+ \frac{n-x_{i}}{(x_{i}+ 1) \cdot F_{\alpha/2, \ 2(x_{i}+ 1), \ 2(n-x_{i})}} \right]^{-1} \end{array} $$

(5)

with F(c,df₁,df₂) the 1 − c quantile from the F distribution with degrees of freedom df₁ and df₂. Although other methods for calculating the binomial proportion confidence intervals exist, we chose the Clopper-Pearson interval as it was based on the cumulative probabilities of the binomial distribution rather than an approximation to the normal distribution (Agresti and Coull 1998). The confidence intervals are shown in Fig. 4.

Fig. 4

Confidence of intervals of population estimates. (Note that districts are ordered in descending order for each criterion)

In model S, these confidence intervals are used to define the indifference, preference and veto thresholds according to Eqs. 1–3. This provides an easy manner of integrating the data uncertainty in the ranking process.

However, model U requires an uncertainty propagation step in order to determine the uncertainty in the global utility values. We tested two error propagation approaches: (i) first order, second moment approximation (FOSM), and (ii) Monte Carlo simulation (MCS). The first approach uses a Taylor series expansion of the random variable, while the second approach generates artificial samples of input random variables in order to evaluate the distribution of the simulated utility value. Both approaches give almost the same confidence intervals of the global utility values. However, while the FOSM approach only estimates the mean and standard deviation of the utility value, the MCS approach provides its full probability distribution. Thus, although it takes a relatively longer time to be completed, we use the MCS approach for hypothesis testing and district ranking.

4.3 Comparison of Rankings

The two MCDA models resulted in similar district rankings (Fig. 5). This convergence between the rankings coincides with results of other studies (Duckstein et al. 1982; Roy and Bouyssou 1986; Goicoechea et al. 1992; Mahmoud and Garcia 2000), whereby rankings obtained by weighted average and ELECTRE-III methods were largely the same.

Fig. 5

Rankings of the 21 districts obtained with models U and S

In both cases, districts of Molo (A3), Kisumu (A12), Nyando (A16) and Uasin Gishu (A19) occupied the leading positions. A closer look at the survey data (Table 3) reveals why these four districts had better WASH services than the rest. For instance, in terms of water supply (c₄), more than 95% of their populations had access to functioning water points; this is 8% higher than the national average. national average. The same is observed with respect to the distance from dwelling to water (c₃): while on average only 40% of the population had access to a water source within 30 minutes or less. The proportion in these four districts was at least 12% higher. In addition, more than 71% of households owned latrines in good hygienic conditions (c₈), far from the national average of 53%. On the other hand, both models placed districts of Garissa (A5), Isiolo (A6), Mwingi (A18) and Mandera (A20) as the lowest rankings. These four districts severely lacked adequate quantities of water for domestic purposes (c₆): only 21%-39% of their populations had access to more than 20 liters of water per capita per day, which is 30% lower than the national average. Furthermore, although access to improved sanitation services was 46% on average (c₇), it was less than 29% in these districts.

The only major divergence between the two models is the position of districts Tana River (A7) and Kajiado (A15): the model U ranked Kajiado higher than Tana River, while the opposite was observed for model S. This reflects the different principles underlying the two models, especially concerning the compensatory nature of their aggregation procedures. The Kajiado district had better services in terms of distance to source, functionality of water supplies, domestic water consumption, household water quality and water treatment (c₃, c₄, c₆, c₈ and c₉), but performed poorly in criteria related to improved water sources and persons responsible for collecting water (c₂ and c₅). Model U, being fully compensatory, placed Kajiado in a higher position as the bad performances in the two criteria are compensated by the rest. In contrast, model S (which is only partially compensatory) limited this compensation, resulting in a lower position for Kajiado district.

Nonetheless, both models lead to similar targeting and prioritization (Fig. 6). These prioritization maps can help understand the inequalities in access to WASH services. In Kenya, there is a serious gap in WASH services in the North Eastern Province, which should thus be targeted in future WASH investments. In the context of limited budgets, this type of targeting and prioritization tools become essential for designing interventions that seek to reduce inequalities in service provision. However, it is particularly important to highlight that, even if the two MCDA models resulted in different rankings — and hence dissimilar prioritization maps —, both are equally relevant and valid. More important than the selection of which model to apply for WASH planning is to fully understand the mathematical model and principles behind it.

Fig. 6

Colour (red/orange/green) depicts prioritization of districts based on their ranking (whereby red is the lowest ranked and green, the highest)

5 Conclusions

Safe WASH services are central to meeting global development goals on health, poverty and economic growth. However, strengthening the role of WASH in poverty alleviation requires evidence-based targeting and prioritization instruments in order to identify and focus on those most in need for better WASH services. In this sense, MCDA can provide insight on priority-setting and development of WASH interventions, but the task of choosing of the most appropriate model can still be challenging. This selection is even more difficult when dealing with uncertainty in the input data, as there is a lack of studies extending MCDA models to integrate data uncertainty.

In this paper, we present and compare two MCDA models, based on MAUT and ELECTRE-III, for targeting and prioritization of WASH services. Unlike other comparisons in the literature, we adapted the MCDA methodological frameworks to address the uncertainty of the input data.

Our main conclusions from this comparison are:

• The two models incorporate uncertainty in the input data in a considerably different manner. Model U, based on MAUT, requires a step of ‘uncertainty propagation’ in order to characterize the uncertainty of global utility values, as well as a step of ‘hypothesis testing’ to determine the ranking of alternatives. Model S, based on ELECTRE-III, presents a more straight-forward ranking procedure, as data uncertainty is directly included through the discrimination thresholds.

• In the WASH sector, household estimates used for targeting and prioritization purposes are inferred from representative samples from the overall population. Therefore, it is important to characterize the precision of the estimated values. A simple way to express uncertainty in the estimates, and its effect on the MCDA models output, is through confidence intervals.

• Both models can be useful decision-aid instruments for targeting and prioritization in the WASH sector. In this case study, the two models yield similar rankings and lead to similar prioritization. However, it is noteworthy to remember that MCDA models should not be used to reveal the ‘correct’ prioritization, but to guide the decision analysis process. While the selection of the MCDA model is important, more emphasis should be given on both defining the decision problem comprehensively and understanding the theoretical principles underlying each technique.