1 Introduction

In organizations, such as companies or local governments, which operate on the basis of a plan, do, check and action (PDCA) cycle, the check process is essential to perform their activities. Moreover, current developments in information technology enable them to evaluate their activities from various sides, and a lot of evaluation methods have been developed including data envelopment analysis (DEA) [1].

In the DEA, organizations are considered as decision-making units (DMUs) and their efficiencies are evaluated by relative comparisons. Specifically, DMUs are assumed to yield the same output items from the same input items, and their efficiencies can be evaluated by the ratio of the virtual input and virtual output values those are calculated by input and output values and their weights. Because each DMU can assign weights so that their own efficiency values are maximized, analysts can evaluate the DMU efficiencies on the basis of their features.

Further, several methods have been developed to evaluate the robustness of efficiency against the changes in input or output items [2, 3]. In several of these methods, the robustness is evaluated using the sensitivity analysis which compares two kinds of efficiency values. The first one is the efficiency value that is calculated on the basis of all input and output items, and the other one is calculated on the basis of the combination that specific input and output items are eliminated. Although, eliminated items are selected subjectively, the robustness considering all the input and output items cannot be evaluated quantitatively using conventional approach.

On the other hand, the hierarchical DEA model was developed to evaluate the efficiency structure of DMUs on the basis of the combinations of input and output items [4]. In this model, by calculating efficiency values based on all combinations of input and output items, analysts can figure out efficiency structures and the characteristic combinations.

Then, we propose a robustness degree for all the input and output items on the basis of the hierarchical DEA model. First, to calculate the proposed degree, the efficiency structure of input and output items is constructed and the characteristic combinations and efficiency values of them are revealed by the hierarchical DEA model. Second, we calculate the proposed robustness degree using the efficiency values, the number of input and output items, and one parameter.

This study introduces the conventional approach and its associated problem in Sect. 2. To overcome the problem, we discuss a hierarchical DEA model in Sect. 3 and a method to calculate a new robustness degree in Sect. 4. The utility of the proposed degree is shown through numerical experiments in Sect. 5 and we conclude our research in Sect. 6.

2 DEA sensitivity analysis

In this section, we introduce the DEA and the conventional sensitivity analysis.

2.1 DEA

The efficiency value of the kth DMU (DMUk) is calculated by the following linear program (LP) (1) [1]. In this formula, it is assumed that there are n DMUs and every DMU yields s output items from m input items. xij and yrj imply the ith input and rth output values of DMUj. vi and ur are weights assigned to them.

$$\begin{gathered} \hbox{max} {\text{ }}{\theta _k}=\sum\limits_{{r=1}}^{s} {{u_r}{y_{rk}}} \hfill \\ s{\text{.t}}{\text{. }}\sum\limits_{{i=1}}^{m} {{v_i}{x_{ik}}} =1 \hfill \\ {\text{ }} - \sum\limits_{{i=1}}^{m} {{v_i}{x_{ij}}} +\sum\limits_{{r=1}}^{s} {{u_r}{y_{rj}}} \geqslant 0{\text{ (}}j=1, \ldots ,n{\text{)}} \hfill \\ {\text{ }}{v_i} \geqslant 0{\text{ (}}i=1, \ldots ,m{\text{), }}{u_r} \geqslant 0{\text{ (}}r=1, \ldots ,s{\text{)}}. \hfill \\ \end{gathered}$$
(1)

In (1), the efficiency value is maximized under two constraints. In the first constraint, virtual input value is calculated as the sum of products of input values and their weights and it is fixed to 1. The second constraint implies that the virtual output value, calculated as the case of virtual input, is not more than the virtual input in all DMUs. The objective function is to maximize the virtual output value under these constraints. In other words, LP (1) implies that weights are assigned to each item so that the efficiency value of DMUk is maximized, and those of all DMUs are less than 1.

If the DMU is evaluated as “efficient”, the efficiency value is 1, otherwise, the DMU is evaluated as “inefficient” and the efficiency value is less than 1. As described above, analysts can evaluate the efficiencies considering features of DMUs and figure out the characteristic items that influence their efficiency values from calculated weights.

2.2 The conventional sensitivity analysis in DEA

Today, organizations such as companies are evaluated from many viewpoints with a lot of items. In these situations, the efficiencies of DMUs tend to be higher in DEA, as the DMUs tends to have characteristics in specific input and output items. Therefore, there are some problems that it is difficult to evaluate the differences in DMU efficiencies and the weights of a few specific items are positive despite those of the rest of the items are 0.

$$\begin{gathered} \hbox{max} {\text{ }}\theta _{k}^{{{I_1},{O_1}}}=\sum\limits_{{r \in {O_1}}}^{{}} {{u_r}{y_{rk}}} \hfill \\ s{\text{.t}}{\text{. }}\sum\limits_{{i \in {I_1}}}^{{}} {{v_i}{x_{ik}}=1} \hfill \\ {\text{ }} - \sum\limits_{{i \in {I_1}}}^{{}} {{v_i}{x_{ik}}} +\sum\limits_{{r \in {O_1}}}^{{}} {{u_r}{y_{rk}}} \geqslant 0{\text{ (}}j=1, \ldots ,n{\text{)}} \hfill \\ {\text{ }}{v_i} \geqslant 0{\text{ (}}i \in {I_1}{\text{), }}{u_r} \geqslant 0{\text{ (}}r \in {O_1}{\text{)}}. \hfill \\ \end{gathered}$$
(2)

To overcome these problems, a sensitivity analysis was developed [2, 3]. In this method, the robustness of efficiency was evaluated by comparing efficiency values based on all input and output items and those based on specific items. Specifically, the efficiency values were calculated by the LP (2) with input items I1 and output items O1.

$$\alpha _{k}^{{{I_1},{O_1}}}=\theta _{k}^{{{I_1},{O_1}}}/{\theta _k}.$$
(3)

Then, the robustness, \(\alpha _{k}^{{{I_1},{O_1}}}\), was evaluated by dividing the efficiency values calculated in (2) by (1) as shown in (3).

It can be evaluated that the closer the value of \(\alpha _{k}^{{{I_1},{O_1}}}\) is to 1, the higher the robustness is, and the closer it is to 0, the lower the robustness is. Although, as this value reflects robustness considering only some specific items, the robustness considering all input and output items cannot be evaluated quantitatively.

3 Hierarchical DEA model

To evaluate the input and output structures related to the efficiency of DMUs, a hierarchical DEA model was developed [4]. In this model, the hierarchical structure is constructed by combinations of input and output items and the efficiency values are calculated on the basis of their items. Moreover, analysts can evaluate the efficiency structure of inputs and outputs by integrating combinations that do not influence on the efficiency. An example of the hierarchical DEA model to one input and three output datasets is shown in Fig. 1. First, each node represents a combination of input and output items and a hierarchical structure is constructed on the basis of all combinations of input and output items so that the upper nodes include items of linked lower nodes. In DEA, as a dataset include at least one input and one output item, the number of nodes is 7 (3 nodes in the 1st layer, 3 nodes in the 2nd layer, and 1 node in the 3rd layer). Second, efficiency values are calculated on the basis of their input and output items. Because of the properties of DEA, the efficiency values of the upper nodes are not less than those of linked lower nodes.

Fig. 1
figure 1

A structure on the basis of all combinations

Full size image

In this structure, if the efficiency value of the upper node is equal to that of the lower node, the added items to the lower node do not influence the efficiency of the upper node, and the combination of the upper nodes can be evaluated as non-characteristic. Therefore, if the efficiency values of the upper and lower nodes are equal, the upper node is integrated into the linked lower ones as is shown in Fig. 2. By integrating the nodes from the upper layer on the basis of this rule, it is possible to evaluate the characteristic combinations of input and output items and their efficiency values.

Fig. 2
figure 2

A structure on the basis of the integrated combinations

Full size image

4 The degree of robustness based on the hierarchical DEA model

It is difficult to evaluate robustness on the basis of all input and output items using the conventional approaches explained in Sect. 2. In this paper, we propose the robustness degree of the efficiency of DMUs, which is based on all items using the hierarchical DEA model. First, it is assumed that the numbers of input items, output items, and DMUs are m, s, and n, respectively, and the input and output data are expressed as xi = (xi1,…, xin) (i = 1,…, m), yr = (yr1,…, yrn) (r = 1,…, s), X = (x1,…, xm), and Y= (y1,…, ys). Moreover, the DMUk’s efficiency value of a node that has α input items, xA, and β output items, yB, is expressed as follows:

$${\theta _k}\left( {{x^A},{y^B}} \right)~~\left( {{x^A} \subseteq X,~{y^B} \subseteq Y,{x^A},{y^B} \ne \emptyset } \right).$$
(4)

Then, we construct the hierarchical structure by combinations of input and output items, calculate the efficiency values of all combinations, and integrate them on the basis of their efficiency values. The node expressed by (4) is located at the α + β − 1th layer in the hierarchical structure.

Moreover, we define the efficiency values \({\varphi _k}\) in consideration of importance of integrated node as follows:

$${\varphi _k}\left( {{x^A},{y^B}} \right)=\left\{ {\begin{array}{*{20}{c}} {{\theta _k}\left( {{x^A},{y^B}} \right)}&{{\text{if}}~\left( {{x^A},{y^B}} \right)~{\text{is~not~integrated}}} \\ {p{\theta _k}\left( {{x^A},{y^B}} \right)~}&{{\text{if}}~\left( {{x^A},{y^B}} \right){\text{~is~integrated~to~others}}} \end{array}} \right..$$
(5)

If a node \(\left({x}^{A},{y}^{B}\right)\)is not integrated, \({\phi }_{k}\) is equal to the original efficiency value, and if this node is integrated, \({\phi }_{k}\) is equal to products of the original efficiency value and p. p is a parameter representing the importance of the integrated node. One can set the parameter p to reflect how important the efficiency values are based on the ratio of non-characteristic combinations to characteristic combinations. The larger the parameter p, the more non-characteristic combinations are emphasized. If p = 0, they are ignored from evaluation, and if p = 1, the efficiency values of all the nodes are considered equally, regardless of whether the node is integrated or not.

In general, the values of input and output items may change due to its environmental changes and it is important to guarantee its efficiency. In such a case, DMUs with many characteristic combinations using a lot of items may be able to guarantee the evaluation than DMUs that depends on a few efficiency items. That is, the former is more robust than the latter. Therefore, DMUs can be evaluated as robust in two situations. The first situation is when the DMU has a lot of characteristic combinations and their efficiency values are high. The other situation is when the decrease in the efficiency value is small if specific items are eliminated. Although it may be easy to evaluate visually how robust the DMU is from calculated efficiency values, it is important to evaluate it quantitatively to compare robustness among DMUs. By taking into account the above two situations, we calculate the degree of robustness as following formula:

$${\tau _k}=\frac{{\mathop \sum \nolimits_{{{y^B} \subseteq Y}}^{{}} \mathop \sum \nolimits_{{{x^A} \subseteq X}}^{{}} \left( {\alpha +\beta - 1} \right){\varphi _k}\left( {{x^A},{y^B}} \right)}}{{\mathop \sum \nolimits_{{{y^B} \subseteq Y}}^{{}} \mathop \sum \nolimits_{{{x^A} \subseteq X}}^{{}} {\varphi _k}\left( {{x^A},{y^B}} \right)}}.$$
(6)

The denominator of (6) is the sum of the products of the number of items and their efficiency values, and the numerator is the sum of the efficiency values. Therefore, the more combinations of characteristic items a DMU has, or the higher its efficiency values are, the higher this degree is, and analysts can evaluate the robustness of DMUs quantitatively by the proposed degree. Moreover, this degree can be considered a center of gravity regarding the efficiency and characteristic combinations of the items.

5 Numerical experiments

5.1 Sample dataset

We show the utility of the proposed degree with the sample dataset shown in Table 1, which was also used in the conventional hierarchical DEA research [4]. In this dataset, there are 35 DMUs that are assumed to yield five output items (y1–y5) from one input item (x1).

Table 1 Sample dataset and results of conventional DEA
Full size table

5.2 Results calculated by conventional approaches

First, we show results calculated using the conventional DEA and sensitivity analysis in the right three columns of Table 1. For the efficiency values calculated using formula (1), five DMUs (5, 7, 9, 15, and 33) were evaluated as efficient, and the others were evaluated as inefficient. The right two columns of Table 1 show the results of the sensitivity analysis. The left side is the efficiency value of x, y1, y2, and y3 (y4 and y5 are eliminated), and the right side is the sensitivity calculated by dividing the above efficiency values by “efficiency values” of all input and output items.

The sensitivity values of 15 DMUs were 1. In other words, their efficiency values did not decrease when y4 and y5 were eliminated despite the sensitivity values of the rest of the DMUs decreasing by 1–10%. Therefore, these 15 DMUs are more robust than the others from the viewpoint of y4 and y5. Although, as the robustness could change if the other items were eliminated, they could not evaluate robustness considering all input and output items from these values.

5.3 Results of the proposed degrees

As explained in Sect. 4, we constructed efficiency structures based on the combinations of input and output items, calculated efficiency values using the dataset associated with each node and reveal the characteristic combinations. We explain the proposed degree through DMU4 and DMU31. Their efficiency structures are shown in Figs. 3 and 4.

Fig. 3
figure 3

Efficiency structure of DMU4

Full size image
Fig. 4
figure 4

Efficiency structure of DMU31

Full size image

Their efficiency values calculated using the conventional DEA are 0.968 and 0.952, respectively, and they can be evaluated as having similar efficiencies. On the other hand, they are different in efficiency structures and it can be analyzed that DMU4 is more robust than DMU31 visually from Figs. 3 and 4. Then, from results of \(\theta _{{}}^{{({x_1},{y_1},{y_2},{y_3})}}\) and \(\alpha _{{}}^{{({x_1},{y_1},{y_2},{y_3})}}\) calculated by the conventional sensitivity analysis, DMU4 are 0.936 and DMU31 are 1.000. Therefore, although DMU31 is more robust than DMU4 in (x1, y1, y2, y3), it cannot be evaluated which DMU is more robust or how robust it is, considering all items.

Then, calculated robustness degrees are shown in Table 2. In this experiment, the parameter p was set to 0.1. Column τ represents the proposed degree and the rest are the efficiency values of each node. Meanwhile, the efficiency values θ (x1, y1, y2, y3, y4, y5) placed at the 5th layer is equal to those in Table 1. Bolded data in Table 2 imply the characteristic combinations of each DMU. As nodes in the 1st layer represent combinations of one input and one output items and they cannot be integrated with others, they are bolded in all the DMUs. From these results, their robustness degrees are different with DMU4 being 2.036 and DMU31 being 1.794 and it can be evaluated that DMU4 is more robust than DMU31 quantitatively.

Table 2 Results of proposed degree
Full size table

6 Conclusion

Although several methods for evaluating robustness have been developed, it has been difficult to analyze how robust the efficiency values calculated by DEA is to all input and output items. Therefore, we proposed a robustness degree using a hierarchical DEA model. First, a hierarchical structure was constructed using all combinations of input and output items, and the efficiency value was calculated for each node. Then, we formulated the degree representing robustness of efficiency, considering that as the number of characteristic combinations of input and output items are more, and the efficiency value of each combinations is higher, the more robustness DMUs are. Moreover, we showed the utility of the proposed degree through the results of numerical experiments.