Introduction

Integrated water resource management (IWRM) is a worldwide accepted policy objective. In India, the National Institution for Transforming India (NITI Aayog), the Ministry of Department of Land Reforms, and the Ministry of Water Resources have emphasized the Integrated watershed management programme (IWMP) through stakeholders’ participation to achieve sustainable economic growth [1]. Despite the high acceptance of IWRM and IWMP principles, the variables for performance evaluation of IWMP projects are relatively amorphous as the project goals are specific to the regional hydrological and social characteristics. This makes IWRM operations exceedingly challenging to comprehend. Subsequently, the NITI Aayog has developed the Composite water management index (CWMI), incorporating nine themes and 28 critical indices for different areas [2]. CWMI report provides an annual snapshot of the water sector status and the water management performance of the other states and UTs in India. CWMI establishes a clear baseline and benchmark for state-level performance on the water indicators, encouraging states and non-profit organizations to highlight participatory governance practices [3]. Accordingly, watershed projects strive to utilize and control water and allied resources in the project area through community-based organizations’ (CBO) participation [4]. Therefore, it is imperative to evaluate the performance of individual watershed projects through a participatory mode.

Researchers have outlined key success performance indicators [5,6,7,8]. Mostly, these indicators are generic and prescribed with universal connotations [7, 9]. In this context, more studies need to be done on the performance assessment of each watershed project in a localized context. In addition, existing watershed evaluation protocols mainly focus on acquiring data from estimated assets or expenditures made for a top-down appraisal where the perspective of the beneficiaries is not taken care of. Commonly, watershed project baseline study, benchmarking, and monitoring and evaluation are entrusted to centralized agencies. The monitoring procedures mainly assessed the structural components of an IWMP project using a top-down method. The monitoring system has also been upgraded using the Geographical Information System (GIS) and Management Information System (MIS) to investigate watershed performance. They seem to have focused more on technical assessment than socio-economic impact assessment [10,11,12]. Contextually, it is necessary to assess the IWMP performance from the perspective of the beneficiaries. There is an absence of a comprehensive framework for a performance assessment that can be used in participatory mode [13]. Therefore, one objective of this study is to construct a performance indexing framework for watershed projects and to validate existing project areas.

Discussing watershed effectiveness, Kenney observed that watershed initiatives are successful if they contribute to achieving on-the-ground objectives defined by prevailing social norms. Discussing 36 watershed studies, he also said that 60% of respondents listed ‘participation by stakeholders’ and about 25% of respondents listed ‘problems with on-the-ground project goals’ as key to success [14]. With that perspective, it is crucial to explore and quantitatively evaluate the community’s expectations, prioritization, and alignment regarding project goals and realized achievements to design an adaptable framework for bridging the gap between watershed performance and planned importance.

Here are some pertinent questions: How can the relevant watershed performance variables be identified, and how can the disparity between performance and planned importance be gauged?

To answer these questions, this study aims to design a participatory performance evaluation framework (PEF) for indexing watershed project performance supported by field-level community information. The research objectives are (a) Mapping the planned importances or targets of selected watershed projects to use them as the variables of the PEF, (b) Introducing a methodology to quantify community perceptions about the project achievements, and (c) Validation of the PEF in selected projects in Brahmaputra Valley, Assam to obtain their performance index (PI).

This study applies a survey in three IWMP projects in the Brahmaputra Valley, Assam, to explore the critical variables (goals) for evaluating watershed performance and examining the community perception of those variables to obtain a performance measuring index. The framework, being applicable in a participatory manner, has the potential to enhance community participation and efficiency in project implementation. Since the watershed programs in Assam are congruent with the IWMP protocols initiated by the Government of India [2, 15], the developed methodology is valuable for watershed projects with analogous socio-economic environments elsewhere in Assam and India. This paper comprises five sections: (i) Introduction, (ii) Methods, (iii) Results and Discussion, and (iv) Conclusion.

Methods

The research methodology adopted includes (i) a literature review and content analysis of watershed project documents to classify the most common watershed developmental goals as PEF variables, (ii) semi-structured focused group interviews of experts for coding goal categories, (iii) inter-coder reliability testing to identify project goals categories, (iv) validating PEF with a questionnaire survey in the study areas and statistical analysis.

Content Analysis for PEF Variables

A comprehensive literature review was conducted to list sustainable watershed projects’ most common characteristics and goals. Guidelines of DoLER [16, 17], NITI Aayog [18], SLNA objectives [19], and detailed project documents of different projects in Assam [15, 20] are reviewed. After cross-checking the validity of the obtained goals list, a thorough qualitative analysis was done to determine the activities planned in the selected watershed projects. The primary project goal categories and the most common IWMP developmental goals are extracted from project reports and field studies [1, 15, 16, 19,20,21,22].

Subsequently, a list of performance variables comprising 75 relevant variables in 8 categories is prepared. Then, focus group discussions were held with three IWMP project stakeholders to finalize the list. The FGD members arrived at a list of 65 variables in 8 categories of goals to be included in the PEF. For validating the goal categories, five watershed experts are invited to code the listed goals and Krippendorff’s inter-coder reliability test (Kalpha reliability) was executed on their observations [23]. (Table 1).

Table 1 The primary project goal categories and the most common IWMP developmental goals
Full size table

Survey Areas

Three IWMP projects were selected based on three primary considerations: (i) covering three different regions with analogous project protocols and goals prescribed by SLNA [15, 20, 22], (ii) covering both banks of the Brahmaputra River with similarity of socio-economic scenarios, and (iii) cooperative attitude and response of the local community. (Fig. 1, Table 2).

Fig. 1
figure 1

(Source: Detailed Project Report of Satpokholi, Turkunijan and Kaldia IWMP)

Maps of the study areas.

Full size image
Table 2 Watershed projects selected for study
Full size table

Survey Methods

Pilot interviews were conducted with five watershed experts to examine the local practice of assessing the performance of IWMP projects and to validate the compiled list of PEF variables. They consented to the identified variables. Subsequently, a structured close-ended questionnaire protocol with a 4-point ratio scale (‘0 for not started’, ‘1 for started’, ‘2 for partially completed’, and ‘3 for completed’) is done amongst the randomly selected project beneficiaries. A ratio scale is quantitative with true zero and equal intervals between neighboring points [24].

Data Analysis

Goal-Wise, Category-Wise Performance Score (CPS) and PI Calculation

The score of each goal is calculated as given below.

Let, in any goal category ‘A’ having n numbers of goals, the score of the i-th goal is denoted by GAi, where i = 1, 2, 3…. n. For each goal, respondents gave a mark between 0 and 3. Now, if the number of respondents giving 0, 1, 2, and 3 scores are R0, R1, R2, and R3, respectively, then,

$${\text{Gai }} = \, \left( {\text{Marks given to that goal}} \right)/\left( {\text{Maximum mark for that goal}} \right)$$
$$= \, \left( {{3}*{\text{R}}_{{3}} + { 2}*{\text{R}}_{{2}} + {1}*{\text{R}}_{{1}} + 0*{\text{R}}_{0} } \right)/ \, \left( {{\text{R}}_{{3}} + {\text{R}}_{{2}} + {\text{R}}_{{1}} + {\text{R}}_{0} } \right) \, *{3}$$
(1)

Then, Category-wise Performance Score of category A (CPSA) would be the summation of marks given to all the goals in that category, i.e.

$${\text{CPS}}_{{\text{A}}} = \, \sum {\text{Gai}}$$
(2)

The different measurements of the evaluation criteria should be reduced to a standardized scale to be manipulated to treat them as standard measures. Heathcote suggested standardization and imposing weights for effective evaluation criteria for fully utilizing available community information and priorities, considering differences between alternatives to make a clear-cut final choice [25].

The scores are grouped to impose weights for PI calculation (Table 3).

Table 3 Score levels and weights imposed for PI calculation
Full size table

The maximum performance score (MPS) of a project = Total Number of goals in the project*Maximum weight. Since we have 65 goals in each project, MPS = 65*10 = 650.

$${\text{The performance index }}\left( {{\text{PI}}} \right) \, = \, \left( {{\text{TPS}}/{\text{ MPS}}} \right) \, \%$$
(3)

The maximum PI of a watershed project can be 100%. For the projects under study, individual PI is calculated.

Breakup of Opinions and Percentage of Zero Score (PZS)

This study has used four score levels denoted by Si (Here, i = 0, 1, 2, and 3). The breakup of opinions is calculated as follows:

Percentage of zero scores (Si) = 100*Number of responses with ith score / Total Response. Accordingly, the percentage of zero scores (S0), percentage of one score (S1), percentage of two scores (S2), and percentage of three scores (S3) are obtained. For a good project, S3 should be high, and S0 should be low.

The percentage of zero scores (PZS denoted by S0) seems valuable as the zero scores are marking and do not indicate abstention. The higher the PZS, the lower the performance. For detailed investigation, the PZS can be calculated goal-wise, category-wise, and project-wise. Results derived from the survey were cross-referenced amongst community members and watershed managers for validation.

Results and Discussion

The Krippendorff’s Alpha for inter-coder reliability testing is 0.8, which indicates the acceptability of the goal category list.

Performance Scores and Index

The goal-wise performance score of the three IWMP projects is calculated (Appendix 1: Goal-wise Performance Scores). The goal-wise and category-wise performance scores (Fig. 2) reveal stakeholders’ opinions on performance. It would tell watershed managers how each goal category is performing.

Fig. 2
figure 2

Category-wise performance score of All IWMPs

Full size image

Of all the projects, the goal category “Social-capital Development’ has the highest performance (about 27–33%). This is obvious as users’ demand for different goals in this category is high. The category “Socio-economic upgradation” has the second highest performance in all the projects (about 16–24%). The three low-performing categories are “Naturalisation and Afforestation,” “Water Resources Management,” and “Associated Infrastructure Development.” One reason may be that the goals under these categories involve higher costs than the other categories.

Then, project-wise performance and PI are calculated (Table 4).

Table 4 The project-wise PI
Full size table

From the definition (see Sect. "Goal-wise, Category-wise Performance Score (CPS) and PI Calculation"), the maximum PI of a watershed project can be 100%. However, the IWMP projects studied have very low PIs. The Turkunijan IWMP has scored the maximum PI value of 29.54%. Out of 65 planned targets, a maximum of 22 targets have only a 20–30% score. The Kaldia IWMP has a PI value of 18.15% and a maximum of 30 targets with a 0–10% score. Here, 62 targets out of a total of 65 targets obtained scores below 30%. The PI of Satpokholi IWMP is very low (10.62%), and all 65 targets scored below 20%.

Thus, the developed performance assessment framework has given us a clear picture of the functioning of the watershed projects studied to find out significant performance parameters (Table 5).

Table 5 A few significant performance parameters
Full size table

Policymakers might use the category-wise performance score to compare, remodel, and manage best practices. ‘Social Capital Development’ is the highest-performing category. This is reasonable because this category includes building community organizations, the primary objective of IWMP. In Satpokholi and Turkunijan IWMP, ‘Associated Infrastructure Development’ is the minor scoring category. This category is also anticipated because it is not a primary IWMP objective.

Percentage of Responses at Different Score Levels

The breakup of opinions against different score levels (Fig. 3) is shown below.

Fig. 3
figure 3

Breakup of opinions with different scores Level-All IWMPs

Full size image

No person has given a maximum mark (3) to any goal in all three projects. This shows that project performances are far from the maximum. In Kaldia and Satpokholi, the maximum number of people gave zero scores to the goals (S0 values are the highest, 63 and 89%, respectively), which shows their dissatisfaction with performances. In Turkunijan, S1 and S2 are the highest, and S0 is the lowest compared to the other two projects. It reflects better performance in Turkunijan compared to Satpokholi and Kaldia.

Similarly, the category-wise PZS (S0) values (Fig. 4) show stakeholders’ opinions on non-performance in different goal categories.

Fig. 4
figure 4

Category-wise percentage of zero score for All IWMPs

Full size image

From the PZS values on goal categories, it is seen that the category “A” (Naturalisation and Afforestation) has performed better (PZS being below 7% in all projects). The performance of categories “C” (Soil and Land Management) and “H” (Associated Infrastructure and Development) is also of the same level, except in Turkunijan IWMP, where the public expectation might be higher than the achieved. The categories “G” (Social capital Development) and “E” (Agricultural Production) have shown higher non-performance (with the PZS values ranging from 11 to 31%). Looking at this result, project managers may go for a detailed analysis of the performance score of the pertaining goals (already obtained by the applied methodology) to detect the bottlenecks (as described in Sect. "Performance Scores and Index").

Project-wise, Kaldia has two lowest performing categories, “G” and “E” (PZS 26.33 and 23.01% respectively). Here, category “H” and category “A” show better performance by PZS (2.28 and 5.61% respectively). In Satpokholi, category “G” is the worst performer, with the highest PZS of 23.71%. Here, category “H” is the highest performer (PZS = 3.09%). In Turkunijan, the worst performer is the category “E” (PZS = 31.45%). Here, the two top-performing categories are “C” (PZS = 5.67%) and “A” (PZS = 6.75%).

Some high and low-performing categories by PZS analysis are shown below (Table 6).

Table 6 Some high and low-performing categories by PZS analysis
Full size table

This concurs with the fact that Turkunijan is a recently completed project, and work on Satpokholi is delayed due to field-related reasons. Here, all 65 goals scored below 20%. In the Kaldia IWMP (PI 18.15%), a maximum of 30 goals has a 0–10% score. Validators agreed that works in Kaldia are different from the plan. According to validators, the PEF would facilitate baseline study and benchmarking of IWMP projects as desired by NITI Aayog (Table 7).

Table 7 Goal-wise performance score for Kaldia IWMP
Full size table

Conclusion

This study aimed to design a participatory performance evaluation framework (PEF) for bridging the gap between achievement and planning goals in a watershed project. Per the research objective, a framework (methodology) is developed and validated in IWMP projects with similar protocols to show how participatory performance indexing is feasible to obtain a quantified performance score (Table 8). The framework variables are identified by analyzing the goals of the three selected projects with analogous settings operating in Brahmaputra Valley, Assam. Using the framework, the performance indices (PI) of the Kaldia, Satpokholi, and Turkunijan IWMP projects are obtained to be 18.15, 10.62, and 29.54, respectively. Thus, the developed performance assessment framework presents a quantitative expression of the performances of the watershed projects studied. Hence, the developed methodology fulfills two necessities of watershed managers: (i) Expressing performance by a quantitative value and (ii) incorporating community perceptions for assessing project outcomes. Usually, watershed programs are assessed by accounting for the number of assets created or fund flow to the project in a top-down procedure. However, from the point of watershed effectiveness, performance evaluation should focus on the parity of the project goals and the accomplishment against them. Moreover, since watershed initiatives are community-based collaborative programs, meaningful evaluation procedures should incorporate the community’s perspective. Expressing performance quantitatively would help watershed policy-makers to dynamically calibrate and compare the effectiveness of different projects in the same regional settings (Table 9).

Table 8 Goal-wise performance score for Satpokholi IWMP
Full size table
Table 9 Goal-wise performance score for Turkunijan IWMP
Full size table

Besides providing project-wise performance scores and indexing, the developed framework has the potential to provide very essential community perceptions about various goal categories.

Although the NITI Aayog [2, 18] has proposed watershed guidelines with broadly relevant indicators to facilitate participatory practices in watershed management, there still needs to be a formulation for ground-level evaluation incorporating community perspectives. Contextually, the framework developed in this study would be helpful for watershed project managers.

Presently, the developed methodology is applied to accommodate the perception of the watershed beneficiary group. Similarly, this methodology can also be used to capture the perceptions of different stakeholder groups.

The variables for the framework are selected with a broader view over a region to incorporate the most common and universal variables. Therefore, the framework and goal categories can be used for performance indexing of various projects in an analogous socio-economic setting with similar IWMP protocols. However, separate survey for data collection would be necessary for individual projects.

In evaluating performance, this study has refrained from setting any threshold or goal-post values. Therefore, the obtained results cannot be used for inter-project comparison. However, policymakers can arbitrarily choose such values for similar watershed projects to make the framework a performance-comparing tool. A series of in-depth case studies on various watershed projects should be launched to verify further and enrich the applicability and reliability of this PEF to enhance watershed effectiveness.