Agro-environmental sustainability assessment using multicriteria decision analysis and system analysis

Abstract

Transparency and reproducibility remain challenges for sustainability assessment, particularly in developing world contexts where formal scientific information is often limited. We posit that even in such contexts, sustainability assessment can be productive and informative if the underlying assumptions about sustainability are made transparent. Thus, the process of assessment can be as instructive as the results, if not more so. In this article, we describe and discuss how we combined multicriteria decision analysis and system analysis as a unified approach to sustainability assessment. This approach is transparent, practical, flexible, and reproducible; it also facilitates the development of recommendations for enhancing sustainability. We illustrate the approach with examples from a recent environmental sustainability assessment of irrigated commercial maize production in Sinaloa, Mexico.

Introduction

Indicator-based sustainability assessments have been widely adopted to aid in decision-making processes of planning and policy: they help inform decision makers, managers, and the public about the current state of a system, and measure progress toward or away from a goal (Parris and Kates 2003; Pope et al. 2004). Although there may be “no recipes” in sustainability (Wals and Jickling 2002), producing an assessment that is transparent and reproducible remains a challenge (Binder et al. 2010; Pischke and Cashmore 2006). By transparent, we mean that to be able to understand and use the results, one should also understand the quality of the data, the contributions of different participants to the process, the assumptions and uncertainties, as well as how and why each decision was made and by whom. By reproducible, we mean that progress toward sustainability may be subsequently monitored with relative ease. Ideally, some flexibility for the assessment inputs should be possible—given that any system in question and data availability is likely to change over time—without altering the fundamental structure of the assessment, or undermining the ability to compare the assessment inputs over time (Reed et al. 2005).

Transparency and reproducibility are especially important in contexts where formal data and resources for monitoring and assessment may be limited. For example, much of the growth in agricultural productivity is expected to occur in the developing world (Foley et al. 2011) where data and resources for assessment and monitoring often are limited. In these contexts, the decision process of the assessment is as important as the results. Nevertheless, to date the literature on agricultural sustainability assessment has not widely addressed or documented the decision process (Binder et al. 2010; Bosshard 2000; Gasparatos et al. 2008).

In this article, we describe a methodological approach to sustainability assessment that responds to the challenges of transparency and reproducibility: the combination of multicriteria decision analysis (MCDA) (Lahdelma et al. 2000; Lootsma 1999; Triantaphyllou 2000) and system analysis (Scott 2000). MCDA helps to rigorously identify context-specific decision criteria through stakeholder engagement. System analysis is a useful method for illuminating the relationships of system variables and their influence in the system. These methods are well established and have been used in sustainability assessment, particularly for environmental management (e.g., Chung and Lee 2009; Liu 2007; Mendoza and Prabhu 2005) and technology assessments (e.g., Linkov and Seager 2011; Wiek et al. 2008), although less often in agricultural contexts. We illustrate this combination of methods with examples from an environmental sustainability assessment of irrigated commercial maize production in Sinaloa, Mexico. Our purpose here is to present the assessment process and the methods; the full results of the assessment of maize production in Sinaloa are documented separately (Bausch 2011a).

In the discussion, we follow Binder and colleagues’ (2010) framework to analyze the normative, systemic, and procedural dimensions of our approach to agricultural sustainability assessment. Aspects of the normative dimension of a sustainability assessment include (1) the underlying sustainability concept, or the problem to be assessed, (2) the sustainability goals for the assessment, and (3) assessment type, or how the sustainable state of indicators is defined (Binder et al. 2010). The systemic dimension of sustainability assessment refers to the representation of the “main structures, processes, and functions of the economic, ecological and social fields of the system studied” (Binder et al. 2010, p. 74). For the procedural dimension, or methods of sustainability assessment, Binder and colleagues (2010) call for methods that are reproducible, comprehensive, transparent, applicable, and involve stakeholders. They identify 3 main tradeoffs typical of sustainability assessment procedure: “(1) Benchmarking vs. system specific analysis; (2) ‘easy’ and ‘fast’ assessment vs. regional specific and applicable results; and (3) easy, clear understandable message (aggregation) vs. system based trade-off analysis” (p. 79). We discuss how these dimensions and tradeoffs are evident in our approach.

Background of methods

MCDA

MCDA is a well-established branch of decision theory that rigorously identifies context-specific decision criteria through stakeholder engagement. Typically, problems with multiple criteria do not have a single best solution, which is why in MCDA, decision makers’ preferences become part of the solution process (Lootsma 1999). MCDA is inherently transdisciplinary, as it requires input from decision makers, stakeholders and researchers with systemic, as well as disciplinary, perspectives. By definition, sustainability challenges involve multiple criteria, as they include multiple scales, domains, stressors, tradeoffs, perspectives, and stakeholders (Kates et al. 2001).

System analysis

There are a wide variety of approaches to and applications of system analysis in sustainability research (e.g., Checkland and Poulter 2006; Meadow 2008; Mitchell 2009). In this study, our approach to system analysis focused on the identification of relationships among system variables, as well as the influence that each system variable had on the other variables. In some literature, such applications are referred to as network analysis (i.e., Mitchell 2009; Scott 2000).

Assessment context and objectives

We applied the above methods, described in greater detail in the sections that follow, to an environmental sustainability assessment of agriculture in the Mexican state of Sinaloa. Sinaloa has a long history of large-scale, irrigated agriculture (Ortega Noriega 1999), and is noted as one of the greatest successes of the Green Revolution (Wright 2005). The state stretches along the northwest Pacific coast of Mexico. The coast is an important ecological asset with high species diversity (Carvalho et al. 1996; Páez-Osuna et al. 2007; Rubio Rocha and Beltrán Magallanes 2003) that supports the economic activities of fishing and tourism (Cruz-Torres 2004; Gobierno del Estado de Sinaloa 2010; Trujillo Félix and Gaxiola Carrasco 2010). Today, 25 % of land in Sinaloa is dedicated to agriculture (Gobierno del Estado de Sinaloa 2010), located primarily on the coastal plains. Most commercial agriculture in the state is irrigated with surface water channeled from 12 river dams. The agricultural sector represents 15 % of Sinaloa’s GDP (Gobierno del Estado de Sinaloa 2009). There are approximately 152,000 farmers in the state, representing 5.6 % of Sinaloa’s population (Gobierno del Estado de Sinaloa 2010).

Farmers in this state did not historically produce maize on a commercial scale. Prior to 1990, Sinaloa’s dominant commercial crops in terms of area planted were sesame, safflower, and a rotation of soy and wheat. Starting in 1990, maize production in Sinaloa expanded rapidly, from 141,000 ha planted in 1989 to a peak of 607,000 ha in 2008 (Servicio de Información Agroalimentaria y Pesquera (SIAP) 2010). During the winter growing season, maize is now a monoculture in the northern part of the state (Eakin et al. 2014).

While there had been previous research on the economic and social dimensions of the dramatic expansion of maize production in Sinaloa (Aguilar Soto 2000, 2004, 2007; Maya Ambía and Ponce Conti 2010; Trujillo Félix and López Cervantes 2007), as well as the agronomic dimension (Díaz Valdés 2006; Díaz Valdés et al. 2008; Ojeda-Bustamante et al. 2006), little attention had been paid to the environmental dimension of this crop (de Ita Rubio 2003; Nadal 1999). The objectives of our assessment were thus: (1) to address a lacuna in the literature on environmental aspects of commercial maize production in the state of Sinaloa, (2) to develop an assessment approach that could later be expanded to incorporate social and economic dimensions of sustainability; and (3) to produce an assessment that could be useful to stakeholders.

Assessment process and results

Guiding normative principles

The concept of agricultural sustainability is still being formed, thus there are diverse perspectives, objectives, methods, and evaluative criteria for its assessment (Binder et al. 2010; Gasparatos et al. 2008; Hansen 1996; Rigby and Caceres 1997; Thompson 2007). Given this diversity, we selected normative principles appropriate to our objectives and the Sinaloa context to guide our selection of system variables, and serve as the evaluative criteria by which we assessed the sustainability of the indicators. To identify the guiding normative principles, we (the assessment team) reviewed the literature on environmental sustainability of agriculture, and agriculture in Sinaloa. The guiding principles we chose were (1) the long-term provisioning of basic human food needs; and (2) the enhancement of environmental quality and natural resources (American Society of Agronomy 1989).

Stakeholder engagement strategy

Part 1. In July 2010, we conducted semi-structured interviews with 26 stakeholders, including farmers, agronomists, researchers, government employees, environmentalists, fertilizer distributors, and representatives from farmers’ associations, all of whom had a stake in Sinaloa’s environment and/or maize production. To identify stakeholders, we looked at the websites of farmer associations, government agencies, and universities, and as well as published scientific literature. We also used a respondent-driven sampling strategy, in which we interviewed experts and practitioners who then referred us to other experts and practitioners. The objectives of these interviews were to improve our understanding of Sinaloa’s maize system, and to consult stakeholders about assessment inputs. We prepared a general interview outline to ensure that the topics of interest were introduced during the conversation. We consulted with stakeholders about assessment inputs such as system variables, indicator selection, and data availability. During these interviews, we administered a questionnaire (“Appendix A”) to systematically capture feedback from stakeholders (see “Selecting system variables” and “System variable weights” below).

Part 2. Once we had the initial results of the assessment, we organized two workshops in Culiacán, Sinaloa in July 2011. The objectives of the workshops were to (1) share our methods, results, and recommendations with stakeholders; and (2) receive critical feedback from stakeholders on our representation and analysis of the maize agro-environmental system in Sinaloa, as well as our recommendations for enhancing sustainability. We invited the participants from part 1 of our engagement strategy, but attendance was open to any interested stakeholders. Fifteen stakeholders attended the first workshop, representing farmers, government agencies for agriculture, growers associations, university students, university professors, input companies, and rural finance agencies. At the second workshop, 22 stakeholders attended, representing agronomists/agricultural consultants, and government agencies for water and agriculture. Between the two workshops, there were 37 participants. We also made the presentation slides and the assessment report available online (Bausch 2011b).

Selecting system variables

The first step in the system analysis was to select variables to represent the current agro-environmental system of maize production in Sinaloa. We define a system variable as a feature of a system that is critical to the system’s sustainability. Each variable could be measured in multiple ways. For example, potential measures or indicators of water quality include pH, dissolved solids, and biological oxygen demand, which are all very different from each other. According to our study’s objective and guiding normative principles described above, we focused our system analysis on indicators of agro-environmental sustainability within the commercial irrigated maize sector in the state.

To select the system variables, we first did an extensive review of agricultural sustainability assessments to identify which variables and indicators have been used to represent and measure environmental sustainability in agricultural systems. We also reviewed existing literature on agriculture in Sinaloa, and maize production in particular to ensure that the variables we selected were relevant to the case. We sought to represent multiple scales, and both active and passive variables (drivers and outcomes) to aid in the system analysis.

We selected 12 system variables (see Table 1): erosion, soil quality, water quality, pest and disease incidence, maize yield, terrestrial ecosystem, aquatic ecosystem, nitrogen fertilizer use, irrigation use, fossil energy use, pesticide use, and agricultural land use. We represented multiple scales by including some farm level indicators (e.g., soil quality, fertilizer use), and regional indicators (e.g., agricultural land use, terrestrial ecosystem). Examples of passive variables include water quality and terrestrial ecosystem health; active variables include agricultural land use and pesticide use, for example. For more information about how we defined the system variables, see Bausch (2011a).

Table 1 System variables and variable indicators

To verify that the variables we selected were representative of the Sinaloa maize system, we consulted with stakeholders during part 1 of our stakeholder engagement strategy. We administered a questionnaire (in Spanish) during interviews and at a meeting of agronomists for a total of 41 responses. On the questionnaire, we asked stakeholders whether the 12 variables we selected were appropriate for the case. The questionnaire also asked stakeholders to identify indicators for the variables. We provided space on the questionnaire (“Appendix A”) for comments and/or recommendations on what additional variables should be included in the assessment, not limited to the environmental dimension of maize production (see “Appendix E” in Bausch 2011a). The stakeholders confirmed our selection of system variables.¹

MCDA: current state analysis and assessment

We analyzed and assessed the current state of the system with MCDA (Lahdelma et al. 2000; Lootsma 1999; Triantaphyllou 2000). MCDA describes sustainability (S) mathematically as a relationship between system variable weights (importance to sustainability in a particular system) and the current state of the variable indicators. We assessed the current state of the variables in terms of sustainability using the technique known as the distance to the ideal point. The “ideal point” is an abstract condition possessing the most desirable state of each indicator (Lootsma 1999; Szidarovsky et al. 1986). This approach entailed three steps (described in detail below): (1) defining system variable weights; (2) defining the distance of the current state of each variable indicator from the ideal point; and (3) aggregating the weights and distances to derive a sustainability score for the system. The final result is a measure of the current state of each variable indicator in terms of sustainability, as well as a measure of the current state of the system as a whole in terms of sustainability.

System variable weights

During part 1 of our stakeholder engagement strategy, using the same questionnaire with which we asked stakeholders to verify the system variables, we asked stakeholders to draw on their own experience and knowledge to numerically rank the variables from 1 to 12 by their importance to the sustainability of maize production in Sinaloa.

Because the ordinal scale cannot be used for mathematical operations, for each questionnaire response, we transformed the ordinal scale to weights (w _ij ) in a scale with ratio properties, using the rank-order centroid method (Noh and Lee 2003):

$$w_{ij} = \frac{1}{n}\sum\limits_{k = i}^{n} \frac{1}{k}$$

(1)

where i is the index of variables, j is the index of stakeholders, n is the number of variables, and k is the rank of the sustainability variable assigned by the stakeholder.

After calculating the weights for each questionnaire response, we aggregated the weights by taking the geometric mean of the weights for each variable, then normalizing the geometric means so that the sum of the weights of all 12 variables was equal to 1.²

We compared the results of the weighting across stakeholder groups (maize growers, agronomists, researchers, government employees, environmentalists, and growers association representatives) to identify points of convergence and divergence. We found that the results were relatively consistent in terms of how the groups ranked what was most and least important in the system. The groups consistently ranked soil quality and irrigation as the most important components of sustainability, and yield as the least important (see Fig. 3). The exception was the group of maize growers, who ranked irrigation low, and yield much higher than the other groups (Bausch 2011a).

Characterizing variables and indicators

In MCDA, the standard approach to characterizing the system variables involves four steps: (1) selecting indicator(s) for each variable, (2) identifying the current state of each indicator, (3) identifying the ideal and anti-ideal state for each indicator, and (4) defining the value function for each indicator.

• (1) Selecting variable indicators. An indicator is a measurable proxy for a variable or concept that is difficult to monitor directly (Rigby et al. 2001). It must measure the variable of interest within a systemic context (Van Cauwenbergh et al. 2007), and be theoretically and contextually appropriate (Rigby et al. 2001). It is important to consider what an indicator communicates about the sustainability of the variable it is intended to measure, what role it plays in the system, and what it suggests about how to improve sustainability. To measure and assess the current state of each system variable, we selected indicators for each variable based on stakeholder feedback during part 1 of our engagement strategy (the interviews and questionnaire), as well as data availability (Table 1). All indicators were intended to reflect the state of the commercial maize system and the environment in which it was embedded. The spatial scale of the variables ranged from the farm scale (e.g., average pesticides used/hectare/year) to the state scale (e.g.,  % maize area planted but not harvested), depending on the variable itself and data availability.

• (2) Identifying current states. We identified the current states for each indicator using primary and secondary data sources. A farm survey, implemented by a research team associated with the authors, provided data for three of the indicators.³ We relied primarily on secondary data sources and stakeholder knowledge for the remaining data inputs. Secondary sources included data from government agencies and academic literature focused on the region. For two variables (irrigation, soil quality), we could not access formal data to indicate the current state. In these cases, we asked 3 local experts to estimate the current state and ideal state. In part 2 of our stakeholder engagement strategy, we consulted with the larger group of stakeholders about whether these data and estimates seemed accurate to their knowledge, and if needed, revised them accordingly (see “Cross-checking results and recommendations” below).

• (3) Identifying ideal and anti-ideal states. The sustainable or ideal state of an indicator can be derived by comparing it to another system, identifying a threshold, target value, or range of values, applying a legal or regulatory value, or a combination of these options (Binder et al. 2010; Van Cauwenbergh et al. 2007). We derived the ideal and anti-ideal state (i.e., undesirable state) of each variable indicator from the literature, best management practices for Sinaloa, and/or expert opinion, as well as the guiding normative sustainability principles. As our assessment was designed to be practical and relevant to stakeholders, we identified ideal states that are realistically achievable within 5 years, given the current state and local cultural and environmental context. In part 2 of our stakeholder engagement strategy, we asked the stakeholders to verify that the ideal and anti-ideal states were appropriate for the Sinaloa context, and revised them where needed.

• (4) Defining value functions. After identifying the current state, ideal state, and anti-ideal state of each variable indicator, we defined the distance of the current state of each indicator from the sustainable state in a value function graph (Beinat 1997) (“Appendix B”). A value function normalizes the value of an indicator in a natural scale to a dimensionless scale from 0 (anti-ideal condition) to 1 (ideal condition) that represents the current state of the indicator in terms of sustainability. In each value function, the abscissa represented the units of the indicator in their natural scale, and the ordinate represented the current state of the indicator as related to sustainability in the dimensionless scale. The dimensionless scale made it possible to compare the current state of all the indicators in terms of sustainability within the same scale.

To identify the appropriate value function for each variable indicator, we consulted academic literature on agroecosystem dynamics in which thresholds associated with agroecosystem change and/or sustainable outcomes had been identified. To identify the value functions that could not be found in the literature, we presented generic value functions to experts who identified the appropriate curve according to their knowledge of the indicator’s behavior in relation to sustainability. We developed the value functions in a Microsoft Excel (2008) file and adjusted them to best fit the expected behavior of each of the variable indicators in relation to sustainability in the Sinaloa context.

To help communicate the results of the current state analysis, we coded the current state value of each indicator in the dimensionless scale according to its distance from sustainability as far, close, or very close to a sustainable state. We categorized a value between 0 and 0.49 as far from a sustainable state. A value between 0.5 and 0.75 is close to a sustainable state. A value of 0.76–1 is very close to a sustainable state. This categorization follows the Weber–Fechner’s Law of psychophysics, which uses knowledge of the psychological response of people to variation of a physical stimulus to define the appropriate data ranges to correspond to peoples’ understanding of the concepts of ‘far’ and ‘close’ (Lootsma 1999). During part 2 of our stakeholder engagement strategy, we verified with stakeholders that the current state analysis of each variable indicator and of the aggregate system seemed accurate to their knowledge.

Example: irrigation

To illustrate this process, consider the indicator for the variable irrigation: net centimeters of irrigation water use per hectare per year (net cm/ha/year). Irrigation has a positive relationship with maize production in Sinaloa, in that evapotranspiration of maize plants exceeds annual precipitation, making irrigation necessary for achieving commercial yields (Ojeda-Bustamante et al. 2006). So, at the farm scale, the value function for the sustainability of irrigation use is an increasing convex curve (Overman and Sholtz 2002). However, at the regional scale there are tradeoffs and opportunity costs for irrigation efficiency (e.g., provisioning more water to ecosystem functions, or having enough water to plant in both winter and spring seasons, vs. using most/all available water in one season), meaning the sustainability of irrigating maize decreases past the threshold of 65 cm/ha/year, the estimated ideal state (expert opinion). The value function representing irrigation water use in maize production at the regional scale is thus bell shaped, reflecting the diminishing sustainability of excess irrigation use. The estimated current state of the indicator is 105 cm (expert opinion), which is 0.48 in the dimensionless scale (Fig. 1). The current state is therefore far from a sustainable state.

Fig. 1

Value function for the indicator for irrigation (annual water use per hectare)

Aggregation

We combined the variable weights ($w_{i}^{{\prime }}$) with the value of each indicator in the dimensionless scale ($x_{i}^{{\prime }}$) to determine the aggregate agro-environmental sustainability score of the Sinaloa maize system (S) (Lootsma 1999):

$$S = \sum\limits_{i = 1}^{n} {w_{i}^{{\prime }} x_{i}^{{\prime }} }$$

(2)

The aggregate sustainability score for the Sinaloa maize system was 0.43, or far from a sustainable state (see Fig. 3).

System analysis

We used system analysis to identify and visualize relationships and influence among the 12 system variables. We developed an impact matrix (Table 2) and analyzed it with the software UCINET (Borgatti et al. 2002). In the impact matrix, each indicator is listed along both the rows and columns. We identified whether each row item influences each column item using a binary measure of relations: 1 represents direct influence, while 0 represents no influence. The assessment of influence was done qualitatively by the authors, in consultation with experts and scientific literature on agroecosystem processes. We used Freeman’s out-degree centrality to measure the direct influence of each variable. We used Freeman betweenness centrality to measure indirect influence (Table 3). We then generated a centrality diagram, a type of system graph, with NETDRAW (Fig. 2) (Borgatti 2002). This analysis illustrated which variables were most influential in terms of both direct influence (out-degree centrality) and indirect influence (betweenness centrality). For example, in our case, the variable agricultural land use (defined as area planted in maize) was most directly and indirectly influential for agro-environmental sustainability, followed by input use (pesticides and irrigation) in terms of direct influence, and yield and pesticides in terms of indirect influence.

Table 2 Impact matrix

Table 3 Direct influence (Freeman’s out-degree centrality) and indirect influence (Freeman betweenness centrality) of agro-environmental system variables

Fig. 2

Centrality map of direct influence (Freeman’s out-degree centrality) among environmental system variables for Sinaloa maize production. The size and location of each circle is indicative of direct influence, i.e., the larger the circle and closer to the center of the map it is, the more direct influence it has on other system variables. Prepared using the software NETDRAW (Borgatti 2002)

Intervention points

Our approach yielded 3 bodies of information to consider for identifying the most effective intervention points for enhancing sustainability in the system (Fig. 3). First, the variable weights captured the relative importance of each variable in the system. Second, the current state analysis captured which variables were closest to and furthest from a sustainable state. Finally, the system analysis captured how much the system variables influenced and were influenced by the other variables in the system. Analyzing these datasets together provided the basis for a nuanced evaluation of which intervention points would be most strategic for enhancing overall systemic sustainability. Using this approach, we identified irrigation, fertilizer use, and soil quality as the most effective intervention points for enhancing sustainability in Sinaloa maize production. Our analysis is limited to agro-environmental variables; however, there are political, institutional, economic, and cultural aspects of the system that need to be considered in conjunction with our results to contextualize the intervention points within local practices and constraints.

Fig. 3

Summary of results. The weights, and direct and indirect influence results (normalized between 0 and 1) are plotted on the primary vertical axis. The current state results are plotted on the secondary vertical axis

To illustrate how we used these data to identify intervention points, consider again the example of the variable of irrigation. Through weighting, stakeholders identified irrigation as highly important to the sustainability of maize production (see Fig. 3). The current state analysis revealed that irrigation was far from a sustainable state: experts estimated that farmers were using nearly double the amount of water they needed (105 vs. 65 net cm/ha/year), which suggested that there was plenty of room for reducing water use over the course of the season (see Fig. 1). Furthermore, reducing water use could be achieved relatively quickly, unlike slower acting (but also strategic) variables such as soil quality. Finally, the system analysis revealed that the variable of irrigation had high direct and indirect influence in the broader agro-environmental system (see Table 3). This means that an improvement in the sustainability of irrigation also means an improvement in the variables that irrigation influences: soil quality, water quality, agricultural land, erosion, aquatic ecosystem, terrestrial ecosystem, and yield (see Fig. 2). For more information about intervention points in the Sinaloa maize system, see Bausch (2011a).

Crosschecking results and recommendations

After completing our initial analysis, we consulted with stakeholders again. During part 2 of our stakeholder engagement, we organized 2 workshops (see “Stakeholder engagement strategy” above) to communicate our results to the community, crosscheck our analysis with local experts, and further incorporate their perspectives. At the beginning of each workshop, we provided the stakeholders with 2 paper handouts. The first was a summary of the methods and results that included the results of the stakeholders’ weighting of the system variables, the current state analysis, and the system analysis diagram. The handout did not include the value functions. The second handout was for written feedback, which we collected from the stakeholders at the end of the workshops. This handout included the current and ideal state of each variable indicator. We asked, “Do the indicators seem appropriate for the agro-ecological system variable they represent? Do the current and ideal states of each indicator seem correct to you? If not, what would you suggest?”

During the workshops, we presented the evaluative criteria (normative sustainability principles), methods, results, intervention points, and recommendations (Bausch 2011b). We then opened the floor to the workshop participants for their questions and comments, followed by general discussion. While the workshop participants confirmed most of our findings, they also provided valuable observations that we incorporated into our analysis, in particular regarding the implications of the assessment results for local decision making, and the communication of the results.

Discussion

Both MCDA and system analysis are well-established methods. Nevertheless, as far as we know, the combination of these methods has not been used for agricultural sustainability assessment. Based on our experience with this approach assessing the environmental sustainability of maize production in Sinaloa, we posit that combining MCDA and system analysis provides advantages for developing transparent, reproducible assessments that are capable of eliminating the tradeoffs identified by Binder and colleagues (2010), and facilitate a systematic approach to identifying intervention points for enhancing sustainability. Here, we discuss our assessment of Sinaloa maize in terms of Binder and colleagues’ (2010) three dimensions (normative, procedural and systemic) of sustainability assessment, as well as the strengths and limitations of the approach, and the applicability of the assessment results.

Normative dimension

The distance-from-the-ideal-state approach is goal oriented,⁴ in which “sustainability (is) interpreted as a property of agriculture developed in response to concerns about threats to agriculture, with the goal of using it as a criterion for guiding agriculture as it responds to change” (Hansen 1996, p. 117). Value functions accommodate assessing indicators with various “reference values” or ideal states, such as (as summarized by Van Cauwenbergh et al. 2007) targets, thresholds, averages, trends, comparisons among sectors, and ranges.

We conceptualized the ideal state for each indicator as a short-term goal, achievable within 5 years. This means that in 5 years, or as these goals are achieved, new goals or ideal states should be identified. Furthermore, the variables, indicators, and their weights may be reviewed and revised as needed every 5 years to reflect changes in the system and stakeholder values (see “Multidimensionality” below). This approach concurs with the idea that sustainability is an iterative and dynamic process (Leach et al. 2010). Assessing this dynamic process requires a baseline or snapshot of the state of the system in terms of sustainability against which future progress can be assessed.

In terms of the normative dimension, an advantage of the approach of combining MCDA and system analysis is that it can accommodate various perspectives (e.g., those of local decision makers, community groups, government, researchers, etc.), as well as concepts, goals, and spatial and temporal scales, as demonstrated, for example, by Ingold (2011). A similar approach can be used for long-term planning and visioning.

We relied on expert knowledge, and sought to include diverse stakeholder perspectives. The process of stakeholder identification and engagement will be unique in each context; these are among the many important decisions an assessment team will make in the process of assessment. We were fortunate that the stakeholders and experts we approached were very accommodating and supportive of our efforts, and contributed to the normative goals of the assessment through their weighting of system variables. However, in any context there is always the possibility that certain stakeholders or groups may decline to participate. In other instances, it is also possible that a particular stakeholder or stakeholder group may try to dominate the process, silencing other participants’ views (Mosse 2001). Nevertheless, this should not be a limiting factor for the assessment, as long as their views are represented through secondary data or proxy experts, and the types of stakeholders who did participate are clearly documented. If stakeholders decline to participate, the reason for declining should be investigated.

Systemic dimension

Assessments of agricultural sustainability typically do not capture interactions among variables or indicators; many describe sustainability as the linear outcome of multiple, semi-aggregated, weighted indicators (e.g., B. Hansen et al. 2001; Rigby et al. 2001; Taylor et al. 1993). We used system analysis to provide a conceptual map of relationships among system variables. This helped us to infer the most effective intervention points based on our analysis, which may enhance the applicability of the assessment, or at least spark discussion about strategies moving forward. While there are other, more complex means of identifying variable relationships, our approach provided a simple and transparent system model. We found the system analysis useful for communicating our understanding of the system relationships to a diverse audience. Developing the impact matrix requires a basic understanding of how the system works, which is sufficient in most cases for identifying effective intervention points. This suggests that the system analysis could be implemented with stakeholders in a participatory process. The impact matrix can also be an input for more formal modeling approaches that might capture how and/or how much variables or their indicators impact or respond to changes in other variables (e.g., Mendoza and Prabhu 2005), depending on the availability of information. Such a model would provide additional material in support of sustainability interventions, and, like system analysis, is highly compatible with MCDA (e.g., Brans et al. 1998; Santos et al. 2002). However, in our experience, a simple model is sufficient for identifying intervention points and developing practical recommendations for enhancing sustainability.

Multidimensionality

Multidimensionality can be understood as the social, economic, and environmental facets of a system, and/or its assessment. As Binder and colleagues (2010) point out, agricultural sustainability assessments tend to focus on environmental aspects of a system—neglecting economic and social aspects—and thus fail to reflect the multidimensionality of the system. This neglect raises a normative concern regarding what aspects of the system are prioritized, as well as the need for a multidimensional approach to assessment. While we concur with this observation in principle, the reality is that a multidimensional assessment can be quite difficult to achieve in practice, as we found while assessing maize production in Sinaloa. There were no previous analyses of environmental sustainability of maize production specific to the region; thus, we considered the environmental domain to be the appropriate place to start. In focusing on environmental sustainability, we ran the risk that, in terms of system analysis, the variables and/or indicators we analyzed might not be compatible with the relevant social and economic variables of the system, which could inhibit the ability to expand on our analysis in those domains, and identify tradeoffs. However, we agree with Reed et al. (2005) that sustainability assessment is an iterative and evolving process. The approach we describe is sufficiently flexible that alternative indicators may be used to monitor existing environmental variables that are more compatible with social and economic concerns, while additional variables of the environmental, social, and/or economic domains may be added as needed. The weight of any new variables could be established by administering a revised ranking questionnaire, which should be done periodically anyway, as system conditions and stakeholder values are likely to change over time (see “Normative dimension” above).

Other researchers have used MCDA and system analysis to great effect within the social and economic domains of sustainability (e.g., Ingold 2011; Munda 2004). MCDA has been used successfully to evaluate the social dimensions of agricultural vulnerability in Mexico and Central America (Eakin and Bojórquez-Tapia 2008; Eakin et al. 2011); a similar procedure would be applicable for sustainability assessment. Identifying the relevant social and economic variables might be more complicated, given the relative paucity of existing scientific literature on these aspects of agricultural sustainability, and thus would require considerable stakeholder input to capture the specific context of production and to relate these to more general variables relevant to sustainability outcomes and principles. Nevertheless, the general procedure would be the same. The MCDA would be useful for hypothesizing, and eventually defining, the relationships of specific social and economic variables in relation to sustainability through value functions. It would also systemically identify the relative contribution of each domain to the overall sustainability of the system when all the domains are aggregated. Similarly, system analysis would provide a means of conceptualizing the interaction of variables in all three domains, capturing the interaction of economic variables or producer knowledge, for example, with ecological processes.

Analyzing variables

We chose to use variables for the system analysis, in contrast with the typical practice in sustainability assessment of using indicators only, for a few reasons. First, indicators often have little direct meaning for sustainability if they are not associated with a concept and context. Because we were consulting with stakeholders, for communication purposes we used terms and concepts that could be understood without much explanation. Thus, we developed the system variables as interrelated, but individually defined sustainability concepts that we measured and assessed with indicators. Second, in many contexts where agricultural sustainability assessments are needed, data are scarce, of poor quality, and collected on an irregular basis. By focusing on variables rather than specific indicators, there is flexibility with data inputs. Alternative indicators of the system variables can be used in subsequent assessments as knowledge improves without significantly altering the system analysis because the variables can remain constant. Third, we posit that using clearly defined variables, as well as translating the current state analysis into a dimensionless scale with value functions, facilitates comparisons across assessments and contexts. Comparison is more difficult when specific indicators are used without reference to a broader variable that is salient in diverse contexts. This addresses the first tradeoff identified by Binder et al. (2010) between benchmarking and system-specific analysis: both are feasible with this approach.

Procedural dimension

With this approach to assessment, we have sought to enhance reproducibility and transparency, important characteristics that have been highlighted in sustainability assessment literature (e.g., Binder et al. 2010; Pischke and Cashmore 2006). The data inputs, their sources, and the assumptions built into the assessment are clearly documented and traceable, and therefore testable. The value functions formalize and visualize the hypothesized relationships of variables to sustainability. Once created, they can be discussed, tested, and revised as needed. System analysis is based on a simple matrix, which can be visually interpreted in a centrality map that makes the basic relationships among system variables visually explicit. Therefore, we conclude that these methods are useful for enhancing transparency and reproducibility.

While our overall approach was quantitative, the quantitative results were informed by the rich qualitative data that we gathered through our interviews. It would be possible to do a sustainability assessment using MCDA and system analysis without such qualitative data; however, it would likely fail to capture the nuances of the local system that would help make it relevant to stakeholders and decision makers. We thus encourage the incorporation of qualitative data through interviews and other ethnographic methods (e.g., participant observation) in sustainability assessment, in particular for the approach we describe.

Indicator measurement and assessment

In many contexts, sustainability assessments are constrained by a lack of available scientific data, peer-reviewed information, and systemic understanding. Nevertheless, by definition, the urgency of sustainability challenges and the implications of today’s decisions for sustainable outcomes in the future require consideration of the best available information in decision making (Sarewitz et al. 2010). Our assessment was based on an optimal combination of available published secondary data, expert knowledge, and informed extrapolations from the existing literature. While compiling this information can be time consuming, it is a relatively low-cost and expedient approach. Further benefits of collecting existing data and seeking expert knowledge are that (1) it identifies available data sources that can be used to monitor sustainability over time at no additional cost; (2) it gives local experts a concrete stake in the assessment process; and (3) it puts the current state of knowledge in one place, calling attention to areas in need of more research. A disadvantage of using expert knowledge and existing data is that the assessment team has little control over data quality. Nevertheless, crosschecking data with different sources, as well as highlighting data requiring further precision in future research, are all productive ways to counter this issue. The accuracy of the data can also be tested in subsequent research (see “Critical reflections and future research” below).

Relying primarily on existing regional data was our solution to the second tradeoff that Binder and colleagues (2010, p. 79) identified between “‘easy’ and ‘fast’ assessment” and “regional specific and applicable results”. However, we only partially agree that this tradeoff exists. In our approach, making local contacts, meeting with experts and stakeholders, gathering data, and sorting, crosschecking, analyzing, and assessing that information can still be a time-consuming and challenging process. Furthermore, whether the assessment is regional specific and applicable is not a question of whether the assessment was easy and/or fast, but rather one of data availability, quality, and accessibility, as well as stakeholder input, and transparency in the assessment process and results. Whether there is a need to collect primary data at the scale of choice, and whether time and resources are devoted to this, are choices that the assessment team must make based on the interaction of these factors.

We assert that the combination of MCDA and system analysis allowed us to address the third tradeoff identified by Binder et al. (2010, p. 79) between an “understandable message (aggregation)” and “system based trade-off analysis”. Both of these goals are feasible with our approach. The approach allows an analysis of tradeoffs within the system by considering together the variable weights, current states, and system influence (centrality and betweenness) (see Fig. 3), while providing an aggregate sustainability score for the system.

In the Sinaloa assessment, one of the biggest challenges of using MCDA for the current state analysis was identifying the anti-ideal states of the indicators for the value functions, because this kind of threshold is hardly discussed in the literature. Thus, for most indicators we relied on expert opinion, or made educated guesses about the anti-ideal state. However, we found that thinking through the anti-ideal state of each indicator lead us to a deeper understanding of the indicators’ relationship to sustainability. In future applications of this approach, an open discussion with stakeholders on how to conceptualize the anti-ideal state could be a useful way to enhance understanding of the relationship of indicators to sustainability among researchers, experts, and stakeholders alike.

Applicability

Wiek and colleagues (2012) have shown in their review of exemplary cases that the contribution of assessments to transformational change toward sustainability can be limited and elusive. In our case, it was beyond the scope of our research to determine how applicable or transformational the results of our analysis would be. Consequently, our assessment of Sinaloa maize should be considered a baseline study with which progress toward sustainability could be measured and compared in the future. The advantages of our general approach are that it is transparent, systematic, and rigorous, providing a foundation from which to compare a future state of the system.

Critical reflections and future research

In this article, we have presented an approach to sustainability assessment that lays bare the assumptions underlying the analysis and assessment of a system. These assumptions—particularly the untested value functions and untested data inputs—may be considered hypotheses that can be scientifically tested in future research. We suggest that the variables that the stakeholders weighted as highly important to sustainability be the starting point for this effort. A second line of research would be to do a follow-up study on the impact of the assessment for decision making in the region. A third line of research might expand the scope of assessment beyond the environmental domain into the social and economic domains to test and discuss the feasibility (see “Multidimensionality” above). Finally, if this approach to assessment is applied in other contexts, there will be opportunities for comparison to illuminate how systems, stakeholder priorities, and strategies vary across contexts. It would also be possible to examine how specific assessment design decisions, such as stakeholder sampling strategies, or weighting method, might change the assessment outcomes.

Conclusions

In this article, we described and discussed combining MCDA and system analysis as a unified approach to agricultural sustainability assessment, drawing on examples from an environmental sustainability assessment of irrigated commercial maize production in Sinaloa, Mexico. We framed and discussed this approach in terms of the normative, systemic, and procedural dimensions of sustainability assessment, as outlined by Binder and colleagues (2010). We found that this approach provides a practical, flexible, systematic, transparent, and reproducible approach to documenting, analyzing, and assessing agricultural sustainability, and developing recommendations for enhancing sustainability. The approach has the capacity to advance how we understand sustainability by making explicit the assumptions and data sources that go into the analysis. We believe that it could be useful for assessing other sustainability challenges as well.

With mounting pressure to enhance global food security, there is a clear need to improve our understanding of the sustainability implications of agricultural production in diverse contexts. Much of the growth in agricultural productivity and planted area is expected to occur in regions of the world where data collection is relatively poor, and resources for monitoring and assessment are limited. The approach we described here is designed to pragmatically address the need for assessment approaches that capture and communicate the normative, systemic, and procedural dimensions of sustainability assessments, accommodate the variable data quality and the state of knowledge in the contexts where it is applied, and leverage that knowledge to support decision making. In these contexts, by making assumptions transparent and facilitating a discussion about the dimensions and variables that are critical for sustainability, the process of assessment can be as instructive as the results, if not more so.

Appendices

Appendix A: Questionnaire for ranking system variables (translated from Spanish)

• (1) Considering the sustainability of maize cultivation in Sinaloa as related to the environment, please rank the following environmental issues according to their importance, with #1 as the most important, and #12 as the least important. If a good indicator for any of the environmental issues occurs to you, please write it in the third column.

  Environmental issues	Ranking	Indicator?
  Soil erosion
  Soil quality (e.g., organic material, salinity, etc.)
  Water quality (e.g., pollution, agricultural runoff, etc.)
  Incidence of pests, weeds, and/or disease
  Yield/yield loss (e.g., Tons/ha, surface area not harvested, etc.)
  Natural terrestrial ecosystems (e.g., forests, sand dunes, etc.)
  Natural aquatic ecosystems (e.g., rivers, lakes, the ocean; e.g., eutrophication, dissolved solids, fish kills, etc.)
  Nitrogen fertilizer use (e.g., efficiency, volume used, etc.)
  Irrigation water use (e.g., allocation of water, volume of water used per season, etc.)
  Fossil fuel use (e.g., diesel consumption per season, etc.)
  Pesticide use (e.g., toxicity of pesticides, volume of pesticides applied per season, etc.)
  Land use (e.g., crop diversity, land use change, etc.)

• (2) Are there other issues that are not listed that should be considered? What are they, what would be a good indicator, and where would you put them in the ranking of importance to environmental sustainability?

Appendix B: Value functions

A value function is a mathematical expression that is used to normalize values of a variable in a common scale (Beinat, 1997). They involve a transformation from a natural scale to a scale of 0 (anti-ideal) to 1 (ideal). In general, there are two types of value functions: nominal and continuous. Nominal value functions are used to represent the level of satisfaction provided by different states denoted by names, such as soil type. Continuous value functions are used to represent the level of satisfaction provided by the states of continuous variables, such as percent, or hectares. Because they are continuous, the functions form a family of continuous curves. In sustainability assessment, the level of satisfaction (v) refers to the proximity of the value of the variable in its natural scale to the ideal state.

Increasing

The level of satisfaction increases as the value of the variable increases, reaching its ideal value at the highest point of the range. There are two types of increasing functions:

$${\text{Concave:}}\,v\text{ } = \text{ }\frac{{e^{yx} - y^{ - } }}{{y^{ * } - y^{ - } }}$$

(A.1)

$${\text{Convex:}}\,v\text{ } = \text{ }\frac{{1 - e^{-\gamma x} - y^{ - } }}{{y^{ * } - y^{ - } }}$$

(A.2)

when $\gamma \text{ } = \text{ } - \log \left( {\frac{{\log (1.1\text{ } + \text{ }0.88(10\text{ } - \text{ }\beta )}}{{\log (x_{{\text{max}} } )}}} \right)^{2}$ (A.3) where γ is the modulator of the exponential function (1/γ estimates the interval when the function doubles in value), β is the saturation factor that determines the depth of the curve, $y^{ - }$ and $y^{ * }$ are the minimum and maximum that can be obtained in the value function, and x _max is the maximum value of the variable in its natural scale (see Figs. 4, 5).

Fig. 4

Increasing concave value function

Fig. 5

Increasing convex value function

Decreasing

The level of satisfaction decreases as the variable increases, reaching the ideal value at the lowest point of the range. There are two types of decreasing functions:

$${\text{Concave: }}v\text{ } = \text{ }\frac{{e^{-\gamma x} - y^{ - } }}{{y^{ * } - y^{ - } }}$$

(A.4)

$${\text{Convex:}}\,v\text{ } = \text{ }\frac{{1 - {\text{e}}^{{ - \left( {\frac{x - 30}{\delta }} \right)}} - y^{ - } }}{{y^{ * } - y^{ - } }}$$

(A.5)

when $\delta \text{ } = \text{ }10^{{\left( {\frac{3}{{10(\log (x_{{\text{max}} } - \beta ))}}} \right)}}$where δ is the modulator of the exponential function (see Figs. 6, 7).

Fig. 6

Decreasing concave value function

Fig. 7

Decreasing convex value function

Optimum

This family of curves includes the bell function, in which the level of satisfaction increases as the variable increases to a point in the middle of its range where it reaches its ideal point, after which the level of satisfaction decreases as the variable continues to increase until reaching the highest point in its range:

$${\text{Bell:}}\, v\text{ } = \text{ }\frac{{{\text{e}}^{{ - \left( {\frac{{x - x_{{\text{max}} } }}{\alpha }} \right)^{2} }} - y^{ - } }}{{y^{ * } - y^{ - } }}$$

(A.6)

when $x_{{\text{min}} } < x^{ * } < x_{{\text{max}} }$

where α is the maximum extent of the bell, x _min is the minimum value of the variable in its natural scale, and x ^* is the value of the ideal point of the variable in its natural scale (see Fig. 8).

Fig. 8

Optimal maximum value function

In addition to the bell curve, this family of curves includes sigmoid relationships:

Increasing sigmoid:

$${\text{Optimal maximum:}}\,v\text{ } = \text{ }\frac{{{\text{e}}^{{ - \left( {\frac{{x - x_{{\text{max}} } }}{\alpha }} \right)^{2} }} - y^{ - } }}{{y^{ * } - y^{ - } }}$$

(A.7)

when x ^* = x _max

$${\text{Optimal minimum:}}\,v\text{ } = 1-\frac{{{\text{e}}^{{ - \left( {\frac{{x - x_{{\text{min}} } }}{\alpha }} \right)^{2} }} - y^{ - } }}{{y^{ * } - y^{ - } }}$$

(A.8)

when $x^{ - } \text{ } = \text{ }x_{{\text{min}} }$

where $x^{ - }$ is the value of the anti-ideal of the variable in its natural scale (see Figs. 9, 10).

Fig. 9

Increasing sigmoid optimal maximum value function

Fig. 10

Increasing sigmoid optimal minimum value function

Decreasing sigmoid:

$${\text{Optimal maximum}}:v\text{ } = \text{ }\frac{{{\text{e}}^{{ - \left( {\frac{{x - x_{{\text{max}} } }}{\alpha }} \right)^{2} }} - y^{ - } }}{{y^{ * } - y^{ - } }}$$

(A.9)

when $x^{*} \text{ } = \text{ }x_{{\text{min}} }$.

$${\text{Optimal minimum}}:v\text{ } = 1-\frac{{{\text{e}}^{{ - \left( {\frac{{x - x_{{\text{min}} } }}{\alpha }} \right)^{2} }} - y^{ - } }}{{y^{ * } - y^{ - } }}$$

(A.10)

when $x^{ - } \text{ } = \text{ }x_{{\text{max}} }$ (see Figs. 11, 12).

Fig. 11

Decreasing sigmoid optimal maximum value function

Fig. 12

Decreasing sigmoid optimal minimum value function