Keywords

1 Introduction: Curricular Reform Strategies to Address Sustainability

For sustainable engineering to effectively contribute to global sustainability, engineering curricula must be updated to properly train sustainability-conscious engineers. Two common methods for effective incorporation of sustainability concepts into university curricula include horizontal and vertical integration. Horizontal integration is a strategy where sustainability concepts are incorporated into several courses across a curriculum, while vertical integration involves the addition of new sustainability courses into an existing curriculum (Ceulemans and De Prins 2010). Dissemination of a new course with sustainability content is essential for teaching students about fundamental concepts and principles related to sustainability (Peet and Mulder 2004). However, vertical integration alone may be insufficient because only teaching students about sustainability separate from core engineering concepts does not encourage them to incorporate sustainability into their professional designs and practices (Peet and Mulder 2004). Rather, integration of sustainability into existing courses may aid students in viewing sustainability in a systemic and holistic manner by demonstrating how sustainability and technical content can be blended to create sustainable designs (Peet and Mulder 2004; Ceulemans and De Prins 2010).

1.1 Sustainability Assessments

Cmaps are innovative assessment tools that can be used to assess student sustainability understanding. Cmaps are student-generated graphical tools for organizing knowledge in which concepts related to a particular knowledge domain are directionally connected using descriptive linking lines (Novak and Canas 2006). Students are provided with a focus question and asked to transcribe their internal knowledge into a cmap that can be easily reviewed (Ruiz-Primo 2000). Thus, cmaps allow students to explicitly reveal knowledge content, while also demonstrating how that content is mentally organized. Sustainability is a rapidly-evolving and complex knowledge domain, in which highly interconnected economic, environmental, social, temporal, and spatial concepts are very important. As a result, concept-map-based assessment tools are ideal for identifying concepts that students associate with sustainability, as well as quantifying the interrelationships between sustainability dimensions. However, practical methods for scoring cmaps are needed before concept-map-based assessment tools are widely applied (e.g., Besterfield-Sacre et al. 2004).

1.2 Project Scope

The goal of this project was to compare the impact of different strategies for integrating sustainability into undergraduate engineering curricula on student sustainability knowledge. The Georgia Institute of Technology (GT) has added a sustainability-focused course into their curriculum (vertical integration), while efforts to add sustainability concepts into existing courses (horizontal integration) is left to the discretion of the instructor. In contrast, James Madison University (JMU) has included sustainability and sustainable design as key elements that are woven throughout their interdisciplinary engineering curriculum. Concept maps created by upperclassmen at these two differing institutions were collected and analyzed to address the following research questions: (1) How do differences in sustainability integration strategies impact the structure of student knowledge? (2) What are the differences in the knowledge content of students from institutions with different sustainability curricula?

2 Background Information: Concept Maps

Concept maps (cmaps) can be used to capture the structure and content of student knowledge in a given domain. Several scoring methods are available to extract data from cmaps. As theoretically-grounded tools, cmaps are used as assessment strategies in a variety of fields, including sustainable engineering.

2.1 Function and Structure

Cmaps are graphical tools for organizing knowledge. Construction of a cmap is completed by enclosing concepts related to a central topic in boxes and using connecting lines, as well as linking phrases, to depict relationships between concepts (Novak and Canas 2006). The basic unit of a cmap is a proposition, which includes two concepts joined by a descriptive linking line. Propositions that include the cmap topic define the map hierarchies, and the level of hierarchy is defined by the number of concepts in the hierarchy. Cross-links, which are important for depicting connectedness, are descriptive linking lines that create propositions by joining two concepts from different map hierarchies (Watson et al. 2014).

2.2 Theoretical Bases

Use of cmaps is supported by cognitive psychological research in the area of semantic memory theory. Semantic memory refers to an organized database of concept-based knowledge, such as meanings, understandings, and images (Tulving 1972). Semantic memory theory posits that knowledge networks are formed by creating directed links between related concepts. Some researchers have proposed that networks are structured hierarchically with broad concept categories being divided into more specific sub-categories (Collins and Quillian 1970), while other researchers have rejected this assumption (e.g., Ruiz-Primo 2000). Nevertheless, interconnectedness within the structure is an important network characteristic, since it increases one’s ability to access concepts (Turns et al. 2000) and is a key feature that differentiates expert and novice knowledge frameworks (Ruiz-Primo 2000). Since cmaps mimic the structure of internal semantic networks, student-generated constructs may be used to infer a student’s domain understanding.

2.3 Use as Assessment Tools

Cmaps are an alternative to traditional assessment tools for characterizing knowledge content and structure. One significant challenge in using cmaps as assessment tools is identification of a robust scoring method (Besterfield-Sacre et al. 2004). Several scoring methods, including the commonly-used traditional approach, have been summarized elsewhere (Watson et al. 2014).

The traditional method (Besterfield-Sacre et al. 2004) involves quantifying the number of components in each cmap. For instance, the number of concepts, the number of hierarchies and highest level of hierarchy, and the number of cross-links are used to determine sub-scores for knowledge breadth, depth, and connectedness, respectively (Table 1). Some authors advocate for sub-scores being condensed into one metric using weightings for component sub-scores. For instance, Novak and Gowin (1998) propose that each proposition and example should receive 1 point, each level of hierarchy should receive 5 points, and each cross-link should receive 10 points. Alternatively, Bayram (1995) assigned one point for each proposition and cross-link, while each hierarchy was multiplied by its level (e.g., 2 points for a hierarchy with two levels). Novak and Gowin (1998) and Bayram (1995) each calculate the overall cmap score as the sum of weighted points. In contrast, Markham et al. (1994) argue that component sub-scores are more valuable when analyzed independently. Nevertheless, component-level scoring can provide an objective method for quantitatively scoring cmaps.

Table 1 Rubric for traditional cmap scoring approach (Besterfield-Sacre et al. 2004)
Full size table

2.4 Applications in Sustainability Education

Several authors have used cmaps to characterize student sustainability understanding. Segalàs et al. (2008) investigated the effectiveness of six sustainability courses by comparing student cmaps before and after course delivery (Segalàs et al. 2008). Evaluation of cmaps revealed that complexity of cmaps resulting from courses employing constructive and community-based pedagogies was higher than from courses using more traditional instructional strategies (Segalàs et al. 2010). Similarly, Borrego et al. (2009) analyzed cmaps before and after a green engineering course using the holistic scoring method and found that the comprehensiveness, correctness, and organization of student maps increased after course delivery. Use of cmaps as assessment tools were also used to monitor student learning in capstone engineering courses (Watson et al. 2013). Thus, cmaps are beginning to be applied as tools for studying student sustainability knowledge.

3 Research Methods

3.1 Student Populations

Student sustainability knowledge was investigated at GT and JMU for students beginning their capstone design experiences. Seniors (4th year) in Civil and Environmental Engineering (CEE) at GT were recruited to construct cmaps documenting their sustainability knowledge. GT has taken a largely vertical integration approach to incorporating sustainability into the CEE curricula, with primary exposure occurring in a required junior-level (3rd year) systems course. For comparison, juniors enrolled in an interdisciplinary engineering program at JMU were also invited to construct sustainability cmaps. JMU has embraced a horizontal integration approach, as sustainability is an integral part of their entire curriculum from the freshman introductory course through the two-year capstone design experience. Most of the students in the JMU sample had not yet completed the two required senior-level sustainability science and lifecycle courses. Additional information on the curricula at GT (Watson et al. 2013) and JMU (Nagel et al. 2013; Pierrakos et al. 2008) are available.

3.2 Concept Mapping Assessment

The concept mapping assessment was conducted as outlined by Watson et al. (2014). Before completion of the sustainability concept mapping task, students participated in a brief concept mapping workshop to familiarize them with the construction of cmaps using CmapTools. Afterward, students were asked to create a concept map on the focus question: “What is sustainability?”. Students were provided with up to three hours to complete their cmaps at GT, although most students were finished after thirty minutes. Students at JMU were provided with approximately thirty minutes to construct cmaps and all students finished within that timeframe.

3.3 Concept Map Analysis

Three judges were trained to use the traditional scoring method. Judges practiced scoring approximately 10 cmaps and discrepancies were discussed to promote future interrater reliability. Krippendorf’s alphas for the training sessions were at least 0.67, which is appropriate for exploratory research (Krippendorff 2004). Further details on scoring calibration are available (Watson et al. 2014).

After scoring calibration, judges scored cmaps generated by JMU and GT students. Two judges scored each submission. First, judges individually quantified traditional scoring parameters. Inter-rater reliability, based on Krippendorff’s alpha, was deemed to be acceptable (Krippendorff 2004) for all parameters (Table 2). Discrepancies in scores were discussed by the judges and consensus scores were used in all subsequent statistical analyses. Since data was determined to be non-normal (Shapiro-Wilk test), non-parametric Kruskal-Wallis tests were used to compare cmap scores based on institution. Significant relationships were identified for p ≤ 0.05.

Table 2 Inter-rater reliability for Cmap scoring
Full size table

In order to compare the students’ depth and breadth of sustainability knowledge, the content of cmaps was analyzed using word clouds, a strategy used by others for content analysis (Huynh et al. 2013). Concepts were extracted from the cmaps and used to generate word clouds with WordleTM. For consistency, the extracted concepts were not modified in any way (e.g., spelling could have been corrected). Further, default settings were used for language (remove numbers, remove common English words, leave words as spelled) and a horizontal layout was selected for both GT and JMU word clouds. For each word cloud, the frequency of concepts was tallied using the show count function. Given the large number of words, only concepts with a count greater than five were recorded. Concepts were then coded in accordance with a four pillar conceptualization of sustainable engineering—economic, environmental, social, and technical pillars. Concepts that did not directly match one of the four categories or could easily be coded as more than one pillar were coded as “other”. The resultant word clouds and content themes are described in Sect. 4.2.

4 Results

A study was conducted to compare sustainability cmaps between student populations at institutions which initiated either horizontal (JMU) or vertical (GT) integration of sustainability content into undergraduate curricula. The structure of cmaps was analyzed using the traditional scoring approach, while cmap content was examined using word clouds.

4.1 Traditional Cmap Scores

The structure of JMU students’ cmaps was more complex than those constructed by GT students (Table 3). Specifically, knowledge breadth was greater for JMU students than for GT students, given that the median number of concepts was 23 and 12, respectively (p ≤ 0.001). Similarly, JMU students demonstrated more substantial knowledge depth than did GT students, with the median highest hierarchy being 4 versus 3 for each group, respectively (p ≤ 0.001). Even still, JMU students prepared cmaps that were significantly more interrelated in structure than those submitted by GT students, with the median number of cross-links recorded as 4 versus 2, respectively. Due to the broader, deeper, more inter-related structure of JMU cmaps, overall scores were nearly twice as high (Med = 120.98) as for GT cmaps (Med = 61.35) (p ≤ 0.001).

Table 3 Comparison of traditional scores across institutions
Full size table

4.2 Content Analysis

Visual and thematic analysis of the word clouds allowed qualitative comparisons of depth and breadth of content coverage (Figs. 1 and 2; Table 4). Visually, the word clouds illustrate a clear environmental theme in the GT concept maps, which is often expected of students’ sustainability knowledge (e.g., Segalàs et al. 2010). JMU concept maps showed a more balanced conceptualization with greater emphasis on the social pillar.

Fig. 1
figure 1

GT sustainability world cloud

Full size image
Fig. 2
figure 2

JMU sustainability word cloud

Full size image
Table 4 Comparison of concept map content from word clouds
Full size table

Table 4 shows that GT students demonstrated unbalanced representations of sustainability with environmental concepts dominating the other categories in terms of both diversity of concepts and frequency. From JMU concept maps, the social dimension demonstrated the greatest diversity of terms amongst the four categories, although environmental concepts were included with greater frequency (30 vs. 27 % for social). For both GT and JMU concept maps, the economic dimension was least well represented, both in terms of number of concepts within that category (8 and 6 respectively) and frequency (11 and 12 % respectively).

5 Discussion

5.1 Structure of Student Knowledge

Examining cmap structure clearly shows that students completing a curriculum with horizontal integration of sustainability content displayed broader, deeper, and more inter-connected sustainability knowledge, as compared to students enrolled in a vertically-integrated curriculum. This finding aligns with previous publications that underscore the importance of blending sustainability content with existing engineering coursework (Peet and Mulder 2004; Ceulemans and De Prins 2010). Examining knowledge breadth, a group of sustainability experts was reported to include an average of 19.8 concepts in their sustainability cmaps (Coral 2009), as compared to medians of 12 and 23 for GT and JMU students, respectively (Table 3). Thus, JMU students demonstrated more expert-like knowledge breadth than GT students. Furthermore, the greater level of concept connectedness displayed by JMU students is especially significant, since connectedness of knowledge is known to increase student ability to access concepts and is a key feature that differentiates expert and novice knowledge frameworks (Turns et al. 2000; Ruiz-Primo 2000). Overall, data from the current and previous studies suggest that horizontal integration has a positive impact on students’ sustainability knowledge networks.

5.2 Content of Student Knowledge

Examining word clouds suggests that horizontal integration of sustainability into undergraduate curricula may encourage development of more balanced sustainability knowledge frameworks, as compared to vertical integration. Notably, “environment/al” was frequently represented in all student cmaps, although students from JMU clearly included the term “social” more often than students from GT (Figs. 1 and 2). Further, JMU students distinguished between individual and community dimensions of the social pillar. Prior research on students’ conceptual understanding of sustainability has overwhelmingly suggested that students often emphasize the environmental dimension of sustainability, while largely neglecting social aspects (e.g., Watson et al. 2014). Interestingly, one study has proposed that social considerations, especially related to end-user safety and well-being, are often evident in student designs, even if cmaps demonstrate little knowledge of social sustainability (e.g., Watson et al. 2013). Authors cite that students may not recognize that many routine aspects of engineering design promote social sustainability, at least to some extent. Consequently, perhaps integration of sustainability content into the unique sequence of design courses at JMU contributed to students awareness of social aspects of sustainability. It is important to note that the majority of JMU students participated in a client-based sophomore design project and are exposed to identifying stakeholder needs and impacts over a project’s lifecycle during that course experience. The “equity” dimension of social sustainability remains a weak area for students’ conceptualizations and applications of sustainability knowledge, and the content of cmaps in this study did not indicate otherwise. Likewise, economic analysis could be further developed to enable appropriate and realistic trade-offs amongst the four dimensions. Nevertheless, the increased balance among sustainability dimensions demonstrated by JMU students is desirable, since promoting sustainability inherently requires an understanding of all inter-related dimensions (Davidson et al. 2007).

5.3 Limitations and Implications for Research

Several limitations are inherent in the design of this project. Foremost, only one institution exemplifying each integration strategy was investigated. Vertical and horizontal integration are two very broad strategies that can encompass a variety of educational interventions. For instance, a program with a different vertically-integrated sustainability course (e.g., more comprehensive, more active, emphasis on different topics) may have proved to be more or less effective than the GT case. Consequently, investigation of integration strategies across multiple institutions may provide more generalizable results.

Second, there were several differences between student groups in the current study that could have contributed to differences in cmap scores, beyond just exposure to different integration strategies. Specifically, students were in different stages of their academic careers (juniors at JMU and seniors at GT) and engaged in a non-discipline specific curriculum at JMU versus a CEE curriculum at GT. However, student academic development may have actually been more similar than expected, since both groups were entering their first comprehensive, independent design experience. Even still, junior JMU students demonstrated more comprehensive sustainability knowledge than senior GT students, despite completing fewer semesters of the prescribed curriculum. Despite these differences in student samples, it is expected that significant differences in student sustainability knowledge demonstrated in this study is due to inherent differences in the curricula of the two institutions. The methods and results of this study do not address how students’ conceptualizations of sustainability translate to performance on the capstone design projects, which is an area for future research.

6 Conclusions

A study was conducted to evaluate the efficacy of horizontal versus vertical integration of sustainability content into undergraduate engineering curricula. Cmaps were collected from students in a horizontally-integrated, interdisciplinary engineering program, as well as a vertically-integrated, CEE program. The structure of student knowledge was analyzed using the traditional scoring approach, while specific content was evaluated using word clouds. The following conclusions were made based on the results:

  1. 1.

    Horizontal integration resulted in student cmaps with greater breadth, depth, and inter-connectedness, as compared to vertical integration.

  2. 2.

    Students participating in a horizontally-integrated curriculum demonstrated more balanced understanding of sustainability, including the social dimension, as compared to students from a vertically-integrated program. To some extent, economic sustainability was a weakness of both programs.

Results from this study align with previously-published suggestions that horizontal integration of sustainability is important for student development. If sustainability is only taught in isolation from core engineering fundamentals, then it is possible that students will view sustainability as an afterthought during the design process. Given that current undergraduate students will soon be responsible for local and global development projects that will impact both humans and the environment, it is important to equip them with the knowledge and skills necessary to engage in sustainable design.