Overview

Today we have students from all over the world sharing online and brick-and-mortar classrooms, either in their own educational environment or through interaction with other institutions and organizations via study abroad programs. The diversity found in the classrooms of most higher education programs is a consequence of the globalized marketplace and worldwide immigration (Banks 2004), with different nationalities coming together in a classroom environment to learn how to better perform in a market that is similarly multicultural and ethnically diverse (Malekzadeh 1998). This learning environment now mandates that students develop a deeper and more comprehensive understanding of different cultures and ways of doing business in different parts of the world compared to earlier decades. The goal is to develop global awareness and citizenship, leading to a consequent employability improvement as well as providing the global community with future leaders and effective problem-solvers. In response to these factors, higher education institutions have incorporated language within their institutional missions expressing their intent to develop global/international awareness and intercultural competency among their students, to create graduates who are responsible and effective global citizens (Hovland 2014). Educators are now faced with the challenge of designing and assessing programs that can achieve this goal.

While one might assume this emphasis on global citizenship derives its primary value in that it can enable graduates to be more “marketable” or “job ready” in today’s global work environment—thus providing students with clear and tangible “private” benefits—promotion of this educational goal often coincides with the promotion of other institutional learning goals that are more frequently assumed to create “public” value; for example, social responsibility, environmental sustainability, public service, and community leadership. Indeed, many of today’s higher education programs that aim to provide these public benefits invariably define them within a global context. However, while the social mandate exists, this imperative is most often expressed as a response to factors related to the workplace environment (e.g., a more nuanced understanding of what creates effective global business operations, and increased diversity and mobility within the workplace) as well as factors related to the classroom environment (e.g., a more diverse student body population). Indeed, as Jones (2015) notes:

[There is a] growing awareness that the intercultural competence required for global contexts is equally important for living and working in today’s increasingly diverse and multicultural societies. Research indicates a rising demand by employers for university graduates with enhanced global perspectives and intercultural competence , and students themselves are showing increased interest in international and intercultural experience. Internationalization thus has both global and more local intercultural interests at its heart. (p. xii)

Defining and Assessing

With this new global reality in place, schools and universities around the world are compelled to provide students (and professionals who want to continue their studies) with a strong exposure to what happens in different cultures and countries. A careful examination of these efforts necessitates a review of how institutions are defining such terms as “global citizenship” and “intercultural competency,” as well as how they are attempting to measure their achievement. The motivation for the development and implementation of comprehensive student learning assessment within international or globally focused curricula derives from the recognition that the typical processes for operational evaluation (e.g., collecting data related to the numbers of students studying abroad or pursuing an internationally focused course of study) are insufficient if the institution wishes to learn if and to what extent its students are actually achieving the institution’s learning goals , as compared to simply participating in the internationally focused curricular activities. This point is emphasized in Green (2013), as follows:

Of central importance is the impact on students. How much, and what kind of learning does a particular course, program, or experience produce? The existence of a given set of institutional activities, and/or participation rates in various courses or programs, does not truly tell institutions what students are learning. For example, an institution may see rising study abroad participation, but that increase may or may not relate to the program’s quality or its impact on students. Similarly, the creation of new internationally focused courses or programs does not ensure that students will acquire global competencies by taking them. (p. 5)

Beelen and Jones (2015) observe that “the articulation and assessment of internationalized learning outcomes remains relatively under-reported” (p. 74). This research aspires to provide the groundwork for filling at least some part of that gap in the literature.

Institutions desiring to utilize student learning assessment tools within their international programming are faced with a broad and varied collection of options. A common choice is to utilize one of the many commercially available tests designed to assess a variety of aspects related to global and intercultural awareness (Fantini 2009; Intercultural Communication Institute 2014; University of Michigan 2014). These tests have been designed to fulfill a variety of purposes, including: assessing readiness for an intercultural experience, diagnosing areas of strength as well as those that require further development, assessing aptitude or potential for learning in an intercultural context, assessing attitude and disposition, assessing competency within a specific area or skill, assessing against a specific criterion or comparison group, assessing one’s relative ability in a bilingual context, assessing one’s intercultural development at different moments in time, and assessing achievement measured against a benchmark level of mastery or understanding (Fantini 2009). For the tests attempting to measure behavioral or attitudinal characteristics, these tools typically utilize self-reported data from respondents to evaluate their development or progress along a given set of criteria or established scale. As a result, one criticism of the use of many of these tools is their dependence on self-reported data. Thus, they can create only indirect measures of achievement.

Student learning assessment efforts at many institutions focus on measuring global knowledge and intercultural attitudes at the course and program level: the former often by direct measures (e.g., tests of factual understanding or language proficiency tests), and the latter using indirect means (e.g., surveys or questionnaires). Equally important, however, is the measurement of student attainment of global and intercultural skills and competencies. Often this type of measurement proves more challenging to implement, particularly if the goal is to obtain direct measures of attainment. One approach well suited for assessing student gains from international internships is to obtain performance evaluation data from the overseas internship supervisors (Gordon 2013). Standardized rubrics designed to measure global awareness and/or intercultural perspectives (either developed “in-house” or by consortia of university faculty, such as the VALUE rubrics mentioned below) can be utilized to assess students’ written work, such as responses to essay prompts derived from a case study focusing on a complex global issue (Landorf and Doscher 2013). Another approach is to utilize rubrics to analyze student performance on capstone projects completed either during or after an international experience (Gordon 2013).

Particularly challenging for many institutions desiring this sort student learning assessment data is developing assessment strategies that obtain direct measures of global learning and intercultural competency. One approach increasingly utilized within US higher education is the use of the “Global Learning” and “Intercultural Knowledge and Competence” VALUE rubrics, developed by the Association of American Colleges and Universities (Association of American Colleges and Universities 2014). While these rubrics are considered by many to be potentially quite useful, their ability to generate meaningful data requires that an institution create well-designed and appropriately implemented activities/assignments that will generate useful “artifacts” of student learning to be assessed using these instruments.

Student learning assessment experts are quick to note that a crucial first step in the development of effective learning assessment tools is the identification of a clearly stated working definition for the learning outcome being assessed. Thus, for this discussion it is important to consider the operational definitions of “global learning” and “intercultural competence” typically utilized by institutions. Unfortunately, there is no single widely recognized definition for either term. As stated in Deardorff (2011), “Fantini (2009) found a variety of terms being used [to refer to the latter], both within the literature and in regard to assessment tools. Among them are multiculturalism, cross-cultural adaptation, intercultural sensitivity, cultural intelligence, international communication, transcultural communication, global competence, cross-cultural awareness, and global citizenship” (p. 66). Possible explanations for the variety of terms used are provided by Green (2012): “The terms international, intercultural, and global are sometimes used synonymously, in spite of their differences. In other cases, value judgments are ascribed to particular terms, such as globalization , causing people to avoid using the term. Conversely, some prefer globalization to internationalization , ascribing more sweep and currency to the former” (p. 2). [Italics in the original]. Hovland (2014) and Whitehead (2015) observe that institutions use this language to promote a variety of goals, including a social justice mission, a worldview focused on the increased complexity and interconnectedness of the world community, greater levels of student engagement that can be achieved from the implementation of this “high-impact practice,” or to emphasize the new expectations and realities that graduates will face in their careers. Furthermore, Hovland (2014) argues that an institution’s understanding of what it means by these terms is enhanced by—and likely motivated by—discussions connected to the development of comprehensive student learning assessment processes and procedures.

For many institutions, this discussion around student learning assessment is occurring in conjunction with their consideration of the AAC&U Essential Learning Outcomes as defined by their LEAP initiative (Association of American Colleges and Universities 2013 ) . Hovland (2014) observes that while the LEAP outcomes do not explicitly reference global learning, “they are consistent with national calls for innovative curricular and cocurricular designs to advance such learning” (p. 5). Utilizing the phrasing provided in the articulation of the LEAP outcomes provides a framework for proposing a definition of global learning:

Knowledge of human cultures and the physical and natural world…focused by engagement with big questions; intellectual and practical skills…practiced across the curriculum; personal and social responsibility…anchored through active involvement with diverse communities and real-world challenges; [and] integrative and applied learning…demonstrated in new settings and in the context of complex problems. (Hovland 2014, pp. 5–6)

Whitehead (2015) notes that the use of the term global learning “shift[s] attention to a greater focus on issues connecting the United States to the rest of the world. … [G]lobal learning focuses on issues that can be examined by all disciplines and that affect individuals all over the globe” (p. 9). Similarly, Deardorff (2011) proposes a definition for intercultural competence that focuses on the “effective and appropriate behavior and communication in intercultural situations, which … can be further detailed in terms of indicators of appropriate behavior in specific contexts.” (p. 66) [Italics in the original.]

Transformative Learning

Organizations typically develop diversity and inclusion (D&I) initiatives in order to foster this kind of intercultural competence among its members. The intent of this focus is the belief that developing this kind of competence will enable the organization to market itself successfully across a variety of cultures, improve the management and productivity of an increasingly diverse workforce, develop a respectful climate within the organization, and increase the recruitment and retention of underrepresented groups. Achievement of this competence implies displaying interculturally sensitive behaviors as well as possessing an intercultural mindset and skill set that allows one to operate effectively in cultural contexts outside one’s own. Thus, intercultural competence is achieved through the development of appropriate knowledge, attitudes, and behaviors (Bennett and Bennett 2004; Leask 2015).

For educators charged with fostering a global perspective and intercultural skills and abilities within their students, the question remains: How best to achieve this? For many educators, this question has been answered most effectively by focusing on creating transformative learning opportunities for students based on the ground-breaking work of Mezirow (1991). As Killick (2015) notes, “[t]he kinds of personal change which are implicated in developing the global self make ‘ perspective transformation’ a helpful construct” (p. 101) for developing the global student. This approach advocates that meaningful intercultural learning is facilitated through a “transformative” experience: Students face a “disorienting dilemma” that initiates critical reflection, motivating dialogue with others who have negotiated a similar change, which results in students putting this new, transformed perspective into action in their lives through newly redefined roles and relationships (Mezirow 1991). “[Mezirow] eventually named this process perspective transformation to reflect change within the core or central meaning structures (meaning perspectives) through which we make sense of the day-to-dayness of our experiences. Perspectives are made up of sets of beliefs, values, and assumptions that we have acquired through our life experiences” (Dirkx 1998, p. 4). [Italics in the original.]

Also noteworthy is the observation that Mezirow’s original research focused on women reentering higher education via community college enrollment in the 1970s and thus is particularly relevant for educational programs for working adults (Dirkx 1998). In addition, transformational learning provides opportunities to create what educators refer to as student-centered learning , an approach that values student experience, interaction, and self-reflection as mechanisms for fostering both intellectual and personal growth. This approach enables educators to build a learning environment that can explicitly incorporate and value the differing emotional responses, life experiences, and cultural perspectives that a diverse student body represents (Killick 2015).

Creating and Measuring the Transformation

Central to the transformative process envisioned by Mezirow (1991), however, is the first step: experiencing some sort of “disorienting dilemma” which propels the self-examination and critical evaluation of one’s prior beliefs, assumptions, and perspective in order to achieve the transformed perspective and world view. Thus, educators are challenged to create an environment where students are pushed into this kind of self-evaluative cycle. Experienced educators will observe that while it might be relatively straightforward to place students into situations that “push them,” caution must be taken in order to do so in such a way to motivate the students to do the hard work implied by Mezirow’s transformative steps. The situation must be personally compelling to them, something that they see as worthwhile and worth doing. It must be relevant to their daily lives, and something they see as significant or meaningful to their futures. As Dirkx (1998) observes:

Knowledge … arises within the social acts of trying to make sense of novel experiences in the day-to-dayness of our lives. To be meaningful, what is learned has to be viewed as personally significant in some way; it must feel purposive and illuminates qualities and values of importance to the person or group. … Through educative experiences learners engage and confront novel situations which question their existing assumptions, beliefs, values, or images of themselves or the world. (pp. 9–10)

Killick (2015) suggests that this sort of personally compelling scenario can be achieved in a cross-cultural experiential context. Indeed, it is in such a setting that one is likely to experience “the shock of cross-cultural contact , the crisis of engagement, that stimulates the learning necessary for intercultural literacy … Without cross-cultural contact, the learning can only ever be about another culture” (Heyward 2002, pp. 15–18, as quoted in Killick 2015, p. 158).

How, then, does the educator know if the transformation had occurred, and to what extent does it reflect the goals of global learning and intercultural competency? A widely used approach for assessing this kind of transformation is provided by Milton Bennett’s Developmental Model of Intercultural Sensitivity, or DMIS (Bennett 1993), which forms the basis for the widely used Intercultural Development Inventory (IDI), an assessment tool for intercultural learning developed by Mitchell Hammer. “More than 1400 Qualified Administrators [of the IDI] in more than 30 countries have extensively applied the IDI in academic and nonacademic contexts. In addition, IDI-related literature is rapidly expanding and currently consists of more than 60 published articles and book chapters as well as over 42 PhD dissertations” (Hammer 2012, p. 117).

The DMIS model was created to provide a framework for explaining students’ experiences as they confronted cultural difference to become more competent intercultural communicators. A person’s development of increasing levels of intercultural sensitivity progresses through six stages, divided into two categories: the initial ethnocentric stages of denial, defense, and minimization, followed by the ethno-relative stages of acceptance, adaptation, and integration (Bennett 1993; Bennett and Bennett 2004). The model is not intended to describe changing attitudes and behaviors, but rather focuses on a person’s cognitive development , which then manifests itself in terms of observed attitudes and behaviors. “The underlying assumption of the model is that as one’s experience of cultural difference becomes more sophisticated, one’s competence in intercultural relations increases. Each stage is indicative of a particular worldview configuration, and certain kinds of attitudes and behavior are typically associated with each such configuration” (Bennett and Bennett 2004, p. 152). [Italics in the original.]

The IDI questionnaire is constructed to provide results that can be mapped along a metric based on the DMIS, referred to as the Intercultural Development Continuum (IDC) . Briefly, the IDC is based on the first five stages of the DMIS (the sixth, integration, is omitted as it is a description of a person’s cognition development after intercultural competence is attained rather than a characterization of its development‚ which is what the IDI is intending to measure). If a person’s results lie within the first three ethnocentric stages , they are described as having a monocultural mindset . If a person’s results place them within the last two stages, they are considered to possess an intercultural mindset .

A common setting for utilizing the IDI tool is the assessment of learning gains for students who participate in study abroad experiences, both short-term (1–6 weeks in length) and those of longer duration. As noted by Hammer (2012), “[p]olitical, business, and international education leaders often support study abroad opportunities based on the view that immersion in another culture will lead to students increasing their intercultural competence —their capability to shift cultural perspective and adapt behavior to cultural context” (p. 124). Despite this long-standing and pervasive assumption, Hammer (2012) further observes that “typical” study abroad experiences do not necessarily create increased intercultural competence (as measured by the IDI) as:

… being in the vicinity of an event in another culture does not mean that one has an intercultural experience merely by being exposed to it. For many students, being immersed in a foreign culture does not necessarily demonstrate that they are learning how to shift cultural perspective or adapt behavior; even those enrolling in programs of longer, rather than shorter, duration are, on average, showing only marginal gains in intercultural development when left to their own devices. (p. 126)

In response, Hammer (2012) outlines what he refers to as the “IDI Guided Development,” which is a set of suggested curricular and programmatic components that educators can use to structure and organize their study abroad programs. Many of these programmatic components are designed to provide students an opportunity to experience what we have earlier described as stages within the transformative learning process, particularly those focusing on reflection and critical self-evaluation. For example, “[c]ultural mentoring … involves guided reflection on the students’ cultural experience [and] is a foundational development strategy of IDI Guided Development” (Hammer 2012, p. 130). In addition, Hammer (2012) observes that “unexamined cultural experiences do not facilitate intercultural competence development. Rather, experience plus cultural reflection result in greater cultural insights and increase students’ intercultural competence” (p. 131). [Italics in the original.] Furthermore, empirical evidence suggests that these specific components increase intercultural development as measured by the IDI (Hammer 2012).

Given this evidence, one might postulate that the effectiveness of these components is at least in part a consequence of the extent to which they replicate the key features of transformative learning . However, as stated earlier, the most important feature of the transformative learning process is the initial “disorienting dilemma” that motivates the transformative experience for the student. This is not emphasized (or explicitly mentioned) in the IDI Guided Development framework. Some of the components that are included—e.g., “becom[ing] involved in the day-to-day life of host country nationals” (p. 132)—might create these kinds of experiences, but they do not appear to be explicitly designed to do so. Transformational learning is predicated on experiencing some sort of “disorienting” event and merely living in a new environment and “reflecting on it” does not guarantee this kind of experience will occur. Further, while the use of the term “disorienting” might suggest that the dilemma would be an unwelcome or unpleasant experience, as Killick (2013) notes, the “triggering ‘dilemma’ may include ‘positive, joyful incidents’—‘events that are fulfilling rather than distressing’” (Brookfield 1987, p. 31, as quoted on p. 184). Thus educators seeking to cultivate transformational learning can do so by providing opportunities for students that are stretching, challenging, and personally fulfilling.

An increasingly popular phenomenon within both secondary and post-secondary education is the emergence of international service learning (ISL) , which combine aspects of short-term study abroad programs with a service learning or community service focused experience. These sorts of experiences provide students opportunities to expand their international awareness while actively engaging with a local community in ways that they find meaningful and personally rewarding (Lewin 2010). To the extent that these experiences also provide students an emotionally engaging and potentially challenging learning opportunity, they can also provide students the impetus necessary for them to begin the process of transformational learning. Clearly, integrating what Hammer (2012) described as “cultural mentoring” and the reflective consideration of the students’ experience that this entails would enable those participating in an ISL experience to move through the stages of transformational learning. Also important would be providing opportunities for students to translate their experiences and evolving perspective into action, for example, upon return to their home communities.

As noted earlier, capstone projects completed during or after an international experience also provide educators an opportunity to measure global learning and gains in intercultural competence (Gordon 2013). Integrating cross-cultural interpersonal interactions within these kinds of project-based experiences (e.g., building teams with a diverse group of students, either from the home institution or in coordination with institutions in other countries) can provide students numerous opportunities to identify and resolve the challenges inherent in complex project-based learning . When travel is either logistically difficult or cost prohibitive, Leask (2015) suggests a variety of ways that information and communication technologies can be utilized to facilitate cross-cultural interaction, either synchronously or asynchronously. Examples include inviting an expert in the field from another country to lead a discussion on an engaging international topic, asking students to interview students from other cultures or those located in other countries, and facilitating “mixed-culture online tutorial groups which examine ways in which particular cultural interpretations of social, scientific, or technological applications of knowledge may include or exclude, advantage, or disadvantage people from different cultural groups” (p. 84). In order to obtain a tangible direct measure of a student’s achievement of intercultural learning, a student’s reflective writing produced as a consequence of any of these experiences can be assessed utilizing a rubric based on Bennett’s DMIS. One such example was mentioned earlier: the Intercultural Knowledge and Competence VALUE rubric, which was developed to utilize the framework outlined in Bennett (2008).

Conclusion

In response to the increasingly diverse and multicultural communities extant within higher education and the world in which our students live and work, educational leaders have transformed their educational missions to include goals designed to develop global/international awareness and intercultural competency among their students, to create graduates who are responsible and effective global citizens. “Internationalization now features as a goal in the mission statements of universities around the world, and internationalization debates no longer occupy the periphery; they have firmly moved into the mainstream” (Ryan 2013, p. 1). As a result, educators across the globe, already challenged to fulfill prior expectations for learning specific to their discipline and/or institution, are now asked to also foster a global perspective and intercultural skills and abilities within their students.

Not surprisingly, much attention is being directed at how best to create learning opportunities for students that will fulfill this goal and do so in ways that can be measured so as to ensure that the learning is both meaningful to the student and realizes the expectations outlined in the institution’s mission. Many have observed that this kind of learning is “transformational” by its very nature, and thus, the literature describing and providing measuring tools appropriate to transformational learning as outlined by Mezirow (1991) and others provides a useful framework for designing, implementing, and assessing these new teaching and learning strategies. Measuring tools based on Milton Bennett’s Developmental Model of Intercultural Sensitivity (Bennett 1993) provide useful behavioral and attitudinal constructs for measuring the development of increasingly more intercultural and global perspectives within students.