Immersive Learning Environments

The capabilities and possibilities of being able to create educational experiences that are immersive in nature have been growing and evolving throughout the past decade. Immersive education, which can be defined as learning through environments that give participants a sense of ‘being there’ even when physical presence is not possible (Gardner and Elliott 2014), can be achieved through a variety of mixed reality experiences. Growing capabilities of computing systems, enhanced sensor and software development technologies, and simpler interfaces to create and interact with Virtual Learning Environments (VLEs) systems have driven the adaptation of virtual, augmented, and mixed reality (VR, AR, and MR, respectively) in a variety of learning environments. Likewise, affordability, encouragement for creative learning methods, and wide acceptance of VLEs as engaging tools for learning are some of the social factors driving the growing use of VLEs. These environments have been associated with opportunities to enhance community, presence, and authentic situated learning experiences. According to Dawley and Dede (2014), the range of VLEs, which they also refer to as immersive environments, align with constructivist learning theory (Vygotsky 1978), as they position the learner within imaginary or real-world contexts that facilitate participatory and metacognitive learning processes.

This paper will explore the learning affordances and potentialities of an immersive, mixed reality environment that is under development at Virginia Tech through the Institute for Creativity, Arts and Technology (ICAT). The Mirror Worlds (MW) project, which combines virtual and physical realities with upcoming developments in gesture and full-bodied interfaces is an experiment in fusality which involves the fusion of real and virtual realities (Polys et al. 2015). This paper will, therefore, (a) describe the unique attributes of the Mirror Worlds project that are aimed at engaging learners in active, authentic and embodied learning environments and (b) identify the potential learning affordances of such immersive technological environments. Due to the complex, multi-faceted nature of the project, and its ongoing development, it is difficult to specifically map out features and translate them to different environments; however, for the sake of illustrating potentialities of this system, project information can be accessed at the following website: http://icat.vt.edu/impact/project/mirror-worlds. The impact of this project has the potential to contribute significantly to the literature on immersive learning and mixed reality environments in education and training due to its capabilities of connecting people through multiple realities and representations.

Mirror Worlds Defined

The Mirror Worlds project represents an exploratory leap for the future of VLEs. Funded by a National Science Foundation grant, several researchers affiliated with Virginia Tech’s Institute for Creativity, Arts, and Technology (ICAT) have built the infrastructure necessary to support a large scale interactive environment which can digitally represent groups of people as well as objects and locations in real time.

The Mirror Worlds system consists of multiple sensors built into the physical environment. The entire Moss Arts Center building at Virginia Tech is used in this case (Polys et al. 2015). Several sensors are placed strategically around the building to capture motion, position, sound, and heat from entities present in the physical environment. There is area specific sensing with varying numbers of sensors placed in the gallery, lobby, rooms, and corridor within the building. The data produced from the sensors are sent across to a server in real-time, which are then processed and displayed on monitors as seen in Fig. 1.

Fig. 1
figure 1

Technology overview of the Mirror Worlds setup at Virginia Tech. (Polys et al. 2015)

Full size image

The project development team had previously created a 3D model of the building upon which the processed data are appended. Hence, the monitors display 3D models of the building along with the real-time position, motion, and sound detected by the sensors. There are several displays placed within the building for local people to access and interact with. Additionally, virtual displays and the web technologies support non-local users to interact with the system. This way, both local and remote users share the space, events, and corresponding interactions.

Up to this point, mirror worlds have predominantly been fodder for highly theoretical forecasts rather than the empirical research that can be now be undertaken given the large-scale instance in place at VT. Originally coined by Gelernter (1993), “Mirror Worlds” were described as a real-time mapping of real world objects onto a software system. While this argument is useful in differentiating other virtual representation from mirror worlds, it does not address the issue that typically arises with mirrors - the perceptive, social interaction between the physical world and the “mirrored” view especially by the person whose view is being reflected. Hence, we extend Gelernter’s view to define Mirror Worlds as the real-time digital representation of real world objects and the interactions between those objects. The digital representations may be of varying fidelity but they, to some extent, represent the truthfulness of real-time interaction between the objects present in the real world.

Gelernter knew that his vision was many years away from feasibility when he initially coined the term. The existing literature lists certain qualities of mirror worlds. Tilden et al. (2011) defined existential correspondence, ontological correspondence, spatial correspondence, temporal reflection, and persistence as the qualities required of Mirror Worlds. Existential correspondence implies that real world people should exist in a certain form in the digital world. In the case of the Mirror World system at ICAT, there can be various representations of people. The representation can be ephemeral to emphasize physical movement, as well as permanent blobs, which represent individuals as seen from the top-view. Likewise, ontological correspondence requires that the objects and attributes from the real world are represented in the virtual world like the real-time point-cloud system developed by Hong et al. (2009). This representation can have varying degrees of fidelity but the objects should be perceivable by the user. In the ICAT installation, as seen in Fig. 2, the objects such as tables and chairs are represented with a high-degree of fidelity. Likewise, spatial correspondence refers to accuracy with which the physical object’s orientation and position are reflected in the virtual world. In Fig. 1, tables, chairs, and doors are represented as they are in the rooms at ICAT. By temporal reflection, we mean that real time events that may occur in the physical world have to be reflected in the virtual world. In the ICAT installation, the blobs change as people move. Like the physical world, the virtual world has to exist even in the absence of users, which Tilden et al. (2011) term as the persistence quality of Mirror Worlds. The ICAT installation runs perpetually and provides real-time representation of events when they occur.

Fig. 2
figure 2

Two different possible representations of people and objects in the ICAT Mirror Worlds installation. The left image shows people as ephemeral blue blobs whereas the right image displays a permanent view

Full size image

In addition, the installation at ICAT supports simultaneous immersion and digital representation of multiple locations of learners. This supports social interaction of physically separate people in the digital world. We have identified that capturing social interactions is an important requirement of Mirror Worlds. We term this quality as social correspondence. This social correspondence quality, along with the capabilities to bring in other layers of interactivity and connect multiple mirror world participants, provides exciting possibilities for the future. The current MW instance reflects the fusion of real and virtual realities, and has been developed to operationalize the concepts of immersion and fusality in many different ways. The complexity of this environment provides many avenues for varied and rich opportunities for social connection and authentic interactions. Fusality can be seen through the following opportunities for interaction in the Virginia Tech Mirror Worlds VLE.

Fusality Defined

Given the complexity of the infrastructure of the MW environment, the following paragraphs will further define and illustration the connections that can be made, and the interactions that can occur, across the real and virtual worlds.

People and Objects in Real World Mapped into Virtual World

As previously described, the Moss Arts Center at Virginia Tech, in which the Mirror Worlds project is located, is equipped with sensors that translate physical objects and persons in the environment into the virtual world, hence providing opportunities for authentic, location- specific activities to be translated into the virtual world. As described by Polys et al. (2015), Mirror Worlds is manifest through “sensory locales, which populate the physical space to serve as bi-directional conduits for interactions and events between occupants of both the real and virtual spaces” (p. 171). This capability, which is a departure from typical virtual worlds that exist unconnected to the real world, provides a unique capability of the MW environment to immersive users in a connected, and hence authentic environment.

Participants Connect from Remote Locations

In addition to enabling location-specific interactions between the physical and virtual worlds, the virtual world provides opportunities for those located elsewhere to connect and participate in the Mirror Worlds interactions. As with most virtual environments, this capability enables connections amongst participants who are not physically co-located. However, in this instance, participants connecting from elsewhere become privy to virtually rendered spaces that more closely emulate the physical environment. Participants from remote locations are represented as avatars within the MW system. The system enables different types of avatars with varying levels of representational fidelity.

Tangible, Gesture-Based Interactions

Beyond blending the physical and virtual worlds in a mirrored environment, the notion of fusality within the MW environment is becoming operationalized through the use of tangible, gesture based and full-body interactions. While currently under development, the use of tangible and full body interactions allows users to participate in this environment in novel and unique ways. Providing participants with the ability to interact with physical materials via tangible and full-bodied interfaces creates connections to embodied forms of learning and cognition (Eisenberg and Pares 2014).

Mirror Worlds and Learning

Given the multiple avenues for interacting within the mixed reality mirror worlds environment, there are multiple pedagogical and theoretical possibilities present when considering learning affordances within such a complex and rich environment. To add social correspondence to Tilden et al.’s (2011) list of the qualities of mirror worlds opens up the potential of these environments to facilitate rich socio-cognitive learning. The following paragraphs will therefore explore some of the ways in which this environment has the potential to provide unique opportunities for rich and authentic interactions, which in turn could lead to the development of engaging and interactive learning experiences. Although not a comprehensive list, attempting to connect learning affordances with specific functions and features of innovative environments lays the groundwork for future empirical inquiry while providing fodder for considering implications of the development of mixed reality environments.

Social Correspondence

Social correspondence can be best understood in terms of learning affordances through a situated cognition and social constructivist framework. While theories of situated cognition argue that learning is best achieved when knowledge is situated within authentic contexts (Brown et al. 1989), constructivist learning assumes knowledge is actively constructed by the learner (Doolittle and Camp 1999; Driscoll 2005; Mayer 1999). Both theoretical constructs, therefore, contend that students construct mental representations of the world around them by using their knowledge in real situations (Doolittle and Camp 1999; Driscoll 2005; Mayer 1999). Through personal experiences, social interactions and metacognition, students learn to reason, think critically and build understanding (Driscoll 2005).

In general, constructivist learning is concerned with developing knowledge that is useful, not learning for the sake of learning. What is learned should have a practical application in the world we live in (Doolittle and Camp 1999). Constructivist learning needs an authentic context to demonstrate to students the practicality and application of what they are learning. Students should have opportunities to work with their peers to understand concepts by building knowledge from prior experiences and experiences shared by others. Finally, reflection activities will improve students’ metacognitive skills.

Applying the constructivist approach to Mirror Worlds situates the learning environment in social and intellectually challenging domains. The constructivist framework can provide structure for instructors designing learning activities in Mirror Worlds. Through multiple connection points in real and virtual worlds, Mirror Worlds can also make unique, expensive, or location based authentic activities available in an affordable manner with a unique interactive component courtesy of the distinctive immersive experience. Interactions provide an outline for social building of knowledge and construction of problem solving. Sharing thoughts and interacting with others in the learning environment provides a discourse that allows students to make connections to authentic situations.

Presence

Presence is a multifaceted concept, defined in a variety of ways by different authors. Seminal authors Short et al. (1976) defined it as “the degree of salience of the other person in a mediated communication, and the consequent salience of their interpersonal interactions” (p. 65). Presence can perhaps be summed up in normal terms as a feeling that someone else is really “there” in an interaction.

In order for learners to effectively interact in environments in which they actively construct knowledge, they must feel fully present, or a sense of ‘being there’, despite being connected virtually. Therefore, the construct of presence also plays an essential role in any constructivist approach (Huang et al. 2010). Construction of knowledge requires ownership and personal connections be made during the experience of learning and in some cases doing something as part of the learning process. To make connections and be part of the overall experience, students need to feel as though they are truly a part of the virtual experience.

There are a variety of factors that are said to contribute to feelings of social presence. For instance, immediacy, initially coined by Wiener and Mehrabian (1968), and intimacy, as was initially described by Argyle and Dean (1965) are two factors that are prevalent in the presence literature. Immediacy is a term that seems to be used in multiple ways, with some authors referring to immediacy in the temporal sense as a property of synchronous interactions at a distance. But according to Tu & McIsaac, immediacy is “the psychological distance between a communicator and the recipient of the communication” (Tu and McIsaac 2002, p. 134), and is barely distinguishable from intimacy (in the educational sense). Intimacy, on the other hand, is seen as a combination of eye contact, physical proximity, facial expressions, and other factors in a face-to-face conversation, all of which must exist in equilibrium. If one component is changed, the others must adjust accordingly in order for the participants in the interaction to remain comfortable.

The potential for the Mirror Worlds environment to facilitate levels of presence, which have been shown to have predictive validity on teacher-student relationship satisfaction, class satisfaction and perceived knowledge gain (Kim et al. 2016), is high. The different types of interactions across real and virtual environments provide opportunities for participants to experience presence in different ways. For instance, learners connecting at a distance are able to experience the effects of immediacy by being able to participate in all local interactions in real time, as participants in the physical location are. The distance between the teacher and the student can be closed if verbal immediacy is taken seriously (Baker 2010). Immediate behaviors include asking questions, telling reflective personal stories, using language that is inclusive (we and us), offering praise and communicating attentiveness. Synchronous activities within the learning environment also support students’ belief of instructor presence (Allmendinger 2010; Baker 2010). If students feel the instructor is present in the learning environment, students have positive feelings about their learning and show greater cognition and motivation (Baker 2010; Kožuh et al. 2015). Social correspondence is therefore operationalized within the Mirror Worlds environment within a social constructivist paradigm that allows for participants to experience presence through a greater sense of intimacy and immediacy.

Embodied Learning Via Tangible and Full-Bodied Interfaces

Beyond what we have defined as social correspondence, the MW is being developed in alignment with Tilden et al.’s (2011) qualities of existential correspondence, ontological correspondence, spatial correspondence, temporal reflection, and persistence through the use of tangible and full-bodied interfaces. Beyond what most virtual environments offer in terms of learning affordances, the MW interface will soon boast the capabilities of tangible and full-bodied interfaces. The capability to use gestures and body language supports cognition (Allmendinger 2010), which therefore adds a layer of functionality that has the potential to impact learning and performance in this virtual world. Microsoft Kinect and Leap Motion-based sensors, which recognize hand and body positioning, allow users to interact with object within the environment with a high level of fidelity and accuracy. Although most virtual worlds are credited with their capabilities to provide embodied social interactions within the environment via their avatar (Dalgarno and Lee 2010), the MW environment allows users to act and interact via physical movements and gestures. This layer of interactivity, which goes beyond most virtual world capabilities, provides for a type of interaction that opens up the potential for different types of learning to occur.

Goldin-Meadow (2003) contends that the act of gesturing continuously informs and alters verbal thinking and is part of a coordinated activity that has been selected for its specifically cognitive virtues. Feedback in the form of gestures provides cues to anticipate important information or provide a visual representation to aid in connecting new ideas. Gestures also add interest and stimulate attention during discussions between peers or groups. This type of feedback is in essence scaffolding learning. Therefore, the role of gesture-based computing can be a significant enhancement to environments that have been constructed with social correspondence in mind.

In Mirror Worlds in particular, avatars that reflect real-time gestures support natural scaffolding while creating a strong sense of presence that will support learning in the environment. Body language and facial expressions further support a sense of presence and improve communication and collaboration needed for social construction of knowledge. Emotions and attitudes are shared through this non-verbal communication mode. As noted earlier, a personal connection brought to discussions in online learning environments can promote a sense of presence. Personal connections such as gestures, body language, and facial expressions also provide feedback of attention, interest, and scaffolding (Allmendinger 2010). These communication aids have the potential to create a natural sense of collaboration thus making social interactions stronger and supporting elements that are characteristic of a constructivist learning environment.

Implications and Future Directions of Mirror Worlds

As has been explicated in the previous paragraphs, the Mirror Worlds infrastructure holds the potential to provide students with learning opportunities in an immersive, mixed reality environment that extends beyond traditional VE or VR environments into an environment where participants interact within a variety of mixed real and virtual spaces.

The sense of presence students are afforded through the variety of features in the environment supports collaboration, making it feel like a joint effort (Baker 2010; Hill 2012; Miyake and Kirschner 2014). To make collaboration successful in Mirror Worlds, instructors need to plan activities that will support communication not only between the instructor and students but also between students themselves (Baker 2010; Huang et al. 2010; Kožuh et al. 2015). Aids for communication that cue distance learners’ attention and nonverbal communication will help facilitate collaboration in a natural sense (Allmendinger 2010).

In face-to-face learning environments, nonverbal communications play a strong role in communicating ideas, sharing concepts for co-construction in the collaboration and provide an avenue for informal personal reflection. In a learning environment such as Mirror Worlds, nonverbal communication can still play a leading role in the collaboration of the community of learners. Mirror Worlds holds the technological potential to provide realistic avatars that simulate actual space in real-time and affords distance users active real-time engagement with other learners. Avatars that act and react as a true reflection of the person it represents provides a higher sense of immersion and sense of social presence (Allmendinger 2010; Grinberg et al. 2014). If a Mirror Worlds avatar can mimic gestures, facial expressions, and reaction patterns that are true to the user, the sense of immersion also supports similar nonverbal communication cues naturally observed in face-to-face learning environments. With this technological advantage, Mirror Worlds brings an element of communication that builds community and belonging within a collaborative group of learners.

There are three areas in which Mirror Worlds can expect to experience near-future growth as the infrastructure allows: higher fidelity representations, multiple connected physical spaces, and incorporation of other associated technologies. As mentioned above, increasing the fidelity of the digital representations within Mirror Worlds should increase learners’ sense of presence - their perceived connection to each other and environment. As Mirror Worlds infrastructures continue to develop and mature, it is reasonable to expect much more realistic digital depictions of space and individuals, thus increasing presence and, with it, student satisfaction, engagement, and motivation. Decisions can also be made to focus learners’ attention on certain aspects of the digital environment by increasing fidelity on some objects while not increasing it on others, or in terms of choosing fidelity levels based upon desired instructional outcomes (McMahan et al. 2012).

Researchers can also expect further developments in Mirror Worlds’ effect on social correspondence and presence once multiple physical spaces are connected in the same digital environment. Presence, then, becomes a construct not just involving groups of co-located learners in digital spaces, but also learners who are sharing digital space but not physical space. Questions to ask at that point include how connected learners feel to those who are not co-located with them, as well as how or whether learners feel a part of the entire class, not just the portion in their shared physical space.

Finally, Mirror Worlds can expect advances as other technologies are incorporated. Already in development and mentioned above, Microsoft Kinect and Leap Motion-based sensors can recognize hand and body positioning while AR technologies like the Microsoft HoloLens provide exciting potential as wearable devices that may allow students to see digital representations of people/objects layered on top of their view of the real world.

With the goal of establishing a sense of presence through natural communication, reflective avatars, and the feeling that an instructor is also available in the learning environment, Mirror Worlds has the potential to bring hands-on activities to students that span classroom spaces. A characteristic of constructivist learning is using the knowledge gained in activities that make the knowledge real and tangible. In these authentic activities, students are given opportunities to practice their knowledge and make it their own while working with others to improve and expand on their understanding and transferability of knowledge (Hill 2012). Mirror Worlds’ capabilities can allow activities that are resource heavy, expensive, or location-based to not only be shared with other learners but for those learners to also be involved in the activity. Distance learners may be able, through the Mirror World learning environment, to work alongside peers to build a prototype for a learning activity as if they were sitting at the table with the same tools and resources as their co-located peer. Mirror Worlds can bring a true sense of immersed learning through engagement and involvement to hands on activities by removing the passive feeling of watching activities occur at a distance or simulated through vicarious learning experiences.