Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

The education of school teachers usually involves a capstone experience, called an internship, where each has the opportunity to deliver instruction to real children either as the sole instructor or in an apprentice role. This puts children (and prospective teachers) at risk while skills, especially soft skills associated with human-to-human interaction, are still being learned. Virtual environments provide an opportunity for teachers to practice without interfering with the education of children and without placing themselves at the mercy of these same children while their classroom management, pedagogy and even content skills are still developing. Unfortunately, purely virtual worlds (ones driven by programmed behaviors) are not yet adaptive enough to provide realistic responses to verbal and non-verbal interactions, and are not capable of handling the almost random directions that the conversations may or should take. Similarly, the demands imposed by virtual worlds that deal with protective strategies, e.g., for college freshmen or pre-teens facing enormous peer pressure, and other applications that need to reflect the subtleties of human interaction, are not presently met through programmed behaviors, even those encompassing evolutionary changes in those behaviors.

This chapter focuses on how a human-in-the-loop paradigm can provide the realism needed to address the areas mentioned above, as well as other applications involving intense human-to-human verbal and non-verbal interaction. Specifically, we focus first on TeachLivE (Teaching and Learning in a Virtual Environment), a system in current use at over 55 universities and four school districts in the U.S. The TeachLivE system provides in-service and pre-service teachers the opportunity to practice skills and reflect on their own performances. The reflection component is achieved through an integrated after-action review system that supports automated and semi-automated tagging of events, along with the use of these tags to select video sequences that demonstrate the teacher’s performance in different contexts. The system’s scalability is enhanced by a micropose-based network protocol and the use of just one human inhabitor, called an interactor, to orchestrate the performances of multiple avatars (human surrogates) and agents (software controlled characters).

In addition to teacher rehearsal, this chapter discusses the uses of avatar-mediated interaction for development of protective strategies for the wellness of self and others, and the employment of the underlying virtual settings for other complex interpersonal interactions such as interviewing prospective employees or being interviewed by prospective employers. Finally, we discuss how the system can prepare people for the complex task of leading debriefing sessions; in effect, showing how the paradigm can be used to train people on how to effectively use reflection, one of the most useful features of our system called AMITIES (Sect. 2).

2 The AMITIES Framework

AMITIES™ is an acronym for Avatar-Mediated Interactive Training and Individualized Experience System. This is the technical framework on which all our avatar-based systems are built. AMITIES supports a blend of agent-based and avatar-based behaviors. AMITIES encapsulates all the software components, including device interaction, needed to remotely deliver, receive and observe avatar-based role-playing experiences. This infrastructure permits the creation of a broad range of detailed realistic and compelling interactive scenarios for training, education, wellness, and similar areas. AMITIES also includes components to help its users reflect on their performance (e.g. for after action review), with or without a coach providing guidance during such reflection.

All experiences presented in this chapter are built on the AMITIES framework. Each involves scenarios that include virtual characters whose personalities are set, but whose actions are heavily influenced by the behaviors of users who take part in the experience. For example, a virtual character may proactively make a personalized comment (such as requesting details about a user’s profession) to increase engagement of the seemingly less-involved user during conversation. A key to the AMITIES paradigm is the use of an interactor who orchestrates the interaction of a troupe of avatars and agents in their communications with one or more human users. Technical details of the AMITIES framework are presented in Nagendran, et al. (2014). For our purposes, the brief description above is sufficient.

3 Teacher Education: Experiences in TeachLivE

TeachLivE™ stands for Teaching and Learning in a virtual Environment. This refers to the virtual classroom environment that we use for preparing teachers for the challenges of working in K-12 classrooms. TeachLivE is supported by the underlying infrastructure of AMITIES (described above). Its primary use is to provide teachers the opportunity to rehearse their classroom management, pedagogical and content delivery skills in an environment that neither harms real children nor causes the teacher to be seen as weak or insecure by an actual classroom full of students. TeachLivE also uses the AMITIES framework to support reflective learning following an experience in the virtual classroom.

3.1 TeachLivE Overview

Consider the situation of walking into a classroom setting where the students are virtual and yet exhibit personalities that are appropriate for members of their age group (Dieker et al., 2014). This is what a TeachLivE experience provides, currently supporting classes of middle (Fig. 1) and high school students, with elementary students coming in fall 2015.

Fig. 1.
figure 1

TeachLivE middle school classroom

Full size image

The members of these classes have personalities that are diverse enough to represent the population of a typical classroom, even though there are only have five students in each such setting. The basis for the selection of personalities comes from the research of Long 1988. Essentially, we have two major pairs of factors: aggressiveness and passiveness, and independence and dependency. This provides four personality types. When we address the middle school classroom students (Fig. 2), we find that Ed (first row, left side) is a passive dependent, who wants your approval but is far too polite to push for it. Sean (first row, right side) is an aggressive dependent whose need for approval by the teacher leads to his wanting to answer nearly every question, whether posed to him or not. Maria (second row, left side) is a passive independent who is an aspiring engineer with no self-perceived need for your assistance or attention. CJ (second row, middle) is an aggressive independent who is the class leader and tends to not care about school or rules. Kevin (second row, right side) is CJ’s most loyal follower who wants her approval for everything he does. The dynamics between Sean and CJ are a bit tense as he openly disapproves of her inappropriate behaviors (e.g., texting during class) and she blatantly views him as being the ultimate uncool kid. Sean’s need for attention leads some teachers to spend all their time with him and others to angrily try to shut him up. Maria rarely opens her mouth, and so many teachers never engage her in conversation, missing out on discovering and encouraging her wonderful intellect. Ed is very easy going and has clear goals but needs some assistance especially in mathematics. Kevin also gets ignored sometimes as his focus on creating YouTube videos does not culturally connect him to some teachers.

Fig. 2.
figure 2

TeachLivE middle school kids – Ed, Sean, Maria, CJ and Kevin

Full size image

The most important thing about these characters and, in fact, all characters that exist within the AMITIES framework is that they always remain true to their personalities, backstories and needs. Thus, if you are a teacher in their classroom, your only hope for change is in your own way of approaching them, not in their essential ways of dealing with you. As one might expect, the high school versions of these characters are much less rigid, and the elementary school versions are more compliant. Thus, as in real life, the middle school classroom is the most challenging but potentially the most rewarding.

3.2 The Roles of the Interactor and User

The key to achieving consistency in the personalities of characters in TeachLivE and other AMITIES-based experiences is the use of interactors. Our interactors are like talented orchestra leaders. While the orchestra members have their own skill sets, it is this orchestration that brings harmony or, if desired, discordance to the performance. What makes interactors different from traditional orchestra leaders is that the performances they direct are interactive and unscripted. The direction of the dialogue and non-verbal behaviors (facial and body) depends on the actions, verbal and non-verbal, of the audience members (generally, but not necessarily, an audience of one). The interactors achieve this level of control in any scenario via a highly customized user-interface that is designed in accordance with their cognitive comfort during the development phase. The process itself is a very iterative one with programmers, artists and interactors all working together to achieve the best blend of control for faithful reproduction of selected behaviors.

The user (trainee teacher) has free movement in a TeachLivE experience. As he or she walks towards or away from a large flat screen display of the classroom, the virtual camera moves in ways that match the user’s actions. Thus, if the user moves towards Maria, the camera position moves towards her as well, thereby compressing space and allowing the user to move as if in the same space as the virtual characters. Bending down provides the user an eye-to-eye view of Maria. This ability to achieve proximity and intimacy provides a sense of place illusion (I’m in the same space as these students) which can also be reflective of the degree of immersiveness of the system. This is critical to the effectiveness of the experience (Slater, 2009). The mapping of the user motion to smooth camera motion is achieved via standard data filtering techniques implemented on the depth stream and skeletal data from a Kinect.

3.3 Reflection

Learning through any experience is greatly aided if the learner has an opportunity to reflect on his or her performance. Such reflection can occur alone, sometimes just mentally replaying the details of the experience, or with a coach who provides a guided replay of one’s actions, offering alternative choices that may lead to more desirable outcomes. This latter coach-assisted reflection is often called after-action review (AAR) or debriefing. One approach, called Debriefing with Good Judgment (Rudolph et al., 2007), is particularly effective when dealing with team performance but also offers excellent guidance for single-user coaching.

Independent of whether reflection is guided or not, the availability of objective data is critical to the process’s success. Our observation is that inexperienced (and often experienced) teachers who are asked if they spent an approximately equal amount of time with each student will invariably offer the subjective answer of yes and the objective answer is invariably no.

A transcript of a sample dialogue during a Math Lesson between a trainee teacher and students in the classroom is provided below:

AMITIES provides support for reflection in the form of automated and manually entered tagging of events. Automated data can include time spent in front of the class, time spent in proximity of each character, percentage of time spent in each zone of the classroom and time spent talking by the user versus that spent by the virtual characters (Fig. 3). Data that presently requires human judgment includes (i) number of high-level questions asked, (ii) number of low-level questions asked, (iii) specific praise offered to students, (iv) general praise offered to students and (v) time from asking a question to giving an answer. As we progress in our research, we are finding that we are gathering enormous amounts of data that we can now mine for other actions that correlate with success. A recent study has shown positive correlation between body posture and perceived success, which appears to also correlate to better performance (Barmaki 2014). Other studies are looking at metrics related to perceived social presence (Okita et al. 2008) and its correlation with performance (Hayes et al. 2014).

Fig. 3.
figure 3

Example of automated feedback

Full size image

3.4 TeachLivE Effectiveness

Over the year, the TeachLivE team ran a series of experiments to determine its effectiveness in conveying new strategies to teachers. Our focus was on math instruction in middle schools in the United States. These students are typically in the age range from 11 to 14 years-old. The experiment started 157 in-service teachers at 10 distinct sites across the country. Due to attrition, 22 dropped out of the experiment resulting in a final total of 135 participants.

Each participating teacher received four levels of professional development, including computer simulation, synchronous online instruction, and lesson resources based on the Common Core standards (2011). We initially observed the teachers in their classrooms, then in the TeachLivE simulation and then back in their classrooms. The goals were to see if improvement occurred in the simulation and if that improvement continued once the teachers returned to their classroom settings. The first specific skill addressed was the use of describe/explain versus short-response versus yes/no questions; here we want high-order questions that involve students in the process of analysis and thinking about their own learning processes. The second was the provision of specific versus general feedback; here, we wanted feedback that relates to the student’s actual performance rather than generalities.

Results indicated that four 10-minute professional learning sessions in the TeachLivE classroom simulator improved targeted teaching behaviors while in the simulator, and those improvements were seen in the real classroom as well. Improvements in frequencies of Describe/Explain questions and Specific Feedback across the four sessions can be seen in Fig. 4. Details of the study can be found in (Straub et al., 2014).

Fig. 4.
figure 4

Teacher practice changes over successive times in TeachLivE

Full size image

3.5 Users and Interactors

AMITIES always involves a minimum of two stations: an interactor station and a user station. The matching up of a user to an interactor occurs through a simple service architecture in which the user publishes a need for a specific scenario at a particular level of intensity and an interactor chooses to provide those services to this specific user. The matching agent also verifies identities against a database of users and interactors and, where appropriate, a time schedule previously agreed upon. This approach is taken to insure that matching is appropriate – the user has arranged for the service and the interactor has the right skill set and is approved to provide that service.

Once a match is made, the user and interactor are in direct communication, where the interactor can see and hear the user, but not vice versa. There is also a mode, where the interactor sees and hears an anonymized version of the user. That mode is only used in very sensitive training situations or as part of research projects where it is important that the interactor not be aware of the gender and race of the user.

The interactor typically orchestrates actions of all virtual characters in a gross manner, e.g., by setting the moods of groups of individuals or controlling the behavior of a specific character. Actions are mapped to those of the character in a manner where gestures are associated with poses, animations and facial expressions. The specific mapping of gestures to poses is calibrated individually by each interactor to reduce his or her physical and cognitive loads. Data associated with these poses is transmitted as a set of weights that are interpreted at the user and interactor sites to achieve pose and facial gesture displays that are consistently viewed at both sites. This lightweight protocol is a key element in achieving low lag and bandwidth requirements. We refer a reader to Nagendran et al., 2014 for further details.

Although we have discussed only one interactor and one user so far, the AMITIES paradigm actually supports multiple interactors, multiple observers and multiple users, provided one is chosen as the lead for purposes of movement of the virtual camera. This drives the scalability aspects of the system. For example, one can imagine a classroom of 10 students being driven by 2 interactors, displayed on a large projection screen. When more than one interactor is operating, these interactors typically control non-overlapping sets of virtual characters. However, there is nothing in AMITIES that limits more than one interactor from controlling a single character and, in fact, we use this capability when a master interactor is remotely coaching a novice or intermediately skilled interactor. Observers, in contrast to interactors, can only watch an experience take place. In most cases, an observer only sees the virtual scene, but hears voices of both the user and the virtual characters in order to protect the identity of the trainee. However, AMITIES can also support a special class of observers who see the user as well. This is reserved for coaches and coders who are tagging events for after-action review.

4 Other Applications of AMITIES

4.1 CollegeLivE

CollegeLivE is an acronym for College Life in a virtual Environment. It refers to the virtual role-playing environment we have created to help college freshmen develop protective strategies for self and others. These skills can be used to effectively deal with the challenging choices that these young adults face in what is often their first time experiencing independence from home and family. As with TeachLivE, CollegeLivE uses the capabilities of AMITIES, including its tools for reflective learning. Figure 5 depicts a scene from CollegeLivE.

Fig. 5.
figure 5

CollegeLivE scene

Full size image

Protective strategies in our context refer to both self-protection and protection of others. Self-protection is the ability to exercise strategies that help an individual protect themselves from dangers that arise from peer pressure. Protection of others refers to strategies where we intervene when a person other than ourselves is being pressured to do something against his or her wishes. This can be an explicit assault, but it can also be a situation where the person is not in complete control, e.g., under the influence of alcohol.

In CollegeLivE, users are placed in situations typical of parties that take place in college settings. Here, as with TeachLivE, each participant is largely in control of the directions that the experience heads. If the participant chooses to drink heavily (in the virtual environment) then opportunities evolve where they might choose to walk or drive home while still intoxicated. Most think that walking is safe, but this is not so in highly travelled areas as those that surround our university. Death and injury to intoxicated pedestrians are unfortunately far too common. Those who choose to drink only moderately or not at all are often put in situations for protecting fellow party-goers, with typical examples including the opportunity to stop others from driving or walking home while intoxicated or intervening when another is being pushed to act in ways that are counter to their best interests. This can involve sexual advances or even pushing someone to drink heavily who chooses not to do so.

4.2 Interviewing and Being Interviewed

TeachLivE has given rise to numerous other applications of the AMITIES framework. These include using the system to prepare teachers for parent-teacher meetings and principals for interviewing prospective teacher candidates (Fig. 6). In both these applications, the user’s position is not tracked as we are simulating one-on-one dialogues across an intervening table. We, however, set this so the physical table at which the user sits and the virtual table at which the interviewer or interviewee sits blend into a single physical-virtual scene.

Fig. 6.
figure 6

Prospective teacher candidate

Full size image

This same paradigm has also been used to help prepare young adults diagnosed with Autism Spectrum Disorders (ASD) for their first job or college interviews. Recent studies (Trepagnier et al., 2011, Alcorn et al. 2011) have shown the effectiveness of the use of Virtual Environments in interventions for those with ASD. Our system builds upon these results and leverages the human-in-the-loop to enhance the natural interactions during such interventions thereby keeping a human subject more immersed and engaged in the conversation. A key component of this use is the presence of a coach with strong experience addressing the needs of this community.

4.3 Debriefing with Good Judgment

Outside of the K-12 school domain, we have also employed AMITIES for helping team trainers rehearse their debriefing skills. This is particularly useful with the efforts of the US Office of Veterans Affairs to keep their trainers current in both hard and soft skills. One approach that they use is to bring trainers into their SimLEARN facility in Orlando, Florida, where these trainers are put into live simulation of emergency situations, e.g., airway crisis management. Trainers from different clinical professions are assembled as members of teams. It is their job to organize rapidly to take on the roles necessary to carry out the procedures on a patient mannequin that would, if done right, save a human life in similar circumstances. Once the simulation is done, a SimLEARN master trainer carries out a debriefing exercise intended to help each person reflect on his or her performance as an individual and as a team member. The goal is to teach a process known as Debriefing with Good Judgment (Rudolph et al., 2007). After another day or two of simulations, coaching and debriefing, these trainers are expected to return to their home bases and employ the same techniques to strengthen the technical and team skills of their fellow clinical staff members. Experience shows that these individuals are very strong on retaining the learned technical skills but that their debriefing skills deteriorate rapidly.

To meet the needs of these trainers, we have developed experiences that allow rehearsal of debriefing skills in a contextually valid environment. Here, we show the trainers a video of a team performance and then the virtual counterparts of those team members come alive in a TeachLivE-like experience, except now the virtual characters are clinical team members and the context is a conference room, as is typically used for such debriefings. Whatever personalities were observed and whatever conflicts were seen in the video directly influence how the virtual characters interact with each other. Those interactions present challenges to the trainer, e.g., soothing the ego of a nurse who feels abused by the perceived arrogance of an ER physician without disrespecting the strengths that the physician brings to the team. Figure 7 shows a sample scene from such an interactive session. As with interviews, movement of the virtual camera is not considered necessary, in fact, it is probably detrimental for the effectiveness of this type of experience. This movement is not just distracting but can be perceived to break the concept of place illusion since the point of view must be held fairly static when a human subject is seated (and not moving) in the scene.

Fig. 7.
figure 7

Debriefing members of a clinical team

Full size image

5 Future AMITIES-Based Experiences

While TeachLivE’s primary use is for teaching education, we have found this paradigm potentially useful for helping young students to develop a greater command of material and an increased confidence in their own abilities to communicate by having them become peer tutors for virtual children. While our research in this area is in a nascent state, early indications are that this approach has great promise. In particular, the human peer tutors seem to naturally show a desire to find alternative ways to present material when the virtual child (or children) does not appear to be catching on. We hypothesize that such activities build new pathways in the mentor, thereby strengthening their content knowledge and their ability to retain this knowledge.

The concept of protective strategies seen in CollegeLivE is now being employed in the preparation of people who are first-line advocates for those who are experiencing extreme stress, ranging from academic issues to physical or psychological abuse. These include teachers, counselors, law enforcement and military personnel.

Our strategies for preparing young people for job interviews can be equally effective in preparing foundation personnel for the tricky job of convincing people to become donors. As with many other multi-stage activities, job shadowing, a common approach, can be successful but is very time-consuming and may miss many complex situations that arise only rarely. In contrast, scenarios can easily be developed that cover the rare as well as the common. In fact, our university’s Foundation now uses our AMITIES-based role-playing as a regular part of their preparing and evaluating new staff members.

Debriefing exercises as described in our VA example are of use in almost all clinical settings, e.g., nurse and physician training, as well as in non-clinical activities such as project planning and even software walkthroughs.

In essence, we see an almost unlimited set of applications of the AMITIES framework and hope that the diversity discussed in this chapter will convince others of this.