1 Introduction

Teaching methods based on the principles of Behavior analysis have proven successful for improving skills, and decreasing challenging behavior in children with Autism Spectrum Disorder (ASD) [12, 31]. Such methods however, need to be correctly implemented. Otherwise the risk of reduced progress will increase [19]. It is therefore particularly critical to properly train those individuals responsible for teaching and enhancing the skills of children with ASD [34].

An important aspect of proper training is to provide the trainees with the opportunity to practice their teaching skills with students while receiving feedback from a supervisor. Unfortunately, due to lack of resources, it is not always possible to provide trainees with these opportunities [34]. For example, a proper clinic might not be a part of a university setting, making practical training difficult for university students. It is also important to consider the possible risk involved for the child in letting teachers train teaching skills on them, before demonstrating a certain level of mastery. Some of these challenges can be addressed with a computer based training program such as Dtkid [8], but an interesting alternative is to provide full situational training using Virtual Reality (VR). With high-fidelity VR technology becoming more affordable, an immersive training setup that allows the trainee to directly manipulate props and face an animated virtual child, has become a possibility.

This paper presents work on a training environment that supports teachers trying to master a teaching method called Discrete Trial Training (DTT). While the environment was first introduced in [25], a new in-depth study, following single-subject design [14], using real teachers in training, is presented here. The goal of the study is to confirm that this approach is viable for real training. This research is the first to examine the effects of a VR environment in training DTT, and paves the way for further development of immersive situational training for special education teachers.

2 Background and related work

2.1 Discrete trial training

Since children with ASD often have a difficult time learning skills from observation or modeling, they sometimes require a more restrictive learning environment [37]. Researchers have examined different methods with the goal of enhancing learning opportunities for children with autism. One method, Discrete Trial Training (DTT) has been studied extensively and is incorporated into many early intervention programs for children diagnosed with autism [34, 37]. DTT is characterized by a highly structured teaching setting where skills are broken down into smaller units with multiple learning opportunities in a brief period of time [10, 12].

Teaching with the DTT method involves three fundamental steps:

  • The teacher delivers a clear instruction (e.g. by saying “Point at the car”). This is called a discriminative stimulus or cue.

  • The child responds correctly or incorrectly (e.g. by pointing at the car or at something else).

  • The teacher delivers an appropriate consequence (e.g. a reward for a child that correctly pointed at the car or implements an error correction procedure for a child that did not point to the car).

Throughout the teaching session the teacher delivers prompts. These prompts are then systematically faded out until the student is independent in the skill that is being taught. The teacher collects data and monitors the child’s progress throughout the sessions [37].

An example of skills taught in a DTT format is word recognition, such as pointing to the word “car” when the teacher says “point to word car”, or discriminating between different community helpers. These skills and other skills that are taught with a DTT method are important since they are prerequisites for many cognitive, communication, social, academic, work, and self-care skills.

2.2 Computer based DTT teacher training

As previously noted, it is extremely important that staff is trained well before implementing DTT with children. Ineffective training of staff could result in less progress as well as an increase in maladaptive behavior [19]. An effective training of DTT and other teaching methods usually involve the teacher receiving instruction about what the skill entails and an opportunity to practice the skill while receiving feedback from a skilled professional. This training is sometimes referred to as behavior skills training (BST). BST involves three necessary steps: (1) Instructions about the skill and why it is important; (2) Modeling, where the participant is shown how to implement the skill; (3) and role play with feedback [33]. A similar training procedure, called practice based coaching (PBC), likewise involves instructions, direct observation and feedback [24]. BST and PBC have been effective when training teachers to conduct DTT [9, 24, 33, 34]. Nonetheless, despite being an effective method one disadvantage of these training procedures is the reliance on scarce resources, specifically for the role-play as well as for the feedback from a skilled professional [28].

One option that has been used to decrease the amount of time needed for training, as well as to decrease the number of professionals needed, is to use computer based training. Several studies have demonstrated the effectiveness of using computer based training to teach DTT [8, 11, 13, 28]. Garland, Vasquez and Pearl [11] included feedback, demonstration and practice in a virtual classroom when teaching four participants the methods of DTT. The participants received written and verbal feedback for their performance after implementing a DTT session in a virtual classroom. Accuracy for all participants in implementing the steps of DTT increased after the training package.

Eldevik and colleagues [8] used an interactive computer simulation program (Dtkid) to teach 12 practitioners DTT skills. The authors evaluated the participants’ teaching performance as well as knowledge in DTT before and after training in the Dtkid simulation. For all participants there were significant improvements in teaching performance as well as knowledge in DTT after the Dtkid simulation training. In addition, the participants also demonstrated improvements in dealing with novel situations, or programs they had not been trained in previously. Although computer based training is effective some researchers have demonstrated that to be effective for all participants it is important to include a feedback component.

Pollard and colleagues [28] used interactive computer training where the trainees received an instruction manual with information regarding the DTT method as well as questions regarding DTT. In addition, the participants received video examples of how to implement DTT. Finally, the authors presented a pre- and post-test where they tested factual as well as application related questions regarding DTT. The authors found that although the interactive computer training resulted in improved implementation of DTT, one participant (out of 4) needed feedback to meet DTT criterion. Similarly, Higbee and colleagues [13] found that when training eight participants with a similar method as Pollard and colleagues, majority of the participants needed additional feedback to reach DTT mastery criterion. Thus it is possible that including feedback in computer based training is necessary for novice instructors to master the skill of implementing DTT.

Taken together, the studies mentioned above demonstrate that computer based training is a viable and an effective method in training novel staff to implement DTT, but providing better feedback might be advantageous.

2.3 Benefits of virtual reality training

Virtual Reality is a technology that allows its users to experience and interact with a simulated environment in a way that feels almost real to them. In essence, its users can be transported into any situation that can be generated by a computer, where direct first person interaction with the simulated entities replaces traditional abstract user interfaces in 2D computer applications. This direct experience paradigm lead to early adoption of VR as an important experiential training tool, especially in the aerospace and military domains, where frequent training with real life equipment and personnel proved cost prohibitive. In fact, the development of early VR systems, based on interactive graphic techniques and immersive display hardware, was intimately linked to the development of these training simulations [27].

While VR has provided a more cost effective alternative to expensive live exercises, it has also made it possible to address some of the other limitations of live training, including being bound to a specific location and time (e.g. a scheduled event at a training facility) and being limited in the extent dangerous and difficult scenarios (e.g. near mortal wounds and catastrophic events) can be recreated [39].

Going beyond potential practical benefits of VR, the paradigm also offers unique affordances that are associated with effective learning environments. Those include a direct non-symbolic experiential interaction with the material, which invites exploration and shifts in perspective (often associated with constructivist learning theory and emphasis on active learning); Intrinsic motivation and engagement with the subject matter (in part because of how responsive the environment is and in part because external stimuli is blocked out); Multi-modal interaction, which addresses more of the senses (e.g. haptic feedback from touching things in the environment) and contextualization of learning by immersing the learner in a complete virtual setting (e.g. learning about teaching in a class room setting) [1, 6, 21, 23, 26, 32]. In addition to these affordances, it has been pointed out that since all behavior of the learner is being tracked in real-time (e.g. where the learner is looking and what they are manipulating), there is an unique opportunity for analyzing this data to build a better model of the learner and adapt the training to address individual behavior (e.g. by directing attention or providing feedback on things that seem particularly salient to the learners) [1].

Given all these benefits, it is no surprise that many VR-based training applications have been built in the last 30 years. These cover a wide variety of professional disciplines including medical and surgical training, emergency and safety training, shipboard training, nurse training and even training for mental-disease professionals [18, 23, 27, 38, 39]. In many of these cases VR was picked for practical reasons, while evidence of superior learning outcomes with VR remained scarce early on. One reason for this is the presence of possible side-effects, such as simulation sickness [36], which plagued earlier systems in particular [39]. This is changing with better technology, and more recent VR training research is showing stronger behavioral and cognitive effects. For example, recall of information presented in a disability simulation was shown to be greater in an immersive version than a non-immersion version [5], and earthquake safety behavior was shown to be significantly better after VR-based safety training than a traditional video course [22].

VR-based training is still, for the most part, an emerging technology. We are getting closer to realizing its potential for dramatically improving the learning experience, but work still remains. Compared to on-screen learning, the unique affordances of VR represent a paradigm shift. It is this shift and these new opportunities - going beyond the screen - that primarily motivate the development of new training applications, including teacher training.

2.4 Virtual agents for specialists training

Ever since the virtual agent STEVE helped trainees grasp shipboard procedures in a Virtual Reality ship simulation [16], virtual pedagogical agents have proven their worth in a range of disciplines [15]. While effective as tutors, virtual agents have also taken on other roles in virtual learning environments to enhance training. For example, virtual patients have been around for a while to help medical students practice various medical procedures.

However, only with recent advances in embodied conversational agents [4], have these patients become fully interactive, supporting face-to-face conversations about their symptoms and feelings [29]. This increased social and psychological capacity has even produced virtual patients with psychological disorders, such as PTSD, for therapist training [17].

Virtual children have also been created as pedagogical play mates or learning-peers [3, 30], which in particular have shown potential as intervention for children with autism [40]. However, virtual children to train special education teachers in applying certain teaching methods have, to our knowledge, not yet been built and studied.

Fig. 1
figure 1

The trainee sits across from a virtual child and can manipulate the various Discrete Trial Training props on the table. On the left, a star, functioning as a reinforcer (reward) can be picked up and dropped into the child’s reward bucket. In the middle, three objects can be arranged in any order and then pointed to. On the right, a clipboard holds a data sheet (form) that the trainee is supposed to refer to and fill out during the session

Full size image

3 Approach and implementation

3.1 Equipment and software

In aiming for maximum transfer of skills to a real world setting, the virtual setting and the trained procedures are kept very close to the authentic source material. This includes immersing the trainee (the teacher in training) in an interactive training environment using fully tracked VR. For this project, we used the Oculus Rift CV1 with the Oculus Touch hand controllers to deliver the experience.

The virtual environment is a furnished special education room, built in part from scratch using the SketchUp 3D modeling system, and in part from available 3D assets. A child is sitting at a table in the middle of the room (see Fig. 1). The child is our virtual agent, and was put together in Fuse, software used for the creation of 3D models. Its animation was created in Blender, an open-source 3D graphics and animation software. The child loops an idle animation, moves legs and moves a little around in the seat. In addition to idling, the child can also point at objects laid out on the table. During this first phase of the project, the focus was on the DTT teaching method itself, rather than modeling a real child’s behavior in depth. Future work will extend the behavior range of the virtual child, including a model of attention and frustration.

The project was put together in the Unity 3D game engine, using the NewtonVRFootnote 1 script library for implementing natural prop interaction with the Oculus Touch controllers. All code was written in C#, and data, gathered about trainee’s actions throughout a session, is stored in an SQLite database. When trainees begin training, they are asked to log in using their name. The database then fetches all previous training sessions, so the trainees can see how well they have been doing in the past. For every session, trainee actions during each trial are stored, along with any errors they make. When errors are made, the trainees also receive automatic feedback inside the environment through text and visual indicators. A supervisor can always review a trainee’s progress by accessing the database. This data about the trainees could be used to determine whether trainees need more practice, or if they are ready to enter the real world.

Fig. 2
figure 2

The system uses the Oculus Rift CV1 head mounted display for full immersion with head tracking and two Oculus Touch VR controllers for picking up and manipulating objects in the virtual environment

Full size image

3.2 The training procedure

After logging in and starting a new training session, the trainee will be sitting in front of the virtual child, who is their student. There are three objects on the table in front of the trainee (see Fig. 1). A data sheet (a form) is placed on their right and a reward for the student, in the form of a star, is placed on their left. The first thing the trainee is supposed to do is review the data sheet to get an overview of the upcoming session. Some basic information about the child and the procedure has already been filled out on the form. The main part of the data sheet is a table where the trainee logs the child’s response to each instruction (a trial). The table is composed of rows which denote the different trials of the session.

Each session consisted of nine trials. At the beginning of each trial the trainee uses the VR touch controller to pick up and places the three objects in a row in front of the child (objects A, B and C). According to the DTT method (and as indicated on the data sheet), the objects in the first trial should be arranged with A on the trainee’s left, then B, in the middle, and C on the trainee’s right. A target stimulus is the object that the child is supposed to correctly identify in each trial. The target stimulus for the first trial is object A. After the child has tried to identify A (see below), the second trial starts by removing all the objects from the table and rearranging them. The order is now C, A, B and B is the new target stimulus. The rotation is repeated before each new trial. After each trial, the object on the far right is placed on the far left, while the other objects are moved one spot to the right. The data sheet displays the correct ordering of objects for each trial, as well as what object should be the target stimulus.

After the objects have been ordered correctly, the trainee provides a prompt by pointing with the VR touch controller to the target stimulus. The virtual child responds by pointing at one of the objects. It is then up to the trainee to decide if the child’s response was correct, in which case they should reinforce the behavior by giving the child a star, and log the information onto the data sheet accordingly. If the child’s response is not correct then the trainee should remove all objects, placing them to the side. Next the trainee should place the objects in front of the child again, in the same order as before their removal. This time the trainee should help the child point at the correct stimulus, taking their hand and leading it to the it. Now the response is correct and the behavior should be reinforced by giving the child a star. If the trainee does not follow the proper procedure, feedback is presented through text, visible in the environment, and highlighted objects, for example objects glowing red when they are incorrectly ordered.

4 Experiment

The purpose of this study was to test how effective the VR environment is in teaching Discrete Trial Training to novice instructors. Participants of this study were four women, special education teachers, who work at a school for children with developmental disabilities. These women were chosen as they were about to learn DTT as part of their professional training. We compared multiple baseline performance across participants, using an in-depth single subject design (see discussion on this design below).

4.1 Apparatus

The experiment was conducted on a VR ready desktop computer attached to an Oculus Rift CV1 Head Mounted Display (HMD) and two Oculus Touch VR controllers (see Fig. 2). The HMD has a resolution of 1080 \(\times \) 1200 pixels for each eye, and a field of view of 110 degrees. Frame rate was kept above 80 FPS to reduce danger of simulation sickness and the entire experience was seated without any locomotion, for maximum comfort.

4.2 Participants and setting

Participants were four teachers Sandy, Hanna, Beth and Sue, 36–42 years old, who worked at a school for children with disabilities.Footnote 2 The participants had 7-10 years of experience working with individuals with disabilities. However, none of the participants were familiar with DTT procedures. All participants went through baseline, lecture and VR training, as described in more detail below. Sessions for baseline were conducted in privacy in a room at the school the participant worked at. After completing the baseline phase, the participant listened to a lecture about DTT where the procedures were described and examples provided. The lecture was also provided at the school where the participants worked. The VR training and the performance measure after the training were conducted in a low-traffic computer science lab at a university. The experiment was conducted with each participant around the same time of day, approximately between 10 am to 4 pm.

4.3 Response measurement

The primary dependent variable was the participant accuracy in implementing six DTT teaching steps in the right order. The steps were:

  1. 1.

    Examine the data sheet.

  2. 2.

    Present stimuli (objects) in the right order.

  3. 3.

    Use an appropriate instruction phrase for each stimuli (e.g.“point at the car”).

  4. 4.

    Use an appropriate prompt (e.g. point towards the correct object).

  5. 5.

    Provide reinforcer (i.e. a reward) at the right time.

  6. 6.

    Log the child’s success on the data sheet.

Data on participants’ behavior was collected with a performance checklist. The performance checklist was based on similar criteria developed by Severtson and Carr [34] where they defined the behavior that was necessary in implementing DTT. The experimenter translated this performance checklist and collected data on the participants’ accuracy on each step. In each trial of a session the participant had to complete those six steps. After baseline, lecture, and VR training, the participant was instructed to perform the tasks of DTT with the experimenter serving as a child. The experimenter checked the boxes of the checklist simultaneously as the participant implemented the tasks. Accuracy was calculated by dividing the total number of correct performance steps with the total possible correct performance steps. The resulting ratio was then converted to a percentage. The mastery criterion was 100% of steps (6 out of 6) performed correctly in two consecutive session. The participants only got correct for their performance on each step if they implemented the step successfully and in the right order.

Fig. 3
figure 3

Mean percentage of correct responses following baseline, lecture, and VR training across all participants

Full size image

4.4 Experimental design

This study follows a single subject design, which is a robust experimental method to demonstrate causal relation, and more importantly, in developing and identifying evidence based practice [14]. Single subject design is mostly used in behavior analysis and clinical research where the purpose is typically to demonstrate the effect of a specific intervention [20]. In the current study, a multiple baseline design across participants was used to demonstrate the effects of the VR environment on skill acquisition for four teachers. Multiple baseline across participants design is a type of single subject design where the effects of the independent variable is demonstrated by systematically introducing it at different points in time for each participant (see for example the different number of baseline sessions between our participants in Fig. 4). Experimental control is demonstrated when the change in the dependent variable is observed only when the independent variable is introduced. This change is then repeated systematically across participants [20]. The independent variable is implemented at different times to replicate the effects of the independent variable as well as to control for other possible variables that could be responsible for the behavior change. Horner and colleagues [14] reviewed research questions that were appropriate for single subject research design. The authors noted that single subject research design was especially appropriate when the research question involved examining causal relations and/or when comparing the effects of two independent variables. In the current study, single subject research design is particularly appropriate since the purpose is to examine the effects of the VR environment on skill acquisition and to compare the effects of the VR environment to the more common teaching method of lectures.

Multiple baseline across participants design has been used in multiple studies where the purpose is to examine the effects of a technique in teaching various skills. For example when examining the effects of different methods in teaching DTT skills to teachers, when improving daily occupational performance and when examining the effects of a computer assisted tutoring program on caregiver skills [2, 7, 35].

4.5 Procedure

Baseline Participants were briefed about how the study would progress. During baseline the participants were provided with a data sheet and told they had up to 1 minute to read it. After the minute passed they were provided with a pen, a little paper star, instructional stimuli cards, and the same data sheet from before. Before each session the experimenter provided instructions to the participants in the following manner: “Here is a pen, the objects, a reinforcer, and the data sheet. Now do the task as best as you can, as you think it is supposed to be done”. The participants did not get any feedback from the experimenter on their performance or on their questions when they asked if they were doing the task right. After at least three trials or when the participants said that they did not know what to do, the baseline session ended.

Lecture After the baseline phase each participant went to a lecture with a specialist at the school. The specialist presented an hour long lecture about DTT. The lecture included the rationale for using DTT, the DTT procedure (including all the steps that were necessary to master the skills) and examples of what the procedure looked like. This lecture is a typical training method for teachers and other professionals. After the lecture the experimenter repeated the same performance test procedure as in the baseline.

VR Training After the lecture the participants went to the science lab for VR training. The participants were shown the VR device and its interface explained. Next, the participants started the VR training. After completing 9 teaching trials (one session) in VR, the participants conducted a performance test procedure in the same fashion as was conducted in baseline and after the lecture. The instructions were changed in the following manner: “Here is a pen, the objects, a reinforcer, and the data sheet. Now do the task as best as you can, just like you trained to do in the VR”. Just like in the baseline, and after the lecture phase, the participants were asked to repeat the task with the experimenter.

4.6 Results

Performance Figure 3 depicts the average performance after each step of the trained procedure.Footnote 3 After the lecture and VR training, performance improved across all steps. However, performance in looking at data sheet was the only step that improved to 100% performance after the lecture. Putting the stimuli in the right order, using the appropriate phrase, using the appropriate prompt, and logging on the data sheet improved up to 40% after the lecture. Reinforcement accuracy improved to 60% accuracy after the lecture. Performance improved dramatically after VR training compared to baseline and after the lecture. Performance across all steps increased on average from 75 to 100% accuracy after having experienced training in the VR environment.

Session-by-session results for Sandy, Hanna, Beth and Sue are depicted in Fig. 4. For all participants a low and stable performance was observed in the baseline condition. After the lecture, performance for all participants increased. After VR training all participants reached mastery criterion. The experimental design, which introduces intervention at different times (showed by the vertical dashed line in Fig. 4), makes the effect of introducing the VR environment quite clear.

Experience A follow-up survey with four questions about the experience of using the system was sent out to the four participants, to get some subjective feedback. Three of the four participants provided answers. The first question asked was “How did you experience the virtual reality DTT training?”.Footnote 4 All participants related a positive experience saying it was “fun”, “good” and “exciting”. One however also mentioned that not being able to wear her prescription glasses made it somewhat harder to read text. The second question was “Is there anything about virtual reality, compared to say regular computer programs, that makes the training more effective in your opinion?”. All participants mentioned that “being in the situation” was something they considered very effective. They noted that this was something they were not used to experiencing using regular software. The third question was “Was there anything repellent or uncomfortable about the virtual reality, which could reduce learning effectiveness?”. Two of the three could not mention anything that came to mind, while the third explained that a technical glitch had caused some delay. She also pointed out that the current interaction feels mechanical in comparison with the real world. She still saw the value in the rehearsal. The final question was “When this kind of VR training has completed development, would you choose to use it?”. Two of the three gave a definite “yes” answer, while the third, said that she was not sure how she would utilize this training unless new training scenarios would get added, in which case she would want to see them here first before encountering them in the real world.

Fig. 4
figure 4

Results for Sandy, Hanna, Beth and Sue depicting the percentage of steps performed correctly in baseline, after the lecture, and after VR training

Full size image

4.7 Discussion

Training in the VR environment was sufficient for all participants to correctly implement basic DTT skills. All participants reached mastery after going through the VR training phase. All performance except for Sandy’s increased after the VR training session, but her performance increased after conducting two sessions. Perhaps a more relevant result for those that train teachers is that none of the participants reached mastery criterion after attending a lecture about the DTT method. Although we did not examine what aspect of the VR training improved performance, it is likely that being able to practice the skill is an important component when training staff.

Based on the follow-up feedback, the training experience of the participants was positive, raising their interest in this learning approach. As with any new technologies, and in particular brand new paradigms, one should assume a positive bias in the subjective responses, due to a strong novelty effect. However, it is particularly encouraging that none of the three participants, that provided answers, mentioned any averse side-effects or usability issues, apart from not being able to wear glasses.

5 Conclusions and future work

To our knowledge, this research is the first to test the viability of VR for teaching Discrete Trial Training. Apart from being a viable teaching method, as shown by our results, there are many other benefits of applying VR in this context. One benefit is that there is no real danger when the trainee makes errors during training. The system, as well as a human supervisor if present, can make sure that the trainees receive feedback after making errors without it impacting a real child. Another benefit of VR is the fact that it provides a more controlled situation than a real-world setting. For example it is possible to manipulate the difficulty level based on each trainee’s progress by introducing a more difficult child and more complex situations. This is a particularly interesting direction, and the subject of ongoing research. Another benefit is the time it took to train the participants. In the current experiment it took approximately 30 minutes to reach mastery criterion, which is relatively short for DTT training.

It should be noted that the training provided by the current system addresses only basic DTT skills - the performance checklist covered six aspects of DTT. Other skills that have to be learned to implement DTT successfully include the right timing of stimuli presentation, appropriately securing the child’s attention, delivering appropriate instruction, appropriately increasing or decreasing the prompt based on the child’s performance, and error correction [35]. These are skills that we plan to support in the virtual environment in the future.

It is necessary to examine the generalization of these skills in more naturalistic teaching conditions, for example when teaching children with autism. In the current study we only tested the skill with the researcher where she acted as the child in a session. Thus we cannot conclude that the teachers conducted the skills accurately when working directly with the children. In addition it is necessary to compare the effectiveness of VR training to other successful training methods. For example Severtson and Carr [35] used a 53 page self-instruction manual, and a quiz plus feedback on participants’ performance before they implemented DTT on a confederate. Those who did not reach mastery criterion after reading the manual watched a 41 minute long video of the same material as in the manual. In the following DTT session the participants received feedback on their performance. Half of the participants had to go through all phases of the experiment (manual, feedback, video). The other half only had to complete the self-instruction manual to reach mastery criterion.

It is important to examine the advantages or disadvantages of using VR training over some of these other training methods. For example, it would be interesting to examine if participants would reach mastery criterion without receiving a lecture before practicing the skill in a VR environment. Finally, although the results of the study where promising one should take them with caution due to the small number of participants. It is important to examine this procedure with a larger sample size to improve the generalizability of these results.

In conclusion, we encourage researchers to examine the different ways in which VR can be used to train teachers and parents of children with special needs. The demand for such training is high, and these results give us a reason to be optimistic about VR having real impact.