Introduction

Autism spectrum disorders (ASD) have been clearly identified as a heterogeneous set of disorders spanning a broad level of strengths and needs (American Psychiatric Association 1994). Yet one consistent area of need for students with ASD is in the area of social competence, particularly in regards to understanding and using basic rules of social engagement. Deficits in this area can be particularly notable and problematic for individuals who are considered high functioning, given misplaced expectations for a presumed ability to navigate their daily environment with equal prowess to their intellectual abilities. As a result, individuals with high-functioning autism (HFA) desire the benefits derived from positive social functioning, but often suffer significant consequences from a lack of social competence (Attwood 2007).

Deficits in social competence can have severe and long-lasting consequences if left untreated. Immediate issues such as the inability to make and maintain friendships can result in social isolation (Eaves and Ho 1997), inability to properly handle bullying (Stichter et al. 2010), and low self-esteem (Myles and Simpson 2002) to name a few. Diminished success in the pursuit of post-secondary education as well as the ability to obtain and maintain employment continue to be signs of longer-term impacts of social competence gaps (Howlin 2000; Howlin et al. 2000; Newman et al. 2011). Assisting students with ASD through social competence interventions, regardless of location and access to services, is clearly needed in order to improve the well-being and outcomes of youth with ASD.

The Case for Supporting Rural Schools

In recent years, schools have been increasingly charged to meet the needs of all students through the use of highly qualified teachers delivering evidenced-based practices (NCLB 2002; Simpson et al. 2004; Jimerson 2005; Cook et al. 2008; Torres et al. 2012). In the area of autism spectrum disorders and specifically social competence, this task has been rather unique. The Centers for Disease Control (Baio 2012) reported that in 2008, 1 in 88 children were identified as having an ASD, and given the exponential increase in identification, consider ASDs an urgent and growing public health concern. The prevalence rate is less clear for High Functioning Autism (HFA); however, conservative estimates indicate an approximate increase from .26 per 1,000 children (Fombonne 2007) to upwards of .48 cases per 1,000 (Fombonne 2009). With the increased incidence of HFA students, schools are further challenged to provide uniquely qualified teachers and provide access to the appropriate evidenced-based practices (Cook et al. 2008; Torres et al. 2012). This need is found across all types of school settings; rural schools are regularly identified as challenged to provide expert teachers and targeted programs for students with special needs (Berry et al. 2011; Johnson and Strange 2007). These schools typically struggle to provide qualified teachers to meet traditional curriculum needs (Strange et al. 2012). For example, while access to education specialists in rural areas may be available, these specialists most often travel from school to school and depending on location and rural nature of their location, the frequency and duration of those visitations and consultations can vary widely. As such, the opportunity to consistently provide and build capacity in the use of evidence-based practices such as social competence can be problematic. However, distance education can be an approach to bridge gaps that constrain consistent use of best practices.

Distance Education and 3D Virtual Learning

Distance education (DE) and other distance support tools have been used for well over two decades to support teachers and students in rural areas. DE is an umbrella term for education that occurs when the teacher and student(s) are geographically distant from each other (Keegan 1996). Activities such as telementoring, teletraining, and telemedicine have been used since the 1980s to effectively provide technology-mediated consultation and education to rural and remote environments that need support in delivering evidence-based practices (Kendall 1992; Rule et al. 2006). In addition, DE has also been used to offer a wide variety of courses to students. Given the nearly 10 million students attending rural schools, a 2005 survey conducted by the National Research Center on Rural Education Support investigated issues related to educational access (Hannum et al. 2009). Distance education technology was identified as a key strategy being used by rural schools to provide a full range of courses and to overcome difficulties in attracting and maintaining qualified and experienced teachers. Furthermore, the 2005 survey showed that 85 % of the surveyed districts had used distance education, and all indications suggest continued growth since. More recent national surveys have shown substantial increases in the use of online education in K-12 reaching a reported 1.5 million students in the 2009–2010 school year alone (Picciano 2009; Wicks 2010). Typically rural and small schools use DE technology to meet the needs of students for courses such as foreign language and advanced credit. However, for rural and small schools DE may also help them enhance their ability to provide high quality, evidence-based programs for students with special needs such as social competence programs for students with ASDs. Unfortunately, typical DE is limited in how it provides support for affective and social learning (Rice 2006) which is often critical to addressing students with special needs.

A form of DE that has been identified as highly engaging and social is 3-dimensional virtual learning environments (3D VLEs). 3D VLEs are defined as “computer-generated, three-dimensional simulated environments that are coupled with well-defined learning objectives” (Schmidt 2010, p. 4). These three-dimensional environments use an “Alice-in-Wonderland” style interface (Dede 2005) in which the student enters a 3D world through their avatars using a networked computer. Students report a sense of being immersed in another world and the perceptions of their own bodies begin to mesh with their avatar (Mennecke et al. 2011). For collaborative 3D VLEs, students who are in world together can often see each other’s avatars, text chat, manipulate objects in world, interact with each other via their avatars, and use their voices to speak to each other. The students do not have to be located in the same room, school, or even state to speak and collaborate together in world. In this manner, the medium allows students to experience and perform actions through their avatar (Hew and Cheung 2010) and, as such, 3D VLEs hold great promise for learning needs that require both cognitive and behavioral practice such as social competence. These collaborative 3D VLEs have potential for addressing the needs of rural districts for DE programming in that these environments bring students together for peer interaction, experiential learning through collaborative effort, and guidance by an expert teacher.

Research supports that learning can take place through the use of 3D VLEs, both for individuals with neurotypical development as well as individuals with ASDs. Some of the most prominent research that provides evidence of learning through 3D VLEs is based on science inquiry environments for middle school-age students such as River City (Ketelhut and Schifter 2011; Metcalf et al. 2009), Quest Atlantis (Barab et al. 2005; Hickey et al. 2009) and EcoMUVE Metcalf et al. 2009, (2011). However, most of the implementation of 3D VLEs to date has been as supplements to in-class, teacher-led activity rather than for DE. There is also evidence that students with ASDs can learn targeted social skills in 3D VLEs. Research supports that these individuals can learn and generalize social appropriateness in social settings (Leonard et al. 2002; Parsons and Mitchell 2002; Parsons et al. 2005; Rutten et al. 2003), and other targeted social skills such as enhanced empathy (Cheng et al. 2010), emotion recognition through avatar representations (Moore et al. 2005), and positive social behaviors such as eye contact and attending (Cheng and Ye 2010). In addition to supporting learning outcomes, this medium has been found to be highly engaging and motivating, both for neurotypical peers (Arici 2009; Choi and Baek 2011; Dede et al. 2005) as well as for individuals with ASDs (Mineo et al. 2009).

Challenges in the Use of 3D VLEs

3-Dimensional virtual learning environments are poised, through DE, to support schools to further meet needs for students with special needs. Nevertheless, in practice there are both technological and curricular challenges. Technologically 3D VLEs require highly capable computers and networks. Although rural schools generally have many computers, their computers are unlikely to meet the requirements of advanced applications such as 3D virtual learning. Additionally the US Department of Education (Gray and Lewis 2009) noted that although 100 % of districts were connected to the Internet, only 12 % had an ISP connection of T3 or DSL3, and that rural communities are 3 times more likely to have lower performance networks than more urban districts. Further, a Federal Communications Commission 2010 survey of E-Rate funded schools (Horrigan 2010) reported that nearly 80 % of respondents reported their broadband connections were inadequate to meet their current needs. The DOE and FCC data show that the technology infrastructure for schools and especially rural schools may need to be upgraded or supplemented to benefit from 3D VLEs.

From a curriculum perspective, 3D VLEs must be designed to elicit the cognitive and behavioral requirements of learning as well as enable pedagogical and behavioral management practices. Some of the natural affordances of a physical classroom such as physical prompting or tools such as paper and pencil are not possible in a 3D VLE and must be replaced by new mechanisms. Similarly, the natural practices of a teacher to elicit and observe behavior are then mediated through the 3D VLE and the behaviors of avatars, and as such, the affordances and practices that are designed into traditional classroom curriculum need to be translated into the 3D VLE. For example, special educators have methods, including teaching strategies and physical room structure, for helping small groups of students stay together and interact during instruction. These methods can break down when all the educator can see are the student avatars within a wide and open virtual space. However, virtual devices such as learning spaces and pods with rules to govern student movement can be built into virtual worlds to shape student behavior. To implement a 3D VLE every object and action must be envisioned, designed and implemented in the world. This provides challenges in living up to the requirements of the evidenced-based practices for teaching and learning. For example, if a student does not have a sense of participating with others in activities then they will not practice turn taking or sharing ideas in ways that elicit the cognitive and behavioral expectations of the curriculum.

Benefits and Affordances of 3D VLEs

The use of 3D VLEs has the potential to provide engaging, supportive and social DE for meeting curriculum needs (Arici 2009; Iqbal et al. 2010; Metcalf et al. 2011). These environments offer many affordances for students who need support and social interaction as part of their learning process. For example, students can learn about constructs via multiple-media rather than simply text, and be engaged interactively in tasks. Standen and Brown (2006) noted three key areas in which 3D VLEs are well suited to youth with ASD as well as other special needs. First, 3D VLEs allow users to learn by making mistakes but without suffering real world consequences. Youth with ASD are often denied real-world experiences because caregivers and educators are forced to do a risk assessment and often make restrictive decisions in regards to independence for physical and emotional safety reasons. Secondly, VLEs are endlessly plastic in that they can be manipulated in ways the real world cannot. For example, scaffolding in the form of suggestions that may appear in the interface or highlighting of certain features in the scene can be provided at the beginning of a task and then withdrawn as the user proceeds. Thirdly, rules and constructs can be conveyed through experience, not simply words or models of what others can do. For example, rules for how to greet a person or interpret facial expressions can be experienced and practiced in contexts that offer high fidelity to natural settings. While not noted by Standen and Brown, our own work suggests that the naturally reinforcing aspects of 3D VLEs support motivation and engagement with the curricular tasks in ways that allow the instructor to spend less effort on delivering the curriculum and be more able to individualize responses and specific delivery to students. Thus when the challenges of design and delivery are met, or diminished, educators will have new powers to represent ideas and engage students for teaching and learning over the Internet so as to reduce barriers to access.

The iSocial 3D VLE

The iSocial 3D VLE is a Distance Education program that implements the Social Competence Intervention (SCI-A) curriculum (Stichter et al. 2010; Schmidt and Stichter 2012; Schmidt et al. 2011b) within a collaborative 3D VLE. The SCI-A curriculum is designed for adolescents age 11–14 with HFA to increase social competence and is based on several evidenced-based practices including Applied Behavior Analysis, Cognitive Behavior Intervention, and scaffolded instruction (see Stichter et al. 2010 for a full description). The distance education structure of iSocial provides a highly trained educator, located in a university lab, to guide a cohort of students through the SCI-A curriculum within the iSocial virtual worlds. A design-based research process (Laffey et al. 2010a) was undertaken to translate SCI-A into iSocial with careful attention to fidelity in teaching such as explaining concepts, providing opportunities for practice and giving feedback. This process began with extensive conversations between the SCI-A team and the iSocial team, which led to lesson planning and system prototyping. Once a working prototype was achieved for a first set of lessons a systematic set of laboratory-based studies were undertaken to establish the feasibility of implementing the curriculum via a virtual environment (Laffey et al. 2011a; Schmidt et al. 2011a). These studies also examined and led to the refinement of design features of iSocial that supported reciprocal interaction among the students and supported the online teacher’s ability to manage and facilitate social behavior in the environment (Schmidt et al. 2012; Laffey et al. 2011a, b, 2010a, b, c).

The Current Study

The purpose of iSocial is to address the limited access of schools to evidence-based practices through the use of DE technology and curriculum to meet the unique needs of students with ASDs to gain social competence. It was built using the foundation of what is understood as the affordances of engaging, motivational aspects of 3D VLEs while addressing the limitations and constraints through design, development, and implementation practices. This study was a first field test of a full implementation of iSocial designed to examine its potential utility in the following areas: (a) the impact on social competence for students with HFA as measured by both descriptive and performance assessment; (b) the degree to which the iSocial 3D VLE implementation has fidelity to the SCI-A curriculum; and (c) the degree to which students, teachers and parents found the experience socially valid.

Methods

Setting and Participants

Three separate school districts located within 150 miles of one another in the same Midwestern state were recruited to participate. Two of the districts had a school population of approximately 18,000 and one district had a school population of 4,500. The districts ranged between 40 and 70 % of student population qualifying for free and reduced lunch. One of the larger districts was considered local, near both the university and location of the 3D VLE server. The smallest district was approximately 60 miles away, and the final district about 120 miles from the server. The focus of this field test was to examine the promise of delivering a social competence intervention over the Internet. As a result, one rural and two suburban districts (GAO 2004) were selected to provide a desired sample range for this stage of system development. iSocial is delivered over the Internet from a university-based server through the school district firewall and to students at individual computers. The schools must have high-speed networks with no packet loss to the server and the individual computers (Windows 7–64 bit or Macintosh OS X) must have a powerful graphics cards. Most school districts have networks and computers that meet the specifications with the exception that schools typically buy computers that do not have sufficient graphics capability and, in practice, school networks have many interruptions and faults. Having a variety of school delivery sites allowed the research team to monitor and compare the technological implementation across school sites.

Within each district eligible students were referred to iSocial by cooperating school staff after obtaining parent consent. A maximum of six students from each site could be accommodated as a single group. Eligibility criteria were as follows: (a) a clinical/medical diagnosis of an ASD; (b) age 11–14; (c) a full scale IQ score above 75; and (d) access to neurotypical peers for at least part of their day. Consistent with previous applications of the SCI-A curriculum, gold standard of diagnostic assessments, the Autism Diagnostic Observation Schedule (ADOS; Lord et al. 2003) or Autism Diagnostic Interview-Revised (ADI-R; Lord et al. 1994) were captured for each student. The cut off scores were not used for eligibility but the scores were documented in the event that any unaccounted variances might be further illuminated through these assessments. This was not needed for this particular data set.

A total of three school districts encompassing four middle or junior high schools participated. Each school district had its own “course” of iSocial delivered. Therefore, all students in each course were from that particular district. These unique courses were used to simplify coordination and isolate district specific technology issues for this pilot implementation. Within some courses students were placed in separate building and in all cases, students were physically distanced from one another during lessons to simulate the DE experience. Each student had their own computer setup, all the while, wearing headsets and interacting with each other and with the university-based educator within the VLE. The local district included four students, the smallest district included three students, and the distant district included four students. A total of 11 students across the three districts participated. Information about students’ characteristics is found in Table 1. For these students, their respective districts chose iSocial as their designated social programming in lieu of any alternate school-based programming. As a result no other planned social skills programmed occurred for students during the school day.

Table 1 Participant characteristics (n = 11)
Full size table

SCI-A Curriculum and iSocial Modifications

Social competence intervention for adolescents is a social competence curriculum for adolescents (see Stichter et al. 2010). It encompasses 31–45-min lessons over 5 units. The curriculum covers recognition of facial expressions, sharing ideas with others, turn taking in conversations, recognizing feelings and emotions of self and others, and problem solving. Each unit takes about 2 weeks, presuming an alternating-day delivery schedule. The structure for each unit consists of the following: (a) reviewing a previously learned skill and introducing a new skill in an instructional and group discussion format; (b) skill modeling; (c) opportunities to practice the skill in structured and naturalistic practice activities; and (d) some type of closing activity or review. To maintain fidelity with the original SCI-A curriculum, iSocial delivered a scaffolded approach as concepts that were learned in previous units were revisited and expanded upon in later units in addition to the new content. Also consistent with previously published SCI-A curriculum reports, the iSocial implementer was a Masters level educator. In the application of iSocial, this individual was referred to as the ‘Online Guide.’ Additional details on the social competence intervention, targeted objectives and curricular scaffolding are addressed in Stichter et al. (2010).

iSocial followed the same five-unit structure and delivered the curriculum across 31 lessons as in SCI-A. In addition, all teaching and learning practices of the SCI-A curriculum had to be translated into virtual practices. This section gives examples of these translations and how the virtual world designers collaborated with the curriculum specialists to create appropriate activities that worked fluidly through this medium to ensure that targeted core objectives were met without any learning modification of those objectives, and maintain fidelity with the cognitive and behavioral requirements of the SCI-A curriculum. The examples illustrate how the unit and lesson structure were maintained and how the student practices for reviewing lessons, practicing skills and implementing skills were matched between SCI-A activities and iSocial activities so as to assure fidelity while accommodating the DE medium.

Examples of iSocial Modifications to Reviews and Modeling

In the face-to-face SCI-A review activities, students utilized paper and pencil to answer guided questions to facilitate self-reflection on progress towards social competence goals. While in theory students could utilize paper and pencil while using iSocial, they would need to be taken out of the virtual medium in order to do so, and the teacher would not be able to see their progress or written answers on those questions. For that reason, this component was modified for the 3D VLE to take advantage of the developmental affordances and address the writing for reflection limitations. A type of question-and-answer “sheet” mechanism was built into the 2-D interface of iSocial. Students could open up their selected sheet to engage in self-reflection and type or select their answers, and submit those answers to the teacher. These sheets were also utilized as replacements for other types of reflection and rating tasks such as watching video modeling of social behaviors. As students watch the video models in world, they refer to the questions on their sheet and rate the behaviors. As each student submits answers, the teacher sees the student reflections on a summary sheet and can then use these reflections to engage students in a discussion regarding the targeted objectives.

As another lesson example, students needed to review recognizing facial expressions in situations, but rather than displaying a set of flat 2D interface images that did not take advantage of the 3D VLE, the recognition tasks were embedded in a game-like problem solving task where students needed to “dig” their way under a moat to find the King in his castle. By choosing the correct facial expressions to given scenarios, the group dug their way under the castle wall and arrived on the other side of the castle where the remaining lessons took place. The same learning objectives were accomplished from the original curriculum, but in a way that took advantages of the 3D VLE interactive and engaging affordances.

Examples of iSocial Modifications to Structured and Naturalistic Practice

Structured and naturalistic practices in SCI-A allowed students to practice the previously taught skills in a given activity. In one face-to-face SCI-A activity, students collaborated to plan a vacation together as a way to practice taking turns in a conversation. They decided where they wanted to go, what to take in the suitcase, and then planned the vacation by choosing transportation, meals, and activities. While in the classroom, the teacher needed to get the students enthused about the make-believe travel in order for the turn taking to be a part of authentic discussion. One of the major affordances of a 3D VLE was to actually put the students into a task within which they had to work together. iSocial replaced the travel planning scenario with the opportunity to build a restaurant. While the students worked together to identify what type of restaurant and menu items they wanted, the restaurant was built so that each succeeding question or puzzle was set in the context of their evolving restaurant. Their collective decisions became immediate and were visual. The same educational objectives were addressed with similar attention to the cognitive and behavioral requirements of the lesson, but the constraints of the real world were mitigated and the affordances of the virtual world emphasized.

Another example comes from the unit in which students’ abilities are developed to use their social competencies to solve problems together. The naturalistic practice activity of this unit was an egg drop that the students performed in the face-to-face SCI-A curriculum. The students were given a budget to buy certain materials, they made a plan for how they were going to construct the materials into a vessel to protect their egg, then they built the vessel and performed the egg drop. However in the 3D VLE employing the principles of physics, to test the egg drop vessels, would have been a difficult technical challenge within the current system. For that reason, the social competence objectives were addressed by utilizing other affordances of the environment. Instead of an egg drop, students were located in a medieval castle world, and worked together to help find the King’s missing items and return them to the King. They read clues and planned their course of action. In doing so, they took advantage of interactive planning mechanisms, affordances of movement, large spaces, and interactive objects to experience working together and moving throughout the castle in order to accomplish this problem solving activity together.

Whenever possible, the components of the curriculum were kept the same, however when necessary components were modified, it was done to most accurately achieve fidelity as well as maximize the medium’s affordances for engagement.

Measures

Consistent with previous applications of the SCI-A curriculum, all eleven students participating in iSocial were administered a full battery of assessments specific to program participation approximately 2 weeks before (pre) and 2 weeks following (post) the intervention. Assessments included others’ reports of students’ behaviors and performance-based measures. For each student, one parent/guardian (same person at pre and post) completed written questionnaires about their perception of the students’ skills and challenges. One teacher, identified by the school as one of the student’s primary general educators, also completed a written questionnaire at pre and post. Although aware that student participants were involved in a social competence intervention, neither the identified teachers nor parents completing assessments were provided direct access to the curriculum or its core constructs. Finally, project staff administered a performance battery with each student lasting between 75 and 90 min.

Parent and Teacher Ratings

The Social Responsiveness Scale (SRS; Constantino and Gruber 2005) was completed by each student’s parent/guardian and by one teacher to evaluate social abilities in the home and school environments, respectively. The SRS is a standardized 65-item rating scale (4-point Likert scale) that measures social impairments associated with autism across five domains: social awareness, social cognition, social communication, social motivation, and autistic mannerisms. Higher scores reflected greater social impairment in that domain. T-scores derived from large-scale norming were provided in the scoring manual, however in order to better represent variations among very high scores, raw scores served as the unit of analysis in this study (consistent with other intervention research). Prior psychometric analysis (Constantino and Gruber 2005) indicated all subscales have demonstrated acceptable internal consistency (α = .77–.92), high construct validity compared to the ADI-R (rs = .65–.70), and contrasted groups validity (i.e., scores distinguish between ASD/non-ASD children).

One parent and one teacher per participant completed the Behavior Rating Inventory of Executive Function (BRIEF; Gioia et al. 2000), a standardized measure used to rate executive function behaviors in the participants’ environment (i.e., home/school) on a 3-point Likert scale. The eight BRIEF subscales can be combined to create two broader indexes, behavior regulation (BRI) and meta-cognition (MI). These two indices were also combined to create an overall score, the global executive composite (GEC). In each index, higher scores indicated increased executive functioning deficits. The t-scores, derived from norms reported in the administration manual, served as the unit of analysis. All subscales/indices demonstrated good internal consistency (α = .84–.97) and significant levels of convergent validity in comparison with other scales of attention/behavioral difficulties (rs = .35–.83) in clinical and normative samples (Gioia et al. 2000).

Performance Measures

The Reading the Mind in Eyes test (Baron-Cohen et al. 2001) was used to assess how well participants could attribute a mental state in another person using this facial feature alone. Although an advanced test of theory of mind, this measure also requires participants to infer emotional states based on subtle yet complex cues in the eye region; therefore we used this measure to assess participants’ ability to recognize facial expressions/emotions. Participants examined a series of 28 pictures of the eye region and were required to indicate which of the four mental state words/phrases corresponded to the picture. Scores were calculated as the number of correct identifications, with higher scores indicating greater ability to interpret mental states based on cues from the eyes. The Mind in Eyes test demonstrated discriminant validity in its ability to distinguish adults with and without an ASD diagnosis (Baron-Cohen et al. 2001).

The Faux Pas Stories (Baron-Cohen et al. 1999) consisted of ten short narratives that did or did not include a Faux Pas (an embarrassing or tactless remark or action which violates accepted, oftentimes unwritten social rules/cues). Five stories included a social Faux Pas; the other five included a social interaction but no Faux Pas (control). For each story, students had to identify if a Faux Pas was present, what the Faux Pas was if present, a non-social detail about the story (to assess comprehension) and if the character had some key knowledge about the social interaction. Students had to answer all four questions correctly to receive credit for the item. Faux Pas was scored as the number of correctly identified scenarios out of ten, with higher scores indicating a greater accuracy of Faux Pas identification. The Faux Pas Stories demonstrated discriminant validity in its ability to distinguish children with and without an ASD diagnosis (Baron-Cohen et al. 1999).

The Strange Stories (White et al. 2009) consisted of eight stories that required students to infer about either the character’s mental (four stories) or physical (four stories) states. The mental state stories assess students’ ability to attribute beliefs, intent or desire; physical state stories function as non-social controls. Each story was scored on a 0–2 scale with higher scores indicating more accurate responses with a maximum of eight points available for each type of story. The measure’s authors indicated the discriminant validity of the Strange Stories to distinguish between children with and without ASD.

The diagnostic analysis of non-verbal accuracy-2, child facial expressions (DANVA-2-CF; Nowicki and Carton 1993) measured recognition of facial expressions in photos of children. This computer-based assessment sequentially displayed 24 photos of children showing varying intensities of happy, sad, angry or fearful emotions. Participants selected which one of these four emotional labels they thought corresponded to the photo. Scores were calculated as the number of correct identifications, with higher scores indicating more accurate facial expression recognition. Nowicki and Carton (1993) reported acceptable internal consistency (α = .74–.76).

Delis–Kaplan executive functioning system (D-KEFS; Delis et al. 2001) is a battery of neuropsychological assessments designed to test student performance on multiple dimensions of executive functioning. We utilized a series of tasks within four distinct tests: Verbal Fluency, Design Fluency, Trail Making, and Color-Word Interference with each student. As a whole these tasks assess dimensions of: initiation, cognitive flexibility, set shifting, inhibition, simultaneous processing, and attention maintenance. All D-KEFS subtests have good internal consistency (>.70) and have been associated with other performance measures of executive functioning, suggesting convergent validity (Delis et al.).

Conners’ continuous performance test-II (CPT-II; Conners and Staff 2000) is a computer-based assessment that measures students’ attention span and inhibition. Students are shown one random letter at a time on the screen and asked to tap a button as fast as possible, except when they see a particular letter (i.e., X). Two scores of interest in this study are the number of omission errors (failure to tap the button for non-X letters) and commission errors (tapping the button for an X rather than inhibiting that response). Fewer errors of either type indicate better attention and inhibition. Conners and Staff (2000) indicated that split-half reliability and test–retest reliability were both adequate (.73–.95 and .55–.84, respectively) and the CPT-II has demonstrated performance differences between individuals with ADHD and neurotypical controls.

Measures of Intervention Fidelity and Social Validity

Across each district, three graduate student observers used an a priori fidelity checklist that was parallel to that used in SCI-A but updated to reflect pertinent iSocial modifications. To code the online guide’s fidelity of implementing the lesson plan as directed each teaching behavior was classified into one of four categories considered to be the core features of the SCI-A/iSocial curriculum: content (skill, concepts, activities), process (instructional methods like facilitation and response clarification), behavior management (use of the designated behavior management system), and specific verbal feedback to students. Fidelity of each teaching behavior was recorded on a 0–2 scale: 0 indicating that the teaching behavior did not occur, 1 indicating the behavior was observed but needs improvement, and 2 indicating full and accurate implementation of the designated teaching behavior.

Students, one parent/guardian, and the identified general education teachers who participate in the program were asked to complete a social validity assessment after the completion of iSocial. As the intention was to develop a 3D VLE, school-based social competence intervention that is feasible and sustainable, it is also important to understand the acceptability of this program for the involved stakeholders. A modified version of the Intervention Rating Profile (IRP-15; Martens and Witt 1982) was implemented to assess social validity among school personnel and a satisfaction survey (Wheeler et al. 2002) was used to assess parent and students’ perceptions of social validity. Students responded on a 3-point Likert scale; parents and teachers responded on a 6-point Likert scale. For all reporters, higher scores indicated greater perceived benefits and satisfaction with the iSocial experience.

For all pre-post assessments (i.e., parent/teacher reports, student performance tasks), changes in scores from pre to post were examined via paired t tests to detect group mean differences. For all analyses, a criterion p value of .05 (or less) was used to indicate statistical significance. It is important to note that as a pilot study, the presented results and criterion p values are intended for use in a more descriptive sense to show data trends and patterns rather than in generalizable sense as would be expected with larger datasets.

Results

Fidelity of Implementation

Across all three districts, the online guide was able to implement the curriculum in the iSocial environment at a high level of fidelity. A total of 95 lessons were coded, with 32 % of the lessons double coded. Inter-observer agreement (IOA) was calculated on all double coded lessons using Kazdin’s (2010) method for calculating percentage of agreement [# agreements/(# agreements + # disagreements)]. Inter-observer agreement on the coding of fidelity scores was also high. Specifically, out of a maximum of two points, implementation scores were as follows: content >1.97 (IOA > 99.00 %), process >1.93 (IOA > 99.33 %), behavior management >1.97 (IOA > 98.85 %), and specific verbal feedback >1.97 (IOA > 99.42 %). These fidelity and IOA results are comparable with forthcoming SCI-A data, and both iSocial fidelity and IOA data are above industry standards (Borrelli et al. 2005).

Parent and Teacher Reports of Student Skills and Behaviors

A summary of parents’ and teachers’ reports of student improvement can be found in Table 2. Overall, parents’ reports of students’ social behaviors and interactions via the SRS were significantly improved from pre to post intervention. Results of t tests indicated significant improvement in overall social responsiveness (t = 3.72, p < .01), particularly in the domains of social cognition, social communication and social motivation. Parents’ reports also suggest improvements in students’ executive functioning at home (t = 2.43, p < .05), particularly in the metacognitive domains such as planning, monitoring and working memory.

Table 2 Summary of pre and post intervention scores on parent/teacher reports and student performance tasks
Full size table

Teachers’ reports of social behaviors and interactions did show change in the hypothesized direction, however these changes did not obtain statistical significance. Similarly, teachers’ reports of students’ executive functioning in the school setting also changed in the hypothesized direction, but not at a statistically significant level.

Students’ Task Performance

On the assessments related to recognition of others’ perspectives in social situations (e.g., Faux Pas Stories, Strange Stories Mental States), students’ performance was highly variable and overall results did not indicate improvement pre to post intervention. Concerning recognition of facial features and facial expressions, students’ scores changed in the hypothesized direction but this change did not achieve statistical significance.

On the performance assessments related to aspects of students’ executive functioning abilities, students did demonstrate improvements on tasks that assessed inhibition of dominant responses (e.g., D-KEFS Color-Word Interference, CPT-II Commission Errors). However, these improvements were not statistically significant. Students’ scores also did not improve on tasks that assessed aspects of cognitive switching, except in the D-KEFS Design Fluency test (t = 2.23, p < .10). Students did demonstrate significant improvement on tasks that assessed cognitive flexibility and generativity (D-KEFS Design Fluency Total, t = 3.82, p < .01).

Overall Satisfaction in the iSocial Experience

Using the 6-point Likert scale, the mean rating of satisfaction and acceptability was 5.38 (SD = 0.47) for parents across all items; teachers’ ratings equaled 5.10 (SD = 0.16) overall. iSocial appeared to be an acceptable and appropriate match to the type of social problems the student was experiencing according to parents (M = 5.45, SD = 0.52) and teacher (M = 5.40, SD = .52). Parents reported a higher perception that their child benefitted from iSocial (M = 5.55, SD = 0.69) than did teachers (M = 4.70, SD = 1.34). Students also reported positive perceptions about their experiences in iSocial (M = 2.67 out of 3.00, SD = 0.42). Specifically, they indicated that the skills they learned would help them get along with others (M = 2.73, SD = 0.47) and that they found the iSocial experience relatively easy to work through (M = 2.64, SD = 0.51).

Discussion

iSocial is a 3D VLE environment designed to increase access to evidence-based practices targeting social competence for youth in schools. Schools are increasingly using distance education (DE) to provide access to important specialized curriculum (Hannum et al. 2009). Unfortunately typical DE formats of asynchronous activity oriented to content delivery, such as is common in course management systems like Blackboard, are not a good fit for meeting needs such as social competence. Given clear requirements for social and experiential learning, 3D VLE is a promising new form of DE for delivering evidence-based practices to schools with typically limited access and to students with particularly high needs. This study was the first of its kind to explore the effects of a full curriculum delivered via 3D VLE as DE to students with HFA across multiple cohorts and schools.

In response to research question one, a general review of the assessments suggest that students did experience some degree of benefit relative to their social behaviors, although parents noted this at a much higher level than did teachers. Future studies would ideally also include direct observations of students’ interactions with peers to assist in understanding these variances. Additionally, measures of some of the underlying skills that may promote social interaction, such as facial expression/emotion recognition and social perspective taking, did not indicate significant improvement. Improvements in aspects of students’ executive functioning (e.g., inhibition) were less pronounced. Again, parents noted improvements in students’ metacognitive skills at home, but teachers’ reports of these skills did not reach statistical significance. In direct performance assessments with students, some improvements in particular domains of executive functioning (e.g., cognitive flexibility) were noted, but other aspects such as attention or response inhibition did not change over time.

Several factors were considered in reviewing how this initial field test compared to previous applications of the SCI-A program (research question two). The student assessments utilized in this study were consistent with previous research on the SCI-A. As noted in these studies, some of these measures are not anticipated to demonstrate change within one semester and for each subscale. For full disclosure and to enhance the work, all data are presented. In general, the results of the student outcome data directionally are aligned with intended expectations. Achieving a larger sample may assist in further clarification and confirmation of the impact of iSocial, particularly in the larger constructs such as social competence and executive functioning. Additional student outcome results did not show an impact from iSocial as assessed. This implies the need for further research and the need to explore whether their exists some inherent deficiencies in addressing these types of skills through 3D VLE, or rather simply that improvements are needed in the translation of iSocial to better address these skills.

Many patterns of the data were, albeit not as strong, similar to previous studies using the SCI-A curriculum in face-to-face settings (Stichter et al. 2010). This is particularly noteworthy in the context of the second research question focused on the degree to which iSocial could be implemented with fidelity with the original SCI-A curriculum. Successful evidence-based practices depend on multiple constructs including implementation fidelity (Carroll et al. 2007). Governing research institutes (e.g., NIH, IES) have called for implementation fidelity yet limitations to current fidelity measures exist. Traditional methods often obtain fidelity through the employment of checklists or implementer ratings (Eames et al. 2008), thus, may lack ability to draw conclusions of effectiveness in the absence of quantitative components. Additionally, traditional measures may fail to measure multi-dimensional components such as adherence to both content and process of intervention, dosage, participant responsiveness, and program differentiation (Carroll et al. 2007). Cost of inadequate fidelity could lead to not only rejection of powerful programs but also acceptance of weaker programs (Borrelli et al. 2005). In this pilot, fidelity of implementation captured content, process, (dosage was included within these) as well as student responsiveness within behavior management plan and differentiation through specific verbal feedback. Fidelity was very high with a range of 1.93–1.97 out of 2 across all areas of fidelity. Although there remains a great deal to understand concerning translation and implementation of curricula via a 3D VLE, the ability of iSocial to support the learning environment and online guide to deliver the curriculum with integrity is highly promising.

The third aim of the study was to assess the social validity of iSocial. The iSocial experience was a highly novel one for students, teachers and parents. Although students by and large were rather versed with computers and even avatar use, neither students nor school staffs had previously ever participated in any type of synchronous interactive experience with an online teacher and peers for purposes of an educational curriculum. As a result, educational staff members monitored the students during their iSocial experience had to learn how to be real world helpers for virtual world learning. To maintain consistency in programming for students, educational staff (i.e., para professionals) assigned to support these students in traditional classrooms, were then assigned as real world helpers during iSocial. These real world helpers were trained to setup the unique computer stations for iSocial and to support any technology-based issues or personal behavior issues that occurred in the physical world of the school. These helpers did not have a role in delivering the instruction or in technology or behavioral issues that occurred in the virtual world. For the purpose of the study, teachers and parents who filled out ratings on the iSocial experience were not highly versed in the curriculum nor its inner workings. As a result, many of the general education teachers who completed social validity forms had varied levels of day-to-day interaction with the students, which may have impacted their ability to comment on the specific fit of iSocial for the students. Nevertheless, despite all of the new components required to carry out iSocial and its 3D VLE affordances, acceptance and support for the program was overarchingly positive, particularly for students and parents.

In summary, the student progress combined with the positive fidelity and social validity data show significant promise. iSocial and the use of 3D VLE show promise for meeting the needs of youth with ASD for developing social competence as well as being encouraging for the general premise that 3D VLE can be an important approach to meeting special needs through DE through schools.

Limitations

Overall, the results from the inaugural voyage of iSocial indicate the feasibility of the delivery of a social competence curriculum within schools utilizing a 3D VLE. Initial data also indicated general acceptance of this delivery and potential for student gains. In spite of these findings, there were a number of limitations to this study that should be considered. First, although three separate and unique districts were represented in this study, a total N of 11 students creates limitations for statistical analysis, generalization of results and the potential for Type 1 error. Second, although improvements on the SRS and BRIEF were either significant or in a positive direction from pre- to post, these measures were both descriptive measures and maybe in some ways impacted by the reporter’s knowledge of the program and child. For example, both assessments were garnered from general education teachers who had varied experience with the student prior to pre assessment. Both assessment tools highly recommend that the responder have strong familiarity with the student. This is difficult to control in school settings at the beginning of a new semester in secondary settings. Additionally, findings may be affected by parents’ knowledge of their child’s iSocial participation. Results in the area of Theory of Mind were not as noteworthy as desired. Despite the use of the most-often cited measures of this construct, it has been noted in previous literature that this area is ripe for enhanced measures to better capture growth in these and related constructs (Stichter et al. 2012). Future work in this area will be charged to continue to test additional potential measures that provide standardized procedures and norms while still capturing a range of social competence changes, subtle and broad.

Additionally, few models exist for this type of 3D VLE. As indicated previously, most prior work in this area has used 3D VLE to enhance classroom instruction. iSocial is designed to provide a full stand-alone curriculum potentially supplemented by face-to-face classroom instruction. The novelty of this work provides an extensive landscape for growth in understanding the technology, integration of the delivery with schools and the impact on students and educators. Hence the limitations of the scope of the current study are numerous, yet the potential for continued work in this area are extensive.

Summary

There exists a significant amount of research on key strategies to promote learning such as the use of didactic instruction, opportunities to respond and fluency building strategies (Carnine 1976; Sutherland and Wehby 2001). As a result, these strategies can be more readily conceptualized into the traditional delivery of a curriculum with relative confidence that goodness of fit is achieved to support the transfer of content from acquisition toward fluency. Less is known about this transfer within a 3D VLE. As a pilot, this study provides a great deal of fodder for additional research to determine what instructional strategies readily translate to 3D VLE, which strategies require more work to find effective translations, and which strategies may be a poor fit for 3D VLE. The initial delivery of iSocial suggests that instructional strategies with a high dependence on social interaction among students and between the online guide and students can be achieved in DE. Similarly our team did substantial work in designing and building virtual environmental supports that enabled the online guide to manage student behavior at a distance so as to achieve curriculum objectives within the required time limits of a lesson (Laffey et al. 2010b, c, 2012). The limitations of this study caution against generalizing beyond the results for these specific youth, and the findings raise several challenges for understanding how to achieve strong outcomes in all the areas targeted by the curriculum.

However, given these cautions this study reports 4 important albeit early stage findings. First, an evidenced based program dependent upon highly social and experiential teaching and learning was translated into 3D VLE and delivered with fidelity to students. Second, the 3D VLE was delivered as DE to schools which shows the potential of iSocial and programs like iSocial to address critical limitations of access for students with special needs. Third, findings for learning outcomes and social validity suggest that the iSocial approach has promise for benefits to youth, families and schools. Fourth, lessons about fidelity, achieving access, and outcomes provide a basis for improvements in how we may bring the affordances of DE and 3D VLE to address gaps in our current abilities within traditional brick and mortar settings. The above findings combined with the engaging aspect of 3D VLEs for youth, DE and 3D VLEs is a ripe area for future research and holds great promise for being a medium that can continue to facilitate educational outcomes for youth.