Introduction

Autism Spectrum Disorder (ASD) is an umbrella term which refers to a range of neurodevelopmental disorders encompassing the formerly known autistic disorder, Asperger’s Disorder, childhood disintegrative disorder, and pervasive development disorder not otherwise specified (PDD-NOS; APA 2013). Given the estimated prevalence in the United States of 1 in 68 (CDC 2014), effective evaluation of cognitive functioning is an important clinical and public health issue. ASD has been found to be associated with specific executive functioning deficits in planning, topic shifting, strategy selection, and impulsivity monitoring (Semrud-Clikeman et al. 2010). For college students with ASD, the transition into adulthood can be a challenging time (Hendricks and Wehman 2009). Beyond the challenge of academics, college students with ASD may be affected by executive functioning deficits that can have debilitating effects on the everyday life activities such as ability to work, functional independence, structuring their time, and appropriate social relations (Adreon and Durocher 2007; Fleischer 2012; Gelbar et al. 2014; Pillay and Bhat 2012). These executive functioning issues can also underlie ineffectual approaches to school work such as inadequate planning, excessive time spent on assignments, and/or undue focus on details (Shmulsky and Gobbo 2013).

Whilst there is increasing evidence for an executive dysfunction hypothesis in ASD, study results have been mixed (Hill and Bird 2006). For example, individuals with ASD tend to perform less well than neurotypical controls on tests of planning like the Tower tests (e.g., Tower of London; Tower of Hannoi; Just et al. 2007, 2012; Ozonoff and Jensen 1999). However, assessments of active inhibition of a habitual response (e.g. Stroop) have had less success in discriminating between individuals with ASD and neurotypical controls (Hill 2004). For example, studies using the classic Stroop task reveal little support for impaired inhibition in ASD (Christ et al. 2007; Goldberg et al. 2005; Ozonoff and Jensen 1999).

While studies of inhibition of prepotent responses report unimpaired inhibition in individuals with autism, studies of resistance to distractor inhibition tend to reveal difficulties (Adams and Jarrold 2012; Christ et al. 2007, 2011; Remington et al. 2009). For example, Christ et al. (2007) compared individuals with ASD to neurotypical controls on tasks aimed at assessing prepotent response inhibition, resistance to distracter interference, and resistance to proactive interference. They found individuals with ASD had difficulty ignoring distracting visual information, but prepotent response inhibition and resistance to proactive interference were relatively intact. Building on this paradigm, Adams and Jarrold (2012) compared performance of individuals with autism to neurotypical controls on a prepotent response inhibition task and a resistance to distractor inhibition task. While no differences were found for in a prepotent response inhibition stop-signal task, significant differences were found for resistance on a distractor inhibition task. The authors conclude that these results suggest a specific deficit in resistance to distractor inhibition for individuals with ASD.

While these findings help refine the executive functioning profile of individuals with ASD in laboratory settings, there is little evidence that these results are generalizable to everyday functioning in real-world environments. Consequently, there has been a movement toward alternative assessment of ASD and other neurodevelopmental disorders utilizing emerging simulation technologies (Goodwin 2008). Although there is an emerging literature investigating the use of virtual reality environments for individuals with autism (Rajendran et al. 2011; Parsons et al. 2004; Parsons et al. 2009), few studies have measured the prepotent response inhibition and/or impaired resistance to distractor inhibition of people with ASD with these virtual environments. In this paper, research is reported that combined a VR-based assessment of prepotent response inhibition with ecologically valid distractors in a virtual classroom environment.

Virtual Reality and Autism Spectrum Disorders

Some researchers have suggested that virtual reality may be especially useful in populations with ASD. A number of studies have been conducted that confirm the utility of using simulation technologies to assess ASD (Parsons et al. 2009), and Parsons and Cobb’s (2011) review of the state of virtual reality use in populations with ASD and concluded that current research supports this claim. Virtual environment studies of individuals with ASD have found that participants were able to attend to comply with examiner requests (Strickland et al. 1996; Strickland 1997). Additionally, individuals with ASD with decreased motivation (a common trait in individuals with ASD) may be more motivated by simulation technologies due to a reported attraction to computers (Trepagnier et al. 2005; Hart 2005). While the use of simulation technology has been investigated predominately in the context of therapy and rehabilitation, investigation into its use as an assessment instrument in populations with ASD is somewhat less common. The arguments for using simulation technology for assessment of ASD are similar to those for using simulation technology for therapy and rehabilitation. Individuals with ASD may be more effectively assessed using simulation technology due to findings in Ozonoff’s (1995) study suggesting individuals with ASD perform differently (i.e., better) on computer-based tasks than the same tasks that are human-administered. This finding combined with the enhanced ecological validity offered by virtual environments suggests virtual environments may be ideal for the assessment of ASD. In a more recent study Rajendran et al. (2011) used a Virtual Errands Task (VET) to investigate multitasking in high-functioning adolescents with ASD. Results revealed that adolescents with high functioning ASD had significantly greater deficits in planning flexibility, inhibition, and prospective memory than neurotypical adolescents.

While there has been an increasing interest in the use of virtual reality for children and adolescents with ASD, there is still a large gap in VR studies in adults with ASD. An exception can be found in Kandalaft et al.’s (2013) investigation of the feasibility of virtual reality for the training of social cognition in young adults. Following a 5 week intervention they found significant increases on social cognitive measures of the theory of mind, emotion recognition, and real life social and occupational functioning. Likewise, Cook et al. (2014) used virtual reality in adults with high functioning autism to investigate interference effects of action observation. Findings suggest that individuals with ASD demonstrated atypical interference effects.

Virtual Classroom Environments for Neurocognitive Assessment

One virtual reality platform that is increasingly used for assessment of individuals with neurodevelopmental disorders is the virtual classroom (Díaz-Orueta et al. 2014; Iriarte et al. 2012; Lalonde et al. 2013; Parsons et al. 2007; Parsons 2014). Within Virtual Classroom environments, participants are “seated” in a desk near the center of the classroom and are surrounded by desks, students, a teacher, a window, and a blackboard. While immersed in the virtual environment, target stimuli are presented on the blackboard. Virtual Classroom environments present distractors in various areas of the simulated classroom. Audio-visual distractors include a school bus driving by, a car driving by, a book dropping to the floor, students passing notes, a student raising his hand, the teacher answering the classroom door, and the principal entering the room. Visual distractors include a paper plane flying through the room. Audio distractors include the sound of paper crumpling, a pencil hitting the floor, an airplane passing overhead, a voice from the intercom, the bell ringing, a sneeze and a cough. These distractors are dispersed throughout the left, center, and middle of the classroom using surround sound speakers. Virtual Classroom environments may be especially beneficial for the assessment of individuals with ASD for a number of reasons. First, the ambiguity of the literature on executive functioning in ASD highlights the need for more sensitive and specific assessments of executive functioning. Second, individuals with ASD may be especially motivated by novel computerized measures, leading to a more accurate record of their attentional capabilities. Third, the Virtual Classroom is a novel test of both motor and cognitive inhibition which may be impaired in ASD (Parsons 2014).

While most Virtual Classrooms initially superimposed continuous performance task stimuli onto the blackboard board in the virtual environment (Díaz-Orueta et al. 2014; Iriarte et al. 2012; Parsons et al. 2007), a more recent iteration includes bimodal Stroop stimuli (Lalonde et al. 2013). The Virtual Classroom allows participants to respond physically via a single response key, which enables the assessment of motor inhibition. The bimodal presentation of stimuli in the Virtual Classroom Stroop paradigm enables the assessment of the participant’s ability to control interference from both external (e.g., environment) and internal (e.g., judgments) sources (Kipp 2005). The addition of distractors in the virtual environment allows for the assessment of external cognitive inhibition by requiring participants to resist distractions in the environment (Lalonde et al. 2013).

Two studies were conducted to investigate the effectiveness of the Virtual Classroom Bimodal Stroop Task for the assessment of motor and cognitive inhibitory control. In the first study, we examined performance of a large typically developing college sample on the Virtual Classroom Bimodal Stroop Task. In Study 1, we aimed to assess: (1) does the Virtual Classroom Bimodal Stroop elicit an interference effect typical of classic Stroop modalities (e.g., paper-and-pencil; and computer automated)? And (2) do distractors within the Virtual Classroom environment affect user performance in a typically developing sample?

Study 1

Method

Participants

Fifty undergraduate students (mean age = 20.37; range = 18–30; 78 % female) from a university in the southwestern United States participated in Study 1 (see. Table 1 for demographic information). For all participants, inclusion criteria were the following: all participants were required to be aged 18 years or older, with normal to corrected vision. Participants could not have an acute psychiatric condition, ADHD, or other Axis I psychopathology (diagnosed or suspected). Exclusion criteria included: epilepsy, intellectual disability (IQ < 70), and neurological impairments impacting motor movements.

Table 1 Demographic characteristics of participants in study 1
Full size table

Upon arriving to the lab, participants were instructed in the study’s procedure, risks and benefits, and alternative options (non-participation). Prior to participation, they signed written informed consent approved by the university’s institutional review board. After informed consent was obtained, basic demographic information was gathered and participants responded to questions designed to measure computer experience and usage activities: how frequently participants use a computer (e.g., “How many hours per week do you spend on the computer?”); their perceived level of computer skill on a Likert scale (1—not at all to 5—very skilled); e.g., “How many hours per week do you spend playing video games?”; and what type of games they play (e.g., role-playing, strategy, sports, etc.). All participants were administered the same battery of tests.

Procedures and Materials

All participants completed informed consent before participating in the study. Testing was conducted in dedicated testing rooms. All participants were administered (counterbalanced) the same neuropsychological battery and were assessed in the Virtual Classroom.

Cognitive Functioning

Participants were administered the Wechsler Test of Adult Reading (WTAR) to assess intellectual functioning (Wechsler 2001).

Delis Kaplan Executive Functioning System Color Word Interference Test

The Color Word Interference Test from the Delis-Kaplan Executive Functioning System (D-KEFS; Delis et al. 2001) is a multi-item paper-and-pencil presentation of the classic Stroop test. Each participant was presented with 4 cards administered in the following order: (1) “color naming card” consisting of 50 blocks of color (green, red, and blue), (2) “word reading” care with 50 color names (green, red, and blue) printed in black ink, (3) “color-word interference” card with 50 color name (green, red, and blue) printed in a discrepant ink color, and (4) “color-word interference/switching” card in which the participant performs the same task as in the color-word interference condition unless words are enclosed in a box. Per the D-KEFS manual, the examiner timed each stimulus card and noted incorrect (including self-corrected) responses. Reaction time is calculated by diving completion time by the number of stimuli completed (maximum 50). Accuracy is calculated as the number of correct responses (not including self-corrected responses) divided by the total stimuli number of stimuli presented (50).

Automated Neuropsychological Assessment Metrics Stroop Task

The Automated Neuropsychological Assessment Metrics (ANAM) Stroop task (Johnson et al. 2008) is a single-item presentation Stroop task that requires a participant to respond by pressing a computer key labeled red, green, or blue to identify a presented stimulus. Each participant used a keyboard to respond to the ANAM Stroop task. Before each test, a short 10 item practice test is given to ensure participant comprehension. The actual test is composed of color-naming, word-reading, and color-word interference conditions. In the word-reading condition, the words RED, GREEN, and BLUE were presented individually in black type. The examinee was instructed to read each word out loud and press a corresponding key for each word (1 = “red”; 2- “green”; 3 = “blue”). In the color-naming condition, a series of XXXXs were presented in either red, green, or blue type. The examinee was instructed to say the color of the XXXXs out loud and press the corresponding key. Finally, in the color-word interference condition, a series of color names (“RED”, “GREEN”, or “BLUE”) were presented individually in incongruently colored type. The examinee was instructed to say the color of the word out loud instead of reading the word and press the key corresponding to the color of the word. Examinees were instructed to respond as quickly as possible without making mistakes. Each stimulus appeared only after the examinee correctly answered the previous stimulus. The ANAM Stroop (2007) calculated the following scores: (1) Color–word score: calculated by (a) multiplying the number of correct colors named by the number of correct words named; and (b) dividing the product by the sum of number of correct colors named plus the number of correct words named; and (2) Interference score- calculated by subtracting the color-word score from the number correct on the interference task.

Virtual Classroom Bimodal Stroop task

Participants viewed the Virtual Classroom Bimodal Stroop task (ClinicaVR: Lalonde et al. 2013; Parsons 2014) using a head-mounted display (HMD; eMagin Z800 with an InterSense InteriaCube 2+ attached for tracking; see Fig. 1). The Visual stimuli could be viewed at all 360 ° around the participant. Within the Virtual Classroom Bimodal Stroop task, participants were “seated” in a desk near the center of the environment surrounded by desks, students, a teacher, a window, and a blackboard.

Fig. 1
figure 1

Participants in Study 1 viewed the virtual classroom bimodal stroop task (digital media works; ClinicaVR) using a head-mounted display (HMD; eMagin Z800 with an InterSense InteriaCube 2+ attached for tracking)

Full size image

The Virtual Classroom Bimodal Stroop task builds on the classic Stroop effect which measures cognitive interference. This task also has go/no go components (assessing motor inhibition) and external interference control (accomplished via visual and auditory distractors). The Virtual Classroom Bimodal Stroop task consists of two conditions: a block-based condition and a word-based condition. In the block-based conditions, a series of colored rectangles (red, blue, and green) appear on the blackboard within the environment while a female voice states the names of colors (red, blue, and green). Participants are instructed to click a mouse button as quickly as possible when the spoken color matches the color of the rectangle on the virtual blackboard, and to withhold a response if the colors do not match. A total of 144 stimuli are presented, with 72 targets and 72 non-targets. Participants completed a non-distraction and distraction condition of this task. Order of distraction and non-distraction tasks was counterbalanced across participants. The duration of the block-based condition was 4.8 min with a 1000 ms inter-stimulus interval (ISI).

In the word-based condition, color words are presented on the virtual blackboard (red, blue, and green) in different ink colors (red, blue, and green). These stimuli are congruent (e.g., the word “blue” in blue ink) and incongruent (e.g. the word “blue” in red ink). The colors are stated in a female voice as in the block-based condition. Participants are instructed to click the mouse when the stated word matches the color of the word presented on the virtual blackboard, and to withhold a response if the stated word and presented ink color do not match. Participants also complete both distraction and non-distraction conditions. The word-based condition was designed to measure cognitive interference in addition to the external interference control and motor inhibition assessed by the block-based condition. The duration of this condition is 4.8 min with a 1000 ms ISI. A total of 144 stimuli are presented, with 72 targets and 72 non-targets.

The complete duration of the Virtual Classroom Bimodal Stroop task is 19.2 min. Data collected in each condition include: (1) mean total reaction time, (2) mean response time (reaction time for correct responses), (3) reaction time variation (standard deviation), (4) response time variation (standard deviation), (5) count of correct responses, (6) commission error count, and (7) omission error count.

Treatment of Data

Behavioral data collected as dependent variables for Stroop tasks are traditionally reaction time (e.g., latency of response) and accuracy (number of correct responses). To compare across stimulus modalities (paper-and-pencil: DKEFS; computerized: ANAM; and virtual reality: Virtual Classroom), a throughput algorithm was used as it takes into consideration mean response time for correct responses (mean RT) and accuracy (percent correct). Throughput units are reported as correct responses per minute of available response time. (Thorne 2006).

A MATLAB scoring program was employed for data acquisition (Wu et al. 2013). This allowed for key events in the environment to be logged and time stamped with millisecond temporal accuracy. The scoring program that was adapted specifically for this study enabled the assessment of performance validity (suboptimal effort) and screening for outliers to establish data integrity: (a) identification of outliers as observations exceeding three standard deviations from the median reaction time; (b) exclusion of observations that are in both the top 1 % in speed and simultaneously in the bottom 1 % of accuracy; and (c) filtering and pattern recognition assessment for establishing feature sets using support vector machine classifiers. No outliers were identified in the current study results.

Hypothesis #1

The Virtual Classroom Bimodal Stroop task without distractors would elicit a classic “Stroop effect” similar to that found in traditional Stroop tests.

A 3 way repeated measures analysis of variance (ANOVA) was conducted to examine performance across Virtual Classroom Bimodal Stoop conditions for overall performance (i.e., throughput). A main effect of condition was observed, F (2, 98) = 70.146, p < .001, partial η2 = .589. The pairwise comparison of color-naming (M = 85.43, SD = 15.52) and interference (M = 68.33; SD = 14.75) was significant, p < .001, d = 1.13. The pairwise comparison of word-reading (M = 83.86, SD = 16.28) and interference (M = 68.33; SD = 14.75) was significant, p < .001, d = 1.00. These results indicate the Virtual Classroom Bimodal Stroop task successfully elicits an interference effect. Participants showed degraded performance in the interference condition relative to both the color naming and word reading conditions (see Fig. 2).

Fig. 2
figure 2

A comparison of Stroop modalities in Study 1. All Stroop modalities elicited an interference effect

Full size image

Comparisons were also made for each condition using reaction time and accuracy data. A 3 way repeated measures ANOVA examining accuracy revealed a main effect of condition, F (2, 98) = 8.069, p = .001, partial η2 = .141. The pairwise comparison of color-naming (M = 96.93, SD = 4.81) and interference (M = 91.61; SD = 12.85) was significant, p = .005, d = .548. The pairwise comparison of word-reading (M = 96.64, SD = 5.38) and interference (M = 91.61; SD = 12.85) was significant, p = .003, d = .511. A main effect of condition was also found for reaction time data, F (2, 98) = 105.46, p < .001, partial η2 = .683. The pairwise comparison of color-naming (M = 694.38, SD = 108.40) and interference (M = 822.51; SD = 104.40) was significant, p < .001, d = 1.204. The pairwise comparison of word-reading (M = 706.96, SD show = 108.38) and interference (M = 822.51; SD = 104.40) was significant, p < .001, d = 1.086. These results also show that the Virtual Classroom Bimodal Stroop task elicits poorer performance in the interference condition than the color naming and word reading conditions.

Hypothesis #2

Participant performance would be negatively affected by the presence of distracting stimuli.

A repeated measures ANOVA was conducted to examine the effects of distracting stimuli on overall performance (throughput) in the Virtual Classroom Bimodal Stroop Task. No significant difference between conditions was observed, F (1, 49) = .675, p = .451. Furthermore, significant differences were not observed for reaction time, F (1, 49) = 3.398, p = .071, or accuracy, F (1, 49) = 2.398, p = .128. Thus, it seems that performance of neurotypical participants is not significantly affected by environmental distractions.

Discussion

The primary results from Study 1 indicate that the Virtual Classroom Bimodal Stroop task may be validly used to examine interference control in a typically developing population. The Classroom Stroop task elicited an interference effect similar to those found in classic Stroop tasks. Furthermore, the presence of distracting stimuli did not significantly affect performance in our neurotypical participants, indicating that our sample was able to exercise external interference control effectively. Notably, none of the participants in Study 1 (using the HMD) reported experiencing simulator sickness. Overall, the results of Study 1 indicate that the Virtual Classroom Bimodal Stroop task shows promise for the ecologically valid assessment of cognitive and motor inhibitory control.

Study 2

Study 2 is the first to compare individuals with ASD and neurotypical controls using a prepotent response inhibition task with ecologically valid distractors in a virtual classroom environment. As such, the specific design parameters of the study (sample size, inclusion and exclusion criteria, etc.) correspond to the early stage of this tool’s development. The exploratory nature of the study and results is underscored. Given previous studies, we hypothesized the following:

  1. 1.

    The Virtual Classroom Bimodal Stroop task without distractors would elicit a classic “Stroop effect” similar to that found in traditional Stroop tests. This “Stroop effect” would be apparent in both ASD and neurotypical groups.

  2. 2.

    Whereas ASD-related impairment would not be evident on the Virtual Classroom Bimodal Stroop without distractors (prepotent response inhibition), impaired resistance to distractors would be present in the distracting condition.

Method

Participants

Eight adults diagnosed with high functioning autism (mean age = 22.88; range = 18–34) and ten neurotypical adults (mean age = 18.8; range = 18–20) participated in the Study 2 (see Table 2 for demographic information). For all participants, inclusion and exclusion criteria were the same as Study 1. Diagnoses of participants with ASD were made independently of this study and confirmed by the university’s Office of Disability Accommodations and primary psychologists (Autism Diagnostic Observation Schedule; Lord et al. 2002).

Table 2 Demographic characteristics of participants in study 2
Full size table

Eligible participants were recruited from agencies in the Dallas/Fort Worth area and the community. Prospective participants were contacted by email after receiving a flyer describing the study. Upon agreement to participate, prospective participants were described the study’s procedure, risks and benefits, and alternative options (non-participation). Prior to participation, they signed written informed consent approved by the university’s institutional review board. After informed consent was obtained, demographic information and computer use information identical to that collected in Study 1 was gathered. All participants were administered the same battery of tests as Study 1 with the addition of the WASI-II in place of the WTAR for a more comprehensive cognitive profile.

Procedures and Materials

All participants completed informed consent before participating in the study. Testing was conducted in dedicated testing rooms at the university. All participants were administered the same neuropsychological battery and were assessed using a 27 inch monitor to display the Virtual Classroom (see Fig. 3). Hence, the HMD was not used in Study 2. During the Virtual Classroom Bimodal Stroop task, a tracker was used to follow the head movements of each individual to map participant’s head movements to a corresponding shift in their virtual perspective on-screen. While the HMD was not used in the current study in order to minimize participant discomfort, the tracker allowed for the mapping of the participant’s head movements to a corresponding shift in their virtual perspective on the monitor. Further, the Wechsler Abbreviated Scale of Intelligence- Second Edition (WASI—II) was administered to evaluate all participants’ cognitive functioning. There were no group differences for performance on the WASI-II (Wechsler and Hsiao-pin 2011). See Table 2 for information on participant cognitive profiles.

Fig. 3
figure 3

Participants in study 2 viewed the virtual classroom bimodal stroop task (Digitial Media Works; ClinicaVR) using a 27 inch monitor

Full size image

Results

To compare participants with ASD to unaffected controls on demographic variables, one way analysis of variance (ANOVA) and Chi square analyses were conducted (see Table 2). Although no significant differences were found for intelligence, gender, computer use, or education, a significant difference emerged for age. As such, all analyses controlled for age using mixed model analysis of covariance (ANCOVA). Whilst ANCOVA has been generally found to be robust to the assumption (e.g., normality and homoscedasticity) violations that frequently arise with the analysis of reaction time data, we also utilized Mann–Whitney U tests that do not rely on such assumptions. Given that the resulting pattern of statistically significant and nonsignificant findings was identical to that found with ANCOVA, the ANCOVA results are reported herein. Due to the pilot nature of the study, significance levels for multiple tests were not adjusted. Analyses were performed using SAS (SAS System for Windows, version 9.1.3, SAS Institute Inc., Cary, North Carolina).

For all analyses of variance (ANOVAs and ANCOVAs) that included a variable with more than 1 ° of freedom, pairwise comparisons were used to identify the precise nature of any main effects. Additionally, all significant main effects and interactions were followed up with pairwise comparisons in order to determine the nature of these effects. Effect sizes were proffered for each analysis. Correlational analyses were conducted to assess for similarities across modalities. A sequentially rejective test procedure based on a modified Bonferroni inequality was used on significant results to prevent inflation of Type I error rates (Rom 1990). Additionally, a Greenhouse–Geisser correction was used for all reported main effects and interactions with greater than 1 ° of freedom.

Hypothesis #1

The Virtual Classroom Bimodal Stroop task without distractors would elicit a classic “Stroop effect” similar to that found in traditional Stroop tests. This “Stroop effect” would be apparent in both ASD and neurotypical groups

For groupwise comparisons, a 2 (Group: ASD and Control) by 3 (Classroom Condition: Color, Word, Interference) mixed analysis of covariance (ANCOVA) controlling for age was conducted in order to examine differences in overall performance (i.e., throughput) among Stroop conditions. No main effect of group was observed F (1, 15) = .192, p = .667. A main effect of Stroop conditions was observed, F (2, 30) = 8.08, p = .002, partial η2 = .35. The pairwise comparison of color-naming (M = 88.47, SD = 1.63) and interference (M = 68.85; SD = 2.55) was significant, p < .001, d = 2.19. The pairwise comparison of word-reading (M = 84.01, SD = 1.72) and interference (M = 68.85; SD = 2.55) was significant, p < .001, d = 1.64. Although both groups showed poorer performance in the interference condition than the color naming and word reading conditions, a 2 (Group: ASD and Control) by 3 (Stroop modality type: DKEFS, ANAM, Classroom) mixed ANCOVA controlling for age revealed no significant differences between groups for any Stroop modality, F(1,15) = 2.50, p = .134. All Stroop modalities successfully elicited an interference effect in both groups. See Fig. 4a for a comparison of Stroop modalities across conditions for the neurotypical controls. Figure 4b illustrates a comparison of Stroop modalities across conditions for the ASD group. Groupwise comparisons of response time and accuracy (percent correct) data were also conducted. A 2 (Group: ASD and Control) by 3 (Classroom Condition: Color, Word, Interference) ANCOVA controlling for age examining accuracy revealed no main effect of condition, F(2, 30) = 1.25, p = .303 and no main effect of group, F(1, 15) = .400, p = .134. A similar 2 × 3 ANCOVA controlling for age was conducted to examine response time. A main effect of condition was observed, F (2, 30) = 5.92, p = .007, partial η2 = .283. The pairwise comparison of color-naming (M = 680.24, SD = 52.39) and interference (M = 806.96; SD = 70.26) was significant, p < .001, d = 2.045. The pairwise comparison of word-reading (M = 716.27, SD = 58.31) and interference (M = 806.96; SD = 70.26) was significant, p < .001, d = 1.405.

Fig. 4
figure 4

a A comparison of Stroop modalities across conditions. All Stroop modalities successfully elicited an interference effect in the control group. b A comparison of Stroop modalities across conditions. All Stroop modalities successfully elicited an interference effect in the ASD group

Full size image

Hypothesis #2

Whereas ASD-related impairment would not be evident on Virtual Classroom Bimodal Stroop without distractors (prepotent response inhibition), impaired resistance to distractors would be present in the distracting condition.

A 2 (Group: ASD and Control) by 2 (Condition: with and without distractions) mixed ANCOVA controlling for age was conducted to investigate the effects of the inclusion of distracting stimuli within the Virtual Classroom Bimodal Stroop on participants with ASD and neurotypical controls. The ANCOVA revealed an interaction of group and condition, F (1, 15) = 5.09, p = .039, partial η2 = .25. Specifically, performance scores of the group with ASD (M = 59.66; SD = 12.35) were significantly lower than performance scores of the neurotypical control group (M = 71.02; SD = 3.04), F (1, 15) = 5.99, p = .026, partial η2 = .27, in the Virtual Classroom Bimodal Stroop with distractions. The presence of distracting stimuli appears to have negatively impacted performance in the ASD group relative to the neurotypical control group (see Fig. 5).

Fig. 5
figure 5

ANOVA results revealed a significant interaction between group and VR Classroom condition. While no significant differences were found between groups in the No Distraction condition, distracting stimuli significantly degraded performance in the ASD group, but not in the control group

Full size image

No difference was observed between groups in the without distraction condition F (2, 17) = .039, p = .962. Furthermore, a repeated measures ANOVA was conducted to examine within-subject effects. Unaffected participants did not significantly differ in their overall performance in the without distraction condition (M = 68.88, SD = 10.70) and distraction condition (M = 71.02, SD = 7.19), F (1, 9) = 1.349, p = .275. Results showed no significant difference between overall performance in the without distraction (M = 68.73, SD = 3.552) and distraction (M = 59.66, SD = 4.37) condition for participants with ASD, F (1, 7) = 4.210, p = .079.

Response time and accuracy data were separately investigated. A 2 (Group: ASD and Control) by 2 (Condition: with and without distractions) mixed ANCOVA controlling for age revealed no main effect of condition for response time data, F (1, 15) = 3.850, p = .069, and no main effect of group, F (1, 15) = .467, p = .505. A similar ANCOVA was conducted to investigate accuracy. No main effect of condition was observed, F(1,15) = 2.586, p = .129. However, results showed a main effect of group, F (1, 15) = 6.401, p = .023, partial η2 = .299, and a significant interaction between condition and group, F (1, 15) = 8.89, p = .009, partial η2 = .372. Specifically, individuals with ASD (M = 86.81; SD = 13.42) were significantly less accurate in their responses than unaffected participants (M = 97.64; SD = 1.61) only in the distraction condition, F (1, 17) = 9.70, p = .007.

Thus, it seems that when under conditions of distraction, individuals with ASD are compromised in their ability to activate external distractor inhibition, though their response time may not suffer.

Study 2 Discussion

In this initial pilot study, we aimed to compare the Virtual Classroom Bimodal Stroop task with paper-and-pencil (D-KEFS) and computerized (ANAM) Stroop tasks. We hypothesized ASD-related impairment would not be evident on the paper-and-pencil Stroop, computer automated Stroop, or the Virtual Classroom Bimodal Stroop without distractors (prepotent response inhibition). Instead, the differentiating factor between individuals with ASD and neurotypical controls would be impaired resistance to distractors in the ASD group during the distracting condition of the Virtual Classroom Bimodal Stroop task. The primary results were (a) the classic Stroop pattern found in traditional modalities (e.g., paper-and-pencil; computer automated) was observed in the Virtual Classroom Bimodal Stroop task; (b) significant differences were not observed between individuals with ASD and neurotypical controls on the paper-and-pencil, computerized, and Virtual Classroom Bimodal Stroop task without distractors; and (c) individuals with ASD performed significantly more poorly on the Virtual Classroom Bimodal Stroop task with distractors than without distractors. Their performance accuracy seemed to be most significantly affected. These results suggest the potential of the Virtual Classroom Bimodal Stroop task to distinguish between prepotent response inhibition (non-distraction condition) and resistance to distractor inhibition (distraction condition) in adults with high functioning autism. Our results are particularly interesting in light of the nature of the study’s participants. Our sample of individuals with ASD is quite high functioning, with subtle cognitive deficits. One might expect to find similar or more pronounced results in more low functioning population.

The Virtual Classroom Bimodal Stroop Elicited the Classic “Stroop Effect”

The non-distraction condition of the Virtual Classroom Bimodal Stroop was performed in a manner consistent with paper-and-pencil and computerized versions of the Stroop task. In both groups, participants performed better (i.e., greater accuracy and faster response time) on color naming and word reading than they did on interference conditions. No main effect was found for Stroop type in analyses, and no interaction of Stroop type and Stroop condition was observed. These findings are consistent with previous studies comparing traditional (paper-and-pencil and computerized) stroop modalities to virtual reality versions of the Stroop (Armstrong et al. 2013; Henry et al. 2012; Lalonde et al. 2013; Parsons et al. 2013).

Likewise, all three Stroop modalities found no difference in performance between groups. This finding is consistent with findings that the Stroop does not consistently discriminate between individuals with ASD and neurotypical controls (Hill 2004). In fact, a number of studies have found similar inhibition performance between individuals with ASD and unaffected controls (e.g., Goldberg et al. 2005; Ozonoff and Jensen 1999; Ozonoff and Strayer 1997). It is important to note that while there were no group differences across conditions (color naming, word reading, interference) for the non-distraction condition of the Virtual Classroom Bimodal Stroop in our ASD and unaffected participants, studies of the Virtual Classroom Continuous Performance Task have shown performance differences between participants with ADHD and unaffected controls. These results may reflect findings from studies using Stroop and continuous performance tests to compare participants with ADHD and autism. Johnson et al. (2007) found that while an ADHD group showed clear deficits in response inhibition and sustained attention as measured by the Sustained Attention to Response Task (Robertson et al. 1997), the high functioning autism group showed no sustained attention deficits.

Virtual Classroom Bimodal Stroop without distractors did not reveal prepotent response inhibition, but impaired resistance to distractors was present in the distracting condition.

Adults with high functioning autism performed comparably to neurotypical controls on traditional (paper-and-pencil and computer) versions of the Stroop paradigm. While ASD-related impairment was not evident during the Virtual Classroom Bimodal Stroop without distractors (prepotent response inhibition), impaired resistance to distractors was present during the distracting condition. These results reflect growing support for distinguishing between prepotent response inhibition and resistance to distractor inhibition in autism (Adams and Jarrold 2012; Christ et al. 2007, 2011; Geurts et al. 2008; Remington et al. 2009). The distractor condition in the Virtual Classroom Bimodal Stroop task is reflective of Adams and Jarrold’s (2012) modification of prepotent response inhibition tasks (e.g., the stop-signal task), and resistance to distractor inhibition tasks (e.g., the Eriksen flanker task). Such tasks enabled the investigators to better investigate resistance to distractor inhibition in individuals with autism. Similar to our findings with the distractor condition of the Virtual Classroom Bimodal Stroop task, they also found that individuals with autism had impaired resistance to distractor inhibition, but unimpaired prepotent response inhibition.

General Discussion

A number of practical implications can be inferred from the current study’s findings. First, we demonstrated in Study 1 that the Virtual Classroom Bimodal Stroop Task successfully replicates the interference effect of classic Stroop tasks and may be used to assess cognitive and motor interference control. While paper-and-pencil and computerized measures such as the DKEFS and the ANAM may provide adequate control, predictive value may be compromised by the lack of ecological validity. These measures do not replicate the complexity of the worlds in which individuals live and interact. Therefore, outcomes of this type of assessment may not accurately represent deficits in cognitive functioning. Virtual environments may improve upon the shortcomings of paper-and-pencil and computer-based assessment by providing a highly ecologically valid yet scientifically controlled testing environment (Parsons et al. 2007).

Second, we demonstrated in Study 2 that simulation technology may be successfully used in individuals with ASD for purposes of assessment and potentially for rehabilitation. To date, most use of simulation technology in populations with ASD has been geared toward rehabilitation and social skills training (e.g., Parsons and Mitchell 2002; Parsons et al. 2006; Kandalaft et al. 2013). The current study demonstrates that using simulation technology for the assessment of interference control in individuals with ASD is also beneficial. Particularly, the Bimodal Virtual Classroom Stroop task was sensitive enough to differentiate between internal (prepotent thought responses) and external (environmental distractions) interference control. As in Study 1 (using the HMD) participants in Study 2 (using the monitor only) did not report nausea/discomfort. It is important to note that the participants in Study 2 did not use the HMD. Instead they used a monitor. The use of an HMD for the population in Study 2 (persons with high functioning autism) is an area that requires further research.

Our results indicated that individuals with high functioning ASD are differentially impacted by everyday distractions in their environment. That is, cognitive functioning may suffer in conditions of external distraction in which neurotypical individuals may be unaffected. Prior research has highlighted the significance of distractor type in studies of ASD. For example, Sasson et al. (2008, 2011) demonstrated evidence of abnormal attention allocation in individuals with ASD. In these studies, toddlers and children with autism displayed reduced exploratory attention in visual displays, but increased perseveratory attention to high autism interest (HAI) objects within these displays. Sasson et al. described HAI objects as objects associated with the target of circumscribed interests in ASD. Critically, without HAI objects, visual attention patterns between ASD and unaffected participants did not differ. This may reflect a specific pattern wherein individuals with ASD tend to perseverate on HAI objects, exploring them in more detail, while missing other visual information. HAI objects included items such as trains, blocks, planes, road signs, and electronics.

Furthermore, Riby et al. (2012) demonstrated that attention allocation to faces is also abnormal in individuals with ASD. Whereas faces tend to capture unaffected individuals’ attention more than objects, individuals with ASD do not display this pattern. Furthermore, Ribey et al. showed that level of functioning in individuals with ASD was significantly correlated with level of distraction by faces. Faces attracted more attention from higher functioning individuals than lower functioning individuals.

In light of these and similar studies, it is important to understand if specific types of distractions (i.e., trains, electronics, faces) differentially impact cognitive functioning. Specific information on the types of distractors that may impact cognitive functioning in people with ASD will aid caretakers and service providers in optimizing the environments in which people with ASD learn (e.g., classrooms, therapy rooms). Moreover, college students with high functioning ASD, such as those in our sample, will likely benefit from academic accommodations. Increased time for test-taking in private rooms is a common accommodation for individuals with ADHD, and our results suggest this may also be beneficial to individuals with high functioning ASD.

Importantly, this study highlights an important advantage of using virtual environments in psychological assessment: enhanced ecological validity. Rather than testing patients in sterile laboratory or clinical environments that do not reflect the complexity of the real world, virtual environments allow for the patient to be assessed in a situation with the complexity of the real world and the control of a laboratory.

Limitations and Projections for Future Studies

Our findings should be understood in the context of some limitations. First, although the current study counterbalances order of presentation and distractor conditions, future studies are needed to assess the mediating effects of conflict processing differences between distractors that are low and high in social content. Further, the high social distractors versus low-social distractors could be separated in future studies to include a high-social-distractor condition in addition to the current non high-neutral-distractor and non-distractor conditions.Second, recent research indicates that healthy, undergraduate research subjects in a “neuropsychological experiment” may put forth suboptimal effort (An et al. 2012). For the Virtual Classroom Bimodal Stroop, we examine performance using scoring algorithms to screen for outliers and assess data integrity (Wu et al. 2013). It is important to note that the data are not free of potential confounds due to poor effort. Screening was done at the time of testing and also during the data analysis process to eliminate obvious poor effort, but data may still contain some individuals who provided less than optimal effort. This situation is true for most normative neuropsychological data and particularly true for computerized testing given the reduced interaction with the examiner. Future studies should make attempts at including well standardized effort tests with other cohorts (individuals with ASD and neurotypical controls) to assess the sensitivity and specificity of the Virtual Classroom Bimodal Stroop. Furthermore, while Study 1 had 50 participants, the findings from Study 2 are based on a relatively small sample size. Although Study 1 gives a clear picture of functioning in a healthy college sample, the results of Study 2 would be aided by a larger sample of participants. Although our findings need to be replicated to assess potential impact of sample size, effect size analysis revealed strong effects for all analyses, which reduces the likelihood of anomalous results relative to sample size.

Whilst the Virtual Classroom Bimodal Stroop needs to be further validated, current findings provide data regarding the potential of the Virtual Classroom Bimodal Stroop for fractionation of inhibition of prepotent responses and resistance to distractor inhibition, which is particularly impressive in our high-functioning sample. The Virtual Classroom Bimodal Stroop replicated and extended the Stroop effects found in paper-and-pencil (e.g., D-KEFS) and computer-automated (e.g., ANAM) versions of the Stroop. Furthermore, the increased ecological validity of the Virtual Classroom Bimodal Stroop may be useful in creating a greater amount of variability in performance levels of participants for investigations into inhibition of prepotent responses and resistance to distractor inhibition in real-world environments.