Therapist treatment integrity is important, and routine evaluation of clinical skills is recommended (Shore et al. 1995). Therapist behavior may be influenced by the presence of an observer, and treatment integrity may be greater as compared to those sessions when an observer is absent (Kazdin 1977, 1979). With the increase in the number of online programs, it behooves researchers to identify effective ways to provide remote supervision with the goal of maintaining treatment integrity in staff members’ performances.

In a study conducted by Brackett et al. (2007), job coaches were trained to decrease the completion of work tasks for their assigned workers. Results indicated that conspicuous observations produced higher treatment integrity as compared to inconspicuous observations. A self-monitoring recording system increased treatment integrity during inconspicuous observations. Other researchers have incorporated teleconsultation, an Internet-based consulting method, into supervision and observations during functional analyses (Frieder et al. 2009; Machalicek et al. 2009). Board certified behavior analysts used teleconsultation to provide real-time feedback to teachers conducting functional analyses. The teachers were able to maintain high assessment integrity, which the authors attributed, in part, to the Web-based consultation and feedback. Wacker and his colleagues (2013) conducted a study wherein teleconsultation was used to successfully teach parents to teach their children diagnosed with autism functional communication training.

Programs like Skype™ provide video and audio feedback in real time, allowing two people to communicate from remote locations. Skype™ supervision is being utilized by some programs; however, videoconferencing has not been evaluated as a way of monitoring or improving treatment integrity, and this study attempted to address this need. The purpose of this study was to compare the effects of an Internet communication-based observation in real-time, via Skype™, in vivo observations and covert observations on the treatment integrity of three direct care staff working with children diagnosed with autism.

Three therapists in an applied behavior analysis (ABA) clinic participated in this study. One week prior to baseline, the clinic supervisor, who was also the primary author of the current study, taught the therapists the error-correction procedures separately, using behavior skills training (BST) (Nigro-Bruzzi and Sturmey 2010). First, the supervisor reviewed a written protocol of the error-correction procedure. Then, the supervisor modeled the error-correction procedure. Next, the therapists implemented the error-correction procedure. Finally, the supervisor provided the therapists feedback on their performances. The mastery criterion was 100 % correct implementation of the error-correction procedure for three consecutive trials across three consecutive days. Therapists targeted the same programs across all three conditions (covert, in vivo, and Skype™) to ensure that programs were equivalent across all experimental conditions.

Two months prior to training the error-correction procedure, a camcorder was positioned on a bookshelf in the front of the room. The staff was told that the camera was going to be used by the supervisor to record group sessions. Prior to the therapists’ arrival the following day, the recording indicator light on the camcorder was disabled, allowing the camcorder to record without the visual cue of the red light.

During baseline/covert observations, the video camcorder was used to covertly record the therapists’ sessions. It was turned on and off in the absence of the therapists. The positioning of the camcorder on the shelf remained constant throughout the study. The baseline data were collected by covertly scoring the integrity of the error-correction procedure. The supervisor walked in and out of the cubicle and remained within view when outside of the cubicle. The supervisor did not provide direct feedback on the error-correction procedure and did not carry the materials used during in vivo and Skype™ observation sessions.

During the in vivo observation condition, the supervisor entered the cubicle and informed the therapist that an observation would be occurring for about 15 min. The supervisor sat approximately 5 to 8 ft from the therapist with a pen and clipboard. The supervisor observed the therapist working with the student and collected data on the error-correction procedure. The therapist was not able to see the error-correction data collection sheet and was unaware that data for the error-correction procedure were being collected. If the therapist incorrectly implemented a step, the supervisor would interrupt the procedure and provide corrective verbal feedback and would record this step as incorrect. Subsequent steps of the error-correction procedure were counted correct if completed correctly. Praise was delivered on an FR1 schedule of reinforcement for the error-correction procedure and on a VR5 schedule for other treatment procedures.

During the Skype™ condition, the supervisor used the Skype™ program installed on an iPad2™ or the laptop with the addition of the webcam to video call the therapist from a remote location. The supervisor’s remote location ranged from a building outside of the school within walking distance to a separate city 45 miles away. The therapist answered the call and positioned the iPad2™ so the supervisor could see the therapist, student, and working space. The supervisor was also visible to the therapist during the session. The observation was started with the supervisor confirming the audio and visual feedback. Afterward, the supervisor collected data on the error-correction procedure. A flip cam was also set up out of the therapist’s view to record sessions for interobserver agreement (IOA) purposes. Feedback was provided throughout the session in a manner procedurally identical to the in vivo observation sessions.

All the participants in the study were debriefed 1 week after the study. Participants were shown their individual data, and the purpose of the study was explained. The supervisor noted that the study results and their participation were strictly for research purposes and would have no effect on their annual performance review for the clinic.

The results of this study demonstrate that both in vivo and Skype™ overt observation methods resulted in similarly high levels of treatment integrity in three ABA therapists with varying levels of experience and training. All the participants’ responses reached the mastery criterion for the error-correction procedure within 7 days. Both the in vivo and Skype™ observation methods produced higher treatment integrity for the error-correction procedures, as compared to the covert observation.

Figure 1 displays the treatment integrity data for the error-correction procedures in baseline and across observation conditions for all three participants. Elizabeth’s baseline performance was between 52 and 76 %. Treatment integrity ranged from 85 to 97.2 % during Skype™ observations and from 63 to 100 % during in vivo observations. Performance during covert observations during treatment was below baseline ranging from 36 to 67 %. Minnie’s baseline performance was between 18.6 and 64 %. Baseline trends continued following the introduction of treatment ranging from 37 to 60.2 %, supporting the occurrence of reactivity during supervisor’s overt observations. Treatment integrity ranged from 79 to 96 % during Skype™ observations and from 74 to 97 % during in vivo observations. Michelle’s baseline performance was from 40 to 63 %. Treatment integrity ranged from 92 to 96 % during Skype™ observations and from 89 to 96 % during in vivo observations.

Fig. 1
figure 1

Percentage of steps implemented correctly for the error-correction procedure during overt observations, in vivo observations, and Skype™ observations for Elizabeth, Minnie, and Michelle

Full size image

Treatment integrity was higher in both overt observation conditions as compared to the covert condition for all the participants. While there was a clear and steady separation of the data paths between the overt and covert observation conditions for Minnie and Michelle, there appeared to be a trend toward convergence of the data paths for Elizabeth. Elizabeth was the most experienced (level 2) therapist, and it may be the case that treatment integrity during covert observations increases over time in those individuals with greater clinical experience.

A limitation of the study is that the remote supervision via Skype™ was never conducted from more than 45 miles away. Courses offered over the Internet may be across much larger distances which may affect connection or visual and audible clarity. Future research may consider examination of concomitant performance of the clients across conditions to see whether these differences in treatment integrity produce significant differences in client learning. Remote supervision is coming into greater demand, and this study demonstrates one way to effectively supervise students’ and clinicians’ long distance.

Method

Three therapists working at an outpatient clinic offering ABA therapy participated in the study. Each therapist worked with one young student diagnosed with autism. The participants were one level 2 therapist (Elizabeth) and two level 1 therapists (Minnie and Michelle). The level 2 therapist has more clinical experience and in-house trainings than the level 1 therapists. A supervisor from the clinic observed the therapists throughout the study. The supervisor had been working at the clinic for 11 months, was enrolled in a master’s program in behavior analysis, and was a board-certified assistant behavior analyst (BCaBA) at the time of the study. The experimenters received approval to use covert observations of the research participants in the study from the University of Houston–Clear Lake, Committee for the Protection of Human Subjects (CPHS). The study was conducted in an outpatient ABA clinic. Each participant had her own 6 ft × 6 ft cubicle which contained a child-sized table, two chairs, and materials to conduct clinical training sessions.

Two iPad2™ units were used throughout the study, one by the supervisor and one by the therapists. Both iPad2™ units were equipped with Skype™. A laptop was also equipped with the Skype™ program for use by the supervisor if the iPad2™ was unavailable. A detachable webcam was used to transmit video feed between the laptop and the iPad2™. A flip video camera was used to covertly record Skype™ sessions and to calculate IOA. A camcorder and mount was used to record sessions.

Skype™ videoconferencing software is an efficient and cost-effective option. Skype™ is a widely used teleconference program and uses physical, technical, and administrative safeguards including encryption to protect user privacy. However, despite these safeguards, Skype™ is not Health Insurance Portability and Accountability Act (HIPAA) compliant. In the current study, the participants signed consent to use Skype™. Research participants and clients need to sign consent if Skype™ is going to be used. If Skype™ is not an option, there are other videoconference software programs available (Lee et al. 2015). Users may also consider limiting the types of practices when using Skype™ (Legal and Regulatory Affairs Staff 2014).

The study employed a multiple baseline across participants design with an embedded multi-element design. Therapist treatment integrity of the error-correction procedure was the primary dependent variable. Therapists implemented an error-correction procedure, which included 17 steps for motor responses (e.g., receptive identification) and 12 steps for verbal responses (e.g., answering questions). It was a most-to-least prompting procedure that was presented consistently as described below. Error-correction for the motor responses included the following: (1) present discriminative stimulus (SD), (2) physical prompt, (3) verbal praise, (4) repositioning materials, (5) present SD, (6) modeling response, (7) verbal praise, (8) repositioning materials, (9) present SD, (10) gesture prompt, (11) verbal praise, (12) repositioning materials, (13) present SD, (14 and 15) two maintenance trials (mastered tasks), (16) present SD, and (17) tangible reinforcer for correct responses.

The error-correction procedure for verbal responses included the following: (1) present SD, (2) full verbal prompt, (3) verbal praise, (4) present SD, (5) partial verbal prompt, (6) verbal praise, (7) present SD with opportunity to answer independently, (8) verbal praise, (9 and 10) two maintenance trials, (11) present SD, and (12) tangible reinforcer for correct responses. Data were collected on the first four motor and first four verbal error-correction procedure trials of the observation. If corrective feedback was provided for a step in the error-correction procedure, that step was counted incorrect. Data were recorded on a trial-by-trial basis with each trial yielding a percentage for the number of steps implemented correctly out of the total number of steps of the error-correction procedure. Interobserver agreement was calculated for 33 % of all conditions. The agreement was 93 % (range 86–100 %) for Elizabeth, 92 % (range 84–98 %) for Minnie, and 94 % (range 88–100 %) for Michelle.