In the last decade, progress has been made in the development and identification of evidence-based treatments (EBT) for a range of psychosocial difficulties, including those associated with child abuse. Two national panels of experts on child maltreatment recently identified EBTs (Saunders et al. 2003) and “Best Practices” for the treatment of child abuse (Kaufmann Foundation 2004). Parent–Child Interaction Therapy (PCIT; Eyberg 1999; Hembree-Kigin and McNeil 1995) was included in both expert reports as a promising treatment approach for child physical abuse.

The next step for EBT, including PCIT, is disseminating and implementing them in community settings, which will require the training of community-based practitioners (Kauffman Foundation 2004; Levant 2005). Currently, no standards exist for the best way to train practitioners (Addis et al. 1999) and little is known about effective training methods (Miller et al. 2004). Accordingly, the purposes of the current investigation were: (a) to investigate the effectiveness of a treatment manual for disseminating PCIT, (b) to evaluate the effects of didactic and experiential workshop formats on participant outcomes, and (c) to provide preliminary information on therapist characteristics associated with learning PCIT.

To date, two common postgraduate learning formats have been used to disseminate EBT: written materials (e.g., treatment manuals) and workshops. While treatment manuals have been essential to evaluating innovative interventions due to their detailed, session-by-session descriptions of therapy activities, they also have sparked considerable debate (e.g., Lambert 1998; Wilson 1998). Some studies have found that the use of a treatment manual is associated with improved patient outcomes (e.g., Frank et al. 1991; Schulte et al. 1992); however, others have suggested that treatment manuals neglect the idiographic study of the individual patient, overlook the importance of client and therapist variability, and de-emphasize and stunt clinical innovation (Davison and Lazarus 1994, 1995). Therapy process research supports the importance of the therapeutic alliance (e.g., Castonguay et al.1996; Najavitis and Strupp 1994), which treatment manuals have been criticized as ignoring (Silverman 1996).

Many have hypothesized that manuals will be helpful as a dissemination tool (e.g., Addis et al. 2006; Kendall and Beidas 2007); however, treatment manual investigations have focused on treatment outcomes rather than the manuals’ effectiveness as a dissemination technique. Therefore, their usefulness remains unclear (Addis and Waltz 2002). Sholomskas et al. (2005) recently offered the first study examining potential changes in clinicians’ knowledge and ability to implement new approaches after only reading a treatment manual. They found that some improvement occurred for clinicians after reading a treatment manual; however, the improvement was smaller and more short-lived than for clinicians in seminar- or web-based training. A treatment manual was helpful, but not sufficient in this study which considered Cognitive Behavior Therapy for adult substance abuse.

Other practitioner training efforts typically consist of one-half to two-day workshops without competency evaluations or follow-up. Labeled by some as the ‘train and hope’ approach (Henggeler et al. 2002), this training format is commonly used in continuing education (CE). While less is known about the effectiveness of mental health CE (VandeCreek et al. 1990), research focused on medical CE indicates that these workshops do little to change behavior. Instead, extended formats, interactive techniques (e.g., case discussion, role-play), and sequencing sessions lead to greater skill acquisition (Davis et al. 1999, 1992). Workshops on EBTs have produced mixed results. Some studies suggest that 1-day workshops are beneficial (Rubel et al. 2000) whereas others do not (Miller and Mount 2001). Recent training studies (Miller et al. 2004; Sholomskas et al. 2005) have indicated that didactic workshops are helpful if supplemented with feedback and continued consultation.

There also has been a limited focus on practitioners within EBT implementation; however, existing studies demonstrate that community-based practitioners are most often experienced, eclectic, masters- and bachelor-level practitioners (Herschell et al. 2008), which is in contrast to previous training studies that have primarily included inexperienced therapists or doctoral-level participants (Alberts and Edelstein 1990) and to EBT studies that include masters and doctoral level cognitive-behavioral and behavioral treatment providers. Among master’s level professionals there are several type of degrees that could be obtained to treat patients (e.g., counseling, educational, or clinical psychology, social work), all of which are typically different than degrees obtained by EBT developers, researchers, and disseminators. Several therapist characteristics have been examined in the research literature that may impact EBT dissemination (e.g., age, degree type, gender, pre-training competence, professional background). However, inconsistencies have been noted in study findings. For example, some studies have found that professional background matters (e.g., Fadden 1997), whereas other studies have found that it does not matter (e.g., DeViva 2006; Hawkins and Sinha 1998).

Parent–Child Interaction Therapy

PCIT is a two-stage (CDI or Child Directed Interaction and PDI or Parent Directed Interaction) intervention that was developed and originally tested for the treatment of disruptive behavior disorders in young children (Herschell et al. 2002). CDI emphasizes the quality of the parent–child relationship by incorporating behaviorally based play therapy techniques. Parenting skills such as praise, reflection, imitation, description, and enthusiasm are emphasized whereas questions, commands, and criticisms are discouraged. PDI concentrates on establishing a structured and consistent discipline program. For each phase of treatment, CDI and PDI, parents attend one didactic session during which the therapist describes the skills of the interaction and provides the rationales for their use. Following the initial didactic session, parents and their child attend weekly coaching sessions together. The treatment protocol is assessment-driven and is not time-limited; progress in the parent-child interactions is coded at each session, and treatment is completed when parents have mastered the skills of CDI and PDI and the child’s behavior is within normal limits (Hembree-Kigin and McNeil 1995).

Treatment outcome data from studies including children with disruptive behavior disorders indicate that completion of PCIT decreases child behavior problems, increases parent skill, and decreases parent stress (see Gallagher 2003; Herschell et al. 2002 for reviews). Similarly, data indicate that parent-child dyads with child physical abuse histories respond well to PCIT (Chaffin et al. 2004; Timmer et al. 2005), perhaps, in part, because of the high comorbidity between physical abuse and disruptive behavior (see Kolko 2002 for a review). A recently completed randomized trial with 110 physically abusive parent-child dyads tested the efficacy of PCIT in preventing re-reports of physical abuse (Chaffin et al. 2004). Findings indicated that at a median follow-up time of 850 days, 19% of participants in the PCIT condition had a re-abuse report, whereas 36% of participants in the PCIT plus individualized, enhanced services condition had a re-abuse report and 49% of participants in the community group had a re-abuse report.

Similar to other parent management training programs, PCIT’s content requires therapists to have a strong working knowledge of behavioral principles, developmental psychopathology, and general therapy skills (e.g., rapport building); however, PCIT’s format is different from other models. Most notably, PCIT therapists ‘coach’ parents to skill mastery levels, which involves a parent wearing a bug-in-the-ear device while a therapist is behind a one-way mirror observing the parent interact with his\her child. Via the bug-in-the-ear device, the therapist provides the parent with detailed feedback about how to increase parent skills, and thereby effectively manage the child’s behavior. Progress in the parent–child interaction is coded at each session, and treatment phases are completed once parents have mastered skills, which are measured by pre-defined skill mastery criteria (Eyberg and Calzada 1998). Given that PCIT involves a unique, skill-based, ‘mastery’ treatment format with the use of specialized equipment (i.e., bug-in-the-ear device, one way mirror) and techniques (i.e., coaching), it may require a more experiential and a longer training time to learn than other therapies with more traditional formats.

Three main questions were addressed by the present study: (a) Can community practitioners read a treatment manual and acquire and master PCIT knowledge and skills? (b) Is an experiential training format more useful than a didactic format? and (c) Are particular therapist characteristics associated with training success? It was hypothesized that reading a manual would increase knowledge, but not skills (Alberts and Edelstein 1990; Beutler and Kendall 1995; Luborsky 1990; Sholomskas et al. 2005). It was predicted that both the experiential group and didactic groups would evidence similar gains in knowledge and satisfaction; however, the experiential group would demonstrate greater gains in skills (Alberts and Edelstein 1990; Beutler and Kendall 1995; Davis et al. 1999; VandeCreek et al. 1990). It also was predicted that participants’ theoretical orientation and educational level would influence their skill acquisition and knowledge gain such that those who reported a behavioral orientation would evidence greater gains relative to participants reporting another orientation. Educational level (type of masters-degree) was included given the attention paid to degree type in EBT discussions (e.g., Addis and Krasnow 2000); however, it was unclear what type of degree might be associated with training success so no directional predictions were made a priori. To examine these questions, participants were assigned to either the didactic or experiential training group and assessed at four times: baseline, after reading a treatment manual, after one didactic training day, and after completion of a second training day that was didactic or experimental. This protocol was approved by two Institutional Review Boards (IRB), the University of California and West Virginia University IRBs. Each participant and supervisor was informed of the rights of research participants; particular attention was paid to detailing that participation was voluntary and confidential (information that could be identified would not be released to supervisors or other agency personnel) and that participants had the right to withdraw at any time.

Method

Participants

Selection

The Child and Adolescent Abuse, Resource, and Evaluation Center (CAARE Center), University of California Davis Children’s Hospital is devoted to medical and mental health treatment, training, and research with abused and neglected children. In 1997, the CAARE Center began providing PCIT to maltreated and at-risk children and their parents with the support of funding through the California Governor’s Office of Emergency Services. After some initial success, the Office of Emergency Services contracted with the CAARE Center to develop similar PCIT programs in 13 community-based mental health agencies serving child maltreatment populations beginning in 2000. The 13 agencies were identified through a request for applications from the Office of Emergency Services (www.oes.ca.gov). Agency directors submitted applications, and 13 agencies were selected to participate in a year-long, five phase training effort based on the quality of their application and geographic diversity. This study represents the first steps in that training effort, training phases one and two. Phase two occurred approximately 12 weeks after phase one. Each of the first two training phases focused on the first stage of PCIT, CDI. The second stage of PCIT, PDI, was introduced in the third training phase, after clinicians had some time to practice CDI skills. This was done because it is important for clinicians (and parents) to master CDI before progressing to PDI.

Due to contractual agreements and community relationships, random assignment to groups was not possible. Two main factors prohibited randomization. First, prior to study development, the CAARE Center provided a description of training to the Office of Emergency Services, which was included in application materials that went to agency directors. The training was described as active and experiential. Second, within their applications, agency directors identified 2 staff members who would attend training if the agency was selected to participate in training. Prior to study development, community partners expected that there would be only one training group and that only 2 staff members per agency would participate in training. These agreements and expectations could not be reduced. Instead, agencies were extended an invitation to invite additional staff members to training. This increased the number of participating staff so that those staff members could be randomized to groups. Also, groups were stratified by educational level to reduce the likelihood of pre-training group differences.

Two of the 13 agencies were excluded from study participation because they had already received some training in PCIT. Initially, it was expected that 22 therapists from 11 agencies would receive training and be invited to participate in the study. Instead, in order to increase the potential number of participants and allow for some randomization, agencies were invited to send as many therapists as possible to participate in training. In phone contacts with each agency prior to training, it was recommended that persons invited to participate in training be direct service providers and be as similar as possible to the original two persons identified for training.

To ensure groups were balanced by agency and education, assignment to condition was based on three factors: (1) agency, (2) identification in the original application (because of contract obligations), and (3) educational level. Assignment to condition occurred at the beginning of the first training day for each agency. First, the two persons identified in the applications were assigned to the experiential group. Next, two persons from the same agency with the same educational degree (bachelors, masters in social work, masters in arts or sciences, doctoral) were assigned to the videotape didactic. If an agency sent more than four participants, the remaining participants were randomly assigned to a group.

Sixty-one therapists attended some training and data originally were collected on all therapists; however 13 people participated in only phase one of training and were, therefore, not considered in this report.Footnote 1 Of the remaining 48 participants, 6 were selected for exclusion from the experiential group after data were collected so that the groups could be balanced for number and educational level. Of the participants selected for exclusion, five participants possessed a doctoral degree (4 Ph.D., 1 Psy.D) and one participant had received a Masters in Social Work. This imbalance occurred because agencies sometimes had only one or two doctoral-level staff, and those staff members were identified in their application. Because of their identification in the application, they had to be assigned to the experiential group. Participants were selected for exclusion in a two-step process. First, the participant numbers for all participants in the experiential group with a doctoral degree were determined and written on pieces of paper. Each piece of paper was placed in a box. Next, five numbers were randomly drawn from the box by a research assistant. Participants with matching participant numbers were excluded from this project. Similarly, all participant numbers for participants in the experiential group with Masters in Social Work were identified, written on pieces of paper, and placed in a box. The same research assistant selected one piece of paper from the box. The participant whose participant number matched the number drawn from the box was excluded from this project.

Description

As is indicated in Table 1, a total of 36 women (86%) and 6 men participated in this study. They reported an average age of 40.50 years (SD = 10.74). Ethnicity was reported as: 64% European American, 26% Hispanic, 2% African American, 2% Indian, 2% Native American, and 2% preferred not to disclose. Forty-five percent of participants spoke a second language, and 19% of participants used English as their second language. In terms of education, highest degrees obtained were: 5% Ph.D., 36% M.A., 17% M.S., 33% M.S·W., and 10% B.A. Participants reported receiving their degrees, on average, 8.03 years prior to pre-training assessment (SD = 7.69) as well as having a number of hours of supervised practice and years experience working with clinical populations in general (M = 9.54; SD = 7.46), and disruptive behavior disorder (M = 8.67; SD = 7.53) and child maltreatment populations (M = 7.69; SD = 6.60) in particular.

Table 1 Demographic characteristics of combined, didactic, and experiential groups
Full size table

Seventy percent of participants reported their predominant professional activity was direct patient contact. Remaining participants reported their primary responsibilities were administrative (20%), other (5%), research (5%), and teaching (5%). Overall, participants reported spending an average of 18.85 h per week (SD = 11.75) in direct client contact. Participants reported adherence to differing theoretical orientations including 33% cognitive behavioral, 24% family systems, 24% psychodynamic/analytic, 5% existential/humanistic, 5% don’t know, 2% behavioral, 2% interpersonal, 2% social learning, and 2% post-modern/narrative. Almost all clinicians believed that the interventions they used were at least somewhat helpful; most (62%) believed that the interventions they used were highly useful (rated 4 or 5 on a five-point Likert scale).

The majority of participants (83%) reported thinking that PCIT would be useful. On average clinicians expected to devote 17.56 h per week to learning and using PCIT (SD = 15.44). Compared to a national sample of practicing psychologists (Addis and Krasnow 2000), slightly more of the clinicians in this sample had heard of a treatment manual (90% compared to 77%) and used different treatment manuals on a semi-regular basis (57% compared to 43%). Generally, participants reported a more positive (M = 3.08; SD = .52) than negative (M = 2.37; SD = .54) attitude about treatment manuals. Participants in the experiential groups were compared to those in didactic group via independent samples t-tests and Pearson’s Chi squares for all pre-training variables. Given the lack of randomization and atypical procedures of assigning clinicians to groups, it seemed possible that the experiential group could have included more experienced or skilled clinicians; however, groups were not significantly different on any variable.

Measures

Information was obtained to assess four primary domains: participant characteristics, skill acquisition, knowledge gain, and satisfaction. Training integrity also was assessed. Table 2 includes a summary of dependent variables highlighting the assessment domain, strategy, and scoring system for each measure as well as information pertaining to their mastery score (if applicable), development status (original, modified, developed for the study), reliability assessment, and when each measure was completed. Significant attempts were made to find psychometrically sound measures; however, because this is an emerging area of study, tools to measure clinician behavior were not always available. If appropriate measures were not available, attempts were made to make as few modifications as possible to existing psychometrically sound measures for them to be useful in the current study. If a measure was not found that could be used directly or with slight modifications, attempts were made to ensure that created measures were sound.

Table 2 Dependent variable summary
Full size table

Participant Characteristics

Demographic Information Survey

A demographic information survey was developed to include variables hypothesized to affect learning of EBTs (Addis and Krasnow 2000) as well as details regarding age, ethnicity, gender, postgraduate training experiences, applicability of training to clinical practice, preferred learning formats, satisfaction with currently used interventions, interest in PCIT training, and attitudes toward a behavioral orientation. Participants completed this measure prior to training (assessment one).

Attitudes Toward Treatment Manuals

This 17-item self-report questionnaire was designed to assess attitudes toward treatment manuals (Addis and Krasnow 2000). More specifically, items were included to assess two factors: Negative Process and Positive Outcome. The Negative Process factor represents therapists’ concern for freedom and flexibility in sessions as well as concern for the potential negative effects of manuals on the therapeutic relationship. Conversely, the Positive Outcome factor represents therapists’ belief that manuals can enhance treatment outcomes (Addis and Krasnow 2000). Practitioners completing this questionnaire were asked to rate their agreement with each of the 17 items on a five-point Likert scale ranging from 1 (strong disagreement) to 5 (strong agreement). Sample items include: (1) Manuals make therapists more like technicians than caring human beings. (2) Following a treatment manual will enhance therapeutic outcomes by ensuring that the treatment being used is supported by research. Participants were asked to complete this measure two times: once at assessment point one and once at assessment point four.

Skill Acquisition

Dyadic Parent–Child Interaction Coding System (DPICS; Eyberg and Robinson 1983)

Acquisition of CDI skills was measured by requiring each participant to interact with a confederate for a 5-min, videotaped, structured behavioral observation. This was meant to be analogous to the parent–child assessment conducted at the beginning of each PCIT treatment session. The confederate, a 30-year old male, interacted with each participant for 5-min, 30-s. The first 30 s were not coded so that the participant had time to adjust to the role-play setting. During subsequent minutes, the confederate acted in ways to elicit target behaviors (e.g., using praise, descriptions) from participants. To ensure that the confederate randomized and engaged in each specified behavior for 1 min, he wore a tape recorder with a pre-recorded audiotape to cue him. Each behavior observation was videotaped and later coded using the DPICS.

The DPICS was originally designed to assess the quality of parent–child interactions through observations of dyads in three standardized laboratory situations (Child-Directed Interaction, Parent-Directed Interaction, Clean-Up). Twenty-four categories of parent and child behaviors are assessed through frequency counts. Normative data are available (Eyberg & Robinson). Reliability ratings for parent behaviors range from .67 to 1.0, with a mean of .91 (Aragona and Eyberg 1981; Eyberg and Matarazzo 1980: Robinson and Eyberg 1981). The DPICS has distinguished between pre- and post-treatment data (Eisenstadt et al. 1993; McNeil et al. 1991), different treatments (Eyberg and Matarazzo 1980), and interaction patterns among diverse family populations (Aragona and Eyberg 1981).

In the current study, only eight of the 24 behaviors were considered: unlabeled and labeled praise; critical, reflective, and descriptive statements; indirect and direct commands; and questions. Abbreviated definitions for these behaviors are included in Table 3. These codes were chosen because they are verbal behaviors that are used clinically to determine if an adult has reached mastery skills criteria in PCIT. At the time of this study, in order to advance to the second stage of PCIT as well as reach mastery of CDI, an adult had to demonstrate the following behaviors in the 5-min CDI observation: 25–50 descriptions and reflections (reflecting at least half of all child verbalizations), 15 praises (8 of which must be labeled), and no more that 3 critical statements, commands, or questions (Eyberg and Calzada 1998). In the current study, the DPICS was applied to therapists because they must master the skills themselves before they can instruct parents.

Table 3 Dyadic Parent–Child Interaction Coding System (DPICS) abbreviated definitions
Full size table

One total score was derived based on therapist performance in each of the eight behavior categories as compared to mastery criteria in CDI skills. Participants received one point toward their total score for each of the categories in which they reached mastery. Total scores ranged from 0 to 7. Examples of scoring appear in Table 4. This system was used at each assessment point.

Table 4 Sample scoring of CDI skill acquisition
Full size table

Coach Coding

Acquisition of coaching skills was assessed by asking participants to view a videotape of the first author interacting with a child and to “coach” her in the use of CDI skills. Participant verbalizations were audiotaped as they coached into a microphone connected to an audiotape recorder. Four videotapes were developed. Participants were asked to coach one tape at each assessment point. The order in which participants received the tapes was randomized. The first author interacted with the same child in each tape, a six year old, Caucasian male who was clinic-referred for a disruptive behavior disorder and history of child maltreatment. During these tapes, the first author approximated parent behaviors (e.g., praise, questions). Participants were instructed that there would be a 30-s period for them to observe the interaction. After a visual cue on the video, they were asked to coach the first author for 5 min as if she was a mother referred for PCIT with her son. Audiotaped coaching samples were collected at each assessment point.

A point system was designed for the current study to assess the quality of coaching after observing multiple coaching sessions involving different coaches. Similar to DPICS coding (Eyberg and Robinson 1983), coaching verbalizations were defined by the “one sentence rule” in which one sentence equals one unit of behavior. Each coaching statement was given a score of −1 through 3 based on the sophistication of the statement. Coaching errors were scored as a −1 whereas advanced coaching statements were scored as a 3. For example, criticizing the parent (e.g., “That was the wrong thing to say to him.”), directing the parent to do something in the avoid category of CDI skills (e.g., “He just said the blue block was red. Tell him that’s wrong.”), or praising the parent for an inappropriate behavior (e.g., “Nice command” after the parent gives the child an indirect command.) was assigned a −1. Conversely, providing feedback to the parent that involved the interaction between the parent and child (e.g., “I noticed that when you are really polite with him, he is also really polite with you.”), providing feedback that commented on a qualitative aspect of parent behavior or a positive parent behavior not included in the CDI skills (e.g., “Your voice tone is really warm. He can tell how much you care and that you mean what you say by the tone of your voice.”), or providing information on how the playtime affects the parent-child relationship (e.g., “He seems to be responding to you very positively. I think his attitude toward you and your relationship is changing.”) received a score of 3.

Mastery of coaching skills was calculated by asking five experienced PCIT clinicians to complete the coaching assessment. Five clinicians completed the assessment who ranged from bachelors- to doctoral-level practitioners, averaged four years of experience conducting PCIT (range 2–7), and saw 8 to 10 PCIT clients per week at the time of the assessment. After completion of the coaching assessment, audiotapes were coded, and scores were obtained for each of the five clinicians. The average coaching score for the PCIT clinicians was 58 (range 47–76); therefore, a score of 58 or higher was considered a mastery criterion for PCIT coaching.

Knowledge Gain

CDI and DPICS Knowledge Questionnaire

Four versions of a 20-item quiz were developed to assess participants’ knowledge of CDI and DPICS. Each version contained 10 CDI knowledge and 10 DPICS knowledge items. The CDI section contained an equal number of items targeting didactic (e.g., “Hembree-Kigin and McNeil (1995) mention five advantages of direct coaching, please list three.”), and coaching skills (e.g., “During a coaching session, a child turns to his mother and says, “I made a tall tower.” As a coach, what might you instruct the parent to say?”). The DPICS section contained an equal number of items targeting definitions (e.g., “please provide a DPICS definition for a descriptive statement.”) and the application of definitions to clinical scenarios (e.g., Please code the parent verbalizations in the following parent-child interaction. Parent: “Let’s draw a cloud in the sky.” Child” I don’t know how to draw a cloud.”). Each participant completed all questionnaires; one questionnaire was completed at each assessment point in randomized order. Each questionnaire yielded three scores: CDI Knowledge, DPICS Knowledge, and Total PCIT Knowledge.

To ensure that the forms were equal in difficulty level and content, three expert PCIT clinicians independently reviewed these forms to assess their face validity. Afterward, a small pilot study was conducted. Ten clinicians with varying levels of PCIT expertise completed the forms. After modifications were made to the original form, a second administration yielded similar scores across forms, and variation in scores were consistent with clinicians’ level of PCIT experience. Mastery of CDI and DPICS knowledge was defined as scoring 80% or higher given that this was the average score of five experienced PCIT clinicians in the pilot study.

In order to score CDI and DPICS knowledge questionnaires, it was proposed that research assistants would be provided a detailed answer key as well as didactic training and practice in coding “mock” questionnaires that would be completed by four clinicians: two who were and two who were not considered trained to mastery level in PCIT. It was anticipated that proficiency in scoring would take approximately 4 h of training; however, after several attempts to train research assistants to score these measures, it was determined that this would not be possible. Because the questionnaires included open-ended questions, the variety of responses provided by participants was extremely variable and scoring required a high level of sophistication in PCIT skills as well as theoretical knowledge and general therapy skills. Therefore, a research assistant re-assigned participant numbers to all questionnaires so that the investigator was blind to the participant number and assessment point for each questionnaire. After this re-assignment was complete, the first author scored each of the questionnaires. Twenty-five percent of the questionnaires were scored twice for reliability purposes (ICC = .99). Final scores were obtained before returning the original participant numbers and assessment points to the questionnaires.

Satisfaction

Modified Therapy Attitude Inventory

The original Therapy Attitude Inventory (TAI; Eyberg 1974) was designed to assess consumer satisfaction with parent training, parent-child treatments, and family therapy (Eyberg 1993). Reported reliability coefficients (Cronbach’s alpha) have ranged from .88 to .91, with good stability over a 4-month period (.85) (Brestan et al. 1999; Eisenstadt et al. 1993; Eyberg and Matarazzo 1980). Low to moderate correlations (.36–.49) were reported between the TAI scores and changes during treatment.

In the current study, the TAI was modified to assess therapists’ (rather than parents’) satisfaction with PCIT. The original 10 items and response choices remained. Changes were made to sentence structure so that items would be appropriate for therapists rather than parents. (Sample item “Compared to other approaches I have used, I feel that child compliance after treatment will be: considerably worse, somewhat worse, neutral, somewhat improved, greatly improved.”) Participants completed this questionnaire at assessment points two, three, and four.

Satisfaction with Training

A 19-item questionnaire was developed to assess participants’ satisfaction in three areas: training content, format, and presenters. Fifteen of the 19 questions were rated on a 5-point Likert scale ranging from 1 (strongly disagree or poor) to 5 (strongly agree or excellent). Of the remaining four questions, which were not considered in the total satisfaction score, two were open-ended, and two asked for the participants’ training methods preferences. Participants completed this measure at assessment points three and four.

Training Integrity

Training Integrity Checklists

Checklists were used to assess trainers’ implementation of each training day. A score of 90% or greater was considered accurate curriculum implementation. Integrity data were coded for 33% of the trainings; scores ranged from 91% to 100%. Twenty-five percent of the training videotapes were scored twice for reliability purposes (ICC = .94).

Procedure

Participant Training

Training was conducted in five phases: program development, PCIT fundamentals, intensive skill building, advanced skill building, and consultation and supervision. Also included in training were quality assurance, quarterly regional meetings, and one PCIT national conference. This study reports on phases one and two of the larger California Governor’s Office of Emergency Services Training Project. Phase one, program development, included individualized assistance and consultation in building a PCIT program. Consultation was provided on practical aspects of establishing and maintaining a PCIT program (e.g., securing a referral base, installing equipment). Approximately three months later, phase two, PCIT fundamentals, included a 2-day consecutive workshop at each agency site or region, which provided an overview of PCIT and detailed information on CDI. Eight separate trainings were held in sites across the state (e.g., Redding, Oakland, Los Angeles, San Diego).

On the first workshop day each participant received the same training experience. All information was presented didactically and through videotape and live modeling. Participants had no scheduled opportunity during the day to practice skills discussed. On the morning of the second day, participants were divided into two groups: an experiential and didactic group. Each group received the same informational content, but the format in which the information was delivered differed. Members of the “hands-on” experiential group participated in role-plays, individually practiced coding videotapes, and received frequent, individualized feedback on their performance. The didactic group reviewed client session videotapes, discussed PCIT skills, and coded videotapes as a group. Didactic group members were not provided opportunities to practice or receive feedback on skill use during training. Therefore, the difference between groups was that the experiential group participants were required to observe, discuss, practice and receive feedback on individual skill performance whereas the didactic group observed and discussed live and videotaped examples of PCIT concepts.

Six trainers were involved in Program Development and PCIT Fundamentals training. To control for potential trainer effects, trainers rotated between the two groups. The order in which trainers conducted each type of training was randomized. Also, the number of trainers present in each of the two training groups was controlled so that two trainers were present at all times.

Assessment Schedule

Prior to the initiation of training, the first author visited each agency to explain the study to participants, receive informed consent, and conduct the pre-training assessment, which included participant characteristics, skills, and knowledge (assessment one). Afterward, all participants were asked to read pages 1–69 of the provided treatment manual, Parent–Child Interaction Therapy (Hembree-Kigin and McNeil 1995), and an abbreviated DPICS manual (Eyberg and Robinson 1983). Participants were asked to keep a reading log to determine if they read assigned materials. Each participant was mailed a letter 2 weeks prior and phoned one week prior to PCIT Fundamentals training as a reminder that the second assessment would occur before training started and include information from the readings. Prior to the start of PCIT fundamentals, participants completed skills, knowledge, and satisfaction assessments (assessment two) again so that the first hypothesis could be tested. This same assessment occurred at the end of that training day (assessment three). A final assessment was conducted at the end of the second day of training and included measures of participant characteristics, skills, knowledge, and satisfaction (assessment four).

Coding of Observational Data

Research assistants who were unaware of study hypotheses, group assignment, and assessment point coded audio and videotapes. Prior to coding, they received 40 h of training and achieved minimum criteria of 80% agreement on three consecutive practice tapes using the first author as the reliability check. Twenty-five percent of all audiotapes were double-coded yielding an intraclass correlation of .98. Videotapes were first coded to assess participant behavior. Overall interrater reliability on the DPICS, calculated for 31% of the data, indicated 85% agreement. Videotapes were coded a second time to determine if the confederate engaged in specified behaviors. Interrater reliability of confederate behavior, which was calculated for 25% of the data, was .84 (Cohen’s kappa).

Results

Impact of Reading a Treatment Manual on Participant Knowledge and Skill

For participants who reported reading the first half (p. 1–69) of the PCIT treatment manual (69%) (Hembree-Kigin and McNeil 1995), paired-comparison t-tests were conducted comparing assessment point one to assessment point two mean scores on knowledge and skill measures. In order to control for the potentially high familywise error rate, the Bonferroni inequality was applied, with an alpha level of .05/13 (.004) or less considered to be significant. As predicted, statistically significant increases were evident in Total PCIT Knowledge t(24) = −3.79, P = .001 and DPICS Knowledge t(24) = −3.86, P = .001; however, no significant increase in CDI Knowledge was revealed (Table 5). Increases were revealed in Labeled Praise t(26) = −3.63, P = .001 and Total Coaching Score t(28) = −4.02, P < .001. Decreases were evident in Questions t(26) = 5.25, P < .001 and Indirect Commands t(26) = 3.14, P = .004.

Table 5 Comparison of knowledge and skill scores pre-training and after reading the PCIT manual
Full size table

Didactic Versus Experiential Training

The sample was divided into two training groups (i.e., didactic and experiential) after the third assessment point. To ensure that groups were equal immediately prior to the experimental manipulation, independent samples t-tests were performed to detect differences between groups on knowledge, skill, and variables related to if clinicians read assigned readings. The Bonferroni inequality was applied again (P < .004). No group differences were detected, including for those who did and did not read the treatment materials; therefore, all participants were included in subsequent analyses.

Three analyses of training success were conducted using 2 × 2 multivariate analysis of variance (MANOVAs) with one between groups factor (group) and one repeated factor (time). One MANOVA was conducted for skill, a second for knowledge, and a third for satisfaction scores at assessment points three and four. An examination of skill variables revealed no group × time interaction or group main effect. A time main effect was present F(10, 23) = 5.93, P < .001. Univariate analyses reveal significant increases in scores on labeled praise F(1, 32) = 20.50, P < .001; descriptions F(1, 32) = 9.01, P = 005.; reflections F(1, 32) = 14.29, P = .001; CDI mastery F(1, 32) = 12.45, P = .001; and total coaching scores F(1, 32) = 37.81, P < .001 as well as decreases in scores on questions F(1, 32) = 17.99, P < .001 (Table 6).

Table 6 Comparison of training groups on skill acquisition, knowledge gain, and satisfaction at assessment points three and four
Full size table

Inspection of knowledge variables revealed no group × time interaction effect or group main effect. A significant time main effect was revealed F(3, 38) = 17.71, P < .001. Univariate tests (Table 6) revealed significant increases in scores on CDI Knowledge F(1, 40) = 26.84, P < .001 and DPICS Knowledge F(1, 40) = 33.27, P < .001. In examining satisfaction variables, a group × time interaction effect was not revealed, nor was a group main effect but, a significant time main effect was present F(2, 38) = 13.10, P < .001. Table 6 highlights that univariate tests revealed significant increases in scores on the Therapy Attitude Inventory F(1,39) = 14.91, P < .001 and the Satisfaction with Training total score F(1,39) = 19.93, P < .001.

Therapist Characteristics Associated with Higher Skill Acquisition

To identify factors that predict mastery of PCIT concepts, logistic multivariate regression analyses were conducted. Two main predictors, theoretical orientation and degree type, were explored. Because a small number of doctoral- and bachelor-level participants were in the current study, they were excluded from this analysis. Instead, participants with a Masters degree of Arts or Science in Psychology or Education (MA/MS) were compared with participants with a Masters degree in Social Work (MSW). Because there is no existing empirical literature comparing these groups, no directional predictions were made.

A multivariate logistic regression analysis was performed to examine if degree type and theoretical orientation predicted mastery of PCIT knowledge, neither of which was found to be predictive. In a second logistic regression analysis to examine if the same variables predicted mastery of CDI skills, type of degree predicted mastery of CDI skills (see Table 7). Participants who had an MSW were 15 times more likely than participants with a MA/MS to reach mastery of CDI skills. Theoretical orientation did not predict mastery of CDI skills. Due to the low number of participants who reached mastery of coaching skills (4 of 42, 10%) and mastery of a combination of all three knowledge and skill measures, PCIT knowledge, CDI skills, and coaching (2 of 42, 5%), logistic regression analyses were not completed for these variables. Considering these variables separately, 31% of participants demonstrated mastery of PCIT knowledge, 17% demonstrated mastery of CDI skills, and 10% demonstrated mastery of coaching skills.

Table 7 Results of logistic regressions predicting CDI skill mastery by degree type and theoretical orientation
Full size table

Discussion

Impact of Reading a Treatment Manual on Participant Knowledge and Skill

Although not originally hypothesized, reading a treatment manual resulted in significant improvements in both participants’ knowledge and skills. While skill and knowledge improvements were statistically significant, their clinical significance is questionable. Mastery of PCIT knowledge or skills was not obtained after reading the book for any participant. Therefore, a treatment manual may serve as a useful “first step” in implementation that must be followed by more intensive training for treatment mastery (e.g., Ducharme and Feldman 1992; Henggeler and Schoenwald 2002; Kelly et al. 2000; Rubel et al. 2000). This finding is consistent with a number of studies indicating that reading often resulted in knowledge changes, but the changes were short-lived and smaller than those of therapists participating in more intensive trainings (e.g., Sholomskas et al. 2005).

Didactic Versus Experiential Training

It was expected that participants in the experiential group would score better on outcome measures than participants in the didactic group; however, no differences were found between the two groups. In hindsight, this lack of group differences may be because the didactic group received more than traditional training. For each of the PCIT concepts practiced in the experiential group, participants in the didactic group viewed a videotape and discussed the concepts through the video case examples. Additionally, specific strategies were highlighted by trainers conducting role-plays with each other. Therefore, the description of the training as “didactic” may be misleading, and a better name for the group might have been the “videotape modeling group.” Considerable research on a parent training program, the Incredible Years Training Series (e.g., Webster-Stratton 1994), has demonstrated that videotape role modeling is an effective training method for parents, teachers, and family service workers (Webster-Stratton et al. 2001). Training provided in the current study was similar in content and format to Webster-Stratton’s program.

Perhaps also contributing to a lack of group differences was the assessment process. Over the 2-day consecutive workshop, participants completed three assessments. Each assessment provided an opportunity for participants to practice PCIT skills in situations analogous to PCIT sessions. Therefore, members of the didactic group were able to practice the PCIT skills during the assessment portions of the training. Perhaps the greatest difference between the two groups was that the experiential group participants received individualized feedback from trainers on their skill performance during training and the didactic group participants did not receive feedback from trainers. Instead, participants in the didactic group received feedback on their suggestions for CDI skills application through discussions with other trainees in their group. Another possible reason for the lack of group differences may be group size; the didactic groups were small enough to allow them to be “intensive” without being experiential.

Therapist Characteristics Associated with Higher Skill Acquisition

Therapist characteristics hypothesized to affect EBT implementation were examined in the current study (e.g., Addis 2002; Addis and Krasnow 2000; Barlow et al. 1999); however, findings should be interpreted cautiously considering the small number of study participants and the even smaller number of participants who met mastery criteria. Reported theoretical orientation was not found to have predictive power for post-training increases in knowledge or skill, which is consistent with Hawkins and Sinha (1998).

Interestingly, degree type predicted CDI mastery; however, this finding is preliminary and would need to be replicated before strong conclusions can be drawn. Of the seven participants who reached mastery criteria in CDI skills, five of them had Master’s degrees in Social work, one had a Master’s degree in Psychology, and one had a Bachelor’s degree in Sociology. The seven participants represented six different agencies. In order to account for why MSW participants were 15 times more likely than MA/MS participants to reach CDI mastery, participants’ reported graduate programs and the program’s accreditation status were examined. MSW participants appeared to be a homogeneous group; they all attended university based, terminal degree programs, and many attended the same state system of programs (e.g., state universities in California). The MA/MS participants were heterogeneous in terms of type of school (University versus Professional School), graduate programs attended, specialty area pursued, and stages of their degree (e.g., terminal masters versus in route to a doctorate). Participants reported no differences in their graduate programs’ accreditation status. Future studies should solicit information about participants’ licensure and quality of graduate training.

Improvement Versus Mastery of Knowledge and Skills

While it is a positive finding that participation in either the experiential or didactic trainings resulted in improvements in clinician knowledge and skill, a concern remains. After a 2-day intensive training, participants improved their skills, but very few demonstrated mastery of skills. Group means appeared promising; however, when individual participants were considered, few (5%) reached mastery level on all three PCIT specific measures (PCIT knowledge, CDI skills, and coaching). Considering these measures individually, only 31% of participants demonstrated mastery of PCIT knowledge, 17% demonstrated mastery of CDI skills, and 10% demonstrated mastery of coaching skills. While it is important to have reasonable expectations for training, the PCIT knowledge and skills measured are the basics of PCIT and likely are essential for successful implementation of the program. Also, it seems that the analog assessment sessions in the current study were optimal for demonstration of knowledge and skill because the “parent” and “child” in the role-play assessments were likely more skillful than the typical client prior to treatment. Implementing PCIT skills in a clinical setting with extremely challenging children and parents would likely decrease performance. The distinction between improvements versus mastery of skills is important in that it may highlight the that clinicians may learn a new treatment approach, but implementing it with the level of skill that was originally intended is more difficult.

Limitations

The results of the current study should be interpreted cautiously due to several limitations including the selection and number of participants, frequent assessment, assessment of basic skills, a lack of standardized and validated dependent variables, and generalizability. Many of these limitations were imposed by the ‘real-world’ nature of this investigation, which is both a strength and weakness of the current study. Many have mentioned the need to balance rigor with relevance in dissemination and implementation studies, which leads to necessary compromises at all levels (cf. Minkler 2004; Shoultz et al. 2006; Williams et al. 2005).

Selection and Number of Participants

Agencies participating in the larger training project had to apply for training funding through California Governor’s Office of Emergency Services. Once awarded funding, agency directors selected members of their staff to participate in training. Each of these situations is ‘real-world,’ but invites a potential selection bias. It is possible that the sample is representative of above average agencies and clinicians, for which performance expectations might be higher. Also concerning is the small number of participants. Having only 21 participants in each group reduced statistical power for detecting group differences and the generalizability of findings. Each training included a small number of participants (average 6) with a low ratio of trainers to trainees (2:3), which does not approximate typical continuing education training. A further limitation is that data on agency-level variables, including general staff characteristics or human resources information, were not collected. Collection of this type of information would have allowed a clearer statement about the representativeness of the sample as well as would have provided important contextual information.

Despite these concerns, the current study offered improvements to methodological challenges mentioned in reviews (Alberts and Edelstein 1990). Previous training studies have been criticized for including clinicians with either little training (Alberts and Edelstein) or doctoral training (Henry et al. 1993). In contrast, the current study included masters-level professionals with considerable work experience. The current sample included only 42 participants, but it is not uncommon for practitioner-focused studies to have small sample sizes (e.g., Henry et al. included 16 participants).

Assessment

The number of assessment points relative to the amount of time spent in training may have increased the likelihood of practice effects. Over the course of a 2-day workshop, participants completed three assessments that involved practicing PCIT skills. These repeated assessments may have contributed to a lack of group differences by allowing the didactic group to practice skills. Outcomes may also have been impacted because the amount of training was decreased to accommodate assessment procedures. The training and assessments were reported by some therapists as tiring, which may have impacted their performance. Also, the lack of a follow-up assessment is a weakness, which could have revealed training differences.

This study included assessment of only basic PCIT skills contained in first phase of PCIT, CDI. Arguably, CDI is consistent with a wide array of theoretical approaches and palatable to a large number of therapists because it combines developmental, interpersonal, and behavioral theories and utilizes play therapy techniques. The second treatment phase, PDI, focuses on compliance training, relies on behavioral principles, and is often considered to be more difficult to implement due to the directive and swift pace of coaching. Perhaps an assessment of basic versus advanced skills would yield more interesting findings on the utility of experiential versus didactic training; experiential training might be more helpful when skills are complex and difficult.

Lack of Standardized and Validated Dependent Variables

Significant attempts were made to find psychometrically sound measures. If appropriate measures were not available, attempts were made to make few modifications to existing psychometrically sound tools. If a measure was not found that could be used directly or with slight modifications, attempts were made to ensure that created measures were sound. However, it was beyond the scope of this study to standardize and validate new measures. Results based on measures created solely for this investigation should be interpreted with caution.

A significant measurement weakness is in the definition and measurement of clinician mastery, a key component of the study. Given that there are no pre-defined, empirically-based mastery criteria for clinicians implementing PCIT skills, mastery criteria established for parents was adapted for clinicians. For knowledge, experienced PCIT clinicians’ knowledge test scores from a pilot study were averaged to establish mastery criteria. While these strategies to develop mastery criteria may posses some face validity, criterion and predictive validity are lacking, which is a weakness. It is possible that a lack of group difference in mastery could be related to weak measurement of mastery.

Generalization

The unique therapy format of PCIT may limit the generalizability of this study. It is possible that the dose and format of training necessary for skill mastery may vary among EBTs. While the training protocol utilized in this study was not sufficient for PCIT mastery, it may be for another treatment approach utilizing a different therapeutic format.

Conclusions

This study offers a contribution to the currently scarce literature in that it is a “first step” in empirically investigating methods of disseminating an EBT to community practitioners. Results confirm three main findings regarding PCIT dissemination: (a) reading a treatment manual is useful but not sufficient, (b) for basic skills, both experiential and didactic training (with videotape modeling) can be useful, and (c) training success is associated with degree type, not theoretical orientation. Also noteworthy is that practitioners with a variety of theoretical orientations accepted the treatment as demonstrated by reported high satisfaction. Each of these findings cautiously, but optimistically, supports the idea that PCIT can be widely disseminated and implemented. Caution must be taken considering that the current study demonstrated that increases in knowledge and skills are likely for a majority of participants; however mastery of knowledge and skills is likely for only a minority of participants after reading a treatment manual and attending two consecutive days of training. Additional instruction and the study of that instruction are necessary to determine how much and what type of training is required for mastery of basic and advanced PCIT skills.

Future studies should continue to explore therapist characteristics associated with training success, improvements in versus mastery of specific EBT knowledge and skills, and the role of masters-level practitioners in EBT delivery. Also, new areas of investigation should be explored such as acquisition of basic versus advanced therapy skills. The application of findings from such investigations will assist program developers and researchers in transporting EBTs to community-based centers, where the majority of children and families receive services, and where the need for effective services is paramount.