Introduction

Recent mental health policy initiatives have called for quality improvement interventions to ensure that evidence-based care is practiced (New Freedom Commission on Mental Health 2003). Quality improvement efforts in psychotherapy have been hampered because there are no easily obtained measures of the extent to which psychotherapy is evidence-based. In this paper, we present preliminary findings regarding a short, self-report measure assessing therapist adherence to three types of psychotherapies for depression.

We focus on depression because it is a prevalent and debilitating disorder, with evidence-based psychotherapy treatments. Lifetime prevalence of major depression is 16.6%, making it the single most prevalent psychological disorder (Kessler et al. 2005). According to World Health Organization estimates, by the year 2020, major depression will become the second leading cause of disability worldwide (World Health Organization 2001). Depression costs employers about $44 billion a year, primarily due to loss of productivity at work and lack of high quality depression care (Stewart et al. 2003). Further, patients with affective disorders have a high degree of morbidity and low health utility compared to most chronic health conditions (Wells and Sherbourne 1999). While antidepressant treatment is an efficacious option, most patients prefer psychotherapy (Alvidrez and Azocar 1999; Dwight-Johnson et al. 2000). Given the prevalence and burden of depression, it is likely the most common condition seen by psychotherapists, thus making depression an important target.

Substantial empirical support exists for the efficacy of two forms of psychotherapy for treating depression in adults: cognitive behavioral therapy (CBT) and interpersonal therapy (IPT; Chambless et al. 1998). Previous works suggests that there has been a lag between discovering efficacious forms of treatment and incorporating them into routine patient care (Institute of Medicine 2001). Efforts to improve the quality of psychotherapy for depression rely on the ability to measure the extent to which either CBT or IPT are practiced.

One indication of how a therapist might treat depression that could readily be available to potential patients or a behavioral health plan hoping to improve the quality of depression care is the therapist’s therapeutic or theoretical orientation. Unfortunately, endorsing a therapeutic orientation (e.g., cognitive-behavioral) may not translate into implementing key components of the model when practicing psychotherapy. While roughly a quarter of psychologists identify themselves as cognitive-behavioral and another quarter as psychoanalytic/psychodynamic, a full one third of therapists identify themselves as eclectic/integrative—indicating their use of techniques from more than 1 orientation (Norcross et al. 2002). National data on therapeutic orientation across disciplines are not available. There is also evidence of striking discrepancies between theoretical belief and clinical practice (Buckley et al. 1979). Further, therapists may endorse a particular therapeutic orientation (e.g., cognitive-behavioral) to increase their appeal when applying to join managed care panels who want therapists practicing evidence-based care. The availability of information about a therapist’s actual therapeutic practice would enable quality improvement efforts to identify the extent to which a particular psychotherapy is practiced.

Current clinical research measures of the extent to which a therapy follows an evidence-based practice are not feasible for quality improvement efforts in mental health settings. These approaches have relied on trained observers to code audio or videotapes of sessions to determine adherence to a particular therapy model. Several process rating scales have been developed to assess adherence to a treatment model. For example, the Cognitive Therapy Scale (Dobson et al. 1985) is a checklist that evaluates the extent to which therapists adhere to CBT and requires administration by trained professionals listening to tapes of sessions. In this study, we reviewed existing ratings scales and psychotherapy literature to develop a clinician self-report measure that could easily be administered as part of a quality assessment and improvement effort. This study represents a first step toward a clinician-reported instrument that could prove useful in describing the extent to which therapists adhere to CBT, IPT or psychodynamic therapy. We developed a clinician-report psychotherapy questionnaire that assesses the frequency that the clinician used cognitive-behavioral (CBT), interpersonal (IPT), and psychodynamic (DT) techniques in treating a particular patient with depression. The goals of the following analyses were to reduce the length of the pilot questionnaire and begin to understand the validity of the hypothesized subscales.

Methods

We sought to develop a clinician-report instrument to describe three psychotherapy approaches used in the treatment of depression. To develop the item content for the pilot questionnaire, we reviewed relevant literature on the essential components of three therapy approaches: cognitive behavioral, psychodynamic, and interpersonal. To evaluate the questionnaire, we analyzed data collected from clinicians associated with a large managed behavioral health care organization. The goals of the analyses presented here were to refine the questionnaire and to begin to understand the validity of the items.

Measure

The instrument, the Psychotherapy Practice Scale—Clinician Depression Care Version (PPS Clinician), was designed to provide a tool to describe the psychotherapeutic techniques used in the treatment of depression. The instrument focuses on three therapeutic approaches: CBT, IPT, and DT. CBT and IPT techniques were included because these approaches have strong empirical support for the treatment of major depression (Butler et al. 2006; Chambless et al. 1998; Chambless and Ollendick 2001; Weissman et al. 2000). While DT is inherently difficult to subject to randomized controlled trials and thus has limited empirical support (Tanenbaum 2005), a recent survey indicated that roughly a quarter of psychologists identify themselves as psychoanalytic/psychodynamic (Norcross et al. 2002), providing support for including dynamic therapy as well.. Content coverage and item development was guided by review of clinical literature on efficacy and essential therapy components (Barlow 2007; Beck 1995; Jacobson et al. 1996; Klerman et al. 1984), existing observation coding tools of adherence and competence (Barber et al. 2003; Dobson et al. 1985; Hill et al. 1992; Weersing et al. 2002), and consultation with clinical experts to determine the key therapeutic techniques essential to each type of therapy when treating depression. Items often include clinical terminology that would be easily recognizable by clinicians who use a particular technique. We also developed a companion patient report instrument (not reported here) that removed clinical terminology and focused on concrete, observable clinician behaviors (Miranda et al. 2010).

The instrument assesses how frequently a therapist uses specific techniques in the course of treating a patient. The full version of the questionnaire included 28 items to assess techniques that are key components of CBT (8 items), DT (11 items), and IPT (9 items). Items are listed in Table 2, while the full formatted instrument with instructions is available from the first author. The instrument asked the clinician to select a single, recently treated adult patient diagnosed with depression. Clinicians were asked to rate the frequency that they used each technique with that patient on a 7-point scale ranging from ‘never’ (1) to ‘always’ (7). In an effort to minimize patterned responding or endorsing only items from a particular orientation, technique items are not labeled or grouped by therapy approach. An additional question asked whether the clinician’s approach in treating this patient was typical of the approach used with other depressed patients.

Clinicians were also asked to select their “main theoretical orientation” when treating depression. Options included cognitive behavioral, interpersonal, psychoanalytic, psychodynamic, supportive, and other. Some clinicians selected multiple orientations, so we created a “primary orientation” variable where clinicians could fall into only one category. We created “Multiple Orientations/Eclectic” to reflect clinicians who selected more than one option. Due to low frequencies, we also combined the psychoanalytic and psychodynamic orientation into a single dynamic orientation and combined supportive with other. Clinicians also reported the number of patients with depression they had treated with at least five sessions of psychotherapy in the past 6 months.

Recruitment and Respondents

A total of 2,200 “high volume” clinicians (600 Psychiatrists, 600 Psychologists and 1,000 Masters-level therapists) that had treated at least ten adult patients in the past year were randomly selected from the clinical panel of United Behavioral Health (UBH), a large, national managed behavioral health organization (since renamed OptumHealth Behavioral Solutions). Clinicians received a packet containing an introductory letter, 1-page study description, informed consent form, and questionnaire via standard mail. Clinicians were offered an educational CD-ROM allowing them to earn 1 h of continuing education credit for returning the questionnaire. Non-respondents received an identical follow-up mailing approximately 2 weeks after the initial mailing. The study was approved by the RAND Institutional Review Board.

A total of 687 clinicians returned the questionnaire by mail for a raw response rate of 31%. To determine an adjusted response rate, we removed 21 sampled clinicians who were ineligible because they no longer practiced, did not treat adult clients, or had died; therefore, the adjusted response rate was 32% (687/2,179). In addition to the mailing to the sampled clinicians, electronic copies of all materials were posted on UBH’s clinician website, allowing any clinician in their network to participate. An additional 40 clinicians completed the questionnaire by obtaining materials through the website. While online respondents reported fewer years of clinical practice (12.6 vs. 16.1, P < 0.05) and a higher proportion of master’s degrees (100 vs. 42%, P < 0.001) compared to mail respondents, there were no demographic differences in gender, age, or ethnicity (Caucasian). Based on these analyses, we determined that the online sample was sufficiently comparable to be combined with the mail sample. The combined sample included a total of 727 respondents.

Comparisons of demographic characteristics for responders (both mail and online, n = 727) and non responders (n = 1,492) yielded some differences (Table 1). Responders were more likely to be older, Caucasian, female, and have a master’s degree. There was no difference in number of years of clinical practice. The pattern of these results was the same when the online responders were excluded.

Table 1 Characteristics of responders vs. non-responders
Full size table

Responding clinicians reported seeing an average of 24 patients (SD = 15.4) for at least five psychotherapy sessions in the prior 6 months. Clinicians reported a range of primary theoretical orientations. Over half endorsed cognitive-behavioral (51%), followed by eclectic/multiple orientations (21%), psychodynamic/psychoanalytic (12%), other (10%), and interpersonal (7%). The majority of clinicians (95%) reported that the approach used with the selected patient was the approach that they typically used with their depressed patients.

Analyses

Approach to Refining the Instrument

We conducted analyses to determine the psychometric properties of the instrument and inform instrument revisions. We believed that items assessing the three therapeutic approaches would form three subscales (or multi-item composites). We conducted exploratory factor analyses using oblique (promax) rotation with varimax prerotation and squared multiple correlations as communality estimates priors to determine the initial composites. The number of factors was determined using eigenvalues and the interpretability of the rotated factor pattern matrix. There were three eigenvalues over one (7.25, 2.98, 1.13, 0.78, 0.57, 0.46, and so on) with an average eigenvalue of 0.44. When items were assigned to a factor that had standardized regression coefficients of at least 0.30, the three factor solution was interpretable and represented the three therapy approaches. There were some notable secondary loadings, but we proceeded with the hypothesized subscales.

We incorporated several pieces of information to determine recommendations for item deletion including internal consistency reliability, corrected item total correlations, item discrimination, subscale intercorrelations, item missing, and limited observed variability. Specifically, we evaluated internal consistency reliability using Cronbach’s alpha, with the goal of achieving at least an alpha of .7 for each of the 3 hypothesized scales (Cronbach and Warrington 1951). Each item’s impact on internal consistency estimates was also noted. We also examined the relationships among the items using a multi-trait, multi-item correlation matrix, which allowed us to determine how highly an item correlated with its hypothesized scale and how that correlation related to the item’s correlation with other scales. We also examined the correlation between hypothesized composites. We expected that the IPT and DT composites would be most highly correlated given that IPT was developed using some concepts from DT. We also expected that the CBT and DT composites would have the lowest correlation given that these two therapies have quite distinct therapeutic approaches. Although we expected the composites to have low to moderate intercorrelations, we believed that each of the composites would provide unique information. We also noted whether an item had higher rate of missing or lower standard deviation than other items.

Approach to Evaluating the Shortened Instrument

The shortened instrument was re-evaluated using the analytic approach describe above, with 1 addition. We also conducted confirmatory factor analyses to evaluate the factor structure of the instrument. We anticipated that the items represented three distinct, but correlated factors of CBT, IPT, and DT. In addition to testing a 3-factor model, we evaluated competing 1- and 2-factor models to determine whether the 3-factor model provided a significant increase in model fit over the competing, more parsimonious models. A well-fitting 1-factor model would suggest that the items do not assess distinct types of therapy, but instead reflect a general approach to treating depression. The 2-factor model was specified so that the IPT and DT items loaded on the first factor (given the conceptual relationships between these therapies), and the CBT items loaded on the second factor. If this model provided a superior fit, it would suggest that the CBT items were distinct, but the IPT and DT items demonstrated a great degree of overlap. A priori, we favored the 3-factor model as conceptually superior and supporting the clinical literature that indicates these to be distinct therapies. To judge model fit, we used CFI (comparative fix index) and RMSEA (root-mean-square error of approximation) (Loehlin 1998). These analyses were conducted in SAS and Mplus (Muthen and Muthen 1998–2004).

In summary, to determine the items retained in the shortened instrument, we evaluated internal consistency reliability, corrected item total correlations, item discrimination, subscale intercorrelations, item missing, observed variability and factor models to select a subset of items that best represented each of the three therapy approaches.

Preliminary Validity Analyses

We created subscores designed to reflect CBT, DT, and IPT. Individual items were standardized (z-scores with a mean of 0 and standard deviation of 1), to ensure that an item’s variability did not impact its weight in the subscore, and then averaged. Scores were transformed into T-scores, with a mean of 50 and standard deviation of 10 for ease of interpretation. We compared scores on each of the three subscales by clinician-reported primary theoretical orientation using t-tests. In addition, we evaluated the correlation of each subscore with the number of years the clinician had been practicing.

Results

Evaluation of Fielded 28-Item Instrument

Table 2 provides a summary of key quantitative and qualitative results for the fielded 28-item instrument. This evidence guided the selection of a subset of 16 better performing items.

Table 2 Evaluation of 28 item version
Full size table

Confirmation of the Shortened Composite Structure

Based on the findings from the pilot test, we revised and shortened the PPS-Clinician. The shortened version presented here included 16 items assessing CBT (6 items), DT (6 items), and IPT (4 items). These items are in bold in Table 2.

The shortened instrument demonstrated good psychometric properties. The multi-trait, multi-item correlation matrix showed appropriate primary correlations (e.g., CBT technique items correlated most highly with the CBT subscale), although some moderate secondary correlations between IPT and DT items were observed (Table 3). Internal consistency reliability was acceptable (above 0.7) for each subscale (Table 4). Further, the correlations among the subscales supported the a priori hypotheses about the relationship among the different therapeutic approaches (Table 4). That is, the highest correlation was observed between DT and IPT, while the lowest correlation was observed between CBT and DT.

Table 3 Multi-trait multi-item correlation matrix
Full size table
Table 4 Properties of shortened composites
Full size table

Confirmatory factor analyses supported a 3-factor solution representing CBT, DT, and IPT. These analyses provided adequate fit for a 3-factor model (CFI = 0.87, RMSEA = 0.089) and improved fit over the 1-factor model (CFI = 0.53, RMSEA = 0.167) and the 2-factor model that specified DT and IPT on the same factor (CFI = 0.78, RMSEA = 0.114).

Preliminary Evaluation of Validity

There is also preliminary support for the validity of the instrument. We compared scores on each of the three subscales by the primary theoretical orientation as reported by the clinician. Figure 1 presents mean t-scores for each of subscores by clinician self-reported primary orientation. Clinicians who indicated their orientation (using the primary orientation variable) to be cognitive-behavioral scored significantly higher on the CBT subscale (P < .0001) and significantly lower on the DT subscale (P < .0001) than non-cognitive-behavioral therapists. Dynamic therapists scored significantly higher on the DT subscale (P < .0001) and significantly lower on the CBT subscale (P < .0001) than non-Dynamic therapists. IPT therapists had significantly lower scores on the CBT subscore (P < .0001), but did not differ on the IPT subscore compared to non-IPT therapists. However, when therapists were allowed to endorse multiple orientations, instead of only a single primary orientation, those who endorsed IPT did score significantly higher on the IPT subscore (P < .01). These subscore-by-orientation analyses provide preliminary support for the validity of the CBT and DT subscales, but they also suggest that the relationship between self-reported orientation and scores on the questionnaire may be more tenuous for IPT therapists than for therapists who endorse either CBT or DT.

Fig. 1
figure 1

Mean T-scores by primary orientation

Full size image

Additional analyses suggested that a higher frequency of using CBT techniques was significantly associated with fewer years of practice (r = −0.20, P < .0001), while use of IPT and DT techniques were not associated with years of practice.

Discussion

This study provides preliminary support for the reliability and validity of the three subscales assessed in the Psychotherapy Practice Questionnaire—Clinician Depression Care Version. The revised clinician instrument includes 16 items that assess the use of CBT (6 items), IPT (4 items), and DT (6 items) techniques with a single adult patient with depression. These 16 items were selected from a larger set of 28 items based on results from the pilot test. The revised PPS-Clinician demonstrates good psychometric properties. The factor structure of the instrument corresponds to the intended domains of CBT, IPT, and DT, and, in general, subscores relate to therapist orientation as expected. We believe that this 16-item version of the questionnaire would likely take 10–15 min for a clinician to complete. Our results suggest that a clinician-report questionnaire may be a promising, feasible approach to describing psychotherapy for depression in usual care and an efficient way to measure the extent to which clinicians practice evidence-based depression care.

Our results also provide some information about usual care psychotherapy for depression. While it may not be surprising that about half of responding clinicians endorsed cognitive-behavioral as their primary orientation, a substantial portion (21%) endorsed multiple orientations, suggesting that it is important to understand how clinicians implement eclectic approaches that blend techniques from multiple orientations and how these approaches relate to outcomes. Our results also suggest an inverse relationship between years of practice and CBT delivery. These findings may reflect efforts to disseminate CBT by emphasizing this approach in graduate programs for training new therapists.

While we believe that our approach is a promising direction in the development of a tool to describe psychotherapy for depression in usual care, there are limitations to our work. A chief concern is the unknown validity of clinician report of use of various psychotherapy techniques. It is unknown whether clinicians can reflect back on their treatment of a particular patient and accurately report on how frequently they used different techniques. Clinicians may also be susceptible to social desirability biases, such as overreporting using techniques from an approach endorsed by a managed care organization. Further, our use of self-reported therapeutic orientation as a preliminary evaluation of validity has limitations because it is unclear how this variable relates to actual practice. Therefore, an essential step in the validation of this approach would be to compare responses to these questionnaire items with ratings of audiotapes by trained raters. Future research will address the relationship between this clinician-report questionnaire, the patient-report version, and expert ratings of audiotapes.

The response rate we obtained (32%) is not uncommon in surveys of managed care network clinicians and highlights the challenges in conducting survey-based quality monitoring. Nevertheless, our response rate may limit our ability to generalize our findings to non-responding clinicians.

We also believe, however, that development work within a large national managed behavioral health care organization (MBHO) is a strength because it provides an opportunity to understand the practice of therapy for depression from the perspective of the many and diverse clinicians delivering care to the general population. Approximately 95% of American workers are covered by employer-sponsored managed care plans (Kaiser Family Foundation and Health Research and Educational Trust 2003). In addition by 2002, 58% of the Medicaid population were enrolled in managed care (Centers for Medicare and Medicaid Services 2002). MBHO’s are also likely to be interested in tools to facilitate quality assessment and quality improvement efforts. For example, recent research has shown that an MBHO-based depression program that identifies and promotes evidence-based treatment for depression identified in employees can significantly improve clinical and workplace outcomes (Wang et al. 2007). MBHOs, with their large provider panels, are in the position to monitor the quality of care provided to their members, and they have the organizational infrastructure to provide valuable information and feedback to a large number of geographically dispersed clinicians, including information on clinical outcomes, and the relationship of outcomes to treatment processes such as type of psychotherapy.

We believe that this clinician reported questionnaire has the potential to fill an important gap in understanding usual care psychotherapy. Questionnaire-based methods such as the one we describe here could provide a more practical, less costly alternative to expert rater methods. A practical tool would be valuable in describing usual care psychotherapy and in evaluating how use of these techniques predicts patient outcomes. As MBHOs have become more accountable to members and customers for quality of care, such a tool would allow for greater monitoring and could inform the development of quality improvement interventions.