Introduction

The idea of pay for performance (P4P), in which providers receive financial incentives for attaining quality or efficiency standards, is currently being examined as a potential evidence-based practice. In general health care, several initiatives including the Leapfrog Group and the “Rewarding Results” program are testing approaches (Dudley et al., 2004; Dudley, Miller, Korenbrot, & Luft, 1998; Rosenthal, Fernandopulle, & Landon, 2004). In behavioral health care, there have been few applications of P4P, and the evidence is primarily descriptive (Bachman, 2004). One of the most prominent is the purchaser’s (Massachusetts Medicaid) contract with the behavioral health carve out (MBHP), that set 123 performance standards between 1996 and 2004, with each standard generating from $333,000 to $400,000 per year (Lane, 2005). While incentive projects such as those are intended to steer the broad activities of an organization, they do not directly incentivize individual clinicians. We report here a before-and-after study to gather evidence around incentives for behavioral health counselors to reduce drop out and improve retention in treatment.

Client dropout is a recurrent problem in behavioral health care because it limits the effectiveness of treatment. In outpatient care, a series of observational studies found that longer durations of treatment were associated with better outcomes (DiClemente & Scott, 1997; Hubbard, Craddock, & Anderson, 2003; Hubbard et al., 1989). In residential care, a randomized trial confirmed a similar finding, where the investigators found that a longer planned duration of treatment of up to 6 months was more effective than shorter durations (McCusker et al., 1997). In lay terms, the famous actor, writer, and director Woody Allen immortalized this finding with the words: “Eighty percent of success in life is showing up.”

Showing up is not easy, however, for most clients in substance abuse treatment. Stark, Campbell, and Binkerhoff (1990), and Stark (1992) indicated that the majority of clients experienced difficulty continuing or completing treatment for substance abuse, regardless of demographic characteristics, personality traits, and the drug of choice. Noting the widespread nature of drop outs, Onken, Blaine, and Boren (1997) wrote that “a clinical trial of a drug addiction treatment without the problems created by dropout is, at this point, a fantasy.” Improving client retention in treatment, particularly in the first few sessions, is a key both for favorable outcomes and valid research. As Bachman (2004) noted, retention is a process indicator that lends itself more readily to financial incentives than do outcomes.

Little research has been done on the effect of counselor incentives on client retention. Theoretical support for counselor incentives is provided by a growing recognition that client retention may be more a reflection of treatment providers than clients (Carroll, 1997). In cases where well-established indicators of quality care can be identified, positive monetary incentives to providers or care produce consistently better outcomes (Dudley et al., 1998; Hanchak, Schlackman, & Harmon-Weiss, 1996).

Methods

Setting

Spectrum Site

Counselor incentives were examined at Spectrum Addiction Services in Westboro, MA, one of the two research sites in a randomized controlled trial of aftercare described below. Spectrum is a large, multi-site and multi-level program that provides treatment of drug and alcohol addiction. The aftercare program involved a new group of counselors and a new location.

RP Aftercare Program

Eligible Spectrum clients were enrolled in this randomized trial from June, 1994 through August, 1996. Program completion was defined as participating in 12 individual counseling sessions. A minimally adequate dose of treatment was considered to be attendance at five sessions. Sessions were usually scheduled once weekly, with the possibility of rescheduling if requested by the client or, rarely, by the counselor. Clients sometimes notified counselors in advance when they could not keep appointments and to reschedule, but often they would be no-shows and the counselor would need to take the initiative to pursue and contact the client to reschedule. The counselors were paid per session with a minimum nominal fee for clients who did not show.

Clients

The clients in this randomized, controlled study were primarily a Medicaid or uninsured population, all receiving treatment at the Spectrum site. Of the 220 participants in this study, 124 were randomized to receive relapse prevention aftercare (RP), and the remainder were randomized to receive standard aftercare, which was the control group. One client died before he completed all 12 sessions and is excluded from that analysis. Spectrum clients faced numerous personal challenges. All had used illegal drugs (primarily heroin); most had extensive criminal records, and, due to the severity of their addiction and personal problems, were sent initially to residential rehabilitation. Table 1 summarizes the clients’ major demographic and drug history characteristics.

Table 1 Demographic characteristics of clients (N = 123)
Full size table

Inadequacy of Previous Approaches to Retention

Several previous studies have examined client incentives as a means for increasing retention and modifying behavior in drug treatment programs (Higgins & Budney, 1997; Higgins, Wong, Badger, Ogden, & Dantona, 2000; Piotrowski et al., 1999; Rothfleisch, Elk, Rhoades, & Schmitz, 1999). For example, Higgins and colleagues (1997, 2000) demonstrated that client incentives almost doubled rates of treatment completion and improved abstinence over 27 months of follow up among cocaine-dependent clients. In the general medical literature, a study of an HIV/STD counseling intervention (Kamb et al., 1998) found that small monetary incentives paid to clients were more effective than non-monetary incentives of similar value on enrollment and treatment completion rates, but had no effect on post-enrollment STD rates.

In an attempt to mitigate the impact of clients’ barriers, the aftercare program incorporated several client incentives into the design from the outset. We reimbursed clients for bus fare when they came to treatment, paid $25 to compensate for transportation expenses and inconvenience for each of 5 follow up research interviews, and offered a $5 fast food voucher for the first aftercare session. After 6 months, to further strengthen recruitment and retention, we increased client incentives by providing $10 gift certificates for music CDs for attendance at each of 5 specified sessions, and awarded a framed certificate of completion for finishing of the full aftercare program of 12 sessions (Shepard et al., 2001). Although we lacked baseline data to determine the efficacy of these client incentives, we were concerned that retention remained far below our target of 80% completing our experimental program. At the midpoint of enrollment, 36% of subjects never attended a single aftercare session, and 67% failed to attend at least 5 sessions. As treatment researchers, we were concerned that drop outs would so dilute the anticipated effects of RP aftercare that they would be rendered undetectable.

Intervention

Counselor Incentives

We introduced the counselor incentive program to increase client retention in the face of the inadequacy of existing measures, including client incentives. The incentive program for counselors was initially introduced to prevent counselor turnover. Counselors delivering relapse prevention aftercare (the experimental group) were notified on November 1, 1995 that they would receive a bonus of $100 for each of their assigned clients who attained five sessions (considered a “milestone” session) after date and $50 for each client who completed the full course of 12 sessions, in addition to their normal compensation of $22 an hour for a one hour session. Counselors continued to receive $11 for no-shows. For clients who switched counselors in the course of their aftercare, the counselor who conducted the “milestone” fifth or twelfth session was the one eligible for the corresponding bonus.

Counselors

Table 2 summarizes the characteristics of the 11 counselors who participated in the study. One of the counselors delivered aftercare as part of his regular job at another agency under contract to Spectrum. The other counselors performed aftercare counseling as an additional task for which they received extra compensation. Their regular job involved counseling in Spectrum’s Residential, Detoxification, or Outpatient programs. Outpatient counselors were already familiar with a type of piecework compensation, as they received a base annual salary of $19,000 plus an additional payment for each paid session beyond a specified quota.

Table 2 Characteristics of counselors (N = 11)
Full size table

Analysis

Statistical Analysis

We first determined the number of days required from randomization to RP for each client to attain the milestone of one session, five sessions, and twelve sessions for those clients who actually reached the milestone. Based on data from this subset of clients, we calculated the 90th percentile time for each milestone (24 days for first session; 75 days for the fifth session; and 150 days for the 12th session). These 90th percentile values were imputed for those who dropped out before reaching their respective milestones by adding the 90th percentile times to their date of randomization into the program to calculate the date by which they would theoretically have attained the milestone if they had remained in the program. For those clients randomized during the time period when the counselor incentives were being offered, they could have potentially been retained in the program if the incentives had worked.

Dependent Variable

There were three dependent variables in this study: (1) client completed one session (Milestone 1); (2) client completed five sessions (Milestone 5); and (3) client completed 12 sessions (Milestone 12). One session was also a significant milestone because almost one third of the clients were lost before the first session. Five sessions was considered an adequate dose of treatment, and 12 sessions signaled successful completion of the program, a milestone attained by only 20% of the participants.

Independent Variables

The independent variable of greatest interest was the dose variable, a continuous variable measuring the dose of client exposure (or potential exposure) to counselor incentives in percentage terms. If all of the actual or potential number of days from randomization to a milestone occurred before the counselor incentives took effect on November 1, 1995, that client’s incentives dose was 0%; if the client’s entire period of time was actually or potentially entirely after the counselor incentives were introduced, then the dose value was 100%. Otherwise it was calculated as the percentage of the time to milestone which occurred during the time when counselor incentives were in place.

For descriptive analyses, we dichotomized this variable at the median exposure to incentives for each milestone. Each client whose counselor’s time with incentives below the median was classified as “Little or no incentives,” while those with at least the median time in incentives were classified as “Offered incentives.” To characterize clients, we defined three client independent variables: site (client assigned to RP in Worcester, the main site, vs. other), education (two dummy variables to indicate the three categories in Table 1), and long-term residential rehabilitation (client spent 30 days or more in residential rehabilitation prior to aftercare, vs. a shorter stay).

Logistic Regression

We conducted logistic regressions analyzing the relationship between dose of counselor incentives (forced into the regression) and achieving Milestone 1, Milestone 5, and Milestone 12. To control for possible confounding by change in client characteristics concurrent with the introduction of incentives, we used stepwise multivariate logistic regression for the possible inclusion of each of the three client characteristics. These analyses ensured that possible increases over time in the share of clients from more successful sites, those who were better educated, or those who received longer rehabilitation treatment were not inappropriately attributed to the success or failure of counselor incentives.

Results

Figure 1 graphs the achievement of each milestone under the two incentive conditions. Substantially higher shares of clients reached the Milestones of 1 and 5 sessions under incentives than with little or no incentives. Overall, the rate of achieving Milestone 5 was 41%. The proportion reaching Milestone 12 was also somewhat higher under incentives, but this change was not statistically significant.

Fig. 1
figure 1

Relation of incentives to milestones

Full size image

Under the multivariate analyses, none of the other independent variables were statistically significant, indicating that none of the three client characteristics affected retention. The results of the logistic regression analyzing the impact of counselor incentives on achievement of the milestones are summarized in Table 3. Even though a $50 incentive was given for Milestone 12, it did not have a significant effect. However, the incentive of $100 was very significant at Milestone 5.

Table 3 Binary logistic regression with only deaths (n = 1) censored
Full size table

Discussion

Overall Effectiveness

Despite modest incentives for clients in aftercare treatment for drug abuse, prior to the introduction of counselor incentives client retention in at least five treatment sessions was only 33%—far below the 80% retention we targeted. After adding a counselor incentive program that encouraged intensive follow-up, client retention for at least five sessions jumped to 59%.

Traditionally, efforts to explain attrition in clinical trials and treatment programs for substance abusers have focused on client characteristics associated with poor retention, such as demographic characteristics, social instability, and low motivation (Carroll, 1997). The importance of the counselor on client retention was shown by Luborsky, McLellan, Woody, O’Brien, and Auerbach (1985) and Luborsky, Barber, Siqueland, McLellan, and Woody (1997), who identified those characteristics associated with retention and outcomes of substance-abuse clients, including the therapist’s interest in helping, skill, and ability to form a good working relationship (therapeutic alliance). Focusing on the therapeutic relationship has led to studies of the importance of achieving a good fit between therapist and client in terms of personality and attitude (Beutler, Zetzer, & Yost, 1997). Greater understanding of factors which strengthen the therapeutic alliance has led to underscoring the counselor’s role and activities which facilitate the relationship with their client and thus promote retention (Newman, 1997; Liese & Beck 1997; Luborsky et al. 1997). The findings of this study suggest that, regardless of the counselor’s assignment, his or her effectiveness was strengthened by counselor incentives.

Provider Satisfaction

In previous studies, incentives targeted to promote quality care are associated with increased provider satisfaction (Grumbach, Osmond, Vranizam, Jaffe, & Bindman, 1998). Anecdotally, counselors in this study were very pleased with the incentives. It is plausible that clients recognized their counselor’s enthusiasm and were thus more likely to attend their scheduled sessions.

The counselors in this study, like those in many settings, needed to balance time scheduling sessions and meeting client needs against other professional and personal demands in their own lives. Previous research has found that client retention is improved by flexible scheduling, reminding clients of upcoming appointments, anticipating and accommodating for client difficulties in keeping appointments, emphasizing the importance to the client of completing an adequate amount of treatment, following up and pursuing clients by phone who miss appointments, and persisting in contacting clients beyond lapses in treatment (Carroll, 1997; Newman, 1997). Anecdotally, counselors reported greater efforts on all these activities after incentives.

Cost-effectiveness: Counselors’ and Payers’ Perspectives

A useful heuristic for both counselors and payors around incentives is an S-shaped curve, similar to the compressed cumulative normal curve or cumulative logistic, with “effort” on the horizontal axis and the rate of success on the vertical axis. Due to compression, the curve does not necessarily reach 100%. The ranges of effort in which the curve rises most steeply (the midpoint between the lowest and highest rates of success) are likely those most responsive to incentives. As Milestone 5 in this study had success rates of 33% before incentives and 59% after incentives (a difference of 26 percentage points), it corresponded to the steep slope in this heuristic. Thus, the divergent result between Milestones 5 and 12 can probably be attributed to the counselors’ figuring that the extra measures which must be taken to encourage a client to achieve Milestone 12 would have required an almost impossible effort (in view of the lower baseline rate) for a lower incentive.

In examining P4P incentives, payers consider not only the impact on quality, but also the impact on program costs (Commons, McGuire & Riordan, 1997; Kamis-Gould, 1987). The cost to payers of offering a meaningful incentive to each client is sometimes appreciable, particularly in relation to the $1,169 average cost of an episode of outpatient substance abuse treatment (Shepard, Beaston-Blaakman, & Horgan, 2003). In studies by Higgins and Budney (1997) and Higgins et al. (2000), incentives increased costs by about $600 per client, with a possible maximum of $998 per client. In the study by Piotrowsky et al. (1999), participants earned an average of $120 with a possible maximum of $755 per client. Based on the success rate in this study, the average cost per client for counselor incentives was $70 ($59 for the 5-session milestone and $11 for the 12-session milestone)—a somewhat lower amount. Research could be done on varying levels of incentives, as was recently reported for client incentives (Ritter et al., 2005).

It is instructive to contrast counselor incentives against other tools of evidence-based practices—performance measurement alone. R.H. Beinecke, D.S. Shepard, and C.L. Hurley (2006) summarize a presentation by Horgan and Garnick at the at the Brandeis/Suffolk forum “Implementing Evidence-Based Mental Health Practices and Performance Measures in Massachusetts.” Performance measurement can calibrate performance, but does not necessarily improve it. In fact, results from a meta-analysis showed that audit and feedback alone did not significantly reduce rates of non-compliance (Jamtvedt, Young, Kristoffersen, Thomson O’Brien, & Oxman, 2005). Incentives are one of the promising approaches towards improving performance.

Potential Confounders

The before-and-after study design used in this study has known weaknesses. Other changes occurring around the same time as the introduction of incentives could confound the results. In this study, we were aware of only one potential confound. About a month prior to the start of counselor incentives, Spectrum moved the office of the researchers evaluating relapse prevention from a different building to the building used for residential rehabilitation (the program from which clients enrolled). This change may have disposed the clients more generally towards the research study, but the likely effect on retention in aftercare treatment was small or zero. Retention in follow up in the research study ranged from 75% to 80% of planned interviews both before and after the research office moved. Furthermore, the retention measure to which the incentives applied was retention in aftercare counseling sessions. Counseling was never performed by research staff, aftercare counseling was never conducted in the research office, and the location of aftercare counseling sessions did not change with incentives.

Although we were not able to evaluate client incentives in this study, retention remained inadequate despite moderate incentives averaging of $5 per session. This comparison suggests that counselor incentives are not only one of the more promising approaches to improving retention, but are likely to be very cost-effective as well.