Introduction

The current study evaluated the effectiveness of the following three approaches for treating dual disorder clients (i.e., individuals that have both a severe mental illness and a substance abuse disorder) who were homeless at intake: integrated assertive community treatment (IACT), assertive community treatment only (ACTO), and standard care (SC). A primary focus of this paper is the identification of the appropriate analysis of longitudinal data from randomized intervention studies. In addition, statistical analyses were undertaken to identify mediators of significant treatment effects (MacKinnon et al. 2007).

Substance Use and Severe Mental Illness

About one-half of people with severe mental illness also have a co-occurring substance abuse disorder (Regier et al. 1990). Compared to similar individuals who have no substance use disorders, dual disorder individuals are more prone to higher relapse rates, more physical health problems, greater violence, higher incarceration rates, more frequent hospitalizations, and higher treatment costs. Furthermore, dual disorder individuals who become homeless are often less responsive to treatment interventions than other homeless individuals (Drake et al. 2001).

Most service systems are poorly equipped to respond to the multiple and serious needs of people with dual disorders. Many communities operate a “parallel treatment” system for persons with dual disorders. In such systems, individuals must go to one agency for mental health treatment and another agency for substance abuse treatment. Research suggests that client outcomes in parallel treatment systems are generally poor (Drake et al. 2001).

Over the past two decades, service providers and researchers have attempted to craft more effective treatment approaches for people with co-occurring disorders. Considerable attention has been directed toward developing “integrated treatment” approaches for the problems of people with dual disorders. With integrated treatment approaches, the client receives treatment for both mental health and substance abuse disorders concurrently from the same clinician or team of clinicians. As it has evolved, integrated treatment has come to emphasize (a) assertive outreach; (b) motivational interventions; (c) a stages-of-treatment approach; (d) cognitive behavioral counseling; (e) interventions to strengthen social networks supportive of recovery; and (f) a long-term perspective (Drake et al. 2001). Although there is some research to support the effectiveness of integrated treatment, many earlier studies had inadequate research designs and the size of the treatment effects in most studies have been modest (Drake et al. 1998b, 2004).

Considerable research has shown that assertive community treatment (ACT) produces better outcomes than other treatments for individuals with severe mental illness, especially for housing and consumer satisfaction (Bond et al. 2001; Mueser et al. 1998). Consequently, some researchers have argued that the ideal way to serve dual disorder individuals is to combine integrated treatment with ACT (Phillips et al. 2001). One study compared integrated treatment within an assertive community treatment team against standard case management (Drake et al. 1998a). Clients in the integrated treatment/assertive community treatment condition had better outcomes on three substance abuse measures than standard case management, but there were no significant differences between treatment conditions on three other substance abuse measures or for psychiatric symptoms.

The current project compared SC, ACTO, and IACT over a 30-month period. We predicted that ACTO and IACT would have better outcomes than SC on client satisfaction, days in stable housing, and psychiatric symptoms. In addition, we predicted that IACT would have better outcomes on the substance use variable than ACTO and SC because the IACT team would be providing both substance abuse and mental health treatment in an integrated manner.

Mediation Analyses

Mediators are intervening variables that occur after participants have been assigned to treatment, but before the measurement of the outcome variable; thus, mediating variables attempt to explain how the treatment variable affected the outcome variable. The study of mediators allows researchers and clinicians to better explain how an intervention produces its effects. Variables such as the number of sessions, the therapeutic alliance, the type of services received are examples of mediating variables used in evaluating mental health interventions. More specifically, the ACT model argues that clients with severe mental illness need: (1) frequent services (intensity), (2) a wide range of services (specificity), and (3) services need to be provided continuously (Brekke et al. 1997). A longitudinal study of schizophrenic clients in three service programs has provided modest support for all three tenets (Brekke et al. 1997). Clients that were provided more frequent contact, a wider range of services and over a longer period of time were more likely to be living independently. Below we provide the specific rationale for the mediators that we chose to investigate in comparing the effectiveness of SC, ACTO, and IACT on consumer satisfaction, housing outcomes, psychiatric symptoms, and substance use.

Consumer Satisfaction

Morse et al. (1994) reported that ACT clients were more satisfied with their treatment program than clients from other programs. They also found that program contact, counseling services, and assistance with daily living were significant mediators of client satisfaction. The present project hypothesized that program contact, help with emotional problems, substance abuse contacts, and assistance with daily living would be significant mediators of any treatment effect on consumer satisfaction.

Stable Housing

Two previous studies found that ACT clients had better housing outcomes than clients in the other programs (Kenny et al. 2004; Morse et al. 1994). In addition, both studies found that financial assistance and assistance in searching for housing mediated a positive increase in the number of days in stable housing. The Morse et al. study (1994) also found that assistance with daily living and help with emotional problems were also significant mediators of the housing effect. The proposed study hypothesized that substance abuse contacts from the assigned program, help finding permanent housing, assistance with daily living, and help with emotional problems would be significant mediators of any housing effect found in the current study.

Psychiatric Symptoms

Although Kenny et al. (2004) found that ACT produced better outcomes than brokered case management in reducing psychiatric symptoms, none of the eight potential mediators mediated the psychiatric symptoms effect. ACT had no significant effect on psychiatric symptoms in the Morse et al. study (1994). Despite the paucity of previous research we predicted that the following service utilization variables would be significant mediators of change in psychiatric symptoms: number of program contacts, number of substance abuse contacts from the assigned program, assistance with daily living, and help with emotional problems.

Substance Abuse

Neither Morse et al. (1994) nor Kenny et al. (2004) found a significant effect of ACT on substance use, so neither study performed a mediation analysis for this outcome variable. The proposed project hypothesized that IACT would be superior to both ACTO and SC in reducing substance use. The authors also predicted that program contacts, substance abuse contacts, help with emotional problems, and assistance with daily living services would be significant mediators of the substance use effect.

Methods

Sample and Research Design

To be eligible for this study individuals had to meet the following criteria: (1) be homeless (e.g., currently staying in a shelter, living in an abandoned building, sleeping in a car or a public place); (2) have a severe mental illness such as schizophrenia, atypical psychosis, bipolar disorder, recurrent major depression, schizo affective disorder, or delusional disorder; (3) have a DSM-IV substance use disorder; and (4) not be currently enrolled in an intensive case management program. The Structured Clinical Interview for DSM-IV AXIS I Disorders (SCID) developed by First et al. (1996) was used to obtain both the psychiatric and substance use diagnoses.

The 191 eligible participants were randomly assigned to one of the three conditions: (1) IACT, (2) ACTO, or (3) SC. Participants were followed for a period of 30 months. The participants had the following demographic characteristics: 80% of the sample was male; 71% of the participants were minorities (primarily African American) and 28% Caucasian; the mean age was 40 (SD = 9.13) years and ranged from 18 to 66 years; 42% of the participants failed to graduate from high school; 54% were never married. The average Global Assessment of Functioning score (American Psychiatric Association 2000) was 43.77 indicating considerable impairment. All of the participants had one or more substance use disorders. Forty percent had an alcohol-only diagnosis, 18% had a drug-only diagnosis, and 42% had both drug and alcohol disorders. Cocaine (usually crack) was the most frequently used drug (29%) followed by cannabis (22%).

Procedures

Potential participants were approached in a variety of settings (e.g., emergency shelters, soup kitchens, psychiatric hospitals, and street locations frequented by homeless people). Once suspected eligible individuals were screened and formally qualified for the project, they provided informed consent via Institutional Review Board approved procedures and were paid $10 to complete the SCID and answer the other eligibility questions. Eligible participants were interviewed monthly for 30 months. Participants were paid $5 for the shorter interviews which took less than 30 min and $10 for the longer quarterly interviews which lasted about an hour. Individuals who were not eligible for the project were referred to other agencies for services.

Treatment Conditions

A new IACT clinical team was created for this project, although several team members had prior experience providing ACT. The ACTO condition was implemented in two other agencies that served persons with severe mental illness, including homeless individuals. Both the IACT and ACTO teams received training and follow-up consultation regarding ACT treatment principles and practices. Additionally, experts provided the IACT team with training and consultation on integrated treatment principles and services. The IACT team had a substance abuse specialist on staff and provided outpatient substance abuse counseling and bi-weekly treatment groups. The ACTO team referred clients to other community providers for outpatient or individual substance abuse services and to 12-step groups. Participants assigned to SC were shown a list of community agencies that provided mental health and substance abuse treatment. Research staff provided these participants with information about treatment openings and assisted individuals in making their initial contact with an agency.

Treatment Fidelity

Research staff assessed treatment fidelity of the IACT and ACTO conditions using a revised and expanded version of the Dartmouth Assertive Community Treatment Scale (DACTS) at two points in time, 12 and 24 months after project initiation. This 35-item instrument contained 26 items that focused on fidelity to the ACT model (Winter and Calsyn 2000) and nine new items that focused on implementation of substance abuse treatments including motivational interviewing, skills approach to substance abuse counseling, and comprehensive substance abuse assessment. Both the IACT and ACTO conditions had moderate to high scores on fidelity to the original ACT model. The IACT condition, as expected, scored higher than the ACTO condition on the nine substance abuse items, but treatment fidelity of the substance abuse components of the IACT condition was less than desirable. More information on treatment fidelity for the project can be found in Morse et al. (2006). The mediator analyses in this study also provide additional information on treatment implementation.

Outcome Measures

Consumer Satisfaction

Every 3 months participants reported their satisfaction with the assigned treatment program using a 10-item scale developed for this project. For each item, participants indicated their degree of satisfaction on a six-point scale, with higher scores indicating greater satisfaction. Coefficient alphas for this scale ranged from .83 to .92 at the various time points.

Stable Housing

Every month participants reported on their housing situation, including days living in stable housing (i.e., living in one’s own apartment or a boarding home). Monthly scores were averaged across quarters.

Psychiatric Symptoms

The 24-item Brief Psychiatric Rating Scale was used to assess psychiatric symptoms (Lukoff et al. 1986). For each item, scores can range from 1—‘no symptoms’ to 7—‘extremely severe’. The average of the 24 items was used as the scale score in the analysis.

Substance Abuse

Every 3 months the research interviewer assessed the severity of both alcohol and drug use with two five-point scales that have been used in many previous studies (Carey et al. 1996), with 1—‘client has not used alcohol (or drugs)’, to 5—‘meets criteria for severe use plus related problems are so severe that make non-institutional living difficult’. Because many clients only abused one substance, these data were skewed with “1” being the most frequent response chosen. To reduce skewness, we created a “highest substance abuse” rating for each participant, depending on which substance (alcohol or drugs) the participant abused the most.

Mediators

The mediators were service utilization variables. On a monthly basis participants were asked to report on the number of days that they had contact with their assigned treatment program (program contacts), the number of days that they discussed substance abuse problems with their assigned program (substance abuse contacts), and the number of days that they spoke with their assigned program on the phone (phone contacts). Participants were also asked whether their assigned program helped them with the following: (1) finding permanent housing, (2) activities of daily living (ADL), including emergency food assistance, help budgeting, help cooking or housecleaning, (3) emotional problems, (4) medication, and (5) transportation. For analysis purposes, the monthly data were aggregated into quarters by averaging data for every 3 months together. Therefore, the scores on the contact variables could range from 0 to 30. The questions regarding help received were dichotomous variables (yes or no) at the monthly level. However, because scores were averaged across quarters, they could take on several values between 0 and 1.

Analytic Strategy

We integrate a multilevel approach to analyzing longitudinal data (Singer and Willet 2003) with traditional methods for assessing mediation (MacKinnon et al. 2007) to test the study hypotheses. We used the NLME package for R (Pinheiro and Bates 2000; Pinheiro et al. 2006) for the multilevel analyses. The basic steps include, analyze the dependent variable (DV) to understand (1) amount of variance attributable to persons and time (i.e., between and within persons), and (2) to model the change in the DV over time. Next we assess the effects of the intervention using a contrast coding scheme on the change in DV over time. That is, we ask, “Does the intervention account for changes in the DV” and “are the intervention effects differentially related to changes in the DV?” Finally, we are interested in how various mediators—program services—account for the relationship between the intervention and the change in the DV.

We generalized traditional methods of mediation testing (e.g., Baron and Kenny 1986; MacKinnon et al. 2002) for the present longitudinal study. We first demonstrate that the intervention has a relationship with the mediator. This relationship could be a main effect (i.e., an effect at a single point in time), or a change process (i.e., treatment * time interaction). Second, we demonstrate that the mediator is related to the outcome. Again, the relationship could be a main effect or change related. These two steps define the paths a and b in the path analytic approach (i.e., computation of indirect effect) and steps 2 and 3 from the Baron and Kenny (1986) procedure. However, the generalization is blurred by the multiple paths (i.e., static and dynamic representations of a and b); therefore, no indirect effect is estimated. The joint significance of a and b coupled with the interpretation of any change in the treatment to outcome relationship (i.e., total effect) is used as evidence of mediation.

Particular decisions made relevant to the analyses here include: (1) we allowed for random intercepts, but did not allow for random slopes; (2) we determined a 3rd order polynomial best described the data via a tear-down procedure (Cohen et al. 2003) and used orthogonal polynomials to reduce effects of multicollinearity (Cohen et al. 2003; Ployhart et al. 2002) recognizing the limitations associated with their interpretation (Biesanz et al. 2004); (3) based on previous literature (Kenny et al. 2004) and model comparisons involving various error structures, we modeled a first-order autoregressive error structure; (4) we used maximum likelihood estimation; and (5) the treatment conditions were contrast coded to represent (ACTO & IACT vs. SC) and (ACTO vs. IACT).

In addition to the statistical analyses, we utilized a graphical approach to analyzing the data (Cleveland 1993; Tukey 1977). We used a lowess (locally weighted scatterplot smoother) to graph the relationships over time to facilitate the interpretation of the statistical results (Cohen et al. 2003). For all graphs, we set the y-axis limits equal to the 1st and 3rd quartiles of the particular variable being graphed. A more technical description of the models assessed is available from the first author.

Results

Descriptives

Table 1 displays the means and standard deviations for all dependent variables and mediators by experimental condition for each of three time periods. Sample size varies across variables due to some clients’ refusal to answer questions at some time periods and some variables were not measured at baseline (i.e., consumer satisfaction, program contact, phone contact, and substance abuse contact). Focusing on the eight variables for which we have baseline data, the sample size only decreases slightly over time with the exception of the last time period. Over all time periods, only 6.9% of the data are missing, with an average of 22.5% missing at 30 months. Tests of independence (χ2) show that neither missingness (overall missing data) nor attrition (missingness over time) is related to study condition.

Table 1 Means and standard deviations at three time periods for outcomes and all program services
Full size table

Table 2 displays the correlation among the mediators (program services) for three-time periods (3, 15, 30 months). It can be noted that (1) correlations are higher for points closer in time, and (2) several of the services are related to one another but both empirically and conceptually distinct (e.g., transportation and phone contact). Table 3 displays the correlations among the outcomes with the mediators for three-time periods. This table demonstrates the very modest relationships that exist between the mediators and the outcomes in this study.

Table 2 Correlations among mediators over time
Full size table
Table 3 Correlations between mediators and outcomes over time
Full size table

Null Models

The null models are useful for determining the intra-class correlation, or proportion of variance between individuals. The ICCs are computed from the variance components from the null models. For consumer satisfaction, τ00 = .65, σ2 = .46, and the ICC = .58. Approximately 58% of the variance in consumer satisfaction is between individuals, whereas about 42% of the variance in consumer satisfaction is attributable to change over time. It is important to note here that the treatment effects have implications for both the between individual differences (e.g., different individuals assigned to different conditions) and within person change (e.g., treatment affects individuals over time). For stable housing, τ00 = 66.5, σ2 = 101.9, and the ICC = .39, thus 39% of variance is between individuals and 61% is within individuals over time. For psychiatric symptoms, τ00 = .10, σ2 = .26, and the ICC = .28, thus 28% of variance is between individuals and 72% is within individuals over time. For substance abuse rating, τ00 = .39, σ2 = 1.05, and the ICC = .28, thus 28% of variance is between individuals and 72% is within individuals over time.

Intervention Effects on Outcomes: Y = X

We present the results in graphical form with textual descriptions of the statistical results. Complete tables containing the technical results are available from the first author. We describe the more prominent effects related to the intervention and mediation as hypothesized rather than every significant effect.

Consumer Satisfaction

Overall, consumer satisfaction gradually declines over time (i.e., negative linear trend) until about mid-way through the intervention at which point the decline in satisfaction becomes steeper (i.e., negative quadratic trend). The negative linear and quadratic trends are apparent in Fig. 1. The treatment contrast (ACTO & IACT vs. SC) is significant and positive (P = .02). This indicates a main effect at about the mid-point of the intervention (when the orthogonally coded time is zero). Both treatment groups are more satisfied than the SC group (see Fig. 1). The treatment contrast (ACTO & IACT vs. SC) marginally interacts with the linear time effect (P = .06).Footnote 1 This indicates a difference between the treatment groups and the SC group in the initial rate of change in satisfaction. The variance explained at level-1 (\( R_1^2 \)) by this model is .04 (see Snijders and Bosker 1994).

Fig. 1
figure 1

Differential changes in outcomes by treatment condition. Graphs depict lowess (locally weighted scatterplot smoother) of the raw data rather than actual values

Full size image

Stable Housing

There is an overall significant linear and quadratic effect for stable housing (see Fig. 1). There is continued improvement through about 15 months at which point the rate of change begins to asymptote. The treatment contrast (ACTO & IACT vs. SC) is significant and positive (P = .01). Both treatment groups enjoy a higher rate of stable housing than does the SC group. The treatment contrast (ACTO & IACT vs. SC) marginally interacts with the linear time effect (P = .09) and the quadratic time effect (P = .09). The treatment groups both have a higher initial rate of improvement (see Fig. 1). Further, the SC group reaches asymptote earlier, at about 12 months in comparison with the ACT treatment group, at about 17.5 months. The treatment contrast (ACTO vs. IACT) interacts with the quadratic time effect (P = .09). This interaction can be seen in Fig. 1, where there is a departure in the slopes for the two groups after about 10 months (i.e., ACTO continues to improve, but IACT slows in rate of improvement). The variance explained at level-1 (\( R_1^2 \)) by this model is .17 (see Snijders and Bosker 1994).

Psychiatric Symptoms

For the model involving psychiatric symptoms as the DV, all three polynomials for time are significant (P < .01). Neither treatment contrast is significant as a main effect (Ps > .1). That is, at the midpoint of the intervention, there are no differences in treatment groups versus SC nor any differences between IACT or ACTO with respect to psychiatric symptoms. However, the treatment contrast (ACTO vs. IACT) significantly interacts with the cubic time effect (P = .02). As seen in Fig. 1, ACTO levels off at about 7.5 months and then begins to improve again at about 17.5 months; however, IACT has a much steadier rate of improvement throughout the 30-month period—any leveling in improvement is much more gradual for IACT than ACTO. The variance explained at level-1 (\( R_1^2 \)) by this model is .08 (see Snidjers and Bosker 1994).

Substance Abuse

For the model involving substance abuse as the DV, all three polynomials for time are significant (P < .01). Once again, neither treatment contrast is significant as a main effect (Ps > .1). That is, at the midpoint of the intervention, there are no differences in treatment groups versus SC nor any differences between IACT or ACTO with respect to substance abuse ratings. However, the treatment contrast (ACTO vs. IACT) significantly interacts with the quadratic time effect (P = .03). Both groups begin with the same level of improvement (i.e., non-significant interaction with linear), but ACTO begins to level off at about 7.5 months (see Fig. 1), whereas IACT continues to improve until about 17.5 months before leveling off. The treatment contrast (ACTO vs. IACT) also interacts with the cubic time effect (P = .07). This interaction denotes the two bends in ACTO, but only one bend in the IACT group. The variance explained at level-1 (\( R_1^2 \)) by this model is .05 (see Snijders and Bosker 1994).

To summarize the effects of the intervention on these outcomes, ACTO and IACT are significantly different from SC (as a main effect) for consumer satisfaction and stable housing, but not for psychiatric symptoms or substance abuse rating. There is no main effect for any of the outcomes in comparing ACTO and IACT. The main effect reflects a difference in these contrasts at a single point in time (i.e., study mid-point). With respect to change in the outcomes, ACTO and IACT have a different change pattern than the SC condition for consumer satisfaction, stable housing, and psychiatric symptoms. Furthermore, ACTO has a different change pattern than IACT for stable housing, psychiatric symptoms, and substance abuse rating.

Intervention Effects on Mediators: M = X

In the spirit of traditional mediation testing, we also assessed the effects of the intervention on each of the mediators. For all eight, the treatment intervention was either related to the mediator as a main effect (i.e., the contrast code was significant indicating a mean difference mid-way through the intervention), or as an interaction with time. Additionally, we assessed the change in each of the mediators over the 30-month period.

Overall, there is no significant change in contacts, phone contacts, or medication. There were overall changes in substance abuse contact, help finding permanent housing, help with activities of daily living, transportation assistance, and help with emotional problems. Each of these trajectories highlights the fact that the program services were dynamic, changing as needed throughout the 30-month period. It is important to note that these descriptions of change are ignoring any differences due to the intervention. Some of the change processes are cancelled out when collapsing across treatment conditions (i.e., change in different directions across treatment programs).

In this study, examining the treatment contrast main effects on the mediators and the treatment by time interactions on the mediators also provide information relevant to treatment implementation and treatment fidelity. The treatment contrast ACTO & IACT vs. SC examines differences between SC versus both assertive community treatment conditions. As expected both ACTO and IACT had higher mean levels than SC on all of the mediators: program contacts, phone contacts, substance abuse contacts, permanent housing assistance, help with activities of daily living, transportation assistance, help with emotional problems and help with medication. These data provide strong evidence that ACTO and IACT were much more comprehensive interventions than SC as well as some evidence regarding treatment fidelity to the ACT model. There are also treatment by time interactions on the mediators (see Fig. 2). Perhaps the most notable effect occurs on substance abuse contacts. Both ACTO and IACT follow a negative linear trend over time, whereas the SC group follows an inverted-U shaped trajectory (i.e., significant negative interaction between the quadratic time effect and the contrast [ACTO & IACT vs. SC], P = .04). The pattern for transportation assistance is also quite different for the groups. Both ACTO and IACT have an initial upward trend and then a leveling off, whereas SC has an initial downward trend and then a leveling off (i.e., significant interaction between the linear, quadratic and cubic time effects and the contrast [ACTO & IACT vs. SC], Ps < .01).

Fig. 2
figure 2

Differential changes in program services by treatment condition. Note for permanent housing, the SC line does not appear in the figure but should resemble a relatively straight line at zero between 0 and 30 months. Graphs depict lowess (locally weighted scatterplot smoother) of the raw data rather than actual values

Full size image

The treatment contrast of ACTO vs. IACT examined differences between ACTO and IACT. We had hypothesized that IACT would have higher scores on substance abuse contacts than ACTO. However, this effect only reached marginal significance (P < .07), indicating some problems with treatment implementation of the IACT condition. We did not expect there to be significant differences between ACTO and IACT on the other mediators. However, there were significant differences between ACTO and IACT on the following mediators: program contacts, phone contacts, permanent housing assistance, help with activities of daily living, transportation assistance, and help with emotional problems (Ps < .05). These results indicate that, in general, the ACTO condition provided more services than the IACT condition in this particular study. However, it is also important to note that there were treatment by time interactions which affected the mediators: Time * (ACTO & IACT vs. SC) for substance abuse contact, help with activities of daily living, transportation assistance, and help with emotional problems (Ps < .05); Time * (ACTO vs. IACT) for program contacts, phone contacts, and help with activities of daily living (Ps < .05). These interactions in change trajectories can be seen graphically in Fig. 2. In summary, the treatment conditions affected the all program services investigated in this study either as main effects or through change processes.

Tests of Mediation: Y = M + X

In the following section, we first describe the effects of the program services (mediators) on the outcomes across conditions, controlling for the intervention and time effects. Then we illustrate how the mediators modify each of the intervention effects. Table 4 summarizes the relationship of the eight mediators to each on the four outcome variables for each of the treatment effect contrasts and controlling for the intervention and time effects. Figures 3–6 provide graphs of the outcomes over time with and without mediators included in the models.

Fig. 3
figure 3

Consumer satisfaction and three example intervening variables: program contact, phone contact, and substance abuse contact. Upper left cell is the fitted values of the model without accounting for any intervening effects. Graphs depict lowess (locally weighted scatterplot smoother) of the fitted models rather than actual values

Full size image
Fig. 4
figure 4

Stable housing and three example intervening variables: substance abuse contact, phone contact, and permanent housing. Upper left cell is the fitted values of the model without accounting for any intervening effects. Graphs depict lowess (locally weighted scatterplot smoother) of the fitted models rather than actual values

Full size image
Fig. 5
figure 5

Psychiatric symptoms and three example intervening variables: assistance with daily living, transportation help, and help with emotional problems. Upper left cell is the fitted values of the model without accounting for any intervening effects. Graphs depict lowess (locally weighted scatterplot smoother) of the fitted models rather than actual values

Full size image
Fig. 6
figure 6

Substance abuse and three example intervening variables: substance abuse contact, transportation help, and help with emotional problems. Upper left cell is the fitted values of the model without accounting for any intervening effects. Graphs depict lowess (locally weighted scatterplot smoother) of the fitted models rather than actual values

Full size image
Table 4 Summary of mediation effects
Full size table

Consumer Satisfaction

The following program services (mediators) were related to consumer satisfaction across treatment conditions (i.e., after controlling for the intervention and time effects): program contacts, phone contacts, substance abuse contacts, help with activities of daily living, transportation assistance, and help with emotional problems. In addition, phone contacts interacts with the quadratic time effect (P = .05), indicating that the effect of phone contacts on satisfaction changes curvilinearly over time.

Recall that the primary relationship of the intervention to satisfaction was through the contrast (ACTO & IACT vs. SC). Only when program contacts is entered into a model is this contrast reduced to non-significance. Finally, the linear interaction with the contrast (ACTO & IACT vs. SC) is reduced modestly for models involving five of the services (program contact, substance abuse contact, help with activities of daily living, transportation assistance and help with emotional problems). This interaction was increased for the model including phone contact. Figure 3 shows the fitted models for consumer satisfaction with (1) no mediators, (2) program contact, (3) phone contact, and (4) substance abuse contact to illustrate the size and direction of a sample of the mediated effects.

Stable Housing

The following program services were related to stable housing regardless of treatment condition (i.e., after controlling for the intervention and time): program contacts, substance abuse contacts, help finding permanent housing (negatively), help with activities of daily living, and help with medication (Ps < .05). In addition, help finding permanent housing interacts with the cubic time effect (P = .02), indicating that the effect of housing help on stable housing changes curvilinearly (with at least two bends) over time. Finally, transportation assistance interacts with the linear time effect (P = .00) indicating a linear change in the relationship of the mediator to the outcome.

With respect to the main effect (ACTO & IACT vs. SC), the contrast is not reduced to non-significance by including any of the mediators into the model. However, the effect is modestly reduced in the models involving the following services: program contact, assistance with activities of daily living, and help with medication. With respect to the linear time by contrast (ACTO & IACT vs. SC) interaction, the effect was reduced in models involving the following services: program contacts, substance abuse contacts, assistance with activities of daily living, transportation assistance, and help with medication. None of the services greatly reduced the quadratic time by contrast (ACTO & IACT vs. SC) interaction effect. Models involving the following services reduced the quadratic time by contrast (ACTO vs. IACT) effect: program contacts, substance abuse contacts, and transportation assistance. Figure 4 shows the fitted models for stable housing with (1) no mediators, (2) substance abuse contact, (3) phone contact, and (4) help finding permanent housing to illustrate the size and direction of a sample of the mediated effects.

Psychiatric Symptoms

The following program services were negatively related to psychiatric symptoms regardless of treatment condition (i.e., after controlling for the intervention and time): transportation assistance and help with medication (Ps < .01). Both assistance with activities of daily living and transportation assistance interacted with the quadratic time effect (Ps < .05) indicating a curvilinear relationship with psychiatric symptoms. Finally, assistance with activities of daily living, transportation assistance, and help with emotional problems interacted with the cubic time effect (Ps < .05) indicating a complex change relationship with psychiatric symptoms.

The intervention was related to psychiatric symptoms via an interaction with the cubic time effect and the contrast (ACTO & IACT vs. SC). This interaction effect was reduced to non-significance only for models involving assistance with activities of daily living and help with emotional problems. The interaction effect was only modestly reduced for models including transportation assistance and help with medication. Figure 5 shows the fitted models for psychiatric symptoms with (1) no mediators, (2) assistance with activities of daily living, (3) transportation assistance, and (4) help with emotional problems to illustrate the size and direction of a sample of the mediated effects.

Substance Abuse

The relationship among the program service mediators and substance abuse outcomes controlling for the intervention and time effects is somewhat complex. Substance abuse contact and transportation assistance negatively interacted with the quadratic time effect indicating a non-linear relationship with the outcome (Ps < .01). Contrary to expectations, help with emotional problems was positively associated with substance abuse rating as a main effect and interaction with the linear time effect (Ps < .05).

The intervention was related to substance abuse ratings via the interaction with the quadratic and cubic time effect with the contrast (ACTO vs. IACT). For the quadratic interaction, the model with transportation assistance resulted in a reduction to non-significance; however, the model involving substance abuse contact resulted in an increase in the interaction effect. The converse was true for the cubic interaction effect: substance abuse contact resulted in reduction to non-significance and transportation assistance resulted in an increase in the effect. The model involving help with emotional problems resulted in only slight downward shifts in these interaction effects. Thus, these three program services do at least partially account for the complex changes in substance abuse ratings resulting from differences in the ACTO and IACT treatment programs. Figure 5 shows the fitted models for substance abuse ratings with (1) no mediators, (2) substance abuse contact, (3) transportation assistance, and (4) help with emotional problems to illustrate the size and direction of a sample of the mediated effects.

Discussion

Substantive Effects

Most clients improved over time on the dependent variables in all three treatment conditions. Much of that change occurred in the first 6–12 months of treatment. Also, regardless of treatment condition, all the service variables (i.e., the mediators) had direct positive effects on one or more of the outcome variables when controlling for the intervention effects. The pattern of results supports the conclusion of Brekke et al. (1997) that service intensity, service specificity, and continuous services are all important in improving the outcomes of persons with severe mental illness, but the best service mix is different for each outcome variable. For example, six of the eight service variables had a positive effect on consumer satisfaction. On the other hand, only transportation assistance and help with medication had a direct effect on psychiatric symptoms. Unfortunately, none of the service activity variables had a direct effect on substance abuse outcomes.

Below we discuss both the intervention effects and the mediation effects for each outcome variable.

Consumer Satisfaction

Clients in both ACTO and IACT reported greater satisfaction with their intervention than SC clients. There was no significant difference between ACTO and IACT, however. It is also important to note that the level of satisfaction is quite high early in treatment and then decreases over time, regardless of treatment condition. Consistent with previous research the superior outcomes on consumer satisfaction achieved by ACTO and IACT were mediated by program contacts, help with activities of daily living, and help with emotional problems (Morse et al. 1994). In addition, transportation assistance and substance abuse contacts were also mediators of the consumer satisfaction effect in this study.

Stable Housing

Clients in both ACTO and IACT increased their days in stable housing more than clients in SC. There was no significant difference on stable housing between ACTO and IACT. Improvement in stable housing is steady during the first half of the intervention and then begins to level off. At 30 months the distribution of scores on the stable housing variable has become fairly bi-modal. Fifty-four percent of the clients in the SC group, 42% of the clients in the ACTO group, and 31% of the clients in the IACT group report 0 days in stable housing during the previous month. At the other end of distribution, 34% of the SC group, 24% of the ACTO group, and 27% of the IACT group report stable housing for the entire month. From a practitioner’s point of view, it is clear that there are a significant number of clients whose permanent housing situation has not been helped by either ACT or SC.

The following variables mediated the ACTO & IACT vs. SC effect on stable housing: program contacts, substance abuse contacts, help with activities of daily living, transportation assistance, and, help with medication. These mediation results both support and conflict with previous research. Morse et al. (1994) had also found that assistance with activities of daily living mediated the stable housing effect. However, Morse et al. did not find mediation effects for program contacts, transportation assistance, or medication assistance. Morse et al. (1994) and Kenny et al. (2004) also reported that permanent housing was a mediator of the stable housing effect.

Psychiatric Symptoms and Substance Abuse

Although clients on average reduced both their psychiatric symptoms and substance abuse over time, there were no significant treatment main effects. Given that there were no main effects of treatment, there were no mediation effects using the more restrictive definition proposed by Baron and Kenny (1986). When considering change in these two variables however, individuals in different treatment groups experienced different trajectories. As such, several services accounted for these differential change patterns (see Table 4).

Treatment Fidelity and Research Design Issues

Several factors limit the generalizability of our study. Like most treatment outcome studies, our interventions were confounded by agency and staff effects, i.e., different staff and agencies were used in the three treatment conditions. It simply was not feasible to use the same treatment staff across all conditions. Similarly, because this was not a multi-site study, we did not have the luxury of replicating our design across multiple sites. As noted previously, the substance abuse treatment components of the IACT intervention were not implemented as completely as designed. Moreover, the ACTO intervention did provide some direct substance abuse treatment later in the project (treatment diffusion), which may have reduced the differences in the two interventions (Morse et al. 2006). These treatment implementation problems may account for the lack of differences on the substance abuse outcomes, but treatment implementation problems cannot explain the failure of the ACTO and IACT conditions to produce better outcomes on psychiatric symptoms than SC. These results are consistent with prior research (Bond et al. 2001; Essock et al. 2006). ACT programs, including integrated ACT, have often failed to demonstrate significant treatment effects on substance use and psychiatric symptoms. Although some of the failure of prior studies to find significant treatment effects in favor of ACT may be due to treatment fidelity problems, it is also clear that ACT is not a particularly strong intervention in affecting substance use and psychiatric symptoms. Nevertheless, more research is needed before concluding the IACT does not produce better substance abuse outcomes than ACT only or SC.

With respect to sample size concerns, we present Table 5, which shows the average sample size across eight of the study variables over time by condition. We chose to exclude consumer satisfaction, program contact, phone contacts, and substance abuse contacts because we lacked measurements at baseline for these variables. Further, we acknowledge that for these four variables, there should be more missing data in the SC condition by design than in the ACT conditions. Individuals could not report satisfaction if they did not have agency contacts during the reporting period. However, with respect to attrition, we found no evidence that individuals left the study at a different rate due to condition for any of the variables. That is, tests of independence (χ2) did not show significant signs of attrition over time by condition. The table shows that sample size remains rather robust for most of the time periods, with slight fluctuations across conditions. Attrition did not become evident until the last two time periods (i.e., 10–20% of individuals lost).

Table 5 Average sample size across eight variables over time
Full size table

Statistical Issues

In addition to the substantive questions addressed, the present study made several methodological contributions. First, we combined graphical methods of analyzing longitudinal data with statistical procedures. The models presented in this paper and indeed the change processes occurring in this investigation were complex. The graphical procedures greatly enhanced our interpretation of these change processes as well as facilitated the understanding of the mediation effects across time. Second, we utilized random coefficient modeling to model change in four outcomes associated with an investigation of treatment interventions along with a host of specific services hypothesized to mediate the treatment effects. In modeling change in this way, we were able to identify and assess the intervening effects of these mediators. To our knowledge, no one has published on the generalization of previous methods and practice to assessing mediation to a longitudinal context involving random coefficient modeling. Finally, we were able to address unique questions and identify relationships between the treatment interventions, the specific program services and the set of outcomes investigated that could not have been addressed in cross-sectional data, pre-test/post-test data, or within a repeated measures ANOVA framework. Therefore we make the following three conclusions based on our methodology:

  1. 1.

    Though main effects of the treatment intervention were not always present, we highlighted the need to assess differences in change process, which were frequently present. If two treatment interventions work with varying degrees of efficiency, there will be inevitable differences in outcomes associated with the change at some point in time. Choice of when to assess the main effects may be the determinant of discovering differences in treatment more so than if there are differences.

  2. 2.

    The treatment interventions were not equally effective at all time points. The change processes were not linear. For example, psychiatric symptoms improved more steadily and for a longer duration for individuals receiving IACT than for either ACTO or SC. With ACTO and SC, there were early and late periods of progress with some stagnancy in the middle months. A mirrored pattern emerged for substance abuse.

  3. 3.

    The recommended application of program services (i.e., the study’s mediators) is perhaps not as straightforward as previously thought. For example, we noted suppression effects over time in some instances (e.g., phone contacts with consumer satisfaction).

Before making any sweeping conclusions we highlight three concerns for practitioners and future researchers. First, we coded time using orthogonal polynomials at the risk of difficulty in interpretation. Without doing so, the analyses would have suffered from multicollinearity. We acknowledge the difficulty in the numeric interpretation of the orthogonally coded time metric, but recognize that the statistical analyses can be directly compared to the graphical analyses. The two (graphical and statistical) sets of analyses augmented one another. Second, we acknowledge that the coding of time directly affected the interpretation of any main effects. Throughout these analyses, the main effect was interpreted as differences at the study midpoint. Other researchers could have chosen to assess time differently and the interpretation of main effects would likewise have been different. We could not assess time in all possible forms. Further, our focus was on change due to treatment. Therefore, we believe our coding scheme is consistent with current research practice and useful in the current investigation. Finally, the mediators assessed in the current study were themselves time varying—proving somewhat elusive in the analyses. That is, as we investigated the effects of the treatments on the change in the outcomes over time we had to recognize that the mediators were continuously changing too. We believe our analytical strategy facilitated our understanding of these time varying mediators. However, we encourage researchers to replicate these change and mediation findings to ensure the generalizability and robustness of the present study’s findings.

Conclusion

In summary, both ACTO and IACT produced better outcomes on housing and consumer satisfaction than SC for clients with severe mental illness. However, neither ACTO nor IACT had superior outcomes to SC on psychiatric symptoms and substance abuse. In addition, this study also identified service activity variables that mediated treatment outcomes. Finally, this study found that the service activity variables (mediators) had direct and specific effects on the outcome variables after controlling for intervention and time effects.