Introduction

Homelessness continues to be a major social issue in North America. Recent estimates suggest that about 3.5 million adults will experience homelessness during any given year in the U.S.1,2 and that approximately 8,000 Canadians are affected by homelessness every night, a likely underestimate of the magnitude of the problem.3 Individuals with no fixed address experience worse health than the general population. They are more likely to suffer from chronic health conditions including tuberculosis, HIV/AIDS, and diabetes, are at greater risk for mortality, experience greater challenges accessing health care, and report more difficulties staying on medical regimens than their housed counterparts.3–5 Homeless individuals have high levels of mental illness with estimates ranging from 20% to 40% experiencing a severe and persistent psychiatric disability.6–12 Further complicating the health of the homeless is the fact that about 50–70% of homeless adults who report suffering from a mental illness also use or abuse substances.13–15 Overall, North American estimates suggest that 10–20% of the homeless population experience co-occurring mental health and substance use disorders,13,16 although some believe that the actual prevalence is far greater.17–19

Homelessness combined with concurrent mental health and substance use disorders (CDs) pose unique challenges in the treatment of adverse health conditions.20–22 Community-based health and social service organizations, because of their accessibility and range of available services, are often a preferred site for health and social needs addressed by homeless individuals.23 Given the complexities of the often chronic social and health problems facing homeless individuals, strong evidence is needed on why and how programs focused on CDs engage, retain, and successfully reduce the mental health and substance use problems of this marginalized population.

This review was undertaken as part of an academic–community partnership between research faculty and fellows in an inner city health research center and practitioners from urban community health and social organizations that serve low-income populations. A major motivating factor behind the partnership was to promote the uptake and implementation of research. A growing body of evidence suggests that use of research findings increases when stakeholders are involved early on in the research process to shape the direction and relevance of the research.24–30 In order to address a gap in knowledge regarding effectiveness of program components in community-based treatment approaches for CDs, in large part identified by our partners, we focused our review on evaluative research of CD programs for homeless adults based in urban community settings.

Treatment of concurrent mental health and substance use disorders have emerged only within the last two decades and researchers are starting to take stock of the effectiveness of existing treatments in reducing mental health and substance use problems, especially among housed populations.31 This literature, unfortunately, is characterized by poor evaluation designs, short-term follow-up, and heterogeneous interventions and populations. Several types of interventions lack sufficient evidence to determine their effectiveness such as individual counseling, family interventions, and intensive outpatient rehabilitation. Case management has been met with inconsistent findings such that conclusions about effectiveness cannot be made. Those interventions which show promise or are shown to be effective include group counseling for substance abuse, contingency management, and long-term residential programs, but the latter was based on weak evidence.31 However, the populations in these studies have largely been housed and the most effective means of treating CDs among the homeless in community settings has yet to be identified.

Systematic reviews generate strong evidence about treatments or interventions as they summarize a full body of work for particular treatment approaches and can yield useful information about average effect sizes across programs.32–34 While many systematic reviews emphasize the magnitude of program success, there is increasing focus in evidence syntheses on assessing what it is about interventions that make them work. To elucidate what program elements are linked to specific outcomes for treatment of CDs among homeless adults, we drew our methods from a realist systematic review approach. A realist review is particularly appropriate for understanding how complex programs work within particular contexts by exploring what “works for whom, in what circumstances, in what respects and how.”35 More specifically, realist reviews focus on understanding why programs work by identifying underlying theoretical mechanisms while exploring the successes and failures of a particular program. By incorporating a focus on program contexts and underlying mechanisms, realist reviews are particularly valuable for complex social interventions such as those concerned with community health.

The present investigation builds on existing research partnerships with five community-based organizations in Toronto, Canada. We take advantage of Pawson’s39 “realist review approach” to answer the following questions: (1) What community treatment approaches are beneficial for homeless adults experiencing concurrent mental health and substance use disorders? and (2) What is it about these programs that works (or doesn’t work)?

Method

Transdisciplinary Partnership

The Centre for Research on Inner City Health (CRICH) conducts multidisciplinary and transdisciplinary research to impact policies and practice that reduce health inequalities. CRICH involves policy and community partners early on and throughout their research as this approach has been shown to increase uptake of research findings.24–30 Through a community stakeholder assessment, CRICH identified five community health and service agenciesFootnote 1 that requested evidence in the area of services for concurrent mental health and substance use disorders in community settings among marginalized populations. The partnership was comprised of the community agencies and multidisciplinary CRICH faculty and research fellowsFootnote 2 who then worked together for a year to generate evidence on best practices to address mental health and substance use problems within community health and service organizations. This working group determined the scope of the review activities and research question and also worked together on all aspects of the project including dissemination of results.

Given the complexity of the issue in terms of treatment regimens and health and social issues facing marginalized populations, we sought to draw from methods of realist systematic review so that we could include numerous sources of data (e.g., academic and gray literature which was important to our community partners) and uncover the mechanisms by which successful programs work.

Research fellows and faculty conducted the practical steps of the research work, guided by monthly meetings with community partners. These meetings provided ample opportunity for community partners to contribute to the development of the research question, methodology, data interpretation, write-up, and overall direction of the project.

The project was approved by the research ethics board at St. Michael’s Hospital with which CRICH is affiliated.

Focusing the Research Question and Obtaining the Evidence

Table 1 lists the steps involved in our evidence synthesis. In keeping with the realist review approach, many of these steps were implemented in an iterative fashion.

Table 1 Key steps involved in our systematic synthesis
Full size table

Our team first identified the topic and scope of the review (step 1). Early on in our scan of the literature concerning persons experiencing mental health and substance use problems, we discovered studies about several groups of low-income populations and very heterogeneous studies and treatment approaches. Given our original interest in community-located treatment and our need to focus the population further, we decided to narrow the scope to only homeless populations as that was a common population served by all the community partners and also provided the greatest number of references. These decisions were heavily influenced by the evidence needs of our community partners who sought to improve the services they were providing to homeless persons with CDs.

We located scholarly peer-review literature on CDs by searching relevant medical and social science databases (Table 2). All English literature since 1980 with keywords related to interventions and mental health and substance use disorders was considered.

Table 2 Medical and social science databases
Full size table

This initial search resulted in 289 citations focused on treatment programs geared for adults experiencing mental health and substance use problems. Abstracts were independently read and screened for relevance. Abstracts not focused on community-based interventions, treatment, or service programs geared specifically for adults experiencing concurrent mental health and substance use disorders were excluded from the review. This primarily excluded literature on institutionalized populations, such as incarcerated or hospitalized persons and studies of veterans, as well as those that focused either on mental health or substance use disorders but not both.

Next, we read these articles in their entirety and reassessed them for relevance using the same criteria. As mentioned earlier, discussions with community partners during this phase of the research reshaped our scope to focus specifically on homeless adults experiencing concurrent mental health and substance use disorders. Fifty-seven articles focused on homeless individuals experiencing CDs. In these studies, homelessness was defined as people currently experiencing homelessness, those at risk of becoming homeless, formerly homeless (within the past few years), and underhoused people. The 17 peer-reviewed articles that were ultimately included in the review discuss ten community-based interventions geared specifically to CD clients experiencing homelessness, with evaluations assessing outcomes related to mental health or substance use disorders. In order to effectively address our research question of “what works in community-based treatment for concurrent disorders among homeless adults,” we included literature describing programs that were located in the community and incorporated a clear community integration or linkage component for clients. For example, we excluded literature on programs evaluating the effectiveness of highly structured residential treatment and therapeutic community programs, as these approaches did not actively integrate or link clients into the broader community through services, but into a smaller treatment community.

A few of the 17 articles focused on the same program; as a result, information was reviewed for each program, when appropriate. We then searched the literature and the Internet again for additional literature to ensure that we had complete information for each of the programs identified. While we made an active effort to supplement scholarly literature with gray literature, the bulk of the evidence used in this synthesis consisted of peer-reviewed research studies. We also e-mailed corresponding authors to learn more about how each program operated particularly in terms of program contexts and underlying intervention philosophies. Out of the 14 authors we contacted, ten responded to the questions we posed, including two who agreed to be interviewed over the phone. Overall, we included 38 sources of evidence on the ten programs. Figure 1 illustrates our literature search process.

FIGURE 1
figure 1

Evidence gathering.

Full size image

Program Outcomes

For the purposes of this realist review, we focused on research that presented evaluative program data on outcomes related to mental health and substance use disorders among homeless clients with CDs, such as number of psychiatric hospitalizations, psychiatric symptom severity, number of days of substance use, and measures of substance use severity. A number of the studies also explored program effects on other nonclinical outcomes such as housing stability, educational attainment, program retention, and client satisfaction. These outcomes were of particular interest to our community partners. However, not all of the studies included an examination of the same nonclinical outcomes, making it difficult to synthesize the evidence related to these outcomes. Consequently, we chose not to synthesize data on these program outcomes for this review. We instead focused on the relationship between program components and the improvement of mental health and substance use outcomes, as this relationship was most consistently evaluated across the programs we identified.

Quality Appraisal

Quality appraisal often focuses on the methodological rigor of the study.34 While we examined methodological rigor (e.g., sample size and statistical power; presence and strength of the comparison group; use of sound outcome measures; recruitment of the sample of homeless persons), one of our main interests in assessing quality was to determine whether the study yielded information of relevance to our research questions and was “fit for purpose.”34,36–38 Even studies that were determined to be of poorer methodological quality, if they provided “trustworthy nuggets of information to contribute to the overall synthesis,”39 were kept but given a lower weight. As such, we not only assessed the strength of the evaluation presented but also examined information provided on “what works” and why. Furthermore, because some studies examined a broad range of outcomes, we assessed the portions of the study that pertained to our research questions concerning community-based interventions for homeless populations for the outcomes of mental health and substance use problems.

We appraised the quality of evidence on a case-by-case basis during the literature search, extraction, and synthesis process. For research studies, we evaluated the strength of the design components when ample information was available (e.g., statistical power, sampling strategies, strength of the comparison groups and methods of evaluation, internal and external validity). For studies that did not indicate a statistically significant difference between treatment and control conditions, we assessed the level of statistical power available in the study. We did this by employing power calculations using information on reported differences between treatment and comparison groups and the sample size available for the analyses. To assess the rigor of the evaluation design, we assessed the presence of or appropriateness and comparability of the comparison groups as well as the recruitment strategies to determine whether large sources of bias could have been introduced through these two routes.

To address the quality of evidence available to determine what works and why, we considered whether the studies presented “thick” or “thin” descriptions of the program components and their mechanisms according to quality appraisal techniques developed by Popay et al.37 Thick program descriptions typically included a detailed description of the program and its suitability for the targeted population, a description of the factors that affect implementation, and a consideration of the reasons for anomalous results.36–38,40 In our case, we rated studies along the thick/thin continuum in terms of whether the studies provided information on how program components reduced mental health symptoms and/or substance use problems. Information on program components and their mechanisms was gleaned from the methods and discussion sections of scholarly literature and from companion articles describing these components of programs. Concerning assessment of the quality of evidence on program mechanisms, we tended to give greater weight to statements based on data that were collected and analyzed as part of the reviewed studies compared to statements made by authors about program workings that were supported by external literature. We used the quality appraisal to give more weight to the evidence provided by studies that had stronger methodological designs as well as provided thicker descriptions of program mechanisms. Table 3 lists information on the ten programs included in this review and details pertaining to quality appraisal.

Table 3 Program description and quality appraisal
Full size table

Data Abstraction and Synthesis

During the initial stages of data abstraction, we focused on extracting data related to context, paying close attention to the quality of the data and what aspects and under which circumstances programs were successful or not and on identifying any failures or harms associated with programs. By reading and rereading the articles and engaging in discussion about them as a team, we sought to identify those program models that worked and the reasons for program success or failure. This textual approach to synthesizing findings and telling a story based on multiple studies, which do not exclusively focus on the effectiveness of a specific intervention, is similar to narrative synthesis.35 Initially, we grouped programs according to common approaches and common models to synthesize the evidence (e.g., assertive community treatment [ACT], case management). However, within groups of treatment models, there was much heterogeneity in overall approaches. Moreover, almost all studies implemented a combination of strategies. Thus, it was virtually impossible to isolate the effects of a particular model on our outcomes of interest. Yet, through both the synthesis process and the weighing of evidence through quality appraisal, we began to identify program components that were often present in programs that were more successful in addressing mental health and substance use outcomes. Because the evidence was not overwhelmingly and specifically supporting a particular approach, we refer to these identified program components or elements as “promising program strategies” that have the potential to improve the mental health and substance use outcomes of CD clients, as they were most commonly present in programs with greater success. As mentioned earlier, these strategies were not necessarily stand-alone program components. Rather, they were often present within the same program and appeared to work together to impact upon client mental health.

As noted earlier, the realist review approach is about identifying program mechanisms or underlying program theories. From our review, several of the program elements were thought to or could be hypothesized to promote autonomy in the client and consequently improve mental health. The relationship of the program elements to each other, to client autonomy, and to mental health outcomes are depicted in Figure 2. It should be emphasized that this conceptual framework did not emerge directly from any one study nor did any of the studies cover all the strategies contained in the figure. Rather, the studies we reviewed contributed parts of the framework with some evidence for the linkages being stronger (thus the solid arrows) than other areas (dashed arrows). Below, we present these promising program strategies, starting with those that have the strongest evidence and concluding with strategies that have not been as rigorously examined.

FIGURE 2
figure 2

Conceptual framework of proposed processes facilitating improved mental health outcomes among homeless adults experiencing CDs.

Full size image

Findings

Promising Program Strategies

Emphasis on Client Choice in Treatment Decision-Making

Clients served by community-based treatment programs for homeless adults with CDs that incorporated client choice in treatment and program decision-making showed improved mental health outcomes. This element was a feature in six of the ten reviewed programs. Client choice was featured prominently in Choices Center and Pathways to Housing and, to a lesser extent, in Horizon Home and Emerson–Davis programs.41 The Choices Center and Pathways to Housing programs were specifically designed to enhance clients’ sense of autonomy or mastery and control and were organized to promote clients’ priorities and interests. Called “client choice” in some programs, this feature ensured that clients’ choices would be respected, regardless of whether these choices were consistent with treatment priorities. Clients were seen as autonomous individuals, experts of their own lives, who should make their own decisions.42,43 For instance, in the Pathways to Housing program, clients chose their apartments and neighborhoods, and provision of housing was not tied, in any way, to mental health or substance use treatment. In addition to making decisions about treatment, Choices Center clients had significant input into staffing decisions and program elements resulting in a program that was maximally tailored to their own needs. Data from Pathways to Housing showed that clients’ sense of mastery and their perceived level of choice were mediators in the causal pathway between housing and psychiatric symptoms. Thus, the beneficial effect of the Pathways to Housing program and the provision of independent housing “first” on psychiatric symptoms are attributed to the enhanced sense of mastery and control that clients experience as part of their treatment.44

Positive Interpersonal Relationships Between Clients and Providers

Several of the studies reviewed highlighted the quality of interpersonal relationships between clients and providers as a promising strategy related to improved outcomes among homeless individuals experiencing CDs. Calsyn et al.45 assessed reciprocal dynamics between clients and providers and showed in their study that trusting relationships, wherein clients and providers both feel comfortable with each other, were associated with client improvement in psychiatric symptoms. One-sided client–provider relationship dynamics, where respect and dignity were shown by service providers to their clients, was an important theme that emerged from the Pathways to Housing and Choices Centre programs. Respectfulness and dignity toward clients were seen to build trust and client self-esteem, which in turn increased clients’ ability to address their life needs. Peer support staff members who had lived experience with homelessness, substance use, or mental health disorders were found to be particularly effective in the Choices program, as their experiences “increased the empathic response in staff–member relationships beyond that found with more traditional staff patterns… [and] increased sensitivity to the struggles” that clients face.46 Clients noticed the difference, stating that peer support staff “really care” and that they “know how to relate to you as an individual.”43 The findings in the Pathways to Housing program further support this, as Padgett et al.47 found that “acts of kindness” by staff made a real difference for clients mental health and helped engage and retain clients in programs. “Acts of kindness” refer to staff who “treat participants with warmth and humanity and/or made extra efforts on their behalf,” for instance, by going out of their way to provide support for clients during difficult transitions or to connect clients to instrumental supports.47 These recollections stood out for clients because such moments were rare “amidst a norm of routinized ([and] sometimes dehumanizing) encounters with staff” [in other programs].47

Assertive Community Treatment Approaches

The provision of services from an ACT approach emerged in four of the reviewed studies as a promising practice for addressing client outcomes. ACT is an approach to addressing mental health and substance use whereby treatment is delivered by a multidisciplinary team of providers. ACT is assertive, outreach-focused, and highly available in the sense that services are delivered within the community and are available to clients 24 hours a day, 7 days a week.48 Furthermore, many of the ACT programs reviewed incorporated an “integrated ACT” approach in which mental health and substance use treatment are offered in the same program location as opposed to at different agencies48–50 and do not require clients to be abstinent in order to enter the program.50,51 Meisler et al.50 and Essock et al.49 found that an ACT approach was related to reductions in psychiatric hospitalizations (but not mental health symptoms) among CD clients over time. Choices Center and Pathways to Housing also used this approach and their programs had positive outcomes for mental health. Such improved outcomes may be facilitated by the fact that this approach is available at all hours of the day and night to clients in order to meet their needs. However, only one of these studies yielded modest evidence to support the effectiveness of such an approach at reducing substance use problems over time.50 There is, thus, minimal evidence indicating the utility of an ACT approach for improving substance use outcomes among homeless individuals experiencing CD.

Despite the fact that several of the ACT programs delivered clinical services in an integrated manner, only one group of studies, Morse et al.48, specifically examined the effect of “integrated ACT” on mental health and substance use outcomes.48 The authors found no difference between the integrated ACT vs. nonintegrated ACT for mental health or substance use outcomes. The authors argued that lack of fidelity in the implementation of the ACT program models may have led to treatment diffusion in which the experimental groups may have received similar types of programming. Based on the data available in this review, it was not possible to establish the superiority of integrated treatment services within an ACT framework for homeless individuals with CD.

Providing Housing

The provision of housing, along with other services, is an important element in the treatment of homeless persons with CDs; however, the evidence is difficult to synthesize as the definition of “housing provision” and the types of “housing” provided differed greatly among studies. One of the more successful programs, Pathways to Housing, offered independent, scatter-site housing without requirements to attend treatment or to be abstinent. This approach may serve to build client self-esteem and autonomy. Having a safe place to stay where one is not at imminent risk of physical danger or expulsion by authorities and where one is not required to live by the strict rules of a shelter allows for greater independence, greater choice, and an enhanced sense of autonomy. This is supported by the concept of “ontological security,” the “well-being that arises from a sense of constancy in one’s social and material environment which, in turn, provides a secure platform for identity development and self-actualization.”52–54 Dupuis and Thorns’ definition of ontological security includes the condition that home is where “people feel in control of their lives,”54,55 providing further support for housing and independent housing as a promising strategy. While the ACCESS program did not provide clients with independent housing directly, it is suggestive of the positive effect that independent housing can have, as clients who were independently housed showed more significant decreases in substance use than clients who were unstably housed or living in shelters or institutions. Horizon Home, which yielded reductions in psychiatric hospitalizations and higher abstinence rates, housed clients in a supervised group home.

Programs that provided other models of housing also reported positive effects on clients’ mental health and/or substance use. For example, both the Community Connections and Emerson–Davis programs followed a “continuum of housing” approach in which clients were first placed into supportive group living arrangements and gradually moved to less restrictive and more independent types of housing as their recovery progressed. As previously discussed, because these programs did not focus exclusively on provision of housing, it is not possible to attribute improvements to provision of housing alone.

The level of structure within housing programs can influence substance use and/or mental health outcomes. For instance, the program discussed by Burnam et al.56 placed clients into monitored group homes, enforced strict rules, and demanded sobriety.56 The program evaluation found no support for this model in terms of reducing mental health and substance use problems. Burnam et al.56 speculated that their high demand environment and strict rules contributed to the lack of program success. However, none of the reviewed programs specifically examined the independent contribution or rules surrounding housing, so future research in this area is warranted.

Providing Supports for Instrumental Needs

The provision of supports beyond just mental health and substance use treatment, either directly or through referral, emerged as a promising strategy in three of the programs reviewed. The programs that emphasized meeting clients’ needs beyond mental health and substance use treatment tended to make positive gains in substance use and mental health, although there were no evaluative studies that tested the unique effect of this program component. For example, the Pathways to Housing program takes a holistic approach to caring for clients in that their services are not limited to housing support and mental health and substance use treatment, but rather include additional supports such as recreation rehabilitation, money management, community integration, and advocacy. Similarly, the Choices program aimed to meet all of the clients’ needs by providing comprehensive supports including social support, storage lockers, meals, and showers, in addition to clinical care and shelter housing. Finally, the Emerson–Davis Family Development Center Program also embraced a comprehensive treatment approach by addressing legal and family needs, in addition to providing clinical support for mental health and substance use issues. Programs that prioritize assistance with clients’ immediate needs (e.g., food, clothing, personal hygiene, social support) may free clients to work on needs that are less immediate, such as mental health or substance use.

Nonrestrictive Program Approaches

Nonrestrictive, flexible program policies also emerged as a promising strategy in a few of the reviewed programs. For example, in some programs, abstinence was not a requirement for program entry or retention.43,50,51 In the Pathways to Housing program, there is no predetermined end time to treatment. This program follows the “once a client, always a client” philosophy where treatment, housing, and support are offered unconditionally regardless of the clients’ situations. Such a flexible approach is respectful to clients needs and provides stability, consistency, and control as they progress through the program. In contrast, the community-linked residential program discussed by Burnam et al.56 consisted of a relatively short, 3-month intensive treatment period after which clients moved to less intense services. The authors reflected that the short-term nature of the intervention may have contributed, in part, to the lack of success since mental health and substance use problems are typically chronic issues requiring longer-term management and intervention.56

Discussion

In this review of studies on community-based interventions, we drew heavily from the realist review methodology. We learned about program mechanisms from both successful and nonsuccessful programs and by augmenting the evaluative evidence in the studies with information gleaned from “thick” program descriptions and communications with those who implemented and evaluated these programs.

We identified ten distinct community-based or community-linked programs serving homeless individuals experiencing CDs that employed a variety of approaches including ACT, provision of housing, integrated mental health and substance use treatment, and a holistic approach through which many of the clients’ life needs were supported. Most programs delivered a combination of program strategies or took different approaches to the same strategies (e.g., different models of housing provision in Pathways to Housing vs. Community Connections and different degrees of fidelity to the ACT model between programs). There was significant heterogeneity across studies in program approaches and measurement as well as methodological weaknesses in some studies, such as no or weak comparison groups and high attrition over time. These limitations hampered our ability to generate strong evidence describing best practices in the treatment of CDs for homeless individuals. These methodological limitations reflect, in large part, the challenges of studying and following up hard to reach populations such as homeless individuals.

A key component of the realist review is building and uncovering a theory of how the program works—the “mechanism” of the program.39 Through a review of the available evaluative and descriptive evidence and despite the limitations noted above, we identified six promising program strategies for the improvement of CDs. These promising strategies included an emphasis on client choice in treatment decision-making, positive interpersonal relationships between client and provider, ACT approaches, providing independent housing along with other services, providing services beyond mental health and substance use treatment, and nonrestrictive program approaches. Because many of the programs incorporated more than one promising strategy, often delivered in different combinations between programs, we cannot identify one promising strategy as being better than the others.

These promising program strategies showed greater impact upon improvements in mental health outcomes than for substance use problems. Our synthesis suggested pathways by which the six program strategies may exert their effects on improvements in mental health, as presented in Figure 2. In the figure, those pathways that were supported by stronger evidence are indicated by solid arrows while the dashed lines indicate weaker evidence. As suggested by the figure, some strategies were found to have a direct impact on improving mental health such as providing clients with independent housing and positive client–provider relationships characterized by trust. These strategies as well as others such as client choice in program decision-making or providing supports for instrumental needs may work to reduce mental health symptoms by promoting client autonomy. Autonomy or the ability to be self-directed has been strongly linked to improvements in problematic health and other behaviors such as successful diabetes management, long-term weight loss, long-term smoking cessation, positive behavior change among troubled and troubling youth, and even nonhealth issues such as law school performance and satisfaction.57–60 Many of the studies identifying autonomy as a central factor in motivation for and maintenance of behavior change noted that goals which are self-directed vs. goals that are imposed or controlled by external sources (e.g., health care providers) are not only more likely to be attained but that goals imposed from external sources may negatively impact well-being.59 Thus, programs and program strategies that support autonomy and self-determination in treatment and use of services will likely lead to longer-term positive health changes compared to programs that are fixed and are restrictive in content and in modes of administration.61 Figure 2 illustrates possible pathways by which these promising strategies may directly impact mental well-being and also build clients’ sense of autonomy and control over their lives and ultimately lead to mental health improvements. Client autonomy appears to be a promising and perhaps essential component of community-based programs and should not only be considered and studied when initiating new projects, but also more rigorously evaluated in those programs where this model of care currently exists. It would be particularly important to know whether and how client autonomy is best promoted for clients with different types of mental health problems and illnesses.

The studies we reviewed provided weak evidence concerning what works for improving substance use outcomes. A lack of improvement in substance use problems has been reported in prior reviews of research on the effectiveness of case management programs for clients with CDs.31 The lack of improvement in substance use problems that we observed across some studies we reviewed may have been due, in part, to a narrow focus on abstinence as the outcome (e.g.,41). Clients, as well as providers, may experience or observe reduction in or greater control of substance use, but such improvements would have been missed when abstinence measures are used. Future studies might consider adopting measures that incorporate a harm reduction perspective on substance use which would capture more nuanced changes in substance use.

A limitation of our review was the quality of evidence presented in the programs that we reviewed, suggesting that too few evaluations had strong internal validity or yielded “thick” descriptions of program elements. The research design of some of the evaluation studies had considerable methodological limitations. For example, a number of the program evaluation studies did not have a comparison group and/or were statistically underpowered to detect the differences yielded by the interventions. We attempted to gain further information about the program theories or mechanisms as well as to obtain insights into why programs were or were not successful by obtaining qualitative program description information from supplementary literature or by communicating with the corresponding and lead authors of the studies we reviewed. Only a few studies undertook qualitative investigations into program implementation and program workings and we were successful in obtaining such information in too few instances. Thus, many of our studies ended up being thin on descriptions of what works, for whom and why.

An additional limitation is that we focused on only two outcomes, mental health and substance use. However, given the complex health and social needs of homeless clients experiencing CD, a number of other outcomes, some of which were examined in the studies reviewed, also deserve further systematic study. For example, nonclinical client outcomes such as short-term and long-term housing stability, short-term and long-term employment and educational attainment, and program-related outcomes such as retention and client satisfaction all deserve further research attention in order to adequately identify the best interventions to meet the needs of clients. Furthermore, our community partners specifically identified a need for research attention to the effects of cultural sensitivity, harm reduction programming, and discharge planning on outcomes.

We sought to implement a method of synthesizing the research evidence that is appropriate for complex interventions and that the identification of best or promising practices for treating homeless clients in community-based or community-linked settings who have concurrent mental health and substance use problems. We drew heavily from the realist review approach promoted by Pawson.35,39,62,63 We had too little guidance on how to actually implement a realist review as there are few examples, and the examples provide sparse information on their methods. For example, Connelly conducted a realist systematic review of obesity prevention programs but provided no detailed information about how their synthesis addressed the issue of uncovering program mechanisms.64 Greenhalgh and colleagues65 synthesized evidence from an existing systematic review to identify potentially successful program mechanisms but did not supplement with descriptive studies or gray literature which, as the authors note, could have enhanced their review. It is still in the early stages in terms of the implementation of realist systematic reviews and our synthesis contributes to the small but growing set of reviews following a realist approach. Because we were interested in employing the best method for synthesizing the evidence on this complex intervention, we were guided not only by the published and available examples of realist reviews64–66 and texts on realist evaluation and review approaches,35,62–66 but also from literature concerning review approaches appropriate for complex interventions.34,37,67,68

Contextual factors, including the characteristics and circumstances of individuals as well as the cultural climate in which programs and systems are operating, are considered important components of describing theories in a realist review and reviews of complex programs.37,39,65 Notwithstanding our intentions to incorporate contextual factors into the synthesis process, much of the literature was lacking in thick descriptions of population characteristics such as ethnicity, as well as the setting or cultural environment in which programs were being implemented, thus presenting further challenges around quality assessment. Given the complexity of CDs, particularly among marginalized populations, it is crucial that future research addresses the context and cultural setting in which programs are operating, so that such complex needs can be effectively met.

A strength of our review lies in the continuous involvement of community-based agencies in various stages of the research process. While experience and expertise from the community partners was key in the integration of knowledge, the evidence gathering process, as well as the extraction and synthesis phases, we were particularly motivated to retain involvement of these key stakeholders to maximize the chances that the evidence will be used to change or inform current practice or policy.24–30