The gap between science and practice in mental health has been well established, through Federal reports (e.g., President’s Freedom Commission on Mental Health 2003; National Advisory Mental Health Council 1999, 2006), research commentaries and systematic reviews (e.g., Balas and Boren 2000; Fixsen et al. 2005; Weisz et al. 2005), and calls for research (NIH 2006). However, we still see relatively slow development of the science of implementation, the knowledge base about the fit between scientifically-tested interventions and the service systems in which they can be used.

To respond to these challenges, the National Institute of Mental Health organized a workshop on October 28th and 29th, 2004, in Bethesda, MD, entitled, “Advancing the Science of Implementation: Improving the Fit Between Mental Health Intervention Development and Service Systems.” The purpose of the workshop was to (1) identify organizational constructs, measures, and theories relevant to implementation research; (2) discuss the likely fit of mental interventions with the service systems; (3) build integrative research designs around the implementation of evidence-based treatments into intervention trials; and (4) build capacity in the field around multi-level models of implementation.

Over the 2-day workshop, a multi-disciplinary group of key researchers in the fields of interventions research, organizational research, and research methodology, along with clinicians and “real-world implementers” of evidence-based practices to discuss the development of the science underlying the implementation of evidence-based practices and the integration of key constructs across different types of interventions. A total of 35 people (listed in the appendix) participated in the meeting, which defined implementation as “the use of strategies to adopt and integrate evidence-based health interventions and change practice patterns within specific settings” (NIH 2006).

The participants were chosen and discussions were organized according to the principles of “engaged scholarship,” in which academic researchers from multiple disciplines (e.g., organizational behavior, psychiatry, psychology, social work, public health) join clinicians and real-world implementers to “co-produce knowledge that can advance theory as well as practice in a given domain” (Van de Ven and Johnson 2005). Rather than viewing the gap between science and practice as a knowledge transfer problem, “engaged scholarship” recasts the gap as a knowledge production problem—the knowledge developed by science may be incompatible for use in practice and vice versa (Van de Ven and Johnson 2005).

This may be particularly relevant for implementation research, where the challenge is to assess the fit between science and practice. For example, as clinical trials are developed, researchers must make choices in design, such as selection of inclusion and exclusion criteria, site selection, choice of statistical approaches, approach to coping with missing data, time points for patient assessment, and many other decisions that are guided by the scientist’s best judgments (Fitzgerald et al. 1999; Huby and Fairhurst 1998; Westen and Bradley 2005; Westen et al. 2004). As a result, those seeking to apply the results of RCTs to clinical practice through implementation may need to balance an awareness of the choices made in the construction of the trials and the rationales behind them with the choices made by clinicians in delivering care to diverse groups of patients. Developing studies within research teams equally representing the experience and knowledge of science and practice may pave the way for knowledge on the implementation of interventions to improve uptake in diverse mental health service systems.

The 2-day meeting included a series of stimulus presentations on research methods and design applicable to implementation studies. Participants were then divided into three groups, each including intervention scientists, services researchers, intervention implementers, methodologists, and clinicians with experience working with a common intervention. Each team was tasked with three specific challenges. First, teams were asked to identify constructs (e.g., organizational structure, culture, clinical training) that were likely to influence implementation of an effective intervention. Second, teams were asked to use assess how these constructs might be included within a research design and whether measures exist to test their influence. Third, teams were asked to identify methods through which research capacity in the field of implementation science could be built.

Teams were organized to address the challenge of implementing three different classes of clinical interventions (medication management for depression, trauma-focused cognitive-behavioral therapy, and assertive community treatment). The three types of interventions were chosen to represent the breadth of clinical practice, including medication and psychosocial interventions delivered by an individual clinician and a team-delivered psychosocial intervention. This structure was based on the assumption that facilitating individual practice change may vary from influencing clinical team or organizational change.

Breakout Session One: Evidence About the Implementation of Interventions: From Intervention Research to Implementation

The first breakout session was designed to use the experience of each of the groups to reflect on how intervention science and implementation practice could inform implementation research, highlighting relevant research questions for implementation and key constructs suitable for study. Discussion was organized around four key questions: What have we learned from efficacy/effectiveness research about key constructs that impact the implementation of interventions? What have we learned from the experience of real-world implementation about key constructs that impact the implementation of interventions? Are there key organizational constructs that may relate to the adaptation of interventions during implementation? How can available theories and measures enable the inclusion of key constructs related to implementation? While the questions were intended to stimulate discussion, each group was encouraged by its facilitator to depart from the structure as needed to enable additional topics to be introduced.

Group One

The first group represented research and field expertise related to the implementation of team-delivered interventions. Group members provided expertise in the design, testing, and implementation and use of Assertive Community Treatment. This group chose to focus on the articulation of multiple levels of influence on the implementation of team-delivered interventions, describing constructs at the client, supervisor, administrator, and state levels within a single framework. The group argued that all levels were interdependent, and a strong collaboration among mental health system stakeholders was essential to successful implementation. At each level of the system, the group identified key questions to include in an implementation research study, as shown in Table 1.

Table 1 Key research questions at each level of implementation
Full size table

Group Two

The second group focused on lessons learned from the implementation of trauma-focused cognitive behavioral therapy for children and adolescents. Though the experience of the group was specific to a single intervention, the group chose to enumerate the challenges of implementing general individual-delivered psychosocial interventions. Like the first group, discussion centered on key constructs relevant at multiple levels of the system.

At the provider level, this group identified the degree of compatibility (of the intervention) with the “treatment as usual” approach, along with the degree to which implementation of the new intervention would disrupt practice routines. The degree of comfort that the clinician had in accepting new ideas was also mentioned, along with the degree to which providers were engaged in the implementation process. Finally, the group noted that the provider’s social network can influence implementation either positively or negatively.

At the organization level, support for new ideas (and opportunities to use those ideas) within a clinic setting was identified as a key construct. The group posited that organizational structure must support change as “routine,” and that opportunities for supervision and training need to be built into the organizational structure. In addition, organizations can provide incentives for employees to change and recognize the significance of organizational culture and climate (Glisson 2002). Finally, the group discussed important infrastructure needs, including information systems, financial incentives and leadership.

The Competing Values Framework, developed by Quinn and colleagues (Quinn et al. 1991) was highlighted as a possible measure that might be relevant to the measurement of multiple constructs. However, it was noted that the scale is filled out at the individual level, which might pose challenges for aggregation up to organizational level and beyond.

Group Three

The third group, drawing from experience related to medication management interventions for depression, discussed a necessary separation between key clinical and non-clinical outcomes for implementation. The group felt that the clinical outcomes were already fairly well addressed by many of the intervention studies, but that non-clinical outcomes were hard to identify, and perhaps even harder to measure.

Discussants also focused on the need for greater specificity in intervention research, to avoid the “kitchen sink” approach, in which multiple components of an intervention are never tested independently, which leaves clinicians to wonder whether there are certain components that may be harder to implement and are not necessarily as beneficial to the client.

Finally, the group focused on the need to think of implementation as a second, non-clinical intervention. This intervention targets change of clinical behavior directly or system behavior that in turn influences clinical behavior. This “implementation” intervention must exist on top of the clinical intervention. Discussion continued about whether research could focus on both types of interventions within the same study, or whether the implementation research must be predicated on the assumption that the clinical intervention questions have all been answered.

Organizational Measurement

Within each of the group sessions, participants challenged whether suitable measures for implementation-relevant constructs already existed, or whether measure development would need to precede implementation research. Before group discussion proceeded further, several stimulus presentations were given to outline the current status of organizational measures relevant to implementation research.

One presentation was organized around a review of measures from multiple industries that might measure different features related to implementing mental health interventions. The literature review included studies from 1998 to 2004 with search terms of “innovation,” “diffusion,” “implementation,” and “culture.” The results were then narrowed to eliminate innovation done “in house,” as the articles lacked information about how to bring in a new intervention from elsewhere. The majority of studies identified were based on adopting new technologies (Kimberly and Cook 2007).

Briefly, the presenter noted that it may be unwise to take variables one at a time, instead researchers should work with clusters of variables. While taking measures from other industries “off the shelf” may be unwise, several clear constructs are worth considering. These include: system readiness of change, organizational culture, distribution of control, behavior of leaders including top-level management, incentives for change, competing demands within an organization, and the role of champions within organizations.

In closing, the presenter cautioned against assumptions that interventions are static once implemented. In reality, many interventions morph as they are diffused, a concept some theorists refer to as “re-invention” (e.g., Greenhalgh et al. 2004; Gustafson et al. 2003).

The importance of organizational measures within implementation was a dominant of the meeting. As a result, a paper based on this presentation was commissioned. The article, included in this volume (Kimberly and Cook 2007) further develops the review presented at the meeting and adds an analysis of the utility of identified measures to the specific context of mental health services.

Social Network Analysis

The next presentation was a brief introduction to the use of social network analysis to look at relationships within and between organizations. This was particularly important, given the many discussions about leadership, power structure, communication and training and their relevance to implementation. The contribution of a social network perspective is that individuals are not seen in isolation, but in the context of the relationships in which they are embedded. “Actors” within a network can be individuals, groups or organizations. “Ties” in a network include communication, responsibility, trust, advice, workflow, and resource allocation, among others (Scott 2000).

The presenter briefly reviewed a number of key features of social networks, including cohesion, which measures how individual actors are affected by direct ties, and structural equivalence, which focuses on how actors’ positions on a network may be similar to others, and one’s likelihood to adopt an innovation can be influenced by others with similar positions within the network. Triads, which represent actors that are not only linked to one another, but also through a third actor that may reinforce decisions by actors to adopt a new innovation, were also described (Wasserman and Faust 1995).

The presentation stimulated several research questions related to implementation at multiple levels of a service system. At the individual level, do network characteristics of individuals impact their likelihood of implementation? How do social network ties facilitate implementation? How do network characteristics of first adopters impact the diffusion process?

At the team level, how does the existing network structure of a team impact the implementation process? How does a team (e.g., Assertive Community Treatment team) network facilitate its ability to deliver services to patients? Finally at the organizational level, how do ties among organizations influence the implementation process? How can network channels be used to facilitate the adoption process? How does employee resistance to adoption travel through network channels? These questions were assimilated into several of the group discussions as they discussed how to incorporate implementation issues into research designs.

Breakout Session Two: Incorporating Implementation Issues into a Research Design

The second breakout session challenged each group to determine how the constructs previously raised as relevant to implementation could be incorporated into a research design. Specifically, groups were asked to consider whether efficacy and effectiveness studies of mental health interventions could be expanded to incorporate implementation questions. Key questions for discussion included: What measures are candidates to augment current efficacy and effectiveness designs? Does consideration of system-level constructs impact study design? What research questions related to implementation can be asked within existing trials and what questions require separate research studies?

Moving from Implementation Constructs to Implementation Research

The first two groups chose to focus on how a research team could design a study to disseminate interventions. The primary focus of the discussion was on the need for research teams to align the research to the multiple stakeholders impacting (and in turn impacted by) implementation of the intervention. First, the research team would need to define and characterize the intervention. Next, the study team would have to think about all of the stakeholders that might impact the intervention, including clients/consumers, families, providers, teams, supervisors, administrators, organizations, and state agencies.

Participants identified significant factors that a research team would need to consider for each of the stakeholder groups. They discussed the importance of knowing the current status of each stakeholder prior to the intervention, and how would the intervention impact each stakeholder. They also described several major principles of a research approach for implementation, which include qualitative and quantitative methods, multiple time points, and multiple viewpoints. In addition, participants identified specific measures needed to perform implementation research. These measures include sensitivity to change, sensitivity of the intervention to the context, and a method of determining the nature of the interaction between the intervention and the system in which it is introduced.

Finally, the groups listed several areas for implementation research:

  • Observational studies could be done at each level of the system (e.g., client, provider, organization) with the use of existing qualitative measures and the development of new measures to gather empirical data about existing implementation initiatives.

  • Research on providers could focus on what factors (e.g., financial, psychological, social network) affect clinician adoption of interventions.

  • Research on organizations could identify the influence of structure, networks, culture, and climate on implementation

  • Research studies could focus on which dimensions of implementation are malleable and which components of a treatment intervention are flexible, something which is still unknown for many interventions.

  • Research studies could determine which factors distinguish adopters of an intervention from non-adopters.

  • Research could also discover to what extent organizational features become part of an intervention. The group noted that a good example of this is an information system. If use of an intervention becomes aided by an information system to the extent that the information system becomes indispensable, might that suggest that the information system becomes part of the intervention?

  • Research could also target the measurement of contextual variables, link fidelity of implementation to outcomes, study natural experiments of implementation, and assess strategies to establish an intervention in a practice setting versus maintaining an intervention over time.

Expanding Intervention Studies to Include Implementation Constructs: Research Development

The third group focused their discussion on the potential expansion of intervention research to include implementation research questions. They began with a standard equation for Efficacy,

$$ {\text{E (efficacy) = T (the treatment effect) + e (some error term)}} $$

Could a broad effectiveness question be represented by the equation

$$ \begin{aligned}{} {\text{E (real-world effectiveness) = }} & {\text{T (treatment effect) + I (Implementation) + T*I }} \\ & {\text{(interaction term) + e}} \\ \end{aligned} $$

where “I” would be some operationalization of implementation, adoption, fidelity, or perhaps “reach” of an intervention. The group then discussed the specific questions that could identify the predictors of “I”.

  • Will physician training of the intervention (e.g., medication algorithm) influence “I”? How can training be most effective?

  • Can the care coordinator influence physician fidelity to the medication algorithm? What is the role of the care coordinator?

  • Will resources (e.g., money, staff levels, information technology, formularies) influence “I”?

  • Will physician willingness to change (e.g., motivation, behavior adherence) influence “I”?

  • Will levels of local adaptation influence “I”?

Next, the discussion moved to that of split–plot designs, where randomization by multiple training conditions or different levels of care coordination, for example, might be able to account for a large part of “I”. The group also discussed the feasibility of testing out some of the constructs, either between different clinics of an intervention study, or within clinics to test out differing conditions of a particular construct. As it became apparent that context (C) was crucial and perhaps separate from I, the equation was modified to become

$$ {\text{E = T + I + C + e + Interaction Terms (TI + CI + TC + TIC)}} $$

The group noted that the same equation could reflect changes over time, and one possible method would be to view E, T, I, and C at various time points. This revised effectiveness model would allow researchers to measure the degree to which effectiveness of a treatment is dependent upon the ability of that treatment to be effectively implemented within a particular setting (I) and of specific characteristics of the context that affect delivery of the treatment (e.g., staff knowledge, skills, and abilities; financing of the treatment; resources). Threats to effectiveness could come not only from the direct treatment effect but from these other terms as well as their interactions. New studies could include these terms, rather than attempting to neutralize their influence. Thus, more information could be available from these trials around effective ways to implement a treatment and how elements of context affect treatment outcomes.

Increasing the Fit Between Interventions and Service Settings

Following the second breakout session, researchers from the UCLA/RAND services research center presented how services researchers had been able to increase the relevance of research on interventions by working within clinical settings and partnering with practice organizations to conduct research. Presenters described work completed in the previous few years which intended to increase community and healthcare commitment to improving quality of mental health care, address multiple stakeholder perspectives, develop priorities through a partnership approach, and to document the process and impact of quality improvement efforts. An extensive account of the lessons from the community partnership is available in another paper (Jones and Wells 2007).

The presenters then described current work involved in measuring organizational factors promoting adoption and sustainability of family therapy for child conduct disorder and adaptation of cognitive behavioral therapy for trauma in schools. Finally, the presenters set forth a framework for dissemination in health services research that would show how research and evaluation must match each stage of the diffusion process, from the determination of organizational readiness, through to adoption, implementation and sustainability of the intervention. Given the tremendous need of the field to have theoretical frameworks to underlie dissemination and implementation studies, the presenters were asked to expand their talk into a more extensive paper, which is included in this volume (Mendel et al. 2007).

Lessons Learned and Next Steps

The workshop concluded with a broad group discussion focused on what Federal agencies and researchers in the field could do to build capacity for conducting research on implementation, which is still a developing field with relatively few people engaged in studies. First, a review of current opportunities were presented, including conference grants, training institutes, and exploratory developmental research on implementation. The group discussed potential actions for funding agencies, including supplements to existing studies to conduct implementation research, coordination of review committees to ensure appropriate expertise in implementation, and collaboration across NIH Institutes to complement new mechanisms. Finally, methods of developing field capacity to conduct implementation research were discussed, including the possibility of involving young scholar groups studying organizational behavior in health care, and the provision of experiential training for graduate students and the potential organization of a dissemination and implementation research conference.

In the several years that passed since the workshop, the suggestions of next steps from the group were embraced both within the National Institutes of Health and across the field. A revised program announcement on dissemination and implementation research now has the participation of eight NIH Institutes, with separate announcements for small grants (R03), exploratory and developmental grants (R21) and large-scale research grants (R01) (NIH 2006). The program announcements highlight the continuing gaps in methods, measures, and theoretical models for implementation, and offers opportunities for researchers to study the very things that participants stressed as crucial to advancing the science of implementation. Drawing heavily from the discussions within the meeting, the current program announcements highlight the key research issues (listed below) that the field is encouraged to focus on:

  1. 1.

    Analysis of factors influencing the creation, packaging, transmission and reception of valid health research knowledge, ranging from psychological and socio-cultural factors affecting individual practitioners, consumers, primary caregivers and other stakeholder groups to investigations addressing large service delivery systems and funding sources.

  2. 2.

    Experimental studies to test the effectiveness of individual and systemic dissemination strategies, focusing on outcomes related to the direct outcomes of the strategies (e.g., acquisition of new knowledge, maintenance of knowledge, attitudes about the dissemination strategies, use of knowledge in practice decision-making).

  3. 3.

    Studies of systemic interventions to impact organizational structure, climate, culture, and processes to enable dissemination and implementation of clinical information and effective clinical interventions.

  4. 4.

    Studies of efforts to implement prevention, early detection, and diagnostic interventions, as well as treatments or clinical procedures of demonstrated efficacy into existing care systems to measure the extent to which such procedures are utilized, and adhered to, by providers and consumers.

  5. 5.

    Studies of the capacity of specific care delivery settings (primary care, schools, community health settings, etc.) to incorporate dissemination or implementation efforts within current organizational forms.

  6. 6.

    Studies that focus on the development and testing of theoretical models for dissemination and implementation processes.

  7. 7.

    Studies on the fidelity of implementation efforts, including the identification of components of implementation that will enable fidelity to be assessed meaningfully.

  8. 8.

    Development of outcome measures and suitable methodologies for dissemination and implementation approaches that accurately assess the success of an approach to move evidence into practice (i.e., not just clinical outcomes).

  9. 9.

    Longitudinal and follow-up studies on the factors that contribute to the sustainability of research-based improvements in public health and clinical practice.

  10. 10.

    Studies testing the utility of alternative dissemination strategies for service delivery systems targeting rural, minority, and/or other underserved populations.

  11. 11.

    Studies on how target audiences are defined, and how evidence is packaged for specific target audiences.

In the interim, the research field has also made tremendous progress. As evidenced by the recent National Advisory Mental Health Council report, The Road Ahead, we have well-tested interventions that demonstrate clinical benefit but are underused in real-world practice, and a growing interest in implementation science among the services research community (NAMHC 2006). We also have current studies ongoing that test systemic interventions targeted at the easing the implementation process for schools, clinics, and communities (Mendel et al. 2007; Glisson 2002). Many of these studies are described in a separate work (Chambers 2007). However, more work is needed to ensure that the interventions we spend so much effort developing and testing get to the patients and providers who can benefit from them. While the 2004 meeting is a helpful primer to develop implementation science and ongoing research studies (including the papers published within this special section) and the recent NIH program announcements can further advance the field, the continued generation of constructs, theories, additional measures and models are still needed to substantially diminish the gap between science and practice.