In 2006, Ontario’s Policy Framework for Child and Youth Mental Health identified a need to improve the dissemination of what works in mental health practice and put it into practice.1 While an encouraging policy direction, the operationalization of evidence-based practice (EBP) implementation in child and youth mental health systems requires further consideration and the provision of support at the organizational and workforce levels. Thus far, the field of implementation science has identified that the implementation of effective and efficient behavioral health treatment programs involves a number of factors beyond that of consulting the research evidence and committing to the adoption of a particular practice.2–4 The range of factors identified as important in EBP implementation is captured in several frameworks and has been synthesized in a Consolidated Framework for Implementation Research (CFIR).5 This framework provides an overarching typology or meta-theory of relevant implementation constructs: intervention characteristics, outer setting, inner setting, characteristics of the individuals involved, and the process of implementation.5 The CFIR identifies constructs for which evidence supports their influence (positively or negatively, as specified) on implementation but does not specify the interactions between these constructs.

We know little of how the process of implementation unfolds in real-world settings, and thus stand to learn a great deal through practice-based implementation studies done in partnership with community-based providers. Here, we have selected to explore the process through an exploratory case study.6 Given the paucity of literature examining implementation processes within the pediatric behavioral healthcare field, the present study used an exploratory framework to follow a change initiative involving the implementation of multiple EBPs over a four-year period in a large pediatric behavioral health service provider organization. In this paper, we report on staff experiences of the change process, which was captured through annual questionnaires and qualitative interviews conducted in year four of the implementation process. These exploratory data map nicely on to the CFIR, capturing several aspects of the inner setting—organizational leadership, resources, and organizational culture—as well as characteristics of the individual, including practitioner willingness to change, their perceptions of the change process and knowledge of evidence-based practice, across four years of a clinical transformation change process.5 Other aspects of this change process, including an overview, process requirements, and working group challenges are reported elsewhere.7 , 8

Organizational Culture and Climate

There is little systematic research that identifies how constructs related to the inner setting influence EBP implementation or interact with other constructs identified in the CFIR to affect EBP implementation. We know that organizational culture and organizational climate are related but distinct constructs that have bearing on EBP implementation. Organizational culture reflects the norms and values of an organization and embodies the degree to which employees perceive an honest, trusting, and fair workplace.9 , 10 Implicit within these values and norms is a sense of trust and transparency, and a belief that these values and their assumed behaviors are shared amongst the entire group for which they were created. Trust plays an important role in the development and maintenance of organizational culture and, for this reason, is a predictor of cooperative behavior, organizational commitment and employee loyalty; all of which contribute to retaining and attracting skilled staff during change processes.10 , 11 Trust, and its more macro conceptualization—organizational culture—set the tone for the organization. A negative tone can undermine the policies and practices of a workplace and its change initiatives.

Organizational climate has been described as a localized and more tangible manifestation of the largely intangible, overarching culture.9 Climate can vary across teams or units and is typically less stable over time as compared to culture.5 , 11 Damschroder et al.5 provide a nice summary of the sub-constructs related to organizational climate, including absorptive capacity for change, shared receptivity of involved individuals to an intervention12 and the extent to which use of an intervention will be rewarded, supported, and expected within their organization.13 They further identify six sub-constructs that contribute to a positive implementation climate for an intervention: tension for change, compatibility, relative priority, organizational incentives and rewards, goals and feedback, and learning climate.5 Organizations having a positive climate and that engage in practice change while supporting their staff appropriately throughout this process, can look forward to decreased turnover rates and increased staff morale throughout and following the change process.11 , 14 The rationale for this particular outcome is that flexible structures and supportive climates are more conducive to limiting staff anxiety related to the change process.15 Furthermore, employees who perceive their organizational climate as supportive, trusting and transparent are more likely to possess positive attitudes towards organizational change and a willingness to participate in the change agenda.14 , 15

Organizational Readiness, Leadership, and Resources

Another feature of the inner setting is the organization’s commitment or readiness to change,16 and this encapsulates leadership engagement, availability of resources to support the change, and access to information and knowledge about the change process.5 Leadership encompasses the notions of commitment, involvement, and accountability for the implementation,17 , 18 and pertains to leaders at any level of the organization including executive level, middle management, front-line supervisors, and team leaders who have a direct or indirect influence on the implementation.17 , 19 Leadership that is effective in providing necessary resources is important for engaging in effective and sustainable organizational change.19 An abundance of literature highlights the importance of effective leadership in organizational change, particularly related to job satisfaction throughout change process, as well as styles of leadership that are more or less likely to facilitate an effective change process.20 Among leadership styles, the literature focuses on three—instrumental, transactional, and transformational—with the latter garnering the most support as the gold standard for which to strive. Transformational leaders are those that motivate their staff to participate in workplace tasks above what is expected while simultaneously supporting their staff in their work.21 , 22 Transformational leaders are concerned with the long-term objectives and parallel the accomplishment of these objectives with consideration of individual employee needs, their intellectual capabilities, and strengths that they offer to the workplace and change process.22 , 23

Paralleling effective leadership is the importance of managerial patience (taking a long-term view rather than short-term) during the implementation process in order to allow time for the inevitable reduction in productivity until the intervention takes hold.24 At the same time, it is important that leaders ensure the level of resources dedicated for implementation and continued sustainability have been identified and provided, including: money, training, education, physical space, and time.25 , 26 Organizations that lack effective leadership and that are insufficiently resourced may push their staff to reach absorptive capacity too soon, particularly if they fail to recognize that additional time is needed to learn, reflect, and incorporate practice and organizational changes into their practice as usual.27 It is likely that some level of recognition of absorptive capacity is necessary from the service funder as well.

Practitioner Readiness, Knowledge, and Beliefs

Low willingness to engage in the change process is a major obstacle to widespread implementation efforts. Four domains of practitioner attitudes can impact willingness to engage in change implementation and EBP adoption, including the perceived intuitive appeal of the new practice, the perceived organizational specificity of the practice implementation, a general openness to change and innovation, and perceived divergence between current and suggested practice.28

Mental health practitioners are more likely to adopt a particular practice if the evidence and support for the practice is generated by colleagues close to the practitioner.28–30 Thus, the information source of the proposed EBP appears to play an important role in practitioner adoption. Preliminary findings of implementation research with teachers have shown similar results.30 Similarly, practitioners’ willingness to adopt a particular EBP is reflective of their willingness to comply with the required changes and tasks dictated by the new EBP, and more broadly, those of the organization. Willingness to comply with organizational rules and regulations should not be confused with a practitioner’s openness to change.31 Unlike the willingness to comply with the specified practice parameters decided by the organization, being open to change is reflective of a general willingness to try new things despite the constraints that a organization may place on that change—with which the practitioner may or may not comply. If practitioners perceive a large or complete divergence in the proposed practice from practice as usual, they may be less willing to participate in the change process because they do not perceive inherent value in making that change.32 The lack of motivation to change and perception of divergence is compounded by the fact that behavioral health organizations are staffed by individuals who vary in the clinical competencies required for evidence-based practice,33 with practitioners having more education having a stronger likelihood of participating in practice change.28 , 31 , 34

The Change Model and Opportunities for Participation

Guidance for the clinical transformation process at Kinark Child and Family Services (Kinark) was provided by the National Implementation Research Network’s implementation model.35 Developed from a systematic review of the implementation literature, the original NIRN model stipulates six successive stages involved in the implementation process: Exploration and Adoption, Program Installation, Initial Implementation, Full Operation, Innovation, and Sustainability. Each of these stages requires consideration of core implementation drivers—staff selection, pre-service and in-service training, ongoing consultation and coaching, staff and program evaluation, facilitative administrative support, and systems interventions.35 Using the NIRN model as guide for transforming all practice as usual services to evidence-based practices, Kinark developed teams or “working groups” to focus on (1) exploration and adoption, (2) program installation, and (3) initial implementation. Each team was composed of staff representing a broad spectrum of the organization (geography, discipline, and function). Working groups were supported by internal experts in information technology, education, human resources, research and evaluation, and communications, as needed.8 Table 1 illustrates how the NIRN implementation stages were operationalized in Kinark’s application.7

Table 1 NIRN stages of implementation
Full size table

Kinark’s adapted NIRN model steered the implementation of eight different evidence-based practices over the 4 years of study, including: Solution-Focused Brief Therapy,36 Cognitive Behavioral Therapy (CBT),37 , 38 Stop Now and Plan (SNAP®),39 Multisystemic Therapy,40 Dialectical Behavioral Therapy,41 Girls Circle,42 Positive Parenting Program (Triple P),43 and People Places.44 Training in each of the EBPs was provided by either an internal staff member identified as competent by the Kinark Clinical Transformation Project Management Team, or by an EBP purveyor.

This paper provides an analysis of data collected over four years regarding staff knowledge of evidence-based practice, their perceptions of the change process, the effectiveness of working group activities, successes and accomplishments of the change process, and overall satisfaction with the change initiative. Their accounts shed light on EBP implementation, as well as the viability of the NIRN framework for supporting EBP implementation in behavioral healthcare organizations.

Methods

Design

A single exploratory case-study design was used to examine staff perceptions of the implementation of multiple EBPs in a pediatric behavioral health-care organization. In this instance, the single case study represents a critical case for testing the NIRN model.45 The study was approved by the Research Ethics Board at The Hospital for Sick Children. Overall, the following data were collected: observation notes and audio-recording of group meetings, key informant interviews with individuals involved in the change process selected by cluster randomization (setting), annual questionnaires among staff attending all-staff meetings at each of four years, administration of the Organizational Learning Survey46 in Winter 2008 and Fall 2010, and the tracking (thematic and chronological) of meetings, milestones and goals. The present paper only reports on staff perceptions captured in annual questionnaires and individual interviews. Related papers report an overview of the project8 and a process evaluation of team meetings throughout the clinical transformation process.7 Two papers are in development reporting on changes in organizational learning over time and managers perspectives of change.

Sample

Two sampling and data collection methods were used to capture staff experience of the implementation process: the administration and analysis of an annual staff questionnaire and the completion of semi-structured qualitative interviews with staff involved in the implementation change processes. All full-time staff—including secretarial, administrative, clinical, and management—were invited to participate in annual staff meetings at which the annual staff questionnaires were disseminated. The inclusion of all staff roles—including secretarial and administrative—ensured that our exploration of clinical transformation captured the role non-clinical support staff play in supporting EBP implementation.

Eligibility to participate in individual interviews during year four of the study was based on direct participation in the clinical transformation change initiative—each interviewee must have participated in a working group, an installation team, implementation team, the clinical transformation steering committee or the clinical excellence committee. Under these criteria, 182 of Kinark’s 579 full-time employees in 2010 were eligible to participate in individual interviews. Among these, 18 individuals were randomly selected, in collaboration with the Kinark Clinical Transformation Project Management Team; with the intention of including one front-line staff and one manager for each program area. The interviews were intended to explore staff experiences of the clinical transformation process in greater detail than possible via questionnaire. The sample size, while small, took into account staff absorptive capacity for clinical transformation activities, including activities imposed by the research, while providing an adequate sample to meet saturation with respect to emerging themes. It was felt that saturation of qualitative themes was possible with this small sample size given the homogenous and cohesive nature of the potential participants, namely that each individual worked for the same agency and received the same communication messages regarding the purpose and processes of clinical transformation throughout the change initiative.47 Moreover, methodological research concerning theme saturation indicates that data saturation can be derived from as few as six interviews, with the strongest support indicating data saturation and stability of code definitions after the analysis of 12 interviews.48

Data collection procedures

Annual staff questionnaire

A staff questionnaire was disseminated annually to all full-time employees—including secretarial, administrative, clinical, and management—at a general staff assembly. Despite the fact that part-time and relief employees had the option to attend the annual assembly; these individuals were not sampled given their inability to participate in the clinical transformation working groups. The purpose of the annual assembly was to discuss the progress of clinical transformation at the organization as well as to share other relevant news. Full-time employees in attendance were asked to complete the annual questionnaire, which had been developed in consultation with the Kinark Clinical Transformation Project Management Team and informed by the Clinical Transformation Project Charter. Questions assessed communication and organizational learning strategies employed by the clinical transformation project management team, staff understanding of the clinical transformation process, and the nature of staff involvement in the clinical transformation process. At Kinark’s request, the questionnaire was intentionally brief, including only three questions: (1) What, if anything, do you know about clinical transformation at Kinark? (2) How would you define evidence-based practice and treatment? (3) In your own work, do you use any courses of treatment considered to be evidence based? If yes, please name the treatment. As per ethics approval, consent to complete the questionnaire was implied if the participant completed and returned the questionnaire.

Interviews

Staff eligible to participate in interviews (n = 182) were grouped into managers and front line staff. Individuals were assigned a unique identifier corresponding to their program area, and were then randomly selected for interviews via random statistical selection using SPSS software—ensuring one front-line staff and one manager representing each program area. Selected participants were recruited by e-mail invitation specifying the interview process would be completed by a third-party interviewer (trained interviewer at the Hospital for Sick Children) during paid-work time. If a selected interviewee declined participation or did not respond to the e-mail invitation, eligible participants in their program area were re-randomized and another name was selected from the pool of participants and invited. Those consenting participation completed the interview by telephone and all interviews were audio-recorded and transcribed verbatim.

Interviewees were asked open-ended questions to elicit their perspectives on five major areas of interest: (1) understanding of the clinical transformation process, (2) effectiveness of the meeting processes, (3) successes and accomplishments of the clinical transformation process, (4) satisfaction with the transformation process, and (5) participation in clinical transformation-related activities (See Appendix A for the interview guide).

Data analysis

Annual staff questionnaire

Analysis of the annual questionnaires followed a two step process: first, each question was coded as “blank” (the question was left unanswered or the participant wrote “NA”), “don’t know” (for example, participant wrote: “don’t know,” “nothing”, “not much,” “no idea,” or “too new to Kinark”) or “answer given” (participant attempted to answer question, whether correct or incorrect). Second, deductive content analysis determined whether the participant provided a correct or incorrect response. A codebook of acceptable and unacceptable responses was developed in reference to Kinark’s Clinical Transformation Project Charter (KCTPC) prior to the commencement of coding.49 , 50 For example, responses to the question, “How would you define evidence-based practice or treatment?” would be identified as “correct” if the participant referenced some aspect of Kinark’s definition of evidence-based practice within the KCTPC. All questionnaires were reviewed and coded for correspondence or exemplification from acceptable or unacceptable responses51 in a table using Microsoft Word. For each survey year, 25% of the questionnaires were independently double coded by a second coder. Following double coding, the Microsoft Word coding-table files were merged and the degree of rater agreement was calculated for each question by dividing the sum of the matched ratings by the total number of ratings (questions that were left blank were not included in the calculation). This inter-rater calculation presents a percentage of agreement but does not take into consideration the proportion of agreement caused by chance alone. Given the qualitative, collaborative, and exploratory nature of this study, discrepancies in coding were resolved through consensus (consultation with MB and the Kinark Clinical Transformation Project Management Team), and for this reason, kappa calculations of inter-rater agreement are not appropriate for this study.

Interviews

The real-world application of the NIRN implementation model lent itself to inductive content analysis,52 , 53 beginning with open coding and the development of a preliminary codebook that encompassed emergent categories. Following multiple readings and comparisons between codes and their application to the interview texts, themes were grouped into higher-order headings and a description of each heading’s meaning was generated to produce a final code book. Twenty percent of the interviews were independently double coded using the final codebook to ensure integrity of data analysis. Disagreements in coding were resolved through consensus making discussions and in consultation with MB. All interview data were analyzed and managed using a qualitative data management program, Nvivo (QSR International Pty Ltd., version 8, 2008).

Results

Questionnaires

Annual staff questionnaires were administered to all full-time staff in attendance at the annual all-staff meeting in years 2006, 2007, 2008, and 2009. Coding for all questions reached a degree of inter-rater agreement above 80%. What follows is a reporting of the results for each item in the questionnaire across all four years of study. Table 2 presents the number of full-time staff in attendance each year and the number and percentage who responded to each question correctly.

Table 2 Attendance at annual meeting and questionnaire performance
Full size table

Knowledge of the clinical transformation process

Staff knowledge of the clinical transformation process grew over time but never reached over 60%. In 2006, only 36% of responders were able to demonstrate that they understood the purpose of the clinical transformation process. Over the next three years, staff understanding grew from 55% to58%.

Defining evidence-based treatment

In 2006, 58% (N = 122) of responders could define evidence-based practice. Levels of EBP knowledge remained unchanged in 2007 (57%) but rose in the last two years, 2008 (70%) and 2009 (77%).

Use of evidence-based practice

In 2006, 82% (N = 195) of responders indicated they were utilizing an EBP in their own practice, the most common being the Triple P (N = 45), CBT (N = 39), and Intensive Behavioral Intervention (N = 36). Although 2007 saw a decrease in the number of staff reporting use of EBPs (72%), numbers related to EBP use rose in both 2008 and 2009 to 77% and 78%, respectively. The most commonly referenced EBPs—in both years—being the Triple P, CBT, and Applied Behavioral Analysis.

Individual interviews

Participants

Thirteen staff (1 male and 12 females) were interviewed: five at the management level, six in front-line positions, and two in corporate positions. Interviewees had an average term of employment tenure at Kinark Child and Family Services (Kinark) of 10.5 years, with a range of 3 to 25 years of service. As per selection procedures, all interviewees had participated in at least one working group related to one of the three primary stages of the NIRN model—Exploration (Working Group), Installation, and/or Implementation. Specifically, four interviewees participated in one clinical transformation group, one interviewee participated in two clinical transformation groups, four interviewees participated in three clinical transformation groups, three interviewees participated in four clinical transformation groups, and one interviewee participated in eight clinical transformation groups.

Themes

Several themes characterize the staff experience of the clinical transformation process, with the most common relating to understanding the clinical transformation project and stages, having clear leadership, having an inclusive change culture, supplying needed resources, and doing this differently. Present in all 13 interviews, these main themes varied in degree of importance for each interviewee but were persistent within and across interviews. What follows is a reporting of these five over-arching themes and the major sub-themes that permeated each of the interviews.

Understanding clinical transformation: a thoughtful and intentional process

When asked to speak to their understanding of the purpose of clinical transformation in year four, all interviewees could clearly articulate the objective of the clinical transformation project and specifically spoke to the goal of simultaneously ensuring consistent and effective service provision. Interviewees viewed the purpose of clinical transformation as determining the best available evidence in treating their patient groups, and transitioning to provide these best practices. Overall, staff grounded their comments and the entire clinical transformation project in the need to provide consistent service across the organization’s programs.

Interviewees consistently made reference to the clinical transformation process as thoughtful and intentional. They viewed Kinark’s use of a staged approach as a good foundation to guide the EBP implementation, recognizing Fixsen et al.’s35 implementation drivers—such as clear leadership and reporting—as a key for framing their work within respective working groups. Interviewees were satisfied with how each implementation stage for each EBP was strategically guided by templates for task completion and communicated thoroughly throughout the organization through an online project management tool called CT Central. They felt that the project management tool, as well as other communication strategies, were extremely helpful in managing their own involvement in the clinical transformation activities and reflected the organizations broader thoughtfulness with respect to keeping staff informed and involved in the change process.

Clear leadership: effective working groups and effective project management

All interviewees discussed the vital role of effective leadership for the success and maintenance of clinical transformation. They viewed working group leaders as essential in guiding the group’s process, ensuring task completion, re-focusing the group’s work if they got off track, and keeping the group to the timelines assigned to them. Those who felt they experienced poor leadership within their working groups reported that the group would “flounder,” “run around in circles,” and “need to request extended deadlines and support in completing tasks.” Similarly, all interviewees expressed strong appreciation for the clinical transformation project manager, the person responsible for steering the entire clinical transformation project and who was, to some degree, involved in every working group, installation team and implementation team. They viewed the project manager’s extensive knowledge of the clinical transformation project, its objectives, and insights into how other working groups had operated and completed their tasks as useful in guiding their own work within their respective working group. They perceived the project manager as helpful in mediating feelings of being overwhelmed among group members, and as helping the groups to work towards a mutual understanding and process for completing project tasks.

“Having the leadership of the project manager as sort of overseeing the whole process of clinical transformation was really effective. Having that presence early on in the group meetings, that was very effective in sort of setting the stage for the process and how the process works.”

Inclusive culture change

Another common theme across all interviews was a sense of inclusive culture related to the change initiative. Each of the three organizational areas—front-line staff, managers, and corporate staff—felt that one of most effective strategies for ensuring the success of the clinical transformation project was the inclusion of both front-line and management staff in the process. Interviewees felt that forcing changes that would prove to be inconsistent with the reality of the everyday frontline service would be unsuccessful and unsustainable. For example, one manager states,

“For me, I think the most pertinent thing has been the front-line staffs’ involvement in the clinical transformation process. So, having them in the working groups, having them involved in the implementation directly, and have a real say in terms of how the interventions have come in and been developed and incorporated into their work has been imperative and significant to the process.”

Interviewees felt there was real value in involving a diversity of behavioral health disciplines and representation from across the organization’s geographical program areas in each working group. Involvement of various disciplines—social workers, psychologists, and child and youth workers—was useful because it brought varied perspectives on service implementation and how shifting service delivery would affect the different clinical roles within the organization. Furthermore, cross-program collaboration was seen as effective for reducing staff misconceptions of program operations as well as validating individual clinical experiences and challenges within their particular service provision.

Supplying needed resources

There was great appreciation for the release time from main responsibilities provided to those participating in clinical transformation. In their view, had management pushed staff to participate in clinical transformation activities outside their regular work hours, participation would not have been welcomed.

“The management of the agency has been very supportive of our trying to manage CT as well as other commitments throughout the year.”

“It is so easy to have meetings conflict that are equally as important as other things. That hasn’t happened because of the messaging and permission from the agency to make this (CT) a priority.”

Despite the fact that interviewees felt that management recognized the time required for CT activities, there was still a sense that clinical workloads would need to be maintained in spite of CT related commitments. It was commonly felt that managing the demands of clinical work—which for staff took ethical precedence—while juggling the demands of clinical transformation, meant that at some points, work had to be put aside and prioritized. For the most part, it was the clinical transformation work that received lower priority.

“Like when I was starting on that working group, I was also meeting [the demands of] our accreditation process. So, you know, I kind of had to find the balance between doing both of those things, knowing that the one—the accreditation, had a specific timeline. So, I said to the group, you know, we are about to do this (accreditation), and if you want to go on to the next steps, it will have to wait until another time, because these other things are up.”

Doing things differently: appreciating the outcome and making suggestions for change

Despite the perceived strain resulting from a heavy workload, balancing both clinical and transformation-related responsibilities, staff articulated their belief that the benefits of clinical transformation would outweigh any disadvantages. Interviewees’ felt that clients would experience clear and measurable benefits as a result of the transformation to EBPs, and could look forward to consistent organization-wide service provision and peace of mind stemming from the use of evidence-based practices.

In terms of personal implications, they perceived greater confidence in their own skills and abilities to treat their clients effectively. The thorough training they received in a variety of evidence-based practices made them more competitive in the behavioral healthcare field—particularly as they envisioned the possibility of moving to positions outside the organization. Paralleling this shift in staff confidence was an increased confidence in the organization as a leader in behavioral healthcare. Participants were unaware of any other Ontario behavioral healthcare provider organization that had undertaken a similar transformational effort, and this embodied a sense of pride.

When asked what could have been done differently throughout the clinical transformation process, interviewees made three general suggestions—to maximize internal resources, to more carefully consider working group composition, and for the benefit of other organizations that may undergo a similar EBP change process, to make a greater investment in preparing staff for change. Specifically, identify for staff specific time commitments that may be required to participate in the change initiative, clearly articulate any change to case-load expectations for staff who may participate in the project, as well as the anticipated timelines for completion of various implementation stages.

In terms of maximizing internal resources, interviewees recognized that practitioners embody a range of clinical skill level and competencies. Some interviewees believed that the organization could have spent less money hiring external EBP purveyors and utilized their own staff to train and supervise their colleagues.

“My issue is that some of the trainings are not even provided by our own staff. Yes, we need professional training… but using our own resources, like we have done for some of the Triple P training, and if we could do that for some of the other EBP trainings that will be rolled out, that would be good.”

There was perceived need that more careful consideration of group composition for each of the implementation stage working groups would have been beneficial. For example, there was a sense that some staff were over-committed to several clinical transformation activities, and for this reason, their work and contribution to the group process was minimal and strained. Also voiced was an impression that some staff were just not cut out for implementation work, and should stick to providing clinical services. Individuals who were not good with implementation work brought down the morale of their group peers and slowed the group processes.

“There are still people in the agency that are involved in this (CT) that shouldn’t be, because they don’t get it, and they don’t try. It is clear that this isn’t their priority; it’s more just on their plate. Because there are some people that do this really well, and then there are others who struggle, and then the strugglers keep coming, saying I can’t get it, I can’t get it. Well, that’s because you can’t do it.”

Lastly, there was a perception that the organization could have taken greater measures to prepare staff for the changes that would occur throughout the change initiative. Practitioner buy-in is known to be an essential component of effective and sustainable change; if staff do not understand the nature, purpose and process of change then it becomes increasingly difficult to rely on these individuals to meaningfully participate in the change process. Interviewees reported that notwithstanding four years of clinical transformation, some individuals continued to demonstrate resistance to the change process and its intended outcomes.

Discussion

Research in the implementation of evidence-based practices has recognized multiple factors implicated in successful change initiatives.5 The field has called for practice-based implementation research to more closely examine how the change process unfolds in the real world.54 The current study explored EBP implementation and organizational change through an exploratory case study of a large behavioral healthcare organization as it underwent an organization-wide shift from practice as usual to practice informed by the evidence base. This approach provided a critical case for testing the NIRN implementation model.

Staff perspectives captured through questionnaires and interviews support previous research that suggest successful and sustainable clinical transformation within the context of behavioral healthcare is dictated by a number of factors related to broader organizational structure and processes.25 , 28 , 29 , 35 As a critical case for the application of the NIRN model, some evidence is provided of the model’s utility and acceptability in light of the modifications made by the organization in its application to their context. Staff experiences suggest that greater refinement or operationalization of the model is likely warranted while supporting the importance and relevance of the core drivers of practice change. While such factors as staff training, leadership, and internal communications are important considerations for implementation, as identified by the NIRN model, organizations will need to drill down to articulate what has to be done to prepare and monitor these activities over the course of implementation activities. The NIRN model identifies important areas for consideration but greater specificity and detail is needed to provide clear, simple guidelines for how to proceed relative to all the important factors.

Staff perspectives captured via questionnaire paint an interesting picture of shifts in their understanding of evidence-based practice throughout the implementation process, rising steadily from 57% in 2007 to 70% in 2008 and finally, 77% in 2009. This gain in awareness and understanding of what constitutes an EBP could explain why more staff reported using a specific EBP at Kinark in 2006 than in subsequent years (2007, 2008, and 2009); they simply did not know if they were using an EBP or not. As the clinical transformation project evolved in parallel with its communication plan, staff likely gained a more critical understanding of what constitutes an EBP from a research and practice perspective. Thus, what staff previously thought of as an EBP or promising practice in 2006 could have been later understood as not having validated research or field testing—reflecting a more sophisticated understanding of the tenets of EBPs. The relative lack of sophistication in EBP preparedness has been acknowledged elsewhere.33 , 55

Despite the steady increase in EBP understanding among staff, there was less gain in their understanding of what constituted clinical transformation within the organization. Specifically, staff understanding of the clinical transformation project peaked at 58% in 2009. It is possible that the emphasis on EBP implementation and the understanding of EBP more generally, overshadowed the general messaging of clinical transformation and its premise for EBP implementation. In other words, staff have a tendency to focus on the EBP being adopted, related practitioner training, and the implications for clients rather than on implementation as an organizational process. Perhaps, clearer messaging pertaining to clinical transformation and its process role in EBP implementation would be warranted for other organizations looking to implement an EBP change initiative.

The importance of project leadership for EBP implementation was a salient theme in this research and in keeping with the broader organizational change literature, provides further support for the importance of an integrative leadership model for successful implementation.10 , 17 , 20 Staff clearly indicated how essential the role of the project manager was in the development and sustainability of the clinical transformation initiative. Moreover, working group leaders made significant contributions to the success or lack thereof, of each working group depending on their ability to effectively keep the group members on task, accept primary responsibility for the group’s outcomes, and make sense of the complex project management information. Relatedly, staff expressed appreciation for the involvement of front-line staff in leading change. Their role as change leaders was viewed as a logical—but often overlooked aspect of change process—and as essential given the need for front-line staff to navigate administrative and clinical realities. A similar viewpoint on the unique ability of front-line staff to engage in leadership positions has been seen in the literature.29 , 56

Despite the support espoused by all interviewees for clinical transformation, they also identified instances where their colleagues did not align with the change agenda. Staff resistance to change was attributed to a lack of attention or preliminary preparation for impeding change. Research indicates that employees can be resistant to change when they fear that the change process, and its expected outcome, will suggest their ignorance of a previously expected skill set.29 Within the context of behavioral health care, this could mean that a frontline manager may be resistant to change because an expectation of the change process may be for the practitioner to demonstrate a clinical skill set that the practitioner may not actually have, but was perceived to have by the employer. Moreover, staff resistance to change has been paralleled to inadequate preparation for the change process and a lack of clear identification of the benefits of the change initiative.10 , 16 , 19 To alleviate anxieties regarding organizational and practice change would seem to require greater attention to pre-implementation preparation and communication. It appears to be important to spend significant time in preparation, to educate staff regarding the change initiative purpose and process, expectations and anticipated outcomes, and expectations for professional development. To this end, we have worked with the Ontario Centre of Excellence for Child and Youth Mental Health to develop a web-based implementation curriculum for the explicit purpose of preparing behavioral healthcare organizations for EBP implementation.57

The small number of interviews can be viewed as a limitation of the study. This is, however, a reality of research conducted in the real time, in real practice, where care is needed in over-burdening practitioners who have limited time to devote to research activities over and above provision of clinical service and the additional activities related to their participation in the change initiative. The interviews were intended to explore staff perceptions of the change process and thereby lend support to the constructs identified as important in the literature; and to this extent, they largely do so. Difficulty experienced with recruitment is perhaps reflective of the level of burn-out that we hoped to avoid. Moreover, a purposeful convenience sampling strategy was used for the annual all staff questionnaire and so the results may not be entirely reflective of full-time behavioral healthcare staff across the behavioral health care sectors.

Implications for behavioral health

This case study of staff experiences related to EBP implementation and organizational change provides preliminary support and insight regarding the field use of the NIRN implementation model as a guiding framework for behavioral healthcare organizations. Future research of its application in other organizations with varying contexts is warranted. There is an appreciation that the NIRN model shepherds a thoughtful and intentional approach to implementation that takes into consideration the numerous complex factors and multifaceted processes implicit in the change process. Perhaps most importantly, EBP implementation requires leadership having a sound understanding of the change agenda and an ability to effectively and tenaciously communicate the tasks and processes required for large scale change while tolerating the disruption that often emanates from the process. Kinark’s experience represents a unique perspective on the change process and the implementation of EBPs in child and youth behavioral healthcare insofar as it is the largest child and youth behavioral health provider in Ontario, and the first, to our knowledge, to undergo a change of this scale utilizing the NIRN implementation model. Similar studies in other contexts will, in time, support or refute the findings presented here.

Despite growing awareness of organizational change as a complex and labor intensive process, implementation of evidence-based practices is increasingly being encouraged in behavioral health care policy. This has important implications for service provision at the practitioner, organizational, and system levels that must be addressed by educators, providers, and government. Growing recognition of gaps in workforce preparation for the evidence-based practice environment needs to be addressed by institutions of higher learning and by service provider organizations33 Educational institutions must be encouraged to increasingly incorporate EBP knowledge and experience into their curricula, while providers must balance their capacity to offer continued professional development whilst maintaining excellence and efficiency in service delivery. Governments must be encouraged to operationalize their policies to shift to evidence-based care in ways that acknowledge the complexity of the task, the implications of organizational change for service delivery, and the needs of practitioners for support and training.

The application of the NIRN model in this study highlights the importance of staff buy-in, shared leadership, readiness for organizational and practice change, effective communication, and resource availability for implementing and sustaining broad-based organizational change. Through the use of NIRN’s thoughtful and intentional staged process, Kinark’s experience of clinical transformation exemplifies the facilitators and barriers in the actual transformation process. Specifically, Kinark is an example of what is possible—enhancing behavioral healthcare while simultaneously maintaining sustainable and high quality service provision to a population that is already underserviced and underfunded.