Introduction

System of care development has evolved over the past 40 years, stimulated by the recommendations of the Joint Commission on Mental Health of Children (1969), a congressionally-appointed body, that completed a 4-year national study and reported that millions of children were not receiving needed mental health services. More than a decade later Unclaimed Children, Knitzer’s (1982) national study of mental health services for children and youth, revealed serious deficits in services throughout the country. In 1984, the federal response to these findings launched the first phase of service reform through the Child and Adolescent Service System Program (CASSP), which provided funding to the states to begin restructuring children’s mental health services. Descriptions of the evolving reform efforts can be found in the writings of Behar (1985, 2002), Friedman (2005a, b), Lourie (2002), Stroul and Friedman (1986, 1996a, b). The reports of the Surgeon General (1999) and the New Freedom Commission on Mental Health (2003) emphasized the value of this reform in improving services to children with mental health disturbances and their families. System of care has become federal policy, promulgated by the Child, Adolescent and Family Services Branch of the Center for Mental Health Services, Substance Abuse and Mental Health Services Administration, US Department of Health and Human Services (2006).

In 1993, the Comprehensive Community Mental Health Services Program for Children and Their Families (2006) legislation began the second phase of systems reform. This Act provides funds to improve/expand community-based systems of care and to address the needs of an estimated 4.5–6.3 million children with serious emotional disturbances and their families. Systems of care are promoted on the premise that the mental health needs of children, adolescents, and their families can be met within their homes, schools, and communities. The Act offers a philosophy that includes four elements: (1) the mental health service systems should be driven by the needs and preferences of the child and family and addressed through a strength-based approach; (2) the focus and management of services should occur within a multi-agency collaborative environment and should be grounded in a strong community base; (3) the services offered, the agencies participating, and the programs generated should be responsive to the cultural context and characteristics of the populations served; and (4) families should be lead partners in planning and implementing the system of care. Funding has been provided nationally to nearly 22% of the 3,177 counties, parishes, boroughs, independent cities, geographical census areas, geographic regions, and the District of Columbia, and has served over 90,000 children and youth. Grants have also been given to 15 federally recognized tribes. Funding is at the level of approximately $5 million per site over a 6-year period. There are 59 communities (some multi-county, some statewide) currently funded and 83 graduated sites. There are additional federal funds for an independent evaluation, technical assistance and training. Thus, the focus on systems of care addresses a major federal policy with a major investment of funds.

The federal agency responsible for managing the Comprehensive Community Mental Health Services Program for Children and Their Families is the Child, Adolescent and Family Branch, Center for Mental Health Services, Substance Abuse and Mental Health Services Administration, US Department of Health and Human Services. This agency provides communities with funding, policy and practice guidance, and technical assistance to improve and expand community based services into coordinated systems of care. System of care policy is based on a set of principles and provides a framework for organizing and delivering services/interventions. There are specific requirements regarding governance structures, interagency collaboration, parent participation in the design and development of services, and approaches to the development of individualized plans of care. Emphases of practice guidance and technical assistance are on the four elements listed above, plus the use of evidence-based practices specific to the disorders presented by each child. The type(s) of specific treatments to be utilized are the responsibility of each funded community, within these parameters. A recent report (Substance Abuse and Mental Health Services Administration, Unpublished manuscript, 2008) indicates that children, youth and families benefit from services delivered within a system of care by improving their emotional well-being and behavioral functioning, improving school performance, reducing contacts with law enforcement, and reducing their use of inpatient care.

Such community transformation is a complex process that involves many stakeholders, including those from public agencies such as mental health, schools, public health, child welfare and juvenile justice, private providers of health and mental health services, families and youth, and community leaders. Recognizing the complexities of the change process, the federal agency develops cooperative agreements with each community for a 6 year period. The first of these years is a planning year, during which the groundwork for systems change is developed by the community partners. From the inception of the program, the Child, Adolescent and Family Branch has sought to identify strategies and processes that enhance successful implementation of system of care framework and support positive outcomes for children and their families. Ongoing evaluation of these programs by Macro International (Manteuffel et al. 2002, 2006) indicates that some programs do quite well and provide effective treatment and show positive impact on children, but others struggle in terms of their capacity to coordinate and integrate services across community agencies, the number of families they serve, and the progress the children make.

The concept of “community readiness” offers an important contribution to improving the planning and implementation process for communities. Understanding what factors are important to the successful implementation of the system of care framework should help communities assess their own strengths and weaknesses. Further, such understanding could support technical assistance efforts by helping to determine areas of focus and strengthen areas of weakness.

Although systems of care may look quite different across communities, given the differences in community size, characteristics, and culture, there appear to be common elements that underlie their development. There is a meager but growing body of knowledge that is applicable to understanding the complex factors that contribute to the successful development of systems of care for children and adolescents with serious emotional disturbances. Behar et al. (2005) used a case study method of nine successful sites and identified nine important factors, to include: transformational leadership, strong foundation of values and principles, a clear description of the local population, a clear and widely held theory of change, an implementation plan, family choice and voice, individualized, culturally competent and comprehensive approaches/interventions, and an effective governance system.

Similarly, Hodges et al. (2007a, b) used intensive case studies over a 6-year period to identify factors that contribute positively to the development of systems of care, to include: shared values, willingness to change, shared accountability, delegation of authority, strategic use of resources, family empowerment, and information-based decisions. Over the past 3 years, Friedman et al. (2009) have developed a survey instrument based on a conceptual model of 14 factors, built upon the nine factors developed by Behar et al. (2005) considered important to successful implementation of systems of care. Their factors include: family choice and voice, individualized treatment, outreach and access to care, transformational leadership, theory of change, implementation plan, local population of concern, interagency collaboration, values and principles, comprehensive financing, skilled provider network, performance measurement, provider accountability, management and governance. They are in the process of conducting a large sample, county-based study to test these factors. Edwards et al. (2000) point out that, “Communities are at many different stages of readiness for implementing programs, and this readiness is a major factor in determining whether a local program can be effectively implemented and supported by the community.” Their Community Readiness Model was developed to provide communities with a theoretical framework, a process, and specific tools to facilitate readiness. Other efforts to develop readiness assessments include (1) Osher and Huff’s (2007) Family Driven Care and Practice System Self Assessment Tool and The Community Readiness and Assessment Tool, which includes a readiness component that taps participant’s perceptions of the role of families

Reports relevant to systems change, but not focusing directly on readiness, are based in other public systems and focus on implementation strategies. Chinman et al. (2004) have developed guidance for implementation of substance abuse prevention programs and focus on the gap between the positive outcomes of prevention science and the more limited outcomes of prevention practice. They have developed a manual of implementation strategies for “Getting to Outcomes” which offers promise for improving practice. Later work on this topic includes ten principles of empowerment evaluation (Fetterman and Wandersman 2005), which focus on improving implementation and evaluation. These include: improvement, social justice, inclusion, democratic participation, capacity building, organizational learning, community ownership, community knowledge, evidence-based strategies, and accountability. Wandersman (2009) has translated these principles to systems of care implementation.

Another approach to systems change includes a focus on state level changes for building sustainable improvements in public health (Padgett et al. 2005). Using a qualitative, case study design, these authors analyzed strategies used by Turning Point (a Robert Wood Johnson initiative). The strategies included: institutionalization within government, establishing “third sector” institutions, cultivating relationships with significant allies, and enhancing communication and visibility among multiple communities.

The current study has been designed to further the understanding of community readiness. The focus of this paper is on both the findings and the methodology, which brings (1) an efficiency by conducting the study on a large sample over a short time period, and (2) results based on accepted statistical analyses, going beyond some of the earlier work which has been based on case studies and more subjective interpretations.

The study is based on the assumption that those involved with systems of care have insights to offer on the essential elements for success. The study uses a web-based approach to obtain information from (1) professionals who study systems of care, provide consultation and guidance on its development, and those who have managed systems; (2) families who have participated in systems of care; and (3) stakeholders who are involved in the systems change.

Method

In this study, the web-based version of concept mapping,Footnote 1 as developed by Concept Systems, Inc. (2006), was used to develop an understanding of community and systems factors that underlie the concept of community readiness. Information was gathered from national experts and representatives of experienced sites funded to develop systems of care, that is, sites in the 5th and 6th years of implementation. The goal was to better define the elements/factors in this complex area by synthesizing input from stakeholders and national experts across the country.

The statements, as sorted by the national experts, were organized into content areas/domains (clusters) by (1) creating a similarity matrix from the sort data, (2) using multidimensional scaling of the similarity matrix to locate statements as points on a map, and (3) using hierarchical cluster analysis of the multidimensional scaling coordinates to group the points on the cluster map. The information within each cluster was rated by the site representatives according to importance and difficulty of implementing, using five-point rating scales (Kane and Trochim 2007). The scoring of this information identified the concepts that the participants defined as central to readiness and to be the most important and easiest/most difficult to implement. The findings provided a description of community readiness derived from the statistical analyses of the data, as used by Concept Systems, Inc. described below.

Concept Mapping

The technique of concept mapping was developed in the 1970’s (Novak 1998) as a way to visually present the ideas of groups on a topic of interest to them. Concept mapping has evolved through the efforts of social scientists and there are now many methods now available to collect and analyze qualitative information. The method designed by Concept Systems, Inc. (Kane and Trochim 2007; Trochim 1989a; Trochim and Linton 1986) is a mixed-methods (Greene and Caracelli 1997) planning and evaluation approach that integrates familiar qualitative group processes including brainstorming, and sorting and rating of statements, with multivariate statistical analyses to help a group describe its ideas on any topic of interest and represent these ideas graphically through maps. The process requires the participants to brainstorm a large set of statements relevant to the topic of interest, individually sort these statements into categories of similar statements, and rate each statement on one or more dimensions. Concept Systems, Inc. has developed a “next generation” research-based methodology to analyze the data obtained, so that the result is an unbiased and fair description of the participants’ input.

The analyses include multidimensional scaling (MDS) of the sort data, hierarchical cluster analysis of the MDS coordinates, and computation of average ratings for each statement and cluster of statements. These data are then used to generate the maps which show the individual statements, with more similar statements located nearer each other, forming a cluster map. Bridging analyses can also be conducted to understand the placement of items within clusters, to clarify how frequently an item is placed in one cluster versus multiple clusters. Analyses of the rating data yield rankings of items within clusters and ranking of items. The latter analysis yields a “go-zone” or “focus-zone” map reflecting the interactive ratings on multiple dimensions.

The Concept Systems, Inc. approach has been used effectively to address substantive issues across a wide range of fields, including public health, human services, higher education and industry (Kane and Trochim 2007; Trochim 1989b; Trochim et al. 2003). Data obtained through concept mapping has been used to develop rating scales (Rosas 2008). Federal and state government agencies, such as the Center for Disease Control and Prevention and the Hawaii Department of Health, have used Concept Systems’ web-based program successfully for brainstorming, sorting and rating (Trochim et al. 2003; Graham et al. 2008). System of care sites using concept mapping for planning, development of logic models, and evaluation include “commUNITYcares” in Mississippi, “Circle of Hope” in Missouri, and three sites of “Integrating Families, Communities, and Providers (IFCAP)” in Florida.Footnote 2

Participants

Two groups, (n = 223) were invited to participate in this project. The first group consisted of grant communities in their 5th and 6th year of funding from the Center from Mental Health Services, Child, Adolescent and Family Branch. Invitations to participate were sent to 27 sites, including three tribal communities. Those invited included project directors, principal investigators, clinical directors, lead family coordinators, youth coordinators, cultural and linguistic coordinators, technical assistance coordinators, and social marketers (N = 155). The second group of participants was comprised of a panel of national experts, selected by the investigators. The experts included people from graduated sites and those who have served as consultants, evaluators, trainers, and leaders in the design and development of systems of care (N = 68). Invitations were sent by the investigators directly to these individuals.

Procedure

Using the Concept Systems, Inc. web-based CS Global© system, input about indicators of community readiness were obtained from the participants as described above. The two-part process took place during the period of April 24, 2008–August 17, 2008. Participants’ input was collected in two phases. Phase 1 consisted of brainstorming, and involved generating a list of community and systems factors. Phase 2 consisted of organizing those factors (sorting) and rating them for Importance and Difficulty of Implementation (rating). The Concept System computer software version 4.147 was used for the analysis and generation of the cluster maps.

Phase 1 (Generating Statements)

This first part of the study was completed from April 24 to May 30, 2008. Members of group 1 and 2 were asked to participate. Of the 223 people invited from both groups, 135 (61%) participated and of these, 115 (85%) completed the task, resulting in a response rate of 52%. All participants were asked to complete a demographic form. Using the web-based program for the brainstorming activity, participants were asked to complete the following focus statement by typing statements into a text box: “To be ready to develop a system of care, the following specific characteristics and functions are essential to be in place before an application for funding can be completed.” The instructions were that each participant could enter 5–6 statements. The group produced 275 statements. The investigators reviewed each statement and separated those that contained more than one idea, resulting in 336 statements. The 336 statements were reviewed for duplication, resulting in 109 statements.

Phase 2 (Organizing and Prioritizing Statements)

This second part of the study was completed during the period of June 30–August 17, 2008. Using the web-based program, group 1 was asked to rate the 109 statements according to their Importance and Difficulty of Implementation. Group 2 was asked to sort the 109 statements into categories of similar statements and to provide their own labels for those categories. Groups 1 and 2 were created because the investigators thought that it was too much to ask participants to do both tasks, as the ratings took 30–40 minutes, and the sorting took 45 minutes to 1 hour. Responses were anonymous.

Rating

Group 1 participants rated each of the 109 statements first on the dimension of Difficulty of Implementation and second on Importance. The ratings were based on a five-point scale with 1 indicating very easy to implement and 5 indicating extremely difficult to implement or 1 indicating not at all important and 5 indicating extremely important. This task took on average 30–40 minutes. Of the 155 people invited to participate in group 1, 84 (54%) accepted and went to the website. Of these, 69 (84%) completed the first rating task, resulting in a response rate of 45%; and 65 (77%) completed the second rating task, resulting in a response rate of 42%. For these tasks, there was representation from 25 of the 27 program sites.

Sorting

Each of the group 2 participants was presented with a list of the 109 statements and was instructed to use a “drag and drop” method of arranging the statements. Each sorted the statements by grouping them into categories of ideas that were similar to each other. The participants were asked to label the categories. This task took on average 50–60 minutes. Of the 68 people invited to sort the statements into groups/domains, 39 (57%) participated and of those 39, 36 (92%) completed the sorting task, resulting in a response rate of 53%.

Results

Demographics

For both phases of the study (Brainstorming and Sorting/Rating) participants were asked to complete a demographics form which covered four areas: age, gender, ethnic identity, and role in the system of care. Responses were anonymous. Of the participants, 22% chose not to fill out the demographics form. Of the other 78, 74% were female and 26%, male. The distribution by age was 25–34, 8%; 35–44, 22%; 45–54, 33%; and 55+, 36%. The reported ethnic identity included European American/White, 68%; African American/Black, 13%; Hispanic/Latino, 9%; Native American Indian/Alaska Native, 5%; and Mixed, 5%.

In the question relating to role in the system of care, Administrators, which referred to the principal investigator and project director, had the largest representation (28%), followed by Outside Experts/Consultants (13%). This latter group was referenced above as group 2, those who served as consultants, trainers, evaluators, or who had managed successful systems of care. The combination of parents (4%) and parent coordinators (8%) makes the parent representation the next largest group at 12%, followed by Technical Assistance Coordinators, (10%). The remaining 33% of the participants were essentially equally divided among Clinical Supervisors, Cultural and Linguistic Coordinators, Principal Investigators, Social Marketing Coordinators, Youth Coordinators, and representatives of community partner agencies/service providers. Overall, there was broad representation from the possible range of respondents.

Results of the Sorting Process

The concept mapping analysis used data collected through the sorting task to determine the configuration of the clusters (domains) that form the concept map. Statistical analyses yielded a visual configuration of the statements that the participants placed together most often. The stress value is the statistic used in this type of analysis to indicate the goodness of fit. A lower stress value indicates a better fit. In a study of the reliability of concept mapping, Trochim (1993) reported that the average stress value across 33 projects was .285, with a range from .155 to .352. The stress value in this analysis was .280.

No mathematical criteria are available to select the appropriate number of clusters. In the sorting process, the number of categories used by the participants ranged from 4 to 21. The investigators began with the highest number of categories (21). Then, they examined successively merging clusters, making a judgment at each stage about whether the merger seemed to combine similar concepts and whether important discrete concepts were lost in the merger. The results of this review yielded an eight cluster solution, as this provided the most discrete clusters that did not contain overlapping ideas. Figure 1 shows the eight cluster solution.

Fig. 1
figure 1

Eight cluster solution

Full size image

Each of the group 2 participants had been asked to provide a name for each of their groupings of statements. The Concept Systems, Inc. software generates cluster labels based on an analysis of frequency and similarity of the names selected by the participants.

The clusters created by the participants are consistent with principles of the system of care policy promulgated by the federal agency. The clusters are similar to the common factors that Behar et al. (2005), Hodges et al. (2007a, b) identified in their case studies of systems of care sites. Friedman et al. (2009) report similar preliminary findings of seven factors using a survey method. The current study builds on earlier works (Behar et al. 2005; Hodges et al. 2007a, b) and validates those findings by using measurable/quantifiable concepts. The Concept Systems, Inc. methodology provides for statistical analyses of data, which is a step beyond the earlier studies. The earlier studies were based on summaries from interviews and observation. The current study uses a quantitative “next generation” method, and provides new information.

A study of the cluster map reveals that the central cluster, Shared Goals, is a bridging cluster in that it “holds together” or links surrounding clusters. According to this map, the other seven clusters are organized around the Shared Goals and it is the items in this cluster that bring the other clusters together.

The cluster map shows that the clusters of Collaboration and Leadership are both densely populated with statements that are located very near each other on the map. Collaboration is more tightly put together, meaning that these items are distinct and participants very often placed them together. On the other hand, the Leadership cluster suggests that many ideas (actions) came together as a broader set of ideas. In other words, there are more differing concepts in the Leadership cluster than in the Collaboration cluster, which has more similar ideas.

A bridging analysis indicates that these two clusters, Collaboration and Leadership, have the lowest bridging values of the eight clusters, indicating that the items in these clusters were most frequently placed together and infrequently placed in other clusters. The scores for a bridging analysis range from 0 to 1.00. The average scores for these two clusters were each less that .25, with the average bridging value for Collaboration being .15 and for Leadership, .22. The other cluster with a low average bridging value is Family and Youth as Partners (.28). The low bridging values suggest that these three clusters are the “cleanest” clusters.

The Network of Partners cluster has items that are placed in two somewhat separate areas. The items at the top of the cluster, closer to the Families & Youth as Partners cluster, involve network issues and having families in the network. Examples of these items are, “An advisory or leadership board should be established that has at least 1/3 parent participation and they should have input on the writing of the proposal,” and “There should be active participation from families, youth and front-line workers from public and private sectors in the implementation of the system.” The items in the lower part of the cluster involve network partners and issues of collaboration. Examples of these items are, “All partners should have a sense of community identification and buy in to the System of Care mission and principles,” and “All community partners must work collaboratively to include strong parental engagement, blended and flexible funding, and shared success and liability.” The “Appendix” shows the highest rated items by cluster, as rated by importance and difficulty of implementation.

Results of the Rating Process on the Ranking of Clusters

After the group completed the sorting process, they rated the statements on a five-point scale for Difficulty of Implementation and Importance. The average Difficulty of Implementation or Importance rating for a cluster is the average of the statements within the cluster. It is the ratings of the items that determine the rankings of the clusters. Therefore, the clusters that contain more statements and higher averages are the clusters that were rated as more important or harder to implement. Table 1 shows the ratings of the clusters, in descending order, indicating the highest to the lowest average rating. Note that it is the rating of the items (action steps) within the clusters that form the basis for the ranking of the clusters.

Table 1 Cluster rating for importance and difficulty of implementation
Full size table

Table 1 indicates that, in general, ratings for Difficulty of Implementation were lower than ratings for Importance. This finding is not unusual, as groups frequently rate their issues in this way, when they consider the efforts involved in accomplishing the tasks they consider important. The participants rated Leadership and Network of Local Partners as very important and also the most difficult to implement.

On the other hand, they rated Plan to Expand Services and Evaluation as the least important and also the easiest to accomplish. Note that there is a small range for the rankings for Importance, indicating that all clusters are considered important.

It is noteworthy that the Family & Youth as Partners cluster was rated fifth of eight on Difficulty of Implementation, and therefore not viewed as the most difficult to implement. The score of 3.21 (on a five-point scale) suggests that the participants viewed this as a fairly easy set of items to implement. Note that items related to family and youth are well integrated in other clusters, as reflected in the list of items in the Shared Goals cluster above, which includes, “The community must demonstrate that child serving agencies have been meeting regularly along with family/youth participation to review children with serious emotional disturbances in their community and in need of more intensive community resources.” Overall, the information regarding the Family & Youth as Partners cluster conveys that family and youth are clearly a part of the overall design of systems of care and their involvement is not seen as something difficult to achieve. This position may be interpreted to reflect real progress on family and youth participation in systems of care.

Results of the Rating Process for Items; Comparing the Dimensions of Importance and Difficulty of Implementation

The 109 statements generated in the brainstorming session were rated on a five-point scale for Importance and Difficulty of Implementation. The statements receiving high ratings on Importance reflect higher priorities for action, and therefore the most important steps for action. However, if the concept of difficulty of implementation is also considered, the priorities for focusing action are changed, as they are tempered by what will require more effort. The areas that would be most essential to pursue, that is, the areas that will require the most attention and effort are those judged both important and difficult to implement.

A way to depict the most important and most difficult next steps is to employ a “focus zone” map as shown in Fig. 2. The “focus zone” map, which is part of the Concept Systems, Inc. data outputs, indicates which aspects of system change should receive the most attention during an implementation process. This method of mapping divides the items into four quadrants and displays the relationship between Importance and Difficulty of Implementation. The upper right quadrant is considered the major “focus zone,” and includes items that the participants considered to be most important yet the most difficult to implement. These are the items that will require the most attention in preparing the community to implement a system of care.

Fig. 2
figure 2

Plot of focus zones

Full size image

A non-pictorial way of presenting the focus zone information is to present the most highly rated statements for the two dimensions, Difficulty of Implementation and Importance. These are the statements in the upper right quadrant. Although there are numerous statements in this quadrant; only the most highly rated on both dimensions will be addressed.

To summarize the process, the items were derived from the brainstorming process, in which the participants identified 109 action steps. Using a five-point scale, group 1 participants indicated the steps that they thought were the most important and the most difficult to implement for developing a system of care. The two sets of ratings of these statements reflect the most important and most difficult to implement items, and translate into action steps for a community on which to concentrate their efforts for developing a system of care. When these two sets of ratings are combined statistically, the result is the action steps that reflect what is most important and most difficult to implement and thus represent the important areas in which to concentrate efforts. In reviewing the combined ratings, there appears to be a slight break at the rating of the top five statements for the combined scores of Difficulty of Implementation and Importance. Note that this list includes six statements because of one tied ranking. These statements are presented in Table 2.

Table 2 Five highest statements rated for difficulty of implementation and importance
Full size table

It is interesting to view the Importance dimension separately, as the community members’ ratings indicate what they consider the most important tasks in implementing a system of care, regardless of the difficulty of these actions. The leadership may wish to focus on the most important actions, that is, those that are both easy and difficult to implement. Table 3 displays the top five statements for Importance, which includes six statements, because of tied rankings.

Table 3 Five highest statements rated for importance separately
Full size table

In addition to determining those steps that are the most important and most difficult to implement overall, each cluster can be examined to determine the statements within that cluster with the highest ratings for Importance and Difficulty of Implementation. This analysis allows for a focus on action steps by cluster (domain), which may be useful as communities plan and implement their projects. For example, communities may develop committees to address the action steps for each cluster/domain.

Discussion

There are several aspects of this study that seem worthy of further discussion; these include the results, the method of collecting data, the limitations of the study, and next steps.

Results

The participants in this study of community readiness identified eight factors (clusters) that they considered essential for the development of systems of care: Leadership, Network of Local Partners, Shared Goals, Collaborations, Families and Youth as Partners, Accountability, Evaluations, and Plans to Expand Services. Within each of these factors, the participants also identified specific action steps (items) and rated these actions by their level of importance and difficulty of implementation. As the conceptual framework for the system of care has evolved, policy guidance, technical assistance, and training have focused on factors related to its successful development. It has become clear that much effort must be devoted to community transformation, focusing on community partnerships, families and youth as equal participants, individualized care, and culturally responsive services.

The eight factors (clusters) identified in this study that define community readiness are similar to the important factors of systems of care identified by Behar et al. (2005), Hodges et al. (2007a, b), Friedman et al. (2009), and Padgett et al. (2005). The similarities are reassuring that despite different methodologies and different purposes, similar descriptors have been obtained. Of these studies, the current study and that by Friedman et al. (2009) have produced findings derived from accepted statistical methods; the other studies have relied on intensive, high quality case studies, and the findings of each study have reinforced the others.

The identification of factors that define “community readiness” offers an important and practical contribution to improve the planning and implementation process for communities. Being able to understand from the very beginning what factors are important to the successful implementation of a system of care should help communities assess their own strengths and weaknesses, and address the areas of weakness. Further, such understanding could support the technical assistance efforts funded by the federal agency to better determine areas of focus for their technical assistance to the sites. This current study augments earlier work to fill an important gap in knowledge, and has used more advanced techniques to collect and analyze the data, providing information that depends less on inference and more on statistical analysis. The current study builds on earlier works and validates those findings by using measurable/quantifiable concepts and provides more new and useful information to understand and to assess community readiness.

The findings of the study can be useful to communities as they plan to develop systems of care, whether they are at the stage of writing an application for funding or in the early stages of implementation. The clusters that resulted from this study define the domains/factors where efforts should be directed. Within those domains, there are specific action steps (statements) that guide what needs to be done. The action steps are rated for how important they are to the successful implementation of a system of care and how difficult they are to implement.

Method

Although concept mapping, as a method of gathering information, has been used for decades, Concept Systems, Inc. has improved upon the process by developing a research-based methodology and software to analyze the data obtained. This approach is a “next generation” tool that uses sound methods of analysis of the data gathered from the participants, so that the end result is an unbiased and fair description of their input. Concept Systems, Inc. has also improved upon this method by developing a web-based method of gathering data, thus providing a cost-efficient technique for gathering information from a large group of people located in many different sites, in a short period of time. In this study, invitations to participate in the three tasks were issued to over 450 people to ensure that a broad representation of the children’s mental health field had an opportunity to respond and 285 provided data.

Limitations

The response rate for this study might be viewed as a limitation. For the three parts of the study, the response rate ranged from 42 to 53%, depending on the task. However, it should be noted that people from 25 of 27 invited programs responded, providing broad national representation of input. From each program, there were at least 2–3 respondents, rather than the 8 that were invited. The purpose of a broad-based invitation was to be sure that everyone felt included. A second purpose was to ensure that parents’ perspectives were obtained. The parents and parent coordinators at 12% were the third largest group represented; very close to the Outside Experts/Consultants (13%) and the Technical Assistance Coordinators (10%).

The response rate of 53% for the 36 of 68 members of the expert panel invited to sort the statements into groups/domains was also low, but sufficient. This group was much more homogenous; thus the demographic characteristics were less relevant. By the selection process, a cadre of “experts” was identified. These were all people who had studied systems of care, provided consultation and training, or were leaders of “graduated” sites, that is, those sites that had successfully completed their 6 years as demonstration projects. Data were gathered over the summer vacation months from groups 1 and 2; thus a lower rate of response was anticipated.

The actual number of participants is sufficient, even robust, according to Concept Systems, Inc., for this methodology. Trochim (1993), in summarizing meta-analyses of 38 projects, reported an average of 14 sorters and raters in each project. The large number of people invited to participate improved the likelihood of obtaining responses from a wide demographic group representing all levels involved in implementing a system of care. As discussed in the Results section, the demographic characteristics of the study sample indicate good representation of parents and professionals from varying levels of employment. The numbers of participants in this study exceeded what Trochim (1993) indicated are required for a sound concept mapping study.

A second limitation was the somewhat low response to the demographic questions, with 22% of the sample not completing these items. The absence of this information precluded understanding the characteristics of the entire group that responded. Of course, with this information missing for those who chose not to reply, a comparison could not be made of those who refused the invitation with those who accepted. The investigators believed that promising anonymity would increase the number of respondents, while understanding that anonymity would also make identification of those who refused more difficult.

Next Steps

The next steps are to assess the importance or impact of these factors by studies designed to empirically test these factors and then to refine the action statements to provide a basis for community assessment, that is, a rating scale for community readiness. Such an instrument would allow a large number of community stakeholders to rate their community’s readiness to develop a system of care, whether they are in the pre-application stage, or in the stage of being funded and in the planning phase. Once the community stakeholders assess their readiness, the resulting information of their strengths and weaknesses could provide direction for their implementation efforts and for technical assistance efforts. A follow-up rating after 10–12 months, using the same rating scale, would reflect their progress especially in areas of weakness. These data could also be compared with other outcome measures of the success of funded sites to determine the predictive value of readiness to overall success. These could be important contributions, given the high priority and major investment that the federal government has placed on the development of systems of care to serve children and adolescents with serious emotional disturbances and their families.

The current study was designed to define the important elements of community readiness for projects focusing on transformation of the ways services to children with mental health disorders are delivered to them and their families. This study offers a view of how the advanced method of concept mapping can be applied to other initiatives that focus on change, and to directly involve stakeholders in the process of defining actions and building consensus for the change process.