Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Conducting Large-Scale Evaluation Studies to Identify Characteristics of Effective Comprehensive School Counseling Programs

For several decades now, the call for heightened educational accountability continues to grow from multiple camps and key stakeholders, including policy makers and legislators as well as parent/caregiver and educator groups. Many prominent school counseling leaders and researchers also championed this position, advocating for results-based programming and stricter professional accountability (e.g., Atkinson, Furlong, & Janoff, 1979; Aubrey, 1984; Borders & Drury, 1992; Dahir & Stone, 2003; Erford, 2015a; Gysbers & Henderson, 2012; Myrick, 1984; Trevisan, 2001). Although accountability research has been conducted for decades (e.g., Kranzler, 1968; Tamminen & Miller, 1968), small- and large-scale evaluation studies are considered an essential method to appraise the utility and cost-effectiveness of educational programming and associated interventions (e.g., Carey & Dimmitt, 2008; Erford, 2015b; Lee & Goodnough, 2015; Sink, 2009). As discussed in a later section, since the 1990s , a series of state-level evaluation studies investigating the efficacy of comprehensive school counseling programs (CSCPs) have been published in respected journals and other scholarly venues in an attempt to meet these long-term objectives : (a) to develop consistent and meaningful policies supporting CSCP design, implementation, and revisions, (b) to discern which counseling- and program-related practices are most beneficial for the preponderance of students and their families, and (c) to detect counselor competence in facilitating program services, activities, and interventions (Gysbers & Henderson, 2012). Given their range, the considerable expense, and the logistical challenges of these CSCP evaluations, the number of extant studies is understandably sparse to fully accomplish these purposes. As such, interested readers must glean from these investigations the most consequential findings, interpreting them cautiously, particularly in light of their research caveats.

With this chapter, we intend to raise awareness of statewide CSCP evaluations and their value to policymakers to identify structural and operational characteristics of effective school-based counseling programs. To do so, we first review three major orientations to comprehensive school counseling programs in the USA , outlining their theoretical/conceptual underpinning, key components, and issues related to program evaluation. Second, survey research methodologies are elucidated. Third, 11 state-level studies are summarized, focusing on their research methods and implications for CSCP policy . We identify areas of coherence and disagreement as well as call attention to the major research limitations. In closing, applications to advancing knowledge of CSCP efficacy and to policymaking and recommendations for future research are explored.

Conceptual Underpinnings and Componential Characteristics

The inklings of a more structured, systemic, and proactive orientation to guidance counseling – now referred to as professional school counseling – emerged, in various forms, as far back as the 1950s and 1960s. Guidance leaders and professionals at that time expressed concern that the existing approach to support students, namely, the reactive pupil personnel services (i.e., position focused) model, was largely ineffective (Ballast & Shoemaker, 1978; Dinkmeyer & Owens, 1969; Gysbers & Henderson, 2012; Hoyt, 1965; Tamminen & Miller, 1968). Taking its cue from these sources and researchers, leadership in the American School Counselor Association (ASCA, 1974) issued a policy statement suggesting that the profession change course. Despite the lack of training available to school counselors, ASCA policy and professional publications encouraged school counselors to implement guidance and counseling program in their districts. Albeit oversimplified, these varying perspectives lacked a well-articulated, systemic approach to school counseling work (Cinotti, 2014), opening the door in the 1970s and 1980s for two major and relatively similar conceptual orientations to CSCPs to materialize. Literature generally refers to them as the Life Career Development Model (Gysbers & Henderson) and Developmental Guidance and Counseling Model (Myrick, 2011; see Brown & Trusty, 2005 and Schmidt, 2008, for a summary of the historical roots of the school counseling profession and CSCPs). By the turn of the century, features of each were modified and distilled into a third organizational framework, the ASCA (2003, 2012) National Model. To orient the rest of the chapter, we comment on the frameworks’ defining characteristics, including their conceptual or theoretical grounding as well as those configural features that are most amenable to program evaluation and accountability research.

Life Career Development Model

The most well-known and researched framework was developed by career guidance scholar Norman Gysbers and his colleagues (e.g., Gysbers & Moore, 1974, 1981, 1988; Gysbers, 1990; Gysbers & Henderson, 1988, 2012; Mitchell & Gysbers, 1978). By 2000, Gysbers and Henderson’s systemic approach to school guidance and counseling was established in most, if not all, Missouri school districts (Missouri Department of Elementary and Secondary Education, 1998) and adapted and employed across numerous states and individual school districts and buildings (MacDonald & Sink, 1999). The framework has been implemented internationally as well (e.g., Lee, Suh, Yang, & Jang, 2012). As a matter of record, the Missouri framework was originally referred to as a career guidance program. In time, this designation was expanded to Missouri Comprehensive Guidance Program (MCGP; e.g., Lapan, Gysbers, & Sun, 1997), suggesting to policymakers and educators that students ought to receive a broader range of school-based counseling services, including career, personal-social, and educational assistance.

Since the Life Career Development Model as operationalized by the MCGP is fully documented in the literature (see, e.g., Gysbers, 1990; Gysbers & Henderson, 2012; Gysbers, Lapan, Blair, Starr, & Wilmes, 1999; Gysbers, Stanley, Kosteck-Bunch, Magnuson, & Starr, 2011; Starr & Gysbers, 1993), we need to only summarize its basic premises, characteristics, and components. First, the model’s hallmarks include: (a) program content involving measureable developmental standards or competencies designed to maximize positive student functioning; (b) a clear organizational configuration, including structural components (program definitions, assumptions, and rationale), program components (classroom guidance, individual student planning, responsive services [e.g., 1:1 and small group counseling], and system support); (c) adequate resources (personnel, financial, and political); and (d) process elements (e.g., program development, management, and accountability activities). In fact, the evolution of a fully functioning district-wide CSCP, like the MCGP , requires upward of 5 or more years and progresses through a clearly specified sequence of overlapping phases: planning, designing, implementing, evaluating, and revising (Gysbers & Henderson, 2012). In short, an effective school counseling program should be developmental in nature, systemic, as well as comprehensive (K-12) in scope and structure. They should also be standards-based, highly collaborative, and lead by advocates for change (Cinotti, 2014; Gysbers, 1990).

Gysbers and Henderson’s (2012) theoretical grounding looks to be a confluence of vocational guidance and career counseling theory, systems thinking, and well-accepted principles of human development. Regrettably, there are only a few indications of how these influences informed their model. Specific theories and their originators are largely absent from the narrative. However, the program’s conceptual-organizational scaffold and its four central components (guidance curriculum, individual student planning, responsive services, and systems support) are well delineated. Sharply put, specific foundational theories and their contributions to the program were given short shrift in lieu of useful discussions outlining its conceptual framework, structural components, and implications for school counselor practice.

Returning to the subject of program evaluation, Gysbers and Henderson (2012) made this area one of the framework’s mainstays, particularly as an avenue to foster program improvement and school guidance personnel accountability . Regular appraisal of program outcomes is thought to influence policymaking from the local to the national level (Gysbers, Lapan, & Jones, 2000). What makes the Missouri Guidance and Counseling Model so appealing to researchers and practitioners are its measureable program outcomes and well-thought out counselor and student competencies. For counselors conducting action research with the Missouri Model, its multiple user-friendly assessment tools are commendable. Nevertheless, the lack of theoretical detail renders the model challenging for researchers to evaluate the guidance program’s theoretical fecundity and fidelity. Without explicit documentation of the model’s theoretical foundation, gauging its real-world operationalization and application is unnecessarily problematic (MacDonald & Sink, 1999). Policymakers and model administrators are at a loss, especially when considering programmatic and practical revisions that are aligned with various theoretical constructs.

Developmental Guidance and Counseling Model

Around the same period that Gysbers and his collaborators were formulating their guidance model, a number of school counseling scholars affirmed the value of incorporating developmental theory and research into school-based counseling activities and services (e.g., Blocher, 1968; Dinkmeyer, 1966; Dinkmeyer, Dinkmeyer, & Caldwell, 1970; Gum, 1979; Gum, Tamminen, & Smaby, 1973; Peters & Farwell, 1959; Sprinthall, 1972, 1974; Super, 1964; Tuma, 1974; Zaccaria, 1965). Although Gysbers and Henderson (2012) took into account developmental psychology, other school counseling scholars (e.g., Dinkmeyer, 1966; Gum, 1969; Myrick, 1989, 1993, 2011; Paisley & Benshoff, 1996) were far more explicit in their discussions, citing recognized developmental theories (e.g., Erikson, Kohlberg, Piaget, Super) and tasks (e.g., Havighurst, 1953) that should guide all facets of comprehensive school counseling programs. In fact, ASCA’s (1979) governing board embraced these ideas characterizing developmental guidance as follows:

… is that component of all guidance efforts which fosters planned intervention within the educational and other human development services programs at all points in the human life cycle to vigorously stimulate and actively facilitate the total development of individuals in all areas—personal, social, emotional, career, moral-ethical, cognitive, aesthetic—and to promote the integration of the several components into an individual lifestyle. (as cited in Myrick, 2003, p. 83)

By the early 1990s, the developmental approach to comprehensive school counseling was firmly established in the literature and not much later integrated into elementary school counseling practice (Borders & Drury, 1992; Gum, 1969; Myrick, 2003, 1997, 2011; Paisley & Peace, 1995). Myrick’s (2011) well-known textbook and its previous editions are perhaps the most complete summary of this CSCP orientation, stressing both theoretical and practical dimensions. In contrast to Gysbers and Henderson (2012), Myrick’s model operationalization is more loosely structured and detailed. That is to say, as an organizational framework for systemic practice, Myrick’s developmental guidance and counseling approach and its overall conceptual blueprint are less than fully explained. More specific guidance and counseling activities and services, however, are coherently presented and practitioner oriented.

A developmental guidance and counseling orientation includes a number of essential features (e.g., Myrick, 2003, 2011; Paisley & Peace, 1995). Similar to Gysbers and Henderson (2012), it rests upon well-researched and documented developmental assumptions, needs, and principles (see Myrick, 2003, 2011, for an extensive overview). The developmental guidance component, reflecting ASCA’s (1979) position statement quoted above, is for all students, involves all school personnel, has an organized and planned curriculum, and is integrated into total educational enterprise (Myrick, 2003).

Perhaps most importantly in terms of program evaluation, development guidance and counseling supports more effective and efficient student learning. That is, guidance and counseling personnel are encouraged to understand students’ cognitions, emotions, and behaviors through lens of developmental stage theories (e.g., cognitive, moral, psychosocial, and career). In particular, all humans progress at varying rates through hierarchical phases of growth, with each stage having its own developmental tasks to accomplish.

Myrick (2003) also proposed eight developmental goals that characterize almost all developmental guidance and counseling programs. Generally stated these include students learning to navigate the school environment, understanding themselves and others, and knowing about their attitudes and the impact of their behavior. Other goals speak to community involvement and participation in career and educational planning. From these goals, general and specific student objectives are structured.

Finally, the developmental approach focuses on the expanded role and functions of school counselors. As developmental specialists, counselors are to provide six basic interventions (called responsive services in Gysbers & Henderson, 2012), including individual and small group counseling, large group guidance, peer facilitator training, consultation, and coordinator of guidance services. Moreover, school counselors should conduct accountability studies to measure the impact of their work on the students they serve.

It is important to reiterate here one of the key differences between the Missouri Model (Gysbers & Henderson, 2012) and the one depicted by Myrick (2003, 2011) and other developmentalists; the latter approach explicates the developmental features in considerable depth, making a program evaluation of the model’s theoretical underpinnings more feasible. However, Myrick’s developmental guidance and counseling model has only a skeletal organizational structure, making program audits more complicated. As with Missouri Model, this approach emphasizes the attainment of multiple student goals, operationalized by specific psychosocial, career, and educational objectives. As such, program evaluators looking at outcomes from both models should be able to at least detect significant improvements in student functioning, particularly between those students attending schools with comprehensive school counseling programs and those pupils who do not (e.g., Sink & Stroh, 2003).

ASCA National Model : A Framework for School Counseling Programs

As referred to earlier, ASCA leadership supported the implementation of comprehensive guidance and counseling at least as far back as the 1970s. In 1997 the developmentally based ASCA National Standards for student behavior were published (Campbell & Dahir, 1997). These provided overall standards, competencies, and specific benchmarks for “optimal” student functioning within these domains: personal-social (social-emotional), career-vocational planning, and educational behavior. Subsequently, in an attempt to create a unified national identity for the school counseling profession, ASCA, with substantial input from school counseling scholars and leading practitioners, produced its own framework in 2003, merging primarily elements of the Missouri Model, as described, in large part, by Gysbers and Henderson (1998, 2000) and some elements of the developmental guidance and counseling model (e.g., Dinkmeyer et al., 1970; Myrick, 2003). Brown and Trusty (2005) aptly compared an early version of the National Model to the Gysbers and Henderson framework, reporting only modest differences. For the sake of brevity, only the latest iteration of the National Model (ASCA, 2012) and its defining attributes are reviewed.

Moving away from Gysbers and Henderson’s (e.g., 1998) conceptual scaffold, framers of the ASCA (2012) Model created its own visual depiction and intentionally streamlining program documentation and descriptions. Using a four quadrant diamond shape, the model has these components: (1) foundation (underlying program focus and student [ASCA Student Standards] and professional competencies [ASCA School Counselor Competencies]); (2) management (assessment tools, assorted working documents, etc.); (3) delivery, including direct (e.g., counseling, individual planning, guidance curriculum) and indirect services (e.g., referrals, consultation) on behalf of student and their families; and (4) accountability (e.g., analysis of school profile and use of time, program outcomes, evaluation of counselor competencies, overall program). As a benefit for program evaluators, the National Model requires school counselors to be data driven and evidence based, offering practitioners numerous informal tools to measure specified student and program outcomes.

Similar to the Life Career Development Model as depicted in Gysbers and Henderson (2012), ASCA’s (2012) school counseling framework is a practical document, replete with counselor-focused information. Curiously, after reviewing the three editions of the National Model, each refers to a theoretical basis without providing evidence to support it, citing no specific theories or theorists. For instance, Henderson (2012), who was charged with writing the section for the most recent version, provided a list of questions and associated ASCA Model Principles, suggesting that these describe “the theory base” (p. 137). One assumes, perhaps, that these principles rest upon specific theories but, again, they are not referenced, nor are they available on ASCA’s website. As a program evaluator or accountability specialist, these principles might become testable if they were converted into specific, measurable, attainable, and realistic goals. Put another way, these “theoretical” principles must have concrete criteria for measuring progress toward meeting them as goals.

As an aside, a careful reader of ASCA (2012) National Model will note the National Standards (ASCA, 1997; Stevens & Wilkerson, 2010; also called the ASCA Student Standards) are not included in the manual. These have been discarded and in their stead the ASCA (2014) Mindsets and Behaviors for Student Success are now to be used. To ASCA’s (2014) credit, the writers of the new standards incorporate a large array of college and career readiness research, standards, and best practices particularly as they relate to improving student academic outcomes. Moreover, the current standards are noticeably focused on academic and career development. Of the many sources referenced for the development of the Mindsets and Behaviors, only two were directly linked to research advances in student social and emotional learning and other noncognitive factors influencing academic performance. For evaluators and accountability researchers, the mindset and behavior standards have two significant downsides. First, at present, there are no explicit benchmarks to ascertain whether students are actually meeting the standards and assessment tools are noticeably absent. Second, the standards are only vaguely connected to the National Model (2012). In other words, it is not entirely clear which aspects of the National Model address specific mindset and behavior standards.

Introduction to Large-Scale Evaluation Methods and Analyses

Before specifically addressing the statewide CSCP program evaluations, a brief recap of three major survey designs utilized in these studies is provided.Footnote 1 First, most basic survey approach is descriptive in nature. In this design, survey data (numerical or narrative or mixed) are collected from respondents in hopes that they accurately characterize or portray participants’ actual attitudes, beliefs, or views related to measureable construct or constructs (e.g., motivation to achieve, empathy). Ideally, prior to the survey’s distribution to a wider and representative sample, the instrument should be examined for test reliability and validity and piloted with a small developmental group. Since inferential statistics are not used in basic descriptive investigations, there should be no attempt to generalize the findings to a wider population. With numerical data, descriptive statistics are generally reported, including measures of central tendency (e.g., M, Mdn) and score variability (e.g., SD, range, kurtosis, skew). Moreover, it is commonplace to depict the numerical data as frequency tables and bar graphs with error bars. Qualitative narrative data can be summarized in overarching themes with representative quotes from respondents.

A second survey method commonly used in statewide program evaluations is referred to as a correlational design. Fundamentally, the aim of such a program evaluation is to examine potential relationships among salient variables, such as respondent demographics (e.g., gender, age, grade level, socioeconomic status, ethnicity) and pertinent outcome measures (e.g., grade point average, test scores, motivation, career attitudes, social and emotional variables). This approach incorporates aspects of basic descriptive survey studies, adding another layer of complexity, namely, the use of inferential statistics. Depending on the research questions posed, after reporting the key descriptive statistics, evaluators may choose to only analyze the data using bivariate measures of association (e.g., Pearson’s r) and ordinary least squares regression or opt for advanced multivariate tools, such as canonical correlational analysis, exploratory and confirmatory factor analysis, multiple linear regression (e.g., hierarchical linear regression [HLR]), logistic regression, and hierarchical linear modeling (HLM).

Some of the correlational designs used in statewide CSCP evaluations utilize higher-order correlational/regression methods. Specifically, HLR was deployed in two studies (Carey, Harrington, Martin, & Hoffman, 2012; Carey, Harrington, Martin, & Stevenson, 2012) and HLM in two Missouri evaluation studies (Lapan, Gysbers, & Sun, 1997, Lapan, Gysbers, & Petroski, 2001). HLR is a relatively common multiple regression procedure used to statistically control for the influence of certain predictors variables on the criterion variable, HLM is less well deployed and understood. As such, it warrants some explanation. Albeit oversimplified, Lapan and his colleagues deployed this statistical procedure to analyze hierarchical, or nested, data structures (e.g., data collected from teachers, nested within schools, nested within school districts) (Osborne, 2000). In other words, large-scale evaluation studies typically gather survey data from students, teachers, and caregivers who are not randomly selected from the larger population. Moreover, student and teachers are not randomly assigned to classrooms. When data sets are then aggregated across classrooms, schools, and districts, any group comparisons made on various dependent variables will be problematic, for the assumption of independence of observations is violated. Namely, cross-level data, if not properly dealt with, confounds the survey results, potentially under- or overestimating the observed relationships between variables. In short, HLM allows researchers to test possible associations among, for example, student outcome variables, by statistically controlling for the nesting problem.

Several of the evaluation studies summarized below used a causal-comparative or ex post facto research design (e.g., Sink & Stroh, 2003; Sink, Akos, Turnbull, & Mvududu, 2008). Similar to a correlational design , this quantitative approach incorporates elements of a basic descriptive evaluation study but also attempts to measure potential respondent group differences on various outcome variables following intervention (i.e., CSCP implementation). Primarily due to logistical reasons, there is frequently limited or no pretesting or the collection of participant baseline data prior to CSCP implementation. As a consequence , internal validity can be compromised. Data from outcome instruments (“posttests”) are gathered from students in the CSCP intervention group and from those in non-CSCP schools. Again, based on the research questions, potential group differences and interaction effects are tested using appropriate inferential statistics, ranging from independent or paired t-tests to variations on the multivariate analysis of variance (MANOVA) procedure.

Although the term causal-comparative is used with this third research method, cause and effect cannot be firmly established unless the investigator designs a large-scale experimental or quasi-experimental evaluation study that at least minimizes threats to internal and external validly. Because such a rigorous design is so challenging to conduct in state-controlled and funded schools, none of the 11 studies reviewed below used a quasi- or true experimental design. Should such an approach be possible in the future, evaluators would attempt, at a minimum, to include experimental and control groups (e.g., students, parents, caregivers, teachers) randomly selected from large representative populations. Random assignment to groups would also be optimal as well. To establish baseline data, the participants would be “pretested” (i.e., before the implementation of a school- or district-based CSCP) on desired outcomes. Subsequently, after a sufficient period of time (e.g., 6 months to 1 school year) for the benefits of CSCP implementation to accrue, the survey(s) would be readministered. This posttesting could continue at regular intervals. Germane student-related outcome data could be maintained and collected from school and state records (archival data), allowing for supplemental group comparisons.

Although program evaluation standards for methodological quality are available from a variety of sources, they are not altogether in agreement. Farrington’s (2003) five standards are both readily interpretable and comprehensive and thus applicable to our analysis of the statewide CSCP studies. Beyond internal validity (i.e., relates to the methodological rigor of the study and controlling for confounding variables) and external validity (i.e., generalizability of possible causal relationships and operational definitions of interventions and outcomes), three other standards should be considered. Quantitative evaluation research should also possess adequate (a) construct validity (i.e., adequacy of the operational definition and measurement of the theoretical constructs that underlie the intervention and the outcome variables), (b) statistical conclusion validity (i.e., effect sizes are reported as a way to estimate whether the presumed cause [CSCP implementation] and the presumed effect (survey respondent outcomes) are related), and (c) descriptive validity (i.e., refers to the adequacy of the presentation of the major elements of an evaluation research report). In the research caveats section below, we consider these standards in more detail.

Summary of Statewide Evaluations

With above context in mind, 11 statewide CSCP evaluations are summarized, elucidating primarily their methodological quality, rigor, and reach. The more obvious commonalities, differences, and limitations with reference to program evaluation research are briefly considered as well. A thorough critique of specific research findings is beyond the scope of this chapter. Instead we later explore the manner in which the most germane results can be positioned to influence policymaking at the national, state, and local levels. Readers interested in an extensive summary and analysis of the state-level results will want to consult Carey and Martin (2015).

It should be mentioned that the school counseling profession has a relatively lengthy history of conducting accountability and program evaluations (Gysbers & Henderson, 2012). Several lesser known studies have investigated various dimensions of guidance programs and counseling services (e.g., Kranzler, 2015). For instance, in Minnesota a comprehensive statewide evaluation of guidance functions and their impact were examined (Tamminen & Miller, 1968), reporting negligible positive student outcomes. Not unexpectedly, the most effective guidance input concerned the quality of interactions between school counselors and students and their colleagues. The most meaningful and helpful relationships produced favorable guidance-related outcomes. Mirroring in part the research findings reported decades later (e.g., Lambie & Williamson, 2004; Ockerman, Patrikakou, & Feiker Hollenbeck, 2015), the Minnesota investigation concluded that counselors ought to devote far less time delivering reactive student services. In their place, guidance personnel need to spend more time promoting and facilitating the development of the whole student. Even though these early evaluations were not altogether exacting in design, limited in its purview, and insufficiently documented, they did set a clear precedence for school counseling researchers to engage in program evaluations and, in turn, develop evidence-based practices.

The statewide program evaluations reviewed below are broad based in nature, mostly correlational and ex post facto in design , and assess distal noncausative effects (e.g., overall improvement in grades for high school students, percentage increase in attendance of children designated as being “at risk” for school failure). By approximately 2010, ten states had developed program evaluation systems with only a few requiring stringent standards, processes, and procedures (Martin, Carey, & DeCoster, 2009). At present, we found 11 state-level studies that merited a closer examination. Numerous other accountability studies have been conducted in the USA, but only those that reached these criteria were included here: statewide in reach, demonstrated a reasonable level of methodological quality, and published in relevant peer-reviewed journals.

Commonalities and Differences

Table 10.1 summarizes these investigations utilizing various categories: (a) state where the accountability study was conducted and its authors, (b) the school counseling program model(s) or framework(s) addressed, (c) methodological issues (research design, sample, sampling, and instrumentation) deployed, (d) most salient findings, and (e) major research limitations. Three studies were conducted with Missouri secondary school students using dimensions of the Missouri Comprehensive Guidance Program (MCGP) as their conceptual framework. Two Washington state investigations were published reflecting an assortment of comprehensive school counseling program frameworks; one was elementary focused and the other sampled middle school counselors and students. The authors of the two statewide evaluation studies conducted in Nebraska reported that the MCSP and ASCA National Model were their organizational blueprints. While the 2005 study was directed toward K-12 counselors’ opinions, the 2012 investigation surveyed only high school counselors. The remaining state-level studies published in 2012 were conducted in Connecticut, Rhode Island, Utah, and Wisconsin, evaluating for the most part high school counselors’ perspectives on their respective comprehensive school counseling program. Furthermore, the investigations relied on self-report data, canvasing school counselors either using an investigator-constructed measure or an established school counseling program survey. Some archival data were used in most of the studies. Two Missouri and both Washington state evaluations applied some type of random sampling procedure, with the remainder trusting convenience and purposeful sampling techniques.

Table 10.1 Summary of publish statewide investigations of comprehensive school counseling programs
Full size table

Notable Results

Given the varying research methods deployed in the state-level studies and their inherent complications, as well as other more specific limitations (see below), any overarching conclusions derived from the findings must be cautiously averred and judiciously viewed. Overall, each of the studies showed to some extent that counselors who reported working in CSCP schools appeared to be contributing positively to student development and school climate. In particular, across the 11 investigations that were mainly secondary school focused, certain CSCP elements were associated with improving student outcomes in the academic/educational and career domains. Counselors repeatedly suggested that skill development activities were beneficial. Some of the program evaluation outcomes corroborated these perceptions. Whether these encouraging gains extended to improving students’ social and emotional functioning is questionable. However defined, schools with more fully implemented CSCPs and heightened program-based training for school counselors seem to be yielding the largest improvements in various outcomes (e.g., Sink & Stroh, 2003). Regrettably, the manner in which these beneficial practices were operationalized and measured at the school level was inconsistent. Lastly and perhaps most significant for policymakers, lower school counselor to student ratios in CSCP schools may contribute to better program and student outcomes (Carey, Harrington, Martin, & Hoffman, 2012; Carey, Harrington, Martin, & Stevenson, 2012).

In summary, among 11 studies reviewed, two Missouri (Lapan et al., 1997, 2001) and two Washington state studies (Sink & Stroh, 2003; Sink et al., 2008) were most rigorously designed and, as such, generated the fairly robust results. Each deployed some type of random sampling and surveyed a large sample using multisources (students, teachers, and school counselors), as well as attempted to control for confounding variables using multivariate statistical procedures (multivariate analysis of covariance [MANCOVA] or HLM). A discussion of the research limitations are provided next .

Research Caveats

As alluded to previously, and certainly not unique to these studies, each statewide evaluation was methodologically flawed in some respects. The most glaring caveats should be considered, especially as they confound research findings and limit policymakers’ ability to make evidence-based decisions and recommend program enhancements. First and foremost, in terms of Farrington’s (2003) Methodological Quality Standards for Evaluation Research, each investigation failed to meet all rigorous standards. Construct validity was jeopardized in all the studies because CSCPs’ theoretical underpinnings are largely under-described and operationalized. Furthermore, given their lack of methodological precision, the survey studies generally possessed deficient internal validity. Relatedly, descriptive validity was marginal at best, for several studies grossly underreported sample demographic characteristics, deployed less than optimal sampling procedures, and included anomalies in the data collection process. Socioeconomic status, gender, and ethnic group findings were inconsistently expressed as well. Moreover, where effect sizes could have been generated in the studies, they were not included, reducing statistical conclusion validity. Finally, the external validity standard was compromised in those studies deploying nonprobability sampling. In these evaluations, the representativeness of the respondent samples to the general populations was equivocal. Only four studies with random sampling could external validity be relatively assured.

Other notable research limitations decreased the validity of the findings. For instance, researchers depended largely on counselors’ perceptions of their programs and practices. Self-report data by their very nature are commonly seen as biased, reflecting to a degree social desirability responding. It is understandable that counselors want to be perceived as comprehensive school counseling advocates and supporters, but, as a consequence, they may have overestimated the degree of CSCP implementation. Curiously, to triangulate school counselor and teacher perspectives, student, parent/caregiver, and other stakeholder opinions were seldom reported. The studies were also fraught with potentially confounding variables (i.e., unaccounted sources of variance in student, school, and school district outcome variables), making their findings less than definite.

In conclusion, as school counseling leaders aim to establish new or modify existing CSCP policies, any decisions in this regard should be founded on empirically based research, involving at some level high-quality experimental or quasi-experimental research (Creswell, 2014; Farrington, 2003). As large-scale school-based evaluations apparently cannot, at present, meet this robust standard due to a myriad of district-related logistical concerns as well as various sociopolitical and ethical barriers, investigators had to settle for designs that were descriptive, correlational, causal-comparative, or a combination of one or more. With such methodologies, researchers were only able to posit tentative conclusions .

Implications for Policy-Related Initiatives

Carey, Harrington, Martin, and Stevenson (2012) reiterated the sentiments of earlier researchers (e.g., Borders & Drury, 1992), insisting that state-level program research is beneficial in several respects. For instance, such investigations are able to detect needed CSCP enhancements, leading in turn to profession-wide projects to address minor and serious gaps in program elements and services. The evaluations offer tangible evidence; these programs are advantageous for students and their families. Furthermore, program renewal , on a regular basis, is essential for sustaining effective holistic and individualized school counseling services in response to shifting school demographics and community needs. Put differently, revisiting program outcome data on a regular basis not only allows school counselors to be more responsive and adaptable to stakeholders concerns, it also affords policymakers with current data to inform their decision-making. Without evidence-based practices, professional advocacy efforts may be less likely to effect change in school-based counseling policy (Borders & Drury; Galassi & Akos, 2004; Sink & MacDonald, 1998).

Each of the reviewed program evaluations incorporated salient recommendations that have implications for CSCP policies and their enactment. State and district guidance and counseling leadership would do well to heed this input, particularly as a way to revitalize and amend substandard program infrastructures, procedures, processes, and practices. For instance, several state-level investigations (Missouri and Washington; e.g., Lapan et al., 1997; Sink & Stroh, 2003; Sink et al., 2008) indicated, among other recommendations, that fully implemented CSCPs, however operationalized, were positively correlated with better efficacy-related outcomes. State and local leadership teams , as a result, must ensure that the CSCP policy documents applicable to program implementation, management, and evaluation practices underscore the value of increased fidelity to the model’s conceptual framework and established guidelines for practice. The Utah study (Carey, Harrington, Martin, & Stevenson, 2012) also generated specific recommendations with policy implications, among them composing a State Superintendent of Education-generated School Counseling and Career and Technical Education policy statement to be remitted to all school districts and notifying a statewide interdisciplinary committee with managerial authority to improve the Grade 12 experience.

For those school administrators involved with CSCP management in Nebraska, Rhode Island, Wisconsin, and Connecticut, the evaluators also proposed several modifications to improve policy implementation and assessment. For example, whereas Barnes, Scofield, and Vrbka (2005) advised Nebraskan CSCP leadership to reinforce various system support features in many of the state’s schools, Carey, Harrington, Martin, and Hoffman (2012) intimated that (a) career and technical education (CTE) needs to be integrated into district-level programs aligned with the ASCA National Model, (b) CTE practices should be routinely inspected, and (c) standardized measurement tools are needed, particularly when evaluators appraise and compare stakeholder and program outcomes across Nebraska’s school districts and between state frameworks. Following their analysis of Rhode Island state CSCP data, Dimmitt and Wilkerson (2012) concluded that policymakers must address the disproportionality in program delivery. Namely, higher percentages of students receiving free or reduced-price lunch and from traditionally underserved racial/ethnic groups were not experiencing the level and perhaps the quality of CSCP services as those students attending economically advantaged CSCP schools. Implementation inequity is a social justice issue for organizational leadership to closely attend to and institute requisite changes. After surveying Wisconsin high school counselors, Burkard, Gillen, Martinez, and Skytte (2012) indicated that state and district CSCP policies and their applications should reflect (a) the priority of equal implementation of program services and components across schools, (b) the need for additional uniform school counselor CSCP training, and (c) the importance of program evaluation and accountability work. Finally, the Connecticut study has major implications for policy development. The study emphasized, for example, the importance of lower high school counselor caseloads, for lower ratios were associated with fewer student behavior infractions (Lapan, Whitcomb, & Aleman, 2012). Next, various proposals for enhancing statewide CSCP evaluation ventures particularly as they may influence policy development and implementation are supplied.

Recommendations

From the previous discussion, state-level CSCP evaluation efforts are largely generating favorable student outcomes. Whether they are influencing state CSCP policies and procedures is an open question. Whatever the case may be, school-based counseling experts must continue to advocate for improvements in efficacy and accountability research. The suggestions presented here should be viewed as talking points for state and local educational leaders, counselors, counseling scholars, program administrators, ASCA leadership, and other key stakeholders. First, before CSCPs can be more fully evaluated, their underlying theoretical orientations and constructs need to be explicated in finer detail (MacDonald & Sink, 1999). With the ASCA (2012) National Model, for example, program framers appear to be conflating philosophical assumptions with theoretical statements. Researchers must understand how the varying CSCP components reflect evidence-based theories.

Second, a national coordinated research effort is required to guide large-scale program evaluation work. Whereas the Center for School Counseling Outcome Research (University of Massachusetts, Amherst) has admirably led the profession in this regard, ASCA as a national organization has yet to fully contribute to this initiative. Should these organizations and others like them establish a viable research consortium and closely coordinate their efforts and funding, higher-quality program evaluation studies should result. Additionally, state-level private school CSCP data should be collected. Findings from both private and public schools would then better inform existing and future program policies and practices. In short, improved research collaboration among relevant organizations will enhance policymaking and program efficacy.

A top priority of a coordinated national evaluation initiative should include a systematic appraisal of the latest version of the ASCA (2012) National Model and its impact on students and schools (Martin & Carey, 2014). The Utah and Nebraska investigations of a previous version of the National Model reported on the educational and career planning benefits to high school students (Carey, Harrington, Martin, & Hoffman, 2012). However, for reasons discussed previously, these hopeful findings should be viewed with some circumspection. Researchers must continue to substantiate the value of the National Model’s program implementation, management, and accountability strategies. Moreover, efficacy data remain sparse as to whether the National Model or other state frameworks increase students’ social and emotional functioning and the extent to which these programs improve school and classroom climate. Finally, subsequent evaluations of the National Model must account for differences in student demographics, particularly racial/ethnic and socioeconomic disparities between school communities.

Although cross-sectional survey studies using descriptive, correlational, and causal-comparative designs have their place, forthcoming evaluations should utilize more rigorous and longitudinal research designs and data analysis procedures that attempt to meet Farrington’s (2003) quality standards. For example, surveys used as outcome tools should be reliable and valid, sampling and data collection procedures should be standardized, and the reporting of findings should be enhanced. Whenever feasible probability sampling, and at least, quasi-experimental designs with control or comparison groups should be utilized. Broadening the participant pool to preK-12 students, parents/caregivers, teachers, administrators, and support staff is critical. Mixed methods are also extremely useful, incorporating qualitative data into the evaluation process (e.g., Martin & Carey, 2012). Effect sizes should always be reported with significant statistical findings.

Complex as they are, advancements in statistical methods allow for more effective ways to analyze large data sets. Multivariate statistical procedures such as multidimensional scaling, cluster analysis, multilevel linear modeling, and structural equation modeling are important tools to analyze complicated data sets (Tabachnick & Fidell, 2013). Logistic regression can be used when the evaluator wants to categorize schools as either CSCP or non-CSCP buildings using categorical data.

Finally, state-level program research should employ an evaluation framework to guide the process. Numerous school counseling-based approaches exist, each with own their strengths and aims. The frameworks emphasize results-based evaluations, assessing in systematic ways various CSCP components and their impact on students and other stakeholders (e.g., Astramovich & Coker, 2007; Eschenauer & Chen-Hayes, 2005; Lapan, 2001; Lapan & Kosciulek, 2003; Martin & Carey, 2014). For example, useful evaluation structures for large-scale studies would include Lapan’s approach. Established, in part, on Johnson and Johnson’s (1982) competency-based model, Lapan’s framework stresses the evaluation of (a) CSCP’s structural and program components, as well as aspects of the counselor’s work (e.g., distribution of time and resources); (b) organization of effective school environments; and (c) student development within demographic, cultural, economic, and political contexts. Six planning and evaluation processes are included as well.

Another option for these state-level CSCP evaluations is the logic model of Frechtling (2007). In fact, Martin and Carey (2014) deployed this approach in a general evaluation of the ASCA (2012) National Model. Specifically, this framework allows researchers to analyze CSCP inputs (foundational elements and program resources), school counseling-related activities (e.g., direct and indirect services, school counselor personnel evaluation, management), outputs (e.g., student change, parent involvement, administrator support), and program outcomes (e.g., student achievement, systemic change and school improvement). Logic modeling should be incorporated into CSCP policy development and program revision planning.

With increasing school district restrictions on data collection , two complementary evaluation approaches may be more feasible for counselors to initiate. Astramovich and Coker (2007) developed the Accountability Bridge Counseling Program Evaluation Model primarily as a framework for individual K-12 schools to deploy. In a systematic way, school counselors can assess themselves, looking, for example, at their effectiveness in planning and delivering services to students and whether they have made any impact on school and learner-related outcomes. Similarly, Eschenauer and Chen-Hayes (2005) created an accountability method called the Transformative Individual School Counseling evaluation model. It focuses on counselors conducting functional behavior assessments on students to define problems as well as single-case study designs to document intervention success. Whether the evaluation is relatively small or large in scope, it is incumbent on program evaluation leadership to select the most suitable approach to answer the research questions. With a strong organizational framework to formulate and direct large-scale CSCP evaluations, needed policy revisions and refinements in implementation strategies should be readily discernable.

Conclusion

In this chapter, we argued that school counseling program policy and practice need to be informed by high-quality accountability research findings. State-level evaluation studies considered above have generated encouraging results for students and schools. Despite their importance, this research continues to raise more questions about the effectiveness of CSCPs than they have answered. The ASCA (2012) National Model, for instance, remains understudied and its actual impact on student functioning and school-related outcomes is uncertain. It was suggested that prospective investigations must incorporate advancements in research methods, relying more heavily on quasi-experimental and longitudinal designs and multivariate analysis techniques. Clearly, much of the onus to assist with implementation of these research enhancements falls on school and district leadership. Federal and state educational departments must encourage administrators to be more receptive to large-scale studies, providing schools with tangible inducements to offset their costs. Without a meaningful partnership among national, state, and local counseling and educational entities, these policy-educating studies will continue to be less than optimal. The evaluation horizon is promising and school counselors have much to be proud of; however, this work is still in its infancy and further high-quality research is needed to firmly establish the efficacy of comprehensive school counseling programs for all stakeholders.