Introduction

Recovery is becoming well-established as the foundation for national mental health policy in the United States and many other countries (President’s New Freedom Commission on Mental Health 2003; U.S. Surgeon General 2000). There is also an increasing body of mental health service research demonstrating the effectiveness of specific interventions (Corrigan et al. 2008). The implementation of these evidence-based practices has become a major challenge for mental health systems and its practitioners. Both recovery and evidence-based practice demand new policies, perspectives, and skill sets within our mental health service delivery agencies.

The challenges are many; Hoge et al. (2011) identified contemporary challenges direct workers face. They include: (1) increased unobserved and autonomous service provision due to continuing shifts from facility-based to community-based care; (2) increased caseloads given financial pressures within agencies; (3) complexity of health and mental health needs of clients; (4) increased demands for risk assessment and management; (5) required competency in evidence-based practices collaborating with other helping professionals; and (6) advancement of shared decision making in the context of recovery.

The task is made more formidable by the fact that the majority of services are delivered by people who are not fully credentialed professionals. Rather, they tend to have a bachelor’s degree, with no professional training and very little human service experience (Hoge et al. 2007). The jobs are demanding and complex. Occupational stress, burnout, and job dissatisfaction are high (Finch and Krantz 1991; Gellis et al. 2004; Kirk et al. 1993). Turnover in these positions is frequent (Corrigan et al. 2008).

Proposed solutions have included (1) significantly raising salaries to attract more seasoned and credentialed staff and (2) increasing the training of staff. Unfortunately the former is unaffordable and the research on training effectiveness suggests that educational efforts do not strongly influence health care providers’ practice behaviors (Joyce and Showers 2002; Miller et al. 2006; Torrey et al. 2012). Curry et al. (1994) estimate that only 10–13 % of skills taught are transferred to the working environment.

This article avers that improved supervision of front-line staff is a third approach to improving performance (Rapp 2006). The purpose of this study was to create an instrument that would assess the benefits or results that accrue from different models of supervision. This included client-centered supervision which is not present in other supervisory scales. This article reports on the results of tests for content validity and reliability.

Supervision

Supervision was originally developed as a part of organizational/administrative responsibilities. Duties included overseeing workers’ efficiency/productivity and agencies’ accountability but evolved to also focus on improving the effectiveness and efficiency of the work processes (Greene 1991; Kadushin and Harkness 2002), improving workers’ skills (Poertner and Rapp 2007), and attending to the socioemotional needs of workers (Kadushin and Harkness 2002). Kadushin and Harkness (2002) conceive of supervision as being of three types: administrative (agency management), educational (professional development), and supportive (expressive-supportive leadership). In addition, supporting workers’ efforts to achieve client goals (Harkness 1997; Poertner and Rapp 2007) has received increasing attention given the current climate of client-centered care (Institute of Medicine, Committee on Quality of Health in America 2006).

There is evidence that the quality of care received by clients can be influenced mightily by the quality of supervision that front-line workers receive (Ahearn 1999; Corrigan et al. 2000; Harkness 1995, 1997; Harkness and Hensley 1991). Supervision has been shown to increase the acquisition, retention, and application of knowledge learned in training (Ng and Cheung 2007; Smith et al. 2007). It has also been shown to improve client outcomes through implementation of Trauma Focused Cognitive-Behavioral Therapy (Kitchiner et al. 2007), Cognitive-Behavioral Therapy (Lau et al. 2004), and Multisystemic Therapy (Schoenwald et al. 2009). Bambling et al. (2006) found that alterations in supervision to focus on therapeutic alliance produced a significant effect on working alliance, symptom reduction, and treatment retention. Several reports emanating from the National Evidence-Based Practice Implementation Project (McHugo et al. 2007) found that the quality of supervision affected the attainment of high fidelity implementation of supported employment (Kostick et al. 2010; Marshall et al. 2008); and Integrated Dual Diagnosis Treatment (Brunette et al. 2008). A series of evidence-based practice implementation reports from Kansas suggests that the difference between success and failure may often be determined by the quality of supervision (Rapp et al. 2010a; 2010b). A national study of evidence-based practice site consultants and trainers found substantial agreement as to the supervisory behaviors important to successful implementation (Carlson et al. 2012).

However, majority of studies supporting effectiveness of supervision often fail to address the attributes of influential supervisory behaviors on outcomes, presumably due to a lack of relevant measurement. In addition, most of the literature discusses the administrative and clinical focus of supervision targeting supervisor–supervisee linkage, but the focus of the effect on client outcomes is sparse or not yet well studied. Given the dominant focus, most available supervisory scales evaluate the benefits from working alliance between supervisor and supervisee with underlying assumptions that the positive effect might lead to greater client outcomes (Bambling et al. 2006). However, these scales do not necessarily address the character of supervisor–supervisee working alliance that directly results in support for client outcomes. For example, research using existing scales in supervision have not found a relationship between the character of supervision and client outcomes (White and Winstanley 2010). Both the Manchester Clinical Supervision Scale (MCSS) (Winstanley 2000) and the revised MCSS-26 (Winstanley and White 2011) do not explicitly address the supervisor’s supportive function in addressing workers’ efforts to assist clients in achieving their goals (Harkness 1997; Poertner and Rapp 2007). The Experience of Supervision Scale developed by Parente (2011) used the Kadushin and Harkness’s (2002) framework, but it does not include client-centered core items. These scales did find association between efficiency of clinical supervision and lower rates of burnout and more job satisfaction of mental health professionals (Hyrkäs 2005).

Given the recovery movement and the imperative to implement evidence-based practices, identifying and measuring the critical components of supervision assuring a client-centered approach to care is sought. Such a measure could boost research that seeks knowledge of more direct relationship between client outcomes and supervisory behavior. It could also be an influential tool for improving supervisory practice. The measurement of fidelity in the National Evidence-Based Practice Project, for example, was found to be a key element in improving evidence-based practice implementation (Bond et al. 2009; Rapp et al. 2010b). Poertner and Rapp (2007) include many other examples of measurement influencing performance supporting what the organizational theorist Mason Hare said, “What gets measured gets done.” (Quoted in Peters and Waterman 1982, p. 268). In this study, we were interested in identifying factors consisting of supervisory support that case managers receive, particularly focusing on the potential benefits in supporting client goals and recovery. In addition, we examined the scale score differences between different supervision formats, namely client-centered group supervision, individual supervision, and team meeting.

Methods

Item Generation

The University of Kansas, School of Social Welfare organized an item-generation team to develop the Perceptions of Supervisory Support (PSS) Scale. The team consisted of trainers, practitioners, and researchers for various evidence-based practices. The team generated the pool of items that cover the critical components of supervision from a comprehensive review of the literature as well as their practice experiences in client-centered care. Through the iterative and consensus-building process, twenty-seven items measuring the extent to which front-line staff receives support in supervision were generated. Generated items were reviewed in terms of grammar, response logic, and readability, so the face validity of the candidate scale was confirmed. Items were anchored using a 1–5 scale (“1 = Less than 10 % of the time” to “5 = 91 % and more of the time”) with 5 signifying a higher percentage of time the worker perceived receiving support in supervision.

Scale Administration

In Kansas, three primary forms of supervision are used within Community Support Services programs in mental health: individual supervision (IS), team meeting (TM), and client-centered group supervision (CGS). IS is scheduled dyadic meetings between a front-line worker and their supervisor. Emphasis varies widely but often includes problem-solving around management of caseloads, feedback, with a minority of time devoted to discussing specific cases (Kadushin and Harkness 2002). TM most often covers administrative concerns (e.g., new policies or procedures, job related concerns) and/or brief updates on the status of clients with, perhaps, some suggestions for next steps. The degree of structure for both IS and TM varies dependent on the supervisor’s preference. CGS, like TM, occurs in a group setting but focuses, almost exclusively on client goal attainment. It is highly structured with rules for case presentations, questioning, and brainstorming. Typically, each discussed case situation would consume 20–30 minutes [For a fuller description see Rapp and Goscha (2011)].

Two hundred and sixty-two case managers from 22 of 26 mental health centers in Kansas completed surveys in a group setting where they were asked about their experience of supervision in the past two months. The survey was administered between January and June in 2011. The study was approved by the Institutional Review Board of the University of Kansas.

Analyses

Data analyses involved three steps. First, an exploratory factor analysis (a principal axis factoring method with promax rotation) was conducted to uncover the underlying structures of the PSS scale for those who received CGS. Second, confirmatory factor analyses were conducted to test if the same factor structure holds with other usual types of supervision, namely IS and TM (testing the rigor and applicability of the factor structure for different formats). Finally, repeated measures analysis of variance accounting for the sample dependency (i.e., the same person rating the scales for each of the three forms of supervision if applicable) tested if the scale scores reflect the differences between three types of supervision. SAS ver. 9.2 and Mplus ver. 6 were used for analyses.

Results

The most frequently used supervision format was a combination of CGS and IS (n = 80, 31 %) followed by all three (n = 77, 29 %), TM only (n = 43, 16 %), TM and IS (n = 41, 16 %), TM and CGS (n = 16, 6 %), and only CGS (n = 5, 2 %). No agency used only IS.

Exploratory factor analysis (see Table 1) retained 19 items and extracted three factors: “emotional support (Factor 1),” “support for client goal achievement (Factor 2),” and “educational/professional development support (Factor 3).” Items eliminated were either lower factor loadings for all factors (i.e., <0.30), cross-loadings, freestanding, or highly correlated with other items. The number of factors extracted were determined based on scree tests, Kaiser’s criterion (i.e., eigenvalues greater than one), and interpretability. Kaiser–Meyer–Olkin measure of sampling adequacy was 0.93. Correlations among factors were from 0.3 to 0.6. The internal consistency (Cronbach’s alphas) for each factor were 0.9, 0.9, and 0.7, respectively.

Table 1 Results of exploratory factor analysis and descriptive statistics of the Perceptions of Supervisory Support Scale for client-centered group supervision
Full size table

Confirmatory factor analyses with the three factor model revealed acceptable fit for both IS (Comparative Fit Index [CFI] = 0.94, Tucker-Lewis Index [TLI] = 0.93, Standardized Root Mean Residual [SRMR] = 0.04, Root Mean Square Error of Approximation [RMSEA] = 0.08) and TM (CFI = 0.92, TLI = 0.90, SRMR = 0.06, RMSEA = 0.08) formats, indicating the same factor structure holds across different types of supervision.

Repeated measures analysis of variance (see Table 2) showed significant group differences for all supportive functions [emotional support: F(2, 522) = 11.35, p < 0.001; support for client goal achievement: F(2, 522) = 35.21, p < 0.001; educational/professional development support: F(2, 520) = 57.71, p < 0.001]. Post-hoc mean comparisons with Bonferroni correction revealed that: (1) CGS and IS had higher scores than TM for emotional support (p < 0.001); (2) CGS had higher scores than TM and IS for support for client goal achievement (p < 0.001); and (3) IS had higher scores than CGS and TM for educational/professional development support (p < 0.001).

Table 2 Mean comparisons of the Perceptions of Supervisory Support Scale (sub-scores) across three supervision formats
Full size table

Discussions

This is the first study we are aware of to test the psychometric properties of a scale that included the critical components of client-centered supervision. Our study revealed three factor structures of the PSS scale in the following areas: (1) emotional support; (2) support for client goal achievement; and (3) educational/professional development support. “Emotional support” refers to efforts aimed to acknowledge work performed by the worker, improve relations among team members, reduce worker stress, increase positive feelings from the worker about their work and the agency, etc. “Support for client goal achievement” refers to efforts to give workers ideas on how to help clients achieve goals, help workers better align with the clients’ goal, help the worker gain a new perspective on a client, reprioritize efforts toward the client’s goal, etc. “Educational/professional development support” refers to efforts to provide the worker feedback on their performance, offer coaching and training on job skills, further career development, etc.

The findings are generally congruent with Kadushin and Harkness’s (2002) conceptual framework (i.e., administrative, educational, and supportive), but “support for client goal achievement” was explicitly included as a function of supervision instead of “administrative.” This is reasonable since supervisees rarely perceive of administrative/organizational supervision as helping their work and desire assistance with case situations (Poertner and Rapp 2007). The need for supporting client goal achievement is consistent with the Institute of Medicine’s call for person-centered care (Institute of Medicine, Committee on Quality of Health in America 2006). Further, the factors extracted in this study accommodate the current supervisory practices called “clinical” (Hoge et al. 2011) with supervisory practices supportive of recovery and non-clinical practices. Given the challenges front-line practitioners face in delivering recovery-oriented and evidence-based practices, the PSS scale seems to capture a desired function of supervision, which is not present in other supervisory scales. The three factor model fits well across different supervision formats, indicating that the scale can be used for both individual and group format.

Along with the evaluation of the factor structure of the PSS scale and its consistency across the three different supervision formats, we were also interested in testing another aspect of validity. Analysis of variance tests revealed that the scale scores differentiated the characteristic differences between supervision formats. In particular, case managers who received CGS perceived more support for “client goal achievement” than other formats and equivalent support to IS in “emotional support.” It was anticipated that CGS would be better at helping case managers with work with clients (support for client goal achievement) given its objective. On the other hand, IS is better for providing feedback and for educational purposes. Lastly, it seems that TM is not a winner for any support function. TM often does not have a structured format and is used as a supplement for IS (Kadushin and Harkness 2002). The strengths of TM may be its flexibility when calling for a meeting that is needed for organizational purposes.

This preliminary study has some limitations. First, a convenience sample in one state was used, which limits the generalizability. For the purpose of more rigorous scale development, a future study recruiting a larger random sample will be necessary. Second, we did not measure other relevant variables such as the levels of burnout and job satisfaction, which restricted our ability to test the construct validity of the scale. Third, this was a cross-sectional study, so the stability of the scale over time is unknown. For the usage of future intervention purpose, a longitudinal study testing test–retest reliability along with the sensitivity for intervention will be needed. Finally, the absence of client outcome variables in this study does not speak to whether the type and level of support will actually predict client outcomes. This consideration and testing are particularly important for assuring the predictive validity of the scale in the context of client-centered supervision. Thus, an objective evaluation will be needed examining whether the inclusion of the five client-centered items would address the improvement in recovery and goal-oriented practices. Despite the limitations, the explicit focus on client-centered supervision in the PSS scale would add to the literature supplementing other existing supervisory scales.

Conclusions

The findings confirmed the content validity of the scale and demonstrated that the scale has good internal consistency. The scale can be used for both individual and group formats. Nineteen items (presented in Table 1) are easily administered for use in research and practice. Since supervision at its most fundamental level should enhance the effectiveness (client outcomes) of direct service workers’ efforts, assessing the degree to which this is being accomplished within an agency seems like an important and useful task. The PSS scale could guide support for front-line practitioners in delivering client-centered care and could be useful for future research.