Introduction

With the movement towards recovery-oriented mental health services worldwide, organisations are increasingly required to train or re-train mental health practitioners in recovery-oriented practices. Care planning in recovery-oriented practice should be consumer-centred and focus on personal growth and living a meaningful life, in contrast to setting clinical goals that are largely dictated by professionals (Andresen et al. 2011). Training the mental health workforce represents a significant cost to public and private sector services. However, training in a new method of working often does not readily transfer into practice (for reviews see: Blume et al. 2010; Herschell et al. 2010; Powell et al. 2012). Implementation of new practices can be conceptualised as taking place in four phases: Exploration, Preparation, Implementation and Sustainment (Aarons et al. 2011). Aarons et al. suggested that the success of the implementation phase is dependent on the degree to which the innovation is aligned with the values and tasks of the organisation, practitioners and consumers, while ongoing support and fidelity monitoring are important in the sustainment phase.

This research spans the implementation and sustainment phases by delivering a training program supported by 12 months of follow-up coaching. Follow-up supervision (Sholomskas et al. 2005) or coaching (Miller et al. 2004) following training workshops have been shown to reinforce newly-learned clinical skills. We compared the effects of two different coaching approaches on the implementation and sustainment of a model of recovery-oriented service provision, the Collaborative Recovery Model (CRM; Oades et al. 2009). The CRM is a well established approach consistent with the concept of recovery described by consumers (Andresen et al. 2003). A coaching-style relationship is used to help consumers clarify their values in life and to describe a life vision. The practitioner then collaborates with the consumer to set personal life goals and identify actions that will move them towards their life vision. Working towards such intrinsically valued goals has been shown to be highly motivating (Deci and Ryan 2002; Emmons 1999; Sheldon and Kasser 2001). The CRM utilises a set of three specific tools, collectively called Life Journey Enhancement Tools (LifeJET), which support the articulation of a person’s life vision, strengths and values, goals and action plans (Oades and Crowe 2008). These written records are retained by the consumer, promoting ownership of the goals, and a copy is filed with the service provider.

Training in the CRM has been shown to improve mental health workers’ knowledge and attitudes towards recovery (Crowe et al. 2006) and the quality of goal and action planning after training (Clarke et al. 2009). However, Uppal et al. (2010) found that only 37 % of practitioners had implemented any aspect of the CRM 12 months following training, with an average interval of transfer into practice of 5½ months. Participants cited nine types of barrier to implementing the CRM, which were categorised into organisational, consumer-related and practitioner-related barriers (Uppal et al. 2010). The focus of the current research was on addressing practitioner-related barriers, described as practitioner confidence with the model, practitioner self-management skills, and philosophical opposition (Uppal et al. 2010). To address these issues, following training, participants received 12 months of coaching to increase confidence and motivation for implementing the CRM. Two types of coaching were provided: standard skills acquisition coaching or transformational coaching.

Skills acquisition coaching generally focuses on the goals of the employer, using a didactic approach to teach the coachee new skills and techniques to improve his or her work performance (e.g. Segers et al. 2011). In contrast, transformational coaching explores the coachee’s personal values and goals to promote personal growth and/or professional development. It encourages a change in the coachee’s habitual responses, increasing the motivation to act more consistently with his or her values (Hawkins and Smith 2010). In clarifying intrinsic values, we aim to make explicit those personal values that are consistent with a recovery orientation. Our transformational coaching approach also exploited the concept of ‘parallel process’ in supervision (Crowe et al. 2011). Parallel process is a phenomenon which takes place spontaneously during supervision, in which the practitioner subconsciously re-enacts aspects of their relationship with consumers in interactions with their supervisor (Morrissey and Tribe 2001). Morrissey & Tribe assert that awareness of parallel processes can be utilised during supervision as an aid for teaching and learning. An innovation of the transformational coaching condition is that these processes are explicitly encouraged and reinforced by using the same LifeJET tools in coaching as clinicians are using with consumers (Crowe et al. 2011). Although both forms of coaching were expected to engender greater confidence and skills with the CRM, we expected the values clarification and parallel processes in the transformational condition to also increase motivation and commitment, which would in turn be reflected in higher quality recovery-oriented care planning.

This article presents the results of a quasi-experimental study investigating the quality of recovery-oriented goal and action planning following training and either skills-based or transformational follow-up coaching. An audit of clinical files compared the quality of goal and action planning before training (0–6 months), soon after training (0–6 months) and longer term (6–12 months after training). The two coaching conditions are compared.

Methods

Participants

Ethics approval was granted by the University of Wollongong Human Research Ethics Committee and the ethics committees of the participating mental health organisations. Staff of four community-managed mental health organisations, representing 13 sites across four states of Australia, were trained in the CRM. After the study was described to trainees, written consent was obtained from 188 practitioners. Participants were allocated to receive either skills coaching (‘Skills’ condition), or transformational coaching (‘Transformational’ condition). Because staff worked in teams, coaching condition was randomised by team (n = 20) and blocked by organisation (n = 4). Thus, for the two larger organisations, four teams were allocated to each condition, while for the two smaller organisations, one team was allocated to each condition. Training was rolled out over an 11-month period according to the needs of each organisation.

Procedure

Training in the CRM Model

All staff received standard CRM training for the first 2 days. This was followed by 1 day of training specific to each condition. Both teams engaged in reviewing the Model and skills in the use of the protocols. Those in the Skills condition then focused on identifying and exploring solutions to organisational and personal barriers to implementation of the CRM with their clients. In contrast, those in the Transformational condition focused on activities around the clarification of their personal values, and practiced coaching each other using the LifeJET protocols. All participants then undertook the same general preparation for undertaking coaching. Booster training sessions were conducted at 6 and 12 months following initial training. Participants in both conditions were expected to attend 1-h coaching sessions once per month for 12 months.

Coach Training

Senior practitioners who were experienced in the CRM were selected by service management and trained as coaches by the research team. Characteristics sought in potential coaches were that they were open to change; valued personal and professional development of staff; had good interpersonal skills and were respected by their peers (Deane et al. 2010). Coaches were trained in the use of the GROW model: Goals—setting goals for each coaching session; Reality—exploring the coachees’ current situation; Options—examining their options; and Wrap-Up—evaluating options, creating a plan, and problem solving any foreseen difficulties (Alexander and Renshaw 2005; Whitmore 2002). Within this framework, Skills condition coaches were trained to address problems with the implementation of the CRM that coachees brought to the session. This could include clinical relationship issues, change enhancement strategies or the use of the LifeJET protocols for values clarification, goal setting or action planning. Transformational coaches were trained to use the LifeJET protocols to assist in the coachees’ personal and/or professional development by continuing to explore their personal values, vision and important goals. Transformational coaching paralleled the coaching-style approach participants were using with clients. Coaches were trained to recognise and reflect on parallels between the coaching relationship and the coachee’s relationship with clients, in order to sensitise participants to the clients’ experience of the CRM. Both coaching conditions are described more fully in Deane et al. (2010). Coaches were supported by monthly group-based coach-the-coach sessions with members of the research team.

Participants were allocated a coach who was not their direct line manager, in order to facilitate openness during coaching.

File Audit

At the end of the 12-month study period, ranging from September 2010 to August 2011, an audit was conducted of client files. The quality of implementation was indicated by the standard of completion of the components of goal planning trained within the CRM. Internal audits were conducted on a random sample of client files from each of three time periods: between 0–6 months prior to training (Time 1), 0–6 months post-training (Time 2) and 6–12 months post-training (Time 3).

Selection of Files for Audit

Each organisation supplied the research team with a coded caseload list for each practitioner for each time period. Using an online random number generator, two clients of each practitioner within each time period were selected for the file audit.

Some practitioners joined the research project at the time of their initial training as an employee of the respective service, and thus did not have client files for the period prior to CRM training. Also, due to staff movements and turnover, not all practitioners who undertook initial training had a client load at the later time periods. In addition, individual consumers were not necessarily engaged with services across all time periods. Therefore, client files were selected randomly for each practitioner who had a client load in the particular time period, rather than following practitioner–client pairs across the three time periods. The most recent care plan of the selected client record within the time period of interest was audited.

Measures

Quality of implementation was determined by ratings on the Goal and Action Plan Instrument for Quality (GAP-IQ), an updated version of the Goal Instrument for Quality (GOAL-IQ; Clarke et al. 2009). The GOAL-IQ is an audit tool developed to assess the quality of clinical goal setting and action planning. It was developed specifically to assess the facets of care planning considered essential to the CRM, which can be broadly categorised into the domains of Vision, Goal Setting, Motivation Enhancement, Action Planning and Review. The GOAL-IQ has demonstrated inter-rater reliability and sensitivity to change (Clarke et al. 2009). The GAP-IQ increases the number of items on the GOAL-IQ from 11 to 17 items, in order to capture the action planning components of the care plans. These additional six items had previously had interrater reliability assessed in an independent sample with intraclass correlations ranging from r = 0.81 to 0.91 (Kelly 2007). Items are rated on a 3-point scale (0 = No; 1 = Partial; 2 = Yes), with the exception of the item Collaboration, which is dichotomous (0 = No; 2 = Yes). Alongside each item are detailed descriptors for each rating. Possible scores range from 0 to 34.

Amount of coaching received was indicated by a count of Coaching Record sheets, which were completed by the coach following each session and returned to the researchers.

Analyses

Reliability Testing

Since ethics requirements were that researchers could not access client files, the file audit was conducted by organisational staff. It was therefore necessary to establish the interrater reliability of the GAP-IQ. Services chose one or two staff members to act as internal auditors and these individuals were provided with training in the use of the GAP-IQ. One organisation did not have the resources to provide a staff member to conduct the audit; therefore, de-identified files were audited by a member of the research team. The research auditor received equivalent training to that of the internal auditors.

To assess interrater reliability, the internal auditors supplied the research team with de-identified photocopies of a randomly selected subset of 30 % of the audit files, along with their GAP-IQ ratings. These files were then blind rated by the research auditor to establish reliability.

Audit Analysis

Randomly selected client files were audited to assess quality of care planning at three time points, using two-way between-groups analysis of variance. Data were analysed using the IBM SPSS statistics software.

Results

Files Audited

Only client files from practitioners for whom we had documentary evidence of having received coaching were included in the analysis. This was determined by (a) The return of Coaching Record sheets, (b) inclusion of the practitioner as a coachee on the Coach’s Experience Questionnaire completed at the 12-month time point, and/or (c) The return of Coachee’s Reflections on Coaching Questionnaire by the practitioner. (These latter two records are qualitative questionnaires not otherwise related to this report).

We received documentary evidence that coaching had been received by 123 participants. Coded case lists for 98 of these practitioners were supplied by the partner organisations for random selection of files for audit. Figure 1 shows the breakdown of files requested, received and included in the audit, which ultimately consisted of 298 (61.9 % of requested) files: 153 (67.6 %) for participants in the Transformational condition and 145 (57.9 %) for the Skills condition. For Time 1, 85 files were received (46 Transformational, 39 Skills), For Time 2, 110 files (58 Trans, 52 Skills) and for Time 3, 103 files (49 Trans, 54 Skills). These files represented 91 practitioners and 254 individual consumers.

Fig. 1
figure 1

Flow chart of files included in audit

Full size image

Demographic information of practitioners included in the audit is shown in Table 1.

Table 1 Demographics of participants included in audit (n = 91)
Full size table

Interrater Reliability of GAP-IQ

In total, 82 de-identified copies of files coupled with corresponding GAP-IQs were supplied to the research team for reliability testing. Since not all files had utilised the LifeJET protocols, the GAP-IQ was applied to other types of care plans in some instances, potentially increasing the likelihood of differences between raters.

There was a high correlation between raters on the total score of the GAP-IQ (r s  = 0.92, p < 0.01). Agreement on individual items was determined using percentage agreement and Cohen’s kappa coefficient (κ), which determines agreement allowing for chance. In general, a κ of less than 0.41 is considered unacceptable (Viera and Garrett 2005). Agreement ranged from κ = 0.404 (Collaboration) to κ = 0.904 (Action Confidence) (Table 2). The low κ for Collaboration was coupled with a high percentage agreement at 86.58 %. Examination of the data confirmed that this item was almost always rated 2 (‘Yes’). The resulting low kappa statistic was due to this lack of variability (Feinstein and Cicchetti 1990). Therefore, all items were considered to have acceptable interrater reliability, indicating no systematic bias by the various auditors.

Table 2 Interrater agreement for individual items of GAP-IQ
Full size table

Results of File Audit

Coaching Record Sheets were returned for 82 participants. The mean number returned ranged from 1 to 12, with a mean of 5.13 (SD = 2.95), indicating that many practitioners were not receiving coaching monthly, as required by the study protocol. A Mann–Whitney U test found no significant difference between the coaching conditions in the number of Coaching Record Sheets returned (p > 0.05).

GAP-IQ data were normally distributed. Means of GAP-IQ for both conditions increased over 12 months, as presented in Table 3. Increasing improvement of scores across time can be seen in the Transformational condition, while scores in the Skills condition appear to level off after Time 2. Baseline scores between the two conditions were not significantly different. A two-way between-groups analysis of variance revealed significant main effects for condition (F (1,291) = 7.40, p < 0.01) and time period (F (2,291) = 3.83, p < 0.05). The interaction effect was non-significant. These results are presented graphically in Fig. 2.

Table 3 Mean GAP-IQ scores for each condition by time period
Full size table
Fig. 2
figure 2

GAP-IQ scores across time period for each condition

Full size image

When there is no interaction effect, separate univariate analyses would not normally be conducted. However, the pattern of results (Fig. 2), the novel aspects in the transformational condition, less than ideal rates of implementation and a wish not to prematurely dismiss potential helpful components in the conditions, led us to conduct post hoc univariate analyses for each condition. These revealed that there was a significant improvement in quality of goal planning between Time 1 and Time 3 in the Transformational condition (p < 0.05), but not in the Skills condition. Overall, the results show that training followed by coaching was associated with a significant improvement in care planning over 12 months in both groups. However, when analysed separately, significantly higher scores on goal planning quality were found between Time 1 and Time 3 only in the Transformational condition.

Discussion

The results indicate that training followed by coaching is associated with increases in the quality of recovery-oriented goal setting and action planning, providing preliminary support for the use of coaching to increase the transfer of training into the workplace. Although no interaction effect was found between the two conditions, it may be premature to abandon the strategies we used to enhance coaching. Natural turnover of staff and other forms of attrition led to a reduced sample size, which may have contributed to the lack of an interaction effect. Power would also have been reduced due to the “Time” factor being between groups rather than within groups, also necessitated by attrition. There was also potential for teams receiving different forms of coaching to communicate with each other over the study period, for example, via staff movements between sites, again reducing differences between the conditions.

Therefore, although the main conclusion is that both coaching conditions were associated with increases in the quality of documented care planning over time, post hoc analyses suggest further investigation of the components of transformational coaching is warranted. For example, it remains unclear at present whether values clarification actually reduces “philosophical opposition” (Uppal et al. 2010) or whether parallel process is the key component. Anecdotal experience suggests philosophical opposition to some recovery-oriented practices still exists, with one participant in the current study stating that they could not change their way of working with consumers. The clarification of personal values attempts to tap into those values consistent with a recovery-oriented approach, or, alternatively, reveal conflicting values that may represent potential barriers to change. With regard to the potential of parallel processes to improve the effectiveness of coaching, there is a need to more directly assess whether activities such as using the same tools and strategies with coachees as those used in work with consumers actually sensitises practitioners to the subjective experience of their clients. There is then a need to explore whether this informs their approach to working collaboratively with consumers.

It is noted that although the quality of goal and action planning improved over time, scores are still relatively low, indicating that there remains much room for improvement. Coaching was only received approximately once every 2 months on average—more regular coaching may lead to greater improvements. We are also mindful that the completion of paperwork may not accurately reflect the clinical relationship; however, scoring the GAP-IQ includes seeking evidence of collaboration with the consumer. That is, the language used in the care plan should reflect the consumer’s recovery vision, rather than prescribed clinical goals, and there should be an absence of jargon. In this study, collaboration was observed frequently in both conditions.

Limitations

A limitation of the study was the lack of a ‘no coaching’ comparison group. While there was overall improvement in the quality of care planning following training, it is not clear how much of this improvement would have occurred without follow-up coaching. A further limitation was that the auditors were internal to each organisation and were thus not blind to condition. Although they had no explicit stake in either coaching condition, it is possible they had unstated preferences or expectations that could led to bias in their quality ratings. Any potential bias due to un-blinded auditors was partially mitigated by interrater reliability ratings conducted by one of our research team who was blind to condition for those samples of care plans.

The results may have been impacted by the high level of attrition in the study. Conducting research amongst front-line staff in mental health organisations, particularly when access to client files is required, presents a number of challenges. Due to ethics requirements, researchers were not permitted direct access to the client files, and while there was a key research contact within each organisation, each step of the data collection process involved the cooperation of several personnel. In addition, filing systems made searching for closed client files by practitioner name difficult. Therefore, lack of documentation received by the research team may not always indicate a lack of implementation. Mental health services have high turnover rates, with organisations in this study averaging a 20 % turnover in 12 months. A long lead-in from initial training to the commencement of the project proper resulted in a loss of some participants to the study, and/or changes to research codes, and at least 63 participants resigned, transferred or took long-term leave during the course of the study.

In addition to loss of data, it is evident that coachees did not attend coaching at the rate suggested. Anecdotal reports suggest this was due other organizational and work demands, resistance and dissatisfaction with the coaching implementation process and in some cases a lack of documentary evidence that coaching had actually occurred. Therefore, we limited our analyses to those client files that we could confidently match to participants who had received coaching. The resulting smaller sample size may have contributed to the lack of power in some analyses.

Conclusions

The research has implications for mental health services wishing to implement new recovery-oriented practices with staff. Although methodological limitations warrant some caution in interpretation of the results, the quality of goal and action planning, as advanced by the CRM, improved from pre-training to 12 months post-training when follow-up coaching was provided. There is sufficient evidence to suggest that components of the transformational coaching condition merit further study. Specifically, values clarification exercises, and the use of the same vision and goal planning protocols in coaching as are used in direct work with consumers. The extent to which these components support values-based motivation or parallel processing is yet to be determined. It is also unclear whether there is alignment of these personal values with those that underlie recovery-oriented service provision. Future research could utilise audio or video recordings of coaching sessions to clarify whether these potential mechanisms are evident. Nonetheless, the study supports the value to organisations of reinforcing the implementation of any new practice model by committing to ongoing coaching. Future research needs to explore the specific mechanisms in the coaching process that are associated with improvements in recovery-oriented practice.