Introduction

The routine measurement of patient progress is a critical component in maintaining the health and wellness of patients in a variety of healthcare settings (Goodman et al. 2013). The process is akin to taking routine vital signs to determine if a patient is on track to recovery or is deteriorating from an established path (Lambert et al. 2003). In mental healthcare, it seems logical, and ethical that the same standard of care should apply, yet research has shown that clinicians are not accurately and routinely assessing client progress and have a difficult time detecting client deterioration in treatment (Hatfield et al. 2010). Progress monitoring (PM), the periodic and routine assessment of client progress to evaluate, inform, and enhance the effectiveness of psychotherapy, was first introduced by Howard et al. (1996). PM has also been referred to as routine outcome monitoring (Lambert 2010) and feedback informed treatment (FIT) (Miller et al. 2015). The use of PM shifts the focus of treatment from a reliance on treatment plans that match antecedent factors with a specific treatment approach to ongoing client monitoring that can detect improvement or deterioration and rapidly shift treatment approaches based on emerging client needs (Lambert et al. 2003). National mental healthcare organizations and associations are calling for increased efforts to utilize PM in a variety of mental healthcare settings (American Psychological Association 2009; Group for the Advancement of Psychiatry, Valenstein et al. 2009). In addition, the Netherlands and United Kingdom now mandate its use in mental healthcare delivery (Miller et al. 2015). The push to implement PM centers on enhancing treatment effectiveness and cost efficiency, though more research is needed on how best to implement PM given organizational missions, structure, and treatment foci.

Recent randomized controlled trials (RCTs) show that the use of PM can “(a) increase effect sizes by a factor of two, (b) increase the percentage of clients that experience reliable and clinically significant change, (c) reduce client dropout rates by 50%, (d) reduce the risk of deterioration by one-third, (e) shorten the length of treatment by two-thirds; and (f) reduce the cost of health care” (Miller et al. 2015, p. 449). The reported benefits drive the on-going efforts to implement PM in a variety of treatment contexts, including group care programs aimed at treatment substance use disorder (SUD) (Weems 2011). In SUD treatment for adolescents, PM is still in the early stages of development but is an important consideration given relatively high documented deterioration rates in treatment. For example, studies have shown that up to one-quarter of clients at the early stages of the treatment process are actually doing worse than they were when they entered treatment (Lambert and Shimokawa 2011; van Sonsbeek et al. 2014). Posten and Hanson (2010) conducted a meta-analysis of the effects of psychological testing followed by personalized feedback on therapy process and outcome across 17 studies on a variety of mental health and alcohol abuse treatment studies. They concluded that routine assessment that provided feedback to the clinician on client progress positively impacted treatment, which was evidenced by an overall medium effect size (d = .423). In a preliminary RCT of PM for youth in psychiatric settings, Bickman et al. (2011) found that the clients of clinicians who received feedback based on reliable and routine assessments improved faster than those who did not, and that youth clients reported significant improvement over time with moderate effect sizes (d = .30). In addition, meta-analyses have also shown that PM in SUD treatment contexts can provide for more efficient record keeping, more accurate diagnoses, and appropriate treatment adjustments given client needs (Carlier et al. 2012; Shimokawa et al. 2010). Despite these promising findings, Goodman et al. (2013) concluded that care should be taken when interpreting the conclusions from these meta-analyses because many lack data on client demographics and the reported studies occurred across a wide variety of mental health settings. The meta-analyses do provide broad support that PM can yield positive outcomes, but research has also shown that several barriers impact implementation due to the time, effort, and resources involved in the process.

Mental health providers work in stressful environments, and PM can be seen as yet another burden placed on clinicians and administrators. PM is costly, requires training, and time and effort on the part of clinicians to administer the questionnaires and integrate the results into therapy sessions to provide meaningful and informed feedback (Duncan and Shaw 2012). In addition, Boswell et al. (2015) found privacy considerations around data use and storage and potential competition between therapists as additional barriers. One noted finding from these studies is that therapists do not completely trust the measures and resulting outputs and would rather rely on their own clinical interpretation and instinct (Boswell et al. 2015). Finally, administrator access to therapists’ outcomes could result in the differential comparisons between therapists, creating a potentially competitive and divisive organizational environment (Sundet 2012). Overcoming these barriers take a concerted effort and significant resources to overcome and require buy in from clinicians, youth care staff, and clients.

This study reports the preliminary results of an initiative to implement PM in a SUD treatment program for adolescents called Base Camp operating west of Calgary, Canada. The evaluation project involved the development of instrumentation and PM protocols for Base Camp. Formative and summative evaluation assessed the integrity and usefulness of the approach, staff adherence to protocol, and the potential outcomes of treatment. The following hypotheses were tested in this study: (a) Reliable and practical instrumentation and assessment protocol could be developed and implemented in an adolescent SUD residential treatment program. (b) The instrumentation and protocol could produce practical and reliable process and outcome data that could meet the needs of the program and its stakeholders. The goal of this study was to test these hypotheses to determine the feasibility and practicality of developing PM protocol in hopes of replicating this work in other settings to help researchers, practitioners, and mental health professionals determine if PM is appropriate for other adolescent treatment contexts in which they may be working.

Methods

Intervention and Procedure

Base Camp is a 15-bed residential facility that has been in operation since 1976 and works with adolescents ages 12–17 for treatment of substance use disorder (SUD). The 8-week program is operated by the non-profit agency ENVIROS and clients are referred to the program by Alberta Health Services case workers based on symptoms associated with SUD treatment. Base Camp’s strengths-based treatment milieu integrates family therapy, which includes a family weekend challenge course experience, educational programming for school credit, and daily group and individual therapy sessions. The therapeutic model is described as being an “eclectic,” strengths-based treatment modality that draws from narrative and solution-focused therapies, humanistic psychology, mindfulness practice, nonviolent communication, motivational interviewing and experiential learning (Base Camp 2017). Strengths-based approaches are grounded in the belief that adolescent mental illness is also associated with negative views of the “self” which present significant barriers to effective treatment, including low self-efficacy and self-esteem, a lack of motivation and drive and limited coping behaviors (Toback et al. 2016). Strengths-based interventions aim to enhance self-perception and character strengths and are an effective complement to traditional psychotherapeutic treatments liked those discussed above (Harris et al. 2016; Park 2009). Base Camp’s treatment approach seeks to develop these character strengths by utilizing adventure-based activities and experiences with therapeutic design and intent, commonly referred to as “adventure therapy.” These activities include, but are not limited to, challenge course sessions, solos, group initiatives and extended backcountry trips (see Gass et al. 2012). Finally, Base Camp also intentionally seeks to develop mindfulness-related skills, particularly in through their interactions with clients in day-to-day living situations and through adventure-based activities. These adventure-based experiences begin with clients setting clear therapeutic intentions and are facilitated by staff asking clients to routinely speak to how they are acting with awareness, being non-judgmental, and being non-reactive during the experience (Fernandez et al. 2010; Levin et al. 2014). A typical weekly program schedule would include a mix of individual and group therapy sessions, adventure-based activities, daily school sessions, leisure time, and more open group and community discussions addressing community values and well-being. The treatment team consists of youth care workers, psycho-educational group facilitators, a family therapist and clinician, and floor and shift supervisors who rotate in and out of the program on a routine basis.

The researchers in this study served as evaluators and were intricately involved in the entire process using principles associated with utilization-focused evaluation posited by Patton (2008). Program staff and the evaluators/authors met on three occasions prior to the implementation of the PM protocol to begin the process of developing relationships with the key stakeholders and to establish rapport with program leadership and staff. The first hypothesis being tested in the study required the evaluators to visit the program and develop relationships between staff and key stakeholders and begin the training and implementation process necessary to conduct progress monitoring. The first visit (2014) involved facilitated sessions where the directors, clinicians and staff described the philosophy, purpose, and approach of the treatment model and the overall program architecture. The output from these sessions was an enhanced understanding of the treatment philosophy and framework as the key stakeholders explored, refined and illustrated their approach to SUD treatment for youth. The second round of meetings included sessions where the evaluators illustrated their understanding of the program structure and framework and facilitated discussions on various types of evaluation protocol and instrumentation that would be effective for Base Camp. The third visit involved the evaluators and stakeholders finalizing the protocol, training the staff, and getting buy-in for the agreed upon methodological approach to be used to track key process and outcome variables. This approach was implemented in December of 2015 with support and guidance from the evaluators checking in with the Program Director and key staff on regular video conferences.

The consent process and confidentiality of parents or legal guardians and clients was approved by the Institutional Review Board of Georgia College and State University and maintained through the assignment of a code by the program administrator. These codes were used throughout data collection, analysis and the reporting process. Clients participating in the evaluation were asked to complete the instrument battery at admission, throughout the treatment process, and at discharge. The below instrumentation was adopted or developed using a progress monitoring approach to address the above research questions.

Participants

Participants in the study were adolescent’s ages 12–17 that were referred to Base Camp by mental health professionals associated with Alberta Health Services. Assessment procedures for fit included initial drug and alcohol use screening questionnaires, analysis of the use of previous mental health services and a risk assessment that examines the presence and absence of a variety of socio-cultural factors associated with client well-being. In addition, some clients are referred to Base Camp by mental health practitioners in schools in and around Calgary familiar with the program. Finally, parents may call Base Camp directly and inquire about the program. Parents are then referred to Alberta Health Services for screening. A total of 70 clients were included in the original database provided to the evaluators. Of those clients, 13 indicated no exit date, very little data on them was present, and they were not included in the analysis.

Measures

Process Evaluation

Personal Involvement with Chemicals

To assess substance use and client involvement with drugs and alcohol, the Personal Involvement with Chemicals (PICS) (Winters and Henly 1989) scale was administered at admission. The PICS scale focuses on the behavioral and psychological depth of substance use involvement and related consequences. The PICS asks questions about the reasons and effects of client substance use that fall into the broad domains of: (1) Recreational (using as a perceived from of recreation and leisure), (2) Effects (the degree to which they feel the negative psychological and physical effects of use), (3) Escape (using to escape problems and issues in their lives), and (4) Social Benefits (using to be with friends and feel more social). The Substance Use Frequency Scale (SUFS) is a straightforward assessment of use history that asks clients to think back over the last 3 months and indicate the number of times they have used a variety of substances, including alcohol, marijuana, and other drugs of choice. The scale also assesses when their first use occurred to gauge onset of addictive behaviors. The instrument develops profiles of clients that illustrate the severity of drug and alcohol use which can be used as a subject variable when analyzing outcome variables.

Program Outcome Questionnaire

The Program Outcome Questionnaire (POQ) was developed by the program directors, clinicians, and evaluators to assess the relative degree of importance clients place on various program elements, which was then used to contextualize program outcome. The instrument was used on a weekly basis to assess how specific program components, like Group Work, Home Visits, and Solos, helped clients achieve their weekly treatment goals. A visual analog scale was used such that clients could mark along a 10 cm line how helpful these various elements of the program had been for them in the previous week. This scale was used internally by staff to assess the relative degree of client satisfaction with program elements to aid structured conversations with clients about the importance of these elements in their treatment.

Outcome Evaluation

Youth-Outcome Questionnaire

The self-report Youth-Outcome Questionnaire (Y-OQ® SR 2.0) was utilized as a progress monitoring instrument to detect change in client well-being. The Y-OQ offers adolescent self-reports designed for repeated measurement of client emotional and behavioral symptoms (e.g. at admission, during therapy, at termination, and at follow-up intervals) (Burlingame et al. 2005). The 64 items contained in the Y-OQ are summed across six content areas to produce a total score. The higher the Y-OQ score, the more serious the adolescent’s symptoms. Estimates of the Y-OQ internal consistency range from .74 to .93 with a total scale estimate of .96. Test–re-test reliability scores are also above .70, indicating moderately high temporal stability. Specific subscale reliability coefficients for three of the subscales, (Interpersonal Distress, Interpersonal Relations, and Behavior Dysfunction) are moderate to high (.69–.93) suggesting homogeneity of content within each subscale (Gironda 2000). Internal consistency estimates are lower (.54–.83) for the Social Problems, Somatic and Critical Incidents given the broader content tapped in these scales. These estimates are deemed suitable for making comparisons of subscale scores across clients (Burlingame et al. 2005).

The Y-OQ instrument is easily administered by staff and only takes approximately ten minutes for clients to complete. The measure has not proven too complicated or detailed for respondents, which is an important consideration when working with adolescents. Thus, the Y-OQ assesses the psychological symptomatic and social functioning of the adolescents, which reflects the goals of Base Camp’s treatment approach and is a well-normed and easily administered outcome measure with good internal consistency and test–re-test reliability. The Y-OQ manual also suggests that if a client’s score “decreases by 13 points or more, they have attained a significant amount of symptom reduction” (Burlingame et al. 2005, p. 7), which is also based on the work of Jacobsen and Truax (1991) on clinically meaningful change. In addition, a “cut score” is contained in the Y-OQ, and is indicated by a score of 46 or lower, indicating that the client’s symptoms have returned to a normative level of functioning (Burlingame et al. 2005). These criteria were used in this evaluation to relate the change in adolescent self-report scores from admission to discharge, as well as weekly intervals throughout treatment.

Child and Adolescent Mindfulness Measure (CAMM)

Because a significant portion of Base Camp’s treatment approach is centered on the development of mindfulness related skills, the Child and Adolescent Mindfulness Measure (CAMM) was used to measure change in these skills at admission and discharge (Greco et al. 2011). The CAMM is a developmentally appropriate measure with adequate internal consistency to assess mindfulness in children and adolescents. CAMM scores are positively correlated with quality of life, academic competence, and social skills and negatively correlated with somatic complaints, internalizing symptoms, and externalizing behavior problems (Greco et al. 2011).

Data Analysis

Data were analyzed using the ‘IBM SPSS Statistics for Windows, version 24 (IBM Corp. 2017). Descriptive analysis was performed on all demographic variables, and pair-wise t tests were performed on outcome variables, including change in Y-OQ scores. Clinically significant change in Y-OQ scores was based on Jacobsen and Truax’s (1991) work on meaningful change in psychotherapy research. Effect sizes were also calculated on any dependent variables associated with treatment outcome, including Y-OQ subscales and changes in CAMM scores from admission to discharge. Where possible, effect sizes reported in this study sample were compared to studies reporting similar demographics or instrumentation. Regression analysis was performed to explore relationships between changes in mindfulness scores as assessed by the CAMM and change in Y-OQ scores.

Results

A total of 57 clients were included in the study and averaged 16.6 years of age at the day they entered treatment. The average numbers of days spent in treatment was 59.3 days (SD = 32.1). The majority of clients were male (n = 32; 56.1%) and almost half of the clients identified their ethnicity as white (57.9%) and almost one-third as Aboriginal or Metis (29.9%). A total of 43 clients (75.4%) entered treatment with a diagnosis or were diagnosed at admission, while 14 clients (24.6%) did not have an identified diagnosis. Almost half (45%) of those clients with a diagnosis presented with substance use disorder (SUD), and 26% with Attention-deficit/hyperactivity disorder. The “Other” category (29%) contained clients with oppositional defiant disorder, obsessive compulsive disorder and fetal alcohol syndrome. The results are guided by the two hypotheses tested in this study: (a) Reliable and practical instrumentation and assessment protocol could be developed and implemented in an adolescent SUD residential treatment program. (b) The instrumentation and protocol could produce practical and reliable process and outcome data that could meet the needs of the program and its stakeholders.

Implementation of PM

The three program visits to Base Camp to help the evaluators establish relationships and build rapport with the directors, clinicians and staff helped the stakeholders in the study come to a clear understanding of the philosophy, purpose, and approach of Base Camp’s treatment model and program architecture. The process was crucial in helping program staff and clinicians refine and illustrate Base Camp’s approach to SUD treatment for youth which then be articulated to the evaluators. This allowed for the evaluators to then to begin to facilitate discussions and present options for reliable instrumentation and protocol that could be effective for Base Camp. The third visit allowed the evaluators and stakeholders to finalize and present the protocol to directors, clinicians, and staff, and begin training on how to implement the approach to be used to track key process and outcome variables and have the process be integrated into the clinical approach used by Base Camp. The implementation process was characterized as one that was flexible and adaptive, and would be revisited semi-annually to discuss what was and was not working well. One important result from the implementation of PM for Base Cap was the successful integration of the protocol within the structure of the existing treatment model.

PM was used very specifically in weekly “Navigation Meetings” with clinical staff and the clients. The clients would complete a Y-OQ and a Program Outcome Questionnaire, which were then instantly scored and used as a discussion tool. Given the various total and subscale scores, questions resulting from the evaluation would follow that asked the client to provide some narrative as to their well-being, and to what degree various program elements had worked for them that week and why. The next step would involve treatment planning for the subsequent week, including specific program activities, projects, and the development of specific and achievable goals. The staff relayed to the evaluators that this process was illuminating for the staff and clients, and progress monitoring brought needed structure to these meetings. Clients provided feedback to staff that they felt empowered in these sessions because they believed they had a voice in their treatment process. Staff also reported that these sessions led to a more “reliable” assessment, because clients began to become familiar and more comfortable with the process and were more accurate in their weekly assessments because they knew that “not doing well” was “alright with staff” and knew that a conversation about “why” would follow as well as attention to “where do we go from here?” In addition, program staff were able to carefully examine how various program elements changed over time for individual and the collective group of clients.

For example, when examining the averages of client ratings from the various program elements, it was noted that they tended to trend upwards as treatment progressed. One-on-Ones (individual sessions) with staff and Family Therapy sessions were also rated as very important by clients, especially toward the end of treatment. The clear importance of Home Visits toward the end of treatment was also noteworthy, especially when compared to the initial weeks of treatment. The importance of Group (Group treatment and discussions) was relatively lower than many of the other factors throughout the course of treatment, even as clients were getting ready for discharge. Clients also did not rate the various Ceremonies that the program engages in as very important to their treatment process (fires, passage ceremonies, etc.). Finally, the Progress Monitoring Sessions (clients sit with staff and discuss Y-OQ scores and treatment goals) jumped considerably between weeks 7 and 8 as they neared the end of treatment. These data help shed light on how the various program elements impacted perceived client progress through time.

Another key factor in PM implementation is that it allows staff and clinicians to track individual clients and their well-being on a periodic basis. To illustrate this, profiles were developed using initial Y-OQ assessments and previous substance use histories, to illustrate how various treatment planning strategies worked for different types of clients. Not all adolescents who entered Base Camp did well or were a good fit for the volunteer program and some were referred to more intensive treatment for a variety of reasons, including undiagnosed trauma or violent and maladaptive behavior. Figure two presents three case studies of individual clients that all discharged at different times while in treatment. Case 1 discharged after 35 days with a Y-OQ score that was trending in a negative direction (up) and was in the range of symptoms that reflect adolescents in need of inpatient care based on normed samples (see Burlingame et al. 2005). Case 1 entered treatment with a high PICS score (80), indicating significant involvement with drugs and alcohol and a drug of choice identified as THC. Case 2 also entered treatment with a high PICS score (76), Fentanyl (a dangerous opioid) as a drug of choice, and Y-OQ scores starting very high and increasing in the short time they were in treatment. Case 3 entered treatment with a very high Y-OQ score, but showed continual and gradual improvement through time and remained in treatment for more than 80 days. Case 3 entered treatment with a lower PICS score of 59 and Methamphetamine as a drug of choice. Progress monitoring, as illustrated by these three different cases, allowed the clinical and youth care staff utilize the protocol to track routine progress, monitor what was occurring in treatment, make informed decisions about the next steps for clients who may not be a good fit for treatment, and thus avoid treatment failure (Fig. 1).

Fig. 1
figure 1

Three case studies illustrating Y-OQ scores assessed during progress monitoring at various intervals

Full size image

Outcomes Associated with PM

Substance Use Involvement

The average total score of 58.82 (an average of slightly over 2.0 for the 29 items) on the SUF suggested that clients have extensive psycho-social involvement with substances across the four domains: Recreational, Effects, Escape, and Social Benefits. In addition, almost 90% of all clients in the sample identified their use of alcohol and/or a specific drug of choice at 20 or more times in the 30 days prior to the assessment. Clients had an early onset of use for alcohol and marijuana (lower than Grade 8) and slightly higher grade level for more serious drugs like opiates or cocaine (Grade 9 and above). Recreational was the highest rated subscale (on a scale from 0—Never to 3—Often) (M = 2.8) with an average score that was at or near 3.0. The three items in this subscale included questions that asked “I have used drugs and alcohol to: (a) Have fun, (b) On weekends, holidays, or days off, and (c) To get high. The next highest subscale was Escape (M = 2.33). These eight questions asked, for example, if they used to “Feel calm or happy” to “Get their mind of their problems” and “Avoid family issues.” The Social Benefits and Effects subscales were below 2.0, indicating that they their use in these domains was less frequent.

Youth-Outcome Questionnaire

A total of 46 of the 57 clients (81%) had admission and discharge scores on the Y-OQ and had remained in treatment for at least 3 weeks. A total of nine clients showed an admission score but then no subsequent scores, and were dropped from this analysis. Table 1 reports client Y-OQ scores at admission that were in the clinical range for symptomatology (M = 68.45, SD = 30.60), indicated by a score greater than 46 on the Y-OQ, implying that the referral and choice to enter residential treatment was warranted. At discharge, scores on average were below the cut score of 46 (N = 46; M = 26.48, SD = 27.86). Therefore, clients were in treatment for an average of 59 days, and on average demonstrated clinically and statistically significant change over time (p < .001) with a large effect size reported (d = 1.38) (see Table 1).

Table 1 Pairwise T test of client Y-OQ scores at admission and discharge and clinical and statistical change
Full size table

At admission, girls scored significantly higher than boys on the Y-OQ [t(50) = − 2.71, p = .009] with girls reporting an average intake score of 83.54 and boys 60.18. At discharge, girls scores (M = 35.76) remained higher than boys (M = 19.60) on the Y-OQ, but the differences in scores were not significant [t(38) = − 1.87, p = .069]. Figure 2 illustrates the results of average Y-OQ scores during the course of the treatment period, beginning with time at admission (YOQ SR0) and ending with week 12 (Y-OQ SR12). The number of clients in each cell falls because clients remain in treatment for different durations and may leave at any time on their own volition, or were discharged early by the program. These data reflect the significant change reported above for clients who remained in the treatment program for an average of 59 days, or slightly over 8 weeks.

Fig. 2
figure 2

Client self-report Y-OQ scores at weekly intervals through treatment

Full size image

Six subscales in the Y-OQ assess a variety of intra- and inter-personal symptoms. Table 2 presents the difference in subscale scores from admission to discharge for each of the six subscales. Clinically and statistically significant change was noted across all six subscales. For each subscale, clients presented at admission with scores above the cut score. The largest effect sizes were found for the Social Problems (d = 1.33), Behavioral Dysfunction (d = .91)) and Intrapersonal Distress (d = .87).

Table 2 Pairwise T test of YOQ subscale scores
Full size table
Change in Mindfulness Traits and Their Relation to Outcome

The results indicated that clients reported a statistically significant increase in scores on the CAMM from admission to discharge [t(42) = − 7.55, p < .001]. When examining the relative contribution that changes in Mindfulness scores had on treatment outcome, a series of regressions were performed and are presented in Table 3. The change in CAMM scores, or the development of mindfulness traits, significantly predicted changes in two subscales but not the total Y-OQ change score. The development of mindfulness traits, as evidenced by a significant increase in scores from admission to discharge, predicted changes in the Y-OQ Intrapersonal Distress (ID) subscale [b = − .77, t(46) = − 2.91, p < .007] and the Y-OQ Interpersonal Relations (IR) subscale [b = − .58, t(46) = − 3.08, p < .005]. The development of mindfulness traits also explained a significant amount of variance in each subscale (ID, 21.8%; IR, 27.5%).

Table 3 Regression analysis of change in CAMM scores and Y-OQ total and subscale change scores
Full size table

Discussion

The results presented in this study illustrate how one program PM can be integrated into an existing adolescent treatment program. The two hypotheses were supported by the results of the study, suggesting that PM, when implemented correctly, can be successfully integrated into the treatment process and yield outcomes that can be useful for programs and stakeholders treating adolescents with SUD. The first finding is that the process of developing and implementing a progress monitoring process within an adolescent SUD treatment program for clinical and youth care staff was feasible and successful. Initially, when the evaluators began working with the program there was some reluctance from some veteran staff to utilize these tools and they were slow to ‘buy in” to the process. This was primarily because they thought the process would take away their clinical and intuitive skills developed from years of working with youth in treatment. The process was successful because relationships and rapport was developed between the evaluators and staff, which allowed the evaluators to suggest certain instrumentation that was reliable (Y-OQ, CAMM, PICS and SUFS), while also working with them to develop instrumentation unique to the program (Program Outcome Questionnaire). Without trust and rapport, this may not have been possible. In addition, the evaluators and program staff immediately began conducting process evaluation to determine if the evaluation protocol was working efficiently and meeting the needs of all levels of staff (Patton 2008). This required routine contact and visits, and alterations of the system if the data collection became too burdensome for the clients and staff. Importantly, clinical and program staff saw first-hand the added value of the data as it helped to inform treatment planning and were more likely to make sure data was collected routinely. This process evolved into the progress monitoring being integrated into the treatment process, where clinical and youth care workers would meet as a team, and with clients individually, on a weekly basis, and have discussions about the real-time findings of the evaluation and use this data to treatment plan for the next week. As Goldberg et al. (2016) point out in an examination of PM implementation, one reason PM can lead to enhanced outcomes for clients and psychotherapists is that PM can help develop what Miller and Hubble (2011) call a “culture of excellence.” Clinicians and staff not only reflect on and view their client outcomes individually through the process, but they discuss clients’ progress in a more formal and structured manner that is more than just telling stories. This is especially the case when clients are not making progress, as was illustrated in two of the case studies. In collaboration with other staff and clinicians, and under the guidance of an external evaluator and evidence—based system, the clinicians and staff could gather concrete suggestions for ways of working with difficult cases.

Keeping the patient experience of the PM process is also critical for organizations to consider as they move forward with PM initiatives. For example, Solstad et al. (2017) conducted a systematic review of patient experiences in PM and developed four “meta-themes” that emerged from an analysis of these experiences: (a) client suspicion toward service providers, (b) a need for flexibility and support to capture the complexity of PM, (c) how clients are empowered through the process, and (d) how collaborative practice is developed. Each of these themes relates to the results developed from this study and serve as a guide for programs interested in implementing PM. For example, the results suggested that when discussing the process and instrumentation with clients, staff noted that it was essential that they were presented in a way that was meant to empower them to have a voice in their treatment process (b), and for clients to be aware that the exercise was not merely a bureaucratic process to appease program directors funders (a). Similarly, the flexibility afforded to the evaluators to change and adapt the process during the process evaluation helped staff and clients feel like they were part of the process (d), because many of them witnessed the shift in protocol that was due to their feedback.

PM also offered client assessments of programmatic elements through time. These self-reported assessments showed consistent and positive trends in their perception of the role that various program factors played in the treatment process. This finding could also be the focus of future analysis to identify how these programmatic elements change through time, and which ones are differentially more important than others. This is an area that is important to consider, because little is known about how process factors relate to treatment outcome, be they program specific constructs like those reported here, or broader adventure therapy-based therapeutic factors like those theoretically discussed (see Russell and Farnum 2004) or that are now being quantitatively assessed in scales like the Adventure Therapy Experience Scale (Russell and Gillis 2017). Lee and Barth (2011) also recommended that research studies follow the group care reporting standard and discuss how specific program activities facilitate desired outcomes, both from a programmatic level or a reliable, and evidence based assessment protocol like the Y-OQ. Further, evaluation efforts could focus on the relative value of asking for client feedback on these programmatic elements, and if clients felt empowered through this process because they, in essence, are given voice in their treatment process. In addition, data sets that are generated through PM efforts could be analyzed in using Hierarchical Linear Modeling (HLM) with process scales like the ATES, to better understand how adventure therapy factors reasoned to be present throughout the experience may impact treatment outcome assessed on a routine or periodic basis (see Russell et al. 2017).

Clients entered treatment at Base Camp with significant psychological and emotional symptoms that warranted treatment, including symptoms of substance use disorder that was extensive and was the focus of treatment. Over the course of treatment, clients showed statistically, clinically and practically significant improvement, as evidenced by significant changes and the large effect sizes reported. These positive outcomes were identified as early as the third week of treatment, and were maintained and further developed throughout the course of treatment, where clients, on average, were then discharged below the clinical cut score of established by normalized populations of adolescents. These score changes are consistent with several studies reporting clinically and statistically significant changes in Y-OQ scores in a variety of adolescent treatment settings similar to Base Camp’s residential model that utilizes short adventure based trips as a supplemental treatment process to residential treatment (Bettmann et al. 2016). For example, Bettmann et al. (2016) reported an overall Hedge’s g of .75 in their meta-analysis of fourteen studies of wilderness therapy programs for adolescents that utilized the Y-OQ as a treatment outcome measure. Gillis et al. (2016) reported an overall Hedge’s g of .98 for 11 studies of the Y-OQ-SR for youth in wilderness or non-wilderness treatment. This study’s reported effect size (d = 1.38) is on the upper range of effect sizes reported in the meta-analysis, which also reported higher effect sizes for “open programs” (programs that have clients entering in and out of an existing group) and for programs that demonstrated direct clinical interventions, which reflects Base Camp’s open enrollment model and clinical approach. These findings are also consistent with several other meta-analyses examining the effects of outdoor and adventure therapy programming (Bedard et al. 2003; Bowen and Neill 2013; Cason and Gillis 1994; Gillis and Speelman 2008; Hattie et al. 1997; Wilson and Lipsey 2000).

As evidenced by pre-post changes on the CAMM, clients also developed mindfulness skills, which were related to symptom reduction, particularly in the Intrapersonal Distress and Interpersonal Relations subscales of the Y-OQ. This finding is consistent with Russell et al. (2015), who were the first to report the use of “mindfulness-based experiences” (MBEs) through adventure in SUD treatment. MBEs are one to 4-day or longer nature-based adventure experiences that include camping, hiking, rock climbing, rappelling, white water canoeing, river crossing, solo experiences, and 24-h hikes. The experiences are intentionally designed to provide participants with an opportunity to practice and develop mindfulness skills related to their treatment goals and be fully present in the moment. Being mindful is discussed often and openly by the clinical staff, the adventure leaders, and the adolescent clients as they work through the process. This study is the first to replicate this finding in an adolescent population and will be the focus of further analysis as the database continues to develop.

The results generated from the study also offer several key findings for the funders and key stakeholders involved with Base Camp. The first is that the program now has an enhanced understanding of the clients that are being referred to the program, and can use these initial findings as a baseline to better determine if Base Camp would be a good fit for certain clients referred to the program. Second, with PM, treatment deterioration and potential referral to alternative settings can be detected using evidence-based instrumentation and protocol, reducing risk to the program and clients. Third, various program elements can be continuously monitored to examine which elements are facilitating change in clients at specific time periods throughout treatment, allowing staff to potentially use them more intentionally and strategically. Finally, summative evaluation of treatment outcome can be used to support how the program fits in the continuum of care of services intended to successfully treat this population.

The results and conclusions of the study should be interpreted with caution due to the limitations of the study. First, a relatively small sample size was used in the study due to the relatively low number of clients that are treated at Base Camp over the course of time. Base Camp is a ten-bed facility that averages 8-weeks of treatment. Therefore, no causal or directional conclusions can be drawn and the sample size of the study limits the generalizability of the findings. Second, it should be noted that Base Camp is a volunteer program, as adolescents are not forced to enter treatment against their will. Many adolescent treatment programs, particularly in the United States, are involuntary, which could affect the interactions that clients and staff have with one another, especially in the first few weeks of treatment. This could affect the communication that clinicians and clients have regarding their progress. Finally, no random assignment or control group was used, which also limits the generalizability of the findings. Future research is needed to replicate PM implementation initiatives in other adolescent settings, particularly in SUD treatment contexts.

In conclusion, this study serves as an example of how a program can institute progress monitoring as a therapeutic process monitor client progress through treatment while providing clinicians and staff data that determines whether clients are moving forward, backward, or standing still (no change). By incorporating measures that are consistent with the mission of the program, meaningful for clinicians, useful for field staff, and helpful for clients to “see” their progress on charts or graphs has the potential increase the likelihood of treatment success and enhance recognition of treatment deterioration. The data generated from PM provided feedback on vital signs for each client as to where they were in their treatment plan, and were achieved through minimal modification to routine practice and which facilitated regular discussions between treatment teams and clients about their clinical progress. Lambert (2017) and others have suggested that PM and feedback methods become a standard of practice, and to wait to until discharge to determine whether treatment has been successful or not is being viewed more and more as an unethical form of clinical practice. This is why we encourage programs to strongly consider investing in progress monitoring, routine outcome monitoring, or feedback informed treatment, which is becoming a core standard of evidence-based practice across a wide variety of treatment settings for youth and adolescents.