Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Introduction

There are two tiers of programs (Tier 1 and Tier 2) in the Project P.A.T.H.S. in Hong Kong (Shek, 2006; Shek & Ma, 2006; Shek & Sun, 2009a, 2009b). The Tier 1 Program is a universal prevention initiative in which Secondary 1–3 students take part. Using a structured curriculum, there are 20 h of training in both core and elective programs in each school year for each grade. Students learn competencies based on the 15 positive youth development constructs as identified in the successful programs identified by Catalano, Berglund, Ryan, Lonczak, and Hawkins (2004).

Besides building up psychosocial competencies in adolescents via the Tier 1 Program, students with greater psychosocial needs are helped via the Tier 2 Program. Because research findings suggest that roughly one-fifth of adolescents would need more help, the Tier 2 Program is provided for at least one-fifth of the students who display greater psychosocial needs at each grade (i.e., selective prevention) by the school social work service providers. The Tier 2 Program (Selective Program) targets adolescents with greater psychosocial needs who are identified in the Tier 1 Program and/or via other sources. Students with greater psychosocial needs usually have special needs in the academic, personal (e.g., adjustment, mental health, and values concerns), interpersonal, and family domains. Information based on multiple sources, including objective assessment tools (e.g., Family Assessment Instrument, Life Satisfaction Scale, Hong Kong Student Information Form), teachers’ ratings, student records, and other relevant quantitative and qualitative information based on systematic assessment, is used to identify students for the Tier 2 Program. Throughout the project, different evaluation studies were conducted to examine the effectiveness of the Tier 2 Program utilizing the subjective outcome evaluation approach. Overall speaking, the findings are very positive, showing that most of the participants had positive views of the program, instructors, and benefits (Lee & Shek, 2010; Lee, Shek & Sun, 2010; Shek, Lee, Sun, & Lung, 2008; Shek & Ma, 2010; Shek, Ma, & Merrick, 2010; Shek & Merrick, 2009; Shek & Sun, 2008). Besides, four major types of program were identified in the Tier 2 programs, including programs based on adventure-based counseling (ABC) approach, programs concentrated on volunteer training and services (VTS), programs incorporating both adventure-based counseling and volunteer training elements, and other programs with different foci. Previous evaluation findings generally showed that these four modes of program did not differ in their evaluation in terms of the subjective outcome evaluation.

Based on the consolidated data on the Tier 2 Program collected in the Experimental and Full Implementation Phases of the project from 2005 to 2009 (213 schools with 60,215 respondents), it was found that the participants generally had positive perceptions of the programs, program implementers, and perceived program effectiveness. It was also found that two-thirds (67.7 %) of all programs adopted the ABC approach as part of or the only program theory. However, similar to the findings based on the separate studies conducted previously, results did not show any significant difference among the four types of programs in participants’ views on the program, instructors, as well as the effectiveness of the program. As the ABC approach is a very popular program theory, it is necessary to investigate if there are differences in the participants’ perceptions on the program effectiveness between ABC-related and non-ABC-related programs. This chapter attempts to examine this question based on nine sets of subjective outcome evaluation data.

The Adventure-Based Counseling Approach

As far as adventure-based counseling is concerned, it is an approach which integrates adventure, wilderness, experiential learning, as well as individual and group counseling (Alvarez & Welsh, 1990; Fletcher & Hinkle, 2002; Hopkins & Putnam, 1993; Lee & Mak, 2002; Quezada & Christopherson, 2005). According to this approach, when an adolescent with a disequilibrium in personal development is put in an environment which is strange and requires cooperation (i.e., adventure environment), the tasks designed providing adventure experiences will lead to transformation in the participant, including changes in self-confidence, self-understanding, and cooperation with others (Gass, 1993; Glass & Myers, 2001; Glass & Shoffner, 2001; Lee & Mak, 2002; Priest & Gass, 1997; Quezada & Christopherson, 2005). According to Schoel, Prouty, and Radcliffe (1988), adventure-based counseling promotes life skills in the participants, including communication, cooperation, decision-making, and problem-solving skills. With regard to the effectiveness of adventure-based counseling, Moote and Wodarski (1997) showed that 16 of the 19 studies under review reported some positive effects for the participants, including enhanced self-esteem, self-concept, cooperative behavior, and physical, social, and intellectual growth. They also concluded that “for social workers who provide direct services to adolescents in various settings, adventure-based counseling may be a viable alternative to traditional approaches” (pp. 161–162).

The Volunteer Training cum Service Approach

The second major mode of Tier 2 programs is closely related to volunteering training and services. According to Clary et al. (1998), there are six functions of volunteering. They are (a) enhanced understanding of the world through volunteering (knowledge function); (b) expression of values via volunteering (value expressive function); (c) avoidance of personal issues or undesirable truths about the self via volunteering (ego defensive function); (d) enhancement of self-esteem, competence, and mood (self-enhancement function); (e) facilitation of career and development of a better resume (utilitarian function); and (f) social companionship and socializing with other volunteers (interpersonal function). Clearly, research findings showed that volunteers perceived several benefits of volunteering, including knowledge and skills acquisition, enhancement of occupational and educational opportunities, and social belongingness among peers (Chapman & Morley, 1999; Cheung, 2006; Hansen, Larson, & Dworkin, 2003; Omoto & Snyder, 2002). Other benefits for adolescents engaging in volunteerism include reduction of anticipated distress and negative emotions (Carlo & Randall, 2002; Nelson & Crick, 1999). Based on these findings, it would be expected that volunteer training and services would promote positive youth development.

It is noteworthy that although it is very common for social work agencies to design programs for adolescents with greater psychosocial needs (e.g., adventure-based counseling, volunteer training program), systematic evaluation and documentation of program evaluation have rarely been found in the local social work literature (Shek, Lam, & Tsoi, 2003). Although recent studies provide solid evidence to support the effectiveness of the Tier 2 Program (Shek & Lee, 2012; Shek & Sun, 2010), an examination of the possible differences in the perceived outcomes among different modes of program is worth noting. Against this background, this chapter examines whether participants joining the ABC-related and non-ABC-related programs differ in their evaluation of the program. Obviously, this examination will stimulate discussion on the choice of program theory in future secondary prevention programs for at-risk adolescents. Besides, predictors of subjective outcome evaluation in the Tier 2 Program are also explored.

Methods

Participants and Procedures

From 2005 to 2009, a total of 93,001 participants (48,212 at the Secondary 1 level, 29,644 at the Secondary 2 level, and 15,145 at the Secondary 3 level) joined the Tier 2 Program across 4 years, where 83,378 were student participants who had greater psychosocial needs and 9,623 were their parents and teachers. The basic characteristics of the participants in the different datasets can be seen in Table 1.

Table 1 Description of data characteristics from 2005 to 2009
Full size table

The first author has developed a Subjective Outcome Evaluation Form for Participants (Form C, Shek & Lee, 2012). Participants were invited to respond to the Form C after completing the Tier 2 Program. From 2005 to 2009, a total of 60,215 questionnaires were collected (mean = 43.74 participants per school, range from 3 to 222) with an overall response rate at 64.75 %. To facilitate the program evaluation, the Research Team has developed an evaluation manual with standardized instructions for collecting the subjective outcome evaluation data. In addition, the program implementers had received adequate training in a 20-h training workshop on how to collect and analyze the data collected by Form C. The participants were invited to respond to Form C after completion of the program.

Instruments

The Subjective Outcome Evaluation Form (Form C) was used to measure the participants’ perceptions of the Tier 2 Program, including the program, instructor, and effectiveness. For the quantitative data, the implementers collecting the data were requested to input the data in an Excel file developed by the Research Team which would automatically compute the frequencies and percentages associated with the different ratings for an item. When the schools submitted the reports, they were also requested to submit the soft copy of the consolidated data sheets. After receiving the consolidated data from the funding body, the data were aggregated to “reconstruct” the overall profile based on the subjective outcome evaluation data by the Research Team. Only quantitative data based on the rating scale items were examined in this study.

Results

The basic characteristics of the Tier 2 Program implemented from 2005/2006 to 2008/2009 school year are listed in Table 1. Table 2 presents the characteristics and perceived effectiveness of four different types of programs, including the number of participants, program attendance, number of program aims and constructs, as well as the mean overall effectiveness. Apart from students, some programs also involved parents and teachers. Among the four program approaches, Type A (ABC plus VTS) was the most widely employed approach, which was used in 525 out of 1,326 programs (39.6 %). This was followed by Type B (ABC only), which accounted for 373 programs (28.1 %), and then Type C (VTS only, 220 programs, 16.6 %) and Type D (approaches other than ABC or VTS, 208 programs, 15.7 %). The average number of participants ranged from 50.58 to 71.18, with the average program attendance ranging from 81.15 to 86.06 %.

Table 2 Summary of the characteristics and perceived effectiveness of the Tier 2 Program
Full size table

The quantitative findings based on the closed-ended questions among four different program approaches are presented in Tables 3, 4, and 5. Over four-fifths of responses regarding participants’ views toward program, implementers, and perceived effectiveness were positive. From Tables 3, 4, and 5, it was observed that Type A programs received the highest scores in all items. For example, 86.89 % of participants opined that “the activities were carefully planned” in Type A program (Table 3) but only 85.36, 85.39, and 85.81 % positive responses were recorded for Types B, C, and D programs, respectively. While 91.83 % of Type A participants were satisfied with the worker (Table 4), 89.48, 90.74, and 90.73 % of the participants of Types B, C, and D were satisfied with the worker, respectively. However, regarding possible differences among the four program types in different measures (views toward program, implementers, and perceived effectiveness) of subjective outcome evaluation, results of ANOVA did not show any significant difference (p > .05 in all cases).

Table 3 Comparison of the positive views toward the Tier 2 Program among four different program approaches
Full size table
Table 4 Comparison of the positive views toward the Tier 2 Program workers among four different program approaches
Full size table
Table 5 Comparison of the positive views toward the Tier 2 Program effectiveness among four different program approaches
Full size table

To further examine the effects of adventure-based counseling, the program approaches were further re-categorized as ABC related (i.e., Type A and Type B programs) and non-ABC related (i.e., Type C and Type D programs) for further analysis (Tables 6, 7, and 8). Similarly, no significant difference was found between the two major modes of program in terms of perceptions of the program, implementers, and perceived effectiveness. The mean scores of the key variables for the two major modes of program are presented in Table 9. Regarding the predictors of perceived effectiveness of the Tier 2 Program, analyses showed that perceived quality of the program and instructor positively predicted perceived effectiveness of the program (Table 10).

Table 6 Comparison of the positive views toward the Tier 2 Program between ABC and non-ABC approaches
Full size table
Table 7 Comparison of the positive views toward the Tier 2 Program workers between ABC and non-ABC approaches
Full size table
Table 8 Comparison of the positive views toward the Tier 2 Program effectiveness between ABC and non-ABC approaches
Full size table
Table 9 Mean, standard deviations, Cronbach’s alphas, and mean of inter-item correlations among the variables by ABC and non-ABC program approaches
Full size table
Table 10 Multiple regression analyses predicting program effectiveness
Full size table

Discussion

One unique feature of this study is that the total number of adolescents participating in the Tier 2 Program held between 2005 and 2009 was very large, with a total of 1,326 programs involving 83,378 students. Besides, it is the first systematic study in different Chinese contexts examining the perceived effectiveness of ABC versus non-ABC programs. Overall speaking, the descriptive findings showed that both ABC-related and non-ABC-related programs were well received by the program participants. The participants also had very favorable perceptions of the instructors and benefits of these two modes, and they did not differ in terms of subjective outcome evaluation findings.

There are several plausible explanations for the finding that there were no significant differences between the two different program modes. First, the present study focuses only on client satisfaction. Although the use of a standardized assessment tool with known reliability and validity for conducting client satisfaction survey reduces biases and eliminates many of the problems commonly found in hastily designed questionnaires (Royse, 2004), subjective outcomes for all programs of the project were grouped into common elements, and therefore the questionnaires used may be insensitive to the unique features of different programs. The differentiation power of the standardized measure for the two program modes might be reduced eventually. In short, one cannot kill two birds with one stone.

Second, as reflected in the results of the evaluation of the perceived perceptions of the programs, the program implementers, and perceived program effectiveness for the four different types (Tables 3, 4, and 5) and the two modes (Tables 6, 7, and 8), the results on all items were consistently positive at the higher end with a minimum of 81.18 % and a maximum of 91.83 % positive ratings. Furthermore, the largest difference in all item ratings for the four different types of programs on the three measures was only 8.49 % (81.18–89.47 % in Table 3, 84.98–91.83 % in Table 4, and 80.06–88.55 % in Table 5). The same situation is found in the analysis of the two program modes (81.43–88.42 % in Table 6, 86.02–90.98 % in Table 7, and 80.82–88.11 % in Table 8) with a maximum variation of 7.9 % in all item ratings. The consistently positive ratings with little variation among all items regardless of program types or modes may explain the finding of no difference (i.e., ceiling effect). In fact, this is a limitation of subjective outcome evaluation that the related ratings are commonly found to be on the higher end.

Third, although ABC has been commonly described in terms of its “magical” effect in transforming young people, young people may also be transformed in subtle ways in voluntary work. Through voluntary work, young people can model and learn desirable behavior. Besides, voluntary work can also benefit the overall development of young people in the areas such as self-confidence, self-understanding, self-efficacy, interpersonal skills, and compassion.

One of the limitations of the present study is that it is just a client satisfaction study. However, it is one of many evaluation strategies adopted for evaluating the whole project. Since the project consisted of two tiers, the subjective outcome evaluation is part and parcel of a comprehensive evaluation of the whole project. Moreover, the use of a validated subjective outcome evaluation measure on a large population of Chinese adolescents is a solid contribution in responding to the comments of Royse (2004) that the lack of standardized assessment tools for conducting client satisfaction survey introduces biases for the client satisfaction approach.

Since we could not find any significant difference in the subjective outcomes among the four types of programs, and between the two modes as reported in this chapter, objective outcome evaluation studies are recommended in future research. However, it must be noted that even though the two approaches address the same target systems (knowledge, value, belief, emotion, behavior), they are targeting different positive youth development constructs and are expected to generate different outcomes. For example, the adventure-based counseling is widely adopted as a developmental and therapeutic approach with adolescents and young people. It helps the participants develop group cohesion (Glass & Benshoff, 2002), enhance their sense of well-being and social self-concept, develop general self-efficacy and positive emotions that could be transferred to their daily lives, develop the appropriate inner resources to engage in new challenges, and enhance their spiritual development (Cheung, 2010; Glass & Benshoff, 2002; Kyriakopoulos, 2010). In a meta-analysis of 43 studies, Cason and Gillis (1994) showed an average of 12.2 % improvement for adolescents participating in an adventure program. Similarly, joining volunteer community service program enhances secondary students’ self-esteem (Meinhard & Foster, 2006), young adults’ positive affect and mental health (Pavlova & Silbereisen, 2012), as well as adults’ positive attitude, social interaction, and professional development (Miller et al., 2002).

It must be pointed out that comparative analyses among these 1,326 programs are very complicated because of the variations in the design of programs for adolescents studying in different schools. Nevertheless, for program theory development and a rational choice of the program theory in designing secondary prevention programs for adolescents, further investigation of the possible differences in the subjective and objective outcomes in relation to types and modes according to the espoused theory (Argyris & Schön, 1974) is fully justified. The use of randomized control-delayed treatment design should also be carried out.

Although the ABC approach has been popular among social workers working with children and adolescents, it is comparatively more expensive because of its labor intensiveness and special requirements in staffing and equipment. Compared to the VTS approach, the ABC approach needs much more resource as it requires different levels of specialized training of the coach and a higher participant-to-coach ratio because of safety consideration. ABC intervention program also requires specialized facilities, including appropriate venue and equipment for holding low- or high-risk activities. As the use of ABC approach in general requires a higher cost in program expenditure than the use of the VTS approach, it is necessary to construct cost-benefit analyses for these two modes of intervention in the future.

Consistent with the previous studies, the finding showed that both perceived program attributes and instructor qualities predicted perceived program effectiveness (Shek & Lee, 2012). Nation et al. (2003) pointed out that a comprehensive program and well-trained program implementers are important elements of an effective program. Weissberg (2000) also pointed out that a well-designed program and high-quality program implementers were commonly found in effective school-based social-emotional learning programs. There are also research findings showing that qualities of the program implementation and program implementers are related to program outcomes. For example, Harachi and colleagues (Harachi, Abbott, Catalano, Haggerty, & Fleming, 1999) showed that instructional strategies (proactive classroom management, cooperative learning methods, strategies to enhance student motivation, student involvement and participation, reading strategies, and interpersonal and problem-solving skills training) were related to student social competencies. Tobler and colleagues (Tobler, Lessard, Marshall, Ochshorn, & Roona, 1999) also showed that programs with high peer interaction were more effective than programs with low peer interaction, and that the delivery method instead of the program content determined the success of the program. The present study further showed that both the program and instructor qualities are important determinants of perceived effectiveness of positive youth development programs in the Chinese contexts. The findings suggest that developing high-quality program and implementing them in a quality manner are instrumental to the success of positive youth development programs. Furthermore, as quality of the program implementers is important to program success, systematic and rigorous training for the potential program implementers is important for promoting program success.