Abstract
Youth development programs can achieve positive social outcomes, however studies comparing the influence of different program components are rare. Structural equation modeling of longitudinal, multilevel data (N = 327) from Project K, a multi-component youth development program, assessed how experiences of engagement or support in each component affected social outcomes. Participants reported significant gains in social self-efficacy and sense of community after the program. Engagement in the outdoor adventure and support during the mentoring partnership components significantly contributed to observed social gains, while engagement in the community service component did not. Results confirm youth development programs can positively influence adolescent social development, while highlighting the importance of moving beyond “black box” investigations in order to maximize program impact and efficiency.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Introduction
Humans are inherently social creatures and a sense of belonging appears to be a prerequisite for a healthy and happy life (Baumeister and Leary 1995). Adolescence is an important time for the development of social ties outside the family and cultivating social skills and connection can protect young people from a range of aversive outcomes (e.g., Nilsen et al. 2013). Youth development programs can be important extra-familial social contexts but, with studies that compare the influence of various program components still very rare (Roth and Brooks-Gunn 2016), little is known about how they support participants’ social development. Project K is a New Zealand-based, multi-component youth development program targeted at adolescents with low self-efficacy. This study builds upon previous evidence that Project K contributes to positive social outcomes for youth (Deane et al. 2016) by investigating the relative contributions made by different program components.
The Protective Qualities of the Social World
A meaningful sense of connection with others is argued to be a fundamental human need (Baumeister and Leary 1995; Ryan and Deci 2000). Meaningful connection can be accessed through interpersonal relationships or from a sense of belonging in groups, contexts, and wider communities (Baumeister and Leary 1995; McMillan and Chavis 1986). A sense of meaningful connection with others appears to confer a range of valuable protective qualities during adolescence and is associated with important life outcomes, such as reduced rates of depression and substance abuse (e.g., Mayberry et al. 2009; Nilsen et al. 2013).
Given the protective value of social bonds, cultivating social resources—whether through social skill development, improved social confidence, or directly facilitating connections—is an essential component of youth thriving. A range of published program evaluations indicate that young people can develop essential social resources, including social skills, peer relationships, and group-level belonging, through educational and out-of-school programs (e.g., Deane et al. 2016; Durlak et al. 2010; Thurber et al. 2007). However, less research has focused on how programs achieve such outcomes. Below we discuss evidence that both participant engagement and experiences of support are key drivers of youth program success, before highlighting gaps in current research that this article aims to address.
Promoting Positive Social Development through Youth Programs
Two mechanisms through which youth programs may promote social development are participant engagement and experiences of support. Engagement is conceptualized as consisting of three key components: affective (i.e., interest, enjoyment), behavioral (i.e., effort), and cognitive (i.e., seeing value in the activity; see Conner and Pope 2013; Wang and Eccles 2012). Beyond mere participation, adolescents who are more engaged in program or educational activities typically show greater benefits—both in general (Conner and Pope 2013; Lekes et al. 2010; Low et al. 2014) and specifically in relation to social outcomes (Krauss et al. 2014; Li and Lerner 2011; Mahoney et al. 2007; Ramey et al. 2010; Shernoff 2010). Thus, engagement appears to be a key mechanism through which youth programs achieve positive outcomes for participants.
Support received from adults in program contexts has also been shown to be associated with a range of positive outcomes (e.g., Zeldin et al. 2014). Although social support can take many forms (see Wills and Shinar 2000 for overview), emotional support has been shown to be particularly relevant for adolescents’ engagement and sense of connection (Ruzek et al. 2016). Emotional support includes access to people who express genuine concern and are willing to listen without judgement (Lakey and Sheldon 2000). Feeling supported by contextually relevant adults (such as teachers, mentors, or other program staff) has been especially linked to positive social outcomes (DuBois et al. 2011; Krauss et al. 2014; Ruzek et al. 2016; Wang and Eccles 2012). For instance, mentors typically provide emotional support (Schwartz et al. 2013) and may help facilitate socio-relational development by modeling positive relationships and helping youth to understand and effectively express emotion (Ahrens et al. 2011).
Unfortunately, as highlighted by Roth and Brooks-Gunn (2016), much of what is known about the processes through which programs achieve positive outcomes, including engagement and supportive adult-youth relationships, draws from research on a broad range of organized youth activities, many of which would not be considered youth development programs per se. Futhermore, research that systematically tests the influence of different program components is rarely seen. Understanding exactly how programs achieve results is becoming increasingly important in a climate where resources are stretched and youth development organizations are required to make efficiencies while evidencing positive impact (Arnold and Cater 2011; Roth and Brooks-Gunn 2016). It is difficult to improve programs if the critical aspects remain unknown—a problem of particular relevance for multi-component programs like Project K. The aim of this article, therefore, is to contribute to the advancement of youth development theory and practice by assessing how engagement in, and support experienced, during the different components of one youth development program, Project K, help to promote positive social development among its participants.
Project K: A Youth Development Program for Adolescents with Low Self-Efficacy
Project K is a multi-component youth development program consisting of an outdoor adventure experience, a community service project, and an adult-youth mentoring partnership. Participants are 13–15 year old high school students with low academic and social self-efficacy relative to their peers (see Deane et al. 2016 for a detailed overview of the participant selection process). The program is owned by the Graeme Dingle Foundation, which provides governance, training, materials, and research and evaluation to Community Partners that are licensed to deliver the program in several regions across New Zealand.
The Community Partners are encouraged to adapt the program activities to suit their community context and needs. They are all, however, required to deliver the core features of the program in a consistent manner. Standardized training and implementation manuals and ongoing monitoring are provided by Graeme Dingle Foundation’s National Support Office to ensure fidelity across the program sites. Community Partners are required to deliver the three core program components in the same sequence over a 14 month period. First, participants take part in a 3-week wilderness adventure. At an isolated bush camp they engage in challenge-based and team-building activities with the objective of developing key skills, such as goal-setting, communication, problem-solving, and leadership. Students then put new skills into practice during a 7–10 day wilderness trek, where they take turns as group leader. Next, participants have the opportunity to reinforce and integrate their new skills in their own communities during a community challenge. The young people engage in workshops on health and wellbeing (e.g., relationships, alcohol, nutrition) and are encouraged to explore community resources, meet local leaders, and develop a collaborative project to address an identified community need. Facilitators of both the wilderness adventure and community challenge are trained to have high expectations of all participants and offer efficacy-enhancing and relatedness support where possible. Finally, participants are matched with a trained adult mentor for a 12-month period. Mentors are adult volunteers from the participants’ own communities who have been selected following a screening process that seeks mentors with qualities such as empathy, good listening skills, and the ability to motivate and inspire. After a social event that provides the opportunity for all mentors and mentees to interact, matches are based on both mentee and mentor preferences, with the former taking priority. Mentors take part in a 20 hour training program and receive ongoing support to develop non-judgemental relationships with the young people. They are expected to meet with mentees fortnightly to help them to set and achieve four goals (one of which must be academic and another related to health) over the course of the year.
The Current Study
As previously demonstrated in a randomized controlled trial, Project K participants significantly improve in social self-efficacy (i.e., believing one can succeed, here in the social domain; Bandura 1977) over the course of the program relative to a control group, an effect corroborated by parental observations and maintained 1 year after completing the program (Deane et al. 2016). Nevertheless, the mechanisms proposed to drive such social gains have not yet been empirically tested. Project K’s theory of change (see Deane and Harré 2014a for more detail) suggests that the program promotes growth through an iterative experiential learning cycle that is reinforced with ongoing support across different contexts. The intense wilderness adventure sets this process in motion by providing opportunities to develop mastery in a supportive but unfamiliar social setting away from everyday distractions. It encourages bonding with peers and development of social resources and self-efficacy. The community challenge enables transfer of the wilderness adventure learning to a familiar context and connects students to people and resources in their community. It has been specifically designed to increase the adolescents’ sense of belonging and efficacy within their communities. The adult mentor then dedicates individualized attention to supporting the goals set by the participant during the wilderness adventure. Theoretically, each component should build on the previous one to consolidate prior learning and sustain outcomes. As a whole, the program alternates challenging experiences with self-reflection aimed at consolidating and transferring learning. Participant engagement is placed at the very centre of the experiential learning process proposed to drive Project K program outcomes, while mentor support is theorized to be another key factor in determining program success.
This study empirically tested the contribution made by each of the three Project K program components to positive social outcomes. Given evidence discussed above that engagement and support are two key drivers of program outcomes, we tested how participants’ affective engagement in the wilderness adventure and community challenge components plus experiences of support during the mentoring partnership, both individually and in unison, predicted improvements in social resources over the course of the program. Specifically, it was hypothesized that Project K participants would experience significant social gains over the course of the program (H1), measured by self-reported social self-efficacy (H1A) and sense of community (H1B), and corroborated by parents’ perceptions of social competence (H1C). Given the additive model outlined in Project K’s theory of change (Deane and Harré 2014a), participants’ emotional engagement in the wilderness adventure (H2) and community challenge (H3) as well as their experiences of emotional support from mentors (H4) were all expected to independently explain social gains, after accounting for baseline levels.
Method
Procedure and Participants
All questionnaires were embedded within the Graeme Dingle Foundation’s internal evaluation plan for programs delivered in 2009 and 2010. After securing ethical approval from the University of Auckland’s ethics committee, de-identified data were shared with the researchers for scholarly purposes. Three-hundred and sixty-five Project K youth participants took part in this research over the course of 2009 and 2010. Prior to analysis, 38 cases were removed because they were missing data on more than 40% of the combined survey itemsFootnote 1. The final sample (N = 327) included 183 males and 144 females. Although participant age was not recorded, Project K is implemented in Year 10 of the New Zealand school system; thus the majority of participants were aged between 13 and 15 years at the start of the program. Almost half of participants (153 participants) were of European ethnicity, 49 were Māori (indigenous New Zealanders), 53 of Pacific Island ethnicity, 9 Asian, and 5 identified with Other ethnicities. The remaining 58 identified with more than one ethnicity (including 31 Māori/European, 15 Māori/Pacific, and 5 Pacific/European). In total, 168 program participants completed the surveys in 2009 and a further 159 in 2010, as part of 31 different program deliveries (independent clusters) from 23 high schools across New Zealand. Each Project K delivery aims to serve 12 young people over the 14-month program, of whom on average just under 11 participated in the research (M = 10.55, SD = 1.93).
Pre- and post-program assessment
The pre-program questionnaire was distributed to all participants before the start of the program (Time 0) either by Project K regional directors or trained representatives and assessed baseline levels of relevant social measures. The same questionnaire was also administered at program completion (Time 4). One parent or caregiver for each participant consented to participate in the research and completed a questionnaire assessing their perceptions of their child’s social competence at Times 0 and 4.
In-program assessment
Questionnaires assessing participants’ experiences were administered by trained program staff within 1 week of completing the wilderness adventure (Time 1) and community challenge (Time 2). Finally, questionnaires assessing perceptions of the 12-month mentoring component were administered up to 2 weeks before program graduation (Time 3).
Measures
Measures analyzed in this study (described in detail below) represent a subset of the entire research program; full questionnaires are available upon request. Reverse scored items across the survey package caused problems for factor reliability and have been removed from analyses, a point returned to in the discussion.
Social self-efficacy
Participants self-reported their beliefs about their social abilities using 8 items from the Social Self-efficacy subscale of Moore’s (2005) self-efficacy questionnaire (e.g., “How well can you have a chat with an unfamiliar person of your age?’; 1 = Not at all, 6 = Very well). Higher scores indicate higher social self-efficacy; α = .84 and .86 at times 0 and 4 respectively.
Sense of community
Seven items were adapted from the brief sense of community scale (Peterson et al. 2008) to capture participants’ degree of group-level belonging in their communities (e.g., “I feel connected to this community”; 1 = Strongly disagree, 5 = Strongly agree). Higher scores indicate a stronger sense of community; α = .86 and .90 at times 0 and 4 respectively.
Parental perceptions of social competence
Parents or caregivers responded to six items from the Prosocial/Communication Skills subscale of Corrigan’s (2002) Social Competence Scale (Parent Version) in order to assess their perspectives of their children’s social skills (e.g., “Your child is very good at understanding other people’s feelings”; 1 = Not at all, 5 = Very well). Higher scores indicate that caregivers perceived participants’ to be more socially competent; α = .80 and .81 at times 0 and 4 respectively.
Engagement
Three items from the interest/enjoyment subscale of the Intrinsic Motivation Inventory (SDT 2016) measured participants’ affective engagement in the wilderness adventure and community challenge (e.g., “I enjoyed the Wilderness Adventure/Community Challenge very much”; 1 = Not at all true, 7 = Very true). Higher scores indicate higher affective engagement; α = .83 and .87 for the wilderness and community components respectively.
Support
Perceptions of the emotional support received from mentors was measured with six items from the Emotional Support subscale of the Family Support Scale (Wills et al. 1992). The scale was modified such that “parent” was replaced with “mentor” (e.g., I could share my feelings with my mentor”, 1 = Strongly disagree, 7 = Strongly agree). Higher scores indicate participants’ reported receiving higher levels of emotional support from their mentors; α = .92.
Analysis
Missing values
Missing Value Analysis was carried out using IBM SPSS 23. Little’s MCAR test suggested that data were missing at random, χ 2 = 24,321.12, df = 24,118, p = .177, therefore imputations were carried out using the Expectation-Maximization algorithm. Where imputed values fell outside the original scale boundaries, they were modified to fit within original scale boundaries.
Model specification
Means and Variance Adjusted Maximum Likelihood methods were chosen to estimate model fit in factor analytical work due to moderate skewness in the data (see Maydeu-Olivares and DiStefano 2016). After using exploratory (oblimin rotation) and confirmatory factor analytic procedures in MPlus 7.35, two social outcomes measurement models (Times 0 and 4) and program experience factors were specified and assessed. Both social outcome measurement models (each for Times 0 and 4) and the measurement model for the three program experience factors (wilderness adventure engagement, community challenge engagement, and mentor support, measured at times 1, 2, and 3 respectively) all met established requirements (Fan and Sivo 2007; Hu and Bentler 1999; Voorhees et al. 2016) of model fit and both convergent and discriminant validity. This suggested that the outcomes at each time point represented distinct social constructs, and the program experiences also represented distinct domains.
A longitudinal invariance test was also carried out to test the stability of both versions of the three-factor social outcome measurement model over time. The longitudinal model met the requirements (Cheung and Rensvold 2002; McArdle 2007) for configural, metric, and scalar invariance, suggesting that the three distinct domains of social engagement were stable over time. Therefore, latent factor means could defensibly be compared across time points. Tests of model fit are summarized in Table 1 and in Online Resources 1 and 2.
Results
Promoting Social Development
Changes on social measures were assessed using z-tests, which are equivalent to paired sample t-tests but specifically used to test changes in latent factors. Overall, participants showed large, statistically significant improvements in social self-efficacy from baseline (M = 4.11, SD = 0.93) to post-program (M = 4.68, SD = 0.64), p < .001, Cohen’s d = 0.76. Participants’ also reported a small but significant increase in their sense of community from baseline (M = 3.35, SD = 0.78) to post-program (M = 3.74, SD = 0.79), p < .001, d = 0.34. However, the same gains were not perceived by caregivers, with no significant difference observed between parental perceptions of self-efficacy from baseline (M = 4.12, SD = 0.73) to post-program (M = 4.11, SD = 0.85), p = .87 (effect sizes interpreted in accordance with Hattie 2009 ).
Assessing the Impact of Program Components
Modeling project K
For the purpose of establishing a structural model that demonstrated the effects of engagement in the wilderness adventure and community challenge as well as mentor support on the social outcomes of interest, several auto-regressive structural models were specified (see Table 1). The first structural model included the three Time 0 factors and three Time 4 factors with direct effects only. A second model included all cross-lag effects. Finally, a third structural model included all three midway factors and cross-lag effects. Assessment of model fit indices for these models suggested a general improvement in fit as more paths and factors were included. The robustness of this final model was assessed by exploring a series of alternative nested models in which different non-statistically significant cross-lagged paths were removed (i.e., those between T0 and midway; midway and T4; and T0 and T4 factors). The related series of chi-square difference tests showed no statistically significant shifts in model fit for any of these alternatives. The final model containing all three midway and cross-lag effects was therefore retained as it represented the data well and also accounted for the negligible associated cross-lag effects.
Accounting for program clusters
Although intra-class correlations (presented with design effects in Table 2) suggested that the portion of individual differences in pre-program social resources attributable to clustering of participants was small, cluster effects were identified in relation to engagement in the wilderness adventure and community challenge components. Multilevel analysis was therefore employed (using MPlus 7.35) to account for the effect of clustering within the 31 different program deliveries. Accounting for clustering, all item-factor loadings remained statistically significant (p < .001), and there were no substantive changes in the structural regression and correlational coefficients between factors. Model fit was also acceptable. In addition, the requirements for configural, metric, and scalar invariance for the final model were met across gender and there were no meaningful differences in structural regressions by gender groupFootnote 2 (see Online Resources 3–5).
Program components
Overall, Project K participants reported relatively high levels of affective engagement in both the wilderness adventure (M = 5.73, SD = 1.20) and community challenge (M = 5.26, SD = 1.41) and perceived high levels of emotional support in mentor partnerships (M = 5.72, SD = 1.09). Pre-program parental perceptions of social competence were positively associated with participants’ experiences of mentor support (ß = .18, p < .05), but no other baseline measure had any statistically significant association with experiences within the program. The final model, accounting for program delivery clusters using multilevel modeling, is summarized in Fig. 1.
Accounting for baseline levels and clustering effects, participants’ engagement in the wilderness adventure (ß = .21, p < .01) and their experiences of receiving emotional support from mentors (ß = .13, p < .05) both significantly contributed to their post-program perceptions of social self-efficacy, R 2 = .29, p < .001, f 2 = .41 (large effect size; Cohen 1992). The same pattern of results was observed for post-program sense of community, R 2 = .30, p < .001, f 2 = .43 (large), which, after accounting for baseline levels, was significantly predicted by engagement in the wilderness adventure (ß = .12, p < .01) and mentor support (ß = .25, p < .001). Although engagement in the community challenge was significantly associated with wilderness adventure engagement (Φ = .44, p < .001), experiences in the community challenge did not uniquely predict gains on any social measure. Finally, the model suggested that neither engagement in the wilderness adventure and the community challenge nor mentor support affected parental perceptions of social competence, which did not significantly change over the course of the program.
Discussion
This article examines if and how experiences of engagement or support during each of the three core components of Project K—a wilderness adventure, community challenge, and mentoring partnership—predicted positive social outcomes for the youth participants. By doing so, the results contribute to understanding how youth development programs enhance participants’ social development, a key task of adolescence. First, and consistent with an earlier randomized controlled evaluation, it was found that Project K participants reported significantly improved social skills and connectedness over a 1-year period (supporting H1), controlling for baseline social resource levels and accounting for clustering effects. As predicted, participants reported significant gains in both social self-efficacy (H1A) and sense of community (H1B). This study, however, finds no evidence that such gains were also perceived by caregivers (contrary to H1C). In regard to the core theoretical question of this article, it was found that participants’ affective engagement in the wilderness adventure (H2) and the levels of emotional support they received from mentors (H4) both significantly and uniquely contributed to their observed social gains, with engagement in the wilderness adventure being the stronger predictor of social self-efficacy gains and mentor support being the stronger predictor of improved sense of community. No evidence, however, was found that engagement in the community challenge contributed to participants’ social development (contradicting H3). This result stresses the importance of investigating the unique contributions made by program components in order to maximize impact and inform program improvements.
In relation to the current study, Project K’s theory of change (Deane and Harré 2014a) suggested that all three core components of the program should contribute to positive social outcomes for participants. Results, however, indicate that only the wilderness adventure and mentoring partnership did so. Outdoor adventure programs such as the wilderness adventure have been shown to promote social connection for participants, in part because they create intense experiences through geographic isolation, shared challenge, and interdependence (see Deane and Harré 2014b; McKenzie 2000). Further, interviews previously conducted with Project K participants have revealed that they experienced the wilderness adventure as socially demanding, requiring both leadership and teamwork from them (Hollis et al. 2011). It was therefore anticipated that engagement in the wilderness adventure would be significantly associated with social gains, and this was confirmed by the data. This result corroborates previous studies showing that outdoor adventure programs can promote positive social development in young people (Hattie et al. 1997; Wang et al. 2006; Whittington and Mack 2010). It also demonstrates that engagement, a mechanism shown to promote positive social outcomes for youth more generally (e.g., Li and Lerner 2011; Ramey et al. 2010; Shernoff 2010), is critical in driving the impact of outdoor adventure activities nested within multi-component programs.
Mentoring partnerships provide ongoing, one-on-one relationships that young people may use as templates to cultivate more positive relationships generally (DuBois et al. 2011; Ruzek et al. 2016; Schwartz et al. 2013). Mentors can also broaden mentee horizons by providing access to social capital, exposing youth to new experiences, people and resources in their communities (Rhodes and Lowe 2009). Supportive non-parental adults working with youth, like mentors, should pay attention to the unique characteristics of their mentee, including their cultural and personal background, and thus help to cultivate a sense of belonging (Heinze et al. 2010; Roth and Brooks-Gunn 2003; Urban 2008). Supportive mentoring was expected to contribute to participants’ ability to navigate the social world and to enhance their sense of connection with the wider community, as results here demonstrate. We build upon existing evidence that experiences of support influence program outcomes (e.g., Krauss et al. 2014; Wang and Eccles 2012) and demonstrate how support experienced in just one component of a program can influence overall social outcomes.
It was particularly surprising to find that the community challenge did not also contribute to participants’ social development, given that it was specifically designed to promote a positive connection with wider communities, which was expected to translate into an improved sense of community. One possible explanation for this finding is that, because sense of community is measured in relation to an unspecified “community”, participants may be primed by the context in which the questionnaire was given to consider their program cohort as their most salient community. If so, the intensity of the wilderness adventure could be the solidifying factor that promotes a connection with the cohort. This supposition is strengthened by the moderate positive association between wilderness adventure and community challenge engagement and the observed clustering effects for those experiences, indicating cohort effects affect program experiences. Another explanation is that the Sense of Community measure used was not suitable for youth participants. There have been suggestions that adult measures are not necessarily appropriate for young people (Chipuer et al. 1999; Evans 2007). For example, young people in early or mid-adolescence may be more concerned with safety and less concerned with influence—at least in the sense of changing social structures—than adults (Chipuer et al. 1999). The measure used here included influence and did not include safety, which may have weakened its validity. Furthermore, the community challenge itself was designed to involve students in a project to address a community need. This one-off contribution is unlikely to have shifted their general perceptions of having influence within their communities, which is what was measured. Since this study was designed, a specific youth sense of community scale has been developed and used successfully (Chiessi et al. 2010; Cicognani et al. 2015). Future research would benefit from employing a more youth-oriented measure that both captures the unique concerns and sensitivities of adolescence and is more closely aligned with what the program component was designed to achieve.
Although it is always difficult to separate conceptual from implementation-related causes, earlier qualitative research on Project K suggests that both conceptual and implementation flaws can help explain why the community challenge did not appear effective. Project K participants interviewed by Warren (2005) and program staff involved in Project K’s theory of change development (Deane and Harré 2014a) reported that it was often difficult to engage participants in the community component and to ensure the project was student-driven. A potential mismatch between adolescent community priorities (e.g., safety) and those the community challenge is designed to promote (e.g., influence) could help explain this lack of engagement. Furthermore, several Community Partners have advised the Graeme Dingle Foundation that participating schools have increasing concerns about the time taken away from studies for students to complete the wilderness and community components, which has resulted in some Community Partners deviating from the initial design. We acknowledge that the community challenge component could be improved in line with other evidence-based community service youth programs and may benefit from a special focus on addressing youth concerns over community safety. However, the absence of strong supporting evidence in favor of the community challenge at present, coupled with the pragmatic resource and logistical concerns of Community Partners, has lead to a recent decision by the Graeme Dingle Foundation to redesign and reduce the length of this program component. Youth development programs must necessarily make choices about the use of resources and these will always be made under conditions of uncertainty, using the best evidence available.
Lastly, although an earlier randomized control trial (see Deane et al. 2016) suggested that social gains reported by Project K participants relative to a control group were also perceived by parents, the current study did not corroborate this finding. Research suggests that the protective qualities conferred by social resources depend more on self-perception than objective indicators of belonging (e.g., Pressman et al. 2005). Therefore, although parental corroboration is a valuable objective measure of social functioning, the evidence presented here that participants perceive improvements in their social skills and connectedness is perhaps a more meaningful measure of program success in the context of youth thriving.
Extrapolating beyond the specific context of Project K, this research adds further evidence that youth programs can effectively nurture the development of core social resources in young people (corroborating Krauss et al. 2014; Li and Lerner 2011; Mahoney et al. 2007; Ramey et al. 2010; Shernoff 2010). In particular, answering Roth and Brooks-Gunn’s call (2016) and building upon earlier research (e.g., Low et al. 2014; Zeldin et al. 2014), our longitudinal data provide strong evidence that participants’ experiences of engagement and support in the specific context of a youth development program can drive positive social development. At a broader level, our findings highlight the importance and value of program research and evaluations moving beyond so called “black box” methodologies (assessing pre-to-post changes) and seeking to understand how programs achieve observed results. This is particularly important in the current resource-constrained climate where evidence-based decisions to increase efficiencies are encouraged (Arnold and Cater 2011; Roth and Brooks-Gunn 2016).
We have tested the relative influence that experiences within each of the three core components of Project K exert on social outcomes. This is only the first step toward a richer understanding of how, why, for whom, and under which conditions youth development programs work. It is not generally possible to test all of the theoretical links proposed in a program theory within a single study (see Weiss 2000). The focus here has therefore been on engagement and support, two mechanisms that have been shown to have a positive influence on youth program outcomes elsewhere (e.g., Conner and Pope 2013; Krauss et al. 2014; Low et al. 2014; Zeldin et al. 2014). Other mechanims of change remain to be tested, as do the experiences and program aspects that themselves create engagement and feelings of support. Our data show that program engagement varies systematically across deliveries. Deane and colleagues’ (2016) randomized controlled trial study previously demonstrated that program outcomes vary according to gender, ethnicity, and the affluence of the school community. Identifying moderators is therefore likely to be a fruitful avenue for future research. Through empirically testing the links proposed in program theories, as has been started with Project K in this instance, a sound evidence base can be built for future youth development initiatives to draw upon.
It is a strength of the current research that measures were collected longitudinally from before, during, and after the program. Cross-lagged modeling to assess changes in social resources over time allowed us to investigate the additive and potentially causal contributions that Project K components made to observed changes. Furthermore, multilevel modeling was employed to account for the fact that participants’ experiences within the program were at least partially influenced by the climate of the particular cohort. A few limitations also warrant discussion. First, Project K’s survey data did not include a measure of participants’ affective engagement in the mentoring partnership, making direct comparison of the role of engagement across all three components impossible. While theoretically distinct, emotional support has been observed to be related to engagement across a range of contexts (e.g., Ruzek et al. 2016) and can therefore be presumed to be positively associated with participants’ interest in and enjoyment of the mentoring component of the program. Next, reverse scored items from the program experience measures had to be removed during factor analysis. The reverse scoring of items, commonly used to guard against positive response bias or inattention by semantically reversing item wording, has sometimes been found to reduce measurement reliability (Barnette 2000). For future research it is recommended to mix response scale direction rather than item wording and therefore ensure scales only use positively worded statements to make it easy for adolescents to respond accurately. Finally, there were systematic differences in ethnicity and baseline social self-efficacy between those who were removed from analyses due to substantive missing data and those who were retained. This precludes generalizing the findings to Pacific youth and those beginning the program with higher levels of social self-efficacy. Future research should endeavor to understand why such students are less likely to complete the program surveys.
Findings discussed here have important implications for practice. Our results indicate that Project K’s community challenge component did not operate as designed, highlighting the importance of critically considering and assessing a program’s theory of change for the purposes of program impovement. It is especially important to monitor the fidelity of program implementation against the component’s design and theory because flaws in implementation may inhibit success. As we discuss in relation to measures of sense of community, new evidence emerges constantly. Youth development practitioners should align program design with best practice, endeavor to keep abreast of new research that can inform their work, and strive to test any program components that are novel or deviate from empirical evidence. Such approaches will ensure youth development programs remain both efficient and as effective as possible for the young people they serve.
Conclusion
Project K participants reported significant gains in social self-efficacy and sense of community over the course of the 14-month program, explained in part by their engagement in an outdoor adventure component and the support they received during a mentoring component. This article demonstrates empirically that youth development program experiences can positively influence adolescent social development, which confers a range of health and wellbeing benefits. Further, results show that program components may have varying (and perhaps unexpected) influences on outcomes. Understanding which components are essential to success is of critical importance for maximization of program impact and efficiency.
Notes
No difference in gender representation was found between participants excluded due to missing data (n = 38) and those retained (n = 327). Comparing participants identifying as Māori, Pacific, or European ethnicity, chi square analyses revealed that European youth were more likely, χ 2 (1) = 17.20, p < .001, while Pacific youth were less likely to be retained, χ 2 (1) = 15.59, p < .001. Due to low response frequencies (<5 for excluded data) it was not possible to compare Asian, other, or combined ethnicities. Finally, independent t-tests with a Bonferroni correction were employed to assess baseline social resources between groups. Those excluded due to missing data reported significantly higher levels of social self-efficacy at baseline (M = 4.15, SD = 0.91) than those retained in the analyses (M = 3.84, SD = 0.89), p = .01. The implications of these systematic group differences are addressed in the discussion.
Sample sizes for gender allowed for a robust assessment of the difference in program effects across groups (144 females to 183 males; both n > 100). Other groupings were not balanced and consisted of smaller sample sizes (for example, only the sample of European ethnicity had n > 100) or contained too small a range to make comparisons meaningful (e.g., age). Therefore, in accordance with conventions of invariance testing (Thompson and Green 2006), other group comparisons were not carried out.
References
Ahrens, K. R., DuBois, D. L., Garrison, M., Spencer, R., Richardson, L. P., & Lozano, P. (2011). Qualitative exploration of relationships with important non-parental adults in the lives of youth in foster care. Children and Youth Services Review, 33, 1012–1023. doi:10.1016/j.childyouth.2011.01.006.
Arnold, M. E., & Cater, M. (2011). From then to now: Emerging directions for youth program evaluation. Journal of Youth Development, 6, 82–94.
Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84, 191–215. doi:10.1037/0033-295X.84.2.191.
Barnette, J. (2000). Effects of stem and likert response option reversals on survey internal consistency: If you feel the need, there is a better alternative to using those negatively worded stems. Educational and Psychological Measurement, 60, 361.
Baumeister, R. F., & Leary, M. R. (1995). The need to belong: Desire for interpersonal attachments as a fundmental human motivation. Psychological Bulletin, 117, 497–529.
Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9, 233–255. doi:10.1207/S15328007SEM0902_5.
Chiessi, M., Cicognani, E., & Sonn, C. (2010). Assessing sense of community on adolescents: Validating the brief scale of sense of community in adolescents (SOC-A). Journal of Community Psychology, 38, 276–292.
Chipuer, H. M., Pretty, G. H., Delorey, E., Miller, M., Powers, T., & Rumstein, O., et al. (1999). The neighbourhood youth inventory: Development and validation. Journal of Community & Applied Social Psychology, 9, 355–368.
Cicognani, E., Mazzoni, D., Albanesi, C., & Zani, B. (2015). Sense of community and empowerment among young people: Understanding pathways from civic participation to social well-being. Voluntas: International Journal of Voluntary and Nonprofit Organizations, 26, 24–45.
Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155–159. doi:10.1037/0033-2909.112.1.155.
Conner, J., & Pope, D. (2013). Not just robo-students: Why full engagement matters and how schools can promote it. A Multidisciplinary Research Publication, 42, 1426–1442. doi:10.1007/s10964-013-9948-y.
Corrigan, A. (2002). Social Competence Scale – Parent Version [Fast Track Project Technical Report]. Retrieved from http://fasttrackproject.org/techrept/s/scp/scp2tech.pdf.
Deane, K. L., & Harré, N. (2014a). Program theory-driven evaluation science in a youth development context. Evaluation and Program Planning, 45, 61–70. doi:10.1016/j.evalprogplan.2014.03.009.
Deane, K. L., & Harré, N. (2014b). The youth adventure programming model. Journal of Research on Adolescence, 24, 293–308. doi:10.1111/jora.12069.
Deane, K.L., Harré, N., Moore, J., & Courtney, M.G.R. (2016). The impact of the project K youth development program on self-efficacy: A randomized controlled trial. Journal of Youth and Adolescence, 1-22. 10.1007/s10964-016-0463-9
DuBois, D. L., Portillo, N., Rhodes, J. E., Silverthorn, N., & Valentine, J. C. (2011). How effective are mentoring programs for youth? A systematic assessment of the evidence. Psychological Science in the Public Interest, 12, 57–91. doi:10.1177/1529100611414806.
Durlak, J., Weissberg, R., & Pachan, M. (2010). A meta-analysis of after-school programs that seek to promote personal and social skills in children and adolescents. American Journal of Community Psychology, 45, 294–309. doi:10.1007/s10464-010-9300-6.
Evans, S. D. (2007). Youth sense of community: Voice and power in community contexts. Journal of Community Psychology, 35, 693–709. doi:10.1002/jcop.20173.
Fan, X., & Sivo, S. (2007). Sensitivity of fit indices to model misspecification and model types. Multivariate Behavioral Research, 42, 509–529. doi:10.1080/00273170701382864.
Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. London: Routledge.
Hattie, J., Marsh, H. W., Neill, J. T., & Richards, G. E. (1997). Adventure education and outward bound: Out-of-class experiences that make a lasting difference. Review of Educational Research, 67, 43–87. doi:10.3102/00346543067001043.
Heinze, H. J., Jozefowicz, D. M. H., & Toro, P. A. (2010). Taking the youth perspective: Assessment of program characteristics that promote positive development in homeless and at-risk youth. Children and Youth Services Review, 32, 1365–1372. doi:10.1016/j.childyouth.2010.06.004.
Hollis, H., Deane, K. L., Moore, J., & Harré, N. (2011). Young maori perceptions of a youth development programme. Kotuitui: New Zealand Journal of Social Sciences Online, 6, 50–61.
Hu, L. T., & Bentler, P. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1–55. doi:10.1080/10705519909540118.
Krauss, S., Collura, J., Zeldin, S., Ortega, A., Abdullah, H., & Sulaiman, A. (2014). Youth–adult partnership: Exploring contributions to empowerment, agency and community connections in Malaysian youth programs. A Multidisciplinary Research Publication, 43, 1550–1562. doi:10.1007/s10964-013-0027-1.
Lakey, B., & Sheldon, C. (2000). Social support theory and measurement. In S. Cohen, B. H. Gottlieb, & L. G. Underwood (Eds.), Social support measurement and intervention: A guide for health and social scientists (pp. 29–52). Cary, NC: Oxford University Press.
Lekes, N., Gingras, I., Philippe, F. L., Koestner, R., & Fang, J. (2010). Parental autonomy-support, intrinsic life goals, and well-being among adolescents in China and North America. Journal of Youth and Adolescence, 39, 858–869.
Li, Y., & Lerner, R. M. (2011). Trajectories of school engagement during adolescence: Implications for grades, depression, delinquency, and substance use. Developmental Psychology, 47, 233–247. doi:10.1037/a0021307.
Low, S., Ryzin, M., Brown, E., Smith, B., & Haggerty, K. (2014). Engagement matters: Lessons from assessing classroom implementation of steps to respect: A bullying prevention program over a one-year period. Prevention Science, 15, 165–176. doi:10.1007/s11121-012-0359-1.
Mahoney, J. L., Parente, M. E., & Lord, H. (2007). After-school program engagement: Links to child competence and program quality and content. Elementary School Journal, 107, 385–404.
Mayberry, M., Espelage, D., & Koenig, B. (2009). Multilevel modeling of direct effects and interactions of peers, parents, school, and community influences on adolescent substance use. Journal of Youth and Adolescence, 38, 1038–1049. doi:10.1007/s10964-009-9425-9.
Maydeu-Olivares, A., & DiStefano, C. (2016). Maximum likelihood estimation of structural equation models for continuous data: Standard errors and goodness of fit. Manuscript in preparation.
McArdle, J. J. (2007). Factor analysis of longitudinal data. In R. Cudeck, & R. MacCallum (Eds.), Factor analysis at 100: Historical developments and future directions (pp. 99–130). Mahwah, NJ: Erlbaum.
McKenzie, M. (2000). How are adventure education program outcomes achieved?: A review of the literature. Australian Journal of Outdoor Education, 5, 19–27.
McMillan, D. W., & Chavis, D. M. (1986). Sense of community: A definition and theory. Journal of Community Psychology, 14, 6–23.
Moore, J. (2005). Self-efficacy and health behaviours: A test of measures to assess the effectiveness of a positive youth development program. (Unpublished Master’s Thesis), University of Auckland, Auckland.
Muthén, B. O. (2014). Means for latent variables [Online forum comment, September 4]. Retrieved from http://www.statmodel.com/discussion/messages/9/5825.html?1459292652.
Muthén, B. O., & Muthén, L. K. (2016). Mplus 7 Base Program. Los Angeles, CA: Muthén & Muthén.
Nilsen, W., Karevold, E., Roysamb, E., Gustavson, K., & Mathiesen, K. S. (2013). Social skills and depressive symptoms across adolescence: Social support as a mediator in girls versus boys. Journal of Adolescence, 36, 11–20. doi:10.1016/j.adolescence.2012.08.005.
Peterson, N. A., Speer, P. W., & McMillan, D. W. (2008). Validation of a brief sense of community scale: Confirmation of the principal theory of sense of community. Journal of Community Psychology, 36, 61–73. doi:10.1002/jcop.20217.
Pressman, S. D., Cohen, S., Miller, G. E., Barkin, A., Rabin, B. S., & Treanor, J. J. (2005). Loneliness, social network size, and immune response to influenza vaccination in college freshmen. Health Psychology, 24, 297–306.
Ramey, H. L., Busseri, M. A., Khanna, N., & Rose-Krasnor, L. (2010). Youth engagement and suicide risk: Testing a mediated model in a Canadian community sample. Journal of Youth and Adolescence, 39, 243–258. doi:10.1007/s10964-009-9476-y.
Rhodes, J. E., & Lowe, S. R. (2009). Mentoring in adolesence. In R. M. Lerner, & L. Steinberg (Eds.), Handbook of adolescent psychology. Hoboken, NJ: Wiley. Vol. 2.
Roth, J. L., & Brooks-Gunn, J. (2003). What exactly is a youth development program? Answers from research and practice. Applied Developmental Science, 7, 94–111.
Roth, J. L., & Brooks-Gunn, J. (2016). Evaluating youth development programs: Progress and promise. Applied Developmental Science, 20, 188–202. doi:10.1080/10888691.2015.1113879.
Ruzek, E. A., Hafen, C. A., Allen, J. P., Gregory, A., Mikami, A. Y., & Pianta, R. C. (2016). How teacher emotional support motivates students: The mediating roles of perceived peer relatedness, autonomy support, and competence. Learning and Instruction, 42, 95–103. doi:10.1016/j.learninstruc.2016.01.004.
Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55, 68–78.
Schwartz, S., Rhodes, J. E., Spencer, R., & Grossman, J. (2013). Youth initiated mentoring: Investigating a new approach to working with vulnerable adolescents. American Journal of Community Psychology, 52, 155–169. doi:10.1007/s10464-013-9585-3.
SDT. (2016). Intrinsic motivation inventory (IMI). Retrieved from http://selfdeterminationtheory.org/intrinsic-motivation-inventory/.
Shernoff, D. J. (2010). Engagement in after-school programs as a predictor of social competence and academic performance. American Journal of Community Psychology, 45, 325–337. doi:10.1007/s10464-010-9314-0.
Thompson, M. S., & Green, S. B. (2006). Evaluating between-group differences in latent variable means. In G. R. Hancock, & R. O. Mueller (Eds.), A second course in structural equation modeling (pp. 119–169). Greenwich, CT: Information Age Publishing.
Thurber, C. A., Scanlin, M. M., Scheuler, L., & Henderson, K. A. (2007). Youth development outcomes of the camp experience: Evidence for multidimensional growth. Journal of Youth and Adolescence, 36, 241–254. doi:10.1007/s10964-006-9142-6.
Urban, J. B. (2008). Components and characteristics of youth development programs: The voices of youth-serving policymakers, practitioners, researchers, and adolescents. Applied Developmental Science, 12, 128–139. doi:10.1080/10888690802199400.
Voorhees, C. M., Brady, M. K., Calantone, R., & Ramirez, E. (2016). Discriminant validity testing in marketing: An analysis, causes for concern, and proposed remedies. Journal of the Academy of Marketing Science, 44, 119–134. doi:10.1007/s11747-015-0455-4.
Wang, M.-T., & Eccles, J. S. (2012). Social support matters: Longitudinal effects of social support on three dimensions of school engagement from middle to high school. Child Development, 83, 877–895. doi:10.1111/j.1467-8624.2012.01745.x.
Wang, C. K. J., Liu, W.-C., & Kahlid, A. (2006). Effects of a five-day outward bound course on female students in Singapore. Australian Journal of Outdoor Education, 10, 20–28.
Warren, J.K. (2005). Adolescent well-being: Effects of time and intervention. (Unpublished doctoral dissertation), University of Auckland, Auckland.
Weiss, C. H. (2000). Which links in which theories shall we evaluate?. New Directions for Evaluation, 87, 35–45.
Whittington, A., & Mack, E. (2010). Inspiring courage in girls: An evaluation of practices and outcomes. The Journal of Experiential Education, 33, 166–180.
Wills, T. A., & Shinar, O. (2000). Measuring perceived and received social support. In S. Cohen, B. H. Gottlieb, & L. G. Underwood (Eds.). Social support measurement and intervention : A guide for health and social scientists (pp. 86–135). Cary, NC: Oxford University Press.
Wills, T. A., Vaccaro, D., & McNamara, G. (1992). The role of life events, family support, and competence in adolescent substance use: A test of vulnerability and protective factors. American Journal of Community Psychology, 20, 349–374. doi:10.1007/BF00937914.
Zeldin, S., Krauss, S., Collura, J., Lucchesi, M., & Sulaiman, A. (2014). Conceptualizing and measuring youth–adult partnership in community programs: A cross national study. American Journal of Community Psychology, 54, 337–347. doi:10.1007/s10464-014-9676-9.
Acknowledgements
The authors gratefully acknowledge the Graeme Dingle Foundation and their Community Partners for their part in implementing the evaluation and supporting the research. We also thank two anonymous reviewers for their helpful comments on an earlier draft of this article. The authors received no financial support for the research, authorship, and/or publication of this article.
Authors’ Contributions
K. D. designed the original study with support from J. M. and under the supervision of N. H., and all authors contributed to aspects of the research design for this article. J. M. coordinated and managed the data collection. M. C. led the statistical analyses and drafted the methods and results with K. D., while C. C., K. D. and N. H. all contributed to the interpretation of results. C. C. took primary responsibility for writing the full article with support from K. D. and N. H. All authors reviewed and approved the final version.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of Interest
J. M. is the Research and Evaluation Manager for the Graeme Dingle Foundation, the organization that owns the Project K program. She was involved in the study design and data collection; however, she had no involvement in the data analysis or reporting of the findings. C. C., K. D., N. H. and M. C. declare that they have no competing interests.
Ethical Approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional research committee and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed Consent
Informed consent was obtained from all individual participants included in the study.
Electronic supplementary material
Rights and permissions
About this article
Cite this article
Chapman, C.M., Deane, K.L., Harré, N. et al. Engagement and Mentor Support as Drivers of Social Development in the Project K Youth Development Program. J Youth Adolescence 46, 644–655 (2017). https://doi.org/10.1007/s10964-017-0640-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10964-017-0640-5