Introduction

Humans are inherently social creatures and a sense of belonging appears to be a prerequisite for a healthy and happy life (Baumeister and Leary 1995). Adolescence is an important time for the development of social ties outside the family and cultivating social skills and connection can protect young people from a range of aversive outcomes (e.g., Nilsen et al. 2013). Youth development programs can be important extra-familial social contexts but, with studies that compare the influence of various program components still very rare (Roth and Brooks-Gunn 2016), little is known about how they support participants’ social development. Project K is a New Zealand-based, multi-component youth development program targeted at adolescents with low self-efficacy. This study builds upon previous evidence that Project K contributes to positive social outcomes for youth (Deane et al. 2016) by investigating the relative contributions made by different program components.

The Protective Qualities of the Social World

A meaningful sense of connection with others is argued to be a fundamental human need (Baumeister and Leary 1995; Ryan and Deci 2000). Meaningful connection can be accessed through interpersonal relationships or from a sense of belonging in groups, contexts, and wider communities (Baumeister and Leary 1995; McMillan and Chavis 1986). A sense of meaningful connection with others appears to confer a range of valuable protective qualities during adolescence and is associated with important life outcomes, such as reduced rates of depression and substance abuse (e.g., Mayberry et al. 2009; Nilsen et al. 2013).

Given the protective value of social bonds, cultivating social resources—whether through social skill development, improved social confidence, or directly facilitating connections—is an essential component of youth thriving. A range of published program evaluations indicate that young people can develop essential social resources, including social skills, peer relationships, and group-level belonging, through educational and out-of-school programs (e.g., Deane et al. 2016; Durlak et al. 2010; Thurber et al. 2007). However, less research has focused on how programs achieve such outcomes. Below we discuss evidence that both participant engagement and experiences of support are key drivers of youth program success, before highlighting gaps in current research that this article aims to address.

Promoting Positive Social Development through Youth Programs

Two mechanisms through which youth programs may promote social development are participant engagement and experiences of support. Engagement is conceptualized as consisting of three key components: affective (i.e., interest, enjoyment), behavioral (i.e., effort), and cognitive (i.e., seeing value in the activity; see Conner and Pope 2013; Wang and Eccles 2012). Beyond mere participation, adolescents who are more engaged in program or educational activities typically show greater benefits—both in general (Conner and Pope 2013; Lekes et al. 2010; Low et al. 2014) and specifically in relation to social outcomes (Krauss et al. 2014; Li and Lerner 2011; Mahoney et al. 2007; Ramey et al. 2010; Shernoff 2010). Thus, engagement appears to be a key mechanism through which youth programs achieve positive outcomes for participants.

Support received from adults in program contexts has also been shown to be associated with a range of positive outcomes (e.g., Zeldin et al. 2014). Although social support can take many forms (see Wills and Shinar 2000 for overview), emotional support has been shown to be particularly relevant for adolescents’ engagement and sense of connection (Ruzek et al. 2016). Emotional support includes access to people who express genuine concern and are willing to listen without judgement (Lakey and Sheldon 2000). Feeling supported by contextually relevant adults (such as teachers, mentors, or other program staff) has been especially linked to positive social outcomes (DuBois et al. 2011; Krauss et al. 2014; Ruzek et al. 2016; Wang and Eccles 2012). For instance, mentors typically provide emotional support (Schwartz et al. 2013) and may help facilitate socio-relational development by modeling positive relationships and helping youth to understand and effectively express emotion (Ahrens et al. 2011).

Unfortunately, as highlighted by Roth and Brooks-Gunn (2016), much of what is known about the processes through which programs achieve positive outcomes, including engagement and supportive adult-youth relationships, draws from research on a broad range of organized youth activities, many of which would not be considered youth development programs per se. Futhermore, research that systematically tests the influence of different program components is rarely seen. Understanding exactly how programs achieve results is becoming increasingly important in a climate where resources are stretched and youth development organizations are required to make efficiencies while evidencing positive impact (Arnold and Cater 2011; Roth and Brooks-Gunn 2016). It is difficult to improve programs if the critical aspects remain unknown—a problem of particular relevance for multi-component programs like Project K. The aim of this article, therefore, is to contribute to the advancement of youth development theory and practice by assessing how engagement in, and support experienced, during the different components of one youth development program, Project K, help to promote positive social development among its participants.

Project K: A Youth Development Program for Adolescents with Low Self-Efficacy

Project K is a multi-component youth development program consisting of an outdoor adventure experience, a community service project, and an adult-youth mentoring partnership. Participants are 13–15 year old high school students with low academic and social self-efficacy relative to their peers (see Deane et al. 2016 for a detailed overview of the participant selection process). The program is owned by the Graeme Dingle Foundation, which provides governance, training, materials, and research and evaluation to Community Partners that are licensed to deliver the program in several regions across New Zealand.

The Community Partners are encouraged to adapt the program activities to suit their community context and needs. They are all, however, required to deliver the core features of the program in a consistent manner. Standardized training and implementation manuals and ongoing monitoring are provided by Graeme Dingle Foundation’s National Support Office to ensure fidelity across the program sites. Community Partners are required to deliver the three core program components in the same sequence over a 14 month period. First, participants take part in a 3-week wilderness adventure. At an isolated bush camp they engage in challenge-based and team-building activities with the objective of developing key skills, such as goal-setting, communication, problem-solving, and leadership. Students then put new skills into practice during a 7–10 day wilderness trek, where they take turns as group leader. Next, participants have the opportunity to reinforce and integrate their new skills in their own communities during a community challenge. The young people engage in workshops on health and wellbeing (e.g., relationships, alcohol, nutrition) and are encouraged to explore community resources, meet local leaders, and develop a collaborative project to address an identified community need. Facilitators of both the wilderness adventure and community challenge are trained to have high expectations of all participants and offer efficacy-enhancing and relatedness support where possible. Finally, participants are matched with a trained adult mentor for a 12-month period. Mentors are adult volunteers from the participants’ own communities who have been selected following a screening process that seeks mentors with qualities such as empathy, good listening skills, and the ability to motivate and inspire. After a social event that provides the opportunity for all mentors and mentees to interact, matches are based on both mentee and mentor preferences, with the former taking priority. Mentors take part in a 20 hour training program and receive ongoing support to develop non-judgemental relationships with the young people. They are expected to meet with mentees fortnightly to help them to set and achieve four goals (one of which must be academic and another related to health) over the course of the year.

The Current Study

As previously demonstrated in a randomized controlled trial, Project K participants significantly improve in social self-efficacy (i.e., believing one can succeed, here in the social domain; Bandura 1977) over the course of the program relative to a control group, an effect corroborated by parental observations and maintained 1 year after completing the program (Deane et al. 2016). Nevertheless, the mechanisms proposed to drive such social gains have not yet been empirically tested. Project K’s theory of change (see Deane and Harré 2014a for more detail) suggests that the program promotes growth through an iterative experiential learning cycle that is reinforced with ongoing support across different contexts. The intense wilderness adventure sets this process in motion by providing opportunities to develop mastery in a supportive but unfamiliar social setting away from everyday distractions. It encourages bonding with peers and development of social resources and self-efficacy. The community challenge enables transfer of the wilderness adventure learning to a familiar context and connects students to people and resources in their community. It has been specifically designed to increase the adolescents’ sense of belonging and efficacy within their communities. The adult mentor then dedicates individualized attention to supporting the goals set by the participant during the wilderness adventure. Theoretically, each component should build on the previous one to consolidate prior learning and sustain outcomes. As a whole, the program alternates challenging experiences with self-reflection aimed at consolidating and transferring learning. Participant engagement is placed at the very centre of the experiential learning process proposed to drive Project K program outcomes, while mentor support is theorized to be another key factor in determining program success.

This study empirically tested the contribution made by each of the three Project K program components to positive social outcomes. Given evidence discussed above that engagement and support are two key drivers of program outcomes, we tested how participants’ affective engagement in the wilderness adventure and community challenge components plus experiences of support during the mentoring partnership, both individually and in unison, predicted improvements in social resources over the course of the program. Specifically, it was hypothesized that Project K participants would experience significant social gains over the course of the program (H1), measured by self-reported social self-efficacy (H1A) and sense of community (H1B), and corroborated by parents’ perceptions of social competence (H1C). Given the additive model outlined in Project K’s theory of change (Deane and Harré 2014a), participants’ emotional engagement in the wilderness adventure (H2) and community challenge (H3) as well as their experiences of emotional support from mentors (H4) were all expected to independently explain social gains, after accounting for baseline levels.

Method

Procedure and Participants

All questionnaires were embedded within the Graeme Dingle Foundation’s internal evaluation plan for programs delivered in 2009 and 2010. After securing ethical approval from the University of Auckland’s ethics committee, de-identified data were shared with the researchers for scholarly purposes. Three-hundred and sixty-five Project K youth participants took part in this research over the course of 2009 and 2010. Prior to analysis, 38 cases were removed because they were missing data on more than 40% of the combined survey itemsFootnote 1. The final sample (N = 327) included 183 males and 144 females. Although participant age was not recorded, Project K is implemented in Year 10 of the New Zealand school system; thus the majority of participants were aged between 13 and 15 years at the start of the program. Almost half of participants (153 participants) were of European ethnicity, 49 were Māori (indigenous New Zealanders), 53 of Pacific Island ethnicity, 9 Asian, and 5 identified with Other ethnicities. The remaining 58 identified with more than one ethnicity (including 31 Māori/European, 15 Māori/Pacific, and 5 Pacific/European). In total, 168 program participants completed the surveys in 2009 and a further 159 in 2010, as part of 31 different program deliveries (independent clusters) from 23 high schools across New Zealand. Each Project K delivery aims to serve 12 young people over the 14-month program, of whom on average just under 11 participated in the research (M = 10.55, SD = 1.93).

Pre- and post-program assessment

The pre-program questionnaire was distributed to all participants before the start of the program (Time 0) either by Project K regional directors or trained representatives and assessed baseline levels of relevant social measures. The same questionnaire was also administered at program completion (Time 4). One parent or caregiver for each participant consented to participate in the research and completed a questionnaire assessing their perceptions of their child’s social competence at Times 0 and 4.

In-program assessment

Questionnaires assessing participants’ experiences were administered by trained program staff within 1 week of completing the wilderness adventure (Time 1) and community challenge (Time 2). Finally, questionnaires assessing perceptions of the 12-month mentoring component were administered up to 2 weeks before program graduation (Time 3).

Measures

Measures analyzed in this study (described in detail below) represent a subset of the entire research program; full questionnaires are available upon request. Reverse scored items across the survey package caused problems for factor reliability and have been removed from analyses, a point returned to in the discussion.

Social self-efficacy

Participants self-reported their beliefs about their social abilities using 8 items from the Social Self-efficacy subscale of Moore’s (2005) self-efficacy questionnaire (e.g., “How well can you have a chat with an unfamiliar person of your age?’; 1 = Not at all, 6 = Very well). Higher scores indicate higher social self-efficacy; α = .84 and .86 at times 0 and 4 respectively.

Sense of community

Seven items were adapted from the brief sense of community scale (Peterson et al. 2008) to capture participants’ degree of group-level belonging in their communities (e.g., “I feel connected to this community”; 1 = Strongly disagree, 5 = Strongly agree). Higher scores indicate a stronger sense of community; α = .86 and .90 at times 0 and 4 respectively.

Parental perceptions of social competence

Parents or caregivers responded to six items from the Prosocial/Communication Skills subscale of Corrigan’s (2002) Social Competence Scale (Parent Version) in order to assess their perspectives of their children’s social skills (e.g., “Your child is very good at understanding other people’s feelings”; 1 = Not at all, 5 = Very well). Higher scores indicate that caregivers perceived participants’ to be more socially competent; α = .80 and .81 at times 0 and 4 respectively.

Engagement

Three items from the interest/enjoyment subscale of the Intrinsic Motivation Inventory (SDT 2016) measured participants’ affective engagement in the wilderness adventure and community challenge (e.g., “I enjoyed the Wilderness Adventure/Community Challenge very much”; 1 = Not at all true, 7 = Very true). Higher scores indicate higher affective engagement; α = .83 and .87 for the wilderness and community components respectively.

Support

Perceptions of the emotional support received from mentors was measured with six items from the Emotional Support subscale of the Family Support Scale (Wills et al. 1992). The scale was modified such that “parent” was replaced with “mentor” (e.g., I could share my feelings with my mentor”, 1 = Strongly disagree, 7 = Strongly agree). Higher scores indicate participants’ reported receiving higher levels of emotional support from their mentors; α = .92.

Analysis

Missing values

Missing Value Analysis was carried out using IBM SPSS 23. Little’s MCAR test suggested that data were missing at random, χ 2 = 24,321.12, df = 24,118, p = .177, therefore imputations were carried out using the Expectation-Maximization algorithm. Where imputed values fell outside the original scale boundaries, they were modified to fit within original scale boundaries.

Model specification

Means and Variance Adjusted Maximum Likelihood methods were chosen to estimate model fit in factor analytical work due to moderate skewness in the data (see Maydeu-Olivares and DiStefano 2016). After using exploratory (oblimin rotation) and confirmatory factor analytic procedures in MPlus 7.35, two social outcomes measurement models (Times 0 and 4) and program experience factors were specified and assessed. Both social outcome measurement models (each for Times 0 and 4) and the measurement model for the three program experience factors (wilderness adventure engagement, community challenge engagement, and mentor support, measured at times 1, 2, and 3 respectively) all met established requirements (Fan and Sivo 2007; Hu and Bentler 1999; Voorhees et al. 2016) of model fit and both convergent and discriminant validity. This suggested that the outcomes at each time point represented distinct social constructs, and the program experiences also represented distinct domains.

A longitudinal invariance test was also carried out to test the stability of both versions of the three-factor social outcome measurement model over time. The longitudinal model met the requirements (Cheung and Rensvold 2002; McArdle 2007) for configural, metric, and scalar invariance, suggesting that the three distinct domains of social engagement were stable over time. Therefore, latent factor means could defensibly be compared across time points. Tests of model fit are summarized in Table 1 and in Online Resources 1 and 2.

Table 1 Model fit for measurement and structural models
Full size table

Results

Promoting Social Development

Changes on social measures were assessed using z-tests, which are equivalent to paired sample t-tests but specifically used to test changes in latent factors. Overall, participants showed large, statistically significant improvements in social self-efficacy from baseline (M = 4.11, SD = 0.93) to post-program (M = 4.68, SD = 0.64), p < .001, Cohen’s d = 0.76. Participants’ also reported a small but significant increase in their sense of community from baseline (M = 3.35, SD = 0.78) to post-program (M = 3.74, SD = 0.79), p < .001, d = 0.34. However, the same gains were not perceived by caregivers, with no significant difference observed between parental perceptions of self-efficacy from baseline (M = 4.12, SD = 0.73) to post-program (M = 4.11, SD = 0.85), p = .87 (effect sizes interpreted in accordance with Hattie 2009 ).

Assessing the Impact of Program Components

Modeling project K

For the purpose of establishing a structural model that demonstrated the effects of engagement in the wilderness adventure and community challenge as well as mentor support on the social outcomes of interest, several auto-regressive structural models were specified (see Table 1). The first structural model included the three Time 0 factors and three Time 4 factors with direct effects only. A second model included all cross-lag effects. Finally, a third structural model included all three midway factors and cross-lag effects. Assessment of model fit indices for these models suggested a general improvement in fit as more paths and factors were included. The robustness of this final model was assessed by exploring a series of alternative nested models in which different non-statistically significant cross-lagged paths were removed (i.e., those between T0 and midway; midway and T4; and T0 and T4 factors). The related series of chi-square difference tests showed no statistically significant shifts in model fit for any of these alternatives. The final model containing all three midway and cross-lag effects was therefore retained as it represented the data well and also accounted for the negligible associated cross-lag effects.

Accounting for program clusters

Although intra-class correlations (presented with design effects in Table 2) suggested that the portion of individual differences in pre-program social resources attributable to clustering of participants was small, cluster effects were identified in relation to engagement in the wilderness adventure and community challenge components. Multilevel analysis was therefore employed (using MPlus 7.35) to account for the effect of clustering within the 31 different program deliveries. Accounting for clustering, all item-factor loadings remained statistically significant (p < .001), and there were no substantive changes in the structural regression and correlational coefficients between factors. Model fit was also acceptable. In addition, the requirements for configural, metric, and scalar invariance for the final model were met across gender and there were no meaningful differences in structural regressions by gender groupFootnote 2 (see Online Resources 3–5).

Table 2 Intra-class correlation coefficients and design effects
Full size table

Program components

Overall, Project K participants reported relatively high levels of affective engagement in both the wilderness adventure (M = 5.73, SD = 1.20) and community challenge (M = 5.26, SD = 1.41) and perceived high levels of emotional support in mentor partnerships (M = 5.72, SD = 1.09). Pre-program parental perceptions of social competence were positively associated with participants’ experiences of mentor support (ß = .18, p < .05), but no other baseline measure had any statistically significant association with experiences within the program. The final model, accounting for program delivery clusters using multilevel modeling, is summarized in Fig. 1.

Fig. 1
figure 1

Auto-regressive model regressing post-program social resources on experiences of engagement and support in the project K program. Note. N = 327; T0 and T4 factors modeled as parallel factors over time; R 2 values for endogenous factors provided in italics in larger font to right of circles; non-statistically significant inter-factor paths not pictured in model; estimated means and standard deviations carried out in accordance with Muthén (2014). Observed Ms and SDs provided for each scale inside circles. * p < .05, ** p < .01 *** p < .001

Full size image

Accounting for baseline levels and clustering effects, participants’ engagement in the wilderness adventure (ß = .21, p < .01) and their experiences of receiving emotional support from mentors (ß = .13, p < .05) both significantly contributed to their post-program perceptions of social self-efficacy, R 2 = .29, p < .001, f 2 = .41 (large effect size; Cohen 1992). The same pattern of results was observed for post-program sense of community, R 2 = .30, p < .001, f 2 = .43 (large), which, after accounting for baseline levels, was significantly predicted by engagement in the wilderness adventure (ß = .12, p < .01) and mentor support (ß = .25, p < .001). Although engagement in the community challenge was significantly associated with wilderness adventure engagement (Φ = .44, p < .001), experiences in the community challenge did not uniquely predict gains on any social measure. Finally, the model suggested that neither engagement in the wilderness adventure and the community challenge nor mentor support affected parental perceptions of social competence, which did not significantly change over the course of the program.

Discussion

This article examines if and how experiences of engagement or support during each of the three core components of Project K—a wilderness adventure, community challenge, and mentoring partnership—predicted positive social outcomes for the youth participants. By doing so, the results contribute to understanding how youth development programs enhance participants’ social development, a key task of adolescence. First, and consistent with an earlier randomized controlled evaluation, it was found that Project K participants reported significantly improved social skills and connectedness over a 1-year period (supporting H1), controlling for baseline social resource levels and accounting for clustering effects. As predicted, participants reported significant gains in both social self-efficacy (H1A) and sense of community (H1B). This study, however, finds no evidence that such gains were also perceived by caregivers (contrary to H1C). In regard to the core theoretical question of this article, it was found that participants’ affective engagement in the wilderness adventure (H2) and the levels of emotional support they received from mentors (H4) both significantly and uniquely contributed to their observed social gains, with engagement in the wilderness adventure being the stronger predictor of social self-efficacy gains and mentor support being the stronger predictor of improved sense of community. No evidence, however, was found that engagement in the community challenge contributed to participants’ social development (contradicting H3). This result stresses the importance of investigating the unique contributions made by program components in order to maximize impact and inform program improvements.

In relation to the current study, Project K’s theory of change (Deane and Harré 2014a) suggested that all three core components of the program should contribute to positive social outcomes for participants. Results, however, indicate that only the wilderness adventure and mentoring partnership did so. Outdoor adventure programs such as the wilderness adventure have been shown to promote social connection for participants, in part because they create intense experiences through geographic isolation, shared challenge, and interdependence (see Deane and Harré 2014b; McKenzie 2000). Further, interviews previously conducted with Project K participants have revealed that they experienced the wilderness adventure as socially demanding, requiring both leadership and teamwork from them (Hollis et al. 2011). It was therefore anticipated that engagement in the wilderness adventure would be significantly associated with social gains, and this was confirmed by the data. This result corroborates previous studies showing that outdoor adventure programs can promote positive social development in young people (Hattie et al. 1997; Wang et al. 2006; Whittington and Mack 2010). It also demonstrates that engagement, a mechanism shown to promote positive social outcomes for youth more generally (e.g., Li and Lerner 2011; Ramey et al. 2010; Shernoff 2010), is critical in driving the impact of outdoor adventure activities nested within multi-component programs.

Mentoring partnerships provide ongoing, one-on-one relationships that young people may use as templates to cultivate more positive relationships generally (DuBois et al. 2011; Ruzek et al. 2016; Schwartz et al. 2013). Mentors can also broaden mentee horizons by providing access to social capital, exposing youth to new experiences, people and resources in their communities (Rhodes and Lowe 2009). Supportive non-parental adults working with youth, like mentors, should pay attention to the unique characteristics of their mentee, including their cultural and personal background, and thus help to cultivate a sense of belonging (Heinze et al. 2010; Roth and Brooks-Gunn 2003; Urban 2008). Supportive mentoring was expected to contribute to participants’ ability to navigate the social world and to enhance their sense of connection with the wider community, as results here demonstrate. We build upon existing evidence that experiences of support influence program outcomes (e.g., Krauss et al. 2014; Wang and Eccles 2012) and demonstrate how support experienced in just one component of a program can influence overall social outcomes.

It was particularly surprising to find that the community challenge did not also contribute to participants’ social development, given that it was specifically designed to promote a positive connection with wider communities, which was expected to translate into an improved sense of community. One possible explanation for this finding is that, because sense of community is measured in relation to an unspecified “community”, participants may be primed by the context in which the questionnaire was given to consider their program cohort as their most salient community. If so, the intensity of the wilderness adventure could be the solidifying factor that promotes a connection with the cohort. This supposition is strengthened by the moderate positive association between wilderness adventure and community challenge engagement and the observed clustering effects for those experiences, indicating cohort effects affect program experiences. Another explanation is that the Sense of Community measure used was not suitable for youth participants. There have been suggestions that adult measures are not necessarily appropriate for young people (Chipuer et al. 1999; Evans 2007). For example, young people in early or mid-adolescence may be more concerned with safety and less concerned with influence—at least in the sense of changing social structures—than adults (Chipuer et al. 1999). The measure used here included influence and did not include safety, which may have weakened its validity. Furthermore, the community challenge itself was designed to involve students in a project to address a community need. This one-off contribution is unlikely to have shifted their general perceptions of having influence within their communities, which is what was measured. Since this study was designed, a specific youth sense of community scale has been developed and used successfully (Chiessi et al. 2010; Cicognani et al. 2015). Future research would benefit from employing a more youth-oriented measure that both captures the unique concerns and sensitivities of adolescence and is more closely aligned with what the program component was designed to achieve.

Although it is always difficult to separate conceptual from implementation-related causes, earlier qualitative research on Project K suggests that both conceptual and implementation flaws can help explain why the community challenge did not appear effective. Project K participants interviewed by Warren (2005) and program staff involved in Project K’s theory of change development (Deane and Harré 2014a) reported that it was often difficult to engage participants in the community component and to ensure the project was student-driven. A potential mismatch between adolescent community priorities (e.g., safety) and those the community challenge is designed to promote (e.g., influence) could help explain this lack of engagement. Furthermore, several Community Partners have advised the Graeme Dingle Foundation that participating schools have increasing concerns about the time taken away from studies for students to complete the wilderness and community components, which has resulted in some Community Partners deviating from the initial design. We acknowledge that the community challenge component could be improved in line with other evidence-based community service youth programs and may benefit from a special focus on addressing youth concerns over community safety. However, the absence of strong supporting evidence in favor of the community challenge at present, coupled with the pragmatic resource and logistical concerns of Community Partners, has lead to a recent decision by the Graeme Dingle Foundation to redesign and reduce the length of this program component. Youth development programs must necessarily make choices about the use of resources and these will always be made under conditions of uncertainty, using the best evidence available.

Lastly, although an earlier randomized control trial (see Deane et al. 2016) suggested that social gains reported by Project K participants relative to a control group were also perceived by parents, the current study did not corroborate this finding. Research suggests that the protective qualities conferred by social resources depend more on self-perception than objective indicators of belonging (e.g., Pressman et al. 2005). Therefore, although parental corroboration is a valuable objective measure of social functioning, the evidence presented here that participants perceive improvements in their social skills and connectedness is perhaps a more meaningful measure of program success in the context of youth thriving.

Extrapolating beyond the specific context of Project K, this research adds further evidence that youth programs can effectively nurture the development of core social resources in young people (corroborating Krauss et al. 2014; Li and Lerner 2011; Mahoney et al. 2007; Ramey et al. 2010; Shernoff 2010). In particular, answering Roth and Brooks-Gunn’s call (2016) and building upon earlier research (e.g., Low et al. 2014; Zeldin et al. 2014), our longitudinal data provide strong evidence that participants’ experiences of engagement and support in the specific context of a youth development program can drive positive social development. At a broader level, our findings highlight the importance and value of program research and evaluations moving beyond so called “black box” methodologies (assessing pre-to-post changes) and seeking to understand how programs achieve observed results. This is particularly important in the current resource-constrained climate where evidence-based decisions to increase efficiencies are encouraged (Arnold and Cater 2011; Roth and Brooks-Gunn 2016).

We have tested the relative influence that experiences within each of the three core components of Project K exert on social outcomes. This is only the first step toward a richer understanding of how, why, for whom, and under which conditions youth development programs work. It is not generally possible to test all of the theoretical links proposed in a program theory within a single study (see Weiss 2000). The focus here has therefore been on engagement and support, two mechanisms that have been shown to have a positive influence on youth program outcomes elsewhere (e.g., Conner and Pope 2013; Krauss et al. 2014; Low et al. 2014; Zeldin et al. 2014). Other mechanims of change remain to be tested, as do the experiences and program aspects that themselves create engagement and feelings of support. Our data show that program engagement varies systematically across deliveries. Deane and colleagues’ (2016) randomized controlled trial study previously demonstrated that program outcomes vary according to gender, ethnicity, and the affluence of the school community. Identifying moderators is therefore likely to be a fruitful avenue for future research. Through empirically testing the links proposed in program theories, as has been started with Project K in this instance, a sound evidence base can be built for future youth development initiatives to draw upon.

It is a strength of the current research that measures were collected longitudinally from before, during, and after the program. Cross-lagged modeling to assess changes in social resources over time allowed us to investigate the additive and potentially causal contributions that Project K components made to observed changes. Furthermore, multilevel modeling was employed to account for the fact that participants’ experiences within the program were at least partially influenced by the climate of the particular cohort. A few limitations also warrant discussion. First, Project K’s survey data did not include a measure of participants’ affective engagement in the mentoring partnership, making direct comparison of the role of engagement across all three components impossible. While theoretically distinct, emotional support has been observed to be related to engagement across a range of contexts (e.g., Ruzek et al. 2016) and can therefore be presumed to be positively associated with participants’ interest in and enjoyment of the mentoring component of the program. Next, reverse scored items from the program experience measures had to be removed during factor analysis. The reverse scoring of items, commonly used to guard against positive response bias or inattention by semantically reversing item wording, has sometimes been found to reduce measurement reliability (Barnette 2000). For future research it is recommended to mix response scale direction rather than item wording and therefore ensure scales only use positively worded statements to make it easy for adolescents to respond accurately. Finally, there were systematic differences in ethnicity and baseline social self-efficacy between those who were removed from analyses due to substantive missing data and those who were retained. This precludes generalizing the findings to Pacific youth and those beginning the program with higher levels of social self-efficacy. Future research should endeavor to understand why such students are less likely to complete the program surveys.

Findings discussed here have important implications for practice. Our results indicate that Project K’s community challenge component did not operate as designed, highlighting the importance of critically considering and assessing a program’s theory of change for the purposes of program impovement. It is especially important to monitor the fidelity of program implementation against the component’s design and theory because flaws in implementation may inhibit success. As we discuss in relation to measures of sense of community, new evidence emerges constantly. Youth development practitioners should align program design with best practice, endeavor to keep abreast of new research that can inform their work, and strive to test any program components that are novel or deviate from empirical evidence. Such approaches will ensure youth development programs remain both efficient and as effective as possible for the young people they serve.

Conclusion

Project K participants reported significant gains in social self-efficacy and sense of community over the course of the 14-month program, explained in part by their engagement in an outdoor adventure component and the support they received during a mentoring component. This article demonstrates empirically that youth development program experiences can positively influence adolescent social development, which confers a range of health and wellbeing benefits. Further, results show that program components may have varying (and perhaps unexpected) influences on outcomes. Understanding which components are essential to success is of critical importance for maximization of program impact and efficiency.