Introduction

Community involvement in all stages of program development and implementation is now a standard of public health practice. Programs that respect and appeal to local values, consider organizational capacity, and draw from indigenous knowledge are more likely to be successfully implemented, supported and sustained (Feinberg et al. 2008; Hausman 2002; Minkler 2000; Wallerstein and Duran 2006). Community-Based Participatory Research (CBPR) is increasingly recognized as a research paradigm that supports the collaboration of community and academic program developers. CBPR principles include equal partnerships among stakeholders and community empowerment through shared information and co-learning (Israel et al. 2008; Wallerstein and Duran 2006).

CBPR is not a set of research methods; instead, CBPR is an approach to research (Wallerstein and Duran 2008). Indeed, CBPR is particularly well-suited to address the exigencies of specific community settings that often challenge standard research practice, particularly in the context of evaluation studies. As a result, interest in participatory methods has erupted, spurred on by both scholars and funders focused on CBPR as one way to bridge the gap between research and practice (Glasgow et al. 2012). Significant progress has been made on methods that facilitate CBPR, such as how to build and sustain community collaborations (e.g., Israel et al. 2008) and how to include relevant stakeholders, particularly youth, in research (Jacquez et al. 2013). CBPR also offers innovative avenues for dissemination of research findings, especially related to programming for families and youth (Vaughn et al. 2013). Important new work has gone one step further, showing how rigorous experimental design can be imbedded into the context of participatory research to develop and test interventions (Nastasi et al. 2000; Ozer and Douglas 2013).

Though considerable progress has been made advancing the use of participatory methods in community-based evaluation research, CBPR has not yet been consistently applied to the conceptualization and measurement of intervention outcomes. Even in the context of program effectiveness research using a CBPR approach, for example, the outcome measures available usually reflect the “external” (etic) academic researchers’ view of program effect and the policy-makers’ needs for decision-making (Hausman 2002; Jackson et al. 1998; Miller and Shinn 2005). Specifically, academic researchers generally ask community stakeholders to use academically-derived, psychometrically sound outcome measures rather than collecting information about “internal” (emic) or community-defined expectations for prevention programs, usually because quantitative measures for such outcomes do not exist.

Using available measures helps to build an evidence base that communicates to academic and policy partners, but the resulting information may be less relevant to community stakeholders. Programs that cannot successfully demonstrate value to local participants will likely be unsustainable, regardless of the extent of community participation in the development phase (Hausman 2002; Miller and Shinn 2005; Wallerstein and Duran 2006). In addition, programs developed with community partners but evaluated only from the perspective of academic and policy partners may miss important program effects. Following CBPR principles of co-learning and knowledge sharing (Wallerstein and Duran 2006), assessing program impact in ways that explicitly reflect community-relevant dimensions has the potential to increase the effective and impactful dissemination of program findings across stakeholder groups.

The value of knowing more about community-relevant outcomes related to youth violence has been recognized. A qualitative study by Yonas et al. (2007) revealed a number of neighborhood level factors that community residents perceive as being relevant to youth violence, including groups of congregating youth, lack of locally owned businesses, and trash. Perkins, Meeks, and Taylor (1992) and Caughy et al. (2001) have used systematic social observations to capture locally relevant characterizations of residents’ neighborhoods and build measures that can assess and study these environmental conditions across neighborhoods. These findings help to bring a culturally sensitive perspective to mechanisms of youth violence, with a particular focus at the group or environmental level. However, they do not directly address the problem of how to evaluate the effect of violence prevention programming on community-defined outcomes at the individual level.

Our intention in presenting this work is to begin a process by which community-defined outcomes of youth violence prevention programming can be meaningfully conceptualized, measured and evaluated. Specifically, we investigate how existing, standardized measurement instruments often used to evaluate such programs could be retooled into reliable and valid measures of community-defined outcomes. Long term, developing measures of such community perspectives on program success can not only build theories of etiology which will inform program development but also increase the capacity of program evaluations to meet the information needs of all stakeholders. In the short term, this work can demonstrate that academically-derived, psychometrically sound outcome measures are capable of capturing community perspectives when utilized in a modified way. Namely, this methodological innovation of CBPR could allow researcher-stakeholder teams to utilize preexisting datasets to effectively evaluate community-defined outcomes of ongoing programs.

Methods

Context of the Study

This study is a secondary analysis of data gathered in the context of a community-based youth violence prevention program conducted by the Philadelphia Collaborative Violence Prevention Center (PCVPC), a CDC-funded Urban Partnership Academic Center of Excellence (2006–2011). PCVPC is built upon a basis of CBPR and supported by a multidisciplinary and multi-institutional infrastructure of faculty and community members. Strong links with the community targeted by the prevention program are represented by the integration of community members in research projects, center leadership positions, and advisory capacities. Community and academic partners collaborated to design and pilot the youth intervention component of the Center, the PARTNERS Program: Participatory Action Research to Negotiate Every Response. PARTNERS is a violence prevention and leadership promotion program for youth 10–14 years old and has been implemented in a systematic stepped-wedge cluster randomized trial across six community after-school sites (Brown and Lilford 2006; Leff et al. 2010).

The PARTNERS program evaluation employed a variety of psychometrically sound scales selected collaboratively by PCVPC community and academic partners to assess emotional and behavioral functioning, social information processing, attitudes towards violence, self-esteem, community involvement and leadership (Crick 1996; Eyberg and Pincus 1999; Leff et al. 2006; Leff 2012; Oman et al. 2002b). Data were collected from youth and their caregivers at baseline and post-intervention time points.

For the work presented here, using data collected in the context of PARTNERS provided several key advantages. First, we did not add to the community participants’ research burden, an underlying value consistently expressed by the community. A second advantage was that the PARTNERS evaluation used a variety of established psychometric scales with known reliability and validity to evaluate youth outcomes associated with violence prevention programming. While the focus on available data is expedient, we recognize that the analyses are limited to the dimensions that the existing measures offer rather than creating new items that may better reflect the community-defined constructs. As such, the work presented here is a proof of concept, investigating whether these well-established tools contain elements of the community-defined constructs and if, through reconfiguration, these tools can measure them. We hope to demonstrate an innovative method of measurement development that can bridge the evaluation gap between community and academic stakeholders, and which can be adopted by others quickly and in the context of pre-existing program trials.

Available baseline data from youth and their caregivers were utilized in this methodological study with all identifying information removed. A protocol for the secondary analyses of the existing data was approved by the researchers’ university Institutional Review Board.

Defining Community-Defined Indicators of Program Success

The process of formulating the community-defined indicators of program success began during research conducted during the first, planning year of the PARTNERS project. The research was conducted by a team of academic and community researchers. Rigorous qualitative research methods, including focus groups and participant validation sessions, more fully described elsewhere (Hausman et al. 2009), were conducted and analyzed using iterative coding and consensus processes. Participants were asked to articulate what their community would look like if the violence prevention program were successful. Community-defined indicators of success of youth violence prevention programs were derived and validated through member checking sessions. Finally, an interactive review and feedback method was conducted by the research team and community advisors to add to the validation of the derived indicator constructs. All told, 43 indicators were identified through the qualitative inquiry.

Matching Survey Items to Indicators

The next goal was to craft measures to evaluate the identified indicators. Thus, the community-academic team conducted a multi-stage matching process between the 43 community-identified indicators and the items from the psychometrically sound scales used in the PARTNERS evaluation. The goal of this process was to see if, among all of the scales being used to evaluate youth- and parent-reported outcomes, there were items that could be considered as candidates to populate measures of the community-defined indicators. Items from the existing scales were assigned to the community-defined constructs through an iterative process including reviewing all 208 items, making an initial sorting into constructs, and then having a community-academic panel agree or disagree with the initial sort. Patterns of agreement were reviewed and discussed, and retention of an item into an indicator construct was reached by consensus. This process is more fully described elsewhere (McDonald et al. 2012). Of the original 43 indicators, 17 were matched to preexisting PARTNERS evaluation data.

Initial, simple correlation analyses showed that among these 17 indicators, only a few had sufficient numbers of items and enough data collected to warrant additional evaluation. For the purpose of investigating whether community-defined constructs can be measured using items from preexisting measures, we focused our analyses on two constructs that had reasonable initial internal consistency (all Cronbach alphas >0.70 as per Nunnally 1978, p. 245) and had data from scales administered to both youth and adults. These constructs were labeled by the community-academic panel as “More Parental Involvement” and “Showing Kids Love.”

More Parental Involvement

An extensive literature base demonstrates that parenting is critical to positive youth development (NICHD 2001). Parental involvement is an important aspect of parenting that typically includes parents becoming informed about and engaged in their children’s various activities across family, school, extracurricular, and peer settings. Parental involvement has been consistently associated with improved outcomes, including better academic achievement (Hill and Tyson 2009), less delinquent behavior (Hoeve et al. 2009) and less involvement in violence and crime (Li et al. 2000). In line with this empirical research, our community focus group participants clearly defined parental involvement as a need for parents to be more connected to their children by knowing where they are, who their friends are, and what their interests are (e.g., music, television). Although these ideas generally mirror parental involvement as it is typically defined and measured, our community focus group participants also described how parents need to be leaders at home. Importantly, they need to take initiative to actively intervene and be advocates for their children. The examples below show how parents can become involved in their children’s lives by assisting them with academic and extracurricular involvement and success. For example, parents can initiate communication with schools to find out about potential problems and assist in developing effective solutions.

You can get phone calls from the school … you don’t need a phone call, you call up the school…is there a problem up there? Okay, all right, I’ll be up there to have a meeting. A lot of parents forget that. (Participant #24, male)

Additionally, parents can show leadership with their children by strongly encouraging them to participate in programs and new experiences rather than letting the children back away from these opportunities.

I was saying that even if you did know about the program, it’s no parents that push the kids to go. Because, see, my children, I had to make them go until they found out they liked it and they kept on going themselves… but you know, you got to push them and some of the parents just won’t push them. (Participant #25, female)

Showing Kids Love

Giving and receiving love was put forth by community informants as a powerful change agent within troubled communities. Youth need to know that they are loved. Adults within their family and the wider community, especially in the absence of the parent, need to demonstrate a commitment to youth and show them that they are loved. Adults need to show youth they have value and by example, show kids how to love. Importantly, it was clearly stated that adults set the conditions by which youth feel, and then give, respect and love. Furthermore, adults can be effective with youth because they show and expect love from youth. Coined here as Showing Kids Love, this construct is clearly related to parental involvement, but emphasizes the attitude behind specific actions and elevates the sense of commitment to and caring for youth, and extends this beyond immediate parents or caregivers. These quotations provide a sense of how Showing Kids Love is reflected by the community.

But what happens is we have to reach out because to me, the reason why there is so much violence and drugs and sex is they don’t get the love that they need. All plants need water to grow, okay. And children are the same and when they don’t have it, they will find a way to get something to make them feel like they are appreciated. (Participant #16, female)

They come flopping over to my house. I be laying there trying to figure out why you want to come over here? Why you all, they feel love. I didn’t realize, it’s not material, is not no PlayStation. Oh, yeah, okay, you all finding that love in this house. (Participant #17, female)

Sample

Data from the baseline administration of PARTNERS evaluation tools were used to assess the psychometric properties of each community-identified construct (i.e., More Parental Involvement and Showing Kids Love). Baseline data from 125 youth and 95 parents participating in the PARTNERS research initiative were available by combining intervention and control participants. The average age of the youth sample was 11.04 years; 61.6 % were female; and all identified as members of racial/ethnic minority groups: 80.8 % African American, 2.4 % Latino, 0.8 % Asian American, 0.8 % Native American, and 15.2 % mixed. Youth reported living with their mothers (n = 72, including 10 who also live with stepfathers and 18 who also live with extended family); their fathers (n = 4, including 3 who also live with extended family); both parents (n = 39, including 5 who also live with extended family); extended family (n = 9); and a foster family (n = 1). When both the youth and caregiver from a family provided data in the context of the PARTNERS research initiative, the caregivers who reported on the youth breakdown as follows: mothers (n = 57), fathers (n = 9), both mothers and fathers (n = 3), parent unspecified (n = 1), aunt (n = 7), grandfather (n = 1), foster mother (n = 1), left blank/don’t know (n = 16). Although specific family information was not able to be collected, it appeared that youth and families approximated the urban communities in which we were working where more than 70 % of the population has a high school diploma, GED or higher education; nearly 75 % of residents are African-American; over 25 % of residents live below federal poverty levels and 40 % of adults are in the labor force (Neighborhood Information System 2006).

Measures

The PARTNERS team selected a battery of ten standardized instruments for data collection with youth and four for parents. All of the items from this battery were available to the community-academic panel during the matching process, but More Parental Involvement and Showing Kids Love were derived from only three of the measures. The Alabama Parenting Questionnaire Child and Parent Forms (APQ; Shelton et al. 1996) evaluate dimensions of parenting including parental involvement, positive parenting, poor monitoring/supervision, inconsistent discipline, corporal punishment, and other discipline practices from the perspectives of the youth and the parent. For this study, it is important to note that as a result of community feedback, the three items included in the corporal punishment subscale were not administered as part of the evaluation battery. In addition, the language in the item “Your mom/dad drives you to a special activity” was modified to be more appropriate to an urban setting where many families walk or use public transportation. The APQ has an extensive literature base documenting its reliability and validity, and it appears especially appropriate for evaluating parenting practices associated with antisocial and delinquent behavior (Essau et al. 2006). The Hare Area Specific Self-Esteem Scale (HSES; Shoemaker 1980) measures youth’s self-esteem, or self-perceptions of worth and importance, across peer, home, and school contexts. Research, including work with urban, African-American adolescents, supports the reliability and validity of the HSES (Hare 1977; Vacek et al. 2010). An abbreviated, ten-item version of the HSES was utilized in this study as part of the effort to minimize the burden on participants. Lastly, the Youth Asset Survey (YAS; Oman et al. 2002b) evaluates eight youth developmental assets, three of which were utilized in this study, including family communication, responsible choices, and community involvement. Again, we note that based on community feedback, the family communication subscale item assessing communication about sexual behavior was not administered as part of the evaluation battery. The YAS has adequate psychometric properties and has been demonstrated to be inversely associated with youth risk behavior, including substance use, fighting, truancy, sexual activity, and involvement with the police (Oman et al. 2002a, b).

Analytic Approach

Our goal was to evaluate the internal consistency of each of the measures created through the matching process (i.e., More Parental Involvement and Showing Kids Love) utilizing Cronbach alpha and to better understand the underlying dimensional structure (construct validity) of the items comprising the measures utilizing exploratory factor analysis (EFA). The item-to-construct matching process allowed individual items to be matched to multiple constructs. In order to maintain the separate nature of the community-defined indicators, we ran EFAs separately for each construct. Though this resulted in 7 shared items between constructs, which is apparent in some high correlations between subscales, we felt that the separate analyses were warranted. All analyses were completed using SAS software, version 9.3 (SAS Institute Inc. 2011). EFA of each scale was accomplished in several steps. The initial extraction of factors was accomplished either with principal factors or principal components methods. For the principal factors method, squared multiple correlations (SMC), which serve as the initial estimates of communality, appeared in the diagonal entries of the item correlation matrices. Final communality estimates are defined as the variance of observed items that are accounted for by the retained common factors. These are computed by squaring the variables’ factor loadings for all retained factors and summing these squares. We obtained the percent of variance explained by the factors by dividing the sum of squared loadings by the total variance. Because we factor analyzed correlation matrices, total variance was obtained by summing the number of items under consideration. Although we attempted to factor analyze each extant scale, when the reduced correlation matrices with SMC on the diagonal were determined to be singular by the software, we used “ones” on the diagonal instead. This is tantamount to running a principal components analysis (PCA) instead of an EFA. Although in theory there are substantive differences between EFA and PCA, both techniques nonetheless satisfied our primary goal, which was to identify and interpret the constructs that presumably underlie a given set of variables. In the results section we used the term “factor” to describe the constructs even when they were identified with PCA.

After the initial extraction, eigenvalues >1.0, scree plots, and maximizing percent of explained variance were used as criteria for determining how many factors to retain. When two or more factors were retained, the Promax technique was used to rotate the correlated factors to satisfy the following conditions: meaningful factor loadings were defined as 0.40 or greater; there are at least three items with meaningful loadings on each retained factor; items on different factors were perceived to be measuring different constructs and rotated factor structure patterns demonstrated simple structure. Simple structure is defined as relatively high loadings on one factor and near zero loadings on others, and most factors have relatively high loadings for some variables, and near zero loadings for the remaining variables. Finally, interpretability of factors was also a major consideration in satisfying the criteria for final solution. Two community members partnering in this research from the outset contributed to the interpretation of the derived factors through an interactive review process of the factor analysis results with two of the academic researchers. Together, the community-academic research panel reviewed the individual items and discussed what the factors identified in the factor analysis might mean. Through this discussion, individual factors were given names and preliminary interpretation of meaning. As such, the opinion of the community partners reviewing the factor analyses as to the saliency of the item groupings played a significant role in determining the solution. For the most part, these criteria were met in our final solutions, though exceptions were occasionally made (see below for details).

Finally, the items that were identified with EFA/PCA were summed to obtain unit weighted composite subscale scores. Pearson correlations were run separately on these subscale scores for youth only, adults only, and finally between youth and adult informants.

Listwise deletion was utilized for all analyses. For the APQ, youth were instructed that they could answer the “Dad” questions about whomever they considered their “Dad,” or if they did not have a Dad at home, they could skip those questions. Eight youth responded to these directions by not answering any of the “Dad” questions. Because the APQ items were utilized in both measures, these 8 youth were listwise excluded from all analyses of the youth data.

Results

More Parental Involvement

Self-reported by Youth

One hundred and eight youths’ responses on 22 items were included for this analysis. The Cronbach alpha for this grouping of items was 0.84, demonstrating strong internal consistency. For the initial extraction, SAS reported a singular correlation matrix, which prevented convergence on a factor solution; therefore, a PCA was run instead. Four factors were retained in the final solution accounting for 61 % of the total variance (see Table 1). The Promax (oblique) rotated factors revealed relatively low inter-factor correlations (rs = 0.03–0.34), and produced relatively simple structure and good interpretability.

Table 1 Factor structure of youth-reported More Parental Involvement
Full size table

The semi-partial correlations (loadings) between factors and variables revealed at least 3 variables loading on each factor. Although the loading of 0.31 for APQ17 “Your parents don’t know the friends you are with” did not meet our 0.40 criterion for “meaningful” loading, it still seemed reasonable to group this item with the other three that defined the 4th factor because this item’s loadings on the other three factors were essentially zero (0.03, −0.07 and 0.03, respectively).

Factor Interpretation

The community partners had the following interpretations of the factors drawn from the youth data on More Parental Involvement. Not surprisingly, items for Moms and Dads loaded separately. Factor 1 was labeled “Relation and Connection with Dad,” and this factor included both daily and special activities, as well as capturing the flavor of the relationship with Dad. Factors 2 and 3 both relate to Mom and separate out daily activities (Factor 2; “Daily Interactions with Mom”) from special activities (Factor 3; “Special Activities with Mom”). For Mom, there are the activities that she does with the child when she sees him/her (Factor 2) and those she might plan or desire to do, if she had time (Factor 3). The community perspective on this separation of activities focused on the fact that many moms try to do special things with their children but they often do not have the time. Many work multiple jobs and/or are responsible for all household management tasks. Interestingly, Factor 2, “Daily Activities with Mom,” includes one general parent item: APQ5 “Your parents reward or give something extra to you for behaving well.” In part, this might reflect the majority of mother-headed households in this sample of youth. For Dad, the grouping of daily and special activities was interpreted as also possibly reflecting youths’ living situations. With most mothers being the usual primary caregiver, the activities with Dad may seem to be more of a highlight for the youth respondents and somehow more distinctive because of when they occur. Factor 4 captures “Attitudes and Practices of Both Parents” and focuses on the notion that parents need to know about their children’s lives in order to be effective advocates.

Self-reported by Adults

Ninety adults’ responses on 17 items were utilized in this analysis. Because the correlation matrix with SMC on the diagonal was singular, a PCA was undertaken here as well. Two factors were retained in the final solution, which accounted for 57 % of the total variance (see Table 2). The oblique rotation revealed an inter-factor correlation of 0.21 and factor loadings demonstrated relatively simple structure. The Cronbach alpha was 0.77. APQ5, “You reward or give something extra to your child for obeying or behaving well” had only a 0.33 loading on its factor, but it was considered as belonging with this set because its loading on the other factor was only 0.03.

Table 2 Factor structure of adult-reported More Parental Involvement
Full size table

One item that was originally matched to this indicator, APQ29 “You don’t tell your child where you are going,” not only did not significantly load on either factor but was also considered to be a poorly worded question by the community partners. Their thoughts were that in this community, parents do not generally tell their children their plans for the day at that level of detail, although they would expect details from the child about the child’s day. Additional thoughts were that a parent might not want to admit to a researcher that they do not follow this practice of providing details to their children and so would not answer truthfully. Alternatively, it could be irrelevant if the child spends much or all of his or her time with the parent, again making it difficult to answer accurately. The community partners felt that the idea behind this question was important to include in the More Parental Involvement scale, but that it should be re-worded. Thus, APQ29 was dropped and the analysis was re-run with the 16 remaining variables. This group of variables achieved a Cronbach alpha of 0.79, demonstrating acceptable internal consistency.

Factor Interpretation

These two factors were named “Activity with Child” and “Parenting Practices,” respectively. The first factor, “Activity with Child,” includes a range of daily and special activities, and also includes relationship-related activities such as having a friendly chat. Also included is the willingness to volunteer for activities that are supportive or important to the child. The low loading of APQ5, “You reward or give something extra to your child for obeying you or behaving well,” was interpreted by the community as reflecting a poor fit of this particular question for the PARTNERS study population. On the one hand, there is an expectation that children will behave well, so no extra reward is necessary. However, extra rewards can occur when the child exhibits exceptionally good work, so the concept of “reward” is not completely irrelevant. Thus, the question was worth keeping in the measure, although interpreted with caution.

The second factor was named “Parenting Practices” because it captures the things parents “should do” to be both protective of their child and to be more aware of what is going on in their lives. Importantly, this factor includes one element of advocacy that was clearly stated by community focus group participants as important for parents: attending school meetings. While the community partners agreed with the importance of asking the questions in this factor, they expressed the need for a measure to ultimately be able to separate out what parents desired to do from what they are able to do. Parents have a lot to do and often do not get to focus on their children until late at night. Parents do want to be there to do homework with their children but addressing basic needs often take priority.

Showing Kids Love

Self-reported by Youth

One hundred and eleven youths’ responses on 14 items were utilized in these analyses. A true EFA was run using principal factors procedure with SMC on the diagonal to extract three factors, which accounted for 66 % of the variance. Promax was used to rotate the factors to a final solution (see Table 3). Although three factors provided the clearest grouping of variables, YAS1, “How often does your mother or father (or another adult in your house) try to understand your point of view,” loaded rather consistently on all three factors (0.26, 0.21 and 0.23). This item was dropped and the analyses repeated on the remaining 13 items, which achieved a Cronbach alpha of 0.76 and accounted for 66 % of the variance. Problematically, only two variables had meaningful loadings on the third factor. Although not an ideal solution, alternative solutions by retaining different number of factors or deleting items did not improve the situation. For instance, excluding YAS1 from the analysis did not alter the doublet structure of the third factor, and retaining only two factors produced several variables with meaningful loadings on both factors. The inter-factor correlations revealed a relatively high correlation between the first two factors (r = 0.51) but relatively low correlations between the first two factors and the third (rs = 0.15 and 0.26).

Table 3 Factor structure of youth-reported Showing Kids Love
Full size table

Factor Interpretation

The community partners named these three factors “Feelings/Perceptions of Love and Regard,” “Demonstration of Love,” and “Dad Love.” The panel was clear about the notion of “regard” as different from love but clearly relevant to positive youth development. The distinction between feeling love and acting out love is particularly poignant. The earlier quotations point to the perception that youth feel the love that adults can give in subtle and passive ways, such as letting them “hang out” in their homes, rather than by discrete actions such as hugging or kissing. In Factor 2, “Demonstration of Love,” we see a reflection of the expectations for rewards for good behavior as noted in the parenting construct. From the community perspective, children generally should not expect rewards or praise for behaving well as this parental behavior is not a norm. However, if they do something exceptional, or act beyond what is considered their usual responsibility, then a reward is appropriate and should be expected. For Factor 3, “Dad Love,” the community partners suggested that as moms are the primary parents in this neighborhood, when Dad does something like ask about a child’s day, it seems special and noteworthy.

Self-reported by Adults

Using 10 items from 91 adults, only one factor was extracted, which accounted for 78 % of the variance (see Table 4). Together, these 10 items achieved a Cronbach alpha of 0.84. Rotation was not possible with one factor, but the factor pattern revealed meaningful loadings for all items except for APQ5, “You reward or give something extra to your child for obeying you or behaving well” (0.34), which was only slightly below our meaningful criterion.

Table 4 Factor structure of adult-reported Showing Kids Love
Full size table

Factor Interpretation

This factor was interpreted as “Demonstration of Love” and includes both daily and special activities. The items reflect actions that the parent/caregiver takes that show the child they are loved. It is interesting to note that the items here partially mirror the items in the same factor as measured in youth, with the exception of the friendly chat with Dad. It seems that the adults feel that the chat or asking about activities for the day are signs of love similar to demonstrating affection or doing activities with the youth, while the youth see it as something different. As noted above, parents (and here they are mostly Moms) desire and plan to do all of these activities, but may not actually get to do them. Here again, we see that the item APQ5, “You reward or give something extra to your child for obeying you or behaving well,” which was matched to More Parental Involvement, also has a weak loading. This again suggests that the word “reward” doesn’t work well for the community.

Correlations Across Informants and Constructs

Unit weighted composite subscales were created by summing the items as indicated by the factor solutions. Pairwise Pearson correlations across informants (youth and adults) and constructs (More Parental Involvement and Showing Kids Love) are presented in Table 5. Correlations appear to cluster by reporter, with medium to large effect sizes noted within informant report subscales across constructs. Notably, the two youth report subscales related specifically to fathers evidenced the weakest correlations with the other youth report subscales. Lastly, the correlations between youth and adult report across constructs were largely of small effect sizes. There was, however, a pattern of medium effect sizes noted between youth reported subscales related specifically to mothers and demonstration of love and adult reported subscales related to activities with child and demonstration of love. These patterns likely reflect the same themes evidenced in the factor interpretations; namely, that mothers frequently play the role of primary caregiver in this community, while fathers’ interactions with their children may be both less frequent and more “special.”

Table 5 Correlations between EFA/PCA derived subscales across reporter (youth and adult) and construct (More Parental Involvement and Showing Kids Love)
Full size table

Discussion

This study was a proof of concept effort to demonstrate a methodological innovation for CBPR that blends the participatory process with program evaluation. Specifically, we demonstrate that measures can be co-constructed by academic researchers and community members to evaluate community-defined outcomes of violence prevention programming for youth. In addition, we describe a participatory process that led to creating those measures by re-organizing items from existing standardized psychometrically sound scales and validating them through the use of existing data. Although generating brand new measures for the community-defined outcomes is preferable in an ideal world, we demonstrate that existing data can be retooled to meet community interests in evaluation. The resulting More Parental Involvement and Showing Kids Love measures were reliable, based on alpha coefficients, and the subscales derived through the use of EFA were valid, based on community interpretation of the factors. Given the expense of community-based evaluations and the sheer number of already-funded intervention studies, this innovative approach allows researcher-stakeholder teams to “recycle” their data into an evaluation plan that equally considers both emic and etic perspectives. In this way, our methods are innovative, replicable and add a measurement tool to the CBPR toolbox.

What We Learned

Asking community partners in evaluation studies to identify locally relevant outcomes is standard practice. Similarly, it is common to ask community partners to review standardized scales and make small modifications to address cultural or language conflicts, such as described above in the PARTNERS evaluation. In contrast, our innovation was to create new quantitative constructs with our partners that more accurately reflected their experience and perceptions. The two community-defined indicators presented in this study appear to be more complicated than traditional outcome measures. For example, the list of items for More Parental Involvement was longer than the Involvement subscale of the APQ (Shelton et al. 1996), suggesting that the community-defined construct of More Parental Involvement was more inclusive than that of parent involvement as defined by the APQ. Similarly, Showing Kids Love is about adults developing personal relationships with youth in the community, no matter whose children they are. Showing Kids Love may also be qualitatively different than other measures of similar constructs, as there is an implied expectation that adults should seek out and offer support to the youth, rather than reacting to a specific request from a youth. Recent qualitative research with youth gang members supports this notion that youths’ perspectives on perceived adult helpfulness is important to understanding the role of adult social control in youth behavior and suggests that additional work to develop measures that reflect the youths’ perspective needs to be done (Wilkinson 2007). Furthermore, even fully CBPR processes do not always include youth perspectives, and the evidence in our work here that youth/adult views are overlapping but not identical supports a wider inclusion of youth in measurement development research (Jacquez et al. 2013).

Some of the difference between the community and academic conceptualizations of important constructs may come from a cultural or economic framework reflected in the psychometric scales that does not fully capture the experience of the community. For example, within the inner city, low-income, African American community characterized by this study, structural issues such as competing priorities for parents’ time limiting the time they can spend with their children do not appear to reflect well in some of the items used from the available scales. There are also some underlying cultural assumptions made by the APQ, such as expectations for children’s good behavior and how that is rewarded (or not), that may also suggest that APQ items do not fully capture parenting constructs as the community perceives them. In light of these conclusions, it is interesting that the participatory process we used to identify items, re-organize them, and then interpret them yielded reliable and valid measures of the constructs that were acceptable to the community. Perhaps more importantly, the process also taught us more about the original scales and how they might be limited in their utility, at least with the community characterized by this study. This demonstrates the mutual benefit gained by the researcher and the community of using CBPR in the context of evaluation studies; specifically, CBPR can not only inform and strengthen traditional methods, but also generate new procedures that channel the strengths, knowledge, and capacity of the community.

Whether the actual constructs of More Parental Involvement and Showing Kids Love are generalizable to other communities is yet to be determined. These constructs have emerged in other studies of demographically similar but distant communities in the same city and in the same community 6 years apart (Hausman et al. 2005). Thus, we can begin to ask the question of generalizability through replication in distinctly different communities. At the same time, the process we used to develop these measures was intentionally specific to the local context. In this way, we achieved the most important goal: the constructs are well suited to the locality where the violence prevention program was designed and implemented. An important next step related to developing these locally specific and culturally valid measures will be first to compare them to the traditional measures with an eye toward elucidating the advantages and disadvantages of each approach. We have started this step related to the conceptualization and measurement of parenting, with a particular focus on the APQ (Baker and Hausman 2012). In addition, we also plan to evaluate the impact of PARTNERS on these two community-defined outcomes using the measures we have developed here. Further, we are pursuing the development of a new measurement tool for constructs that relate to youth-adult relationships that will be applicable to a much wider range of communities (Hohl and Hausman 2013).

Challenges

Earlier publications presenting the details of how the constructs were derived and the items were matched to the constructs considered the limitations related to those processes (Hausman et al. 2009; McDonald et al. 2012). Most relevant to the work presented here is that only scales being used by the PARTNERS evaluation were considered for the measurement development process. We intentionally did not create new questionnaires to specifically reflect the constructs of the indicators since one of the underlying goals was to utilize existing data so to not increase the research burden on community participants. The advantage of relying on the tools being used by the evaluation is that data were immediately available for measurement testing, allowing us to demonstrate the feasibility of constructing community-derived measures from pre-existing data. Using existing and available measures is supported by the relative success of matching items from the PARTNERS evaluation to at least a subset of community-defined indicators. The APQ alone, for example, contains items that fit well with More Parental Involvement (youth and adult measures) and at least the adult measure of Showing Kids Love. It may be that one application of the APQ can be used to provide results on both standard psychometric and community-defined outcomes, saving valuable time and resources.

Another limitation is that reviewers considered each item individually during the matching process (i.e., subscales previously determined to exist were not considered as meaningful groups of items) and the items included in the analysis were determined by the matching process (i.e., not all items from the standardized scale were used). Additionally, some aspects of the community-derived indicator definitions were left to interpretation of the community and academic panelists who performed the match, potentially leading to inconsistent ideas of their meanings. Yet, as our previous analyses demonstrated (McDonald et al. 2012), there were patterns of high agreement among raters and results indicating acceptable internal consistency at least among the reviewers. Furthermore, the same community members were involved in all phases of the project; thus, in line with CBPR principles, there has been consistent interpretation within this specific community context. Though our samples are sizable for community-based research, our reliance on the PARTNERS evaluation data did mean that they were somewhat smaller than we might have preferred when running factor analyses. As noted above, some of our data are systematically missing those youth (8 out of 125) who chose not to answer questions about Dad. However, of these 8, 5 reported living with their mothers, 2 reported living with their mothers in addition to extended family members, and 1 reported living with his/her mother in addition to his/her stepfather, reflecting similar proportions to the overall sample. Thus, these systematically missing data are unlikely to affect the overall results.

It is also important to note that not all of the adult respondents of the survey are the biological parents of the paired child. This may be an indication of another limitation of some standardized scales and perhaps the similarity of the two constructs of More Parental Involvement and Showing Kids Love: the use of “parent,” rather than “caregiver” might blur distinctions of expectations of behaviors of adults on the part of their own children or other youth with whom they have some relationship (Showing Kids Love). Similarly, it is important to recognize that the father constructs assessed here (e.g., “Relation and Connection with Dad” and “Dad Love”) are construed broadly and relate to male role models rather than biological fathers specifically. Thus, culturally-informed evaluation in low-income, urban areas must take into consideration non-traditional caregiver roles (McAdoo 1997; Sanders 1996; Thomas et al. 2008).

Summary

In sum, this study suggests that new measures need to be developed to fully evaluate community perspectives on important youth violence prevention outcomes, and that community representatives must be included in that process. In the context of this study, community definitions of adult relationships with youth (e.g., More Parental Involvement and Showing Kids Love) are similar but not synonymous with traditional definitions of these constructs. The differences are nuanced but important, and they are worthy of continued investigation. Existing scales can be used to capture some of this nuance although it requires re-organizing items and re-analyzing data. Indeed, the participatory process we described in this study to retool existing measures addresses several key CBPR principles and is a method that can be used in any community by any investigator team that uses standardized instruments and has existing data. The defined outcome constructs are community-driven potentially leading to increased ownership of the standardized measures by community partners. They also teach the traditional academic researcher about the lived experience, and they provide a way to show how an intervention might be able to have impact on outcomes owned by the community. While the locally relevant outcomes might not apply elsewhere, the methods we used might facilitate participatory projects everywhere.