Introduction

The Elluam Tungiinun (Toward Wellness; ET) and Yupiucimta Asvairtuumallerkaa (Strengthening our Yup’ik Identity; YA) studies were designed to address the paucity of evidence on what constitutes effective intervention with American Indian and Alaska Native (AIAN) youth to reduce alcohol use disorder (AUD) and suicide risk (Gone 2007; Gone and Trimble 2012). The existing epidemiological evidence indicates that the consequences of these twin behavioral health concerns of AUD and suicide constitute an enormous source of health disparity among Alaska Native (AN) people (Allen et al. 2011). The limited existing research literature on suicide and AUD prevention efforts seeking to address these disparities with AI/AN populations, and the IndigenousFootnote 1 model of protection guiding the current intervention is reviewed by Allen et al. (2014a).

People Awakening Program of Research

This report evaluates the feasibility of an intervention model based in over 15 years of community based participatory research (CBPR) with AN communities. The original aim of the People Awakening program of research was to describe an indigenous model of protection (Mohatt et al. 2004; Allen et al. 2006) and natural recovery (Mohatt et al. 2007) from AUD. Over time, a long-term aim became to develop a cultural intervention to build the protective factors we had described. We term this resulting community intervention (Trickett et al. 2011) cultural intervention, which we define as an extension of multi-level, culturally situated intervention (Schensul and Trickett 2009). Cultural intervention contrasts with culturally situated intervention in that culture is both a central focus of all intervention activities, and the underlying theory guiding intervention is also indigenous to the culture.

Through this program of research, we accumulated evidence that this Indigenous model of protection from AUD also provided protection from suicide risk among AN youth (Allen et al. 2014b). In response to deeply held community concerns regarding the ongoing epidemic of AN youth suicide (Allen et al. 2011), the CBPR partnership determined to expand the focus of intervention development to prevent both AUD and suicide (Allen et al. 2014a). With this, the process of CBPR moved from qualitative and mixed methods discovery-based research seeking to describe an indigenous model of protection, to the intricacies of culturally appropriate measurement development described by Gonzalez and Trickett (2014). Allen et al. (2014b) then report on the next step in this programmatic research effort, which was to test the operating characteristics and further refine these measurement instruments. These resulting instruments tapped new constructs and adapted existing constructs for measures of intervention outcome. This work also permitted an empirical test of selected key elements of the proposed indigenous model of protection. Alongside these developments, work also proceeded on elaborating and implementing a multi-level cultural intervention described in Rasmus et al. (2014).

Conceptualizing Culturally and Contextually Responsive Outcomes

Defining the outcomes of intervention required negotiation across several crucial contextual and ethical concerns (Gonzalez and Trickett, 2014). In rural Alaska, a majority of communities have declared themselves “dry,” meaning possession of alcohol is illegal (Berman et al. 2000). These communities have small populations (e.g., population 100–1,200) with 30–100 youth. Acknowledgement of drinking on typical alcohol use measures could create double jeopardy for rural AN youth. In many communities local option laws make drinking illegal, and the risks of deductive disclosure of sensitive information are higher in smaller communities. Not surprisingly, youth were reluctant to answer direct questions about alcohol use truthfully. Past instances where reporting of research results stigmatized local communities and the region as a whole also argued against the use of direct questions about alcohol use as outcome variables, and led to development of an alternative construct. Umyuangcaryaraq is a Yup’ik word translatable as “reflecting” and is one component of a broader cultural value of awareness of interconnections between people, animal, and spirit worlds, and of the resulting consequences of one’s actions, described by the Yup’ik as a part of ellangneq or awareness—literally “awake” (Allen et al. 2006; Mohatt et al. 2011). As an ultimate outcome variable, reflective processes (Allen et al. 2012) taps a narrow element of ellangneq—youth awareness about the broad potential negative consequences of alcohol use; this assessment of a reflective capacity was based on previous work with AN adults on factors protective from alcohol (Allen et al. 2006).

Similarly, Gonzalez and Trickett (2014) describe local community concerns about the advisability of asking their youth first person questions about suicide. Yup’ik cultural beliefs include a profound respect for the person; such intrusive questioning can represent a violation of this respect. In addition, the deep cultural regard held by many Yup’ik for the power of words acknowledges the ways in which the use of words can influence the likelihood of events. From this cultural frame, questioning about suicide might by itself influence youth life choices and decisions related to these choices. And more deeply, within traditional Yup’ik cultural understandings, all things, including suicide, have a spirit essence. To speak of it can “feed” the spirit, making it more powerful. The Brief Reasons for Living Inventory for Adolescents (Osman et al. 1996) taps reasons why a youth would not end life when feeling suicidal. While drawing from the factor structure of this existing construct, we instead emphasized cultural beliefs and experiences that make life enjoyable, worthwhile, and provide meaning for young people, as Reasons for Life, without reference to the presence or absence of suicidal feelings. Yuuyaraqegtaar, Reasons for Life, or literally, “a way to live a very good, beautiful life”—described in greater detail in Allen et al. (2014b)—provided a second culturally appropriate ultimate outcome variable.

The Current Study

Using these two new constructs as ultimate outcomes, this paper aims to address two questions: (1) is the proposed intervention implementable in rural Alaska Native communities, and (2) does the intervention produce measureable effects? This first question is explored through assessments of adherence, quality, dosage, and reach. The second question, about measureable effects, is distinct from a test of intervention efficacy, which would instead require a full prevention trial using an appropriate experimental or quasi-experimental design. The goal of feasibility evaluation is instead to provide evidence that the intervention is sufficiently promising to warrant the considerable expense and effort of a full prevention trial. This second aim also involves questions about (a) adequacy of instrumentation, and (b) impact of contextual factors on implementation. Accordingly, in addition to providing data relevant to our two main questions, this feasibility evaluation has two secondary aims: (3) further extension of the existing measurement development work to produce measures that were maximally sensitive to change, and (4) comparison of the implementation processes and the resulting effects across two different communities.

With regards to the measurement development aim, classical scale construction focuses upon the internal consistency of a scale, typically assessed through the coefficient alpha statistic. This is useful when the purpose is the detailed mapping of a construct in order to allow for testing of a theoretical model (e.g., Allen et al. 2014b). However, measures of static constructs may not be useful for measuring change (Collins and Cliff 1990; Nunnally and Bernstein 1994, p. 245), which is the goal in a prevention outcome study. This feasibility study provides an assessment of alternative approaches to adapt existing measures mapping stable traits into measures that are maximally sensitive to change.

With regards to the implementation process, this project implemented the intervention in two communities. CBPR with distinct AI/AN communities often creates distinct interventions. Contextual differences on the community level interacted with differences in resources aligned with different funding mechanisms, and the ET and YA projects unfolded in different ways. Standardizing interventions by the underlying functions an intervention serves (functions in the current intervention were the protective factors promoted) instead of its form or components (replicating specific activities) can assist in describing CBPR interventions and evaluating their effects across different contexts, including culturally distinct settings (Henry et al. 2012). Using this conceptual framework, we sought to describe the impact of context through systematic exploration of intervention differences potentially responsible for differences in the measureable effects we observed.

Based in earlier findings from a test of the theoretical model for intervention (Allen et al. 2014b), we hypothesized that (a) individual, family, peer, and community level intermediate variables would increase post intervention, and (b) ultimate variables of Reflective Processes and Reasons for Life would also increase. Feasibility assessment represents a first step in developing data that can lead to evidence based preventive cultural interventions for AN youth.

Methods

Participants

ET Study

Sixty-one youth were recruited to participate in the intervention from approximately 100 12–17-year olds residing in a community of approximately 650 total population. Sixty youth completed Wave 1 assessments, 46 completed Wave 2, 43 completed Wave 3, and 61 completed Wave 4, resulting in 37 youth completing all four waves of assessment (T1–T4), 8 that completed three assessments, and 10 that completed two assessments. Because analyses with small samples are particularly susceptible to influence from outlying observations, we identified multivariate outliers using hierarchical cluster analysis (Henry et al. 2005), a statistical method that detects homogenous clusters of cases by iteratively grouping cases together based on distance computation. Using this approach we explored scores across the measures for multivariate outliers, and we identified one youth who was distant from others. In addition, five youth who completed only the final assessment were dropped from the analysis, resulting in 54 participants. Demographic data are presented in Table 1 for this sample of 54 participants used in the analysis. Mean age of the sample at Wave 1 was 14.2 years. The gender distribution was 31 females and 23 males, and there was no significant age difference between males and females (t(58) = 1.02, ns). All youth reported Yup’ik ethnicity.

Table 1 Youth demographic characteristics
Full size table

YA Study

Fifty-three youth were recruited to participate in the intervention from the approximately 100 12–17-year olds residing in a community of 530 total population. Of the 53 youth, 48 completed four waves of assessment, and five completed three waves. Using hierarchical cluster analysis, we identified and excluded one multivariate outlier, a youth who was distant from others across the measures, resulting in data from 52 youth to be analyzed. Table 1 presents demographics for these 52 youth; the mean age of the sample was 14.6 years at program entry, the gender distribution was 25 females and 27 males, and there was no significant age difference between males and females (t(50) < 1, ns). All youth were Yup’ik.

Measures

Measure Development Process

One aim of this study was to develop unidimensional scales that are maximally sensitive to change, using as a start point the scales used in the test of the measurement model for intervention.Footnote 2 Here we describe procedures used to convert these longer classical test theory based construct mapping scales into brief measures of change. Samejima’s (1997) Graded Response Model, an item response theory (IRT) approach, allowed us to assess both the functioning of these scales and the coverage of their items across the full range of the latent trait found among individuals being measured. IRT generally, and the Graded Response Model in particular, lends itself particularly well to creating scales that are maximally sensitive to change using fewer items than scales created through classical test theory.

We aimed for scales whose items covered the entire range of the latent trait, and whose response scaling provided information from each anchor. The computer-administered surveys used an analog scale for responding: respondents slid an image of a Yukon salmon in a continuous motion across a horizontal sea blue background with three semantic anchors placed below (salmon are a valued subsistence food). For most questions, at the suggestion of our Yup’ik linguistic advisors, these anchors read, ‘Not at all,” “Somewhat,” and “A lot.” For analysis, these analog responses were divided into five segments. IRT assisted in locating cut-offs optimizing the function of response options. Family Characteristics used a true/false format and Peer Influences used a four alternative radio button response format to use the same response format from the original instrument from which these measures were adapted.

The results of the IRT modeling produced scales that included a mix of items distributed along the continuum of item difficulty, from “easy” (e.g., participants with low levels of the latent trait endorsed the item) to “difficult” items (e.g., only participants with high levels of the latent trait endorsed the item). When two or more items covered the same part of the scale, we retained the item with the higher discrimination index, i.e., items whose scores correlated more strongly with the latent trait. Thus, the resulting outcome measures were composed from the best functioning items from Allen et al. (2014b) at graded levels of item difficulty, were much shorter in length than those used in testing the measurement model, and were designed to function as measures maximally sensitive to change. Table S1 (available in the online Appendix S1) lists the constructs, scale and subscales tapping the construct, the final number of items composing the change measure scale, and variable status as intermediate or ultimate variable.

Elluarrluni Piyugngariluni: “Learning in the Mind of Doing Things in a Masterful Way”: Individual Characteristics

This measure consists of three subscales based on the Communal Mastery: Family and Communal Mastery: Friends subscales from the Multicultural Mastery Scale (MMS; Fok et al. 2011a, b). Using the item stems from the five best functioning items of the Communal Mastery scale (Jackson et al. 2000) adapted for Alaska Native youth, five items tap the young person’s belief that he or she can face life’s problems successfully through joining with family, and these five items are then reworded to tap this dimension with regards to friends. Because self-mastery in our test of measurement model test predicted little variance in the ultimate variables (Allen et al. 2014b), we dropped the MMS Mastery-Self Scale, a measure of personal control mastery, from outcome analyses.

Elluarrluteng Ilakelriit: “Nurturing family”: Family Characteristics

This 19-item version of the Brief Family Relationship Scale (BFRS; Fok et al. 2011a, b) includes subscales for Cohesion, Expressiveness, and Conflict. The 19 BFRS items were selected as the best functioning items from the 25 items used in the Cuqyun study, originally adapted from the relationship dimensions of the Family Environment Scale (Moos and Moos 1994).

Nunamta: “Our community”: Community Characteristics

The Youth Community Protective Factors Scale was adapted from the Yup’ik Protective Factors scale that was developed for adults (Allen et al. 2006). Items were derived from statements in qualitative life history transcripts of abstainers and non-problem drinkers that exemplified important components of a Protective Factors model for Alaska Native adults. The items from this measure describing elements of protective communities were adapted into a measure for youth of protective Community Characteristics. The measure consists of a four-item and a three-item subscale tapping Support and Opportunities for youth in the community.

Maryarta: “One who leads”: Peer Influences

This 10-item measure was adapted from two scales from the American Drug and Alcohol Survey (Oetting and Beauvais 1990). The Peer Discouragement of Alcohol, Tobacco, and Other Drug (ATOD) Use Scale (Discourage) and the Disapproval of Peers’ ATOD Use Scale (Disapproval) have been used extensively in research with youth in American Indian tribal communities (Beauvais 1990). The scales were adapted for understandability and relevance to rural Alaska Native youth, and measure peer attitudes that discourage alcohol and other drug use, conceived as protective peer influences in the young person’s social environment.

Umyuangcaryaraq: “Reflecting”: Reflective Processes

The five best functioning items were assembled as an outcome measure from the 12-item Reflective Processes scale (Allen et al. 2012) used in the Cuqyun study. These items were adapted from the adult Yup’ik Protective Factors scale, and the item set taps of reflective processes involving thinking over the potential negative consequences of alcohol use and abuse.

Yuuyaraqegtaar: “A Way to Live a Very Good, Beautiful Life”: Reasons for Life

The five best functioning items were assembled as an outcome measure using a Graded Response Model from the 14-item Reasons for Life scale used in the Cuqyun study. This new measure is an extension of constructs tapped by the Brief Reasons for Living Inventory for Adolescents (Osman et al. 1996) and the adult Reasons for Living Inventory (Linehan et al. 1983). The Reasons for Life scale taps beliefs and experiences that make life enjoyable and worthwhile.

Psychometric Operating Characteristics of Brief Measures of Change

Table 2 reports number of items, coefficient alpha internal consistency, item separation, person separation, and test–retest reliability for each of the outcome measures. Based on a Rasch analysis (Bond and Fox 2001), item separation reliabilities index the extent to which the sample of people is adequate to scale the items and person separation reliability measures the extent to which the items are able to separate people according to their levels of the latent trait. The item separation reliability values (.69–.77) were acceptable for Reflective Processes but the retest reliability and person separation reliability values were suboptimal and close to its alpha values. This suggests that the Reflective Processes items possess limitations in their ability to discriminate between persons at different levels of the latent trait underlying the measure, along with less stability over time than would have been desirable. Results on the Reflective Processes scale should be interpreted with this in mind.

Table 2 Psychometric properties of outcome measures
Full size table

Intervention Procedures

Planning, community development work, and development of prevention activity modules that would constitute the future intervention as implemented was conducted by community members with the assistance of local and university project staff over a one (YA) or two (ET) year time period prior to any module delivery in the intervention activities. The resulting prevention modules were then implemented over a 1-year time period.

Recruitment

Participant recruitment in each community was initiated through announcements at tribal council meetings, followed by meetings with youth service providers and schoolteachers, and through posters placed throughout the community announcing the project. Then, our local prevention staff, which knew and had grown up with most community members, personally approached every parent of an age-appropriate child in the community, inviting the parent and the child to participate in the project.

The Qungasvik: A Toolbox for Community Intervention

Community planning groups developed the interventions with representation from youth, parents, community leadership, Elders, and university researchers through a process described in Rasmus et al. (2014). This work, in the ET community, resulted in the Qungasvik (toolbox; Alakanuk Community Planning Group et al. 2008). Rather than offering a prescriptive manual of specific intervention activities, the Qungasvik was developed as a process manual for intervention. It provides a flexible format, selection from a range of activities at different levels (individual, family, community), and basic outlines for prevention activities, all situated within a community development framework.

Form Versus Function of Intervention Activities

Henry et al. (2012) describe the Qungasvik prevention manual approach as grounded in the underlying function of each intervention activity. This distinction of function, as distinguished from form, or intervention components, has emerged over the past decade in community intervention research (Hawe et al. 2004). Though communities throughout the Yukon-Kuskokwim region of Southwest Alaska are all majority Yup’ik, each is distinct in regards to many local customs. In response, the Qungasvik framework facilitates development of interventions contextualized to these customs. This allows for local variation in form while preserving underlying function—delivery of the same set of protective factors. This process approach to intervention is also intended to maximize community ownership and control.

Multilevel Intervention Activities

The resulting Qungasvik intervention toolbox is organized into 36 modules, and each module addresses 2–5 protective factors. Though each was conceived as an individual, family, or community intervention activity, the actual protective factors to be delivered in a Qungasvik activity can cross levels. For example, the Qasgiq module is a community level module, in which community members constructed a sacred place for learning and teaching the way of life to young people and for community decision-making. One of the five protective factors emphasized in this activity is communal mastery. Although communal mastery involves joining with significant others in one’s family and community to solve problems, the agency experience of mastery occurs on the individual level (Fok et al. 2011a, b), hence it is an individual level protective factor.

Ripple Effects

Additionally, several community activities were stimulated by the process of delivering modules and created ripple effects that went beyond the intent of the Qungasvik. For example, there is a prayer walk module described in the Qungasvik; after the activity was first introduced by the project it was spontaneously repeated multiple times in the community. There was a request to assist the community in developing a proposal for state funding to expand the program, and to advocate for more police protection, both of which were facilitated through the collaboration of community members with the university researchers. In summary, the impact of the development of the toolbox and the intervention activities spread through ripple effects leading to additional community activities beyond the specific modules reported here that were part of the direct intervention activities.

Dose

Table S2 (available in the online Appendix S2) provides the interested reader detailed synopsis of the intervention delivered in each community–describing each Qungasvik module delivered, the protective factors each module was designed to deliver, number of times (0, 1, 2) delivered by the ET or the YA project. The analysis of dose that will be presented in the results accordingly simply charts the impact of amount of exposure to intervention modules on measured change in protective factors as it varies for each individual youth. We seek to understand if a particular level of dose exposure is required to initiate change in the data, and to enhance the internal validity of claim regarding the potential for the intervention to be producing these measureable effects in a dose related fashion. For more detailed reporting of the dose exposure, Henry et al. (2012) provide a latent class analysis of patterns of protective factors exposure among youth who participated in the intervention.

ET Dose

The ET project delivered 26 Qungasvik prevention modules over a total of 52 sessions with youth, families, and the community. Ten of these modules were community level, two family level, and fourteen individual level. However, consistent with cultural values, in practice these events, including those focused on the individual level and therefore designed for youth, were open to all who wished to contribute. For example, siblings, parents, and other family members participated in 10 of the 14 individual level modules designed for youth participation.

YA Dose

YA delivered 15 prevention modules. Six modules were community modules, four were individual youth modules, and five were family modules. However, as in the case in the ET intervention, family members participated in modules that were at the individual level. Also, as in the case of the ET intervention, protective factors delivered in these Qungasvik activities often crossed levels. Seven of the modules delivered community protective factors, three provided experiences in family protective factors, while 10 delivered individual protective factors.

Cross-Community Intervention Differences

The ET and YA projects differed in two important respects. While both projects offered compensation for participating in assessment, in ET but not YA, youth received modest compensation for participating in each intervention model they attended. This was related to ET having more funding due to the differences in the two grant awards. These differences also allowed for more paid local staff and total staff effort in the ET community. These differences in resources allowed ET to deliver more intervention modules, and to assess a subset for fidelity of implementation. Eleven additional modules were offered in ET, in contrast to YA, resulting in greater opportunities for protective factors exposure, as can be seen in Table 3, which reports the protective factors delivered in each community. However, differences in protective factors were focused at the individual level where ET provided opportunities for four times the protective factors as YA. Beyond the individual level, the YA intervention provided greater opportunities for family level protective experience and only 5 fewer community level protective experiences. For the ET but not YA project, youth entered into the intervention program in groups at different times, resulting in varying numbers of assessments at varying time frames. Time of entry was also related to systematic factors such as level of motivation of parents to involve their youth and youth differences in initial interest and openness to engagement. For these reasons, in the analysis, we treated these time of entry groups in ET as different cohorts. Twenty youth who entered the program in November 2006 are referred to as Cohort 1, and 26 youth who entered about 3 months later are Cohort 2. Cohort 3 included 8 youth who entered late in the program, and were assessed only twice, at Wave 3 and at Wave 4.

Table 3 Protective factors delivered through intervention by community
Full size table

Results

Feasibility of implementation of this intervention in rural Alaska was evaluated through an assessment of adherence, quality, protective factors delivery, and reach using observational data from five randomly selected intervention modules. The ability of the intervention to produce measureable effects was assessed through the relation of intervention dose to change over time.

Evaluation of Adherence, Quality, Protective Factors Delivery, and Reach

Adherence, quality, protective factors delivery, and reach were assessed in five randomly selected modules delivered as part of the ET program. All ET modules were video recorded, and recordings of the Qasgiq (community level), Yup’ik Kinship (family level), and Murilkelluku Cikuq—Watch the Ice, Surviving Your Feelings, and Healthy Adolescent Relationships (individual level) modules were assessed. Two observers independently coded the five module activities for adherence, quality, and protective factors delivery.

Adherence

Adherence measures how close the intervention as actually delivered was to its planned implementation model. Adherence was coded by two raters as “yes” or “no” for each of the key processes outlined within the Qungasvik toolbox in each of the five modules, with “no” indicating the process was not implemented following the Qungasvik outlines. AC1 statistics (Gwet 2001) were used for an inter-rater reliability analysis as there was low prevalence in one of the adherence coding categories, and kappa has known sensitivity to a substantial imbalance in the table’s marginal totals. AC1 ranged from .87 to 1.00, with a mean AC1 of .95. Any disagreements between two coders were then resolved by discussion with a third observer. Overall, 88 % of categories of key processes were endorsed as “adhered” by independent observers. This suggested implementers closely followed procedures outlined in the Qungasvik and completed critical content elements of the intervention modules in creating protective experiences.

Quality

While adherence measures whether key processes and content were delivered, quality measures how well these processes and content were delivered. The quality of delivery was coded for each the categories of key processes for each module using a three-point scale (Unacceptable = 0, Acceptable = 1, Ideal = 2). Acceptable was coded when delivery was judged by raters as effective, but recommendations were provided staff for further improvement. Mean inter-rater reliability kappa for raters was .77 (range = .68–.81). Quality scores indicated more than four-fifths of these key processes and content were delivered at either acceptable (29 %) or ideal (57 %) levels.

Protective Factors Delivery

We define this as an assessment of whether protective factors content was actually delivered in an intervention module. Within the five selected modules, two independent observers counted frequency with which the intended protective factors components of the module were implemented. This involved evaluation of whether the experience occurred, whether it was discussed and named in debriefing with the youth following the experience, and whether the underlying cultural values associated with the protective factor were discussed with youth by parents, elders, and staff as part of the module. Overall inter-rater reliability kappa for these ratings was .80. Protective characteristics at each of the three levels (community, family, and individual) were delivered across all five modules but their emphasis varied by module. Across these five modules, individual characteristics (66 %) were implemented more often than family (16 %) or community (18 %) characteristics. Protective factors delivered most frequently per module across the five modules assessed were ellangneq (median = 4, range = 2–6) and communal mastery (median = 2, range = 1–3).

Reach

Reach refers to the participation rate of youth for which the intervention is intended and the representativeness of these participants, as measured by their attendance in the five modules assessed. In keeping with Yup’ik cultural values that emphasize profound respect for individual autonomy and choice, community leadership and parents insisted youth attendance was on a voluntary basis and entirely a youth or parent’s choice. Approximately 60 % of eligible youth in the community and at least one of their parents or an adult mentor enrolled in the program. Sixty-one participants (39 youths, 22 adults; 40 females, 21 males) attended at least one module, and an average of thirty-three participants attended per module. The attendance rate of enrolled participants varied from 46 % (Qasgiq; a family—enrolled youth and parent—module) to 66 % (Murilkelluku Cikuq; a youth module) with an average attendance rate of 54 % across all five modules. Each participant attended an average of 2.74 modules out of five modules.

Measureable Effects

Data Analysis

The analytic model we constructed allowed us to address design and implementation inconsistencies typical to community intervention, and in particular, to work in remote, geographically dispersed global health locations such as rural arctic Alaska. To evaluate intervention effects, we created mixed effects regression models (see Hedeker and Gibbons 2006 for an overview) that accounted for the clustering of observations within individuals. This method, also known as hierarchical linear modeling (HLM; Raudenbush and Bryk 2002; Pinheiro and Bates 2004), permitted use of all data from youth who participated in two or more waves of assessment. The intervention effect of interest was dose, defined as attendance in intervention modules at each of the four time points of assessment; this approach measured effects in response to increasing levels of exposure to the intervention.

The impacts of three potentially confounding factors were evaluated in the model: (1) pre-existing protection, (2) the duration of each individual’s participation in chronological time, and (3) in the case of the ET but not YA project, the cohort of youth with whom the individual entered the intervention. Pre-existing protection was estimated using Wave 1 scores on the Support subscale of the Community Protective Factors scale. We selected this variable based on path analytic findings (Allen et al. 2014b) that indicated the Support subscale scores accounted for the largest proportion of variance among the two ultimate variables of Reflective Processes and Reasons for Life. To approximate pre-existing level of protection, we decided to use only one variable at one time point for two reasons: we sought to preserve four time points in the remaining protective factor assessments to maximize power, and we concluded the single best predictor provided the best strategy for minimizing measurement error in this estimation, given the overlapping domains of error across levels. Regarding time, because participants entered the intervention at different times, time (in days) was centered at the date each individual started the intervention, and was included to model the effects of length of involvement in intervention apart from intervention dose. The interaction between time and dose was entered to explore change in dose effects over time, later in the intervention year. And finally, to evaluate impact of cohort (in the ET study only), the cohort in which each youth began the intervention was represented by two dummy codes that compared Cohort 1 with Cohort 3 participants, and with Cohort 2 participants, the largest group. Thus, each model was set up to evaluate the effect of intervention dose while controlling for individual variation on each of these potential confounds. All appropriate interactions among dose, time, pre-existing protection, and cohort were entered into the models, along with random effects accounting for variation in individual intercepts.

Prior to analysis, we identified and removed multivariate outliers. Range standardized summed scale scores of these outcome variables were used in all analyses (e.g., the minimum was subtracted from each score then divided by the range, expressing each score as a proportion of the range). In initial runs of the model, we examined residuals and where necessary, we used square-root transformations to normalize residual distributions. Models were fit with the SPLUS LME package (Insightful Corporation 2005).

At level 1, the outcome variable at each of the four time points was predicted from an individual intercept, linear time slope, and the interaction between dose and time. At level 2, the individual level, the level 1 coefficients were predicted by time 1 protective factors (preexisting protective factors), and when the individual became involved in the intervention (cohort). In HLM notation, the model may be expressed as followsFootnote 3:

$$ \begin{aligned} {\text{Level 1 }}\left( {\text{time}} \right): \, & {\text{Y}}_{\text{ij}} = {\text{ B}}_{{0{\text{j}}}} + {\text{B}}_{{ 1 {\text{j}}}} \left( {\text{time}} \right)_{{}} + {\text{B}}_{{ 2 {\text{j}}}} \left( {\text{dose}} \right)_{{}} + {\text{B}}_{{ 3 {\text{j}}}} \left( {{\text{time}}*{\text{dose}}} \right) \, + {\text{e}}_{\text{ij}} \\ {\text{Level 2 }}\left( {\text{individual}} \right): \, & {\text{B}}_{{0{\text{j}}}} = {\text{G}}_{00} + {\text{G}}_{0 3} ({\text{high vs}}.{\text{ low protection}})_{{}} + {\text{G}}_{0 4} \left( {{\text{cohort 2 vs}}.{ 1}} \right)_{{}} \\ & \quad \quad \quad + {\text{G}}_{0 5} \left( {{\text{cohort 3 vs}}.{ 1}} \right)_{{}} + {\text{u}}_{{0{\text{j}}}} \\ & {\text{B}}_{{ 1 {\text{j}}}} = {\text{G}}_{ 10} + {\text{G}}_{ 1 3} ({\text{high vs}}.{\text{ low protection}})_{{}} + {\text{G}}_{ 1 4} \left( {{\text{cohort 2 vs}}.{ 1}} \right)_{{}} \\ & \quad \quad \quad + {\text{G}}_{ 1 5} \left( {{\text{cohort 3 vs}}.{ 1}} \right)_{{}} + {\text{u}}_{{ 1 {\text{j}}}} \\ & {\text{B}}_{{ 2 {\text{j}}}} = {\text{G}}_{ 20} + {\text{G}}_{ 2 3} ({\text{high vs}}.{\text{ low protection}}) + {\text{G}}_{ 2 4} \left( {{\text{cohort 2 vs}}.{ 1}} \right)_{{}} \\ & \quad \quad \quad + {\text{G}}_{ 2 5} \left( {{\text{cohort 3 vs}}.{ 1}} \right)_{{}} + {\text{u}}_{{ 2 {\text{j}}}} \\ & {\text{B}}_{{ 3 {\text{j}}}} = {\text{G}}_{ 30} + {\text{G}}_{ 3 3} ({\text{high vs}}.{\text{ low protection}})_{{}} + {\text{G}}_{ 3 4} \left( {{\text{cohort 2 vs}}.{ 1}} \right)_{{}} \\ & \quad \quad \quad + {\text{G}}_{ 3 5} \left( {{\text{cohort 3 vs}}.{ 1}} \right)_{{}} + {\text{u}}_{{ 3 {\text{j}}}} . \\ \end{aligned} $$

ET Study

Table 4 displays descriptive statistics of the scales by study and by wave. This descriptive data additionally provides evidence for limited attrition among participants in the intervention across the four waves of assessment and the 1-year time period of each program’s direct work with youth. The subdiagonal of Table 5 displays correlations among the variables; with the exception of Reasons for Life with Community Characteristics, all were low to moderate.

Table 4 Mean scale and subscale scores at Waves 1–4 measurement points
Full size table
Table 5 Elluam Tungiinun (N = 54) and Yupiucimta Asvairtuumallerkaa (N = 51) study correlations among measures at Wave 1
Full size table

Results of the HLM analysis are presented in Table 6. Due to the complexity of the model described in the data analysis section, outcomes for the intermediate and ultimate variables are reported only for (1) dose, expressed as number of intervention sessions in which each individual youth participated, and (2) dose × protection, which examined whether the effects of dose changed with the pre-existing protection level. We report slope, standard error, t statistic, lower and upper bounds of the 95 % CI, p value, and Cohen’s d as a measure of effect size. Given this study is a small sample feasibility trial that explores if the intervention produces measurable effects (not intervention effectiveness), we focus our reporting here on the effect sizes produced, while also reporting exact probabilities for each effect and their accompanying confidence intervals.

Table 6 Elluam Tungiinun study summary of mixed model results (N = 54)
Full size table

Intervention Effects on Intermediate Variables

To test our first hypothesis that the intermediate variables of Individual, Family, and Community Characteristics, and Peer Influences would increase as a result of the intervention, we explored the effect of dose, defined as number of prevention activities attended, on growth in these protective factors. Controlling for the effects of pre-existing protection, the amount of time the individual participated in the intervention, and cohort membership, higher intervention dose was associated with positive slopes and moderate effect sizes resulting in higher levels of these characteristics. Table 6 reports medium-sized slope effects for Individual (d = .46, p = .01), Family (d = .38, p = .03), and Community (d = .45, p = .06) Characteristics, and Peer Influences (d = .3, p = .10) across measurement Waves 1–4. These effects did not depend upon the pre-existing protection level, as can be inferred by the low dose × protection effect sizes and their accompanying high p levels. The 95 % CIs of the dose slope effects for Individual, Family, and Community Characteristics did not include zero. The confidence limits for the dose slope estimates for Peer Influences (95 % CI −.0045 to .0755) suggest marginally significant effects of intervention dose on this variable. Thus, we found partial support for the first hypothesis, with clear measureable intervention effects produced for Individual, Family, and Community Characteristics, but not Peer Influences.

Intervention Effects on Ultimate Variables

Our second hypothesis was that the ultimate variables of Reflective Processes and Reasons for Life would increase with higher intervention dosage. As Table 6 shows, we found effects similar in size to the effects on the intermediate variables. Increasing intervention dose was associated with positive slopes of moderate size on Reflective Processes (d = .35, p = .05) and Reasons for Life (d = .32, p = .08). The 95 % CIs for the dose effects on Reflective Processes did not include zero, but the interval for Reasons for Life did (95 % CI −.0009 to .0357). These effects did not depend upon the pre-existing protection level, as demonstrated by the low dose × protection effect sizes and their accompanying high p levels. In summary, these findings partially supported our second hypothesis, with measureable intervention effects produced for Reflective Processes, and trending effects for Reasons for Life.

YA Study

The lower panel of Table 4 displays the descriptive statistics of the scales analyzed in this study by wave, which also evidences limited attrition among participants in the intervention. As can be seen in the superdiagonal of Table 5, correlations among the variables were low to moderate. Results of the HLM analysis are presented in Table 7 for (1) dose, and (2) dose × pre-existing protection, and include linear slope, standard error, t statistic, lower and upper bounds of the 95 % CI, p value, and Cohen’s d (Cohen 1988) as a measure of effect size.

Table 7 Yupiucimta Asvairtuumallerkaa study summary of mixed model results (N = 52)
Full size table

Intervention Effects on Intermediate Variables

We explored the effect of dose on growth in protective factors to test our first hypothesis that levels of the intermediate variables would increase with increasing intervention dose. Controlling for the effects of pre-existing protection and the amount of time the individual participated in the intervention, higher intervention dose was associated with positive slopes for Individual Characteristics that were moderate in magnitude. As Table 7 shows, the linear slope effect size for Individual Characteristics across four waves of measurement was moderate (d = .30, p = .07). In contrast, for Family and Community Characteristics, and Peer Influences, dose effects were small. There was a small dose × protection effect for Individual Characteristics (d = −.27, p = .10) and for Peer Influences, (d = .34, p = .04). This medium size interaction effect suggests that those with higher levels of pre-existing protection appeared to benefit more from the intervention in terms of Peer Influences discouraging alcohol use. In summary, we found mixed support for this hypothesis in the YA study, with a moderate effect for dose on Individual Characteristics only, and negligible effects of dose on the other variables.

Intervention Effects on Ultimate Variables

Our second hypothesis was that there would be significant increases in the ultimate variables of Reflective Processes and Reasons for Life in response to the intervention. Table 7 reports that Reflective Processes and Reasons for Life increased with dose, but the effect sizes were small and not significant (Reflective Processes: d = .24, p = .14; Reasons for Life: d = .25, p = .14). However, even though these dose effects were non-significant, the asymmetry of the confidence limits about zero suggests possible practical significance. In addition, there was a moderate dose × protection effect for Reasons for Life (d = −.34, p = .04). Intervention dose was associated with higher scores on Reasons for Life among those with fewer pre-existing protective factor resources. No significant effects on Reflective Processes were observed.

Discussion

The goals of the ET and YA studies were to determine if (1) prevention research with this type of intensive community intervention was feasible in rural AN communities, and (2) the effects of intervention were measureable and sufficiently promising to warrant a full prevention efficacy trial. Two secondary aims included (3) development of measures that were maximally sensitive to change, and (4) cross-community comparison of the implementation process and its outcomes.

Feasibility

Several lines of evidence provided support that the intervention was implementable within rural Yup’ik Alaska Native settings. There was little attrition among youth, as evidenced through continued participation across the dose counts at the four waves of assessments to evaluate impact. In addition, later in the intervention, in the ET community, two new groups of youth who heard about the program and its activities enrolled, further suggesting that program activities offered something of sufficient value to warrant continued involvement. Additionally, evaluation of five selected modules in one of the communities provided evidence for adherence to the intervention model in implementation, acceptable quality in delivery of key processes and content, delivery of protective factors exposure, and reach through adequate and representative participation of intended youth.

This evidence in selected modules of adherence, quality, protective factors delivery, and reach, along with completion of the intervention program in two communities, together indicate this approach to intervention implementation appears feasible in the rural AN community context. The approach involves community direction of the program development wherein university-based interventionists co-facilitate an adaptive implementation process with local staff and community advisors. This flexible, locally controlled implementation approach is matched to this rural, culturally distinct ecological context. These settings are characterized by important within group local cultural differences, and flexibility in implementation has important implications for facilitating local ownership of intervention.

The perspective of Hawe et al. (2004) on standardization in controlled designs provides a means to conceptualize this as an intervention through underlying function rather than form (Henry et al. 2012). In this community intervention, the key functions that intervention activities deliver across settings is the replicable element instead of repetition of identical intervention components across communities. Thus, the form of the intervention components can be tailored to local context while the underlying function remains the same. As one example, coastal Yup’ik may hunt seal, whereas upriver Yup’ik may hunt moose in prevention modules with the same underlying function. Similarly, two communities may have different local cultural protocols for conducting the qasgiq module. This approach allows contextualizing of each module to these important local differences between communities, while also maximizing community ownership by acknowledging the competence and expertise of the community. We believe this approach holds promise in other Indigenous and immigrant settings undergoing colonial relationships, external controls over life and society, and/or involuntary acculturation.

Measurable Effects

Regarding measureable impact, collectively, the studies identified several instances of medium slope effect sizes associated with intervention dosage (number of intervention modules attended) on intermediate and ultimate outcome variables. In the ET study, higher intervention dose was associated with positive slopes and moderate effect sizes for the intermediate variables of Individual, Family, and Community Characteristics, and Peer Influences. In the YA study, there was a moderate linear slope effect size for Individual Characteristics only, and very modest effects on Peer Influences moderated by pre-existing protective factor levels. In ET, increasing intervention dose was associated with positive slopes of moderate size on the ultimate outcome variables of Reflective Processes and Reasons for Life. YA results suggest youth with fewer pre-existing protective factors may have benefited more on Reasons for Life, but no effects were noted on Reflective Processes. These effects persisted even when time in the intervention, pre-existing levels of protection, and cohort of program entry were entered as potentially confounding variables.

The effects identified in response to this cultural intervention were linked to an indigenous, culture-specific theoretical model of protection identified through our previous collaborative research. The model specified mechanisms and modes of action guiding intervention. These findings were only possible because of the careful, long-term theory driven measurement development effort that followed a program of qualitative research designed to generate an indigenous multilevel model of protection (Allen et al. 2006, 2014b; Mohatt et al. 2004). We cannot overemphasize the crucial importance of attention to cultural variables in this work and the extensive psychometric testing that was completed as part of the development of outcome measures. Our goal was to create measures that were understandable and fit local dialect and usages, grounded to local context and ecological setting, and culturally appropriate. However, none of the many standard measures that we pilot tested functioned adequately with this population and efforts to adapt existing measures (e.g., Family Environment Scale; Moos and Moos 1994), resulted in such significant departures in item content and composition that we essentially created new measures (Allen et al. 2014b). Extensive repeated iterations of focus group and cultural expert consultative work with pilot testing were required to develop new measures of constructs. Continued attention to the multitude of unresolved tensions inherent in this work is needed (Gonzalez and Trickett 2014),

Measures of Change

We also highlight here an important distinction in measurement, between measures for construct elaboration and theoretical modeling, and measures of change for outcomes assessment. We found that measures developed for the purpose of testing the theoretical model (Allen et al. 2014b) were not optimally sensitive to change in assessing intervention impact. IRT approaches to scale construction, and in particular the Graded Response Model (Samejima 1997), hold promise in crafting brief, locally acceptable, yet highly sensitive indices of change for outcome studies with culturally distinct groups. This also resulted in shorter measures with less participant burden.

Implications of Contrasting Findings across Two Communities

The differential impact of the intervention dose response effect across the two studies highlights important implications regarding the intensity of youth and community involvement required for effective intervention. Because of the limitations of the developmental grant mechanism that funded YA, the YA intervention was funded at approximately half the level of the ET study. The most evident impact of this funding discrepancy is the smaller number of intervention modules and associated protective factors delivered in YA. The contrasting results can be understood on one level as the direct impact of lower levels of dose in the YA community. With half the funding, the YA project had less than half the staff, the community planning group was smaller, there was less involvement of Elders, and overall, there was less community involvement. Fewer community members knew about the program in a detailed way. When the sole full-time staff person on the YA project became injured and required 6 months of medical leave, there were no additional staff with the accumulated knowledge base to carry the project onward. This led to a significant interruption of the project just as it was gaining momentum.

Other important factors affecting dosage and effects included the priorities and goals of each community. Despite the high rates of suicide in the region, the YA community had not experienced a suicide in over 30 years during the time of this intervention. However, alcohol use among youth was a rising community concern, and the intervention this community created prioritized development of the same protective factors but focused primarily on alcohol, not suicide.

In contrast, the ET community had been devastated by successive waves of youth suicide. Elders noted that they had been waiting for this project to come. Urgency only increased when, at the start of the intervention, another cluster of young adult suicides rocked the community. In response to this devastating prehistory, the community mobilized and then developed plans for 36 prevention modules, and succeeded in delivery of 26 modules through a total of 52 actual sessions. Community planning group members engaged in over 90 planning meetings, requiring up to 4 h of time each, to create these prevention activities. These planning meetings were attended by large numbers of community members spanning a diversity of generational and other social locations, including Elders, tribal and community leadership, parents, and youth. This description highlights the intensity of community leadership and commitment required for effective implementation of this intervention, and emphasizes the need for sufficient resources to support the time required for these levels of community input.

Methods to Address Challenges in Research with Rural and Small Population Culturally Distinct Groups

These results underscore how small sample sizes typical to research with rural and culturally distinct groups can be impacted by the limitations of current statistical methodologies, providing important insights into a number of methodological challenges. Though we found substantively meaningful medium slope effect sizes for intervention dose, our analyses were underpowered. In research with rural and culturally distinct groups, small samples and enormous logistical challenges are the norm. Many recent advances in multivariate statistics require large samples, and state of the art methods often call for rigid design parameters rarely attainable within the logistical constraints of community research in the arctic and in other rural, remote locations.

Research in rural contexts challenges researchers on many fronts. In rural Alaska, individuals come and go with seasonal patterns of subsistence hunting and gathering. Community intervention itself must be seasonally organized, and is further organized through patterns of ritual breaks, proscribed behavior, respect, timing, funerals, community events, and community tragedies. Storms, cold temperatures, inclement weather, and equipment breakdown interact with geographic isolation to close air travel and access to a community. Any of these factors can stop progress for weeks at a time. Assessments and program implementation can drag past deadline much more frequently than in more controlled environments. All of this requires extraordinary effort and commitment on the part of a research team, and flexibility in the research methods.

For the purpose of analyses, we were therefore required to find a methodology that could tease out effects of a complex intervention in an environment with significant complexity of its own, defying the kind of exacting timelines typical in many design frameworks. Multi-level models provide one partial solution in that they treat individuals as a random factor. This allows the analysis to take into account the amount of information supplied by each participant, as well as participants’ pooled information, while also making complete use of participant data when incomplete or missing in part. Similarly, multi-level models are also capable of treating the dimension of individual time flexibly (cf., Singer and Willett 2003, p. 181ff). Because of this, we were able to center time at each individual participant’s entry into the intervention, while also, in the case of the ET study, modeling for the possible effects of entry into the program though one of three different cohort groups. Finally, given each youth entered into the program with differing pre-existing levels of protection, and we suspected this might differentially effect both future protective factor growth and ability to benefit from the intervention, we also modeled pre-existing levels of protection as a covariate at the individual level. This allowed the intervention process to naturally unfold in each community, while at the same time allowing us to model variation in how it unfolded. In the end, we believe this offered a more precise estimation of effects.

Henry et al. (2012) demonstrated use of an innovative analytic framework for the study of function as the unit of analysis in fidelity to this intervention. Most intervention research is based on the Neyman–Rubin causal model, which assumes every individual receives exactly the same intervention. This Stable Unit Treatment Value Assumption (SUTVA; Rubin 1974) is violated in multisite CBPR studies when each community can design local intervention. Henry et al. used latent class analysis to help interpret the complex patterns of protective factors constructed through this adaptive intervention process. Results provided evidence that the specific protective factors emphasized in each community’s intervention reflected local community values, history, and priorities, and latent classes identified subgroups of youth whose selection of activities led to distinct configurations of protective factor exposure. This approach permits unified analysis of data from a multisite CBPR intervention study as in this intervention.

There is a need for continued refinement of statistical methodologies to fit the requirements of ecological settings, rather than requiring settings to attempt to fit the needs of methodology. The solutions we adopted in modeling intervention effects provide a metaphor for the intervention research process required in these types of settings; it requires the research team to adopt nimble and flexible methods to respond to community realities and direction. All too often, despite the pressing need for research in rural and culturally distinct settings, and for community based research in health disparities settings, conventional design requirements are not amenable to the realities of setting and community priorities, limiting the feasibility of conducting this important research. A constructivist philosophy of science, more consistent with a CBPR worldview, emphasizes the need for innovation and flexibility in methods and outcomes that incorporate careful naturalistic description of the local conditions under which the findings are actually generated.

In summary, the level of community acceptance and involvement, the level of participation of youth, the findings regarding adherence, quality, protective factors delivery, and reach, and the measureable effects observed together suggest this intervention warrants further testing in a prevention trial. Due to local concerns expressed regarding the potential impracticality and cultural inappropriateness of randomized controlled trial (RCT) designs, such a trial could make use of one of the promising quasi-experimental alternatives to RCTs, such as dynamic wait-listed designs, as used in other youth suicide prevention trials (Brown et al. 2006).

These two studies are analogous to a successful Phase II Exploratory trial, effectively positioning this intervention for a Phase III trial, according to the Medical Research Council (2000) framework for development and evaluation of trials for complex interventions. However, our ongoing efforts to establish this intervention as an evidence based practice for AN people presents a new dilemma: will this decade and a half CBPR process now simply become an instrument in the service of translational research? Or can the research methods used to establish evidence base instead be developed or adapted to benefit from and be consistent with local knowledge and cultural values (Trickett 2011)? The later option would acknowledge the competence and expertise of the community in their development of a complex intervention to address a locally defined health priority.