Background

There are promising findings for the efficacy of eating disorder (ED) prevention programs (e.g., Shaw et al. 2009; Stice and Shaw 2004; Stice et al. 2007). This includes both Internet-based approaches (Taylor et al. 2006) and face-to-face interventions (Stice et al. 2012). Yet, although effective interventions are available, the implementation of these interventions in the real-world is challenging. In the majority of efficacy trials, recruitment is facilitated by offering financial compensation or course credits. Without such incentives, willingness to participate in an ED prevention program is usually very low, and motivating the target population to participate is challenging (Atkinson and Wade 2013).

As a result, the majority of prevention programs never find their way into routine care. This challenge is not specific for ED but valid for prevention programs across all mental health conditions (e.g., Frantz et al. 2015). Research often focuses on the development and evaluation of programs. However, this is not sufficient, and it must be ensured that as many individuals as possible may have access to the intervention in order to maximize its public health impact (Muñoz et al. 2012). From a public health point of view, the impact of an intervention equally depends on its efficacy and its reach (Glasgow et al. 2003). The relevance of effective dissemination strategies to maximize an intervention’s reach has been increasingly acknowledged in recent years (Glasgow et al. 2012). However, systematic research on the dissemination of prevention programs is still sparse.

Internet-based prevention programs are associated with a set of properties, which seem to offer promising potentials to mental health services. Among others, accessibility, scalability, and sustainability permit service provision to large populations and to individuals with limited access to regular care at relatively low cost (O’Connell et al. 2009). Potentially, the reach of e-health interventions is large, but similar to face-to-face programs, there is a lack of studies investigating their actual reach (Bennett and Glasgow 2009).

Until now, the majority of studies that explore the effects/success and costs of different dissemination strategies compared different online recruitment strategies and conventional advertisement with print materials. Most common online advertisement strategies were Google advertisement (i.e., a link to the page is shown at the search results in a separate section), banner advertisement (i.e., an advertisement for the page is shown in banners of other pages), and Facebook advertisement (i.e., a Facebook page for the program is created and advertised on Facebook; optionally, a target population can be defined based on e.g., gender, age group, city, etc.). Usually, there is a cost per click on the advertised link, and prices vary according to the search terms, the target population, and the demand for advertisement with the same search terms or for the same target population (Morgan et al. 2013).

Based on the literature, dissemination costs for Internet-based prevention programs vary substantially from less than $10 to nearly $600 per participant: In a pragmatic trial exploring different online recruitment strategies for an Internet-based depression prevention intervention, Google advertisement was the most effective recruitment strategy, at an average cost of AUD $12 per participant (Morgan et al. 2013). Costs of on average $209.34 per participant were reported for online recruitment strategies for an Internet-based smoking cessation program targeting Latinos in the USA (Graham et al. 2012). Studies comparing online and traditional dissemination strategies report mixed findings. A study that applied both online and offline recruitment strategies for an online smoking cessation program reported costs of $41.35 (online ads) and $56.23 (offline materials) per participant for the most cost-effective strategies (Buller et al. 2012). Contradictory, comparing costs and effects of conventional (e.g., flyers, posters, etc.) and online recruitment strategies also for an online smoking cessation program, Graham et al. (2008) found conventional strategies to be more cost-effective, with $5–$8 per participant. Online advertisement resulted in on average $35 per participant (range $7–$476; Graham et al. 2008).

In another study that explored recruitment costs for an online tobacco cessation program, Google advertisement yielded to costs of $6.70 per participant (Gordon et al. 2006). Costs per participant were higher for all other strategies: $36 for direct mailings to previous study participants, $36 for sending promotion materials to interested organizations and professionals, $92 for media campaigns, $115 for newspaper advertisements, and $597 for direct mailings to purchased lists (Gordon et al. 2006).

Until now, research studying the cost of different recruitment strategies for Internet-based interventions have been conducted in a pragmatic way, which may bias the results as the different strategies are not independent and might have influenced each other. E.g., if individuals remember the program from a flyer and look it up on Google at home, they might use the link from Google advertisement to access the program although they originally learned about it from a flyer. Studies that compare online dissemination strategies usually apply different strategies consecutively and compare the number of registrations/page visits during these time periods. Although these kinds of studies provide valuable data on the costs of online dissemination strategies, these designs are prone to multiple sources of bias. So far, only a limited number of dissemination strategies (predominantly online strategies and conventional strategies such as flyers and posters) have been investigated.

Another strategy to disseminate prevention programs, which has already been applied for ED prevention, is to approach social systems like high schools or colleges/universities (e.g., Becker et al. 2008; Becker et al. 2009). Compared to online advertisement, an advantage of this strategy is that not only students that actively search for information on ED (which is the case for at least some online advertisement strategies) are informed about the intervention. When adolescents and young adults are the target population for prevention, these strategies seem promising as large groups can be reached simultaneously and can be informed about the program. Students interested in participation learn about the program and how to access it. Yet, data on costs and effects for school-based approaches are lacking.

The current study bridges this gap by investigating the cost-effectiveness of five different school-based dissemination strategies for an Internet-based intervention for the prevention and early intervention in eating disorders (ProYouth; Bauer et al. 2013) within a randomized trial.

Methods

Design

Based on a complete list of schools provided by the state of Baden-Wuerttemberg (Germany), 395 secondary high schools were randomly assigned to one out of five dissemination strategies stratified by type of school and the responsible contact person. Contact persons were Bachelor level psychologists with a part-time employment at the research center. One responsible contact person was assigned to each school. The efficacy study was approved by the Ethics Committee of the Medical Faculty of the University of Heidelberg, and separate approval for the dissemination of the program was not needed.

Intervention

ProYouth is an Internet-based program for the prevention and early intervention in ED. It is based on the program Essprit developed in Germany, which was evaluated with respect to its feasibility, acceptability (Bauer et al. 2009; Lindenberg et al. 2011), and efficacy (Lindenberg and Kordy 2015) in previous studies. The program contains various modules of different intensity, e.g., psychoeducational materials, chat counseling, forums, news blogs, and self-monitoring (for a comprehensive description of the intervention see Bauer et al. 2013). In order to use the program, participants have to fill in a screening questionnaire and register with a valid email address. After the screening, they receive an individualized feedback: Only individuals with an elevated risk to develop an ED and those who report slight ED symptoms are explicitly encouraged to register. Individuals that report severe symptoms and/or already are in treatment because of an ED, as well as individuals who do not show an elevated risk receive the message, and that ProYouth might not be appropriate for them. Nevertheless, anyone may register as the program is publically available on the Internet.

Dissemination Strategies

The target population was students of age 14 and older, i.e., at least grade 8. Five school-based dissemination strategies were studied:

Strategy 1 (advertisement materials only) included mailing a box with flyers and information material to the school. In the cover letter, schools were asked to disseminate the materials among the target population (grades 8 and higher). The research team had no contact with the school prior to sending materials. Strategy 2 (phone call plus advertisement materials) included a personal contact with the school (phone call) asking whether they would be willing to disseminate ProYouth materials. If the school agreed, the same materials as in the first strategy were sent with the same cover letter. In strategy 3 (student representatives/peers), student representatives (e.g., members of the student parliament or the school’s student newspaper) were asked to inform their peers about the program, again using printed information materials and flyers. In case their contact information was not available on the school’s homepage, the school was approached and asked for the contact information of a student representative. Strategy 4 (introductory presentation) consisted of a presentation for students at their high school. Schools were contacted via telephone and/or email and—if they were interested—a member of the study team visited the school, gave a 1-h presentation about eating disorders in general (risk and protective factors, ED symptoms, help-seeking and treatment options, etc.), introduced the ProYouth program, and invited the students to access the website. Strategy 5 (introductory presentation plus workshop) included the same presentation as strategy 4. In addition, a workshop in the school’s computer room was conducted in which interested students could directly register to ProYouth. The workshop took place immediately after the introductory presentation. Students decided on a voluntary basis whether or not they wanted to participate in the events. Only students who provided written informed consent were allowed to participate in the workshops in the computer room. In case of minors, parents also had to provide informed consent. Activities in strategies 4 and 5 took place during the regular teaching time. If possible, classes were merged in order to minimize organizational efforts.

Figure 1 gives an overview of the recruitment. Thirty schools were excluded so that the final sample consisted of 365 schools.

Fig. 1
figure 1

Flow of schools. Note. Strategy 1: advertisement materials only; strategy 2: phone call and advertisement materials; strategy 3: student representatives/peers; strategy 4: introductory presentation; strategy 5: introductory presentation plus workshop

Full size image

The distribution of school types in the five strategies and the total number of students in these schools were comparable across strategies (Table 1).

Table 1 Distribution of school types and total number of students in schools
Full size table

Assessment of Cost

Documented costs included printing costs, mailing costs, travel costs for school visits, and staff time. Staff time included the time spent to prepare, pack, and send information materials, to communicate with the schools (phone calls, emails, etc.) and to conduct school visits (incl. travel time).

Assessment of Effect

The effects of the dissemination activities were operationalized by the number of individuals visiting the website of the ProYouth program, the number of completed screenings, and the number of registered users.

The effects were measured in a non-reactive manner. For each dissemination strategy, we used a different web address (URL). All five URLs were redirected to the homepage of ProYouth and recorded in the database. Thus, for visitors, individuals who completed the online screening, and individuals who registered to the program, the respective strategy that attracted them to the program could be identified. All actions on the page were tracked. Page visits of individuals who returned to the page following their registration were subtracted from the total number of page visits to ensure that each individual was counted only once. The software to analyze the page visits was developed by the study team, as available web analytics software did neither comply with the study’s requirements nor with the desired privacy.

The study was conducted immediately after the launch of the ProYouth initiative. During the study period, no other dissemination strategies were applied in order to avoid that the URLs used for the study get spread through other channels (e.g., online communities). Data collection was ended 3 months after the last school was approached.

In addition, due to comprehensive real-time monitoring, it could be verified that within the study period, none of the participants accessed ProYouth via a link on another website or a link provided in an email. Ongoing referrer link analyses and monitoring would have detected if participants used such links to access ProYouth, i.e., such sources of bias could be controlled during the present study.

Statistical Analyses

Total costs and effects were calculated for each of the five dissemination strategies. Confidence intervals for cost estimations were obtained by bootstrapping to account for the non-normal distribution of cost data (5.000 repetitions). Analyses were conducted using SPSS21.

Results

The more intense the advertisement strategy, the less schools agreed to participate. In total, 166 schools (45.5 %) did not agree to support the dissemination of ProYouth. In strategy 5 (introductory presentation and workshop), only 5 schools (6.9 %) agreed to participate; in strategy 4 (introductory presentation), 17 schools (23.0 %) participated; in strategy 3 (student representatives/peers), 34 schools (50.0 %) participated; in strategy 2 (phone call plus advertisement materials), 63 schools (88.7 %) participated. As in strategy 1 (advertisement materials only), schools were sent the materials without prior contact, and they did not have the possibility to refuse participation, so all 80 schools (100 %) were considered as participating. Main reasons for not participating were no interest (38.6 %, n = 64) and no reply after three contacts (37.3 %, n = 62). In addition, 10.2 % (n = 17) did not want to invest the time and effort, and 13.9 % (n = 23) mentioned other reasons.Footnote 1

Costs

Table 2 displays the total costs associated with the five dissemination strategies. In sum, cost for strategy 4 (presentation) was the highest, and cost for strategy 3 (student representative) was the lowest. Most expenses accrued for staff time and for strategies 1 and 2 materials were the largest matter of expense.

Table 2 Dissemination costs for the 5 strategies in € (95 % confidence interval)
Full size table

Effect

More intense dissemination strategies resulted in higher numbers of page visits, screenings, and registrations (Table 3). In strategies 1, 2, and 3, less than 50 individuals visited the page, less than half of them completed the screening, and only 2 in strategies 1 and 3 and 8 in strategy 2 registered to the intervention. Most visitors could be attracted to the page by strategies 4 and 5.

Table 3 Effects of the 5 strategies
Full size table

The percentage of visitors that completed the screening and registered to the intervention differed between the five strategies (χ 2(8) = 163.09; p < 0.001). In strategy 5, out of the 941 individuals, who visited the page, 85.7 % did the screening and 41.2 % registered to ProYouth. In strategy 4, 65.5 % did the screening and 24.0 % registered. Lowest registration rates were found in strategies 1 (4.1 %) and 3 (7.7 %) (Table 3).

Table 4 displays the cost-effect ratios. The average cost per visit (2.83€), screening (3.30€), and registration (6.86€) were lowest in strategy 5.

Table 4 Cost-effect ratios for the 5 strategies
Full size table

Discussion

To our knowledge, this is the first study that systematically investigated the effects and cost-effectiveness of different school-based strategies aimed at the dissemination of a program for the prevention and early intervention of a mental illness. Overall, the dissemination of the program via schools proved to be challenging. Costs for one registered participant ranged from 6.86€ (strategy 5) to 431.10€ (strategy 1) and costs for one screening assessment varied between 3.30€ (strategy 5) and 75.66€ (strategy 3). Although fewer schools agreed to participate in the more intense dissemination strategies, more time-consuming and thus more expensive dissemination strategies proved to be more cost-effective. Sending advertisement materials only (strategy 1) resulted in very few registrations, contacting the schools prior to sending such materials (strategy 2) resulted in overall lower costs and similar effects. Thus, a personal contact seems to be a useful add-on to sending information materials mainly because schools, which are not interested in participation and not willing to disseminate the materials, can be excluded and do not receive the materials. Although strategy 3 (student representatives/peers) was the cheapest strategy, it was even less effective than strategies 1 and 2. The more intense strategies 4 and 5 with personal introductory presentations at the schools showed the highest costs and were most successful (i.e., had the largest effects). Yet, compared to strategy 4, strategy 5 was less costly and more effective with respect to visits, screenings, and registrations. Strategy 5 had the best cost-effect ratios, i.e., presentations and workshops in the computer room proved to be the most promising school-based dissemination strategy. Incremental cost-effect ratios were not calculated, because the strategies investigated in this study are not necessarily competing, exclusive strategies. In practice, they might also complement each other.

There is limited research from other fields to which the findings of the present study may be compared. The cost-effect ratios of the dissemination strategies of the present study are in the range of those reported for online dissemination (e.g., Gordon et al. 2006; Graham et al. 2008, 2012; Morgan et al. 2013). Available research shows that online dissemination strategies are not as effective and cheap as one would intuitively expect. In most of the published studies, they were even more expensive than the strategies identified as successful in the present study. However, previously available evidence stems mostly from research on smoking cessation and the comparison of dissemination costs for smoking cessation programs and ED prevention programs is of course questionable. Thus, data on costs for online dissemination of ProYouth would clearly be an interesting complement for the present study. Additional research is also needed because for the most common online strategies (Google ads and Facebook ads) costs depend on the “market.” The higher the demand for online advertisement related to specific key words, the more expensive advertisement becomes. Thus, prices are not stable and vary over time and between countries (e.g., Gross et al. 2014).

Research on smoking cessation programs has also revealed that different recruitment strategies may attract different target populations and influence compliance with the intervention (Smit et al. 2012; Stanczyk et al. 2014). Future studies will need to investigate these effects for online, face-to-face, and school-based dissemination strategies for mental health interventions such as ProYouth.

Compared to conventional face-to-face interventions, successful dissemination strategies are even more crucial for the cost-effectiveness of Internet-based interventions. Usually, cost-effectiveness analyses comparing two conventional interventions do not take fixed costs into account, i.e., costs for the development of the interventions and their maintenance are not included in the analyses. For Internet-based interventions these costs are high and independent of the number of participants. Their consideration results in relatively high break-even points (e.g., Ruby et al. 2013), i.e., developing and providing Internet-based interventions under routine care conditions is not feasible without effective dissemination strategies because otherwise the costs per participant would be extremely high.

In the present study, dissemination costs per participant for strategies 1, 2, 3, and 4 were higher than the average maintenance costs of ~15€ per year and participant to actually provide the program and to deliver the intervention (Minarik et al. 2013). This highlights the importance of successful dissemination strategies if large populations should be addressed in a cost-effective manner. In light of the limited compliance and high drop-out rates in both Internet-based interventions (Melville et al. 2010) and preventive interventions (O’Connell et al. 2009), cost-effective dissemination strategies become even more important in order to involve large populations in such programs. Therefore, appropriate strategies and a specific budget need to be incorporated in every dissemination plan for Internet-based programs.

So far, research on dissemination strategies for Internet-based interventions is sparse. In part, this can be explained by the specific challenges in the assessment of dissemination effects for publically available interventions. Online accessibility requires specific methods to clearly identify which dissemination strategy attracted a specific user. Online visibility of an Internet-based intervention (e.g., links to the program on other websites or in forums) increases over time and can result in direct or time-delayed accesses to the intervention which may bias the results of dissemination studies. In order to obtain precise estimates of the effects, it is necessary to assess which strategy attracted a specific participant and whether a visit to the page is the first visit of that user or not. None of the currently available web analytic programs fulfilled the study requirements. The software specifically developed for this project could reliably identify the strategy that attracted individuals to the ProYouth program. Thus, potential biases could be minimized which clearly represents a strength of this study.

However, even though the effects were assessed reliably, the study has some limitations. The effects of the dissemination strategies were only assessed in total and not on a school level because the organizational effort would have been too high. As a result, no confidence bounds for the effects but only for the costs could be calculated. Travel cost for strategies 4 and 5 were influenced by the location of the research center and cannot be generalized to other areas. For strategies 1 to 3, it remains unclear how and to which extent schools and student representatives actually engaged in distributing materials or promoting ProYouth. Involving student representatives to support the initiative (strategy 3) might require previous training of the students in order to improve their understanding of the program and strengthen their motivation (Becker et al. 2009). No training was provided in the present study, and student representatives were only asked to distribute information materials to their peers. Furthermore, an obstacle for several schools randomized to strategy 3 was the fact that schools were reluctant to provide contact information of student representatives (e.g., due to privacy concerns). In these cases, student representatives could only be contacted via the school’s secretary, which made communication slower and more complicated and impeded the dissemination process. In the future, it might therefore be more promising to contact umbrella associations of student representatives (e.g., regional associations) instead of schools to explore the potential of this strategy.

Despite these shortcomings, the present study significantly adds to the available literature on the dissemination of mental health interventions by investigating five school-based approaches within a large-scale randomized trial. The findings show that more intense, expensive dissemination strategies involving personal contact were more cost-effective. The results also put into perspective the common assumption that Internet-based interventions per se have a large reach at low cost. Although geographically, the reach of these interventions is without any doubt large, the reach from a public health point of view is not necessarily large. Without successful and cost-effective dissemination strategies, these interventions cannot tap their full potential. The present findings indicate that the integration of an Internet-based program and face-to-face school-based activities is a promising and cost-effective strategy to facilitate the dissemination of ED prevention efforts. In addition, although initially, only a minority of schools agreed to participate in the more intense strategies, a number of schools are now visited on a regular basis. These longer-term effects of the more intense advertisement strategies resulted in a solid network of schools, which regularly host workshops. In this sense, the more intense dissemination strategies also turned out to be promising means for capacity building and sustainability of the ProYouth initiative.

Clearly, further research is needed to improve our understanding of strategies that allow us to successfully disseminate programs directed towards the prevention and early intervention in mental illness. The development and evaluation of dissemination strategies is crucial for the broad and sustained implementation of such programs into routine care. Studies like the one presented in this paper may make an important contribution to reducing the research-practice gap and increasing the public health impact of preventive interventions.