Introduction

Men who have sex with men (MSM) are disproportionately impacted by HIV/AIDS in the U.S. In 2014, MSM accounted for an estimated 67% of all diagnosed HIV infections despite only representing about 2% of the population (CDC, 2015a, 2016). The number of new infections among MSM increased 9% between 2010 and 2014 (CDC, 2015c). This is the only population subgroup that experienced an increase in number of HIV infections. Among MSM, there are also disparities by age and race and ethnicity. Young MSM (YMSM) ages 25–34 represent the greatest number of new infections, and YMSM ages 13–24 the second highest (CDC, 2015c). The number of diagnoses among White YMSM decreased by 6% and recently stabilized at a high rate for Black YMSM, but diagnoses of HIV increased 13% among Latinos between 2010 and 2014 (CDC, 2015b).

Despite the disproportionate burden of HIV among YMSM and calls for greater understanding of disparities in HIV infection, there are few proven HIV prevention programs that are tailored to this population (CDC, 2017; Mustanski, Newcomb, Du Bois, Garcia, & Grov, 2011). Moreover, researchers have identified increasing challenges to recruiting populations that experience the highest prevalence of HIV—racial and ethnic minority MSM—into observational and intervention studies (Jenkins, 2012; Sullivan et al., 2011).

Although newer recruitment methods (e.g., Internet approaches) contribute to the recruitment toolbox along with more established methods (e.g., venue-based approaches), they have not consistently mitigated the barriers to recruiting diverse samples of MSM into research studies. As a result of these challenges, racial and ethnic minority populations continue to be underrepresented in HIV research despite being most affected by HIV (Halkitis, Wolitski, & Millett, 2013). To illustrate, research has demonstrated high willingness among MSM to participate in HIV vaccine trials (76–79%) with no differences across racial and ethnic groups; however, underrepresentation of minorities persists in HIV research trials. This may be a result of venue and demographic characteristics of the catchment and recruitment area (Dhalla & Poole, 2014).

Venue-based and time–space sampling have been utilized as recruitment approaches for over two decades of HIV behavioral risk surveillance studies with MSM (Lemp et al., 1994; MacKellar, Valleroy, Karon, Lemp, & Janssen, 1996; Stueve, O’Donnell, Duran, San Doval, & Blome, 2001), including the National HIV Behavioral Surveillance System (NHBS) (CDC, 2017). Since 2000, HIV researchers also have assessed the feasibility of the Internet as a recruitment tool (Bowen, Williams, & Horvath, 2004; Bull, Lloyd, Rietmeijer, & McFarlane, 2004; Fernandez et al., 2004; Mustanski, 2001; Rhodes, DiClemente, Cecil, Hergenrather, & Yee, 2002; Ross, Tikkanen, & Mansson, 2000), and research has flourished with the Internet as a setting for surveillance, basic research, and behavioral intervention studies (Grov et al., 2016b). The advantages of Internet-based recruitment identified by early adopters include high reach, low cost, and low time involvement (Bull et al., 2004; Fernandez et al., 2004; Mustanski, 2001; Pequegnat et al., 2007; Raymond et al., 2010; Rhodes et al., 2002). However, disadvantages include a lack of a sampling frame from which to establish external validity (Hernandez-Romieu et al., 2014) (for an example of an exception see Grov, Cain, Rendina, Ventuneac, & Parsons, 2016a), as well as practical considerations of ensuring data quality and integrity due to risks associated with distance data capture (e.g., duplicate responders and computer robots) (Sullivan, Grey, & Simon Rosser, 2013). Further, debate remains about whether or not samples recruited on the Internet are representative of the greater target population (Raymond et al., 2010), and if Internet-based recruitment can reach populations experiencing the greatest burden of HIV infection given research showing that MSM of color have been systematically underrepresented in online HIV prevention studies (Sullivan et al., 2011). Researchers have suggested that enrollment quotas based on race and ethnicity may improve representation in research, but may also increase selection bias in the research (Sullivan et al., 2011).

In consideration of recruitment venues for HIV research, specific settings provide unique cultural contexts and social norms for venue participants. As described by Grov (2012), social norms in bars, online settings, and bathhouses differentially impact how MSM communicate about and engage in risky or protective sexual behaviors. When seeking to understand venue-associated HIV risk, it is not uncommon for researchers to focus on a single venue as a means to address venue-specific HIV prevention needs (Grov, 2012). In contrast, when aiming to enhance generalizability of research findings, participant recruitment and sampling may include multiple approaches across multiple venues. There is a growing body of HIV research with MSM that evaluates similarities and differences among participants recruited from diverse venues, in particular, online versus offline settings. Identifying these differences is important in informing HIV prevention efforts and understanding the generalizability of study findings (Grov, 2012; Parsons, Vial, Starks, & Golub, 2013).

There may be demographic differences among individuals recruited from different venues, which of course will be highly dependent on the specific venues used for recruitment. In terms of overall trends for age, research is mixed, but studies tend to suggest that individuals recruited online are older than those recruited offline (Grov, 2012; Hernandez-Romieu et al., 2014; Parsons et al., 2013). However, two studies found that Internet-based recruitment yielded younger samples compared to time-location sampling in two NHBS studies (Raymond et al., 2010; Sanchez, Smith, Denson, Dinenno, & Lansky, 2012). With regard to race and ethnicity, the research is clear that White individuals are more likely to be recruited from the Internet across diverse websites (e.g., Adam4Adam.com, Craigslist.org, Facebook.com, Gay.com), than Black and Latino individuals (Parsons et al., 2013; Raymond et al., 2010; Sanchez et al., 2012; Sullivan et al., 2011; Vial, Starks, & Parsons, 2014).

As there are demographic differences among individuals recruited from different venues, individuals may also differ in their behavioral risk profile according to recruitment venue. Indeed, there is ever-mounting evidence that research participants recruited from the Internet report higher HIV sexual risk-taking behavior than participants recruited from offline, field-based venues (Grov, 2012; Parsons et al., 2013; Raymond et al., 2010; Sanchez et al., 2012; Vial et al., 2014). A recent meta-analysis also found that gay and bisexual identified men recruited online reported more sexual risk behaviors than those recruited from other venues (Yang, Zhang, Dong, Jin, & Han, 2014). In an effort to understand these findings, it has been speculated that Internet-based studies may have oversampled individuals from higher risk websites (e.g., Craigslist.com, Manhunt.com) and apps (e.g., Grindr) (Hernandez-Romieu et al., 2014). However, rather than viewing these findings as study limitations, it has been suggested that Internet-based recruitment methods actually may sample individuals not reached through traditional venue-based approaches (Sanchez et al., 2012). In contrast to this body of research, a few studies have found no differences in risk (Chiasson et al., 2007; Hernandez-Romieu et al., 2014) and have recommended that HIV surveillance and research studies may utilize online recruitment, specifically via Facebook, with limited evidence of selection bias (Hernandez-Romieu et al., 2014).

Finally, individuals may also differ in their attainment of research milestones, from screening eligible to long-term retention, as a function of their recruitment venue. Research is limited in this area, but tends to suggest that MSM recruited offline are more likely to meet preliminary eligibility criteria, provide contact information, and be reached for full eligibility screening compared to individuals recruited across a range of diverse online venues (Parsons et al., 2013). Other research has also shown that individuals recruited online are least likely to participate in research compared to those recruited through newspaper ads, friends, or health professionals (Hatfield et al., 2010). For White MSM in this study, those recruited by online ads had lower than expected attendance rate; for Latino MSM, those recruited by online ads had the smallest attendance rate. Of note, African-American MSM did not show differential attendance rates by recruitment method (Hatfield et al., 2010). With regard to retention in HIV surveillance and research studies, researchers have found no differences in long-term retention between groups recruited online and offline (Hernandez-Romieu et al., 2014). Research is limited for attainment of research milestones as a function of recruitment venue for Internet-based HIV interventions, but it has been suggested that in-person recruitment may help establish study credibility for online research leading to increased likelihood of participation (Du Bois, Johnson, & Mustanski, 2012; Greene, Madkins, Andrews, Dispenza, & Mustanski, 2016; Pequegnat et al., 2007).

In this article, we use data from the randomized control trial (RCT), Keep It Up! 2.0, to examine the impact of recruitment venue on participant demographics, behavioral HIV risk profile, and enrollment. We hypothesized that dropout during enrollment would be lowest among participants recruited from community-based organizations (CBOs) given past research linking face-to-face recruitment with greater commitment to studies and that these participants would also be at lower risk for HIV given that they were recruited from a venue where HIV prevention is encouraged. This study is unique in that it uses a variety of recruitment approaches to enroll a diverse sample of YMSM into an online HIV prevention intervention. Few studies have examined the impact of recruitment venue on participant characteristics and likelihood to enroll in an online RCT for YMSM. Such research is necessary to evaluate the impact that utilizing a variety of recruitment venues may have on generalizability of study findings and ability to recruit participants at greatest risk for infection.

Method

Participants

Data were taken from Keep It Up! 2.0 (KIU! 2.0), a multisite two-arm RCT in Atlanta, Chicago, and New York assessing the efficacy of an online HIV prevention intervention for YMSM. KIU! 2.0 was based on principles of e-learning and informed by the Information-Motivation-Behavior Skills model of HIV risk behavior change (Clark & Mayer, 2003; Fisher & Fisher, 2002; Fisher, Fisher, Williams, & Malloy, 1994). The intervention was tailored to YMSM and used a holistic approach that incorporated HIV information, role modeling of skills to build self-efficacy, and contextualizing the application of HIV prevention skills in different situations to maintain emotional, physical, and sexual health. Intervention content was presented through animation, videos, and games. The control condition used slide sets to present foundational HIV prevention information. Additional information about the intervention development and content can be found in previously published manuscripts (Greene et al., 2016; Mustanski, Garofalo, Monahan, Gratzer, & Andrews, 2013; Mustanski et al., 2017b).

Participants completed a baseline assessment, intervention or control modules, and follow-ups assessments at 3, 6, and 12 months post-intervention. Sexually transmitted infection (STI) testing for gonorrhea and chlamydia at the urethral and rectal sites was administered for all participants at baseline and 12-month follow-up. Primary outcomes of interest were the count of condomless anal sex (CAS) acts and incidence of STIs. This study was approved by the Institutional Review Boards at Northwestern University, Emory University, and Hunter College.

Potential participants were assessed for eligibility through a brief screener. To screen as eligible, participants initially had to be between the ages of 18 and 24, be male assigned at birth and currently identify as male, report CAS with a male partner in the previous 6 months, not be in a behaviorally monogamous relationship lasting longer than 6 months, and have received an HIV negative test result from a study site or remote HIV testing. Participants were also required to read English at least at an 8th grade level and have an e-mail address that could be used for research contact.

Eligibility requirements were modified twice during the recruitment phase of the study. First, the age of eligibility was expanded from a maximum of 24 to 29 years in order to include older YMSM who are also greatly affected by HIV/AIDS and to increase the pool of eligible YMSM participants. Second, enrollment of White participants was capped in the final year of recruitment to focus on enrolling Latino and Black participants. This was done to increase diversity and meet the study goal of enrolling a sample that was approximately equal among White, Black, and Latino.

HIV testing clinics and mobile testing units of partner CBOs primarily serving diverse YMSM in Atlanta, Chicago, and New York City were the initial source of recruitment for participants as prior research suggested that linking study recruitment to face-to-face HIV testing by trusted staff increased participation of racial and ethnic minorities (Du Bois et al., 2012). However, as a result of an insufficient pace of enrollment at CBOs, study recruitment strategies were expanded to include three venues shown to be successful at reaching YMSM at risk of acquiring HIV: Facebook, geosocial dating applications (apps), and outreach events (Grov, 2012; Grov, Rendina, Jimenez, & Parsons, 2016c; Rendina, Jimenez, Grov, Ventuneac, & Parsons, 2014). The recruitment and eligibility screening process differed by recruitment venue.

Community-Based Organizations

YMSM who tested HIV negative at CBOs were informed of KIU! 2.0 at the conclusion of their testing session by CBO staff and asked if they were interested in learning more about the study. Interested clients were given a brief description of the study and asked to complete an online or paper screener to assess eligibility. Eligible participants were e-mailed a link to the baseline assessment.

Facebook and Geosocial Dating Applications

Study staff ran local and national advertisement (ad) campaigns on Facebook and geosocial dating applications. Staff used filters to target Facebook users who indicated in their profiles that they were men, between 18–29 years old, and lived within 25 miles of the local study site (Chicago, Atlanta, or New York). After eligibility requirements changed to include ethnic and racial restrictions, the study teams added filters for Black or Latino targeted ads. Staff were only able to use location filters on dating apps. Interested participants who clicked ads were routed to an online preliminary screener to access eligibility. Preliminarily eligible participants recruited from local ads were contacted by phone and e-mail to schedule study visits to complete HIV testing and full eligibility screening. Participants in Atlanta and New York City were scheduled to visit Emory or Hunter, respectively, while participants in Chicago were scheduled to visit one of the local CBO partner sites. Participants who screened as eligible at Emory and Hunter were also offered enrollment during the same study visit.

Interested and preliminarily eligible participants recruited from national ads were sent an FDA-approved, at-home, oral fluid OraQuick HIV test kit (OraSure Technologies, Bethlehem, PA). They were required to provide study staff with a photo of their at-home HIV test stick and complete full eligibility screening over the phone with study staff. Participants who were recruited through local Facebook and dating app ads but unable to visit Emory, Hunter, or a Chicago CBO site were also offered the opportunity to complete at-home HIV testing. Upon confirmation of their HIV negative status and eligibility, participants were e-mailed a link to the baseline assessment.

Outreach Events

Study staff attended events (e.g., Pride Fests and Drag Balls) and venues (e.g., bars and clubs) frequented by YMSM. In New York, potential participants were approached by study staff to complete preliminary eligibility screening on a handheld electronic device. In Atlanta and Chicago, staff collected contact information of individuals who were interested in participating and contacted them to complete preliminary eligibility screening over the phone or through an online screener via a link sent by study staff. Participants who preliminarily screened as eligible were offered the same full eligibility screening and HIV testing options as participants recruited from Facebook and dating apps.

In addition to these four main recruitment strategies, participants were recruited from friend referrals, Chicago health department STI clinics, and research participant registries. These participants followed recruitment and enrollment processes similar to those described above. They were not included in analyses due to the small number recruited from these sources.

Regardless of recruitment venue, there were four stages of enrollment: (1) screening as eligible, (2) completing the baseline assessment, (3) completing baseline STI testing, and (4) beginning the intervention. Participants were regarded as “enrolled” when they began the intervention and were exposed to study content.

Measures

Demographics

Participants self-reported their age, race/ethnicity, sexual orientation, and relationship status during eligibility screening. The study team also manually entered recruitment venue information and used contact addresses provided by participants to categorize them as residing in the West, Midwest, South, or Northeast region of the country. The regions were based on the US Census Bureau’s groupings (U.S. Census Bureau, 2015).

Sexual Behaviors

The HIV-Risk Assessment for Sexual Partnerships (H-RASP) assessed sexual behaviors in the previous 3 months on a partner-by-partner level for participants’ three most recent partners, as well as in the aggregate for additional partners (Mustanski, Starks, & Newcomb, 2014; Swann, Newcomb, & Mustanski, 2018). Items of particular interest were frequency of CAS acts and the number of sex partners with whom condoms were not used. Follow-up questions differentiated between insertive and receptive anal sex. A sample question is, “How many times did you have sex without using a condom during anal sex where you were the bottom with this partner?”

Condom Errors

An abbreviated version of the Condom Use Errors and Problems Questionnaire was used to measure how often participants experienced condom failure (e.g., breakage during sex), condom errors (e.g., using oil-based lubricants), or erection loss that occurred prior to or during sex with a condom (Mustanski, Ryan, & Garofalo, 2014a). Participants responded using a 5-point Likert scale (1 = never, 5 = always). A sample question is, “When you used condoms during anal sex in the last 3 months, how often was space left at the end of the condom when putting it on?”

Sexual Health History

Study staff developed a short questionnaire to assess participants’ sexual health history. Participants were asked about previous STI diagnoses, HIV testing, and past participation in HIV prevention programs. A sample question is, “Have you ever been told by a doctor or nurse that you had a sexually transmitted infection, such as chlamydia, trichomoniasis (trich), syphilis, gonorrhea (clap), genital herpes, genital warts (HPV), or pubic lice (crabs)?” In addition, participants completed at-home STI testing for urethral and rectal chlamydia and gonorrhea at baseline (see Mustanski, Feinstein, Madkins, Sullivan, & Swann, 2017a for details of testing protocol).

Analytic Plan

Univariate and bivariate analyses were conducted to assess the frequency of completion for each stage of enrollment and HIV sexual risk behaviors by recruitment venue. For categorical variables, chi-square tests were used to examine differences in attrition during the enrollment process and HIV sexual risk behaviors. Post hoc analyses were used to interpret chi-square contingency table test results (Beasley & Schumacker, 1995). Kruskal–Wallis H test was used for non-normally distributed continuous variables. Logistic regressions were used to calculate odd ratios for the completion of each stage of enrollment. All analyses were conducted in SPSS Statistics 23.

Results

Participants

A total of 2984 potential participants were approached for the study. Few participants were uninterested and declined to complete the eligibility screener (see Fig. 1). Nearly all of the participants (98%) who declined to complete the eligibility screener were recruited from CBO partner sites. Due to the small number of incomplete eligibility screeners and concentration among CBO sites, eligibility screener completion was not included in later analyses.

Fig. 1
figure 1

Participant Recruitment and Enrollment Flow Diagram, Keep It Up! 2.0, 2013–2015

Full size image

The majority of participants (63.7%) were recruited from CBOs, while the fewest were recruited from Facebook (6.9%). There were significant differences in demographics by recruitment venue, particularly when focusing on non-CBO sites (see Table 1). The smallest proportion of White participants were recruited from Facebook, while nearly twice as many Black and Latino participants were recruited from the social media website. The significantly smaller number of White participants recruited from Facebook was the result of staff using Facebook’s ethnic affinity targeting feature during recruitment. This allowed staff to target ads to users who liked content that Facebook categorized as having an “African-American” or “Latino” affinity. The dating apps that were used for recruitment did not have a similar feature that allowed for targeting of non-White participants. A significantly larger proportion of White participants were recruited from dating apps, while significantly fewer Latino and more Black participants were recruited from outreach events. When looking at age, nearly twice as many 18–24 year olds were recruited from outreach events while significantly more 25–29 year olds were recruited from dating apps. Fewer bisexual identified participants, compared to gay participants, were recruited from Facebook, but the difference was not statistically significant. Most of the participants approached during recruitment were not in relationships, but of those in a relationship, a significantly higher proportion of participants in serious non-monogamous relationships were recruited from dating apps while significantly fewer participants in relationships lasting less than 6 months were recruited from apps.

Table 1 Demographics of participants screened for eligibility by recruitment venue (N = 2724), Keep It Up! 2.0, 2013–2015
Full size table

Most participants (84.6%) completed eligibility screening in Atlanta, Chicago, and New York. Of those who were not in the primary three cities, most lived in nearby or adjacent states (e.g., Indiana, New Jersey, and Florida). Few participants were recruited from the West Coast (n = 53) and, as a result, were excluded from later analyses of geographic differences. Recruitment varied across the university study sites and regions. The universities focused on recruitment strategies that proved particularly successful for their sites. Midwest recruitment, led by Northwestern, primarily occurred at local CBOs. In the other two regions, recruitment occurred primarily through online ads posted by both Hunter and Emory, and at outreach events attended by Emory staff.

When looking specifically at the enrolled sample, the only significant demographic differences in recruitment venue was by race/ethnicity (χ2[9] = 54.18, p < .001). Nearly 50% of participants enrolled from dating apps were White, while only 22% of participants enrolled from Facebook identified as White. Nevertheless, the final enrolled sample was ethnically and racially diverse: 37% White, 29% Latino, 24% Black, and 10% Other/Multiracial. The overall mean age of enrolled participants was 24.2 years. Most identified as gay (86%), not in a relationship (71%), having at least some college education (88%), and being employed at least part time (78%). Enrolled participants were recruited from a variety of venues: 44.2% were recruited from dating apps, 23.2% from CBOs, 20.3% from Facebook, and 12.3% from outreach.

Retention

Univariate analyses showed significant differences at three of the four stages of enrollment by recruitment venue and at all stages by the region from which participants were recruited (see Table 2). Participants recruited from non-CBO venues or from regions outside of the Midwest had greater odds of completing stages of enrollment. Odds ratios for eligibility by recruitment venue were not calculated for two reasons. First, there were changes to eligibility during recruitment that could affect the ability to confidently interpret odds ratios. Second, the enrollment process for participants recruited through ads on Facebook and dating apps included preliminary screening for eligibility. Only preliminarily eligible participants from these recruitment venues proceeded to complete HIV testing and the full eligibility screener. As a result, nearly all of the participants recruited from Facebook and dating apps were eligible to participate.

Table 2 Regional and recruitment venue differences in screened participants completing enrollment stages, Keep It Up! 2.0, 2013–2015
Full size table

When controlling for age, race/ethnicity, sexual orientation, relationship status, and region, non-CBO participants were significantly more likely to complete the baseline assessment (Facebook: OR = 20.99, p < .001; dating app: OR = 12.95, p < .001; outreach: OR = 2.36, p < .01) and STI testing (Facebook: OR = 12.25, p < .001; dating app: OR = 5.17, p < .001; outreach: OR = 5.56, p = .001). When controlling for age, race/ethnicity, sexual orientation, relationship status, and recruitment venue, only participants recruited from the Northeast maintained their higher odds of completing the baseline assessment (OR = 3.66, p < .001).

When looking at demographics, only age had an effect on the likelihood of completing more than one stage of enrollment. As shown in univariate analysis results in Table 3, older participants had significantly greater odds of screening as eligible and completing the baseline assessment. When age was included in a multivariable model, older participants only remained significantly more likely to complete the baseline assessment (OR = 1.40, p < .05). There were fewer differences by race/ethnicity, sexual orientation, and relationship status. Participants in a serious, long-term non-monogamous relationship had greater odds of screening as eligible compared to participants not in a relationship. Bisexual identified participants had lower odds of completing STI testing compared to gay participants. Black and Latino participants had lower odds, compared to White participants, of enrolling in the intervention. These differences persisted when all variables (age, race/ethnicity, sexual orientation, relationship status, region, and recruitment venue) were included in a multivariable model (serious long-term relationship: OR: 1.58, p < .05; bisexual: OR = 0.50, p < .05; Black: OR = 0.35, p < .01; Latino: OR = 0.46, p < .05).

Table 3 Demographic differences in screened participants completing enrollment stages, Keep It Up! 2.0, 2013–2015
Full size table

Among fully enrolled participants, there were differences by race/ethnicity (χ2[3] = 14.70, p = .002) and recruitment venue (χ2[3] = 12.33, p = .006) for completion of all three sessions of the intervention. Participants who identified as non-White were all less likely to complete the intervention than White participants (Black: OR = 0.36, p = .001; Latino: OR = 0.37, p = .002; other: OR = 0.33, p = .004). Specifically, 94.8% of White participants completed the intervention while 87.3% of Latino participants, 86.8% of Black participants, and 85.9% of participants who identified as another race completed the intervention. When race/ethnicity was included in a multivariable model with age, sexual orientation, relationship status, region, and recruitment venue, non-White participants remained significantly less likely to complete the intervention. Participants recruited from dating apps had twice the odds of completing the intervention than those recruited from CBOs (OR = 2.13, p = .018). This higher odds of completing the intervention did not hold when controlling for age, race/ethnicity, sexual orientation, relationship status and region.

Sexual Risks

There were no significant differences by recruitment venue in sexual risk behaviors. Engaging in condomless anal sex in the previous 3 months, having a history of STIs, and testing positive for either gonorrhea or chlamydia at baseline did not differ by recruitment venue in univariate analyses (see Table 4). In addition, there were no significant differences in number of condomless partners (χ2[3] = 6.41, p = .09) and number of condom errors (χ2[3] = 0.08, p = .99) in the previous 3 months.

Table 4 Sexual risk behaviors at baseline of enrolled participants by recruitment venue (N = 901), Keep It Up! 2.0, 2013–2015
Full size table

Discussion

At the start of the Keep It Up! 2.0 intervention, participants were recruited solely from CBO partner sites. To accelerate enrollment, recruitment was expanded to include other strategies. The goal of this paper was to examine if the expansion of recruitment strategies had an impact on participant demographics, completion of the enrollment process, and in HIV sexual risk behaviors among enrolled participants. Understanding such differences can inform recruitment plans for future eHealth HIV prevention programs that seek to recruit diverse samples.

The populations that were approached at each particular venue differed by age, race/ethnicity, sexual orientation, and relationship status. A greater proportion of younger participants were recruited offline at CBOs and outreach events compared to online venues. This is similar to the findings of other studies comparing recruitment venues (Grov, 2012; Parsons et al., 2013). While these differences were significant, it is important to contextualize these findings by noting that the entire sample was YMSM and therefore differences represent approaches that were stronger in recruiting younger participants within the late adolescent to emerging adult developmental periods. Future ehealth studies should consider examining differences in recruitment by ages within these developmental periods.

The findings related to race and ethnicity differed from past studies. While fewer Black and Latino participants were recruited on dating apps compared to White participants, more were recruited from Facebook. This contrasts with past studies that found that more White participants than non-White participants were recruited from across general social networking websites like Facebook and dating specific apps. The smaller number of White participants recruited from Facebook in this study was the result of a newer Facebook feature that allows researchers to use ethnic affinity targeting during ad campaigns. This feature allowed staff to target recruitment ads to users who liked content that Facebook categorized as “African-American” or “Latino.” Although some users who do not identify as African-American or Latino were reached by these ads, the majority of people who saw the ads on their Facebook feed and went on to complete the study eligibility screener did identify as such.

A smaller proportion of bisexual identified participants, compared to gay identified participants, were recruited from Facebook despite users who listed being “interested in men and women” being among those targeted by ads. While there has been ongoing interest in the study of bisexual identified men in HIV prevention and sexual health research, we are aware of no other studies that have compared recruitment rates of bisexual identified participants across sources. Past studies, however, do suggest that recruitment from gay-oriented spaces is more likely to yield participants whose bisexual identity is transitory or contrary to their sexual behaviors (Dodge, Reece, & Gebhard, 2008). To improve recruitment of bisexual identified men, future studies should include recruitment venues that are specifically for a bisexual identified audience.

When looking specifically at differences by recruitment venue in who became enrolled, the only statistical difference was in race/ethnicity. Nearly 50% of participants enrolled from dating apps were White while over two-thirds of participants enrolled from Facebook were Black or Latino. This finding aligns with past studies that found that White YMSM were more likely to be enrolled from dating apps (Goedel & Duncan, 2015; Huang, Marlin, Young, Medline, & Klausner, 2016; Landovitz et al., 2013; Vial et al., 2014) and the recent few that have found better success in enrolling YMSM of color specifically from Facebook (Prescott et al., 2016; Young, Szekeres, & Coates, 2013). Changes in technology are the likely explanation for this study’s success in reaching YMSM of color on Facebook. At the time of recruitment for this study, staff were able to use Facebook’s ethnic affinity targeting feature. This allowed staff to target ads to users who liked content that Facebook categorizes as “African-American” or “Latino.” In contrast, the dating apps only allow targeting by location. The targeting is on a city-wide level versus zip code level, the latter of which could be useful for reaching YMSM of color living in segregated cities. These findings suggest that without the option to target specific demographics, YMSM of color will continue to be under-enrolled from other online venues like dating apps (Grosskopf, Harris, Wallace, & Nanin, 2011; Sullivan et al., 2011). In contrast, the absence of other statistically significant demographic differences by recruitment venue suggests that while there may differences in who is initially approached, YMSM of similar ages, sexual orientation, and relationship status can be enrolled into interventions.

There were few differences by recruitment venue in the likelihood of participants completing the four stages of enrollment. When controlling for age, race/ethnicity, sexual orientation, relationship status, and region in a multivariate model, participants recruited from Facebook, dating apps, and outreach events were more likely than those from CBOs to complete the baseline assessment and STI testing. However, similar to other recent studies, there were no differences by recruitment venue among participants who completed enrollment into the study by starting the intervention (Frandsen, Thow, & Ferguson, 2016; Hernandez-Romieu et al., 2014; Parsons et al., 2013). Our hypothesis that participants recruited from CBOs would be more likely to enroll was unsupported. These results suggest that there may be fewer enrollment differences by recruitment source when a simplistic enrollment process after eligibility screening is eschewed in favor of a multistep process that requires participants to expend more effort to become enrolled. The only significant difference in a multivariate model predicting enrollment was race/ethnicity with Black and Latino participants being less likely to enroll in the study. This finding is consistent with past research that has shown that YMSM of color are less likely to enroll in Internet-based HIV behavior studies (Sullivan et al., 2011). Black and Latino participants were also less likely to complete all three sessions of the intervention. These findings suggest that YMSM of color may need to be oversampled during recruitment in order to have them enroll and complete interventions at a similar rate to that of White YMSM. It also suggests the need for further research to assure eHealth HIV prevention programs are relevant and engaging to YMSM of color.

Among enrolled participants, there were no differences by recruitment venue in self-reported HIV sexual risk behavior. This contrasts with our hypothesis and most of the findings from other studies where, compared to men enrolled from offline venues, men enrolled from online sites reported more behaviors which could put them at risk for HIV infection (Hernandez-Romieu et al., 2014; Parsons et al., 2013; Sanchez et al., 2012). A meta-analysis of 14 studies found that MSM recruited online, regardless of whether the app or site was dating or hook-up oriented, reported more HIV risk behaviors than those recruited from offline venues (Yang et al., 2014). A potential explanation for the similarity of risks and STI prevalence among participants is presented by other previous studies that have found that high percentages of MSM reached online now also visit offline venues regularly and vice versa (Hernandez-Romieu et al., 2014; Raymond et al., 2010; Saxton, Dickson, & Hughes, 2013). In addition, all participants had to self-report condomless anal sex in the past 6 months in order to screen as eligible for the study. As a result, there may be a floor effect for risky sexual encounters reported by participants.

There were also no significant differences by recruitment venue in STI prevalence among enrolled participants. This differs from the findings of previous studies. It is worth nothing that few studies have included prevalent cases of laboratory confirmed STIs in their analyses. Instead, most use self-report of past STIs as disease outcomes (Grov, Rendina, & Parsons, 2014; Lehmiller & Ioerger, 2014). Results from these studies are mixed regarding the impact of recruitment venue on STI diagnosis. One of the few studies that used prevalent cases of STIs as an outcome found that MSM recruited from venues such as bars, clubs, and Pride events were significantly more likely to be infected with a rectal STI than MSM recruited on Facebook (Hernandez-Romieu et al., 2014). Additional research comparing prevalent cases of STIs, both urethral and rectal, across recruitment venue will help to assess the comparability of participant health outcomes.

The results of this study should be considered in light of several limitations. Sexual risks were self-reported and therefore susceptible to social desirability bias. This bias may have, however, been moderated since data on sexual risks were collected online and not in person by study staff. The use of pre-screening for participants enrolled from online venues but not for those recruited from offline venues, as well as changes to eligibility during recruitment, limited our ability to assess for differences in rates of completing the eligibility screener along with screening as eligible. Participants from the West region of the country were not included in analyses because of the small number recruited. It is possible that in addition to less intensive recruitment in this region, YMSM from the West clicked online ads less frequently for the study due to lower name recognition of the research partners. In addition, there was overlap between the venue and region from which participants were recruited. This overlap may have limited detection of regional differences. More participation from YMSM living in the West and greater variety of recruitment venues within each region would have allowed for a fuller understanding of potential venue and regional differences.

Despite these limitations, the current study has multiple strengths. It is one of few studies to compare recruitment venues for an online HIV prevention intervention tailored to YMSM. Most of the previous research into recruitment venue differences was based on one time surveys or offline interventions. In addition, our study suggests that it is possible to enroll a racially and ethnically diverse sample of YMSM with similar risk profiles despite recruitment from a wide range of venues.

Conclusions

This study has shown that recruitment from a variety of venues is feasible for online HIV prevention programs tailored for racially and ethnically diverse YMSM. Despite the expansion of study recruitment beyond CBO partner sites, YMSM at similar risk for HIV infection at baseline were enrolled into the study. There were also no differences in rate of enrollment by recruitment venues. The differential racial and ethnic makeup of recruitment venues continue to be a factor for researchers to consider. The lower number of YMSM of color approached on online dating apps highlights the need to include targeted recruitment of these populations as a strategy. Recruiting from venues, such as Facebook, which allow ads to be shown to specific racial and ethnic groups may serve as a counterbalance to predominately White venues. This study also shows the need for additional research to understand why, once approached for recruitment, Black and Latino YMSM remain less likely to enroll in HIV prevention studies. It is vital that researchers improve their ability to engage Black and Latino YMSM as they continue to be most affected by HIV.