Introduction

Both cross-sectional and cohort studies have documented a rapid increase of HIV prevalence among men who have sex with men (MSM) in China (from 1.4% in 2001 to 6.3% in 2011) [1,2,3], which is especially notable in the nation’s capital, Beijing [1, 4,5,6,7,8,9]. Multiple sexual partners, inconsistent condom use, and use of alcohol and aphrodisiac substances are associated with increasing HIV transmission risk among Chinese MSM [1, 2, 5,6,7,8, 10,11,12,13,14]. MSM in Beijing have the option of visiting voluntary counseling and testing (VCT) clinics for HIV testing. Though MSM are advised to test every 6 months [15,16,17], frequent VCT among MSM in China is uncommon [18, 19] even in communities with good HIV risk awareness and social tolerance towards homosexuality [20,21,22]. Frequent testing could find those living with HIV as soon as possible, and subsequent ART treatment can reduce transmission to others. However, HIV-negative MSM are more likely to continue risk behaviors when they push their luck. One study cited an increase of high risk behaviors among HIV-negative MSM who tested for HIV regularly, so effective risk reduction counselling must be a part of VCT services [23]. Both governmental and non-governmental sources emphasize that the efforts and investments of interventions to reduce HIV transmission for MSM must be evaluated to see what works to curb the growing HIV epidemic [11, 24]. Recognizing MSM at highest risk of HIV/AIDS transmission can contribute to more efficiently targeted interventions, either specific to or independent of Chinese sexual cultural context [25, 26].

Developing risk scores for one disease using clinical prediction rules can generate valuable public health tools [27,28,29,30]. Risk scores for HIV infection have used traditional variable selection of backward selection or stepwise procedures in maximum likelihood estimation [30,31,32,33,34,35], or manual variable selection and model comparisons based on the associations between HIV infection and each potential predictor [36, 37]. Identifying MSM at greatest risk for HIV infection can permit better to focus on prevention resources and interventions, such as enhanced counseling and/or antiretroviral pre-exposure prophylaxis (PrEP) [38, 39]. HIV prevention clinical trial designs can be made more efficient if higher risk persons are identified and recruited preferentially. We sought to develop a risk score for HIV infection based on Chinese MSM using Harrell’s regression model strategies [40], which may reduce prediction error compared to traditional maximum likelihood estimation.

Methods

Study Population and Setting

Our NIH-funded combination HIV prevention project included two components: (1) promoting HIV testing in Beijing among MSM, including transgender women; and (2) a pilot randomized clinical trial of interventions for enhancing linkage to HIV care among newly diagnosed HIV-infected MSM. Data for this paper were drawn from the first component, a cross-sectional study. MSM participants were recruited mainly from the regular clients who visited four study clinics or through short message service, peer referral, web-advertisement and community outreach [41, 42]. Participant recruitment was conducted between March 2013 and March 2014 in four HIV VCT clinics in Beijing, including Beijing Municipal Center for Disease Control and Prevention (CDC), Xicheng District CDC, Chaoyang District CDC, and Jingcheng Skin Disease Hospital.

The inclusion criteria for study participation were: man or transgendered woman who self-reported having sex with a man in the last 12 months; age ≥ 18 years; living in Beijing; do not know their own HIV status or ever diagnosed as HIV negative in the last HIV testing (in order to find newly-diagnosed HIV positive MSM for second component of whole project); able and willing to provide written informed consent to questionnaire surveys and serological testing; and not having participated previously in HIV testing in this study. For identifying the latter, cell phone numbers of potential participants were used to screen for possible duplicate testing. This study protocol and informed consent form were reviewed and approved by the institutional review boards of the National Center for AIDS/STD Control and Prevention (NCAIDS) of the Chinese Center for Disease Control and Prevention (FWA00002958) and Vanderbilt University Medical Center (FWA00005756).

Data Collection and Laboratory Tests

An interviewer-administered structured questionnaire was used to collect respondent potential risk predictors. All interviewers were trained of interview skills, protocol and questionnaires before study recruitment and all had prior work experience within HIV/AIDS-related programs. One private room was provided for face-to-face interview between one trained interviewer and each potential participant, which could prevent participants from telling a lie due to stigma. Negative results of both HIV and syphilis rapid tests using finger or venous blood at clinics were provided for participants following interview completion on the same day. Pre- and post-test counseling was provided to participants, including the significance and possible limitations of the test (e.g., HIV window periods) and safer sex practices. Further laboratory confirmatory tests of HIV and syphilis were performed within 2 weeks for positive screening test results. If venous blood was collected for rapid tests, the remaining of the blood sample was used for serological confirmatory tests. If a finger stick was used for rapid tests, and a positive result on either test was received, a venous blood sample was collected for confirmatory tests of both HIV and syphilis.

HIV rapid tests were performed using Alere Determine™ HIV1/2 (Colloidal Selenium Device; Alere Medical Co, Ltd., USA). Venous specimens were further screened for HIV antibody using ELISA (HIV ELISA testing kit 1, ZHUHAI LIVZON Diagnostics Inc., China). If positive by ELISA screening, the specimen was double tested using the same ELISA kit and another ELISA kit (HIV ELISA testing kit 2, Beijing WANTAI Biological Pharmacy Enterprise Co. Ltd., China). A specimen with a positive reaction in both or either ELISA testing kits was confirmed by Western blot test (HIV Blot 2.2 WB; MP Biomedicals Co, Ltd., China). One Step Anti-TP (Treponema pallidum/Syphilis) Test (Colloidal Gold Device, Standard Diagnostics, Inc., Korea) was used as a syphilis rapid test. For rapid test positive persons, syphilis serology was determined through Syphilis Toluidine red unheated serum test (Beijing WANTAI Biological Pharmacy Enterprise Co. Ltd., China) and confirmed by the Treponema palladium particle assay (Fujirebio Inc., Tokyo, Japan).

Candidate Variables

Potential predictors were considered for exposure to HIV risk (original questionnaire in Appendix S1) that could also be regularly collected among MSM in general HIV prevention and research settings. Nineteen candidate variables were used based on plausibility of risk association (Appendix S2). The number of knots specified according the estimated “power” of each predictor. The ten candidate variables (items) were included in model as continuous variables using restricted cubic splines with four knots, which offer an adequate fit of the model and are a good compromise between flexibility and loss of precision caused by overfitting a small sample [40, 43]. If continuous data had low variability, the number of knots was reduced to 3, or continuous data were categorized as ordinal or dichotomous variables, where relevant. Default knot locations chose quantiles or tertiles of each predictor’s distribution. Two items were collected using ordinal scales (Appendix S2).

Statistical Modeling and Sample Size Justification

The primary study outcome was HIV sero-status. The associations between potential predictors and HIV infection were evaluated using a logistic regression model. Sample size for prediction modeling must be large enough or have a conservative list of candidate predictors to achieve reproducibility in future populations. Specifically, 10 infections should be observed per parameter numbers estimated in the model. Our proposed model has 10 continuous variables, 6 dichotomous variables, and 3 ordinal variables that yield 40 parameters. With 455 HIV infections, we allowed 45 parameters to avoid overfitting.

Statistical Modeling of the Full Model and Reduced-form Model

The following analyses were conducted using R-software version 3.3.1 (R statistical software, Institute for Statistics and Mathematics, Vienna, Austria) [44, 45]. All pre-specified variables described above were included in a full logistic regression model of HIV infection using shrinkage of estimated regression coefficients, which might have larger predictive capability when used for future similar population. If selecting variables only through univariate screening and expert experiences, bias in estimating regression coefficients might be introduced [40, 46,47,48]. In order to correct for potential overfitting of regression coefficients, we applied penalized maximum likelihood estimation (PMLE), which maximized the log-likelihood of conventional logistic regression, provided smaller prediction errors, and preserved the discriminative ability of the model while shrinking each predictor for over-optimism [40, 43]. No predictor interactions were examined during modeling due to the complexity of developing and interpreting a model with interaction terms. To reduce the number of variables in the above-mentioned full model and to identify a subset of items that could be used in future surveys of assessing HIV infection risk among MSM, a reduced-form model was fitted using backward variable selection at an alpha criterion of 0.25 with bootstrap validation. Odds ratios (OR) 95% confidence intervals (CI), and nomograms were summarized for both the full and reduced-form models. Online evaluation tools were published using Harrell application by RStudio (http://shiny.rstudio.com/).

Model Calibration and Discrimination

Model accuracy regarding both calibration and discrimination was assessed [46, 49]. Calibration is the ability of our model to predict either future observations or the current survey data. Calibration was evaluated by graphing the predicted probabilities (x-axis) against observed outcomes (y-axis). Since observed outcomes are binary (1 = HIV-infected and 0 = not infected), we used locally weighted scatterplot smoothing (LOWESS) to estimate the observed probabilities of the outcome in relation to the predicted probabilities and compared this line with a 45° line which represents perfect calibration. Predictive discrimination was assessed using the c-index, which is a rank-order statistic to estimate the probability of concordance between the predicted and observed outcomes and to measure how well the model discriminates between different outcomes.

Internal Bootstrap Validation

Bootstrap validation was used to assess the predictive ability and reproducibility of the model [43, 46, 49, 50]. Internal bootstrap validation was performed as follows: (1) estimate the apparent predictive ability with the original sample dataset used to fit the model; (2) draw 300 bootstrap samples with replacement for fitting the model and measuring the predictive ability for each bootstrap sample; (3) test the accuracy measures by applying the bootstrap model to the original sample; (4) calculate the optimism in the predictive ability for each bootstrap model as the difference in performance between the bootstrap model and the original dataset; and (5) use the average of the optimism over all 300 bootstrap samples as a correction factor to estimate a validated performance measure. Thus, bootstrap-corrected estimates of the c-index were estimated for discrimination along with calibration curves for the full and reduced-form models [43].

Results

General Characteristics of Study Participants

During recruitment, 3751 MSM were screened for HIV infection. Fully 95.7% (n = 3,588) met inclusion criteria for our study and were used for HIV infection analyses. Participants (n = 163) were excluded due to the following three reasons: (1) duplicate HIV testing using the same cell phone numbers in different study clinics or on different dates (n = 126); (2) diagnosed as HIV-infected prior to participation in this study since those who have knew their HIV positive status may change their behaviors (n = 30); (3) refusing venous or finger blood collection (n = 5); (4) incomplete questionnaire interview (n = 2).

The actual HIV prevalence was 13.4% among 3620 MSM after excluding 126 duplicate HIV testers and 5 refusing blood collection. Among 3588 eligible MSM who do not know their HIV status or HIV negative in the last HIV testing, and 12.7% tested positive for HIV. The mean age was 29.8 [standard deviation (SD), 7.8) years with the range of 18–75 years; 93.7% identified as Han ethnicity; 15.0% were currently married to a woman; 71.9% had obtained a bachelor’ degree or higher; 80.6% were employed full time; 24.8% had Beijing permanent resident cards (called Hukou), without which MSM may have some restrictions, including buying apartment and cars, right to vote and to be elected, their children to school, and so on (protective impact of local residents on HIV risk was also observed in other Chinese big cities [1, 7, 51, 52]). Table 1 presents descriptive summaries of possible predictors and their associations with HIV infection estimated using univariate logistic regression models. Married currently with a woman, having Beijing Hukou, less years of living in Beijing, nitrite inhalants use (also called Rush), illegal drug use (including methamphetamine, MDMA, and ketamine), numbers of lifetime male sexual partners ≥ 10, unprotected receptive anal sex with male sexual partners, alcohol consumption often before sex, no sex with female sexual partners, consistent condom use with female sexual partners, and syphilis infection were associated with HIV acquisition (P < 0.05).

Table 1 Univariate associations of potential predictors with HIV infection risk
Full size table

Full Model

HIV infection was modeled using penalized multivariable logistic regression with the 19 potential predictors (Table 2). Eleven variables were associated with HIV infection, including Beijing Hukou (yes vs. no; OR 0.55; 95% CI 0.38–0.82), years of living in Beijing (10 vs. 2 years; OR 0.66; 95% CI 0.51–0.86), illegal drug use in the past 3 months (yes vs. no; OR 4.34; 95% CI 2.40–7.87), alcohol use in the past 3 months (≤ 1 time/month vs. never; OR 1.36; 95% CI 1.04–1.77), alcohol consumption before sex in the past 3 months (often vs. never; OR 2.98; 95% CI 1.45–6.10), number of lifetime male sexual partners (20 vs. 4; OR 1.45; 95% CI 1.15–1.83), number of male sexual partners in the past 3 months (2 vs. 1; OR 0.84; 95% CI 0.72–0.98), frequency of receptive anal sex in the past 3 months (3 vs. 0; OR 1.75; 95% CI 1.30–2.35), proportion of condom use during receptive anal sex in the past 3 months (100% vs. 0%; OR 0.53; 95% CI 0.37–0.77), frequency of insertive anal sex in the past 3 months (4 vs. 0; OR 0.70; 95% CI 0.52–0.93), and syphilis infection (yes vs. no; OR 2.87; 95% CI 2.11–3.91).

Table 2 Full and reduced-form prediction rule models and penalized odds ratios for the associations of potential predictors with HIV risk
Full size table

Reduced-form Model

Backward step-down variable selection of risk variables from the full model was used to obtain a reduced-form model for HIV infection (Table 2, Fig. 1, Appendices S3), including Beijing Hukou (yes vs. no; OR 0.55; 95% CI 0.38–0.81), years of living in Beijing (10 years vs. 2 years; OR 0.67; 95% CI 0.52–0.86), illegal drug use in the past 3 months (yes vs. no; OR 4.35; 95% CI 2.43–7.76), alcohol consumption before sex in the past 3 months (often vs. never; OR 2.83; 95% CI 1.48–5.39), time since the first sex (12 years vs. 4 years; OR 1.35; 95% CI 1.02–1.78), number of lifetime male sexual partners (20 vs. 4; OR 1.41; 95% CI 1.13–1.75), number of male sexual partners in the past 3 months (2 vs. 1; OR 0.85; 95% CI 0.73–0.99), frequency of receptive anal sex with male sexual partners in the past 3 months (3 vs. 0; OR 1.67; 95% CI 1.25–2.24), proportion of condom use during receptive anal sex with male sexual partners in the past 3 months (100% vs. 0%; OR 0.49; 95% CI 0.34–0.69), frequency of insertive anal sex with male partners in the past 3 months (4 vs. 0; OR 0.78; 95% CI 0.60–1.01), sex with female partners in the past 3 months (yes vs. no; OR 0.53; 95% CI 0.34–0.81), and syphilis infection (yes vs. no; OR 2.89; 95% CI 2.13–3.93). The reduced-form model yielded the bias-corrected c-indices of 0.71 (calibration curves in Appendices S4).

Fig. 1
figure 1

Reduced-form sexual risk score PMLE model nomogram for HIV infection

Full size image

Nomogram

The nomogram for penalized reduced-form model of risk score (Fig. 1) may be more convenient to evaluate HIV predicted probability for MSM in practice. The Points grid-line at the top of the nomogram is assigned to point values for each sexual variable by aligning a vertical line from the grid-line to the line adjacent to the sexual variable. The individual variable points are summed and a vertical line from the summed points on the Total Points grid-line to the Predicted Probability provides an estimated probability of HIV infection. For example, the corresponding probability of HIV infection is approximately 88.6% (95% CI, 74.2–95.4%) for a MSM who has no Beijing Hukou, lived in Beijing for 10 years before recruitment, illegal drug use, has 10 years of sexual experiences, has 20 male sexual partners in total and 5 male sexual partners in the past 3 months, 15 episodes of receptive anal sex with male sexual partners, 50% condom use during receptive anal sex with male partners, 5 episodes of insertive anal sex with male sexual partners, alcohol consumption often before sex, no sex with female partners in the past 3 months, and current syphilis infection. For sake of facilitating the calculation for a potential participant, our calculation is further illustrated using Shiny in Fig. 2 (available at https://meridithblevins.shinyapps.io/sexualrisk/).

Fig. 2
figure 2

Screenshot of calculating predicted probability for each MSM using Shiny app

Full size image

Discussions

We have developed a discrete set of predictors for HIV risk among Chinese MSM who do not know their HIV status or HIV negative in the last HIV testing using Harrell’s regression modeling strategies. The impact of each predictor might not be obvious on increasing HIV risk, but Harrell’s regression includes several potential predictors together to obtain stronger prediction abilities [40]. The predictors selected for our reduced-form risk score model are independently associated with HIV infection and included variables commonly measured in HIV prevention studies among Chinese MSM. The reduced-form model seems to be more competitive than the full model for HIV infection among Chinese MSM, and corresponding nomogram using Shiny is more convenient for health workers or education counsellors during routine prevention and intervention. To our knowledge, this is the first risk score derived from Chinese MSM, obtaining bias-corrected c-index of 0.71 in reduced-form model. Some risk scores have been developed and further simplified to predict the HIV infection probability showing c-index from 0.66 to 0.74 among American MSM [32, 33, 35, 53].

As we know, prediction errors and over-optimism derived from traditional regression models are inevitable. In addition, categorizing continuous variables might produce arbitrary cutoff values for some predictors when they were used in other studies. Harrell’s strategies utilizing PMLE, restricted cubic splines, and LOWESS have been confirmed to improve prediction capability of the model [40, 50], and used successfully to predict pediatric acute asthma for hospitalization [43]. In order to facilitate calculating the predicted probability of HIV infection for each MSM, our online risk tool is provided for clinic counsellors and physicians. A high predicted probability of HIV infection could influence the efficient allocation of prevention resources and careful selection of appropriate interventions after comprehensive consideration and assessment for the balance of benefits versus potential toxicity, especially enhanced counselling and PrEP [38, 39].

Although our risk score was derived from a cross-sectional study and validated internally using 3588 MSM in Beijing, China, external validation to MSM in other regions should be considered before widespread deployment. We did not develop this score to differentiate MSM in the process of pre- and post-counselling because a low score did not indicate zero HIV risk in our analyses. All MSM should be counselled about risk reduction strategies, including consistent condom use, the treatment of sexually transmitted infections (STIs), and exclusive relationships with sexual partners, regardless of female or male sexual partners. We think that the greatest utility of the scores will be in repeated assessments to track changes in HIV risk among MSM at set time intervals (e.g., every 3 or 6 months), helping to provide feedback for health workers or counsellors alike. It may also help identify sub-populations with a need for more intense intervention, as well as helping identify highest risk persons for HIV prevention clinical trial recruitment. Additionally, definitive HIV test confirmation delays the final determination of sero-status. For example, in China it commonly takes 1–2 weeks to receive Western Blot results. Some HIV-infected patients are sometimes not linked to care after diagnosis due to this notification delay or due to a lack of identifiable information. Using this score evaluation might help health workers identify high risk MSM, whose identity and contact information could be collected at clinic even before infection status is confirmed.

The HIV prevalence in our study (13.4%) is no doubt concerning as it is much higher than previous studies among MSM in Beijing [1, 3,4,5,6,7,8,9], which was consistent with Beijing Sentinel Surveillance on the same year (unpublished data). The increasing HIV epidemic might be substantially worse due to an increase of testing coverage provided by various large testing programs targeting MSM using significant monetary incentives [1, 2]. In addition, WHO issued a new guideline in July 2013 recommending earlier treatment and raised the threshold for starting ART from a CD4 of 350 cells/µL to 500 cells/µL [54], and Chinese government guidelines also loosened CD4 criteria for ART initiation at the end of 2013. Hence, the number of people living with HIV are increasing substantially which has stimulated great debate over disease burden and treatment costs in China.

It is worth mentioning that those with more alcohol use were not likely to have higher acquisition risk in our MSM participants like previous studies [12, 14, 55]. However, alcohol use before sex showed a positive dose–response association with higher acquisition risk in our study, which also was included in reduced-form model. Additionally, heavy alcohol use is not very common among Chinese MSM [12, 14, 41, 55]. Though the positive association between heavy alcohol use and HIV infection was reported in our study [41], heavy alcohol use defined by alcohol use disorder identification test-consumption (AUDIT-C) was not used in most MSM studies so far. Therefore, AUDIT-C was not considered in developing our prediction model.

Our analysis has specific limitations. First, all questionnaire data were based on self-reporting; study participants may have provided responses based more on social desirability, especially involving sensitive questions, such as sexual risk behaviors. Second, bias or spurious associations might be introduced because our participants in this study are volunteers recruited using non-randomized sampling, who instead felt they had recently engaged in risky behaviors or could differ on any number of other factors. Hence, the study sample may not represent the entire MSM community in Beijing or in China. Third, our risk score was validated only using our study, and further validation should be completed using data from other MSM studies in other locations prior to widespread implementation [27, 28, 56, 57]. Fourth, we simply measured prevalence at a single point in time among MSM of varying ages who happened to seek testing at our VCT clinics. Many variables included in the model were time-varying, such as age at the time of recruitment and number of lifetime male sexual partners, meaning that they would be expected to change throughout the course of an individual’s life thereby changing their predicted probability of HIV infection. Additionally, correlating a time-varying variable with a cross-sectional HIV test isn’t valid because nobody knows how many lifetime male sexual partnerships an individual had at the time he acquired HIV. Fifth, the recall period of recent risky sexual behaviors was set in the past 3 months in our study. Moderator numbers of Chinese MSM may change short-term sexual behaviors due to personal reasons, including feeling sick, career promotion, job change, and so on. Longer recall period might be preferred to find risk behaviors, since our study design is unable to collect risky behaviors when they infected. Our prediction model might also be altered a little if this recall period was increased to 6 months or decreased to 1 month in other studies. Finally, our cross-sectional study design has potential methodological limitations to detect the causality between exposure and outcome.

Conclusions

It remains challenging to provide early diagnosis, linkage to care, and early treatment to HIV-infected individuals. A simple, quantitative risk score could provide a robust and useful approach to discriminate MSM at highest risk for HIV acquisition and to evaluate changes of HIV risk over time in clinical, public health, or research settings. Use of a validated risk score may also help improve research efficiency in prevention studies such as PrEP by discriminating higher from lower risk persons.