Introduction

Efforts to prevent HIV infection and estimate HIV-related risk often rely on self-report data to identify, direct and evaluate interventions and programs. Similarly, research attempting to focus on populations at elevated risk for HIV often rely on self-report surveys to identify eligible participants. Survey items that ask about HIV risk perception (eg., chances of getting HIV) and risk behavior (eg., number of partners, ages of partner(s), sex events, event type and use of condoms) [1] have a limited validation evidence base, despite common use. Work conducted to date suggests that many of these items may be particularly prone to under-reporting due to self-presentation bias [2] and under-reporting of socially “undesirable” behaviors (eg., remaining sexually active during STI treatment) may in fact increase in the context of added pressure (eg., counseling promoting the desired behavior) [3]. Other issues such as problems in comprehension of items [4] or responses or their translation and local interpretation [5] may also erode accuracy in reporting. A critical aspect of validation is the establishment of a shared, clear understanding of what is being asked and what the answer options mean, as well as minimal demand for altering answers to manage self-presentation or mitigate perceived negative outcomes of choosing one answer over another [6]. One method to determine the extent to which survey items and answer choices have a clear and consistent meaning to those answering the questions is cognitive interviewing [7,8,9,10].

Cognitive interviewing is a strategy used in survey science to understand comprehension of questions, demands in terms of memory retrieval or cognitive tasks to move from question to answer selection, decision making processes involved which may include modifying initial answer choices to manage potential consequences of one choice over another [7, 9, 10]. This strategy has been used with a wide variety of topics within public health research, including adaptation of surveys of sex behavior to youth in Zimbabwe [11] and pediatric adherence items in Kenya [12]. This and related work has helped to position cognitive interviewing as a crucial part of survey development and adaptations in sub-Saharan Africa particularly for behavior that may be sensitive in nature and uncommonly openly discussed, like sexual behavior [2, 13, 14].

HIV risk and behavioral items that require respondents to aggregate across months of behaviors (eg., sex events over the past 3 months), estimate percentages (eg., percent of time condoms were used) or relative proportions (e.g., about how much of the time were condoms used) may also create challenges due to numeracy and cognitive demands inherent in abstract and summative exercises. For youth and young adults, who have unique social and developmental characteristics, assessing sex behavior and HIV-risk is particularly worthy of additional exploration especially in high HIV-incidence areas.

To explore the meaning youth make of commonly used sexual behavior and HIV-risk assessment items, we conducted cognitive interviews with young (18–24) women and men who have sex with men (MSM) in two high-incidence, highly researched areas in South Africa- Cape Town and rural KwaZulu-Natal (KZN). The Cognitive Interviewing Project focused on a variety of assessment items to explore comprehension, understanding, and self-presentation concerns, and collected recommendations for improvements. The full project included three rounds of cognitive interviews. Here we present the findings from items assessing sexual behavior and HIV-risk perception from our first round of interviews.

Methods

The Cognitive Interviewing Project (CIP) is a joint research project involving the University of Michigan, Michigan, USA; Center for the AIDS Programme of Research in South Africa (CAPRISA), Kwazulu-Natal; and the Desmond Tutu HIV Centre (DTHC), Cape Town, South Africa. The objective of CIP is to explore the meaning young women and MSM ascribe to commonly used survey items in HIV-research studies in high research areas. The first round of interviews focused on a set of items assessing HIV risk behavior and estimation of HIV risk taken from audio computer assisted self-interviews (A-CASI), CASI without audio, and interviewer collected tools from trials and research in the HIV Prevention Trials Network (HPTN) and local studies at CAPRISA and DTHC [15, 16]. Both CAPRISA and DTHC served as recruitment and implementation sites. Each are active clinical research sites, located in areas with ongoing HIV epidemics, with robust histories of engagement in large-scale HIV prevention and treatment studies, and have longstanding relationships with communities of youth who presently or have engaged in HIV-related.

Population and Procedures

Youth (ages 18 to 24) were recruited at outreach and site events, consented and engaged in one audio-recorded cognitive interview (see measures below). Inclusion criteria included: self-reported female or male sex, between the ages of 18–24 years, self-reported sexual activity with a male partner within the past 3 months, willing/available for a 60 min interview, willing to be audio recorded, and speaks English or local language (isiZulu or isiXhosa). Age and sex were recorded among those who enrolled. Although current or prior experience as a research participant was not an enrollment criteria, recruitment strategies (through site based activities and inquiring among youth attending visits for other studies) enriched for youth how had experience with study research procedures, such as interviewer or computer administered surveys focused on sexual health and behavior.

Enrollment targeted 15 cis-females from each of the two participating sites (total n = 30) and 10 cis-males identifying as MSM (total n = 20), for a total of 50 interviews. Interested youth were consented using IRB study procedures and forms. Interviews were conducted in private locations, recorded, transcribed and translated (to English). Participants were reimbursed 70 RAND (equivalent to slightly under $5.00 USD) and provided snacks during the interview. No identifying information (name or contact number) was retained after scheduling and completing the interview, and not collected at all for individuals interviewed immediately following consent. De-identified transcripts were shared with the coding team. Audio recordings, only retained locally for cross-checking transcription, were destroyed at the completion of the final round of data collection in CIP. All procedures and material were reviewed and approved by all participating site IRBs.

Interviewers and Training

A total of 5 interviewers (2 in Cape Town and 3 in KZN) were trained to conduct cognitive interviews, and each interviewer completed a series of mock interviews, for which feedback was provided before study launch. Neutral interviewing approaches were emphasized with multiple opportunities for practice.

Measures

Survey Questions and Responses

Items for cognitive interviewing included sets of items assessing perception of HIV risk, sexual behavior (events), partner types, and other items known to be associated with higher risk for HIV (transactional sex and age of first sex). Items and response options were selected to reflect commonly used risk assessment questions and answer choices, which we drew from measures used in a number of computer assisted surveys from various studies [15,16,17]. Item selection for inclusion underwent a series of core team and external expert reviews. For each set of items (assessing risk perception and general risk and assessing behaviors associated with elevated HIV risk), 5 primary items were selected, although some items had a series of follow up questions that were translated from English to Zulu and isiXhosa with cross-translation and refinement for language accuracy by the research team. Items within each set were then randomized to create unique interview guides that maintained the same order in terms of set presentation (risk perception questions followed by risk behavior questions), but within each set the 5 items had a random order of presentation to minimize potential order effects. Items and response options are detailed in the results.

Cognitive Interviewing

A semi-structured interview guide was used to assess concordance between item intent and meaning attributed to each item (including specific question, response options and any lead-in or explanations provided for or within the questions). The interviews were conducted in two phases with practice prior to each phase. In the first phase, participants were asked to use the “think aloud” process as they read and responded to each of the questions. Interviewers would provide guidance and support for articulating one’s thoughts during this process but would not probe for explanation or interpretations. After completing all 3 sets, the interviewer implemented phase 2, where participants returned to each question and responded to probes regarding: comprehension, retrieval, judgment and response. Comprehension reflected the participants’ understanding of the question, which included identifying the information being sought. Retrieval involved the way in which respondents recall information. Judgment encompassed the way in which participants summarize the information including determining relevance of memories and drawing inferences to provide a response. Response emulated the way participants report back the information, including mapping summarized information onto the response category. For participants reporting living with HIV at the time of the interview, their perceptions and recommendations for HIV risk-perception items were collected in phase 2 of the interview. These two main processes, (think-aloud and active-probing) are common cognitive interviewing strategies, often used in the improvement of survey items [18]. At the conclusion of the probing section of the interview, the participants were asked whether they would prefer to complete the survey as an in-person interview, or whether they believed using a tablet or computer would be better to address the survey items.

Analyses

De-identified translated transcriptions of each interview were analyzed by a trained team of three coders and the project PI using an adaptation [19] of framework analysis [20]. Content of discussions were first organized into frames containing responses to each item presented in the cognitive interview. These frames were then combined across the three distinct item sets (risk perception items, HIV risk behavior items and adherence items). Item sets were then iteratively reviewed to create a codebook identifying content themes within each set through thematic analysis [21, 22]. Identification of types of issues emerging in the areas of comprehension, retrieval, judgment and response formed the basis of summaries across items, followed by detailing item specific issues [23]. Coders met regularly and discussions of themes and refinement of them involved the full CIP team (all investigators, coders, and on-site interviewers). Coders met minimum criteria for applying the first framing pass (80% or greater agreement via the Dedoose program’s test sub-program). Consistency in application of thematic codes was achieved through team discussion to reach consensus on any identified discrepancies to ensure negotiated agreement [24]. Main themes were then further discussed, distilled as having enough saturation to suggest a main (versus less consistent) theme, and main themes were characterized with example quotes. Emerging content that presented opportunities for insights despite being less common in discourse were also considered, although these are presented separately.

Results

Sample

A total of 50 interviews were collected between July and November 2016 for round 1 of the cognitive interviews. Participants were an average of 21 years of age (standard deviation of 1.76), ranging between 18 and 24. About 2/3 were female (60%) per protocol, and 26 interviews were conducted at CAPRISA and 24 at DTHC. Most participants (82%) disclosed an HIV negative status, 8 (16%) disclosed living with HIV, and 1 (2%) interviewee’s HIV status was not clear. Most participants (73%) had current or recent experience with participating in HIV-focused research studies, of these the most common type of study (72%) was in biomedical prevention (eg., oral PrEP).

Themes Identified

Across items focused on risk perception and risk behavior, 5 main observations emerged (Table 1): (1) unclear and confusing terms or specific words; (2) language or phrasing of questions mismatched to local language or common in-group language creating misunderstanding or misinterpretation of questions; (3) confusion over and dislike of items that required self-tailoring (selecting the term that matched the person’s sexual practices or anatomy); (4) social desirability and presentation concerns; and (5) reflecting on partner’s behavior and trustworthiness when asked to estimate one’s own HIV risk, particularly for young women. Each theme is detailed with example quotes in Table 1.

Table 1 Themes (across items)
Full size table

By item, specific concerns and recommendations were identified (Table 2) during the think-aloud and active probing phases of the interview. Although all items had some potential for improvements per participant discussions, items varied in terms of whether the main concerns were word choices or phrasing and the extent to which social desirability concerns would erode accuracy in selecting responses. Items using mixed terms to allow for a single question to be used for females and males who have sex with men that required “self-tailoring” were generally disliked. Additionally, the manner in which participants assumed their answers would be interpreted by others as reflections on one’s personality or enduring traits was noted throughout, suggesting that response options would be driven in part by self-presentation concerns (what a given response would “say” about the person, their values or attributes).

Table 2 Item specific concerns
Full size table

Discussion

Results suggest that despite frequent use of items the same as, or similar to those we evaluated, discrepancies between intended and interpreted meaning and variability between participant interpretation of items may erode accuracy in data collection around HIV risk perception and behaviors. Issues identified largely centered on lack of clarity of language and phrasing, social desirability, and factors impacting confidence in one’s evaluation of their risk for HIV, such as partner trust or lack of access to information about what partners are actually doing that may subsequently elevate participant risk. Specific recommendations for changes to items and answer options were provided or extracted from discourse, which are the focus of subsequent rounds of cognitive interviewing in CIP and could be explored in large sample studies for potential impact on accuracy. Current results suggest potential promise in the following modifications and recommendations:

  • Move from open-field (enter an age or a number) to a range or numeric selection for counts or reports that evoke self-presentation concerns (e.g., provide age ranges for age of first sex that intentionally start below expected lowest age).

  • Avoid self-tailored or combined items and instructions (e.g., “meaning, a condom was on at all times when your penis/your partner’s penis was inserted in your anus or vagina/your partner’s anus or vagina”) and use sex and partner specific item sets (e.g., filter participants to appropriate items about insertive anal sex, receptive anal sex, insertive vaginal sex, or receptive vaginal sex separately).

  • Consider offering definitions of various sex behaviors only as needed (e.g., click a button or some other action to receive specific definition) versus provided to all to reduce long explanations.

  • Exercise caution when using “permission statements” (e.g., “It’s OK to take your best guess”) as these may be interpreted as permission to present in a favorable manner.

  • Consider shorter recall periods for youth when asking for counts of events (e.g., use 1 month rather than 3 months).

  • Work with local communities to better represent gradations of agreement and disagreement to use for response options and consider response scale modifications if gradations lack cultural relevance.

  • Consider alterations to the phrasing of HIV risk perception items (e.g., ask about general feelings of safety and/or ask about worry or concern around HIV) to lower self-presentation demands.

Even with these suggestions, validation in terms of comparison of item response to objective measures is require. While cognitive interviewing provides valuable information about comprehension and meaning or interpretation of items and response options, it is limited by not addressing concurrent or predictive validity of items. It is difficult to imagine strong validity for items that have substantial issues in comprehension or interpretation, however, it is possible to have items that raise some concerns but nonetheless perform well in predicting outcomes.

For well over a decade, single variable and predictive models of risk for HIV infection among MSM have used self-report variables (eg., condomless receptive anal intercourse, number of partners) [25,26,27,28]. Because reliable biomarkers of HIV-risk remain unavailable in most settings, [29] self-report, with its well-recognized inaccuracies, [29, 30] will likely continue to represent a key measure of HIV risk. As such, there is a critical role for cognitive interviewing to improve one of the most basic principles in survey development– a respondent should know what is being asked and what the answer choices mean [31].

It is additionally important to note that participants in the CIP were all young women or MSM from South Africa. Findings could lack generalizability, however observations from other studies conducted in sub-Saharan Africa do appear to be in line with our findings [11, 32,33,34]. The role of cultural context and self-presentation in understanding and responding to questions about anal sex among women [11], frustration with confusing items and challenges in estimating events and partners [32] as well as risk of HIV infection [33] has been identified in other work including other groups and countries. Results of reviews of sexual behavior assessments focused on adolescents, largely in the US [34], also support several of the situational and interpretation findings we identified. Thus, we do have a degree of confidence in the generalizability of the issues we identify as relevant to many young adult populations. However, concerted efforts to combine cognitive interviewing results from diverse populations may help in the creation of measures that have wide-scale appeal. Attending to local sample and community while also prioritizing harmonization of measures can be particularly challenging. The alternative of using surveys that have consistent items but variable interpretation across communities in multi-site trials and projects may present an even greater challenge in data interpretation.

Recent work to identify predictive risk measures for adolescent and young adult populations include the evaluation of the VOICE risk score [35] among youth in participating in HPTN 068 [36]. Despite its validated performance with adult women in Africa, the tool, which uses a mix of demographic (age), observed (sexually transmitted infection) and self-reported factors (eg., whether one’s partner has other partners, alcohol use), did not perform well in their sample of young women as a predictor of HIV infection or as a method to identify PrEP candidates [37]. The authors called attention to the need for additional work in the area of identifying more germane risk items, and further suggested that even if objective risk tools could be optimized, questions concerning perceptions of HIV-risk remain critically important for engaging youth in prevention. Questions and answers that could prompt discussion, screen for potential interest in new or better prevention strategies, or gauge one’s experience of HIV risk remain important to implementation and scientific understanding of risk perception dynamics. Our results from cognitive interviews with youth living in HIV endemic, highly researched areas provides some guidance for item and answer construction that can be used in ongoing efforts to develop valid risk assessments and screening tools. Moreover, attending to how participants receive, experience and react to questions often used in HIV-related research has considerable merit in its own right. Efforts to phrase items and offer response options or instructions that minimize non-essential content associated with negative, anxiety provoking or stressful reactions should be prioritized.

Conclusion

Cognitive interviewing to disentangle areas where HIV risk survey items may create confusion, variable interpretation or social demands is an important step in building accurate approaches to identify and monitor HIV risk. Based on the current results, revised items and additional probes are suggested and will be further evaluated in subsequent rounds of cognitive interviewing in the CIP. Developing risk perception items that explicitly recognize risks outside of one’s control (e.g., partner(s) behavior), use of number selection versus fill-in options, avoidance of terms that lack cultural relevance or create translation issues, and careful attention to potential misinterpretation of add-in definitions or permission statements are important areas for future consideration. Cognitive interviewing provides important information for item construction but does not ensure accuracy. Large sample evaluations of accuracy between objective behaviors and self-report for items that have demonstrated consistent and clear concordance through cognitive interviewing is needed.