Background

HIV-associated neurocognitive disorders (HAND) remain prevalent despite effective combination antiretroviral therapy (cART) (Clifford 2008). The current diagnosis of HAND requires impaired performance on at least two cognitive domains within a neuropsychological (NP) test battery (Antinori et al. 2007). While NP batteries yield comprehensive information, they require examiner expertise and are time-consuming. In the pre-cART era, batteries for the evaluation of HIV-associated dementia often required >3 h (Gendelman 2012). While testing to fulfill the most recent HAND diagnostic criteria can be achieved in less time, these modern batteries still typically require at least 1.5–2 h (Antinori et al. 2007) .

Conversely, while many abbreviated cognitive assessment tools have been studied in HIV-infected individuals, their overall performance is not optimal, particularly in cases of milder impairment. For example, the HIV dementia scale (HDS) is relatively sensitive in the context of demented HIV-infected patients who are hospitalized (Berghuis et al. 1999; Valcour et al. 2011). However, most individuals with HAND in the cART era are outpatients with milder cognitive impairment, and in this context, the HDS is not sensitive for the detection of neurocognitive impairment (Bottiggi et al. 2007). Other rapid tests, including the International HIV dementia scale (IHDS), the mini mental status exam (MMSE), and the Montreal cognitive assessment (MOCA), have also been shown to be relatively inaccurate for the diagnosis of HIV-associated neurocognitive impairment with sensitivity and specificity results in the 50–70 % range (Haddow et al. 2013; Overton et al. 2013; Skinner et al. 2009). Thus, there is an ongoing need for methods to assess neurocognition in HIV-infected individuals which are relatively accurate but can be performed in a condensed period of time without examiner expertise (Brouillette et al. 2015).

We performed a preliminary study in which a novel computerized cognitive assessment device (NCAD) was compared to traditional neurocognitive testing in order to gauge the NCAD’s promise as a diagnostic tool for HIV-associated neurocognitive impairment. The NCAD was first developed as a cognitive assessment tool through a partnership between Emory University and the Georgia Institute of Technology. In previous studies of elderly individuals without HIV infection, the device was shown to be sensitive to age-related cognitive decline with results that correlated closely with traditional NP testing (Wright et al. 2010; Wright et al. 2011).

Methods

Participants were enrolled between March 2011 and February 2015 at the Emory University Center for AIDS Research (CFAR) clinical core site in Atlanta, Georgia. Individuals with chronic HIV between 18 and 59 years of age were considered for participation. Individuals were excluded from the study for any of the following: (1) history of any neurologic disease known to affect memory (including stroke, malignancy involving the brain, traumatic brain injury, and AIDS-related opportunistic infection of the central nervous system); (2) ongoing substance use (marijuana use in the last 7 days OR cocaine, heroin, methamphetamine, or other non-marijuana illicit drug use in the last 30 days); (3) heavy alcohol consumption in the last 30 days (defined as >7 drinks per week for women and >14 drinks per week for men); or (4) serious mental illness including schizophrenia and bipolar disorder (depression was not excluded if participants were well controlled on treatment). Participants were also screened for hypothyroidism (with serum TSH level), vitamin B12 deficiency (with serum B12 level), and cryptococcal disease if CD4+ T cell count was <100 cells/μl (with serum cryptococcal antigen) and excluded if any were found to be abnormal. Participants with a history of treated syphilis and a persistently positive RPR titer of 1:4 or less were eligible for the study if there was a decrease in RPR of at least fourfold at 6 months after treatment and there were no neurological symptoms at initial syphilis presentation. Lastly, participants were excluded in the event that significant cognitive symptoms had occurred precipitously in the last 30 days in order for further medical workup to be undertaken. The study was approved by the Emory University Institutional Review Board, and written consent was obtained from all the participants.

In addition to the HIV dementia scale, the NP battery included the following eight tests that are used commonly in studies of cognition in HIV-infected participants (Robertson and Yosief 2014): (1) Trailmaking part A (processing speed); (2) Trailmaking part B (executive function); (3) Hopkins Verbal Learning Test-Revised (HVLT-R) total recall (verbal learning); (4) HVLT delayed recall (verbal memory); (5) grooved pegboard-dominant hand (fine motor); (6) Stroop color/word (executive function); (7) controlled oral word association test (verbal fluency); and (8) animal fluency (verbal fluency). These tests were selected in order to examine at least five domains as recommended in the most recent nosology of HAND criteria (Antinori et al. 2007). Additionally, the American National Adult Reading Test (AMNART) was administered to estimate pre-morbid function. The tests were administered by study staff after intensive training by an experienced, board certified neuropsychologist (DL).

For the purposes of comparisons with the NCAD (for which demographically corrected norms are not available at this time), scaled scores without correction for demographic characteristics were identified based on published literature (Heaton et al. 2004). As per the most recent nosology of HAND criteria (Antinori et al. 2007), neurocognitive impairment was considered to be present when the result from at least one test from two different domains was at least 1 standard deviation (SD) below the population mean (again, for the purposes of this study, uncorrected scaled scores were used). For the domains with two individual tests (executive function and verbal fluency), participants were judged impaired in that domain when results on at least one test was >1 SD below the mean. In addition, T scores were calculated by adjusting all individual tests for demographic characteristics using published norms (Heaton et al. 2004; Norman et al. 2011). A composite neuropsychological test score (NPT-8) was then calculated by averaging individual T scores. Global deficit score (GDS), a validated measure of HIV-associated neurocognitive impairment based on multiple neuropsychological tests, was also calculated based on T scores (Carey et al. 2004).

Each participant also underwent NCAD testing at the same visit. NCAD testing requires approximately 20 min. In brief, the participant wears a headset unit with video display and noise canceling headphones (see Fig. 1) and uses a handheld input piece with two buttons (one for each thumb). Before the assessment begins, the device provides a brief verbal explanation on how to use the handheld input piece. Depending on the subtest, the buttons signal “Yes” and “No” OR “Right” and “Left.” Therefore, each question in each subtest has two possible responses. The full NCAD assessment includes seven subtests that are administered sequentially with an automated voice-over (see Fig. 2 for visual representation). These subtests include 1. selective visual word recall (with two trials at the beginning of the full assessment and one delayed trial at the end of the full assessment), 2. sequential colored shapes (Yes/No if each shape is the same as the initial reference shape), 3. colored arrow direction (for blue arrow, the participant pushes the button the arrow is pointing towards, for red arrow, the participant pushes the button the arrow is pointing away from), and 4. faces (first one-back sequence and then two-back sequence). Each subtest is a modified version of an established NP test (complex symbol reaction, selective reminding memory, go/no-go executive function, and N-back working memory) (Wright et al. 2010). These NCAD tests cover multiple domains, including processing speed, episodic memory, working memory, and executive function. Results from each participant are stored automatically in a de-identified computer database. The NCAD predictive score is a proprietary 10-item scale that incorporates the individual subtests and reflects the probability of impairment from 1 (normal/no impairment) to 10 (definite impairment). The algorithm that generates the predictive score was derived from a multivariable predictive ordinal regression model and validated using a tenfold cross-validation approach in a cohort of approximately 400 participants (Wright et al. 2011). In the current study, the predictive score was calculated if all subtests were completed and registered in the computerized database.

Fig. 1
figure 1

Novel cognitive assessment device (NCAD) apparatus

Full size image
Fig. 2
figure 2

Summary of NCAD subtests

Full size image

Normality was assessed using the Shapiro-Wilk test. Pearson correlation (r) was used for variables with normal distribution by Shapiro-Wilk while Spearman correlation (rho) for variables with non-normal distribution by Shapiro-Wilk. All statistics, including sensitivity/specificity and area under the curve, were calculated using SAS 9.4 and GraphPad Prism 6.

Results

Thirty-nine HIV-infected participants were enrolled for this study. Demographic/disease information is presented in Table 1. The median age was 48 years, and the participants were 79.5 % male and 64.1 % African-American. Median number of years since HIV diagnosis was 17 (interquartile range 7–22). The majority of the participants (74.4 %) were receiving cART, and the majority of the participants (71.8 %) had undetectable plasma HIV RNA levels (defined as <40 copies/ml). Median CD4+ T cell count was 354 cells/μl, and median CD4 nadir was 34 cells/μl. Only one participant had a positive RPR titer (1:2) which was chronic in the setting of a distant history of syphilis treatment. Four participants had confirmed hepatitis C virus (HCV) co-infection by RNA detection in blood, and four participants had confirmed hepatitis B virus (HBV) co-infection by detectable HBV surface antigen in blood. Other common co-morbidities included cigarette smoking, hypertension, depression (controlled on treatment), diabetes mellitus, and peripheral neuropathy. Of the 29 participants on cART, the most common regimens were as follows: 1. efavirenz/tenofovir/emtricitabine (6), 2. atazanavir/ritonavir/tenofovir/emtricitabine (5), and 3. atazanavir/ritonavir/ abacavir/lamivudine (3). Median years of education was 14 (range 9–18), and mean AMNART score after adjusting for demographic variables was 90.89 (SD = 7.17).

Table 1 Demographic and disease characteristics
Full size table

On NP testing, mean HIV dementia scale was 12.79, mean NPT-8 score was 46.07, and mean GDS was 0.59. The mean total subtest accuracy (correct answers divided by total answers) on the NCAD was 72.46 %. NCAD total subtest accuracy correlated significantly with NPT-8 (Pearson correlation r = 0.59, p < 0.0001, Fig. 3) as well as GDS (Spearman’s rho = −0.36, p = 0.02). Twenty-nine participants had available results on all NCAD subtests and therefore had NCAD predictive score results available (ten participants had one missing subtest result due to error in automated data upload after test administration and therefore could not have predictive score calculated). Mean NCAD predictive score for these participants was 2.59. NCAD predictive score also correlated significantly with NPT-8 (Spearman’s rho = −0.5601, p = 0.0016) and GDS (Spearman’s rho = 0.45, p = 0.0144).

Fig. 3
figure 3

Correlation between NCAD total subtest accuracy and NPT-8 composite score

Full size image

In terms of clinical measures, current CD4+ T cell count was not significantly associated with average subtest accuracy (Spearman’s rho 0.17, p = 0.31) or with NCAD predictive score (Spearman’s rho −0.21, p = 0.27). Current CD4+ was also not significantly associated with any of the NCAD subtests. There were also no significant associations between nadir CD4+ T cell count and average subtest accuracy, individual subtest accuracy, or NCAD predictive score.

Twenty-seven out of 39 participants (69.2 %) met criteria for neurocognitive impairment given the presence of at least two domains with scores >1 SD below the mean. For the detection of neurocognitive impairment, receiver operator curves (ROC) were calculated for HIV dementia scale, NCAD total subtest accuracy, and NCAD predictive score. For HIV dementia scale, a cutoff of >9.0 yielded a sensitivity of 85.19 %, but area under the curve (AUC) was not statistically significant at 0.508 (p = 0.930). In contrast, the AUC for NCAD total subtest accuracy was 0.7562 (p = 0.012). A <75.44 % cutoff for NCAD total subtest accuracy yielded the most balance between sensitivity (66.67 %) and specificity (83.33 %) for the detection of neurocognitive impairment (see Table 2 for the range of sensitivity/specificity values). For the NCAD predictive score (n = 29), the AUC was 0.7381 (p = 0.051) for detection of neurocognitive impairment. A >2.015 cutoff value on the NCAD predictive score yielded the most balance between sensitivity (66.67 %) and specificity (75.00 %).

Table 2 Corresponding sensitivity/specificity results for NCAD total subtest accuracy detection of neurocognitive impairment. Cutoff value in bold corresponds to highest combined sensitivity and specificity
Full size table

Discussion

In this preliminary study, we found that performance on a novel computerized cognitive assessment device (NCAD) correlated well with standard NP testing and had promising accuracy for the diagnosis of neurocognitive impairment in HIV-infected individuals. Given the challenges involved with administration of a full neuropsychological test battery, there is an unmet need for rapid, efficient tools for neurocognitive assessment in HIV-infected individuals, who continue to show high rates of HAND. This new device has a number of advantages as a screening tool, including automation that minimizes test administration variability and allows for immediate digital storage and interpretation of data.

We acknowledge the preliminary nature of this study, and that a larger study is needed to better characterize the performance of the NCAD for the detection of HAND. Many participants in our study had a long history of HIV or had low current/nadir CD4+ T cell counts. It is not clear if our results would be similar in a population with a short history of HIV and without a history of significant immunocompromise. There were no significant associations between current/nadir CD4+ and NCAD results, but given the small sample size of the study, there is a risk of type II error when examining these associations.

We did not obtain formal measures of functional status in our participants. Given that assessment of functionality is needed in order to make an official determination of HAND status (Antinori et al. 2007), we are unable at this time to report on the utility of the NCAD for this purpose. The NCAD has not been validated in populations who are not primarily English speakers, though there is potential for the device to be adapted to such groups. Also, demographically corrected scores (accounting for age in particular) are not available at this time for the NCAD, and it is possible that our results could change significantly if NCAD results were corrected for demographics. The NCAD was originally studied in older individuals as a way to detect dementia. Unfortunately, there is not currently sufficient NCAD testing data in younger individuals to use as reference for our study, in which the median age was 48 years. This is a limitation of the current study. Ideally, an HIV-specific predictive score based on a large sample of HIV-infected individuals (including individuals both with and without neurocognitive impairment) would be available to interpret NCAD scores compared to an HIV-negative population with similar demographics. A much larger study would be required to generate these data.

While the device allows for eyeglasses, it requires adequate vision as well as the ability to hold and operate the keypad. Thus, individuals with severe visual impairment not ameliorated with visual aids or who have severe impairment in manual dexterity are poor candidates for assessment with the NCAD. As detailed in the results, there were ten participants with one missed subtest result, and for these participants, there was no predictive score (which is calculated based on subtest results) available. Based on quality assessment, this issue arose from lack of automated data upload in the case of those particular subtests. This hindered our ability to fully assess the diagnostic accuracy of the NCAD predictive score, for which the AUC was similar to NCAD total subtest accuracy but p value was just above 0.5, possibly due to lack of power. Further technical refinements to ensure reliable data upload have been made with the next generation of the device, which is available for new research studies (now known as integrated display enhanced testing for cognitive impairment and traumatic brain injury or iDETECT) (LaPlaca 2015).

While the NCAD is not yet in commercial production, it is available to the investigator completely assembled. Aside from the limitations of the device in terms of need for relatively intact manual dexterity, vision, and hearing, the device is straightforward for clinic staff to administer. Other than checking with the patient at the beginning of the test to make sure that the headpiece is comfortable and that the patient is able to clearly see the screen and hear the audio, there is no other need for the clinic staff. The entire test is automated (including the directions for the patient), and at the completion of the test, the results are automatically uploaded. Individual de-identified scores (including composite predictive score) are immediately available for review on a password-protected website. Again, since the NCAD is not yet in commercial production, it is not clear at this time what revenue model will be used by the manufacturer. However, affordability is a strong consideration as is financial assistance for resource-limited settings.

Lastly, given that not all domains are tested (such as verbal memory), the NCAD is not designed to be a replacement for a comprehensive neuropsychological battery. However, it has potential as a more accurate, more automated screening method. Further study with the NCAD that includes evaluation of functional status is needed to determine if this device could be useful for the assessment of HIV-infected individuals in broader clinical settings.