Introduction

With the development of combination antiretroviral therapy (cART), the prevalence of HIV-associated dementia (HAD) has declined but less severe HIV-associated neurocognitive disorders (HAND) have become prominent [1]. The recognition of subtler manifestations of HIV on cognition, including mild neurocognitive disorder (MND) and asymptomatic neurocognitive impairment (ANI), proves to be challenging in a busy clinic environment, particularly given the ability of individuals to compensate for mild impairment. While these latter entities are more difficult to recognize than HAD, timely diagnosis is critical since even ANI can impair capacity to adhere to medical regimens, safely operate a motor vehicle, complete basic activities of daily life, and maintain employment [2–5]. Furthermore, with the transformation of HIV into a chronic medical illness, advanced age and HIV infection may act synergistically to increase the prevalence of HAND [6]. Recent cohort studies have reported the prevalence of HAND to range from 39 to 69% of subjects on cART [7, 8]. The need for brief and psychometrically sound methods to evaluate neurocognitive function is important to identify HAND and limit progressive impairment in one’s capacity to complete activities of daily life independently [2–5].

Some available screening tools used to diagnose neurocognitive impairment, such as the International AIDS Dementia Screen or the Mini-Mental Status Exam, are insensitive to early manifestations of functional impairment [9–11]. Conversely, formal neurocognitive testing is time-consuming and requires special training and thus, cannot realistically be used in the typical outpatient setting, where the necessary time, staffing, space, and funding are generally not available. CogState, a commercially available product (http://www.cogstate.com), is a computerized cognitive test battery designed to measure psychomotor performance, attention, memory, and executive functioning: domains frequently impaired in persons with early neurocognitive disorders [12, 13]. The battery consists of brief tasks in the form of card games to minimize language and cultural differences. It has previously been validated in persons with HAD [13] and been used in clinical cohort studies to measure change in neurocognitive function and performance [14]. In this analysis, we sought to determine the correlation between CogState and formal neuropsychological testing to detect neurocognitive impairment in a subset of healthy, autonomous HIV-infected persons who are followed in two cohort studies at the Washington University Outpatient HIV Clinic.

Methods

Washington University in St. Louis is a site for two prospective cohort studies evaluating complications of HIV and HIV therapy including cognitive function, one with a validated traditional neurocognitive testing battery and the other with CogState. To determine the utility of the CogState computerized battery as a screening tool, subjects enrolled in both studies were identified to compare the two neurocognitive batteries. Subjects were eligible for participation in this study if they completed an assessment in CHARTER within 6 months of the baseline SUN assessment. Enrollment criteria for the studies have been outlined previously [15, 16]. Both studies were approved by the Washington University IRB and all subjects provided written informed consent.

SUN Study

The Study to Understand the Natural History of HIV and AIDS in the Era of Effective Therapy (The SUN Study) is a CDC-funded multi-site prospective observational cohort monitoring complications of antiretroviral therapy and HIV [15]. At baseline and each 6 month visit thereafter, participants are evaluated for neurocognitive ability using CogHealth© CogState Ltd., Melbourne, Australia [13]. The computerized battery requires 12–15 min to complete and consists of six individual tests: two tests of detection evaluating simple reaction times (DET1 and DET2) assess psychomotor function and speed of processing domains; a test of identification as a choice reaction time (IDN) assesses the domains of visual attention and vigilance; the one back test (ONB) assesses the domains of attention and working memory; the monitoring test is a measure of divided attention (MON) that assesses the domain of attention; and the associate learning test (ASSL) assesses the domain of visual learning and memory. Each test is scored based on time to complete the task (speed) and error rate (accuracy).

CHARTER Study

The CNS HIV Antiretroviral Therapy Effects Research (CHARTER) study is an NIH-funded multi-site cohort to explore HIV neurological complications in the context of emerging antiviral treatments such as cART [16, 17]. Participants receive comprehensive neuromedical, neurocognitive, and laboratory examinations. The neurocognitive battery performed in CHARTER requires approximately 1 h to assess the following domains: attention/psychomotor speed (Trailmaking Test Part A, Symbol Digit Test and Symbol Search Test from WAIS-III, Paced Auditory Serial Addition Task); fine motor speed skills (Dominant and Non-dominant Hand Pegboard Test); learning and memory (Brief Visuospatial Memory Test–Revised, Hopkins Verbal Learning Test–Revised, Figure Memory Learning Test and Story Memory Learning test); executive functioning including working memory (Wisconsin Card Sorting Test, Letter-Number Sequence from WAIS-III), fluency (Controlled Oral Word Association Test and Animal Category Test), and set shifting/response inhibition (Trailmaking Test Part B) [18–30].

Statistical Analysis

Data from the computerized battery were evaluated for normality of data distribution; reaction time measures were log10 transformed due to a positive skew of the distribution and accuracy measures were transformed using Arcsine-root transformation [31]. The raw scores from the neurocognitive tests from CHARTER were converted into T scores corrected for demographic data to minimize the impact of education, age, race/ethnicity, and sex. The Global Deficit Score (GDS) is a composite score calculated by converting the T scores from the CHARTER battery into one summary deficit score ranging from 0 (normal) to 5 (severe neurocognitive impairment) [32]. For the evaluation of construct validity, a correlation matrix was created to determine the correlation between the CogState measures and the CHARTER study standard neuropsychological measures, including the GDS, using Pearson’s correlation coefficient.

To determine the potential value of the CogState battery as a screening tool, we performed a stepwise linear regression analysis with GDS (log transformed to approximate a normal distribution) as the dependent variable and the CogState measures as independent variables. The stepwise regression was performed using the default selection method (forward in, backward out) in SAS. The stepwise regression was validated using 1,000 bootstrapping replicates. The significant Cogstate variables from the stepwise regression were used as independent variables in a multivariable regression model with GDS as the dependent variable to determine the model R2. The Receiver Operating Characteristic (ROC) curve was produced to determine the sensitivity and specificity of the combination of the significant factors identified from the stepwise regression model for predicting whether an individual’s GDS was greater than or less than 0.5, the threshold for neurocognitive impairment.

Results

Forty-six subjects enrolled in both cohort studies were eligible for the present analysis. Table 1 outlines the clinical parameters of the subjects at the time of baseline CogState evaluation between 2004 and 2006. Median CD4 count was 424.5 c/mm3 with 74% of the cohort on cART and 61% with HIV viral load <400 cp/ml. The median GDS, as calculated from the CHARTER study, for the cohort was 0.47 (range 0.00–2.79), with 24 subjects (52%) having normal function (GDS < 0.5) and 22 (48%) having mild to moderate impairment (GDS ≥ 0.5).

Table 1 Baseline demographics of study participants, the study to understand the natural history of HIV and AIDS in the era of effective therapy (The SUN Study) and the CNS HIV antiretroviral therapy effects research (CHARTER) study
Full size table

The correlation matrix showing the Pearson product-moment correlation coefficient between each COGSTATE index and the dependent variables from the CHARTER clinical exam, including GDS is shown in Table 2. The speed measures for both simple detection tests (DET1, DET2) and the identification task (IDN), a more complex reaction time task, correlated with the GDS and had the highest levels of correlation across the tests performed in the CHARTER battery. The accuracy components of the CogState battery generally correlated poorly with the individual neurocognitive tests in the CHARTER battery. The accuracy measures for the complex tasks of One-Back Memory (ONB) and the Measure of Divided Attention (MON) were the only accuracy measures that correlated with the GDS derived from the CHARTER testing.

Table 2 Correlation matrix between CogState tests and CHARTER neurocognitive tests
Full size table

The stepwise regression analysis identified the following potential independent correlates with GDS: the accuracy and speed of the two simple detection tests (DET1lmn, DET2acc) and three more complex tasks: the accuracy of the associate learning (ASSLacc), the accuracy of monitoring tasks (MONacc), and the accuracy of the One Back test (ONBacc) (see Fig. 1). A regression model using GDS as the dependent variable and these measures as independent variables yielded a model R2 of 0.53 (P < 0.0001) indicating that approximately 53% of the variance in the GDS is explained by these five CogState variables. In the validation analyses using results from the stepwise regression analysis of 1,000 bootstrap replicates MONacc appeared in 96% of the replicates, DET1lmn appeared in 76% of the replicates, DET2acc appeared in 71% of the replicates, ASSLacc appeared in 57% of the replicates and ONBacc appeared in 51% of the replicates. A regression model using GDS as the dependent variable and MONacc, DETlmn and DET2acc (the three variables that appear in more than 70% of the replicates) yields a model R2 of 0.39 (P = 0.0002).

Fig. 1
figure 1

Correlation between parameters from the CogState computerized battery. Correlation between different components of the CogState battery and GDS from CHARTER testing was determined by Pearson product-moment correlation coefficient. The five components displayed here were included in the final multivariate regression model

Full size image

To further explore the utility of the composite score derived from the computer battery tests to detect cognitive impairment based on GDS, we performed an area under ROC analysis using the combination of five significant test parameters identified from the multivariate regression model. The area under ROC curve for the specified model, including a composite of ASSLacc, ONBacc, MONacc, DET1lmn, and DET2acc was 0.90 (95% CI: 0.81–0.99, P < 0.0001; Fig. 2).

Fig. 2
figure 2

Receiver operating characteristic (ROC) curve of five parameters from the CogState computerized battery for Global Deficit Score from CHARTER. CHARTER refers to the area under the curve for the CogState composite score to predict dichotomized GDS score was 0.90 (95% CI 0.81–0.99)

Full size image

Discussion

In this study, we compared a brief, self-administered computerized screening battery with formal neurocognitive assessment. The best correlations were with the simple reaction tests, which measure functional speed as represented numerically in the correlation matrix and visually in the scatterplots. Cognitive slowing is a prominent feature of HAND, and thus, these results are consistent with previous comparisons of the CogState battery to formal neurocognitive functioning although the associations are not as robust for less severe cognitive impairment [13]. With the exception of reaction tests, the other individual tests in the computerized battery correlated poorly with formal neurocognitive testing.

While these results illustrate that a single brief screening test may be insensitive to identifying people with neurocognitive impairment, particularly those with mild impairment, the modeling of several parameters from the CogState battery against the Global Deficit Score highlights that there is utility to a brief computerized neurocognitive screening tool. Notably, the use of the five significant test parameters from the regression analysis to create a composite score correctly classified 90% of individuals as cognitively impaired or not. While these results require additional validation, they suggest clinical applicability for such a computerized battery. Future prospective studies are needed to determine the sensitivity of the CogState battery to identify HAND. Identifying persons with either ANI or MND with a relatively simple screening tool is particularly important for prevention to be effective; the early recognition of cognitive impairment will provide opportunities for timely intervention as well as ongoing evaluations for progression and the need for additional services to manage activities of daily life. If the screening tool proves to be sensitive with established valid cutpoints, this battery could be used to identify persons with mild cognitive impairment. However, in the absence of cutpoints that denote clinical significance, the battery will not achieve optimal clinical utility.

We recognize the limitations of this analysis. It consists of a small group of individuals and lacks a control group. No persons with advanced cognitive impairment were included. Another limitation to these findings was the lack of correlation between the Associate Memory test and the measures of learning and memory. The CogState battery evaluated here was insensitive to these measures and thus may impact the ability of Cogstate to identify persons with impaired learning ability. Additionally, there was a lack of correlation between CogState indices and the Trailmaking Part B test, which is widely used to reflect executive function with important implications regarding one’s ability to perform activities of daily living independently [33]. Our data fail to confirm the previous work by Cysique et al. [13], in which the strongest correlations were with the Trail Making tests. However, their work focused on persons with HAD. While the Trailmaking Part B test does not necessarily capture all of the components of executive control important for daily living, our findings suggest that CogState may not provide critical information related to early decline in functional independence, though it should be noted that executive function is a heterogeneous construct.

In summary, we found that a compilation of the tests from a brief computerized screening tool for neurocognitive function was correlated to traditional neurocognitive testing among HIV-infected persons and a composite score of five parameters could discriminate cognitive impairment. These findings confirm previous reports of correlation between brief computerized CogState battery and standard neuropsychological examination [13], especially for identifying cognitive slowing, a central feature of HIV-associated neurocognitive disorders, although there were differences regarding different domains potentially related to differences in the severity of impairment in the studies [34]. Nevertheless, additional research is needed to fully evaluate the utility of this battery, specifically to assess the composite score based upon the significant parameters identified in the regression analysis.