Introduction

Since the introduction of highly active antiretroviral therapy (HAART), human immunodeficiency virus (HIV)-associated dementia has markedly declined (Ances and Ellis 2007). However, subtle forms of neurocognitive impairment have become more prominent. A recent multi-site study demonstrated that 52 % of HIV-infected adults had neurocognitive impairment (Heaton et al. 2011). These neurocognitive changes can greatly impact day-to-day functioning and cause significant morbidity and mortality (Vivithanaporn et al. 2010). The continued prevalence of neurocognitive impairment in the HAART era may reflect prolonged patient survival (Valcour et al. 2011a, b), chronic inflammation due to the inability of certain HAART regimens to adequately cross the blood–brain barrier (Cysique et al. 2011), and/or medication-induced neurotoxicity (Marra et al. 2009).

To adequately evaluate the extent of neurocognitive impairment, a battery of neuropsychological tests is typically administered (Heaton et al. 2010). However, these neuropsychological tests are neither cost-effective nor time-efficient in the outpatient clinical setting as they often require several hours to complete and can be labor intensive, requiring additional trained personnel to administer and correctly score (Koski et al. 2010; Robinson-Papp et al. 2009).

A number of relatively simple tests have been developed to assess neurocognitive impairment in the office (Valcour 2011). The HIV Dementia Scale (HDS) consists of a brief series of tests that evaluates motor speed, memory, constructional praxis, and executive function (Davis et al. 2002; Power et al. 1995). The original 16-point HDS was modified for evaluations in international settings with limited resources (Lawler et al. 2010). The resulting International HIV Dementia Scale (IHDS) is scored out of a maximum score of 12, with less than 10 indicating dementia (Joska et al. 2011). While the HDS and IHDS can adequately identify more severe forms of neurocognitive impairment, specifically HIV-associated dementia, these tools are relatively insensitive in differentiating milder forms (Valcour et al. 2011a). In addition, these screening tests are influenced by the education level of the person (Waldrop-Valverde et al. 2010). Another method that evaluates multiple cognitive domains is a condensed version of formal neuropsychological testing in which scores from eight screening tests are combined into a single neuropsychological performance called the neuropsychological z score (NPZ)-8 (Clifford et al. 2002). This battery creates an aggregate score by averaging z scores from the eight different tests after correcting for age and education. This approach has been used extensively in clinical trials for ease of data management, but thresholds to identify neurocognitive impairment have not been established (Schifitto et al. 2006).

Screening tools that have been used to assess neurocognitive impairment in other neurodegenerative disorders (e.g., Alzheimer’s disease (AD), Parkinson’s disease, and Huntington’s disease) may be useful in the HIV clinic as many patients are living longer due to HAART (Wendelken and Valcour 2012). In particular, the AD-8 is an eight-item informant-based screen that reliably discriminates cognitively normal persons from demented persons, even at a very mild stage, and is sensitive to the earliest signs of neurocognitive impairment as reported by the individual person or an informant (often a spouse or life partner) (http://alzheimer.wustl.edu/About_Us/PDFs/AD8form2005.pdf; Galvin et al. 2005, 2006). The items focus on the individual’s current and previous level of functioning and attempt to determine changes attributable to neurocognitive impairment (Galvin et al. 2007). An individual serves as his/her own control, and this eliminates the need for preceding baseline evaluation. The AD-8 takes less than 2 min to complete and can be performed in the clinic waiting room (Galvin et al. 2010). The person or his/her informant is asked to report changes in memory, problem solving, orientation, and daily activities.

Another screening test that has been used by the AD community is the Montreal Cognitive Assessment (MoCA). The MoCA takes approximately 10 min to complete and consists of 30 items that encompass eight cognitive domains (Nasreddine et al. 2005). The MoCA is in the public domain, has been translated into multiple languages to facilitate global clinical application, can be administered by lay personnel, is easy to score, and requires a simple correction for years of education (www.mocatest.org; Valcour et al. 2011a). It is more sensitive than the mini-mental status exam in differentiating neurocognitive impairment (Freitas et al. 2011; Paul et al. 2011). However, few studies have examined the utility of the MoCA in HIV-infected populations (Koski et al. 2010; Chan et al. 2012).

In this study, we evaluated the utility of these two brief screening tools (AD-8 and MoCA) in assessing neurocognitive impairment within a cohort of 200 HIV-infected persons seen in the infectious disease outpatient clinic of a single institution. In particular, we assessed the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of the AD-8 and MoCA compared to the more formal condensed neuropsychological performance testing using the NPZ-8.

Methods

Study population

A total of 200 HIV-infected persons aged 18 to 65 years were enrolled at the outpatient infectious disease clinic at Washington University in St. Louis. All persons were on HAART and were virologically suppressed (plasma HIV viral load <400 copies/mL). Each person provided a written consent approved by the Human Research Protection Office at Washington University in St. Louis. Persons with a history of confounding neurological disorders including epilepsy, stroke, head injury with loss of consciousness greater than 30 min, history of opportunistic central nervous system infection, brain tumor, AIDS-defining opportunistic infections within 45 days prior to study entry, major psychiatric disorders, or active substance abuse were excluded from participation. Demographic, clinical, and laboratory data were extracted from the clinic’s electronic medical record. Specifically, information regarding past medical history was gathered by medical chart abstraction and by self-report. Depression screening was performed using the PHQ-9, a standard screening tool for depression (Kroenke et al. 2001).

Neuropsychological performance evaluation

All persons completed a condensed battery of neuropsychological tests. This battery included the Trail Making tests A and B, the revised Hopkins Verbal Learning Test (HVLT-R), the Stroop naming and color interference task, the Timed Gait test, Grooved Pegboard for dominant and nondominant hands, CalCAP Choice and Sequential Reaction Time tests, and the digit symbol substitution test (Clifford et al. 2002). These tests examine multiple cognitive domains (including learning and memory, psychomotor speed, motor skills, and executive function) and have previously been used to assess neurocognitive impairment (Heaton et al. 2010; Van Gorp et al. 1989). Raw scores from each test were standardized using demographically (age, gender, race, and education) adjusted normative means (Heaton et al. 2011). A standardized z score was calculated by subtracting the appropriate normative mean from the raw score and then divided by the normative standard deviation. A person was identified as having neurocognitive impairment if they had impairment in cognitive functioning in at least two domains as documented by z scores ≤−1.0 (Antinori et al. 2007). The NPZ-8 was calculated from the average z scores from the following eight tests (CalCAP Choice and Sequential Reaction times, Trail Making A and B, digit symbol substitution test, Timed Gait, and Grooved Pegboard nondominant hands, and HVLT-R delayed component).

Montreal Cognitive Assessment

Each person completed the MoCA with scoring performed according to published instructions. The MoCA consists of 13 tasks organized into eight cognitive domains including visuospatial, executive, naming, memory, attention, language, abstraction, delayed recall, and orientation. A total score was generated by summing scores across eight domains. One point was added for persons with 12 or fewer years of formal education (Nasreddine et al. 2005). The maximum possible score is 30 points, with a score <26 considered as having neurocognitive impairment.

Alzheimer’s disease-8

Each person also completed the AD-8. This test consists of eight ‘Yes/No’ questions (repeats self (memory), reduced interest in hobbies and activities (apathy), trouble with judgment (executive), trouble operating appliances (executive), forgets correct month/year (orientation), trouble with finances (executive), forgets appointments (memory), daily problems with memory/thinking (general). The AD-8 yields a score from 0 to 8, depending on the number of positive responses. A score of ≥2 indicates neurocognitive impairment (Galvin et al. 2005).

Data analysis

Baseline parameters for the neurocognitive impairment and cognitively normal persons, based on the criteria formulated by Antinori et al. (2007), were compared for possible differences using Student’s t test, Chi-square test, or Mann–Whitney U tests. The following measures of binary classification were generated to evaluate the utility of the two screening tools (MoCA and AD-8) compared to formal testing: sensitivity (the proportion of actual positives correctly identified), specificity (the proportion of actual negatives correctly identified), positive and negative predictive values (PPV, NPV) (measures of precision) and receiver operating characteristic (ROC) curves (a measure of the trade-off between sensitivity and specificity) (Metz 1978; Zweig and Campbell 1993). The Pearson product–moment correlation was obtained by comparing the NPZ-8 with the AD-8 and MoCA (SPSS version 18, Chicago, IL, USA). We also evaluated different cutoff values for the MoCA to optimize sensitivity as has previously been performed for AD (Lee et al. 2008).

Results

The mean age of the 200 HIV-infected persons was 43 years, with 72 % being male and African–Americans comprising the largest racial/ethnic group (67 %). The median CD4 cell count was 538 cells/mm3 (IQR 361, 695) with all participants virologically suppressed on HAART (<400 copies/mL). Using formal neuropsychological testing, 127 of the 200 persons (64 %) met the criteria for neurocognitive impairment (z score ≤−1.0 in at least two domains). Neurocognitively impaired persons were more likely to be African–American, have lower education, currently use tobacco, and have higher total cholesterol and low-density lipoprotein (LDL) compared to cognitively normal persons. Relevant demographic and clinical characteristics of the cohort are summarized in Table 1.

Table 1 Baseline demographics of HIV-infected (HIV+) individuals
Full size table

The MoCA identified 101 of 200 persons (51 %) as neurocognitively impaired, while the AD-8 identified 113 of 200 persons (57 %) with neurocognitive impairment. Using the MoCA, 80 of the 127 neurocognitively impaired persons were identified by formal testing, and 52 of the 73 cognitively normal persons were correctly identified, yielding a sensitivity of 63 %, a specificity of 71 %, a PPV of 79 %, and an NPV of 53 % (Table 2). When comparing the AD-8 to the formal testing results, 77 of the 127 neurocognitively impaired persons and 37 of the 73 cognitively normal persons were correctly identified, yielding a sensitivity of 61 %, a specificity of 51 %, a PPV of 68 %, and an NPV of 43 % (Table 3).

Table 2 Comparison of the MoCA to formal neuropsychological testing
Full size table
Table 3 Comparison of the AD-8 to formal neuropsychological testing
Full size table

Evaluation of the two tests using ROC curves demonstrated that the MoCA differentiated neurocognitive impairment from normal cognition better than the AD-8 when compared with a condensed neuropsychological battery. The area under the curve with 95 % confidence interval (CI) was 0.67 (0.59, 0.75) for the MoCA and 0.56 (0.47, 0.64) for the AD-8 (Fig. 1). These values are considered fair and poor, respectively, by a recent review of prognostic indicators (Yourman et al. 2012).

Fig. 1
figure 1

Receiver operator characteristic (ROC) curves for the Montreal Cognitive Assessment (MoCA) and Alzheimer’s disease-8 (AD-8) for the identification of neurocognitive impairment. The MoCA had a significantly greater area under the curve (0.67; 95 % CI (0.59–0.75)) compared with the AD-8 (0.56; 95 % CI (0.47–0.64))

Full size image

Both brief screening tests correlated with neurocognitive impairment as measured by the NPZ-8. In particular, the MoCA was positively correlated with the NPZ-8 (R = 0.65, p < 0.001) and accounted for 43 % of the variation in NPZ-8 scores, while the AD-8 score was negatively correlated with the NPZ-8 (R = −0.29, p = 0.001) and accounted for only 8 % of the variation in the NPZ-8 scores.

To improve the utility of the MoCA as a screening tool for HIV-associated neurocognitive disorders (HAND), we evaluated the sensitivity, specificity, PPV, NPV, and ROC with different thresholds for detecting neurocognitive impairment (23, 24, 25, 26, and 27). Data to evaluate the different cutoffs are displayed in Table 4. By lowering the threshold to 23, we reduced false-positive tests (i.e., improved specificity) at the cost of sensitivity. We found that raising the cutoff to 27 for neurocognitive impairment for the MoCA increased the sensitivity to nearly 90 % for identifying impairment. This marked improvement in sensitivity was accompanied by an expected loss of specificity. Interestingly, the areas under the ROC curves for these various MoCA threshold values (23, 24, 25, 26, or 27) were not notably different with overlapping 95 % CIs (Table 4).

Table 4 Evaluation of neurocognitive impairment for MoCA cutoff scores
Full size table

We also examined whether a combination of both tests was better in discriminating neurocognitively impaired persons than conducting either of the tests alone. A combination of the MoCA and AD-8 identified 51 of 127 cognitively impaired persons and 60 of 73 cognitively normal persons for a sensitivity of 40 %, specificity of 82 %, PPV of 80 %, and NPV of 44 %. The lower sensitivity of this combined approach likely reflects the fact that the two tests were weakly correlated (R = 0.140, p = 0.048).

Discussion

Among a group of HIV-infected persons who were virologically suppressed with HAART, neurocognitive impairment was highly prevalent (64 %) when determined by a condensed battery of formal neuropsychological performance testing. Neurocognitive impairment was more prevalent within HIV-infected persons of African–American race and persons with lower educational attainment, higher cholesterol levels, and currently using tobacco. Both the MoCA and the AD-8, brief screening tests originally developed for AD, correlated with formal testing, but neither was particularly sensitive as a screening tool (63 % for the MoCA and 61 % for the AD-8). The combination of tests fared no better.

The relatively high proportion of persons identified with neurocognitive impairment in this study at a single site was similar to previous reports (Tozzi et al. 2007; Robertson et al. 2007). Others have identified a lower prevalence (Heaton et al. 2011; Cysique et al. 2004), reflecting either differences in the cohorts studied or neuropsychological battery used. For instance, individuals with significantly greater educational attainment than the present cohort comprised the former study, and the latter study excluded all persons with HIV-associated dementia, thus reducing the overall prevalence of neurocognitive impairment. More importantly, the shift from HIV-associated dementia to milder forms of neurocognitive impairment highlights the need for effective screening tests to identify these more subtle forms (Valcour et al. 2011a). Early identification is of critical importance, with the recent recognition that even these less severe forms of neurocognitive impairment are associated with subsequent progressive cognitive decline and other health implications (Heaton et al. 2012; Watkins and Treisman 2012; Wendelken and Valcour 2012). Cost-effective screening tools which require minimal performance time are clearly needed.

A challenge that has prevented the development of effective screening tools for HAND is that multiple etiologies (including aberrant immune response (Cysique et al. 2011), medication neurotoxicity (Marra et al. 2009), comorbidities (diabetes and hypertension) (McCutchan et al. 2012), or genetic factors (apolipoprotein E4 allele) (Chang et al. 2011)) may contribute. While this study was not specifically designed to evaluate other possible etiologies of neurocognitive impairment, the homogeneity of the sample population with regard to virologic suppression provided the opportunity for us to look at other factors. We observed that the African–American racial background, education level, total and LDL cholesterol, and current smoking use were associated with neurocognitive impairment. The relationship of race and education may reflect socioeconomic factors that are well known to impact cognitive functioning and likely serve as a marker of cognitive reserve. The cognitive reserve hypothesis suggests that individuals with greater cognitive reserve, i.e., higher education, exhibit higher resistance to neuropathologic damage (Roe et al. 2010; Roe et al. 2007). The associations with cholesterol and smoking are intriguing and suggest that atherosclerosis may contribute to neurocognitive impairment. An association between metabolic factors, most notably obesity, and cognitive impairment was recently reported in another large HIV-infected cohort (McCutchan et al. 2012). The pro-inflammatory milieu, attributable to HIV and obesity, likely engenders atherosclerotic changes and leads to an underlying vascular component that contributes to neurocognitive impairment (McMurtray et al. 2008). Cigarette smoking is much more prevalent among HIV-infected persons than the general population (Durazzo et al. 2007). The negative association of tobacco on cognition provides additional ammunition for care providers to counsel patients to stop smoking (Vellozzi et al. 2009; Chen et al. 2012). Further research is needed to understand how these different factors impact cognition in this at-risk population.

The search for an effective screening tool for neurocognitive impairment remains a challenge. Our data illustrate that simple screening tools developed for AD may not provide the best discriminator in an HIV-infected population. We observed reduced sensitivity and specificity for the AD-8 compared to previous AD literature (Galvin et al. 2005). Given that different factors likely contribute to neurocognitive impairment in the current HIV-infected population, it is not surprising that the AD-8, a battery that assesses for functional impairment, performed poorly. The nature of the questions within the AD-8, specifically orientation and memory, makes it less applicable to a subcortical process like HIV infection. The AD-8 has been validated in the AD community for either the person or their surrogate informant to complete, and thus, this tool lends itself to a third party assessment of the person’s functional change. In contrast, HIV-infected persons often attend clinic appointments alone (Valcour et al. 2011a) and often have limited social support networks (Shippy and Karpiak 2005).

The MoCA performed better than the AD-8 as manifested within the ROC curves, yielding a fair prognostic score (0.60–0.69) compared with the poor prognostic score of the AD-8 (0.50–0.59) (Yourman et al. 2012). The MoCA tests eight cognitive domains and is less likely to be affected by education, comorbidities, or reporting bias (Nasreddine et al. 2005). However, this screening tool alone failed to adequately distinguish HIV-infected persons with neurocognitive impairment from those with normal cognition. Unlike previous studies that have used the MoCA in an HIV-infected cohort (Koski et al. 2010; Chan et al. 2012), ours is the first to compare this brief screening test to more formal neuropsychological testing. Notably, the sensitivity of the MoCA was improved to 90 % by changing the threshold, but the prognostic score remained in the fair range. While improved sensitivity may be preferable to the clinician, it will increase the number of persons referred for formal neurocognitive testing. Alternatively, the MoCA could be used to identify persons that should be repeatedly monitored. This tactic has been used with other screening tools with average sensitivity such as cervical cancer screening (Wheeler 2007).

There are several limitations to the current study. This study was cross-sectional, included persons from a single clinical site, and enrolled only persons whose HIV was suppressed with HAART. First, the use of a single site provided certainty that the testing was performed in a similar manner for all persons. Second, the inclusion of only HIV-infected persons who were virologically well controlled was specifically performed in an attempt to limit any confounding that could occur with uncontrolled HIV infection or undetected opportunistic infections. In addition, we did not include persons uninfected with HIV within this study. Future studies are needed that would include a wide range of HIV-infected persons as well as healthy controls. While the large proportion of African–Americans may limit the generalizability, the inclusion of this minority population is highly relevant to the ongoing US HIV epidemic. We did not formally assess activities of daily living and could not make a formal diagnosis of HAND. Using recently developed criteria, asymptomatic neurocognitive impairment is characterized by mild neuropsychological impairment in at least two cognitive domains but without a functional decline, whereas mild neurocognitive disorder is defined by the impairment of at least two domains and reported mild functional decline (Chen et al. 2012). HIV-associated dementia is defined as having deficits in at least two domains and a significant impairment in activities of daily living. While we cannot classify individuals into different HAND categories (asymptomatic neurocognitive impairment, mild neurocognitive disorder, and HIV-associated dementia), our results do not detract from the primary analysis to evaluate these batteries as screening tools for neurocognitive impairment.

In summary, our study confirms that a large proportion of HIV-infected persons have a neurocognitive impairment as assessed by formal neuropsychological testing. The MoCA and AD-8 correlated with formal neurocognitive testing, but the sensitivity of each of these tests was lower than what is desired for a single screening test. The sensitivity of the MoCA can be improved using a different threshold value although the number of false positives also increases. Furthermore, like other cognitive screening tests, performance on the MoCA does not define the etiology of neurocognitive impairment. HIV-infected persons with performances <26 on the MoCA may still require a referral for more comprehensive neuropsychological assessments.