Keywords

Throughout this book, there is an emphasis upon the importance of (1) enhancing ecological validity via a move from construct-driven assessments to tests that are representative of real-world functions—it is argued that this will proffer results that are generalizable for prediction of the functional performance across a range of situations; (2) the potential of computerized neuropsychological assessment devices (CNADs) to enhance: standardization of administration, accuracy of timing presentation and response latencies, ease of administration and data collection, and reliable and randomized presentation of stimuli for repeat administrations; and (3) novel technologies to allow for precise presentation and control of dynamic perceptual stimuli—provides ecologically valid assessments that combine the veridical control and rigor of laboratory measures with a verisimilitude that reflects real-life situations.

Neuropsychologists are increasingly interested in the potential for advanced psychometrics, online computer-automated assessment methods, and neuroinformatics for large-sample implementation and the development of collaborative neuropsychological knowledgebases (Bilder 2011; Jagaroo 2009). In this chapter, the focus will be upon using technology to develop repositories for linking neuropsychological assessment results with data from neuroimaging , psychophysiology, and genetics . It is argued that clinical neuropsychology is ready to embrace technological advances and experience a transformation of its concepts and methods. To develop this aspect of Neuropsychology 3.0, clinical neuropsychologists should incorporate findings from the human genome project, advances in psychometric theory, and information technologies. Enhanced evidence-based science and praxes is possible if neuropsychologists do the following: (1) develop formal definitions of neuropsychological concepts and tasks in cognitive ontologies ; (2) create collaborative neuropsychological knowledgebases ; and (3) design novel assessment methods. These focus areas have been reinforced by a growing body of literature calling for increased emphasis upon neuroinformatics in the future of clinical neuropsychology (Bilder 2011; Jararoo 2009; Nichols and Pohl 2015).

1 Formal Definitions of Neuropsychological Concepts and Tasks in Cognitive Ontologies

1.1 Covariance Among Neuropsychology Measures of Differing Domains

An important growth area for neuropsychology is the capacity for sharing knowledge gained from neuropsychological assessments with related disciplines. Obstacle to this shared knowledge approach includes the covariance among measures and the lack of operational definitions for key concepts and their interrelations. The covariance that exists among neuropsychological measures designed to assess overlapping cognitive domains limits categorical specification into well-delineated domains. As such, neuropsychological assessment batteries may be composed of multiple tests that measure essentially the same performance attributes. According to Dodrill (1997), poor test specificity may be revealed in the median correlations for common neuropsychological tests. For example, Dodrill asserts that while the median correlation within domain groupings on a test was 0.52, the median correlation between groupings was 0.44. From this, Dodrill extrapolates that the tests are not unambiguously domain specific because the median correlations should be notably higher for the within groupings and lower for the between groupings. Consequently, the principal assessment measures used by practitioners may not be quantifying domains to a level of specificity that accounts for the covariance among the measures (Dickinson and Gold 2008; Parsons et al. 2005). Future studies should look at multivariate approaches found in neuroinformatics that will allow for elucidation of the covariance information latent in neuropsychological assessment data.

1.2 Lack of Back-Compatibility in Traditional Print Publishing

An additional concern is that revisions found in traditional print publishing like the Wechsler Adult Intelligence Scale (WAIS) and the Wechsler Memory Scale (WMS) have fallen short in terms of back-compatibility that may invalidate clinical interpretations. Loring and Bauer (2010) contend that the significant alterations in the structure and content of the WAIS-IV/WMS-IV may decrease the potential for accurate patient classification if algorithms developed using earlier versions (e.g., WAIS III) are employed. While the revisions to the Wechsler scales offer enhanced psychometric validity, there are presently insufficient clinical data in neurologic populations to assure that they are appropriate for use in neuropsychological evaluations. This lack of back-compatibility is especially disappointing given the fact that even modest updates to paper-and-pencil batteries take years. The use of modern psychometric theory and neuroinformatics enables preservation of robust back-compatibility with prior test versions and enables the introduction of new content and new constructs. This will become increasingly important as evidence-based medicine is systematically adopted within the context of clinical neuropsychology (e.g., Chelune 2010). There is increasing interest among neuropsychologists in adopting consolidated standards for the reporting of clinical trials that include neuropsychological endpoints that will ultimately serve to strengthen the empirical evidence and scientific base of neuropsychology (Loring and Bowden 2014; Miller et al. 2014).

1.3 Neuropsychology’s Need for Cognitive Ontologies

In addition to novel stochastic approaches to limiting covariance among measures, there is need for formalizing neuropsychological concepts into ontologies that offer formal descriptions of content domains. While ontologies abound in other biomedical disciplines, neuropsychological assessment lags in its development of formal ontologies (Bilder et al. 2009; Jararoo 2009; Price and Friston 2005). The idea of “ontologies” in neuroinformatics reflects the formal specification of entities that exist in a domain and the relations among them (Lenartowicz et al. 2010). A given ontology contains designations of separate entities along with a specification of ontological relations among entities that can include hierarchical relations (e.g., “is-a” or “part-of”) or spatiotemporal relations (e.g., “preceded-by” or “contained-within”). These knowledge structures allow for consistent representations across models, which can facilitate communication among domains by providing an objective, concise, common, and controlled vocabulary. This consistency also allows for enhanced interoperability and provision of links among levels of analysis. Such ontologies have become central within many areas of neuroscience. In the realm of neuropsychology, several projects have been initiated to cognitive ontologies at the Consortium for Neuropsychiatric Phenomics (www.phenomics.ucla.edu). This consortium aims to enable more effective collaboration and facilitation of knowledge sharing about cognitive phenotypes to other levels of biological knowledge (Bilder et al. 2009).

Although there is great need for neuropsychology to develop cognitive ontologies for data garnered from neuropsychological assessment data, there are some roadblocks. Neurocognitive functioning involves interrelated dimensional constructs that refer to a hypothetical cognitive resource whose existence is inferred from research findings (Burgess et al. 2006). Take for example “working memory.” Instead of being a unitary concept, working memory is a latent construct that is estimated in a given study by one or more neuropsychological measures (directly observable behavioral output). Sabb et al. (2008) developed a collaborative annotation database (www.Phenowiki.org) to facilitate representation and sharing of empirical information. As a proof of concept, they examined over 478 articles to evaluate the relationship among estimates of heritability, behavioral measures, and component constructs of executive function. After applying a phrase search algorithm to isolate key terms most commonly used in the neuropsychology literature, they isolated a set of five terms that summarized the literature on executive functions: “cognitive control,” “response inhibition,” “response selection,” “task switching,” and “working memory.” The latter four terms were selected because of their frequency of co-occurrence with the term “cognitive control.” Further, these terms revealed high internal consistency for their indicators and associated heritability measures. This suggests that they may effectively capture distinct components of executive functions. Conversely, Sabb et al. (2008) also found that the same indicators associated with the term “cognitive control” had also been associated with the other constructs. While some suggest that this may reflect shared neural systems that can be important in establishing distinct cognitive constructs (Lenartowicz et al. 2010), there is also the possibility that the neurocognitive assessments themselves may be to blame. Although for some measures there was good consistency across studies in the use of a specific indicator (e.g., Digit Span Backwards only used one indicator: correct recall of digits), other measures had considerable variation in the specific indicator used (e.g., each of three studies reporting Go/No-go performance used different indicators and versions of the test). The result is an inconsistency that greatly increases the difficulty of pooling data and interpreting results across studies. That said, the work of Sabb and colleagues (2008) to develop a collaborative annotation database offers an approach to collaborative knowledge building. Such work enables neuropsychologists in their selection and prioritization of endophenotypes for translational research.

2 Web 2.0 and Collaborative Neuropsychological Knowledgebases

In addition to shared definitions of neuropsychological constructs, the development of ontologies enables systematic aggregation of neuropsychological knowledge into shared databases. Technological developments in the use of the Internet for collective knowledge building are apparent in Web 2.0 practices that involve specialized Web tools (O’Reilly 2007). Web 2.0 represents a trend in open platform Internet use that incorporates user-driven online networks and knowledgebases. While there are no large Internet-based repositories for neuropsychological data , efforts have been made to allow users to input individual test scores via a Web portal (www.neuropsychnorms.com) and obtain immediate reports comparing a patient’s neuropsychological data to published findings (Mitrushina et al. 2005). Another online application is the Cognitive Atlas (http://www.cognitiveatlas.org/) which was developed to address the need for a collaborative knowledgebase that captures the comprehensive collection of conceptual structures within neuropsychology (Miller et al. 2010). To do this, the Cognitive Atlas project describes the “parts” and processes of the cognitive functioning in a manner similar to descriptions of the cell’s component parts and functions in gene ontology. While this project does offer promise, it is still in its infancy and requires development if it is to provide a solid basis for annotation of neuropsychological (e.g., neuroimaging of cognitive processes) data. Further, like other collaborative knowledgebases (e.g., Wikipedia), its realization will depend on the involvement of a large number of interested neuropsychologists (Poldrack 2010).

Some neuropsychologists are calling for greater inclusion of Web-based neurocognitive assessments (Bilder 2011; Jagaroo 2009). The exponential increase in access to computers and the Internet across the life span allows for interactive Web-based neuropsychological assessments (Bart et al. 2014; Raz et al. 2014; Wagner et al. 2010). It is important to note that some neuropsychologists have concerns that Web-based neuropsychological assessment devises will somehow replace human neuropsychologists and/or overlook significant clinical information. However, Web-based neuropsychological assessments are just tools for enhanced presentation and logging of stimuli. When properly used, Web-based assessments can greatly enhance the assessment in terms of precision and rapid implementation of adaptive algorithms. Online neurocognitive assessment may enhance dissemination of testing because tests could be administered in a variety of settings (e.g., office, home, school), at different times of the day, and by multiple persons at the same time.

A distinct improvement of computer timing precision is that it allows for enactment of procedures from the human neurosciences that rely on more refined task operations and trial-by-trial analyses that may be more sensitive and specific to individual differences in neural system function. Web-based computerized assessments have the capacity for adaptive testing strategies that are likely to multiply efficiency in construct measurement (Bilder 2011). In one study, Gibbons and colleagues (2008) found that the use of a computerized adaptive test resulted in a 95 % average reduction in the number of items administered. The potential of adaptive Web-based neurocognitive assessment protocols can be seen in the capacity to large sample of participants in relatively short periods of time. While there may be concerns that the neuropsychologist cannot be sure of the identity of the test-takers or that they are performing tasks as instructed, validity indicators, online video surveillance, and anthropometric identifiers can be included to remove these concerns. For example, algorithms can be implemented that allow for item-level response monitoring and automated consistency checks. Further, neuroinformatics algorithms are available that will allow for the detection of outlying response patterns of uncertain validity.

One of the most widely used and validated of the new Web-based neuropsychology batteries is WebNeuro . The battery includes assessments of the following domains of neurocognitive function: sensorimotor, memory , executive planning, attention , and emotion perception (social cognition; Mathersul et al. 2009; Silverstein et al. 2007; Williams et al. 2009). In addition to cognitive assessments, the WebNeuro battery also includes affective assessments of emotion recognition and identification: immediate explicit identification followed by implicit recognition (within a priming protocol). The WebNeuro protocol was developed as a Web-based version of the IntegNeuro computerized battery . The IntegNeuro battery was developed by a consortium of scientists interested in establishing a standardized international called Brain Resource International Database (BRID; Gordon 2003; Gordon et al. 2005; Paul et al. 2005). IntegNeuro and WebNeuro are part of the BRID project’s aim to move beyond outdated approaches to aggregating neuropsychological knowledgebases and develop standardized testing approaches that facilitate the integration of normally independent sources of data (genetic, neuroimaging, psychophysiological, neuropsychological, and clinical). The WebNeuro platform allows for data to be acquired internationally with a centralized database infrastructure for storage and manipulation of these data.

While the normative findings for a Web-based neuropsychological assessment are promising (Mathersul et al. 2009; Silverstein et al. 2007; Williams et al. 2009), the greatest potential appears to be WebNeuro’s relation to Brain Resource International Database. The data from WebNeuro are linked to insights and correlations in a standardized and integrative international database (see www.BrainResource.com; www.BRAINnet.com). The WebNeuro data are incorporated as additional (in addition to other markers: genetic, neuroimaging , psychophysiological, neuropsychological, and clinical) clinical markers that can be incorporated into databases as new marker discoveries emerge from personalized medicine.

3 Construct-Driven and Function-Led Redux

Earlier in this book, we discussed the limitations of construct-driven tasks and the need for function-led neuropsychological assessments. While the original goal of many neuropsychological assessments was lesion localization , there is increasing need for assessments of everyday functioning . This new role for neuropsychologists has resulted in increased emphasis upon the ecological validity of neuropsychological instruments (Chaytor et al. 2006). As a result, neuropsychologists have been compelled to move beyond the limited generalizability of results found in construct-driven measures to tests that more closely approximate real-world function. The function-led approach allows the neuropsychologist to move from construct-driven assessments to tests that are representative of real-world functions and proffer results that are generalizable for prediction of the functional performance across a range of situations.

A number of investigators have argued that performance on traditional neuropsychological construct-driven tests (e.g., Wisconsin Card Sorting Test, Stroop) has little correspondence to activities of daily living (Bottari et al. 2009; Manchester et al. 2004; Sbordone 2008). According to Chan et al. (2008), most of these traditional construct-driven measures assess at the veridicality level and do not capture the complexity of response required in the many multistep tasks found in everyday activities. The “function-led approach” to creating neuropsychological assessments will include neuropsychological models that proceed from directly observable everyday behaviors backward to examine the ways in which a sequence of actions leads to a given behavior in normal functioning and the ways in which that behavior might become disrupted. These neuropsychological assessments should meet the usual standards of reliability , but discussions of validity should include both sensitivity to brain dysfunction and generalizability to real-world function.

However, this raises a problem for the development of cognitive ontologies and collaborative knowledgebases . These approaches rely on construct-driven and computer-automated neuropsychological assessment devices. One approach would be to have two streams of research: (1) use traditional (paper-and-pencil and computerized neuropsychological assessment devices) construct-driven assessments to develop cognitive ontologies and collaborative knowledgebases and (2) use virtual environment-based function-led assessments for return-to-capacity decisions. While this approach seems to have promise in the short term, it seems to miss the rich data that could come from an approach that used simulation technologies as a special case of computerized neuropsychological assessment devise.

3.1 Virtual Environments for Assessing Polymorphisms

While little neuroinformatic work has included virtual reality immersion, neuroimaging studies utilizing virtual environments have been used to delineate brain circuits involved in sustained anxiety to unpredictable stressors in humans. In a study of contextual fear conditioning, Alvarez et al. (2008) used a Virtual Office and fMRI to investigate whether the same brain mechanisms that underlie contextual fear conditioning in animals are also found in humans. Results suggested that contextual fear conditioning in humans was consistent with preclinical findings in rodents. Specifically, findings support hot affective processing in that the medial aspect of the amygdala had afferent and efferent connections that included input from the orbitofrontal cortex. In another study using a Virtual Office, Glotzbach-Schoon et al. (2013) assessed the modulation of contextual fear conditioning and extinction by 5HTTLPR (serotonin-transporter-linked polymorphic region) and NPSR1 (neuropeptide S receptor 1) polymorphisms. Results revealed that both the 5HTTLPR and the NPSR1 polymorphisms were related to hot affective (implicit) processing via a fear potentiated startle. There was no effect of the 5HTTLPR polymorphism on cold cognitive (explicit) ratings of anxiety. Given the ability of virtual environments to place participants in experimentally controlled yet contextually relevant situations, there appears to be promise in applying this platform to future translational studies into contextual fear conditioning. The ability to differentiate cold cognitive from hot affective processing could be used to develop ontologies for future research.

3.2 Neuropsychological Assessment Using the Internet and Metaverse Platforms

A further development of the emerging Neuropsychological Assessment 3.0 paradigm for clinical neuropsychologists may be found in the expanding Metaverse of virtual worlds found on the Internet. Technological advances in computing and the World Wide Web in the last couple decades (Abbate 1999) have allowed for Internet-based virtual worlds testing with potentially more diverse samples in respect to socioeconomic status, sex, and age than traditional samples that are often drawn from undergraduate university students (Gosling et al. 2004). Virtual worlds are made up of online communities in which persons interrelate in simulated environments. The continued progress in the development of robust technologies such as more rapid and secure Internet connections has led to the ever-increasing interest in social networks (Boulos and Wheeler 2007). Parsons (2012) described a number of advantages of Internet-delivered online virtual worlds:

  • Systematic presentation of cognitive tasks targeting both construct-driven and function-led neuropsychological performance beyond what is currently available using traditional computerized neuropsychological assessment devices.

  • Reliability of function-led neuropsychological assessment and treatment of affective and cognitive disorders can be enhanced in online virtual worlds by better control of the perceptual environment, more consistent stimulus presentation, and more precise and accurate scoring.

  • Online virtual worlds may also improve the validity of neurocognitive measurements via the increased quantification of discrete behavioral responses, allowing for the identification of more specific cognitive domains.

  • Online virtual worlds could allow for cognition and affect to be assessed and treated in situations that are more ecologically valid. Participants can be evaluated in an online virtual world that simulates the real world, not a contrived testing environment.

  • Online virtual worlds offer the option to produce and distribute identical “standard” simulation environments to large samples in which performance can be measured and treated.

Within such digital scenarios, normative data can be accumulated for performance comparisons needed for assessment/diagnosis and for treatment/rehabilitation purposes. In this manner, reusable archetypical online virtual worlds constructed for one purpose can also be applied for applications addressing other clinical targets.

One example is Second Life (SL; Linden Lab, San Francisco, Calif.), which proffers tools (i.e., scripting and graphics) and environments that facilitate the creation of virtual environments that can be made available to potentially thousands of research subjects in an economical manner (Bainbridge 2007). The population of users in Second Life has reached more than six million virtual citizens (Boulous 2007). Virtual worlds provide users to experience social interaction as they participate in individual and group activities. The virtual world Second Life proffers multiple medical and health educational projects (Boulos et al. 2007). Although these programs focus primarily on the dissemination of medical information and the training of clinicians, a handful of private islands in Second Life (e.g., Brigadoon for Asperger’s syndrome; Live2give for cerebral palsy ) have been created for therapeutic purposes. In a recent article by Gorini et al. (2008), the authors describe such sites and the development and implementation of a form of tailored immersive e-therapy in which current technologies (e.g., virtual worlds; bio and activity sensors; and personal digital assistants) facilitate the interaction between real and 3-D virtual worlds and may increase treatment efficacy. In a recent article in science, Bainbridge (2007) discussed the robust potential of virtual worlds for research in the social and behavioral sciences. For social and behavioral science researchers, virtual worlds reflect developing cultures, each with an emerging ethos and supervenient social institutions (for a discussion of supervenience see Hare 1984). In addition to the general social phenomena emerging from virtual world communities, virtual worlds provide novel opportunities for studying them. According to Bainbridge (2007), virtual worlds proffer environments that facilitate the creation of online laboratories that can recruit potentially thousands of research subjects in an automated and economically feasible fashion. Virtual worlds like Second Life offer scripting and graphics tools that allow even a novice computer user the means necessary for building a virtual laboratory. Perhaps even more important is the fact that social interactions in online virtual worlds (e.g., Second Life) appear to reflect social norms and interactions found in the physical world (Yee et al. 2007). Finally, there is the potential of virtual worlds to improve access to medical rehabilitation. Klinger and Weiss (2009) describe the evolution of virtual worlds along to two dimensions: (1) the number of users and (2) the distance between the users. According to Klinger and Weiss, single user and locally used virtual worlds have developed into three additional venues: (1) multiple users located in the same setting, (2) single users remotely located, and (3) multiple users remotely located. According to Klinger and Weiss, single user, locally operated virtual worlds will continue to be important for rehabilitation within a clinical or educational setting. However, the literature, to date, has been limited to descriptions of system development and reports of small pilot studies (Brennan et al. 2009). It is anticipated that this trend is changing and future years will see evidence of the effectiveness of such virtual worlds for therapy.

4 Computational Neuropsychology

4.1 Virtual Environments as a Special Case of Computerized Neuropsychological Assessment Devices

In this book, we have looked at the development of technologies for neuropsychological assessment. An argument has been made that ecologically valid assessments should include both cognitive constructs and functions in affect sensitive simulations that represent real-world functions. One approach suggested in Neuropsychological Assessment 3.0 was to extend computerized neuropsychological assessment devices to virtual environments. Virtual environments represent a special case of computerized neuropsychological assessment devices because they have enhanced computational capacities for administration efficiency, stimulus presentation, automated logging of responses, and data analytic processing. Since VEs allow for precise presentation and control of dynamic perceptual stimuli, they can provide ecologically valid assessments that combine the veridical control and rigor of laboratory measures with a verisimilitude that reflects real-life situations. Additionally, the enhanced computation power allows for increased accuracy in the recording of neurobehavioral responses in a perceptual environment that systematically presents complex stimuli. Such simulation technology appears to be distinctively suited for the development of ecologically valid environments, in which three-dimensional objects are presented in a consistent and precise manner (Schultheis et al. 2002). VE-based neuropsychological assessments can provide a balance between naturalistic observation and the need for exacting control over key variables (Campbell et al. 2009; Parsons 2011). In summary, VE-based neuropsychological assessments allow for real-time measurement of multiple neurocognitive abilities in order to assess complex sets of skills and behaviors that may more closely resemble real-world functional abilities (Matheis et al. 2007).

4.2 Extending Computer-Automated Construct-Driven Ontologies to Virtual Environments

A possible place to start for extending cognitive ontologies from computer-automated neuropsychological assessment devices is to extend the annotations to construct-driven virtual environments like the virtual apartment Stroop task . The virtual apartment superimposes construct-driven stimuli (e.g., Stroop and CPT) onto a large television set in a virtual living room (Henry et al. 2012). A series of construct-driven stimuli appear on the flatscreen television with in the environment, while a female voice states the names of colors (red, blue, and green). Participants are instructed to click a mouse button as quickly as possible when the spoken color matches the color of the rectangle on the virtual flatscreen television and to withhold a response if the colors do not match. A total of 144 stimuli are presented, with 72 targets and 72 nontargets. During the task, distracters appear in different field of view locations in the environment. Some distracters are audio–visual: School bus passing on the street and a sports utility vehicle viewed through window on the right; iPhone ringing and vibrating on the table (in front of participant); and toy robot moving and making noise on the floor (center). Auditory distractors include the following: crumple paper (left); drop pencil (left); doorbell (left); clock (left) vacuum cleaner (right); jack hammer (right); sneeze (left); and jet noise (center). Visual distractors include the following: paper airplane (flying from left to right in front of the participant) and woman walking in the kitchen (center). The conditions were designed to assess reaction times (simple and complex), selective attention (matching the auditory and visual stimuli), and external interference control (environmental distracters). Computational neuropsychologists could compare construct-driven virtual environment performance with and without distractors to computer-automated neuropsychological assessment devices.

4.3 Construct-Driven and Function-Led Virtual Environments for Hot and Cold Processing

Recently, virtual environments have been applied to the assessment of both “Cold” and “Hot” processes using combat-related scenarios (Armstrong et al. 2013; Parsons et al. 2013). The addition of virtual environments allows neuropsychologists to move beyond the ethical concerns related to placing participants into real-world situations with hazardous contexts. The goal of these platforms is to assess the impact of hot affective arousal upon cold cognitive processes. For example, Parsons et al. (2013) have developed a Virtual Reality Stroop Task (VRST) in which the participant is immersed in a simulated High Mobility Multipurpose Wheeled Vehicle (HMMWV) and passes through zones with alternating low threat (driving down a deserted desert road) and high threat (gunfire, explosions, and shouting among other stressors), while construct-driven stimuli (e.g., Stroop stimuli) were presented on the windshield. They found that the high-threat zones created a greater level of psychophysiological arousal (heart rate, skin conductance , respiration) than did low-threat zones. Findings from these studies also provided data regarding the potential of military relevant virtual environments for the measurement of supervisory attentional processing (Parsons et al. 2013). Analyses of the effect of threat level on the color–word and interference scores resulted in a main effect of threat level and condition. Findings from the virtual environment paradigm support the perspective that (1) high information load tasks used for cold cognitive processing may be relatively automatic in controlled circumstances—for example, in low-threat zones with little activity; and (2) the total available processing capacities may be decreased by other hot affective factors such as arousal (e.g., threat zones with a great deal of activity). In a replication study, Armstrong et al. (2013) established the preliminary convergent and discriminant validity of the VRST with an active duty military sample.

In addition to virtual environment-based neuropsychological assessments using driving simulators, a number of other military relevant virtual environments have emerged for neurocognitive assessment of Cold and Hot processes. For example, Parsons et al. (2012, 2014) immersed participants into a Middle Eastern city and exposed participants to a cold cognitive processing task (e.g., paced auditory serial addition test) as they followed a fire team on foot through safe and ambush (e.g., hot affective—bombs, gunfire, screams, and other visual and auditory forms of threat) zones in a Middle Eastern city. In one measure of the battery, a route-learning task, each zone is preceded by a zone marker, which serves as a landmark to assist in remembering the route. The route-learning task is followed immediately by the navigation task in which the participants were asked to return to the starting point of their tour through the city. Courtney et al. (2013) found that the inclusion of hot affective stimuli (e.g., high-threat zones) resulted in a greater level of psychophysiological arousal (heart rate, skin conductance, respiration) and decreased performance on cold cognitive processes than did low-threat zones. Results from active duty military (Parsons et al. 2012) and civilian (Parsons and Courtney 2014) populations offer preliminary support for the construct validity of the VR-PASAT as a measure of attentional processing. Further, results suggest that the VR-PASAT may provide some unique information related to hot affective processing not tapped by traditional Cold attentional processing tasks.

While the above virtual environments fall short of a truly function-led environment in that they include construct-driven stimuli, they go beyond other construct-driven virtual environments because they include affective stimuli. It may be possible to take this paradigm and extend it to the ontological annotation of other executive control tasks. This would involve new neuroinformatic approaches to deal with the massively enhanced dataset. That said, neuroinformatics for neuropsychologists is in its earliest stages and it will not be long before collaborative knowledgebases will be developed for construct-driven assessments. A computational neuropsychological approach that uses artificial neural networks and machine learning could allow for enhanced assessment of construct-driven stimuli embedded in virtual environments . The end result would be a collaborative knowledgebase of function-led virtual environments to describe both the functions and the operational constructs involved in everyday activities.