Abstract
Today, Virtual Reality (VR) and Augmented Reality (AR) are the new communication tools readily available to consumers. Because of the increasing availability of AR and VR, communication and neuroscience researchers are showing increasing interest in the use of VR systems for studies in collaboration, communication, and basic neuroscience. Beyond relying on self-reported or behavioral measures, psychophysiological or functional neuroimaging measurements sensing brain waves (e.g. EEG) or brain hemodynamics (e.g. fNIRS) are powerful techniques for measuring brain activity while interacting with virtual reality stimuli or environments. However, using these measures with virtual reality systems can be difficult due to physical and technical constraints. Both Functional Near-Infrared Spectroscopy (fNIRS) and Electroencephalography (EEG) need multiple channels to measure brain activity, a combination of cables and probes must be attached to a head cap. However, this setup obstructs wearing head-mounted display (HMD) in a VR environment and the challenge varies with the design of the HMD. To overcome these limitations, we introduce the design and development of the M.I.N.D. brain measurement cap specifically adapted for research with virtual reality system. We discuss the design process as well as the advantages and limitations of the current iterative design of the cap. Generally, we anticipate that this measurement system will expand the potential of influence of cognitive neuroscience contribute on VR research by making it easier for researchers to use a breadth of tools.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction
Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) are regarded as the new frontier in digital communication media and use of these new technology tools is rapidly increasing [1].
The development and increased availability of advanced brain measurement methods allows researchers to explore human behaviors and minds in different ways than were previously available. Researchers do not have to rely on entirely on the results from self-reported measurements, which have known limitations [2,3,4] or behavioral measures only. The emergence of neurophysiological measurements has opened up new fields and allowed others to grow because the tools allow us to explore previously unanswerable questions. For the reasons above, neurophysiological research has become increasingly popular.
This has been possible because the brain can be directly observed by non-invasive electrophysiological recording and the data shows accurate and high-resolution data.
2 A Review of Functional Neuroimaging and Electrophysiological Measurement
A cognitive neuroscience-based measurement is an advanced and improved technique to expand our knowledge of human behaviors and minds. These measurements are designed for non-invasively interpreting brain functioning at high temporal and/or spatial resolution. By capturing electric fields on the surface of the head induced by neuronal activity in the brain, we can specify human brain function.
This concept has been developed with the trust that the brain mainly controls physical and psychological activities [5]. For instance, this basic concept has widely contributed to develop brain-computer interfaces (BCI) using the brain as an addition input device for broader users including disable patients [6].
In addition, psychophysiological approaches to the brain have contributed to explore human minds deeper and wider without getting direct responses from participants. For example, neuroimaging studies have proven that increased amygdala activity when viewing fear [7], sad, angry [8] or happy facial expressions [7]. In addition, sensors are also able to measure mental workload [9], which is more effective and accurate than estimating workload from post-test questionnaires. The reason is first, it does not instantly reflect the result at the time of exposure (delay in response). In most cases, the questionnaire is completed after exposure, not during exposure of stimuli. Therefore, to complete a post-test questionnaire, a participant needs to recall the time of the experiment. Secondly, the result from the 5 or 7 scale questionnaires does not represent the details of participants’ minds and thoughts because human’s psychological process is not readily available for introspection. On the contrary, the brain imaging tool using a psychophysiological method continuously captures user’s cognitive process on a millisecond scale. For these reasons, psychophysiological measurements have been widely used even though it requires more time and cost to set the experimental environment up.
Then, what types of brain waves we can measure and what do they represent? There are technically different methods to measure brain waves, and each method has a different approach to explore human minds.
There are several physiological measurements (i.e. PET, fMRI or TMS), however we discuss only methods that measure brain waves with a head cap and sensors since a purpose of this paper is to suggest a new design of the cap for brain wave measurement system.
2.1 Electroencephalogram (EEG)
In contrast to other physiological measures such as heart rate, skin-conductance level, or facial electromyography (EMG), Electroencephalogram (EEG) is a direct measure of central nervous system activity from the active brain. In particular, the EEG can see the changes in electrical discharge of cortical neuronal populations; it measures the capacity or performance of cortical information [10]. In other words it can measure the degree of brain concentration [11]. It can measure the arousal dimension of human emotions from peripheral signals [12] which may include alpha, theta and/or frequencies greater than 16 Hz [13]. In addition, it is ease of use and low set-up cost. It generally uses from 2–256 channels to see each area of the brain by wearing a head cap and putting measuring sensors (electrodes) into grommets over the cap. Grimes et al. showed 99% accuracy in working memory states (WM) and four WM states with up to 88% accuracy with EEG [14]. Because of this accuracy, military also uses EEG to monitor pilots’ mental states while they are in the air [15] (Fig. 1).
2.2 Functional Near-Infrared Spectroscopy (fNIRS)
Functional near-infrared spectroscopy (fNIRS) is another advanced neuroimaging technology for mapping the functioning human cortex. It was originally designed for medical use and has been first clinically used in the mid-1980s. However, it is now playing an important role in both neuroscience and communication research [16]. The fNIRS uses a measurement of concentration changes in both oxygenated and deoxygenated hemoglobin (Hb). It is also a non-invasive imaging method to see the changes in blood oxygenation, that can represent levels of brain activation while EEG captures electrical waves associated with the activation potential of neurons. The fNIRS uses optical fibers placed on the scalp that send light in the wavelength range of 650–850 mm in to the targeted area on the head, where the infrared light is re-emitted based on the amount of oxygenated and deoxygenated hemoglobin from the issue during brain function from multiple channels (up to 256).
This method has been proven by many researchers as a valid method to measure hemodynamic levels originating from prefrontal cortex (PFC) activation [17] in emotion induction [18] and cognitive functions such as problem solving or memory related to mental workload [19] (Fig. 2).
3 Benefits of Psychophysiological Measures to Understand New Media
The brain imaging tools we discussed above have several common advantages for research. First, they accurately measure human mental states or workload with sub-second or even millisecond level precision. Secondly, those are popular, non-invasive and widely proven techniques in research. Third, EEG or fNIRS can measure brain waves in more ecologically valid conditions than other sensors. For example, using a PET or fMRI require participants to be stationary or lie in restricted positions.
For those reasons, EEG and fNIRS have been widely used in human computer interaction (HCI) studies, including usability testing and user-centered design for better human performance [9]. In addition, EEG and fNIRS are also suitable for new media research when scientists are interested in the cognitive mechanisms behind media effects [20, 21].
3.1 Virtual Reality
The concept of Virtual Reality (VR) is to experience a certain unreal environment as real. With the technological development in graphics, it is possible to depict a realistic environment in 3D. By wearing Head-mounted display (HMD), it is possible for a user to be in a virtual world without seeing other real objects near the user that may distract experiencing immersive VR.
In addition, other functionalities that increase users’ emersion, such as head-tracking or motion-tracking system, make a virtual environment more realistic so that users feel “they are actually there” [22,23,24]. Therefore, in recent years communication researchers and sociologists have studied to see their minds when they interact with other objects or people in VR [25, 26].
3.2 Challenges in Recording EEG and fNIRS in Virtual Reality
As we discussed above, it is essential to see users’ cognition when they interact in VR. It is essential to see how users accept and process information they get and interact with in virtual environments.
However, combining VR system and EEG or fNIRS is difficult because the HMD mounts interfere with the probes of the devices. There were some studies that see some social effects of virtual reality with electroencephalography (EEG) or fNIRS previously [27, 28], however they just see the effect of virtual object shown in a 2D display, not an HMD.
As shown in the Fig. 3, both HTC VIVE and Oculus Rift use a 3-axis headband for a perfect fit. In addition, the headband should be tightened enough to cut off the light from outside of the HMD, so the user can fully focus in virtual environment. However, once the user puts the brain measurement cap on, they are unable to wear the HMD over the cap since the cables and sensors attached to the cap block placement of the HMD. Since the HMD head bands are made with a flexible rubber, it puts pressure on the sensors. This pressure displaces the probes, which results in poor sensor contact with the head and consequently, poor data quality. Also, to measure prefrontal cortex activity in VR, the HMD cannot be worn correctly because the HMD obstructs the area over the prefrontal cortex. If the HMD is worn below the cap, then the HMD lens and eye are not aligned and centered, that finally results in a failure of setting up the virtual environment. Figure 4 shows the experimental setup of fNIRS. As shown in the picture, it is impossible to wear the HMD over the cap because of cables.
For those restrictions, it has always been a difficulty for scholars to research an immersive virtual environment using sensors like the fNIRS.
3.3 Previous Solutions Suggested by Other Researchers
Figure 5 shows a method to use fNIRS in VR setup suggested by Seraglia et al. [29] in 2011. They developed a VR cap that does not interfere fNIRS cables. However, the configuration limits the regions of the brain that can be measured, which functionally prohibits research on topics like cognitive load. For instance, prefrontal cortex activity seeing many higher cognitive functions [30] cannot be measured with this setup. Many researchers investigated prefrontal cortex activity in the simulated environment created with computer graphics [31, 32]. Therefore, it was highly needed to develop a new brain measurement cap to solve this limitation.
4 M.I.N.D. Brain Cap, Measurement for VR Research: Design and Iterations
4.1 M.I.N.D. Brain Cap, Initial Design
To solve issues and restrictions of brain measurements in VR discussed above, we designed an innovative head cap that makes it possible to wear the head-mounted display over the brain measurement cap (Figs. 6 and 7).
There are several advantages to this design. First, researchers do not need to limit the number of channels to use because the cap is designed to make most channels available in the VR experimental setup.
Secondly, this cap saves time in setting up an experimental environment because subjects need to wear only one cap that incorporate (1) the HMD, and (2) the brain sensor array. In a previous setup, a subject wears the brain measurement cap first and then wear the HMD. Third, the cap design works with both EEG and fNIRS. Senor sockets can be easily replaced for each experimental measurement (Fig. 8).
However, there are also limitations of the cap. First, even though the main material of the cap is made of rubber, expansion of the cap could not cover all head sizes. For instance, optical signals showed poor in the front forehead area when the cap was worn on a smaller head. Secondly, connectors to attach HMDs are not universal, therefore we need to design a cap for each HMD (e.g. Oculus rift version, or HTC VIVE version). Thirdly, modifying the channel configuration and probe placement is difficult.
4.2 M.I.N.D. Brain Cap: Second Version
To solve the issues of the first version of the cap, we made several design modifications. First, we choose a modular design to address the size issue of the cap.
As shown in Fig. 9, the cap consists of three different parts and each part can be easily connected to other modules (probe holders) with several plastic clips. By doing so, a researcher can easily find a perfect fit (size) for participant’s head by configuring different size of cap modules. Therefore, we have a higher probability of accurate probe placement and better-quality data. Secondly, the researcher can easily rearrange the measurement area by choosing various size of modules, which is a standard option for non-VR research. For instance, standard configuration of the modular consist of 4 × 4 and 3 × 3 channels. If a researcher wants to concentrate on a certain area, then he or she can use 4 × 4, and choose 3 × 3 for the rest area in fNIRS (See Fig. 10 for details).
5 Conclusion
The emergence of VR and AR as a readily available communication tool and media interface has pushed researchers to begin exploring the cognitive neuroscience associated with VR/AR experiences. However, conducting studies is difficult because of technical challenges associated with integrated HMDs and sensors. The M.I.N.D. brain measurement cap is the first attempt at addressing these problems; we hope our design makes it easier for researchers to combine VR/AR and EEGs and fNIRS.
References
Ryan, M.-L.: Narrative as Virtual Reality 2: Revisiting Immersion and Interactivity in Literature and Electronic Media. Johns Hopkins University Press, Baltimore (2015)
Donaldson, S.I., Grant-Vallone, E.J.: Understanding self-report bias in organizational behavior research. J. Bus. Psychol. 17, 245–260 (2002). https://doi.org/10.1023/A:1019637632584
McDonald, J.D.: Measuring personality constructs: the advantages and disadvantages of self-reports, informant reports and behavioural assessments. Enquire 1, 1–19 (2008)
Stone, A.A., Bachrach, C.A., Jobe, J.B., Kurtzman, H.S., Cain, V.S.: The Science of Self-report: Implications for Research and Practice. Lawrence Erlbaum, Mahwah (1999)
Coles, M.G.H.: Modern mind-brain reading: psychophysiology, physiology, and cognition. Psychophysiology 26, 251–269 (1989). https://doi.org/10.1111/j.1469-8986.1989.tb01916.x
Girouard, A., Solovey, E.T., Hirshfield, L.M., Peck, E.M., Chauncey, K., Sassaroli, A., Fantini, S., Jacob, R.J.K.: From brain signals to adaptive interfaces: using fNIRS in HCI. In: Brain-Computer Interfaces Applying our Minds to Human-Computer Interaction, pp. 221–237 (2010)
Breiter, H.C., Etcoff, N.L., Whalen, P.J., Kennedy, W.A., Rauch, S.L., Buckner, R.L., Strauss, M.M., Hyman, S.E., Rosen, B.R.: Response and habituation of the human amygdala during visual processing of facial expression. Neuron 17, 875–887 (1996). https://doi.org/10.1016/s0896-6273(00)80219-6
Johnson, B.T., Eagly, A.H.: Effects of involvement on persuasion: a meta-analysis. Psychol. Bull. 106, 290–314 (1989). https://doi.org/10.1037/0033-2909.106.2.290
Hirshfield, L.M., Chauncey, K., Gulotta, R., Girouard, A., Solovey, E.T., Jacob, R.J.K., Sassaroli, A., Fantini, S.: Combining electroencephalograph and functional near infrared spectroscopy to explore users’ mental workload. In: Schmorrow, D.D., Estabrooke, I.V., Grootjen, M. (eds.) FAC 2009. LNCS (LNAI), vol. 5638, pp. 239–247. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-02812-0_28
Klimesch, W.: EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis. Brain Res. Rev. 29, 169–195 (1999). https://doi.org/10.1016/S0165-0173(98)00056-3
Liu, N.H., Chiang, C.Y., Chu, H.C.: Recognizing the degree of human attention using EEG signals from mobile sensors. Sens. (Switz.) 13, 10273–10286 (2013). https://doi.org/10.3390/s130810273
Chanel, G., Kronegg, J., Grandjean, D., Pun, T.: Emotion assessment: arousal evaluation using EEG’s and peripheral physiological signals. In: Gunsel, B., Jain, A.K., Tekalp, A.M., Sankur, B. (eds.) MRCS 2006. LNCS, vol. 4105, pp. 530–537. Springer, Heidelberg (2006). https://doi.org/10.1007/11848035_70
Bonnet, M.H., Carley, D., Consultant, M.C., Consultant, P.E., Chairman, C.G., Harper, R., Hayes, B., Hirshkowitz, M., Keenan, S., Consultant, M.P., Roehrs, T., Smith, J., Weber, S., Westbrook, P., Administrative, A., Bruce, S.: EEG arousals: scoring rules and examples: a preliminary report from the Sleep Disorders Atlas Task Force of the American Sleep Disorders Association. Sleep 15, 173–184 (1992)
Grimes, D., Tan, D.S., Hudson, S.E., Shenoy, P., Rao, R.P.N.: Feasibility and pragmatics of classifying working memory load with an electroencephalograph. In: Proceedings of the 26th SIGCHI Conference on Human Factors in Computing Systems, p. 835 (2008). https://doi.org/10.1145/1357054.1357187
Karim, H., Schmidt, B., Dart, D., Beluk, N., Huppert, T.: Functional near-infrared spectroscopy (fNIRS) of brain function during active balancing using a video game system. Gait Posture. 35, 367–372 (2012). https://doi.org/10.1016/j.gaitpost.2011.10.007
Liu, N., Mok, C., Witt, E.E., Pradhan, A.H., Chen, J.E., Reiss, A.L.: NIRS-based hyperscanning reveals inter-brain neural synchronization during cooperative Jenga game with face-to-face communication. Front. Hum. Neurosci. 10 (2016). https://doi.org/10.3389/fnhum.2016.00082
Sato, H., Yahata, N., Funane, T., Takizawa, R., Katura, T., Atsumori, H., Nishimura, Y., Kinoshita, A., Kiguchi, M., Koizumi, H., Fukuda, M., Kasai, K.: A NIRS–fMRI investigation of prefrontal cortex activity during a working memory task. Neuroimage 83, 158–173 (2013). https://doi.org/10.1016/j.neuroimage.2013.06.043
Herrmann, M.J., Ehlis, A.C., Fallgatter, A.J.: Prefrontal activation through task requirements of emotional induction measured with NIRS. Biol. Psychol. 64, 255–263 (2003). https://doi.org/10.1016/S0301-0511(03)00095-4
Izzetoglu, K., Bunce, S., Onaral, B., Pourrezaei, K., Chance, B.: Functional optical brain imaging using near-infrared during cognitive tasks. Int. J. Hum. Comput. Interact. 17, 211–227 (2004)
Bolls, P.D., Lang, A., Potter, R.F.: The effects of message valence and listener arousal on attention, memory, and facial muscular responses to radio advertisements. Commun. Res. 28, 627–651 (2001). https://doi.org/10.1177/009365001028005003
Park, B.: Psychophysiology as a tool for HCI research: promises and pitfalls. In: Jacko, J.A. (ed.) HCI 2009. LNCS, vol. 5610, pp. 141–148. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-02574-7_16
Biocca, F.: The Cyborg’s dilemma: progressive embodiment in virtual environments minding the body, the primordial communication medium. JCMC 3, 1–29 (1997). https://doi.org/10.1111/j.1083-6101.1997.tb00070.x
Heeter, C.: Being there: the subjective experience of presence. Presence 1, 262–271 (1992). https://doi.org/10.1109/VRAIS.1995.512482
Reeves, B., Nass, C.: How People Treat Computers, Television, and New Media Like Real People and Places (1998)
McCabe, K., Houser, D., Ryan, L., Smith, V., Trouard, T.: A functional imaging study of cooperation in two-person reciprocal exchange. Proc. Natl. Acad. Sci. 98, 11832–11835 (2001). https://doi.org/10.1073/pnas.211415698
Rilling, J.K., Gutman, D.A., Zeh, T.R., Pagnoni, G., Berns, G.S., Kilts, C.D.: A neural basis for social cooperation. Neuron 35, 395–405 (2002). https://doi.org/10.1016/S0896-6273(02)00755-9
Baumgartner, T., Valko, L., Esslen, M., Jäncke, L.: Neural correlate of spatial presence in an arousing and noninteractive virtual reality: an EEG and psychophysiology study. CyberPsychol. Behav. 9, 30–45 (2006). https://doi.org/10.1089/cpb.2006.9.30
Schilbach, L., Koubeissi, M.Z., David, N., Vogeley, K., Ritzl, E.K.: Being with virtual others: Studying social cognition in temporal lobe epilepsy. Epilepsy Behav. 11, 316–323 (2007). https://doi.org/10.1016/j.yebeh.2007.06.006
Seraglia, B., Gamberini, L., Priftis, K., Scatturin, P., Martinelli, M., Cutini, S.: An exploratory fNIRS study with immersive virtual reality: a new method for technical implementation. Front. Hum. Neurosci. 5 (2011). https://doi.org/10.3389/fnhum.2011.00176
Miller, E.K., Cohen, J.D.: An integrative theory of prefrontal cortex function. Annu. Rev. Neurosci. 24, 167–202 (2001). https://doi.org/10.1146/annurev.neuro.24.1.167
Calhoun, V.D., Pekar, J.J., McGinty, V.B., Adali, T., Watson, T.D., Pearlson, G.D.: Different activation dynamics in multiple neural systems during simulated driving. Hum. Brain Mapp. 16, 158–167 (2002). https://doi.org/10.1002/hbm.10032
Matsuda, G., Hiraki, K.: Sustained decrease in oxygenated hemoglobin during video games in the dorsal prefrontal cortex: a NIRS study of children. Neuroimage 29, 706–711 (2006). https://doi.org/10.1016/j.neuroimage.2005.08.019
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Kim, G., Jeon, J., Biocca, F. (2018). M.I.N.D. Brain Sensor Caps: Coupling Precise Brain Imaging to Virtual Reality Head-Mounted Displays. In: Schmorrow, D., Fidopiastis, C. (eds) Augmented Cognition: Intelligent Technologies. AC 2018. Lecture Notes in Computer Science(), vol 10915. Springer, Cham. https://doi.org/10.1007/978-3-319-91470-1_11
Download citation
DOI: https://doi.org/10.1007/978-3-319-91470-1_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-91469-5
Online ISBN: 978-3-319-91470-1
eBook Packages: Computer ScienceComputer Science (R0)