Keywords

1 Introduction

The functional mental state of operators is a determinant factor in task performance. It depends on various cognitive factors including mental fatigue, mental workload, and mental engagement, etc. In operational environments where high cognitive attention is required, these factors can contribute to lapse in attention directly or indirectly, which consequently lead to performance degradation [1]. Task-related behavioral measures like errors or response time, or subjective evaluations like NASA task load index are simple means in assessing performance variations [1, 2]. In recent years, many researchers have attempted to identify and evaluate reliable metrics in electroencephalography (EEG) to assess various cognitive states of operators [1, 3, 4].

Challenges remain in building portable, ease-to-use, and reliable EEG recording systems to be used in real-world situations and in developing novel computational algorithms in extracting useful information from recordings of such EEG systems. In comparison to classic EEG systems with electrodes using gel, wireless EEG systems designed to collect high-density EEG data (64, 128 or 256 channels) at high sampling frequencies using dry electrodes have been recently pursued [5]. These systems enable data collection in naturalistic environments with minimal interference to users.

In most EEG studies on assessing mental states and/or cognitive functions in real-world situations, frequency-specific oscillatory activities at channels of interest are major targets to be investigated in identifying patterns. Besides limiting the assessment to pre-defined channels, these methods usually emphasize only on temporal patterns and cannot reveal complete spatial patterns of activities of interests. Furthermore, channel-level EEG signals suffer from superimposition of multiple neural sources due to the volume conduction effect [6].

In this paper, we summarized three of our recent studies in monitoring mental/cognitive functions from lab bench experimental protocol, i.e. a speeded color-word matching task, to low-fidelity simulation protocol, i.e. training oriented air traffic control (ATC) task, and then to high-fidelity ATC simulation protocol. We implemented a spatial filtering method called independent component analysis (ICA) [7], based on the principle of statistical independence of source components in EEG data, which are potentially associated with distinct neural substrates. The method has been further advanced into the time-frequency ICA in one of our recent studies [6]. Our current results indicate that oscillatory activities in EEG components of interest in each study showed consistent spatial patterns across studies and indicated increasing dynamic patterns with the time-on-task effect.

2 Methods

2.1 Experimental Design

Figure 1 illustrates the three studies in the order of varied fidelity and realistic configurations. The details of each study are described as follows. Written informed consents were obtained from all subjects prior to their participation in the study.

Fig. 1.
figure 1figure 1

Illustration of tasks in three studies

Full size image

Study I. Ten healthy college students (20−29 years, all males) were recruited at the University of Oklahoma. Subjects performed a speeded color-word matching Stroop task [7] implemented using E-prime (for further details see [8]). They were required to judge the congruency between word meaning and ink color, and respond by pressing a button within 1400 ms. Each subject took part in two sessions of about 20 min each with a pseudo-randomized sequence of 390 trials, divided into 3 blocks.

Study II. Ten subjects (21−33 years, all males) were recruited at the University of Oklahoma. Subjects participated in two 2-h sessions of ATC tasks in low-fidelity simulation generated by CTEAM V2.0 [9, 10]. They were required to perform control actions (i.e., change direction of heading, speed and altitude level) on aircraft to regulate and maintain smooth air traffic flow using mouse.

Study III. Five certified professional controllers (54−60 years, all males) were recruited at FAA CAMI, Oklahoma City. Subjects performed 2-h ATC tasks in an high-fidelity en route environment on En route automation modernization (ERAM) interface [11]. They were asked to safely navigate aircrafts across designated airspace using slew-ball, keyboard and voice commands.

2.2 Data Analysis

In Study I and II, 128-channel EEG data were recorded at sample frequency of 250 Hz using an Amps 300 amplifier (Electrical Geodesics, Inc., OR, USA). In Study III, 64- channel EEG data were recorded at sample frequency of 1000 Hz using a BrainAmp amplifier (Brain Products GmbH, Munich, Germany).

All EEG data were offline filtered with a band-pass filter of 0.5−30 Hz. Preprocessing steps of bad channel removal, bad epoch removal and artifactual independent components (ICs) subtraction, were sequentially performed at each session data. In Study I, data in epochs from 1500 ms before to 2300 ms after stimulus onset were selected. In Study II and III, data points at peaks of global field power (GFP) data were selected for each session. The selected data from each study were combined and a group-wise ICA method was applied to identify ICs of interest [12]. Similar brain related IC patterns of interest among three studies were selected from individual studies based on spatial patterns. Then oscillatory power dynamics from each IC were calculated and transformed to dB to probe the time-on-task effect as follows. In Study I, an epoch of 700 ms pre-stimulus to stimulus onset for each trial was selected. The spectral powers of all epochs in each block were averaged and repeated measures analysis of variance (ANOVA) was performed using blocks as an independent variable. In Study II, each IC’s spectral-temporal dynamic was fitted with a linear regression line for each session and the detection of significant positive slopes of the fitted lines was tested via a binomial test. In Study III, the period of first (T1) and last (T2) 10 min of the task were selected and their spectral powers in each IC were compared using t test for individuals.

3 Results

3.1 ICs of Interest

Two distinct brain activities related ICs (named as the frontal IC and the parietal IC; Fig. 2) were similarly identified in terms of spatial and spectral patterns in each study. The frontal IC indicated a spectral peak in the theta band (5- < 8 Hz), while the parietal IC indicated a spectral peak in the alpha band (8 - < 13 Hz). Consequently, subsequent analysis was only performed on these frequency bands for each IC.

Fig. 2.
figure 2figure 2

The spatial and spectral patterns of two selected ICs

Full size image

3.2 Dynamics of EEG Components

In Study I, the oscillatory dynamics from all ICs augmented within each session, which revealed the tendency of significance in ANOVA tests (Table 1). For the frontal IC, t tests indicated a significant increase of the theta power from Block 1 to Block 3 (p = 0.01) and Block 2 to Block 3 (p = 0.02) in session 1 and from Block 1 to Block 3 (p = 0.04) in session 2. Similarly, for the parietal IC a significant increase of the alpha power was observed in both sessions from Block 1 to Block 3 (p < 0.001) and Block 2 to Block 3 (p < 0.005).

Table 1. Average and std of spectral powers (dB) of identified ICs in all blocks of two sessions for Study I and the detections of PRS in Study II.
Full size table

In Study II, the linear regression analysis indicated a significant (p < 0.05) regression model of positive slope (PRS) in most subjects for the frontal IC (8 out of 10 subjects in both sessions) and for the parietal IC (all subjects in session 1, and 8 out of 10 subjects in session 2), which were significant from a binomial test (Table 1).

In Study III, for the frontal IC, t tests indicated a significant increase of the theta power from T1 to T2 in subject 3 and 4, while a significant decrease in subject 2. For the parietal IC, a significant increase of the alpha power from T1 to T2 was detected in subjects 1, 4 and 5 (Table 2).

Table 2. Averaged spectral powers (dB) of identified ICs in two time periods (T1 and T2: first and last 10 min respectively).
Full size table

4 Discussion and Conclusion

In this paper, we presented our recent studies in monitoring mental/cognitive performance with tasks of different fidelity levels and at various experimental conditions. We explored IC-level oscillatory patterns from EEG signals for the feasibility of evaluating mental state, especially mental fatigue. Three studies with different cognitive demands were conducted, in which subjects were required to utilize sufficient cognitive reasoning and mental attention to accomplish the task goals. The tasks varied from simple single decision making in a repetitive cognitive task, to low- and high-fidelity task requiring processing of sensory information from multiple inputs and making complex cognitive decisions. The subjects in Study II were college students who are novice in ATC task, while the subjects in Study III were experienced ATC professionals. These studies from basic to real-world simulations and from novice to professionals enable us to develop technologies to inch closer to be deployable real-world task monitoring. In this paper, we presented the first evidence that similar brain patterns can be observed across various cognitive tasks, whether they are well-controlled or close to real-world situations. We also presented results related to the time-on-task effect, which is also known as the time-on-task mental fatigue. This is achieved with the novel EEG signal processing techniques (i.e. ICA), which is able to extract EEG component signals that arise from distinct brain regions. Variations in oscillatory dynamics of identified EEG components suggest the time-on-task effect in each study.

In the present study, two ICs representing distinct brain functions: cognitive control (i.e., the frontal IC) and attention (i.e., the parietal IC), were identified in all three studies. The frontal theta activity has been observed in many cognitive tasks that require concentration, attention, and short-term working memory [13], while the parietal alpha is thought to be critical in attention control mechanism during a complex human-computer interaction [14]. Two IC-based oscillatory dynamics suggest gradual increasing patterns in dominant spectral band with time on task in all studies. The increasing parietal alpha power suggests deteriorated attention levels in subjects while the increasing theta power indicates that more cognitive resources needed to fulfill cognitive demands from the tasks. Both phenomena have been reported as indicators of mental fatigue [15]. More complex dynamic patterns of the frontal theta power in professionals in Study III might suggest the effect of their training and working experience in battling the time-on-task mental fatigue in tasks that have zero tolerance to errors and failures. Furthermore, our recent data using engagement indices based on the ratio of the beta to the sum of alpha and theta band power showed consistent dynamic patterns across subjects [16], which suggests better biomarkers might be developed with information from multiple frequency bands.

Compared to Study I, both Studies II and III, which mimicked realistic ATC tasks and duration, allowed us to evaluate these phenomena in real-world condition. Our findings from these three studies indicate the consistent pattern of the time-on-task effect on different tasks, similar to some previous studies [15]. It is noted that, although we only reported data related to the time-on-task effect here, other behavioral metrics like response time, mouse clicks or key presses have been used to evaluate our EEG component data [6, 8, 10, 17, 18], in which significant correlations have been identified and indicated that these phenomena can be used to study other factors in monitoring mental/cognitive functions, e.g., mental workload and effort. In the future, more studies, including real ATC tasks, should be conducted to further explore the capability of the proposed technique and associated discoveries from its implementation.

EEG is a direct measure of neuronal electrical activities, and has been largely used to evaluate mental states of human being in neuroergonomics [4]. Compared to behavioral data, EEG data has the advantage that it can be obtained and analyzed continuously. Our recent efforts on computational data analysis method development have advanced the capability of using EEG in real-world situations. Together with recent non-prep sensor and portable and wireless hardware developments, this technology can be translated to in-field uses in many areas. In terms of monitoring mental/cognitive functions, our recent data suggest that brain processes including working memory, performance monitoring and decision-making from a brain network are required to accomplish cognitive tasks. Our technique reveals oscillatory dynamics of EEG components indicative of these brain processes. The present results indicate that the proposed technique can assess mental state of, such as, ATC operators in real-world conditions that need to be error free for public safety.