Cognitive Workload Assessment Based on the Tensorial Treatment of EEG Estimates of Cross-Frequency Phase Interactions

Abstract

The decoding of conscious experience, based on non-invasive measurements, has become feasible by tailoring machine learning techniques to analyse neuroimaging data. Recently, functional connectivity graphs (FCGs) have entered into the picture. In the related decoding scheme, FCGs are treated as unstructured data and, hence, their inherent format is overlooked. To alleviate this, tensor subspace analysis (TSA) is incorporated for the parsimonious representation of connectivity data. In addition to the particular methodological innovation, this work also makes a contribution at a conceptual level by encoding in FCGs cross-frequency coupling apart from the conventional frequency-specific interactions. Working memory related tasks, supported by networks oscillating at different frequencies, are good candidates for assessing the novel approach. We employed surface EEG recordings when the subjects were repeatedly performing a mental arithmetic task of five cognitive workload levels. For each trial, an FCG was constructed based on phase interactions within and between Frontal^θ and Parieto-Occipital^α2 neural activities, which are considered to reflect the function of two distinct working memory subsystems. Based on the TSA representation, a remarkably high correct-recognition-rate (96%) of the task difficulties was achieved using a standard classifier. The overall scheme is computational efficient and therefore potentially useful for real-time and personalized applications.

Introduction

Recent advances in neuroengineering have facilitated the use of EEG signals or “brain waves” for establishing efficient communication modes between humans and machines. There is a rapidly growing interest, in this meta brain computer interface (BCI) era, for methodologies that will not only help in restoring the communication and control capabilities of disabled people, but also will support novel applications for healthy subjects (e.g., cognitive monitoring).54 This is a multi-disciplinary research area in which engineers and computer scientists interact with experts from medicine, neuroscience and psychology. Moreover, the convenience of modern EEG machinery has amplified further the development of far-reaching applications, that extend well beyond BCI,1 ^, 43 such as neurofeedback systems18 ^, 36 and cognitive training.32 ^, 35

Neurofeedback-based games could be used for conserving or even boosting particular cognitive functions. EEG-monitoring during arithmetic tasks can be useful for understanding and treating related disorders like dyscalculia (where the subjects face great difficulties in learning and comprehending arithmetic operations),49 attention deficit hyperactivity disorder (ADHD)6 and autism spectrum disorders (ASD).51 A neurofeedback system, based on mental arithmetic tasks, aiming at improving the working performance of healthy adults has recently appeared.53

There is an important body of literature, in which emphasis is put on discriminating arithmetic tasks based on features extracted from the recorded EEG signal. Among others, fractal dimension (FD)53 and the coefficients of autoregressive model have been reported as efficient features for accomplishing the classification of mental arithmetic tasks.34 In a previous study, we attempted to differentiate arithmetic tasks by deriving network metric time series (NMTS).13 This was an approach deviating from all the published work on the topic, and in accordance with the emerging concept of exploiting functional connectivity and its dynamics in BCI.4 ^, 6 ^, 10 ^, 35 ^, 55

In general, functional connectivity captures deviations from statistical independence between distributed and often spatially remote neuronal units. Statistical dependence may be estimated by measuring correlation or covariance, spectral coherence or phase-locking. Functional connectivity estimates from any kind of synchronization measure and between every possible pair of EEG recording sites are employed to form functional connectivity graphs (FCGs). Of note, previous brain decoding studies based on FCGs typically treated the obtained FCGs as vectors in a high-dimensional space42 ^, 46 and handled in a standard pattern-analytic fashion. The main drawback of this approach is that it overlooks the inherent format of FCGs. In fact, since each FCG has a straightforward tabular representation, it can be considered as a second order tensor. The relationship stored in the corresponding matrix constitute important features, that reflect ordered associations between brain areas, and hence should be faithfully preserved in a low dimensional representation. To this end, we treat FCGs as tensors and employ tensor subspace analysis (TSA)23 as a suitable and convenient feature extraction strategy in this work.

Although two-way or multi-way tensors have already been used in neuroscience studies for improving classification accuracy,31 in most cases the focus was on extracting consistent patterns in frequency domain within a recording condition or from a population based original multichannel signals.8 ^, 16 ^, 33 To the best of our knowledge, there is only one fMRI study, in which dynamic FCGs were modeled via a group-based 3D-tensorial approach and in an attempt to associate particular connectivity patterns with different brain states.33 In a preliminary version of this work,17 we have realized a single-subject study aiming at differentiating deviant workload levels (two levels) based on connectivity patterns recovered from signals recorded over parieto-occipital brain areas. Based on the above promising results, we attempted here to correctly identify the workload levels among five different cognitive states by incorporating estimates of both intra and inter-frequency phase coupling within a single FCG representation.

Motivated by previous experimental findings regarding working memory (WM) subsystems26 ^, 28 ^, 44 and a recent modelling study,7 we sought a functional connectivity description that could encompass both inter and intra-frequency interactions. WM-involved tasks (e.g., complex arithmetic operations) require not only distinct functions such as storage (parieto-occipital regions) and central executive control (frontal regions), but also the coordination of the individual subtasks. Hence, it was necessary to adopt an integrated perspective. EEG rhythms (mostly θ and α) are known to appear over distributed brain regions during WM tasks, but the cross-frequency coupling between task-relevant regions has not been sufficiently studied. Here, we examined how θ and α ₂ frequency bands interact within and between the two WM subsystems located over frontal (F) and parieto-occipital (PO) areas correspondingly.26 ^, 44 ^, 52

In the present paper, we introduce and extensively evaluate a tensorial approach to encapsulating the characteristics of WM and describing their modulations induced by different levels of mental workload. TSA-based learning deduces the essence of functional connectivity that is hidden within the high-dimensional domain formed by all the estimates of pairwise coupling. In this way, the reliable quantification of mental workload can be performed efficiently and with minimal computational complexity.

To enhance both the introduction of tensorial treatment of FCG and also the incorporation of intra and inter frequency in a single FCG, we adopted the below comparison approach. Firstly, TSA based learning scheme was compared with vectorial treatment of original FCG with and without a feature extraction approach. Secondly, we compared the introduced intra/inter format of FCG with: (a) intra-frequency couplings independently in frontal and parieto-occipital brain regions in θ and α frequency band correspondingly and (b) with the incorporation of both subregions in a single FCG which communicate with either θ or α frequency band. The above approach will reveal the significance of cross-frequency coupling between the two subregions and the importance of studying each subsystem oscillating on its prominent frequency. The objective criterion in the aforementioned comparative approaches was the classification performance based on the adopted scheme.

The novelty and contribution of our work can be summarized via the following features: (1) A novel representation of intra-frequency and inter-frequency couplings from multi-site recordings. (2) A tensorial dimensionality-reduction technique for functional connectivity patterns that significantly boosts the performance of subsequent classification. (3) A refined detection of cognitive workload level that can be performed, at single-trial level, very fast and may be potentially useful in real-time applications.

Materials and Methods

Experimental Data

Subjects

The present study concerned 16 right-handed volunteers [9 males and 7 females, aged: 21–26 years, with a mean age of 21.5 (SD = 1.5) years], who were recruited from the National University of Singapore. All of the participants signed an informed consent form after the procedures were explained to them, had normal or corrected-to-normal vision and reported no history of verbal or non-verbal learning disability. The investigation was approved by the Institutional Review Board of the National University of Singapore.

EEG Recordings

We analysed functional connectivity patterns derived from EEG recordings in which the subjects were performing a mental arithmetic task (addition) with five levels of difficulty. EEG data were recorded from 64 channels at 256 Hz with an ActiveTwo Biosemi system and referenced using average reference. The experiment was segmented into blocks of 1 min, with rest periods of 30 s between blocks. Within a block, the delivered problems were on the same difficulty level. The blocks were presented in randomized order. The rest period had been introduced to avoid excessive fatigue, so that differences between block would be due to workload only. A schematic diagram of the experimental protocol is presented in Fig. 1.

Figure 1

A schematic depiction of the experimental timecourse. Blocks of stimuli (of same difficulty level) were presented in random order

The task consisted of summing mentally two numbers presented on the computer screen, retaining the result in memory and comparing it to a proposed answer, also presented on screen.41 There were 5 possible difficulty levels (denoted as Lv1, Lv2, etc.). At level 1, the problems consisted of summing two one-digit numbers, and each subsequent level included an extra digit such that at level 5 the problems consisted of summing two three-digit numbers. After three repetitions of all the difficulty levels, a slide-show of landscape pictures (one picture every 30 s) was presented for 5 min to allow the participant to relax. The whole session was then repeated [15 blocks (3 blocks per difficulty level, 3 × 5 = 15) of mental arithmetic and 5 min of relaxation between them].

With the gradually increased difficulty levels, the subject would need more time to conduct the calculation. Given that the duration for each block is constant across CWLs, the number of trials would significantly decrease with the increase of CWLs. The average number of trials across the entire set of subjects, for each CWL, was as follows (mean ± SD): CWL¹ = 225 ± 48, CWL² = 155 ± 33, CWL³ = 101 ± 32, CWL⁴ = 73 ± 24 and CWL⁵ = 55 ± 19.

Preprocessing

Artifact reduction was performed based on independent component analysis (ICA).11 After concatenating the responses from the entire set of blocks, we used EEGLAB11 to zero the components that were associated with artifactual activity from eyes, muscle, and cardiac interference. The estimated mixing matrix was used to reconstruct the multichannel signal from the rest independent components (ICs).

Determining Frequency-Bands and Recording-Sites of Interest

Taking into consideration previous works,26 ^– 28 ^, 38 ^, 44 ^, 47 which have identified brain waves within particular frequency bands and originated from specific brain regions to play an instrumental role in WM and mental arithmetic tasks, we confined our functional connectivity analysis within selected regions of frequency and spatial domain.

First, we experimentally verified that higher cognitive loads are mostly associated: (i) with increased power in θ (5–6 Hz) and α ₂ (10–13 Hz) bands over frontal sites bilaterally (Fig. 2a) and (ii) with increased θ and α ₂ power over parieto-occipital (PO) regions (Fig. 2b). Then we selected the corresponding sensors that showed this tendency for both rhythms (FZ, FP1, AF3, F3, F7, FC5, FC1, FC6, FC2, F4, F8, FP2, AF4, PZ, P7, P8, P5, P6, PO7, PO8, PO3, O1, OZ, O2, and PO4).

Figure 2

Signal POWER (averaged across subjects) of brain activity in θ band (5–6 Hz) and in α₂ band (10–13 Hz) as a function of workload level. The measurements from individual sensors have been averaged for frontal sites (a) and for parieto-occipital sites (b). Significant differences between successive levels are denoted with ‘*’ (Wilcoxon rank sum test p < 0.01; Bonferroni corrected p′ < 0.01/4)

Within and Cross-Frequency Phase Synchronization

Phase locking value (PLV) estimates of phase synchronization were derived, on a single-trial basis, based on a time window W that begun with the onset of stimulus (numbers to add appeared onscreen) and ended with the last peak of theta (6 Hz) cycle.¹

PLVs were calculated, in a pairwise fashion, from the brainwaves recorded at k, l sensor and associated respectively with the f1, f2 frequency band according to the following formula (provided in a generic setting):

$$ PLV\left( {{}^{f1}x_{k} ,{}^{f2}x_{l} } \right) = \left| {\frac{1}{{N_{\text{w}} \cdot N_{\text{s}} }}\sum\limits_{n = 1}^{{N_{\text{w}} }} {\sum\limits_{{s_{1} ,s_{2} }}^{{N_{\text{s}} }} {\exp \left( {i\left( {\phi_{k} \left( {s_{1} ,n} \right) - \phi_{l} \left( {s_{2} ,n} \right)} \right)} \right)} } } \right|. $$

(1)

The x _k (n) denotes a single-trial segment of N _w samples extracted from a single addition. The ϕ _k (s ₁, n) denotes the corresponding instantaneous phase measurements (for the N _s1 scales associated with the frequency band f1). The phases were calculated by means of wavelet transform using Morlet function with a Gaussian envelope in the time domain (characterized by standard deviation σ _t) and a complex exponential located around a centre frequency f and characterized by resolution σ _f in frequency domain.50

Within-frequency phase synchronization measures, PLV(^θ x _k , ^θ x _l ) and PLV(^α2 x _k , ^α2 x _l ), were computed so as to estimate functional connectivity among Frontal (F) brain regions in θ band and among Parieto-occipital (PO) brain regions in α ₂ band. Phase-phase cross-frequency coupling (CFC) measures, namely PLV(^θ x _k , ^α2 x _k ), were computed for every electrode from F and PO sites. In addition, PLV(^θ x _k , ^α2 x _l ) measures were also computed as indices of coordination between F and PO regions that would reflect cross-frequency coupling mechanisms (between distinct WM subsystems). All the aforementioned estimates of phase interaction were tabulated in a single matrix as shown in Fig. 3, which was the overall single-trial descriptor for the task related functional couplings.

Figure 3

Topographic illustration of the FCG construction that incorporates intra and inter frequency couplings between frontal (F) and parieto-occipital brain areas (PO). The matrix in the right-most end corresponds to the tabular equivalence of all the previous five topographies. Each colour corresponds to a specific type of functional connectivity (intra or inter frequency) in one of the two brain areas (F or PO). We defined intra and inter frequency couplings between and within brain regions with $PLV_{{{\text{Brain}}\;{\text{waves}}\;}}^{{{\text{Brain}}\;{\text{region}}}}$. The superscript denotes the brain region(s) over which the interactions took place while the subscript the frequency band(s) in which the interactions occur

The initial formation of functional connectivity patterns was followed by a statistical refinement step, which was applied on a single-subject basis and for each workload level independently. It is common practice to trim the initial estimates of functional connectivity so as to null out insignificant couplings that always appear due to random fluctuations in the time series. Based on a Rayleigh test for the uniformity of PLV values, we calculated the significance of each value (significance is calculated as p = exp(−N _trials $PLV_{{}}^{2}$).21 To correct for multiple testing, the false discovery rate (FDR) method was adopted.3 ^, 15 A threshold of significance was set such that the expected fraction of false positives was restricted to q ≤ 0.01. The PLV(x _k, x _l ) values surviving that thresholding were used to form the final single-trial FCGs. The percentage of PLV-values survived from this thesholding across the entire set of subjects, for each CWL, was as follows (mean ± SD): CWL¹ = 81.09 ± 9.82%, CWL² = 83.47 ± 10.14%, CWL³ = 84.54 ± 7.94%, CWL⁴ = 79.18 ± 10.24%, and CWL⁵ = 75.34 ± 7.53%.

TSA Learning of CFC Patterns

The single-trial FCGs were represented based on a recent methodology that blends ideas from multi-linear algebra and manifold data learning.23 In a nutshell, given some FCGs sampled from the space of functional connectivity patterns, we can approximate the underlying manifold by (i) first building a nearest neighbour graph that captures the proximity relationships among the FCGs and (ii) then deriving a tensor subspace that faithfully represents these relationships. TSA provides an optimal linear approximation to the FCG manifold.

Linear Dimensionality Reduction Problem in Tensor Space

Let $X \in \Re^{{n_{1} \times n_{2} }}$ be a FCG of size $n_{1} \times n_{2}$. Mathematically, X can be thought as a 2nd order tensor (or 2-tensor) in the tensor space $\Re^{{n_{1} }} \otimes \Re^{{n_{2} }}$. The generic problem of linear dimensionality reduction in the second order space is the following. Given a set of tensors (i.e., matrices) $X_{1} , \ldots ,X_{m} \in \Re^{{n_{1} }} \otimes \Re^{{n_{2} }}$ find two transformation matrices U of size $n_{1} \times l_{1}$ and V of $n_{2} \times l_{2}$ that maps these m tensors to a set of tensors $Y_{1} , \ldots ,Y_{m} \in \Re^{{_{l1} }} \otimes \Re^{{_{l2} }} (l_{1} < n_{1} ,l_{2} < n_{2} )$, such that Y _i “represents” X _i, where $Y_{\text{i}} = U^{\text{T}} X_{\text{i}} V$. The method is of particular interest in the special case where X ₁ , X ₂ ,…, X _m ∈ M and M is a nonlinear sub-manifold embedded in $\Re^{{n_{1} }} \otimes \Re^{{n_{2} }}$.

Optimal Linear Embedding

The “true” domain of FCGs most probably forms a nonlinear sub-manifold embedded in the ambient space of 2nd order tensors. With the adopted TSA, we attempt to find a linear subspace approximation to the sub-manifold in the sense of local isometry. The adopted technique is actually the tensorial counterpart of locality preserving projection (LPP).

Given a set of m tensors $\left\{ {X_{i} } \right\}_{i = 1:m}$, with each one being the tabular version of a single-trial FCG and having associated the cognitive load level as class label, TSA starts by building an [m × m] weight-matrix S that represents the nearest neighbour graph G among the tensors. In our implementation, the element S _ij was set as

$$ {S_{ij}} = \left\{ {\begin{array}{ll} {\exp \left( { - ||{X_i} - {X_j}|{|^2}/t} \right)}&{{\text{condition}}(1)} \\ 0&{{\text{otherwise}}} \end{array}} \right\} $$

(2)

The functional in Eq. (2) is known as heat kernel (here is employed with frobenius norm); t is a control-parameter usually referred as “radius of influence” and condition(1) states that X _i, X _j should share the same class label and anyone of them is among the κ-nearest neighbors of the other.

Then TSA seeks two transformation matrices U and V, such that when applied to each tensor to result in a mapping that would preserve the neighborhood relations encoded in G. Mathematically this is formulated in the form of the following objective function:

$$ \mathop {\hbox{min} }\limits_{U,V} \sum\limits_{\text{ij}} {\left\| {U^{\text{T}} X_{\text{i}} V - U^{\text{T}} X_{\text{j}} V} \right\|^{2} S_{\text{ij}} } , $$

(3)

that incurs a heavy penalty if neighbouring tensors X _i and X _j of the same class are mapped far apart. By denoting with D the diagonal matrix with elements $D_{\text{ii}} { = }\sum\nolimits_{\text{j}} {S_{\text{ij}} }$, the above optimization problem is reformulated as two coupled problems of eigenvector analysis23:

$$ \begin{array}{*{20}c} {\left( {D_{U} - S_{U} } \right)v = \lambda D_{U} v} \\ {D_{U} = \sum\nolimits_{\text{i}} {D_{\text{ii}} X_{\text{i}}^{\text{T}} UU^{\text{T}} X_{\text{i}} } } \\ {S_{U} = \sum\nolimits_{\text{ij}} {S_{\text{ij}} X_{\text{i}}^{\text{T}} UU^{\text{T}} X_{\text{j}} } } \\ \end{array} , $$

(4)

$$ \begin{array}{*{20}c} {\left( {D_{V} - S_{V} } \right)u = \lambda D_{V} u} \\ {D_{V} = \sum\nolimits_{\text{i}} {D_{\text{ii}} X_{\text{i}} VV^{T} X_{\text{i}}^{\text{T}} } } \\ {S_{V} = \sum\nolimits_{\text{ij}} {S_{\text{ij}} X_{\text{i}} VV^{\text{T}} X_{\text{j}}^{\text{T}} } } \\ \end{array} . $$

(5)

The optimal U should be the generalized eigenvectors of (D _V  − S _V , D _V ) and the optimal V should be generalized eigenvecrtors of (D _U  − S _U , D _U ). However, it is difficult to simultaneously compute the optimal U and V since the matrixes D _V , S _V , D _U , and S _U are not fixed. Both U and V were iteratively compute as follows. We first fix U, then V can be computed by solving the Eq. (4). Once V is obtained, U can be updated by solving the Eq. (5). The optimal U and V can then be obtained by iteratively computing the generalized eigenvectors of (4) and (5) (after initializing U with the identity matrix). Matrices D _U , D _V , D _U  − S _U , and D _V  − S _V are all symmetric and positive semi-definite. For further details of how to iteratively solve the objective function an interest reader can see the algorithmic steps in the Appendix.

In the present study, the dimensionality of the reduced tensors (i.e., the numbers of eigenvectors for the mapping Y _i = U ^T X _i V) was optimized, via cross-validation, for each subject independently so as to achieve the highest classification performance. The numbers of neighbors and the heat parameter were set in a similar way.

Learning Machines for the Assessment of Cognitive Workload

With the U and V matrices at hand, the quantification of CWL (from a given single-trial FCG) proceeds in the standard pattern recognition way of deriving the reduced tensor (i.e., the TSA-represented FCG) and comparing it with instances of known CWL. We found experimentally that the descriptive power of TSA was so high that even simple learning machines could be effective at this step.

The proposed scheme of CWL assessment incorporates the k-nearest-neighbors (k-NN) algorithm and employs the frobenius norm as the distance metric for matching the input FCG against the members of a database that includes single-trial FCGs of known classification (CWL).

To evaluate the performance of our approach, we followed a cross-validation scheme. The overall set of single-trial FCGs (including additions of all five difficulty levels) were randomly partitioned into two unequally sized subsets (corresponding to 90 and 10% respectively). The bigger one served as the training set (the database of FCGs of known level) and the smaller as the test set (trials for which the level had to be predicted). A correct recognition rate was estimated, as the proportion of trials in the test set for which the correct CWL was predicted. The cross-validation scheme was repeated 100 times and the mean value of the recognition rate was estimated for each subject separately.

Apart from the proposed scheme, which denoted as “TSA + k-NN”, we also employed a more conventional classification scheme denoted as “LDA + k-NN”. In the latter, the FCG-related tensors were first vectorized (i.e., represented as high dimensional vectors by traversing the corresponding matrices in a systematic way), then dimensionality was reduced via LDA20 (linear discriminant analysis) and classification was performed via standard k-NN algorithm (implementing majority vote strategy).

In all the above experiments, we defined the number k of nearest-neighbors as the one that maximizes the group-averaged classification performance.

Computation of FCGs and Statistical Analysis

All computations related with functional connectivity estimates and statistical analysis were performed using in-house software developed in Matlab (version R2011a, MathWorks) in a PC running Windows 7 with Intel i7 of 6 cores.

Results

Behavioral-Data Analysis

We first include the results from the statistical analysis of behavioral data, namely the subject’s response time and performance. Wilcoxon rank-sum test was employed for the statistical comparisons between consecutive difficulty-levels in terms of response time and also accuracy performance. As for all post hoc tests reported in this paper, corrections for multiple testing were incorporated based on Bonferonni adjustment.

Regarding response time, we detected a main effect of difficulty (p < 0.001, Bonferroni Corrected, p′ < 0.001/4) with higher difficulty leading to longer response times (Fig. 4a). The accuracy measurements showed a trend, associating the increased difficulty with deteriorating performance, but it did not survive the statistical testing (for p < 0.001) (Fig. 4b).

Figure4

Response times (a) and Accuracy scores (b) as a function of CWL

Sub-Graph Strength Level Across Cognitive Workloads

To illustrate the differences between the five different CWLs in terms of functional connectivity estimates, group-averaged topologies of FCGs were demonstrated and a statistical comparison over the average PLV across CWLs was performed. Figure 5 includes a topographical representation of the PLVs for the studied relationships (sites corresponding to F, PO brain regions and inter/within-frequency-band(s) couplings associated with θ and α ₂ brain waves). For each difficulty level, a group-averaged FCG is shown after thresholding each type of coupling separately. The three thresholds, corresponding to θ-coupling, α ₂-coupling and θ-α ₂ interactions, are indicated over distinct colorbars. They were estimated as the mean + SD from the measurements of the 5th level.

Figure 5

The topography of important functional couplings for the five cognitive workload levels. Three different thresholds have been applied based on the type of coupling (F/θ, PO/α ₂ cross-frequency coupling)

To examine the influence of CWL on the three types of functional coupling in a more quantitative manner, we performed statistical analysis (at group level) of the average PLV value estimated over subgraph interactions as this was defined for each one of the two subgroups of selected sites (F and PO ones). Figure 6 includes the obtained results. A progressive decrease of the $\overline{PLV}$ ² can be observed as we move from the easiest to the most difficult additions. All level-dependent differences, shown in Fig. 6a, were statistically significant (Wilcoxon rank sum test, p < 0.01; Bonferroni corrected p′ < 0.01/10). Additionally, the sub-graph $\overline{PLV}$ over F^θ areas was higher than the $\overline{PLV}$ over PΟ^α2 sites, as can be seen in Fig. 6b, This tendency was present at every cognitive workload level and statistical significant (Wilcoxon rank sum test, p < 0.01). Apart from these, within-frequency, measurements, we also examined the $\overline{PLV}$ for the cross-frequency couplings (data not shown here). The $\overline{PLV}$ was found decreasing during the three first levels, but then remained almost unaltered for the last two ones.

Figure 6

$\overline{PLV}^{{}}$ over frontal (F^θ) and parieto-occipital (PΟ^α2) brain regions as a function of difficulty level. (a) Statistical comparison of $\overline{PLV}^{{}}$ between every possible pair of cognitive workload levels (CWL) independently for F^θ and PΟ^α2 (*Wilcoxon rank sum test p < 0.01; Bonferroni corrected p′ < 0.01/10) (b) Statistical comparison of $\overline{PLV}^{{}}$ between F^θ and PΟ^α2 in every CWL (*Wilcoxon rank sum test p < 0.01; Bonferroni corrected p′ < 0.01/5)

Performance of Various Strategies for CWL-Assessment

To further establish that the reported improved performance is a combined effect of both the augmented FCG representation and the tensorial handling of the connectivity patterns, we compared our approach against a standard signal-analytic method (power indices of WM-relevant EEG rhythms) and a conventional pattern-analytic technique (LDA applied to vectorized versions of FCGs). Furthermore, to show that cross-frequency phenomena are indeed important in brain decoding, we measure the performance of CWL estimation based on isolated single-rhythm (either θ or α) measurements.

We present on this subsection results from a strategy that included signal characteristics and behavioral data. In the feature-extraction step, signal-power measurements (the power in θ band P _θ , the power in α ₂ band P _α2 and the corresponding ratio P _θ /P _α2) were measured from all selected channels and for each single-trial segment (i.e., an addition). This battery of features was further augmented by including the reaction-time. By means of a normalization step, that had been performed for each feature independently, all features were brought within [0–1] range. Subsequently, the dimensionality of the feature space was reduced via principal component analysis (PCA). Finally a k-NN classifier served as the predictor of CWL. The whole classification scheme was run for each subject independently and its performance was evaluated with the cross-validation scheme (described in “Learning Machines for the Assessment of Cognitive Workload” section). The measured performance, expressed as correct-recognition-rate and averaged across subjects, was found equal to 75.12 ± 9.35%.

We then present the results from strategies that incorporate functional-connectivity measurements for assessing the CWL during a single-trial (Table 1). Apart from our suggestion TSA + k-NN we also include, for comparison purposes, the results from LDA + k-NN and also the results from a variant that did not incorporate any cross-frequency phenomena. The group-averaged performance of TSA + k-NN scheme was 95.56 ± 2.35%. The corresponding correct-recognition-rate for the “vectorizing” approach, LDA + k-NN, was 85.44 ± 2.98%. When we restricted our TSA + k-NN strategy to the F^θ and PΟ^α2 subgraphs the measured group averaged performance was respectively θ: 73.45 ± 8.36% and α ₂: 75.39 ± 7.61%. The vectorial treatment of F^θ and PΟ^α2 with LDA + k-NN scheme was respectively θ: 69.32 ± 7.38% and α ₂: 70.61 ± 8.94% (Table 1).

Table 1 Classification performance (averaged across subjects) for original and bootstrap-sampling procedure with vectorial, tensorial and LDA treatments of FCGs over different frequency and spatial dependent representations

Additionally, we manipulated FCGs based on (F–PΟ)^θ and (F–PΟ)^α2 as vectors and as tensors in order to further justify the importance of estimating the connectivity profile of each brain sub-region oscillates on the preferred frequency and the role of CFC. The TSA + k-NN strategy gave a group-averaged classification performance of 81.12 ± 8.12 and 79.12 ± 7.96% for (F–PΟ)^θ and (F–PΟ)^α2 respectively. The LDA + k-NN strategy showed a group-averaged classification performance 75.43 ± 9.06 and 77.86 ± 6.92% for (F–PΟ)^θ and (F–PΟ)^α2 correspondingly. Table 1 summarizing the classification performance of vectorial, LDA and tensorial treatment of FCGs over different frequency and spatial dependent representations. The results clearly demonstrated the superiority of tensorial manipulation of FCGs compared to vectorial and also the importance of incorporating both intra and inter-frequency phase coupling within FCG. Additionally, both TSA and LDA dimensionality reduction techniques outperformed the treatment of original FCGs.

Finally we should mention that the performance of TSA analysis is influenced by the parameters t (heat kernel) and κ (κ-nearest neighbors). We searched over a range of possible values for these two parameters ($\kappa \in \left\{ {1,2, \ldots ,30} \right\}$ and $t \in \left\{ {1,2, \ldots 30} \right\}$). That experimentation showed that the classification performance was insensitive to the heat kernel parameter t and that parameter was set to 1. On the other hand, the optimal κ (subject-adapted value) was ranging between 6 and 10.

Handling the Classification of Unbalanced Datasets

Our classification scheme based on unbalanced datasets since the classification categories (CWLs) are not approximately equally represented. Unbalanced datasets can cause seriously negative effect on classification performance of machine learning algorithms. To deal with classification of unbalanced data sets in the current experiment, we extracted multiple random (partially overlapped) bootstrap-samples with size equal to size corresponding to the most difficult CWL 5 (with the lower number of trials), repeatedly measure the classification accuracy and derive the corresponding averaged score (+SD). The above procedure was repeated 100 times for each subject then we estimated the subject-average and finally the group-average + SD. The estimated classification performances are tabulated in Table 1.

Discussion

Based on our experimental validation, the tensorial treatment of FCGs increased the discriminability among the five Cognitive Workload Levels significantly (10% with respect to LDA). This notable improvement can be associated both with the fact that TSA considers FCGs as matrices (and not as vectors) and the ability of TSA to learn sufficiently the underlying manifold of connectivity measurements. Moreover, classification strategies based on power spectrum measurements or FCGs related with F^θ, PO^α2,(F–PO)^θ and (F–PO)^α2 subbands showed inferior performance. The whole analysis demonstrated the importance of tensorial treatment of FCG and the gain in information representation by incorporating cross-frequency synchronization phenomena (Table 1). The relationships between the row and column vectors of the connectivity matrix constitute important features and hence should be exploited for deriving a suitable low-dimensional representation. Additionally, the unstructured representation of all pairwise interactions, as a high-D vector, disrupts the natural representation of connectivity pattern. Considering the high classification performance that can be achieved in minimal time, the most appealing outcome of this study is that the obtained results were obtained based on scalp–recorded EEG (instead of measurements based on non-portable and costly equipment like MEG and fMRI).

Unbalanced datasets can often cause negative effects on the classification performance. In the present work, classification categories (different cognitive workload levels) are not approximately equally represented and a solution to this problem is to rebalance them artificially by oversampling and/or under-sampling.30 Here, we extracted multiple random (partially overlapped) bootstrap-samples with size equal to the most difficult level 5 (with the lowest number of trials), repeatedly measure the classification accuracy and derive to corresponding group-averaged score. The classification performances based on rebalanced datasets further support the remarkably high correct-recognition-rate of the task difficulty based on available initial training dataset.

Previous neuroimaging studies7 ^, 26 ^, 28 ^, 44 have revealed the significant contributions of specific brain regions to distinct processes related to WM-involved tasks like complex arithmetic operations. However, a clear picture about the coordination between the WM-related brain regions involved in such tasks44 has not been established yet.

Visual working memory (VWM) is the ability to maintain relevant to the task visual information for a short time period and is well-known to have a limited-capacity.37 Numerous studies identified frontal, parietal and visual regions as neural candidates for VWM maintenance.26 ^, 52 There is evidence that these distributed regions communicate through: (a) local within-frequency phase interactions and (b) long-range synchronization based on frequency-specific phase interactions of the F^θ and PO^α2 oscillations, as shown by human scalp-recorded (EEG) studies during several WM tasks, such as mental calculations.28 ^, 44 ^, 45 Taking advantage by the current neuroscience knowledge regarding the distributed WM network between two functions, namely, the central executive and storage buffer functions,2 ^, 7 ^, 26 we constructed FCG that tabulated both intra (local) and inter (long-range) frequency functional connectivity. An integrative view of memory-related phase synchronization mechanisms includes the phase-amplitude coupling, the m:n phase coupling and frequency restricted phase synchronization.19

The design of the experiment and the adopted methodology established a novel direction for many situations where brain activity in humans has to be assigned to different mental states.22 Mental states can be related to individual’s state of reading, writing, listening, and feeling. Brain decoding is required for dream reading based on visual imagery,24 for monitoring/enhancing human performance, for neurofeedback6 ^, 53 treatment in patients with psychological disorders etc.

Our attempt was more difficult compared to previous studies since the task differed on the number of digits of the addend numbers. We estimated the within and between frequency regional synchronizations and CFC of the θ and α ₂ rhythms demonstrate that the dynamic linkage between θ oscillations and modality-specific α ₂ oscillations mediates communication between the central executive functions and storage buffer functions in WM (Fig. 7).

Figure 7

Schematic illustration of the working memory (WM) manipulations of the visual representations. The external visual information is conveyed, in the modality-specific posterior region, by the α ₂ rhythm. The CFC synchronization between θ and α ₂ connect the frontal (F) executive regions with parieto-occipital (PO) regions where the visual and number related information is stored. Moreover, θ and α ₂ interact within both F and PO regions

Multivariate features from brain networks may provide increased discriminatory power for brain state classification12 instead of well-known univariate features (e.g., power spectrum, activated foci in fMRI). Machine learning and pattern analysis techniques were introduced and applied to brain connectivity graphs with main scope to decode the different brain states based on the functional connectivity patterns.42

Machine learning methods are an excellent choice for compensating the high variability in EEG when analyzing single trial data in real-time.39 Until now numerous studies have presented various preprocessing and classification schemes for efficient EEG-based brain-computer interfacing (BCI)39 and mental state monitoring applications.5 ^, 29 From the famous Berlin brain-computer interface (BBCI), which can be operated with minimal subject training and succeeds communication speeds of 6–8 letters/min39 to monitor of mental states and decoding of covert user states5 ^, 29 different strategies have been suggested but all of them based on time–frequency EEG analysis. In the present study, we exploit, for the first time, functional connectivity as a source of information that can lead to high-performance CWL decoding, and without inducing a computational load that will be prohibitive for real-time applications.

The execution time for computing the FCG over the selected 25 sensors was ~0.7 s in the case of CWL-5 (most delayed response time) and almost 0.1 s for acting upon a test-tensor with the pre-calculated U, V matrices (see Sect. “TSA Learning of CFC Patterns”). By taking advantage of the multi-core processor and adopting parallel computation the execution time for FCG computation drops to ~0.2 s and this results to a total time of ~0.3 s (for representing a single-trial) which is almost identical to the execution time of power spectrum estimations conventionally employed in BCI applications.18 ^, 36

In functional connectivity studies, it is important to address the degree of connectivity that can also be explained by volume conduction. Volume conduction is an electrical field produced at near the speed of light by an electrical dipole and thus exhibits approximately zero phase lag everywhere in the field.40 Zero phase delay is one of the important properties of volume conduction, when separated generators exhibit a stable phase difference of, for example, 30° then this can not be explained by volume conduction. We examined the distribution of instantaneous phase differences for many pairs of electrodes (in particular those of strongest phase coupling) and confirmed that it was not centered around 0 or ± π. Moreover checking for phase distribution that does not peak around 0 or π does not guarantee that phase coupling will not be affected by volume conduction but it may be less sensitive.9 ^, 48 In multichannel time–frequency EEG analyses it has been shown that there is a huge amount of 0 or π phase synchronization. To address this issue, we studied the phase difference derived from the entire set of trials over CWLs and subjects. By taking into account the whole group of subjects and the entire set of trials from the 5 CWLs, we derived a histogram of the phase differences (data not shown here). We mention here that less than 8% of phase differences was centered near 0 and π and almost 5% of phase differences was centered near ±90° difference. Additionally, in Fig. 6a $\overline{PLV}$ was presented over F^θ and PΟ^α2 brain regions as a function of difficulty level and clearly demonstrated significant differences between CWLs. The above trend could not be explained by volume conduction effects.

Our EEG study is the first one that classify five levels of WM Load with high performance based on the same task. Present study together with a previous preliminary work14 demonstrated the crucial role that phase synchrony plays in mental calculations. Our next goal is to built a unified classifier from all the subjects that can also differentiate the correct from wrong responses. In addition, we will attempt to incorporate both anatomical and functional/effective connectivity patterns and especially of the latter estimated in the source space to further improve the discriminating power of brain decoding in a future exploration.25

Appendix: Algorithmic steps of the TSA