Keywords

1 Introduction and Motivation

Brain networks operate over a broad range of spatial and temporal scales. Our ability to capture brain network activity is limited by the spatial and temporal resolution of the tools that are available. The most common noninvasive imaging modalities are blood-oxygenation-level-dependent (BOLD) functional magnetic resonance imaging (fMRI), magnetoencephalography (MEG), and electroencephalography (EEG). Each of these modalities provides a distinct, but limited, window onto brain network activity. Researchers are therefore interested in integrating information obtained from these different modalities in order to obtain a more detailed picture of true underlying brain dynamics.

The following chapter addresses some of the motivations, methodology, difficulties, and results of integrating EEG and fMRI. EEG and fMRI are widely used for multimodal integration since they make up for each other’s spatial and temporal limitations. EEG is sensitive to temporal dynamics on the millisecond timescale but has very limited spatial resolution. FMRI, in contrast, is sensitive to spatial differences on the order of millimeters but can only capture temporal changes on the order of seconds. MEG provides comparable spatial and temporal resolution to EEG with the additional advantage that magnetic source activity is not spatially filtered by the volume conduction properties of the scalp, skull, and brain. In addition, EEG and MEG are each sensitive to activity at comparable spatial scales or volumes of the cortex. However, MEG, due to the direction of magnetic field lines, is preferentially sensitive to cortical activity oriented tangential (or sulcal) to the scalp, while EEG is preferentially sensitive to radial (or gyral) activity (Cohen and Cuffin 1983). EEG and MEG each provide relatively distinct measures of cortical source activity. This motivates methodological approaches that can integrate information both between EEG-fMRI and between MEG-fMRI.

In addition to their complementary spatial and temporal sensitivities, fMRI and EEG differ in terms of the aspects of neural activity that they are most sensitive. EEG is sensitive to synchronous neural electrical potentials primarily along cortical gyri (Cohen and Cuffin 1983). BOLD fMRI, in contrast, is sensitive to neural metabolic processes via its coupling with changes in local blood oxygenation. EEG therefore provides a measure of neural activity directly along the cortical surface, while fMRI provides an indirect measure of neural activity throughout the entire brain.

The different spatiotemporal and neural sensitivities of fMRI and EEG raise caution in assuming a direct one-to-one correspondence between the two. It is a strong assumption that fMRI responses represent the spatial location of the observed EEG directly or that EEG responses directly reflect the temporal dynamics of responsive fMRI spatial locations. Instead EEG and fMRI are sensitive to the different aspects of neural activity that operate over their respective temporal and spatial scales. EEG may reflect the activity within only a subset of activated voxels, or EEG may reflect the activity within brain networks that covary in a complex manner with fMRI networks.

Different analysis approaches impose different assumptions on the relationship between EEG and fMRI. One line of research assumes a direct relationship between them, for example, by constraining EEG sources or inverse solutions to BOLD fMRI activations or structural MRI locations within radially oriented cortical gyri (e.g., Ahlfors and Simpson 2004; Lin et al. 2005). Alternatively, another line of research focuses on common temporal modulations within each modality irrespective of their spatial overlap. In this instance, fMRI voxels may be associated with EEG responses measured anywhere over the scalp and vice versa. The linked EEG-fMRI responses reveal brain networks that overlap after incorporating the broader range of spatial and temporal scales available within each modality (Siegel et al. 2012).

2 Physiological Considerations in EEG-fMRI

The sections below provide a broad overview of the physiology underlying EEG and BOLD fMRI responses. These physiological differences are an important consideration in EEG-fMRI study design and in the subsequent approach to EEG-fMRI analysis. The differences also provide important context for interpretation of the EEG-fMRI findings reviewed in the sections of the chapter that follow.

2.1 The Neural Basis of EEG

The first human EEG recordings were reported by Berger (1929) in his seminal paper. His initial observations were met with skepticism within the scientific community, and even Berger himself was wary of the findings. The initial skepticism was rightfully warranted, as it is difficult to imagine that small changes in brain activity would propagate through the head, generating measurable electrical potentials on the surface of the scalp.

EEG measures microvolt differences in scalp electric potentials that emerge from the aggregate activity of a large number of cortical pyramidal neurons. Synaptic inputs to pyramidal cells generate small sources and sinks along the cell membrane. These sources and sinks are space averaged over cortical areas that approximate the size of cortical columns. Pyramidal neurons are aligned parallel to each other along the cortex, forming a patch of neural tissue that approximates a dipole moment vector or more realistically a dipole layer . Scalp EEG is thought to reflect the average extracellular current generated from these pyramidal synaptic potentials. In order for the current to propagate to the scalp, the net charge of an individual patch of tissue must be oriented perpendicular to the scalp and must not be completely canceled out by opposing charges within neighboring tissues. A single EEG electrode reflects dynamic fluctuations in neural activity over an at least cm2 sized patch of the cortex. (Note that the voltage at a single electrode reflects the difference in potential between that electrode and a reference electrode. The electrode is commonly re-referenced to the average of all electrodes.) EEG responses can be distinguished as “local” or “global” by comparing the raw cortical potential with its spatially filtered representation (e.g., with surface Laplacian or current source density (CSD) analysis). Local sources are located underneath the electrode and are consistent with the assumption of a single dipole source. Global sources are present over many electrodes, correspond to either large areas of cortical activation or deep sources, and are inconsistent with the dipole assumptions of source localization (Nunez 2000; Srinivasan 2005).

Currents move in opposite directions at any given moment along certain locations of the scalp, forming sources and sinks. The overall current moving perpendicular in one direction along the scalp equals the current moving in the other direction. The movement of currents, and the spatial location of source and sinks, depends on the skull conductivity. Skull conductivity differs across the head due to differences in skull thickness and the nature of the bone tissue. Thus, scalp sources and sinks are more likely to appear over the locations with increased skull conductivity (Chauveau et al. 2004; Cuffin 1993; Nunez and Srinivasan 2006). These locations may not directly overlap with the location of cortical activity.

EEG responses reflect cortical potentials conducted through cerebrospinal fluid (CSF), the skull, and the scalp. The resistivity of these tissues contributes to the volume conduction properties of the head, effectively forming a head transfer function (Nunez and Srinivasan 2006). Theoretical studies suggest that these volume conduction properties emphasize large dipole layers over small dipole layers (Srinivasan et al. 1996). This low-pass spatial filtering property of the head effectively acts as a low-pass temporal filter as well, since larger areas of activation are associated with greater transmission delays and increased transmission delays render it difficult to sustain high-frequency oscillations (e.g., within gamma-band responses appearing at 40 Hz and above). Thus, low-frequency EEG responses between 1 and 12 Hz (e.g., incorporating the delta, theta, and alpha bands) are often global or widespread (Nunez and Srinivasan 2006).

In summary, it should be clear that there are a number of nuances to consider along with the statement that “EEG reflects synchronous cortical electrical fluctuations.” Notable nuances include the orientation of the cortical source, the degree in which cortical sources are cancelled out by neighboring tissues, the distance between the cortex and the electrode, and the choice of reference. In addition, the spatial location of EEG is influenced by differences in electrical conductivity over the skull, and the observed potentials reflect a low-pass spatially (and temporal) filtered representation of the underlying cortical sources. Some of these issues with EEG are absent in MEG recordings and are thus an important consideration when comparing findings from EEG-fMRI and MEG-fMRI.

2.2 The Neural Basis of BOLD fMRI

Increases in neural activity within a particular brain area result in increased blood flow to that same area. For example, tapping your finger for a few seconds will result in increased blood volume within vessels that supply the motor cortex. The enhanced blood flow response carries oxygen to the activated neural tissue, although the amount of oxygen available to the tissues exceeds the tissues’ needs. It has been said that the excessive increase in blood volume is akin to a gardener “watering the entire garden for the sake of one thirsty flower” (Malonek and Grinvald 1996).

The mechanism and function of the large increase in blood flow is a topic of ongoing research. One hypothesis is that the large increase in blood flow may help maintain a constant tissue oxygen pressure (pO2) (Buxton 2010). This hypothesis emphasizes the importance of pO2 in oxygen metabolism, which is interesting in light of the observation that tissue pO2 appears to approximate the level of pO2 in the atmosphere when oxygen metabolism first arose on earth. Regardless of the functional role, however, there is no debate that the large increase in blood flow is fortuitous, since it is a phenomenon on which the majority of functional neuroimaging studies are based.

Blood oxygenation levels serve as a proxy for underlying changes in neural activity. The relationship between neural activity and blood oxygenation is complex and indirect. Neural activity leads to an increase in cerebral oxygen metabolism (CMRO2) and an increase in cerebral blood flow (CBF). These two effects contribute to the measured fMRI response in opposite ways. A sudden increase in oxygen metabolism leads to a decrease in oxygenated hemoglobin, which, due to its magnetic properties, disrupts the magnetic field and reduces the BOLD fMRI response. The increase in CBF replaces deoxygenated hemoglobin with oxygenated hemoglobin, which reduces the magnetic field distortion and contributes to increased fMRI responses. (Note that the term “BOLD” is not technically accurate since the response depends upon deoxygenated hemoglobin.) The neural mechanisms that lead to decreased CMRO2 may differ somewhat from the mechanisms that lead to increased CBF. Relatedly, the ratio of CMRO2 and CBF changes can differ within the same brain area across subjects, across brain areas within a single subject, and even within the same brain area in response to different stimuli. This means that the observed percent signal change can differ in situations where neural activity is the same (for reviews see Buxton 2010; Gauthier and Fan 2018).

The BOLD response is most sensitive to aspects of neural activity that are associated with increased aerobic metabolism. Attwell and Laughlin (2001) estimate that the majority of the brain’s energy is devoted to restoring postsynaptic ion gradients. This supports the notion that BOLD fMRI more closely reflects synaptic integration than neural spike rate, as demonstrated empirically by stronger correlations between BOLD fMRI and the local field potential (LFP) than with microelectrode measures of spiking activity (Logothetis et al. 2001) (for exceptions see Ekstrom 2010). The sensitivity to synaptic integration means that the BOLD signal is sensitive to inputs to a particular area, without directly depending upon whether or not those inputs were effective at generating spikes (i.e., outputs) to other areas.

In addition, the observed BOLD response can be conceptualized as the neural metabolic process convolved with a hemodynamic response function (HRF). The HRF filter peaks about ∼6 s following the onset of the initial neural/metabolic event. The ∼6 s delay accounts for the sluggishness of blood flow changes in response to neural activity. It is this delay, and the limited sampling rate of fMRI, that contribute to the reduced temporal resolution of fMRI recordings.

2.3 Physiological Overlap Between EEG and fMRI

The finding that EEG and fMRI are sensitive to different aspects of neural activity does not make EEG-fMRI integration a futile endeavor. Instead, if EEG and fMRI completely overlapped in their neural and spatiotemporal sensitivities, then their integration would be redundant and pointless. Instead, linking the two provides an improved window onto the brain’s spatiotemporal dynamics by incorporating their non-overlapping range of spatial and temporal sensitivities. The resulting EEG-fMRI networks indicate that synaptic activity changes (coupled with metabolism and blood flow) at fMRI spatial locations are related to synchronous cortical potentials (from pyramidal cells) at certain EEG frequencies.

EEG and fMRI have many important commonalities. The sensitivity of fMRI to synaptic metabolism overlaps well with the sensitivity of EEG to synchronous cortical potentials. For example, both EEG and fMRI appear to overlap more with the low-frequency spectrum of multiunit activity (e.g., up to 250 Hz) compared to the high-frequency spectrum (e.g., from 500 to 1000 Hz). The low-frequency spectrum (i.e., the local field potential or LFP) is thought to represent integrative perisynaptic processes, while the high-frequency spectrum reflects “multiunit” spiking activity. The processes generating LFPs thus overlap with the processes generating EEG and the metabolic processes thought to drive BOLD fMRI (for reviews see Heeger and Ress 2002; Logothetis 2008) (for exceptions see Ekstrom 2010). However, the direct relation between fMRI and LFPs is less straightforward since spiking activity is often correlated with both fMRI and LFPs. This association is strengthened by cases where correspondence is observed between fMRI and LFP in the absence of spiking activity. Similar correspondence (e.g., between spiking activity and fMRI in the absence of LFPs) is rarely observed (Goense and Logothetis 2008; Logothetis et al. 2001). With regard to EEG, the physiologically interesting frequencies observed in LFPs overlap reasonably well with the frequencies commonly studied in EEG. For example, the characteristics of the alpha frequency band (e.g., 8–12 Hz) have also been examined in visual LFP recordings (Bollimunta et al. 2011; Mo et al. 2011).

An additional similarity between EEG and fMRI is that they are sensitive to responses that occur over similar volumes of brain tissue. It has been estimated that at least a cm2 cortex must be synchronously active to generate electrical activity observable on the scalp (Nunez and Srinivasan 2006). This volume of the cortex overlaps pretty well with the size of fMRI voxels, which typically range from about 27 to 64 mm3. The voxels are subsequently smoothed with their neighbors in order to enhance the signal-to-noise ratio (SNR) of fMRI responses. This smoothing also brings a closer correspondence between the effective volume of fMRI voxels and the minimum volume of cortex for EEG. In either case however, EEG and fMRI each represent an aggregate measure of activity from the collective dynamics that emerge from millions of neurons (Fig. 1).

Fig. 1
figure 1

EEG and fMRI are each primarily sensitive to synaptic activity. The factors that contribute to the non-overlap between EEG and fMRI are listed on top

Full size image

The aggregate window on brain activity provided by EEG and fMRI likely contributes to their utility in understanding cognition and perception. For example, cognition and perception are thought to emerge from the dynamic interactions between multiple brain areas (Bola and Sabel 2015; Siegel et al. 2012; Varela et al. 2001). These dynamic interactions likely overlap within the timescales of EEG, in the sense that the timescale of changes in our perceptual experience overlaps well with the timescales of fluctuations in EEG. EEG, for example, can separate the early visual response to sensory inputs from the subsequent visual response to the same input following reciprocal interactions with other brain areas (Lamme and Roelfsema 2000). FMRI provides a limited picture of these aspects of neural dynamics. However, fMRI is capable of measuring neural responses throughout the whole brain, providing a window on the brain areas that integrate together over second-by-second timescales.

Synchronization directly contributes to EEG and likely comprises synaptic integration processes that contribute to fMRI. The sensitivity of each measure to synaptic integration suggests that they also provide an aggregate measure of neural excitability, since neural excitability appears to coincide with the phase of synchronous activity (Klimesch et al. 2007). These coordinated bursts of activity help ensure that neurons influence other brain areas in a coordinated, efficient manner. Thus, the spatiotemporal scales and neural sensitivities of EEG and fMRI appear relevant to understand the brain’s integrative processes guiding cognition and perception.

3 Approaches to EEG-fMRI Integration

We now turn our focus to a subset of different approaches that have been utilized to integrate EEG and fMRI and review the findings revealed through each approach. Associations between the two modalities time courses are considered, as examined by correlation or general linear modeling (GLM) of the time courses, by deconvolution of the EEG and fMRI time courses, or by independent component analysis (ICA) of multi-subject EEG and fMRI datasets. These approaches are insensitive to whether there is a direct causal relationship between EEG and fMRI. Thus, there is no implicit assumption that EEG reflects a measure of the neural activity that contributes directly to the BOLD fMRI response.

3.1 Overview of Correlation and GLM-Based Findings

The most straightforward approach to integrating concurrent EEG and fMRI is by either correlating the time courses or by including the EEG time course as a predictor in a general linear model (GLM) analysis. In either case, the EEG time course is divided into non-overlapping epochs and converted to its frequency representation (e.g., by Fourier analysis), returning complex-valued coefficients for each frequency and epoch. The coefficients are absolute valued, returning the amplitude of each frequency within a given epoch. The EEG epochs are chosen such that each amplitude value (within a given frequency) corresponds in time to a concurrently recorded fMRI acquisition. Broadly, this approach examines whether fluctuations within a given EEG frequency are related to fluctuations within a given fMRI voxel.

Temporal delays between the EEG and fMRI time courses are not directly accounted for in traditional correlation or general linear model (GLM) analysis, since they focus on the instantaneous relationship between variables (see Labounek et al. 2019 for a recent exception). Therefore, the delay in the hemodynamic response must be accounted for prior to analysis. The characteristics of the hemodynamic delay are well described by the hemodynamic response function (HRF) (for review see Buxton et al. 2004). The assumption is that the BOLD fMRI response reflects a low-pass delayed representation of the underlying neural activity. The characteristics of the filter are incorporated within the HRF shape. For example, the HRF peaks at a delay of ∼6 s, which reflects the delayed increase in blood oxygenation following neural/metabolic events. The low-pass characteristic of the filter incorporates the temporal smearing that results from sluggish hemodynamics. These properties are accounted for by either convolving the EEG time course with the canonical HRF or by deconvolving the fMRI time course with the canonical HRF.

The initial EEG-fMRI studies focused on correlations between individual fMRI voxel time courses and the amplitude time course of EEG frequencies. This approach can generate an unmanageable number of statistical comparisons if univariate tests are conducted separately for each of thousands of fMRI voxels and for dozens of EEG electrodes and frequency bands. The number of statistical comparisons is typically reduced by focusing a priori on a subset of EEG frequency bands and/or on a subset of fMRI regions of interest (i.e., ROIs). Data decomposition approaches have also been quite successful at reducing the data to a few underlying sources (Eichele et al. 2009).

Initial EEG-fMRI studies focused on fMRI responses associated with the EEG alpha band (e.g., 8–12 Hz). The emphasis on the alpha band was motivated by its robust presence in individual recordings; alpha activity can be observed by an untrained experimenter in unprocessed EEG. The robust presence of alpha activity is particularly important in concurrent EEG-fMRI since the scanner environment introduces substantial artifacts in the EEG (for review see Ritter and Villringer 2006). The salience of alpha activity in EEG recordings likely contributes to their “salience” in the EEG literature, as decades of research have been conducted on the generators and characteristics of the EEG alpha rhythm. It was appropriate that the first EEG-fMRI studies focused on the alpha band as well.

Alpha oscillations appear predominantly over occipital electrodes and demonstrate a robust increase when individuals close their eyes, are drowsy, or engage in mental arithmetic (Klimesch et al. 2007). These tasks involve a lesser degree of visual cortical activity; thus, increased occipital alpha activity is thought to reflect cortical inactivity. This inactivity reduces the ability of visual areas to influence areas of the brain that support current cognitions or tasks. For example, increases in alpha activity are associated with reduced resting-state connectivity between early visual areas and the rest of the brain (Scheeringa et al. 2012). Increased visual inactivity is also synonymous with increased synchrony across visual areas, increased dependence across areas, and an overall reduction in visual complexity (Edelman and Tononi 2000). These processes are also likely associated with reduced cortical metabolism, and the sensitivity of BOLD fMRI to metabolic processes allowed the unique ability to test this theory.

Early EEG-fMRI studies have indeed demonstrated negative relationships between alpha activity and occipital, parietal, temporal, and fontal fMRI responses (Bridwell et al. 2013; de Munck et al. 2009; Goldman et al. 2002; Laufs et al. 2003; Sadaghiani et al. 2010; Scheeringa et al. 2011) and positive relationships between alpha and the thalamus (Bridwell et al. 2013; de Munck et al. 2009; Goldman et al. 2002). The negative correlation is consistent with the idea that increased alpha activity reflects reduced cortical metabolism and a subsequent reduction in the BOLD fMRI response. Equivalently, increased metabolism is associated with increased fMRI responses and a reduction in alpha. This interpretation was further supported by Moosmann et al. (2003) by demonstrating a negative relationship between changes in deoxy hemoglobin (measured by near-infrared spectroscopy (NIRS)) and alpha EEG. The main findings from selected “resting-state” EEG-fMRI studies are demonstrated in Table 1. The majority of studies demonstrate a negative relationship between fMRI and EEG alpha activity. Thus, this finding is one of the most consistent and reproduced findings in the EEG-fMRI literature. It can serve as a useful “sanity check” in EEG-fMRI.

Table 1 Main findings from select EEG-fMRI studies
Full size table

3.2 Background and Advantages of ICA in EEG-fMRI

One of the most difficult challenges in multimodal integration is extracting meaningful information from high-dimensional datasets. BOLD fMRI responses are obtained within tens of thousands of voxels, and each EEG epoch contains information within multiple frequency bands over dozens of electrodes. Integrating the EEG channel by frequency information with the fMRI voxel information with the traditional correlation or GLM approach ignores the rich structure within each dataset, is computationally demanding, and generates an unmanageable number of statistical comparisons. These limitations can be alleviated with blind source separation (BSS) approaches such as spectral ICA (Bridwell et al. 2013; Wu et al. 2010), principle component analysis (PCA), and temporal ICA (for a review see Makeig et al. 2004), as well as semi-BSS approaches such as functional source separation (FSS) (Porcaro et al. 2010, 2011). These approaches decompose each observation as the linear sum of a small number of underlying sources.

Among the data decomposition techniques described above, spatial ICA has demonstrated to be particularly informative and useful in fMRI analysis. For example, ICA (implemented with the infomax algorithm) can emphasize sparse-independent spatial fMRI maps, which aligns with the assumption that cognitive activation is sparse and distributed and with the sparse and spatially specific nature of cardiac and motion artifacts (McKeown et al. 1998). Temporal ICA is commonly utilized for EEG data, and the assumptions for temporal ICA align well with the theoretical generation of EEG. For example, the decomposition of a time course as a linear sum of independent temporal sources aligns well with the assumption that the response at a single electrode reflects a linear mixture of independent scalp sources (for review see Makeig et al. 2004). (Note that “sources” here refers to the independent sources estimated through ICA. These sources are different from the cortical “equivalent dipole sources” thought to generate EEG.) ICA can also be conducted on EEG spectra, revealing spectral sources that peak within characteristic EEG frequency bands (Bridwell et al. 2013). The stability of these sources has been evaluated using different BSS decomposition algorithms applied to real and realistic simulated data (Bridwell et al. 2018), and similar sources are present across different experimental paradigms (Labounek et al. 2018).

BSS approaches are particularly advantageous when EEG and/or fMRI are measured in the absence of an explicit task. For example, BSS algorithms such as ICA utilize the inherent structure in the data to extract underlying spatiotemporal activity patterns. These coherent patterns of activity likely result from activity within somewhat distinct brain modes or sources. The coherent nature of unique modes or sources suggests that they may also be described as distinct brain networks. The unique networks observed with ICA may demonstrate functionally distinct properties. For example, “resting-state” ICA can reveal sources which overlap with brain areas with greater activation during “internal” mental states (e.g., the so-called “default mode” areas). Other sources overlap with brain areas with greater activation during “external” attentive states (Corbetta et al. 2008). Of course, it is difficult or impossible to infer the functional role of networks that are present in the absence of explicit tasks since the individuals’ cognitions are unknown to the experimenter.

ICA is routinely used to extract independent spatial fMRI sources to link with concurrent EEG (for a review see Eichele et al. 2009). EEG is then associated with temporal fluctuations in fMRI spatial sources, rather than individual voxels or clusters. This is advantageous since it separates the voxel response at each point in time by the separate contribution of multiple independent sources. However, only a few EEG-fMRI studies have additionally conducted ICA on the EEG (Bridwell et al. 2013; Eichele et al. 2009; Wu et al. 2010; Yu et al. 2016). Thus, BOLD fMRI sources are often linked with EEG spectral information that potentially contains the combined contribution of multiple sources with overlapping frequency bands and spatial locations. A spectral EEG decomposition (with ICA) may reveal sources with distinct peaks that correspond to the traditional EEG frequency bands. This data-driven approach can validate the presence of distinct EEG frequency bands, improving the ability to link fMRI with EEG activity within each band.

It can be particularly important to decompose EEG spectra within the alpha band, as previous studies demonstrate that it contains the combined contribution of multiple distinct networks which may overlap spectrally and/or spatially. The 8–12 Hz alpha band has been subdivided by its upper and lower frequencies and overlaps in frequency with the central mu rhythm. These different alpha sources demonstrate distinct spatial topographies, spectral peaks, and/or sensitivities to experimental manipulation (Niedermeyer 1997; Nunez et al. 2001), and the average 8–12 Hz activity represents the combined contribution of these multiple independent sources. The presence of multiple sources with overlapping spectral characteristics is also suggested by the difficulty identifying the boundaries between EEG frequency bands within the average EEG spectrum and the presence of high correlations between the different frequency bands (de Munck et al. 2009; Mantini et al. 2007).

Conducting an independent group ICA within each modality can provide an improved measure of fMRI or EEG network activity while also helping to incorporate as much information as possible within each modality. The approach reduces the need to restrict the analysis to only a subset of fMRI networks (e.g., the default mode) or to restrict analysis to a subset of EEG electrodes or frequencies. An important consequence of this restriction is that it helps guarantee the frequency specificity of the results. Consider the negative relationship between alpha EEG and fMRI as an example. The presence of this relationship can be strengthened by demonstrating that similar relationships do not exist for other EEG frequency bands. For example, fluctuations in the alpha band likely reflect both broad fluctuations in the EEG spectral baseline and fluctuations specific to the alpha band. This possibility can be directly addressed by including additional frequencies as covariates in a GLM (de Munck et al. 2009) or multiple linear regression (e.g., PPI) (Scheeringa et al. 2012), by reporting results obtained separately for multiple frequencies, and/or by extracting frequency-specific sources with blind source separation (Bridwell et al. 2013; Scheeringa et al. 2008, 2011). In either case, considering multiple frequency bands helps acknowledge the full constellation of fMRI and EEG networks that may be present at any given moment (Mantini et al. 2007; Siegel et al. 2012).

3.3 Multi-subject Extensions of ICA

ICA can extract spatiotemporal patterns within EEG or fMRI data when individuals are not engaged in an explicit task (i.e., “during rest”). Generalization of these results across subjects can be more challenging with ICA than a traditional GLM analysis. For example, GLM analysis can be conducted on fMRI data for each individual subject, and the beta weight associated with the experimental time course is utilized as an independent observation in the second-level group analysis. Generalization across subjects is straightforward since beta weights can correspond to the same experimental condition across all subjects. ICA decomposes the multivariate fMRI data into a set of independent spatial sources and their associated time courses. Thus, ICA essentially estimates the unknown time courses of functionally distinct spatial maps (in accordance with the assumptions of the ICA algorithm) (Fig. 2). Researchers are then faced with the challenge of pairing up common sources across individuals. This problem can be addressed by incorporating information from multiple subjects within a single ICA decomposition and then examining the subject-specific parts (Beckmann and Smith 2005; Calhoun et al. 2001; Esposito et al. 2005; Guo and Pagnoni 2008; Schmithorst and Holland 2004). We focus here on the group ICA technique implemented in Calhoun et al. (2001) and in the GIFT software package (http://mialab.mrn.org/software/gift/).

Fig. 2
figure 2

Comparing the GLM and ICA for fMRI. The GLM estimates the contribution of each modeled time course to the observed data by deriving beta \( \left(\widehat{\beta}\right) \), the “activation map.” ICA models the observed data as a linear mixture of underlying spatially independent sources S. (Adapted from Calhoun et al. (2009))

Full size image

The typical ICA model assumes that each observation can be described as a linear mixture of independent sources. This can be demonstrated in an example with two observations represented by X = (x1, x2)T, begin generated from the following model:

$$ X= AS. $$

S = (s1, s2)T is the estimated sources, and A is the estimated mixing matrix. The mixing matrix describes the contribution of each source at each observation. ICA estimates the matrix inverse of A, which is denoted as the unmixing matrix W. The unmixing matrix applies a spatial transformation of the observations to arrive at the estimated sources:

$$ Y= WX, $$

which approximates the “true” sources S. ICA algorithms can emphasize the normality, independence, and complexity of the derived sources when estimating the unmixing matrix. For example, the infomax ICA algorithm iteratively changes the unmixing matrix in order to maximize the entropy of the estimated sources, which also maximizes their independence (Bell and Sejnowski 1995). Further details on ICA algorithms can be found in Stone (2004) and Hyvarinen et al. (2001).

Group ICA extends the ICA implementation described above in order to decompose data from multiple subjects. Group ICA estimates group sources based upon the aggregate group data and enables evaluation of individual subject differences via individual back-reconstructed components (Beckmann and Smith 2005; Calhoun et al. 2001; Erhardt et al. 2011). The individual data Xi is first compressed through principle components analysis (PCA), as expressed by

$$ {Y}_i={F}_i^{-1}{X}_i. $$

\( {F}_i^{-1} \) is the reducing matrix derived from PCA for subject i. The reduced data from M subjects is concatenated in order to form an aggregate group matrix which, in the case of fMRI, is [[time × M] by voxels]. The aggregate group matrix is compressed with PCA into the number of desired group components:

$$ Y={G}^{-1}\left[\begin{array}{c}{F}_1^{-1}{X}_1\\ {}\dots \\ {}{F}_M^{-1}X\end{array}\right]. $$

The reducing matrix G−1is a [components by [time × M]] matrix derived from PCA. The resulting matrix Y is decomposed through ICA (e.g., \( Y=\widehat{A}\,\widehat{S} \)) in order to derive the [component by voxel] matrix of group sources \( \widehat{S} \). The individual subject loadings (i.e., time courses for spatial ICA) are derived by matrix multiplication of the individual partition of the PCA reducing matrix Fi by the individual partition of the aggregate reducing matrix Gi and \( \widehat{A} \) (Calhoun and Adali 2012; Calhoun et al. 2001; Erhardt et al. 2011).

The group ICA steps described above implement ICA on a data matrix containing the aggregate data from all of the subjects. Spatial group ICA is commonly applied to fMRI data. In this instance the data are concatenated temporally such that each column corresponds to the same spatial location across subjects. This approach assumes common aggregate spatial maps across subjects while allowing flexibility in the estimated time courses for each subject.

3.4 Group ICA Applied to EEG and fMRI

Spatial group ICA has been particularly effective with fMRI data collected in the absence of tasks (for review see Calhoun et al. 2009) or in cases where the experimental models may not necessarily be known in advance (Calhoun et al. 2002). Group ICA has recently been extended to time-locked EEG (i.e., event-related potentials (ERPs)) analysis during tasks (Eichele et al. 2011) and spatiospectral EEG during rest. For example, Bridwell et al. (2013) decomposed 2D frequency by channel spectral maps into a set of group frequency by channel sources. The incorporation of frequency and channel information ensures that the decomposition utilizes as much of the data as possible, without restricting analysis to a single frequency band or electrode. The group sources correspond well with the characteristic frequency bands in EEG, and the temporal modulation of the group source is conceptually similar to the envelope of the response within the particular frequency band.

Group ICA can be conducted independently on EEG data and fMRI data collected concurrently. The data matrices are constructed so that the temporal modulations of the fMRI sources correspond in time with the temporal modulations of the concurrent EEG sources. EEG and fMRI can then be linked by focusing on relationships between the modulations within the two time courses. For example, the time courses may be correlated with each other after convolving the EEG time course with the canonical HRF or deconvolving the fMRI time course with a canonical HRF. This approach is less than optimal, however, as deviations in the assumption of a fixed HRF can reduce the sensitivity to instantaneous covariations between each modality. These assumptions can be relaxed by deconvolving the fMRI time course against the EEG time course, generating an estimated impulse response function (IRF). This approach treats the fMRI response as the output of the EEG response convolved with the unknown estimated filter (de Munck et al. 2009). If the neural activity measured with EEG overlaps with the neural activity that contributes to fMRI, then the estimated IRF will likely resemble the HRF. Estimation of the IRF from the data directly can account for the variation in HRF shape observed across individuals and over different brain regions (Aguirre et al. 1998; Handwerker et al. 2004; Steffener et al. 2010).

An advantage of applying group ICA independently to fMRI and EEG is that the number of possible statistical tests reduces from [voxels × electrodes × frequencies] to [fMRI sources × EEG sources]. Figure 3 indicates 56 group fMRI sources (in a) and 10 group EEG sources (in b). The results from all 560 comparisons are indicated in the [56 × 10] matrix in c. Positive associations (indicated by significant deviations in the estimated IRF) are indicated in white, and negative associations are indicated in black. In general the majority of positive associations are present within the lower (e.g., delta and theta) and upper (e.g., high beta and low gamma) EEG frequencies, while the negative associations were primarily restricted to two of the five alpha components.

Fig. 3
figure 3

Spatial fMRI components and their relationship to concurrent frequency by spatial EEG. 56 BOLD fMRI components are z-scored, thresholded, and displayed in (a). The spectrum and topography of the 2D EEG sources are indicated in (b). The relationship between the sources is indicated by the [fMRI source  ×  EEG source] matrix in (c). Significant positive associations are indicated along the white grayscale axis, and significant negative associations are indicated along the black grayscale axis. (Adapted from Bridwell et al. 2013)

Full size image

The widespread nature of the findings in Fig. 3 may be related to improved measurements of frequency-specific activity by decomposing underlying EEG sources at the group level and by relaxing the assumption of a fixed relationship (e.g., the assumption of a canonical HRF) between EEG and fMRI (as in de Munck et al. 2007, 2009). This is particularly applicable for the theta band, since estimated theta IRFs less clearly resemble the canonical HRF (de Munck et al. 2007) and theta IRFs tend to be more variable across subjects compared to the alpha band (de Munck et al. 2009).

Variability in the IRF can contribute to the variability of results observed in the literature. For example, the relationship between fMRI and theta EEG is less consistent than the relationship with the alpha band. Scheeringa et al. (2008) indicates that frontal theta activity is negatively correlated with many fMRI regions during rest, including inferior frontal, medial frontal, inferior parietal, and medial temporal areas. The negative correlation with theta and medial frontal areas is also supported by (Mizuhara et al. 2004). Figure 3 primarily indicates positive associations between theta and fMRI, which agrees with positive associations that have been reported while individuals perform mental arithmetic tasks (Mizuhara et al. 2004; Sammer et al. 2007).

4 Further Considerations

4.1 The Importance of Concurrent Recording

The fMRI environment introduces substantial artifacts within EEG recordings. The fluctuating magnetic field induces electric current in EEG, which appears as EPI artifacts. Current is also induced by movement of EEG wires within the static magnetic field with each heartbeat (i.e., the ballistocardiogram artifact). The EEG can also introduce artifacts within MRI (Ilhalainen et al. 2015; Klein et al. 2015; Luo and Glover 2012). Researchers must therefore consider whether the benefits of concurrent recordings outweigh the costs associated with these artifacts. It may sometimes be the case that concurrent recordings are not necessary. For example, EEG responses are often averaged together after time-locking to an external event (i.e., with ERP analysis). This approach discards the trial-by-trial fluctuations within each EEG epoch, which strongly reduces the need to measure fMRI concurrently. Instead, ERPs can be measured outside of the scanner environment, and fluctuations within ERPs across subjects can be related to fluctuations in fMRI maps across subjects (Calhoun et al. 2006). Fluctuations in ERP and fMRI maps across subjects may be directly compared with results from MEG, which can further improve the ability to spatiotemporally characterize brain activity (Plis et al. 2010).

Concurrent recordings are particularly advantageous when examining the epoch-by-epoch fluctuations within each modality. For example, this approach demonstrates that fluctuations within a particular EEG frequency are associated with fluctuations within a particular fMRI spatial location. Concurrent recordings are important in this instance, since they can reveal the characteristics in which the brain dynamically integrates distant spatial locations in cognition and behavior (Debener et al. 2006).

4.2 Intrinsic Connectivity Networks

The spatiotemporal patterns that emerge from EEG or fMRI data are thought to reflect the brain’s inherent structure or intrinsic connectivity. One might imagine that these networks describe a particular brain state and that this particular state is involved in an aspect of cognition such that fluctuations within that state are associated with fluctuations in that cognition. These networks can be identified in the absence of explicit tasks (e.g., during “rest”), and research is beginning to focus on how the networks identified during rest can potentially inform individuals’ ability to perform tasks (Carter et al. 2010; Deco et al. 2011).

The idea that “resting-state ” networks can predict performance is reasonable, since the cognitions that individuals experience during “rest” likely overlap with cognitions experienced during tasks. For example, attention is likely facilitated by enhanced activity within a subset of networks and suppressed activity within another subset. Tasks can promote attention, which promotes the ability to identify the subset of networks that facilitate attention. These same networks are likely present during “rest” since overlapping attentional processes likely occur during the “resting state.”

Thus, the “resting state” should not be thought of as inherently distinct from tasks. Instead, it simply reflects the broad range of cognitions that can emerge when individuals are unconstrained by an explicit task. Broadly, this supports the idea that the intrinsic connectivity networks identified during rest might inform the degree in which individuals utilize attention and memory processes that underlie tasks.

An additional implication, however, is that the relationship between EEG-fMRI networks observed during rest may overlap closely with the relationship between EEG-fMRI networks identified during tasks. This brings up a distinction between the extent in which an area is activated and the coupling of that area with other areas or modalities (O’Reilly et al. 2012). Consider the negative correlation between occipital fMRI voxels and EEG alpha. Individuals may perform tasks which suppress EEG alpha activity (e.g., reduces its extent), but the relationship between EEG alpha and occipital fMRI voxels would likely remain intact. In this instance, one would anticipate an overall reduction in EEG alpha and an overall increase in occipital fMRI responses. However, the relationship between alpha activity and occipital responses may remain the same, such that the two measures maintain the same correlation, and the estimated IRFs do not differ across the two conditions. This type of scenario is expected if the EEG-fMRI networks reflect the intrinsic structure of brain activity. Cognitive processes may modulate the extent in which a particular area is activated, but the inherent intrinsic structure would likely remain intact.

5 Conclusions

Combining the spatial information of fMRI and the spectral information of EEG can provide an improved picture of brain dynamics. These EEG-fMRI networks can be revealed even though each modality is sensitive to unique aspects of neural activity. The initial EEG-fMRI integration studies focused largely on fMRI responses associated with the EEG alpha band and utilized correlation and GLM-based approaches. Decomposing the information within each modality (e.g., with ICA) can provide an improved ability to isolate distinct networks, which can facilitate subsequent EEG-fMRI or MEG-fMRI integration. Within this context, it can be particularly important to account for differences in the hemodynamic response across individuals and across brain areas. The resulting EEG-fMRI networks can supplement findings in MEG-fMRI. Overall, combining information within each modality provides an improved ability to isolate brain networks, which may help clarify their potentially distinct roles in cognition and behavior.