Keywords

1 Introduction

1.1 Meta-Analysis in Neuroscience

There is an ever increasing number of scientific publications in many research fields in general, and in neuroscience in particular. Hundreds of articles are published every month, with a considerable amount devoted to functional magnetic resonance imaging (fMRI) ([7,12]). When comparing results obtained with a particular experimental setup with those reported in the existing literature, one may validate, integrate or confront different theories. This analysis is usually performed in a rather human-intensive manner, through the use of dedicated curators, e.g. [14,15]. The development of tools able to synthesise and aggregate such large-data can then be seen as crucial.

Meta-analysis of neuroscience research would clearly benefit from direct access to their original data sets. This is often not possible, due to the unavailability of such data. Yet, albeit of poorer quality, there is a plethora of summarising information, readily available in many published reports. Its analysis is the main topic of the current manuscript. That information is encoded both in text structures, as well as in image content, providing ample scope for mining information at various levels. The extraction of relevant information is not a simple task, and constitutes a major subject of information retrieval and data mining [11].

1.2 Previous Work

As stated above, previous approaches often used a considerable amount of curator work, with researchers reading from several sources, and extracting by hand the relevant information (cf., [14]). This severely limits the range of possible analyses. It is, therefore, of significant importance that robust automated information retrieving approaches be added to the current attempts to build functional neuro-atlases. A recent, fully automated approach was proposed by [21]. Their framework combines text-mining, meta-analysis and machine-learning techniques, to generate probabilistic mappings between cognitive and neural states. One drawback of this method is that it addresses only text mining, and requires the presence of activation coordinates in the articles analysed. Those peak-coordinates and some text tags are the only representation of the activations, which results in the discarding of valuable information from the neural activity.

We see our approach as a complementary way to tackle the problem, when image information, rather than text, is automatically harvested from published data.

1.3 Default Mode Network

An open field of research with increasing interest in neuroscience is the resting state and default mode networks (RSN & DMN, respectively). These networks comprise areas such as the occipital, temporal and frontal areas of the brain. They are active when the individual is not performing any goal-oriented task, and suppressed during activity [6,17]. In spite of the great attention to those networks, scientific research of brain’s “resting state” still poses various conceptual and methodological difficulties [19]. A commonly topic of study consists in investigating the differences and commonalities in the activity of healthy brains when compared to, e.g., Alzheimer or schizophrenic brains. Specifically, how different is the composition of RSN and DMN, in healthy and pathological brains, and how do these differences influence cognitive and functional performances.

1.4 Proposed Approach

In this chapter, we propose a complementary framework to text analysis, focusing instead on image information. It relies on the automatic extraction and characterisation of image information in fMRI literature. Such information often takes the form of activation/suppression of activity in the brain, in a variety of image settings, orientations and resolutions. This framework aims to open different means to building and improving functional atlases of the human brain, based solely on the large number of images published in neuroscientific articles.

We demonstrate the feasibility and results of our method in studies of the resting state and default mode networks, and highlight three outcomes of such research. The first is the identification of common neuronal activity across all subjects. Several regions are expected to participate in the DMN structures, in spite of possible existence of any of the aforementioned diseases. The second outcome focuses on differences between the activation patterns of healthy subjects and unhealthy ones, which can be explained with information already reported in articles within the data set used. Finally, we aim at identifying also variations in activity, not reported in the literature, and which could constitute evidence for proposing new research questions.

In the following sections, we will describe the procedure used for the extraction of reported fMRI images and subsequent mapping of functional activity patterns to a common brain template. Then we demonstrate the results obtained when mining information from a collection of articles related to the DMN. Using those results, we subsequently compare brain activity in healthy, Alzheimer and schizophrenic brains. We conclude the article with some remarks about the proposed approach, its limitations and possible future work.

2 Methodology

2.1 Data

The first step of our research consisted in the construction of a database of relevant publications. With this in mind, we searched for neuroscientific publications published online, in which the topic of discussion was related to the default mode network. This search was carried out using a keyword based search, with words such as DMN, Alzheimer, fMRI, cognitive impairment, Schizophrenia and resting state.

We gathered 183 articles in pdf format, from journals such as NeuroImage, Human Brain Mapping, Brain, Magnetic Resonance Imaging, PNAS and PLOS ONE. The time-frame for these articles ranged from early 2000 to June 2013. All papers were then separated according to the specificity of the analysis carried therein (see Table 1), distinguishing between studies on healthy brains (132), Alzheimer (29) and Schizophrenia (18) research.

Table 1 Number of articles used in this study, separated by type of study, as well as figures, images and blobs, obtained by our method
Full size table

2.2 fMRI Activity

Consider typical images of fMRI activity, as shown in Fig. 1. In a brief glance, it is easy to identify several features of relevance, such as the kind of section of the image (axial in this case, as opposed to sagittal or coronal), various anatomical features of the section, as well as the functional activity regions or ‘blobs’ within the section. We do that by relating the image to an internal representation of our anatomical and physiological knowledge of the brain. This relation takes into account physical and geometrical properties of the underlying structural image, as well as of the superimposed blob. In addition to the activity location, other features, such as intensity, area, perimeter or shape can be used to fully characterise the activity, c.f. ([1,18]). Other non-pictorial features, such as the text in the caption, could also be used to characterise said images.

Fig. 1
figure 1

Examples of images presented in fMRI reports. On the leftmost image (adapted from [13]), activity is present in the occipital, left temporal and frontal areas of the brain, and the activity is reported using the hot colour scale. The activity on the second image (adapted from [9]) is shown in three different uniform colours, while the third image (adapted from [22]) shows a combination of hot and cold colour scales, for increase and decrease of activity when compared to the reference

Full size image

In Fig. 1, one can also see the various reporting styles illustrated, including various underlying, gray-scale, structural images, colours and formats. The leftmost image shows a typical example where a slight increase of activity when compared to the reference corresponds to dark red while a big increase is depicted in bright yellow, which is typically called the hot colour scale. On the rightmost image, the decrease of activity when compared to the reference is shown in a gradation of blue, from dark to bright, corresponding to a small to big decrease. The image on the middle shows an example where the authors only chose to report the areas of difference in activation, without giving intensity information.

2.3 Image Extraction Procedure

In Fig. 2, we show a flowchart of our framework. We start by extracting figures from the PDF files of publications, using an open source command-line utility pdfimages, running in Linux.

Fig. 2
figure 2

Flowchart describing the blob mining procedure. First, figures are retrieved from articles (images adapted from Johnson et al.(2007)). This is then followed by the detection of possible objects containing fMRI activity reports. After processing and retrieval of these images, they are cleaned of artifacts, such as lines and text, allowing for a final stage of blob identification

Full size image

For each journal publication there is a pre-defined, common reporting style, but, as shown before, different authors produce their figures with different styles. They have non-homogeneous content, such as multiple image frames per figure, other plots, annotations or captions. Since we desire a clear image in order to accurately isolate the fMRI activity of interest, it is necessary to morphologically process those figures.

The first stage is the object identification. Many figures have a simple background colour, like black or white, but others have different colours, e.g., gray. Hence, the background colour needs to be detected, which is done through histogram and border analysis. The possible background colours are detected from the borders of the image, and the one with highest number of pixels is selected.

To detect different objects in a figure, and after background detection, figures are converted to black (background) and white (objects) colour. In those binary images, the white areas correspond to the smallest rectangle enclosing a object. Objects in the border of the respective figure, as well as those composed of only a few pixels are discarded. The next step is to analyse the images that are left inside the remaining objects. After extracting said images, we need to identify and extract the ones that correspond to fMRI reports. This is done using various properties, such as:

  • a minimum perimeter of the image, which we have set to 80 pixels, to allow a sufficient processing resolution;

  • a minimum and maximum number of image/background pixel ratio, between 0.1 and 97.5, to avoid non-brain images;

  • percentage of colour pixels in the image between 0 and 40 % of coloured pixels, filtering out non-fMRI images or images with activity all over the brain;

  • image aspect ratio between 0.66 and 1.6, typical of a brain image;

  • one image should occupy more than 50 % of the frame, to eliminate multiple images in the same object frame.

Regarding the last property, we repeated the object identification procedure when objects included several images, until no more images could be found.

In the example shown in Fig. 2, the object frame containing the figure colour map is discarded, due to the aspect ratio. Two of the brain images are also discarded since they don’t have colour present, therefore not being considered as originating from an fMRI study.

The following step removes undesired annotations. In Fig. 2, these correspond to coordinate axis as well as letters ‘L’ and ‘R’. This stage is done by removing all images inside the frame, except for the biggest one. Also any lines in 0 or 90 degree angles are removed, using the Hough transform [8,20] on each frame. Pixels belonging to vertical/horizontal lines that are present in more than two thirds of the height/width of the object are replaced with an average intensity of the surrounding pixels.

2.4 Volume and Section Identification

Once the activity images have been retrieved and cleaned, the type of template used in the images, i.e. volume type, and sections are identified, to estimate the three-dimensional coordinates of the activated regions. To represent the three dimensional changes in brain activity, views from three different planes are used to represent them in two dimensions. Thus we have axial sections, along the transversal plane that travels from the top of the brain to bottom; the sagittal section that travels along the median plane, from left to right; and the coronal section, along the frontal plane, that travels from front to back. To do a proper characterisation of the images, instead of focusing on the internal features of each section, the symmetry characteristics of the section shapes are used, as show in Fig. 3.

Fig. 3
figure 3

Section identification—Top column contains example fMRI activity images (after conversion to grey scale) and below them their corresponding binary masks. From left to right, we have axial, coronal and sagittal sections

Full size image

The images are again converted to binary images, thereby outlining the respective shape of the section. Simple symmetry allows for a suitable distinction between sections. One axial section is mostly symmetric about both the horizontal and vertical axis (Fig. 3a, d). The coronal section displays some symmetry only with respect to its vertical axis (Fig. 3b, e) while the sagittal section is asymmetric (Fig. 3c, f).

Most researchers map the activity changes found onto either SPM [10] or Colin [3] volume templates. Colin volumes contain higher resolution sections, when compared to SPM. Regarding the spatial separation between adjacent sections, SPM volume templates uses 2 mm, whereas that distance is 1mm for Colin templates.

To detect the volume type, one can use a complexity measure of the images. We used a Canny filter, [4], to detect the voxels corresponding to contrast edges. This is done for both template volumes, i.e. Colin and SPM, and for all the image slices from the section identified before. The volume template we select corresponds to the one with the minimal difference between the analysed image and the volume template images. This difference is calculated for the whole image and for a centred square with half the image size. We then average both values and use this as the difference measure.

To identify which slice of the template’s volume corresponds to the extracted image, we compare that image with all of the template’s slices. This comparison is performed using a combination of correlation and scale invariant feature transform (SIFT,  [16,20]). If there is a slice with a correlation of more than.9 with the extracted image, then that slice is selected. Otherwise, we select the slice which obtains the smallest distance of SIFT features as the correct mapping. Once this information is found, the complete coordinate set is identified for the reported image.

2.5 Blob Information Mapping

Once the geometrical considerations of the image have been dealt with, we can now characterise in the more detail the regions reported therein.

Activity regions are generated in response to stimulation. The properties of these regions largely define the fMRI activity and hence it is crucial that an analysis of the coloured blobs is carried out. Since we assume that only activations are color coded, these regions are easily segmented based on hue information (cf. ‘blob identification’ box in Fig. 2).

As mentioned before, the reporting style of different researchers can vary. This variety of reporting methods restricts the analysis that can be performed, since the same article can contain images with different colour scales. We tried to obtain intensity information from each fMRI image by using a colour map detection procedure, through histogram analysis. Since some images showed both increased and decreased activity, this step comprised from mild human intervention, aimed at fixing some wrongly detected colour maps. This was only applied to the rare cases where the automatic histogram analysis couldn’t detect the correct colour scale, and was performed rather easily.

Using the Colin brain template as a reference to our own reporting, we mapped all blob intensity information to their respective coordinates. We sum all intensities found in the data, for each voxel. Then those intensities are normalised to a scale from 0 to 1, where one corresponds to the highest possible common activity.

This produces a three-dimensional intensity map, where each voxel displays the intensity corresponding to the average activity in the data, for the respective voxels. Since this intensity map was built using two-dimensional images, we also performed a 3D smoothing, using a Gaussian ellipsoid with dimensions corresponding to 5 % of the template size.

In our reporting, we decided to use the jet colour scale for the summarising intensity map. There, colours go from dark blue to dark red, covering also green and yellow. One big difference between our scale and typical fMRI reports is that we don’t distinguish between increase or decrease of activity, when compared to a reference, but consider any coloured report as “interesting”. Therefore the dark blue corresponds to locations with very low reporting of activity (positive or negative) while dark red is used for locations with many reports. To avoid showing all the brain in dark blue, we show only intensities for locations where the number of reported blobs is more than 10 % of the total.

By superimposing this intensity map over the template volume, we can obtain a visual summary of all the results found in the articlesFootnote 1.

3 Results

3.1 Extracted Information

Table 1 shows how many articles, figures, images and blobs were found according to the publications analysed. Note that the number of samples for the unhealthy cases is quite small when compared to the healthy brains. This bias might affect the quality of the results, but the same problem would occur to any other meta-researcher, investigating brain activity in Alzheimer/Schizophrenia, due to the smaller sample of research dealing with these cases, when compared to the healthy controls.

Regarding the accuracy of the method, a simple visual inspection shows that once the volume and type of section are identified, the section coordinates were typically accurate within 1 voxel of distance. Also, the cleaning procedure of images mentioned in Sect.  2.3 doesn’t remove all artifacts from images, e.g., when the letters are inside the brain. Nonetheless we found that leftover artifacts rarely affected activity detection and subsequent mapping.

3.2 Meta-Analysis

After the compilation of all the results and the creation of the three-dimensional activity maps, one can perform analyses on the different types of brains studied.

3.2.1 Healthy Brain Activity

Figure 4 shows the brain activity reported for healthy subjects, displayed on axial section of the standard Colin reference. The highest areas of activity are the typical subsystems that compose the DMN: the posterior cingulate/precuneous, the medial pre-frontal cortex and the inferior parietal lobes. Note that, in the majority of the reports, including Alzheimer and Schizophrenia, most subjects presented the bulk of the activity in these major areas.

Fig. 4
figure 4

Average brain activity reported in publications dealing with healthy brains, superimposed on a Colin-based brain template, shown at various axial heights. Most of the activity is reported on the occipital, temporal and frontal areas of the brain, which correspond to the typical default mode network areas

Full size image

3.2.2 Alzheimer vs Healthy Brain Activity

We can now focus on the comparison between healthy, Alzheimer and Schizophrenia DMN activity, for example at axial height 114 of Colin’s standard brain (see Fig. 5). According to [2], one would expect that older brains have larger areas of activity than younger ones. We can see this in the posterior cingulate and in the inferior parietal lobes for Alzheimer when compared to the healthy brain image. On the other hand, the aged brain image shows somewhat less spread activity on the frontal lobe, when compared to the other areas of DMN. This seems counter-intuitive in light of the referred work. One may say that the lack of samples could cause this phenomenon, but our results seem rather consistent for the other areas. To find out a possible reason for this discrepancy, we can search for corroborating evidence in one of the articles analysed. In Fig. 1 of [5], there is a similar decrease in activity for aged brains, compared to healthier ones, confirming our own results.

Fig. 5
figure 5

Brain activity reported for healthy (a), Alzheimer (b) and schizophrenic (c) brains, at height 114 of the colin standard brain. The reports on brains affected by Alzheimer show a smaller intensity of activity in the pre-frontal cortex, when compared to the other DMN areas, unlike the reports for healthy and schizophrenic brains

Full size image

3.2.3 Schizophrenia vs Healthy Brain Activity

Another analysis that can be performed with our method relates to finding areas of the brain with different activities between unhealthy brains and healthy ones. In Fig. 6, one can see images for axial height 130, where publications dealing with healthy brains report a bigger area of activity in the posterior cingulate area (PCC), when compared with brains suffering from Alzheimer, and even more so on schizophrenic brains.

Fig. 6
figure 6

Brain activity reported for healthy (a), Alzheimer (b) and schizophrenic (c) brains, at height 130 of the colin standard brain. (a) image shows wider activation in the posterior cingulate area, suggesting that both Schizophrenia and Alzheimer might play a big role in this area of the brain

Full size image

3.2.4 Overall Comparison between Healthy, Alzheimer and Schizophrenia DMN Activity

To have a better overall perspective of the reported brain activations/deactivations, we can look at the 3D images of the intensity map, as depicted in Fig. 7. In this figure, we can clearly see that the areas reported on the healthy brains correspond exactly to the ones normally expected for studies of the DMN. On the brains suffering with Alzheimer, the intensity values decrease when compared to the healthy brain, as we suggested already in Fig. 5, although the areas reported are still the same as the from normal controls. Regarding the brains with Schizophrenia, we can see an area increase in the frontal region of the brain, while several smaller foci of activation appear, e.g., near the cerebellum.

Fig. 7
figure 7

Three dimensional images of brain activity reported for healthy (top row), Alzheimer (middle row) and schizophrenic (bottom row) brains. All reported images show the expected main DMN areas, although the reports on Alzheimer show a decreased intensity and the brains suffering with Schizophrenia report a more distributed activity pattern

Full size image

4 Discussion

We gathered more than 180 articles studying the default mode network, and analysed the images contained therein, in order to get a summarising overview of their results. Our main goal was to automatically map the results of studies reported by several researchers, onto a standard brain, and use this mapping to analyse the differences between healthy and unhealthy brains. This task would involve a tremendous amount of work and time if done by a human curator, whereas our method retrieves most information in a uniform and almost automatic manner.

The complete procedure is done in approximately 1 min per article (including human intervention if needed), while it takes 30–60 min when done by a curator, as in [15]. In that publication, the researchers went through 13 publications to obtain the information they desired. Using our method, not only it would save a considerable amount of manual work, it would enable them to find other fMRI studies related to the areas they are interested in.

Looking at the results, it seems clear that our method performs remarkably well, suggesting that it could be used to help creating a comprehensive functional brain atlas. Since we only performed a rough analysis of a particular research topic, we didn’t aim at a complete report of all brain activities that might be studied.

There are some problems with our approach, that also occur in other automatic data-mining approaches. First, by using only image information we are giving the same weight to all publications, irrespectively of the number of subjects studied. Furthermore, statistical thresholds and analysis methods vary in every publication, hence we cannot claim to make a thorough statistical analysis. Also, the number of articles dealing with the unhealthy cases is quite small when compared to the healthy brains. All these problems will affect quantitatively our analysis, although we may still draw valuable information from the data. We also expect their influence to decrease with an increasing number of analysed publications.

We showed that with a clear topic in mind, it is possible to obtain results of high relevance. As an example, we have seen that most reports on DMN, regardless of the health condition of the subjects show activity on the posterior cingulate/precuneous, the medial pre-frontal cortex and the inferior parietal lobes. On the other hand, the pre-frontal activity of Alzheimer subjects is shown to be spatially restricted. Corroborating evidence for this finding can be traced back to the original published reports. Due to the reduced sample statistics for the unhealthy brains, we can’t guarantee that there is a ‘real’ lack of activity, or just the absence of reports, but it suggests a possible area of investigation.

As stated before, there is a considerable variability in how each researcher displays their results. In the future, and to mitigate the lack of availability of original data, our method could be included in online submission systems for publication, after authors have uploaded their document. With minimal manual effort, the authors could validate the proposed summarising data, and hence improve the quality of the information gathered.

Lately there have been more and more efforts to increase data availability, either through common databases or by submitting the data at the same time as the article. Naturally, when available, this would allow for a much better analysis of the data, avoiding all the problems of detecting fMRI images or which colour scale they have. Nevertheless, these databases are still rather rare.

Despite the specificity of the method regarding fMRI images, we believe the principles behind it could be easily ported to other areas of investigation, such as weather reports or earthquake maps.

We hope to further refine our method by combining it with a text-mining approach, and test it in situations where there is either a clear agreement between different research reports, or a challenge between theories. The former is a key aspect to the construction of functional neuro-atlases, whereas the latter may lead to true findings in neuroscience.

5 Appendix–Articles Database

figure a
figure b