Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

The human connectome and its visualization are one of the hottest topics in the field of neuroscience and medical scientific visualization [35, 51, 58]. Hereby, the term connectome describes the entirety of all neural connections of the brain, addressed at different types [57] and scales. We focus on anatomical connectivity analysis at the scale of millimeters within the living brain, for which magnetic resonance imaging (MRI) and diffusion-weighted MRI (dMRI) are excellent imaging techniques. dMRI measures the strength of the diffusion of water molecules along multiple orientations and allows to estimate major white matter tracts, as diffusion is directed along cell structures defining those tracts. The process of reconstructing such tracts is called tractography and became an important tool in neuroscience over the past years. Many approaches have been proposed to extract neural tracts at high accuracy from dMRI data [32]. In addition, neuroscience experts need anatomical context to evaluate tractography data.

Among tractography algorithms, probabilistic diffusion tractography is a widely used and promising tool to probe for white matter tracts in diffusion data [32, 50]. Probabilistic tracts are volumetric scalar maps of connectivity scores ranging from zero to one, which relate a seed region or seed point to other areas in the brain. If the score is close to one there is good evidence that there is a physical connection of this point to the seed region. Otherwise, if the score is close to zero, there is high evidence that there is no physical connection between this point and the seed region. Therefore, the existence of a physical connection between every point of the brain to this seed region is expressed by this connectivity score. For a review on tractography and probabilistic tractography, including its limitations, see [6, 32].

Margulies et al. recently presented a very valuable overview of visualization techniques currently used in the neuroscience community for connectome visualization [39]. He state that there is a serious lack of visualizations combining image modalities. To fill this gap, we analyze the combination of various imaging modalities and probabilistic tractography data. Therefore, we survey typical datasets neuroscientists are working with and typical visualization techniques used in anatomical space. We conclude by giving recommendations on how to combine techniques to better analyze structural connectivity data.

2 Background and Related Work

We first want to provide an overview of visualization techniques for probabilistic tracts. Then, we relate our aim to recent survey articles and discuss techniques in terms of perception useful for our analysis. Finally, we discuss recent work based on the different data types, which is the basis for our classification and provides the structure of this paper.

2.1 Visualizing Probabilistic Tracts

The most complete and up to date overview of visualization techniques for probabilistic tractography data is given by Margulies [39]. The techniques working in anatomical space can be classified into two groups: three-dimensional or slice-based. Three-dimensional techniques typically employ three-dimensional structures, such as surface estimations, and slice-based techniques, such as axis planar cutting planes, using only two dimensions for representation.

Slice based techniques typically superimpose some texture or colormap of the tract over anatomical context. A common technique is the heat-map (also known as hot-iron colormap), which can be used as overlay over structural MRI data [49], see Fig. 1a. There, the color is made transparent in case of no connectivity, to dark red for low connectivity and then gradually to bright yellow for highest connectivity.

Fig. 1
figure 1

Overview of visualization techniques used for visualizing probabilistic diffusion tractography data. All figures render the same tract using the same camera position. (a) Heat-map of the probabilistic tract over T1 image. (b) LIC of major diffusion orientation filtered by the probabilistic tract. (c) Same as (b) but with T1 background and colored LIC with heat-map. (d) Red Fiber-Stipples with boundary curves as context. (e) DVR of the tract, with transfer function mapping low scores to white shadow and the core of the tract to opaque red. (f) Isosurface, with isovalue set to 50 %. (g) Three nested isosurfaces. (h) Probabilistic fibers defining the connectivity scores

Full size image

Line integral convolution (LIC) [11], generates a texture from a vector field by smearing a noise texture along integral lines. This can be used to depict major diffusion directions in regions of the tract (Fig. 1b and c). Tract density imaging (TDI) [12] produces similar results by partially projecting fibers onto a slice.

Fiber-Stippling [26, 29], generates small line stipples to depict diffusion direction. The connectivity score is encoded in stipple density and opacity. Anatomical context is given as contours as well as usual structural MRI underlay, see Fig. 1d. We consider Fiber-Stipples an important tool because it was used in a Technical Spotlight in the European Journal of Neuroscience to reveal the structural bases of cerebellar networks within the basal ganglia which were previously only known from studies using transneuronal virus tracers in macaque monkeys [50].

In the category of three-dimensional techniques, direct volume rendering (DVR) assigns a certain color and opacity to each volume sample which are then compiled into one image. Hereby, the color mapping is modeled by a transfer function [60], see Fig. 1e for an example. Another three-dimensional technique uses isosurfaces to show outlines of the volumetric connectivity score. The isosurfaces are typically represented as triangle meshes and depict only a particular connectivity score (isovalue) of the tract, see Fig. 1f. Berres et al. [7] proposed nested isosurfaces to display multiple values, see Fig. 1g. Brecheisen et al. [10] propose similar results, but with an illustrative rendering. Some probabilistic tracking algorithms repeatedly start a randomized deterministic tracking from the seed region. The number of tracts hitting a voxel is similar to the connectivity score. Polylines from such trackings are called probabilistic fibers [16], see Fig. 1h.

2.2 Surveys

For identifying important image modalities and most commonly used visualization techniques we consulted recent surveys [1, 38, 39, 55]. As probabilistic diffusion tractography originates from dMRI and structural MRI, especially other MRI modalities, such as functional MRI (fMRI) or magnetic resonance angiography (MRA) [59], might be of interest. While fMRI gives information about functional activity and connectivity, MRA data provides structural information of blood vessels. Each image acquisition technique generates different raw data from which additional data might be derived. The surveys state that most common visualization techniques in anatomical space use textured (axis planar) slices, glyphs, polylines, or surfaces.

2.3 Color and Perception

When combining different techniques, the perception of shape and color plays an important role. Ware [61] provides a good overview on the use and perception of colors. In addition, he also provides useful guidelines on choosing colors in various settings. For example, Ware suggests to use Bauer’s method [5] to compute the most discriminative color to a given set of fixed colors, cf. Paragraph “Fixed Colors”. Additionally Ware propose to enhance luminescence when chromatic contrast is exhaused, cf. Sect. 4.1.1, or outlining glyphs, cf. Fig. 6, to further increase visual contrast. Additionally, Kapri et al. [60] evaluates optimal coloring for probabilistic tract in virtual environment regarding uncertainty.

2.4 Data

For a systematic investigation, we first need an overview of what data is available and of which visualization techniques are typically applied. In the following paragraphs we briefly introduce various imaging modalities and types of data neuroscientists are working with. We then collect information on typical visualization techniques and summarize the information in Table 1 which serves as basis for our study. Even though they have proven to be very helpful to neuroscientists, to focus this paper, we exclude visualization techniques outside of anatomical space such as dendrograms for hierarchical agglomerative clusterings, colored matrices depicting functional correlation, and radial graphs with force-directed edge bundling, cf. [9, 34, 39].

Table 1 Each row corresponds to a specific data type, named in the first column. The remaining columns then identify common visualization techniques used for this data type
Full size table

2.4.1 CT and Structural MRI

Usually computed tomography (CT) [36] and structural MRI data (T1,T2) [14] are displayed on axis-aligned slices (coronal, sagittal, or axial) using a monochrome texture representing the measured data. Since small nuances of gray are important (see Fig. 2), contrast and brightness manipulators are often used for contrast enhancement. Besides monochrome textures, different tissues (e.g. gray matter) or structures (e.g. lesions, tumors, blood vessels, ventricular system) need to be highlighted. Contrast improvement can be done using contrast agents during the measurement, but some structures that are visible in the raw data can be extracted in a postprocessing step. T1 data, for example, is also rendered on inflated cortical surface maps or used to segment brain structures such as the ventricular system. Furthermore, MRA data is used to reconstruct three-dimensional surfaces as well. As segmentations and surfaces are derived data, we will discuss them separately in Sects. 2.4.6 and 2.4.8.

Fig. 2
figure 2

These figures show the difficulty of reading structural MRI data at the example of the internal capsule (IC) which consists of mainly three parts: Its anterior limb, posterior limb, and the genu. In (a) red labels indicate their positions on an axial slice of a T2 image. (b) Same subject, same axial slice position but with T1 data textured. With prior knowledge from (a) you hopefully grasp the IC in (b) as well

Full size image

2.4.2 Diffusion Images

Diffusion imaging measures diffusion strength along multiple orientations, also called gradients or angles. For a typical high angular diffusion image (HARDI), there are more than 30 different gradients, each resulting in a three-dimensional scalar image representing the diffusion strength along its orientation [67]. Such data is rarely visualized directly (if so, then on slices) and is typically used for image registration processes, artifact reduction schemes, and, most importantly, to construct data representing some model of the diffusion process.

The diffusion tensor is one of the oldest and most popular diffusion models [4, 32] and founded the diffusion tensor imaging (DTI). Although there are limitations to this model, for example not being able to resolve crossings of even major white matter bundles, it still has a very big (clinical) importance [1]. Compared to other diffusion models, this model is a rather simplistic approach, which also might be a reason for its ongoing success. The most common visualization of all diffusion models is using glyphs [3, 30], often placed on axis-aligned slices. One of the biggest limitations of DTI is that it is not able to resolve fiber crossings locally. Even worse, Wedeen found white matter organization to be highly affected by fiber crossings [63]. This triggered a new demand of more complex diffusion models such as diffusion spectrum imaging (DSI) or q-ball imaging (QBI), cf. [32]. Nonetheless their direct visualizations remain glyph centric.

One of the most important parts of DTI is the major diffusion orientation computed from the eigensystem of the second order tensor. A straight-forward visualization technique is to map orientations into a color space. The most popular coloring technique is the RGB-coloring scheme [48]. It maps a three-dimensional orientation vector to RGB color space, so superior–inferior orientations mapped to blue, left–right to red and finally anterior–posterior orientations are mapped to green. Besides this, other coloring schemes have been developed [15]. Other approaches depict the major diffusion orientation using gray-scale textures generated from LIC or by a small line per voxel, which is called vector-plot [24].

Diffusion indices or biomarkers are important scalar values generally derived from diffusion models. They have been successfully used for certain diagnoses and brain development studies and hence are important for brain analysis. The most popular diffusion indices are derived from the rather simplistic diffusion tensor and have been used in various clinical studies to study specific diseases, aging, and developmental processes, e.g. [17, 53]. For example the fractional anisotropy (FA), apparent diffusion coefficient (ADC), mean diffusivity (MD) or radial diffusivity (RD). Beside DTI based diffusion indices, new generalized diffusion indices, such as peak fractional anisotropy (PFA), generalized anisotropy (GA), and generalized fractional anisotropy (GFA), have been conceived [23].

2.4.3 Tractography

As described earlier, tractography is the process of reconstructing neural tracts out of diffusion data. The algorithms can be classified as follows: Deterministic tractography methods generate polylines [21] whereas probabilistic tractography methods often generate volumetric data. For an overview on probabilistic tractography, we refer to Sects. 1 and 2.1. In the following we focus on deterministic tractography.

Deterministic tracts are often visualized as pure lines or tubes [39] and might be colored globally or locally. Global coloring assigns the color of its main orientation to a polyline, while local coloring schemes assign individual colors to each line segment. Typical coloring schemes are the same as for directional color coding mentioned earlier in Sect. 2.4.2. To increase depth-perception, polylines may be enhanced with illumination and shadows [20]. Besides rendering the lines directly, there are also techniques super-sampling the density of deterministic tracts producing LIC-like textures such as TDI.

2.4.4 Functional Activity

Functional MRI (fMRI) is used to locate neural activity indirectly through metabolism activity measured by the blood oxygen level dependent (BOLD) factor [52]. The resulting data are scalar maps indicating regions of brain activity. This activity is often rendered in the same way as probabilistic tracts by using heat-map or rainbow colormaps on slices with structural MRI background.

Electroencephalography (EEG) [47] and Magnetoencephalography (MEG) [2] are other very popular techniques for measuring brain activity. These techniques capture time-dependent electric or magnetic field strengths, respectively, with sensors placed along the scalp. The measurement delivers plots of electric or magnetic activity over time, induced by neural activity inside of the brain. The spatial location of the activation areas at a time can be computed by so-called source reconstruction. This reconstruction leads to scalar activation maps that, from a visualization point of view, can be treated similarly to fMRI activation maps and are therefore not listed separately in Table 1.

2.4.5 Functional Connectivity

Functional connectivity typically refers to a statistical concept of the correlation between spatially distinct units e.g. between cortex areas. The most common visualization within anatomical space is to connect cortex regions with a strong correlation by a straight line or splines. Other approaches try to sample glyphs on the gray matter–white matter interface denoting the anatomical direction of their functional correlations [9].

2.4.6 Segmentation

One of the big goals of neuroscience is to classify brain regions based on functional units, which are important on the cortex as well as in white matter tissues. For example finding a cortex parcellation that decomposes the cortex into functional units or finding white matter clustering schemes that decompose white matter tracts into major white matter fiber bundles are both hot topics in the community. The visualization techniques labeling different brain regions are usually limited to employ most discriminative colors to each single unit [27, 66]. Besides coloring, the visualization of segmentation data uses three-dimensional surfaces [25, 31] as well as slice-based techniques [62]. For highlighting segmentation results sometimes also contours are used to outline cortex areas or the gray matter–white matter interface.

2.4.7 PET and SPECT

Positron emission tomography (PET) or single-photon emission computed tomography (SPECT) measures photons from the radioactive decay of an injected radionuclide, which has been aggregated by different metabolic systems. Other applications are the analysis of brain hemodynamics by tracking chemicals in blood flow [13] or so called amyloid imaging, which visualizes the presence of senile plaques linked to Alzheimer’s disease (AD) [45]. The so gained three-dimensional images localize areas of high tracer presence and are typically visualized in anatomical space on slices using rainbow colormaps [56].

2.4.8 Surface Data

Neuroscientists frequently use surface data, ranging from cortex estimations [22, 65], over boundaries of lesions or tumors, the ventricular system, or the blood system [46] to boundary surfaces on brain segmentations. Most visualization systems represent these surfaces using triangular mesh structures or solid surface renderings using colored and illuminated facets.

3 Methods

The data and visualization techniques we are working with in this paper are summarized in Table 1. Even though the list of data types might be incomplete and might lack data used to answer specific questions, to the best of our knowledge, it lists the most popular data types. Other imaging techniques may fall into a similar category as one of the listed modalities. Also please note that this table only lists common visualization techniques, not all possible techniques. This is also why there is no column for direct volume rendering, as it is rarely used in the neuroscience community. We compiled the table based on recent publications, which have been already discussed in the related work.

We implemented methods for the combined visualization of probabilistic tractography (as seen in Fig. 1) with state-of-the-art visualization techniques for other modalities as shown in Table 1 and compared the results.

4 Results and Discussion

In this section, we discuss possible combinations of probabilistic tractography with data from Table 1. For each column in this table we will discuss the scenarios in separate subsections. This groups similar data and simplify the overview on how can it combined with probabilistic tractography data. Please note, other data not listed in Table 1 may also be assigned to one of the columns and thus may be threatened similarly.

4.1 Textured Slices

Textures are very often used with slice based visualization techniques. Although other geometry may be used for texturing, for example inflated cortical surface maps textured with T1 data, slices are still very prominent. One reason for that is, that slices deliver detailed context. This does not imply that three-dimensional visualizations are useless, other visualizations may still profit from their strength in communicating the overall structure. However, most multi-modal visualizations using textures on slices, destroy this valuable context by using opaque overlays. Also semi-transparent renderings introduce the problem of color mixture, which may lead to wrong interpretation of both foreground and background color. All techniques for probabilistic tract visualization share the same problem: Rendering the tracts along with other data will occlude it more or less. Fiber-Stipples is a (slice-based) technique for rendering probabilistic tracts, which tries to minimize exactly this occlusion and hence is in that point superior to all other techniques. This is the reason why we suggest Fiber-Stipples for visualizing probabilistic tracts in combination with other textures on slices, except for directional textures as discussed in Sect. 4.1.3.

4.1.1 Monochrome Backgrounds

There are many monochrome images providing important information for neuroscientists. CT, structural MRI, angiography data, diffusion MRI, and diffusion indices are data sources for such images. As mentioned earlier (cf. Sect. 2.4.2), direct visualization of the diffusion gradients is rare and neuroscientists so far focus on providing gray-scale images as anatomical context to their technique, either taking measured data (such as T1 or T2 images) or by projecting their data into a reference frame and using reference data as context [60]. Axis aligned slices of T1 and T2 images are the most commonly used reference data for all techniques displaying probabilistic tracts. This is mainly because probabilistic tractography and T1 and T2 images are MRI based techniques.

A problem when using such underlays, and especially with monochrome underlays, is, that they typically have bright and dark regions. Hence the luminescence of the background conflicts with the luminescence of the foreground color. Ware [61] (pp. 111–112) states that fine details may then be hard to perceive and proposes to increase luminance contrast. This is of special importance to Fiber-Stipples, which encode most information in such fine details of the texture. This can be solved by controlling (i.e., reducing) brightness and contrast of the background image. Of course this will have the drawback of loosing detailed features in the background. Another possibility is to adapt the luminescence of the stipples depending on the local background. This removes the possibly of using quantitative color-coding. Fortunately, Fiber-Stipples provides two channels to communicate the connectivity score: the stipple coloring, typically by controlling opacity, and secondly, the stipple density, which remains unchanged. An example of such a contrast enhanced stipple coloring is given in Fig. 3. However, such an adaptive color coding may lead to misinterpretation of the data and needs further investigation. In addition, fully opaque stipples might distract the user in regions of low connectivity scores. The problem is less severe when the background data provides sufficient contrast along the probabilistic tract. As an example, fractional anisotropy (FA) which is a prominent scalar value derived from diffusion information (diffusion index), is also used as termination criteria for many tractography algorithms or as a indicator on bundle integrity [41]. While major white matter bundles will often follow regions of high FA, most of the stipples will pass bright regions and the color can be chosen accordingly. When white matter tracts also project into cortex, which is a region of low FA and thus rendered in dark colors, a uniform coloring might be suboptimal then, see Fig. 4 for images dealing with diffusion indices.

Fig. 3
figure 3

Fiber-Stipples on T1 images. (a) Close up view of original Fiber-Stipples. (b) Full opaque stipples with fixed maximal luminescence. (c) Adaptive luminescence with full opacity. This way, the stipples have higher contrast

Full size image
Fig. 4
figure 4

Fiber-Stipples over diffusion indices: axial diffusivity (a) and fractional anisotropy (b). Other prominent indices such as radial diffusivity or mean diffusivity look very similar and are not depicted here

Full size image

Beside, the combination of structural MRI or diffusion indices with probabilistic tracts, we think angiography data (from CT or MR) might be worth to combine with probabilistic tracts. An application might be surgical planning, where the location of important blood vessels and white matter tracts may become critical. On the other side, pure CT data might not provide enough soft tissue contrast for probabilistic tracts. The optimal visualization of probabilistic tracts in such a scenario is not known and needs further investigation. Contextual information from angiography data can be represented in slices as contours or colormaps, as well as three dimensional surfaces or volume renderings.

4.1.2 Colored Background

Beside monochrome textures, neuroscientists are also working with colored textures for superimposed data. As an example, fMRI data is colored with the heat-map as overlay for T1 data as context. Another example is the RGB-coloring of diffusion orientations (cf. Sect. 2.4.2) which may be used directly as texture, but also with additional filters such as the fractional anisotropy (FA), see Fig. 5.

Fig. 5
figure 5

Fiber-Stipples on colored background representing RGB-coloring scheme of the major diffusion direction filtered by FA. In all images the stipples are fully opaque. (a) White stipples on unmodified RGB-colored FA background. (b) Outlined white stipples with decreased background luminance from 100 % to 80 %

Full size image

Some colored textures may use the whole color space (e.g. rainbow colormap) and some are limited to a fixed number of colors or transitions between only two or three colors (e.g. heat-map). As visualization techniques for probabilistic tracts usually use some color, a combination with colored background generally depends on whether there are unused colors left which also provide discernible contrast. Techniques not using color but texture are LIC or TDI, which are discussed separately in Sect. 4.1.3. Due to the same motivation as given in Sect. 4.1 regarding the occlusion, Fiber-Stipples are suggested for combinations of probabilistic tracts with colored backgrounds as well. However, the encoding of connectivity score in stipple opacity may be of no use, as the color blending might change colors and hence lead to misinterpretation.

4.1.2.1 Arbitrary Colored Backgrounds

When a texture is arbitrarily colored it may use almost the whole colorspace and any color used for rendering the probabilistic tract might fail. Prominent examples are the rainbow colormap [8] or directional color codings [15, 48] (filtered or unfiltered). With disabling the opacity, black or white may sound as a good choice then for coloring the stipples, see Fig. 5. While the luminescence of the background may still influence the visualization, we further suggest to limit or increase (depending on data) the background luminescence as well, see Fig. 5b. To further increase contrast we propose to outline (cf. also Ware [61] page 123) the stipples as depicted in Figs. 5b and 6. This way the combination of probabilistic tract data with underlying colored texture may still be perceived well.

Fig. 6
figure 6

White Fiber Stipples over arbitrary colored background. To ease the perception of directionality of Fiber-Stipples, a small outline around each stipple is used. In (a) no such outline is present on fully opaque white Fiber-Stipples, while in (b) the outline substantially improves the shape. This outlining technique can also be applied to Fiber-Stipples with variable opacity encoding the connectivity score (c)

Full size image
4.1.2.2 Colormaps Defined by Few Colors

When working with colormaps defined by only two or three colors and their continuous transitions, a most complementary color may be practical for coloring the probabilistic tract. For example we believe, when working with the heat-map, used e.g. for indicating BOLD activity in fMRI, blue might be a good choice as it provides good chromatic contrast, see Fig. 7. The combination of fMRI data with probabilistic tracts is a very interesting one, as fMRI data communicates functional connectivity and the probabilistic tract communicates structural connectivity. This contributes to the visualization of effective connectivity [19, 57]. Other combinations with PET data are possible as well [44], but, from a visualization point of view, the combinations are then very much the same as for fMRI data, due to the same type of data.

Fig. 7
figure 7

FMRI Activation Maps are often visualized as colormap overlay onto T1 images as in (a), and (b) shows a close up view. For such activation maps the so called heat-map is very common

Full size image

Another possibility to combine visualizations of probabilistic tracts with colormap overlays using only few colors, is to apply the colormap directly on Fiber-Stipples, see Fig. 8 for an example. Although it seems a natural solution for combining Fiber-Stipples with such colormaps, important topological information may not be rendered. A tract not crossing a center of neural activation may then need additional topological information for better interpretation. Another solution would be to combine two three-dimensional techniques, e.g. use nested isosurfaces [7] to combine the modalities. One surface highlights the probabilistic tract while another may highlight the fMRI activity region. However, when anatomical context is very important we suggest Fiber-Stipples again for visualization.

Fig. 8
figure 8

Fiber-Stipples color coded with the fMRI BOLD activity signal. (a)—bright yellow stipples indicate not high connectivity score, as this is encoded in stipple density, but a high BOLD signal activity. In (b), the BOLD signal is thresholded, so blue colored stipples represent stipples below this BOLD threshold

Full size image
4.1.2.3 Fixed Colors

Finally, textures with few fixed colors are very common in neuroscience to label segmented structures. For example several white matter atlases label segmented structures with different colors, see Fig. 9. As same as in the previous section, it would be practical to have a most discriminative color, used for Fiber-Stipples representing the probabilistic tract. Bauer et al. [5] revealed, that the most discriminative colors should lie outside of the convex hull defined by the given colors in CIE color space [64]. Unfortunately, the more colors are used, the more difficult it is, to find such a color. Furthermore, it is also important to adapt luminescence contrast as well for better color discrimination, as the colors used for labeling structures might have very high or low luminance as well. Although it sound possible to change color labels of white matter structures, so that the set of fixed colors allows a better choice for the stipple color, we do not encourage such a change, as scientists may be used to the given colors for specific atlases. Last but not least, the visualization technique depicting the probabilistic tract may also employ the given set of colors as labels for the structures (the very same way as outlined in Paragraph “Colormaps Defined by Few Colors”) which imposes the same problem of missing topological information. This would be important to answer specific questions such as, to determine if a given tract just barley hits the structure or if it is projecting right into its center.

Fig. 9
figure 9

Segmentation from the JHU white matter tractography atlas [42] labeling certain white matter regions. Here parts of the forceps minor, internal capsule, external capsule and parts of the corpus callosum in the frontal lobe are labeled with different colors

Full size image

4.1.3 Directional Textures

A special case of textures are textures indicating directional information using fine patterns, with line integral convolution (LIC) being the best known representative of this group, cf. Fig. 1b. Beside LIC, tract density imaging (TDI) [12] and other fabric-like visualizations have been proposed to display tensor information.

In order to combine such textures with probabilistic tracts we think semi-transparent colormaps are most useful here, see Fig. 1c. The directional textures are rather versatile when it comes to applying the pattern to different planes or surfaces. Their main disadvantage is the restless pattern that makes it harder to see small structures in front of it (cf. [61]). These disadvantages are depicted in Fig. 10c.

Fig. 10
figure 10

Combinations of Fiber-Stipples with directional textures. (a) shows red Fiber-Stipples over LIC background computed from DTI’s major diffusion orientation. (b) shows two different tracts which cross. The red tract is part of the corpus callosum while the blue is part of the cortico spinal tract. T1 is used as background. (c) Close up view of (b) but with LIC of the second diffusion orientation (computed by FSL BedpostX) as background instead of T1. (d) Same as (c) but LIC replaced with (yellow) Fiber-Stipples

Full size image

However, these textures transport some directionality (from vector field) and eventually also some density (cf. TDI). Both properties may be depicted with Fiber-Stipples again, leaving room for anatomical context which would be occluded by such textures otherwise, see Fig. 10d. There, two crossing tracts are depicted by the major diffusion orientation. The FSL software package offers a technique to estimate a second diffusion direction out of DTI data by using Bayesian estimators implemented in BedpostX [33], which helps to better understand crossings. Such a second direction is then depicted as LIC in Fig. 10c and with yellow Fiber-Stipples instead in Fig. 10d. Please note that, other visualization techniques, such as the vector-plot, cannot communicate density.

4.2 Glyphs

Glyphs have been used in many settings before, and can be combined with surface and volume representations [40]. Special care has to be taken when glyphs are not sampled along a plane or surface, but express volumetric information. Then, occlusion leading to a perceived change in glyph density may distract the viewer.

So, combining glyphs with probabilistic diffusion tractography would be best on a plane or surface. When placing glyphs on an arbitrary surface, e.g. on cortex [9] or on an isosurface of a probabilistic tract, glyph placement should be performed as proposed by Kratz et al. [37] to get optimal results. In short, they employ an anisotropic Voronoi cell rendering to obtain best packaging on two-manifold domains.

When combining glyphs with slice based techniques, they will typically occlude some visualization parts. As there are also slice based techniques using glyphs for the visualization of the probabilistic tract (Fiber-Stipples), mixing different glyph types might be very confusing, especially when they need to maintain a density property. On the other hand, glyphs of the same type might be combined. For example, Fiber-Stipples can also be regarded as a specific glyph-based technique to render probabilistic tract information. As the technique resembles manual drawings of stipples [54], they may be safety combined (up to a certain extend) with other Fiber-Stipples, as long as they have discernible colors.

Last but not least, integrating the tract into the given glyph might also be an option. To summarize this, it is hard to tell from which combination such a visualization would benefit and we vote for keeping it simple and use either Fiber-Stipples or a plain colormap to communicate the tract as an underlay for the glyphs.

4.3 Line Data

Neuroscientists use various line data in their visualizations. Examples are, polylines or tubes representing white matter fiber tracts, outlines of structures, splines as connections between regions and so on.

4.3.1 Contours

Contours outline specific parts of T1 data, such as cortex parcellations, white matter segmentations, or lesions. Often only one or two contour lines are used, e.g. Fiber-Stipples uses one contour line for outlining the gray matter and another for outlining the gray matter–white matter interface, see Fig. 11a.

Fig. 11
figure 11

All images render the same probabilistic tract at the same slice with same density. (a) shows the original Fiber-Stipples technique with enabled opacity encoding which is disabled in (b)–(d) (b) combines the fMRI activation from Fig. 7 with Fiber-Stipples and renders contour lines (90 %, 80 %, 70 %, 60 %, and 50 %) of the activity field in blue. (c) same as (b) but with T1 background and contours colormapped from light blue (90 %) to dark blue (50 %). (d) same as (c) but using the heat-map for contours as well

Full size image

As contours are often used on surfaces we focus on combinations of slice based techniques for probabilistic tracts with contours. Such combinations might be useful to see if a tract is affected by a lesion, or if it projects into outlined regions. The combination of colormaps as overlays and contours would clearly lead to desired results, although the overlays occlude anatomical context. On the other hand, contours might be hard to perceive along with directional patterns such as LIC or TDI. Likewise, Fiber-Stipples with boundary curves have problems (to many contours with different meaning), see Fig. 11b. If we could find a way to depict contours, probabilistic tract and context at once, this would then be the optimal combination for such data. As an example we look back at the combination of Fiber-Stipples with fMRI data, where stipples are colored with the heat-map of BOLD activity data (cf. Paragraph “Colormaps Defined by Few Colors” and Fig. 8). There was a lack of topological information, which could be closed by using additional information, such as contours. This leads to a visualization of Fiber-Stipples without boundary curves over T1 background, where the Fiber-Stipples are colored with fMRI texture (heat-map) and additional contours, for the fMRI data, provide the missing topological information. We must admit, it is not easy to select the right colors to avoid the whole scene to become visually cluttered. Examples of such visualizations are given in Fig. 11. Please note that this introduces a new and alternative visualization for probabilistic tracts as well, see Fig. 12. In summary, we would here for Fiber-Stipples again, as it preserves valuable context on slices while still being able to work with contour data arising from other modalities.

Fig. 12
figure 12

Probabilistic tract rendered with contour lines, 100 %, 85 %, 70 %, etc. using the heat-map

Full size image

4.3.2 3D Polylines

In neuroscience, three-dimensional polylines are often used for representing neural tract data, cf. deterministic tractography [32, 43]. As such polylines visualize a three-dimensional course through the brain, we believe it best to use three-dimensional visualizations of probabilistic tracts to combine them. Solutions are semi-transparent isosurfaces Fig. 13a, or illustrative techniques for confidence intervals [10].

Fig. 13
figure 13

Combinations of probabilistic tractography with three-dimensional visualizations. (a) 3D polylines (rendered as tubes) from deterministic tractography representing left cingulum (CNG) bundle, partially enclosed by an isosurface. (b) Cortex approximation surface (semi-transparent gray) and probabilistic tract (red) with seed region in gray matter–white matter interface of the precentral gyrus. (c) The same white matter atlas [42] as used for segmentation as in Fig. 9 but using a different viewing perspective, (d) with the segmentation represented by surfaces, some of them are opaque and some are semi-transparent, and (e) a close up view of (d)

Full size image

Other applications of such polylines or splines are representations of functional connectivity [9]. However, such a combination should be considered carefully as though the line’s endpoints reside in anatomical space, their course does not have an anatomical foundation.

4.4 Surfaces

Neuroscientists work a lot with surfaces from brain structures, blood vessels but most importantly, cortex estimations. Due to the same fact, that those surfaces typically represent three-dimensional objects we recommend to use three-dimensional tract representations as well. As an example see Fig. 13b with a semi-transparent isosurface for the cortex and an opaque isosurface for the tract. Although, it is possible to combine slice based visualizations, such as Fiber-Stipples (Fig. 13c–e), the interpretation of the data will remain very difficult. For example the stipple orientation and density might not be perceived correctly from different viewing perspectives. A better slice based communication of the tract with the surface is to use cutting planes with colormaps, see Gorbach [28] (Fig. 1). Please note, in case of using those cutting planes the three-dimensional course might be lost. As an example, the fornix tract is mostly diagonal to all three (sagittal, axial and coronal) cutting planes.

5 Conclusion

In this work we gave an overview of current visualization techniques for probabilistic diffusion tractography data and its combination with other image modalities from the neuroscience domain. We summarized common data types and visualization techniques from the neuroscience community to systematically investigate multi-modal visualization with probabilistic tractography data.

In Fig. 2, we pointed out why anatomical context is of utmost importance when analysing white matter tracts. It turned out, that the Fiber-Stipples technique is a very flexible visualization technique for probabilistic tracts as it supports various superimposed data while still maintaining precise and detailed context. The resulting images typically incorporate a lot of information, thus, often the simple sparse and effective visualizations will stand out from their competitors. We showed that a combination of Fiber-Stipples with contour lines may communicate complex multi-modal visualizations. As an example we show the combination with fMRI activation regions. Additionally, we showed, that contour lines serve as simplistic visualizations of probabilistic tracts as well. Furthermore, we observed that color plays an important role when combining modalities and gave certain guidelines regarding contrast and color as well as improvements such as outlining to enhance the resulting visualization.

However, slice-based visualization techniques have drawbacks when three-dimensional objects, such as cortex surface approximations, demand for a combined visualization. In such cases we believe that three-dimensional techniques, e.g. isosurface renderings, may be superior to slice-based visualizations.

6 Future Work

The presented techniques should be considered a guideline for the creation of meaningful and useful visualizations focusing on probabilistic tractography data. With improved or new acquisition techniques arising daily, this list should not be considered complete but should be enhanced in the future. Whereas we focus on probabilistic data, an overview including other specific tasks would be beneficial. Although we reasoned design decisions wherever possible, we will evaluate them in a user study.