Introduction

Augmented reality (AR) is different in its approach than virtual reality (VR); in the former, computer-generated objects supplement the physical, real-world environment [1,2,3,4,5,6,7,8,9] while in the latter, the entire real world is replaced by a computer-generated, virtual environment [10,11,12,13,14,15,16]. Both techniques have found initial applications in teaching and anatomical education [4, 10, 12] and have been explored for aiding surgical procedures [7, 14,15,16].

With increasing complexity of neuroimaging data, there is a need to convey this information to the end user, who in many cases may not be an expert in AR visualization or neuroimaging in general. A quick and easy approach for displaying the result and the most pertinent information will therefore be of great advantage as well as a simplicity of interaction with this complex data. With the advent of consumer-grade head-mounted AR devices, AR technology has been made available to imaging experts without the need of being an expert in AR computer algorithms or AR hardware.

Here, we report first experiences with a commercially available AR device (Hololens, Microsoft Inc.). The MS Hololens is a self-contained holographic computer in the shape of a wearable headset that allows the user to see, hear, and interact with computational objects projected into the environment such as a living room, office space or, potentially a surgical suite. The user can navigate this augmented scene by gestural interaction. There is no need to connect the Hololens to a PC, instead, all data required to display the computational objects is transferred to and stored in the Hololens directly. It uses high-definition lenses with small computer screens positioned in front of the eyes of the user together with spatial sound technology to create an immersive interactive holographic environment. We present an algorithm entirely based on freely available software to create AR objects from magnetic resonance imaging (MRI) data. Selected examples illustrate the application of the algorithm.

Materials and Methods

  1. 1.

    Augmented reality device

The AR device (Hololens, Microsoft Inc.) was acquired as part of a Developer’s edition. One of the device-specific apps, the 3D Viewer beta, allows importing 3D objects.

  1. 2.

    Image data processing pipeline

The following sections describe the conversion of the neuroimaging data into a format that is compatible with the AR device. A flow chart as summary is provided as well (Fig. 1).

Fig. 1
figure 1

Flow chart to construct 3D objects for import into the AR device for the four discussed examples in the text. Cortical surface: T1-weighted (T1W) high-resolution anatomical images were converted into NIfTI format and imported into Freesurfer where pial surfaces of the right and left hemisphere were extracted in STL format, which could be directly imported into the Paraview software. fMRI BOLD Activation maps: Field echo echo planar images (FE EPI) were acquired during the execution of a bladder voiding task. fMRI activation maps were obtained with the AFNI software with T1W high-resolution anatomical images as references. The 3D IO plugin of ImageJ allowed storing both the cortical surface and the BOLD activation in the VTK format readable by the Paraview software. White matter tracts: based on DTI images acquired for 15 distinct diffusion gradient directions, white matter tracts were reconstructed by the diffusion toolkit that allowed (via the trackvis software) storing these tracts in the VTK file format. Functional Connectivity: FE EPI images and averaged fMRI BOLD activation maps were combined (see text) to create a graph of FC during task performance (bladder voiding) that was visualized in the individual anatomical space of one subject and stored in VTK format. Once displayed in Paraview, 3D STL and VTK Files were exported as scenes (X3D format) and rendered in the Blender software from where they could be imported into the 3D Viewer beta app of the AR device (as FBX files)

Full size image

MRI image data, both anatomical (structural and diffusion) and functional, was retrospectively collected from an IRB-approved human research study involving repetitive bladder voiding [17, 18] in DICOM format [19] (3T Ingenia, Philips).

  1. (a)

    Anatomical image data—brain surface

High-resolution T1-weighted MRI images of the brain of one subject (axial, 1 mm isotropic resolution) were imported into Freesurfer (version 5.3.0) and separate pial surfaces of the left and the right hemispheres were obtained in the stereolithographic (STL) format. A 3D scene combining surfaces from both hemispheres was created Paraview (version 5.2, Kitware Inc.) and stored in the X3D format, which was previously also reported to be used for medical applications [20].

  1. (b)

    fMRI BOLD activation maps

Task-based fMRI images were analyzed by a standard processing pipeline (AFNI software, version 16.3.18) [21]. Group analysis yielded BOLD activation maps in Talairach space (p < 0.05, NifTI format [22]) to identify brain areas activated during the initiation of bladder voiding [18]. Converted into the VTK format (www.vtk.org, Kitware Inc.) by ImageJ {version 1.50c4, 3D IO plugin [23]) the 3D BOLD maps were overlaid onto a mesh of the cortical surface created from the T1-weighted skull-stripped images. This 3D scene including both BOLD activation information and the cortical surface was exported in X3D format (using Paraview).

  1. (c)

    Diffusion tensor images—white matter tracts

White matter tracts were reconstructed in VTK format from a diffusion tensor acquisition (axial, 15 directions, slice thickness 2.5 mm, in-plane resolution 1.75 mm; diffusion toolkit and trackvis software, versions 0.6.3 and 0.6.0.1, retrospectively, www.trackvis.org/dtk [24], minimum tract length 21 mm). Due to size restrictions in the AR device software, the 3D scene (X3D format) created from this dataset was reduced by the application of the ‘Decimate’ filter in Paraview (target reduction of 90%).

  1. (d)

    Functional connectivity—brain networks

Functional connectivity (FC) during repetitive bladder voiding was determined by an algorithm described previously [25]. With the correlation coefficient between time courses as edge weights, individual FC networks were created and visualized in the anatomical space of one subject as a polydata dataset (using Paraview), where edges were color-coded by strength of activation of the vertices they connect. For improved visualization, edges with low connectivity strength (edge weights <0.8) were omitted in the final 3D scene (X3D) and a mesh of the cortical surface was included.

All 3D scenes (X3D) were loaded into the Blender software (version 2.77, blender.org) and rendered with standard settings (in FBX format). FBX files are readable by the 3D Viewer app of the AR device.

Results

The AR devices succeeded in creating a feeling of actual physical presence of the computational object, which is difficult to convey by the simultaneously recorded photos (Fig. 2) or videos (supplementary materials 1 and 2).

  1. 1.

    Anatomical image data—cortical surface

Fig. 2
figure 2

Rendered 3D scenes in Paraview compared with the actual view through the AR device (internal photo capture)

Full size image

The user was able to walk around the freely suspended pial cortical surfaces and inspect them from varying viewpoints. Entering the structure, revealed (limited) interior features as outer layers were automatically removed by the AR device (Fig. 3).

  1. 2.

    fMRI BOLD activation maps

Fig. 3
figure 3

a Illustration of the “pealing off” effect at a selected location of the DTI dataset (gray box in top view) while “walking through” the 3D object (lower views, closer proximity from top to bottom as indicated by the arrow). Gaps in lateral tracts reveal details of more medially located structures. b Streaming capability of the AR device allows for view sharing with others. Top view represents an internal view capture from the device and bottom view a screen capture (from screen visible in top view marked by asterisk) making the internal view accessible to the audience. c “Peal off” effect for pial cortical surfaces. Medial part of the surface calculated from Fressurfer is visible (left), but other structures are missing (right)

Full size image

The ability to inspect the location of activated brain regions in a large 3D display by visual inspection facilitated the understanding of the global fMRI BOLD activation pattern also by users, who had no prior introduction to the AR device or the software. No additional instructions were needed and immediate acceptance was observed.

  1. 3.

    Diffusion tensor imaging dataset

    Presenting the DTI dataset as a “floating” structure, where layers of tracts can be “pealed off” by simply entering into the computer-generated object, greatly improved the understanding of the spatial relationship of different tract groups to each other (Fig. 3). In addition, the AR device allows simultaneously streaming of the view that is presented to the user to a nearby computer screen (Fig. 3). This feature allowed the imaging professional to guide the inexperienced user.

  2. 4.

    Functional connectivity—brain networks

As observed for the previous models, the ability of walking “into” the computational model allowed to selectively remove layers of edges thereby gaining an intuitive understanding of the connectivity of selected brain areas. The network under investigation was found to consist of two large sub-networks, one in the frontal brain and one that connected sensory regions with deeper brain structures including the micturition center located in the pons (Fig. 2).

Discussion

First reports of AR to aid in surgical intervention or for educational purposes have been reported as early as 20 years ago [1, 2, 4, 26, 27]. Recently, easy access to powerful portable and at the same time commercially available AR devices has renewed interest in this topic [28,29,30,31].

In this technical note, an algorithm and first impressions of a head-mounted AR device (Hololens, Microsoft Inc.) for displaying complex medical image data are presented. In all cases, the application of the device was successful as an impression was created whereby the computational object seemed to exist in the actual real space in which it was displayed. Once positioned, users with no experience of the AR device could interact with these objects by simple inspection guided by simultaneous streaming of the AR view to a computer screen. This greatly aided in the acceptance of this technology and may facilitate applications in clinical research and in patient education. AR provides a new venue, where the imaging expert can interact with the research subject or patient directly. Once established, the here presented algorithm for converting the image data into 3D objects that can be displayed by the AR device can then be easily extended to other kinds of 3D data, potentially to provide guidance in surgical interventions.

Our algorithm is easy to implement, as it makes use of well-established and tested software and data transfer mechanisms. There is no need to write additional software as its components work well together but also provide sufficient flexibility to modify the appearance of the created computational objects. At the same time, this flexibility introduces some kind of variability into the data manipulation. Collecting more feedback from different user groups will help establish further guidelines how to streamline and standardize the processing and display of medical image data. It is anticipated that different user groups will have varying requirements, all of which may not be fulfilled with the here presented algorithm. Once feedback from these different groups is available, additional development efforts aimed at enhancing innovation of the AR environment will then be successful in fostering the application of wearable AR devices for visualizing medical image data.

One great potential of the wearable AR devices such as the MS Hololens is the creation of unique collaborative experiences. For example, co-located users can see shared 3D virtual objects that they interact with, or a user can annotate the live video view of a remote worker, enabling them to collaborate at a distance [30]. Means to identify objects traditionally used with tablets or large computer screens such as mouse pointers can be largely replaced by gestures that make it easier and faster to interact in particular with less training and knowledge of the underlying computer system. There is a direct benefit of integrating decision makers that are not computer experts due to the simplicity of interaction once an AR scene is set up in the device. The Hololens also allows virtual annotations to be incorporated into a computer-created scene. This feature together with view independence has recently been shown to reduce task completion time [31]. The interaction with the computational objects comes naturally. Further improvement of the efficiency of gestures will aid in this process.

Limitations

Restriction of the size of the 3D model, i.e., number of vertices, was experienced as a serious limitation and compromises (i.e., decimating the computational meshes) were necessary. Another shortcoming consisted in the gestures and verbal commands needed to interact with the AR device. Visualization session were therefore separated into two parts; in the first part, the imaging expert positioned the virtual model in the real space, after which the AR device was handed over to the inexperienced user, who then viewed the object without any further interaction with the device.

Conclusion

First positive experiences using a commercially available head-mounted AR device are reported. An algorithm is presented visualizing complex three-dimensional objects derived from medical image data. Potential applications include enhancing understanding of the complexity contained in these objects and advanced education in neuroimaging research.