Introduction

The amount of data recorded before, during and after a surgery has drastically increased in recent years, due to the availability of new imaging technologies as well as better equipment for many non-imaging modalities. Supplementary information such as segmentations and annotations can be generated and data from other surgeries and patients can be used for comparison. Ideally, all of these information sources should be considered when planning the treatment of a patient. Managing the expanding amount of information will be increasingly difficult in the foreseeable future, and displaying the data to surgeons in a meaningful context will require new visualization and rendering systems.

An important portion of the acquired data is inherently three-dimensional, such as CT and MRI multi-slice scans. Usually, this data is viewed as individual slices on conventional 2D screens, resulting in the loss of depth perception, even though it has been shown that viewing data in 3D greatly enhances the comprehension of the presented structures [12, 19]. When planning the treatment of a patient, however, it is vital that the surgeons understand the precise spatial location of tumors, organs and vessels to optimize a possible surgical procedure and minimize patient risk (Fig. 1).

Fig. 1
figure 1

Left: Traditionally, 2D slices of CT scans are used for planning. Right: In complex scenarios such as this one, where the tumor (yellow) and the various vessel trees (veins, arteries and bile ducts) lie close together, a 3D visualization could greatly benefit physicians in analyzing patient data (screenshot of the virtual reality application)

Full size image

We present the IMHOTEPFootnote 1 framework, which meets the need to visualize very inhomogeneous data types in an intuitive, simple to navigate virtual reality (VR) environment. The framework provides tools for loading, viewing and manipulating patient-specific data. By rendering the scene in stereo (one image for each eye, with slightly shifted perspective between the images), depth perception is increased. We use head-mounted displays (HMDs) to expand the available workspace to a full \(360^\circ \) virtual environment. By using modern gaming technology, we achieve frame rates high enough for real-time interaction.

The aim of the framework is to be a development and research platform for VR tools in clinical scenarios. To this end, we provide many standard functions which facilitate the use of a VR environment in conjunction with medical data. The core functionalities of the framework are open source and are available online.Footnote 2 The framework’s central design choices are discussed below, while implementation details and instruction for usage can be found in the project’s documentation. Readers are encouraged to experiment with the project and add custom tools to develop their own medical VR applications.

Areas of application

Preoperative planning Complex surgical operations require individual planning on a per-patient basis. Many factors have to be considered, such as patient age, allergies and medication taken by the patient, as well as the patient history and previous surgeries. By visualizing the data in VR, the surgeons are able to quickly view relations in the data within one single application and to review the anatomical situation, even before performing the first incision.

Simulation Understanding complex dynamic 3D scenes is demanding. By providing the tools to port such animations into VR, the framework can be used to display simulations in stereo 3D, allowing users to view the animation almost as if they were present in real life. The framework’s intuitive input tools can be used to manipulate the simulation settings and the effects can be viewed in real time.

Fig. 2
figure 2

The virtual scene. The patient has been configured to show the surface rendering of the liver, liver vessels and tumor (yellow), along with a volumetric rendering of the rib cage. CT slice images are displayed on the virtual 2D screen for reference. On the bottom left, part of a handheld tool palette can be seen, displaying a tool to control organ transparency

Full size image

Teaching and learning in virtual reality Current teaching material usually only shows projections of anatomical structures, which can make them difficult to understand. Simple learning tasks can be integrated into the framework quickly. By expanding the framework with animations and simulations, it can aid in the understanding not only of 3D but also of 4D (3D plus time) processes.

Related work

The potential of VR in the field of data visualization has been widely recognized [4, 7, 18].Footnote 3 VR data visualization systems have been created for the areas of climate research [23], social media feed data [20] and software analysis [26]. Companies such as Kineviz,Footnote 4 Virtualitics.Footnote 5 and DynamoidFootnote 6 already offer VR data visualization applications for businesses, science and engineering. They use data mining algorithms and machine learning with the goal of making big data easier to analyze.

In the medical domain, the usage of VR applications has quickly increased in recent years [14], mainly due to the development of low-cost hardware. Many of these applications are focused on treating patients with traumas [2, 16, 25] or chronic pain [22]. These applications place patients into an immersive virtual scene, often as a form of exposure therapy.

In contrast, VR applications for physicians often focus on medical education. These range from visualizations of anatomical structures [11] and tools for surgical training [13, 17] to virtual scenarios which aim to enhance the interaction between clinical personnel and patients [10, 27]. In another project, 3D video captured during surgeries is played back and experienced in VR [5]. These systems are usually used with a fixed data set (such as a generic patient model or prerecorded video) and do not support the examination of patient-specific data for clinical use. They are not focused on clinical day-to-day routine and usually rely on older data sets.

Only recently, the first VR systems have been developed which are designed for diagnostic purposes. Examples are tools for the analysis of pathology slides [9] and tools which visualize the changes in sizes and shapes of tumors over time in a 3D virtual setting [21]. Ard et al. [1] have developed a VR system which visualizes brain MRI scans as well as several augmentations of the data. While these projects show the significance of new VR tools in medicine, they are aimed only at single areas of application and lack a broader, flexible and reusable architecture.

Egger et al. [8] implemented a VR plug-in for the medical image processing software MeVisLab. The plug-in focuses on the rendering of the virtual scene and relies on MeVisLab for loading and management of the data. However, this method only moves small portions of the available data into VR, so relationships between the inhomogeneous data sources cannot be visualized and context awareness may be lost.

Fig. 3
figure 3

Left: Volumetric rendering of a CT scan of a patient’s abdomen. The chosen transfer function extracts the skeleton and some organs, but can be modified to display different parts of the volume. Right: Schematic top view of the workspace layout. The 3D data are displayed in the center of the virtual room (a). The user (b) stands on a platform on the side of the room, surrounded by a large curved transparent 2D screen (c). The user has a tool palette (d) which follows them around. A virtual laser pointer (e) is used to interact with the UI and the data

Full size image

Using our framework, Loewe et al. [15] have created an interactive VR ablation simulation which allows physicians to explore the effects which ablation lesions have on atrial arrhythmias. Their work focuses on the usage of the framework for one specific application. In contrast, this article describes the underlying framework itself, its system components and design as well as a more general evaluation.

Methods

VR-based surgical assistance tools have several requirements. They need to

  • handle common clinical file formats and protocols,

  • smartly position the inhomogeneous data types,

  • render the virtual environment in stereo to allow for true 3D vision,

  • feature an intuitive user interface tailored for clinical day-to-day usage,

  • handle interaction with the VR hardware

  • ensure a high frame rate to create an enjoyable VR experience.

Since these conditions are independent of the exact application, a platform which abstracts away the required functionality can considerably speed up the development of VR tools for clinical usage. Our framework combines techniques from modern gaming technology with medical viewers. It provides functions to control the asynchronous loading, visualization and organization of patient data and handles user input and interaction with the hardware.

Patient data: from disk to visualization

At the core of the framework lies the loading and visualization of patient-specific data sets. These data sets can be comprised of textual information, 2D images, 3D volumes and 3D surface meshes (Fig. 2).

For the visualization of the 3D data, the framework provides two methods: If 3D volumes (such as the slices resulting from a CT or MRI scan) are available, they can be displayed via volumetric rendering (Fig. 3, left). This is realized by a custom GPU shader program which renders 3D textures using a slice-based approach [6] and a Blinn-Phong lighting model [3]. Transfer functions assign colors to the individual voxels in the data. By changing the transfer functions, different parts of the data can be visualized and highlighted. The rendering process is done twice for each time frame, once for each eye, generating stereo vision. The volumetric data is loaded in the common DICOMFootnote 7 standard via the ITKFootnote 8 library.

If previously segmented surface meshes exist, these can be displayed in the virtual scene (Fig. 1, right). The meshes (representing structures such as organs, vessels and tumors) are generated using external tools. They are loaded into the Blender3D software—which can import many common file formats—and are then saved in Blender’s native .blend file format. Individual colors can be assigned to the various meshes before they are opened and displayed by the framework.

Additional data (such as laboratory results and patient history) can be loaded and displayed alongside the image data. Optionally, text can be formatted in HTML to allow text coloring, font formatting and individual text layouts depending on the application.

Fig. 4
figure 4

IMHOTEP architecture overview. The framework builds on the Unity3D engine. It uses ITK to parse DICOM data, reads Blender3D surface meshes and communicates with VR hardware through OpenVR. The various data types can be manipulated by tools and visualized using various methods. Demanding tasks can be outsourced to separate threads. Asynchronous tasks can communicate across modules by firing events. Tested with Unity3D Version 2017.2, SimpleITK Version 1.0.1, OpenVR/SteamVR Version 1.2.1 and Blender3D Version 2.78

Full size image

Data organization: introducing workspaces

The type of information which is accumulated before, during and after a surgery is extremely diverse. First, there is data which is best presented as text, such as patient data (name, age, sex, diagnosis). Next, there is data which is inherently 2D, such as images (slices from MR- and CT scans, photographs or images from endoscopic video), graphs (development of patient data over time) and flowcharts. Finally, there is 3D data such as the surface meshes of the patient’s organs, along with annotations. Additionally, most of the tools for data manipulation need space in which to display their input elements and current status. In order to make best use of the available space and to present each data type in an intuitive way, the virtual scene is divided into three distinct workspaces (Fig. 3, right).

3D workspace To fully exploit the benefits of the 3D display techniques available in VR, the largest workspace in the center of the spherical virtual room is dedicated to displaying the 3D patient data. This automatically draws the initial attention of the user to the 3D visualization of the patient’s organs. Since the space is so large, the organs can be scaled to a larger-than-life size, allowing the viewer to focus on small details.

2D screens To supplement the 3D data with additional 2D information, a curved screen is introduced, which surrounds the viewer in the virtual scene. The resulting area is much larger than that of conventional 2D screens and the user can view the entire screen simply by turning their head. Positions of the various types of data (such as images or patient meta-data) can be pre-defined so that similar data will always be found on a similar position on the screen.

Tool palette Most of the available tools have settings which can be manipulated by the user. These settings can be accessed by choosing the tool from the tool ring, a user interface element fixed to one of the input controllers. Once a tool has been chosen, its options are displayed in the tool palette next to the controller. In this way, active tools are always easy to access, while inactive tools do not use up any of the available screen space.

Data control tools

The intention of the framework is to be used in very different applications. To implement these, researchers can add their own tools which build on the framework’s interface. Some commonly used standard tools are already present in the framework:

Annotation control Annotations are a simple way of marking structures and positions in 3D space and are commonly used in 3D design and visualization. The framework contains an annotation tool which can be used to highlight important positions, areas and volumes. Each marker can be linked to a textual label to add additional information. To minimize overlap between the labels and the patient data, hedgehog labeling [24] is used to automatically place the labels.

View control Surgeons and radiologists are used to viewing data from certain orientations, such as the sagittal, transverse and coronal views. The view control tool can be used to pre-define such views so that the data from different patients will be rotated and aligned to standard views automatically.

User interaction

The HTC Vive’s tracked controllers are used as the main input system. While other methods, such as tracking the user’s hands or gestures, might seem more natural at first, they were less suited for productive work, since precise selection and positioning of objects was not possible. By linking their movement to actions like rotating the patient organs or zooming the 2D DICOM images, user interaction becomes very intuitive. To precisely point at positions and select objects, a virtual laser pointer tool is attached to one controller.

Implementation details

In order to achieve real-time performance, the framework uses the Unity3D engine as the rendering back-end. The engine is well suited for the task since it is aimed at game development (an area where interactive frame rates are necessary). To ensure extendability, the framework’s architecture is composed of individual modules which exchange information through application programming interfaces and threads (Fig. 4).

Supported hardware and modes The system supports both the Occulus Rift as well as the HTC Vive and auto-detects which of the two HMDs is available. Depending on the supported functions of the detected hardware, the framework switches between Room-Scale-VR where the user can walk around the scene and seated mode. Furthermore, a development mode allows to use the program without any headset.

Events and threads IMHOTEP builds upon an event system which limits module inter-dependencies and realizes extendability. If one module fires an event, an arbitrary number of other modules can be notified and react individually. The available core events can be extended with other, tool-specific events. Since essential functions in a medical visualization framework require a lot of processing time, such as the loading of large data sets or calculation of a simulation response, expensive tasks are outsourced to background threads. By using the threading system in conjunction with the event system, tasks can be run asynchronously and then fire an event to notify other modules upon completion.

Fig. 5
figure 5

Participants’ responses to the questions: By using the tool, can a complex surgical situation be understood better (\(\mu =4.22\), \(\sigma =0.79\)) and faster (\(\mu =4.09\), \(\sigma =0.85\)) when compared to traditional software? Given values represent the mean \(\mu \) and standard deviation \(\sigma \) of the responses

Full size image
Fig. 6
figure 6

Participants’ responses to the questions: How high do you estimate the potential of this technology for student education (\(\mu =4.39\), \(\sigma =0.81\)) and postgraduate education for physicians (\(\mu =4.22\), \(\sigma =0.79\))? Given values represent the mean \(\mu \) and standard deviation \(\sigma \) of the responses

Full size image
Fig. 7
figure 7

Screenshot of the ablation simulation of the human atrium [15]. The color of the surface mesh is animated to show the activation wave (yellow), the refractory regions (green) and regions which have not yet been activated (white). The user can draw ablation lesions (red) and study their influence on the wave’s propagation

Full size image

Results

User study Using a patient data set for a complex hepatectomy, a user study was performed with 77 test persons (25 resident physicians, 30 medical students and 22 clinical staff). After viewing the data in VR, the test persons were asked to assess the application in terms of clinical and educational usefulness. Most participants stated that the application is beneficial for the planning of complex surgery, both in terms of quality as well as speed at which the anatomical situation can be understood (Fig. 5). The participants also believed the tool to be useful for the education of medical students and postgraduates (Fig. 6). Of the 77 participants, most stated that the experience in the virtual scene was comfortable (37) or very comfortable (33).

Operation planning To evaluate the feasibility in a clinical preoperative planning setting, a pilot study was conducted using the data of a 75-year-old female patient with a large central hepatocellular carcinoma comprising the segments I, IVa, V and VIII. The segmented 3D surface models were displayed semi-transparently to allow for the visualization of all target and risk structures without the need to change the view. CT images, laboratory parameters and the patient diagnoses and current medication were loaded into the system. Before surgery, the surgeons revisited the patient information first in the classical way in the hospital information system (HIS) and picture archiving and communication system (PACS). Their judgment of the case was based on the information in these systems. In addition, they used the system to visualize the patient’s data and records in VR.

The surgeons who evaluated the system confirmed qualitatively that it may lead to a faster and possibly better preoperative reassessment of relevant clinical data. They believed the system was able to show all relevant information of the patient in the VR environment at point of care. The data could be generated in a reasonable time frame considering the pilot study character of the evaluation. Visualizations could be handled easily to virtually generate the planned direct line-of-sight of the surgeons during operation.

Extendability The aforementioned real-time capable ablation simulation system [15] was built on top of our framework. The authors added their own fast-marching algorithm to simulate excitation propagation, as well as tools which allow the user to add and remove ablation lesions, handle tissue types, manipulate pacemakers and control simulation time. By relying on our framework’s ability to work with patient-specific data, their aim is to provide a tool to study the effect of lesions on arrhythmic behaviors in the human atrium for diagnosis as well as therapy planning (Fig. 7). Their user study showed that the application has a short learning curve due to the intuitive user interface and leads to a greater understanding of arrhythmias.

Discussion

The clinical evaluation of the framework has shown a very high acceptance for the use of VR in the context of surgical applications among medical personnel and students. Most users found the interaction with the data very intuitive and natural.

The framework has successfully been used to build an operation planning tool as well as an ablation simulation tool. Various other projects are being developed on top of the framework, including a resection planning system to allow the placing of resection planes and the calculation of residual volume. These results are encouraging when considering the future of VR in medical applications.

The studies performed here were conducted using qualitative questionnaires and can only summarize subjective opinions of the test persons. To objectively confirm the benefit of using a VR system when viewing complex data, further studies have to be carried out. Ideally, these should evaluate criteria such as cognitive load on the users, their ability to interpret a complex anatomical situation, time needed for operation planning and operation outcome and should be compared to results from a control group.

The framework can be used as-is to realize many medical applications. For more advanced usages, additional functionalities would be needed:

Muli-user-environment For complex cases, a multi-user extension of the framework would allow doctors to view and discuss the treatment with experts from all over the world in a virtual tumor board. In this area, open questions are the real-time interaction with the visualizations by multiple users at once, secure transfer of data and automatic anonymization.

Educational uses Participants in the user study expect a high benefit from using the VR environment for educational purposes. To aid in the development of such tools, functions common to various educational applications should be implemented in the framework such as quiz and scoring systems, the rendering of realistic tissue textures and gaming elements.

Clinical integration To make the system easier to use in a clinical day-to-day scenario, it should be integrated into the hospital information system (HIS), to allow rendering patient data directly from hospital servers. This is currently limited since required data—such as surface meshes—is usually not present for most patients, but would make the usage of VR in hospitals much more feasible in the near future.