Keywords

8.1 Introduction

The world around us is changing rapidly as we increasingly rely on technology. New generations cannot even imagine the world without it. The technology to support mixed, augmented, and virtual reality has already become a part of our world. Nowadays, medicine is changing rapidly compared to past centuries when none of these digital health technologies were available.

Mixed Reality (MR) brings together the real and virtual worlds. In mixed reality, physical and digital objects interact in real time but have no specific place in a virtual or real world. One can say it is a mix of augmented reality and virtual reality in either two or three dimensions (meaning adding virtual objects to the real environment). In 1994, Paul Migram described mixed reality as a scale of reality—a virtual continuum where mixed reality covers every state between the real and virtual worlds (Milgram et al. 1994).

Although mixed reality is still mainly associated with entertainment, various other industries have implemented it for different purposes. So far, it has been already used with success in education, military training, remote working, architecture, interior design, and product content management, among many others. In the case of medicine and its wide range of possible applications, the use of mixed reality stands out as a very promising tool in healthcare as well.

In practicality, to accomplish a mixed reality environment, there is myriad of digital tools from mobile devices or wearable technology to entire rooms designed for it. The usefulness of smart glasses in medicine has been especially evaluated and proven to be of benefit in many studies even when, in many cases, the evaluated devices were not medical devices and were not designed for medical purposes. Analytics from Market Data Forecast predict the market compound annual growth rate of global mixed reality to be 47.9% during 2020–2025 and the medical holography market to grow from only USD 500 million in 2021 to over USD 2 billion in 2026. This growth will be driven by improvement in technology and by access to it. This chapter will highlight potential benefits, opportunities, and some challenges to the application of mixed reality in medical practice and medical education.

8.2 Possible Use of Mixed Reality in Medicine

Mixed reality devices often come in the form of headsets or smart glasses. Smart glasses are usually web-connected wearable computing devices that allow the transmission and projection of various types of data in the field of vision. Early models could perform basic tasks and display some pictures and figures in the field of vision. Such visualization could be useful in everyday medical practice. However, with technological progress, other devices have rapidly appeared that are capable of displaying three-dimensional objects with which users can interact in real time. These devices use special mobile applications dedicated for specific tasks.

In general, smart glasses can display all kinds of information: patient data, test results, imaging studies reconstructions etc. Almost a decade ago, Google Glass™ was one of the first models of smart glasses to be used in medicine. These efforts were not only popularized in the media, but also were described in numerous investigational studies and reports. Using a wireless platform, smart glasses present practically no obstruction to human interactions and movements. The other advantage is the short learning curve and the fact that smart glasses can run on the well-known android system. Other early smart glasses had low wearability, a longer learning curve, and were obstructive to human-to-human interactions. O. J. Muensterer is a pediatric surgeon and a Glass explorer. He wore Google Glass in LMU Munich Children’s Hospital for 4 consecutive weeks, each in different clinical situations, and kept a diary of his experience. He focused on how well the device is tolerated by the user, checking features like wearability, battery life, and audiovisual quality. Patients’ and their families’ responses to the device were also assessed (Muensterer et al. 2014).

Google Glass allows the projection of various data. This concept was used by Jeroudi et al. to investigate the accuracy of electrocardiogram interpretation by Google Glass. Each of 10 compared electrocardiograms was visualized in four formats: as viewed by Google Glass; picture taken by Google Glass; paper version; and a picture taken by the camera. The researchers then compared differences in the interpretations of the electrocardiograms. Although users were not satisfied with the images compared to the paper version, this study among others is an example of using such a device in telemedicine (Jeroudi et al. 2015).

Smart glasses also allow the sharing of information among specialists, whether as a consultation from within the hospital or from anywhere in the world. Authors from Yale University attempted to show the application of Google Glass for teleconferencing in emergency medicine. In their project, a team of paramedics performing triage during mass accidents consulted with an emergency medicine specialist. The results revealed some obstacles but overall performance was not decreased; however, it took the users more time to perform their tasks. This study showed that with some technical improvements, smart glasses could be used in medical emergencies (Cicero et al. 2015).

There are few mixed reality head-mounted displays available; the most commonly used platform for mixed reality currently is Microsoft HoloLens (Redmond, WA, USA). This system projects holographic three-dimensional images in the user’s field of vision and runs on Windows operating system, which is familiar to users worldwide, making it extremely easy to navigate. It contains an internal battery and features Wi-Fi and Bluetooth connectivity.

This head-mounted mixed reality device is light (566 grams) and comfortable to wear, with an adjustable headband and fits over eyeglasses, which is another advantage.

Many companies and start-ups from around the world are developing specific applications for Microsoft HoloLens, some for medical use. Some applications provide access to medical data, remote patient care tools, life streaming, educational tools and, of course, 3D reconstructions for surgical planning and assistance.

Almost every field of medicine from anatomic pathology (viewing 3D specimens, navigation through specimen slices, telepathology) (Hanna et al. 2018), primary care physicians, oncology, and radiotherapy (mixed reality-guided patient positioning systems) (Li et al. 2022) among others can take advantage of MR.

With the global COVID-19 pandemic, another advantage of using MR devices has emerged. The pandemic led to acceleration in the implementation of digital tools and telemedicine solutions in many countries. The main goal was to provide safety for patients and medical staff when accessing medical professional diagnosis and treatment.

A pilot study was conducted at Imperial College London Hospitals using mixed reality during COVID-19 patients’ consultations. In that study, a single senior staff member entered the COVID-19 ward for rounds and patient care wearing a Microsoft HoloLens2 while other practitioners joined the rounds and took part in the process virtually. This reduced the risk of coronavirus transmission by minimizing physical interaction between hospital staff and infected patients. Total reduction of exposure time in all participating teams was 222.98 h/week. In addition, a significant reduction in the use of personal protective equipment (PPE) was noticed (approximately 3100 fewer items of PPE used per week). When staff members were questioned about their experience using the mixed reality device and its impact on their work, 75% said it was easy to navigate and more than 70% noted that it is comfortable to wear. HoloLens facilitated the work, the rounds were less time-consuming, and teamwork was improved (Martin et al. 2020).

A similar study conducted by Jeremy B Levy et al., also in London, reported similar results. In this study, COVID-19 patients were asked about their views on the use of the mixed reality headset during medical rounds. No patient claimed that the device disturbed their medical care or their interaction with medical staff (Levy et al. 2021).

In another implementation of mixed reality concerning COVID-19 patients, 3D holograms with mixed reality techniques were used to assess pulmonary lesions in COVID patients. The study showed that compared to standard CT scans, mixed reality 3D holographic images can be helpful to evaluate pulmonary lesions especially by less experienced doctors (Liu et al. 2021).

8.3 AR and MR in Surgery

Smart glasses react to simple voice commands, eye movements, or gestures. The hands-free system is particularly helpful in surgical practice and other fields of medicine that require practitioners to work manually, sometimes in sterile field. Numerous proofs of concept for smart glasses in surgery have been proposed.

The concept of projecting test images in the field of vision has been studied widely. For example, Wu et al. used Google Glass to facilitate ultrasound-guided central venous access. In this study, the Google Glass user had fewer additional head movements (Wu et al. 2014).

Three-dimensional pictures are routinely applied in the preoperative evaluation of surgical patients. This is usually a simple 3D reconstruction of images viewed by the surgeon on a plane screen. However, in some complex cases this is not adequate. Spatial understanding is crucial to achieve surgical precision and avoid complications. There is growing interest of 3D printing in preoperative planning, which would allow the surgeon to see and touch the printed organ and even practice the surgical procedure beforehand. This permits a better understanding of a patient’s anatomy, thereby improving safety and accuracy. Using augmented reality and 3D reconstruction, for example, holographic images could be as useful but cheaper and faster than 3D printing. Additionally, with mixed reality, the surgeon can interact with the anatomical reconstructions in real time during the procedure and remain sterile in the surgical field. The holograms can be rotated, sliced, or scaled freely by the surgeon and can be placed anywhere in the visual field. The user can even “step inside” the target organ virtually, an experience never before possible. Most applications created to display holographic reconstructions of a patient’s anatomy work with different imaging techniques recorded in the DICOM standard (Digital Imaging and Communications in Medicine) such as computed tomography (CT), magnetic resonance imaging (MRI), angiography, or 3D ultrasonography, and some have the capability to connect directly to imaging devices, for example, echocardiography devices, to visualize the images in real time (Fig. 8.1).

Fig. 8.1
A screenshot depicts a Camalife Holo window with a C T scan of a human on the left and the hologram of the pelvic, spine, and rib cage on the right.

Process of rendering CT scans to hologram. Photo: Tomasz Jędrzejewski, Medical University of Warsaw press office

Full size image

The application of Microsoft HoloLens has been described in various surgical fields, e.g., orthopedic surgery, plastic surgery, neurosurgery, oncological surgery, and many others. Practitioners from all fields of medicine see the prospects of mixed reality in their everyday work and are trying to explore its possibilities in their practice.

Mixed reality tools in preoperative planning might be of great benefit in cases where the patient’s anatomy differs such as congenital diseases or in complicated oncological cases. Brun et al. described one of the first examples of preoperative planning using mixed reality in congenital heart disease. The suggestions for the surgical repair were made based on 3D mixed reality reconstructions. The holograms were easy to interpret and helped the surgeons solve challenging tasks intuitively and were rated highly by all users (Brun et al. 2019). Some of the applications for HoloLens provide specific tools for surgical planning (Kumar et al. 2020).

Research relating to the use of mixed reality in the spatial understanding of liver anatomy showed that it decreases the time to correctly identify lesions in the liver and, in some localizations, also increases accuracy (Pelanis 2020).

Dimitrios Chytas and Vasileios S Nikolaou in their literature review outlined the state of the use of mixed reality in orthopedic surgery (Chytas and Nikolaou 2021). They described numerous implementations found in the literature. For example, reverse shoulder arthroplasty performed with the aid of an MR headset to better visualize the anatomy (Gregory et al. 2018), cervical spine complex fracture procedure where MR was used for preoperative planning and perioperatively (Wu et al. 2018), and total hip arthroplasty with the use of both mixed reality and 3D printing. These are only a few examples of countless proof–of-concept studies conducted in many areas of surgery (Lei et al. 2019).

MR is a tool that can help personalize treatment and could help in the implementation of new methods with better visualization and accurate planning. An example is a study conducted in Cracow where patients with unresectable pancreatic or liver tumors had irreversible electroporation or microwave ablation treatment with the assistance of Microsoft HoloLens 2. MR was used preoperatively for planning and during the procedures to support the decision-making process. Additionally, the researchers assessed the remote connection with a team of specialists (Wierzbicki et al. 2022).

3D reconstructions for surgical planning with the use of MR can potentially increase the surgeon’s precision, help with the expertise, and increase the safety of the procedure. Having the opportunity to see the organ in 3D and to interact with it proved to be the most advantageous for trainees and less-experienced doctors in particular.

Another step toward the future will be totally augmented or mixed reality-guided surgery, where anatomical reconstructions would not help plan the procedure and understand the anatomy, but the surgery would be navigated by a mixed reality platform.

One of the best examples of augmented reality image-guided surgery is spine surgery (pedicle screw insertions) with Augmedics Xvision Spine system, a wireless surgical navigation platform that allows visualization of a patient’s spine anatomy through the skin and tissue using a minimally invasive percutaneous procedure. The system helps in navigating implants while looking at the surgical field with no need to look on the other monitors for imaging study results. The platform consists of a transparent near-eye-display headset; otherwise, it is similar to currently used traditional navigation systems. Researchers on a cadaveric proof-of-concept study determined its accuracy. Additionally, with no need to look at separate screens it eliminates attention shift (Molina et al. 2019).

The advantage of using smart glasses during surgery also lies in the possibility for the surgeon to consult and interact with other specialists, and to ask them for advice. For example, while performing reverse shoulder arthroplasty with the aid of a mixed reality HoloLens, Gregory et al. shared the procedure video in real time with four other specialists for expertise (Gregory et al. 2018).

Another interesting implementation of augmented reality in the operating room is using it for following the surgical safety checklist developed by the World Health Organization. Because many medical errors happen in the operating room, the surgical safety checklist can be lifesaving. Many surgeons read the checklist from a list hanging on a wall or do it from memory, which can lead to skipping some important steps. Thomas Boilat and Homero Rivas developed the Digital Checklist Box (DCB), which can be projected directly onto the draped patient and completed and verified before starting the surgical procedure, preventing the surgeon from missing steps (Boilat and Rivas 2021).

8.4 MR in Endovascular Procedures: One of the Greatest Examples of Its Usefulness

Endovascular surgery, interventional radiology, and interventional cardiology are notable examples of medical fields that could benefit most from these MR technologies.

One of the major challenges of endovascular procedures is working with two-dimensional images of a three-dimensional anatomy. Oftentimes, vascular anatomy is complex, requiring the surgeon to use their spatial imagination; nevertheless, sometimes many angiographic images have to be taken in order to insert catheters or wires in the right position, which raises concerns about radiation exposure for both patient and surgeon, and the use of iodine contrast.

One of the first implementations of the HoloLens on larger scale was made by interventional cardiologists. Opolski et al. performed 15 percutaneous coronary interventions for chronic total occlusions with the assist of MR and showed lower contrast exposure compared to procedures without MR assist (Opolski et al. 2017).

The author of this chapter had the opportunity to use Microsoft HoloLens during endovascular aortic aneurysms repairs (EVAR). We thought that using 3D holograms of a patient’s anatomy could make orientation within the vascular anatomy easier.

The EVAR procedure involves radiation exposure and iodine contrast agent, which can cause acute kidney injury. Dealing with more complex aneurysms using fenestrated or branched stent-grafts also involves increased radiation, more contrast use, and a prolonged procedure. In those cases, mixed reality could be most useful.

We used the Carna Life Holo application created by Polish company, MedApp. Holograms of the patient’s anatomy are created from preoperative standard cross-sectional DICOM computer tomography images that are segmented and processed. The stent-graft implantations were successful, and we observed no adverse events during follow-up. Seeing the patient’s vascular anatomy reconstructions precisely in three dimensions certainly helped us navigate the vascular tree (Fig. 8.2). To our knowledge, this was one of the first implementations of holographic visualization during an EVAR procedure in the world (Fig. 8.3) (Wrzesińska n.d.).

Fig. 8.2
A photograph depicts the holography of a human rib cage with the pelvic bone, along with the reality of a surgical field and several medical tools.

Holographic visualization of vascular tree projected on a surgical field

Full size image
Fig. 8.3
A photograph depicts the two doctors, who wear gloves, masks, and glasses with tubes on their hands, and two people standing behind them.

Visualization during mixed reality-assisted EVAR procedure. Photo: Tomasz Jędrzejewski, Medical University of Warsaw press office

Full size image

To see the anatomy and position of the catheters and stents in real time would revolutionize the field of endovascular interventions. Authors from Germany proposed a mixed reality guidance system for the EVAR procedure with the use of a HoloLens display. They used a special artificial human torso with an aortic aneurysm phantom to assess the electromagnetic tracking system (sensors were attached on the catheters’ tips and navigated by ultrasound), and display this information on the HoloLens in real time (García-Vázquez et al. 2018). Those techniques with technological improvements might result in shorter procedure time and a decrease of radiation and contrast use in the future.

One way to obtain visual data in real time is by using 3D ultrasonography. Currently, this is easier to apply in cardiac procedures as many applications allow the creation of holograms directly from echocardiography.

An example of a system that creates holographic images in real time is RealView Imaging, an Israeli company. According to the manufacturer, this is the first medical holographic system in the world. It was initially targeted to support interventional cardiology procedures. This system provides 3D holographic images projected with the use of special patented Digital Light Shaping™ technology. There is no need for smart glasses or other wearable devices as images are projected in the air. The system has FDA clearance for clinical use.

Researchers from Israel conducted a feasibility study using this holographic system during cardiac catheterization procedures. The system uses intraprocedural data from live 3D transesophageal echocardiography and 3D rotational angiography to make real-time holographic reconstructions. Eight patients were enrolled in the study. All anatomical landmarks during the procedures were identified successfully with no adverse events (Bruckheimer 2016).

8.5 MR in Education

Given its interactive features, mixed reality has an enormous potential in medical education.

One of the first and most spectacular examples of bringing mixed reality to teaching is a program conducted by Case Western Reserve University. In their project, medical students study anatomy via MR with the use of Microsoft HoloLens. The device enables students not only to see the anatomical structures in 3D but also to interact with holographic images and even whole-body holographic mannequins. This teaching method was compared with traditional anatomy classes on cadavers. Upon examination, there was no statistical difference between the scores of students taught using MR and those taught learning on cadavers (Stojanovska et al. 2019).

Another prospective study of anatomy students compared anatomy course study time and effectiveness between two groups: those using an MR learning platform and those using traditional methods (cadaveric dissection). The results indicated that while learning time was shortened using MR, there was no difference in the groups’ examination score (Ruthberg et al. 2020).

One more example of using MR in anatomy training was described for plastic surgeons. Researchers used a mixed reality HoloLens platform and virtual face models reconstructed from pictures taken from different angles. The device was able to project individual layers of the face anatomy (Kumar et al. 2021). These and other research studies prove that mixed reality can be successfully implemented in teaching anatomy.

With the use of smart glasses that have potential to bring demonstrations, recording, and live-streaming videos at any given location globally, education becomes not only easier but also more attractive. This attribute of mixed and augmented reality is particularly exploitable in surgical training. Traditionally, when performing surgical procedures there are some limitations on the number of persons that can participate in or watch. Those physical limitations end when the procedure can be transmitted, recorded, and explained. Harnessing mixed reality in the medical education process allows students and trainees to interact with the content they are studying, which by all means is more effective than just observing. Using mixed reality methods, the teaching surgeon can reach many trainees all over the world at the same time, whereas traditional methods limit the number of trainees.

Telementoring is another interesting application for augmented or mixed reality, allowing students or residents to perform some procedures on a patient by themselves while being constantly monitored and controlled by a supervising person. This can be done regardless of distance between trainee and supervisor as they can work, for example, in different hospitals (Mitsuno et al. 2019). Residents and more experienced surgeons learning new techniques could also benefit (Guraya 2019). Telementoring can help in potential guidance for a complex procedure not performed by a surgeon on a daily basis or when the patient is in a critical condition and cannot be transported to a specialized surgical center.

MR can also be used to simulate scenarios of potential clinical situations. This is a particularly popular way of teaching procedural and technical skills. Simulation-based training has been confirmed to be safe and effective, and to reduce the rate of complications. Trainees using MR simulations learn tasks mimicking relevant clinical situations. Assorted studies have shown that the results of such training are similar to traditional one but that the lower cost and improvement of patient safety are a benefit of MR simulations (Barsom et al. 2016; Huang et al. 2018).

As mentioned earlier, the global COVID-19 pandemic not only had an impact on providing healthcare but also on education, making it necessary to implement remote access in many places in the world. Likewise, medical education, which is based on interactions with patients, had to change dramatically during lockdowns. Again, augmented, or mixed reality in such a case presented a fine solution (Kassutto et al. 2021).

8.6 Patients Also Can Use MR

Not only healthcare practitioners can benefit from using mixed reality in their work but patients can also directly benefit from it. Using MR can empower patients in the sense of interaction between them and their physicians, telemedicine, simplifying hospital or outpatient visits, and patient’s education before surgery among many other functions. Companies make applications for patient use in pain management, rehabilitation, or to plan pharmacological treatment. Mixed reality can help patients with chronic diseases like Parkinson’s disease or with chronic pain (Wrzesińska 2015).

8.7 Challenges

With all those possible implementations, MR seems like a solution to many problems by the facilitation and betterment of existing solutions (Table 8.1). So, when will it deliver its promises and why does it take so long? Though medicine is becoming more technology-dependent, innovations in medicine take longer than in other areas. Because medicine has strict rules and restrictions, the speed of technological progress is limited. Every new method and device has to proceed along a path from its inception to its implementation in medical practice; indeed, questions arise about the use of mixed reality in medicine.

Table 8.1 Current applications of mixed reality smart glasses in healthcare
Full size table

The first and most important limitation that may hinder the use of mixed reality in medicine is the law because most of the augmented/mixed reality devices are not medical devices. Although the devices do not need to be certified for medical use, the medical applications of them oftentimes do. The legislation process in medicine takes a long time, especially if the innovation is something completely new, and the slow FDA approval process is not flexible for a prompt digital revolution.

We have to address also issues of data protection before using MR tools as this could be the main inhibitory factor in implementing these technologies in everyday clinical use. Of course, all medical software applications for MR systems must deal with the issue of confidentiality and adapt to local regulations.

For now, research is dominating the medical holography market according to a Market Data Forecast report. Geographically, the North American region is holding the largest market share in medical holography. This might be attributed to better support in medical research and better funding as well as good accessibility to and acceptance of innovative technologies (Market Data Forecast 2021).

Various technical aspects may inhibit MR implementation in healthcare. If the technology is to be used in everyday practice, then it has to be not only safe and effective but also ergonomic and not obstructive. Even though most devices can operate wirelessly, poor battery life is reported in many studies. For example, Google Glass has a reported battery life of 40 min (Muensterer et al. 2014; Chimenti and Mitten 2015). With Microsoft HoloLens, the battery life is longer (up to 5.5 h according to some studies), with around 3 h of active use and up to 2 weeks of standby time according to the manufacturer, and the device is fully functional when the battery is charging (Gregory et al. 2018).

Hence, because those devices were not designed for medical purposes, some technical limitations are not a surprise. Tao Zhan et al. addressed some technical issues like brightness, panel resolution, or vergence–accommodation conflict in existing MR and VR systems (Zhan et al. 2020).

With HoloLens, the user can control the amount of information in order to make the cognitive load tolerable. The quality and stability of the image do not cause motion sickness, and the voice and gesture commands are easy to use. In many studies, users were questioned about the comfort and ergonomics of mixed reality platforms (Gregory et al. 2018; Léger et al. 2017).

The gap between research and clinical work is caused to a significant extent by costs. Hopefully, this will change over time as these devices become more popular and, therefore, more affordable. However, when compared to 3D printing, mixed and augmented reality applications cost less.

It seems that the younger generation of practitioners is more willing to try new technologies than older doctors. There is also a problem of mindset of medical practitioners who are taught to be risk-averse and to use only proven methods; hence, they are oftentimes reticent to try new things. What has to be changed to move an innovation into practice is the mindset among clinicians.

In conclusion, it will take some time before we witness the use of MR in healthcare on a daily basis, but it is inevitable. It represents a phenomenal opportunity to adapt new technologies in medicine in order to improve patients’ outcome and safety. Despite some challenges that for now delay its wider use, I am sure that mixed reality will become a part of future medical and surgical practice as it is already a part of our everyday life.