Augmented Reality in Medical Education and Training: From Physicians to Patients

Abstract

This chapter discusses the intersection of Augmented reality (AR) and the education of medical students, physicians, surgeons, and patients. For medical students tasked with absorbing a large body of medical knowledge in a short period of time, AR technology can help them learn anatomy, view interactions with patients from new perspectives, and observe novel procedures alongside their mentors. Physicians and surgeons can continue using the AR technology that helped them through medical school as well, particularly for laparoscopic procedures, neurosurgery, and cardiology. Augmented reality technology has also been utilized to guide patients through treatments when they lack access to medical professionals, inform patients on medical procedures, and allow patients to educate themselves about their conditions. As AR is still in its infancy, this chapter will explore AR’s current capabilities as well as potential future applications in medical training and education with respect to both health practitioners and patients.

1 Introduction

Research studies of preliminary Augmented reality (AR) prototypes reveal that AR has potential as a unique educational tool, especially in medicine. AR applications may not only enhance medical education at each stage of training but may also help patients become more involved and informed in their medical decision-making. For instance, in medical school, students are required to mentally convert 2D textbook images of human anatomy to 3D models that they can later apply in a clinical setting. With AR technology, medical students will be able to more accurately and swiftly detect malformations, educate patients, and understand anatomy. In the surgical field, AR simulations may provide a risk-free environment for residents to practice without increasing patient risk and may also be used to deliver additional information to surgeons intraoperatively. Lastly, physicians may turn to AR applications to help patients better understand their treatments, thus empowering patients to take an active role in their medical treatment plan and participate in medical decision-making.

To better understand the value of AR in medical education, it is first important to understand how it differs from its closely related, and more well-known sister technology—Virtual reality (VR). VR refers to the generation of an entirely artificial, computer-simulated 3D image or environment with which users can interact in real-time (Khor et al. 2016). Everything the user sees during a VR experience is computer-generated. However, AR involves overlaying an interactive, digital layer of information on top of the physical environment (Barsom et al. 2016). For instance, AccuVein superimposes a map of vasculature on the surface of a patient’s skin to help medical professionals find veins before drawing blood or inserting an intravenous line (Fig. 7.1) (Khor et al. 2016). Thus, AR modifies or enhances the user’s real-life experience, while VR places the user in a completely simulated environment.

Fig. 7.1

A medical professional using AccuVein to visualize veins before drawing blood (AccuVein Inc., NY, USA)

As technology advances, applications and simulations involving AR are becoming increasingly incorporated into various aspects of the medical field. Although AR technology is still in its infancy, this chapter discusses the beginnings of AR in medicine and its current and potential future applications in the field of medical education.

2 Augmented Reality in Medical School

Medical students are tasked with becoming experts in human anatomy, adept at diagnosing disease, proficient in creating beneficial treatment plans, and effective at guiding and comforting their future patients throughout their medical journey. While medical school only lasts four years, learning is a career-long process for physicians, especially since the body of medical knowledge is growing exponentially. One study examining the challenges facing medical education estimates that by 2020, the current body of medical knowledge will double every 73 days (Densen 2011). Thus, medical professionals are lifelong learners, constantly working to remain knowledgeable about the latest medical breakthroughs and discoveries.

Augmented reality shows great potential in facilitating medical students’ learning, whether by aiding in studying for anatomy exams by providing intuitive 3D models, improving fine motor skills, or helping students see interactions from a patient’s perspective. Augmented reality can make learning faster, more efficient, and more engaging. Thus, it gives medical students new ways to fully understand the complex structures described in their textbooks and apply that learning in realistic settings. However, it has also been argued that AR, though efficient and engaging, cannot replace interactions with real patients and can even act as a distraction that hinders, rather than aids, a medical student’s education. As advancements are made in the medical field, it is important to keep the patient at the forefront of a medical student’s education and ensure new technology does not create a barrier between the patient and physician. If used carefully, AR has great potential to better connect medical students to the material they are learning and to the patients they serve.

2.1 Augmented Reality and Anatomy

For medical students—who are learning a high volume of information about the human body, often for the first time—any innovation that makes learning more efficient is welcome. Medical students reported that learning with mobile AR technology, specifically 3D video animations, 3D human anatomy models, and diagrams on their phones, requires less cognitive effort than textbook learning. Mobile AR technology (Fig. 7.2) also allows them to access materials whenever they want, giving them flexibility in their studying. These mobile AR applications, or “apps” integrate virtual learning objects into the real world and allow students to interact directly with the environment (Kucuk et al. 2016). Medical students who studied anatomy using a mobile app with AR technology were able to learn more items in an allotted time period than students who studied using textbooks alone (Albrecht et al. 2013). In addition to increasing efficiency, AR technology gives students a better understanding of spatial relationships, better captures their attention, and reduces failure rate once a student is actually applying what they learned (Zhu et al. 2014). Thus, AR gives students a comprehensive introduction to the human body before they work directly with patients or cadavers, making the transition from textbook pictures to real structures easier (Kugelmann et al. 2018).

Fig. 7.2

a AR Anatomy app used to help medical students memorize organs. Photo courtesy of Zed Interactive. b AR Anatomy app used to help medical students with spatial orientation and learning parts of the skull. Photo courtesy of Wikimedia Commons. No changes were made. The license for this photo may be found at https://creativecommons.org/share-your-work/licensing-considerations/compatible-licenses

AR provides real-time feedback that captures medical students’ attention and immerses them in the learning experience (Kamphuis et al. 2014). When compared to students who studied anatomy using Virtual reality (VR) and tablet-based learning, students who used AR scored similarly on anatomy tests but had increased engagement and learner immersion without the adverse effects of VR, such as dizziness or nausea (Moro et al. 2017). The employment of AR technology such as a “Magic Mirror” (Fig. 7.3), which uses a computed tomography, or CT dataset to allow a student to view the anatomical structures in their own body, and “Gunner Goggles,”¹ which add links, videos, and 2D images to a traditional textbook, further advances this engagement in learning anatomy. A vast majority of medical students felt the “Magic Mirror” had educational value (Ma et al. 2015) as did the “Gunner Goggles,” and found AR to be a viable, and sometimes better alternative to previous exam review methods (Wang et al. 2016).

Fig. 7.3

a A man using the “Magic Mirror” to view his ribcage and organs. b The CT dataset corresponding to the red circle in image a (Bork et al. 2019)

By providing medical students with an immersive experience, AR allows them to practice patient care in a realistic setting without putting any patients in the care of inexperienced students. Augmented reality that superimposes images from anatomical datasets onto simulated patients allows medical students to visualize anatomical structures and interact directly with anatomy in a way that they can easily apply to a clinical setting (Fig. 7.4).

Fig. 7.4

Using AR technology to project anatomical dataset onto a simulated patient. Photo courtesy of CAE Healthcare

This can also be replicated using radiological images to give students a view of spatial anatomy or ultrasound images to visualize blood flow. Augmented reality can also be combined with haptic technology to give students tactile feedback, allowing them to feel the difference between tissues without touching a patient before they are experienced enough to do so (Khor et al. 2016). The applicability of AR to anatomy education extends even further, with studies showing students using AR to visualize patient-specific idiosyncrasies and deformations in the lungs (Kamphuis et al. 2014) or accurately measure a patient’s kidney in preparation for an ultrasound (Ebner et al. 2019).

This technology can be used to aid students in observing procedures for the first time, providing them with patient-specific diagrams of anatomy for learning purposes. For example, students working with the same cadaver regularly have the opportunity to really get to know that particular body’s normal anatomy. However, with AR goggles or mobile AR technology, students could then view that anatomy overlaid with various abnormal pathologies that can teach them about various diseases and how they present. Thus, AR technology provides students with early opportunities to learn about patient-specific disease presentation.

2.2 Augmented Reality and Mentoring

AR fosters interactions between medical students and their mentors in remote locations, allowing students to virtually experience procedures they are not yet skilled enough to perform themselves. In recent years, AR has developed to the point that a remote surgeon could superimpose their hands onto another surgeon performing a procedure with an AR headset. Similarly, Microsoft Hololens² (Fig. 7.5) employs AR technology to project a hologram of someone in a different location, which allows surgeons worldwide to work in the same operating space with holographic projections of each other. These technologies could similarly be employed to train medical students in their clinical years and help them become more comfortable performing new procedures alongside their mentors and peers (Khor et al. 2016).

Fig. 7.5

A man wearing Microsoft Hololens. Photo courtesy of Tyler Ferguson—Creation Media. No changes were made

2.3 Augmented Reality and Clinical Skills

AR also has the potential to help medical students improve their clinical skills and better interact with patients. Typically, medical students assess their clinical skills by working with simulated patients and watching the interaction afterward from a “bird’s eye view” camera in the corner of the room. Students have expressed that this bird’s eye view does not always allow them to properly assess their ability to make eye contact or adjust their facial expressions throughout the conversation.

To remedy this, Google Glass,³ AR technology on glasses that can record from the point of view of the wearer, was used to help medical students improve their interactions with patients (Fig. 7.6). One study allowed medical students to perform a clinical simulation while watching a video of a simulated patient in respiratory distress through Google Glass. In another study, students were recorded giving a terminal cancer diagnosis to a standardized patient who wore Google Glass to record the interaction. A majority of medical students felt the Google Glass would be valuable to the clinical skills training program, though some students found it distracting or felt self-conscious because of it (Tully et al. 2015). Overall, students felt positively about the simulation and felt it was realistic, though some cited technical difficulties such as difficulty connecting to a wireless network, short battery life, and overheating equipment as a challenge (Chaballout et al. 2016).

Fig. 7.6

a A woman wearing Google Glass. Photo courtesy of Antonio Zugaldia. No changes were made. The license for this photo may be found at https://creativecommons.org/licenses/by/2.0/. b A man using Google Glass to perform a clinical simulation in an ambulatory setting. Photo courtesy of Institut Informatique de Gestion. No changes were made. The license for this photo may be found at https://creativecommons.org/licenses/by-sa/4.0/deed.en

2.4 Limitations of Augmented Reality for Medical School

Despite its many benefits to medical students, AR still has its limitations, and medical schools are still determining the best way to implement this new technology. Though engaging, AR places a larger extraneous cognitive load on students because it forces them to orient physical models in their minds. A study done in 2019 shows that students studying neuroanatomy who worked with physical cross sections of the brain showed significantly more improvement in test scores than students who worked with AR (Henssen et al. 2019). Thus, the classical method of learning with a physical model may sometimes be superior to AR learning methods, as a classical model incorporates a tactile component into learning and allows students to rotate a model physically. It has been argued that though AR can bring students valuable immersive experience, it still can never compare to the tactile learning that results from working with a cadaver (Kamphuis et al. 2014). However, medical students in these studies expressed a compromise: AR could be used to prepare students for the learning they will do in a dissection room and supplement, rather than replace, that tactile learning with cadavers that is so important in medical education. Augmented reality allows students to see not just what normal looks like, but also abnormal, which may be difficult to observe using only cadavers. Thus, this new technology may be the key to exposing more medical students to abnormal pathologies earlier in their medical education.

AR is still finding its place in medical education, engaging students with a holistic view of anatomy and a patient’s view of their clinical skills. As these students progress in skill, graduate medical school, and become residents, we will begin to see the impact of AR on these students’ abilities. Until then, we will continue to discover ways that AR can have a place in their learning.

3 Augmented Reality for Residents and Attendings

Augmented reality applications have attracted public and scientific interest due to their gradual incorporation into medical practices. Medical professionals have already begun using AR to enhance global collaboration, aid patients in rehabilitation, and improve surgical training and outcomes (Khor et al. 2016).

A review of approximately 2500 published studies revealed that AR applications have helped healthcare providers better understand spatial relationships, strengthen cognitive-psychomotor abilities, and prolong concept retention (Zhu et al. 2014). They have also proven especially useful for training surgical residents: while current surgical curricula heavily emphasize the acquisition and perfection of technical skills, there is a dearth of situational awareness training for residents (Barsom et al. 2016). As no two patients are the same, attention to detail and an ability to adapt to unexpected variations are crucial for residents to avoid making errors during surgery (Graafland et al. 2014). To better prepare residents for rare cases or unexpected complications, AR applications are a promising way to more optimally and realistically train residents (Barsom et al. 2016). Increasing student engagement and situational immersion during training also helps enhance conceptual understanding, knowledge retention, and automatic skills (Birt et al. 2018).

The incorporation of AR in medicine also has the potential to revolutionize the world of surgery beyond training. On an individual level, surgeons would no longer need to break scrub to check test results or consult patient scans, as the information can be virtually delivered to them, thus increasing efficiency and reducing the risk of infection (Fig. 7.7) (Khor et al. 2016). On a global scale, AR technology has the ability to connect physicians on opposite sides of the globe and make international surgical collaborations easier. For instance, Virtual Interactive Presence and Augmented Reality (VIPAR) is an AR application that allows surgeons to project their hands into the displays of other remote surgeons wearing headsets (Fig. 7.8). Other platforms like Proximie have also enhanced the global dissemination of surgical experience, as surgeons in developing countries can watch experts in other parts of the world perform operations (Khor et al. 2016).

Fig. 7.7

With AR technology, patient data could be delivered directly to the surgeon’s field of view without requiring the surgeon to break scrub. Photo courtesy of Brother UK. No changes were made. The license for this photo may be found at https://creativecommons.org/licenses/by/2.0/

Fig. 7.8

a The remote surgeon’s hand (yellow arrow) can be seen in the center while the operating surgeon’s hand is on the left and right (red arrows). b A remote surgeon’s hand is projected into the surgical field of view to guide the operating resident (Baker et al. 2015). Photo courtesy of The Orthopedic Journal of Harvard Medical School

Due to its ability to effectively fill the gap between virtual and physical reality, AR has quickly risen as a promising technology to be integrated into the medical field. The following sections examine three common surgical areas that have already begun to utilize AR in more detail.

3.1 Laparoscopic Surgery

3.1.1 Challenges of the Field

Laparoscopic surgery, which involves the insertion of a miniature camera into a patient’s abdomen to perform a procedure, has several benefits, including reduced scar size, low postoperative morbidity, and a shorter recovery time (Nicolau et al. 2011). While open surgeries put patients at greater risk compared to minimally invasive ones, endoscopic procedures are more challenging for surgeons due to reduced depth perception, a limited field of view, and an altered sense of touch transmitted through an instrument—known as the fulcrum effect (Fig. 7.9) (Kamphuis et al. 2014). Augmented reality can help physicians overcome these challenges in part by increasing their intraoperative vision, or how much they can see during a procedure. More specifically, AR applications can help surgeons see into a patient prior to laparoscope insertion and provide additional information about hidden structures during the surgery after the instrument has been inserted. However, to achieve the seamless integration of virtual learning experiences and real-life physical context, several systems must first be developed and synchronized.

Fig. 7.9

A surgeon uses laparoscopes to perform a minimally invasive laparoscopic surgery. Photo courtesy of Seaman Jordan Bair. No changes were made. This photograph is considered public domain and has been cleared for release

3.1.2 How Augmented Reality Applications Work

In order for an AR application to be routinely incorporated into laparoscopic surgeries, four systems must work in unison to deliver the final, desired blend of virtual and real data: a 3D visualization system, a display system, a tracking system, and a registration system (Nicolau et al. 2011).

First, preoperative data must be converted into intraoperative information. Patient-specific data gathered from CT and magnetic resonance imaging (MRI) scans is used to generate 3D models of patients either through direct volume rendering or surface rendering. Direct volume rendering is best for visualizing malformations detected through contrast imaging but less useful for procedures involving the removal of all or part of an organ, as image volume cannot be computed. On the other hand, surface rendering is useful for planning resections and preoperative simulations but often takes longer and is more difficult (Nicolau et al. 2011).

After creating a virtual model of the patient, there are several ways to display AR data, each with its own benefits and drawbacks. See-through displays place a semi-transparent mirror in front of a surgeon’s gaze: while still being able to see their patient through the mirror, surgeons are also able to see AR data displayed on a screen and reflected by the mirror into the surgeon’s eye. Although this method allows surgeons to combine the quality of human sight—something that cameras cannot replicate—with virtual information, this system is expensive and requires real-time, accurate tracking of pupil positions. Projection-based display systems are another option, where AR information is directly projected onto the patient’s skin. Even though this technique provides one of the most real feelings of patient transparency, it is also one of the most inaccurate because of significant perception errors. The location of a surgeon’s head and the patient’s skin must be tracked, and even then, this technique would still only be accurate for structures that lie directly under the skin. Video-based displays work by superimposing preoperative patient data onto the live video stream. In other words, AR information is directly displayed on the endoscopic image (Fig. 7.10). Thus far, this method seems most appropriate for laparoscopic surgeons (Nicolau et al. 2011).

Fig. 7.10

Systems for displaying AR data. a In a video-based display, preoperative data is superimposed on a live video stream. b In a see-through display, data is reflected in the surgeon’s eyes. c In a projection-based display, AR data is projected directly onto the patient (Nicolau et al. 2011). (Reprinted from Surgical Oncology, 20, Nicolau S, Soler L, Mutter D, Marescaux J, 189–201, (2011), with permission from Elsevier.)

To maximize the efficiency and accuracy of AR applications, a tracking system is also needed to monitor both the endoscopic instrument and the location and direction of the surgeon’s head. Current techniques use optical infrared tracking, normal calibrated cameras, or electromagnetic tracking. Such devices are necessary for the real-time tracking of position and orientation (Nicolau et al. 2011).

Lastly, a registration system is responsible for superimposing the intraoperative information at the correct physical position. It is of utmost importance that this system be as accurate as possible, as any errors can result in the incorrect guidance of a surgeon and dangerous surgical movement. The two most common methods are interactive registration AR and automatic registration AR. Interactive Augmented Reality (IAR) requires users to manually identify anatomical landmarks on both the real patient and the 3D patient model while automatic AR (AAR), as the name suggests, is a fully automated process to register anatomical landmarks. To date, researchers have had more success with IAR. Projector-based IAR has been reported to improve port placement in laparoscopic gastric surgery and reduce operative time and X-ray exposure in kidney stone removals. Camera-based IAR has also helped surgeons resect tumors on the adrenal gland, liver, pancreas, and parathyroid (Nicolau et al. 2011). Although more applications currently use IAR, these systems not only require additional medical personnel but also heavily depend on the skill of the user, thus limiting its clinical feasibility. Researchers will eventually need to overcome the challenges posed by AAR in order to create AR applications that can realistically be incorporated into medical practice.

For laparoscopic surgeons, AR information would ideally be displayed directly in the endoscopic view (Fig. 7.11). But before this can be done, AR applications first must account for natural human movements, like breathing motions and heartbeats, as well as real-time changes to organs made by the surgeon during the operation. Ultimately, even though significant progress has been made on the separate systems, the different pieces of the AR puzzle have yet to be perfected and seamlessly assembled into an accurate, cost-effective, user-friendly AR application.

Fig. 7.11

AR technology is used to superimpose vein and tumor locations gathered from preoperative images onto the liver during a tumor resection (Talbot et al. 2015; Haouchine et al. 2013)

3.1.3 Uses of Augmented Reality in Laparoscopic Surgical Training

Despite the need for further technical development, AR applications still have several educational benefits that allow for experiential learning (Dalgarno and Lee 2010). For laparoscopic surgery specifically, “part-task practice” has been shown to be the most effective way to master the complex psychomotor skills required in procedures. Through “part-task practice,” trainees repeatedly practice certain motions until they become ingrained in muscle memory and reach a certain level of automation (Kamphuis et al. 2014). Residents and newer surgeons can practice on mannequins that have been overlaid with anatomical AR information (Fig. 7.12). To minimize simulation dissonance, AR applications must be easy to use and have high simulation quality, design, and realism. Given the technical challenges and high costs of developing AR applications, educators must be able to decide which surgical skills are worth investing time and money into developing simulations for (Birt et al. 2018). However, it is still important to remember that simulation training should be seen as a supplement, rather than a replacement, for hands-on training.

Fig. 7.12

A model without (a) and with (b) anatomical information projected onto it allows residents to study and practice techniques (Aaskov et al. 2019)

3.2 Neurosurgery

3.2.1 Challenges of the Field

Neurosurgery is the medical specialty concerned with the diagnosis and surgical treatment of illnesses associated with the human nervous system, including the brain and the spinal cord. Neurosurgery requires an enormous amount of knowledge, near-perfect technical skill, and meticulous preparation. In addition to being limited to a minuscule surgical field to maneuver in, surgeons must also avoid the fragile blood vessels and critical brain structures scattered throughout the surgical field, mere millimeters away from their surgical instruments (Pelargos et al. 2017). Even the smallest error can potentially render patients blind, deaf, or paralyzed.

Given the gravity of technical error in neurosurgery (Matthew et al. 2014), it is especially important for trainees to receive rigorous and thorough training before beginning to operate on live patients. Current education models are largely based on models created in the 1890s and focus mostly on mentorship, number of operations performed, and time spent in the hospital (Pelargos et al. 2017). However, with recent laws placing working-hour restrictions to not only improve resident working conditions but also patient safety, the neurosurgical field needs to take advantage of technological advancements to create higher quality, more time-efficient training paradigms (Matthew et al. 2014).

3.2.2 Uses of Augmented Reality in Neurosurgical Training

Unlike the laparoscopic surgical field that has already begun incorporating AR applications into surgeries on live patients, AR applications in neurosurgery are mostly limited to use in education, training, and preoperative rehearsals. Given the current state of AR technology, AR applications are most valued in this field for their ability to increase opportunities for residents to gain surgical experience without a corresponding increase in risk for patients. Even though AR systems in neurosurgery are still significantly underdeveloped compared to their VR counterparts, they can still be used to project anatomical information gathered from pre-scanned CT, MRI, or ultrasound data onto the surgical field, thus illuminating structures hidden intraoperatively. Lastly, they could possibly be used to develop a proper, standardized set of evaluation criteria for freshly minted trainees and experienced surgeons alike (Pelargos et al. 2017). Specific procedures that scientists and physicians have begun using AR applications for include percutaneous vertebroplasty, thoracic pedicle screw placement, percutaneous spinal needle placement, facet joint injection, and spinal needle insertion (Pfandler et al. 2017).

3.2.3 Limitations of Augmented Reality in Neurosurgery

A systematic review of published studies on AR applications in neurosurgery revealed that the majority of study participants gave positive feedback and commented on the realistic feel of the simulations. However, these results have not been completely validated due to major design flaws identified in the majority of the studies, including a lack of controls, validity assessments, and randomized study designs. Inconsistencies in how studies define “validity” and “reliability” also make it difficult to compare and validate one study’s results with another’s. In general, current studies on AR applications in neurosurgery are too different in terms of study design, quality, and outcome measures to allow for any meaningful conclusions to be made. Ultimately, more rigorous, standardized studies need to be conducted before the benefits of AR simulation in neurosurgery can be confirmed (Matthew et al. 2014).

3.3 Cardiology

Researchers and physicians alike have begun exploring how to apply AR applications to various areas in cardiology, including physician education, cardiac rehabilitation, pre-procedural planning, and intraoperative use. For surgeries, AR applications have the potential to help physicians visualize hidden structures in 3D, manipulate digital images in real-time, and improve control of surgical tools.

One early feasibility study evaluated the use of an AR guidance system in conjunction with Microsoft Hololens during aortic valve replacement and device placement procedures. With AR technology, physicians no longer needed to perform aortic arch angiograms–which involves injecting a dye into the patient’s arteries and using X-rays to observe how it travels–and patients experienced less exposure to X-rays. Researchers optimistically concluded that AR applications are clinically feasible, reduce surgery time, and decrease patient risk (Southworth et al. 2019).

Another study looked at the use of a first-person point of view AR application meant to guide needle insertion during catheter placement. Surgeons found the AR technology realistic, easy to use, and helpful. They also thought the integration of AR into medical education would benefit future students. However, they criticized the fit of the AR goggles, the restricted allowable head movement, and the cost of implementation. Despite these limitations, outcomes indicate that AR applications have much potential: with AR guidance, experienced and non-experienced participants were equally likely to identify anatomical landmarks and insert needles correctly (Rochlen et al. 2017).

Other assessments of the effects of AR simulations on echocardiography training revealed similar sentiments (Platts et al. 2012; Weidenbach et al. 2009). In addition, a significant number of participants believed that, due to the unpredictable and imperfect nature of the clinical environment, simulation training would not be able to completely replace hands-on, clinical experience. Thus, these budding technologies may prove to be more useful during the initial training stages (Platts et al. 2012).

Ultimately, AR provides surgeons with a multitude of new ways to improve surgical outcomes, educate residents, and increase efficiency and patient safety. Though nothing can fully replace hands-on learning, AR has the potential to provide a valuable supplement to surgical experiences. With new applications continuing to be developed, AR applications can also be used to help patients better understand their medical treatments and surgical procedures. As healthcare delivery becomes increasingly patient-centered, utilizing technology to allow patients to become more active in developing their treatment plans will become essential.

4 Patient Education

In recent years, the medical field has begun shifting toward a more patient-centered approach instead of a more traditional, disease-centered approach. While the latter is effective when a patient has one primary illness or concern, it quickly becomes less so when a patient is experiencing multiple chronic conditions (Tinetti et al. 2016). This shift to the patient-centered approach—which focuses on identifying the patient’s individual health goals and what they are willing to undertake in order to reach those goals—is timely considering the aging of the U.S. population, which will inevitably lead to an increase in the number of patients with multiple chronic conditions (Colby and Ortman 2014).

A critical factor in successfully employing this approach is an active doctor-patient relationship where the patient is able to communicate their goals and the physician is able to inform the patient about their condition(s) and potential treatments. In fact, empowering patients to become active participants in their health care has numerous benefits, such as increased patient satisfaction and trust in their physicians, decreased patient anxiety, and improved adherence to medical regimens (Vahdat et al. 2014). Due to its interactive and visual format, there is a growing belief that AR is particularly suited to engage and educate patients at various stages of their treatments, from the preoperative consultation and the inpatient experience to the recovery and rehabilitation process (Zucker et al. 2018; Courtney et al. 2015).

4.1 Preoperative Education

Surgery, most likely due to its immediate results and clear risks, is one area of medicine where patients are particularly motivated to have a comprehensive understanding of their options and the potential results. The preoperative consultation is the surgeon’s opportunity to educate their patients and ensure they have the tools and information necessary to make informed decisions; in some fields, such as urology, physicians heavily rely on preoperative images when discussing significant procedures with their patients (Wake et al. 2019). However, learning about the complex structure of human anatomy through 2D images can be difficult, which is only exacerbated by the stress of deciding whether or not to undergo surgery (Fig. 7.13).

Fig.  7.13

2D image (left) versus a 3D image (right) both illustrating the outcome of jaw surgery to a patient. Photo provided courtesy of Dr. Michael Malek and Akh Linz. The top of the original photo was cropped for the purpose of this chapter. Otherwise, no changes were made. The license for this photo may be found at https://creativecommons.org/licenses/by-nc-nd/2.0/.

There has been a long-standing interest in utilizing AR to provide patients with a 3D visualization of their specific anatomy. In 2010, a team comprised of computer scientists and plastic surgeons in Germany developed a “Magic Mirror” for patients preparing for breast augmentation. Wearing a target tracker on their shoulder, patients could stand in front of a screen where their body would be virtually reconstructed and projected, illustrating the potential breast augmentation. When the team consulted with multiple surgeons regarding this software, all of the surgeons stated there was a need for more advanced teaching tools for patients, and most of the surgeons expressed interest in employing the “Magic Mirror” in their own practice, assuming the visualizations were accurate (Wucherer et al. 2010). More recently, an AR tool aptly called RAD-AR was developed to assist in the various facets of radiotherapy treatments. In addition to providing physicians with more detailed 3D datasets and improving the accuracy of patient positioning during treatment, RAD-AR may also be used for educating patients on the importance of pre-treatment preparation. It is common to use 2D sketches of patients’ anatomy in these educational sessions, but a 3D visualization may be more intuitive and more likely to improve patient understanding and compliance with pre-treatment procedures (Cosentino et al. 2017).

The conversation regarding AR’s potential application in preoperative consultation was reignited in 2018 by a letter written by Benjamin Zucker, Dr. Paris Tekkis, and Dr. Christos Kontovounisios, which was published in a surgical medical journal. The authors argued that AR’s intuitive visualization—compared to traditional 2D imaging such as a CT scan or MRI—and innate interactive nature would allow patients to gain a better understanding of surgical procedures and enable patients to direct their educational experience. However, a study conducted in 2019 comparing the effectiveness of 3D printed models, AR, and 3D computer visualizations (Fig. 7.14) in educating surgical patients found that AR may not be the optimal tool, currently, for patient education. The 3D printed model was found to be the most beneficial overall, increasing patients’ understanding of their anatomy, disease, and treatment plan as well as making patients feel more comfortable with their treatment. Despite the fact that patients perceived AR to be useful, it did not seem to actually improve their understanding (Wake et al. 2019).

Fig. 7.14

Kidney cancer models that are a 3D printed, b 3D computer-generated, and c AR. The tumor is shown in white in the 3D printed and 3D computer-generated models and purple in the AR model. The artery is shown in red, the vein in blue, and the collecting system in yellow (Wake et al. 2019). The bottom of the original figure was cropped for the purpose of this chapter. Otherwise, no changes were made. The license for this photo may be found at https://creativecommons.org/licenses/by/4.0/

Although the results of this study raise doubts regarding the efficacy of AR in patient education, this is currently the only relatively large-scale study published on this topic and there is still much more research to be done. Augmented reality’s optimal role may be serving as a supplementary resource, as opposed to a primary one. Furthermore, its potential will grow as AR evolves and both physicians and patients develop their proficiency in utilizing the technology.

4.2 The Pediatric Inpatient Experience

Going to the hospital can be a stressful experience, particularly for children who are less likely to understand why they are at the hospital or why they should allow the doctor to run a test. To help alleviate children’s potential fear and anxiety, some children's hospitals have begun incorporating VR and AR games into medical procedures and the inpatient visit. While the majority of these experiences utilize VR and focus mainly on entertaining and distracting the child during a procedure, some games use AR to familiarize the patient with a specific medical procedure or the inpatient visit overall.

The Alder Hey Children’s Hospital in the United Kingdom recently released an application called “Alder Play” for both Android and iOS (Fig. 7.15). It can be downloaded prior to a child’s visit and was designed to prepare children for their hospital stay, guide them through potentially frightening moments, and reward them for their courage (Alder Play). Users choose a virtual animal guide who leads them through the hospital, familiarizing them with the facility by using AR. They can even message with their guide who is equipped with an artificially intelligent chatbot powered by IBM’s Watson. The app also supplies video tutorials on common procedures, such as blood tests, so children are less intimidated when they experience the procedure. Doctors can also award their patients in-game stickers as positive reinforcement (Jimenez et al. 2019; Debczak 2017).

Fig. 7.15

The Alder Play children’s guide gives a child a tour of the hospital playroom (Debczak 2017)

Lucile Packard Children’s Hospital Stanford is another hospital applying this technology through their Packard Children’s Childhood Anxiety Reduction through Innovation and Technology (CHARIOT) program. The majority of their initiatives utilize VR to engross patients in virtual adventures as they undergo procedures, but they have begun using AR to allow curious patients to observe and even be an active participant in their procedures (Jimenez et al. 2019; CHARIOT Program 2018; DeTrempe 2017). For those children who may be more inquisitive than afraid, the latter fulfills the patient’s curiosity while also making it easier for medical staff to execute their procedures. Through these experiences, children are able to learn more about their hospital stay, understand their procedures, and feel more comfortable in a hospital setting.

4.3 Telemedical Applications

Augmented reality’s potential for patient education is not restricted to the four walls of the hospital or the doctor’s office. Access to health care, especially in rural areas, is an ongoing and major issue in the U.S. A study published in 2015 reported that rural communities were still facing considerable barriers to accessing healthcare, including limited health care services and public transportation, understaffing of trained physicians, and a lack of internet access (Douthit et al. 2015). Additionally, a report by the Georgetown University Health Policy Institute noted that the rural population is more likely to be older, poorer, and uninsured as well as experience higher rates of injury and suicide (Wagnerman 2019). Telemedicine, the use of technology to provide medical services to a patient in a distant location, has been a key element in the medical field’s campaign to deliver health care to all in need of it. Augmented reality is now being integrated into telemedicine in order to increase both the physician’s and patient’s ability to interact and communicate with each other.

In 2014, two pilot studies were published regarding two telemedicine systems supported by AR and their effectiveness in training laypersons to accomplish a specific task. The first was a preliminary test for a telerehabilitation system called Ghostman. This system was designed to allow physical therapists to remotely instruct, correct, and monitor their patients’ progress by placing them in their patient’s point of view (Fig. 7.16). Ghostman was first tested by having an instructor guide a participant using chopsticks. Participants received either a face-to-face session with the instructor or a remote Ghostman session with the instructor. There was no difference between the groups with respect to frequency of errors or skill retention over one week, which indicated that Ghostman was as effective as the traditional in-person session (Chinthammit et al. 2014).

Fig. 7.16

An example of teacher instructing a student on how to use chopsticks through Ghostman (Chinthammit et al. 2014). No changes were made. The license for this photo may be found at https://creativecommons.org/licenses/by/3.0/

The second study sought to assess AR and telemedicine’s potential to guide untrained persons to correctly use an electrocardiogram (ECG) device to successfully conduct an ECG on a patient. The AR application, developed using ARToolKit, an open-sourced AR library, was specifically designed to navigate the user with a combination of audio messages and visuals such as a text-box, pointers, and spots superimposed on the user’s field of view (Fig. 7.17). Participants were able to successfully conduct an ECG on a real patient with an average positioning error of less than 7 mm (Bifulco et al. 2014). However, as noted in the study, these tests were conducted in an extremely controlled environment, and the system’s ability to account for fluctuations in both the patient’s and environment’s characteristics needs to be developed before it is employed in real-life scenarios.

Fig. 7.17

Point of view of a user being guided by the AR application by correctly placing electrodes on a mannequin or human body to carry-out an ECG (Bifulco et al. 2014). No changes were made. The license for this photo may be found at https://creativecommons.org/licenses/by/4.0/

Dr. Brett Ponce and colleagues at the University of Alabama at Birmingham School of Medicine took the next step by examining if a telemedical system known as the virtual interactive presence (VIP) system could be used to guide patients through postoperative care when supported by AR. The postoperative period is a crucial time for healing and recovery and generally necessitates follow-ups with providers, as services such as redressing wounds and wound evaluation are commonly required. However, many people are simply unable to attend their follow-ups for logistical or financial reasons. The VIP system, available as a free iOS mobile application, creates a hybridized feed of the patient and physician viewpoints, allowing them to virtually interact (Fig. 7.18). The study found that both patients and surgeons found the system useful for wound inspection, dressing, and medical equipment management, and felt it was superior to communication via phone, text, or email. Unfortunately, there were two instances where patient-participants attempted to contact their surgeon via the application during a medical emergency instead of seeking immediate, physical attention, but both were eventually resolved without long-term effects. Technical difficulties were another issue, as 20 of the 50 participants were unable to complete their session due to set-up troubles, poor network connections, inability to use the application, or incompatible devices (Ponce et al. 2016). As this technology develops, so will its potential applications.

Fig. 7.18

This photo illustrates how a dermatologist may virtually interact with their patients, with the VIP system, by remotely examining a patient’s arm. Photo courtesy of Help Lightning

While the integration of AR and telemedicine shows great promise, it currently faces several limitations. As mentioned at the beginning of this section, a considerable portion of rural areas do not have internet access (Eighth Broadband 2012). Without a reliable and adequate internet connection, this technology is rendered unusable. Additionally, these systems need to optimize inclusivity. For example, when designing them as mobile applications, they should be compatible with both iOS and Android devices. The VIP system was only compatible with iOS, excluding a considerable portion of the U.S. population. Given that Android users are 86% more likely to live in rural areas and iOS users are 27% more likely to live in urban areas, a system compatible with both Android and iOS devices is necessary to reach more of the rural population (Alexander 2019). Security is another major concern; any information transmitted during these sessions should be heavily encrypted, and the databases storing all the information must be closely monitored. Developers should be wary of using open-source libraries to develop their system as there are no requirements regarding who can contribute to the code, and mal-intentioned or poorly written code may be uploaded, potentially introducing security flaws. Given adequate attention and resources, these obstacles are manageable and can be overcome.

4.4 General Patient Education

In addition to facilitating physician to patient education, AR applications are also being created for individual, self-directed use. The EyeDecide App is a free mobile application for iOS devices, which allows users to examine the anatomy and structure of the eyeball at 360°. According to an online publication for medical professionals, users may be able to self-diagnose by compiling their symptoms into the application’s simulation. It even provides a list of recommended, local eye-specialists (Lewis 2013; EyeDecide App). Researchers also developed an AR game for Android mobile devices to support and educate pediatric patients with diabetes (Fig. 7.19). Players advance through three levels, each focusing on a different food group, and must select the correct carbohydrate content level corresponding to the food visualized on the physical plate. To move on to the next level players must score 70% or higher. One study found that children who played the game improved their knowledge surrounding the carbohydrate content of foods displayed in the game regardless of age or gender (Calle-Bustos et al. 2017).

Fig. 7.19

An outline of each of the three levels along with screenshots of the gameplay (Calle-Bustos et al. 2017). No changes were made. The license for this photo may be found at https://creativecommons.org/licenses/by/4.0/

5 Conclusion

Ultimately, as the field of medicine changes and evolves, incorporating new technologies to better serve patients, AR will find its place educating physicians and patients alike. It is important that medical schools take the time to understand these new technological advancements, utilizing them to improve the education of the next generation of physicians and supplement traditional learning. Though nothing can truly replace the tactile learning experience of working with cadavers or actual patients, AR technology can similarly engage and inform medical students and physicians. Tools such as the “Magic Mirror” and Google Glass can provide an immersive, holistic learning opportunity for medical students that they can carry with them throughout their training. Utilizing AR for laparoscopic surgeries, neurosurgery, and cardiac procedures have allowed physicians to perform minimally invasive, high-risk procedures in a more efficient and safe way. Augmented reality aids physicians in educating patients of any age about their anatomy, their condition, and what their options are, giving them control over their own care during an often stressful time.

Despite the many positive implications of widespread AR use in the medical field, it is not without its drawbacks. Cost and availability are still of great concern, as AR technology is expensive, requiring powerful microcomputers that are not yet widely available and specially trained instructors to implement AR technology in hospital settings (Zahiri et al. 2018). These devices must be functional, light, durable, and comfortable for the wearer performing procedures that require a high level of concentration. As AR technology is more widely implemented in medical education and the medical field, this issue of availability and cost may be reduced due to higher demand and increased comfort using AR technology among doctors. Confidentiality and data management are also important to consider, as bringing technology into the operating or exam room runs the risk of breaching HIPAA (Khor et al. 2016). However, as medical professionals have become increasingly comfortable using technology in their practice, the medical field has also improved the protection of patient privacy through technological advancements and will continue to do so with AR.

In an increasingly digitally-centered world, where even our youngest members of society are engaged with AR technology through games like Pokémon GO, AR will no doubt continue to make its way into the medical field. Though there are many avenues still left to explore, current innovations in AR have already made care more cost-effective and patient-centered. By using this new technology to more fully understand anatomy and patient interactions, medical students are able to enter residency more prepared and confident. Surgeons can not only improve outcomes for their patients long-term but also help them fully understand their procedure and condition like never before. Patient education through AR can improve compliance, stop preventable conditions before they start, and ease patients’ concerns. Patients may one day even be able to communicate with providers long-distance, using AR to understand and discuss their condition. Innovations improving and implementing AR technology are greatly beneficial to doctors and patients alike, providing new perspectives that truly place the patient at the forefront of care.