Keyword

Introduction

Augmented reality involves the overlay of digital imagery onto the real world in real time [1, 2]. This has been used in a wide variety of applications and stems from virtual reality in which a user is able to interact with a virtual environment.

From a historic perspective, a “predictor” of the possibility of visually augmented reality may be seen in the work of Galileo Galilei in the sixteenth century. Early reports described a device of Galilei, referred to as a celatone, that was a face-mounted construct, which held a telescope while allowing basic vision, affording more precise visualization of celestial bodies to assist in navigation. Modern wearable technology began to take form in the 1970s when Steve Mann, a Canadian engineer, began wearing computers to improve his vision and was credited with being the first cyborg [3]. Subsequently there has been a surge in the number of manufacturers of devices as microchips have become smaller and “Heads Up” displays have become more delicate. At this time there are at least 45 companies, ranging from Sony (Minato, Japan) and Google (Mountain View, CA) to crowd funded start-ups such as Technical Illusions (Seattle, WA), with various forms of augmented reality devices in development and a handful with devices available for purchase. These range from $200 to $125,000 offering a range of capabilities with a wide variety of intended applications.

More recent augmented reality efforts began with smartphones, utilizing integrated cameras in order to merge app-based information and the real world. However, this hands-on approach is clearly not ideal for medical and certainly surgical applications. Over the past decade, physicians and surgeons have adopted this technology by way of heads-up displays both in training and to employ patient specific imaging in order to reference or guide intervention [4, 5].

While physicians, surgeons and other health care providers have identified potential applications for this technology, devices developed have suffered from high cost, uncomfortable form factors, suboptimal battery life, or lack of an app-based developer ecosystem [1]. The recent, limited release of wearable technologies using an eyeglass form-factor has begun to address these issues [6–8]. The advent of these wearable, consumer-based, augmented reality devices may provide economical, rapid communication, documentation, and consultation among clinicians.

Technology

The technology behind providing a heads-up display is based upon the use of a combiner. A combiner overlays a virtual image of an object with the actual object in order to provide a new augmented version of the object to the viewer. Combiners can be either curved mirror-based or waveguide-based (diffractive waveguide, holographic wave guide, polarized waveguide, etc.). Combiners vary in their thickness, weight as well as in the field of view provided. The Star™ 1200 manufactured by Vuzix (Rochester, NY) uses a curved mirror combiner (www.vuzix.com). Google® Glass by Google (Mountain View, CA), for example, uses a monocular Flat Combiner (45°) in order to provide a virtual image that is still translucent (Fig. 6.1). Though it is effective, this technology provides a medium field of view, has a medium eye box, and suffers from a relatively thick combiner. On the other end of the spectrum, Innovega (Bellevue, WA) has implemented a tapered opaque light guide that necessitates the use of a contact lens and glass in order to provide a thin, binocular glass, with a large eye box and a very large field of view (nearly 120°), and allows for true three-dimensional full augmented reality (innovega-inc.com).

Fig. 6.1
figure 1

Google Glass became available to the general public in May of 2014 and provides users with a natural language voice command user interface in order to provide a wearable hands-free computing experience

Full size image

Devices

Augmented reality is the more modern term for mediated reality where our perception of the world is modified in some way. The simplest of these devices could be a rear-view mirror on a car; however, cutting edge augmented reality as we think of it today began in 1958 with Heads-up displays (HUD) for fighter pilots [9]. This overlay of computer-generated imagery has progressed since that time and has become infused into our vehicles, our smart phones and now our eyeglasses.

Smart Phones

Augmented reality for the smart phone became a reality with the integration of a camera. This allowed for the overlay of app-based information onto an image of the real world. Various apps used for locating restaurants and stores were the first to take advantage of this form of AR. Satellite AR (Analytical Graphics, Inc., Exton PA) even allows for the AR visualization of satellites as they orbit the earth (Fig. 6.2).

Fig. 6.2
figure 2

A screen shot from Satellite AR (Analytical Graphics, Inc., Exton PA) shows the digital overlay of a satellite’s current position and trajectory about the earth. (Source: https://play.google.com/store/apps/details?id=com.agi.android.augmentedreality. Permission granted under the terms of the GNU Free Documentation License, Version 1.2.)

Full size image

Head Mounted Displays

As previously stated, the original head mounted displays (HMD) were pioneered by the military for use by fighter pilots (Fig. 6.3). Though many companies have entered the race to engineer the ideal heads-up display, not all will be applicable for medical or surgical applications.

Fig. 6.3
figure 3

The Heads-Up Display from the cockpit of an F-16 fighter plane displays altitude, artificial horizon, and magnetic heading along with other information to the pilot, without ever losing site of the external environment. (Source: http://www.defenseindustrydaily.com/39m-to-keep-f16-huds-aok-02131/. Permission granted under the terms of the GNU Free Documentation License, Version 1.2.)

Full size image

For example, Sony released the Glasstron™ in 1997, which did not have any transmissivity and was intended for viewing of multimedia such as movies. Since that time, Sony and most of its competitors have moved toward AR compatible eyewear with varying AR compatibility.

Though many companies have thrown their hat into the head mounted display (HMD) arena, not all seek the same goal. Canon (Jamesburg, NJ) has developed their Mixed Reality eyewear with a price of $125,000, with the hope of providing a professional grade AR for use in multiple industries. Civil engineers would have the ability to visualize gas, water, and electrical lines as they exist or even as they are proposed prior to ever breaking ground.

Physicians and surgeons may be able to use the same device clinically and fully integrate the smart glass into the electronic medical record (EMR) of choice. This integration may allow for documentation, chart reviews, intraoperative communication, and information augmentation.

Not all groundbreaking work is being performed by tech based companies though. In 2007, the 3D Visualization and Imaging System Lab at the University of Arizona developed a unique polarized HMD and was subsequently awarded Army phase I and II Smart Business Innovation Research (SBIR) grants (www.sbir.gov) for its work in the field.

User Interface

As some authors have suggested, a HMD may cause information overload [10]. The user interface (UI) becomes important in order to tailor the content provided by the device. Google Glass uses a touchpad located on the side of the glass and voice commands in order to interact with the content of glass. Most of the newer devices have Bluetooth® (Bluetooth SIG, Kirkland, WA) connectivity and will allow for interface with a smartphone and computer, which can be used as the UI. The Technology Partnership (Melbourne, UK) has integrated electrodes capable of interpreting extraocular muscle activity as a proxy for eye tracking, which is implemented as the UI. Olympus Optical (Shinjuku, Japan) is one of the many users that have experimented with hand gesture recognition as the UI.

Google recently received recognition for their consumer smart glass (Google Glass) that did not require linkage to a device in order to display information. Google Glass is a computer with an optical head mounted display comparable to a 25″ HD television viewed from 8 ft. It uses bone conduction for audio, stores 16GB, possesses a 5MP camera, Wi-Fi, Bluetooth, voice recognition and has roughly 4 h of continuous, active battery life. It is incorporated into a familiar eyeglass form factor (Fig. 6.4).

Fig. 6.4
figure 4

Google Glass. One eye consists of a see-through monocle and integrated camera with bone-conducting or direct in-ear microphone. It connects to the Internet via Bluetooth and mobile phone or directly via WiFi

Full size image

Inventors continue to push the envelope. Babak Parviz, an affiliate professor of electrical engineering at the University of Washington, has already created a contact lens with a single LED, and it is powered wirelessly with radio-frequency (RF) energy (Fig. 6.5) [11]. This is just the beginning. More recent works have shown the creation of contact lenses capable of sensing tear glucose concentrations [12].

Fig. 6.5
figure 5

A mock up of an augmented reality contact lens pioneered by Babak A. Parviz, PhD, a bionanotechnologist at the University of Washington. (Source: B. Parviz. 2009, September. Augmented Reality in a Contact Lens [Online]. Illustration by Emily Cooper. Available at: http://spectrum.ieee.org/biomedical/bionics/augmented-reality-in-a-contact-lens/eyesb1. Permission granted under the terms of the GNU Free Documentation License, Version 1.2.)

Full size image

Within our own group, the smart phone changed the way we performed telemedicine [6]. With the advent of Google Glass, we were able to further integrate telemedicine into our practice.

Intraoperative Case Example

As a surgical limb salvage group, we consulted with another surgical colleague and discussed requirements for resection and subsequent admixture of an antimicrobial bioactive implant to deliver into a previously infected bony defect following a combined vascular-soft tissue reconstructive limb salvage procedure (Fig. 6.6a–d). The Google Hangouts™ application was managed entirely by the operating surgeon using hands-free voice control. Additionally, real time diagrams and MRI measurements were developed as a picture in picture, permitting two colleagues to effectively consult without the surgeon’s eyes leaving the operative field to view intraoperative imaging along with surgical anatomy.

Fig. 6.6
figure 6

(a–d) Visible defect on main screen with consultant clinician in lower right (a), real-time photograph of admixture of antibiotic-demineralized bone matrix (b) and measured delivery into defect (c and d). (a–d: Used with permission from Armstrong DG, Rankin TM, Giovinco NA, Mills JL, Matsuoka Y. A Heads-Up Display for Diabetic Limb Salvage Surgery: A View Through the Google Looking Glass. J Diabetes Sci Technol September 2014; 8: 951–956.)

Full size image

A consultation for the above patient was continued in clinic the next day. The clinician performed a dressing change along with the virtual consultant. The consultant was able to be virtually present for the first postoperative dressing change in the clinic, satisfactorily managing postoperative care.

Intraoperative Education

In a different patient at high risk for wound complications, we utilized Google Glass as an educational adjunct. A junior resident donned Google Glass during a scheduled delayed primary closure of a plantar defect and was engaged in an interactive “screen share” feature, which fed detailed descriptions on retention suture technique. This allowed for real-time, visual instruction in collaboration with a senior attending surgeon (Fig. 6.7a, b). This approach maximized hands on experience and autonomy for the resident. This was performed using bandwidth from a standard 3G CDMA connection using Glass-Bluetooth-Phone. A similar procedure followed demonstrating compartmental anatomy and assisting in planning a surgical decompression of a limb-threatening infection (Fig. 6.8a, b). Similarly, investigators have shown that AR education can shorten the learning curve during the acquisition of laparoscopic skills [13].

Fig. 6.7
figure 7

(a and b) During a delayed primary closure of a high-risk plantar wound, real-time descriptions with instant “screen share” were fed through Glass to a junior resident during the surgical procedure to assist instruction by the senior attending surgeon. (a and b: Used with permission from Armstrong DG, Rankin TM, Giovinco NA, Mills JL, Matsuoka Y. A Heads-Up Display for Diabetic Limb Salvage Surgery: A View Through the Google Looking Glass. J Diabetes Sci Technol September 2014; 8: 951–956.)

Full size image
Fig. 6.8
figure 8

(a and b) View during Intraoperative consultation for plantar deep space infection to assist in incision planning, exploration, and decompression using senior author’s manuscript’s figures as case example. (a and b: Used with permission from Armstrong DG, Rankin TM, Giovinco NA, Mills JL, Matsuoka Y. A Heads-Up Display for Diabetic Limb Salvage Surgery: A View Through the Google Looking Glass. J Diabetes Sci Technol September 2014; 8: 951–956.)

Full size image

The above example is merely one of many ways that physicians are beginning to scratch the surface of what can be made possible.

Current Work

Surgeons are constantly faced with the task of mentally integrating two-dimensional radiographs and the three-dimensional surgical field, which is the very reason that augmented reality is so attractive. Researchers have been attempting to overcome this complexity for the last two decades [14].

Trauma

Trauma is the leading cause of mortality in patients aged 1–44 in the USA (CDC.gov). The ability to save lives is predicated upon a well-established system of responding to trauma patients and transporting them for definitive treatment. First responders are constantly faced with navigating unknown neighborhoods and homes in order to rescue injured patients. AR navigation could not only facilitate transportation to the distress call but also has been shown by researchers in France to make navigation within low-visibility environments, such as fires, safer [15]. This technology could be further modified to make the cellular signal of the 911 caller a beacon in order to accelerate the localization of the patient.

Orthopedic Surgery

Investigators have also been able to successfully apply AR to orthopedic trauma surgery by integrating the live feed/fluoroscopic radiographs onto the patient in a geometrically appropriate fashion so that the surgeon can visualize the underlying osseous structures during surgical intervention and manipulation (Fig. 6.9) [16]. Trauma surgeons have also utilized hybrid navigation [17].

Fig. 6.9
figure 9

The correctly oriented radiograph from a C-arm is displayed over the surgical field. (Used with permission from Weidert, S., L. Wang, A. von der Heide, et al., [Intraoperative augmented reality visualization. Current state of development and initial experiences with the CamC]. Unfallchirurg 2012; 115(3): 209–13.)

Full size image

Maxilofacial surgeons in Vienna, Austria, began applying this technology in the late 1990s in order to correct posttraumatic deformities of the zygomatic arch [18]. The use of patient specific CT imaging presented in a head mounted display allowed for accurate reconstruction of complex facial fractures, while minimizing the number of incisions typically required to do so.

Otolaryngology

Otolaryngologists from the University of Toronto have evaluated the effect of AR on simulated intraoperative performance during an endoscopic approach to the skull base [19]. Though the AR was not based upon an HMD, the two experimental groups were provided either a full AR experience with critical structures visually highlighted or a standard display with an AR submonitor where the same information could be obtained but was not fully integrated into the primary display. Fifty otolaryngologists performed the task and they found that there was no significant difference in the speed or precision with which the task was completed. However, subjects were not informed that there would be a foreign body within the surgical field during the task. The group provided with full AR displayed inattentional blindness and did not identify the foreign body 68 % of the time vs. 40 % of standard display users.

Hepatobiliary Surgery

Trials have already been underway since 2009 investigating the utility of AR for liver resection. Seven hundred sixty nine patients have been modeled, leading to 50 operations. The authors believe that this technology may have increased the surgical eligibility of their patients. However, the authors comment that the issue in open surgery is deformation of organs. Other investigators have used a video see-through display whereby a short rigid stereoscope is used in to capture the operative field and then preoperative CT images are integrated on the 2D or 3D monitor (Fig. 6.10) [20]. The solution they found to organ deformation was the ability to quickly register defined points on the organ by using an infrared marker to co-register fiducial points. They used both soft tissue landmarks as well as blood vessels and found no significant difference in the accuracy of the two methods. These methods had a mean error of 6–10 mm, which is acceptable for open abdominal surgery, though it is not sensitive enough for neurosurgical applications (Fig. 6.11). The increasing availability of the hybrid OR may also be a solution to this problem as the patient could be reimaged once the surgical field was opened and retractors placed [21]. The surgical demands of a head mounted display are similar in some respects to those of the soldier, wherein the surgeon requires a simple design, with low head supported weight and low cost [22].

Fig. 6.10
figure 10

Surgeons take advantage of the 3D, real time images captured by a stereoscope, which is then further augmented by the overlay of patient specific CT data upon the surgical field during a pancreatic dissection. (Used with permission from Onda, S., T. Okamoto, M. Kanehira, et al. Short rigid scope and stereo-scope designed specifically for open abdominal navigation surgery: clinical application for hepatobiliary and pancreatic surgery. J Hepatobiliary Pancreat Sci, 2013; 20(4): 448–53.)

Full size image
Fig. 6.11
figure 11

Augmented reality created by the overlay of patient specific anatomy obtained from CT imaging and real-time, 3D imaging of the surgical field seen from a stereoscope. (Used with permission from Onda, S., T. Okamoto, M. Kanehira, et al. Short rigid scope and stereo-scope designed specifically for open abdominal navigation surgery: clinical application for hepatobiliary and pancreatic surgery. J Hepatobiliary Pancreat Sci 2013; 20(4):448–53.)

Full size image

Neurosurgery

Neurosurgeons at the University of Florida have applied mixed reality or augmented reality in a slightly different way. Two hundred and sixty residents performed a ventriculostomy on a mixed reality model (real simulated skin/3D printed skull, but virtual brain) [23]. In this trial, augmented reality was not meant to guide the subject, but to evaluate the subject after the completion of the procedure. In this way, potential trainees have the advantage of very real physical feedback, and the detailed performance feedback typical of a virtual system. Junior and senior residents outperformed interns as would be expected, but, more importantly, this mixed simulation was able to distinguish levels of training. As a validated model, this reflects the potential of mixed reality simulators in order to educate our residents who are faced with the responsibility of an ever-growing body of knowledge and fewer hours in which to master that knowledge.

The Future and Potential of AR

AR has the potential to increase surgical precision, increase patient safety, and facilitate physician education, but researchers have found limitations of the technology as it currently exists. During intracranial AVM surgery, AR was not as useful due to the complexity of AVMs; however, the authors felt that real-time information concerning hemodynamics would be more useful [24]. As investigators continue to push the envelope of this technology, its utility will become more defined.

Investigators have also looked at integrating other modalities separate from axial imaging. The use of an infrared camera allowed for the real-time visual display of cardiac ischemia in a pig model [25]. This could have direct implications during cardiac surgery or microvascular free tissue transfer. Depending on the sensitivity of the camera, this might even provide critical care specialists with real-time data concerning distribution of flow to myocardium or even pulmonary tissue. Theoretically, such technology could allow for the rapid diagnosis of pulmonary emboli or threatened limb ischemia. The military has also explored the use of ultra-wide band radars and stepped frequency continuous wave radars in order to detect the human heart beat at very long distances [26]. This is an additional technology that is being explored for noninvasive medical applications and could be used for evaluating real time, functional information.

While these procedures already are based largely on various imaging modalities—i.e., fluoroscopy, CT, or MRI—the ability to overlay for the operator additional information in the image field may allow for more meaningful interpretation of a given situation. For example, while focused on performing an interventional procedure and seeing a third dimension via an intraluminal ultrasound image that is now overlayed on hemodynamic data, ones clinical decision-making may be enhanced. As outlined for surgical procedures, the ability to telemeter or broadcast this information to fellows, residents, students or other health care personnel may further enhance the efficacy of a given procedure and will serve as an augmented teaching modality.

The areas that AR appears to be most useful in are minimally invasive procedures where the tumor and or pathology are not palpable. This seems to be most obvious in the brain and skull base operations, arthroscopic procedures, and procedures within solid organs and the retroperitoneum.

A Word of Caution

It would be prudent to suggest caution in the use of the standard approach to research and development for AR technologies for medical applications, as we have seen high costs and slow progress with this approach. Though it may not integrate into our current systems readily, it will be of utmost importance to allow for a developer based ecosystem to drive the development of new software for these devices as this will result in lower costs and ultimately more rapid evolution.

Surgeons have been employing augmented reality in medicine and surgery for the last 20 years [27]. The key to successful introduction of new technologies is to seamlessly integrate them with existing technologies. The ability of oncologists to precisely eradicate a tumor using stereotactics, high-definition imaging, and focused radiation has had a significant impact on patient safety and survival [28]. Many new technologies suffer from high cost and size, and, therefore, widespread application is not possible. Until recently, augmented reality had neither the correct form factor to make intraoperative and in-clinic use practical, nor the efficacy to justify its application [29, 30].

In addition to size and cost, accessibility is key to the proliferation of a new technology. The cost of development for a commercially available technology is spread over an enormous consumer base. Most new medical technologies are slow to develop due to a relatively small consumer base—resulting in high costs—and proprietary restrictions. Google Glass, for example, is part of an app-based developer ecosystem, which greatly accelerates the maturation process as seen by mobile applications and the smart phone. Just as with smart phones a decade ago, the application ecosystem for Glass is rather small. However, the form factor and subsequent iterations of it allow for arguably enormous growth.

Summary

Although many of the issues concerning AR in surgery and medicine overall need further investigation, AR is here to stay, and surgeons should champion this technology in their quest to improve patient care. Despite these issues, it is clear this technology has the potential to enhance communication. Smart glasses and other forms of AR may increase individual physician efficiency, especially for surgeons, so that more efficient patient care can be provided. Surgeons now have the ability to provide or receive more meaningful consultation from their colleagues without having to leave the operating room, office, or clinic. Even intraoperative team communication could be improved. Each member of the operative team could be outfitted with smart glasses. The surgeon has heads-up real-time patient vitals, information and imaging. The anesthesiologist is able to administer critical medications, without ever losing sight of the operative field or communication with the surgeon. The surgical resident and medical students always have an ideal view of the operative field, which may help facilitate instruction. The operating room (OR) nurse knows exactly what tools the surgeon needs, even while outside of the OR: communication is never lost.

Concerning patient safety, many parallels might be drawn between the commercial airline industry and medicine. Surgical theaters have adopted checklists similar to pilots prior to take off, in order to make the journey safer. Smart glasses may even serve as an early “black box” in medicine. If every action of a physician, surgeon, and nurse is recorded, it could impact litigation, but it could also provide critical insight into surgical decision-making, technique and safety. Furthermore, anatomic “no fly zones” [31, 32] could be projected, like a road map, onto a head mounted display, which could reduce the risk of intraoperative complications. Immediate future works from our group include evaluating use of these “no fly zones” as well as merging with image overlay/anatomic registry to use techniques like indocyanine green angiography to identify tissue viability in real time [33, 34].

While the above points seem laudable, we must consider that the quest for increased efficiency could paradoxically result in decreased quality of care. While most people assume these devices are intended to increase the amount of information perceived, Steve Mann has also addressed the utility of degrading visual stimuli presented to the user. The low-tech version of this idea is sun glasses or a welding helmet that degrade the intensity of light presented in order to protect the user and improve visual acuity. This may be of utility to surgeons who use lasers in their practice.

Other industries have investigated the concept of cognitive blindness or in-attentional blindness [10]. Some authors concluded that, intraoperatively, similar objects seen during a challenging task are likely to not be differentiated. Given the difficulty or perceived difficulty of a surgical operation, particularly during surgical training, surgeons would be a high-risk population for this effect. Even early simulation conducted by NASA revealed a risk of in-attentional blindness during the use of head-up displays while landing aircraft [35].

We are ultimately now addressing the possibilities associated with implementing these systems. The application of AR is user- and application-specific [22]. It is unlikely that there will be one perfect system or technology that will suit every surgical specialty.

There are a number of issues that have not been addressed adequately. For example, we have not yet investigated how the additional stimulus of augmented reality impacts performance or whether it might prove too distracting [19]. We ultimately need to identify if AR positively impacts patients’ outcomes. We look forward to further work in this area that might seek the requisite balance. We believe that the future in this area is promising. Enhancing communication, facilitating education, increasing safety and improving care are not virtual goals, but are rather virtuous and ultimately realistic.