Introduction

The use of image guidance as an adjunct for surgical procedures has been an area of significant interest in the surgical community, with research pursuing application of several modalities as potential venues for improvement in overall surgical outcomes and training of complex surgical procedures. Two-dimensional (2D) fluoroscopic and three-dimensional (3D) computed tomography (CT) renderings have been employed to guide operative approach in the preoperative setting [1,2,3]. Recently, 3D printing has also been used to aid in preoperative planning of complex procedures [3,4,5,6]. Stereotactic navigation using intraoperative 3D CT imaging has been used to provide real-time guidance during challenging operations [7, 8]. Augmented reality (AR) has been studied and applied in multiple surgical specialties, including interventional radiology [9, 10], orthopedic surgery [11,12,13], neurosurgery [14, 15], general surgery [16, 17], ophthalmology [18], otolaryngology [19, 20], urology [21], and plastic surgery [22]. AR has also been utilized to improve intraoperative performance of difficult procedures [23,24,25,26], and in surgical simulations [27, 28]. Previous efforts in preoperative planning, stereotactic navigation, and AR have traditionally been displayed on a monitor that requires the operator to shift his field of vision away from the surgical field.

Head-mounted displays (HMDs) have recently been utilized for the display of images in virtual reality (VR), completely immersing the user in a virtual environment, and obscuring the wearer’s real-world field of view [29]. Head-mounted display augmented reality (HMD-AR) technology further enables the projection of virtual 2D or 3D images into the field of vision of the user. The Microsoft HoloLens is a highly advanced HMD-AR headset in terms of contrast perception, task load, and frame rate [30]. The HoloLens differs from VR by its ability to project holographic images into the field of view of the wearer, rather than completely immersing the wearer in a digital environment. Even in its infancy, HoloLens applications in the medical field have been an area of active research. The HoloLens has been employed to visualize tumor markers using near-infrared fluorescence [31]. The HoloLens was also used in conjunction with an electromagnetic marking system to guide catheter navigation in a vascular tree phantom without the use of real-time fluoroscopy [32]. In both of these studies, however, the focus was on visualization and tracking, rather than percutaneous instrumentation.

A promising area of research for surgical applications of AR is in developing minimally invasive surgical approaches to the spinal column [13]. Surgical cannulation of the vertebral pedicle is an invasive and difficult step in many procedures, including pedicle screw placement [33] and vertebroplasty [34]. The identification of entry points and surgical trajectory is of vital importance in properly performing these procedures. Several investigators continue to focus on developing more reliable and accurate techniques for screw placement [35]. We studied the use of HMD-AR CT image guidance in placing pedicle screw placement guides in an opaque lumbar model with no real-time fluoroscopic guidance or electromagnetic tracking technology. We also report measurement of the accuracy of registration of AR images to the model upon which they were superimposed. This technology is unique because it allows transposition of all underlying anatomical structures visible on CT onto the real structure from which the image was generated, permitting the operator to look without interruption directly at the surgical field and patient. We selected a percutaneous approach for the simulation to ensure that hardware placement was accomplished with reliance on AR, rather than direct visualization of the pedicle. To our knowledge, this is the first peer-reviewed report and evaluation reporting HMD-AR CT guidance technology for pedicle screw placement.

Materials and methods

Lumbar phantom

A lumbar model was created by encasing articulated L1, L2, and L3 human replica Sawbones (Vashon Island, WA, USA) vertebrae in opaque Smooth-On Inc (Macungie, PA, USA) Ecoflex 00-30 super soft platinum silicone material. The vertebrae were anatomically aligned and encased such that no part of the bone was visible through the opaque silicone. The surface of the phantom was smooth and the cross section had a hexagonal shape, which was the shape of the container used as the mold.

CT scanner

CT imaging was obtained of the model using a General Electric (Boston, MA, USA) Light Speed VCT 64-slice scanner. The tube voltage was set at 140 kV and the current 100 mA. The model was scanned with 5-mm-thick scans and reconstructive images were every 2.5 mm to enhance detail and limit noise. The CT scanner was calibrated each day with standard phantoms. Water was set to a density of 0 according to the Hounsfield Scale.

Augmented reality

AR superimposes virtual images into a person’s field of view, creating a composite of virtual and real surroundings. The Microsoft (Redmond, WA, USA) HoloLens AR headset was used in this experiment. This device uses a transparent lens for each eye, with a system of lights and mirrors combined with processor technology to blend virtual images and normal vision. The virtual image is produced by projecting two sets of beams, an object beam and a reference beam, to create a sculptural casting of the light waves on real objects and superimpose virtual images in proper dimension and space to allow the user 3D visualization of holograms in a real environment [36]. The HoloLens is equipped with its own system of sensor cameras, which allow it to maintain its position in 3D space while in a room. The HoloLens automatically maps the environment using these sensor cameras, thereby allowing holographic images to be fixed in a set 3D spatial position regardless of how the user moves around the room [31]. The HoloLens interface can be manipulated using hand gestures and voice commands, freeing the user’s hands when using the HoloLens [37]. The technology includes multiple environmental understanding cameras on the front and sides for tracking surroundings. This technology allows a processed image to be projected in an appropriate field of vision in which both virtual images and real objects can be visualized in 3D space.

The Novarad (American Fork, UT, USA) OpenSight application downloaded onto the HoloLens was used to integrate the CT data with AR. In brief, this software works by rebundling CT or Magnetic Resonance Imaging (MRI) Digital Imaging and Communications in Medicine (DICOM) images into a proprietary file format allowing full integration of the images as one complete 3D volume, instead of individual images. This complete volume rendering is then projected through the HoloLens projection system to allow the user visualization of the CT data as a recreated 3D image. A component of the software affords the user the ability to use the HoloLens voice and hand motion recognition systems to operate the software. The recreated image can be moved in 3D space and either automatically registered to the object of interest by using an algorithm to align the virtual surface area with the real surface area or it can be manually registered. This object can then be fixated in 3D space, allowing the user to view the image in a fixed location even when walking around the image. The software includes interface modes to allow the cross-sectional images of the 3D volume to be viewed in axial, coronal, sagittal, and oblique views. The window level can also be adjusted allowing different densities of material or tissue to be visualized according to the Hounsfield scale. For example, low-density tissue like soft tissue can be dimmed and high-density tissue like bone can be highlighted to allow contrasting visualization of underlying anatomical structures.

Initial registration accuracy verification

To verify proper initial registration of the hologram and the model, a single user placed six 22-gauge cannulated spinal needles into the phantom roughly at the pedicles, but without the guidance of holographic trajectories or imaging. A CT scan was then performed of the phantom with the spinal needles in place. Bright green lines were drawn to resemble virtual trajectories following the exact 3D path of the needles placed using the Novarad Picture Archiving and Communication System (PACS). These trajectories with the CT data were then superimposed holographically in real space over the phantom and registered to the phantom using the OpenSight application on the HoloLens. The images were aligned using an automatic alignment feature of the software to align the virtual CT data reconstructed image to the phantom in 360°. The registration of the virtual image to the phantom was then fine-tuned by manually aligning the virtual image to verify that the object was in the correct position. Accuracy of registration was then assessed by comparing the deviation of the holographic image from the phantom, and a ruler was used to measure the deviation of real needle from the virtual needle at the point of contact with the phantom, measured in 45° increments walking around the model at a radius of approximately 2 feet away from the model in 360° for a total of eight measurements for each needle (Fig. 1).

Fig. 1
figure 1

Measuring the deviation of real needle from virtual trajectory (green line) at point of entry into phantom

Full size image

Preprocedural planning

A CT was performed of the phantom with no needles in place. CT images of the phantom were processed using the Novarad PACS system. Using the PACS to annotate, bright green trajectory lines of appropriate angle and depth were drawn through the center of the pedicle and projecting out of the phantom to allow the preprocedural trajectories to be seen outside of the phantom during the operative phase (Fig. 2). Trajectories for L1 were aligned at 5°, L2 at 10°, and L3 at 15° in the mediolateral direction corresponding to the natural relative variation of pedicle alignment [38, 39]. All preprocedural trajectories were aligned at 0° in the craniocaudal direction. Additionally, small red spheres were drawn at the point of contact of the phantom along the line of the trajectory to aid the user in localizing the proper point of needle contact.

Fig. 2
figure 2

CT imaging of the lumbar phantom using Novarad processing software with preprocedural trajectories (shown as green lines) included. The following views are shown: axial (top left) for both L2 pedicles, coronal (top right) for all six pedicles, sagittal (bottom left) for right L2 pedicle at 10° mediolaterally, and 3D rendering showing all six preprocedural trajectories

Full size image

Percutaneous placement

The combined CT data and trajectories described from preprocedural planning were then integrated using the OpenSight application on the HoloLens to allow visualization of the virtual model, as well as the corresponding trajectories which were then registered onto the real phantom using the OpenSight application. Alignment of the hologram to the model was achieved using the automatic registration feature which follows an algorithm that outlines surface areas, stores them into memory and uses the HoloLens’ sensor cameras to find similar patterns in the environment to snap the holographic image in 3D into the same location as the model. Once grossly aligned, a manual registration interface using hand gestures was used to transpose and rotate the hologram further as needed to further fine-tune the alignment if the automatic registration was not ideal. To maintain spatial accuracy, the OpenSight application has a software element called a spatial anchor which is used to maintain the spatial accuracy of the hologram while the user moves throughout the room.

Once aligned, concurrent visualization of underlying bony structures, the trajectory guides, and the suggested entry point on top of the model was visible. The image format could be changed to show individual 2D slices, which could be scrolled through using hand motions or voice commands to optimize stereotactic and ergonomic performance according to user preference. Needles representing pedicle screw trajectories were then placed using 22-gauge cannulated spinal needles, inserted along the projected holographic trajectories to the point of contact with bone (Fig. 3). A total of 24 needles were placed by 2 medical students, 6 needles by a neuroradiologist, and 6 needles by an orthopedic surgeon, for a total of 36 needles placed. A CT scan of the model was obtained after each trial, and trajectories of the needles were extrapolated to the point of final screw placement within the bone using the Novarad PACS system (Fig. 4).

Fig. 3
figure 3

Images showing the model as well as interactive views using the HMD-AR display to visualize inner structures of the model. a Model before needle placement; b model after needle placement; c HMD-AR of axial views superimposed on model with no needles placed; d HMD-AR of axial views superimposed on model with needles placed; e HMD-AR 3D rendering of model with no needles placed; and f HMD-AR 3D rendering of model with needles placed

Full size image
Fig. 4
figure 4

CT imaging of the lumbar phantom using Novarad processing software with needle placement and measurements performed for the right L2 pedicle. The following views are shown: axial (top left) measuring a total width of 11.20 mm with the needle 5.29 mm from the closest edge in the mediolateral direction, sagittal (bottom left) for the right L2 pedicle measuring a total pedicle height of 13.09 mm with the needle 5.95 mm from the closest edge in the craniocaudal direction, coronal (top right) showing both mediolateral and craniocaudal views, and a 3D rendering showing needle trajectories and measurements

Full size image

Results

Registration accuracy

Assessment of registration accuracy was conducted for each pedicle as described in “Materials and methods.” The needles placed were distributed around the holographic trajectory guide in a roughly circular deviation with average radius of 2.5 mm with standard deviation (SD) of 0.44 mm.

Percutaneous placement

The average diameter of the pedicles in the mediolateral direction was 12.99 mm (range 12.31–13.61 mm) and in the craniocaudal direction was 15.59 mm (range 12.02–18.69 mm) with L1 pedicles being the smallest diameter and L3 pedicles being the largest.

It took an average of 200 s (s) (SD 96.69 s; range 67–464 s) to place one needle to the point of contact with the spinal pedicle, and an average of 1.36 attempts for each needle. An attempt was defined as complete removal of the needle to select a new entry point after having advanced the needle into the silicone.

Extrapolation of needle trajectory into the pedicle showed that of 36 needles placed, 35 (97%) would have remained within the pedicle. In surgeries where pedicle screws are utilized, pedicle screw size typically varies between 4 and 7 mm in diameter [40, 41]. With pedicle screw diameters of 4, 6, and 7 mm, 35 (97%), 32 (89%), and 31 (86%), respectively, would have remained entirely within the pedicle (Table 1). Outcomes were universally worse on placement of needles in the right L3 pedicle compared with the rest of the pedicles. The right L3 needle trajectory showed 83% of needles placed compared to 100% placement of the rest of the needles. At pedicle screw diameters of 4, 6, and 7 mm, the right L3 needle showed a success rate of 83, 83, and 67% compared to 100, 97, and 93% of the rest of the pedicles, respectively (Table 2).

Table 1 Percentage of needle placement attempts which would have been successful (n = 36)
Full size table
Table 2 Percentage of needle placement attempts which would have been successful comparing right L3 pedicle (n = 6) with the rest of the pedicles in the phantom (n = 30)
Full size table

Several measurements were taken to determine the proximity of the needle to the edges of the cortical portion of the pedicle (Fig. 5). The distance from the needle to the nearest mediolateral border of the pedicle had a mean of 4.69 mm (median 5.05 mm, SD 1.44 mm, 95% CI 4.22–5.16 mm). The distance from the needle to the nearest craniocaudal border of the pedicle where the neural foramen lies had a mean of 4.48 mm (median 5.84 mm, SD 1.69 mm, 95% CI 4.93–6.03 mm) (Table 3). The single failed attempt was past the edge of the cortical portion of the pedicle and was recorded as − 0.95 mm in the mediolateral direction and − 1.41 mm in the craniocaudal direction. The deviation of needle placement from the center of the pedicle was recorded. The mediolateral deviation from center had a mean of 1.39 mm (median of 0.94 mm, SD 1.35 mm, 95% CI 1.19–2.00 mm). The craniocaudal deviation from center had a mean of 1.60 mm (median 1.30 mm, SD 1.23 mm, 95% CI 1.19–2.00 mm) (Table 3).

Fig. 5
figure 5

Schematic of a vertebra with cross section through the pedicle, demonstrating the ideal centerline and the pedicle cannulation guide (spinal needle) placement. [a = closest mediolateral distance of spinal needle to edge, b = closest craniocaudal distance of spinal needle to edge, x = ½ of horizontal diameter of pedicle, y = ½ of vertical diameter of pedicle]

Full size image
Table 3 Characteristics of attempts for placement of needles (n = 36)
Full size table

The 2D spatial deviation from preoperative planning was defined as the distance the needle deviated from the preoperative trajectories created on the Novarad PACS software rather than the center of the pedicle. This measurement was taken at the midsection of the pedicle. Mediolateral deviation from pre-op planning had a mean of 1.30 mm (median 0.91 mm, SD 1.22 mm, 95% CI 0.90–1.70 mm). The craniocaudal deviation from pre-op planning had a mean of 1.53 mm (median 1.19 mm, SD 1.22 mm, 95% CI 1.13–1.93 mm) (Table 3). Needle placement of L3 on the right consistently had the highest deviation from pre-op planning. Among all users, L3 on the right was subjectively perceived as the most difficult in regarding user ease of placement (mean of 1.67 compared to overall mean of 2.17). This same pedicle also had the lowest rate of success of potential screw placement with 83% (97% overall) success at 0-mm and 67% (86% overall) success using 7-mm pedicle screw.

The angular deviation was measured using the Novarad PACS system in both the mediolateral and craniocaudal directions. The degree of deviation in the mediolateral direction had a mean of 2.36° (median 1.97°, SD 2.01°, 95% CI 1.170°–3.02°). The degree of deviation in the craniocaudal direction had a mean of 1.82° (median 1.79°, SD 1.08°, 95% CI 1.47°–2.18°).

Discussion

Image guidance procedures are uniquely suited to spine surgical interventions, given the complex anatomy of the vertebral column, devastating consequences of neural and vascular injury, and issues pertaining to radiation exposure for the patient and operating room personnel. Radiation exposure from intraoperative CT O-arm use is considerably higher than fluoroscopy to the patient. While the surgeon and operating room personnel can step away as the scan is being carried out, each scan using a CT O-arm integrated with Stealth navigation is estimated to result in nearly three times the radiation exposure to a patient as the use of conventional fluoroscopy during pedicle screw placement [42]. Another study reported an average of 75.8 s of fluoroscopy radiation exposure time to guide pedicle screw procedures ranging from 2 to 8 total screw placements [43]. A recent study on the use of an AR system to place spinal pedicle screws in cadavers, relying on preprocedural imaging with no intraoperative fluoroscopy, found potential for overall reduction in radiation exposure [12]. Because CT or MRI images are typically obtained prior to spinal surgery, implementing HMD-AR with preprocedural images could eliminate the need for intraoperative O-arm, significantly reducing radiation exposure, and potentially improve surgical accuracy of pedicle cannulation.

An important aspect of adopting new surgical technology is the ability of the technology to translate rapidly to surgeons at different levels of experience and skill. Needle placement was performed by two medical students, a neuroradiologist, and an orthopedic spine surgeon. Operators that performed our approach to the spinal pedicle had minimal prior exposure to HMD-AR. Each operator was given a brief tutorial on the operation of the HMD-AR system, and the procedure that would be performed, lasting about 10 min immediately prior to performing the procedure. The decision to have medical students and a neuroradiologist place the majority of the needles was designed to encourage reliance on the holographic trajectories, with the goal being assessment of the ease of use and precision of the technology, rather than an assessment of operative skill.

The time needed to place a needle varied between individuals, but took an average of 200 s per pedicle. When compared to data published on pedicle screw placement using percutaneous approach without HMD-AR, some users reported an average of 6.92 min for each screw placement [43]. While our simulation did not involve penetration of the pedicle itself, the most arduous and hazardous steps in pedicle screw placement are ensuring the appropriate entry point into the pedicle and proper trajectory within the pedicle [44, 45]. Because the primary objective of this study was to evaluate viability of superimposed holographic guidance with respect to trajectory, and did not include actual placement of pedicle screws, the times presented here are not directly comparable to the time that may be necessary to align and place a pedicle screw. These times do suggest that the system is intuitive to operate and that operators are able to commit to a trajectory in a period of time that would be reasonable for performing actual placement of a pedicle screw. Preparation of CT images for use in a surgical setting is a quick and straightforward process. With current technology, addition of intraoperative trajectories and subsequent upload of images to the HoloLens takes < 30 min, with planning of trajectories accounting for the majority of this time. However, this is a process that could be partially automated in future updates and could be done in a fraction of this time. While not formally measured, it typically took between 2 and 5 min to align the hologram onto the model.

HMD-AR has potential advantages over 2D screen-based AR in the surgical field. Because the device can be operated solely via voice commands or hand gestures, the surgeon’s hands can remain free and sterile throughout the operation. Additionally, because HMD-AR provides 3D visualization capability in the operating field itself, there is potential to improve overall ergonomics for the surgeon during the operation when compared to AR systems using separate monitors.

The measured variability of hologram in space as the wearer of the HoloLens moved in a circle around the model had a roughly circular radius of approximately 2.5 mm, indicating that the image deviates slightly in space as the headset moves. This represents the maximum error that could be attributed to instability of the hologram. However, the deviation from the preplanned holographic path to trajectory of the actual needle was 1.3–1.53 mm. This was less than the radius of the deviation attributed by the HoloLens. This indicates that operators of the HMD-AR system were able to mitigate this inherent holographic instability, and was likely accomplished by viewing the hologram from different angles. The degree of holographic deviation should be improved before this technology is adopted as a guidance system for pedicle screw placement. Nonetheless, it is promising that users in this simulation were able to place needles within the margin of error of the hologram, especially considering likely future technologic advancement of HMD-AR.

A high rate of needle deviation in the right L3 pedicle was seen and was likely due to error in the planning phase while creating the holographic trajectory. This included one complete miss on the right L3 pedicle in which the needle was outside of the pedicle by 0.95 mm mediolaterally and 1.41 mm craniocaudally. Preprocedural trajectories were drawn for L3 at 15° from the midline based on previously published data [46]. However, when visually inspecting the preprocedural trajectory, it is apparent that a different trajectory, specifically with a smaller deviation from midline, would have allowed a more accurate guide to safely traverse the right L3 pedicle. This highlights the fact that pedicle orientation can be variable and patient specific, thereby demonstrating the benefit of dedicated preoperative planning to help guide pedicle trajectory. To avoid potential error in the planning phase, it would be best to visually confirm the ideal trajectory on multiple planes in CT.

There were a few weaknesses of the lumbar spine phantom that led to suboptimal simulation. The phantom was created to encase and obscure the vertebrae but not necessarily to fully resemble the human back. In addition, the model was composed completely of silicone material, which, although designed to have similar properties to soft tissue, has more resistance than human tissue to advancement of a metal instrument, and is less conducive to changing direction after insertion. In vivo, the varying densities of soft tissue of the back allow small changes in direction by adjusting the bevel of the needle when approaching the pedicle, which was not possible or quite challenging in our phantom. It is possible that repeating the process reported here on a more realistic model, cadaver, or live patient, could lead to improvement in entry point identification and modification of trajectory, potentially increasing the overall accuracy of the procedure. Additionally, penetration of the bone has potential to compromise needle trajectory as bony irregularities may slightly alter the established trajectory. Nonetheless, the purpose of this project was to establish whether CT images could be projected over a model using HMD-AR in a manner that allows a user to follow trajectories with an instrument. Further validation in other more realistic models will be necessary in future development of this technology with respect to its use for pedicle screw placement.

There are several limitations to be overcome before this technology can be sufficiently reliable to guide spinal cannulation. While the automatic registration feature was helpful, the hologram still required manual registration to align the hologram with the model. In addition, the CT scans were static images in a fixed 3D column and would therefore not change if a patient were in a different position. The system of registration must be enhanced, and a real-time tracking system should be integrated if this technology is to be used for pedicle cannulation in live patients. There is significant potential to improve this technology by using a form of tracking method for automatic maintenance of registration on the skin surface, which has been previously described in other technologies such as fiducial placement, but there exist potential sources of error even with these automated tracking technologies [47]. Additionally, an inherent part of surgery is disruption and manipulation of tissues. As tissue disruption occurs, the patient’s anatomy will begin to vary from a preoperative image. However, this is less likely to affect bony structures such as vertebrae, which is one reason why they are an attractive target for augmented reality guidance. A number of tissue deformation algorithms and tissue adaptation programs have been developed with the intent of allowing a preoperative image to adapt intraoperatively, but further research is needed before these ideas become practical [48, 49]. An additional consideration related to the HoloLens is that the sensor cameras detect images based on surrounding light reflections, meaning the sensors will be unable to maintain 3D spatial positioning in dark rooms [31]. While most surgeries are performed in bright light, this limitation is worth mentioning for future applications.

Because of its unique ability to provide 3D visualization of underlying structures when directly viewing the operative field, HMD-AR also has potential to aid in training new residents in both simulated and live procedures. It can provide experienced surgeons with enhanced visualization of anatomy to potentially reduce difficulty and increase safety of surgery. Promising potential applications of AR augmentation are numerous and include pedicle screw placement [13] and intracranial procedures [50, 51]. Procedures requiring exacting trajectories such as percutaneous interventional radiology procedures [52] or lumbar punctures may also stand to benefit from HMD-AR technology [53]. In regard to documentation and training, images viewed by the user of the HMD-AR showed variation of the position of the hologram from recorded photos and videos, which has also been previously reported by HoloLens users in other applications [32]. While this has little effect on the performance of the operator, it represents a significant hurdle for documenting and training purposes as both photographs and videos are not displayed in the exact same manner as the operator visualizes the AR rendering.

Conclusion

We have demonstrated that HMD-AR technology projecting reconstructed 3D and 2D CT images can be accurately superimposed over a lumbar model and used to place pedicle cannulation guides that could eventually be used for placement of pedicle screws without the use of real-time fluoroscopy or intraoperative CT O-arm. While HMD-AR technology requires extensive development to become clinically useful, our results suggest that potential does exist to use HMD-AR for image-guided surgical navigation of the complex three-dimensional orientation of the lumbar pedicles. Moreover, this technology opens the door to a variety of possibilities of improved 3D visualization and surgical navigation of minimally invasive image-guided interventional procedures, medical training, and optimization of risk in technically challenging surgeries.