Abstract
We adapted and evaluated the Microsoft Kinect (touchless interface), Hillcrest Labs Loop Pointer (gyroscopic mouse), and the Apple iPad (multi-touch tablet) for intra-procedural imaging review efficacy in a simulation using MIM Software DICOM viewers. Using each device, 29 radiologists executed five basic interactions to complete the overall task of measuring an 8.1-cm hepatic lesion: scroll, window, zoom, pan, and measure. For each interaction, participants assessed the devices on a 3-point subjective scale (3 = highest usability score). The five individual scores were summed to calculate a subjective composite usability score (max 15 points). Overall task time to completion was recorded. Each user also assessed each device for its potential to jeopardize a sterile field. The composite usability scores were as follows: Kinect 9.9 (out of 15.0; SD = 2.8), Loop Pointer 12.9 (SD = 13.5), and iPad 13.5 (SD = 1.8). Mean task completion times were as follows: Kinect 156.7 s (SD = 86.5), Loop Pointer 51.5 s (SD = 30.6), and iPad 41.1 s (SD = 25.3). The mean hepatic lesion measurements were as follows: Kinect was 7.3 cm (SD = 0.9), Loop Pointer 7.8 cm (SD = 1.1), and iPad 8.2 cm (SD = 1.2). The mean deviations from true hepatic lesion measurement were as follows: Kinect 1.0 cm and for both the Loop Pointer and iPad, 0.9 cm (SD = 0.7). The Kinect had the least and iPad had the most subjective concern for compromising the sterile field. A new intra-operative imaging review interface may be near. Most surveyed foresee these devices as useful in procedures, and most do not anticipate problems with a sterile field. An ideal device would combine iPad’s usability and accuracy with the Kinect’s touchless aspect.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Background
The power of computing systems has rapidly advanced. However, the original computer mouse from the 1980s has remained the de facto method of human computer interaction. New devices have been introduced such as the Hillcrest Labs Loop Pointer (Rockville, MD), a gyroscopic mouse, the Apple iPad (Cupertino, CA), a multi-touch tablet, and the Microsoft Kinect (Seattle, WA), a touchless gesture-based input device. These new devices, respectively, free the user from a table for the mouse, from the mouse itself, and from physical contact with a device. By changing the mode of interaction, these new tools may potentially enable a more effective means of manipulating radiological images [1–9], particularly during real-time imaging.
Real-time imaging review is important in radiology, particularly during procedures. Procedures are adapted to intra-operative findings which oftentimes necessitate additional imaging evaluation or change in technique. The standard paradigm of viewing images on a computer workstation sitting atop a table with a mouse and keyboard is ill-suited for intra-procedural use. A tabletop and mouse can occupy substantial space in a limited-sized procedural room and can impede interaction required between the physician and the patient. Moreover, a handheld or hands-free device may be more convenient, freeing a hand during procedures which require re-positioning the patient or operating other devices such as the fluoroscope. Procedural difficulty can be compounded when sterile technique is required such as with endovascular or intra-abdominal procedures. Computer keyboards and mice have been shown to harbor high infectious colonization rates [10]. These new interfaces, especially the Kinect, limit physical contact, which may reduce the risk of contamination and limit transmission of infection [11].
Gesture-based systems have been described in the past [3–6, 12], but their efficacy was limited by then-existing technology. The Kinect brings a new generation of motion tracking with far greater accuracy and response time. Some have described using the iPad and a gyroscopic mouse for surgery or radiology; however, these studies do not provide a comparison across devices [13–15]. The suitability of each of these devices has not been well evaluated for the radiologist, particularly for intra-procedural use, and comparative head-to-head analysis has not been performed [11]. Using qualitative and quantitative survey data, we evaluate the comparative efficacy of each of these devices by 29 radiologists at our institution.
Methods
Interface Setup
We adapted three new computer interfaces—the Microsoft Kinect (Microsoft, Seattle, WA), Hillcrest Labs Loop Pointer (Hillcrest Labs, Rockville, MD), and the Apple iPad (Apple, Cupertino, CA)—for intra-procedural imaging review and compared their efficacy. The Kinect utilizes a camera and depth sensor to track and respond to gestures without a requiring handheld device. The Loop Pointer uses gyroscopic sensors and accelerometers to enable wireless image manipulation without a flat surface. The iPad is a tablet with a multi-touch interface to register distinct positions of inputs to manipulate images without a separate pointing device or keyboard. Figure 1a, b, and c illustrates how to operate each device.
To adapt the Kinect for use, publicly available device drivers from Primesense (Tel-Aviv, Israel) were installed onto a laptop computer with an Intel Core i7-2620M CPU (Santa Clara, CA) and 4 gigabytes of RAM, running Windows 7 Professional 64-bit (Microsoft, Redmond, WA). Custom software was written in the C# programming language, utilizing the OpenNI software (OpenNI.org) framework and skeletal tracking middle-ware module NITE (Primesense). The software enabled interpretation of motion data captured by the Kinect and conversion into mouse and keyboard commands to control the MIM Cloud DICOM viewer (Cleveland, OH). The software also included a large tool bar to enable selection of different tools such as scroll, window level, pan, zoom, and measure.
The Loop Pointer wireless receiver self-installs its own drivers when plugged into a computer’s universal serial bus (USB) and is immediately ready for use. The iPad is a second-generation iPad powered by an Apple A5 processor. The iPad was readily configured for the study by installing and loading the Mobile MIM DICOM viewer app (Cleveland, OH). The Loop Pointer and iPad were placed in plastic bags to simulate sterile covers for intra-procedural use.
The MIM Cloud Viewer was installed on the computer for use with the Kinect and Loop Pointer. MIM Software Mobile MIM was installed as an application on the iPad. Although similar, the MIM Cloud Viewer and MIM Software Mobile MIM have slightly different methods to engage each function, as each is optimized for each device. Both versions are freely available for use, and the latter was the first application approved by the FDA for viewing of DICOM images on an Apple iPad. An included video (Electronic Supplementary Material 1) demonstrates a few of the functions on all three devices.
Participants and Study Design
We recruited 29 participants consisting of radiology residents, fellows, and attendings at our institution to evaluate all three devices. An exemption from Institutional Review Board was approved. Demographic characteristics and technology experience were assessed utilizing a multiple-choice survey that each participant completed.
A routine contrast enhanced axial CT of the abdomen and pelvis consisting of 181 images and containing a single 8.1-cm hepatic lesion (as initially measured by PACS viewer measurement function using standard mouse and keyboard) was selected as a representative test case. All patient identifiers were removed from the images. Approximately 2 to 5 minutes of guided training was provided to each participant on the use of each device until each participant reported that they were ready to proceed to the simulation. For each device, each participant was asked to perform the following five common basic interactions to complete the overall task of measuring the 8.1-cm hepatic lesion: scroll, window, zoom, pan, and measure. The scroll, window, zoom, and pan interactions were used to accomplish a navigation subtask of placing the lesion in the field of view, with a subsequent measurement subtask performed using the PACS viewer measurement function. The users performed the measurement interaction by selecting the measurement function from the toolbar, manually navigating to a point on the perimeter of the lesion (long axis), clicking to place a caliper, then selecting and placing a second caliper on the opposing perimeter. The PACS viewer then reports the measurement above the resultant drawn line. For each user with each device, the authors recorded the time to complete the task and their respective measurements. Each radiologist then anonymously completed a survey of their impressions. For each of the five basic interactions, participants assessed the devices on a 3-point subjective scale, where 1 is lowest score and 3 is the highest score. Subjects were informed of the scale valence by a 3-point-labeled scale of task difficulty on the survey form (1 = difficult, 2 = moderately difficult, 3 = easy). The five individual scores were summed to calculate a subjective composite usability score with a maximum of 15 points. Each user also assessed each device for its potential to jeopardize a sterile field. The authors also identified the number of surfaces that each user touched during the simulation through real-time observation to try to gauge the risk of contamination associated with each device.
Statistics
Frequencies and percentages are used to describe participant demographics and general technology and device experiences. Frequencies and percentages or means and standard deviations (SD) are reported for basic interaction measures. A repeated-measures ANOVA was used to assess differences between basic interaction specific measures. When the repeated-measures ANOVA results indicated significance at p = 0.05, pairwise comparisons were made between the devices and reported as mean differences and associated 95 % confidence intervals (CIs) with associated p values. Because of multiple comparisons, a Bonferroni adjustment was used to define statistical significance (p = 0.017). Comparisons between video game and device experience and basic interaction time were compared with a t test. All statistical analyses were conducted using the IBM SPSS Statistic 19.0 software package (SPSS, Inc., Chicago, IL).
Results
A total of 29 radiologists participated in the study including 11 residents, 4 fellows, and 14 attendings, including 5 interventional radiology attendings. Demographic characteristic are reported in Tables 1 and 2. Technology experience is shown in Table 3. No one had previous experience with the Loop Pointer. Only 4 of those surveyed had experience with the Kinect. A total of 9 (31.0 %) of participants had some experience with an iPad or iPhone, and 12 (41.4 %) had extensive experience with these devices. Experience with technology is reported in Table 3. The majority of participants rarely or never play video games, and almost all reported being comfortable or proficient with computers.
For each device, Table 4 reports the means and standard deviations for the completion times, measurement, deviation from the true measurement, and subjective composite usability score. There was a significant difference between devices for all measures except deviation from the true measurement (p = <0.001 for time and composite score, p = 0.002 for measurement, and p = 0.920 for deviation).
The Kinect had a mean usability score of 9.9 (SD = 2.8), the Loop Pointer 12.9 (SD = 2.9), and iPad 13.5 (SD = 1.8). The differences in mean score between Loop Pointer and iPad were not statistically significant (Kinect vs. Loop Pointer, diff = 3.0, 95 % CI = 1.7, 4.4, p < 0.001; Kinect vs. iPad, diff = 3.6, 95 % CI = 2.1, 5.2, p < 0.001; and Loop Pointer vs. iPad, diff = 0.6, 95 % CI = −0.7, 1.9, p = 0.358). Specific components of the usability scores are reported in Fig. 2.
For completion times, there were significant differences between the Kinect and the other two devices. The Kinect had the longest mean time with 156.7 s (SD = 86.5), and the iPad has the shortest at 41.1 s (SD = 25.3) (Kinect vs. Loop Pointer, diff = 105.2, 95 % CI = 72.8, 137.7, p < 0.001; Kinect vs. iPad, diff = 115.7, 95 % CI = 85.5, 145.9, p < 0.001; and Loop Pointer vs. iPad, diff = 10.5, 95 % CI = −0.3, 21.2, p = 0.055). Figure 3 shows the minimum, mean, and maximum completion times with each device. Figure 4 shows the relationship of video game experience with completion times for each device. There was a significant difference in completion times by video game experience for the Loop Pointer (mean difference = 29.52; 95 % CI = 0.34, 58.70; p = 0.048), but not for the iPad or Kinect (p > 0.05 for both). Figure 5 shows the relationship of a prior experience with each respective device and completion times; the Loop Pointer was excluded since no user had prior experience with this device. Differences in completion times based on experience with either the Kinect or iPad (p > 0.05 for both) were not statistically significant. Figure 6 shows the minimum, mean, and maximum completion times on the Microsoft Kinect based on participant’s previous experience with the device.
The mean hepatic lesion measurement of the Kinect was 7.3 cm (SD = 0.9), and the iPad was 8.2 cm (SD = 1.2), which was the only significant difference identified for this measure (Kinect vs. Loop Pointer , diff = 0.5, 95 % CI = 0.1, 1.0, p = 0.033, NS with Bonferroni correction; Kinect vs. iPad, diff = 0.9, 95 % CI = 0.4, 1.4, p = 0.001; and Loop Pointer vs. iPad, diff = 0.3, 95 % CI = −0.1, 0.8, p = 0.167). The authors observed the following number of surfaces contacted by each user during the simulation with each device: (1) Kinect, none; (2) Loop Pointer, two surfaces (the device and the table upon which it rested) with a single hand; and (3) Apple iPad, three surfaces (both sides of the device and the table) using both hands, one hand cradling the device, and another interacting with it. Most thought each device could be useful in interventional radiology; 55 % for the Kinect, 59 % for the Loop Pointer, and 62 % for the iPad.
Discussion
To our knowledge, this is the first study to perform a head-to-head comparison of these devices, though prior studies analyzing the iPad and Kinect individually have reported a generally positive response towards their utility in the operative environment [11, 13, 14, 16–19]. Most users favored the usability of the iPad (mean score of 13.5) in almost every basic interaction over the Loop Pointer (12.9) and Kinect (9.9; Fig. 2).
The mean usability scores mirrored mean completion times; the iPad had fastest and Kinect slowest mean times (Fig. 7). The mean time with the iPad was almost four times lower than the Kinect (41 vs. 157 s). However, completions times widely varied (Fig. 3). Several users were faster using the Kinect than others were with the other devices. We suspected that a larger number of users with prior experience with the iPad (20 users) versus the Kinect (4) might explain the difference. Though the Kinect had the longest mean basic interaction completion time, prior studies have demonstrated reasonable moving, zooming, and windowing basic interaction completion times with the Kinect, requiring 1.4 times more duration as compared to a mouse/keyboard setup [11]. Although not reaching statistical significance, users with prior experience with the respective devices had decreased completion times (Fig. 5). Users with prior experience with the Kinect also decreased variance in completion times (Fig. 6).
The effect of additional training on performance was not evaluated, but prior experience with the devices may serve as a proxy. Training may be more effective than prior experience since it targeted. Video game experience may also serve as a proxy for training. Individuals with video game experience had decreased completion times for all devices (Fig. 4). This reached statistically significance only for the Loop Pointer. Interestingly, computer experience was not associated with decreased completion times.
Users were more accurate with the iPad. The actual liver lesion measured 8.1 cm. As shown in Table 4, the iPad’s mean measurement was closest and the Kinect farthest. However, precision was comparable across the devices. Differences in deviation from measurement were not statistically significant, and standard deviations were similar. Users tended to under-measure with the Kinect, decreasing accuracy, but still maintaining precision.
The most difficult basic interactions were those requiring fine movements such as measuring lesions and setting the exact window level (Fig. 2). However, the average difficulty level for each of the five basic interactions for almost all the devices was rated as moderate. It is also important to note that we utilized a generic image review interface across all devices, which likely shows the base level of capability of each device. The usability and accuracy of the individual interactions are likely to improve if future studies employ device-specific interfaces leveraging each device’s native abilities (e.g., an iPad app developed specifically for medical imaging review and supporting window and level operations by measuring motions on a screen).
A potential limitation of the composite usability score utilized in our study is that each basic interaction does not necessarily contribute equally in importance to the navigation and measurement subtasks. Further studies could employ an individual assessment on the navigation subtask overall instead of summing the interactions.
Most did not anticipate that the devices would have problems with preserving a sterile field; only 1 individual anticipated a problem with the Kinect; on the other hand, 7 individuals anticipated a problem for the Loop pointer and 10 for the iPad. One person thought our two-handed implementation of the Kinect could affect the sterile field, but use of two hands was a design choice that can be modified.
Assessing risk of infection complications would be difficult, probably requiring a large-scale study with follow-up on infectious complications. Instead, we relied upon subjective evaluation by experienced physicians. In addition, the authors identified the number of surfaces that each user touched during the simulation as a gauge for the risk of contamination. Both measures corresponded; namely, the Kinect has least concern and the iPad has the most concern in jeopardizing the sterile field. Notably, prior studies have employed varying techniques to allow for sterility, including sealing the unit between two pieces of Tegaderm by a sterile operator [13], use of a sterile-wrapped iPad touch-display system mounted on an operation microscope positioned just above the surgeon for neurosurgical application [14], and sterile-bagged iPad intra-operative CT image review during lung segmentectomy.
Comparison was not made to the current standards of practice, either scrubbing out of a procedure and using a mouse or verbally conveying commands to a second party. Such a comparison would depend on factors outside of the devices, such as time required to scrub out and back in or inter-communication abilities. Since nearly everyone is familiar with those scenarios, most probably used that frame of reference to evaluate the devices. Moreover, since multiple factors are involved in determining whether the devices are an improvement over existing technology, an overall subjective evaluation is probably more insightful.
Conclusion
A new intra-operative imaging review interface may be near. Most of the surveyed radiologists could foresee the utility of these devices for procedural imaging review and believe that these devices can preserve a sterile field. A new device may be driven by hospital demands to reduce infectious complication and more complicated procedures. The studied characteristics of usability, speed, accuracy, and preservation of the sterile field are probably important components for that interface. An ideal device would combine the higher usability and accuracy of the iPad with the touchless aspect of the Kinect. Implementations such as 3Gear Systems’ (San Francisco, CA) setup using two Kinects appear to substantially improve the usability and accuracy of the system while preserving the sterile field. Alternative devices like Leap Motion’s (San Francisco, CA) Leap which uses two different cameras for touch-free motion sensing may embody the ideal combination. Future research can be directed to optimizing the user interfaces, improving and standardizing the training, and studying infectious complications arising from intra-procedural use of these devices.
References
Johnson R, O’Hara K, Sellen A, Cousins C, Criminisi A: Exploring the potential for touchless interaction in image-guided interventional radiology. Paper presented at the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vancouver, 2011
Graetzel C, Fong T, Grange S, Baur C: A non-contact mouse for surgeon-computer interaction. Technol Health Care 12(3):245–257, 2004
Kipshagen T, Graw M, Tronnier V, Bonsanto M, Hofmann UG: Touch- and marker-free interaction with medical software. In: Dössel O, Schlegel W Eds. World Congress on Medical Physics and Biomedical Engineering. Springer, Berlin Heidelberg, 2009, pp 75–78. doi:10.1007/978-3-642-03906-5_21. September 7–12, 2009, Munich, Germany, vol 25/6. IFMBE Proceedings
Soutschek S, Penne J, Hornegger J, Kornhuber J: {3D Gesture-Based Scene Navigation in Medical Imaging Applications Using Time-Of-Flight Cameras}. In: Computer Society Conference on Computer Vision O (ed) {2008 I.E. Computer Society Conference on Computer Vision and Pattern Recognition}, Anchorage, AK, 2008.
Wachs J, Stern H, Edan Y, Gillam M, Feied C, Smith M, Handler J: A Real-Time Hand Gesture Interface for Medical Visualization Applications. In: Tiwari A, Roy R, Knowles J, Avineri E, Dahal K Eds. Applications of Soft Computing, vol 36. Advances in Intelligent and Soft Computing. Springer, Berlin Heidelberg, 2006, pp 153–162. doi:10.1007/978-3-540-36266-1_15
Wachs JP, Stern HI, Edan Y, Gillam M, Handler J, Feied C, Smith M: A gesture-based tool for sterile browsing of radiology images. J Am Med Inform Assoc 15(3):321–323, 2008
Padoy NHG: Gesture based surgical manipulation of a da Vinci robot using a Kinect. The John Hopkins University, Baltimore, 2011. Accessed 5/13/2013
Ruppert GCAPH, Moares TF, Ruppert GCAPH, Moares TF, Silva JV: Gesture Interface using Kinect for Medical Imaging Visualization in Surgeries. Renato Archer Center for Information Technology, Campinas, 2011. Accessed 5/13/2013
Tan JUJ, Link K: Kinect Sensor Allows Surgeons to Manipulate 3D CT Images in Midair. Wake Forest University School of Medicine, Winston-Salem, 2011. Accessed 5/13/2013
Hartmann B, Benson M, Junger A, Quinzio L, Rohrig R, Fengler B, Farber UW, Wille B, Hempelmann G: Computer keyboard and mouse as a reservoir of pathogens in an intensive care unit. J Clin Monit Comput 18(1):7–12, 2004
Ebert LC, Hatch G, Ampanozi G, Thali MJ, Ross S: You can’t touch this: touch-free navigation through radiological images. Surg Innov 19(3):301–307, 2012
Tani BS, Maia RS, von Wangenheim A: A Gesture Interface for Radiological Workstations. In: Computer-Based Medical Systems, 2007. CBMS ’07. Twentieth IEEE International Symposium on, pp 27-32. doi:10.1109/cbms.2007.6, 2007.
Murphy AD, Belcher HJA: Novel method for sterile intra-operative iPad use. J Plast Reconstr Aesthet Surg 65(3):403–404, 2012. doi:10.1016/j.bjps.2011.08.037
Soehngen E, Rahmah NN, Kakizawa Y, Horiuchi T, Fujii Y, Kiuchi T, Hongo K: Operation-microscope-mounted touch display tablet computer for intraoperative imaging visualization. World Neurosurg 77(2):381–383, 2012
Weiss DL, Siddiqui KM, Scopelliti J: Radiologist assessment of PACS user interface devices. J Am Coll Radiol 3(4):265–273, 2006
Eguchi T, Takasuna K, Kitazawa A, Fukuzawa Y, Sakaue Y, Yoshida K, Matsubara M: Three-dimensional imaging navigation during a lung segmentectomy using an iPad. Eur J Cardiothorac Surg 41(4):893–897, 2012
Volonte F, Robert JH, Ratib O, Triponez F: A lung segmentectomy performed with 3D reconstruction images available on the operating table with an iPad. Interact Cardiovasc Thorac Surg 12(6):1066–1068, 2011
Hotker AM, Pitton MB, Mildenberger P, Duber C: Speech and motion control for interventional radiology: requirements and feasibility. Int J Comput Assist Radiol Surg 13:13, 2013
Ruppert GC, Reis LO, Amorim PH, de Moraes TF, da Silva JV: Touchless gesture user interface for interactive image visualization in urological surgery. World J Urol 30(5):687–691, 2012
Conflict of Interest
No funding was provided for this project.
Author information
Authors and Affiliations
Corresponding author
Electronic Supplementary Material
Below is the link to the electronic supplementary material.
ESM 1
(WMV 6,857 kb)
Rights and permissions
About this article
Cite this article
Chao, C., Tan, J., Castillo, E.M. et al. Comparative Efficacy of New Interfaces for Intra-procedural Imaging Review: the Microsoft Kinect, Hillcrest Labs Loop Pointer, and the Apple iPad. J Digit Imaging 27, 463–469 (2014). https://doi.org/10.1007/s10278-014-9687-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10278-014-9687-y