Abstract
Artificial intelligence (AI) is gradually changing the practice of surgery with technological advancements in imaging, navigation, and robotic intervention. In this article, we review the recent successful and influential applications of AI in surgery from preoperative planning and intraoperative guidance to its integration into surgical robots. We conclude this review by summarizing the current state, emerging trends, and major challenges in the future development of AI in surgery.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Vitiello V, Lee SL, Cundy TP, Yang GZ. Emerging robotic platforms for minimally invasive surgery. IEEE Rev Biomed Eng 2013; 6:111–126
Troccaz J, Dagnino G, Yang GZ. Frontiers of medical robotics: from concept to systems to clinical translation. Annu Rev Biomed Eng 2019; 21(1): 193–218
Yang GZ. Body Sensor Networks. New York: Springer, 2014
Yang GZ. Implantable Sensors and Systems: from Theory to Practice. New York: Springer, 2018
Shortliffe E. Computer-Based Medical Consultations: MYCIN. Amsterdam: Elsevier, 2012. Vol. 2
Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (NIPS). Lake Tahoe. 2012: 1097–1105
Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, van der Laak JAWM, van Ginneken B, Sánchez CI. A survey on deep learning in medical image analysis. Med Image Anal 2017; 42: 60–88
Khosravi P, Kazemi E, Imielinski M, Elemento O, Hajirasouliha I. Deep convolutional neural networks enable discrimination of heterogeneous digital pathology images. EBioMedicine 2018; 27: 317–328
Chilamkurthy S, Ghosh R, Tanamala S, Biviji M, Campeau NG, Venugopal VK, Mahajan V, Rao P, Warier P. Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study. Lancet 2018; 392(10162): 2388–2396
Meyer A, Zverinski D, Pfahringer B, Kempfert J, Kuehne T, Sündermann SH, Stamm C, Hofmann T, Falk V, Eickhoff C. Machine learning for real-time prediction of complications in critical care: a retrospective study. Lancet Respir Med 2018; 6(12): 905–914
Li X, Zhang S, Zhang Q, Wei X, Pan Y, Zhao J, Xin X, Qin C, Wang X, Li J, Yang F, Zhao Y, Yang M, Wang Q, Zheng Z, Zheng X, Yang X, Whitlow CT, Gurcan MN, Zhang L, Wang X, Pasche BC, Gao M, Zhang W, Chen K. Diagnosis of thyroid cancer using deep convolutional neural network models applied to sonographic images: a retrospective, multicohort, diagnostic study. Lancet Oncol 2019; 20(2): 193–201
Rubinstein E, Salhov M, Nidam-Leshem M, White V, Golan S, Baniel J, Bernstine H, Groshar D, Averbuch A. Unsupervised tumor detection in dynamic PET/CT imaging of the prostate. Med Image Anal 2019; 55: 27–40
Winkels M, Cohen TS. Pulmonary nodule detection in CT scans with equivariant CNNs. Med Image Anal 2019; 55: 15–26
Maicas G, Carneiro G, Bradley AP, Nascimento JC, Reid I. Deep reinforcement learning for active breast lesion detection from DCE-MRI. In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). New York: Springer, 2017: 665–673
Lee H, Yune S, Mansouri M, Kim M, Tajmir SH, Guerrier CE, Ebert SA, Pomerantz SR, Romero JM, Kamalian S, Gonzalez RG, Lev MH, Do S. An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets. Nat Biomed Eng 2019; 3(3): 173–182
Kamnitsas K, Ledig C, Newcombe VFJ, Simpson JP, Kane AD, Menon DK, Rueckert D, Glocker B. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med Image Anal 2017; 36: 61–78
Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston. 2015: 3431–3440
Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. In: Proceedings of International Conference on Medical Image Computing and ComputerAssisted Intervention (MICCAI). New York: Springer, 2015: 234–241
Cicek O, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O. 3D U-Net: learning dense volumetric segmentation from sparse annotation. In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). New York: Springer, 2016: 424–432
Zhou XY, Yang GZ. Normalization in training U-Net for 2D biomedical semantic segmentation. IEEE Robot Autom Lett 2019; 4(2): 1792–1799
Gibson E, Giganti F, Hu Y, Bonmati E, Bandula S, Gurusamy K, Davidson B, Pereira SP, Clarkson MJ, Barratt DC. Automatic multi-organ segmentation on abdominal CT with dense V-networks. IEEE Trans Med Imaging 2018; 37(8): 1822–1834
Wang G, Li W, Zuluaga MA, Pratt R, Patel PA, Aertsen M, Doel T, David AL, Deprest J, Ourselin S, Vercauteren T. Interactive medical image segmentation using deep learning with image-specific fine tuning. IEEE Trans Med Imaging 2018; 37(7): 1562–1573
Laina I, Rieke N, Rupprecht C, Vizca’ino JP, Eslami A, Tombari F, Navab N. Concurrent segmentation and localization for tracking of surgical instruments. In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). New York: Springer, 2017: 664–672
Feng X, Yang J, Laine AF, Angelini ED. Discriminative localization in CNNs for weakly-supervised segmentation of pulmonary nodules. In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). New York: Springer, 2017: 568–576
Bai W, Chen C, Tarroni G, Duan J, Guitton F, Petersen SE, Guo Y, Matthews PM, Rueckert D. Self-supervised learning for cardiac MR image segmentation by anatomical position prediction. In: International Conference on Medical Image Computing and Computer Assisted Intervention. New York: Springer, 2019: 541–549
Balakrishnan G, Zhao A, Sabuncu MR, Guttag J, Dalca AV. VoxelMorph: a learning framework for deformable medical image registration. IEEE Trans Med Imaging 2019: 38(8): 1788–1800
Shen Z, Han X, Xu Z, Niethammer M. Networks for joint affine and non-parametric image registration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach. 2019: 4224–4233
Hu Y, Modat M, Gibson E, Li W, Ghavami N, Bonmati E, Wang G, Bandula S, Moore CM, Emberton M, Ourselin S, Noble JA, Barratt DC, Vercauteren T. Weakly-supervised convolutional neural networks for multimodal image registration. Med Image Anal 2018; 49: 1–13
Miao S, Piat S, Fischer P, Tuysuzoglu A, Mewes P, Mansi T, Liao R. Dilated FCN for multi-agent 2D/3D medical image registration. In: Proceedings of AAAI Conference on Artificial Intelligence. New Orleans. 2018
Sokooti H, de Vos B, Berendsen F, Lelieveldt BP, Išgum I, Staring M. Nonrigid image registration using multi-scale 3D convolutional neural networks. In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). New York: Springer, 2017: 232–239
Liao R, Miao S, de Tournemire P, Grbic S, Kamen A, Mansi T, Comaniciu D. An artificial agent for robust image registration. In: Proceedings of AAAI Conference on Artificial Intelligence. San Francisco. 2017
Cool D, Downey D, Izawa J, Chin J, Fenster A. 3D prostate model formation from non-parallel 2D ultrasound biopsy images. Med Image Anal 2006; 10(6): 875–887
Zhou X, Yang G, Riga C, Lee S. Stent graft shape instantiation for fenestrated endovascular aortic repair. In: The Hamlyn Symposium on Medical Robotics. London. 2017
Zhou XY, Lin J, Riga C, Yang GZ, Lee SL. Real-time 3D shape instantiation from single fluoroscopy projection for fenestrated stent graft deployment. IEEE Robot Autom Lett 2018; 3(2): 1314–1321
Zheng JQ, Zhou XY, Riga C, Yang GZ. Real-time 3D shape instantiation for partially deployed stent segments from a single 2D fluoroscopic image in fenestrated endovascular aortic repair. IEEE Robot Autom Lett 2019; 4(4): 3703–3710
Zhou XY, Riga C, Lee SL, Yang GZ. Towards automatic 3D shape instantiation for deployed stent grafts: 2D multiple-class and class-imbalance marker segmentation with equally-weighted focal U-Net. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Madrid: IEEE, 2018: 1261–1267
Zheng JQ, Zhou XY, Riga C, Yang GZ. Towards 3D path planning from a single 2D fluoroscopic image for robot assisted fenestrated endovascular aortic repair. In: 2019 International Conference on Robotics and Automation (ICRA). Montreal: IEEE, 2019: 8747–8753
Lee SL, Chung A, Lerotic M, Hawkins MA, Tait D, Yang GZ. Dynamic shape instantiation for intra-operative guidance. In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). New York: Springer, 2010: 69–76
Zhou XY, Yang GZ, Lee SL. A real-time and registration-free framework for dynamic shape instantiation. Med Image Anal 2018; 44: 86–97
Zhou XY, Wang ZY, Li P, Zheng JQ, Yang GZ. One stage shape instantiation from a single 2D image to 3D point cloud. In: International Conference on Medical Image Computing and Computer Assisted Intervention. New York: Springer, 2019: 30–38
Mahmood F, Durr NJ. Deep learning and conditional random fields-based depth estimation and topographical reconstruction from conventional endoscopy. Med Image Anal 2018; 48: 230–243
Mahmood F, Chen R, Durr NJ. Unsupervised reverse domain adaptation for synthetic medical images via adversarial training. IEEE Trans Med Imaging 2018; 37(12): 2572–2581
Turan M, Ornek EP, Ibrahimli N, Giracoglu C, Almalioglu Y, Yanik MF, Sitti M. Unsupervised odometry and depth learning for endoscopic capsule robots. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Madrid: IEEE, 2018: 1801–1807
Shen M, Gu Y, Liu N, Yang GZ. Context-aware depth and pose estimation for bronchoscopic navigation. IEEE Robot Autom Lett 2019; 4(2): 732–739
Zhou T, Brown M, Snavely N, Lowe DG. Unsupervised learning of depth and ego-motion from video. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Hawaii. 2017
Zhan H, Garg R, Saroj Weerasekera C, Li K, Agarwal H, Reid I. Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City. 2018
Ye M, Johns E, Handa A, Zhang L, Pratt P, Yang GZ. Selfsupervised siamese learning on stereo image pairs for depth estimation in robotic surgery. In: The Hamlyn Symposium on Medical Robotics. London. 2017: 27
Zhu JY, Park T, Isola P, Efros AA. Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). Venice. 2017: 2223–2232
Turan M, Almalioglu Y, Araujo H, Konukoglu E, Sitti M. Deep endovo: a recurrent convolutional neural network (RCNN) based visual odometry approach for endoscopic capsule robots. Neurocomputing 2018; 275: 1861–1870
Sganga J, Eng D, Graetzel C, Camarillo D. Offsetnet: deep learning for localization in the lung using rendered images. In: 2019 International Conference on Robotics and Automation (ICRA). Montreal: IEEE, 2019: 5046–5052
Mountney P, Stoyanov D, Davison A, Yang GZ. Simultaneous stereoscope localization and soft-tissue mapping for minimal invasive surgery. In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). New York: Springer, 2006: 347–354
Davison AJ, Reid ID, Molton ND, Stasse O. MonoSLAM: real-time single camera SLAM. IEEE Trans Pattern Anal Mach Intell 2007; 29(6): 1052–1067
Mountney P, Yang GZ. Motion compensated SLAM for image guided surgery. In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). New York: Springer, 2010: 496–504
Grasa OG, Bernal E, Casado S, Gil I, Montiel JM. Visual SLAM for handheld monocular endoscope. IEEE Trans Med Imaging 2014; 33(1): 135–146
Turan M, Almalioglu Y, Araujo H, Konukoglu E, Sitti M. A nonrigid map fusion-based direct SLAM method for endoscopic capsule robots. Int J Intell Robot Appl 2017; 1(4): 399–409
Song J, Wang J, Zhao L, Huang S, Dissanayake G. MISSLAM: real-time large-scale dense deformable SLAM system in minimal invasive surgery based on heterogeneous computing. IEEE Robot Autom Lett 2018; 3(4): 4068–4075
Zhou XY, Ernst S, Lee SL. Path planning for robot-enhanced cardiac radiofrequency catheter ablation. In: 2016 IEEE international conference on robotics and automation (ICRA). Stockholm: IEEE, 2016: 4172–4177
Shi C, Giannarou S, Lee SL, Yang GZ. Simultaneous catheter and environment modeling for trans-catheter aortic valve implantation. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Chicago: IEEE, 2014: 2024–2029
Zhao L, Giannarou S, Lee SL, Yang GZ. SCEM +: real-time robust simultaneous catheter and environment modeling for endovascular navigation. IEEE Robot Autom Lett 2016; 1(2): 961–968
Zhao L, Giannarou S, Lee SL, Yang GZ. Registration-free simultaneous catheter and environment modelling. In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). New York: Springer, 2016: 525–533
Mountney P, Yang GZ. Soft tissue tracking for minimally invasive surgery: learning local deformation online. In: Proceedings of International Conference on Medical Image Computing and ComputerAssisted Intervention (MICCAI). New York: Springer, 2008: 364–372
Ye M, Giannarou S, Meining A, Yang GZ. Online tracking and retargeting with applications to optical biopsy in gastrointestinal endoscopic examinations. Med Image Anal 2016; 30: 144–157
Wang R, Zhang M, Meng X, Geng Z, Wang FY. 3D tracking for augmented reality using combined region and dense cues in endoscopic surgery. IEEE J Biomed Health Inform 2018; 22(5): 1540–1551
Bernhardt S, Nicolau SA, Soler L, Doignon C. The status of augmented reality in laparoscopic surgery as of 2016. Med Image Anal 2017; 37: 66–90
Wang J, Suenaga H, Hoshi K, Yang L, Kobayashi E, Sakuma I, Liao H. Augmented reality navigation with automatic marker-free image registration using 3-D image overlay for dental surgery. IEEE Trans Biomed Eng 2014; 61(4): 1295–1304
Pratt P, Ives M, Lawton G, Simmons J, Radev N, Spyropoulou L, Amiras D. Through the HoloLens™ looking glass: augmented reality for extremity reconstruction surgery using 3D vascular models with perforating vessels. Eur Radiol Exp 2018; 2(1): 2
Zhang X, Wang J, Wang T, Ji X, Shen Y, Sun Z, Zhang X. A markerless automatic deformable registration framework for augmented reality navigation of laparoscopy partial nephrectomy. Int J CARS 2019; 14(8): 1285–1294
Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med 2019; 25(1): 44–56
Mirnezami R, Ahmed A. Surgery 3.0, artificial intelligence and the next-generation surgeon. Br J Surg 2018; 105(5): 463–465
Bouget D, Benenson R, Omran M, Riffaud L, Schiele B, Jannin P. Detecting surgical tools by modelling local appearance and global shape. IEEE Trans Med Imaging 2015; 34(12): 2603–2617
Shvets AA, Rakhlin A, Kalinin AA, Iglovikov VI. Automatic instrument segmentation in robot-assisted surgery using deep learning. In: Proceedings of IEEE International Conference on Machine Learning and Applications (ICMLA). Stockholm: IEEE, 2018: 624–628
Islam M, Atputharuban DA, Ramesh R, Ren H. Real-time instrument segmentation in robotic surgery using auxiliary supervised deep adversarial learning. IEEE Robot Autom Lett 2019; 4(2): 2188–2195
Sznitman R, Richa R, Taylor RH, Jedynak B, Hager GD. Unified detection and tracking of instruments during retinal microsurgery. IEEE Trans Pattern Anal Mach Intell 2013; 35(5): 1263–1273
Zhang L, Ye M, Chan PL, Yang GZ. Real-time surgical tool tracking and pose estimation using a hybrid cylindrical marker. Int J CARS 2017; 12(6): 921–930
Ye M, Zhang L, Giannarou S, Yang GZ. Real-time 3D tracking of articulated tools for robotic surgery. In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). New York: Springer, 2016: 386–394
Zhao Z, Voros S, Weng Y, Chang F, Li R. Tracking-by-detection of surgical instruments in minimally invasive surgery via the convolutional neural network deep learning-based method. Comput Assist Surg (Abingdon) 2017; 22(sup1): 26–35
Nwoye CI, Mutter D, Marescaux J, Padoy N. Weakly supervised convolutional LSTM approach for tool tracking in laparoscopic videos. Int J CARS 2019; 14(6): 1059–1067
Sarikaya D, Corso JJ, Guru KA. Detection and localization of robotic tools in robot-assisted surgery videos using deep neural networks for region proposal and detection. IEEE Trans Med Imaging 2017; 36(7): 1542–1549
Kurmann T, Neila PM, Du X, Fua P, Stoyanov D, Wolf S, Sznitman R. Simultaneous recognition and pose estimation of instruments in minimally invasive surgery. In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). New York: Springer, 2017: 505–513
Padoy N, Hager GD. 3D thread tracking for robotic assistance in tele-surgery. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). San Francisco: IEEE, 2011: 2102–2107
Hu Y, Gu Y, Yang J, Yang GZ. Multi-stage suture detection for robot assisted anastomosis based on deep learning. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA). Brisbane: IEEE, 2018: 1–8
Gu Y, Hu Y, Zhang L, Yang J, Yang GZ. Cross-scene suture thread parsing for robot assisted anastomosis based on joint feature learning. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Madrid: IEEE, 2018: 769–776
Aviles AI, Alsaleh SM, Hahn JK, Casals A. Towards retrieving force feedback in robotic-assisted surgery: a supervised neurorecurrent-vision approach. IEEE Trans Haptics 2017; 10(3): 431–443
Marban A, Srinivasan V, Samek W, Ferna’ndez J, Casals A. Estimation of interaction forces in robotic surgery using a semisupervised deep neural network model. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Madrid: IEEE, 2018: 761–768
Ahmidi N, Tao L, Sefati S, Gao Y, Lea C, Haro BB, Zappella L, Khudanpur S, Vidal R, Hager GD. A dataset and benchmarks for segmentation and recognition of gestures in robotic surgery. IEEE Trans Biomed Eng 2017; 64(9): 2025–2041
Fard MJ, Ameri S, Chinnam RB, Ellis RD. Soft boundary approach for unsupervised gesture segmentation in robotic-assisted surgery. IEEE Robot Autom Lett 2017; 2(1): 171–178
Krishnan S, Garg A, Patil S, Lea C, Hager G, Abbeel P, Goldberg K. Transition state clustering: unsupervised surgical trajectory segmentation for robot learning. Int J Robot Res 2017; 36(13–14): 1595–1618
Murali A, Garg A, Krishnan S, Pokorny FT, Abbeel P, Darrell T, Goldberg K. TSC-DL: unsupervised trajectory segmentation of multi-modal surgical demonstrations with deep learning. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA). Stockholm: IEEE, 2016: 4150–4157
Zappella L, Bíjar B, Hager G, Vidal R. Surgical gesture classification from video and kinematic data. Med Image Anal 2013; 17(7): 732–745
Tao L, Zappella L, Hager GD, Vidal R. Surgical gesture segmentation and recognition. In: Proceedings o International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). New York: Springer, 2013: 339–346
Despinoy F, Bouget D, Forestier G, Penet C, Zemiti N, Poignet P, Jannin P. Unsupervised trajectory segmentation for surgical gesture recognition in robotic training. IEEE Trans Biomed Eng 2016; 63(6): 1280–1291
DiPietro R, Ahmidi N, Malpani A, Waldram M, Lee GI, Lee MR, Vedula SS, Hager GD. Segmenting and classifying activities in robot-assisted surgery with recurrent neural networks. Int J CARS 2019; 14(11): 2005–2020
Liu D, Jiang T. Deep reinforcement learning for surgical gesture segmentation and classification. In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). New York: Springer, 2018: 247–255
Padoy N, Hager GD. Human-machine collaborative surgery using learned models. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA). Shanghai: IEEE, 2011: 5285–5292
Calinon S, Bruno D, Malekzadeh MS, Nanayakkara T, Caldwell DG. Human-robot skills transfer interfaces for a flexible surgical robot. Comput Methods Programs Biomed 2014; 116(2): 81–96
Osa T, Sugita N, Mitsuishi M. Online trajectory planning in dynamic environments for surgical task automation. In: Robotics: Science and Systems. Berkeley. 2014: 1–9
Van Den Berg J, Miller S, Duckworth D, Hu H, Wan A, Fu XY, Goldberg K, Abbeel P. Superhuman performance of surgical tasks by robots using iterative learning from human-guided demonstrations. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA). Alaska: IEEE, 2010: 2074–2081
Murali A, Sen S, Kehoe B, Garg A, McFarland S, Patil S, Boyd WD, Lim S, Abbeel P, Goldberg K. Learning by observation for surgical subtasks: multilateral cutting of 3D viscoelastic and 2D orthotropic tissue phantoms. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA). Seattle: IEEE, 2015:1202–1209
Mayer H, Gomez F, Wierstra D, Nagy I, Knoll A, Schmidhuber J. A system for robotic heart surgery that learns to tie knots using recurrent neural networks. Adv Robot 2008; 22(13–14): 1521–1537
De Momi E, Kranendonk L, Valenti M, Enayati N, Ferrigno G. A neural network-based approach for trajectory planning in robot-human handover tasks. Front Robot AI 2016; 3: 34
Kober J, Bagnell JA, Peters J. Reinforcement learning in robotics: a survey. Int J Robot Res 2013; 32(11): 1238–1274
Abbeel P, Ng AY. Apprenticeship learning via inverse reinforcement learning. In: Proceedings of International Conference on Machine Learning (ICML). Beijing: ACM, 2004: 1
Tan X, Chng CB, Su Y, Lim KB, Chui CK. Robotassisted training in laparoscopy using deep reinforcement learning. IEEE Robot Autom Lett 2019; 4(2): 485–492
Ho J, Ermon S. Generative adversarial imitation learning. In: Proceedings of Advances in Neural Information Processing Systems (NIPS). Barcelona. 2016: 4565–4573
Levine S, Finn C, Darrell T, Abbeel P. End-to-end training of deep visuomotor policies. J Mach Learn Res 2016; 17(1): 1334–1373
Thananjeyan B, Garg A, Krishnan S, Chen C, Miller L, Goldberg K. Multilateral surgical pattern cutting in 2D orthotropic gauze with deep reinforcement learning policies for tensioning. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA). Singapore: IEEE, 2017: 2371–2378
Yang GZ, Dempere-Marco L, Hu XP, Rowe A. Visual search: psychophysical models and practical applications. Image Vis Comput 2002; 20(4): 291–305
Yang GZ, Mylonas GP, Kwok KW, Chung A. Perceptual docking for robotic control. In: International Workshop on Medical Imaging and Virtual Reality. New York: Springer, 2008: 21–30
Visentini-Scarzanella M, Mylonas GP, Stoyanov D, Yang GZ. I-brush: a gaze-contingent virtual paintbrush for dense 3D reconstruction in robotic assisted surgery. In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). New York: Springer, 2009: 353–360
Fujii K, Gras G, Salerno A, Yang GZ. Gaze gesture based human robot interaction for laparoscopic surgery. Med Image Anal 2018; 44: 196–214
Nishikawa A, Hosoi T, Koara K, Negoro D, Hikita A, Asano S, Kakutani H, Miyazaki F, Sekimoto M, Yasui M, Miyake Y, Takiguchi S, Monden M. Face mouse: a novel human-machine interface for controlling the position of a laparoscope. IEEE Trans Robot Autom 2003; 19(5): 825–841
Hong N, Kim M, Lee C, Kim S. Head-mounted interface for intuitive vision control and continuous surgical operation in a surgical robot system. Med Biol Eng Comput 2019; 57(3): 601–614
Graves A. Mohamed Ar, Hinton G. Speech recognition with deep recurrent neural networks. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing. Vancouver: IEEE, 2013: 6645–6649
Zinchenko K, Wu CY, Song KT. A study on speech recognition control for a surgical robot. IEEE Trans Industr Inform 2017; 13 (2): 607–615
Jacob MG, Li YT, Akingba GA, Wachs JP. Collaboration with a robotic scrub nurse. Commun ACM 2013; 56(5): 68–75
Wen R, Tay WL, Nguyen BP, Chng CB, Chui CK. Hand gesture guided robot-assisted surgery based on a direct augmented reality interface. ComputMethods Programs Biomed 2014; 116(2): 68–80
Oyedotun OK, Khashman A. Deep learning in vision-based static hand gesture recognition. Neural Comput Appl 2017; 28(12): 3941–3951
Hu Y, Zhang L, Li W, Yang GZ. Robotic sewing and knot tying for personalized stent graft manufacturing. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Madrid: IEEE, 2018: 754–760
Hu Y, Li W, Zhang L, Yang GZ. Designing, prototyping, and testing a flexible suturing robot for transanal endoscopic microsurgery. IEEE Robot Autom Lett 2019; 4(2): 1669–1675
Yang GZ, Bellingham J, Dupont PE, Fischer P, Floridi L, Full R, Jacobstein N, Kumar V, McNutt M, Merrifield R, Nelson BJ, Scassellati B, Taddeo M, Taylor R, Veloso M, Wang ZL, Wood R. The grand challenges of science robotics. Sc Robot 2018; 3(14): eaar7650
Yang GZ, Cambias J, Cleary K, Daimler E, Drake J, Dupont PE, Hata N, Kazanzides P, Martel S, Patel RV, Santos VJ, Taylor RH. Medical roboticsregulatory, ethical, and legal considerations for increasing levels of autonomy. Sci Robot 2017; 2(4): 8638
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Xiao-Yun Zhou, Yao Guo, Mali Shen, and Guang-Zhong Yang declare that they have no conflict of interest. This manuscript is a review article and does not involve a research protocol requiring approval by the relevant institutional review board or ethics committee.
Rights and permissions
About this article
Cite this article
Zhou, XY., Guo, Y., Shen, M. et al. Application of artificial intelligence in surgery. Front. Med. 14, 417–430 (2020). https://doi.org/10.1007/s11684-020-0770-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11684-020-0770-0