With the recent advancements in information communication technology (ICT) and the expansion of unique device identification regulation, the trend toward the “Internet of Things (IoT)” is gaining popularity across various fields, whereby everything in one’s surrounding is connected to the internet [1, 2]. In recent years, attention has been further drawn to new approaches such as Industry 4.0 [3, 4] and the Industrial Internet Consortium [5]. At the heart of this concept is the IoT trend, which is the key to “visualization” and “connectivity” technologies.

The IoT trend is reportedly expanding gradually into the medical field and is being applied in medical devices. For instance, the “smart bed system™” is capable of managing various biometric data centrally through ICT [6]; this is a method of collecting and managing the biometric data of healthy individuals and of patients using a wearable device [7, 8], followed by postoperative data follow-up after bariatric surgery [9]. Furthermore, attempts have been made in the field of surgery to automatically and centrally manage medical devices such as steel instruments [10,11,12,13,14] and to automatically monitor the control of medical devices during intra-abdominal surgical procedure [15,16,17]. Moreover, methods have been examined to create an IoT-based medical information system [18]. For this purpose, it is important to digitize various data using sensors and to search for a means to apply the obtained information.

However, above applications in the medical field are only “experimental” undertakings. Compared with the IoT trend in a general society, which is reaching the level of social implementation, it can be said that the application is not up to par. In particular, the IoT trend has not yet commenced in the field of surgery, and the feasibility of whether the “visualization” of surgical procedure can actually be put into practice through the IoT trend has not yet been demonstrated, while some forceps and disposable surgical equipment used for open surgery are tagged with bar codes and/or radio frequency identifier (RFID). These tags are used mostly for logistic management, and one cannot identify the forceps’ movement and traffic during surgery or determine if the energy device is switched on or off.

In the future, assuming that the IoT trend will definitely reach inside the operating room, questions on what would be the outcome of this technological application, and whether it would truly enable the visualization of surgical procedures by technology connected to sensor technology are raised. To address these questions, we conducted a feasibility study using large animals with a particular focus on whether forceps traffic in laparoscopic surgery (i.e., the frequency by which the forceps in the surgeons left and right hands are used) and activation (i.e., electrocautery on/off) can be visualized.

Materials and methods

Animals

The entire experiment protocol in the present study was approved by the institutional animal care and ethical review board. All procedures were conducted in a standard manner with animals under general anesthesia using 3-month-old female pigs (n = 5; average weight = 35 kg). Each pig was humanely euthanized upon completion of the experiment. Board-certified surgeons conducted laparoscopic cholecystectomy in the supine position. The pigs received no pretreatment.

Surgical procedure

A rigid laparoscopic system (IMAGE1 STM System; KARL STORZ, Tuttlingen, Germany) was used for the surgery, whereas an electrocautery generator (VIO 300D; Erbe-Elektromedizin GmbH; Tuebingen, Germany) was used as the energy device. A total of four ports were used, including one port of 12-mm diameter for a camera and three ports of 5-mm diameter for forceps. For all pigs, the surgery was conducted using the French technique from the caudal end of the pig in a dorsal position [19,20,21,22]. Next, after confirming the critical view of safety, bile duct was clipped and dissected [23, 24] using a standard laparoscopic clip applier.

Surgeon’s forceps tracking by RFID

For the experiment, we used an Ultra-High-Frequency (UHF) RFID tag (WT-A533; FUJITSU FRONTECH, Tokyo, Japan) (Fig. 1A, B), and each RFID tag was embedded with an ID to identify each pair of forceps. For the RFID tag, we used the smallest tags that could be autoclaved during the experiment. On the left and right sides of the operating table, RFID readers (TFU-AN33A; FUJITSU FRONTECH, Tokyo, Japan) were installed to detect the RFID tags (Fig. 1C). The RFID tags were installed on all forceps and electrocautery probe used, and the traffics involving the insertion and removal of the forceps were detected via a short-range wireless communication system by the RFID reader. All surgeons wore a sterilized surgical pocket around their abdomen to store forceps and for management purpose (EndoApron; Daiei, Osaka, Japan). The movement of the forceps right from their removal from the sterilized surgical pocket to its use in the operative field by the left and right hands of the surgeon were captured by the RFID reader and automatically recorded (Fig. 2A).

Fig. 1
figure 1

RFID tag and the output sensor of the electrocautery generator. A An RFID tag was attached to all forceps. B The RFID tag used could be autoclaved successfully. C An RFID antenna was installed to the left and right of the surgeon (the area within the white circle). D A coil to detect the output was installed on the electrocautery generator (the area within the white circle)

Full size image
Fig. 2
figure 2

The trapping of the surgical instrument with RFID and the capture of the usage of surgical energy device. A schematic drawing of the automatic recording method of each device. A The insertion and removal of the forceps in the surgeon’s left and right hands were captured by the RFID readers and then automatically recorded as the associated traffics. B By electrically capturing the activation of electrocautery probe, the on/off rates and the activation time could be automatically recorded

Full size image

Recording of activation of electrocautery probe

The electrocautery generator was attached with a counter electrode plate cable detector coil (Fig. 1D). Using this detector coil, high-frequency output of the electrocautery (0.5–10 MHz) was detected without contact. The probe’s on/off rates and the activation time of the hook-type monopolar probe used in the actual experiment were automatically recorded (Fig. 2B).

All data were accumulated into the prototype of a dedicated software (XTRACE; EXSCION, Tokyo, Japan) (Fig. 3), and post-test analysis was performed on Microsoft excel and spreadsheet (Microsoft Corporation, Washington, United States of America) after converting the data into a standard spreadsheet software.

Fig. 3
figure 3

The visualized behavior of each device. Insertion/removal of the forceps and the activation of the electrocautery were automatically recorded and processed on a dedicated software. The visualized data can clearly show the behavior of each device on a PC screen

Full size image

Results

A study overview

Five surgeons with 9–24 years of experience conducted laparoscopic cholecystectomy on pigs (n = 5). All were board-certified surgeons. For all subjects in the present experiment, we could automatically record the intraoperative forceps traffics and activation of electrosurgical probe via the RFID tags and readers (Table 1). Intraoperative data collection revealed no onset of incidents that could prevent the automatic recording. In the data obtained by automatic recording, all items could be digitized by conversion to a standard spreadsheet. The operative duration was a median of 1828 s (1159–2962 s).

Table 1 Overall study outcomes
Full size table

The insertion time-rate of the forceps according to the left and right hands

We were able to clarify the insertion rate of the forceps in the surgeons’ left and right hands. In the right hands of the surgeons, the electrosurgical probe was used for the longest time [1179 s; 75% (771–1445 s)], followed by the Maryland dissectors [149 s; 10% (107–783 s)], and the clip applier [91 s; 2% (23–140 s)] (Fig. 4A). In the left hand of the surgeon, non-traumatic grasping forceps were used primarily [1780 s; 93% (1048–2755 s)] (Fig. 4B). In the data obtained, the numbers of insertions and removals of the forceps were recorded, which was used to confirm which surgeon used which forceps, the associated number of insertions and removals of each forceps, and the time each forceps were used.

Fig. 4
figure 4

Comparison of the left and right forceps used by the surgeon and the insertion rate of each forceps in the abdominal cavity. A The rate of insertion of the forceps in the surgeon’s right hand. B The rate of insertion of the forceps in the surgeon’s left hand

Full size image

Visualization of the usage of electrocautery

We were able to digitize all usages, including the on/off rates and activation time of electrocautery (Fig. 5). Upon analyzing the usage of the electrocautery probe used most often by the right hand, the actual activation time was a median of 9% of the insertion time of the electric scalpel (Fig. 5A). This tendency was comparable among all surgeons, with no cases showing > 10% of the activation time. The activation time per usage of the electrocautery probe was a median of 0.41 s (0.14–0.57 s), with variation observed among surgeons (Fig. 5B).

Fig. 5
figure 5

The ON/OFF rates and the activation time of the electrocautery. A The activation time-rate per overall duration of the electrocautery usage by each surgeon. B The activation time per electric scalpel usage by each surgeon [median: 0.41 s/time (range 0.14–0.57 s/time)]

Full size image

Discussion

The IoT is currently one of the most-discussed topics in academic and industrial societies. The internet has become indispensable for everyday objects around us and has therefore gained unprecedented popularity [25]. The IoT can be of great advantage to businesses such as in the optimization of processing, complex autonomous systems, and decision-making analysis of sensor drives [26]. At General Electric (GE), when a jet engine functions poorly or malfunctions, information is collected using several different sensors attached to the engine for managing and repairing important engine parts in advance [27]. IoT is thus expected to have a major impact on people’s lives and businesses [28]. In the medical field, various advancements have already been reported with the use of IoT [6, 8,9,10,11,12,13,14,15]. However, in the field of surgery, particularly for devices associated with surgical procedures, the advancement is considerably lagging.

The question is why medical and surgical fields are lagging behind in the IoT trend. One of the reasons could be because of the inherent difference between industrial circles and businesses as compared to medical fields, particularly surgeries, is that sterilization is needed in the latter [29]. Disinfection and sterilization are indispensable to prevent transmission of infectious pathogens from medical activities or surgical instruments to the patient, and, if the guidelines based on scientific grounds are not followed, several disease outbreaks can occur [30,31,32,33,34,35,36]. Moreover, there is currently no technique to sterilize sensors capable of surgical application, which has severely limited the use of sensors in surgical devices. Furthermore, it is possible that the regulatory agencies of each country affect problems and poor awareness of physicians. Another technical hurdle to overcome is the crosstalk in the wireless system [37].

The present experiment is the first global success in visualizing specific movements during a surgical procedure. As reported, data on aspects that indicate the surgical instrument behavior, the devices held by the surgeon’s left and right hands, and the manner by which the electrosurgical probe is used would enable live feedback to the surgeon by ways of visualization and standardization. Furthermore, this technique can study the correlational aspects of various factors that affect the performance of a surgery, which would help refine surgical performance and increase patients’ safety.

As a result of this study, most of the devices used with the right hand were electrocautery probes in all the operations. However, the actual activation time of the electrocautery probe was short. From this, it turned out that the electrocautery probe of the right hand was used for various purposes such as blunt dissection and ductal tunneling besides coagulation, hemostasis, and cutting. This can be consistent with the fact that when the surgeon uses energy devices, it conducts not only coagulation and cutting but also blunt dissection and other tissue handling. Furthermore, in this study, surgeon A performed laparoscopic cholecystectomy as a mentor surgeon. He intentionally used different forceps depending on the situation, for e.g., Maryland dissector for periductal dissection and hook-type electrocautery for gallbladder dissection from liver bed. He had to change instruments frequently mostly for educational and/or demonstration purposes. As a result, the total use time of the electrocautery probe decreased and the traffic of the surgical devices increased. Meanwhile, surgeons (B, E) showed more practical performance, demonstrating the full use of electrocautery probe for multiple purposes. As a consequence, the activation time per usage of the electrocautery probe became longer, whereas the traffic become smaller. Further “big data” analysis may demonstrate that surgical technique might be different depending on the function of operating surgeons.

In the present experiment, only a small volume of study data (n = 5) was involved. However, this experiment paved the way for large-scale study with a huge data set and, subsequently, a big data analysis. Accumulating surgical data can help identify the collective rule for data analysis. From the perspective of data characteristics, although the data obtained for an individual patient provides few evaluation items, it characteristically has a large number of samples. However, increasing the number of subjects will help identity diversity in individualized patterns. Such data can be applied as “benchmarks” in various procedures as well as used for maintaining the technical quality of procedures and for learning, training, and credentialing purposes.

Furthermore, linking of such a huge data set automatically to a hospital information system (HIS) and incorporation of artificial intelligence (AI) would give rise to several new possibilities in the future. It is believed that the decision-making abilities of surgeons are worthy of trust, and that, at present, every important step in a surgery is performed at the discretion of the surgeon. Conversely, AI makes judgements prospectively based on the correlation obtained from data. Therefore, if information from AI is included in intraoperative judgements, more appropriate perioperative care can be obtained as compared to those through conventional method via controlling various risks during surgery and providing suitable advices to the surgeon while predicting postoperative progress in advance. In fact, it has been reported that AI can provide a supporting role to the surgeon [38], and combining AI with a huge data set can be of immeasurable benefit [39].

Conversely, the present study was only a proof of concept (POC) with very small data and had several limitations. First, the function of the tip of the forceps (end-effector motion) could not be visualized. Furthermore, for the energy device, only data pertaining to the electrocautery switching on and off were digitized, while the mode of generator used remained unknown. Second, the surgical technique was not linked with image data. Third, the relationship between vital signs, the state of anesthetic devices such as the mechanical ventilator, and drug administrations were not examined. Fourth, the sensors were still too large and could have interfered with surgery; thus, they cannot be used practically at present. Finally, when using such technologies in the operating room, problems related to ensuring information security and protecting personal information may arise; however, in the present study, security issues were not examined. Further studies that take security into account are thus warranted. In recent years, sensor technology has rapidly improved, enabling the development of smaller, multi-functional sensors with higher performance. In the near future, we may be able to ascertain the true meaning of forceps traffic. Furthermore, in the field of dermatology and gastroenterology, it has been suggested that diagnosis can be made by coordinating image data and AI [38, 40, 41], and in the near future, it may be possible to coordinate surgical imaging with AI. Moreover, coordination with HIS could help correlate outcomes and might provide high-level clinical evidence. The present study demonstrated a POC, and we are planning a clinical study in the future.

The IoT trend in surgical devices enables intraoperative events, and the traffic of devices can be automatically recorded, which suggests the possibility that procedures can be visualized and standardized. Thus, it is expected that creating big data and including such data for analyses by AI technology will help present the most-suitable usage of surgical devices, find the proper usage of the energy devices, improve the safety of surgery, and make them standardized. Moreover, we believe that combining HIS with the postoperative outcomes will contribute to the improvement of the quality of surgical treatment overall.