Introduction

High-precision manipulation is essential for retinal surgery due to high level of safety required to handle delicate tissue in a small constrained workspace. Factors such as physiological hand tremor, fatigue, poor kinesthetic feedback, patient movement, and the absence of force sensing could potentially lead to surgeon’s misoperations and subsequently iatrogenic injury. During retinal surgery, the surgeon inserts small instruments (e.g., 25 Ga, \(\phi =0.5\) mm) through the sclerotomy port (\(\phi <1\) mm) to perform delicate tissue manipulations in the posterior of the eye as shown in Fig. 1. The sclerotomy port continuously sustains variable tool-to-sclera forces. Extreme tool maneuvers applied to the port could result in excessive forces and cause scleral injury. One example of such a challenging retinal surgery task is retinal vein cannulation, a potential treatment for retinal vein occlusion. During this operation, the surgeon needs to pass and hold the needle or micropipette through the sclerotomy port and carefully insert it into the occluded retinal vein. After insertion, the surgeon needs to hold the tool steadily for as long as two minutes for clot-dissolving drug injection. In such a delicate operation, any unintentional movements of the surgeon’s hands may put the eye at high risk of injury. Therefore, the success of retinal surgeries is highly dependent on the surgeon’s level of experience and skills, a requirement that could be relaxed with the help of more advanced robotic assistive technology.

Continuing efforts are being devoted to the development of the surgical robotic systems to enhance and expand surgeon’s capabilities in retinal surgery. Current technology can be categorized into teleoperative manipulation systems [1,2,3], hand-held robotic devices [4], untethered microrobots [5], and flexible micromanipulators [6]. Recently, two robot-assisted retinal surgeries have been performed successfully on human patients [7, 8], demonstrating the clinical feasibility of robotic microsurgery.

Our team developed the “steady hand” eye robot (SHER) capable of human–robot cooperative control [9]. Surgical tools can be mounted on the robot end-effector, and user manipulates the tool in a collaborative way with the robot. The velocities of SHER follow the user’s manipulation force applied on the tool handle measured by an embedded end-effector force/torque sensor. Furthermore, our group has designed and developed a series of “smart” tools based on fiber Bragg gratings (FBGs) sensors [10, 11], in which the multi-function sensing tool [12] can measure scleral force, tool insertion depth, and tool tip contact force. Scleral force and insertion depth are the key measurements employed in this work. The measured force is used for force control, and haptic or audio feedback to the surgeon to enhance safety [13, 14].

The aforementioned robotic devices can give passive support to surgeons, i.e., filtering hand tremor, improving tool location accuracy, and providing operation force information to surgeons. However, the existing robotic systems cannot actively counteract or prevent iatrogenic accidents (e.g., from extreme scleral forces) caused by surgeon’s misoperations or fatigue, since it is generally difficult to quantify and react on time to the surgeon’s next move.

Fig. 1
figure 1

Illustration of retinal microsurgery. a Two tools are inserted into eyeball via sclerotomy port, and the sclera sustains constant manipulation force. b The surgeon manipulates the instruments under the microscope to perform retinal surgery

Full size image
Fig. 2
figure 2

Active interventional robotic system (AIRS). The robotic manipulator is activated to move along the direction of scleral force at a certain velocity once the predicted force status is unsafe. The RNN is trained offline in advance as the predictor. A force sensing tool is developed to measure scleral force (\(F_x\) and \(F_y\)) and insertion depth

Full size image

Predicting the surgeon’s following movement based on their prior motions can potentially resolve the above-mentioned problems. The predicted information can be used to identify unsafe manipulations that might occur in the near future. Then, robotic system could actively take actions, e.g., retreating the instruments or pausing the surgeon’s operation to prevent high forces applied to delicate tissue. To this end, we propose an active interventional robotic system (AIRS) summarized in Fig. 2. For an initial study, we focus on scleral safety. A force sensing tool is developed to collect scleral force. A recurrent neural network (RNN) with long short-term memory (LSTM) units is designed to predict and classify the impending scleral forces into safe and unsafe, using the notion of a safety boundary. An initial linear admittance control taking the predicted force status as input is applied to actuate the robot manipulator to reduce future forces. Finally, AIRS is implemented using SHER research platform and is evaluated by performing “vessel following,” a typical task in retinal surgery, on an eye phantom. The experimental results prove the advantage of AIRS in robot-assisted eye surgery, i.e., the unsafe force proportion is kept below 3% with AIRS compared to 28.3% with SHER and 26.6% with freehand for all participated users.

Background

Fig. 3
figure 3

Force sensing tool. a Overall dimension of the tool. The sensing part contains nine FBG sensors, which are located on three segments along tool shaft. b The radial positions of FBG fibers on tool shaft. c The calibration results of scleral force. The measurement RMS errors of sclera force are 3.8 mN. d The calibration results of insertion depth. The measurement RMS errors of insertion depth are 0.3 mm

Full size image

Fiber Bragg gratings sensors

FBGs [15] are a type of fiber optic strain sensors. The Bragg gratings work as a wavelength-specific reflector or filter. A narrow spectral component at this particular wavelength, termed the Bragg wavelength, is reflected, while the spectrum without this component is transmitted. The Bragg wavelength is determined by the grating period, which depends on the strain generated in the fiber:

$$\begin{aligned} \lambda _\mathrm{{B}} = 2n_\mathrm{{e}} \varLambda \end{aligned}$$

where \(\lambda _\mathrm{{B}}\) denotes the Bragg wavelength, \(n_\mathrm{{e}}\) is the effective refractive index of the grating, and \(\varLambda \) denotes the grating period.

We use FBGs as force sensor, since they are small enough (60–200 \(\upmu \mathrm{{m}}\) in diameter) to be integrated into the tool without significantly increasing the tool’s dimensions. Besides, FBG sensors are very sensitive, and they can detect strain changes of less than 1 \(\upmu \epsilon \). They are lightweight, biocompatible, sterilizable, multiplexable, and immune to electrostatic and electromagnetic noise.

Force sensing tool

The force sensing tool was previously designed and fabricated as shown in Fig. 3, which is employed to measure the scleral force and insertion depth in this work. The tool is composed of three parts: tool handle, adapter, and tool shaft (sensing part). The tool shaft is made of a stainless steel wire with the diameter of 0.635 mm. It is machined to contain three longitudinal V-shaped grooves. An optical fiber is carefully glued and positioned inside each groove. Each fiber contains three FBG sensors (Technica S.A, Beijing, China) which are separately located in segments I, II, and III. Hence, nine FBG sensors are embedded in the tool shaft in total.

The tool is calibrated using previously developed algorithm [12] to measure sclera force and insertion depth. The sclera force can be calculated using Eq. (1) :

$$\begin{aligned} F _\mathrm{{s}} = \frac{M _{II}-M _{I}}{\varDelta l } = \frac{K _{II }\varDelta S _{II}-K _{I }\varDelta S _I}{\varDelta l } \end{aligned}$$
(1)

where \(\varDelta l = l _{II} - l _{I}\) is the constant distance between FBG sensors of segments I and II as shown in Fig. 3. \(F _\mathrm{{s}} = [F _x, F _y]^T\) is the sclera force applied at sclerotomy port. \(M _i = [M _x, M _y]^T\) denotes the moment attributed to \(F _\mathrm{{s}}\) on FBG sensors of segment i. \(K _i\) (\(i = \)I, II) are \(3 \times 2\) constant coefficient matrices, which are obtained through the tool calibration procedures. \(\varDelta S _i = [\varDelta s _{i1}, \varDelta s _{i2}, \varDelta s _{i3}]^T\) denotes the sensor reading of FBG sensors in segment \(i \), which is defined as follows:

$$\begin{aligned} \varDelta s _{ij} = \varDelta \lambda _{ij} - \frac{1}{3}\sum _{j=1}^{3}\varDelta \lambda _{ij}, \end{aligned}$$
(2)

where \(\varDelta \lambda _{ij}\) is the wavelength shift of the FBG, \(i =\) I, II is the FBG segment, \(j = 1,2,3\) denotes the FBG sensors on the same segment. The insertion depth can be obtained from the magnitude ratio of the moment to the force:

$$\begin{aligned} d =l _i - \frac{||M _i||}{||F _\mathrm{{s}}||} \end{aligned}$$
(3)

where \(||\cdot ||\) denotes the vector Euclidean norm. When the magnitude of the sclera force \(F _\mathrm{{s}}\) is small, the insertion depth calculated using Eq. (3) can be subject to a large error, so Eq. (3) is only valid when the sclera force \(F _\mathrm{{s}}\) is bigger than a preset threshold (e.g., 10 mN).

Active interventional robotic system

Our proposed system consists of four main parts: the force sensing tool, an RNN, an admittance control algorithm, and the SHER research platform as depicted in Fig. 2. The tool held by the user is attached to the robot which can also manipulate it to intervene when needed. The scleral force and the insertion depth are measured by the force sensing tool in real time. These parameters along with the robot Cartesian velocities are recorded and fed into the RNN as input. The network predicts the scleral force a few hundred milliseconds away from the current moment. The predicted results are used to implement the admittance control. If the predicted forces are about to exceed the safe boundaries, the admittance control is activated and the robot makes an autonomous motion to reduce the forces.

RNN network design

RNNs are suitable for modeling time-dependent tasks such as a surgical procedure. Classical RNNs typically suffer from the gradient vanishing problem when trained with back propagation through time, due to its deep connections over long time periods. To overcome this problem, the LSTM model [16] was proposed, which can capture long-range dependencies and nonlinear dynamics and model varying length sequential data, achieving good results for problems spanning clinical diagnosis [17], image segmentation [18], and language modeling [19].

Fig. 4
figure 4

Proposed RNN. The network gets input from data history with h timesteps in past and outputs the probabilities of each force status at time t + n, where t is the current timestep. Then, the one with the highest probability is selected as the final force status, which is further fed into the admittance control

Full size image

We assume that the scleral force characteristics can be captured through a short time history of sensor measurements (e.g., the last few seconds). An LSTM network [16] is constructed to make predictions based on such history as shown in Fig. 4. The network is based on memory cells composed of four main elements: one input gate, one forget gate, one output gate, and one neuron with a self-recurrent connection (a connection to itself). The gates serve to modulate the interactions between the memory cell itself and its environment. The input gate determines whether the current input should feed into the memory, the output gate manages whether the current memory state should proceed to the next unit, and the forget gate decides whether the memory should be cleared. The following standard equations describe recurrent algebraic relationship of the LSTM unit:

$$\begin{aligned}&f _t = \sigma (W _f \cdot [h _{t-1} , x _t] + b _f ), \end{aligned}$$
(4)
$$\begin{aligned}&i _t = \sigma (W _i \cdot [h _{t-1} , x _t] + b _i ), \end{aligned}$$
(5)
$$\begin{aligned}&\tilde{C }_t = \phi (W _C \cdot [h _{t-1} , x _t] + b _C ), \end{aligned}$$
(6)
$$\begin{aligned}&C _t = f _{t}\odot C _{t-1} + i _t\odot \tilde{C }_t, \end{aligned}$$
(7)
$$\begin{aligned}&o _t = \sigma (W _o \cdot [h _{t-1} , x _t] + b _o ), \end{aligned}$$
(8)
$$\begin{aligned}&h _t = o _t\odot \phi (C _t), \end{aligned}$$
(9)

where \(h _{t-1}\) stands for the memory cells at the previous sequence step, \(\sigma \) stands for an element-wise application of the sigmoid (logistic) function, \(\phi \) stands for an element-wise application of the tanh function, and \(\odot \) is the Hadamard (element-wise) product. The input, output, and forget gates are denoted by i, o, and f, respectively, while C is the cell state. \(W_\mathrm{{f}}\), \(W_\mathrm{{i}}\), \(W_\mathrm{{C}}\), and \(W_\mathrm{{o}}\) are the weight for forget gate, input gate, cell state, and output state, respectively. \(b_\mathrm{{f}}\), \(b_\mathrm{{i}}\), \(b_\mathrm{{C}}\), and \(b_\mathrm{{o}}\) are the bias for forget gate, input gate, cell state, and output state, respectively. In this work, the LSTM unit uses memory cells with forget gates but without peephole connections.

To perform sensor reading predictions, a fully connected (FC) layer with softmax activation function as shown in Eq. (10) is used as the network output layer after the LSTM units, which outputs the normalized probabilities for each label.

$$\begin{aligned} y_i = \frac{e^{x_i}}{\displaystyle \sum \nolimits _{j=1}^n e^{x_j}} \end{aligned}$$
(10)

where \(x_i\) is the output of FC layer, \(y_i\) is the normalized probability, and n is the number of classes, i.e., force status.

The proposed RNN network takes the scleral force, the insertion depth, and the robot manipulator’s Cartesian velocities in past h timesteps as the input and outputs the probabilities of the future scleral force statuses, i.e., safe/unsafe t+n timesteps in the future, where t denotes the current time and n is the prediction time in future. Then, the one with the highest probability is selected as the final force status and is further fed into the admittance control. The groundtruth labels are generated based on the forces within a prediction time window \((t,t+n)\), as shown in Eq. (11):

$$\begin{aligned} label(t) = {\left\{ \begin{array}{ll}0 &{} F_x(t)^*<F_\mathrm{{gate}}\text { and }F_y(t)^*<F_\mathrm{{gate}} \\ 1 &{} F_x(t)^*<F_\mathrm{{gate}}\text { and }F_y(t)^*>F_\mathrm{{gate}} \\ 2 &{} F_x(t)^*>F_\mathrm{{gate}}\text { and }F_y(t)^*<F_\mathrm{{gate}} \\ 3 &{} F_x(t)^*>F_\mathrm{{gate}}\text { and}F_y(t)^*>F_\mathrm{{gate}} \end{array}\right. } \end{aligned}$$
(11)

where \(F_\mathrm{{gate}}\) is safety threshold of scleral force, which is set as 60 mN referred to our previous work [20]. Labels that are assigned as 0 represent safe status, otherwise unsafe status. \(F_x(t)^*\) and \(F_y(t)^*\) are the maximum scleral forces within the prediction time window shown as follows:

$$\begin{aligned} F_x(t)^{*}= & {} \mathrm{{max}}(|F_x(t)|,|F_x(t+1)|,\ldots ,|F_x(t+n)|), \end{aligned}$$
(12)
$$\begin{aligned} F_y(t)^{*}= & {} \mathrm{{max}}(|F_y(t)|,|F_y(t+1)|,\ldots ,|F_y(t+n)|) \end{aligned}$$
(13)

where \(|\cdot |\) denotes the absolute value.

Admittance control method

An admittance robot control scheme is proposed using the approach described in [21] as a starting point. During cooperative manipulation, the user’s manipulation force which is applied on the robot handle is measured and fed as an input into the admittance control law as shown in Eq. (14):

$$\begin{aligned} {\dot{x}}_{hh}= & {} \alpha F _{hh} , \end{aligned}$$
(14)
$$\begin{aligned} {\dot{x}}_{rh}= & {} A {} d _{g_{rh}}{\dot{x}}_{hh}, \end{aligned}$$
(15)

where \({\dot{x}}_{hh}\) and \({\dot{x}}_{rh}\) are the desired robot handle velocities in the handle frame and in the robot frame, respectively. \(F _{hh}\) is the user’s manipulation force measured in the handle frame, \(\alpha \) is the admittance gain tuned by the robot pedal, \(A {} d _{g_{rh}}\) is the adjoint transformation as shown below, and it is associated with the coordinate frame transformation \({g_{rh}}\).

$$\begin{aligned} A {} d _{g_{rh}} = \left[ \begin{array}{ccc} R _{rh} &{}\quad \hat{p }_{rh}{} R _{rh} \\ 0 &{}\quad R _{rh} \end{array} \right] \end{aligned}$$
(16)

where \(R _{rh}\) and \(p _{rh}\) are rotation and translation component of the frame transformation \({g_{rh}}\) and \(\hat{p }_{rh}\) is the skew symmetric matrix that is associated with the vector \(p _{rh}\).

The linear admittance control scheme is activated when the predicted scleral forces turn to the unsafe status. The desired robot handle velocities in Eq. (14) change as follows:

$$\begin{aligned} {\dot{x}}_{hh}= \alpha W F _{hh}+V \end{aligned}$$
(17)

where W is diagonal admittance matrices which can be set as \(diag([0,0,1,1,1,1]^T)\) and V is compensational velocity to reduce scleral force and it can be written as follows:

$$\begin{aligned} V = [sign(F_x)\cdot c,\quad sign(F_y)\cdot c,\quad 0,0,0,0]^T \end{aligned}$$
(18)

where c is set as a constant value.

The robot motion mode is switched back to the original control mode as shown in Eq. (14) when the predicted forces return to the safe status.

Experiments and results

Experimental setup

The experimental setup is shown in Fig. 5a and includes SHER, the force sensing tool, and an eye phantom. Besides, an FBG interrogator (SI 115, Micron Optics Inc., GA, USA) is utilized to monitor signals of FBG sensors within the spectrum from 1525 to 1565 nm at 2 kHz refresh rate. A microscope (ZEISS, Germany) and a monitor are used to provide magnified view. A Point Grey camera (FLIR Systems Inc., BC, Canada) is attached to the microscope for recording the user’s interaction with the eye. A shared memory architecture is implemented to integrate the RNN predictor running in python into the robotic control system running in C++. We focus on vessel following as a representative surgical task.

Force sensing tool calibration

The force sensing tool is mounted on SHER and used to collect the scleral force and the insertion depth. It is calibrated based on Eq. (1) using a precision scale (Sartorius ED224S Extend Analytical Balance, Goettingen, Germany) with resolution of 1 mg. The details of the calibration procedure were presented in our previous work [12]. The calibration matrices used in Eq. (1) are obtained as follows:

$$\begin{aligned} K _I = \left[ \begin{array}{lll} 0.1566 &{}\quad -\,0.2113\\ 0.1059 &{} \quad 0.2273\\ -\,0.2625&{} \quad -\,0.01605 \end{array} \right] , K _{II} = \left[ \begin{array}{ccc} 0.1853 &{} \quad -\,0.2315\\ 0.1114 &{} \quad 0.2515\\ -\,0.2967&{} \quad -\,0.0200 \end{array} \right] , \end{aligned}$$

The tool validation experiment is carried out with the same scale to test the calibration results. The validation results of the calculated force and the groundtruth values are shown in Fig. 3b, c. The measurement Root Mean Square (RMS) errors of the sclera force and the insertion depth are calculated to be 3.8 mN and 0.3 mm, respectively.

Fig. 5
figure 5

Experimental setup. a The sensing tool is mounted on the SHER end-effector, and the eye phantom is attached on a stage. b The eyeball is made of silicon rubber and is fixed into a socket. c The vessel following task consists six phases including approaching sclerotomy port, insertion, forward trace the curve, backward trace the curve, retraction, and move away from the eyeball

Full size image

Eye phantom

An eye phantom is developed using silicon rubber and is placed into a 3D-printed socket as shown in Fig. 5b. A printed paper with four curved lines representing the retinal vessels is glued on the eyeball inner surface. The curved lines are painted with different colors, and all lines intersect at the central point called “home” position.

Vessel following operation

Based on the recommendation of our clinical lead, “vessel following” which is a typical task in retinal surgery was chosen for the validation experiments. Vessel following can be performed using the eye phantom and is mainly comprised of 6 phases as shown in Fig 5c: (1) moving the tool to approach sclerotomy port, (2) inserting the tool into the eyeball through sclerotomy port to reach the home point, (3) following one of the colored curved vessels with the tool tip without touching the vessel, (4) tracing the curve backward to the home point, (5) retracting the tool to the sclerotomy port, and(6) move the tool away from the eyeball.

Fig. 6
figure 6

Data flow of the systems. Real-time prediction is n-\(\varDelta t\) time ahead, where n is the RNN prediction time and \(\varDelta t\) is the elapsed time in one prediction

Full size image

Shared memory

The RNN network runs in python at 100 Hz, while the robotic system runs in C++ at 200 Hz. To integrate the RNN network into the robotic systems, a shared memory architecture is used as the bridge to transmit the actual user operation data and network prediction results between two programs as shown in Fig. 6. Considering the elapsed time per single prediction \(\varDelta t\), the time of predicted results reaching robotic systems turns out to be \(T=t+n-\varDelta t\), where n is the prediction time.

Network training

The above-mentioned vessel following operation is performed 50 times by a SHER familiar engineer, and the collected data are used to train the RNN. The groundtruth labels are generated using Eq. (11). Successful training critically depends on the proper choice of network hyper-parameters [22]. To find a suitable set of the hyper-parameters, i.e., network size and depth, and learning rate, we apply cross-validation and random search. The learning rate is chosen as a constant number 2e−5, and the LSTM layer is set as 100 neurons. We use the Adam optimization method [23] as the optimizer and the categorical cross- entropy as the loss function. Note that for training the network, we cannot shuffle the sequences of the dataset because the network is learning the sequential relations between the inputs and the outputs. In our experience, adding dropout does not help with the network performance. The training dataset is divided into mini batches of sequences of size 500. We normalized the dataset value into the range of 0 to 1. The network is implemented with Keras [24], a high-level neural networks API. Training is performed on a computer equipped with Nvidia Titan Black GPUs, a 20-cores CPU, and 128 GB RAM. Single-GPU training takes 90 minutes.

The performances of different RNN networks are shown in Table 1, where accuracy is the true positive of prediction result, and successful rate is the prediction result excluding the false negative. Stacked LSTM model with the absolute value of the input data has the highest accuracy which is 89%, but its single prediction takes longer time which is 25 ms than LSTM model, and the latter takes 11 ms. We finally apply LSTM model in our experiments to get the best trade-off between prediction accuracy and timeliness. The chosen LSTM model obtains 89% prediction successful rate.

Table 1 Performance of the RNN networks
Full size table
Fig. 7
figure 7

Forces landscape with predictions. When “label = 1” is on, \(F_x\) in the imminent 200 ms would possibly go over the safety threshold \(F_\mathrm{{gate}}\). Same situation suits for “label = 2” and \(F_y\). The linear admittance control is activated to actuate the robot manipulator to reduce scleral force when label emerges

Full size image

Active interventional robotic system evaluation

The feasibility of AIRS is evaluated in real time by performing the vessel following operations. The research study was approved by the Johns Hopkins Institutional Review Board. Three non-clinician users took part in the study. With the assistance of AIRS, users are asked to hold the force sensing tool which is mounted on SHER to carry out vessel following task on the eye phantom. The operation is repeated 10 times by each user. Meantime, the benchmark experiments are also performed in freehand operation and in SHER-assisted operation, respectively.

With the assistance of AIRS, the force landscape in one typical vessel following operation is depicted as shown in Fig. 7. The tool tip trajectory in the robot coordinate frame during this operation is shown accordingly in Fig. 8. When the label turns to be on, the scleral force in the next 200 ms will possibly run over the safety threshold \(F_\mathrm{{gate}}\). At this moment, AIRS takes action to suppress the scleral force to prevent the unsafe forces from happening. The evaluation results of AIRS as well as two benchmark group are shown in Table 2. Four metrics are calculated including the maximum scleral forces, the unsafe force (force’s absolute value is larger than the safety threshold) duration, the total duration of the experiments, and the unsafe force proportion which is the ratio of the unsafe force duration to the total duration. It should be mentioned that the metrics are obtained from the conjunct data of ten trails for each user in each condition. The results show that the maximum forces, the unsafe force duration, and the unsafe forces proportion with AIRS are less than the ones with SHER for all three users and less than the ones with freehand for user 1 and user 2. The total duration with AIRS and with SHER is similar, but both of them are larger than the one with freehand.

Fig. 8
figure 8

Typical tool trajectory in vessel following operation. The six phases of the tool tip motion, i.e., approaching the eyeball, insertion, forward tracing the curve, backward tracing the curve, retraction, and move away from the eyeball are drew in different colors

Full size image
Table 2 Evaluation results
Full size table

Discussion

Robotic assistance could help in eliminating hand tremor and improving precision [7, 8]. However, it could affect surgeon’s tactile perception and in turn cause larger manipulation forces compared to freehand due to mechanical coupling and stiffness. Moreover, robotic assistant could prolong surgery time compared to freehand maneuver because a robotic assistant could restrict the surgeon’s hand dexterity. This is evident from our empirical results, where the fraction of unsafe forces with SHER are similar (for user 1) or even larger (for user 2 and user 3) than the ones with freehand, and the total procedure durations are longer with SHER than those with freehand for all users, as shown in Table 2. These results are consistent with our previous study [25]. These phenomena could potentially weaken the advantage of robotic assistance in retinal surgery and lower the enthusiasm to robotic technology for users who possess unique ability and skills to maintain steady motions with their hands (e.g, user 3).

However, the addition of our proposed AIRS system could remedy the limitation of the existing robotic assistant by capturing the user’s operation and eliminate potentially imminent extreme scleral forces before they occur. As shown in Table 2, the maximum forces and the unsafe force proportion with AIRS decrease sharply compared to the ones with robot and are comparable (for user 3) or less than (for user 1 and user 2) the value with freehand. These results demonstrate that AIRS could enhance safety of manipulation using robotic assistance.

We chose the LSTM model as the predictor in AIRS which obtained 89% prediction successful rate. Although the predictor has 11% failure rate to capture the unsafe scleral force in advance, which might lead AIRS to omit a small portion of unsafe cases, AIRS significantly enhances the existing SHER by keeping keep scleral force in a safe range as shown in Table 2. Currently, only the manipulation data from a SHER familiar engineer are used to train the network, which may lead to improper output for a surgeon’s operation. Therefore, a remaining limitation is that our system requires larger datasets from multiple users including surgeons and further the development of the learning models in order to provide trustworthy confidence intervals during inference. This will be the focus of future work.

Conclusion

We implemented AIRS by developing a force sensing tool to collect scleral forces and the insertion depth, used along with robot kinematics as inputs to an RNN network to learn a model of user behavior and predict the imminent scleral force unsafe mode, which are then used by an admittance control method to enable the robot to take action to prevent the sclera injury. The feasibility of AIRS is evaluated with multiple users in a vessel following task. The results show that AIRS could potentially provide safe manipulation that can improve the outcome of the retinal microsurgery. The presented method could also be applied in other types of microsurgery.

This work was only considered sclera forces. Future work will incorporate the predicted insertion depth together with the predicted scleral force to implement multiple variable admittance controls, to enable more versatile modeling of potential tissue damage. It is also critical to involve a larger set of users and different surgical tasks. Our force sensing tool can also measure the force at the tool tip, which will also be incorporated into the network training in future work to provide more information for robot control. Finally, while in this initial study we applied a linear non-smooth admittance control in AIRS as the first step to evaluate the system feasibility, nonlinear admittance control method will be explored in the future to achieve smooth robot motion.