Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Multi-fingered robot hands are favored end effectors of slave robot enabling the teleoperation systems interacting with remote environment flexibly. However, because of the multiple degrees of freedom in fingers, robot hand manipulation in unknown environments becomes a challenge [1–3]. Haptic glove (Fig. 1) is mostly chosen as a master device to control a robot hand and to provide force feedback to operator. While, most robot hand used in engineering is not humanoid hand, so the kinematic structures of haptic glove and robot hand are different, it is required to find intuitive grasp and force mapping methods [4, 5]. And there is still a design bottleneck for a reliable haptic glove that it is hard to find the ideal small actuator placed in limited human hand space [6]. Besides, many sensors and actuators in the haptic glove reduce the transmission rate of system which lowers the force feedback available to the operator [7]. Desktop haptic device is rarely used for robot hand manipulation although it is usually the master device for robot arm control in many teleoperation systems. In [8], the haptic device is used to position final desired grasp poses in the virtual environment for real robot to execute.

Fig. 1.
figure 1

Different haptic devices (a) and (b) used in three-fingered robot hand teleoperation.

Full size image

This paper presents a three-fingered robot hand teleoperation system based on desktop haptic device. The haptic device in the system is a parallel mechanism device with a ball handle (Fig. 1). It has similar kinematic structure with three-fingered robot hand and can provide stable and high frequency force feedback to operator. The three-dimensional device motions are intuitively mapped to the fingers and spread motion of robot hand. When operator grasping the handle of haptic device, it is just like the robot hand grasping objects.

During robot hand grasping, proper grasp force is critical for a safe and stable grasp (Fig. 2). Insufficient grasp force causes repeated pre-grasps. And excessive grasp force may deform even damage the objects. This problem is addressed in laparoscopic grasp control [9]. In order to control the grasp force of robot hand in teleoperation system, force feedback is necessary since the visual feedback is unreliable when grasping objects of different properties. In our system, the grasp forces of the robot hand are transformed to proper feedback force in haptic device. Operators control the three-fingered robot hand to safely grasp and hold different objects depending on the feedback force.

Fig. 2.
figure 2

The effects of different grasp forces.

Full size image

We raise two hypotheses related to the proposed system: (1) the operator can control the robot hand to grasp a wide range of objects based on the feedback force, (2) the proposed force feedback based grasping control is more reliable and effective than vision feedback control.

The main contribution of this paper is that we present an intuitive three-fingered robot hand teleoperation system by using desktop haptic device as a master device and propose a simple force feedback method to realize safe and stable robotic hand grasping. And the following section introduces our system design. Section 3 describes our experiment for the system and its results analysis. Section 4 presents our conclusion and future work.

2 The Robot Hand Teleoperation System Design

2.1 System Setup

Figure 3 shows the teleoperation system structure. The robot hand and haptic device are connected to networked computers respectively. Communication is realized by a UDP/IP connection in a local area network where time delay is neglected. The slave side receives the motion commands from the master side and sends the finger position and grasping force information back. A USB webcam is installed in slave side to capture video information of robot hand transmitted to the master side. The video information and a virtual robot hand simulating the real robot hand are displayed in the Graphical User Interface (GUI) for operator. The Graphical User Interface (GUI) is developed using VC++ and OpenGL library. The frequency of the haptic rendering and network transmission are both 1 KHz.

Fig. 3.
figure 3

The three-fingered robot hand teleoperation system.

Full size image

The three-fingered robot hand used in the system is BarrettHandTM BH8-280 (Barrett Technology Inc.). A strain gage installed about the distal joint of each finger measures the torque applied at the fingertip over a range of ±1 N-m. The maximum load at the tip of each finger is 2 kg. The haptic device used in the system is Novint Falcon (Novint Technologies Inc.) with the maximum Force Capabilities of 2 lbs.

2.2 Grasp Mapping

The kinematic structure of Falcon and BarrettHand are similar to some extent. When operator grasps the handle, it is just like the robot hand grasping objects. So we develop the grasp mapping between these two devices with the principle that the operation should be as simple and intuitive as feasible. There are only two kinds of basic motions for BarrettHand: fingers and spread close or open. Therefore, z direction movement of Falcon maps to the finger’s motion. Three of the four buttons on the handle represent three fingers respectively. Spread motion is mapped to y direction movements from intuition point of view (Fig. 4). Operators take control of the fingers or spread by pressing the corresponding buttons.

Fig. 4.
figure 4

Motion mapping of Falcon and BarrettHand.

Full size image

Repeated calibration is used in our mapping method to solve the problem of different workspaces. Operator releases the control of robot hand by pressing the button again and relocates Falcon device just like using computer mouse. However, in order to guarantee the continuity of the feedback force, this operation is invalid when the hand is contacting with objects.

The mapping relation of the haptic device position and the robot hand joint angle is as follows:

$$ P_{RHi} (k) = k_{p} \sum\limits_{t = 0}^{k} {\varDelta P_{Falcon\_z} } + P_{RHi} (0) $$
(1)

where P RHi (k) is the current robot hand position of the finger i (i = 1, 2, 3, spread) at k moment and P RHi (0) is the initial position of the finger. ΔP Falcaon_z is the z position (or y position for spread) difference of haptic device between current moment and previous moment. k p is the mapping coefficient which is 14.65 rad/m in our system. This value is calculated based on the workspace of haptic device and the robot hand. It is adjusted according to precision requirement of robot hand grasp.

2.3 Force Feedback

The strain gage properties of BarrettHand represent the torques applied on the fingers. The grasp force is calculated by these properties. The grasp force on the fingertip F RHi is calculated using the strain gage value S sgi (2), assuming the contact point is exactly at the fingertip. All the grasps in the experiment are fingertip grasps that only the fingertip can contact the object.

$$ F_{RHi} = k_{f} S_{sgi} $$
(2)

where i (i = 1, 2, 3) represents the fingers of BarrettHand. The unit of strain gage value obtained from the BarrettHand is count. The value ranges from 0 to 4000 and the initial value for each finger without load is about 2000. k f is a coefficient which converts the count number of strain gage value to grasp force.

The feedback force F Z in master side is calculated by averaging the grasp forces of three fingers.

$$ F_{z} = \frac{1}{3}\sum\limits_{i} {F_{RHi} } $$
(3)

The feedback force of haptic device is designed to one-dimensional force along the z direction during grasping control (Fig. 5). Operator feels the contact force feedback when the slave robot hand grasps an object. Based on the feedback force and previously learned knowledge in grasping, they control the robot hand to grasp objects.

Fig. 5.
figure 5

One-dimensional feedback forces.

Full size image

3 Experiment

User experiment compares the proposed force feedback based grasping with visual feedback based grasping in robot hand teleoperation (Fig. 6).

Fig. 6.
figure 6

The experimental scene.

Full size image

3.1 Participants and Task

Ten students, 6 male and 4 female in the laboratory, aged from 21 to 30, were invited to participant in the experiment. They are all right handed and familiar with the haptic device. All the participants never controlled a robot hand to grasp objects by using glove or haptic device before.

There are ten objects need to be grasped in the experiment for each participant (Fig. 7). These objects are common things in our daily life and their properties vary in a wide range. Their properties are well known by participants. Each participant needs to grasp the same object twice based on different feedback information.

Fig. 7.
figure 7

Objects in the experiment.

Full size image

The experimental task is using the haptic device to control the robot hand to grasp and hold the objects. All the grasps are fingertip grasp where the objects could not contact with the palm of the hand. Participants should first adjust the robot hand to a proper pre-grasp pose according to the video feedback. Then, they begin to grasp the object to find a proper grasp force to hold the object based on force feedback (F mode, no visual feedback) and visual feedback (V mode, no force feedback) respectively in random order. Finally, the robot hand is moved up and down to see if the object is stably grasped. During the process, any drops are not allowed. Because the robot hand has not been connected to the arm yet, we cannot change its position to reach an object. So we manually put the object close to the hand where it can grasp it through pose adjusting based on visual feedback.

3.2 Evaluation

After each grasp, the grasp force is recorded. The performance of grasp is evaluated by relative grasp force. Relative grasp force is calculated as follows:

$$ p_{relative\_gf} = \frac{{F_{g} - F_{\hbox{min} \_obj} }}{{F_{\hbox{min} \_obj} }} \times 100\% $$
(4)

where F g is the grasp force recorded each trail which equals to F z . F min_obj is the minimum grasp force necessary for robot hand to grasp the object. F min_obj of each object is obtained before experiment through increasing grasp force until no slippery is happened. The relative grasp force reflects the level of safety grasp and 10–30 % is regarded as safety margin for bare hand grasp [10].

3.3 Results and Discussion

Figure 8 shows the results of participants’ grasp forces based on visual and force feedback (Fig. 8). It can be seen that the force feedback based grasping shows a much smaller variance in grasping force for most objects. It can be concluded that the force feedback based grasping is more stable than visual feedback based grasping. We remove the outliers in our dataset and calculate the relative grasp forces for each grasp and the average relative grasp forces of each participant.

Fig. 8.
figure 8

Grasp forces based on visual feedback (a) and force feedback (b).

Full size image

Figure 9 shows the average relative grasp forces of each participant based on force feedback and visual feedback. The ANOVA results show that the differences of relative grasp force between force and visual feedback based grasping are significant (p = 0.00 < 0.05). Most of the relative grasp forces in force feedback based grasping control are smaller than the forces in visual feedback control. But it is the opposite results when grasping light and flexible objects (plastic bottle, paper cup, foam rubber) whose minimum grasp forces are lower than about 1 N. The reason is that the grasp forces for these objects are too small for operator to be perceived at the beginning in force feedback mode. But the object deformation can be easily noticed by participants through visual feedback.

Fig. 9.
figure 9

Average relative grasp forces for each participant.

Full size image

From the experiment results we get the conclusion that operator grasped a wide range of objects by using the proposed three-fingered robot hand teleoperation system. For most objects grasping, the force feedback based grasping control was proved more reliable and effective compared with visual feedback based control. These results demonstrated the hypotheses proposed in Sect. 1.

However, there is no participant who can successfully grasp all the objects in the experiment. For force feedback based grasping control, it is hard to grasp plastic bottle without deformation. The grasp force in robot hand is calculated by the strain gage properties. Since the finger itself has gravity, this torque value is not zero when the fingers close or open without load. So the grasp force cannot be distinguished when grasping materials of large elasticity in our system. When participant felt the force, the objects already had deformation. This also explained why the average of grasp forces for the flexible objects using force feedback based control are larger than the forces using visual feedback based control. For visual feedback based grasping control, the problem of excessive grasp force is inevitable especially grasping stiff objects. Operator relies more on the force feedback to control the robot hand grasp force which is similar with human hand grasp. So the force feedback based control is necessary. For engineering application, the two feedback modes can be combined to complement each other.

4 Conclusion and Future Work

This paper presents a three-fingered robot hand teleoperation system to grasp objects in unstructured environment based on force feedback, by using desktop haptic device as the master manipulator. The grasp mapping and force feedback methods are developed for this system. In particular, we transform the grasp forces of the robot hand to one-dimensional feedback force in master side. The system is intuitive and easy-to-realize for robot hand teleoperation application.

To evaluate the effectiveness of our system, an experiment was conducted in which subjects were instructed to grasp 10 objects of different stiffness, based on force feedback and visual feedback respectively. Our results demonstrated that the subjects grasped a wide range of objects with a proper grasp force without slippage or damage by using the proposed three-fingered robot hand teleoperation system. The force feedback based grasping control was proved more reliable and effective compared with visual feedback based control. However, the results also showed that not all objects in the experiment can be safely grasped because of the effect of force feedback accuracy.

In future work, we will further evaluate the system on the effectiveness of distinguishing the objects with different materials and weights, and how difficult it is for operator to adapt this kind of force feedback. Furthermore, we are interested in the difference of grasp forces when people grasp objects with teleoperated robot hand and the bare hand. This will be an important basis to design proper force feedback to improve the transparence of robot hand teleoperation.