Keywords

1 Introduction

In recent years, there has been remarkable progress towards more direct collaboration between humans and robots, enabled by technological advances in robot hardware [1]. The current trend of Industry 4.0 envisions shared environments, where robots interact with their surroundings, humans, and other agents [2]. In addition, the recent COVID-19 pandemic has increased the demand for autonomous and collaborative robots in environments such as care homes and hospitals [3]. Human workers can potentially benefit from robotic assistants due to several advantages, such as transferring repetitive, low-skill, and ergonomically unfavourable tasks to robots. In this context, robot handover is of great significance [4].

Fig. 1.
figure 1

Vision-based tactile sensor for handover framework

Full size image

The human-robot handover task has been studied in several works. One of these pioneering works is [5], in which the authors proposed a system that comprises a three-finger gripper attached to a robot that makes decisions regarding when to open the gripper, close it or adjust one of the fingers in case the contact between the gripper and the object is lost. This approach uses a combination of information regarding joint angles, contact with the object, and kinematics to assess the grasping stability. The authors showed that handover is closely related to grasp stability since the robot must hold the object until the human is ready to receive it. In contrast, Edsinger and Kemp demonstrated that humans adapt how they hand objects over to a specific configuration of the robot’s gripper [6].

There are handover approaches based on using the user’s hand velocity to determine when the robot should release an object [7]. However, visual information can be unreliable if the object is occluded or the lighting conditions are not appropriate. Besides that, some objects, such as soft or irregularly shaped objects, may be challenging to hand over. On the contrary, tactile sensors provide more precise and accurate information to the robot, enabling it to adapt according to the object’s physical properties. This allows the robot to perform the handover of a broader range of objects with varying shapes, sizes, and textures and ensures a more robust grasp of an object whose surface friction is unknown [8]. Despite the benefits of tactile sensing and its implementation under various operation principles (e.g., pressure, vibration, temperature, texture, or shape), the use of vision-based sensors for human-robot handover tasks has not yet reached its full potential.

In this paper, we present a vision-based tactile sensor for object handover frameworkFootnote 1, aiming to solve the problem of coordination and timing in human-robot handover. The framework is based on a sensor comprising a latex rubber layer with a marker on the interior surface, a camera, and a chamber with variable pressure. The contact movement is estimated by tracking the marker on the latex skin’s surface. This information is then used to decide if opening the robot’s gripper is required. We investigate several aspects of the framework, such as success rate with an exploratory user study in which ten participants complete the task of handing over eight objects. Each object has a different size, shape, and stiffness. For our experiments, we built an interface for experiments that allows the user and the robot to interact during the handover tasks. Our contributions are summarised as follows: (i) a framework that allows human-robot handover with a vision-based tactile sensing principle and (ii) a method for calibrating the sensor’s sensitivity based on the user’s experience.

The rest of the paper is organised as follows. Section 2 discusses related work on tactile sensors and human-robot handover. Then, Sect. 3 presents our vision-based tactile sensor for the object handover framework. In Sect. 4, we explain the experimental setup used to validate the framework. Section 5 discusses the results. Finally, we conclude this paper and propose potential future work in Sect. 6.

2 Related Work

A handover is defined as a collaborative joint action in which one agent (the giver) gives an object to another agent (the receiver). The physical exchange begins when the receiver first touches the object held by the giver and ends when the giver completely hands over the object to the receiver. Human-robot handover, a frequent collaborative action among humans, requires a concerted effort of prediction, perception, action, learning, and adjustment by both parties. Implementing an object handover that is as efficient and fluent as the exchange among humans is an open challenge in robotics [9].

There is a significant amount of literature on human-robot object handover with the potential to enhance robot capabilities performed from a range of aspects, such as visualising robot intent [10], adaptiveness to user preference [11], visual perception for handover [12], affordance-based handover [13], and gripper effort control [7]. These approaches are based on specific aspects, such as the human hands, objects, contact points, and contact pressure. On the other hand, our framework does not depend on contact force estimation. Instead, it utilises the direct measurement of the object’s movement by tracking the dot placed on the internal face of the layer.

Among the different robotic applications and tactile sensing types, slip detection is a typical application [14]. An important early approach in this area was proposed in [15], where the authors describe grasping behaviours and grip forces. Similar vision-based optical tactile sensors have been studied in [16, 17], with the purpose of detecting slip. For example, the sensor proposed in [18], covered with an opaque surface skin made of latex, provides slip detection by producing high-resolution force arrays. Another popular device for tactile sensing is the BioTac sensor [19]. For example, in [20], two BioTac sensors are used to detect slip. BioTac sensors are also used in [21], where the authors defined three types of tactile estimation: finger forces-based, slip detection-based, and slip classification-based.

Despite the success of the aforementioned approaches, the most commonly used object shapes to test are cylindrical (i.e. bottle) or rectangular (i.e. box). These objects are easier to hand over than deformable and fragile objects; Hence, the generalisation of handovers to a variety of objects with different shapes and stiffness has not yet reached its full potential.

3 Vision-Based Tactile Sensor for Object Handover Framework

In this section, we present the vision-based tactile sensor for object handover framework (Fig. 1), which aims to solve the problem of human-robot handover. The framework comprises the following modules: segmentation, dot tracking, and decision.

The segmentation module’s input is the video stream image captured by the camera inside the sensor. This module is in charge of segmenting the dot printed at the centre of the latex layer. For this purpose, the original image is transformed into a negative version of it and then into grayscale. This module is implemented using Python and OpenCV.

The dot tracking module utilises the output of the segmentation module as input and transforms the image into coordinates of the white dot’s centre. First, the dot tracking module sets an initial position once the gripper has closed and is holding an object. Then, the centre coordinates of the dot moving with respect to the initial position are stored in the vector v, which is the output of this module.

The decision module controls the sensitivity-pressure pair of the sensor and the open and close actions of the gripper. The accumulated displacement of the object being pulled by the user while the gripper holds it is given by the following :

$$\begin{aligned} M=\sum _{i=T-n}^{T} \left| \frac{||v_i-v_{(i-1)}||}{ (t_i-t_{(i-1)})}\right| , \end{aligned}$$
(1)

where \({t_i}\) and \(t_{(i-1)}\) denote the current and past time, and \(v_i\) and \(v_{(i-1)}\) are the current and past positions of the dot, respectively. Here, i takes values within the range \((T-n,..., T)\). The value of M increases proportionally to the accumulated displacement of the dot, and the bigger the value of n, the more past information is considered. The decision module can decide if opening the gripper is necessary based on the following:

$$\begin{aligned} {\begin{matrix} Gripper_{action}= {\left\{ \begin{array}{ll} \text {Open}, &{} M>S\\ \text {No action},&{} \text {otherwise} \end{array}\right. } \end{matrix}} \end{aligned}$$
(2)

where S is the sensitivity and acts as a threshold that determines when the robot should release an object.

Fig. 2.
figure 2

The image displays the experimental setup in which the robot is handing over a cup to a user.

Full size image

4 Methodology

To investigate the optimality of the sensitivity-pressure pair that is more comfortable from the user’s perspective when using our framework, we conducted an experiment in which users rated how easy or difficult it was to hand over an object from the robot. Additionally, users rated the degree of damage the object sustained after completing the task. The time taken by the user to attempt to take the object from the gripper until it was released (sensing time) was also measured for all the handover tasks. The experimental setup (Fig. 2) consisted of a KUKA® LBR IIWA 14 robot arm with 7 degrees of freedom and a Robotiq® 2-finger gripper with a vision-based optical tactile sensor attached to its fingers. Moreover, the user followed instructions displayed on a screen placed next to the robot, indicating when to place or take the object from the gripper.

4.1 Participants

We asked 10 participants from Cardiff University, including eight males and two females aged 24–30, to do the experiments. There was no compensation for the participants. Among them, 2 participants had previous experience with robots, while 8 participants had never interacted with a collaborative robot.

Fig. 3.
figure 3

A human and a robot are performing a handing-over task with raspberry (a) and dough (b) during the experiments. In (a) and (b), when the colour of the line is red, the gripper is closed. The yellow colour indicates that the user is trying to get the object. The green colour indicates that the gripper is open. (Color figure online)

Full size image

4.2 Experimental Procedure

The experiment took place at the Robotics Lab of Cardiff University under the supervision of our 2 experimenters. Participants stood in a designated position before a robotic arm and started the object handover task. Participants first read the instructions and then signed the consent form. After reading the instructions, the experimenter provided information about the experimental process by reading from a script and collected basic demographic information, such as gender, through a short questionnaire.

Table 1. Parameters used for the sensitivity-pressure pairs during the experiments.
Full size table

For the experiments, we set nine sensitivity-pressure pair values, as shown in Table 1, and used eight objects with different shapes and stiffnesses (paper, fabrics, a cup, dough, strawberries, raspberries, a cable, and a prism). Aiming to investigate several aspects of our framework, we designed a questionnaire (Table 2) that consists of two questions. Q1 has a scale of 0–10, where 0 - too sensitive, 5 - ideal, and 10 - too difficult. Q2 has a scale from 0–10, where 0 - the object is intact, and 10 - the object is damaged. The participants perform the following experiments:

  1. 1.

    Each participant performs the handover tasks in a random order without knowing the sensitivity-pressure pair values.

  2. 2.

    After each handover, the participant answers Q1 (Table 2) for the object in turn.

  3. 3.

    After each handover, the participant answers Q2 (Table 2) for the object in turn.

  4. 4.

    If the robot fails to hand over the piece, the participant marks the experiment as invalid and is not required to answer questions Q1 and Q2.

For each experiment, in turn, we aim to answer the following research questions:

  1. 1.

    Despite the random order of objects and sensitivity-pressure pairs in which the participants are asked to perform the handover tasks, is it likely that the participants will agree on which sensitivity-pressure combination results in a more pleasant handover?

  2. 2.

    Which sensitivity-pressure pair produces a more pleasant human-robot handover from the participant’s point of view?

  3. 3.

    Which sensitivity-pressure pair reduces or increases the damage to the objects after handing them over?

  4. 4.

    What is the success rate of the proposed framework?

To validate the questionnaire, we conducted a pilot study at the beginning with three participants who carried out the experiments and answered Q1 and Q2. For this analysis, the Cronbach’s alpha value of each question was calculated, such that the statistical output was 0.775 and 0.852 for Q1 and Q2, respectively. Since both values are greater than 0.7, the questions are considered good [22].

Table 2. Questionnaire used for the exploratory user study.
Full size table

The described experiments involve the use of a screen placed on the side of the participants to provide instructions for handling various objects, as shown in Fig. 2. Once the participants verbally indicate they are ready, the researchers manually start the robot program. The positioning of the screen is strategic, as it allows the participants to receive clear visual guidance while completing the handover task (see Fig. 3). The screen will show red, indicating that placing an object is unsafe and may cause damage or harm. In this case, the participant will be instructed to refrain from placing the object and wait for further instructions. When the user indicator displays orange, it signifies that the participant can place the object between the gripper. Once the object is in place, the experimenter can close the gripper, and the participant can continue the task. Once the indicator displays a green signal, the participant can attempt to take the object.

Moreover, the instructions are essential to ensure that the participants follow the correct procedure for each object, which is critical to ensure the safety of the objects and the participants involved.

Fig. 4.
figure 4

This figure displays the results of each sensitivity-pressure pair for Q1 and Q2, as well as the sensing time for all objects. The box plots are displayed in the upper row, while the means are shown in the bottom row.

Full size image

5 Results and Discussion

5.1 Results

We have organised the data in a hierarchical order to evaluate the experimental results. First, we sorted the participants’ ratings for Q1, Q2, and the sensing time for all the objects. Then, we analysed the results of Q1 by considering each object type. Lastly, we examined the findings of Q2 by considering the different categories of objects.

Table 3 summarises the results of Q1, Q2, and the sensing time for all the objects (Fig. 4). For Q1, the results show that the participants found the handover task more comfortable for \(S_3\). At the same time, the score deviates from 5 (5 being the ideal score as defined in the questionnaire), which indicates that the scores react proportionally to the value of S(for example, \(S_1P_2=6.28\) and \(S_3P_2=5.27\)). At this point, the pressure value P does not seem to have a significant impact on the participants’ sensation of fluency during the handover tasks. The reason behind this behaviour is that the latex layer maintains its elasticity regardless of pressure changes, which does not affect the marker’s movement. For Q2, the participants scored higher values that correspond to higher pressure values, with the exception of \(S_3P_1\), \(S_3P_1\) and \(S_3P_1\). This is because \(S_3\) provokes the robot to open the gripper faster and produce less damage to the object. In terms of sensing times, the participants perceived a more comfortable handover when the time is below 500 ms. However, the high values for the standard deviation indicate that the time measurement requires improvement.

Table 3. Results of each sensitivity-pressure pair for Q1 and Q2, as well as sensing time for all objects and participants.
Full size table

In Table 4, the participants’ ratings for Q1 with respect to each object are summarised. Among all the objects, raspberries and cables received the highest scores for being difficult to hand over. On the contrary, the rest of the objects obtained a rating of around 6. For all objects (Table 3), the score is closer to 5 when the sensitivity value S is lower.

Table 4. Results of each sensitivity-pressure pair for Q1 obtained in the experiments.
Full size table

Table 5 shows the participants’ ratings for Q2 with respect to each object. It can be observed that as the pressure increases for P, the participants tend to rate a higher value for Q2, indicating that the object is more damaged after the handover. Among all the objects used during the experiments, raspberries had the highest rates of damage, suggesting that the handover task using our approach is more challenging to execute with a fragile object like a raspberry.

Table 5. Results from the experiment for every sensitivity-pressure pair for Q2.
Full size table

In terms of success rate (Table 6), the cable got the lowest success rate, followed by the Raspberry with 86% and 90%, respectively. At the same time, the rest of the objects obtained a similar success rate of above 95% despite their shapes or sizes.

Table 6. Success rates of the proposed framework for each object and the total.
Full size table

5.2 Discussion

Based on the above results, it was found that, from the participants’ perspective, a lower value of S (\(S_3=0.001\)) produces a more comfortable handover. Additionally, a higher pressure value of P caused more damage to the objects after being handed over. However, there is no clear indication that the change in pressure enhanced the user’s sensation of fluency while handing over the objects during the experiments. In terms of success rates, our approach encountered more difficulty with reduced-diameter objects, such as the cable, because, depending on the object’s initial position while being grasped, the dot at the centre of the latex layer did not move as it did with the other objects. As a consequence, the dot-tracking module could not detect any movement of the object. Another challenging objects were the raspberries because, among all the objects, the participants noticed damage to them after performing the task, which also reflected in the success rate. Despite the shortcomings, the robot using our framework managed to hand over different objects with different sizes and stiffness with a 94% success rate.

6 Conclusion and Future Work

In this paper, we proposed an object handover framework using the vision-based tactile sensor. Overall, the experiments achieved a success rate of 94%. The exploratory user study revealed that users found the handover task more comfortable when the sensitivity value was lower. The limitations of this paper, found during the experiments, are related to reduced diameter objects (e.g., cables) and highly fragile objects (e.g., raspberries), which seem to be the most challenging objects to hand over using our approach. However, the reliability of our framework and sensor was demonstrated for handing over objects such as paper, fabrics, dough, and strawberries. For future work, we plan to improve the sensor by adding more markers to the latex layer and exploring the effects and benefits of using tactile sensing with variable pressure. Additionally, to better evaluate the performance improvement achieved by our approach, a comparison with tactile-only methods is necessary. By addressing this issue, we aim to enrich our research and establish a robust foundation for future advancements in tactile sensing systems for human-robot interaction.