Wiping the mouth using the robot arm with information of RGB-D sensor

Abstract

In recent years, research and development of robots in the field of welfare has been actively conducted. This study focused on a meal support in welfare. For healthy subjects, the action on meals is simple, but also includes difficult movements for those with impaired upper limbs. Furthermore, in recent years, it has become difficult to support each person due to the lack of manpower at welfare sites. Therefore, it can be expected that further support robots will be needed in the future. A meal support is not all that carries food. There is a need to support before and after that. For example, making it easier to swallow food, making food easier to eat, attaching seasoning, wiping off stains on the mouth are also among the dietary support. This study focuses on support before and after meal support. In this study, we aimed to promote swallowing gymnastics before meals to make it easy to swallow food, and to do the work of wiping the mouth after eating with a robot arm.

1 Introduction

In recent years, research and development of robots in the field of welfare has been actively. For example, some power assist systems are proposed for heavy physical work or walking. In this study, we focused on a help with eating by robot arm. The task of eating is necessary to survival. For healthy subjects, a motion on meals is easy. However, it is difficult to make the eating motion for handicapped person or elder person. They need to help and support to eat every day. It takes many time and the burden of nursing care. Furthermore, in recent years, it has become difficult to support each person due to the lack of manpower in the field of welfare. It can be expected support by robots. The support robots are proposed that carried some food to the mouth in the researches on the support robots and mechanisms [1]. However, we thought that task of eating support includes not only the carrying the food, but also cutting the food to eat easy, the mouth exercises before eating and wiping the stains around the mouth after meal. In this report, we propose the help with eating before and after meal supporting by robot system. We aimed to promote swallowing exercises before eating foods. It relaxes the muscles of the face and neck, then person can swallow food easy. In the previous research, wiping the mouth by robot system after eating was presented [2]. A technology of relating to feature quantity extraction in image [8, 10]. In addition, research of face recognition by camera [7, 9] is developed. We detect the mouth position in the visual information using the one of the program of “face-tracking function” of a KINECT. The KINECT is one of the RGB-D sensors which can output the depth information and color information simultaneously. Development of recognition system using Kinect with such functions [11]. This function outputs the face position, direction, and some feature points on the face. The proposed robot system uses these information. It is performed according to instructions of motion by displaying the arrows on the front monitor in swallowing exercise before eating the meal. For mouth wiping motion, the proposed system finds the mouth position using face-tracking function of KINECT. The robot-desired positions are given these sensor information. Then, the desired angle of each motor calculated by inverse kinematics [3]. In addition, the robot arm wipes out the stains moving around the mouth position. The subject performs the final fine adjustment and wipes the mouth. Then, we propose an interactive cooperative system between a person and a robot. We confirm that these proposed support systems are useful by the experiment results.

Fig. 1

RGB-D sensor of KINECT v2

Fig. 2

Representative point of target

2 Information of RGB-D sensor and motions

2.1 Color recognition and position information

In this research, we adopted the system configuration with KINECT v2 (Fig. 1). In the depth sensor of KINECT v2, the measurement method by Time-of-Fright (TOF) method is adopted. In addition, the resolutions of RGB and depth have also increased, and the performance has improved from KINECT v1. The system constructed with some software and libraries, “Kinect for Windows SDK” distributed by Microsoft and used “Open CV” for visualization of data obtained from each sensor [4]. In this experiment, the origin of coordinate system was set the RGB camera viewpoint of KINECT. We set the some assumptions as follows.

• The red handkerchief means the dry cloth.

• The green handkerchief means the moistened cloth.

• A blue vinyl tape is wrapped around the robot hand.

Figure 2 shows the examples, the blue area means the robot hand position, red area means the dry cloth, and green area means the moistened one. This proposed system detects the positions of some targets using the color recognitions. Color information is acquired for each pixel. The color within the screen area is judged by preset R, G, and B thresholds. The R, G, and B are represented by a plurality of points, respectively. Calculate the average of the pixels at the plurality of points detected from the threshold information of the color to be detected and make it representative point. Acquire the position from the representative point. A representative point of each color area was decided to calculate the average of point group. A position information, x and y, are calculated by the pixel number of representative point. The distance per pixel is given by the depth information of representative point. The position information of x and y are developed for pixel number. In addition, the position information of z is given from depth data of KINECT.

Fig. 3

Output of face-tracking function

Fig. 4

Output of HD face tracking

2.2 Face tracking

“Face-tracking function” detect the human face when there is an entire face within measurable area. This software detects and follows the face position. This function is configured to find outline of face and some parts of face, such as the eyes, mouth, and nose. The KINECT provides two type face-tracking systems, as shown in Figs. 3 and 4, i.e., “face tracking” and “HD face-tracking” functions. This function can be selectively used depending on the application. In HD face tracking, 1347 feature points on a face can be acquired.

Since this study aims to achieve the wiping the mouth, we pay attention to feature points of the mouth part. In this experiment, we do the two type motions of wiping mouth. One motion is the wiping the mouth with moving from left side of mouth to right side. This method finds the two points, left side of mouth and right side of it, in the feature points. Another motion is the wiping the mouth with making circular motion around the mouth. This method finds the four points, left, right, upper, and bottom points of mouth.

Fig. 5

Face tracking and outputs

Figure 5 shows the results of a preliminary experiment of face recognition. The characters displayed under the face in the image, which display the plot of the feature point and the result of the face recognition. In the face recognition program, the five feature points, eyes, nose, left, and right mouth corners, are detected. Then, it calculates the face angles of roll, pitch, and yaw based on these points. Furthermore, the distance from the RGB-D camera to the face was determined by depth information of mouth corners. By bringing the robot hand at right angles to the position of this mouth corner, we make a motion to bring a handkerchief to the mouth. Although the error of the position information has been within $\pm 20$ [mm], in this experiment, it is aimed at cooperative work of a subject and a robot. It has no problems, because the final adjustment is done by subject moving.

2.3 Swallowing exercise

As a support before eating meals, exercises are warm-up for eating and swallowing food [5]. This exercise is what is commonly done before meals at the nursing home etc. In this study, we incorporated this swallowing exercise in help with eating as a support of before eating. Swallowing exercise has various kinds of motions such as neck exercise, shoulder, arm, and vocal exercise. In this research, we focused on instructions on the warm-up the movement of the neck muscle. The proposed system shows the arrows which mean the direction of next movement. The subject moves his neck with instruction arrows. This system confirms that changing of face direction using the “face-tracking function”. If he did not make the neck motion adequately, this system shows the same direction arrow again. These situations of swallowing exercise are shown in Fig. 6. From this result, we confirm that the subject is moving her head for warm-up the neck muscle.

Fig. 6

Swallowing exercise using the RGB-D sensor

2.4 Wipe the mouth

The way you wipe your mouth varies depending on how you wipe. Therefore, as a method of wiping the mouth, we performed a motion to wipe the mouth by reciprocating to the left and right and a motion to wipe the mouth so as to draw an arc [6]. Prepare two handkerchiefs of different colors. Using color recognition, it detects the handkerchief is wet or not. In mouth wiping motion, at first, robot arm uses wet green handkerchief, then dry red handkerchief to wipe around the mouth. The robot arm uses these mouth points as the desired positions. The experimental robot composed five motors. The five motors are numbers ID1–ID5. The four motors, ID1–ID4, are used to control the position of end effector. One motor of ID5 is used to open or close the hand.

2.5 Voice recognition

The wiping motion is started to call the “start” by voice recognition function. The microphone uses the microphone array mounted on KINECT. Activate the arm with the word “start”. In addition, when activating by speech recognition, malfunctions due to people’s voices can be considered. The check items were created to prevent malfunction to confirm the speech direction and specify the speaker. After that, confirm that the speaker is facing front with respect to KINECT. The eyes are open, and the meal is finished. This is evaluated by the mouth is moving or not. These check items prevented malfunctions due to the voices of surrounding people. In the environment, where speech recognition is performed, assuming the daytime of the room, the music which is about 50 (dB) right under the KINECT. This volume was measured with a decibel gauge.

2.6 Experiment environment

It is intended for help with eating, people sited on a chair. We assume that a person who needs some help with eating has handicapped. In this experiment, it was used a wheelchair indeed of a chair. KINECT is fixed on a tripod and it is placed in front of the subject. The angle of KINECT shall be horizontal. On the left of KINECT, set a monitor to use the swallowing exercises. Fix the decibel gauge which measures the volume right under KINECT. A robot arm fixed to the base of the tree board. The robot arm was placed on the right side a subject. A rubber mat of 2 (mm) in thickness was laid down for vibration absorption and displacement prevention. Let the boundary between this rubber mat and the table be the initial position of the arm. Figures 7 and 8 show the arrangement. The handkerchief finished to wipe storage container was secured on the rubber mat. They placed, as shown in Fig. 9. When wiping is completed, it stored the handkerchief in each zone. Since each handkerchief is recognized by color information, it has no need to set the condition in the left or right place. The positional relationship between the KINECT and the robot arm is shown in Figs. 7, 8 and 9.

Fig. 7

Front view of experiment desk

Fig. 8

Side view of experiment room

Fig. 9

Position relationship between KINECT and robot

3 Experiment

3.1 Outline of experiment

Using the KINECT, as shown in Fig. 1, positions of x, y, and z (mm) of the mouth are acquired. Based on the position, the joint angles of robot are calculated from mouth position by inverse kinematics, where link parameters are defined in Figs. 10 and 11. Joint angles, $\theta _1-\theta _4$, are calculated as follows.

Fig. 10

Robot arm

Fig. 11

Link parameters

$$\begin{aligned} \theta _1= & {} \arctan 2(P_y, P_x), \end{aligned}$$

(1)

$$\begin{aligned} \theta _2= & {} \frac{\pi }{2} - \phi - \alpha , \end{aligned}$$

(2)

$$\begin{aligned} \theta _3= & {} \pi - \beta , \end{aligned}$$

(3)

$$\begin{aligned} \theta _4= & {} \frac{\pi }{2} - \theta _2 - \theta _3. \end{aligned}$$

(4)

Here,

$$\begin{aligned} \alpha= & {} \arctan 2\left(\cos {\alpha }, \sqrt{1-{\cos {\alpha }}^2}\right), \\ \cos {\alpha }= & {} \frac{{L_3}^2+(P_0 P^{\prime })^2-{L_4}^2}{2{L_3}(P_0 P^{\prime })}, \\ \beta= & {} \arctan 2\left(\cos {\beta }, \sqrt{1-{\cos {\beta }}^2}\right), \\ \cos {\beta }= & {} \frac{{L_3}^2+{L_4}^2-(P_0 P^{\prime })^2}{2{L_3}{L_4}}, \\ \phi= & {} \arctan 2\left(\sqrt{{P_{x}^{\prime }}^2+{P_{y}^{\prime }}^2}, P_z-L_1-L_2\right). \end{aligned}$$

As an action plan, after grasping the cloth, move only the ID 1 motor and rotate the arm. Pass the defined space, then output the joint angle obtained by calculation, bring the cloth closer to the mouth. All places to start wiping are set as the right end of the mouth. When wiping around the mouth, move the defined point around counterclockwise.

The robot arm waits for inputting the starting words at the initial position. When he says the starting word, the position of the mouth and the cloth is calculating from KINECT’s position information. Then, it is confirmed whether or not the mouth and the cloth exist within the movable area of the arm. When confirmation is completed, the robot arm moves to the operation of grasping the cloth. In the beginning grab and hold the green handkerchief, which is a cloth moistened with water. Then, by inverse kinematics calculation from the position information of the mouth, bring the hand tip of the robot arm closer to the right end of the mouth. When reaching the target position, shift to mouth wiping operation. After that, it separates from the mouth, and a green handkerchief is stored on the back side of the container.

Then move on to grasp the red handkerchief which is a dry cloth. Just grab it up like a water wipe, bring the hand tip of the robot arm close to the right end of the mouth. Moving to mouth wiping operation when reaching the target position. After that, it separates from the mouth, the red handkerchief is stored on the front side of the container, and the arm returns to the initial position and stops. Ring the alarm calling caregivers at the same time as stopping.

3.2 Experiment results and discussion

Fig. 12

Experiment flow

Table 1 Results of preliminary experiment about subject A

Table 2 Results of three linear reciprocating motion wiping out the stains of cake

Figure 12 shows the experiment flow of wiping motion. By this figure, we explain the proposed mouth wiping system using the RGB-D sensor.

• (a) At first, when she finished the meal, she turns to the front and say “start”.

• (b) The support system check the mouth and handkerchief position which are within the movable range of the robot arm by the RGB-D sensor information. Here, we limit the depth of mouth position between 500 and 750 (mm) for safety. The depth is under the 500[mm] too near the robot arm. In addition, it is over the 750[mm], then robot arm may touch the body of subject.

• (c) Robot arm moves the represented point on wet handkerchief by inverse kinematics.

• (d) Robot arm moves to the mouth position by inverse kinematics.

• (e) The robot executes the primitive motion, round motion, or reciprocating motion, for wiping the mouth.

• (f) Release handkerchief from mouth.

• (g) Store the handkerchief in the box.

• (h) Robot arm move the next handkerchief position.

• (i) Robot arm moves the represented point on dry handkerchief by inverse kinematics.

• (j) Robot arm moves to the mouth position by inverse kinematics.

• (k) The robot executes the primitive motion, round motion, or reciprocating motion, for wiping the mouth.

• (l) Release handkerchief from mouth.

• (m) Store the handkerchief in the box.

• (n) Robot arm move the primary position, and the support system signals the buzzer.

Table 1 summarizes preliminary experiments conducted on Subject A. In the table, “$\times$” means leave to wipe off, “$\triangle$” means that oil stain remains, although there is no visible wipe remaining, “$\bigcirc$” means that wiping can be carried out firmly. Preliminary experiments prepared three kinds of food, cake, curry, and spaghetti. As a movement, we made the two type primitive motion. One is the to wipe the mouth moving a circle in counterclockwise around the mouth. Another motion is the moving the linearly toward the mouth corners. In addition, I tried the operation to repeat from 1 to 3 times. From the questionnaire of subject “A” that the discomfort remained after wiping was finished, although the one with strong oil stain like curry and spaghetti is wiped out in appearance. It was more effective to repeat the operation, and it was confirmed that sufficient wiping was obtained in about three times.

Based on this result, wiping experiment was further carried out for subjects B–F with cake and wiping action with three reciprocating movements between left and right mouth corners. The results are shown in Table 2. It was confirmed that 6 people including A in the preliminary experiment can wipe off the cake stain with three reciprocating movements.

We asked three men in their teens and 20s to eat the cake and wiped the stains of the mouth. From this experiment, contamination around the mouth before and after mouth wiping of subjects A, B, and C were confirmed. The photographs are shown in Figs. 13, 14 and 15. Comparing the figures before and after the experiment, it can be seen that the cream stains are gone. There were many opinions that stain on the cake had no feeling bad after wiping out stain on the appearance.

4 Summary

In this experiment, we developed a help with eating system focused on support before and after meals. In swallowing exercise before meals, it was possible to display the arrow defined by the movement of the face, and as a result of the face detection by KINECT, it was possible to detect the movement of the face and switch the instruction. In this experiment, it was only movement of the neck up and down, left, and right, but it is thought that more effective pre-meal exercise can be implemented by incorporating exercises other than the neck.

In mouth wiping motion after eating, we detected mouth and confirmed that we remove stain by moving robot arm to specific mouth position defined in this study. We also confirmed that even a plurality of people with different sitting height automatically recognize the position of the mouth on the image based on the result of recognition and wipe the stain on the mouth.

In this study, we think that if a subject can move such a little motion, he should move himself. In addition, we focused on before and after meal. Then, we proposed the cooperative meal support system with robot arm and RGB-D sensor.

Fig. 13

Experimental result of subject A

Fig. 14

Experimental result of subject B

Fig. 15

Experimental result of subject C