The Estimation Method of Force by Using Kinect Camera

Abstract

The purpose of this paper is an estimation of driving torque by using an external sensor. This paper deals with an estimation method of driving torque and physical parameters. This research have been estimated the driving torque of each joints as the force. Future, this research will design an estimation system of a myoelectric signal using an external sensor. Moreover, we aim to design an evaluation system of the ability to exercise using obtained biosignal.

1 Introduction

Due to the aging of society, people who need nursing care are increasing [1]. Herewith, the burden, which is received a caregivers instructing rehabilitation, is thought to increase. In addition, when trainer of newcomers issue instructions for exercise, a detailed exercise cannot be identified because experience is insufficient, the instruction becomes ambiguous. Generally, these are classify into an evaluation problem of ability to exercise. Thus, due to formulating it, to enable a rehabilitation effectively and an identification detailed exercise.

In terms of the evaluation problem of ability to exercise, studies had been conducted on using biosignal [2,3,4]. These studies had assessed the ability to exercise using biosignal, such a force and myoelectric potential. However, the contact sensor as exemplified by a force sensor and a myoelectric sensor occur physical disorder feeling on the ground of wearing the sensor. This is a cause of obstruct for movement. In order to solve this problem, a non-contact sensor is used to obtain the biosignal. It can expect to ease up a limitation of movement. Therefore, a method that obtaines the biosignal using non-contact sensor is proposed.

To obtain a biosignal using a non-contact sensor, Nakamura and colleagues had estimated muscle tension by inverse dynamics calculation [5]. In previous study, Nakamura and colleagues had estimated arm power and myoelectric potential by non-contact sensor Kinect [6]. However, the techniques that estimate physical parameters other than angular velocity and angular acceleration had not been proposed. In addition, force and myoelectric potential with respect to the leg had not been estimated.

Wherein, this research aims to estimate the physical parameters based on the information obtained from Kinect, calculate the driving torque and myoelectric potential of the whole body, and estimate the ability to exercise. This paper describes the driving torque estimation method by the Newton Euler method. After that, it describes the estimation method of physical parameters required for the Newton Euler method and explains the estimation result.

2 Measurement Condition

This section describes the biological signal acquisition method and measurement environment. In this research, Kinect which is a non-contact sensor is used as a biological signal acquisition device. Kinect is an RGB-D camera, and it is able to estimate joint coordinates. Figure 1 shows the measurement environment using Kinect. Figure 2 shows the joint point in the camera coordinate system obtained by Kinect. In the Fig. 2, ${\varvec{p}}_{j}(j = 0,1,\cdots ,24)$ is the coordinate on the camera coordinate system $O_{k}$, $\varvec{p}_{0}$ to ${\varvec{p}}_{3}$, ${\varvec{p}}_{20}$ the trunk, ${\varvec{p}}_{8}$ to ${\varvec{p}}_{11}$, ${\varvec{p}}_{23}$, ${\varvec{p}}_{24}$ are the right arm, ${\varvec{p}}_{4}$ to ${\varvec{p}}_{7}$, ${\varvec{p}}_{21}$, ${\varvec{p}}_{22}$ are the left arm, ${\varvec{p}}_{16}$ to ${\varvec{p}}_{19}$ are the right leg, and ${\varvec{p}}_{12}$ to ${\varvec{p}}_{15}$ are the joints of the left leg. Figure 3 shows the coordinate system of each joints. $j_{n}(n=1,\cdots ,8)$ means the joint from the shoulder to the hand, and the angle of $j_{n}$ is $\theta _{n}$. Similarly, numbers $n=9,\cdots ,15$ are used to number the joints from the hip joint to the toes. Provided that $j_1$ to $j_3$ indicate shoulder joints, $j_5$ to $j_7$ indicate wrist joints, $j_{10}$ to $j_{12}$ indicate hip joints, $j_{14}$, $j_{15}$ indicate ankle joints. Here, Tables 1 and 2 shows DH parameters defined for each joints of the right arm and the right leg. However, dummy is a provisional joint. These are obtained from Fig. 3.

Fig. 1.

Measurement environment.

Fig. 2.

Camera point.

Fig. 3.

Coordinate system of right arm and right leg.

3 The Driving Torque Estimation Method by Using Newton Euler Method

A driving torque of each joints was estimated by the Newton Euler method in this research. Therefore, this section describes about the driving torque estimation method using the Newton Euler method. Now suppose that the each joint angle ${\varvec{\theta }}_{i}$, the angular velocity ${\varvec{\dot{\theta }}}_{i}$ and the target angular accelelation ${\varvec{\ddot{\theta }}}_{i}$ is known. It calculates the angular velocity ${\varvec{\omega }}_{i}$, the angular acceleration ${\varvec{\dot{\omega }}}_{i}$, translational velocity ${\varvec{\dot{p}}}_{i}$ and translational acceleration ${\varvec{\ddot{p}}}_{i}$ that subtend the base coordinate system of each links from the base link to the end link. After that, it derives the translational force ${\varvec{\hat{f}}}_{i}$ and the rotational moment ${\varvec{n}}_{i}$ need to subject that realize such motion by the Newton’s equtaiton of motion and Euler’s one. Moreover, suppose that the force ${\varvec{f}}_{E}$ and the moment ${\varvec{n}}_{E}$ subject to the force the end link is given, it calculates the force ${\varvec{f}}_{i}$ and the moment ${\varvec{n}}_{i}$ that subject to each joints from the base to the end. Based on that, the driving torque $\tau _{i}$ that subject to each joints is obtained [7]. The driving torque corresponds to the force in human motion. Table 3 shows the parameter that used in the Newton Euler method. Provided that the upper left superscript indicates the base coordinate system and the under right superscript indicates link number. Equations (1)–(9) show a forward path and a backward path that calculated by the Newton Euler method. The driving torque $\tau _{i}$ is obtained by calculating the forward path from 1st to nth joints and the backward path from nth to 1st joints.

$$\begin{aligned} \nonumber ~&[Forward~path] \\&^i{{{\varvec{\omega }}}}_{i}\!\!=\! {^i{{{\varvec{\omega }}}}_{{i-1}}} \!+\! {\varvec{e}}_z \dot{\theta }_{i} \end{aligned}$$

(1)

$$\begin{aligned}&{^i{\dot{{\varvec{\omega }}}}_{i}} \!\!=\! {^i\dot{{{\varvec{\omega }}}}_{{i-1}}} \!+\! {\varvec{e}}_z \ddot{\theta }_{i} \!+\! ({^i{{{\varvec{\omega }}}}_{{i-1}}}) \times {\varvec{e}}_z \dot{\theta }_{i}\end{aligned}$$

(2)

$$\begin{aligned}&^i{\varvec{\ddot{p}}}_{{i}}\!\!=\! {^i{\varvec{{\ddot{p}}}}_{i-1}} \!+\! {^i{{\dot{{\varvec{\omega }}}}_{{i-1}}}} \!\!\times \! {^{i}{{\varvec{{\hat{p}}}}_{{i}}}} \!+\! {^{i}{{{{\varvec{\omega }}}}_{{i\,-\,1}}}}\times ({^i{{{\varvec{\omega }}}}_{{i\,-\,1}}} \times {^i{\varvec{{\hat{p}}}}_{i}})\end{aligned}$$

(3)

$$\begin{aligned}&{^{i}{\varvec{{\ddot{s}}}}_{{i}}}\!\!=\! {^{i}{{\varvec{{\ddot{p}}}}_{{i}}}} + {^{i}{{\dot{{\varvec{\omega }}}}_{{i}}}} \times {^{i}{{\varvec{{\hat{s}}}}_{{i}}}} + {^{i}{{{{\varvec{\omega }}}}_{{i}}}} \times ({^{i}{{{{\varvec{\omega }}}}_{{i}}}} \times {^{i}{{\varvec{\hat{s}}}_{i}}}) \end{aligned}$$

(4)

$$\begin{aligned} \nonumber ~&[Backward~path] \\ ~&^{i}{\varvec{\hat{f}}}_{i} = m_{i} {^i{\varvec{\ddot{s}}}_{i}} \end{aligned}$$

(5)

$$\begin{aligned} ~&^i{\varvec{\hat{n}}} _{i} = {^iI_{i}} {^i{\dot{{\varvec{\omega }}}}_{i}} + {^i{{{\varvec{\omega }}} _{i}}} \times ({^iI_{i}} {^i{{\varvec{\omega }}} _{i}})\end{aligned}$$

(6)

$$\begin{aligned} ~&{^{i}{{\varvec{{f}}}_{{i}}}} = {^{i}{{\varvec{{f}}}_{{i+1}}}} + {^{i}{{\varvec{{\hat{f}}}}_{{i}}}} \end{aligned}$$

(7)

$$\begin{aligned} ~&{^{i}{{\varvec{{n}}}_{{i}}}} = {^{i}{{\varvec{{n}}}_{{i+1}}}} \!+\! {^{i}{{\varvec{{\hat{n}}}}_{{i}}}} \!+\!{^{i}{{\varvec{{\hat{s}}}}_{{i}}}} \times {^{i}{{\varvec{{\hat{f}}}}_{{i}}}} + {^{i}{{\varvec{{\hat{p}}}}_{{i+1}}}} \times {^{i}{{\varvec{{f}}}_{{i+1}}}}\end{aligned}$$

(8)

$$\begin{aligned} ~&{\tau }_{{i}} = {{{\varvec{e}}}_{z}}^{T} {^{i}{{\varvec{{n}}}_{{i}}}} \end{aligned}$$

(9)

Table 1. DH parameter (Arm)

Table 2. DH parameter (Leg)

Table 3. Number of Newton-Euler method

Calculating the above, the driving torque that require to realize human motion for each joints is estimated. The next section explains the estimation method that obtains physical parameter required the Newton Euler method.

4 Physical Parameter Estimation

This section describes the method to estimate a joint angle, an angular velocity, an angular acceleration and mass, and inertia tensor. The joint angle is calculated based on inverse kinematics. After that, joint angular velocity and angular acceleration are obtained using differential filter. Farther, the mass and the inertia tensor are estimated by regarding the human body as a simple model.

4.1 Joint Angle Estimation

Taking the right arm as an example, this subsection describes the joint angle estimation method based on inverse kinematics calculation. $\theta _{s}$ is defined as a formed angle between the straight line passing from the upper part of the breastbone point ${\varvec{p}}_{20}$ to the right shoulder point ${\varvec{p}}_{8}$ in Fig. 4 and the X axis on the camera coordinate system. Here, $\theta _{s}$ is calculated by Eq. (10).

$$\begin{aligned}&\theta _{s}\! = \! \tan ^{-1}\left( \frac{p_{8z}-p_{20z}}{p_{8x}-p_{20x}}\right) \end{aligned}$$

(10)

${\varvec{p}}_{8}$ corresponds the $j_{1}$’s coordinate system in Fig. 3 to rotate ${\varvec{p}}_{i}$ on Y axis of the camera coordinate system by $-\theta _{s}$. Moreover, to define rotation matrix ${\varvec{R}}(\cdot )$ that rotate on the Y axis of the camera coordinate system, the point ${\varvec{p}}'_{i}$ on the right arm whose origin is ${\varvec{p}}_{8}$ calculate with Eq. (11).

$$\begin{aligned}&{\varvec{p}}'_{i}\! = \! {\varvec{R}}_{\theta _{s}}({\varvec{p}}_{i}-{\varvec{p}}_{8}) \end{aligned}$$

(11)

Angle $\theta _{1}$ which made by $X-Z$ plane of joint $j_{1}$’s coordinate system in Fig. 5 and the point ${\varvec{p}}'_{9}$ is obtained by Eq. (12).

$$\begin{aligned}&\theta _{1} \! = \! \tan ^{-1}\left( \frac{p'_{9z}-p'_{8z}}{p'_{9x}-p'_{8x}}\right) \end{aligned}$$

(12)

After obtaining $\theta _{1}$, rotate all joints to the leading joints from the joint $j_{1}$ by $-\theta _{1}$ (Eq. (11)). In the same procedure, joint angle $\theta _{2}$–$\theta _{8}$ are calculated to each axis like Figs. 6 and 7. Provided that, after calculating shoulder angles $\theta _{1}$–$\theta _{3}$, translate coordinate system to elbow coordinate system according to DHparameter which were defined. Similarly, after calculating elbow angles $\theta _{4}$, it translate to wrist coordinate system. Joint angles $\theta _{i}(i=2,\cdots ,8)$ is calculated with Eqs. (13)–(19).

$$\begin{aligned}&\theta _{2} \! = \! \tan ^{-1}\left( \frac{p'_{9y}-p'_{8y}}{p'_{9x}-p'_{8x}}\right) \end{aligned}$$

(13)

$$\begin{aligned}&\theta _{3} \! = \! \tan ^{-1}\left( \frac{p'_{10y}-p'_{9y}}{p'_{10z}-p'_{9z}}\right) \end{aligned}$$

(14)

$$\begin{aligned}&\theta _{4} \! = \! \tan ^{-1}\left( \frac{p'_{10z}-p'_{9z}}{p'_{10x}-p'_{9x}}\right) \end{aligned}$$

(15)

$$\begin{aligned}&\theta _{5}\! = \! \tan ^{-1}\left( \frac{p'_{11z}-p'_{10z}}{p'_{11x}-p'_{10x}}\right) \end{aligned}$$

(16)

$$\begin{aligned}&\theta _{6}\! = \! \tan ^{-1}\left( \frac{p'_{11y}-p'_{10y}}{p'_{11x}-p'_{10x}}\right) \end{aligned}$$

(17)

$$\begin{aligned}&\theta _{7}\! = \! \tan ^{-1}\left( \frac{p'_{23y}-p'_{11y}}{p'_{23z}-p'_{11z}}\right) \end{aligned}$$

(18)

$$\begin{aligned}&\theta _{8}\! = \! \tan ^{-1}\left( \frac{p'_{24y}-p'_{11y}}{p'_{24x}-p'_{11x}}\right) \end{aligned}$$

(19)

Fig. 4.

Projection of shoulder and spine shoulder as X-Z plane.

Fig. 5.

Projection of shoulder and spine shoulder as X-Z plane.

Fig. 6.

Projection of shoulder and elbow as X-Y plane.

Fig. 7.

Projection of shoulder and elbow as Y-Z plane.

Fig. 8.

A Human Model

Fig. 9.

Result of estimated center of gravity.

Joint angler velocity and acceleration are calculate from joint angle that obtained by differential filter. Angler velocity calculate using differential filter for obtained angle once. Similarly, angler acceleration calculate using differential filter bidirectional.

4.2 Mass Estimation Method

This subsection explains the estimation method of the each parts of human body. Human body is defined as combination of cylinders such as Fig. 8. Using that mass is estimated from volume and density. Each radius of cylinders are defined as the width from joints to human surface. Moreover, height is defined as a length between joints. Table 4 shows mass that estimated the density of each cylinders as $1.0\times 10^{3}\,\mathrm{kg/m}^3$.

Table 4. Mass of link

Table 5. Moment of inertia

As a result of comparing the sum of estimated mass and measured mass, mass was estimated about 90 % accuracy.

4.3 Inertia Tensor Estimation

This subsection describes the estimation method of the center of gravity and inertia tensor of human body by using chandler estimation equation [9]. Inertia tensors of each cylinders are divided into term of product of inertia and one of inertia moment. In this research, the axis of rotation on the joint coordinate system correspond with the one on the center of gravity coordinate system. For such occasion, term of product of inertia is 0. Therefore, inertia tensor is able to estimate by inertia moment. The gravity center of the each cylinders was estimated by chandler estimation equation. Vertex is the point of human surface on the $p_{3}$ in Fig. 2. Moreover, X, Z of the Heel is same as the value of $p_{18}$, and Y is same as $p_{19}$. Figure 9 shows the result of estimating the center of gravity. Figure 9 shows that the center of gravity is appropriate for each parts of human. Inertia moment is calculated by the estimated center of gravity. The inertia moment of the cylinder is obtained by Eqs. (20)–(22) [9].

$$\begin{aligned}&J_x = \frac{M}{12}(3R^2+h^2) + Md_{x}^2 \end{aligned}$$

(20)

$$\begin{aligned}&J_y = \frac{M}{12}(3R^2+h^2) + Md_{y}^2 \end{aligned}$$

(21)

$$\begin{aligned}&J_z = \frac{M}{2}R^2 + Md_z^2 \end{aligned}$$

(22)

Fig. 10.

Capture arm motion (1)

Fig. 11.

Capture arm motion (2)

Fig. 12.

Capture leg motion (1)

Fig. 13.

Capture leg motion (2)

Table 5 show inertia moment of each cylinders. Inertia tensor is calculated by obtained inertia moment.

5 Verification of Estimating Result

This section describes the estimating result of joint angle and torque. Figures 10, 11, 12 and 13 shows appearance of measuring. Further, Figs. 14, 15, 16 and 17 shows the joint data reconstructed from obtained log data. The arm has default position as $\theta _{2} = -\frac{\pi }{2}$ and $\theta _{3} = \frac{\pi }{2}$. This motion rotates in order to shoulder’s Z axis of camera coordinate system by $\frac{\pi }{2}$ rad, elbow’s one by $\frac{\pi }{2}$ rad, elbow’s one by $-\frac{\pi }{2}$ rad, and shoulder’s one by $-\frac{\pi }{2}$ rad. The leg has default position as $\theta _{9} = \theta {10} = \theta {11} = \theta {12} = \theta _{13} = \theta {14} = 0$. This motion rotates in order to hip’s Z axis of camera coordinate system by $\frac{\pi }{4}$ rad, knee’s one by $-\frac{\pi }{2}$ rad, knee’s one by $\frac{\pi }{2}$ rad, and hip’s one by $-\frac{\pi }{4}$ rad. Figures 10, 11, 12, 13, 14, 15, 16 and 17 shows that the result of estimation joint correspond to each motion. Figures 18 and 19 shows the result of arm and leg angle estimation using measured log data. Figures 18 and 19 shows that it is appropriate that the default position and motion of shoulder, elbow, hip and knee. Therefore, arm and leg angle was estimated. Figures 20 and 21 shows results of driving torque estimation by using Newton Euler method. Moreover, Figs. 22 and 23 shows the joints which was moved. The torque was increased that shown by Figs. 20 and 21. Additionally, the driving torque for the motion shown by Figs. 22 and 23 was estimated. For verification of appropriateness, coefficient of correlation between angular velocity and driving torque was calculated by Newton’s law. Tables 6 and 7 shows coefficient of correlation between angular velocity and driving torque which obtained by estimation. The coefficient of correlation of joints 4–9 of arm was calculated ±0.9–1 (Table 6). The reason why it is minus was implicated that the anti-torque occurred to keep the posture. The one of leg was calculated 1 except joints 10 and 11. The reason why it is minor on the joints 10, 11 was implicated that the anti-torque has occurred to bow knee joints keeping hip posture. Hereafter, detail accuracy of torque would be verified by acceleration sensor or motion capture system.

Fig. 14.

Arm’s log (1)

Fig. 15.

Arm’s log (2)

Fig. 16.

Leg’s log (1)

Fig. 17.

Leg’s log (2)

Fig. 18.

Estimation arm angle.

Fig. 19.

Estimation leg angle.

Fig. 20.

Result of torque estimation (Arm).

Fig. 21.

Result of torque estimation (Leg).

Fig. 22.

Joint2 and Joint4’s angle and torque.

Fig. 23.

Joint10 and Joint13’s angle and torque.

Table 6. Correlation (arm)

Table 7. Correlation (leg)

6 Conclusion

The angle of each joints of human body have been estimated based on inverse kinematics calculation from joint position which obtained by Kinect. Additionally, mass and center of gravity of each human body parts have been estimated. And the torque of arm and leg have been estimated by physical parameters. Hereafter, this research aims to accuracy improvement of driving torque estimation and design the system of estimation the myoelectric potential. Moreover, it aims to design appraisation system of ability to exercise by driving torque and the myoelectric potential.