Keywords

1 Introduction

The “Report on the Chinese Stroke Prevention 2017” shows that the number of stroke patients who are over 40 years of age in China is 12.24 million. Also the trend of young patients with stroke is obvious. Stroke is one of the three highest fatal diseases in our country, with 75% of survivors having varying degrees of disability and loss of ability to work. However, the number of professional therapists for stroke patients is relatively small. Because there are more and more patients, the evaluation of upper limb movement function is arduous.

Traditional Upper limb motor function assessment methods usually include Rom [3], the Wolf motor function test [2], Fugl-Meyer [1], Brunnstrom stage [4] and so on. However, these methods are highly dependent on occupational therapists and need more time [5]. During rehabilitation, the rehabilitation recipe depends on the physician’s assessment; because the assessment of rehabilitation is not quantitative, different doctors may make different assessment decisions [4].

The Kinect 3D Somatosensory Camera developed by Microsoft has been used to capture the movement of players in three-dimensional space [6]. Using Kinect Zhao propose a rule-based human motion tracking rehabilitation exercise that provides automated real-time assessment, feedback, and guidance for users performing rehabilitation exercises at home without physical therapist supervision [7]. Su et al. developed a Kinect-enabled system for the patient to do the exercise and recorded as a base for evaluating the patient’s rehabilitation exercise at home [8].

The exercise space of the upper extremities was measured, to evaluate the functional impairment of the upper extremities caused by some neurological diseases [9, 10]. Kurillo et al. focus on the technical aspect and accuracy of using Microsoft Kinect for assessment of reachable workspace as a potential outcome measure for the upper extremity [11].

Rapid advance in virtual reality technologies has been gaining a wide field in the motor rehabilitation process. Virtual reality technologies could assist the patients during unsupervised rehabilitation by providing an empathic feedback to improve their adherence to the treatment. Virtual reality [12] can create a highly interactive and immersive virtual environment, and Kinect can provide real-time motion status feedback for subjects. Virtual reality will improve the user’s enthusiasm for the experiment; reduce the dependence on the doctor [13, 14].

In this paper, virtual environment technology is used to set up vivid virtual scenes for evaluating test experiments. Visual feedback and auditory feedback increase the interactivity and immersion of the assessment. Different protocol was selected adaptively according to the arm activity to eliminate patients’ psychological stress, which comes from the mismatched movement protocols during the measurement. Subjects follow the instructions of the instructor in different directions to wave the limbs, while Kinect collects hand joint information and shoulder joint information, then the upper limb reachable workspace of the stroke patient was evaluated and analyzed.

2 Method

2.1 Platform

As shown in Fig. 1, the experiment platform contains Microsoft Kinect camera (version 1) [15], tripod, laptop, 31.5-in. display monitor, loudspeaker and so on.

Fig. 1.
figure 1

The experiment platform for upper limb

Full size image

Kinect has three autofocus cameras: two infrared cameras optimized for depth detection and one standard visual-spectrum camera used for visual recognition. The Kinect SDK (Software Development Kit) for Windows provides detailed location and orientation information for up to two players standing in front of the Kinect sensor array. Previous devices have difficulty tracking human motion using a camera without body sensors; Kinect is a noninvasive and markerless method for motion tracking. In this paper Kinect 1.0 was used to collect the movement information of subjects; the tripod was used to place Kinect; the monitor provided a large field of view for the patient to watch easily. Unity 3d was used to design the reachable workspace experimental platform on the computer.

Overall, when a subject came to use the system to do experiment, Kinect built a visual character (avatar) to match him. The protocol level was selected adaptively. Then the instructor in the video began to show what actions the subject should follow. Moreover, the virtual environment will give subject a visual stimulation by the avatar standing next to the video that will copy the subject’s actions when he follows the instructor. Finally, the joint data was recorded and analyzed to evaluate the upper limb reachable workspace.

2.2 Human Skeleton Matching

The skeleton of the subject collected by Kinect is bound to the skeleton of the virtual character model. Figure 2 shows the skeleton point collected by Kinect and the vivid virtual character model. So that the subject can control the character model to move in real time. This increases human-computer interaction and improves patient participation in the trial. Considering the age of patient, the mirror control was used for convenient. Just like the subject stood in front of the mirror and waved his arm. Soothing background music was played while measurement.

Fig. 2.
figure 2

Human skeleton matches virtual characters

Full size image

2.3 Protocol

According to the range of the shoulder flexion extension, outreach adduction, internal rotation and external rotation. Moreover, depending on the patient’s condition, the upper limb reachable workspace motion protocol was designed in three levels, as shown in Table 1, where the range and speed of the first level are the smallest for the severe dyskinesia patient, the range and speed of the third-level are maximum. The upper limb was moved in different ranges by different azimuth and elevation angles in the horizontal and vertical directions.

Table 1. Upper extremity reachable workspace protocol
Full size table

The starting position is the arm naturally put down and arm straight palm inward. The upper limb moves in the horizontal and vertical directions with the shoulder joint as the origin. One of the movement position is shown in Fig. 3. This paper chose to define a coordinate system similar to that used by the Microsoft Kinect. The Kinect is placed on the right side, and the subject is standing facing the Kinect sensor. The shoulder joint O is the origin, P is the recording point-the position of the hand, θ is the azimuth, φ is the altitude. Different angles are set to measure the movement of upper limbs. When the altitude is 30, 60, 90 or 120, the Cheers sound is played to increase the patient’s enthusiasm.

Fig. 3.
figure 3

Movement protocol demonstration

Full size image

2.4 Adaptive Selection of Protocol Level

In order to reduce the psychological burden of the patient and avoid the negative emotion of the patient during the measurement, different levels of motion protocols were set adaptively.

Before the start of the experiment, the subject did the maximal arm outreach three times with the azimuth is zero. The reachable workspace coordinate system of right limb is shown in the Fig. 4(a). The stretching angle θ is shown in Fig. 4(b). It is calculated by the following formula.

Fig. 4.
figure 4

The reachable workspace coordinate system and four quadrants I, II, III, and IV of right limb

Full size image
$$ \uptheta = \cos^{ - 1} \frac{{\overrightarrow {CP} \cdot \overrightarrow {SH} }}{{\overrightarrow {{\left| {CP} \right|}} \overrightarrow {{\left| {SH} \right|}} }} $$
(1)
$$ \overrightarrow {CP} = \left( {Px - Cx,Py - Cy} \right) $$
$$ \overrightarrow {SH} = \left( {Hx - Sx,Hy - Sy} \right) $$

C stands for shoulder center, P stands for Spine, S stands for Shoulder right, and H stands for Right hand.

The system recorded the data automatically and calculated the average of the three abduction angles. The exercise protocol level was selected adaptively according to the average value. When the angle is in the range of 0 to 60°, the motion protocol is the first level, the speed of the demonstrator’s arm is slow. When the angle is in the range of 60–120°, the motion protocol is the second level, and the arm speed is Medium speed. When the angle is within 120–180°, the action protocol level is level 3, and the demonstrator’s arm speed is fast.

Table 1 shows the key motion of the experiment. The movement mainly in vertical and horizontal directions, the path is set by different azimuth angle and altitude angle. The experiment requires subject’s hand to stretch as far as possible and to keep the elbow outstretched. When the subject’s movement is good, the cheers sound is played, giving auditory feedback and encouraging the subject to continue the exercise.

3 Data Analysis

With the shoulder joint as the center of the sphere and the arm length as the radius, the upper limb stretches and draws arcs in different planes. The reachable workspace of upper limb is part of the sphere. The workspace is divided into four quadrants (I, II, III and IV) as shown in the Fig. 4, each quadrant corresponds to 1/4 sphere. The right upper limb is taken as an example.

The trajectory data that was collected by Kinect was filtered with a 6th-order low pass Butterworth filter with a cut-off frequency of 30 Hz. The data was rejected when the movement speed (tangential velocity) was lower than 50 mm/s. The nonlinear least squares method was used to fit the trajectory; then, the fitted trajectory data was projected into the spherical coordinate system, and the α-shape geometry was used to extract contour edges of the trajectory. In order to make the boundary of the reachable workspace more smooth, Hermite spline interpolating was used to interpolate the boundary points. Next, the trajectory data was projected back into the Cartesian coordinates. Then, the corresponding accessible surface patches were extracted. At last, the surface area was normalized. Figure 5 shows the data processing.

Fig. 5.
figure 5

Data processing

Full size image

4 Results

Patient clinical characteristics are described in Table 2. It shows the affected arm, time after stroke, stroke category (cerebral hemorrhage or infarction) and Fugl-Meyer score.

Table 2. Patient Clinical Characteristics
Full size table

Fugl-Meyer score (100) is equal to the total score of upper extremity function (66) plus the total score of lower extremity function (34). According to the clinical significance of the Fugl-Meyer assessment (FMA), human motor function is divided into four stages: severe dyskinesia (<50), significant dyskinesia (50–84), moderate dyskinesia (85–95), and mild dyskinesia (96–99). According to the proportion of upper extremity motor function scores, the clinical significance of dividing the upper extremity FMA score is: severe dyskinesia (<33), obvious dyskinesia (33–55), moderate dyskinesia (56–62) and mild dyskinesia (63–65). The experimental scene in Nanjing Tongren hospital is shown in Fig. 6.

Fig. 6.
figure 6

Experiment scene

Full size image

Figure 7(a1), (a2) and (a3) shows the analysis of reachable workspace obtained in a healthy subject. Figure 7(a1) shows the 3D trajectory data after least squares fitting, the red dot is the ball center. Figure 7(a2) shows the trajectory data projected to the spherical coordinates, and the outer boundaries of the concave bounding polygon is obtained as shown by the blue line. Figure 7(a3) shows the envelope of the reachable workspace obtained by back projecting trajectory data to three dimensional space and fitting a spherical surface. Figure 7(b1), (b2) and (b3) shows the analysis of reachable workspace obtained in the patient 4. Figure 7(c1), (c2) and (c3) shows the analysis of reachable workspace obtained in the patient 2. The Relative surface area (RSA) is represented in different colors in each quadrant, respectively, area1 red, area2 green, area3 purple, and area4 pink.

Fig. 7.
figure 7

Reachable workspace (a1) 3D trajectory after least squares method of healthy subject (a2) Boundary of healthy subject (a3) Graphical visualization of 3d reachable workspace of healthy subject. (b1) 3D trajectory after least squares method of patient 4 (b2) Boundary of patient 4 (b3) Graphical visualization of 3d reachable workspace of patient 4. (c1) 3D trajectory after least squares method of patient 2 (c2) Boundary of patient 2 (c3) Graphical visualization of 3d reachable workspace of patient 2. (Color figure online)

Full size image

It can be seen from RSA in Fig. 7(a3), (b3) and (c3), healthy arm could naturally move to each quadrant; the upper limb of the patient 4 could reach the fourth quadrant, and its active space in the second and third quadrants was less, but it could hardly move to the first quadrant. As can be seen from the figure, as the injury of the upper extremities increases, the smoothness of the trajectory decreases, which indicates that the movement quality of the upper limb is decline and the movement becomes more and more awkward.

The average RSA plot by Fugl-Meyer level is shown in Fig. 8. The value of RSA is the average of each RSA for patients in the same level. As can be seen from the figure, with the decline in the Fugl-Meyer level, the RSA is gradually reduced. The upper limb of patients in level I are only active in the fourth quadrant. The patients in level IV are not much different from the RSA of healthy people.

Fig. 8.
figure 8

The RSA plot by Fugl-Meyer level

Full size image

There is a partial absence of the patient’s upper limb working space. Overall, with the increase of upper limb injury, the RSA of upper limb reachable workspace was decreased, the smoothness of space trajectory was reduced, and the irregularity of path was increased. These figures suggest that compared with healthy subject, stroke patients have less RSA, both in the quadrant and in the total area. Patients in different Fugl-Meyer score also has different RSA.

5 Conclusion

The reachable workspace RSA in stroke patients based on Kinect was measured and analyzed. Patients with different Fugl-Meyer score were participated in this assessment.

The protocol with virtual environment can increase the interactivity and immersion of the assessment. The adaptive selection of protocol can improve the efficiency of the test, and prevent the patient from producing psychological pressure effectively.

Reachable workspace RSA has a certain relationship with the motor function disorder classification based on Fugl-Meyer score, and can be used as a potential quantitative assessment method to assess the motor function of the upper limb, and provide further basis for clinical treatment of post-stroke patients.