1 Background

The relationship between improving motor functions of stroke patients and the intensity of the rehabilitation program that the patients receive is well established [30, 35]. To overcome many obstacles (economic, transportation, physiological, etc.) that prevent stroke patients from continuing their rehabilitation treatment, researchers have proposed virtual reality exergames that could help patients perform their rehabilitation exercises at home [7, 11, 15, 18]. Many of these researches have focused on assistive therapy devices that can serve as interface between the patients and the virtual environment [1, 25]. Furthermore, these devices could provide the patients and the therapists with a scientific feedback that helps evaluating the treatment program. Artificial intelligence algorithm is being used in [11] to push the patients to their limits while they are performing the exercises without losing control of their pace. In [1], a novel multiple regression model is used to analyze the clinical data obtained from the GENTLE/S robot rehabilitation system. The majority of the robotic therapy devices provide certain level of satisfaction [12]; nonetheless, these devices have been suffering from some limitations due to their bulky shape, high price, and the complexity of their deployment. This interprets the fact that these robotic-assisted therapy devices are not widely used in home rehabilitation.

Several studies have shown that there is a correlation between the strength and the kinematics of the upper limb movements [4, 24]. The relationship between the strength of the upper extremity and the throwing speed were investigated in [23]. The results suggested a direct relationship between, in particular, the strength of the elbow extension and wrist extension movements and throwing speed. Moreover, Cohen et al. [4] showed that the serve’s speed of elite tennis players could be increased by strengthening the muscles of their upper limbs. The relationship between the kinematics and dynamics of nineteen upper limb’s daily activities were generated in [24]. Xiao et al. [36] captured the force myographic data signals of the forearm and used them to predict the upper-extremity posture in real-time. More recently, Nathan et al. [21] have investigated the energy expenditure of the skeletal muscles during motion by using Kinect sensor. They use two well know equations for external and internal work of the body that find a relationship between the kinematics of the skeletal and the potential and kinetic energy. Although these studies suggest that such relationship between the strength of muscles and the kinematics of the upper extremity exists, further investigations are needed to confirm the obtained results. The reason is that most of the experiments were conducted on professional athletes that knew how to coordinate their upper extremity with lower extremity and truncate activity while they were performing the tasks [4, 23]. Moreover, even though there are many studies that use Kinect to capture the kinematic data from live gestures of a subject [7, 26], we could not find any research that tackles the correlation between the kinematics of the upper limb of stroke patients and the strength of their muscles. In addition, to our knowledge, there were no attempts on finding the strength of the muscles using a camera tracking sensor (Kinect).

In this study, we have designed an experiment in order to address the relationship between the kinematics of human’s upper limbs and the strength of the corresponding muscles. Five stroke patients and eight healthy subjects are volunteered to participate in this study. Kinect sensor is used along with a FSR glove and two straps to collect real-time upper limb joints data of the subjects and the strength of muscles while they are under rehabilitation exergame. The nearest correlation matrix is found between the force and the kinematics of the upper limb. The matrix is used later to calculate the muscles forces given the kinematics measurements of the upper limb. We have analyzed the matrices characteristics of the healthy subjects and the patients. Moreover, we have found the correlation between multiple regression matrices that are taken at different times of the same patient. Finally, we have conducted several experiments to asses objectively the accuracy of the obtained matrix in finding the correlation between the strength and the kinematics of the upper limb.

2 Proposed system

2.1 Kinect sensor

Kinect sensor is a free-hand, low-cost game controller that provides a natural interface in a computer game environment. Being able to capture the positions, depths of players, and surrounding environment in a 3D space, Kinect can build an augmented reality environment where players interact with real and virtual worlds. Many studies evaluated the correctness of the obtained 3D positions of the body joints from Kinect [3, 22, 32]. The evaluation was done by comparing the values captured by Kinect to those captured by an advanced multiple-camera 3D motion tracking. In [22], 43 markers were placed on a human body and tracked by nine cameras. The obtained points were connected together to form a skeleton. Another skeleton was estimated by using the Kinect sensor. Results showed that Kinect could replace high-cost tracking systems for controlled body exercises. However, it suffered from occlusion when the subjects were performing the exercises while they were sitting in a wheelchair. Clark et al. [3] have reached similar results when they tested Kinect pose estimation with twelve camera Vicon MX motion tracking. Moreover, it has been found in [32] that the Kinect interface has advantages over the Nintendo Wii and Sony PlayStation Move interfaces.

More recently, Kinect2 has been used to measure the human balance by tracking the center of mass and skeleton of the body [16, 17]. The obtained results were compared with a Wii balanced balance board and it was shown that the most accurate results were obtained by kinect2 sensor.

2.2 FSR glove

Five Force Sensing Resistors (FSRs) are mounted on the fingertips of a glove as illustrated in Fig. 1.

Fig. 1
figure 1

Five FSR sensors mounted on finger tips of a glove

Full size image

This type of sensors is widely used in data gloves (http://www.cyberglovesystems.com/), because its force sensitivity is optimized for the use in human touch control of electronic devices (https://www.sparkfun.com/datasheets/Sensors/Pressure/fsrguide.pdf). The glove is also equipped with a Picaxe 14 M2 microcontroller and a BlueSmirf bluetooth modem. The microcontroller receives eight analog input voltages, corresponding to the amount of pressure on each FSR sensor, through eight Analog to Digital Converter (ADC) channels, processes, and sends a relevant serial output raw data to the computer through the Bluetooth modem. The modem passes this data with a baud rate of 9600 with a sample rate of 29 samples per second. The glove circuit is powered by a 3.3 V rechargeable battery. More details about the design of the FSR glove can be found in [29].

2.3 Strap position

Extensors and flexors muscles are originated from just above the elbow (Common extensor tendon and common flexor tendon). These muscles cross the elbow, forearm, and wrist and are inserted in the fingers’ bone. Although they originate just above the elbow, their function (movement and stabilization) is on the wrist and the fingers. A sensor has to be placed around these muscles in order to measure their strength. Moreover, in our test, the subject grasps and moves a cup in a vertical direction which includes moving and stabilizing the elbow. Hence, the main muscles that act on the elbow biceps (flexion and supination), triceps (extension), brachialis (flexion), and brachioradialis (flexion) are also involved in our experiment. Another strap of sensors is needed around the arm in order to measure the strength of these muscles.

Twenty FSR sensors are placed on two hook-and-loop straps (ten FSR sensors on each). Each strap is 29 cm long and 2 cm wide as shown in Fig. 2.

Fig. 2
figure 2

Ten FSR sensors mounted on a strap

Full size image

Half centimeter is the distance that separates two consecutive FSR sensors. The design is very similar to the one found in [36] except that we have added two extra sensors in order to: (1) get the measurements of most of the muscles that are engaged in the experiment and, (2) minimize the effect of rotation of the strap on the final results of the experiment. One of the straps is wrapped around the forearm 4 cm away from the proximal radioulnar joint. The other strap is wrapped around the biceps and triceps of the arm. It is worth noting that the experiment was conducted under a direct supervision of an orthopedic sub-specialist; the blood circulation was normal in the arm and the subjects were not at risk at any time.

2.4 Data capturing setup

An analog to digital converter converts the analog data that have been captured by the FSRs sensor into digital one. The average forces of the glove’s five sensors is determined and then sent through a Bluetooth communication channel to the computer interface. The same operation is repeated for the other two straps located on the forearm and the arm respectively. Meanwhile the kinematics of the upper limb are also captured by the Microsoft Kinect depth camera. The positions of the joints are recorded every 35 ms. The average velocity is obtained by dividing the total distance (the sum of all sub-distances) by the total time. Researchers have found that “Jerk Cost,” the time integral of the squared magnitude of the hand jerkiness, is more effective in identifying the smoothness of the arm movement in the space [8]. For this reason, we have calculated the jerk cost in this study instead of the jerkiness (simple derivative of the acceleration).

Computer software is developed to manage and control the received data from the FSR sensors as well as from the Kinect sensor. Five arrays are created to store the recorded data (three arrays for the FSR sensors, one for the Kinect sensor, and one for the time stamp). The program calculates the instantaneous velocity, instantaneous acceleration, and the jerk cost according to Eqs. 1, 2, and 3 respectively, and then stores all the data (time stamp, velocity, acceleration, jerkiness, glove FSR, strap one FSR, and strap two FSR values in SPSS data file for further analysis.

$$ \boldsymbol{v}=\frac{\Delta \boldsymbol{x}}{\Delta \boldsymbol{t}} $$
(1)
$$ \boldsymbol{a}=\frac{\Delta \boldsymbol{v}}{\Delta \boldsymbol{t}} $$
(2)
$$ \boldsymbol{J}=\frac{1}{\boldsymbol{N}}\ {\displaystyle {\sum}_{\boldsymbol{i}=1}^{\boldsymbol{N}}\Big(\frac{1}{2}\times {\displaystyle {\int}_0^{\boldsymbol{T}}{{\boldsymbol{a}}^{\boldsymbol{\prime}}}^2\left(\boldsymbol{t}\right)\boldsymbol{d}\boldsymbol{t}}}\Big) $$
(3)

Where v is the velocity, a is the acceleration, J is the jerk cost, N is the number of times the user has performed the movement, T is the time interval, and a’(t) is the rate of change of the acceleration.

2.5 Correlation between strength and kinematics with least-squares regression

Correlation techniques based on least square regression procedures are one of the most commonly used methods that deal with multi-input multi-output systems’ problems. Such systems usually have more equations than unknowns in which an exact solution cannot be found. However, we can use the least squares technique in order to find an estimation of the solution with minimum sum of squares of errors.

In stroke rehabilitation literature, linear regression models have been used to find the relationship between different parameters related to patients’ performance [5, 10, 20, 28, 37]. In [5], three linear regression techniques namely, ordinary least squares (OLS), Tobit, and censored least absolute deviation, were used to find a mapping between general health outcome variables and EQ-5D weights among stroke patients. Results clearly indicated that the OLS technique had produced the best results for predicting the EQ-5D variables. In [20], Menon et al. evaluated five regression techniques namely, physiological based model, OLS, regularized least squares linear regression model, support vector machine, artificial neural network, and locally weighted projection regression, that were used to predict the isometric joint torque by acquiring the surface electromyography signals. One more time, OLS model produced the best fit for estimating the isometric torque among the five regression models that have been evaluated. In [37] OLS model was also used to investigate the relationship between arm’s muscle strength variation and its bio-impedance. The results obtained in [37] as well as the previous results obtained in [5] and [20] encouraged us to use the least- square regression model to investigate the relationship between the strength of the patients’ muscles and their corresponding kinematics.

Consider the system shown in Fig. 3.

Fig. 3
figure 3

Mapping Kinematics to Forces

Full size image

It has three kinematics inputs, K = (Velocity V, Acceleration A, Jerkiness J), and twenty five muscles’ force outputs (ten from each strap and five from the glove), F = (f1, f2,. ..., f25). The mathematical model of the system can be described by the following set of equations:

$$ \left\{\begin{array}{c}\hfill {f}_1={b}_{1,1}V+{b}_{1,2}A+{b}_{1,3}J\hfill \\ {}\hfill \dots \hfill \\ {}\hfill {f}_{25}={b}_{25,1}V+{b}_{25,2}A+{b}_{25,3}J\hfill \end{array}\right. $$
(4)

or in vector form:

$$ \mathrm{F}=BK $$
(5)

where:

$$ F\circeq {\left[{f}_1,\dots, {f}_p,\dots, {f}_{25}\right]}^T $$
(6)
$$ K\circeq {\left[V,A,J\right]}^T={\left[{k}_1,{k}_2,{k}_3\right]}^T $$
(7)

and

$$ \mathrm{B}\circeq \left[\begin{array}{ccc}\hfill {b}_{1,1}\hfill & \hfill {b}_{1,2}\hfill & \hfill {b}_{1,3}\hfill \\ {}\hfill \begin{array}{c}\hfill .\hfill \\ {}\hfill .\hfill \end{array}\hfill & \hfill \begin{array}{c}\hfill .\hfill \\ {}\hfill .\hfill \end{array}\hfill & \hfill \begin{array}{c}\hfill .\hfill \\ {}\hfill .\hfill \end{array}\hfill \\ {}\hfill {b}_{25,1}\hfill & \hfill {b}_{25,2}\hfill & \hfill {b}_{25,3}\hfill \end{array}\right] $$
(8)

In our setup, we have captured the kinematics by the Kinect sensor and the forces by the glove and the straps. The relationship between the kinematics and forces which is represented by the matrix B can be calculated depending on the least-square regression formula. The p th row of Eq. (4) can be written as follows:

$$ {f}_p={\boldsymbol{K}}^{\boldsymbol{T}}{\boldsymbol{b}}_{\boldsymbol{p}} $$
(9)

where

$$ {\boldsymbol{b}}_{\boldsymbol{p}\circeq}\left[{b}_{p1}{b}_{p2}{b}_{p3}\right] $$
(10)

Moreover, let us further define the following two matrices:

$$ {\mathbf{\mathcal{F}}}_{\boldsymbol{p}}\circeq \left[\begin{array}{c}\hfill {x}_{i(1)}\hfill \\ {}\hfill .\hfill \\ {}\hfill \begin{array}{c}\hfill .\hfill \\ {}\hfill {x}_{i(p)}\hfill \\ {}\hfill \begin{array}{c}\hfill \begin{array}{c}\hfill .\hfill \\ {}\hfill .\hfill \end{array}\hfill \\ {}\hfill {x}_{i(n)}\hfill \end{array}\hfill \end{array}\hfill \end{array}\right] $$
(11)
$$ \mathrm{k}\circeq \left[\begin{array}{ccc}\hfill {V}_{(1)}\hfill & \hfill {A}_{(1)}\hfill & \hfill {J}_{(1)}\hfill \\ {}\hfill .\hfill & \hfill .\hfill & \hfill .\hfill \\ {}\hfill \begin{array}{c}\hfill .\hfill \\ {}\hfill \begin{array}{c}\hfill {V}_{(p)}\hfill \\ {}\hfill \begin{array}{c}\hfill .\hfill \\ {}\hfill .\hfill \end{array}\hfill \end{array}\hfill \\ {}\hfill {V}_{(n)}\hfill \end{array}\hfill & \hfill \begin{array}{c}\hfill .\hfill \\ {}\hfill \begin{array}{c}\hfill {A}_{(p)}\hfill \\ {}\hfill \begin{array}{c}\hfill .\hfill \\ {}\hfill .\hfill \end{array}\hfill \end{array}\hfill \\ {}\hfill {A}_{(n)}\hfill \end{array}\hfill & \hfill \begin{array}{c}\hfill .\hfill \\ {}\hfill \begin{array}{c}\hfill {J}_{(p)}\hfill \\ {}\hfill \begin{array}{c}\hfill .\hfill \\ {}\hfill .\hfill \end{array}\hfill \end{array}\hfill \\ {}\hfill {J}_{(n)}\hfill \end{array}\hfill \end{array}\right]=\left[\begin{array}{c}\hfill {\boldsymbol{K}}_{(1)}^{\boldsymbol{T}}\hfill \\ {}\hfill .\hfill \\ {}\hfill \begin{array}{c}\hfill .\hfill \\ {}\hfill \begin{array}{c}\hfill .\hfill \\ {}\hfill .\hfill \\ {}\hfill {\boldsymbol{K}}_{\left(\boldsymbol{n}\right)}^{\boldsymbol{T}}\hfill \end{array}\hfill \end{array}\hfill \end{array}\right] $$
(12)

The bracketed subscript (p) denoting the p th set of measurement p = (1,  … , n). Consequently, the above n sets of measurements satisfy for the i th output:

$$ \begin{array}{ccc}\hfill {x}_{i(1)}\hfill & \hfill =\hfill & \hfill {\boldsymbol{K}}_{(1)}^{\boldsymbol{T}}{\boldsymbol{b}}_{\boldsymbol{i}}\hfill \\ {}\hfill .\hfill & \hfill .\hfill & \hfill .\hfill \\ {}\hfill \begin{array}{c}\hfill .\hfill \\ {}\hfill {x}_{i(p)}\hfill \\ {}\hfill \begin{array}{c}\hfill .\hfill \\ {}\hfill .\hfill \\ {}\hfill {x}_{i(n)}\hfill \end{array}\hfill \end{array}\hfill & \hfill \begin{array}{c}\hfill .\hfill \\ {}\hfill =\hfill \\ {}\hfill \begin{array}{c}\hfill .\hfill \\ {}\hfill .\hfill \\ {}\hfill =\hfill \end{array}\hfill \end{array}\hfill & \hfill \begin{array}{c}\hfill .\hfill \\ {}\hfill {\boldsymbol{K}}_{\left(\boldsymbol{p}\right)}^{\boldsymbol{T}}{\boldsymbol{b}}_{\boldsymbol{i}}\hfill \\ {}\hfill \begin{array}{c}\hfill .\hfill \\ {}\hfill .\hfill \\ {}\hfill {\boldsymbol{K}}_{\left(\boldsymbol{n}\right)}^{\boldsymbol{T}}{\boldsymbol{b}}_{\boldsymbol{i}}\hfill \end{array}\hfill \end{array}\hfill \end{array} $$
(13)

or, in matrix form:

$$ {\mathrm{\mathcal{F}}}_i=\mathrm{k}{b}_i $$
(14)

Since we cannot find the exact element of b i of equation 14, we are going to assume \( \widehat{{\boldsymbol{b}}_{\boldsymbol{i}}} \) to be an estimate of b i . Equation 14 can be rewritten as:

$$ \widehat{{\mathrm{\mathcal{F}}}_i}=\mathrm{k}\widehat{b_i} $$
(15)

where \( \widehat{{\mathbf{\mathcal{F}}}_{\boldsymbol{i}}} \) is the estimation of ℱ i .

The best estimate of b i , in a least square regression sense is obtained by applying the techniques of matrix calculus of trace functions to the above expression:

$$ \Big({\mathrm{k}}^{\boldsymbol{T}}\mathrm{k}{\widehat{\boldsymbol{b}}}_{\boldsymbol{i}}^{*}={\mathrm{k}}^{\boldsymbol{T}}{\mathbf{\mathcal{F}}}_{\boldsymbol{i}} $$
(16)

where \( {\widehat{\boldsymbol{b}}}_{\boldsymbol{i}}^{*} \) is the best estimate of b i such that:

$$ {\widehat{b}}_i^{*}={\left({\mathrm{k}}^T\mathrm{k}\right)}^{-1}.{\mathrm{k}}^T{\mathrm{\mathcal{F}}}_i=\left({\widehat{b}}_{1i}^{*},{\widehat{b}}_{2i}^{*},{\widehat{b}}_{3i}^{*}\right) $$
(17)

3 Experiment study with the patients

3.1 Clinical study

The motor functions of the upper limbs of the patients were evaluated before starting the experiments by a physician and a professional physiotherapist. They used a well-known test, the Action Research Arm (ARM) test [19], in order to measure the subjects’ strength. For healthy subjects the maximum score of the test is 57. This score is obtained by fully completing the 19 sub-exercises that the test consists of. Table 1 shows the results of the ARM test for all the patients during three weeks.

Table 1 Results of Action Research Arm Test taken by patients at week one, week eight, and week seventeen
Full size table

3.2 Experiment

Eight healthy volunteers (mean age: 31 ± 10.9; mean forehand circumference: 28.2 ± 2.1; mean biceps circumference: 28 ± 1.6) and five stroke patients (mean age: 52.4 ± 20.8; mean forehand circumference: 26.56 ± 1.3; mean biceps circumference: 26.78 ± 1.7) participated in the experiment. The healthy volunteers (Table 2) were divided into two groups, the basic persons (five subjects) who do not practice on a weekly basis and the average persons (three subjects) who practice regularly every week.

Table 2 Healthy Subjects Statistics
Full size table

The patients (Table 3) were having chronic stroke ( > 8 months) with no serious cognitive problems, not fully bounded to a wheelchair, and had not been hospitalized for 24 h a day during the time of conducting the experiments (they were out patients). All patients were given a copy of an informed consent form to take home and read carefully, and to decide whether to sign it or not, and hence take part in the experiment. We made sure that the informed consent forms had been signed by the patients before we started the experiments. Moreover, the experiment was approved by the local ethics committee in the rehabilitation center. Finally, to confirm the reliability of the resultant data, the medical conditions of the patients were unknown to the experimenting team.

Table 3 Stroke Patients Statistics
Full size table

The exercises took place in a 5 m × 7 m room inside the rehabilitation center. In the following, descriptions of the test protocols are illustrated.

  1. 1.

    Prior to the first virtual-reality rehabilitation session, all the patients were asked to watch a pre-recorded video that describes the task they were going to perform.

  2. 2.

    As per Kinect requirement to be between 2 and 3 m from the subject, 2.2 m was the distance from the patient to the Kinect camera.

  3. 3.

    Therapist helps the patients put on the straps as described in section 2.3 (one of the straps is wrapped around the forearm 4 cm away from the proximal radioulnar joint. The other strap is wrapped around the biceps and triceps of the arm). Moreover, before starting the exercise, the patient should grasp the cup in a proper way to ensure a correct reading of the fingers’ forces.

  4. 4.

    The patients had tested the system before they performed the official task. That was the right time for us to talk to the patients; to discuss the different steps of the task, and to emphasize the importance of their cooperation.

  5. 5.

    The patients had four sessions in each week for a period of 24 weeks. Each session lasted for 30 min. However, we stopped some sessions before the planned time, because the patients were tired or got uncomfortable.

  6. 6.

    The data collected between week one and week seventeen was used to build the regression matrix, while the rest of the collected data was used in the validation process.

A moving cup experiment was designed to investigate the relationship between the kinematics of a moving hand and its strength. The participants were asked to reach and move a real cup in the vertical direction (Fig. 4).

Fig. 4
figure 4

A healthy subject conducting the experiment

Full size image

With the help of the Kinect camera, an augmented reality environment containing a virtual cup and a vertical straight line was created. The straight line represented the moving path between the start and the end points that the user would follow. The virtual cup represented the end point. The task started when the participant grasped and moved the real cup. We setup a timer to capture the period from the moment the participant touched the real cup to the moment the real cup reached the virtual one.

3.3 Statistical analysis

Statistical analysis were conducted by using SPSS version 19 [9] and the mean difference was set to be significant at the 0.05 level or below. Separately for each participant, the average values of the kinematics and the forces were calculated and used to represent his/her overall performance. The healthy subject were divided into two groups (a) basic group who do not practice regularly, (b) average group who practice on a weekly basis. ANOVA test, a well-known statistical test used for comparison of means between different groups [6, 27, 31], was used to investigate the significant difference between the kinematics values and the forces values of the two groups of the healthy subjects. The same test (ANOVA) was also used to determine the significant difference between the calculated parameters (kinematics and forces parameters) at the first, the eight and the seventeenth week of the stroke patients. Changes over time in the strength of the forces and the kinematics of the patients were evaluated by using the paired t-test.

The regression between the kinematics of the upper limb and the extracted forces for both the stroke patients and healthy participants were calculated using the least square regression technique. The resulted regression matrices were normalized for each individual at the first, eight, and the seventeenth weeks. Moreover, we calculated the correlation between corresponding columns of the matrices that obtained from the same patients during the period of rehabilitation.

4 Results

In this experiment, the least square regression matrix consists of three columns and twenty five rows (from the multiplication theory of matrices, our matrix should have the same number of columns of the kinematic matrix - three kinematic variables - and the same number of rows of the sensors matrix - twenty five data sensors - see equation 19). Before we discuss the calculated regression matrices of the stroke patients, it is worth to state that the variance of the obtained matrices of the same healthy subject were statistically significant (P = 0.001), and hence we could not find a unique relationship between the kinematics and the corresponding muscles of the healthy subjects.

Let M1, M2, and M3 be the least square regression matrices at week one, week eight and week seventeen respectively. We have conducted Pearson correlation to find the strength of association that exists between the three matrices M1, M2, and M3. As shown in Table 4, M1, M2, and M3 are highly correlated and hence, we could use any of them in order to calculate the values of the forces. However, after testing the three matrices, the minimum error between the real values captured by the sensors and the calculated values of the upper limb forces was found by using M3.

Table 4 Correlation between the regression matrices M1, M2, and M3
Full size table

The average values of the measured forces of the arm, the forearm, and the fingers were greater than those derived using equation 19 (Fig. 5).

Fig. 5
figure 5

Mean forces of the upper limb muscles extracted by the FSR forces and calculated by using the regression matrix between week 18 and week 24. a arm force, b forearm force, c fingers force

Full size image

However, there were no statistically significant difference in the measured values of the forces and the calculated ones (P = 0.234 for arm; P = 0.224 for forearm; and P = 0.349 for fingers). The smaller difference between the average of the measured values and the calculated values of the forces was recorded at week 18, whereas the larger difference was recorded at week 19.

The average force of each finger of the healthy participant followed the same trend over the four sessions during the first week; the forces dramatically increased at the beginning of the contact between the sensors and the cup, and then they almost reached a stable value (Fig. 6a).

Fig. 6
figure 6

Muscles forces and hand velocity versus time. a fingers forces of healthy subject, b arm forces of healthy subject, c hand velocity of healthy subject, d fingers forces of stroke patient, e arm forces of stroke patient, f hand velocity of stroke patient

Full size image

The minimum registered force (mean 2.21 N) was exerted by the little finger, while the thumb exerted the maximum force (mean 5.6 N). Although the forces exerted by the fingers of the stroke patients were not uniform, the minimum force (mean 0.61 N) and the maximum force (mean 1.81 N) were, just like the healthy subjects, registered by the little finger and the thumb respectively as depicted in Fig. 6d. The obtained results indicated that the mean values of the fingers of the healthy subjects were significantly different than those of stroke patients (P < 0.001). As compared with the strength of the fingers at week one, strength of the grooming, middle and ring fingers of the patients at week seventeen reported a non-significantly variance in the captured forces (P = 0.21). However, there were a statistically significant difference between the thumb and the little finger of the patients in week one compared to week seventeen (P < 0.02).

The forces generated by the healthy subjects’ arm muscles have, to a certain extent, a certain pattern depending on the location of the sensor on the muscle (Figs. 6b). On the other hand, the curves of the forces of the stroke patient at week one are irregular; they do not follow a certain trend or pattern (Fig. 6d). After seventeen weeks of rehab the curves look smoother and a certain form of pattern can be detected and the overall forces exerted are greater in the seventeenth week than that in first week.

The corresponding velocity curves for the healthy and patients participant are shown in Fig. 6c. The velocity profile for the healthy subjects shows slightly negative skewed bell-shaped curve (single peak); the maximum velocity (v = 0.084 cm/ms) is registered a 500 ms and the total duration of time is 750 ms. However, in week one the velocity profile of the stroke patient has multiple peaks and the maximum velocity is lower (v = 0.0029 cm/ms). In week seventeen, the number of the peaks decreased while the maximum velocity increased (v = 0.0046 cm/ms). Although the mean velocities of all the patients increased, the amount of change was not statistically significant between week one and week seventeen. However, when the t-test was conducted on the jerkiness and zero-cross values of week one and week seventeen, it was found that the group of patients were associated with statistically significant mean difference (Table 5).

Table 5 Summary of different variables of the first, the eight, and the seventeenth week
Full size table

The increase of the kinematics and the forces measurements was consistent with the results obtained from the clinical tests (Table 1, and Table 5). Moreover, such consistency was emphasized when we conducted the clinical test again at the end of the study (week 24).

5 Discussion

In this study we found the relationship between kinematics and strength of upper limb of stroke patients. The velocities of the hand movement of stroke patients were characterized by a larger degree of variation than those of healthy subjects. Moreover, the time needed to finish the task by the patients was longer than that of healthy subjects. The collected measurement data show that the trajectories of the movements of the patients were not smooth and precise. Indeed, the large number of both, the peaks of the jerkiness and the zero-cross of the patients’ curves confirms that (see Table 3). The kinematics values obtained from the patients were highly correlated with the severity of the stroke attack. In particular, data from patients with high severe attacks indicate less velocity, less acceleration, but more variations in the jerkiness.

Most studies show that the performance of the upper limb stroke patients is correlated with the severity of the stroke attack [2, 14]. Patients who had a severe stroke attack suffer from muscle weaknesses which lead to decrease of the kinematics. As expected, we found that there was a significant relation between the strength of the muscles of the fingers, forearm, arm and the kinematics of the upper limb. Moreover, our study has suggested that the strength of the muscles of the fingers and that of the forearm were significantly correlated during the first 500 ms of the task (time needed to start moving the cup in the vertical direction). After that, the forces of the muscles of the arm started to increase with overall values larger than those of the forearm. This indicates that the forearm muscles are more engaged in the grasping activity than the arm muscles. This result confirms a previous study by Sara et al. [34] who showed that there was no significant change in the arm muscles activity at the beginning of the task. However, during the object transport in the vertical plane maximum activity of the arm muscles was recorded.

The large variation in jerkiness measurements of the upper limb of the same patient is due to the lack of muscle strength in the group muscles of the fingers, forearm, and arm. Indeed, as shown in Fig. 6, the values of the forces exerted by the muscles of the fingers’ arm are very small compared to those exerted by healthy subjects. Moreover, the inconsistency of these values (note the lack of a trend in the obtained data) indicates that there is difficulty planning, coordinating, and controlling the movements of the upper limb. Such difficulty is significantly related to the severity of the stroke. In addition, we have noticed that there is a significant variation between the groups of data obtained from the different patients. For example, the magnitude of the standard deviation (3.5 s) of the time needed to complete the task is clinically significant. By the same token, the magnitude of the standard deviation (53) of the number of zero-cross is also clinically significant. In contrast, the kinematics measurements (including jerkiness) and the values of the muscles forces of the healthy subjects were consistent during the entire upper limb analysis session. Our results are compatible with previous researches that have been conducted by Stewart et al. [29] and Xiao and Menon [36]. In [29] the time needed to complete a task of moving a cup was relatively small, and the trajectory profile of the velocity was a bell-shaped. In [36], eight FSR sensors mounted on a strap and placed around the forehand of a healthy subject. The captured forces of the muscles of the forehand exhibited patterns that can correctly classify six postures associated to a drinking task.

The least square correlation matrix showed to be a good tool in calculating the forces of the muscles giving the upper-limb’s kinematics of the stroke patients. The same result could not be proven when it comes to healthy subjects. In fact, and since the rehabilitation exercises in this study require a little effort to perform, the healthy subjects could control the kinematics and the strength of their muscles at their will (they could change the measurement values of the forces without changing the kinematics and vice versa). Back to the regression matrix of the patients, we could notice that the more severe the stroke is, the higher is the correlation between the corresponding columns of the M1, M2 and M3. For example, patients four and five, with the most severe stroke patients in our study, have the largest correlation between the matrices obtained at week one, week eight, and week seventeen (Table 5 and Table 4). The correlation increases between the corresponding columns as we move from right to left (Table 4).

Although the number of subjects is relatively small to be conclusive, the confidence level of the results is very promising. Moreover, the obtained regression matrices for the patients are independent of each other; their values depend on the severity of each patient. Such result suggests that the increase of the number of patients would not highly affect the relationship between the regression matrices. Finally, it is worth to say that in literature the number of subjects in similar studies is relatively small [13, 33].

The small difference between the measured values of the forces and the calculated values indicates that we could use an unobtrusive device, Microsoft Kinect camera in our case, to find the strength of the forces of the upper-limb of a stroke patient. To the best of our knowledge, this is the first work that finds the regression matrix between the kinematics and the associated forces for stroke patients. The majority of the current approaches so far have used the wearable-based systems to measure these forces. However, our approach will not cause discomfort to the patients while they are preforming their rehabilitation exercises because they do not have to wear tracking devices or forces measuring devices and yet the values of the forces can still be estimated.

In conclusion, although the FSR sensors are part of the system, the patient has to wear them only for a specific period of time until the regression matrix is built. As the results show, the regression matrix can be built within the first week of the rehabilitation period. After that, Kinect camera can be used alone to estimate the strength of the upper limb muscles.

6 Conclusion and future work

This study investigates the relationship between kinematics and forces of the upper limb. A solid mathematical model (the least-square regression matrix) is used to find such relationships. The most innovative component, in addition to the correlation matrix, is the ability to derive the strength of the muscles from the kinematics values through the least square regression matrix. The clinical assessment was, to a certain extent, consistent with the obtained kinematics and forces measurements during the period of the rehabilitation study. However, the time spent on training patient four and patient five on the rehabilitation system before the actual recorded session started was much longer than the rest of the patients: both, patient four (77 years old) and patient five (66 years old), are senior citizens, both lack the rudiments of computer literacy. In our future work, we aim to deploy the whole system in ten patients’ homes and conduct a long term study (1) to determine the accuracy of the system over a long period of time, and (2) to investigate the potential of time series matching algorithms and forecasting algorithms in predicting the progress of the patients.