Keywords

1 Introduction

The considerable number of cases of patients with brain injuries that lead to problems of the musculoskeletal system shows that rehabilitation fulfills a necessary and important function, since the lack of movement induces a high risk of contracting metabolic syndromes and chronic diseases that decline the quality of life [1,2,3,4,5,6]. A neurorehabilitation entails exploiting motor learning by involving the patient in an intervention that generates favorable conditions to stimulate the modification of the residual neural networks of the brain [3, 7, 8]; therapeutic exercise techniques have been shown to correct deterioration and improve the functions of the musculoskeletal system in a variety of disease states. Some exercises performed during a rehabilitation process are based on natural movements of joints that are intended to strengthen or maintain the existing strength of specific muscles; this may delay the progression of weakness or proactively prevent painful musculoskeletal syndromes associated with lack of mobility and atrophy due to disuse [2, 6, 9, 10].

Studies indicate that traditional physical therapy processes, which include strengthening exercises, have been successful in cases of brain damage, however, fundamental problems have been identified that may prevent achieving the goal of rehabilitation. The most notorious ones are related to the fact that they do not consider the individual differences in motor and cognitive capacity that can influence differently the reaction of each patient, as well as being a complex and long-term process with repetitive exercises [6,7,8,9, 11, 12]. As a consequence, psychological problems may arise in the patient that diminish his motivation and interest in performing the exercises, this becomes even more problematic when dealing with children, who generally lack enthusiasm in participating in the tedious process of rehabilitation. Because of this inconvenience, it becomes necessary to implement alternative solutions that motivate patients to actively perform the exercises [8, 12,13,14,15].

In recent years there has been a growth in research and development of technological systems that aim to help the patient in the rehabilitation process; these include complex and expensive systems such as the implementation of robotic systems, as well as low-cost systems such as the development of motion-based games through motion sensor technology that stimulate the patient to participate in the exercises intentionally caused by the games. Systems with virtual environments can be used to optimize motor learning because it is possible to explicitly involve motivation and learning mechanisms based on sensory feedback [4, 7, 8, 10, 11, 13, 14, 16,17,18]. Added to this is the way in which users interact with the game with the integration of non-invasive and low-cost devices such as the Microsoft Kinect, which captures skeletal movement without compromising the freedom of limb movement [3, 5, 6, 15, 19,20,21]. Several applications are presented using this device, e.g., related to evaluation and postural control, identification of injuries, improving mobility, etc. Many of these applications are complemented by methods of classification and recognition of gestures or body postures for the fulfillment of their purposes, achieving satisfactory results [1, 3, 12, 14, 17, 21, 22].

In this context, this paper presents the development of a virtual assistance system in the rehabilitation process focused on the strengthening of lower extremities, which is not intended to replace the work and supervision of the expert professional. This low-cost system uses the Microsoft Kinect 2.0 as a unique and main input device, which digitally captures in real time the movements of the user while he exercises the muscles of his lower limbs with an exercise machine approved by the expert. The digital data of the movements of the body of the user are processed by the algorithms implemented so that the virtual interface recognizes the exercises and executes graphic animation subroutines. In the implementation of the virtual interface are considered the computer-aided design, CAD, and graphics engines as a complement of Unity software that allow to obtain graphics and animations of high quality in 3D virtual games. Each of the 3D virtual games developed have three levels of complexity. These characteristics make the use of the application attractive and intuitive, so the user will perceive that he performs an activity that emulates a real situation, which requires and demands the concentration of his mind and the movement of his extremities to be completed successfully. Additionally, the system is complemented with subroutines for recording relevant data in a local server, which allow the rehabilitator to analyze the progress of the therapies. Figure 1 indicates the components and functionality of the proposed system.

Fig. 1.
figure 1

General scheme of the proposed virtual system that uses the Microsoft Kinect device.

Full size image

The work is organized in IV Sections including the Introduction. Section 2 presents the development of the system based on the use of Unity and the Microsoft Kinect 2.0 SDK to create the intuitive virtual interface and the algorithms, respectively. The experimental results and discussion that validate the implementation, functionality and the achieves of the proposal system are presented in Sect. 3. Finally, the conclusions are found in Sect. 4.

2 Development of the Virtual Rehabilitation System

2.1 Virtual System Operation

The system is designed to be used by one user at a time. The Kinect 2.0 captures the movements of the user while exercising using an exercise machine. The digital data corresponding to the movements of the user are used in an algorithm that recognizes when an exercise has been properly performed. The recognition of exercises is used to execute movement animations of 3D graphics in a virtual interface developed and programmed in the Unity software. In this way, virtual applications are created that monitor and stimulate the patient to properly perform rehabilitation activities, while capturing their attention with game objectives and positive reinforcement.

In Fig. 2, the flow diagram of the general operation of the implemented application is indicated. The system is structured mainly in the functions of the routines programmed in C# language in the Unity software, where a virtual interface is developed, which is the stage of the system that users perceive and interact with. The rehabilitation user intervenes, who defines the parameters that govern the difficulty and the objectives of the virtual game, and the patient user, who performs his rehabilitation session through the challenges required by the virtual game.

Fig. 2.
figure 2

General operation scheme of the algorithm of the proposed virtual system.

Full size image

The operation of the system is fundamentally based on the subroutine of recognition of movements of the body of the user. The Microsoft Kinect 2.0 device, which acts as a single input device, acquires the data of the 25 points of the body that it is able to identify, these data are processed as quaternions values that define the rotation of each point in a three-dimensional space. The position and position variations of these quaternions are compared with data from a predefined pattern gesture, which are found in a created gesture database that is accessed through the Microsoft Kinect 2.0 SDK. The rest of the subroutines correspond to animations, data logging, game administration, menus, 3D object collision detectors, among others.

In Unity, a programming based on classes and objects is handled, which is why the so-called Game Objects are created, which are associated with scripts in C# that control and define the operation of the application. Several Game Objects depend on the data and the operation of others, which resembles a parallel execution. In a Unity project you will also find software packages necessary for the compilation of the application. A more specific outline of the elements that operate in Unity is shown in Fig. 3.

Fig. 3.
figure 3

Specific diagram of the hardware and software operation in the virtual system developed in Unity.

Full size image

The acquisition of data from the Microsoft Kinect input device occurs through the Body Tracking object, this object is associated with Unity-specific Kinect Plugins that allow the management of the functions associated with the official Microsoft Kinect SDK. In addition, the Body Tracking object is directly related to the animations control scripts of the 3D characters that operate according to the captured movements of the user.

The main object, Custom Gesture Manager, is responsible for accessing the database of gestures, processing the comparison and providing the results of whether a predefined gesture has been identified (built in the Visual Gesture Builder software) to the other objects that depend on that result to execute their routines. The database of gestures corresponds to a *.gbd file that must be stored in the Unity project, specifically in a folder called StreamingAssets. It is also necessary to add the Kinect VisualGestureBuilder Plugin of Unity in the project for the compilation of the codes.

The objects that contain their respective scripts that control the animations have been modeled with the Blender software, these 3D objects, like the avatars, are characterized mainly by having a structure or skeleton that allows the individual rotation of their joints. According to the gesture recognition phase, actions such as the rotation and translation of 3D objects in three-dimensional space, the appearance of sounds, or executions of animation frames configured in the Blender software are produced.

The interaction between the application and the user is also produced by the objects of type Canvas that write on the screen data and information messages and motivation, as well as read instructions and input data and instructions through buttons and text boxes.

By last, an object is oriented only to the execution of scripts that manage the insertion of data, generated in the game, creating a communication channel with a script *.php, which runs on a local server (managed and created by XAMPP software) and directly controls a MySQL database from the same local server. The fields of the database and the scores in the games have been implemented based on the analysis of which data are useful for recording the progress of the patient in each rehabilitation session. These correspond to the number of correct exercises accounted for, the times in which the challenges have been completed, the number of correct and incorrect decisions as appropriate, date and time of the session; as well as patient data, name and surname, identity card, age and pathology presented.

2.2 Gestural Recognition Through Microsoft Kinect 2.0

Each implemented virtual game is based on a specific exercise, so that the application interprets these exercises as Boolean commands in a software subroutine, it is necessary to decompose the exercise into different movements that will be interpreted as a set of gestures reproduced in an orderly and periodic way. In this work, machine learning has been implemented through the use of Microsoft Visual Gesture Builder software, VGB, where it is necessary to load pre-recorded files of the gestures to be recognized. The Fig. 4 indicates a sequence of images of the learning process in the VGB software of the three exercises considered in this work: movement of the legs alternately as a pedaling while sitting (left pedaling and right pedaling), in a similar position but pedaling with the legs together (gestures legs up and legs together below) and in a lying position on the back performing an alternate movement of the legs similar to pedaling (left leg down and right leg down).

Fig. 4.
figure 4

Utilization of the Microsoft VGB software to build the gestures to be acknowledged by the games.

Full size image

Next, a portion of the code that is executed in an object in the application made in Unity is indicated to access the database (*.gbd), where the information of each gesture is stored. When accessing the database of gestures, a frame captured by Kinect 2.0 is entered, which contains data on the skeletal movement of the user, so that the comparison with the data of each pre-established gesture is executed.

figure a

The next portion of the code that is presented corresponds to the access to the result of the comparison that has occurred. The result is presented with a floating confidence value between 0 and 1. It is necessary to extract the confidence value of each one of the gestures that form an exercise, in this way it can be identified which of them has happened or has not happened to later execute the corresponding animations and other subroutines of the application.

figure b

2.3 Virtual Interface Functionalities

The virtual environments have been designed with children’s content, i.e., with colorful, eye-catching and animated 3D graphics, so that the games promote concentration in the virtual activity that the three-dimensional characters are performing, trying to make the experience of performing the exercises entertaining. Each virtual game has different objectives; challenges; animations; indicators of scores, errors and time; informative and stimulating messages; sounds controlled by the corresponding algorithms of the application.

The application also has intuitive user menus, where you can define parameters such as the desired level of the game and/or change the number of repetitions of exercises as appropriate; so too with Canvas that contain buttons that command the data record of the game session in the database that operates with the XAMPP software.

In the Fig. 5 the corresponding environment is indicated for the game that is designed for the user to relate his presence to that of an avatar that goes on a bicycle, as the avatar pedals an advance of the bicycle towards the front takes place. Animation scripts validate if the exercise has been completed and is being done correctly, i.e., if it is pedaled alternately, so that if the exercises are done improperly the bicycle does not advance and even slows down when a new correct exercise reading has not been recorded.

Fig. 5.
figure 5

Virtual environment of the game 1 (bike).

Full size image

In the second virtual game, indicated in Fig. 6, we have included objects, textures and animations that give the effect of water movement in an artificial lagoon on which a pedal boat sails. The animations that cause the avatar to push and contract the pedals are related to those produced by the advance of the boat towards the front and the effect of the navigation movement. In the same way the exercises are validated through the reading of gestures, so that the braking animations of the boat are produced when the correct execution of the exercise is not identified.

Fig. 6.
figure 6

Virtual environment of the game 2 (pedal boat).

Full size image

Finally, the swimmer’s game consists of an avatar that is in a pool which reproduces animations of movement of his limbs imitating the action of swimming on his back. The effects of movement and animation are based on the method applied in the two games described above. The environment developed in Unity of the virtual game of the swimmer is indicated in Fig. 7.

Fig. 7.
figure 7

Virtual environment of the game 2 (swimmer).

Full size image

In all the games, the main objects (bicycle-avatar, boat-avatar, swimmer avatar) are moved by defined routes within the virtual environment. The control of movement from one point to another is done through the identification of collisions of the main characters with invisible objects that are placed at strategic points of the three-dimensional environment.

3 Results and Discussion

The first results of the project presented in this section are divided in three aspects, the first one is related to the experimentation of the proposed virtual system, i.e., it is shown the software routines functioning of the virtual application. Secondly, the results of a test applied to the users who participated in the test of the proposed system are presented and analyzed. Finally, it is realized an analysis of the exercises that the proposed system motivates to the user to do, and the benefits that these exercises represent in a rehabilitation process.

3.1 Experimentation of the Virtual Rehabilitation System

For the execution of the proposed system is used a computer with the following features: Intel core i7-7500 2.9 GHz of seventh generation, 16 GB of RAM, a Graphic Card Intel HD 620, Windows 10 Home operative system of 64 bits. The implemented algorithms and animations of the application run satisfactorily, however, the response of the execution can be improved with a computer with advanced features, specially related to the graphic card.

The user is located in front of the Microsoft Kinect 2.0 positioning to perform an exercise with the Woder Core Smart machine. At the beginning it may be necessary to wait a while until the Kinect device recognizes the body shape of the user and tracking occurs, an indicator of this action will appear on the screen, and at that time the game can be executed by pressing a start button. The rehabilitator will be guiding the patient at all times, and he will define the repetitions, times, challenges and/or levels of play using the interactive and intuitive menu that is part of the application. The menu also allows you to select one of the three virtual games and their corresponding levels that are presented in this experiment.

The first virtual game contains an environment of a park with a cycling track, an avatar is on a bicycle, as the user performs the exercise of pedaling (alternating left and right legs) the animations and the progress of the bicycle until a defined route is fulfilled; the application measures time and counts the exercises performed by each lower limb. The sequences of images of the operation of virtual game 1 are presented in Fig. 8.

Fig. 8.
figure 8

Images sequence of the game 1 functioning.

Full size image

The second virtual game consists of driving a pedal boat. There is a virtual environment of an artificial lagoon in a park; the move of the boat occurs when the user performs the exercise of stretching and contracting both legs at the same time using the exercise machine, as shown in Fig. 9. The application identifies and records the number of exercises performed correctly and the time it takes for a complete tour.

Fig. 9.
figure 9

Images sequence of the game 2 functioning.

Full size image

The third virtual game contains an avatar that must swim backwards and meet a certain number of laps in the pool. The virtual environment is designed so that the user perceives in a visual and sound way a pool surrounded by nature. The exercises that the user performs with the exercise machine resemble the movements performed by the legs during swimming. When performing the exercise correctly, alternating the two legs, the animation that allows you to advance to the virtual avatar is produced, as shown by the sequences of images in Fig. 10. The application identifies and records the number of exercises performed by each leg in a way correct, the number of objectives achieved (laps in the pool) and the time in which this objective is completed.

Fig. 10.
figure 10

Images sequence of the game 3 functioning.

Full size image

3.2 Usability of the Virtual Rehabilitation System

The weighting of the usability of the system has been obtained from a survey of users, which correspond to five children aged between seven and eleven years (the results of each question are presented in Table 1). The questions have been formulated according to a child’s level of comprehension, i.e., simple, concrete and easy to understand questions that only have the option to choose one of the five answers (bad, regular, acceptable, good, excellent). Specifically, the orientation of the survey is centered on three relevant aspects: (i) The characteristics and operation of the application, which refers to whether environments, animations, 3D graphics, sounds, menus and indicator messages were considered intuitive, pleasant and did not generate annoyance. (ii) The level of concentration and attention that the application captures, i.e., if the challenges and environments generate some level of immersion that focuses the great part of the attention of the user on the video game while exercising. (iii) The opinion of the user regarding the utility and the benefits that the proposed system can provide.

Table 1. Result of the applied usability test.
Full size table

Each question has a maximum score of 4 points (excellent) and a minimum of 0 points (bad), so that the maximum weight of the survey is given over 40 points. With the five surveys carried out, as a result of the usability of the system, an average weight of 37.6 is obtained, which corresponds to 94% of the ideal score defined in this work.

The score reflects the acceptance of the usability of the system by the users who participated in the experiment; in addition, the results of the survey indicate that the participants agree that the methods of rehabilitation, presented in this work, can positively stimulate the correct performance of rehabilitation activities by promoting entertainment and eliminating the monotony of traditional methods. In contrast, minimum indices have been obtained that the aspect to be improved corresponds to increasing the level of immersion of the user in the virtual environment, provided that the objective is to obtain a higher level of capture of the concentration of the user in the game, which it does not allow the patient to think that he is doing a therapy session.

3.3 Discussion of the Rehabilitation Benefits Achieved

The most important aspect to highlight corresponds to the impact of the positive reinforcement generated by the proposed system in the patient, since it is evident, through the results of the survey, that rehabilitation therapies can be perceived as a video game that avoids monotony and the consequences of tiredness that conventional methodologies can generate, which could lead to psychological problems. In addition, in the experiment performed that tests the operation of the proposed system, users were consulted about the areas of their extremities where they felt the physical effort. Based on this evaluation, the muscles involved in the movements of the limbs that stimulate the virtual games implemented were analyzed.

In the game of the bicycle (Fig. 8) the muscles of the thigh are exercised in the zones corresponding to the sartorius, long adductor, rectus femur, and the greatest effort is concentrated in the femoral quadriceps; while the involved muscles of the leg are biceps femoris, external twin, internal twin, sinew of the twins, peroneus longus, and the ligament of the patella is included. In the pedal boat game (Fig. 9) the same muscles described above are involved, however, the effort is intensified by exercising both legs at the same time. Finally, in the swimmer’s game (Fig. 10) the thigh muscles corresponding to the femur square, adductor minimums, adductor magnus are exercised, and the greatest intensity is felt in the femoral biceps.

In addition to the muscles that benefit from the exercises that the proposed system induces in the experimentation, it is important to consider that there is no probability of generating repercussions in the joints involved in the movement of the exercise, because they do not suffer impact and must not support excessive weights.

4 Conclusions

This work presents the implementation of a non-invasive system based on a virtual application, which is programmed to assist and stimulate the patient in rehabilitation therapies that contemplate the muscular strengthening of upper and lower extremities. In the experimentation the correct accomplishment of the exercises that stimulate the effort of groups of muscles of thighs and legs has been promoted. The virtual application, developed in the Unity software, contains games with environments and intuitive, high-quality three-dimensional graphics that are animated to emulate a real activity with challenging objectives, this allows capturing the patient’s attention and motivating them to correctly carry out rehabilitation activities, according to the affirmation of the result of the usability test applied to 5 users, which reached an acceptability of 94%. The use of the Microsoft Kinect device to digitally capture the movement of the body of the user, and the algorithms implemented based on machine learning, which is enhanced by Microsoft Visual Gesture Builder software, has the advantage of immediately implementing different exercises, decomposing them as different gestures of the body to be recognized in the corresponding subroutines. As future work, adding animations, subroutines and devices that increase the level of immersion of the application that the user perceives is proposed.