Keywords

1 Introduction

In the labor sphere, certain tasks are performed repeatedly which entails to that a large number of workers suffering from injuries, increasing the number of people with disabilities and access early retirement [1, 2]. A similar study shows that unnatural postures, irregular or violent movements, body contractions and the repetition of movements produce negative consequences for health [3]. According to global statistics over the course of a year there is a high percentage of fatal work accidents and daily there is much influence of injuries due to occupational accidents [4]. Ergonomic factors within a work environment must govern by rules in order to provide the worker with the necessary comfort to perform their tasks. If these factors are not optimal, they can cause injuries, affecting both the health of the personnel and the productivity of the companies; that is because measures are taken for the recovery of the staff through rehabilitation therapies allowing patients to recover the lost functionalities.

Physical rehabilitation has a positive impact on the quality of life of people, helping them to recover the mobility of their joints and avoiding muscle atrophy [5]. Depending on the technological progress in the field of rehabilitation. Currently there are different therapies such as conventional rehabilitation in which the collaboration of a physiotherapist is needed to supervise the patient’s movements throughout the rehabilitation session; and on the other hand the inclusion of technological devices, e.g. robots or virtual systems, among which several studies can be highlighted, such as a robot that assists in the rehabilitation of stroke patients and a manual assistance robot with multiple degrees of freedom for rehabilitation therapies [6, 7]. Certain service robots contribute to the rehabilitation through compensating the movement, allowing reaching assisted movements and with more repetitions. Others are responsible for providing specific training for the patient to perform the movements correctly, such as virtual systems [7].

The use of virtual environments in rehabilitation therapies often empowers the patient to perform movements that the clinical environment does not allow, helping the patient to feel more interested in their treatment and immersing it in their daily environment [8, 9]. Virtual reality (VR) is an immersion of an individual in a new world created based on real environments, allowing him to interact through the use of virtual devices, e.g. glasses, audio helmets, gloves, movement traction sensors etc. [9, 10]. Some projects that have been proposed in VR work in open loop, i.e., that do not have a feedback system, as a result the rehabilitation phase will depend a lot on the endeavor that the patient is willing to put in order to perform the correct movements [1, 9, 11]. For rehabilitation tasks, it is necessary that the system is fed back in order to confirm that the correct movements are made and the number of repetitions necessary, as can be seen in the diagram of Fig. 1. By implementing a feedback system, the error is reduced by completing the rehabilitation movements of the patient and allowing their recovery to be faster. One of the ways to close the loop is the implementation of sensors that acquire electromyographic signals (EMG) to obtain muscle signals and recognize that the indicated movement was execute.

Fig. 1.
figure 1

Closed loop system

Full size image

As described earlier in this article, the development of a VR system that allows the patients to immerse themselves in a daily or work environment to perform rehabilitation tasks and prevent recovery from being a monotonous activity is proposed. For this a hand functional orthosis will be used, designed in a software CAD (Computer Aided Design) and printed in 3D with flexible materials that allow the comfort that the patient requires [12]. For the feedback system it is proposed to use EMG signals from the Myo Armband sensor which contains eight myoelectric sensors that are sent to a computer via Bluetooth, these EMG signals are processed in real time and each of the gestures are labeled and that are performed with the aim of having a closed loop system and checking that the movements are correct at the time of rehabilitation [13].

This document is divide into five Sections, including the Introduction. The second section presents a general description of the problem and proposed solution. In the third section, it is explained how the project is conformed to the use of a multilayer scheme. The results of the tests after the implementation in the system analyzed in the fourth section, and finally in the fifth section, all the conclusions of the project are presented.

2 Problem Formulation

Upper extremity traumatisms are complex injuries, many of these injuries are due to monotonous tasks and prolonged efforts. Between the diseases by repetitive efforts in the hand exist the CTS (Carpal Tunnel Syndrome), tenosynovitis, trigger finger, among others; At present, this type of lesion is common, which originates in some work environments [14,15,16].

CTS is one of the most common entrapment neuropathies in the upper extremity [17]. Among the symptoms that can be had is severe pain, tingling, numbness and loss of motor control, reducing the ability to grip [18]. The high-pressure damages the blood flow in the median nerve, this pressure increase occurs in the carpal tunnel region shown in Fig. 2a. The treatments of this pathology are complex, which is why many experts agree to perform the following methods: orthoses, corticoids or surgical treatment. Each one exposed according to the severity of the disease [19]. From the presented solutions, it will be necessary to know the movements and postures to achieve the rehabilitation of the hand, either through an active functional orthosis or according to postoperative movements. Several of these movements must be repeated for periods indicated by the physiotherapist. The movements are observed in Fig. 2b.

Fig. 2.
figure 2

Carpal tunnel syndrome

Full size image

As for Quervain’s tenosynovitis, it is defined as one of the main skeletal muscle diseases linked to overload [20]. Described as an acute inflammation in the tendons that occupy the first extensor compartment of the hand, long abductor and short extensor of the thumb. The diagnosis is based on the Finkelstein test shown in Fig. 3a; when closing the fist and performing the maneuver of ulnar deviation, intense pain is perceived [15]. Some of the movements recommended for the recovery of Quervain’s tenosynovitis are shown in Fig. 3b.

Fig. 3.
figure 3

De Quervain’s tenosynovitis

Full size image

The trigger finger or flexor tenositis is a disease that acts on the flexor tendons of the fingers, producing pain and limiting the movement of the finger when flexing or extending the finger as shown in Fig. 4a. This pathology is attributed to factors such as diseases and excessive effort of the hand; however, the causes of appearance of the trigger finger are also due to advanced age and occur more frequently in women than in men [19, 21]. Among the movements that help in the rehabilitation of this trauma, it is important to focus mainly on the affected finger, see Fig. 4b.

Fig. 4.
figure 4

Trigger finger

Full size image

For instance, this work has been previously described the development of an application in VR oriented to fine motor rehabilitation is proposed. The application considers two virtual environments: (i) Industrial process, which has activities related to different areas of work, e.g. engineering, medicine, industry, etc.; and (ii) Rehabilitation game, this environment will have entertainment activities that will help at the same time in the rehabilitation of hand. Each developed environment will allow the patient to execute guided actions through visual instructions, which according to the type of injury present will be informed the movement to be emulated. To achieve the movement, a functional hand orthosis is used, manipulated according to signals obtained from a mathematical algorithm that will classify the signals of the Myo armband. The link of these control processes, which will allow responding to operations of several processes attached to the virtual devices in the Unity environment, see Fig. 5.

Fig. 5.
figure 5

Component interrelation diagram

Full size image

The scene simulation module contains the VR programming, where the 3D model of the hand links the motion controller; this module also presents the necessary configuration of the physical properties, which simulate the rehabilitation movements according to the selected environment.

In the Input Module is consider the Oculus Rift device, which allows observing the virtual environment, in addition to hand tracking devices that allow interaction with the environment such as Leap Motion and Myo Armband. When making use of different input units it is required that the structure of the code is general, being compatible with several platforms, avoiding the reconstruction of the project, managing to detect automatically all the devices.

The SCRIPTS Module is responsible for managing communication with all control blocks, input devices and output devices through the application controller, providing the virtual reality environment with the necessary functionality. In addition, it has the user interface, which allows selecting the rehabilitation environment according to the necessary characteristics of the injury; on the other hand, the motion controller and the Leap Motion Plugin serve as a link to the simulation module. For the interaction of the virtual environment and the patient, there are the voice recognition modules and Oculus VR Plugin. While the TCP module transfers data from the mathematical software.

The Mathematical Software Module is implemented the algorithm of classification of movements that receives the signals of the Myo Armband sensors, processes them and determines the movement made by the patient sending signals through the serial module to a controller that will activate the motors of the orthosis hand.

Finally, in the Output Module, the feedback is produced by means of the orthosis that moves according to the predicted rehabilitation exercise. It also provides audio and visual feedback to the patient to monitor movements within the VR environment.

3 Virtual Interaction and Immersion

CAD software is a design tool that allows users to create virtual mechanisms through an apprehensible graphical environment. An example is SolidWorks, which is a package of mechanical tools that allows the modeling of objects in 2D and 3D; it also has CAE (Computer Aided Engineering) tools, which allow to perform analyzes and simulations of the model designed with real parameters [22].

Within the present document, a multi-layered scheme proposes for the development of the application in a virtual environment with the purpose of providing greater immersion and interaction to patients in rehabilitation tasks for hand injuries caused by repetitive activities see Fig. 6.

Fig. 6.
figure 6

Virtualization model

Full size image

Layer 2 establishes the relationship between the 3D model and the Unity3D graphics engine 3DS Max software is used. The interaction process establishes that hierarchies of the model must be obtained according to the elements, positions and restrictions of the designed object, it is also necessary to determine reference points (pivot) that will serve as location and orientation of the model. The model visualized in the virtual environment is that of a hand, the model of the orthosis is not used so that the patient has greater sensitivity when observing a real hand. Finally, the process performed in the link software will have an .fbx output file, which is compatible with the Unity3D software, allowing applications to be developed for the VR environment. One of the applications is an industrial environment, specifically a tannery, in which the patient can develop work activities while rehabilitating his hand; another application has to do with motor games designed.

On the other hand, Layer 3 performs the configuration of inputs for the interaction of the moving parts that make up the 3D model and the user interface using the Oculus Rift device for the generation of the virtual environment, while the Leap motion device tracks the position of the hand at the moment of making movements by the patient and transfers them to the virtual environment obtaining a visual feedback on the rehabilitation movements already established, see Fig. 7. To the Hand Movement Execution block comes the signal coming from the mathematical algorithm in order to that in the virtual environment the movements made are visualized; the App Controller block ensures that the mathematical algorithm classifies the signals in order to complete the desired movement, this block also contains the instructions for the movements to be made, for which the user accesses by means of the voice command “Help” and a panel is show in which is indicated the movement that the patient must make step by step. Once the instructions are understood, the voice command “Close” is used, the instruction window is hidden, and rehabilitation begins.

Fig. 7.
figure 7

System device

Full size image

In Layer 4 the communication of the Myo Armband is developed, whose function is to obtain signals from the sensors and send them to the computer, where the algorithm predicts the movement executed according to the training and programming carried out in the mathematical software. These signals are sent to the motors of the orthosis in order to assist in the different movements to be performed by the patient. Layer 4 can be subdivided into 3 stages, in the first stage the Myo Armband is described, which is a non-invasive surface sensor that records the electrical signals of the muscles through its 8 sensors, see Fig. 8. These signals are sampled at a speed of 200 Hz and with a resolution of 8 bits for each sensor; these signals are taken from the muscles of the forearm because these are responsible for the movements of the different parts of the hand. The Myo Armband has an inertial measurement unit with 9 degrees of freedom (accelerometer, gyroscope and orientation, in the three axes X, Y and Z); all registered data is sent via Bluetooth to the computer. Both amplitude and spectrum of the signals coming from the armband vary according to the thickness of the skin, its temperature, the blood flow and the percentage of body fat of the person, it will also depend on the location of the sensors. Fatigue, neuromuscular diseases and aging affect the quality of electromyographic signals.

Fig. 8.
figure 8

Surface sensor: Myo Armband.

Full size image

Regarding the second stage, the mathematical algorithm of classification of movements was developed that was carried out through five phases: 1. Acquisition of signals, made by the Myo armband; 2. Preprocessing, in this phase the signal obtained is rectified to avoid that in the average sum of each channel of the signal is zero, in addition a low pass filter is applied to reduce the noise and soften the channels; 3. Characteristics extraction, here we define a characteristics matrix that contains the 200 samples of each channel, these samples are obtained after observing the EMG signals through a 1 s window with a sampling frequency of 200 Hz; 4. Classification, for this phase the rule of the nearest k-neighbors is used, which is a supervised classification method together with the algorithm of dynamic temporal alignment that measures similarities between sequences that vary in time; 5. Post Processing, in this last phase the model allows to classify the superimposed windows of the same gesture by delivering different labels [13]. For its training, the rehabilitation movements of Figs. 2b, 3b and 4b were used, with this a matrix is established according to the environment, disability and movement to be performed, presented in Table 1.

Table 1. Rehabilitation movements according to the disability and environment present.
Full size table

Finally, in the last stage, a functional orthosis is used which is designed in CAD software, based on anthropometric measurements of the hand to provide the necessary ergonomics to the patient; It is considered that the measurements of the fingers vary from one person to another and proceeds to make a design that allows an easy exchange of the fingers of the orthosis in order to have a correct fit on the patient’s fingers, see Fig. 9. The orthosis was printed in 3D for which it is considered a preliminary study on the characterization of flexible materials printed in 3D, within which standardized test pieces were used based on ASTM D638, these test pieces were printed from a thermoplastic elastomer (NinjaFlex) and the testing it was carried out by means of a Universal Testing Machine MT-50, being able to reach the conclusion that the best printing direction to withstand greater efforts and forces is in the XZ plane [12].

Fig. 9.
figure 9

Modelo CAD de la Órtesis

Full size image

4 Results and Discussion

In this section, the results obtained from the project are presented, for which first the Classification of Movements is exposed, where the interface of the mathematical algorithm that allows classifying the signals coming from the Myo Armband is shown, in addition, the Virtual Application is shown, which contains two virtual environments, which offers the patient a more interactive rehabilitation session motivating the patient with their treatment and recovery.

4.1 Classification of Movements

The classifier of movements has a graphical interface, which has three modules: Initialization, Control and Display. The Initialization Module allows entering user information e.g. name, age, gender, measurement of the perimeter of the arm, distance from the ulna to the elbow and distance of the Myo Armband to the elbow. This information will be used to create a database, used in the training of the classifier.

The Control Module has four buttons: Start, it initialize the program and store the user’s data in the database; Restart, it allows access to the database and continue with the collection of signals from a previously entered user; Record, it capture the signals from the Myo Armband with a sampling frequency of 200 Hz; Repeat, it erase the signals of the last captured movement.

To observe the captured signals, the interface has the Display Module, it consisting of 8 windows that indicate the signal of each sensor, see Fig. 10.

Fig. 10.
figure 10

Graphical interface for training

Full size image

In the training, 50 people are considered to execute 4 gestures: Fist, Open, IMC and RPC. The parameters for the training are in the Initialization Module, having 50 samples by default for each gesture with a recording time of 5 s, plus 10 samples of the movement called Relax, that consists of not realize any action, necessary for training the classification algorithm since it allows to distinguish when each movement begins.

The windows of the signals recorded according to each movement are shown in Fig. 11. Wave In and Wave Out movements are not trained as they are controlled in open loop.

Fig. 11.
figure 11

Signals obtained from each gesture

Full size image

4.2 Virtual Application

Displays the virtual environment that offers the patient two different applications, the one an industrial process for which it was considered a tannery of hides and the other a game of rehabilitation that allow to perform tasks and exercises for rehabilitation of hand injured because of repetitive exercises, Fig. 12a shows the application start up screen. The onscreen Game and Industrial Process accessing an information panel which gives to know which activities can make the patient within the selected scenario, see Fig. 12b.

Fig. 12.
figure 12

Virtual Application

Full size image

4.2.1 Rehabilitation Game

This application allows entertainment, also a better rehabilitation for the patient for the reason that it is not restricted movements focused on activities of an industrial process, the rehabilitation game includes movements such as IMC, and RPC to treat injuries of CTS and Trigger Finger, see Fig. 13.

Fig. 13.
figure 13

Start screen of the rehabilitation game

Full size image

The environment is a musical video game application in which the patient must perform the specified movement the exact moment the signal is shown on the screen, so that the patient performs, the movement is given a time of 5 to 10 s in order that the movement is completed correctly and can continue to the next movement. The application begins to accumulate points in the case of a correct movement with the aim of presenting at the end of the exercise a table where the progress of the patient is observed, see Fig. 14.

Fig. 14.
figure 14

Rehabilitation game

Full size image

4.2.2 Industrial Process: Tannery

Once entered into the industrial process, as in the game, the panel with the activities of the applications that the patient can perform is shown first, see Fig. 15a. The following screen shows the industrial environment: a tannery where the process of transformation of animal skins to leather is simulated, in Fig. 15b the operations carried out in the tannery where it includes the Leather Squeeze Machine, drums and actuators are shown for each process [23]. The environment also has the respective instructions to comply with the activity.

Fig. 15.
figure 15

Virtual application

Full size image

The activity proposed in this environment has to do with the activation of the Leather Squeeze machine responsible for the process of eliminating excess fat in the leather, this activity consists of four actions in which rehabilitation movements are developed. The actions are: open the process drive box, press the start button, press the stop button to finish the tanning process and close the drive box. Figure 16 shows the activity described.

Fig. 16.
figure 16

Industrial environment

Full size image

5 Conclusions

The interaction of the patient with an environment different from the conventional one produces motivation to carry out the rehabilitation sessions, since they are entertaining activities that help the patient in their progress. That is why it was necessary to create the mathematical algorithm of classification of movements, having to perform gestures such as RPC and IMC that are not recognized by the Myo Armband software; the algorithm has a high gesture recognition accuracy, but it can be improved by training it with a greater number of users. In addition, the virtual rehabilitation environment demonstrates a good alternative for rehabilitation sessions, since it allows the user to correct their fine motor skills and reduce the progress of injuries inherent to the patient.