Keywords

1 Introduction

In this days and age, in early childhood centers and in inquiry pediatrics is frequent to find boys with several problems in orofacial region, it is due to childhood neurological pathologies such us: cerebral palsy (CP), syndromes or simply because the infant was born premature [1]. On the one hand, in the different pathologies, voluntary movements are carried out through extensor and hypertonic expressive reactions, that slow or prevent the free movement of the joints, causing permanent atrophies in certain muscle groups of the face. On the other hand, the main characteristic of these syndromes is its muscular hypotonia that affect the following areas: labial, lingual, pharyngeal and the whole of the orofacial muscles; in both cases, therapy is recommended to regulate the proper functioning of the orofacial area [2, 3].

First of all, among the affectations that can occur in children by not having a good functioning of the orofacial area, are: disorders in breathing, chewing, swallowing, poor diet and language [4, 5]. Feeding is essential for the growth of an infant, but in most cases due to the affectation they have at the neurological level, they can present poor nutrition. For instance, within the comorbidities that accompany CP, eating disorders. and swallowing (SWT) are prominent as a source of morbidity and mortality [6, 7]. Researches carried out show high percentages of prevalence of dysphagia in this condition: 43% [8], and 99% [9]. Diet allows child to consume food adequately, achieving progress in weight and height or in turn maintaining them [10], If there is a change in feeding, it may be due to any difficulty in sucking, biting, chewing, handling food in the oral cavity, controlling spittle, and swallowing [11].

In addition, Development of oral language in the early childhood education stage is very important, since it is the instrument that will allow children to carry out a satisfactory school learning, on which all subsequent knowledge will be based [12]. Moreover, orofacial exercises are part of a speech therapy and seek to stimulate vibration, and stretch of muscles, to strengthen the weak functions of language and facilitate feeding as it works on sucking, swallowing and chewing [13]. Orofacial exercises include gesticulation, vocalization, breath exercises, absorption, lip and tongue. Therapists devise a series of games for children to improve their orofacial motor skills, like: use of the dice to imitate facial gestures of an image, blowing bubbles, inflating the cheeks are the most common exercises in orofacial therapy [14,15,16].

Nowadays, state-of-the-art technologies have been gaining ground in terms of virtual rehabilitation focused mainly on the upper and lower limbs [17,18,19,20], orofacial therapies have few studies [21,22,23] and some research are focus specifically on tongue training [24]. The use of devices such as the Kinect has given good results in oral rehabilitation and in tongue training specially for patients suffering from dysphagia and dysarthria [25, 26], as it is a non-invasive device, it allows to the systems developed for virtual rehabilitation may be used by any patient with different pathologies or syndromes [25,26,27].

In this context, the present work presents the development of a virtual system that uses the Microsoft Kinect 2.0 as a single and main input device which digitally captures the movements of the user’s face in real time. Furthermore, digital data of the user’s face movements is processed by the algorithms implemented so that the virtual interface recognizes defined facial gestures and run subroutines of graphical animations. The implementation of the virtual interface is based on the use of computer-aided design, CAD, and graphics engines as a complement to Unity software for high-quality graphics and animations in 3D virtual games controlled by facial gestures. These features allow the reproduction of attractive and intuitive games that catch the patient’s attention so that the orofacial physiotherapy process is not monotonous. Besides, the system consists of software routines that allow the selection of levels of complexity and the storage of the data of the results of the rehabilitation session, as a function that facilitates, to the expert professional, the future analysis of the patient’s evolution.

This work is organized in IV Sections including the Introduction. Section 2 presents the development of the algorithms used in the system based on the use of Microsoft Kinect 2.0 SDK for face tracking and Unity functions. Section 3, presented the experimental results and discussion that validate the implementation, functionality and the achieves of the proposed system, and finally in Sect. 4, are the conclusions.

Fig. 1.
figure 1

General scheme of the proposed virtual system.

Full size image

2 Development of the Virtual Orofacial Physiotherapy System

2.1 Virtual System Operation

The main element of the system is the face of an avatar, which has been designed to interact with the user. The Kinect 2.0 device captures the movements of the user’s face while mimicking the avatar, the general scheme of the proposed virtual system is shown in Fig. 1. The data corresponding to the movements of the face are used in an algorithm that recognizes when a facial gesture has been made correctly. Facial gesture recognition is used to execute 3D graphic movement animations in a virtual interface developed and programmed in the Unity software. The virtual applications created allow monitor and stimulate the user to exercise the muscles of his face by performing different gestures, meanwhile at the same time capturing their attention in the fulfilment of goals and positive feedback.

The routines programmed in the Unity software based in C # language is the main structure of the system. The virtual interface developed allows the system to interact with users. The rehabilitation user is involved, who defines the parameters governing the difficulty and objectives of the virtual game, and the patient user, who carries out his rehabilitation session through the challenges required by the virtual game.

With the help of the Microsoft Kinect 2.0 device, data from strategic points on the user’s face is acquired. The subroutine implemented for the operation of the system allows the recognition of the human face. The variations of the facial expressions made by the user are compared with the data of the movement of the face of a virtual avatar. Animations, menus, user data log, game options, 3D object collision detectors, and others, are found in the rest of the subroutines.

In the software Unity, the so-called Play Objects are created, which is nothing more than using a programming based on classes and objects, they are also associated with C # scripts that define and control the operation of the game. The data and some objects of the game depend on the operation of another game, similar to a parallel execution.

2.2 Recognition of Face Gestures Through Microsoft Kinect 2.0

The application is designed to promote the movement of a person’s facial muscles in an entertaining way; Microsoft Kinect 2.0 device is used to capture the movement of strategic points on the human face in real time which are associated with the movements of a virtual avatar’s face. The person will exercise the muscles of their face by making different gestures (smile, sadness, surprise, open your mouth, raise your eyebrows, among others) to overcome the series of challenges contained in virtual games.

A mesh of 1347 points or vertices, which forms the face of the avatar, to associate them with the points that Microsoft Kinect 2.0 is capable of reading. Figure 2 illustrates the associated points.

Fig. 2.
figure 2

Points or vertices, which form the face of the avatar

Full size image

Algorithms are implemented to associate the data provided with the Kinect of human face points with the movements of the avatar’s face area; algorithms that recognize that the person is making a certain (smile, sadness, surprise, open your mouth, raise your eyebrows, among others), algorithms that count hits and misses.

figure a

2.3 Virtual Interface Functionalities

The virtual environment has been designed based on content for children, it has very colorful and animated 3D graphics, the three-dimensional games and characters easily capture the child’s attention, this causes their experience with the system to be lively and entertaining and also motivates them to children perform facial exercises from a traditional therapy. In each virtual game there are different challenges and objectives to be met, the animations, score or error time indicators depend on each game environment, as well as positive feedback, informative and stimulating messages are shown. Sounds are generated in each environment, which is feedback controlled by application algorithms.

Figure 3 Shows the main menu that the application has, it is divided into 3 sections: the registration of user data, the indication of date and time and finally the selection of games; each game in turn has intuitive menus where you can define parameters such as the desired level and certain game actions.

Fig. 3.
figure 3

Virtual system main menu

Full size image

Figure 4 shows the game (i) related to mimic facial gestures, it is designed to capture the user’s attention so that they relate their presence to a child’s avatar as if it were a virtual mirror, so that they forget that you are in a therapy session and concentrate only on imitating the gestures requested by the game. The aim of the game is to pay attention to the gesture requested and to carry it out. All requested gestures must be completed in the shortest possible time. The game counts the number of gestures completed correctly (mark as correct), as well as the life of the game which decreases and decreases faster if the requested gesture is not made (marked as incorrect). The patient (game user) must stand in front of the Kinect device for the game to start automatically. Randomly requested facial gesture indicators will appear and the person should imitate them until a green bar is one hundred percent full.

Fig. 4.
figure 4

Virtual environment of the game of imitating facial gestures

Full size image

The set (ii) shown in Fig. 5, it is designed so that the user controls the movements of the bicycle (front, left and right) by making certain facial gestures, so that they collect as many prizes as possible, avoid obstacles and reach the goal in the shortest possible time.

Fig. 5.
figure 5

Virtual environment of the game to control a bicycle by facial gestures

Full size image

The game (iii) that controls a hot-air balloon, is designed so that the user controls the movements of the hot-air balloon (lifting, front, back, left and right) by doing certain facial gestures, so that they should collect the requested number of prizes in the shortest possible time.

The patient (game user) must stand in front of the Kinect device for the game to start automatically. They must perform the gestures configured to control the direction of movement of the hot-air balloon and meet the objective of the game. Also, when they are touching the prize, they must make the gesture configured to collect it.

The gambling will validate if the exercise has been completed and is being done correctly and counts this as a success as well as the time it takes for the hot-air balloon to reach the finish line. Game environment is shown in Fig. 6.

Fig. 6.
figure 6

Virtual environment of the game of controlling a hot-air balloon by facial gestures

Full size image

The movement control of the main objects (avatar, bicycle and balloon) of each game is associated with the data provided with the Kinect of points of the human face and with the movements of the avatar’s face area, whose recognize that the person is making a certain gesture (smile, sadness, surprise, opening the mouth, raising the eyebrows, among others).

3 Results and Discussion

Currently, virtual reality has been gaining ground as a complement to traditional rehabilitation therapies [21,22,23]. This section shows the initial results of the work divided into three aspects: the first shows the operation of each game and the related facial movements. In the second section, the results obtained from the application of the SUS usability test to six children are analyzed. Finally, it is analyzed whether the application is entertaining and motivates users to perform the exercises correctly, generating benefits for the rehabilitation process.

3.1 Experimentation of the Virtual Rehabilitation System

A computer with the following characteristics is used to run the proposed system: Intel core i7–7500 2.9 GHz seventh generation, 16 GB RAM, an Intel HD 620 graphics card, 64-bit Windows 10 Home operating system. The implemented algorithms and animations of the application are executed satisfactorily, however, the response of the execution can be improved with a computer of advanced characteristics, especially related to the graphic card [17].

Once the user is positioned in front of Microsoft Kinect 2.0 to perform the exercise, it may be necessary to wait for some time until the Kinect device recognizes the user’s facial gestures and a follow-up is performed. The game will start automatically, activity carried out by the patient will be guided by the therapist at all times and will define the repetitions, challenges and/or levels of the play through the interactive and intuitive menu that is part of the application. The menu also allows you to select one of the three virtual games and their corresponding levels. Facial gestures made by the user must be well defined otherwise it will count as a mistake.

The first game is like a virtual mirror, they can see how the avatar imitates the user’s gestures automatically, while a picture and the message of the gesture to be made appear on the right side of the screen. The images sequence of the operation of the first virtual game is shown in Fig. 7.

The second virtual game consists of driving a bicycle. The virtual environment represents a path through a forest; movement of the bicycle occurs when the user makes the predetermined facial gestures that will control the bicycle, as shown in Fig. 8. The application identifies and records the time spent and the number of prizes collected during the trajectory.

Fig. 7.
figure 7

Images sequence of the game 1 functioning.

Full size image
Fig. 8.
figure 8

Game 2 image sequence working.

Full size image

The third virtual game is based on a hot-air balloon, whose movements will be controlled by the gestures set in the menu. The virtual environment represents a city and you can feel the sensation of the hot-air balloon rising. When they perform the facial gestures correctly, the animation is produced that allows you to advance the balloon, as shown in the image sequences in the Fig. 9. Application identifies and records the time elapsed, the number of awards achieved and the crashes produced during the journey.

Fig. 9.
figure 9

Image sequence of game 3 working.

Full size image

3.2 Usability of the Virtual Rehabilitation System

The system was initially validated by a speech therapist. The traditional therapy specialist uses graphic cards to mimic gestures to strengthen the facial muscles of his patients. The use of the system by the specialist is acceptable. Interaction, environments, animations, 3D graphics, sounds, menus and feedback through indicator messages are the factors that the specialist highlights as interesting in the system, because they achieve the stimulation of children. The specialist’s experience concludes that the use of technology in traditional therapies provokes more interest in children and therefore better benefits for their recovery.

The System Usability Scale (SUS) was applied to six children between the ages of eight and ten to obtain the weighting of the system’s usability. The questions asked focused on aspects such as animation, interaction and functionality of the games (Table 1).

Table 1. Evaluation of the usability of the system.
Full size table

Ten questions were asked, where each question can be scored from 1 to 5, where 1 indicates total disagreement and 5 total agreements. To obtain the results of the six surveys carried out, the average results obtained were added together and then the SUS scale was applied to obtain the final number 38.67, which multiplied by 2.5 gives a SUS score value of 96.7. Given that the theoretical maximum is 100 points, this result indicates that the proposed system reflects a high acceptance by the users who participated in the experiment. In addition, the survey showed that the participants are motivated to play; the environments, messages, sound and the challenge proposed in each game are interesting for each of them.

Both the specialist and the participants consider that the proposed system can stimulate to interactively perform a traditional speech therapy as well as a correct performance in rehabilitation activities.

From the observations made by the specialist, it is necessary to consider the duration of each game, in order to match the rehabilitation time of a traditional therapy.

Finally, it can be stated that the objective of the present work has been achieved, as the user can perform rehabilitation therapies in an entertaining and enriching way and not in a boring and monotonous way as traditional therapies usually are.

3.3 Discussion of the Rehabilitation Benefits Achieved

The positive feedback generated by the proposed system from users is one of the most important aspects to highlight. The use of virtual reality as part of traditional rehabilitation therapy especially for children, it causes interest in the use of these systems and that the movements and activities planned for therapy are carried out in an integrated manner.

From the results obtained, it can be determined that the proposed games activate the concentration of the children and motivate them to carry out a series of orofacial exercises based on the fulfilment of challenges. These activities that in traditional therapy were monotonous and boring become entertaining.

In the experiment carried out it could be observed that children feel pressure in the area of the face but in spite of this they continue to carry out the activity with the aim of achieving the desired score. Furthermore, this indicates that the muscle tone of the face is being strengthened.

4 Conclusions

This paper presents the implementation of a non-invasive system based on a virtual application, which is programmed to assist and stimulate the patient in orofacial physiotherapy. In the testing it has been possible to promote the performance of exercises such as swell cheeks, blowing, kissing, smiling, etc. They favor the good functioning of the orofacial area. Virtual environments developed in Unity software contain high-quality animated graphics and 3D environments. The games created present challenges that cause the user to feel motivated and interested in performing rehabilitation activities. The SUS Usability test yielded a weighting of 96.7, which reflects that the proposed system has a high acceptability by the users who participated in the experiment. The use of the Microsoft Kinect 2.0 device allows the tracking of the movement of the user’s face while allowing the same movement to be associated with a 3D facial model, and makes it possible to implement algorithms based on readings of confidence values of movements of a set of specific points of the face positioned over the eyes, mouth and cheeks. These characteristics of the device can be used in future work, for the construction of other facial gestures to be recognized according to the requirement and indications of the physiotherapist, enhancing the effectiveness of recognition with machine learning algorithms.