Keywords

1 Introduction

The Social Assistive Robot (SAR) MARIA T21 (shown in Fig. 1) developed at UFES/Brasil brings together embedded technologies with the aim of being an aid tool in therapies with children with Autistic Spectrum Disorder (ASD) and Down Syndrome (DS), mainly applying the concept of Serious Games (SG) [1, 2]. Some traditional therapies for these children may require many hours of dedication per week, in addition to great commitment from professionals, reducing the number of children served. For this reason, many of these therapies have a high cost. In this way, the incorporation of robots in therapies aims to create an alternative of more playful and efficient interventions.

Fig. 1.
figure 1

Robot MARIA T21

Full size image

The robot MARIA T21 robot integrates a set of systems capable of interacting with the child. All these processes are managed by an opensource framework called Robot Operating System (ROS). From there, the entire network communication among sensors, actuators, and software are established, as well as the recording of data obtained. Figure 2 presents an overview of all the systems incorporated in the robot MARIA T21, as well as their approximate positioning on the robot’s body.

Fig. 2.
figure 2

MARIA T21’s systems

Full size image

The proposal of this work is to present some of the main sensors embedded in the robot MARIA T21. We describe the process that makes them functional tools for application in therapies for children with ASD and DS, and how the incorporation of sensors allows the addition of observational information, such as the degree of social interaction, repetitive and stereotyped movements, and psychomotricity, in an automatic way.

This work is organized as follows. In Sect. 1, an overview of the guiding themes of the research is given. In Sect. 2, Computer Vision and Tactile sensor systems will be described in detail and correlated with their respective therapeutic evaluation parameters and the SG that makes use of it. In Sect. 3, possibilities and limitations are discussed, as well as difficulties encountered in implementing the sensors. Finally, in Sect. 4 the conclusions are presented.

2 Sensors and Evaluative Parameters

2.1 Tracking System—LIDAR

The LIDAR (Light Detection And Ranging) sensor used in the robot is capable of estimating the distance of obstacles around it in two dimensions, using a rotating perpendicular scan of a laser light beam that reaches the obstacle and returns to the sensor. In this way, the estimated distance between the sensor and the obstacle is proportional to the time between the emission and reception of the light beam. The emitted laser pulses have a wavelength that varies between 0.7 and 1000 \(\upmu \)m, being contained in the infrared region of the electromagnetic spectrum. The LIDAR sensor used here is the RP LIDAR A1, developed by SLAMTEC, which was chosen because of its low cost when compared to other models available.

The 2D data obtained by this sensor can be used for mapping, localization and modeling of the environment around it. However, for the applications proposed by this work, a good part of these data can be discarded, as they refer to regions outside the zone of interest for the analysis of proxemics (explained in next section), where the serious games is beeing played by the child. Thus, data’s pre-processing is necessary, in order to remove unwanted data, making the child’s detection easier. Once the filtering process is carried out, the x and y position of the child in the zone of interest is calculated through the average of the remaining points. This measurement considers that there is no object or person in this zone besides the child. A pseudo-algorithm 1 describes this process in a few words.

Algorithm 1
figure a

Preprocessing and classification of LIDAR points cloud for child position identification

Full size image

Tracking Aplication—Proxemic Proxemics is a fundamental principle of interaction that represents the distance between two agents during an interaction applied for both human agents and human-robot interaction [3]. Initially defined by Edward Hall [4], it seeks to describe the spatial behavior of individuals, which can be seen as a form of non-verbal and implicit communication, that is, how people use the space around them, and how this influences the interaction and communication with other people in the nearby [5] space. Proxemics theory has a social and human-centered focus, seeking to define the space around a person as different zones with different radii, namely: intimate, personal, social and public [6]. These areas of proxemics theory do not have a fixed radius, as they can change according to factors such as the age and culture of the subjects.

Research in human-robot interaction is often inspired by Hall’s theory with the purpose of understanding and improving how robots and humans approach each other and find the right distance [5,6,7].

From the data obtained from the LIDAR, it is possible to build the graph of Fig. 3, which refers to an excerpt of a child’s interaction with the robot, with the position (0, 0) being the robot, and the blue dot the child. The pink dashed line shows the path taken by the child during the interaction. It is possible to observe that the child traveled through all the zones defined by the proxemics, arriving twice in the intimate zone, which demonstrates the child’s receptivity to the robot. This type of chart can be used by therapists to assess the long-term evolution of the child’s social interaction with the robot.

Fig. 3.
figure 3

Proxemic’s graphic

Full size image

Tracking Aplication—Sound Repeat A SG named “Sound Repeat” was designed to be use with the robot MARIA T21 in front of the child, and has a simple interface containing only four squares of different colors, an indication of where to step, and a scoring system in the upper right corner in the form of stars. Once started, sounds are played with a different musical note selected randomly while the coresponding squares flashes. The child must then step on the blinking squares, repeating the sequence shown, Fig. 4. On the first level, only one item flashes; at level two, two items flash or one item flashes twice, and so the sequence continues to increase. In this way, each level is more complex than the previous one, and for each correct execution, the child earns a star. In case of child’s fail, the game returns to the initial level. This SG exercises the conceps of divided and shared attention, working memory and discrimination of audiovisual stimuli.

Fig. 4.
figure 4

Serious game—Sound Repeat

Full size image
Fig. 5.
figure 5

Child 2 path in SG Sound Repeat

Full size image

In furtherance of the child to interact with the game, the robot MARIA T21 uses the information of the child’s x and y position, obtained by processing the data from the LIDAR sensor. So that, to activate the desired color, the child must move to the top of the corresponding square. In such wise, the game processing is able to identify the square chosen by the child. The choice is confirmed by an increase in the square’s color brightness and by the emission of the corresponding sound. The graphics in Fig. 5a and Fig. 5b demonstrate the path covered by two children during the game’s application. In the figures it is possible to observe the shaping of a figure with 4 points, which refers precisely to the 4 squares of the game. Another point that can be highlighted is the high dispersion of the path taken by the child in Fig. 5b, when compared to the path taken by the child in Fig. 5a. Considering that they occured in two different days, is is possible to infer that this difference demonstrates the child’s lack of focus on performing the task.

2.2 Computer Vision

The computer vision system of the robot MARIA T21 is formed by a USB Logitech C920 PRO camera, which has a diagonal field of view (FOV) of 78\(^\circ \), and a maximum resolution of 1080p@30fps. The camera is attached to the upper part of the robot’s face to capture frontal images of the child’s movement during therapy. For the applications proposed in this work, the main function of the computer vision system is to identify the child’s body movement. For this, an open source library, called OpenPose, developed by Carnegie Mellon University (CMU), based on convolutional neural network and supervised learning was used [8]. It is able to capture information from images of human bodies and provide spatial information about joints.

Computer Vision Application—Center of Mass Postural balance involves the coordination of sensorimotor strategies including many physiological systems and cognitive processing to stabilize the body’s center of mass during self-initiated and externally triggered disturbances in postural stability [9]. In individuals wich ASD and with DS, there is evidence of a diversity of motor difficulties and atypical sensory profiles, which individually or in combination can result in postural stability challenges [10, 11].

Travers et al. [10] examined the postural balance of sixteen young people with ASD, and twenty one with typical development. It was found that the group with ASD showed greater postural sway, and also that the sway was related to the Intelligence Quocient (IQ) of the subjects. The results suggest that children with lower IQ suffer from postural stability, and lay the groundwork for investigation of the underlying mechanisms of poorer balance across the autistic spectrum [10]. It is worth mentioning that impairment in postural control can have a substantial impact on the development of motor and social skills in individuals with ASD [12].

The child’s center of mass was calculated considering the midpoint between keypoints 1 and 8, as shown in Fig. 6. It can be seen in yellow, the trajectory that this point performed from a stretch of the child’s execution of the SG.

Fig. 6.
figure 6

Child keypoints and center of mass trajectory

Full size image

From the graph of Fig. 7, it is possible to observe the center of mass horizontal variations with two movements of greater amplitude to the right in the intervals [0, 100] and [600, 700], In addition, a movement of lower amplitude to the left, in the interval [400, 500], and in the other frames, small oscillations in relation to the central axis can also be observed.

Fig. 7.
figure 7

Center of mass horizontal variations

Full size image

Computer Vision Application—Scarlet Parrot The SG Scarlet Parrot is projected onto the floor by the robot and has the mechanics of controlling a bird through the child’s torso and arm movements. For this, a natural environment full of mountains, waterfalls, green and red trees was designed on the ground. This environment consists of the home of the game’s characters: a Scarlet macaw, and her chicks. The child’s task is to control the bird in search of food for the chicks. For this, the child must place himself/herself in front of the robot, at the edge of the projection, and raise the arms so that the macaw flies, as shown in Fig. 8a.

Fig. 8.
figure 8

SG Scarlet Parrot

Full size image

If the child wants the animal to go up or down, he/she must raise or lower the arms; to go to the left or right, the child’s torso must be tilted to the side that he/she wants to direct the animal’s flight; to maintain the course, it is enough for the child to remain with the trunk vertically with open arms. A total of 4 fruits must be collected from the red trees, and only then return to the nest and deliver fruits to the chicks. This SG aims to exercise divided and shared attention, proprioception, motor coordination, and postural balance, which are important aspects for children with ASD and DS. From the movements required by the SG and through the keypoints processed by OpenPose, it is possible to evaluate aspects of psychomotricity such as balance, as shown in Fig. 7. It is also possible to evaluate precision and amplitude of movements, shown in Fig. 8b, where the keypoints of the hands trajectory are shown by the yellow and green lines.

2.3 Tactile System

The robot MARIA T21 has a tactile system built from a copper film independently applied in the robot’s arms, belly and head. Copper acts as the electrode of a capacitive circuit connected to an analog port of an ESP32 microcontroller. Above the copper foil, the robot still has a white plastic adhesive film, covering the entire robot, which has aesthetic purpose. Such plastic coating does not significantly interfere with the capacitive circuit, since the touch of human skin on the robot’s surface changes the circuit’s capacitance, generating a variation in the signal received by the microcontroller.

Copper coating was performed in order to separate the robot into 3 tactile zones: Arms, Belly and Head, as shown in Fig. 9. The system is only capable of differentiating the touch between zones.

Fig. 9.
figure 9

Touch zones

Full size image
Fig. 10.
figure 10

Touch zones’ history

Full size image

Haptic Communication Similar to the concept of proxemics, interpersonal haptic communication refers to any communication system that supports the mediation of touch between two or more people, in order to assess the degree of social interaction [13, 14]. This type of communication is crucial for interpersonal development as a manner of expressing affection, intention or emotion, and its development is even more necessary given that many of ASD children have an aversion to touch [15].

Therefore, therapies that encourage physical contact have a very important function. Despite this, the use of the concept of haptic communication is little explored in therapies that use robots [16, 17]. However, from the capacitive sensors described above, the robot MARIA T21 is able to recognize touches and stimulate the children through speech or changes in the face. For instance, this is used in situations where the child is outside the social zone (according to concepts of proxemic), where the robot tilts its head down, changes its expression to sad, and emits acoustic phrases expressing sadness. At that moment, the child is usually encouraged to go towards the robot and hug it, with a positive change in the feeling simulated by the robot as a reward. The graph in Fig. 10 shows the reading of these sensors in an interaction of this type.

3 Discussion

The results of this research demonstrate that the use of sensors in the socially-assisted robot MARIA T21 can offer a way for health professionals and family members to quantitatively complement the results of therapies in children with ASD and DS with greater precision, identifying more precisely social communication behaviors in addition to psychomotricity factors. In addition, monitoring and stimulating elements such as proxemics and haptic communication has the potential to develop fundamental skills for children with ASD. On the other hand, the quantification of psychomotricity factors and stereotypies can help therapists in directing interventions as well as supervising the evolution of children submmited to therapy.

However, it is worth highlighting some specific characteristics of the sensors. For instance, the camera onboard the robot needs a good lighting contrast with the low lighting requirement for the game’s projection system. This system, in particular, is a computational factor for identifying the two factors that impair the vision of both quality points. Regarding the LIDAR sensor, the overlapping of objects or children in the interest zone can generate blind regions, which are zones where the sensor is not able to correctly identify the child’s position. To work around this issue, the child must stay alone in this zone. In the touch detection system, it would be essential to identify the touch intensity for a better analysis of haptic communication. However the type of sensor used in the robot is not capable of recognizing such a parameter.

4 Conclusion

The work demonstrated the sensor’s potential embedded in the robot MARIA T21 to act as a complementary tool in the quantification of data commonly obtained only through visual observation by therapists. For future work, new systems are under development, such as a facial expression recognition system, as well as an attention monitoring system. In this way, it is expected the incorporation of this new data as input parameters for the SG, better adapting to the child’s needs.