Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Life expectancy has strikingly increased and produced the need of offering new technological solutions that allow people to live a better old age. Ageing reduces the perception capabilities and also entails some cognitive and physical problems. Nevertheless, the lost or impairment of some sense, such as the sense of sight, is not only due to ageing. There are other reasons that provoke this impairment. Moreover, as noticed in a recent paper [1], physical activity, cognitive training and social engagement improve the quality of life and reduce cognitive decline in older adults. Thus, in order to provide solutions that improve the quality of life of these impaired people, assistive technologies bring technological proposals that allow them to improve or reduce the effect of their disabilities. These technologies offer new ways to rehabilitate their lost capabilities or to carry out some physical or cognitive exercises in a funny and engaging way, just as when playing video games.

In general, the rehabilitation process could involve the movement of bones and joints (physical) or brain processing (cognitive). Nevertheless, despite the fact that cognitive rehabilitation includes some movements necessary to carry out the task, these movements are not meant to improve other physical disabilities. Thus, merging these two types of rehabilitation in one tool is a promising field. In any case, although these technological proposals show high potential in the rehabilitation process [13], they must take into account other troubles, such as the loss of abilities in the perception of external stimuli.

This paper presents an improvement of a previous system [9] designed to aid in the cognitive and physical rehabilitation of elderly people, combining in one therapy the two disabilities. The main advantage of the new proposal is the incorporation of multisensory stimulation. In this case, the inclusion, not only visual information but also aural and haptic, enhances the perception of the object included in a specific therapy. With this new information, the therapist can adapt the sensory cues that the user could use to successfully complete the therapy. Thus, after reviewing the assistive technologies used in cognitive and physical rehabilitation, as well as some others that use vibrotactile information to guide the user, we present this new proposal. First, the supported cognitive therapies are described. After that, we describe our multisensory proposal explaining how the specialist configures the therapy, including several senses cues, and how the user carries out the therapy by using these new stimuli. Finally, some conclusion and future works are presented.

2 Use of New Technologies in Rehabilitation

The use of technology applied to human activities is increasingly frequent. It offers support and facilitates the completion of daily tasks. Rehabilitation is not an exception. In fact, the use of technology in rehabilitation offers not only the possibility to control its execution, but it also allows carrying out these activities without the physical presence of the therapist. Moreover, the patients can accomplish their rehabilitation activities at home. In the context of cognitive impairment, there are already some systems that help patients in their rehabilitation process. The cognitive impairments faced may be due to the degradation resulting from age, or from an injury or stroke that affects cognitive abilities. It is impossible to define a unique way of dealing with these problems. Indeed, each type of brain disorder requires different methods and specialists for its treatment [8].

Although the use of Kinect to control the patients’ movements in a cognitive rehabilitation therapy is not so frequent, some examples are available. The Tango:H system [4] focuses on hospitalised children with disabilities. It offers some funny games in which children must use some body parts to select objects that allow them to achieve the correct answer. In the older people rehabilitation domain, a system uses Kinect to improve memory by performing mental activities and physical exercises at the same time [?]. The authors design several exercises of different levels of complexity that are adapted depending of the mental state of the patient. Moreover, the effectiveness of this rehabilitation therapy has been analysed [14]. It is concluded that, although this evaluation does not show an improvement of the cognitive impairment measured by means of the Mini–Mental State Examination, cognitive performance does not decline and the impairment progression is slow. A very recent work [3] presents a system for the rehabilitation of stroke-affected patients, which includes four immersive games. One of them is a quiz activity, which not only makes the person perform the physical exercise, but also focuses on cognitive aspects to answer a basic set of general knowledge questions. Finally, the base of the work introduced here presents a tool for designing cognitive therapies that allows the therapists to design from scratch their own cognitive therapies associated with pair and multiple association or categorisation exercises [9]. The capability to design their own exercises allows the therapists to adapt the therapies to each specific patient. In addition, these cognitive exercises are complemented with physical movements of the patient, this way combining cognitive and physical rehabilitation in one single therapy.

3 Haptic Guidance for Visually Impaired People

As noticed recently [10], there are only a few proposals in the rehabilitation domain that include haptic cues for improving the task. As an example, a novel rehabilitation system is presented [12]. It includes a glove that has several vibration motors to increase the feeling of immersion. The proposal uses a Kinect sensor to track the movement of the hand and several bend sensors to control the specific movement of the user’s fingers. In this approach, a glove is used to place the vibration motors on the hand. The system locates the different actuators by creating a grid that is used to transmit some movement feeling. Although this proposal locates several vibration motors on the user’s hand, the actuators are used to improve the user’s perception when interacting with virtual objects, but for guiding the interaction they only use the visual cue. A rehabilitation system that uses a vibrotactile armband communicating with Kinect wireless has been created [10]. The vibrotactile actuators are only used to instantly notify when the selected body part is raised to the required angle of elevation during a physiotherapy session.

Moreover, vibrotactile actuators are also used to guide user movements. Thus, a paper [2] presents a vibrotactile system that measures arm motions using a Kinect sensor and provides vibration feedback when the user deviates from a desired trajectory, indicating the movement. The vibration algorithm is designed so that the user moves toward the stimulus or away from the stimulus. Each band consists of four actuators. The system activates them to indicate that the arm has to be relocated in a correct place. In another work [6], a system that uses accelerometers and gyroscopes to control the patient’s movements is introduced. In addition, it includes two coin eccentric mass pager motors for transmitting the vibrotactile stimulus. The user is instructed to move in the direction of the vibrotactile sensation until the vibration is eliminated. Also, a wearable haptic system, in which the user wears a full body suit, densely embedded with vibrotactile actuators, has been presented [5]. The movements of the user are tracked through a motion capture system. When the user deviates more than a predefined distance from the target, actuators on the associated limb segment provide feedback. It helps the users to move that part of his/her body toward the correct configuration. In this case, the vibration feedback from each actuator is intended to be a repulsive cue to push the user away.

As we have noticed, although there are previous proposals that provide support to the user when he/she has to imitate a specific movement, these proposals are all dedicated to the domain of physical rehabilitation therapies. Moreover, in general, they include several actuators distributed in different body parts. Finally, the goal of these systems is only to guide the user in his/her movements, but in our system we need to include some vibrotactile stimuli that allow the user to identify the specific element reached with the movement of his/her hand. Thus, in the next section we present our proposal, highlighting its main contributions.

4 Vi-SMARt: Our Cognitive Rehabilitation System

The proposed system (Vi-SMARt) is basically aimed at cognitive rehabilitation of people. In addition, it combines cognitive rehabilitation with physical exercises. Thus, together with exercising the brain, the user also moves some parts of his/her body, as defined by the therapist, to improve his/her rehabilitation. To achieve this goal, an open rehabilitation system has been created, in which the therapist designs all the rehabilitation exercises from scratch. This allows creating exercises based on well-known therapies, albeit adapted to the specific impairments of each particular patient.

Each exercise incorporates a series of meaningful elements with specific locations in the interaction space. During the design of an exercise, the therapist locates in the interaction space as many elements as he/she wants and selects some features for each element, such as size, position, orientation and colour. Thus, the patients should recognise the meaning of all these elements and carry out a specific cognitive process to successfully complete the exercise. Currently, our system allows designing three types of exercises: Pair Matching, Multiple Matching and Categorisation. These exercises have been selected as they are widely applied in rehabilitation of many cognitive deficits of the front executive function of the brain [8]. In addition, Vi-SMARt offers a possibility of free exercise, in which the therapist can design other types of exercises by freely defining all the available options.

When a patient wants to carry out an exercise, he/she should be located in front of a Kinect sensor that captures his/her body joints. In the previous version of our system (see [9, 11]), an image of the user body and the elements defined by the therapist appear on the monitor. In addition, the instructions that guide the patient in the performance of the exercise are also displayed on the screen. The patient’s mission is to select several elements by taking into account a specific rule defined by the therapist. To select each element, the patient places on it a particular part of his/her body. The allowed parts of the body used to select an element are set by the therapist at the moment of creating the exercise. As it can be seen, the visual channel is very relevant in this previous version, so that visually impaired people have enormous problems to use it.

4.1 Cognitive Exercise Authoring

The Cognitive Exercise Authoring component allows the therapist to design all the steps of a specific task or exercise. When the specialist creates a new exercise, the four types of exercises supported by the system appear. After selecting the desired type, the exercise edition screen is displayed (see left of Fig. 1). The exercise composer interface is divided into five areas. At the top, the therapist introduces the name and the goal of the exercise. These labels should be descriptive enough so that the patient can understand what he/she should carry out to successfully solve the exercise. At the bottom, the elements that the therapist can use to design the exercises are shown (numbers from 0 to 9). In this case, due to the fact that the exercise represented in Fig. 1 is Pair Matching, when the user selects one number at the bottom of the screen, two elements with the same image appear in the working area. On the left side of the screen, the elements already located in the working area appear. The user can assign several specific properties by selecting one element of the list (see right part of Fig. 1). Finally, the right side of the screen shows the Steps that make up the exercise. An exercise may consist of one or more steps. At each step, the elements displayed are totally different from the other steps.

Fig. 1
figure 1

Exercise edition interfaces. Left Exercise composer. Right Object properties

Full size image

To select an element from the working area, the patient must locate a part of his/her body on it and maintain the position at least for 1 s. As we use Kinect for tracking the user body, the allowed joints are those controlled by this device. Thus, the therapist must define for each elements which joint should be monitored to activate its selection (see Fig. 1).

As this system is required to be used by visually impaired people, other stimuli different from the visual one are needed. In this case, we need to assign an additional meaning to each object, in order to provide an additional way to identify each object. As it can be seen in Fig. 1, our system allows the therapists to define two additional meanings to each object. They can add a specific sound and/or and specific haptic feedback to each object. Thus, each object could be represented by an image, a sound, a haptic feedback, or a combination of them. In addition, the therapist can define other options to establish the way the user must carry out the exercise.

In the case that the rehabilitation specialist determines that one object must have sound, he/she selects a representative sound for it. When the patient enters into this object, the sound assigned to this object is reproduced. In this way, the visually impaired patient is able to determine the object. For example, in the exercise illustrated in this paper (see Figs. 1 and 2) the sound assigned to the element is the number represented in the visual image.

Fig. 2
figure 2

Patient performing an exercise

Full size image

4.2 Therapy Execution Environment

The Vitaki system [7] is used for the sake of designing and providing haptic sensations to patients. This system, which is composed of a software and a hardware component that are described below, has been created in our lab.

4.2.1 VITAKI Hardware

The VITAKI hardware is used to bring haptic sensations to the user. VITAKI is connected to a computer via Bluetooth protocol, and the computer sends the haptic stimulation that the actuator must reproduce. In this way, it has the lowest possible latency, since it only has to adapt the signal to stimulate the haptic actuators. This device has 9 outputs to which a haptic actuator can be connected. For our rehabilitation system, we use 8 outputs of the device, 4 for each haptic glove. These actuators are placed on the patient’s hands, using haptic gloves. In addition, the device uses a battery in order to get a minimal intrusion when a user is performing an exercise. In this way, the user should not worry about any wire connection between PC and VITAKI, so he/she performs the exercise totally free.

4.2.2 VITAKI Software

The VITAKI software is used to design the haptic feedback, so it can be used by the therapist to design a specific stimulus for each object. With this application, we can design the haptic stimulus that the user will receive when entering into an object. We can create complex stimuli by designing their waveforms. An example of this program is shown in Fig. 3. The system allows us to perform any desired waveform for stimulation. The therapist may create complex waveforms from the simple waveforms that are in the system or use previously defined ones. In addition, he/she can also create waveforms libraries for later use.

Fig. 3
figure 3

VITAKI software

Full size image

Although this allows us to provide haptic feedback to the user with greater accuracy, the therapist may wish to provide simple haptic signals to the patient. For this, we can use the rehabilitation system itself, instead of the Vitaki Software. In this way, the therapist will only have to define the duty cycle (period and pulse duration) to create the haptic stimulus instead of designing the waveform. Using this method, we can create distinctive stimulus for each object in the scenario in a simple way.

The patient’s part is simpler than the therapist’s one. At the beginning of the exercise, the user is offered the name and the objective set by the therapist at the top of the screen (see Fig. 2). Moreover, this information is read out by the system, allowing the visually impaired people to understand what they have to do during that exercise. This explanation must be sufficiently clear so that the user does not need any additional information for carrying out the full exercise. In the central part of the screen, the first Step created by the therapist appears, as well the figure of the patient in the background, are displayed. This image is captured by a Kinect sensor placed in front of him/her. On the right side of the interface, the number of hits, cognitive and physical failures, the duration of the exercise, and the patient’s physiological data (e.g., heart rate or concentration) are also highlighted.

In order to sense the haptic stimuli, the user has to place the Vitaki device on his/her chest or waist, and a glove with haptic actuators on each hand. When the exercise starts, the patient receives a haptic stimulus at each of his/her hands, indicating the location of the nearest haptic elements of the working area.

On the gloves (see Fig. 4) 4 vibrotactile actuators are placed. They are located on the thumb, on the index finger, on the pinky and on the palm of the hand. This actuators’ situation allows the user to draw a cross, pointing to the four possible movements of the user’s hand to find an element in the working area (see Fig. 4). The vibration of these actuators indicate in which direction the patient should move his/her hand (i.e., the vibration of these actuators on the patient’s right hand indicates that he/she has to move the hand to the left, up, to the right or down, respectively). The vibration of the actuator becomes lower when the patient’s hand comes close to the object. This helps visually impaired people to find objects in the interaction area.

Fig. 4
figure 4

Haptic glove

Full size image

Secondly, the patient receives a specific pulse when he/she enters inside each object. The pulse is created by using the VITAKI tool discussed in the previous section. The haptic pulse is provided at the same time in the 4 actuators of the hand, and the intensity of this pulse is defined by the size of the object in the virtual world.

The relation between the ‘haptic object’ and the stimulus is not necessarily representative, because there is no direct relation between object meaning and haptic sensation in most cases. However, there are some exceptions to this fact. This is the case, for example, in Categorisation exercises, where the user has to sort the objects. In this type of exercises, the number of pulses can be used to identify the number of the object. In this way, the patient receives a pulse for the first ‘haptic object’, two pulses for the second one, and so on. In the exercise used as an example in this paper (see Figs. 1 and 2), the haptic pulse assigned to each object can be a single pulse for objects with number 1, and three pulses for objects with number 3, and the pulse being more intense in large objects, and less intense in small objects. In this way, haptic meaning is assigned to objects in the virtual world, which enables patients with visual impairment to distinguish one object from another.

5 Conclusions

In this article, we have presented a multimodal cognitive rehabilitation system for visually impaired patients. After introducing some current computer-aided rehabilitation systems and their target patients, the rehabilitation system implemented in our research team has been described. Due the wide range of cognitive injuries that may occur, it is impossible to design a system that covers all of them. Therefore, the developed system has been focused on a small group of exercises, namely Pair Matching, Multiple Matching and Categorisation.

After choosing the type of exercise, the therapist just needs to put some elements in the working area and specify the stimuli associated to each one. As described in the paper, besides the visual sense we can send information to the user by aural and haptic channels. In the first case, standard PC peripherals such as display and speakers are used. But, in the second one, we use the VITAKI system developed in our laboratory, which allows us to manage several vibrotactile actuators that transmit a previously designed signal. As for the patient’s view of the system, we have tried to minimise the cognitive load. In fact, the patient only needs to move his/her body to select some elements of the working area. In addition, to knowing where the elements of the working area are, we have design a glove composed by four actuators that guides the user in the exploration task. Finally, this glove allows the user to receive some information about the meaning of the each element.