Keywords

1 Introduction

Mental healthcare with the aid of robots is fast becoming an exciting niche in human-robot interaction (HRI) studies. Recent studies have covered the role of robots in clinical settings and care facilities for people with dementia, Alzheimer and also children with special needs. Robot-based solutions are desirable as they are specific, repetitive, motivating and can be made to fit current therapy models of the target population.

1.1 Post-stroke Depression

Depression is present in 25–30% among stroke patients [1]. Stroke survivors may experience stress, worry, sadness and hopelessness in different degrees. They may also experience low mood, feelings of hopelessness, withdrawal from daily social activities and even suicide. Several factors that could accelerate symptoms of depression in a stroke patient are lack of psychosocial support, inadequate of nutrients, intrapersonal coping skills and society integration. The dearth of goal-oriented behaviour that guides the patient towards community integration may perpetuates it further.

Post-stroke depression has negative effects on functional recovery [2]. Thus it needs to be treated effectively. Existing treatments include anti-depressant nortriptyline (a type of drug), social care through peer support group and animal-assisted therapy. Anti-depressant helps to improve mood. Social participation and integration reduces emotional distress and improves functional independence. However, community integration may only start after inpatient rehabilitation process took place. Animal-assisted therapy (AAT) is a helpful addition to a rehabilitation regimen. AAT helps motivate and make therapy more enjoyable and less stressful for stroke patients [3]. Nevertheless, the use of animals for therapy is unsuitable for patients whom are afraid of animals or have allergies. Also, the maintenance care will be costly plus the risk of zoonotic infection.

1.2 Animal Robots as Therapy Medium

The creation of robots which is capable to interact with human being and can show facial expression has been popular in the history of robotic [4]. Animal robots is suggested to replace real animals in AAT programs because it can prevent the case of scratches, bite, and allergy but still remain effective and give positive impact. Most robotic animals are modelled after common animals such as dogs and cats. Yet, their behaviour can be unsuccessful in meeting human’s anticipations. PARO the seal robot (Fig. 1) has recorded much success to increase mood, make people happy and encourages human-to-human interaction [5].

Fig. 1.
figure 1

PARO the baby seal robot

Full size image

1.3 Multi-disciplinary Rehabilitation Centre

When a stroke, brain injury or other traumatic event requires that patients spend time in rehabilitation, a working multidisciplinary team is fundamental in delivering effective care across the journey to recovery [6]. Instead of a hospital gown, patients wear their own clothes, and incorporate taking care of themselves as a part of the rehab experience. The amount and types of therapy are prescribed by experts in the rehabilitation unit to help patients set goals and regain the same level of function before their injury.

SOCSO Tun Razak Rehabilitation Center (TRRC) in Melaka is a rehabilitation complex that combines medical and vocational rehabilitation with an allied health institute. Under the SOCSO’s ‘Return to Work’ program, disabled patients undergo physical and vocational rehabilitation in order to rejoin the workforce. Patients in rehabilitation centres need constant mind conditioning and motivation. However, psychologists can only spend a limited time with the patients in a week. Thus, the aim of this pilot study is to propose a robotic animal as an adjunct tool to help provide mental support to the patients and increases mood during the in-patient rehabilitation period.

2 PARO the Baby Seal Robot

2.1 PARO Technical Specifications and Behaviour

PARO is a seal robot designed and developed by Professor Takanori Shibata in 1993. It weighs around 2.8 kg and equipped with several sensors and actuators. PARO features a soft furry coat with built-in intelligence providing psychological, physiological and social effects through physical interaction with humans. It reacts to petting and stroking (Fig. 2) by blinking its eyes and moving its flipper. It cries out when handled roughly, but it reacts positively to soft petting. It acts as a surrogate for a real pet to help reduce stress levels and improve social skills of patients [7]. PARO is very cute and can give facial expressions through the movement of its eyelids [8]. Its battery lasts for one hour and can be charged with a charger resembling a pacifier.

Fig. 2.
figure 2

PARO being stroked and hugged by a patient at the psychosocial department of TRRC

Full size image

PARO’s model is inspired after a baby harp seal to increase interest among users. Unfamiliar animals are hypothesized to have minimum expectations in the human agent during interaction [4]. Non-technical people (i.e. psychologists, therapists) are receptive to robots as assistive technology; however they require systems that require low technical skills (like programming). PARO is a robot readily equipped with processors and tactile sensors making it to be directly integrated into current therapy regime.

2.2 PARO’s Role to Help Manage Depression

Antidepressants are effective psychological interventions and are easily available in Malaysia [9]. Nevertheless, drugs only act as mood stabilizer and do not cure the illness. These drugs sometimes cause insomnia, fatigue and blurred vision. Patients with depression at centres like TRRC undergo rehabilitation treatment before they are declared stable. Then they can return to their family. In this study, PARO is proposed as an augmentative strategy to elevate mood for a patient with depression during the early stages of the rehabilitation journey after stroke occurred. Early treatment is vital as it helps to decrease the morbidity and mortality associated with depression [10].

3 Robot as Therapy Medium for Post-stroke Depression

3.1 Experimental Protocol

The experimental procedure serves as platform to investigate the interaction between the patient and PARO and to compare the changes before and after robotic exposure. The overall experimental flow is shown in Fig. 3. First, approval was obtained from the UTeM Research Ethics Committee to safeguard the wellbeing of the patients and researchers involved and to ensure that their rights are always protected. Then, suitable patients at TRRC were selected. The inclusion criteria are: diagnosed with depression after stroke, currently an in-patient at the centre, able to converse in Malay language or English, able to stroke an animal and can hear clearly. At the time of study, only one patient fulfils the said criteria at TRRC. Then, a short session with psychologist was conducted to explain about the experiment flow. Another aspect that we have taken into consideration is the “Conditions to Abort Procedures” which include conditions where:

Fig. 3.
figure 3

Overall flow of methodology for the study

Full size image
  1. a.

    The patient becomes restless/uncooperative/show abnormal behaviour.

  2. b.

    The patient is scared of PARO.

  3. c.

    The patient requests to abort the interaction.

The next step is the pre-experimental evaluation by a rehabilitation psychologist at SRC using the Hamilton Depression Rating Scale (HAM-D), Hamilton Anxiety Rating Scale (HAM-A), Columbia-suicide Severity Rating Scale (C-SSRS) and Pittsburgh Sleep Quality Assessment (PSQI). HAM-D is a validated screening instrument for post-stroke depression [11] to provide indication of depression, guilt, suicide, insomnia (sleep problem) and as guide to evaluate recovery. HAM-A measures the severity of anxiety symptoms and is widely used in both clinical and research settings. C-SSRS rates suicidal assessment on a scale. It is a semi constructed, rater-based interview to prospectively asses the severity and frequency of suicidal ideation and behaviors [12]. PSQI measures the quality and patterns of sleep in adults.

Then, the first interaction with PARO took place. Each session lasted for 20 min every day for 3 consecutive days. During the interaction, the patient was given time to familiarize and was encouraged to interact with PARO under a psychologist’s supervision. In the post-experimental stage after the third session, the same four screening instruments: HAM-D, HAM-A, C-SSRS and PSQI were administered. Recorded videos were analysed using OpenCV software. With facial emotion detection, algorithms detect faces and sense micro expressions of smile. The number of smiles were recorded and stored as indication of positive emotions.

3.2 Subject

The single subject in this pilot study was a 43-year old female patient. Before admission as an in-patient at TRRC, she was diagnosed with hemorrhagic stroke in February 2016. When she was in TRRC, she displayed symptoms of depression in which she was referred to psychiatric department in the general hospital of Melaka. Psychiatric findings indicated that the patient has major depressive disorder (MDD). She was prescribed with anti-depressant to stabilize her depression symptoms; to help her sleep better. However, the anti-depressant does not cure depression. It enables patient to return back to previous functioning level. It also does not trigger patients to smile.

3.3 Method of Interaction

Detailed experimental steps using PARO in the therapy session were designed based on advice from the psychologist. The patient was currently undergoing a rehabilitation program that mainly includes physical therapy activities. Three days of therapy with PARO in a week shall complement and not affect the current program. First, the therapy room in TRRC were set-up with HD video camera, voice recorder and projector as shown in Fig. 4. Next, the patient was invited to enter the room. Before the first interaction session, the psychologist administered the four screening instruments. Then, she was given brief introduction about the experiment, her consent was taken and a short video of PARO was shown.

Fig. 4.
figure 4

Room set-up in the rehabilitation centre where the experiment was conducted (left) and layout of therapy room arrangement. Control area only applicable in the third session (right)

Full size image

Then only the 10-min session can take place to obtain recording of the session without PARO. During this period, the psychologist began to introduce general topics to talk with the patient such as her favourite food, her activities during weekends and her family members. Indirectly, the psychologist was using cognitive behaviour techniques and goal directed behaviour during the therapy. Then, interaction with PARO began. PARO was brought into the room by the experimenter and was put on the patient’s lap as she sat in her wheelchair. With PARO in the room, the consultation therapy continues with PARO as companion to the patient. The psychologist continued to have conversations with the patient. The 10-min therapy without PARO and 10-min with PARO cycle was repeated for all three sessions. At the end of each session, the patient was informed that PARO will be taken away. During the third session, before the therapy with PARO ended, the patient was left alone with PARO to observe her behaviour towards the animal robot without human intervention. Patient was observed by the psychologist and experimenter from the next room equipped with a two-way mirror (Fig. 4). Duration of was limited to 20 min because post-stroke patients cannot withstand long therapy sessions and can easily get tired.

3.4 Video Processing Using OpenCV

The sessions without PARO and with PARO were recorded for analysis purposes. To evaluate the patient’s positive emotion, specifically the number of smiles, recorded videos were processed using OpenCV software. OpenCV is an open source computer vision library for commercial and research use [13]. It is one of the most widely used libraries in image processing, it is simple to use and has extensive user network.

For smile detection algorithm, Adaptive Boosting (AdaBoost) was used as learning algorithm [14]. Using AdaBoost improves the accuracy of learning algorithm where the output of multiple “weak classifiers” is combined into a weighted sum that represents the final output of the boosted classifier. Smile detection was targeted at the mouth region (Fig. 5). In the smile detection mechanism, the videos were processed frame by frame and converted into grey scale. Smile detection determines ‘where’ in a face image the ‘smile’ is located and this is done by scanning the different face image scales and extracting the exact patterns to detect smile. Classifier uses a single feature to define images as smiles or non-smiles which are stored in cascade data.

Fig. 5.
figure 5

Mouth region for image processing (left) and snapshots during therapy sessions (right)

Full size image

4 Results and Discussion

The first session of the patient-robot interaction was focused on getting the patient to familiarize herself with PARO. Snapshots from the recorded videos are shown in Fig. 5. Based on the video record of the first session, during the first 10 min, the patient responded to the psychologist about general topics and about her family. It was observed that the patient really missed her family. She looked sad and cried when talking about her family. Then, PARO showed up for the first time. The patient got curious about the robot. She stopped crying and the intensity of her previous sad emotions had gone down. When stroking PARO, she kept saying that PARO was cute. With the presence of PARO, it is observed that the patient’s mood had changed. PARO reacted to petting and stroking by the patient by blinking its eyes and moving its flipper. This, in turn, continuously prompted the patient to give respond to PARO.

The following day, the session continues with 10 min of session introduction without PARO and 10 min of robot interaction with PARO. The second session started with the discussion on general topics and then about the patient’s pet. The patient expressed that she missed her pets which were six cats at her home. She said PARO reminded her of them. She voluntarily talked to PARO by saying ‘I love you PARO’. And the end of the session she mentioned that she would like to see PARO again. The patient was observed to be more comfortable with PARO on the second day. On the third session, the topic introduced by the psychologist was about her weekend activities, visits from family members, her plans after the rehabilitation program is complete and about return-to-work. She mentioned that she felt calmer. She wanted to get better so she can take care of her sick father. She expressed that she missed PARO after the second session and that it would be good to have PARO to accompany her before sleep. It was also observed that the patient showed more verbal communications with PARO.

4.1 Results of Screening Evaluations

Results on evaluations using the HAM-D, HAM-A, C-SSRS and PSQI instruments are tabulated in Table 1. For HAM-D, the scoring instructions are 0–7 = Normal, 8–13 = Mild Depression, 14–18 = Moderate Depression, 19–22 = Severe Depression and ≥23 = Very Severe Depression. For HAM-A, a total score of <17 = Mild Severity, 18–24 = Mild to Moderate Severity and 25–30 = Moderate to Severe. For HAM-D, score of 13 in pre-experimental falls in the high end of mild depression category. The screening evaluation suggests mild insomnia (difficulty to sleep), mild depressed mood in terms of sadness, anxiety and pessimism about the future. For post-experimental, score reduced to 9 (lowest end of mild depression) with reduced anxiety, worrying and fear of future. In terms of depressed mood: tendency of crying had reduced too. Patient’s sleep had improved where difficulty of falling asleep had reduced in post experiment. Patient reported that she fell asleep in less than 30 min. For HAM-A, the scores were 17 (pre) and 6 (post). Both fall under the mild severity anxiety. This is consistent with HAM-D (pre) results where patient reported moderate to severe anxious mood, tension, fear and concentration difficulty. In post-experiment HAM-D reported mild anxious mood, tension and mild depressed mood. Overall behaviour observation using HAM-A showed that in the pre-experimental, patient reported restlessness and agitation but in the post-experimental, patient only exhibited mild restlessness and agitation.

Table 1. Results from screening instruments
Full size table

The C-SSRS instrument was used to further support HAM-D in the suicide section. In the pre and post screening, C-SSRS results support the scores using HAM-D where the patient denied suicide. This reflects that she had positive insight to recover and become better. C-SSRS scores also support findings from the image processing tool that detects the frequency of her smile. This will be covered in the next section. PSQI instrument that measures the quality of sleep also reflected congruency with results in HAM-D where evaluation result improved from mild disturbance before the experiment to reduced disturbance after the experiment. The level of depression and anxiety had reduced. Hence, the quality of sleep had improved after interaction with PARO. Even though the duration of three days was short, positive changes were observed.

4.2 Results of Video Processing

Sample of output results from the OpenCV smile detection mechanism is shown in Fig. 6. The graph in Fig. 6 differentiates the frequency of smiles in the 10-min therapy duration without PARO followed by the last 10-min therapy with PARO. This duration was chosen as this was when the patient has familiarized herself and felt comfortable with PARO. The results show that there is an increase in number of smiles with and without PARO in the first, second and third session. Findings from the image processing tool is consistent with results from C-SSRS scores where the patient still relate to reality and has good judgment. Thus her smiles were natural and within normal limits. Patients with depression rarely smile. Thus, even though the total number of smiles were small in the pre and post-experiment, the finding is significant. The frequency of smile was the same in the first two sessions and increased in the third session suggesting that the patient felt calmer when interacting with PARO.

Fig. 6.
figure 6

Example of OpenCV output (left) and bar graph comparing the number of smiles (right)

Full size image

4.3 Overall Observation

The screening results suggest that PARO appeared to help the patient to manage her psychological distress. In the pre and post period, there was no other psychological intervention administered to the patient besides PARO to minimize the co-founding factors. The interactions with PARO show that patient could express her emotion and also personal issue easier during therapy with PARO. This might explain why the symptoms of anxiety and depression reduced. PARO acted as a platform to enable the patient to express herself. Though the number of smiles detected was few (maximum of 4 in the final session), it is considered as positive outcome for a patient with depression. The patient had enjoyed the companionship of PARO robot over the three days of exposure.

5 Conclusion

This study suggests that PARO effectively uplifts mood and helps the patient to be calm. PARO reacted to stroking by the patient by continuously blinking its eyes and moving its flipper. This continuous trigger prompted the patient to give respond to PARO throughout the session. Even though the patient was prescribed with anti-depressant, it was only meant to improve her depression symptoms and will not trigger her to smile. So, the increase in the number of smiles is a natural respond and within the normal limits of a typical person. For future works, studies with a larger sample of patients are needed as the effect of repetitive interaction with PARO needs to be investigated. Depression is a serious mental illness but often under-recognized [9]. Embedding state-of-the-art robotics technology into depression therapy shall give new hope to people affected with depression all over the world.