Keywords

1 Introduction

Individuals across the globe are living longer. A consequence of this is that people develop complex comorbidities and social problems. Dementia is currently one of the major causes of disability and dependency amongst older adults and affects the ability to perform daily activities such as cooking, washing, and maintaining personal hygiene [12]. Technology is an important factor in supporting people with dementia to live independently in their homes. One response by carers is to provide reminders or cues [19] using various assistive technologies.

IntraVox [27] is a novel voice-based assistive technology, which has the potential to prompt older adults with dementia to change their actions. In collaboration with a city council, we conducted an initial field study of IntraVox in a home setting. The system is composed of a speaker and a Raspberry Pi 4B computer with various smart home sensors attached. Based on the sensor data collected and using a personalized human voice, IntraVox verbally sends prompts and reminders to end-users to conduct tasks according to their needs and aspirations (e.g., enhance continence). The system introduces a high level of personalization as the human voice can be that of e.g. a carer, a family member, or a friend. All prompts and reminders verbalized by IntraVox are tailored to the individual’s needs and capacity to understand the instructions (e.g., tailored keywords and sentence structure). Whilst popular voice assistants (e.g., Amazon Alexa, Google Home) can provide prompts and reminders using standard voices [34], the IntraVox system is unique in that it uses a personalized human voice for delivering prompts, which could have benefits of security and comfort [27].

The work was conducted during the COVID-19 pandemic, at which time many research studies had stagnated [4, 25]. Despite these challenges, we recruited one end-user, their family supporter, and three carers. We installed IntraVox in the end-user’s home for seven days and asked participants to interact with it in an uncontrolled way. We wished to address the following research questions: RQ1: Does IntraVox have the potential to prompt people with dementia to change their behaviour? and RQ2: Do the tailored prompts used by IntraVox have an emotional impact on people with dementia and their carers?

This paper is structured as follows. We start by providing an overview of assistive technologies, highlighting also the importance of conducting short field evaluation studies. We then present the study conducted to evaluate IntraVox. Finally, we discuss the contributions we bring to the Human-Computer Interaction (HCI) field.

2 Background

In this section, we present the current research regarding assistive technologies and we highlight the importance of conducting short field evaluation studies.

Voice Assistive Systems. Most voice assistive systems are composed of off-the-shelf devices such as Amazon Alexa and Google Home. These have been introduced in the home as they have the potential to support independent living [4, 11, 15, 22]. For example, Shalini et al. [29] developed a customized voice assistant system using the two devices for older adults to use in the home. The system was developed as a consumer interface for the end-users and their family members that can provide health information on-demand, based on spoken queries. Simpson et al. [30], on the other hand, used an open source framework to design a conversational agent and provide companionship to older adults. The device engages in a casual conversation with the user regarding a past memory in their life or by suggesting activities that might be of interest to them. Despite the benefits, most users face challenges in interacting with the assistants. Whilst most users start by interacting in a natural language, they are later forced to learn how to speak to the assistant in order to successfully enable and interact with it, i.e., limit themselves to specific keywords, remember the proper commands and verbalize them clearly, loudly and in a short amount of time in the case of Amazon Alexa and Google Home [10, 18].

Prompting and Reminding. Cognitive assistive technologies supporting adults with dementia can provide them with an increased level of independence by considering their specific needs, emotions, and preferences [22, 27]. For example, systems using sensors, smart bulbs, and pressure detectors can direct individuals to the bathroom at night [1], whereas other systems can guide individuals to prepare meals [5], wash hands and prepare a cup of tea [20]. König et al. [16] highlight that a one-size-fits-all style of prompting should be avoided as people with dementia have different emotional responses to prompts, thus the user’s background and “sense of self and identity” should be considered when designing assistive technologies. Furthermore, Boyd et al. [6] concluded that prompts for people with dementia that are delivered in a familiar and explicit language can increase the success of a tailored intervention.

Short Field Evaluations. Technology evaluation is essential in HCI. Despite being complex and time-consuming, it is important to conduct short field evaluations as laboratory-based ones could leave mistakes unrevealed [14, 17]. Rogers et al. [24] argue that laboratory studies are “poor at capturing the context of use” and highlight that in-situ evaluations can indicate how people interact with technology in their intended setting. Bacchetti et al. [2] highlight that studies of new technologies and ideas “often must start small (sometimes even with n of 1) because of cost and feasibility concerns.”. Similarly, Caine [9] notes that “small” sample size studies can bring important findings to the HCI community by informing future studies and revealing the most obvious usability problems.

3 Short Field Evaluation Study

Together with a city council team, we identified an opportunity for IntraVox to be piloted with one of the city council’s customers. The end-user was a woman with advanced dementia, who lived alone at home (referred to as ‘the mother’). A family supporter (her son) stayed with her during evenings and mornings and three carers (city council employees) attended daily to prepare meals (lunch and dinner) and provide medication.

The study had three phases: Phase(1) - a pre-study interview with the son to understand the requirements and how IntraVox could provide an intervention; Phase(2) – an uncontrolled field study of seven days when IntraVox was installed in the mother’s home together with additional questionnaires the son and the three carers were asked to complete, and Phase(3) – a post-study interview to explore the son’s views regarding the system and his views of the impact it had on his mother. Ethical approval was obtained from the University and the city council to install the system in the mother’s house. Informed consent was attained by the son and carers, whereas the son acted as a consultee for the involvement of his mother who lacked the capacity to consent [31].

3.1 Phase 1 – Pre-study Interview

The interview was composed of open-ended questions to understand the son’s experience with smart home devices and how IntraVox could support the mother (to answer RQ1). The son was also presented with videos depicting scenarios where IntraVox could support end-users with complex needs [27].

Findings. The son had previously installed technology in his mother’s house. This included smart cameras to monitor the carers [23], USB-controlled gas, smart switches, and an Amazon Alexa for controlling lighting. His mother was unable to interact with the devices as a result of her advanced dementia.

With the deterioration of the mother’s cognitive abilities, she was in need of more personal care. One aspect that also impacted the son was the mother urinating in the utility room rather than her bathroom. The son suggested that IntraVox could “remind her that the utility room is not the bathroom and that she should go upstairs to use the toilet.”. Regarding the voice, the son stated: “Her dementia is very advanced, but she is strong-willed, and she would listen to a familiar voice, especially a carer’s voice. She doesn’t always listen to me, but she follows the carer’s instructions really well. Accents that she recognises make her feel more comfortable.”.

3.2 Phase 2 - Short Field Study

In collaboration with the son and three city council officers (an occupational therapist (OT), a dementia carer who has previously supported the mother, and an IT representative familiar with the mother’s situation), we held a separate discussion to explore how we could install the system in the mother’s home in compliance with COVID-19 pandemic restrictions, and to agree on the intervention. It was agreed to use IntraVox to provide prompts to use the bathroom rather than the utility room for voiding. It was decided that IntraVox would be installed in the utility room. Whenever motion would be detected (i.e., the mother entered the room), a prompt would be played: “Mary, the toilet is upstairs.”. As advised by the son, the prompt would be repeated twice. As the mother cooperates when recognising a familiar accent, we used the OT’s voice for the prompt. IntraVox was deployed for seven days to collect sufficient qualitative and quantitative data [13].

Development. IntraVox was comprised of a Raspberry Pi 4B computer, embedded in a case, with speakers attached to play the prompt (Fig. 1 left). As per the city council officers’ recommendation, we used the Samsung SmartThings motion sensor [28] for detecting motion in the utility room. The city council officers suggested the use of a Message Queue Telemetry Transport (MQTT) protocol [21], a standard Internet of Things protocol designed for transferring messages. Python software was used to connect to the city council’s server and to subscribe to the topic of interest (motion in a room). Whenever a message was received (motion detected), the prompt played.

Fig. 1.
figure 1

IntraVox installed inside a cabinet (left) and the motion sensor (right).

Full size image

Procedure. The system was installed following COVID-19 safety measurements [25]. As per the son’s advice, the system was installed in a hidden cabinet to avoid it being disturbed by the mother, and the motion sensor was placed on top of a light switch (Fig. 1 right). Data was collected in the background to log the time motion was detected and a prompt was played.

The son and the three carers were asked to complete two questionnaires, every day. The first questionnaire had two open-ended questions and a 5-Likert-scale question to capture their views regarding IntraVox. The questionnaire also included an emotion wheel to capture their emotion and emotional distress when interacting with the system [8]. We used Baillie et al.’s emotion wheel [3] to capture their emotions when entering the room and the prompt was played. For each emotion wheel, participants were asked to provide a primary and secondary emotion, with each emotion being grouped into its corresponding quadrant [3]. The second questionnaire, containing an open-ended question and an emotion wheel, aimed to capture the mother’s reaction and emotions when hearing the prompt (to answer RQ2). Participants were asked to view and describe the mother’s interaction with the system. The scope was to make sure the audio messages were not causing any emotional distress, e.g. being perceived as auditory hallucinations or making the mother think someone is in the house.

Findings. Due to their busy schedules, the son and the three carers managed to complete the two questionnaires only once. Positive emotions were provided when filling out the emotion wheels, with the majority being in the Very Passive – Very Positive quadrant (Table 1). This indicates that participants had a pleasant interaction with the system.

Inconsistencies were, however, noticed in the feedback. Despite indicating positive feelings when entering the utility room and hearing the prompt, participants provided mixed feedback on the 5-Likert-scale question if the prompt was annoying, i.e., the son disagreed, Carer 1 neither agreed nor disagreed, whilst Carer 2 and 3 agreed.

Table 1. Participants’ emotions when hearing the prompt played by IntraVox.
Full size table

Positive emotions were described by the son and carers with regards to the mother’s interaction with the system, with the majority being in the Very Passive – Very Positive quadrant (Table 2). She had a positive reaction to the system and no signs of distress when hearing the prompt. This contradicts previous findings indicating that IntraVox might only be suitable for people with mild and moderate dementia as the familiar voice might cause confusion [27].

Table 2. The mother’s emotions when hearing the prompt played by IntraVox.
Full size table

3.3 Phase 3 – Post-study Interview

The second interview with the son was composed of open-ended questions and aimed to understand his views regarding IntraVox and whether it was beneficial to the mother (to answer RQ1).

Findings. IntraVox received positive feedback from the son who stated: “The idea is massive. The study was 100% successful as we did not have any incident in the utility room since IntraVox was installed.”. Confirming previous findings [27], the son felt that the human voice contributed to a sense of security and comfort, and helped his mother adhere to a routine. The son also believed that IntraVox can improve the quality of life of both of them, stating that: “It can definitely improve her quality of life as sometimes she feels embarrassed of her actions. It gives me more time to spend with my mother, rather than cleaning around the house. The relief is amazing.”. When asked about the mother’s interaction with the system, the son declared that: “She would open the door and stop when she would hear the voice. She acknowledged that somebody was talking, she listened and paid attention to the second prompt. It didn’t scare her, she didn’t seem upset. She didn’t seem surprised either.”.

The son highlighted that the familiar voice “was annoying to me and the other carers who were in and out for 4 - 5 times.”. The son believed that IntraVox might be “better for people living alone”, or, as suggested by the carers, to trigger the prompts only for the mother. However, in other circumstances, the son declared he would like to hear the voice all the time: “If the sensor was installed on the front door, then I would definitely like to hear it and have an alert on my phone.”. All in all, despite the annoyance the familiar voice can sometimes trigger to non-end-users, the son added that: “The benefit of having it outweighs that, and the voice would not stop me from installing it.”.

The log data collected also indicates a need to distinguish between individuals. As advised by the son, “only triggers between 1 pm - 6 pm would be accurate” as the mother was visited by carers during the day. Focusing on this time interval, the prompt was triggered on average 8.75 times per day. Importantly, there were no incidences of voiding in inappropriate locations during the study. This suggests that IntraVox was an effective prompt.

4 Discussion

Here we discuss the outcomes of the study and how they address our RQs:

RQ1: Does IntraVox have the potential to prompt people with dementia to change their behaviour? Results show that IntraVox has the potential to prompt a lifestyle change by prompting end-users with dementia to change their routines and behaviours. Whilst embodied conversational agents also have great potential in supporting older adults [32], IntraVox is unique in that it uses a personalized human voice for delivering prompts and reminders to users with dementia with the purpose of supporting them in their daily activities. Moreover, IntraVox does not require any interaction as the system is not composed of any voice assistive device available on the market (Amazon Alexa, Google Home). We argue that overcoming the need to verbalize commands and remembering syntaxes results in an increased usability. IntraVox can always be adjusted and its purpose can be changed over time following an evaluation of the end-user’s needs. For example, IntraVox can also be used as a guiding system by using a motion sensor and a smart bulb and/or bar. Similar to previous findings [6, 16, 27], the results indicate that assistive technologies using tailored prompts to the end-user’s needs and the use of a human voice can be beneficial to people with dementia. Researchers designing voice assisting interfaces for people with cognitive impairments should consider using a familiar voice (e.g., a carer, a family member, or a friend) as this could create a sense of security and comfort.

RQ2: Do the tailored prompts used by IntraVox have an emotional impact on people with dementia and their carers? Prompts and reminders are important in dementia care [19]. The positive feedback and the emotions provided demonstrate that IntraVox was not received as a negative experience by the mother, it did not cause any emotional distress and has the potential to improve the quality of life of both the end-user and their carers. The son and carers were pleased that previous voiding in inappropriate places ceased, without intervention from them. This enabled their interaction to refocus on being together and enjoying positive interaction rather than focusing on cleaning the utility room. We believe that IntraVox’s main outcome demonstrated here was the improvement in the quality of life of the cared-for and carers, together with a reduced carer burden. People with dementia live with a range of symptoms and their response to the use of IntraVox will differ. Hence, careful assessment is required to ensure that technology is useful, appropriate, and, importantly, does not enhance anxiety.

We conducted the study as an experience-centred project [33] focusing on the mother’s interaction with IntraVox and her reactions and emotions when hearing the prompt. Having no incident during the study represented a huge achievement for her. This field study highlights the importance of designing assistive technologies based on the end-users’ needs and aspirations. Similar to [7], we believe that the findings could inform future larger studies focusing on evaluating assistive technologies for prompting and supporting end-users with dementia. Below we present additional key points arising from the study’s findings:

Multiple evaluation forms might need to be applied when evaluating assistive technologies and their emotional impact on users. Inconsistencies were noticed in the feedback provided, i.e., despite providing positive feelings when entering the utility room and hearing the prompt, participants found the prompt to be annoying. Similar to [26], this inconsistency can indicate that researchers may want to consider using more than one technique when assessing the success of an assistive technology and its emotional impact on users.

Understanding the end-user and the environment can improve the user experience when interacting with assistive technologies. When technology migrates to real-life settings, more usability problems are discovered than during laboratory evaluations [14, 17, 24]. The findings from this study indicate a need to distinguish between individuals and play the prompts accordingly. This is due to the fact that IntraVox does not recognize the person entering the room and plays the prompt whenever motion is detected. This led to the carers and the son sometimes being irritated by the voice repetition, whilst the mother was not. This indicates that researchers can design an assistive technology that may seem annoying to a regular user but not to the intended end-user.

5 Limitations and Future Work

The main limitation is the small number of participants taking part in the study. We would like to conduct more field studies with a higher number of participants to determine whether IntraVox would be suitable for people experiencing various cognitive disabilities. To develop the system further, we would introduce sensors that could differentiate between different users, as suggested in the discussion. In the future, thanks to recent developments in generative Artificial Intelligence, a synthetic voice might also be created to mimic a known human voice.

6 Conclusion

In this paper, we present the short field evaluation we conducted to evaluate IntraVox, a novel voice-based interaction system that uses a highly personalized human voice to send prompts to older adults with dementia. Results show the system has the potential to prompt a lifestyle change and increase the quality of life of the end-user and their carers. Despite the small sample, we believe that the findings can be generalized to individuals experiencing a wide range of cognitive disabilities. Moreover, the lessons learned can inform larger studies focusing on evaluating assistive technologies for prompting and reminding end-users with complex needs to conduct various daily activities.