1 Introduction

The research reported in this paper investigates the potential of Wearable Mixed Reality (MR) technology, particularly HoloLens, for persons with Alzheimer’s Disease (AD). Wearable MR enables users to exploit head-mounted displays to experience the surrounding real-world environment while computer-generated visual elements are overlaid on the world visualization. HoloLens is the most advanced product in wearable MR commercial technology. In HoloLens applications, the overlaid digital items are conventional digital items as well as 3D holographic objects, the latter providing realistic effects that can be smoothly integrated in the view of the physical environment. Since its launch in 2016, this device has been used in several domains such as data visualization, industrial training, tourism, education, and videogames. Still, HoloLens has been seldom exploited in the arena of mental health [5] and, to our knowledge, it has never been explored among persons with Alzheimer’s Disease.

AD is a form of Dementia. The latter is a collective name for a heterogeneous group of chronical neuro-degenerative diseases that are characterized by the progressive degeneration of the structure and function of the central nervous system or peripheral nervous system. This degeneration severely affects mental functions such as thinking, remembering, and reasoning skills and reduce the person’s ability to carry out the simplest everyday tasks. Dementia ranges in severity from the “mild” stage, when it is just starting to affect a person’s functioning, to “severe”, when the person must depend completely on others for basic care [4]. The global impact of dementia in our society is increasing as a direct consequence of the increased age of population. Some studies report that in 2015, almost 46.8 million people worldwide were living with Dementia, and this number will probably reach 131.5 million in 2050 [28].

AD is characterized by memory loss and impairments in cognitive abilities such as word-meaning matching, vision/spatial awareness, and appropriate judgment formulation [2, 3]. Most of the current interventions attempt to mitigate the AD effects using medications, which help people with AD to manage some behavioral symptoms [26]. Specialists are also exploring new directions to intervene at the level of cognitive functioning and to delay mental decline, e.g., engaging AD subjects in cognitive training exercises (also referred to as “brain training”) [29]. Cognitive training tasks are devoted to practice memory, problem-solving skills, and language ability [2]; they employ physical materials (e.g., paper & pencil and cardboard cards) or “general” (i.e., not natively created for AD) commercial applications for brain training available on personal computer, tablets and smart phones. To our knowledge, the use of more advanced forms of digital interactive technology in the treatment of AD is largely unexplored.

In collaboration with a team of AD specialists we created MemHolo, an innovative HoloLens application for elders with an initial stage of AD. MemHolo is intended to be used as a cognitive training tool that complements regular therapeutic interventions and enables patients to practice short-term and spatial memory skills in a safe and controlled virtual environment.

The use of Wearable MR in general, and HoloLens in particular, is associated to several challenges, particularly when we address persons with AD. This technology is relatively new, and few guidelines exist on how develop usable interfaces, how to minimize physical side effects that are typical of Virtual Reality (VR) experiences in head-mounted displays (e.g., motion sickness and digital eye strain), and how to create digital contents objects whose design is appropriate for the integration in the view of the user’s physical world. In addition, it is hard to predict how the persons with AD would react to the use of the device. It is generally acknowledged that older adults struggle with new technologies [7], and this problem could be further exacerbated by AD impairments.

To our knowledge, MemHolo is the first example of HoloLens application natively born for the elderly with AD. Our main contribution in this paper is to exemplify how traditional brain training activities can be translated into wearable MR experiences for this target group. In this respect, our work can be useful for future designers of MR applications for cognitive disability, also beyond AD. Our research also sheds a light on how people with AD perceive HoloLens applications, and pinpoints potential benefits and drawbacks for these persons, paving the ground for new forms of treatments in this field.

The rest of the paper is organized as follows. After a review of the state of the art in VR and MR technology, and their application for persons with cognitive impairments like AD, we discuss the needs of therapists and persons with AD that we elicited with the team of AD specialists, and how we address such needs during the design process. We then present the initial prototypes of MemHolo and two explorative studies that we performed during the design and development activity. We discuss how the insights from this empirical work informed the design of the last version of MemHolo, which was evaluated in a more systematic empirical study at a care center involving 11 elderlies in an initial stage of AD. The last section draws the main conclusions and outlines the future directions of our work.

2 Related work

2.1 Mixed reality and Microsoft HoloLens

The term MR was used for the first time in 1994 by Paul Milgram and Fumio Kishino in their paper “A taxonomy of Mixed Reality Displays” [23]. They used the concept of the virtual continuum to define the space between two opposite realities: physical reality (the real world in which we exist) and digital reality (the world of computers and artificial contents). As shown in Fig. 1b, on the extreme left-hand side we have Augmented Reality (AR) – interactive experiences that superimpose video or graphics on the view of the physical world; on the extreme right-hand side we have VR – interactive experiences that make the user feel immersed in a digital environment [35].

Fig. 1
figure 1

a Microsoft HoloLens device (a.1) with the clicker (a.2); b the MR spectrum (b)

Full size image

MR is the blending of these two extremes, as it happens in applications based on Microsoft HoloLens (Fig. 1a), which integrate 3D holograms with the surrounding real world (seen by the user through the head-mounted device) as if they were truly there.

To create holographic objects “in the real world” with a high degree of realism, the HoloLens device has lenses that use an optical projection system and generate 3D full-color holograms. Furthermore, it contains cameras and sensors to capture information about what the users are doing and the environment around them. HoloLens provides various interaction modes since it can interpret gestures [17], gaze [16], and voice [33], as well as the input of a “clicker (Fig. 1a.2). For the purpose of MemHolo, we considered all these interaction modes, about which we provide some details in section 5 (“MemHolo”). Thanks to spatial sound-synthesizer, the user can hear sound effects from anywhere in the physical space [19]. HoloLens can be connected (via Wi-Fi or USB cable) to the Microsoft device portal [32], and allows other people (such as therapists, family members or other patients) to see on an external digital display what the user is viewing inside the device.

2.2 Virtual reality for persons with cognitive impairments

Existing MR applications for persons with cognitive impairments such as AD are located in the extreme right-hand side of the MR spectrum (Fig. 1b), i.e., Virtual Reality (VR).

Some VR systems in the literature use interaction devices such as a joystick, keyboard input, or touch displays [6, 10, 11, 22, 31]. Still, even these standard interaction paradigms seem to confuse elderly people, who have limited or no experienced in using such devices [34]. Few researches in interactive technology for mental health investigate the use of Wearable Immersive Virtual Reality (WIVR), which enables users to experience 3D virtual spaces using Head Mounted Displays (HMDs) and supports the feeling of being “inside” these digital environments. An example is reported in [21]. The authors made an empirical study to explore whether persons with cognitive disability could learn to navigate in a simple WIVR world (a virtual supermarket) that represent an everyday life environment, and whether this experience could improve their capability of moving inside the corresponding “real” physical space. The results show that the study participants could understand the virtual environment and improve their navigational skills inside the VR world, but there was no evidence of cognitive skills generalization to real life contexts.

The project by Alzheimer’s Australia Vic and Opaque Multimedia [1] developed a WIVR application that exploits a virtual forest created from real omnidirectional photographs to provide an engaging experience for people with dementia. Waving their arms, patients can make the sun rise or the wind blow through the trees, or can change the weather conditions from sunshine to snow; a clap of the hands causes the leaves to fall or the birds to fly up. This work differs from MemHolo in terms of both the technology used (wearable VR and not wearable MR) and the application goal: The aim of the “forest experience” is not to delay the symptoms of dementia, but to trigger a sense of wonder and fun that eventually people may remember, and feel again, when visiting an Australian real forest.

Existing studies highlight two main advantages of VR experiences over traditional practices: for therapists, the possibility of offering a variety of controlled stimuli and tasks, the availability of tools for higher control on such stimuli and for monitoring users’ responses automatically; for patients, the opportunity to perform therapeutic activities in a safer and less boring environment [23].

At the same time, some researchers (e.g., [14, 15]) pinpoint the potential drawbacks of immersive virtual environments, which are associated to physical and cognitive side-effects such as “motion sickness”, “digital eye strain”, and “isolation effect” [19]. Motion sickness is caused by the disagreement between vestibular system’s sense of movement our body and the visually perceived movement in the simulated world [27]. Digital eye strain is defined as eye fatigue (the feeling that our eyes are burning, itchy, and tired) and is induced by the exposure to digital displays. Isolation effect is the feeling of being too much inside your own mental and emotional world and “away” from the physical environment and the people surrounding you. The work described in [15] investigates WIVR in interventions for Neurodevelopmental Disorder (NDD). The authors developed a WIVR application for a commercial low cost VR viewer (Google Cardboard) that provides simple games in a storytelling environment and performed an exploratory study at a care center involving 11 children with NDD. The above mentioned physical side effects were noticed initially in several subjects, but disappeared after children familiarized with the device.

3 Requirements elicitation

The starting point of our research was a meeting and two focus groups with a team of AD experts (neurological doctors, psychologists, and caregivers) from the Gerontological Division of a large Italian Hospital (University Hospital in Palermo), who were looking for an innovative way to involve elders with AD in cognitive training tasks.

The needs and requirements elicitation process of MemHolo unfolded along several steps: User profiling (to understand the salient characteristics of persons with AD); Overview of current therapeutic practices using brain training; Analysis and selection of the MR technology to use; Target Group selection; UX and Tasks design.

At the beginning of our work, the specialists explained us the physical, mental, and attitudinal characteristics of the elderly person with AD and how these increase the complexity of introducing new technology-based treatments: sensory impairments (such as vision and hearing deficit), memory loss, decline of motor skills and motor control, and a biased, afraid attitude towards new technologies [22].

Then the experts presented examples of cognitive training activities adopted in their current practice. They mainly focus on memory exercises and use paper-based materials such as cards or drawings. Because of the limited number of physical materials and tasks, this approach suffers from several problems: the progressive decrease of engagement (the experience becomes boring and repetitive soon), the limited number of customization options, physical material deterioration, the difficulty of keeping the patient seated for a relatively long time during task execution, and the burden of recording manually the relevant data about the patient’s evolution, e.g., task accomplishment time and number of right/wrong answers.

Specialists had the hope that these drawbacks could be mitigated through the adoption of a VR-based approach, of which they had heard from TV programs or web news. We presented examples of different VR technologies: AR on tablet, Wearable Immersive VR (using different headsets such as Google Cardboard, Samsung Gear, and HTC Vive), and Microsoft HoloLens MR. After the AD experts tried these devices themselves and understood how they work, we discussed the potential benefits and drawbacks of each different solution, and eventually selected HoloLens.

Several reasons support this choice. A HoloLens experience combines the benefits of VR with the ones granted by the interaction with the real world: it enables the user walk around and explore the surrounding physical space while perceiving limits and obstacles in the room; it offers the same visual stimuli as the real space, but extends them with new stimuli created by 3D holograms (and other digital contents). According to the specialists, these features could add engagement and realism to the experience, could help users maintain the awareness of the surrounding environment, and could reduce the risks of the side effects discussed in section “Related Work”. In addition, HoloLens applications can perform automatic data gathering on performance features extracted from log data, which are helpful for patient’s monitoring and assessment.

Concerning the initial target group, the AD experts suggested to address persons with an initial form of AD, for whom early treatments has higher change to lead to improvement or at least to suspend mental decline (compared with persons with severe forms of AD).

The AD specialists invited us to refer to a simple brain training exercise that they frequently use in their practice: a therapist puts on the desk a number of cards representing different objects, asks the patient to look at them for a specific amount of time and then to close her/his eyes, removes a card, and finally asks the patient to look at the desk again and remember which item is missing. This exercise aims at training short-term and spatial memory skills, particularly “allocentric” spatial memory (which encodes the location of an object with respect to another object in the scene, as opposed to “egocentric” spatial memory, which encodes the location of an object relative to the body axes [9].

The AD experts also suggested us to take into account some heuristic rules that they apply during the execution of the above exercise, for example, to design tasks at different levels of complexity, in which the number of objects to remember can be changed by the therapist, up to a maximum of 15, and to enable the therapist use objects with different visual characteristics (shape, color, position).

4 MemHolo

The design and development of MemHolo were an iterative process that involved two progressive prototypes explored in two studies on the field, and a final prototype which was evaluated in a more systematic way. Since the beginning, the system provided three slightly different activities at increasing difficulty levels. They are inspired by the brain training exercise discussed during the requirement elicitation process, and can be customized by the therapist in terms of complexity in order to address the specific needs of each single person with AD. In addition, MemHolo keeps track automatically the global completion time for each activity as well as the number of right and wrong answers, and save these data both on the device and on the therapist’s Microsoft OneDrive account.

The MR environment in which all interactions and tasks take place is the physical room in which the patience is using the device. In all progressive prototypes, the application starts by showing a fluctuating main menu which allows the therapist to “select” the desired activity among the three available ones (Fig. 2a), and in the proper configuration.

Fig. 2
figure 2

Pictures of the fluctuating main menu with the three activities’ buttons (a) and the reference object, which the user had to remember, with the instruction to start the activity (b) of the second prototype of MemHolo

Full size image

4.1 First activity

This activity is the simplest one and is meant to be used for both warming up and as an initial training of short-term memory. The patient’s goal is to find the box where the reference objects is hidden.

The application displays, just in front of the user, a virtual 3D object randomly chosen for each session. This is the “reference object”, i.e., the objects that the patient should memorize (Fig. 2b). When s/he is ready to start the task, the user must “confirm” to proceed with the experience. The application displays a grid of 8 fluctuating boxes aligned on 2 rows. Each box contains a different object (from a set of 8 objects) which is assigned at random each time the activity starts. The user must “open” a box (by “selecting” it) to see the content. A message asks whether the object inside the box is the reference one or not (Fig. 3a). If the user’s answer is wrong, the box is shut, the count of wrong answers increase, and the patient can continue the exercise; if the answer is correct, the activity is completed, the count of right answers increases, the completion time is recorded, and user can proceed by selecting a new activity or repeating the current one.

Fig. 3
figure 3

Pictures of the activities from the point of view of the HoloLens user. The first image (a) shows the moment in the first activity when the patient has opened a box and have to confirm if the inner object coincides with the reference one. The second (b) shows the same moment, but in the second activity; here there are two boxes to open and find the right couple of objects. Finally, there is the third activity when all the boxes are spread on the room’s floor and table (c) and when the patient has opened two boxes (d)

Full size image

4.2 Second activity

The second activity is inspired by the popular “Memory Game”. The patient’s goal is to discover two identical objects among those ones hidden inside the boxes (aligned in 2 rows, as in the first activity - Fig. 3b); s/he must open two boxes and to confirm if the objects inside both of them match or not.

4.3 Third activity

The logic of this activity is the same as the one of the second activity (finding pairs of identical objects); still, the spatial allocation of boxes is not regular nor predictable: the boxes are no longer in a fixed position (aligned on a grid) but are randomly scattered in the surrounding real space. The patient must explore the space to find the boxes and must moves close to a box in order to open it. After disclosing an object inside a box, the user must look for the second identical object (Fig. 3d), and open another box: if the object inside is not “correct” (i.e., not identical to the first one disclosed) the boxes of both the first and the second object are closed and the “first object to match” must be selected again. To increase complexity of the exercise, the activity set up can be “upgraded” so that in case of failure the user must look for exactly the same previous object as “first object to match”.

This activity is the most complex one from a cognitive perspective, since it requires the user to remember not only the object to match but also the object position in the space. It trains both short term and allocentric spatial memory. In addition, it stimulates spatial awareness and motor skills, since the tasks involve moving in the physical environment and avoiding obstacles.

4.4 Interaction modes

In the real world, people look at an object when they are interested to and would interact with it. MemHolo uses the information about the user’s gaze in the MR space to determine the person’s focus of attention or interest and to identify what s/he is “selecting” (or “targeting”). Then MemHolo exploits all three built-in interaction modes (gesture, voice, and clicker) to “activate” the selected object.

It is important to highlight that the built-in gaze technology of HoloLens uses the position and orientation of user’s head, not eye tracking, to determine the gaze vector [16]. Following existing design rules about HoloLens’ gaze indicator [12], we decided to keep the targeting cursor always visible in our scenes. We also provide some visual interactive effects to mitigate the risk of “feeling lost in the VR space” and to increase the user’s understanding of the interaction state. A different visual feedback (changing the cursor’s shape and color) is used to distinguish if the user is selecting an object or not. We change the cursor’s shape (the blue little hand-shaped symbol in Fig. 2a) also when HoloLens detects a hand in the ready-state; finally, we highlight the “targeted” object by making its color and brightness more intense.

To “execute” the targeted objects, i.e., to trigger the visual or audio feedbacks associated to the selected items, we explored air-tap interaction in the first prototype, air tap and voice interaction in the second prototype, and added the use of the clicker in the third prototype.

Air tap gesture consists of a “press” followed by a “release”. The user makes a fist in front of her/himself, with the hand back facing her/him and with the elbow bent. Then, s/he raises the index finger upwards and, finally, flexes it down and then back up [17]. HoloLens can recognize these movements by tracking the hands’ position that are visible to the device. It only sees hands when they are in either the ready state (a fist with index finger up) or the pressed state (a fist with the index finger down).

For voice recognition, MemHolo uses the HoloToolkit Keyword-Recognizer, which recognizes a given array of string commands to listen for [33]. Unfortunately, HoleLens voice recognition works correctly only among users who can read and pronounce English words well (English is the only language supported by HoloLens at the date of writing of this paper).

Clicker-based interaction exploits a physical button-shaped controller (Fig. 1a.2) that is connected to the headset via Bluetooth and can be used in place of air-tap to activate or move a selected interactive element.

4.5 Spatial mapping

The third activity requires the creation of an accurate digital representation of the environment surrounding the user room and its physical surfaces (e.g., tables, floor, and walls), in order to place the boxes dynamically in such a way that their position satisfy the proper spatial relationship (e.g., “on” or “near”) with these surfaces. For this purpose, we use an interesting and powerful built-in feature of HoloLens: spatial mapping [30], exemplified in Fig. 3c.

Thanks to spatial mapping, the digital visualization of the room in which the execution of this third activity takes place is generated automatically and in a relative short time (of the order of seconds), depending on the room’s dimensions and the desired level of detail. Spatial mapping can be performed by therapists autonomously, at the beginning of a therapeutic session, or in advance (if the room’s setting does not change), saving the room mapping and re-using it as many times they want.

When the third activity is launched, MemHolo starts a new scansion of the physical space for 20 s. It automatically recognizes whether the real room is one previously saved or a new one. In the first case, it will reuse it. In the second case, it creates a new map and saves it. Once this step is completed, the digital boxes appear in the real space view. MemHolo recognizes floor and tables, and draws a colored plane on them (blue and green respectively, as can be seen in Fig. 3c). If the therapist or the patient prefer to see the real surfaces, the planes and the wireframe mesh on the background (Fig. 3d) can be set transparent. The application is implemented in such a way that the objects are positioned in a visible and easy to reach positions (for example, not under tables, nor behind big furniture, nor too far away from the patient), considering that the possible motor impairments of our target group.

5 Exploratory studies

Two exploratory studies, respectively using the first and the second running prototype, focused on the usability of the device and MemHolo interaction mechanisms.

5.1 Participants

Overall, in the two studies we recruited 30 persons (21 men and 9 women) at a local retirement house, aged 64–67, which is the average age when the AD symptoms are more frequent. Still, these persons were not yet diagnosed with AD but they were considered a good sample for the purpose of our study, since AD does not affect a person’s functional capability of interacting with a device like HoloLens, and the subjects had an age compatible with the one of our main target group.

We randomly assigned participants to two homogeneous groups each one composed of 15 persons. The first group was involved in the first study and used the first prototype with air-tap gesture-based interaction. The second group used the second prototype, in which we implemented some design improvements based on the results of the first study and made available both air-tap gesture and voice interaction.

5.2 Procedure

In both studies, the experience with MemHolo took place in the same room, a space that participants were familiar with. Each subject was initially introduced to what the application’s aim was about, to the tasks of each activity, and received an explanation about the interaction mechanism (the use of the gaze to target objects, how to perform a selection, and the position of the hand to air-tap) with a demo by the evaluators. Then each subject tried the application autonomously, performing the three activities starting from the first one. We adopted a “thinking aloud” method for data gathering, asking all participants to think aloud during the experience. Observers from the design team took notes during the sessions.

5.3 Results of exploratory study 1: observations and suggestions for re-design

All participants in our first study wore the HoloLens headset with no problem for the whole session and none of them manifested motion sickness or eye fatigues. This result suggests that this device could be more comfortable and induce less physical side-effects compared to other WIVR equipment (such as Google Cardboard, discussed in [15], although wider studies are needed to validate this hypothesis.

Overall, all users showed a positive feeling towards HoloLens and MemHolo. Many participants used the terms “entertaining, “surprising”, and “engaging” to describe the experience. The tasks were defined “easy to understand”.

Still, some participants also mentioned that some objects were too small to immediately identify their meaning, and that they could not easily see the instructions in the room (Activity 3). In addition, when moving around objects, users could not see the instructions correctly, especially in the third activity, because the instructions’ messages were placed in a fixed position above the boxes instead of “moving” with respect to the user and consistently with his/her point of view.

Concerning the interaction mechanism, all participants could easily point with gaze to “select” an object. The objects “execution” raised some usability problems. Only two users accomplished correctly all tasks in the three activities. Even if all participants declared they had understood the interaction mechanisms after the explanation, i.e., before the actual use of the system, all of them found difficult to perform the air-tap gesture, which sometimes caused frustration or confusion. They did not position their fingers in the right way (not rose the index enough) or they placed the hand out of the HoloLens field of view. They forgot that their hand had to be in the HoloLens’ field of view.

Considering the results of the first study, in the second prototype we implemented the following improvements:

  • we increased the boxes and objects size, and the distance between among them, to improve visibility and prevent possible errors in gaze targeting

  • we modified the cursor so that it changes its shape not only when is over an object but also when the user’s hand is in the field of view of HoloLens (Fig. 4b)

  • we included sound feedbacks when an action is impossible or it has been already performed (such as opening an already opened box)

  • we added voice interaction, enabling users to use voice commands while performing a task (e.g. “open box” or “close box” to see/hide the box’s content, or “yes” or “no” for answering question or confirming actions)

  • we included textual instructions that “follow” the user. To do this we used two HoloToolkit scripts: Billboard and Tagalong [18]. They modify automatically the orientation and the position of the instructions panel so that messages always face the user and are always at a constant distance in his/her field of view (Fig. 4d).

Fig. 4
figure 4

First (a) and second (b) version of the main menu and a comparison between the first (c) and second (d) version of MemHolo during the activities where can be seen the improved readability of the instructions and the objects

Full size image

5.4 Results of exploratory study 2: observations and suggestions for re-design

Explorative study 2, involving the second prototype of MemHolo, provided encouraging findings. We recorded similar positive results as in the first study, in terms of engagement and lack of physical side effects. In addition, all participants were able to accomplish all tasks. Only one person had some initial difficulties to find objects in the first activity. This was due to the user’s limited field of view of HoloLens caused by the wrong wearing of the device. The problem disappeared when the headset was set correctly.

Considering the interaction method, all participants initially tried both air-tap and voice commands (in the first activity). Then all of them preferred to use voice commands, observing that it was “more intuitive”, which suggests that voice-based interaction, natively supported by HoloLens, can be particularly effective for the elderly.

The users’ performance with MemHolo was independent from the participants’ level of familiarity with tablets, PCs, or smart phones. Even participants had never or rarely used a computer were able to accomplish the proposed tasks correctly. This result - which deserves further investigation - seems to support a hypothesis that the neurologists in our team formulated since the beginning of our work, namely, that “the users’ pre-existing knowledge of digital technology does not influence their behavior with and their attitude for this new device”.

Ten participants mentioned that they appreciated the increasing difficulty of the activities and particularly liked the third one (in which the spatial-memory component is more important, and there is the remarkable effect of holographic object overlay in the real space). In the second prototype, instructions did not overlap the scene and were always readable and in the user’s field of view and all participants could easily read the instructions. The result indicates that this design solution was effective, and designers of future HoloLens applications may consider applying it.

We observed that some participants easily moved in the room to look for the two identical objects. Still, most of them maintained their initial position for the whole session, because of motor impairments or because of a limited attitude to movement that is frequently found in the elderly. If they could not see an object (e.g., because it was in a box of the behind row) their movements were usually just a rotation of the head or the body, but not a walk towards the object. We had not considered this issue when we allocated the objects in the MR space. We took it into account, together with other improvements suggested by AD specialists after analyzing the study results, in the third prototype.

Instead of placing objects horizontally on an imaginary plane, in the revised version of MemHolo we put them vertically so that the user could see them all together, even if s/he was seated, as shown in Fig. 5d.

Fig. 5
figure 5

Pictures of the third prototype of MemHolo. After choosing the desired activity from the main menu (a), the user has to remember the reference object (a rose, in this case) and confirm using the clicker (b). Then, all objects are visible for the defined amount of time (c) and then hidden in the boxes (d). When the user opens a box, he/she must decide whether the inner object is the correct one or not (e); at the end, the results on task execution are shown (f)

Full size image

In order to further stimulate memory and facilitate task execution, some therapists suggested to show all objects, their positions, and the associated instructions, before the start of an activity, for a period that depends on the profile of each specific patient. We implemented this feature and added a new parameter in the main menu that can be changed by the therapist for each user. The yellow highlighted number in Fig. 5a represents the number of seconds during which all instructions and objects are visible (Fig. 5c) before hiding them into the boxes (Fig. 5d). The red and green buttons under it enable to decrease and increase its value respectively.

Lastly, we integrated the HoloLens’ clicker in MemHolo as an additional interaction mechanism that can substitute or complement air-tap gesture and voice commands in case patients have difficulties in performing the air-tap or cannot express instructions in English. The introduction of this control device required some new interface features, namely, virtual buttons which can be activated using the clicker. As shown in Fig. 5b, the initial instruction panel was modified to include a light blue button which allows starting the activity. Similarly, the confirmation panel (Fig. 5e) in the third prototype has two buttons (one green and one red) to confirm or deny whether the displayed object corresponds to the reference one or not.

6 Summative evaluation

In this section, we discuss the evaluation of the final (third) prototype, which involved subjects at an initial stage of AD.

6.1 Research goal and data gathering method

This study involved patients at the first stage of AD, i.e., persons who manifest AD initial symptoms – mild forms of mental decline and memory loss – and have a psychological rigidity by effect of the disease, which causes suspect or resistance towards any unknown situations (especially if they do not have a high educational level) [7].

The goal of the study concerned several aspects. First of all, we were interested to observe the attitude towards MemHolo. More precisely, we wanted to investigate if our target group manifests curiosity towards this new technology and is willing to use HoloLens and MemHolo. This is a pre-condition for the adoption of this tool and for the effectiveness of this novel form of brain training treatment.

In addition, we wanted to explore the degree at which the interaction modes of HoloLens, and the specific interaction and interface features of MemHolo, could be understood and learned by this target, or if instead the experience with MemHolo would be frustrating, leading to abandoning the cognitive training tool.

We measured the above aspects using qualitative and quantitative methods:

  • a questionnaire that each patient had to fill at the end of the session. The questionnaire focused on the device acceptability. The willingness to use MemHolo, and the general feeling towards the experience. The questionnaire included a simplified version of System Usability Scale (SUS) test [8], considering only the most relevant questions, adapting them to our case, and removing all negative statements. We used a Likert-type scale with five options (from 1: strongly disagree, to 5: strongly agree with the sentence) for each question.

  • therapists’ and observers’ annotations to measure “task accomplishment”, “needed support from caregiver”, patients’ reactions and behaviors during the use of MemHolo

6.2 Participants

The study took place at “Filo di Arianna” center for elders in Milan. After a focus group with the local caregivers, we recruited 11 guests who attend the center regularly and had a similar age as the participants to the previous studies, i.e., between 68 and 94 (84.18 ± 7.24 years old).

Table 1 shows the participants’ characteristics and considers gender, age, educational level, English speaking capability, and MMSE score (24.18 ± 4.07). MMSE (Mini-Mental State Examination [13]) values indicate the participants’ clinical profile. MMSE is a 30-point test widely used in clinical and research fields as quantitative assessment of cognitive impairments in adults [20, 24]. A score greater than 25 is usually considered normal cognition, 24 is the borderline value, between 23 and 19 mild cognitive impairment and below 18 moderate and severe impairment [25].

Table 1 Profile of the participants of the empirical evaluation
Full size table

As shown in Tables 1, 6 participants have a MMSE score which is “normal” or higher. Still, they were considered as part of our target group since they manifested initial AD symptoms, even if the disease had not yet affected cognition at a level that could be captured by MMSE measures.

6.3 Procedure

Evaluation sessions took place in a room where the study participants perform recreational activities, and which was associated with pleasant experiences.

At the beginning of the study, a member of the design team presented the project to the participants, seated in circle in front of her. She explained the goal of the research and gave a demonstration of MemHolo while HoloLens MR contents were visualized on an external digital display.

In the following days, we organized the sessions of use of MemHolo, one for each patient, and defined the execution protocol.

The user seated next to the PC, wore HoloLens and the clicker, received a short explanation of MemHolo activities, and was invited to perform them one after the other, “thinking aloud” during the experience and communicating how s/he felt and whether s/he experienced tiredness or distress. A caregiver was sitting side-by-side answering questions and giving suggestions when needed (Fig. 6). Since the tasks in the third activity could be too complex for people who have sever motor impairments or use a wheelchair, we did not ask these people to perform these tasks. At the end of the session, the user filled the questionnaire, while we arranged the set-up for the next user.

Fig. 6
figure 6

Third study. All participants used MemHolo autonomously, with caregiver sitting nearby to answer questions and listen to comments and feedbacks

Full size image

In the initial design of the study, we planned to have each participant use MemHolo alone at the presence of his or her caregiver and observers only, for approximately 15 min. Still, during the initial presentation of the project we noticed that the show of HoloLens contents on a shared display attracted the persons’ curiosity and was perceived as a pleasant and socially engaging activity. People shared comments on what appeared in the MR space and on the performance of the person using the device, giving him/her encouragement and suggestions. To preserve the effects created by this setting, we redefined the session format. We organized participants in small groups, and each person used the application (see Fig. 6, top-left) while the others looked at the MR contents on the external display and talk. A group was composed of 2–3 persons, to avoid long queuing and allowing a better visibility of the external display.

6.4 Main results

As can be seen from the questionnaire results in Fig. 7, the majority of the participants assigned a positive or very positive score to the experience and seemed to be willing to use the system again.

Fig. 7
figure 7

Questionnaire aggregated results showing the “number of patients’ answers” for each item

Full size image

Among the main findings emerging from observers’ note, a frequent difficulty was related to the use of the clicker, since it requires fine-motor control skills that were weak in some persons. This problem can be alleviated by asking the user to practice with the mechanics of the clicker before using MemHolo. One participant had some difficulties in reading the instructions, but this was due to her eyeglasses (the bigger the distance between HoloLens’ lenses and the user’ eyes, the smaller the field of view).

During our initial focus groups, therapists mentioned the risk that the MR context could have an influence in patients with an initial form of AD on the ability to distinguish what is real and is virtual. Indeed, this did not happen among our study participants. All persons could understand that holograms were “fictitious” while the elements of the environment were the ones in the room where they were playing. They also enjoyed the graphics and visual effects of the different virtual objects, expressing comments such as “Wow, look at this, … it looks like a cartoon watched by my grandchildren! They should try MemHolo!”

To assess “task accomplishment”, we refined this concept into finer grained elements referred to as “goals”. The goals are: “completing the activity”, “understanding the aim of the activity” (remembering the reference object, finding it in a box and confirming the answer), “pointing” (seeing the cursor and controlling the gaze vector) and “selecting” (using the clicker to interact with the application).

Then we evaluated a goal achievement in terms of “needed support”, i.e., considering the amount of aid required from the caregiver. The data for these measures were extracted from the notes taken by observers. For each goals, we used the following 4-points scale:

  • 1: the user cannot achieve the goal

  • 2: the user achieved the goal, but needed continuous verbal support or even a physical aid (for example: the caregiver moved his/her head to allow them to see the boxes)

  • 3: the user achieved the goal and needed only a small amount of help (asking few times if s/he was doing correctly)

  • 4: the user achieved the goal autonomously, after the initial explanation.

Figure 8 shows the results obtained from the data gathered during the validation sessions, which confirm the trend emerged from the questionnaire data regarding gaze and clicker.

Fig. 8
figure 8

Quantitative evaluation of sub-tasks completion in terms of “support from caregiver”. The values represent the number of patients who complete the sub-task with the specific amount of help

Full size image

7 Discussion and design highlights

The results of our empirical studies are still preliminary and do not meet the requirements of statistically significant research, and should be taken with caution. There are limitations due to the limited number of participants, their heterogeneous profiles, and the short time of participants’ exposure to MemHolo. Still, our findings could be interesting for future research in the field and may trigger further work in the domain of MR for mental health.

Further insights were offered by the AD specialists in our team during a final workshop which was organized to draw the main conclusions about the project, and to identify positive aspects, weaknesses, and suggestions for future work.

One of the main pros of MR that AD experts acknowledged was the possibility to interact, at the same time, with the virtual and the real world, reducing the risk of triggering the “isolation effect” [19] that may occur in wearable VR. HoloLens technology surrounds the user with interactive virtual content, that could be designed for several different experiences, without the isolation effects. Especially for people with AD, the presence of a caregiver or a family member is important. They also needed to see a familiar environment to avoid getting scared or confused.

At the beginning of the project, caregivers at the center were wary of the elders’ willingness to take part in the research, overcoming the psychological resistance that they usually manifest towards a new situation. In contrast, MemHolo created a surprising level of curiosity and engagement in participants, and improved socialization. When a user wore the HoloLens, s/he could hardly believe that s/he could see virtual 3D objects and usually her/his feelings moved from curiosity to surprise or astonishment. The level of detail of the 3D models (which is, in our opinion, one of the most important component of a MR experience) captured the participants’ attention and increased the engagement. For instance, when we asked to a participant what the reference object was (a rabbit), she answered in an unexpected way: “a funny rabbit with emerald-colored eyes and big teeth”.

At the same time, we suggest designing carefully the degree of realism of the virtual elements when creating MR applications for people with AD. As already mentioned, before starting the empirical study therapists wondered if patients could have difficulties in distinguishing “the real” from “the virtual”, and could get confused. To mitigate this risk, we designed the virtual objects as something that could be easily identified as “not real”, because of their visual characteristics or their inconsistent semantics with respect to the current physical context (e.g., a rabbit or a rose). Moreover, we did not place virtual objects on the real surfaces (e.g., on the table) but left them fluctuate in front of the user to further reduce the realism. This solution seemed to be effective, since none of the patients manifested confusion or real-virtual cognitive mismatch during our tests.

Another interesting highlight was that presenting this research as a group activity, even though a person per time used HoloLens, created a collaborative environment where the others could help the subject during the activities and performed themselves a form of cognitive training. While waiting for their turn during sessions, participants talked about the experience with MemHolo and gave suggestions to the current user. In addition, this interest persisted after the sessions, as MemHolo was the subject of conversation among participants and with families in other moments of their life at the center.

The opportunity of using HoloLens (a “futuristic device”, according to their words that “neither grandchildren had ever used”) and contributing to a scientific work in collaboration with a research institution made participants feel a sense of usefulness and increased their self-esteem. This socio-cultural benefit, which go beyond the functional and pragmatic effects of our application, deserves further reflection on the role of advanced technology among the elderly.

Some final requirements for improvements concern both the design of the future versions of MemHolo and how to perform the session of use of MemHolo:

  • Two times it happened that the users stuck on looking at the initial instructions. This indicates that instructions might become a distracting component rather than a helpful resource. Therapists suggested to make instructions disappear after a certain amount of time.

  • The limited field of view of HoloLens was a relevant restriction. Even though the “tagalong” feature kept the activity objects in the field of view of the user, at the beginning the patient could not find all boxes, particularly if they wore eyeglasses (that further limit the visibility). This problem could be mitigated by training the patient to move the head instead of the eyes in these situations, since it is the head movement and not the gaze movement that determines the change of the field of view.

  • We used a 13” PC monitor to show the experience of the user wearing HoloLens. A bigger display would be better considering the possible sight impairments of the elderly. A technical difficulty for the therapist’s work during the session was introduced by the synchronization delay of the Microsoft device portal preview originated by a non-optimal infrastructure set up. Since we did not have a fast Wi-Fi connection, the PC display was connected to HoloLens via USB cable, and showed what the user was seeing in the MR environment with several seconds of delay. Due to this temporary mismatch, it happened that therapists gave wrong instructions to the user. Therefore, we employed the external monitor only to show the activity to the other patients but we asked the patent to explain aloud what s/he was doing, so that the therapist could provide contextualized support

8 Conclusions and future work

To our knowledge, our research is the first one that explores HoloLens applications for people with AD. During the development of MemHolo, we faced several challenges such as the limited number of design principles for MR applications – particularly those for old adults (with or without AD), the relative novelty of HoloLens technology, and the lack of guidelines in running empirical studies on MR for this target.

Our main contribution is to exemplify how MR technology, particularly HoloLens, can be designed for persons with AD, to discuss potential benefits and drawbacks of HoloLens applications for this specific target, and to provide highlights on some socio-organizational aspects related to the use of these systems in interventions for these people. Our work can also benefit the designers of HoloLens applications for the elderly, pinpointing pitfalls that they may encounter, and offering some suggestions on how to avoid them.

Concerning our future research, the natural follow-up in the short term will be to perform a longer and wider evaluation, with a larger number of participants who will be involved more frequently and for a much longer time span - at least 6 months. This work will provide a stronger empirical validity for the results of our exploratory studies. In parallel, we will improve the interaction and visual features of MemHolo to meet the new requirements emerged from AD specialists’ suggestions and from field study observations, we will extend the set of memory-training activities, and we will re-engineer the technical architecture of the system to make it more robust and performant.