Background

The prevalence and effects of dementia

Currently, the German Statistical Institute predicts an increase of people older than 65 years from 17 + million in 2015 to 22 + million by 2060 [21]. These numbers are alarming, as the likelihood of becoming an elderly person with dementia (PwD) correlates with the age of the individual. It is estimated that some 1.6 million PwD are living in Germany. In turn, this number could rise to 2.9 million or more PwD by 2060 [5]. Dementia is defined as a major loss of cognitive functions caused by a distinctive and long-term functional impairment of the brain [7]. Subsequently, impaired autobiographical memory function can lead to a loss of identity [1]. At the moment there is no known therapy that can stop the progression or reverse the effects of Alzheimer’s disease (AD) and although medical research is working to find a cure [19], modern non-medication-based reminiscence therapy (RT) is a viable way to preserve the personhood of PwD.

Activating and occupying PwD via person-centered RT

In order to counteract the consequences of the memory decline, modern dementia care and the accompanying research aim to keep the PwD active and occupied as long as possible and with that to keep the quality of life (QoL) on a steady level. This is in line with the person-centered care approach [12], which propagates to foster and preserve the personhood of the individuals in their day to day care and with that maintain their well-being. As a consistent measure, RT sessions can elicit various activating effects (behavioral and/or verbal) that in turn generate a positive impact on the cognition [22], identity and QoL of PwD. With reminiscence also being referred to as an emotion-based intervention, research suggests that it can evoke positive emotional effects on interest, interaction and mood as well as reduce unwanted behavioral effects [6]. To further clarify, the RT approaches mentioned in this article have no psychotherapeutic aims. They are equivalent to the German term “Erinnerungspflege” (e. g [9]). This is important, because the international and the German terminologies slightly differ and the underlying goals and methods can be different.

Because the planning and implementation of RT sessions is time-consuming and stressful for all stakeholders, the use of potentially disencumbering multimedia information and communication technologies (ICT) accompanying the entire process, which is a much sought after request [20], is investigated by a multidisciplinary research team in the scientific project Interactive Memories (InterMem; see [13]). The field research presented in this article was partially embedded in the InterMem project infrastructure. Additionally, a preliminary ICT study indicating positive effects when using surface computers (SC) for the occupational therapy of PwD was conducted in 2015 [16].

Research questions and objectives

As the research to date (especially) regarding large touchscreen systems in combination with specific multimedia content for ICT in RT is scarce, this article summarizes three similar dementia ICT studies and aims to generate a common synthesis to further the understanding of such systems by addressing a specific subset of questions. Multimedia ICT interventions displayed on either small surface devices (9.7’’ Samsung Galaxy Tab A touchscreen tablet PC), medium surface (32’’ LG 32LS359S non-touchscreen TV) or large surface (40’’ SC Samsung SUR 40 with Microsoft PixelSense and touch technology; see Fig. 1 for the experimental set-up) and the accompanying contents were developed as (potentially) added value for RT and finally evaluated in the studies. Each system aimed at triggering activation and passively or actively occupying the PwD in slightly different ways while trying to find an answer to one of the following main research questions:

  • Study S1: can a multimedia natural user interface (NUI) program running on a SC be used to facilitate enjoyment as an added value, further evoking reminiscence and activating PwD? If so, how do the PwD interact with the different tangible and touch-based input modalities?

  • Study S2: will personalized multimedia content being shown on a SC add value to and be adopted/accepted as a meaningful asset to RT by the care personnel as well as the PwD in a dementia special care home?

  • Study S3: are non-personalized YouTube (YT) movies (in contrast to self-filmed movies) presented on tablet PCs or TVs suitable to evoke reminiscence and activate PwD and if so, which option is more effective regarding the whole process?

Fig. 1
figure 1

Examples of images used in the studies: a SC aquarium program as used in study S1, b personalized SC photo book as used in study S2, c television display and tablet PC used in study S3

Full size image

Categorizing the research question outcomes

In order to strengthen the comparability between studies S1, S2 and S3, the outcomes answering the research questions can be classified into the overarching categories of “reactions to” and “interactions with” the ICT interventions. Thus, (positive) verbal statements or non-verbal gestures and expressions from the PwD are some of the expected reactions pointing towards well-being (including emotions such as joy), reminiscence and interest-activating effects of the respective system as well as the general acceptance towards it. Adding to that, the interactions between the PwD and the system itself, between the PwD and the carer as well as intragroup interactions (where applicable), indicate if the ICT intervention provides a suitable medium to facilitate multimedia RT where the participants can physically engage with and thus actively choose their own reminiscence triggers.

State of the art and related work

Augmenting RT with multimedia ICT

In order to facilitate the planning and implementation process, enrich the conventional RT as well as unburden the care personnel, multimedia RT is thought to add value to the whole process and all parties involved. In this respect, a review [14] has concluded that using multimedia ICT to promote reminiscence and other related activities for PwD can have beneficial effects.

Next to commercial projects, such as the media dementia tablet (mediadementia.de), several academic projects besides InterMem have addressed diverse aspects of multimedia ICT RT in recent years, most notably the larger scale computer interactive reminiscence and conversation aid (CIRCA) project [2, 4].

In ICT approaches to RT, e.g. as reviewed by Joddrell and Astell [11], the visual multimedia content varies from dynamic video types to static types, such as photographs, text or non-moving graphics. The display of the visual contents usually takes place on small (e. g. smartphone, tablet PC) to large-screened monitors. Furthermore, the auditory channel is addressed with accompanying audio output, such as background sounds or music, commonly known songs or voices. Some systems go beyond mere audio-video-touchscreen interaction and address other sensory channels, e. g. augmented physical objects that additionally stimulate the tactile sense [10].

To control the multimedia output, touchscreen user interfaces are used as input devices in many cases [11]. Going a step further from, but potentially still including touch interfaces, natural user interaction (e. g. through gestures or tangibles) should also be feasible for PwD, as it can evoke familiarity through using native motions [18].

Technological aspects as applied in the studies

The three studies presented in this article offer a mixture of the aforementioned output and input aspects addressing multiple sensory and motor channels. Study S1 focuses on audio-video output as well as touch and object input and thus on the interaction with a multimodal NUI. To avoid frustrating effects caused by the combination of a small screen and the sensory limitations of elderly PwD, an animated “mixed reality” virtual 3D aquarium (including background sounds) was built for the PixelSense SC [8]. The recreational program can be playfully used by a group of PwD via its 50 touch-point screen as well as through theme-appropriate augmented objects (i. e. fish food can, boats). Furthermore, a sensor kit was used to add object interaction via a “real” augmented fish food can (to feed the fish) and hand clapping (to scare the fish). The system was designed according to the responding, enabling, augmenting, failure-free (REAFF) design principles for PwD [3].

As a follow-up to the work by Murko and Kunze [16], study S2 also used audio-video output and touch input on the PixelSense SC, albeit in a less interactive way than study S1: a virtual personalized 2D photo book (including appropriate sounds) was displayed to trigger reminiscence in single RT sessions. The touchscreen allowed interaction with the virtual book similar to the interaction with a real photo book (i. e. turning pages).

Study S3 uses audio-video output, but no designated direct input, as it solely focuses on the content: self-filmed movies are tested against standard YT movies (filmed by other persons) in sessions with a TV or tablet PC display (which is not intended to be controlled by PwD). It can be seen as an extension to the YT study of O’Rourke et al. [17].

Methods

Different studies sharing a common aim

The three studies share a common aim in the context of accompanying research for InterMem and RT: designing and evaluating effective user-centered multimedia ICT that can be put into practice in the near future to maintain the QoL of PwD. With this in mind, they all implement diverse forms of RT supported by different audio-visual multimedia content which can be manipulated by touchscreen input (in most cases). In consequence, each study deals with slightly different research questions, experimental designs and technological aspects.

While the study S1 focuses primarily on the interaction of PwD with the NUI technology and creates a comprehensive technological 3D graphics application that also allows the use of augmented physical objects as input, the other studies S2 and S3 put the main focus on the content and the modality (i. e. photo books or videos) of the presentation.

Common external conditions and research methods

Each research work was conducted in form of short-term qualitative field studies with similar external conditions. Among other things, ethical implications regarding evaluative work with PwD and the tight working schedule of dementia care institution personnel made it difficult to recruit and empirically test larger samples of PwD. Conversely, the care institutions provided an authentic setting for the multimedia ICT RT sessions. Following extensive literature reviews and expert interviews, direct as well as participant observations were conducted, transcribed, analyzed and interpreted in order to assess the specific needs of PwD regarding RT. These were in turn fed into the ICT prototype creation. The prototypes were finally evaluated using similar observation forms created by the three authors.

Details of study S1

As a first step, expert interviews with three professional carers were conducted. In parallel, eight design ideas were created and presented as scenario-accompanying mock-ups in order to get a validated final design, the aquarium. The analyzed NUI principles included input interaction modalities (e. g. touch, tangibles), 2D/3D perspectives as well as usability and user experience aspects. Subsequently, the prototype program was implemented and finally evaluated. With that, the 16 PwD test subjects (15 females and 1 male aged between 76–95 years), among them 6 with mild, 6 with medium and 4 with severe dementia, were observed for 5 min each (on account of their attention span) encompassing a test period of 2 days. The first day consisted of 11 single sessions whereas the second day consisted of 4 group sessions. A self-provided observation form (filled out by the author while the caretaker instructed and prompted the PwD to use the technology) was used to measure the reactions and the interaction behavior.

Details of study S2

After conducting five expert interviews, the behavior of five female PwD being presented a personalized SC photo book based on their own biographies, as well as the interaction with it, was evaluated via a direct observation in 29 RT sessions (each 7–45 min) spanning 2 weeks. The members of the test group were aged from 80–94 years. One participant had mild dementia whereas all the other participants were in mederate dementia stages. The photo books were put together by the author and based on biographical sheets and observations of the PwD daily routine.

The outcomes, regarding personal traits after Kitwood’s person-centered care approach [12], and verbal as well as non-verbal behavior were measured with a self-provided observation form, which was then evaluated according to Mayring’s qualitative data analysis method [15] and interpreted corresponding to a hermeneutical approach.

Details of study S3

In study S3, similar topics for both video types were chosen based on the analysis of the PwD biographies. The self-created videos were filmed in the Black Forest region by the study author herself and afterwards cut using her PC. Before conducting the test RT sessions, the short films were validated in an expert workshop and once more edited by the study author. In light of the subject matter, impaired vision and hearing capabilities were used as exclusion criteria when choosing the PwD sample. In a 3-week test phase, 12 participants (6 male and 6 female) consisting of 4 mildly, 4 medium and 4 severely impaired persons aged 75–97 years (only 9 of them finished the complete test) were evaluated in 50 sessions of 15 min each.

Outcomes regarding the form on the day, the environmental quality, and verbal as well as non-verbal statements were observed via a participant observation and with that a self-provided observation form. A summary regarding the detailed parameters of each study can be seen in Table 1.

Table 1 Main parameters of studies S1, S2 and S3
Full size table

Results

Common results

The results regarding the reactions to, interactions with and effects of the ICT RT interventions on reminiscence and emotion could be mainly interpreted as positive but, given their qualitative nature with a relatively low number of test subjects and non-validated survey instruments, they are not representative.

Reactions to the ICT systems

The first reactions to the aquarium intervention consisted of verbal comments in roughly half of the cases, e. g. “How nice, an aquarium”, “This is witchcraft!” or “These are fish, remind me of home”. Conversely, roughly one third of the test subjects did not show an obvious first reaction as they stared at the screen without any gestural or mimic expression. In contrast, the rest of the group still showed a gestural or mimic expression, e. g. by smiling or joyfully looking at the display.

The PixelSense SC used in study S2 also seemed to be well accepted by the PwD and their carers. The PwD assumed their seats and showed no reticence towards the system, resting their arms and hands on the table and supporting themselves on the hardware. Furthermore, all test persons touch-interacted with the photo books in the application.

Reactions to the movies in study S3 could be noted as positive interactions between the test person and the carer, which mainly consisted of reminiscing past situations. A part of the PwD joyfully smiled and started humming to background music or even commenced impromptu musical interludes.

Interactions with and effects of the ICT systems

Study S1 showed that the touch-based input modality seemed to be a well-working form of interaction leading to moments of enjoyment, as the test persons were able to interact with the virtual fish without significant difficulties (although most of them still needed some kind of prompting by the carer). While using the augmented fish food can, roughly half of the PwD were activated verbally as well as non-verbally, e. g. by smiling, while fully being able to use and turn the fish food can so that the fish moved into the middle of the screen. On the other hand, some of the PwD had problems tilting the fish food can due to motor issues or were simply disinterested. As the correct movement of the fish food could also generate an accompanying sound and the whole virtual process closely resembled the real world process, some of the PwD were seemingly immersed and asked why no food came out of the can itself.

In the course of the S2 study main test phase, the PwD narrated more and more stories pertaining to the photo book content and thus apparently delved deeper into their memories. With time, an initially confused test person seemed to engage in a more and more orderly way of telling stories. The participants became more verbally active in their everyday lives and also seemed to remember parts of the last RT session and the photo book contents. In accordance to their presumed need of permanence and orientation, engaging in reminiscence using consistent interactive photo books was perceived positively by the participating PwD and their carers.

The results of study S3 show that biography-based RT films can be used with PwD at all stages of dementia. No signs of excessive demands or states of agitation after showing the film could be observed within the test population; however, it became evident that not all PwD were activated by the proposed movies, as some seemed to show no interest at all. Furthermore, the assessment of some reactions (e. g. unintelligible utterings) was difficult because of the inability of those PwD to clearly express their thoughts. The higher proportion of the test persons seemed to show positive reactions as the PwD started to narrate, sing, repeat single words, smile and/or look interested. Beyond that, the form on the day of the PwD and the environmental factors (e. g. the “feel good factor” and/or the ambient sound level) seemed to play a leading role relating to the awareness when watching the films. In any case, both the short films from YouTube and the self-created movies led to positive resonance.

Comparison of group vs. single sessions

The comparison between the group and single session types in study S1 showed a higher level of activation in the group setting: the social interactions and group dynamics between the PwD led to a livelier atmosphere: “If she can do it, I can do it too”. The PwD who did not interact with the S1 system in single sessions were triggered by the rest of the group and thus were encouraged to engage in interactions themselves.

Discussion

General synthesis and shortcomings

As the three studies focus on (slightly) different aspects and systems of delivering personalized as well as non-personalized multimedia ICT and its content in the context of RT, they cannot be fully compared and subsequently this article can only give a superficial and descriptive insight into the subject matter.

In all three studies, the prototype systems seemed to activate at least some of the PwD cognitively, sensorimotor-related and/or verbally. The lessons learned encompass proposals that can be used in order to create activating user interfaces as well as application and content types for multimedia RT systems; hence, a synthesis towards a guideline for value-adding, potentially effective RT multimedia interaction systems and content can be created.

Lessons learned regarding the interaction interface

The lessons learned from the S1 evaluation indicate that SC can be used to activate PwD when using value-adding as well as familiar, recreational multimedia content. In this particular example, a virtual aquarium that lets the PwD play with the fish includes well-known elements, but is not likely to be found in the real world with such a high level of interactivity.

As observed, NUI interaction works with PwD, but some aspects, e. g. the haptic object interaction, which is adding value in most situations, may be unfitting when not adjusted to the needs and dementia levels of the individual PwD. People with severe dementia grades and thus impaired sensorimotor abilities could not correctly position the objects to activate the underlying “usage metaphors” in the first place. In contrast, people in mild/early dementia stages could still effectively interact with the same objects. As PwD in late stages have more difficulties with the interaction, friendly (and competent) assistance from the care person is all the more important.

In essence, the touch input modality seems well-suited for most cases. Depending on the design of the multimedia ICT RT system, a combination of different primary (e. g. touch or object interaction) and secondary (e. g. different gestures, ideally already known by the PwD) input modalities should be implemented to avoid unusable interfaces for some individual PwD. In any case, the lag between the triggering of the interaction and the outcome of the interaction itself should be minimized so that the PwD can directly connect the system’s output to their input.

Lessons learned regarding the content

Study S2 showed that personal photo books were used to different extents and could elicit reminiscence in the form of narration. The added value of the multimedia photo book consists of a combination of audio-visual triggering via a matching content combination of pictures, text and audio. In line with the findings of study S1, the system was accepted and used as best as possible by the PwD.

In order to bolster the acceptance, the multimedia content intervention should be created with respect to conventional RT processes taking place in the care institution and be adjusted to the needs of elderly PwD. Hence, the multimedia RT intervention is not to exceed a certain amount of time and the PwD should be put in a familiar environment as well as kept comfortable throughout the whole session. The digital photo books should be “easy on the eyes and ears” of the PwD, e. g. the photos and the background should have a high contrast ratio as well as be large enough in size (the same is the case in the S1 study aquarium). Also, the accompanying background audio should be loud enough to be clearly perceived by the PwD.

If short films are used as content type at the carer’s option, a direct interaction of the PwD with the movie content is likely not intended. As a matter of fact, the interaction modalities have no prominent role in that case. Conversely, the type and content of the movie itself is more important: study S3 showed that both personalized and non-personalized short films (shown on both small and large screens) are equally suited to elicit verbal reminiscence, that is if the theme of the films matches the interests of the PwD. Even the aquarium can be useful in the context of RT, albeit not being personalized. The fish themselves as well as the setting can function as memory triggers: it became apparent that some of the PwD possessed real aquaria in their past lives and perceived the virtual fish as their old pets while telling stories about them.

Designing effective ICT RT systems

Summarizing the aforementioned findings, it can be assumed that ICT systems providing reminiscent moments may effectively add value if the daily form of the PwD as well as their environments are acceptable, if the system’s interaction as well as audio-visual output modalities take the individual PwD sensorimotor skills into account and, most importantly, if the content and/or game nature of the multimedia intervention is interesting and thus “hits the spot” of the PwD. In addition, group sessions seem to foster further interactions between the (previously inactive) PwD.

Outlook

The next step has to include the formulation of a guideline and based on that a refined holistic, value-adding multimedia RT system adhering to the aforementioned lessons learned. As the examined studies are of a qualitative explorative nature, more quantitative research with a sufficiently high number of test persons (and potentially on a longer term) is crucial to validate the lessons learned, hypotheses and interpretations. In any case, the findings seem promising and may pave the way to memory-activating and at the same time fun multimedia ICT RTs of the future.