Keywords

1 Introduction

This chapter proposes the use of a multisensory system into a smart home environment; this multisensory system allows the connectivity of all the sensors that could be installed in a smart home in order to know more about the requirements of elderly people, specifically for avoiding social isolation and incrementing physical activity to improve their healthy conditions. Those problems are considered extremely important thus must be solved since elderly population is vulnerable. In addition, this multi-sensor structure is focused on the decision fusion stage that is based on the information that is coming from all the sensors and evaluated for an adaptive neural network fuzzy inference (ANFIS). This proposal also describes how the elderly people can be engaged to improve their quality of life by a gamification strategy and a fuzzy logic system, which are running on a human machine interface (HMI). Figure 1 describes the general proposal for data fusion on HMI at smart homes.

Fig. 1
figure 1

General proposal for data fusion on HMI at smart homes

Full size image

1.1 Elderly People

By 2018, for the first time, the population of people over 65 years old outnumbered children under 5 years old. Additionally, the United Nations expects that by 2050, there will be 1.5 billion people aged 65 and will outnumber the adolescents and youth population from 15 to 24 years old (1.3 billion). Thus, the proportion of elderly people in the world is projected to reach from 9% to 16% in 2050 [1].

Figure 2 shows a graphic of the estimated and projected global population by broad age group from 1950 to 2100, where population over 65 will increase while population under 24 will decrease.

Fig. 2
figure 2

Estimated and projected global population by broad age group from 1950 to 2100 [1]

Full size image

Elderly people can be seen as individuals with a wealth of life experience, with interests and aspirations in their later life, as limitations and losses [2]. Aging involves biological changes, such as cognitive deterioration, physical strength diminished or languishing sensory perception and detection; and social changes, like social isolation and loss of loved ones, or as a sense of loss of value, purpose or confidence [3, 4]. Furthermore, one of the challenging changes in aging is the loss of autonomy in daily life, this causes a modification of the living environment. Thus, social inclusion and social exclusion allow access to social engagement and participation through social relationships, civic activities, local services and financial resources [5]. Moreover, social relationships are related to better health and well-being in late life. Elderly people use technology to shape their social contexts; however, its use depends on the positive or negative evaluation of a device. Table 1 shows the theoretical and practical implications of the conceptual model of technology use and social context in late life [6].

Table 1 Theoretical and practical implications of the conceptual model of technology use and social context in later life from [6]
Full size table

In that regard, since 1988, gerontechnology emerged as a task to solve problems and challenges found by aging people. Ambient Assisted Living (AAL) technologies support elderly people to maintain and continue their daily life more independently [7]. Moreover, the arising of technology allows connecting devices and systems to exchange communication with individuals and collect the data derived from that interaction [8].

There is an increase of life expectancy in the world, due to technological and medical improvements; it is relevant to develop and implement new strategies and technologies for the elderly that improves health care, independence, social inclusion, among others. In that regard, smart homes may allow elderly people to stay comfortable by monitoring their health and social inclusion with unobtrusive and non-invasive remote devices [7].

1.2 Smart Homes

In 1984 the American Association of House Builders introduced the concept of smart homes in terms of “wired homes” [9]. However, the term is often defined based on technological aspects and usage. For instance, the construction sector defined a smart home as a residence equipped with computing and information technology, which anticipates and responds to the needs of the occupants, working to promote their comfort, convenience, security and entertainment through the management of technology within the home and connections to the world beyond [9], or as a living environment that has the technology to allow devices and systems to be controlled automatically [10]. The health care sector define it as a residence equipped with technology that facilitates monitoring of residents’ health status and/or that promotes independence, prevents emergencies, and increases their quality of life or as an assisted interactive dwelling house [10]. This type of home collects and analyzes data, gives information to the habitants and manage different domestic appliances [11]. Figure 3 shows some common household appliances used in a smart home as Smart TV, electric stove, coffee maker, interior lighting, washing machine, refrigeration, and connected thermostats. Furthermore, smart household appliances could be accepted if [10]:

Fig. 3
figure 3

Household appliances

Full size image

  • Elderly know that those type of appliances exist.

  • The appliances and devices can be quickly and cheaply obtained.

  • Those products demonstrate they reduce or eliminate physical demands for their operation.

  • The usability of the product considers characteristics as maneuverability.

Since the adoption of technology, the development of the smart home for the elderly has been researched. Figure 4 shows the timeline of the smart house development for the elderly. In 1998, the AID project (Assisted Interactive Dwelling) proposed to use homes that consider equally the non-disabled, elderly or disabled tenants through the concept of barrier-free design by taking advantage of the smart technology at that time. They used a demo that considered a motor-driven window, door and curtain control, enhanced heating controls, ‘keyless’ door locks, a video-entry system, an enhanced security system, and infra-red bathroom controls. The idea was to raise awareness amongst caring agency, housing providers, architects, product manufacturers and end-users to promote the use of technology in homes [12]. In 2002, the Aware Home project proposed to use a Digital Family Portrait to support awareness of the long-term health, activity, and social well-being of senior adults living by themselves. This portrait tracked the daily activity of the users and showed the last six actions performed by them; it also included the weather, indoor and outdoor temperature and the number of room-to-room transitions in 15-min increments [13]. The next year, in 2003 the LARES project [14] consisted of an intelligent human-friendly residential system that implemented an intelligent bed robot for the elderly and the handicapped where an arm was attached to the bed for transporting objects; the human-oriented interface informed the users intention to the bed robot; and the home network was equipped to transmit and share information between each device. In 2009, the VAALID project (Accessibility and Usability Framework for AAL Interaction Design Process) began to develop new tools and methods to make easier and more dynamic the creation, design, construction, implementation and evaluation processes for technological solutions within ALL to ensure the accessibility and usability of the environment for the elderly [15]. In 2012 the U-Care project applied the Korean government method where they use the activity data alone to effectively analyze the status of the solitary elderly people; the home was equipped with a gas leak detector, gateway, absence button, smoke detector and activity sensor to analyze the user’s activity level, which are monitored with five-second intervals, and the activity data calculated every hour [16]. Since 2015 the ORCATECH (Oregon Center for Aging and Technology) project of aging developed an advanced platform for assessing the health of people living within their homes using passive sensing technologies to enable accurate assessment of cognitive and physical health [17, 18]. In 2016 the SPHERE project (Sensor Platform for Healthcare in a Residential Environment) aimed to develop a smart home platform of non-medical network sensors, capable of gathering and integrating multiple types of data about the home environment and the behaviors of its residents to understand a range of healthcare needs [19]. In 2017 the SMARTA project consisted of developing and testing a personal health system that integrated standard sensors as well as innovative wearables and environmental sensors to allow home telemonitoring of vital parameters and detection of anomalies in daily activities, thus supporting active aging through remote healthcare [20]. In 2018, the European Commission funded the ALADIN project (Smart Home-Care solutions for the Elderly) to create smart furniture solutions for the elderly and the nursing homes that take care of the elderly; ALADIN is a homecare assistant which increases the independence of elderly people living alone [21, 22]. Recently, the HABITAT project (Home Assistance Based on the Internet of Things for the Autonomy of Everybody) developed an IoT-based platform for assistive and reconfigurable spaces that integrates RFID, wearable electronics, wireless sensor networks, and artificial intelligence. This project has the purpose of assisting needy people in their homes in safe conditions, helping them to conduct autonomously most of the activities tied to the satisfaction for their primary needs, sustaining actions focused on hospitalization and home care [7]. Finally, this year, the Tecnologico de Monterrey and UC Berkeley are developing the Gamified HMI project. This first stage takes advantages of the camera and Alexa to train an ANFIS model to propose a tailored gamified HMI that teaches, engages and motivates elderly people to keep in touch with their peers, caregivers, doctors, and family members to promote social inclusion and happiness and avoid social isolation and depression; the following steps will consider the smart household appliances to use them as social products that will interact with the end-user and the devices and between devices [23].

Fig. 4
figure 4

Timeline of Smart Home projects for the elderly people

Full size image

2 Multi Sensors: Data Fusion

When multiple sensors are collecting information, which later is combined, they can make accurately inferences than could not be achieving by a single sensor; moreover, if these sensors are placed under a reference framework that is able to map the value of the property or attribute to a quantitative measurement in a consistent and predictable manner, this could be extremely attractive as a framework because it also includes functions which can be described in terms of compensation, information processing, communication and integration. This framework is called multisensor data fusion [24,25,26]. The multisensor data fusion concept is based on fundamental tasks done by animals and humans since they use multiple sensors to improve their ability to survive when they have to identify potential threats and the have to perform some functions; for instance, the five human senses used in daily tasks are hear, touch, sight, smell and taste.

The primarily functions are:

  • Compensation when sensors respond to environmental changes by self-diagnostic tests, self-calibration and adaption, a compensation process is taking place.

  • Information processing is the stage linked to signal conditioning, data reduction, event detection and decision-making.

  • Communications this stage is based on implementing a standardized inference protocol for connecting the sensor and the outside world.

  • Integration is generated when the sensing and computation processes are linked under the same silicon chip or system.

Besides, a decision making stage is a primary stage in the multisensory system because it could be considered as the central stage in which the information from sensors are processed in order to create a decision according with the sensed data.

Thus, the concept of data fusion is not new, but the evolution of innovative sensors, digital systems, machine-learning algorithms, advanced processing techniques and real time processing devices allow creating a high-performance data fusion system. Figure 5 represents that concept; data from Alexa and camera are collected to put them on the ANFIS system to analyze the data and represent the information in one form. For instance, the output could be the required features to propose a tailored interface.

Fig. 5
figure 5

Data fusion legend

Full size image

2.1 Multisensor Configuration

The configuration of sensor is focused on two main detections that are presented below.

  1. 1.

    Detection of facial expression and body posture for detecting the mood of an elderly person as well as physical limitations.

  2. 2.

    Detection of nonconventional behavior through a voice survey; for getting this information an Alexa system was used in which a survey proposed by Yesavage [27] was deployed in order to rate depression in the elderly. This scale is known as Geriatric Depression Scale (GDS); from 0 to 9 is a not depressed user, from 10 to 19 a mild depressive individual and from 20 to 30 a severe depressive person. This questionnaire is usually completed in 10 min or less and was initially validated with depressed patients and with elderly people without any history of mental disorder [4].

2.2 Multisensor: Decision Fusion

The representation of knowledge could be achieved by an inference system, which could be normally generated through an inference system [28]. When it was proposed to use in this multisensory system into the decision making a fuzzy logic system, it was described as an extension from binary values (0 and 1). However, an ANFIS could be trained and generate automatically the linguistic rules so a knowledge base is crated [24].

2.3 ANFIS: Adaptive Neuro-Fuzzy Inference Systems

Sometimes, conventional mathematical modeling algorithms do not deal with vague or uncertain information. Thus, Fuzzy systems using linguistic rules (IF-THEN) have the strength and ability to reason as humans, without employing precise and complete information. However, a problem arises, how to transfer human knowledge to a fuzzy system. Several proposals have been made, such as the combination of artificial neural networks with fuzzy systems. Artificial neural networks have the ability to learn and adapt from experience, thus complementing fuzzy systems. Among the most important techniques is the ANFIS, an adaptive neuro-fuzzy inference system proposed by Jang [24] in 1993, which generates fuzzy IF-THEN rule bases and fuzzy membership functions automatically. ANFIS is based on adaptive networks which is a super set of feedforward artificial neural networks with supervised learning capabilities as stated by Jang in [24, 29]. It is a topology of nodes directionally connected, almost all the nodes depend on parameters that are changed according certain learning rules that will minimize an error criteria. The most used learning rule is the gradient descent method; however, Jang proposed a hybrid learning rule that incorporates least square estimation.

2.4 Multisensor Topology Proposed: Detection of Nonconventional Behavior

The multisensory system is based on two elements; the first one is an Alexa voice smart home controller [30] and a face detection for human emotions.

Alexa applies a survey to the end-user in order to know more about his mood, a classification between 0 and 1 according with the responses is done using the survey presented in Table 2; this survey was presented in [4]. The basic internal structure of Alexa is shown in Fig. 6.

Table 2 Survey for elderly people in order to detect social isolation problems (depression)
Full size table
Fig. 6
figure 6

Alexa internal structure

Full size image

3 Emotions Classification: Detection of Facial Expression

An online detection of emotions based on facial expressions is achieve using a webcam and a PC running Python with the OpenCV library. The PC has an Intel Core i7-7500U dual core processor with 8 GB of RAM and an integrated camera HP wide vision HD of 0.92 MP and resolution of 1280 × 720 px.

The process for the emotion detection consists mainly in the use of two separate artificial neural networks. The first one is a deep neural network that extracts the faces from the image. For the second part a convolutional neural network is used to classify the extracted face within seven different levels of emotions: angry, disgust, fear, happy, sad, surprised, and neutral.

The overall process is listed next:

  1. 1.

    Get one frame from the webcam and resize to 300 × 300 pixels

  2. 2.

    Extract the biggest face detected using a DNN

  3. 3.

    Resize the face to 48 × 48 pixels in grayscale

  4. 4.

    Use a CNN to detect the emotions

  5. 5.

    Plot the level of each emotion detected.

3.1 Face Detection

The idea for the deep networks is that they can extract low, middle and high-level features in a multi-layer architecture, hence the common trend to stack layers going “deeper” in the net. But deep networks are hard to train because when they start to converge a degradation problem shows, as the gradient is back-propagated it gets smaller because of the repetitive multiplications. So, as the network goes deeper it gets saturated. As a solution to this, the residual networks were introduced in which an identity shortcut connection is used to skip one or more layers to perform an identity mapping and their outputs are added to the outputs of the stacked layers as shown in Fig. 7.

Fig. 7
figure 7

Residual block

Full size image

For the face detection part, a model included in the OpenCV library was used based on a single-shot-multibox detector and a ResNet-10 architecture as backbone. This model was already trained with images available on the model zoo of the Caffe framework [31]. The results of this network are shown in the Fig. 8.

Fig. 8
figure 8

Detection and extraction of detected face

Full size image

3.2 Emotion Detection

Convolutional neural networks (CNN) is a deep learning algorithm which from an input image, it assigns importance to various features by decomposing the image and compressing it into simple features hence it can differentiate one from another [32]. The CNN architecture use layers that convolve the inputs with filters and compresses them, the objective of the convolution operation is to extract the low-level features from the input. Then a pooling layer responsible for reducing the spatial size of the convolved features to decrease the computational power required to process the data. Lastly a fully connected (FC) layer with a softmax classification technique for learning non-linear combinations of the extracted features. This architecture is shown in the Fig. 9.

Fig. 9
figure 9

Convolutional neural networks (CNN) architecture

Full size image

For the emotion detection part of this work a VGG-like network is used. A VGG net is a deep convolutional network developed by Oxford’s Visual Geometry Group [33] which is publicly available online. The specific architecture used is shown in Table 3.

Table 3 VGG-like network for emotion classification
Full size table

The training of the network was made with two merged public databases. The first one is the FER2013 dataset from the Kaggle competition [34], it contains seven facial expressions in 35,887 images. The second is the KDEF database [35] (Karolinska Directed Emotional Faces) which contains 4900 pictures on facial emotional expressions for 70 different models. The KDEF database images had to be adjusted to have the same format of the FER2013 database, that meant to extract the face in B&W and resize it to 48 × 48 pixels. Figure 10 shows the training graph through 1134 epochs.

Fig. 10
figure 10

Training graph for VGG-like network

Full size image

The results obtained with this network are shown in the Fig. 11.

Fig. 11
figure 11

Emotion classification results

Full size image

The proposed configuration effectively gives a grade for each of the facial expressions detected and works with no noticeable delay on the live camera recording. Therefore, the grade of happiness or sadness detected can be used to train the ANFIS system. Therefore, this configuration helps as a social connector that can promote social interaction between their relatives and friends; besides, this emotion classification may detect daily activities or moods to address any unusual activity and addressing those rare activities in early stages [36].

4 Gamification

Since the 1980s decade, there are references of ‘gamifying’ applications; however, since the 2000s decade, several definitions of Gamification terms have been defined. In 2003, Pelling used gamification to create game-like interfaces for electronic devices [37]. In 2008, Terill defined the concept as taking game mechanics and applying to other web properties to increase engagement [38]. In 2011, Huotari and Hamari defined it as a process of enhancing a service with affordances for gameful experiences in order to support the user’s overall value creation [39]. In 2011, Deterding et al. defined it as the use of game elements and game-design techniques in non-game contexts [40]. In 2015, Chou defined it as the craft of deriving fun and engaging elements found typically in games and thoughtfully applying them to real-world or productive activities [41]. Therefore, the main goal of gamification is to increase the motivation of users by using game-like techniques and applying them effectively in the real world to influence user’s behavior and improve user’s skills, competencies and creativity [42].

Besides, the use of gamification within a device can improve enjoyment, health care and promote social interaction in elderly people [43]. Ponce et al. [23] proposed including social factors in the design process by implementing a gamification strategy to send stimuli to change consumer behavior. Mendez et al. [44] developed a three-step framework to propose a tailored HMI using a fuzzy logic system in connected thermostats to teach, engage and motivate users to save energy. Moreover, that framework was adapted to take advantage of that connected thermostat platform to promote social interaction and physical activity for the elderly people [45].

The Internet maintains and enhances social relationships through email, instant messaging, social networks, discussion forums and blogs have positive adoptions among elderly people. Besides, technology can change daily life in three ways [8]:

  1. 1.

    It should shape social contexts regarding age-specific needs.

  2. 2.

    It should improve the contact’s quality and create meaningful relations.

  3. 3.

    It should compensate for losses and burdens that individuals may face throughout adulthood.

4.1 Kaleidoscope Framework

Kappen proposed the kaleidoscope framework that has intrinsic and extrinsic motivation to promote physical activity in the elderly people over 50 years old [46].

  • Intrinsic Motivation

    • Autonomy: Customization, purpose, independence. Related to the improvement of fitness performance, comfortable routines, incremental progression, rein-forcing success, internalizing rewards and responsibility. Freedom of modulating fitness routines, structured routines, modularity, choice of changing goals, reinforcing profession through visual and verbal feedback.

    • Competence: Engagement based, achievement-based, performance-based. It is related to the complexity of activity routines, challenges with the repetitions of the activity, focus on remembering activity steps, and ease of understanding.

    • Relatedness: Relationships, sharing, preferences. This element is related to fostering social connections. Sharing achievements and experiences, setting an example for peers within the fitness activity domain, exchanging feedback with peers and trainers, and being validated for performance by the trainer and doctors.

  • Extrinsic Motivation: Encompass factors of external regulation, identification, and integration. Moreover, these motivations are not as valued by older adults, and tangible rewards are mostly related to food.

    • Rewards

    • Incentives

    • Leaderboards

    • Points

    • Badges.

4.2 Octalysis Framework

Chou [41] proposed a complete framework based on extrinsic, intrinsic, positive, and negative motivation. Figure 12 displays the framework proposed by him.

Fig. 12
figure 12

Octalysis framework proposed by Chou [41]

Full size image

  • Extrinsic motivation: People are motivated because of external recognition or economic rewards.

  • Intrinsic motivation: People are motivated due to inner motivation; the activity itself is rewarding on its own without tangible goals to achieve.

  • Positive motivation: The activity is entertaining because people feel successful, happy, and powerful.

  • Negative motivation: People engages in the activities because the user fears lose something.

  • Core 1: Epic meaning and calling: People believe that what they do is more significant than themselves.

  • Core 2: Development and accomplishment: People believe they are succeeding, progressing, developing skills, achieving mastery, among others.

  • Core 3: Empowerment of creativity and feedback: People realize a creative process by trying several combinations to achieve goals.

  • Core 4: Ownership and possession: This core is known as the desire core. People believe and feel they are in control of something.

  • Core 5: social influence and relatedness: People are motivated due to social elements.

  • Core 6: Scarcity and impatience: People want something because it is challenging to have it.

  • Core 7: Unpredictability and curiosity: People are engaged due to the uncertainty of what is going to happen next. This core is behind the gambling addiction.

  • Core 8: Loss and avoidance: People try to prevent something terrible to happen.

Figure 13 shows a brief timeline of four projects for the elderly that included game elements within the platform.

Fig. 13
figure 13

Timeline of daily activity projects that considered game elements

Full size image

5 Proposal

Figure 14 shows the trained ANFIS system for the detection of facial expression and nonconventional behavior through voice survey to propose a gamification strategy to engages the elderly people to promote social inclusion and avoid depression or sadness in those individuals. The premise is that by taking advantage of household appliances, it is possible to promote social interaction in elderly people and avoid activities that cause sadness in the elderly.

Fig. 14
figure 14

Trained ANFIS using image and voice as input elements and the output the gamified elements required for the HMI

Full size image

Figure 15 shows that the input values of the ANFIS system are the camera and the Alexa device; the input membership function for both devices are ranged from the sad face (from 0 to 0.25), a bit sad (from 0 to 0.5), neutral (from 0.25 to 0.75), a bit happy (from 0.5 to 1) to a happy face (from 0.75 to 1). Besides the sad face uses a Z-shape membership function, the faces from a bit sad to a bit happy uses the triangular membership function and the happy faces uses the S-Shape membership function. The expected output membership functions ranged from extrinsic, negative, positive to intrinsic motivations. Furthermore, the output values are more biased to the negative side of the lower values because it is considered that the elderly people tend to feel isolated [47]; thus our proposal looks for social inclusion and physical activity. Therefore, the tailored gamified HMI will be more focused on showing extrinsic motivation through messages, videos, rewards and tips about physical activity that will depend on how active or inactive, and sad or happy the elderly people are.

Fig. 15
figure 15

Input membership functions of the trained ANFIS

Full size image

Consequently, Fig. 16 shows how the proposal works. Six types of interactions are used to avoid elderly isolation: user-house, house-product, product-product, product-user, product-interface, and user-interface. As the elderly user spends most of the time at home, this user is continuously in touch with the household elements, such as doors, windows, furniture, household appliances, among others (interaction user-house). Then, the interaction between house-product determines which household appliances are mostly used. Moreover, when the elderly consumer interacts with the products, it is profiled the user’s routines and activities, which are also monitored by the HMI (interaction product-interface), while Alexa and the camera take decisions for the user to get more comfortable the life inside the house (interaction product-user); for instance, Alexa can diminish the light intensity, or turn off the TV (interaction product-product). Through, the multi-sensor system, Alexa and the camera analyzes the elderly facial expressions and communicate by voice with the user to monitor any change in the elderly mood. Finally, throughout the ANFIS system (see Table 4), it is shown which gamification elements should be displayed in the tailored interface to promote social inclusion and physical activity in the elderly (interaction user-interface). The adjustment and feedback between the tailored HMI, the household appliances, and the camera and Alexa are required to continue engaging and motivating the user to be in touch with friends and family members, to improve habits, to exercise. Therefore, with this proposal, the elderly quality of life may improve.

Fig. 16
figure 16

Proposal diagram

Full size image
Table 4 25 rules from the fuzzy system of the trained ANFIS
Full size table

Therefore, the tailored gamified HMI must consider mainly the extrinsic motivation to promote social inclusion and promote happiness in elderly people. The gamification elements required in the interface are: points, badges, rewards, leaderboard, progress bar, higher meaning, narrative, elitism, virtual goods, build from scratch, avatar, monitoring through social connectors as video call [41, 46]. If the elderly people are neutral either in voice and image, the gamification elements can also include progress loss, visual storytelling, random rewards, status quo, mentorship that helps the user become happier and social included. If the elderly user is a bit happy, then the interface can include real-time control, instant feedback, voluntary autonomy, and choice perception, social invite, group quest. Finally, if the user is totally happy, then the elements should include visual storytelling, random rewards, status quo, mentorship, real-time control, instant feedback, voluntary autonomy, choice perception, social invite, group quest.

6 Results

The literature review indicates that the most common gamification elements for the elderly to promote social interaction and physical activity are: feedback, social sharing, challenges, leaderboard, rewards, social connector, monitoring, and a profile [36, 41, 42, 45, 46, 48, 49]. Table 5 displays the gamification elements in a mobile device for the elderly. Those elements consider the extrinsic, intrinsic, positive, and negative motivation based on the Octalysis and Kaleidoscope framework. Figure 17 proposes a typical layout for the elderly user without considering any customization. This dashboard is a proposal for a connected thermostat, it shows the most common gamification elements that may appear in an interface whose primary purpose is to promote social interaction and physical activity. The right side of the screen has the purpose that elderly user can track his/her progress, compare with friends, check their weekly and monthly challenges, and see how rewards are available to be achieved. In the right upper side of the dashboard, it is displayed the daily challenge to promote physical activity in the elderly. The Tips button advises the elderly user on how to improve and learn more about physical activity and its benefits as wells as the kind of household appliances are in his/her home. The Be a HERO button works as an interaction button where the elderly user gives or receives advises of his friends. Finally, the video conference screen works as a social connector because the elderly user can call his/her family member or friends. With this video call screen, it is possible to determine through the face detection if the elderly user is sad or happy.

Table 5 Gamification elements to be used for the tailored HMI
Full size table
Fig. 17
figure 17

General dashboard for a connected thermostat layout without any personalization

Full size image

Based on the ANFIS proposal, in Fig. 18, it is displayed a tailored HMI based on an elderly user that is a bit happy. The first image is the interface for the smart home, which initially has connected the thermostat, television and lighting in the house. The second and third image displays an interface for a bit happy user as it includes real-time control, instant feedback, social invite, the points, badges, leaderboard and the monitoring through social connector (Alexa, and video call). Finally, the last image displays the connected thermostat interface, where it also includes positive and extrinsic motivation.

Fig. 18
figure 18

Tailored HMI proposal for a bit happy elderly user

Full size image

Moreover, if the elderly user is sad or depressed, the gamified interface will display elements that promote social inclusion as video calls or social sharing like feedback their peers through challenges. If the elderly user is neutral, the interface will display a sober layout to determine and recognize the user’s interest; for instance, Fig. 19 shows the proposal of layout for this type of user; the left image shows the general display for the smart home, this type of user prefers a more sober layout where it is easy to have a call with friends or with Alexa.

Fig. 19
figure 19

Tailored HMI proposal for a neutral user

Full size image

Thus, the gamified application monitors the elderly mood in order to propose a tailored interface. If the user does not engage, then the interface re-adapts to propose other elements. The Octalysis and Kaleidoscope frameworks help as a guideline to promote the gamification elements based on the motivations (intrinsic, extrinsic, positive, and negative), Table 5 displays those gamification elements. Besides, this proposal is just considering the gamified elements to be displayed on the interfaces; however, the design and distribution of this HMI may be improved by applying the ten heuristics proposed by Nielsen [50].

For the last three decades, it is proposed the inclusion of elderly people in the smart home by tracking their daily activities, detecting falls, remembering to them activities. However, to the best of the author’s knowledge, any of these proposals have considered a tailored interface based on the elderly people mood to promote social inclusion and avoid depression by using devices as Alexa that allows having a conversation with the elderly or through video calls to promote social relatedness. This proposal aims to teach, engage and motivate users to feel included in the society by promoting interaction with Alexa, their peers and family.

7 Discussion

This chapter proposes the inclusion of Alexa and cameras to track the elderly people and check their daily status, their mood in order to improve their quality of life by promoting social inclusion and physical exercise. The multi-sensor system is used within a smart home environment to identify the physical characteristics of elderly people. Thus, the voice and face detection are evaluated on an ANFIS system to propose the personalized gamified elements that run in an HMI needed each type of user.

The emotions detected are ranged from a 5 type-scale: sad (the lowest membership function value), a bit sad, neutral, a bit happy, and happy (the highest membership function value). Those emotions are measured with the survey for elderly people presented in Table 2 [4]; these questions are asked to the elderly people through Alexa. Thus, the initial interaction begins during this questionnaire. During the initial tests that face detection was done using the webcam from the PC, and the facial expression concorded with the results. Therefore, the face detection can be used from the mobile or tablet camera the elderly individual is using.

Then, based on those ranges the ANFIS system, that is biased to the lower values to propose a gamified interface focused more on the extrinsic motivation, as this type of motivation is more accessible to measure through metrics, of how many times the individual is using the application or even the daily physical activities the individual is doing and how much time the user spends exercising [41, 46]. Besides, the application displays the gamification elements from Octalysis and Kaleidoscope framework that engage the elderly people to improve their social skills and physical activity. This initial approach takes in consideration that one family member is continuously in contact with the elderly user, thus if any urgency occurred, the application would contact the family member immediately to check if their elderly familiar is fine.

However, a clear disadvantage of this proposal is that the elderly user and the family members require to accept that the house will be permanently monitored. Moreover, the type of mobile phone or tablet that the elderly user has may be inconvenient, and the socioeconomic level where this type of product can be used. Besides, it is required that the elderly accept Alexa as an initial manner to interact. Another possible failure is the face detection; the face may display another feeling rather than what the program is detecting; however, the face detection algorithm can be updated with an artificial neural network that can train the user’s faces and detect more accurately the user feelings. As it was mentioned on the Results section, the gamified interface can be improved by applying the ten Nielsen heuristics [50]: visibility of system status, match between system and the real world, user control and freedom, consistency and standards, error prevention, recognition rather than recall, flexibility and efficiency of use, aesthetic and minimalist design, help users recognize, diagnose, and recover from errors, and help and documentation.

This proposal intends to change the way of how the products are used, and to take advantage of the household appliances to make them more social, it means, to produce a social interaction between the user and the product. A product that can take into account the needs of the elderly to promote in and the unconscious way social interaction and therefore, improve their physical activity. With the multisensor system, it is possible to take advantage of sensors, or household appliances that use sensors to analyze the elderly pattern and propose, for instance, a customized application that best fit to the user.

8 Conclusion

In this chapter, a Multi-sensor system for helping elderly people by using gamification and artificial intelligence within a tailored HMI have been proposed. Elderly people will continue increasing, whereas younger people will decrement, since last year, the elderly population became more than the younger population. Therefore, the elderly people require to be considered, and as some of their problems have led to social isolation, it is needed a strategy that improves their quality of life. This strategy considers the interaction of the elderly people and the products; with the interaction between household appliances and Alexa, and gamification features within an application it is possible to engage and motivate the user to be socially included and improve their quality of life, too. Nowadays, it is not enough to have connected products that are not able to profile the elderly user and take actions based on the user tasks that benefit them without making intrusive decisions. With this proposal, it is intended to profile and know better the user in order to propose an accurate application that improves their social and physical skills without affecting their personality or without affecting their freedom. This HMI set the opportunity of creating an atmosphere where the connected products can interact with the elderly and propose social and physical activities that will help the user to feel included.