1 Introduction

According to recent projections from the United Nations [45], the percentage of people aged 65 and older in the world was expected to grow from 9.1 in 2019 to 15.9% in 2050 worldwide.

According to the same projections by the United Nations (UN) [45], the region of Latin America and the Caribbean is projected to have a percentage of 19% of its population ageing 65 and over by 2050, growing from a percentage of 8.7% in 2019. These projections show that Latin America and the Caribbean have followed the worldwide population ageing trend.

The UN population projections [45] estimated that, in 2020, Brazil was the largest country in terms of population in the region of Latin America and the Caribbean, accounting for 32.5% (approximately a third) of the region’s population.

In terms of population ageing in Brazil, a study from the Brazilian Institute of Geography and Statistics (IBGE) [18] in 2018 estimated that 21.9% of Brazilian population will be aged 65 and over in 2050. Those results from IBGE are in line with the UN population projections for Latin America and the Caribbean, showing that Brazil is following the regional population ageing trend.

Population growth and changes in lifestyles and health conditions have contributed to the worldwide ageing process. With such changes, the design of new technologies must be thought out appropriately for elderly users.

Difficulties encountered by older people in using technologies (such as mobile phones and computers) are usually related to their limited previous experience with these features. Besides, there are difficulties encountered due to age-related conditions, which may vary with age or even change during the same day due to external factors that may extenuate or soften them. These difficulties involve more than one area, and their factors are related to cognitive, perceptual, and physical losses associated with ageing [16, 25]. Analyzing the accessibility for older and elderly people is different from analyzing accessibility for a specific type of disability [28, 38], as there is not a single set of characteristics that are common to all older people.

Furthermore, to ageing characteristics, some difficulties can compromise the use of interactive technologies. Examples of such difficulties include the lack of technology experience; lack of knowledge to perform daily activities using sites and applications; and lack of awareness of all the benefits one can obtain from using technological resources, and barriers to access caused by poorly designed technology [27]. For these reasons, it is essential to know the needs and preferences of target users, so that designers consider their particularities and cultural differences that influence the use of current technologies.

Being concerned about affordable design is very important for people to have access to all technological resources, without exclusion due to financial conditions. Technology has the potential to offer a better quality of life for everyone, especially for people with disabilities and older people. In particular, recent developments have brought notable possibilities for improving daily life aspects, such as sensors and devices can communicate with each other thanks to the Internet of Things (IoT) paradigm. With the advancement of these technologies, the smart homes research field has evolved rapidly and allows its users greater freedom and independence. A wide range of applications employs intelligent home devices, such as activity recognition, environmental monitoring, health assessment and care, home automation, and energy efficiency [12, 28, 38].

In the context of smart homes, an electronic device control system developed with a focus on elderly people may be able to assist them in their daily activities, increasing their independence. Thus, it is essential to conduct studies on how to implement accessibility features in computational applications such as Web and mobile applications—in particular, investigating the needs of specific user groups in developing countries such as many in Latin America, and particularly as in Brazil is crucial. This kind of investigation can significantly help to design the interaction of such systems focused on the particularities of those users in the region.

In a previous study [43], the authors of the present paper performed a preliminary study analyzing the main points raised in two focus groups with ten elderly people in Brazil about the needs related to intelligent home control systems. The focus groups sought to cover elderly people’s perceptions regarding technology and especially the wishes and expectations of a home control system. That preliminary study focused primarily on needs-finding, without examining the interaction with any system. During the sessions, the researchers exposed participants only to a video demo of a mobile smart home system.

Despite having contributed to eliciting valuable knowledge about the needs of elderly users in a countryside town in Brazil regarding interaction with smart home systems, the preliminary study [43] was still limited. That study did not aim to reveal implications for the design of interactive mobile smart home systems focused on elderly people from their actual interaction with prototype systems.

The present study consisted on an expanded piece of research, encompassing the adaptation of a prototype of a mobile interactive smart home system according to the findings from the preliminary focus group study [43], and analysis of user evaluation with a different group of elderly users in Brazil.

The main contribution of the present study concerns the application of the knowledge acquired from focus groups from our previous study [43] and relating them to the results from observations of usability evaluations of a prototype mobile smart home system with elderly users in Brazil. From this analysis, we aimed to provide recommendations for the creation of interfaces and functionalities for home control systems for the elderly.

The remainder of this paper is organized as follows. Section 2 presents the main concepts related to accessibility, interaction in the context of smart homes and elderly people, and related work. Section 3 describes the study design, participants, procedures for data collection, and analysis. Section 4 describes the results obtained, followed by discussions in Section 5. Finally, Section 6 presents conclusions and future work.

2 Background

In this section, we describe fundamental concepts on accessibility for the use of technologies by older people, the context of smart homes as a way to provide a better quality of life for older people and related work.

2.1 Accessibility and elderly users

Over the years, people tend to have a significant decline in cognitive, perceptual, and motor skills. The United Nations [45] projected an expected growth of the elderly population in Latin America from 8.7 in 2019 to 19% in 2050. Elderly people also make up the fastest growing group of Internet users [36, 47]. In Brazil, Internet adoption by the elderly population grew by 56% between 2015 and 2017 [6]. Besides, the use of smartphones for internet access grew by 107.7% between 2015 and 2017 by the elderly population [5, 6].

Older people often find application and website interfaces challenging to use. Many of these technologies do not take into account in their design process the limitations that this specific group may face due to ageing, such as their fitness, and physical or cultural factors [16, 25]. Thus, it is necessary to be concerned with the accessibility of such interfaces [27].

Accessibility is defined by ISO 9241-171 as “the usability of a product, service, environment or feature by people with the widest range of skills” [19]. This standard encompasses issues associated with creating accessible software for people with varying physical, sensory, and cognitive abilities, including those who are temporarily disabled and elderly people. Good usability and accessibility provide increased efficiency and effectiveness of applications, productivity, acceptance, support for users with low technology literacy or disability, and error reduction and training needs [32].

Several studies have analyzed accessibility guidelines for building applications for older people [16, 20, 24, 27, 29], with indications of good practices and design recommendations.

The World Wide Web Consortium’s (W3C) guidelines for making Web content and browsers more accessible to people with disabilities are important in providing access to specific disability issues. However, these guidelines do not adequately address issues specific to older users [16].

Nevertheless, there are other sets of guidelines designed for older people. We can cite the National Development Institute (NIA)/National Library of Medicine (NLM) [33] and other design guidelines focused on elderly people derived from research studies [26]. However, even such guidelines do not address specific accessibility issues related to emerging interactive technologies, such as those using sensors and the Internet of Things (IoT).

2.2 Smart homes and elderly people

Advances in smart home technologies offer considerable possibilities to improve the health and well-being of older people in the context of “Ageing-In-Place,” which aims to provide means for older people to remain at their homes instead of care houses. These advances can also help caregivers of the most vulnerable elderly people who need special care. Information and communication technology (ICT) is perhaps the richest field for producing new ideas for health care for the elderly [11]. “ICTs are used to monitor, manage and motivate a new generation of health care for older people” [17].

The development of interfaces for older people in smart homes should take the necessary steps to ensure good acceptance [35]. Due to lack of experience, insecurity in manipulating new technologies, and limitations due to age, older people should have a comfortable, clear, and natural interaction. In general, people often use technologies that add benefits and are easy to understand and use.

There are several general-use mobile home automation applications available for home use. However, there is limited availability of such applications in languages such as Brazilian Portuguese, which makes it difficult for users with little command of English, which is often the case for Brazilian elderly people. A previous study [42] revealed accessibility issues for elderly users in some of those applications when considering specific guidelines for this user profile. In addition to technologies that help control household devices, there is substantial research into features that can be used to assist elderly people in their daily household activities [3, 10, 15, 21, 22].

2.3 Related work

2.3.1 Studies on the accessibility of smart homes

The constant evolution and interest in smart homes have led to the construction of new products from different manufacturers, requiring different forms of interaction. This evolution makes the design more challenging for developers to meet the usability and accessibility needs of all potential users. There has been a growth in studies aiming at providing more accessible smart home solutions to different user groups.

Almeida et al. [1], in partnership with Bosch Thermotechnology S.A. in Portugal, carried out a study aimed at the proposal of a smart home architecture capable of multimodal interactions. The paper presents a proof of concept of the developed wizard, addressing a user-centered design. A set of personas and scenarios was used to define the requirements, considering the various modalities and forms of interaction, so that the assistant is accessible to several groups of users, including people with disabilities.

Queirós et al. [38] conducted a study with the objective of reviewing the smart homes area. The literature mapping study sought to raise aspects of the usability of smart home systems and how end-users are involved in the development process. The study also sought to survey the main products, technologies, and services in the smart home area. The article also shows that there is a greater focus on studies of smart homes with elderly people.

In another study, Duarte et al. [13] sought to investigate the state of the art of assistive technologies for elderly people. The paper highlights an overview of assistive technologies, systems development patterns, and the challenges encountered in the area. The study also highlights the absence of a reference architecture for the development of intelligent assistive systems, as well as issues such as data privacy, usability, and accessibility.

Buzzi et al. [9] aimed to develop guidelines for smart home Web interfaces accessible through screen readers. For this, the authors carried out a series of data collection through interviews and questionnaires [28] in Italy, in addition to tests with a system available on the market [8]. With the results, the researchers conducted prototype development and evaluation with real users. In the end, the paper presented guidelines or good practices to help researchers and developers create more accessible projects for people with visual impairments.

2.3.2 Elderly people using IoT

Peek et al. [37] developed a qualitative field study that involved home visits to older people with different health status, living conditions, and experience levels with technology in order to collect information about participants’ chronic conditions and the reasons for the level of technology utilization. The research showed that the perceptions of the elderly and the use of technology are incorporated in their personal, social, and physical contexts. Awareness of these psychological and contextual factors is necessary to facilitate ageing through the use of technology.

Silva et al. [41] developed an interactive TV platform that provides informative content about public and social services. The purpose of this platform is to promote info-inclusion and quality of life for elderly Portuguese people. The study encompassed two phases of implementation of a high-fidelity prototype and user testing. The evaluations were performed through field tests at the participants’ residences. The results obtained by the evaluation focused on the usability and usefulness of the platform for the elderly public. The users’ perception was positive and allowed the analysis and evolution of the prototype in question.

Ketsmur et al. [23] developed a conversation assistant for controlling a smart home accessible to a Portuguese family. They considered the different accessibility needs that a family may have due to the diversity of inhabitants concerning age, interests, and permanent or temporary disabilities. Smart homes usually do not provide information on consumption and spending and have limitations on the European Portuguese language. The study focused on meeting these needs. The authors used IBM Watson for wizard development and simulated a real environment using personas.

Oliveira et al. [35] conducted a comparative study of users’ perceptions of smart home technologies before installation and after using the system for 1 year. With the participation of 21 families, the researchers conducted a series of interviews to collect participants’ expectations, desires and knowledge. After the system was installed and used for 1 year, the researchers again collected data to compare with pre-installation data. Thus, it was possible to elaborate improvements in the developed smart home system, in addition to formulating design recommendations for the development of smart home systems for elderly people.

Tsuchiya et al. [43] presents a study to survey characteristics such as expectations and perceptions of the elderly in the Brazilian context about interactive technologies and smart home systems. The researchers sought to extract requirements that met the specific needs of elderly users. For this, focus groups were held with a total of 10 participants addressing the main technological aspects (difficulties, experiences, and desires) as well as demographic (limitations and domestic tasks more difficult to perform). As a result, the article proposed recommendations for the development of interfaces for home control systems with a focus on elderly people.

By analyzing the related studies presented in this section, we can observe that important advancements have been made to propose technologies to improve living conditions to elderly people and disabled people. However, there are still limitations to understand the needs and interaction characteristics of elderly people in specific regions, such as in the interior of Latin American countries as Brazil. Observing elderly people using prototypes of such technologies can help gain a substantial understanding of their actual perception of such technologies, and implications for the design of new technologies considering their needs and experiences in use.

3 Methods

This section presents the main methodological approaches used in this study. It describes the study design, prototype development, the evaluation procedure, the participants’ characteristics, and the procedure used for data collection and analysis.

3.1 Study design

The study aimed to design and implement a mobile Web application prototype for smart home control, followed by a user evaluation involving elderly users from the interior of Brazil. The goal was that the application was designed with a focus on usability and accessibility for elderly people. The development of the interface was based on the findings of the previous study [43], which involved focus groups with elderly users.

The evaluation aimed to identify points for improvement of the prototype and the reactions and user experience with the software. These results, along with the results from the previous focus group study, aimed to provide implications for the design of applications to control smart home systems for elderly users.

The usability tests were carried out with ten elderly participants, who attempted to perform twelve tasks in the prototype of the mobile smart home control system, operating a TV, a lamp, and a fan through a smartphone.

After completing each task, participants assigned a grade from 1 to 5 for the level of difficulty in performing it. In the end, we collected demographic information, users’ levels of experience with technology, and their experience with the tested application. The System Usability Scale (SUS) [7] standardized questionnaire was also applied to measure aspects of system usability. To measure levels of satisfaction, motivation, and feeling of control, the Self Assessment Manikin (SAM) [4] questionnaire was applied.

The Research Ethics Committee of the Federal University of Lavras approved the project. Participants agreed to the Informed Consent Form (ICF), signed before starting the evaluations, and authorized to perform audio and video recordings for further analysis. The evaluation sessions were conducted using simple language, with no technical terms. Sessions were conducted with the aid of a previously prepared script.

3.2 Development of the prototype interface

Contact with potential system users in a previous study [43] has allowed us to identify several important points we took into consideration in system development.

The implementation made was concentrated at the interface level, using CSS (Cascading Style Sheets) and HTML (Hypertext Markup Language). Contact and help pages have been added to bring more reliability to the system, and instructions on how to use the application.

The colors used in the system have been chosen to provide better contrast between text and background. WCAG criterion 1.4.6 Contrast (Enhanced), which refers to a contrast ratio of at least 7:1, was used as a reference. This criterion meets the highest compliance level (AAA level) of the Web Content Accessibility Guidelines (WCAG) 2.1 [46]. Some of the nomenclatures used to avoid the use of technical terms that made understanding difficult were not used. Fonts and button sizes have been increased, as well as spacing changes to improve mobile usage. Figure 1 shows examples of the screens implemented in the prototype.

Fig. 1
figure 1

Prototype pages implemented

Full size image

3.3 Participants

The participants involved in this study consisted of people who were at least 60 years old. Participants were required to be lucid, so they would be able to legally provide their informed consent independently to take part in the study.

Recruitment was performed using contact information obtained during the focus group performed at a partner institution in the city where a previous study was conducted [43]. We also invited other participants following an indication from other previous participants.

All participants had basic experience using smartphones and computers.

Participants were recruited to take part in the evaluations at the ALCANCE Group laboratory in sessions during October 2018 and September 2019.

3.4 Usability test procedure

The conduction of the usability evaluation sessions started with a brief explanation about the project and its objective to verify the usability and accessibility of the prototype so that older people can use it without barriers. In this way, it would be possible to identify improvement points and to verify the reactions and the users’ experience with the application. Researchers stressed that the test intended to evaluate the system, not the participants as individuals, but whether the design was usable to them.

We clarified to participants that the tests and data collected did not intend to disclose any personal information of each participant. For this reason, all personal data collected would be confidential, and the questionnaires and results obtained would be kept anonymous. Participation was entirely voluntary, so they would not be required to answer any of the questions presented and could feel free to ask any questions to the researchers. After clarification, each participant was asked to sign the ICF.

The sessions involved video recording with two cameras: one capturing the content displayed on the screen at the time of the interaction and another camera recording the participant’s facial expressions and comments in audio. We elaborated a set of twelve tasks. The tasks involved essential procedures related to the simulation of daily activities and the navigation of participants by every room in the house. We estimated that each task would last 5 min each at most. In the following, we list the full description of the tasks:

  • Task 1: “You had a busy day at work today and came home very tired. Then, you decide to go to your bedroom to lie down for a while to rest before taking a shower. When you enter the room, you remember that your favourite TV show is on and decide to turn on the television. However, the remote control is too far from the bed, and you are too lazy to get up to reach for it. So, you decide to use the home assistant app to turn on the TV in the bedroom. Turn on your bedroom TV using the app.”

  • Task 2: “Your neighbours are making a lot of noise, and the TV volume is low, so you are unable to understand the TV program. Turn up the volume of the bedroom TV using the app.”

  • Task 3: “You are in the commercial break for the TV show you were watching. However, since you don’t like commercials, you change the TV channel to look for a new show. Change the bedroom’s TV channel using the app.”

  • Task 4: “You finished the TV program you were watching, and then you decide to turn off the TV and turn off the lights in the room to get some sleep. Turn off your bedroom TV using the app.”

  • Task 5: “Summer is coming, and it is extremely hot today. Then, you wake up, go to your living room and turn on the fan to cool down a little. Turn on the room fan using the app.”

  • Task 6: “You need to prepare lunch, but you remember that you don’t have any ingredients at home and you need to go to the supermarket to shop. But before you leave the house, you turn off the fan in the room. Turn off the fan in the room using the app.”

  • Task 7: “You are attending a conference in Rio de Janeiro and are staying at the Copacabana Palace hotel. When checking in at the hotel, the receptionist explains to you that the devices in your room are controlled through the home assistant app. After many hours of flight, you arrive at the room and go to take a shower. So, you decide to turn on the bathroom lights using the home assistant app. Turn on the bathroom light using the app.”

  • Task 8: “After the shower, you turn off the bathroom lights using the home assistant app and sit next to the bedroom table to finish making the slides about the paper you are going to present at the conference. Turn off the bathroom light using the app.”

  • Task 9: “Daylight saving time has started, and this is a good time of the year to save energy. You spent all day at work and got home at 6 pm. As it is summertime, at 6 pm it is still bright, and you do not need to turn on the house lights. Therefore, you check if you have any house lights on using the home assistant app. If any lights are on, you turn them off to save energy. Turn off the house lights, if any of them are on.”

  • Task 10: “Your nephew came to your house and said that the kitchen was too dark. So he turned on a kitchen light using the switch. Check if you have received any notification messages informing you of the status of the kitchen light through the PushOver application installed on your phone.”

  • Task 11: “You are in the living room of your house sitting on the sofa browsing your cell phone. You feel hungry, look at the time and realize that it is the time you normally eat. You turn on the kitchen lights using the home assistant app, then head to the kitchen. Turn on the kitchen light using the app.”

  • Task 12: “Since you don’t have proper cooking practice, you put a frozen lasagna to heat up in the microwave. After the lasagna is ready, you have dinner and then wash the dirty dishes. Then you go back to the living room and turn off the kitchen light using the home assistant app. Turn off the kitchen light using the app.”

Participants were asked to use the “think-aloud” protocol, a well-established method in the field of human–computer interaction. This method involves asking users to “think out loud” while trying to perform a task on the computer. This method allowed the researchers to better understand the cognitive process by which users perform tasks and thus to identify how to improve the system for users [40].

During the tests, users were told that they could indicate at any time when they encountered a problem. The problems were recorded in a form by the researcher. The facilitator sought to interfere as little as possible with the performance of tasks by each participant.

At the end of each task, the researcher asked users to give a rating for the difficulty on a Likert [31] scale from 1 to 5, with 1 being very easy, 2 easy, 3 neither easy nor hard, 4 hard, and 5 very hard.

After the evaluation, the participants were asked to answer a short questionnaireFootnote 1 with questions concerning demographic data (age, educational level, and physical limitations) and about the use of technologies (experience, frequency of use, and possible difficulties). Besides, they were also asked to inform their perceptions about the application by providing criticism and suggestions for improvements.

To obtain general usability perceptions and acceptance, we adopted a standardized and widely used questionnaireFootnote 2 in several countries. The questionnaire was the System Usability Scale (SUS) [7]. For each statement, participants assigned a rating with “1” meaning “strongly agree” and “5” meaning “strongly disagree.” Although being a simple questionnaire, the instrument has being pointed as having properties for initial indications of satisfaction in the use of systems.

Finally, participants were asked to indicate their emotional reactions in a questionnaireFootnote 3 using the satisfaction, motivation, and sense of control scales, assigning a score from 1 to 9 for each of the aspects measured in the SAM [4].

3.5 Data analysis

After the usability tests, we analyzed the usability problems encountered. We also obtained quantitative results through the questionnaires and qualitative results regarding the participants’ impressions. With the input obtained, we prepared a report with the most critical accessibility and usability problems and indications for improvement and solution of the problems.

Data were analyzed using content analysis techniques of the videos recorded with the screen content at the time the tasks were performed and users’ reactions and comments. From this analysis, the report incorporated problems reported directly by users during the sessions and other usability and accessibility issues noted by researchers that may not have been reported by participants directly.

Content analysis was performed by two of the authors in two independent sets of usability tests, under the supervision of a senior researcher from the research group. Initially, a subset of the videos was analyzed with the researchers and cross-checked by the supervisor. When conflicts or disagreements emerged, the researchers discussed until they reached an agreement. After the initial analysis rounds with discussions, and after an agreed coding was reached, the two researchers in charge of the analysis proceeded with the full analysis of the remaining videos.

Besides, researchers analyzed whether the tasks requested from users were completed in order to record data on the success rate in performing the tasks. Data on task difficulty and satisfaction questionnaires data were aggregated to generate satisfaction statistics and improvements that can be made in the system.

For the task difficulty rate, we performed an analysis on each answer obtained from the participants, as well as a view of the degree of difficulty based on the highest rating scores. The scores of each SUS questionnaire applied were calculated following the calculation scheme provided by the authors of that questionnaire. With these numbers, we obtained an average score. This average score was then compared with the average of this questionnaire reported in other studies, as provided by the authors of the questionnaire. For the SAM questionnaire, a comparison of each response obtained during the tests was performed.

4 Results

This section presents the main results obtained from usability evaluations with elderly users. The following subsections present the analysis of the participants’ demographics, performance, usability problems, and a summary of the results of the questionnaire applied.

4.1 Demographics

Study participants were given a questionnaire to inform demographic information and questions about their experience with technology and interest related to the tested application. Table 1 summarizes the participants’ demographic information.

Table 1 Participants sample characterization
Full size table

Among the 10 participants, the sample included seven women (70%) and three men (30%), with a mean age of 73.1 years. Most participants lived with their family or spouse (70%), and three other participants reported living alone (30%). Regarding the level of education, one participant only attended elementary school (10%), five completed high school (50%), and four completed higher education (40%).

Regarding the use of technology, all participants used smartphones. Five participants mentioned the use of the mobile phone and computer combination, and two mentioned the use of the mobile phone and tablet. Only one participant had contact with all mentioned technologies: mobile, computer, and tablet. Most participants used these technologies daily (90%), and only one used it a few times a week (10%). Six participants considered their experience level to be average, while four said they had low familiarity with technologies.

4.2 Experience with technologies

The pre-test questionnaireFootnote 4 also contained five open-ended questions that addressed participants’ difficulties and shortcomings, difficulties, and suggestions for improvements to the use of current technologies, and, finally, the interest in acquiring a system that would control their home appliances by phone or computer.

Regarding the difficulty of performing any daily activity, or any disability that the participant considered relevant to mention, be it perceptive, physical, or cognitive, only one participant did not mention any characteristics. The other participants reported difficulties, mainly concerning vision. Most participants needed to wear glasses daily. Aspects such as spinal problems, which cause difficulty to lower and to lift weight, were also raised. One participant mentioned having difficulty remembering more recent facts and sometimes finding it easier to remember older memories. Regarding specific disabilities, one participant mentioned having arthrosis, a degenerative disease, usually associated with ageing, that affects the cartilage and tissues that protect the joints. The disease causes discomfort, pain, inflammation, and deformation, making movement difficult and even impossible [14].

Five of the participants stated they had no difficulty using current technologies. Participants who mentioned having a little difficulty detailed that this happened in more specific activities, in which they were not previously assisted by other people, or who had no previous knowledge. Besides, participants mentioned difficulties related to the excess of information present in the systems, as well as the learning difficulty of new technologies, thus generating a lack of interest in their use.

The use of technologies was considered a very recent phenomenon for many of them. The suggestions they provided to improve the use of technology were predominantly related to the lack of practice and contact with these technologies. They also reported on the high complexity in the control of the systems. They discussed the possibility of taking courses that would help them and also the availability of people who could help them more often. Another possibility raised was the abstraction and simplification of information in the technologies used. They also indicated the difficulty in reporting errors to people responsible for systems, such as communication with providers when the internet is not working correctly.

Nine of the ten participants expressed interest in using a system such as the one tested at their own home. Some of them showed great enthusiasm by commenting on how it could facilitate their daily lives. One participant was concerned about the difficulty of installation of the system, despite showing interest in using it. Only one participant showed no interest and justified that he prefers the essential use of technologies and not its application in the domestic context.

4.3 Activity performance

In performing each of the tasks described in the test scenarios, each participant was asked to rate the difficulty level of the task on a scale: very easy; easy; neither easy nor difficult; difficult; very difficult.

The results obtained for each task in each of the tests are shown in the graph in Fig. 2, where the vertical axis represents the tasks and the horizontal axis the number of users participating.

Fig. 2
figure 2

Difficulty notes of tasks assigned by participants

Full size image

There is a good portion of ratings considered easy (51.67 %), followed by very easy (31.67 %); difficult (9.16 %); and neither easy nor difficult (7.5 %). None of the activities was considered as very difficult. These numbers show that a vast majority (83.34 %) were positively positioned regarding the difficulty level of the tasks, considering the intuitive and easy navigation system.

Based on the ratings obtained for each of the tasks, we analyzed the ratings with the highest incidence. Of the 12 tasks presented, two can be considered easy, representing 16.6% of the sample, nine as very easy, framing 75%, and only one task framed as neither too easy nor too difficult.

In total, 120 tasks were performed during the ten usability tests, resulting in 110 completed tasks and ten unfinished tasks. All participants proceeded until the end of the evaluation and considered each of the twelve activities as completed. However, in one of the sessions, although the activities were considered complete, the participant did not perform the activities correctly. This task was not completed because the application simulates a different room on each page, and there was no room separation in the lab. This problem might have caused some confusion to this particular participant.

4.4 Usability problems

Analysis of the usability tests yielded a list of the problems found, in order to relate them to previous studies and propose new solutions for the prototype. The problems identified are described in the following subsections. We organized the problems according to the parts of the system in which they were encountered.

4.4.1 Help page

The help page had many usability problems that need to be considered. The feature did not help users as initially thought in the previous study [43].

Participants who reached this point in the navigation were confused and could not play the explanation video. This problem could be due to the video player being external to the application used. Every time someone tried to play the video, it ended up moving from the application to the video player application. Although the application allowed playing the video on the same system window, it was possible to trigger the behavior of running it on an external application, depending on the position the video was clicked on.

One of the participants went to the stage in which he had explanatory images on how to navigate the system. However, when viewing images similar to the screen, he tried to click on the tutorial images themselves. At this point the participant reported: “I am clicking to turn on the TV, but nothing happens.” The two participants who entered the help page did not obtain the expected feedback and chose to try navigating the prototype again, and eventually were able to perform the activities requested by the session facilitator.

4.4.2 Activities in the system

Some difficulties were identified for the execution of actions within the prototype. To control devices, before clicking the “ON” and “OFF” buttons for a particular device, attempts were made to click on other elements, such as clicks on device images, and text with the current status that appears right after identification of the device. After some attempts, the users then used the buttons. This behavior was even more evident for controlling the television. Since this device has more actions available at first, not all possible options are shown at the same time. The on and off buttons are the last ones listed visually.

Also, four users turned on the fan instead of turning on the lights. This problem was possibly due to the color of the images illustrating the “light” and “fan” functionalities.

One participant was slow to realize that the link on the homepage was a clickable element, despite the use of different formatting as used on most sites.

4.4.3 Screen scrolling

One user encountered difficulties in scrolling through system pages. During the first minutes of the test, the participant did not try to slide the screen up and, for this reason, could not find the television, the device to be manipulated in the first scenario. After a while of trying, navigating through the prototype pages, the participant realized that scrolling was necessary. He said that it was inattentive not to try to find other elements that were not present on the screen.

4.4.4 House room identification

One of the participants failed in some attempts for not changing rooms as requested in the activities. However, he successfully performed the first six tasks, which were in two separate rooms. From this point on, the user no longer switched rooms and reported not being sure if he was handling the right appliance, as there was only one on the screen.

Some other users also performed the requested actions in the room already open at the time of reading the following scenario. However, soon after, they identified that the current was not the requested room and then navigated to the page of the correct room, leading to the accomplishment of the task.

4.5 Acceptance: SUS

To measure aspects related to satisfaction related to system usability, we used the SUS questionnaire, created by John Brooke in 1986 [7]. It is considered a flexible and fast tool, widely used to measure the usability of systems.

SUS has ten questions that are rated on a scale from 1 to 5. Table 2 presents the scores obtained by the SUS questionnaire for each participant.

Table 2 SUS results provided by each participant
Full size table

Overall, the system obtained an average of 68.75. Some researchers reported the use of larger datasets with the application of SUS in order to establish a classification scale that would allow the comparison of results obtained with the application of the questionnaire [2, 39, 44].

We noted that some participants obtained scores below 60 (P7, P8, and P10). During the tests, P7 and P10 emphasized that despite the applicability of the control system, they still prefer manual and technology-free control of appliances. Participant P8 obtained a low score due to lack of familiarity with the smartphone device.

With the average obtained in usability tests, the score obtained by the developed prototype can be a satisfactory result. According to Lewis and Sauro [30], a SUS score above 68 would be considered above average.

4.6 Emotional reactions

To measure the reactions and perceptions of the evaluation participants, the SAM [4] questionnaireFootnote 5 was applied at the end of the test.

SAM is an evaluation method that uses pictograms and addresses issues related to the affective quality of a computer system. From the scale, it is possible to identify three dimensions: pleasure, arousal, and dominance of a person when using a computer system. A scale represents each dimension with values from 1 to 9, and the user must choose those that best represent his/her emotions [4].

Figure 3 shows the results obtained. The vertical axis represents the number of users who chose the option, and the horizontal axis represents the value scale.

Fig. 3
figure 3

Questionnaire results SAM

Full size image

The answers obtained were positive, concentrating the scores above 7 on a scale from 1 to 9, with the maximum score representing the most positive aspects for each reaction. In terms of satisfaction, most participants considered themselves extremely satisfied with the system. The participants also felt very motivated. Some participants reported on the well-being coming from the possibility to collaborate with the research, which may have intensified this sense of motivation. The feeling of control was the factor that was lowest on the scale. Despite a good position, it was the only reaction that received a neutral rating. Participants often did not feel in control of the situation due to fear of using an application they had never been in contact with before, even showing concern about not having an initial explanation on how to use it.

5 Discussion

This section discusses the findings from the study, focusing on design implications, critical difficulties encountered in performing tasks, and participants’ reactions, as well as limitations to the study.

The study elicited valuable feedback during the analysis of the evaluations. This feedback emerged from the tests performed and from the data obtained through the application of the questionnaires. Although the success and satisfaction rates obtained were considered adequate, the suggestions for improving usability were crucial.

The tenth task (receiving a notification) was the most difficult for participants due to the need to use another application together with the developed system.

Other problems were caused by the need to scroll the screen and by the difficulty in identifying the actions available on the system. Possible options could include using hyperlinks that clearly show the possibility to use all devices at the beginning, without the need to scroll the page, or in a feature that indicates to the user that information is not yet visible.

The clear presentation of a list of available rooms (Fig. 4) was another need for modification raised. This problem was motivated by the fact that users did not use the menu options initially. Thus, from the first contact, the user would already have the notion that the system separates the devices by rooms.

Fig. 4
figure 4

Available room options

Full size image

These aspects relate to the content and navigation terms considered in the Inclusive Web Design Checklist (IWDC) [29] guidelines sets. Often, both experienced and inexperienced users tend not to scroll, not to exceed the visible screen boundary. Another factor considered is the lack of visual treatment and explicit labelling to identify which hyperlinks are clickable, as they might not be as evident as a button, for example. Avoiding the need for scrolling is also one of the criteria proposed by the IWDC. When scrolling is necessary, including icons that highlight scrolling is a possible alternative to avoid confusion.

The study also showed that only the title at the top of the page is not ideal for indicating the location of the system, especially since sliding the screen disappears. This problem may be associated with the need to reduce short-term memory load [40]. Since room location information is only at the top of the page, and they all look very similar, there is a time when it is not apparent to the user what room they are at the moment. An example of this situation can be analyzed in Fig. 5.

Fig. 5
figure 5

Room identification problem

Full size image

One of the activities asked the users to check if they received a notification indicating that the kitchen light was on. One user noted that despite receiving the message, the application showed the light turned off. This correction is a necessary improvement in the system. Although users can control the appliances, their current situation is not identified if there is no user action. The application must show the user the status of the appliances.

Regarding the availability of a tutorial to help users with questions about how to proceed with system activities, we need to study the best way to enable this feature. The tutorial must be well integrated with the interface, without disruption when opening the help system.

During the tests, participants initially showed some insecurity. This insecurity was possibly due to not receiving an explanation about how to use the system, to not having had any contact with the application before. Nevertheless, the excitement was remarkable when they could manipulate some of the devices. They expressed contentment when they identified that they had been able to perform the task and were willing to explore the application even before instructions for the next tasks.

The study design with the abstract separation of the rooms without the actual physical separation in the laboratory may have confused some of the participants. The organization of the rooms in the system might have been the cause of the actions that were not completed due to the participant not having recognized he/she was not in the correct room in the application, as mentioned in the task.

There were many issues related to items that seemed to be clickable to participants but were only illustrations or informative texts. The application used commonly used interface components present in many mobile applications used for Web and Android devices. It could be that younger participants that are more used to such technologies would have noticed more quickly which elements are clickable or not. However, it seems like older users needed more explicit affordances of what was clickable and what was not clickable.

5.1 Implications for design

From the usability problems elicited in our study, as well as from the data gathered in the focus groups studies performed in a previous paper [43], we have summarized the main implications for the design of mobile applications to control smart homes focused on elderly people.

  • Whereabouts in the house on the screen: Designers should take special care in designing navigation schemes to enable elderly people to locate appliances in the respective rooms of the house. Correspondence with the actual physical environment is essential to enable elderly users to know their whereabouts in the system and perform the correct actions with the right appliances.

  • No information is hidden under scrolling: As shown in previous studies [24, 26, 27], scrolling can be awkward for elderly users, as they might not notice that more information is available on the screen. Smart home applications can have a substantial amount of information shown on the screen, especially for devices with more features, such as TVs. Avoiding scrolling when possible, or making it clear that more information is available on the same screen is essential to enable appropriate interaction.

  • Help: What to do with images and videos?: Help is essential to aid elderly people in using interactive systems. In the interaction with smart homes, novel ways of interacting can be frightening, so videos and images can be helpful to make instructions more concrete. However, many elderly people can be confused about whether they can interact with demo screens in videos or images and make them even more confused if they do not work. Making clear that the information on videos is just part of a demonstration, or trying to make the screens on instructions interactive may be a reasonable possibility to improve such issues.

  • Context: limit the windows: Opening new windows to manage at the same time can be complicated to manage for elderly users. Smart home systems have to limit the number of windows they open, especially when users are not fully aware of that behavior.

  • Affordance: what is clickable?: Smart home mobile applications have several illustrations of devices, explanatory texts, and buttons. Although some commonly used buttons (such as flat buttons) can be usable by young users, elderly users can have severe difficulties in recognizing what is clickable and what is not clickable with so many pictorial representations on the screen. Using clear button layouts to show what is clickable is essential to avoid confusion.

  • Proximity on the screen: The Gestalt principle of proximity has to be taken seriously in the design of smart home mobile control systems for elderly users. Although making elements, descriptions, and action buttons close enough helps all users, the study showed that elderly users were particularly affected by difficulty with visually recognizing the grouping of rooms and appliances in the house context in the application.

5.2 Limitations

Although the usability evaluation included a small sample of participants, the study raised valuable implications. According to Nielsen [34], five participants is an adequate number to reveal at least one instance of the overall usability issues that would be encountered with a sample using more participants. This way, even though the results from the present study cannot be representative in quantitative terms, the results with 10 participants had good potential to reveal relevant usability problems at least once.

The usability evaluations and the focus groups conducted in the previous study [43], with a different group of 10 elderly participants, may not be considered statistically representative of Brazil or of the town in which the study was conducted, with a population of around 100,000. However, we had an adequate variability in terms of education levels, use of technology, and age.

We also acknowledge that the results of the study could be widely variable in different regions of Brazil and Latin America. Brazil itself is a very diverse country, with varying levels of human development and access to technology by elderly people. The south of Minas Gerais state, where this study was conducted, is a reasonably well-developed region. It would be interesting to expand this study to other regions of Brazil and Latin America with a lower level of access to technology and education to elderly people.

Although we noticed problems regarding the low availability of mobile applications to control smart homes in Portuguese, we acknowledge that other more profound language issues could impact Brazil and Latin America. There are many regions with language minorities in Brazil and Latin America, especially involving indigenous peoples and regions with intensive migration processes. Southern Brazil, for example, has large groups of elderly people who are immigrants from European or Asian countries who still are not fluent speakers of Portuguese. This situation also occurs in many other Latin American countries. Investigating the language impact of such systems to enable access to elderly people would be an essential research topic to be investigated.

6 Conclusion

The present study aimed to analyze the implications for the design of smart home control systems on mobile devices for elderly people in Brazil. The results were based on the evaluation of a prototype of a Web and mobile home control system by elderly people.

As the main conclusions, we observed that the prototype obtained satisfactory results through the tests performed. The difficulty levels of the tasks were considered low, meaning that the prototype had an acceptable level of learnability for users to navigate without the need for an initial explanation of how to use it. The success rate of execution of activities was above 90%, considering that the test environment did not allow the simulation of a real environment, in which users could differentiate the rooms of the house for the manipulation of devices.

We also observed that, in addition to the interest in new technologies, older people are enthusiastic and see benefits in their adoption. The use of specific guidelines directed to elderly users and usability heuristics in the development of an application allowed their inclusion as a target audience, allowing them to offer more comfort and independence.

Questionnaire responses showed that elderly users in Brazil have considerably lower knowledge of smart home technologies than people in more developed countries, possibly because of the costs and availability of the market. However, even with less access, the participants showed that they were very open to experimenting with these technologies, as long as they could meet their needs.

As implications for the design of mobile smart home systems for elderly users, from the lessons learned from the evaluation with elderly users in the interior of Brazil, we can highlight the following: the importance of care in designing help systems with explanatory images and videos; care with opening new windows and changing the focus of elderly users; attention with the perceptible affordance of clickable and not clickable items; the positioning of pictorial house elements and their descriptions for easy identification; attention with the use of scrolling for recognition of available elements on the screen; and precise identification of the location of elements in the house.

As future work, we intend to perform usability evaluations with other user profiles. Among the intended improvements, we can mention the visualization of devices in each room of the house, the reduction of the need for scrolling screens, and improvements in the help session. We also intend to expand the current study with participants from other regions in Brazil and other countries in Latin America, in order to compare results and analyze possible cultural differences in the use of this type of technology.