1 Introduction

In the past decade, research on smart homes has been moving towards applying the principles of ubiquitous computing [8]. The smart home adjusts its functions to the inhabitants’ needs according to the information it collects from the inhabitants, the computational system, and the context. In this kind of intelligent environment, information processing and networking technology is hidden away, and interaction between the home and its devices takes places via advanced, “natural” user interaction techniques, such as speech [1].

Invisibility of computational technology is a major advantage of the future home, but it may also be its weakness. Users accustomed to manual access to their home devices might not be ready to interact with their familiar environment through new interaction techniques [13]. It is also questionable whether the users want the technology to be hidden or just reshaped.

Rodden and Benford [12] propose that the future smart home must evolve from our existing homes. In general, current homes are not designed with the requirements of ubiquitous computing in mind. Thus, even though most technical means of ubiquitous computing exist already, the transition to actual, everyday use of such computing will take time and effort.

To intermesh new technology with the old and familiar home environment, we need more research on domestic settings regarding the roles that various home devices can have in a smart home environment. Rodden and Benford [12] divide smart home devices into the following three types: interactive household objects, which are familiar, existing household objects incorporating new possibilities for interaction; augmented furniture, which has interaction capabilities embedded in it; and information appliances, which are existing household appliances with standardized interaction and communication facilities and devices layered on them.

In this study, our aim was to explore the possibilities of using information appliances already in the home environment to control a selection of familiar household objects. Three familiar information appliances—a PC, a media terminal (enabling interaction via TV by remote control), and a mobile phone—were used as user interfaces (UIs) to selected smart home functions. The usability and acceptability of these functions were analyzed through focus groups, laboratory tests/interviews, and an ethnographic study of actual use and living experience in a real smart home.

In previous research by others, user experience of smart home solutions has been studied, for example, with tests and walkthroughs of Wizard-of-Oz-type prototypes [3] and brief field trials [10]. Also, longer trial periods have been used, but the users have often been a part of the design team [6, 9]. The special value of our research lies in its empirical settings that allowed comparison of three functional UIs in an actual living environment over a long time period (six months).

2 Research on smart home living experience

2.1 The research context

The Smart Home Usability and Living Experience project [7] was carried out during May 2002–March 2003 at the Institute of Software Systems at Tampere University of Technology (TUT), with the support of Nokia mobile phones, Pikosystems (accessibility solutions), and Tekes (National Technology Agency of Finland). The project focused on the usability and acceptability of interaction solutions for smart home environments.

Usability aspects were studied ethnographically in an eHome, a two-room apartment with basic infrastructures for an intelligent environment, built for research purposes by the TUT Institute of Electronics. The apartment is located in a standard but new apartment building area near the center of Tampere (a city of 200,000 inhabitants) in southern Finland.

The technologies in the eHome included automated/controlled lighting and everyday “smart objects”, such as moving curtains and status-aware plant pots. The functions were controlled via the above-mentioned three UI devices. The PC (a laptop) was used mainly in the living room, the media terminal was placed also in the living room (attached to the TV), and the mobile phone was, of course, operable both inside and outside the apartment.

The above UI devices were connected via various network technologies (WLAN, Ethernet LAN, and GPRS) to the main computer, which worked as a bridge, whereas the “smart objects” were connected to a serial HUB. The sockets and lights were controlled via a commercial LINET control system. All network technology was hidden from the users.

The research evaluated the usability and acceptability of the three device/UI entities, which, in this paper, are referred to as “user interfaces.” The focus is on the entities (not only on software UIs or only on physical devices) because users do not differentiate between software and hardware, but, instead, they see these software/hardware combinations as entities for performing tasks.

2.2 The research process and methods

For our research guidelines, we adapted the principles of the human-centered design set with standard ISO 13407. Our study consisted of the following three main phases:

  • The definition phase, during which we collected the basic UI user requirements

  • The design and implementation phase, during which we tested and iterated the UI prototypes.

  • The evaluation phase, during which we gathered and analyzed data on user experience

In the definition phase, we collected user requirements of interface design from the literature and through contextual inquiry [2], theme interviews, and focus groups [11]. Contextual inquiries and interviews in people’s homes helped us identify their daily living patterns and needs for home technology, define their home environment, and write realistic scenarios of life in a smart home. The scenarios were then presented in focus group sessions to help collect information about user attitudes and possible prejudices to new kinds of interaction in a future home. For a wide range of views, we selected subjects for our inquiries, interviews, and focus groups (a total of 22) from among young adults, families with children, middle-aged couples, and the elderly. None of the subjects had any previous experience with smart home solutions.

In the design and implementation phase, three alternative smart home UIs were prototyped, and we then evaluated their usability by heuristic analysis and usability tests. The prototypes were iterated according to the feedback. Four to five subjects tested each solution in the laboratory to help us determine at least 80% of the possible usability problems. The subjects were mainly young adults, because the end-users in the ethnographic study, that is, the inhabitants of the smart apartment, were also young adults.

In the evaluation phase, we installed the three UIs in the eHome one at a time. The PC UI was installed first, followed two months later by the media terminal, and the mobile phone at the beginning of the third month. All three interfaces were, thus, available for the users for the last three months of the study. During the research phase, we also allowed the inhabitants to access the critical smart home functions manually, to make sure that the home functioned in case the inhabitants had problems with the technology. We studied the inhabitants’ use of the UIs’ ethnographically, using contextual interviews and participatory walkthroughs. In particular, we sought to evaluate the UIs’ usability in their actual context and to find the requirements that the inhabitants might have for accepting the devices for various smart home tasks.

The tenants of the eHome were selected from among those who applied to the apartment house. Mainly, we required that the applicants be neither technically oriented nor resistant to novel ways of using information appliances. The selected couples were a 26-year-old woman (historian) and a 27-year-old man (biologist). The ethnographic phase lasted six months.

3 Definition of user requirements for home UIs

In the definition phase, we used contextual inquiries, interviews, and focus group sessions to collect user attitudes about UI use, home activity scenarios, and UI solutions.

3.1 Nature of home activities

In a familiar environment, human behavior assumes certain regularities. Doing everyday chores often turns into chains of action, which assume patterns, such as doing the laundry at a particular time and in a particular place and way. According to Crabtree et al. [4], these chains of action occur in various action centers. In this study, we found that, in particular, families with children and the elderly had established activity patterns. The subjects thought that the patterns helped create a stable growth environment for their children. Furthermore, the patterns also contributed to a sense of belonging, particularly in the lives of single adults, who achieved a sense of belonging by forming patterns around different media. In fact, media devices, such as the TV and PC, have become significant action centers at home. For example, in front of TV sets, people do chores (for example, fold clothes), entertain or baby-sit children (for example, with cartoons), and make their plans (for example, according to weather forecasts). In addition, as the TV and PC can also be used for lamps, radios, clocks, etc., their adjustability makes them natural smart home interaction and communication devices.

Because they are scattered between activity centers, actual home tasks often require mobility [4]. In addition, multitasking is common in a home environment, and various simultaneous tasks require varying amounts of activity. For example, the subjects could make dinner, water house plants, supervise their children, and do laundry, all at the same time. Clearly, their interaction follows a sequence of steps they can resume and build upon [1]. This indicates that there is a need for several communication devices that are provided throughout the home. On the other hand, the communication/control device itself can be mobile. A mobile device, for example, the mobile phone, has the advantage of becoming a personal device and, in addition, enabling remote control while the person is not at home.

Multiple users constitute a challenge to home interaction because they may want to interact simultaneously. The focus groups expressed, especially, the need to be able to easily shift between privacy and togetherness. Such shifts become possible if the various UIs collaborate: the TV is a medium for a group, whereas mobile phones are personal.

3.2 Expectations about UI techniques

We presented new interaction techniques to the focus groups with scenarios and descriptions about their use and selected the UI options based on already existing interaction techniques; speech [14, 16], gesture [5], graphical UI (GUI), and automation [9].

The subjects ranked automation as the most wanted interaction technique. However, they did not want full automation but chains of functions they could program or set up themselves. Users would like to set the causalities of context-aware functions themselves: “When this alarm stops, I want that lamp to turn on.” But, even though automation could be context-aware, subjects felt they are not ready for it, for they fear that technology does not understand the various functions in the home. Thus, besides accepting the technology in practical terms, people may have problems accepting socially and emotionally what is “smart” in their homes. For example, in a romantic mood, the user may want the lights to dim, even though that may otherwise be impractical.

The focus groups thought the speech interface was a handy way to interact in small tasks and especially in the kitchen, for example, to answer the phone while the person was baking. Nevertheless, the subjects generally assumed that the commands should be short and curt. They thought natural speech interfaces would be risky: “What if one says something, and the system interprets the words as a command? It would be an awkward mess if the guests had to be advised about what they could say in the kitchen.” The focus groups considered the GUI as efficient and easy to use as the speech interface. According to them, the visibility of the GUI’s functions made it easy to perform tasks. The participants’ experiences with mobile phone speech interfaces had obviously affected their attitude towards speech interaction. Therefore, they trusted the GUI more than the speech solution.

In the definition phase, the subjects thought gestures alone were not very natural for interaction. However, they were not given scenarios where gestures would be part of a multimodal input (for example, speech and gesture [3]). Multimodality may, in fact, have been acceptable to them.

The focus groups were not fully satisfied with any of the presented interaction techniques, and the conversation in all groups soon moved to another option, centralized remote control; that is, a remote control with which to interact with an increasing number of domestic devices. Older participants thought that one or two remotes would be enough for a home, whereas younger participants wanted each inhabitant to have their own personalized control device.

4 UI solutions for empirical study

With the evolution phase in mind, we focused on evaluating user experience of the PC, media terminal on the TV, and the mobile phone as interaction devices and excluded “natural” UIs in this study. We selected the information appliances based on their ability to support the nature of home activities and assumed that the appliances could be adopted as action centers in a smart home.

4.1 UI prototypes

The UI on the PC is a GUI in which the interaction style is direct manipulation (Fig. 1). The user could control dialog by moving visual objects with the mouse or by touching the screen with a pen or finger. The media terminal (Fig. 2) and mobile phone UIs (Fig. 3) were both menu-based and, thus, allowed users to navigate the menu hierarchies with the arrow buttons on their remote or phone. For example, to set the lighting via the media terminal, the user chose “Lights,” then selected the room, and, with the left and right arrows, adjusted the brightness of the lights.

Fig. 1
figure 1

Example of a GUI on a PC

Full size image
Fig. 2
figure 2

Example of a media terminal UI on a TV

Full size image
Fig. 3
figure 3

Example of a mobile phone UI

Full size image

On the basis of the usability tests, we made some minor changes in the UIs, but left the overall interaction modes as they were in the prototypes. However, the interviews we conducted after the actual test sessions gave us valuable information about user expectations. From the comments, we could deduct that the main problem with the PC and media terminal was their fixed location. The inhabitants thought that the GUI should be also accessible while they were mobile in the house, as well as outside the house. In addition, they said the devices should be constantly turned on. On the other hand, though they found mobility appealing and useful, they were also concerned about reliability, especially about controlling appliances from a distance (for example, if a home appliance was really turned off as the interface indicated).

4.2 Functions for ethnographic study of living experience

To gain empirical knowledge about the suitability of the UIs for smart home control, a set of functionalities were implemented in the two-room smart apartment.

A few simple everyday tasks were chosen as the functionalities to be controlled via the three UIs (Table 1), for simple tasks work as overall measures of user acceptance of smart home technology. When users gain confidence with simple tasks, they are ready for more complex tasks.

Table 1 Functionalities controlled via user interfaces
Full size table

The focus groups revealed that the smart home solutions that were easiest to accept contribute to security and saving energy. The prototype had security built-in in the form of the possibility to monitor the state of the home’s electrical appliances. An appliance could be turned off by switching off the circuit in a wall socket via the UI.

Another major expectation about smart home technology was possible freedom from housework; in practice, for example, care of the home from a distance. This becomes a concern especially during holiday seasons when people do not necessarily spend time at home. Interfaces may then provide access to functions, for example, to monitor moisture, temperature, and lighting of the plants at home.

In addition to their practical purposes, the rooms in the home can also express the inhabitants’ moods. The impression of space that the home imparts varies according to who the inhabitants are and what they do, and this mood can be made visible with elements such as light and sound. We took this aspect into account by enabling the inhabitants to create their own lighting states by stepless control of lights and curtains. The lights could be controlled individually or in groups, and the states could be saved and named for later use.

In addition, the interest towards user-controlled automation was acknowledged in the design of the functionality of lights and curtains. The GUI on the PC provides a possibility to set automation by creating timings. The inhabitants could define a time for specific lighting (with lights and curtains) or select a previously defined lighting mode. The same timing could also be used repeatedly, daily, weekly, or only during weekends. In addition, timings could be created and stored, and the stored ones activated later on.

5 Findings of the ethnographic study

This section summarizes the results of our smart home study, the six-month living experience of a couple in the smart apartment.

The couple used all three UIs freely and willingly, though no single UI or access technique fully replaced the old (manual) access modes. The couple had the most additional value from the luxury provided by the mobile phone to control home devices in new contexts (for example, from bed) and from the pleasure of automating the timing of the lights and curtains (for example, the lights could be set to turn on just before they came home).

5.1 Requirements for the UIs based on activity patterns

Our ethnographic study brought to light further user requirements of smart home UIs. The home tasks can be categorized in terms of reaction time. On one hand, users need functions to handle familiar and recurrent activity patterns; that is, tasks that require pattern control. On the other hand, they would like to have functions for impulsive and unexpected tasks; that is, tasks that call for instant control.

The need for pattern control is associated with functions that users can set with time—hours, days, or even weeks earlier. Pattern control enables them to prepare their home for everyday routines, such as workday morning rituals (for example, turn on lights to dress up, open curtains to have breakfast). By pattern control, the user can plan and program automation.

Instant control can be described in terms of “right now” and “right here.” The “right now” principle demands constant standby of the interaction device and simple task performance with only few action steps (for example, shortcuts). On one hand, the “right here” principle requires centralized means for the user to control all the different devices that are affecting the action center of “right here.” For example, while watching TV, the user can adjust the lights and close the curtains in that space. On the other hand, the principle can be determined as a possibility for the user to—without moving from “right here”—use centralized control to similar devices or functions, which are scattered among the different action centers. For example, the user may adjust the lights all over the home at once.

When these means of control are enabled by the smart home system, the user experience can follow the natural activity patterns at home. The next section presents how the three UIs supported these requirements.

5.2 Living experience with the three UIs

Our couple liked the controlled automation in the apartment and appreciated its potential. They used automation to optimize the home environment for particular action patterns. The couple felt that the automation was not necessary for action itself, but for preliminary work, and they used it to prepare their home for whatever next action phase they were expecting. For example, they would program some lights to turn off or dim at bedtime. The GUI, directly manipulated via the PC, was well suited for setting timings and the automation of pattern control. According to these experiences, a PC can easily act as a central controlling unit for the functions that support the familiar and constant patterns that are not so time-limited and situation-critical.

The couple had some problems with the invisibility of automation. They would forget that they had, for example, timed the lights. Surprises occurred especially because more than one person operated the devices, and they were not always aware of each other’s settings. Another aspect they did not get used to during the six-month test period was that their home could “live a life of its own.” The home was not exactly the same in the evening when they came home as it was in the morning when they left it. For example, a non-smart home with the lights on and the inhabitants not in would suggest an uninvited visitor. Towards the end of the test period, the couple grew used to the independence of their home and the possibilities of controlling and adjusting things in it from a distance—by mobile phone—to make homecoming more pleasant.

The couple had quite high expectations of the media terminal. They assumed it would be a natural controlling device, because they were accustomed to keep the TV on even when they did not actually watch it. Their problem was basically the mobility of the tasks. Even though the remote control provided some mobility, control was tied up with the media terminal and the room it was in. In addition, switching from passive TV viewing to active instant control was not as easy as the couple had thought, for they had to find the remote first and point it to the media terminal. Furthermore, because the remote functioned as a single-user input device, both users could not use it simultaneously. On the other hand, the media terminal with the TV screen offers ample joint use potential for group-centered output.

Before they tried it, the couples were quite skeptical about the mobile phone UI and thought that they would not use the device at home. Nevertheless, during the test period, the mobile phone seemed to be always within their reach, partly because they did not have a fixed-line phone. Consequently, they used the mobile phone so actively, especially for in-home control, that it became their primary and most frequently used UI. They found it suitable even for demanding instant control. The mobile phone became their preferred device because, unlike the PC or the media terminal, they always kept it on. Thus, even though they had to turn on the application, they felt the device was always ready to perform tasks.

The couple considered their mobile phone a personal device, and accessing home functions while they moved inside or outside the home felt like luxury to them as they could control their home appliances from almost anywhere. For example, they would check if the coffee maker was switched off or turn, for example, the lights on while they were in bed. The study showed that to gain the most out the mobile phone UI, all family members should have their own phones; otherwise, the situation indoors would be comparable to what we said above about the media terminal remote.

The mobile phone is flexible in its potential for use, and its users are aware that new series and models come with new additional functions. This study confirms that the use of mobile phones could be easily extended to the home environment.

5.3 User confidence in smart home technology

People gain confidence in a new technology through using it, through testing its functions, and evaluating the results. The couple’s trust in the devices and interfaces in their apartment was not complete, but varied from day-to-day. In general, they used the interfaces only when they could confirm visually that the results were as they expected, or to perform non-crucial tasks. Thus, at first, they were doubtful about controlling things from a distance and apprehensive about the risks of leaving devices active for extended periods. They were even used to disconnecting the TV when they left home or went to bed. Though they knew that the devices did not consume much power and were not a fire risk, they were subconsciously worried. However, controlling household things from a distance intrigued the couple, and we can assume that, with time, they would have become confident of such control.

6 Discussion and summary

This paper is a report on the study of smart home usability and living experience conducted in Tampere, Finland, in May 2002–March 2003. We aimed to evaluate alternative UIs for control of devices in a smart home environment by investigating users’ expectations and requirements of a smart home, and by conducting an ethnographic study of living in a smart home.

This paper built on the hypothesis that experience of a smart home evolves stepwise. A home must feel safe, and its safety and reliability grow along with the inhabitants’ familiarity with the things in their home. Thus, we focused on the communication devices already present in many homes—PCs, TVs, and mobile phones—around which smart home interaction could be expected to evolve. The study shows that the subjects welcomed the smartness that was added to their homes with the devices already there (for example, the TV).

Future smart homes are targeted towards a variety of users. In our ethnographic study, the users were young non-technical professionals, and their experience differs naturally from that of, for example, the elderly and the disabled. Nevertheless, understanding home activity patterns is independent of the target group. The needs and expectations we discovered about smart homes suggest that they would be similar among similar users in similar cultural settings.

The following constitute our main findings:

  1. 1.

    We identified two main types of activity patterns that require different UI solutions.

  2. 2.

    Pattern control enables users to control the automation of tasks that can be predetermined and planned. In this case, the PC, with its diverse input methods, is an ideal central control unit, while the media terminal with a GUI offers an alternative for direct manipulation of interaction.

  3. 3.

    Instant control enables immediate control in a real-use context. As a central device for instant control, the mobile phone is ideally suited for mobile tasks.

  4. 4.

    Subjects were interested in centralized control. The mobile phone could become the primary centralized remote control while its personalization capabilities could add to its usability. In addition, advanced UIs with overlapping functions and optimized for different home tasks could be made available.

Considering the evolution of the home, we should point out that home is not only a product or a physical environment to support functions. The home is also resonant with hidden purposes and meanings, which must be protected as technology is reaching its doorstep. But as homes are also stages upon which people express their identities, they may not remain completely outside technology. In the future, control can be exerted with the input and output devices that already exist in the home or with natural interfaces, though full automation of the home’s functions are an unlikely option. Even Mark Weiser, the father of Ubicomp, makes it clear that it is a myth that the computer knows what it is like to be human [15]. Therefore, computers should not make choices for users, but the other way around. A crucial question is how options should be provided. This study proposes two ways of interaction—pattern control and instant control—by means of the familiar communication devices already there in domestic environments.