Keywords

1 Introduction

The evolution of computers has emerged a new generation of users whose expectations, in terms of computer interaction, have been raised. This new generation of users is not only seeking for the information, but they also would like to be able to create, collaborate, and interact with the digital world. Augmented Reality (AR) [1] is one of the human-computer interaction technologies that has shown great potential in the recent decade. The goal of AR is to provide users the enhanced version of the real world by overlaying virtual computer-generated objects to the real physical environment. It aims to seamlessly blend the virtual objects with the real world environment in order to bring a new improved experience for the users [2]. The term “Mixed Reality” (MR) [3] and AR found to be used interchangeably – despite some minor distinctions – however, MR is designed to combine both features of the Virtual Reality (VR) [4] and AR.

In recent years, AR has faced an increasing potential and acceptance in different domains including the activities that need continuous and smart assistance. The superimposed virtual objects are able to guide the users to find the location of the real objects within the domestic environment, and provide them a complete step by step 3D animated instruction – with visual and voice feedbacks – on how to complete a task in regards to each of the related object. This could bring strong advantage in the field of Smart Homes [5] and Ambient Assisted Living (AAL) [6], in which inhabitants’ constant support in performing the Activities of Daily Living (ADLs) [7] is required. The paradigm of smart home encompasses multiple methods and techniques to represent all the house appliances in a connected network of smart devices in which they are able to transmit and exchange data and information.

This work describes an AR framework, called “HoloHome”, within the smart house in order to investigate a new means of interaction for the inhabitants to manage and regulate the domestic environment and to perform their daily activities in a more convenient and independent way to fulfill the requirements of an AAL environment. HoloHome is part of the Italian project “Future Home for Future Communities (FHfFC)” [8], which aims to develop the “house of the future” by integrating multiple technological paradigms to promote inhabitants’ comfort, safety, and independence while providing them continuous support in performing different ADLs. The “house of the future” has been implemented inside the STIIMA’s Living Lab in Lecco and is currently under development phase.

Although some ADLs within the house might seem to be simple and intuitive for most of us, but some people like elderlies and people with cognitive impairments face a lot of challenges to perform regular daily tasks [9]. These tasks could vary from operating a kitchen appliance, opening and closing the window, or a simple act of turning on/off the light.

In the context of AAL, it is of pivotal importance to enable Ambient Intelligence (AmI) [10] to equip the domestic environment with Context-Aware (CA) [11] devices which would guarantee smart services for the inhabitants. The aim of these services is to help people with special needs to live more independently, in a safe and healthy environment. In this regard, providing a continuous interactive guide and support for the users to complete their daily activities is crucial.

The remainder of this paper is organized as follows: Sect. 2 highlights some of the notable works in the field of smart homes and AmI; Sect. 3 delves into the detailed architecture of the HoloHome framework; Sect. 4 depicts two use cases in which HoloHome features are illustrated; and finally the Conclusion summarizes the main outcomes of this paper in addition to the future works.

2 Related Works

Although there have been many studies concerning the AR in general, little study has been conducted in regards to the exploitation of AR in the field of smart homes. AR was first been reviewed by Azuma in 1997 [1] alongside with describing the characteristics of AR systems and its limitations. Ever since that date, there has been vast amount of research on AR within different fields. With release of Microsoft HoloLens smart glasses in 2016, many researchers started exploiting HoloLens in several contexts. Evans et al. [12] described a thorough evaluation on the HoloLens functionalities on the guided assembly instruction. Recently, Kučera et al. [13] depicted the modality of the connection between the AR applications and the smart devices through microcontrollers. There exists variety of research works related to the Smart Homes and simulation of the domestic environment; however, little study has been conducted in regards to the managing the smart home through AR. In the field of Smart Home, Mahroo et al. [14] investigated the possibility of deploying a Smart Home and enabling a CA system, exploiting the protocol of Internet of Things (IoT) and semantic web technologies. In another study [15] smart home and architecture of connected ubiquitous devices for increasing the customized comfort metrics has been discussed.

3 The Smart Home Architecture of FHfFC

This section describes the architecture of the HoloHome, which enables the users to interact with the smart home. As mentioned in Sect. 1, the possibility to interact with the home appliances and to receive visual feedback from them would bring a huge benefit in the context of AAL environment. In this regard, this work proposes an AR application installed on the Microsoft Hololens smart glasses that is capable of interacting with real objects within the physical environment. HoloLens is able to keep track of the spatial environment around the Hololens in order to understand the real world position of each object. Spatial mapping provides a mesh representation of the real objects surfaces within the environment, allowing developers to implement more realistic AR applications [12]. Knowing the spatial mapping of the domestic environment, the HoloHome is able to seamlessly align the virtual objects to real world objects.

Although the proposed AR application must be flexible enough to adjust to different positioning of the real objects within the environment, it also needs to be rigid enough to have the virtual objects always anchored to the real appliances as well.

HoloHome must be capable of positioning each hologram – virtual object made by Hololens – on the corresponding real-world object with the exact same positioning and precise scale alignment to create the ultimate realistic feelings for the users. In this regard, HoloHome exploits the Vuforia’s image processing library which allows HoloLens to track a target image from the front-facing camera’s coordinate system. In this way, the HoloHome is able to find the accurate location of each appliance within the real world environment and instantiate the virtual version of that object in a way that both real and virtual objects align on each other perfectly.

Moreover, the HoloHome leverages the ability to control and regulate the home appliances exploiting IoT system to bring the new way of interaction between the users and the smart home platform (Fig. 1).

Fig. 1.
figure 1

Conceptual model of different technologies involved in HoloHome & their interactions.

Full size image

In the following subsections, the technical details of the development phase of AR framework are discussed.

3.1 Physical Infrastructure

In this work, the “house of the future” is simulated within the Living Lab of Lecco as part of the national project FHfFC. The lab environment comprises kitchen major appliances, while their functionalities are simulated via AR. The kitchen environment consists of a refrigerator, a washing machine, a dishwasher, a sink, a pantry, few cabinets, and a desk. In addition to the home appliances that are needed in this project lab, the physical infrastructure of this smart house also encompasses a network of interconnected and interrelated smart devices – sensors and actuators – to enable the transmission and exchange of data over the internet exploiting IoT. In order to allow the inhabitants to interact with the “house of the future”, a pair of AR glasses is needed to be used in this environment. One of the best AR devices on the market is the Microsoft HoloLens which is chosen to use in this project. HoloLens has a pair of translucent screens for its eye-pieces that allows the injection of the holograms into the user’s line of sight without completely blocking the user off the world.

HoloLens uses an accelerometer (to measure the speed of the head while moving), a gyroscope (to measure the tilt and orientation of the head), and a magnetometer (to function as a compass) to provide an ideal AR experience. HoloLens is also equipped with depth sensing camera and a standard camera to collect data about the surrounding area in order to determine where surrounding physical objects are located and to draw a digital picture of the surroundings.

3.2 AR Application on HoloLens

The HoloHome is implemented with Unity 3d software [16] for the Universal Windows Platform (UWP). The application exploits the Microsoft Mixed Reality Toolkit to handle the AR environment, and Vuforia SDK [17] to augment the holograms on the target images registered in the Vuforia portal (Vuforia is described further in the Sect. 3.3). Although it is important to be able to move the real object and re-instantiate the corresponding virtual object alongside with that – as it may happen in real life scenarios to redecorate the house and change the place of home appliances – it is also not convenient to locate the objects with Vuforia each time the user launches the application. As a result, HoloHome has been developed in such way to save the real-world location of the objects – after image target is being detected by Vuforia – and load the previously anchored positions of the objects next time the user launches the application. In order to meet both requirements simultaneously, HoloHome adopted a particular approach which foresees two different options at the very first scene when the application is launched; asking the user if she/he would like to locate the home appliances via Vuforia target image tracker, or she/he would like to restore the location of the objects as it was anchored and saved in the previous session. If the user chooses the option to turn on the Vuforia and locate the objects, HoloHome application would start with no virtual object, waiting for the image targets to be detected by HoloLens front camera to locate the virtual objects on the image target. However, if the user chooses the other option to turn off the Vuforia and restore the last object positions as in the last session, HoloHome would turn off the HoloLens front camera – which makes the Hololens camera free for other uses – and load the latest locations of the objects anchored the most recent time HoloHome had run (Fig. 2).

Fig. 2.
figure 2

The overall architecture of HoloHome and its interaction with Vuforia engine.

Full size image

3.3 Vuforia Image Processing Engine

Vuforia is the most widely used platform to develop AR applications with robust tracking and performance. Vuforia engine is a marker-based AR system in which the markers are the images or 3D objects registered with the application – via the Vuforia library – that act as information triggers in your application.

When HoloLens camera detects any of the registered markers within the real environment (while running HoloHome), this prompts the display of virtual objects over the real world position of the marker in the camera view. Marker-based tracking can use a variety of different marker types, including QR codes, physical reflective markers, Image Targets, and 2D tags. In this work, we use Image Target which is the simplest and most common type of markers. There exist few images uploaded on the Vuforia image processing portal, in which each of them is associated with a specific kitchen appliance. When the user launches HoloHome via the HoloLens, she/he can look for the image target in the real environment allowing the Hololens camera to recognize the marker and augment the virtual object associated with that particular image.

Another advantage that Vuforia would bring to the application is the fact that any real-world object’s position becomes dependent on the tracked image target. Not only does this approach improve the hologram stability, but also it allows the user to relocate the real object and be able to easily re-instantiate the hologram as well.

3.4 Arduino Interaction with AR Platform

In order to develop the idea of the smart home and regulating the heterogeneous devices within the domestic environment through an automatic remote system, the need for deploying an IoT network is crucial. The “house of future” forms a network of WiFi-enabled devices – preferably in a star typology where there is one central hub and several nodes or devices connected to the hub which makes it easier to add or remove any device without affecting the rest of the network.

In order to allow the interaction between the home appliances and smart devices through the AR platform, a solid connection between the smart devices network and the HoloHome application is required. In this network, each device is capable of both sending data captured from the environment to HoloHome application, and receiving the prompt action from the application to actuate the device. As a result, in this project Arduino [18] microcontroller and Arduino WiFi shield [19] – for Serial to WiFi data transmission – have been employed. Arduino microcontroller is sketched to turn on or turn off the connected light, whenever it receives the related prompt data coming from HoloHome. However, HoloHome sends this trigger to the Arduino when the user clicks the proper virtual button. In this way, the “house of future” provides its inhabitants with the possibility of manipulating the real devices within the domestic environment using the AR platform.

4 Use Case Scenario

The HoloHome is designed and implemented to provide continuous help and support for the smart house inhabitants for performing ADLs. Combining various technological paradigms, HoloHome enhances user interaction with the smart home platform. Considering the complexity of the system, it is important to define some use cases to illustrate the interaction between the dwellers and the smart house AR platform; to demonstrate how HoloHome boosts the user experience while interacting with their smart house; and finally to demonstrate how HoloHome help the dwellers in performing their ADLs in a more convenient and independent way. In the following use cases, a middle-aged male user is considered, who is 58 years old and suffers from mild executive function deficit [20], which causes him difficulties in performing sequential and executive tasks.

4.1 User Turning on/off the Light

The first use case depicts how the inhabitant of the “house of future” is able to interact with the house devices in general and with the lights in particular. HoloHome is able to create a virtual version of the home appliances and overlay the virtual object over the real object. The HoloHome application then provides the dwellers, visual step by step instruction and feedback over each appliance object. Each time a dweller launches the HoloHome application and has the virtual objects located in place – either by reading the image targets or by restoring the previous locations –, HoloHome is ready to provide help and support regarding each activity he is gazing at. When the dweller looks around the room wearing the HoloLens, each time he stops moving his head and looks at each specific device – his eye gaze collides with the hologram –, the virtual images, animations, sound, or any necessary information appears on that particular object to help him regulate the device.

4.2 User Looking for the Specific Object with the Help of HoloHome

The second use case demonstrates how the HoloHome supports the dweller to find different objects he needs. This can be done by providing him with visual hints to find the place of the ingredients he needs for his meal recipe, or the particular appliance and/or its functionality that he is looking for. Considering the fact that people with mild cognitive impairments tend to suffer from amnesia and face difficulties remembering the order of execution sequences, HoloHome offers features and functionalities to provide some hints and clues in a form of hologram graphics, to guide the user in finding the anticipated object faster and easier. Assume the user would like to turn on the washing machine with specific settings. He has already located the washing machine using Vuforia so the HoloHome knows the exact location of the washing machine, its components, and the control panel. When he looks at the washing machine, a virtual panel appears on the washing machine which aligns on the exact same position as the real washing machine’s control panel and its buttons. Each button or setting will be lit with different colors in a sequential order to avoid the user’s confusion by facing too many buttons at the same time. HoloHome also provides the explanation in the form of virtual text or voice command on each setting. When he sets the first setting by clicking on the virtual button, the first setting’s hologram disappears and the second setting that needs to be done would be lit with another color.

5 Conclusion and Future Works

This work introduces HoloHome, an AR application deployed on the HoloLens to manage and regulate the “house of the future”. Exploiting the protocol of IoT, AAL, and AR technologies, this framework aims to provide continuous guide and support for the inhabitants to complete their ADLs. The AR framework investigates the possibility of aligning virtual objects over the real objects to offer specific services for each domestic devices in AR form. Moreover, HoloHome is able to communicate with real devices and regulate them through WiFi-enabled Arduino microcontrollers.

Future works foresee the deployment of the domestic appliances within the “house of the future”, and connecting all the appliances to the HoloHome application. It is also envisioned to validate the HoloHome framework in the near future to evaluate the usability of the application amongst diverse group of people.