Keywords

1 Introduction

The Internet of Things has the vision to pervade our daily environment. One of the main characteristics of such environments is the growing amount of smart devices and their complexity. Indeed, due to the weaknesses of existing user interfaces, when interacting with intelligent environments, users are overloaded by the multitude of networked devices [3]. Identifying and activating the right device or service from a huge amount of existing devices to perform a specific task is a challenge for users in these environments. Moreover, devices can disappear in the background and can be invisible to users requiring new forms of interaction [1]. Researchers are dealing with the design of interfaces enabling the interaction through different devices and in different forms depending on the characteristics of the ambient environment [13]. In [1], a systematic review on meta-interfaces to interact with the environment points out that the most relevant unexplored issues are related to the lack of spontaneous discoverability [6] of such interfaces and the lack of adequate feedback [5, 11]. When Artificial Intelligence (AI) takes the orchestration of Smart Environment devices, this results into Ambient Intelligence (AmI). AmI should satisfy human requests by piloting the devices in a context-aware manner [4]. The context, in this case, refers to both the environment and user circumstances. For example, a context could depend on smart devices availability, environmental conditions, and user position. Thus, the need to let AmI understand the user context and requirements demands for new interactive Human-Computer Interfaces (HCIs). The idea and architecture of an utterly tactile interface device that interacts with an AmI were already presented in [2]. The proposed device, named TactCube, promotes a tactile-only interface that could allow human interaction with AmI without visual or audio means. Furthermore, an effort was made to design a small wireless cube-shaped hardware device that could be used with one hand only and to be kept in a pocket. This work aims to investigate the modalities of admissible interactions by users with AmI through a tangible mobile interface. The TactCube device was initially introduced to users, and three smart devices (smart bulb, speaker and curtain) were selected to conduct a guessability study. Guessability studies have been widely used to define sets of touch or motion gestures on various types of devices equipped with multitouch screens, both large-screened devices and common smartphones [7, 9]. For each device, the study allowed us to investigate users’ preferences among the set of admissible TactCube interactions to be mapped onto a set of device behaviors governed by an AmI system. Finally, the analysis of the responses led us to define a set of design implications that designers should consider when developing tangible mobile interfaces for ambient intelligence systems.

2 Interactive Cube

The idea behind TactCube was to design a fully tactile interface, helping users to communicate with AmI avoiding video and audio media. For the sake of brevity, we do not report here all the details on the TactCube prototype implementation, for which we address the reader to [2], and instead focus on what is relevant for the present study. TactCube design aims to offer a non-intrusive tangible interface, which let user communicate with an AmI system his/her contextual needs through simple gestures, just holding the small device in one hand. A users may start the interaction holding the cube with the fingers, as in Fig. 1, and may perform the following 10 gestures: 1. a single tap of index finger on one of cube faces; 2. a double tap of index finger on one of cube face; 3. the forward rotation followed by a single tap; 4. the forward rotation followed by a double tap; 5. the backward rotation followed by a single tap; 6. the backward rotation followed by a double tap; 7. the right rotation (clockwise) followed by a single tap; 8. the right rotation followed by a double tap; 9. the left rotation (anticlockwise) followed by a single tap; 10. left rotation followed by a double tap.

Besides these ten gestures, five types of haptic feedback are also admissible, namely: 1. one short vibration; 2. two short vibrations; 3. a combination of short and long vibrations 4. a face of the device warming up; 5. a face of the device cooling down.

TactCube deploys WiFi connectivity with AmI and a Bluetooth Low Energy (BLE) solution that unveils the smart devices in proximity, helping the AmI system to recognize the user position inside the environment. TactCube communicates to the AmI contextual information about the users to pilot the smart devices and support their needs. As an example, if the user is turning the cube forward in a dark room, and AmI is adequately trained, by the context and the position of the user, it may infer that more light is needed and then switch the lamps on. If the same action was made in the proximity of a curtain and the light was already on, given the different context and user’s position, the AmI system may infer that the request was related to the desire to open the curtain, instead.

Fig. 1.
figure 1

The TactCube prototype and a user interacting with it.

Full size image

3 The User Study

The idea behind TactCube was to design a fully tactile interface, helping users to communicate with AmI avoiding TactCube provides users with multisensorial feedback involving touch, vibrations and temperature changing and a large number of motion and touch interactions to be mapped onto the commands. In our study, we investigate on the one hand which are users’ favorite TactCube interactions for precise commands and feedback in a smart environment and, on the other, if such preferences may depend on the kind of smart device under control. To do so, we set up a mixed-design experiment performed with 60 participants equally distributed intro 3 different groups. We considered 2 factors that are respectively: Factor 1 the 12 Tasks as a within-subject variable, and Factor 2 the 3 devices as a between-subject variable. Participants were selected from a group of 130 volunteers IT students who had passed one of the computer science exams dealing with the discipline of human computer interaction, at the authors’ university. Based on a short questionnaire, we were able to select the 60 students, who best grasped the goal of the study, all of them being technologically skilled enough to interact with a tangible device like TactCube without distorting the experimental results. The chosen number of subjects was suitable to get preliminary insights and in line with similar experiments present in the scientific literature [9, 10, 12]. We divided our subject participants into three subgroups, properly Gr1, Gr2 and Gr3. To each subgroup we assigned a different device to interact with using TactCube – precisely a smart bulb, a smart speaker and a smart curtain, which are common everyday devices. Table 1 shows 20 guessability tasks for each device, including 12 navigation/action items and 8 feedback items. The tasks are split into three categories, specifically, Action, Navigation and Feedback. The items of each category are distributed into two subcategories: AmI Environmental system and Smart Device. Then, we assign a label to each item depending on its category and subcategory, precisely EA and DA for Action, EN and DN for Navigation, EF and DF for Feedback. The categories are adapted from the work presented in [10] which considers the generic tasks in a smartphone system environment. Apart from the categories Action and Navigation, we also added the category Feedback originally missing in [10] but necessary, as recommended also in [10], considering that also the feedback must be mapped correctly in the system to be interpreted by users. For each task, the participants were required to propose a specific gesture or feedback to associate to the prototype chosen from the 10 basic available gestures and the 5 kinds of feedback. The experiment took place in a controlled environment, that is the usability research lab of the University of XXXX, the participants were scheduled all along three days and only one participant at a time could access the laboratory, also due to Covid-19 restrictions. A workstation was set up for the participants, then, they were asked to sit and hold the TactCube device and interact with it. When the experiment session started, an application running in a computer desktop showed for each task the specific device behavior and asked for an interaction preference. Figure 1 shows a user during the experiment. To better evaluate the potential and rationale behind the specific interaction choices, a “think-aloud” technique was used. Moreover, users were photographed and recorded, in order to allow the researchers a systematic analysis of the experiment. At the end of the experiment, we asked participants to discuss any possible suggestion or issue. Each session lasted on average 30 min. During it, the authors took notes to discuss at the end of the session and of the whole experiment.

Table 1. The guessability tasks for the smart bulb, speaker and curtain.
Full size table

4 Analysing the Results

In the experiment, we involved 60 participants divided into three distinct groups. We asked each user to propose one interaction on the prototype for 12 tasks and select one kind of feedback from those eligible for further 8 tasks. Each group was also assigned to a distinct scenario in which to execute the tasks with a different smart device, specifically a bulb, a speaker, and a curtain. To evaluate the degree of consensus among participants we adopted the process of calculating an agreement score for each task [12, 13]. An agreement score, At, reflects in a single number the degree of consensus among participants. Wobbrock [12] provides a mathematical calculation for agreement, where: \({A}_{t}={\sum }_{{P}_{i}}{\left(\left|\frac{{P}_{i}}{{P}_{t}}\right|\right)}^{2}\). In equation, t is a task in the set of all tasks T, Pt is the set of proposed gestures for t, and Pi is a subset of identical gestures from Pt. The range for A is [0, 1]. Table 2 shows the agreement score reached by groups for each action and in relation to the specific smart device.

Table 2. Agreement scores for each item and group
Full size table

A consideration that can already be made is that in all the proposed tasks a minimum consensus has always been found by all the groups, indeed the agreement score is always greater than 0.1. Anyway, five tasks related to Actions and Navigation achieved the highest degrees of consensus, which are EA1, DA1, DA2, DA3, EN1 and DN6 in Table 2. Indeed, their degrees of consensus are higher than 0.4. This means that in a group, more than 40% of the proposals were for a given interaction on average. On the other side, the agreement scores for the 8 feedback items are generally higher. In particular, EF1, DF1, DF3, DF4, DF5, DF6 and DF7 are higher or close to 0.5 as shown in Table 2. Figure 2 shows a graphical comparison of the choices made by the three groups for each task in the Action and Navigation categories. Regarding the first category, EA1 is about taking action to access an intelligent environment. In this case, the proposals by users of all groups focused almost exclusively on two actions, single tap and double tap with a small predominance for the latter. In particular, double tap was selected by all the users 37 times and single tap was selected 20 times. This indicates an almost total consensus among the three groups for actions that involve only the tap, excluding actions that involve rotations of the cube. DA1 concerns the switching on of a smart device. Gr1 and Gr2 had a similar behavior proposing mostly double tap and single tap. On the other hand, although Gr3 participants proposed a certain number of both double and single tap, they slightly preferred the right rotation followed by a single tap. Similarly, for DA2, which is about turning off a smart device, double and single tap are generally predominant preferences, however, Gr3 slightly preferred left rotation followed by a single tap. Some of the users across all groups explained their interaction preference commenting it. Many justified the tap or double tap because they suggest the action of turning an electronic device on using a conventional but- ton, however it should be noted that double tap has often been the predominant choice. A participant from Gr1 explained: “I chose the double tap because the tap reminds me of the click on the light button, but the double tap gives me the certainty of not pressing it by mistake”. Some participants from Gr3 chose right or left rotation because this recalls the gesture of turning a crank with which to open or close a shutter or a Venetian blind. This suggests that in order to turn on and off a smart device both tap, and double tap are interactions generally understandable by all the users regardless of the type of device, but certain devices can also suggest a more specific interaction connected to their particular affordance, as in the case of the smart curtain. DA3 is related to the command stop, play or act on a program of a smart device. As in the previous cases, there is a predominance in the use of tap, although this time among the three groups the most common choice was the single tap. However, the predominant choice relative to Gr1 is the forward rotation followed by a single tap. In this case, one participant explained that the rotation suggests the action of starting the car rotating the key, and another that it is linked to the idea of a two-way switch. Regarding the Navigation category, only EN1 and DN6 showed significant agreement where recurrent interaction patterns can be appreciated. As regards to the EN1 item, for all the groups, the two most proposed actions are right rotation to the right followed by a single tap, and double tap. The rotation was proposed 34 times and the double tap 21 times. Then, both the rotation and the double tap are interactions considered understandable by the three groups regardless of the target smart device. In choosing this gesture, some users imagined moving through a wheel control on a menu to select the desired device. Participants who used the double tap or the tap stated that they were imagining choosing the device with a “click” being close to it. Finally, in DN6, related to the programmed shutdown, the combination of backward rotation and double tap is the main preference, chosen 41 times by all.

Fig. 2.
figure 2

Comparison of preferences in the Action and Navigation categories.

Full size image

Concerning the tasks involving the preference of the type of feedback, the degree of consensus among the participants of the different groups is particularly high and homogeneous among the 3 groups. Figure 3 shows a comparison of the users’ preferences for each item in the Feedback category. More specifically, DF7, which is related to the feedback of a device error, received a very high level of consensus among all groups on a combination of short and long vibrations. Indeed, 54 users indicated a sequence of vibrations as it suggests “a shock”, “an impact” or “a movement that cannot be completed”. Regarding the feedback of a successful action (DF6) the participants of the three groups, equally distributed, chose 31 times a short vibration and 29 times two short vibrations. Users explained their choice by associating it with the behavior of their phone. When a task ends, the phone shortly vibrates one or more times or makes a short beep. DF5 is the feedback status “loading”. In this case, the choices of the participants were polarized on the face of the cube warming up. One participant said: “In an electronic device the heating is often a clue that it is working”. Another added: “an object that is working may heat up slightly”. On the other hand, regarding DF4, which is related to a device switching off, a majority of 48 participants proposed the cube face cooling down. Then, a smart device cooling down is perceived by the users as a device that stops working. Similar considerations can be made for DF3, which is the feedback related to the device switching on. The participants’ preferences are distributed almost halfway between a short vibration and the face of the cube warming up. Also, in this case the warming suggests to users the idea of a device that gives off heat because it works, while the vibration suggests the correctness of an action or a state, due to previous experiences with mobile phones. Finally, DF1 and EF1, which refer to the feedback connected to the spontaneous discoverability of the smart device and the intelligent ambient, polarize the preferences of all groups around one short vibration or two short vibrations. Moreover, to understand whether the user preferences for each task is directly linked to the device chosen for the scenario, we ran the Kruskal-Wallis one-way ANOVA by ranks [8] on the participants’ preferences expressed on each task. This analysis is a non-parametric test used to compare three or more sets of data and can be considered when the samples are independent. The association between two variables is statistically significant if the resulting P-value is less than 0.05. In our experiment, the test was executed in the IBM SPSS Statistics software. Except for DN4 (≈ 0.00), all the P-values calculated by the statistical method are higher than 0.05. Moreover, the DN4 item can be questionable, indeed the users’ preferences are variegated. This means that, even if many users’ comments suggest a link between the smart device form and the interaction preference, statistically no evidence proves it, and more investigation is needed to understand such potential links.

Fig. 3.
figure 3

Comparison of preferences in the Feedback category.

Full size image

5 Design Implications and Final Remarks

The guessability study presented here leads to implications that can be useful when designing tangible interfaces for devices interacting with the AmI.

  1. a)

    Design the interactions to be meaningful and consistent with everyday life experience of your users. Even if some preferences are suggested to users by the affordance of the controlled smart device, mainly the preferences are led by other factors such as previous experiences in other contexts such as mobile phones, or common buttons as light switches and the affordance of the device controller. An example is given by the results in the category Action: the interactions preferences for tasks such as turning on or off, and play can be related to the form factor of the smart device that suggests specific manipulations, although generally single or double taps are commonly understood by all users for this kind of tasks.

  2. b)

    Use the double tap without combination or the single tap if you are combining the tap with other gestures. In combination with rotation, the single tap is the gesture chosen more frequently, nevertheless, when the gesture is not combined, double-tapping can be preferred as it gives users the certainty of not performing it accidentally.

  3. c)

    Use rotations to navigate the device parameters and settings. In the Navigation category the main preferences are given to rotation even if in different directions and in different combinations with the tap, then it is not easy to find a recurrent pattern. Anyway, right rotation appears as a consistent preference for selecting a smart device in the environment, forward rotation can be indicated when increasing a device parameter, and backward rotation for programming shutdown time.

  4. d)

    Consider tactile feedback as strongly related to the real world. In our experiment, the types of possible feedback are transversal to the various smart devices and are linked to behaviors related to the real world. d1. As in mobile phones, a short vibration or two short vibrations express something positive, such as the correctness of an action, a functioning state of the device, or also the discovery of a close smart device. d2. Heat perception is connected to something running on the device, and therefore to the switching on of the device or to a loading state. d3. Cold suggests that the device is not working and therefore can be used to indicate its shutdown. d4. Combinations of long vibrations are interpreted by users as something problematic and thus adapt to warn users.

The above implications should be used to design interactions that are truly meaningful to users. However, this study has some limitations to consider. It was developed within a static controlled environment and involving a limited number of smart devices and users. To refine and confirm our outcomes, it will be necessary to introduce the TactCube in the wild through a field study involving a larger number of devices and different environments. Finally, in the present work we explored only the 10 basic gestures provided by the TactCube prototype, while we plan in the future to consider a memorability study to show if a larger set of gestures can benefit or not the user interaction experience with such device.