Keywords

1 Introduction

Technology advancements are generating devices that are increasingly more accessible and affordable than the previous generation, introducing new possibilities for designing interactive systems. In this ever-changing framework, designers are needed to develop interactive digital products to support the way people communicate and interact in their daily and working lives [1]. Since ubiquitous computing technology is becoming widespread [2, 3], designers should also have available tools and methods to design new interactive systems. Nowadays, the most significant challenge for designers is creating interfaces and experiences that could extend and enhance the real world. Augmented Reality (AR) technologies [4] have a great potential for the design and reinterpretation of interactive systems, with profound implications for use, aesthetics, and function. The potential of AR to shape the future of both design education [5] and product design [6] lies in enabling a new generation of interfaces and interactions.

AR integrates 3D virtual objects superimposed on the real environment in real-time, augmenting reality rather than replacing it completely [7]. Nowadays, the development of AR technologies is proceeding at a very fast pace [8] with an increasing interest in investigating the employ of AR technologies in different fields [9]. AR technology is also spreading into domains where advanced technologies are not traditionally used, such as household appliances.

By definition, a household appliance is a machine that assists users in household functions such as cooking. This paper aims to test the feasibility of introducing AR technology in this field. To this extent, the research started by identifying a case study through a survey of potential household appliances users.

Through semi-structured questionnaires, we surveyed a sample of 463 participants to figure out which household appliances could benefit from AR technologies. The results showed that the kitchen machine is one of the best candidates. A kitchen machine is a small appliance, generally used on a table, for cooking. This appliance was originally designed to mix and knead food, reducing time and manual effort. Kitchen machines evolved from the first hand-operated prototypes of mechanical mixers invented in the early 1900s to the nowadays models featured with electric motors, microprocessors, digital displays, and smartphone apps.

The form in which recipes are presented to users evolved in parallel with the household appliance. Starting from paper-based recipes, characterized by black-and-white 2D text and figures, nowadays recipes are in form of digital content such as web pages and video tutorials. The latest kitchen machines integrate a recipe book system that users can use to prepare dishes by exploiting digital content. The step-by-step recipe information is displayed via the small LCD integrated monitor in form of text and static images, but still preserving the issue of paper-based recipes. The issue with these recipe formats lies in the separation between the workspace in the kitchen and the information conveyed by the recipe [10].

AR technology aims to overcome this issue by conveying information about the dish creation process through virtual content such as CAD models, animations, videos, images, and text directly registered in the real world. Thus, AR provides a rapid spatial mapping between the recipe information and the task location to be performed.

We believe that applying AR technologies to kitchen machines, could improve user performance in terms of task completion time, error rate, and system usability.

Using a case study, we describe the application of the human-centred approach to integrating AR technologies to develop interactive systems for household appliances.

The remaining paper is structured in five sections. The first describes the state-of-the-art related to AR technologies applied to kitchen appliances. The second describes the human-centred design approach deployed with AR integration. The third describes the case study of an AR-enabled kitchen machine. The fourth describes the AR-enabled kitchen machine demo. Lastly, we report our conclusions and future works.

2 Related Work

In the literature, there are few contributions in the field of AR technologies applied to kitchen appliances.

Kellogg et al. highlighted that kitchen is by nature a candidate for AR interfaces because there is a great need for users to stay in touch with physical reality while using a variety of sophisticated tools that benefit from digital information [11]. In addition, kitchen appliances are suitable for free-hand interaction for both hygiene and safety reasons. While cooking, users usually consult recipes touching other surfaces, and then handle ingredients. Sometimes, cooking can also be considered a dangerous task because of the tools that must be handled (e.g., sharp knives) while consulting recipes.

Gausemeier et al. [12] developed an AR-PDA system consisting of an AR-based user interface that allows users to receive information and configure appliances. The case study concerns the oven, augmented with a virtual telescoping cart. The prototype allows recognition and tracking of moving objects, scene augmentation using animated 3D objects or images, and personalized user interaction by touching the display with a pen or using Personal Digital Assistant (PDA) buttons. In addition, AR-PDA is used to prevent errors or malfunctions on home appliances by providing users with 3D assembly instructions and augmented real-world animations.

Mahroo et al. [13] implemented the HoloHome, an AR framework that aims to provide new means of interaction with the Smart Home and its components. The goal of the system is to provide a Mixed Reality (MR) environment implemented on Microsoft HoloLens, to enable user interaction with Smart Home devices and appliances through augmented objects. HoloHome addresses the problem of locating real-world objects and aligning virtual objects with real ones within the spatial environment, using the Vuforia image processing engine. Also, the HoloHome leverages the ability to control and adjust household appliances by leveraging the Internet of Things (IoT) system to bring a new way of interaction between users and the smart home platform.

Walczak et al. [14] presented several VR and AR systems for interactive visualization of household appliances. The goal is to support wholesale sales of a variety of products configured and then ordered in a manufacturer’s showroom and individual sales by allowing customers to view different versions of products in their home environment before making a purchase decision. The system also provides training and support for appliance maintenance. The visualization software, created using a game engine, leverages a wide range of visualization hardware, such as a smartphone, a Head-Mounted Display (HMD), and a Powerwall, to enable its use of indifferent settings. Users can configure and manipulate the components of the virtual appliance directly in a 3D space. Case studies cover, for example, an immersive service guide for an induction cooktop and an interactive animated virtual oven, displaying features that are difficult to show in the real world without AR.

Ergün et al. [15] developed AR-SI, an induction cooking appliance to augment food preparation support experiences, with a future perspective of replacing people with robotic servers. The induction technology used integrates IoT and a supported virtual assistant. To assist the user in interacting with food and induction devices, AR-SI includes a virtual assistant in AR that suggests to the user how to follow the recipe and cooking procedures step-by-step, considering weight control and presenting 3D visualization on the real stove and pot. They used a game engine to build and develop the AR application. The marker includes the 3D model of the stove, and they used a Scale-Invariant Feature Transform (SIFT) to track the physical model during the AR interaction. Beginner users in cookery art can quickly and accurately prepare even the most complicated meals with AR support. In addition, the induction cooking appliance can be useful for cooking school students by practicing with the AR stand.

As literature findings revealed, new emerging concepts in household appliance design are recently developing due to new technological advances such as those of the Industry 4.0 paradigm [16]. Some operations, such as cooking, have evolved with new trends that mix IoT, AR, Virtual Reality (VR), and even robotics [15].

3 Human-Centred Design Approach

The Human-Centred Design (HCD), as reported in the ISO 13407:1999, is an approach to interactive systems development that aims to make systems usable and useful by focusing on the users, their needs, and requirements, and by applying human factors/ergonomics, and usability knowledge and techniques [17].

ISO 13407:1999 is an international standard that describes best practices in user-centered design, then withdrawn and expanded into ISO 9241-210:2010, later withdrawn and expanded into ISO 9241-210:2019. The standard guides the design activities that take place during the life cycle of interactive systems, ensuring that the development and use of interactive systems take user needs into account. During the process, many needs and expectations of users emerge having an impact on interaction design, as designers refine their understanding of users.

A correct HCD approach consists of the following iterative steps:

  1. 1.

    Identification of the context of the use, and research on users.

  2. 2.

    Definition of user needs and requirements, and system requirements.

  3. 3.

    Development of design solutions that can meet the needs and requirements.

  4. 4.

    Evaluation of the design solution against requirements.

In this paper, AR is embedded in interactive systems with the HCD approach to promote a higher level of innovation, reducing related problems, and increasing interactivity and ease of use. Through AR, virtual graphics can transform the physical world, guiding and assisting users to use the system properly. Thus, the adoption of AR technology becomes a powerful tool to design interactive systems that are more and more user-friendly and accessible, increasing user knowledge.

3.1 Context of Use and User Research

Understanding and specifying the context of use includes an extensive discussion of the environments of use in which the interactive system will be deployed. In addition, it is necessary to know the users and the tasks for which they will use the system. Users’ information is collected and analyzed through a proper user analysis process that allows collecting user needs and preferences. User data can be gathered using qualitative or quantitative methods (e.g., interviews, surveys or questionnaires, field studies, focus groups, etc.).

3.2 User Needs, User Requirements, and System Requirements

The HCD approach first identifies User Needs, then formalizes them into measurable User Requirements, and finally maps them into System Requirements.

User Needs (UN) are identified through user research, based on behavioral analysis of users in their natural environment, and within the reference context.

User Requirements (UR) concern the set of user-generated statements that describe the functions, performance, and capabilities that the system will bring to them as they use it.

System Requirements (SR) are a statement of what the system must do and/or be to meet the user’s requirements.

Exploiting the HCD approach in the development of interactive systems, with the integration of AR technology, could solve real-world problems, guide the user in its use, and improve information visualization and user experience [18].

3.3 Design

Producing design solutions starts with designing personas, scenarios, and story-boarding useful for clearly visualizing the final interactive system.

Personas represent user needs by creating fictional individuals to describe the typical user based on the user profile. Their purpose is to represent a group of end-users during design discussions and keep everyone focused on the same goal. Personas are usually defined by identity and photo, status, goals and tasks, skill sets, requirements and expectations, and relationships [19].

Scenarios present detailed realistic examples of how users may perform their tasks in a context. The main purpose is to provide examples of future use to understand and clarify user requirements and improve a basis for subsequent usability testing. The scenario represents a story that describes how a particular persona completes a task or behaves in any given situation. The scenario is usually defined by the settings, actors, objectives, sequence of events, and outcomes [19].

Storyboards are drawn tables that tell the scenarios related to the target user demonstrating the relationship between user inputs and system outputs. A storyboard is a powerful means for the designer because, by telling a story about (parts of) the interaction(s), it allows the user to gain access to ideas expressed on two levels, communication, and experience [20].

The next phase of the HCD approach is the deployment of the prototype, which can be as simple as a sketch or static mock-up or as complicated as a fully functional interactive system with full functionality.

3.4 Evaluation

Design evaluation starts with assessing the design against the requirements based on users’ perspectives. This evaluation step can be used to:

  • Gather new information about User Needs.

  • Provide feedback on the strengths and weaknesses of the design solution from the user’s perspective (to improve the design).

  • Evaluate whether User and System Requirements have been achieved.

  • Establish baselines or make comparisons between designs.

Two widely used approaches to human-centred evaluation are (1) user-based testing and (2) inspection-based evaluation using usability and accessibility guidelines or requirements. The interactive system will be implemented at the end of this iterative process.

4 AR-Enabled Kitchen Machine Case Study

In this case study, following the HCD approach, we design and implement a user interface to guide the user in executing AR-guided recipes. We describe it in the following sections.

4.1 User Research

We analyze the context of use by investigating the kitchen machine’s role within the kitchen environment. Thus, we create a questionnaire using Google Forms distributed to a sample of potential users for 8 days (n = 463; 95% female and 5% male). The questionnaire provides quantitative data about user behavior and habits regarding cooking activity through 30 open and closed questions using the 7 point-Likert Scale.

As an interesting result, the user analysis demonstrates that most of the users own a kitchen machine (64%). Also, many users like to cook (mean = 6 ± 1,9 on a 7 point-Likert Scale), appreciating their kitchen machine.

Users’ level of familiarity with AR technologies is quite high (mean = 5 ± 0,7 on a 7 point-Likert Scale). In addition, 58% of users would be willing to wear an HMD if it is “light and non-invasive” to increase the kitchen machine potential.

Several issues emerge from the questionnaire and are later converted into user needs.

4.2 User Needs, User Requirements, and System Requirements

After user research, we define User Needs (see Table 1), User Requirements (see Table 2), and System Requirements (see Table 3), which are fundamental to setting goals for the interactive system.

Table 1. User needs
Full size table
Table 2. User requirements
Full size table
Table 3. System requirements. Priority: (S) the system should have; (M) the system must have
Full size table

4.3 Design

Following the HCD approach, we use personas to identify target users and simulate behaviors and goals to design the interface [21].

Elisabeth is the primary persona, a woman who agrees to use the AR-enabled kitchen machine: a modern woman, successful in life and family, mother of three children, an elementary school teacher. She loves to cook but she is always very busy with her job. She owns a kitchen machine, but she has tried several times to use it, without getting great results, preferring traditional recipes made manually.

In this scenario, AR can help Elizabeth in the preparation of the dishes, thanks to the step-by-step guidance of a virtual assistant. The scenario involves cooking a dish using the AR-enabled kitchen machine. After wearing an HMD (AR glasses), Elizabeth turns on her kitchen machine. Through an AR interface, the Graphical User Interface (GUI) shows three preset programs: “Empty the fridge” (system visualizes the ingredients in the fridge and suggests a recipe), “Let me inspire you” (system suggests a recipe), and “Search” (a recipe). She selects the preset program “Empty the fridge” and the only two ingredients available are zucchini and eggs. Then, the humanoid avatar suggests some recipes with these two ingredients, and she selects the zucchini omelet. The AR interface integrates a humanoid avatar that suggests step-by-step the preparation of the dish through the augmentation of 3D virtual information spatially registered in the real kitchen environment (see Fig. 1).

Fig. 1.
figure 1

The case study storyboard and its interactions: (a) Turn on the kitchen machine; (b) Select the preset program (Empty the fridge); (c) Discover the ingredients available in the fridge; (d) Choose a recipe based on the ingredients (zucchini omelet); (e, f) Perform the operations with animations and auxiliary lines; (g) Mix the ingredients; (h) Dish preparation finished.

Full size image

5 AR-Enabled Kitchen Machine Demo

The demo is developed to be experienced using both an AR HMD (HoloLens 2) and a Hand-Held Display (Android tablet) with a table stand. The application software is implemented in Unity 3D engine leveraging the Vuforia AR Software Development Kit for the tracking capability of the system.

Exploiting Vuforia Virtual Buttons we implement a Tangible User Interface (TUI). Virtual Buttons provide a useful mechanism for making the target interactive. Once the target image is occluded with the real ingredient, the Virtual Buttons generate a canvas that contains information about that ingredient (see Figs. 2 and 3).

In order to achieve visual coherence between the kitchen machine and the virtual information shown to the user, the kitchen machine is continuously tracked.

The kitchen tools (see Fig. 4) are tracked with a marker that augments virtual information useful in preparing and handling ingredients.

Fig. 2.
figure 2

a) Kitchen machine tracked and switched on; b) Preset programs

Full size image
Fig. 3.
figure 3

a) Empty the Fridge preset program; b) Avatar suggests recipes based on ingredients

Full size image
Fig. 4.
figure 4

Ingredients handling with the aid of virtual tools: a) Animations; b) Auxiliary lines

Full size image

The AR interface includes a humanoid avatar (embodied conversational agent) that aims to leverage some properties of human communication to make the interface more effective (see Fig. 5). The objective of the avatar is to create a feeling of “company” in the physical environment while guiding the user in the recipe execution. Also, virtual assistants may provide better learning of cooking procedures [15].

Fig. 5.
figure 5

A) Ingredients mixing operation; b) Dish preparation finished and celebration

Full size image

The interface design is highly spatial, dynamic, and playful, with bright colors, particle systems, 3D animations, sound effects, and voice commands. Spatial gestures and virtual floating icons allow the user to interact with the machine.

6 Conclusions and Future Works

Designers need to devise interactive systems to support the way people live and interact due to technological advancements. From this perspective, the ever-widespread AR technology can become a tool for designers. One of the domains that could benefit from AR is household appliances, namely kitchen machines.

In this paper, we use an HCD approach to develop new interactive systems with AR technology integration. Furthermore, we present a case study following the HCD approach that relates to the development of an AR-enabled kitchen machine to guide the user in the cooking activity.

To complete the last step of HCD (i.e., evaluation), we planned a user study that aims to evaluate:

  • the user’s experience through a User Experience Questionnaire (UEQ).

  • the cognitive load related to the recipe preparation task, through the NASA Task Load Index (NASA-TLX).

  • the usability of the AR system, through the System Usability Scale (SUS).

  • the error rate during the recipe preparation.

Although the user study is still an ongoing phase, this paper demonstrates the feasibility of integrating AR applications in a specific domain (e.g., household appliances), exploiting the HCD approach.