A smart cooking device for assisting cognitively impaired users

Abstract

In our ageing world, a rising number of people suffer from cognitive deficit, which most of the time leads to a reduced autonomy. Even with their impaired capacities, these persons often stay at home or they go live with a relative. They then have to perform important daily tasks (such as cooking) using devices and appliances designed for healthy people, which do not take into consideration their cognitive impairment. Using these devices is risky and may lead to a tragedy (e.g. fire). A potential answer to this challenge is to provide automated systems, which perform tasks on behalf of the impaired user. However, clinical studies have shown that encouraging users to maintain their autonomy greatly help to preserve health, dignity, and motivation. Therefore, we present in this paper a new smart range prototype allowing monitoring and guiding a cognitively impaired user in the activity of preparing a meal. This new original prototype is capable of giving adapted prompting to the user in the completion of several recipes by exploiting load cells, heat sensors and electromagnetic contacts embedded in the range. Our system is also able to detect risky situations and is able of taking preventive actions accordingly. It includes a state-transition recognition algorithm incorporating a model of the main cognitive errors. Finally, we present several experiments with the prototype and a study conducted with the targeted users, with companies, public organisms and professionals.

1 Introduction

In our ageing society [59], the capacity of a person to prepare his own meal constitutes one of the most basic affirmations of autonomy [53]. Indeed, cooking is a very rewarding activity that gives fulfilling sensation of achievement, and which does not require much physical exertion. It is the best way for an adult of keeping a healthy diet, which helps reducing the risk of developing chronic health problems such as heart disease, diabetes and high blood pressure [18]. Cooking also promotes a good quality of life and it is a great way for seniors to maintain a sense of independence, to stay active and to socialize. In addition, one can eat better for a cheaper price by preparing his own meal. From that perspective, allowing older people staying at home and/or in a senior residence to cook is desirable for social, health and economic reasons. Preparing a meal is one of the oldest activities in the world. In short, it consists in following a recipe, which is a set of instructions that describes how to prepare a dish. It usually involves the following steps [56]:

• Fetching the raw ingredients (e.g. from the refrigerator).

• Preparing the raw ingredients for proper use (e.g. peeling or cutting).

• Mixing them adequately and in the right order.

• Using cooking devices (mixers, range, burner, microwave, etc.) to mix and cook the dish.

• Serving the dish on plate.

As we all know, cooking involves certain risks. The main one is fire. For instance, the U.S. National Fire Protection Association [46] reports that three (3) in ten (10) home fires start in the kitchen, more than any other room in the house. In most of the case, the risk of fire seems to be mostly inherent in the use of the range, cooktop and the oven. As we can see in Fig. 1, the NPFA also states that human factor (e.g. equipment unattended) is the main cause.

Fig. 1

Leading factors in home cooking fires [46]

In the specific case of elders using the equipment, the risks are increased. As an example, the Federal Emergency Management Agency (FEMA) reports that people over the age of 65 have a 2.5-times greater risk of dying in a kitchen fire than the general population. We can multiply these risks by 10 or 20 when seniors suffer from mild or moderate cognitive impairment [37]. Despite of these important risks, most elders want to remain at home as long as possible, and still want to be able to cook and remain as independent as possible. Therefore, the challenge is how to give to seniors, even with mild cognitive troubles, the possibility to cook while minimizing the risks associated?

First, one needs to understand what the kind of troubles these persons experience. Seniors with cognitive impairments mostly suffer from weakening executive functions, sporadic losses of memory, and problems on focusing their attention on a specific task [10]. Therefore, a distraction (e.g. phone call, unfamiliar noise, etc.) or a memory lapse can lead a person to perform actions in the wrong order, to skip some steps of an activity, to perform actions that are not even related to the originally planned goal, or to completely forget what she was doing (e.g. forgetting something in the range’s oven). However, the person’s capacity to perform a simple action (without many steps) remains relatively unaffected [34]. Therefore, this situation requires supervision of the user and ad hoc interventions on the part of, for instance, a caregiver. When continued support is provided to such person, in the form of cognitive aide, a cognitively impaired user can be able to perform his activity (such as cooking). In addition, keeping the person cognitively active contributes to slow the degeneration process of the disease (e.g. Alzheimer) [11].

At the light of these issues, a lot of scientists in the field [3, 12, 44, 58] consider that smart technology presents itself as a viable avenue of solution, carrying a lot of hopes to help this kind of people performing their ADL safely. One potential approach to solve this issue is to develop efficient automated systems, which perform tasks on behalf of the resident. However, clinical studies have shown that encouraging users to maintain a certain level of autonomy greatly helps to preserve health, dignity and motivation [16]. In that sense, automated systems had the inconvenience of entirely removing the autonomy of the user. An alternative approach consists in developing assistive systems (instead of automated systems), which are able to track an activity of a cognitively impaired user to identify his erroneous or risky actions, and which are able to give adequate prompts (hints, suggestions or reminders), thus, increasing the probability of a desired behavioral outcome [19]. The aim of these systems is to provide appropriate guidance to the user to allow him to complete, by himself, his ADL safely.

In this paper, we present such a new assistive system, which takes the form of a smart range prototype [15] allowing to monitor the cooking activity of a cognitively impaired user and to give adapted guidance [36] in the completion of a recipe. Our system is also able to detect risky situations (e.g. a dangerous state that may lead to fire) and is able to take preventive actions accordingly. The originality of the device is to combine, in real time, the inputs coming from load cells, heat sensors and electromagnetic contacts embedded in the range to infer the current state of an ongoing activity. The system also identifies the main types of errors characterizing cognitively impaired users [8, 55]. The artificial intelligence (AI) model of the prototype relies on a stochastic representation of each activity with a state-transition model [25], which is included in a knowledge base. We deposed a declaration of invention at our university for this device and we also obtained a provisional patent (2015) covering North America on this invention. We conducted several experiments with the prototype, giving promising results that will be presented in this paper, showing the interest of this device. We also conducted, with the help of a valorization firm, an exhaustive study with the targeted users, with the companies, the public organisms and the professionals. For this study, 42 individual interviews have been conducted and 148 companies and organisms have been directly consulted. The highlights of the result of this huge study is also presented in this paper.

The remaining of the paper is structured has follows. Section 2 summarizes the concept of cognitive errors and describes the different categories that exist according to studies. This section will help the reader understand the specific kind of behavioral problems related to cognitively impaired users and to see how we can address them with an assistive device. Section 3 presents the recent works related to similar assistive systems dedicated to population with cognitive deficits. In this section, we specifically present systems developed for helping people cooking and related systems dedicated to other kinds of activities. Section 4 presents in details our new prototype of smart device for assisting cognitively impaired users in performing their cooking activities. This section also covers the implementation of the systems (hardware and software), the architecture of the software assistive agent, the tracking algorithms for objects based on load cells, the passive and active monitoring of recipes, the model of artificial intelligence used for detecting errors, etc. Section 5 then presents experimental results of a series of tests performed on the prototype to validate its functionality and its efficiency. The whole system and several subsystems have been tested separately. Section 6 presents the highlights of the results of the exhaustive study we conducted with the targeted users, with companies, with public organisms and with professionals in the field. These results allow seeing the market potential for the device, to understand better the users’ needs, to envision how to deploy the device in real context, etc. The study also allows showing clearly the path for future works. Finally, Sect. 7 concludes the paper. In this section, we summarize our contribution, we discuss the limitations and the advantages of our new devices and we, of course, propose several avenues to investigate in future works.

2 Categories of cognitive errors

The behavior of persons suffering from cognitive deficits is characterized by the presence of sporadic incoherencies in their actions, which appear when they perform complex activities involving cognitive skills [47]. Indeed, people without such impairment can act incoherently too. The main difference is that a healthy person is, most of the time, able to recognize their behavioral errors and to correct them by themselves. Moreover, healthy people do not act incoherently on a regular basis. In contrast, a person suffering, as an example, from Alzheimer’s disease, will certainly act incoherently, even while performing familiar tasks, and his behavior will become increasingly incoherent as the disease evolves. In the literature, the notion of cognitive disorders and their impacts on the performance of ADL has been well studied [34]. Many categorizations exist to describe the kind of cognitive errors. However, all these categorization systems are similar and they cover most of types of cognitive errors. For instance, the Naturalistic Action Test (NAT) [55] is a standard assessment test proposing a categorization system based on object-oriented behavior. Another example is the Kitchen Task Assessment (KTA) [8], which is a well-known functional measure used by therapists to record the level of cognitive support required by a person to complete a cooking task successfully. The KTA proposes a categorization system divided in six types of errors:

• 1. Initiation Initiation errors happen when the person is, for any reason, unable to begin their task. For example, if a therapist indicates to an Alzheimer’s patient that he must take his medication right now, the patient may answer “OK, I’m going to take it now” but he might still do nothing. Literally, the person is unable to “initiate” the task because he does not know how (he forgot).

• 2. Organization Organization errors happen when the person performs some steps of an activity in an inappropriate way. For instance, the person can use the wrong type of spoon, or even a knife, to mix up the ingredients of a recipe.

• 3. Execution Execution errors happen when a person has a distraction (cognitive surcharge) or a memory lapse, which leads them to perform actions that have nothing to do with their original goal, or to skip some steps of their activity. For example, a user can put a bowl of soup in the microwave oven to heat it while forgetting to start the microwave and, a few minutes later, eat the soup thinking that it is hot while it is still cold.

• 4. Sequence Sequence errors correspond to some disorganization in the course of the activity’s steps. For instance, the user may try to change the television channel without having turned it on beforehand. It is literally a problem in the sequence of actions.

• 5. Judgment Judgment errors happen when the person performs a task in an unsafe way, like manipulating a hot frying pan without wearing gloves.

• 6. Completion Completion errors happen when the person is unable to finish their task, because they stop in the middle of it, or because they indefinitely repeats one or more steps of the task. For instance, a user may want to open a kitchen cupboard to take a can of soup but, instead, may begin to repeat the action of opening and closing the cupboard for an indefinite period of time.

This classification system aims to cover all types of common errors characteristic of persons suffering from cognitive impairments. In this paper, we use the KTA and the NAT classification system as the foundation for our artificial intelligence agent in charge of monitoring cognitive errors and providing adequate assistance.

3 Related work

In the last several years, many research teams proposed new assistive systems aiming to help disabled persons performing their everyday tasks. The literature on the subject is vast and varied [3, 12, 24, 44, 58], but it can be divided in two streams. The first stream concerns work on assistive systems for helping specifically cognitively impaired users performing ADLs [1, 2, 20, 21, 21, 32, 49], but in various context, not necessarily with cooking tasks. The second stream concerns work that have been done directly on smart cooking assistants [4, 41, 45, 54], but for various different kinds of targeted users (with or without cognitive deficits).

3.1 Assistive systems for cognitively impaired users

Many works have been done in the last 2 decades in the field of Ambient Assisted Living (AAL) regarding, in particular, assistive systems dedicated to cognitively impaired users [38]. We cannot review all of them, of course, but we will take a look at the most important ones and the ones closest to our proposal. Two of the earliest systems that can be found in the literature are the well-known Autominder prototype [50] and the Independent LifeStyle Assistant (I.L.S.A.) system [32]. These systems are considered by the community as the foundation of modern assistive devices for persons with cognitive impairments, and still serve as a template for today modern prototypes. We will begin by reviewing both.

The Autominder system [50] is a reference in the field. It has been developed at Cornell University by the renowned Martha E. Pollack [49, 50]. This prototype aims to provide reminders to a user for ADLs completion using three key components: a plan manager, a client modeler and a reminder module. The plans are modeled with a symbolic approach as disjunctive temporal problems (DTPs) [42] and the recognition system use a backtracking algorithm [27]. The reminder module reasons about inconsistencies between what the user is supposed to perform and what he is currently doing, and determines what reminders to issue through an iterative refinement process. Thus, the Autominder system is able to consider situations where the user performs multiple activities, thanks to multiple sensors installed, and to prompt reminders when some erroneous behaviors, mainly temporally related (wrong moment to perform an action), are detected. This system has been deployed in a prototype form on a mobile robot assistant to assist elderly individuals with mild cognitive and physical impairments, and to support nurses. Nevertheless, this system presents several constraints. For instance, it is complex and expensive to manually specify the rewrite rules and evaluation function, because to accomplish the goal of personalization, they would have to be redesigned for each user. In addition, this prototype is limited in the fact that it does not distinguish the type of cognitive errors committed by the users, for which it is important to adapt the prompting strategy.

The Independent LifeStyle Assistant (I.L.S.A.) is an early well-known prototype developed by Karen Zita Haigh [32] at the laboratory of the company Honeywell. It presents a multi-agents system integrating a unified activity detection model, situation assessments, response planning, instantaneous response generation, and machine learning. This prototype main focus is on monitoring the taking of medication and the mobility of elders to issue alerts and information to family caregivers through communication technologies. The ADL model exploits the Geib’s hybrid hierarchical plan recognition model [28], which uses a Bayesian reasoning approach for its task tracking component where each action/plan is represented as a variable with a probability [7]. However, a big limitation was that the hardware of I.L.S.A. is too complex and requires many hours of testing, calibration and active debugging, as well as multiple visits onsite, to be able to deploy it adequately. More recently, several new promising assistive systems incorporating cutting edge technologies and modern artificial intelligence approaches have caught the attention of the community. One of the most well known of these recent assistive systems for cognitively impaired users is certainly COACH (Cognitive Orthosis for Assisting with aCtivites in the Home) [21, 22, 43]. It has been developed for several years now at the University of Toronto by the team of the renowned Dr. Alex Mihailidis at the ATSL lab. This system aims to actively monitor an Alzheimer’s patient attempting a specific bathroom task, for instance, hand washing, and to offer assistance in the form of guidance (e.g. prompts or reminders) when it is most appropriate. It uses a camera to obtain as observations a set of state variables, such as the location of patient’s hands, to determine the completion status of the task according to a handcrafted model of the activity. If a problem occurs, such as an error being made or the patient seeming to be confused, the system computes the most appropriate solution to finish the task, using a probabilistic approach based on Partially Observable Markov Decision Processes (POMDP), [39], and then guides the person in the completion of his activity. Hence, this approach is an adaptive system that learns how to guide, in the best way, the user using POMDP. Clinical trials conducted with the COACH system, including Alzheimer’s patients and therapists, have shown very good results in monitoring a single pre-established activity and in providing adequate assistance at the right moment [21]. Nevertheless, an important limitation of this prototype is that it relies on a complex and very sensitive sensor: a single camera. Although, in principle, the data captured by the camera should be as useful as that captured by the key human senses of sight and hearing, in practice the task of extracting features from such rich low-level representations has proven to be very challenging and not very robust when generalized [17]. For instance, the camera is sensible to many changes, such as fluctuation in brightness, color variations, form of the objects, etc. Moreover, the presence of a camera in the bathroom affects the privacy of the user and causes ethical issues. Finally, COACH does not address the fundamental task of cooking at home and focuses uniquely on bathroom tasks.

Another interesting recent prototype is the Mobile Social Computing prototype of Afridi [1]. This project focused on an approach of mobile social computing to offer assistance to enhance the care for elderly. The cares, which are considered, are divided into three categories: physical needs, emotional needs and task or functional needs. These needs are supported through social media and software (e.g. applications on a smartphone, robots, etc.) and ubiquitous care software information. The system itself incorporates only a small AI system with seems to be a simple rules-based inference engine [30]. This technology has been developed to make easy the relation of elderly with their family and to ensure the collaboration of family members to take care of them despite the distance and the lack of time. Therefore, the elderly care becomes a social activity and responsibility. However, this system has some weaknesses. First, it can be difficult to protect the privacy of the family using social networking technologies. Second, the elders feel less comfortable with social networking. More importantly, the issue targeted here can be seen as secondary or higher level needs. The cognitively impaired people have more basic needs, related to fundamental ADLs (cooking, bathing, etc.), requiring to be addressed first.

Finally, another very interesting recent system is “CASAS: A Smart Home in a Box (SHiB)”, developed by Dr. Diane J. Cook at the University of Washington [2, 20]. The system aims to procure a lightweight smart home design that is easy to install and provides smart home capabilities out of the box with no customization or training. It is based on a set of small Zigbee wireless mesh sensors that can easily deploy in the home by the user. CASAS’ SHiB prototype includes an activity recognition software, called AR, providing real-time activity labeling as sensor events arrive in a stream. More precisely, they exploit an approach based on a support vector machine (SVM) algorithm [40] for real-time activity recognition. The system can also provide activity-aware health assistance in the form of prompting individuals to initiate important daily activities such as taking medicine, exercising, or calling their children. A machine learning algorithm is trained to identify when an individual performs an activity. This system is really a great achievement. They tested the system in more than 32 smart homes and had very promising results. However, the SHiB prototype suffers from a very important limitation, which is related to its ability to recognize activity at a useful level of granularity. Indeed, the system can recognize activity, but only high-level activities such as “bathing” or “cooking”. It is unable to follow, for instance, the performance of recipe, step by step, while preparing a meal. Therefore, it cannot be used for helping cognitively impaired people cooking.

Of course, there are a lot of other similar examples, in the literature, of prototype systems aiming to assist persons with disabilities. For a thorough literature review on the subject, the reader may refer to [3, 5, 12, 24, 38, 58]. However, in summary, we can see that the vast majority of existing systems suffer from the same limitations: using complex or non-robust enough sensors, not taking into account the type of cognitive errors performed by the user, recognizing activities with an insufficient level of granularity, hard to deploy, etc. Moreover, there are very few systems specifically addressing the issue of assisting a cognitively impaired user in cooking tasks.

3.2 Existing systems for helping people cooking

In the last several years, with the democratization of sensors’ technology, a growing interest in the scientific community has been seen in developing smart cooking assistant based on ubiquitous sensors. Many teams [4, 41, 45, 54] have recently developed and deployed real prototypes. In this section, we review a few of the most representative of these systems to position our work among this stream of research.

The first system worth noticing is the MimiCook assistant [54], developed by the University of Tokyo and Sony Computer Science Laboratory. The system consists of a computer, a depth camera, a projector, and a scaling device. It displays step-by-step instructions directly onto the utensils and ingredients, giving a display environment with Augmented Reality (AR) [9]. It controls the guidance display in accordance with the user’s situations. Recipes are embodied in a kitchen counter with AR. MimiCook also displays guidance directly onto the object of interest and controls the guidance display in accordance with the user’s situation. The depth camera recognizes objects’ existence at specific places to judge whether the user is following the instructions. The recognition approach is based on the depth map. A first experiment with the prototype has given mixed results. This kind of system suffers from several limitations. First, basing a system on image recognition is always tricky, knowing that the conditions (e.g. light, form of an object, etc.) always change. Second, even if MimiCook recognize mistake in performing recipe, it does not identify cognitive errors, which is one of the key objectives of our project.

Another interesting recent prototype is “KogniChef” [45], a cognitive cooking assistive system that provides users with interactive, multi-modal and intuitive assistance while preparing a meal. That system augments common kitchen appliances with a wide variety of sensors and user interfaces, interconnected internally to infer the current state in the cooking process and to provide smart guidance. In fact, this architecture is really similar to ours [15]. However, their system aims to evaluate the processing and reasoning skills of the cook, and to provide assistance similar to an expert chef for the user to become really good at cooking. Therefore, the targeted users are healthy persons with good cooking skills that will perform complex recipes. Our targeted users are cognitively impaired people, which would want to prepare simple common meals. Thus, the system does not consider cognitive errors. Also, an important part of our system is based on load cells, which is not used in the KogniChef prototype.

The Smart Cueing Kitchen project [41] is certainly one of the closest initiatives to our prototype. The Smart Cueing Kitchen is intended to be a cognitive orthosis with advanced sensing and prompting tools, designed to satisfy the need of cognitively impaired persons for cooking. In their 2014 paper [41], the team of the Human Engineering Research Laboratories of the University of Pittsburgh described in details the design rationale for deployment of different system technologies in the kitchen and proposed future developments strategies. This project, while being really interesting and close to ours, seems to be still at an early stage of development and our prototype is more advanced.

Of course, many other emerging initiatives are on the way, such as the project “Pic2Dish: A Customized Cooking Assistant System” [4], an all-around cooking assistant on smartphone application developed to help users who would like to cook a dish but neither know the name of dish nor has cooking skill. Basically, by inputting a picture of the dish and the list of ingredients at hand, Pic2Dish automatically recognizes the dish name and recommends a customized recipe together with video clips to guide user on how to cook the dish. Unfortunately, no real AI system is incorporated and no monitoring and errors detection system is included. The private sector also has few initiatives, such as Google Cooking Assistant with Google Home (support.google.com/googlehome/) or IBM Chef Watson (http://www.ibmchefwatson.com). However, these initiatives are not designed for populations with cognitive impairment. Finally, there is the work of [35] which recently provided guidelines for designing requirements for a smart kitchen dedicated for cognitively impaired persons. Our prototype tries to follow their guidelines that are applicable to our context.

In the next section, we present our new assistive smart range that we developed, taking the form of a functional prototype of a smart range. This system is able to monitor a user during the performance of a chosen recipe, to identify and categorize cognitive errors, and to send adapted prompting aiming to guide the user to the safe completion of his meal.

4 The smart range: an assistive cooking device

The smart range was developed by our team, at the Laboratory of Ambient Intelligence and Recognition of Activities (LIARA). The starting point of the project was to build a prototype using a standard low-priced range available commercially. We choose a main stream commercially available electrical range from well-known company LG. It should be noted that, in Quebec, almost all the population use electrical appliances and it is very rare to see, for instance, a gas range. It is due to the fact that the hydroelectricity produced by the public company Hydro Quebec is very affordable and, therefore, people often buy electrical appliances, which is cheaper at the end. Although our assistive system cannot be installed straightforwardly by non-professional, it is standardized to be independent from the brand or the model of range selected. Any standard device would work with the system developed.

A schema of the prototype is presented in Fig. 2. The figure shows the hardware additions that are installed on the range. As the reader can see, several sensors were added, the control panel was replaced by an Android tablet, and a computer was installed in the drawer compartment. The sensors added have been minimized to keep the cost low, and the installation procedure simpler. Each of them has their own role to play in the overall system; a role which will be further described in the next sections of this paper. The load cells are used to estimate, with signal analysis [48], the position and the nature of the objects placed on the range. The infrared sensors, combined with the load cells, are used to detect fire. Heat sensors are used to estimate the appropriate cooking time of an item, to anticipate situations that may lead to a fire, etc. Finally, the electromagnetic sensor allows knowing when something is put inside the range’s oven.

Fig. 2

Schema of the smart cooker and the add-ons

The total price of building the prototype was 3178 Canadian dollars. The bulk of the cost, however, comes from the Uno Xpro industrial computer and the APAX automaton that were used for the prototype. The whole prototype was built with industrial grade components to produce a robust “proof of concept”. Figure 3 shows the industrial computer and the wiring in the drawer. Alternatively, the team is working on using commercial grade technology to build a cheaper version using, for example, system on chip such as Raspberry Pi or BeagleBoard. The use of this technology will reduce the cost of our smart range significantly.

Fig. 3

The computer and the wiring as they were built for the first prototype

4.1 Implementation of the system

The system architecture follows a workflow in three levels (see Fig. 4). First, the sensors and the actuators (anything that can be controlled on the range: light, buzzer, heating apparatus) are wired to a programmable APAX-5570 automaton located in the bottom drawer (see Fig. 4).

Fig. 4

Implementation architecture

The APAX automaton harvests information in real time from all sensors. Then it sends the data to the Uno XPro which run on a Windows operating system. The technological choice was made initially to enable the team to use big libraries and not restraint us from using complex algorithms.

The data are processed by a C# application and then it is stored on a SQL Server database. The database was mostly exploited for long-term storage and for compatibility with our smart home infrastructure. Indeed, when a smart range is detected in the smart home, the database is automatically integrated to the smart home environment. The database works exactly as the one implemented of our larger smart home infrastructure, which is composed of hundreds of sensors (see [14]). Therefore, any algorithm working in our smart home automatically adjusts to the new information provided from the smart range.

Finally, this information is extracted every 200 ms by the artificial intelligence (AI). The AI uses it to ensure a safe use of the device and to monitor step by step recipes done with the apparatus. The AI is designed with the philosophy of promoting autonomy of the users, but if the range is in a state that could endanger the user, the power will be cut automatically. Moreover, in addition to the simple monitoring, the AI uses the cognitive errors defined in KAT/NAT, which were described in Sect. 2. When an error of that type is detected, the AI plays the role of a cognitive orthosis and tries to help the user to recognize and address his mistake.

4.2 The assistant software

The control panel of the range has been replaced by a touch screen, or, to be more precise, by an Android tablet. This tablet communicates with the system to either collect information or to command the appliance. It serves as the main human–computer interaction (HCI) component for the user. In some context, it could be considered impractical to have the control panel behind the cooking plate. Despite behind the standard position on an average range, in our smart cooker the HCI could be deported to a different device as needed. The reason is that the Android tablet is simply communicating through a private Wi-Fi network with the system. Therefore, the user could have the app on his smartphone or on another device and interact with the smart cooker from anywhere in his house, as long as it is in the Wi-Fi range of the smart cooker. Figure 5 shows the app in “Free mode”.

Fig. 5

The free mode enabling to manually control the smart cooker

The “Free mode” allows the user to control the smart cooker without following any type of instructions from the assistive system. Basically, it is the same as using the range without all the smart features. However, the security measures are still working in background in in this mode. Therefore, it should detect the basic dangerous behaviors the user could produce (e.g. start a fire, or forget the cooking plate). The goal is always to provide a safe appliance that foster the autonomy of the user. Other than using the smart cooker in free mode, the user can be assisted in his activities. Figure 6 shows the main user interface of the application.

Fig. 6

User interface: selecting a recipe

The main screen of the app is voluntarily very light and refined to minimize the volume of information shown to the user. The targeted population, different types of cognitively impaired persons, has a limited capacity to absorb information and may feel overwhelmed by new knowledge to integrate [31]. On that screen, the user can do three things. First, he can click a button to operate the device in the “free mode”, which was described previously. Second, he can consult a list of assisted recipes and choose one to make. The recipes in this list are activities followed step by step by the assistive system using the sensors and the information on the state of the smart cooker. If a step in a recipe is impossible to detect automatically, the user is asked to confirm completion. As a side note, none of the recipes implemented in this first prototype required a completion confirmation and we believe it should be avoided as much as possible. Indeed, this could add to the cognitive charge of the user, which might not be beneficial to his condition [31]. Figure 7 shows an example of screen when a recipe is ongoing.

Fig. 7

Carrying a recipe: the preheating step of the cooking a chicken

The recipes in the assistant are generally simple. They contain only a few steps and are purposely created to avoid details. The targeted population is assumed to a group of persons who want to cook to be autonomous (feed themselves) and, therefore, is unlikely to create complex meals with the device. This last statement is not a scientific fact, but only an assumption drawn from our experience with people suffering from cognitive impairments. A complex recipe can be divided into simpler sub-recipes in the assistant if needed. For example, chicken Alfredo spaghetti could be two recipes: cook chicken and prepare Alfredo pastas. The assistant allows for more than one recipe to be done at the time. More details on the workflow of the assistant will be provided in Sect. 4.5.

As a side note, the app can be configured in different language settings. It can be French with French buttons, or in French with English controls. It can also use independently the metric system or the imperial system. The recipes automatically adjust to the parameters set in the system. The reason for this is that the elders are generally familiar with the English buttons and the imperial system despite being native French speakers.

4.3 Tracking objects on the smart range

The potentially most important component of the smart cooking device is the module tracking the objects in real time on the appliance. This functionality exploits recent methods from the field of signal processing [48]. This module is based on the use of the signal coming from four standard load cells installed under each leg of the range. Load cells were chosen to track the objects for three important reasons. First, this type of sensor is designed for industrial applications and is generally used in chain production, or in other context requiring continuous work. Therefore, they are highly robust which makes them perfect for a device destined to cognitively impaired people. Second, these sensors are cheaper than the other alternative that could be exploited for tracking such as camera or radio-frequency technology. Finally, their usage is simpler in this context. Using RFID would require one to add tags to each of the objects to track and to ensure that these tags are heat resistant. Using computer vision with camera would require security & privacy measure, and a complex process to build recognition models.

4.3.1 The tracking algorithm

The idea of the algorithm is to analyze the variation and the distribution of the weight on the range. Since there are four load cells under each corner of the appliance, the distribution of the weight is a clear indication of the position of the objects on the range. Suppose that we have the set $W=\{w_1,w_2,w_3,w_4\}$ containing a weight history for each leg associated with an iteration number i. If the object $\alpha$ is installed on the apparatus after a certain time $\varDelta t$, then the weight of $\alpha =\sum _n^4w_n[i -\varDelta t]$. In other words, the weight of the object is the sum of the new weight of the legs before $\varDelta t$. This part is actually trickier than it seems. An object weight is actually added over several iterations and sometime the weight can go a bit higher than the real weight. As a consequence, the algorithm cannot simply use the max variation in weight over a window of time. The algorithm works like an oscillator. It actually evaluates the general stability when the appliance is being used without any object added or removed. Then using this information, when a variation above the threshold is detected, the previous stability point is selected to determine the weight of the object.

There is a list of objects for the five possible positions on the range; the four hubs and the range’s oven. The actual tracking works with a moving Gaussian average [52] on the weight of each legs of the appliance. Once it is decided that an object has been added, depending on the distribution of the new weight the object is added to the list corresponding to its assumed position. In addition, the module provides a qualitative level of certainty (low, medium, high) to any other services using it. In that regards, a sensitive service could discard any low certainty information and act differently. For example, let us suppose that the user puts a new object on the range. Then let us assume that the module determines that the weight has increased mostly in the front on the left side of the range. In that case, the object is added to the corresponding hub’s list with a medium certainty. If the front left hub is on, then the certainty could go to high. In opposition, if the front right hub is on and the front left hub is off, then the certainty would go to low. An object would be added to the range’s list if the new weight is well balanced. The certainty would be high if two door events were observed in between (door of the range opened and then closed). Using this general method, the AI can track several objects at the time. Table 1 illustrates an example of scenario.

Table 1 Example scenario of weight variation on the smart cooker

This example gives a good idea of the algorithm behind the object tracking. It has been overly simplified to make it shorter and easier to understand. In reality, the weight actually changes every iteration. It also changes over several iterations when an object is added making the task a little bit more complex.

Table 2 Accuracy of the tracking experiments

4.3.2 Calibration

The tracking module enables the AI to get a precise idea of when, where and how many objects were deposited on the appliance. This functionality is crucial for the recipe assistant and for the security services. However, to work properly, every time the device is moved at new physical location, it has to be calibrated. The calibration is an automatic process which is done by any system admin (see Sect. 4.6). Prior to calibration, the admin should make sure that the appliance is on level with the floor. If the calibration is done while the device is tilted, the accuracy of the tracking will be affected. To calibrate the device, a single button has to be pressed and the device should not be touched for 20 seconds. During that time, an average of the weight for each leg will be computed over the hundred value collected (the load cells information is collected every 200 ms). Hence, the system will memorize the normal weight expected on each leg when the range is free of any object. A good calibration is necessary because a range is a heavy appliance. Whenever it is moved, the weight distribution can change for two reasons: the new location’s floor is tilted differently, or the legs’ adjustment has changed. A bad calibration will influence the accuracy of the different intelligent services. Luckily, once installed, a range is rarely moved if ever. Moreover, if it is moved only to be put back at the same spot (e.g. for cleaning under), it is likely that the calibration will not be necessary (i.e. the variation will be minimal).

4.3.3 Validation of the tracking system

To validate our weight tracking algorithm, we designed a short experiment done at our smart home laboratory. The validation of this module was very important since the whole smart cooking device depends on it. Three types of object were used: 0.45 kg, 1.13 kg and 4.5 kg. For each of the tests, between 1 and 4 objects were added on the appliance for approximately 10 s. The objects were added on each hub and in the range’s oven. All the tests’ variation was done totaling 285 tests and more than 21,000 data samples. Table 2 summarizes the results obtained in terms of accuracy.

There are few things that can be observed from this table. First, tracking one object at the time is very easy for the system. Second, the heaviest object was more difficult to track when it was not put in the range’s oven. Since it was heavy, it generally took more time to the tester to place it on the hubs. Therefore, the system often detected the consecutive placement of two objects on the hubs instead of one. This type of error, called addition, accounted for the most part of the errors for the 4.5 kg objects. Figure 8 summarizes the categories of errors that were observed during the experiments.

Fig. 8

Number of errors in the tracking system for each category

As a side note, very rarely more than one heavy object would be placed on the hubs and more than likely such an object would be in the range’s oven, which does not cause this problem. Third, most of the errors observed occurred when four objects were placed on the range. These errors were caused by the inability of the system to accurately measure the weight change when the fourth object was removed. Very often, the weight spiked low and then stabilized to a higher level, therefore, resulting in an early removal of two objects from the hubs’ lists instead of one. In only one case, an object was not detected when installed on a hub. In that case, an object was added in the range’s oven and then quickly added on the front right hub. The system could not detect the variation of weight since it had not finished stabilizing.

4.3.4 Discussion on the tracking system

The experiments we designed covered extreme cases of utilization of the apparatus. Despite this, the results showed accurate and robust tracking of object in real time. In realistic context, the objects used for cooking will have more similar weight which will help the system to track. Moreover, since the targeted population is suffering from cognitive impairment, it is unlikely that they will perform complex recipes resulting in the need to track several objects at the same time. There are some parametrizations to the tracking algorithm. In particular, there is a trade-off between how quickly it can detect a new object and the stability of the signal. We empirically parameterized this, but a learning of the best value could be done in the future. Finally, the tracking system has troubles detecting very light objects. The load cells are sensible to the weight variation, the problem is that such variation occurs naturally when the smart cooker is being used whether or not an object has been added. Therefore, the tracking has to ignore any variations below a certain threshold.

4.4 Passive monitoring of dangerous events

Whenever the smart cooking device is being used, security services run behind the scene. These services are simple ad hoc algorithms that are built to be very robust. The prompting/actions chosen by the system to intervene depend on how dangerous the situation is and can go from a simple warning (e.g. a beep) to a complete shutdown of the range. It should be noted that cutting the power of the apparatus do not shut down the AI. There are currently three of these passive services that help keep the user from unsafe behaviors. The first service monitors if the range’s oven and its door are adequately used. If the oven is being used (i.e. is heating), the door should never stay open for an extended period of time. Using the electromagnetic contact, the service will calculate the time it has been opened to warn the user he may have forgotten the door. After 5 min, a sound is emitted to get his attention and the light will start blinking. After another 5 min, it will beep again and shows a demonstration video on the screen. It will keep trying, and after 20 min the range’s oven will be turned off automatically.

The second passive service ensures that none of the cooking apparatus are on without being in use. For the oven, this is straightforward; it simply checks if the range’s oven has been at the set temperature for a while without any activities (e.g. open door, object added, object removed). For the hubs, it is slightly more difficult since the system only knows the temperature and not their on/off state. The service keeps in memory the believed state of each hub and updates it every time it executes (every 200 ms). If a hub temperature does not gradually go down, the service pools the tracking module and ask if an object could currently be on the hub. When few minutes have passed with the hub heating and no object detected, the assistance process begins. The assistance is progressive; it uses the same protocol of the first service, but for the hubs the power needs to be cut in the last resort. The last service is the flame detection. This service is one that we hope is not going to serve and is complementary to the standard smoke detector and/or fire sprinklers. The service uses infrared (IR) sensors to do so. The usual method to detect fire is by monitoring smoke or heat. It would not be possible in our case since any meal preparation could generate smoke. Moreover, we want a starting fire to be detected as quickly as possible. The service also needs to avoid false positives; resident tend to disable systems that are triggered too often. According to a report from the government of Quebec in Canada, as much as 30% of the smoke detectors did not function properly when a fire occurred [29]. To work properly, the fire detection relies on the evolution of the infrared signal and on the tracking of objects on the range. Basically, an open flame will generate a lot of infrareds which will be captured by the IR sensors. If it happens, before triggering, the system makes sure the IR are not generated by an uncovered working hub by pooling the tracking module. If a flame is detected, the power is immediately cut, an alarm is activated and security is called.

4.4.1 Experiments on the flame detection service

The first two passive services are straightforward and were very easy to validate in our lab. The fire detection is, however, more complex to validate since we cannot empirically observe it in realistic usage. The team collaborated with the security service of the University to design an experiment with controlled fires on the range. In this experiment, the goal was to check the detection accuracy depending on the flame size and the sensibility level of the sensors. The IR sensors can indeed be parameterized to be more or less sensible to the IR. We qualitatively evaluated the sensibility to the minimal, the median and the maximal value. Three types of fire were done on the range. The first one was a simple flame of approximately 5 cm from a conventional lighter. The second type was made by burning a small paper ball. The flames were between 5 and 15 cm. The third and last type of fire was also made by burning a paper ball. However, the paper ball was bigger and installed in a saucepan to simulate a real fire. We conducted five tests for each category, or 45 simulated fires in total. Table 3 summarizes the results.

Table 3 Accuracy of the tracking experiments

4.4.2 Discussion on the passive monitoring

The results of our experiments with the fire detection system are very encouraging. They seem to suggest that the best course would be to set the sensors to very sensitive since it could even detect the smaller flame from a lighter. However, since we want to avoid false positives, it is actually better to select the less sensitive value. The plus value of this fire detection is to avoid false positives compared to a standard smoke detector. During the different tests that were done with the smart cooking device, no false positives were observed. Long term experiments will be necessary to confirm the preliminary hypothesis that our fire detection is more accurate and does cause very few false positives. There is also a known limit to the fire detection with IR sensors. Other source of IR can exist and cause the sensors to spike. Direct sunlight on the sensors, for example, generates as much IR as a fire. The system is able to detect that the IR are caused by the sun, and not a fire, but while the sensors are overloaded by the sun, they could miss an actual fire. There is no perfect fire detection method to the best of the authors’ knowledge, and adding a smoke detector would not solve this type of problem (whenever the range is used, smoke would be detected). Still, this is an open question for future work in this area.

4.5 Monitoring the recipes with AI

The last module that needs to be discussed is the one that enables step by step monitoring of a recipe. The monitoring module also aims to detect the errors a user with cognitive impairments could perform when carrying his/her activities. The task relates to the activity recognition problem which consists of 1. take low-level sensor data and transform them into meaningful information, 2. merge and associate this information with atomic action that could represent a step in a plan, and 3. circumscribe a plan library with the recognized actions to decide on the current activity. In our case, step 3 can be ignored since it is assumed that the user preselects the recipe from the main menu. This type of activity recognition is nonetheless difficult, since it cannot be assumed that the user will be rational at all time. Moreover, it is necessary to have a monitoring with close to a hundred percent accuracy since otherwise it could result in confusion toward the assisted person. In consequence, the team decided to avoid using learning-based activity recognition methods, despite being among the most promising one in this field [60]. Figure 9 shows the different modules which intervene in the monitoring of the recipes.

Fig. 9

An abstract representation of the main modules implemented in the system

The recipes are recorded in a knowledge base accessible by the AI. Each recipe is modeled by a stochastic state-transition model [25], which includes information about time, weight, sequence and action. In these models, probabilities are used to infer which transition is taken from a state when all the conditions are met. This enables the AI to choose accurately between two or more transitions if all conditions seem to be properly filled. While the monitoring algorithm allows the system to properly navigate through the state-transition models, another algorithm, illustrated on Algorithm 1, aims to determine if the state of the user is normal or abnormal. The abnormal state is described using a mix of the cognitive error system described in the Kitchen Task Assessment by Baum and Edwards [8] and the one described in the Naturalistic Action Test (NAT) by Schwartz [55]. Some of these errors are easy to recognize/categorize, but others, such as Action addition are often impossible. A discussion on this topic can be found in Sect. 6.2. Finally, a simple priority is also associated automatically with the current step corresponding to how dangerous the situation is (low, medium, high). Whenever the AI determines that the user is in an abnormal state, an assisting solution is constructed using the prompting model described in [36].

4.5.1 XML interpreter

One of the challenges of activity monitoring with a state-transition model is the difficulty to encode the library of activities [26]. Such model can be complex to build and it limits the ability of users to customize the cookbook. The system has been built with an XML interpreter that can convert recipes expressed in a simple XML structure to a state-transition model. There are currently 5 types of tags defined to construct a recipe. The RECIPE tag which contains information about the recipe (e.g. name, png file for the miniature). The STEP tag which specifies the name, description and the optional time required to complete the step. A step can be specified as optional. The CONDITION tags are a bit more complex. They are used to describe the conditions that should be respected in the realization of a step. Conditions can be on sensors, on time spent, buttons pressed, on steps or on other conditions (propositional logic is used to create condition of conditions). The conditions can activate a transition or an ACTION. The action tag is simply used to trigger a specific prompt when some conditions are met. Finally, the last tag (QUESTION) is used to specifically ask a simple question to the user to better plan the recipe. While it has not been used in our tests, an example could be to ask if the pizza has a thin crust to automatically adjust the cooking time. However, when designing a recipe, question should be avoided as much as possible to reduce the cognitive charge placed upon the user. Any recipe expressed in that language is automatically added to the cookbook. Currently, the expressiveness of the interpreted language written in XML is lower than the real representation of the recipes as a state-transition model, but the team aims to continue developing it to enable the user to easily construct new recipes through a graphical interface generating the XML. With a creator software, any end users could add recipes to the system and built upon the library. Figure 10 shows a part of a recipe written in the language. The recipe has been translated from French and simplified for the purpose of this paper.

Fig. 10

A simplified example of a recipe in XML

4.6 Administration mode

The system installed on the tablet can be used in administrator mode (see Fig. 11). To do so, a panel can be opened to enter a username and a password. The administrator mode is mainly used to diagnose the apparatus and see the raw values of the sensors in real time. The administrator can test the alarm, the light and also enable or disable the power on the range. The passive monitoring services can also be disabled or enabled in that mode. Finally, the administrator mode can be used to configure the system. The recipes can be loaded from the repository.

Fig. 11

The GUI of the administration screen

5 Experiments with the smart cooking device

In addition to the validation of each functionality of the smart cooking device, the team decided to implement three cooking scenarios (see Sect. 5.1) and validate them in a realistic simulation context in our smart home infrastructure. Although these simulations were done by the lab members, we claim that everything was done to make them as realistic as possible. Moreover, our infrastructure is composed of every element a real kitchen possesses and could provide all the facilities for a family to live. For the experiments, the participants were given instructions on the broad objectives of the research and how the scenarios would go. For each test series, four different activities had to be performed. In total, 24 scenarios of performing recipes were done. For each scenario, the instructions were provided before the beginning, and then no help would be provided during the execution. Finally, before each test series, the load cells were automatically calibrated by the system. Good calibration of the load cells is essential to the monitoring of the on-going scenario. The reader should be aware that calibration would be scheduled to be performed automatically when the apparatus is not used for a long period of time (e.g. during the night).

5.1 Methodology and scenarios

The first scenario consisted of cooking a chicken. This scenario, which is illustrated in Fig. 12, was performed twice. The first time, no instructions, other than telling the person to cook a chicken, were provided. The goal was to familiarize the user with the system and he was not asked to perform any mistake. Even so, in two cases, the system detected minor errors (correctly) where the user performed the step putting the chicken in the range’s oven before waiting for the preheating to be done. As a side note, for the conviviality of the experiment, the food was replaced with something of the same weight and the cooking times were shortened to be able to proceed with all scenarios in an acceptable timeline. For the second scenario, we asked the subjects to perform the same recipe again, but this time by proposing them a number of possible errors and asking them to choose at least two of their choices. A total of 8 errors were proposed such as inversing steps, add an action or omit one of the steps. For the third scenario, the participants were asked to prepare pastas. The normal steps (in grey in Fig. 13) to perform the activity were simply explained to the participants. In this case, we explained the type of errors that could be done according to the KTA/NAT categories [8, 55].

For the last scenario, the subjects had to cook a pizza. The whole scenario was provided for this last test, including four errors to be performed. The goal was to test a specific type of cognitive errors with the variability induced by the participant but in a controlled manner (Fig. 14).

Fig. 12

Scenario corresponding to the cooking of a chicken activity (blue states are potential errors)

Fig. 13

Scenario of preparing pastas with the type of errors performed

Fig. 14

Scenario of cook a pizza with the exact errors to be performed

Fig. 15

Error recognition by categories

5.2 Cognitive error recognition

Following the completion of all the experiments, we compiled and analyzed the collected data. In Fig. 15, we have classified the various errors under the KTA and NAT categories. First, it should be mentioned that for the execution of the normal scenario of the recipe Cook a Chicken, the subjects made two inversion errors in total; the chicken was put in the range’s oven before the preheating step was completed. In both cases, the error was detected and reported to the user. One of them withdrew the chicken and waited until the range’s oven was completely preheated. These results are not surprising since the subjects had no cognitive impairment and they had not received the indication to make deliberate errors. The second performance of this same recipe, however, had a good number of errors. Of these, two action addition errors were not correctly identified by the software. In the first case, it is a false positive that has been triggered by a problem with the tracking system. The addition of an object on a hub was perceived when it had no place to be. This error was caused by the delay in perceiving the chicken as it was installed on the grid by the subject. In the second case, it is a false negative. The subject changed the cooking temperature of the chicken and the artificial intelligence did not perceive it as an additional action.

For the third recipe, most of the errors were of the “Omission” type. Many subjects forgot to turn on the hub at the beginning of the recipe and/or turn it off at the end. We also had two cases of forgetting the pastas on the hub when they were cooked, including a case where the hub remained on (potentially dangerous situation). During that recipe, the software made three mistakes that were false negatives. In the first case, the pasta was added before the water boiled, but the difference in weight did not allow the tracking technology to observe the step. On the other hand, this error was also committed by another subject and this time it was detected. As a reminder, the sensitivity of the tracking technology was deliberately reduced during these experiments to ensure stability and robustness. The second false negative was caused when a subject simply added the pan lid to another cooking hub (the latter was not normally useful for the good completion). Finally, the third and last is a false negative of the type Spatial estimation. This type of error was committed by three different subjects when making this recipe. In all three cases, the subject placed the pan on the wrong cooking hub rather than the one he had turned on. In one case, the pan was placed on the hub used to keep the food warm in the middle of both rear cooking hub. However, our system does not track objects on this hub, so it selected one of the rear hubs and in this case, the choice stopped on the active hub resulting in a false negative.

Finally, for the recipe Cook a pizza, all the errors were detected without difficulty. There were no false negatives or false positives. It should be noted that this scenario was the only one that made it possible to test perseveration type errors. In this case, all the subjects repeated the installation step of the pizza at least three times completely or partially. This is a type of error that is usually easy to detect if it involves the tracking technology (weight variation) and does not require knowing the current recipe. However, it takes more than two repetitions to have good results. Additionally, of the ten categories of KTA/NAT, three have not been listed in this chart. These are the categories “Gesture”, “Tool” and “Substitution”. This is because for these three categories, the majority of potential errors are undetectable with the smart cooking device. We can, therefore, consider that the false negative rate would be very high if not 100%. Also, we have covered the initiation error. This type of error is very easy to detect and has not caused any problems. In fact, in no case do we need to know the current recipe to achieve it. Overall, the error detection rate when performing the 24 recipes was 92%.

5.3 Discussion and limitations of the recognition system

The results obtained are very encouraging for the potential of the cooking appliance. These tend to confirm the hypotheses our research team formulated at the beginning of this project. Many types of errors are very easy to identify when the user is cooking with the assistance system. In particular, the errors of omission, inversion, anticipation, perseverance and initiation do not cause many challenges. The good performance in detecting errors largely depends on the performance of the sensing techniques (mostly the tracking system). In fact, a sensing error will likely result in error in the ongoing recipe monitoring. It should also be noted that the results of these experiments were collected in a context of recipe simulation. Although these are an encouraging sign for the future of our prototype, experiments in a living lab context are needed.

Another limitation regarding the experiment is about the Add Action errors. Although our results were satisfactory, in general, action addition detection is a type of error that is more difficult to detect due, among other things, to the very large number of possibilities. In particular, if more than one activity is done in parallel, the system must validate whether the action is part of one or the other of the plans, including erroneous versions thereof. On the other hand, it is difficult to decide automatically whether adding the action requires some form of intervention. For example, if the user lowers the temperature of a plate to avoid an overflow, it seems to be a relevant action addition. On the other hand, if it does so for no reason and the temperature is too low for proper cooking, then adding this action would require intervention. In short, it is clear that many cases of “Add Action”, by their ambiguous nature, would cause uncertainty.

Finally, these experiments allowed us to see two important elements. First, the step-by-step tracking of recipes and the real-time assistance require a proper script writing and a good deal of calibration. The calibration is especially important for the object tracking system, since a bad calibration would cause the system to fail in monitoring the recipes. Second, we observed that several errors are repeated in much the same way from one recipe to another. We, therefore, think that it is not necessary to precisely script each recipe. Indeed, if the focus had been put on the errors leading to dangerous situations, we anticipate that it would be possible to create a generic assistance model for all the recipes. This would require further investigation.

6 Market study with targeted users and potential buyers

For this project, we deposed a declaration of invention to our university. The declaration was accepted and a valorization process of our new device, funded by the university, began. The first step of this process was to fill a patent declaration to the U.S. Patent and Trademark Office (Patent no. 20150099245). The provisional patent declaration is entitled “Method for Monitoring an Activity of a Cognitively Impaired User and Device Therefore”. The reputed firm Norton Rose Fulbright was hired by the university to guide us in the process. The second step was to proceed to an extensive market study to clearly investigate the potential of the device in multiple aspects. The questions we wanted to answers were, for instance: 1. Is there a real need for such kind of device? 2. What are the interesting market segment? 3. What users and buyers think of the device in terms of features, design, etc. For this step, we obtained a grant from the Natural Sciences and Engineering Research Council of Canada (NSERC) to fund our marked study. Then, with the help of a valorization firm named SOVAR, and with the assistance of the market research firm Zins Beauchesne & cie, we conducted an exhaustive study with the targeted users, with the potential companies, with public organisms and with professionals. For this study, 42 individual interviews have been conducted and 148 companies and organisms have been directly consulted. More precisely, here the list of all people and organism that has been interviewed:

• (7) Appliance manufacturers (OEM);

• (70) Private residences for elderly;

• (9) Suppliers and distributors of equipment and solutions to assist a user with a loss of autonomy to remain in their home;

• (12) Professional from the public health network;

• (2) Representatives of programs for home care for the elderly;

• (35) Occupational therapists working with seniors, including (2) working in the private sector;

• (10) Agencies and service companies for home support;

• (1) Representative of an association of appliance manufacturers;

• (1) Representative of the Canadian Alzheimer Society;

• (1) Representative of the Quebec industrial cluster “Biotech Santé”, whose research and development initiatives are geared toward home care for the elderly.

6.1 Size of the potential market

First, we wanted to have some information about the scale of the market in terms of senior residences. At the Quebec’s province level (  8 million inhabitants), there are actually 102,101 units in retirement homes. These are broken down as follows:

• 43,743 single rooms and semi-private rooms;

• 12,101 studios;

• 33,984 units consisting of a bedroom;

• 12,273 units consisting of two rooms

The last three categories include ranges, or 58,358 ranges. Therefore, we can conclude that there is approximately 2 569 175 potential units in retirement home needing a smart range in North America by assuming the proportion are similar in the U.S. Of course, there is much more potential individual buyers because, overall, seniors living in home residences mostly have access to a range. Often, as soon as the first symptoms of cognitive impairment appear, procedures are started to remove the access to the appliance. As this medical condition is a sensitive topic for the resident (e.g. the acceptance of illness may be difficult for some), some companies may not be comfortable about imposing behaviors on the resident inside their own apartment, considering this space of the private domain. They would prefer to act as advisers to their clients, residents and their families.

6.2 Product design assessment

The smart range seems to be appreciated In the results of the consultation, the vast majority of respondents find the idea of having a smart cooking device able to help people prepare their meal in a safe way very attractive. More specifically, the feature of automatic power cutting in case of emergency, the possibility of giving access for only a certain period of time, and the step by step assistive feature were mentioned as very interesting.

Concerns about the complexity of utilization Many respondents had concerns about the complexity of the device. Comment such has “my mother only knows how to push the bake button and nothing else, how can she use such an advanced smart device?” were frequent. Some managers of senior residences had concerns about the complexity of supervising their clients when they will use the device. Finally, few people had concerns about the complexity of adapt your recipe with the device.

Concerns about the potential price The main concern of most respondents was the potential price of the device. Seniors people are often poor and the managers of residences are concerned about the cost/benefit value of buying such of device themselves for their residents. The study points out that most respondents think that a price situated between 50$ and 250$ seems to be acceptable for the new smart features. Therefore, a smart device like ours should not be more than 250$ more expensive than a regular cooking device if one wants to commercialize it.

Reducing the insurance price The assurance companies could play an important role in the decision of acquiring a smart range. The vast majority of respondents think that a reduction in their insurance price (e.g. for fire hazard) would constitute a good argument toward the decision of buying such device.

What professionals think? The results of our study show that health professionals think that this kind of device could really help people stay at home longer. Their main interest in the device reside in the safety features allowing preventing fire and minor incidents with the range.

What features are missing? According to respondents, several missing smart features could add values to the product. Here the list of the main ones proposed by the respondents.

• 1. Automatic fan activation when required.

• 2. Possibility of setting a usage schedule.

• 3. Possibility of locking the device or certain parts (e.g. the range).

• 4. Adding control buttons on the front of the device.

• 5. More visual cues with light.

• 6. Automatic clock adjustment.

Interest for a retrofit version of the device Most respondents were interested by a concept of a smart device that can be installed on a regular range. That kind of kit could come with some wireless sensors, a screen, and some pieces of hardware. However, our actual solution is fully embedded in a range and it was designed to be so. In the future, we want to work on a lighter version of the device that can be adapted for a retrofit kit.

7 Conclusion and future works

People suffering from a loss of autonomy caused by cognitive deficits experience many problems while trying to perform their activities of daily living [31]. They have to live in environments that have not be designed for their needs, and to use (at their own risk) appliances designed for healthy people. In this paper, we presented a new assistive system, which takes the form of a smart range prototype [15], allowing monitoring the cooking activity of a cognitively impaired user and to give adapted guidance [36] in the completion of a recipe. The system that we developed has the capacity to detect risky situations (e.g. a dangerous state that may lead to fire) and is able to take preventive actions accordingly. The originality of our invention is to combine, in real time, the inputs coming from load cells, heat sensors and electromagnetic contacts embedded in the range to infer, using an artificial intelligence state-transition recognition model, the current state of an on-going activity. The prototype is also able to identify the main types of errors characterizing cognitively impaired users [8, 55]. The artificial intelligence relies on a stochastic state-transition representation [25, 26] of each activity. We conducted several experiments with the prototype, giving promising results and showing the interest of this device. We also conducted, with the help of a valorization firm, a study with the targeted users, companies, public organisms and professionals. For this study, 42 individual interviews have been conducted and 148 companies and organisms have been directly consulted.

Indeed, a lot of work still has to be done to reach an adequate level of robustness and readiness to be deployed in real users’ homes. First, at the light of the study with targeted users and companies, we will need to adjust the functionalities of the devices to better fit with the needs. Second, we will need to conduct more tests in controlled environments to calibrate and ensure the effectiveness of the device. Third, we will need to proceed to an evaluation with several persons by deploying the device in a real in-vivo environment. We already have two partnerships with institutions that have access to prototypes of real smart residences where the device could be deployed. Finally, according to the result of the study, we will need to design a stand-alone retrofit version of the device capable of being installed on already existing standard appliances.