Keywords

1 Introduction

Recent technological advancements in Internet of Things (IoT) and Cyber-Physical Systems (CPS) are fostering the diffusion of smart environments relying on sensor networks. Smart environments have created an upward trend in long-term monitoring systems demand for the future. The deployment of such sensor-based research trend is to address the necessities of people in many different environments like e.g., a home, an office or a shop mall, just to mention few examples. These devices usually rely on different types of sensors capable of continuously producing data about, the status of a particular environment or the status of a person. Sensors are capable of gathering data that can be processed by external services with different purposes. In this context, the continuous improvement of sensing devices in terms of accuracy and reliability as well as the reduction of manufacturing costs and their combination with ICT technologies are paving the way for new research challenges and business opportunities. The management of a continuous flow of data with the need of properly representing and processing such data still represents an open issue and an important research trend. Specifically, there is a lack of integrated solutions for both sensor data representation, data interpretation and knowledge extraction. Also, there are several sensor-based applications that have been successfully developed in different contexts, each of which often leverages its own “internal” representation of sensor data as well as its own processing mechanisms.

This paper presents a recently started research initiative called KOaLa (the Knowledge-based cOntinuous Loop) that tries to propose an innovative approach to deal with sensors, data processing and decision making. KOaLa initiative is a follow up of a previous research project called GiraffPlus [8] whose aim was to realize an integrated system composed by a telepresence robot, a sensor network and personalized services for fostering social interactions and long-term monitoring of senior users living alone in their home. KOaLa aims at enhancing GiraffPlus data analysis services by integrating Artificial Intelligence (AI) techniques to create a continuous robot control loop leveraging automatic reasoning features on sensor data. Specifically, the pursued research objective is to introduce “cognitive” capabilities that allow a system to proactively support the daily home living of seniors. The key contribution of KOaLa is to realize a sensor-based reasoning mechanism capable of continuously analyzing sensor data and dynamically make decisions accordingly. KOaLa leverages knowledge representation and reasoning techniques to recognize events/activities within the sensorized living environment and then it leverages automated planning and execution techniques to autonomously decide actions to be executed for either supporting activities or reacting to events.

Sensor data processing and knowledge extraction are performed by leveraging semantic technologies and the Web Ontology Language (OWL) [4]. In particular, KOaLa proposes an holistic approach which relies on an ontology which was defined as an extension of the Semantic Sensor Network (SSN) ontology [7]. The information collected through the sensor network is modeled pursuing an ontological context-based approach to characterize the different types of sensor that can be used, their capabilities as well as the related observations. In addition, the KOaLa ontology introduces concepts that model the environment and the behaviors or situations that may concern the elements composing an application scenario, senior adults and also activities an agent (either a user or a telepresence robot) can perform over time. This paper provides a general description of the envisaged KOaLa cognitive architecture. It focuses on the key role of the ontology and the context-based approach with respect to sensor data management and the knowledge extraction mechanism to realize a flexible and proactive assistive robot.

2 Sensor-Based Applications and Knowledge Extraction

This work covers different fields of AI such as decision making, goal reasoning, autonomous control, knowledge representation and ontological reasoning. There are many works in the literature that are directly or indirectly related to the pursued research objective. Several works deal with the problem of interpreting sensor data to extract useful knowledge about a particular application scenario and leverage such knowledge to realize complex services. The work [2] proposes a knowledge-driven approach based on a ontology defined as extension of the SSN ontology. The aim of the work is to reason about changes in the detected qualities of the pervasive air in a sensorized kitchen. The defined ontology models high-level knowledge about odours, their causes and relations to other phenomenon. Then, an incremental reasoner leverages such knowledge for data processing via Answer Set Programming (ASP). The works [1, 13] propose an ontology-based approach for activity recognition and context-aware reasoning for a home-care service, and a constraint-based approach for proactive human support. Some works deal with the problem of realizing complex architectures for socially-interacting robots capable of “dealing with” humans in robust and flexible way. Some examples are the works [10, 12] that perform human-aware planning by dynamically inferring context and goals by leveraging constraint programming techniques. In particular, [10] proposes an online planning setting capable of dynamically inferring planning goals in order to continuously satisfy human preferences. They rely on the concept of Interaction Constraints (IC) which defines a clear model of the interactions between the activities of a human and the activities of other agents. Some other works address the problem of endowing autonomous agents with cognitive capabilities to represent knowledge about contexts and leverage such knowledge to improve the flexibility of control processes in robotics. The work [3] is particularly interesting as it points out the importance of social-norms for domestic service robots. It focuses on the concept of Functional Affordance and proposes a tight integration of OWL-DL with hierarchical planning to realize robots capable of interacting with the environment in a flexible and robust way. Some other examples are [5, 9] that propose the integration of knowledge processing mechanisms with Hierarchical Task Network (HTN) planning to improve the efficiency and performance of control processes, and the works [11, 14] that propose the integration of knowledge representation with machine learning to improve human-robot interactions as well as flexibility and performance of robot behaviors.

The novelty of our approach consists in the design and development of a cognitive architecture relying on a holistic approach to knowledge representation through a well-defined ontology. The key idea is to realize a control architecture capable of leveraging knowledge processing mechanisms to infer a general and abstract model about the application context and, then integrate complex services that can leverage the generated knowledge for different purposes. Specifically, we consider the integration of automated planning and execution techniques to realize a proactive and autonomous robotic service.

2.1 The GiraffPlus Research Project

The GiraffPlus research project [8] represented a successful example of continuous monitoring of older people using sensor networks, intelligent software and a telepresence robot. The objective of GiraffPlus to realize a system capable of supporting elderly people directly at their home through a composition of several services. There was possible to distinguish between services decsigned for primary users (i.e., virtual visits, reminders, messages) and services designed for secondary users (i.e., real-time monitoring visualizations, reports, alarms, warnings). The services at home are provided by the telepresence robot Giraff. The robot allowed older people to communicate to their friends and family, through audio messages and videocalls as well. Seniors could interact with the robot either by means of a touch screen (part of the robot equipment) or by means of vocal commands. Interaction abilities are extremely important to improve the emotional engagement between the user and the robot. For this reason, different interaction abilities were taken into account. GiraffPlus combined the presence of a robot with a network of sensors. The network could be composed by physical sensors (i.e., blood pressure sensors) and environmental sensors (i.e. motion sensor, actuators). Physical (wearable) sensors monitored the health status of users. Environmental sensors monitored both the house and user behaviors according to their deployment.

2.2 Data Needs Semantics

Data coming from sensors must be properly represented and managed in order to produce useful information that can be used for different purposes and services. Thus, sensor data must be associated with a schema or a semantics clearly defining the general rules and properties that must be followed to properly interpret sensor data and extract useful information. Knowledge processing mechanisms are in charge of leveraging such semantics to process data and dynamically generate knowledge about a particular application scenario. There is a lack of standard approaches for representing and managing sensors data. Different sensor manufacturers can use different formats and protocols to represent and communicate data. As a result, sensor-based applications strictly depend on the particular types of sensor and communication protocol used. The data processing logic is highly coupled to the particular physical sensors available and the “syntactic” features of generated data. Such a dependency does not facilitate the deployment of these applications to different contexts with different sensors as well as does not facilitate interoperability among different applications.

Standard semantic technologies for knowledge representation and processing like e.g., the Web Ontology Language (OWL) [4], are well-suited to characterize the different properties and features of sensor data. Such technologies can be used to extend sensor-based applications by introducing a semantic layer between a physical layer which produces data through a particular sensor network and an application layer which extracts and processes information coming from the physical layer. In this way, the application layer and the related knowledge processing mechanisms depend only on the semantics of the gathered data. Namely, the data processing logic of a sensor-based application will no longer be coupled to the physical features of the deployed sensors but only to the structure and properties of the extracted information. Semantics plays a key role when dealing with sensors and sensor-based applications. Thus, the ontological approach of KOaLa wants to characterize information, properties and capabilities of physical sensors in order to generate/infer and represent knowledge in a “standard way”.

3 KOaLa: Knowledge-Based Continuous Loop

The KOaLa cognitive architecture relies on the integration of two core AI-based modules into a closed-loop control cycle. A semantic module deals with data interpretation and knowledge extraction. An acting module deals with plan synthesis and execution by leveraging the timeline-based approach [6] and the PLATINUm planning and execution framework [15]. Figure 1 shows a conceptual view of the architecture by taking into account a typical application scenario of the GiraffPlus project.

Fig. 1
figure 1

The KOaLa cognitive architecture with respect to the GiraffPlus case study

Full size image

The control flow starts with data coming from the sensors deployed inside the house. The KOaLa Semantic Module continuously processes sensor data in order to extract knowledge and dynamically build/refine a Knowledge Base (KB) characterizing the status of the home environment. The KB is analyzed to recognize goals i.e., high-level activities the system must perform to proactively support the daily home living of the primary user. This module relies on a dedicated ontology (the KOaLa ontology) and implements a knowledge processing mechanism to provide sensor data with semantics. Such a process allows the semantic module to dynamically infer activities a primary user is performing and/or events occurring inside the house. Then, according to the particular combinations of events and/or activities recognized (i.e., situations) a goal recognition process identifies the high-level supporting tasks that must be performed.

Then, the KOaLa Acting Module dynamically synthesizes and executes a timeline-based plan according to the received goals. The Planning and Execution module “runs” the timelines of the synthesized plan by dispatching commands to the system and by properly managing execution feedbacks. This module implements environment control capabilities in order to actually carry out the high-level activities needed to support a user. A problem formulation process receives goals from the semantic module and generates a timeline-based problem specification. The planning and execution module synthesizes a set of timelines that must be executed over time. Each timeline characterizes the sequence of actions/commands a particular feature of the environment must perform to proactively carry out the desired supporting activities.

3.1 A Context-Based Ontological Approach

The semantic module of Fig. 1 is in charge of realizing the cognitive capabilities needed to interpret sensor data and internally represent the knowledge about the application scenario. The reasoning mechanism leverages a context-based ontological approach which characterizes the knowledge according to three contexts: (i) the sensor context; (ii) the environment context; (iii) the observation context. The sensors context characterizes the knowledge about the sensing devices that compose the environment, their deployment and the properties they may observe. This context directly extends the SSN ontology in order to provide a more detailed representation of the particular types of sensor available and their features/properties. This context allows the cognitive architecture to dynamically recognize the actual monitoring capabilities of the system but also the set of operations that can be performed according to the detected configuration of the environment. The environment context characterizes the knowledge about the structure and the physical elements composing a home environment. It characterizes the properties of the different elements that can be observed and the deployment of sensors on these elements. Thus, this level of abstraction provides a complete representation of the particular configuration of the considered home environment by taking into account the specific deployment of the sensor network. The observation context characterizes the knowledge about the features of the environment that can actually produce information and the possible events and activities that can be recognized accordingly. This level of abstraction extends the detected configuration of the environment by characterizing the particular types of event and activity that can be actually monitored/recognized, given the particular types of sensor deployed.

Fig. 2
figure 2

The KOaLa data processing pipeline

Full size image

A data processing mechanism leverages these contexts to process sensor data and incrementally build/refine the knowledge about the application scenario. Figure 2 shows the main steps fo this data processing mechanism together with their correlations with the sensor, environment and observation contexts. Each step applies a set of dedicated rules that elaborate incoming data and refine the KB by inferring additional knowledge when possible. The Configuration Detection and Data Interpretation step first generates an initial KB by analyzing the “static” information about the configuration of the environment. Then, it refines the KB by interpreting sensor data coming from the environment. The Feature Extraction step elaborates the KB by extracting the observable features and properties of the environment according to the configuration. This step processes sensor data in order to properly infer observations and refine the KB by taking into account the particular observed values that characterize some property of a features of the home environment. Finally, the Event and Activity Detection step further analyzes the inferred observations by taking into account the involved elements, and the related properties. For example, it is possible to infer the event low temperature in a particular room of the home environment when the observed temperature of the room is below a known threshold. Different inference rules have been defined to detect different events or activities like e.g., cooking, sleeping or watching tv. Each rule takes into account different “patterns” of observed properties, values and environment features.

3.2 Linking Knowledge Processing and Planning

Given the KB obtained through the data processing mechanism briefly described in the previous section, a goal recognition process completes the analysis of the KB in order to identify situations that require the execution of some actions by the system. The goal recognition process is the key architectural element responsible for linking knowledge reasoning with automated planning and execution at runtime, and therefore it is responsible for providing the an assistive robot like e.g., the GiraffPlus robot with proactivity. It can be seen as a background process which analyzes the KB and generates goals that trigger the KOaLa Acting Module. The actual set of operations the system can perform (i.e., the set of goals that can be generated) depends on the particular configuration of the environment and the related capabilities. Each goal is associated to a particular situation which represents a particular combination of events and/or activities occurring inside the house. The generated goals and the involved operations may range from general supporting activities like therapy reminders or comfort management to emergency management and health monitoring. For example, if the system recognizes low temperature inside a particular room (e.g., the kitchen), and if the system detects the presence of the user inside the same room then, the goal recognition can generate a goal to heat the room in order to improve the comfort of the user. Similarly, if an event like fallen or a very high hearth-rate is detected then, the goal recognition can generate a goal to perform an emergency call and proactively alert user’s relatives and/or caregivers.

The signals generated by the goal recognition process represent high-level assistive tasks the assistive robot must perform to react to some events or support some activities concerning the user. The acting module is responsible for the synthesis and the execution of the set of actions needed to perform all the desired assistive tasks (i.e., all the goals generated by the semantic module). The acting module leverages timeline-based planning and execution technologies [6, 15] to generate and maintain a temporal plan which specifies operations/actions to execute. The key objective of the KOaLa Acting Module is to dynamically control the environment through a temporal plan characterizing the behavior of a monitored primary user over time and the corresponding assistive tasks needed. These behaviors are represented in shape of timelines that describe the sequences of activities or states a particular feature of the domain (e.g., the primary user or the assistive robot) performs or assumes over time. The behavior of the overall system (i.e., the assistive robot, the sensing devices and the primary user) is represented by a set of temporally flexible timelines that are dynamically executed and adapted according to the feedbacks received from the environment.

4 KOaLa and Giraff Working Together

The envisaged cognitive architecture and the resulting tight interaction between a semantic module and an acting module realize an adaptive and flexible assistive system capable of uniformly manage assistive goals and real-time goals. A system capable of dealing with assistive tasks to support the daily-home living according to the “expected behavior” of a person and simultaneously monitoring the environment to dynamically recognize events/activities that require a proactive support. Real-time goals concern operations the system must perform to proactively react to an event or support an activity triggered by the goal recognition process. Assistive goals instead represent operations that can be planned in advance according to the detected configuration and the expected behavior of a user which can be given as an “external timeline” to the acting module.

Fig. 3
figure 3

A timeline-based plan integrating assistive tasks and real-time goals

Full size image

Figure 3 shows an example of a generated timeline-based plan containing activities concerning both assistive tasks and real-time goals. The red timeline represents the behavior of the primary user. The green timeline represents the assistive tasks and the real-time goals generated to support the user. Other timelines represent the actions the system perform to achieve the desired goals through the assistive robot (i.e., the GiraffPlus robot). As the plan shows, the system initially detects that the human (i.e., the primary user) is having a meal (see the cooking activity followed by the eating activity) when the assistive task HandleCookingDetection is generated in order to suggest some recipes that comply with the dietary restriction of the user and remind the therapy. Specifically, the robot navigates the home environment to reach the kitchen, and then reproduces an audio message to remind dietary restrictions to the user (see the ReproduceReminder activity). When the meal is finished (i.e., after the eating activity), the robot starts a video call in order to make the user in contact with his/her relatives and check the therapy. After a while, the user falls and a real-time goal is generated to promptly support the “emergency situation” (see the HandleFallDetection activity). The robot moves from the kitchen and navigates the home environment to reach the detected position of the user. Then, the location is reached, the robot starts a video call to promptly allow the user to ask for help by calling his/her relatives or associated caregivers.

5 Final Remarks and Future Developments

This paper presented a cognitive architecture integrating sensing, knowledge representation and planning to constitute a cognitive control loop capable of enhancing the proactivity of an assistive robot. It relies on a tight integration of a semantic module and an acting module. A semantic module leverages a dedicated ontology to process sensor data and build a KB. An acting module leverages the timeline-based planning approach to control the overall assistive system. A goal recognition process connects these two modules and provides the key enabling feature to endow the robot with a suitable proactivity level. At this stage, some tests have been performed to show the feasibility of the approach. Further work is ongoing to perform more extensive integrated laboratory tests and better assess the performance and capabilities of the overall system.