Keywords

1 Introduction

Reasoning about an environment in the context of smart homes gained significant attention in the last two decades. Researchers have been developing concepts and tools to employ miscellaneous artificial intelligence (AI) techniques for ambient assisted living (AAL) environments [13]. Such systems target the well-being of the inhabitants [18], via health monitoring [8], energy-efficient appliance operation [11] and personalised service adaptation [12]. AI applications in AAL environments perceive the inhabitant via the surrounding sensors and automatically adapt or make decisions. This component eventually exhibits reasoning capabilities and systematic decision support; see [14] for a recent survey. An essential aspect of those systems is detecting and recognising human activities or situations (such as left the apartment) and detecting emergencies (such as falls). It is of high relevance for the ageing population, where an AAL system [9] would notify or act promptly in scenarios of significance. However, such ecosystems’ complexity requires high maintenance costs and complex relations to achieve semantic interoperability to recognise activities in focus. There can be many situations worth identifying using sensors in a single room, ranging from “is someone present” to “is the water boiling”. Considering an entire home, we may end up with hundreds of such situations, and an office building could have thousands. This leads to increasing numbers of sensors to cover all the above situations, driving the economic and maintenance costs to higher levels.

This paper focuses on the reusability and repurposing of existing day-to-day sensors in the environment (e.g., light switches, contact sensors). As such, they do not directly offer the property in need (e.g., situation or activity) but allow their incorporation in an inference task, as “lifted” contextual information. One with a naive knowledge of physics may exercise a hypothesis evaluation about the situation using sensor results as evidence. We offer a solution to encode this Commonsense Knowledge (CK) in rules with the different combinations of sensors while forming a model for recognising the situation in need. The rules mentioned above contain a form of uncertainty in their definition. More specifically, we employ a Markov Logic Network (MLN) for developing a probabilistic model to support the uncertain varieties of different compositions of sensors, expressed as commonsense logic rules, for identifying the situation in need. The inference task also contains a set of meta-rules, which encode the interaction between the sensor events and their interpretation effects. The paper gives a quick overview of the approach and a discussion over state of the art in reasoning in AAL environments. We conclude with a short outlook and future steps.

2 Proposed Approach

This section will briefly describe our approach to representing the situation recognition from “lifted” contextual information (i.e., sensor information) using CK and its uncertainty aspects. In our case, we use the term “CK” to metaphorically argue behind the transfer of knowledge one has behind a naive understanding of physics of how sensors work. For general definitions of CK, see [6]. For example, one knows that a light bulb in operation, in simple words, emits light and gets hotter over time; this accounts for commonsense. As such, using environmental properties (of light intensity and ambient temperature), we may reason behind the followings situations via explanations of the sensor readings (accounts as knowledge): (i) a light switch is flipped on, (ii) someone is present in the room, (iii) but could also possibly mean that there is a fire in the room; and many others. Specifically, for the situation in (iii), the readings of a light sensor and a temperature sensor may not be enough. An additional sensor (e.g., an air quality sensor) will increase the explanation’s accuracy (“there is fire”). One may observe a factor of uncertainty regarding the reasoning behind some sensor readings, as much as a process of Commonsense Reasoning (CR) does.

We automate this form of CR, using symbolic representations of sensor data in logic-based rules over continuous time. MLN is a robust framework that combines both logical and probabilistic reasoning [16]. It allows us to declare a stochastic model at a high level using first-order logic. Besides, we use the well-defined temporal formalism of Event Calculus (EC), a many-sorted predicate calculus, to reason about events and their effects [10]. A hybrid approach of the two was presented by Skarlatidis et al. [19], developing a dialect of EC to model the inertia laws for recognising complex events in an annotated video surveillance dataset. Their work inspired us to select the technologies for dealing with uncertain knowledge and extract situations of interest from continuous sensor data.

The construction of a first-order knowledge base is expressive, powerful and uses unambiguous semantics for its syntactic rules. Constructing an MLN Knowledge Base (KB) is a set of tuples \(\langle w,\, F \rangle \), where w is the confidence value to the rule formula F. Each first-order formula contains different atoms connected with logical operators. Each atom is a predicate symbol applied to a tuple of terms, representing an object in the domain. A term can be a constant (e.g., a sensor type - ContactSensor, MotionSensor), a variable (which ranges over a domain - a range of constants) or a function (applied over terms also).

Using the domain-independent predicates from the EC dialect (MLN-EC) in [19], we create CK formulas that reflect different sensors’ compositions to recognise a situation of interest. EC’s main components are the event and the fluent (a property whose value changes over time). In our system, the events are “lifted” contextual information from the sensor data. They are low-level symbolic representations of sensor data, matching primitive shape-based patterns - we name them shapeoids. The fluents are the monitored situations whose value persists over time. The meta-rules of EC encode the so-called inertia laws [17], which dictate that something continuous to hold unless it is indicated otherwise (e.g., terminated by an event). The variables and functions start with a lowercase letter. The predicates and constants with an uppercase letter.

The offline statistical relation framework of MLN, combined with the meta-rules of Event Calculus, offers a formal, but at the same time, a powerful probabilistic logic-based method for complex event recognition [19]. Open-source implementations of MLN exists, such as AlchemyFootnote 1, TuffyFootnote 2, LoMRFFootnote 3. For our purpose, we use LoMRF as its implementation is in Scala, matching the language of any modern data processing framework (e.g., Apache BEAMFootnote 4), to realise a holistic architecture for online inferences from streaming data. Also, LoMRF has the most recent development cycle than any of the other tools.

Finally, we envisage an architecture for our approach and turn it into a holistic system and a complete pipeline that spawns over the following steps/components:

  • Modelling CK and representing a different set of sensor compositions as alternatives for monitored situations.

  • Extract low-level symbolic representations of streaming sensor data, matching primitive shape-based patterns - named shapeoids.

  • Initiate dynamic inference processes (i.e., inference in MLN) on incoming streaming shapeoid events while posing queries for the recognition.

3 Related Work

Several different approaches have been pursued over the years to encode CK [6] and perform CR [7]. In AAL, the notion of CR is applied when there is a creation of a symbolic knowledge base and perform reasoning upon these symbols. In a recent survey [4], the authors contacted a study for augmenting the situation of AAL and the kinds of solutions applied in such environments. The pervasiveness of sensor devices in smart environments enables systems to “read” and understand the environmental behaviour from sensor data [3, 20, 21]. The task of reasoning in AAL, bases mostly on the probes of information installed in the environments (i.e., sensors) and the context they are applied for (e.g., the inhabitant, daily activities etc.). A fundamental step in a reasoning task, is the initial representation of the entities and their connections (if any). Many use the concept of ontologies [5] as a shared conceptual model of the world, to facilitate the core knowledge representation. The authors in [15] distinguish between lightweight ontologies, storing only the formal hierarchies and relationships, and heavyweight ontologies, adding inference rules for semantic interpretations. Nonetheless, we concur that some situations do not exist implicitly in the data or are challenging to collect and annotate.

The inference process for reasoning with ontologies is monotonic; a new observation will not change already inferred knowledge. For instance, in [1], the authors use a hybrid system incorporating an ontological representation of data and a non-monotonic inference process using answer set logic for sensor data [2] targetting AAL environments. However, their approach does not handle rule uncertainty, although they model their semantics in a model that supports fallible logic. Besides, their approach does not support a formal grounded theory at the meta-level, dictating logic and supporting the CR mode [7]. We opt for integrating fundamental domains from formal grounded theories, probabilistic theory and modern dynamic data processing solutions to automate the CR in smart environments.

4 Conclusion and Future Steps

This paper introduced an approach to use “lifted” contextual information from sensor data and introduce sensor data interpretation via CR rules. A naive knowledge of physics coupled with background domain knowledge of a smart environment opts for detecting occurrences of certain situations. As such, declaring these logical “inference” sentences results in a human-readable form of reasoning that incorporates commonsense logic.

Due to the uncertain nature of making a hypothesis about given observations, we use MLN to soften the constraints against the possible world where the logical sentences are satisfied. The inertia rules must remain as hard constraints. The choice of a bounded environment, such as AAL, narrows down the available knowledge for encoding it with our method. The approach does not foresee an infinite amount of encoded inference rules, as inference in MLN may become intractable if we make too many open-world assumptions in the CK rules. The complexity of the system relies on the definition of the CK rules. However, our approach’s novelty is the redundancy in detecting the desired situation via alternatives from ones’ CK with the available nearby sensors, considering that we normally use direct means for sensing (e.g., use a contact sensor to detect if the door is open).

As future steps, we want to demonstrate in a dynamic scenario of how the sensor data patterns (shapeoids) relate to semantic interpretations (CK rules). Moreover, an evaluation with a real dataset is foreseen to examine the scalability of the approach.