Distributed Context-Aware Applications by Means of Web of Things and Semantic Web Technologies

Abstract

Ambient Assisted Living aims for providing context-aware and adaptive applications to assist elderly and impaired people in their everyday living environment. This requires the recognition of user intentions and activities by means of multi-modal and heterogeneous sensing devices. An unresolved problem is the lack of interoperability and extendibility of the setting. Moreover, to achieve adaptivity, a context-aware environment requires to consider user impairments as well as capabilities and to monitor non-stop user activities. This complicates an on the fly integration of new sensing devices and applications. Furthermore, a flexible and expressive domain model for describing and processing user profiles, intentions and activities, is required. Our approach to overcome these integration and modeling problems, is to use the Web of Things and Semantic Web technologies. Another unresolved problem concerns the security of the collected sensitive data. To avoid the manipulation of applications by an unauthorized access, we introduce ontology based security policies for context-aware applications, considering their managed context data.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction/Motivation

Context-aware platforms and assisting applications -summarized under the term Ambient Intelligence (AmI)- have mainly the goal to recognise the user intention, to achieve adaption. The objective is that users are not required to learn how to use the assisting system, but the assisting system does have to learn and to adapt itself to the user and his/her needs [2]. The environmental data have to be sensed and to be mapped into a machine interpretable domain model, enriched usually with semantics so that applications can process these data. In AmI use cases this is a common applied approach to allow context-awareness and adaption. A subset of AmI is Ambient Assisted Living (AAL). The target group of AAL are elderly and impaired people who have problems to overcome the complexity of daily flows. For this reason, a context-aware application must be able to recognise the user activity and to infer by means of the activity and context recognition the user intention to provide an adequate assistance by means of actions. The user intention specifies the current need and objective of the user in an appropriate context while actions enable changes in the environment. The challenge here is to provide a domain model of user intentions and activities and to enable, a reasoner by this model to deduce the matching actions. Currently there seems to be no approach which is addressing adaption by means of user intention and activity recognition. Moreover, a context-aware environment consists of multi-modal and heterogeneous devices for sensing user-specific and environmental data. The concerned devices can be separated into wearable and stationary devices. A wearable device is worn in common by the user to sense for instance his/her vital parameters and to monitor his/her health state while stationary devices are sensing state changes in the environment. For this reason, an additional requirement of a context-aware environment is to provide a standardised description of these devices to enable their integration. But current approaches are lacking of mechanisms for a simplified integration and interoperability. Our approach to use Web of Things (WoT) and Semantic Web allows to integrate devices on the fly. However not just devices have to be integrated on the fly, but also context-aware applications.

In this work, we consider context-aware applications as a composition of services which execute appropriate tasks to assist according to the user characteristics, activities and intentions. It is self-explanatory that applications impact the user context by actions which they initiate to accomplish state changes in the environment. For this reason, we need an approach to express their functionality by means of action descriptions and to relate them with the device functionality as well as with the user intention. From the perspective of a context-aware application, a user intention has always to lead to at least one action which is satisfying the user intention. In our previous work [5], we present a service description which is linking user intentions to matching actions. The advantage is that the knowledge and functionality of context-aware applications can be increased by new incorporated service descriptions, serving for appropriate use cases.

We also have to consider that context-aware applications collect the entire time data about the user and his/her context. These data are sensitive data as they allow derivations about the user and his/her current activity and situation or even about his/her health state. This implies that we need a mechanism to protect these data from unauthorized access and manipulation. Current attempts to provide ontology-based security policies do not satisfy the requirements of an AmI environment. The overall objective of this work is to overcome the mentioned problems and challenges and to enhance state of the art approaches.

Section 2 gives an overview of state of the art techniques to distinguish this work from current approaches. Section 3 deals with the general problems of context-aware environments and introduces our contribution for developing distributed context-aware applications by means of WoT and Semantic Web technologies. Section 4 presents the research methodology and the technical approach of this proposal. Section 5 gives a brief overview of the current work state. Section 6 introduces the evaluation plan of our work and Sect. 7 summarizes the discussed topics and gives an outlook to further open issues and research questions.

2 State of the Art

There are a lot of efforts in developing context-aware AAL environments. For instance, Hristova et al. introduce in [4] an Ambient Home care system which implements context-aware services with reasoning functionalities. The framework uses no ontology for modelling the AAL environment and its participating instances (users, devices, services, etc.). The problems of this approach are that every application developer does have to learn different APIs to implement their application logic. It would be more comfortable if every developer just needs to follow one standarised API, which is extendible in a semantic way. The presented use cases of [4] do not evaluate user capabilities and impairments. The reasoning is done by a rule-based engine, modelling the context with key-value pairs. The engine in [4] is very restrictive because the context of a user is more complex than simple rules can cover by key-value pairs. Hristova et al. also state that privacy and security are not supported by their framework. We want to address this issue with our approach to model ontology-based security policies.

Bacciu et al. discuss in their work [1] an AAL platform for prohibiting sedentariness and unhealthy dietary habits by means of activity recognition. The sensed data is aggregated to allow the platform to recognise social and physical activities [1]. However the mentioned platform does not consider user intentions and profiles. An adaption mechanism is not given. In contrast to our approach, the given activity recommendations are restrictive because they are based on general expert assumptions and do not consider the individuality of the user. The DemaWare platform in [8] aims for supporting people with dementia, i.e. the platform and its ontology is addressing a very specific domain of AAL. According to [8] the platform is restricted in terms of the used hardware and includes a weak provision of context information. The used ontology is just matching to people with Dementia. So the field of application is very specific and restricted while our approach aims at being applicable in the entire field of AAL.

Regarding security policies, the work in [7] presents an approach to compose privacy policies, based on Semantic Web technologies. The engine in [7] generates composition rules and deduces implicit terms by means of the data usage context. However the mentioned approach provides no rules for other contexts as for the presented one. Moreover their approach provides no storage location regarding the appropriate data [7]. Considering our AAL use case, it is necessary to enable storage location information, because every context-aware application is distributed due to security aspects and provides and manages its context data itself. If an application wants to access the data of a device or another application, the access must be regulated. The approach in [7] provides no user feedback if the policy composition fails. As a consequence to the mentioned problem, our approach aims at giving the user an appropriate feedback, if the security policies are not obeyed.

3 Problem Statement and Contributions

There are criteria and aspects for implementing a context-aware infrastructure in the AAL domain. Security is such an aspect. If the platform is attacked, the entire system and the privacy of the user is concerned and in danger. An intended manipulation of these sensitive data can have serious matters for the user. An offender could infer by means of the gathered data the activities and habits of the user and exploit this knowledge to mean ill. The contribution of this proposal is to devise semantical security policies, for the handling of collected and distributed context data. Another aspect is that an assistive environment consists of different sensing devices and user interfaces which can be used through different communication channels like voice, gaze and haptic interaction. The usage of these devices and user interfaces can require different abilities which the user might not have because of his/her impairments. Therefore the recognition of and adaption to user intentions and activities requires to happen in a natural and intuitive way by means of sensed activities and user profiles. Considering these aspects and challenges as well as our work in [5], this work answers the following research questions:

• 1. Do ontology-based security policies for context-aware applications in the AAL domain prevent misuse and manipulation of sensitive context data?

• 2. Do lightweight ontology languages such as RDF(S) provide the necessary expressivity for representing devices and their functionalities in AAL environments? See also [5]

• 3. How can user intentions, user activities as well as environmental aspects such as device functionalities be semantically linked together using lightweight ontological semantics to improve the integration of devices and the user intention and activity recognition? See also [5]

With respect to these research questions, we present a platform that facilitates the integration and interoperability of heterogeneous devices by means of the WoT and Semantic Web technologies. Furthermore, we devise a context-aware application called Sherlock that recognises the user intention and activity by means of the mentioned Service Description. Moreover, an objective is to enable the Sherlock application to implement ontology based security policies.

4 Research Methodology and Approach

The methodology of this work comprises to define the target group and to understand by means of workshops and interviews with the user group and domain experts, the lifecycle of AAL environments and use cases. Moreover, we analyse Ethical Legal Social Implications (ELSI) to infer general user requirements of assistive systems. Considering ELSI and state of the art approaches, we determine which aspects an assisting system needs to satisfy and which of these aspects are not yet achieved by current approaches. In a next step, we create a domain model by means of a light-weight ontology. For this reason, we introduce -according to the WoT concept- the term Things of Interest. Things of Interest are physical or abstract objects of the intended use cases. The semantical description of these Things is the basis for modelling an AAL domain ontology which is considering the impairments and capabilities of the user as well as the functionality of applications and devices. For a future evaluation of our approach, we implement a WoT server and create by means of the proposed domain model a machine-processable AAL environment. We cite the Light Switching use case as an example use case. The implementation of this use case requires a Light Service description, which is used by the Sherlock application to deduce, if the user does have the intention to turn a lamp on or off. The objective is to enhance the functionality of Sherlock by rule description languages (such as SWRL¹ or SPIN²). For this reason, we need to consider and analyse existing rule description languages, regarding their expressivity. There might be sometimes fuzzy situations, i.e. rule-based languages are not suitable in this cases. So we have to investigate in fuzzy logics and in pattern recognition methodologies. But before we can apply these rules, we require to model sequences of user activities. Therefore, we have to consider different models such as Donald Norman’s Seven Stage Model, which are describing established user activities.

The planned security policies also require to follow rules. These rules enable the Sherlock application to recognise security violations.

We devised a first version of a semantical domain model, describing Things of Interest. For this reason, we considered different existing ontologies such as the SSN³ and DUL⁴ ontology. But we noticed that these ontologies do not cover all aspects which are necessary for the considered AAL application domain.

Figure 1 depicts our approach to model a Lamp device and the linking of its inherent functionalities and location. The location of Lamp1 references further relevant devices (AmbientLightSensor1), which are sensing relevant context data for the participating applications. The composition of the sensed observations allows the derivation of the user context.

Fig. 1.

A snippet of the device ontology as graph representation [5]

For the sake of simplicity, it is necessary to provide a tool for generating ontology instances which can be accessed immediately during runtime. The approach in this proposal is to use Semantic MediaWiki (SMW) because it provides forms and templates for creating semantically annotated data. Context-aware applications get the possibility to request and subscribe for context data depending on their need and interest without knowing the (details of the) different context resources. The knowledge of the domain is distributed by different context-aware applications and devices, managing their own context and information. To accomplish this, every service requires to provide at least one Service Description, which the applications can use for reasoning. This Service Description is based on the WoT recommendation. Every Thing whether physical or abstract in the WoT recommendation is described by [6]:

• (i) properties, describing the Thing composition

• (ii) events, the Thing is interested in or which can be triggered by a Thing

• (iii) actions, which can be invoked on a Thing, if matching events are occurring

Fig. 2.

The Light Service description linking user intentions with actions [5]

Furthermore, the WoT approach allows the exchange of data by open Web standards. The goal of this approach will be to use the WoT for describing and managing Things of Interest to make them findable, sharable, accessible and composable [3]. The knowledge of the AAL environment is distributed in this way by a net of smart applications, managing and sharing their acquired context data. Figure 2 depicts the Light Service description. The Light Service consists of Capabilities, the service provides and Intentions the application is interested in. The service description expresses by linking Capabilities and Intentions, the ability of a service to serve for an appropriate task and that this task is depending on the linked user intention. The advantage of the mentioned approach is, that an application can request various service descriptions to expand its knowledge about the domain and context. A user Intention is described by Actions and Rules. Every Rule consists of pre-conditions (triples) which are denoting possible events. In Fig. 3, the SwitchLightOff intention references the SwitchLightOff action and three triples describing the context conditions to recognise the presented Intention. Moreover, our approach aims at achieving the security and privacy of this context-aware infrastructure by distributing the acquired data to various context-aware applications which are managing and creating their own context data. Therefore every application requires to follow semantical security policies. For the evaluation of our approaches, we extend the Sherlock application with the proposed functionality. Moreover, in future steps, we adapt and enhance the work of [7] so that Sherlock might follow the devised semantical security policies and warn the user if a security policy was violated by external influences. Furthermore, we want to extend our domain ontology to allow the semantical description and evaluation of user profiles. A user profile constitutes of relevant user characteristics, such as user capabilities and disabilities. One possibility is to define and integrate semantical user profiles based on the WHO International Classification of Diseases Guidelines ⁵ to derive compensating actions in appropriate situations. Our objective is to enable context-aware applications to include user profiles in their reasoning process.

Fig. 3.

The Light Switching user intention with its rules. See also [5]

5 Prelimenary Results

The target group and involved people were interviewed in two workshops to determine by means of ELSI the technical requirements and to specify useful use cases. Three use cases are defined.

• 1. a Light Switching use case, controlling lamps by the user intention and activity.

• 2. the recognition of emergency situations (for instance fire in the living environment).

• 3. the controlling of an autonomous wheelchair by gaze patterns and user context.

A first prototype of the Sherlock application and the components in Fig. 4 were already implemented and have to be extended. Furthermore, the Light Switching use case was implemented to demonstrate our approach. The Sherlock engine is running on a mobile phone and is communicating with the user. In a first step, Sherlock requests the Light Service description from a Query Generator component. This component uses the vocabulary of our introduced light-weight ontology and generates from the application requests SPARQL queries. The Triple Store imports the RDF instances of the SMW. The Configurator transforms created device descriptions from the SMW into appropriate IoT system configurations to introduce the devices. After Sherlock has received its service description, it registers at the WoT Server for appropriate sensor events. An Eye Tracking Glass (ETG) is measuring the gazes of the user who is sitting in an autonomous wheelchair. If the user focusses for instance a Lamp, this sensor event is sent to the WoT. The WoT uses different Adapters to transform events into a standard structure, which the Sherlock engine can understand. Sherlock requests after the sensor event from the WoT server further context information according to its Light Service description and the contained intention rules. By means of this context information, Sherlock derives the current intention of the user and asks the user, if the inference was right. The user confirms Sherlocks conclusion and Sherlock triggers the appropriate action, which is forwarded to the WoT server for switching the appropriate devices, i.e. Lamp1.

Fig. 4.

The proposed architecture of our approach. See also [5]

A first version of the presented domain ontology with the service description was defined and inserted in SMW. A general message structure was already defined for exchanging messages between different components to abstract the different resources and to achieve a common understanding. The message contains all relevant information of the underlying ontology.

6 Evaluation Plan

The evaluation will be conducted as part of the AICASys⁶ project. We plan to make lab and field trials with people of the target group in real living environments. An objective is to evaluate the usability, feasibility and reliability of the context-aware Sherlock application by measuring and statistically evaluating the success factor of the user intention and context recognition. We want to show in our evaluation that (a) the semantically specified security policies prevent the misuse of sensitive data, (b) the user intention detection is improved (at least 80 % reliability) by means of introducing user profiles and (c) the integration of various devices and applications is simplified for not technically minded persons compared to other related work approaches. It is planned to compare the improvements or degradations with other similar approaches to show that the presented approach was successful or not.

7 Conclusion

We introduced service descriptions which are providing the knowledge of our context-aware applications. Context-aware applications need these service descriptions to link user intentions to appropriate actions. We discussed that the detection of user intentions and activities needs also to consider user characteristics by means of user profiles. In our approach context-aware applications are distributed by running independent on different devices and managing their own context data. The exchanged context and user data is sensitive and requires to be protected from unauthorized access. For this reason, we proposed to integrate security policies in our ontology, so that every application needs to follow these policies. The utilisation of the WoT concept offers the solution for overcoming integration problems of heterogeneous data sources. Moreover, the Semantic Web opens a way to map the real world into a machine understandable format so that reasoning techniques can be applied to infer the context and intention of an appropriate user. For demonstration purposes we introduced our context-aware and extendible Sherlock application. However, this proposal does not address to solve problems concerning the realtime reaction of applications to different events. Furthermore, it is was not discussed how to enhance the quality of context data. In an AAL environment it is necessary to evaluate and assure the trustworthiness of data sources to warrant a reliable context-awareness. In future work we have to consider what happens if fuzzy and uncertain situations are detected. A conceivable approach is to apply machine learning and pattern recognition to combine it with rule-based approaches. One enhancement can be the implementation of a mood classifier. This would allow the Sherlock engine to learn from context data to motivate the user to do some activities. But the mentioned challenges remain for future research work as part of this Ph.D.