Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Decline in daily functioning usually involves the reduction and discontinuity in daily routines; entailing a considerable decrease of the quality of life (QoL). This is especially relevant for people that need assistance, as for instance elderly or disabled people [1]. Sometimes it may also hide pathological (e.g. Alzheimer) and/or mental (e.g. depression or melancholia) conditions.

To remotely monitor and support this kind of people, especially those that live alone, novel and intelligent systems are required. In particular, ambient and assisting living systems must be deeply investigated to facilitate improving autonomy, safety and social participation of people with special needs, normally elder and/or disabled. Therefore, there is the need of systems that allow monitoring indoor and outdoor activities of users to detect emergencies, recognize activities, send notifications, and provide a summary of all the relevant information related to user daily activities. Ambient assisting living solutions normally use sensor-based solutions [11]. Through the sensors, a lot of information are gathered and suitable support given to the end-user, accordingly. In fact, once data have been analyzed, the system has to react and perform some actions. On the one hand, the user (e.g. elderly or disabled people) needs to be keep informed about emergencies as soon as they happen and s/he has to be in contact with therapists and caregivers to change habits and/or to perform some therapy. On the other hand, monitoring systems are very important from the perspective of therapists, caregivers, and relatives. In fact, those systems allow them to become aware of user context by acquiring heterogeneous data coming from sensors and other sources.

In this chapter, we present a novel solution that provides all the above-mentioned functionalities. In fact, it advances existing telemonitoring systems because it combines wireless off-the-shelf sensors, activity recognition, and anomaly/emergency detection providing a daily summary of the relevant performed activities in real time. This information is automatically shared with the therapists to allow decision making and defining personalized rules according to the user profile. The overall telemonitoring system is part of the BackHome projectFootnote 1 and it has been developed according to a user-centered design approach in order to collect requirements and feedback from all the actors. The system was tested in two end-user’s home in Belfast. The overall experience in using the system by people with severe disabilities is discussed in the chapter.

The rest of the chapter is organized as follows. Section 2 summarizes relevant related work in the field of telemonitoring focusing in particular on intelligent monitoring solutions. Section 3 illustrates the proposed solution presenting all the main components. In Sect. 4, the BackHome experience is presented focusing on all the implemented services and the corresponding results. Section 5 ends the chapter with conclusions.

2 Background

2.1 Telemonitoring

In the literature, various studies and systems aimed at monitoring and supporting people that need assistance, especially detecting and overwhelming the worsening in daily activities, have been proposed. Several methods are limited to measuring daily functioning using self-report such as with the modified Katz ADL scale [21] or a more-objective measurement method as the assessment of motor and process skills [8]. Recently, solutions have been proposed to unobtrusively monitor activities of people that need assistance. In particular, sensor-based approaches are normally used [15].

Sensor-based systems rely on a conjunction of sensors, each one devoted to monitor a specific status, a specific activity or activities related to a specific location. Binary sensors are currently the most adopted sensors [18], even if they are prone to noise and errors [17]. Once all of the data have been collected, intelligent solutions that incrementally and continuously analyze the data to all the involved actors (i.e. therapists, caregivers, relatives, and end-users themselves) are required. Moreover, it is then necessary to identify if the person needs a form of assistance since an unusual activity has been recognized. This requires the adoption of machine learning solutions to take into account the environment, the performed activity and/or some physiological data [4].

2.2 Activity Recognition in Telemonitoring Systems

There is a large literature on recognition of activities at home [19, 24]. At the same time, we find a great variability in the settings of the experiments either in the number of sensors and their type, individuals involved or the duration thereof. Also noteworthy is the large amount of recognition techniques (supervised, either generative or discriminative; and unsupervised).

A former study [18] already points out some of the difficulties in discriminating daily life activities based only on binary sensors activities. The automatic recognition system was based on rules defined from the context and the duration of the activities to identify. The data of the study were obtained from 14 days of monitoring activities at home. Although promising accuracies were achieved for some activities, detection tasks such as “leaving home” were nothing less than satisfactory with 0.2 of accuracy. This was because the activities were represented by rules directly defined on the firings outputted by single sensors (i.e. door switches); so they did not contemplate that could be activated for other reasons and in varying times, which made reduce their discriminating power.

A more exhaustive work regarding the use of switch and motion sensors for tracking people inside home is found in [23]. Tests were done with up to three simultaneous users. High performances were reported by the trained tracking models. However it is interesting to note that this type of sensors experimented occasional lag between “entering” a room and triggering a sensor; making to decrease the performance of the tracking models.

In [9] a more complex template learning model (SVM) was used to automatically recognize among 11 different home activities. The proposed technique was integrated in different sliding window strategies (e.g. weighting sensor events, dynamic window lengths, or two levels of window lengths). They used 6 months of data from three different homes in which activities such as “entering” or “leaving home” were monitored. From the best experimental settings the authors claimed accuracy for “entering” home about 0.80 of F1-score but around 0.4 for “leaving” home tasks.

In a more extensive work [3] they use Naïve Bayes (NB), Hidden Markov (HMM) models and conditional random fields (CRF) for the activity recognition problem. In this study, seven smart environments were used and 11 different data sets were obtained. Several activities were attempted to be recognized. Among others, we highlight “entering” and “leaving home” as relevant for our approach. Although they did not report specific accuracies for these activities, authors claimed an overall recognition performance on the combined dataset of 0.74 for the NB classifier, 0.75 for the HMM model, and 0.72 for the CRF using 3-fold cross validation over the set of annotated activities.

In [14], authors proposed a hybrid approach to recognize ADLs from home environments using a network of binary sensors. Among the different activities recognized “leaving” was one of them. The hybrid system proposed was composed by using an SVM to estimate the emission probabilities of an HMM. The results showed how the combination of discriminative and generative models is more accurate than either of the models on their own. Among the different schemes evaluated, the SVM/HMM hybrid approach obtains a significant 0.7 of F1-score a notable better performance than the rest of approaches.

Finally, detecting “multiple” people in single room by using binary sensors was already studied in an early work [22]. In that work, authors proposed a method based in expectation maximization Monte Carlo algorithm. In a more recent article [13], high accuracy (0.85) were reported on detecting visits at home using binary sensors. In that approach, they used an HMM algorithm over the room events although not all rooms of the home were monitored.

3 The Telemonitoring and Home Support System

To monitor users’ activities, we develop a sensor-based telemonitoring and home support system (SB-TMHSS) able to monitor the evolution of the user’s daily life activity. The implemented system is able to monitor indoor activities by relying on a set of home automation sensors and outdoor activities by using Moves.Footnote 2 Information gathered by the SB-TMHSS is also used to provide context-awareness by relying on ambient intelligence [2]. Monitoring users’ activities through the SB-TMHSS gives us also the possibility to automatically assess quality of life of people [20].

The high-level architecture of the SB-TMHSS is depicted in Fig. 1. As shown, its main components are: home; middleware; intelligent monitoring system; and healthcare center.

Fig. 1
figure 1

Main components of the sensor-based system focused on the intelligent monitoring

3.1 Home

At home, a set of sensors are installed. In particular, we use presence sensors (i.e. Everspring SP103), to identify the room where the user is located (one sensor for each monitored room); a door sensor (i.e. Vision ZD 2012), to detect when the user enters or exits the premises; electrical power meters and switches, to control leisure activities (e.g. television and pc); and pressure mats (i.e. bed and seat sensors) to measure the time spent in bed (wheelchair). The system is also composed of a network of environmental sensors that measures and monitors environmental variables like temperature, but also potentially dangerous events like gas leak, fire, CO escape and presence of intruders. All the adopted sensors are wireless z-wave.Footnote 3 They send the retrieved data to a collector (based on Raspberry piFootnote 4). The Raspberry pi collects all the retrieved data and securely redirects them to the cloud where they will be stored, processed, mined, and analyzed. The proposed solution relies on z-wave technology for its efficiency, portability, interoperability, and commercial availability. In fact, on the contrary of other wireless solutions (e.g. ZigBee), z-wave sensors are able to communicate with any z-wave device. Moreover, we adopt a solution based on Raspberry pi because it is easy-to-use, cheap, and scalable.

We are also using the user’s smartphone as a sensor by relying on Moves, an app for smartphones able to recognize physical activities and movements by transportation. Among the activity trackers currently on the market, we select Moves because it does not need user intervention being always active in background.

3.2 Middleware

As mentioned above, the telemonitoring system, by definition needs to be interoperable, extensible and scalable. In order to deal with such requirements the design of the system architecture was based on the service oriented architecture framework (SOA).Footnote 5 Thus, each component of the system was built as web service itself and communicate through the REST protocol. This protocol is selected since it works over HTTP/s and allows relaxed interoperability between different components. The middleware, which is the set of core components of the architecture, is composed by independent web services that provides not only scalability and interoperability but also securely interconnect the main components of the system, i.e. the home, the intelligent monitoring system, as well as the healthcare center. Its main functional components are:

  • The API façade, which encapsulates a set of information services provided by the intelligent monitoring system in order to be consumed by external applications (i.e. the healthcare center). Internally, this component securely dispatches all requests of information from outside and redirects them to the enterprise service bus. This service contains policies and protocols to handle load balancing and concurrency.

  • The enterprise service bus (ESB), which orchestrates the communication between internal components avoiding ad hoc communication. The ESB makes the system more extensible and flexible to changes by doing all components communicate through it.

  • The security manager (SM), which provides mechanisms to user authentication, and services authorization. This component manages the session IDs to the system as well as the UUIDs that identify uniquely the users and the services which have access to.

  • Notification service (NS), which is a mechanism for sending asynchronously data (e.g. events, emergencies, rule actions) generated from the consumer (e.g. intelligent monitoring) to a receiver or client (e.g. health care center).

3.3 Intelligent Monitoring

In order to cope with data necessities of the actors of the system (i.e. therapists, caregivers, relatives, and end-users themselves), an intelligent monitoring (IM) system has been designed. It is aimed to continuously analyzing and mining the data through five-dimensions: detection of emergencies, activity recognition, event notifications, summary extraction, and rule triggering. In order to achieve these objectives, the intelligent monitoring system is composed of the following modules (see Fig. 2): PP, the pre-processing module to encode the data for the analysis; ED, the emergency detection module to notify, for instance, in case of smoke and gas leakage; AR, the activity recognition module to identify the location, position, activity- and sleeping-status of the user; EN, the event notification module to inform when a new event has been detected; SC, the summary computation module to perform summaries from the data; and RA, the risk advisement module to notify risks at runtime.

Fig. 2
figure 2

The flow of data and interactions among the modules in the proposed approach

3.3.1 Pre-processing

IM continuously and concurrently listens for new data. The goal of PP is to pre-process the data iteratively sending a chunk c to ED and also RA according to a sliding window approach. Starting from the overall data streaming, the system sequentially considers a range of time \(|t_{i} -t_{i+1}|\) between a sensor measure \(s_{i}\) at time \(t_{i}\) and the subsequent measure \(s_{i+1}\) at time \(t_{i+1}\). Thus, the output of PP is a window c from \(t_{s}\) to \(t_{a}\), where \(t_{s}\) is the starting time of a given period (e.g. 8:00 a.m.) and \(t_{a}\) is the actual time. Thus, each chunk is composed of a sequence of sensor measures s; where s is a triple \(\langle \textit{ID}, v, t\rangle \), i.e. the sensor ID, its value and the time in which a change in the sensor status is measured. Figure 3 shows an example of a chunk composed by four sensors measures.

Fig. 3
figure 3

Example of a chunk composed of four sensor measures

3.3.2 Emergency Detection

ED module aims to detect and inform about emergency situations for the end-users and sensor-based system critical failures. Regarding the critical situations for the end-users, an emergency is risen when specific values appear on c (e.g. gas sensor ID, smoke sensor ID). Regarding the system failures, ED is able to detect when the end-user’s home is disconnected from the middleware as well as a malfunctioning of a sensor (e.g. low battery). The former is implemented by a keepalive mechanism in the Raspberry pi. If no signals are received from the Raspberry pi after a given threshold, an emergency is risen. The latter is implemented by using a multivariate gaussian distributions of sensor measurements on c. If the corresponding total number of measures is greater than a given threshold, an emergency is risen.

Each emergency is a pair \(\langle s_{i}, l_{\varepsilon i}\rangle \) composed of the sensor measure \(s_{i}\) and the corresponding label \(l_{\varepsilon i}\) that indicates the corresponding emergency (e.g. fire, smoke). Once the ED finishes the analysis of c, the list of emergencies \(\varepsilon \) is sent to the middleware, whereas c, filtered from the critical situations, is sent to AR.

3.3.3 Activity Recognition

In the current implementation, the system is able to recognize if the user is at home or away and if s/he is alone; the room in which the user is (no-room in case s/he is away, transition in case s/he moving from a room to another); the activity status (i.e. active or inactive); and the sleeping status (i.e. awake or asleep).

Fig. 4
figure 4

The hierarchical approach in the activity recognition module

To recognize if the user is at home or away and if s/he is alone, we implemented a solution based on machine learning techniques [16]. The adopted solution is a hierarchical classifier (see Fig. 4) composed of two levels: the upper is aimed at recognizing if the user is at home or not, whereas the lower is aimed at recognizing if the user is really alone or if s/he received some visits. The goal of the classifier at the upper level is to improve performance of the door sensor. In fact, it may happen that the sensor registers a status change (from closed to open) even if the door has not been opened. This implies that AR may register that the user is away and, in the meanwhile, activities are detected at user’s home. On the contrary, AR may register that the user is at home and, in the meanwhile, activities are not detected at user’s home. Thus, we first revise the data gathered by AR searching for anomalies, i.e.: (1) the user is away and at home some events are detected and (2) the user is at home and no events are detected. Then, we validate those data by relying on Moves, installed and running on the user smartphone, and the supervision of the user. Using those as an “oracle”, we build a dataset in which each entry is labeled depending on the fact that the door sensor was right (label “1”) or wrong (label “0”). The goal of the classifier at the lower level is to identify whether the user is alone or not. The input data of this classifier are those that has been filtered by the upper level, being recognized as positives. To build this classifier, we rely on the novelty detection approach [10] used when data has few positive cases (i.e. anomalies) compared with the negatives (i.e. regular cases); in case of skewed data.

To measure the activity status, we rely on the home automation sensors. By default, we consider as “active” the status of the user when s/he is away (the corresponding positions are saved as “no-room”). On the contrary, when the user is at home, AR recognizes s/he as “inactive” if the sensor measures at time \(t_{i}\) that user is in a given room r and the following sensor measure is given at time \(t_{i+1}\) and the user was in the same room, with \(t_{i+1} - t_{i}\) greater than a given threshold \(\theta \). Otherwise, the system classified the user as “active”.

Finally, sleeping is currently detected by relying on the presence sensor located in the bedroom and the pressure mat located below the mattress. In particular, we consider the presence of the user in that room and no movements detection (i.e. the activity status is “inactive”) together with the pressure of the mattress.

Thus, the output of AR is a triple \(\langle t_{s}, t_{e}, {l}\rangle \), where \(t_{s}\) and \(t_{e}\) are the time in which the activity has started and has finished, respectively, and l is a list of four labels that indicates: the localization (i.e. home, away, or visits), the position (i.e. the room, no-room, or transition), the activity status (i.e. active or inactive), and the sleeping status (i.e. awake or asleep). To give an example, let us consider Fig. 5 where the same chunk of Fig. 3 has been processed by AR.

Fig. 5
figure 5

Example of a chunk after the AR processing

3.3.4 Event Notification

By relying on a set of simple rules, EN is able to detect events to be notified. Each event is defined by a pair \(\langle t_{i}, l\rangle \) corresponding to the time \(t_{i}\) in which the event happens together with a label l that indicates the kind of event. In particular, according to user requirements and therapist’s focus groups, we decided to detect the following kind of events: leaving the home, going back to home, receiving a visit, remaining alone after a visit, going to the bathroom, going out of the bathroom, going to sleep, and awaking. These events allow to study activity degradation as well as improvement/worsening of the overall quality of life. Nothing prevents to consider further notification and/or to change them in case requirements change or further needs arise. Following the example, in Fig. 3, an event is the pair \(\langle \)2014-02-24 10:31:55\(, \textit{going}\,\textit{to}\,\textit{the}\,\textit{bathroom}\rangle \).

3.3.5 Summary Computation

Once all the activities and events have been classified, measures aimed at representing the summary of the user’s monitoring during a given period are performed. In particular, two kinds of summary are provided: historical and actual. As for the historical summary, we decided to have a list of the activities performed during (i) the morning (i.e. from 8:00 a.m. to 8:00 p.m.), (ii) the night (i.e. from 8:00 p.m. to 8:00 a.m.), (iii) all the day, (iv) the week (from Monday morning to Sunday night), as well as (v) the month. In particular, we monitor: sleeping time; time spent outdoors; time spent indoors; time spent performing indoor activities; time spent performing outdoor activities; number of times spent in each room; and number of times that the user lefts the house. As for the actual summary, we are interested in monitoring: the room in which the user is; if the user is at home, or not; the number of times that s/he leaves the home; sleeping time; activity time; and number of visits per room.

As a final remark, let us note that all emergencies, activities, notifications, and summaries are stored in a database to be available to all the involved actors.

3.3.6 Risk Advisement

RA is aimed at advising therapists about one or more risky situations before they happen. The module executes the corresponding rules at runtime according to the sequence of sensor measures coming from the PP as well as the summary provided by the SC. Figure 6 sketches how RA works and which are its main components and interactions. Through the healthcare center, therapists access to an ad-hoc user interface to define the rules corresponding to risks. Those rules are automatically coded in a suitable language, namely ATML [7], and then translated in DRL by the Rule Builder and stored in the knowledge-based of rules (KBR). RA continuously processes data coming from the other modules of the IM and acts according the defined rules in KBR. In particular, it analyzes the entire chunk c from PP, the list of activities a from AR, and the complete summary \(\varSigma \) from SC. The actions \(\{a_{r}\}\) triggered by RA are sent to the middleware that is in charge of actuate in consequence sending the corresponding advisement to the healthcare center.

Fig. 6
figure 6

The RA module functioning

To implement the RA we relied on Drools,Footnote 6 a rules management system that provides a rule execution server, and a web authoring and rules management application. A rule is a quadruple \(\langle i, v, o, a_{r}\rangle \), where i is the item that has to be verified (e.g. a room, the number of slept hours) according to a given value v (e.g. bedroom, 4 slept hours); o is the logic operator (i.e. and, or, not) and a “null” operator in case there is only one term; and a is the action to be performed (i.e. send a notification, an alarm, or an email).

3.4 The Healthcare Center

The healthcare center receives notifications, summaries, statistics, and general information belonging to the users through a web application. Its goal is to keep informed therapists and caregivers about emergencies as soon as they happen and to proactively inform them about changes in user habits and/or to perform some therapy. In this way, therapists and caregivers become aware of user context by acquiring heterogeneous data coming from sensors and other sources.

The healthcare center has been implemented as a Web-based application and provides user management, rehabilitation task management, therapy assessment, rule definition, statistics on system usage, as well as for communication between therapists and user.

A modular approach was considered from the very starting point of the healthcare center design, which led to the definition of a loosely coupled system where each of its components keeps its logic as self-contained as possible [6]. This design strategy is crucial in order to manage changes while reducing its overall impact in the rest of the platform. As a result, each of the main functionality is encapsulated in a self-contained module, which in turn, is managed by the platform infrastructure services. Those base infrastructure services are in charge of the definition of a common application context where every module is registered while providing cross-platform functionality as well.

4 The BackHome Experience

The overall system presented in this chapter is part of the EU project BackHome.Footnote 7 The project is aimed at moving brain computer interfaces (BCIs) from being laboratory devices for healthy users toward practical devices used at home by people with limited mobility. This requires a system that is easy to set up, portable, and intuitive. Thus, BackHome aims to develop BCI systems into practical multimodal assistive technologies to provide useful solutions for communication, Web access, cognitive stimulation and environmental control, and to provide this technology for home usage with minimal support. These goals are attained through three key developments, each of them advancing the current state of the art [5, 12]: (i) practical electrodes, with the delivery of novel BCI equipment which sets a new standard of lightness, autonomy, comfort and reliability; (ii) easy-to-use software tailored to peoples needs, with a complete range of highly desirable applications finely tuned for one-click command and adaptive usage; and (iii) telemonitoring and home support to remotely assist independent use, with remote services to plan and monitor cognitive rehabilitation and pervasively assess the use of the system and the quality of life of the individual. The development followed a user-centered design approach in order to collect requirements and feedback from all the actors.

A total of four participants were recruited in Belfast for a 6-week home based evaluation of the system. For the sake of anonymity let us refer to the final users as Home User 1 (HU1), Home User 2 (HU2), Home User 3 (HU3), and Home User 4 (HU4). Actually, only two of them concluded the 6-week evaluation period. In fact, attempts to set up the system in HU2’s home failed due to the restricted Internet connection; whereas HU4 became ill after the installation of the BCI and did not recover in time to participate in the home based testing. Thus, the evaluation has been performed only with HU1 and HU3.

In each home, we installed a presence sensor in each room a door sensor in each entrance, and two power meter switches to control a light and a radio through the BCI. Environmental sensors have not been used in BackHome because of the user requirements, moreover, pressure mat sensors have not be installed due to privacy constraints inside the project.

Regarding the IM and due to the BackHome user requirements and end-user characteristics, in addition to PP (that is essential), AR, SC and RA was implemented.

AR was first evaluated with 2 able-bodied users in Barcelona, as reported in [16]. The hierarchical classifier showed an improvement of 15 % of accuracy with respect to a rule-based solution (see Table 1). To highlight the performance of the proposed approach, let us consider the Fig. 7 that shows a comparison between the real data, labeled during the validation phase (on the left), and the data classified by relying to the approach proposed in this chapter (on the right).

Table 1 Results of the overall hierarchical approach with respect to the rule-based one
Fig. 7
figure 7

Comparison between real labeled data and data classified by the hierarchical approach

Fig. 8
figure 8

Indoor user habits recognized by the SB-TMHSS for HU1 (the two graphs on the top) and HU3 (the two graphs on the bottom) for a period of 1 month

Fig. 9
figure 9

An example of data coming from the sensors as shown to the therapist

Fig. 10
figure 10

Trigger definition therapist interface

Figure 8 shows the results belonging to 1 month of indoor monitoring; outdoor monitoring is reported similarly. In particular, for each user the following information is given: when the user was at home, away, or received visits day by day and a summary of the different locations in which the user was during a period of 1 month.

The Healthcare center in BackHome, called Therapist Station, allows therapists to remotely manage the end-user. Among the overall set of functionality provided by the therapist station, let us consider here the most relevant for monitoring and supporting.

Thanks to the IM, the Therapist Station daily receives the summary of the information regarding the daily-life activities of the user (computed by SC). Figure 9 shows an example of data coming from the sensors regarding both indoor and outdoor activities.

Moreover, therapists defined simple rules, such the one shown in Fig. 10. The therapist set a rule to raise an alarm if the user spends more than ten hours at bed in a day.

Overall the healthcare center was viewed in a positive light and considered to be an asset to daily practice. On average, the 36.63 % of the therapists evaluated as positive (4) the overall healthcare center and the 44.22 % as very positive (5), making a total of 80.86 % of positive and very positive evaluation. The Therapist Station was thought to be “modern and is relatively easy to use” and that the “site is laid out well”. The focus of the BackHome therapist station was considered to be a “very useful starting point when a client returns home from hospital and is very dependent”. Additionally, it might be useful to think of other populations and applications for the therapist station, “the Therapist Station is an excellent platform, could be used with a range of clients; in paediatrics would be good for assigning home programmes etc”.

5 Conclusions

In this chapter, we presented a sensor-based system aimed at detecting emergencies, recognizing activities, sending notification as well as collecting the information in a summary and executing actions triggered by means of a rule-based engine. The goal of the implemented system was to monitor and support people that need assistance and to constantly give a suitable feedback to therapists and/or caregivers. They have access to information about the evolution of the status, behavior and habits of the corresponding user, thanks to a web-based application. Under the umbrella of BackHome, the system has been installed and tested in two disabled people’s homes where is currently running and remotely tested by about 80 therapists.

The BackHome telemonitoring features have proven to be an effective way of remotely reporting information about end-user habits, quality of life and detailed usage of the BCI environment. Those telemonitoring capabilities of the BackHome system provide tools and means for therapists and technical experts to support end-users and caregivers at home in a reactive but also in a proactive way.