1 Introduction

The old-age dependency ratio indicator is one of the most widely used indicators for monitoring changes in populations. According to prospects of the United Nations, by 2050, 16.66% of the world population will be over the age of 65, up from 9.09% in 2019 (United Nations 2019). This situation can be mainly explained by the fertility decline and increased longevity combined with other factors such as the pappy-boom effect in Europe (i.e. post-war children who are now reaching retirement age). High formal healthcare costs and population aging will put increased financial pressure on old-age public and private support systems. In the U.S., about 80% of elderly people are living with at least one chronic disease, and an estimated 5.4 million senior citizens are suffering from Alzheimers disease (Parisa and Alex 2013). To face the growing demand for assistance and healthcare, it is now unanimously acknowledged that information and communication technologies (ICTs) will undoubtedly play a determining role. A recent survey on ICT and the Internet of Things (IoT) concluded that for most diseases and behavioral disorders, current technologies can meet the requirement in terms of reliability, connectivity, data rate, and latency (Muhammad Mahtab et al. 2018). However, healthcare systems are required to provide advanced diagnostic and prognostic solutions and move towards more proactive approaches such as prevention and smart home care services. For a wide end-user acceptance (Debajyoti et al. 2018), smart homes are preferred by elderly people for several reasons including independent living even with manageable disabilities, the promotion of the overall welfare, the considerable reduction in the length of stays in hospitals and healthcare institutions, the feeling of being self-sufficient despite complex health situations, etc.

Ambient intelligence (AmI) is an emerging discipline that promises to revolutionize everyday human life. In AmI, devices are spreading everywhere (ubiquity), enabling humans to interact with their environments and including context management in a smart and transparent way. In the e-health realm, it is much applied to enhance the daily life activities of people (Hill et al. 2015; Rahman et al. 2016; Mshali et al. 2018). AmI research field is based on advances on sensor networks, ubiquitous computing, and artificial intelligence algorithms. For instance, home and wearable sensors help to recognize the person’s activities and more data can be acquired by integrating the sensors in a more complete Internet of Things (IoT) system (Valentina et al. 2019). Elderly people smart homes are designed to track the person’s behavior and detect various distress situations like falling, changes in the heart rate and abnormal behaviors (Salisu Wada et al. 2019; Abubaker et al. 2019). Such environments help to gain valuable knowledge with a rich set of contextual data (Wickramasinghe et al. 2017). According to the complexity of the occupant’s behavior, retrieving meaningful information from a large amount of data is a task that is proven to be complex and sensitive. The heterogeneity and ubiquity of data make the context as a key concept for modeling a system that requires a relevant description of the environment. To design a context-aware system, the information must be well defined and modeled in a suitable form. In this context, knowledge-based methods have been exploited to improve suitability and adaptability for heterogeneous contextual situations. These methods were used to formalize, represent and structure the context of the activity such as logic-based approaches (Aysenur et al. 2016), rule-based systems (Thanos et al. 2015), and ontological models (Georgios and Ioannis 2017). The approaches can be classified from the less sophisticated one such as key-value modeling to the most powerful and complex such as ontology-based modeling (Rodríguez et al. 2014). Ontologies-based modeling provides a semantic representation of the contextual information of activities in a hierarchical way with a specification of the reasoning mechanism (Ye et al. 2015; Noor et al. 2018; Hossain et al. 2017; Gayathri et al. 2017; Bobillo and Straccia 2016; Wemlinger and Holder 2018).

Despite their advantages, ontology-based modeling can be computationally expensive and have other limitations that must be tackled (Rodríguez et al. 2014). For instance, the semantics cannot directly manage vague pieces of knowledge which are inherent to several real-world problems (Bobillo and Straccia 2016); they required a good knowledge of engineering tasks to define a formal semantic of human activities (Bae 2014); they have a limited capacity for changing contexts ; they require several means of operators using the ontological language. In smart environments, as the context is captured by sensors, if one sensor data is lacking the recognition, the ontology-based model can not be qualified to discover the missing activity.

In order to overcome the previous modeling issues, the Elderly people by room activities description language (EBRADL): a context-sensitive descriptive language for designing the behavior of a person evolving in a smart space is proposed. EBRADL allows non-technical users to encode essential knowledge about user activities and provide intelligent reasoning. In order to represent and structure the activities-context, the language considers several criteria such as time, location and object. Besides the proposed language focuses primarily on describing the behavior of elderly people, the important criterion in structuring daily activities which is the person’s geriatric profile is considered. The profile contains contextual data such as diseases, age, and autonomy. The flexibility EBRADL is an asset in considering different categories of users and different description levels in support of independent living regardless of the age of monitored persons. Bearing in mind that the goal is to understand and recognize the behavior based on these sensory data, this could be achieved by using common sense knowledge of the person’s daily activities. In this work, a focus is made on defining modeling based on a meaningful, relevant and useful representation of activities of daily living (ADL). For that purpose, a new modeling language is proposed for activities, scenarios, and behavior based on a well-defined set of rules and criteria. Generally, to build a complete activity recognition system it is necessary to consider the structuring and recognition of the behavior together. The main purpose of recognizing the activities is to detect or recognize the usual activities and behaviors of persons in their daily lives. Several recognition models were proposed in order to predict and recognize the behavior: probabilistic and statistical techniques (e.g. Hidden Markov-based models) (Yu et al. 2018; Ronao and Cho 2017; Wu et al. 2016), computational intelligence techniques (e.g. neural networks) (Hussein et al. 2014; Liu et al. 2015; Liouane et al. 2016a, c), and also reasoning techniques such as fuzzy logic reasoning (Yuan and Herbert 2014; Das et al. 2013).

A successful model should effectively recognize and predict sensitive data and be adapted to the structure representing the knowledge. Subsequently, given the widespread use of Markovian approaches with satisfactory results (Suryadevara et al. 2013; Liouane et al. 2016b) and in order to integrate hierarchical and temporal reasoning, the Temporal Hierarchical Hidden Markov Model (THHMM): a new hybrid Markov model is proposed. The model is adapted to the reasoning of the proposed language which aims to recognize the behavior of the elderly people, which can be complex and changing, with high accuracy.

In this work, we aim to provide answers to the following questions:

  • How to make smart homes user-oriented and provide both technical and non-technical stakeholders with a tool that easily accommodates their work with monitored persons.

  • How to capture and express the essential knowledge and provide intelligent reasoning regarding human daily activities.

  • How to integrate the several key criteria of smart home monitoring such as time, locations and objects into one description language.

  • How to process complex scenarios with different autonomy levels and succeed to organize, highlight and structure raw data collected by several sensors.

  • How to provide a human activity recognition framework able to detect the daily behavior and anticipate risky situations with high accuracy.

This paper is organized as follows. Section 2 presents a brief survey of related works regarding the knowledge methods and recognition models. In Sect. 3, a new knowledge approach is defined based on natural language. In Sect. 4, a behavior modeling is proposed based on the hybrid markovian approach. The application of the proposed model is carried in Sect. 5, also the performance of the proposed model is evaluated. Finally, Sect. 6 concludes the paper.

2 Related works

In this section, the related works in the areas of knowledge and recognition models are summarized. The principal focus is on the application of such models to human behavior and especially to the accurate monitoring of activities of daily living in smart homes.

2.1 Knowledge methods

A great effort has been devoted to identify techniques to effectively model human behavior in smart environments. Many techniques based on machine learning and pervasive computing were used to achieve a better prediction of human activities and scenarios. Ta Minh et al. (2018) and Abdelrahman and Wang (2019) proposed a key-value modeling. This type of knowledge modeling is based on the simplest data structure to describe a given activity based on flexible units which represent sensor data. However, the capture of sophisticated context dimensions remains limited to capacities. To overcome the problems of the key-value methods, Francesco et al. (2019) proposed the exploitation and integration of hierarchical structures into multi-label classification systems with the form of a hierarchical deep neural network (HDNN). On the other hand, other approaches used graphical modeling in the design step (Asad Masood et al. 2014; Elena et al. 2018). Unfortunately, such techniques are flat information models that are limited concerning interoperability in contextual systems.

Object-oriented modeling is another method used to represent context information by using programming principles such as abstraction, inheritance, and aggregation (Mansur Kazemi et al. 2014). Despite the good performances of such techniques, they remain limited in regarding the interoperability issues. Logic-based modeling is used to provide a clear and elegant semantics in the description of contextual information but it has yet to represent uncertain context and inflexibility regarding the user habits (Sanjari et al. 2017; Ferilli 2014). Hossain et al. (2017) proposed a cluster-based learning model for activity recognition in smart homes. They claimed that the model is better than the traditional active learning approaches in the discovery of new activities with optimal accuracy. Bae (2014) used a model based on reasoning with an ontology for activities to design a knowledge base about the environment hence create a flexible recognition model. To facilitate the recognition and tracking of human activities, Liu et al. (2016) focused on human actions by using a classification of activities via temporal patterns. The work demonstrated the accuracy of the proposed model regarding complex activity recognition.

Wen and Wang (2017) built a general activity model with labeled data and suggest an online prediction by combining Adaboost with graphical models such as Conditional Random Fields (CRF) to smooth out the outliers. Compared to supervised and semi-supervised models, higher recognition accuracy was demonstrated with a low amount of labeled data. An integration of a probabilistic model with ontologies was proposed in Gayathri et al. (2017). Using the probabilistic reasoning, uncertainty of data was tackled. The proposed system showed an increase in the recognition of the activities through a Markov logic network model. A fuzzy logic model was combined with ontologies to overcome the challenges of classical ontologies-based approaches and in particular the uncertainty of data captured by a defecting sensor system (Bobillo and Straccia 2016; Noor et al. 2018). To provide accurate recognition with a reasonable level of confidentiality, a fusion of collected data was applied in the context-aware reasoning for sensor-based applications.

2.2 Recognition models

Several models have been proposed in the literature to perform an efficient recognition of human behavior. The probabilistic theory was a widely used approach in dealing with the uncertainty of data collected from sensors. It provides a mathematical representation for degrees of belief. For example, Yu et al. (2018) developed a Hidden Markov-based model to provide an automatic system for the detection of falls. The HMM model is used to detect falls based on data collection by motion sensors. The work highlighted the benefits of the proposed model based on the accuracy rate of detection. In order to build a pervasive environment, the recognition of ADL is a key task. Kabir et al. (2016) proposed a two-layer HMM to design an ADL monitoring system. The results showed that the two layers reach an effective recognition performance if compared to the conventional model. In Yiyan et al. (2016), a hierarchical model was proposed to accurately detect the complex walking activities based on decision trees, random forest and HMM. The decision tree is used to detect the coarse-grained distinction of the human motion mode. The random forest is used to perform the motion mode of the fine-grained distinction. The HMM is used to obtain a robust walking recognition. The results confirmed the accuracy of HMM which performed 93.8% of accuracy. Wang et al. (2016) proposed a new model named sliding window-based HMM (SW-HMM) in order to recognize the activities of daily living. First, a sequence of activities is divided into several fixed size, then, each sub-sequence is evaluated using HMM. The results showed that the performance of SW-HMM is better than the conventional model. Kim et al. (2015) proposed a new model based on HMM, named HMM ensemble (HMME), in order to recognize the human’s activities. The recognition of the human activities of daily living was achieved using smartphone sensors. If compared with traditional recognition activities models, the results showed the best performance of HMME regarding the recognition. The work discussed in Ronao and Cho (2017) proposed a new model called Hierarchical Continuous Hidden Markov Model (HCHMM). The results showed that the hierarchical structure increases the performance of the activity recognition with a better performance regarding the accuracy with 93.18%.

Further recent techniques were discussed in support of independent living and ADL recognition (Salisu Wada et al. 2019; Abubaker et al. 2019). The approaches identify abnormalities activities by detecting any significant deviation from usual routines. The ability to adjust normality scores makes the proposed system more flexible (Salisu Wada et al. 2019). In a multi-sensor space, measuring the progressive trends in the person’s behavior allows providing relevant indicators to health stakeholders (Abubaker et al. 2019).

3 A new approach based on natural language

In order to overcome the previous challenges of markup scheme modeling, graphical modeling, key-value modeling, object-oriented modeling, logic-based modeling, and ontology modeling, a new knowledge-based method is proposed using a new descriptive language for the daily activities. Indeed, The proposed language uses the characteristics of natural languages. the description of the activities of an elderly person is an essential task that requires precision and clarity in the context of a home monitoring system. A scenario can be defined as a set of activities and actions performed by the person over time. Therefore, it is possible to consider that a scenario is a descriptive diagram of the activities. The structure of a scenario can be very complex and heterogeneous from the point of view of knowledge and data. Therefore, in order to simplify, organize, highlight, structure and remove any ambiguity over the information schematized by a scenario, the proposed language descriptions concern activities of elderly people based on the characteristics of the natural language.

3.1 Characteristics of a natural language

People predominantly use a language to communicate such as English, French, etc. Two aspects characterize a language: its meaning and its structure. The meaning of a language is materialized by the ideas it conveys which is called its semantic content. The structure of a language contains grammatical rules used to construct it and form the syntactic aspect. When someone is learning a new language, he must master the vocabulary, grammar, and semantics.

In fact, {The lexicon:} is the vocabulary which represents the words of the dictionary, the declensions, etc. {The syntax:} it is the grammar or how to compose correct sentences regarding the form and structure. For example, the sentence the cat eats the house is grammatically correct. {The semantics:} is the meaning of words and sentences. Generally, learning semantics is achieved at the same time as the vocabulary and the grammar: the process learns the correct words, their meaning, and their grammatical constructions.

Based on these highlighted characteristics, a new knowledge language, called “Elderly people by room activities description language (EBRADL)” is defined based on terms, rules, constraints, and well-considered relationships. EBRADL provides non-technical users a mean to encode knowledge about the daily activities and ensures an intelligent reasoning. In the following, the aims of EBRADL and its field of knowledge are described.

3.2 Elderly people by room activities description language (EBRADL)

The recognition of daily activities plays an important role in the realm of e-health services. A large number of sensors deployed in the smart space provides a safe and comfortable environment. However, the complexity of the information captured by sensors influences the accuracy of the recognition model. In this work, the objective is to simplify the complexity of this information. To this aim, a behavioral description language is proposed in order to structure, describe the occupant’s activities and make it less complex. A global, clear and specific understanding and characterization of the targeted activities for an elderly adult in a smart home is a critical task to understand the links and the relationships between the different daily activities. The standard formalization of the contextual information, defined in the conceptualization process, is a key step in enabling effective designs.

3.2.1 The aims of EBRADL

Given the enormous amount of information and descriptive data regarding the person’s activities, describing the person’s behavior in an adaptable and customizable way is a complex and difficult task that requires a precise description and modeling of the activities of the person. Therefore, the main objective is to provide a clear modeling which is simple and easy to understand while respecting the existing rules and constraints, also a powerful and reliable descriptive language is considered to characterize the activity based on their concept and context. A formal algorithm that represents the instructions and the basic sequence is used to build the proposed language, is presented in below.

Basic instructions of the EBRADL language: the main instructions of EBRADL ensure the following: identify the area of knowledge; identify the lexical/vocabulary field; analyze the vocabulary (terms); identify the vocabulary terms and normalize their meanings; identify concepts, relationships, and constraints; build a structure/hierarchy that represents the language. This simple modeling of the behavior of the elderly adult is an essential task for an adequate selection of the data collected by the sensors and for a relevant system design.

3.2.2 Semantic fields of EBRADL

The conceptualization process of EBRADL is primarily based on terms that can be formalized to be understandable by the computer. Thus, it is a step towards a fully detailed and automated business process. A lexical field is a set of words and terms related to the same context, it is the vocabulary used in the proposed language. As previously noted, the EBRADL language is a language dedicated to elderly persons with difficulties (who represent a sensitive population) living alone at home. In this context, it is required to precisely and clearly define the terms and words of the language. A term is considered as a word of the language that designates a context. More specifically, in order to fix and identify the terms of EBRADL, we try to answer the following question: what the language will describe, who and where? In this case, the following three terms are used to define:

What is described?—Answer: the activities of daily living.

Who is described?—Answer: the person evolving in a smart environment.

Where is the context of the descriptions?—Answer: at home.

From the previous answers, it is possible then to distinguish that the lexical field focuses on three basic terms that are: [home, person, and activities]. Each basic term consists of several sub-terms. Table 1 represents each category with the associated terms.

Table 1 The basic terms according to their sub-terms
Full size table

Therefore, the descriptive language is specified by the appropriate terms: the home, the person and the activities.

3.2.3 Syntax of EBRADL

In this part, the focus is made on the specific definition of the syntax of EBRADL. The syntax is usually defined to indicate the relationships between the rule-based terms and the meaning of terms. Lexical fields of the EBRADL language are based on three basic terms that are the home, the person and the activity. Moreover, the relationships between these terms are defined. Each term has relationships with others:

Relationship 1: House–Person \(\lbrace Relationship\ 1.1: Person-Location\ (room)\rbrace \lbrace Relationship\ 1.2: Person-Object\rbrace \)

Relationship 2: House–Activity \(\lbrace Relationship\ 2.1: Activity-Location\rbrace \lbrace Relationship\ 2.2: Activity-Object\rbrace \)

Relationship 3: Person–Activity \(\lbrace Relationship\ 3.1: Person-Activity\rbrace \lbrace Relationship\ 3.2: Person-Time.\rbrace \)

From the previous relationships, it is easy to notice that each term designates a relationship with another one. In addition, each relationship defines a link between the terms. In general, defining the relationships is not enough to provide an adaptable and customizable description. As a result, the rest of this section will focus on defining rules and constraints based on the constraints of everyday life. Generally, in everyday life, people apply rules learned from birth. For example, constraints related to objects, constraints related to places, constraints related to times, etc. In a smart home environment, there are many types of contextual information that can be used to characterize an activity, such as interactions with objects, locations, and time. Thus, object, location, and time are widely used concepts to describe and distinguish activities.

An Object usually refers to the types of activities being made. For example, if the TV set is the object, it is possible to deduce that the activity is a watching television while taking a broom can lead to an activity like sweeping the floor.

The location is the specific place where an activity occurs. For example, preparing the meal takes place in the kitchen and taking a shower occurs in the bathroom. In addition, the location can also help to identify the involved objects.

Time is another key feature in the description of a given activity. Indeed, the user can perform an activity at different times, but some activities occur only at specific times of the day. For example, the activity preparing breakfast usually occurs in the morning, prepare dinner usually occurs in the evening and take a shower once or twice a day can occur after waking or before going to bed.

Other context information, useful for characterizing activities, includes the person’s profile and habit. Since the proposed language focuses on elderly people and since this kind of population can suffer sometimes from limited autonomies (such as physical autonomy, mental autonomy, etc.), the constraints based on the rules of everyday life are defined. The constraints help to distinguish elderly persons and improve the syntax of the proposed language. Five constraints are defined, called:

\(\lbrace Constraint 1: Location-Object\rbrace , \lbrace Constraint 2: Location-Activity\rbrace ,\lbrace Constraint 3: Activity-Time\rbrace ,\lbrace Constraint 4: Activity-Habit\rbrace , and \lbrace Constraint 5: Person-Profile\rbrace \)

Constraint 1

(Location–Object): this constraint represents the entire specification of the object and the link that exists between a specific location and objects used in that location. Every place at home is equipped with specific objects. Therefore, the objects are relative to the place. For example, kitchen objects cannot be used in the bathroom. Table 2 shows a classification of objects regarding their locations based on Constraint 1.

Table 2 Objects by location
Full size table

Constraint 2

(Location–Activity): this constraint represents the specification of the activity according to the type of location. Every activity of daily life is relative to its place. This constraint represents the same principle as the constraint 1 (Location–Object), i.e. each activity is linked to its location and can not be realized in another place. For example, in the normal course, a person cannot sleep in the kitchen. Table 3 represents a classification of activities by location.

Table 3 Person’s activities by location
Full size table

Constraint 3

(Activity–Time): another very important constraint concerns the descriptive language is the constraint (activity/time) which represents the relationship between the activity and the time of realization. In general, approximately every daily activity is relative to a specific time slot. This means that an activity is bounded by a minimum and a maximum amount of time. For a given person and profile, this time interval can be defined by experts and sometimes can be estimated based on the routine of a person’s life. For example, it is possible to consider that it does not make sense to prepare a breakfast during 5 h, or to take a shower in a time slot equals to 8 h.

Constraint 4

(Activity–Habit): this constraint plays a major role in the recognition phase, which facilitates the detection of abnormal behaviors of the person based on their habit. This constraint consists of two sub-constraints which are: the habit–time constraint and the habit–order constraint of the activities.

Habit–time: this constraint is related to the constraint 3 (Activity–time) but it also depends on the person’s routine in the sense that the time of realization of an activity changes from one person to another. The usual time required by humans is considered to define the relationship between the time required for a given activity in a general setting and the time required for an activity that concerns the person being monitored in the smart space.

Habit–order of activities: this constraint concerns the order of activities and the habit of the person. The constraint helps to structure the behavior of the person and subsequently to distinguish each new behavior change. For example, the usual order of activities during three months for a given person can be: get up, then, take a shower, then, make his toilet, then, prepare his breakfast, then, to watch TV. If a change is identified regarding the order of daily activities, this may indicate a change in behavior that sometimes highlights a change regarding the health state of the person.

Constraint 5

(Person–Profile): in this description, the focus of the target subjects is made on elderly persons. This constraint is based on the person’s profile in order to improve the language and gain an adaptable and customizable description. The profile of the person is considered as follows: Profile = [Name/Surname − Age − Location − Diseases − SMAF Geriatric Profile − Weight − Height]. In order to handle the profile of the person within the proposed fifth constraint, a focus is made on three basic information (age, diseases, and SMAF profile) which is the most significant information in the proposed description. The purpose of the Constraint 5 is to define a specific link between these terms and which is easy to interpret in order to have a clear level of description according to the profile of the person. The term “SMAF Profile” indicates the level of autonomy of the person using the SMAF geriatric scale (Mshali et al. 2018). SMAF is a very detailed autonomy measurement system that consists of 14 profiles, ordered starting from independent persons to totally dependent persons. In order to facilitate the use of this information, the 14 profiles are classified into four categories of profiles. The term “age” is an important criterion for the elderly people. This criterion can help to determine the level of descriptions in the language. The term “diseases” is also a significant term. It also helps to improve the way of descriptions. From these terms, a level of description of the proposed language is proposed to take into account the profile of the person and his level of autonomy. The principle of this classification is as follows: three levels of description are provided: a simple description, a detailed description, and a more detailed description. This classification allows to eliminate the presence of unnecessary content in the descriptions and avoid the clutter of information if it is not necessary.

  • Level 1: simple descriptions of elderly persons with an age \(\in \) [50...70].

  • Level 2: detailed description elderly persons with an age \(\in \) [70...80].

  • Level 3: more detailed description elderly persons with an age \(\in \) [80...90].

Table 4 presents the three levels of description according to the age, SMAF profile, and diseases.

Table 4 Levels of description according to age, SMAF profile, and diseases
Full size table

An assessment of the person’s level is made at the beginning of the person’s monitoring in order to provide an adequate description. This level periodically requires control according to the usual data and according to the Constraint 4. In addition, if the behavior of the person undergoes a noticeable change, the level of autonomy defined beforehand will be updated.

3.3 Structure of EBRADL

After having defined the lexical field and the syntax of the proposed language, this part focuses on the designation of an adaptable and customizable structure based on the various constraints proposed in the syntax part. Based on constraints 1, 2, and 3, it is easy to conclude that a daily activity is proportional to the place and objects involved in each activity are relative to the place. Hence, the location includes the activity and the object proportionately. Consequently, the descriptive structure of the language takes into account the notion of place as a basic notion. For this reason, a classification based on the type of location of the home is proposed. In this way, a specification can be added to to EBRADL in order to have a simple, adaptable and relevant description. Table 5 presents the notion of classification.

Table 5 Classification of Activities/Objects/Time slot by location
Full size table

After providing a relevant and meaningful classification to the proposed terms and constraints, a well organized structure of the language can now be define. Indeed, the description of the daily activities of a person takes the following form:

Person # is in # the place # performs # activity # with the use of # object [\(objetc_{1}\), \(objetc_{2}\),..., \(objetc_{n}\)] # at t = # start time.

Example: Paul # is in # the bathroom # performs # his toilet with the use of # [Sink] # at t = # 9: 35h

Considering the constraint 5 (Person–Profile), three descriptive structures are defined according to the profile of the person.

Level 1 (simple descriptions): Person # is in # a place # make # activity.

Level 2 (detailed descriptions): Person # is in # place # performs # activity # with use of# object [\(object_{1}\), \(object_{2}\),..., \(object_{n}\)] # at t = # Start time.

Level 3 (more detailed description): Person # is in # a place # performs # activity # with use of # object [\(object_{1}\), \(object_{2}\),..., \(object_{n}\)] # at \(t_{s} =\)# Start time # to \(t_{e} =\) # End time.

Our descriptive language EBRADL focused on elderly persons and is dedicated to offer an adequate selection of the data collected by sensors. Therefore, it is crucial to model and automate the proposed language in order to recognize and predict the behavior of the monitored person.

4 Elderly people behavior modeling

Most of the research that has been done to deal with the recognition and prediction of behavioral activity is done using statistical techniques. These techniques are used to determine the dependence and correlations between time and data generated by the sensors to identify the behavior of the occupant. Probabilistic models are classified among the best techniques used for identifying the human behavior because they are able to represent random variables, dependencies, and temporal variations in the data. In this work, in order to model and automate the proposed language, an efficient hybrid model is proposed. The goal is to recognize and predict the behavior of the person in a flexible and relevant way that allows monitoring the person afterward.

4.1 Recognition hybrid model

The Hidden Markov Model (HMM) is one of the statistical models in which a system uses a Markov process with unknown parameters. It is widely used to identify a user’s activity from sensor data. EBRADL is characterized by a hierarchical structure and well-defined constraints that represent robust relationships between terms and the logical aspect that considers the profile of the person, the architecture of the house, etc. Thus, relevant modeling requires a model that meets the needs of the proposed language. The targeted model must be effective from the point of view of the recognition and perdition and must take into consideration the hierarchical notion and the temporal notion together. To this aim, a new hybrid model called “Temporal Hierarchy Hidden Markov Model (THHMM)” is proposed.

4.2 The THHMM model

The Temporal Hierarchy Hidden Markov Model (THHMM) is based on a combination of the hidden hierarchical markovian model (HHMM) (Kong et al. 2018) and the hidden semi-markovian model (HSMM) (Dong and He 2007) in order to support the hierarchical and temporal reasoning.

In the considered hierarchy, the number of levels increases from the highest level to the lowest level. A state \(S_{i}^{n}\) represents a state (internal transmitter) with i represents the index and n the level in the hierarchy. A parent state at level n emits a sequence of symbols to stimulate a child state at level n + 1. These states may also be composed of HMM or HHMM. There are two types of transitions: a vertical transition connecting the states of the same level and a horizontal transition that connects a state of the level n to level n + 1. To estimate the model’s parameters, the three HMM conventional algorithms are derived for the evaluation, decoding and learning problems.

The forward–backward algorithm is applied to solve the evaluation problem and determine the probability that the observations are generated. The Viterbi algorithm is used to determine the most likely state sequence that produced the observations. For the learning problem, to find a method to adjust the model parameters and maximize the probability of a given observation sequence O, the Baum–Welch algorithm is used based on an iterative method. The re-estimation of the probability of being in the state i at time \( t = 1 \) is represented by \(\bar{\pi _{i}}\) where \(\bar{\pi _{i}}=\gamma _1(i),1\leqslant i\leqslant N \) and \(\gamma _{t}(i)=P(S_t=i\mid O,\lambda )\) is the probability with O being the observation sequence with the hierarchical model \(\lambda \).

Since the hierarchy structure does not take into account the temporal notion for each state, this dimension is integrated into the proposed model from the semi-Markov model to find explicit modeling and estimate the amount of time in a given state. The time criterion is of paramount importance in e-health monitoring services and allows us to discriminate the duration of each activity and distinguish any change in the routine of the person. The explicit parameterization of a state’s duration is defined by the estimation of a probabilistic parameter \(P_i(d)\) with \(1\leqslant d \leqslant D \), where D represents the maximum duration (limit) by state and \(P_i\) represents a probabilities vector of discrete duration. Two distribution classes are used for the state durations: a discrete distribution (a Poisson distribution) and two continuous distributions (gamma and Gaussian distributions).

The general form of the THHMM model can be represented by \(\lambda =(A^S, B^S, \pi ^S, P_S(d), N)\) where for a given level: \(A^S\) represents the transition matrix between states, \(B^S\) represents the observations matrix, \(\pi ^S\) represents the initial state, N represents the states, and \(P_S(d)\) is the probability that the model remains in a state S for a duration d. Thus, the recognition and learning algorithms (Baum–Welch and basic forward–backward) undergo the reformulation in order to apply the hierarchical structure and time in THHMM. The resulting THHMM model integrates explicit management of the duration of each state by the estimation of a temporal variable. These time variables are connected to each other by arcs in a way that the current states are triggered at the end of the previous state. The Markovian recognition model makes it possible to recognize the hierarchical states according to the level in the hierarchy structure. If a state of level n is triggered, the model operates as a conventional HMM and seeks to find the most probable state. Subsequently, the most likely state stimulates sub-states that are in the n + 1 level and are linked with a vertical link between the levels. Figure 1 shows the architecture of the THHMM model.

Fig. 1
figure 1

Structure of the Temporal Hierarchy Hidden Markov Model (THHMM) model

Full size image

4.3 Application of THHMM for the recognition of the behavior

In order to recognize the behavior of an elderly person evolving in a smart environment, the THHMM Model is applied. The following part discusses the principles of EBRADL modeled by the THHMM model. First, the structure of the monitored person’s space is defined. In this study, the example of a smart home consisting of four rooms is considered. The rooms are: the bedroom (BR), the living room (LR), the bathroom (Bth-R) and the kitchen (K). Each room is equipped with several types of sensors. Prior knowledge of the environment allows us to define the structure of the model and define the number of states. This step is used to model the proposed language. The specific form of THHMM becomes \(\lambda \) with \(\lambda = (A^s, B^s, \pi ^s, \hbox {S}, \hbox {T}, \hbox {N})\), where: T represents the time where the model is stopped at a given state level, N represents the states of the model which are defined by the set S with \(S_{t} = {S_{1}, S_{2}, S_{3}}\) where \(S_{1}\) represents the rooms in where the occupant is at time t, \(S_{2}\) represents the activities that the person is doing at the moment t, \(S_{3}\) represents the objects used at time t.

In order to test the THHMM model, a supervised scenario to simulate the behavior of the activities of daily living (ADL) of the elderly person is performed. The THHMM can be seen as a discrete set of nodes (or states) and transitions (or arcs) connecting these states to each other. The states of the model are hierarchically organized. An appropriate modeling is developed in order to structure the EBRADL language by the THHMM hybrid model. In order to recognize the behavior of the person and to automate the EBRADL language modeled by THHMM, the proposed model is composed of 4 hierarchical levels which are the Root level, the Location level, the Activity level, and the Object level. Figure 2 illustrates the structure of the EBRADL language modeled by THHMM. It shows the activities of the person according to the places and objects with the proposed hierarchical structure. Level 1 represents the root environment. Level 2 represents the main environment. Levels 3 and 4 represent the activities and objects respectively.

Fig. 2
figure 2

EBRADL structure modeled by THHMM

Full size image

5 Results assessment

In this part, the results of the recognition and prediction obtained by the THHMM model are simulated and interpreted. The accuracy of THHMM based on the EBRADL language is evaluated. The goal is to identify the capacity of THHMM in terms of prediction accuracy. The evaluation considers the prediction error, the root mean squared error (RMSE) and the state error percentage during the prediction step to evaluate the performance of the proposed model. Also, the achieved results are compared with the existing research to evaluate the efficiently of the proposed recognition model.

5.1 Experiment context

For the purpose of conducting various experiments to evaluate the propositions presented in this work, it is required to process a dataset that includes a substantial period of monitoring and which is as close as possible to the requirements and richness of expressivity of EBRADL such as the consideration of the information about the geriatric assessment and the integration of daily activities, places, objects and time (Sect. 3). In this work, the assessment is based on data collected from the e-health monitoring open data project integrating the CASAS Aruba and Tulum datasets (Mshali et al. 2018; Liouane et al. 2018; Diane et al. 2013). Raw data provide a set of activities and actions achieved by the occupant in chronological order. Data represent the activities as defined in the geriatric SMAF model. The provided data describes several scenarios throughout a long period. The different periods of monitoring include: five periods each one of one year for one resident (with a worsening geriatric profile), a period of 7 months for one resident with the same profile and a period of 4 months for two residents with the same profile. The format of each daily living event includes the beginning and the end time of each activity or basic action. For instance, Table 6 summarizes the activities with a specific category and code for each activity for an autonomous inhabitant who belongs to the first profile of the SMAF model. Parallel processing, with MATLAB, was performed on data using the four cores of a DELL Precision M6700 machine with an Intel®Quad-Core TM i7-3940XM, 3.20 GHz processor with a RAM of 32 GB.

Table 6 Daily living activities and actions
Full size table

The periods of daily scenarios are processed to build a behavior model of the older adult. The activities and actions achieved by the inhabitant are represented with a generalized temporal relationship form: [day, start-time, end-time, activities-code] (Liouane et al. 2018). An example of data collected from a home with multiple sensors is depicted on Table 7. Collected data are processed and transformed into a continuous time-series format with the classification of activities, objects and time slots by location (Sect. 3.3; Table 5).

Table 7 Time series of the daily living scenarios
Full size table

5.2 Behavior prediction

Scenarios prediction represents a very important task as part of a home monitoring system. The objective is to predict future activities in advance and anticipate required interventions in case of a risky behavior change or emergency. The prediction also allows updating the profile information of the person in the case of significant changes. In the following, the prediction’s accuracy of THHMM while benefiting from the model’s hierarchical structure (i.e. level 2: locations and level 3: activities) is evaluated. First, the focus is made on the prediction of activities (Figs. 3, 4): what are the expected overall activities that the resident will perform at home and those expected in a specific place? Secondly, the focus is made on the prediction of the duration that the monitored person could spend in achieving the daily activities (Figs. 5, 6): how long does it take to perform the activities regardless of the location and for a specific location?

Figures 3 and 4 present the results of the predicted behavior using the THHMM model at level 2 (i.e. location) and level 3 respectively over the real behavior. The blue curve shows the real behavior and the red one shows the predicted behavior as estimated by the THHMM model in both Figs. 3 and 4. The simulation results regarding the interpretation of the person’s behavior using THHMM during a day (Fig. 3) show that the behavior estimated by the THHMM model seems to correspond to the real behavior of the person. The same observation applies in Fig. 4.

Fig. 3
figure 3

The real and predicted behavior obtained by THHMM at level 2

Full size image
Fig. 4
figure 4

The real and predicted behavior related to the Kitchen obtained by THHMM at level 3

Full size image
Fig. 5
figure 5

The real and predicted time for each state (Livingroom, Kitchen, Bathroom, and Bedroom) obtained by THHMM during a day

Full size image
Fig. 6
figure 6

The real and predicted time for each state (preparing meal, washing the dishes, and using an object), level \(\hbox {n} = 3\)

Full size image

In the context of a monitoring system, the prediction of durations is of paramount importance. Indeed, the criterion of time helps to detect a behavioral change or an abnormal situation in particular for dependent persons. The time spent in each space and the time spent in achieving a given activity can guide to model behaviors related to the person’s habits and distinguish the changes based on those habits. This information can be used for medical purposes. For example, to assess the person’s autonomy or to identify some activity-related diseases such as Alzheimer’s, diabetes, and depression. As a result, the behavioral model THHMM is mainly based on the temporal criterion. Figure 5 presents the prediction results regarding the time spent in each location obtained by the THHMM model at level 2, states: living room, kitchen, bathroom, and bedroom. Figure 6 presents the prediction results of the time spent to perform each activity (e.g. preparing meal) associated with the kitchen state obtained by THHMM at level 3. Each curve shows the real-time as well as the estimated time for each state throughout a day. The blue curve shows the real-time while the red one shows the estimated time. The simulation results show that the time estimated by the THHMM model for each state appears to fit well with the real-time spent in each location.

5.3 Performance evaluations

The prediction error of the proposed model is evaluated throughout 1 day. Table 8 presents the results regarding 10 days for the RMSE assessment. The obtained results confirm the accuracy of THHMM. Indeed, the average of the RMSE is about 9.6392 × \(10^{-2}\) per day.

Table 8 Root mean square error (RMSE) per day
Full size table

In the following, the error percentage is evaluated of each state in level 2 which represent the bedroom, bathroom, living room, and the kitchen. Figure 7 shows the error rate for each state belonging to level 2 in the prediction step. From this figure, it is observed that the error percentage related to the bathroom state is 0.2. For the kitchen, the error percentage is approximately 0.5. The percentages of errors are 0.44 and 0.31 for the living room and the bedroom respectively. Concerning the “level 3” of the hierarchy (i.e. activities), specifically the kitchen state, the percentage of error per state is 0.03 for the state using an object. For the washing the dishes state and the preparing meal state, the error percentages are 0.06 and 0.12 respectively (Fig. 8).

Fig. 7
figure 7

Prediction error based on states (level 2)

Full size image
Fig. 8
figure 8

Prediction error based on states (level 3)

Full size image

From the previous results, the performance evaluations of the THHMM model highlight the behavior prediction accuracy and at the same time show the time prediction accuracy which confirm the efficiently of the proposed recognition model.

5.4 Comparative study

In order to approve the performance of the proposed approach, the results of the description language EBRADL and the THHMM model are evaluated together. First, the benefits of EBRADL compared to existing knowledge methods are appreciated. On the other hand, the utility, the considered activities, and the benefits of THHMM are evaluated and compared with the recognition models based on HMM that exist in the literature. Generally, a deep knowledge study is very important to provide a good and understanding knowledge method. For this purpose, Table 9 shows the comparison of benefits between the existing knowledge methods and the proposed knowlegde method (EBRADL). From Table 9, it is observed that the proposed method is more understandable and powerful to design and model the activities of daily living. In fact, EBRADL ensures several criteria such as adaptability, simplicity (structure), data abundance, and effectiveness.

Table 9 The benefits comparison between the existing knowledge methods and the proposed language
Full size table

To ensure a good recognition and prediction of the generated knowledge (regarding ADL and IADl activities) provided by the proposed language, several criteria must be checked in the used model. For this purpose, a hybrid markovian model is proposed to improve the recognition method and the prediction accuracy. Table 10 presents a comparative study between the proposed model and existing markovian models.

Table 10 A comparison of characteristics between the existing markovian models and the proposed model (EBRADL/THHMM)
Full size table

From Table 10, it is easy to conclude that the proposed model (THHMM) provides a good performance to ensure the needs of monitoring systems taking into account two important criteria which are: the temporal dimension and the hierarchical notion. Also, THHMM considers both of the ADL and the IADL activities to track the behavior of the person, so that is more intense from the data knowledge perspective. Finally, another important benefit of THHMM is the major prediction accuracy.

6 Conclusion

Describing and recognizing the daily activities of a senior monitored in a smart space is a very important task in a health monitoring system. Unfortunately, the heterogeneity and variety of sensory data collected make tasks description, recognition and prediction difficult to achieve. Therefore, for an efficient context-aware modeling, a description language named “Elderly people by Room Activities Description Langauge (EBRADL)” was proposed based on the ADL and IADL activities, person’s profile, constraints, and rules. The proposed language serves to provide a specific and clear understanding and characterization of daily activities according to the person’s level of autonomy. In addition, a standard formalization of the contextual information that represents the behavior of the person was achieved. Subsequently, a hybrid approach based on the Markovian model was proposed in order to automate the EBRADL language and to predict the future behavior of the person. EBRADL is characterized by two basic criteria: a hierarchical structure and the temporal dimension, so the relevant modeling requires a model that meets the needs of these characteristics. The proposed hybrid model called Temporal Hierarchy Hidden Markov (THHMM) is characterized by its hierarchical structure and the time integration. The simulation results show that the proposed model offers promising results regarding the prediction accuracy. In addition, a comparative study between the proposed recognition process (the combined knowledge method with the recognition model) and existing works was provided. The evaluation confirmed that the proposed description language is more efficient than existing knowledge models and, hence, it allows more benefits such as simplicity, clarity, adaptability, and flexibility. With respect to the proposed recognition model, the comparison results confirmed that THHMM is more suitable to recognize and predict the knowledge related to persons activities and behavior. Moreover, THHMM provides a better prediction accuracy compared with other related research.