Abstract
This paper outlines the steps for a specification to create a language for defining human behavior in the context of complex activities, with a specific focus on Activities of Daily Living (ADLs). The proposed specification to create a language for defining human behavior is used to develop a framework for representation of (1) macro-level tasks or actions associated with any complex activity, (2) the context parameters on which these tasks or actions are performed and (3) sequence of movements of various body parts associated with performing these actions on the context parameters in the context of the given complex activity. To evaluate the efficacy of this proposed work, it has been implemented on a dataset of ADLs. The results presented and discussed uphold the relevance for practical implementation of the proposed human behavior definition language in real-time settings for addressing various challenges and utilizing the full potential of activity centric computing for improving the quality of life and user experience during ADLs as well as for various other applications.
Access provided by Autonomous University of Puebla. Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction
In this era of Industry 4.0 where modernization and advancement of technologies are taking place at a very fast pace, people are surrounded by technology all the time. The essence of these technology-laden pervasive and ubiquitous environments lies in effective communication and coordination between various components for sharing, tracking, monitoring and analyzing user data to facilitate collaborate learning, collaborative knowledge and collaborative discovery [1]. Semantic understanding of human behavior during complex activities [2] has immense potential for addressing this challenge. Recent works [3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25] in the field of activity recognition have several limitations. The primary limitation being discrete approaches – for instance video-based recognition, machine learning, probability theory etc., being used for development of the activity recognition models, which makes integration and communication of these systems with other ‘smart’ environment parameters a challenge. There is a need for development of an approach for activity recognition that can be seamlessly understood and integrated with any other systems or technologies in an Internet of Things (IoT)-based environment, to allow the utilization of the immense potential of activity centric computing for real-time development of various applications, for instance – behavior interventions, task recommendations, rehabilitation systems, training of experts for skilled professions, behavior monitoring etc. This serves as the main motivation for this work.
2 Literature Review
This section outlines the recent works in this field. A hierarchical framework for activity recognition using multiple sensors was developed by Azkune et al. [3]. A system that consisted of multiple layers with distinct functionalities for each to study and analyze video data for activity recognition was proposed by Cheng et al. [4]. An infrared sensor powered system using artificial neural networks for activity recognition was developed by Skocir et al. [5]. The system was able to detect different kinds of enter and exit events in an IoT-based smart home environment. An environment specific task recommendation system was proposed by Doryab et al. [6] to improve performance of medical practitioners in hospital and similar environments. Using wireless sensor technology, Abascal et al. [7] developed an indoor navigation system to assist users to perform different activities. A smart home monitoring system was developed by Chan et al. [8] to monitor the behaviors of elderly people as they carried out different activities. An architecture for an intelligent kitchen environment was proposed by Yared et al. [9] with an aim to reduce accidents in the kitchen area. Deen [10] developed a physiological signal tracking system to monitor users and their performance during different activities. The work done by Civitarese et al. [11] involved a system which would track user interactions to evaluate if the user has successfully been able to complete the given activity. Iglesias et al. [12] developed a health information monitoring system to monitor the health status of users in the context of ADLs. In a similar work done by Angelini et al. [13], the authors developed a smart bracelet to collect data about the health of its user as well as remind them of their routine medications and daily tasks. Khosla et al. [14] developed an assistive robot to help elderly people perform their daily routine tasks. A similar work was done by Sarkar [15] where an intelligent robot called ‘Nursebot’ comprised of a scheduler to schedule routine medications. The works of Thakur et al. [16,17,18,19,20,21,22,23,24,25] have also contributed extensively to this field.
3 Proposed Work
The work presented in this section extends one of the previous works [26] in this field – ‘A Complex Activity Recognition Algorithm’ (CARALGO). The steps to develop this framework comprise the following:
-
1.
Identify the macro and micro level tasks associated with the given complex activity
-
2.
Identify the environment parameters or context attributes on which these tasks need to be performed for successful completion of the activity
-
3.
Track the user’s behavior to study when the complex activity started and when it ended – this includes tracking (i) when the user successfully reached the goal and (ii) when it was a false start
-
4.
Record the most important context attribute [25] and the action or actions performed by the user on it
-
5.
Study the user’s behavior associated with performing each of the atomic activities while interacting with the context attributes:
-
a.
Develop a methodology to represent the skeletal tracking of the user using Microsoft Kinect Sensors
-
b.
Identify specific feature points on the skeletal tracking
-
c.
Define these feature points according to their locations and functions
-
d.
Analyze characteristics of these feature points – which include joint distance, joint movements, joint angle and joint rotation speed
-
e.
Record the sequence in which the characteristics of these feature points change or influence one another
-
a.
-
6.
Compile the sequence of the characteristics of human behavior from (3) in the CARALGO analysis of the complex activity
Microsoft Kinect Sensors provide real-time skeletal tracking of any user and upto 20 joint points on the skeletal [27]. For defining these joint points as per the associated movements and functions, we review [28]. This definition is presented below (Fig. 1):
Prior to implementation of this framework for analyzing a complex activity, we present an example to study a specific macro-level task. For instance, if the user has to answer the phone, then in this context the most important context attribute is the phone. The behavior associated with interacting with the phone would involve bringing the phone close to the ear – this involves the distance between joint point pairs (6,4), (7,4), (8,4), (6,3), (7,3), (8,3) or (10,4), (11,4), (12,4), (10,3), (11,3), (12,3) getting less.
4 Results and Implementation
To evaluate the efficacy of this framework, a dataset was developed based on the work done Ordóñez et al. [29]. This work by Ordóñez et al. involved recording different ADLs performed by a user in the premises of a smart home. The work consisted of a smart IoT-based environment which comprised of multiple sensors that were used to perform activity recognition to sense ADLs in a smart home, for 24 h over a period of 22 days. The different ADLs that were a part of this dataset consisted of the complex activities of Sleeping, Showering, Eating Breakfast, Leaving for work, Eating Lunch, Eating Snacks and Watching TV in Spare Time. Multiple complex activities from this dataset were analyzed according to this framework and analysis of one of the complex activities – ‘Eating Lunch’ is being presented here. In this representation, by change we mean change in distance between the joint point pairs:
As can be observed from Table 1, not all atomic activities are associated with joint point pairs experiencing a change. However, for certain atomic activities, for instance At6, several joint points pairs experience a change and it also depends on the user diversity – in terms of the user being left-handed or right-handed. It is important to track and analyze the joint points that undergo a change and also study the sequence in which multiple joint points undergo this change. This is because any atomic activity can be broken down into sub-atomic activities that need to follow a specific sequence as outlined in [23]. This analysis of changes experienced by joint points can be seamlessly communicated to other systems or technology-laden environments in the given IoT space to facilitate collaborative learning and discovery of human behavior.
5 Conclusion and Future Work
The proposed framework for development of a language to define human behavior has several characteristics: (1) It helps to study the macro-level tasks associated with any complex activity, (2) It provides a definition of different feature points for skeletal tracking, (3) It discusses a method to study, track and analyze the changes in characteristic features of these feature points with changing human behavior and (4) It also discusses the relevance of tracking the sequence of these changes for each atomic activity associated with the given complex activity. To the best knowledge of the authors, no similar approach has been done in this field yet. The results presented uphold the relevance and potential of this approach for defining human behavior in the future of IoT-based Smart Home environments for improving the quality of life and user experience during ADLs. Future work would involve implementation of this framework in real-time in an IoT-setting.
References
Lampropoulos, G., Siakas, K., Anastasiadis, T.: Internet of Things in the context of Industry 4.0: an overview. Int. J. Entrep. Knowl. 7(1), 4–19 (2019)
Davies, N., Siewiorek, D.P., Sukthankar, R.: Activity-based computing. Pervasive Comput. IEEE 7, 20–21 (2008)
Azkune, G., Almeida, A., López-de-Ipiña, D., Chen, L.: Extending knowledge driven activity models through data-driven learning techniques. Expert Syst. Appl. 42 (2015). https://doi.org/10.1016/j.eswa.2014.11.063
Cheng, Z., Qin, L., Huang, Q., Jiang, S., Yan, S., Tian, Q.: Human group activity analysis with fusion of motion and appearance information. In: Proceedings of the 19th ACM International Conference on Multimedia, Scottsdale, Arizona, USA, pp. 1401–1404, 28 November–01 December 2011 (2011)
Skocir, P., Krivic, P., Tomeljak, M., Kusek, M., Jezic, G.: Activity detection in smart home environment. In: Proceedings of the 20th International Conference on Knowledge Based and Intelligent Information and Engineering Systems, 5–7 September 2016 (2016)
Doryab, A., Bardram, J.E.: Designing activity-aware recommender systems for operating rooms. In: Proceedings of the 2011 Workshop on Context-awareness in Retrieval and Recommendation, 13 February 2011 (2011)
Abascal, J., Bonail, B., Marco, A., Sevillano, J.L.: AmbienNet: an intelligent environment to support people with disabilities and elderly people. In: Proceedings of ASSETS 2008, Halifax, Nova Scotia, Canada, 13–15 October 2008 (2008)
Chan, M., Campo, E., Bourennane, W.¸ Bettahar, F., Charlon, Y.: Mobility behavior assessment using a smart-monitoring system to care for the elderly in a hospital environment. In: Proceedings of PETRA 2014, Island of Rhodes, Greece, 27–30 May 2014 (2014)
Yared, R., Abdulrazak, B., Tessier, T., Mabilleau, P.: Cooking risk analysis to enhance safety of elderly people in smart kitchen. In: Proceedings of PETRA 2015, Corfu, Greece, 01–03 July 2015 (2015)
Deen, M.J.: Information and communications technologies for elderly ubiquitous healthcare in a smart home. Pers. Ubiquitous Comput. 19, 573–599 (2015)
Civitarese, G., Bettini, C., Belfiore, S.: Let the objects tell what you are doing. In: Proceedings of Ubicomp/ISWC 2016 Adjunct, Heidelberg, Germany, 12–16 September 2016 (2016)
Iglesias, R., de Segura, N.G., Iturburu, M.: The elderly interacting with a digital agenda through an RFID pen and a touch screen. In: Proceedings of MSIADU 2009, Beijing, China, 23 October 2009 (2009)
Angelini, L., Nyffeler, N., Caon, M., Jean-Mairet, M., Carrino, S., Mugellini, E., Bergeron, L.: Designing a desirable smart bracelet for older adults. In: Proceedings of UbiComp 2013, Zurich, Switzerland, 8–12 September 2013 (2013)
Khosla, R., Chu, M.-T., Kachouie, R., Yamada, K., Yoshihiro, F., Yamaguchi, T.: Interactive multimodal social robot for improving quality of care of elderly in australian nursing homes. In: Proceedings of MM 2012, Nara, Japan, 29 October–2 November 2012 (2012)
Sarkar, D.P.: A nurse bot for elderly people. In: Proceedings of UbiComp/ISWC 2018 Adjunct, Singapore, 8–12 October 2018 (2018)
Thakur, N., Han, C.Y.: An improved approach for complex activity recognition in smart homes. In: Reuse in the Big Data Era. Lecture Notes in Computer Science, vol 11602, pp. 220–231. Springer, Cham (2019)
Thakur, N., Han, C.Y.: Framework for a personalized intelligent assistant to elderly people for activities of daily living. Int. J. Recent Trends Hum. Comput. Interact. (IJHCI) 9(1), 1–22 (2019)
Thakur, N., Han, C.Y.: Framework for an intelligent affect aware smart home environment for elderly people. Int. J. Recent Trends Hum. Comput. Interact. (IJHCI) 9(1), 23–43 (2019)
Thakur, N., Han, C.Y.: A context-driven complex activity framework for smart home. In: Proceedings of the 9th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON) 2018, Vancouver, Canada, 1–3 November 2018 (2018)
Thakur, N., Han, C.Y.: A hierarchical model for analyzing user experiences in affect aware systems. In: Proceedings of the 9th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON) 2018, Vancouver, Canada, 1–3 November 2018 (2018)
Thakur, N., Han, C.Y.: An approach to analyze the social acceptance of virtual assistants by elderly people. In: Proceedings of the 8th International Conference on the Internet of Things (IoT) 2018, Santa Barbara, California, 15–18 October 2018 (2018)
Thakur, N., Han, C.Y.: Methodology for forecasting user experience for smart and assisted living in affect aware systems. In: Proceedings of the 8th International Conference on the Internet of Things (IoT) 2018, Santa Barbara, California, 15–18 October 2018 (2018)
Thakur, N., Han, C.Y.: An activity analysis model for enhancing user experiences in affect aware systems. In: Proceedings of the IEEE 5G World Forum Conference (IEEE 5GWF) 2018, Santa Clara, California, 09–11 July 2018 (2018)
Thakur, N., Han, C.Y.: A virtual wisdom mining ‘pan’ for connecting retired experts with currently active professionals. In: Proceedings of IT Research Symposium, University of Cincinnati, 10 April 2018 (2018)
Thakur, N., Han, C.Y.: A complex activity based emotion recognition algorithm for affect aware systems. In: Proceedings of IEEE 8th Annual Computing and Communication Workshop and Conference (IEEE CCWC) 2018, Las Vegas, 08–10 January 2018 (2018)
Saguna, A.Z., Chakraborty, D.: Complex activity recognition using context driven activity theory and activity signatures. ACM Trans. Comput.-Hum. Interact. 20(6), Article 32 (2013)
Microsoft, Coordinate Spaces. https://msdn.microsoft.com/enus/library/hh973078.aspx
Chakraborty, S., Han, C.Y., Zhou, X., Wee, W.G.: A context driven human activity recognition framework. In: International Conference on Health Informatics and Medical Systems HIMS 2016 (2016)
Ordóñez, F.J., de Toledo, P., Sanchis, A.: Activity recognition using hybrid generative/discriminative models on home environments using binary sensors. Sensors 13, 5460–5477 (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Thakur, N., Han, C.Y. (2021). Towards a Language for Defining Human Behavior for Complex Activities. In: Ahram, T., Taiar, R., Langlois, K., Choplin, A. (eds) Human Interaction, Emerging Technologies and Future Applications III. IHIET 2020. Advances in Intelligent Systems and Computing, vol 1253. Springer, Cham. https://doi.org/10.1007/978-3-030-55307-4_47
Download citation
DOI: https://doi.org/10.1007/978-3-030-55307-4_47
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-55306-7
Online ISBN: 978-3-030-55307-4
eBook Packages: EngineeringEngineering (R0)