Abstract
Human action recognition is an active topic of research in computer vision and machine learning. Its application in the industrial domain is even more challenging since workers can handle multiple objects and follow different assembly sequences, and only a few datasets are target-oriented. However, the availability of low-cost cameras capable of extracting high-level information about human posture and movement opens up new possibilities. This work compares four state-of-the-art graph neural networks working with skeletal data to recognize the actions in the HA4M dataset, where subjects perform an assembly task. Videos are divided into clips of consecutive frames that form the input skeletal graphs of the networks. Then, an algorithm for action segmentation is proposed to assess each action’s exact starting and ending instants. Results show that the best performance is achieved by a two-stream Adaptive Graph Convolutional Network trained with input clips 77 frames long.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Al-Amin, M., Qin, R., Moniruzzaman, M., Yin, Z., Tao, W., Leu, M.C.: An individualized system of skeletal data-based CNN classifiers for action recognition in manufacturing assembly. J. Intell. Manuf. (2021)
Berg, J., Reckordt, T., Richter, C., Reinhart, G.: Action recognition in assembly for human-robot-cooperation using hidden Markov models. Procedia CIRP 76, 205–210 (2018). In: 7th CIRP Conference on Assembly Technologies and Systems
Chen, C., Wang, T., Li, D., Hong, J.: Repetitive assembly action recognition based on object detection and pose estimation. J. Manuf. Syst. 55, 325–333 (2020)
Cicirelli, G., D’Orazio, T.: A low-cost video-based system for neurodegenerative disease detection by mobility test analysis. Appl. Sci. 13(1), 278 (2022)
Cicirelli, G., Marani, R., Romeo, L., Domínguez, M.G., Heras, J., Perri, A.G., D’Orazio, T.: The HA4M dataset: multi-modal monitoring of an assembly task for human action recognition in manufacturing. Sci. Data 9(1), 745 (2022)
Duan, H., Wang, J., Chen, K., Lin, D.: PYSKL: Towards Good Practices for Skeleton Action Recognition (2022)
Carreira, J., Zisserman, A.: Quo vadis, action recognition? A new model and the kinetic dataset. In: Proceedinfs of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299–6308 (2017)
Jegham, I., Khalifa, A.B., Alouani, I., Mahjoub, M.A.: Vision-based human action recognition: an overview and real world challenges. Forensic Sci. Int.: Digit. Investig. 32, 1–17 (2020)
Kobayashi, T., Aoki, Y., Shimizu, S., Kusano, K., Okumura, S.: Fine-grained action recognition in assembly work scenes by drawing attention to the hands. In: 15th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS), pp. 440–446 (2019)
Kong, Y., Fu, Y.: Human action recognition and prediction: a survey. Int. J. Comput. Vis. 130, 1366–1401 (2022)
Liu, J., Shahroudy, A., Perez, M., Wang, G., Duan, L., Kot, A.C.: NTU RGB+D: a large scale dataset for 3D human activity analysis. IEEE Trans. Pattern Anal. Mach. Intell. 42, 2684–2701 (2020)
Liu, Z., Zhang, H., Chen, Z., Wang, Z., Ouyang, W.: Disentangling and unifying graph convolutions for skeleton-based action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 143–152 (2020)
Mahbub , U., Ahad, M.A.R.: Advances in human action, activity and gesture recognition. Pattern Recognit. Lett. (2021)
Feng, M., Meunier, J.: Skeleton graph-neural-network-based human action recognition: a survey. Sensors 22 (2022)
Microsoft: Azure Kinect DK Documentation (2021). https://docs.microsoft.com/en-us/azure/kinect-dk/. Accessed March 2022
Özyer, T., Ak, D.S., Alhajj, R.: Human action recognition approaches with video datasets - a survey. Knowl.-Based Syst. 222, 1–36 (2021)
Romeo, L., Marani, R., Cicirelli, G., D’Orazio, T.: Video based mobility monitoring of elderly people using deep learning models. IEEE Access 11, 2804–2819 (2023)
Romeo, L., Marani, R., Malosio, M., Perri, A.G., D’Orazio, T.: Performance analysis of body tracking with the Microsoft Azure Kinect. In: 2021 29th Mediterranean Conference on Control and Automation (MED), pp. 572–577. IEEE (2021)
Sarkar, A., Banerjee, A., Singh, P.K., Sarkar, R.: 3D human action recognition: through the eyes of researchers. Expert Syst. Appl. 193, 116424 (2022)
Shahroudy, A., Liu, J., Ng, T.T., Wang, G.: NTU RGB+D: a large scale dataset for 3D human activity analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1010–1019 (2016)
Shi, L., Zhang, Y., Lu, H.: Skeleton-based action recognition with multi-stream adaptive graph convolutional networks. IEEE Trans. Image Process. 29, 9532–9545 (2020)
Shi, L., Zhang, Y., Cheng, J., Lu, H.: Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12018–12027 (2019)
Tao, W., Al-Amin, M., Chen, H., Leu, M.C., Yin, Z., Qin, R.: Real-time assembly operation recognition with fog computing and transfer learning for human-centered intelligent manufacturing. Procedia Manuf. 48, 926–931 (2020)
Wang, L., Gao, R., Vancza, J., Krüger, J., Wang, X.V., Makris, S.: Symbiotic human-robot collaborative assembly. CIRP Ann. Manuf. Technol. 68, 701–726 (2019)
Wang, Z., Qin, R., Yan, J., Guo, C.: Vision sensor based action recognition for improving efficiency and quality under the environment of Industry 4.0. Procedia CRP 80, 711–176 (2019). In: 26th CIRP Life Cycle Engineering (LCE) Conference
Yan, S., Xiong, Y., Lin, D.: Spatial temporal graph convolutional networks for skeleton-based action recognition. Comput. Res. Repos. (CoRR) (2018)
Chen, Y., Zhang, Z., Yuan, C., Deng, Y., Hu, W.: Channel-wise topology refinement graph convolution for skeleton-based action recognition. In: Proceeding of IEEE/CVF International Conference on Computer Vision, pages 13359–13368 (2021)
Zamora-Hernandez, M.A., Castro-Vergas, J.A., Azorin-Lopez, J., Garcia-Rodriguez, J.: Deep learning-based visual control assistant for assembly in Industry 4.0. Comput. Ind. 131, 1–15 (2021)
Zhang, J., Wang, P., Gao, R.X.: Hybrid machine learning for human action recognition and prediction in assembly. Robot. Comput. Integr. Manuf. 72, 1–10 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Maselli, M.V., Marani, R., Cicirelli, G., D’Orazio, T. (2024). Continuous Action Recognition in Manufacturing Contexts by Deep Graph Convolutional Networks. In: Arai, K. (eds) Intelligent Systems and Applications. IntelliSys 2023. Lecture Notes in Networks and Systems, vol 825. Springer, Cham. https://doi.org/10.1007/978-3-031-47718-8_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-47718-8_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-47717-1
Online ISBN: 978-3-031-47718-8
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)