Abstract
The paradigm for robot usage has changed in the last few years, from a scenario in which robots work isolated to a scenario where robots collaborate with human beings, exploiting and combining the best abilities of robots and humans. The development and acceptance of collaborative robots is highly dependent on reliable and intuitive human-robot interaction (HRI) in the factory floor. This paper proposes a gesture-based HRI framework in which a robot assists a human co-worker delivering tools and parts, and holding objects to/for an assembly operation. Wearable sensors, inertial measurement units (IMUs), are used to capture the human upper body gestures. Captured data are segmented in static and dynamic blocks recurring to an unsupervised sliding window approach. Static and dynamic data blocks feed an artificial neural network (ANN) for static, dynamic, and composed gesture classification. For the HRI interface, we propose a parameterization robotic task manager (PRTM), in which according to the system speech and visual feedback, the co-worker selects/validates robot options using gestures. Experiments in an assembly operation demonstrated the efficiency of the proposed solution.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Yang G-Z, Bellingham J, Dupont PE, Fischer P, Floridi L, Full R, Jacobstein N, Kumar V, McNutt M, Merrifield R, Nelson BJ, Scassellati B, Taddeo M, Taylor R, Veloso M, Wang ZL, Wood R (2018) The grand challenges of science robotics. Science Robotics 3:14
Johannsmeier L, Haddadin S (Jan 2017) A hierarchical human robot interaction planning framework for task allocation in collaborative industrial assembly processes. IEEE Robot Autom Lett 2(1):41–48
Sadrfaridpour B, Wang Y (2017) Collaborative assembly in hybrid manufacturing cells: an integrated framework for human-robot interaction. IEEE Trans Autom Sci Eng PP(99):1–15
Kaipa K, Morato C, Liu J, Gupta S (2014) Human-robot collaboration for bin-picking tasks to support low-volume assemblies. Robotics Science and Systems Conference
Matsas E, Vosniakos G-C, Batras D (2017) Effectiveness and acceptability of a virtual environment for assessing human–robot collaboration in manufacturing. Int J Adv Manuf Technol 92(9):3903–3917
Makrini IE, Merckaert K, Lefeber D, Vanderborght B (2017) Design of a collaborative architecture for human-robot assembly tasks. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1624–1629
Ende T, Haddadin S, Parusel S, Wüsthoff T, Hassenzahl M, Albu-Schäffer A (2011) A human-centered approach to robot gesture based communication within collaborative working processes. IROS 2011, San Francisco
Sheikholeslami S, Moon A, Croft E (2017) Cooperative gestures for industry: Exploring the efficacy of robot hand configurations in expression of instructional gestures for human-robot interaction. Int J Robot Res 36 (5-7):699–720
Rouanet P, Oudeyer P, Danieau F, Filliat D (2013) The impact of human-robot interfaces on the learning of visual objects. IEEE Trans Robot 29(2):525–541
Radmard S, Moon AJ, Croft E (2015) Interface design and usability analysis for a robotic telepresence platform. In: 2015 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pp 511–516
Simao M, Neto P, Gibaru O (2016) Natural control of an industrial robot using hand gesture recognition with neural networks. In: IECON 2016 - 42nd Annual Conference of the IEEE Industrial Electronics Society, pp 5322–5327
Wolf MT, Assad C, Vernacchia MT, Fromm J, Jethani HL (2013) Gesture-based robot control with variable autonomy from the JPL biosleeve. in: 2013 IEEE International Conference on Robotics and Automation. IEEE, pp 1160–1165
Neto P, Pereira D, Pires JN, Moreira AP (2013) Real-time and continuous hand gesture spotting: An approach based on artificial neural networks, 2013 IEEE International Conference on Robotics and Automation, pp 178–183
Gleeson B, MacLean K, Haddadi A, Croft E, Alcazar J (2013) Gestures for industry intuitive human-robot communication from human observation, in 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp 349–356
Goldin-Meadow S (1999) The role of gesture in communication and thinking. Trends Cogn Sci 3(11):419–429
Feldman RS (1991) Fundamentals of nonverbal behavior. Cambridge University Press, Cambridge
Waldherr S, Romero R, Thrun S (2000) A Gesture Based Interface for Human-Robot Interaction. Auton Rob 9(2):151–173
Quintero CP, Fomena RT, Shademan A, Wolleb N, Dick T, Jagersand M (2013) Sepo: Selecting by pointing as an intuitive human-robot command interface. In: 2013 IEEE International Conference on Robotics and Automation (ICRA), pp 1166–1171
Burke M, Lasenby J (2015) Pantomimic gestures for human–robot interaction. IEEE Trans Robot 31 (5):1225–1237
Okuno Y, Kanda T, Imai M, Ishiguro H, Hagita N (2009) Providing route directions: Design of robot’s utterance, gesture, and timing, pp 53–60
Salem M, Kopp S, Wachsmuth I, Rohlfing K, Joublin F (2012) Generation and Evaluation of Communicative Robot Gesture. Int J Soc Robot 4(2):201–217
Huang C-M, Mutlu B (2013) Modeling and evaluating narrative gestures for humanlike robots. In: Inproceedings of Robotics: Science and Systems, pp 15
Wongphati M, Osawa H, Imai M (2015) Gestures for manually controlling a helping hand robot. Int J Soc Robot 7(5):731–742
Shao Z, Li Y (2015) Integral invariants for space motion trajectory matching and recognition. Pattern Recogn 48(8):2418–2432
Simao MA, Neto P, Gibaru O (2017) Unsupervised gesture segmentation by motion detection of a real-time data stream. IEEE Trans Ind Inf 13(2):473–481
Alon J, Athitsos V, Yuan Q, Sclaroff S (2009) A unified framework for gesture recognition and spatiotemporal gesture segmentation. IEEE Trans Pattern Anal Mach Intell 31(9):1685–99
Yang R, Sarkar S, Loeding B (2010) Handling movement epenthesis and hand segmentation ambiguities in continuous sign language recognition using nested dynamic programming. IEEE Trans Pattern Anal Mach Intell 32(3):462–77
Burger B, Ferrané I, Lerasle F, Infantes G (2011) Two-handed gesture recognition and fusion with speech to command a robot. Auton Robot 32(2):129–147
Villani V, Sabattini L, Riggio G, Secchi C, Minelli M, Fantuzzi C (2017) A natural infrastructure less human robot interaction system. IEEE Robot Autom Lett 2(3):1640–1647
Wu D, Pigou L, Kindermans PJ, LE N, Shao L, Dambre J, Odobez JM (2016) Deep dynamic neural networks for multimodal gesture segmentation and recognition. IEEE Trans Pattern Anal Mach Intell PP (99):1–1
Ordonez FJ, Roggen D (2016) Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors 16:1
Field M, Stirling D, Pan Z, Ros M, Naghdy F (2015) Recognizing human motions through mixture modeling of inertial data. Pattern Recogn 48(8):2394–2406
Song Y, Demirdjian D, Davis R (2012) Continuous body and hand gesture recognition for natural human-computer interaction. ACM Trans Interact Intell Syst 2(1):1–28
Monnier C, German S, Ost A (2015) A Multi-scale Boosted Detector for Efficient and Robust Gesture Recognition. Springer International Publishing, Cham, pp 491–502
Mei K, Zhang J, Li G, Xi B, Zheng N, Fan J (2015) Training more discriminative multi-class classifiers for hand detection. Pattern Recogn 48(3):785–797
Pedersen MR, Krüger V (2015) Gesture-based extraction of robot skill parameters for intuitive robot programming. J Intell Robot Syst 80(1):149–163
Rossi S, Leone E, Fiore M, Finzi A, Cutugno F (2013) An extensible architecture for robust multimodal human-robot communication. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp 2208–2213
Cacace J, Finzi A, Lippiello V, Furci M, Mimmo N, Marconi L (2016) A control architecture for multiple drones operated via multimodal interaction in search rescue mission. In: 2016 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), pp 233–239
Møller MF (1993) A scaled conjugate gradient algorithm for fast supervised learning. neural Netw 6(4):525–533
Safeea M, Bearee R, Neto P (2018) End-Effector Precise Hand-Guiding for collaborative robots. Springer International Publishing, Cham, pp 95–101
Madsen O, Bøgh S, Schou C, Andersen RS, Damgaard JS, Pedersen MR, Krüger V (2015) Integration of mobile manipulators in an industrial production. Ind Robot: Int J 42(1):11–18
Funding
This work was supported in part by the Portuguese Foundation for Science and Technology (FCT) project COBOTIS (PTDC/EME-EME/32595/2017), and the Portugal 2020 project DM4Manufacturing POCI-01-0145-FEDER- 016418 by UE/FEDER through the program COMPETE2020.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Neto, P., Simão, M., Mendes, N. et al. Gesture-based human-robot interaction for human assistance in manufacturing. Int J Adv Manuf Technol 101, 119–135 (2019). https://doi.org/10.1007/s00170-018-2788-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00170-018-2788-x