Keywords

1 Introduction

Robots can provide accurate, repetitive and fast working, but it does not have flexibility and adaptability like humans. Hence, human–robot collaboration (HRC) is a recent trend of robotics research to include capabilities of human and robot both in the industry (Wang et al. 2019). In manufacturing industry, human–robot collaboration (HRC) focuses on working of human and robot simultaneously in the workspace (Wang et al. 2017).

2 Human–Robot Collaboration (HRC)

In different scenarios like coexistence, cooperation, interaction and collaboration, relationships of the humans and robots are different. It can be evaluated based on the following criteria (Wang et al. 2019):

  • Workspace: Working area.

  • Contact: Direct physical contact with each other.

  • Working task: Same operation towards the same working objective.

  • Resources: Available equipment and machines.

  • Simultaneous process: Working on same time but different tasks.

  • Sequential process: No overlapping of task, one by one task.

Based on above criteria, different human–robot relationships can be defined as follows (Wang et al. 2020) (Table 1):

Table 1 Different kinds of (human–robot relationship Wang et al. 2020)
Full size table

In Fig. 1, possible cases and roles of human–robot collaboration have been given.

Fig. 1
figure 1

Possible cases and roles in human–robot collaborations (Wang et al. 2020)

Full size image

Human–robot collaborative system should have following characteristic:

  • It should follow safety standards and operational regulations

  • Flexibility and adaptability for specific functions

  • Potential of improvement in productivity and product quality

  • Support, assistance and collaboration with human in hazardous, tedious, non-ergonomic and repetitive tasks.

3 Gesture Recognition in Human–Robot Collaboration (HRC)

In manufacturing with robots, robots in the industries are pre-programmed with rigid codes, and reprogramming is a challenging task as it is tedious and time consuming (Liu et al. 2018). Although by programming with demonstration, reprogramming of robots for a different task has been made easy. Rigid code-based robotic control cannot support human–robot collaboration. To solve this problem, intuitive programming with communication channels like gesture, poster, voice, and haptic can be used to implement in human–robot collaboration.

As shown in Fig. 2, gesture recognition for human–robot collaboration can consist of sensor data collection, gesture identification, gesture tracking, gesture classification and gesture mapping (Liu and Wang 2018).

Fig. 2
figure 2

Model for human–robot collaboration (Liu and Wang 2018)

Full size image

There are three types of gesture that can be used as a communication channel as follows:

  • Head and face gesture

  • Hand and arm gesture

  • Full body gesture.

Hongyi Liu and Lihui Wang have reviewed gesture recognition for human–robot collaboration and identified research trend as follows (Liu and Wang 2018) (Fig. 3).

Fig. 3
figure 3

Past work and future trends in gesture recognition (Liu and Wang 2018)

Full size image

4 Computer Vision as a Sensor Technology in Human–Robot Collaboration

To implement human–robot collaboration, proper communication channel must be implemented. Computer vision-based communication channels can be applied effectively for the same (Liu and Wang 2018). For gesture recognition, sensor data collection needs to be done. As shown in Fig. 4, various sensors can be used for the same (Liu and Wang 2018).

Fig. 4
figure 4

Different type of sensors for gesture recognition (Liu and Wang 2018)

Full size image

In image-based sensor, single camera and depth camera are widely used in recent gesture recognition research. Depth camera will also give depth information along with RGB data. The depth and visual information provided by the depth sensor (like Kinect sensor) can be used in object tracking and recognition, human activity analysis, hand gesture analysis and indoor sematic segmentation for human–robot collaboration (Papadopoulos et al. 2014; Pham et al. 2016). Fig. 5 shows basic structure and working of depth sensors like Kinect (Han et al. 2013).

Fig. 5
figure 5

Basic structure and working of depth sensors like Kinect (Han et al. 2013)

Full size image

For effective human–robot collaboration, human gesture should be correctly understood by robot and so human gesture should be interpreted mathematically (Nakamura 2018). In HRC system, mathematical modelling of sequence of human operations and motions is very important.

Liu H. has modelled sequence of human motion in assembly line using existing motion recognition techniques and applied hidden Markov model (HMM) to predict human motion by generating a motion transition probability matrix (Liu and Wang 2017). Example of task-level representation in an assembly line for a case study is shown in Fig. 6.

Fig. 6
figure 6

Example of task-level representation in an assembly line (Liu and Wang 2017)

Full size image

Liu H has also applied recurrent neural network (RNN) to predict human motion by generating a motion transition probability matrix (Zhang et al. 2020). Overview of RNN for motion prediction in human–robot collaboration (HRC) is shown in Fig. 7.

Fig. 7
figure 7

Overview of RNN for motion prediction (Zhang et al. 2020)

Full size image

Mainprice and Berenson (2013) proposed framework based on early prediction of the human’s motion. These generate a prediction of human workspace occupancy by computing the swept volume of learned human motion trajectories. This framework is based on generation of a prediction of human workspace occupancy by calculating the swept volume of learned human motion trajectories (Mainprice and Berenson 2013). In Fig. 8, human workspace occupancy has been explained by example.

Fig. 8
figure 8

Example of human occupancy in human–robot collaboration (Mainprice and Berenson 2013)

Full size image

In recent years, multimodal approach has also been widely investigated. This approach is multidisciplinary including fields like artificial intelligence (AI), image processing, automation control, sensor networks and path planning (Liu et al. 2018). For human–robot collaboration, multimodal programming by deep learning approach has been explained in Fig. 9.

Fig. 9
figure 9

Multimodal conceptual framework for (human–robot collaboration Liu et al. 2018)

Full size image

5 Human Safety in Human–Robot Collaboration (HRC)

In implementing human–robot collaboration (HRC) for industry, the most important characteristic of HRC is human safety. The seven elements of the collision events are pre-collision, detection, isolation, identification, classification, reaction and collision avoidance (Haddadin et al. 2017). Possible causes of collision can be categorized into three categories as shown in Fig. 10.

Fig. 10
figure 10

Possible causes of collision (Haddadin et al. 2017)

Full size image

In computer vision-based human–robot collaboration (HRC), some research like Schmidt implements a depth camera-based method for collision detection (Mohammed et al. 2017).

Depth camera (like a Kinect sensor) can be used to calculate relative distance for collision detection as shown in Fig. 11.

Fig. 11
figure 11

Depth camera-based collision avoidance system (Mohammed et al. 2017)

Full size image

6 Conclusion

This paper explains research in computer vision-guided human–robot collaboration for Industry 4.0 in brief. After introducing the human–robot collaboration, different types of human–robot relationship and roles have been explained. In the second section of the paper, gesture recognition in human–robot collaboration has been studied as the most important element of human–robot collaboration systems. In the third section of the paper, sensor data collection needs to be done, and computer vision-based gesture recognition has been identified as the best suitable method. In the fourth section of the paper, recent research in computer vision-based human–robot collaboration has also been reviewed. In the fifth section of the paper, human safety for human–robot collaboration has been reviewed.