Keywords

1 Introduction

During the construction and operation of the Mir Space Station and the International Space Station, the man machine cooperation mode of the space manipulator has played an important role.

Twice extravehicular activities of the Mir Space Station in 1991 and 1996, the Russian astronauts respectively installed 14 m long, 45 kgs weighs of telescopic Boom “Strela Boom” in the Mir Space Station core module (Base block) column section of one side and Kvant another side, as shown in Fig. 1. Each boom is able to support 700 kgs weighs, used to move a large solar array, and the astronauts can be moved from one place to another place. The boom manipulator is manually operated and the astronaut will move the manipulator around the base by turning a pair of cranks [1, 2].

Fig. 1.
figure 1

Mir space station strela boom

Full size image

The Canada arm and Canada arm 2, known as the Space Shuttle and International Space Station have played an irreplaceable role in the process of construction, operation and maintenance of the International Space Station. The astronauts can observe the operation and operating environment of the cabin through the three LCD displays on the Robot Workstation or directly through the porthole of the space shuttle. RWS unit has 2 sets of joystick: a set of rotary manual joystick, a set of translational manual joystick. The space shuttle Canada arm mainly includes autonomous mode, astronaut manual mode and single joint mode. Autonomous mode mainly includes pre-programmed trajectories and direct control of astronaut directives. The astronaut manual control mode allows the astronaut to adjust the position and pose of the end of the robot via the hand controller, mainly used for crawling, capturing non-parked loads; tracking and capturing free flight loads; coarse positioning. In the case of manipulator joints drive hardware failure, the control mode can be switched to direct drive and passive drive derating using fail-safe mode [3] (Fig. 2).

Fig. 2.
figure 2

Shuttle Remote Manipulator System preliminary system structure [3]

Full size image

Compared to the United States, Russia, Germany and Japan [4], Space manipulator technology in China starts late, but develops rapidly in recent years. In Tiangong-2 Space Laboratory mission, Chinese first space manipulator and dexterous hand man machine cooperation on orbit experiments have completed successfully. The experiments have been carried out to verify the operation of the space manipulator and the dexterous hand in the Space Laboratory, local use of commercial products, with the smallest resources and funding to obtain on-orbit data and technical accumulation. This paper gives a brief introduction to the basic system composition, control mode and working mode of the space manipulator and dexterous hand man machine cooperation experiments, the mathematical model of man machine cooperation and the method of ground verification, The correctness of the man machine cooperation scheme design and the ground verification system of the space manipulator proposed in this paper is verified through the implementation on orbit. It has accumulated some experiences for Chinese space robot technology through this experiments on orbit.

2 System Composition and Man Machine Cooperation Mode

2.1 System Composition

Space manipulator and dexterous hand man machine cooperation experiments are equipped with a manipulator, a dexterous hand, a hand-eye camera, a controller, global cameras, orbit replaceable units. The system block diagram is shown in Fig. 3.

Fig. 3.
figure 3

Manipulator and dexterous hand system structure

Full size image

The manipulator, dexterous hand and hand-eye camera are powered by the controller. The controller is the information center of the system. Control instructions and data acquisitions of manipulator, dexterous hand, hand-eye camera are emitted and incepted by the controller. The controller controls the operation of system actions. Global cameras measurement data are collected and processed through the PC, transmission by cable to controller for the use of system closed-loop control.

The personal computer acts as a man machine interface to the astronauts, and is connected to the controller via Ethernet. The astronauts can send control commands via the interaction software on the personal computer or monitor the system state. In the astronauts manual mode of operation, the astronauts can teleoperate the manipulator and dexterous hand respectively by space mouse and cyberglove.

The controller is set an emergency stop button for the astronauts for system emergency stop.

2.2 Control Mode

The system consists of three basic control modes: position control, impedance control and visual servo control.

Position control

Position control block diagram is shown in Fig. 4. In ground test, in order to unload the manipulator’s own gravity, to improve the dynamic control performance, to control accuracy, it is set the gravity term G(q) in the manipulator control loop. The gravity compensation term G(q) is set to zero during orbital operation. The control mode is the basic control mode for the manipulator operation.

Fig. 4.
figure 4

Position control of manipulator

Full size image

Impedance control

The manipulator is used an position control inner loop based impedance controller [5, 6], as shown in Fig. 5. The outer impedance control loop uses each joint torque sensor data for end effector force computed through force Jacobian, the desired position of the end of the manipulator is calculated by the admittance link. At last, the amount of change in each joint to the closed-loop feedback is calculated by inverse kinematics. This mode is mostly used in contact tasks.

Fig. 5.
figure 5

Impedance control of manipulator

Full size image

When the stiffness K in the impedance link is set to zero, the impedance control can be set to zero force control. This mode is used astronauts teaching playback.

Visual servo control

The visual servo controller receives the camera-identified position and pose information for closed-loop control, combined position control or impedance control. In the screwing screw task, if there is also a slight error between the end of the power tool and the screw hole, this small error makes the power tool head cannot accurately be put into the screw hole, but be on the edge of the screw hole. At this time, the impedance controller of the manipulator is adjusted according to the condition of the end force feedback, and the control variable of the desired position from the visual servo controller is corrected, so that the manipulator terminal will be inserted into the screw hole compliantly. The control block diagram is shown in Fig. 6. The control mode is used in the autonomous task of the manipulator system.

Fig. 6.
figure 6

Vision servo control fused impedance control

Full size image

2.3 Operating Mode

Normal conditions

The astronauts command manipulator pre-programming instructions by operating the interaction software button on the personal computer, manipulator acts according to the pre-set action. Astronauts observe the entire experiment process, confirm each step to ground at any time. Based on telemetry and astronaut feedback, ground confirms the on-orbit experiment status [7, 8].

Backup conditions

When an unexpected failure occurs during the experiment, astronauts immediately press the emergency stop button. When the ground sends instructions cannot complete the test, astronauts also press the emergency stop button or ground send the “system standby” command. At this time, astronauts operate space mouse and cyberglove to operating manipulator and dexterous hand system.

After astronauts have adjusted the position and posture of the manipulator, the operator and ground will confirm the current position and posture of the manipulator terminal. The joint angle data will be saved in the PC control software by the astronauts. The data are sent via PC to the controller, to complete the experiment task.

Fail safe conditions

When an astronaut judges that the manipulator will harm his personal safety, or if the astronaut judges that the manipulator is about to intervene with the equipment in the cabin, the astronaut can stop the system by pressing the emergency stop button.

When faced with ground path planning difficult task, the ground will switch the manipulator control mode to zero force control mode, manipulator will be driven directly by the astronauts on orbit to adjust the terminal to proper position and posture. Ground records and analyzes the position and posture by telemetry parameters.

3 Man Machine Cooperation Operation Mathematical Model

3.1 Kinematic Model of Manipulator

The manipulator and dexterous hand are modeled and controlled separately. The manipulator has six joints, each joint corresponds to a joint coordinate system.

The joint coordinate system of the manipulator uses the modified D-H method to establish the joint coordinate system of the manipulator, as shown in Fig. 7.

Fig. 7.
figure 7

Manipulator D-H reference frame definition

Full size image

Substituting the link parameters into the link transformation general formula (1), the relative transformation matrix \( {}_{i}^{i - 1} T \) of each link can be obtained, \( i = 1,2, \cdots 5,6 \).

$$ {}_{i}^{i - 1} T = \left[ {\begin{array}{*{20}c} {c\theta_{i} } & { - s\theta_{i} } & 0 & {a_{i - 1} } \\ {s\theta_{i} c\alpha_{i - 1} } & {c\theta_{i} c\alpha_{i - 1} } & { - s\alpha_{i - 1} } & { - d_{i} s\alpha_{i - 1} } \\ {s\theta_{i} s\alpha_{i - 1} } & {c\theta_{i} s\alpha_{i - 1} } & {c\alpha_{i - 1} } & {d_{i} c\alpha_{i - 1} } \\ 0 & 0 & 0 & 1 \\ \end{array} } \right] $$
(1)

The transformation matrix is multiplied by right in turn, it can be obtained the transformation matrix of the end of the manipulator relative to the origin of the coordinate system, as shown in Eq. (2).

$$ {}_{6}^{0} T = {}_{1}^{0} T(\theta 1){}_{2}^{1} T(\theta 2){}_{3}^{2} T(\theta 3){}_{4}^{3} T(\theta 4){}_{5}^{4} T(\theta 5){}_{6}^{5} T(\theta 6) $$
(2)

Sufficient conditions for the existence of closed solutions for the inverse kinematics of the manipulator (Pieper criterion): the three joints of the end of manipulator intersect at one point, Paul’s inverse transformation method can be used [9]. When the end posture is given by the RPY angle, the corresponding rotation matrix is obtained. Combined with the end position input, that is, the relative transformation matrix of the given manipulator end relative to the base coordinate system. By giving the above matrix, closed solution of joint 1 to 6 can be solved by the inverse transform method.

3.2 Kinematic Model of Dexterous Hand

The dexterous hand has five fingers, each finger structure is exactly the same. Every single finger can be seen as a joint robot with 4 joints, 3 degrees of freedom. The fourth joint and the third joint have a similar 1:1 coupling relationship, satisfying the following equation. Single finger coordinate system is shown in Fig. 8 [10]. Substituting the link parameters into the link transformation matrix general formula (1), a transformation matrix between the links can be obtained.

Fig. 8.
figure 8

Dexterous hand fingers reference frame definition

Full size image

As a finger has only three degrees of freedom, and the range of joint movement of the fingers is limited in mechanical dimensions, there is only one finger gesture at the same fingertip position. If the position of the fingertip in the base coordinate system (x, y, z) is known, according to the principle of inverse kinematics in robotics, the only inverse solution in the joint space of the finger can be obtained.

3.3 Man Machine Cooperation Mapping Model

Man machine cooperative manipulator mapping model

In this experiment, space mouse is used to control three directions of the translation and three directions of rotation of end tip of the manipulator, as shown in Fig. 9. Space mouse movement and rotation range are tiny, so in this experiment space mouse output is set to the end increment of the manipulator [8].

Fig. 9.
figure 9

Space mouse

Full size image

Space mouse and personal computer exchange data using USB2.0 port. Astronauts use the space mouse to control the end position and posture of the manipulator, the astronauts - manipulator interface logic as shown in Fig. 10.

Fig. 10.
figure 10

Space mouse cartesian space control logistic chart

Full size image

When the astronauts operate manipulator and dexterous hand, the interface program detects the space mouse event, reads the incremental value of the six directions of space mouse, to accumulate the current actual position of the manipulator. If this position does not exceed the movement range of the manipulator and is not singular, the human-machine interface program will perform inverse kinematics calculations. Through filting multi-solution, the control variables of the joint space of the manipulator are finally generated.

If the control data of the space mouse will make the manipulator out of the range of motion, the program returns the Cartesian posture to the last value, until the data entered by the space mouse again causes the manipulator to be in the range of motion.

Man machine cooperative dexterous hand mapping model

The commercial cyber gloves are selected for dexterous hand man-machine cooperative operating. The information exchange interface between the cyber glove and the personal computer are USB 2.0. At each finger, there are 3 joint buckling sensors, 4 joint outreach sensors, 1 palm sensor, and 2 measurements of joint flexion and joint abduction sensors. The 19 joint angle sensors which are operated by man-machine cooperation. During the ground test, the joint sensors measurement angle of cyber gloves are calibrated according to the astronauts’s palm. The control of the dexterous hand uses the position mapping of the joint space. For each joint of the dexterous hand, a first order linear function is used for mapping as shown in Eq. (3) [11].

$$ G_{0} = AGi + B $$
(3)

In Eq. (3):

Gi :

——Cyber glove sensors measurement matrix (5 × 3);

G0 :

——Calibrated dexterous hand control matrix (5 × 3);

A:

——Calibration gain coefficient matrix (3 × 5);

B:

——Calibration bias matrix (5 × 3).

The motion relationship of the astronaut controlling dexterous hand by cyber glove is shown in Fig. 11. Cyber gloves measure the angle of the joints of the astronaut’s fingers, control the movement of the dexterous hand, through the joint position mapping to dexterous hand joints space.

Fig. 11.
figure 11

Astronaut and dexterous hand cooperation mapping relation based cyber glove

Full size image

4 Ground Test

In this paper, two kinds of ground test to complete the space manipulator system man machine on-orbit cooperation experiment verification work. First of all, through simulation verification to ensure that the manipulator system trajectory planning is reasonable. Through the ground flight system 1:1 physical test to ensure that experiment of the flight procedures are reasonable, the failure schemes are feasible.

4.1 Simulation Verification

The simulation system is worked by VC++. Using the 3D model of the design phase to transform into the 3D model of Open Inventor.

The root of the simulation environment is located at the center of the base joint of the robot arm, which is the origin of the kinematic base coordinates of the manipulator. And then the various parts of the model are loaded in program with the Open Inventor class library function, placed into the scene for assembly. The relationship between the two models follows the parent-child relationship, that is, the movement of the parent node affects his child nodes, and the movement of the child nodes cannot affect the parent node. Such as between any joints, as shown in Fig. 12.

Fig. 12.
figure 12

Simulation models assembly relation and data drive relation

Full size image

In this relationship, it is formed a control chain from the base to the dexterous hand fingertip. The control chain is as a node joined the virtual scene. The center point of the base joint is as the origin point, the aircraft platform, the orbital replacable units are located in the scene by the actual position [12]. A simulation system for the space manipulator system based on graphical predictive emulation is shown in Fig. 13.

Fig. 13.
figure 13

Manipulator and dexterous hand system man machine cooperation simulation

Full size image

A pre-programmed trajectory instruction is used to drive the simulation system, or the man-machine interaction device cyber gloves and a space mouse to drive the manipulator and dexterous hand directly in the simulation model.

4.2 Test Verification

Before on orbit implementation, 1:1 test verification is carried out using ground flight system. As shown in Fig. 14, The ground technicians were operating the manipulator and dexterous hand through space mouse and cyber gloves.

Fig. 14.
figure 14

Manipulator and dexterous hand system man machine cooperation test validation

Full size image

Besides further verifying the manipulator and dexterous hand operating trajectory, in test verification, it is more important to verify the real dynamic response and tracking performance of the manipulator and dexterous hand hardware system during manual operating, the correctness of the fault plan switching, the ergonomic performance.

5 On-Orbit Implementation

Space manipulator system man machine cooperation has been tested successfully in Tiangong-2 Space Laboratory Missions on-orbit operation display task.

The astronauts operated the dexterous hand by the cyber gloves, and successfully completed the gestures specified in the flight procedure as shown in Fig. 15, the trajectory following by dexterous hand is smooth and stable as shown in Fig. 17a.

Fig. 15.
figure 15

Joint angle data of dexterous hand man machine cooperation on-orbit experiment

Full size image
Fig. 16.
figure 16

Joints following error of manipulator man machine cooperation on-orbit experiment

Full size image
Fig. 17.
figure 17

Telemetry images of manipulator man machine cooperation on-orbit experiment

Full size image

As shown in Fig. 17b, the astronauts operated the manipulator by the space mouse, and successfully completed the movement requirements specified in the flight procedure. As shown in Fig. 16, the joins error of manipulator is not greater than 0.01°, the tracking performance of manipulator is well.

6 Conclusion

Through the successful implementation of the on-orbit experiment, it is proved that the system configuration and control mode presented in this paper are reasonable, and the man-machine cooperation mode is reasonable, and the ground verification method combining simulation and physical test is feasible.

The experiment has opened up the manipulator man machine cooperation mode in our country, and has carried on the beneficial exploration to the system configuration structure, the necessary control mode and the ground verification technology system of this model. The experiment has laid the technical foundation for the man-machine cooperation mode of astronauts and manipulators in the China’s space station mission. The overall layout of the experiment is in the cabin, it has carried on space manipulator technology verification. The subsequent part of the manipulator transplanted outside the cabin for operation missions, space manipulator composition structure, operation mode, control mode and ground test methods can be inherited.

In subsequent missions, when the astronauts in cabin control the manipulator outside the cabin, it is noteworthy to control the time delay in the closed-loop system and set up the necessary cabin cameras to meet the astronauts’ observation requirements.