1 Introduction

Stroke is one of the major causes of death and disability worldwide. Approximately 70 % to 80 % of the patients who experience strokes are subsequently suffer from varying degrees of physical dysfunction; timely and effective rehabilitation training is conducive to the functional recovery of these patients. For patients with hemiplegia after stroke, the functional recovery of their upper and lower limbs is significantly different because the gross motor of the lower limbs recovers faster. By contrast, the upper limbs are relatively slower and more difficult to recover, because they primarily perform more flexible, coordinated skills, occupying a large projection area in the somatic sensation and movement functional zones in the cerebral cortex. The disorders in the motor function of hands greatly affect the daily living activities of patients. The hand is one of the most important structures of human body. The hands can not only complete many gross motor skills but also engage in various sophisticated activities that are closely related to daily life. Rehabilitation of the hand function has become a challenge and hotspot of modern rehabilitation medicine research.

In recent years, the principle of neural function recovery has been gradually clarified in the process of rehabilitation. In patients with central nervous system damage, the brain (including the spinal cord) has a certain plasticity and reconstruct characteristic of regional function. Stroke rehabilitation treatment is based on the principle of plasticity and functional reorganization of the nervous system, to promote the upper center of the movement control, and to inhibit the abnormal and original reflex activities, so as to reconstruct normal movement patterns and enhance muscle strength. The structure and function of the central nervous system have the ability of compensatory and functional reorganization in patients with stroke after nerve function injury, namely, the plasticity of brain; meanwhile, other brain cells usually take the regeneration of axons, sprouting of dendrites and synaptic threshold changes as the foundation of the physiological, biochemical and morphological changes of brain plasticity, but this kind of plasticity is always obtained depend on the special function exercise and the repeated practice activity. Training can effectively wake up the residual function of neurons to participate in activities, so as to contribute to the recovery and reconstruction of sensory feedback, prompting constantly modification of the quality of the action via the central nervous system of the brain, eventually result in the restoration of the original function or the establishment of a new area of compensation, to achieve functional remodeling and to improve the skills of the body [10, 14, 19, 24, 26].

People began using the virtual reality (VR) technology for rehabilitation training as early as the 1990s. VR technology is a computer-simulated life that is replicated from physical presence in the real world. Users can interact with the virtual environment via sensor devices, which helps achieve a direct natural interaction between users and the virtual environment. The feasibility and effectiveness of the VR technology in the field of rehabilitation have been preliminarily confirmed [12]. The VR technology can simulate the 3D world, but the virtual environment created is entirely separated from the real-life scenario. A much better unity of real and virtual scenarios may be achieved if they are combined.

On the basis of the VR technology, augmented reality (AR) is developed as a new technology to improve the perception of real world via the information or real-world elements augmented by the computer system. This technology applies virtual information to the real world, overlays virtual objects or scenes generated by the computer to the real scenario, and thus achieves reality enhancement. Users can achieve real-time 3D interaction between real and virtual objects in such an environment with a more natural viewpoint. Applied to hand rehabilitation, the AR technology has more advantages in perception, in addition to the merits of rehabilitation techniques based on VR. The users are not isolated from the real surroundings; instead, they can interact with the real scene and virtual objects in a more natural way, thereby enhancing the authenticity, interactivity, and practicability.

Applying the AR technology to hand rehabilitation is currently a cutting-edge technology and a hotspot in community. Based on the familiar movements in our daily lives, the AR technology provides goal active movements that can be repeated. The repeating goal active movements in the therapeutic process can effectively reawaken neuronal cells with residual functions to participate in activities, thus providing vision, hearing, and a variety of feedback for patients. The patients are then motivated to exercise repeatedly, and psychological therapy is run through the rehabilitation process. In this way, patients can train their hand functions in the AR environment and transfer the movement skills they learned into the real environment. The data obtained through this training can directly show the hand rehabilitation of patients, and the cost is lower than that of hand detection equipment, e.g., the hand motion detection glove [7]. This study designs a training system for hand rehabilitation based on the AR technology, which provides an interesting and natural environment for the rehabilitation of patients. In this system, we use tangible objects to enable patients to touch them and interact with the real and virtual environments. Consequently, patients can obtain better tactile feedback during the training process; such feedback benefits the restoration of muscle strengths. In Section II, the status of applying the AR technology to hand rehabilitation is shown. In Section III, the framework and realization of the system are introduced. In Sections IV and V, four training scenes and the usability research result of the system are presented, respectively. Finally, in Section VI, the contents of this paper are summarized, and the research direction for future work is provided.

2 Related work

Several attempts have been made to apply the AR technology to the circle of hand rehabilitation [18]. Luo et al. developed a training environment that integrates AR with assistive devices for post-stroke hand rehabilitation [15, 16]. The AR element of the environment utilizes a head-mounted displayer and virtual objects for a reach-and-grasp task training. The assistive device consists of either a body-powered orthosis (BPO) or a pneumatic-powered device (PPD), both of which are incorporated into gloves. For the BPO, assistance is controlled by the voluntary movement of the unaffected arm of the user; for the PPD, assistance is controlled by a combination of electromyography (EMG) signal with the difference between present and desired hand -opening angles. A therapist, who can be either on-site with the user or watching off-site through a video camera feed, supervises the movement of the user. The therapist can dynamically modify the virtual scene to efficiently meet the needs of the user.

Sucar et al. developed a web-based hand treatment system [23], in which patients can perform arm movements and regularly interact with therapists at home or in the clinic. At a low cost (only two webcams and a computer), the vision algorithm in this system can locate and track the hand of patients by collecting color and movement information. For improved rehabilitation, this system also simulates a real-life scene. However, the system cannot perceive the bending of fingers. Thus, this designed movement scene is only suitable for upward and downward movements.

Shen et al. developed a rehabilitation system for hand movements [22]. Two rehabilitation stages according to the physical conditions of patients, namely, a severe stage (in which the hand of a patient cannot be moved at all) and a minor stage (in which the hand of the user can be moved with effort), are provided in this system. In the rehabilitation process designed for the severe stage, one wireless 5DT data glove is worn as an input device on the normal functioning hand of the patient or the hand of an assistant. A virtual hand is superimposed onto the affected hand, and the movements of the input glove can be mapped to the virtual affected hand. By using this approach, the system can provide cognitive stimuli to give the patient the feeling that the affected hand can be moved. For the minor rehabilitation stage, an exercise scenario is designed to recover the motor skills of the fingers, and it covers a range of motions and movement speeds. In the scenario, five virtual keys are rendered above the fingers, and the patient has to flex a finger to touch the key that is randomly selected.

Zhang et al. developed an AR system to virtually play a piano [25], which can be utilized to train finger dexterity. A computer, a webcam, and a self-made wireless data glove are used as hardware for this system. The webcam can capture the scene in the real world, and the glove is marked. The bending sensor on the glove can detect the angle of finger bending and transfer the data via Bluetooth. Given that the virtual hand overlaps the marks in the real environment, it can interact with the virtual keyboard when covered on the real hand. At the beginning of the game, users can select the degree of difficulty they prefer and press the corresponding button by following the given hint. Scores are given based on the degree of difficulty, degree of accuracy, and response time. The system also provides visual and auditory feedback. However, such a system can detect neither the side way movement of a finger nor the thumb movement, although it can detect the curvature of fingers.

Alamri et al. proposed a novel approach based on AR technologies that can increase the involvement of a stroke patient in the rehabilitation process [1, 2]. The framework provides natural-force interaction with the daily environment by adopting a tangible-object concept. This framework contains a patient subsystem and a therapist subsystem. The hardware of the patient subsystem comprises a head -mounted displayer and a computer. Once the system is successfully logged in, it will check the treatment plan and present a training scenario for the patient. A typical feature of this framework is the decision-support engine in the therapist subsystem, which designs a decision model based on eight elements extracted from the treatment process of the patient. A proper model can continuously and automatically supervise the progress of the patient and then send it to the therapist.

Aung et al. developed an upper-limb rehabilitation system based on the AR technology [3, 4], which consists of two parts, namely, a rehabilitation training module and a real-time muscle-detection module. This system collects the EMG signal of patients and extracts the relevant information. These two modules can be combined for use in the rehabilitation training of patients in hospitals. As an independent domestic rehabilitation system, the rehabilitation training comprises four training programs, namely, collecting balloons (for shoulder, elbow, and wrist movements), playing table tennis (for curvature movement), passing objects (for flexion and horizontal adduction of shoulder joints and relevant muscular movements), and feeding animals (for extending the range of arm movement and muscular strengths).

Hondori et al. also developed a rehabilitation system based on AR [13]. The installation of this system includes a computer, a webcam, and a program. The system can provide a platform to program various virtual objects on it when a user sits in front of the table. The webcam can then track the tested hand movement when the user is interacting with the virtual object. This system can be used both locally and remotely. Such a system also develops a variety of vision algorithms and uses color markers to locate and track the tested hand and movement information. On the basis of daily activities, the system designs four tasks, namely, hand stretching, slant, pointing, and capture, in which training scenes can be modified by therapists.

Burke et al. designed two upper-limb rehabilitation games based on AR [6]. One game is the Brick-a-Break, which offers players a row of bricks above the court. The player needs to control the racket by moving a real object (such as a cube) attached with a tag and to rebound the moving ball on the court by the racket. If the ball bumps against the above bricks, such bricks will break and disappear, while the ball will rebound in the opposite direction. The other game is the Shelf Stack, in which people can place several real objects of different shapes, sizes, and weights with tags attached. Each real object has a corresponding virtual object on a computer screen. The users must correctly select and place the object in the required position. However, the system does not have the function of data storage and analysis.

Regenbrecht et al. proposed a hand rehabilitation system based on AR [11, 20, 21]. The treatment system includes two black boxes with curtains, two cameras installed inside the boxes, and a computer. The user sits in front of the computer and puts both hands into the boxes, and the movements of the hands can be captured by the cameras in the boxes. During the training, the training scene is shown on the display, i.e., small actual hand movements lead to perceived large movements. A therapist needs to sit next to the user to help complete the training.

Choi et al. developed a novel ubiquitous rehabilitation system [8], in which a novel design of data glove (eGlove) for stroke rehabilitation is presented to engage stroke patients in an AR environment of a mobile phone. A patient wears an eGlove, and the hand motion captured by the glove is transmitted via Bluetooth to a mobile phone. The captured motion data can be transmitted to the other mobile phone of another patient or a therapist at a different place. AR software generates a real-time 3D hand from a mathematical model of a hand. Therefore, each patient can observe the hand motion of other patient in the AR display of his mobile phone. A therapist can also use the eGlove to show a teacher movement to a patient at a remote place because the movement can be synchronized in the AR scene.

Most of the existing hand rehabilitation systems need to be equipped with auxiliary equipment (such as head-mounted display and data gloves), the system cost is high, and the training project is relatively small. This study specially designs a rehabilitation training system based on AR for hand rehabilitation. Hand pendency, lift, touch, push, press, grasp, pinch, and other movements are included in the system. The system provides diverse forms of training that feature flexible and abundant contents. The neurons with residual function of patients can be effectively aroused to participate in activities by repeating the active movement of the hands, thereby achieving the desired outcome of rehabilitation. The system cost is low because it requires only a computer and a camera, conveniently facilitating the training of the trainee at hospital or home. During the training process, we adopt some feedback, record the data in real time, and send information to the therapist. In the light of the needs of the hand rehabilitation training, we also design a color marker for hand rehabilitation training.

3 Rehabilitation system

3.1 System framework

The system framework is shown in Fig. 1.

Fig. 1
figure 1

System framework diagram

Full size image

A patient needs to select a program for training when logged in. The system then establishes a virtual environment in front of the patient and loads it to a real scene captured. The position and direction of this virtual environment are set by markers. In the training process, the relevant data are recorded, which are then sent to a therapist to formulate the specific training scheme for the patient after the training process.

3.2 Implementation

In the hand rehabilitation training system proposed in this study, the user is required to hold the marker in hand while moving, but the marker may be occluded by the fingers, thereby affecting the registration of the marker. Among the systems described above, most of them adopt Artoolkit marker, which, however, cannot be occluded in use. For this reason, on the basis of Hamming, we design a colorful marker suitable for the hand rehabilitation training, known as color Hamming marker (CH-marker), to solve the problem that the marker may be partially occluded in a dynamic scene.

The CH-marker consists of 7 × 7 squares, and each square is equivalent to a binary code. Following the Hamming coding principle, the first, second, and fourth bits of each row are verification bits, and the third, fifth, sixth, and seventh bits are information bits, for a total of 7 × 4 = 28 information bits.

The codeword of the CH-marker is not rotation invariant. Therefore, the square of row 1, column 1 represents the marker direction. When the binary code is “1”, yellow is used to fill; when the binary code is “0”, white is used to fill.

As mentioned above, the CH-marker is divided into two parts to encode. Yellow and white are used to fill the 3 × 3 inner squares. Yellow represents binary bit “1”, and white represents binary bit “0”. Red and blue are used to fill the outer squares. Blue represents binary bit “1”, and red represents binary bit “0” (Fig. 2).

Fig. 2
figure 2

Part of the CH-markers. If the boundary squares of this marker are partially occluded, when using the CH-marker, we can use the Hamming code to detect the occluded bits of a codeword to solve the occlusion problem

Full size image

The decoding method of CH-markers consists of four steps, namely, the threshold process of the original image, extracting quadrilateral contours, binary decoding, and 3D registration.

The color images of video frames are first converted into binary images. After threshold processing, quadrilateral contour detection is performed by using the Canny operator to detect edges and to extract the contours of images. The shape of each contour is then analyzed, and the contours that enclose small areas are removed. The contour is evaluated whether it is concave or convex, and the quadrilateral contours that meet the conditions are selected.

The quadrilateral images are first converted into rectangular ones by affine transformation. The rectangular images are then converted into a hue-saturation-value (HSV) color space. A significant distinguishing method exists for each color area of a CH-marker in the HSV color space. The standard ranges for H, S, and V are 0° to 360°, 0 to 1, and 0 to 1, respectively (see Table 1).

Table 1 Color range in this study
Full size table

A marker consists of 7 × 7 squares. The inner 3 × 3 squares are first decoded. Yellow represents binary bit “1,” and white represents binary bit “0.” Therefore, the color of this area is easy to detect. The color of the outer part is then detected. A blue square represents “1,” and a red square represents “0.” After the colors are detected, the marker direction is determined. In this study, direction is determined by decoding the area of row 1, column 1, i.e., whether it is “1” or “0” or whether it is yellow or white. If the area is neither white nor yellow, then the binary code of the marker will be shifted counterclockwise until the area is either yellow or white. If the area is either yellow or white, then the codeword of each row of the marker will be compared with 16 possible codewords. The Hamming code distances of the codeword of each row are measured. If the seven code distances are all 0, then the decoding is successful.

The CH-marker solves the problem that registration fails when markers are partially occluded in dynamic scenes. As shown in Fig. 3, a CH-marker and an Artoolkit marker are placed in the same background for tracking and registration. When the outer borders of two markers are occluded by the fingers of the user, the CH-marker is still able to be stably registered, but the Artoolkit marker failed. The maximum occluded area ratio of the CH-marker is 36.73 %. Experiments show that the proposed marker is effective, reliable, and capable of meeting the application demand of AR. The details of the experimental process are not presented in this paper.

Fig. 3
figure 3

CH-marker and Artoolkit marker registered in the same background. (a) Registration effects of the two markers under normal conditions. (b) Registration effects of the two markers occluded by fingers

Full size image

4 Rehabilitation training program

The proposed system can synthesize computer-generated virtual objects and a real environment, as well as enhance or extend a scene. In this way, patients can train their hand functions in the enhanced real environment and apply these functions to the real environment. The system consists of a computer, a webcam, and some objects with markers; it has the advantages of simple equipment, low cost, strong practicability, and ease of promotion. This paper presents four training programs as follows.

4.1 Trajectory training

Users should conduct hand rehabilitation movements, including hand pendency, lift, and pinch, alongside the given trajectory. Such movements are conducive to the movement scope of shoulders, elbows, and wrists.

The Artoolkit marker in the training module is used to control the position of the virtual trajectory. Users can select a fixed position and remove the marker once the trajectory is adjusted, but the webcam position should remain fixed during the training process. The CH-marker needs to be placed on the object handled by the users.

Two trajectories exist in the foregoing trajectory, namely, a straight-line mode and a curve mode. Each mode on the screen includes a virtual straight-line trajectory or a virtual curve trajectory, a virtual circular ring as end position, and a virtual ball as moving object. When the virtual ball touches the trajectory, the color of the ball turns red, which shows that the ball has been on the trajectory. When the ball reaches the end position, the ring also turns red. The bottom of the screen shows the ready time, and a timer is used for data acquisition, as shown in Fig. 4. Trajectory length, width, height, direction, and other parameters can be set according to the requirement.

Fig. 4
figure 4

Trajectory Training. This training is divided into (a) straight -line and (b) curve trajectory training. Each training exercises hand pendency, lift, and pinch movements. Ready(s) represents the 10 -s countdown time, and Time(s) represents the time from the beginning to the end of the training

Full size image

When a user starts training, if all virtual objects are recognized, then the system performs a 10 s countdown. During that time, the user should move the CH-marker to the initial position of the trajectory, and the ball starts to turn red.

After 10 s, the training begins. The user should grasp the real objects with CH-markers, control the virtual ball on the screen, and move the ball along the virtual trajectory. If the ball stays on the designated trajectory, it will keep red.

A circular ring is on the terminal of this trajectory. The ball will move along the trajectory until it runs through the circular ring. If the circular ring turns red, both the training process and the system timing system will be stopped. The trajectory completed by the user is also recorded.

4.2 Shelf training

Compared with trajectory training, shelf training is more difficult. Users should place a mug in a special position, as instructed. This training can practice the functions of hand stretching, grasp, lift, and muscle release.

Shelf training consists of a virtual three-layer shelf and a virtual mug, as shown in Fig. 5. The Artoolkit marker is used to determine the position of the virtual shelf in the screen. Once the best position is determined, it can be fixed, and the marker can be removed. The CH-marker is used to control the moving of the virtual mug. The number of layers, height, width, and other parameters of the shelf in the scene can also be set. The material object in the training can be a wood particle or any other real object with markers.

Fig. 5
figure 5

Shelf Training. The trainer needs to move a virtual mug according to the direction of the red arrow. The system gives the correct prompt when the trainer is moving in the wrong direction. The training also has the ready time and timer

Full size image

When a user starts shelf training, the system performs a 10 s countdown. After 10 s, the user should grasp the wood particle and move the virtual mug to the circular ring that is randomly designated on the shelf. If the virtual mug bumps against the specified circular ring, the ring will turn green. The system then gives the corresponding instructions on the upper left of the screen to guide the user in placing the mug in the ring. Once the mug is accurately placed in the circular ring, the ring will turn red, and the system will give a cheer. After moving into the designated circular ring, the virtual arrow immediately points at the subsequent circular ring, and the training is repeated. The system will record the number of the successful placement of the mug during a specified time (e.g., 1 min).

4.3 Batting training

User grasps an object attached marker, with each hand in the batting training. This rehabilitation training can train hand -bending capacity and movement flexibility. Compared with the two training programs described above, this program needs to be accomplished using two hands, which is conducive to the promotion of synchrony, bending capacity, movement flexibility of two hands, and the synchrony of hands and eyes.

A virtual table is registered on the Artoolkit marker. During the training, a small virtual ball appears on the table. We also register two small cubes on two CH-markers. To distinguish these cubes, we set up one for the blue and the other for the gray. The users need to grasp an object with a marker attached in each hand, which is represented by two cubes on the screen (see Fig. 6a and b).

Fig. 6
figure 6

Batting Training. Two CH-markers are used to register the hit box, and the Artoolkit marker is used to build the desktop scenes in (a) and (b). In (c), when the training starts, a small ball appears on the table. The user hits the ball with both hands. If the left hand hits the ball, then LeftHit will be added; if the right hand hits the ball, then RightHit will be added

Full size image

The ball appears in the table center at the beginning of the training. The system presents 10 s countdown. After 10 s, the ball moves at certain speed and direction. The velocity reversal and size of the ball are assigned at the initialization (see Fig. 6c). When a user moves the cubes to hit the ball, the ball will rebound. If the ball bumps against the four sides of the table, it will bounce back.

The training cycle can be set according to the degree of injury and recovery of the hand. In the training process, the system records the number of ball hits by these two blocks and the training time and then sends feedback to the user.

4.4 Spile training

The spile training in this system simulates a nine-hole peg test [9, 17]. This training can improve the synchrony of upper limbs, finger grip, the ability to oppose the limbs, and the accuracy and speed of upper limb movement.

To make the sense of touch real in the training, we employ a crabstick (100 mm in height, 12 mm in diameter) and attach the marker on the top of the cylindrical stick to track and locate the drawn virtual stick.

In the training, a marker is first used to locate the virtual board (Fig. 7), the size of which is 160 mm × 160 mm × 20 mm. The crabsticks, which are placed near the hand of the subject, are put in the container. The subject is then asked to take a stick out of the container at each time and insert it into the hole in the virtual board shown on the display. When the stick is inserted correctly in place, a sound prompt will be heard. Every time after the nine crabsticks are inserted, one of them is immediately pulled out and placed in the container. The operation time is counted from the beginning of the operation command to the moment the last stick is placed in the container.

Fig. 7
figure 7

Spile Training. The trainer holds a virtual crabstick and places it on the designated position of the board according to the red arrow. This training mainly exercises the strength and flexibility of hand grip

Full size image

5 Usability study

This section presents the result of a usability study we conducted on our system. The objective of this study was to verify that our system is a motivating tool for rehabilitation.

5.1 Experimental setup

Twenty subjects (10 males and 10 females; 22–36 years old, average age of 28) participated in this experiment. All subjects have self-reported a normal or corrected-to-normal vision and a normal sense of touch. They were new to the designed system and AR prior to the test. The subjects did not take part in any medical examination or review before participating in the test. Every subject needed to carry out four training programs in turn, and each program was requested to be trained 5 times. Subjects could take a 10-min break between each training program. In addition to the batting training with both hands, other training programs were done with the right hand. Every subject was requested to fill out a questionnaire score designed for this purpose once he/she completed the test. We describe the scoring rules and some scoring items in Appendix A of this paper [5]. To perform a test, a subject was asked to sit 10–15 cm from the edge of a regular computer desk facing an LCD screen. The experiments were taken on a PC running Windows 7 with an Intel(R) Core(TM) i7-2600 CPU running at 3.4 GHz (Fig. 8). An ANC HD720P USB webcam was used.

Fig. 8
figure 8

Training scene

Full size image

5.2 Results

Figures 9, 10, 11 and 12 present the questionnaire results after the trajectory training, shelf training, batting training, and spile training, respectively.

Fig. 9
figure 9

Data record interface of the trajectory training

Full size image
Fig. 10
figure 10

Data record interface of the shelf training

Full size image
Fig. 11
figure 11

Data record interface of the batting training

Full size image
Fig. 12
figure 12

Data record interface of the spile training

Full size image

The results of one of the subjects in the trajectory training are shown in Fig. 9. He performed 5 times of straight and 5 times of curve trajectory training. In the figure below, the average time and standard deviation of the 5 times of straight and 5 times of curve trajectory training were respectively calculated. If one of the records was selected, then the actual movement track could be seen.

In the shelf training, the number of the mugs successfully placed by each subject within 1 min was recorded. In Fig. 10, the performance by one subject who finished 5 times was recorded, and the mean and standard deviation were calculated.

The system interface of batting training is shown in Fig. 11. The left part is the personal information of the user and the time records of the batting training. The related data and charts for this training are shown on the right part.

If the user selects a record of the checkbox on the left, the right will display the number of ball hits by two hands at this training time and the related histogram. If the user selects multiple records on the left, the right side will calculate the average value of this multiple training and the standard deviation. We can understand the hand rehabilitation of the user from these two values and then adjust the training cycle to achieve high-efficiency training.

In batting training, the number of both hands hitting the ball was recorded in 1 min, as shown in the following figure.

In the spile training, the completion times for every subject were recorded, as shown in Fig. 12.

The results of the questionnaire score for the four training programs are shown in Figures 13, 14, 15 and 16.

Fig. 13
figure 13

Questionnaire results of trajectory training

Full size image
Fig. 14
figure 14

Questionnaire results of shelf training

Full size image
Fig. 15
figure 15

Questionnaire results of batting training

Full size image
Fig. 16
figure 16

Questionnaire results of spile training

Full size image

5.3 Discussion

The evaluation parameter commonly used by therapists is the task-completion time, which was considered in the four designed training programs. In the trajectory training, the time that the trainees took to complete the straight and curve tracks was recorded, and the mean and standard deviation of multiple training could be selected and calculated. This process was conducive to conforming the difference in the moving ability of the trainees among different tracks. The shelf training and batting training mainly tested the performance of the trainees in completing a task within a given time. In the spile training, the completion time of each subject was recorded. The completion time of the subjects was found to be longer than that taken by using a real nine -hole peg test. This result was due to the fact that in an AR environment, a subject needs to hold the crabstick in hand while observing the position of the holes on the board shown on the display, which requires extensive hand–eye coordination.

For the scores of the subjects in the four training programs, the average scores for Easy to understand and Portable of the four training programs were 10 points because all subjects believed that these training programs were movements easy to understand. As a result, they showed an active attitude in the trainings and tried their best to complete them; compared with wearable training equipment, the instrument used in this training was convenient to carry. Among the four training programs, the average scores for AR environment and Interaction ranged from 8 to 9 points because most of the subjects recognized the layout of the AR scene, which was suitable for the short movement of hand; the training was designed for convenient interaction and strong sense of immersion. The scores for Enjoyable and Motivating were relatively high because all subjects felt comfortable and safe during the trainings. These training programs could ignite their enthusiasm toward their treatment and efforts to achieve better results. Moreover, we could add some pleasant sounds or vivid animations in the trainings as feedback to the user to increase their interest in the trainings. Clear instructions were an important indicator evaluating the trainings. Among the four training programs, the average score of the shelf training was the highest because the users could place the mug to the specified position according to the system instruction. By contrast, the instruction in the batting training was relatively simple, i.e., to bat the ball by moving two blocks in a plane, and not much instruction was needed. The average scores for the trajectory training and the spile training were the lowest because these two took longer time of training, and the users wanted to acquire a more precise instruction. For instance, in the trajectory training, the correct direction should be hinted after the path was deviated; in the spile training, the correct direction should be hinted when the stick approached the hole. Before the training, some audio instruction or demo video about the training purpose could be played. According to the scores for Operability, the scores for the shelf training were lower, and two subjects felt fatigue in their hands and arms during the training because this task required to lift the virtual mugs for a long time. In the spile training, two of the subjects believed that it was difficult to operate because the previously placed sticks could be knocked down when inserting a new one. They hoped to increase the intervals among the sockets.

In general, all subjects unanimously recognized and expressed keen interest to the four AR-based hand rehabilitation trainings, looking forward to more training programs designed for the patients.

Hand function recovery is generally divided into three stages, the first is the restoration of the amplitude of the joint, and then the recovery of force, the third is the tactile recovery[26]. In the course of our training test, each participant is healthy, always using the same small piece of wood. A corresponding improvement according to the situation should be performed for patients with different stages of hand dysfunction. Hand joint amplitude is relatively small in patients with hand weakness, corresponding hand function recovery should be started by non-impedance active joint amplitude training. It is hence necessary to select some lighter objects as the carrier, choosing the trajectory training as the first stage of the hand rehabilitation training to practice simple lifting and hanging movements of the hand, etc.. The second stage is the restoration of force, when patients recover joint amplitude, there is a need for hand griping training. Relied on shelf training, the hand ability of the patient can be trained by holding labeled objects from light to heavy lifting to improve hand griping ability. Before entering the third stage, spile training and batting training can be practiced focused on the training of hand griping, pinching, griping, and so on, to enhance the flexibility of the hand. Each patient may have different request for the scene, patients’ hand lifting, movement and hanging can be fully trained by setting the camera to adjust the size of the scene, and even in a room with a large scene.

6 Conclusions and future work

This study combines the methods and design principles of occupational therapy in hand impairment rehabilitation, as well as follows the suggestions and instructions of doctors. Four training programs for hand rehabilitation are designed, namely, trajectory training, shelf training, batting training, and spile training. Movement trajectory may be developed from a regular and linear trajectory into an irregular and nonlinear trajectory, which can train hand functions and the coordination ability of hands and eyes.

To achieve effective application in an AR scene, this study designs a color marker that is suitable for hand rehabilitation training. Following the Hamming coding principle, this color marker restores the occluded binary bits and solves the problem of tracking registration failure when part markers are occluded in the dynamic scenes.

Finally, the effectiveness of the developed system is evaluated via experiments, with all indicators evaluated by questionnaires and analysis. The analysis result shows that the rehabilitation programs, which comprise obvious advantages in comparison with traditional rehabilitation methods, can be applied to hand rehabilitation training in daily life.

Applying the AR technology to rehabilitation training remains a new research field, such that this study only introduces some foundational works. The present system can be improved through the following steps:

  1. A.

    Practicable training programs in the existing hand rehabilitation system may not be sufficient. In the future, the virtual scenes of the system will be adjusted to be more comprehensive by including wrist, metacarpus, and phalanx movements and an overall coordination training of hands.

  2. B.

    The system compatibility will be improved to achieve good interaction with therapists when it comes to exercise training, psychological therapy, function evaluation, and plan formulation.

  3. C.

    The natural interaction of unlabeled human hands in real time will be more effectively utilized in the direct 3D interactions between the natural behavior made by human hands and virtual information.

  4. D.

    Considerable information will be incorporated into the system. A comprehensive natural AR environment will be established in combination with perceptual knowledge and conceptual knowledge. The channel of human –computer interaction will be expanded, and the rehabilitation effect will be improved.