Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Robot-assisted rehabilitation therapy is an emerging form of rehabilitation treatment for motor recovery after neurologic injuries such as stroke and spinal cord injury. Robotic devices can help patients achieve the intensive, repetitive practice needed to stimulate neural recovery, reducing the need for supervision and improving cost-benefit profiles [1]. They can also provide quantitative parameters to characterize the therapy. Initial clinical studies have found that training with robotic therapy devices typically matches or exceeds the therapeutic benefits possible with conventional therapy approaches [2–4].

With these promising results, there has been a proliferation of device designs. For example, a recent review of robotic therapy devices for the upper extremity found over 150 devices in the literature [5]. However, even if numerous robot-assisted devices have been proposed, the rationale for the design of each device is still largely improvised, because there is limited scientific knowledge of why robotic therapy is effective.

The objective of any neurorehabilitation system is to exploit neuroplasticity and motor learning, involving the patient in an intervention that recreates the favorable conditions that can induce the modification of the residual neural networks of the brain. However, despite progress, a full understanding of the neuroplasticity and motor learning mechanisms that are involved in beneficially modifying brain networks is not currently available. The objective of this chapter is to review current state of knowledge of the mechanisms that might cause robotic therapy to improve motor recovery. To organize this chapter, we identify three categories of mechanisms:

  • Parameters of the therapeutic experience: therapy dosage, task specificity, and challenge level.

  • Types of motor learning: learning from augmented feedback, Hebbian learning, and reinforcement learning

  • Approaches to human-machine interaction: haptic guidance, error augmentation, and machine-enhanced motivation

We discuss experimental evidence for each mechanism. We also interpret this evidence with respect to its implications for current and future robotic therapy device design. We primarily focus the discussion on stroke rehabilitation, which is the largest potential user group of robotic therapy. Many of the results likely can be extended to other patient populations, including people with spinal cord injury and cerebral palsy.

2 Quality of the Therapeutic Experience and Robotic Therapy Design

2.1 Dosage

2.1.1 Evidence for a Dose-Response Effect

Perhaps the primary rationale for using robots in rehabilitation therapy has been to increase the dosage of therapy that can be delivered. Dosage in rehabilitation therapy can be defined in different ways, including the amount of therapy in minutes per day, the number of exercise repetitions achieved during therapy, or the number of therapeutic sessions in the rehabilitation process. The term “dosage” can also include the concept of therapy “intensity”, such as the amount of external work and/or power the patient produces during training [6].

What is the evidence that a dose-response mechanism exists in rehabilitation movement therapy? At least in the case of stroke, evidence is relatively strong that there is an overall effect of dose, as summarized in a recent review [7]. Improved dose can improve activities of daily living (ADL) performance [8–12], strength [9, 13, 14], and shorten the length of stay in a rehabilitation center [15]. However, there are some questions about the persistence of the dose-response effect, as well as its specificity.

For example, in one key study, an increase in therapy dose was achieved by giving longer therapeutic sessions (usually 1.5-2 times than what the control group is receiving) [8]. This increased dose improved the ADL scores in the beginning of the therapy (at 4 months), but this effect was transient, since, after 12 months of therapy the increased dose group had similar ADL scores as the control group. A similar trend was observed in other studies [9–12]. This suggests that an increase in rehabilitation dose in the first six months may not be mandatory for long-term rehabilitation outcomes but allows the patient to regain earlier a better performance in activities of daily living, thus justifying the use of robotics to increase dose early in therapy.

Other studies have suggested that increasing dose affects different activities of daily living differently. For example, the correlation between repetition and improvement has been suggested to be stronger for occupational therapy than for physical therapy activities [12], or for discipline-specific therapy than for combined therapy across disciplines [16]. In another study, increasing dosage improved stair climbing and 6-minutes of walking in a sustained way, but not timed up-and-go test [13].

2.1.2 Implication of Dose for the Design of Rehabilitation Robotics

The dose-response mechanism is likely a key reason that robotic rehabilitation therapy has been successful. For example, a recent review stated that, when robotic therapy and conventional therapy were applied with the same duration and intensity, there were similar improvements in outcome [4]. This has also been confirmed in other studies, where the patients with the highest robotic therapy dose (here expressed as hours of therapy per week) had the best improvement in motor function [17, 18].

Therefore, a key consideration in the design of robotic therapy devices is how to make therapy (1) less demanding for the rehabilitation therapist, and/or (2) possible for patients to complete without continuous one-on-one supervision from the therapist. Thus, ideally, devices must be designed to allow the patient to exercise as independently as possible, so that rehabilitation dosage can be increased. This has strong implications that robotic therapy devices must be designed for ease of use, and must include engaging rehabilitation games or other motivation strategies that facilitate extended, semi-autonomous practice. Designing devices that can provide gradable amounts of work or power, and adjustable number of repetitions per second, also appears to be important due to the dose-response recovery mechanism.

2.2 Task Specificity

2.2.1 Evidence for Importance of Task Specificity

The development of new robotic therapy devices in the last ten years has been driven by the finding in rehabilitation research that task-related, functional training leads to better outcomes than non-functional training [19, 20]. Indeed, rehabilitation after stroke has evolved during the last 20 years from mostly analytical rehabilitation methods to task-oriented training approaches [21]. By “analytical methods” we mean methods that address single-joint movements that are not directly linked to skills. These are usually movements without a goal and in one plane. In contrast, task-oriented approaches involve training of skills and activities aimed at increasing the subject’s participation. Task-oriented training consists of either multi-joint simple movements not directly related to activities of daily life (e.g. moving blocks from one location to the other) or movements with a clear functional goal (e.g. washing dishes or dressing) [21].

A key rationale for task-specific training is that transfer of motor learning has often been found to be limited [22]. Therefore, practicing the tasks that one wishes to be able to participate in during daily life ensures the greatest learning on those tasks. With regards to upper-extremity training, it has also been shown that practicing tasks that are meaningful for the person increases cortical reorganization. Moreover, the effects of task-related training were found to be long-lasting compared to the effects of traditional therapies [23].

Regarding lower-extremity training, there has also been a strong preference in rehabilitation practice for task-specific training, particularly training focused on walking. Thus, both manual body-weight support treadmill therapy and robot-aided treadmill training have received great attention in the rehabilitation world. One of the main reasons is that task-specific training provides locomotion-relevant afferent input to spinal central circuitries that generate rhythmic stepping behavior [24].

2.2.2 Implication of Task-Specificity for the Design of Rehabilitation Robotics

Because of the importance of functional training, robotic therapy devices have become increasingly complex, with the inclusion of more degrees of freedom (DOF) [5]. With these devices, task-oriented exercises that resemble activities of daily life are possible and they are usually combined with virtual-reality software that mimics activities of daily living [25, 26]. This increasing complexity has typically meant a trend to build exoskeletal devices, rather than end-effector based devices, to more closely mimic the structure of the limbs, for safety and comfort during the complex movements required for ADLs (e.g. ARMin [27], BONES [28]). Recently, hand modules have also been increasingly integrated into exoskeletal devices, allowing the integrated use of the arm and hand [5]. Among the lower-extremity robots, we can find that the robotic gait trainers that allow functional training of walking are usually the exoskeletal devices that have actuated hip and knee joints (Lokomat [29], LOPES [30], ALEX [31]), which are increasingly incorporating pelvic movement for balance training (PAM [32], LokomatPro FreeD). There has also been rapid development of legged exoskeletons that allow practice of overground walking, with several commercial products now available (Ekso, HAL [33], ReWalk [34], Indego [35]).

It is unclear however, at least for the upper extremity, if the increased complexity has been necessary. For example, a recent study with an upper-limb robotic device (BONES) showed that multi-joint, task-related training was not superior to single-joint training in a group of 20 chronic stroke patients. It seems that better learning of the movement occurred when the task was decomposed in simpler parts as opposed to practicing only the whole [25]. Likewise, a recent clinical study with a more complex exoskeletons seemingly did not produce better results than previous studies with less complex robotic devices [26]. Indeed, it has been hypothesized previously that more severely impaired patients would benefit more from impairment-based training than from functional training [36]. It may be the case that, until patients have developed the whole repertoire of movements required to complete a task, they might not fully benefit from functionally-based robotic rehabilitation approaches.

For the lower extremity, while functional robotic gait therapy is effective in stroke patients [37], there is also evidence that home-exercise programs, with the aim of enhancing flexibility, strength, coordination or balance, are equivalent to locomotor training (LEAPS trial [38]). In the case of walking, it has been clearly shown that the locomotor capacity correlates well with strength of leg muscles, like hip flexors or extensors [39, 40]. In this case, therefore, training aimed at improving the strength of some target muscles of the lower limbs could be more beneficial than walking with a robotic gait trainer alone [41]. Another interesting example is the single-joint training provided by an ankle robot that trained dorsi/plantarflexion and inversion/eversion movements, where patients improved velocity and distance walked [42]. No comparison with functional training was made in this study.

In summary, it is currently difficult to draw a definitive conclusion on how mechanically complex it is to make robotic therapy devices for the purposes of retraining function. At present, then, robotic-therapy designers must rely on clinical wisdom. One clinical framework that may be useful is the concept that sensory-motor training should present a total package, consisting of several stages [43]:

  • training of basic physical capabilities that are prerequisites for skilled movement (e.g. muscle force, range of motion, tonus, coordination)

  • skill training (cognitive, associative and autonomous phase)

  • improvement of endurance on muscular and/or cardiovascular level.

In this framework, task-related training and analytical training are complementary, and different robots can be designed to account for the different needs of the patients: from very complex multi-DOF exoskeletons to more simple, but as important, single-joint devices.

2.3 Optimal Level of Challenge

2.3.1 Evidence for the Importance of Optimal Challenge

A key aspect of the recovery of motor function is to challenge patients during training according to their skill level [44]. Too low a level of challenge can make a treatment boring, and not encourage motor learning. On the other hand, too high a level of challenge can be frustrating or overwhelming, and also make it difficult for a patient to garner new information needed to learn. In motor-learning research, these ideas have been captured in the Challenge Point Theory, which posits an optimal challenge level for learning based on the skill level of the trainee [45].

The idea of optimal challenge can also be related to studies of the role of guidance in motor learning. In one seminal study, always guiding a person during movements reduced motor learning in a task where participants had to learn to position the elbow at a desired location without vision [46]. In contrast, faded guidance, which is a time-scheduled reduction of guidance, was found to be approximately as effective as no assistance, and significantly better than a fixed amount of assistance. Reducing guidance therefore seems to promote optimal levels of challenge.

Regarding neurorehabilitation, the upper-limb function of stroke subjects has been reported to improve after robot-mediated treatments that require the patient’s effort during the entire movement and assist only to complete the task [47]. Patients seem to benefit from a progressive reduction of the assistance, although definitive conclusions cannot be drawn yet [47]. Results in robotic gait training also seem to suggest that devices that “over-assist” patients produce lower therapeutic benefits [48–50].

2.3.2 Implication of Challenge for the Design of Rehabilitation Robotics

A key advantage of robotic therapy devices is that they can provide varying degrees of assistance, balancing the difficulty of training, theoretically maximizing the rehabilitation outcome because of the challenge-dependence of motor learning. However, how best to manage the trade-off between training variables, and how to determine the level of optimal assistance that the robot must provide to help the patient without replacing his volitional movement, are still open issues [51]. What is clear is that robotics technology allows for a range of approaches, from a passive robot that follows the patient [52], to an active device that carries the patient and assumes the control [47, 53].

One strategy for providing challenge is the assistance-as-needed paradigm, which lets the patient execute the movement and tracks the performance error to provide support only when required. This is similar to faded guidance in motor-learning research, except the guidance is adjusted based on real-time measurement of performance, rather than based on a fixed-time schedule of reduction. In this paradigm, the participant’s effort is encouraged, and only self-initiated movements can be performed. This can be done, for example, by allowing some error variability around the desired movement using a deadband (an area around the trajectory in which no assistance is provided), and triggering the assistance only when the participant achieves a force or velocity threshold [53]. As another example, Emken et al. developed in 2007 a mathematical algorithm for fading robotic guidance based on a measure of ongoing error [54]. This “guidance-as-needed” robotic assistance helped people learn to form an internal model of a novel force field that was applied to the leg while walking. This strategy reduced performance errors while it allowed participants to progressively experience more of the actual task to be learned. Nevertheless, the main challenge that the assistance-as-needed controllers still need to face is the optimal choice of the measured process variable, as well as the algorithm for adapting the system parameters to align to the requirements of the patient.

Several robotic neurorehabilitation systems have been developed based on the assistance-as-needed paradigm. The first robotic system to be clinically tested was the MIT-Manus, which allows a free movement of the arm in the horizontal plane with low friction. The impedance selection allowed the treatment to change according to the performance of the patient and it has provided positive results on its repeated tests with stroke patients [55–57]. Wolbrecht et al. developed in 2008 a controller to learn the patient’s abilities and complement them with the robot, by reducing the force exerted on the upper limb when the errors in the task execution were small [58]. GENTLE/A, a robot designed to rehabilitate the upper-limb function in point-to-point and single-axis movements, also estimates the contribution of the participant during the treatment and adapts its assistance/resistance accordingly, automatically tuning the difficulty of the task to challenge the patient [59].

In summary, the trend of the robotic controllers in recent years is to continue to improve the human-robot interaction, adapting the robot behavior and cooperation in an attempt to make the communication more natural and the challenge level optimal [44]. For example, the robot can even be made to anticipate to the user’s actions, to provide more effective assistance-as-needed [60].

3 Learning Mechanisms and Robotic Therapy Design

3.1 Learning from Augmented Feedback

3.1.1 Evidence for the Importance of Augmented Feedback

Augmented feedback refers to providing artificial feedback of movement parameters [61], given in addition to intrinsic feedback, defined as the natural information from internal sensory processes like vision, proprioception and hearing [62]. Training with augmented feedback in rehabilitation is generally recognized to be more effective than training without [42, 62, 63], but the neurological mechanisms underlying its effects still remain to be clarified. It could be that either new pathways are developed, or old persisting cerebral and spinal pathways are mobilized by introducing the auxiliary feedback loop [64]. In this model, visual and auditory feedback activate unused or underused synapses in executing motor commands. As such, continued training could establish new sensory motor memories that help patients perform tasks without feedback [65]. Biofeedback may also enhance neural plasticity by engaging auxiliary sensory inputs [63]. This could be the case for patients with injuries of the central or peripheral nervous system where perception is often disturbed or missing due to lack of appropriate afferent input from the receptors. In this case, artificial sensors can be used for recording the signals to be fed back to the subject [66]. Moreover, it is known that an effective rehabilitation training should be intensive, repetitive, task-oriented and of long duration [67] and feedback can potentially enhance these aspects by increasing the level of attention, reducing the mental fatigue of executing the task [42, 63, 68, 69].

Augmented feedback can be conveyed in two paradigms: knowledge of performance feedback delivers information about the whole performed action, whereas knowledge of results feedback informs only about the final outcome [62]. Augmented feedback can be further classified according to the display modality: visual, auditory, haptic or a combination of these [42, 63, 67, 69, 70]. Visual feedback is the most used display modality [71, 72] and it can span from a very simple display using lines or colors to convey information [61], to more complex representations, such as an avatar displayed on a screen [66, 67] in a Virtual-Reality (VR) environment. Auditory feedback can also be played back in response to an action or an internal state of a robotic therapy system [73]. Auditory tones can be used with a rewarding function (“positive tone”) or continuously to map a particular characteristic of a movement (e.g. smoothness or distance to the target in a reaching task) [63]. Sounds can be also used to augment realism in a VR environment [73]. Haptic information feeds back kinaesthetic sensations that are important for task performance and it highly enhances the immersiveness of VR environments [63]. A trend in rehabilitation robotics is to combine the three modalities in VR applications that provide a more immersive feedback for task-related training [63, 74].

The optimal feedback modality is thought to be dependent on the information to convey and on the characteristics of the population involved in the training. Visual feedback seems to be more effective for information related to spatial aspects of the task, while haptic cues are thought more suitable for conveying timing information [75, 76]. Auditory feedback can be employed to emphasize small kinematic errors, which are not visible due to limited resolution of video feedback. Also, sound is very suitable to display velocity-related information [73]. It has been hypothesized that the optimal feedback is different for upper limb (visual) and lower limb (haptic) motor tasks [66]. Variables such as the site or size of the brain lesion, the patient’s motivation during therapy, and his/her cognitive ability may also influence the effectiveness of biofeedback or any therapy [63]. It is important that the feedback is neither overloading nor distracting, since distraction reduces effort [71]. Moreover, it is important to consider the transfer from exercise to real life, and not only the short-term effects of feedback training. Patients are capable of learning motor tasks in a virtual environment and the acquired skills can be transferred to real life [42, 70], but it may be important to fade the feedback or to provide it more intermittently to prevent the subjects to rely on it [71]. Healthy subjects performing motor-learning tasks showed improved retention if the feedback was given at the end of the task or if no-feedback trials were included during the learning phase [72].

3.1.2 Implications of Biofeedback for Rehabilitation Robotic Therapy Design

Robotic therapy devices are particularly suitable to deliver augmented feedback since they are equipped with sensors, as well as visual, audio, and haptic display capabilities. Further, augmented feedback is highly recommended in robotic therapy because, unlike in traditional therapy, the psychological, relational, verbal, and touch contact between therapist and patient is missing. This lack creates the need for additional channels to provide feedback on the performance and to improve the patient’s motivation. However, although many interesting studies have paved the way for a more systematic use of biofeedback in robotic rehabilitation as summarized above, evidence on the best paradigm remains to be established.

Feedback training for the upper limbs has in general an added value to conventional therapy in stroke rehabilitation but its optimal characteristics have not been determined yet [62, 77]. VR exercises with a robotic device could engage the participants for a longer period leading to potentially better therapeutic outcomes [78]. Several studies showed the benefits of robotic devices for upper-limb training that make use of VR and feedback [26] but further studies are needed to prove the benefits attributed to augmented feedback itself.

In lower-limb training, augmented feedback was found superior both to conventional therapy and to therapist’s feedback, and these benefits were maintained also long term [79]. Visual feedback enhanced active participation in robotic therapy [42, 69, 80]. Positive effects on gait parameters were found after feedback training [42, 81] but the lack of systematic studies prevents drawing definitive conclusions.

In summary, then, augmented feedback during robotic therapy appears at minimum to increase motivation, thus leading potentially to better therapeutic outcomes. Nevertheless, evidence on short- and long-term effects of training with augmented feedback and on the optimal feedback modality for different patient populations is incomplete.

3.2 Hebbian Learning

3.2.1 Evidence for the Importance of Hebbian Learning

As stated in the Introduction, the objective of any neurorehabilitation system is to exploit neuroplasticity and motor learning, involving the patient in an intervention that recreates the favorable conditions that can induce the modification of the residual neural networks of the brain. Donald Hebb introduced in 1949 a fundamental concept of how residual networks are modified, which is that cells that fire together, wire together. This statement summarizes the increase of synaptic connection between neurons that is produced by their simultaneous activation [82]. Further research has shown that this important mechanism of neuroplasticity involves not only synaptic potentiation, but also structural changes, like axon sprouting and the formation and stabilization of new dendritic spines [83].

Recent work has shown that Hebb’s rule can be used to artificially induce neuroplasticity, modifying a neural network by imposing coactivation firing patterns on target neurons that an experimenter wishes to wire together [83, 84]. It has been suggested that this paradigm could be used as a treatment to shift the function of a destroyed area of the brain to another area that can adapt to perform the new function [85]. A similar idea can be applied to the most common robotic therapy paradigm, which is to have the robotic device assist patients in completing target movements. In this paradigm, the patient attempts to move, causing efferent activity. Then, the robot assists, causing time-correlated afferent activity. The convergence of the afferent activity with the efferent activity in residual sensory motor centers would then be expected to cause plasticity in those centers via Hebbian learning. Note that in this scenario, the robot actually enhances afferent activity, since the patient is weak and impaired and cannot move well without the robot. This robot-enhanced afferent activity may in turn enhance Hebbian learning. To our knowledge, however, this rationale has not yet been experimentally verified in robotic therapy.

Hebbian learning is also the underlying rationale of two widely-used neurorehabilitation paradigms: motor imagery and movement observation. These paradigms have direct relevance for robotic therapy, and so we summarize them briefly here.

Motor imagery is a key application of Hebb’s rule to neurorehabilitation, and consists of activating with imaginary movements the areas of the brain that are involved in movement preparation and execution. The effects of motor imagery have been widely stablished as a training method for athletes, and its impact on neurological patients has been studied for the past decades [86], including studies with stroke patients [87], incomplete spinal-cord patients [88], and children with cerebral palsy [89]. The studies that are centered in neurophysiology suggest that the brain responses to movement imagination and execution seem to have the same duration; however, the amplitude of the responses suggests that imaginary movements produce a weaker degree of activation of the central nervous system [90, 91]. Furthermore, not all subjects (particularly stroke subjects) are able to focus intensively and for long periods when imagining a movement; this inability has been termed chaotic motor imagery [90, 91]. Recent studies are also proposing ways of quantitatively assessing the degree of effective motor imagery performance, measured as a suppression of the sensorimotor rhythm [92].

Finally, the application of movement observation to robotics has primarily been in the use of mechanical guidance to teach patients how to perform a movement. We refer the reader to the section on haptic guidance for further discussion.

3.2.2 Implications of Hebbian Learning for Rehabilitation Robotic Therapy Design

If robotic assistance indeed stimulates afferent activity that in turn promotes Hebbian learning, this would provide a rational framework for the design of robotic therapy devices, since the devices could then be explicitly designed to promote activity parameters optimal for Hebbian learning. Thus, a major research direction for robotic therapy research should be to determine the role of Hebbian learning in the therapeutic effects seen with active assistive robotic therapy.

The design of robotic therapy devices could also potentially benefit from incorporating ideas from other rehabilitation techniques inspired by Hebbian learning. For example, the mental imagery of a motor task can be the input of a Brain-Computer Interface (BCI) that commands the robot, fully integrating the patient into his rehabilitation [87]. This goal-oriented setup guarantees that the patient focuses on producing the motor task, which presumably will increase neuroplasticity and enhance motor recovery [87]. Many studies have already shown the feasibility of BCI in the control of external devices [93]. BCI techniques have also been shown capable of classifying imaged grasping movements of the paralyzed hand of stroke subjects [94]. With respect to incorporation of BCI into robotic therapy, Ang et al. recently conducted a randomized controlled trial with 21 hemiplegic stroke patients that commanded hand opening and closing actions to a haptic knob robot for arm rehabilitation [95]. Before the study, the patients who underwent the robot-assisted rehabilitation were screened for their ability to operate the BCI, and after 6 weeks, they significantly improved motor recovery with respect to control subjects. Likewise, priming brain activity with a BCI-controlled robotic therapy device before rehabilitation therapy improved the patient outcomes, for individuals with severe impairment after stroke [96].

Another approach to exploit Hebbian learning in robotic therapy is through Neuromuscular Electrical Stimulation (NMES), which has already been shown to reinforce neuron synapses and enhance motor relearning when combined with simultaneous voluntary effort [97]. NMES and robots have been traditionally used separately for rehabilitation of neurological patients, but the disadvantages of each might be mitigated by combining them [98, 99]. For instance, the robot normally uses an externally applied torque to produce movement on the limb, whereas NMES activates the muscle to generate the force; on the other hand, NMES applications usually have difficulties at controlling the speed, trajectory and smoothness of the movement, which could be mitigated with a robotic device [98]. An EMG-driven robot system combined with NMES was recently proposed for wrist training after stroke [98], and was tested on five subjects. Results showed that the robot assistance improved movement accuracy, whereas NMES reduced the excessive muscular activations of the elbow joint.

3.3 Reinforcement Learning

3.3.1 Evidence for Importance of Reinforcement Learning

Reinforcement learning is a type of biological learning [100], which has also inspired extensive work in machine learning approaches [101]. The key idea is that the learning agent measures a parameter associated with reward that results from its actions, then changes its actions in order to find the optimal policy that maximizes the estimates of future cumulative rewards. A reinforcement-learning system must therefore solve a credit-assignment problem, which is to determine how to adjust many internal parameters (e.g. neural connection strengths in a biological system) based on a scalar measure of reward produced by its actions. It must also balance exploitation of what it already knows with the exploration of new actions that might improve the policy in the future. Reinforcement learning in biological movement control has been strongly tied to the dopaminergic system [100].

Reinforcement learning has recently been used in computational neuroscience as a way to model the mechanisms of rehabilitation therapy. Han et al. used reinforcement learning to model the learned non-use typical of stroke subjects, suggesting from this model that upper-limb rehabilitation must aim not only at improving motor control of the paretic limb, but also at reaching the point where the patient spontaneously uses the weakened limb on his daily life. Otherwise, it is just a matter of time until the paretic limb goes back to the initial deteriorated point [102]. Reinforcement learning was also recently used to model the recovery of movement strength in stroke patients, obtaining results that mimic the strength-recovery curve, residual capacity, and the influence of therapy timing and impairment level on the recovered strength [103].

3.3.2 Implications of Reinforcement Learning for Rehabilitation Robotic Therapy Design

Reinforcement learning is important for rehabilitation robotic design in two key ways. First, a full understanding of how the motor system uses reward-based teaching signals will open new ways to improve the design of robotic therapy after stroke. For example, the study by Han et al. suggests that robotic exoskeletons that are worn throughout the day might help a patient by encouraging spontaneous use of the weakened limb, resulting in a self-training, therapeutic effect. At the minimum, a recent study highlights the importance of considering reward in robotic therapy design. Ten chronic stroke patients underwent a clinical pilot study of nine sessions, where they trained ankle plantar- and dorsi-flexion with an impedance-controlled ankle robot [104]. The subjects were divided in two groups, receiving either high or low reward. The enhanced rewards were in the form of game scores, positive social interaction, and monetary rewards. The group with high reward had significantly faster learning curves, smoother movements, reduced contralesional-frontoparietal coherence, and reduced left-temporal spectral power, with respect to the low-reward group. Additionally, only the high-reward group increased the non-paretic step length.

In targeting reinforcement during robotic therapy, there are some factors to consider. Reward can be considered to be a scalar biofeedback that assists the patient in improving his performance, and increases motivation. However, use of a scalar reward might lead to negative compensatory movements that undermine motor relearning [105]. Indeed, as proposed by Kitago et al., a stroke patient who wants to reach an object will obtain the same reward whether he does it with his arm or by leaning with his trunk [105]. Therefore, the robot-enhanced reinforcement learning protocol must be carefully defined if compensation is not the goal. Another factor is that a scalar reward might discourage subjects when they cannot achieve the target on the first trials. To solve this problem, Sans-Muntadas et al. developed a system that measured the subjects’ execution with respect to their best performance, adapting the reward levels to the real abilities of the subject [106]. This algorithm was tested on 21 healthy subjects that simulated impairment, and was reported to provide a motivating workspace where virtually-impaired subjects could relearn how to move an impaired limb without feeling discouraged by the process. However, the a reward also reduced the subject’s willingness to explore other motor tasks, which would over time slow the learning process.

A second way that reinforcement learning is relevant to robotic therapy devices is that such devices can use reinforcement learning approaches for their control systems, to create adaptive controllers that change according to the users’ needs. The use of reinforcement learning is particularly useful when the policy must be learned from maximizing a simple reward signal, which in robotic therapy could be the amount of patient learning from trial to trial or a measure of effort, for example. A controller based on reinforcement-learning algorithms successfully controlled a robotic arm during a double-target reaching task with two monkeys using a body-machine interface [107], which has the potential to become an upper-limb rehabilitation treatment. The controller used binary feedback with information about the previous robot performance (good/bad) to quickly learn to control the robot, providing a stable control through several sessions and robustly adapting to perturbations of the neural inputs. Tamei et al. also used a reinforcement learning algorithm to guarantee stable control of a robot that based its decisions on EMG signals, modelling the scenario as a Markov decision process where the learning agent and the human shared the same goal [108].

4 Human-Machine Interaction and Robotic Therapy Design

4.1 Haptic Guidance

4.1.1 Evidence of the Importance of Haptic Guidance

In motor rehabilitation, the demonstration of the correct movement trajectory is often addressed by manually moving the patient’s limbs as the patient attempts to move. This “active-assist” technique is thought to support motor learning by means of demonstrating the task and providing a feeling of the correct movement [22], and perhaps by stimulating Hebbian learning (see section above). In human-robot interaction, this strategy is termed “haptic guidance”. In this context, a robot moves the user’s limbs through a correct kinematic pattern in order to reduce errors and, in some applications such as gait training or surgery, to enhance safety during practice. Furthermore, haptic guidance has the capacity to deliver more movement repetitions than conventional training protocols [109].

With healthy subjects, several studies have studied learning a novel arm movement with and without haptic guidance (e.g. [76, 110]). Haptic guidance increased especially timing accuracy of the learned movement. Recently, a study showed that haptic guidance in combination with interspersed free trials was able to shape the movement pattern of a novel complex sport-specific motor task, and these changes persisted after seven days without any further training [111]. Therefore, there is potential for the use of haptic guidance in teaching generic movement trajectories.

However, a number of studies have found that physically guiding a movement may actually decrease motor learning for some tasks. This phenomenon relates to the “guidance hypothesis” [53], which is that guiding a movement may reduce the burden on the learner’s motor system to discover the principles necessary to perform the task successfully. The dynamics of movement are also fundamentally different when a human or machine trainer guides limbs. Thus, training with haptic guidance is paradoxical: it may be helpful for reducing performance errors during training, but the experienced task is dynamically different from the actual one to be learned, and this may impair learning [112].

4.1.2 Implications of Haptic Guidance for Rehabilitation Robot Therapy Design

Haptic guidance in robotic therapy may be beneficial for the reasons outlined above – providing a feel for the movement, especially timing, increasing safety, and perhaps stimulating Hebbian learning. However, there is also some evidence in rehabilitation robotics that haptic guidance can be less effective than conventional training. For instance, patients with motor incomplete spinal cord injury who walked in a gait training robot that was controlled with an impedance-based assistive controller consumed 60 % less energy than in traditional manually-assisted therapy [48]. Likewise, stroke patients who were assisted by an adaptively-controlled, compliant robot that had the potential to “take over” a reaching task for them decreased their own force output, letting the robot do more of the work of lifting their arm [113].

In summary, the benefits and pitfalls of haptic guidance for rehabilitation robotic therapy are still under debate [53, 112, 114]. Most robotic therapy systems that have undergone clinical testing have used robotic guidance, and have shown benefits for improving motor recovery of the arm following acute and chronic stroke [2, 3]. However, it is not clear whether haptic guidance in rehabilitation is better than conventional rehabilitation treatments or just provides an alternative treatment possibility [72]. It will be desirable in this area to achieve consensus about appropriate outcome measures in order to quantify the motor re-learning benefits of haptic guidance [115].

4.2 Error Augmentation

4.2.1 Evidence for Importance of Error Augmentation

Another strategy used to enhance motor learning with robotic devices is error augmentation, which derives from the fact that many forms of learning are error-driven processes. By artificially increasing performance error in the course of learning, it has been hypothesized that the motor system could be driven in a way that makes it adapt more completely [114, 116]. This section briefly discusses error augmentation strategies, including resistive exercise, error amplification, and noise force disturbance.

Resistive exercise refers to the therapeutic strategy of providing resistance to the participant’s hemiparetic limb movements during exercise. There is a reasonable amount of evidence now from multiple non-robotic studies stating that resistive exercises that require higher effort from the impaired limb can indeed help stroke subjects improve motor function [53]. An alternative control strategy is to apply a performance-based resistance that amplifies error, based on subjects’ online performance.

A recent study assessed whether amplification of error or haptic guidance induced more motor learning, during a timing-based task with health subjects [117]. Both training conditions promoted learning. However, when dividing subjects based on their skill level, error-amplification training benefited learning more for the skilled subjects while it seemed that haptic-guidance training was more effective for the less skilled subjects. It appears that this result supports the challenge point theory, proposed by Guadagnoli et al. [45], which speculated that greater learning is achieved when an optimal challenge is provided to the individuals based on their skill level. The optimal level of challenge can determined from the ability of the performer, the complexity of the task, and the conditions of practice.

Kao et al. recently investigated whether performance-based robotic training using an error-augmentation algorithm better facilitated short-term changes of a typical gait pattern in healthy individuals compared to robotic training employing an error-reduction algorithm [118]. In the results, neurologically intact subjects were able to walk with stepping patterns closer to a prescribed template that required a higher than normal step height. Matching the target template was substantially better in persons receiving error-augmenting forces compared to error-reducing forces.

Another approach to error amplification is noise disturbance, i.e., randomly-varying feedforward forces that disturb subjects’ movements during training. A published study reported that training with noise disturbance resulted in better tracking than unassisted training and than training with a more conventional error-amplification strategy (repulsive forces proportional to tracking errors) [119]. In a more recent study, experiments under different training modes were performed, including exercises with haptic guidance, without guidance, with error amplification (repulsive forces proportional to errors), and in noise-force disturbance mode (with a randomly varying force disturbance added to the no haptic guidance mode) [120]. Moreover, adding random force disturbances during training appeared to increase attention, and therefore improve motor learning. Another recent study with robotic training of virtual golf putting found that error augmentation can decrease motivation for training, in a way that persists days after the experience of the error augmentation [121].

4.2.2 Implications of Error Augmentation for Rehabilitation Robot Therapy Design

Patton et al. explored the features of motor adaptation in chronic stroke survivors during the execution of planar multi-joint movements that are disturbed by a force field (forces as a function of hand position and/or hand velocity) [114]. They found that enhancing trajectory errors by the use of force fields induced better learning compared to reducing trajectory errors (haptic guidance) or providing no force field, in individuals with stroke. Using a similar paradigm, another study also suggested that a two-week training program of error enhancing trajectory seemed to provide the most benefit to the least impaired individuals, whereas active assistance during target reaching tended to be more helpful for the most impaired individuals [122]. Training with error augmentation was recently shown in a randomized controlled trial to produce better motor outcomes in chronic stroke patients than training without error augmentation [123].

In summary, error augmentation is a promising strategy for inducing a therapeutic response in robotic therapy. More research is needed to determine under what conditions error augmentation is appropriate, and for what kind of patients.

4.3 Motivation

4.3.1 Evidence for Importance of Motivation

Motivation can be defined as the “forces acting on or within a person to initiate a behavior” [124]. Patient “engagement” is a construct that is driven by a patient’s motivation; in motor rehabilitation it is the effort to regain movement capabilities executed through active, effortful participation during therapy [125] resulting in increased physical activation. Motivation can be regarded as intrinsic if it comes from doing an activity for its inherent satisfactions rather than for some separable consequence, or as extrinsic if, on the contrary, the activity is done in order to attain some separable outcome [126]. In motor rehabilitation, patients already have a personal, extrinsic motivation to regain their movement capabilities. This motivation could, however, further be increased by turning boring, repetitive training into enjoyable and entertaining therapy sessions [127].

Motivation is a multifaceted concept, which has been shown to be linked to features inherent to the prescribed regimen, personality traits of the patient, physician, and therapist, and characteristics of the broader social environment [128]. Important factors that have a role in improving motivation are: setting rehabilitation goals that are perceived as relevant by the patient, providing information about rehabilitation, and accessing and using the patient’s cultural norms [128].

Rehabilitation professionals have long suspected that a patient’s motivation plays an important role in determining the outcome of a therapy. Indeed, motivation is recognized to be one of the critical modulators of neuroplasticity, together with salience and attention [129]. In particular, dopamine production favors plasticity of the brain and it is enhanced by performing enjoyable training, such as game-like exercises [130]. Furthermore, a high degree of motivation leads to an active behavior during the training. Active training is more effective than passive training, leading to better motor outcomes and higher degrees of activation at the cortical level [131, 132]. Motivating exercises potentially also allow patients to perform longer training with more repetitions, therefore increasing therapy dosage. Thus, motivation, acts on three different levels: it can modulate neuroplasticity, it can elicit more intense motor effort (e.g. development of higher muscular forces during training) and it potentially results in longer training time.

4.3.2 Implications of Motivation in Rehabilitation Robotics Therapy Design

Enhancing motivation is particularly relevant in rehabilitation robotics, where there is a possibility that the patient might “slack” due to the assistance provided by the device [53]. In order to prevent this behavior different strategies have been proposed. One consists in the use of an assist-as-needed controller with a forgetting term that constantly tries to reduce the assistance provided by the device in order to challenge the patient [53]. In this way the patient is “forced” to be active because he cannot rely on the guidance of the device. Another essential approach to enhancing motivation is the use of game-like feedback and virtual reality to provide a more entertaining environment for the therapy.

Making sure robotic therapy devices are motivating is particularly important for children involved in robotic rehabilitation programs: diversification, fun and motivation are essential because children are generally not able to find enough “extrinsic” motivation in a boring and repetive training task [69, 133]. As also discussed in Sect. 3.1, studies that compared conditions with virtual-reality feedback or game-like exercises found better outcomes when the subjects were actively involved in the training and motivated by the additional feedback [42, 62, 78].

Interestingly, patients seem to enjoy particularly training with robotic devices and this high acceptance can enhance even more the efficacy of robot-based training strategies. In particular, it must be considered that robots allow patients with very severe impairments to perform movements that otherwise would be unattainable, leading to a strong positive feeling that could potentially affect the therapy outcomes. Finally, the development of sensitive and valid assessment tools for motor recovery that can be implemented in robots [134] are important to promote motivation in patients, so that they can be more aware of the effects of a therapy and increase their motivation.

5 Conclusions

Robot-assisted rehabilitation is a relatively new type of intervention for motor recovery, introducing benefits such as the possibility of the patient working semi-autonomously during training, and giving the patient and the therapist quantitative measurements of performance improvements. The first generation of robots has already provided hints of the potential benefits these devices can generate in the rehabilitation of movement after neurological impairment. Optimizing these benefits will require a thorough consideration of the mechanisms of motor recovery triggered by robotic therapy, as we have argued in this chapter.

We reviewed nine recovery mechanisms related to the parameters of the therapeutic experience (therapy dosage, task specificity, and challenge level); types of motor learning (learning from augmented feedback, Hebbian learning, and reinforcement learning); and approaches to human-machine interaction (haptic guidance, error augmentation, and machine-enhanced motivation). Based on this evidence, it seems clear that robotic-therapy devices will be most effective if they are designed to promote high levels of therapy dosage, optimal challenge, and high levels of motivation. For example, a positive loop can be established in which biofeedback from biomechanical data recorded through the robot sensors is used to enhance the motivation and involvement of the patient, therefore increasing the intensity of the training and the number of repetitions. Robot-aided rehabilitation has great potential for increasing patient’s motivation: virtual-reality games can be easily implemented in a robotic setting, sensors can be used to assess the patient’s performance, and the games can feed back the information to them. At the same time, robots can guarantee an intensive training since they are able to take over the physical demand required to the therapists in conventional physical rehabilitation. The level of challenge can be automatically adapted by the control algorithm of the robotic device to constantly match the patient’s status.

On the other hand, the role of task specificity in robotic therapy training is less clear. Recent studies have challenged the hypothesis that task-related training is undoubtedly superior to other kinds of training [25, 41] and further research will lead to the formation and consolidation of a stronger rationale behind the concept and design of future rehabilitation robots. Either way, in rehabilitation robotics there is the possibility to combine single and multi-joint or task-related training.

With respect to learning mechanisms, reinforcement-learning techniques are a promising direction for further research, either by providing faithful mathematical models that prescribe how to enhance motor recovery using new rehabilitation paradigms, or by customizing human-machine interfaces based on key outcome measures that can be sensed in real time. Understanding the role of Hebbian learning in robotic therapy is also a key direction for further investigation.

Finally, with respect to human-machine interaction, haptic guidance is a promising technique, but with some caveats, such as the potential to produce slacking by the patient. At the least, it can be used to enhance safety of tasks such as walking; it may also enhance motivation. Error-augmentation techniques are an exciting direction for future research, as they could be incorporated into a wide variety of robotic therapeutic exercises by exploiting the robot’s ability to sense a variable of interest and then to provide an action that augments the error under consideration.

Further research should provide a quantitative assessment of the relative importance of these recovery mechanisms in a short- and long-term time span and. an evaluation of the outcomes that might come from their combination. Such evidence will promote the optimal design for novel rehabilitation robots.

About the Authors

Noelia Chia Bejarano is a Bioengineering PhD student in the Neuroengineering and medical robotics Laboratory (NearLab), at Politecnico di Milano, Italy. Her research focuses on the development of a neurorehabilitation system for motor relearning during gait based on inertial sensors and functional electrical stimulation, designed for neurological patients and older adults. Contact: noelia.chia@polimi.it

Serena Maggioni studied Biomedical Engineering at Politecnico di Milano and she is now a PhD student at the Sensory-Motor Systems Lab at ETH Zürich and at Hocoma (Switzerland). Her research focuses on the development of a robot-aided gait assessment method that can be integrated in the robotic gait trainer Lokomat. Contact: serena.maggioni@hocoma.com

Laura De Rijcke studied Electro-mechanical Enigineering (Mechatronics) and is now a PhD student at the Robotics and Multibody Mechanics group of the Vrije Universiteit Brussel (VUB), Belgium. Her research focuses on wearable robotics, mainly assistive exoskeletons for workers. Contact: lderijck@vub.ac.be

Carlos A. Cifuentes is a postdoctoral fellow of the Robotics and Industrial Automation Group from Universidade Federal do Espirito Santo, Brazil. He is broadly interested in wearable sensors, human-robot interaction and rehabilitation robotics. Contact: carlos.garcia@ele.ufes.br

David J. Reinkensmeyer is a professor at the University of California at Irvine. His research focuses on the development of robotic devices and wearable sensors for rehabilitation. Contact: dreinken@uci.edu