Introduction

The central nervous system (CNS) regulates muscular contraction by adjusting the recruitment number and firing rate of motor units (MUs). Actuation of the MU signals is initiated by the motor neurons in the spinal cord. Collective action of MUs results in composite action potentials, which are eventually recorded as the myoelectric signals (electromyograms, EMG), as shown in Fig. 1. The signal can be detected either invasively using needles (intramuscular EMG, iEMG), or non-invasively using surface electrodes (surface EMG, sEMG) (Merletti et al. 2010a; Farina and Aszmann 2014).

Fig. 1
figure 1

The generation of myoelectric signals

Full size image

As an electrical manifestation of muscle contraction, the EMG signal contains rich information about the neural activities streaming from the spinal cord to muscle fibers. Specific patterns of neural signaling to muscles can be accurately decoded from sEMG signals (Farina et al. 2010). Due to its ease of use, noninvasive acquisition, and highly relevant physiological basis, the EMG has been widely adopted to detect abnormalities of neuromuscular function by estimating the MU activation level/pattern, resulting EMG signals, and correlating with motion biomechanics (Merletti et al. 2010b). Human-body motion analysis also shows that, by measuring EMG, the onset of motion can be detected about 100 ms ahead of physical movement, making sufficient time for controlling active prostheses (Wentink et al. 2013). In this context, the myoelectric signal control (MSC, also termed as EMG control) can be roughly defined as a set of methods to extract necessary motion instructions from myoelectric signals used for driving external device, such as a prosthesis or an exoskeleton. The methods may include specific strategies on EMG data acquisition, signal processing, feature extraction, coding/decoding, pattern recognition, regression, etc.

In 1948, the EMG was first used to drive hand prostheses (Cloutier and Yang 2013). Since then, the EMG control has mainly evolved through four stages: (1) one degree of freedom (DOF) proportional MSC (Fig. 2a); (2) multi-DOF sequential MSC (Fig. 2b) (Dalley et al. 2012); (3) pattern recognition-based discrete MSC (Fig. 2c) (Oskoei and Hu 2007; Kuiken et al. 2016); and (4) regression-based simultaneous MSC (Fig. 2d) (Hahne et al. 2014). In early stages, the user can only control one DOF of the prosthesis, such as hand open/close or wrist pronation/supination; while recently, the pattern recognition-based MSC (PR-MSC) and regression-based MSC (R-MSC) can drive multiple DOFs at the same time, by either discrete or continuous means (Fougner et al. 2012). Sometimes, an encoding strategy (Pons et al. 2005), like Morse code, can also be utilized to replace the pattern recognition algorithm in the PR-MSC. This encoding-based MSC (E-MSC) can be seen as a variant of PR-MSC, with low-cost configuration and simple algorithm, but unnatural control feelings. Currently, a novel synergy-based MSC (S-MSC) is also proposed, which aims to extract muscle synergies (Castellini and van der Smagt 2013; Ison and Artemiadis 2014) from multi-channel EMG to drive all DOFs of the hand prosthesis simultaneously (Fig. 2e). The S-MSC can be seen as a variant of R-MSC, because the proportional signals extracted in R-MSC are re-used to control the ratio of several joint coupling relationships (eigenpostures) of the hand, rather than the joints themselves (for more details, please refer to Sect. 2.1).

Fig. 2
figure 2

Five types of myoelectric control. In a only one DOF can be controlled at one time. The degree of the control objective, hand aperture or rotational speed, is proportional to the amplitude of the EMG signal. In b a switching signal, realized by co-contraction EMG activity or a manual trigger, is used to alternate the control modes (hand gestures or active DOFs) consequentially from all candidates (finite state machine). In c the intended motion can be directly abstracted from multi-channel EMG signals through feature extraction and classification. In d two DOFs, hand open/close and wrist pronation/supination, are controlled simultaneously from two channels of analog signals extracted from EMG signals using multivariate regression algorithms. The method shown in e is a variant of d, as the individual DOFs are replaced with two principal components (eigenpostures) extracted from hand gesture dataset. Driving signals (in the figure, synergy 1 and synergy 2) extracted from EMG signals are used to determine how these two eigenpostures are mixed

Full size image

Despite great progress, currently the MSC methods still have many limitations on its functionality (what actually can it do?), robustness (is it reliable enough against confounding clinical conditions) and adaptability (is it smart enough that can adapt to changing environment?). At present, many MSC methods have a very low functionality, with which the prosthetic hand cannot fully embody its mechanical dexterity. It is obvious that the single-DOF proportional MSC cannot realize sophisticated hand operations (Belter et al. 2013; Liu et al. 2016a). Meanwhile, the switch-based sequential MSC can only accomplish several fixed grasp patterns. This method does not follow the natural motor control pathway and generally needs more training time. In contrast, the PR-MSC adopts an intuitive control strategy that resembles its natural counterpart. However, since only a handful of predefined patterns (grasps/motions) can be realized, the practical dexterity of the hand prostheses is largely discounted. From the viewpoint of control continuousness, the R-MSC method is approaching the closest to natural multi-DOF hand operations. However, only two or three DOFs (mostly, wrist pronation/supination, wrist flexion/extension and hand open/close) can be controlled simultaneously, prohibiting its successful use on dexterous hand prostheses. For addressing this problem, the output of the R-MSC can be used, as a synergy, to drive a set of hand joints together under a specific coupling ratio. Different joint sets with different coupling ratios are termed as eigenpostures, which can be abstracted from human hand grasp datasets using principal component analysis (PCA). In this way, the eigenpostures are the major two or three principal components, as shown in Fig. 2e. Considering the hand’s motion redundancy, this synergy-based MSC can achieve desired hand movements with acceptable accuracy.

Besides the functionality, the robustness of MSC against clinical conditions is still unacceptable. Most experiments reported in the literature were carried out under ideal laboratory conditions, but not in real living environment. Similarly, many studies tend to report their results on healthy, young subjects. The differences of the environments and subjects could be reflected in the usage of different EMG stages and confounding factors such as force variations, body/arm postures (Khushaba et al. 2014), inadvertent EMG activations, etc. The result is that scientific gains reported do not always apply or land themselves to proven clinical and practical results.

Another big limitation of current MSC is that they cannot adapt well to changes. Such changes include psychological/physiological variations, such as muscle atrophy, skin impedance, and fatigue, as well as those coming from outer environment, such as temperature/humidity, electrode shift (Pan et al. 2015), sweating, etc. In an ideal data-collection protocol, these variations are rarely considered that the MSC will gradually lose its efficacy during long-term use. Although the repetitive training can be used, however, it would also degrade the user experience and increase the rate of rejection.

Great efforts have been made to tackle these issues from various aspects. For enhancing the functionality, the targeted muscle reinnervation (TMR) was used to graft residual nerves to the chest muscles for producing sufficient control signals (Kuiken et al. 2009) and intuitive feedback (Merad et al. 2016). Invasive needles or implanted electrodes were used to collect intramuscular EMG (iEMG) directly inside muscles, for providing more pure, accurate information about neural activities (Smith and Hargrove 2013; Kamavuako et al. 2014; Smith et al. 2014). Electrode grids/arrays were used to collect high-density EMG signals (HD-EMG) with preferable contents and resolution, for providing more control possibilities (Daley et al. 2012; Zhang and Zhou 2012; Pan et al. 2015; Stango et al. 2015). For improving the robustness, it is viable to use special electrode configurations (Yinfeng and Honghai 2014), or novel data-acquisition protocols (Ameri et al. 2014c) embracing various EMG variations, such as different signal stages (Lorrain et al. 2011; Yang et al. 2012), various body/arm postures (Shin et al. 2016), and varying muscle force (Al-Timemy et al. 2016). Lastly, for promoting the adaptability, advanced machine learning techniques were tested, such as the online self-recovery or self-enhancing schemes used to hold the PR-MSC’s accuracy (Huang et al. 2010; Chen et al. 2013; Pilarski et al. 2013; Tommasi et al. 2013; Zhang et al. 2016), mode adaptation strategies used to increase user applicability (Liu et al. 2015), and deep learning networks used to improve the feature representation (He et al. 2015b) and classification reliability (Atzori et al. 2016; Geng et al. 2016).

However, these methods can only be found in scientific literature. At present, there is still a significant gap between scientific research and clinical practice (Scheme and Englehart 2011; Castellini et al. 2014b). A fact is that the most available prosthetic hand on the market are still cosmetic or single-DOF grippers. Even dexterous prosthetic hands are available, many users are not inclined to use them (Biddiss and Chau 2007; Castellini et al. 2014b; Atzori and Muller 2015). Commercial dexterous hand prostheses (Belter et al. 2013), like BeBionic™ and i-limb™, mainly use sequential MSC without any sensory feedback (Lewis et al. 2012). The contrast between extensive scientific research and scarce clinical practice has attracted much attention in both biomedical engineering and robotics communities (Castellini et al. 2014b). It has been advocated that a higher priority of MSC research should be given to transferring laboratory studies into clinical practice (Peerdeman et al. 2011; Scheme and Englehart 2011; Ning et al. 2012).

In particular, Farina et al. (Farina et al. 2014) have articulated the problem of reliability of a prosthetic limb control system into two parts: (1) the robustness to instantaneous changes, such as electrode shift and arm posture variation; and (2) the adaptability to slow changes, such as muscular fatigue and skin impedance variation. Castellini et al. also pointed out that, to achieve a clinical breakthrough, research should be focused on improving the robustness, adaptability, and awareness of the MSC methods for hand prostheses (Castellini et al. 2014a). Based on these findings, we deem that, besides weak robustness and insufficient adaptability, the lack of control functionality is also an inducement of the limited applicability of MSC. In this aspect, we start with reviewing different approaches to prosthetic hand control (DOF configuration, discrete or simultaneous, etc.), and how well the control is performed (accuracy, response, intuitiveness, etc.). Besides, in the aspect of robustness, we continue reviewing the confounding factors (limb positions, electrode shift, force variance, and inadvertent activity) that affect the stability of the control performance, as well as their potential solutions. Lastly, in the aspect of adaptability, we review the strategies that can automatically adjust the classifier for different users in long-term usages. We try to clarify the state-of-the-art of current MSC systems and present different implementations, inspirations and new research directions to bridge the gap between research and clinical practice related to MSC.

Status and limitations

In this paper, we consider the performance of MSC from three aspects: functionality, robustness, and adaptability. Regarding the functionality, we try to answer questions such as, what functions can be realized, in which they are implemented, and how well they are performed? Regarding the robustness, we attempt to examine the MSC’s capability to keep its functionality despite various interferences, such as limb positions, electrode shift, force variations, etc. A good robustness shows that the MSC may be transferred directly from laboratory results to clinical applications. Regarding the adaptability, we intend to test the MSC’s generalization capability to the signal changes caused various intrinsic (psychological/physiological) or extrinsic (environmental) inducements. Here, since confounding factors, such as electrode shift, limb movement, etc., also lead to signal changes that, the MSC having a good adaptability can also be asserted robust. Therefore, to separate these two notions, the adaptability is specially defined as an active character of the MSC, indicating its continual learning ability to adapt to various time-varying conditions; while the robustness is defined as a passive, intrinsic character of the MSC against external/internal signal changes.

Functionality

In this section, we discuss the MSC functionality from the perspectives of DOF configuration, control continuity (discrete or simultaneous), accuracy, response, intuitiveness, and system complexity.

DOF configuration

In early days of MSC development, the single-DOF proportional control (Fig. 2a) was widely used. The amplitude of the myoelectric signal was utilized to drive a single DOF’s movement, such as open/close of the hand or flexion/extension of the elbow. When multi-DOF hand prostheses (bebionic, i-limb, etc.) (Belter et al. 2013) became available, a multi-functional MSC was needed to match the hand’s dexterity. However, so far the MSC functionality is still far behind the mechanical evolvement of dexterous hand prostheses. There is still no mature EMG control method that can freely manipulate all the active joints of the dexterous prostheses on the market. The methods of sequential switch (Dalley et al. 2012), selective encoding (Pons et al. 2005), and pattern recognition (Oskoei and Hu 2007) are all pseudo methods for achieving so-called dexterous operations. Generally, a manual switch is needed to select a proper motion from a handful of candidates. These motion candidates can be selected as individual motions of each DOF (individual-DOF mode), and compound motions of all DOFs (compound-DOFs mode). In the individual-DOF mode, the motion of each DOF constitute a motion class. All active DOFs, such as individual finger flexion/extension and wrist pronation/supination, are separated and, at any time, only one can be freely controlled. In the compound-DOFs mode, all DOFs work together to achieve a default motion class, such as cylindrical grasp, tripod grasp, or lateral grasp. Sometimes, for achieving a higher dexterity, motions of individual-DOF mode and compound-DOFs mode often appear in one PR-MSC (Yang et al. 2009). In this way, a big training dataset and a long preparation time are needed.

Most PR-MSC methods focus on the classification of 3-DOF wrist movements and 1-DOF hand open/close. Probably for transradial amputees, these DOFs are the most important ones urgently to be restored. When multi-fingered prosthetic hands became available, the focus began to be shifted from these DOFs to hand grasps (Zecca et al. 2002; Castellini et al. 2009; Castellini and van der Smagt 2009; Li et al. 2010) and individual finger movements (Tenore et al. 2009; Anam and Al-Jumaily 2017; Celadon et al. 2016). However, it is still not clear whether hand dexterity can be well restored by individual finger movements. At present, a widely accepted motion set for the PR-MSC is the combination of 1-DOF wrist movements (pronation/supination or flexion/extension) and multi-DOF hand grasps (spherical, cylindrical, lateral, key, etc.). However, the damaged neuromuscular system might not able to provide sufficient EMG signals to detect these motions. To improve the MSC functionality, the motion primitives that largely contribute to hand dexterous operation have to be clarified first.

A study on hand motion analysis argued that a single-DOF hand with wrist flexion/extension allows operational dexterity comparable to that of a multi-fingered hand (Montagnani et al. 2015). Thus, if we shift the dexterity from individual fingers to the wrist, the capability of transradial prostheses for performing daily activities could also be well preserved. In another study, the authors further indicated that, if the thumb can be opposable, the independent long fingers are not essential anymore for grasps on the whole (Montagnani et al. 2016). Long fingers provide a measureable advantage only when precision grasps are involved. These important findings provide important design guidelines for advanced prostheses, especially when considering the control capability of the amputee and the producing cost. It can be foreseen that the thumb adduction/abduction together with the wrist flexion/extension and pronation/supination will be another focus in future MSC research (Yang et al. 2014a). More specific, a new set of hand motions, including wrist pronation, wrist supination, wrist flexion, wrist extension, thumb adduction, thumb abduction, hand open, and hand close (termed SIG-8 motion set, Fig. 3), will probably become a new criterion in the PR-MSC. Various MSC implementations can be compared on this motion set and thus real progress can be made. This SIG-8 motion set is relatively easy and more practical when compared with the ultimate goal of an MSC, that is, simultaneously decoding all movements during the preparation phase (hand posture adjustment and pre-shaping) and the executing stage (various grasps and operations) of a common reach-to-grasp task.

Fig. 3
figure 3

The eight significant motions (SIG-8 motion set) for hand grasp restoration

Full size image

Research on the human nervous system also shows that the skeletal muscles are regulated by CNS in a synthetic way. This strategy of synthesized motor signals from our neural system, that is, simultaneously coordinating several muscles together to accomplish a common task, is called synergy. Marco et al. (2013) categorized the synergy into three types: neurophysiology synergy (cortical and spinal neurons), kinematic synergy (postures and movements), and force synergy (EMG activities). The gain from the postural synergies of hand grasps can be found in the actuation design of bionic hand prostheses (Brown and Asada 2007; Xu et al. 2013; Catalano et al. 2014; Liu et al. 2016c); while the knowledge from EMG synergy (Castellini and van der Smagt 2013) can be utilized in the synergistic control of multi-DOF dexterous robot hands (Ficuciello et al. 2011; Wimbock et al. 2011; Rombokas et al. 2013; Kent et al. 2014). These approaches can help reduce the system’s complexity while enhancing the prosthesis’s functionality. As mentioned before, for extracting the kinematic synergies, the PCA is generally utilized on a comprehensive dataset of hand movements. However, the PCA only considers the whole joint variance but not the importance of each joint. Thus, it is not clear how the residual variance, particularly for precise movements, influences the hand’s operation. In addition, the best way to construct a comprehensive dataset that integrates all representative hand movements is still an open question in the robotics community.

A similar implementation is to utilize the correlation contents within the trajectories of different joints during a specific reach-to-grasp task (Lacquaniti and Soechting 1982; Liarokapis et al. 2012). For example, from the healthy shoulder’s movement, the missing elbow’s motion was predicted and further utilized in its restoration (Merad et al. 2016). However, whether consistent correlations do exist for all joints in various reach-to-grasp tasks should be examined first. Moreover, the accuracy of this compensation mechanism should be fully verified before use in clinical practice.

Control continuality

Both the sequential MSC (Fig. 2b) and the PR-MSC (Fig. 2c) are a type of discrete control, i.e., at each time point, only one DOF (or, a fixed relationship of DOFs) can be driven, as shown in Fig. 4a. For further improving functionality, a new concept of simultaneous control is proposed (Fougner et al. 2012), which can voluntarily drive multiple DOFs at the same time, as shown in Fig. 4b. This method imitates the human’s motor coordination skills, and largely exploits the prosthetic hand’s dexterity. However, how to extract reliable motor wills for driving each DOF is still challenging. Of all means, regression algorithms from machine learning are widely used (Ameri et al. 2014a; Hahne et al. 2014). Similar to the PR-MSC, the R-MSC also requires adequate training samples. However, collecting the ground-truth targets for the samples from amputees is extremely difficult. At present, the training labels are often obtained from virtual training paradigms (Ameri et al. 2014b) or self-elicited data; lots of experiences are required but limited accuracy can be obtained. An urgent need is to develop a new device (Yang et al. 2018), or a framework (Ju and Liu 2014), to measure, or to deduce, the multi-DOF forces elicited by the hand as the ground-truth labels for training. By adopting those devices such as force-gloves or robotic exoskeletons, various linear or non-linear regression methods can be verified and compared, and an essential approach to improve the performance of simultaneous control could be found.

Fig. 4
figure 4

The discrete and continuous myoelectric signal control. At each time step, in a, only a fixed pattern can be controlled; while in b, both the DOFs and their ratio can be simultaneously controlled

Full size image

Kuiken et al. (2016) thoroughly compared the direct MSC (sequential switch-based, only one DOF is activated at one time, as in Fig. 2b) and the PR-MSC (a fixed hand motion is selected at one time, as in Fig. 2c). The PR-MSC was highly advocated for future MSC research and application. Furthermore, on the basis of the Fitts’ law assessment procedure, Wurth et al. thoroughly compared the direct control, discrete PR-based EMG control, and simultaneous EMG control (Wurth and Hargrove 2014). The authors designed a test that is capable of comprehensively describing the prosthetic control strategies in real time. Of particular note, significant improvements (p < 0.01) were found in throughputs and path efficiencies when comparing the simultaneous control with the direct control or discrete PR. Attributed to its flexibility, the simultaneous MSC generally becomes a new research focus in the community. However, at present this method is mainly limited to wrist movement decoding (two or three DOFs). Only a few of investigations focus on hand movement decoding (5 fingers, 15 joints and 21 DOFs) (Ma et al. 2015). At this point, constructing a hand posture dataset and then extracting 2–3 postural synergies (principal components) as the simultaneous control targets may be a practical solution for the multi-fingered hand’s control (Catalano et al. 2014; Ison and Artemiadis 2014) (Fig. 2e), especially when the actuation system of the prosthetic hand is also designed according to those postural synergies.

It is well known that the power of the EMG signal is highly correlated to the intensity of the neural drives to muscles (Farina et al. 2010). Based on this strong relationship between the EMG power and muscular force, a strategy of force control, rather than position control, can be implemented (Castellini and van der Smagt 2013). With help of multi-variant regression, force control has its solid neurological basis and thus performs superior-to-discrete PR-based MSC. However, the force control needs dynamic muscular contractions, due to which the static training protocol and steady-state EMG used in common PR-MSC is no longer viable (Lorrain et al. 2011; Scheme and Englehart 2013). Since the regression algorithm needs to extract continuous force information inside EMG data, both the steady-state and transient EMG should be collected (Yang et al. 2012). According to the different neurological basis of isometric and isotonic contractions (Merletti et al. 2010a), the features extracted from the steady-state and transient EMG signals may also be different. This instability character of EMG will largely affect the force regression accuracy and thus increase the implementation difficulty of simultaneous control. Developing strategies that can extract accurate multi-DOF force information from multi-channel EMG signals will become a new focus in the simultaneous MSC.

Control accuracy

Accuracy of various MSC implementations has different implications depending on the method used. For the 1-DOF proportional control, because it does not have to select one mode from candidates, the accuracy is assessed as the consistency between the real outputs and its expectation. Because it is difficult to quantify the actual size of the driving motor, this kind of accuracy can only be obtained by indirect metrics of specific experiments.

For the E-MSC, the accuracy can be defined as the hitting rate of each coding, i.e., the reciprocal of the average number of attempts to achieve a desired mode. If activating a specific mode generally needs five attempts, then the control accuracy for this mode is 20% (1/5). As the number of attempts required for activating a mode increases, the control accuracy reduces while the system delay increases.

For PR-MSC, the control accuracy refers to the classification success rate, i.e., the percentage of correctly classified samples, a standard index in PR studies. It is largely expected that the classification accuracy (CA) can reach to an ideal 100%. However, since the data-collection paradigms cannot ensure the purity and completeness of the training samples, the EMG patterns may not be fully learned by a classifier. To prevent an over-fitting learning, a compromise should be made between the classifier’s CA and generalization capability. In the PR-MSC, it means that we should not only consider validating the classifier through offline CA, but also checking its real-time performance during online experiments (Hargrove et al. 2007a). In fact, because of various confounding factors, the offline CA of a PR-MSC cannot be used alone to represent its real performance. In addition, since we generally have more than two motion classes, an overall CA is not adequate to detail the result; rather, a confusion matrix is needed to detail the CA of each class. It is greatly expected that the CA of each motion class will have a reasonable value. If a motion has a relatively low CA, it will be not included in the control, even though it is vital to realize some important hand functions. In practice, when we obtain a resultant label indicating it belongs to class A, it could be either a true positive (it is indeed A), or a false positive (it is not A, refer to receiver operating characteristic analysis, ROC) 49. For a specific class, the true positives rate (TPR) is the proportion of correctly classified samples with originally positive labels, while the false positives rate (FPR) is the proportion of misclassified samples with originally negative labels. A better PR-MSC demands high TPR and low FPR, to output more correct motion commands while introducing less motion artifacts (Yang et al. 2017a).

For the simultaneous MSC, since there are no discrete classes anymore, it is no longer suitable to use the hitting rate to represent control accuracy. Inspired by the Fitts’ law in graphic user interfaces (GUI) (MacKenzie 1992), the metrics, such as throughput, path efficiency, reaction time, overshoot, stopping distance, average speed, and completion time, are proposed to evaluate the performance of simultaneous MSC. The basic idea is to describe the simultaneous MSC as a multi-dimensional target-approaching problem, and then quantify the control performance by examining the execution path, including the overshot (throughout), the time needed (completion time), and the disparity between the real path and the planned one (path efficiency) (Ameri et al. 2014a), as shown in Fig. 5. For the offline validation, the correlation coefficient is also used to describe the similarity degree between an extracted control will and its ground-truth prompt. However, a good coefficient is not sufficient to realize a simultaneous MSC; rather, the intensity of the control wills for each DOF should also be efficiently estimated.

Fig. 5
figure 5

The target-approaching task for a 3-DOF simultaneous control. Note that the x position, y position and angle of the cursor constitute the three DOFs of the control. In this figure, The cursor is required to move from the left-down corner to the right-upper position. Once the target has moved into the acceptance area(s), the target can be said as approached. The completion rate is the percentage of approached targets within a dwell time (say, 10 s). The completion time is the average time to approach a target. The path efficiency is the average of the ratio between the expected trajectory and the traveled distance. The overshoot is the average number of the times the target is approached but lost before completion of the dwell time

Full size image

For clinical application of an MSC, it is also necessary to establish some metrics on how well it performs on real tasks. Light et al. (1999) firstly provided a comprehensive review of the existing hand assessment procedures from both research and clinical points of view. When evaluating these procedures, the authors found that there is still a lack of uniformity and inability to quantify the prosthetic hand’s functionality. The completion time for accomplishing common grasp-and-transfer tasks is widely adopted, such as in the Southampton Hand Assessment Procedure (SHAP) (Light et al. 2002; Kyberd et al. 2009; Kyberd 2011), to indirectly reflect the control accuracy for a prosthetic hand. In addition, in terms of stability, the rate of dropping the objects during the transport phase can also be measured to describe the control’s performance (Yang et al. 2014b). Besides, the methods used to assess the residual functions (Jebsen et al. 1969; Smith 1973) of a damaged hand can also be employed to evaluate prosthetic hands. However, since all of these evaluations are influenced by individual differences like amputation levels, acceptance rates, and training periods, it is not easy to find a comprehensive, fair enough indicator to justify an MSC’s functionality. Considering its application conditions, the MSC cannot be separately evaluated without its user, environment, and prosthetic hand. We say an MSC is “superior” to the others, properly because it can significantly improve the prosthetic hand’s performance, or just because one likes its user experience. In this case, the control accuracy is not anymore a determinant factor for evaluating the MSC. It is worth noting that, we are not intending to find an all-around method for every potential user, but rather an optimal one that fits to the individual situation very well.

System response

Our neuromuscular system requires a few milliseconds to transmit voluntary motion intentions from the brain to skeletal muscles. This is known as neuromuscular lag, and is caused by inefficiencies in signal transmission. In addition, there will be at least 100 ms from the EMG onset to any detectable physical movements (Wentink et al. 2013), known as actuation delay, and caused by the neuromuscular biomechanical properties. Attributed to these intrinsic delays, it was suggested that a reasonable delay for MSC is within 300 ms (Hudgins et al. 1993). Here, the control delay is defined as the duration from the motion initiation (or the start of muscle contraction, i.e., EMG onset) to the prosthetic hand’s physical movement. More commonly, in the PR-MSC, the control delay is defined as the sum of the length of data-processing window and the duration of signal processing for feature extraction and recognition. A large delay leads to slow system response, and affects the control intuitiveness to a certain extent even the CA is high. Some methods, like moving average and majority voting (Englehart and Hudgins 2003; Oskoei and Huosheng 2008), can smooth the outputs and even improve the CA; however, their influence on the system response is also obvious and just equivalent to using large data sets to get more reliable outputs.

To reduce the delay, some methods attempt to use short windows (Farrell and Weir 2007; Smith et al. 2011). A radical approach is to use the features extracted directly from instant EMG samples (Geng et al. 2016). Since the EMG is a type of non-stationary signal, shortening the processing window might magnify the signal’s stochastic feature that reduces the classification accuracy. Thus, there could be a compromised value for the length of the processing window (Smith et al. 2011; Earley et al. 2016). At present, it is widely accepted that a 200 ms-length EMG signal can help acquire stable time/frequency domain (TD/FD) features that show good performance in the PR-MSC. However, it should be clearly noted that the approach that increases the overlap of the moving window might not help reduce the control delay, because, each time a window is used, a control delay is intrinsically introduced by the length of the window, no matter how much the window overlaps.

For the E-MSC, the time delay is determined by the number of bits and the lasting of each bit, represented by values such as three bits and 50–100 ms as in the Manus Hand’s control method (Pons et al. 2005). Generally, a 150–300 ms delay would be introduced to actuate a specific mode. As the number of bits increases, the number of the achievable functions also increases, as well as the control delay and misclassification rate. In addition, the user needs much more training time to adapt to this unintuitive method. Considering the non-intuitiveness of this control strategy, both the number of bits and the length of each bit should be carefully selected.

For the dual-site direct MSC, the control delay largely comes from the size of the window, which is quite small in a single-DOF prosthesis control. However, when a multi-DOF operation is applied, the switching time between different control modes/DOFs largely determines the system response and usability, especially for the sequential switch control shown in Fig. 2b.

For the simultaneous MSC, a processing window is still needed to apply a regression algorithm to extract the driving components. Since the control is continuous, the regression outputs should be as fast as possible to guarantee the continuity. Different from PR-MSC, this simultaneous control needs real-time feedback about every DOF’s performance, based on which the muscular system can rapidly adjust itself in the closed-loop. This rapid adjustment requires the control delay of the simultaneous control largely smaller than the one of PR-MSC, such that the length of its processing window should be greatly reduced (say, less than 50 ms). A rapid algorithm to extract reliable multi-DOF kinematic information is the ultimate goal of simultaneous MSC.

For all MSC methods, the calculation of various algorithms (filtering, encoding, feature extraction, pattern recognition, etc.) also needs some time that affects the system response. At present, the calculation cost of some effective methods, such as TD features and linear discriminant analysis (LDA), stands well for these methods’ online implementation with a suitable system response. However, new featured algorithms, such as continuous wavelet transform (CWT) and multi-layers deep learning network, will be no longer competent to their online implementation considering a limited hardware configuration. At this point, the delay introduced by intensive calculation will become significant and even make online experiments impractical. From this point of view, although these complex methods could achieve better control accuracy or robustness, they may not be applicable for clinical usage and thus have very limited reference value.

Control intuitiveness

The control scheme of MSC should comply with its original neuromuscular pathway, such that the external device is controlled in a way that is similar to its biological counterpart. The psychological effect of this similarity is called control intuitiveness. Both the PR-MSC and simultaneous MSC can be said to be intuitive. From the physiological perspective, the intuitive control method, which conforms to the original neuromuscular pathway, is able to maintain the activity of the undamaged neuromuscular system and brain division, thus effectively avoiding various forms of atrophy. From a psychological perspective, the intuitive control methods can enhance the user’s ownership feeling of the prosthesis, and thus help recover the patient’s body image and self-confidence. It is better for the user to treat the prosthesis as an organ of his/her body, rather than a tool that is operated externally.

Since it is very difficult to compare the intended and physical movements, the control intuitiveness is usually evaluated using subjective methods rather than quantitative means. Especially for the amputees, the ground-truth hand motion parameters cannot be measured because of hand deficiency. It is well known that both the control accuracy and system delay largely affect the control intuitiveness. However, quantitative indicators of these impacts are still missing, and the level of the control intuitiveness of an MSC is largely evaluated by questionnaire ratings.

It is notable that, since the amputation affects the completeness of the neuromuscular system, some patients may no longer be able to operate or perceive a particular DOF. In this case, a remapping strategy can be used to reconnect a perceivable DOF to an expired one, for instance, a wrist ulnar/radial deviation from the amputee’s body could be used to control the prosthetic hand’s thumb adduction/abduction. Although this method loses partial control intuitiveness, the patient can adopt well to these mechanisms after a period of adaptation. The adaptation process of our nervous system behind this remapping mechanism requires further investigation.

As a biomimetic approach where the natural motor control system is activated and reused in the system, the intuitive MSCs may not require long-term training. In contrast, the E-MSC generally needs its user to devote a large mental effort to achieve skilled operation of the prosthesis, and thus a long time is required for adaptation. After training, high operation efficiency can still be obtained, making those methods acceptable in spite of the feeling that the hand is just a tool but not a part of the body. Only, it might have nothing to do with the rehabilitation of the human nervous system.

Functional transparency

It is well known that sensory feedback is an essential part of prosthetic hand control (Peerdeman et al. 2011; Scheme and Englehart 2011; Ning et al. 2012; Castellini et al. 2014b). Current feedback methods include biological sight, mechanical vibration, electrical stimulus, etc. Of all means, biological sight feedback can only provide vision information, but not force or contact, about the interaction. Instead, mechanical vibration or electrical stimulus can be used as a sensory substitute in the efferent neural path of the myoelectric hand. In this case, the MSC should fully support various forms of feedback without noticeable interference. The degree of the MSC for supporting various feedback approaches can be termed its functional transparency. An MSC with a high-degree of transparency can co-exist easily with a large variety of feedback means, to complete information flow during the manipulation. However, this ultimate goal is largely subject to limitations of current feedback approaches and the human body’s perceptual bandwidth. For example, when the MSC is applied together with a transcutaneous electric nerve stimulus (TENS), interactive noise would be produced, as the location of electrical stimulation is adjacent to EMG acquisition (Jiang et al. 2014).

The development of sensory feedback for hand prostheses is largely lagging behind the control strategy. Due to limited human perception bandwidth, usually the number of feedback channels is much less than that of control channels. For instance, when implementing a PR-based individual finger control, sensory feedbacks (position, force, contact, etc.) for individual five fingers are seldom seen in the literature. Rather, only a few of channels (less than three) is used to feedback the overall interaction effect, such as the grasp force. For the direct control proposed in (Smith et al. 2014), since the control signals (iEMG) are directly collected from agonist/antagonist muscles of a joint, the user’s residual proprioception (force or skin tension) of this joint can be utilized as a direct feedback. From this point of view, we can say that the transparency of the iEMG-based direct control is higher than that of PR-based individual finger control.

System complexity

This metric considers both the complexity of the hardware (data-collection configuration) and the software (algorithm calculation cost), and can be described as:

$${\text{System}}\,{\text{complexity}}=\left( {{\text{Hardware}}\,{\text{cost}} \times {\text{Software}}\,{\text{cost}}} \right)/{\text{Control}}\,{\text{achievement.}}$$

To achieve a particular control performance, we expect the system complexity to be as low as possible, i.e., the data-collection setup and the algorithm are both of ultimate simplicity. For a PR-MSC, when the same PR algorithm (feature extraction and classification) is used, less EMG channels means a favorable system complexity. In general, since no invasive facilities, like needles and wires, are needed, the cost of using sEMG is much lower than that of iEMG, resulting in a high acceptance rate. Surely, the HD-EMG can acquire detailed neural activities; however, its high configuration complexity and computational cost are also obvious. To reduce the data scale, data compression methods, such as PCA and non-linear projection, are widely utilized to remove the redundancy while retain the desired information (Zhang and Zhou 2012). Another approach is to utilize deep learning strategies, to find intrinsic features and reliable patterns inside the big data of myoelectric signals automatically (Atzori et al. 2016). Both approaches largely increase the software cost. If a similar achievement (number of controllable DOFs, control accuracy, response, and intuitiveness) is finally obtained, then the system using HD-EMG can be said more complex than the one using a few of channels of sEMG.

Robustness

The factors that affect the MSC’s robustness can be divided into two categories, external factors and internal factors. The external factors include changes in data-acquisition conditions, such as different limb postures, and electrode shift; while the internal factors are generally from user, such as dynamic force changes, inadvertent movements, etc.

Limb postures

In most previous studies, the EMG PR was performed on fixed arm postures. Due to pursuing the goal of high accuracy, different body postures and dynamic limb movements were seldom considered. In fact, the body posture is a very annoying factor largely affecting the accuracy of PR-MSC (Scheme et al. 2011). It is known that the structure of our muscular system (such as the tendon’s length, overlapping forms, etc.) slightly changes under different body postures, which alters the temporal and spatial composition of the EMG signals, and in turn affects the distribution of EMG samples in its feature space. The algorithm tested in a fixed posture cannot deal with this change, resulting in considerable performance degradation. Studies on PR-MSC indicated that training the classifier on several static body/arm postures (five in Fougner et al. 2011, five in; Geng et al. 2012, and three in; Jiang et al. 2013) can alleviate the negative effects caused by posture variations. However, to acquire suitable accuracy, this method needs much more time to collect the EMG from a large variety of postures, which may introduce fatigue and reduce the usability.

Training of the classifier with various static limb postures does not ensure it will work well in other scenarios, such as dynamic arm movements. Considering clinical practice, the PR-MSC should work properly both on static limb postures and during dynamic arm movements. At present, it is still unknown how the EMG signals change along with the dynamic limb postures. In addition, it is still not clear that if the EMG signals collected from several static body/limb postures can encompass all EMG variations in dynamic limb movements. To make the MSC more robust, it is reasonable to include both dynamic limb and body movements during training, thus increasing the integrity of the training samples. Liu et al. explored a multi-condition training scheme that collected data from several arm conditions to improve the robustness of the EMG PR (Liu et al. 2014). Yang et al. also proposed a dynamic training paradigm that collects myoelectric signals with dynamic arm movements and varying contraction levels, rather than on steady postures, to mitigate the classification error (Yang et al. 2017b). However, how to choose suitable, representative motion primitives, and effectively integrate them into the training is still an open question.

Electrode shift

For those MSCs that employ sEMG, the surface electrodes need to be suitably attached to the skin by a clamping force. The electrodes may somehow shift on the residual limb when the contact condition changes, because of an external load, donning and doffing, and varying environmental parameters. This shift may be vital, since it could totally overwhelm the preset, ideal data-collection condition. Especially for the PR-MSC, the influence of electrode shift on the CA is crucial. It is even possible that a few channels of the surface EMG signals could be lost.

Using special electrodes and regulated inter-electrode distance, the negative effects of electrode shift on CA can be mitigated (Young et al. 2011, 2012). It is also viable to combine HD-EMG together with advanced algorithms, such as spatial correlation (Stango et al. 2015) and common spatial patterns (Pan et al. 2015), to increase the control robustness against unstable data-collection conditions. In addition, the development of some special training strategies, such as one that trains the system to recognize plausible displacement locations (Hargrove et al. 2008), can also be helpful. However, one extreme situation of electrode shift is doffing and then donning the electrodes with a loose pairing. A robust configuration or algorithm that can resist the attribute changes, or even some failure, of the electrodes is largely pursuit in the MSC research.

Force variance

In the early stages of EMG PR research, the steady-state EMG signals were largely used to achieve favorable CA. The EMG signals under dynamic muscular contraction (dynamic EMG, hereafter) were seldom considered. This EMG variation can also be termed the force variation, since the elicited force of a joint is largely dependent on the strength of the muscular contraction. Different from steady-state EMG, the dynamic EMG signals contain lots of transient patterns that cannot be well recognized by a static classifier learned from steady-state EMG signals (Lorrain et al. 2011). This is largely attributed to the gap between the training and testing samples. In other words, EMG samples collected separately for training and testing may not have identical distributions. Including the transient EMG into the training dataset may achieve a classification that effectively reduces the error appearing in the initial and transitional stage of the movement (Yang et al. 2012), thus resulting in an improved system response. If different limb positions and dynamic contraction forces (30%-50% MVC) are both involved in the training, then the EMG PR can effectively promote its robustness (Scheme and Englehart 2013). New feature sets can also be used to remove the force information but keep the mode information for improving the CA (He et al. 2015b; Al-Timemy et al. 2016). However, taking full advantage of the force and mode information inside the EMG signals to realize a simultaneous regression and classification is still challenging.

Inadvertent EMG

Caused by involuntary muscular contractions, inadvertent EMG activation has also been seen as one of the main reasons for frustration during clinical testing (Scheme et al. 2013). For a transradial amputee, the most remarkable inadvertent activation happens in the interference of wrist movement with hand motions (Earley and Hargrove 2016). When the hand moves, involuntary motion compensation (especially, wrist pronation/supination and flexion/extension) will produce some adjunctive muscular activities at the sites of EMG collection, and these will largely contaminate the EMG for hand motion classification. The EMG signals produced by wrist motions and finger movements may have crosstalk at the sites of collection that makes accurate classification of them very difficult (Yang et al. 2014a). In particular, this inadvertent EMG activation is obvious in partial-hand amputation (Earley and Hargrove 2016), largely because the patients rely on the healthy wrist to adjust their hand postures for grasping. Co-activation of wrist-related and finger-related muscles further prevents accurate recognition. By designing special feature sets (Khushaba et al. 2016), extending data-acquisition area to include intrinsic and extrinsic hand muscles (Adewuyi et al. 2016), dynamically adjusting the window length (Earley et al. 2016), and improving the classification’s confidence score, the robustness of the MSC for hand motions can be effectively promoted against wrist movements.

Adaptability

In this paper, the MSC’s adaptability mainly refers to its generalization ability to the signal changes caused by individual differences and long-term usages. When applying the MSC on different users, this indicator is used to evaluate the MSC’s generalization performance among users with a few of adaption, or even zero training (Liu et al. 2015). For the same user, this indicator is used to describe the stability of the MSC when applying it in a relatively long term.

Individual difference

The individual differences of human bodies make the MSC perform differently for different users. The special structure of our neuromuscular systems, various amputation levels, skin impedance, physiological status or psychological activities, fatigue, and other such factors have a compound impact on the MSC. To obtain a wide range of application, it is not possible to verify a new method only on a single subject such as in a case study. To acquire a reasonable, unified method for a large majority of users, researchers are expected to report the average experiment results from multiple subjects. However, our previous study showed that a preferable MSC implementation (including the motion patterns, the specific approaches for encoding, on/off switch or pattern recognition) might vary significantly from person to person (Yang et al. 2014b). Although some methods can achieve reasonable CA, it does not mean that all users would like to use them in real life. On the one hand, we expect that the selected algorithm could have the adaptability to meet the requirements of a majority of potential users; while on the other hand, we hope the algorithm could fully exploit the user’s potential, that is, to be the most suitable to each individual. Clearly, a compromise should be made between these two requests.

Seeking a generalized approach or architecture from a group of subjects and then customizing it to its end user according to individual characteristics can effectively simplify the configuration procedure of MSC. Tommasi et al. proposed a general method to reuse past experiences, in the form of common models synthesized from previous subjects, to boost the MSC’s adaptability (Tommasi et al. 2013). Comprehensive tests were conducted on databases recorded from healthy subjects in either controlled or non-controlled conditions. Their results revealed that the method could significantly improve the CA over the baseline (non-adaptive) case. This promising approach might be utilized to pre-train the prosthesis before shipping it to the end user.

Long-term usability

During a long-term use of MSC, there will be some signal variations inevitably caused by psychological and physiological changes that lead to accuracy degradation (He et al. 2015a). The phenomenon whereby the probability distribution of target data changes along time is called concept drift, where the term concept refers to the probability distribution of an EMG pattern in feature space (see Fig. 6 for an example). The adaptability of an MSC largely depends on how long it can effectively maintain the control accuracy. It is argued that the MSC could adapt during longer-term use, thus to ensure that the users do not have to frequently return to the hospital to re-calibrate the algorithm (Sensinger et al. 2009). We believe that keeping the MSC effective in a single day’s use can well improve the system’s acceptance (Yang et al. 2013). This requires an intelligent strategy that automatically adjusts the parameters of the control method and adapts with the user along with the usage.

Fig. 6
figure 6

An example for concept drift of EMG patterns and adaptive learning. Each dataset was collected at 30-min intervals. The varying EMG pattern is denoted using different colors according to their life span. An adaptive concept for this EMG pattern (denoted in blue line) was learned using support vector domain description (one-class SVM), given the original dataset and incremental samples

Full size image

In general, this strategy can be mainly classified into two categories, supervised and unsupervised methods (Sensinger et al. 2009). The supervised methods can carefully control the MSC’s learning process, since the ground-truth labels for the samples used for re-training can be obtained, although this method requires cumbersome training trials. Meanwhile, the unsupervised methods attempt to adapt without any knowledge about the labels of the intended movements, being friendly to the user but at the cost of reduced accuracy and efficacy. The authors conducted a comprehensive comparison of four supervised and three unsupervised adaptation paradigms. Their result indicated that, compared with the non-adaptive classifier as a baseline, all supervised adaptation paradigms can significantly reduce the error over time (at least 26%); while the unsupervised adaptation paradigms provide very limited error reduction due to the uncertainty of the label of the real-time samples. Thus, the authors argued that supervised adaptation should be considered for incorporation into any clinically viable PR controller.

Oskoei et al. proposed adaptive schemes for the EMG-based human–machine interface (HMI) in a video game (Oskoei and Hu 2009). These adaptive schemes can modify the online classification criteria and thus keep a stable control performance. As the support vector machine (SVM) was used as the classifier, its incremental training method was utilized in both supervised and unsupervised manners to update the online training dataset. Kawano et al. adjusted the size of the training set online according to the sample’s Kuhn-Tucker condition and constrained the learning procedure in the interval of two recognized motions (Kawano et al. 2009). By doing so, the prediction and adaptation phases of SVM were separated that the calculation cost was largely reduced. In lights of these studies, we further extended this method using support vector domain description (SVDD) (Tax and Duin 1999) instead of SVM. One merit of SVDD is that it describes the motion classes using one-class strategy, that is, each motion class has its own boundary that is friendly to its online adaptation. Considering the sample’s lifespan, a forgetting factor was further applied to regulate the speed of the online learning (Yang et al. 2013).

Using the adaptive learning, the robustness of EMG PR can be improved to a certain extent. However, in the process of adaptation, only a priori knowledge of the features is considered. In general, the environmental changes are not included in the online learning process. To this end, Pilarski et al. proposed an adaptive decoding method based on environmental adaptation and historical experience (Pilarski et al. 2013). The method adopted a general evaluation function to describe the probability distribution of previous classification results. The assessment of the previous classification and the sensed environmental information were utilized as an evaluation index to guide the updating of the classification algorithm. Zhang et al. evaluated several kinds of online algorithms in classifying hand motions (Zhang et al. 2016). Their results showed that second-order online learning algorithms generally outperformed first-order algorithms, indicating that online learning methods with large margin and confidence weight can generate good results.

A classifier with a good discrimination of multiple EMG patterns should act as a basis for achieving effective adaptations. Generalized information extracted from the signals that are collected from repeated sessions can somehow encompass the signal variations, thereby reducing the time for re-training and calibration. Liu et al. proposed a domain adaptation (DA) structure that can automatically extract this useful information from the old models (Liu et al. 2016b). When combined with a polynomial classifier or LDA, this DA structure can significantly promote the CA. The authors also assumed that, for a specific pattern, there would be some certain invariant features in the long-term EMG signals (Liu et al. 2015). They proposed use of a framework of common model component analysis (CMCA), wherein a projection optimization is utilized to minimize the dissimilarity among multiple classifiers (LDA) trained on different days. The CMCA has better generalization ability over unseen data when compared with the baseline (non-adaptation) method. The CMCA holds great potential in the effort of developing zero re-training methods of myoelectric pattern recognition.

The CA degradation of EMG PR in long-term usage can also be mitigated though intensive subject training, i.e., to regulate the subject’s outputs into more consistent, repeatable signal patterns. He et al. found that, when the classifier was trained and tested on two consecutive days, the classification error decreased exponentially but plateaued after 4 days for able-bodied subjects and 6–9 days for amputees (He et al. 2015a). The relative changes in EMG features over time became progressively smaller when the number of training days are increased. The learning curves, for both healthy and disabled subjects, could be modeled as an exponential function. Their results provide important insights into the user adaptation characteristics during practical long-term myoelectric control applications, as well as meaningful implications for the design of an adaptive PR system.

Solutions and directions

At present, an effective, widely accepted MSC that meets all needs on functionality, robustness and adaptability is still missing. Though issues have been well recognized, there seems to be no all-purpose solution that can tackle these problems at the same time. In fact, there also seems to be no research priority on these aspects. Functionality is the footstone of the MSC, it largely determines what a hand prosthesis can actually do with the MSC; robustness is an index of the MSC’s usability, i.e., to which extent the MSC can be qualified in real, complex clinical scenarios; while, the adaptability is a metric of the MSC’s intelligence, it largely determines how long should the user have to calibrate the system. Rather than priority, we advocate an order (functionality→ robustness →adaptability) when proposing a new MSC design. This order can also be adopted to evaluate and compare different MSC methods.

Functionality

From the perspective of functionality alone, there are still many issues to be resolved. Currently, most MSC methods are still confined to the hand’s grasp control, i.e., applying a suitable form or force closure to the object (Napier 1956). To realize a dexterous within-hand operation, how to use MSC to coordinate multi-finger movements is still an open question. An operational MSC can afford the prosthetic hand powerful and precise manipulation functionality. However, due to the numerous hand DOFs and limited EMG signals, it seems impossible to realize a high-level dexterous control. The methods to resolve this issue include new grasp taxonomy (Bullock et al. 2015; Feix et al. 2015) that helps understand the hand’s grasp/operation functionality, new motion primitive derivation (Jenkins and Mataric 2002) that helps simplify the control process, and novel approaches that consider the hand manipulation as a transition phase between different grasps (Elliott and Connolly 1984; Li et al. 1989).

In addition, it is also possible to increase the depth (iEMG) and breadth (HD-EMG) of the EMG representations to improve the control functionality. Early studies showed that, compared with sEMG, using iEMG might have no significant CA difference in the PR-MSC (Hargrove et al. 2007b). However, recent investigations clearly indicated that a combination of iEMG and sEMG together can achieve a higher CA than sEMG alone (Kamavuako et al. 2014). Since the iEMG has nature properties of anti-interference, anti-crosstalk, and precise anchoring, it is still very promising in the multi-DOF MSC (Smith and Hargrove 2013; Smith et al. 2014). Attributed to the increased dimension and resolution of the HD-EMG signal, we can get more recognizable patterns and higher success rate (Daley et al. 2012; Zhang and Zhou 2012). We can even infer the MUCP’s firing process (Merletti et al. 2008), which is useful for understanding the neurophysiological processes of a hand motion. The HD-EMG collection largely expands the dimension of the feature space. In general, it cannot be directly used in classification without dimension reduction. At this point, how to quantify a dimension reduction method for ensuring the compressed data’s efficacy for classification is still unknown. Recent studies have attempted to use deep learning from artificial intelligence, which feeds HD-EMG signals as a two-dimensional image into a multi-layer network, implicitly collecting useful information for feature extraction and classification, and directly outputting the class probabilities. The merit of using deep learning framework is that it does not need any prior knowledge about the EMG PR; instead, it treats the myoelectric signals directly as images. However, this method generally requires the developers to have plenty of knowledge/experience on network structuring and training/validating, as well as powerful hardware such as large-volume GPU and high-speed CPU.

When few EMG signals are available, additional sensing modalities can be introduced to help improve the control functionality. A large variety of bio-sensing technologies can be used, such as sonomyography, mechnomyography, electroneurography (ENG), electroencephalograhy (EEG), electrocorticography (ECoG), intracortical neural interfaces, near infrared spectroscopy, magnetoencephalography, and functional magnetic resonance imaging (Fang et al. 2015). Considering their complementary features, fusion of different signal modalities can provide new feasibilities in MSC. However, of all these means, the myoelectric interfacing seems to be the only viable way to control external devices in commercial and clinical systems because of its easy access and use (Ning et al. 2012; Farina et al. 2014).

Research on robot vision can also help improve the MSC’s functionality, on providing information on the object’s category, shape, and dimension. This information can be utilized in the closed-loop control of a prosthetic hand for automatically adjusting grasp postures. For instance, a suitable set of points and axis can be determined by robot vision (Saxena et al. 2008; Sahu et al. 2017) for determining the grasp direction; or, the object shape/size information (cube, cylinder, sphere or key) can be extracted and used to assist the prosthetic hand in selection of a suitable grasp pattern (Tucker and Ellis 2001). Another attractive approach is use of radio frequency identification (RFID) to achieve an automatic grasp selection when the hand approaches the tags on different objects (Morph, Infinite Biomedical Technologies, USA) (Trachtenberg et al. 2011). Clearly, this approach reduces the burden of extracting the mode information from EMG signals. However, whether this local intelligence confuses the user and reduce the sense of embodiment of the prosthetic hand in their minds still needs further study.

To be frank, from the view of dexterous operation, current MSCs still have many limitations. In robotics, one pressing question is identification of the most important motions that a prosthetic hand should have. Many studies select the motions of their own experiences, such as wrist movements, grasps, or individual finger movements, a system which lacks sufficient support from academic research and statistics. This random selection also creates difficulties in comparing various approaches. To address this problem, Bullock et al. proposed a vision system that can capture various hand activities performed by housekeepers and machinists in their daily life (Bullock et al. 2015). They counted out the grasps types and their occurrence frequency from a relatively long-term data collection. This statistic on human hand movements provides a valuable reference for selecting a proper motion set for the MSC. Standardizing and categorizing various grasp patterns and then unifying the motion selection should be a priority in current MSC studies. In general, we believe that the motion patterns should be determined according to the level of amputation and the significance of the joints. For instance, the individual finger movements are important for restoring finger dexterity and should be considered first in partial-hand prostheses; while, wrist motions and hand grasps are essential for accomplishing reach-to-grasp functions that are required by transradial hand prostheses. In general, the wrist’s pronation/supination and flexion/extension are recognized more important than its ultra/radial deviation, from the view of providing rapid, accurate hand postures. The thumb’s adduction/abduction together with the whole hand’s close and open can accommodate most hand functions, while individual finger movements seem to be important to realize dexterous hand operations. Here, we propose a motion set as wrist pronation, wrist supination, wrist flexion, wrist extension, thumb adduction, thumb abduction, hand open, and hand close (SIG-8 set), for the myoelectric control of transradial prostheses. Furthermore, for transhumeral prostheses, the elbow extension/flexion should be restored to enable the hand to reach for the object. For shoulder disarticulation, motions coming from the shoulder should also be considered. In this case, because the relevant muscles might not exist anymore, extra neural signals like EEG and ECoG (Hotson et al. 2016), etc.), surgery like TMR (Kuiken et al. 2007), and multi-modality fusion strategy (McMullen et al. 2014) should be introduced to leverage the functionality. Based on the synergistic movements of the forearm, coordinated control for multiple joints might be an efficient way to solve the control issue of dexterous prostheses. Of course, an effective training with acceptable regression accuracy must be built first on sufficient samples and credible labels.

As mentioned before, current MSC studies are difficult to compare, since there still is no standard evaluation framework. For a given population of potential users, what is the “best” MSC method? Are there any generalized, standardized procedures to make the “best” method available? To answer these questions, MSC studies in the future should also be devoted to new procedures, platforms (both hardware and software), and strategies for evaluating the hand’s performance.

Robustness

The low robustness of MSC will continue to annoy researchers for some time. Previous studies of PR-based MSC highlighted the CA but neglected the stability of the accuracy under confounding factors, which are fatal to the system in clinical practice. At present, there are three main approaches to solve this problem: (1) new hardware configuration; (2) training protocol refinement; and (3) robust feature extraction. The first approach emphasizes improving the reliability of signal acquisition, to avoid electrode shift or to control the electrochemical reaction between the electrode and the human body. The second approach tries to integrate practical factors, such as, various limb positions and different EMG signal stages, into the training procedure to improve the representation and completeness of training samples. The third approach attempts to extract the common information from individuals, thus simplifying the repeated training process among different users. Despite these approaches, current studies on the MSC robustness rarely consider multiple confounding factors together. Thus, it is not clear how well an MSC method will perform in practice. We firstly considered multiple EMG variations together (limb posture, force variation, inadvertent EMG, and wrist supination/pronation) in a sets of dynamic data-collection protocols (Yang et al. 2017a). Our results showed that the protocol with dynamic limb postures and dynamic muscle contractions obtained the highest accuracy. It clearly showed that, to achieve good performance, it is not necessary to include all variance in the training samples.

The inertial measurement unit (IMU) can also be employed to measure limb postures. Appending limb posture information into the EMG data expands the dimension of the feature space, which may address the crosstalk problem (overlapping samples in the original feature space) caused by limb movements. A pioneer study have shown this feasibility (Fougner et al. 2011). Moving forward, if we have a corresponding sensor to detect any specific EMG variation, and then successfully integrate these sensors’ reads into the EMG features, undoubtedly the robustness of the MSC could be further improved.

Adaptability

Due to the influence of changing variables, the MSC is not always effective during use. Some MSC methods integrated with adaptive learning can maintain their accuracy by adapting to these changes. However, the adaptation capabilities vary largely with the algorithms. Since we know little about the changing process, the ground-truth labels used for calibrating the intelligent machine are not easy to decipher. This is true for both classifier and regressor, whether the labels are discrete classes or continuous forces. This creates a paradox, that, on one hand, since we have no idea about the ground-truth labels, we cannot know how to calibrate the classifier; on the other hand, if we know the labels well, it means that the classification is good and we do not need to retrain the classifier. To find a real-time method of determining whether the samples are classified correctly is equivalent to finding an adaptive strategy for learning the changes. These two tasks are somehow the same in terms of maintaining the control accuracy. Hence, to improve the control’s adaptability, the focus should be on finding a method that can effectively pick up the misclassified samples and resign them to the correct labels. It is extremely difficult, especially for an unsupervised approach.

A more general method is to endow the samples with an attribute of lifespan, making the classifier always try to learn from the most recent samples (Yang et al. 2013). In this strategy, the labels used for re-training are given by the classifier online. This method hypothesizes the change of current samples but cannot escape the boundaries defined by the previous classifier. Thus, for those slow signal changes caused by the outer environment (temperature, moisture, etc.), this adaptation strategy works quite well. However, this method cannot adapt to rapid/sudden signal changes such as electrode shift (doffing & donning), force variation, etc.

Another approach is constructed from the morphological distribution of the samples in the feature space. It employs unsupervised learning (cluster) strategies to determine the feasible labels, and then update the classifier based on these new labels (Huang et al. 2017). This strategy assumes that there must be a certain similarity among the samples belonging to the same class, thus making them distribute as a tight cluster within the feature space. Since creating a cluster needs a proper number of samples, the batch learning is generally required to update the classifier in the intervals of motions.

If an evaluation function was built to assess current performance of the MSC, then a reinforcement learning method could be established by introducing rewards on different adaptation policies. The adaptive actions that maximize the reward could be selected to determine the present sample’s label. However, this evaluation function of the MSC can only be implicitly displayed. The online assessment of the efficacy of hand operation is very difficult, whether in the field of robotics or biomedical engineering.

It is encouraging, state-of-the-art deep learning framework provide the MSC more possibilities. One day we may be able to collect sufficient, detailed EMG activities using HD-EMG, and then easily and completely extract useful information from those signals using deep learning networks. Sparse representations (Betthauser et al. 2017) for describing the motion pattern or force size could also be robust against confounding factors and reliable during long-term use. All these together could promise a suitable functionality for MSC in clinical practice.

Conclusions

Among different approaches like EEG, ECoG, and ENG, the EMG interfacing currently might be the only viable way to control external devices in commercial systems. An intuitive control method for a multi-DOF prosthetic hand with powerful functionality for dexterous operation, proper robustness against confounding clinic factors, and reliable adaptability over different users and long-term usages, is a common goal in the literature. Despite the massive research, few prostheses adequately meet the requirements of functionality, robustness and adaptability at the same time. Research argue that the focus of MSC study should be transferred to clinic practice. However, the current methods may not be ready for clinical application. The differences between ideal laboratory conditions and complex living conditions mean that the good results reported in the literature are difficult to reach in real lift. In this paper, we propose a framework for evaluating the performance of the MSC, from three perspectives: functionality, robustness, and adaptability. We suggest focusing current investigations on robustness and adaptability. Because, compared with a slight promotion on the offline control accuracy, we are more caring about how to maintain a reasonable online accuracy in clinic practice.