Closed-Loop Acquisition of Training Data Improves Myocontrol of a Prosthetic Hand

Abstract

Modern myocontrol of prosthetic upper limbs employs pattern recognition models to map the muscular activity of the residual limb onto control commands for the prosthesis. The quality of pattern-recognition-based myocontrol, and that of the resulting user experience, depend on the quality of the data used to build the model. Surprisingly, the prosthetic community has so far given marginal attention to this aspect, especially as far as the involvement of the user in the data acquisition process is concerned. This work shows that closed-loop data acquisition strategies using a feedback-aided approach outperform the standard open-loop acquisition by helping users detect areas of the input space that need more training data. The experiment was conducted in realistic settings, involving one prosthetic hand and tasks inspired by activities of daily living.

This work was partially supported by the DFG project Deep-Hand, CA1389/1-2.

1 Introduction

Pattern-recognition techniques allow controlling poliarticulated prostheses using non-invasive measurements of the muscular activity, such as surface electromyography (sEMG). This is done by training a myocontrol model from a set of muscular activations labelled with the corresponding hand postures. Despite producing models with considerable predictive capabilities, the performance of these approaches depends on how well the training data reflect the actual testing conditions.

Among other factors, changes in the limb position introduce variability in the recorded muscular activity and hinder myocontrol performance. Solutions to this problem involve acquiring training data for the desired hand posture in multiple areas of the reachable space, i.e., in multiple arm configurations [1]. A possibility is to move the arm during the data acquisition to reduce the amount of time and the physical effort required to cover the reachable space [2].

The efficiency of dynamic training data acquisition relates to the arm trajectory that the subject must follow. The trajectory should enforce arm configurations in which the myocontrol model’s predictions tend to be less precise. Identifying those critical arm configurations before the experiment seems unfeasible because they depend on the subject’s characteristics and the performed hand gesture. In this regard, standard open-loop data acquisition, which is based on predefined movement routines, may fail to reinforce the model where needed. We argue that it is possible to identify model weak points during the data acquisition, by monitoring the instantaneous prediction error generated by the stream of training samples.

For this reason, we propose a closed-loop dynamic data acquisition paradigm in which critical arm configurations are identified in real-time and signalled to the subject with acoustic feedback, leading subjects to enforce movements in critical arm configurations. Here we compare the myocontrol models obtained from one open-loop and two closed-loop dynamic acquisition procedures. All the procedures build myocontrol models incrementally and online. One closed loop scheme also exploits a sample selection criterion to reduce the number of model updates performed.

2 Methods

2.1 Data Acquisition

Simultaneous and proportional myocontrol of a prosthetic hand was implemented with an instance of incremental Ridge Regression with Random Fourier Features [3]. The model was trained online during the data acquisition with a stream of 8-channel sEMG samples labelled with normalized velocity commands for the prosthesis fingers.

The acquisition procedure focused on three hand gestures, namely a power grasp, a resting hand, and a pointing index. Each of the proposed acquisition procedures took 45 s for each hand gesture, 135 s in total. The resulting regression model was capable of predicting graded velocity commands, despite being trained on a discrete set of hand gestures.

2.1.1 Open-Loop Dynamic Data Acquisition (OL-DA)

was based on the dynamic data acquisition procedure presented in [2]. Subjects performed the target hand gestures while moving their arm in a helical trajectory spanning the reachable space. The movement proceeded with constant speed from the waist level to the head with the palm oriented downward, and in the opposite direction with the palm upward. The movement was repeated twice uninterruptedly. All the recorded training samples were used to build the myocontrol model in real-time.

2.1.2 Closed-Loop Dynamic Data Acquisition (CL-DA)

combined OL-DA with an acoustic feedback signal dependent on the prediction error obtained for the incoming training samples. Subjects were instructed to perform the same trajectory as in OL-DA, reducing the arm speed anytime the feedback volume increased. They were not required to complete two entire cycles of the helical trajectory in the given 45 s. The prediction error for a labelled training pair $\left\{ \mathbf {x},\mathbf {y}\right\}$ was $e_p=|\mathbf {y}-\hat{\mathbf {y}}|^2$, with $\hat{\mathbf {y}}$ being the model’s prediction for the sample $\mathbf {x}$. The acoustic feedback was related quadratically to the prediction error so to emphasize significant model inaccuracies. The feedback’s intensity varied from 0 (silent) to the maximum volume of the control laptop’s speakers.

2.1.3 Closed-Loop Dynamic Data Acquisition with Sample Selection (CLSS-DA)

was formally equivalent to CL-DA but included a criterion to discard redundant training samples in real-time. All the training samples were used to generate the feedback signal, but only those which returned a prediction error $e_p>\theta$ were used to update the model. $\theta$ was set to 5% of the highest possible prediction error, corresponding to predicting a power grasp instead of a resting hand gesture.

2.2 Experimental Evaluation

The effectiveness of the acquisition protocols was assessed by engaging 12 non-disabled subjects ($20-32$ years old) in a series of realistic manipulation tasks using a commercially available prosthetic hand. A Thalmic Labs’ Myo armband provided 200 Hz 8-channel sEMG measurement of the forearm activity. An Össur’s i-LIMB revolution prosthetic hand was attached to a standard wrist/hand orthotic splint, Fig. 1. A control laptop collected, rectified, and low-pass filtered the sEMG, and used it to train the myocontrol model.

The tasks included serving food and water, sweeping and tidying up (completing a pegboard), and dialling phone numbers. The task sequence was repeated three times for each acquisition protocol. Two repetitions allowed subjects to familiarize themselves with the system, while the third was used to measure myocontrol performance based on the tasks’ completion times. Each subject tested all the acquisition protocols in random order, to counterbalance learning effects.

Fig. 1

The experiment setup included a Myo armband for sEMG, an i-LIMB revolution prosthetic hand, a computer, and several household objects

The performances of different protocols were compared using a Friedman test followed by post-hoc Wilcoxon signed-rank tests with Holm-Bonferroni adjustment of the p-value. The significance level was set to $\alpha =0.05$.

3 Results

Figure 2 shows the duration of the third repetition of the task sequence. The model produced by CL-DA allowed the subjects to complete the tasks significantly faster than that provided by OL-DA (average task sequence duration of 163 s versus 206 s, $W=6$, Holm-Bonferroni adjusted $\hat{p}=0.029$). The average performance of CLSS-DA (168 s) was equivalent to CL-DA and better than OL-DA, but this result was not supported by statistical evidence.

Although each acquisition protocol collected an equivalent amount of training data (about 29000 samples), CLSS-DA trained the model only using about 25% of it.

4 Discussion

We compared one standard open-loop data acquisition procedure, used to reduce the influence of the limb position on myocontrol, to two novel closed-loop acquisition protocols. During closed-loop data acquisition, information about the model’s accuracy was fed back to the subjects using an acoustic signal, a design choice easily integrable in daily routines for myocontrol training and recalibration. The feedback signal ultimately guided subjects to enforce training signals that strengthened the model. One of the closed-loop procedures also implemented a selection criterion to discard unnecessary input samples.

Our experiment showed that closed-loop acquisition improved myocontrol performance in challenging manipulation tasks for non-disabled subjects. The sample selection criterion could reduce the computational load of the training phase, but should be carefully tuned not to hinder the model’s performance. Furthermore, the presented approach heavily relies on the user’s capacity to maintain the correct gesture during training, which could be more challenging in amputees due to a reduced proprioceptive awareness: this topic will be further investigated.

Fig. 2

Subjects completed the task sequence significantly faster in the CL-DA condition compared to the OL-DA condition. The performance of CLSS-DA did not differ significantly from those of the other acquisition strategies