Keywords

1 Introduction

Neural prosthetic hand is a typical human-machine interface application, which is promising in the rehabilitation of amputee. Though pattern recognition based prosthesis has made great success over the past few years [1, 2], the clinical applicability of it is limited. There remains many significant challenges in bio-signal controlled prosthetic hands, one of which is the simultaneous and proportional force estimation. It is a challenging task to establish a force prediction model with strong anti-interference ability and robustness based on bio-signal.

Surface electromyography (sEMG) is a popular neuromuscular interface [3,4,5]. Due to reflecting the activation degree of muscle contraction, sEMG signal is usually used in force estimation [6,7,8]. Choi et al. [9] applied ANN algorithm to estimate pinch force by sEMG and achieved a promising result with NRMSE of \(0.081\pm 0.023\). Cao et al. [10] adopted extreme learning machine (ELM) to predict hand grip force by sEMG signals from forearm muscles, and the achieved root mean squared error (RMSE) and correlation coefficient were \(1.165\pm 0.475\) and \(0.991\pm 0.007\) respectively. Though the aforementioned studies have obtained high accuracy, the problem with them is not taking the non-stationary property of sEMG into consideration. SEMG signal is susceptible to muscle fatigue, electrode displacement and sweating, which limits its long-term use. Electrode displacement and sweating can be avoided by strictly controlling the external environment, but muscle fatigue is inevitable.

Ultrasound signal is another biosensing method. The principle of ultrasound generation is that the ultrasound probe sends ultrasound pulses into human tissue and the pulses will be reflected at tissue interface. These reflected echoes can be used to detect the morphological changes in muscle deformation. As reported by [11], there is a linear relationship between finger force and spatial first-order features extracted from ultrasound images of the forearm. Although the research of human-machine interface based on B-mode ultrasound image has made great progress [12, 13], the B-mode ultrasound device is bulky and not portable. In contrast, A-mode ultrasound apparatus [14] is more realistic to be wearable and embedded into prosthetic hands. Zhou et al. [15] utilized A-model ultrasound to predict finger movement and obtained excellent accuracy. Yang et al. [16] successfully applied A-mode ultrasound signal to predict grasp force with average NRMSE of \(0.102\pm 0.037\) and \(R^2\) of \(0.905\pm 0.057\). The aforementioned studied proved that force regression accuracy on A-mode ultrasound signal was acceptable. However, whether A-mode ultrasound based force estimation is robust against muscle fatigue remains uncertain.

In order to further improve the performance of human-machine interface, multi-modality fusion has gradually become a research trend. Guo et al. [17] developed a hybrid EMG/NIRS sensor system, and recognition performance were improved. Leeb et al. [18] developed the multimodal fusion approach of EEG and EMG, which yielded accurate and more stable performance even in muscle fatigue condition. Inspired by these work, this paper explores the difference between sEMG and ultrasound signals in force regression accuracy and fatigue robustness, and investigates the fusion of sEMG and ultrasound modality. It is expected that their combination can exert the complementary advantages of muscle morphology information and electrophysiological information, breaking through the bottleneck of single modality and attaining both high accuracy and robustness against muscle fatigue.

2 Muscle Fatigue Robustness Evaluation

2.1 Subjects

Seven healthy subjects (referred as S1–S7, 23–29 years old, six males, one female) without any neuromuscular diseases history were recruited to participate in this experiment. The maximum voluntary contraction (MVC) of each subject was measured for experiment configuration.

2.2 Experimental Apparatus

Fig. 1.
figure 1

The overview of experimental platform: (a) The employed signal acquisition apparatus, including hybrid sEMG/A-mode ultrasound system (HEUS) and hand grip dynamometer; (b) The exhibition of four-channel sensor arm band, each small module is embedded with a pair of sEMG electrodes and an ultrasound probe; (c) The real-time acquisition of ultrasound signals (left side) and sEMG signals (right side); (d) The curve of the actual hand grip force (green line) collected by dynamometer and an estimated force (red line).

Full size image

A portable hybrid sEMG/A-mode ultrasound system was employed in this study. The detail of this hardware system can be seen from [14]. The system is capable of simultaneously acquiring four channels of sEMG signals and A-mode ultrasound signals. Owing to the four-channel sensor arm band shown in Fig. 1(b), these two signals can be collected in the same position, which makes comparasion of sEMG signals and ultrasound signals impartial and meaningful. Hand grip dynamometer (G100, Biometrics Ltd., UK) was applied to evaluate the MVC of subjects and record the hand grip force when exacting contraction. It should be noticed that the sample rate of sEMG, ultrasound and force sensors 1000 Hz, 10 Hz, 1000 Hz respectively.

2.3 Experimental Protocols

During the experiment, the subjects were asked to sit in front of a PC screen with their elbow laid comfortably on the desk. The four sensor modules in the arm band were placed on four muscles: flexor digitorum superficialis (FDS), flexor carpi ulnaris (FCU), extensor carpi ulnaris (ECU) and extensor digitorum (ED). The subjects hold the grip dynamometer with their dominant hand.

Firstly, the subject gave the most strength to grasp the dynamometer for 10 s, the average force between 2.5 s and 7.5 s was defined as his MVC. The experiment configuration in the following stage varied between subjects according to the MVC.

Fig. 2.
figure 2

The demonstration of the experimental protocols: (a) The demonstration of indicative force curve on the computer screen for the whole trial (15–20 repetitions), (b) the indicative force curve of one repetition.

Full size image

Secondly, after evaluating the MVC, the subject was instructed to exert grip force in the shape of triangular curve. As present in Fig. 1(d), there was a moving green line indicating the real-time grip force and the subject was asked to precisely control their grip force to track the red line of triangular curve. The subject was asked to output 15 to 25 repetitions of triangular wave force according to the instructions on the screen. The number of repetitions varies according to each person’s fatigue resistance. Each repetition lasted 30 s. In the first 5 seconds and the last 5 seconds, the subject rested. Between 5 seconds and 15 seconds, the subject’s grip strength was slowly increased from 0 to 80\(\%\) MVC, and between 15 seconds and 25 seconds, the subject’s grip strength was slowly reduced from 80\(\%\) MVC to 0. The indicative force curve of one repetition is shown in Fig. 2(b). The subjects repeated the repetitions until he felt intense muscle fatigue and his grip force was not sufficient to track the indicator force curve on the screen. Whether the highest point of the force curve in one repetition was less than 70\(\%\) of MVC was regarded as a sign of muscle fatigue. The procedure repeated 3 trials, with an enough time rest between each trial in order to allow muscles to recover from fatigue.

In each trial, during the first six repetitions, the subject’s muscles were considered to be in non-fatigue state. The data collected from first six repetitions were used as the training set. The muscle status of the following three repetitions was considered to be similar to that of the first six repetitions and can also be considered as non-fatigue state. The data collected from this three repetitions were used as the testing data on non-fatigue state (referred as Test-NF). The last three repetitions before the subject’s grip strength is insufficient to track the screen’s indicator force curve were designated as fatigue state. The data collected from this three repetitions were used as the testing data on fatigue state (referred as Test-F). The illustration of split method for selecting the training data, Test-NF, Test-F is shown in Fig. 2(a).

2.4 Signal Processing and Feature Extraction

Fig. 3.
figure 3

The process of signal alignment and preprocessing on sEMG, ultrasound and force signals: (a) force signal and its slide window, (b) sEMG signal and its slide window, (c) original A-mode ultrasound signal.

Full size image

Firstly, samples needed to be separated and extracted from the original sEMG, ultrasound and force signals. The process of signal alignment and preprocessing on sEMG, ultrasound and force signals are shown in Fig. 3. Since the sampling rate of sEMG signal 1000 Hz and the frame rate of ultrasound signal is 10 frames per second, ultrasound and sEMG signals need to be unified and aligned. The sliding window and step of sEMG was set to 250 ms and 100 ms respectively, consequently, the sEMG samples is separated 10 times per second by slide window. Through this approach the matching of the ultrasound samples and sEMG samples is achieved. In addition, the same sliding window operation is performed on the force signal to maintain the same samples number. Concretely, Fig. 3(a) and Fig. 3(b) represent the force signal curve and sEMG signal curve respectively. Force signals and sEMG signals are one-dimensional time series signals. Different from them, ultrasound signals can be regarded as one-dimensional images, which are updated every 100 ms. Figure 3(c) displays several different frames of ultrasound signals and zooms in on two of the frames.

For sEMG samples, TD-AR6 feature is employed in this paper. TD-AR6 feature is a combination between four time domain features (mean absolute value (MAV), waveform length (WL), zero crossing number (ZC), and slop sign changes (SSC)) and 6th order AR coefficient of autoregressive model (AR6). For ultrasound, each frame is composed of 1000 sample dots. The first 20 dots and last 20 dots were discarded as they contains no meaningful information. The raw ultrasound signals are processed by time gain compensation, band-pass filtering, envelope detection, and log compression [16]. Then the processed ultrasound signals are divided into 48 segmentations. The MSD [16] feature, which represented mean value and standard deviation, is extracted from each segmentation of ultrasound signals. The TD-AR6 features and MSD features across all channels are regarded as sEMG features and ultrasound features respectively. The stacked features of these two signals are used as fusion features of sEMG and ultrasound.

2.5 Linear Regression Model

This article applied linear regression (LR) model to hand grip force by the features of bio-signals. The ability of the sEMG/ultrasound signal features to characterize the grip strength was evaluated through the force estimation accuracy. The LR model is defined as Eq. 1:

$$\begin{aligned} \hat{y}_i=\hat{x}_i^T(X^TX)^{-1}X^Ty \end{aligned}$$
(1)

where y means labels of train data, and \(\hat{x}_i\), \(\hat{y}_i\) represent feature vector and predicted label of test data respectively. The train dataset is represented as a matrix X of size \(m*(n+1)\), and m, n represent number of train samples and feature dimension respectively. Each row of the matrix X represents an instance and each column represents a feature.

2.6 Evaluation Metrics

The aforementioned regression models were employed to estimate the exerted hand grip force. The normalized root-mean-square error (NRMSE) were applied to quantify the force estimation precision, which were defined as

$$\begin{aligned} NRMSE=\frac{\sqrt{\sum _{i=1}^m(y_i-\widetilde{y}_i)^2/m}}{y_{max}-y_{min}} \end{aligned}$$
(2)

where m means the total number of test samples and \(y_i\), \(\widetilde{y}\), \(y_{max}\), \(y_{min}\) represent predicted force, actual force, maximum of actual force, minimum of actual force respectively. The NRMSE of non-fatigue state and fatigue state are compared to evaluate the robustness to muscle fatigue.

Fig. 4.
figure 4

The comparison of hand grip force estimation accuracy on Test-NF and Test-F across different modalities (separate sEMG, separate ultrasound and combined sEMG/ultrasound). Error bars represents the standard deviation.

Full size image

3 Results

3.1 Force Estimation Performance and Robustness Evaluation

The accuracies of different modalities (separate sEMG, separate ultrasound and combined sEMG/ultrasound) for hand grip force estimation are investigated in this study. The statistical results are shown in Fig. 4. The symbol Test-NF and Test-F represent testing on non-fatigue state and fatigue state respectively. Considering force estimation accuracy on Test-NF, separate sEMG signal always outperforms separate ultrasound signal, and there is little difference between accuracies of combined sEMG/ultrasound signal and separate sEMG signal. The average NRMSE of sEMG, ultrasound, fusion sEMG/ultrasound are 0.0869, 0.1280, and 0.0798. However, as for force estimation accuracy on Test-F, NRMSE of separate sEMG signals fell to 0.1321, and NRMSE of separate ultrasound signals and combined sEMG/ultrasound signals are 0.1445 and 0.0949. The force estimation accuracy on Test-F of combined sEMG/ultrasound signal outperforms that of separate sEMG signal. The declines of these three modalities are 0.0452 (52.01%), 0.0165 (12.89%), and 0.0151 (18.92%). It shows that the robustness of ultrasound and combined sEMG/ultrasound are better than that of separate sEMG.

Fig. 5.
figure 5

Typical demonstrations of force estimation result: (a) non-fatigue state with sEMG feature; (b) non-fatigue state with ultrasound feature; (c) non-fatigue state with fusion sEMG and ultrasound features; (d) fatigue state with sEMG feature; (e) fatigue state with ultrasound feature; (f) fatigue state with fusion sEMG and ultrasound features.

Full size image

Based on the analysis of the results in Fig. 4, it can be concluded that the force estimation precision of the sEMG signal under non-fatigue state is excellent, but once the muscle fatigue induced, the precision decreases significantly, which indicates that there is a strong correlation between sEMG signals and force, but sEMG signals are very sensitive to muscle fatigue. Although ultrasound signal is not as accurate as sEMG signal in force estimation, it is more robust to muscle fatigue than sEMG signal. As for combined sEMG/ultrasound signal, under non-fatigue condition, the force prediction accuracy of the combined signals is comparable to that of the sEMG signal, and when muscle fatigue occurred, the accuracy of the combined signal is significantly better than separate sEMG signal. These results demonstrate the superior performance of the combined sEMG/ultrasound signals in both force prediction accuracy and robustness against muscle fatigue. Figure 5 shows a typical force estimation result, from which sEMG modality’s sensitivity to muscle fatigue, fusion sEMG/ultrasound modality’s excellent precision and enhanced fatigue robustness can be clearly demonstrated.

4 Conclusion

In a conclusion, this paper validates that the force estimation accuracy of sEMG signal is superior, but sEMG signal is susceptible to muscle fatigue. Although the force estimation accuracy of the ultrasound signal cannot be compared with the sEMG signal, its robustness against muscle fatigue can make up for the short board of the sEMG signal. The fusion sEMG and ultrasound signals have the characteristics of high force prediction accuracy and excellent fatigue robustness.