Keywords

1 Introduction

Individuals with upper extremity amputations can be equipped with bionic hands, which are conventionally controlled using sEMG signals acquired from the users’ residual limbs (Ravindra and Castellini 2014; Atzori and Müller 2015). Although available bionic hands have multiple Degrees of Freedom (DOFs) and offer high levels of dexterity (Atzori and Müller 2015), EMG-driven hands, have as high as a 75% rejection rate by the end-users (Biddiss and Chau 2007) because one of the reasons is that the control is based on non-intuitive series of muscle contractions and requires an extensive training period with experts (Castellini et al. 2014). In order to reduce this rejection rate, individuals with amputations need to be equipped with a control interface that is intuitive and reliable, and allows the users to dexterously and naturally control their bionic hands.

This study aims to investigate the feasibility of increasing robustness of natural control of upper extremity prosthesis by using FMG as a synergist to the conventional sEMG. This was accomplished by embedding multiple FSRs and two EMG electrodes in a custom prosthetic socket. The intended upper-extremity residual limb motions were captured using the FMG and sEMG sensors to distinguish between different grip patterns using pattern recognition, and then to control a commercially available robotic prosthetic hand, Bebionic3 hand by the Steeper Group UK. These advanced control strategies would potentially increase the adoption of the robotic prosthetic hand and thus provide a better quality of life to amputee users.

2 Background

When a limb is replaced with prosthesis, the user and the prosthesis start an intimate relationship in which human-machine interface is a fundamental component. Unfortunately, the link between the user and their prosthesis is the weakest link in the chain. The conventional non-invasive HMI is based on surface-Electromyography (Atzori and Müller 2015a).

2.1 Surface Electromyography

sEMG sensors detect the electrical potentials created by the motor units from the human skin surface of skeletal muscles. These sEMG signals have a wide variety of uses such as identifying neuromuscular or motor control diseases to various kinesiology studies.

EMG sensors have been used to control robotic prostheses since the 1950s (Wirta et al. 1963). Most commercially available prostheses use control strategies based upon open-close state detected by two sEMG sensors. This control strategy is robust for commercial myoelectric two-configuration grippers. Unfortunately for advanced robotic prostheses this control strategy not only limits the DOFs but also requires non-intuitive Morse code like sequences to obtain multiple DOFs, which requires an extra mental interpretation between the intended gesture to the open-close control command (Yang et al. 2014).

The research community has addressed this problem extensively by investigating Pattern Recognition (PR) based surface-electromyography (sEMG) control (Castellini et al. 2014). This control strategy has been recently commercialized however has yet to show clinical adoption (Atzori and Müller 2015). One of the reasons identified in the literature is due to the sEMG signals that are affected by sweating, electrode shift, ambient noise, fatigue, cross-talk between adjacent muscles, signal drifting, and force level variation (Merletti et al. 2010; Atzori et al. 2014; Al-Timemy et al. 2015). Therefore several alternative sensing techniques including pressure sensors have recently been explored (Fang et al. 2015). Ravindra and et al. compared three non-invasive HMIs (sEMG, ultrasound imaging, pressure sensing) and concluded that pressure sensing represented a valid alternative/augmentation to sEMG because of its potential to provide the highest PR prediction accuracy, signal stability over time, wearability, simplicity in socket embedding, and affordability of cost.

2.2 Pressure Sensing Techniques

The use of pressure sensors as a sensing technique has been termed in various ways in the literature such as: residual kinetic imaging (Phillips and Craelius 2005), pressure distribution map (Radmand et al. 2014a), topographical force map (Ravindra and Castellini 2014), force myography, and muscle pressure mapping (Wininger et al. 2008; Li et al. 2012; Cho et al. 2016).

It is a technique involving the use of different types of pressure sensors such as Force Sensitive Resistors (FSRs) on the surface of the limb to detect the volumetric changes in the underlying musculo-tendonous complex. Each term shares the same fundamental principle and are described in the following section.

3 Materials and Methods

3.1 Subject

A participant with a transradial amputation, was recruited via Barber Prosthetics Clinic (BPC) to help assessing the feasibility of the control technology developed in this study. The participant acquired his transradial amputation in 1980 and has used a body-powered prosthesis since. Although significant time has passed since the amputation, the test pilot is extremely active and has ample sensation and control in his residual limb. All the methods within this study were in compliance with the declaration of Helsinki and were approved by the Simon Fraser University (SFU) Office of Research Ethics (#2015s0143).

3.2 Prosthetic Prototype

A conventional socket design with a locking liner was fabricated by Barber Prosthetics Clinic (BPC) and connected to the BeBionic3 robotic hand to be donned by the participant with a transradial amputation. The custom socket is comprised of a transparent thermoplastic inner and outer socket. The outer socket houses an opening for easy accessibility of the internal electronics, cut-outs for the sEMG sensors, slits for the FSR bands and is covered using a 3D printed enclosure. The setup is shown in Fig. 1.

Fig. 1.
figure 1

Prosthesis with 3D printed housing - distal end (Left), proximal end (Right)

Full size image

3.3 FSR Bands

Five FSR Bands were used in total. FSR bands were custom-printed and each contains 16 FSR sensors (individual sensors have same properties as FSR 402 from Interlink Electronics). Four bands were placed linearly from the proximal to the distal of the inner socket and the fifth band was placed circumstantially. This provided a total of 48 active FSR sensors whereby active is defined as in contact with the residual limb, the non-active sensors were not sampled.

3.4 EMG Sensors

OttoBock 13E68 sEMG electrodes were used in the study which provide an amplified, bandpass-filtered and Root-Mean-Square (RMS) rectified version of the raw sEMG signal. The electrodes amplification gain was set at about 14,000. Two extrusions were created in the socket such that the sEMG sensors would receive the highest muscle activity, and highest signal extraction, from the extensor and flexor muscle bellies on the residual limb. Thus, optimizing EMG positioning over FMG positioning.

3.5 Data Acquisition Platform

The 3D printed enclosure contained a BeagleBone Black (BBB) Rev C development board with a custom made cape/shield. The sEMG sensors were directly connected to the cape while the FSR bands were interfaced with vertical zero-insertion force (ZIF) connectors. Since the frequency of human hand motion is typically less than 4.5 Hz (Xiong and Quek 2006; Amft et al. 2006), a 10 Hz sampling frequency was deemed sufficient. Each FSR band was supplied through a BBB general purpose input\output (GPIO) with 3.3 volts. Bands’ outputs were connected to voltage dividers using 5 kΩ and 6 kΩ resistors and individual sensors on a band were sampled individually (in series). The network used is similar to that of Rasouli et al.

4 Experimental Protocol

4.1 Protocol

It has been determined that most functional activities of daily living can be performed through four primary grips: power (also known as force), tripod, finger point, and the key grips (Peerdeman et al. 2011; Yang et al. 2014). In this study three different grip sets are examined. First, the six grips containing the four primary grips above and also relax hand open, and open palm grips. The second set contains only the opposed-thumb primary grips: relax hand open, open palm, power, and tripod. Finally, the third set contained the non-opposed thumb primary grips: relax hand open, open palm, finger point, and key grip. The opposed and non-opposed grip sets were examined because they match the grip patterns made available by the Bebioinc3 opposed and non-opposed hand configurations.

The experiment included three trials, each having three repetitions. Each repetition included 11 grip patterns performed in a stationary position with the pilot’s elbow off the table, and a 5-min rest period between repetitions. The 11 grip patterns used in this study are shown in Fig. 2.

Fig. 2.
figure 2

11 grip patterns. Grip patterns 3 to 6 are considered the primary grip patterns (Peerdeman et al. 2011; Yang et al. 2014). Grip patterns 1 to 4 constitute the opposed-thumb primary grips.

Full size image

The FSR and sEMG sensors were sampled at 10 Hz and 100 samples, respectively. For each grip pattern the test pilot first created the bilateral mirrored contraction for the desired grip pattern and once he felt it was stable he initiated data acquisition by pressing a button on the user interface (UI) with his sound limb. Each recording lasted 10 s.

4.1.1 Trial 1: Without Robotic Prosthetic Hand

In Trial 1, three repetitions were performed without connecting the prosthesis to the BeBionic3 robotic hand. This trial served two purposes: to investigate the effects the weight of the hand has on classification accuracy and to increase the pilot’s residual limb volume due to muscle fatigue.

4.1.2 Trial 2: Complete System

In Trial 2, three repetitions were performed with the prosthesis connected to the BeBionic3 robotic hand. These trials closely resemble a real-life scenario.

4.1.3 Trial 3: EMG Threshold Force Levels

In Trial 3, three repetitions were performed with the complete system. Various force levels were used in different repetitions to establish the low, medium, and high force levels from the sEMG sensor values to investigate using EMG force level to trigger FMG pattern recognition analysis. In Trial 3 Repetition 1, for the low force (41.6% of maximal voluntary contraction (MVC)), the pilot provided the minimal amount of force he felt required to create the desired grip pattern with his residual limb. In Trial 3 Repetition 2, the medium force (64.5% of MVC), the pilot created each grip pattern with his residual limb using the normal amount of force he would use with his sound limb. In Trial 3 Repetition 3, the high force, the pilot used his maximum strength when creating each grip pattern with his residual limb. Since each grip pattern in Trial 3 Repetition 3 was performed with maximal force, it was used as the MVC level.

A collected sample has the structure (FSR1, ..., FSR48, EMG1, EMG2). The data from the last trial was used to create 3 different force thresholds for data filtering. For each repetition, the mean EMG value was obtained from both EMG sensors during each of the 11 grip patterns. Once the 3 thresholds were created, it could be used to filter the raw data of Trials 1 and 2 to remove the FSR bias values associated with an EMG value below the set threshold. The filtered data could then be classified to investigate the effects of EMG threshold filtering on classification accuracy.

4.2 Machine Learning Algorithm

To classify the grip patterns, a Linear Discriminant Analysis (LDA) model was used offline. LDA is a robust machine learning algorithm that finds a linear combination of features to separate two or more classes of objects. Additionally, LDA has been identified as to have real-time applicability and provides better or equal classification results when compared to more complex algorithms (E. Scheme and Englehart 2011; Scheme et al. 2013; Zhang et al. 2013; Amsuss et al. 2014). For Trials 1 and 2, Leave-One-Out Cross-Validation (LOOCV) method was used to obtain the classification accuracy. To implement LOOCV for each trial, the first repetition was used as the test data set against the second and third repetitions which were pooled and used as the training data set. The data sets were fed into the LDA model and the classification accuracy was recorded. This process was repeated to test the accuracies of repetition 2 then 3. The average of the three classification accuracies were then averaged as the classification accuracy of the trial.

5 Experimental Results

5.1 Single FSR Bands

Data from a single FSR band was used to classify 4 and 11 grip patterns, the resulting classification accuracies are shown in Table 1. The average single band classification accuracy for 4 (opposed) grip patterns was 63.8% and 42.8% without and with the robotic hand respectively. The average single band classification accuracy for 11 grip patterns was 32.0% and 25.7% without and with the robotic hand respectively.

Table 1. Single FSR band classification accuracy
Full size table

5.2 Multiple FSR Bands

In trial 2, data from a different number of FSR bands was used to classify 4 and 11 grip patterns. The average accuracy resulting from using all permutations of different number of bands can be seen in Table 2.

Table 2. Multiple FSR band classification accuracy. The variance was calculated considering all the possible permutations of the number of active bands (e.g. [write the simplest example])
Full size table

5.3 EMG Features

For these results the values from the two sEMG sensors were treated as additional features and they were included with all 5 FSR bands in Trial 2 data for classification. The results for the accuracy with and without EMGs and with and without EMG filtering for the different grip pattern groups can be seen in Table 3. In addition, confusion matrices were created for the 4 grips (non-opposed and opposed) to understand the increase in classification accuracy. These can be seen in Fig. 3 where the x axis is the actual class and the y axis is the machine learning algorithm prediction.

Table 3. Classification Accuracy of 5 FSR Bands without EMG, with EMG as Features, and with EMG as Features and Medium Threshold Filtration
Full size table
Fig. 3.
figure 3

5 FSR band 4 grip opposed and non-opposed with and without EMG as features confusion matrix; opposed w/o EMG (top left), opposed w/EMG (top right), non-opposed w/o EMG (bottom left), and non-opposed w/EMG (bottom right); (1 - relax hand open, 2 - open palm grip, 3 – power Grip, 4 – tripod grip, 5 – finger point, and 6 – column grip

Full size image

5.4 EMG Filtration

Trial 3 created 3 sets of grip pattern specific threshold levels to filter the data of Trial 2 in order to increase classification accuracies. During all three trials the average EMG values were recorded for both EMG sensors per grip pattern. For low, medium and high filtration, several adaptations of the thresholds were applied such as using: only EMG 1 threshold, only EMG 2 threshold, EMG 1 or EMG 2 threshold, and EMG 1 and EMG 2 thresholds. It was found that a medium level threshold, assessed as in Sect. 4.1.3, for both EMG sensors was optimal. The accuracy results from classification using all 5 FSR bands, both EMG sensors as features and as a filter can be seen in Table 3.

6 Discussion

6.1 Single FSR Band

The FSR single analysis shows the validity of the positioning for the 5 FSR bands. Looking at Table 1, we can see that the classification accuracy does not differ significantly with the standard deviation ranging between 5.3% to 9.7% depending on the configuration. The classification accuracy differences between the bands can be attributed to the placement and the number of active FSR sensors per band. In addition, the comparatively lower accuracy results for FSR Band # 1 can also be attributed to the specific style of socket chosen. Band #1 is positioned between the flexor and extensor on the exterior surface and thus when the elbow was bent into the stationary position a space was formed between the liner and the socket near the proximal end of the socket.

The testing configuration whether the pilot had the robotic prosthetic hand attached or not alters the classification accuracy. For any single band a drop in classification accuracy was observed when the pilot wore the robotic prosthetic hand. The classification accuracy dropped 21.0% and 6.2% for 4 grips and 11 grips respectively. The accuracy drops are due to the weight of the robotic prosthetic hand on the distal end of the socket that causes a torque about the socket’s center of mass. The torque applied pre-loads the sensors at the exterior of the proximal end and the anterior of the distal end. The opposite occurs where a gap is produced at the anterior of the proximal end and the exterior of the distal end. Although the robotic prosthetic hand reduces classification accuracy, the paper continues to focus on Trial 2 (with the robotic hand) as they are most similar to real life use for the participant.

6.2 Multiple FSR Bands

For 4 or 11 grips grip pattern classifications, there is an increased accuracy with the addition of another FSR band. The results from Table 2 show the average classification accuracy of all permutations. On average, each additional FSR band increased the classification accuracy by 8.4% and 6.2% for 4 and 11 grip patterns respectively. As a result of using all 5 FSR bands with 48 sensors the classification accuracy nearly doubled from 42.8% to 76.2% and 25.8% to 50.4% for 4 and 11 grip patterns respectively. The remainder of this discussion will mainly focus on the four primary grips as they are also the first four grips of the BeBionic3 robotic hand.

6.3 EMG Features

The two EMG sensors’ data were added to the 48 FSR sensors’ data and used as additional features to increase classification accuracy. From Table 3 we see that the classification accuracy for 4 grip (opposed) increased from 76.2% to 81.9% due to the addition of the EMG sensors as features. Investigating the confusion matrices in Fig. 3, we see a significant reduction in misclassification between grips 3 and 4, force and tripod respectively. From Fig. 3, it can be found that the grip pattern classification accuracy increased from 79.3% to 94.3% and 65.7% to 73.3% for force and tripod grips respectively. By adding the EMG sensors as additional features the increased classification accuracy of 81.9% was achieved for the four chosen classes.

6.4 EMG Filtration

It was found that using both EMG 1 and EMG 2 values for medium thresholds increased the 4 grips (non-opposed) classification accuracy using 5 FSR bands and both EMGs as input features from 81.9% to 93.5%. In comparison, the 11 grips with 5 FSR bands and two EMGs as input features decreased slightly: from 50.6% to 47.7%. In both cases, the data set size was reduced by an average of 79.0% with the medium threshold level. Even with the smaller data set, the 11 grips classification accuracy dropped only by 2.9% yet the 4 grips accuracy rose 11.6%.

7 Conclusions

For this study, a prosthetic socket was produced for our pilot with a transradial amputation. Five customized FSR bands were embedded in the socket providing 48 sensors that were used to create a pressure map in addition to two sEMG sensors. 11 different grip patterns were examined and classified offline using LDA machine-learning algorithm. The average accuracies using single FSR bands were 42.8% and 25.8% for 4 (opposed) and 11 grips respectively. By using all 5 FSR bands the accuracy rose from 42.8% to 76.2% and 25.8% to 50.4% for 4 (opposed) and 11 grips respectively. Using both sEMG sensors as additional features to the 5 FSR bands, accuracy changed from 76.2% to 81.9%, 60.3% to 60.0%, 62.4% to 63.3%, and 50.4% to 50.6% for 4 (opposed), 4 (non-opposed), 6, and 11 grips respectively. Finally, by applying data filtration of the previous configuration using EMG 1 and EMG 2 medium level thresholds, the accuracy changed from 81.9% to 93.5%, 60.0% to 93.3%, 63.3% to 63.4%, and 50.6% to 47.7% for 4 (opposed), 4 (non-opposed), 6, and 11 grips respectively. The results indicate the potential benefit of a multi-sensor approach for controlling robotic prostheses.