Multiclassifier System Using Class and Interclass Competence of Base Classifiers Applied to the Recognition of Grasping Movements in the Control of Bioprosthetic Hand

Abstract

In this paper the problem of recognition of patient’s intent to move hand prosthesis is addressed. The proposed method is based on recognition of electromyographic (EMG) and mechanomyographic (MMG) biosignals using a multiclassifier (MC) system working with dynamic ensemble selection scheme and original concept of competence measure. The concept focuses on developing competence and interclass cross- competence measures which can be applied as a method for classifiers combination. The cross-competence measure allows an ensemble to harness information obtained from incompetent classifiers instead of removing them from the ensemble. The performance of MC system with proposed competence measure was experimentally compared against six state-of-the-art classification methods using real data concerning the recognition of six types of grasping movements. The system developed achieved the highest classification accuracies demonstrating the potential of MC system for the control of bioprosthetic hand.

1 Introduction

Hands in a human life play a role not only of a skillful manipulator which allows grasping and manipulating a variety of objects, but also of the sensor in order to determine the type of object being touched. The loss of even a single hand significantly reduces the human activity. The people who have lost their hands are doomed to permanent care. Restoring to these people even a hand substitute makes their life less onerous. The hand transplantations are still in a medical experiment, mainly due to the necessity of immunesuppression [21]. An alternative is “cyborgization”, i.e. equipping the armless patient with the prosthetic hand. At present, the construction of a multi-joint anthropomorphic mechanical structure that can copy even very complicated movements of the human hand poses no problem. Also the motion control of such a structure to accomplish defined finger postures is well known. The basic problem lies however in controlling the movement of prosthetic hands so as to enable their users to grasp and manipulate objects dexterously [20].

At the decision level this control can be reduced to the recognition of the patient’s intent on the basis of biosignals coming from the patient’s body. Electrical potentials accompanying skeleton muscles (called EMG signals) are an example of such biosignals. Through the tensing of these muscles, the disabled person may express his/her intentions as to the workings of the prosthesis [3, 9, 18, 24]. Nevertheless, reliable recognition of intended movement is a serious problem. A natural solution to overcome this difficulty and increase the efficiency of the recognition stage may be achieved through the following actions [16]:

• 1. by introducing the concept of simultaneous analysis of different types of biosignals which are the carrier of information about the performed hand movement – the fusion of electromyographic signals (EMG signals) and mechanomyographic signals (MMG signals) is considered in this study;

• 2. through improving the recognition method – authors propose to use the multiclassifier system with dynamic ensemble selection scheme [2].

Multiclassifier (MC) systems combine responses of a set of base classifiers. For the classifier combination two main approaches used are classifiers fusion and classifiers selection [13]. In the first method, all classifiers in the ensemble contribute to the decision of the MC system, e.g. through sum or majority voting. In the second approach, a single classifier is selected from the ensemble and its decision is treated as the decision of the MC system. The selection of classifiers can be either static or dynamic. In the static selection scheme a classifier is selected for all test objects, whereas the dynamic classifier selection (DCS) approach explores the use of different classifiers for different test objects [2]. Recently, dynamic ensemble selection (DES) methods have been developed which first dynamically select an ensemble of classifiers from the entire set (pool) and then combine the selected classifiers by majority voting rule [12]. In this way a DES based system takes advantage of both selection and fusion approaches. In most of the methods, the base classifiers are selected from the pool on the basis of their individual accuracy measure called competence in a local region of the feature space. These methods differ in algorithms for determining classifier competence and ways of defining the local regions [2, 12, 13, 25, 26, 30]. Regardless of the interpretation, competence measure evaluates classifier ability to correct activity (correct classification) in a defined region.

In this paper a new method for calculating the competence of a classifier in the feature space is developed and applied to the classifying user’s intent of upper-limb prosthesis motion based on EMG and MMG biosignals. The proposed competence measure evaluates both the local class-dependent probabilities of correct classification and probabilities of interclass misclassification using concept of Randomized Reference Classifier [26] and a local fuzzy confusion matrix [23]. Such idea of cross-competence measure allows the ensemble to exploit even activity of incompetent classifiers instead of removing them from the ensemble.

The paper arrangement is as follows. Section 2 includes the concept of prosthesis control system based on the recognition of patient’s intention and provides an insight into biosignals acquisition procedure and the method of feature extraction. Section 3 presents the key recognition algorithm based on the multiclassifier system working in dynamic ensemble selection scheme with original concept of competence measure. The experiments conducted and the results with discussion are presented in Sect. 4. The paper is concluded in Sect. 5.

2 Bioprosthetic Hand Control System

The bioprosthesis control performed by recognizing patient’s intention involves three stages:

• 1. acquisition of signals;

• 2. reduction of dimensionality of their representation;

• 3. classification of signals.

As already mentioned, in this study the fusion of electromyography (EMG) signals and mechanomyography (MMG) signals is the basis for recognition of patient’s intent. Myopotentials (EMG signals) can be detected through the skin by means of surface electrodes located above selected muscles. EMG signals measured on skin are the superposition of electrical potentials generated by recruited motor units of contracting muscles [4, 10]. The MMG signals are mechanical vibrations propagating in the limb tissue as the muscle contracts. They have low frequency (up to 200 Hz) and small amplitude and can be registered as a “muscle sound” on the surface of the skin using microphones [11].

After the acquisition stage, the recorded signals have the form of strings of discrete samples. Their size is the product of measurement time and sampling frequency. For a typical motion, that gives a record of size between 3 and 5 thousand of samples (time of the order of 3–5 s, and the sampling of the order of 1 kHz). This “primary” representation of the signals hinders the effective classification and requires the reduction of dimensionality. This reduction leads to a representation in the form of a signal feature vector.

Former experimental research showed [14,15,16, 24] that the effective method as regards to the recognition error and the calculation costs in the biosignal analysis are the sequence of two techniques: autoregressive (AR) model and principal component analysis (PCA).

The AR model belongs to a group of linear prediction methods that attempt to predict an value $y_n$ of a time series of data $\{y_n\}$ based on the previous values $(y_{n-1}, y_{n-2},\ldots )$. Several estimators of AR coefficients are well known in the field of signal processing. In the experimental investigations we choose the Burg algorithm because of its many remarkable advantages (it does not apply window data, minimizes forward and backward prediction errors, gives high resolution for short data records, always produces a stable model) [22]. The Burg algorithm estimates the AR coefficients by fitting an auto-regressive linear prediction filter model of a given order to the signal.

Although as a classifier construction different methodological paradigms can be used, we suggest to use multiclassifier systems with the dynamic ensemble selection method using procedure of fusion/selection based on original competence measure. Details of the classification stage are presented in the next section.

3 Multiclassifier System

3.1 Preliminaries

In the multiclassifier (MC) system we assume that a set of trained classifiers $\varPsi =\{\psi _1, \psi _2, \ldots , \psi _L\}$ called base classifiers is given. A classifier $\psi _l$ is a function $\psi _l\;:\;\mathcal {X}\,\rightarrow \,\mathcal {M}$ from a feature space $\mathcal {X} \subseteq \mathcal {R}^d$ to a set of class labels $\mathcal {M}=\{1,2,\ldots ,M\}$. Classification is made according to the maximum rule

$$\begin{aligned} \psi _l(x)=i\Leftrightarrow d_{li}(x)=\max _{j\in \mathcal {M}}\ d_{lj}(x), \end{aligned}$$

(1)

where $[d_{l1}(x), d_{l2}(x), \ldots , d_{lM}(x)]$ is a vector of class supports (classifying functions) produced by $\psi _l$. Without loss of generality we assume that $d_{lj}(x)\ge 0$ and $\sum _j d_{lj}(x)=1$.

The ensemble $\varPsi$ is used for classification through a combination function which, for example, can select a single classifier or a subset of classifiers from the ensemble, it can be independent or dependent on the feature vector x (in the latter case the function is said to be dynamic), and it can be non-trainable or trainable [13].

In this paper, we propose MC systems which use a dynamic ensemble selection scheme and trainable combining methods based on a competence measure $c_{ij}(\psi _l|x) (i,j \in \mathcal {M})$ of each base classifier ($l=1,2,...,L$) evaluating the class-dependent competence (for $i=j$) and interclass (cross-) competence (for $i \ne j$) of classifier $\psi _l$ at a point $x \in \mathcal {X}$. For training methods of combining base classifiers it is assumed that a validation set

$$\begin{aligned} \mathcal {V}=\{(x_1,j_1), (x_2,j_2), \ldots ,(x_N,j_N)\}; \ \;x_k \in \mathcal {X},\ j_k \in \mathcal {M} \end{aligned}$$

(2)

containing pairs of feature vectors and their corresponding class labels is available. In the MC systems developed in this study, combining algorithm is implemented at the classification function (support) level which allows to determine supports provided by the MC system to different classes, as a weighted sum of supports of base classifiers, namely:

$$\begin{aligned} d_j(x)= \sum _{l=1}^L \sum _{i=1}^M d_{li}(x) c_{ij}(\psi _l|x), \ j \in \mathcal {M}. \end{aligned}$$

(3)

In the next section two methods for calculation of competences $c_{ij}(\psi _l|x)$ of base classifiers will be developed.

3.2 Competence Measures

A natural concept of competence measure $c_{ij}(\psi _l|x)$ is probability that object given by the feature vector x belonging to the j-th class is assigned by $\psi _l$ to the ith class [27], namely:

$$\begin{aligned} c_{ij}(\psi _l|x) = P_l(i|j,x). \end{aligned}$$

(4)

In other words, probabilities (4) denote class-dependent probabilities of correct classification (for $i =j$) and misclassification (for $i\ne j$). A high value of class-dependent competence $c_{ii}(\psi _l|x)$ denotes that classifier $\psi _l$ is capable of providing the correct classification of objects from the ith class, whereas the high value of cross-competence $c_{ij}(\psi _l|x)$ clearly shows that the investigated classifier tends to misclassify objects from the jth class to the ith class. In the proposed method the above mentioned indicators can be utilized to correct the response of a classifier that tends to commit systematic errors.

Unfortunately, for deterministic base classifiers $\psi _l$ probabilities (4) are equal to 0 or 1, unlike the randomized classifiers for which these probabilities belong to the interval [0, 1] [1]. We do not accept, however, impractical assumption that base classifiers assign labels under a stochastic scheme because all classifiers used in real examples operate in a deterministic manner. For this reason, a direct approach to calculating probabilities (4) is not used in this study. Instead, indirect methods for solving this problem and fully utilizing the combining model (3) are applied. In the first approach classifier $\psi _l$ is modeled by the equivalent randomized reference classifier (RRC). In the second approach we will use a local confusion matrix which is built from the validation objects creating fuzzy neighborhood of point x. Details are described in the next two subsections.

The Method Using Randomized Reference Classifier (MC1). The proposed method of evaluation of the probabilities (4) is based on the original concept of a hypothetical classifier called Randomized Reference Classifier (RRC). The RRC, originally introduced in [26], is a stochastic classifier defined using a probability distribution over the set of class labels $\mathcal {M}$. The RRC uses the maximum rule (1) and a vector of class supports $[\delta _{l1}(x), \delta _{l2}(x), \ldots , \delta _{lM}(x)]$ for the classification of object x, where the j-th support is a realization of a random variable (rv) $\varDelta _{lj}(x)$. The probability distributions of the rvs are chosen in such a way that the following conditions are satisfied:

$$\begin{aligned} \varDelta _{lj}(x) \in [0, 1],\ \ \sum _{j \in \mathcal {M}} \varDelta _{lj}(x)=1, \end{aligned}$$

(5)

$$\begin{aligned} E[\varDelta _{lj}(x)]=d_{lj}(x), \ j=1,2,\ldots , M, \end{aligned}$$

(6)

where E is the expected value operator. From the above definition it follows that RRC can be considered as equivalent to the classifier $\psi _l$ for the feature vector x since it produces, on average, the same vector of class supports as the modeled base classifier $\psi _l$.

Since the RRC performs classification in a stochastic manner, it is possible to calculate the probability of classifying a validation object $x_k$ to the i-th class:

$$\begin{aligned} P^{(RRC)}_l(i|j_k,x_k)=Pr[\forall _{m=1,\ldots ,M, \ k\ne i} \ \varDelta _{li}(x_k)> \varDelta _{lm}(x_k)]. \end{aligned}$$

(7)

The formula (7) denotes class-dependent probability of correct classification (for $i=j_k$) or misclassification (for $i \ne j_k$) of RRC classifier $\psi ^{(RRC)}_l$ at a validation point $x_k$.

The key element in the modeling presented above is the choice of probability distributions for rvs $\varDelta _{lj}(x), j \in \mathcal {M}$ so that the conditions (5)–(6) are satisfied. In this study, the beta distribution is selected – the justification of such a choice can be found in [26] and furthermore the MATLAB code for calculating probabilities (7) was developed and it is freely available for download [28].

Since the RRC can be considered equivalent to the modeled base classifier $\psi _l$, it is justified to use the probability (7) as the competence $c_{ij_k}(\psi _l|x_k)$ of the classifier $\psi _l$ at the validation point $x_k \in \mathcal {V}$, i.e.

$$\begin{aligned} c_{ij_k}(\psi _l|x_k) \approx P^{(RRC)}_l(i|j_k,x_k). \end{aligned}$$

(8)

The competence values for the validation objects $x_k \in \mathcal {V}$ can be then extended to the entire feature space $\mathcal {X}$. To this purpose the following normalized Gaussian potential function model was used [26]:

$$\begin{aligned} c_{ij}(\psi _l|x)=\frac{\sum _{x_k \in \mathcal {V}: j_k=j} c_{ij_k}(\psi _l|x_k) exp(-dist(x,x_k)^2)}{\sum _{x_k \in \mathcal {V}: j_k=j} exp(-dist(x,x_k)^2)}, \end{aligned}$$

(9)

where $dist(x,x_k)$ is the Euclidean distance between x and $x_k$.

The Method Using Local Fuzzy Confusion Matrix (MC2). In the second approach we will use confusion matrix for evaluation of probability (4). A confusion matrix gives the complete picture of correct and incorrect classification made by classifiers $\psi _l$ for separate classes [6]. The rows (columns) correspond to the true classes (results of classification made by classifier $\psi _l$), as shown in Table 1.

Table 1. The multiclass confusion matrix of classifier $\psi _l$.

The value $\varepsilon _{ij}^{(\psi _l)}$ is determined from validation set (2) as the following ratio ($|\cdot |$ is the cardinality of a set):

$$\begin{aligned} \varepsilon _{ij}^{(\psi _l)} = \frac{|\mathcal {V}_j \cap \mathcal {D}_i^{\psi _l}|}{|\mathcal {V}_j|}, \end{aligned}$$

(10)

where $\mathcal {V}_j = \{x_k \in \mathcal {V} : j_k=j\}$ denotes the set of validation objects from the jth class and $\mathcal {D}_i^{\psi _l} = \{x_k \in \mathcal {V} : \psi _l(x_k)=i\}$ is the set of validation objects assigned by $\psi _l$ to the ith class.

Since we want to estimate probabilities $P_l(i|j, x)$ at a point x, values of confusion matrix $\varepsilon _{ij}^{(\psi _l)}(x)$ should be calculated on the base of local (for x) validation objects. A typical method is to define a neighborhood of an object x and only validation objects belonging to this neighborhood are used to calculate $\varepsilon _{ij}^{(\psi _l)}(x)$. Such an approach, however, has a major drawback: the method is very sensitive to the size of the neighborhood. As the neighborhood size increases, the sense of “locality” concept decreases, and as this size decreases, the risk that $\varepsilon _{ij}^{(\psi _l)}(x)=0$ increases. In order to avoid this problem, we define validation objects creating the neighborhood of the point x as a fuzzy set:

$$\begin{aligned} \mathcal {V}(x) = \left\{ (x^{(k)},\mu _{\mathcal {V}(x)}(x^{(k)}) ): x^{(k)} \in \mathcal {V} \right\} , \end{aligned}$$

(11)

whose membership function is equal to 1 for $x^{(k)}=x$ and decreases with increasing the distance between $x^{(k)}$ and x. In the further experimental investigations, the Gaussian membership function was applied:

$$\begin{aligned} \mu _{\mathcal {V}(x)}(x^{(k)})= c \, \exp ({-dist(x,x^{(k)})^2}), \end{aligned}$$

(12)

where $dist(x,x^{(k)})$ is the Euclidean distance and c denotes normalizing coefficient.

From (10), (11) and (12) directly results the formula for determining values of local confusion matrix:

$$\begin{aligned} \varepsilon _{ij}^{(\psi _l)}(x) = \frac{|\mathcal {V}_j \cap \mathcal {D}_i^{\psi _l} \cap \mathcal {V}(x)|}{|\mathcal {V}_j \cap \mathcal {V}(x)|}, \end{aligned}$$

(13)

where $|\cdot |$ is the cardinality of a fuzzy set [17] and $\mathcal {V}_j$ and $\mathcal {D}_i^{\psi _l}$ are treated as fuzzy sets defined in (10) with membership function equal to 1. Finally, normalizing (13) we get estimation (4):

$$\begin{aligned} c_{ij}(\psi _l|x) = \frac{\varepsilon _{ij}^{(\psi _l)}(x)}{\sum _{j \in \mathcal {M}} \varepsilon _{ij}^{(\psi _l)}(x)}. \end{aligned}$$

(14)

4 Experimental Investigations

4.1 Experimental Setup

Performance of the MC systems developed was evaluated in experiments using real data. The experiments were conducted in the Matlab environment using PRTools 4.1 and Signal Processing Toolbox.

In the recognition process of the grasping movements, 6 types of grips (tripoid, pinch, power, hook, column and mouse grip) were considered. Our choice is deliberate one and results from the fact that the control functions of simple bioprosthesis are hand closing/opening and wrist pronantion/supination, however for the dexterous hand these functions differ depending on grasped object [3] (Fig. 1).

Fig. 1.

Types of grips.

Fig. 2.

The layout of the integrated sensors (EMG electrodes and MMG microphones) on the underside (A) and top side (B) of the forearm. Examples of EMG and MMG signals from channel 2.

The experiments were carried out on healthy persons. Biosignals were registered using 8 integrated sensors (containing EMG electrode and MMG microphone in one casing) located on a forearm (vide Fig. 2) and specially designed 16-channel measuring circuit with sampling frequency 1 kHz. For further processing the following sensors (channels) located above the most active muscles during grasping movements were selected [4]: 1 (sensors located above pronator quadratus muscle), 2 (flexor digitorium supercialis), 3 (flexor digitorium profundus), 5 (extensor pollicis brevis) and 8 (supinator).

The dataset used to test of proposed classification method consisted of 600 measurements, i.e. pairs EMG and MMG signals segment/movement class. Each segment lasted 6 s and was preceded with a 10 s break. The coefficients of AR function for different order of AR model ($p=20, 50, 80$ per signal and per channel) were considered as the primary feature vector. Next, primary features were subjected to the PCA feature extraction procedure with the number of PC’s determined by 95% of the total variation rule. The training and testing sets were extracted from each dataset using two-fold cross-validation. For combining the MC system, a two-fold stacked generalization method [29] was used. Three experiments were performed which differ in the biosignals used for classification (EMG signals, MMG signals, both EMG and MMG signals).

The experiments were conducted using heterogeneous ensemble with the following ten base classifiers [8]: (1–2) linear (quadratic) classifier based on normal distributions with the same (different) covariance matrix for each class, (3) nearest mean classifier, (4–6) k-nearest neighbors classifiers with $k=1,5,15$, (7) naive Bayes classifier (8) decision-tree classifier with Gini splitting criterion, (9–10) feed-forward back-propagation neural network with 1 hidden layer (with 2 hidden layers).

In the experiment MC1 and MC2 systems were compared against six state-of-the-art multiclassifier systems: (1) The single best (SB) classifier in the ensemble [13]; (2) Majority voting (MV) of all classifiers in the ensemble [13]; (3) Dynamic classifier selection – local accuracy (LA) method [30]; (4) Dynamic ensemble selection – KNORA method (KE) [12]; (5) Randomized reference classifier (RRC) method [26, 27]; (6) Multiclassifier with fuzzy inference system (MCF) [15].

4.2 Results and Discussion

Classification accuracies (i.e. the percentage of correctly classified objects) for methods tested in the experiments are listed in Table 2. The accuracies are average values obtained over 10 runs (5 replications of two-fold cross validation). Statistical differences between the performances of MC1 and MC2 systems and the six multiclassifier systems were evaluated using 5x2cv F test [7]. The level of p < 0.05 was considered statistically significant. In Table 2, statistically significant differences are given as upper indices of the method evaluated, e.g. for the dataset with p = 20 and EMG signals the MC1 system produced statistically better classification accuracies from SB, MV and LA methods.

Table 2. Classification accuracies of classifiers compared in the experiment (description in the text). The best score for each dataset is highlighted. (p denotes the order of AR model).

Fig. 3.

Average ranks of multiclassifier systems. Thick interval is the critical rank difference (2.686) calculated using the Bonferroni – Dunn test ($p < 0.05$).

Statistical differences in rank between multiclassifier systems were obtained using Friedman test with Iman and Davenport correction combined with a post hoc Holm stepdown procedure [5]. The average ranks and a critical rank difference calculated using a Bonferroni – Dunn test [5] are visualised in Fig. 3. The level of $p < 0.05$ was considered as statistically significant.

These results imply the following conclusions:

• 1. The MC1 and MC2 systems produced statistically significant higher scores in 60 out of 108 cases (9 datasets $\times$ 6 classifiers compared $\times$ 2 MC systems);

• 2. The MC1 (MC2) classifier:

  • for EMG signals outperformed, on average, the SB, MV, LA, KE, RRC and MCF systems by 2.9%, 3.6%, 4.4%, 1.2%, 0.8% and 2.6% (by 2.8%, 3.5%, 4.3%, 1.1%, 0.7% and 2.5%), respectively;

  • for MMG signals outperformed, on average, the SB, MV, LA, KE, RRC and MCF systems by 5.6%, 5.4% and 5.3%, 0.3%, 1.4% and 3.0% (by 5.7%, 5.5% and 5.4%, 0.4%, 1.5% and 3.1%), respectively;

  • for EMG and MMG signals outperformed, on average, the SB, MV, LA, KE, RRC and MCF systems by 3.6%, 2.6%, 3.0%, 0.4%, 0.9% and 0.7% (by 4.3%, 3.3%, 3.7%, 1.1%, 1.6% and 1.4%), respectively;

• 3. MC1 and MC2 methods have statistically higher average rank than MCF, MV, LA and SB methods;

• 4. The multiclassifier systems using both EMG and MMG signals achieved the highest classification accuracy for all datasets;

• 5. When the order of AR model increases then the accuracy of all methods investigated also increases.

5 Conclusion

The classic methods of analysis of biosignals in the bioprostheses control systems are widely discussed in the literature [10, 11, 19, 24]. However, the classification stage still poses a challenge for researching new solutions enabling the reliable recognition of human intention. In this study a novel method for recognition of grasping movements is proposed. The method, combining base classifiers into multiclassifier system and taking into account the class and interclass competence of base classifiers, brings new possibilities to biosignal analysis. Results obtained in experimental investigations imply that it is worth trying solution that improves recognition efficiency.

The introduced approach constitutes the general concept of the human-machine interface, that can be applied for the control of a dexterous hand and an agile wheelchair as well as other types of prostheses, exoskeletons, etc. This, however, requires a further study, mainly in the experimental phase, which would allow to assess and verify the effectiveness of the adopted concept.