Keywords

1 Introduction

The importance of hands in human life cannot be estimated. The loss of even a single hand significantly reduces the human activity. The people who have lost their hands are doomed to permanent care. Restoring to these people even a hand substitute makes their life less onerous. The hand transplantations are still in a medical experiment, mainly due to the necessity of immunosuppression [17]. An alternative is to equip these people with cybernetics prostheses.

The activity of human organism is reflected in characteristic biosignals, which can be measured and next can be applied to the control of the work of technical devices. Electrical potentials accompanying skeleton muscles (called EMG signals) are an example of such biosignals. Through the tensing of these muscles, the disabled person may express his/her intentions as to the workings of the prosthesis [1, 2, 5, 15, 18, 22]. Nevertheless, reliable recognition of intended movement using only the EMG signals analysis is a hard problem hence any attempt to obtain better classification methods and algorithms is fully justified.

According to the author’s recent experience [11, 13, 18, 19], increasing the efficiency of the recognition stage may be achieved through the following activities:

  1. 1.

    by introducing the concept of simultaneous analysis of two different types of biosignals, which are the carrier of information about the performed hand movement—the EMG and mechanomiographic (MMG) signals;

  2. 2.

    by using sequential classification scheme which is based on decomposition of hand movement on a sequence of elementary actions with Markov model;

  3. 3.

    through the use of multiclassifier system with base classifiers dedicated to the particular steps of sequential recognition procedure.

The bioprosthesis control system developed in this study includes the above mentioned ideas within a common concept in contrast to the earlier author’s works where above suggestions were considered separately. Taking into account above ideas, the paper aims to solve the problem of recognition of the patient’s intention to move the multiarticulated prosthetic hand during grasping and manipulating objects in a skillful manner, by measuring and analyzing multimodal signals coming from patient’s body. The adopted solution takes into consideration the advantages given by the fusion of the EMG and MMG signals in the original sequential MC system based on the Bayes paradigm and Markov model of dependences among elementary actions.

In the proposed MC system new method of dynamic fusion of base classifiers is developed. The method is dedicated to the sequential recognition scheme with probabilistic model. This specificity of the MC system is visible through the pool of base classifiers which are associated with the particular stages of classification process and the trainable mechanism of fusion based on probabilistic properties of base classifiers.

The paper arrangement is as follows. Chapter 2 includes the concept of prosthesis control system based on the recognition of patient’s intention in the sequential scheme and provides an insight into steps of the whole decision control procedure. Chapter 3 presents the key sequential recognition algorithm based on the multiclassifier system with Markov model of trainable combining algorithm. The experiments conducted and the results with discussion are presented in Chap. 4. The paper is concluded in Chap. 5.

2 Bioprosthetic Hand Control System

As mentioned above, the bioprosthesis control is performed by recognizing its intended movement on the base of classification of EMG and MMG signals from user arm stump. This requires the development of three stages: (1) acquisition of signals; (2) reduction of dimensionality of their representation; (3) classification of biosignals (recognition of patient’s intention).

Biosignal acquisition and analysis processes influence essentially on the reliability of recognition of prosthesis motion control decisions. The acquisition process should take into account the nature of the measured signals and their measurement conditions [3]. For the needs of experimental research presented in Chap. 4 the special EMG/MMG biosignals measuring system was constructed. The system fully meets the above requirement, mainly due to the use of differential amplifiers, which eliminate interferences in EMG signals and special casing, which isolate the microphone from the external sound sources for MMG signals [14].

After the acquisition stage, the recorded signals have the form of strings of discrete samples. Their size is the product of measurement time and sampling frequency. For a typical motion action, that gives a record of size between 5 and 7 thousand of samples per channel (time of the order of 5–7 s, and 1 kHz sampling). This primary representation of the signals hinders the effective classification and requires the reduction of dimensionality. This reduction leads to a representation in the form of a signal feature vector. In this study, the sequence of autoregressive (AR) model and principal component analysis (PCA) is proposed as a feature extraction and reduction methods, respectively. Former experimental research showed, that both AR and PCA algorithms are effective methods in respect of the recognition error and the calculation costs in the biosignal analysis [7, 11].

In the considered control concept we assume that each prosthesis operation consists of specific sequence of elementary actions, and the patient’s intention means his will to perform a specific elementary action [18]. Thus, prosthesis control is a discrete process where at the nth stage (\(n=1,2,\ldots , N\)) occurs successively:

  1. 1.

    the measurement of EMG and MMG signal parameters (results of AR and PC analysis) \(x_n\) (\(x_n \in \mathcal {X} \subseteq \mathcal {R}^d\)), that represents patient’s will \(j_n\) (\(j_n \in \mathcal {M}=\{1,2,\ldots ,M\}\)) (the intention to take a particular action);

  2. 2.

    the recognition of this intention (the result of recognition at the nth stage will be denoted by \(i_n \in \mathcal {M}\));

  3. 3.

    the realization of an elementary action \(a_n \in \mathcal {A}\), uniquely defined as a recognized intention. This means that there is M number of elementary actions \(\mathcal {A}=\{a^{(1)}, a^{(2)},\ldots , a^{(M)}\}\)—an exemplary meaning of elementary actions in relation to a dexterous hand prosthesis is defined in Sect. 4.

The assumed character of control decisions (performing an elementary action) means that the task of bioprosthesis control is reduced to the recognition of the patient’s intent in successive stages on the basis of the available measurement information. Since the patient’s current intention depends on history, the specificity of the investigated classification task reveals in the form of input data, which are not associated only with the direct EMG and MMG signals parameters that manifest the current intention, but comprise up to an extend the historic information that regards the preceding course of control process. In the general case, we suppose that the decision algorithm at the nth instant takes into account the K-instant-backwards-dependence (\(K<n\)). It means, that decision at the nth instant is made on the base of vector of features

$$\begin{aligned} \bar{x}_n^{(K)}=(x_{n-K}, x_{n-K+1}, \ldots , x_{n-1}, x_n). \end{aligned}$$
(1)

In consequence, the classification algorithm at the nth instant is of the following form:

$$\begin{aligned} \Psi _n(\bar{x}_n^{(K)})=i_n, \ i_n \in \mathcal {M}. \end{aligned}$$
(2)
Fig. 1
figure 1

System of bioprosthesis control via sequential recognition of patient’s intentions

Full size image

Figure 1 shows the block diagram for the complete dynamic process of bio-prosthesis control in the explored sequential decision problem. In this study, multiclassifier systems will be applied as classifiers (2) for the particular instances of sequential recognition. In the proposed MC systems, both the pool of base classifiers and the combining mechanism will be constructed using the supervised learning procedure, what leads to the assumption that a learning set \(\mathcal {S}\) and a validation set \(\mathcal {V}\) are available [8]. In the considered sequential decision problem, the learning set \(\mathcal {S}\) consists of m training sequences:

$$\begin{aligned} \mathcal {S}=\{\mathcal {S}_1, \mathcal {S}_2, \ldots , \mathcal {S}_m\}, \end{aligned}$$
(3)

where a single sequence

$$\begin{aligned} \mathcal {S}_k=((x_{1,k}, j_{1,k}), (x_{2,k}, j_{2,k}),\ldots , (x_{N,k}, j_{N,k})) \end{aligned}$$
(4)

denotes a single-patient sequence of prosthesis activity that comprises N EMG and MMG signals observation instants, and the patient’s intentions.

Similarly, the validation set V consists of r validation sequences \(\mathcal {V}= \{\mathcal {V}_1, \mathcal {V}_2, \ldots , \mathcal {V}_r\}\) and a single sequence \(\mathcal {V}_k\) has the same form as in (4). The next section, describes the procedure of determining the original MC systems (2) using learning set \(\mathcal {S}\) and validation set \(\mathcal {V}\), in detail.

3 Multiclassifier System

3.1 Preliminaries

The proposed multiclassifier system is built as a combination of the two following probabilistic paradigms:

Markov Model. We will treat the sequential recognition task as a discrete dynamical process, in which the patient’s intents in successive stages \(j_1, j_2, \ldots , j_N\) are observed values of sequence of random variables \(\mathbf J _1, \mathbf J _2, \ldots , \mathbf J _N\) modeled by first-order Markov chain. The probabilistic formalism for such a dependence is given by the initial probabilities

$$\begin{aligned} p_{j_1}=P(\mathbf J _1=j_1) \end{aligned}$$
(5)

and by the transition probabilities

$$\begin{aligned} p_{j_n,j_{n-1}}=P(\mathbf J _n=j_n|\mathbf J _{n-1}=j_{n-1}). \end{aligned}$$
(6)

Meta Bayes Classifier. In the concept of Meta Bayes Classifier (MBC), which originally was introduced in [12] we suppose that a base classifier \(\psi \) is given, which maps feature space into a set of class numbers, viz.

$$\begin{aligned} \psi : \mathcal {X} \longrightarrow \mathcal {M}. \end{aligned}$$
(7)

The MBC \(\psi ^{MBC}\) constitutes the specific probabilistic generalization of base classifier (7) which has the form of the Bayes scheme built over the classifier \(\psi \). This means, that \(\psi ^{MBC}\) takes the decision according to the maximum a posteriori probability rule:

$$\begin{aligned} \psi ^{MBC}(\psi (x)=k)=i \longleftrightarrow P(i|\psi =k)=\max _{l \in \mathcal {M}} P(l|\psi =k). \end{aligned}$$
(8)

3.2 Fusion of Base Classifiers

Suppose first, that we have the set of N trained base classifiers:

$$\begin{aligned} \psi _1(x_1), \psi _2(x_2), \ldots , \psi _N(x_N), \end{aligned}$$
(9)

which classify the patient’s intents at the 1st, 2nd, ..., Nth instant, respectively.

The MC system (2) for nth instant is defined as the MBC classifier (8) constructed over the set of base classifiers (9) for nth, \((n-1)\)th, ..., \((n-K)\)th instants, namely:

$$\begin{aligned} \Psi _n(\bar{x}_n^{(K)})=\psi ^{MBC}(\psi _{n-K}(x_{n-K})=i^{'}_{n-K}, \ldots , \psi _{n-1}(x_{n-1})=i^{'}_{n-1}, \psi _n(x_n)=i^{'}_n). \end{aligned}$$
(10)

The MC system (10) produces the decision about the patient’s intent at the nth instant according to the generalized rule (8):

$$\begin{aligned} \nonumber \Psi _n(\bar{x}_n^{(K)})=i_n \longleftrightarrow P(i_n|\psi _{n-K}(x_{n-K})=i^{'}_{n-K}, \ldots ,\psi _n(x_n)=i^{'}_n)= \end{aligned}$$
$$\begin{aligned} =\max _{l \in \mathcal {M}}P(l|\psi _{n-K}(x_{n-K})=i^{'}_{n-K}, \ldots ,\psi _n(x_n)=i^{'}_n), \end{aligned}$$
(11)

where:

$$\begin{aligned} P(i_n|\psi _{n-K}=i^{'}_{n-K},\ldots ,\psi _n=i^{'}_n)=\frac{P(i_n,\psi _{n-K}=i^{'}_{N-k},\ldots ,\psi _n=i^{'}_n)}{P(\psi _{n-K}=i^{'}_{N-k},\ldots ,\psi _n=i^{'}_n)}. \end{aligned}$$
(12)

Since denominator in (12) has no influence on the classification result of algorithm (11), classifying function of (11) reduces to the nominator, which—assuming that base classifiers (9) are conditionally independent—after simple calculations has the following form:

$$\begin{aligned} \nonumber P(i_n,\psi _{n-K}=i^{'}_{n-K},\ldots ,\psi _n=i^{'}_n)= P(\psi _n=i^{'}_n|i_n) \times \end{aligned}$$
$$\begin{aligned} \times \sum _{j_{n-1}}P(\psi _{n-1}=i^{'}_{n-1}|j_{n-1}) p_{i_{n} j_{n-1}}\times \cdots \times \sum _{j_{n-K}}P(\psi _{n-K}=i^{'}_{n-K}|j_{n-K}) p(j_{n-K}). \end{aligned}$$
(13)

The key element in the algorithm (13) presented above is the calculation of probabilities \(P(\psi _{n}=i_n|j_n)\), i.e. class-dependent probabilities of correct classification and misclassification for base classifiers (9).

The proposed method of evaluation of these probabilities is based on the original concept of a hypothetical classifier called Randomized Reference Classifier (RRC) [20]. The RRC is a stochastic classifier defined by a probability distribution which is chosen in such a way, that RRC acts, on average, as an modeled base classifier. It means, that RRC can be considered equivalent to the modeled base classifier, and therefore it is justified to use the class-dependent probabilities of correct classification (misclassification) of RRC as appropriate probabilities for the evaluated base classifier. In the computational procedure, first these probabilities are calculated for validation points and then they are generalized on the whole feature space. Details of the method can be found in [20]. Similarly, initial (5) and transition (6) probabilities in (13) are estimated using validation set \(\mathcal {V}\).

4 Experiments

4.1 Experimental Setup

Performance of the MC system developed was evaluated in experiments using real data. The experiments were conducted in the Matlab environment using PRTools 4.1 and Signal Processing Toolbox.

In the control process the grasping of 6 types of objects (a pen, a credit card (standing in a container), a computer mouse, a cell phone (laying on the table), a kettle and a tube (standing on the table)) were considered. Our choice is deliberate one and results from the fact that the control functions of simple bioprosthesis are hand closing/opening and wrist pronation/supination, however for the dexterous hand these functions differ depending on grasped object [2].

In the considered examples, seven steps (elementary actions) can be distinguished in the process of grasping with a hand [18]: \(a_{0}\)—rest position; \(a_{1}\)—grasp preparation; \(a_{2}\)—grasp closing; \(a_{3}\)—grabbing; \(a_{4}\)—maintaining the grasp; \(a_{5}\)—releasing the grasp; \(a_{6}\)—transition to the rest position.

The experiments were carried out on healthy persons. Biosignals were registered using 8 integrated sensors (containing EMG electrode and MMG microphone in one casing) located on a forearm (vide Fig. 2). EMG and MMG signals were registered in specially designed 16-channel biosignals measuring circuit with sampling frequency 1 kHz.

Fig. 2
figure 2

The layout of the integrated sensors (EMG electrodes and MMG microphones) on the underside a and top side b of the forearm. Examples of EMG and MMG signals from the channel 2

Full size image

On the base of anatomical analysis of forearm muscles [1], for further processing the following channels (sensors) were selected: channel 1, channel 2, channel 3, channel 5 and channel 8.

The dataset set used to test of proposed classification method consisted of 2940 measurements, i.e. pairs EMG and MMG signals and segment/movement class forming 420 sequences (4). Each sequence lasted 6 s and was preceded with a 10 s break. The coefficients of AR function for different order of AR model (\(p=20, 30, 50, 80\) per signal and per channel) were considered as primary feature vector. Next, primary features were subjected to the PCA feature extraction procedure with the number of PC’s determined by the 95 % of the total variation rule.

The training and testing sets were extracted from each dataset using two-fold cross-validation. For combining the MC system, a two-fold stacked generalization method [21] was used. In this way, the class-dependent probabilities of correct classification/misclassification for base classifiers and initial/transition probabilities of Markov chain are calculated for all objects in the original training set, but the data used for the calculation are unseen during the classifier training.

The experiments were conducted using three different recognition algorithms as base classifiers (4): (LC) Linear classifier based on normal distribution with the same covariance matrix for each class; (k-NN) k-nearest neighbors classifier (k after trials was set to 3); (ANN) feed-forward back-propagation neural network with 1 hidden layer.

The performace of the proposed MC system for \(\mathrm{K}=1\) (MCS-1) and \(\mathrm{K}=2\) (MCS-2) in the sequential scheme was compared against the following six sequential classifiers:

  • the probabilistic algorithm based on the first (second) order Markov dependence (Markov-1, Markov-2) [10];

  • the fuzzy algorithm based on the Mamdani inference scheme with 1- (2-)instant-backward-dependence (Mamdani-1, Mamdani-2) [18];

  • the fuzzy algorithm based on the fuzzy relation with 1- (2-)instant-backward-dependence (FRealtion-1, FRelation-2) [9].

4.2 Results and Discussion

Classification accuracies (i.e. the percentage of correctly classified objects) for methods tested are listed in Table 1. The accuracies are average values obtained over 10 runs (5 replications of two-fold cross validation). Statistical differences between the performances of the MC systems and the six sequential classification methods were evaluated using \(5 \times 2\) cv F test [4]. The level of \(p < 0.05\) was considered statistically significant. In Table 1, statistically significant differences are given under the classification accuracies as indices of the method evaluated, e.g. for the dataset with \(\mathrm{p} = 20\), MCS-1(LC) system produced statistically better classification accuracies from the Mamdani-1, FRelation-1 and FRelation-2 methods.

Table 1 Classification accuracies of classifiers compared in the experiment
Full size table

These results imply the following conclusions: (1) The MC systems produced statistically significant higher scores in 87 out of 144 cases (4 datasets \(\times \) 6 classifiers compared \(\times \) 6 MCS’s); (2) The MCS-2 system with ANN base classifiers achieved the highest overall classification accuracy averaged over all datasets it outperformed the Markov-1, Markov-2, Mamdani-1, Mamdani-2, FRelation-1, FRelation-2 systems by 1.2, 0.6, 5.8, 4.6, 13.1, 12.3 %, respectively. This results confirm the effectiveness of the use the multiclassifier system in the recognition of patient’s intent; (3) There occurs a common effect within each classifier (MC system) type: 1-instant-backwards-dependence is always worse than 2-instant-backwards-dependence. This confirms the effectiveness of the decomposition of decision procedure into sequence of simpler classification tasks; (4) When the order of AR model increases then the accuracy of all methods investigated also increases.

5 Conclusion

The classic methods of analysis of biosignals in the bioprostheses control systems are widely discussed in the literature [6, 7, 12, 16]. However, the classification stage still poses a challenge for researching new solutions enabling the reliable recognition of human intention. In this study a novel method for recognition of sequence of elementary actions of grasping movements is proposed. The method, combining the meta-Bayes concept and Markov model into multiclassifier system and taking into account the K-instant-backwards-dependence among elementary actions, brings new possibilities to biosignal analysis. Results obtained in experimental investigations imply that it is worth trying solution that improves recognition efficiency.

The introduced approach constitutes the general concept of the human-machine interface, that can be applied for the control of a dexterous hand and an agile wheelchair as well as other types of prostheses, exoskeletons, etc. This, however, requires a further study, mainly in the experimental phase, which would allow to assess and verify the effectiveness of the adopted concept.