1 Introduction

Feature selection [1,2,3,4,5] aims at selecting some informative features from feature set, which is an important method of dimensionality reduction and has widespread applications, such as text processing [6, 7], steganalysis [8, 9], underwater objects classification [10], network anomaly detection [11], information retrieval [12], and image classification [13, 14]. Feature selection algorithms are divided into three categories: filter, embedded, and wrapper methods. Since classification accuracy of classifier is taken as the metric, embedded and wrapper methods are time-consuming and not robust. Filter methods take less time at the cost of the decline in classification results. It is advisable that the datasets with high dimensional features be dealt with filter methods [15].

The metrics commonly used in filter methods include consistency, distance and MI. Since MI has the capacity of measuring linear and non-linear correlation as well as its invariance under space transformations [16], feature selection based on MI and TDMI is widely investigated. Mutual information maximization (MIM) [17] is a basic feature selection algorithm based on MI. It calculates MI between the class label and features, and selects some features with greater values. On the basis of MIM, feature selection algorithms based on relevance and redundancy are proposed, such as minimum-redundancy maximum-relevance (mRMR) [18], conditional mutual information (CMI) [19], and MIFS-CR [20]. These algorithms adopt MI between features and the class label to describe relevance and exploit MI between features to describe redundancy. Owing to considering relevance and redundancy simultaneously, feature selection performance of these algorithms is improved.

Some feature selection algorithms further consider TDMI to improve the performance and algorithms based on TDMI are proposed, such as interaction weight based feature selection (IWFS) [21], joint mutual information maximization (JMIM) [22] and maximizing independent classification information (MRI) [23]. Since these algorithms consider TDMI among features and the class label and ignore TDMI among features, their objective functions might miss some useful information and the performance of these algorithms is influenced.

Considering the above problem, this paper investigates feature selection based on TDMI among features. Firstly, to select the features that provide more useful information, based on the maximal relevance minimal redundancy (MRMR) criterion, joint mutual information (JMI) among the class label and feature set as well as MI between feature set is employed to describe relevance and redundancy separately. Then, JMI among the class label and feature set as well as MI between features is decomposed, and TDMI among features is adopted. Furthermore, both performance and computation are considered, and an objective function is achieved. Finally, a feature selection algorithm based on CMI is proposed.

The main contributions of this paper are as follows. (1) The maximal relevance minimal redundancy criterion is adopted in selecting features. (2) Our algorithm takes special consideration of three-dimensional mutual information among features. (3) The proposed algorithm takes both performance and computation into account. (4) Our algorithm can achieve better feature selection performance at the expense of more time-consuming.

The rest of this paper is organized as follows. Section 2 gives the knowledge of mutual information. Related works are analyzed in Section 3. Section 4 presents the proposed algorithm. Experimental results and analysis are given in Section 5. Section 6 is conclusions and future work.

2 The knowledge of mutual information

Assuming x and y are two discrete random variables. p(x) and p(y) are the probability of x and y separately. Information entropy is exploited to measure information and H(X) is defined by (1):

$$ H(X) = - \underset{x \in X}{\sum} {p(x)\log p(x){\text{ }}} $$
(1)

Conditional entropy H(Y |X) is the entropy of Y when X is given and it can be expressed as (2):

$$ H(Y|X) = - \underset{x \in X}{\sum} {\underset{y \in Y}{\sum} {p(x,y)\log p(y|x)} {\ }} $$
(2)

where p(y|x) is the conditional probability and p(x,y) is the joint probability.

MI is employed to quantify the information that two variables share and MI I(X;Y ) is defined as (3):

$$ I(X;Y) = \underset{x \in X}{\sum} {\underset{y \in Y}{\sum} {p(x,y)\log \frac{{p(x,y)}}{{p(x)p(y)}}{\ }}} $$
(3)

Greater MI value suggests more information that the two variables share. MI has the relationship with information entropy and conditional entropy as (4).

$$ I(X;Y) = I(Y;X) = H(Y) - H(Y|X) = H(X) - H(X|Y){\ } $$
(4)

TDMI is a supplement of MI and it includes CMI, JMI and three-way interaction information. CMI is the reduction in the uncertainty of the other variable due to the knowledge of another variable when one variable is given. CMI I(X;Y |Z) is defined by (5):

$$ I(X;Y|Z) = \underset{x \in X}{\sum} {\underset{y \in Y}{\sum} {\underset{z \in Z}{\sum} {p(x{\text{,}}y,z)\log \frac{{p(x,y|z)}}{{p(x|z)p(y|z)}}} {\ }} } $$
(5)

JMI is utilized to measure the information that two variables share with the other variable. JMI I(X,Y ;Z) has the relationship with I(Y ;Z) and I(X;Z|Y ) as (6).

$$ I{\text{(}}X{\text{,}}Y{\text{;}}Z{\text{)}} = I{\text{(}}Y{\text{;}}Z{\text{)}} + I{\text{(}}X{\text{;}}Z|Y{\text{)}} {\ } $$
(6)

Three-way interaction information I(X;Y ;Z) has the relationship with I(X;Z|Y ) and I(X;Z) as (7) [24].

$$ I{\text{(}}X;Y;Z{\text{)}} = I{\text{(}}X;Z|Y{\text{)}} - I{\text{(}}X;Z{\text{)}} {\ } $$
(7)

3 Related works

MIM is a basic feature selection algorithm based on MI and the objective function is presented in (8).

$$ MIM = \arg \underset{{f_{i}} \in X}{\max} \left[ {I\left( {c{\text{;}}{f_{i}}} \right)} \right]{\ } $$
(8)

MIM calculates MI between the class label c and each candidate feature fi, and selects some features with greater values from the candidate feature set X.

Some feature selection algorithms consider redundancy between features further and algorithms based on relevance and redundancy are proposed, such as mutual information feature selection (MIFS) [25], MIFS-U [26], mRMR, normalized mutual information feature selection (NMIFS) [16], CMI and MIFS-CR. These algorithms exploit MI between candidate features and the class label to describe relevance, and employ MI between selected features and candidate features to describe redundancy. Objective functions are the key of these algorithms. The objective functions of MIFS and MIFS-U are given below.

$$ MIFS = \arg \underset{{f_{i}} \in X}{\max} \left[ {I\left( {c{\text{;}}{f_{i}}} \right) - \beta \underset{{f_{s}} \in S}{\sum} {I\left( {{f_{s}}{\text{;}}{f_{i}}} \right)} } \right]{\ } $$
(9)
$$ MIFS - U = \arg \underset{{f_{i}} \in X}{\max} \left[ {I\left( {c{\text{;}}{f_{i}}} \right) - \beta \underset{{f_{s}} \in S}{\sum} {\frac{{I(c;{f_{s}})}}{{H({f_{s}})}}I\left( {{f_{s}}{\text{;}}{f_{i}}} \right)} } \right]{\ } $$
(10)

where β is a parameter, fs is a selected feature and S is the selected feature set.

Since MIFS and MIFS-U have the problem that β is uncertain, mRMR adopts the reciprocal of the number of selected features \({\left | S \right |}\) to replace β and the objective function is shown in (11).

$$ mRMR = \arg \underset{{f_{i}} \in X}{\max} \left[ {I\left( {c{\text{;}}{f_{i}}} \right) - \frac{1}{{\left| S \right|}}\underset{{f_{s}} \in S}{\sum} {I\left( {{f_{s}}{\text{;}}{f_{i}}} \right)} } \right]{\ } $$
(11)

Since mRMR is considered to have bias toward the features with greater MI values, MI between a candidate feature and a selected feature is normalized and NMIFS is proposed. Its objective function is given in (12).

$$ NMIFS = \arg \underset{{f_{i}} \in X}{\max} \left[ {I\left( {c;{f_{i}}} \right) - \frac{1}{{\left| S \right|}}\underset{{f_{s}} \in S}{\sum} {\frac{{I\left( {{f_{i}};{f_{s}}} \right)}}{{\min \left( {H\left( {{f_{i}}} \right),H\left( {{f_{s}}} \right)} \right)}}} } \right] $$
(12)

CMI and MIFS-CR are proposed, and their objective functions are shown below.

$$ CMI = \arg \underset{{f_{i}} \in X}{\max} \left[ {I\left( {c{\text{;}}{f_{i}}} \right) - \frac{{H({f_{i}}|c)}}{{H({f_{i}})}}\underset{{f_{s}} \in S}{\sum} {\frac{{I(c;{f_{s}})I\left( {{f_{s}}{\text{;}}{f_{i}}} \right)}}{{H({f_{s}})H(c)}}} } \right]{\ } $$
(13)
$$ MIFS - CR = \arg \underset{{f_{i}} \in X}{\max} \left[ {I\left( {c;{f_{i}}} \right) - \frac{1}{2}\underset{{f_{s}} \in S}{\sum} {\left( {\frac{{I(c;{f_{s}})}}{{H({f_{s}})}} + \frac{{I(c;{f_{i}})}}{{H({f_{i}})}}} \right)I\left( {{f_{s}};{f_{i}}} \right)} } \right] $$
(14)

Furthermore, considering that mRMR has the problem that the feature with the maximum difference is not always the feature with minimal redundancy maximal relevance, a method of equal interval division is adopted to process the case where the objective function of mRMR is not accurate. Finally an algorithm based on equal interval division and minimal-redundancy-maximal-relevance (EID-mRMR) [27] is proposed. Since relevance and redundancy are both considered, these algorithms based on relevance and redundancy can achieve better feature selection performance.

There are some feature selection algorithms based on TDMI, such as dynamic weighting-based feature selection (DWFS) [28], IWFS, conditional mutual information maximization (CMIM) [29], JMI [30], maximal conditional mutual information (MCMI) [31] and MRI. DWFS and IWFS belong to the same category, and they utilize symmetrical uncertainty that normalizes MI to describe relevance.

JMI, CMIM, MCMI, and MRI fall into a different category, and they have different objective functions. Except for the four algorithms above, the kind of algorithms include conditional informative feature extraction (CIFE) [32], JMIM, CFR [33], and Dynamic Change of Selected Feature with the class (DCSF) [34], and their objective functions are presented below.

$$ JMI = \arg \underset{{f_{i}} \in X}{\max} \left[ {I\left( {c{\text{;}}{f_{i}}} \right) - \frac{1}{{\left| S \right|}}\underset{{f_{s}} \in S}{\sum} {I\left( {{f_{s}}{\text{;}}{f_{i}}} \right)} + \frac{1}{{\left| S \right|}}\underset{{f_{s}} \in S}{\sum} {I\left( {{f_{s}}{\text{;}}{f_{i}}|c} \right)} } \right]{\ } $$
(15)
$$ CMIM = \arg \underset{{f_{i}} \in X}{\max} \left[ {\mathop {\min }\limits_{{f_{s}} \in S} \left( {I\left( {c;{f_{i}}|{f_{s}}} \right)} \right)} \right] $$
(16)
$$ MCMI = \arg \underset{{f_{i}} \in X}{\max} \left[ {\mathop {\max }\limits_{{f_{s}} \in S} \left( {I\left( {c;{f_{i}}|{f_{s}}} \right)} \right)} \right] $$
(17)
$$ MRI = \arg \underset{{f_{i}} \in X}{\max} \left[ {I\left( {c{\text{;}}{f_{i}}} \right) + \underset{{f_{s}} \in S}{\sum} {I\left( {c{\text{;}}{f_{i}}|{f_{s}}} \right)} + \underset{{f_{s}} \in S}{\sum} {I\left( {c{\text{;}}{f_{s}}|{f_{i}}} \right)} } \right]{\ } $$
(18)
$$ CIFE = \arg \underset{{f_{i}} \in X}{\max} \left[ {I\left( {c{\text{;}}{f_{i}}} \right) - \underset{{f_{s}} \in S}{\sum} {I\left( {{f_{s}}{\text{;}}{f_{i}}} \right)} + \underset{{f_{s}} \in S}{\sum} {I\left( {{f_{s}}{\text{;}}{f_{i}}|c} \right)} } \right]{\ } $$
(19)
$$ JMIM = \arg \underset{{f_{i}} \in X}{\max} \left[ {\mathop {\min }\limits_{{f_{s}} \in S} \left( {I\left( {{f_{i}},{f_{s}};c} \right)} \right)} \right] $$
(20)
$$ CFR = \arg \underset{{f_{i}} \in X}{\max} \left[ { \underset{{f_{s}} \in S}{\sum} {I\left( {c{\text{;}}{f_{i}}|{f_{s}}} \right)} + \underset{{f_{s}} \in S}{\sum} {I\left( {c{\text{;}}{f_{s}}{\text{;}}{f_{i}}} \right)} } \right]{\ } $$
(21)

In (15)–(21), since TDMI among features is not employed, feature selection effectiveness of these algorithms can be affected.

4 The proposed algorithm

This section decomposes JMI among feature set and the class label as well as MI between feature set and attains an objective function. Then, based on the objective function, the proposed algorithm is presented.

4.1 The proposed objective function

The aim of some existing feature selection algorithms based on MI and TDMI is to select the feature set that have maximum MI with the class label, and these algorithms only consider relevant information satisfying the maximum. However, selecting a candidate feature fi introduces not only relevant information, but also redundant information. To introduce maximal relevant and minimal redundant information, we formulate this issue as (22).

$$ \arg \underset{{f_{i}} \in X}{\max} \left[ {I{\text{(}}S{\text{,}}{f_{i}}{\text{;}}c{\text{)}} - I{\text{(}}S{\text{;}}{f_{i}}{\text{)}}} \right] {\ } $$
(22)

where S is the selected feature set and X is the candidate feature set. The total of relevant information that is introduced is I(S,fi;c) and that of redundant information is I(S;fi). The greater the difference between relevant and redundant information, the more informative the candidate feature is. Adopting (22) can ensure that maximal relevant and minimal redundant information is introduced, thus guaranteeing feature selection effectiveness of selected features. I(S,fi;c) satisfies (23).

$$ I{\text{(}}S{\text{,}}{f_{i}}{\text{;}}c{\text{)}} = I{\text{(}}S{\text{;}}c{\text{)}} + I{\text{(}}{f_{i}}{\text{;}}c|S{\text{)}} {\ } $$
(23)

I(fi;c|S) satisfies (24).

$$ I{\text{(}}{f_{i}}{\text{;}}c|S{\text{) = }}I{\text{(}}c{\text{;}}{f_{i}}{\text{)}} - I{\text{(}}S{\text{;}}c{\text{) + }}I{\text{(}}S{\text{;}}c|{f_{i}}{\text{)}} {\ } $$
(24)

Equation (25) is derived by adding (23) to (24).

$$ I{\text{(}}S{\text{,}}{f_{i}}{\text{;}}c{\text{)}} = I{\text{(}}c{\text{;}}{f_{i}}{\text{) + }}I{\text{(}}S{\text{;}}c|{f_{i}}{\text{)}} {\ } $$
(25)

By combining (25) with (22), (26) is obtained.

$$ \arg \underset{{f_{i}} \in X}{\max} \left[ {I{\text{(}}c{\text{;}}{f_{i}}{\text{) + }}I{\text{(}}S{\text{;}}c|{f_{i}}{\text{)}} - I{\text{(}}S{\text{;}}{f_{i}}{\text{)}}} \right] $$
(26)

I(S;c|fi) satisfies (27).

$$ I{\text{(}}S{\text{;}}c|{f_{i}}{\text{)}} = \frac{1}{{\left| S \right|}}\underset{{f_{s}} \in S}{\sum} {I\left( {{f_{s}}{\text{;}}c|{f_{i}}} \right)} $$
(27)

By combining (25) with (27), (28) is derived.

$$ I{\text{(}}S{\text{,}}{f_{i}}{\text{;}}c{\text{)}} = I{\text{(}}c{\text{;}}{f_{i}}{\text{) + }}\frac{1}{{\left| S \right|}}\underset{{f_{s}} \in S}{\sum} {I\left( {{f_{s}}{\text{;}}c|{f_{i}}} \right)} {\ } $$
(28)

I(S;fi) satisfies (29).

$$ I{\text{(}}S{\text{;}}{f_{i}}{\text{)}} = \frac{1}{{\left| S \right|}}\underset{{f_{s}} \in S}{\sum} {I\left( {{f_{i}}{\text{;}}{f_{s}}} \right)} + \frac{1}{{\left| S \right|\left| {S - 1} \right|}}\underset{{f_{s}} \in S}{\sum} {\underset{{f_{j}} \in S,{f_{j}} \ne {f_{s}}}{\sum} {I\left( {{f_{j}}{\text{;}}{f_{i}}|{f_{s}}} \right)} } $$
(29)

Since it is quite time-consuming to calculate the second part of (29), we replace (29) by (30).

$$ \frac{1}{{\left| S \right|}}\underset{{f_{s}} \in S}{\sum} {I\left( {{f_{i}}{\text{;}}{f_{s}}} \right)} + \frac{1}{{\left| S \right|\left| {S - 1} \right|}}\underset{{f_{s}} \in S}{\sum} {\underset{{f_{j}} \in S,{f_{j}} \ne {f_{s}}}{\sum} {I\left( {{f_{j}}{\text{;}}{f_{s}}|{f_{i}}} \right)} } $$
(30)

By combining (28) and (30), (31) is obtained.

$$ \begin{array}{@{}rcl@{}} \arg \underset{{f_{i}} \in X}{\max} &&\left[ {I{\text{(}}c{\text{;}}{f_{i}}{\text{)}} - \frac{1}{{\left| S \right|}}\underset{{f_{s}} \in S}{\sum} {I\left( {{f_{i}}{\text{;}}{f_{s}}} \right)} {\text{ + }}\frac{1}{{\left| S \right|}}\underset{{f_{s}} \in S}{\sum} {I\left( {{f_{s}}{\text{;}}c|{f_{i}}} \right)} } \right.\!\!\\ &&\left. { - \frac{1}{{\left| S \right|\left| {S - 1} \right|}}\underset{{f_{s}} \in S}{\sum} {\underset{{f_{j}} \in S,{f_{j}} \ne {f_{s}}}{\sum} {I\left( {{f_{j}}{\text{;}}{f_{s}}|{f_{i}}} \right)} } } \right] \end{array} $$
(31)

Figure 1 gives a brief description of the determination of (31). Equation (31) considers not only MI between features and the class label, CMI among features and the class label as well as MI between features, but also CMI among features, while the objective functions of other feature selection algorithms do not consider TDMI among features. Compared with the objective functions of other algorithms, (31) contains more useful information and selecting the feature satisfying (31) can guarantee that maximal relevant and minimal redundant information is obtained. To guarantee introducing maximal relevant and minimal redundant information, (31) is taken as an objective function.

Fig. 1
figure 1

Determination of (31)

Full size image

4.2 Algorithmic implementation

Based on (31), a feature selection algorithm based on CMI for MRMR (CMI-MRMR) is proposed and the flow chart is presented in Fig. 2.

Fig. 2
figure 2

Flow chart of CMI-MRMR

Full size image

In Fig. 2, we first initialize X and S. Then, we calculate MI between features and the class label, and select the feature with maximum. We calculate (32) that does not have the fourth part of (31) and select the feature satisfying the condition. Following that, we calculate (31) and select the feature that meets the requirement until a specified number of features are selected.

$$ \arg \underset{{f_{i}} \in X}{\max} \left[ {I{\text{(}}c{\text{;}}{f_{i}}{\text{)}} - \frac{1}{{\left| S \right|}}\underset{{f_{s}} \in S}{\sum} {I\left( {{f_{i}}{\text{;}}{f_{s}}} \right)} {\text{ + }}\frac{1}{{\left| S \right|}}\underset{{f_{s}} \in S}{\sum} {I\left( {{f_{s}}{\text{;}}c|{f_{i}}} \right)} } \right] $$
(32)

The pseudo-code of CMI-MRMR is shown in Algorithm 1.

figure e

CMI-MRMR consists of three parts. The first part (lines 1-6), S and X are initialized. Then, MI between the class label and features is calculated, and the feature fk is selected; in the second part (lines 7-12), \({I\left ({{f_{i}}{\text {;}}{f_{s}}} \right )}\) and \({I\left ({{f_{s}}{\text {;}}c|{f_{i}}} \right )}\) are calculated. Then, (32) is calculated and the feature fl that satisfies the condition is selected from X; in the third part (lines 13-25), \({I\left ({{f_{j}}{\text {;}}{f_{s}}|{f_{i}}} \right )}\), \({I\left ({{f_{s}}{\text {;}}c|{f_{i}}} \right )}\), and \({I\left ({{f_{i}}{\text {;}}{f_{s}}} \right )}\) are calculated. Then, (31) is calculated and the feature fm meeting the requirement is selected. Following the above steps, the process ends when the number of selected features is N.

5 Experimental results

To validate the performance of CMI-MRMR, mRMR, IWFS, JMIM, MCMI, MRI, CFR, and DCSF are compared.

5.1 The datasets and experimental settings

The datasets in Table 1 are from UCI machine learning repository [35] and Arizona State University (ASU) feature selection datasets [36]. For all the datasets, N is set to 50. Minimum description length discretization method [37] is exploited to transform the numerical features into discrete ones. Three popular classifiers, J48, IB1, and Naive Bayes are employed and their parameters are set to Waikato environment for knowledge analysis (WEKA)’s [38] default values. ASU feature selection software package [39] is utilized.

Table 1 Description of the datasets
Full size table

5.2 Experimental results and analysis

To reduce the influence of randomness on the final results, ten times of 10-fold cross-validation are employed, and the mean value and standard deviation of ten results are taken as the final results. Classification accuracy of features selected by these algorithms with J48, IB1 and Naive Bayes is presented in Tables 2–4. To determine whether the effectiveness of experimental results is significant, a one-sided paired t-test at 5% significance level is performed, and the number of the datasets that CMI-MRMR performs better than/equal to/worse than other algorithms is shown in Win/Tie/Loss (W/T/L). Average performance of algorithms with the three classifiers is given in Fig. 3. Furthermore, the optimal top several features from 1 to 50 selected by CMI-MRMR is compared with all features, and the comparison result with three classifiers is shown in Tables 5 and 6.

Fig. 3
figure 3

Average performance comparisons of algorithms with the three classifiers

Full size image
Table 2 Classification accuracy (%) of selected features with J48
Full size table

As shown in Table 2, the Avg. values show that mRMR, JMIM and CMI-MRMR achieve better results. For the W/T/L values, the number of datasets that mRMR, DCSF and CMI-MRMR can obtain better feature selection performace.

In Table 3, mRMR, DCSF and CMI-MRMR obtain greater Avg. and W/T/L values with IB1. In comparison with Table 2, CMI-MRMR outperforms mRMR, IWFS and MCMI with more performance gain in the Avg. values and it achieves more advantages than IWFS in the W/T/L values.

Table 3 Classification accuracy (%) of selected features with IB1
Full size table

The Avg. and W/T/L values in Table 4 show that mRMR, MRI, DCSF and CMI-MRMR obtain better feature selection effectiveness. Compared with Tables 2 and 3, in terms of the Avg. values, CMI-MRMR has more advantage than mRMR, IWFS, JMIM, and MCMI. For the W/T/L values, CMI-MRMR can obtain better performance gain than other algorithms except DCSF.

Table 4 Classification accuracy (%) of selected features with Naive Bayes
Full size table

As shown in Fig. 3, CMI-MRMR achieves better feature selection effectiveness in the majority of datasets, while other algorithms cannot obtain the desired results in some datasets. We take mRMR and CFR as examples, mRMR cannot handle well in lung_discrete and orlraws10P. CFR does not achieve the desired feature selection performance in lung_discrete and arcene.

As shown in Tables 5 and 6, the Avg. values show that the optimal features selected by CMI-MRMR obtain higher accuracy than all features. In comparison with these datasets, the number of the datasets that the features selected by CMI-MRMR perform better than all features is 8 with J48, 4 with IB1, and 5 with Naive Bayes. Overall, although CMI-MRMR only selects the top 50 features, it can have fairly good performance with all features.

Table 5 Classification accuracy (%) of the optimal features selected by CMI-MRMR and all features
Full size table
Table 6 The number of the optimal features selected by CMI-MRMR and all features
Full size table

6 Conclusions and future work

This paper investigates feature selection based on TDMI among features and proposes a feature selection algorithm named CMI-MRMR. To verify the performance, we apply it to three classifiers, four UCI datasets, and twelve ASU datasets, and compare results with those from several algorithms based on MI and TDMI. Experimental results validate that CMI-MRMR can achieve better feature selection effectiveness. Furthermore, the optimal feature set selected by CMI-MRMR are compared with all features, the comparison results show that CMI-MRMR can achieve fairly good performance with all features in the majority of datasets, even better than all features in some datasets.

Considering that CMI-MRMR can achieve better feature selection performance, it can be applied in many fields, such as text processing, underwater objects recognition and classification, network anomaly detection, gene expression, and image classification. Classification accuracy of the top optimal several features from 1 to 50 selected by CMI-MRMR is compared with all features, since classification results of the top optimal several features are worse than all features in some datasets, the determination of the number of selected features will be investigated in the next stage.