Abstract
A hybrid filter/wrapper feature subset selection algorithm for regression is proposed. First, features are filtered by means of a relevance and redundancy filter using mutual information between regression and target variables. We introduce permutation tests to find statistically significant relevant and redundant features. Second, a wrapper searches for good candidate feature subsets by taking the regression model into account. The advantage of a hybrid approach is threefold. First, the filter provides interesting features independently from the regression model and, hence, allows for an easier interpretation. Secondly, because the filter part is computationally less expensive, the global algorithm will faster provide good candidate subsets compared to a stand-alone wrapper approach. Finally, the wrapper takes the bias of the regression model into account, because the regression model guides the search for optimal features. Results are shown for the ‘Boston housing’ and ‘orange juice’ benchmarks based on the multilayer perceptron regression model.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Guyon, I., Elisseeff, A.: An Introduction to Variable and Feature Selection. Journal of Ma-chine Learning Research 3, 1157–1182 (2003)
Kurgan, L.A., Cios, K.J.: CAIM Discretization Algorithm. IEEE Transactions on Knowl-edge and Data Engineering 16, 145–153 (2004)
Kohavi, R., John, G.H.: Wrappers for Feature Subset Selection. Artificial Intelligence 97, 273–324 (1997)
Van Dijck, G., Van Hulle, M.M., Wevers, M.: Hierarchical Feature Subset Selection for Features Computed from the Continuous Wavelet Transform. In: 2005 IEEE Workshop on Machine Learning for Signal Processing, pp. 81–86 (2005)
Cover, T.M., Thomas, J.A.: Elements of information theory. John Wiley & Sons, New York (1991)
Schreiber, T., Schmitz, A.: Surrogate Time Series. Physica D 142, 346–382 (2000)
Kraskov, A., Stögbauer, H., Grassberger, P.: Estimating Mutual Information. Phys. Rev. E. 69, 66138 (2004)
Francois, D., Wertz, V., Verleysen, M.: The Permutation Test for Feature Selection by Mutual Information. In: European Symposium on Artificial Neural Networks, pp. 239–244 (2006)
John, G., Kohavi, R., Pfleger, K.: Irrelevant Features and the Subset Selection Problem. In: Proc. of the Eleventh Int. Conf. on Machine Learning, pp. 121–129 (1994)
Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification. John Wiley & Sons Inc., New York (2001)
Bishop, C.M.: Neural Networks for Pattern Recognition. Oxford University Press Inc., New York (1997)
Narendra, P.M., Fukunaga, K.: A Branch and Bound Algorithm for Feature Subset Selection. IEEE Trans. Computers 26, 917–922 (1977)
Pudil, P., Novovicova, J., Kittler, J.: Floating Search Methods in Feature Selection. Pattern Recognition Letters 15, 1119–1125 (1994)
Michalewicz, Z.: Genetic Algorithms + Data Structures = Evolution Programs, 3rd edn. Springer, Heidelberg (1996)
Kudo, M., Sklansky, J.: Comparison of Algorithms that Select Features for Pattern Recognition. Pattern Recognition 33, 25–41 (2000)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Van Dijck, G., Van Hulle, M.M. (2006). Speeding Up the Wrapper Feature Subset Selection in Regression by Mutual Information Relevance and Redundancy Analysis. In: Kollias, S.D., Stafylopatis, A., Duch, W., Oja, E. (eds) Artificial Neural Networks – ICANN 2006. ICANN 2006. Lecture Notes in Computer Science, vol 4131. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11840817_4
Download citation
DOI: https://doi.org/10.1007/11840817_4
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-38625-4
Online ISBN: 978-3-540-38627-8
eBook Packages: Computer ScienceComputer Science (R0)