Abstract
Mutual Information (MI) is a powerful concept from information theory used in many application fields. For practical tasks it is often necessary to estimate the Mutual Information from available data. We compare state of the art methods for estimating MI from continuous data, focusing on the usefulness for the feature selection task. Our results suggest that many methods are practically relevant for feature selection tasks regardless of their theoretic limitations or benefits.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
Keywords
- Feature Selection
- Mutual Information
- Kernel Density Estimation
- Histogram Estimation
- Kernel Density Estimation Method
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Khan, S., Bandyopadhyay, S., Ganguly, A.R., Saigal, S., Erickson, D.J., Protopopescu, V., Ostrouchov, G.: Relative performance of mutual information estimation methods for quantifying the dependence among short and noisy data. Physical Review E 76, 026209 (2007)
Scott, D.W.: Multivariate density estimation: theory, practice and visualization. John Wiley & Sons, New York (1992)
Cellucci, C.J., Albano, A.M., Rapp, P.E.: Statistical validation of mutual information calculations: Comparison of alternative numerical algorithms. Physical Review E 71(6), 066208 (2005)
Silverman, B.W.: Density Estimation for Statistics and Data Analysis. Chapman and Hall, London (1986)
Suzuki, T., Sugiyama, M., Sese, J., Kanamori, T.: A least-squares approach to mutual information estimation with application in variable selection. In: Proceedings of the 3rd Workshop on New Challenges for Feature Selection in Data mining and Knowledge Discovery (FSDM 2008), Antwerp, Belgium (2008)
Kraskov, A., Stögbauer, H., Grassberger, P.: Estimating mutual information. Physical Review E 69, 066138 (2004)
Battiti, R.: Using mutual information for selecting features in supervised neural net learning. IEEE Transactions on Neural Networks 5(4), 537–550 (1994)
Kwak, N., Choi, C.H.: Input feature selection by mutual information based on parzen window. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(12), 1667–1671 (2002)
Asuncion, A., Newman, D.: UCI machine learning repository (2007), http://archive.ics.uci.edu/ml/
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Schaffernicht, E., Kaltenhaeuser, R., Verma, S.S., Gross, HM. (2010). On Estimating Mutual Information for Feature Selection. In: Diamantaras, K., Duch, W., Iliadis, L.S. (eds) Artificial Neural Networks – ICANN 2010. ICANN 2010. Lecture Notes in Computer Science, vol 6352. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-15819-3_48
Download citation
DOI: https://doi.org/10.1007/978-3-642-15819-3_48
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-15818-6
Online ISBN: 978-3-642-15819-3
eBook Packages: Computer ScienceComputer Science (R0)