Abstract
In this paper, we first survey the theoretical and historical backgrounds related to ensemble neural network rule extraction. Then we propose a new rule extraction method for ensemble neural networks. We also demonstrate that the use of ensemble neural networks produces higher recognition accuracy than do individual neural networks. Because the extracted rules are more comprehensible. The rule extraction method we use is the Ensemble-Recursive-Rule eX traction (E-Re-RX) algorithm. The E-Re-RX algorithm is an effective rule extraction algorithm for dealing with data sets that mix discrete and continuous attributes. In this algorithm, primary rules are generated, followed by secondary rules to handle only those instances that do not satisfy the primary rules, and then these rules are integrated. We show that this reduces the complexity of using multiple neural networks. This method achieves extremely high recognition rates, even with multiclass problems.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
Keywords
References
Breiman, L.: Bagging predictors. Mach. Learn. 24, 123–140 (1996)
Freund, Y., Schapire, R.: Experiments with a new boosting algorithm. In: Proc. of the Thirteenth International Conference on Machine Learning, Bari, Italy, pp. 148–156 (1996)
Zhang, G.P.: Neural networks for classification: A Survey. IEEE Trans. Systems, Man and Cybernetics–Part C: Applications and Reviews 30(4), 451–462 (2000)
Rokach, L.: Ensemble-based classifiers. Artificial Intelligence Review 33, 1–39 (2010)
Yao, X., Islam, M.: Evolving artificial neural network ensembles. IEEE Computational Intelligence Magazine 3(1), 31–42 (2008)
Cybenko, G.: Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems 2, 303–314 (1989)
Setiono, R.: A penalty-function approach for pruning feedforward neural networks. Neural Comp. 9(1), 185–204 (1997)
Bishop, C.: Neural Networks for Pattern Recognition. Oxford University Press (1995)
Setiono, R., Baesens, B., Mues, C.: Recursive neural network rule extraction for data with mixed attributes. IEEE Trans. Neural Netw. 19, 299–307 (2008)
Akhand, M.A.H., Murase, K.: Neural Network Ensembles. Lambert Academic Publishing (LAP) (2010)
Alpaydin, E.: Multiple Neural Networks and Weighted Voting. IEEE Trans. on Pattern Recognition 2, 29–32 (1992)
University of California, Irvine Machine Learning Repository, http://archive.ics.uci.edu/ml/
Hara, A., Hayashi, Y.: Ensemble neural network rule extraction using Re-RX algorithm. In: Proc. of WCCI (IJCNN) 2012, Brisbane, Australia, June 10-15, pp. 604–609 (2012)
Mitra, S., Hayashi, Y.: Neuro-Fuzzy Rule Generation: Survey in Soft Computing Framework. IEEE Trans. Neural Netw. 11(3), 748–768 (2000)
Gallant, S.I.: Connectionist expert systems. Commun. ACM 31, 152–169 (1988)
Saito, K., Nakano, R.: Medical diagnosis expert systems based on PDP model. In: Proc. IEEE Int. Conf. Neural Netw, San Diego, CA, pp. I.255–I.262 (1988)
Hayashi, Y.: Neural expert system using teaching fuzzy input and its application to medical diagnosis. Inform. Sci.: Applicat. 1, 47–58 (1994)
Hayashi, Y.: A neural expert system with automated extraction of fuzzy if-then rules and its application to medical diagnosis. In: Lippmann, R.P., Moody, J.E., Touretzky, D.S. (eds.) Advances in Neural Information Processing Systems, pp. 578–584. Morgan Kaufmann, Los Altos (1991)
Hudson, D.L., Cohen, M.E., Anderson, M.F.: Use of neural network techniques in a medical expert system. Int. J. Intell. Syst. 6, 213–223 (1991)
Takagi, H.: Fusion technology of fuzzy theory and neural network—Survey and future directions. In: Proc. Int. Conf. Fuzzy Logic and Neural Networks, Iizuka, Japan, pp. 13–26 (1990)
Buckley, J.J., Hayashi, Y., Czogala, E.: On the equivalence of neural nets and fuzzy expert systems. Fuzzy Sets Syst. 53(2), 129–134 (1993)
Hayashi, Y., Buckley, J.J.: Approximation between fuzzy expert systems and neural networks. Int. J. Approx. Res. 10, 63–73 (1994)
Buckley, J.J., Hayashi, Y.: Numerical relationship between neural networks, continuous function and fuzzy systems. Fuzzy Sets Syst. 60(1), 1–8 (1993)
Buckley, J.J., Hayashi, Y.: Hybrid neural nets can be fuzzy controllers and fuzzy expert systems. Fuzzy Sets and Syst. 60, 135–142 (1993)
Golea, M.: On the complexity of rule extraction from neural networks and network querying. In: Proc. the AIBS 1996 Workshop on the Rule Extraction from Trained Neural Networks, Brighton, UK, pp. 51–59 (1996)
Roy, A.: On connectionism, rule extraction, and brain-like learning. IEEE Trans. Fuzzy Systems 8(2), 222–227 (2000)
Zhou, Z.-H.: Rule Extraction: Using Neural Networks or for Neural Networks? J. Comput. Sci. & Technol. 19(2), 249–253 (2004)
Buckley, J.J., Hayashi, Y.: Fuzzy neural networks: A survey. Fuzzy Sets Syst. 66, 1–13 (1994)
Hayashi, Y., Buckley, J.J., Czogala, E.: Fuzzy neural network with fuzzy signals and weights. Int. J. Intell. Syst. 8(4), 527–573 (1993)
Ishibuchi, H., Kwon, K., Tanaka, H.: A learning algorithm of fuzzy neural networks with triangular fuzzy weights. Fuzzy Sets Syst. 71, 277–293 (1995)
Feuring, T., Buckley, J.J., Hayashi, Y.: A gradient descent learning algorithm for fuzzy neural networks. In: Proc. IEEE Int. Conf. Fuzzy Syst. FUZZ-IEEE 1998, Anchorage, AK, pp. 1136–1141 (1998)
Bologna, G.: Is it worth generating rules from neural network ensembles? J. of Applied Logic 2, 325–348 (2004)
Bologna, G.: A study on rule extraction from several combined neural networks. Int. J. Neural Syst. 11(3), 247–255 (2001)
Bologna, G.: A model for single and multiple knowledge based networks. Artificial Intelligence in Medicine 28, 141–163 (2003)
Zhou, Z.-H.: Extracting symbolic rules from trained neural network ensembles. AI Communications 16, 3–15 (2003)
Zhou, Z.-H.: Ensemble neural networks: Many could be better than all. Artificial Intelligence 137, 239–263 (2002)
Hara, A., Hayashi, Y.: A new neural data analysis approach using ensemble neural network rule extraction. In: Villa, A.E.P., Duch, W., Érdi, P., Masulli, F., Palm, G. (eds.) ICANN 2012, Part I. LNCS, vol. 7552, pp. 515–522. Springer, Heidelberg (2012)
Hansen, L.K., Salamon, P.: Neural network ensembles. IEEE Trans. Pattern Analysis and Machine Intelligence 12, 993–1001 (1990)
Adeodato, P.J.L., et al.: MLP ensembles improve long term prediction accuracy over single networks. Int. J. Forecasting 27, 661–671 (2011)
Hornik, K., et al.: Multilayer feedforward networks are universal approximators. Neural Netw. 2(5), 359–366 (1989)
Chorowski, J., Zurada, J.M.: Extracting Rules from Neural Networks as Decision Diagrams. IEEE Trans. Neural Netw. 22(12), 2435–2446 (2011)
Augasta, M.G., Kathirvalavakumar, T.: Reverse engineering the neural networks for rule extraction in classification problems. Neural Processing Letters 35, 131–150 (2012)
Barakat, N., Bradley, A.P.: Rule extraction from support vector machines: A review. Neurocomputing 74, 178–190 (2010)
Liu, S., et al.: Combined rule extraction and feature elimination in supervised classification. IEEE Trans. Nanobioscience 11(3), 228–236 (2012)
Zhou, Z.H.: Medical diagnosis with C4.5 rule preceded by artificial neural network ensemble. IEEE Trans. Information Technology in Biomedicine 7(1), 37–42 (2003)
Zhou, Z.H., et al.: A statistics based approach for extracting priority rules from trained neural networks. In: Proc. IEEE-INNS-ENNS Int. Conf. Neural Netw., Como, Italy, vol. 3, pp. 401–406 (2000)
We, X., et al.: Top 10 algorithms in data mining. Knowledge and Information Systems 14, 1–17 (2008)
Robert, B., et al.: Boosting of the margin: A new explanation for the effectiveness of voting methods. The Annals of Statistics 26(5), 1651–1686 (1998)
Duch, W., Setiono, R., Zurada, J.M.: Computational intelligence methods for rule-based data understanding. Proceedings of the IEEE 92(5), 771–805 (2004)
Tickle, A.B., et al.: The truth will come to light: Directions and challenges in extracting the knowledge embedded within trained artificial neural networks. IEEE Trans. Neural Netw. 9, 1057–1068 (1998)
Lin, C.T., Lee, C.S.G.: Neural fuzzy systems—A neuro–fuzzy synergism to intelligent systems. Prentice-Hall, Englewood Cliff (1996)
Khosravi, A., et al.: Comprehensive review of neural network-based prediction intervals and new advances. IEEE Trans. Neural Netw. 22(9), 1341–1356 (2011)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Hayashi, Y. (2013). Neural Data Analysis: Ensemble Neural Network Rule Extraction Approach and Its Theoretical and Historical Backgrounds. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing. ICAISC 2013. Lecture Notes in Computer Science(), vol 7894. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-38658-9_1
Download citation
DOI: https://doi.org/10.1007/978-3-642-38658-9_1
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-38657-2
Online ISBN: 978-3-642-38658-9
eBook Packages: Computer ScienceComputer Science (R0)