Summary
This chapter provides an introduction to the main methods and issues in the combination of Artificial Neural Nets. A distinction is made between ensemble and modular modes of combination, and the two are then treated separately. The reasons for ensemble combination are considered, and an account is provided of the main methods for creating and combining ANNs in ensembles. This account is accompanied by a discussion of the relative effectiveness of these methods, in which the concepts of diversity and selection are explained. The review of modular combination outlines the main methods of creating and combining modules, depending on whether the relationship between the modules is co-operative, competitive, sequential or supervisory. An overivew of the chapters in the book forms the conclusion section.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
D.E. Eckhardt and L.D. Lee. A theoretical basis for the analysis of multiversion software subject to coincident errors. IEE Transactions on Software Engineering, SE-11(12), 1985.
A.J.C. Sharkey. On combining artificial neural nets. Connection. Science. Special Issue on Combining Artificial Neural: Ensemble Approaches, 8 (3 & 4): 299–314, 1996
L.K. Hansen and P. Salamon. Neural network ensembles. IEEE Transaction on Pattern Analysis and Machine Intelligence, 12 (10): 993–1000, 1990
C. Bishop. Neural Networks for Pattern Recognition. Clarendon Press, Oxford, 1 edition, 1995.
R.R. Brooks, S.S. Iyengar, and N.S.V. Rao. Sensor fusion survey: Sensors, statistics, signal processing and neural networks. In Neural Networks and Their Applications: NEURAP’97, pages 183–190. March 1997
J.A. Fodor. The Modularity of Mind: An Essay on Faculty Psychology. A Bradford Book, MIT Press, London, England, 1983
B. Grofman and G. Owen. Editors, Information Pooling and Group Decision Making. Jai Press Inc, Greenwich, Connecticut, 1986
J. von Neumann. Probabilistic logics and the synthesis of reliable organisms from unreliable components. In C.E. Shannon and J. McCarthy, editors, Automata Studies, pages 43–98. Princeton University Press, 1956
J.M. Bates and C.W.J. Granger. The combination of forecasts. Operations Research Quarterly, 20: 451–468, 1969
N.S.V Rao. Fusion rule estimation in multiple sensor systems with unknown noise densities. Journal of Franklin Institute, 331B (5): 509–530, 1995
A. Avizienis and J.P.J Kelly. Fault diagnosis by design diversity: Concepts and experiments. IEEE Comput, 17: 67–80, 1984
S. Geman, E. Bienenstock, and R. Doursat. Neural networks and the bias/variance dilemma. Neural Computation, 4 (1): 1–58, 1992
B. Parmanto, P.W. Munro, and H.R. Doyle. Reducing variance of committee prediction with resampling techniques. Connection Science. Special Issue on Combining Artificial Neural: Ensemble Approaches, 8 (3 & 4): 405–426, 1996
N. Intrator and Y. Raviv. Bootstrapping with noise: an effective regularization technique. Connection Science. Special Issue on Combining Artificial Neural Nets: Ensemble Approaches, 8 (3 &4): 355–372, 1996
B. Rosen. Ensemble learning using decorrelated neural networks. Connection Science. Special Issue on Combining Artificial Neural: Ensemble Approaches, 8 (3 & 4): 373–384, 1996
A. Krogh and J. Vedelsby. Neural network ensembles, cross validation and active learning. In G. Tesauro, D.S. Touretzky, and T.K. Leen, editors, Advances in Neural Information Processing Systems 7. MIT Press, 1995
K. Kim and E.B. Bartlett. Error estimation by series association for neural network systems. Neural Computation, 7: 799–808, 1995
G. Rogova. Combining the results of several neural network classifiers. Neural Networks, 7 (5): 777–781, 1994
D.H. Wolpert. Stacked generalization. Neural Networks, 5: 241–259, 1992
R.A. Jacobs. Methods for combining experts’ probability assessments. Neural Computation, 7: 867–888, 1995
B.Littlewood and D.R. Miller. Conceptual modelling of coincident failures in multiversion software. IEE Transactions on Software Engineering, 15(12), 1986.
J.G. Gueard Jr. and R.T. Clemen. Collinearity and the use of latent root regression for combining gnp forecasts. Journal of Forecasting, 8: 231–238, 1989
A.J.C. Sharkey and N.E. Sharkey. How to improve the reliability of artificial neural networks. Technical Report CS-95–11, University of Sheffield, Department of Computer Science, University of Sheffield, 1995
A.J.C. Sharkey and N.E. Sharkey. Combining diverse neural nets. The Knowledge Engineering Review, 12 (3): 231–247, 1997
L. Breiman. Bagging predictors. Machine Learning, 26 (2): 123–140, 1996
B. Efron and R. Tibshirani. An Introduction to the Bootstrap. Chapman and Hall, New York, 1993
A.J.C. Sharkey, N.E. Sharkey, and G.O. Chandroth. Neural nets and diversity. Neural Computing and Applications, 4: 218–227, 1996
K. Tumer and J. Ghosh. Error correlation and error reduction in ensemble classifiers. Connection Science. Special Issue on Combining Artificial Neural: Ensemble Approaches, 8 (3 &4): 385–404, 1996
R.E. Schapire. The strength of weak learnability. Machine Learning, 5: 197–227, 1990
H. Drucker, C. Cortes, L.D. Jackel, Y. LeCun, and V. Vapnik. Boosting and other ensemble methods. Neural Computation, 6 (6): 1289–1301, 1994
Y. Freund and R. Schapire. Experiments with a new boosting algorithm. In Proceedings of the Thirteenth International Conference on Machine Learning, pages 149–156. Morgan Kaufmann, 1996
L. Breiman. Arcing classifiers. Technical Report 460, Statistics Department, University of California, Berkeley, 1996
C. Genest and J.V. Zidek. Combining probability distributions: a critique and annotated bibliography. Statistical Science, 1: 114–148, 1996
L. Xu, A. Krzyzak, and C.Y. Suen. Methods of combining multiple classifiers and their applications to handwriting recognition. IEEE Transactions on Systems, Man and Cybernetics, 22 (3): 418–435, May 1992
Peter A. Zhilkin and Ray L. Somorjai. Application of several methods of classification fusion to magnetic resonance spectra. Connection Science. Special issue on Combining Artificial Neural Nets: Ensemble Approaches, 8 (3 & 4): 427–442, 1996
M.P. Perrone and L.N. Cooper. When networks disagree: Ensemble methods for hybrid neural networks. In R.J. Mammone, editor, Neural Networks for Speech and Image Processing, chapter 10. Chapman-Hall, 1993
S. Hashem. Optimal Linear Combinations of Neural Networks. PhD thesis, School of Industrial Engineering, Purdue University, 1993
S. Hashem. Effects of collinearity on combining neural networks. Connection Science: Special Issue on Combining Artificial Neural Networks: Ensemble Approaches, 8 (3 &4): 315–336, 1996
S. Hashem. Optimal linear combinations of neural networks. Neural Networks, 10 (4): 599–614, 1997
K. AI-Ghoneim and B.V.K. Vijaya Kumar. Learning ranks with neural networks. In Applications and Science of Artificial Neural Networks: Proceedings of the SPIE, volume 2492, pages 446–464. 1995
K. Tumer and J. Ghosh. Order statistics combiners for neural classifiers. In Proceedings of the World Congress on Neural Networks, pages I:31–34. INNS„Press, Washington DC, 1995
L. Breiman. Stacked regression. Technical Report 367, Statistics Department, University of California, Berkeley, 1993
M. LeBlanc and R. Tibshirani. Combining estimates in regression and classification. Paper available from ftp site: ustat.toronto.edu, 1993
J.F. Kolen and J.B. Pollack. Backpropagation is sensitive to initial conditions. Technical Report TR 90-JK-BPSIC, 1990
J. Denker, D. Schwartz, B. Wittner, S. Solla, R. Howard, L. Jackel, and J. Hopfield. Large automatic learning, rule extraction and generalisation. Complex Systems, 1: 877–922, 1987
N.E. Sharkey and A.J.C. Sharkey. An analysis of catastrophic interference. Connection Science, 1995(7):313–341, 3 
mp; 4 1995
P. Sollich and A. Krogh. Learning with ensembles: How overfitting can be useful. In D.S. Touretzky, M.C. Mozer, and M.E. Hasselmo, editors, Advances in Neural Information Processing Systems 8. MIT Press, 1996
D.W. Optiz and J.W. Shavlik. Actively searching for an effective neural network ensemble. Connection Science. Special Issue on Combining Artificial Neural: Ensemble Approaches, 8 (3 & 4): 337–354, 1996
R. Murray-Smith and T.A. Johansen. Multiple Model Approaches to Modelling and Control. Taylor and Francis, UK, 1997
K.S. Narendra, J. Balakrishnan, and K. Ciliz. Adaptation and learning using multiple models, swtiching and tuning. IEEE Control Systems Magazine, pages 37–51, June 1995
R.A. Brooks. A robust layered control system for a mobile robot. IEEE Journal of Robotics and Automation, RA-2: 14–23, 1986
N.E. Sharkey. Artificial neural networks for coordination and control: the portability of experiential representations. Robotics and Autonomous Systems, 22: 345–360, 1997
N.E. Sharkey and A.J.C. Sharkey. A modular design for connectionist parsing. In M.F.J. Drosaers and A. Nijholt, editors, Proceedings of Workshop on Language Technology, pages 87–96. 1992
T. Catfolis and K. Meert. Hybridization and specialization of real-time recurrent learning-based neural networks. Connection Science. Special Issue on Combining Artificial Neural Nets: Modular Approaches, 9 (1): 51–70, 1997
Y. Bennani and P. Gallinari. Task decomposition through a modular connectionist architecture: A talker identification system. In I. Aleksander and J. Taylor, editors, Third International Conference on Artificial Neural Networks, volume 1, pages 783–786. North-Holland, Amsterdam, 1992
W. Marslen-Wilson and L.K. Tyler. Against modularity. In J.L. Garfield, editor, Modularity in Knowledge Representation and Natural Language, pages 37–62. Bradford Book, MIT Press, 1987
T.J.Prescott. Spatial representation for navigation in animats. Adaptive Behaviour, 4(2), 1996.
P. Gallinari. Modular neural net systems: training of. In M.A. Arbib, editor, The Handbook of Brain Theory and Neural Networks, pages 582–585. Bradford Books: MIT Press, 1995
T. Hrycej. Modular Learning in Neural Networks. John Wiley, Chichester, 1992
L.Y. Pratt, J. Mostow, and C.A. Kamm. Direct transfer of learned information among neural networks. In Proceedings of the Ninth National Conference on Artificial Intelligence (AAI-91), pages 584–589. Anaheim, CA, 1991
B. Lu and M. Ito. Task decomposition and module combination based on class relations: A modular neural network for pattern classification. Technical Report BMC TR-98–1, Bio-Mimetic Control Research Centre, Nagoya, Japan, 1998
J.B. Hampshire and A.H. Waibel. The M,eta-P,i network: Building distributed representations for robust multisource pattern recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14 (7): 751–769, 1992
A. Waibel, H. Sawai, and K. Shikano. Modularity and scaling in large phonemic neural networks. IEEE Transactions on Acoustics, Speech and Signal Processing, 37: 1888–1898, 1989
L. Ohno-Machado and M.A. Musen. Modular neural networks for medical prognosis: Quantifying the benefits of combining neural networks for survival prediction. Connection Science. Special Issue on Combining Artificial Neural Nets: Modular Approaches, 9 (1): 71–86, 1997
W.G. Baxt. Improving the accuracy of an artificial neural network using multiple differently trained networks. Neural Computation, 4: 772–780, 1992
R. Anand, K.G. Mehrotra, C.K. Mohan, and S. Ranka. Efficient classification for multiclass problems using modular neural networks. IEEE Trans Neural Networks, 6: 117–124, 1995
R.A. Jacobs, M.I. Jordan, S.J. Nowlan, and G.E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3: 79–97, 1991
M.I. Jordan and R.A. Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural Computation, 6 (2): 181–214, 1994
F. Peng, R.A. Jacobs, and M.A. Tanner. Bayesian inference in mixtures-ofexperts and hierarchical mistures-of-experts models with an application to speech recognition. Journal of the American Statistical Association, 1995
R. Luo and M. Kay. Data fusion and sensor integration: State-of-the-art 1990’s. In Abidi and Gonzales, editors, Data Fusion in Robotics and Machine Intelligence, pages 7–136. Academic Press, Boston, 1992
C. McCormack. Adaptation of learning rule parameters using a meta neural network. Connection Science. Special Issue on Combining Artificial Neural Nets: Modular Approaches, 9 (1): 123–136, 1997
R.R. Murphy. Biological and cognitive foundations of intelligent sensor fusion. IEEE Transactions on Systems, Man and Cybernetics, pages 42–51, January 1996
A.J.C. Sharkey, N.E. Sharkey, and S.S. Cross. Adapting an ensemble approach for the diagnosis of breast cancer. In Proceedings of ICANN98. Springer-Verlag, 1998
D.J. Chalmers. Syntactic transformation on distributed representations. Connection Science, 2: 53–62, 1990
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1999 Springer-Verlag London Limited
About this chapter
Cite this chapter
Sharkey, A.J.C. (1999). Multi-Net Systems. In: Sharkey, A.J.C. (eds) Combining Artificial Neural Nets. Perspectives in Neural Computing. Springer, London. https://doi.org/10.1007/978-1-4471-0793-4_1
Download citation
DOI: https://doi.org/10.1007/978-1-4471-0793-4_1
Publisher Name: Springer, London
Print ISBN: 978-1-85233-004-0
Online ISBN: 978-1-4471-0793-4
eBook Packages: Springer Book Archive