Abstract
Multiple Classifier Systems (MCSs) allow evaluation of the uncertainty of classification outcomes that is of crucial importance for safety critical applications. The uncertainty of classification is determined by a trade-off between the amount of data available for training, the classifier diversity and the required performance. The interpretability of MCSs can also give useful information for experts responsible for making reliable classifications. For this reason Decision Trees (DTs) seem to be attractive classification models for experts. The required diversity of MCSs exploiting such classification models can be achieved by using two techniques, the Bayesian model averaging and the randomised DT ensemble. Both techniques have revealed promising results when applied to real-world problems. In this paper we experimentally compare the classification uncertainty of the Bayesian model averaging with a restarting strategy and the randomised DT ensemble on a synthetic dataset and some domain problems commonly used in the machine learning community. To make the Bayesian DT averaging feasible, we use a Markov Chain Monte Carlo technique. The classification uncertainty is evaluated within an Uncertainty Envelope technique dealing with the class posterior distribution and a given confidence probability. Exploring a full posterior distribution, this technique produces realistic estimates which can be easily interpreted in statistical terms. In our experiments we found out that the Bayesian DTs are superior to the randomised DT ensembles within the Uncertainty Envelope technique.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Breiman, L., Friedman, J., Olshen, R. and Stone, C.: Classification and Regression Trees, Wadsworth, Belmont, CA, 1984.
Chipman, H., George, E. and McCullock, R.: Bayesian CART model search, J. Am. Stat. 93 (1998), 935–960.
Denison, D., Holmes, C., Malick, B. and Smith, A.: Bayesian Methods for Nonlinear Classification and Regression, Willey, 2002.
Kuncheva, L.: Combining Pattern Classifiers: Methods and Algorithms, Willey, 2004.
Dietterich, T.: Ensemble Methods in Machine Learning, in J. Kittler and F. Roli (eds.), First International Workshop on Multiple Classifier Systems, Lecture Notes in Computer Science, Springer Verlag, New York, 2000, pp. 1–15.
Fieldsend, J. E., Bailey, T. C., Everson, R. M., Krzanowski, W. J., Partridge, D. and Schetinin, V.: Bayesian Inductively Learned Modules for Safety Critical Systems. Computing Science and Statistics, 35th Symposium on the Interface, Salt Lake City, 2003.
Green, P.: Reversible Jump Markov chain Monte Carlo computation and Bayesian model determination, Biometrika 82 (1995), 711–732.
West, D. B.: Introduction to Graph Theory, 2nd edn, Prentice-Hall, Englewood Cliffs, 2000.
Blake, C. L. and Merz, C. J.: UCI Repository of machine learning databases http://www.ics.uci.edu/~mlearn/MLRepository.html. University of California, Department of Information and Computer Science, Irvine, CA, 1998.
Ripley, B.: Neural networks and related methods for classification, J. R. Stat. Soc., B 56(3) (1994), 409–456.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Schetinin, V., Fieldsend, J.E., Partridge, D. et al. Comparison of the Bayesian and Randomised Decision Tree Ensembles within an Uncertainty Envelope Technique. J Math Model Algor 5, 397–416 (2006). https://doi.org/10.1007/s10852-005-9019-9
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10852-005-9019-9