Abstract
Decision Trees are considered to be one of the most popular approaches for representing classifiers. Researchers from various disciplines such as statistics, machine learning, pattern recognition, and Data Mining have dealt with the issue of growing a decision tree from available data. This paper presents an updated survey of current methods for constructing decision tree classifiers in a top-down manner. The chapter suggests a unified algorithmic framework for presenting these algorithms and describes various splitting criteria and pruning methodologies.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Keywords
References
Almuallim H., An Efficient Algorithm for Optimal Pruning of Decision Trees. Artificial Intelligence 83(2): 347–362, 1996.
Almuallim Hv and Dietterich T.G., Learning Boolean concepts in the presence of many irrelevant features. Artificial Intelligence, 69:1–2, 279–306, 1994.
Alsabti K., Ranka S. and Singh V., CLOUDS: A Decision Tree Classifier for Large Datasets, Conference on Knowledge Discovery and Data Mining (KDD-98), August 1998.
Attneave F, Applications of Information Theory to Psychology. Holt, Rinehart and Winston, 1959.
Baker E., and Jain A. K., On feature ordering in practice and some finite sample effects. In Proceedings of the Third International Joint Conference on Pattern Recognition, pages 45–49, San Diego, CA, 1976.
BenBassat M., Myopic policies in sequential classification. IEEE Trans. on Computing, 27(2): 170–174, February 1978.
Bennett X. and Mangasarian O.L., Multicategory discrimination via linear programming. Optimization Methods and Software, 3:29–39, 1994.
Bratko I., and Bohanec M., Trading accuracy for simplicity in decision trees, Machine Learning 15: 223–250, 1994.
Breiman L., Friedman J., Olshen R., and Stone C.. Classification and Regression Trees. Wadsworth Int. Group, 1984.
Brodley C. E. and Utgoff. P. E., Multivariate decision trees. Machine Learning, 19:45–77, 1995.
Buntine W., Niblett T, A Further Comparison of Splitting Rules for Decision-Tree Induction. Machine Learning, 8: 75–85, 1992.
Catlett J., Mega induction: Machine Learning on Vary Large Databases, PhD, University of Sydney, 1991.
Chan P.K. and Stolfo S.J, On the Accuracy of Meta-learning for Scalable Data Mining, J. Intelligent Information Systems, 8:5–28, 1997.
Crawford S. L., Extensions to the CART algorithm. Int. J. of ManMachine Studies, 31(2): 197–217, August 1989.
Dietterich, T. G., Kearns, M., and Mansour, Y., Applying the weak learning framework to understand and improve C4.5. Proceedings of the Thirteenth International Conference on Machine Learning, pp. 96–104, San Francisco: Morgan Kaufmann, 1996.
Duda, R., and Hart, P., Pattern Classification and Scene Analysis, New-York, Wiley, 1973.
Esposito E, Malerba D. and Semeraro G., A Comparative Analysis of Methods for Pruning Decision Trees. EEE Transactions on Pattern Analysis and Machine Intelligence, 19(5):476–492, 1997.
Fayyad U., and Irani K. B., The attribute selection problem in decision tree generation. In proceedings of Tenth National Conference on Artificial Intelligence, pp. 104–110, Cambridge, MA: AAAI Press/MIT Press, 1992.
Ferri C, Flach P., and Hernandez-Orallo J., Learning Decision Trees Using the Area Under the ROC Curve. In Claude Sammut and Achim Hoffmann, editors, Proceedings of the 19th International Conference on Machine Learning, pp. 139–146. Morgan Kaufmann, July 2002
Fifield D. J., Distributed Tree Construction From Large Datasets, Bachelor’s Honor Thesis, Australian National University, 1992.
Freitas X., and Lavington S. H., Mining Very Large Databases With Parallel Processing, Kluwer Academic Publishers, 1998.
Friedman J. H., A recursive partitioning decision rule for nonparametric classifiers. IEEE Trans, on Comp., C26:404–408, 1977.
Friedman, J. H., “Multivariate Adaptive Regression Splines”, The Annual Of Statistics, 19, 1–141, 1991.
Gehrke J., Ganti V., Ramakrishnan R., Loh W., BOAT-Optimistic Decision Tree Construction. SIGMOD Conference 1999: pp. 169–180, 1999.
Gehrke J., Ramakrishnan R., Ganti V., RainForest-A Framework for Fast Decision Tree Construction of Large Datasets, Data Mining and Knowledge Discovery, 4,2/3) 127–162, 2000.
Gelfand S. B., Ravishankar C. S., and Delp E. J., An iterative growing and pruning algorithm for classification tree design. IEEE Transaction on Pattern Analysis and Machine Intelligence, 13(2): 163–174, 1991.
Gillo M. W., MAID: A Honeywell 600 program for an automatised survey analysis. Behavioral Science 17: 251–252, 1972.
Hancock T. R., Jiang T, Li M., Tromp J., Lower Bounds on Learning Decision Lists and Trees. Information and Computation 126(2): 114–122, 1996.
Holte R. C, Very simple classification rules perform well on most commonly used datasets. Machine Learning, 11:63–90, 1993.
Hyafil L. and Rivest R.L., Constructing optimal binary decision trees is NP-complete. Information Processing Letters, 5(1):15–17, 1976
Janikow, C.Z., Fuzzy Decision Trees: Issues and Methods, IEEE Transactions on Systems, Man, and Cybernetics, Vol. 28, Issue 1, pp. 1–14. 1998.
John G. H., Robust linear discriminant trees. In D. Fisher and H. Lenz, editors, Learning From Data: Artificial Intelligence and Statistics V, Lecture Notes in Statistics, Chapter 36, pp. 375–385. Springer-Verlag, New York, 1996.
Kass G. V, An exploratory technique for investigating large quantities of categorical data. Applied Statistics, 29(2):119–127, 1980.
Kearns M. and Mansour Y., A fast, bottom-up decision tree pruning algorithm with near-optimal generalization, in J. Shavlik, ed., ‘Machine Learning: Proceedings of the Fifteenth International Conference’, Morgan Kaufmann Publishers, Inc., pp. 269–277, 1998.
Kearns M. and Mansour Y, On the boosting ability of top-down decision tree learning algorithms. Journal of Computer and Systems Sciences, 58(1): 109–128, 1999.
Kohavi R. and Sommerfield D., Targeting business users with decision table classifiers, in R. Agrawal, P. Stolorz & G. Piatetsky-Shapiro, eds, ‘Proceedings of the Fourth International Conference on Knowledge Discovery and Data Mining’, AAAI Press, pp. 249–253, 1998.
Langley, P. and Sage, S., Oblivious decision trees and abstract cases. in Working Notes of the AAAI-94 Workshop on Case-Based Reasoning, pp. 113–117, Seattle, WA: AAAI Press, 1994.
Last, M., Maimon, O. and Minkov, E., Improving Stability of Decision Trees, International Journal of Pattern Recognition and Artificial Intelligence, 16: 2, 145–159, 2002.
Li X. and Dubes R. C, Tree classifier design with a Permutation statistic, Pattern Recognition 19:229–235, 1986.
Lim X., Loh W.Y., and Shih X., A comparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms. Machine Learning 40:203–228, 2000.
Lin Y. K. and Fu K., Automatic classification of cervical cells using a binary tree classifier. Pattern Recognition, 16(1):69–80, 1983.
Loh W.Y.,and Shih X., Split selection methods for classification trees. Statistica Sinica, 7: 815–840, 1997.
Loh W.Y. and Shih X., Families of splitting criteria for classification trees. Statistics and Computing 9:309–315, 1999.
Loh W.Y. and Vanichsetakul N., Tree-structured classification via generalized discriminant Analysis. Journal of the American Statistical Association, 83: 715–728, 1988.
Lopez de Mantras R., A distance-based attribute selection measure for decision tree induction, Machine Learning 6:81–92, 1991.
Lubinsky D., Algorithmic speedups in growing classification trees by using an additive split criterion. Proc. AI&Statistics93, pp. 435–444, 1993.
Martin J. K., An exact probability metric for decision tree splitting and stopping. An Exact Probability Metric for Decision Tree Splitting and Stopping, Machine Learning, 28,2–3):257–291, 1997.
Mehta M., Rissanen J., Agrawal R., MDL-Based Decision Tree Pruning. KDD 1995: pp. 216–221, 1995.
Mehta M., Agrawal R. and Rissanen J., SLIQ: A fast scalable classifier for Data Mining: In Proc. If the fifth Int’l Conference on Extending Database Technology (EDBT), Avignon, France, March 1996.
Mingers J., An empirical comparison of pruning methods for decision tree induction. Machine Learning, 4(2):227–243, 1989.
Morgan J. N. and Messenger R. C, THAID: a sequential search program for the analysis of nominal scale dependent variables. Technical report, Institute for Social Research, Univ. of Michigan, Ann Arbor, MI, 1973.
Muller W., and Wysotzki F, Automatic construction of decision trees for classification. Annals of Operations Research, 52:231–247, 1994.
Murthy S. K., Automatic Construction of Decision Trees from Data: A Multi-Disciplinary Survey. Data Mining and Knowledge Discovery, 2(4):345–389, 1998.
Naumov G.E., NP-completeness of problems of construction of optimal decision trees. Soviet Physics: Doklady, 36(4):270–271, 1991.
Niblett T. and Bratko I., Learning Decision Rules in Noisy Domains, Proc. Expert Systems 86, Cambridge: Cambridge University Press, 1986.
Olaru C, Wehenkel L., A complete fuzzy decision tree technique, Fuzzy Sets and Systems, 138(2):221–254, 2003.
Pagallo, G. and Huassler, D., Boolean feature discovery in empirical learning, Machine Learning, 5(1): 71–99, 1990.
Peng Y., Intelligent condition monitoring using fuzzy inductive learning, Journal of Intelligent Manufacturing, 15(3): 373–380, June 2004.
Quinlan, J.R., Induction of decision trees, Machine Learning 1, 81–106, 1986.
Quinlan, J.R., Simplifying decision trees, International Journal of Man-Machine Studies, 27, 221–234, 1987.
Quinlan, J.R., Decision Trees and Multivalued Attributes, J. Richards, ed., Machine Intelligence, V. 11, Oxford, England, Oxford Univ. Press, pp. 305–318, 1988.
Quinlan, J. R., Unknown attribute values in induction. In Segre, A. (Ed.), Proceedings of the Sixth International Machine Learning Workshop Cornell, New York. Morgan Kaufmann, 1989.
Quinlan, J. R., C4.5: Programs for Machine Learning, Morgan Kaufmann, Los Altos, 1993.
Quinlan, J. R. and Rivest, R. L., Inferring Decision Trees Using The Minimum Description Length Principle. Information and Computation, 80:227–248, 1989.
Rastogi, R., and Shim, K., PUBLIC: A Decision Tree Classifier that Integrates Building and Pruning, Data Mining and Knowledge Discovery, 4(4):315–344, 2000.
Rissanen, J., Stochastic complexity and statistical inquiry. World Scientific, 1989.
Rounds, E., A combined non-parametric approach to feature selection and binary decision tree design, Pattern Recognition 12, 313–317, 1980.
Schlimmer, J. C., Efficiently inducing determinations: A complete and systematic search algorithm that uses optimal pruning. In Proceedings of the 1993 International Conference on Machine Learning: pp 284–290, San Mateo, CA, Morgan Kaufmann, 1993.
Sethi, K., and Yoo, J. H., Design of multicategory, multifeature split decision trees using perceptron learning. Pattern Recognition, 27(7):939–947, 1994.
Shafer, J. C, Agrawal, R. and Mehta, M., SPRINT: A Scalable Parallel Classifier for Data Mining, Proc. 22nd Int. Conf. Very Large Databases, T. M. Vijayaraman and Alejandro P. Buchmann and C. Mohan and Nandlal L. Sarda (eds), 544–555, Morgan Kaufmann, 1996.
Sklansky, J. and Wassel, G. N., Pattern classifiers and trainable machines. SpringerVerlag, New York, 1981.
Sonquist, J. A., Baker E. L., and Morgan, J. N., Searching for Structure. Institute for Social Research, Univ. of Michigan, Ann Arbor, MI, 1971.
Taylor P. C, and Silverman, B. W., Block diagrams and splitting criteria for classification trees. Statistics and Computing, 3(4):147–161, 1993.
Utgoff, P. E., Perceptron trees: A case study in hybrid concept representations. Connection Science, 1(4):377–391, 1989.
Utgoff, P. E., Incremental induction of decision trees. Machine Learning, 4: 161–186, 1989.
Utgoff, P. E., Decision tree induction based on efficient tree restructuring, Machine Learning 29,1):5–44, 1997.
Utgoff, P. E., and Clouse, J. A., A Kolmogorov-Smirnoff Metric for Decision Tree Induction, Technical Report 96-3, University of Massachusetts, Department of Computer Science, Amherst, MA, 1996.
Wallace, C. S., and Patrick J., Coding decision trees, Machine Learning 11: 7–22, 1993.
Zantema, H., and Bodlaender H. L., Finding Small Equivalent Decision Trees is Hard, International Journal of Foundations of Computer Science, 11(2): 343–354, 2000.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer Science+Business Media, Inc.
About this chapter
Cite this chapter
Rokach, L., Maimon, O. (2005). Decision Trees. In: Maimon, O., Rokach, L. (eds) Data Mining and Knowledge Discovery Handbook. Springer, Boston, MA. https://doi.org/10.1007/0-387-25465-X_9
Download citation
DOI: https://doi.org/10.1007/0-387-25465-X_9
Publisher Name: Springer, Boston, MA
Print ISBN: 978-0-387-24435-8
Online ISBN: 978-0-387-25465-4
eBook Packages: Computer ScienceComputer Science (R0)