Abstract
This paper investigates the use of meta-learning to estimate the predictive accuracy of a classifier. We present a scenario where meta-learning is seen as a regression task and consider its potential in connection with three strategies of dataset characterization. We show that it is possible to estimate classifier performance with a high degree of confidence and gain knowledge about the classifier through the regression models generated. We exploit the results of the models to predict the ranking of the inducers. We also show that the best strategy for performance estimation is not necessarily the best one for ranking generation.
Chapter PDF
Similar content being viewed by others
Keywords
- Predictive Accuracy
- Performance Estimation
- Kernel Method
- Mean Absolute Deviation
- Normalise Mean Square Error
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
H. Bensusan. God doesn’t always shave with Occam’s Razor — learning when and how to prune. In Proceedings of the 10th European Conference on Machine Learning, pages 119–124, 1998.
C. Blake and C. Merz. UCI repository of machine learning databases, 1998. http://www.ics.uci.edu/.
W. Cohen. Fast efective rule induction. In Proceedings of the 12th International Conference on Machine Learning, pages 115–123. Morgan Kaufmann, 1995.
R. O. Duda and P. E. Hart. Pattern classi-cation and scene analysis. John Wiley, 1973.
Y. Freund and R. E. Schapire. Experiments with a new boosting algorithm. In Proceedings 13th International Conference on Machine Learning, pages 148–146, 1996.
J. Gama. Discriminant trees. In Proceedings of the 16th International Machine Learning Conference (ICML’99), pages 134–142, 1999.
J. Gama and P. Brazdil. Characterization of classification algorithms. In Proceedings of the 7th Portugese Conference in AI, EPIA 95, pages 83–102, 1995.
C. Giraud-Carrier et al. A meta-learning assistant for providing user support in data mining and machine learning, 1999-2001. http://www.metal-kdd.org.
A. Kalousis and T. Theoharis. Noemon: Design, implementation and performance results of an intelligent assistant for classifier selection. Intelligent Data Analysis, 3(5):319–337, 1999.
C. Köpf, C. Taylor, and J. Keller. Meta-analysis: from data characterisation for meta-learning to meta-regression. In P. Brazdil and A. Jorge, editors, PKDD’2000 Workshop on Data Mining, Decision Support, Meta-Learning and ILP, 2000.
D. Michie, D. J. Spiegelhalter, and C. C. Taylor, editors. Machine Learning, Neural and Statistical Classification. Ellis Horwood, 1994.
H. Neave and Worthington P. Distribution Free Tests. Unwin Hyman, London, UK, 1992.
B. Pfahringer, H. Bensusan, and C. Giraud-Carrier. Meta-learning by landmarking various learning algorithms. In Proceedings of the Seventeenth International Conference on Machine Learning, pages 743–750, 2000.
J. R. Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann, 1993.
J. R. Quinlan. Bagging, boosting, and C4. 5. In Proceedings of the Thirteenth National Conference on Artificial Intelligence and the Eighth Innovative Applications of Artificial Intelligence Conference, pages 725–730, 1996.
R. Quinlan. An overview of cubist. Rulequest Research, November 2000. http://www.rulequest.com/.
R.B. Rao, D. Gordon, and W. Spears. For every generalization action, is there really an equal and opposite reaction? In Proceedings of the 12th International Conference on Machine Learning, 1995.
C. Soares and P. Brazdil. Zoomed ranking: Selection of classification algrorithms based on relevant performance information. In Proceedings of the 4th European Conference on Principles of Data Mining and Knowledge Discovery, pages 126–135. Springer, 2000.
S. Y. Sohn. Meta analysis of classification algorithms for pattern recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(11), 1999.
L. Todorovski, P. Brazdil, and C. Soares. Report on the experiments with feature selection in meta-level learning. In Proceedings of the Workshop on Data Mining, Decision Support, Meta-learning and ILP at PKDD’2000, pages 27–39, 2000.
L. Torgo. Inductive Learning of Tree-based Regression Models. PhD thesis, Department of Computer Science, Faculty of Sciences, University of Porto, Porto, Portugal, September 1999.
D Wolpert. The existence of a priori distinctions between learning algorithms. Neural Computation, 8:1391–1420, 1996.
D Wolpert. The lack of a priori distinctions between learning algorithms. Neural Computation, 8: 1341–1390, 1996.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Bensusan, H., Kalousis, A. (2003). Estimating the Predictive Accuracy of a Classifier. In: De Raedt, L., Flach, P. (eds) Machine Learning: ECML 2001. ECML 2001. Lecture Notes in Computer Science(), vol 2167. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44795-4_3
Download citation
DOI: https://doi.org/10.1007/3-540-44795-4_3
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-42536-6
Online ISBN: 978-3-540-44795-5
eBook Packages: Springer Book Archive