Abstract
Stability has been explored to study the performance of learning algorithms in recent years and it has been shown that stability is sufficient for generalization and is sufficient and necessary for consistency of ERM in the general learning setting. Previous studies showed that AdaBoost has almost-everywhere uniform stability if the base learner has L 1 stability. The L 1 stability, however, is too restrictive and we show that AdaBoost becomes constant learner if the base learner is not real-valued learner. Considering that AdaBoost is mostly successful as a classification algorithm, stability analysis for AdaBoost when the base learner is not real-valued learner is an important yet unsolved problem. In this paper, we introduce the approximation stability and prove that approximation stability is sufficient for generalization, and sufficient and necessary for learnability of AERM in the general learning setting. We prove that AdaBoost has approximation stability and thus has good generalization, and an exponential bound for AdaBoost is provided.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Alon, N., Ben-David, S., Cesa-Bianchi, N., Haussler, D.: Scale-sensitive dimensions, uniform convergence, and learnablity. J. ACM 44(4), 615–631 (1997)
Bickel, J.P., Ritov, Y., Zakai, A.: Some theory for generalized boosting algorithms. J. Mach. Learn. Res. 7, 705–732 (2006)
Bousquet, O., Elisseeff, A.: Stability and generalization. J. Mach. Learn. Res. 2, 499–526 (2002)
Breiman, L.: Prediction games and arcing classifiers. Neural Comput. 11(7), 1493–1517 (1999)
Devroye, L.P., Wagner, T.J.: Distribution-free performance bounds for potential function rules. IEEE Trans. Inform. Theory 25, 601–604 (1979)
Freund, Y., Schapire, R.E.: Game theory, on-line prediction and boosting. In: Proc. of 9th COLT, Desenzano sul Garda, Italy, pp. 325–332 (1996)
Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. 55(1), 119–139 (1997)
Freund, Y., Schapire, R.E.: Response to “Evidence contrary to the statistical view of Boosting”. J. Mach. Learn. Res. 9, 171–174 (2008)
Friedman, J., Hastie, T., Tibshirani, R.: Addtive logistic regression: A statistical view of boosting (with discussion). Ann. Statist. 28, 337–407 (2000)
Kearns, M., Ron, D.: Algorithmic stability and sanity-check bounds for leave-one-out cross-validation. Neural Comput. 11, 1427–1453 (1999)
Kutin, S., Niyogi, P.: The interaction of stability and weakness in Adaboost. Technical Report 30, Department of Computer Science, University of Chicago, Chicago, IL (2001)
Kutin, S., Niyogi, P.: Almost-everywhere algorithmic stability and generalization error. In: Proc. of 18th UAI, Edmonton, Canada, pp. 275–282 (2002)
Mason, L., Baxter, J., Bartlett, P.L., Freman, M.R.: Boosting algorithms as gradient descent. In: Solla, S.A., Leen, T.K., Müller, K.-R. (eds.) Advances in NIPS, vol. 12, pp. 512–518. MIT Press, Cambridge (1999)
McDiarmid, C.: On the method of bounded differences. In: Surveys in Combinatorics, pp. 148–188. Cambridge University Press, Cambridge (1989)
Mease, D., Wyner, A.: Evidence contrary to the statistical view of boosting with discussion. J. Mach. Learn. Res. 9, 131–201 (2008)
Mukherjee, S., Niyogi, P., Poggio, T., Rifkin, R.: Learning theory: Stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimzation. Adv. Comput. Math. 25, 161–193 (2006)
Rakhlin, A., Mukherjee, S., Poggio, T.: Stability results in learning theory. Anal. Appl. 4, 397–417 (2005)
Rätsch, G., Onoda, T., Müller, K.R.: Soft margins for Adaboost. Mach. Learn. 42, 287–320 (2001)
Reyzin, L., Schapire, R.E.: How boosting the margin can also boost classifier complexity. In: Proc. of 23rd ICML, Pittsburgh, PA, pp. 753–760 (2006)
Rudin, C., Schapire, R.E., Daubechies, I.: Precise statements of convergence for adaboost and arc-gv. In: Proc. of AMS-IMS-SIAM Joint Summer Research Conference: Machine learning, Statistics and Discovery, Snowbird, Utah, pp. 131–145 (2007)
Schapire, R., Freund, Y., Bartlett, P.L., Lee, W.: Boosting the margin: A new explanation for the effectives of voting methods. Ann. Statist. 26, 1651–1686 (1998)
Shalev-Shwartz, S., Shamir, O., Srebro, N., Sridharan, K.: Learnability and stability in the general learning setting. In: Proc. of 22nd COLT, Montreal, Canada (2009)
Shalev-Shwartz, S., Shamir, O., Srebro, N., Sridharan, K.: Stochastic convex optimation. In: Proc. of 22nd COLT, Montreal, Canada (2009)
Vapnik, V.N.: Statistical Learning Theory. John Wiley & Sons, New York (1998)
Wang, L., Sugiyama, M., Yang, C., Zhou, Z.-H., Feng, J.: On the margin explanation of boosting algorithm. In: Proc. of 21st COLT, Helsinki, Finland, pp. 479–490 (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Gao, W., Zhou, ZH. (2010). Approximation Stability and Boosting. In: Hutter, M., Stephan, F., Vovk, V., Zeugmann, T. (eds) Algorithmic Learning Theory. ALT 2010. Lecture Notes in Computer Science(), vol 6331. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-16108-7_9
Download citation
DOI: https://doi.org/10.1007/978-3-642-16108-7_9
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-16107-0
Online ISBN: 978-3-642-16108-7
eBook Packages: Computer ScienceComputer Science (R0)