Abstract
Vapnik-Chervonenkis (VC) bounds play an important role in statistical learning theory as they are the fundamental result which explains the generalization ability of learning machines. There have been consequent mathematical works on the improvement of VC rates of convergence of empirical means to their expectations over the years. The result obtained by Talagrand in 1994 seems to provide more or less the final word to this issue as far as universal bounds are concerned. Though for fixed distributions, this bound can be practically outperformed. We show indeed that it is possible to replace the 2∈ 2 under the exponential of the deviation term by the corresponding Cramér transform as shown by large deviations theorems. Then, we formulate rigorous distributionsensitive VC bounds and we also explain why these theoretical results on such bounds can lead to practical estimates of the effective VC dimension of learning structures.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Alexander, K.: Probability Inequalities for Empirical Processes and a Law of the Iterated Logarithm. Annals of Probability 4 (1984) 1041–1067
Azencott, R.: Grandes Déviations, in Hennequin, P.L. (ed.): Ecole d’Eté de Probabilités de Saint-Flour VIII-1978. Lecture Notes in Mathematics, Vol. 774. Springer-Verlag, Berlin Heidelberg New York (1978)
Bartlett, P., Lugosi, G.: An Inequality for Uniform Deviations of Sample Averages from their Means. To appear (1998)
Cohn, D., Tesauro, G.: How Tight Are the Vapnik-Chervonenkis Bounds? Neural Computation 4 (1992) 249–269
Cohn, D.: Separating Formal Bounds from Practical Performance in Learning Systems. PhD thesis, University of Washington (1992)
Devroye, L.: Bounds for the Uniform Deviation of Empirical Measures. Journal of Multivariate Analysis 12 (1982) 72–79
Devroye, L., Györff, L., Lugosi, G.: A Probabilistic Theory of Pattern Recognition. Springer-Verlag, Berlin Heidelberg New York (1996)
Haussler, D.: Sphere Packing Numbers for Subsets of the Boolean n-Cube with Bounded Vapnik-Chervonenkis Dimension. Journal of Combinatorial Theory, Series A 69 (1995) 217–232
Haussler, D., Kearns, M., Seung, H.S., Tishby, N.: Rigorous Learning Curve Bounds from Statistical Mechanics. Machine Learning (1996) 195–236
Hoeffding, W.: Probability Inequalities for Sums of Bounded Random Variables. Journal of the American Statistical Association 58 (1963) 13–30
Kearns, M.J., Vazirani, U.V.: An Introduction to Computational Learning Theory. MIT Press, Cambridge Massachussets (1994)
Ledoux, M., Talagrand, M.: Probability in Banach Spaces. Springer-Verlag, Berlin Heidelberg New York (1992)
Lugosi, G.: Improved Upper Bounds for Probabilities of Uniform Deviations. Statistics and Probability Letters 25 (1995) 71–77
Massart, P.: Rates of Convergence in the Central Limit Theorem for Empirical Processes. Annales de l’lnstitut Henri Poincaré, Vol. 22, No. 4 (1986) 381–423
Parrondo, J.M.R., Van den Broeck, C.: Vapnik-Chervonenkis Bounds for Generalization. J. Phys. A: Math. Gen. 26 (1993) 2211–2223
Pollard, D.: Convergence of Stochastic Processes. Springer-Verlag, Berlin Heidelberg New York (1984)
Schuurmans, D.E.: Effective Classification Learning. PhD thesis, University of Toronto (1996)
Stroock, D.W.: Probability Theory, an Analytic View. Cambridge University Press (1993)
Talagrand, M.: Sharper Bounds for Gaussian and Empirical Processes. The Annals of Probability, Vol. 22, No. 1 (1994) 28–76
van der Vaart, A.W., Wellner, J.A.: Weak Convergence and Empirical Processes. Springer-Verlag, Berlin Heidelberg New York (1996)
Vapnik, V.N., Chervonenkis, A. Ya.: On the Uniform Convergence of Relative Frequencies of Events to their Probabilities. Theory of Probability and its Applications, Vol.XVI, No. 2 (1971) 264–280
Vapnik, V.N., Chervonenkis, A. Ya.: Necessary and Sufficient Conditions for the Uniform Convergence of Means to their Expectations. Theory of Probability and its Applications, Vol. XXVI, No. 3 (1981) 532–553
Vapnik, V. N.: Estimation of Dependences Based on Empirical Data. Springer-Verlag, Berlin Heidelberg New York (1982)
Vapnik, V.N., Levin, E., Le Cun, Y.: Measuring the VC Dimension of a Learning Machine. Neural Computation 6 (1994) 851–876
Vapnik, V.N.: The Nature of Statistical Learning Theory. Springer-Verlag, Berlin Heidelberg New York (1995)
Vayatis, N.: Learning Complexity and Pattern Recognition. PhD thesis, Ecole Polytechnique. To appear (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1999 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Vayatis, N., Azencott, R. (1999). Distribution-Dependent Vapnik-Chervonenkis Bounds. In: Fischer, P., Simon, H.U. (eds) Computational Learning Theory. EuroCOLT 1999. Lecture Notes in Computer Science(), vol 1572. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-49097-3_18
Download citation
DOI: https://doi.org/10.1007/3-540-49097-3_18
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-65701-9
Online ISBN: 978-3-540-49097-5
eBook Packages: Springer Book Archive