Abstract
The Vapnik-Chervonenkis (V-C) dimension is an important combinatorial tool in the analysis of learning problems in the PAC framework. For polynomial learnability, we seek upper bounds on the V-C dimension that are polynomial in the syntactic complexity of concepts. Such upper bounds are automatic for discrete concept classes, but hitherto little has been known about what general conditions guarantee polynomial bounds on V-C dimension for classes in which concepts and examples are represented by tuples of real numbers. In this paper, we show that for two general kinds of concept class the V-C dimension is polynomially bounded in the number of real numbers used to define a problem instance. One is classes where the criterion for membership of an instance in a concept can be expressed as a formula (in the first-order theory of the reals) with fixed quantification depth and exponentially-bounded length, whose atomic predicates are polynomial inequalities of exponentially-bounded degree. The other is classes where containment of an instance in a concept is testable in polynomial time, assuming we may compute standard arithmetic operations on reals exactly in constant time.
Our results show that in the continuous case, as in the discrete, the real barrier to efficient learning in the Occam sense is complexity-theoretic and not information-theoretic. We present examples to show how these results apply to concept classes defined by geometrical figures and neural nets, and derive polynomial bounds on the V-C dimension for these classes.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Alt, H., Behrends, B., & Blömer, J. (1991). Approximate Matching of Polygonal Shapes.Procs. of the 1991 ACM Symposium on Computational Geometry pp. 186–193.
Anthony, M., & Biggs, N. (1992).Computational Learning Theory: an Introduction, Cambridge University Press, 1992.
Attalah, M.J. (1983). A Linear Time Algorithm for the Hausdorff-distance between Convex Polygons.Information Processing Letters 17 pp. 207–209.
Baum, E.B., & Haussler, D. (1988). What Size Net Gives Valid Generalization?Neural Computation 1, pp. 151–160.
Ben-David, S., & Lindenbaum, M. (1993). Localization vs. Identification of Semi-Algebraic Sets.Proceedings of the 6th Annual ACM Conference on Computational Learning Theory, pp. 327–336.
Ben-Or, M. (1983). Lower Bounds for Algebraic Computation Trees.Proceedings of the 15th Annual ACM Symposium on the Theory of Computing, pp. 80–86.
Blumer, A., Ehrenfeucht, A., Haussler, D., & Warmuth, M.K. (1987). Occam's Razor.Information Processing Letters 24 pp. 377–380.
Blumer, A., Ehrenfeucht, A., Haussler, D., & Warmuth, M.K. (1989). Learnability and the Vapnik-Chervonenkis Dimension.Journal of the Association for Computing Machinery 36 No. 4, pp. 929–965.
Dudley, R.M. (1978). Central Limit Theorems for Empirical Measures,Annals of Probability 6, pp. 899–929.
Ehrenfeucht, A., Haussler, D., Kearns, M., & Valiant, L.G. (1989). A General Lower Bound on the Number of Examples Needed for Learning.Information and Computation 82, pp. 247–261.
Goldberg, P. (1992). PAC-Learning Geometrical Figures.PhD thesis, Department of Computer Science, University of Edinburgh (1992).
Hall, M. (1967).Combinatorial Theory, Blaisdell, Waltham MA (1967).
Haussler, D., Littlestone, N., & Warmuth, M.K. (1988). Predicting {0,1} functions on randomly drawn points.Proceedings of the 29th IEEE Symposium on Foundations of Computer Science, pp. 100–109.
Laskowski, M.C. (1992). Vapnik-Chervonenkis Classes of Definable Sets.J. London Math. Society (2) 45, pp. 377–384.
Linial, N., Mansour, Y., & Rivest, R. (1991). Results on Learnability and the Vapnik-Chervonenkis Dimension.Information and Computation 90, pp. 33–49.
Maass, W. (1992). Bounds for the Computational Power and Learning Complexity of Analog Neural Nets.Insts. for Information Processing Graz, report 349; Oct. 1992. Proceedings of the 25th Annual ACM Symposium on the Theory of Computing (1993), pp. 335–344.
Macintyre, A. & Sontag, E.D. (1993). Finiteness Results for Sigmoidal “Neural” Networks,Proceedings of the 25th Annual ACM Symposium on the Theory of Computing, pp. 325–334.
Milnor, J. (1964). On the Betti Numbers of Real Varieties.Procs. of the American Mathematical Society 15, pp. 275–280.
Natarajan, B.K. (1991)Machine Learning: A Theoretical Approach. Morgan Kaufman Publishers, Inc., ISBN 1-55860-148-1
Renegar, J. (1992). On the Computational Complexity and Geometry of the First-Order Theory of the Reals. Part 1 (of 3).Journal of Symbolic Computation 13, pp. 255–299.
Sontag, E.D. (1992). Feedforward Nets for Interpolation and Classification,Journal of Computer and System Sciences 45, pp. 20–48.
Steele, J.M. & Yao, A.C. (1982). Lower Bounds for Algebraic Decision Trees.Journal of Algorithms 3, pp. 1–8.
Stengle, G., & Yukich, J.E. (1989). Some New Vapnik-Chervonenkis Classes, Annals of Statistics17, pp. 1441–1446.
Valiant, L.G. (1984). A Theory of the Learnable.Communications of the ACM 27 No. 11, pp. 1134–1142.
Valiant, L.G. (1985). Learning Disjunctions of Conjunctions.Procs of the 9th International Joint Conference on AI, pp. 560–566.
Valiant, L.G. (1991). A View of Computational Learning Theory.NEC Research Symposium: Computation and Cognition (ed. C.W. Gear), SIAM, Philadelphia, 1991.
Vapnik, V.N., & Chervonenkis, A. Ya. (1971). On the uniform convergence of relative frequencies of events to their probabilities.Theory of Probability and its Applications 16, No. 2 pp. 264–280.
Warren, H.E. (1968). Lower Bounds for Approximation by Non-linear Manifolds.Trans. of the AMS 133, pp. 167–178.
Wenocur, R.S., & Dudley, R.M. (1981). Some special Vapnik-Chervonenkis classes.Discrete Mathematics 33, pp. 313–318.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Goldberg, P.W., Jerrum, M.R. Bounding the Vapnik-Chervonenkis dimension of concept classes parameterized by real numbers. Mach Learn 18, 131–148 (1995). https://doi.org/10.1007/BF00993408
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.1007/BF00993408