Regularization Theory, Radial Basis Functions and Networks

Abstract

This paper consists of two parts. In the first part we consider the problem of learning from examples in the setting of the theory of the approximation of multivariate functions from sparse data. We first will show how an approach based on regularization theory leads to develop a family of approximation techniques, including Radial Basis Functions, and some tensor product and additive splines. Then we will show how this fairly classical approach has to be extended in order to cope with special features of the problem of learning of examples, such as high dimensionality and strong anisotropics. Furthermore, the same extension that leads from Radial Basis Functions (RBF) to Hyper Basis Functions (HBF) also leads from additive models to ridge approximation models, such as some forms of Projection Pursuit Regression.

In the second part we consider the problem of estimating the complexity of the approximation problem. We first briefly discuss the problem of degree of approximation, that is related to the number of parameters that are needed in order to guarantee a generalization error smaller than a certain amount. Then we consider certain Radial Basis Functions techniques, and, using a lemma by Jones (1992) and Barron (1991, 1993) we show that it is possible to define function spaces and basis functions for which the rate of convergence to zero of the error is O $\left( {\frac{1} {{\sqrt n }}} \right)$ in any number of dimensions. The apparent avoidance of the “curse of dimensionality” is due to the fact that these function spaces are more and more constrained as the dimension increases. Examples include spaces of the Sobolev type, in which the number of weak derivatives is required to be larger than the number of dimensions.