Convergence with Probability One: Martingale Difference Noise

Abstract

Much of the classical work in stochastic approximation dealt with the situation where the “noise” in each observation Y _n is a martingale difference, that is, where there is a function g _n (·) of θ such that E [Y _n |Yi, i < n, θ₀] = g _n (θ _n ) [17, 40, 45, 47, 56, 79, 86, 132, 154, 159, 169, 181]. Then we can write Y _n = g _n (θ _n ) + δM _n where δM _n is a martingale difference. This “martingale difference noise” model is still of considerable importance. It arises, for example, where Y _n has the form Y _n = F _n (θ _n , ψ _n ) where ψ _n are mutually independent. The convergence theory is relatively easy in this case, because the noise terms can be dealt with by well-known and relatively simple probability inequalities for martingale sequences. This chapter is devoted to this martingale difference noise case. Nevertheless, the ODE, compactness, and stability techniques to be introduced are of basic importance for stochastic approximation, and will be used in subsequent chapters.