Global Optimisation

Abstract

The purpose of this chapter is to consider various ways in which the fast second-order training algorithms discussed in Chapters 2 and 3 can be modified so that they are more likely to converge to global minima — rather than local minima — in the MLP error surface. Local minima are known to be a serious obstacle to successful training when MLPs are applied to many, if far from all, practical tasks (see the discussion in Section 1.2.2). In this respect, the significance of the benchmark test results presented in Chapter 5 is that they suggest that local minima are an even more serious obstacle for certain second-order training methods — notably the quasi-Newton methods of Section 3.3.2 and (to a lesser extent) the conjugate gradient methods of Section 3.3.4 — than they are for the conventional backpropagation-related training methods of Section 1.3.