Unlearning and Its Relevance to REM Sleep: Decorrelating Correlated Data

Abstract

We review the performance of Hebbian unlearning in a neural network with parallel dynamics and stationary patterns. One of the main problems of Hebbian learning, and any local coding procedure, is that it cannot store extensively many patterns with different activities — the synaptic coding problem. The reason behind it is that undesirable correlations are introduced during learning. Hebbian unlearning is an unsupervised procedure whose main function is to remove these correlations. Its characteristic features are: (i) A solution of the synaptic coding problem which is so good that the storage capacity is near to or even saturating the theoretical upper bound, (ii) The existence of an optimal and a critical number of dreams at and above which, respectively, storage and retrieval are optimal (D _opt ) or break down completely (D _c ) so that all information is lost, (iii) The existence of a critical resemblance, or overlap, m_c so that for an initial overlap to, greater than m _c the network always converges to a pattern whereas for m_i less than m_c it relaxes to a state with blinking neurons, thus signaling that it had not seen this state before, (iv) For neurons with self-interaction the convergence time, which is to be interpreted as the duration of a dream, diverges at D _opt < D _c so that it is hard to do too much unlearning. The consequences for ordinary REM sleep are discussed.