Keywords

1 Introduction

The hunt for understanding human brain includes searching for neuronal features related to behaviour and trying to correlate them. In computational neuroscience, establishing a correlation means to model a link between activity and behaviour. The selection of accurate model, from a class of models, depends on the researcher’s preference. In this paper, we have focussed upon recurrent neural networks and discussed how it can be useful as collaborator between complex system of brain and interpretable hypothesis.

RNNs are amongst a group of computational models, used for exploring neurological phenomena and as a mechanism for solving machine learning problems [1,2,3]. These are such types of networks in which a neuron receives signal/input from many other neurons in its network, without the consideration of crust or trough of the signal. As a result, the functionality of neurons is not only affected by stimulus current but due to the current state of the network also. Such features make these networks more suitable to use for computations that can reveal the crucial properties of brain like holding items in memory or recalling patterns towards a problem and what can be the mechanisms leading to memory failure.

The logic for studying RNNs as model in our work originates from the fact that human brain is a complex structure of tremendous number of distinct neurons, connected with each other to perform some activities like memory building, recalling and recognition, moods and behaviour, movement and motor system. Our work is concerned with memory building and how an RNN can represent a network of connected neurons, responsible for memory building. Here, we have presented a review of how RNNs can be used to explain the dynamical system along with techniques which can be incorporated to improve the performance of the network.

A detailed overview of RNNs and the techniques incorporated with RNNs to study problems related to cognition and memory is given below.

Martens et al. [4] proposed a novel damping approach for training RNN with Hessian Free (HF) optimisation for a set of two challenging issues. First, the pathological data set could not be used for optimization, due to its very long-term dependencies such as gradient descent, and second is real-world sequential data set dependent on long short-term memories. Experimental results established a solution to the problem related to long-term dependencies. The HF optimisation approach is capable of training RNNs more effectively. However, the total computation time required by the proposed approach was considerably higher than the expected. Therefore, some other mechanisms can be incorporated to reduce the training time.

Bitzer et al. [5] proposed a fusion of RNN with Bayesian inference approach to explore nonlinear dynamical system. In this strategy, RNN is used as a production model of dynamic process caused by the stimulus, and Bayesian equations are derived to decode its resultant. This combination of RNN with Bayesian equation is called as recognising RNNs. Experimental results show that these updates not only improved the efficiency of conventional RNN but also computes prediction with fast decoding and robustness. However, to explain coordinated movements, multistable dynamics may also be incorporated, which is an important phenomenon for explaining coordinated movements. For this purpose, rRNN can be upgraded to gRNN, which shows required multistable dynamics.

Brian et al. [6] proposed an approach of using firing-rate dynamics to train RNN of spiking neurons model. They proposed it for purposes like producing dynamical patterns and generating structural and temporal outputs, based in network input. The proposed methodology proved to be a success in constructing spiking networks. With few minor changes in the number of neurons, continuous variable network mimics the original spiking model. However, when it comes to explain the nonlinearity and connectivity of different populations of neurons, it could not work effectively.

Ingmar et al. [7] proposed a neural solution to simultaneous location and mapping (SLAM) problem. The proposed model of recurrent LSTM network performs a class of two-dimensional navigation. The experimental results prove the worth of training recurrent neural network and proposes to extend its work to be incorporated in robotics. But, this favour comes with a price. Since this approach only considered movement, it does not include memorisation. It would be interesting to see if constraints like non-negative neural activations, dale’s law or architecture of hippocampus circuit can be incorporated with brain’s navigation system.

Peter et al. [8] presented a train-and-constrain approach that portraits the results of artificial neurons with spiking neurons. Firstly, the training process includes RNN with back-propagation with respect to time; then, weights are discretised, and then, finally, the results are converted to spiking RNN. Experimental results show that artificial neurons mimic the responses of spiking neurons. To prove the accuracy of the proposed approach, the mapping of NLP, i.e. question classification is done. This device used for demonstration is TrueNorth, and IBM’s Neurosynaptic system. TrueNorth puts some restrictions on synaptic connectivity and its parameters. The hardware-restricted model generated 74% accuracy in NLP question classification task. However, there is a performance drop due to synaptic weights taken in the model are taken in the form of discrete. This discretisation leads to performance decrease and hence can be expected to be the topic of further research.

Rajan et al. [9] proposed a partial in-network training algorithm and used it to modify small portions of connections, starting from random connections, to show that distorted RNN can generate sequences and working memory effectively. The proposed algorithm proved to be efficient when the sequences generated by the collaboration of recurrent synaptic interconnections and external inputs were successfully propagated. However, neuronal sequences may also arise through largely connected unstructured network.

Alexander et al. [10] proposed trained recurrent neural networks to study task-related neural dynamics. In this, a mean-field theory was created to get multiple fixed-point attractors. Experimental results show that dynamics connected to network’s output, in the boundaries of attractors, are determined by low-order linear differential equation. Using the equations, network can be accessed, and prediction of success and failure can also be made. However, the network generates failure when a nonlinear sigmoid activation function was used with rectified linear units. The proposed methodology forecasts a state-dependent recurrent selectivity in the network response.

Omri Barak [11] suggested to methodologies for designing and training RNNs, the first one includes implementation of low-D intuition, and second one incorporates reverse engineering in place of low-D structures. The low-D intuition methodology assumes that the activity of the neurons is not affected by the input current only, but also by the state of the network. But, this assumption is quite expensive because training an RNN is a difficult process. Here, the connectivity in the network was trained repeatedly with many example trials by recasting connectivity according to certain learning rule. Reverse engineering, on the other hand, does not include low-D intuition. It assumed that saddle points, line attractors, and other areas of phase space could be the main describing points of computation, which may provide some insights to the process existing in blackbox.

Umut et al. [12] proposed a recurrent neural network model for modelling the dynamics and activities of human brain. The authors claim that there has been substantial work on creating a better feature model; a very limited work has been performed on response model. Here, in this work, they have investigated that RNN models can be used to predict the sequences based on feature-invoked responses, depending on their internal storage for nonlinear processing of arbitrary sequences. Experimental results show that the RNN model proposed in this work can outstrip the existing response model by correctly evaluating long-term dependencies.

Hupkes et al. [13] proposed a methodology to explain how neural network can process and learn language with hierarchical structure. For this purpose, an artificial task of processing nested arithmetic expression was given to each type of neural network and computed their results. Surprisingly, it was RNN, which came out with a generalised solution to the arithmetic expression. It was seen that RNN stored the results of sub-expressions in the form of stacks and used those to solve the entire expression. However, this methodology worked well with limited size of arithmetic expression. It did not work well when the length of the expression was increased.

Table 1 Succinct review of the related work is presented in this table including different techniques used to study RNNs
Full size table

2 Conclusion

Recurrent neural networks (RNNs) are useful tool for computational and theoretical neuroscience research. Designing and training such networks has unfolded alternative schema to develop neural systems. In addition to providing ample set of sub-networks to specific task, this type of network can give insights to new scientific questions. Like, does it provide correct measure of naturalism whilst modelling the brain? What would be the extent of solution for a given problem? What would be the behaviour of the network towards biological constraints? Will we be able to understand the learning process happening in the black box? In order to gather all these benefits, a robust theoretical foundation is required and is expected to be developed in near future. For understanding the dynamical system of human brain, it is quite important to train and design the network, the same way human brain is doing. In biological neural network, training takes place on a framework of physiological and anatomical constraints. These constraints can be thought of as elements in a network. My assumption is that if the network is able to represent the dynamical system of human brain, with all constraints satisfied, it may be able to solve the scientific questions.

This review targets recurrent neural network as an overall framework to understand the dynamical system of biological network, the connectivity and communication amongst a population of different connected neurons.