Keywords

1 Introduction

Currently, there are many poorly formalized problems that are badly solved by existing methods (detection and recognition of objects in conditions of significant data shortage, control of unstable systems, control of the behavior of mobile agents in a volatile environment, etc.).

One of the most promising common approaches to solving such problems is artificial neural networks (ANN), in particular, deep neural networks (DLN), which are now actively developing. This is due, in particular, with the advent of new hardware (NVIDIA graphics accelerators [1], specialized processors (BrainScaleS [2, 3], SpiNNaker [4], NIDA [5], DANNA [6], Neurogrid [7], IBM TrueNorth [8]), which allow efficient numerical calculations on the basis of the mathematical apparatus of the DLN, and the direction of neuromorphic systems, whose architecture and design are based on the principles of the work of the biological neural structures of the nervous system. This is a fairly broad interpretation, in which the deep learning fit well. Possible successes of neuromorphic systems are associated, first of all, with the biological plausibility of their basic neuron component and its hardware implementation. In this sense, some specialized processors (in particular, IBM TrueNorth) refer specifically to processors of the neuromorphic type.

2 Overview of Deep Neural Network Architectures

Today, the practical application of neural networks is most intensively developed in the trend of deep learning.

There is a large number of networks within this trend [9]. The basic architectures, from which all the main implementations are obtained:

  • Feed forward (FF) (Perceptron, Autoencoders [10]);

  • Fully connected networks (FCN) (Markov Chain [11], Hopefield network [12], Boltzmann Machine [13];

  • Convolutional neural networks (CNN) (LeNet [14], VGG-19 [15], Google Inception [16]);

  • Recurrent neural networks (RCN) (LSTM [17], Deep Residual Network (ResNet) [18,19,20]);

There are separately presented architectures such as growing neural networks, in which the following widespread types can be distinguished:

  • Networks based on Kohonen maps (SOM [21], ESOM [22], GHSOM [23], SOS [24]);

  • SOINN, ESOINN [25];

  • Neural Gas Network [26] and its derivatives GNG [27], IGNG [28], GCS [29], TreeGCS [30], PGCS [31] and others.

Relatively new works are devoted to the implementation of spiking neural networks, based on the above architectures [32,33,34]. The advantages of deep spiking neural networks are firstly declared in the significant energy savings in the case of hardware implementation.

If we consider the achievements of neural networks from the point of view of solving particular problems, great progress has been made in this direction. So, according to the results of the competition in recent years, DLN have been won in most computer vision tasks (pattern recognition, object detection, segmentation, etc.). It is important to note that such networks are effective in problems in which there are high local correlations in the input data.

Also, there is the big problem of combining a set of private solutions, formed by neural networks to solve common problems of controlling agent behavior in a complex environment. In other words, the solution, for example, of object detection problem, converts the space of high-dimensional input data into a space of low dimensionality of the classes of objects to be detected. If it is necessary to create a flexible control system for the behavior of the agent (robot) in a volatile environment, we are forced to operate with a number of such particular solutions. This naturally limits the agent in adaptability to changes in the environment. Part of this problem is solved in growing networks.

Despite the fact that ANN were originally based on the analogy with the nervous system, the majority of neural networks in their topology, training rules and principles of functioning as a whole is very different, and the trend away from biological likelihood is growing. In particular, the development of networks follows the path of increasing the number of layers, but not the complexity of the functional element of neural networks - the neuron; and growing neural networks are based on the addition of neurons and layers, in contrast to change in the structure of a neuron dendritic tree in a biological system, where each dendrite provides complex information processing.

If we compare the known features of the nervous system and ANN (assuming that the advantages of the still disjointed architectures of ANN will be unified), then following table can be made (Table 1).

Table 1. Comparison of the features of artificial neural networks and the nervous system
Full size table

It seems promising to consider the possibility of complicating the model of the neural networks functional element with an emphasis on the possibilities of network structural adaptation, in the trend of the neuromorphic approach.

3 Neuron Models

There are many widespread models of neurons. By the level of abstraction, models can be divided into:

  • Biological (biophysical)-models based on the modeling of biochemical and physiological processes, which, as a consequence, lead to a certain behavior of the neuron in certain modes of operation (the Hodgkin–Huxley model [35]).

  • Phenomenological-models describing certain phenomena of the behavior of a neuron in certain modes of operation as a “black box” (the Izhikevich model [36]).

  • Formal-models with the highest level of abstraction, describing only the basic properties of the neuron (formal neuron [37]).

Each model can correspond to several features from this classification. In the framework of ANN in general, and DLN, in particular, modifications of formal neuron models, with different activation functions (Sigmoid, hyperbolic tangent, ReLU and its derivatives [38]) are used. Spiking variations of deep networks basically contain such models of neurons as variations of the threshold integrator model [39], the Izhikevich model mentioned above.

One of the promising options for implementing the model of an element of neuromorphic systems is the phenomenological model of a dynamic spike neuron with the ability to describe the spatial structure of the dendritic apparatus [40]. This model allows us to describe the variable topology of a neural network, based on the principles of neural structure formation known from neurophysiology [41].

4 Discussion

The main feature of the nervous system, which is still not considered in the ANN archives, is a great potential in structural (topological) restructuring. Structural adaptation in the nervous system is largely based on the high complexity of a single element of the network - the neuron.

The analysis allows to identify the following areas of development of ANN in the framework of the neuromorphic approach:

  • Complicating the neuron model, adding the possibility of describing the structure of the membrane (as generalizing and binding elements) of the neuron.

  • Development of learning algorithms, taking into account the modification of the structure of the generalizing and binding elements of the neuron.

  • Development of ANN architectures that allow training and data output simultaneously in multiple contexts.