Keywords

1 Introduction

Artificial Neural Network (ANN) and Evolutionary Algorithm (EA) are relatively young research areas that were subject to a steadily growing interest nowadays; represent two evolving technologies that are inspired by biological information science.

ANN is derived from brain theory to simulate learning behavior of an individual, which is, used for approximation and generalization, while EA is developed from the evolutionary theory raised by Darwin to evolve the whole population for better fitness. Evolutionary Algorithm is actually used as an optimisation algorithm, and not a learning algorithm.

In particular, EO has been used to search for the design and structure of the network and to select the most relevant features of the training data. It is well known that to solve nonlinearly separable problems, the network must have at least one hidden layer; but determining the number and the size of the hidden layers is mostly a matter of trial and error. EOs has been used to search for these parameters, to generate the appropriate network to solve specific problems.

On the other hand, ANN has yield many benefits solving a lot of problems in fields as diverse as biology, physics, computer science and engineering. In many applications, the real-time solutions of optimisation problems are widely required. However, traditional algorithms may not be efficient since the computing time required for a solution is greatly dependent on the structure of the problems and there dimension.

A promising approach to solving such problems in real time is to employ artificial neural networks based on circuit implementation [133]. ANNs possess many desirable properties such as real-time information processing. Therefore, neural networks for optimisation, control, and signal processing received tremendous interests. The theory, methodology, and applications of ANNs have been widely investigated.

The present work focuses on Evolutionary Algorithms (EO), Artificial Neural Networks (ANN) and their joint applications as a powerful tool to solve challenging problems for the academia and the industry. In this section the essentials and importance of both methodologies are presented.

1.1 Evolutionary Algorithms

Since 1960s, there has been an increasing interest in emulating evolutionary process of living beings to solve hard optimisation problems [34, 101]. Simulating these features of living beings, yields stochastic optimisation procedures called Evolutionary Optimisation (EO). EO belongs to global search meta-heuristics methods since, by its own nature, explores the whole decisional space for global optima.

Evolutionary algorithms (EAs) are a class of stochastic and probabilistic optimisation methods that are inspired by some presumed principles of evolution; attempt to emulate the biological process of evolution, incorporating concepts of selection, reproduction, and mutation.These techniques, inspired in Darwinian evolution postulates, consider a population of individuals on which selection and diversity generation procedures are performed, guaranteeing better fitted individuals to survive through successive iterations [12, 46, 47]. Each individual is a potential solution of the optimisation problem, so it belongs to decision space. Every iteration (generation), individual features are combined by means of recombination operators such selection, crossover and mutation, driving solutions to global optima. By mimicking this process, EAs are able to evolve solutions to real world problems, if they have been suitably encoded.

Evolutionary Optimisation has demonstrated to be effective in engineering and science optimisation problems in several fields such as: Aerospatiale applications [15], energy [9], transport planning, RAMS [87, 96, 97, 111], task scheduling and so on.

The applications mentioned above usually results in high dimensional search spaces, highly non-linear optimisation problems, non-convex optimisation, highly constrained problems, uncertainty effects and/or multicriteria paradigm. These numerical difficulties are commonly tackled by these meta-heuristics, which usually outperforms traditional optimisation strategies with lower probability of being stacked in local optima, and being able to yield a Pareto set in a single run of the algorithm (multi-objective approach). Because of this, Evolutionary Optimisation has been an important R&D matter in the last decade.

Among the variety of EO meta-heuristics, the most relevant nowadays are: Genetic Algorithm (GA), Evolutionary Strategy (ES), Evolutionary Programming (EP), Collective Intelligence, Memetic Algorithms, and Differential Evolution.

In recent years, there has been an increase in the use of evolutionary approaches in the training and optimisation of artificial neural networks(ANNs). Different works are presented in Sect. 4.2.1

1.2 Artificial Neural Networks ANN

Artificial neural networks (ANNs) are biologically inspired computer programs, inspired from the morphological and biophysical properties of neurons in the brain. ANNs are designed to simulate the way the human brain processes information.

Neural networks are similar to the human brain in the following two ways:

  1. 1.

    A neural network acquires knowledge by learning.

  2. 2.

    The knowledge of a neural network is stored in the connections between neurons known as synaptic weights.

[98] were the first patterned biological neurons from the binary automata. A second generation of neurons integrates a nonlinear activation function that has allowed growing up the interest in ANNs [69], allowing to solve nonlinear problems.

The power and utility of artificial neural networks have been shown in several applications including speech synthesis, diagnostic problems, medicine, business and finance, control, robotics, signal processing, computer vision and many other industrial problems that are included in the category of pattern recognition. But knowing that there exists a suitable network for a specific problem is interesting, finding it proved to be difficult. Although there exist some algorithms to set the weights by learning from data training given a fixed topology, even if get stuck in local minima. To lead to good results, they strongly depend on problem specific parameter settings and on the topology of the network.

Training procedures of neural networks are optimisation algorithms aim to minimize the global error output respect to connection weights under conditions of noise, qualitative uncertainty, nonlinearity, etc. Therefore training procedures of neural networks provide a common approach to the optimisation task in process optimisation and control applications [149]. However, this is an important issue because there are strong biological and engineering evidences to support that an ANN is determined by its architecture. Using EOs as a procedure to assist neural network design and training seems to be a straightforward idea. Branke in [23] explain how EO improve design and training ANNs.

Today, significant progress has been made in the field of neural networks, enough to attract a lot of attention and research funding. Research in the field is advancing on many fronts. New neural concepts are emerging and applications to complex problems developing. Clearly, today is a period of transition for neural network technology.

ANNs are inherently parallel architectures which can be implemented in software and hardware. One important implementation issue is the size of the neural network and its weight adaptation. This makes the hardware implementation complex and software learning slower.

ANNs has two distinct steps [172];

  1. 1.

    Choosing proper network architecture.

  2. 2.

    Adjusting the parameters of a network so as to minimize certain fit criterion.

Even if, the most of problems treated in real study are complex, so the unknown architecture of the ANN is set arbitrarily or by trial and error [121], and small networks cannot achieve the solution in much iteration, but if the network is too large, it leads to overfitting and a bed generalization, and the majority of neural networks suffers of premature convergence and low global convergence speed etc. In order to overcome these limitations, some improvements were made for EOs in the last decade on ANNs.

2 Different Use of ANNEO and EOANN

In the last decade, there has been a great interest combining learning and evolutionary techniques in computing science to solve complex problems for different fields. Different works presented how ANNs are an optimisation tools [33, 120, 131, 136, 173, 175]. In this paper we just limit this review to several ways in which EOs and ANNs may be combined.

The next section presents different approaches of combining EOs and ANNs.

2.1 The Use of EOs in ANNs: EOANN

In recent years, evolutionary algorithms (EAs) have been applied to the ANN’s optimisation. The first applications of EAs to ANN parameter learning date back to the late 80 s in the fields of Genetic Algorithms (GAs) [45, 47] and Evolutionary Programming (EP) [47, 103]. EAs are powerful search algorithms based on the mechanism of natural selection. Since the 90s, EAs have been successfully used for optimizing the design and the parameters of ANNs [132, 150]. This special class of ANNs in which evolution is another fundamental form of adaptation in addition to learning creating an EOANN [40, 167]. EOANN are used to find the best data subset which optimizes the ANN training for a specific problem.

The result of EOANN is an ANN with optimal performance to estimate the value of one or more variables and the estimation error is strongly related to the quality of the training set in terms of size and treatment of the possible outputs. Another approach is the evolutionary weight training process, to optimizing connection weights, learning rules, and optimizing the network architecture by identifying how many inputs neurons, hidden neurons, hidden layers we have to use, to get a good performance [18, 39, 88, 92, 128]. Considerable research has been conducted on the evolution of topological structures of networks using evolutionary algorithms [3, 6, 13, 14, 20, 73, 84].

In the other hand, an essential issue is to improve generalization of the neural network training models. Early stopping, weight decay and curvature-driven smoothing and other techniques are been used to resolve this problem, another approach is including an additional term in the cost function of learning algorithms, which penalizes overly high model complexity. Regularization of neural training was treated with Eos in [1, 75].

In general, constructing neural network consists of two major steps, design and training component networks, combining of the component networks predictions to produce the neural networks solutions. ANN training method has some limitations associated with overfitting, local optimum problems and slow convergence rate. In order to overcome the limitations, some scientist proposed particle evolutionary algorithm to train ANN.

The research use EOs to evolve and design the structure architecture or the selection of the training algorithms and optimisation of its synaptic weight initialization, thresholds, training ratio, momentum factor, etc., of neural network roundly. The scientist object is to accelerate the convergence speed of network and optimize the result in case of trapping into local optimal value, and a better searching space is found out in the solution space.

The initial set of weights to be used in learning of ANN has a strong influence in the learning speed and in the quality of the solution obtained after training. An inadequate initial choice of the weight values may cause the training process to get stuck in a poor local minimum or more time to converge. Inappropriate topology selection and learning algorithm are frequently bed; there is little reason to expect that one can find a uniformly best algorithm for selecting the weights in an ANN [94]. This kind of topology was chosen for the following main reasons:

  1. 1.

    Additional links that skip adjacent layers allow the genetic algorithm to remove a whole layer of nodes while still keeping the network functional.

  2. 2.

    Some additional shortcut connections, e.g., those between the network input and output, may ease training and therefore the whole genetic topology optimisation may become faster.

There are different ways to evaluate weights of the component networks. For example Jimenez [74] use weights determined by confidence of the component networks. Zhou [184] utilize the genetic algorithm to find proper weights for each member of an ensemble. In [143], present and define many operators and crossover applied to weights of an ANN. The importance of a good choice for the initial set of weights is stressed by Kolen and Pollak [83].

ANNs strongly depend on the network topology, the neurons activation function, the learning rule, etc. optimisation for these factors are usually unknown a priori because they depend mainly on the particular training set to be considered and on the nature of the solution [137]. So, for practical purposes, the learning rule should be based on optimisation techniques that employ local search to find the optimal solution [124, 126].

Evolutionary approaches have been shown to be very effective as an optimisation technique, their efficiency could he exploited in training and constructing neural networks, there architecture/design and learning, they can evolve towards the optimal architecture without outside interference, thus eliminating the tedious trial and error work of manually finding an optimal network, adapting the connection weights and learning algorithms according to the problem environment. Many considerable efforts at obtaining optimal ANNs based on EAs have been reported in the literature [4–7, 10, 13, 24–28, 31, 52, 53, 58, 63–67, 82, 99, 117, 122, 132, 139, 144, 166, 168–171].

The EAs are using diver methods to encode ANNs for the purpose of training and design. The common approach is to encode the ANN weights into genes that are then concatenated to build the genotype. Encoding methods can be divided in three main groups according to the process of creating the network from the encoded genome: direct, parametric and indirect encoding. A real coded genetic algorithm is used to optimize the mean square of the error produced by training a neural network established by Aljahdali in [11]. Benaddy et al. [17] present a real coded genetic algorithm that uses the appropriate operators type to train feed-forward neural network. Larranaga et al. [86] describes various methods used to encode artificial neural networks to chromosomes to be used in evolutionary computation.

Another important point to note is the use of EOs to extract rules from neural networks trained. Rule extraction from neural networks is attracting wide attention because of its computational simplicity and ability to generalize [49–51, 141].

2.2 The Use of ANNs in EO: ANNEO

In real world applications, sometimes it is not easy to obtain the objective value. Therefore we needs complicated analysis or time consuming simulation to evaluate the performance of design variables [147]. As ANNs represent a nonlinear robust modeling technique which are developed, or trained, based on analytical or simulated results of a subset of possible solutions [35], give to ANNs an important role in solving problems with extremely difficult or unknown analytical solution. ANNs can be used with a huge reduction of cost in terms of objective function evaluations [22]. One of the first pioneers in ANNEO were Hopfield and Tank who presented an ANN for solving combinatorial problems, that was mapped into a closed-loop circuit [68, 133], named Hopfield Neural Network (HNN). HNN is a continuously operating model very close to analog circuit implementation. Since 1985 a wide variety of ANNs have been designed for improving the performance of HNN.

In ANNEO fast objective functions evaluations are performed using pre-trained ANN. Normally in iterative process subsets of objective function values obtained using exact procedures are used in an embedded EOANN algorithm, and some of the new objective function evaluations are performed using the ANN. The result is an EO which evolves faster than conventional ones but special care must be paid to the selection of appropriate training subsets and the number of objective functions evaluated using ANN in order to avoid convergence problems. Acceleration of the convergence speed is done in [176] as an ANN model trained to approximate the fitness function according to an adaptive scheme for increasing the number of network fitness calculation.

EOs usually needs a large number of fitness evaluations before a satisfying result can be obtained. And as an explicit fitness function does not exist, or his evaluation is computationally very expensive. It is necessary to estimate the fitness function by constructing an approximate model or presenting an interpolation of the true fitness function via some interpolation technique as ANNs [22, 55] employ a Feedforward neural networks for fitness estimation, reduce the number of expensive fitness function evaluations in evolutionary optimisation. The idea of the implementation of an ANN, that approximates the fitness function, comes from the universal approximation capability of multi-layer neural networks [69]. An artificial neural networks model is used in order to reduce the number of time-consuming fitness evaluations [55].

Real-time solutions to resolve problems are often needed in engineering applications. Solve many problem of optimisation in real time usually contain time-varying parameters, such in signal processing, robotic, time series, etc., and we have to reduce and optimize the performance. The numbers of decision variables and constraints are usually very large and large-scale optimisation problems are even more challenging when they have to be solved in real time to optimize the performance of dynamical systems. For such applications, Conventional numerical methods may not be effective at all due to the problem dimensionality and stringent requirement on computational time [16]. The employment of ANNs techniques as Recurrent Neural Networks (RNN) [21, 33, 48, 79, 89, 91, 113, 115, 127, 133, 145, 146, 160, 165, 177] (Papers below proposed neural networks guaranteed to be globally convergent in finite time to the optimal solutions), Fuzzy Neural Network (FNN) are a promising approach to resolve this inefficiency [123].

Application of ANNs algorithms receive increase interests for optimisation as we see in [21, 32, 33, 37, 79, 93, 105–107, 118, 133, 134, 151, 154, 156, 157, 182], using gradient and projection methods [91, 104], Bouzerdoum and Pattison [21] presented a neural network for solving quadratic optimisation problems with bounded variables only, which constitutes a generalization of the network described by Sudharsanan [127]. Rodrguez-Vzquez et al. [118] proposed a class of neural networks for solving optimisation problems, in which their design does not require the calculation of a penalty parameter. To avoid using finite penalty parameters, many other studies have been done in [21, 56, 70, 78, 118, 134, 148, 153, 156, 157, 159, 182, 183]. In [151–154], the authors presented several neural networks for solving linear and quadric programming problems with no unique solutions, which are proved to be globally convergent to exact solutions, and in which there is no variable parameter to tune.

Romero [119] approached optimisation problems with a multilayer neural network. Da Silva [36] coupled fuzzy logic with Hopfield Neural Networks to solve linear and non-linear optimisation problems. Case studies with convex and nonconvex optimisation problems are studied in illustrate the approach with a trained Multilayer neural networks [143]. Xia et al. [155, 158, 160] proposed a general projection neural network, that includes projection neural network, the primal-dual neural network, and/or the dual neural network, as special cases, for solving a wider class of variational inequalities and related optimisation problems.

A reliability network reflect a network optimized whose topology is optimist, at minimal cost, under the constraint that every pair of neurons can communicate with each other. Abo El fotoh et al. [2] presents an ANN for solving this problem o reliability, by constructing an energy function whose minimization process drives the neural network into one of its stable states.

An ANN aided with other algorithms as simulated-annealing (SA) algorithm, can be usefully used to resolve optimisation problems as they do in [85]. Chaotic artificial neural are studied and established as optimisation model in [8, 85]. Another point to be mentioned is the use of ANNs to re-optimisation online as it presented in [138] .

3 Some Applications Using ANNEO and EOANN

Many academic papers show the applicability of EOANN to optimize different parameters of ANN, to improve their training and their stability [13, 20, 43, 71–73, 76, 84] and the papers cited below in Sect. 4.2.2. Other EOANN applications were performed in several fields such as Financial Engineering [41, 59], grammatical inference [19], Chemical Reaction [178], Hydrochemistry [142], Time series prediction [77, 81, 180], Classification Process [29, 30, 44, 80, 90, 95, 109, 112, 135], Medicine [54], Diagnosis problems [17, 72, 129, 130], Diverse Engineering Applications [57, 100, 125, 135, 174], Robotic [161], Monitoring [38], Traffic Prediction [114], Control Systems [181], Neutron spectrometry and Dosimetry research areas [108], Multi-agent systems [81], Regression problems [61], Chaos dynamics Problem [179, 180], Reliability [62], etc.

Also, the role of ANNs in EO is presented in many academic papers as in [22, 147, 176], and in many fields like Reliability Systems [35], Electromagnetics optimisation [22], pressure vessel design problem [147], aerodynamic design optimisation [73], Electric-Field optimisation [85, 116, 164], Cybernetics [162], Design optimisation [102, 163], Diagnosis [60], Power Technology [110, 140], Resource Management optimisation [42], etc.

4 Conclusions

We can deduce that if the purpose of using ANNs is to find the best network configuration for solving particular problems, is has been possible employing EOs, as it have mentioned above in several works. EOs provides good approximation to get success and speed of training of neural network based on the initial parameter settings, such as architecture, initial weights, learning rates, and others.

In the other hand, ANNs based objective function allows the fitness to be evaluated in a small fraction of the time taken to perform first principles analysis and permits the EAs to complete in a reasonably small amount of time.

Due to the last advances in both methodologies, there are several chances for future develops of joint procedures especially when complex industrial applications are addressed. Anyway the use of big computing facilities will continue being still necessary for that applications.