Evolving Connectionist Systems: From Neuro-Fuzzy-, to Spiking- and Neuro-Genetic

Abstract

This chapter follows the development of a class of neural networks (GlossaryTerm

NN

) called evolving connectionist systems (GlossaryTerm

ECOS

). The term evolving is used here in its meaning of unfolding, developing, changing, revealing (according to the Oxford dictionary) rather than evolutionary. The latter represents processes related to populations and generations of them. An GlossaryTerm

ECOS

is a neural network-based model that evolves its structure and functionality through incremental, adaptive learning and self-organization during its lifetime. In principle, it could be a simple GlossaryTerm

NN

or a hybrid connectionist system. The latter is a system based on neural networks that also integrate other computational principles, such as linguistically meaningful explanation features of fuzzy rules, optimization techniques for structure and parameter optimization, quantum-inspired methods, and gene regulatory networks. The chapter includes definitions and examples of GlossaryTerm

ECOS

such as: evolving neuro-fuzzy and hybrid systems; evolving spiking neural networks, neurogenetic systems, quantum-inspired systems, which are all discussed from the point of view of the structural and functional development of a connectionist-based model and the knowledge that it represents. Applications for knowledge engineering across domain areas, such as in bioinformatics, brain study, and intelligent machines are presented.

1 Principles of Evolving Connectionist Systems (ECOS)

Everything in Nature evolves, develops, unfolds, reveals, and changes in time. The brain is probably the ultimate evolving system, which develops during a lifetime, based on genetic information (Nature) and learning from the environment (nurture). Inspired by information principles of the developing brain, GlossaryTerm

ECOS

are adaptive, incremental learning and knowledge representation systems that evolve their structure and functionality from incoming data through interaction with the environment, where in the core of a system is a connectionist architecture that consists of neurons (information processing units) and connections between them [1]. An GlossaryTerm

ECOS

is a system based on neural networks and the use of also other techniques of computational intelligence (GlossaryTerm

CI

), which operates continuously in time and adapts its structure and functionality through continuous interaction with the environment and with other systems. The adaptation is defined through:

• 1. A set of evolving rules.

• 2. A set of parameters (genes) that are subject to change during the system operation.

• 3. An incoming continuous flow of information, possibly with unknown distribution.

• 4. Goal (rationale) criteria (also subject to modification) that are applied to optimize the performance of the system over time.

GlossaryTerm

ECOS

learning algorithms are inspired by brain-like information processing principles, e. g.:

• 1. They evolve in an open space, where the dimensions of the space can change.

• 2. They learn via incremental learning, possibly in an on-line mode.

• 3. They learn continuously in a lifelong learning mode.

• 4. They learn both as individual systems and as an evolutionary population of such systems.

• 5. They use constructive learning and have evolving structures.

• 6. They learn and partition the problem space locally, thus allowing for a fast adaptation and tracing the evolving processes over time.

• 7. They evolve different types of knowledge representation from data, mostly a combination of memory- based and symbolic knowledge.

Many methods, algorithms, and computational intelligence systems have been developed since the conception of GlossaryTerm

ECOS

and many applications across disciplines. This chapter will review only the fundamental aspects of some of these methods and will highlight some principal applications.

2 Hybrid Systems and Evolving Neuro-Fuzzy Systems

2.1 Hybrid Systems

A hybrid computational intelligent system integrates several principles of computational intelligence to enhance different aspects of the performance of the system. Here we will discuss only hybrid connectionist systems that integrate artificial neural networks (GlossaryTerm

NN

) with other techniques utilizing the adaptive learning features of the GlossaryTerm

NN

.

Early hybrid connectionist systems combined GlossaryTerm

NN

with rule-based systems such as production rules [3] or predicate logic [4]. They utilized GlossaryTerm

NN

modules for a lower level of information processing and rule-based systems for reasoning and explanation at a higher level.

The above principle is applied when fuzzy rules are used for higher-level information processing and for approximate reasoning [5, 6, 7]. These are expert systems that combine the learning ability of GlossaryTerm

NN

with the explanation power of linguistically plausible fuzzy rules [10, 11, 8, 9]. A block diagram of an exemplar system is shown in Fig. 40.1, where at a lower level a neural network (GlossaryTerm

NN

) module predicts the level of a stock index and at a higher level a fuzzy reasoning module combines the predicted values with some macro-economic variables representing the political and the economic situations using the following types of fuzzy rules [2]

$$\mathit{\text{IF <the predicted by the NN module stock value}}\mathit{\text{in the future is high>}}\mathit{\text{AND}}\mathit{\text{<the economic situation is good>}}\mathit{\text{AND}}\mathit{\text{<the political situation is stable>}}\mathit{\text{THEN <buy stock>}}.$$

(40.1)

Fig. 40.1

A hybrid NN-fuzzy rule-based expert system for financial decision support (after [2])

Along with the integration of GlossaryTerm

NN

and fuzzy rules for a better decision support, the system from Fig. 40.1 includes an GlossaryTerm

NN

module for extracting recent rules form data that can be used by experts to analyze the dynamics of the stock and to possibly update the trading fuzzy rules in the fuzzy rule-based module. This GlossaryTerm

NN

module uses a fuzzy neural (GlossaryTerm

FNN

) network for the rule extraction.

Fuzzy neural networks (GlossaryTerm

FNN

) integrate GlossaryTerm

NN

and fuzzy rules into a single neuronal model tightly coupling learning and fuzzy reasoning rules into a connectionist structure. One of the first GlossaryTerm

FNN

models was initiated by Yamakawa and other Japanese scientists and promoted at a series of IIZUKA conferences in Japan [12, 13]. Many models of GlossaryTerm

FNN

s were developed based on these principles [14, 15, 2].

2.2 Evolving Neuro-Fuzzy Systems

The evolving neuro-fuzzy systems further extended the principles of hybrid neuro-fuzzy systems and the GlossaryTerm

FNN

, where instead of training a fixed connectionist structure, the structure and its functionality evolve from incoming data, often in an on-line, one-pass learning mode. This is the case with evolving connectionist systems (GlossaryTerm

ECOS

) [1, 16, 17, 18, 19].

GlossaryTerm

ECOS

are modular connectionist-based systems that evolve their structure and functionality in a continuous, self-organized, on-line, adaptive, and interactive way from incoming information [17]. They can process both data and knowledge in a supervised and/or unsupervised way. GlossaryTerm

ECOS

learn local models from data through clustering of the data and associating a local output function for each cluster represented in a connectionist structure. They can learn incrementally single data items or chunks of data and also incrementally change their input features [18].

Elements of GlossaryTerm

ECOS

have been proposed as part of the early, classical GlossaryTerm

NN

models, such as Kohonen’s self organising maps (GlossaryTerm

SOM

) [20], redical basis function(GlossaryTerm

RBF

) [21], FuzyARTMap [22] by Carpenter etal and Fritzke’s growing neural gas [23], Platt’s resource allocation networks (RAN) [24].

Some principles of GlossaryTerm

ECOS

are:

• Neurons are created (evolved) and allocated as centers of (fuzzy) data clusters. Fuzzy clustering, as a means to create local knowledge-based systems, was stimulated by the pioneering work of Bezdek, Yager and Filev [27, 28, 29, 30].

• Local models are evolved and updated in these clusters.

Here we will briefly illustrate the concepts of GlossaryTerm

ECOS

on two implementations: evolving fuzzy neutral networks (GlossaryTerm

EFuNN

) [16] and dynamic neuro-fuzzy inference systems (GlossaryTerm

DENFIS

) [25]. Examples of GlossaryTerm

EFuNN

are shown in Figs. 40.2 and 40.3 and of GlossaryTerm

DENFIS

in Figs. 40.4 and 40.5. In GlossaryTerm

ECOS

, clusters of data are created (evolved) based on similarity between data samples (input vectors) either in the input space (this is the case in some of the GlossaryTerm

ECOS

models, e. g., GlossaryTerm

DENFIS

), or in both the input and output space (this is the case, e. g., in the GlossaryTerm

EFuNN

models). Samples that have a distance to an existing node (cluster center, rule node, neuron) less than a certain threshold are allocated to the same cluster. Samples that do not fit into existing clusters, form (generate, evolve) new clusters. Cluster centers are continuously adjusted according to new data samples, others are created incrementally. GlossaryTerm

ECOS

learn from data and automatically create or update a local (fuzzy) model/function in each cluster, e. g.,

$$\mathit{\text{IF}}<\mathit{\text{data is in a (fuzzy) cluster Ci >}}\mathit{\text{THEN}}<\mathit{\text{the model is Fi>}},$$

(40.2)

where Fi can be a fuzzy value, a linear or logistic regression function (Fig. 40.5), or an GlossaryTerm

NN

model [25].

Fig. 40.2

A simple, feedforward EFuNN structure. The rule nodes evolve from data to capture cluster centers in the input space, while the output nodes evolve local models to learn and approximate the data in each of these clusters

Fig. 40.3

An EFuNN structure with feedback connections (after [16])

Fig. 40.4 a,b

Learning in DENFIS uses the evolving clustering method illustrated on a simple example of $2$ inputs and $1$ output and $11$ data clusters evolved. The recall of the DENFIS for two new input vectors x ₁ and x ₂ is illustrated with the use of the $3$ closets clusters to the new input vector (after [25]). (a)  Fuzzy role group 1 for a DENFIS. (b)  Fuzzy role group 2 for a DENFIS

Fig. 40.5

An example of the DENFIS model (after [26]) for medical renal function evaluation

GlossaryTerm

ECOS

utilize evolving clustering methods. There is no fixed number of clusters specified a priori, but clusters are created and updated incrementally. Other GlossaryTerm

ECOS

that use this principle are: evolving self-organized maps (GlossaryTerm

ESOM

) [17], evolving classification function [18, 26], evolving spiking neural networks (Sect. [4]).

As an example, the following are the major steps for the training and recall of a GlossaryTerm

DENFIS

model:

Training::

• (a) Create or update a cluster from incoming data.

• (b) Create or update a Takagi–Sugeno fuzzy rule for each cluster:

IF x is in cluster Cj THEN yj$=$fj (x),

where: yi$=\beta 0+\beta 1$ x1$+\beta 2$ x2$+\mathrm{\cdots }+\beta$q.

The function coefficients are incrementally updated with every new input vector or after a chunk of data. Recall – fuzzy inference for a new input vector:

• 1. For a new input vector x$=$[x1, x2, $\mathrm{\dots },$ xq] GlossaryTerm

  DENFIS

  chooses m fuzzy rules from the whole fuzzy rule set for forming a current inference system.

• 2. The inference result is

  $$y=\frac{{\mathrm{\Sigma }}_{\text{i}=1,\text{m}}\left[\omega \text{i}\text{fi}\right(\text{x}1,\text{x}2,\mathrm{\dots },\text{xq}\left)\right]}{{\mathrm{\Sigma }}_{\text{i}=1,\text{m}}\omega \text{i}},$$

  (40.3)

  where i is the index of one of the m closets to the new input vector x clusters, ωi$=1-\text{di}$ is the weighted distance between this vector the cluster center, fi(x) is the calculated output for x according to the local model fi for cluster i.

2.3 From Local to Transductive (Individualized) Learning and Modeling

A special direction of GlossaryTerm

ECOS

is transductive reasoning and personalized modeling. Instead of building a set of local models fi (e. g., prototypes) to cover the whole problem space and then using these models to classify/predict any new input vector, in transductive modeling for every new input vector x a new model fx is created based on selected nearest neighbor vectors from the available data. Such GlossaryTerm

ECOS

models are neuro-fuzzy inference systems (GlossaryTerm

NFI

) [31] and the transductive weighted neuro-fuzzy inference system (GlossaryTerm

TWNFI

) [32]. In GlossaryTerm

TWNFI

for every new input vector the neighborhood of the closest data vectors is optimized using both the distance between the new vector and the neighboring ones and the weighted importance of the input variables, so that the error of the model is minimized in the neighborhood area [33]. GlossaryTerm

TWNFI

is a further development of the weighted-weighted nearest neighbor method (GlossaryTerm

WWKNN

) proposed in [34]. The output for a new input vector is calculated based on the outputs of the k-nearest neighbors, where the weighting is based on both distance and a priori calculated importance for each variable using a ranking method such as signal-to-noise ratio or the t-test.

Other GlossaryTerm

ECOS

were been developed as improvements of GlossaryTerm

EFuNN

, GlossaryTerm

DENFIS

, or other early GlossaryTerm

ECOS

models by Ozawa etal and Watts [35, 36, 37], including ensembles of GlossaryTerm

ECOS

 [38]. A similar approach to GlossaryTerm

ECOS

was used by Angelov in the development of the (GlossaryTerm

ETS

) models [39].

2.4 Applications

GlossaryTerm

ECOS

have been applied to problems across domain areas. It is demonstrated that local incremental learning or transductive learning are superior when compared to global learning models and when compared in terms of accuracy and new knowledge obtained. A review of GlossaryTerm

ECOS

applications can be found in [26]. The applications include:

• Medical decision support systems (Fig. 40.5)

• Bioinformatics, e. g., [40]

• Neuroinformatics and brain study, e. g., [41]

• Evolvable robots, e. g., [42]

• Financial and economic decision support systems, e. g., [43]

• Environmental and ecological modeling, e. g., [44]

• Signal processing, speech, image, and multimodal systems, e. g., [45]

• Cybersecurity, e. g., [46]

• Multiple time series prediction, e. g., [47].

While classical GlossaryTerm

ECOS

use a simple McCulloch and Pitts model of a neuron and the Hebbian learning rule [48], evolving spiking neural network (GlossaryTerm

eSNN

) architectures use a spiking neuron model, applying the same or similar GlossaryTerm

ECOS

principles.

3 Evolving Spiking Neural Networks (eSNN)

3.1 Spiking Neuron Models

A single biological neuron and the associated synapses is a complex information processing machine that involves short-term information processing, long-term information storage, and evolutionary information stored as genes in the nucleus of the neuron. A spiking neuron model assumes input information represented as trains of spikes over time. When sufficient input spikes are accumulated in the membrane of the neuron, the neuron’s post-synaptic potential exceeds a threshold and the neuron emits a spike at its axon (Fig. 40.6a,b). Some of the-state-of-the-art models of spiking neurons include: early models by Hodgkin and Huxley [49], and Hopfield [50]; and more recent models by Maass, Gerstner, Kistler, Izhikevich, Thorpe and van Ruller [51, 52, 53, 54]. Such models are spike response models (GlossaryTerm

SRM

s), the leaky integrate-and-fire model (GlossaryTerm

LIFM

) (Fig. 40.6), Izhikevich models, adaptive GlossaryTerm

LIFM

, and probabilistic IFM [55].

Fig. 40.6

(a) LIFM of a spiking neuron. (b) The LIFM increases its membrane potential $u\left(t\right)$ with every incoming spike at time t until the potential reaches a threshold, after which the neuron emits an output spike and its potential is reset to an initial value

3.2 Evolving Spiking Neural Networks (eSNN)

Based on the GlossaryTerm

ECOS

principles, an evolving spiking neural network architecture (GlossaryTerm

eSNN

) was proposed in [26], which was initially designed as a visual pattern recognition system. The first GlossaryTerm

eSNN

s were based on Thorpe’s neural model [54], in which the importance of early spikes (after the onset of a certain stimulus) is boosted, called rank-order coding and learning. Synaptic plasticity is employed by a fast supervised one-pass learning algorithm. An exemplar GlossaryTerm

eSNN

for multimodal auditory-visual information processing on the case study problem of speaker authentication is shown in Fig. 40.7.

Fig. 40.7

An exemplar eSNN for multimodal auditory-visual information processing in the case study problem of speaker authentication (after [56])

Different GlossaryTerm

eSNN

models use different architectures. Figure 40.8 shows a reservoir-based GlossaryTerm

eSNN

for spatio-temporal pattern recognition where the reservoir [57] uses the spike-time-dependent plasticity (GlossaryTerm

STDP

) learning rule [58], and the output classifier that classifies spatio-temporal activities of the reservoir uses rank-order learning rule [54].

Fig. 40.8

A reservoir-based eSNN for spatio-temporal pattern classification (after [55])

3.3 Extracting Fuzzy Rules from eSNN

Extracting fuzzy rules from an GlossaryTerm

eSNN

would make GlossaryTerm

eSNN

not only efficient learning models, but also knowledge-based models. A method was proposed in [59] and illustrated in Fig. 40.9a,b. Based on the connection weights w between the receptive field layer L1 and the class output neuron layer L2 fuzzy rules are extracted.

Fig. 40.9

(a) A simple structure of an eSNN for 2-class classification based on one input variable using six receptive fields to convert the input values into spike trains. (b)  The connection weights of the connections to class Ci and Cj output neurons, respectively, are interpreted as fuzzy rules${}_{:}$ IF(input variable v is SMALL) THEN class Ci; IF(v is LARGE)THEN class Cj

3.4 eSNN Applications

Different GlossaryTerm

eSNN

models and systems have been developed for different applications, such as:

• GlossaryTerm

  eSNN

  for spatio- and spectro-temporal pattern recognition – http://ncs.ethz.ch/projects/evospike

• Dynamic GlossaryTerm

  eSNN

  (GlossaryTerm

  deSNN

  ) for moving object recognition – [60]

• Spike pattern association neuron(GlossaryTerm

  SPAN

  ) for generation of precise time spike sequences as a response to recognized input spiking patterns – [61]

• Environmental and ecological modeling – [44]

• GlossaryTerm

  EEG

  data modeling – [62]

• Neuromorphic SNN hardware – [63, 64]

• Neurogenetic models (Sect. 40.4).

A review of GlossaryTerm

eSNN

methods, systems and their applications can be found in [65].

4 Computational Neuro-Genetic Modeling (CNGM)

4.1 Principles

A neuro-genetic model of a neuron was proposed in [41, 66]. It utilizes information about how some proteins and genes affect the spiking activities of a neuron such as fast excitation, fast inhibition, slow excitation, and slow inhibition. An important part of the model is a dynamic gene/protein regulatory network (GlossaryTerm

GRN

) model of the dynamic interactions between genes/proteins over time that affect the spiking activity of the neuron – Fig. 40.10.

Fig. 40.10

A schematic diagram of a computational neuro-genetic modeling (CNGM ) framework consisting of a gene/protein regulatory network (GRN ) as part of an eSNN (after [41])

A GlossaryTerm

CNGM

is a dynamical model that has two dynamical sub-models:

• GlossaryTerm

  GRN

  , which models dynamical interaction between genes/proteins over time scale GlossaryTerm

  T1

• GlossaryTerm

  eSNN

  , which models dynamical interaction between spiking neurons at a time scale GlossaryTerm

  T2

  .

The two sub-models interact over time.

4.2 The NeuroCube Framework

A further development of the GlossaryTerm

eSNN

and the GlossaryTerm

CNGM

was achieved with the introduction of the NeuroCube framework [67]. The main idea is to support the creation of multi-modular integrated systems, where different modules, consisting of different neuronal types and genetic parameters correspond in a way to different parts of the brain and different functions (e. g., vision, sensory information processing, sound recognition, motor-control) and the whole system works in an integrated mode for brain signal pattern recognition. A concrete model built with the use of the NeuroCube would have a specific structure and a set of algorithms depending on the problem and the application conditions, e. g., classification of GlossaryTerm

EEG

, recognition of functional magneto-resonance imaging (GlossaryTerm

fMRI

) data, brain computer interfaces, emotional cognitive robotics, and modeling Alzheimer’s disease.

A block diagram of the NeuroCube framework is shown in Fig. 40.11. It consists of the following modules:

• An input information encoding module

• A NeuroCube module

• An output module

• A gene regulatory network (GlossaryTerm

  GRN

  ) module.

Fig. 40.11

The NeuroCube framework (after [67])

The main principles of the NeuroCube framework are:

• 1. NeuroCube is a framework to model brain data (and not a brain model or a brain map).

• 2. NeuroCube is a selective, approximate map of relevant to the brain data brain regions, along with relevant genetic information, into a 3-D spiking neuronal structure.

• 3. An initial NeuroCube structure can include known connections between different areas of the brain.

• 4. There are two types of data used for both training a particular NeuroCube and to recall it on new data: (a) data, measuring the activity of the brain when certain stimuli are presented, e. g., (GlossaryTerm

  EEG

  , GlossaryTerm

  fMRI

  ); (b) direct stimuli data, e. g., sound, spoken language, video data, tactile data, odor data, etc.

• 5. A NeuroCube architecture, consisting of a NeuroCube module, (GlossaryTerm

  GRN

  )s at the lowest level, and a higher-level evaluation (classification) module.

• 6. Different types of neurons and learning rules can be used in different areas of the architecture.

• 7. Memory of the system is represented as a combination of: (a) short-term memory, represented as changes of the neuronal membranes and temporary changes of synaptic efficacy; (b) long-term memory, represented as a stable establishment of synaptic efficacy; (c) genetic memory, represented as a change in the genetic code and the gene/protein expression level as a result of the above short-term and long-term memory changes and evolutionary processes.

• 8. Parameters in the NeuroCube are defined by genes/proteins that form dynamic GlossaryTerm

  GRN

  models.

• 9. NeuroCube can potentially capture in its internal representation both spatial and temporal characteristics from multimodal brain data.

• 10. The structure and the functionality of a NeuroCube architecture evolve in time from incoming data.

4.3 Quantum-Inspired Optimization of eSNN and CNGM

A GlossaryTerm

CNGM

has a large number of parameters that need to be optimized for an efficient performance. Quantum-inspired optimization methods are suitable for this purpose as they can deal with a large number of variables and will converge in much faster time that any other optimization algorithms [68]. Quantum-inspired GlossaryTerm

eSNN

(GlossaryTerm

QeSNN

) use the principle of superposition of states to represent and optimize features (input variables) and parameters of the GlossaryTerm

eSNN

including genes in a GlossaryTerm

GRN

 [44]. They are optimized through a quantum-inspired genetic algorithm [44] or a quantum-inspired particle swarm optimization algorithm [69]. Features are represented as qubits in a superposition of $1$ (selected), with a probability α, and 0 (not selected) with a probability β. When the model has to be calculated, the quantum bits collapse in $1$ or $0$.

4.4 Applications of CNGM

Various applications of GlossaryTerm

CNGM

have been developed such as:

• Modeling brain diseases [41, 70]

• GlossaryTerm

  EEG

  and GlossaryTerm

  fMRI

  spatio-temporal pattern recognition [67].

5 Conclusions and Further Directions

This chapter presented a brief overview of the main principles of a class of neural networks called evolving connectionist systems (GlossaryTerm

ECOS

) along with their applications for computational intelligence. GlossaryTerm

ECOS

facilitate fast and accurate learning from data and new knowledge discovery across application areas. They integrate principles from neural networks, fuzzy systems, evolutionary computation, and quantum computing. The future directions and applications of GlossaryTerm

ECOS

are foreseen as a further integration of principles from information science-, bio-informatics, and neuro-informatics [71].