Keywords

JEL Classification

Introduction

Artificial neural networks (ANNs) constitute a class of flexible nonlinear models designed to mimic biological neural systems. Typically, a biological neural system consists of several layers, each with a large number of neural units (neurons) that can process the information in a parallel manner. The models with these features are known as ANN models. Such models can be traced back to the simple input–output model of McCulloch and Pitts (1943) and the ‘perceptron’ of Rosenblatt (1958). The early yet simple ANN models, however, did not receive much attention because of their limited applicability and also because of the limitation of computing capacity at that time. In seminal works, Rumelhart, McClelland and PDP Research Group (1986b) and McClelland, Rumelhart and PDP Research Group (1986) presented the new developments of ANN, including more complex and flexible ANN structures and a new network learning method. Since then, ANN has become a rapidly growing research area.

As far as model specification is concerned, ANN has a multi-layer structure such that the middle layer is built upon many simple nonlinear functions that play the role of neurons in a biological system. By allowing the number of these simple functions to increase indefinitely, a multi-layered ANN is capable of approximating a large class of functions to any desired degree of accuracy, as shown in, for example, Cybenko (1989), Funahashi (1989), Hornik, Stinchcombe and White (1989, 1990), and Hornik (1991, 1993). From an econometric perspective, ANN can be applied to approximate the unknown conditional mean (median, quantile) function of the variable of interest without suffering from the problem of model misspecification, unlike parametric models commonly used in empirical studies. Although nonparametric methods, such as series and polynomial approximators, also possess this property, they usually require a larger number of components to achieve similar approximation accuracy (Barron 1993). ANNs are thus a parsimonious approach to nonparametric functional analysis.

ANNs have been widely applied to solve many difficult problems in different areas, including pattern recognition, signal processing, and language learning. Since White (1988), there have also been numerous applications of ANN in economics and finance. Unfortunately, the ANN literature is not easy to penetrate, so it is hard for applied economists to understand why ANN works and how it can be implemented properly. Fortunately, while the ANN jargon originated from cognitive science and computer science, they often have econometric interpretations. For example, a ‘target’ is, in fact, a dependent variable of interest, an ‘input’ is an explanatory variable, and network ‘learning’ amounts to the estimation of unknown parameters in a network. The purpose of this article is thus twofold. First, it introduces ANN using familiar econometric terminology and hence serves to bridge the gap between the fields of ANN and economics. Second, it provides an overview of ANN modelling approach and its implementation methods. For an early review of ANN from an econometric perspective, we refer to Kuan and White (1994).

This article proceeds as follows. We introduce various ANN model specifications and the choices of network functions in section “ANN model specifications”. We present the ‘universal approximation’ property of ANN in section “ANN as an universal approximator”. Model estimation and model complexity regularization are discussed in section “Implementation of ANNs”. Section “Concluding remarks” concludes.

ANN Model Specifications

Let Y denote the collection of n variables of interest with the t-th observation yt (n × 1) and X the collection of m explanatory variables with the t-th observation xt (m × 1). In the ANN literature, the variables in Y are known as targets or target variables, and the variables in X are inputs or input variables. There are various ways to build an ANN model that can be used to characterize the behavior of yt using the information contained in the input variables xt. In this section, we introduce some network architectures and the functions that are commonly used to build an ANN.

Feedforward Neural Networks

We first consider a network with an input layer, an output layer, and a hidden layer in between. The input (output) layer contains m input units (n output units) such that each unit corresponds to a particular input (output) variable. In the hidden layer, there are q hidden units connected to all input and output units; the strengths of such connections are labelled by (unknown) parameters known as the network connection weights. In particular, γh = (γh.1,…, γh,m)′ denotes the vector of the connection weights between the h-th hidden unit and all m input units, and βj = (βj,1;…; βj,q)′ denotes the vector of the connection weights between the j-th output unit and all q hidden units. An ANN in which the sample information (signals) are passed forward from the input layer to the output layer without feedback is known as a feedforward neural network. Figure 1 illustrates the architecture of a three-layer feedforward network with three input units, four hidden units and two output units.

Artificial Neural Networks, Fig. 1
figure 121figure 121

A feedforward network with three input units, four hidden units and two output units

Full size image

This multi-layered structure of a feedforward network is designed to function as a biological neural system. The input units are the neurons that receive the information (stimuli) from the outside environment and pass them to the neurons in a middle layer (that is, hidden units). These neurons then transform the input signals to generate neural signals and forward them to the neurons in the output layer. The output neurons in turn generate signals that determine the action to be taken. Note that all information from the units in one layer is processed simultaneously, rather than sequentially, by the units in an ‘upper’ layer. (This concept, also known as parallel processing or massive parallelism, differs from the traditional concept of sequential processing and has led to a major advance in designing computer architecture.)

Formally, the input units receive the information xt and send to all hidden units, weighted by the connection weights between the input and hidden units. This information is then transformed by the activation function G in each hidden unit. That is, the h-th hidden unit receives x′tγh and transforms it to G(x′tγh). The information generated by all hidden units is further passed to the output units, again weighted by the connection weights, and transformed by the activation function F in each output unit. Hence, the j-th output unit receives \( {\sum}_{h=1}^q{\beta}_{j,h}G\left({\mathbf{x}}_t^{\prime }{\gamma}_{\mathrm{h}}\right) \) and transforms it into the network output:

$$ {o}_{t,j}=F\left(\sum_{h=1}^q{\beta}_{j,h}G\left({\mathbf{x}}_t^{\prime }{\gamma}_{\mathrm{h}}\right)\right),\ \ \ j=1,\dots, n. $$
(1)

The output Oj is used to describe or predict the behaviour of the j-th target Yj.

In practice, it is typical to include a constant term, also known as the bias term, in each activation function in (1). That is,

$$ {o}_{t,j}=F\left({\beta}_{j,0}+\sum_{h=1}^q{\beta}_{j,h}G\left({\gamma}_{j,0}+{\mathbf{x}}_t^{\prime }{\gamma}_{\mathrm{h}}\right)\right),j=1,\dots, n, $$
(2)

where γh,0 is the bias term in the h-th hidden unit and βj,0 is the bias term in the j-th output unit. A constant term in each activation function adds flexibility to hidden-unit and output-unit responses (activations), in a way similar to the constant term in (non)linear regression models. Note that when there is no transformation in the output units, F is an identity function (that is, F(a) = a) so that

$$ {o}_{t,j}={\beta}_{j,0}+\sum_{h=1}^q{\beta}_{j,h}G\left({\gamma}_{j,0}+{\mathbf{x}}_t^{\prime }{\gamma}_{\mathrm{h}}\right),\ \ \ j=1,\dots, n. $$
(3)

It is also straightforward to construct networks with two or more hidden layers. For simplicity, we will focus on the three-layer networks with only one hidden layer.

While parametric econometric models are typically formulated using a given function of the input xt, the network (2) is a class of flexible nonlinear functions of xt. The exact form of a network model depends on the activation functions (F and G) and the number of hidden units (q). In particular, the network function in (3) is an affine transformation of G and hence may be interpreted as an expansion with the ‘basis’ function G.

The networks (2) and (3) can be further extended. For example, one may construct a network in which the input units are connected not only to the hidden units but also directly to the output units. This leads to networks with short-cut connections. Corresponding to (2), the outputs of a feedforward network with short cuts are

$$ {o}_{t,j}=F\left({\beta}_{j,0}+{\mathbf{x}}_t^{\prime }{\boldsymbol{\upalpha}}_j\sum_{h=1}^q{\beta}_{j,h}G\left({\mathbf{x}}_t^{\prime }{\upgamma}_{\mathrm{h}}\right)\right),\ \ \ j=1,\dots, n, $$

where αj is the vector of connection weights between the output and input units, and, corresponding to (3), the outputs are

$$ {o}_{t,j}={\beta}_{j,0}+{\mathbf{x}}_t^{\prime }{\boldsymbol{\upalpha}}_j\sum_{h=1}^q{\beta}_{j,h}G\left({\gamma}_{j,0}+{\mathbf{x}}_t^{\prime }{\gamma}_{\mathrm{h}}\right),\ \ \ j=1,\dots, n. $$

Figure 2 illustrates the architecture of a feedforward network with two input units, three hidden units, one output unit and short-cut connections. Thus, parametric econometric models may be interpreted as feedforward networks with short-cut connections but no hidden-layer connections. The linear combination of hidden-unit activations, \( {\sum}_{h=1}^q{\beta}_{j,h}G\left({\upgamma_h}_{,0}+{\mathbf{x}}_t^{\prime }{\upgamma}_{\mathrm{h}}\right), \)in effect characterizes the nonlinearity not captured by the linear function of xt.

Artificial Neural Networks, Fig. 2
figure 122figure 122

A feedforward neural network with short cuts

Full size image

Recurrent Neural Networks

From the preceding section we can see that there is no ‘memory’ device in feedforward networks that can store the signals generated earlier. Hence, feedforward networks treat all sample information as ‘new’; the signals in the past do not help to identify data features, even when sample information exhibits temporal dependence. As such, a feedforward network must be expanded to a large extent so as to represent complex dynamic patterns. This causes practical difficulty because a large network may not be easily implemented. To utilize the information from the past, it is natural to include lagged target information yt−k, k = 1,…, s, as input variables, similar to linear AR and ARX models in econometric studies. Yet such networks do not have any built-in structure that can ‘memorize’ previous neural responses (transformed sample information). The so-called recurrent neural networks overcome this difficulty by allowing internal feedbacks and hence are especially appropriate for dynamic problems.

Jordan (1986) first introduced a recurrent network with feedbacks from output units. That is, the output units are connected to input units but with time delay, so that the network outputs at time t−1 are also the input information at time t. Specifically, the outputs of a Jordan network are

$$ {\displaystyle \begin{array}{l}{o}_{t,j}=F\left({\beta}_{j,0}+\sum \limits_{h=1}^q{\beta}_{j,h}G\left({\gamma}_{j,0}+{\mathbf{x}}_t^{\prime }{\upgamma}_{\mathrm{h}}+{\mathbf{o}}_{t-1}^{\prime }{\boldsymbol{\updelta}}_h\right)\right),\hfill \\ {}\ \ \ j=1,\dots, n,\hfill \end{array}} $$
(4)

where δh is the vector of the connection weights between the h-th hidden unit and the input units that receive lagged outputs ot−1 = (ot−1,1,…,ot−1,n)′. The network (4) can be further extended to allow for more lagged outputs ot−2, ot−3,.. ..

Similarly, Elman (1990) considered a recurrent network in which the hidden units are connected to input units with time delay. The outputs of an Elman network are:

$$ {o}_{t,j}=F\left({\beta}_{j,0}+\sum_{h=1}^q{\beta}_{j,h}{a}_{t,h}\right),\ \ \ j=1,\dots, n,\ \ {a}_{t,h}=G\left({\gamma}_{h,0}+{\mathbf{x}}_t^{\prime }{\upgamma}_{\mathrm{h}}+{\mathbf{a}}_{t-1}^{\prime }{\boldsymbol{\updelta}}_h\right),h=1,\dots, q, $$
(5)

where at−1 = (at−1,1,…, at−1,q)′ is the vector of lagged hidden-unit activations, and δh here is the vector of the connection weights between the h-th hidden unit and the input units that receive lagged hidden-unit activations at−1. The network (5) can also be extended to allow for more lagged hidden-unit activations at−2, at−3, Figure 3 illustrates the architectures of a Jordan network and an Elman network.

Artificial Neural Networks, Fig. 3
figure 123figure 123

Recurrent neural networks: Jordan (left) and Elman (right)

Full size image

From (4) and (5) we can see that, by recursive substitution, the outputs of these recurrent networks can be expressed in terms of current and all past inputs. Such expressions are analogous to the distributed lag model or the AR representation of an ARMA model (when the inputs are lagged targets). Thus, recurrent networks incorporate the information in the past input variables without including all of them in the model. By contrast, a feedforward network requires a large number of inputs to carry such information. Note that the Jordan network and the Elman network summarize past input information in different ways and hence have their own merits. When the previous ‘location’ of a network is crucial in determining the next move, as in the design of a robot, a Jordan network seems more appropriate. When the past internal neural responses are more important, as in language learning problems, an Elman network may be preferred.

Choices of Activation Function

As far as model specifications are concerned, the building blocks of an ANN model are the activation functions F and G. Different choices of the activation functions result in different network models. We now introduce some activation functions commonly employed in empirical studies.

Recall that the hidden units play the role of neurons in a biological system. Thus, the activation function in each hidden unit determines whether a neuron should be turned on or off. Such an on/off response can be easily represented using an indicator (threshold) function, also known as a heaviside function in the ANN literature, that is,

$$ G\left({\gamma}_{h,0}+{\mathbf{x}}_t^{\prime }{\gamma}_{\mathrm{h}}\right)=\left\{\begin{array}{l}1,\ \ \ \mathrm{If}\ \ \ {\gamma}_{h,0}+{\mathbf{x}}_t^{\prime }{\gamma}_{\mathrm{h}}\ge c,\hfill \\ {}0,\ \ \ \ \mathrm{If}\ \ \ {\gamma}_{h,0}+{\mathbf{x}}_t^{\prime }{\gamma}_{\mathrm{h}}<c,\hfill \end{array}\right. $$

where c is a pre-determined threshold value. That is, depending on the strength of connection weights and input signals, the activation function G will determine whether a particular neuron is on \( \left(G\left({\gamma}_{h,0}+{\mathbf{x}}_t^{\prime }{\gamma}_{\mathrm{h}}\right)=1\right) \) or off \( \left(G\left({\gamma}_{h,0}+{\mathbf{x}}_t^{\prime }{\gamma}_{\mathrm{h}}\right)=1\right). \)

In a complex neural system, neurons need not have only an on/off response but may be in an intermediate position. This amounts to allowing the activation function to assume any value between 0 and 1. In the ANN literature, it is common to choose a sigmoid (S-shaped) and squashing (bounded) function. In particular, if the input signals are ‘squashed’ between 0 and 1, the activation function is understood as a smooth counterpart of the indicator function. A leading example is the logistic function:

$$ G\left({\gamma}_{h,0}+{\mathbf{x}}_t^{\prime }{\gamma}_{\mathrm{h}}\right)=\frac{1}{1+\exp \left(-\left[{\gamma}_{h,0}+{\mathbf{x}}_t^{\prime }{\gamma}_{\mathrm{h}}\right]\right)}, $$

which approaches 1 (zero) when its argument goes to infinity (negative infinity). Hence, the logistic activation function generates a partially on/off signal based on the received input signals.

Alternatively, the hyperbolic tangent (tanh) function, which is also a sigmoid and squashing function, can serve as an activation function:

$$ G\left({\gamma}_{h,0}+{\mathbf{x}}_t^{\prime }{\gamma}_{\mathrm{h}}\right)=\frac{\exp \left({\gamma}_{h,0}+{\mathbf{x}}_t^{\prime }{\gamma}_{\mathrm{h}}\right)-\exp \left(-\left[{\gamma}_{h,0}+{\mathbf{x}}_t^{\prime }{\gamma}_{\mathrm{h}}\right]\right)}{\exp \left({\gamma}_{h,0}+{\mathbf{x}}_t^{\prime }{\gamma}_{\mathrm{h}}\right)+\exp \left(-\left[{\gamma}_{h,0}+{\mathbf{x}}_t^{\prime }{\gamma}_{\mathrm{h}}\right]\right)}. $$

Compared with the logistic function, this function may assume negative values and is bounded between − 1 and 1. It approaches 1 (−1) when its argument goes to infinity (minus infinity). This function is more flexible because the negative values, in effect, represent ‘suppressing’ signals from the hidden unit. See Fig. 4 for an illustration of the logistic and tanh functions. Note that for the logistic function G, a re-scaled function \( \tilde{G} \) such that \( \tilde{G}(a)=2G(a)-1 \) also generates values between −1 and 1 and may be used in place of the tanh function. (A choice of the activation function in classification problems is the so-called radial basis function. We do not discuss this choice because its argument is not an affine transformation of inputs and hence does not fit in our framework here. Moreover, the networks with this activation function provide only local approximation to unknown functions, in contrast with the approximation property discussed in section “ANN as an universal approximator”.)

Artificial Neural Networks, Fig. 4
figure 124figure 124

Activation functions: logistic (left) and tanh (right)

Full size image

The aforementioned activation functions are chosen for convenience because they are differentiable everywhere and their derivatives are easy to compute. In particular, when G is the logistic function,

$$ \frac{dG(a)}{da}=G(a)\left[1-G(a)\right]; $$

when G is the tanh function,

$$ \frac{dG(a)}{da}={\left[\frac{2}{\exp (a)+\exp \left(-a\right)}\right]}^2=\mathsf{\sec} {\mathsf{h}}^{\mathsf{2}}(a). $$

These properties facilitate parameter estimation, as will be seen in section “Model estimation”. Nevertheless, these functions are not necessary for building proper ANNs. For example, smooth cumulative distribution functions, which are sigmoidal and squashing, are also legitimate candidates for activation function. In section “ANN as an universal approximator”, it is shown that, as far as network approximation property is concerned, the activation function in hidden units does not even have to be sigmoidal, yet boundedness is usually required. Thus, sine and cosine functions can also serve as an activation function.

As for the activation function F in the output units, it is common to set it as the identity function so that the outputs of (3) enjoy the freedom of assuming any real value. This choice suffices for the network approximation property discussed in section “ANN as an universal approximator”. When the target is a binary variable taking the values zero and one, as in a classification problem, F may be chosen as the logistic function so that the outputs of (2) must fall between 0 and 1, analogous to a logit model in econometrics.

ANN as an Universal Approximator

What makes ANN a useful econometric tool is its universal approximation property, which basically means that a multi-layered ANN with a large number of hidden units can well approximate a large class of functions. This approximation property is analogous to that of nonparametric approximators, such as polynomials and Fourier series, yet it is not shared by parametric econometric models.

To present the approximation property, we consider the network function element by element. Let \( {f}_{G,q}:{\mathbb{R}}^m\times {\Theta}_{m,q}\to \mathbb{R} \) denote the network function with q hidden units, the output activation function F being the identity function, and the hidden-unit activation function G, that is,

$$ {f}_{G,q}\left(\mathbf{x};\boldsymbol{\uptheta} \right)={\beta}_0+\sum_{h=1}^q{\beta}_hG\left({\gamma}_{h,0}+{\mathbf{x}}^{\prime }{\gamma}_{\mathrm{h}}\right), $$

as in (3), where Θm,q is the parameter space whose dimension depends on m and q, and θ ∈ Θm,q (note that the subscripts m and q for θ are suppressed). Given the activation function G, the collection of all fG,q functions with different q is:

FG=⋃q=1∞{fG,q:fG,qxθ=β0+∑h=1qβhGγh,0+x′γh}

when the union is taken up to a finite number N, the resulting collection is denoted as \( {\mathcal{F}}_G^N \). Intuitively, ℱG is capable of functional approximation because fG,q can be viewed as an expansion with the ‘basis’ function G and hence is similar to a nonparametric approximator.

More formally, we follow Hornik (1991) and consider two measures of the closeness between functions. First define the uniform distance between functions f and g on the set K as

$$ {d}_K\left(f,g\right)=\underset{\mathbf{x}\in K}{\sup}\ \ \left|f\left(\mathbf{x}\right)-g\left(\mathbf{x}\right)\right|. $$

Let K denote a compact subset in ℝm and C(K) denote the space of all continuous functions on K. Then, when the activation function G is continuous, bounded and nonconstant, the collection ℱG is dense in C(K) for all K in ℝm in terms of dK (Theorem 2 of Hornik 1991). (Hornik 1991, considered the network without the bias term in the output unit, that is, β0 = 0. Yet as long as G is not a constant function, all the results in Hornik 1991, carry over; see Stinchcombe and White 1998, for details.) That is, for any function g in C(K) and any ε > 0, there is a network function fG,q in ℱG such that dK (fG;q − g) < ε. As \( {\mathcal{F}}_G^N \) is not dense in C (K) for any finite number N, this result shows that any continuous function can be approximated arbitrarily well on compacta by a three-layered feedforward network fG,q, provided that q, the number of hidden units, is sufficiently large.

Taking x as random variables, defined in the probability space with the probability measure ℙ, we consider the Lr-norm of f(x)−g(x):

$$ {\left\Vert f-g\right\Vert}_r{\left({\int}_{{\mathbb{R}}^m}{\left|f\left(\mathbf{x}\right)-g\left(\mathbf{x}\right)\right|}^rd\mathrm{\mathbb{P}}\left(\mathbf{x}\right)\right)}^{1/r}, $$

1 ≤ r < ∞. For r = 2 (r = 1), this is the well-known measure of mean squared error (mean absolute error). Then, when the activation function G is bounded and nonconstant, the collection ℱG is dense in the Lr space (Theorem 1 of Hornik 1991). That is, any function g (with finite Lr-norm) can also be well approximated by a three-layered feedforward network fG,q in terms of Lr-norm when q is sufficiently large.

It should be emphasized that the universal approximation property of a feedforward network hinges on the three-layered architecture and the number of hidden units, but not on the activation function per se. As stated above, the activation function in the hidden unit can be a general bounded function and does not have to be sigmoidal. Hornik (1993) provides results that permit even more general activation functions. Moreover, a feedforward network with only one hidden layer suffices for such approximation property. More hidden layers may be helpful in certain applications but are not necessary for functional approximation.

Barron (1993) further derived the rate of approximation in terms of mean squared error \( {\left\Vert f-g\right\Vert}_2^2. \)It was shown that three-layered feedforward networks fG,q with G a sigmoidal function can achieve the approximation rate of order O(1/q), for which the number of parameters grows linearly with q (with the order O(mq)). This is in sharp contrast with other expansions, such as polynomial (with p the degree of the polynomial) and spline (with p the number of knots per coordinate), which yield suitable approximation when the number of parameters grows exponentially (with the order O(pm)). Thus, it is practically difficult for such expansions to approximate well when the dimension of the input space, m, is large.

Implementation of ANNs

In practice, when the activation functions in an ANN are chosen, it remains to estimate its connection weights (unknown parameters) and to determine a proper number of hidden units. Given that the connection weights of an ANN model are unknown, this network must be properly ‘trained’ so as to ‘learn’ the unknown weights. This is why parameter estimation is referred to as network learning and the sample used for parameter estimation is referred to a training sample in the ANN literature. As the number of hidden units q determines network complexity, finding a suitable q is known as network complexity regularization.

Model Estimation

The network parameters can be estimated by either online or offline methods. An online learning algorithm is just a recursive estimation method which updates parameter estimates when new sample information becomes available. By contrast, offline learning methods are based on fixed training samples; standard econometric estimation methods are typically offline.

To ease the discussion of model estimation, we focus on the simple case that there is only one target variable y and the network function fG,q. Generalization to the case with multiple target variables and vector-valued network functions is straightforward. Once the activation function G is chosen and the number of hidden units is given, fG,q is a nonlinear parametric model for the target y; the network with multiple outputs is a system of nonlinear models. If we take mean squared error as the criterion, the parameter vector of interest θ* thus minimizes

$$ \mathbb{E}{\left[y-{f}_{G,q}\left(\mathbf{x};\boldsymbol{\uptheta} \right)\right]}^2. $$
(6)

It is well known that

$$ \mathbb{E}{\left[y-{f}_{G,q}\left(\mathbf{x};\boldsymbol{\uptheta} \right)\right]}^2=\mathbb{E}{\left[y-E\left(y|\mathbf{x}\right)\right]}^2+\mathbb{E}{\left[\mathbb{E}\left(y|\mathbf{x}\right)-{f}_{G,q}\left(\mathbf{x};\boldsymbol{\uptheta} \right)\right]}^2. $$

As \( \mathbb{E}\left(y|\mathbf{x}\right) \) is the best L2 predictor of y, θ* must also minimize the mean squared approximation error: \( \mathbb{E}{\left[\mathbb{E}\left(y|\mathbf{x}\right)-{f}_{G,q}\left(\mathbf{x};\boldsymbol{\uptheta} \right)\right]}^2 \). This shows that, among all three-layered feedforward networks with the activation function G and q hidden units, fG,q(x;θ*) provides the best approximation to the conditional mean function.Given a training sample of T observations, an estimator of θ* can be obtained by minimizing the sample counterpart of (6):

$$ \frac{1}{T}\sum_{t=1}^T{\left[{y}_t-{f}_{G,q}\left(\mathbf{x};\boldsymbol{\uptheta} \right)\right]}^2, $$

which is just the objective function of the nonlinear least squares (NLS) method. The NLS method is an offline estimation method because the size of the training sample is fixed. Under very general conditions on the data and nonlinear function, it is well known that the NLS estimator is strongly consistent for θ* and asymptotically normally distributed (see, for example, Gallant and White 1988).

In many ANN applications (for example, signal processing and language learning), the training sample is not fixed but constantly expands with new data. In such cases, offline estimation may not be feasible, but online estimation methods, which update the parameter estimates based solely on the newly available data, are computationally more tractable. Moreover, online estimation methods can be interpreted as ‘adaptive learning’ by biological neural systems. It should be emphasized that when there is only a given sample, as in most empirical studies in economics, recursive estimation is not to be preferred because it is, in general, statistically less efficient than the NLS method in finite samples.

Note that the parameter of interest θ* is the zero of the first order condition of (6):

$$ \mathbb{E}{\left[\nabla {f}_{G,q}\left(\mathbf{x};\boldsymbol{\uptheta} \right)\left(y-{f}_{G,q}\left(\mathbf{x};\boldsymbol{\uptheta} \right)\right)\right]}^2=0, $$

where \( \nabla {f}_{G,q}\left(\mathbf{x};\boldsymbol{\uptheta} \right) \) is the (column) gradient vector of fG,q with respect to θ. To estimate θ*, a recursive algorithm proposed by Rumelhart et al. (1986a) is

$$ {\widehat{\boldsymbol{\uptheta}}}_{t+1}={\widehat{\boldsymbol{\uptheta}}}_t+{\eta}_t\nabla {f}_{G,q}\left(\mathbf{x};\widehat{\boldsymbol{\uptheta}}\right)\left[{y}_t-{f}_{G,q}\left(\mathbf{x};\widehat{\boldsymbol{\uptheta}}\right)\right], $$
(7)

where ηt > 0 is a parameter that re-scales the adjustment term in the square bracket. It can be seen from (7) that the adjustment term is determined by the gradient descent direction and the error between the target and network output: \( {y}_t-{f}_{G,q}\left({\mathbf{x}}_t;{\widehat{\boldsymbol{\uptheta}}}_t\right) \), and it requires only the information at time t, that is, yt, xt, and the estimate \( {\widehat{\boldsymbol{\uptheta}}}_t \). (The algorithm (7) is analogous to the numerical steepest-descent algorithm. However, (7) utilizes only the information at time t, whereas numerical optimization algorithms are computed using all the information in a given sample and hence are offline methods.)

The algorithm (7) is known as the error back-propagation (or simply back-propagation) algorithm in the ANN literature, because the error signal \( \left[{y}_t-{f}_{G,q}\left({\mathbf{x}}_t;{\widehat{\boldsymbol{\uptheta}}}_t\right)\right] \) is propagated back through the network to determine the change of each weight. The underlying idea of this algorithm can be traced back to the classical stochastic approximation method introduced in Robbins and Monro (1951). White (1989) established consistency and asymptotic normality of \( {\widehat{\boldsymbol{\uptheta}}}_t \) in (7). Note that the parameter ηt in the algorithm is known as a learning rate. For consistency of \( {\widehat{\boldsymbol{\uptheta}}}_t \) it is required that ηt satisfies \( {\sum}_{t=1}^{\infty }{\eta}_t=\infty \) and \( {\sum}_{t=1}^{\infty }{\eta_t}_2<\infty \), for example, ηt = 1/t. The former condition ensures that the updating process may last indefinitely, whereas the latter implies \( {\eta_t}_2\to 0 \) so that the adjustment in the parameter estimates can be made arbitrarily small. (In many applications of ANN, the learning rate is often set to a constant ηo; the resulting estimate \( {\widehat{\boldsymbol{\uptheta}}}_t \) loses consistency in this case. Kuan and Hornik (1991) established a convergence result based on small −ηo asymptotics.)

Instead of the gradient descent direction, it is natural to construct a recursive algorithm with a Newton search direction. Kuan and White (1994) proposed the following algorithm:

$$ {\widehat{\mathbf{H}}}_{t+1}={\widehat{\mathbf{H}}}_t+{\eta}_t\left[\nabla {f}_{G,q}\left({\mathbf{x}}_t;{\widehat{\boldsymbol{\uptheta}}}_t\right)\nabla {f}_{G,q}{\left(\mathbf{x};\widehat{\boldsymbol{\uptheta}}\right)}^{\prime }={\widehat{\mathbf{H}}}_t\right],{\widehat{\boldsymbol{\uptheta}}}_{t+1}={\widehat{\boldsymbol{\uptheta}}}_t+{\eta}_t\;{\widehat{\mathbf{H}}}_{t+1}^{-1}\nabla {f}_{G,q}\left({\mathbf{x}}_t;{\widehat{\boldsymbol{\uptheta}}}_t\right)\left[{y}_t-{f}_{G,q}\left({\mathbf{x}}_t;{\widehat{\boldsymbol{\uptheta}}}_t\right)\right], $$
(8)

Where \( {\widehat{\mathbf{H}}}_{t+1} \) characterizes a Newton direction and is recursively updated via the first equation. Kuan and White (1994) showed that \( {\widehat{\boldsymbol{\uptheta}}}_t \) in (8) is \( \sqrt{1} \) -consistent, statistically more efficient than \( {\widehat{\boldsymbol{\uptheta}}}_t \) in (7), and asymptotically equivalent to the NLS estimator. The algorithm (8) may be implemented in different ways; for example, there is an algorithm that is algebraically equivalent to (8) but does not involve matrix inversion. See Kuan and White (1994) for more discussions on the implementation of the Newton algorithms.

On the other hand, estimating recurrent networks is more cumbersome. From (4) and (5) we can see that recurrent network functions depend on θ directly and also indirectly through the presence of internal feedbacks (that is, lagged output and lagged hidden-unit activations). The indirect dependence on parameters must be taken into account in calculating the derivatives with respect to θ. Thus, NLS optimization algorithms that require analytic derivatives are difficult to implement. Kuan et al. (1994) proposed the dynamic back-propagation algorithm for recurrent networks, which is analogous to (7) but involves more updating equations. Kuan (1995) further proposed a Newton algorithm for recurrent networks, analogous to (8), and showed that it is \( \sqrt{1} \) -consistent and statistically more efficient than the dynamic back-propagation algorithm. We omit the details of these algorithms; see Kuan and Liu (1995) for an application of these estimation methods for both feedforward and recurrent networks.

Note that the NLS method and recursive algorithms all require computing the derivatives of the network function. Thus, a smooth and differentiable activation function, as the examples given in section “Choices of activation function”, are quite convenient for network parameter estimation. Finally, given that ANN models are highly nonlinear, it is likely that there exist multiple optima in the objective function. There is, however, no guarantee that the NLS method and the recursive estimation methods discussed above will deliver the global optimum. This is a serious problem because the dimension of the parameter space is typically large. Unfortunately, a convenient and effective method for finding the global optimum in ANN estimation is not yet available.

Model Complexity Regularization

Section “ANN as an universal approximator” shows that a network model fG,q can approximate unknown function when the number of hidden units, q, is sufficiently large. When there is a fixed training sample, a complex network with a very large q may over fit the data. Thus, there is a trade-off between approximation capability and over-fitting in implementing ANN models.

An easy approach to regularizing the network complexity is to apply a model selection criterion, such as Schwarz (Bayesian) information criterion (BIC), to the network models with various q. (Alternatively, one may consider testing whether some hidden units may be dropped from the model. This amounts to testing, say, βh = 0 for some h. Unfortunately, the parameters in that hidden-unit activation function (γh,0 and γh) are not identified under this null hypothesis. It is well known that, when there are unidentified nuisance parameters, standard econometric tests are not applicable.) As is well known, BIC consists of two terms: one is based on model fitness, and the other penalizes model complexity. Hence, it is suitable for regularizing network complexity; see also Barron (1991). A different criterion introduced in Rissanen (1986, 1987) is predictive stochastic complexity (PSC) which is just an average of squared prediction errors:

$$ \mathrm{PSC}=\frac{1}{T-k}\sum_{t=k=1}^T{\left[{y}_t-{f}_{G,q}\left({\mathbf{x}}_t;{\widehat{\boldsymbol{\uptheta}}}_t\right)\right]}^2, $$

where \( {\widehat{\boldsymbol{\uptheta}}}_t \) is the predicted parameter estimate based on the sample information up to time t−1, and k is the total number of parameters in the network. Given the number of inputs, the network with the smallest BIC or PSC gives the desired number of hidden units q*. Rissanen showed that both BIC and PSC can be interpreted as the criteria for ‘minimum description length’ in the sense that they determine the shortest code length (asymptotically) that is needed to encode a sequence of numbers. In other words, these criteria lead to the least complex model that still captures the key information in data. Swanson and White (1997) showed that a network selected by BIC need not perform well in out-of-sample forecasting, however.

Clearly, PSC requires estimating the parameters at each t. It would be computationally demanding if the NLS method is to be used, even for a moderate sample. For simplicity, Kuan and Liu (1995) suggested a two-step procedure for implementing ANN models. In the first step, one estimates the network models and computes the resulting PSCs using the recursive Newton algorithm, which is asymptotically equivalent to the NLS method. When a suitable network structure is determined, the Newton parameter estimates can be used as initial values for NLS estimation in the second step. This approach thus maintains a balance between computational cost and estimator efficiency.

Concluding Remarks

In this article, we introduce ANN model specifications, their approximation properties, and the methods for model implementation from an econometric perspective. It should be emphasized that ANN is neither a magical econometric tool nor a ‘black box’ that can solve any difficult problems in econometrics. As discussed above, a major advantage of ANN is its universal approximation property, a property shared by other nonparametric approximators. Yet, compared with parametric econometric models, a simple ANN need not perform better, and a more complex ANN (with a large number of hidden units) is more difficult to implement properly and cannot be applied when there is only a small data-set. Therefore, empirical applications of ANN models must be exercised with care.

See Also