Multi-variable Volterra kernels identification using time-delay neural networks: application to unsteady aerodynamic loading

Abstract

In the last decades, the Volterra series theory has been used to construct reduced-order models of nonlinear systems in engineering and applied sciences. For the particular case of weakly nonlinear aerodynamic and aeroelastic systems, the Volterra series theory has been tested as an alternative to the high computing costs of CFD methods. The Volterra series model determination depends on identifying the kernels associated with the respective convolution integrals. The Volterra kernels identification has been tried in many ways, but the majority of them addresses only the direct kernels of single-input, single-output nonlinear systems. However, multiple-input, multiple-output relations are the most typical case for many dynamic systems. In this case, the so-called Volterra cross-kernels represent the internal couplings between multiple inputs. Not many generalizations of the single-input kernel identification methods to multi-input Volterra kernels are available in the literature. This work proposes a methodology for the identification of Volterra direct kernels and cross-kernels, which is based on time-delay neural networks and the relationship between the kernels functions and the internal parameters of the network. Expressions to derive the pth-order Volterra direct kernels and cross-kernels from the internal parameters of a trained time-delay neural network are derived. The method is checked with a two-degree-of-freedom, two-input, one-output nonlinear system to demonstrate its capabilities. The application to a mildly nonlinear unsteady aerodynamic loading due to pitching and heaving motions of an airfoil is also evaluated. The Volterra direct kernels and cross-kernels of up to third order are successfully identified using training datasets computed with CFD simulations of the Euler equations. Comparisons between CFD simulations and Volterra model predictions are presented, thereby ensuring the potential of the method to systematically extract kernels from neural networks.

1 Introduction

The functional series of Volterra is a powerful representation for nonlinear continuous, causal, time-invariant, fading memory systems [1]. Volterra series comprises an infinite expansion of multi-dimensional convolution integrals of increasing order. This approach has been used in a vast range of engineering and applied sciences problems [2,3,4,5].

Survey papers since Billings [6] to the most recent by Cheng et al. [7] present the progress on Volterra functional-based identification and applications in engineering. The Wiener series [8,9,10] attained modeling accuracy similar to the Volterra-type series and are solved using an expansion into a series of mutually orthonormal polynomial functionals, the so-called G-functionals. Although the Wiener models provide a systematic approach to identification problems, the excessive number of required coefficients, even for lower-order nonlinear systems, undermines the applicability of the method. Moreover, nonlinear systems identification methods were also conceived on block-oriented models [11, 12], which use combinations of linear and nonlinear static subsystems. The LNL cascade systems [13] consist of the combination of a linear subsystem followed by a static nonlinearity, and another linear subsystem, which allows nonlinear identification from Wiener and Hammerstein cascade models [14]. The block-oriented approaches were developed solely for random processes, in particular for white Gaussian inputs, to systematically assess the parameters of the identified models for the associated class of nonlinear systems. The Adomian decomposition method was also used as an alternative to Volterra series [15].

A common drawback of the aforementioned methodologies is the excessive number of terms necessary to identify a nonlinear dynamic behavior. In turn, system identification based on the Volterra theory requires the calculation of the kernel functions. Again, most of the proposed kernel identification methods suffer from a large number of expansion terms [7]. Cheng et al. [7] present a comprehensive review of kernel identification methods in both time and frequency domain, in which investigations mostly concern nonlinear SISO dynamics. Marmarelis and Naka [16] were the first to address the issue of multi-input Volterra model. Since then, a relatively small amount of work in multi-variable Volterra theory is observed. The Volterra theory employed to multi-variable systems requires the calculation of the so-called cross-kernels, which identify the system internal nonlinear couplings [17].

Generalized models from the existing direct-kernel identification of nonlinear SISO systems have been proposed. Among the methods used for cross-kernels assessment, Worden et al. [18] expanded the harmonic probing algorithm, and Chatterjee and Vyas [19] developed harmonic functions to the analysis and identification of Volterra higher-order direct kernels and cross-kernels. For both cases, a large number of terms are involved in the kernels identification.

Alternative ways to overcome the cumbersome development of expansion terms of Volterra direct kernels and cross-kernels have been tried. The application of artificial neural networks theory is an example. Neural networks to identify Volterra kernels were proposed by Wray and Green [20]. They proved that particular network architectures are equivalent to a Volterra series representation of a dynamic system. Moreover, the kernels of any order can be extracted from the network parameters. Following similar developments, recent attempts to extract Volterra kernels from neural networks were attained in the frequency domain [21, 22] and with Taylor series expansions of a conventional feedforward architecture [23]. The systematic way of dealing with a black-box-type model through supervised training schemes and extracting the kernels from the network parameter can be considered an appealing form to assess a Volterra model.

Volterra theory has been successfully used to assess reduced-order models (ROM) in the field of unsteady aerodynamics and aeroelasticity [3]. Silva was the first to present a comprehensive application of Volterra series to unsteady aerodynamic ROM [24, 25]. The approach was also tried by other researchers [26,27,28], where direct kernels were identified using numerical solutions of fluid mechanics equations for mildly nonlinear transonic aerodynamics. Cross-kernels identification of multi-variable unsteady aerodynamics was only tackled more recently [29, 30] with multi-impulse inputs and sparse Volterra series. In de Paula et al. [31], the direct-kernel identification using time-delay neural networks was performed for an aerodynamic system. The neural network was based on the proposed kernel extraction method by Wray and Green [20].

This paper addresses the identification of Volterra direct kernels and cross-kernels from time-delay neural networks by generalizing the Wray and Green [20] approach and presents an application to the unsteady aerodynamic loads prediction. To the authors’ best knowledge, the identification of multi-variable Volterra cross-kernels via artificial neural networks has not been investigated in the literature. The method considers a polynomial expansion of the neuron activation function. The neural network training follows conventional techniques, and the final converged network furnishes the internal parameters used for direct kernels and cross-kernels determination. To illustrate the method, a two-degree-of-freedom, second-order nonlinear system is used, and the counterpart Volterra-type model is analyzed. For the unsteady aerodynamics problem, the compressible flow around an airfoil is computed with an Euler CFD code. The unsteady pitching moment predicted with the Volterra model admits simultaneous pitch and heave motions and relies upon higher-order direct kernels and cross-kernels assessment.

2 Volterra kernels identification

2.1 Volterra series

Volterra series [1] are nonlinear functionals developed as a generalization of the Taylor series expansion for a function. The basic premise of the Volterra functional series approach is that a representation for a continuous, nonlinear, physically realizable (or causal), time-invariant, single-input, single-output (SISO) system is given by an infinite series of multi-dimensional convolution integrals of increasing order,

$$\begin{aligned} y(t)= & {} \sum ^{\infty }_{p=0} \int ^{+\infty }_{-\infty } \cdots \int ^{+\infty }_{-\infty } h_{p} (\tau _1, \tau _2, \ldots , \tau _p ) \nonumber \\&\times \prod ^{p}_{i=1} u(t - \tau _i) \ \mathrm{d} \tau _1 \ \mathrm{d} \tau _2 \ldots \mathrm{d} \tau _p, \end{aligned}$$

(1)

where y(t) is the system response, $u(\cdot )$ is the input to the system, and $h_p(\cdot )$ is the pth-order Volterra kernel.

The zeroth-order Volterra kernel $h_0$ is the zero-input response of the system, whereas the first-order Volterra kernel $h_1$ represents the linear response of the system to a single impulse input. The higher-order kernels are the system multi-dimensional impulse responses, and they lead to measures of the nonlinearity degrees, or the relative influence of a previous input on the current response, that characterizes the temporal effect in the nonlinear dynamics [6, 7]. From the system causality, it results that the kernel $h_{p}(\tau _{1}, \ldots , \tau _{p})$ is zero if any $\tau _{i} < 0$, thereby leading the lower limits of the integrals in Eq. (1) to be equal to zero. The causality hypothesis additionally implies that the system response cannot depend on future input values, making it possible to replace the upper limit of the integrals by t. It is also possible to assure that each kernel $h_{p}(\tau _{1}, \ldots , \tau _{p})$ is symmetric with respect to any permutation of $\tau _{1}, \ldots , \tau _{p}$.

A convenient form to present the Volterra functional series for causal, finite memory, time-invariant, and discrete-time systems is,

$$\begin{aligned} y[n]\simeq & {} h_0 + \sum ^{p}_{j=1} \left\{ \sum ^{n}_{k_1=0} \cdots \sum ^{n}_{k_j=0} h_j [k_1, \ldots , k_j] \right. \nonumber \\&\left. \prod ^{j}_{i=1} u[n-k_i] \right\} , \end{aligned}$$

(2)

where n is the discrete-time variable, and $h_j[k_1, k_2, \ldots , k_j]$ is the jth-order Volterra kernel.

The Volterra theory presented previously concerns only SISO systems, but it can be extended to multiple-input, multiple-output (MIMO) systems. Each lth-output, $y_l[n]$, of a nonlinear system due to q inputs can be modeled using a pth-order, multi-input Volterra series [29]. For a discrete-time representation of the nonlinear system, the multi-input Volterra series is,

$$\begin{aligned} y_l[n]\simeq & {} h_0^{(l)} +\sum _{j=1}^{q}\left\{ \sum _{k=0}^{n}h_1^{(l:j)}[k]u_j[n-k] \right\} + \nonumber \\&+ \sum _{j_1=1}^{q}\sum _{j_2=1}^{q} \left\{ \sum _{k_1=0}^{n}\sum _{k_2=0}^{n} h_2^{(l:j_1j_2)}[k_1,k_2] u_{j_1}\right. \nonumber \\&\left. [n-k_1]u_{j_2}[n-k_2] \phantom {\sum _{k=0}^{n^{\sum }}}\right\} + \cdots \nonumber \\&+ \sum _{j_1=1}^{q} \ldots \sum _{j_p=1}^{q}\left\{ \sum _{k_1=0}^{n} \ldots \sum _{k_p=0}^{n} h_p^{(l:j_1 \ldots j_p)} \right. \nonumber \\&\left. [k_1,\ldots ,k_p] u_{j_1}[n-k_1] \ldots u_{j_2}[n-k_2]\phantom {\sum _{k=0}^{n^{\sum }}}\right\} , \end{aligned}$$

(3)

where n is the discrete-time variable, $u_1[\cdot ], u_2[\cdot ], \ldots , u_q[\cdot ]$ are the system inputs, and $h_p^{(l:j_1 \ldots j_p)}[k_1,\ldots ,k_p]$ is the pth-order, multi-input, discrete-time Volterra kernel. The first superscript of the kernels notation (before the colon) denotes the output of the system and the secondary superscripts (after the colon) identify to which inputs the kernel corresponds. If the secondary superscripts are equal ($j_1 = j_2 = \cdots = j_p$), the kernels are the so-called direct kernels. On the other hand, for combined inputs that differ, i.e., at least one secondary superscript is distinct from the others, the kernels are named the cross-kernels. The cross-kernels differentiate the multi-variable Volterra series from the single-input formulation by accounting for the intermodulation terms that arise when a nonlinear system is excited with more than one input.

The zeroth-order kernels are the steady-state responses of the system and can be subtracted from the total system response. Therefore, $h_0^{(l)}$ can be neglected in Eq. (3) without loss of generality.

2.2 Multi-variable Volterra kernel identification via time-delay neural networks

Wray and Green [20] have introduced a method to identify the direct kernels of a single-input Volterra series from the internal parameters of a trained time-delay neural network. The method offers a general expression to identify the pth-order Volterra kernel. Recently, de Paula et al. [31] have successfully applied the method to aerodynamic systems and identified direct kernels up to the fourth order.

The novelty of the present work is a generalized TDNN approach to systematically identify the multi-variable Volterra direct kernels and cross-kernels. To derive the equations for a system with multiple inputs and multiple outputs, a time-delay neural network (TDNN) architecture is considered, such as the one illustrated in Fig. 1. The TDNN is composed of $(q + \sum _{k=1}^q M_k)$ input units, a hidden layer with N hidden nodes and L output nodes. The input units are grouped into blocks corresponding to each particular input, $u_k$ (for $k = 1, 2, \ldots , q$). Each block contains $M_k + 1$ time-delayed samples of the kth-input signal. The value $M_k$ is a measure of the memory length and different values can be assumed for each particular input. In a dynamic system, each particular input corresponds to a degree of freedom (DOF) that is excited. The connecting links between the ith-input unit and the jth-hidden neuron are associated with a synaptic weight $w_{ij}^{(k)}$, where the superscript index indicates with which particular input the weight is correlated. The numbering of the index i also reflects the block structure in which the input units are grouped. The value of i starts from zero in each new block and goes up to $M_k$ (for $k = 1, 2, \ldots , q$). The connecting links between the jth-hidden unit and the lth-output node are weighted by the amounts $c_{jl}$. Such a TDNN applies to a system with q inputs and L outputs.

Fig. 1

Time-delay neural network architecture with multiple inputs and multiple outputs

The input into each hidden node (a neuron unit) of the network illustrated in Fig. 1 is a weighted sum over the time-delayed input units in all degrees of freedom,

$$\begin{aligned} x_j[n] = \sum _{k=1}^{q} \sum _{i=1}^{M_k} w_{ij}^{(k)}u_k[n-i]. \end{aligned}$$

(4)

The basic operation performed by a hidden neuron unit is to process the weighted sum of the inputs (cf. Eq. (4)) with a nonlinear activation function, $\phi (\cdot )$, as shown in the schematic of Fig. 2. A bias, $b_j$, is also applied to the linear combination of the inputs. A typical sigmoidal activation function given by a hyperbolic tangent function is assumed. The Taylor series expansion of the activation function around the bias gives an equivalent polynomial representation, $p_j$,

$$\begin{aligned} p_j(x_j+b_j)= & {} \sum _{p=0}^{\infty } \frac{\tanh ^{(p)}(b_j)}{p!} (x_j)^p \nonumber \\= & {} \sum _{p=0}^{\infty } a_{pj}(x_j)^p, \end{aligned}$$

(5)

where $\tanh ^{(p)}$ represents the pth derivative of $\tanh$ and $a_{pj}$ are the coefficients of the polynomial expansion. The calculation of the $a_{pj}$ values requires the determination of the derivatives of the $\tanh$ function, which can be obtained analytically or by using a symbolic manipulation software.

Fig. 2

Nonlinear model of a neuron

The output neurons have linear activation functions, as it is usual for function fitting problems. Assuming their biases as zero, each output of the neural network is described by a linear combination of the output of each hidden unit, each hidden unit corresponding to the response in a degree of freedom of the system,

$$\begin{aligned} y_l[n] = \sum _{j=1}^{N} c_{jl} \ p_j(x_j) = \sum _{j=1}^{N}\sum _{p=0}^{\infty } c_{jl} \ a_{pj}(x_j)^p. \end{aligned}$$

(6)

If it is admitted that the polynomial of Eq. (6) has finite order, the output of the neural network can be expanded (using Eq. (4)) as

$$\begin{aligned} y_l[n]= & {} \sum _{j=1}^{N} c_{jl} a_{0j} + \sum _{j=1}^{N} c_{jl} a_{1j} \left( \sum _{k=1}^{q}\sum _{i=0}^{M_k} w_{ij}^{(k)} u_k[n-i]\right) + \nonumber \\&+ \sum _{j=1}^{N} c_{jl} a_{2j} \left( \sum _{k=1}^{q}\sum _{i=0}^{M_k} w_{ij}^{(k)} u_k[n-i] \right) ^2 + \cdots + \nonumber \\&+ \sum _{j=1}^{N} c_{jl} a_{pj} \left( \sum _{k=1}^{q}\sum _{i=0}^{M_k} w_{ij}^{(k)} u_k[n-i]\right) ^p, \end{aligned}$$

(7)

where p is an integer value.

The terms related to the power of a sum in Eq. (7) can be further expanded and rearranged as

$$\begin{aligned} y_l[n]= & {} \sum _{j=1}^{N} c_{jl} a_{0j}+ \sum _{k=1}^{q} \nonumber \\&\left\{ \sum _{i=0}^{M_k} \left( \sum _{j=1}^{N} c_{jl} a_{1j} w_{ij}^{(k)}\right) u_k[n-i] \right\} \nonumber \\&+ \sum _{k_1=1}^{q}\sum _{k_2=1}^{q} \left\{ \sum _{i_1=0}^{M_{k_1}} \sum _{i_2=0}^{M_{k_2}} \right. \nonumber \\&\left( \sum _{j=1}^{N} c_{jl} a_{2j} w_{i_1j}^{(k_1)}w_{i_2j}^{(k_2)}\right) \nonumber \\&\left. u_{k_1}[n-i_1] u_{k_2}[n-i_2] \right\} + \cdots \nonumber \\&+ \sum _{k_1=1}^{q} \ldots \sum _{k_p=1}^{q} \left\{ \sum _{i_1=0}^{M_{k_1}} \ldots \sum _{i_p=0}^{M_{k_p}}\right. \nonumber \\&\left( \sum _{j=1}^{N} c_{jl} a_{pj} w_{i_1j}^{(k_1)} \ldots w_{i_pj}^{(k_p)}\right) \nonumber \\&\left. u_{k_1}[n-i_1] \ldots u_{k_p}[n-i_p] \right\} . \end{aligned}$$

(8)

The form of Eq. (8) is equivalent to the multi-variable Volterra series shown in Eq. (3). The Volterra kernels correspond to the terms shown inside parentheses, that is,

$$\begin{aligned}&h_0^{(l)} = \sum _{j=1}^{N} c_{jl} \ a_{0j} \end{aligned}$$

(9)

$$\begin{aligned}&h_1^{(l:k)}[i] = \sum _{j=1}^{N} c_{jl} \ a_{1j} \ w_{ij}^{(k)} \end{aligned}$$

(10)

$$\begin{aligned}&h_2^{(l:k_1,k_2)}[i_1,i_2] = \sum _{j=1}^{N} c_{jl} \ a_{2j} \ w_{i_1j}^{(k_1)} \ w_{i_2j}^{(k_2)} \end{aligned}$$

(11)

and the general expression for the pth-order kernel is

$$\begin{aligned} h_p^{(l:k_1,\ldots ,k_p)}[i_1,\ldots ,i_p] = \sum _{j=1}^{N} c_{jl} \ a_{pj} \ w_{i_1j}^{(k_1)} \ldots w_{i_pj}^{(k_p)}. \end{aligned}$$

(12)

Note that Eqs. (9)–(12) represent the direct kernels and the cross-kernels. The kernels secondary superscripts, which are related to the combination of inputs, are determined by the block inputs associated with the respective synaptic weights, $w_{ij}^{(k)}$.

The total number of kernels of a specific order depends on the number of input combinations and can be very large for multi-degree-of-freedom systems. However, because of existing symmetries, the number of cross-kernels to be assessed can be significantly reduced [19]. The Volterra kernel formulae based on the time-delay neural network reflect this symmetry as, for the same set of inputs, the kernel will be the same irrespective of the order in which the synaptic weights are multiplied. For example, for a system with two inputs and a single output (MISO-type system), the multi-variable Volterra series of third order can be written as

$$\begin{aligned} y[n]= & {} \sum _{j=1}^{N} c_{j} a_{0j} + \left\{ \sum _{i=0}^{M_1} \left( \sum _{j=1}^{N} c_{j} a_{1j} w_{ij}^{(1)}\right) \right. \nonumber \\&\left. \times u_1[n-i] + \sum _{i=0}^{M_2}\left( \sum _{j=1}^{N} c_{j} a_{1j} w_{ij}^{(2)}\right) u_2[n-i] \right\} \nonumber \\&+ \left\{ \sum _{i_1=0}^{M_{1}} \sum _{i_2=0}^{M_{1}} \left( \sum _{j=1}^{N} c_{j} a_{2j} w_{i_1j}^{(1)}w_{i_2j}^{(1)}\right) \right. \nonumber \\&\left. \times u_{1}[n-i_1] u_{1}[n-i_2] + \sum _{i_1=0}^{M_{2}} \sum _{i_2=0}^{M_{2}} \right. \nonumber \\&\times \left( \sum _{j=1}^{N} c_{j} a_{2j} w_{i_1j}^{(2)}w_{i_2j}^{(2)}\right) u_{2}[n-i_1] u_{2}[n-i_2] \nonumber \\&+ 2 \sum _{i_1=0}^{M_{1}} \sum _{i_2=0}^{M_{2}} \left( \sum _{j=1}^{N} c_{j} a_{2j} w_{i_1j}^{(1)}w_{i_2j}^{(2)}\right) \nonumber \\&\left. \times u_{1}[n-i_1] u_{2}[n-i_2] \right\} \nonumber \\&+ \left\{ \sum _{i_1=0}^{M_{1}} \sum _{i_2=0}^{M_{1}}\sum _{i_3=0}^{M_{1}} \left( \sum _{j=1}^{N} c_{j} a_{3j} w_{i_1j}^{(1)} w_{i_2j}^{(1)} w_{i_3j}^{(1)}\right) \right. \nonumber \\&\times u_{1}[n-i_1] u_{1}[n-i_2] u_{1}[n-i_3]\nonumber \\&+ \sum _{i_1=0}^{M_{2}} \sum _{i_2=0}^{M_{2}}\sum _{i_3=0}^{M_{2}} \left( \sum _{j=1}^{N} c_{j} a_{3j} w_{i_1j}^{(2)} w_{i_2j}^{(2)} w_{i_3j}^{(2)}\right) \nonumber \\&\times u_{2}[n-i_1] u_{2}[n-i_2] u_{2}[n-i_3] \nonumber \\&+ 3 \sum _{i_1=0}^{M_{1}} \sum _{i_2=0}^{M_{1}} \sum _{i_3=0}^{M_{2}} \left( \sum _{j=1}^{N} c_{j} a_{3j} w_{i_1j}^{(1)} w_{i_2j}^{(1)} w_{i_3j}^{(2)}\right) \nonumber \\&\times u_{1}[n-i_1] u_{1}[n-i_2] u_{2}[n-i_3] \nonumber \\&+ 3 \sum _{i_1=0}^{M_{1}} \sum _{i_2=0}^{M_{2}}\sum _{i_3=0}^{M_{2}} \left( \sum _{j=1}^{N} c_{j} a_{3j} w_{i_1j}^{(1)} w_{i_2j}^{(2)} w_{i_3j}^{(2)}\right) \nonumber \\&\times \left. u_{1}[n-i_1] u_{2}[n-i_2] u_{2}[n-i_3] \right\} , \end{aligned}$$

(13)

where the square of a sum and the third power of a sum (appearing in Eq. (7)) can also be expressed as

$$\begin{aligned} \left( \sum _{k=1}^q z_k \right) ^2= & {} \sum _{k=1}^q z_k^2 + 2 \sum _{k_1=1}^q\sum _{k_2=1}^{k_1-1}z_{k_1} z_{k_2} \end{aligned}$$

(14)

$$\begin{aligned} \left( \sum _{k=1}^q z_k \right) ^3= & {} \sum _{k=1}^q z_k^3 + 3 \sum _{k_1=1}^q\sum _{k_2=1}^{k_1-1} (z_{k_1} z_{k_2}^2 + z_{k_1}^2 z_{k_2}) \nonumber \\&+\, 6 \sum _{k_1=1}^q\sum _{k_2=1}^{k_1-1}\sum _{k_3=1}^{k_2-1} z_{k_1} z_{k_2} z_{k_3} \end{aligned}$$

(15)

in which $\sum _{i=0}^{M_k}w_{ij}^{(k)}u_k[n-1]$ is taken as $z_k$ and $k_1, k_2, k_3$ are positive integer values. The coefficients 2 and 3 multiplying the terms relative to the cross-kernels are the result of the existing symmetries [18], namely

$$\begin{aligned}&\left\{ h_2^{(l:j_1j_2)}[k_1,k_2] + h_2^{(l:j_2j_1)}[k_2,k_1] \right\} \nonumber \\&\quad \ \Rightarrow \ 2 h_2^{(l:j_1j_2)}[k_1,k_2] , \end{aligned}$$

(16)

and, similarly

$$\begin{aligned}&\left\{ h_3^{(l:j_1j_1j_2)}[k_1,k_2,k_3]\right. \nonumber \\&\qquad \left. + h_3^{(l:j_1j_2j_1)}[k_1,k_3,k_2] + h_3^{(l:j_2j_1j_1)}[k_3,k_2,k_1]\right\} \ \nonumber \\&\quad \Rightarrow \ 3 h_3^{(l:j_1j_1j_2)}[k_1,k_2,k_3]. \end{aligned}$$

(17)

3 Kernels identification for a MISO nonlinear system

An arbitrary two-degrees-of-freedom, second-order coupled nonlinear system with two inputs and a single output is used to illustrate the features of the kernel identification via a neural network. Quadratic terms of each degree-of-freedom are included to ensure weakly smooth nonlinearity. The MISO system dynamics is represented by the following set of coupled nonlinear ODEs:

$$\begin{aligned}&\ddot{y}_1 + \dot{y}_1 + y_1 + 0.2 y_1 y_2 + 0.2 y^2_1 = u_1 \end{aligned}$$

(18)

$$\begin{aligned}&\ddot{y}_2 + \dot{y}_2 + 0.1 y_1 y_2 + 0.15 y^2_2 = u_2 \end{aligned}$$

(19)

$$\begin{aligned}&y = y_1 + y_2 \end{aligned}$$

(20)

where $y_1$ and $y_2$ are the system internal variables that compose the system output y, and $u_1$ and $u_2$ are independent inputs.

Fig. 3

Training and generalization results compared with the direct simulation of Eqs. (18)–(20)

Equations (18)–(20) are numerically evaluated using a third-order, fixed-step of one millisecond, Runge–Kutta solver [32] to establish a database for the neural network training process. The training inputs $u_1$ and $u_2$ comprised random signals with a Gaussian distribution. The TDNN architecture parameters used for training were taken after a trial-and-error scheme and subsequent inspection of the network predictive performance. The TDNN architecture parameters used for training were defined by inspection of the predictive performance with increasing neural network complexity, i.e., number of time delays and hidden neurons, such as the error analysis presented in de Paula et al. [31]. A feedforward neural network with a total of $M = 50$ time delays per input and $N=3$ hidden neurons has shown a suitable performance. The TDNN was trained with the Bayesian regularization algorithm, and the regularization scheme is performed within the Levenberg–Marquardt algorithm. It constrains the magnitudes of the network parameters in order to force a smoother response, as well as to prevent overfitting and to enhance the generalization properties [33]. The stopping criterion considers the convergence of the mean squared error between the predicted and the numerically simulated outputs.

Figure 3 depicts the comparison between the simulated and predicted signals after completion of the time-delay neural network training and generalization testing (cf. Fig. 3a, b). The training results in Fig. 3a, c demonstrate that the neural network provides a good match to the direct simulation of Eqs. (18)–(20). Figures 3b, d present the network performance in predicting a signal different from those used for the training. Such results ensure that the time-delay neural network satisfactorily represents the MISO two-degree-of-freedom system. The next step concerned the extraction of the Volterra kernels from the neural network internal parameters. The manipulation of Eq. (5), followed by applying Eq. (12), leads to the Volterra direct kernels and cross-kernels.

Fig. 4

First-order kernels for the MISO 2-DOF system

Volterra kernels up to the second order were extracted for this case through the neural network approach. The first-order direct kernels related with each input signal are presented in Fig. 4a and b, respectively. The asymmetry of the system can be observed from the differences between the kernels. Figure 5 presents the second-order Volterra kernels obtained. The identified second-order direct kernels and cross-kernels also indicate the asymmetry.

Fig. 5

Second-order direct kernals and cross-kernels for the MISO 2-DOF system

Fig. 6

Influence of the identified Volterra kernels in predicting the MISO 2-DOF system response

Once the Volterra kernels are identified, they can be used as a reduced-order model to predict the system dynamics. To test the reduced-order model built with the computed kernels, a pair of inputs of sinusoidal form is used. The inputs considered are: $u_1(t) = \cos (t)$ and $u_2(t) = \cos (0.8 t)$. The numerically simulated signal is compared with various combinations of the identified kernels, which were used in the convolutions in Eq. (3). The results are summarized in Fig. 6. Figure 6a shows how the Volterra reduced model behaves for different kernels combinations, which include some or the totality of kernel functions. It is clear the importance of each higher-order kernel to predict the nonlinear behavior of the system. By examining the time histories, it is possible to conclude that minimum prediction is attained with the first-order kernels. Moreover, the intrinsic asymmetry of the MISO 2-DOF system could only be approximated with the inclusion of the second-order kernels. Table 1 presents the calculated $L^2$-norms of the error between the numerical simulation of the MISO 2-DOF system against the Volterra model prediction for each kernel combination, respectively. Lower error values are obtained when more kernel functions are included in the Volterra model. Figure 6b confirms the Volterra model performance enhancement in predicting the MISO 2-DOF system as the second-order direct and cross-kernels are added.

Table 1 Error measurements influenced by the number of kernels in the system response prediction

4 Kernel identification for unsteady nonlinear aerodynamic loading model

Unsteady aerodynamic loads can be assessed using numerical techniques pertaining to the branch of computational fluid dynamics (CFD). However, CFD solutions of practical, complex aerodynamic flow fields are still cumbersome. For that reason, reduced-order models have great appeal for unsteady aerodynamic applications [3]. In this section, the Volterra model with direct kernels and cross-kernels identified via time-delay neural networks is tested for an aerodynamic system. The unsteady aerodynamic loading under the effects of compressible flow around an airfoil is considered. To obtain the signals required for training and testing the Volterra model, an inviscid Euler CFD code [34] was used. The CFD code consists of a finite volume 2D solver based on an explicit second-order time marching scheme. The computational domain was set on the NACA 0012 airfoil with an unstructured triangular O-type mesh comprising 19,886 elements, and extending 20 chords from the profile in all directions.

The unsteady pitching moment coefficient response, $C_{m}(\tau )$, resulting from the airfoil pitching, $\alpha (\tau )$, and heaving, $h(\tau )$, motions is simulated with the CFD code, where $\tau =(tU_\infty )/c$ is the dimensionless time. The variables denote the instantaneous time, t, the free-stream speed, $U_{\infty }$, and the airfoil chord length, c. Computational grid, domain extension, and time step were adjusted following the solutions presented by Camilo et al. [34]. The neural network training and generalization test signals consist of random-phase multi-sine pitching and heaving motions function, namely

$$\begin{aligned} \alpha (t) = \ \text{ or } \ h(t) = \sum _{k=1}^{F}A_k \cos (2 \pi f_0 k t + \phi _k), \end{aligned}$$

(21)

where $A_k$ is the randomly distributed motion amplitude (for pitching or heaving motions, respectively), k is the reduced frequency¹, $\phi _k$ is the randomly distributed phase shift in the interval $[0,2\pi )$, F is the index related to the value of the maximum excitation frequency, and $f_0=1/(N_s \Delta t)$ is the frequency band obtained from the number of time samples of the signal, $N_s$, and the time step, $\Delta t$.

The simulations based on the inputs given by Eq. (21) were carried out with a dimensionless integration time step of $\Delta \tau = 6\times 10^{-4}$. A mildly nonlinear loading condition at Mach number 0.6 was chosen for evaluation of the unsteady pitching moment coefficient response. At that compressible regime, weak effects of shock excursion are observed [34] and there is a reasonable amount of nonlinear behavior for testing the Volterra model. Such flow regime has also been used in other work on aerodynamic modeling via Volterra series [25, 29].

Similarly to the approach in the previous example, the neural network architecture was chosen with $M = 25$ time-delayed inputs per input and $N=5$ hidden neurons. The training set consisted of a total of 1,997 points sampled from the CFD full simulation signal at a period of $\Delta \tau = 6\times 10^{-4}$. The training process was performed using the same algorithm as for the previous example given in Sect. 3, namely the Bayesian regularization. After training, a generalization test was carried out and the results of the comparison of the neural network prediction against the CFD simulation are depicted in Fig. 7. The results demonstrate that the neural network prediction features ensure an adequate fitting of the CFD simulation signal, thereby allowing the kernels identification.

Fig. 7

Training results compared with the CFD simulation

Despite the weak nonlinear behavior of the aerodynamic loading, it is expected a more complex dynamics as that of the previous study (cf. Sect. 3). For this reason, the highest kernel order examined for the aerodynamic loading prediction is 3rd-order. Indeed, more complex first-order direct kernels can be observed in Fig. 8a and b for the pitch and plunge degrees of freedom, respectively. Figure 9 depicts the second-order direct kernels and cross-kernels, which also have complex shapes and contribute to modeling asymmetries in the response. As for the third-order kernels, a major difficulty lies on graphically representing them due to the larger-than-three dimensionality. Previous studies [29, 35] with kernel synthesis based on multi-impulse inputs have also revealed non-smooth aerodynamic kernel functions. In de Paula et al. [31], the equivalence between the neural network and the multi-impulse-kernels was established in the frequency domain. The analysis also pointed out that the kernels extracted from the TDNN might be inflicted with high-frequency numerical noise, which impacted the shape of the kernel functions in the time-domain, but not the convolved ROM response.

Fig. 8

First-order kernels for the aerodynamic loading prediction with Volterra series

Fig. 9

Second-order direct kernels and cross-kernels for the aerodynamic loading prediction with Volterra series

To test the ability of the identified set of kernels to assess a Volterra model for the unsteady aerodynamic loading prediction, a simulated test case based on the AGARD experimental data is considered. The conditions of the CT2 case [36] were considered by Balajewicz et al. [29] to establish the following prescribed simultaneous oscillation in pitching and heaving, respectively: $\alpha (\tau ) = 3.16^\circ + 1^\circ \sin (2k_\alpha \tau )$ and $h(\tau ) = 0.14c \sin (2k_h \tau )$, where $k_\alpha = 0.08$, $k_h \approx 0.8k_\alpha$, and $M_\infty = 0.6$.

Fig. 10

Pitching moment response comparison between CDF simulation and Volterra series with the direct kernels only

Fig. 11

Pitching moment response comparison between CDF simulation and Volterra series with the direct kernels and cross-kernels

Figures 10 and 11 can be used to observe the influence of the kernel functions on the $C_{m}(\tau )$ prediction in comparison with the CFD simulation. Figure 10 shows the aerodynamic loading approximation including only the direct kernels. It is reasonable to infer that a Volterra series with kernels of first to third order (referenced as third-order kernels in Fig. 10) can more accurately represent the system behavior. The results confirm such feature, but also highlight the need for more kernel functions to reduced local discrepancies within the $C_{m}(\tau )$ time history. Moreover, it should be observed that the model including only the direct kernels yields a response equivalent to the superposition of the individuals responses in each degree of freedom.

The incorporation of cross-kernel functions in the Volterra model is essential to represent the system internal nonlinear couplings when simultaneous inputs are applied. Figure 11 depicts the effect of the cross-kernels inclusion in the Volterra model. Admitting all direct kernels already as part of the Volterra model, the results in Fig. 11 ensure the importance of additionally considering the cross-kernels, particularly the third-order one. A multi-input Volterra series with direct kernels and cross-kernels up to the third order leads to a satisfactory predicting model of the unsteady pitching moment. To summarize the results in Figs. 10 and 11, error measures based on $L^2$-norm are presented in Table 2 and confirm the importance of the cross-kernels for the Volterra series to accurately predict the aerodynamic loading under the effects of internal couplings.

Figure 12 shows the power spectra of the respective signals of Figs. 10 and 11. It is observed that the first-order kernels capture the fundamental frequencies, that is, $k/{k_\alpha } = 0.8$ (regarding $k/{k_h} = 1$) and $k/{k_\alpha } = 1$. The 2nd- and 3rd-order direct kernels capture the $2\times$ and $3\times$ harmonics of the fundamental frequencies, which are observed at $k/{k_\alpha } = 1.6$ and $k/k_{\alpha } = 2$, and at $k/{k_\alpha } = 2.4$ and $k/k_{\alpha } = 3$. Moreover, the second- and third-order components lead, respectively, to a nonlinear contribution to the static response ($k/{k_\alpha } = 0$) and to contributions in the fundamental frequencies.

Table 2 Error measurements influenced by the number of kernels in the aerodynamic loading prediction

Fig. 12

Power spectra comparison between the Volterra series with different combinations of direct kernels and cross-kernels and for the CFD simulation

The second-order cross-kernels bring the first intermodulation terms, namely $k/{k_\alpha } = 0.2$ and $k/{k_\alpha } = 1.8$ regarding the combinations $(k_\alpha + k_h)/{k_\alpha }$ and $(k_\alpha - k_h)/{k_\alpha }$. The third-order cross-kernels are responsible for terms as $(2k_h - k_\alpha )/{k_\alpha }$ (resulting in $k/{k_\alpha } = 0.6$), $(2k_\alpha - k_h){k_\alpha }$ (resulting in $k/{k_\alpha } = 1.2$), $(2k_h + k_\alpha )/{k_\alpha }$ (resulting in $k/{k_\alpha } = 2.6$), and $(2k_\alpha + k_h)/{k_\alpha }$ (resulting in $k/{k_\alpha } = 2.8$). It is possible to observe that the frequency combinations in the terms of intermodulation are related to the set of inputs associated with the cross-kernels. For instance, in the case of second-order cross-kernels, there is only one occurrence of each of the inputs, so the intermodulation frequencies correspond to the combination of the fundamental frequencies. In the case of third-order cross-kernels, there are two occurrences of a particular input and only one of the second input. Therefore, the intermodulation frequencies associated with third-order cross-kernels are always determined by the combinations between $2\times$ harmonics and the remaining fundamental frequency.

5 Concluding remarks

This work presented an application of time-delay neural networks for the identification of Volterra direct kernels and cross-kernels. The applicability of the identified kernels to the construction of reduced-order models for aerodynamic loading prediction was tested for the case of a mildly nonlinear unsteady aerodynamic regime. In the proposed method, kernel synthesis is based on the internal parameters of a TDNN after regular supervised training is completed. The extracted direct kernels and cross-kernels are subsequently used in a discrete-time multi-input Volterra series model.

The methodology has been tested with a MISO 2-DOF system to check its potential. Kernels up to second order were identified and applied to a Volterra model to predict the system response to new inputs. Results demonstrated the impact of including the cross-kernel functions in the Volterra series for the prediction of internal system couplings. The direct kernels alone were not capable of accurate approximations of the simulated system response.

The same kernel identification procedure was applied to the prediction of the unsteady pitching moment of an airfoil under compressible flow at Mach number 0.6. The database used to train the neural network was obtained through CFD simulations. In this case, direct kernels and cross-kernels up to third order were necessary to attain an accurate approximation of the aerodynamic system. Moreover, the second- and third-order cross-kernels were essential to represent the features associated with the internal nonlinear couplings resulting from the simultaneous excitation in the pitch and plunge degrees of freedom. Frequency domain analyses of the predicted responses also revealed the importance of the cross-kernels in capturing the intermodulation harmonics of the fundamental frequencies.