Optimal LQG control for discrete time-varying system with multiplicative noise and multiple state delays

Abstract

This paper is concerned with the optimal linear quadratic Gaussian (LQG) control problem for discrete time-varying system with multiplicative noise and multiple state delays. The main contributions are twofolds. First, in virtue of Pontryagin’s maximum principle, we solve the forward and backward stochastic difference equations (FBSDEs) and show the relationship between the state and the costate. Second, based on the solution to the FBSDEs and the coupled difference Riccati equations, the necessary and sufficient condition for the optimal problem is obtained. Meanwhile, an explicit analytical expression is given for the optimal LQG controller. Numerical examples are shown to illustrate the effectiveness of the proposed algorithm.

1 Introduction

The control problem for time-delay systems have received extensively attention since 1950s because of its wide applications in networked control system, intelligent cruise control system, finance, cable-driven manipulators and so on [1,2,3,4,5,6]. There have been a lot of researches on the optimal control and stabilization problem of time-delay systems in recent years, and many results have been surveyed, which concern with single input/state delay or multiple input/state delays [7,8,9,10]. For example, Yue et al. [7] proposed a Lyapunov–Krasovskii functional approach to design the delayed feedback controller of uncertain systems with time-varying input delay, by introducing some relaxation matrices and turning parameters. Lee et al. [9] studied the robust $\mathrm{H}_{\infty }$ control problem for uncertain linear systems with a state-delay. Based on the obtained delay-dependent bounded real lemma, the delay-dependent condition for the existence of a robust controller was presented. Due to the existence of multiple delays, the optimal controller is related to the past variables, which makes the control problem more challenging.

On the other hand, stochastic uncertainties exist in many control processes, and some results have been shown in [11,12,13,14]. Qi et al. [12] presented the optimal estimation and the optimal output feedback controller of the discrete-time multiplicative noise system with intermittent observations by virtue of coupled Riccati equations. The stabilization condition for this system was developed in the mean square sense. Rami et al. [13] considered the discrete-time stochastic LQ problem subject to state and control-dependent noises. A necessary and sufficient condition for the existence of the optimal control was identified in terms of the solution to the proposed difference Riccati equation. As to meet the actual demand in different areas, the control systems with both stochastic uncertainties and time delay(s) have been thoroughly studied [15,16,17,18,19]. Zhang et al. [15] obtained the optimal linear quadratic regulation (LQR) controller for discrete-time system with input delay and multiplicative noise via the Riccati-ZXL difference equation, while the additive noise was not considered in this reference. In [16], Liang et al. took the state- and control-dependent noise, additive noise and input delay into account, and the optimal controller and the suboptimal linear state estimate feedback controller for the linear quadratic Gaussian (LQG) system were both derived, with only single time delay in the input. Besides, when there are multiplicative noise and multiple delays in the input, Li et al. [19] presented the optimal controller and the optimal cost under the necessary and sufficient condition. However, additive noise was not considered. It is obviously shown that the system models described in the above literatures are all discrete time-invariant and input-delay(s) systems. Moreover, the optimal problem involving simultaneously multiplicative noise, additive noise and multiple state delays are not mentioned. In addition, when the additive noise is related to the multiplicative noise, the analysis and synthesis for the control problem remain challenging.

Different from the existing research systems, the system considered in this paper contains simultaneously multiplicative noise, multiple state delays and additive noise, which is more complex than before. It should be emphasized that the additive noise and the multiplicative noise are dependent, and the coefficients in this paper are time-varying. The LQG control for our paper is much more sophisticated and unsolved. The main contributions of this paper are as follows: (1) The relations between the state and the costate in terms of the discrete time-varying LQG problem is given by lots of inductive calculations, which is also the solution to the forward and backward stochastic difference equations (FBSDEs). (2) If and only if a sequence of matrices are all positive definite, the optimal controller and the associated cost function will be obtained via the coupled difference Riccati equations, and the explicit expression of the unique controller is presented, which is obviously more complicated than LQR controller in [19]. Our approach is based on the stochastic maximum principle, and the key technique is the solution to the FBSDEs.

The rest of this paper is organized as follows. In Sect. 2, the discrete time-varying stochastic LQG control problem is described. In Sect. 3, the key tool to the solution is presented, and the necessary and sufficient condition for the optimal LQG control problem is shown. The solutions to the general LQG problem are derived. Numerical examples are shown in Sect. 4. Conclusions are provided in Sect. 5. Proofs of the Lemma and the Theorem are described in Appendixes.

Notation ${\mathbb {R}}^n$ denotes the n-dimensional real Euclidean space. I presents the unit matrix of appropriate dimension. The superscript  $'$ denotes the transpose of the matrix. $\{\varOmega , {\mathcal {F}},$ ${\mathcal {P}},\{{\mathcal {F}}_k\}_{k\geqslant 0}\}$ denotes a complete probability space on which random variable $\nu _k$ and $\mu _k$ are defined such that $\{{\mathcal {F}}_k\}_{k\geqslant 0}$ is the natural filtration generated by $\nu _k$ and $\mu _k$, i.e., ${\mathcal {F}}_k=\sigma \{\nu _0,\ldots ,\nu _k,\mu _0,\ldots ,$ $\mu _k\}$, augmented by all the ${\mathcal {P}}$-null sets in ${\mathcal {F}}$. A symmetric $A>0\,(\geqslant 0)$ means that it is a positive definite (positive semi-definite) matrix. $\theta _{a,\,b}$ is the usual Kronecker function, i.e., $\theta _{a,\,b}=0$ if $a\ne b$, and $\theta _{a,\,b}=1$ if $a=b$. $\mathrm{Tr}(P)$ represents the trace of matrix P.

2 Problem formulation

Consider the discrete time-varying stochastic LQG system with state delays and multiplicative scalar noise:

$$\begin{aligned} x_{k+1}= \sum \limits _{i=0}^d\big [(\mathcal {C}_i(k) + \nu _k\bar{\mathcal {C}}_i(k))x_{k-i}\big ] +(\mathcal {D}(k) + \nu _k\bar{\mathcal {D}}(k))u_k+\mu _k, \end{aligned}$$

(1)

where $x_k\in {\mathbb {R}}^n$ is the state, $u_k\in {\mathbb {R}}^m$ is the input control, the positive integer d is the state delay, $\nu _k$ is the scalar noise with zero mean and variance $\gamma$, $\mu _k\in {\mathbb {R}}^n$ is random variable satisfying $\mathrm{E}[\mu _k|{\mathcal {F}}_{k-1}]={\bar{\mu }}_k$ and $\mathrm{E}[\mu _k\mu _k'|{\mathcal {F}}_{k-1}]=Q_{\mu _k}$. The coefficient matrices with compatible dimensions $\mathcal {C}_i(k),\bar{\mathcal {C}}_i(k),\mathcal {D}(k)$ and $\bar{\mathcal {D}}(k)$ with $i=0,\ldots ,d$ are time-varying. $\nu _k$ and $\mu _k$ are correlated, satisfying $\mathrm{E}[\nu _k\mu _k'|{\mathcal {F}}_{k-1}]=\tau$, $\mathrm{E}[\nu _k\mu _l'|{\mathcal {F}}_{k-1}]=0,$ $k\ne l$. The initial states $x_i$ for $i=-d,\ldots ,0$ are deterministic and known.

Consider the associated cost function for system (1):

$$\begin{aligned} J_N = \mathrm{E} \left\{ \sum \limits _{k=0}^N \left[ x_k'Q_kx_k + u_k'R_ku_k\right] + x_{N+1}'{\mathcal {P}}_{N+1}^0x_{N+1} \right\} , \end{aligned}$$

(2)

where $Q_k$, $R_k$ and ${\mathcal {P}}_{N+1}^0$ are positive semi-definite matrices with appropriate dimensions, and N is the horizon length. In view of the fact that $x_k$ depends on $\nu _{k-1}, \nu _{k-2}$, $\ldots , \mu _{k-1}, \mu _{k-2}, \ldots$, and the controller obeys the causality constraint, $u_k$ must be $\mathcal {F}_{k-1}$-measurable (see [13]). Then, the problem to be addressed is stated as follows.

Problem 1

Find the unique ${\mathcal {F}}_{k-1}$-measurable state feedback controller $u_k$, $k=0,\ldots ,N$, for system (1) such that the cost function (2) is minimized.

3 Main results

For simplicity, we make the system (1) to be

$$\begin{aligned} x_{k+1}=\sum \limits _{i=0}^d\mathcal {C}_k^i(k)x_{k-i}+\mathcal {D}_k(k)u_k+\mu _k, \end{aligned}$$

(3)

where

$$\begin{aligned} \mathcal {C}_k^i(k)&=\mathcal {C}_i(k)+\nu _k\bar{\mathcal {C}}_i(k), \quad i=0,\ldots ,d,\\ \mathcal {D}_k(k)&=\mathcal {D}(k)+\nu _k\bar{\mathcal {D}}(k). \end{aligned}$$

Following the similar approach in [19], we apply stochastic Pontryagin’s maximum principle [20] to system (3) with the cost function (2) to yield the costate equations:

$$\begin{aligned} \zeta _N&=\mathcal {P}_{N+1}^0x_{N+1}, \end{aligned}$$

(4)

$$\begin{aligned} \zeta _{k-1}&=\mathrm{E}\big [\sum \limits _{m=0}^d(\mathcal {C}_{k+m}^m)'(k+m)\zeta _{k+m}|{\mathcal {F}}_{k-1}\big ]+Q_kx_k, \end{aligned}$$

(5)

$$\begin{aligned} 0&=\mathrm{E}\big [\mathcal {D}_k'(k)\zeta _k|{\mathcal {F}}_{k-1}\big ]+R_ku_k,~~ k=0,\ldots ,N, \end{aligned}$$

(6)

where $\zeta _k$ is the costate variable with $\zeta _k=0$ for $k>N$.

For further study, we define the following Riccati coupled equations and make the backwards recursion for $k=N, N-1, \ldots , 0$:

$$\begin{aligned} {\mathcal {P}}_k^j&=\sum \limits _{i=0}^{d-j}\bigg [\mathcal {C}_i'(i+ k){\mathcal {P}}_{i+k+1}^0\mathcal {C}_{i+j}(i + k)\nonumber \\&\quad +\gamma \bar{\mathcal {C}}_i'(i + k){\mathcal {P}}_{i+k+1}^0\times \bar{\mathcal {C}}_{i+j}(i + k) \nonumber \\ \nonumber&\quad + \mathcal {C}_i'(i + k){\mathcal {P}}_{i+k+1}^{j+i+1}+({\mathcal {P}}_{i+k+1}^{i+1})' \mathcal {C}_{i+j}(i + k) \nonumber \\&\quad -(N_{i+k}^i)'\varOmega _{i+k}^{-1}N_{i+k}^{j+i}\bigg ]+ \theta _{j,0}Q_k, \end{aligned}$$

(7)

$$\begin{aligned} \varOmega _k&=R_k+\mathcal {D}'(k){\mathcal {P}}_{k+1}^0\mathcal {D}(k)+\gamma \bar{\mathcal {D}}'(k){\mathcal {P}}_{k+1}^0\bar{\mathcal {D}}(k), \end{aligned}$$

(8)

$$\begin{aligned} N_k^j&=\mathcal {D}'(k){\mathcal {P}}_{k+1}^{j+1}+\mathcal {D}'(k){\mathcal {P}}_{k+1}^0\mathcal {C}_j(k)+\gamma \bar{\mathcal {D}}'(k){\mathcal {P}}_{k+1}^0\bar{\mathcal {C}}_j(k). \end{aligned}$$

(9)

The terminal values of the above matrix sequences ${\mathcal {P}}_k^j$, $\varOmega _k$ and $N_k^j$, $j=0,\ldots ,d$ are given by

$$\begin{aligned} \left\{ \begin{aligned}&{\mathcal {P}}_{N+1}^j=0, ~~ j=1,\ldots ,d+1,\\&{\mathcal {P}}_{N+i}^j=0, ~~ i=2,\ldots ,d+1, \ j=0,\ldots ,d+1,\\&\varOmega _{N+i}=I, ~~N_{N+i}^j=0, ~~ i=1,\ldots ,d, \ j=0,\ldots ,d. \end{aligned}\right. \end{aligned}$$

(10)

It should be emphasized that the recursion will stop unless assuming that $\varOmega _k$ is invertible. To give the main results of Problem 1, we need to obtain the solution to the FBSDEs (3) and (4)–(6), and then the following lemma is proposed.

Lemma 1

Assuming that $\varOmega _k$ are positive definite, i.e., $\varOmega _k>0$, for $k=0,\ldots ,N$, then the following equation

$$\begin{aligned} \zeta _{k-1}=\sum \limits _{j=0}^d{\mathcal {P}}_k^jx_{k-j}+\varPhi _k \end{aligned}$$

(11)

is the solution to FBSDEs (3) and (4)–(6), where

$$\begin{aligned} \varPhi _k&=\sum \limits _{i=0}^d\Bigg [\big (\mathcal {C}_i'(k+i)-(N_{k+i}^i)'\varOmega _{k+i}^{-1}\mathcal {D}'(k+i)\big )\nonumber \\&\quad \times (\varPhi _{k+i+1}+{\mathcal {P}}_{k+i+1}^0{\bar{\mu }}_{k+i})+\big (\bar{\mathcal {C}}_i'(k+i)\nonumber \\&\quad -(N_{k+i}^i)'\varOmega _{k+i}^{-1}\bar{\mathcal {D}}'(k+i)\big ){\mathcal {P}}_{k+i+1}^0\tau \nonumber \\&\quad +({\mathcal {P}}_{k+i+1}^{i+1})'{\bar{\mu }}_{k+i}\Bigg ] \end{aligned}$$

(12)

with the terminal value $\varPhi _{N+1}=0$, and ${\mathcal {P}}_k^j$, $N_k^j$, $\varOmega _k$ satisfy the coupled equations (7)–(10).

Proof

The proof of Lemma 1 is in Appendix A.$\square$

Remark 1

It is noted that the system model in this paper is discrete time-varying, and contains simultaneously multiplicative noise, additive noise and multiple state delays. Meanwhile, the multiplicative noise is related with the additive noise. Thus, the problem of optimal LQG control is particularly difficult.

Remark 2

We have defined ${\mathcal {P}}_k^j$, $N_k^j$ with $j\in [0,d]$ by the equation (10). As using the notations ${\mathcal {P}}_k^j$, $N_k^j$ for $j>d$, we extend the definition ${\mathcal {P}}_k^j=0$, $N_k^j=0$ for $j>d$. Besides, the coefficient matrices $\mathcal {C}_j(k)$, $\bar{\mathcal {C}}_j(k)$ are set to be 0 for $j>d$.

Now, we are in the position to present the solution to Problem 1. The results are stated in the following theorem.

Theorem 1

Problem 1has a unique ${\mathcal {F}}_{k-1}$-measurable $u_k$ if and only if $\varOmega _k$, for $k=0,\ldots ,N$, are positive definite. In this context, the optimal controller $u_k$ is calculated by

$$\begin{aligned} u_k=-\varOmega _k^{-1}\sum \limits _{j=0}^dN_k^jx_{k-j}-\varOmega _k^{-1}\varSigma _k. \end{aligned}$$

(13)

The minimum performance index is as

$$\begin{aligned} J_N^*&=x_0'{\mathcal {P}}_0^0x_0+2x_0'\sum \limits _{j=1}^d{\mathcal {P}}_0^jx_{-j}+\sum \limits _{j=1}^d\sum \limits _{i=1}^d \sum \limits _{l=0}^{d-1}x_{-j}'\Big [\mathcal {C}_{j+l}'(l)\nonumber \\&\quad \times {\mathcal {P}}_{l+1}^0\mathcal {C}_{i+l}(l)+\gamma \bar{\mathcal {C}}_{j+l}'(l){\mathcal {P}}_{l+1}^0\bar{\mathcal {C}}_{i+l}(l)+({\mathcal {P}}_{l+1}^{j+l+1})'\nonumber \\&\quad \times \mathcal {C}_{i+l}(l)+\mathcal {C}_{j+l}'(l){\mathcal {P}}_{l+1}^{i+l+1}-(N_l^{j+l})'\varOmega _l^{-1}N_l^{i+l}\Big ]x_{-i}\nonumber \\&\quad+2x_0'\varPhi _0-\sum \limits _{k=0}^N\varSigma _k'\varOmega _k^{-1}\varSigma _k+2\sum \limits _{k=0}^N{\bar{\mu }}_k'\varPhi _{k+1}\nonumber \\&\quad +\sum \limits _{k=0}^N{\rm Tr}[{\mathcal {P}}_{k+1}^0Q_{\mu _k}], \end{aligned}$$

(14)

where

$$\begin{aligned} \varSigma _k=\mathcal {D}'(k)(\varPhi _{k+1}+{\mathcal {P}}_{k+1}^0{\bar{\mu }}_k)+\bar{\mathcal {D}}'(k){\mathcal {P}}_{k+1}^0\tau , \end{aligned}$$

(15)

while ${\mathcal {P}}_k^j$, $N_k^j$, $\varOmega _k$ satisfy the coupled equations (7)–(10).

Proof

The proof of Theorem 1 is in Appendix B.$\square$

Remark 3

Different from the existing work [19], the difficulties caused by the additive noise are mainly as follows. First, this paper considers the optimal LQG control problem with both multiplicative noise and correlated additive noise, which is more challenging than [19]. Second, due to the existence of the additive noise, the key technique to this optimal control problem, i.e., the solution to the FBSDEs (11) is quite more difficult than that of [19]. Besides, the optimal controller $u_k$ satisfying (13) and the associated optimal cost (14) are more difficult to obtain, and the expression of $u_k$ and $J^*_N$ are more complicated than [19].

Remark 4

For a stochastic discrete-time system with no state delays, i.e., $d=0$ in system (3), it is obviously obtained that the coupled Riccati difference equations:

$$\begin{aligned} {\mathcal {P}}_k^0&=\mathcal {C}_0'(k){\mathcal {P}}_{k+1}^0\mathcal {C}_{0}(k)+\gamma \bar{\mathcal {C}}_0'(k){\mathcal {P}}_{k+1}^0\bar{\mathcal {C}}_{0}(k)\\&\quad -(N_{k}^0)'\varOmega _{k}^{-1}N_{k}^{0}+Q_k, \end{aligned}$$

where

$$\begin{aligned} \varOmega _k&=R_k+\mathcal {D}'(k){\mathcal {P}}_{k+1}^0\mathcal {D}(k)+\gamma \bar{\mathcal {D}}'(k){\mathcal {P}}_{k+1}^0\bar{\mathcal {D}}(k),\\ N_k^0&=\mathcal {D}'(k){\mathcal {P}}_{k+1}^0\mathcal {C}_0(k)+\gamma \bar{\mathcal {D}}'(k){\mathcal {P}}_{k+1}^0\bar{\mathcal {C}}_0(k). \end{aligned}$$

The optimal controller reduces to

$$\begin{aligned} u_k=-\varOmega _k^{-1}N_k^0x_{k}-\varOmega _k^{-1}\varSigma _k, \end{aligned}$$

where $\varSigma _k$ as (15). In this context, the solution to the FBSDEs is as

$$\begin{aligned} \zeta _{k-1}={\mathcal {P}}_k^0x_{k}+\varPhi _k \end{aligned}$$

with

$$\begin{aligned} \varPhi _k&=\big (\mathcal {C}_0'(k)-(N_{k}^0)'\varOmega _{k}^{-1}\mathcal {D}'(k)\big )(\varPhi _{k+1}+{\mathcal {P}}_{k+1}^0{\bar{\mu }}_{k})\\&\quad +\big (\bar{\mathcal {C}}_0'(k)-(N_{k}^0)'\varOmega _{k}^{-1}\bar{\mathcal {D}}'(k)\big ){\mathcal {P}}_{k+1}^0\rho +({\mathcal {P}}_{k+1}^{1})'{\bar{\mu }}_{k}. \end{aligned}$$

Remark 5

From Theorem 1, when the disturbance term $\mu _k$ and the multiplicative noise $\nu _k$ are independent, i.e., $\tau =0$, and when the additive noise is Gaussian white noise, i.e., ${\bar{\mu }}_k=0$, the Riccati difference equations are as (7)–(10), and the matrices can be rewritten as

$$\begin{aligned} \varPhi _k&=\sum \limits _{i=0}^d\big (\mathcal {C}_i'(k+i)-(N_{k+i}^i)'\varOmega _{k+i}^{-1}\mathcal {D}'(k+i)\big )\varPhi _{k+i+1}, \end{aligned}$$

(16)

$$\begin{aligned} \varSigma _k&=\mathcal {D}'(k)\varPhi _{k+1}. \end{aligned}$$

(17)

As the terminal value $\varPhi _{N+1}=0$, it is obviously obtained that $\varSigma _k$ and $\varPhi _k$ in (16), (17) always equal to be zero for $k=0,\dots ,N$. Then, the optimal controller reduces to

$$\begin{aligned} u_k=-\varOmega _k^{-1}\sum \limits _{j=0}^dN_k^jx_{k-j}, \end{aligned}$$

and the solution to the FBSDEs is as

$$\begin{aligned} \zeta _{k-1}=\sum \limits _{j=0}^d{\mathcal {P}}_k^jx_{k-j}. \end{aligned}$$

In this case, the associated optimal cost is given by

$$\begin{aligned} J_N^* =&x_0'{\mathcal {P}}_0^0x_0+2x_0'\sum \limits _{j=1}^d{\mathcal {P}}_0^jx_{-j}+\sum \limits _{j=1}^d\sum \limits _{i=1}^d\sum \limits _{l=0}^{d-1}x_{-j}'\big [\mathcal {C}_{j+l}'(l)\\&\times {\mathcal {P}}_{l+1}^0\mathcal {C}_{i+l}(l)+\gamma \bar{\mathcal {C}}_{j+l}'(l){\mathcal {P}}_{l+1}^0\bar{\mathcal {C}}_{i+l}(l)+({\mathcal {P}}_{l+1}^{j+l+1})'\\&\times \mathcal {C}_{i+l}(l)+\mathcal {C}_{j+l}'(l){\mathcal {P}}_{l+1}^{i+l+1}-(N_l^{j+l})'\varOmega _l^{-1}N_l^{i+l}\big ]x_{-i}\\&+\sum \limits _{k=0}^N{\text{Tr}}[{\mathcal {P}}_{k+1}^0Q_{\mu _k}]. \end{aligned}$$

In view of obtaining the scalar case of optimal LQG control system (3), we derive the results to the general system with multiple delays and multiplicative noise.

Consider the following case of discrete time-varying system:

$$\begin{aligned} x_{k+1}&=\sum \limits _{i=0}^d\Big[ \mathcal {C}_i(k)+\sum \limits _{m=1}^f\nu _k(m)\bar{\mathcal {C}}_{i,m}(k)\Big] x_{k-i}\nonumber \\&\quad +\Big[ \mathcal {D}(k)+\sum \limits _{m=1}^f\nu _k(m)\bar{\mathcal {D}}_{m}(k)\Big] u_{k}+\mu _k, \end{aligned}$$

(18)

where $\mathcal {V}_k=(\nu _k(1) \ldots \nu _k(f))'$ is a f-dimensional white noise defined on a complete probability $\{\varOmega , {\mathcal {P}}, {\mathcal {F}}\}$. $\mathcal {V}_k$ satisfies the variance $\gamma$, i.e.,

$$\begin{aligned} \mathrm{E}[\mathcal {V}_k\mathcal {V}_k']=\gamma =\begin{bmatrix}\gamma _{11} \cdots \gamma _{1f}\\ \vdots \qquad \vdots \\ \gamma _{f1} \cdots \gamma _{ff}\end{bmatrix}\in {\mathbb {R}}^{f\times f},~~\gamma \geqslant 0. \end{aligned}$$

Here ${\mathcal {F}}_k$ is the natural filtration generated by $\mathcal {V}_k$ and $\mu _k$, i.e., ${\mathcal {F}}_k$ is the $\sigma$-algebra generated by $\{\mathcal {V}_0,\ldots ,\mathcal {V}_k,\mu _0,$ $\ldots ,\mu _k\}$. Then, the general case of discrete time-varying LQG control problem is stated as follows.

Problem 2

Find the unique $\mathcal {F}_{k-1}$-measurable state feedback controller $u_k$, $k=0,\dots ,N$, for system (18) such that the cost function (2) is minimized.

To solve Problem 2, we derive the definition as

$$\begin{aligned}&\mathcal {\mathcal {C}}_k^i(k)=\mathcal {C}_i(k)+\sum \limits _{m=1}^f\nu _k(m)\bar{\mathcal {C}}_{i,m}(k),\\&\mathcal {\mathcal {D}}_k(k)=\mathcal {D}(k)+\sum \limits _{m=1}^f\nu _k(m)\bar{\mathcal {D}}_{m}(k), \end{aligned}$$

and the coupled difference Riccati equations (7)–(9) extend to

$$\begin{aligned} {\mathcal {P}}_k^j&=\sum \limits _{i=0}^{d-j}\bigg [\mathcal {C}_i'(i+k){\mathcal {P}}_{i+k+1}^0\mathcal {C}_{i+j}(i+k)+\sum \limits _{a=1}^f\sum \limits _{b=1}^f\gamma _{ab}\nonumber \\&\quad \times \bar{\mathcal {C}}_{i,a}'(i+k){\mathcal {P}}_{i+k+1}^0\bar{\mathcal {C}}_{i+j,b}(i+k)+\mathcal {C}_i'(i+k)\nonumber \\&\quad \times {\mathcal {P}}_{i+k+1}^{j+i+1}+({\mathcal {P}}_{i+k+1}^{i+1})'\mathcal {C}_{i+j}(i+k)-(N_{i+k}^i)'\nonumber \\&\quad \times \varOmega _{i+k}^{-1}N_{i+k}^{j+i}\bigg ]+\theta _{j,0}Q_k, \end{aligned}$$

(19)

$$\begin{aligned} \varOmega _k&=R_k + \mathcal {D}'(k){\mathcal {P}}_{k+1}^0\mathcal {D}(k) + \sum \limits _{a=1}^f\sum \limits _{b=1}^f\gamma _{ab} \bar{\mathcal {D}}_a'(k){\mathcal {P}}_{k+1}^0\bar{\mathcal {D}}_b(k), \end{aligned}$$

(20)

$$\begin{aligned} N_k^j&=\mathcal {D}'(k){\mathcal {P}}_{k+1}^{j+1} + \mathcal {D}'(k){\mathcal {P}}_{k+1}^0\mathcal {C}_j(k) + \sum \limits _{a=1}^l\sum \limits _{b=1}^l\gamma _{ab}\nonumber \\&\quad \times \bar{\mathcal {D}}_a(k)'{\mathcal {P}}_{k+1}^0\bar{\mathcal {C}}_{j,b}(k),~~k=0,\dots ,N. \end{aligned}$$

(21)

Based on the above definitions, the solution to Problem 2 is derived in the following theorem.

Theorem 2

Problem 2has a unique ${\mathcal {F}}_{k-1}$- measurable $u_k$ if and only if $\varOmega _k$, for $k=0,\dots ,N$, are positive definite. In this case, the optimal controller $u_k$ is given by

$$\begin{aligned} u_k=-\varOmega _k^{-1}\sum \limits _{j=0}^dN_k^jx_{k-j}-\varOmega _k^{-1}\varSigma _k, \end{aligned}$$

and the associated optimal cost function is as

$$\begin{aligned} J_N^*&=x_0'{\mathcal {P}}_0^0x_0+2x_0'\sum \limits _{j=1}^{d}{\mathcal {P}}_0^jx_{-j} + \sum \limits _{j=1}^d\sum \limits _{i=1}^{d}\sum \limits _{l=0}^{d-1}x_{-j}'\bigg [\mathcal {C}_{j+l}'(l)\\&\quad \times {\mathcal {P}}_{l+1}^0\mathcal {C}_{i+l}(l)+ \sum \limits _{a=1}^f\sum \limits _{b=1}^f\gamma _{ab}\bar{\mathcal {C}}_{j+l,a}'(l){\mathcal {P}}_{l+1}^0\bar{\mathcal {C}}_{i+l,b}(l)\\&\quad +({\mathcal {P}}_{l+1}^{j+l+1})'\mathcal {C}_{i+l}(l)+\mathcal {C}_{j+l}'(l){\mathcal {P}}_{l+1}^{i+l+1}-(N_l^{j+l})'\\&\quad \times \varOmega _l^{-1}N_l^{i+l}\bigg ]x_{-i}+2x_0'\varPhi _0-\sum \limits _{k=0}^N\varSigma _k'\varOmega _k^{-1}\varSigma _k\\&\quad +2\sum \limits _{k=0}^N{\bar{\mu }}_k'\varPhi _{k+1}+\sum \limits _{k=0}^N{\rm Tr}[{\mathcal {P}}_{k+1}^0Q_{\mu _k}], \end{aligned}$$

where ${\mathcal {P}}_k^j$, $N_k^j$, $\varOmega _k$ satisfy the coupled equations (19)–(21).

Remark 6

It is obviously that the multiplicative noise $\mathcal {V}_k$ in the general LQG control system (18) is expanded by multiple dimensions of white noises. The existence of multi-dimensional white noise has no essential influence on the optimal control problem, and we can treat it as a whole. Then, the approach of Theorem 1 is also applied to the general situation. Thus, combining the mathematical characteristics of ${\mathcal {V}}_k$, Theorem 2 is derived as the above.

4 Numerical examples

Example 1

Consider the scalar case of time-varying LQG control system (3) in Theorem 1, as the additive noise $\mu _k$ correlated with $\nu _k$. Let the associated parameters be as

$$\begin{aligned}&\mathcal {C}_0(0)=1,~~ \mathcal {C}_1(0)=-1, ~~ \mathcal {C}_2(0)=1,\\&\bar{\mathcal {C}}_0(0)=-4,~~ \bar{\mathcal {C}}_1(0)=3,~~ \bar{\mathcal {C}}_2(0)=2,\\&\mathcal {C}_0(1)=2,~~ \mathcal {C}_1(1)=3, ~~ \mathcal {C}_2(1)=-2,\\&\bar{\mathcal {C}}_0(1)=2,~~ \bar{\mathcal {C}}_1(1)=-2,~~ \bar{\mathcal {C}}_2(1)=1,\\&\mathcal {D}(0)=4,~~ \bar{\mathcal {D}}(0)=1, ~~ \mathcal {D}(1)=-1,~~\bar{\mathcal {D}}(1)=-1, \\&\gamma =1,~~\tau =1,~~ Q_{\nu _k}=1, ~~ Q_{\mu _k}=1, ~~{\bar{\mu }}_k=0.2, \end{aligned}$$

and the cost function (2) with

$$\begin{aligned}&d=2,~~ N=1, ~~ \mathcal {P}^0_{N+1}=1, ~~ Q_k=1, ~~ R_k=1. \end{aligned}$$

By direct calculation , it yields

$$\begin{aligned}&\mathcal {P}^0_0=64.55,~~ \mathcal {P}^0_1=-48.73,~~ \mathcal {P}^0_2=-25.93,\\&N^0_0=2.67,~~ N^0_1=-6.33,~~ N^0_2=22,\\&\mathcal {P}^1_0=3.67,~~ \mathcal {P}^1_1=0.67,~~ \mathcal {P}^1_2=-0.67,\\&N^1_0=-4,~~ N^1_1=-1,~~ N^1_2=1\\&\varSigma _0=6.6,~~ \varSigma _1=-1.2, ~~ \varOmega _0=63.33,~~ \varOmega _1=3. \end{aligned}$$

It is obviously known that $\varOmega _k$ is positive definite for $k=0,1$. Thus, from Theorem 1, there exists a unique $u_k$, which is given by

$$\begin{aligned}&u_0^*=-0.0421\times x_0+0.1\times x_{-1}-0.3474\times x_{-2}-0.1042;\\&u_1^*=0.0632\times x_1+0.0158\times x_0-0.0158\times x_{-1}+0.0189. \end{aligned}$$

Second, we shall illustrate that $u_k^*$ can minimize the cost function (2). Let the controller be arbitrary. For example,

$$\begin{aligned}&{\hat{u}}_0=0.34\times x_0+2\times x_{-1}+0.8\times x_{-2}-0.2;\\&{\hat{u}}_1=2\times x_1+1\times x_0-0.2\times x_{-1}-1. \end{aligned}$$

Compare the cost function under $u_k^*$ and ${\hat{u}}_k$ with different initial values as follows:

$$\begin{aligned}&\text {1)~} x_0=1,\ x_{-1}=1,\ x_{-2}=2,\ J^*=3.56,\ {\hat{J}}=21.88;\\&\text {2)~} x_0=-1,\ x_{-1}=2,\ x_{-2}=2,\ J^*=61.65,\ {\hat{J}}=141.16;\\&\text {3)~} x_0=0,\ x_{-1}=-1,\ x_{-2}=-3,\ J^*=22.99,\ {\hat{J}}=36.79;\\&\text {4)~} x_0=2,\ x_{-1}=1,\ x_{-2}=-2,\ J^*=23.16,\ {\hat{J}}=42.09. \end{aligned}$$

Hence, $u_k^*$ and $J^*$ are optimal. This demonstrates the correctness of our results.

Example 2

Consider LQG control system (3) with $x_k\in {\mathbb {R}}^2$, $u_k\in {\mathbb {R}}^2$, $d=1$, $\gamma =1$, $\tau =[1,1]'$, and the coefficient matrixes are time-invariant with

$$\begin{aligned}&\mathcal {C}_0=\begin{bmatrix}-1&{}1.2\\ 0.8&{}1\end{bmatrix}, \ \bar{\mathcal {C}}_0=\begin{bmatrix}-0.2&{}0\\ 1&{}-0.5\end{bmatrix}, \ \mathcal {C}_1=\begin{bmatrix}-1&{}~2\\ 0&{}~1\end{bmatrix},\\&\bar{\mathcal {C}}_1=\begin{bmatrix}0.2&{}-2\\ 1&{}1\end{bmatrix},\ \mathcal {C}_2=\begin{bmatrix}-1&{}2\\ -2&{}-0.4\end{bmatrix}, \ \bar{\mathcal {C}}_2=\begin{bmatrix}0.6&{}-2\\ 1&{}0\end{bmatrix},\\&\mathcal {C}_3=\begin{bmatrix}0&{}1.5\\ -0.7&{}1\end{bmatrix},\ \bar{\mathcal {C}}_3=\begin{bmatrix}0.8&{}-1.3\\ 1&{}0.4\end{bmatrix}, \ \mathcal {D}=\begin{bmatrix}0.3&{}~0\\ -1&{}~0.2\end{bmatrix},\\&\bar{\mathcal {D}}=\begin{bmatrix}1&{}1.2\\ 0.3&{}-1\end{bmatrix},\ Q_{\nu }=I,\ Q_{\mu }=I, \end{aligned}$$

and the cost function (2) with

$$\begin{aligned}&N=3,~~ Q=I,~~ R=I, ~~ \mathcal {P}^0_{N+1}=0. \end{aligned}$$

Then by direct calculation, the solution to coupled difference Riccati equations is yielded as

$$\begin{aligned}&\mathcal {P}^0_0=\begin{bmatrix}-6.28&{}-11.33\\ -0.41&{}-37.19\end{bmatrix}, ~~ \mathcal {P}^0_1=\begin{bmatrix}1.43&{}-1.29\\ 1.73&{}-2.84\end{bmatrix},\\&\mathcal {P}^0_2=\begin{bmatrix}2.46&{}-0.02\\ -0.72&{}2.78\end{bmatrix}, ~~ \mathcal {P}^0_3=\begin{bmatrix}1&{}~0\\ 0&{}~1\end{bmatrix},\\&\mathcal {P}^1_0=\begin{bmatrix}-2.11&{}5.09\\ 4.08&{}-6.45\end{bmatrix}, ~~ \mathcal {P}^1_1=\begin{bmatrix}1.72&{}-5.66\\ 0.15&{}-3.70\end{bmatrix},\\&\mathcal {P}^1_2=\begin{bmatrix}0.81&{}-1.77\\ -0.84&{}0.29\end{bmatrix}, ~~ \mathcal {P}^1_3=\begin{bmatrix}0&{}~0\\ 0&{}~0\end{bmatrix},\\&N^0_0=\begin{bmatrix}1.10&{}3.96\\ 0.52&{}-1.53\end{bmatrix}, ~~ N^0_1=\begin{bmatrix}-2.24&{}-2.27\\ -3.12&{}1.85\end{bmatrix},\\&N^0_2=\begin{bmatrix}-1&{}-0.79\\ -1.08&{}0.7\end{bmatrix}, ~~ N^0_3=\begin{bmatrix}0&{}~0\\ 0&{}~0\end{bmatrix},\\&N^1_0=\begin{bmatrix}-0.45&{}-6.19\\ 0.94&{}1.46\end{bmatrix}, ~~ N^1_1=\begin{bmatrix}-0.19&{}-3.54\\ -1.93&{}-9.9\end{bmatrix},\\&N^1_2=\begin{bmatrix}0.2&{}-2.1\\ -0.76&{}-3.2\end{bmatrix}, ~~ N^1_3=\begin{bmatrix}0&{}~0\\ 0&{}~0\end{bmatrix},\\&\varOmega _0=\begin{bmatrix}4.11&{}~0.99\\ 0.99&{}~4.53\end{bmatrix}, ~~ \varOmega _1=\begin{bmatrix}6.71&{}~1.32\\ 2.23&{}~8.33\end{bmatrix},\\&\varOmega _2=\begin{bmatrix}3.18&{}0.7\\ 0.7&{}3.48\end{bmatrix}, ~~ \varOmega _3=\begin{bmatrix}1&{}~0\\ 0&{}~1\end{bmatrix}. \end{aligned}$$

Obviously, $\varOmega _k>0$ for $k=0,\ldots ,N$, therefore, there exist a optimal controller for Problem 1. In addition, when the initial values are

$$\begin{aligned}&x_0=\begin{bmatrix}-0.5\\ 0.8\end{bmatrix},~~ x_{-1}=\begin{bmatrix}0.3\\ -0.7\end{bmatrix}, \end{aligned}$$

the optimal controller can be calculated as

$$\begin{aligned} u_0^*=\begin{bmatrix}2.52\\ -1.36\end{bmatrix}, \ u_1^*=\begin{bmatrix}-3.68\\ 1.58\end{bmatrix},\ u_2^*=\begin{bmatrix}-3.89\\ -4.6\end{bmatrix}, \ u_3^*=\begin{bmatrix}0\\ 0\end{bmatrix}. \end{aligned}$$

Accordingly, the associated cost function is $J^*=148.8$.

5 Conclusions

In this paper, the discrete time-varying LQG control problem with both multiplicative noise and multiple state delays has been studied. We obtain the solution to the FBSDEs for the discrete time-varying systems. A necessary and sufficient condition for the existence of a unique optimal controller is proposed. The basis of this approach is the stochastic maximum principle and the key is the relationship between the state and costate. In the future works, we expect that the results in this paper shall pave new ways for networked control system with both state delays and packet dropout.

Appendices

Appendix A

With the stochastic maximum principle (4)–(6) to LQG control system (3) involving multiple state delays and multiplicative noise, we can obtain for $k=N$,

$$\begin{aligned} 0&=\sum \limits _{i=0}^d\big (\mathcal {D}'(N){\mathcal {P}}_{N+1}^0\mathcal {C}_i(N)+\gamma \bar{\mathcal {D}}'(N){\mathcal {P}}_{N+1}^0\bar{\mathcal {C}}_i(N)\big )x_{N-i}\\&\quad +\big (\mathcal {D}'(N){\mathcal {P}}_{N+1}^0D(N)+\gamma \bar{\mathcal {D}}'(N){\mathcal {P}}_{N+1}^0\bar{\mathcal {D}}(N)+R_N\big )\\&\quad \times u_N+\mathcal {D}'(N){\mathcal {P}}_{N+1}^0{\bar{\mu }}_N+\bar{\mathcal {D}}'(N){\mathcal {P}}_{N+1}^0\tau . \end{aligned}$$

Using Eqs. (8) and (9), the optimal controller $u_N$ is as

$$\begin{aligned} u_N=&-\varOmega _N^{-1}\sum \limits _{i=0}^dN_N^ix_{N-i}-\varOmega _N^{-1}\varSigma _N, \end{aligned}$$

where $\varSigma _N=\mathcal {D}'(N){\mathcal {P}}_{N+1}^0{\bar{\mu }}_N+\bar{\mathcal {D}}'(N){\mathcal {P}}_{N+1}^0\tau$. From (4) and (5), we also have

$$\begin{aligned} \zeta _{N-1}&=\mathrm{E}\Bigg [(\mathcal {C}_N^0)'(N){\mathcal {P}}_{N+1}^0x_{N+1}|{\mathcal {F}}_{N-1}\Bigg ]+Q_Nx_N\\&=\mathrm{E}\Bigg [\Bigg (\sum \limits _{i=0}^d(\mathcal {C}_N^0)'(N){\mathcal {P}}_{N+1}^0\mathcal {C}_N^i(N)-(N_N^0)'\varOmega _N^{-1}N_N^i\Bigg )\\&\quad \times x_{N-i}- (N_N^0)'\varOmega _N^{-1}\mathcal {D}_N'(N){\mathcal {P}}_{N+1}^0\mu _N + (\mathcal {C}_N^0)'(N)\\&\quad \times {\mathcal {P}}_{N+1}^0\mu _N|{\mathcal {F}}_{N-1}\Bigg ]+Q_Nx_N. \end{aligned}$$

Substituting (7), $\zeta _{N-1}$ yields

$$\begin{aligned} \zeta _{N-1}&=\sum \limits _{i=1}^d{\mathcal {P}}_N^ix_{N-i}+{\mathcal {P}}_N^0x_N+(\mathcal {C}_0'(N){\mathcal {P}}_{N+1}^0{\bar{\mu }}_N\\&\quad +\bar{\mathcal {C}}_0'(N){\mathcal {P}}_{N+1}^0\tau -(N_N^0)'\varOmega _N^{-1}\mathcal {D}'(N){\mathcal {P}}_{N+1}^0\\&\quad \times {\bar{\mu }}_N-(N_N^0)'\varOmega _N^{-1}\bar{\mathcal {D}}'(N){\mathcal {P}}_{N+1}^0\tau )\\&=\sum \limits _{j=0}^d{\mathcal {P}}_{N}^jx_{N-j}+\varPhi _N, \end{aligned}$$

where $\varPhi _N$ satisfied (12) with the terminal values being zero.

Now, we have verified (11) for $k=N$. Supposing that $\zeta _{k-1}$ are as (11) for all $k\geqslant n+1$, we will show that (11) also holds for $k=n$. For $k=n+1$, with (3) and (11), $\zeta _n$ can be calculated as

$$\begin{aligned} \zeta _n&=\sum \limits _{j=0}^d{\mathcal {P}}_{n+1}^jx_{n+1-j}+\varPhi _{n+1}\nonumber \\&=\sum \limits _{j=1}^d{\mathcal {P}}_{n+1}^jx_{n+1-j}+{\mathcal {P}}_{n+1}^0\sum \limits _{i=0}^d\big (\mathcal {C}_n^i(n)x_{n-i}+\mathcal {D}(n)u_{n}\nonumber \\&\quad +\mu _{n}\big )+\varPhi _{n+1} . \end{aligned}$$

(22)

Inserting $\zeta _n$ to (6), (6) will become

$$\begin{aligned} 0&=\mathrm{E}\Bigg [\sum \limits _{j=0}^d\big (\mathcal {D}_n'(n){\mathcal {P}}_{n+1}^{j+1} + \mathcal {D}_n'(n){\mathcal {P}}_{n+1}^0\mathcal {C}_n^j(n)\big )x_{n-j} + \mathcal {D}_n'(n)\\&\quad \times \!{\mathcal {P}}_{n+1}^0\mathcal {D}_n(n)u_n \!+\! \mathcal {D}_n'(n){\mathcal {P}}_{n+1}^0\mu _n \!+\! \mathcal {D}_n'(n)\varPhi _{n+1}|{\mathcal {F}}_{n-1}\Bigg ] \!+\!R_nu_n\\&=\sum \limits _{j=0}^d N_{n}^jx_{n-j}+ \varOmega _nu_n + \mathcal {D}'(n)\left( {\mathcal {P}}_{n+1}^0{\bar{\mu }}_n + \varPhi _{n+1}\right) \\&\quad + \bar{\mathcal {D}}'(n){\mathcal {P}}_{n+1}^0\tau . \end{aligned}$$

Thus, the optimal controller is given by

$$\begin{aligned} u_n=-\varOmega _n^{-1}\sum \limits _{j=0}^dN_{n}^jx_{n-j}-\varOmega _n^{-1}\varPhi _n. \end{aligned}$$

(23)

In virtue of equations (3), (5) and (23), $\zeta _{n-1}$ yields that

$$\begin{aligned}&\zeta _{n-1}\\&\quad =\mathrm{E}\Bigg [ \sum \limits _{m=0}^{d-1} (\mathcal {C}_{n+m}^m)'(n + m)\zeta _{n+m} + (\mathcal {C}_{n+d}^d)'(n + d)\\&\qquad \times\Big (\sum \limits _{j=1}^d{\mathcal {P}}_{n+d+1}^j x_{n+d+1-j}+{\mathcal {P}}_{n+d+1}^0\Big (\sum \limits _{i=0}^d\mathcal {C}_{n+d}^i(n + d)x_{n+d-i}\\&\qquad +\mu _{n+d}+\mathcal {D}_{n+d}(n + d)u_{n+d}\Big )+\varPhi _{n+d+1}\Big )|{\mathcal {F}}_{n-1}\Bigg ]+Q_nx_n\\&\quad =\mathrm{E}\Bigg [\sum \limits _{m=0}^{d-1}(\mathcal {C}_{n+m}^m)'(n + m)\zeta _{n+m} + \sum \limits _{j=0}^d\big ((\mathcal {C}_{n+d}^d)'(n + d){\mathcal {P}}_{n+d+1}^{j+1}\\&\qquad +(\mathcal {C}_{n+d}^d)'(n + d){\mathcal {P}}_{n+d+1}^0\mathcal {C}_{n+d}^j(n + d)-(N_{n+d}^d)'\varOmega _{n+d}^{-1}\\&\qquad \times N_{n+d}^j\big )x_{n+d-j}-(N_{n+d}^d)'\varOmega _{n+d}^{-1}\varSigma _{n+d}+(\mathcal {C}_{n+d}^d)'(n + d)\\&\qquad \times ({\mathcal {P}}_{n+d+1}^0\mu _{n+d}+\varPhi _{n+d+1})|{\mathcal {F}}_{n-1}\Bigg ]+Q_nx_n \\&\quad =\mathrm{E}\Bigg [\sum \limits _{m=0}^{d-2}(\mathcal {C}_{n+m}^m)'(n + m)\zeta _{n+m}+\sum \limits _{j=0}^d\big ((\mathcal {C}_{n+d-1}^{d-1})'(n + d - 1)\\&\qquad \times {\mathcal {P}}_{n+d}^{j+1}+(\mathcal {C}_{n+d-1}^{d-1})'(n + d - 1){\mathcal {P}}_{n+d}^0\mathcal {C}_{n+d-1}^j(n + d - 1)\\&\qquad +(\mathcal {C}_{n+d}^d)'(n + d){\mathcal {P}}_{n+d+1}^{j+2}+(\mathcal {C}_{n+d}^d)'(n + d){\mathcal {P}}_{n+d+1}^0\\&\qquad \times \mathcal {C}_{n+d}^{j+1}(n + d)-(N_{n+d}^d)'\varOmega _{n+d}N_{n+d}^{j+1}+({\mathcal {P}}_{n+d}^d)'\\&\qquad \times \mathcal {C}_{n+d-1}^j(n + d - 1)\big )x_{n+d-1-j}+(N_{n+d-1}^{d-1})'u_{n+d-1}\\&\qquad - (N_{n+d}^d)'\varOmega _{n+d}^{-1}\varSigma _{n+d} + (\mathcal {C}_{n+d}^d)'(n + d)({\mathcal {P}}_{n+d+1}^0\mu _{n+d} \\&\qquad + \varPhi _{n+d+1})+(\mathcal {C}_{n+d-1}^{d-1})'(n + d - 1)({\mathcal {P}}_{n+d}^0g_{n+d-1}+\varPhi _{n+d})\\&\qquad +({\mathcal {P}}_{n+d}^{d})'\mu _{n+d-1}|{\mathcal {F}}_{n-1}\Bigg ]+Q_nx_n\\&\quad =\mathrm{E}\Bigg [\sum \limits _{m=0}^{d-3}(\mathcal {C}_{n+m}^m)'(n + m)\zeta _{n+m} + \sum \limits _{j=0}^d\sum \limits _{i=d-2}^d \big ((\mathcal {C}_{n+i}^i)'(n + i)\\&\qquad \times {\mathcal {P}}_{n+i+1}^0\mathcal {C}_{n+i}^{i+j-d+2}(n + i)+(\mathcal {C}_{n+i}^i)'(n + i){\mathcal {P}}_{n+i+1}^{i+j-d+3}\\&\qquad +({\mathcal {P}}_{n+i+1}^{i+1})'\mathcal {C}_{n+i}^{j+1}(n+i)-(N_{n+i}^i)'\varOmega _{n+i}^{-1}N_{n+i}^{i+j-d+2}\big )\\&\qquad \times x_{n+d-2-j}-\sum \limits _{i=d-1}^d \big ((N_{n+i}^i)'\varOmega _{n+i}^{-1}\varSigma _{n+i}+(\mathcal {C}_{n+i}^i)'(n + i)\\&\qquad \times ({\mathcal {P}}_{n+i+1}^0\mu _{n+i}+\varPhi _{n+i+1})+({\mathcal {P}}_{n+i+1}^{i+1})'\mu _{n+i}\big )|{\mathcal {F}}_{n-1}\Bigg ]\\&\qquad +Q_nx_n. \end{aligned}$$

Plugging (3) and (23) into the above equation for times d, we can calculate $\zeta _{n-1}$ as follows:

$$\begin{aligned}&\zeta _{n-1}\\&\quad =\mathrm{E}\Bigg [\sum \limits _{j=0}^d\sum \limits _{i=0}^d \Bigg ((\mathcal {C}_{n+i}^i)'(n + i){\mathcal {P}}_{n+i+1}^0\mathcal {C}_{n+i}^{i+j}(n + i) \\&\qquad + (\mathcal {C}_{n+i}^i)'(n + i) {\mathcal {P}}_{n+i+1}^{i+j}+({\mathcal {P}}_{n+i+1}^{i+1})'\mathcal {C}_{n+i}^{j+1}(n+i)\\&\qquad -(N_{n+i}^i)'\varOmega _{n+i}^{-1}N_{n+i}^{i+j}\Bigg ) x_{n+1-j}+\sum \limits _{i=0}^d\Bigg ((-N_{n+i}^i)'\varOmega _{n+i}^{-1}\varSigma _{n+i}\\&\qquad +(\mathcal {C}_{n+i}^i)'(n+i) ({\mathcal {P}}_{n+i+1}^0\mu _{n+i}+\varPhi _{n+i+1})\\&\qquad +({\mathcal {P}}_{n+i+1}^{i+1})'\mu _{n+i}\Bigg )|{\mathcal {F}}_{n-1}\Bigg ]+Q_nx_n, \end{aligned}$$

where ${\mathcal {P}}_{n+i+1}^{i+j}=0$ for $i+j>d$ from Remark 1, and then

$$\begin{aligned}&\zeta _{n-1}\\&\quad =\sum \limits _{j=0}^d\sum \limits _{i=0}^{d-j}\big (\mathcal {C}_i'(n + i){\mathcal {P}}_{n+i+1}^0\mathcal {C}_{i+j}(n + i)\\&\qquad +\gamma \bar{\mathcal {C}}_i'(n + i){\mathcal {P}}_{n+i+1}^0 \bar{\mathcal {C}}_{i+j}(n + i)+\mathcal {C}_i'(n + i){\mathcal {P}}_{n+i+1}^{i+j+1}\\&\qquad +({\mathcal {P}}_{n+i+1}^{i+1})'\mathcal {C}_{i+j}(n + i)-(N_{n+i}^i)'\varOmega _{n+i}^{-1}N_{n+i}^{i+j}\big )x_{n-j}\\&\qquad +\sum \limits _{i=0}^d\big ((-N_{n+i}^i)'\varOmega _{n+i}^{-1}\varSigma _{n+i}+\mathcal {C}_i'(n + i)({\mathcal {P}}_{n+i+1}^0{\bar{\mu }}_{n+i}\\&\qquad + \varPhi _{n+i+1}) + \bar{\mathcal {C}}_i'(n + i)({\mathcal {P}}_{n+i+1}^0\tau + ({\mathcal {P}}_{n+i+1}^{i+1})'{\bar{\mu }}_{n+i}\big )+Q_nx_n. \end{aligned}$$

After inserting (7), we can summarize that

$$\begin{aligned} \zeta _{n-1}=\sum \limits _{j=0}^d{\mathcal {P}}_{n}^jx_{n-j}+\varPhi _{n}. \end{aligned}$$

This completes the proof of the lemma.

Appendix B

(Necessity)   Suppose that there exists the unique ${\mathcal {F}}_{k-1}$-measurable $u_k$ to make the cost function (2) minimized. We will show that $\varOmega _k, k=0,\dots ,N$ are positive definite by induction and the optimal controller can be designed as (13). Define

$$\begin{aligned} J(k)= \mathrm{E}\Bigg [\sum \limits _{i=k}^Nx_i'Q_ix_i+u_i'R_iu_i+x_{N+1}'{\mathcal {P}}_{N+1}^0x_{N+1}\Bigg ]. \end{aligned}$$

When $k=N$, J(N) is presented as

$$\begin{aligned} J(N)&= u_N'\varOmega _Nu_N + 2u_N(\mathcal {D}(N){\mathcal {P}}_{N+1}^0{\bar{\mu }}_N + \bar{\mathcal {D}}(N){\mathcal {P}}_{N+1}^0\tau )\\&\quad + {\rm Tr}[{\mathcal {P}}_{N+1}^0Q_{\mu _N}], \end{aligned}$$

where $x_N=0$ and $x_{N-j}=0$ for $j=0,\ldots ,d$ as the uniqueness of the optimal controller is unrelated with $x_k$.

As J(N) can be expressed as a quadratic function of $u_N$, and the performance index must be positive, it can be obviously know that $\varOmega _N>0$, i.e., $\varOmega _k$ is positive definite for $k=N$. Assuming $\varOmega _k>0$ for all $k\geqslant n+1$, we will prove that $\varOmega _n>0$. With (3), (5) and (6), for $k\geqslant n+1$, we construct that

$$\begin{aligned}&\mathrm{E}\Bigg [x_k'\zeta _{k-1}-x_{k+1}'\zeta _k\Bigg ]\nonumber \\&\quad =\mathrm{E}\Bigg [x_k'\mathrm{E}[\sum \limits _{m=0}^d\mathcal {C}_{k+m}^m(k+m)\zeta _{k+m}|{\mathcal {F}}_{k-1}]+x_k'Q_kx_k\nonumber \\&\qquad -\Bigg(\sum \limits _{i=0}^d\mathcal {C}_k^i(k)x_{k-i}+\mathcal {D}_k(k)u_k+\mu _k\Bigg )'\zeta _k\Bigg ]\nonumber \\&\quad =\mathrm{E}\Bigg [x_k'Q_kx_k-u_k'\mathrm{E}[\mathcal {D}'(k)\zeta _k|{\mathcal {F}}_{k-1}]-\mu _k'\zeta _k\Bigg ]\nonumber \\&\quad =\mathrm{E}\Bigg [x_k'Q_kx_k+u_k'R_ku_k-\mu _k'\zeta _k\Bigg ]. \end{aligned}$$

(24)

To obtain the form of J(N), we add both sides of (24) from $k=n+1$ to $k=N$, we have

$$\begin{aligned} \mathrm{E}[x_{n+1}'\zeta _n-x_{N+1}'\zeta _N]= \sum \limits _{k=n+1}^N \mathrm{E}\big [x_k'Q_kx_k+u_k'R_ku_k-\mu _k'\zeta _k\big ]. \end{aligned}$$

Then,

$$\begin{aligned}&\mathrm{E}\Bigg [\sum \limits _{k=n+1}^N\Bigg (x_k'Q_kx_k+u_k'R_ku_k\Bigg )+x_{N+1}'{\mathcal {P}}_{N+1}^0x_{N+1}\Bigg ]\\&=\mathrm{E}\big [x_{n+1}'\zeta _n-\sum \limits _{k=n}^N\mu _k'\zeta _k\big ]. \end{aligned}$$

Compared with (2), it yields that

$$\begin{aligned} J(n) = \big [x_n'Q_nx_n+u_n'R_nu_n\big ]+\mathrm{E}\Big [x_{n+1}'\zeta _n + \sum \limits _{k=n+1}^{N} \mu _k'\zeta _k\Big ]. \end{aligned}$$

(25)

Setting $x_n=0$ and $x_{n-i}=0$, for $i=0,\ldots ,d$ as the same as the condition $k=N$, and plugging (11) into (25), we obtain

$$\begin{aligned} J(n)&=\mathrm{E}\Bigg [u_n'R_nu_n+u_n'D_n'(n)\zeta _n+\sum \limits _{k=n}^N\mu _k'\zeta _k\Bigg ]\\&= \mathrm{E}\big [u_n'R_nu_n+u_n'\mathcal {D}_n'(n)({\mathcal {P}}_{n+1}^0\mathcal {D}_n(n)u_n+\mu _n)\\&\quad +u_n'\mathcal {D}_n'(n)\varPhi _{n+1}+\sum \limits _{k=n}^N\mu _k'\zeta _k\big ]\\&=u_n'\varOmega _nu_n+u_n'\big (\mathcal {D}'(n)({\mathcal {P}}_{n+1}^0{\bar{\mu }}_n+\varPhi _{n+1})\\&\quad +\bar{\mathcal {D}}'(n){\mathcal {P}}_{n+1}^0\tau \big )+\sum \limits _{k=n+1}^N\mu _k'\zeta _k. \end{aligned}$$

Similarly to the case $\varOmega _N>0$ above, we obviously get $\varOmega _n>0$ for all $k=0,\ldots , N$. This completes the proof of necessity.

(Sufficiency)   Suppose that $\varOmega _k>0$ for $k=0,\dots ,N$ is ture, we will show the existence of the unique ${\mathcal {F}}_{k-1}$-measurable $u_k$ to minimize (2). Make the definition:

$$\begin{aligned}&V(x_k)\\&\quad =\mathrm{E}\Bigg [x_k'{\mathcal {P}}_k^0x_k+2x_k'\sum \limits _{j=1}^{d}{\mathcal {P}}_k^jx_{k-j}\\&\qquad +\sum \limits _{j=1}^{d}\sum \limits _{i=1}^{d}\sum \limits _{l=0}^{d-1}x_{k-j}' \Bigg [\mathcal {C}_{j+l}'(k + l) {\mathcal {P}}_{k+l+1}^0\mathcal {C}_{i+l}(k + l)\\&\qquad +\gamma \bar{\mathcal {C}}_{j+l}'(k + l){\mathcal {P}}_{k+l+1}^0 \bar{\mathcal {C}}_{i+l}(k + l)+\mathcal {C}_{j+l}'(k + l){\mathcal {P}}_{k+l+1}^{i+l+1}\\&\qquad +({\mathcal {P}}_{k+l+1}^{j+l+1})' \mathcal {C}_{i+l}(k + l)-(N_{k+l}^{j+l})'\varOmega _{k+l}^{-1}N_{k+l}^{i+l}\Bigg ]x_{k-i}\\&\qquad +2x_k'\varPhi _k\Bigg ]. \end{aligned}$$

First, as $k=k+1$, using the equivalent substitution $l=l+1$, $j=j-1$, and $i=i-1$ in turn, the $V(x_{k+1})$ can be calculated as

$$\begin{aligned}&V(x_{k+1})\\&\quad =\mathrm{E}\Bigg [x_k'\big ((\mathcal {C}_k^0)'(k){\mathcal {P}}_{k+1}^0\mathcal {C}_k^0(k)+(\mathcal {C}_k^0)'(k){\mathcal {P}}_{k+1}^1\\&\qquad +({\mathcal {P}}_{k+1}^1)' \mathcal {C}_k^0(k)\big )x_k+2x_k'\sum \limits _{j=1}^d\big ((\mathcal {C}_k^0)'(k){\mathcal {P}}_{k+1}^0\mathcal {C}_k^j(k)\\&\qquad +(\mathcal {C}_k^0)'(k) {\mathcal {P}}_{k+1}^{j+1}+({\mathcal {P}}_{k+1}^1)'\mathcal {C}_k^j(k)\big )x_{k-j}\\&\qquad +\sum \limits _{j=1}^d\sum \limits _{i=1}^dx_{k-i}' \big ((\mathcal {C}_k^i)'(k) {\mathcal {P}}_{k+1}^0\mathcal {C}_k^j(k)+(\mathcal {C}_k^i)'(k){\mathcal {P}}_{k+1}^{j+1} \\&\qquad +({\mathcal {P}}_{k+1}^{j+1})'\mathcal {C}_k^i(k)\big )x_{k-j}+2u_k'\sum \limits _{j=0}^dN_k^jx_{k-j}+u_k'(\varOmega _k - R)u_k\\&\qquad +\sum \limits _{j=0}^{d-1}\sum \limits _{i=0}^{d-1}\sum \limits _{l=1}^dx_{k-j}' [\mathcal {C}_{j+l}'(k + l){\mathcal {P}}_{k+l+1}^0\mathcal {C}_{i+l}(k + l)\\&\qquad + \gamma \bar{\mathcal {C}}_{j+l}'(k + l){\mathcal {P}}_{k+l+1}^0 \bar{\mathcal {C}}_{i+l}(k + l)+\mathcal {C}_{j+l}'(k + l){\mathcal {P}}_{k+l+1}^{i+l+1}\\&\qquad +({\mathcal {P}}_{k+l+1}^{j+l+1})'\mathcal {C}_{i+l}(k + l)-(N_{k+l}^{j+l})'\varOmega _{k+l}^{-1}N_{k+l}^{i+l}]x_{k-i}\\&\qquad +2\mu _k'\sum \limits _{j=0}^d{\mathcal {P}}_{k+1}^0\mathcal {C}_k^j(k)x_{k-j}+2\mu _k'{\mathcal {P}}_{k+1}^0\mathcal {D}_k(k)u_k+\mu _k'{\mathcal {P}}_{k+1}^0\mu _k \\&\qquad +\sum \limits _{j=0}^d\mu _k'{\mathcal {P}}_{k+1}^{j+1}x_{k-j}+2x_{k+1}'\varPhi _{k+1}\Bigg ]. \end{aligned}$$

Constructing the form $V(x_k)-V(x_{k+1})$, we have

$$\begin{aligned}&V(x_k)-V(x_{k+1})\\&\quad =\mathrm{E}\Bigg [x_k'Q_kx_k+u_k'R_ku_k - \Bigg (u_k+\varOmega _k^{-1}\sum \limits _{j=0}^dN_k^jx_{k-j}\Bigg )'\varOmega _k\Bigg (u_k\\&\qquad +\varOmega _k^{-1}\sum \limits _{j=0}^dN_k^jx_{k-j}\Bigg )-2u_k'\varSigma _k+2x_k\varPhi _k-2\sum \limits _{j=0}^dx_{k-j}'\\&\qquad \times \Bigg ((\mathcal {C}_k^j)'(k)\Bigg ({\mathcal {P}}_{k+1}^0\mu _k+\varPhi _{k+1}\Bigg )+{\mathcal {P}}_{k+1}^{j+1}\mu _k\Bigg )-2\mu _k'\varPhi _{k+1}\\&\qquad -\mu _k'{\mathcal {P}}_{k+1}^0\mu _k\Bigg ]. \end{aligned}$$

Denote

$$\begin{aligned} \phi _k^i&=\big (\mathcal {C}_i'(k)-(N_k^i)'\varOmega _k^{-1}\mathcal {D}'(k)\big )({\mathcal {P}}_{k+1}^0{\bar{\mu }}_k+\varPhi _{k+1})\\&\quad +\big (\bar{\mathcal {C}}_i'(k)-(N_k^i)'\varOmega _k^{-1}\bar{\mathcal {D}}'(k)\big ){\mathcal {P}}_{k+1}^0\tau +({\mathcal {P}}_{k+1}^{i+1})'{\bar{\mu }}_k. \end{aligned}$$

We can obviously know that $\varPhi _k=\sum \limits _{i=0}^d\phi _{k+i}^i$. Then,

$$\begin{aligned}&\mathrm{E}\Bigg [2x_k\varPhi _k - 2\sum \limits _{j=0}^dx_{k-j}'\big ((\mathcal {C}_k^j)'(k)({\mathcal {P}}_{k+1}^0\mu _k + \varPhi _{k+1})\nonumber \\& + {\mathcal {P}}_{k+1}^{j+1}\mu _k\Bigg ]\nonumber \\&\quad =2\sum \limits _{j=0}^dx_{k-j}'\phi _k^j-2\sum \limits _{j=0}^d\big (\phi _k^j+(N_k^j)'\varOmega _k^{-1}\varSigma _k\big )\nonumber \\&\quad =-2\sum \limits _{j=0}^dx_{k-j}'(N_k^j)'\varOmega _k^{-1}\varSigma _k. \end{aligned}$$

(26)

By virtue of (26), the following equation becomes

$$\begin{aligned}&V(x_k)-V(x_{k+1})\\&\quad =\mathrm{E}\Bigg [x_k'Q_kx_k+u_k'R_ku_k - \big (u_k+\varOmega _k^{-1}\sum \limits _{j=0}^dN_k^jx_{k-j}\big )'\varOmega _k\big (u_k\\&\qquad +\varOmega _k^{-1}\sum \limits _{j=0}^dN_k^jx_{k-j}\big )-2u_k'\varSigma _k-2\sum \limits _{j=0}^dx_{k-j}'(N_k^j)'\varOmega _k^{-1}\varSigma _k\\&\qquad -2\mu _k'\varPhi _{k+1}-\mu _k'{\mathcal {P}}_{k+1}^0\mu _k\Bigg ]\\&\quad =\mathrm{E}\big [x_k'Q_kx_k+u_k'R_ku_k - \big (u_k+\varOmega _k^{-1}\sum \limits _{j=0}^dN_k^jx_{k-j}\big )'\varOmega _k\big (u_k\\&\qquad +\varOmega _k^{-1}\sum \limits _{j=0}^dN_k^jx_{k-j}\big )-2u_k'\varSigma _k-2\sum \limits _{j=0}^dx_{k-j}'(N_k^j)'\varOmega _k^{-1}\varSigma _k\\&\qquad -\varSigma _k'\varOmega _k^{-1}\varSigma _k+\varSigma _k'\varOmega _k^{-1}\varSigma _k-2\mu _k'\varPhi _{k+1}-\mu _k'{\mathcal {P}}_{k+1}^0\mu _k\big ]\\&\quad =x_k'Q_kx_k + u_k'R_ku_k - \big (u_k + \varOmega _k^{-1}\sum \limits _{j=0}^dN_k^jx_{k-j}+\varOmega _k^{-1}\varSigma _k\big )'\\&\qquad \times \varOmega _k\big (u_k+\varOmega _k^{-1}\sum \limits _{j=0}^dN_k^jx_{k-j}+\varOmega _k^{-1}\varSigma _k\big ) + \varSigma _k'\varOmega _k^{-1}\varSigma _k\\&\qquad -2{\bar{\mu }}_k'\varPhi _{k+1}-{\rm Tr}[{\mathcal {P}}_{k+1}^0Q_{\mu _k}]. \end{aligned}$$

Adding from $k=0$ to $k=N$, the following equation is obtained:

$$\begin{aligned}&V(x_0)-V(x_{N+1})\\&\quad =\sum \limits _{k=0}^{N}\Bigg [x_k'Q_kx_k + u_k'R_ku_k - \Bigg (u_k + \varOmega _k^{-1}\sum \limits _{j=0}^dN_k^jx_{k-j}\\&\qquad + \varOmega _k^{-1} \varSigma _k\Bigg )'\varOmega _k\Bigg (u_k + \varOmega _k^{-1}\sum \limits _{j=0}^dN_k^jx_{k-j} + \varOmega _k^{-1}\varSigma _k\Bigg )\\&\qquad + \varSigma _k'\varOmega _k^{-1} \varSigma _k-2{\bar{\mu }}_k'\varPhi _{k+1}-{\rm Tr}[{\mathcal {P}}_{k+1}^0Q_{\mu _k}]\Bigg ]. \end{aligned}$$

Then, the cost function (2) becomes

$$\begin{aligned} J_N&=V(x_0)+\sum \limits _{k=0}^N\Bigg [\Bigg (u_k+\varOmega _k^{-1}\sum \limits _{j=0}^dN_k^jx_{k-j}+\varOmega _k^{-1}\varSigma _k\Bigg )'\\&\quad \times \varOmega _k\Bigg (u_k+\varOmega _k^{-1}\sum \limits _{j=0}^dN_k^jx_{k-j}+\varOmega _k^{-1}\varSigma _k\Bigg )\\&\quad -\varSigma _k'\varOmega _k^{-1} \varSigma _k+2{\bar{\mu }}_k'\varPhi _{k+1}+{\rm Tr}[{\mathcal {P}}_{k+1}^0Q_{\mu _k}]\Bigg ]. \end{aligned}$$

As $\varOmega _k>0$, the unique optimal controller is

$$\begin{aligned} u_k^*=-\varOmega _k^{-1}\sum \limits _{j=0}^dN_{k}^jx_{k-j}-\varOmega _k^{-1}\varSigma _k, \end{aligned}$$

which minimized the cost function (2), and the optimal cost is

$$\begin{aligned} J_N^*=V(x_0) + \sum \limits _{k=0}^{N}\Bigg (2{\bar{\mu }}_k'\varPhi _{k+1} - \varSigma _k'\varOmega _k^{-1}\varSigma _k+ {\rm Tr}[{\mathcal {P}}_{k+1}^0Q_{g_k}]\Bigg ). \end{aligned}$$

Now, the proof of Theorem 1 is completed.