Abstract
In this paper some noise models for stochastic control systems are described that differ from the well known model of Brownian motion. Some of the processes are Gaussian such as the family of fractional Brownian motions and other processes are non-Gaussian, especially Rosenblatt processes. These processes have a long range dependence property that can be described by a slow decay of the covariance process. A stochastic calculus for these processes exists that is more limited than the calculus for Brownian motion that is inherited from the martingale property but nevertheless is sufficient for addressing some stochastic control problems. In many physical situations the data demonstrates a long range dependence which can justify a choice from these non-Brownian processes. Explicit solutions of stochastic control problems with quadratic cost functionals having driving noise from the Rosenblatt processes are given.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
Linear–quadratic stochastic control problems are formulated and the optimal controls are explicitly described where the noise driving the systems are Rosenblatt processes. A major difficulty for the control solution is typically finding an explicit optimal control for the controlled stochastic models. The absence of the martingale property for the noise processes presents a major difficulty to determine optimal controls and complicates the stochastic calculus. Nonetheless it is shown here that for some families of optimal controls, which are fairly natural for the noise models considered, optimal controls can be explicitly described. A major motivation for considering these noise models is that often the physical data does not support an assumption of a Brownian motion noise. The data often does not even justify a Gaussian assumption for the noise model.
Fractional Brownian motions are a family of Gaussian processes indexed by the Hurst parameter \(H \in (0,1)\). The process for the value \(H=\frac{1}{2}\) denotes a Brownian motion and for \(H> \frac{1}{2}\) the Gaussian process has a long range dependence which can be described as a slow decay of the covariance function as the time separation for the random variables in the covariance becomes large. These processes were defined by Kolmogorov and the initial empirical identification of these processes was made by H. E. Hurst in his study of rainfall along the Nile River Valley. A Gaussian generalization of these processes is called Gauss-Volterra processes because they are constructed from Brownian motion by a singular integral operator as is the family of fractional Brownian motions. A non-Gaussian example is the family of Rosenblatt processes that were introduced by Roseblatt in [12]. Some control problems with Rosenblatt process noise are solved here. Not only control problems but also stochastic differential games can be explicitly solved [11].
2 Gauss-Volterra Processes
Initially some Gaussian processes are introduced that have a long range dependence. Gauss-Volterra processes are a family of Gaussian processes that includes a number of interesting Gaussian processes such as fractional Brownian motions for \(H > \frac{1}{2}\). These processes are obtained by a Wiener integral with a singular kernel. The definition and some examples are given now.
The process \((b(t), t \ge 0)\) is a centered Gauss-Volterra process, which is described by the covariance. The three conditions on the kernel of the Wiener integral are given now.
-
(K1)
\(K(t,s) = 0\) for \(s > t\), \(K(0,0)=0\), and \(K(t,\cdot ) \in L^2(0,t)\) for each \(t \in {\mathbb {R}}_+.\)
$$\begin{aligned} R(t,s) = {\mathbb {E}} b(t) b(s) := \int ^{\min (t,s)}_0 K(t,r)K(s,r)dr \end{aligned}$$where the kernel \(K:{\mathbb {R}}^2_+ \rightarrow {\mathbb {R}}\) satisfies
-
(K2)
There are positive constants \(C, \beta \) such that for each \(T > 0\)
$$\begin{aligned} \int ^T_0 (K(t,r) - K(s,r))^2 dr \le C |t-s|^{\beta } \end{aligned}$$where \(s,t \in [0,T]\).
-
(K3)
(i) \(K=K(t,s)\) is differentiable in the first variable in \(\{0<s<t<\infty \}\), both K and \(\frac{\partial }{\partial t}K\) are continuous and \(K(s+,s)=0\) for each \(s\in [0,\infty )\)
-
(ii)
\(|\frac{\partial K}{\partial t}(t,s)| \le c_T(t-s)^{\alpha -1} (\frac{t}{s})^{\alpha }\)
-
(iii)
\(\int ^t_0 K(t,u)^2 du \le c_T(t-s)^{1-2\alpha }\)
on the set \(\{0< s< t<T\},\ T<\infty ,\) for some constants \(c_T>0\) and \(\alpha \in (0,\frac{1}{2})\).
For simplicity it is assumed that there is a real-valued Wiener process \((W(t), t\ge 0)\) such that \((b(t), t \ge 0)\) satisfies
To indicate the usefulness of this family of processes, some examples are provided now. The first one is a fractional Brownian motion with the Hurst parameter \(H \in (\frac{1}{2},1)\).
(i) A fractional Brownian motion (FBM) with the Hurst parameter \(H>\frac{1}{2}\). In this case
where \(C_H\) is a constant depending on H. The kernel satisfies conditions (K1)–(K3) with \(\alpha = H-\frac{1}{2} >0\).
(ii) A Liouville fractional Brownian motion (LFBM) for \(H >\frac{1}{2}\), in which case
satisfies (K1)–(K3) with \(\alpha = H - \frac{1}{2}\).
(iii) A multifractional Brownian motion (MBM) [2]. A simplified version analogous to LFBM in (ii) is considered. The kernel \(K:{\mathbb {R}}_+ \times {\mathbb {R}}_+ \rightarrow {\mathbb {R}}_+\) is defined as
where \(H:{\mathbb {R}}_+ \rightarrow [\frac{1}{2},1)\) is the “time-dependent Hurst parameter”. It is assumed that \(H\in C^1({\mathbb {R}}_+)\) and (a) there exists a constant \(\varepsilon \in \big (0,\frac{1}{2}\big )\) such that \(H(t) \in [\frac{1}{2} + \varepsilon ,1)\), \(t\in {\mathbb {R}}_+\)
(b) For each \(t >0\) there is a constant \(C_{\varepsilon ,t}\) such that
It has been shown that the conditions (K1)–(K3) are satisfied in this case (the latter with \(\alpha = \varepsilon \)).
Clearly all of these processes are Gaussian and also have a long range dependence which can be defined by the covariance function.
3 Rosenblatt Processes
Rosenblatt processes are a family of non-Gaussian processes that have a long range dependence and provide a non-Gaussian alternative for modeling noise processes. These processes are defined as follows.
Let \((u)_+ = \max (u,0)\) be the positive part of u and define
for \(H \in (\frac{1}{2},1), u\in \mathbb {R}\) and \(y=(y_1,y_2,\ldots ,y_k)\in \mathbb {R}^k\).
Initially a fractional Brownian motion is given in terms of h.
Definition Let \(H\in (1/2,1)\). The fractional Brownian motion \(B^H=(B^H_t)_{ t\in \mathbb {R}}\) is defined by
for \(t\ge 0\) (and similarly for \(t<0\)) where \(C_H^B\) is a constant such that \(\mathbb {E}(B_1^H)^2=1\) and W is a standard Wiener process (Brownian motion).
Definition Let \(H\in (1/2,1)\). The Rosenblatt process \(R^H=(R^H_t)_{ t\in \mathbb {R}}\) is defined by
for \(t\ge 0\) (and similarly for \(t<0\)) where \(C^R_H\) is a constant such that \(\mathbb {E}(R_1^H)^2=1\) and the integral is the Wiener-Itô multiple integral of order two with respect to the Wiener process W.
The normalizing constants \(C_H^B\) and \(C_H^R\) are defined so that these processes have second moment one at \(t=1\) are given explicitly by
where \(\mathrm {B}\) is the Beta function. For the Itô-type formula [4] used below, it is also convenient to denote
and
where \(\Gamma \) is the Gamma function.
The Rosenblatt processes have a useful stochastic calculus [4] with some prior development in [1, 15].
4 Linear–Quadratic Control with Roseblatt Noise
A major motivation for the use of a Rosenblatt noise model is that empirical evidence provided in [5] shows that for many control systems that have operations in various locations in the world a Gaussian model for the noise is not appropriate. The Rosenblatt processes have continuous sample paths and have a useful stochastic calculus with a stochastic integration and a change of variables formula. Specifically they have a stochastic integration theory that has developed in recent years [1, 4, 15] so that the integrals of a suitable family of functions using a Rosenblatt process as integrator have expectation zero and are Skorokhod integrals. Furthermore there is a change of variables formula for smooth functions of these processes which is explicit though more complicated than the well known formula for a Brownian motion or other continuous martingales [4]. While a suitable family of random functions can be integrated with respect to a Rosenblatt process, these integrals are not martingales though they are Skorokhod integrals and thereby have expectation zero. Fractional Brownian motions are probably the closest family of Gaussian processes to Rosenblatt processes because such a Gaussian process can be described as a singular integral with respect to a single Brownian motion. Rosenblatt processes can be expressed as double singular integrals with respect to a Brownian motion. Since Rosenblatt processes are not martingales the family of controls are restricted to be functions of only the current state because otherwise the controls would be functionals of the past of the observed process. This family of controls has been used to obtain optimal controls for systems with fractional Brownian motions [8] and also Gauss-Volterra processes [7]. An ergodic control problem has been explicitly solved for a scalar system with a Rosenblatt noise [3]. For scalar systems some stochastic differential games have been solved with a Rosenblatt noise where the strategies of the two players are linear transformations of the current state [11]
The problem considered in this paper is the quadratic cost control of an n-dimensional linear stochastic system where the n-dimensional noise is a vector of n real-valued independent Rosenblatt processes with the same parameter H. It seems that no results for the optimal control of multidimensional stochastic equations driven by Rosenblatt processes or other continuous non-Gaussian and non-Markovian processes are available. However a result for ergodic control of a scalar system has been obtained [4] and a two player scalar system with an ergodic payoff has been solved [11]. Since the Rosenblatt processes are not Markov processes, Hamilton–Jacobi–Bellman equations are not applicable as well as other methods for Markov processes. Furthermore a stochastic maximum principle with forward-backward stochastic differential equations is not available. Thus it seems necessary to apply a direct method to determine optimal controls that has been successfully used for linear–quadratic control problems with Brownian motions and fractional Brownian motions e.g. [6, 9], and control with Gauss-Volterra noise [7]. However the extension of this direct method is not immediate or even clear because of the stochastic calculus for Rosenblatt processes. In addition to the stochastic calculus used here for Rosenblatt processes there is another notion of stochastic integrals for Rosenblatt processes that can be considered the analog of Stratonovich integrals for Brownian motion [13].
5 Optimal Control Problem
The control problem considered here is formulated now using an n dimensional stochastic system driven by a Rosenblatt process and having a quadratic cost. The controlled stochastic system satisfies the following linear stochastic equation
where \(X(t) \in {\mathbb {R}}^n, A \in {\mathcal {L}}(\mathbb {R}^n, \mathbb {R}^n), B \in {\mathcal {L}}(\mathbb {R}^n, \mathbb {R}^n) \text { is } B= I, \) \((R_H(t), t \ge 0)\) is a standard n dimensional Rosenblatt process defined as having the same parameter \(H \in \big (\frac{1}{2},1\big )\) for all independent components of the n dimensional Rosenblatt process. It is noted here that the noise components can be correlated and the components can have different H values. These extensions are fairly straightforward. All of the random variables are defined on the complete probability space \((\Omega , {\mathcal {F}}, \mathbb {P})\).
The quadratic cost, \(J_T(U)\), is
where Q and R are symmetric and positive definite linear transformations and \(T>0\) is fixed. There could also be easily a final time cost term.
The family of admissible controls, \({\mathcal {U}}\), is the collection of constant linear feedbacks of the state X, that is,
This family of feedback controls is quite natural from the result for a Brownian motion noise. However allowing the controls to be adapted to the past of the state process in this case would imply functional optimal controls with functional dependence on the past of the state because the control would be predicting the future of the state process as is the case with fractional Brownian motions [9]. Such controls are not easily implementable. Furthermore the approach in this paper has been successful for scalar linear systems driven by a Rosenblatt process and for multidimensional linear and bilinear systems driven by fractional Brownian motions and Gauss-Volterra processes [3, 10].
A change of variables (Itô formula) is used for the optimal control solution that is verified in [4]. The subsequent change of variables formula contains the following two differential operators,
where D is the Malliavin derivative and
and \(\alpha>0, \alpha _1>0, \alpha _2 > 0\) are constants. These operators reflect the singular integral definition of a Rosenblatt process.
6 Optimal Feedback Control
The Riccati equation that is used for some computations here is the one used for a Brownian motion noise so it is not intrinsic for a Rosenblatt noise but it suffices for some computations. It is the following equation.
The solution of the optimal feedback control for the finite time horizon control problem described by (3) and (5) is given in the following theorem.
Theorem 6.1
The stochastic control problem with the stochastic equation (3) and the quadratic cost (5) has an optimal feedback control, \(K^*\), given by the minimum of the following expression which can be obtained by differentiation. The expression is strictly convex in K so the optimal K is determined by the unique zero of the derivative.
where P is the unique solution of the Riccati equation (11) and
Proof
Initially a change of variables formula for Rosenblatt processes is applied to \((\langle P(t)X(t),X(t) \rangle , t \in [0,T]) \) using the result in [4]. This change of variables result is also stated in [3]. While the result in [4] is given for a scalar process, using n linear functionals that form a basis in \(\mathbb {R}^n\), the change of variables is reduced to considering a sum of real-valued processes. The result is the following:
and tr is the trace of a linear transformation and \(B^H\) and \(R^H\) are vectors of n independent fractional Brownian motions and Rosenblatt processes respectively.
The two stochastic integrals in the above equality are Skorokhod integrals [14] so they have expectation zero. It is necessary to compute \(\nabla ^{\frac{H}{2},\frac{H}{2}}X_t(u,u)\). This term is the analog of the second derivative in the change of variables formula for a Brownian motion noise. Initially the process X in the second derivative term is replaced by the Rosenblatt process \(R^H\) to determine \(\nabla ^{\frac{H}{2},\frac{H}{2}}X_t(u,u)\) because X is a linear transformation of \(R^H\). Thus compute \(\nabla ^{\frac{H}{2},\frac{H}{2}}R^H\) where \(R^H\) is an n-vector of independent real-valued Rosenblatt processes each with the same parameter H. It follows from computations in [3] that
where the constant, \({\tilde{C}}_H \), is given by
and B is the Beta function. Note that the integral on the RHS of (16) is an n-vector each element having the same integrand. Let \(\Xi _t(u) = \nabla ^{\frac{H}{2},\frac{H}{2}}X_t(u,u)\) for notational simplicity. Then it follows from the solution of (3) by linearity of the above differential operator that
because the operator \(\nabla ^{\frac{H}{2},\frac{H}{2}}X_t(u,u)\) is symmetric. Solving this integral equation, it follows directly from the linearity of (18) using (16) that
which by an elementary change of variables letting \(t=u=s\) that
Note that the term \(|u-r|^{2H-2}\) in (19) is an n vector which has this same scalar term in all elements. Since \(A+KB + A^T + B^TK^T \) is symmetric, it can be diagonalized. Fix K and diagonalize the linear operator \(A+KB + A^T + K^TB^T \) which is denoted \(\text {diag}(a_1,\ldots , a_n)\).
Substituting the Riccati equation (11) in (15) and taking expectation, the following equation results.
This completes the proof. \(\square \)
It is briefly noted how an ergodic cost functional can be addressed. Initially consider a limit of the inner integral for the last term on the RHS of the above equality, that is,
where \((a_i, i=1\ldots ,n)\) are the eigenvalues of the symmetric transformation \((A+BK +A^T + K^TB^T)\). Clearly averaging of this result as \(\frac{1}{T} \int _o^T\) converges to the same value. Now divide the previous equality by T and let \(T \rightarrow \infty \).
The result in this paper allows for the use of a Rosenblatt noise for an n dimensional linear–quadratic control problem so that the noise can better model the noise observed in control systems for physical systems. It is important to consider the case where the Rosenblatt noise components are correlated and have different H parameters. It is also important to address some computational questions about these control problems with Rosenblatt noise as well as to provide numerical studies to justify Rosenblatt processes as opposed to other non-Gaussian processes. The authors thank the referee for his/her comments that clarified some items in the paper.
References
Arras, B.: From forward integrals to Wick-Itô integrals: the fractional Brownian motion and the Rosenblatt process cases (2016)
Čoupek, P., Maslowski, B.: Stochastic evolution equations with Volterra noise. Stoch. Proc. Appl. 127, 877–900 (2017)
Čoupek, P., Duncan, T.E., Maslowski, B., Pasik-Duncan, B.: An infinite time horizon linear-quadratic control with a Rosenblatt process. In: Proc. IEEE Conf. on Decision and Control, Miami (2018)
Čoupek, P., Duncan, T.E., Pasik-Duncan, B.: A stochastic calculus for Rosenblatt processes. Stoc. Proc. Appl. (2021). https://doi.org/10.1016/j.spa.2020.01.004
Domański, P.D.: Non-Gaussian properties of the real industrial control error in SISO loops. In: 19th International Conference on System Theory, Control and Computing (ICSTCC), pp. 877–882 (2015)
Duncan, T.E., Maslowski, B., Pasik-Duncan, B.: Linear-quadratic control for stochastic equations in a Hilbert space with fractional Brownian motions. SIAM J. Control Optim. 50, 207–231 (2012)
Duncan, T.E., Maslowski, B., Pasik-Duncan, B.: Stochastic linear-quadratic control for bilinear evolution operators driven by Gauss-Volterra processes. Appl. Math. Optim. 80, 369–389 (2019)
Duncan, T.E., Maslowski, B., Pasik-Duncan, B.: Some ergodic control problems for linear stochastic equations in a Hilbert space with fractional Brownian motion. Banach Center Publ. 105, 91–102 (2015)
Duncan, T.E., Pasik-Duncan, B.: Linear-quadratic fractional Gaussian control. SIAM J. Control Optim. 51, 4604–4619 (2013)
Duncan, T.E., Pasik-Duncan, B.: Some linear-quadratic stochastic differential games driven by state dependent Gauss-Volterra processes. In: Yin, G., Zhang, Q. (eds.) IMA Stochastic Control, Computational Methods and Applications, pp. 173–180. Springer, New York (2019)
Duncan, T.E., Pasik-Duncan, B.: A scalar linear-quadratic two player stochastic differential game with a Rosenblatt process noise. In: Proc. 59th IEEE Conf. Decision and Control, S. Korea (2020)
Rosenblatt, M.: Independence and dependence. In: Proceedings of the 4th Berkeley Symposium Mathematical Statistics and Probability, pp. 431–443. University of California Press (1961)
Russo, F., Vallois, P.: Forward, backward and symmetric stochastic integration. Probab. Theory Relat. Fields 97, 403–421 (1993)
Skorokhod, A.V.: On a generalization of a stochastic integral. Theory Probab. Appl. 20, 219–233 (1975)
Tudor, C.A.: Analysis of the Rosenblatt process. ESAIM 12, 230–257 (2008)
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors have no conflict of interest to declare that are relevant to the content of this article.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Research supported by AFOSR Grant FA9550-20-1-0336.
Rights and permissions
About this article
Cite this article
Duncan, T.E., Pasik-Duncan, B. Advances in Noise Modeling for Stochastic Systems in Optimal Control. La Matematica 1, 685–695 (2022). https://doi.org/10.1007/s44007-022-00025-y
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s44007-022-00025-y