Optimal Control for a Linear Quadratic Problem with a Stochastic Time Scale

Abstract

We consider a linear-quadratic control problem where a time parameter evolves according to a stochastic time scale. The stochastic time scale is defined via a stochastic process with continuously differentiable paths. We obtain an optimal infinite-time control law under criteria similar to the long-run averages. Some examples of stochastic time scales from various applications have been examined.

1. INTRODUCTION

In the present paper, we study the problem of a linear-quadratic controller for the case when the change in the time parameter is described by a stochastic process. The procedure of random change of time [1, 2] is widely used in modeling system dynamics and decision making in various fields of applications, see [3,4,5,6,7,8]. In this case, the introduction of a stochastic time scale leads to a control system with time-varying random coefficients. In particular, linear controllers without additive noise and with a random time parameter were previously considered in [9], where the scale was defined as the sum of random variables. In [10], it was assumed that stochastic time scale is associated only with the choice of controls and belongs to the class of subordinators. It should be noted that in the control problems in [9, 10], the minimization of the expected values of cost functionals was considered, and the pathwise optimality (optimization with probability $1$) was not analyzed. The general framework of a linear control system with random coefficients was studied in [11] on a finite horizon. In the general case of nonstationary coefficients, passing to infinite horizon setting turns out to be difficult due to the unboundedness of the cost functionals and the need to study the existence of solutions of backward stochastic differential equations. However, as will be shown in this paper, these difficulties can be avoided when considering a linear control system arising from the assumption of a stochastic time scale. The corresponding optimality criteria used on an infinite time horizon will include both criteria based on expected values (with deterministic normalization) and pathwise criteria with random normalization based on a stochastic time scale process. The article is organized as follows. Section 2 describes the underlying model under study and sets up the problem. Section 3 contains the main result on the form of the optimal control law. Section 4 provides examples of stochastic time scales from various applications as well as an example of a scalar control system.

2. SPECIFICATION OF THE MODEL AND STATEMENT OF THE PROBLEM

2.1. Preliminaries

Assume that on a complete probability space $\{\Omega ,\mathcal {F},\mathbf {P}\}$ with filtration $(\mathcal {F}_t)_{t\geqslant 0}$, we are given a scalar stochastic process $\alpha _t$, $t\geqslant 0$, having continuous and positive paths with probability $1$. Then the stochastic time scale is defined as an almost surely (a.s.) increasing process $\tau _t=\int \nolimits _0^t \alpha _v\thinspace dv$, $t\geqslant 0$, or, in the differential form, as

$$ d\tau _t=\alpha _t\thinspace dt,\quad \tau _0=0. $$

(1)

For $\alpha _t$, $t\geqslant 0$, one can take processes of diffusion type, see [5], or, for example, a random variable $\alpha _t=\bar \alpha >0$ with absolutely continuous distribution and finite moments that determines the scaling factor of the of time scale, see [4]. The process $\tau _t$, $t\geqslant 0$, is referred to as the “internal” time as opposed to physical or real time $t$. The terms “operational,” “business” time, “informational” time scale, “biological” or “molecular clock,” etc. are also used depending on the field of application.

Assumption $\mathcal {A}$. A stochastic process $\alpha _t>0$, $t\geqslant 0$, defining a time scale in (1) has continuous (with probability $1$ and in mean square) sample paths with $\int \nolimits _0^t \alpha _v\thinspace dv\to \infty$ as $t\to \infty$ a.s.

It should be noted that by the monotone convergence theorem, see, e.g., [12, Theorem 1.1, p. 15], the condition $\int \nolimits _0^t \alpha _v\thinspace dv\to \infty$ as $t\to \infty$ a.s. being satisfied also implies $\int \nolimits _0^t E\alpha _v\thinspace dv\to \infty$ as $t\to \infty$. As will be shown below, incorporation of a stochastic time scale $\tau _t$ into the control system, known as stochastic linear quadratic controller, leads to dynamics equations and a cost functional with random coefficients.

2.2. Statement of the Problem

Let $\tilde W_t$, $t\geqslant 0$, be a $d$-dimensional standard Wiener process with respect to $(\mathcal {F}_t)_{t\geqslant 0}$. The evolution of the system state $Y_{\tau }$, $\tau \geqslant 0$, in the internal time $\tau$ is determined using an $n$-dimensional controlled stochastic process with the dynamics

$$ d Y_{\tau }=AY_{\tau }d{\tau }+B\tilde U_{\tau }d{\tau }+Gd\tilde W_{\tau } , \qquad Y_0=x ,$$

(2)

where $x$ is a nonrandom initial state, $\tilde U_{\tau }$ is the $k$-dimensional vector of an admissible control (to be defined below), and $A,B,G\neq 0$ are constant matrices of appropriate dimensions. The assumption of the deterministic initial state has been made because of the subsequent consideration of the control system on an infinite horizon. In the case of nondegenerate disturbances, the contribution of the initial state diminishes over time under an optimal control.

If the value of $T$—the length of planning horizon in real time—is given, then the corresponding $\mathcal {T}(T)=\int \nolimits _0^T \alpha _t\thinspace dt$. For the internal time scale, the cost functional has the form

$$ J_{\mathcal T}(\tilde U)=\int \limits _0^{\mathcal {T}}\left (Y^\mathrm {T}_{\tau }QY_{\tau }+\tilde U^\mathrm {T}_{\tau }R\tilde U_{\tau }\right ) d\tau ,$$

(3)

where $Q\geqslant 0$ and $R>0$ are symmetric matrices, $^\mathrm {T}$ is the transpose sign, and the notation $A\geqslant B$ ($A>B$) for matrices means that the difference $A-B$ is positive semidefinite (positive).

To state the problem, we transform (2)–(3) taking into account (1). It can readily be noticed that $\tilde W_{\tau _t}$ is an $\mathcal {F}_t$-martingale with the quadratic variation of each component equal to $\int \nolimits _0^t \alpha _s\thinspace ds$. Then, according to [13, Lemma 2], there exists a Wiener process $W_t$, $t\geqslant 0$, such that $\tilde W_{\tau }=\int \nolimits _0^t\sqrt {\alpha _s}dW_s$. Assuming that $X_t=Y_{\tau }$, $U_t=\tilde U_{\tau }$, and $J^{(\alpha )}_T(U)=J_{\mathcal T}(\tilde U)$, we obtain the control system with random coefficients

$$ dX_t=\alpha _tAX_tdt+\alpha _tBU_tdt+\sqrt {\alpha _t}GdW_t , \quad X_0=x,$$

(4)

$$ J^{(\alpha )}_T(U)=\int \limits _0^T \alpha _t\left (X^\mathrm {T}_tQX_t+U^\mathrm {T}_tRU_t\right ) dt, $$

(5)

where the admissible controls $U_t$, $t\geqslant 0$, are $\mathcal {\bar F}_t$-adapted processes $\mathcal {\bar F}_t=\sigma \{W_s,\alpha _s, s\leqslant t\}$ such that Eq. (4) has a solution. (Here $\sigma (\cdot )$ denotes a $\sigma$-algebra.) We denote the set of admissible controls by $\mathcal {U}$. Linear systems of the form (4) with random coefficients (in the absence of control actions) were studied earlier in modeling in physics [7], finance [3], and mechanics [8]. Note that in economics and finance, (4) is often used to specify the dynamics of deviations of variables from their equilibrium values, as well as economic indicators that can be of both signs (inflation, return, budget balance, and so on). Obviously, the process $\alpha _t$, $t\geqslant 0$, is by no means always available for direct observation. In economics and finance, there are approaches permitting one to use information about known variables to determine the dynamics of the stochastic time scale. The process $\alpha _t$ is associated with economic (or market) activity, and there are various capturing indicators: trading activity (number of transactions and their volume), volatility of key financial variables and related derivatives, specific indices of economic activity, etc.; see the overview part in [14] as well as [15]. In physics, the $\alpha _t$ describes, for example, the inhomogeneity of a medium, see [16], and makes it possible to observe the corresponding characteristics. Establishing the relationship between specific observables and the process $\alpha _t$ is a separate problem that, when stated with mathematical rigor, leads to more complex models with incomplete information, which are not considered in this paper. In the above situations, an important point is the assumption that the time speed $\alpha _t$ of the stochastic time scale is independent of random disturbances (of the process $W_t$) in the dynamic equation (4). It is also worth noticing that when studying (4) for linear control laws, one can use results on the conditional Gaussian property of $X_t$ with respect to $\mathcal {F}^{(\alpha )}_t=\sigma \{\alpha _s,s\leqslant t\}$ if $\alpha _t$ is a diffusion process and the coefficients of the underlying stochastic differential equations meet some requirements; see [17, Sec. 12] and the example in Sec. 4. As $T\to \infty$, we consider the control problems

$$ \limsup _{T \to \infty }\left ({ \mathrm {E}J^{(\alpha )}_T(U)}\Bigg /{ \mathrm {E}\left (\displaystyle \int \limits _0^T\alpha _t dt\right )}\right )\to \inf _{U\in \mathcal {U}}$$

(6)

and

$$ \limsup _{T \to \infty }\left ({J^{(\alpha )}_T(U)}\Bigg /{\displaystyle \int \limits _0^T\alpha _t dt}\right )\to \inf _{U\in \mathcal {U}} \quad \text {with probability~$1$}.$$

(7)

The solution of problem (7) is understood in the following sense: if $U^{*}$ is an optimal control and $J^{*}=\limsup _{T \to \infty }\Big \{J^{(\alpha )}_T(U^{*}) \big (\int \nolimits _0^T\alpha _t dt\big )^{-1}\Big \}$, then for each admissible control $U{\thinspace \in \thinspace }\mathcal {U}$ one will almost surely have $\limsup _{T \to \infty }\Big \{J^{(\alpha )}_T(U)\big (\int \nolimits _0^T\alpha _t dt\big )^{-1}\Big \}\geqslant J^{*}$. We will see in what follows that, with probability $1$, the value of $J^{*}$ is equal to a constant; i.e., the values of the criterion are compared with a constant for each outcome $\omega \in \Omega$. Here we can also characterize the design procedure for the criterion in problem (6). We use the same principle as the one on which the form of long-run average is based: the normalization of the expected value is selected in accordance with the behavior of $\mathrm {E}J_T(U^{*})$ on the control $U^{*}$ as ${\thinspace T\to \thinspace }\infty$. It is useful to notice that in the internal time (without taking (1) into account) problems (6)–(7) would have the form of control problems with long-run averages $\limsup _{\mathcal {T}{\thinspace \to \thinspace } \infty } \{ \mathrm {E}J_{\mathcal {T}}(U)/\mathcal {T}\} \to \inf _{U\in \mathcal {U}}$ and $\limsup _{\mathcal {T} \to \infty }\{J_{\mathcal {T}}(U)/\mathcal {T}\}{\thinspace \to \thinspace } \inf _{U\in \mathcal {U}}$ a.s. This observation allows us to assume that the existence of a well-known stable feedback control law of the form $U^{*}=-R^{-1}B^\mathrm {T}\Pi X^{*}$ ( $\Pi \geqslant 0$ is a solution of the algebraic Riccati equation) can also be sufficient to derive an optimal strategy in (6)–(7). Suppose that the matrix $Q$ in the functional (3) has the form $Q=C^\mathrm {T}C$, where $C$ is some square matrix. We introduce the following assumption.

Assumption $\mathcal {P}$. The pair of matrices $(A,B)$ is stabilizable; the pair of matrices $(A,C)$ is detectable.

Recall that a pair of matrices $(A,B)$ is said to be stabilizable if there exists a matrix $K$ such that the matrix $A+BK$ is exponentially stable, and detectability is the property dual to stabilizability. More precisely, the detectability for $(A,C)$ implies the stabilizability of $(A^\mathrm {T},C^\mathrm {T})$; see [18, p. 168].

3. MAIN RESULT

In view of Assumption $\mathcal {P}$, there exists a symmetric matrix $\Pi \geqslant 0$ that is a unique positive semidefinite solution of the algebraic Riccati equation

$$ \Pi A+A^\mathrm {T}\Pi -\Pi BR^{-1}B^\mathrm {T}\Pi +Q=0 ,$$

(8)

with the matrix $A-BR^{-1}B^\mathrm {T}\Pi$ being exponentially stable; see [19, Theorem 3.7, p. 275]. Then we can define the control law

$$ U^{*}_t=-R^{-1}B^\mathrm {T}\Pi X^{*}_t, $$

(9)

where the process $X^{*}_t$, $t\geqslant 0$, satisfies the equation

$$ dX^{*}_t=\alpha _t\left (A-BR^{-1}B^\mathrm {T}\Pi \right ) X^{*}_tdt+\sqrt {\alpha _t}GdW_t , \quad X^{*}_0=x . $$

(10)

It will be shown below that a $U^{*}$ of the form (9)–(10) is a solution of problems (6) and (7). Equation (10) is a linear stochastic differential equation (SDE) with random coefficients, and, by virtue of Assumption $\mathcal {A}$, see also [2, Corollary 4.6], its solution exists and can be written in closed form as

$$ X^{*}_t=\Phi (t,0)x+\Phi (t,0)\int \limits _0^t \Phi (0,s)\sqrt {\alpha _s}G dW_s ,$$

(11)

where the matrix $\Phi (t,s)=\exp {\left \{\left (A-BR^{-1}B^\mathrm {T} \Pi \right )\int \nolimits _s^t \alpha _v dv\right \}}$ admits, with probability $1$, the estimate $\|\Phi (t,s)\|\leqslant \kappa _0 \exp {\left (-\kappa \int \nolimits _s^t \alpha _v dv\right )}$, $s\leqslant t$, for some nonrandom constants $\kappa _0,\kappa >0$ (here $\|\cdot \|$ is the Euclidean matrix norm). Several asymptotic properties of the process $X^{*}_t$, $t\to \infty$, that we will need in the sequel are presented in the following lemma.

Lemma.

Let Assumptions $\mathcal {A}$ and $\mathcal {P}$ be true. Then there exist constants $\bar c_1, \bar c_2>0$ such that

$$ \limsup _{t\to \infty }\left ({\mathrm {E}\|X^{*}_t\|^2}\Bigg /{\displaystyle \ln \left ({\mathrm {E}}\int \limits _0^t \alpha _s ds+e\right )}\right )<\bar c_1 , $$

(12)

and, with probability $1$ , the inequality

$$ \limsup _{t\to \infty }\left ({\|X^{*}_t\|^2}\Bigg /{\ln \left (\displaystyle \int \limits _0^t \alpha _s ds+e\right )}\right )<\bar c_2 $$

(13)

holds, where $e$ is the base of the natural logarithm.

The proofs of the Lemma and the Theorem are given in the Appendix.

It should be noted that deriving (4)–(5) from (2)–(3) with a deterministic change of time allows for the straightforward use of well-known results on optimal control for time-invariant systems, see [20]. The considered case of a stochastic time scale requires a separate analysis. The results of this analysis are stated in the following assertion.

Theorem.

Let Assumptions $\mathcal {A}$ and $\mathcal {P}$ be satisfied. Then the control law $U^{*}$ defined in (9)–(10) is a solution of problems (6) and (7). Furthermore,

$$ \lim _{T\to \infty }\left ({ \mathrm {E}J^{(\alpha )}_{{T}}(U^{*})}\Bigg /{{\mathrm {E}}\left (\displaystyle \int \limits _0^T\alpha _t dt\right )}\right )= \lim _{T \to \infty }\left ({ J^{(\alpha )}_{ T}(U^{*})}\Bigg /{\displaystyle \int \limits _0^T\alpha _t dt}\right )=\mathrm {tr}\thinspace (G^\mathrm {T}\Pi G) \quad \text {a.s.}$$

(where $\mathrm {tr}\thinspace (\cdot )$ is the notation for the trace of a matrix).

Remark 1.

The condition $\alpha _t>0$ a.s., $t\geqslant 0$, has been necessary to switch from system (1)–(3) to system (4)–(5) by incorporating the time scale into analysis. If the processes (4)–(5) are already given, then we can introduce the weaker condition $\alpha _t\geqslant 0$, $t\geqslant 0$, into Assumption $\mathcal {A}$.

Remark 2.

In the case of a deterministic system evolving in the internal time, i.e., when $G=0$ in (2), the control law $U^{*}$ will be a solution of the problems

$$ {\limsup _{T \to \infty } \mathrm {E}J^{(\alpha )}_T(U){\;\to \;}\inf _{U\in \mathcal {U}} }\quad \text {and}\quad {\limsup _{T \to \infty }J^{(\alpha )}_T(U)\to \to \inf _{U\in \mathcal {U}} }\quad \text {a.s.}$$

In this case, $\lim _{T \to \infty } \mathrm {E}J^{(\alpha )}_T(U^{*})=\lim _{T \to \infty }J^{(\alpha )}_T(U^{*})=x^\mathrm {T}\Pi x$.

Remark 3.

Along with problems (6)–(7), one can also consider the problem

$${\limsup _{T \to \infty } \mathrm {E}\left \{J^{(\alpha )}_T(U)\left (\int _0^T\alpha _t dt\right )^{-1}\right \}\to \inf _{U\in \mathcal {U}}},$$

in which the criterion has been obtained from the long-run average known for a deterministic system under constant nonrandom perturbations. By analogy with what has been proved in the Theorem, a control law $U^{*}$ of the form (9)–(10) will be a solution, and the value of the criterion in this case will also be equal to $\mathrm {tr}\thinspace (G^\mathrm {T}\Pi G)$.

4. EXAMPLES OF STOCHASTIC TIME SCALES AND A SCALAR CONTROL SYSTEM

We used stochastic normalization in the control problem (7); however, in many examples, it proves possible to replace random normalization with a deterministic function. In applications, when describing the process $\alpha _t$, $t\geqslant 0$, the requirement of “comparability” of the time scale $\mathcal {T}(T)=\int \nolimits _0^T \alpha _t dt$ and the actual planning horizon $T$ as $T\to \infty$ is often introduced (see, e.g., [21, Theorem 6.1, p. 174]); i.e., $\mathcal {T}(T)/T\to \mathrm {const}$ a.s. In the following remark we describe the possibility of transition to nonrandom normalizations and the form of the corresponding control problems.

Remark 4.

• 1. Suppose that for a stochastic process $\alpha _t$, $t\geqslant 0$, we have

  $$ \limsup _{T\to \infty }\left \{\int _0^T\alpha _t dt/\Gamma ^{(+)}_T\right \}=c^{(+)}>0\quad \text {or}\quad {\liminf _{T\to \infty }\left \{\int _0^T\alpha _t dt/\Gamma ^{(-)}_T\right \}=c^{(-)}>0}$$

  with probability $1$; ${\Gamma ^{(+)}_T},$ $\Gamma ^{(-)}_T$ are positive deterministic functions, and $c^{(+)}$ and $c^{(-)}$ are constants. Then, instead of (7), we can consider the problems

  $$ \limsup _{T \to \infty }\cfrac {J^{(\alpha )}_T(U)}{\Gamma ^{(+)}_T}\to \inf _{U\in \mathcal {U}} \quad \text {or}\quad \liminf _{T \to \infty }\cfrac {J^{(\alpha )}_T(U)}{\Gamma ^{(-)}_T}\to \inf _{U\in \mathcal {U}} .$$

  In this case, the values of the criteria on the optimal control $U^{*}$ will be, respectively,

  $$ {\limsup \limits _{T \to \infty }\left \{\frac {J^{(\alpha )}_T(U)}{\Gamma ^{(+)}_T}\right \}=c^{(+)}\mathrm {tr}\thinspace (G^\mathrm {T}\Pi G)}\quad \text {and}\quad {\liminf _{T \to \infty }\left \{\frac {J^{(\alpha )}_T(U)}{\Gamma ^{(-)}_T}\right \}=c^{(-)}\mathrm {tr}\thinspace (G^\mathrm {T}\Pi G)}. $$

  If the process $\alpha _t$ is ergodic, i.e., if

  $$ {\lim _{T\to \infty }\left \{T^{-1}\int \limits _0^T\alpha _t dt\right \}=\lim _{T\to \infty } \left \{T^{-1}\int \limits _0^T \mathrm {E}\alpha _t dt\right \}}\quad \text {a.s.}, $$

  then the criteria in problems (6)–(7) become the long-run averages.

• 2. Let $T^{-1}\int \nolimits _0^T\alpha _t\to \bar \alpha$ a.s., and let $T^{-1}\int \nolimits _0^T \mathrm {E}\alpha _t\to \mathrm {E}\bar \alpha$, $T\to \infty$, where $\bar \alpha >0$ is some random variable. Then (6)–(7) are replaced by problems with criteria given by long-run averages,

  $$ \limsup _{T \to \infty }\cfrac { \mathrm {E}J^{(\alpha )}_T(U)}{T}\to \inf _{U\in \mathcal {U}} \quad \text {and}\quad \limsup _{T \to \infty }\cfrac {J^{(\alpha )}_T(U)}{T}\to \inf _{U\in \mathcal {U}};$$

  however, here

  $$ {\lim _{T \to \infty } \left \{T^{-1} \mathrm {E}J^{(\alpha )}_T(U^{*}) \right \}=( \mathrm {E}\bar \alpha )\mathrm {tr}\thinspace (G^\mathrm {T}\Pi G)}\enspace \text {and}\enspace {\lim _{T \to \infty } \left \{ T^{-1}J^{(\alpha )}_T(U^{*}) \right \}=\bar \alpha \mathrm {tr}\thinspace (G^\mathrm {T}\Pi G)};$$

  i.e., the deterministic normalization leads to a difference between the values of the two criteria on $U^{*}$; one of the long-run averages will be a random variable.

In all the examples considered below, by $\bar W_t,$ $t\geqslant 0$, we denote a scalar Wiener process.

Example 1.

In financial and physical applications (see [3, 7]), the so-called CIR-process (Cox–Ingersoll–Ross process) is often used as the change of time. This model admits a generalization to the case of time-varying coefficients in the equation. Let $\alpha _t=\xi _t$, where $\xi _t$, $t\geqslant 0$, is given by the equation

$$ d\xi _t=\mu \rho _t(\theta -\xi _t)dt+\sigma \sqrt {\rho _t}\sqrt {\xi _t}d\bar W_t,\quad \xi _0=\bar \xi >0 , $$

(14)

with constants $\mu ,\theta ,\sigma >0$ and $2\mu \theta \geqslant \sigma ^2$, where the deterministic monotone function $\rho _t>0$, $t\geqslant 0$, is such that $\int \nolimits _0^t \rho _s ds\to \infty$, $t\to \infty$. It is easy to notice, see, e.g., [22, Theorem 8.5.7, p. 190], that $\xi _t=\tilde \xi _{\nu _t}$, where $\nu _t=\int \nolimits _0^t \rho _s ds$, and the process $\tilde \xi _\nu$ is a standard CIR-process with constant parameters, i.e., a solution of the equation $d\tilde \xi _{\nu }=\mu (\theta -\tilde \xi _{\nu })d\nu +\sigma \sqrt {\xi _{\nu }}d\tilde W_{\nu }$, $\tilde \xi _0=\bar \xi$, where $\tilde W_{\nu }$ is some Wiener process. Then the condition $2\mu \theta \geqslant \sigma ^2$ implies that $\tilde \xi _{\nu }>0$ a.s., $\nu \geqslant 0$ (see, e.g., [23, Sec. 6.3.1, p. 357]), and consequently, $\xi _t>0$ with probability $1$, $t\geqslant 0$. Thus, the stochastic time scale process $\tau _t=\int \nolimits _0^t \xi _s ds=\int \nolimits _0^t \tilde \xi _{\nu _s}ds$ is given by a double change of time. Since the statistical characteristics of $\tilde \xi _{\nu }$, $\nu \geqslant 0$, are well known, see, e.g., [23, Sec. 6.3.3], we can use a change of time to determine that $\mathrm {E}\xi _t\to \theta$ and $\mathrm {E}(\xi _t- \mathrm {E}\xi _t)^2 \to \theta \sigma ^2(2\mu )^{-1}$ as $t\to \infty$. In this case, for the covariance function $K(t,s)= \mathrm {E}(\xi _t\xi _s)- \mathrm {E}\xi _t \mathrm {E}\xi _s$ one has the estimate

$${\big \|K(t,s)\big \|\leqslant c_{\xi }\left (\exp {\left \{-\mu \int \limits _0^{\tilde t}\rho _v dv\right \}}+\exp {\left \{-\mu \int \limits _{\tilde s}^{\tilde t}\rho _v dv\right \}}\right )},$$

where $c_{\xi }>0$ is some constant and the variables are $\tilde t=\max (t,s)$ and $\tilde s=\min (t,s)$. To study the behavior of the normalization $\mathcal {T}(T)$, $T\to \infty$, we use [24, Theorem A, p. 154], according to which, for the process to be ergodic, it suffices to have the estimate

$$ {\chi _T=\int \limits _0^T\int \limits _0^T \big \|K(t,s)\big \| ds dt\leqslant \bar c T^{\gamma }} $$

for some constants $0\leqslant \gamma <2$ and $\bar c>0$. It can readily be noticed that $\chi _T\leqslant \bar c T$ for a nondecreasing $\rho _t$ for large $T$, because

$$ {\big \|K(t,s)\big \|\leqslant c_{\xi }\left (\exp {\left \{-\mu \rho _0\tilde t\right \}}+\exp {\left \{-\mu \rho _0\left (\tilde t-\tilde s\right )\right \}}\right )}. $$

If $\rho _t\to 0$ as $t\to \infty$ under the constraint $\rho _tt^{\beta }\to \infty$ as $t\to \infty$ for some $\beta <1$, then we can take the constant $\gamma =\beta +1$, because in this case the limit (being found by l’Hôpital’s rule) is equal to $\lim \limits _{T\to \infty }\{\chi _T/T^{\gamma }\} =\lim \limits _{T\to \infty }\{1/(\rho _TT^{\gamma -1})\}=0$. Consequently, $\left (\int \nolimits _0^T \xi _t dt-\int \nolimits _0^T \mathrm {E}\xi _t dt\right )T^{-1}\to 0$ a.s. as $T\to \infty$, and the normalizations of the criteria in problems (6)–(7) will be equal to $T$ (see also item 1 in Remark 4).

Example 2.

Freris et al. [25] proposed a network model of “clock” with a time change rate characterized by an exponential Ornstein–Uhlenbeck process. More precisely, $\alpha _t =\lambda _t \exp {(\xi _t)}$, where $\xi _t = \sigma \exp (-at)\int \nolimits _0^t \exp (at)d\bar W_t$ with a constant $a > 0$ and $\lambda _t = (\exp ( \mathrm {E}\xi ^2_t/2))^{-1}$; i.e., $\mathrm {E}\alpha _t=1$ by virtue of the lognormal distribution for $\exp (\xi _t)$, $t\!\geqslant \!0$. Accordingly, $\lim _{T\to \infty }{T^{-1}\int \nolimits _0^T \mathrm {E}\alpha _t dt}\!=\!1$. Then the process $\mathcal {Y}_t=\alpha _t- \mathrm {E}\alpha _t$ is considered under a pathwise analysis of the time scale; the covariance function $K(t,s)=\exp {\{\rho (t,s)\}}-1$ of this process is determined, where $\rho (t,s)= \mathrm {E}\xi ^2_{\min (t,s)}\exp {\{-a|t-s|\}}$; and the estimate $\chi _T=\int \nolimits _0^T\int \nolimits _0^T K(t,s) ds dt\leqslant \bar c T$ with some constant $\bar c>0$ is established. Then, see [24, Theorem A, p. 154], $\lim _{T\to \infty }(\mathcal {Y}_T/T)=0$ a.s., and as a consequence, we have the relation $\lim _{T\to \infty }{T^{-1}\int \nolimits _0^T \alpha _t dt}=1$. In this case, the criteria in problems (6)–(7) have the form of long-run averages.

Example 3.

When assessing financial instruments, Xia [26] used the “business time” $\tau _t=\lambda _1 t+\lambda _2\int \nolimits _0^t \bar W^2_s ds$ ( $\lambda _1,\lambda _2>0$ are constants). Here $\alpha _t=\lambda _1+\lambda _2\bar W^2_t$ and it is known, see, e.g., [27], that

$$ {\liminf \limits _{T\to \infty }\left \{\left (\Gamma ^{(-)}_T\right )^{-1}\int \limits _0^T \bar W^2_t dt\right \}=1/8, } \quad {\limsup \limits _{T\to \infty }\left \{\left (\Gamma ^{(+)}_T\right )^{-1}\int \limits _0^T \bar W^2_t dt\right \}=8/\pi ^2 } $$

for the functions $\Gamma ^{(-)}_T=T^2(\ln \ln T)^{-1}$ and $\Gamma ^{(+)}_T=T^2\ln \ln T$ and also that $\mathrm {E}\bar W^2_t=t$. Thus, the time scale in this example does not possess the ergodic property. Consequently, when switching from the random normalization to the deterministic one, instead of (7), we can consider two problems with different criteria including the normalizing functions $\Gamma ^{(-)}_T$ and $\Gamma ^{(+)}_T$; see item 1 in Remark 4.

Example 4.

Let $\alpha _t=\int \nolimits _0^t \exp {(-as+\sigma \bar W_s)} ds$, where the constant $a>0$. Since $a>0$, we have $\alpha _t \to \bar \alpha$ as $t\to \infty$ with probability $1$, where the random variable $\bar \alpha$ has the inverse gamma-distribution; see [28]. The finiteness of $\mathrm {E}\bar \alpha$ is ensured under the condition $\sigma ^2/2-a<0$, and $\mathrm {E}\bar \alpha ^2<\infty$ for $\sigma ^2-a<0$. Then, according to item 2 in Remark 4, the control problems (6) and (7) can be replaced by problems with criteria given by the long-run averages. For $\sigma ^2/2-a \geqslant 0$, one has $T^{-1}\int \nolimits _0^T \mathrm {E}\alpha _t\to \infty$ as $T\to \infty$, and one needs a normalization in (6) that grows faster than $T$: power-law normalization $T^2$ for the case of $a=\sigma ^2/2$ or exponential normalization $\exp {\{(\sigma ^2/2-a)t\}}$ for $a<\sigma ^2/2$. It should be noted that the pathwise long-run average instead of the criterion in (7) is preserved in this case.

The results obtained earlier are illustrated by the example of a scalar control system. Various properties of the process under an optimal strategy are also determined.

Example 5.

Consider the model of control of the velocity of a particle in a inhomogeneous medium, for example, in the field of cell biology. We start from the dynamics equation in [29] with a “diffusion diffusivity,” which alters the time scale in the velocity equation (see the introduction in [16]) and is modeled using the CIR-process (14) with constant parameters, where the impulse towards a cell membrane in [30] can serve as an example of a factor with such a dynamics. A dynamic equation of the form $dX_t=\xi _tU_t dt+G\sqrt {\xi _t}dW_t$, $X_0=x$, and the cost functional $J^{(\alpha )}_T(U)=\int \nolimits _0^T(X^2_t+U^2_t) dt$ correspond to (4)–(5) with the coefficients $A=0$, $B=1$, $Q=R=1$, and $\alpha _t=\xi _t$. The processes $\bar W_t$ in (14) and $W_t$ are assumed to be independent, $t\geqslant 0$. It follows from the results in the Theorem and Example 1 that the control law $U^{*}_t=-X^{*}_t$ is optimal by the criteria of long-run averages. Using the conditional Gaussian property of the process $X^{*}_t$, we can write the expressions

$$ { \mathrm {E}X^{*}_t = \mathrm {E} \left (\exp {\left \{- \int \limits _0^t \xi _v dv\right \}}\right )x}\quad \text {and}\quad { \mathrm {E}(X^{*}_t)^2 = \mathrm {E} \left (\exp {\left \{-2 \int \limits _0^t \xi _v dv\right \}}\right )x^2 + G^2/2}; $$

see [17, Theorem 12.1]. It is well known, see, e.g., [23, Corollary 6.3.4.2], that for $\lambda >0$ one has

$$ { \mathrm {E}\left (\exp {\left \{-\lambda \int \limits _0^t \xi _v dv\right \}}\right )\sim \exp {\left \{-\theta \mu \sigma ^{-2} \left (\sqrt {\mu ^2+2\lambda \sigma ^2}-\mu \right )t\right \}}}; $$

i.e., for two moments of the process $X^{*}_t$ one has the exponential rate of convergence to constant values (zero and $G^2/2$). By virtue of the ergodicity of the process $\xi _t$, it follows from the Lemma that the paths $X^{*}_t$ are a.s. majorized by a function proportional to $\sqrt {\ln t}$ as $t\to \infty$.

5. CONCLUSIONS

In the present paper, we have studied the linear control system (2) with the quadratic cost functional (3) over an infinite time horizon under the assumption of the stochastic nature of the time scale (1) (see also Assumption $\mathcal {A}$). The incorporation of (1) into the analysis leads to system (4)–(5) with random coefficients, for which the control problems (6) and (7) are stated with the criteria serving as analogs of long-run averages. It is shown that in this case, the optimal control strategy can be chosen in the form of the well-known linear law $U^{*}$ (see (9)–(10) and the assertion in the Theorem). It should be noted that, unlike the problems of synthesis of stochastic linear controllers with deterministic coefficients (see, e.g., [20, 31]), the normalizations in the criteria for the two control problems (6) and (7) for system (4)–(5) are different. In the general case, the ergodicity of the time scale process $\tau _t=\int \nolimits _0^t \alpha _s ds$ does not take place, and a significant difference can be observed in the orders of growth of the cost functional $J_T(U^{*})$ and its expectation $\mathrm {E}J_T(U^{*})$ on the optimal control $U^{*}$ (see Examples 3 and 4). As can be seen, switching to a stochastic time scale in linear control systems with constant coefficients preserves the key properties of stabilizability/detectability, stability and, as a consequence, the infinite horizon optimality of the linear feedback control law. This remark allows the suggestion that the form of the optimal strategy may also prove invariant under random change of time in other optimal control problems for linear systems, in particular, when using the so-called “risk-sensitive” cost functional $\exp {(\theta J_{\mathcal {T}}(\tilde U))}$ (where $J_{\mathcal {T}}(\tilde U)$ is given in (3) and $\theta$ is a constant). For the direction of further research we can indicate the analysis of the situation of nonmonotone stochastic time scale encountered in applications for models in the fields of statistics, metrology, and computer science.

APPENDIX

Proof of the Lemma. First, the assertions in the Lemma are proved for the case of Eq. (10) with $A-BR^{-1}B^\mathrm {T}\Pi =-\kappa I$; $\kappa >0$ is a constant, and $I$ is the identity matrix. Then it is shown that the underlying properties of the process $X^{*}_t$, $t\to \infty$, in Eq. (10) do not change for an arbitrary exponentially stable matrix $A-BR^{-1}B^\mathrm {T}\Pi$. Consider the process $X^{*}_t=\hat X_t$, $t\geqslant 0$, with the dynamics

$$ d\hat X_t=-\kappa \alpha _tI \hat X_t+\sqrt {\alpha _t}GdW_t , \quad \hat X_0=0 . $$

Denoting the $i$th component of the process $\hat X_t$, $i=1,\dots ,n$, by $\hat X_{it}$, one can readily obtain the representation $\hat X_{it}=c_{i}M_{it}\left (\left \langle M_{it}\right \rangle +1\right )^{-1/2}$, where the martingale

$$ {M_{it}={c_{i}}^{-1}\int \limits _0^t \sqrt {\alpha _s}\exp {\left \{\int \limits _0^s \kappa \alpha _v dv\right \}}\left (\thinspace \sum _{j=1}^{d} G_{ij}dW_{js}\right )} $$

has the quadratic variation $\langle M_{it}\rangle =\exp {\big \{2\kappa \int \nolimits _0^t \alpha _v dv\big \}}-1$; further, $c_{i}=\big ((2\kappa )^{-1}\sum _{j=1}^{d}G^2_{ij}\big )^{1/2}$, where the $G_{ij}$ are the entries of the matrix $G$ ($i=1,\dots ,n$, $j=1,\dots ,d$); $W_{jt}$ is the $j$th component of the Wiener process $W_t$ ($j=1,\dots ,d$). It was established in [32, Lemma 2.3] that

$$ {\Big \|M_{it}\big (\langle M_{it}\rangle +1\big )^{-1/2}\Big \|\leqslant N(\omega )\sqrt {\ln \ln \big (\langle M_{it}\rangle +e^{e}\big )}},$$

where $N(\omega )\geqslant 0$ is a.s. a finite random variable. Therefore, for the process $\|\hat X_t\|^2$ with probability $1$ one has the relation

$$ \|\hat X_t\|^2\leqslant cN^2(\omega )\ln \left (\int \limits _0^t \alpha _v dv+e\right ) .$$

(A.1)

Then it follows from the estimate (A.1) and the Jensen inequality that

$$ \mathrm {E}\|\hat X_t\|^2\leqslant \tilde c\ln \left (\mathrm {E}\int \limits _0^t\alpha _v dv+e\right ) , $$

(A.2)

where $c$ and $\tilde c$ in (A.1) and (A.2) denote some positive constants whose particular values are inessential and can vary from formula to formula. It can readily be noticed that relation (13) is an obvious consequence of the above representation for the components of $\hat X_t$ and of the law of iterated logarithm for martingales; see, e.g., [33]. Further, we introduce the process $Z_t=X^{*}_t-\hat X_t$ with the dynamic equation

$$ dZ_t=\alpha _t\left (A-BR^{-1}B^\mathrm {T}\Pi \right ) Z_tdt+\alpha _t\left (A-BR^{-1}B^\mathrm {T}\Pi +\kappa I\right )\hat X_t , \quad Z_0=x,$$

which has the solution $Z_t=\Phi (t,0)x+\int \nolimits _0^t\Phi (t,s) \alpha _s(A-BR^{-1}B^{\mathrm {T}}\Pi +\kappa I)\hat X_s ds$. In view of the upper bound for $\|\Phi (t,s)\|$ (see the comment on (11)) and the Cauchy–Schwarz inequality for the process $Z_t$, we have the estimate

$$ \begin {aligned} \|Z_t\|^2&\leqslant 2\kappa _0^2{ \exp {\left \{-2\kappa \int \limits _0^t \alpha _v dv\right \}}}\|x\|^2 \\ &\qquad {}+c\thinspace { \exp {\left \{-\kappa \int \limits _0^t \alpha _v dv\right \}}}\int \limits _0^t \alpha _s{\thinspace \exp {\left \{\kappa \int \limits _0^s \alpha _v dv\right \}}\|\hat X_s\|^2 ds}. \end {aligned} $$

(A.3)

Applying (A.1) to (A.3) gives the relation

$$ \|Z_t\|^2\leqslant 2\kappa _0^2{\thinspace \exp {\left \{-2\kappa \int \limits _0^t \alpha _v dv\right \}}}\|x\|^2+cN^2(\omega ) \ln \left (\int \limits _0^t \alpha _v dv+e\right ).$$

Taking the expectation on both sides in this relation combined with the Jensen inequality leads to the estimate $\mathrm {E}\|Z_t\|^2\leqslant \tilde c+\tilde c \ln \left (\mathrm {E}\int \nolimits _0^t \alpha _v dv+e\right )$; this then implies (12) in the lemma being proved. Inequality (13) is also easy to obtain in a well-known way; see, e.g., the proof in [31, Theorem 2], if one notices that $h_t=\ln \left (\int \nolimits _0^t \alpha _v dv+e\right )$ is a nondecreasing function. Then dividing (A.3) by $h_t$ in the subsequent estimation of the integral on the right-hand side (while using the result for $\|\hat X_t\|^2$) gives a bounded limit. The proof of the lemma is complete.$\quad \blacksquare$

Proof of the Theorem. For $U\in \mathcal {U}$, we write a representation for the difference of cost functionals,

$$ J^{(\alpha )}_T(U^{*})-J^{(\alpha )}_T(U)=2x^\mathrm {T}_T\Pi X^{*}_T-\int \limits _0^T \alpha _t\left (x^\mathrm {T}_tQx_t+u^\mathrm {T}_tRu_t\right )dt-2\int \limits _0^T \sqrt {\alpha _t}x^\mathrm {T}_t\Pi GdW_t , $$

(A.4)

where the variables are $x_t=X^{*}_t-X_t$ and $u_t=U^{*}_t-U_t$ with $dx_t=\alpha _t A x_t dt+\alpha _t B u_t dt$, $x_0=0$. Since the pair $(A,C)$ is observable, it follows that there exists a matrix $F$ such that the matrix $A+FC$ is exponentially stable. Then

$$ {\|x_t\|\leqslant c \exp {\left \{-\bar \kappa \int \limits _0^t \alpha _v dv\right \}}\int \limits _0^t \exp {\left \{\bar \kappa \int \limits _0^{s} \alpha _v dv\right \}}\alpha _s\big (\|Cx_s\|+\|u_s\|\big )ds}$$

for some constant $\bar \kappa >0$. After squaring and applying the Cauchy–Schwarz inequality as well as the conditions $Q=C^{\mathrm {T}}C$, $R>0$, we have

$$ {\|x_t\|^2\leqslant \tilde c \exp {\left \{-\bar \kappa \int \limits _0^t \alpha _v dv\right \}}\int \limits _0^t \exp {\left \{\bar \kappa \int \limits _0^{s} \alpha _v dv\right \}}\alpha _s\left (x^\mathrm {T}_sQx_s+u^{\mathrm {T}}_sRu_s\right )ds}. $$

Further, using integration by parts, we show that

$$ {\int \limits _0^T \|x_t\|^2 ds\leqslant \tilde c\bar \kappa ^{-1} \int \limits _0^T\alpha _t\left (x^\mathrm {T}_tQx_t+u^{\mathrm {T}}_tRu_t\right )dt}. $$

Accordingly, we obtain the estimate

$$ {\|x_T\|^2+\int \limits _0^T \|x_s\|^2 ds\leqslant c_0\int \limits _0^T \alpha _t\left (x^\mathrm {T}_tQx_t+u^\mathrm {T}_tRu_t\right )dt}$$

with $T>0$ and some constant $c_0>0$. Then, considering the elementary inequality $2ab\leqslant a^2\bar c+b^2/\bar c$, which holds for any numbers $a,b$ and $\bar c>0$, the expression on the right-hand side of (A.4) is estimated in the form

$$ J^{(\alpha )}_T(U^{*})-J^{(\alpha )}_T(U)\leqslant c_1 \|X^{*}_T\|^2-c_2\int \limits _0^T \alpha _t\|x_t\|^2 dt-2\int \limits _0^T \sqrt {\alpha _t}x^\mathrm {T}_t\Pi GdW_t ,$$

(A.5)

where $c_1,c_2>0$ are some constants. After taking the expectation on both sides in (A.5) and dividing by ${\mathrm {E}}(\int \nolimits _0^T \alpha _t dt)$, in the limit as $T\to \infty$ we use the result (A.2) in the Lemma; this leads to the relation

$$ \limsup _{T \to \infty }\left ({ \mathrm {E}J^{(\alpha )}_T(U^{*})}\Bigg /{ \mathrm {E}\left (\displaystyle \int \limits _0^T\alpha _t dt\right )}\right )\leqslant \limsup _{T \to \infty }\left ({ \mathrm {E}J^{(\alpha )}_T(U)}\Bigg /{ \mathrm {E}\left (\displaystyle \int \limits _0^T\alpha _t dt\right )}\right ).$$

In the pathwise analysis of (A.5), introducing the notation

$$ {M_T=-2\int \limits _0^T \sqrt {\alpha _t}x^\mathrm {T}_t\Pi GdW_t, } $$

we write an estimate for (A.5) in the form

$$ J^{(\alpha )}_T(U^{*})\leqslant J^{(\alpha )}_T(U)+\mathcal {R_T} ,$$

where $\mathcal {R}_T=-c_3\langle M_T\rangle +M_T$ for some constant $c_3>0$; $\langle M_T\rangle =\int \nolimits _0^T \|\sqrt {\alpha _t}G^{\mathrm {T}}\Pi x_t\|^2 dt$ is the quadratic variation of $M_T$. Note that $\limsup _{T\to \infty }g_T\mathcal {R}_T\leqslant 0$ a.s. for any monotone function $g_T$ with the property $g_T>0$ and $g_T\to \infty$ as $T\to \infty$ (see [34]); in particular, $g_T=\left (\int \nolimits _0^T \alpha _t dt\right )^{-1}$ (see also Assumption $\mathcal {A}$). It is also obvious that (13) implies $\|X^{*}_T\|^2\left (\int \nolimits _0^T \alpha _t dt\right )^{-1} \to 0$ a.s. as $T\to \infty$. Therefore, with probability $1$ we have the relation

$$ \limsup _{T \to \infty }\left ({J^{(\alpha )}_T(U^{*})}\Bigg /{\left (\int \limits _0^T\alpha _t dt\right )}\right )\leqslant \limsup _{T \to \infty }\left ({J^{(\alpha )}_T(U)}\Bigg /{\left (\int \limits _0^T\alpha _t dt\right )}\right ). $$

By the Itô formula, in a standard manner we establish that

$$ {J^{(\alpha )}_T(U^{*})=x^\mathrm {T}\Pi x-(X^{*}_T)^{\mathrm {T}}\Pi X^{*}_T+\mathrm {tr}\thinspace (G^\mathrm {T}\Pi G)\int \limits _0^T \alpha _t dt+2\int \limits _0^T \sqrt {\alpha _t}(X^{*}_t)^\mathrm {T}\Pi GdW_t}. $$

Then the value of the criterion in (6) for $U^{*}$ is equal to

$$ {\lim _{T\to \infty }\left \{ \mathrm {E}J^{(\alpha )}_T(U^{*}) \mathrm {E}\left (\int \limits _0^T\alpha _t dt\right )^{-1}\right \}=\mathrm {tr}\thinspace (G^\mathrm {T}\Pi G)}.$$

Applying the iterated logarithm law to the martingale $\mathcal {M}_T=\int \nolimits _0^T \sqrt {\alpha _t}(X^{*}_t)^{\mathrm {T}}\Pi GdW_t$ yields the estimate $\|\mathcal {M}_T\|\leqslant c \sqrt {\langle \mathcal {M}_T\rangle \ln \ln \langle \mathcal {M}_T\rangle }$ for large $T$, and the use of (13) allows one to pass to the ineqaulity $\langle \mathcal {M}_T\rangle \leqslant \tilde c \left ( \int \nolimits _0^T \alpha _t dt \right )\ln \left ( \int \nolimits _0^T \alpha _t dt\right )$. Consequently, $\|\mathcal {M}_T\| \left (\int \nolimits _0^T \alpha _t dt\right )^{-1} \to 0$ as $T\to \infty$ a.s.; then

$$ {\lim _{T\to \infty }\left \{J^{(\alpha )}_T(U^{*})\left (\int \limits _0^T\alpha _t dt\right )^{-1}\right \}=\mathrm {tr}\thinspace (G^\mathrm {T}\Pi G) }$$

with probability $1$. The proof of the theorem is complete. $\quad \blacksquare$