On the Relation Between Turnpike Properties for Finite and Infinite Horizon Optimal Control Problems

Abstract

We show that, under appropriate regularity conditions, a finite horizon optimal control problem exhibits the turnpike property, if and only if its infinite horizon counterpart does. We prove the result for both undiscounted and discounted problems and also provide a version which incorporates quantitative information about the convergence rates.

1 Introduction

The turnpike phenomenon is a property of trajectories of optimally controlled systems that has long been observed in optimal control, even back to early work by von Neumann [1]. The turnpike property describes the fact that an optimal trajectory “most of the time” stays close to an equilibrium point, as illustrated in Fig. 1, for finite horizon optimal trajectories. This property attracted significant interest, particularly in the field of mathematical economics [2], because it directly leads to the concept of optimal economic equilibria and thus provides a natural economic interpretation of optimality. The name “turnpike property” was coined in 1958 in the book by Dorfman et al. [3], who compared the phenomenon to the optimal way of driving by car from a point A to a point B using a turnpike or highway, which consists of three phases: driving to the highway (i.e., approaching the equilibrium), driving on the highway (i.e., staying near the equilibrium) and leaving the highway (i.e., moving away from the equilibrium).

Recently, the turnpike property has also attracted interest in areas different from mathematical economics; see, e.g., [4,5,6,7,8,9]. This interest stems from the fact that this property considerably simplifies the computation of (approximately) optimal trajectories in all areas of optimal control, either directly by constructive synthesis techniques, as in [10], or indirectly via a receding horizon approach, as in economic model predictive control [11, 12].¹ Moreover, the turnpike property can also be rigorously established in control systems governed by partial differential equations [13], significantly enlarging the classes of systems for which these methods are applicable. In the context of economic model predictive control, strict dissipativity can be used as an alternative to assuming the turnpike property [14]. However, as [9] shows, these two properties are almost equivalent, in the sense that under a controllability and a reachability condition, strict dissipativity holds if and only if a robust version of the turnpike property holds. “Robust” here refers to the fact that not only optimal but also approximately optimal trajectories exhibit the turnpike phenomenon, though in a relaxed form. This robust version of the turnpike property has additional structure, making it more suitable for rigorous mathematical proofs, which is why we also use it in this paper. We will, however, use neither strict dissipativity nor controllability assumptions in this paper.

Many of the papers and books discussed above provide sufficient (and sometimes also necessary) conditions for the occurrence of turnpike behavior. However, most results apply to either finite horizon or infinite horizon optimal control problems. In contrast to this, in this paper we study the relation between turnpike properties for these two classes of optimal control problems. More precisely, we show that under suitable regularity conditions, the turnpike phenomenon occurs in the finite horizon problem if and only if it occurs in the infinite horizon problem with the same stage cost. The study is carried out for discrete-time systems, mainly because this somewhat reduces the analytic overhead and simplifies some arguments. However, we expect that a similar reasoning is also possible in the continuous-time setting. Related results can be found in the work of Zaslavski, where, for continuous-time systems [5, Theorem 3.1.4] and [6, Theorem 3], provide conditions whereby finite horizon turnpikes imply, and are implied by infinite horizon turnpikes. In the discrete-time setting of this paper [7, Theorem 2.2] provides sufficient conditions for an infinite horizon turnpike property to imply a finite horizon turnpike property. In contrast to [7], in this paper we do not require that the underlying state space be compact, nor even finite dimensional, nor do we require explicit regularity assumptions on the stage cost in the optimal control problem, and we do not impose any endpoint constraints. Moreover, we also present results for discounted optimal control problems with arbitrary discount rate $\beta \in {]0,1[}$ as well as quantitative bounds on the convergence rate of the trajectories to the equilibrium.

The paper is organized as follows. In Sect. 2, we describe the optimal control problem we are considering. In Sect. 3, we show the relation between the finite and the infinite horizon property for undiscounted problems. In Sect. 4 we repeat the analysis for discounted problems since these problems require significantly different assumptions and proof techniques. In Sect. 5, we discuss turnpike properties which also include quantitative bounds on the convergence rate of the trajectories to the equilibrium. Section 6 concludes the paper.

2 Setting and Preliminaries

We consider possibly discounted discrete-time optimal control problems

$$\begin{aligned} \mathop {\hbox {minimize}}_{u\in {\mathbb {U}}^N(x_0)} J_{N}(x_0,u) \end{aligned}$$

(1)

where

$$\begin{aligned} J_{N}(x_0,u):= & {} \sum _{k=0}^{N-1} \beta ^k \ell (x(k),u(k)),\nonumber \\ x(k+1)= & {} f(x(k),u(k)), \quad x(0) = x_0, \end{aligned}$$

(2)

$N\in {\mathbb {N}}$ or $N=\infty$, $\beta \in {]0,1]}$, $f:X\times U\rightarrow X$, and $\ell :X\times U \rightarrow {\mathbb {R}}$ for metric spaces X and U, state and input constraints ${\mathbb {X}}\subseteq X$, ${\mathbb {U}}\subseteq U$, and admissible control sets

$$\begin{aligned} {\mathbb {U}}^N(x_0) := \left\{ u(\cdot )\in U^N:\, \begin{array}{l} x(n)\in {\mathbb {X}}\; \forall n=0,\ldots , N \text{ and }\\ u(n) \in {\mathbb {U}}\; \forall n=0,\ldots ,N-1\end{array}\right\} . \end{aligned}$$

In what follows, for simplicity of exposition we assume ${\mathbb {U}}^N(x_0)\ne \emptyset$ for all $x_0\in {\mathbb {X}}$ and all $N\in {\mathbb {N}}$ (which implies ${\mathbb {U}}^\infty (x_0)\ne \emptyset$). If this is not the case, all results remain true if we restrict ourselves to initial conditions $x_0$ for which ${\mathbb {U}}^\infty (x_0)\ne \emptyset$ and to control functions from ${\mathbb {U}}^\infty (x_0)$. We define the optimal value function

$$\begin{aligned} V_{N}(x_0) := \inf _{u\in {\mathbb {U}}^N(x_0)} J_{N}(x_0,u). \end{aligned}$$

As we will see, both the proofs and the assumptions on the problem differ considerably between the undiscounted case $\beta =1$ and the discounted case $\beta \in {]0,1[}$. We therefore treat these two cases in two separate sections and start with the undiscounted case. In what follows, we denote the cardinality of a set $S \subset {\mathbb {N}}$ by $\mathop {\text {card}}S$.

3 The Undiscounted Case

In this section, we consider the undiscounted case; i.e., the case $\beta =1$. In this case, it is not guaranteed that $\sum _{k=0}^\infty \ell (x(k),u(k))=\lim _{K\rightarrow \infty } J_K(x_0,u)$ exists for all trajectories and controls. In order to avoid the introduction of complicated constraints on the set of controls over which we minimize, we use $J_\infty (x_0,u) = \limsup _{K\rightarrow \infty } J_K(x_0,u)$ in this case. Still, in order to obtain a meaningful optimal control problem, we need to ensure that $|V_\infty (x)|$ is finite for all $x\in {\mathbb {X}}$. A class of optimal control problems for which this can be achieved are dissipative optimal control problems satisfying a certain controllability property.

Definition 3.1

The optimal control problem is called strictly dissipative at an equilibrium $(x^e,u^e)\in {\mathbb {X}}\times {\mathbb {U}}$ if there exists a so-called storage function $\lambda :{\mathbb {X}}\rightarrow {\mathbb {R}}$, which is bounded from below, and a function² $\rho \in {{{\mathcal {K}}}}$ such that, for all $x\in {\mathbb {X}}$ and $u\in {\mathbb {U}}$ with $f(x,u)\in {\mathbb {X}}$, the inequality

$$\begin{aligned} \ell (x,u) - \ell (x^e,u^e) + \lambda (x) - \lambda (f(x,u)) \ge \rho (d(x,x^e)) \end{aligned}$$

(3)

holds. The optimal control problem is called dissipative if the same condition holds with $\rho \equiv 0$.

While we will not explicitly use strict dissipativity in our subsequent considerations (particularly, we do not require knowledge of a storage function $\lambda$), we will see that our results only apply to strictly dissipative systems; cf. Remark 3.1, below. Hence, it is not restrictive to assume this property. Many examples can be found in the literature of dissipative and strictly dissipative optimal control problems; see for instance the discussions and examples in [14,15,16,17]. Hence, there are many examples to which the following considerations apply.

Dissipativity implies $V_\infty (x)>-\infty$ for all $x\in {\mathbb {X}}$, provided we assume $\ell (x^e,u^e)=0$. This assumption can be made without loss of generality by replacing $\ell (x,u)$ with $\ell (x,u)-\ell (x^e,u^e)$. Then (3) with $\rho \equiv 0$ implies

$$\begin{aligned} J_K(x_0,u) = \sum _{k=0}^{K-1} \ell (x(k),u(k)) \ge - \lambda (x(0)) + \lambda (x(K)), \end{aligned}$$

and since $\lambda$ is bounded from below we obtain the existence of a constant $M\in {\mathbb {R}}$ such that $J_K(x_0,u) \ge -\lambda (x_0) + M$ for all $K\in {\mathbb {N}}$ and $u\in {\mathbb {U}}^\infty (x)$, implying $V_\infty (x_0) \ge - \lambda (x_0) + M$.

For the converse inequality $V_\infty (x)<\infty$—again under the assumption $\ell (x^e,u^e)=0$ and assuming that $\ell$ is continuous—it is sufficient to ensure that, for each initial condition $x(0)=x_0\in {\mathbb {X}}$, we can find a control which yields sufficiently fast convergence $x(k)\rightarrow x^e$ and $u(k)\rightarrow u^e$ as $k\rightarrow \infty$. Clearly, this convergence and the continuity of $\ell$ imply $\ell (x(k),u(k)) \rightarrow \ell (x^e,u^e)=0$ and “sufficiently fast” means that, additionally, $\limsup _{K\rightarrow \infty }\sum _{k=0}^{K-1}\ell (x(k),u(k))<\infty$ holds. For instance, if the cost $\ell$ is a polynomial (or at least bounded from above near $(x^e,u^e)$ by a polynomial p with $p(x^e,u^e)=0$), then exponential convergence $x(k)\rightarrow x^e$ and $u(k)\rightarrow u^e$ would suffice. This is an exponential controllability condition that is a standard condition satisfied by many examples in the literature. In finite dimensions it is guaranteed by stabilizability of the linearization of f at $(x^e,u^e)$.

In summary, there are many examples in which the infinite horizon undiscounted optimal control problem is well defined and, in what follows, we will restrict ourselves to such problems.

We now precisely define the turnpike properties under consideration.

Definition 3.2

(Finite horizon turnpike property) Optimal control problem (1) has the finite horizon robust turnpike property at an equilibrium $x^e\in {\mathbb {X}}$ if, for each $\delta >0$, each $\varepsilon >0$, and each bounded set ${\mathbb {X}}_b\subset {\mathbb {X}}$, there is a constant $C^\mathrm{fin}_{\delta ,\varepsilon ,{\mathbb {X}}_b}\in {\mathbb {N}}$ such that all trajectories (x(k), u(k)) with $x_0\in {\mathbb {X}}_b$, $u(\cdot )\in {\mathbb {U}}^N(x_0)$ and arbitrary $N\in {\mathbb {N}}$ satisfying $J_{N}(x_0,u) \le V_{N}(x_0) + \delta$ satisfy

$$\begin{aligned} \mathop {\text {card}}\Big \{ k\in \{0,\ldots ,N\}:\, d(x(k),x^e) \ge \varepsilon \Big \} \le C^\mathrm{fin}_{\delta ,\varepsilon ,{\mathbb {X}}_b}. \end{aligned}$$

(4)

In words, this definition demands that, given $\varepsilon >0$ and $\delta >0$, for every $\delta$-optimal trajectory starting in ${\mathbb {X}}_b$, all but at most $C^\mathrm{fin}_{\delta ,\varepsilon ,{\mathbb {X}}_b}$ points on the trajectory lie in an $\varepsilon$-neighborhood of $x^e$. The important property of the constant $C^\mathrm{fin}_{\delta ,\varepsilon ,{\mathbb {X}}_b}$ is that it does not depend on N; i.e., the bound on the number of points outside of the $\varepsilon$-neighborhood of $x^e$ is independent of N. Figure 1 shows finite horizon optimal trajectories on different horizons N which exhibit the turnpike property. For the details of the optimal control problems behind these figures we refer to [12].

Fig. 1

Finite horizon optimal trajectories $x(\cdot )$ (dashed) for different optimization horizons $N=2,4,\ldots ,30$ (left) and $N=2,4,\ldots ,20$ (right) for two examples from [12]

Definition 3.3

(Infinite horizon turnpike property) Consider optimal control problem (1) with $N=\infty$ and $|V_\infty (x)|<\infty$ for all $x\in {\mathbb {X}}$. Then problem (1) has the infinite horizon robust turnpike property at an equilibrium $x^e\in {\mathbb {X}}$ if, for each $\delta >0$, each $\varepsilon >0$, and each bounded set ${\mathbb {X}}_b\subset {\mathbb {X}}$, there is a constant $C^\infty _{\delta ,\varepsilon ,{\mathbb {X}}_b}\in {\mathbb {N}}$ such that all trajectories (x(k), u(k)) with $x_0\in {\mathbb {X}}_b$, $u(\cdot )\in {\mathbb {U}}^\infty (x_0)$ satisfying $J_{\infty }(x_0,u) \le V_{\infty }(x_0) + \delta$ satisfy

$$\begin{aligned} \mathop {\text {card}}\Big \{ k\in {\mathbb {N}}:\, d(x(k),x^e) \ge \varepsilon \Big \} \le C^{\infty }_{\delta ,\varepsilon ,{\mathbb {X}}_b}. \end{aligned}$$

(5)

Note that Definition 3.3 implies $\lim _{k\rightarrow \infty } d(x(k),x^e) =0$, because otherwise there are $\varepsilon >0$ and $k_j\rightarrow \infty$ with $d(x(k_j),x^e)\ge \varepsilon$ for all $j\in {\mathbb {N}}$. But this implies $\mathop {\text {card}}\Big \{ k\in {\mathbb {N}}: \, d(x(k),x^e) \ge \varepsilon \Big \} = \infty$, and hence the property from Definition 3.3 cannot hold. Therefore, the infinite horizon turnpike property implies convergence of the respective trajectories to the equilibrium $x^e$. However, the rate of convergence can be arbitrarily slow, since we do not make any assumption about the size of the time instant k in $\mathop {\text {card}}\Big \{ k\in {\mathbb {N}}: \, d(x(k),x^e) \ge \varepsilon \Big \}$. We will address this issue in Sect. 5.

In order to establish a relation between Definitions 3.2 and 3.3, we make the following regularity assumptions on the problem.

Assumption 3.1

We assume the following for optimal control problem (1).

• (i) For each bounded subset ${\mathbb {X}}_b\subseteq {\mathbb {X}}$ there exists $C>0$ such that $|V_N(x)|\le C$ holds for all $x\in {\mathbb {X}}_b$ and all $N\in {\mathbb {N}}\cup \{\infty \}$; and

• (ii) for each $\varTheta >0$ there is a bounded set ${\mathbb {X}}_\varTheta \subseteq {\mathbb {X}}$ such that for all $N\in {\mathbb {N}}\cup \{\infty \}$ the inequality $J_{N}(x_0,u) \le \varTheta$ implies $x(k)\in {\mathbb {X}}_\varTheta$ for all $k=0,\ldots ,N$.

Part (i) of Assumption 3.1 is a boundedness condition which demands that the optimal value functions are uniformly (with respect to N and including $N=\infty$) bounded on bounded sets, both from above and from below. It is needed to rule out degenerate behavior caused by unbounded accumulated cost. This assumption can be seen as a finite horizon variant of the assumption $|V_\infty (x)|<\infty$, and just like this inequality, it can, for example, be guaranteed by dissipativity and (sufficiently fast) controllability with respect to an equilibrium point $(x^e,u^e)$; cf. the discussion after Definition 3.1. Similar to the discussion there, the condition $\ell (x^e,u^e)=0$ is necessary for (i) to hold. If this is undesirable, we could replace the condition by “there exists $C>0$ and $D\in {\mathbb {R}}$ such that $|V_N(x)-ND|\le C$ holds for all $x\in {\mathbb {X}}_b$ and all $N\in {\mathbb {N}}\cup \{\infty \}$,” because then (i) holds if we replace $\ell (x,u)$ by $\ell (x,u)-D$, which does not change the optimal trajectories of the problem.

Part (ii) effectively states that trajectories with bounded values stay in bounded sets. There are (at least) two easy ways to ensure that this condition holds: On the one hand, we may assume that ${\mathbb {X}}$ itself is bounded, in which case we can always choose ${\mathbb {X}}_\varTheta ={\mathbb {X}}$. Alternatively, we may assume the existence of constants $C_1,C_2,C_3\in {\mathbb {R}}$ with $C_2>0$ and a point ${\hat{x}}\in {\mathbb {X}}$ such that the inequalities $\ell (x,u) \ge C_1+C_2 d(x,{\hat{x}})$ and $V_N(x)\ge C_3$ hold for all $x\in {\mathbb {X}}$, all $u\in {\mathbb {U}}$, and all $N\in {\mathbb {N}}\cup \{\infty \}$. In this case, the existence of $k\in {\mathbb {N}}$ with $k\le N$ with $d(x(k),{\hat{x}})>\varDelta$ implies

$$\begin{aligned} J_N(x,u)= & {} J_k(x,u) + \ell (x(k),u(k)) + J_{N-k-1}(x(k+1),u(\cdot +k+1)) \\> & {} C_3 + C_1 + C_2\varDelta + C_3. \end{aligned}$$

Hence, $J_N(x,u)\le \varTheta$ implies $\varDelta \le (\varTheta -C_1-2C_3)/C_2$, and thus ${\mathbb {X}}_\varTheta$ can be chosen as the closed ball with radius $(\varTheta -C_1-2C_3)/C_2$ around ${\hat{x}}$.

The following theorem now gives the main result of this section.

Theorem 3.1

Consider optimal control problem (1) satisfying Assumption 3.1. Then the finite horizon turnpike property from Definition 3.2 holds if and only if the infinite horizon turnpike property from Definition 3.3 holds.

Proof

“Definition 3.2 $\Rightarrow$ Definition 3.3”: Assume that the problem has the finite horizon turnpike property from Definition 3.2. We show that the problem then also has the infinite horizon turnpike property from Definition 3.3. To this end, we consider a trajectory satisfying the conditions from Definition 3.3. That is, we pick $\delta >0$, $\varepsilon >0$, a bounded subset ${\mathbb {X}}_b\subseteq {\mathbb {X}}$, and an infinite trajectory with $x_0\in {\mathbb {X}}_b$ satisfying $J_{\infty }(x_0,u) \le V_{\infty }(x_0) + \delta$.

Next we verify that the trajectory also satisfies the conditions from Definition 3.2. For this purpose, let C denote the bound from Assumption 3.1(i), which implies $|V_{N}(x) - V_{\infty }(x)|\le K = 2C$ for $x\in {\mathbb {X}}_b$. Then Assumption 3.1(ii) implies that $x(k) \in {\mathbb {X}}_\varTheta$ for all $k\in {\mathbb {N}}$ and a bounded set ${\mathbb {X}}_\varTheta$ with $\varTheta =K+\delta$ which, by Assumption 3.1(i), yields the existence of ${\widetilde{K}}>0$ with $V_{\infty }(x(k)) \ge -{\widetilde{K}}$ for all $k\in {\mathbb {N}}$. For all $N\in {\mathbb {N}}$ we have

$$\begin{aligned} V_{\infty }(x_0) + \delta\ge & {} J_{\infty }(x_0,u) \; = \; J_{N}(x_0,u) + \beta ^N J_\infty (x(N),u(N+\cdot ))\\\ge & {} J_{N}(x_0,u) - \beta ^N {\widetilde{K}} \end{aligned}$$

implying

$$\begin{aligned} J_{N}(x_0,u) \le V_{\infty }(x_0) + \delta + \beta ^N {\widetilde{K}} \le V_N(x_0) + K + \delta + {\widetilde{K}}. \end{aligned}$$

Thus, the conditions from Definition 3.2 are satisfied, and since by assumption the problem has the finite horizon robust turnpike property, from (4) we obtain

$$\begin{aligned} \mathop {\text {card}}\Big \{ k\in \{0,\ldots ,N\}: \, d(x(k),x^e) \ge \varepsilon \Big \} \le C^\mathrm{fin}_{\delta +K+{\widetilde{K}},\varepsilon ,{\mathbb {X}}_b} \end{aligned}$$

for all $N\in {\mathbb {N}}$, which implies (5) with

$$\begin{aligned} C^{\infty }_{\delta ,\varepsilon ,{\mathbb {X}}_b} = C^\mathrm{fin}_{\delta +K+{\widetilde{K}},\varepsilon ,{\mathbb {X}}_b}, \end{aligned}$$

and thus the infinite horizon robust turnpike property according to Definition 3.3.

“Definition 3.3 $\Rightarrow$ Definition 3.2”: We proceed similarly as above for the converse direction and consider a trajectory satisfying the conditions from Definition 3.2. To this end, fix $\delta >0$, $\varepsilon >0$, $N\in {\mathbb {N}}$, a bounded subset ${\mathbb {X}}_b\subseteq {\mathbb {X}}$, and a trajectory of length N with $x_0\in {\mathbb {X}}_b$ satisfying $J_{N}(x_0,u) \le V_{N}(x_0) + \delta$.

Now we construct an extended trajectory which satisfies the conditions from Definition 3.2. Letting $K=2C$ denote the bound on the difference $|V_N(x)-V_\infty (x)|$ from Assumption 3.1(i), by Assumption 3.1(ii) we can conclude the existence of a bounded set ${\mathbb {X}}_\varTheta$ with $x(N)\in {\mathbb {X}}_\varTheta$ and hence, again by Assumption 3.1(i), of a constant ${\widetilde{K}}$ with $V_\infty (x(N)) \le {\widetilde{K}}$. Picking a control function ${\tilde{u}}$ satisfying $J_\infty (x(N),{\tilde{u}}) \le {\widetilde{K}} + \delta$ and defining

$$\begin{aligned} {\hat{u}}(k) := \left\{ \begin{array}{l@{\quad }l} u(k), &{} k=0,\ldots ,N-1,\\ {\tilde{u}}(k-N), &{} k = N, N+1, \ldots , \end{array}\right. \end{aligned}$$

we thus obtain

$$\begin{aligned} J_\infty (x_0,{\hat{u}})= & {} J_{N}(x_0,u) + \beta ^N J_\infty (x(N),{\tilde{u}}) \le V_{N}(x_0) + \delta + {\widetilde{K}} + \delta \\\le & {} V_{\infty }(x_0) + K + {\widetilde{K}} + 2\delta . \end{aligned}$$

Hence, by (5) the extended trajectory satisfies

$$\begin{aligned} \mathop {\text {card}}\Big \{ k\in {\mathbb {N}}: d(x(k),x^e) \ge \varepsilon \Big \} \le C^{\infty }_{2\delta +K+{\widetilde{K}},\varepsilon ,{\mathbb {X}}_b}, \end{aligned}$$

which implies finite horizon turnpike (4) with $C^\mathrm{fin}_{\delta ,\varepsilon ,{\mathbb {X}}_b} = C^{\infty }_{2\delta +K+{\widetilde{K}},\varepsilon ,{\mathbb {X}}_b}$. $\square$

Remark 3.1

(i) Using [9, Lemma 3.9(a) and the implication “(b)$\Rightarrow$(c)” of Theorem 4.1], one sees that Assumption 3.1(i) together with the finite horizon turnpike property implies strict dissipativity. Thus, there is a close connection between our result and dissipativity theory. Particularly, in conjunction with Theorem 3.1 this observation immediately implies that Assumption 3.1 together with the infinite horizon turnpike property also implies strict dissipativity, an implication which to the best of our knowledge has not previously been observed in the literature.

(ii) We emphasize that despite the fact that there is a close connection between strict dissipativity and Theorem 3.1, we do not explicitly use strict dissipativity in our assumptions. Particularly, neither an explicit expression for the storage function $\lambda$ nor of the lower bound $\rho$ in Definition 3.1 is needed. If the finite or the infinite horizon turnpike property is established by another sufficient condition (see, e.g., [7] for a variety of such methods), we can apply our results without having to construct the functions involved in the strict dissipativity inequality.

4 The Discounted Case

We now turn our attention to the discounted case with $\beta \in {]0,1[}$. For our analysis, the decisive difference with the undiscounted case is that the discount factor $\beta ^k$ tends to 0 as k tends to infinity. This means that if a trajectory has a large deviation from the optimal trajectory, then this large deviation may nevertheless be barely visible in the cost functional, provided it happens sufficiently late. For this reason, it is unreasonable to expect that one can see the turnpike behavior for trajectories satisfying $J_N(x,u) \le V_N(x) + \delta$. In order to fix this problem, we need make two changes to the robust turnpike Definitions 3.2 and 3.3. First, we need to restrict the time interval on which we can expect to see the turnpike phenomenon and second, we need to limit the difference $\delta$ between the value of the trajectory under consideration and the optimal value. In the following definitions, the first will be taken care of by introducing the discrete-time interval $\{0,\ldots ,M\}$ and the second by defining the bound $\delta ^\mathrm{fin}_{\varepsilon ,M,{\mathbb {X}}_b}$.

Definition 4.1

(Finite horizon turnpike property) Optimal control problem (1) has the finite horizon near-optimal approximate turnpike property if, for each $\varepsilon >0$ and each bounded set ${\mathbb {X}}_b\subset {\mathbb {X}}$, there is a constant $C^\mathrm{fin}_{\varepsilon ,{\mathbb {X}}_b}>0$ such that for each $M\in {\mathbb {N}}$ there is a constant $\delta =\delta ^\mathrm{fin}_{\varepsilon ,M,{\mathbb {X}}_b}>0$ such that for all $N\in {\mathbb {N}}$ with $N\ge M$, all trajectories (x(k), u(k)) with $x_0\in {\mathbb {X}}_b$, $u(\cdot )\in {\mathbb {U}}^N(x_0)$ and $J_{N}(x_0,u) \le V_{N}(x_0) + \delta$ satisfy

$$\begin{aligned} \mathop {\text {card}}\Big \{ k\in \{0,\ldots ,M\}: \, d(x(k),x^e) \ge \varepsilon \Big \} \le C^\mathrm{fin}_{\varepsilon ,{\mathbb {X}}_b}. \end{aligned}$$

(6)

Definition 4.2

(Infinite horizon turnpike property) Optimal control problem (1) has the infinite horizon near-optimal approximate turnpike property if, for each $\varepsilon >0$ and each bounded set ${\mathbb {X}}_b\subset {\mathbb {X}}$, there is a constant $C^\infty _{\varepsilon ,{\mathbb {X}}_b}>0$ such that for each $M\in {\mathbb {N}}$ there is a constant $\delta =\delta ^{\infty }_{\varepsilon ,M,{\mathbb {X}}_b}>0$ such that all trajectories (x(k), u(k)) with $x_0\in {\mathbb {X}}_b$, $u(\cdot )\in {\mathbb {U}}^\infty (x_0)$ and $J_{\infty }(x_0,u) \le V_{\infty }(x_0) + \delta$ satisfy

$$\begin{aligned} \mathop {\text {card}}\Big \{ k\in \{0,\ldots ,M\}: \, d(x(k),x^e) \ge \varepsilon \Big \} \le C^{\infty }_{\varepsilon ,{\mathbb {X}}_b}. \end{aligned}$$

(7)

We note that in both definitions the level $\delta$ which measures the deviation from optimality depends on M. In both definitions, $\delta \rightarrow 0$ may be required if $M\rightarrow \infty$. It is, however, easily seen that the definitions imply (5) for the optimal trajectories (i.e., for $\delta =0$), provided they exist. We also note that Definition 4.2 implies $\lim _{k\rightarrow \infty } d(x^*(k),x^e) =0$ for the optimal trajectory, again provided it exists.

Similar to the definitions of the turnpike property, we also need to adapt Assumption 3.1 to the discounted case.

Assumption 4.1

We assume the following for optimal control problem (1).

• 1. $V_N \rightarrow V_\infty$ as $N\rightarrow \infty$ uniformly on bounded subsets of ${\mathbb {X}}$; and

• 2. for each ${\tilde{\varepsilon }}>0$ and each bounded set ${\mathbb {X}}_b\subseteq {\mathbb {X}}$ there is $N_0\in {\mathbb {N}}$ with the following property: For each $N'\ge N_0$ there is ${\tilde{\delta }}>0$ such that for all $x_0\in {\mathbb {X}}_b$, all $N\in {\mathbb {N}}\cup \{\infty \}$ with $N\ge N'$, and all $u\in {\mathbb {U}}^N(x_0)$ satisfying the inequality $J_N(x_0,u)\le V_N(x_0)+{\tilde{\delta }}$, the inequality $\beta ^{N'}|V_{N''}(x(N'))|\le {\tilde{\varepsilon }}$ holds for all $N''\in {\mathbb {N}}\cup \{\infty \}$.

Assumption 4.1(i) states that the two operations “taking the infimum of $J_N(x,u)$ with respect to u” and “passing to the limit for $N\rightarrow \infty$” can be interchanged without changing the value. While this would be a rather strong assumption for undiscounted problems, for discounted problems it is always satisfied if, e.g., the stage cost is bounded along the optimal trajectories. In this case, due to the exponential decay of $\beta ^k$, the value of a tail of an optimal trajectory becomes arbitrarily small, and hence also the difference between minimizing $J_N$ and $J_\infty$ becomes arbitrarily small. Therefore, Assumption 4.1(i) is always satisfied if, e.g., $\ell$ is bounded on ${\mathbb {X}}$ or at least on a set containing the optimal trajectories starting in a bounded set.

Assumption 4.1(ii) is relatively technical, but, again, since $\beta ^k\rightarrow 0$ as $k\rightarrow \infty$, if we know that the modulus of the optimal value functions $|V_N|$ for $N\in {\mathbb {N}}\cup \{\infty \}$ is bounded along the trajectories $x(\cdot )$, say by a constant C, then it suffices to choose $N'$ sufficiently large that $\beta ^{N'}C\le {\tilde{\varepsilon }}$ holds. Again, this boundedness holds, e.g., if the $|V_N|$ are uniformly bounded on the whole set ${\mathbb {X}}$ or if they are bounded on bounded sets and the near-optimal trajectories $x(\cdot )$ stay in bounded sets up to the time $N'$. Since the last two properties are implied by Assumption 3.1(i) and (ii), Part (ii) of Assumption 4.1 can be seen as a relaxation of Assumption 3.1.

The counterpart of Theorem 3.1 for the discounted case now reads as follows.

Theorem 4.1

Consider optimal control problem (1) satisfying Assumption 4.1. Then the finite horizon turnpike property from Definition 4.1 holds if and only if the infinite horizon turnpike property from Definition 4.2 holds.

Proof

“Definition 4.1 $\Rightarrow$ Definition 4.2”: Similar to the first part of the proof of Theorem 3.1 we consider a trajectory satisfying the conditions of Definition 4.2 and show that it also satisfies the conditions of Definition 4.1, from which we then conclude (7). However, we require some preliminary considerations in order to determine the bound on $\delta$ in Definition 4.2. To this end, fix $\varepsilon >0$, a bounded set ${\mathbb {X}}_b\subseteq {\mathbb {X}}$, and $M\in {\mathbb {N}}$, and let $\delta ^\mathrm{fin}=\delta ^\mathrm{fin}_{\varepsilon ,M,{\mathbb {X}}_b} >0$ be the level of accuracy needed in Definition 4.1. We set ${\tilde{\delta }} := \delta ^\mathrm{fin}/4$ and pick $N_0\in {\mathbb {N}}$ from Assumption 4.1(ii), and from Assumption 4.1(i) we choose $N\ge \max \{N_0,M\}$ sufficiently large that $|V_N(x_0)-V_\infty (x_0)| \le {\tilde{\delta }}$ for all $x_0\in {\mathbb {X}}_b$. For this N, we take ${\tilde{\varepsilon }}>0$ from Assumption 4.1(ii) and set $\delta := \min \{{\tilde{\varepsilon }},\delta ^\mathrm{fin}/2\}$. Now we consider a trajectory satisfying $J_{\infty }(x_0,u) \le V_{\infty }(x_0) + \delta$.

Then, using Assumption 4.1(ii) with $N=N'$ and $N''=\infty$ we obtain

$$\begin{aligned} V_{N}(x_0) + \delta + {\tilde{\delta }}\ge & {} V_{\infty }(x_0) + \delta \; \ge \; J_{\infty }(x_0,u)\\= & {} J_{N}(x_0,u) + \beta ^N J_\infty (x(N),u(N+\cdot ))\\\ge & {} J_{N}(x_0,u) + \beta ^N V_\infty (x(N)) \; \ge \; J_{N}(x_0,u) - {\tilde{\delta }} \, ; \end{aligned}$$

i.e.,

$$\begin{aligned} J_{N}(x_0,u) \le V_{N}(x_0) + \delta + 2{\tilde{\delta }} \le V_N(x_0) + \delta ^\mathrm{fin}. \end{aligned}$$

This implies the condition for the finite horizon near-optimal approximate turnpike property in Definition 4.1, and thus (6) yields the inequality

$$\begin{aligned} \mathop {\text {card}}\Big \{ k\in \{0,\ldots ,M\}: \, d(x(k),x^e) \ge \varepsilon \Big \} \le C^\mathrm{fin}_{\varepsilon ,{\mathbb {X}}_b} \end{aligned}$$

for all sufficiently large $N\in {\mathbb {N}}$. From this we obtain the infinite horizon near-optimal approximate turnpike property (7) with $C^{\infty }_{\varepsilon ,{\mathbb {X}}_b}=C^\mathrm{fin}_{\varepsilon ,{\mathbb {X}}_b}$ and $\delta ^{\infty }_{\varepsilon ,M,{\mathbb {X}}_b}=\delta$.

“Definition 4.2 $\Rightarrow$ Definition 4.1”: Similar to the second part of the proof of Theorem 3.1, we consider a trajectory satisfying the conditions of Definition 4.1 from which we construct an extended trajectory satisfying the conditions of Definition 4.2. As in the first part of the proof, we need to take care of the bounds of $\delta$ in these definitions.

Fix again $\varepsilon >0$, a bounded set ${\mathbb {X}}_b\subseteq {\mathbb {X}}$, and $M\in {\mathbb {N}}$, and let $\delta ^{\infty }=\delta ^{\infty }_{\varepsilon ,M,{\mathbb {X}}_b} >0$ be the level of accuracy needed in Definition 4.2. We set $\delta := \delta ^{\infty }/8$ and pick $N_0\in {\mathbb {N}}$ from Assumption 4.1(ii). From Assumption 4.1(i) we can find $N_1\ge N_0$ such that $|V_{N'}(x_0)-V_N(x_0)| \le \delta$ and $|V_{N'}(x_0)-V_\infty (x_0)| \le \delta$ for all $x_0\in {\mathbb {X}}_b$ and all $N,N'\ge N_1$. Moreover, we may pick $N_1$ sufficiently large that ${\tilde{\varepsilon }}$ from Assumption 4.1(ii) satisfies ${\tilde{\varepsilon }} < \delta ^\infty /4$. Finally, we set $N'=\max \{M,N_1\}$.

Then, for arbitrary $N\ge N'$ we pick a control sequence satisfying the conditions of Definition 4.1; i.e., with $J_{N}(x_0,u) \le V_{N}(x_0) + \delta$. This implies

$$\begin{aligned} V_{N'}(x_0) + 2\delta\ge & {} V_{N}(x_0) + \delta \; \ge \; J_{N}(x_0,u)\\= & {} J_{N'}(x_0,u) + \beta ^{N'} J_{N-N'}(x(N'),u(N'+\cdot ))\\\ge & {} J_{N'}(x_0,u) + \beta ^{N'} V_{N-N'}(x(N')) \; \ge \; J_{N'}(x_0,u) - {\tilde{\varepsilon }}, \end{aligned}$$

and thus $J_{N'}(x_0,u) \le V_{N'}(x_0) + 2\delta + {\tilde{\varepsilon }}$. Picking another control sequence ${\tilde{u}}$ satisfying $J_\infty (x(N'),{\tilde{u}}) \le V_\infty (x(N')) + \delta$, and defining

$$\begin{aligned} {\hat{u}}(k) := \left\{ \begin{array}{l@{\quad }l} u(k), &{} k=0,\ldots ,N'-1\\ {\tilde{u}}(k-N'), &{} k = N', N'+1, \ldots \, ,\end{array}\right. \end{aligned}$$

we thus obtain

$$\begin{aligned} J_\infty (x_0,{\hat{u}})= & {} J_{N'}(x_0,u) + \beta ^{N'} J_\infty (x(N'),{\tilde{u}})\\\le & {} V_{N'}(x_0) + 2\delta + {\tilde{\varepsilon }} + \beta ^{N'} V_\infty (x(N')) + \delta \\\le & {} V_{\infty }(x_0) + 4\delta +2{\tilde{\varepsilon }} \; = \; V_{\infty }(x_0) + \delta ^{\infty }. \end{aligned}$$

Hence, the extended trajectory satisfies the condition of Definition 4.2 and thus (7) yields

$$\begin{aligned} \mathop {\text {card}}\Big \{ k\in \{0,\ldots ,M\}: \, d(x(k),x^e) \ge \varepsilon \Big \} \le C^{\infty }_{\varepsilon ,{\mathbb {X}}_b}. \end{aligned}$$

This implies (6) and thus the finite horizon near-optimal approximate turnpike property from Definition 4.2 for $N\ge N'$ with $C^\mathrm{fin}_{\varepsilon ,{\mathbb {X}}_b} = C^{\infty }_{\varepsilon ,{\mathbb {X}}_b}$ and $\delta ^\mathrm{fin}_{\varepsilon ,M,{\mathbb {X}}_b}=\delta$. For arbitrary N we thus obtain (6) with $C^\mathrm{fin}_{\varepsilon ,{\mathbb {X}}_b} = \max \{ N',\, C^{\infty }_{\varepsilon ,{\mathbb {X}}_b}\}$. $\square$

Remark 4.1

For discounted problems, all sufficient conditions for turnpike properties of which we are aware work only for $\beta$ sufficiently close to 1; see, e.g., [18, 19] and also the discussion and references in Remark 3.1 in [18]. In contrast to this, our result holds for all discount factors $\beta \in {]0,1[}$.

Example 4.1

We consider a basic growth model in discrete time, which goes back to [20]. The problem is originally a maximization problem which can be written as a minimization problem of the form (1), (2) with ${\mathbb {X}}=]0,\infty [$ and

$$\begin{aligned} x(k+1) = u(k), \qquad \ell (x,u) = -\ln (Ax^{\alpha }-u). \end{aligned}$$

Here, $Ax^\alpha$ is a production function with constants $A>0$, $0<\alpha <1$, capital stock x, and control variable $u > 0$. The difference between output and next period’s capital stock (given by u) is consumption. For discount factor $\beta \in {]0,1[}$ the exact solution to the infinite horizon problem is known (see [21]) and is given by

$$\begin{aligned} V_\infty (x)=-B-C\ln x \end{aligned}$$

with

$$\begin{aligned} C=\frac{\alpha }{1-\alpha \beta } \;\; \text{ and } \;\; B=\frac{\ln ((1-\alpha \beta )A)+\frac{\beta \alpha }{1-\beta \alpha }\ln (\alpha \beta A)}{1-\beta }. \end{aligned}$$

The unique optimal equilibrium for this example is given by $x^e=1/\root \alpha -1 \of {\beta \alpha A}$, and using dynamic programming, one easily identifies the infinite horizon optimal control $u_\infty ^*$ in feedback form $u^*(k)= F(x(k))$ with

$$\begin{aligned} F(x) = \mathop {\text {argmin}}_{u \in {]0,\infty [}} \left\{ -\ln (Ax^{\alpha }-u) + \beta V(u)\right\} = \frac{\beta A C x^\alpha }{\beta C-1}. \end{aligned}$$

(8)

To the best of our knowledge, explicit solutions for the corresponding finite horizon problem are not known. However, using the explicit formulas for the infinite horizon problem, one checks that the problem has the infinite horizon turnpike property according to Definition (4.1) for all $\beta \in {]0,1[}$. Indeed, one easily checks that $F(x)<x$ for $x > x^e$ and $F(x)>x$ for $x<x^e$, from which convergence of the optimal solution to $x^e$ follows. Due to the fact that the expression to be minimized in (8) is strictly convex in u, near-optimal controls u(k) are close to optimal controls $u^*(k)$, which implies the turnpike property for near-optimal trajectories. The other assumptions of Theorem 4.1 are checked in a similar way. Hence, we can conclude the finite horizon turnpike property.

5 Turnpike with Transient Estimates

As already mentioned, the turnpike definitions so far do not allow for estimating how fast the trajectories approach the equilibrium $x^e$. They also do not allow for bounds on the trajectories during the time in which they are not close to $x^e$. In this section, we propose definitions for finite and infinite horizon turnpike properties that provide this information. Here, the infinite horizon definition was inspired by the usual notion of asymptotic stability (in its formulation via ${{{\mathcal {K}}}}{{{\mathcal {L}}}}$-functions, which has become standard in nonlinear control, see [22]), while the finite horizon definition can be seen as an extension of the exponential turnpike property established in [15] under a strict dissipativity condition. Like in the previous sections, we will then be able to show that these two conditions are equivalent under suitable regularity conditions on optimal control problem (1). In order to streamline the presentation, we limit ourselves to a set of assumptions suitable for the undiscounted setting from Sect. 3; i.e., to $\beta =1$.

For the following definitions, we recall (cf. Footnote 2) that ${{{\mathcal {K}}}}$ is the space of functions $\alpha :{\mathbb {R}}_0^+\rightarrow {\mathbb {R}}_0^+$ which are continuous and strictly increasing with $\alpha (0)=0$ and that ${{{\mathcal {K}}}}{{{\mathcal {L}}}}$ is the space of functions $\phi :{\mathbb {R}}_0^+\times {\mathbb {R}}_0^+\rightarrow {\mathbb {R}}_0^+$ which are continuous, $r\mapsto \phi (r,t)$ is a ${{{\mathcal {K}}}}$-function for each $t\ge 0$, and $t\mapsto \phi (r,t)$ is strictly decreasing to 0 for each $r>0$. The space ${{{\mathcal {L}}}}_{{\mathbb {N}}_0}$ denotes all functions $\gamma :{\mathbb {N}}_0\rightarrow {\mathbb {R}}_0^+$ which are strictly decreasing to 0.

Definition 5.1

(Finite horizon) Optimal control problem (1) has the finite horizon robust ${{{\mathcal {K}}}}{{{\mathcal {L}}}}$-turnpike property at an equilibrium $x^e\in {\mathbb {X}}$ if, for each bounded set ${\mathbb {X}}_b\subset {\mathbb {X}}$, there are $\phi \in {{{\mathcal {K}}}}{{{\mathcal {L}}}}$, $\omega \in {{{\mathcal {K}}}}$, and $\gamma \in {{{\mathcal {L}}}}_{N_0}$ such that for each $\delta >0$, $N\in {\mathbb {N}}$, and all trajectories (x(k), u(k)) with $x_0\in {\mathbb {X}}_b$, $u(\cdot )\in {\mathbb {U}}^N(x_0)$, and satisfying $J_{N}(x_0,u) \le V_{N}(x_0) + \delta$ the inequality

$$\begin{aligned} d(x(k),x^e) \le \phi (d(x_0,x^e),k)+ \omega \Big (\delta + \gamma (N) + \gamma (N-j)\Big ) \end{aligned}$$

(9)

holds for all $j=0,\ldots ,N$ and all $k=0,\ldots ,j$.

Definition 5.2

(Infinite horizon) Consider optimal control problem (1) with $N=\infty$ and $|V_\infty (x)|<\infty$ for all $x\in {\mathbb {X}}$. Then problem (1) has the infinite horizon robust ${{{\mathcal {K}}}}{{{\mathcal {L}}}}$-turnpike property at an equilibrium $x^e\in {\mathbb {X}}$ if, for each bounded set ${\mathbb {X}}_b\subset {\mathbb {X}}$, there are $\phi \in {{{\mathcal {K}}}}{{{\mathcal {L}}}}$ and $\omega \in {{{\mathcal {K}}}}$ such that for each $\delta >0$ and all trajectories (x(k), u(k)) with $x_0\in {\mathbb {X}}_b$, $u(\cdot )\in {\mathbb {U}}^\infty (x_0)$, and satisfying $J_{\infty }(x_0,u) \le V_{\infty }(x_0) + \delta$, the inequality

$$\begin{aligned} d(x(k),x^e) \le \phi (d(x_0,x^e),k)+ \omega (\delta ) \end{aligned}$$

(10)

holds.

We note that (10) implies that optimal trajectories $x^\star (k)$ starting at $x=x^e$ satisfy $x^\star (k)=x^e$. Hence, in order to ensure that $V_\infty (x^e)$ is finite, we need that $\min _{u\in {\mathbb {U}}, f(x^e,u)=x^e}\ell (x^e,u)=0$, which implies $V_\infty (x^e)=0$. We may thus assume $V_\infty (x^e)=0$ without loss of generality in the remainder of this section. We note that this assumption does not imply $V_N(x^e)\approx 0$, even for large N.

In order to show equivalence of Definitions 5.1 and 5.2, in addition to Assumption 3.1 we need the following assumption.

Assumption 5.1

For optimal control problem (1) we assume that there is $K\in {\mathbb {R}}$ such that for any bounded set ${\mathbb {X}}_b\subseteq {\mathbb {X}}$ there is $\rho \in {{{\mathcal {L}}}}_{{\mathbb {N}}_0}$ such that for all $x\in {\mathbb {X}}_b$ the inequality

$$\begin{aligned} | V_\infty (x) - V_N(x) + K | \le \rho (N) \end{aligned}$$

holds.

The intuition behind Assumption 5.1 is as follows: assume the infinite horizon problem has an optimal equilibrium $(x^e,u^e)$ with $\ell (x^e,u^e)=0$. Then we have $V_\infty (x^e)=0$, but since on finite horizons $(x^e,u^e)$ will typically not be an optimal equilibrium, in general $\lim _{N\rightarrow \infty }V_N(x^e)=0$ will not hold. In this case, this limit value is the candidate for the value K for which Assumption 5.1 holds. The following lemma shows that this reasoning can be made precise under rather mild conditions if the turnpike property holds.

Lemma 5.1

Consider optimal control problem (1) and assume that the problem exhibits the turnpike property according to Definitions 3.2 and 3.3 and $V_\infty (x^e)=0$. Assume moreover that the limit $\lim _{N\rightarrow \infty } V_N(x^e)$ exists and that the optimal value functions $V_N$ are continuous at $x^e$ uniformly in $N\in {\mathbb {N}}\cup \{\infty \}$ in the following sense: There exists $\sigma \in {{{\mathcal {K}}}}$ and $\nu \in {{{\mathcal {L}}}}_{{\mathbb {N}}_0}$ such that the inequality

$$\begin{aligned} |V_N(x)-V_N(x^e)| \le \sigma (d(x,x^e)) + \nu (N) \end{aligned}$$

holds for all $x\in {\mathbb {X}}$ and $N\in {\mathbb {N}}\cup \{\infty \}$, with the convention $\nu (\infty )=0$. Then Assumption 5.1 is satisfied.

Proof

We show that the assertion follows for $K=\lim _{N\rightarrow \infty } V_N(x^e)$. We choose $\eta \in {{{\mathcal {L}}}}_{{\mathbb {N}}_0}$ such that $|V_N(x^e)-K|\le \eta (N)$ for all $N\in {\mathbb {N}}$ and fix a bounded set ${\mathbb {X}}_b\subseteq {\mathbb {X}}$. Moreover, we note that it is sufficient to prove the assertion for sufficiently large N, because the continuity assumption implies boundedness of $V_N$ and $V_\infty$ on bounded sets, which ensures existence of $\rho (N)$ for finitely many N.

We start by showing that there exists $\rho _1\in {{{\mathcal {L}}}}_{{\mathbb {N}}_0}$ for which the inequality $V_\infty (x) \le V_N(x^e) - K + \rho _1(N)$ holds for all $x\in {\mathbb {X}}_b$. To this end, fix $\delta _0>0$, let $\delta \in {]0,\delta _0[}$, $x\in {\mathbb {X}}_b$, and consider a control $u^\delta$ with $J_N(x,u^\delta )\le V_N(x) + \delta$.

Then, for sufficiently large $N\in {\mathbb {N}}$ and $\varepsilon >0$ the constant $C_{\delta _0,\varepsilon ,{\mathbb {X}}_b}^\mathrm{fin}$ from Definition 3.2 satisfies $C_{\delta _0,\varepsilon ,{\mathbb {X}}_b}^\mathrm{fin} \ge N/2$. We pick $\varepsilon =\varepsilon (N)>0$ minimal such that this inequality holds. Then, since for each $\varepsilon >0$ there is $N\in {\mathbb {N}}$ such that $C_{\delta _0,\varepsilon ,{\mathbb {X}}_b}^\mathrm{fin} \ge N/2$ holds, it follows that $\varepsilon (N)\rightarrow 0$ as $N\rightarrow \infty$. Hence, there is ${\tilde{\varepsilon }}(\cdot )\in {{{\mathcal {L}}}}_{{\mathbb {N}}_0}$ with $\varepsilon (N)\le {\tilde{\varepsilon }}(N)$; e.g., ${\tilde{\varepsilon }}(N)=\sup _{K \ge N} \varepsilon (K) + 2^{-N}$. For each N we now pick the minimal $k^*\in \{0,\ldots ,N\}$ satisfying $d(x_{u^\delta }(k^*),x^e)< \varepsilon (N)$, which because of $C_{\delta _0,\varepsilon ,{\mathbb {X}}_b}^\mathrm{fin} \ge N/2$ satisfies $N-k^*\ge \lfloor N/2 \rfloor$. We pick a control ${\hat{u}}^\delta$ satisfying $J_\infty (x_{u^\delta }(k^*),{\hat{u}}^\delta ) \le V_\infty (x_{u^\delta }(k^*)) + \delta$ and set $u(k)=u^\delta (k)$, $k=0,\ldots ,k^*-1$ and $u(k)={\hat{u}}^\delta (k+k^*)$, $k\ge k^*$. Then we can estimate

$$\begin{aligned} V_\infty (x)\le & {} J_\infty (x,u)\\= & {} J_{k^*}(x,u^\delta ) + J_\infty \left( x_{u^\delta }(k^*),{\hat{u}}^\delta \right) \\\le & {} J_N(x,u^\delta ) - J_{N-k^*}\left( x_{u^\delta }(k^*), u^\delta (k^*+\cdot )\right) + V_\infty \left( x_{u^\delta }(k^*)\right) + \delta \\\le & {} V_N(x) + \delta - V_{N-k^*}\left( x_{u^\delta }(k^*)\right) + V_\infty \left( x_{u^\delta }(k^*)\right) + \delta \\\le & {} V_N(x) - K + V_\infty (x^e) + 2\sigma (\varepsilon (N)) + \nu (N-k^*) + \eta (N) + 2\delta . \end{aligned}$$

Since $\delta >0$ was arbitrary, $N-k^* \ge \lfloor N/2 \rfloor$, and $V_\infty (x^e)=0$, this shows the claim with $\rho _1(N)=2\sigma ({\tilde{\varepsilon }}(N)) + \nu (\lfloor N/2 \rfloor ) + \eta (N)$.

The converse inequality $V_N(x) \le V_\infty (x^e) + K + \rho _2(N)$ is obtained similarly, starting from a $\delta$-optimal trajectory for the $\infty$-horizon problem and extending it after the “turnpike time” $k^*$ by a $\delta$-optimal trajectory for the problem with horizon $N-k^*$. Together this yields the assertion with $\rho =\max \{\rho _1,\rho _2\}$. $\square$

The first equivalence theorem for Definitions 5.1 and 5.2 now uses Assumption 5.1.

Theorem 5.1

Consider optimal control problem (1) and assume that

• 1. $|V_{\infty }|$ is bounded on bounded subsets of ${\mathbb {X}}$;

• 2. Assumption 5.1 holds; and

• 3. for each $\varTheta >0$ there is a bounded set ${\mathbb {X}}_\varTheta \subseteq {\mathbb {X}}$ such that for each $N\in {\mathbb {N}}\cup \{\infty \}$ the inequality $J_{N}(x_0,u) \le \varTheta$ implies $x(k)\in {\mathbb {X}}_\varTheta$ for all $k=0,\ldots ,N$.

Then Definition 5.1 holds if and only if Definition 5.2 holds.

Proof

“Definition 5.1 $\Rightarrow$ Definition 5.2”: Consider a trajectory $x(\cdot )$ with control $u(\cdot )$ and initial value $x_0$ satisfying the conditions of Definition 5.2. Then for all $j\in {\mathbb {N}}$ we obtain

$$\begin{aligned} J_j(x_0,u) + V_\infty (x(j)) \le J_\infty (x_0,u) \le V_\infty (x_0) + \delta . \end{aligned}$$

Then from (ii) with ${\mathbb {X}}_b={\mathbb {X}}_\varTheta$ from (iii), for arbitrary $N\in {\mathbb {N}}$ with $j\le N$ we obtain

$$\begin{aligned} J_j(x_0,u)\le & {} V_\infty (x_0) - V_\infty (x(j)) + \delta \le V_N(x_0) - V_{N-j}(x(j)) \\&+ \,\delta + \rho (N) + \rho (N-j). \end{aligned}$$

Now taking the control u(k) for $k=0,\ldots ,j-1$ and extending it with an $\varepsilon$-optimal control for horizon $N-k$, arbitrary $\varepsilon >0$ and initial value x(j) yields a control ${\tilde{u}}$ satisfying

$$\begin{aligned} J_N(x_0,{\tilde{u}}) \le J_j(x_0,u) + V_{N-j}(x(j)) +\varepsilon \le V_N(x_0) + \delta + \rho (N) + \rho (N-j) + \varepsilon . \end{aligned}$$

Hence, Definition 5.1 with $\delta +\rho (N)+\rho (N+j)+\varepsilon$ in place of $\delta$ implies the estimate

$$\begin{aligned} d(x(k),x^e)\le & {} \phi \left( d(x_0,x^e\right) ,k) \\&+\,\omega \Big (\delta + \rho (N) + \gamma (N) + \rho (N-j) + \gamma (N-j) + \varepsilon \Big ) \end{aligned}$$

for all $k=0,\ldots ,j$. Fixing k and letting $\varepsilon \rightarrow 0$, $N\rightarrow \infty$ and $j:=\lfloor N/2 \rfloor \rightarrow \infty$, continuity of $\phi$ and $\omega$ and the fact that $\rho \in {{{\mathcal {L}}}}_{{\mathbb {N}}_0}$ and $\gamma \in {{{\mathcal {L}}}}_{{\mathbb {N}}_0}$ yield the desired inequality

$$\begin{aligned} d(x(k),x^e) \le \phi \left( d\left( x_0,x^e\right) ,k\right) + \omega (\delta ). \end{aligned}$$

“Definition 5.2 $\Rightarrow$ Definition 5.1”: Consider a trajectory $x(\cdot )$ of length N with control $u(\cdot )$ and initial value $x_0$ satisfying the conditions of Definition 5.1. Then for all $j=0,\ldots ,N$ we obtain

$$\begin{aligned} J_j(x_0,u) + V_{N-j}(x(j)) \le J_N(x_0,u) \le V_N(x_0) + \delta . \end{aligned}$$

Then from (ii) with ${\mathbb {X}}_b={\mathbb {X}}_\varTheta$ from (iii) we obtain

$$\begin{aligned} J_j(x_0,u)\le & {} V_N(x_0) - V_{N-j}(x(j)) + \delta \le V_\infty (x_0) \\&- V_\infty (x(j)) + \delta + \rho (N) + \rho (N-j). \end{aligned}$$

Now taking the control u(k) for $k=0,\ldots ,j-1$ and extending it with an $\varepsilon$-optimal control for infinite horizon for arbitrary $\varepsilon >0$ and initial value x(j) yields a control ${\tilde{u}}$ satisfying

$$\begin{aligned} J_\infty (x_0,{\tilde{u}}) \le J_j(x_0,u) + V_\infty (x(j)) + \varepsilon \le V_\infty (x_0) + \delta + \rho (N) + \rho (N-j) + \varepsilon . \end{aligned}$$

Hence, using Definition 5.2 with $\delta +\rho (N)+\rho (N+j)+\varepsilon$ in place of $\delta$ yields the estimate

$$\begin{aligned} d(x(k),x^e) \le \phi \left( d\left( x_0,x^e\right) ,k\right) + \omega \Big (\delta + \rho (N) + \rho (N-j)+\varepsilon \Big ) \end{aligned}$$

for all $k=0,\ldots ,j$. For $\varepsilon \rightarrow 0$, continuity of $\omega$ yields the desired inequality for $\gamma =\rho \in {{{\mathcal {L}}}}_{{\mathbb {N}}_0}$. $\square$

Using Lemma 5.1 we can obtain a variant of Theorem 5.1 avoiding the use of Assumption 5.1.

Corollary 5.1

Consider optimal control problem (1) and assume that

• 1. $V_\infty (x^e)=0$ and $\lim _{N\rightarrow \infty } V_N(x^e)$ exists;

• 2. the optimal value functions $V_N$ are continuous at $x^e$ uniformly in the horizon $N\in {\mathbb {N}}\cup \{\infty \}$ in the following sense: There exists $\gamma \in {{{\mathcal {K}}}}$ and $\nu \in {{{\mathcal {L}}}}_{{\mathbb {N}}_0}$ such that the inequality

  $$\begin{aligned} |V_N(x)-V_N(x^e)| \le \gamma (d(x,x^e)) + \nu (N) \end{aligned}$$

  holds for all $x\in {\mathbb {X}}$ and $N\in {\mathbb {N}}\cup \{\infty \}$ with the convention $\nu (\infty )=0$; and

• 3. for each $\varTheta >0$ there is a bounded set ${\mathbb {X}}_\varTheta \subseteq {\mathbb {X}}$ such that for all $N\in {\mathbb {N}}\cup \{\infty \}$ the inequality $J_{N}(x_0,u) \le \varTheta$ implies $x(k)\in {\mathbb {X}}_\varTheta$ for all $k=0,\ldots ,N$.

Then Definition 5.1 holds if and only if Definition 5.2 holds.

Proof

We note that (i) and (ii) imply boundedness of $V_\infty$ and $V_N$ on bounded sets. Moreover, we note that the turnpike property from Definition 5.1 implies that of Definition 3.2 and that the property from Definition 5.2 implies that of Definition 3.3. Since by Theorem 3.1 the properties from Definitions 3.2 and 3.3 are equivalent under the conditions of the corollary, we obtain that if either Definition 5.1 or Definition 5.2 holds, then both Definitions 3.2 and 3.3 follow. Hence we can apply Lemma 5.1 in order to conclude that Assumption 5.1 holds. The assertion then follows from Theorem 5.1. $\square$

We illustrate the use of Theorem 5.1 by the following well-known class of optimal control problems.

Example 5.1

Consider an undiscounted linear quadratic optimal control problem with

$$\begin{aligned} f(x,u)=Ax+Bu \qquad \text{ and } \qquad \ell (x,u) = (x^T,u^T) \, G \left( \begin{array}{c}x\\ u\end{array}\right) , \end{aligned}$$

where (A, B) is stabilizable and the matrix G is symmetric and positive definite. It is well known that for such a problem the optimal trajectories converge to the origin exponentially fast and that the infinite horizon optimal value function is of the form $V_\infty (x) = x^T Q_\infty x$ for a symmetric and positive definite matrix $Q_\infty$. Moreover, the optimal control is available in linear feedback form; i.e., $u^* = Fx$ and $V_\infty$ is a quadratic Lyapunov function. More precisely, the inequality

$$\begin{aligned} V(Ax+BFx) \le V(x) - \ell (x,Fx) \end{aligned}$$

holds for all $x\in {\mathbb {R}}^n$. For all trajectories (x(k), u(k)) satisfying the inequality $J_{\infty }(x_0,u) \le V_{\infty }(x_0) + \delta$ and all $k\in {\mathbb {N}}$ it holds that

$$\begin{aligned} J_\infty (x(k),u(k+\cdot )) \le V_\infty (x(k))+ \delta , \end{aligned}$$

which implies

$$\begin{aligned} V_\infty (x(k+1))\le & {} J_\infty (x(k+1),u(k+1+\cdot ))\\= & {} J_\infty (x(k),u(k+\cdot )) - \ell (x(k),u(k)) \\\le & {} V_\infty (x(k)) - \ell (x(k),u(k)) + \delta . \end{aligned}$$

From this inequality a standard Lyapunov argument yields the existence of $C_1,C_2>0$ and $a\in {]0,1[}$ such that

$$\begin{aligned} V_\infty (x(k)) \le \max \left\{ C_1 a^k V_\infty (x(0)),\, C_2\delta \right\} \end{aligned}$$

for all $k\in {\mathbb {N}}$, which implies

$$\begin{aligned} \Vert x(k)\Vert \le \phi (\Vert x(0)\Vert ,k) + \omega (\delta ) \end{aligned}$$

with $\phi (r,k) = C \sqrt{a}^k r$ and $\omega (r) = C\sqrt{\delta }$ for an appropriate constant $C>0$. Hence, the problem has the infinite horizon robust ${{{\mathcal {K}}}}{{{\mathcal {L}}}}$-turnpike property according to Definition 5.2.

In order to show that the problem also has the finite horizon robust ${{{\mathcal {K}}}}{{{\mathcal {L}}}}$-turnpike property according to Definition 5.1, we now check that the problem satisfies the conditions of Theorem 5.1. Condition (i) is obviously satisfied, since $V_\infty$ is a quadratic function. Condition (ii) follows with $K=0$ from the fact that $V_N(x) = x^TQ_Nx$ and the matrices $Q_N$ are defined via the Riccati difference equation and converge exponentially fast toward $Q_\infty$, which is the solution of the discrete-time algebraic Riccati equation. The exponential convergence moreover implies that $\rho$ can be chosen to be of the form $\rho (N) = Db^N$ with $D>0$ and $b\in {]0,1[}$. Condition (iii) follows immediately from the fact that G is positive definite, implying the existence of $C_\varTheta >0$ such that $\ell (x,u)>\varTheta$ whenever $\Vert x\Vert \ge C_\varTheta$. From this condition (iii) follows with ${\mathbb {X}}_\varTheta =\{x\in {\mathbb {R}}^n: \, \Vert x\Vert \le C_\varTheta \}$.

Thus, all conditions of Theorem 5.1 hold and we can conclude the finite horizon ${{{\mathcal {K}}}}{{{\mathcal {L}}}}$-turnpike property with functions $\phi (r,k) = C \sqrt{a^k} r$, $\omega (r) = C\sqrt{r}$, and $\rho (N)=D b^N$; i.e., both $\phi$ and $\rho$ are exponentially decaying in time k or in the horizon N, respectively.

Since our results hold on arbitrary metric spaces, we can extend the reasoning of this example to infinite-dimensional systems; for instance, to discrete-time linear quadratic optimal control problems whose state dynamics are described by a linear semigroup of operators on an infinite-dimensional Hilbert space. For such problems, the properties used above hold in an analogous way (see, e.g., [23, 24]), and thus our results allow one to conclude the finite horizon robust ${{{\mathcal {K}}}}{{{\mathcal {L}}}}$-turnpike property with exponentially decaying $\phi$ and $\rho$ for this class of infinite-dimensional systems.

6 Conclusions

In this paper, we have investigated the relationship between turnpike properties for finite and infinite horizon optimal control problems with the same stage cost. Specifically, we have shown that, under mild technical assumptions, these properties are equivalent. Furthermore, this relationship has been demonstrated for optimal control problems involving both undiscounted and discounted stage costs, making the results applicable to commonly studied problems in both engineering and mathematical economics.

Furthermore, we have proposed a definition of a turnpike property that incorporates information about the rate of convergence for optimal trajectories approaching an optimal equilibrium, as well as a bound on how far such trajectories can be from this equilibrium during the time when they are not close. This robust ${{\mathcal {K}}}{{\mathcal {L}}}$-turnpike property provides a potential route to sharper quantitative results in problems involving turnpikes, similar to the modern use of comparison functions in stability theory.