Stability of Solutions to Hamilton–Jacobi Equations Under State Constraints

Abstract

In the present paper, we investigate stability of solutions of Hamilton–Jacobi–Bellman equations under state constraints by studying stability of value functions of a suitable family of Bolza optimal control problems under state constraints. The stability is guaranteed by the classical assumptions imposed on Hamiltonians and an inward-pointing condition on state constraints.

1 Introduction

In the optimal control theory, very often we deal with partial differential equations called Hamilton–Jacobi–Bellman equations. In the literature, there are several concepts of the generalised solutions of Hamilton–Jacobi equations. In order to deal with continuous solutions of the Hamilton–Jacobi–Bellman equation, the notion of viscosity solution was introduced; see, e.g. Crandall and Lions [1] for the definition of viscosity solution. This paper concerns first-order Hamilton–Jacobi equations with convex Hamiltonian under state constraints and investigates robustness of the viscosity solutions with respect to perturbations of the Hamiltonian and the constraints. We establish stability of solutions of Hamilton–Jacobi–Bellman equations under state constraints by investigating stability of value functions of a suitable family of Bolza optimal control problems under state constraints. It is well known that, in the absence of state constraints, the value function of the Bolza problem satisfies the Hamilton–Jacobi–Bellman equation in a generalised sense. Namely, under some technical assumptions, the value function is a unique viscosity solution of the Hamilton–Jacobi equation. For the case with no state constraints, there is large literature, where under appropriate assumptions it is proved that the value function is the unique viscosity solution of Hamilton–Jacobi–Bellman equation; see, e.g. [2, 3]. Several papers were devoted to Hamilton–Jacobi–Bellman equations under state constraints; see, e.g. [4, 5]. The uniqueness of solution of Hamilton–Jacobi–Bellman equation was proved by different authors under the hypotheses, which include the so-called inward-pointing condition (IPC). The existence of solutions is known under hypotheses that include an inward-pointing condition, in the literature dealing with discontinuous value functions; see, e.g. Crandall-Lions [1], an outward-pointing condition is imposed instead. Soner [6] has considered inward-pointing condition for the constraint set having a smooth boundary and investigated the infinite horizon optimal control problem. The inward-pointing condition is an important property in investigation of uniqueness of solutions to Hamilton–Jacobi–Bellman equation under state constraints, because it allows to approximate (in the sense of uniform convergence) feasible trajectories by trajectories staying in the interior of the constraint set; see, for example [7–9] for the most recent neighbouring feasible trajectories (NFT) theorems concerning such approximations. In order to investigate the discontinuous solutions to Hamilton–Jacobi–Bellman equation, Ishii and Koike [10] have expressed the inward-pointing condition using “inward” trajectories of a control system, which is not simple to verify. In general, the value function of the Bolza optimal control problem may be not continuous (even if all data are smooth). Frankowska and Plaskacz [11] have proved the uniqueness results for Hamilton–Jacobi–Bellman equation by extending the inward-pointing condition to constraints having nonsmooth boundary.

In the present paper, we investigate stability of solutions of Hamilton–Jacobi–Bellman equations by investigating stability of value functions of Bolza problems. The stability is guaranteed by the classical assumptions imposed on Hamiltonians and an inward-pointing condition on state constraints. We show that, under appropriate assumptions, the value function is a unique viscosity solution to Hamilton–Jacobi–Bellman equation. This allows us to conclude that solutions are stable with respect to Hamiltonians and state constraints. The novelty of this paper is that it establishes stability of solutions of the Hamilton–Jacobi equations under more general conditions that those required to obtain stability from known results. The stability analysis arises in various engineering applications, in which the perturbations correspond to the difference between idealised models and the real world. Nevertheless, we do not investigate such applications in the present paper.

The outline of the paper is as follows: In Sect. 2, we recall some notions and introduce some notations. In Sect. 3, we investigate the stability of value functions of Bolza problems. In Sect. 4, we associate with a Hamilton–Jacobi–Bellman equation (with the Hamiltonian convex in the last variable) a Bolza optimal control problem. In Sect. 5, we prove the uniqueness of solutions of Hamilton–Jacobi–Bellman equation and their continuous dependence on data.

2 Preliminaries and Notations

The notation $B(x_{0}, R)$ stands for the closed ball in $\mathbb {R}^{n}$ of centre $x_{0} \in \mathbb {R}^{n}$ and radius $R \ge 0$ and $RB :=B(0, R)$, $B:=B(0,1)$. We denote by $\langle p,v \rangle$ the scalar product of $p,\, v \in \mathbb {R}^n$ and by $\vert x \vert$ the Euclidean norm. For a bounded function $f : \Omega \rightarrow \mathbb {R}$, we define $\Vert f \Vert _{\infty } :=\sup \left\{ \,\left| f(x)\right| :x\in \Omega \,\right\}$. For a set $X \subset \mathbb {R}^{n}$, denote by conv(X) its convex hull. For an extended real-valued function $f : \mathbb {R}^{n} \rightarrow \mathbb {R} \cup \lbrace \pm \infty \rbrace$, $f \mid _{K}$ stands for the restriction of f to K. Let A be a metric space with the distance d and X be a subset of A. The distance from $x \in A$ to X is defined by

$$\begin{aligned} \mathrm{d}(x,X) :=\inf _{ y \in X} \mathrm{d}(x,y), \end{aligned}$$

where we have set d$(x, \emptyset ) = + \infty$. We denote by $\partial X$ the boundary of X.

Let $\{X_{i}\}_{i \ge 1}$ be a family of subsets of A. The subset

$$\begin{aligned} Limsup_{i \rightarrow \infty } X_{i} := & {} \lbrace x \in A: \liminf _{i \rightarrow \infty } \mathrm{d}(x, X_{i}) = 0 \rbrace \\ \quad= & {} \lbrace x \in A : \mathrm{for \, every \, open \, neighbourhood} \ U \ \mathrm{of} \ x, \\&\qquad U \cap X_{i} \ne \emptyset \ \mathrm{for \, infinitely \, many} \ i\rbrace , \end{aligned}$$

is called the upper limit of the sequence $X_{i}$, and the subset

$$\begin{aligned} Liminf_{i \rightarrow \infty } X_{i} := & {} \lbrace x \in A: \limsup _{i \rightarrow \infty } \mathrm{d}(x, X_{i}) = 0 \rbrace \\ \quad= & {} \lbrace x \in A : \ \mathrm{for \, every \, open \, neighbourhood} \ U \ of \ x, \\&\qquad U \cap X_{i} \ne \emptyset \ \mathrm{for \, all \, large \, enough} \ i\rbrace , \end{aligned}$$

is called its lower limit. A subset X is said to be the (Kuratowski) set limit of the sequence $X_{i}$ iff

$$\begin{aligned} X = Liminf_{i \rightarrow \infty } X_{i} = Limsup_{i \rightarrow \infty } X_{i} =: Lim_{i \rightarrow \infty } X_{i}. \end{aligned}$$

For arbitrary subsets X, Y of $\mathbb {R}^{n}$, the extended Hausdorff distance between X and Y is defined by

$$\begin{aligned}\mathcal {H}aus(X,Y) := \mathrm{max} \lbrace \sup _{x \in X} \mathrm{d}(x, Y), \sup _{x \in Y} \mathrm{d}(x, X) \rbrace \in \mathbb {R}\cup \lbrace + \infty \rbrace ,\end{aligned}$$

which may be equal to $+ \infty$ when X or Y is unbounded or empty.

It is well known that, if $X_i$ are subsets of a given compact set, then

$$\begin{aligned} X = Lim_{i \rightarrow \infty } X_{i} \Leftrightarrow \lim _{i \rightarrow \infty } \mathcal {H}aus(X_i,X)=0. \end{aligned}$$

Let $T > 0$, $F(\cdot , \cdot ) : [0, T] \times \mathbb {R}^{n} \rightrightarrows \mathbb {R}^{n}$ be a multifunction with compact and nonempty values. Consider $t_{0} \in [0,T[$ and the following differential inclusion

$$\begin{aligned} \dot{x}(t) \in F(t,x(t)), \ \mathrm{a.e.} \ t \in [t_{0},T]. \end{aligned}$$

(1)

Solutions to differential inclusion (1) are understood in the Carathéodory sense, i.e. absolutely continuous functions verifying (1) almost everywhere. We denote by $\bar{S}_{[t_{0}, T]}(x_{0})$ the set of absolutely continuous solutions $x(\cdot )$ of (1), defined on $[t_{0}, T]$ and satisfying the initial condition $x(t_{0}) = x_{0}$. Let $K \subset \mathbb {R}^{n}$ be a closed and nonempty set. Consider the following state constrained differential inclusion

$$\begin{aligned} \dot{x}(t)\in & {} F(t,x(t)), \ \mathrm{a.e.} \ t \in [t_{0},T], \nonumber \\ x(t)\in & {} K, \ \forall t \in [t_{0},T]. \end{aligned}$$

(2)

The very proof of Theorem 2.3, [8] implies the following result, the so-called neighbouring feasible trajectories (NFT) theorem, stated in a slightly different way than Theorem 2.3, [8].

Theorem 2.1

(NFT) Let $r_{0} > 0$. Assume that for some positive constant $c > 0$ and for $R = e^{cT}(r_{0} + 1)$, the following hypotheses hold true

• 1. $\max _{v \in F(t,x)} \vert v \vert \le c (1+ \vert x \vert ),$ for any $x \in \mathbb {R}^{n}$ and for $t \in [0,T]$.

• 2. There exists $c_{R}(\cdot ) \in L^{1}(0,T)$ such that, for all $x, x^{\prime } \in RB$ and a.e. $t \in [0,T]$,

  $$\begin{aligned} F(t,x^{\prime }) \subset F(t,x) + c_{R}(t) \vert x - x^{\prime } \vert B. \end{aligned}$$

• 3. (IPC) Inward-pointing condition. There exist $\varepsilon >0, \eta > 0$ such that, for any $(t,x) \in [0,T] \times (\partial K + \eta B) \cap RB \cap K$, we can find $v \in co F(t,x)$ satisfying $x^{\prime } + [0, \varepsilon ](v + \varepsilon B ) \subset K,$ for all $x^{\prime } \in (x + \varepsilon B) \cap K.$

• 4. For an absolutely continuous function $a_{R}:[0,T] \rightarrow \mathbb {R}$ and for any $x \in K \cap RB$ and $0 \le s < t \le T$,

  $$\begin{aligned} F(s,x) \subset F(t,x) + \int _{s}^{t} a_{R}(\tau ) \mathrm{d} \tau B. \end{aligned}$$

  Then, there exists $C > 0$ depending only on $\varepsilon , \eta , c, c_{R}(\cdot )$ and $a_{R}(\cdot )$ such that, for any $t_{0} \in [0, T[$ and any solution $\hat{x}(\cdot )$ of (1), with $\hat{x}(t_{0}) \in K \cap (e^{ct_{0}}(r_{0}+1)-1)B$, we can find a solution $x(\cdot )$ of (2) satisfying $x(t_{0}) = \hat{x}(t_{0})$, $x(t) \in int K$ for all $t \in ]t_{0}, T]$ and

  $$\begin{aligned} \vert \hat{x}(\cdot )- x(\cdot ) \vert _{C([t_{0},T], \mathbb {R}^{n} )} \le C \max _{t \in [t_{0}, T]} \mathrm{dist}(\hat{x}(t),K). \end{aligned}$$

Definition 2.1

Let $i \ge 1$ and $K_{i} \subset \mathbb {R}^{n}$ be closed and nonempty sets. For $T>0$ consider $V_{i} : [0,T] \times K_{i} \rightarrow \mathbb {R}$. We say that $V_{i}$ are equicontinuous uniformly in i, iff for any $\varepsilon >0$, there exists $\delta > 0$, such that for any i and any $x, y \in K_{i}$, $t, s \in [0,T]$ with $\vert x-y \vert + \vert t-s \vert \le \delta$,

$$\begin{aligned} \vert V_{i}(t,x)-V_{i}(s,y) \vert \le \varepsilon . \end{aligned}$$

Definition 2.2

Let $\phi : \mathbb {R}^{n} \rightarrow \mathbb {R} \cup \lbrace \pm \infty \rbrace$.

• 1. $Dom(\phi )$ is the set of all $x_{0} \in \mathbb {R}^{n}$, such that $\phi (x_{0}) \ne \pm \infty$.

• 2. The epigraph of $\phi$ is defined by $epi(\phi ) := \lbrace (x,a): x \in \mathbb {R}^{n}, a \in \mathbb {R}, a \ge \phi (x) \rbrace .$ The hypograph of $\phi$ is defined by $hyp(\phi ) := \lbrace (x,a): x \in \mathbb {R}^{n}, a \in \mathbb {R}, a \le \phi (x) \rbrace .$

• 3. The subdifferential of $\phi$ at $x_{0} \in Dom(\phi )$ is defined by

  $$\begin{aligned} \partial _{-} \phi (x_{0}) := \left\{ p \in \mathbb {R}^{n}: \ \liminf _{x \rightarrow x_{0}} \dfrac{\phi (x)-\phi (x_{0})- \langle p, x-x_{0} \rangle }{ \vert x- x_{0} \vert } \ge 0 \right\} . \end{aligned}$$

  The superdifferential of $\phi$ at $x_{0} \in Dom(\phi )$ is defined by

  $$\begin{aligned} \partial _{+} \phi (x_{0}) := \left\{ p \in \mathbb {R}^{n}: \ \limsup _{x \rightarrow x_{0}} \dfrac{\phi (x)-\phi (x_{0})- \langle p, x-x_{0} \rangle }{ \vert x- x_{0} \vert } \le 0 \right\} . \end{aligned}$$

• 4. The contingent epiderivative of $\phi$ at $x_{0} \in Dom(\phi )$ in the direction $u \in \mathbb {R}^{n}$ is defined by

  $$\begin{aligned} D_{\uparrow } \phi (x_{0})(u) := \liminf _{h \rightarrow 0+, v \rightarrow u} \dfrac{\phi (x_{0}+hv)-\phi (x_{0})}{h}. \end{aligned}$$

  The contingent hypoderivative of $\phi$ at $x_{0} \in Dom(\phi )$ in the direction $u \in \mathbb {R}^{n}$ is defined by

  $$\begin{aligned} D_{\downarrow } \phi (x_{0})(u) := \limsup _{h \rightarrow 0+, v \rightarrow u} \dfrac{\phi (x_{0}+hv)-\phi (x_{0})}{h}. \end{aligned}$$

Definition 2.3

Let $X \subset \mathbb {R}^{n}$. We call $d \in \mathbb {R}^{n}$ a tangent direction (in the sense of Bouligand) to X at point $x \in X$, iff there exist sequences $\lbrace x_{k} \rbrace _{k \in \mathbb {N}}$ and $\lbrace t_{k} \rbrace _{k \in \mathbb {N}}$ such that

$$\begin{aligned} \lbrace x_{k} \rbrace \subset X, \ \ t_{k} \downarrow 0, \ \ \dfrac{x_{k}-x}{t_{k}} \rightarrow d. \end{aligned}$$

The set of tangent directions to X at x is called a tangent cone (in the sense of Bouligand) for X at x and is denoted by $T_{X}(x).$

Definition 2.4

We define a normal cone (in the sense of Bouligand) to a set $X \subset \mathbb {R}^{n}$ at point $x \in X$ by $N_{X}(x) := \lbrace p \in \mathbb {R}^{n}: \langle p, v \rangle \le 0, \forall v \in T_{X}(x) \rbrace .$

Lemma 2.1

Let K be a closed set in $\mathbb {R}^{n}$ and $F : K \rightrightarrows \mathbb {R}^{n}$ be lower semicontinuous with closed images. Then, the following are equivalent

• (i) $F(x) \subset T_{K}(x)$, for any $x \in K$.

• (ii) $F(x) \subset cl(conv(T_{K}(x)))$, for any $x \in K.$

Proof

• (i) $\Rightarrow$ (ii), is immediate.

• (ii) $\Rightarrow$ (i), from Theorem 4.1.10, [12], we deduce that

  $$\begin{aligned} Lim inf_{x \rightarrow _{K} x_{0}} F(x) \subset Lim inf_{x \rightarrow _{K} x_{0}} cl(conv(T_{K}(x))) \subseteq T_{K}(x_{0}), \end{aligned}$$

  where $\rightarrow _{K}$ denotes the convergence in the set K. As F is lower semicontinuous, $F(x_{0}) \subset Lim inf_{x \rightarrow x_{0}} F(x).$ This ends the proof. $\square$

Lemma 2.2

Let $\phi : \mathbb {R}^{n} \rightarrow \mathbb {R}$, $x_{0} \in \mathbb {R}^{n}$. Then, $p \in \partial _{-} \phi (x_{0})$ iff, for any $v \in \mathbb {R}^{n}$,

$$\begin{aligned} D_{\uparrow } \phi (x_{0}) (v) \ge \langle p, v \rangle , \end{aligned}$$

and $p \in \partial _{+} \phi (x_{0})$ iff, for any $u \in \mathbb {R}^{n}$,

$$\begin{aligned} D_{\downarrow } \phi (x_{0}) (u) \le \langle p, u \rangle . \end{aligned}$$

Proof

We do not provide a proof, since in [3], the complete proof is provided. $\square$

3 Stability of the Value Functions of Bolza Problems

Let U be a compact metric space, K and $K_{i}$ be nonempty and closed subsets of $\mathbb {R}^{n}$ for $i = 1, 2, \ldots$, controls $u(\cdot )$ be Lebesgue measurable maps on [0, T] taking values in U, where $T>0$. Let $y_{0} \in \mathbb {R}^{n}$ and $\varphi :\mathbb {R}^{n} \rightarrow \mathbb {R}$, $\varphi _{i}:\mathbb {R}^{n} \rightarrow \mathbb {R}$ be equicontinuous. Consider continuous functions $f:[0,T] \times \mathbb {R}^{n} \times U \rightarrow \mathbb {R}^{n}$, $f_{i}:[0,T] \times \mathbb {R}^{n} \times U \rightarrow \mathbb {R}^{n}$, $l:[0,T] \times \mathbb {R}^{n} \times U \rightarrow \mathbb {R}$, $l_{i}:[0,T] \times \mathbb {R}^{n} \times U \rightarrow \mathbb {R}$, $i =1, 2,\ldots$ and the following Bolza optimal control problems:

$$\begin{aligned}&(P) {\left\{ \begin{array}{ll} \min \int _{0}^{T} \! l(s,x(s),u(s)) \, \mathrm {d} s + \varphi (x(T)), \\ \dot{x}(s)=f(s,x(s),u(s)), \ u(s) \in U \ \mathrm{a.e.} \ \mathrm{in} \ [0,T],\\ x(0)=y_{0}, \\ x(s) \in K, \ \forall s \in [0,T]. \end{array}\right. } \end{aligned}$$

(3)

$$\begin{aligned}&(P_{i}) {\left\{ \begin{array}{ll} \min \int _{0}^{T} \! l_{i}(s,x(s),u(s)) \, \mathrm {d} s + \varphi _{i} (x(T)), \\ \dot{x}(s)=f_{i}(s,x(s),u(s)), \ u(s) \in U \ \mathrm{a.e.} \ \mathrm{in} \ [0,T],\\ x(0)=y_{0}, \\ x(s) \in K_{i}, \ \forall s \in [0,T]. \end{array}\right. } \end{aligned}$$

(4)

We impose the following assumptions on f and l.

(A1) For any $R >0$, there exist an integrable function $c_R : [0,T] \rightarrow \mathbb {R}_+$ and an absolutely continuous function $a_{R}: [0,T] \rightarrow \mathbb {R}$ such that, for all $t,s \in [0,T]$, $x,y \in RB$, $u \in U$,

$$\begin{aligned}&\vert f(t,x,u)-f(t,y,u) \vert + \vert l(t,x,u)-l(t,y,u) \vert \le c_{R}(t) \vert x-y \vert ,\\&\vert f(t,x,u)-f(s,x,u) \vert + \vert l(t,x,u)-l(s,x,u) \vert \le \vert a_{R}(t)-a_{R}(s) \vert . \end{aligned}$$

(A2) There exists a positive constant $c > 0$ such that $\vert f(t,x,u) \vert \le c(1+\vert x \vert )$ for all $(t,x,u) \in [0,T] \times \mathbb {R}^{n} \times U$.

For any $(t_{0}, y_{0}) \in [0,T] \times \mathbb {R}^{n}$ denote by $S_{[t_{0},T]}(y_{0})$ the set of all trajectory–control pairs of the control system under state constraint

$$\begin{aligned} \dot{x}(s)= & {} f(s,x(s),u(s)), \ u(s) \in U \ \mathrm{a.e.} \ \mathrm{in} \ [t_{0},T],\nonumber \\ x(t_{0})= & {} y_{0}, \nonumber \\ x(s)\in & {} K, \ \forall s \in [t_{0},T], \end{aligned}$$

(5)

and by $S^{i}_{[t_{0},T]}(y_{0})$ the set of all trajectory–control pairs of the following control system under state constraint

$$\begin{aligned} \dot{x}(s)= & {} f_{i}(s,x(s),u(s)), \ u(s) \in U \ \mathrm{a.e.} \ \mathrm{in} \ [t_{0},T],\nonumber \\ x(t_{0})= & {} y_{0}, \nonumber \\ x(s)\in & {} K_{i}, \ \forall s \in [t_{0},T]. \end{aligned}$$

(6)

For all $(t_{0}, y_{0}) \in [0,T] \times \mathbb {R}^{n}$, the value function of the Bolza optimal control problem (P) is defined by:

$$\begin{aligned} V(t_{0},y_{0}) := \inf \left\{ \int _{t_{0}}^{T} \! l(s,x(s),u(s)) \, \mathrm {d} s + \varphi (x(T)): (x,u) \in S_{[t_{0},T]}(y_{0}) \right\} . \end{aligned}$$

(7)

Similarly, for all $(t_{0}, y_{0}) \in [0,T] \times \mathbb {R}^{n}$, the value function of the Bolza optimal control problem $(P_{i})$ is defined by:

$$\begin{aligned} V_{i}(t_{0},y_{0}) :=\inf \left\{ \int _{t_{0}}^{T} \! l_{i}(s,x(s),u(s)) \, \mathrm {d} s + \varphi _{i}(x(T)): (x,u) \in S^{i}_{[t_{0},T]}(y_{0}) \right\} . \end{aligned}$$

(8)

In the above, we set $V(t_{0}, y_{0}) = + \infty$, if $S_{[t_{0}, T]}(y_{0}) = \emptyset$ and, respectively, we set $V_{i}(t_{0}, y_{0}) = + \infty$, if $S^{i}_{[t_{0}, T]}(y_{0}) = \emptyset .$

We assume that the closed sets K and $K_{i}$ are defined by the multiple inequality constraints, namely let $g_{i}^{j}:\mathbb {R}^{n} \rightarrow \mathbb {R}$ and $g^{j} : \mathbb {R}^{n} \rightarrow \mathbb {R}$, $j=1,\ldots ,m$, $i = 1,2,\ldots$ be given continuously differentiable functions satisfying

(A3) Regularity.

• 1. For any $R >0$, there exists $A_{R} >0$ such that $\vert \nabla g_{i}^{j}(x) \vert \le A_{R}$ for any $x \in RB$ and $\nabla g_{i}^{j}$ is $A_{R}$-Lipschitz on RB, $i = 1,2,\ldots$, $j = 1,2,\ldots ,m$.

• 2. $\nabla g_{i}^{j} \rightarrow \nabla g^{j}$ uniformly on compacts and $g^{j}_{i}(0) \rightarrow g^{j}(0)$, when $i \rightarrow \infty$, for any $j = 1,\ldots ,m.$

Consider closed sets

$$\begin{aligned} K_{i}:= & {} \bigcap _{j = 1}^{m} \lbrace x: g_{i}^{j}(x) \le 0 \rbrace , \end{aligned}$$

(9)

$$\begin{aligned} K:= & {} \bigcap _{j = 1}^{m} \lbrace x: g^{j}(x) \le 0 \rbrace . \end{aligned}$$

(10)

For any $x \in \mathbb {R}^{n}$, let us now denote by I(x) the set of active indices at x, for $g(\cdot ) = (g^{1}(\cdot ),\ldots ,g^{m}(\cdot ))$, i.e.

$$\begin{aligned} I(x) := \lbrace j : g^{j}(x) = 0 \rbrace . \end{aligned}$$

(A4) Inward-pointing condition.

For any $R >0$, there exists $\rho _{R} > 0$ such that, for every $x \in K \cap RB$ with $I(x) \ne \emptyset$ and every $s \in [0,T]$,

$$\begin{aligned} \inf _{v \in co f(s,x,U)} \max _{j \in I(x)} \langle \nabla g^{j}(x),v \rangle \le - \rho _{R}. \end{aligned}$$

Lemma 3.1

Let $K, K_{i} \subset \mathbb {R}^{n}$ defined above be nonempty and (A2), (A3), (A4) hold true. If $f_{i}$ converge to f uniformly on compacts, then for every $R >0$, there exist $\eta _{R} >0$, $\varepsilon >0$, $i_{0} \ge 1$ such that, for all $i \ge i_{0}$, $t \in [0, T]$ and $x \in (\partial K_{i}+ \eta _{R} B) \cap RB \cap K_{i}$ we can find $v_{x, t} \in co f_{i}(t,x, U)$ satisfying $x^{\prime } + [0, \varepsilon ](v_{x,t} + \varepsilon B) \subset K_{i}$, for all $x^{\prime } \in (x+\varepsilon B) \cap K_{i}$.

Proof

The proof follows by a straightforward, but somewhat technical, contradiction argument. $\square$

Proposition 3.1

Let the assumptions of Lemma 3.1 hold true. Then, for any $\delta > 0$, there exists $i_{0}$ such that for any $i \ge i_{0}$

$$\begin{aligned} K \cap RB \subset (K_{i} \cap (RB+\delta B) ) + \delta B. \end{aligned}$$

Proof

The proof follows by a straightforward, but somewhat technical, contradiction argument. $\square$

Proposition 3.2

Let the assumption (A3) holds true. Then, for any $R >0$, $\delta > 0$, there exists $i_{0}$ such that, for any $i \ge i_{0}$

$$\begin{aligned} \overline{K^{c}_{i} \cap RB} \subset (\overline{K^{c}} \cap RB) + \delta B. \end{aligned}$$

Proof

The proof follows by a straightforward, but somewhat technical, contradiction argument. $\square$

Proposition 3.3

Let the assumption (A3) holds true. For all $x_{0} \in Int K$ and $r >0$ such that $x_{0} + rB \subset K$ there exists $i(x_{0})$ satisfying $x_{0}+\frac{r}{2} B \subset K_{i}$ for all $i \ge i(x_{0})$.

Proof

The proof follows by a straightforward, but somewhat technical, contradiction argument. $\square$

For all $(t,x) \in [0,T] \times \mathbb {R}^{n}$ and $i \ge 1$ define

$$\begin{aligned} G_{i}(t,x) := \lbrace (f_{i}(t,x,u), l_{i}(t,x,u)+r): u \in U, r \ge 0 \rbrace . \end{aligned}$$

Theorem 3.1

Let (A3), (A4) hold true and assume that $G_{i}(t,x)$ is convex and closed for all $i \ge 1$, $(t,x) \in [0,T] \times \mathbb {R}^{n}$ and f, $f_{i}$, l, $l_{i}$ satisfy (A1), (A2) with the same integrable functions $c_{R}(\cdot )$, absolutely continuous functions $a_{R}(\cdot )$ and $c > 0$. Assume that $f_{i}$ converge to f, $l_{i}$ converge to l and $\varphi _{i}$ converge to $\varphi$ uniformly on compacts, when $i \rightarrow \infty$, and that for some $M_{R} >0$ and all $(t,x,u) \in [0,T] \times RB \times U$, we have $\vert l(t,x,u) \vert + \vert l_{i}(t,x,u) \vert \le M_{R}$. Then, for all $x_{0} \in Int K$ and $r > 0$ such that $x_{0} + rB \subset K$, $V_{i} \mid _{[0,T] \times B(x_{0}, \frac{r}{2} ) }$ converge uniformly to $V \mid _{[0,T] \times B(x_{0}, \frac{r}{2} ) },$ when $i \rightarrow \infty$. Furthermore, for any $Q >0$, $V_{i} \mid _{[0,T] \times (B(0, Q) \cap K_{i})}$ are equicontinuous uniformly in i.

Proof

We first show that $V_{i} \mid _{[0,T] \times (B(0, Q) \cap K_{i})}$ are equicontinuous uniformly in i, for any $Q >0$. Fix $Q >0$. Let us now prove that there exist increasing, continuous functions $\omega ^{\prime } : [0, + \infty [ \rightarrow [0, + \infty [$ and $\omega ^{\prime \prime } : [0, + \infty [ \rightarrow [0, + \infty [$ with $\omega ^{\prime }(0)=0$ and $\omega ^{\prime \prime }(0) = 0$ such that, for any $t_{1}, t_{0} \in [0,T]$ and $y_{1}, y_{2} \in B(0,Q) \cap K_{i},$

$$\begin{aligned}&\vert V_{i}(t_{0},y_{1})-V_{i}(t_{0},y_{2}) \vert \le \omega ^{\prime }(\vert y_{1}-y_{2} \vert ),\nonumber \\&\vert V_{i}(t_{1},y_{1})-V_{i}(t_{0},y_{1}) \vert \le \omega ^{\prime \prime }(\vert t_{1}-t_{0} \vert ). \end{aligned}$$

(11)

We have $\varphi _{i}(\cdot )$ are equicontinuous on compact subsets of $\mathbb {R}^{n}$. Thus, for any compact set $\Omega \subset \mathbb {R}^{n}$, there exists an increasing, continuous function $\omega _{\Omega } : [0, + \infty [ \rightarrow [0, + \infty [$ such that $\omega _{\Omega }(0)=0$ and $\vert \varphi _{i}(x) - \varphi _{i}(y) \vert \le \omega _{\Omega }(\vert x-y \vert )$, for all $x, y \in \Omega$. Let us prove that there exists a modulus of continuity $\omega ^{\prime }(\cdot )$ such that, for all i and for any $y_{1}, y_{2} \in B(0, Q) \cap K_{i}$ and $t_{0} \in [0,T]$,

$$\begin{aligned} \vert V_{i}(t_{0},y_{1})-V_{i}(t_{0},y_{2}) \vert \le \omega ^{\prime }(\vert y_{1} - y_{2} \vert ). \end{aligned}$$

Let $i(y_{1})$ be as in Proposition 3.3. It is well known that, taking into account Lemma 3.1, under assumptions of Theorem 3.1, there exist $(y_{i}(\cdot ), u_{i}(\cdot )) \in S^{i}_{[t_{0},T]}(y_{1})$, for any $i \ge i_{0}$ such that

$$\begin{aligned} V_{i}(t_{0},y_{1}) = \int _{t_{0}}^{T} \! l_{i}(s,y_{i}(s),u_{i}(s)) \, \mathrm {d} s + \varphi _{i} (y_{i}(T)). \end{aligned}$$

Then, $y_{i}(\cdot )$ is a trajectory of the following system

$$\begin{aligned} \dot{y_{i}}(s)= & {} f_{i}(s,y_{i}(s), u_{i}(s)), \\ \dot{\bar{z}}_{i}(s)= & {} l_{i}(s,y_{i}(s),u_{i}(s)), \\ y_{i}(t_{0})= & {} y_{1}, \\ \bar{z}_{i}(t_{0})= & {} 0, \end{aligned}$$

satisfying $y_{i}(s) \in K_{i}$, for all $s \in [t_{0},T]$. Now consider the solution $(x_{i}(\cdot ), z_{i}(\cdot ))$ of

$$\begin{aligned} \dot{x}_{i}(s)= & {} f_{i}(s,x_{i}(s),u_{i}(s)), \\ \dot{z}_{i}(s)= & {} l_{i}(s,x_{i}(s),u_{i}(s)), \\ x_{i}(t_{0})= & {} y_{2}, \\ z_{i}(t_{0})= & {} 0. \end{aligned}$$

Let $R > 0$ be such that, for every trajectory–control pair (x, u) of the control system $\dot{x}(t) = f_{i}(t,x(t),u(t))$, $u(t) \in U$ satisfying $x(t_{0}) \in B(0, Q)$ for some $t_{0} \in [0, T]$ we have $x_{i}(T) \in B(0,R)$. We would like to underline that B(0, R) depends on Q. As $Q > 0$ is fixed, for the simplicity, we will omit the subindex B(0, R) for $\omega _{B(0,R)}(\cdot )$. By Lemma 3.1 and Theorem 2.1 applied to

$$\begin{aligned} F(t,x,z) = \lbrace (f_{i}(t,x,u), l_{i}(t,x,u)), u \in U \rbrace , \end{aligned}$$

there exists $C >0$ independent from i such that, for all $i \ge 1$, we can find absolutely continuous $(\tilde{x}_{i}(\cdot ), \tilde{z}_{i}(\cdot ))$ such that $(\dot{\tilde{x}}_{i}(t), \dot{\tilde{z}}_{i}(t)) \in F(t, \tilde{x}_{i}(t), \tilde{z}_{i}(t))$ a.e. $\tilde{x}_{i}(t_{0})=y_{2}$, $\tilde{z}_{i}(t_{0})=0$ satisfying the following state constraints $(\tilde{x}_{i}(t),\tilde{z}_{i}(t)) \in K_{i} \times \mathbb {R},$ for all $t \in [t_{0}, T]$ such that

$$\begin{aligned} \Vert \tilde{z}_{i}- z_{i} \Vert _{\infty } + \Vert \tilde{x}_{i}- x_{i} \Vert _{\infty } \le C \max _{s \in [t_{0},T]} \mathrm{dist}(x_{i}(s), K_{i}). \end{aligned}$$

By the Gronwall inequality, for any $s \in [t_{0}, T]$ and for a constant $E >0$, we have $\Vert x_{i}(s) - y_{i}(s) \Vert _{\infty } \le E \vert y_{2} - y_{1} \vert .$ Using Gronwall’s inequality, we deduce

$$\begin{aligned} \max _{s \in [t_{0},T]} \mathrm{dist}(x_{i}(s), K_{i}) \le \max _{s \in [t_{0},T]} \vert x_{i}(s) - y_{i}(s) \vert \le E \vert y_{2}-y_{1} \vert . \end{aligned}$$

By Filippov theorem, Theorem 8.2.10, [12], for some measurable $\tilde{u}_{i}(\cdot ) : [t_{0},T] \rightarrow U$ we have

$$\begin{aligned} \dot{\tilde{x}}_{i}(s)= & {} f_{i}(s,\tilde{x}_{i}(s),\tilde{u}_{i}(s)) \ \mathrm{a.e.} \ \mathrm{in} \ [t_{0}, T],\\ \dot{\tilde{z}}_{i}(s)= & {} l_{i}(s,\tilde{x}_{i}(s),\tilde{u}_{i}(s)) \ \mathrm{a.e.} \ \mathrm{in} \ [t_{0}, T]. \end{aligned}$$

We have that $\vert \varphi _{i}(\tilde{x_{i}}(T))-\varphi _{i} (x_{i}(T)) \vert \le \omega (\vert \tilde{x_{i}}(T)-x_{i}(T) \vert )$ and also

$$\begin{aligned} \left| \int _{t_{0}}^{T} \! l_{i}(s,\tilde{x_{i}}(s),\tilde{u_{i}}(s)) \, \mathrm {d} s - \int _{t_{0}}^{T} \! l_{i}(s,x_{i}(s),u_{i}(s)) \, \mathrm {d} s \right| \le \Vert \tilde{z}_{i}- z_{i} \Vert _{\infty } \le C E \vert y_{2} - y_{1} \vert . \end{aligned}$$

By the definition of $V_{i}(t_{0},\cdot )$, we have

$$\begin{aligned} V_{i}(t_{0},y_{2})-V_{i}(t_{0},y_{1})\le & {} \int _{t_{0}}^{T} \! l_{i}(s,\tilde{x_{i}}(s),\tilde{u_{i}}(s)) \, \mathrm {d} s + \varphi _{i} (\tilde{x_{i}}(T))\\&- \int _{t_{0}}^{T} \! l_{i}(s,y_{i}(s),u_{i}(s)) \, \mathrm {d} s \\ -\varphi _{i} (y_{i}(T)) \!= & {} \! \int _{t_{0}}^{T} \! (l_{i}(s,\tilde{x_{i}}(s),\tilde{u_{i}}(s)) - l_{i}(s,y_{i}(s),u_{i}(s))) \, \mathrm {d} s + (\varphi _{i} (\tilde{x_{i}}(T))\\&- \varphi _{i} (y_{i}(T)) ). \end{aligned}$$

Hence,

$$\begin{aligned}&V_{i}(t_{0},y_{2})-V_{i}(t_{0},y_{1}) \le \int _{t_{0}}^{T} \! (l_{i}(s,\tilde{x_{i}}(s),\tilde{u_{i}}(s)) - l_{i}(s, x_{i}(s),u_{i}(s))) \, \mathrm {d} s \\&\quad + \int _{t_{0}}^{T} \! (l_{i}(s, x_{i}(s),u_{i}(s)) - l_{i}(s,y_{i}(s),u_{i}(s))) \, \mathrm {d} s + \varphi _{i} (\tilde{x_{i}}(T))- \varphi _{i} (x_{i}(T)) \\&\quad + \, \varphi _{i} (x_{i}(T))- \varphi _{i} (y_{i}(T)). \end{aligned}$$

From (A1), we deduce that

$$\begin{aligned}&V_{i}(t_{0},y_{2})-V_{i}(t_{0},y_{1}) \le C E \vert y_{2}- y_{1} \vert + \int _{t_{0}}^{T} \! c_{R}(s)\vert x_{i}(s)-y_{i}(s) \vert \, \mathrm {d} s\\&\quad + \, \omega (\vert \tilde{x_{i}}(T) - x_{i}(T) \vert ) +\omega ( \vert x_{i}(T)-y_{i}(T) \vert ). \end{aligned}$$

Thus, for some $\bar{M} >0$ and $\bar{c} >0$ independent from i and for all $y_{1}, y_{2} \in B(0, Q) \cap K$,

$$\begin{aligned} V_{i}(t_{0},y_{2})-V_{i}(t_{0},y_{1}) \le \bar{M} \vert y_{2}-y_{1} \vert + \omega (\bar{c} \vert y_{2}-y_{1} \vert ). \end{aligned}$$

For all $s \ge 0$ define $\omega _{Q}^{\prime }(s) := \bar{M} s + \omega (\bar{c} s)$. Therefore,

$$\begin{aligned} V_{i}(t_{0},y_{2})-V_{i}(t_{0},y_{1}) \le \omega _{Q}^{\prime }(\vert y_{2}-y_{1} \vert ). \end{aligned}$$

Interchanging the roles of $y_{1}, y_{2}$, we deduce that

$$\begin{aligned} \vert V_{i}(t_{0},y_{2})-V_{i}(t_{0},y_{1}) \vert \le \omega _{Q}^{\prime }(\vert y_{2} - y_{1} \vert ), \end{aligned}$$

(12)

this ends the proof of the first inequality in (11).

Now let us show that $V_{i} \mid _{[0, T] \times (B(0, Q) \cap K_{i})}$ are equicontinuous with respect to the time variable too. Consider $y_{0} \in B(0, Q) \cap K_{i}$, $0 \le t_{0} < t_{1} \le T$ and $(x_{i}(\cdot ), u_{i}(\cdot )) \in S^{i}_{[t_{0},T]}(y_{0})$ such that

$$\begin{aligned} V_{i}(t_{0},y_{0}) = \int _{t_{0}}^{T} \! l_{i}(s,x_{i}(s),u_{i}(s)) \, \mathrm {d} s + \varphi _{i} (x_{i}(T)). \end{aligned}$$

We have that for some $\bar{c} >0$ independent from i and $y_{0}$,

$$\begin{aligned} \vert x_{i}(t_{1})-y_{0} \vert = \vert x_{i}(t_{1}) -x_{i}(t_{0}) \vert \le \bar{c} \vert t_{1} -t_{0} \vert . \end{aligned}$$

Since $y_{0} \in B(0, Q) \cap K$, we deduce that $x_{i}(t_{1}) \in B(0, Q + \bar{c} T)$. By the dynamic programming principle

$$\begin{aligned} V_{i}(t_{0},y_{0}) = \int _{t_{0}}^{t_{1}} \! l_{i}(s,x_{i}(s),u_{i}(s)) \, \mathrm {d} s + V_{i} (t_{1},x_{i}(t_{1})). \end{aligned}$$

Hence,

$$\begin{aligned} V_{i} (t_{1},x_{i}(t_{1})) - V_{i}(t_{0},y_{0}) = - \int _{t_{0}}^{t_{1}} \! l_{i}(s,x_{i}(s),u_{i}(s)) \, \mathrm {d} s. \end{aligned}$$

(13)

Therefore,

$$\begin{aligned} \vert V_{i}(t_{0},y_{0})- V_{i}(t_{1},y_{0}) \vert\le & {} \vert V_{i}(t_{0},y_{0})- V_{i} (t_{1},x_{i}(t_{1})) \vert + \vert V_{i} (t_{1},x_{i}(t_{1}))- V_{i}(t_{1},y_{0}) \vert \\\le & {} \int _{t_{0}}^{t_{1}} \! \vert l_{i}(s,x_{i}(s),u_{i}(s)) \vert \, \mathrm {d} s + \vert V_{i} (t_{1},x_{i}(t_{1}))- V_{i}(t_{1},y_{0}) \vert . \end{aligned}$$

We have that $l_{i}$ are equibounded on compacts. Therefore, for some $c>0$ independent from i by (13), we deduce that

$$\begin{aligned} \int _{t_{0}}^{t_{1}} \! \vert l_{i}(s,x_{i}(s),u_{i}(s)) \vert \, \mathrm {d} s \le c\vert t_{1}-t_{0} \vert . \end{aligned}$$

Let $\bar{Q}$ be such that every trajectory $x(\cdot )$ of the control system $\dot{x}(t) = f_{i}(t,x(t),u(t))$, $u(t) \in U$ with $x([0,T]) \cap B(0,Q) \ne \emptyset$ satisfies $x([0,T]) \subset B(0, \bar{Q}).$ According to (12), it follows that

$$\begin{aligned} \vert V_{i}(t_{1},x_{i}(t_{1}))- V_{i}(t_{1},y_{0}) \vert \le \omega _{\bar{Q}}^{\prime } ( \vert x_{i}(t_{1})-y_{0} \vert ). \end{aligned}$$

Thus,

$$\begin{aligned} \vert V_{i}(t_{0},y_{0}) - V_{i}(t_{1},y_{0}) \vert \le c \vert t_{1} -t_{0} \vert +\omega _{\bar{Q}}^{\prime } ( \vert x_{i}(t_{1}) -y_{0} \vert ). \end{aligned}$$

Therefore,

$$\begin{aligned} \vert V_{i}(t_{0},y_{0}) - V_{i}(t_{1},y_{0}) \vert \le c\vert t_{1}-t_{0} \vert + \omega _{\bar{Q}}^{ \prime } ( \bar{c} \vert t_{1}-t_{0} \vert ). \end{aligned}$$

We set $\omega ^{\prime \prime }(s) := c s + \omega ^{\prime }_{\bar{Q}}(\bar{c} s)$ for all $s \ge 0$. Hence, we have proved that, for all $0 \le t_{0} < t_{1} < T$,

$$\begin{aligned} \vert V_{i}(t_{0},y_{0}) - V_{i}(t_{1},y_{0}) \vert \le \omega ^{\prime \prime }(\vert t_{1}-t_{0} \vert ) . \end{aligned}$$

Therefore, $V_{i} \mid _{[0,T] \times (B(0, Q) \cap K_{i})}$ are equicontinuous uniformly in i, for any $Q >0$.

Fix $(t_{0},x_{0}) \in [0,T] \times intK$. Let $r >0$ be such that $x_{0}+rB \subset K$ and $y_{0} \in x_{0} + \dfrac{r}{2} B$. We claim that

$$\begin{aligned} \lim _{i \rightarrow \infty } V_{i}(t_{0},y_{0}) = V(t_{0},y_{0}). \end{aligned}$$

First we will show that

$$\begin{aligned} V(t_{0},y_{0}) \le \liminf _{i \rightarrow \infty } V_{i}(t_{0},y_{0}). \end{aligned}$$

Let $i(y_{0})$ be as in Proposition 3.3. It is well known that, taking into account Lemma 3.1, under assumptions of Theorem 3.1, there exist $(x_{i},u_{i}) \in S^{i}_{[t_{0},T]}(y_{0})$, for any $i \ge i_{0}$ such that

$$\begin{aligned} V_{i}(t_{0},y_{0})= \varphi _{i}(x_{i}(T))+\int _{t_{0}}^{T} \! l_{i}(s,x_{i}(s),u_{i}(s)) \, \mathrm {d} s . \end{aligned}$$

Consider a subsequence $V_{i_{j}}$ such that

$$\begin{aligned} \liminf _{i \rightarrow \infty } V_{i}(t_{0},y_{0})=\lim _{j \rightarrow \infty }V_{i_{j}}(t_{0},y_{0}). \end{aligned}$$

By (A2), we may assume that $i_{j}$ are such that $x_{i_{j}}$ converge uniformly on $[t_{0},T]$ to an absolutely continuous function $x \in W^{1,1}([t_{0},T];\mathbb {R}^{n})$, $\dot{x}_{i_{j}}(\cdot )$ converge weakly in $L^{1}$ to $\dot{x}$, and

$$\begin{aligned} \xi _{j}(\cdot ) := l_{i_{j}}(\cdot ,x_{i_{j}}(\cdot ),u_{i_{j}}(\cdot )) \end{aligned}$$

converges weakly in $L^{1}$ to some $\psi (\cdot )$. Then,

$$\begin{aligned} \int _{t_{0}}^{T} \! l_{i_{j}}(s,x_{i_{j}}(s),u_{i_{j}}(s)) \, \mathrm {d} s \rightarrow \int _{t_{0}}^{T} \! \psi (s) \, \mathrm {d} s. \end{aligned}$$

By our assumptions for any $R >0$ and for every $\varepsilon >0$, there exists $i_{0} \ge 1$ such that for any $i \ge i_{0}$, $t \in [0,T]$, $x \in RB$, $u \in U$ and $\varepsilon >0$ we have that $\vert l_{i}(t,x,u)-l(t,x,u) \vert \le \varepsilon$, $\vert f_{i}(t,x,u)-f(t,x,u) \vert \le \varepsilon$.

We fix $\varepsilon >0$ and denote $G_{\varepsilon }(t,x) := G(t,x)+\varepsilon B.$ Then, $G_{\varepsilon }(t,x)$ is closed and convex. As $x_{i_{j}}(\cdot ) \rightarrow x(\cdot )$ uniformly on $[t_{0},T]$, there exists $R >0$ such that $\Vert x_{i_{j}}(\cdot ) \Vert _{\infty } \le R$ for all j. Using Lipschitzianity assumptions (A1), we deduce that for all sufficiently large j and all $t \in [t_{0},T]$,

$$\begin{aligned} (f_{i_{j}}(t,x_{i_{j}}(t),u_{i_{j}}(t)), l_{i_{j}}(t,x_{i_{j}}(t),u_{i_{j}}(t))) \in G_{\varepsilon }(t,x(t))+2c_{R}(t) \vert x_{i_{j}}(t)-x(t) \vert B. \end{aligned}$$

For all $t \in [t_{0},T]$, the sets $Q_{\varepsilon }(t) := G_{\varepsilon }(t,x(t))+2c_{R}(t) \varepsilon B$ are convex and closed. Thus, the set $\lbrace v(\cdot ) \in L^{1}([t_{0},T];\mathbb {R}^{n}): v(t) \in Q_{\varepsilon }(t),\forall \ t \in [t_{0}, T] \rbrace$ is convex and closed in $L^{1}$. By the Mazur theorem (applied in $L^{1}$), it follows $(\dot{x}(s), \psi (s)) \in Q_{\varepsilon }(s)$ a.e., since $\varepsilon >0$ is arbitrary, we get $(\dot{x}(s), \psi (s)) \in G(s,x(s))$ a.e. in $[t_{0},T]$. By the measurable selection theorem, there exist a measurable selection $u(s) \in U$ and $\lambda (s) \ge 0$

$$\begin{aligned} \dot{x}(s)= & {} f(s,x(s),u(s)), \\ \psi (s)= & {} l(s,x(s),u(s))+\lambda (s). \end{aligned}$$

Since $\psi (\cdot ) \in L^{1}$ and l is bounded on compacts, $\lambda (\cdot )$ is integrable. Note that, as $(x_{i},u_{i}) \in S^{i}_{[t_{0},T]}(y_{0})$, for any $i \ge i_{0}$ and $x_{i_{j}}(\cdot ) \rightarrow x(\cdot )$ uniformly on $[t_{0},T]$, hence $x(t) \in K$, for any $t \in [t_{0},T]$. We have that

$$\begin{aligned} \lim _{j \rightarrow \infty }V_{i_{j}}(t_{0},y_{0})=\varphi (x(T))+\int _{t_{0}}^{T} \! l(s,x(s),u(s)) \, \mathrm {d} s+ \int _{t_{0}}^{T} \! \lambda (s) \, \mathrm {d} s \ge V(t_{0},y_{0}). \end{aligned}$$

We show next that $V(t_{0},y_{0}) \ge \limsup _{i \rightarrow \infty } V_{i}(t_{0},y_{0}).$ Let $(\bar{x}(\cdot ), \bar{u}(\cdot )) \in S_{[t_{0},T]}(y_{0})$ be such that

$$\begin{aligned} V(t_{0},y_{0}) = \varphi (\bar{x}(T))+ \int _{t_{0}}^{T} \! l(s,\bar{x}(s),\bar{u}(s)) \, \mathrm {d} s, \end{aligned}$$

and for almost all $s \in [t_{0},T]$,

$$\begin{aligned} \dot{\bar{x}}(s)= & {} f(s,\bar{x}(s),\bar{u}(s)), \\ \dot{z}(s)= & {} l(s,\bar{x}(s),\bar{u}(s)), \\ \bar{x}(t_{0})= & {} y_{0}, \\ z(t_{0})= & {} 0, \\ \bar{x}(s)\in & {} K, \ s \in [t_{0}, T]. \end{aligned}$$

Then, $(\bar{x}(s),z(s)) \in K \times \mathbb {R}$, for all $s \in [t_{0},T]$. Consider the solutions $x_{i}(\cdot )$ of

$$\begin{aligned} \dot{x}_{i}(s)= & {} f_{i}(s,x_{i}(s),\bar{u}(s)), \\ \dot{z}_{i}(s)= & {} l_{i}(s,x_{i}(s),\bar{u}(s)), \\ x_{i}(t_{0})= & {} y_{0}, \\ z_{i}(t_{0})= & {} 0, \end{aligned}$$

for $i = 1,2,\ldots$ Observe that for any $\varepsilon > 0$, there exists $\bar{i}_{0}> 0$, such that for any $i > \bar{i}_{0}$, we have $\vert x_{i}-\bar{x} \vert _{\infty } \le \varepsilon$ and $\vert z_{i}-z \vert _{\infty } \le \varepsilon$. Let R be such that $R > \vert \bar{x} \vert _{\infty }$. For any $\delta >0$ and for any sufficiently large i, by triangular inequality, it follows that

$$\begin{aligned} \mathrm{dist}((x_{i}(s),z_{i}(s)),( K_{i} \times \mathbb {R} ) )= & {} \mathrm{dist}(x_{i}(s), K_{i}) \le \mathrm{dist}(x_{i}(s),K_{i} \cap (RB+ \delta B)) \\\le & {} \mathrm{dist}(x_{i}(s),(K_{i} \cap (RB+ \delta B))+ \delta B) + \delta . \end{aligned}$$

Fix $\delta >0$. From Proposition 3.1, it follows that there exists $i_{0} > 0$, such that, for any $i > i_{0}$,

$$\begin{aligned} \mathrm{dist}(x_{i}(s),(K_{i} \cap (RB+ \delta B))+ \delta B) \le \mathrm{dist}(x_{i}(s),K \cap RB). \end{aligned}$$

Hence, for all sufficiently large i

$$\begin{aligned} \mathrm{dist}((x_{i}(s),z_{i}(s)),( K_{i} \times \mathbb {R} ) ) \le \mathrm{dist}(x_{i}(s),K \cap RB) +\delta . \end{aligned}$$

Consequently, for any $\delta >0$, there exists $i_{0}$, such that, for any $i > i_{0}$,

$$\begin{aligned} \mathrm{dist}((x_{i}(s),z_{i}(s)),( K_{i} \times \mathbb {R})) \le 2 \delta . \end{aligned}$$

By Lemma 3.1 and Theorem 2.1 applied to

$$\begin{aligned} F(t,x,z) = \lbrace (f_{i}(t,x,u), l_{i}(t,x,u)), u \in U \rbrace \end{aligned}$$

there exists $C >0$ independent from i such that, for all sufficiently large i, we can find absolutely continuous $(\tilde{x}_{i}(\cdot ), \tilde{z}_{i}(\cdot ))$ such that $(\dot{\tilde{x}}_{i}(t), \dot{\tilde{z}}_{i}(t)) \in F(t, \tilde{x}_{i}(t), \tilde{z}_{i}(t))$ a.e. in $[t_{0}, T]$, $\tilde{x}_{i}(t_{0}) = y_{0}$, $\tilde{z}_{i}(t_{0}) =0$ satisfying state constraints $(\tilde{x}_{i}(t),\tilde{z}_{i}(t)) \in K_{i} \times \mathbb {R},$ such that

$$\begin{aligned} \Vert \tilde{z}_{i}- z_{i} \Vert _{\infty } + \Vert \tilde{x}_{i}- x_{i} \Vert _{\infty } \le C \max _{s \in [t_{0},T]} \mathrm{dist}(x_{i}(s), K_{i}). \end{aligned}$$

We have that for any $\delta > 0$,

$$\begin{aligned} \max _{s \in [t_{0},T]} \mathrm{dist}(x_{i}(s), K_{i}) \le \max _{s \in [t_{0},T]} \mathrm{dist}(x_{i}(s), K_{i} \cap (R + \delta ) B). \end{aligned}$$

From Proposition 3.1, we deduce for any $\delta > 0$ and all sufficiently large i that

$$\begin{aligned} \max _{s \in [t_{0},T]} \mathrm{dist}(x_{i}(s), K_{i}) \le \max _{s \in [t_{0},T]} \mathrm{dist}(x_{i}(s), K \cap RB) + \delta \le \Vert x_{i} - \bar{x} \Vert _{\infty } + \delta \le \varepsilon + \delta . \end{aligned}$$

Thus, taking $\delta = \varepsilon$, we deduce that

$$\begin{aligned} \Vert \tilde{z}_{i}- z_{i} \Vert _{\infty } + \Vert \tilde{x}_{i}- x_{i} \Vert _{\infty } \le 2 C \varepsilon . \end{aligned}$$

Consider measurable $\tilde{u}_{i}(\cdot ) : [t_{0},T] \rightarrow U$ such that

$$\begin{aligned} \dot{\tilde{x}}_{i}(s)= & {} f_{i}(s,\tilde{x}_{i}(s),\tilde{u}_{i}(s)) \ \mathrm{a.e.} \ in \ [t_{0},T], \\ \dot{\tilde{z}}_{i}(s)= & {} l_{i}(s,\tilde{x}_{i}(s),\tilde{u}_{i}(s)) \ \mathrm{a.e.} \ in \ [t_{0}, T]. \end{aligned}$$

For any $\varepsilon >0$, there exists $\tilde{i}_{0} > 0$, such that for any $i > \tilde{i}_{0}$, we have

$$\begin{aligned} \vert \varphi _{i}(\tilde{x_{i}}(T))-\varphi (\bar{x}(T)) \vert \le \varepsilon \end{aligned}$$

and

$$\begin{aligned} \left| \int _{t_{0}}^{T} \! l_{i}(s,\tilde{x_{i}}(s),\tilde{u_{i}}(s)) \, \mathrm {d} s - \int _{t_{0}}^{T} \! l(s,\bar{x}(s),\bar{u}(s)) \, \mathrm {d} s \right| \le \varepsilon . \end{aligned}$$

Hence, we obtain

$$\begin{aligned} V(t_{0},y_{0})= & {} \int _{t_{0}}^{T} \! l(s,\bar{x}(s),\bar{u}(s)) \, \mathrm {d} s + \varphi (\bar{x}(T)) \\\ge & {} \int _{t_{0}}^{T} \! l_{i}(s,\tilde{x_{i}}(s),\tilde{u_{i}}(s)) \, \mathrm {d} s + \varphi _{i}(\tilde{x}_{i}(T)) - 2\varepsilon \ge V_{i}(t_{0},y_{0}) - 2\varepsilon . \end{aligned}$$

Thus,

$$\begin{aligned} V(t_{0},y_{0}) \ge \limsup _{i \rightarrow \infty } V_{i}(t_{0},y_{0}) - 2\varepsilon . \end{aligned}$$

The above being valid for any $\varepsilon > 0$; therefore, we get

$$\begin{aligned} V(t_{0},y_{0}) \ge \limsup _{i \rightarrow \infty } V_{i}(t_{0},y_{0}). \end{aligned}$$

Hence, we have that for any $Q>0$, $V_{i}$ are equicontinuous uniformly in i on $[0,T] \times (B(0,Q) \cap K_{i})$ and converging pointwise to V on $[0, T] \times B(x_{0}, \frac{r}{2})$; we deduce that the convergence is uniform. The proof is complete. $\square$

Corollary 3.1

Let the assumptions of Theorem 3.1 hold true. Then,

$$\begin{aligned}Lim_{i \rightarrow \infty } graph V_{i} = graph V,\end{aligned}$$

where the limit is taken in the Kuratowski sense.

Proof

We will first prove that $graph V \subset Liminf_{i \rightarrow \infty } graph V_{i}.$

Case 1 Let $(t,x) \in [0,T] \times int K$. We will show that

$$\begin{aligned}((t,x), V(t,x)) \in Liminf_{i \rightarrow \infty } graph V_{i}.\end{aligned}$$

Take any (relatively) open neighbourhood $\Omega$ of ((t, x), V(t, x)) in $[0,T] \times \mathbb {R}^{n} \times \mathbb {R}$. It is not restrictive to assume that $\Omega = W_{0} \times U_{0}$, where $W_{0}$ is an open neighbourhood of (t, x) and $U_{0}$ is an open neighbourhood of V(t, x).

By Theorem 3.1 for all $x \in int K$ and $r >0$ such that $x + r B \subset K$, we have $V_{i}( \cdot , \cdot ) \rightarrow V( \cdot , \cdot )$ uniformly on $[0,T] \times B(x, \frac{r}{2})$, when $i \rightarrow \infty$, thus there exists an open neighbourhood $W_{1}$ of (t, x) and there exists $i_{0} \ge 1$ such that, for any $(s,y) \in W_{1}$ and any $i \ge i_{0}$, $V_{i}( s, y) \in U_{0}$. Therefore,

$$\begin{aligned} ((W_{1} \cap W_{0}) \times U_{0}) \cap graph V_{i} \ne \emptyset , \end{aligned}$$

for any $i \ge i_{0}.$ We deduce that $\Omega \cap graph V_{i} \ne \emptyset$, for any $i \ge i_{0}.$ Hence, for any $(t,x) \in [0,T] \times int K$,

$$\begin{aligned} ((t,x), V(t,x)) \in Liminf_{i \rightarrow \infty } graph V_{i}. \end{aligned}$$

Case 2 Let $(t,x) \in [0,T] \times \partial K$. Take any open neighbourhood $\Omega$ of ((t, x), V(t, x)). It is not restrictive to assume that $\Omega = W_{0} \times U_{0}$, where $W_{0}$ is an open neighbourhood of (t, x) and $U_{0}$ is an open neighbourhood of V(t, x). There exists $x_{1} \in int K$, such that $(t, x_{1}) \in W_{0}$ and $V(t, x_{1}) \in U_{0}$ (by continuity of $V(t, \cdot )$ on K). Thus, we can choose $W_{1}$, an open neighbourhood of $(t,x_{1})$, and $U_{1}$, an open neighbourhood of $V(t,x_{1})$, such that $W_{1} \times U_{1} \subseteq W_{0} \times U_{0}$. Consider $\Omega _{1} = W_{1} \times U_{1}$, then $\Omega _{1} \subseteq \Omega$. As $x_{1} \in int K$, by the result of Case 1, we have that there exists $i_{0} \ge 1$ such that $\Omega _{1} \cap graph V_{i} \ne \emptyset$, for any $i \ge i_{0}.$ Therefore, $\Omega \cap graph V_{i} \ne \emptyset$, for any $i \ge i_{0}.$ Hence, for any $(t,x) \in [0,T] \times \partial K$, we have $((t,x), V(t,x)) \in Liminf_{i \rightarrow \infty } graph V_{i}.$

Combining the results of Case 1 and Case 2, we deduce that

$$\begin{aligned} graph V \subset Liminf_{i \rightarrow \infty } graph V_{i}. \end{aligned}$$

(14)

In order to complete the proof, let us now prove that

$$\begin{aligned}graph V \supset Limsup_{i \rightarrow \infty } graph V_{i}.\end{aligned}$$

Take any $\omega \in Limsup_{i \rightarrow \infty } graph V_{i}$, thus for any open neighbourhood $Q \ni \omega$, we have $Q \cap graph V_{i} \ne \emptyset$, for infinitely many i. Thus, $B( \omega , \frac{1}{k}) \cap graph V_{i} \ne \emptyset$, for infinitely many i, where $B( \omega , \frac{1}{k})$ is the ball of centre $\omega$ and with the radius $\frac{1}{k}$, for any $k > 0$. Hence, there exist $v_{i_{k}} \in B( \omega , \frac{1}{k}) \cap graph V_{i_{k}}$, such that $v_{i_{k}} = ((t_{i_{k}},x_{i_{k}}), V_{i_{k}}(t_{i_{k}},x_{i_{k}}))$, for some $(t_{i_{k}},x_{i_{k}}) \in [0,T] \times K$. Therefore,

$$\begin{aligned} \vert ((t_{i_{k}},x_{i_{k}}), V_{i_{k}}(t_{i_{k}},x_{i_{k}}))- \omega \vert < \frac{1}{k}. \end{aligned}$$

Let $v \in \mathbb {R}$ be such that $\omega = ((t,x), v)$, for some $(t,x) \in [0,T] \times \mathbb {R}^{n}$. Hence, for any $k >0$, we have that

$$\begin{aligned}&\vert x_{i_{k}} - x \vert < \dfrac{1}{k}, \nonumber \\&\vert t_{i_{k}} - t \vert < \dfrac{1}{k}, \nonumber \\&\vert V_{i_{k}}(t_{i_{k}}, x_{i_{k}}) -v \vert < \dfrac{1}{k}. \end{aligned}$$

(15)

By (14), it follows that there exist $(\bar{t}_{k}, \bar{x}_{k}) \in [0,T] \times K_{i_{k}}$, $(\bar{t}_{k}, \bar{x}_{k}) \rightarrow (t,x)$, when $k \rightarrow \infty$, such that

$$\begin{aligned} V_{i_{k}}(\bar{t}_{k}, \bar{x}_{k}) \rightarrow V(t,x). \end{aligned}$$

(16)

From (15), we have that when $k \rightarrow \infty$, then $(t_{i_{k}},x_{i_{k}}) \rightarrow (t,x)$. By triangular inequality

$$\begin{aligned}&\vert t_{i_{k}} - \bar{t}_{k} \vert \le \vert t_{i_{k}} - t \vert + \vert t - \bar{t}_{k} \vert , \nonumber \\&\vert x_{i_{k}} - \bar{x}_{k} \vert \le \vert x_{i_{k}} - x \vert + \vert x - \bar{x}_{k} \vert , \end{aligned}$$

(17)

$$\begin{aligned}&\vert V_{i_{k}}(t_{i_{k}},x_{i_{k}}) - V(t,x) \vert \le \vert V_{i_{k}}(t_{i_{k}}, x_{i_{k}})-V_{i_{k}}(\bar{t}_{k}, \bar{x}_{k}) \vert + \vert V_{i_{k}}(\bar{t}_{k}, \bar{x}_{k})-V(t,x) \vert .\nonumber \\ \end{aligned}$$

(18)

Since (by Theorem 3.1) $V_{i_{k}} \mid _{[0,T] \times K_{i_{i_{k}}}}$ are equicontinuous (in the sense of Definition 2.1), then by (18), (17) and (16), we deduce $\lim _{k \rightarrow \infty } V_{i_{k}}(t_{i_{k}}, x_{i_{k}})=V(t,x).$ By (15), we obtain that $v = V(t,x).$ Hence, $((t,x), V(t,x)) = \omega$, thus $\omega \in graph V.$ Thus, $Lim_{i \rightarrow \infty } graph V_{i} = Liminf_{i \rightarrow \infty } graph V_{i} =Limsup_{i \rightarrow \infty } graph V_{i} = graph V$, this ends the proof. $\square$

4 Hamilton–Jacobi–Bellman Equations and the Bolza Optimal Control Problem

Let K be a closed and nonempty subset of $\mathbb {R}^n$. Consider the Hamilton–Jacobi equation

$$\begin{aligned} (\textit{HJB}) {\left\{ \begin{array}{ll} -V_{t}(t,x)+H(t,x,-V_{x}(t,x))=0, \ (t,x) \in [0,T] \times K, \\ V(T,x)=\varphi (x), \end{array}\right. } \end{aligned}$$

(19)

with the Hamiltonian $[0,T] \times \mathbb {R}^{n} \times \mathbb {R}^{n} \ni (t,x,p) \rightarrow H(t,x,p)$.

Definition 4.1

For a map $H : [0, T] \times \mathbb {R}^{n} \times \mathbb {R}^{n} \rightarrow \mathbb {R},$ $H^{*}$ denotes the conjugate of H with respect to the third variable, i.e. for all $(t,x,v) \in [0,T] \times \mathbb {R}^{n} \times \mathbb {R}^{n}$,

$$\begin{aligned} H^{*}(t,x,v) := \sup _{p \in \mathbb {R}^{n}} \lbrace \langle v,p \rangle - H(t,x,p) \rbrace \in \mathbb {R} \cup \lbrace + \infty \rbrace . \end{aligned}$$

Assumptions.

(H1) $H(t,x,\cdot )$ is convex for any $(t,x) \in [0,T] \times \mathbb {R}^{n}$.

(H2) For any $R>0$, there exists an integrable $c_R : [0,T] \rightarrow \mathbb {R}_+$ such that, for all $x,y \in RB$, $t \in [0,T]$ and $p \in \mathbb {R}^{n}$,

$$\begin{aligned} \vert H(t,x,p)-H(t,y,p) \vert \le c_{R}(t) (1+\vert p \vert )\vert x-y \vert . \end{aligned}$$

(H3) There exists $c > 0$ such that

$$\begin{aligned}\vert H(t,x,p)-H(t,x,q) \vert \le c(1+\vert x \vert )\vert p-q \vert \end{aligned}$$

for all $(t,x) \in [0,T] \times \mathbb {R}^{n}$ and $p,q \in \mathbb {R}^{n}$.

(H4) $H^*(t,x,\cdot )$ is bounded on its domain for all $(t,x) \in [0,T] \times \mathbb {R}^n$.

(H5) For every $R>0$, there exists $M_R > 0$ such that, for all $(t,x) \in [0,T] \times RB$ and $v \in dom(H^*(t,x,\cdot ))$ we have

$$\begin{aligned} H^*(t,x,v)=\max _{p \in B(0,M_R)}(\langle v,p\rangle - H(t,x,p)). \end{aligned}$$

(H6) For every $R>0$, there exists an absolutely continuous $a_{R}: [0,T] \rightarrow \mathbb {R}$ such that, for all $x \in RB, \; p \in \mathbb {R}^{n}$ and $t,\,s \in [0,T]$,

$$\begin{aligned} \vert H(t,x,p)-H(s,x,p) \vert \le (1+\vert p \vert ) \vert a_{R}(t)- a_{R}(s) \vert . \end{aligned}$$

Definition 4.2

A continuous function $W: [0, T] \times K \rightarrow \mathbb {R}$ is called a viscosity solution of (19) iff $W(T, \cdot ) = \varphi (\cdot )$ and

(i) for all $(s,x) \in ]0, T[ \times K$ and all $(p_{s}, p_{x}) \in \partial _{-} W (s, x)$,

$$\begin{aligned} -p_{s} + H(s, x, -p_{x}) \ge 0. \end{aligned}$$

(ii) for all $(s,x) \in ]0, T[ \times int K$ and all $(p_{s},p_{x}) \in \partial _{+} W(s, x)$,

$$\begin{aligned} -p_{s} + H(s, x, -p_{x}) \le 0. \end{aligned}$$

Frankowska and Sedrakyan [13] have shown that, if (H1)–(H6) hold true, then there exist $f : [0,T] \times \mathbb {R}^{n} \times B \rightarrow \mathbb {R}^{n}$ and $l:[0,T] \times \mathbb {R}^{n} \times B \rightarrow \mathbb {R}$ satisfying (A1)–(A2) with $U = B$ and such that $f(t,x,B) = dom H^{*} (t,x, \cdot )$,

$$\begin{aligned} H(t,x,p) = \max _{u \in B} (\langle p, f(t,x,u) \rangle - l(t,x,u)). \end{aligned}$$

Moreover, $G(t,x) = \lbrace (f(t,x,u), l(t, x, u) + r): u \in B, r \ge 0 \rbrace$ is convex and closed. Let V be the value function defined in Sect. 2 for f,l and U as above. We impose the following assumption.

(A4) $_{H}.$ For any $R >0$, there exist $\rho _{R} > 0$ such that, for every $x \in K \cap RB$ with $I(x) \ne \emptyset$ and every $t \in [0,T]$,

$$\begin{aligned} \inf _{v \in dom(H^{*}(t,x,\cdot ))} \max _{j \in I(x)} \langle \nabla g^{j}(x),v \rangle \le - \rho _{R}. \end{aligned}$$

Proposition 4.1

Let assumption (A4)$_H$ hold true. Then, for all $(s,x) \in [0, T[ \times K$ and all $(p_{s}, p_{x}) \in \partial _{-} V (s, x)$,

$$\begin{aligned} -p_{s} + H(s, x, -p_{x}) \ge 0. \end{aligned}$$

Proof

Fix $(t_{0},x_{0}) \in [0,T[ \times K$. By a straightforward, but somewhat technical argument, one can deduce that the assumption (A4)$_H$ implies that the optimal control (P) admit solutions. Let $(\bar{x}(\cdot ), \bar{u}(\cdot ))$ be optimal for (P) at $(t_{0}, x_{0})$; therefore

$$\begin{aligned} V(t, \bar{x}(t)) = V(t_{0}, \bar{x}(t_{0}))-\int _{t_{0}}^{t} \! l(s,\bar{x}(s),\bar{u}(s)) \, \mathrm {d} s. \end{aligned}$$

Take $t := t_{0} + h$ with $h >0$ small enough. Hence,

$$\begin{aligned} \dfrac{V(t_{0}+h, \bar{x}(t_{0}+h))-V(t_{0}, \bar{x}(t_{0}))}{h} = - \dfrac{1}{h} \int _{t_{0}}^{t_{0}+h} \! l(s,\bar{x}(s),\bar{u}(s)) \, \mathrm {d} s. \end{aligned}$$

(20)

We shall deduce that for some $(v, \gamma ) \in G(t_{0}, x_{0})$, $D_{\uparrow }V(t_{0}, \bar{x}(t_{0}))(1,v) \le -\gamma .$ For this aim consider $h_{i} \rightarrow 0+$, when $i \rightarrow \infty$ and $v \in \mathbb {R}^{n}$, $\gamma \in \mathbb {R}$, such that

$$\begin{aligned} \dfrac{\bar{x}(t_{0}+h_{i})-\bar{x}(t_{0})}{h_{i}} \rightarrow v, \ \ \ \dfrac{\int _{t_{0}}^{t_{0}+h_{i}} \! l(s,\bar{x}(s),\bar{u}(s)) \, \mathrm {d} s }{h_{i}} \rightarrow \gamma . \end{aligned}$$

(21)

We deduce from the continuity of f, l and (A2) that for any $\varepsilon > 0$, there exists $h_{0} > 0$ such that for any $s \in [t_{0}, t_{0}+h_{0}]$,

$$\begin{aligned}&(f(s, \bar{x}(s), \bar{u}(s)), l(s, \bar{x}(s), \bar{u}(s))) \subset (f(t_{0}, \bar{x}(t_{0}), \bar{u}(s)), l(t_{0}, \bar{x}(t_{0}), \bar{u}(s))) +\varepsilon B \\&\subset G(t_{0}, x_{0}) + \varepsilon B. \end{aligned}$$

Hence, $(v, \gamma ) \in G(t_{0}, x_{0})$. Thus, from Definition 2.2, (20), (21), we deduce that

$$\begin{aligned} D_{\uparrow }V(t_{0}, x_{0})(1,v) \le -\gamma . \end{aligned}$$

(22)

By definition of $G(\cdot , \cdot )$, there exists $u_{0}$ and $r_{0} \ge 0$ such that

$$\begin{aligned} v= & {} f(t_{0}, x_{0}, u_{0}),\nonumber \\ \gamma= & {} l(t_{0}, x_{0},u_{0})+ r_{0}. \end{aligned}$$

(23)

By (22), (23), we obtain $D_{\uparrow }V(t_{0}, x_{0})(1, f(t_{0}, x_{0}, u_{0})) \le -l(t_{0}, x_{0}, u_{0}) - r_{0} \le -l(t_{0}, x_{0}, u_{0}).$ For any $(p_{s}, p_{x}) \in \partial _{-}V(t_{0}, x_{0})$, using the Lemma 2.2, we obtain that

$$\begin{aligned} p_{s} \cdot 1 + \langle p_{x}, f(t_{0}, x_{0}, u_{0}) \rangle \le D_{\uparrow } V(t_{0}, x_{0}) (1, f(t_{0}, x_{0}, u_{0})) \le -l(t_{0}, x_{0}, u_{0}). \end{aligned}$$

Hence, $-p_{s}+ \langle -p_{x}, f(t_{0}, x_{0}, u_{0}) \rangle -l(t_{0}, x_{0}, u_{0}) \ge 0,$ and we obtain

$$\begin{aligned} -p_{s}+\sup _{u \in B} (\langle -p_{x}, f(t_{0}, x_{0}, u) \rangle -l(t_{0}, x_{0}, u)) \ge 0. \end{aligned}$$

Therefore, for any $(p_{s}, p_{x}) \in \partial _{-} V (t_{0}, x_{0})$, we have $-p_{s} + H(t_{0}, x_{0}, -p_{x}) \ge 0.$

Since $(t_{0}, x_{0}) \in [0, T[ \times K$ is arbitrary, we end the proof. $\square$

Proposition 4.2

For all $(s,x) \in [0, T[ \times int K$ and all $(p_{s}, p_{x}) \in \partial _{+} V (s, x)$,

$$\begin{aligned} -p_{s} + H(s, x, -p_{x}) \le 0. \end{aligned}$$

Proof

Fix $u_{0} \in B$ and consider the solution $x(\cdot )$ of

$$\begin{aligned} \dot{x}(s)= & {} f(s, x(s), u_{0}), \\ x(t_{0})= & {} x_{0}. \end{aligned}$$

Then,

$$\begin{aligned} V(t_{0}+h, x(t_{0}+h)) \ge V(t_{0}, x_{0}) - \int _{t_{0}}^{t_{0}+h} \! l(s, x(s), u_{0}) \, \mathrm {d} s. \end{aligned}$$

We have that for $h \rightarrow 0+$,

$$\begin{aligned} \dfrac{x(t_{0}+h)-x_{0}}{h} \rightarrow f(t_{0}, x_{0}, u_{0}) \end{aligned}$$

and

$$\begin{aligned} \dfrac{1}{h} \int _{t_{0}}^{t_{0}+h} \! l(s, x(s), u_{0}) \, \mathrm {d} s \rightarrow l(t_{0}, x_{0}, u_{0}). \end{aligned}$$

By Lemma 2.2, for any $(p_{s}, p_{x}) \in \partial _{+} V(t_{0}, x_{0})$, we have that

$$\begin{aligned} \langle (p_{s}, p_{x}),(1, f(t_{0}, x_{0}, u_{0})) \rangle \ge D_{\downarrow } V(t_{0}, x_{0}) (1, f(t_{0}, x_{0}, u_{0})) \ge -l(t_{0}, x_{0}, u_{0}). \end{aligned}$$

Hence, we have obtained that for any $(p_{s},p_{x}) \in \partial _{+}V(t_{0}, x_{0})$ and $u_{0} \in B$,

$$\begin{aligned} -p_{s}+ \langle -p_{x}, f(t_{0}, x_{0}, u_{0}) \rangle -l(t_{0}, x_{0}, u_{0}) \le 0, \end{aligned}$$

and therefore, for any $(p_{s},p_{x}) \in \partial _{+}V(t_{0}, x_{0})$, we have $-p_{s} + H(t_{0}, x_{0}, -p_{x}) \le 0.$ Since $(t_{0}, x_{0}) \in [0, T[ \times K$ is arbitrary, we end the proof. $\square$

Theorem 4.1

If assumptions (H1)–(H6) hold true, then the value function of the Bolza optimal control problem (3) is a viscosity solution of the Hamilton–Jacobi equation (19).

Proof

By Theorem 3.1, the value function is continuous on $[0,T] \times K$. According to Definition 4.2 and Proposition 4.1, the value function is a viscosity supersolution of Hamilton–Jacobi equation, and by Proposition 4.2, the value function is a viscosity subsolution of Hamilton–Jacobi equation; thus, it is a viscosity solution. This ends the proof. $\square$

5 Uniqueness of Solutions of Hamilton–Jacobi Equation and Their Continuous Dependence on Data

Theorem 5.1

Let assumptions (H1)–(H6) and (A4)$_H$ hold true. Then, there exists the unique viscosity solution of Hamilton–Jacobi equation (19) on $[0, T] \times K$.

Proof

We provide a complete proof, since in [3] there are no state constraints, while in [5] the Mayer problem instead of the Bolza one is considered.

Frankowska and Sedrakyan [13] have shown that, if (H1)–(H6) hold true for H, then there exist $f : [0,T] \times \mathbb {R}^{n} \times B \rightarrow \mathbb {R}^{n}$ and $l:[0,T] \times \mathbb {R}^{n} \times B \rightarrow \mathbb {R}$ satisfying (A1)–(A2) with $U = B$ and such that $H(t,x,p) = \max _{u \in B} (\langle p, f(t,x,u) \rangle - l(t,x,u)).$ Moreover, $G(t,x) = \lbrace (f(t,x,u), l(t, x, u) + r): u \in B, r \ge 0 \rbrace$ is convex and closed. We consider Bolza optimal control problem (3) with $U=B$ and the associated value function. By Theorem 4.1, we know that the value function is a viscosity solution of the Hamilton–Jacobi equation. Let W be a viscosity solution of (19). We will show that $W = V$ on $[0, T] \times K$. We proceed in two steps.

Step 1 We will show first that for any $(t_{0}, x_{0}) \in [0, T] \times K$, it holds true $W(t_{0}, x_{0}) \ge V(t_{0}, x_{0})$. Since W is a viscosity solution, by Definition 4.2, we have

$$\begin{aligned}&\forall (t,x) \in ]0, T[ \times K, \ \forall (p_{t}, p_{x}) \in \partial _{-} W (t, x), \nonumber \\&-p_{t} + \sup \limits _{u \in B} (\langle -p_{x}, f(t,x,u) \rangle - l(t,x,u)) \ge 0. \end{aligned}$$

(24)

If for some $(t,x) \in ]0,T[ \times K$ and $z \ge W(t,x)$, $(p_{t}, p_{x}, q) \in N_{epi(W)}(t,x,z),$ then $(p_{t}, p_{x}, q) \in N_{epi(W)}(t,x,W(t,x))$. By Lemma 4.2, [3], there exist $(t_{i}, x_{i}) \in ]0,T[ \times K$, such that $(t_{i}, x_{i}) \rightarrow (t,x)$, when $i \rightarrow \infty$ and

$$\begin{aligned} (p_{t}^{i}, p_{x}^{i}, q_{i}) \in N_{epi(W)} (t_{i}, x_{i}, W(t_{i}, x_{i})), \end{aligned}$$

(25)

where $q_{i} < 0$ and such that $(p_{t}^{i}, p_{x}^{i}, q_{i}) \rightarrow (p_{t}, p_{x}, q), \ when \ i \rightarrow \infty .$ Therefore, as $q_{i} < 0$, we deduce from (25) that

$$\begin{aligned} \Big ( \dfrac{p_{t}^{i}}{\vert q_{i} \vert }, \dfrac{p_{x}^{i}}{\vert q_{i} \vert }, -1 \Big ) \in N_{epi(W)}(t_{i}, x_{i}, W(t_{i}, x_{i})). \end{aligned}$$

Hence, by Proposition 4.1, [3], we obtain that

$$\begin{aligned} \Big ( \dfrac{p_{t}^{i}}{\vert q_{i} \vert }, \dfrac{p_{x}^{i}}{\vert q_{i} \vert } \Big ) \in \partial _{-}W(t_{i}, x_{i}). \end{aligned}$$

(26)

From (24) and (26), we deduce that the following inequality holds true

$$\begin{aligned} -\dfrac{p_{t}^{i}}{\vert q_{i} \vert } + \sup _{u \in B} \Big ( \langle -\dfrac{p_{x}^{i}}{\vert q_{i} \vert }, f(t_{i}, x_{i}, u) \rangle - l(t_{i}, x_{i}, u) \Big ) \ge 0, \end{aligned}$$

or equivalently, $-p_{t}^{i} + \sup _{u \in B} \Big ( \langle -p_{x}^{i}, f(t_{i}, x_{i}, u) \rangle - \vert q_{i} \vert l(t_{i}, x_{i}, u) \Big ) \ge 0.$ Passing to the limit when $i \rightarrow \infty$, by continuity of f and l, we obtain that

$$\begin{aligned}-p_{t} + \sup _{u \in B} \Big ( \langle -p_{x}, f(t, x, u) \rangle - \vert q \vert l(t, x, u) \Big ) \ge 0.\end{aligned}$$

Therefore,

$$\begin{aligned} p_{t} + \inf _{u \in B} \Big ( \langle p_{x}, f(t, x, u) \rangle + \vert q \vert l(t, x, u) \Big ) \le 0. \end{aligned}$$

(27)

Consider a solution x of

$$\begin{aligned} \dot{x}(s)= & {} f(s,x(s),u(s)), \ s \in [0,T], \ u(s) \in B, \nonumber \\ x(0)= & {} x_{0} \in RB \cap K. \end{aligned}$$

(28)

From (28) and (A2), together with the Gronwall lemma, it follows that there exists $c > 0$ such that $\sup _{t \in [t_{0}, T]} \vert x(t) \vert \le e^{cT} \vert x_{0} \vert < 2 e^{cT} R := \hat{R}.$ Therefore, any solution starting at $x_{0} \in B(0,R)$ and defined on $[t_{0}, T]$ stays in $\mathring{B}(0,\hat{R}).$ For any $(t,x,u) \in [0,T] \times B(0,2 \hat{R}) \times B$ denote by

$$\begin{aligned} M := \max _{(t,x,u) \in [0,T] \times B(0,2 \hat{R}) \times B} \vert l(t,x,u) \vert , \end{aligned}$$

as l is continuous and $[0,T] \times B(0,2 \hat{R}) \times B$ is a compact set; thus, $M >0$ is a constant, such that for any $(t,x,u) \in [0,T] \times B(0,2 \hat{R}) \times B$, we have $\vert l(t,x,u) \vert \le M.$ Define a set-valued map $F^{-} : [0,T] \times \mathbb {R}^{n} \times \mathbb {R} \rightrightarrows \mathbb {R} \times \mathbb {R}^{n} \times \mathbb {R}$ by

$$\begin{aligned} F^{-}(t,x,v) := \lbrace (1, f(t,x,u), -l(t,x,u)-r) \mid u \in B, r \in [0, M-l(t,x,u)] \rbrace , \end{aligned}$$

where M is as above. Note that $F^{-}$ has convex and compact images. Let us prove that

$$\begin{aligned} F^{-}(t,x,v) \cap cl(conv(T_{epi(W)} (t,x,z))) \ne \emptyset , \end{aligned}$$

(29)

for any $(t,x) \in ]0,T[ \times (K \cap B(0, e^{cT} R))$, $z \ge W(t,x).$

We proceed by a contradiction argument. Indeed, if (29) is not satisfied for some $(t,x,v) \in ]0,T[ \times (K \cap B(0, e^{cT} R)) \times B$, then by the separation theorem, there exists $0 \ne (p_{t}, p_{x}, q) \in \mathbb {R} \times \mathbb {R}^{n} \times \mathbb {R},$ such that

$$\begin{aligned}&\inf _{(\alpha , \beta , \gamma ) \in F^{-}(t,x,v)} \langle (\alpha , \beta , \gamma ),(p_{t}, p_{x}, q) \rangle >\nonumber \\&\sup _{w \in cl(conv(T_{epi(W)}(t,x,W(t,x))))} \langle w, (p_{t}, p_{x}, q) \rangle \ge 0. \end{aligned}$$

(30)

Note that, if we assume that the right-hand side of (30) is not equal to 0, then it is equal to $+\infty$ since the supremum is taken over a cone, leading to a contradiction because the left-hand side of (30) is bounded. Thus, we deduce that

$$\begin{aligned} \sup _{w \in cl(conv( T_{epi(W)}(t,x,W(t,x))))} \langle w, (p_{t}, p_{x}, q) \rangle =0. \end{aligned}$$

(31)

Hence, from (30) and (31), we obtain that, for all $r \in [0, M-l(t,x,u)]$,

$$\begin{aligned} p_{t} + \langle p_{x}, f(t,x,u) \rangle + q (-l(t,x,u)-r) > 0. \end{aligned}$$

(32)

From (31), it follows that

$$\begin{aligned} (p_{t}, p_{x}, q) \in N_{epi(W)}(t,x,W(t,x)). \end{aligned}$$

(33)

Therefore, from (33), we deduce that $q \le 0$; thus by (32), we obtain that

$$\begin{aligned} p_{t} + \langle p_{x}, f(t,x,u) \rangle + \vert q \vert (l(t,x,u)+r) > 0. \end{aligned}$$

Let us take $r = 0$, hence $p_{t} + \langle p_{x}, f(t,x,u) \rangle + \vert q \vert l(t,x,u) > 0.$ This leads to a contradiction with (27). Hence, (29) holds true. Consider the control system

$$\begin{aligned} (CS1) {\left\{ \begin{array}{ll} \dot{t}(s) = 1, \\ \dot{x}(s) = f(t_{0} + s, x(s) ,u(s)), \ u(s) \in B, \\ \dot{z}(s) = -l(t_{0} + s, x(s), u(s)) - r(s), \ r(s) \in [0, M-l(s, x(s), u(s)]. \end{array}\right. } \end{aligned}$$

(34)

We have that W is continuous; thus, epi(W) is closed. On the other hand, $F^{-}$ is continuous and has convex compact images; thus by Theorem 3.2.4, [14], and Local Viability Theorem 3.3.4, [14], we deduce that, for any $(t_{0}, x_{0}) \in ]0,T[ \times K$, there exists a solution $(t(\cdot ), x(\cdot ), z(\cdot ))$ of (CS1) on $[0,T-t_{0}]$ such that $t(0) = t_{0}$, $x(0) = x_{0}$, $z(0) = W(t_{0}, x_{0})$ and $(t(s), x(s), z(s)) \in epi (W),$ for any $s \in [0, T-t_{0}[.$ Therefore, we have, for any $s \in [0,T-t_{0}[$, that

$$\begin{aligned} z(s) \ge W(t(s), x(s)). \end{aligned}$$

(35)

By continuity, it holds true also for $s = T-t_{0}.$ Take $s = T- t_{0}$, thus we obtain from (35) that

$$\begin{aligned} z(T-t_{0}) \ge W(t(T-t_{0}), x(T-t_{0})). \end{aligned}$$

(36)

We set $y(t_{0} + s) := x(s),$ therefore we will obtain that $x(T-t_{0}) = y(T)$ and $W(t(T-t_{0}), x(T-t_{0})) = W(T, y(T)).$ From (36), we will deduce that for any $(t_{0}, x_{0}) \in ]0,T[ \times K$,

$$\begin{aligned} W(t_{0}, x_{0}) - \int _{0}^{T-t_{0}} \! l(t_{0}+\tau , y(t_{0}+\tau ), u(\tau )) \, \mathrm {d} \tau \ge \varphi (y(T)). \end{aligned}$$

We set $\hat{u}(t_{0}+s) := u(s)$. Therefore,

$$\begin{aligned} W(t_{0}, x_{0}) - \int _{t_{0}}^{T} \! l(s, y(s), \hat{u}(s)) \, \mathrm {d} s \ge \varphi (y(T)). \end{aligned}$$

Hence, for any $(t_{0}, x_{0}) \in ]0,T[ \times K$,

$$\begin{aligned} W(t_{0}, x_{0}) \ge \varphi (y(T)) + \int _{t_{0}}^{T} \! l(s, y(s), \hat{u}(s)) \, \mathrm {d} s \ge V(t_{0}, x_{0}). \end{aligned}$$

Using that W and V are continuous, we end the proof of Step 1.

Step 2 We will show next that for any $(t_{0}, x_{0}) \in [0, T] \times K$, it holds true $W(t_{0}, x_{0}) \le V(t_{0}, x_{0})$. Since W is a viscosity solution, by Definition 4.2, we have

$$\begin{aligned}&\forall (t,x) \in ]0, T[ \times int K, \ \forall (p_{t}, p_{x}) \in \partial _{+} W (t, x), \nonumber \\&-p_{t} + \sup \limits _{u \in B} (\langle -p_{x}, f(t,x,u) \rangle - l(t,x,u)) \le 0. \end{aligned}$$

(37)

Claim 1 For any $(t,x) \in ]0,T[ \times int K$ and $u \in B$

$$\begin{aligned} (1, f(t,x,u), -l(t,x,u)) \in cl(conv(T_{hyp(W)}(t,x,z))), \end{aligned}$$

for any $z \le W(t,x).$ In order to prove this claim, we proceed by a contradiction argument. Suppose there exists $u_{0} \in B$, such that for $z = W(t,x)$, we have that

$$\begin{aligned}(1, f(t,x,u_{0}), -l(t,x,u_{0})) \notin \bar{co} T_{hyp(W)}(t,x,W(t,x)).\end{aligned}$$

By the separation theorem, we deduce that there exists $0 \ne (p_{t}, p_{x}, q) \in \mathbb {R} \times \mathbb {R}^{n} \times \mathbb {R},$ such that

$$\begin{aligned}&\sup _{w \in cl(conv(T_{hyp(W)}(t,x,W(t,x))))} \langle (p_{t}, p_{x}, q), w \rangle \nonumber \\&<\langle (p_{t}, p_{x}, q), (1, f(t,x,u_{0}), -l(t,x,u_{0}) ) \rangle . \end{aligned}$$

(38)

Note that the left-hand side of (38) cannot be positive, because the maximum over the cone on the left-hand side is bounded. Therefore,

$$\begin{aligned} \sup _{w \in cl(conv(T_{hyp(W)}(t,x,W(t,x))))} \langle w, (p_{t}, p_{x}, q) \rangle =0. \end{aligned}$$

(39)

From (38), we also deduce that $q \ge 0$. Therefore,

$$\begin{aligned} p_{t} + \langle p_{x}, f(t,x, u_{0}) \rangle - q l(t,x, u_{0}) >0. \end{aligned}$$

(40)

By (39), we have $(p_{t}, p_{x}, q) \in N_{hyp(W)}(t,x, W(t,x)).$ By Lemma 4.2, [3] (substituting epigraph by hypograph), there exist $(t_{i}, x_{i}) \in ]0,T[ \times K$, such that $(t_{i}, x_{i}) \rightarrow (t,x)$, when $i \rightarrow \infty$ and

$$\begin{aligned} (p_{t}^{i}, p_{x}^{i}, q_{i}) \in N_{hyp(W)} (t_{i}, x_{i}, W(t_{i}, x_{i})), \end{aligned}$$

(41)

where $q_{i} > 0$ and such that $(p_{t}^{i}, p_{x}^{i}, q_{i}) \rightarrow (p_{t}, p_{x}, q), \ when \ i \rightarrow \infty .$ Therefore, as $q_{i} > 0$, we deduce from (41) that

$$\begin{aligned} \Big ( \dfrac{p_{t}^{i}}{q_{i}}, \dfrac{p_{x}^{i}}{q_{i}}, 1 \Big ) \in N_{hyp(W)}(t_{i}, x_{i}, W(t_{i}, x_{i})). \end{aligned}$$

Hence, by [3], page 267, we obtain that

$$\begin{aligned} \Big ( -\dfrac{p_{t}^{i}}{q_{i}}, -\dfrac{p_{x}^{i}}{q_{i}} \Big ) \in \partial _{+}W(t_{i}, x_{i}). \end{aligned}$$

(42)

From (37) and (42), we deduce that $\dfrac{p_{t}^{i}}{q_{i}} + \langle \dfrac{p_{x}^{i}}{q_{i}}, f(t_{i}, x_{i}, u_{0}) \rangle -l(t_{i}, x_{i}, u_{0}) \le 0,$ or equivalently, $p_{t}^{i} + \langle p_{x}^{i}, f(t_{i}, x_{i}, u_{0}) \rangle -q_{i} l(t_{i}, x_{i}, u_{0}) \le 0.$ Passing to the limit when $i \rightarrow \infty$, by continuity of f and l, we obtain that $p_{t} + \langle p_{x}, f(t,x, u_{0}) \rangle -q l(t,x,u_{0}) \le 0.$ This is a contradiction with (40). This ends the proof of Claim 1.

Claim 2 For any $(t,x) \in ]0,T[ \times intK$ and $z \le W(t,x)$, any $u \in B$

$$\begin{aligned} (1, f(t,x,u), -l(t,x,u)) \in T_{hyp(W)}(t,x,z). \end{aligned}$$

Proof of Claim 2 follows from Lemma 2.1 and Claim 1. Consider the control system

$$\begin{aligned} (CS2) {\left\{ \begin{array}{ll} \dot{t}(s) = 1, \\ \dot{x}(s) = f(t_{0}+s, x(s) ,u(s)), \ u(s) \in B, \\ \dot{z}(s) = -l(t_{0}+s, x(s), u(s)). \end{array}\right. } \end{aligned}$$

(43)

From the proof of Theorem 3.3, [3], (substituting epigraph by hypograph) we deduce that the set $\Psi = hyp(W) \cap ]0,T[ \times intK \times \mathbb {R},$ is locally invariant by the system (CS2), i.e. for any solution $(t(\cdot ), x(\cdot ), z(\cdot ))$ of (CS2) with $t(0) = t_{0} \in ]0,T[$ and $x(0) = x_{0} \in int K$, $z(0) = W(t_{0}, x_{0})$, satisfying $x(s) \in int K$, $s \in [0, \delta ]$, for some $\delta >0$, we have that $(t(s), x(s), z(s)) \in hyp(W).$ Therefore, we deduce that $z(s) \le W(t(s), x(s)).$ Hence,

$$\begin{aligned} W(t_{0}, x_{0}) - \int _{t_{0}}^{t_{0} + \delta } \! l(t_{0} + s, x(s), u(s)) \, \mathrm {d} s \le W(t_{0} + \delta , x(t_{0} + \delta )). \end{aligned}$$

Thus, if a solution $(x,u)(\cdot )$ of (CS2) satisfies $x(s) \in int K$ on $[t_{1}, t_{2}]$, then

$$\begin{aligned} W(t_{1}, x(t_{1})) \le W(t_{2}, x(t_{2})) + \int _{t_{1}}^{t_{2}} \! l(s, x(s), u(s)) \, \mathrm {d} s. \end{aligned}$$

Let $(\bar{x}(\cdot ), \bar{u}(\cdot ))$ be optimal for (P) at $(t_{0}, x_{0}) \in ]0,T[ \times int K$. By Theorem 2.1 applied to (CS2) and $\mathcal {K} = hyp (W)$, there exist controls $u_{\varepsilon }$ such that $x_{\varepsilon }(\cdot )$ corresponding to $u_{\varepsilon }$, converges uniformly to $\bar{x}(\cdot )$ when $\varepsilon \rightarrow 0$, and $z_{\varepsilon }(\cdot )$, defined on $[t_{0},T]$ by $z_{\varepsilon }(t) := W(t_{0}, x_{0}) - \int _{t_{0}}^{t} \! l(s, x_{\varepsilon }(s), u_{\varepsilon }(s)) \, \mathrm {d} s,$ converges uniformly to $z(\cdot )$ given by $z(t) := W(t_{0}, x_{0}) - \int _{t_{0}}^{t} \! l(s, \bar{x}(s), \bar{u}(s)) \, \mathrm {d} s,$ and for all $t \in ]t_{0}, T]$ we have $(t, x_{\varepsilon }(t), z_{\varepsilon }(t)) \in int (hyp (W)).$ Hence, $x_{\varepsilon }(t) \in int K$ on $]t_{0}, T]$. Therefore, we deduce that for any $t \in ]t_{0}, T]$, it holds true $z_{\varepsilon }(t) \le W(t, x_{\varepsilon }(t)).$ Hence, for all small $\tau >0$,

$$\begin{aligned} W(t_{0} + \tau , x_{\varepsilon }(t_{0} + \tau )) - \int _{t_{0} + \tau }^{T} \! l(s, x_{\varepsilon }(s), u_{\varepsilon }(s)) \, \mathrm {d} s \le W(T, x_{\varepsilon }(T)) = \varphi (x_{\varepsilon }(T)). \end{aligned}$$

Taking the limit when $\tau \rightarrow 0+$, we get

$$\begin{aligned} W(t_{0}, x_{\varepsilon }(t_{0})) - \int _{t_{0}}^{T} \! l(s, x_{\varepsilon }(s), u_{\varepsilon }(s)) \, \mathrm {d} s \le W(T, x_{\varepsilon }(T)) = \varphi (x_{\varepsilon }(T)). \end{aligned}$$

Passing to the limit when $\varepsilon \rightarrow 0+$, we deduce that

$$\begin{aligned} W(t_{0}, x_{0}) \le \varphi (\bar{x}(T)) + \int _{t_{0}}^{T} \! l(s, \bar{x}(s), \bar{u}(s)) \, \mathrm {d} s. \end{aligned}$$

We obtain for any $(t_{0}, x_{0}) \in ]0,T[ \times int K$ that $W(t_{0},x_{0}) \le V(t_{0}, x_{0}).$ Since W and V are continuous, we end the proof of step 2. From Step 1 and Step 2, we deduce that the value function of the Bolza problem is the unique viscosity solution of the Hamilton–Jacobi equation on $[0, T] \times K$ (in the class of continuous functions). This ends the proof of Theorem 5.1. $\square$

Theorem 5.2

For every $i \ge 1$ let $K_{i}$, K be closed and nonempty subsets of $\mathbb {R}^{n}$ defined by (9), (10) and (A3) holds true. Consider continuous $H_{i} : [0, T] \times \mathbb {R}^{n} \times \mathbb {R}^{n} \rightarrow \mathbb {R}$ satisfying the assumptions (H1)–(H6) with the same integrable functions $c_{R}(\cdot )$, absolutely continuous functions $a_{R}(\cdot )$ and $c >0$, $M_{R} >0$. Assume that for some $H : [0, T] \times \mathbb {R}^{n} \times \mathbb {R}^{n} \rightarrow \mathbb {R}$, $H_{i} \rightarrow H$ uniformly on compacts, when $i \rightarrow \infty$ and that assumption (A4)$_{H}$ holds true. Consider viscosity solutions $W_{i}$ to Hamilton–Jacobi equation (19) with H replaced by $H_{i}$ and K replaced by $K_{i}$. Let $x_{0} \in int K$, $r >0$ such that $B(x_{0}, r) \subset K$. Then, the restrictions of $W_{i}$ to $[0, T] \times B(x_{0}, \dfrac{r}{2})$ converge uniformly to the restriction to $[0, T] \times B(x_{0}, \dfrac{r}{2})$ of the unique solution W of (19).

Proof

Clearly, H satisfies (H1)–(H6) with the same $c_{R}(\cdot )$, $a_{R}(\cdot )$, c, $M_{R}$. In [13], it is shown that, if (H1)–(H6) hold true for H and $H_{i}$, then there exists $f, f_{i}, l, l_{i}$ satisfying (A1)–(A2) such that $H(t,x,p) = \max _{u \in B} (\langle p, f(t,x,u) \rangle - l(t,x,u))$ and $H_{i}(t,x,p) = \max _{u \in B} (\langle p, f_{i}(t,x,u) \rangle - l_{i}(t,x,u)).$ Moreover, we will also have that $G_{i}(t,x) = \lbrace (f_{i}(t,x,u), l_{i}(t, x, u) + r): u \in B, r \ge 0 \rbrace$ is convex and closed. Let $x_{0} \in int K$ and $r >0$ be such that $B(x_{0}, r) \subset K$. By Theorems 4.1 and 5.1, the value function of the Bolza problem (with $f_{i}$, $l_{i}$) is the unique viscosity solution of the Hamilton–Jacobi equation on $[0, T] \times K_{i}$. As $H_{i} \rightarrow H$ uniformly on compacts, when $i \rightarrow \infty$, thus by Theorem 4.1, [13], we have $f_{i}$ converge to f and $l_{i}$ converge to l uniformly on compacts, when $i \rightarrow \infty$. Proposition 3.3 and Theorem 3.1 end the proof. $\square$

Corollary 5.1

Let the assumptions of Theorem 5.2 hold true. Then,

$$\begin{aligned}Lim_{i \rightarrow \infty } epi W_{i} = epi W,\end{aligned}$$

where W is the unique viscosity solution of (19).

Proof

The proof follows by Corollary 3.1 and from the fact that $W_{i}=V_{i}$ is a bounded family of equicontinuous functions. $\square$

6 Conclusions

In this paper, we have considered Hamilton–Jacobi–Bellman equations under state constraints and our main goal was to study the stability of solutions of HJB equations. For this reason, we have associated with HJB equations a suitable family of Bolza optimal control problems under state constraints and established the stability results of value functions, see Theorem 3.1. We have imposed classical hypotheses on Hamiltonian, under which the HJB equation is characterising the value function of Bolza optimal control problem, and using viability analysis, we have proven that the value function is stable under perturbations. The key technical point is the inward-pointing condition (IPC) and the use of so-called neighbouring feasible trajectories theorem (NFT), see Theorem 2.1. We have also shown that under the classical assumptions on the Hamiltonian, the value function of the Bolza optimal control problem is a viscosity solution of the Hamilton–Jacobi–Bellman equation, see Theorem 4.1. The existence of the unique viscosity solution of HJB equation is proved under the suitable inward-pointing condition on the Hamiltonian in Theorem 5.1. The stability of solutions of HJB equations is proved in Theorem 5.2 using the obtained results of stability of value functions.