Universal Nash Equilibrium Strategies for Differential Games

Abstract

The paper is concerned with a two-player nonzero-sum differential game in the case when players are informed about the current position. We consider the game in control with guide strategies first proposed by Krasovskii and Subbotin. The construction of universal strategies is given both for the case of continuous and discontinuous value functions. The existence of a discontinuous value function is established. The continuous value function does not exist in the general case. In addition, we show the example of smooth value function not being a solution of the system of the Hamilton–Jacobi equation.

1 Introduction

The purpose of this paper is to study the Nash equilibria for a two-player deterministic differential game in the case when the players are informed about the present position. We construct the pair of strategies providing the Nash equilibrium at any initial position from the given compact set. It is natural to say that such pair of strategies is a universal Nash equilibrium for a given compact set. Note that the notion of universality generalizes the concept of strong time consistency (subgame perfectness).

There are two approaches in the literature dealing with this problem (see [8] and the references therein). The first approach is close to the so-called Folk Theorem for repeated games and is based on the punishment strategy technique. This technique makes it possible to establish the existence of Nash equilibrium at the given initial position in the framework of feedback strategies [14, 15] and in the framework of Friedman strategies [21]. The set of Nash equilibria at the given initial position is characterized in [12, 14]. The infinitesimal version of this characterization is derived in [2, 4]. In addition, each Nash equilibrium payoff at the given position corresponds to the pair of continuous functions; these functions are stable with respect to auxiliary zero-sum differential games, and their values at the initial position are kept along some trajectory [3]. Note that in this case, the Nash equilibrium strategies are not universal.

The key idea of the second approach is to find a Nash equilibrium payoff as a solution of the system of the Hamilton–Jacobi equations [5, 11, 13]. In this case, the universal Nash equilibrium can be constructed. However, the existence theorem for the system of the Hamilton–Jacobi equations is established only for some cases of the games in one dimension [6, 7, 9, 10].

In this paper, we consider the Nash equilibrium for deterministic differential games in control with guide strategies. These strategies was first proposed by Krasovskii and Subbotin for zero-sum differential games [17]. In the framework of this formalization, the player forms his control stepwise. It is assumed that the player measures the state of the system only in the times of control correction. At the time of control correction, the player estimates the state of the system using the information about the state of the system at the previous time instants of control correction. Having this estimate and the information about the real state of the system, he assigns the control which is used up to the next control correction. Roughly speaking, the player using control with guide strategies needs instruments to measure the current position and a computer to store the information about the state of the system at the previous time of control correction, and to evaluate the state of the system at the current time of control correction, whereas the player using feedback strategies needs only measuring instruments.

The choice of control with guide strategies is motivated by the following arguments. Even for zero-sum differential game, a universal feedback solution does not exist (feedback strategies solving the game at any position from the given compact) [19]. The universal solution of zero-sum differential games can be found in the class of feedback strategies depending on the precision parameter [16] or in the class of control with guide strategies [17]. However, for nonzero-sum differential games, an existing design of Nash equilibria in the class feedback strategies depending on the precision parameter does not provide the universality.

The paper is organized as follows. In Section 2, we set up the problem and introduce the control with guide strategies. In Section 3, we construct the Nash equilibrium in the control with guide strategies for the case of a continuous value function. This function is to satisfy some viability conditions. Further in Section 3, the properties of a continuous value function are considered. We give the infinitesimal form of viability conditions. After, we compare the value functions satisfying viability conditions and the solutions of the system of the Hamilton–Jacobi equations. The example showing that the continuous value function does not exist in the general case completes Section 3. In Section 4, we generalize the construction of Section 3 for the case of an upper semicontinuous value multifunction. In Section 5, we prove the existence of a value multifunction.

2 Problem Statement

Let us consider a two-player differential game with the dynamics

$$ \dot{x}=f(t,x,u)+g(t,x,v), \ \ t\in [0,T], \ \ x\in\mathbb{R}^{n}, \ \ u\in P, \ \ v\in Q. $$

(1)

Here, u and v are controls of player I and player II, respectively. Payoffs are terminal. Player I wants to maximize σ ₁(x(T)), whereas player II wants to maximize σ ₂(x(T)). We assume that sets P and Q are compacts, and functions f, g, σ ₁, and σ ₂ are continuous. In addition, suppose that functions f and g are Lipschitz continuous with respect to the phase variable and satisfy the sublinear growth condition with respect to x.

Denote

$$\mathcal{U} := \{u:[0,T]\rightarrow P\ \ {\rm{measurable}}\}, $$

$$\mathcal{V} := \{v:[0,T]\rightarrow Q\ \ {\rm{measurable}}\}. $$

If $u\in \mathcal {U}$, $v\in \mathcal {V}$, then denote by x(⋅, t ₀, x ₀, u, v) the solution of the initial value problem

$$\dot{x}(t)=f(t,x(t),u(t))+g(t,x(t),v(t)), \ \ x(t_{0})=x_{0}. $$

We assume that the players use control with guide strategies (CGS). In this case, the control depends not only on a current position but also on a vector w. The vector w is called a guide. The dimension of the guide can differ from n.

The control with guide strategy of player I U is a triple of functions (u, ψ ¹, χ ¹) such that for some natural m, the function u maps [0, T] × ℝⁿ × ℝ^m to P, the function ψ ¹ maps [0, T] × [0, T] × ℝⁿ × ℝ^m to ℝ^m, and χ ¹ is a function of [0, T] × ℝⁿ with values in ℝ^m.

The meaning of the functions u, ψ ¹, and χ ¹ is the following. Let w ¹ be a m-dimensional vector. Further, it denotes the state of the first player’s guide. Player I computes the value of the variable w ¹ using the rules which are given by the strategy U. The function u(t _∗, x _∗, w ¹) is a function forming the control of player I. It depends on the current position (t _∗, x _∗) and the current state of guide w ¹. The function ψ ¹(t ₊, t _∗, x _∗, w ¹) determines the value of the guide at time t ₊ under the condition that at time t _∗, the phase vector is equal to x _∗, and the state of guide is equal to w ¹. The function χ ¹(t ₀, x ₀) determines the initial state of guide.

Player I forms his control stepwise. Let (t ₀, x ₀) be an initial position, and let $\Delta =\{t_{k}\}_{k=0}^{r}$ be a partition of the interval [t ₀, T]. Suppose that player II chooses his control v[⋅] arbitrarily. He can also use his own CGS and form the control v[⋅] stepwise. Denote the solution x[⋅] of Eq. (1) with the initial condition x[t ₀] = x ₀ such that the control of player I is equal to $u\left (t_{k},x_{k},{w_{k}^{1}}\right )$ on [t _k , t _{k + 1}[ by x ¹[⋅, t ₀, x ₀, U, Δ, v[⋅]]. Here, the state of the system at time t _k is x _k , the state of the first player’s guide is ${w_{k}^{1}}$; it is computed by the rule ${w_{k}^{1}}=\psi ^{1}\left (t_{k},t_{k-1},x_{k-1},w_{k-1}^{1}\right )$ for $k=\overline {1,r}$, ${w_{0}^{1}}=\chi ^{1}(t_{0},x_{0}).$

Note that the player needs only the finite number of sampling points (t _k , x _k ) to produce the piecewise constant control on whole interval [t ₀, T]. Certainly, he should use a computer to obtain the values ${w_{k}^{i}}$.

The control with guide strategy of player II is defined analogously. It is a triple V = (v, ψ ², χ ²). Here, v = v(t _∗, x _∗, w ²), ψ ² = ψ ²(t ₊, t _∗, x _∗, w ²), χ ² = χ ²(t ₀, x ₀)); (t _∗, x _∗) is a current position, where w ² denotes the guide of player II, and (t ₀, x ₀) is an initial position. The motion generated by a strategy V, a partition Δ of the interval [t ₀, T], and a measurable control of player II u[⋅] is also constructed stepwise. Denote it by x ²[⋅, t _∗, x _∗, V, Δ, u[⋅]].

We assume that the Nash equilibrium is achieved when the players get the same partition. Let $\Delta =\{t_{k}\}_{k=0}^{m}$ be a partition of the interval [t ₀, T]. Denote the solution x[⋅] of Eq. (1) with the initial condition x[t ₀] = x ₀ such that the control of player I is equal to $u\left (t_{k},x_{k},{w_{k}^{1}}\right )$ on [t _k , t _{k + 1}[, and the control of player II is equal to $v\left (t_{k},x_{k},{w_{k}^{2}}\right )$ on [t _k , t _{k + 1}[ by x ^(c)[⋅, t _∗, x _∗, U, V, Δ]. Here, x _k denoting the state of the system at time t _k ; ${w_{k}^{i}}$ is the state of the i-th player’s guide at time t _k . Recall that $w_{k+1}^{i}=\psi ^{i}\left (t_{k+1},t_{k},x_{k},{w_{k}^{1}}\right )$, ${w_{0}^{i}}=\chi ^{i}(t_{0},x_{0})$, i = 1, 2.

Definition 2.1

Let G ⊂ [0, T] × ℝⁿ. A pair of control with guide strategies (U ^∗, V ^∗) is said to be a control with guide Nash equilibrium on G iff for all (t ₀, x ₀) ∈ G the following inequalities hold:

$$\begin{array}{@{}rcl@{}} &&\lim\limits_{\delta\downarrow 0}\sup\left\{\sigma_{1}\left(x^{2}[T,t_{0},x_{0},V^{*},\Delta,u[\cdot]]\right): d(\Delta)\leq\delta,u[\cdot]\in\mathcal{U}\right\}\\ &&{\kern2pc}\leq \lim\limits_{\delta\downarrow 0}\inf\left\{\sigma_{1}\left(x^{(c)}[T,t_{0},x_{0},U^{*},V^{*},\Delta]\right): d(\Delta)\leq\delta\right\},\\ &&\lim\limits_{\delta\downarrow 0}\sup\left\{\sigma_{2}\left(x^{1}[T,t_{0},x_{0},U^{*},\Delta,v[\cdot]]\right): d(\Delta)\leq\delta,v[\cdot]\in\mathcal{V}\right\}\\ &&{\kern2pc}\leq \lim\limits_{\delta\downarrow 0}\inf\left\{\sigma_{2}\left(x^{(c)}[T,t_{0},x_{0},U^{*},V^{*},\Delta]\right): d(\Delta)\leq\delta\right\}. \end{array} $$

Note that if G is a reachable set from (t ^∗, x ^∗), then the control with guide Nash equilibrium on G is a subgame perfect Nash equilibrium.

Definition 2.2

A function (c ₁, c ₂) : [0, T] × ℝⁿ → ℝ² is called a value function if for any compact set G ⊂ [0, T] × ℝⁿ, there exists a control with guide Nash equilibrium on G (U ^∗, V ^∗) such that for all (t ₀, x ₀) ∈ G

$$c_{i}(t_{0},x_{0})=\lim\limits_{\delta\downarrow 0}\inf\{\sigma_{2}(x^{(c)}[T,t_{0},x_{0},U^{*},V^{*},\Delta]): d(\Delta)\leq\delta\}.$$

Note that for the zero-sum game, the value function is defined in each position independently, and also it can be defined as in Definition 2.2 [20].

3 Continuous Value Function

3.1 Construction of the Nash Equilibrium Strategies

Let (t _∗, x _∗) ∈ [0, T] × ℝⁿ, u _∗ ∈ P, v _∗ ∈ Q.

Define

$$\text{Sol}^{1}(t_{*},x_{*};v_{*}) := \text{cl}\{x(\cdot,t_{*},x_{*},u,v_{*}):u\in \mathcal{U}\}, $$

$$\text{Sol}^{2}(t_{*},x_{*};u_{*}) := \text{cl}\{x(\cdot,t_{*},x_{*},u_{*},v):v\in \mathcal{V}\}, $$

$$\text{Sol}(t_{*},x_{*}) := \text{cl}\{x(\cdot,t_{*},x_{*},u,v):u\in \mathcal{U},v\in\mathcal{V}\}. $$

Here, cl denotes the closure in the space of continuous vector function on [0, T]. Note that the sets Sol¹(t _∗, x _∗;v _∗), Sol²(t _∗, x _∗;u _∗), and Sol(t _∗, x _∗) are compact.

Theorem 3.1

Let a continuous function (c ₁, c ₂) : [0, T] × ℝⁿ → ℝ² satisfy the following conditions:

• (F1) c _i (T, x) = σ _i (x), i = 1, 2;

• (F2) For every (t _*, x _*) ∈ [0, T] × ℝⁿ, u ∈ P, there exists a motion y ²(⋅) ∈ Sol ²(t _∗, x _∗; u) such that c ₁(t, y ²(t)) ≤ c ₁(t _∗, x _∗) for t ∈ [t _∗, T];

• (F3) For every (t _*, x _*) ∈ [0, T] × ℝⁿ, v ∈ Q, there exists a motion y ¹(⋅) ∈ Sol ¹(t _∗, x _∗;v) such that c ₂(t, y ¹(t)) ≤ c ₂(t _∗, x _∗) for t ∈ [t _∗, T];

• (F4) For every (t _*, x _*) ∈ [0, T] × ℝⁿ, there exists a motion y ^(c)(⋅) ∈ Sol(t _∗, x _∗) such that c _i (t, y ^(c)(t)) = c _i (t _∗, x _∗) for t ∈ [t _∗, T], i = 1, 2.

Then, (c ₁, c ₂) is a value function.

The proof of Theorem 3.1 is constructive, and it is based on the Krasovskii–Subbotin extremal shift rule.

Let G ⊂ [0, T] × ℝⁿ be a compact. Denote by E the reachable set from G:

$$ E := \{x(t,t_{*},x_{*},u,v):(t_{*},x_{*})\in G, t\in [t_{*},T], u\in \mathcal{U},v\in\mathcal{V}\}. $$

(2)

Put

$$ K := \max\{\|f(t,x,u)+g(t,x,v)\|:t\in [0,T],x\in E, u\in P, v\in Q\}, $$

(3)

Let L be a Lipschitz constant of the function f + g on [0, T] × E × P × Q, i.e., for all t ∈ [0, T], x′, x″ ∈ E, u ∈ P, v ∈ Q

$$\|f(t,x',u)+g(t,x',v)-f(t,x^{\prime\prime},u)-g(t,x^{\prime\prime},v)\|\leq L\|x'-x^{\prime\prime}\|. $$

Also, put

$$\begin{array}{@{}rcl@{}} \varphi^{*}(\delta) := \sup\{\|f&(t',x,u)+g(t',x,u)-f(t^{\prime\prime},x,u)-g(t^{\prime\prime},x,u)\|:\\ &t',t^{\prime\prime}\in [0,T], |t'-t^{\prime\prime}|\leq\delta, x\in E, u\in P,v\in Q\}. \end{array} $$

Note that φ ^∗(δ) → 0, as δ → 0.

Let us introduce the auxiliary controlled system

$$ \dot{s}=h(t,s,\omega_{1},\omega_{2}), \ \ s\in\mathbb{R}^{n}, \ \ \omega_{i}\in \Omega_{i}. $$

(4)

Below, we consider two cases.

• (i) Ω₁ = P, Ω₂ = Q, h = f + g;

• (ii) Ω₁ = P × Q, $\Omega _{2}=\varnothing$, h = f + g.

Note that in both cases, system (4) satisfies the Isaacs condition.

Put β := 2L, R := max{∥s′−s″∥:s′, s″ ∈ E}, φ(δ) = 4 φ ^∗(δ)R+4K ² δ.

The following lemma was proved by Krasovskii and Subbotins (see [17]).

Lemma 3.1

Let ${s_{1}^{0}},{s_{2}^{0}}\in \mathbb {R}^{n}$, t _∗ ∈ [0, T], $\omega _{1}^{*}\in \Omega _{1},$ $\omega _{2}^{*}\in \Omega _{2}$ satisfy the following conditions:

$$\max\limits_{\omega_{1}\in \Omega_{1}}\min\limits_{\omega_{2}\in \Omega_{2}}\left\langle {s_{2}^{0}}-{s_{1}^{0}},h\left(t_{*},{s_{1}^{0}},\omega_{1},\omega_{2}\right) \right\rangle=\min\limits_{\omega_{2}\in \Omega_{2}}\left\langle {s_{2}^{0}}-{s_{1}^{0}},h\left(t_{*},{s_{1}^{0}},\omega_{1}^{*},\omega_{2}\right) \right\rangle, $$

$$\min\limits_{\omega_{2}\in \Omega_{2}}\max\limits_{\omega_{1}\in \Omega_{1}}\left\langle {s_{2}^{0}}-{s_{1}^{0}},h\left(t_{*},{s_{1}^{0}},\omega_{1},\omega_{2}\right) \right\rangle=\max\limits_{\omega_{1}\in \Omega_{1}}\left\langle {s_{2}^{0}}-{s_{1}^{0}},h\left(t_{*},{s_{1}^{0}},\omega_{1},\omega_{2}^{*}\right) \right\rangle. $$

If s ₁(⋅) is a solution of the initial value problem

$$\dot{s}_{1}=h(t,s_{1},\omega_{1}^{*},\omega_{2}(t)), \ \ s_{1}(t_{*})={s_{1}^{0}}, $$

and s ₂(⋅) is a solution of the initial value problem

$$\dot{s}_{2}=h(t,s_{2},\omega_{1}(t),\omega_{2}^{*}), \ \ s_{2}(t_{*})={s_{2}^{0}}, $$

for some measurable controls ω ₁(⋅) and ω ₂(⋅), then for all t ₊ ∈ [t _∗, T] the following estimate holds:

$$\|s_{2}(t_{+})-s_{1}(t_{+})\|^{2}\leq \|{s_{2}^{0}}-{s_{1}^{0}}\|^{2}(1+\beta(t_{+}-t_{*}))+\varphi(t_{+}-t_{*})\cdot(t_{+}-t_{*}).$$

We assume that the i-th player’s guide w ⁱ is a quadruple (d ⁱ, τ ⁱ, w ^{i, (a)}, w ^{i, (c)}). The variable d ⁱ ∈ ℝ describes an accumulated error, τ ⁱ ∈ [0, T] is a previous time of the control correction, w ^{i, (a)} ∈ ℝⁿ is a punishment part of the guide, and w ^{i, (c)} ∈ ℝⁿ is a consistent part of the guide. The whole dimension of the guide is 2n + 2.

For any (t _∗, x _∗) ∈ [0, T] × ℝⁿ, u ∈ P, v ∈ Q, choose and fix a motion y ²(⋅;t _∗, x _∗, u) satisfying condition (F2), a motion y ¹(⋅;t _∗, x _∗, v) satisfying condition (F3), and a motion y ^(c)(⋅;t _∗, x _∗) satisfying condition (F4).

Now, let us define the strategies U ^∗ and V ^∗. Below, we prove that the pair of strategies (U ^∗, V ^∗) is a control with guide Nash equilibrium on G.

First, put χ ¹(t ₀, x ₀) = χ ²(t ₀, x ₀) := (0, t ₀, x ₀, x ₀).

Let (t, x) be a position, and w ⁱ = (d ⁱ, τ ⁱ, w ^{i, (a)}, w ^{i, (c)}) be a state of the i-th player’s guide. Put

$$ z^{i} := \left\{ \begin{array}{cl} w^{i,(c)}, & \|w^{i,(c)}-x\|^{2}\leq d^{i}\left(1+\beta\left(t-\tau^{i}\right)\right)+\varphi\left(t-\tau^{i}\right)\left(t-\tau^{i}\right), \\ w^{i,(a)}, & \text{ otherwise}. \end{array}\right. $$

(5)

Let us consider two cases.

• i = 1. Choose a control u _∗ by the rule

  $$ \max\limits_{u\in P}\langle z^{1}-x,f(t,x,u)\rangle=\langle z^{1}-x,f(t,x,u_{*})\rangle. $$

  (6)

  Further, let v ^∗ satisfy the following condition:

  $$ \min\limits_{v\in Q}\langle z^{1}-x,g(t,x,v)\rangle=\langle z^{1}-x,g(t,x,v^{*})\rangle. $$

  (7)

  Define u(t, x, w ¹) := u _∗. For t ₊ > t, put ψ ¹(t ₊, t, x, w ¹) be equal to $w^{1}_{+}=\left (d_{+}^{1},\tau _{+}^{1},w^{1,(a)}_{+},w^{1,(c)}_{+}\right )$, where

  $$d_{+}^{1} := \|z^{1}-x\|^{2},\ \ \tau_{+}^{1} := t,\ \ w^{1,(a)}_{+} := y^{1}\left(t_{+};t,z^{1},v^{*}\right),\ \ w^{1,(c)}_{+} := y^{(c)}\left(t_{+};t,z^{1}\right).$$

• i = 2. Let a control v _∗ be such that

  $$ \max\limits_{v\in Q}\langle z^{2}-x,g(t,x,v)\rangle=\langle z^{2}-x,g(t,x,v_{*})\rangle. $$

  (8)

  Choose u ^∗ satisfying the condition

  $$ \min\limits_{u\in P}\langle z^{2}-x,f(t,x,u)\rangle=\langle z^{2}-x,f(t,x,u^{*})\rangle. $$

  (9)

  Set v(t, x, w) := v _∗. For t ₊ > t, put ψ ²(t ₊, t, x, w ²) be equal to $w_{+}^{2}=\left (d_{+}^{2},\tau _{+}^{2},w^{2,(a)}_{+},w^{2,(c)}_{+}\right )$, where

  $$d_{+}^{2} := \|z^{2}-x\|^{2},\ \ \tau_{+}^{2} := t,\ \ w^{2,(a)}_{+} := y^{2}(t_{+};t,z^{2},u^{*}),\ \ w^{2,(c)}_{+} := y^{(c)}(t_{+};t,z^{2}).$$

Note that

$$ c_{j}\left(t_{+},w^{i,(c)}_{+}\right)=c_{j}(t,z^{i}) \ \ \text{for all }~i,j=1,2, $$

(10)

$$ c_{1}\left(t_{+},w^{2,(a)}_{+}\right)\leq c_{1}\left(t,z^{2}\right), \ \ c_{2}\left(t_{+},w^{1,(a)}_{+}\right)\leq c_{2}\left(t,z^{1}\right). $$

(11)

Below, let x ₊ denote the state of the system at time t ₊.

Lemma 3.2

Suppose that z ¹ = z ² = z. If players I and II use respectively the controls u _∗ and v _∗ on the interval [t, t ₊], then $w^{1,(c)}_{+}=w^{2,(c)}_{+}$ and

$$\|x_{+}-w^{i,(c)}_{+}\|^{2}\leq d_{+}^{i}(1+\beta(t_{+}-\tau_{+}))+\varphi(t_{+}-\tau_{+})(t_{+}-\tau_{+}). $$

Proof

The controls u _∗ and v _∗ satisfy the condition

$$\max\limits_{u\in P,v\in Q}\langle z-x,f(t,x,u)+g(t,x,v)\rangle=\langle z-x,f(t,x,u_{*})+g(t,x,v_{*})\rangle. $$

We apply Lemma 3.1 with Ω₁ = P × Q, Ω₂ = ∅, h = f + g. If x(⋅) = x(⋅, t, x, u _∗, v _∗), y ^(c)(⋅) = y ^(c)(⋅;t, z), then

$$\|x(t_{+})-y^{(c)}(t_{+})\|^{2}\leq \|x-z\|^{2}(1+\beta(t_{+}-t))+\varphi(t_{+}-t)\cdot (t_{+}-t). $$

The definition of the strategies U ^∗ and V ^∗ yields that $w_{+}^{i,(c)}=y^{(c)}\left (t_{+}\right )$ for i = 1, 2. By construction of the functions ψ _i , i = 1, 2 we have that $t=\tau ^{i}_{+}$, and $d_{+}^{i}=\|x-z\|^{2}$. This completes the proof of the Lemma. □

Lemma 3.3

If player I uses the control u _∗ on the interval [t, t ₊], then

$$\|x_{+}-w^{1,(a)}_{+}\|^{2}\leq d_{+}^{i}(1+\beta(t_{+}-\tau_{+}))+\varphi(t_{+}-\tau_{+})(t_{+}-\tau_{+}),\ \ i=1,2. $$

Proof

We apply Lemma 3.1 with Ω₁ = P, Ω₂ = Q and h = f + g. The choice of u _∗ (see Eq. (6)) and v ^∗ (see Eq. (7)) yields that the inequality

$$\|x(t_{+})-y^{1}(t_{+})\|^{2}\leq \|x-z^{1}\|^{2}(1+\beta(t_{+}-t))+\varphi(t_{+}-t)\cdot (t_{+}-t) $$

holds with x(⋅) = x(⋅, t, x, u _∗, v) and y ¹(⋅) = y ¹(⋅, t, z ¹, v ^∗). Since $w^{1,(a)}_{+}=y^{1}(t_{+})$, $\tau ^{1}_{+}=t$, and $d_{+}^{1}=\|x-z^{1}\|^{2}$, the conclusion of the Lemma follows. □

We need the following estimate. Let $\Delta =\{t_{k}\}_{k=0}^{r}$ be a partition of the interval [t ₀, T], and let $\{\gamma _{k}\}_{k=0}^{r}$ be a collection of numbers such that

$$ \gamma_{k+1}\leq \gamma_{k}(1+\beta(t_{k+1}-t_{k}))+\varphi(t_{k+1}-t_{k})\cdot (t_{k+1}-t_{k}). $$

(12)

Then,

$$ \gamma_{k}\leq [\gamma_{0}+(1+(t_{k}-t_{0}))\varphi(d(\Delta))]\exp\beta(t_{k}-t_{0}). $$

(13)

Proof of Theorem 3.1

First, let us show that for all (t ₀, x ₀) ∈ G, the following equality is valid:

$$ c_{j}(t_{0},x_{0})=\lim\limits_{\delta\downarrow 0}\inf\left\{\sigma_{j}(x^{(c)}[T,t_{0},x_{0},U^{*},V^{*},\Delta]), d(\Delta)\leq\delta\right\}, \ \ j=1,2. $$

(14)

Let $\Delta =\{t_{k}\}_{k=1}^{r}$ be a partition of the interval [t ₀, T]. Denote the state of the system at time t _k by x _k , the state of the i-th player’s guide by ${w^{i}_{k}}=\left ({d_{k}^{i}},\tau _{k},w^{(a),i}_{k},w^{i,(c)}_{k}\right )$. Also, let ${z^{i}_{k}}$ be chosen by rule (5) at time t _k . We have that τ ₀ = t ₀, τ _{k + 1} = t _k for k ≥ 0. Moreover,

$$z^{1}_{0}=w^{1,(c)}_{0}=w^{1,(c)}_{0}={z^{2}_{0}}.$$

Hence using lemma 3.2 inductively, we get that

$$ {z^{1}_{k}}=w^{1,(c)}_{k}={z^{2}_{k}}=w^{2,(c)}_{k},\ \ d^{i}_{k+1}=\|x_{k}-{z_{k}^{i}}\|^{2}. $$

(15)

and

$$\|x_{k+1}-z^{i}_{k+1}\|^{2}\leq \|x_{k}-{z^{i}_{k}}\|^{2}(1+\beta(t_{k+1}-t_{k}))+\varphi(t_{k+1}-t_{k})(t_{k+1}-t_{k}) $$

for all $k=\overline {0,N}$.

It follows from Eq. (13) that

$$ \|x_{r}-{z_{r}^{i}}\|^{2}\leq [\|x_{0}-{z_{0}^{i}}\|^{2}+(1+(t_{r}-t_{0}))\varphi(d(\Delta))]\exp\beta(t_{r}-t_{0}). $$

Since ${z_{0}^{i}}=x_{0}$, we obtain that

$$ \|x_{r}-{z_{r}^{i}}\|\leq \varkappa(\delta) := \Bigl[(1+(t_{r}-t_{0}))\varphi(\delta)\exp\beta(t_{r}-t_{0})\Bigr]^{1/2}, $$

(16)

where δ = d(Δ). Note that ϰ(δ) → 0, δ → 0.

Let ϕ _j (Y) be a modulus of continuity of the function σ _j on the set E

$$\phi_{j}(\gamma) := \sup\{|\sigma_{j}(x')-\sigma_{j}(x^{\prime\prime})|:x',x^{\prime\prime}\in E, \|x'-x^{\prime\prime}\|\leq\gamma\}.$$

We have that

$$ \|\sigma_{j}(x_{r})-\sigma_{j}({z_{r}^{i}})\|\leq \phi_{j}(\varkappa(\delta)). $$

(17)

Since ${z^{i}_{k}}=w^{i,(c)}_{k}$, it follows from Eq. (10) that $c_{j}\left (t_{k+1},w_{k+1}^{i,(c)})=c_{j}(t_{k},{z^{i}_{k}}\right )=c_{j}\left (t_{k},w_{k}^{i,(c)}\right )$. Therefore, using condition (F1), we get

$$\|\sigma_{j}(x[T,t_{0},x_{0},U^{*},V^{*},\Delta])-c_{j}(t_{0},x_{0})\|\leq \phi_{j}(\varkappa(\delta)) $$

with δ = d(Δ). Passing to the limit, we obtain equality (14).

Now, let us show that for all (t ₀, x ₀) ∈ G

$$ c_{2}(t_{0},x_{0})\geq \lim_{\delta\downarrow 0}\sup\{\sigma_{2}(x^{1}[T,t_{0},x_{0},U^{*},\Delta,v[\cdot]), d(\Delta)\leq\delta,v[\cdot]\in\mathcal{V}\}. $$

(18)

Let $\Delta =\{t_{k}\}_{k=1}^{r}$ be a partition of the interval [t ₀, T], and let v[⋅] be a control of player II. Denote the state of the system at time t _k by x _k , the state of the first player’s guide by ${w^{1}_{k}}=\left ({d_{k}^{1}},\tau _{k},w^{(a),1}_{k},w^{1,(c)}_{k}\right )$. Also, let ${z^{1}_{k}}$ be chosen by rule (5) at time t _k .

We claim that inequality (12) is valid with $\gamma _{k}=\|{z_{k}^{1}}-x_{k}\|^{2}$. Note that $\tau ^{1}_{k+1}=t_{k}$, $d^{1}_{k+1}=\|{z^{1}_{k}}-x_{k}\|^{2}$. If $z^{1}_{k+1}=w^{1,(c)}_{k+1}$, then inequality (12) holds by construction. If $z^{1}_{k+1}=w^{1,(c)}_{k+1}$, then by using Lemma 3.3, we obtain that inequality (12) is fulfilled also.

Therefore, we have inequality (13) with γ ₀ = 0 and $\gamma _{k}=\|{z_{k}^{1}}-x_{k}\|^{2}$. Hence,

$$\|{z^{1}_{r}}-x_{r}\|\leq \varkappa(d(\Delta)). $$

Consequently, inequality (17) is fulfilled for i = 1, j = 2.

It follows from Eqs. (5), (10), and (11) that

$$ c_{2}\left(t_{k+1},z^{1}_{k+1}\right)\leq c_{2}\left(t_{k},{z^{1}_{k}}\right). $$

(19)

Condition (F1) and the equality ${z_{0}^{1}}=x_{0}$ yield the inequality

$$ \sigma_{2}\left({z_{r}^{1}}\right)=c_{2}\left(T,{z_{r}^{1}}\right)\leq c_{2}(t_{0},x_{0}). $$

From this and Eq. (19), we conclude that

$$\sigma_{2}(x^{1}[T,t_{0},x_{0},U^{*},\Delta,v[\cdot]])\leq c_{2}(t_{0},x_{0})+\phi_{2}(\varkappa(\delta)), $$

with δ = d(Δ). Passing to the limit, we get inequality (18).

Analogously, one can prove the inequality

$$ c_{1}(t_{0},x_{0})\geq \lim_{\delta\downarrow 0}\sup\left\{\sigma_{1}\left(x^{2}[T,t_{0},x_{0},V^{*},\Delta,u[\cdot]\right), d(\Delta)\leq\delta,u[\cdot]\in\mathcal{U}\right\}. $$

(20)

Combining equality (14) and inequalities (18) and (20), we conclude that the strategies U ^∗ and V ^∗ form the control with guide Nash equilibrium on G. Moreover, the Nash equilibrium payoff of player i at the position (t ₀, x ₀) is c _i (t ₀, x ₀). Hence, (c ₁, c ₂) is a value function.

3.2 Infinitesimal Form of Conditions (F1)–(F4)

Define

$$H_{1}(t,x,s) := \max_{u\in P}\min_{v\in Q}\langle s,f(t,x,u)+g(t,x,v)\rangle,$$

$$H_{2}(t,x,s) := \max_{v\in Q}\min_{u\in P}\langle s,f(t,x,u)+g(t,x,v)\rangle. $$

Proposition 3.1

Conditions (F2) and (F3) are equivalent to the the following one: the function c _i is a viscosity supersolution of the following equation:

$$\frac{\partial c_{i}}{\partial t}+H_{i}(t,x,\nabla c_{i})=0. $$

(21)

This proposition directly follows from [20, Theorem 6.4].

Further, define a modulus derivative at the position (t, x) in the direction w ∈ ℝⁿ by the rule

$$\begin{array}{@{}rcl@{}} &&{\mathrm{{d}_{abs}}}(c_{1},c_{2})(t,x;w)\\ && := \liminf_{\delta\downarrow 0,w'\rightarrow w}\frac{|c_{1}(t+\delta,x+\delta w')-c_{1}(t,x)|+|c_{2}(t+\delta,x+\delta w')-c_{2}(t,x)|}{\delta}. \end{array} $$

Proposition 3.2

Condition (F4) is valid if and only if for every ${(t,x)\in [0,T]\times \mathbb {R}^{n}}$

$$\inf_{w\in\mathcal{F}(t,x)}{\mathrm{{d}_{abs}}}(c_{1},c_{2})(t,x;w)=0. $$

Proof

Condition (F4) means that the graph of the function (c ₁, c ₂) is viable under the differential inclusion

$$\left(\begin{array}{c} \dot{x} \\ \dot{J}_{1} \\ \dot{J}_{2} \end{array} \right)= \text{co}\left\{\left(\begin{array}{c} f(t,x,u)+g(t,x,v) \\ 0 \\ 0 \end{array} \right):u\in P,v\in Q\right\}. $$

One can rewrite this condition in the infinitesimal form [1, Theorem 11.1.3]: for J ₁ = c ₁(t, x), J ₂ = c ₂(t, x) and some w ∈ co{f(t, x, u) + g(t, x, v):u ∈ P, v ∈ Q}, the inclusion

$$ \left(\begin{array}{c} w \\ 0 \\ 0 \end{array} \right)\in D\text{gr}(c_{1},c_{2})(t,(x,J_{1},J_{2})) $$

(22)

holds. Here, D denotes the contingent derivative. It is defined in the following way. Let $\mathcal {G}\subset [0,T]\times \mathbb {R}^{m}$, $\mathcal {G}[t]$ denote a section of $\mathcal {G}$ by t:

$$\mathcal{G}[t] := \{w\in \mathbb{R}^{m}:(t,x)\in \mathcal{G}\},$$

and let the symbol d denote the Euclidian distance between a point and a set. Following [1], set

$$D\mathcal{G}(t,y) := \left\{h\in\mathbb{R}^{m}: \liminf_{\delta\rightarrow 0}\frac{\mathrm{d}(y+\delta h; \mathcal{G}[t+\delta])}{\delta}=0\right\}. $$

Let J _i = c _i (t, x). We have that (w, Y ₁, Y ₂) ∈ Dgr(c ₁, c ₂)(t, (x, J ₁, J ₂)) if and only if there exist sequences $\{w_{k}\}_{k=1}^{\infty }$ and $\{\delta _{k}\}_{k=1}^{\infty }$ such that $w=\lim _{k\rightarrow \infty }w_{k}$, and

$$Y_{i}=\lim_{k\rightarrow\infty}\frac{c_{i}(t+\delta_{k},x+\delta_{k}w_{k})-c_{i}(t,x)}{\delta_{k}}.$$

Therefore, condition (22) is equivalent to the condition d _abs (c ₁, c ₂)(t, x; w) = 0 for some w ∈ co{f(t, x, u)+g(t, x, v):u ∈ P, v ∈ Q}.

3.3 System of the Hamilton–Jacobi Equations

It is well known that the solutions of the system of the Hamilton–Jacobi equations provide Nash equilibria [5]. Let us show that Theorem 3.1 generalizes the method based on the system of the Hamilton–Jacobi equations.

For any $s\in \mathbb {R}^{n}$, let $\hat {u}(t,x,s_{1})$ satisfy the condition

$$\langle s,f(t,x,\hat{u}(t,x,s))\rangle=\max\{\langle s,f(t,x,u)\rangle:u\in P\}, $$

and let $\hat {v}(t,x,s)$ satisfy the condition

$$\langle s,g(t,x,\hat{v}(t,x,s))\rangle=\max\{\langle s,g(t,x,u)\rangle:u\in P\}. $$

Set

$$\mathcal{H}_{i}(t,x,s_{1},s_{2}) := \langle s_{i}, f(t,x,\hat{u}(t,x,s_{1}))+g(t,x,\hat{v}(t,x,s_{2}))\rangle. $$

Consider the system of the Hamilton–Jacobi equations:

$$ \left\{ \begin{array}{l} \frac{\partial \varphi_{i}}{\partial t}+\mathcal{H}_{i}(t,x,\nabla \varphi_{1},\nabla \varphi_{2}) =0, \\ \varphi_{i}(T,x)=\sigma_{i}(x). \end{array} \right. \ \ i=1,2 $$

(23)

Proposition 3.3

If the function (φ ₁ ,φ ₂ ) is a classical solution of system ( 23 ), then it satisfies condition (F1)–(F4).

Proof

Condition (F1) is obvious.

Since (φ ₁, φ ₂) is the solution of system (23), we have that

$$\begin{array}{*{20}l} 0&=\frac{\partial \varphi_{1}(t,x)}{\partial t}+\max\limits_{u\in P}\langle \nabla \varphi_{1}(t,x),f(t,x,u)\rangle\\ &{\kern12pt}+\langle \nabla \varphi_{1}(t,x),g(t,x,\hat{v}(t,x,\nabla \varphi_{1}(t,x)))\rangle\\ &\geq \frac{\partial \varphi_{1}(t,x)}{\partial t}+\max\limits_{u\in P}\langle \nabla \varphi_{1}(t,x),f(t,x,u)\rangle\\ &{\kern12pt} +\min\limits_{v\in Q}\langle \nabla \varphi_{1}(t,x),g(t,x,v)\rangle\\ &=\frac{\partial \varphi_{1}(t,x)}{\partial t}+H_{1}(t,x,\nabla \varphi_{1}(t,x)). \end{array} $$

The subdifferential of the smooth function φ ₁ is equal to D ⁻ φ ₁(t, x) = {(∂φ ₁(t, x)/∂t, ∇φ ₁(t, x))}. Therefore, φ ₁ is a viscosity supersolution of Eq. (21) for i = 1 [20, Definition (U4)]. This is equivalent to condition (F2).

Condition (F3) is proved in the same way.

$$\begin{array}{*{20}l} &{\rm{d_{abs}}}(\varphi_{1},\varphi_{2})(t,x;w)\\ &=\left|\frac{\partial \varphi_{1}(t,x)}{\partial t}+\langle \nabla \varphi_{1}(t,x),w\rangle\right|+\left|\frac{\partial \varphi_{2}(t,x)}{\partial t}+\langle \nabla \varphi_{2}(t,x),w\rangle\right|. \end{array} $$

Substituting $w=f(t,x,\hat {u}(t,x,\nabla \varphi _{1}(t,x)))+g(t,x,\hat {v}(t,x,\nabla \varphi _{2}(t,x)))$ gives condition (F4).

Generally, there exists a smooth function (c ₁, c ₂) satisfying conditions (F1)–(F4) not being a solution of the system of the Hamilton–Jacobi equations.

Example 3.1

Consider the system

$$ \left\{\begin{array}{cc} \dot{x}_{1}= & -v \\ \dot{x}_{2}= & 2u+v \end{array}\right. $$

(24)

Here, t ∈ [0, 1], u, v ∈ [− 1, 1]. The purpose of the i-th player is to maximize x _i (1).

The function $(c_{1}^{*},c_{2}^{*})$ with $c_{1}^{*}(t,x_{1},x_{2})=x_{1}+(1-t)$, $c_{2}^{*}(t,x_{1},x_{2})=x_{2}+(1-t)$ satisfies conditions (F1)–(F4), but it is not a solution of the system of the Hamilton–Jacobi equations (23). Moreover, $c_{i}^{*}(t,x)> \varphi _{i}(t,x)$ for some solutions of system (23) (φ ₁, φ ₂).

Proof

First, let us write down the system of the Hamilton–Jacobi equations for the case under consideration. Denote ∂φ ₁/∂x _j by p _j , ∂φ ₂/∂x _j by q _j .

The variables $\hat {u}$ and $\hat {v}$ satisfy the conditions

$$\max_{u\in [-1,1]}p_{2}u=p_{2}\hat{u}, \ \ \max_{v\in [-1,1]}(-q_{1}+q_{2})v=(-q_{1}+q_{2})\hat{v}.$$

Hence, the system of the Hamilton–Jacobi equations (23) takes the form

$$ \left\{ \begin{array}{cc} \frac{\partial \varphi_{1}}{\partial t}- p_{1} \hat{v}+p_{2}(2\hat{u}+\hat{v}) & =0 \vspace{4pt},\\ \frac{\partial \varphi_{2}}{\partial t}-q_{1} \hat{v}+q_{2}(2\hat{u}+\hat{v}) & =0. \end{array}\right. $$

(25)

The boundary conditions are φ ₁(1, x ₁, x ₂) = x ₁ and φ ₂(1, x ₁, x ₂) = x ₂.

The function $(c_{1}^{*},c_{2}^{*})$ satisfies conditions (F1)–(F4). Indeed, condition (F1) holds obviously. Condition (F2) is valid with v = 1, and analogously, condition (F3) is valid with u = − 1. Moreover, both players can keep the values of the functions if they use the controls v = − 1, u = 1. This means that condition (F4) holds also.

On the other hand, the pair of functions $\left (c_{1}^{*},c_{2}^{*}\right )$ does not satisfy the system of the Hamilton–Jacobi equations. Indeed,

$$\begin{array}{@{}rcl@{}} &&\partial c_{1}^{*}/\partial x_{1}=p_{1}=1, \ \ \partial c_{1}^{*}/\partial x_{2}=p_{2}=0, \ \ \partial c_{2}^{*}/\partial x_{1}=q_{1}=0, \\ &&\partial c_{2}^{*}/\partial x_{2}=q_{2}=1, \ \ \partial c_{1}^{*}/\partial t=\partial c_{2}^{*}/\partial t=-1. \end{array} $$

Therefore, $\hat {v}=1$. Substitution into the first equation of (25) leads to the contradiction.

Further, consider the functions φ ₁(t, x ₁, x ₂) = x ₁−(1−t), $\varphi _{2}^{\alpha }(t,x_{1},x_{2})=x_{2}+(1+2\alpha )(1-t)$. Here, α is a parameter from [− 1, 1]. Note that if $\hat {v}=1$ and $\hat {u}=\alpha$, then $\left (\varphi _{1},\varphi _{2}^{\alpha }\right )$ is a classical solution of system (25).

We have that for α ∈ [− 1, 0)

$$c_{1}^{*}(t,x_{1},x_{2})> \varphi_{1}(t,x_{1},x_{2}), \ \ c_{1}^{*}(t,x_{1},x_{2})> \varphi_{1}^{\alpha}(t,x_{1},x_{2}).$$

□

3.4 Problem of Continuous Value Function Existence

The continuous function (c ₁, c ₂) satisfying conditions (F1)–(F4) does not exist in the general case.

Example 3.2

Let the dynamics of the system be given by

$$\dot{x}=u, \ \ t\in [0,1], x\in\mathbb{R}, u\in [-1,1].$$

The purpose of the first player is to maximize |x(1)|. The second player is fictitious, and his purpose is to maximize x(1). In this case, there is no continuous function satisfying conditions (F1)–(F4).

Proof

Let a function (c ₁, c ₂) : [0, 1] × ℝ → ℝ² be a value function. Since the payoff of player I does not depend on the control of player II, we have that c ₁(t, x) = |x| + (1 − t) and the Nash equilibrium strategy of the player I U ^∗ = (u, ψ ₁, χ ₁) should satisfy the conditions u(t, x, w ¹) ∈ {− 1, 1} and

$$u\left(t,x,w^{1}\right)=\left\{ \begin{array}{cc} 1, & x>0, \\ -1, & x< 0. \end{array} \right.$$

Therefore, c ₂(t, x) = x + (1 − t) for x > 0 and c ₂(t, x) = x − (1 − t) for x < 0. Note that the value of the function c ₂ at the positions (t,0) is determined only by the condition c ₂(t, 0) ∈ {(1 − t), − (1 − t)}. Thus, there is a nonuniqueness of the value functions. □

The example shows that we need to modify Theorem 3.1 for the case of discontinuous value functions. A natural way is to consider value multifunctions.

4 Value Multifunctions

A multifunction S : [0, T] × ℝⁿ ⇉ ℝ² is called a value multifunction if any of its selector is a value function in the sense of Definition 2.2.

Theorem 4.1

Assume that there exists an upper semicontinuous multifunction S : [0, T] × ℝⁿ ⇉ ℝ² with nonempty images satisfying the following conditions:

• (S1) S (T, x) = {(σ ₁(x), σ ₂(x))}, x ∈ ℝⁿ;

• (S2) For all (t, x) ∈ [0, T] × ℝⁿ , (ⁿJ₁, J ₂) ∈ S(t, x), u ∈ P and t ₊ ∈ [t, T], there exists a motion y ²(⋅) ∈ Sol ²(t, x; u) and a pair (J ₁′, J ₂′) ∈ S(t ₊, y ²(t ₊)) such that J ₁ ≥ J ₁′;

• (S3) For all (t, x) ∈ [0, T] × ℝⁿ, (J ₁, J ₂) ∈ S(t, x), v ∈ Q and t ₊ ∈ [t, T], there exists a motion y ¹(⋅) ∈ Sol ¹(t, x; u) and a pair $\left (J_{1}^{\prime \prime },J_{2}^{\prime \prime }\right )\in S\left (t_{+},y^{1}(t_{+})\right )$ such that $J_{2}\geq J_{2}^{\prime \prime }$;

• (S4) For all (t, x) ∈ [0, T] × ℝⁿ, (J ₁, J ₂) ∈ S(t, x) and t ₊ ∈ [t, T], there exists a motion y ^(c)(⋅) ∈ Sol(t _∗, x _∗) such that (J ₁, J ₂) ∈ S(t ₊, y ^(c)(t ₊)).

Then S is a value multifunction, i.e., for any selector $(\hat {J}_{1},\hat {J}_{2})$ of the multifunction S and a compact set G ⊂ [0, T] × ℝⁿ, there exists a control with guide Nash equilibrium on G such that the corresponding Nash equilibrium payoff at (t ₀, x ₀) ∈ G is $(\hat {J}_{1}(t_{0},x_{0}),\hat {J}_{2}(t_{0},x_{0}))\in S(t_{0},x_{0})$.

Remark 4.1

Let U ^∗, V ^∗ be a Nash equilibrium constructed for the compact G ⊂ [0, T] × ℝⁿ and the selector $(\hat {J}_{1},\hat {J}_{2})$. The value of $(\hat {J}_{1},\hat {J}_{2})$ may vary along the Nash trajectory $x_{*}^{c}[\cdot ]$, that is, a limit of step-by-step motions generated by U ^∗ and V ^∗. However, it follows from Theorem 4.1 that for any intermediate time instant θ, there exists a Nash equilibrium such that the corresponding payoff at $(\theta ,x^{c}_{*}[\theta ])$ is equal to the value of $(\hat {J}_{1},\hat {J}_{2})$ at the initial position.

Analogously, if $x_{*}^{1}[\cdot ]$ is a limit of step-by-by step motions generated by strategy of player I U ^∗ and a control of player II v[⋅], then for any intermediate time instant θ, there exists a Nash equilibrium such that the corresponding payoff at $(\theta ,x^{1}_{*}[\theta ])$ of the player II does not exceed the value of the function $\hat {J}_{2}$ at the initial position.

Remark 4.2

Below, we prove the existence of multifunction satisfying conditions (S1)–(S4) (see Theorem 5.2). Since properties (S1)–(S4) are preserved under pointwise union and closure, there exists the maximal by inclusion multivalued map S _max satisfying conditions of Theorem 4.1. Choose that the value of the selector $(J^{*}_{1}(t,x),J^{*}_{2}(t,x))$ be equal to a Pareto optimal for the set S _max(t, x). The equilibrium corresponding to this selector is an optimal Nash equilibrium achieved in control with guide strategies.

Proof of Theorem 4.1

To prove the theorem, we modify the construction proposed in the Section 3. We add the expected payoff to the guide. The selector $(\hat {J}_{1},\hat {J_{2}})$ is used only at the initial position. The starting value of the expected payoff at (t ₀, x ₀) is equal to $(\hat {J}_{1}(t_{0},x_{0}),\hat {J}_{2}(t_{0},x_{0}))$. In the times of control correction t _k , the expected payoff is recomputed in such way that if both players use Nash equilibrium strategies, then the expected payoff at t _k is equal to the value of the selector at the initial position and belongs to $S\left (t_{k},{z_{k}^{i}}\right )$, where ${z_{k}^{i}}$ is a point close to the state of the system at time t _k .

Thus, the guide consists of the following components: d ∈ ℝ is an accumulated error, τ ∈ ℝ is a previous time of correction, w ^(a) is a punishment part of the guide, w ^(c) is a consistent part of the guide, and Y ₁ ∈ ℝ and Y ₂ ∈ ℝ are expected payoffs of the players.

Let (t, x) ∈ [0, T] × ℝⁿ be a position, t ₊ > t, (J ₁, J ₂) ∈ S(t, x), u ∈ P, v ∈ Q. Let motions y ²(⋅) and y ¹(⋅) satisfy conditions (S2) and (S3), respectively. Denote b ²(t ₊, t, x, J ₁, J ₂, u) := y ²(t ₊), b ¹(t ₊, t, x, J ₁, J ₂, v) := y ¹(t ₊). Also, if y ^(c)(⋅) satisfies condition (S4), then put b ^c(t ₊, t, x, J ₁, J ₂) := y ^(c)(t ₊).

First, let us define the functions

$$\chi_{1}(t,x)=\chi_{2}(t,x) := \left(d_{0},\tau_{0},w^{(c)}_{0},w^{(a)}_{0}, Y_{1,0}, Y_{2,0}\right)$$

by the following rule: d ₀ := 0, τ ₀ := t, $w^{(c)}_{0}=w^{(a)}_{0} := x$, $Y_{1,0} := \hat {J}_{1}(t_{0},x_{0})$, $Y_{2,0} := \hat {J}_{2}(t_{0},x_{0})$.

Now, we shall define controls and transitional functions of the guides. Assume that at time t, the state of the system is x, and the state of the i-th player’s guide is $w^{i}=\left (d^{i},\tau ^{i},w^{(a),i},w^{(c),i}, {Y_{1}^{i}}, {Y_{2}^{i}}\right )$. Define z ⁱ by rule (5). Now, let us consider the case of the first player. Put

$$\left(Y_{1,+}^{1}, Y_{2,+}^{1}\right) := \left\{ \begin{array}{cc} \left({Y_{1}^{i}}, {Y_{2}^{i}}\right), & z^{1}=w^{(c),1} \\ \left(Y_{1}^{\prime\prime}, Y_{2}^{\prime\prime}\right), & z^{1}=w^{(a),1}. \end{array}\right. $$

Here, $\left (Y_{1}^{\prime \prime }, Y_{2}^{\prime \prime }\right )$ is an element of S(t, w ^(a),1) such that $Y_{2}^{\prime \prime }=\min \{J_{2}:(J_{1},J_{2})\in S(t,w^{(a),1})\}$. Choose u _∗ by rule (6) and v ^∗ by Eq. (7). As above, put u(t, x, w) := u _∗ and also set $\psi _{1}\left (t_{+},t,x,w^{1}\right ) := \left (d^{1}_{+},\tau ^{1}_{+},w^{(a),1}_{+},w^{(c),1}_{+}, Y_{1,+}^{1}, Y_{2,+}^{1}\right )$ where

$$\begin{array}{@{}rcl@{}} d_{+}^{1}=\|z^{1}-x\|^{2}, \ \ \tau_{+}^{1}=t, \ \ w^{(a),1}_{+}&=&b_{1}\left(t_{+},t,z^{1}, Y_{1,+}^{1}, Y_{2,+}^{1},v_{*}\right),\\ w^{(c),1}_{+}&=&b_{c}\left(t_{+},t,z^{1}, Y_{1,+}^{1}, Y_{2,+}^{1}\right). \end{array} $$

The case of the second player is considered in the same way. Put

$$\left(Y_{1,+}^{2}, Y_{2,+}^{2}\right) := \left\{ \begin{array}{cc} \left({Y_{1}^{i}}, {Y_{2}^{i}}\right), & z^{2}=w^{(c),2} \\ \left(Y_{1}^{\prime}, Y_{2}^{\prime}\right), & z^{2}=w^{(a),2}. \end{array}\right. $$

Here, (Y1′, Y2′) is an element of S(t ₊, w ^(a),2) such that Y1′= min{J ₁:(J ₁, J ₂) ∈ S(t, w ^(a),2)}. Let v _∗ satisfy condition (8). Also, let u ^∗ satisfy condition (9). Put v(t, x, w) := v _∗. Further, set $\psi _{2}\left (t_{+},t,x,w^{2}\right ) := \left (d^{2}_{+},\tau ^{2}_{+},w^{(a),1}_{+},w^{(c),2}_{+}, Y_{1,+}^{2}, Y_{2,+}^{2}\right )$ where

$$\begin{array}{@{}rcl@{}} d_{+}^{2}=\|z^{2}-x\|^{2}, \ \ \tau_{+}=t,\ \ w^{(a),2}_{+}&=&b_{2}\left(t_{+},t,z^{2}, Y_{1,+}^{2}, Y_{2,+}^{2},v_{*}\right),\\ w^{(c),2}_{+}&=&b_{c}\left(t_{+},t,z^{2}, Y_{1,+}^{2}, Y_{2,+}^{2}\right). \end{array} $$

Let us prove that the following equality holds at any position (t ₀, x ₀) ∈ G:

$$ \hat{J}_{i}=\lim\limits_{\delta\downarrow 0}\inf\{\sigma_{i}(x^{(c)}[T,t_{0},x_{0},U^{*},V^{*},\Delta]), d(\Delta)\leq\delta\}, \ \ i=1,2. $$

(26)

Let $\Delta =\{t_{k}\}_{k=0}^{r}$ be a partition of [t ₀, T], d(Δ) ≤ δ, x ^c[⋅] := x ^c[⋅, t _∗, x _∗, U ^∗, V ^∗,Δ]. Extend the partition Δ by adding the element t _{r + 1} = t _r = T. Denote x _k := x ^c[t _k ]. Let us denote the state of the i-th player’s guide at time t _k by ${w^{i}_{k}}=\left ({d^{i}_{k}},w^{(a),i}_{k},w^{(c),i}_{k}, Y_{1,k}^{i}, Y_{2,k}^{i}\right )$. Let ${z^{i}_{k}}$ be a position chosen by rule (5) for the i-th player at time t _k .

It follows from Lemma 3.2 that the point ${z^{i}_{k}}$ is equal to $w^{(c),i}_{k}$. In addition, $w_{k}^{(c),1}=w_{k}^{(c),2}$, and the following inequality is valid:

$$ \|x_{k}-w^{(c),i}_{k}\|\leq \|x_{k-1}-z_{k-1}^{i}\|^{2}(1+\beta(t_{k}-\tau_{k-1}))+\varphi(t_{k}-\tau_{k-1})(t_{k}-\tau_{k-1}). $$

Applying this inequality sequentially and using the equality ${z_{0}^{i}}=x_{0}$, we get estimate (16) for i = 1, 2. Further, estimate (17) holds for i = 1, 2, j = 1, 2. The choice of ${z_{k}^{i}}$ yields that $(Y_{1,k}^{i}, Y_{2,k}^{i})=(Y_{1,k-1}^{i}, Y_{2,k-1}^{i})$, and $\left (Y_{1,k}^{i}, Y_{2,k}^{i}\right )\in S\left (t_{k-1},z_{k-1}^{1}\right )$ for $k=\overline {1,r+1}$. Also, the construction of the function χ _i leads to the equality $\left (Y_{1,0}^{i}, Y_{2,0}^{i}\right )=(\hat {J}_{1}(t_{0},x_{0}),\hat {J}_{2}(t_{0},x_{0}))$. Hence, $(\hat {J}_{1}(t_{0},x_{0}),\hat {J}_{2}(t_{0},x_{0}))\in S\left (t_{r},{z_{r}^{i}}\right )=\left \{\left (\sigma _{1}({z_{r}^{i}}),\sigma _{2}({z_{r}^{i}})\right )\right \}$. By Eq. (17), we conclude that equality (26) holds.

Now, let us prove that for any position (t ₀, x ₀) ∈ G, the following inequality is fulfilled:

$$ \hat{J}_{2}(t_{0},x_{0})\\\geq \lim\limits_{\delta\downarrow 0}\sup\left\{\sigma_{2}\left(x^{1}[T,t_{0},x_{0},U^{*},\Delta,v[\cdot]]\right), d(\Delta)\leq\delta, v[\cdot]\in \mathcal{V}\right\}. $$

(27)

As above, let $\Delta =\{t_{k}\}_{k=0}^{r}$ be a partition of the interval [t ₀, T], d(Δ) ≤ δ, x ¹[⋅] = x ¹[⋅, t ₀, x ₀, U ^∗,Δ, v[⋅]]. We add the element t _{r + 1} = t _r = T to the partition Δ. Denote x _k := x ¹[t _k ]. Let us denote the state of the first player’s guide at time t _k by ${w^{1}_{k}}=\left ({d^{1}_{k}},w^{(a),1}_{k},w^{(c),1}_{k}, Y_{1,k}^{1}, Y_{2,k}^{1}\right )$. Further, let ${z^{1}_{k}}$ be a point chosen by rule (5) for the first player at time t _k .

The choice of ${z_{k}^{1}}$ (see Eq. (5)) and Lemma 3.3 yield the inequality

$$\|x_{k}-{z_{k}^{1}}\|^{2}\leq \|x_{k-1}-z_{k-1}^{1}\|^{2}(1+\beta(t_{k}-t_{k-1}))+\varphi(t_{k}-t_{k-1})(t_{k}-t_{k-1}). $$

Applying this inequality sequentially and using the equality ${z_{0}^{1}}=x_{0}$, we get estimate (16) for i = 1. Therefore, inequality (17) is fulfilled for i = 1, j = 2. In addition, $Y_{2,k}^{1}\geq Y_{2,k-1}^{2}$. Indeed, if ${z_{k}^{1}}=w^{(c),1}_{k}$, then $\left (Y_{1,k}^{1}, Y_{2,k}^{1}\right )=\left (Y_{1,k-1}^{1}, Y_{2,k-1}^{1}\right )$. If ${z_{k}^{1}}=w^{(a),1}_{k}$, we have that an element $(Y_{1,k}^{1}, Y_{2,k}^{1})$ is chosen so that $Y_{2,k}^{1}$ is the minimum of $\left \{J_{2}:(J_{1},J_{2})\in S\left (t_{k-1},z_{k-1}^{1}\right )\right \}$.

By the construction, we have $\left (Y_{1,k}^{1}, Y_{1,k}^{1}\right )\in S\left (t_{k-1},z_{k-1}^{1}\right )$. Hence, using condition (S1), we obtain that

$$ \hat{J}_{2}(t_{0},x_{0})\geq Y_{2,r+1}^{1}=\sigma_{2}\left({z_{r}^{1}}\right). $$

(28)

Since inequality (17) is valid for i = 1, j = 2, estimate (28) yields inequality (27).

Analogously, for any position (t ₀, x ₀) ∈ G, we have the inequality

$$ \hat{J}_{1}(t_{0},x_{0})\geq \lim\limits_{\delta\downarrow 0}\sup\left\{\sigma_{1}\left(x^{2}[T,t_{0},x_{0},V^{*},\Delta,u[\cdot]]\right), d(\Delta)\leq\delta, u[\cdot]\in\mathcal{U}\right\}. $$

(29)

Equality (26) and inequalities (27) and (29) mean that the pair of strategies U ^∗ and V ^∗ is a Nash equilibrium on G. Moreover, the Nash equilibrium payoff at the initial position (t ₀, x ₀) ∈ G is equal to $(\hat {J}_{1}(t_{0},x_{0}),\hat {J}_{2}(t_{0},x_{0}))$.

5 Existence of Value Multifunction

5.1 Discrete Time Game

In order to prove the existence of a multifunction satisfying conditions (S1)–(S4), we introduce the auxiliary discrete time dynamical game. Let N be a natural number, and let δ ^N := T/N be a time step. We discretize [0, T] by means of the uniform grid $\Delta ^{N} := \left \{{t_{k}^{N}}\right \}_{k=0}^{N}$ with ${t_{k}^{N}}=k\delta ^{N}$.

Consider the discrete time control system

$$\begin{array}{@{}rcl@{}} \xi^{N}\left(t_{k+1}^{N}\right)=\xi^{N}(t_{k})&+&\delta^{N}\left[f\left({t_{k}^{N}},\xi^{N}\left({t_{k}^{N}}\right),u\left({t_{k}^{N}}\right)\right)+ g\left({t_{k}^{N}},\xi^{N}\left({t_{k}^{N}}\right),v\left({t_{k}^{N}}\right)\right)\right],\notag\\ &&k=\overline{0,N-1}, \ \ u\left({t_{k}^{N}}\right)\in P,\ \ v\left({t_{k}^{N}}\right)\in Q. \end{array} $$

(30)

Denote

$$\mathcal{U}^{N} := \{u:[0,T]\rightarrow P: u(t)={u_{k}^{N}}\in P~\text{for}~t\in [{t_{k}^{N}},t_{k+1}^{N}[\hspace{1pt}\}, $$

$$\mathcal{V}^{N} := \{v:[0,T]\rightarrow Q: v(t)={v_{k}^{N}}\in Q~\text{ for }~t\in [{t_{k}^{N}},t_{k+1}^{N}[\hspace{1pt}\}. $$

For t _∗ ∈ Δ^N, ξ _∗ ∈ ℝⁿ, $u\in \mathcal {U}^{N}$, and $v\in \mathcal {V}^{N}$, let $\xi ^{N}(\cdot ,t_{*},\xi _{*},u,v):\Delta ^{N}\cap [t_{*},T]\rightarrow \mathbb {R}^{n}$ be a solution of initial value problem (30), ξ ^N(t _∗) = ξ _∗.

First, we shall estimate ∥ξ ^N(t ₊, t _∗, ξ _∗, u, v)−x(t ₊, t _∗, x _∗, u, v)∥.

Let G ⊂ [0, T] × ℝⁿ be a compact of initial positions. Let E′ ⊂ ℝⁿ be a compact such that x(t, t _∗, x _∗, u, v) ∈ E′, and ξ ^N(t, t _∗, x _∗, u, v) ∈ E′ for all natural N, (t _∗, x _∗) ∈ G, t, t _∗ ∈ Δ^N, $u\in \mathcal {U}^{N}$, $v\in \mathcal {V}^{N}$. Set

$$K' := \max\{\|f(t,x,u)+g(t,x,v)\|:t\in [0,T],x\in E',\ \ u\in P,\ \ v\in Q\}.$$

Denote by L′ the Lipschitz constant of the function f + g on [0, T]×E′×P × Q: for all t ∈ [0, T], x′, x″ ∈ E′, u ∈ P, v ∈ Q

$$\|f(t,x',u)+g(t,x',v)-f(t,x^{\prime\prime},u)-g(t,x^{\prime\prime},v)\|\leq L'\|x'-x^{\prime\prime}\|.$$

Further, set

$$\begin{array}{@{}rcl@{}} \varphi^{\prime}(\delta):&=& \sup\{\|f(t',x',u)-f(t^{\prime\prime},x^{\prime\prime},u)\|+\|g(t',x',v)-g(t^{\prime\prime},x^{\prime\prime},v)\|:\\ &&t',t^{\prime\prime}\in [0,T],x',x^{\prime\prime}\in E',\ \ |t'-t'|\leq\delta,\\ &&\|x'-x^{\prime\prime}\|\leq K'\delta,\ \ u\in P,\ \ v\in Q\}. \end{array} $$

Lemma 5.1

If t _∗, t ₊ ∈ Δ^N, t ₊ ≥ t _∗, (t _∗, x _∗), (t _∗, ξ _∗) ∈ G, $u\in \mathcal {U}^{N}$, and $v\in \mathcal {V}^{N}$, then,

$$\begin{array}{@{}rcl@{}} &&\|x(t_{+},t_{*},x_{*},u,v)-\xi^{N}(t_{+},t_{*},\xi_{*},u,v)\|\notag\\ &&\leq \|x_{*}-\xi_{*}\|\exp(2L'(t_{+}-t_{*}))+\varphi^{\prime}(\delta^{N})\exp(L'(t_{+}-t_{*})). \end{array} $$

(31)

Proof

Let m and r be natural numbers such that $t_{*}={t_{m}^{N}}$, $t_{+}={t_{r}^{N}}$. Denote x(⋅) := x(⋅, t _∗, x _∗, u, v), $x_{k} := x\left ({t_{k}^{N}},t_{*},x_{*},u,v\right )$, $\xi _{k} := \xi ^{N}\left ({t_{k}^{N}},t_{*},\xi _{*},u,v\right )$. We have that

$$\begin{array}{@{}rcl@{}} x_{k+1}&=&x_{k}+\int_{{t_{k}^{N}}}^{t_{k+1}^{N}}[f(t,x(t),u_{k})+g(t,x(t),v_{k})]d t\\ &=&x_{k}+\delta^{N}\left[f({t_{k}^{N}},x_{k},u_{k})+g({t_{k}^{N}}x_{k},v_{k})\right]\\ &&+\int_{{t_{k}^{N}}}^{t_{k+1}^{N}}[f(t,x(t),u_{k})+g(t,x(t),v_{k})-f({t_{k}^{N}},x_{k},u_{k})-g({t_{k}^{N}},x_{k},v_{k})]d t. \end{array} $$

Here, u _k and v _k denote the values of u and v on $[{t_{k}^{N}},t_{k+1}^{N}[$, respectively.

Further,

$$\|x(t)-x_{k}\|\leq K'(t-t_{k}), \ \ t\in [t_{k},t_{k+1}]. $$

Therefore, the following inequality is fulfilled:

$$ \int_{t_{k}}^{t_{k+1}}\left[f(t,x(t),u_{k})+g(t,x(t),v_{k})-f\left({t_{k}^{N}},x_{k},u_{k}\right)-g\left({t_{k}^{N}},x_{k},v_{k}\right)\right]dt \leq \delta^{N} \varphi(\delta^{N}). $$

Hence,

$$ \|x_{k+1}-x_{k}-\delta^{N}\left[f\left({t_{k}^{N}},x_{k},u_{k}\right)+g\left({t_{k}^{N}},x_{k},v_{k}\right)\right]\|\leq \delta^{N} \varphi(\delta^{N}). $$

(32)

Further, we have

$$\begin{array}{@{}rcl@{}} x_{k}&+&\delta^{N}\left[f\left({t_{k}^{N}},x_{k},u_{k}\right)+g\left({t_{k}^{N}},x_{k},v_{k}\right)\right]-\xi_{k+1}\\ &=&x_{k}-\xi_{k}+\delta^{N}\left[f\left({t_{k}^{N}},x_{k},u_{k}\right)+g\left({t_{k}^{N}},x_{k},v_{k}\right)- f\left({t_{k}^{N}},\xi_{k},u_{k}\right)-g\left({t_{k}^{N}},\xi_{k},v_{k}\right)\right]. \end{array} $$

Consequently,

$$\|x_{k}+\delta^{N}\left[f\left({t_{k}^{N}},x_{k},u_{k}\right)+g\left({t_{k}^{N}},x_{k},v_{k}\right)\right]-\xi_{k+1}\|\leq \|x_{k}-\xi_{k}\|+\delta^{N} 2L'\|x_{k}-\xi_{k}\|. $$

This inequality and estimate (32) yield that

$$\|x_{k+1}-\xi_{k+1}\|\leq \|x_{k}-\xi_{k}\|+\delta^{N} 2L\|x_{k}-\xi_{k}\|+\delta^{N}\varphi(\delta^{N}). $$

Applying the last inequality sequentially, we get inequality (31). □

Now, let us prove the existence of a function satisfying discrete time analogs of conditions (S1)–(S4).

Theorem 5.1

For any natural N, there exists an upper semicontinuous multifunction Z ^N : Δ^N × ℝⁿ ⇉ ℝ² satisfying the following properties:

• 1. Z ^N(T, ξ) = {(σ ₁(ξ), σ ₂(ξ))};

• 2. For all (t _∗, ξ _∗) ∈ Δ^N × ℝⁿ, u ∈ P, (Y ₁, Y ₂) ∈ Z ^N(t _∗, ξ _∗) and t ₊ ∈ Δ^N, t ₊ > t _∗, there exists a control $v\in \mathcal {V}^{N}$ and a pair (Y ₁′, Y ₂′) ∈ Z ^N(t ₊, ξ ^N(t ₊, t _∗, ξ _∗, u, v)) such that Y ₁ ≥ Y ₁′;

• 3. For all (t _∗, ξ _∗) ∈ Δ^N × ℝⁿ, v ∈ Q, (Y ₁, Y ₂) ∈ Z ^N(t _∗, ξ _∗) and t ₊ ∈ Δ^N, t ₊ > t _∗, there exists a control $u\in \mathcal {V}^{N}$ and a pair $(Y^{\prime \prime }_{1}, Y^{\prime \prime }_{2})\in Z^{N}(t_{+},\xi ^{N}(t_{+},t_{*},\xi _{*},u,v))$ such that $Y_{2}\geq Y_{2}^{\prime \prime }$;

• 4. For all (t _∗, ξ _∗) ∈ Δ^N × ℝⁿ, (Y ₁, Y ₂) ∈ Z ^N(t _∗, ξ _∗) and t ₊ ∈ Δ^N, t ₊ > t _∗, there exist controls $u\in \mathcal {U}^{N}$ and $v\in \mathcal {V}^{N}$ such that (Y ₁, Y ₂) ∈ Z ^N(t ₊, ξ ^N(t ₊, t _∗, ξ _∗, u, v)).

Proof

In the proof, we fix the number N and omit the superindex N. Denote

$$f_{k}(z,u) := \delta f(t_{k},z,u), \ \ g_{k}(z,v) := \delta g(t_{k},z,v). $$

The proof is by inverse induction on k. For k = N, put Z(t _N , z) := {σ ₁(z), σ ₂(z)}.

Now, let $k\in \overline {0,N-1}$. Assume that the values Z(t _{k + 1}, z),…, Z(t _N , z) are constructed for all $z\in \mathbb {R}^{n}$. In addition, suppose that the functions Z(t _{k + 1},⋅),…, Z(t _N ,⋅) are upper semicontinuous. Define

$$\varsigma^{i}_{k+1}(z) := \min\{ Y_{i}:(Y_{1}, Y_{2})\in Z(t_{k+1},z)\}, \ \ i=1,2.$$

It follows from the upper semicontinuity of the multifunction Z(t _{k + 1},⋅) that the functions $\varsigma ^{1}_{k+1}$ and $\varsigma ^{2}_{k+1}$ are lower semicontinuity.

Set

$$W_{k}(z) := \bigcup\limits_{u\in P,v\in Q}Z(t_{k+1},\xi(t_{k+1},t_{k},z,u,v)), $$

$$ {\varrho_{k}^{1}}(z) := \max\limits_{u\in P}\min\limits_{v\in Q}\varsigma_{k+1}^{1}(\xi(t_{k+1},t_{k},z,u,v)), $$

(33)

$$ {\varrho_{k}^{2}}(z) := \max\limits_{v\in Q}\min\limits_{u\in P}\varsigma_{k+1}^{2}(\xi(t_{k+1},t_{k},z,u,v)). $$

(34)

The multifunction W _k is upper semicontinuous. Indeed, let z ^l → z ^∗, and let $\left ({Y_{1}^{l}}, {Y_{2}^{l}}\right )\in W_{k}(z^{l})$ be such that $\left ({Y^{l}_{1}}, {Y^{l}_{2}}\right )\rightarrow \left (Y_{1}^{*}, Y^{*}_{2}\right )$. We have that $\left ({Y^{l}_{1}}, {Y^{l}_{2}}\right )\in Z(t_{k+1},\xi (t_{k+1},t_{k},z^{l},u^{l},v^{l}))$ for some u ^l ∈ P, v ^l∈Q. We can assume without loss of generality that (u ^l, v ^l) → (u ^∗, v ^∗). By the continuity of the functions f _k and g _k , we get that ξ(t _{k + 1}, t _k , z ^l, u ^l, v ^l) = z ^l+f _k (z ^l, u ^l)+g _k (z ^l, v ^l) → ξ(t _{k + 1}, t _k , z ^∗, u ^∗, v ^∗), as l → ∞. The upper semicontinuity of the multifunction Z(t _{k + 1},⋅) yields that $\left (Y^{*}_{1}, Y^{*}_{2}\right )\in Z(t_{k+1},\xi (t_{k+1},t_{k},z^{*},u^{*},v^{*}))\subset W_{k}(z^{*})$.

Now, let us show that the functions ${\varrho ^{i}_{k}}$ are lower semicontinuous. We give the proof only for the case i = 1. For a fixed u ∈ P, consider the function $z\mapsto \min _{v\in Q}\varsigma _{k+1}^{1}(\xi (t_{k+1},t_{k},z,u,v))$. We shall prove that this function is lower semicontinuous, i.e., for any z ^∗, the following inequality holds:

$$ \liminf\limits_{z\rightarrow z^{*}}\min\limits_{v\in Q}\varsigma_{k+1}^{1}(\xi(t_{k+1},t_{k},z,u,v))\geq \min\limits_{v\in Q}\varsigma_{k+1}^{1}\left(\xi\left(t_{k+1},t_{k},z^{*},u,v\right)\right). $$

(35)

Let $\{z^{l}\}_{l=1}^{\infty }$ be a minimizing sequence:

$$\liminf\limits_{z\rightarrow z^{*}}\min\limits_{v\in Q}\varsigma_{k+1}^{1}\left(\xi\left(t_{k+1},t_{k},z,u,v\right)\right)=\lim\limits_{l\rightarrow \infty}\min\limits_{v\in Q}\varsigma_{k+1}^{1}\left(\xi\left(t_{k+1},t_{k},z^{l},u,v\right)\right). $$

Let v ^l ∈ Q satisfy the condition

$$\varsigma^{k+1}_{1}\left(\xi\left(t_{k+1},t_{k},z^{l},u,v^{l}\right)\right)=\min\limits_{v\in Q} \varsigma_{k+1}^{1}\left(\xi\left(t_{k+1},t_{k},z^{l},u,v\right)\right). $$

Hence, we have

$$ \liminf\limits_{z\rightarrow z^{*}}\min\limits_{v\in Q}\varsigma_{k+1}^{1}\left(\xi\left(t_{k+1},t_{k},z,u,v\right)\right)=\lim\limits_{l\rightarrow \infty}\varsigma_{k+1}^{1}\left(\xi\left(t_{k+1},t_{k},z^{l},u,v^{l}\right)\right). $$

(36)

We can assume without loss of generality that the sequence {v ^l} converges to a control v ^∗∈Q. From continuity of the function ξ(t _{k + 1}, t _k ,⋅, u,⋅) and lower semicontinuity of the function $\varsigma _{k+1}^{1}$, we obtain that

$$\begin{array}{@{}rcl@{}} \lim\limits_{l\rightarrow \infty}\varsigma_{k+1}^{1}\left(\xi\left(t_{k+1},t_{k},z^{l},u,v^{l}\right)\right)&\geq& \varsigma_{k+1}^{1}\left(\xi\left(t_{k+1},t_{k},z^{*},u,v^{*}\right)\right)\\ &\geq& \min\limits_{v\in Q}\varsigma_{k+1}^{1}(\xi(t_{k+1},t_{k},z^{*},u,v)). \end{array} $$

This inequality and equality (36) lead inequality (35).

Since the functions $z\mapsto \min _{v\in Q}\varsigma ^{k+1}_{1}(\xi (t_{k+1},t_{k},z,u,v))$ are lower semicontinuous for each u∈P, the function

$$\varrho_{1}^{k}(z)=\max_{u\in P}\min_{v\in Q}\varsigma^{k+1}_{1}(\xi(t_{k+1},t_{k},z,u,v)) $$

is lower semicontinuous.

Put

$$ Z(t_{k},z) := \left\{(Y_{1}, Y_{2})\in W^{k}(z): Y_{i}\geq {\varrho_{k}^{i}}(z), \ \ i=1,2\right\}. $$

(37)

First, we shall prove that it is nonempty. Let z ∈ ℝⁿ. Let u _∗ maximize the right-hand side of Eq. (33), and let v _∗ maximize the right-hand side of Eq. (34). Choose (Y ₁, Y ₂)∈Z(t _{k + 1}, ξ(t _{k + 1}, t _k , z, u _∗, v _∗)). We have that (Y ₁, Y ₂)∈W _k (z). Further,

$$\varrho^{i}_{k}(z)\leq \varsigma_{k+1}^{i}(\xi(t_{k+1},t_{k},z,u_{*},v_{*}))\leq Y_{i}. $$

Therefore, (Y ₁, Y ₂)∈Z(t _k , z).

The upper semicontinuity of the function Z(t _k ,⋅) follows from Eq. (37), the upper semicontinuity of the multifunction W ^k, and the lower semicontinuity of the function ${\varrho _{k}^{i}}(z)$.

Now, let us show that the function Z satisfies conditions 1–4 of the theorem.

Note that conditions 1 and 4 are fulfilled by the construction. Prove conditions 2 and 3. Let $(t_{*},\xi _{*})\in \Delta ^{N}\times \mathbb {R}^{n}$, t ₊∈Δ^N, t ₊>t, u _∗∈P, (Y ₁, Y ₂)∈Z(t _∗, ξ _∗). It suffices to consider the case t = t _k , t ₊ = t _{k + 1}. By construction of the function Z, we have that $Y_{1}\geq {\varrho _{k}^{1}}(\xi _{*})$. From the definition of the function ${\varrho _{k}^{1}}$ (see Eq. (33)), it follows that

$$Y_{1}\geq\max\limits_{u\in P}\min\limits_{v\in Q}\varsigma_{k+1}^{1}(\xi(t_{k+1},t_{k},\xi_{*},u,v))\geq \min\limits_{v\in Q}\varsigma_{k+1}^{1}(\xi(t_{k+1},t_{k},\xi_{*},u_{*},v)). $$

Let v _∗∈Q be a control of player II such that

$$\min\limits_{v\in Q}\varsigma_{k+1}^{1}(\xi(t_{k+1},t_{k},\xi_{*},u_{*},v))=\varsigma_{k+1}^{1}(\xi(t_{k+1},t_{k},\xi_{*},u_{*},v_{*})). $$

From the definition of the function $\varsigma _{k+1}^{1}$, we get that there exists a pair (Y1′, Y2′) ∈ Z(t _{k + 1}, ξ(t _{k + 1}, t _k , ξ _∗, u _∗, v _∗)) such that $Y_{1}^{\prime }=\varsigma _{k+1}^{1}(\xi (t_{k+1},t_{k},\xi _{*},u_{*},v_{*}))$. Consequently, Y ₁ ≥ Y1′. Hence, condition 2 holds. Condition 3 is proved analogously. □

5.2 Continuous Time Dynamics

Theorem 5.2

There exists an upper semicontinuous multifunction S : [0, T] × ℝⁿ ⇉ ℝ² with nonempty images satisfying conditions (S1)–(S4).

The proof of Theorem 5.2 is given in the end of the section.

First, for each N, define the multifunction S ^N : [0, T] × ℝⁿ ⇉ ℝ² by the following rule:

$$ S^{N}(t,x) := \left\{\begin{array}{lr} Z^{N}({t_{k}^{N}},x), & t\in (t_{k-1},t_{k}),\ \ k=\overline{1,N-1} \\ Z^{N}(t_{k},x)\cup Z^{N}(t_{k+1},x), & t=t_{k},\ \ k=\overline{0,N-1} \\ Z^{N}({t_{N}^{N}},x), & t=T \end{array}\right. $$

(38)

The functions S ^N have the closed graph.

Denote

$$B(\nu) := \{x:\|x\|\leq \nu\}.$$

For Σ : [0, T] × ℝⁿ ⇉ ℝ² set

$$\text{Gr}_{\nu} \Sigma := \{(t,x, Y_{1}, Y_{2}):\|x\|\leq \nu, (Y_{1}, Y_{2})\in \Sigma(t,x)\}. $$

The sets Gr _ν S ^N are compacts. Indeed,

$$M_{i,\nu} := \max\{|\sigma_{i}(x(T,t_{*},x_{*},u,v))|:t_{*}\in [0,T],\|x_{*}\|\leq \nu, u\in\mathcal{U},v\in\mathcal{V}\}<\infty.$$

We have that Gr _ν S _N ⊂[0, T]×B(ν)×[−M _{1, ν} , M _{1, ν} ]×[−M _{2, ν} , M _{2, ν} ].

Consider the Hausdorff distance between compact sets A, B ⊂ [0, T] × ℝⁿ × ℝ²

$$\begin{array}{@{}rcl@{}} h(A,B) := \max\left\{\max\limits_{(t,x, Y_{1}, Y_{2})\in A}\mathrm{d}((t,x, Y_{1}, Y_{2}),B),\max\limits_{(t,x, Y_{1}, Y_{2})\in B}\mathrm{d}((t,x, Y_{1}, Y_{2}),A)\right\}. \end{array} $$

Here, d((t, x, Y ₁, Y ₂), A) is the distance from the point (t, x, Y ₁, Y ₂) to the set A generated by the norm

$$\|(t,x, Y_{1}, Y_{2})\|=|t|+\|x\|+| Y_{1}|+| Y_{2}|. $$

Since for any ν the set [0, T]×B(ν + 1)×[−M _{1, ν} , M _{1, ν} ]×[−M _{2, ν} , M _{2, ν} ] is compact, using [18, Theorem 4.18], we get that one can extract a convergent subsequence from the sequence $\{\text {Gr}_{\nu +1} S^{N}\}_{N=1}^{\infty }$.

Using the diagonal process, we construct the subsequence {N _j } such that for any ν, there exists the limit

$$\lim\limits_{j\rightarrow\infty}\text{Gr}_{{\nu}+1}S^{N_{j}}=R_{\nu}.$$

One can choose the subsequence {N _j } satisfying the property:

$$h(\text{Gr}_{{\nu}+1}S^{N_{j}},R_{\nu})\leq 2^{-j}~\text{ for }~j\geq \nu.$$

Denote $\widetilde {S}_{j} := S^{N_{j}}$.

Lemma 5.2

Let $(Y_{1,l}, Y_{2,l})\in \widetilde {S}_{j_{l}}(t_{l},x_{l})$, ∥x _l ∥ ≤ ν + 1, (t _l , x _l ) → (t ^∗, x ^∗), $(Y_{1,l}, Y_{1,l})\rightarrow (Y_{1}^{*}, Y_{2}^{*})$, as l → ∞. Then $(t^{*},x^{*}, Y_{1}^{*}, Y_{2}^{*})\in R_{\nu }$.

Proof

Consider the set $R_{\nu }\cup \{(t^{*},x^{*}, Y_{1}^{*}, Y_{2}^{*})\}$. This set is closed. We claim that

$$ h(\text{Gr}_{{\nu}+1}\widetilde{S}_{j_{l}},R_{\nu}\cup\{(t^{*},x^{*}, Y_{1}^{*}, Y_{2}^{*})\})\rightarrow 0,\ \ l\rightarrow\infty. $$

(39)

Indeed, $\mathrm {d}((t,x,Y_{1},Y_{2}),R_{\nu }\cup \{(t^{*},x^{*}, Y_{1}^{*}, Y_{2}^{*})\})\leq \mathrm {d}((t,x,Y_{1},Y_{2}),R_{\nu })$ for all $(t,x,Y_{1},Y_{2})\leq \text {Gr}_{\nu +1}\widetilde {S}_{j_{l}}$. Hence,

$$ \max\limits_{(t,x,Y_{1},Y_{2})\in \text{Gr}_{{\nu}+1}\widetilde{S}_{j_{l}}}\mathrm{d}((t,x,Y_{1},Y_{2}),R_{\nu}\cup\{(t^{*},x^{*}, Y_{1}^{*}, Y_{2}^{*})\})\rightarrow 0, ~\text{ as }~l\rightarrow \infty. $$

(40)

Further, the following convergence is valid:

$$\max\limits_{(t,x, Y_{1}, Y_{2})\in R_{\nu}\cup \{(t^{*},x^{*}, Y_{1}^{*}, Y_{2}^{*})\}}\{\mathrm{d}((t,x, Y_{1}, Y_{2}),\text{Gr}_{\nu+1}\widetilde{S}_{j_{l}})\}\rightarrow 0, ~\text{ as }~ l\rightarrow\infty. $$

This and Eq. (40) yield Eq. (39).

Formula (39) means that

$$R_{\nu}\cup\{(t^{*},x^{*}, Y_{1}^{*}, Y_{2}^{*})\}=\lim\limits_{l\rightarrow\infty}\text{Gr}_{{\nu}+1}\widetilde{S}_{j_{l}}=R_{\nu}. $$

This completes the proof.

Lemma 5.3

For r > ν, the following equality holds:

$$R_{r}\cap \left([0,T]\times B(\nu)\times\mathbb{R}^{2}\right)=R_{\nu}\cap \left([0,T]\times B(\nu)\times\mathbb{R}^{2}\right). $$

Proof

Let (t, x, Y ₁, Y ₂)∈R _r , ∥x∥ ≤ ν, and j≥r. There exists a quadruple $(\theta _{j},y_{j},\zeta _{1,j},\zeta _{2,j})\in \text {Gr}_{r+1}\widetilde {S}_{j}$ such that

$$ |t-\theta_{j}|+\|x-y_{j}\|+| Y_{1}-\zeta_{1,j}|+| Y_{2}-\zeta_{2,j}|=\mathrm{d}\left(\left(t,x, Y_{1}, Y_{2}\right),\text{Gr}_{r+1}\widetilde{S}_{j}\right)\leq 2^{-j}. $$

(41)

Therefore, $\|x-y_{j}\|\leq \mathrm {d}((t,x, Y_{1}, Y_{2}),\text {Gr}_{{r}+1}\widetilde {S}_{j})\leq 2^{-j}$. We have that ∥y _j ∥ ≤ ∥x∥+2^−j ≤ ν + 1. Therefore, $(\theta _{j},y_{j},\zeta _{1,j},\zeta _{2,j})\in \text {Gr}_{\nu +1}\widetilde {S}_{j}$. It follows from formula (41) and Lemma 5.2 that (t, x, Y ₁, Y ₂)∈R _ν . Since the quadrable (t, x, Y ₁, Y ₂) satisfies the condition ∥x∥ ≤ ν, we conclude that

$$R_{r}\cap ([t_{0},T]\times B(\nu)\times\mathbb{R}^{2})\subset R_{\nu}\cap ([t_{0},T]\times B(\nu)\times\mathbb{R}^{2}). $$

The opposite inclusion is proved in the same way. □

Define the multifunction $\bar {S}:[0,T]\times \mathbb {R}^{n}\rightrightarrows \mathbb {R}^{2}$ by the following rule: for ∥x∥ ≤ ν

$$\bar{S}(t,x) := \{(Y_{1}, Y_{2}):(t,x, Y_{1}, Y_{2})\in R_{\nu}\}.$$

Note that this definition is correct by virtue of Lemma 5.3. We have that $\text {Gr}_{\nu }\bar {S}=R_{\nu }\cap ([t_{0},T]\times B(\nu )\times \mathbb {R}^{2})$.

Proof of theorem 5.2

We shall show that the function $\bar {S}$ has nonempty images and satisfies conditions (S1)–(S4).

First, we shall prove that the sets $\bar {S}(t,x)$ are nonempty. Let ν satisfy the condition ∥x∥ < ν, and let $(Y_{1,j}, Y_{2,j})\in \widetilde {S}_{j}(t,x)$. Since $\widetilde {S}_{j}(t,x)\subset [-M_{1,\nu },M_{1,\nu }]\times [-M_{2,\nu },M_{2,\nu }]$, there exists a subsequence $\{(Y_{1,j_{l}}, Y_{2,j_{l}})\}_{l=1}^{\infty }$ converging to a pair $\left (Y_{1}^{*}, Y_{2}^{*}\right )$. By Lemma 5.2, we obtain that $\left (Y_{1}^{*}, Y_{2}^{*}\right )\in \bar {S}(t,x)$.

Now, let us prove that the multifunction $\bar {S}$ satisfies conditions (S1)–(S4).

We begin with condition (S1). Let x _∗ ∈ ℝⁿ. Choose ν such that the following conditions hold:

• 1. x(t, T, x _∗, u, v)∈B(ν) for all t∈[0, T], $u\in \mathcal {U}$, $v\in \mathcal {V}$;

• 2. All z such that x _∗ = ξ ^N(T, t, z, u, v) for some natural N, t ∈ Δ^N, $u\in \mathcal {U}^{N}$, $v\in \mathcal {V}^{N}$ belong to B(ν).

Let K _ν be defined by Eq. (3) for E = B(ν + 1).

Let N be a natural number, t _∗ ∈ Δ^N, and ξ _∗∈B(ν). By conditions 1 and 4 of Theorem 5.1, we have that if (Y ₁, Y ₂)∈Z ^N(t _∗, ξ _∗), then there exists $u\in \mathcal {U}^{N}$, and $v\in \mathcal {V}^{N}$ such that

$$ Y_{i}=\sigma_{i}\left(\xi^{N}(T,t_{*},\xi_{*},u,v)\right), \ \ i=1,2. $$

(42)

We have the estimate

$$ \|\xi_{*}-\xi^{N}(T,t_{*},\xi_{*},u,v)\|\leq K_{\nu}(T-t_{*}). $$

(43)

Let $(J_{1},J_{2})\in \bar {S}(T,x)$. This means that there exists a sequence $\left \{\left (t_{j},x_{j}, Y_{1,j}, Y_{2,j}\right )\right \}_{j=1}^{\infty }$ such that $(Y_{1,j}, Y_{2,j})\in \widetilde {S}_{j}\left (t_{j},x_{j}\right )=S^{N_{j}}\left (t_{j},x_{j}\right )$, and t _j → T, x _j → x, Y _{i, j} → J _i as j → ∞. Let θ _j ∈ Δ^N _j be such that $\left (Y_{1,j}, Y_{2,j}\right )\in Z^{N_{j}}\left (\theta _{j},x_{j}\right )$ and t _j ∈(θ _j −δ ^N, θ _j ]. Combining these Eqs. (42) and (43), we conclude that for any j, there exists xj′ ∈ B(ν) such that ∥x _j − xj′ ∥ ≤ K _ν (T−t _j ) and Y _{i, j} = σ _i (xj′), i = 1, 2. We have that xj′ → x, as j → ∞. By the continuity of the functions σ _i , we obtain that

$$J_{i}=\lim\limits_{l\rightarrow\infty} Y_{i,j}=\lim\limits_{j\rightarrow\infty} \sigma_{i}\left(x_{j}^{\prime}\right)=\sigma_{i}(x).$$

Now, we shall prove the fulfillment of condition (S2). Let $(t_{*},x_{*})\in [0,T]\times \mathbb {R}^{n}$, $(J_{1},J_{2})\in \bar {S}(t_{*},x_{*})$, u∈P, t ₊∈[t _∗, T]. We shall show that there exists y ²(⋅)∈Sol²(t _∗, x _∗, u) such that J1′ ≤ J ₁ for some $\left (J_{1}^{\prime },J_{2}^{\prime }\right )\in \bar {S}\left (t_{+},y^{2}(t_{+})\right )$.

There exists a sequence $\left \{\left (t_{j},x_{j}, Y_{1,j}, Y_{2,j}\right )\right \}_{j=1}^{\infty }$ such that $\left (Y_{1,j}, Y_{2,j}\right )\in \widetilde {S}_{j}\left (t_{j},x_{j}\right )=S^{N_{j}}\left (t_{j},x_{j}\right )$, and t _j → t _∗, x _j → x _∗, Y _{i, j} → J _i , as j → ∞. Let θ _j be an element of Δ^N _j such that $\left (Y_{1,j}, Y_{2,j}\right )\in Z^{N_{j}}\left (\theta _{j},x_{j}\right )$ and t _j ∈(θ _j −δ ^N, θ _j ]. Further, let τ _j be the least element of Δ^N _j such that t ₊ ≤ τ _j .

By condition 2 of Theorem 5.1 for each j, there exists a control $v_{j}\in \mathcal {V}^{N_{j}}$, and a pair (Y1, j′, Y2, j′) such that $\left (Y_{1,j}^{\prime }, Y_{2,j}^{\prime }\right )\in Z^{N_{j}}\left (\tau _{j},\xi ^{N_{j}}\left (\tau _{j},\theta _{j},x_{j},u,v_{j}\right )\right )\subset \widetilde {S}^{j}\left (\tau _{j},\xi ^{N_{j}}\left (\tau _{j},\theta _{j},x_{j},u,v_{j}\right )\right )$ and Y1, j′ ≤ Y _{1, j} . By Lemma 5.1, we have that

$$\|x\left(\tau_{j},\theta_{j},x_{j},u,v_{j}\right)-\xi^{N_{j}}\left(\tau_{j},\theta_{j},x_{j},u,v_{j}\right)\|\leq \varphi^{\prime}\left(\delta^{N_{j}}\right)\exp(LT).$$

We may extract a subsequence $\{j_{l}\}_{l=1}^{\infty }$ such that $\left \{x\left (\cdot ,\theta _{j_{l}},x_{j_{l}},u,v_{j_{l}}\right )\right \}_{l=1}^{\infty }$ converges to some motion y ²(⋅), and $\{(Y_{1,j_{l}}^{\prime }, Y_{2,j_{l}}^{\prime })\}$ converges to some pair (J1′, J2′). We have that y ²(⋅)∈Sol²(t _∗, x _∗, u). Lemma 5.2 gives the inclusion $\left (J_{1}^{\prime },J_{2}^{\prime }\right )\in \overline {S}\left (t_{+},y^{2}\left (t_{+}\right )\right )$. We also have

$$J_{1}^{\prime}\leq J_{1}. $$

This completes the proof of condition (S2).

Conditions (S3) and (S4) are proved analogously. □