Degree Theory for Generalized Mixed Quasi-variational Inequalities and Its Applications

Abstract

The present paper is devoted to building degree theory for a generalized mixed quasi-variational inequality in finite dimensional spaces. Then, by employing the obtained results, we prove the existence and stability of solutions to the considered generalized mixed quasi-variational inequality.

1 Introduction

Variational inequalities are a flexible and unifying framework, which incorporates optimization problems, fixed point problems, transportation problems, financial equilibrium problems, migration equilibrium problems, saddle point problems and so on; see, e.g., [1,2,3,4,5,6,7]. This is the reason why there is vast literature studying the theory and applications of variation inequalities. There are two kinds of important extensions of the classic variational inequalities. One is the mixed variational inequality (MVI, also known as Hemivariational inequality), which is characterized by the involvement of a proper, convex and lower semicontinuous function into the classic variational inequality. The mixed variational inequality has been studied by many scholars and has also been applied to many practical problems, such as circuits in electronics, power control problems in ad hoc networks and economic equilibrium problems (see, [8,9,10,11,12,13]). The other important extension of the classic variational inequalities is the quasi-variational inequality (QVI), whose constraint set depends on the decision variables. This characterization of the quasi-variational inequality allows one to model many complex problems, such as impulse control problems, frictional elastostatic contact problems, and power markets and generalized Nash equilibrium problems in game theory (see, [1, 2, 14,15,16,17]). However, the above-mentioned two kinds of extensions of variational inequalities are not enough for the applications of variational inequalities. In some practical situations, we need to consider more general extensions of variational inequalities to model more complicated problems, and thus, it is necessary for us to study them first from the theoretical point of view.

The generalized mixed quasi-variational inequality (GMQVI) considered in this paper is, obviously, a generalization of the above mentioned MVI and QVI. In addition, the GMQVI can be also applied to model the noncooperative game in practice; see Example 2.1.

The degree theory is widely used in the study of differential equations and more general functional equations. Particularly, it is also a powerful way to study the existence and stability of solutions for various kinds of variational inequalities. Facchinei and Pang [2] employ the degree theory to obtain some existence theorems for variational inequalities and quasi-variational inequalities (see Proposition 2.2.3, Theorems 2.3.4, 2.8.3 and Corollary 2.8.4 of [2]). These results give some necessary and sufficient conditions for the existence of solutions of the variational inequalities and the quasi-variational inequalities. Later, some results by Facchinei and Pang are extended by Kien et al. in [3] to the cases of the variational inequality and the generalized variational inequality (GVI) in infinite dimensional reflexive Banach spaces. Also, in [18], Kien et al. build a degree theory for the GVI in finite dimensional spaces by using a different method under different conditions. As an application, they employ the obtained degree theory to prove some results of existence and stability of solutions to the GVI. Recently, Wang and Huang [19] build a degree theory for a so-called generalized set-valued variational inequality which is actually a generalization of the GVI and utilize the obtained results to prove an existence of solutions to the generalized set-valued variational inequality in Banach spaces. For more related works on the applications of the degree theory to variational inequalities, we refer readers to [20,21,22,23,24,25,26,27] and the references therein.

As mentioned above, there are few papers studying the GMQVI theoretically, especially constructing the degree theory for the GMQVI although there are abundant results on the degree theory for variational inequalities and quasi-variational inequalities in the literature. In this paper, by means of the generalized f-projection operator and the Brouwer degree, we focus on the establishment of a degree theory for the GMQVI and then employ the obtained results to prove the existence and stability of solutions to the GMQVI. The results presented in this paper extend and improve some corresponding results in [2, 18].

The rest of the paper is organized as follows. In Sect. 2, we recall some basic notation and preliminary results. Then, in Sect. 3, we build the degree theory for the GMQVI. As applications, by using the obtained results, we show some results on the existence and stability of solutions to the GMQVI.

2 Preliminaries

Throughout this paper, unless otherwise stated, we suppose that the norm and the dual pair in Euclidean spaces $\mathbb {R}^{n}$ are denoted by $\Vert \cdot \Vert$ and $\langle \cdot ,\cdot \rangle$, respectively. Let $f: \mathbb {R}^{n}\rightarrow \mathbb {R}$ be a convex function, $K: \mathbb {R}^{n}\rightrightarrows \mathbb {R}^{n}$ be a set-valued mapping with nonempty, closed and convex values, and $F: \mathbb {R}^{n}\rightrightarrows \mathbb {R}^{n}$ be a set-valued mapping. In this paper, we are interested in the following GMQVI: find $x\in K(x)$ and $x^{*}\in F(x)$ such that

$$\begin{aligned} \langle x^*, y-x\rangle +f(y)-f(x)\ge 0,\quad \text {for all }y\in K(x). \end{aligned}$$

(1)

The GMQVI is a class of general models in the study on variational inequalities, and many kinds of existing problems can be cast as its special cases, some of which are listed as follows:

$\centerdot$:

If $f(x)=0$ for all $~x\in \mathbb {R}^{n}$, then the GMQVI reduces to the following generalized quasi-variational inequality problem (GQVI) : find $x\in K(x)$ and $x^*\in {F}(x)$ satisfying

$$\begin{aligned} \langle x^*, y-x\rangle \ge 0,\quad \text {for all }y\in K(x), \end{aligned}$$

(2)

which is introduced and studied in [1] by Chan and Pang.

$\centerdot$:

If $F: \mathbb {R}^{n}\rightarrow \mathbb {R}^{n}$ is a single-valued mapping and $K(x)=C$ with C being a nonempty, closed and convex subset of $\mathbb {R}^n$, then the GMQVI reduces to the following MVI: find $x\in C$ such that

$$\begin{aligned} \langle F(x), y-x\rangle +f(y)-f(x) \ge 0,\quad \text {for all }y\in C, \end{aligned}$$

(3)

which has been considered by Facchinei and Pang in [2].

$\centerdot$:

If $f(x)=0$ for all $~x\in \mathbb {R}^n$ and $K(x)=C$ with C being a nonempty, closed and convex subset of $\mathbb {R}^n$, then the GMQVI reduces to the GVI: find $x\in C$ and $x^{*}\in F(x)$ such that

$$\begin{aligned} \langle x^*, y-x\rangle \ge 0,\quad \text {for all }y\in C. \end{aligned}$$

(4)

which has been studied by many authors; see, e.g., [1,2,3].

Among many GMQVI’s applications, the following general noncooperative game is used to illustrate its importance in practice.

Example 2.1

Consider a general noncooperative game, in which we suppose that there are N players each of whom has a certain cost function and strategy set which depends on the other players’ strategies. For each $i\in \{1,2,\cdots , N\}$, we use the set-valued mapping $K_i:\mathbb {R}^{n-n_i}\rightrightarrows \mathbb {R}^{n_i}$ to denote the ith player’s strategy and his/her cost function is defined by

$$\begin{aligned}\theta _i(x)=:g_i(x)+f_i(x_i),~\forall \, x_i\in R^{n_i}, ~x=(x_i)_{i=1}^N\in \mathbb {R}^{n_1+n_2+\cdots +n_N}=\mathbb {R}^{n},\end{aligned}$$

where $f_i: \mathbb {R}^{n_i}\rightarrow \mathbb {R}$ and $g_i: \mathbb {R}^{n}\rightarrow \mathbb {R}$ are two convex functions.

It is known that a point $(x_i^*)_{i=1}^{N}$ is called a generalized Nash equilibrium for the general noncooperative game if for each fixed but arbitrary tuple of rival action $x_{-i}:=(x_1^*,\cdots ,x_{i-1}^*,x_{i+1}^*,\cdots ,x_N^*)$, $x_i^*$ solves the cost minimization problem in the variable $y_i$ defined as

$$\begin{aligned} \begin{aligned}&\mathrm{minimize}_{y_i} ~~\theta _i(y_i,x_{-i})=g_i(y_i,x_{-i})+f_i(y_i)\\&\mathrm{subject ~to}~~{y_i}\in K_i(x_{-i}). \end{aligned} \end{aligned}$$

We could prove that a point $(x_i^*)_{i=1}^{N}$ is a generalized Nash equilibrium for the general noncooperative game if and only if for each $i\in \{1,2,\cdots , N\}$,

$$\begin{aligned} 0\in \partial f_i(x_i^*)+ \partial _{x_i} g_i(x_i^*,x_{-i}^*)+N_{ K_i(x_{-i}^*)}(x_i^*), \end{aligned}$$

(5)

where $N_{ K_i(x_{-i}^*)}(x_i^*)$ denotes the normal cone of $K_i(x_{-i}^*)$ at $x_i^*$, and $\partial f_i(x_i^*)$ and $\partial _{x_i} g_i(x_i^*,x_{-i}^*)$ denote subdifferential of $f_i$ and $g_i$ with respect to $x_i$, respectively.

Moreover, the problem (5) is equivalent to finding $x_i^*\in K_i(x_{-i}^*)$, $u^*_i\in \partial _{x_i} g_i(x_i^*,x_{-i}^*)$ and $s^*_i\in \partial f_i(x_i^*)$ such that

$$\begin{aligned} \langle u^*_i, y_i-x_i^*\rangle +\langle s^*_i, y_i-x_i^*\rangle \ge 0, \forall ~y_i\in K_i(x_{-i}^*). \end{aligned}$$

Now, let $F(x)=\prod _{i=1}^N \partial _{x_i} g_i(x)$, $f(x)=\sum _{i=1}^{n}f_i(x_i)$ and $K(x)=\prod _{i=1}^N K_i(x_{-i}).$ As done in the proof of Proposition 1 in [28], the problem (5) can be formulated as follows: find $x^*=(x_i^*)_{i=1}^{N}\in K(x^*)$ and $u^*\in F(x^*)$ such that

$$\begin{aligned} \langle u^*, y-x^*\rangle +f(y)-f(x^*)\ge 0, \forall ~y\in K(x^*), \end{aligned}$$

(6)

which is a form of GMQVI.

Next, we present the definition of the Brouwer degree and its properties, which could be found in [29,30,31]. To this end, we assume that D is an open, bounded set in $\mathbb {R}^{n}$ with the boundary $\partial D$ and the closure ${\text {cl}} D$.

Definition 2.1

Suppose that $\phi : {\text {cl}}D\rightarrow \mathbb {R}^n$ is a continuous mapping and $p\in \mathbb {R}^n{\setminus } \phi (\partial D)$. Then we define $d(\phi , D, p):=d({\hat{\phi }}, D, p)=\sum _{x\in {\hat{\phi }}^{-1}(p)}\text {sgn}(J_{{\hat{\phi }}}(x)$, where $\text {sgn}$ is the sign function, ${\hat{\phi }}: {\text {cl}}D\rightarrow \mathbb {R}^n$ is a continuously differential function such that $\Vert \phi (x)-{\hat{\phi }}(x)\Vert <\text {dist}(p,\phi (\partial D))$ for all $x\in {\text {cl}}D$, and $J_{{\hat{\phi }}}(x)$ denotes the Jacobian determinant of ${\hat{\phi }}$.

Theorem 2.1

Suppose that $\phi :{\text {cl}} D\rightarrow \mathbb {R}^n$ is a continuous mapping and $p\in \mathbb {R}^n{\setminus } \phi (\partial D)$. Then, the Brouwer degree $d(\phi , D, p)$ has the following properties:

• ($a_1$) (Normalization) If $p\in D$, then $d(I,D, p)=1$, where I denotes the identity mapping, that is, $I(x)=x$ for all $x\in ~\mathbb {R}^{n}.$

• ($a_2$) (Existence) If $d(\phi , D, p)\ne 0$, then there is $x\in D$ such that $\phi (x)=p$.

• ($a_3$) (Additivity) Suppose that $D_1$ and $D_2$ are disjoint open subsets of D. If $p\not \in \phi ({\text {cl}}D{\setminus }(D_1\bigcup D_2))$, then

  $$\begin{aligned}d(\phi , D, p)=d(\phi , D_1, p)+d(\phi , D_2, p).\end{aligned}$$

  ($a_4$) (Homotopy) If $\phi _t:[0,1]\times {\text {cl}}D\rightarrow \mathbb {R}^n$ is continuous and $p\not \in \cup _{t\in [0,1]}\phi _t(\partial D)$, then $d(\phi _t,D,p)$ does not depend on $t\in [0,1]$.

• ($a_5$) (Excision) If $D_0\subset {\text {cl}} D$ is closed and $p\not \in \phi (D_0)$, then $d(\phi , D, p)=d(\phi , D{\setminus } D_0, p).$

• ($a_6$) If $p\not \in \phi (\partial D)$, then $d(\phi , D, p)=d(\phi -p,D,0).$

Then, we introduce some basic notation and preliminary results as follows.

Definition 2.2

Let ${\mathcal {O}}\subset \mathbb {R}^n$ be a nonempty set and $F:{\mathcal {O}}\rightrightarrows {\mathbb {R}^{n}}$ be a set-valued mapping.

• ($a_1$) F is said to be upper semicontinuous at $x\in {\mathcal {O}}$, if for any open set $V\subset \mathbb {R}^n{\text { with }}F(x)\subset V$, there exists an open neighborhood U of x such that $F(y)\subset V$ for all $y\in U\cap {\mathcal {O}}$. If F is upper semicontinuous at every $x\in {\mathcal {O}}$, we say that F is upper semicontinuous on ${\mathcal {O}}$.

• ($a_2$) F is said to be lower semicontinuous at $x\in {\mathcal {O}}$, if for any open set $W\subset \mathbb {R}^{n}{\text { with }}F(x)\cap W\ne \emptyset$, there exists an open neighborhood U of x such that $F(y)\cap W\ne \emptyset$ for all $y\in U\cap {\mathcal {O}}$. If F is lower semicontinuous at every $x\in {\mathcal {O}}$, we say that F is lower semicontinuous on ${\mathcal {O}}$.

Suppose that $K: \mathbb {R}^{n}\rightrightarrows {\mathbb {R}^{n}}$ is a set-valued mapping with nonempty, closed and convex values. For each $x\in \mathbb {R}^{n}$, the generalized f-normal cone operator associated with the set K(x) at $x'$ is defined as follows:

$$\begin{aligned} N^f_{K(x)}(x')=\left\{ \begin{array}{ll} \{\varphi \in \mathbb {R}^{n} :\langle \varphi ,y-x'\rangle +f(x')-f(y)\le 0,\forall y\in K(x)\}, &{}{\text { if }}x'\in K(x),\\ \emptyset ,&{} {\text { otherwise. }} \end{array} \right. \end{aligned}$$

Obviously, the problem (1) is equivalent to the following problem: find $x\in K(x)$ such that

$$\begin{aligned}0\in F(x)+N^f_{K(x)}(x).\end{aligned}$$

For any fixed $\rho >0$, let ${\mathcal {N}}\subset \mathbb {R}^{n}$ be a nonempty, closed and convex set, $f:{\mathcal {N}}\rightarrow {\mathbb {R}}\cup \{+\infty \}$ be a proper, convex and lower semicontinuous function, and $G: \mathbb {R}^{n}\times {\mathcal {N}}\rightarrow {]}-\infty ,+\infty ]$ be a function defined as follows:

$$\begin{aligned} G(\varphi , x)=\Vert x\Vert ^2-2\langle \varphi ,x\rangle +\Vert \varphi \Vert ^2+ 2\rho f(x), ~\forall ~\varphi \in \mathbb {R}^{n}, ~\forall ~x\in {\mathcal {N}}. \end{aligned}$$

(7)

Through the rest of this paper, unless otherwise stated, we take $\rho =1$ in (7).

Definition 2.3

[32] We say that $\varPi ^f_{\mathcal {N}}:\mathbb {R}^n\rightrightarrows {\mathcal {N}}$ is a generalized f-projection operator if

$$\begin{aligned}\varPi ^f_{{\mathcal {N}}}\varphi =\{u\in {\mathcal {N}}: G(\varphi ,u)=\text {inf}_{y\in {\mathcal {N}}}G(\varphi , y)\}, \quad \forall ~\varphi \in \mathbb {R}^n.\end{aligned}$$

Remark 2.1

The generalized f-projection operator is a generalization of the resolvent operator for the subdifferential $\partial f$ of a proper, convex and lower semi-continuous functional f; see Remark 3.2 of [33]. In addition, if $f (x) \equiv 0$ for all $x\in {\mathcal {N}}$, then the generalized f-projection operator $\varPi ^f_{\mathcal {N}}$ is equivalent to the following metric projection operator

$$\begin{aligned} \varPi _{\mathcal {N}}(\varphi )=\{u\in {\mathcal {N}}: \Vert u-\varphi \Vert =\text {inf}_{y\in {\mathcal {N}}}\Vert y-\varphi \Vert \},\quad \forall ~\varphi \in \mathbb {R}^{n}. \end{aligned}$$

Lemma 2.1

[32, 34] The following statements hold:

• (i) For any given $\varphi \in \mathbb {R}^{n}$, $\varPi ^f_{\mathcal {N}} \varphi$ is a nonempty and single-valued.

• (ii) For any given $\varphi \in \mathbb {R}^{n}$, $x= \varPi ^f_{\mathcal {N}}\varphi$ if and only if

  $$\begin{aligned}\langle x-\varphi ,y-x\rangle + f(y)- f(x)\ge 0,\quad \forall ~y\in {\mathcal {N}}.\end{aligned}$$

• (iii) $\Vert \varPi ^f_{\mathcal {N}} x-\varPi ^f_{\mathcal {N}}y\Vert \le \Vert x-y\Vert$, for all $x,y\in {\mathcal {N}}$.

Lemma 2.2

[35] Let X be a topological space, Y be a regular topological space and $F:X\rightrightarrows Y$ be an upper semicontinuous mapping with closed values. Then F is closed (i.e., its graph is closed).

Lemma 2.3

[35] Let X and Y be topological spaces and $F:X\rightrightarrows Y$ be an upper semicontinuous mapping with compact values. Then for any compact subset E of X, F(E) is compact.

Definition 2.4

[36] Let O be a nonempty subset of $\mathbb {R}^p$ and $K' : O \rightrightarrows {\mathbb {R}^n}$ be a set-valued mapping.

• ($a_1$) $K'$ is said to be outer semicontinuous at $w_0\in O$, if $\limsup _{w\rightarrow w_0\in O}K'(w)\subset K'(w_0)$. If $K'$ is outer semicontinuous at every $w\in O$, we say that $K'$ is outer semicontinuous on O.

• ($a_2$) $K'$ is said to be inner semicontinuous at $w_0\in O$, if $K'(w_0)\subset \liminf _{w\rightarrow w_0\in O}K'(w)$. If $K'$ is inner continuous at every $w\in O$, we say that $K'$ is inner semicontinuous on O.

• ($a_3$) $K'$ is said to be continuous at $w_0\in O$, if $\liminf _{w\rightarrow w_0\in O}K'(w)=\limsup _{w\rightarrow w_0\in O}K'(w)=K'(w_0).$ If $K'$ is continuous at every $x\in O$, we say that $K'$ is continuous on O.

According to Exercise 5.6 and Theorem 5.19 of [36], $K'$ is inner semicontinuous at a point $w_0\in O$ if and only if $K'$ is lower semicontinuous at the point $w_0\in O$, while the outer semicontinuity of $K'$ at the point $w_0\in O$ is equivalent to its upper semicontinuity at the point $w_0\in O$ and the closedness of $K'(w_0)$, if $K'$ is locally bounded at the point $w_0\in O$. Moreover, it follows from Theorem 5.7 of [36] and Lemma 2.2 that $K'$ is outer semicontinuous, if $K'$ is an upper semicontinuous mapping with closed values. Obviously, the mapping $K'$ is continuous at $w_0\in O$ if and only if for any $\{w_n\} \subset O$ with $w_n\rightarrow w_0\in O$, $K'(w_n)$ Mosco-converges to $K'(w_0)$; see Definition 3.2 of [23] and page 152 of [36]. By Theorem 3.2 of [23], it is easy to obtain the following lemma.

Lemma 2.4

Assume that $K':\mathbb {R}^p\rightrightarrows {\mathbb {R}^n}$ is a set-valued mapping with nonempty, closed and convex values. Suppose that $f: \mathbb {R}^n\rightarrow {]-\infty ,+\infty [}$ is convex, and $K'$ is continuous at $\lambda _0\in \mathbb {R}^p$. Then for each sequence  $\{(u^*_n,\lambda _n)\}\subset \mathbb {R}^n\times \mathbb {R}^p$ such that $u^*_n\rightarrow u^*_0$ and $\lambda _n\rightarrow \lambda _0\in \varLambda$ as $n\rightarrow +\infty$, $\varPi ^f_{K'(\lambda _n)}u^*_n$ converges to $\varPi ^f_{K'(\lambda _0)}u^*_0$.

3 Degree Theory for the Generalized Mixed Quasi-variational Inequality

In this section, we build the degree theory for the GMQVI. To this end, we first give the following approximate continuous selection lemma.

Lemma 3.1

[37] Suppose ${T}:\mathbb {R}^n\rightrightarrows {\mathbb {R}^n}$ is an upper semicontinuous mapping with closed and convex values. Then, for any $\epsilon >0$, there exists a continuous and single-valued mapping $T_{\epsilon }:\mathbb {R}^{n} \rightarrow \mathbb {R}^n$ such that for every $x\in \mathbb {R}^{n}$, there exist $y\in \mathbb {R}^{n}$ and $z\in {T}(y)$ satisfying

$$\begin{aligned} \Vert y-x\Vert< \epsilon \quad \mathrm{and} \quad \Vert z-T_{\epsilon }(x)\Vert <\epsilon . \end{aligned}$$

(8)

Assume that F satisfies the conditions for the existence of an $\epsilon$-approximation. We define $\varPhi _\epsilon :\mathbb {R}^{n}\rightarrow \mathbb {R}^n$ by the following formula

$$\begin{aligned} \varPhi _\epsilon (x)=x-\varPi ^f_{K(x)}(x-f_\epsilon (x)), \end{aligned}$$

where $f_\epsilon$ is an approximate continuous selection of F which satisfies (8).

The following lemma about $\varPhi _\epsilon$ plays an essential role in building the degree theory for the GMQVI.

Lemma 3.2

Let $f: \mathbb {R}^{n}\rightarrow \mathbb {R}$ be a convex function, $K: \mathbb {R}^{n}\rightrightarrows {\mathbb {R}^{n}}$ be a set-valued mapping with nonempty, closed and convex values and $F:\mathbb {R}^{n}\rightrightarrows {\mathbb {R}^{n}}$ be an upper semicontinuous mapping with nonempty, compact and convex values. Besides, suppose that $\varOmega \subset \mathbb {R}^{n}$ is a nonempty, bounded and open set, K is continuous on ${\text {cl}}\varOmega$ and $0\not \in F(\partial \varOmega )+N^f_{K(\partial \varOmega )}(\partial \varOmega )$. Then the following assertions hold:

($a_1$) There exists $\epsilon _1>0$ such that $0\not \in \varPhi _{\epsilon }(\partial \varOmega )$ for all $\epsilon \in ]0, \epsilon _1].$

($a_2$) There exists $\epsilon _2>0$ such that

$$\begin{aligned} d(\varPhi _{\epsilon },\varOmega , 0)=d(\varPhi _{\epsilon '},\varOmega ,0), ~~\forall ~\epsilon ,~\epsilon '\in ]0, \epsilon _2].\end{aligned}$$

Proof

($a_1$) Suppose the conclusion is not true. Then there exists a sequence $\{\epsilon _k\}$ with $\epsilon _k\rightarrow 0^{+}$ and a sequence $\{x_k\}$ with $x_k\in \partial \varOmega$ such that $\varPhi _{\epsilon _k}(x_k)=0$. Due to Lemma 2.1, we know $\varPi ^f_{K(x_k)}(\cdot )$ is single-valued and

$$\begin{aligned} x_k=\varPi ^f_{K(x_k)}(x_k-f_{\epsilon _k}(x_k)).\end{aligned}$$

(9)

In light of compactness of $\partial \varOmega$, we can assume that $x_k\rightarrow {\bar{x}}\in \partial \varOmega$. Since $K:\mathbb {R}^{n}\rightrightarrows \mathbb {R}^{n}$ is continuous on ${\overline{\varOmega }}$ and $x_k\in K(x_k)$, we know ${\bar{x}}\in K({\bar{x}})$. Lemma 3.1 implies that there exist $y_k\in \mathbb {R}^{n}$ and $z_k\in F(y_k)$ such that

$$\begin{aligned} \Vert y_k-x_k\Vert<\epsilon _k\quad \mathrm{and}\quad \Vert z_k-f_{\epsilon _k}(x_k)\Vert <\epsilon _k, \end{aligned}$$

and thus $\{y_k\}$ is bounded and $y_k\rightarrow {\bar{x}}$. Since F is an upper semicontinuous mapping with compact and convex values, Lemma 2.3 implies that ${\text {cl}}(F(\{y_k\}))\subset {\text {cl}}(F({\text {cl}}\{y_k\}))$ is a compact set. By $\{z_k\}\subset {\text {cl}}(F(\{y_k\}))$, without loss of generality, we can assume that $z_k\rightarrow z_0$. Since $\Vert z_k-f_{\epsilon _k}(x_k)\Vert <\epsilon _k$ and $\epsilon _k\rightarrow 0^{+}$, we obtain $f_{\epsilon _k}(x_k)\rightarrow z_0$.

Since F is an upper semicontinuous mapping with closed values, we deduce from Lemma 2.2 and $z_k\in F(y_k)$ that $z_0\in F({\bar{x}})$.

Taking $k\rightarrow \infty$ in (9), Lemma 2.4 implies that ${\bar{x}}=\varPi ^f_{K({\bar{x}})}({\bar{x}}-z_0)$. Now Lemma 2.1 and the definition of $N^f_{K(\cdot )}(\cdot )$ yield

$$\begin{aligned}0\in z_0+N^f_{K({\bar{x}})}({\bar{x}})\subset F({\bar{x}})+N^f_{K({\bar{x}})}({\bar{x}}){,} \end{aligned}$$

which contradicts $0\not \in (F(\partial \varOmega )+N^f_{K(\partial \varOmega )}(\partial \varOmega ))$. The proof of the part ($a_1$) is completed.

($a_2$) Assume that there exist sequences $\{\epsilon _k\}$ and $\{\epsilon '_k\}$ with $0<\epsilon _k<\epsilon ' _k\rightarrow 0$ satisfying

$$\begin{aligned} d(\varPhi _{\epsilon _k},\varOmega , 0)\ne d(\varPhi _{\epsilon '_k},\varOmega ,0). \end{aligned}$$

(10)

Let

$$\begin{aligned}H(t,x)=x-t\varPi ^f_{K(x)}(x-f_{\epsilon _k}(x))-(1-t)\varPi ^f_{K(x)}(x-f_{\epsilon '_k}(x)),~~\forall ~(t,x)\in [0,1]\times {\text {cl}}\varOmega . \end{aligned}$$

This indicates that

$$\begin{aligned}H(0,x)=x-\varPi ^f_{K(x)}(x- f_{\epsilon '_k}(x))=\varPhi _{\epsilon '_k}(x)\end{aligned}$$

and

$$\begin{aligned} H(1,x)=x-\varPi ^f_{K(x)}(x- f_{\epsilon _k}(x))=\varPhi _{\epsilon _k}(x). \end{aligned}$$

If $0\not \in H(t,\partial \varOmega )$ for all $t\in [0,1]$, then, in view of ($a_4$) of Theorem 2.1, we have

$$\begin{aligned} d(H(0,\cdot ),\varOmega , 0)= d(H(1,\cdot ),\varOmega , 0), \end{aligned}$$

which is in contradiction with (10). Hence, for each k, there exists $t_k\in [0,1]$ such that $0\in H(t_k,\partial \varOmega )$. This implies that, for each k, there exists $x_k\in \partial \varOmega$ such that

$$\begin{aligned} x_k= t_k\varPi ^f_{K(x_k)}(x_k-f_{\epsilon _k}(x_k))+(1-t_k)\varPi ^f_{K(x_k)}(x_k-f_{\epsilon '_k}(x_k)). \end{aligned}$$

(11)

By compactness of [0, 1], it is reasonable to suppose that $t_k\rightarrow {\bar{t}}$. Thanks to compactness of $\partial \varOmega$, we assume naturally that $x_k\rightarrow {\bar{x}}\in \partial \varOmega$. Since $K:\mathbb {R}^{n}\rightrightarrows \mathbb {R}^{n}$ is continuous on ${\text {cl}}\varOmega$ and $x_k\in K(x_k)$, we get ${\bar{x}}\in K({\bar{x}})$. Lemma 3.1 implies that there exist $y_k,y'_k\in \mathbb {R}^{n}$; $z_k\in F(y_k)$ and $z'_k\in F(y'_k)$ such that

$$\begin{aligned} \Vert y_k-x_k\Vert< \epsilon _k,~ \Vert z_k-f_{\epsilon _k}(x_k)\Vert<\epsilon _k,~\Vert y'_k-x_k\Vert< \epsilon '_k~\mathrm{and }~ \Vert z'_k-f_{\epsilon '_k}(x_k)\Vert <\epsilon '_k. \end{aligned}$$

Hence $\{y_k\}$ and $\{y'_k\}$ are bounded, and both of them converge to ${\bar{x}}$. Since F is an upper semicontinuous mapping with compact and convex values, it follows from Lemma 2.3 that ${\text {cl}}(F(\{y'_k\}))$ and ${\text {cl}}(F(\{y_k\}))$ are two compact sets. Owing to $\{z_k\}\subset {\text {cl}}(F(\{y_k\}))$ and $\{z'_k\}\subset {\text {cl}}(F(\{y'_k\}))$, without loss of generality, we can suppose that $z_k\rightarrow z_0$ and $z'_k\rightarrow z'_0$. Since $\Vert z_k-f_{\epsilon _k}(x_k)\Vert <\epsilon _k$, $\epsilon _k\rightarrow 0^{+}$, $\Vert z'_k-f_{\epsilon '_k}(x_k)\Vert <\epsilon '_k$ and $\epsilon '_k\rightarrow 0^{+}$, we know $f_{\epsilon _k}(x_k)\rightarrow z_0$ and $f_{\epsilon '_k}(x_k)\rightarrow z'_0$.

Since F is an upper semicontinuous mapping with closed values, $z'_k\in F(y'_k)$ and $z_k\in F(y_k)$, Lemma 2.2 implies that $z'_0\in F({\bar{x}})$ and $z_0\in F({\bar{x}})$.

Letting $k\rightarrow \infty$ in (11), Lemma 2.4 implies that ${\bar{x}}={\bar{t}}\varPi ^f_{K({\bar{x}})}({\bar{x}}-z_0)+(1-{\bar{t}})\varPi ^f_{K({\bar{x}})}({\bar{x}}-z'_0)$, and thus

$$\begin{aligned} {\bar{x}}\in {\bar{t}}\varPi ^f_{K({\bar{x}})}({\bar{x}})-F({\bar{x}}))+(1-{\bar{t}})\varPi ^f_{K({\bar{x}})}({\bar{x}}-F({\bar{x}}))=\varPi ^f_{K({\bar{x}})}({\bar{x}}-F({\bar{x}})). \end{aligned}$$

Therefore $0\in F({\bar{x}})+N^f_{K({\bar{x}})}({\bar{x}})$. Since ${\bar{x}}\in \partial \varOmega$, this is in conflict with $0\not \in F(\partial \varOmega )+N^f_{K(\partial \varOmega )}(\partial \varOmega )$. This completes proof of Lemma 3.2.$\square$

Thanks to Lemma 3.2, we know there exists ${\bar{\epsilon }}>0$ such that $0\not \in \varPhi _\epsilon (\partial \varOmega )$ and $d(\varPhi _\epsilon , \varOmega , 0)=d(\varPhi _{\epsilon '},\varOmega , 0)$ for all $\epsilon ,~\epsilon '\in {]}0,~{\bar{\epsilon }}]$, which allows us to give the following definition.

Definition 3.1

Let $f: \mathbb {R}^{n}\rightarrow \mathbb {R}$ be convex function, $K: \mathbb {R}^{n}\rightrightarrows {\mathbb {R}^{n}}$ be a continuous and set-valued mapping with nonempty, closed and convex values and $F:\mathbb {R}^{n}\rightrightarrows {\mathbb {R}^{n}}$ be an upper semicontinuous mapping with nonempty, compact and convex values. Besides, suppose that $\varOmega \subset \mathbb {R}^{n}$ is a nonempty, bounded and open set, K is continuous on ${\overline{\varOmega }}$ and $0\not \in F(\partial \varOmega )+N^f_{K(\partial \varOmega )}(\partial \varOmega )$. The degree of the GMQVI defined by F and K with respect to $\varOmega$ at 0 is the common value $d(\varPhi _\epsilon ,\varOmega ,0)$ for $\epsilon >0$ sufficiently small and denoted by $d(F+N^f_K, \varOmega , 0)$.

Remark 3.1

(1) Though there have been a large number of papers on the degree theory for set-valued mappings heretofore (see [37,38,39]), we point that to some extent, the degree theory for the generalized mixed quasi-variational inequality in the present paper is different from degree theory for upper semicontinuous and set-valued mappings with compact and convex values. It is established by the single-valued mapping $\varPhi _{\epsilon }$.

(2) If $f(x)=0$ and $K(x)=C~\text {for all } x\in \mathbb {R}^{n}$ with C being a nonempty, closed and convex subset of $\mathbb {R}^{n}$, then Definition 3.1 reduces to Definition 2.1 of [18].

The following theorem gives some properties of the degree for the generalized mixed quasi-variational inequality.

Theorem 3.1

Let $f: \mathbb {R}^{n}\rightarrow \mathbb {R}$ be a convex function, $K: \mathbb {R}^{n}\rightrightarrows {\mathbb {R}^{n}}$ be a set-valued mapping with nonempty, closed and convex values and $F:\mathbb {R}^{n}\rightrightarrows {\mathbb {R}^{n}}$ be an upper semicontinuous mapping with nonempty, compact and convex values. Besides, suppose that $\varOmega \subset \mathbb {R}^{n}$ is a nonempty, bounded and open set, K is continuous on ${\text {cl}}\varOmega$ and $0\not \in F(\partial \varOmega )+N^f_{K(\partial \varOmega )}(\partial \varOmega )$. Then the following assertions hold:

($a_1$) (Normalization) Assume that there exists ${\hat{x}}\in \varOmega \cap (\cap _{x\in \mathbb {R}^{n}} K(x))$ such that $f({\hat{x}})=\mathrm{inf}_{y\in K(x)}f(y)$. If $F=I-{\hat{x}}$ for any $x\in \mathbb {R}^{n}$, then $d(F+N^f_k, \varOmega , 0)=1$.

($a_2$) (Existence) If $d(F+N^f_K,\varOmega , 0)\ne 0$, then there exists ${\bar{x}}\in \varOmega$ such that

$$\begin{aligned} 0\in F({\bar{x}})+N^f_{K({\bar{x}})}({\bar{x}}). \end{aligned}$$

($a_3$) (Additivity) If $\varOmega _1$, $\varOmega _2$ are disjoint open subsets of $\varOmega$ such that $0\not \in (F(\cdot )+N^f_{K(\cdot )}(\cdot ))({\text {cl}}\varOmega {\setminus }(\varOmega _1\cup \varOmega _2))$, then

$$\begin{aligned} d(F+N^f_K,\varOmega ,0)=d(F+N^f_K,\varOmega _1,0)+d(F+N^f_K,\varOmega _2,0). \end{aligned}$$

($a_4$) (Homotopy) For ${\tilde{i}}\in \{1,2\}$, $F_{{\tilde{i}}}:\mathbb {R}^{n}\rightrightarrows \mathbb {R}^{n}$ is an upper semicontinuous and set-valued mapping with nonempty, compact and convex values. Moreover, $0\not \in tF_1(\partial \varOmega )+(1-t)F_2(\partial \varOmega )+N^f_{K(\partial \varOmega )}(\partial \varOmega )$ for all $t\in [0,1]$. Then

$$\begin{aligned} d(F_1+N^f_K,\varOmega ,0)=d(F_2+N^f_K,\varOmega ,0). \end{aligned}$$

($a_5$) (Excision) If $D\subset {\overline{\varOmega }}$ is a closed set such that $0\not \in F(D)+N^f_{K(D)}(D)$, then

$$\begin{aligned} d(F+N^f_K,\varOmega ,0)=d(F+N^f_K,\varOmega {\setminus } D,0). \end{aligned}$$

($a_6$) If ${\tilde{f}}:\mathbb {R}^{n}\rightarrow \mathbb {R}^n$ is a single-valued and continuous mapping such that ${\tilde{f}}(x)\in F(x)$ for all $x\in \mathbb {R}^{n}$, then $d(F+N^f_K, \varOmega , 0)=d(\varPhi , \varOmega , 0)$, where $\varPhi (x)=x-\varPi ^f_{K(x)}(x-{\tilde{f}}(x))$.

Proof

($a_1$) According to Definition 3.1, there exists ${\tilde{\epsilon }}>0$ such that

$$\begin{aligned}d(I+N^f_K,\varOmega , 0)=d(\varPhi _{{\tilde{\epsilon }}}, \varOmega , 0),\end{aligned}$$

where

$$\begin{aligned}\varPhi _{{\tilde{\epsilon }}}(x)=x-\varPi ^f_{K(x)}(x-(x-{\hat{x}}))=x-\varPi ^f_{K(x)}({\hat{x}}), ~~\forall ~x\in \mathbb {R}^{n}.\end{aligned}$$

Since $f({\hat{x}})=\mathrm{inf}_{y\in K(x)}f(x)$, we know that $\varPhi _{{\tilde{\epsilon }}}(x)=x-{\hat{x}}$ for all $x\in \mathbb {R}^{n}.$

Now, it follows from Theorem 2.1 that

$$\begin{aligned}d(I+N^f_K, \varOmega , 0)=d(\varPhi _{{\tilde{\epsilon }}}, \varOmega , 0)=d(I-{\hat{x}}, \varOmega , 0)=d(I, \varOmega , {\hat{x}})=1.\end{aligned}$$

($a_2$) Owing to Definition 3.1, there exists ${\bar{\epsilon }}>0$ such that $d(F+N^f_K, \varOmega , 0)=d(\varPhi _\epsilon , \varOmega , 0)$ for all $\epsilon \in (0,{\bar{\epsilon }}]$. Suppose that $\{\epsilon _k\}$ is a sequence satisfying $\epsilon _k\rightarrow 0^{+}$. Thus, $d(\varPhi _{\epsilon _k}, \varOmega ,0)\ne 0$ for k sufficiently large. From ($a_2$) of Theorem 2.1, it follows that there exists $x_k\in \varOmega$ such that $\varPhi _{\epsilon _k}(x_k)=0$. Therefore,

$$\begin{aligned} x_k=\varPi ^f_{K(x_k)}(x_k-f_{\epsilon _k}(x_k)). \end{aligned}$$

(12)

By similar arguments as the proof of ($a_1$) in Lemma 3.2, we know $x_k\rightarrow {\bar{x}}$, $f_{\epsilon _k}(x_k)\rightarrow z_0$ and $z_k\rightarrow z_0\in F({\bar{x}})$. Letting $k\rightarrow \infty$ in (12), we obtain ${\bar{x}}=\varPi ^f_{K({\bar{x}})}({\bar{x}}-z_0)$ . In light of Lemma 2.1, the definition of $N^f_{K(\cdot )}(\cdot )$ implies that

$$\begin{aligned} 0\in z_0+N^f_{K({\bar{x}})}({\bar{x}})\subset F({\bar{x}})+N^f_{K({\bar{x}})}({\bar{x}}). \end{aligned}$$

Since $0\not \in (F(\partial \varOmega )+N^f_{K(\partial \varOmega )})(\partial \varOmega )$, we have ${\bar{x}}\in K({\bar{x}})\cap \varOmega$.

($a_3$) We show that there exists ${\bar{\epsilon }}>0$ such that $0\not \in \varPhi _{\epsilon }({\text {cl}}\varOmega {\setminus }(\varOmega _1\cup \varOmega _2))$ for all $\epsilon \in {]}0,{\bar{\epsilon }}]$. Indeed, if the conclusion is not true, then there are sequences $\{\epsilon _k\}$ and $\{x_k\}$ with $\epsilon _k\rightarrow 0^{+}$ and $x_k\in {\text {cl}}\varOmega {\setminus }(\varOmega _1\cup \varOmega _2)$ such that

$$\begin{aligned} x_k=\varPi ^f_{K(x_k)}(x_k-f_{\epsilon _k}(x_k)). \end{aligned}$$

(13)

Using similar arguments as the proof of ($a_1$) in Lemma 3.2, we know that $x_k,~y_k\rightarrow {\bar{x}}$, $f_{\epsilon _k}(x_k)\rightarrow z_0$ and $z_k\rightarrow z_0\in F({\bar{x}})$. Taking $k\rightarrow \infty$ in (13), we have

$$\begin{aligned}{\bar{x}}=\varPi ^f_{K({\bar{x}})}({\bar{x}}-z_0).\end{aligned}$$

From Lemma 2.1 and the definition of $N^f_{K(\cdot )}(\cdot )$, we deduce that

$$\begin{aligned} 0\in z_0+N^f_{K({\bar{x}})}({\bar{x}})\subset F({\bar{x}})+N^f_{K({\bar{x}})}({\bar{x}}),\hbox { for some } {\bar{x}}\in {\text {cl}}\varOmega {\setminus }(\varOmega _1\cup \varOmega _2), \end{aligned}$$

which contradicts $0\not \in (F(\cdot )+N^f_{K(\cdot )}(\cdot ))({\text {cl}}\varOmega {\setminus }(\varOmega _1\cup \varOmega _2))$. Thus, $0\not \in \varPhi _\epsilon ({\text {cl}}\varOmega {\setminus }(\varOmega _1\cup \varOmega _2))$ for all $\epsilon \in {]}0,{\bar{\epsilon }}]$. From ($a_3$) of Theorem 2.1, it follows that

$$\begin{aligned} d(\varPhi _{\epsilon }, \varOmega , 0)=d(\varPhi _{\epsilon }, \varOmega _1, 0)+d(\varPhi _{\epsilon }, \varOmega _2, 0). \end{aligned}$$

This implies that

$$\begin{aligned} d(F+N^f_K,\varOmega ,0)=d(F+N^f_K,\varOmega _1,0)+d(F+N^f_K,\varOmega _2,0). \end{aligned}$$

($a_4$) For ${\tilde{i}}\in \{1,2\}$, suppose that $f^{{\tilde{i}}}_{\epsilon }:\mathbb {R}^{n}\rightarrow \mathbb {R}^n$ is an approximate selection of $F_{{\tilde{i}}}$ satisfying the conclusion of Lemma 3.1. Set

$$\begin{aligned} H_{\epsilon }(t,x)=x-\varPi ^f_{K(x)}[x-(tf^1_{\epsilon }(x)+(1-t)f^2_{\epsilon }(x))]. \end{aligned}$$

We assert that there is ${\bar{\epsilon }} > 0$ such that $0\not \in H_{\epsilon }(t,\partial \varOmega )$ for all $t\in [0,1]$ and $\epsilon \in [0,{\bar{\epsilon }}]$.

In fact, if the claim is not true, then there exist a sequence $\{t_k\}$ with $t_k\in [0,1]$ and a sequence $\{\epsilon _k\}$ with $\epsilon _k\rightarrow 0^{+}$ such that $0\in H_{\epsilon _k}(t_k,\partial \varOmega )$. This means that, for each k, there exists $x_k\in \partial \varOmega$ such that

$$\begin{aligned} x_k=\varPi ^f_{K(x_k)}[x_k-(t_kf^1_{\epsilon _k}(x_k)+(1-t_k)f^2_{\epsilon _k}(x_k))]. \end{aligned}$$

(14)

By compactness of [0, 1], assume that $t_k\rightarrow t_0$. By the same methods as the proof of ($a_2$) in Lemma 3.2, it is easy to show that $x_k\rightarrow {\bar{x}}\in \partial \varOmega$, $f^1_{\epsilon _k}(x_k)\rightarrow z_1$ for some $z_1\in F_1({\bar{x}})$ and $f^2_{\epsilon _k}(x_k)\rightarrow z_2$ for some $z_2\in F_2({\bar{x}})$. Letting $k\rightarrow \infty$ in (14), yields ${\bar{x}}=\varPi ^f_{K({\bar{x}})}[{\bar{x}}-(t_0z_1+(1-t_0)z_2)]$. From Lemma 2.1 and the definition of $N^f_{K(\cdot )}(\cdot )$ we can infer that

$$\begin{aligned} 0\in (t_0z_1+(1-t_0)z_2)+N^f_{K({\bar{x}})}({\bar{x}})\subset t_0F_1({\bar{x}})+(1-t_0)F_2({\bar{x}})+N^f_{K({\bar{x}})}({\bar{x}}), \end{aligned}$$

which contradicts $0\not \in tF_1(\partial \varOmega )+(1-t)F_2(\partial \varOmega )+N^f_{K(\partial \varOmega )}(\partial \varOmega )$ for all $t\in [0,1]$. Thus, the assertion is true and ($a_4$) of Theorem 2.1 implies that

$$\begin{aligned} d(H_{\epsilon }(0,\cdot ), \varOmega , 0)=d(H_{\epsilon }(1,\cdot ), \varOmega , 0)\hbox { for all }\epsilon \in ]0,~{\bar{\epsilon }}]. \end{aligned}$$

Therefore

$$\begin{aligned} d(F_1+N^f_K,\varOmega ,0)=d(F_2+N^f_K,\varOmega ,0). \end{aligned}$$

($a_5$) Using the similar arguments as the proof of ($a_1$) in Lemma 3.2, we can prove that there exists ${\bar{\epsilon }}>0$ such that $0\not \in \varPhi _\epsilon (D)$ for all $\epsilon \in (0, {\bar{\epsilon }}]$. Applying ($a_5$) of Theorem 2.1 to $\varPhi _\epsilon$, we know that the property ($a_5$) holds.

($a_6$) Taking $f_\epsilon ={\tilde{f}}$ for all $\epsilon > 0$, we can obtain the desired conclusion.

This completes the proof of Theorem 3.1. $\square$

Obviously, if $K(x)\equiv C$ with C being a nonempty, closed and convex subset in $\mathbb {R}^{n}$, then the condition ($a_1$) can be easily satisfied in Theorem 3.1. Next we give a simple example with K(x) depending on x such that the condition ($a_1$) holds.

For any $x\in \mathbb {R}^{n}$, let $K(x)=\{u\in \mathbb {R}^{n}: \Vert u\Vert \le \Vert x\Vert +2\}$, $f(x)=\Vert x\Vert ^2$ and $\varOmega =\{u\in \mathbb {R}^{n}: \Vert x\Vert \le r ,(r>0)\}$. It is easy to check that ${\hat{x}}=0\in \varOmega \cap (\cap _{x\in \mathbb {R}^{n}} K(x))$ satisfies $f({\hat{x}})=\mathrm{inf}_{y\in K(x)}f(y)$ for any $x\in \mathbb {R}^{n}$.

In order to study the stability of isolated solutions to the GMQVI, we first introduce the following definition.

Definition 3.2

A vector $x_0\in K$ is called an isolated solution of the GMQVI, if there exists a neighborhood $V_1\subset \mathbb {R}^{n}$ of $x_0$ such that $x_0$ is the unique solution of the GMQVI in ${\text {cl}}V_1$.

Theorem 3.2

Assume that $f: \mathbb {R}^{n}\rightarrow \mathbb {R}$ is a convex function, $K: \mathbb {R}^{n}\rightrightarrows {\mathbb {R}^{n}}$ is a continuous and set-valued mapping with nonempty, closed and convex values and $F:\mathbb {R}^{n}\rightrightarrows {\mathbb {R}^{n}}$ is an upper semicontinuous mapping with nonempty, compact and convex values. Suppose that $x_0$ is an isolated solution of the GMQVI and ${\mathfrak {U}}$ is the collection of all open and bounded neighborhoods $V_1$ of $x_0$ such that ${\text {cl}}V_1$ does not contain another solution of the GMQVI. Then

$$\begin{aligned} d(F+N^f_K,V_2,0)=d(F+N^f_K,V_3,0) \end{aligned}$$

for all $V_2$, $V_3\in {\mathfrak {U}}$. The common value $d(F+N^f_K,V_1,0)$ for all $V_1\in {\mathfrak {U}}$ is called the index of $F+N^f_K$ and denoted by $i(F+N^f_K, x_0, 0)$.

Proof

We use the same arguments as in [18] for the proof below. Taking any ${\tilde{V}}\in {\mathfrak {U}}$, we have

$$\begin{aligned}0\not \in F(\partial {\tilde{V}})+N^f_{K(\partial {\tilde{V}})}(\partial {\tilde{V}}).\end{aligned}$$

Therefore $d(F+N^f_K, {\tilde{V}}, 0)$ is well defined. We now assume that $V_2$, $V_3\in {\mathfrak {U}}$. Put $V_1=V_2\cup V_3\in {\mathfrak {U}}$ and $D={\text {cl}}V_2\cap V^c_3$, where $V^c_3=\mathbb {R}^{n}{\setminus } V_3$. We infer that D is a bound and closed set in ${\text {cl}}V_1$ and $0\not \in F(D)+N^f_{K(D)}(D)$. By ($a_5$) in Theorem 3.1, we have

$$\begin{aligned} d(F+N^f_K,V_1,0)=d(F+N^f_K,V_1{\setminus } D,0)=d(F+N^f_K,V_3,0). \end{aligned}$$

Using similar arguments for $D={\text {cl}}V_3\cap V^c_2$, we get that

$$\begin{aligned}d(F+N^f_K,V_1,0)=d(F+N^f_K,V_1{\setminus } D,0)=d(F+N^f_K,V_2,0),\end{aligned}$$

and thus

$$\begin{aligned}d(F+N^f_K,V_2,0)=d(F+N^f_K,V_3,0), \quad \forall ~V_2,V_3\in {\mathfrak {U}}.\end{aligned}$$

This completes the proof of Theorem 3.2. $\square$

Remark 3.2

Theorems 3.1, 3.2 and Definition 3.2 extend the corresponding theorems and definition in [18] for the GVI to the cases of the GMQVI. In addition, if $f(x)=0,$ for all $x\in \mathbb {R}^{n}$ and F is a continuous and single-valued mapping, then thanks to Definition 3.1 and ($a_2$) of Theorem 3.1, it is easy to obtain Theorem 2.8.3 of [2].

4 Applications

In this section, we employ the results presented in Sect. 3 to prove the existence and stability of solutions for the GMQVI.

The following theorem is an extension of Corollary 2.8.4 of [2].

Theorem 4.1

Let $f: \mathbb {R}^{n}\rightarrow \mathbb {R}$ be convex, $K:\mathbb {R}^{n}\rightrightarrows \mathbb {R}^{n}$ be a set-valued mapping with nonempty, closed and convex values and $F:\mathbb {R}^{n}\rightrightarrows \mathbb {R}^{n}$ be an upper semicontinuous mapping with nonempty, compact and convex values. Suppose that there exist a bounded and open set $\varOmega '\subset \mathbb {R}^{n}$ and a vector ${\hat{x}}\in \varOmega '$ satisfying

• (a) the mapping K is continuous on ${\text {cl}}\varOmega '$;

• (b) the vector ${\hat{x}}$ belongs to $\cap _{x\in {\text {cl}}\varOmega '}K(x)$;

• (c) for any $x\in {\text {cl}}\varOmega '$, $f({\hat{x}})=\mathrm{inf}_{y\in K(x)}f(y)$;

• (d) the following holds

  $$\begin{aligned}\{x\in K(x): \inf _{x^*\in F(x)}\langle x^*, x-{\hat{x}}\rangle +f(x)-f({\hat{x}})<0\}\cap \partial \varOmega '=\emptyset .\end{aligned}$$

  Then the GMQVI has a solution.

Proof

Assume that the GMQVI has no solution. Hence $0\not \in F(\partial \varOmega ')+N^f_{K (\partial \varOmega ')}(\partial \varOmega ')$ and thus the degree $d(F+N^f_K,\varOmega ', 0)$ is well defined. Put

$$\begin{aligned} H(t,x)=tF(x)+(1-t)(x-{\hat{x}})+N^f_{K(x)}(x),~~\forall ~(t,x)\in [0,1]\times \overline{\varOmega '}, \end{aligned}$$

which implies that $H(0,x)=x-{\hat{x}}+N^f_{K(x)}(x)$ and $H(1,x)=F(x)+N^f_{K(x)}(x)$.

We show that $0\not \in H(t, \partial \varOmega ')$ for all $t\in [0,1].$ Indeed, if $0\in H(0,\partial \varOmega ')$, then there exists $x_0\in \partial \varOmega '$ such that ${\hat{x}}-x_0\in N^f_{K(x_0)}(x_0)$. Owing to definition of $N^f_{K(x_0)}(x_0)$, we have

$$\begin{aligned} \langle {\hat{x}}-x_0, y-x_0\rangle +f(x_0)-f(y)\le 0,~~\forall ~y\in K(x_0).\end{aligned}$$

(15)

Letting $y={\hat{x}}$ in (15), we obtain

$$\begin{aligned} \langle {\hat{x}}-x_0, {\hat{x}}-x_0\rangle +f(x_0)-f({\hat{x}})\le 0. \end{aligned}$$

(16)

From $f({\hat{x}})=\mathrm{inf}_{x\in K(x_0)}f(x)$, it follows that $x_0={\hat{x}}$. This contradicts the fact that $x_0\in \partial \varOmega '$ and ${\hat{x}}\in \varOmega '$. Hence $0\not \in H(0,\partial \varOmega ')$.

If $0\in H(1,\partial \varOmega ')$ then $0\in F(\partial \varOmega ')+N^f_{K (\partial \varOmega ')}(\partial \varOmega ')$, which is conflict with $0\not \in F(\partial \varOmega ')+N^f_{K (\partial \varOmega ')}(\partial \varOmega ')$.

If there exist $t_0\in (0,1)$ and $x'_0\in \partial \varOmega '$ such that $0\in H(t_0,x'_0)$, then

$$\begin{aligned} 0\in t_0F(x'_0)+(1-t_0)(x'_0-{\hat{x}})+N^f_{K(x'_0)}(x'_0). \end{aligned}$$

(17)

Due to (17), there exists $x^*_0\in F(x'_0)$ satisfying

$$\begin{aligned}-t_0x^*_0-(1-t_0)(x'_0-{\hat{x}})\in N^f_{K(x'_0)}(x'_0).\end{aligned}$$

Thanks to the definition of $N^f_{K(x'_0)}(x'_0)$, we infer that

$$\begin{aligned} \langle t_0x^*_0+(1-t_0)(x'_0-{\hat{x}}), y-x'_0\rangle +f(y)-f(x'_0)\ge 0,~~\forall ~y\in K(x'_0), \end{aligned}$$

and thus

$$\begin{aligned} \langle x^*_0, y-x'_0\rangle +\frac{1}{t_0}(f(y)-f(x'_0))\ge \frac{ 1-t_0}{t_0}\langle x'_0-{\hat{x}},x'_0-y\rangle ~~\forall ~y\in K(x'_0). \end{aligned}$$

(18)

Letting $y={\hat{x}}$ in (18), $t_0\in (0,1)$ and ${\hat{x}}\ne x'_0$, yield

$$\begin{aligned} \langle x^*_0, {\hat{x}}-x'_0\rangle +\frac{1}{t_0}(f({\hat{x}})-f(x'_0))\ge \frac{ 1-t_0}{t_0}\langle x'_0-{\hat{x}},x'_0-{\hat{x}}\rangle >0. \end{aligned}$$

Since $t_0\in (0,1)$ and $f({\hat{x}})=\mathrm{inf}_{x\in K(x'_0)}f(x)$, we know that

$$\begin{aligned}f(x'_0)-f({\hat{x}})\le \frac{1}{t_0}(f(x'_0)-f({\hat{x}})),\end{aligned}$$

and thus

$$\begin{aligned} \mathrm{inf}_{x^* \in F(x'_0)}\langle x^*,x'_0-{\hat{x}}\rangle +f(x'_0)-f({\hat{x}})\le \langle x^*_0, x'_0-{\hat{x}}\rangle +\frac{1}{t_0}(f(x'_0)-f({\hat{x}}))<0. \end{aligned}$$

This conflicts with the condition (d) and thus the claim is true. By ($a_1$) and ($a_4$) of Theorem 3.1, we infer that $d(F+N^f_K, \varOmega ', 0)=d(I-{\hat{x}}+N^f_K,\varOmega ', 0)=1$. In view of ($a_2$) of Theorem 3.1, there exists ${\bar{x}}_0\in \varOmega '\cap K$ such that $0\in F({\bar{x}}_0)+N^f_{K({\bar{x}}_0)}({\bar{x}}_0)$. Lemma 2.1 and the definition of $N^f_{K(\cdot )}(\cdot )$ imply that ${\bar{x}}_0$ is a solution of GMQVI. This completes the proof of Theorem 4.1. $\square$

Remark 4.1

If $f(x)=0,$ for all $x\in \mathbb {R}^{n}$ and F is a continuous single-valued mapping, then Theorem 4.1 reduces to Corollary 2.8.4 of [2].

From Theorem 4.1, it is easy to get the following corollary.

Corollary 4.1

Assume that ${\bar{K}}\subset \mathbb {R}^{n}$ is a nonempty, closed and convex set. Let ${\tilde{F}}: {\bar{K}}\rightrightarrows {\mathbb {R}^{n}}$ be an upper semicontinuous mapping with nonempty, compact and convex values. If there exists a vector ${\hat{x}}\in K$ such that the set

$$\begin{aligned} L_{<}({\hat{x}}):=\{x\in {\bar{K}}: \mathrm{inf}_{x^* \in {\tilde{F}}(x)}\langle x^*,x-{\hat{x}}\rangle < 0\} \end{aligned}$$

is bounded (possibly empty), then the problem (4) has a solution.

Remark 4.2

If $L_{\le }({\hat{x}}):=\{x\in {\bar{K}}: \mathrm{inf}_{x^* \in {\hat{F}}(x)}\langle x^*,x-{\hat{x}}\rangle \le 0\}$ is bounded, then $L_{<}({\hat{x}})$ is bounded. Therefore, Corollary 4.2 extends and improves Theorem 3.1 in [18].

In the following example, we apply Theorem 4.1 to a generalized Nash game problem.

Example 4.1

In Example 2.1, we introduced a generalized Nash game problem in which the objective functions are nonsmooth. We continue this example by showing the associated generalized mixed quasi-variational inequality is solvable. Set $n=3$, $n_1=n_2=n_3=1$, for all $x=(x_1,x_2, x_3)\in \mathbb {R}^{3}$, $X=\sum _{i=1}^{3}x_i$ and

$$\begin{aligned} p(X)={\left\{ \begin{array}{ll} &{}a_1X-b_1,~~~-\infty<X\le \beta _1,\\ &{}a_2X-b_2,~~~\beta _1\le X<+\infty , \end{array}\right. } \end{aligned}$$

where $a_i$ and $b_i$ $(i=1,2)$ are positive numbers such that $a_1<a_2$ and $a_1 \beta _1-b_1=a_2\beta _1-b_2$. For $i\in \{1,2,3\}$, let $f_i(x_i)=|x_i|$, $K_i(x_{-i})=\{u_i:|u_i|\le |x_i|\}$ and $g_i(x)=x_ip(X)$. According to the definition of subdifferential of convex functions, we have

$$\begin{aligned} \partial _{x_i}g_i(x)={\left\{ \begin{array}{ll} &{}p(X)+x_i\{a_1\},~~~-\infty<X<\beta _1,\\ &{}p(X)+x_i[a_1,a_2],~~~X= \beta _1,\\ &{}p(X)+x_i\{a_2\},~~~\beta _1<X<-\infty .\\ \end{array}\right. } \end{aligned}$$

Obviously, for each $x\in \mathbb {R}^{3}$, $F(x)=\prod _{i=1}^3 \partial _{x_i} g_i(x)$ , $f(x)=\mathrm{max}\{4a_2,4\}(\sum _{i=1}^{3}f_i(x_i))$ and $K(x)=\prod _{i=1}^3 K_i(x_{-i}).$ According to Proposition 3 on page 122 of [40], it is easy to know that F is an upper semicontinuous mapping with nonempty, compact and convex values. In addition, it is obvious that, f is convex and K is a continuous mapping with nonempty, closed and convex values. Taking ${\hat{x}}=0$, we know that the conditions (b) and (c) of Theorem 4.1 hold. Since $\liminf _{\Vert x\Vert \rightarrow +\infty }[\inf _{x^*\in F(x),x\in K(x)}\langle x^*, x\rangle +f(x)]=+\infty$, there exists a bounded and open set $\varOmega '\subset \mathbb {R}^{3}$ such that ${\hat{x}}\in \varOmega '$ and the condition (d) of Theorem 4.1 holds. The existence of an equilibrium is a consequence of the application of Theorem 4.1.

In the rest of the section, we give a stability result of solutions to the GMQVI. Assume that M and $\varLambda$ are nonempty subsets of $\mathbb {R}^k$ and $\mathbb {R}^m$, respectively. Let $K:\varLambda \times \mathbb {R}^{n} \rightrightarrows {\mathbb {R}^{n}}$ be a set-valued mapping with nonempty, closed and convex values, $F:M \times \mathbb {R}^{n} \times \mathbb {R}^{n} \rightrightarrows {\mathbb {R}^{n}}$ be a set-valued mapping and $f: \mathbb {R}^{n}\rightarrow \mathbb {R}$ be a convex mapping. We introduce the following parametric generalized mixed quasi-variational inequality: for each given $(\lambda , \mu )\in \varLambda \times M$, find $x\in K(\lambda ,x)$ and $x^{*}\in F(\mu , x)$ such that

$$\begin{aligned} \langle x^*, y-x\rangle +f(y)-f(x)\ge 0,\quad \text {for all }y\in K(\lambda ,x). \end{aligned}$$

(19)

By Lemma 2.1 and the definition of the generalized f-normal cone operator, we know that $x\in K(\lambda ,x)$ and $x^{*}\in F(\mu , x)$ is a solution of the problem (19) if and only if $x\in K(\lambda ,x)$ satisfies

$$\begin{aligned} 0\in F(\mu , x)+N^f_{K(\lambda ,x)}(x), \end{aligned}$$

(20)

where $N^f_{K(\lambda ,x)}(x)$ is the value at x of the generalized f-normal cone operator associated with the set $K(\lambda ,x)$ and $(\mu , \lambda )$ are parameters.

Let $K_1:\varLambda \rightrightarrows \mathbb {R}^{n}$ be nonempty, closed and convex values. If $f=0$, $K(\lambda ,x )=K_1(\lambda )$ for all $(\lambda , x )\in \varLambda \times \mathbb {R}^{n}$, then the problem (20) reduces to the following problem: find $x\in K(\lambda )$ satisfying

$$\begin{aligned} 0\in F(\mu , x)+N_{K_1(\lambda )}(x), \end{aligned}$$

(21)

The problem (21) is called the parametric generalized variational inequalities, studied by Kien et al. [18].

We denote by $S(\mu ,\lambda )$ the solution set of the problems (19) with parameters $(\mu , \lambda )$. Our main aim is now to investigate the behavior of $S(\mu ,\lambda )$ when $(\mu ,\lambda )$ varies around $(\mu _0,\lambda _0)$.

Lemma 4.1

[41] Let X be an arbitrary metric space, A be a closed subset of X, L be a locally convex linear space and ${\mathcal {G}}: A\rightarrow L$ be a continuous mapping. Then there exists a continuous extension ${\mathcal {F}} : X\rightarrow L$ of ${\mathcal {G}}$, i.e, ${\mathcal {F}}(a)={\mathcal {G}}(a)$ for all $a\in A$. Furthermore, ${\mathcal {F}}(X) \subset \mathrm{con ({\mathcal {G}}(A))}$, where $\mathrm{con ({\mathcal {G}}(A))}$ is convex hull of ${\mathcal {G}}(A)$.

Theorem 4.2

Suppose $x_0 \in S(\mu _0,\lambda _0)$ is an isolated solution. Let $X_0$, $\varLambda _0$ and $M_0$ be neighborhoods of $x_0$, $\lambda _0$ and $\mu _0$, respectively. Let $K:\varLambda _0\times \mathbb {R}^{n}\rightrightarrows {\mathbb {R}^{n}}$ be a set-valued mapping with nonempty, closed and convex values, $F:M \times \mathbb {R}^{n} \rightrightarrows {\mathbb {R}^{n}}$ be a set-valued mapping and $f: \mathbb {R}^{n}\rightarrow \mathbb {R}$ be convex. Assume that the following conditions are satisfied:

• (i) $F(\cdot ,\cdot )$ is a lower semicontinuous mapping with nonempty, closed and convex values on $M_0\times X_0$ and $F(\mu _0,\cdot )$ is an upper semicontinuous mapping with nonempty, compact and convex values on $X_0$;

• (ii) $K(\cdot ,\cdot )$ is continuous on $B(\lambda _0,\beta _1)\times X_0,$ where $B(\lambda _0,\beta _1):=\{{\bar{\lambda }}\in \varLambda _0:\Vert {\bar{\lambda }}-\lambda _0\Vert < \beta _1\}$ for some $\beta _1>0$;

• (iii) the index

  $$\begin{aligned}i( F(\mu _0, \cdot )+N^f_{K(\lambda _0,\cdot )\cap {\text {cl}}B(x_0,r_1)}, x_0, 0)\ne 0,\end{aligned}$$

  where ${\text {cl}}B(x_0,r_1)=\{y\in X_0:\Vert y-x_0\Vert \le r_1\}$.

Then there exist a neighborhood $M_1$ of $\mu _0$, a neighborhood $\varLambda _1$ of $\lambda _0$ and an open bounded neighborhood $Q_0$ of $x_0$ such that the following assertions are fulfilled:

• (a) The solution map ${\hat{S}}: M_1\times \varLambda _1 \rightrightarrows {\mathbb {R}^{n}}$ of the problem (19) defined by ${\hat{S}}(\mu ,\lambda )=S(\mu ,\lambda )\cap Q_0$ is nonempty for all $(\mu ,\lambda )\in M_1\times \varLambda _1$ and ${\hat{S}}(\mu _0,\lambda _0)=\{x_0\}$.

• (b) ${\hat{S}}$ is lower semicontinuous at $(\mu _0,\lambda _0)$.

Proof

Thanks to the condition (i) and the continuous selection theorem (see [42]), we infer that there exists a continuous mapping $T_1:M_0\times X_0\rightarrow \mathbb {R}^n$ such that $T_1(\mu ,x)\in F(\mu ,x),~~\forall ~(\mu ,x)\in M_0\times X_0$. According to Lemma 4.1, we can assume that $T_1$ is continuous on $M\times \mathbb {R}^{n}$. For any $(\mu ,\lambda ,x)\in M_0\times \varLambda _0\times X_0$, consider the mapping

$$\begin{aligned} \varPhi _{}(\mu ,\lambda ,x)=x-\varPi ^f_{K(\lambda ,x)\cap {\text {cl}}B(x_0,r_1)}(x-T_1(\mu ,x)). \end{aligned}$$

We now show that $\varPhi$ is continuous on $M_0\times B(\lambda _0, \beta _1)\times {\text {cl}} B(x_0, r_1)$.

In fact, for any given point $({\tilde{\mu }}_0, {\tilde{\lambda }}_0, {\tilde{x}}_0)\in M_0\times B(\lambda _0, \beta _1)\times {\text {cl}} B(x_0, r_1)$ and for any sequence $({\tilde{\mu }}_n, {\tilde{\lambda }}_n, {\tilde{x}}_n)$ in $M_0\times B(\lambda _0, \beta _1)\times {\text {cl}}B(x_0, r_1)$ with $({\tilde{\mu }}_n, {\tilde{\lambda }}_n, {\tilde{x}}_n)\rightarrow ({\tilde{\mu }}_0, {\tilde{\lambda }}_0, {\tilde{x}}_0)$ as $n\rightarrow +\infty$, the continuity of $T_1$ implies that $T_1({\tilde{\mu }}_n,{\tilde{x}}_n)\rightarrow T_1({\tilde{\mu }}_0,{\tilde{x}}_0)$ as $n\rightarrow +\infty$. From the condition (ii) and Definition 2.4, $K(\cdot ,\cdot )\cap {\text {cl}}B(x_0, r_1)$ is continuous on $X_0\times B(\lambda _0,\beta _1)$. By Lemma 2.4, we infer that

$$\begin{aligned}&\varPi ^f_{K({\tilde{\lambda }}_n,{\tilde{x}}_n)\cap {\text {cl}}B(x_0,r_1)}({\tilde{x}}_n-T_1({\tilde{\mu }}_n,{\tilde{x}}_n))\rightarrow \varPi ^f_{K({\tilde{\lambda }}_0,{\tilde{x}}_0)\cap {\text {cl}}B(x_0,r_1)}({\tilde{x}}_0-T_1({\tilde{\mu }}_0,{\tilde{x}}_0)),\\&\quad ~~\mathrm{~as~~ }n\rightarrow +\infty . \end{aligned}$$

Thus, $\varPhi$ is continuous on $M_0\times B(\lambda _0, \beta _1)\times {\text {cl}}B(x_0, r_1)$.

Since $x_0 \in S(\mu _0,\lambda _0)$ is an isolated solution, there exists an open and bounded neighborhood $Q_0\subset X_0$ of $x_0$ such that $x_0$ is the unique solution in ${\text {cl}} Q_0$ of the generalized equation

$$\begin{aligned} 0\in F(\mu _0,x)+N^f_{K(\lambda _0,x)}(x). \end{aligned}$$

Since $x_0$ belongs to the interior of ${\text {cl}}B(x_0,r_1)$, it is also the unique solution in ${\text {cl}}Q_0$ of the following generalized equation

$$\begin{aligned} 0\in F(\mu _0,x)+N^f_{K(\lambda _0,x)\cap {\text {cl}}B(x_0,r_1)}(x). \end{aligned}$$

From the condition (iii) and Theorem 3.2, it follows that

$$\begin{aligned} d( F(\mu _0,\cdot )+N^f_{K(\lambda _0,\cdot )\cap {\text {cl}}B(x_0,r_1)},Q_0, 0)=i(F(\mu _0,\cdot )+N^f_{K(\lambda _0,\cdot )\cap {\text {cl}}B(x_0,r_1)},x_0, 0)\ne 0. \end{aligned}$$

Owing to ($a_6$) of Theorem 3.1, we know that

$$\begin{aligned} d(\varPhi (\mu _0,\lambda _0,\cdot ),Q_0, 0)=d(F(\mu _0,\cdot )+N^f_{K(\lambda _0)\cap {\text {cl}}B(x_0,r_1)},Q_0, 0)\ne 0. \end{aligned}$$

Note that any solution of the equation $\varPhi (\mu _0,\lambda _0,x)=0$ is also a solution of the following problem

$$\begin{aligned}0\in F(\mu _0,x)+N^f_{K(\lambda _0,\cdot )\cap {\text {cl}}B(x_0,r_1)}(x).\end{aligned}$$

Hence, $x_0$ is a unique solution of $\varPhi (\mu _0,\lambda _0,x)=0$ in ${\text {cl}}Q_0$.

We now can use similar arguments as the proof of Theorem 4.1 in [23] (see also Theorem 3.2 of [18]) to complete the proof of the theorem. This completes the proof of Theorem 4.2. $\square$

By Theorem 4.2, it is easy to obtain the stability of solutions of the parametric generalized variational inequalities (21). Below, we give an example to illustrate Theorem 4.2.

Example 4.2

Let $X_0=\{(x_1,x_2):x_1^2+x_2^2\le 100 \}\subset \mathbb {R}^{2}$, $\varLambda =\varLambda _0=[9, 80]\subset R$ and $M=M_0=[-0.5,0.5]$. Let $f(x)=0$ for all $x\in \mathbb {R}^{2}$ and $F:M\times \mathbb {R}^{2}\rightrightarrows \mathbb {R}^{2}$ be defined by

$$\begin{aligned} F(\mu ,x)= & {} \{(u_1, u_2):(x_2-u_1)^2+(x_2^2+\mu x_1-u_2)^2\le 1\},\\&~~\forall ~\mu \in M,~x=(x_1,x_2)\in \mathbb {R}^{2}. \end{aligned}$$

Define the mapping $K:\varLambda \times \mathbb {R}^{2}\rightrightarrows \mathbb {R}^{2}$ by the following formula

$$\begin{aligned} K(\lambda ,x)= & {} \{(v_1,v_2): 2v_1-v_2\le 4x_2+1000,~v_1+v_2=2\lambda x_1\},\\&~~\forall ~\lambda \in \varLambda ,~x=(x_1,x_2)\in \mathbb {R}^{2}. \end{aligned}$$

For each $\epsilon >0$, let $f_\epsilon (\mu ,x)=(x_2, x_2^2+\mu x_1),~~\forall ~\mu \in M,~x=(x_1,x_2)\in \mathbb {R}^{2}$. Thus $f_\epsilon (\mu ,x)\in F(\mu ,x)$ and $\varPhi (\mu ,\lambda ,x)=x-\varPi _{K(\lambda ,x)}(x-f_\epsilon (\mu ,x))$ for all $(\mu ,x)\in M\times \mathbb {R}^{2}.$ Furthermore,

$$\begin{aligned} \varPhi (\mu ,\lambda ,(x_1,x_2))&=(x_1, x_2)-\varPi _{K(\lambda ,x)}[(x_1,x_2)-f_\epsilon (\mu ,(x_1,x_2))]\\&=(x_1, x_2)-\varPi _{K(\lambda ,x)}[(x_1-x_2,x_2-x_2^2-\mu x_1)]\\&=(x_1, x_2)-(\frac{1}{2}x_2^2-x_2+\frac{1}{2}x_1(2\lambda +\mu +1),x_2\\&\quad -\frac{1}{2}x_2^2+\frac{1}{2}x_1(2\lambda -\mu -1))\\&=(x_2-\frac{1}{2}x_2^2+x_1(\frac{1}{2}-\lambda -\frac{\mu }{2}),\frac{1}{2}x_2^2-x_1(\lambda -\frac{1}{2}-\frac{\mu }{2})). \end{aligned}$$

Obviously, $\varPhi (\mu ,\lambda ,(x_1,x_2))=0$ if and only if

$$\begin{aligned} \left\{ \begin{aligned}&x_2-\frac{1}{2}x_2^2+x_1\left( \frac{1}{2}-\lambda -\frac{\mu }{2}\right) =0,\\&\frac{1}{2}x_2^2-x_1(\lambda -\frac{1}{2}-\frac{\mu }{2})=0.\\ \end{aligned} \right. \end{aligned}$$

(22)

The above system of equations admits two solutions $(x_1^1,x_2^1)=(0,0)$ and $(x_1^2,x_2^2)=(\frac{2\lambda -\mu -1}{(2\lambda -1)^2}, \frac{2\lambda -\mu -1}{2\lambda -1})$. Taking $(\mu _0,\lambda _0)=(0,10)$, we infer that $x_0=(0,0)$ and $x_0'=(\frac{1}{19},1)$ are the solutions of the problem (20) at $(\mu _0,\lambda _0)$.

In addition, set $B(\lambda _0,\beta _1)=\{\lambda \in \varLambda _0:\Vert \lambda -10\Vert < 1\}$. Since the mappings F and K are continuous, it is easy to see that the conditions (i) and (ii) of Theorem 4.2 are fulfilled.

Set $r_1=0.4$ and $V=\{(x_1,x_2):x_1^2+x_2^2\le 0.16 \}$. Since

$$\begin{aligned}J_\varPhi (\mu _0, \lambda _0,(x_1,x_2))=\left| \begin{array}{cccc} \frac{1}{2}-\lambda _0-\frac{\mu _0}{2}&{} 1-x_2 \\ -(\lambda _0-\frac{1}{2}-\frac{\mu _0}{2}) &{} x_2 \end{array}\right| =-19x_2+\frac{19}{2},\end{aligned}$$

we know the index $i( F(\mu _0, \cdot )+N^f_{K(\lambda _0,\cdot )\cap {\text {cl}}B(x_0,0.4)}, x_0, 0)=d (\varPhi (\mu _0, \lambda _0,\cdot ), V,0)=1$.

Choosing $M_1=M_0$, $\varLambda _1=B(\lambda _0,\beta _1)$ and $Q_0=V$, we know that ${\hat{S}}(\mu _0,\lambda _0)=S(\mu _0,\lambda _0)\cap Q_0=\{x_0\}$ and ${\hat{S}}(\mu ,\lambda )=S(\mu ,\lambda )\cap Q_0$ is nonempty for all $(\mu ,\lambda )\in M_1\times \varLambda _1$ and lower semicontinuous at $(\mu _0,\lambda _0)$.