On composite vector variational-like inequalities and vector optimization problems

Abstract

In this paper, we introduce a class of (weak) composite vector variational-like inequality problems and establish its relationship with composite vector optimization problem. We also prove the relation of a vector critical point in composite vector optimization problem with its weak efficient point, under the assumption of composite pseudo invexity. Using KKM Lemma, we derive result for existence of solutions of composite vector variational-like inequality problem. Furthermore, we define a gap function for the composite vector variational-like inequality problem and finally, as an application, we study a system of composite vector optimization problems and system of vector variational-like inequality problems, whose solutions imply the solution of Nash equilibrium problem. Examples are provided to illustrate the derived results.

1 Introduction

Let us consider the following scalar convex composite programming problem:

$$\begin{aligned} \begin{array}{lr} \mathbf{(P)}&{}\min ~F(x)\\ &{}\mathrm{subject\,to}\quad x\in R^n,\\ \end{array} \end{aligned}$$

where $F(x)=h(f(x))$ with $h:R^m\mapsto R$ convex and $f:R^n\mapsto R^m$ continuously differentiable. This model has received a great deal of attention in the literature, since it offers a unified framework within which many of the traditional problem occurring in mathematical programming can be studied, for example to study the convergence behavior of various algorithms and Lagrangian conditions in constrained and unconstrained optimization. Rockafellar (1988) pointed out that most common problems of optimization that arise in practice can be reformulated using convexly composite functions.

However, convexity does not appear as a natural property for various functions in the mathematical modelling of real world systems. Hence, there has been several extensions and generalizations for classical convexity. Hanson (1981) generalized convex functions to introduce the concept of invexity, which was a significant milestone as it inspired a great deal of subsequent work which has greatly expanded the role and applications of invexity in optimization theory. Considering the wide applications of composite functions and limitations of convexity, Askar and Tiwari (2009) discussed first-order optimality conditions and duality results for composite multiobjective optimization problems. Recently, Padhan and Nahak (2015) established second order duality results for an invex composite optimization problem.

Vector variational inequalities, which serve as an efficient tool to investigate vector optimization problems, was introduced by Giannessi (1980). The proper use of a problem’s structure can lead to very efficient optimization methods. Considerably, there has been extensive generalization of vector variational inequalities, see for instance (Chen and Huang 2012; Jabarootian and Zafarani 2008; Jahn 2004; Oveisiha and Zafarani 2012; Ruiz-Garzón-Lizana et al. 2004; Verma 1998). Ruiz-Garzón-Lizana et al. (2004) established the relationships between vector variational-like inequality problems and vector optimization problems, under the assumption of generalized invexity. Recently, a new class of exponential form of vector variational-like inequality problems was introduced and studied by Jayswal et al. (2014) under the frame work of (p, r)-invexity.

The above facts serve to establish the importance of introducing the concept of composite vector variational-like inequality problem to study invex composite optimization problems, which covers the class of convex optimization problems as a special case. It is well known that an equilibrium problem, researched by various authors (Ansari et al. 2002; Ansari 2012; Blum and Oettli 1994; Huang et al. 2003; Mehta and Chaudhary 2014; Sun and Li 2013) is a unified model of several problems, namely, variational inequality problem, complementarity problem, fixed point problem, saddle point problem, Nash equilibrium problem, etc. And, gap function is a powerful concept for transforming variational inequalities into optimization problems. Consequently, we study these problems in accordance with composite vector optimization problem and composite vector variational-like problem.

Our paper is organized as follows: In Sect. 2, we recall some definitions and lemmas, which will be applicable in the sequel of the paper. In Sect. 3, the relationship between a composite vector variational-like inequality problem and a composite vector optimization problem is established. In Sect. 4, we prove an existence result for solutions of composite vector variational-like inequality problem. A gap function for composite vector variational-like inequality problem is defined in Sect. 5. In Sect. 6, we introduce the system of composite vector optimization problems and system of vector variational-like inequality problems, obtain relation of these problems with Nash equilibrium problem. Lastly in Sect. 7, we conclude the paper.

2 Notations and preliminaries

Let $R^n$ denote the n-dimensional Euclidean space. Let $x, y \in R^n$, we use the following convention for equalities and inequalities throughout this paper.

• (a) $x\le y\Leftrightarrow x_{i}\le y_{i},\;i=1,\ldots ,n$, with strict inequality holding for at least one i;

• (b) $x\leqq y\Leftrightarrow x_{i}\le y_{i},\;i=1,\ldots ,n$;

• (c) $x= y\Leftrightarrow x_{i}= y_{i},\;i= 1,\ldots ,n$;

• (d) $x< y\Leftrightarrow x_{i}< y_{i},\;i= 1,\ldots ,n$.

Now, we extend the notion of composite invex functions defined in Padhan and Nahak (2015) to composite invex vector and composite psuedo invex vector functions. Throughout this paper, we define $X\subset R^n$ and $Y\subset R^m$. Let $F:X\mapsto Y$ be a vector valued function.

Definition 2.1

A differentiable vector function $g:Y \mapsto R^p$ is said to be composite invex at a point $F(u)\in Y$ with respect to $\eta :Y \times Y\mapsto R^{m}$, if the inequality

$$\begin{aligned} g(F(x))- g(F(u))\geqq \left\langle \nabla g(F(u)), \eta (F(x), F(u))\right\rangle ,\quad \forall \,F(x)\in Y, \end{aligned}$$

holds.

The following example shows that there exists a function which is composite invex but not invex with respect to a given function $\eta$.

Example 2.1

Let $X=[2,3]$ and $Y= R$. Consider the functions $F:X \mapsto Y , g:Y \mapsto R^2$ and $\eta :Y \times Y \mapsto R$ defined as

$$\begin{aligned} F(x)=-x,\quad g(x)=\begin{pmatrix} x^2\\ 2x^2\\ \end{pmatrix}\quad \text {and}\quad \eta (F(x), F(u))= 10\sin (F(x))\sin (F(u)) \end{aligned}$$

respectively. Obviously, we have $\nabla g(x)=\begin{pmatrix} 2x\\ 4x\\ \end{pmatrix}$. Also,

$$\begin{aligned} g(F(x))= g(-x)= \begin{pmatrix} x^2\\ 2x^2\\ \end{pmatrix}\quad \text {and}\quad \nabla g(F(x))=\nabla g(-x)= \begin{pmatrix} -2x\\ -4x\\ \end{pmatrix}. \end{aligned}$$

For $u=2$, we have

$$\begin{aligned} g(F(x))- g(F(u))- \left\langle \nabla g(F(u)), \eta (F(x), F(u))\right\rangle&= \begin{pmatrix} x^2- u^2+ 20u\sin x\sin u\\ 2x^2- 2u^2+40u\sin x\sin u\\ \end{pmatrix}\\&=\begin{pmatrix} x^2- 4+ 40 \sin x \sin 2\\ 2x^2- 8+ 80 \sin x \sin 2\\ \end{pmatrix}\\&\ge 0,\quad \text {for all}\;x\in X. \end{aligned}$$

Therefore, the function g is composite invex function at $F(2)=~-2 \in Y$ with respect to $\eta$. However, g is not an invex function at $u=2,$ with respect to $\eta$, since

$$\begin{aligned} g(x)- g(u)- \left\langle \nabla g(u), \eta (x, u)\right\rangle&= \begin{pmatrix} x^2- u^2- 20u \sin x \sin u\\ 2x^2- 2u^2- 40u \sin x \sin u \end{pmatrix}\\&=\begin{pmatrix} x^2- 4- 40 \sin x \sin 2\\ 2x^2- 8-80 \sin x \sin 2 \end{pmatrix}\\&< 0,\quad \text {for all}\;x\in X. \end{aligned}$$

Definition 2.2

A differentiable vector function $g:Y \mapsto R^p$ is said to be composite strictly invex at a point $F(u)\in Y$ with respect to $\eta :Y \times Y\mapsto R^{m}$, if the inequality

$$\begin{aligned} g(F(x))- g(F(u)) > \left\langle \nabla g(F(u)), \eta (F(x), F(u))\right\rangle ,\;F(x)\ne F(u)\quad \text {and}\quad \forall \;F(x)\in Y, \end{aligned}$$

holds.

Definition 2.3

A differentiable vector function $g:Y \mapsto R^p$ is said to be composite psuedo invex at a point $F(u)\in Y$ with respect to $\eta :Y \times Y \mapsto R^{m}$, if the inequality

$$\begin{aligned} g(F(x))- g(F(u)) < 0 \Rightarrow \left\langle \nabla g(F(u)), \eta (F(x), F(u))\right\rangle < 0 ,\quad \forall \;F(x)\in Y, \end{aligned}$$

holds.

Now, we introduce the composite vector variational-like inequality problem as follows. Let $h :Y\mapsto R^{p \times m}$ be a matrix valued function.

(CVVLIP) A composite vector variational-like inequality problem is to find a point $F(u)\in Y$ such that there exists no $F(x)\in Y,$ satisfying

$$\begin{aligned} \left\langle h(F(u)), \eta (F(x), F(u))\right\rangle \le 0. \end{aligned}$$

Remark 2.1

If $F(x)=x$, then the above (CVVLIP) reduces to vector variational-like inequality problem (VVLIP) defined by Ruiz-Garzón-Lizana et al. (2004).

(WCVVLIP) A weak composite vector variational-like inequality problem is to find a point $F(u)\in Y$ such that there exists no $F(x)\in Y,$ satisfying

$$\begin{aligned} \left\langle h(F(u)),\eta (F(x), F(u))\right\rangle < 0. \end{aligned}$$

Remark 2.2

From the above two definitions it is clear that (CVVLIP) $\Rightarrow$ (WCVVLIP).

The following example shows that there exists a solution for (CVVLIP), but the (VVLIP) given by Ruiz-Garzón-Lizana et al. (2004) is not solvable.

Example 2.2

Let $X=[1, 2]$ and $Y= R$. Consider the functions $F:X \mapsto Y$, $h:Y \mapsto R^{2 \times 1}$, and $\eta :Y \times Y\mapsto R$ defined by

$$\begin{aligned} F(u)=-e^{u},\;h(F(u))=\begin{pmatrix} -F(u)-1\\ -F(u)-1/2\\ \end{pmatrix}\quad \text {and}\quad \eta (F(x), F(u))=F(x)F(u), \end{aligned}$$

respectively. Now, for $u=1,$ we have

$$\begin{aligned} \left\langle h(F(u)),\eta (F(x),F(u))\right\rangle&= \begin{pmatrix} -F(u)- 1\\ -F(u)- 1/2\\ \end{pmatrix}\;F(x)F(u)\\&=\begin{pmatrix} e^{u}-1\\ e^{u}-1/2\\ \end{pmatrix}~e^{x}e^{u}\\&=\begin{pmatrix} e^{2u+x}-e^{u+x}\\ e^{2u+x}-(1/2)e^{u+x}\\ \end{pmatrix}\\&=\begin{pmatrix} e^{2+x}-e^{1+x}\\ e^{2+x}-(1/2)e^{1+x}\\ \end{pmatrix}. \end{aligned}$$

From above, it is clear that for $u=1$, there exists no $F(x)\in Y, \text { where }x\in X$ such that

$$\begin{aligned} \left\langle h(F(u)),\eta (F(x),F(u))\right\rangle =\begin{pmatrix} e^{2+x}-e^{1+x}\\ e^{2+x}-(1/2)e^{1+x} \end{pmatrix}\le 0, \end{aligned}$$

which means that $F(1)\in Y$ is a solution of (CVVLIP). However, the following vector variational-like inequality problem is not solvable, since

$$\begin{aligned} h(u)\eta (x,u)=\begin{pmatrix} -u^2 x-u x\\ -u^2 x-(1/2)u x \end{pmatrix} \le 0,\quad \forall \,x,u\in X. \end{aligned}$$

Consider the following composite vector optimization problem:

$$\begin{aligned} \begin{array}{lr} \mathbf{(CVOP)}&{}\min ~f(x)\\ &{}\mathrm{such\,that}\;x\in X,\\ \end{array} \end{aligned}$$

where $f(x)=g(F(x)),\,g:Y\mapsto R^p$ is a differentiable vector function and $F:X\mapsto Y$ is a vector valued function.

Definition 2.4

Given a function $g:Y\mapsto R^p$, a point $F(u)\in Y$ is said to be an efficient (Pareto) point, if there exists no $F(x)\in Y$ such that $g(F(x))-g(F(u))\le 0$.

Definition 2.5

Given a function $g:Y\mapsto R^p$, a point $F(u)\in Y$ is said to be a weak efficient (Pareto) point, if there exists no $F(x)\in Y$ such that $g(F(x))-g(F((u))<0$.

Definition 2.6

A feasible solution $F(u)\in Y$ is said to be a vector critical point for (CVOP), if there exists a vector $\mu \in R^p$ with $\mu \ge 0$ such that $\mu ^T\nabla g(F(u))=0.$

In order to identify the solutions of (WCVVLIP) with the weak efficient points of (CVOP), we define the following definition on the lines of Definition 2.1 defined by Mohan and Neogy (1995).

Definition 2.7

Let $F(u)\in Y$. A set Y is said to be invex at a point $F(u)\in Y$ with respect to $\eta :Y\times Y \mapsto R^{m}$, if for each $F(x)\in Y$, $0\le t\le 1$,

$$\begin{aligned} F(u)+t\eta (F(x),F(u))\in Y. \end{aligned}$$

The set Y is said to be an invex set with respect to $\eta$, if Y is invex at each $F(u)\in Y$. Obviously, if $\eta (F(x),F(u))=F(x)-F(u),$ then Y is said to be a convex set.

3 Relationship between composite vector variational-like inequality problem and composite vector optimization problem

The following theorems demonstrate the relationship between (CVVLIP) and (CVOP).

Theorem 3.1

Let $g:Y\mapsto R^p$ be a differentiable function. Assume that

• (i) $h=\nabla g$ and g is composite invex function with respect to $\eta$ at a point F(u),

• (ii) F(u) solves (CVVLIP) with respect to same $\eta$.

Then, F(u) is an efficient point of (CVOP).

Proof

Suppose F(u) is not an efficient point of (CVOP), then there exists a point $F(x)\in Y$ such that

$$\begin{aligned} g(F(x))-g(F(u)) \le 0. \end{aligned}$$

(1)

Now, by using composite invexity of g with respect to $\eta$ at the point F(u), we have

$$\begin{aligned} g(F(x))-g(F(u))\geqq \left\langle \nabla g(F(u)),\eta (F(x), F(u))\right\rangle ,\quad \forall \,F(x)\in \,Y, \end{aligned}$$

which by using (1), it follows that, there exists $F(x)\in Y$ such that

$$\begin{aligned} \left\langle \nabla g(F(u)), \eta (F(x), F(u))\right\rangle \le 0. \end{aligned}$$

From the assumption, $h=\nabla g,$ the above inequality yields that, there exists $F(x)\in Y$ such that

$$\begin{aligned} \left\langle h(F(u)), \eta (F(x), F(u))\right\rangle \le 0, \end{aligned}$$

which contradicts our assumption that F(u) solves (CVVLIP). Hence the proof. $\square$

Theorem 3.2

Let $g:Y\mapsto R^p$ be a differentiable function. Assume that

• (i) $h=\nabla g$ and $-g$ is composite strictly invex function with respect to $\eta$ at a point F(u),

• (ii) F(u) is a weak efficient point of (CVOP).

Then, F(u) solves (CVVLIP).

Proof

Suppose F(u) is a weak efficient point of (CVOP) but it does not solve the (CVVLIP), then there exists a point $F(x)\in Y$ such that

$$\begin{aligned} \left\langle h(F(u)), \eta (F(x),F(u)) \right\rangle \le 0. \end{aligned}$$

From the assumption, $h=\nabla g,$ the above inequality gives

$$\begin{aligned} \left\langle \nabla g(F(u)),\eta (F(x),F(u))\right\rangle \le 0. \end{aligned}$$

(2)

Now, by using composite strict invexity of $-g$ with respect to $\eta$ at the point F(u), we have

$$\begin{aligned} g(F(x))-g(F(u))<\left\langle \nabla g(F(u)),\eta (F(x),F(u))\right\rangle ,\quad \forall \,F(x)\in Y\quad \text {and}\quad F(x)\not = F(u). \end{aligned}$$

(3)

On combining inequalities (2) and (3), we can write, there exists $F(x)\in Y$ such that

$$\begin{aligned} g(F(x))-g(F(u))<0, \end{aligned}$$

which contradicts our assumption that F(u) is a weak efficient point of (CVOP). Hence the proof. $\square$

The following example shows that under the assumptions of Theorem 3.2, if there exists a weak efficient point of(CVOP) then it is also a solution of (CVVLIP).

Example 3.1

Let $X= [2,3]$ and $Y= R$. Consider the functions $F:X \mapsto Y$, $g:Y\mapsto R^2$ and $\eta :Y\times Y\mapsto R$ defined by

$$\begin{aligned} F(x)=-x,\;g(x)= \begin{pmatrix} e^{x}\\ 2e^{x}\\ \end{pmatrix}\quad \text {and}\quad \eta (F(x),F(u))=100\cos (F(x)) \sin (F(u)), \end{aligned}$$

respectively. Then obviously, we have

$$\begin{aligned} \nabla g(x)=\begin{pmatrix} e^{x}\\ 2e^{x}\\ \end{pmatrix},\;\nabla g(F(x))=\nabla g(-x)= \begin{pmatrix} e^{-x}\\ 2e^{-x}\\ \end{pmatrix},\;g(F(x))=g(-x)=\begin{pmatrix} e^{-x}\\ 2e^{-x}\\ \end{pmatrix}. \end{aligned}$$

It can be easily verified that $-g$ is composite strictly invex at $F(3)\in Y$ with respect to $\eta$. Now, for $u= 3$ we have

$$\begin{aligned} g(F(x))- g(F(u))= \begin{pmatrix} e^{-x}- e^{-u}\\ 2e^{-x}- 2e^{-u}\\ \end{pmatrix} =\begin{pmatrix} e^{-x}- e^{-3}\\ 2e^{-x}-2e^{-3}\\ \end{pmatrix}. \end{aligned}$$

From above, it is clear that for $u=3,$ there exists no $F(x)\in Y,$ where $x\in X$ such that

$$\begin{aligned} g(F(x))-g(F(u))=\begin{pmatrix} e^{-x}-e^{-3}\\ 2e^{-x}-2e^{-3}\\ \end{pmatrix}< 0, \end{aligned}$$

which shows that $F(3)\in Y$ is a weak efficient point of (CVOP). Further, for $u= 3$, we have

$$\begin{aligned} \left\langle \nabla g(F(u)), \eta (F(x),F(u))\right\rangle&=\begin{pmatrix} e^{-u}\\ 2e^{-u}\\ \end{pmatrix}100\cos (-x)\sin (-u)\\&=\begin{pmatrix} -100e^{-u}\cos x\sin u\\ -200e^{-u}\cos x\sin u\\ \end{pmatrix}\\&=\begin{pmatrix} -100e^{-3}\cos x\sin {3}\\ -200e^{-3}\cos x\sin {3}\\ \end{pmatrix}. \end{aligned}$$

From this, it is clear that for $u=3,$ there exists no $F(x)\in Y,$ where $x\in X,$ such that

$$\begin{aligned} \left\langle \nabla g(F(u)), \eta (F(x),F(u))\right\rangle =\begin{pmatrix} -100e^{-3}\cos x\sin {3}\\ -200e^{-3}\cos x\sin {3}\\ \end{pmatrix}\le 0. \end{aligned}$$

Thus, $F(3)\in Y$ is a solution of (CVVLIP).

Remark 3.1

It is clear that the function $-g$ considered in the above example is not strictly invex with respect to $\eta$ at $u=3$.

Corollary 3.1

Let $g:Y\mapsto R^p$ be a differentiable vector function. Assume that

• (i) $h=\nabla g$ and $-g$ is composite strictly invex function with respect to $\eta$ at a point F(u),

• (ii) F(u) is an efficient point of (CVOP).

Then, F(u) solves (CVVLIP).

Proof

Since, every efficient point is weak efficient point, hence the proof follows from Theorem 3.2. $\square$

The following theorem enables us, under which conditions we might identify solutions of (WCVVLIP) with the weak efficient points of (CVOP).

Theorem 3.3

Assume that, Y is an invex set and $h=\nabla g$. If F(u) is a weak efficient point of (CVOP), then it solves (WCVVLIP).

Conversely, if g is a composite pseudo invex function with respect to $\eta$ at a point F(u) and F(u) solves (WCVVLIP) with respect to the same $\eta$, then it is a weak efficient point of (CVOP).

Proof

Let F(u) be a weak efficient point of (CVOP). Since Y is an invex set, then there exists no $F(x)\in Y$ such that

$$\begin{aligned} g(F(u)+t\eta (F(x),F(u)))-g(F(u))<0,\quad 0<t<1. \end{aligned}$$

Dividing the above inequality by t, taking the limit $t\rightarrow 0,$ and using Taylor’s series, we obtain

$$\begin{aligned} 0&>\underset{t\rightarrow 0}{\lim }\frac{g(F(u)+t\eta (F(x),F(u)))-g(F(u))}{t}\\&={\underset{t\rightarrow 0}{\lim }\frac{g(F(u))+\left\langle \nabla g(F(u)), t\eta (F(x),F(u))\right\rangle + \frac{\left\langle \nabla ^{2} g(F(u)), t^2\eta ^2(F(x),F(u))\right\rangle }{2!}+\cdots \cdots -g(F(u))}{t}.}\\&\Rightarrow \underset{t\rightarrow 0}{\lim } \left( \left\langle \nabla g(F(u)),\eta (F(x),F(u)) \right\rangle +\frac{\left\langle \nabla ^2 g(F(u)),t\eta ^2(F(x),F(u)) \right\rangle }{2!}+\cdots \cdots \right) <0,\\&\quad i.e.,\quad \left\langle \nabla g(F(u)),\eta (F(x),F(u)) \right\rangle <0. \end{aligned}$$

From the assumption, $h=\nabla g$, above inequality implies that, there exists no $F(x)\in Y$ such that

$$\begin{aligned} \left\langle h(F(u)), \eta (F(x),F(u))\right\rangle <0, \end{aligned}$$

which shows that F(u) solves (WCVVLIP).

Conversely, suppose that F(u) is not a weak efficient point of (CVOP). Then there exists $F(x)\in Y$ such that

$$\begin{aligned} g(F(x))-g(F(u))<0. \end{aligned}$$

Since, g is composite pseudo invex vector function at F(u) with respect to $\eta$, it follows that, there exists $F(x)\in Y$ such that

$$\begin{aligned} \left\langle \nabla g(F(u)),\eta (F(x),F(u))\right\rangle <0. \end{aligned}$$

From the assumption $h=\nabla g$, the above inequality gives, there exists $F(x)\in Y$ such that

$$\begin{aligned} \left\langle h(F(u)),\eta (F(x),F(u))\right\rangle <0, \end{aligned}$$

which contradicts our assumption that F(u) solves (WCVVLIP). Hence the theorem. $\square$

The following theorem enables us, under which condition a weak efficient point will be an efficient point of (CVOP).

Theorem 3.4

Let Y be invex set and $g:Y \mapsto R^p$ be a differentiable function. Assume that

• (i) $h=\nabla g$ and g is a composite strictly invex function with respect to $\eta$ at a point F(u),

• (ii) F(u) is a weak efficient point of (CVOP).

Then, F(u) is an efficient point of (CVOP).

Proof

Suppose, F(u) is a weak efficient point of (CVOP) but it is not an efficient point of (CVOP). Then, there exists $F(x)\in Y$ such that

$$\begin{aligned} g(F(x))-g(F(u))\le 0. \end{aligned}$$

(4)

By composite strict invexity of the vector function g with respect to $\eta$ at the point F(u), we have

$$\begin{aligned} g(F(x))-g(F(u))>\left\langle \nabla g(F(u)),\eta (F(x),F(u))\right\rangle ,\;\forall \,F(x)\in Y\;\text {and}\;F(x)\not =F(u). \end{aligned}$$

(5)

On combining inequalities (4) and (5), we get, there exists $F(x)\in Y$ such that

$$\begin{aligned} \left\langle \nabla g(F(u)),\eta (F(x),F(u))\right\rangle <0. \end{aligned}$$

By using $h=\nabla g$, above inequality implies that, there exists $F(x)\in Y$ such that

$$\begin{aligned} \left\langle h(F(u)),\eta (F(x),F(u))\right\rangle <0. \end{aligned}$$

From the above inequality, it is clear that F(u) does not solve the (WCVVLIP). By Theorem 3.3, it follows that F(u) is not a weak efficient point of (CVOP), which contradicts our assumption. Hence the proof. $\square$

The following theorem enables us to establish the relationship between a vector critical point and a weak efficient point of (CVOP).

Theorem 3.5

Let $g:Y \mapsto R^p$ be a differentiable function. Assume that

• (i) g is a composite pseudo invex function with respect to $\eta$ at a point F(u),

• (ii) F(u) is a vector critical point of (CVOP).

Then, F(u) is a weak efficient point of (CVOP).

Proof

Suppose F(u) is a vector critical point of (CVOP) but not its weak efficient point. Then, there exists $F(x)\in Y$ such that

$$\begin{aligned} g(F(x))-g(F(u))< 0. \end{aligned}$$

By using composite pseudo invexity of g with respect to $\eta$ at the point F(u), we have

$$\begin{aligned} \left\langle \nabla g(F(u)), \eta (F(x),F(u))\right\rangle < 0. \end{aligned}$$

Now, by applying Gordan’s Theorem, the above inequality implies that the system

$$\begin{aligned} \mu ^T \nabla g(F(u))= 0 \end{aligned}$$

has no solution for $\mu \in R^p$ with $\mu \ge 0$, which contradicts our assumption that F(u) is a vector critical point of (CVOP). Hence the theorem. $\square$

4 Existence of solutions of the composite vector variational-like inequality problem

In this section, we derive the result for existence of solution of (CVVLIP) using composite proper quasi-monotonicity and KKM Lemma. We need the following definitions and lemmas to prove our existence result. Throughout this section let $Y\subset R^m$ be a real topological vector space.

Definition 4.1

Let $g:Y\mapsto R^p$ be a differentiable function, then $\nabla g$ is said to be composite properly quasi-monotone with respect to $\eta :Y \times Y \mapsto R^{m}$, if for any $\{F(x_1),F(x_2),\ldots , F(x_n)\}\in Y$ and $F(y)\in conv\{F(x_1),F(x_2),\ldots ,F(x_n)\}$, there exists $i\in \{1,2,\ldots ,n\}$ such that

$$\begin{aligned} \left\langle \nabla g(F(y)), \eta (F(x_i), F(y))\right\rangle \ge 0, \end{aligned}$$

where $conv\{F(x_1),F(x_2),\ldots ,F(x_n)\}$ denotes the convex hull of $\{F(x_1),F(x_2),\ldots ,F(x_n)\}$.

Definition 4.2

(Ansari 2012) Let K be an arbitrary subset of a real vector space E. A set-valued map $\Gamma :K\mapsto 2^E$ is said to be a Knaster–Kuratowski–Mazurkiewicz map (KKM map), if for every finite subset $A = \{x_1,\ldots ,x_n\}\subseteq ~K$, it holds

$$\begin{aligned} conv (A)\subset \bigcup ^{n}_{i=1} \Gamma (x_i). \end{aligned}$$

Lemma 4.1

(Ansari 2012) Let $\Gamma :K\mapsto 2^E$ be a set-valued map defined on a subset K of a real topological vector space E and verifying:

• (i) $\Gamma$ is a KKM map,

• (ii) all values of $\Gamma$ are closed and convex.

Then, the family $\{\Gamma (x)\}_{x\in X}$ has the finite intersection property.

If in addition, at least one value $\Gamma (x_0)$ for some $x_0\in X$, is compact, then $\bigcap _{x\in X}\Gamma (x)\ne \phi$.

Theorem 4.1

Let $g:Y\mapsto R^p$ be a differentiable function and $\nabla g$ be composite properly quasi-monotone with respect to $\eta :Y\times Y \mapsto R^{m}$. Assume that

• (i) $h= \nabla g$,

• (ii) the set-valued map $\Gamma :Y \mapsto 2^Y$, defined by

  $$\begin{aligned} \Gamma (F(x))= \{F(y)\in Y:\left\langle \nabla g (F(y)),\eta (F(x),F(y))\right\rangle \not \le 0,~\forall ~ y\in X\} \end{aligned}$$

  is closed and convex valued,

• (iii) for at least one $F(x_0)\in Y$, $\Gamma (F(x_0))$ is compact.

Then, (CVVLIP) has a solution.

Proof

Firstly, we shall show that $\Gamma$ is a KKM mapping. We proceed by contradiction and suppose that $\Gamma$ is not a KKM mapping. Then there exist $\{F(x_1),F(x_2),\ldots ,F(x_n)\}\subset Y$, $\lambda _i\ge 0$, $i=1,2,\ldots ,n$ with $\sum _{i=1}^{n}{\lambda _i}=1$ such that

$$\begin{aligned} F(y)=\sum _{i=1}^{n}{\lambda _{i} F(x_i)}\notin \bigcup ^{n}_{i=1}{\Gamma (F(x_i))}. \end{aligned}$$

Then, for any $i=1,2,\ldots ,n,$

$$\begin{aligned} \left\langle \nabla g(F(y)),\eta (F(x_i),F(y)\right\rangle \le 0, \end{aligned}$$

which contradicts our assumption that $\nabla g$ is composite properly quasi-monotone with respect to $\eta$. Hence, $\Gamma$ is a KKM mapping.

Now, by condition (ii) and (iii), we have $\Gamma (F(x))$ is KKM mapping with closed and convex value and for some $F(x_0)\in Y$, at least one value $\Gamma (F(x_0))$ is compact. Then, by Lemma 4.1, we have

$$\begin{aligned} \bigcap _{F(x)\in Y}{\Gamma (F(x))\ne \phi }. \end{aligned}$$

Thus, it follows that there exists $F(u)\in Y$, for which

$$\begin{aligned} \left\langle \nabla g(F(u)),\eta (F(x),F(u))\right\rangle \not \le 0,\quad \forall \,F(x)\in Y. \end{aligned}$$

By condition (i), we can rewrite the above inequality as, there exists no $F(x)\in Y$ such that

$$\begin{aligned} \left\langle h(F(u)),\eta (F(x), F(u))\right\rangle \le 0. \end{aligned}$$

Hence, $F(u)\in Y$ is a solution of (CVVLIP). $\square$

The following example shows that under the assumptions of Theorem 4.1, (CVVLIP) has a solution.

Example 4.1

Let $Y=R$ and $X=[4,5]$. Consider the functions $F:X\mapsto Y$, $g:Y\mapsto R^2$ and $\eta :Y\times Y \mapsto R$ defined by

$$\begin{aligned} F(x)=-x,\;g(x)= \begin{pmatrix} x^3\\ x^5\\ \end{pmatrix}\quad \text {and}\quad \eta (F(x),F(y))=\sin (F(x))e^{F(y)}, \end{aligned}$$

respectively. Obviously, $\nabla g(x)=\begin{pmatrix} 3x^2\\ 5x^4\\ \end{pmatrix},~\nabla g(F(x))=\nabla g(-x)= \begin{pmatrix} 3x^2\\ 5x^4\\ \end{pmatrix}.$

Since, for any $\{F(x_1),F(x_2),\ldots ,F(x_n)\}\in Y$ and $F(y)\in conv\{F(x_1), F(x_2),\ldots ,F(x_n)\}$ there exists $i\in \{1,2,\ldots ,n\}$ such that

$$\begin{aligned} \left\langle \nabla g(F(y)),\eta (F(x_i),F(y))\right\rangle =\begin{pmatrix} 3y^2\\ 5y^4\\ \end{pmatrix}\sin (-x_i)e^{-y}\ge 0. \end{aligned}$$

Therefore, $\nabla g$ is composite properly quasi-monotone with respect to $\eta$. Now, we have the mapping $\Gamma :Y\mapsto 2^Y$ defined by

$$\begin{aligned} \Gamma (F(x))&=\{F(y)\in Y:\left\langle \nabla g (F(y)),\eta (F(x),F(y))\right\rangle \not \le 0,~\forall ~ y\in X\}\\&=\left\{ F(y)\in Y:\begin{pmatrix} 3y^2\\ 5y^4\\ \end{pmatrix} \sin (-x)e^{-y}\not \le 0,~\forall ~ y\in X\right\} \\&=\left\{ -y\in Y:\begin{pmatrix} -3y^2\sin (x) e^{-y}\\ -5y^4\sin (x) e^{-y}\\ \end{pmatrix}\not \le 0,~\forall ~ y\in X\right\} \\&=[-5,-4]. \end{aligned}$$

It is obvious that set-valued map $\Gamma$ is closed and convex valued and at least for one $F(x_0)\in Y$, $\Gamma (F(x_0))$ is compact. For $u= 4$, we have

$$\begin{aligned} \left\langle \nabla g(F(u)), \eta (F(x),F(u))\right\rangle&=\begin{pmatrix} 3u^2\\ 5u^4\\ \end{pmatrix} \sin (-x)e^{-u} =\begin{pmatrix} -3u^2\sin (x)e^{-u}\\ -5u^4\sin (x)e^{-u}\\ \end{pmatrix}\\&=\begin{pmatrix} -48\sin (x)e^{-4}\\ -1280\sin (x)e^{-4}\\ \end{pmatrix}\ge 0,\quad \forall \, x\in X. \end{aligned}$$

From this, it is clear that for $u=4,$ there exists no $F(x)\in Y,$ where $x\in X,$ such that

$$\begin{aligned} \left\langle \nabla g(F(u)), \eta (F(x),F(u))\right\rangle =\begin{pmatrix} -48\sin (x)e^{-4}\\ -1280\sin (x)e^{-4}\\ \end{pmatrix}\le 0. \end{aligned}$$

Thus, $F(4)\in Y$ is a solution of (CVVLIP).

Remark 4.1

In the above example, we can easily show that $\nabla g$ is composite properly quasi-monotone with respect to $\eta$ but not properly quasi-monotone with respect to the same $\eta$, as shown below. Since,

$$\begin{aligned} \left\langle \nabla g(y),\eta (x_i,y)\right\rangle =\begin{pmatrix} 3y^2\sin (x_i) e^y\\ 5y^4\sin (x_i) e^y\\ \end{pmatrix}. \end{aligned}$$

Therefore, for any $\{x_1,x_2,\ldots ,x_n\}\in X$ and $y\in conv\{x_1,x_2,\ldots ,x_n\},$

$$\begin{aligned} \left\langle \nabla g(y),\eta (x_i,y)\right\rangle \le 0,\quad \forall \,i\in \{1,2,\ldots ,n\}. \end{aligned}$$

5 Gap function for composite vector variational-like inequality problems

Gap function has become a powerful tool in the study of optimization problems as it can reformulate a variational inequality as an equivalent optimization problem. Gap functions can also be used to obtain error bounds for the solutions of vector variational inequalities. In this section, we define a gap function for the introduced class of composite vector variational-like inequality problem.

Definition 5.1

A function $\phi :Y \mapsto R$ is said to be a gap function for the (CVVLIP) if:

• (i) $\phi (F(u))\le 0,~\forall ~F(u)\in Y~\text {and}~\forall ~u\in X,$

• (ii) $\phi (F(u))=0$ if and only if F(u) is a solution of (CVVLIP).

Now, we define $\phi :Y\mapsto R$ as

$$\begin{aligned} \phi (F(u))= \underset{x\in X}{\underset{F(x)\in Y}{\inf }}\underset{1\le i\le p}{\min }(\left\langle h(F(u)),\eta (F(x),F(u))\right\rangle )_{i}, \end{aligned}$$

(6)

where,

$$\begin{aligned} \left\langle h(F(u)),\eta (F(x),F(u))\right\rangle =(\left\langle h(F(u)),\eta (F(x),F(u))\right\rangle _1,\ldots ,\left\langle h(F(u)),\eta (F(x),F(u))\right\rangle _p) \end{aligned}$$

i.e., $(\left\langle h(F(u)),\eta (F(x),F(u))\right\rangle )_i$ is the $i^{th}$ component of $\left\langle h(F(u)),\eta (F(x),F(u))\right\rangle$, $i=1,2,\ldots ,p.$

Theorem 5.1

The function $\phi$ defined in (6) is the gap function for (CVVLIP), if

$$\begin{aligned} \left\langle h(F(u)),\eta (F(u),F(u))\right\rangle =0,\quad \forall \,F(u)\in Y\quad \text {and}\quad \forall \,u \in X. \end{aligned}$$

Proof

Let $\left\langle h(F(u)),\eta (F(u),F(u))\right\rangle =0,~\forall ~F(u)\in Y~\text {and}~\forall ~u\in X$. Now, we have to prove that function $\phi$ defined in (6) is the gap function for (CVVLIP). It is clear that, for any $F(u)\in Y$

$$\begin{aligned} \phi (F(u))\le \underset{1\le i\le p}{\min }(\left\langle h(F(u)),\eta (F(x),F(u))\right\rangle )_i,\quad \forall \,F(x)\in Y. \end{aligned}$$

In particular, if $F(x)=F(u)$, then by the hypothesis, we have

$$\begin{aligned} \phi (F(u))\le \underset{1\le i\le p}{\min }(\left\langle h(F(u)),\eta (F(u),F(u))\right\rangle )_i=0,\quad \forall \,F(u)\in Y\quad \text {and}\quad \forall \,u\in X, \end{aligned}$$

which implies that, $\phi (F(u))\le 0,~\forall ~F(u)\in Y~\text {and}~\forall ~u\in X.$

On the other hand, if $\phi (F(u))= 0$, then we have

$$\begin{aligned} \underset{x\in X}{\underset{F(x)\in Y}{\inf }}\underset{1\le i\le p}{\min }(\left\langle h(F(u)),\eta (F(x),F(u))\right\rangle )_{i}= 0, \end{aligned}$$

$$\begin{aligned} \Leftrightarrow \underset{1\le i\le p}{\min }(\left\langle h(F(u)),\eta (F(x),F(u))\right\rangle )_{i}\ge 0,\quad \forall \,F(x)\in Y\quad \text {and}\quad \forall \,x\in X. \end{aligned}$$

The above inequality can be rewritten as

$$\begin{aligned} \left\langle h(F(u)),\eta (F(x),F(u))\right\rangle \not \le 0,\quad \forall \,F(x)\in Y\quad \text {and}\quad \forall \,x\in X. \end{aligned}$$

Thus, it follows that, there exists no $F(x)\in Y$ such that

$$\begin{aligned} \left\langle h(F(u)),\eta (F(x),F(u))\right\rangle \le 0, \end{aligned}$$

which implies that, $F(u)\in Y$ is a solution of (CVVLIP). Ultimately, we conclude that $\phi (F(u))=0$ if and only if $F(u)\in Y$ is a solution of (CVVLIP). Hence the theorem. $\square$

We present the following example to illustrate the result established in the above theorem.

Example 5.1

Let $X=[1,2]$ and $Y=R$. Consider the functions $F:X\mapsto Y$, $h:Y\mapsto R^{2 \times 1}$ and $\eta :Y\times Y \mapsto R$ defined as

$$\begin{aligned} F(u)=-u,\;h(F(u))=\begin{pmatrix} F(u)\\ 2F(u)\\ \end{pmatrix}\quad \text {and}\quad \eta (F(x), F(u))=F(x)-F(u), \end{aligned}$$

respectively. Now, for $F(x)=F(u)$, we have

$$\begin{aligned} \left\langle h(F(u)),\eta (F(u),F(u))\right\rangle&= \begin{pmatrix} F(u)\\ 2F(u)\\ \end{pmatrix}\;(F(u)-F(u))\\&=0. \end{aligned}$$

The composite vector variational-like inequality problem (CVVLIP) is to find a point $F(u)\in Y$ such that there exists no $F(x)\in Y$, satisfying

$$\begin{aligned} \left\langle h(F(u)),\eta (F(x),F(u))\right\rangle&= \begin{pmatrix} ux-u^2\\ 2ux-2u^2\\ \end{pmatrix}\le 0. \end{aligned}$$

Now, we shall show that $\phi$ defined in (6) is the gap function for (CVVLIP).

• (i) Since,

  $$\begin{aligned} \phi (F(u))&= \underset{x\in X}{\underset{F(x)\in Y}{\inf }}\underset{1\le i\le p}{\min }(\left\langle h(F(u)),\eta (F(x),F(u))\right\rangle )_{i}\\&=\underset{x\in X}{\inf }\min (ux-u^2,2ux-2u^2)\\&=\min (u-u^2,2u-2u^2). \end{aligned}$$

  Thus, we observe that $\phi (F(u))\le 0,~\forall ~F(u)\in Y~\text {and}~\forall ~u\in X.$

• (ii) It is obvious that $\phi (F(u))=0$ for $u=1$. Now, for $u=1$, we have

  $$\begin{aligned} \left\langle h(F(u)),\eta (F(x),F(u))\right\rangle&= \begin{pmatrix} ux-u^2\\ 2ux-2u^2\\ \end{pmatrix}\\&=\begin{pmatrix} x-1\\ 2x-2\\ \end{pmatrix}. \end{aligned}$$

  From above, it is clear that for $u=1$, there exists no $F(x)\in Y$, where $x\in X$ such that

  $$\begin{aligned} \left\langle h(F(u)),\eta (F(x),F(u))\right\rangle = \begin{pmatrix} x-1\\ 2x-2\\ \end{pmatrix}\le 0, \end{aligned}$$

  which means that $F(1)\in Y$ is a solution of (CVVLIP).

6 Applications

The Nash equilibrium problem, in which each player’s strategy may depend on the rival strategies, has attracted growing attention due to its interesting applications in the field of economics, mathematics, engineering and operations research. Facchinei and Kanzow (2007) served a historical note, some relevant applications and also existence results of Nash equilibrium problems. These results stimulate the interest of Nash equilibrium problem in operations research field. For various approaches, see Facchinei and Kanzow (2007), Krawczyk (2007) and Pang and Fukushima (2009). Accordingly, as an application, we define a system of composite vector optimization problems and system of composite vector variational-like inequality problems. Moreover, solutions of these introduced problems imply the solution of Nash equilibrium problem.

Throughout this section, let $I=\{1,\ldots ,n\}$ be a finite index set and for each $i \in I$, $X_i$ and $Y_{i}$ be finite dimensional Euclidean spaces $R^{q_{i}}$ and $R^{p_{i}}$, respectively. Further, we assume $\{K_{i}\}_{i\in I}$ is a family of nonempty convex subsets with each $K_{i}$ in $X_{i}$ and define $K ={\Pi }_{i \in I}K_i$, an element of the set $K^{i}={\Pi }_{j\in I,~i\ne j}K_{j}$ is written as $x^{i}$, therefore, $x\in K$ can be written as $x=(x^{i},x_{i})\in K^{i}\times K_{i}$. We consider the functions, $A_i:K\mapsto Y_{i},~\text {for each}~i\in I$ and $F:K\mapsto K$, where$A_{i}(x)=(A_{i_{1}}(x),\ldots ,A_{i_{p_{i}}}(x))$.

A Nash equilibrium problem is to find an element $u\in K$ such that for each $i\in I,$ there exists no $x_{i}\in K_{i}$, satisfying

$$\begin{aligned} A_i(u^{i},x_{i})-A_{i}(u)< 0. \end{aligned}$$

(SCVOP) The system of composite vector optimization problems is to find a point $F(u)\in K$ such that for each $i\in I$, there exists no $F(x)\in K$, satisfying

$$\begin{aligned} A_i(F(x))- A_i(F(u))< 0. \end{aligned}$$

(7)

Now, we can choose $F(x)\in K$ in such a way that, for each $i\in I$

$$\begin{aligned} F^i(x)=F^i(u)=u^i,\quad F_i(x)=x_i\quad and \quad F_i(u)=u_i. \end{aligned}$$

(8)

Since $F(u)\in K$, we have $F(u)= (F^i(u),F_i(u))=(u^i,u_i)=u\in K$. Then, by using assumption (8), (SCVOP) reduces to find a point $u\in K$ such that for each $i\in I$, there exists no $x_{i}\in K_{i}$, satisfying

$$\begin{aligned} A_i(u^i,x_i)-A_i(u)< 0, \end{aligned}$$

which is the well-known Nash equilibrium problem. Clearly, every solution of (SCVOP) is also a solution of Nash equilibrium problem but converse is not true.

Now, we define the composite invexity for the function $A_{i}:K\mapsto Y_{i},~i\in I$, by keeping the view of generalized invexity, considered in Ansari et al. (2002).

Definition 6.1

For each $i\in I$, $A_{i}:K\mapsto Y_{i}$ is called composite invex at $F(u)\in K$ with respect to a given function $\eta _{i}:K_{i}\times K_{i}\mapsto X_{i}$, if

$$\begin{aligned} A_{i}(F(x))-A_{i}(F(u))\geqq \left\langle \nabla A_{i}(F(u)),\eta _{i}(F_{i}(x),F_{i}(u)) \right\rangle ,\quad \forall \,F(x)\in K. \end{aligned}$$

Ultimately, we introduce the system of composite vector variational-like inequality problems.

(SCVVLIP) To find $F(u)\in K$ such that for each $i\in I$, there exists no $F_{i}(x)\in K_{i}$, satisfying

$$\begin{aligned} \left\langle \nabla A_{i}(F(u)),\eta _{i}(F_{i}(x),F_{i}(u))\right\rangle < 0. \end{aligned}$$

Theorem 6.1

Let for each $i\in I$, $A_{i}$ be composite invex function at $F(u)\in K$ and with respect to $\eta _{i}$. If F(u) solves (SCVVLIP), then it solves (SCVOP).

Proof

Suppose, contrary to the hypothesis, that F(u) solves (SCVVLIP) but does not solve (SCVOP). Therefore, for some $i\in I$, there exists $F(x)\in K$ such that

$$\begin{aligned} A_{i}(F(x))-A_{i}(F(u))< 0. \end{aligned}$$

(9)

By using the invexity of each $A_{i}$ with respect to given $\eta _{i}$ at point F(u), we have

$$\begin{aligned} A_{i}(F(x))-A_{i}(F(u))\geqq \left\langle \nabla A_{i}(F(u)),\eta _{i}(F_{i}(x),F_{i}(u)) \right\rangle ,\quad \forall \,F(x)\in K. \end{aligned}$$

(10)

On combining inequalities (9) and (10), it follows that for some $i\in I$, there exists $F(x)\in K$ such that

$$\begin{aligned} \left\langle \nabla A_{i}(F(u)),\eta _{i}(F_{i}(x),F_{i}(u)) \right\rangle < 0, \end{aligned}$$

which leads to a contradiction that F(u) solves (SCVVLIP). Hence the theorem. $\square$

Remark 6.1

Since every solution of (SCVOP) is also a solution of Nash equilibrium problem. Hence, by Theorem 6.1, we can say that every solution of (SCVVLIP) is also a solution of Nash equilibrium problem.

7 Conclusion

In this paper, we have introduced composite vector variational-like inequality problem and established its relationship with composite vector optimization problem under the condition of composite invexity. We have also established the relation of a vector critical point with a weak efficient point of composite vector optimization problem, using the assumption of composite pseudo invexity. Moreover, we have derived the existence result for solutions of composite vector variational-like inequality problem using composite proper quasi-monotonicity and defined a gap function. For application, we have introduced the system of composite vector optimization problems and system of vector variational-like inequality problems and derived the relation of these problems with Nash equilibrium problem.