1 Introduction

Optimality condition is one of the most important topics in optimization. It gives us a tool to choose the best solution of related mathematical models. To study this topic, the concept of derivatives plays an essential role. For set-valued optimization, the contingent derivative, whose definition is based on the contingent cone of graph of set-valued map, has been considered as the first and the most important notion for the formulation of optimality conditions and duality, see [4, 10,11,12, 19,20,21, 24,25,27]. It extends the Fréchet derivative naturally from smooth single-valued case to set-valued one.

In [9] (see Theorems 4.1 and 4.2), Corley showed that necessary and sufficient optimality conditions in terms of the contingent derivative do not coincide under standard assumptions. To overcome this case, Jahn and Rauh introduced the contingent epiderivative defined by the contingent cone of epigraph of corresponding map, see [16]. Then, Bednarczuk and Song established more applications of this concept in [5]. Inspired by this notion, there have been many concepts of (higher-order) generalized contingent epiderivatives proposed and applied to some topics in optimization recently, see [2, 7, 8, 17, 28, 29]. All these generalized epiderivatives capture local information of concerning maps (using \(t_n\rightarrow 0^+\) in their formulations). To reflect global nature (in the sense of epigraph), Flores-Bazán introduced the radial epiderivative in [13]. Its properties were studied in [14, 18]. In [1], Anh extended this concept to higher orders and obtained its applications to duality of set-valued optimization problems. The above-mentioned (higher-order) radial epiderivative is single-valued map even though related maps are set-valued.

In the paper, we propose the higher-order generalized radial epiderivative for a set-valued map. The main difference between this notion and the concept in [1] is that the higher-order generalized radial epiderivative is set-valued map, while the higher-order radial epiderivative is single-valued one. Besides, we discuss its applications to optimality conditions for set-valued optimization problems.

The layout of this paper is as follows. Section 2 is devoted to some concepts needed in the sequel. In Sect. 3, the higher-order generalized radial epiderivative and its properties are introduced. Then, we establish higher-order optimality conditions for set-valued optimization problems via this epiderivative in Sect. 4.

2 Preliminaries

Throughout the paper, let X, Y be normed spaces and \(C\subseteq Y\) be a pointed closed convex cone. \(Y^*\) stands for the dual space of Y. With the cone C above, the dual cone \(C^*\) is defined by

$$\begin{aligned} C^*:=\left\{ c^*\in Y^*| \langle {c^*,c}\rangle \ge 0, \forall c\in C\right\} . \end{aligned}$$

Definition 2.1

([4]) Let \(A\subseteq Y\) and \(a_0\in A\).

  1. (i)

    \(a_0\in A\) is said to be a Pareto efficient point of A (\(a_0\in \mathrm{Min}_CA\)) if \((A-\{a_0\})\cap (-C\setminus \{0\}) = \emptyset \).

  2. (ii)

    Suppose that \(\mathrm{int}C\ne \emptyset \), \(a_0\in A\) is said to be a weak efficient point of A (\(a_0\in \mathrm{WMin}_CA\)) if \((A-\{a_0\})\cap (-\mathrm{int}C) = \emptyset \).

Definition 2.2

([15, 23]) (i) The cone C is called Daniell if any decreasing sequence in Y having a lower bound converges to its infimum.

  1. (ii)

    A subset A of Y is said to be minorized if there is a \(y \in Y\) so that \( A\subseteq y + C\).

  2. (iii)

    The domination property is said to hold for a subset A of Y if \(A\subseteq \mathrm{Min}_C A + C\).

  3. (iv)

    The space Y is said to be boundedly order complete if every bounded decreasing sequence has an infimum.

Proposition 2.1

([6, 23]) Let \(A\subseteq Y\) be a nonempty subset. If A is closed and bounded (or is minorized), C is a Daniell cone, and Y is boundedly order complete, then \(\mathrm{Min}_CA \ne \emptyset \).

For a set-valued map \(F : X \rightarrow 2^Y\), the domain, image, graph, epigraph of F are defined by, respectively,

$$\begin{aligned} \mathrm{dom}F:= & {} \{x\in X| F(x)\ne \emptyset \},\;\;\;\mathrm{im}F:=\{y\in Y|y\in F(X)\},\\ \mathrm{gr}F:= & {} \{(x,y)\in X\times Y| y\in F(x)\},\;\;\; \mathrm{epi}F:=\{(x,y)\in X\times Y| y\in F(x) + C\}. \end{aligned}$$

Definition 2.3

([1, 3]) (i) Let \(A\subseteq X\times Y\) at \((x_0,y_0)\in \mathrm{cl}A\), the mth-order radial set of A at \((x_0,y_0)\) is defined by

$$\begin{aligned} R^{m}_A(x_0,y_0):=&\left\{ (u,v)\in X\times Y| \exists t_n>0, \exists (u_n,v_n)\right. \\&\left. \rightarrow (u,v), \left( x_0+t_nu_n,y_0+t_n^mv_n\right) \in A\right\} . \end{aligned}$$

(ii) Let \(F:X\rightarrow 2^Y\) and \((x_0,y_0)\in \mathrm{gr}F\), a single-valued map \(ED^{m}_RF(x_0,y_0): X\rightarrow Y\) whose epigraph equals the mth-order radial set of epigraph of F at \((x_0,y_0)\), i.e.,

$$\begin{aligned} \mathrm{epi}ED^{m}_RF(x_0,y_0)=R^{m}_{\mathrm{epi}F}(x_0,y_0), \end{aligned}$$

is called the mth-order radial epiderivative of F at \((x_0,y_0)\).

Definition 2.4

([22]) Let \(e\in \mathrm{int}C\) and \(a\in Y\), the Gerstewitz’s nonconvex separation functional \(\xi _{ea}: Y\rightarrow {\mathbb {R}}\) is defined by

$$\begin{aligned} \xi _{ea} (y) = \mathrm{inf} \left\{ {t \in {\mathbb {R}} | y \in a + te - C} \right\} . \end{aligned}$$

Proposition 2.2

([22]) Let \(e\in \mathrm{int}C\), \(a\in Y\) and \(y\in Y\), we have

  1. (i)

    \(\xi _{ea}(y) < r \Longleftrightarrow y \in a + re - \mathrm{int }\,C\);

  2. (ii)

    \(\xi _{ea}(y) \le r \Longleftrightarrow y \in a + re - C\);

  3. (iii)

    \(\xi _{ea}(y) > r \Longleftrightarrow y \notin a + re - C\);

  4. (iv)

    \(\xi _{ea}(y) \ge r \Longleftrightarrow y \notin a + re - \mathrm{int}\, C\).

Proposition 2.3

([22]) Let \(C\subseteq Y\) be a closed convex cone with \(\mathrm{int} C\ne \emptyset \), \(e\in \mathrm{int} C\) and \(a\in Y\). Then, there exists a subset \(\Gamma \subseteq C^*\setminus \{0\}\) such that, for \(y\in Y\),

$$\begin{aligned} C= & {} \left\{ y\in Y |\langle {y^*,y}\rangle \ge 0, \quad \forall y^*\in \Gamma \right\} ,\\ \xi _{ea} (y)= & {} \mathop {\sup }\limits _{y^* \in \Gamma } \left\{ {{{\left\langle {y^*,y}\right\rangle - \left\langle {y^*,a}\right\rangle } \over {\left\langle {y^*,e}\right\rangle }}} \right\} . \end{aligned}$$

3 The Higher-Order Generalized Radial Epiderivative

Definition 3.1

Let \(F:X\rightarrow 2^Y\) and \((x_0,y_0)\in \mathrm{gr}F\), the mth-order generalized radial epiderivative of F at \((x_0,y_0)\) is a set-valued map \(G\text {-}ED^{m}_RF(x_0,y_0): X\rightarrow 2^Y\) defined by

$$\begin{aligned} G\text {-}ED_R^{m}F(x_0, y_0)(x) := \mathrm{Min}_CH(x), \end{aligned}$$

where \(H(x):=\{y \in Y| (x, y) \in R^{m}_{\mathrm{epi}F}(x_0, y_0)\}.\)

The following example shows the case that \(G\text {-}ED_R^{m}F(x_0, y_0)\) exists, but \(ED_R^{m}F(x_0, y_0)\) does not.

Example 3.1

Let \(X={\mathbb {R}}\), \(Y={\mathbb {R}}^2\), \(C={\mathbb {R}}^2_+\), and \(F: X\rightarrow 2^Y\) be defined by \(F(x):=\{(u,v)\in Y | u+v \ge 0\}\). By calculating, we get that \(ED_R^{m}F(0,0,0)(x)=\emptyset \), while

$$\begin{aligned} G\text {-}ED_R^{m}F(0,0,0)(x)=\{(u,v)\in Y| u+v =0\}. \end{aligned}$$

From Proposition 2.1, we get the existence of the higher-order generalized radial epiderivative directly as follows.

Proposition 3.1

Let \(F:X\rightarrow 2^Y\) and \((x_0,y_0)\in \mathrm{gr}F\). If H(x) is closed and bounded (or is minorized) for all \(x\in \mathrm{dom}H\), C is a Daniell cone, and Y is boundedly order complete, then \(G\text {-}ED_R^{m}F(x_0, y_0)\) exists.

If the higher-order radial epiderivative exists, then Propositions 3.2 and 3.3 in [1] give us relationships between Definitions 2.3(ii) and 3.1 in case of \(Y={\mathbb {R}}\) and (Y, C) being an order complete vector lattice, respectively. We now establish another relationship of these epiderivatives.

Proposition 3.2

Let \(F:X\rightarrow 2^Y\) and \((x_0,y_0)\in \mathrm{gr}F\). If \(ED_R^{m}F(x_0, y_0)\) exists and H(x) satisfies the domination property for all \(x\in \mathrm{dom}H\), then

$$\begin{aligned} \mathrm{epi} G\text {-}ED_R^{m}F(x_0, y_0) = \mathrm{epi} ED_R^{m}F(x_0, y_0). \end{aligned}$$

Proof

“\(\subseteq \)”: Let \((x,y)\in \mathrm{epi} G\text {-}ED_R^{m}F(x_0, y_0)\), then \((x,y)\in R^m_{\mathrm{epi}F}(x_0,y_0)\). It follows from Definition 2.3(ii) that \((x,y)\in \mathrm{epi} ED_R^{m}F(x_0, y_0)\).

“\(\supseteq \)”: Suppose to the contrary, i.e., there is \((x,y)\in \mathrm{epi} ED_R^{m}F(x_0, y_0)\), but \((x,y)\not \in \mathrm{epi} G\text {-}ED_R^{m}F(x_0, y_0)\). Then, we have

$$\begin{aligned} y\not \in G\text {-}ED_R^{m}F(x_0, y_0)(x) + C, \end{aligned}$$

equivalently,

$$\begin{aligned} y\not \in \mathrm{Min}_C\left\{ y\in Y| (x,y)\in R^m_{\mathrm{epi}F}(x_0,y_0)\right\} +C. \end{aligned}$$
(1)

Since \((x,y)\in \mathrm{epi} ED_R^{m}F(x_0, y_0)\), one gets \((x,y)\in R^m_{\mathrm{epi}F}(x_0,y_0)\), which implies that

$$\begin{aligned} y\in \left\{ y\in Y| (x,y)\in R^m_{\mathrm{epi}F}(x_0,y_0)\right\} (= H(x)). \end{aligned}$$

By the domination property, there exist \({\hat{y}}\in \mathrm{Min}_C\{y\in Y| (x,y)\in R^m_{\mathrm{epi}F}(x_0,y_0)\}\) and \({\hat{c}}\in C\) such that \(y={\hat{y}} + {\hat{c}}\), which contradicts (1). \(\square \)

Example 3.2

Let \(X={\mathbb {R}}\), \(Y={\mathbb {R}}^2\), \(C={\mathbb {R}}^2_+\), and \(F: X\rightarrow 2^Y\) be defined by

$$\begin{aligned} F(x):=\left\{ (u,v)\in Y | u+v \ge x^2, u\ge 0, v\ge 0\right\} . \end{aligned}$$

Then, we get that \(ED_R^{2}F(0,0,0)(x)=\{(0,0)\}\), and

$$\begin{aligned}&R^2_{\mathrm{epi}F}(0,0,0)=\bigcup _{x\in X}\left\{ (x,u,v)| u+v \ge x^2, u\ge 0, v\ge 0 \right\} ,\\&H(x):=\left\{ (u,v)\in Y| (x,u,v)\in R^2_{\mathrm{epi}F}(0,0,0)\right\} \\&\qquad \quad \,\,\,=\left\{ (u,v)\in Y| u + v \ge x^2, u\ge 0, v\ge 0\right\} ,\\&G\text {-}ED_R^{2}F(0,0,0)(x):= \mathrm{Min}_C H(x) = \left\{ (u,v)\in Y| u+v =x^2,u\ge 0, v\ge 0\right\} . \end{aligned}$$

It is obvious that \(H(x)\subseteq \mathrm{Min}_C H(x) + C\), i.e., H(x) satisfies the domination property. By Proposition 3.2, one has

$$\begin{aligned} \mathrm{epi} G\text {-}ED_R^{2}F(0,0,0) = \mathrm{epi} ED_R^{2}F(0,0,0). \end{aligned}$$

In the rest of this section, we discuss some properties of the higher-order generalized radial epiderivative.

Proposition 3.3

Let \(F: X\rightarrow 2^Y\) and \((x_0,y_0)\in \mathrm{gr}F\). If \(G\text {-}ED^m_R F(x_0,y_0)\) exists then

  1. (i)

    \(G\text {-}ED^m_R F(x_0,y_0)(\alpha x) = \alpha ^m G\text {-}ED^m_R F(x_0,y_0)(x)\) for all \(\alpha >0\).

  2. (ii)

    Suppose that \(H(x-x_0)\) satisfies the domination property for all \(x\in \mathrm{dom}F\), then

    $$\begin{aligned} F(x) - y_0 \subseteq G\text {-}ED^m_R F(x_0,y_0)(x-x_0) + C. \end{aligned}$$

Proof

  1. (i)

    By Definition 3.1, one gets

    $$\begin{aligned} \begin{array}{lll} \dfrac{1}{\alpha ^m}G\text {-}ED^m_R F(x_0,y_0)(\alpha x)&{}=&{} \mathrm{Min}_C\left\{ \dfrac{1}{\alpha ^m} y \in Y| (\alpha x,y)\in R^m_{\mathrm{epi}F}(x_0,y_0)\right\} \\ &{}=&{} \mathrm{Min}_C\left\{ u \in Y| (\alpha x,\alpha ^m u)\in R^m_{\mathrm{epi}F}(x_0,y_0)\right\} \\ &{}=&{} \mathrm{Min}_C\left\{ u \in Y| (x,u)\in R^m_{\mathrm{epi}F}(x_0,y_0)\right\} \\ &{}=&{} G\text {-}ED^m_R F(x_0,y_0)(x). \end{array} \end{aligned}$$
  2. (ii)

    Let \(y\in F(x)\), then there exist \(t_n=1\), \(y_n = y-y_0\), \(x_n = x-x_0\) such that

    $$\begin{aligned} y_0 + t_n^my_n \in F(x_0 + t_nx_n), \end{aligned}$$

    i.e., \((x-x_0,y-y_0)\in R^m_{\mathrm{epi}F}(x_0,y_0)\). Thus, \(y-y_0 \in H(x-x_0)\). From the domination property, we have

    $$\begin{aligned} y-y_0\in G\text {-}ED^m_R F(x_0,y_0)(x-x_0) + C. \end{aligned}$$

\(\square \)

The domination property is essential for Proposition 3.3(ii) by the following example.

Example 3.3

Let \(X={\mathbb {R}}\), \(Y={\mathbb {R}}^2\), \(C={\mathbb {R}}^2_+\), and \(F: X\rightarrow 2^Y\) be defined by

$$\begin{aligned} F(x):=\{(u,v)\in Y | u\ge 0, v\ge 0\}\cup \{(u,v)\in Y | u \ge x^2\}\cup \{(u,v)\in Y | v \ge x^2\}. \end{aligned}$$

Then,

$$\begin{aligned} R^2_{\mathrm{epi}F}(0,0,0)= & {} \bigcup _{x\in X}\left( \{(x,u,v)| u\ge 0, v\ge 0 \}\cup \right. \\&\quad \left. \{(x,u,v)| u \ge x^2\}\cup \{(x, u,v)| v \ge x^2\}\right) ,\\ H(x)= & {} \{(u,v)\in Y | u\ge 0, v\ge 0\}\cup \{(u,v)\in Y | u \ge x^2\}\cup \\&\quad \{(u,v)\in Y | v \ge x^2\},\\ G\text {-}ED_R^{2}F(0,0,0)(x)= & {} \left\{ \begin{array}{ll} \{(0,0)\},&{}\quad \mathrm{if}\; x\ne 0,\\ \emptyset ,&{} \quad \mathrm{if}\; x=0. \end{array}\right. \end{aligned}$$

We can check that H(x) does not hold the domination property. Then, one has

$$\begin{aligned} F(x)-0 \nsubseteq G\text {-}ED^m_R F(0,0,0)(x-0) + C. \end{aligned}$$

4 Applications to Optimality Conditions

Let X, Y, Z be normed spaces, \(C\subseteq Y\) and \(D\subseteq Z\) be pointed closed convex cones with nonempty interior, consider the following constrained set-valued optimization problem

$$\begin{aligned} \mathrm{(SOP)}\;\;\;\left\{ \begin{array}{ll} &{}\quad \mathrm{WMin}_C F(x)\\ \mathrm{s.t.}&{}\quad x\in S,\\ &{}\quad G(x)\cap (-D)\ne \emptyset , \end{array}\right. \end{aligned}$$

where \(S\subseteq X\), \(F:X\rightarrow 2^Y\), \(G:X\rightarrow 2^Z\) with \(\mathrm{dom}F\cup \mathrm{dom}G \subseteq S\).

The set \(A:=\{x\in S| G(x)\cap (-D)\ne \emptyset \}\) is a feasible set of (SOP). A point \((x_0,y_0)\in \mathrm{gr}F\) is said to be a weak efficient solution of (SOP) if \(x_0\in A\) and \((F(A) - \{y_0\})\cap (-\mathrm{int}C)=\emptyset \).

In this section, we establish some types of optimality conditions for weak efficient solutions of (SOP) via the higher-order generalized radial epiderivative.

Theorem 4.1

(Primal type) Let \((x_0,y_0)\in \mathrm{gr}F\) and \(z_0\in G(x_0)\cap (-D)\). Suppose that \(G\text {-}ED^m_R(F,G)(x_0,y_0,z_0)\) exists.

(i) (Necessary condition) If \((x_0,y_0)\) is a weak efficient solution of (SOP), then for all \(x\in \Omega :=\mathrm{dom}G\text {-}ED^m_R(F,G)(x_0,y_0,z_0)\),

$$\begin{aligned} G\text {-}ED^m_R(F,G)(x_0,y_0,z_0)(\Omega )\cap -\mathrm{int}(C\times D)=\emptyset . \end{aligned}$$

(ii) (Sufficient condition) If \(H'(x-x_0)\) satisfies the domination property for all \(x\in S\) and

$$\begin{aligned} G\text {-}ED^m_R(F,G)(x_0,y_0,z_0)(A-x_0)\cap -(\mathrm{int}C\times D(z_0))=\emptyset , \end{aligned}$$

where \(H'(x-x_0):= \{(y,z)\in Y\times Z| (x-x_0,y,z)\in R^m_{\mathrm{epi}(F,G)}(x_0,y_0,z_0)\}\) and \(D(z_0):=\mathrm{cone}(D+z_0)\), then \((x_0,y_0)\) is a weak efficient solution of (SOP).

Proof

  1. (i)

    Suppose to the contrary, i.e., there is \(x\in \Omega \) such that

    $$\begin{aligned} (u,v)\in G\text {-}ED^m_R(F,G)(x_0,y_0,z_0)(x)\cap -\mathrm{int}(C\times D). \end{aligned}$$

    By Definition 3.1, we get \((x,u,v)\in R^m_{\mathrm{epi}(F,G)}(x_0,y_0,z_0)\), which implies the existence of \(t_n >0\), \((x_n,y_n,z_n)\in \mathrm{gr}(F,G)\), and \((c_n,d_n)\in C\times D\) with

    $$\begin{aligned} \dfrac{y_n + c_n -y_0}{t_n^m}\rightarrow u, \;\;\dfrac{z_n+d_n-z_0}{t_n^m}\rightarrow v, \dfrac{x_n-x_0}{t_n}\rightarrow x. \end{aligned}$$

    For n large enough, one has

    $$\begin{aligned} y_n + c_n -y_0 \in -\mathrm{int}C,\;\; z_n + d_n -z_0 \in -\mathrm{int}D, \end{aligned}$$

    so \(y_n -y_0\in -\mathrm{int}C\) and \(z_n \in -\mathrm{int}D\). Thus, there is \(x_n\in A\) such that \((F(x_n) - y_0)\cap (-\mathrm{int}C)\ne \emptyset \), which contradicts that \((x_0,y_0)\) is a weak efficient solution of (SOP).

  2. (ii)

    Suppose that \((x_0,y_0)\) is not a weak efficient solution of (SOP). Then there exist \({\hat{x}}\in S\), \({\hat{y}}\in F({\hat{z}})\), and \({\hat{z}}\in G({\hat{x}})\cap (-D)\) such that \({\hat{y}} - y_0 \in -\mathrm{int}C\). Thus, we get

    $$\begin{aligned} ({\hat{y}},{\hat{z}}) - (y_0,z_0)\in -(\mathrm{int}C\times D(z_0)). \end{aligned}$$
    (2)

It follows from the assumption and Proposition 3.3(ii) that for all \(x\in S\),

$$\begin{aligned} ((F,G)(x) - (y_0,z_0))\cap -(\mathrm{int}C\times D(z_0))=\emptyset , \end{aligned}$$

which contradicts (2). \(\square \)

Example 4.1

Let \(X=Z={\mathbb {R}}\), \(Y={\mathbb {R}}^2\), \(S=X\), \(C= {\mathbb {R}}^2_+\), \(D={\mathbb {R}}_+\). Consider (SOP) with \(F:X\rightarrow 2^Y\) and \(G:X\rightarrow 2^Z\) defined by

$$\begin{aligned} F(x):=\{(u,v)\in Y| u^2+v^2 \le x^2\},\;\; G(x):= {\mathbb {R}}. \end{aligned}$$

Let \((x_0,y_0,z_0) = (0,0,0)\), by calculating, we get

$$\begin{aligned} G\text {-}ED^2_R(F,G)(x_0,y_0,z_0)(x) = \{(u,v,w)\in Y\times Z| u^2 + v^2 = x^2, u\le 0, v\le 0\}. \end{aligned}$$

Thus

$$\begin{aligned} G\text {-}ED^2_R(F,G)(x_0,y_0,z_0)(x)\cap -\mathrm{int}(C\times D(z_0)) \ne \emptyset . \end{aligned}$$

By Theorem 4.1(i), \((x_0,y_0)\) is not a weak efficient solution of (SOP).

Example 4.2

Let \(X=Z={\mathbb {R}}\), \(Y={\mathbb {R}}^2\), \(S=X\),\(C= {\mathbb {R}}^2_+\), \(D={\mathbb {R}}_+\). Consider (SOP) with \(F:X\rightarrow 2^Y\) and \(G:X\rightarrow 2^Z\) defined by

$$\begin{aligned} F(x):=\{(u,v)\in Y| u+v \ge x^2\},\;\; G(x):= {\mathbb {R}}. \end{aligned}$$

Let \((x_0,y_0,z_0) = (0,0,0)\), we can check that \(H'(x-x_0)\) satisfies the domination property for all \(x\in S\),

$$\begin{aligned} G\text {-}ED^2_R(F,G)(x_0,y_0,z_0)(x) = \{(u,v,w)\in Y\times Z| u + v = x^2\}, \end{aligned}$$

and

$$\begin{aligned} G\text {-}ED^2_R(F,G)(x_0,y_0,z_0)(x)\cap -(\mathrm{int}C\times D(z_0)) = \emptyset . \end{aligned}$$

By Theorem 4.1(ii), \((x_0,y_0)\) is a weak efficient solution of (SOP).

In Examples 4.1, 4.2, the (higher-order) radial epiderivative, which is single-valued map, does not exist. Thus, results in [1, 13, 14, 18] cannot be employed to check the efficiency of \((x_0,y_0)\) in these cases.

Theorem 4.2

(Dual type) Let \((x_0,y_0)\in \mathrm{gr}F\) and \(z_0\in G(x_0)\cap (-D)\). Suppose that \(G\text {-}ED^m_R(F,G)(x_0,y_0,z_0)\) exists.

(i) (Necessary condition) If \((x_0,y_0)\) is a weak efficient solution of (SOP) and \((F-y_0,G)(S)\) is convex, then there exists \((c^*,d^*)\in (C^*\times D^*)\setminus \{(0,0)\}\) such that for all \((y,z)\in G\text {-}ED^m_R(F,G)(x_0,y_0,z_0)(\Omega )\),

$$\begin{aligned} \left\langle {c^* ,y} \right\rangle + \left\langle {d^* ,z} \right\rangle \ge 0, \end{aligned}$$
(3)

and

$$\begin{aligned} \left\langle {d^* ,z_0} \right\rangle = 0. \end{aligned}$$
(4)

If, additionally, there exists \(z\in G(S)\) satisfying \(0\in \mathrm{int}(z+z_0 +D)\), then \(c^*\ne 0\).

(ii) (Sufficient condition) If \(H'(x-x_0)\) (defined in Theorem 4.1(ii)) satisfies the domination property for all \(x\in S\) and there are \(c^*\in C^*\setminus \{0\}\), \(d^*\in D^*\) such that (3) and (4) hold, then \((x_0,y_0)\) is a weak efficient solution of \(\mathrm{(SOP)}\).

Proof

(i) It follows from the weak efficiency of \((x_0,y_0)\) that

$$\begin{aligned} (F-y_0,G)(S)\cap -\mathrm{int}(C\times D) = \emptyset . \end{aligned}$$

Since \((F-y_0,G)(S)\) is convex, there exists \((c^*,d^*)\in (Y^*\times Z^*)\setminus \{(0,0)\}\) such that for all \((y,z)\in (F,G)(S)\), \((c,d)\in -\mathrm{int}(C\times D)\),

$$\begin{aligned} \left\langle {c^* ,y-y_0} \right\rangle + \left\langle {d^* ,z} \right\rangle \ge 0, \end{aligned}$$
(5)

and

$$\begin{aligned} \left\langle {c^* ,c} \right\rangle + \left\langle {d^* ,d} \right\rangle \le 0. \end{aligned}$$

Since C, D are closed convex cones, we get \(c^*\in C^*\) and \(d^*\in D^*\). Taking \(y=y_0\), \(z=z_0\) in (5), we get \(\left\langle {d^* ,z_0} \right\rangle \ge 0\), which implies that \(\left\langle {d^* ,z_0} \right\rangle = 0\) (since \(z_0\in -D\)). Thus, (4) holds.

Let \((y,z)\in G\text {-}ED^m_R(F,G)(x_0,y_0,z_0)(\Omega )\), then there exists \(x\in S\) with \((x,y,z)\in R^m_{\mathrm{epi}(F,G)}(x_0,y_0,z_0)\), i.e., there are \(t_n >0\), \((x_n,y_n,z_n)\in \mathrm{gr}(F,G)\), and \((c_n,d_n)\in C\times D\),

$$\begin{aligned} \dfrac{y_n-y_0 + c_n}{t_n^m}\rightarrow y,\;\;\dfrac{z_n-z_0 + d_n}{t_n^m}\rightarrow z,\;\;\dfrac{x_n-x_0}{t_n}\rightarrow x. \end{aligned}$$

It follows from (5) that

$$\begin{aligned} \left\langle {c^* ,\dfrac{y_n-y_0 + c_n}{t_n^m}} \right\rangle + \left\langle {d^* ,\dfrac{z_n-z_0 + d_n}{t_n^m}} \right\rangle \ge 0, \end{aligned}$$

then \(\left\langle {c^* ,y} \right\rangle + \left\langle {d^* ,z} \right\rangle \ge 0\).

Suppose that there exists \(z\in G(S)\) satisfying \(0\in \mathrm{int}(z+z_0 +D)\). If \(c^* = 0\), from (5), one has \( \left\langle {d^* ,z + z_0 +d} \right\rangle \ge 0\) for all \(d\in D\), i.e., \(0\not \in \mathrm{int}(z+z_0+D)\), which is a contradiction. Hence, \(c^*\ne 0\).

(ii) By the domination property and Proposition 3.3(ii), we get for all \(x\in S\),

$$\begin{aligned} \left\langle {c^* ,y-y_0} \right\rangle + \left\langle {d^* ,z-z_0} \right\rangle \ge 0. \end{aligned}$$
(6)

If \((x_0,y_0)\) is not a weak efficient solution of (SOP), then there exist \({\hat{x}}\in S\), \({\hat{y}}\in F({\hat{x}})\), and \({\hat{z}}\in G({\hat{x}})\cap (-D)\) such that \({\hat{y}}-y_0\in -\mathrm{int}C\). From (6), one has

$$\begin{aligned} \left\langle {c^* ,{\hat{y}}-y_0} \right\rangle \ge \left\langle {d^* ,-{\hat{z}}} \right\rangle \ge 0, \end{aligned}$$

which is a contradiction. \(\square \)

In the next result, we establish optimality conditions in dual type for (SOP) without the convexity.

Theorem 4.3

Let \((e,k)\in \mathrm{int} (C\times D)\), \((x_0,y_0)\in \mathrm{gr}F\) and \(z_0\in G(x_0)\cap (-D)\). Suppose that \(G\text {-}ED^m_R(F,G)(x_0,y_0,z_0)\) exists.

(i) (Necessary condition) If \((x_0, y_0)\) be a weak efficient solution of (SOP), then there exists \((\Gamma , L) \subseteq (C^*\times D^*){\setminus } \{(0,0)\}\) such that for all \((y,z) \in G\text {-}ED^m(F,G)(x_0,y_0,z_0)(\Omega )\),

$$\begin{aligned} C= & {} \{y\in Y | \left\langle {c^*,y}\right\rangle \ge 0, \;\forall c^*\in \Gamma \},\;\; D=\{z\in Z | \left\langle {d^*,z}\right\rangle \ge 0,\; \forall d^*\in L\}, \\&\mathop {\sup }\limits _{(c^* ,d^* ) \in (\Gamma ,L)} \left\{ {{{\left\langle {c^*,y}\right\rangle + \left\langle {d^*,z}\right\rangle } \over {\left\langle {c^*,e}\right\rangle + \left\langle {d^*,k}\right\rangle }}} \right\} \ge 0. \end{aligned}$$

(ii) (Sufficient condition) If \(H'(x-x_0)\) (defined in Theorem 4.1(ii)) satisfies the domination property for all \(x\in S\) and there exists \((\Gamma ,L)\subseteq (C^*\times D^*)\setminus \{(0,0)\}\) such that for all \((y,z) \in G\text {-}ED^m(F,G)(x_0,y_0,z_0)(S-x_0)\setminus \{(0,0)\}\),

$$\begin{aligned} C=\{ y\in Y | \left\langle {c^*,y}\right\rangle \ge 0,\; \forall c^*\in \Gamma \}, \; D=\{ z\in Z | \left\langle {d^*,z}\right\rangle \ge 0,\; \forall d^*\in L\}, \end{aligned}$$
$$\begin{aligned}&\mathop {\sup }\limits _{(c^*,d^*) \in (\Gamma ,L)} \left\{ {{{\left\langle {c^*,0}\right\rangle + \left\langle {d^*,-z_0}\right\rangle } \over {\left\langle {c^*,e}\right\rangle + \left\langle {d^*,k}\right\rangle }}} \right\} = 0, \end{aligned}$$
(7)
$$\begin{aligned}&\mathop {\sup }\limits _{(c^*,d^*) \in (\Gamma ,L)} \left\{ {{{\left\langle {c^*,y}\right\rangle + \left\langle {d^*,z}\right\rangle } \over {\left\langle {c^*,e}\right\rangle + \left\langle {d^*,k}\right\rangle }}} \right\} > 0, \end{aligned}$$
(8)

then \((x_0,y_0)\) is a weak efficient solution of \(\mathrm{(SOP)}\).

Proof

  1. (i)

    It follows from Theorem 4.1(i) and Propositions 2.2, 2.3.

(ii) Suppose to the contrary, i.e., \((x_0,y_0)\) is not a weak efficient solution of (SOP). By Theorem 4.1(ii), there exists \((v,w)\in G\text {-}ED^m(F,G)(A-x_0)\setminus \{(0,0)\}\) with \((v,w)\in -(\mathrm{int}C\times D(z_0))\). So, there exist \(\lambda \ge 0\) and \(d\in D\) such that \(w = -\lambda d - \lambda z_0\). From (7), we have

$$\begin{aligned}&\mathop {\sup }\limits _{(c^*,d^*) \in (\Gamma ,L)} \left\{ {{{\left\langle {c^*,v}\right\rangle + \left\langle {d^*,-\lambda d - \lambda z_0}\right\rangle } \over {\left\langle {c^*,e}\right\rangle + \left\langle {d^*,k}\right\rangle }}} \right\} \\&\quad \le \mathop {\sup }\limits _{(c^*,d^*) \in (\Gamma ,L)} \left\{ {{{\left\langle {c^*,v}\right\rangle + \lambda \left\langle {d^*,-d}\right\rangle } \over {\left\langle {c^*,e}\right\rangle + \left\langle {d^*,k}\right\rangle }}} \right\} \\&\quad + \mathop {\sup }\limits _{(c^*,d^*) \in (\Gamma ,L)} \left\{ {{{\left\langle {c^*,0}\right\rangle + \lambda \left\langle {d^*,-z_0}\right\rangle } \over {\left\langle {c^*,e}\right\rangle + \left\langle {d^*,k}\right\rangle }}} \right\} \le 0. \end{aligned}$$

On the other hand, it follows from (8) that

$$\begin{aligned} \mathop {\sup }\limits _{(c^*,d^*) \in (\Gamma ,L)} \left\{ {{{\left\langle {c^*,v}\right\rangle + \left\langle {d^*,w}\right\rangle } \over {\left\langle {c^*,e}\right\rangle + \left\langle {d^*,k}\right\rangle }}} \right\} >0, \end{aligned}$$

which is a contradiction. \(\square \)