1 Introduction

It is known that scalar-valued composite optimization model is very important in both theory and methodology, it provides a unified framework for studying convergence behaviour of various algorithms and Lagrangian optimality conditions. The study of scalar-valued composite optimization model has recently received a great deal of attention in the literature, see e.g. ( [4, 10, 15, 17, 25, 27] and the references therein).

However, we are rarely asked to make decisions based on only one criterion; most often, decisions are based on several conflicting criteria. Multiobjective optimization model provides the mathematical framework to deal with these situations, there is no doubt that it is a powerful tool in decision analysis. Moreover, it also has found a lot of significant applications in the other fields such as economics, management science and engineering design. Many papers have been published to study optimality conditions, duality theory and topological properties of solution sets of multiobjective optimization problems (see, e.g., [57, 9, 12, 14, 19, 24]).

Composite multiobjective optimization model is broad and flexible enough to cover many common types of multiobjective optimization problems, seen in the literature. Moreover, the model obviously includes the wide class of scalar-valued composite optimization problems, which is now recognized as fundamental for theory and computation in scalar nonsmooth optimization. Recently, some investigations for composite multiobjective optimization models has been proposed in following papers: Jeyakumar and Yang [1618, 26] investigated some first and second order optimality conditions for both nonsmooth and smooth convex composite multi-objective optimization problems, they also obtained some duality results for the problems even when the objective functions were not cone convex. Reddy and Mukherjee studied some first order optimality conditions for a class of composite multiobjective optimization problem with \(V-\rho \)-invexity in [21]. Bot et al. [3] also obtained some conjugate duality results for multiobjective composed optimization problems. It is worth noticing that there are fewer results for the composite multiobjective optimization problems since the complexity of objective functions and the variety of solution sets. Furthermore, to the best of our knowledge, there is no numerical method has be designed for solving composite multiobjective optimization problems, even no conceptual one.

In this paper, we consider the following extended-valued composite multiobjective optimization problem:

$$\begin{aligned} \text{(}CMOP\text{)} \quad&\text{ Min}_C&F(Ax)\\&\text{ s.t.}&x\in S, \end{aligned}$$

where \(S\subset R^n\) is closed and convex. The outer function \(F: R^l\rightarrow R^m\cup \{+\infty _C\}\) is a vector-valued function, \(F_i\) is the \(i\)th components of F, denote by \(domF_i\) the effective domain of \(F_i\), i.e. \(domF_i=\{x\in R^l| F_i(x)<+\infty \}\). The inner function \(A: S\rightarrow R^l\) is a \(l\times n\) matrix such that \(A(S)\subset \cap _{i=1}^m dom{F}_i\). Denote by r(A) and \(A^\top \), the rank and the transpose of the matrix \(A\), respectively. To illustrate the nature of the model \((CMOP)\), let us look at an example.

Example 1.1

Consider the vector approximation (model) problem:

$$\begin{aligned}&\text{ Min}_C ~ (\Vert A_1(x)\Vert _1,\ldots , \Vert A_m(x)\Vert _m)\\&\quad \text{ s.t.} ~ x\in S, \end{aligned}$$

where \(S\subset R^n\) is closed and convex. \(\Vert .\Vert _i, i=1, 2,\ldots , m\) is a norm in \(R^l\), and for each \(i=1, 2,\ldots , m, A_i(x)\) is a \(l\times n\) matrix. Various examples of vector approximation problems of this type that arise in simultaneous approximation are given in [13, 14].

The idea is that by studying the composite model problem \((CMOP)\) a unified framework can be given for the treatment of many questions of theoretical and computational interest in multiobjective optimization. The motivation of this paper is to consider how to design a iterative algorithm for computing the model \((CMOP)\) via asymptotic analysis. Although the inner function \(A(x)\) is linear, the composite structure \(F(A(x))\) captures some elementary characterizations of composite optimization. On the other hand, there are some technical difficulties in computing asymptotic function and subdifferential of a vector-valued composite function, when the inner function is not linear.

The paper is organized as follows. In Sect. 2, we present some concepts, basic assumptions and preliminary results. In Sect. 3, we we characterize the nonemptiness and compactness of the weak Pareto optimal solution set of the problem (CMOP). In Sect. 4, we employ the obtained results to construct a class of proximal-type method for solving the problem (CMOP), convergence analysis is made under some mild conditions. In Sect. 5, we draw some conclusions.

2 Preliminaries

In this section, we introduce various notions of Pareto optimal solutions and present some preliminary results that will be used throughout this paper.

Let \(C=R^m_+\subset R^m\) and \(C_1=\{x\in R^m_+| ~ \Vert x\Vert =1\}\). We define, for any \(y_1, y_2\in R^m\),

$$\begin{aligned}&y_1\le _C y_2 \text{ if} \text{ and} \text{ only} \text{ if} y_2-y_1\in C;\end{aligned}$$
(2.1)
$$\begin{aligned}&y_1\not \le _{intC} y_2 \text{ if} \text{ and} \text{ only} \text{ if} y_2-y_1\not \in int C. \end{aligned}$$
(2.2)

The extended space of \(R^m \) is \(\bar{R}^m= R^m\cup \{-\infty _C, +\infty _C\}\), where \(-\infty _C\) is an imaginary point, each of the coordinates is \(-\infty \) and the imaginary point \(+\infty _C\) is analogously understood (with the conventions \(\infty _C+\infty _C=\infty _C, \mu (+\infty _C)=+\infty _C\) for each positive number \(\mu \)). The point \(y\in R^m\) is a column vector and its transpose is denoted by \(y^\top \). The inner product in \(R^m\) is denoted by \(\langle \cdot , \cdot \rangle \).

It is worth noticing that the binary relation \(\not \le _{intC}\) is closed in the sense that if \(x_k\rightarrow x^*\) as \(k \rightarrow \infty \), \(x_k \not \le _{intC} 0 \), then we have \(x^* \not \le _{intC} 0\). This is because of the closeness of the set \(W= : R^m\setminus \{-intC\}\).

Definition 2.1

[5] Let \(K\subset R^n\) be convex and a map \(F: K\rightarrow R^m\cup \{+\infty _C\}\) is said to be C-convex if

$$\begin{aligned} F((1-\lambda )x+\lambda y)\le _C (1-\lambda )F(x)+\lambda F(y) \end{aligned}$$

for any \(x, y\in K\) and \(\lambda \in [0, 1]. F\) is said to be strictly C-convex if

$$\begin{aligned} F((1-\lambda )x+\lambda y)\le _{int C} (1-\lambda )F(x)+\lambda F(y) \end{aligned}$$

for any \(x, y\in K\) with \(x\not =y\) and \(\lambda \in (0, 1)\).

Definition 2.2

[11] A map \(F: K\subset R^n\rightarrow R^m\cup \{+\infty _C\}\) is said to be C-lsc at \(x_0\in K\) if, for any neighborhood \(V\) of \(F(x_0)\) in \(R^m\), there exists a neighborhood \(U\) of \(x_0\) in \(R^n\) such that \(F(U\cap K)\subseteq V+C\). The map \(F: K\subset R^n\rightarrow R^m\cup \{+\infty _C\}\) is said to be C-lsc on \(K\) if it is C-lsc at every point \(x_0\in K\).

Remark 2.1

In fact, when \(C=R^m_+\), the \(R^m_+\)-lower semicontinuity of \(F=(F_1,\ldots , F_m)\) is equivalent to the (usual) lower semicontinuity of each \(F_i\).

Definition 2.3

[5] Let \(K\subset R^n\) be convex and \(F: K\rightarrow R^m\cup \{+\infty _C\}\) be a vector-valued function. \(x^*\in K\) is said to be a Pareto optimal solution of \(F\) on K if

$$\begin{aligned} (F(K)-F(x^*))\cap (-C\backslash \{0\})=\emptyset , \end{aligned}$$

\(x^*\in K\) is said to be a weak Pareto optimal solution of \(F\) on \(K\) if

$$\begin{aligned} (F(K)-F(x^*))\cap (-intC)=\emptyset , \end{aligned}$$

\(x^*\in K\) is said to be an ideal optimal solution of \(F\) on \(K\) if

$$\begin{aligned} F(x)-F(x^*)\in C, \quad \forall x\in K. \end{aligned}$$

Lemma 2.1

[11] Let \(K\subset R^n\) be a closed set, and suppose that \(W\subset R^m\) is a closed set such that \(W+C\subseteq W\). Assume that \(F: K\rightarrow R^m\cup \{+\infty _C\}\) is C-lsc. Then, the set \(P=\{x\in K |~F(x)-\lambda \in -W\}\) is closed for all \(\lambda \in R^m\).

Definition 2.4

[1] Let K be a nonempty set in \(R^n\). Then the asymptotic cone of the set \(K\), denoted by \(K^\infty \), is the set of all vectors \(d\in R^n\) that are limits in the direction of the sequence \(\{x_k\}\subset K\), namely

$$\begin{aligned} K^\infty =\left\{ d\in R^n|~ \exists t_k\rightarrow +\infty , ~and~ x_k\in K, \lim \limits _{k\rightarrow +\infty }\frac{x_k}{t_k}=d\right\} . \end{aligned}$$
(2.3)

In the case that \(K\) is convex and closed, then, for any \( x_0\in K\),

$$\begin{aligned} K^\infty =\{d\in R^n | ~x_0+td\in K, \forall ~ t>0\}. \end{aligned}$$
(2.4)

Lemma 2.2

[1] A set \(K\subset R^n\) is bounded if and only if its asymptotic cone is just the zero cone: \(K^\infty =\{0\}\).

Definition 2.5

[1] For any given function \(f: R^n\rightarrow R\cup \{+\infty \}\), the asymptotic function of \(f\) is defined as the function \(f^\infty \) such that \(epi f^\infty =(epi f)^\infty \), where \(epi f=\{(x, t)\in R^n\times R | f(x)\le t\}\) is the epigraph of \(f\). Consequently, we can give the analytic representation of the asymptotic function \(f^\infty \):

$$\begin{aligned} f^\infty (d) =\inf \left\{ ~\liminf \limits _{k\rightarrow +\infty }\frac{f(t_k d_k)}{t_k}: t_k\rightarrow +\infty , d_k\rightarrow d\right\} . \end{aligned}$$
(2.5)

When \(f\) is a proper convex and lower semi-continuous (lsc in short) function, we have

$$\begin{aligned} f^\infty (d) =\sup \{ f(x+d)-f(x)|~ x\in domf\} \end{aligned}$$
(2.6)

or equivalently

$$\begin{aligned} f^\infty (d) =\lim \limits _{t\rightarrow +\infty }\frac{f(x+td)-f(x)}{t}=\sup \limits _{t>0}\frac{f(x+td)-f(x)}{t}, ~~\forall ~d\in domf \end{aligned}$$
(2.7)

and

$$\begin{aligned} f^\infty (d) =\lim \limits _{t\rightarrow 0^+}tf(t^{-1}d), ~~\forall ~ d\in domf. \end{aligned}$$
(2.8)

For the indicator function \(\delta _K\), we have that \(\delta ^\infty _K=\delta _{K^\infty }\), where \(K\subset R^n\) is a nonempty set.

Definition 2.6

[23] The function \(f: R^n\rightarrow R\cup \{+\infty \}\) is said to be coercive if its asymptotic function \(f^\infty (d)>0\), for all \( d\ne 0\in R^n\) and it is said to be counter-coercive if its asymptotic function \(f^\infty (d)=-\infty \), for some \( d\ne 0\in R^n\).

Lemma 2.3

[1] Let \(f: R^n\rightarrow R\cup \{+\infty \}\) be proper convex and lower semicontinuous, then the following three statement are equivalent:

  1. (a)

    f is coercive;

  2. (b)

    the optimal set \(\{x\in R^n |~ f(x)=\inf f\}\) is nonempty and compact;

  3. (c)

    \(\lim \limits _{\Vert x\Vert \rightarrow +\infty }\inf \frac{f(x)}{\Vert x\Vert }>0.\)

Definition 2.7

[19] A cone \(C_2\subseteq R^m\) is called \(Daniell\) if any decreasing sequence of \(R^m\) having a lower bound converges to its infimum. For example, the cone \(C=R^m_+\) has the Daniell property.

Definition 2.8

[24] A set \(S\subset R^m\) is said to have the domination property with respect to \(C\), if there exists \(s\in R^m\) such that \(S\subseteq s+C\).

Let \(H: R^n\rightarrow R^m\cup \{+\infty _C\}\) be a vector-valued function, denote by \(EpiF\) the epigraph of \(H\), i.e.

$$\begin{aligned} EpiH=\{(x, y)\in R^n\times R^m| y\in H(x)+C\}. \end{aligned}$$

Similarly, we define the asymptotic function of a vector-valued function.

Definition 2.9

For any given vector-valued function \(H: R^n\rightarrow R^m\cup \{+\infty _C\}\), the asymptotic function of H is defined as the function \(H^\infty : R^n\rightarrow R^m\cup \{+\infty _C\}\) such that

$$\begin{aligned} EpiH^\infty =(EpiH)^\infty . \end{aligned}$$

Proposition 2.1

Let \(H: R^n\rightarrow R^m\cup \{+\infty _C\}\) be a proper, C-lsc and C-convex vector-valued function. One has

$$\begin{aligned} H^\infty (d)=\lim \limits _{t\rightarrow \infty }\frac{H(x+td)-H(x)}{t},\quad \forall d \in domH^\infty , \end{aligned}$$
(2.9)

where x is any vector in \(domH\).

Proof

From the \(C\)-convexity of \(H\), we know that the set \(EpiH\) is also convex in \(R^n\times R^m\). By the definition of the asymptotic cones of \(EpiH\), one has that for any \(x\in domH\),

$$\begin{aligned} (EpiH)^\infty =\{(d,u)\in R^n\times R^m|(x,H(x))+t(d,u)\in EpiH, \forall t>0\}. \end{aligned}$$
(2.10)

That is for each \((d,u)\in (EpiH)^\infty \) if and only if for any \(x\in domH\), we have

$$\begin{aligned} H(x+td)\le _CH(x)+tu, \forall t>0. \end{aligned}$$
(2.11)

From the inequality (2.11), we define a new function \(T: R^n\rightarrow R^m\cup \{+\infty _C\}\):

$$\begin{aligned} T(d)=\sup _{t>0}\frac{H(x+td)-H(x)}{t}\le _C u \end{aligned}$$
(2.12)

and hence

$$\begin{aligned} (EpiH)^\infty =EpiT,\forall ~x\in domH. \end{aligned}$$

On the other side, for any fixed \(x,d\in R^n\) and \(i\in \{1,\ldots ,m\}\), the function \(\frac{H_i(x+td)-H_i(x)}{t} \) is nondecreasing with \(t>0\). Thus we obtain that

$$\begin{aligned} \lim \limits _{t\rightarrow +\infty }\frac{H_i(x+td)-H_i(x)}{t}=u_i, \forall ~ i\in \{1,\ldots , m\}, \end{aligned}$$

and

$$\begin{aligned} \lim \limits _{t\rightarrow +\infty }\frac{H(x+td)-H(x)}{t}=u. \end{aligned}$$

That is

$$\begin{aligned} H^\infty (d)=\lim \limits _{t\rightarrow \infty }\frac{H(x+td)-H(x)}{t}\quad \forall ~ d \in domH^\infty . \end{aligned}$$

The proof is complete.\(\square \)

Remark 2.2

From the statement of Proposition 2.1 and the formula 2.7, we have

$$\begin{aligned} H^\infty (d)=(H^\infty _1(d),\ldots ,H^\infty _m(d)). \end{aligned}$$

Proposition 2.2

Let \(F: R^l\rightarrow R^m\cup \{+\infty _C\}\) be a proper function, let \(A\) be a linear map from \(R^n\rightarrow R^l\) with \(A(R^n)\subset domF\), and let \(G(x)=F(Ax)\) be a proper composite function. If \(F\) is proper, \(C-lsc\) and \(C-convex\). Then, \(G\) is C-lsc and C-convex.

Proof

From Remark 2.1, we know that \(G\) is \(C\)-lsc if and only if \(G_i\) is lsc for any \(i\in \{1,\ldots ,m\}\). Since \(A\) is a linear map and \(F_i\) is lsc for any fixed \(i\in \{1,\ldots ,m\}\). So \(G_i\) is lsc for any \(i\in \{1,\ldots ,m\}\), clearly \(G\) is \(C\)-lsc.

On the other side, let \(x_1, x_2\in S\) and \(\lambda \in [0,1]\). From the assumptions, we obtain

$$\begin{aligned} G(x_1)=F(Ax_1),\quad G(x_2)=F(Ax_2) \end{aligned}$$

and

$$\begin{aligned} G(\lambda x_1+(1-\lambda x_2))=F(A(\lambda x_1+(1-\lambda ) x_2)). \end{aligned}$$

By the \(C-convexity\) of \(F\), we derive

$$\begin{aligned} F(A(\lambda x_1\!+\!(1-\lambda ) x_2))\le _C\lambda F(Ax_1)\!+\!(1\!-\!\lambda )F(Ax_2)\!=\!\lambda G(x_1)\!+\!(1-\lambda )G(x_2).\qquad \end{aligned}$$
(2.13)

That is \(G\) is \(C\)-convex. The proof is complete. \(\square \)

Proposition 2.3

Let the same assumptions as in Proposition 2.2 hold. Then,

$$\begin{aligned} G^\infty (d) =F^\infty (Ad), \quad \forall d\in R^n. \end{aligned}$$
(2.14)

Proof

The proof of Proposition 2.3 is a little bit trivial, so we omit it.

Throughout this paper. we denote by \(\bar{X}\) the weak Pareto optimal solutions set and \(X^*\) the ideal solutions set of problem (CMOP), respectively.

3 Characterizations of weak Pareto solution optimal sets

We denote by

$$\begin{aligned} S_1=\cap _{y\in S }\{u\in S^\infty | F(A(\lambda u+y))-F(Ay)\in -W, \forall \lambda >0\} \end{aligned}$$

and

$$\begin{aligned} S_2=\cap _{y\in S }\{x\in S| F(Ax)-F(Ay)\in -W\}^\infty . \end{aligned}$$

Theorem 3.1

In problem (CMOP), suppose that \(F\) is proper, \(C\)-lsc and \(C\)-convex. If \(\bar{X}\ne \emptyset \), then

$$\begin{aligned} \bar{X}^\infty \subseteq S_1\subseteq S_2. \end{aligned}$$
(3.1)

Furthermore, if the ideal solution set \(X^*\) is nonempty, then \(S_1=\bar{X}^\infty \).

Proof

(1) Taking any \(u\in \bar{X}^\infty \), from the definition of asymptotic cone, we have there exist some sequences \(\{x_k\}\subset \bar{X}\) and \(\{t_k\}\) with \(t_k\rightarrow +\infty \) such that \(\mathop {\text{ lim}}\limits _{k\rightarrow +\infty } \frac{x_k}{ t_k}= u\). By the fact of \(x_k\in \bar{X}\), one has

$$\begin{aligned} F(Ay)-F(Ax_k)\in W, \forall y\in S. \end{aligned}$$
(3.2)

For each \(y\in S\), we have

$$\begin{aligned} F(Ay)\in F(Ay)-C. \end{aligned}$$
(3.3)

By virtue of Proposition 2.2, for any fixed \(\lambda >0\), we have

$$\begin{aligned} F\left(A\left(\left(1-\frac{\lambda }{t_k}\right)y+\frac{\lambda }{t_k}x_k\right)\right)\le _C \left(1-\frac{\lambda }{t_k}\right)F(Ay)+\frac{\lambda }{t_k}F(Ax_k), \end{aligned}$$

when \(t_k\) is sufficiently large. That is

$$\begin{aligned} F\left(A\left(\left(1-\frac{\lambda }{t_k}\right)y+\frac{\lambda }{t_k}x_k\right)\right)\in \left(1-\frac{\lambda }{t_k}\right) F(Ay)+\frac{\lambda }{t_k}F(Ax_k)-C. \end{aligned}$$
(3.4)

From the inequality (2.8), one has

$$\begin{aligned} \frac{\lambda }{t_k}F(Ax_k)\in \frac{\lambda }{t_k}F(Ay)-W \end{aligned}$$
(3.5)

and by the formula (3.3), we derive

$$\begin{aligned} \left(1-\frac{\lambda }{t_k}\right)F(Ay)\in \left(1-\frac{\lambda }{t_k}\right)F(Ay)-C. \end{aligned}$$
(3.6)

Hence, combining (3.4), (3.5) with (3.6), we obtain

$$\begin{aligned} F\left(A\left(\left(1-\frac{\lambda }{t_k}\right)y+\frac{\lambda }{t_k}x_k\right)\right)\in F(Ay)-W. \end{aligned}$$
(3.7)

Taking the limit in (3.7) as \(k\rightarrow +\infty \), we derive

$$\begin{aligned} F(A(y+\lambda u))\in F(Ay)-W. \end{aligned}$$

That is \(u\in S_1\).

(2). Taking any \(d\in S_1\), which means that for any \(x\in S\), we have

$$\begin{aligned} F(Ax)-F(A(x+\lambda d))\in W, \quad \forall \lambda >0. \end{aligned}$$
(3.8)

Without lose of generality, we assume \(x_k=x+\lambda _kd\), where \(\lambda _k\rightarrow +\infty \). From the inclusion (3.8), we have

$$\begin{aligned} F(Ax)-F(Ax_k)\in W. \end{aligned}$$
(3.9)

Choosing \(t_k=\lambda _k\), we obtain

$$\begin{aligned} \mathop {\text{ lim}}\limits _{k\rightarrow +\infty } \frac{x_k}{t_k}=\mathop {\text{ lim}}\limits _{k\rightarrow +\infty }\frac{\bar{x}+\lambda _kd}{\lambda _k}= d. \end{aligned}$$

That is \(d\in S_2\).

(3). Taking any \(d\in S_1\), by the assumption that \(X^*\) is nonempty, we have

$$\begin{aligned} \bar{x}+t_kd\in S, \quad \forall t_k>0, \end{aligned}$$
(3.10)

where \(\bar{x}\in X^*\) is fixed and \(t_k\rightarrow +\infty \). Taking any \(y\in S\), it is easy to check that

$$\begin{aligned} F(Ay)-F(A(\bar{x}+t_kd))=F(Ay)-F(A\bar{x})+F(A\bar{x})-F(A(\bar{x}+t_kd)). \end{aligned}$$
(3.11)

From the definition of \(X^*\), we have

$$\begin{aligned} F(Ay)-F(A\bar{x})\in C, \forall x\in S \end{aligned}$$
(3.12)

and by the definition of \(S_1\), we obtain

$$\begin{aligned} F(A\bar{x})-F(A(\bar{x}+t_kd))\in W. \end{aligned}$$
(3.13)

Combining (3.12) with (3.13), we have

$$\begin{aligned} F(Ay)-F(A(\bar{x}+t_kd))\in W,\quad \forall y\in S. \end{aligned}$$
(3.14)

The formula (3.14) means that \(\bar{x}+t_kd\in \bar{X}\). We denote by \(x_k=\bar{x}+t_kd\) and it follows that

$$\begin{aligned} \mathop {\text{ lim}}\limits _{k\rightarrow +\infty } \frac{x_k}{t_k}=\mathop {\text{ lim}}\limits _{k\rightarrow +\infty }\frac{\bar{x}+t_kd}{t_k}= d. \end{aligned}$$

Hence, we conclude \(d\in \bar{X}^\infty \). The proof is complete.\(\square \)

Remark 3.1

When \(A=I\) is an identical mapping, some corresponding results have obtained in [8, 11].

Next let’s consider some necessary and sufficient conditions for the nonemptiness and compactness of weak Pareto optimal solution sets in the problem (CMOP).

Lemma 3.1

In the problem (CMOP), we assume \(F\) is proper, C-lsc and C-convex. Then we have \(\bar{X}\) is nonempty and compact if and only if

$$\begin{aligned} S^\infty \bigcap \cup _{j=1}^{m}\{d\in R^n| F_j^\infty (Ad)\le 0\}=\{0\} \end{aligned}$$
(3.15)

Proof

Denote by \(argmin_SF_j\) the solution set of the following scalar-valued optimization problem:

$$\begin{aligned} \quad&\text{ min}&F_j(Ax)\\&\text{ s.t.}&x\in S, \end{aligned}$$

where \(j\in [1,\ldots ,m]\). By virtue of Theorem 2.1 of [8], one has that \(\bar{X}\) is nonempty and compact if and only if \(argmin_SF_j\) is nonempty and compact for any \(j\in [1,..,m]\). We observe that the nonemptiness and compactness of \(argmin_SF_j\) is equivalent to

$$\begin{aligned} S^\infty \cap \{d\in R^n| F_j^\infty (Ad)\le 0\}=\{0\} \end{aligned}$$
(3.16)

for each \(j\in [1,..,m]\), see the Theorem 27.1 of [22]. Thus, we obtain the formula (3.15) is equivalent to \(\bar{X}\) is nonempty and compact. The proof is complete.\(\square \)

Remark 3.2

The linearity of \(A(x)\) makes it possible to obtain the analytical expression of the asymptotic function of the vector-valued function (Proposition 2.3). By virtue of the analytical expression of the asymptotic function, Lemma 3.1 generalizes some corresponding results of [8].

4 Proximal-type method for convex composite multiobjective optimization problem

It is known that the following constrained multiobjective optimization problem

$$\begin{aligned} Min_C\{F(Ax)|~ x\in S\}\quad (CMOP) \end{aligned}$$

is equivalent to the unconstrained multiobjective optimization problem

$$\begin{aligned} in_C\{F_0(Ax)|x\in R^n\}\quad (MOP) \end{aligned}$$

in the sense that they have the same sets of Pareto optimal solutions and the same sets of weak Pareto optimal solutions, where

$$\begin{aligned} F_0(Ax)=\left\{ \begin{array}{ll} F(Ax),&\text{ if} x\in S;\\ +\infty _C,&\text{ if} x\notin S.\end{array}\right. \end{aligned}$$

Lemma 4.1

If \(K\subset R^n\) is a convex set and \(F_1: K\rightarrow R^m\cup \{+\infty _C\}\) is a proper \(C\)-convex map, then

$$\begin{aligned} C-ARGMIN_w \{F_1(x)\mid x\in K\}=\bigcup _{z\in C_1}argmin \{\langle F_1(x), z\rangle \mid x\in K\} \end{aligned}$$

where \(C-ARGMIN_w \{F_1(x)\mid x\in K\}\) is the weak Pareto optimal solution set of \(F_1\) on \(K\).

This follows immediately from Theorem 2.1 in [2].

Now we make the following assumption:

(A) the set \(\bar{X}\) is nonempty and compact.

Here we propose the following vector-valued proximal-type method (VPM, in short):

  • Step (1): Taking any \(x_0\in R^n\);

  • Step (2): Given \(x_k\), if \(x_k\in \bar{X}\), then \(x_{k+p}=x_k\) for all \(p\ge 1\) and the algorithm stops, otherwise goes to step (3).

  • Step (3): If \(x_k\notin \bar{X}\), then compute \(x_{k+1}\) satisfying

$$\begin{aligned} x_{k+1}\in C-ARGMIN_w\{F_0(Ax)+ \frac{\varepsilon _k}{2} \Vert x-x_k \Vert ^2e_k\mid x\in \theta _k\} \end{aligned}$$
(4.1)

where \(\theta _k:=\{x\in R^n| F_0(Ax)\le _CF_0(Ax_k)\}\), \(\varepsilon _k\in (0, \varepsilon ], \varepsilon >0\) and goes to step (2).

Next we will establish the main results in this section.

Theorem 4.1

In the problem \((MOP)\), let \(F_0: R^l\rightarrow R^m\cup \{+\infty _C\}\) be proper \(C\)-convex and \(C\)-lower semicontinuous mapping. Further suppose that \(r(A)=n\). Under the assumption (A), any sequence \(\{x_k\}\) generated by the method (VPM) is well-defined and bounded.

Proof

Let \(x_0\in R^n\) be an initial point and we assume the algorithm has reached step \(k\). We will show that the next iterative \(x_{k+1}\) does exist. Defining a new function \(T_k(x): R^n\rightarrow R\cup \{+\infty \}\) with

$$\begin{aligned} T_k(x)=\langle F_0(Ax), \lambda \rangle +\frac{\varepsilon _k}{2} \Vert x-x_k \Vert ^2\langle e_k, \lambda \rangle +\delta _{\theta _k}(x) \end{aligned}$$

where \(\lambda \in C_1\) and \(\delta _{\theta _k}(x)\) is the indicator function of set \(\theta _k\). Denote by \(\bar{X}_k\) the solution set of the following scalar-valued optimization problem:

$$\begin{aligned} \min \{T_k(x)|~ x\in R^n\}\quad (MOP_k). \end{aligned}$$

It is clear that \(\theta _k\) is a nonempty and convex set by its definition. From the assumptions that \(F_0\) is \(C\)-lsc and \(C\)-convex, we know that for any \(\lambda \in C_1\), the function \(\langle F_0(Ax), \lambda \rangle +\delta _{\theta _k}(x)\) is convex and lsc with respect to \(x\). That is \(T_k(x)\) is lsc and convex. By the assumption (A) and the virtue of Lemma 2.3, we have

$$\begin{aligned} (\lambda F_0)^\infty (Ad)>0,\quad \forall d\in R^n ~{ and}~ Ad\ne 0. \end{aligned}$$
(4.2)

From the assumption that \(r(A)=n\), we have the following inequality

$$\begin{aligned} (\lambda F_0)^\infty (Ad)>0,\quad \forall d\ne 0 \in R^n. \end{aligned}$$
(4.3)

From the definition of an indicator function, we know that

$$\begin{aligned} {{\delta _{\theta _k}}}^\infty (d)={{\delta _{{\theta _k}^\infty }}}(d)= \left\{ \begin{array}{ll}0,&\text{ if} d\in {\theta _k}^\infty ;\\ +\infty ,&\text{ if} d\notin {\theta _k}^\infty .\end{array}\right. \end{aligned}$$
(4.4)

Thus, combining (4.3) with (4.4), we obtain that

$$\begin{aligned} (\lambda F_0)^\infty (Ad)+{{\delta _{\theta _k}}}^\infty (d)>0, \quad \forall d\ne 0\in R^n. \end{aligned}$$
(4.5)

On the other side, by the fact of \(\{e_k\}\subset R^m_+\) and the definition of \(\lambda \), we have \(\langle e_k, \lambda \rangle >0\). And by the formula (2.8), we obtain

$$\begin{aligned} (\Vert d-x_k\Vert ^2)^\infty =\lim \limits _{t\rightarrow 0^+}t\left\Vert\frac{d}{t}-x_k\right\Vert^2=\lim \limits _{t\rightarrow 0^+}\frac{\Vert d-x_k\Vert ^2}{t}\ge 0, \quad \forall d\ne 0\in R^n. \end{aligned}$$
(4.6)

From the Proposition 2.6.1 of [1], we derive that

$$\begin{aligned} T_k^\infty (d)=(\lambda F_0)^\infty (Ad)+{{\delta _{\theta _k}}}^\infty (d)+ \frac{\varepsilon _k}{2}\langle e_k, \lambda \rangle (\Vert d-x_k\Vert ^2)^\infty \end{aligned}$$
(4.7)

Combining (4.4), (4.5) with (4.6), we obtain that

$$\begin{aligned} T_k^\infty (d)>0, \quad \forall d\ne 0\in R^n. \end{aligned}$$

By the virtue of Lemma 2.3, we conclude that the set \(\bar{X}_k\) is nonempty and compact for every \(k\in N\). From Lemma 4.1, we have a minimizer of \(T_k(x)\) satisfies 4.1 and can be taken as \(x_{k+1}\).

Next we will show that the sequence \(\{x_k\}\) is bounded as \(k\rightarrow +\infty \). Let’s consider its contrary, assume \(\Vert x_k\Vert =+\infty \) as \(k\rightarrow +\infty \). From the formula (4.2), we know that the function \(\langle F_0(Ax), \lambda \rangle \) is coercive. From the statements (c) of Lemma 2.3, we have

$$\begin{aligned} \lim \limits _{\Vert x_k\Vert \rightarrow +\infty }\inf \frac{\langle F_0(Ax_k), \lambda \rangle }{\Vert x_k\Vert }>0. \end{aligned}$$
(4.8)

However by the definition of the method (VPM), we know

$$\begin{aligned} \lim \limits _{\Vert x_k\Vert \rightarrow +\infty }\inf \frac{\langle F_0(Ax_k), \lambda \rangle }{\Vert x_k\Vert }\le \lim \limits _{\Vert x_k\Vert \rightarrow +\infty }\inf \frac{\langle F_0(Ax_0), \lambda \rangle }{\Vert x_k\Vert }=0 \end{aligned}$$
(4.9)

a contradiction with (4.8). Thus the sequence \(\{x_k\}\) is bounded. The proof is complete.\(\square \)

Remark 4.1

The main statements of Theorem 4.1 are concerned with the existence and the boundedness of sequences. Compared with some corresponding results in [2], our contributions are that we present a quite different method to prove the existence of iterates and the boundedness of sequences via asymptotic analysis. It is worth noticing that when the regular term in (4.1) is not quadratic, the traditional method does not deal with such complex cases. However, the method in this paper does still work.

Lemma 4.2

Let the assumptions in Theorem 4.1 hold and suppose that \(F_0(A(R^n))\) have the domination property. Then, we have

$$\begin{aligned} \lim \limits _{k\rightarrow +\infty }\Vert x_k-x_{k+1}\Vert =0. \end{aligned}$$
(4.10)

Proof

From the method (VPM), we know that if the sequence stops at some iteration, \(x_k\) will be a constant vector thereafter. Now we assume that the sequence \(\{x_k\}\) will not stop finitely. Define \(E\subset R^n\) as follows

$$\begin{aligned} E=\{x\in R^n\mid F_0(Ax)\le _C F_0(Ax_k)\quad \forall k\in N\}. \end{aligned}$$

By the assumption that \(F_0(A(R^n))\) has the domination property, it follows from the Daniell property of \(R^m_+\) that we have E is nonempty. Since \(x_{k+1}\) is a weak Pareto optimal solution of problem (4.1), there exists a \(\lambda _k\in C_1\) such that \(x_{k+1}\) is the solution of the following problem \((MOP_{\lambda _k})\):

$$\begin{aligned} \min \{T_{\lambda _ k}(x)|~ x\in R^n\}, \end{aligned}$$

where \(T_k(x)=\langle F_0(Ax), \lambda _k\rangle +\frac{\varepsilon _k}{2} \Vert x-x_k \Vert ^2\langle e_k, \lambda _k\rangle +\delta _{\theta _k}(x)\) and \(\delta _{\theta _k}(x)\) is the scalar-valued indicator function. Thus \(x_{k+1}\) satisfies the first-order necessary optimality condition of problem \((MOP_{\lambda _k}\!)\). It follows from Theorem 3.23 of [20] that there exists \(\mu _k\in \partial \langle F_0(.), \lambda _k\rangle (Ax_{k+1})\) and \(\nu _k\in \partial \delta _{\theta _k}(x_{k+1})\) such that

$$\begin{aligned} A^{\top }\mu _k+{\varepsilon _k}\langle e_k, \lambda _k\rangle (x_{k+1}-x_k)+\nu _k=0. \end{aligned}$$

Denote by \(\alpha _k={\varepsilon _k}\langle e_k, \lambda _k\rangle \), obviously the sequence \(\alpha _k>0\) for all \(k\in N\). By the fact that \(\langle \nu _k, x-x_{k+1}\rangle \le 0\) for any \(x\in \theta _k\), we have that

$$\begin{aligned} \langle A^{\top }\mu _k+\alpha _k (x_{k+1}-x_k), x-x_{k+1}\rangle \ge 0 \quad \forall x\in \theta _k. \end{aligned}$$

Let \(x^*\in E\). It is obvious that \(x^*\in \theta _k\) for all \(k\in N\) and we deduce that

$$\begin{aligned} \langle A^{\top }\mu _k+\alpha _k (x_{k+1}-x_k), x^*-x_{k+1}\rangle \ge 0. \end{aligned}$$
(4.11)

By the definition of subgradient of \(\langle F_0(Ax_{k+1}), \lambda _k\rangle \), we have that

$$\begin{aligned} \langle F_0(Ax^*)-F_0(Ax_{k+1}), \lambda _k\rangle \ge \langle \mu _k, Ax^*- Ax_{k+1} \rangle . \end{aligned}$$

From the fact that \(x^*\in \theta _k\) for all \(k\in N\), we have \(\langle F_0(Ax^*)-F_0(Ax_{k+1}), \lambda _k\rangle \le 0\). It follows that

$$\begin{aligned} \langle A^{\top }\mu _k, x^*- x_{k+1}\rangle \le 0 \end{aligned}$$
(4.12)

and

$$\begin{aligned} 0\le \alpha _k \langle x_{k+1}\!-\!x_k, x^*-x_{k+1}\rangle \!=\!\alpha _k (\Vert x_k-x^*\Vert ^2\!-\!\Vert x_{k+1}\!-\!x^*\Vert ^2\!-\!\Vert x_k-x_{k+1}\Vert ^2).\qquad \end{aligned}$$
(4.13)

Combining the inequality (4.13) and the fact of \(\alpha _k>0\), we obtain that

$$\begin{aligned} \Vert x_k-x_{k+1}\Vert ^2\le \Vert x_k-x^*\Vert ^2-\Vert x_{k+1}-x^*\Vert ^2 \end{aligned}$$
(4.14)

From the statements of Theorem 4.1, we have the sequence \(\{\Vert x_k-x^*\Vert ^2\}\) is bounded, furthermore by the inequality (4.14), we know \(\{\Vert x_k-x^*\Vert ^2\}\) is a nonnegative and nonincreasing sequence, and hence is convergent. We conclude that

$$\begin{aligned} \lim \limits _{k\rightarrow +\infty }\Vert x_k-x_{k+1}\Vert =0. \end{aligned}$$
(4.15)

The proof is complete.\(\square \)

Theorem 4.2

Let the assumptions in Theorem 4.1 and Lemma 4.2 hold. Then any cluster point of \(\{x_k\}\) belongs to \(\bar{X}\).

Proof

If there exists \(k_0 \ge 1\) such that \(x_{k_0+p}=x_{k_0},\forall p\ge 1.\) Then, it is obvious that \(x_{k_0}\) is a cluster point of \(\{ x_k\}\) and it is also a weak Pareto optimal solution of problem (MOP). Now suppose that the algorithm does not terminate finitely. Then, by Theorem 4.1, we have that \(\{x_k\}\) is bounded and it has some cluster points. Next we will show that all of cluster points are weak Pareto optimal solutions of problem (MOP). Let \(\hat{x}\) be one of the cluster points of \(\{x_k\}\) and \(\{x_{k_j}\}\) be a subsequence of \(\{x_k\}\), which converges to \(\hat{x}\). Let \(\lambda \in C_1\). We define a function \(\psi _\lambda : S\rightarrow R\cup {+\infty }\) as \(\phi _\lambda (x)= \langle F_0(Ax), \lambda \rangle \). Since \(F_0\) is C-lsc and \(C\)-convex, \(\psi _\lambda \) is also lsc and convex, it follows that \(\psi _\lambda (\bar{x})\le \liminf \limits _{j\rightarrow +\infty } \psi _\lambda (x_{k_j})\). By the fact that \(x_{k+1}\in \theta _k\), we can see that \(F_0(Ax_{k+1})\le _C F_0(Ax_k)\) for \(k\in N\). Thus, \(\psi _\lambda (x_{k+1})\le \psi _\lambda (x_{k})\). Therefore,

$$\begin{aligned} \psi _\lambda (\bar{x})\le \liminf \limits _{j\rightarrow +\infty } \psi _\lambda (x_{k_j})= \inf \{\psi _\lambda (x_{k})\}. \end{aligned}$$

Hence, we have that \(\psi _\lambda (\bar{x})\le \psi _\lambda (x_{k})\), which implies \(F_0(A\bar{x})\le _C F_0(Ax_k)\). Assume that \(\bar{x}\) is not the weak Pareto optimal solution of problem (MOP), then there exists \(x^*\in R^n\) such that \(F(Ax^*)\le _{intC}F(A\bar{x})\). Taking \(\lambda _{k_j}\in R^m_+\) as a subsequence of \(\lambda _k\), obviously there exists \(\bar{\lambda }\in R^m_+\) such that \(\bar{\lambda }\) is a cluster point of \(\{\lambda _{k_j}\}\). Without loss of generality, we assume that

$$\begin{aligned} \lim \limits _{j\rightarrow +\infty }\lambda _{k_j}=\bar{\lambda }. \end{aligned}$$

Thus we have that

$$\begin{aligned}&\langle F_0(A x^*) F_0(A\bar{x}), \lambda _{k_j}\rangle \ge \langle F_0(Ax^*)- F_0(Ax_{k_{j}+1}), \lambda _{k_j}\rangle \nonumber \\&\quad = \psi _{\lambda _ {k_j}}(x^*)-\psi _{\lambda _ {k_j}}(x_{k_{j}+1}). \end{aligned}$$
(4.16)

Let \(T_k(x)\) be the function defined in the proof of Theorem 4.1. There exist some \(\xi _{k_j}\in \partial \psi _{k_j}(x_{k_{j}+1})\) and \(\rho _{k_j}\in \partial T_{k_j}(x_{k_j+1})\) such that

$$\begin{aligned} \rho _{k_j}=\xi _{k_j}+\varepsilon _{k_j}\langle e_{k_j}, \lambda _{k_j}\rangle (x_{k_j+1}-x_{k_j}). \end{aligned}$$

It follows that

$$\begin{aligned} \psi _{k_j}(x^*)\!-\!\psi _{k_j}(x_{k_{j}\!+\!1})\!=\!\langle \rho _{k_j}, x^*-x_{k_{j}+1} \rangle \!-\!\varepsilon _{k_j}\langle e_{k_j}, \lambda _{k_j}\rangle \langle x_{k_{j}+1}\!-\!x_{k_j}, x^*-x_{k_{j}\!+\!1}\rangle .\qquad \end{aligned}$$
(4.17)

From the definition of the method (VPM), we have

$$\begin{aligned} \langle \rho _{k_j}, x^*-x_{k_{j}+1} \rangle \ge 0. \end{aligned}$$

That is

$$\begin{aligned} \psi _{k_j}(x^*)-\psi _{k_j}(x_{k_{j}+1})\ge -\varepsilon _{k_j}\langle e_{k_j}, \lambda _{k_j}\rangle \langle x_{k_{j}+1}-x_{k_j}, x^*-x_{k_{j}+1}\rangle . \end{aligned}$$
(4.18)

Similarly denote by \(\alpha _{k_j}=\varepsilon _{k_j}\langle e_{k_j}, \lambda _{k_j}\rangle \), obviously \(\alpha _{k_j}>0\). From the inequality (4.18), we deduce that

$$\begin{aligned} \psi _{k_j}(x^*)-\psi _{k_j}(x_{k_{j}+1})\ge -\alpha _{k_j}\Vert x_{k_{j}+1}-x_{k_j}\Vert \Vert x^*-x_{k_{j}+1}\Vert . \end{aligned}$$
(4.19)

Noted that \(\{x_k\}\) is bounded so that \(\{\Vert x^*-x_{k_{j}+1}\Vert \}\) is also bounded. By virtue of Lemma 4.2, we have \(\lim \limits _{j\rightarrow +\infty }\Vert x_{k_{j}+1}-x_{k_j}\Vert =0\). We conclude that the limit of the rightmost expression in (4.19) as \(j\rightarrow +\infty \) vanishes. Thus, taking limit in (4.16) we obtain

$$\begin{aligned} \langle F_0(A x^*)- F_0(A\bar{x}), \bar{\lambda }\rangle \ge 0 \end{aligned}$$
(4.20)

where \(\bar{\lambda }\) is the cluster point of \(\{\lambda _{k_j}\}\). Then we can conclude that (4.16) contradicts with the facts that \(\bar{\lambda }\in C_1\) and the assumption \(F_0(A x^*)\le _{intC} F_0(A\bar{x})\), thus we can claim that \(\bar{x}\) is a weak Pareto optimal solution of problem (MOP). The proof is complete. \(\square \)

Theorem 4.3

Assume the same assumptions as in Theorem 4.2 Then the whole sequence \(\{x_k\}\) converges to a weak Pareto optimal solution of problem (MOP).

Proof

Suppose to the contrary both \(\hat{x}\) and \(\tilde{x}\) are two distinct cluster points of \(\{x_k\}\) and

$$\begin{aligned} \lim \limits _{j\rightarrow +\infty }x_{k_j}=\hat{x}, \quad \lim \limits _{i\rightarrow +\infty }x_{k_i}=\tilde{x} \end{aligned}$$

From the Theorem 4.2, we know that \(\hat{x}\) and \(\tilde{x}\) are also the weak Pareto optimal solutions of problem (CMOP), \(\Vert \tilde{x}-x_k\Vert \) and \(\Vert \hat{x}-x_k\Vert \) are convergent. So there exists \(\tilde{\alpha }, \hat{\alpha }\in R\) such that

$$\begin{aligned} \lim \limits _{k\rightarrow +\infty }(\Vert \tilde{x}-x_k\Vert - \Vert \hat{x}-x_k\Vert )=\tilde{\alpha }-\hat{\alpha }. \end{aligned}$$
(4.21)

Since

$$\begin{aligned} \Vert x_k-\tilde{x}\Vert ^2=\Vert x_k-\hat{x}\Vert ^2+2\langle x_k-\hat{x}, \hat{x}-\tilde{x}\rangle + \Vert \hat{x}-\tilde{x}\Vert ^2 \end{aligned}$$
(4.22)

and

$$\begin{aligned} \lim \limits _{k\rightarrow +\infty }\langle x_k-\hat{x}, \hat{x}-\tilde{x}\rangle = \frac{1}{2}(\Vert \hat{x}-\tilde{x}\Vert ^2+\hat{\alpha }^2-\tilde{\alpha }^2). \end{aligned}$$
(4.23)

The left-hand of (4.23) vanishes for \(\hat{x}\) is a cluster point of \(\{x_k\}\), it follows that

$$\begin{aligned} \Vert \hat{x}-\tilde{x}\Vert ^2=\tilde{\alpha }^2-\hat{\alpha }^2. \end{aligned}$$
(4.24)

Repeating it again by changing the roles of \(\hat{x}\) and \(\tilde{x}\), we have that

$$\begin{aligned} \Vert \hat{x}-\tilde{x}\Vert ^2=\hat{\alpha }^2-\tilde{\alpha }^2. \end{aligned}$$
(4.25)

Combining (4.24) with (4.25), it is obvious that

$$\begin{aligned} \Vert \hat{x}-\tilde{x}\Vert =0\Rightarrow \hat{x}=\tilde{x} \end{aligned}$$

which contradicts with the assumption that \(\hat{x}\ne \tilde{x}\). We conclude that \(\{x_n\}\) is convergent to a weak Pareto optimal solution of the problem (CMOP) and the proof is complete. \(\square \)

5 Conclusion

In this paper, we defined the asymptotic function of a vector-valued function, obtained the analytical expression of the asymptotic function of a class of cone-convex vector-valued function, characterized the nonemptiness and compactness of the weak Pareto optimal solution sets of a composite multiobjective optimization problem. We then applied the obtained results to construct a proximal-type method for solving the composite multiobjective optimization problem. Under some conditions, we proved that any sequence generated by this method converges to a weak Pareto optimal solution of the problem.