1 Introduction and main results

We consider nonexistence of solutions for the semilinear biharmonic problems:

figure a

and

figure b

where \(B \subset {\mathbb {R}}^N \; (N \ge 5)\) is the unit ball, i.e., \(B=\{x \in {\mathbb {R}}^N: \; |x|<1\}\), \(\nu \) is the unit outward normal vector of \(\partial B\) relative to B and \(1<p\le \frac{N+4}{N-4}\). In the following, we use \(B_r\) to denote the ball of radius r centered at the origin.

The study of the equations in (P) and (Q) plays an important role in conformal geometry [9, 14, 37] and other related fields [18, 24]. The problems, similar to the Yamabe problem, are concerned with the existence of conformal metrics with constant or prescribed Q-curvature. For more results, we refer to [10, 15, 16, 26, 35, 38, 50] and the references therein.

The structure of positive solutions of the equation

$$\begin{aligned} \Delta ^2 u=u^p \;\;\; \text{ in } {\mathbb {R}}^N \; (N \ge 5), p>1 \end{aligned}$$
(1.1)

is considered by many authors recently, see [1, 2, 13, 17, 19, 22, 23, 27, 29, 31, 33, 40, 43, 48]. The classification of positive entire solutions of (1.1) via Morse index has also been obtained, see [12, 36, 40, 41, 49].

Recently, existence and nonexistence of positive supersolutions of the equation

$$\begin{aligned} \Delta ^2 u=g(u) \;\;\; \text{ in } {\mathbb {R}}^N \backslash {\overline{B}} \end{aligned}$$
(1.2)

have been studied in [7]. More precisely, Pérez, Melián and Quaas in [7] obtained that when \(1 \le N \le 4\), (1.2) does not admit any positive classical supersolution u verifying

$$\begin{aligned} -\Delta u>0 \;\;\; \text{ in } {\mathbb {R}}^N \backslash {\overline{B}}, \end{aligned}$$
(1.3)

provided g is continuous and nondecreasing in \([0, \infty )\). When \(N \ge 5\), such supersolutions exist if and only if

$$\begin{aligned} \int _0^\delta \frac{g(s)}{s^{\frac{2(N-2)}{N-4}}} ds<\infty \end{aligned}$$
(1.4)

for any \(\delta >0\). If \(g(u)=u^p\) and \(1<p \le \frac{N}{N-4}\), \(N \ge 5\), we see that (1.2) does not admit any positive classical solution u verifying (1.3). For \(p>\frac{N}{N-4}\), Gazzola and Grunau [22] have obtained that

$$\begin{aligned} u(x)=C(N,p) |x|^{-\frac{4}{p-1}}, \end{aligned}$$

where

$$\begin{aligned} C(N,p)= & {} \frac{8}{(p-1)^4} \Big [(N-2)(N-4) (p-1)^3+2 (N^2-10N+20) (p-1)^2\\&-16(N-4)(p-1)+32 \Big ] \end{aligned}$$

is a positive solution of (1.2) with \(g(u)=u^p\), which satisfies (1.3). It should be pointed out that the nonexistence results in [7] rely on the crucial assumption (1.3) but do not rely on any boundary condition. Under the boundary conditions in (P) and (Q), if \(p>\frac{N+4}{N-4}\), it is known from [28, 34] that (P) and (Q) admit a unique positive radial solution \(u \in C^4 ({\mathbb {R}}^N \backslash B)\) verifying \(\lim _{|x| \rightarrow \infty } \sup |x|^{N-4} u(x)<\infty \) respectively. For \(1<p \le \frac{N}{N-4}\), if we can show that any solution u of (P) satisfies the assumption (1.3), then Theorem 1 in [7] can be applied to derive that such solutions cannot exist. However, by the maximum principle, we see that the crucial assumption (1.3) cannot hold for solution \(u \in C^4({\mathbb {R}}^N \backslash B)\) of the problem (Q). Thus the arguments in [7] cannot be used to obtain the nonexistence result for the problem (Q).

In this paper, we first show that if \(u \in C^4({\mathbb {R}}^N \backslash B)\) is a solution of (P) with \(1<p \le \frac{N+4}{N-4}\), then \(-\Delta u>0\) in \({\mathbb {R}}^N \backslash {\overline{B}}\). Then, by Theorem 1 in [7], we can directly obtain the nonexistence results for (P) with \(1<p \le \frac{N}{N-4}\). We will do further to show that, for \(\frac{N}{N-4}<p \le \frac{N+4}{N-4}\), problem (P) does not admit any classical solution either. This extends the nonexistence range of p in [7]. Moreover, when \(1<p<\frac{N+4}{N-4}\), similar nonexistence results for problem (Q) are also obtained, but the arguments in [7] cannot be applied.

The main results of this paper are the following Liouville type results.

Theorem 1.2

Assume \(N \ge 5\) and \(1<p \le \frac{N+4}{N-4}\). Then problem (P) does not admit any solution \(u \in C^4({\mathbb {R}}^N \backslash B)\).

Theorem 1.2 Assume \(N \ge 5\) and \(1<p<\frac{N+4}{N-4}\). Then problem (Q) does not admit any solution \(u \in C^4({\mathbb {R}}^N \backslash B)\).

Remark 1.3

If \(u \in C^4({\mathbb {R}}^N \backslash B)\) is a nontrivial nonnegative solution to the problem

figure c

we find, by the maximum principle, that \(u>0\) in \({\mathbb {R}}^N \backslash {\overline{B}}\) and u is a solution to (P). Theorem 1.1 implies that problem \((P')\) with \(1<p \le \frac{N+4}{N-4}\) does not admit any nontrivial nonnegative solution.

If \(u \in C^4({\mathbb {R}}^N \backslash B)\) is a nontrivial nonnegative solution to the problem

figure d

we cannot directly conclude that \(u>0\) in \({\mathbb {R}}^N \backslash {\overline{B}}\). However, we will see that the arguments in the proof of Theorem 1.2 can also be used to obtain nonexistence of nontrivial nonnegative solutions of \((Q')\) under the assumptions of Theorem 1.2. Therefore, the results of Theorem 1.2 still hold for problem \((Q')\). For \(p=\frac{N+4}{N-4}\), we can also prove the nonexistence of nontrivial nonnegative radial solutions of \((Q')\), hence the nonnegative classical radial solution of \((Q')\) with \(p=\frac{N+4}{N-4}\) is \(u \equiv 0\).

It seems interesting that neither asymptotic behavior of u at infinity nor symmetric property of u is required in the proof of Theorems 1.1 and 1.2. On the other hand, we make the Kelvin transformation for the solutions of (P) and (Q), that is,

$$\begin{aligned} v(y)=|x|^{N-4} u(x), \;\;\; y=\frac{x}{|x|^2}. \end{aligned}$$

We see from Lemma 3.1 in [33] that \(v \in C^4 ({\overline{B}} \backslash \{0\})\) satisfies the problems

$$\begin{aligned} \left\{ \begin{array}{ll} \Delta ^2 v=|y|^{(N-4)p-(N+4)} v^p\;\;\;&{}\quad \text{ in } B \backslash \{0\},\\ v>0 \;\;\; &{}\quad \text{ in } B \backslash \{0\},\\ v=0, \;\;\; \Delta v-4 \frac{\partial v}{\partial \nu }=0 \;\;\; &{}\quad \text{ on } \partial B \end{array} \right. \end{aligned}$$
(1.5)

and

$$\begin{aligned} \left\{ \begin{array}{ll} \Delta ^2 v=|y|^{(N-4)p-(N+4)} v^p\;\;\;&{}\quad \text{ in } B \backslash \{0\},\\ v>0 \;\;\; &{}\quad \text{ in } B \backslash \{0\},\\ v=\frac{\partial v}{\partial \nu }=0 \;\;\; &{}\quad \text{ on } \partial B \end{array} \right. \end{aligned}$$
(1.6)

respectively, where \(\nu \) is the unit outward normal vector of \(\partial B\) relative to B. Simple calculations imply that \(-\Delta _x u>0\) in \({\mathbb {R}}^N \backslash {\overline{B}}\) is not equivalent to \(-\Delta _y v>0\) in \(B \backslash \{0\}\). However, for (1.5), we can still show that \(-\Delta _y v>0\) in \(B \backslash \{0\}\). This fact does not hold for (1.6) by the maximum principle. The results of Theorems 1.1 and 1.2 can also be used to obtain nonexistence results for problems (1.5) and (1.6).

Remark 1.4

(1) The nonexistence result in Theorem 1.2 for \(p=\frac{N+4}{N-4}\) is still open. When \(1<p<\frac{N+4}{N-4}\), if \(u \in C^4 ({\mathbb {R}}^N \backslash B)\) is a solution of (Q), we can use blow-up arguments to obtain the decay rate of u at infinity. Combining with the Pohozaev identity in the “Appendix”, such decay rate can be used to obtain the nonexistence results in Theorem 1.2. If \(u \in C^4 ({\mathbb {R}}^N \backslash B)\) is a solution of (Q) with \(p=\frac{N+4}{N-4}\), then \(v \in C^4 ({\overline{B}} \backslash \{0\})\) satisfies the problem

$$\begin{aligned} \left\{ \begin{array}{ll} \Delta ^2 v=v^{\frac{N+4}{N-4}}\;\;\;&{}\quad \text{ in } B \backslash \{0\},\\ v>0 \;\;\; &{}\quad \text{ in } B \backslash \{0\},\\ v=\frac{\partial v}{\partial \nu }=0 \;\;\; &{}\quad \text{ on } \partial B, \end{array} \right. \end{aligned}$$
(1.7)

where \(\nu \) is the unit outward normal vector of \(\partial B\) relative to B. We can see that 0 is a non-removable singularity point of v. Otherwise, there is \(R \gg 1\) such that \(u(x)\le C |x|^{4-N}\) for \(x \in {\mathbb {R}}^N \backslash B_R\), which implies, by the Pohozaev identity, that \(u \equiv 0\) in \({\mathbb {R}}^N \backslash {\overline{B}}\). On the other hand, it follows from the maximum principle that we cannot have \(-\Delta _y v>0\) in \(B \backslash \{0\}\). If we put an extra assumption on v:

(A) There is \(0<R<1\) such that

$$\begin{aligned} -\Delta _y v>0 \;\;\text{ in } B_R \backslash \{0\}, \end{aligned}$$
(1.8)

(we think that this assumption holds for v, but we cannot provide a proof here), we can obtain the asymptotic behavior of v at 0 by using Theorem 1.1 in [39]. From this we can obtain the decay rate of u at infinity. Unfortunately, this decay rate of u is not good enough to derive the nonexistence of u by using the Pohozaev identity. For the Navier boundary condition, we can use the moving-plane argument to show that v is radially symmetric and then derive the nonexistence of v by studying the detailed properties of v. However, since 0 is a non-removable singularity point of v for (1.7), the method of moving plane developed in [5] does not work. So we do not know how to use the moving-plane argument for the solution v of (1.7).

(2) We will see from Remark 3.3 below that the assumption (A) holds for radial solutions of problem (1.7). Therefore, we can obtain the nonexistence of radial solutions of problem (1.7) and hence the problem (Q) with \(p=\frac{N+4}{N-4}\) does not admit any radial solution \(u \in C^4 ({\mathbb {R}}^N \backslash B)\).

This paper is organized as follows. In Sect. 2, we give some preliminaries needed in Sect. 3. The main results will be obtained in Sect. 3. In “Appendix A”, we present the Pohozaev identities corresponding to problems (P) and (Q). In “Appendix B”, we estimate the upper bound of singular solutions needed in Sect. 2. Throughout this paper, we denote \(\nu \) the unit outward normal vector of \(\partial B\) relative to B.

2 Some preliminaries

In this section, we first use the blow-up argument to get the decay estimate of u, and then obtain \(\Delta u<0\) in \({\mathbb {R}}^N \backslash {\overline{B}}\) if \(u \in C^4 ({\mathbb {R}}^N \backslash B)\) is a solution of (P). Moreover, we can also obtain the negativity of \(\Delta v\) in \(B \backslash \{0\}\), where \(v(y)=|x|^{N-4} u(x)\), \(y=\frac{x}{|x|^2}\) and v is a solution of (1.5).

Lemma 2.1

Let u be a nonnegative solution of \(\Delta ^2 u=u^p\) in \({\mathbb {R}}^N \backslash B\). Assume that \(1<p<\frac{N+4}{N-4}\), then

$$\begin{aligned} u(x)\le C|x|^{-\frac{4}{p-1}}~~\text {for}\quad |x|>2, \end{aligned}$$
(2.1)

where C is a positive constant depending only on N and p.

Proof

Argue by contradiction that there is a sequence of nonnegative solutions \(\{u_k\}\) of \(\Delta ^2u=u^p\) in \({\mathbb {R}}^N \backslash B\) and a sequence of points \(\{x_k\}\subset {\mathbb {R}}^N \backslash B_{2}\), such that

$$\begin{aligned} M_k(x_k)d(x_k)>2k~~\text {for}~~k=1,2,\ldots , \end{aligned}$$

where \(M_k(x):=(u_k(x))^{\frac{p-1}{4}}\), \(d(x):=dist(x,\partial B)\). By the doubling lemma in [44], there exists another sequence \(\{y_k\}\subset {\mathbb {R}}^N \backslash B_2\) such that

$$\begin{aligned} M_k(y_k)d(y_k)>2k,\quad M_k(y_k)\ge M_k(x_k) \end{aligned}$$

and

$$\begin{aligned} M_k(z)\le 2M_k(y_k)~~~\text {for}~|z-y_k|\le k\lambda _k, \end{aligned}$$

where \(\lambda _k:=M_k^{-1}(y_k)\).

Define

$$\begin{aligned} w_k(x)=\lambda _k^{\frac{4}{p-1}}u_k(y_k+\lambda _kx), ~~x\in B_k. \end{aligned}$$

Thus \(w_k\) is a nonnegative solution of \(\Delta ^2w_k=w_k^p\) in \(B_k\). Note that \(w_k(0)=1\) and \(\max \nolimits _{B_k}w_k\le 2^{\frac{4}{p-1}}\), by elliptic estimates, we may assume, up to a subsequence, that \(\{w_k\}\) converges to w in \(C_{loc}^4({\mathbb {R}}^N)\), where w is a nonnegative solution of \(\Delta ^2w=w^p\) in \({\mathbb {R}}^N\). Using Theorem 1.4 in [43], we see that \(w\equiv 0\), which is a contradiction with \(w(0)=1\). \(\square \)

Proposition 2.2

Let \(1<p \le \frac{N+4}{N-4}\). Assume that \(u \in C^4 ({\mathbb {R}}^N \backslash B)\) is a solution of (P). Then

$$\begin{aligned} \Delta u(x)<0 \;\;\; \forall x \in {\mathbb {R}}^N \backslash {\overline{B}}. \end{aligned}$$
(2.2)

Moreover,

$$\begin{aligned} \frac{\partial u (x)}{\partial \nu }>0, \;\;\; \frac{\partial (\Delta u) (x)}{\partial \nu }<0 \;\;\; \forall x \in \partial B, \end{aligned}$$
(2.3)

where \(\nu \) is the unit outward normal vector of \(\partial B\) relative to B.

Proof

We first show that for \(1<p \le \frac{N+4}{N-4}\),

$$\begin{aligned} |\Delta u(x)| \rightarrow 0 \;\; \text{ as } |x| \rightarrow \infty . \end{aligned}$$
(2.4)

We consider two cases here: (i) \(1<p<\frac{N+4}{N-4}\), (ii) \(p=\frac{N+4}{N-4}\).

For the case (i), by Lemma 2.1, we see that there are \(C:=C(N,p)>0\) and \(R>2\) such that

$$\begin{aligned} u(x) \le C |x|^{-4/(p-1)} \;\; \; \forall x \in {\mathbb {R}}^N \backslash {\overline{B_R}}. \end{aligned}$$
(2.5)

For the case (ii), we cannot use the blow-up argument. We need to use some new arguments to get similar estimates as in the case (i), i.e., there are \(C:=C(N)>0\) and \(R \gg 1\) such that

$$\begin{aligned} u(x) \le C |x|^{-\frac{N-4}{2}} \;\;\; \forall x \in {\mathbb {R}}^N \backslash {\overline{B_R}}. \end{aligned}$$
(2.6)

To this end, making the Kelvin transformation:

$$\begin{aligned} v(y)=|x|^{N-4} u(x), \;\;\; y=\frac{x}{|x|^2}, \end{aligned}$$

it follows from Lemma 3.1 of [33] that \(v \in C^4({\overline{B}} \backslash \{0\})\) satisfies the problem

$$\begin{aligned} \left\{ \begin{array}{ll} \Delta ^2 v=v^{\frac{N+4}{N-4}}\;\;\;&{}\quad \text{ in } B \backslash \{0\},\\ v>0 \;\;\; &{}\quad \text{ in } B \backslash \{0\},\\ v=0, \;\;\; \Delta v-4 \frac{\partial v}{\partial \nu }=0 \;\;\; &{}\quad \text{ on } \partial B, \end{array} \right. \end{aligned}$$
(2.7)

where \(\nu \) is the unit outward normal vector of \(\partial B\) relative to B. Problem (2.7) is the critical Steklov biharmonic problem with critical value 4 in the coefficient of the normal derivatives. The corresponding variational problems have been studied in [3, 4, 6].

Note that \(u=0\), \(\Delta u=0\) on \(\partial B\) and \(\frac{\partial v}{\partial \nu } \le 0\) on \(\partial B\), we see that

$$\begin{aligned} v=0, \;\;\; \Delta v \le 0 \;\; \text{ on } \partial B. \end{aligned}$$
(2.8)

Taking \(p=\frac{N+4}{N-4}\) in (2.23) below, we find that \(v^{\frac{N+4}{N-4}} \in L^1 (B)\). As in the proof of (2.24) in Proposition 2.3, we deduce that \(-\Delta v\) is a superharmonic function in B in the distributional sense. Note that \(-\Delta v \ge 0\) on \(\partial B\), we infer that

$$\begin{aligned} -\Delta v \ge 0 ~~\text {in}~~ B\backslash \{0\}. \end{aligned}$$
(2.9)

Then, \(v \in C^4 (B \backslash \{0\})\) is a solution to the equation in (2.7) satisfying (2.9). Using Proposition 5.2, we obtain

$$\begin{aligned} v(y) \le C |y|^{-\frac{N-4}{2}} \;\;\; \forall y \in B \backslash \{0\}. \end{aligned}$$
(2.10)

By the Kelvin transformation, we find that (2.6) holds.

We next show that (2.4) holds. For \(1<p \le \frac{N+4}{N-4}\) and any \(\lambda >1\), define

$$\begin{aligned} {\overline{u}} (x)=\lambda ^{\frac{4}{p-1}} u(\lambda x). \end{aligned}$$

Then \({\overline{u}}\) is a solution of (P) in \({\mathbb {R}}^N \backslash {\overline{B_{\frac{1}{\lambda }}}}\). By (2.5) and (2.6), we see

$$\begin{aligned} {\overline{u}} (x) \le C \;\;\; \text {for}~~ x \in {\mathbb {R}}^N \backslash {\overline{B_{{R}}}}, \end{aligned}$$
(2.11)

where C and R are the same constants as in (2.5) and (2.6). For any \(x_0 \in {\mathbb {R}}^N \backslash {\overline{B_{10 { R}}}}\), taking \(\lambda =\frac{|x_0|}{5 {R}}\) and \(\xi _0=\lambda ^{-1} x_0\), we see that \(|\xi _0|=5 {R}\). By (2.11) and standard elliptic estimates, we have

$$\begin{aligned} \sum _{k \le 3}|\nabla ^k {\overline{u}}(\xi _0)| \le C. \end{aligned}$$

Rescaling back we obtain that for \(x \in {\mathbb {R}}^N \backslash {\overline{B_{10 { R}}}}\),

$$\begin{aligned} \sum _{k \le 3} |x|^{\frac{4}{p-1}+k} |\nabla ^k u(x)| \le C. \end{aligned}$$
(2.12)

Thus, (2.4) holds.

It follows from (2.4) that for any \(\epsilon >0\), we can find \(R_\epsilon \gg 1\) such that \(\Delta u\le \epsilon \) on \({\mathbb {R}}^N \backslash B_{R_\epsilon }\). Using the subharmonicity of \(\Delta u\) and \(\Delta u=0\) on \(\partial B\), we deduce

$$\begin{aligned} \Delta u (x) \le \epsilon , \;\; \forall 1 \le |x| \le R_\epsilon . \end{aligned}$$
(2.13)

Sending \(\epsilon \) to 0, we find

$$\begin{aligned} \Delta u (x) \le 0, \;\; \forall x \in {\mathbb {R}}^N \backslash {\overline{B}}. \end{aligned}$$
(2.14)

By the strong maximum principle, we see that (2.2) holds. Then (2.3) follows from (2.2) and Hopf’s boundary lemma. This completes the proof of this proposition. \(\square \)

Let u be a solution of the problem

$$\begin{aligned} \left\{ \begin{array}{ll} \Delta ^2 u=u^p \;\;\; &{}\quad \text{ in } {\mathbb {R}}^N \backslash {\overline{B}},\\ u>0 \;\;\; &{}\quad \text{ in } {\mathbb {R}}^N \backslash {\overline{B}},\\ u=\Delta u=0 \;\;\; &{}\quad \text{ on } \partial B. \end{array} \right. \end{aligned}$$
(2.15)

Using the Kelvin transformation:

$$\begin{aligned} v(y)=|x|^{N-4} u(x), \;\;\; y=\frac{x}{|x|^2}, \end{aligned}$$

we find, by Lemma 3.1 in [33], that v(y) satisfies the problem

$$\begin{aligned} \left\{ \begin{array}{ll} \Delta ^2 v=|y|^{(N-4)p-(N+4)} v^p\;\;\;&{}\quad \text{ in } B \backslash \{0\},\\ v>0 \;\;\; &{}\quad \text{ in } B \backslash \{0\},\\ v=0, \;\;\; \Delta v-4 \frac{\partial v}{\partial \nu }=0 \;\;\; &{}\quad \text{ on } \partial B, \end{array} \right. \end{aligned}$$
(2.16)

where \(\nu \) is the unit outward normal vector of \(\partial B\) relative to B. Problem (2.16) is closely related to the study of isolated singularities of polyharmonic equations, see [8, 11] for more details.

Let \(u(x)=u(r, \theta )\) with \(r=|x|\) and \(v(y)=v(\rho , \theta )\) with \(\rho =|y|\). It is easy to check that

$$\begin{aligned} \Delta _x u=\rho ^N \Big (\Delta _y v-4 \frac{v_\rho }{\rho }-2(N-4) \frac{v}{\rho ^2} \Big ), \end{aligned}$$
(2.17)

where \(v_\rho =\frac{\partial v}{\partial \rho }\). From (2.17), we see that \(\Delta _x u<0\) in \({\mathbb {R}}^N \backslash {\overline{B}}\) does not directly imply \(\Delta _y v<0\) in \(B \backslash \{0\}\).

To obtain \(\Delta _y v<0\) in \(B \backslash \{0\}\), we have to present a new proof independently, which is interesting itself.

Proposition 2.3

Let \(1<p \le \frac{N+4}{N-4}\) and \(v \in C^4 ({\overline{B}} \backslash \{0\})\) be a solution to (2.16). Then

$$\begin{aligned} \Delta _y v<0 \;\;\; \text{ in } B \backslash \{0\}. \end{aligned}$$
(2.18)

Moreover,

$$\begin{aligned} \frac{\partial v}{\partial \nu }<0, \;\;\; \Delta _y v<0, \;\;\; \frac{\partial (\Delta v)}{\partial \nu }>0 \;\; \text{ on } \partial B, \end{aligned}$$
(2.19)

where \(\nu \) is the unit outward normal vector of \(\partial B\) relative to B.

Proof

Since \(\frac{\partial u}{\partial \nu }=-\frac{\partial v}{\partial \nu }\) on \(\partial B\), it follows from (2.3) that

$$\begin{aligned} \frac{\partial v}{\partial \nu }<0 \;\; \text{ on } \partial B. \end{aligned}$$
(2.20)

Since \(v(1, \theta ) \equiv 0\) for \(\theta \in S^{N-1}\), we easily see that \((\Delta _\theta v)(1, \theta ) \equiv 0\) for \(\theta \in S^{N-1}\). Thus, \(\Delta _x u=0\) is equivalent to \(\Delta _y v-4 \frac{\partial v}{\partial \nu }=0\) on \(\partial B\), which implies that \(v_{\rho \rho }+(N-5) v_\rho =0\) on \(\partial B\). By (2.20), we obtain

$$\begin{aligned} \Delta _y v=4 \frac{\partial v}{\partial \nu }<0 \;\; \text{ on } \partial B. \end{aligned}$$
(2.21)

It is easy to check that

$$\begin{aligned} (\Delta _x u)_\rho =-\rho ^{N+1} [N \Delta _y v-6(N-2) \rho ^{-1} v_\rho +\rho (\Delta _y v)_\rho -4 v_{\rho \rho } -2(N-2)(N-4) \rho ^{-2} v]. \end{aligned}$$

Due to (2.3) and \(\Delta _y v=4 \frac{\partial v}{\partial \nu }\) on \(\partial B\), we can deduce

$$\begin{aligned} \begin{aligned} \rho (\Delta _y v)_\rho>&4v_{\rho \rho }+6(N-2)\rho ^{-1}v_\rho -N\Delta _y v+2(N-2)(N-4)\rho ^{-2}v\\ =&4v_{\rho \rho }+6(N-2)\rho ^{-1}v_\rho -N\Delta _y v\\ =&4(5-N)v_\rho +6(N-2)v_\rho -4Nv_\rho \\ =&-2(N-4)v_\rho >0\;\;\text{ on } \partial B, \end{aligned} \end{aligned}$$

which implies

$$\begin{aligned} \frac{\partial (\Delta _y v)}{\partial \nu }>-2(N-4) \frac{\partial v}{\partial \nu }>0 \;\; \text{ on } \partial B. \end{aligned}$$
(2.22)

We next prove (2.18). We first claim that

$$\begin{aligned} |y|^{(N-4)p-(N+4)} v^p \in L^1 (B). \end{aligned}$$
(2.23)

To do so, we take the cut-off function \(\eta \in C^\infty ({\mathbb {R}})\) with values in [0,1] satisfying

$$\begin{aligned} \eta (t)={\left\{ \begin{array}{ll} 0, &{}\quad \text {for}~t\le 1,\\ 1, &{}\quad \text {for}~t\ge 2. \end{array}\right. } \end{aligned}$$

Let \(q=\frac{4p}{p-1}\) and define \(\varphi _\epsilon (y)=\eta (\epsilon ^{-1} |y|)^q\), where \(0<\epsilon \ll 1\). Multiplying the equation in (2.16) by \(\varphi _\epsilon (x)\) and integrating by parts, we have

$$\begin{aligned} \int _B |y|^{(N-4)p-(N+4)} v^p \varphi _\epsilon = \int _B v \Delta ^2 \varphi _\epsilon +\int _{\partial B} \frac{\partial (\Delta v)}{\partial \nu } d\sigma _1. \end{aligned}$$

Since

$$\begin{aligned} |\Delta ^2\varphi _\epsilon |\le C\epsilon ^{-4}\varphi _\epsilon ^{1/p}\chi _{\{\epsilon \le |y|\le 2\epsilon \}}. \end{aligned}$$

By Hölder inequality, we have

$$\begin{aligned} \left| \int _B v \Delta ^2 \varphi _\epsilon \right|\le & {} C \epsilon ^{-4} \int _{\epsilon \le |y| \le 2 \epsilon } v \varphi _\epsilon ^{\frac{1}{p}}\\\le & {} C \epsilon ^{N-\frac{N}{p}-4} \Big (\int _{\epsilon \le |y| \le 2 \epsilon } v^p \varphi _\epsilon \Big )^{1/p}\\\le & {} C \epsilon ^{N-\frac{N}{p}-4-(N-4)+\frac{N+4}{p}} \Big (\int _B |y|^{(N-4)p-(N+4)} v^p \varphi _\epsilon \Big )^{1/p}\\\le & {} C \Big (\int _B |y|^{(N-4)p-(N+4)} v^p \varphi _\epsilon \Big )^{1/p}. \end{aligned}$$

Thus

$$\begin{aligned} \int _B |y|^{(N-4)p-(N+4)} v^p \varphi _\epsilon \le C \Big (\int _B |y|^{(N-4)p-(N+4)} v^p \varphi _\epsilon \Big )^{1/p} +\int _{\partial B} \frac{\partial (\Delta v)}{\partial \nu } d\sigma _1, \end{aligned}$$

which implies

$$\begin{aligned} \int _B |y|^{(N-4)p-(N+4)} v^p \varphi _\epsilon \le C. \end{aligned}$$

Letting \(\epsilon \) to 0, we obtain

$$\begin{aligned} \int _B |y|^{(N-4)p-(N+4)} v^p \le C. \end{aligned}$$

We now show that \(\Delta v\) is a subharmonic function in B in the distributional sense. Let \(\psi \in C_c^\infty (B)\) be a nonnegative function. We only need to prove that

$$\begin{aligned} \int _B \Delta v \Delta \psi \ge 0. \end{aligned}$$
(2.24)

Multiplying (2.16) by \(\varphi _\epsilon \psi \) and integrating by parts, we obtain

$$\begin{aligned} 0\le & {} \int _B |y|^{(N-4)p-(N+4)} \varphi _\epsilon \psi v^p\\= & {} \int _B \Delta (\varphi _\epsilon \psi ) \Delta v\\= & {} \int _B \Delta v (\Delta \psi \varphi _\epsilon +2 \nabla \psi \cdot \nabla \varphi _\epsilon +\psi \Delta \varphi _\epsilon ). \end{aligned}$$

Denote \(\zeta =2 \nabla \psi \cdot \nabla \varphi _\epsilon +\psi \Delta \varphi _\epsilon \). Then \(\zeta (y) \equiv 0\) for \(|y| \le \epsilon \) and for \(|y| \ge 2 \epsilon \), and

$$\begin{aligned} |\Delta \zeta (y)| \le C \epsilon ^{-4}. \end{aligned}$$

Thus, we have

$$\begin{aligned} \Big | \int _B \Delta v \zeta \Big |\le & {} \int _B v |\Delta \zeta |\\\le & {} C \epsilon ^{-4} \Big (\int _{\epsilon \le |y| \le 2 \epsilon } v^p \Big )^{1/p} \epsilon ^{N(1-1/p)}\\\le & {} C \epsilon ^{N-\frac{N}{p}-4-(N-4)+\frac{N+4}{p}} \Big (\int _B |y|^{(N-4)p-(N+4)} v^p \Big )^{1/p}\\\le & {} C \epsilon ^{4/p} \rightarrow 0 \;\; \text{ as } \epsilon \rightarrow 0. \end{aligned}$$

Hence, we infer

$$\begin{aligned} \int _B \Delta v \Delta \psi= & {} \lim _{\epsilon \rightarrow 0} \int _B \Delta v (\Delta \psi \varphi _\epsilon +2 \nabla \psi \cdot \nabla \varphi _\epsilon +\psi \Delta \varphi _\epsilon )\\= & {} \lim _{\epsilon \rightarrow 0}\int _B |y|^{(N-4)p-(N+4)} \varphi _\epsilon \psi v^p \ge 0. \end{aligned}$$

Therefore, \(\Delta v\) is a subharmonic function in B in the distributional sense. On the other hand, from (2.21), we see that \(\Delta v<0\) on \(\partial B\). By the maximum principle, we conclude that (2.18) holds and the proof of this proposition is completed. \(\square \)

3 Proof of the main results

In this section, we present the proof of Theorems 1.1 and 1.2. For the subcritical cases, we use the Pohozaev identities and decay estimates to prove Theorems 1.1 and 1.2. The proof of the critical case of Theorem 1.1 needs some new arguments, since the Pohozaev identity cannot be used to deal with this case.

Proof of the subcritical case of Theorem 1.1

As in the proof of Proposition 2.2, we find that, for \(1<p<\frac{N+4}{N-4}\), there are \(C:=C(N,p)>0\) and \(R_*>2\) such that (see (2.12)) for \(|x|>R_*\),

$$\begin{aligned} \sum _{k \le 3} |x|^{\frac{4}{p-1}+k} |\nabla ^k u(x)| \le C. \end{aligned}$$
(3.1)

Thus, for any \(R>R_*\) and \(k=0,1,2,3\),

$$\begin{aligned} |\nabla ^k u(x)|\le C |x|^{-\frac{4+k(p-1)}{p-1}}, \;\;\; \forall |x| \ge R. \end{aligned}$$
(3.2)

By Corollary 4.2, we have

$$\begin{aligned} \Big (\frac{N}{p+1}-\frac{N-4}{2} \Big ) \int _{B_R\backslash {\overline{B}}}u^{p+1}=\int _{\partial B_R} G(u,\Delta u) (x) d \sigma _R+\int _{\partial B}\frac{\partial u}{\partial \nu }\frac{\partial (\Delta u)}{\partial \nu } d \sigma _1, \end{aligned}$$
(3.3)

where

$$\begin{aligned} G(u,w) (x)= & {} \frac{R}{p+1}u^{p+1}-\frac{2}{R}(x\cdot \nabla u)(x\cdot \nabla w) \\&+R \nabla u\nabla w-\frac{N-2}{R} u(x\cdot \nabla w)+\frac{R}{2} w^2 \\&-\frac{N}{2R} (w(x\cdot \nabla u)-u(x\cdot \nabla w)). \end{aligned}$$

Using (3.2), by direct calculations, we deduce

$$\begin{aligned} \Big |\int _{\partial B_R} G(u, \Delta u)(x) d \sigma _R \Big | \le C R^{N-\frac{4(p+1)}{p-1}} \;\;\; \forall R>R_*, \end{aligned}$$
(3.4)

where \(C>0\) is independent of R. Since \(N-\frac{4(p+1)}{p-1}<0\), we see from (3.4) that

$$\begin{aligned} \int _{\partial B_R} G(u, \Delta u)(x) d \sigma _R \rightarrow 0 \;\; \text{ as } R \rightarrow \infty . \end{aligned}$$
(3.5)

Thanks to (3.3) and (3.5), we see

$$\begin{aligned} \Big (\frac{N}{p+1}-\frac{N-4}{2} \Big ) \int _{{\mathbb {R}}^N \backslash {\overline{B}}}u^{p+1}=\int _{\partial B}\frac{\partial u}{\partial \nu }\frac{\partial (\Delta u)}{\partial \nu } d \sigma _1. \end{aligned}$$
(3.6)

By Proposition 2.2, we find

$$\begin{aligned} \int _{\partial B}\frac{\partial u}{\partial \nu }\frac{\partial (\Delta u)}{\partial \nu } d \sigma _1<0. \end{aligned}$$

Since \(\frac{N}{p+1}-\frac{N-4}{2}>0\) and \(u>0\) in \({{\mathbb {R}}}^N\setminus \overline{B}\), this contradicts (3.6). This completes the proof of Theorem 1.1 for the subcritical case. \(\square \)

Proof of Theorem 1.2

As in the proof of Theorem 1.1, we have that, for \(1<p<\frac{N+4}{N-4}\), there are \(C:=C(N,p)>0\) and \(R_*>2\) such that (see (2.12)) for \(|x|>R_*\),

$$\begin{aligned} \sum _{k \le 3} |x|^{\frac{4}{p-1}+k} |\nabla ^k u(x)| \le C. \end{aligned}$$
(3.7)

Thus, for any \(R>R_*\) and \(k=0,1,2,3\),

$$\begin{aligned} |\nabla ^k u(x)|\le C |x|^{-\frac{4+k(p-1)}{p-1}} \;\;\; \forall |x| \ge R. \end{aligned}$$
(3.8)

Due to Corollary 4.3, we have

$$\begin{aligned} \Big [ \frac{N}{p+1}-\frac{N-4}{2} \Big ] \int _{B_R \backslash {\overline{B}}} u^{p+1}=\int _{\partial B_R}G(u, \Delta u) (x)d \sigma _R-\frac{1}{2} \int _{\partial B} (\Delta u)^2 d \sigma _1, \end{aligned}$$
(3.9)

where \(G(u, \Delta u)\) is given in (3.3). Meanwhile, (3.4) and (3.5) hold for \(\int _{\partial B_R} G(u, \Delta u) (x) d \sigma _R\). Then, letting \(R \rightarrow \infty \) in (3.9), we find

$$\begin{aligned} \Big [ \frac{N}{p+1}-\frac{N-4}{2} \Big ] \int _{{\mathbb {R}}^N \backslash {\overline{B}}} u^{p+1}=-\frac{1}{2} \int _{\partial B} (\Delta u)^2 d \sigma _1. \end{aligned}$$
(3.10)

Since \(\frac{N}{p+1}-\frac{N-4}{2}>0\) and \(u>0\) in \({\mathbb {R}}^N\setminus \overline{B}\), this contradicts (3.10). This completes the proof of Theorem 1.2. \(\square \)

Proof of the critical case of Theorem 1.1

Let u be a solution to (P) for \(p=\frac{N+4}{N-4}\). Set \(v(y)=|x|^{N-4} u(x)\), \(y=\frac{x}{|x|^2}\), then v(y) satisfies the problem

$$\begin{aligned} \left\{ \begin{array}{ll} \Delta ^2 v=v^{\frac{N+4}{N-4}}\;\;\;&{}\quad \text{ in } B \backslash \{0\},\\ v>0 \;\;\; &{}\quad \text{ in } B \backslash \{0\},\\ v=0, \;\;\; \Delta v-4 \frac{\partial v}{\partial \nu }=0 \;\;\; &{}\quad \text{ on } \partial B. \end{array} \right. \end{aligned}$$
(3.11)

By Proposition 2.3, we have

$$\begin{aligned} \Delta v<0 \;\;\; \text{ in } B \backslash \{0\}. \end{aligned}$$
(3.12)

Moreover,

$$\begin{aligned} \frac{\partial v}{\partial \nu }<0, \;\;\; \Delta v<0, \;\;\; \frac{\partial (\Delta v)}{\partial \nu }>0 \;\; \text{ on } \partial B. \end{aligned}$$
(3.13)

In the following, instead of showing the nonexistence of u, we show the nonexistence of v. We first claim that 0 is a non-removable singularity point of v. Suppose that 0 is a removable singularity point of v, then \(v \in C^4 ({\overline{B}})\). We can also establish the corresponding Pohozaev identity for (3.11) in B, but we cannot use it directly to derive a contradiction because of the inhomogeneous boundary conditions. On the other hand, since 0 is a removable singularity point of v, we see that \(\lim _{|x| \rightarrow \infty } |x|^{N-4} u(x)=v(0)>0\). Thus there is \(R^* \gg 1\) such that

$$\begin{aligned} u(x) \le 10 v(0) |x|^{-(N-4)} \;\;\; \forall |x| \ge R^*, \end{aligned}$$
(3.14)

which implies

$$\begin{aligned} \sum _{k \le 3} |x|^{N-4+k} |\nabla ^k u(x)| \le C \;\;\; \forall |x| \ge 20 R^*. \end{aligned}$$
(3.15)

Using (3.15) and Corollary 4.2, we have

$$\begin{aligned} \Big (\frac{N}{p+1}-\frac{N-4}{2} \Big ) \int _{B_R\backslash {\overline{B}}}u^{p+1}=\int _{\partial B_R} G(u,\Delta u) (x) d \sigma _R+\int _{\partial B}\frac{\partial u}{\partial \nu }\frac{\partial (\Delta u)}{\partial \nu } d \sigma _1 \end{aligned}$$
(3.16)

and

$$\begin{aligned} \Big |\int _{\partial B_R} G(u,\Delta u) (x) d \sigma _R \Big | \le C R^{4-N} \rightarrow 0 \;\;\; \text{ as } R \rightarrow \infty . \end{aligned}$$

Recall that \(p=\frac{N+4}{N-4}\) and let \(R\rightarrow \infty \) in (3.16), we find

$$\begin{aligned} \int _{\partial B}\frac{\partial u}{\partial \nu }\frac{\partial (\Delta u)}{\partial \nu } d \sigma _1=0. \end{aligned}$$

This contradicts (2.3). \(\square \)

Lemma 3.1

Let \(N \ge 5\) and \(v \in C^4 ({\overline{B}} \backslash \{0\})\) be a solution to (3.11) satisfying (3.12) and (3.13). Then v is radially symmetric about 0.

Proof

The moving-plane method as in [47] is used to prove this lemma. We rewrite the equation of v into a system of equations:

$$\begin{aligned} \left\{ \begin{array}{ll} -\Delta v=z \;\;\; &{}\quad \text{ in } B \backslash \{0\},\\ -\Delta z=v^{\frac{N+4}{N-4}} \;\;\; &{}\quad \text{ in } B \backslash \{0\},\\ v=0, \;\;z=-4 \frac{\partial v}{\partial \nu } \;\;\; &{}\quad \text{ on } \partial B. \end{array} \right. \end{aligned}$$
(3.17)

Due to (3.12) and (3.13), we see that (vz) is a positive solution to (3.17). Moreover, the system (3.17) is cooperative. Note that \(v=0\), \(z=-\Delta v>0\) and \(\frac{\partial v}{\partial \nu }<0\), \(\frac{\partial z}{\partial \nu }<0\) on \(\partial B\). However, since (3.17) is an inhomogeneous boundary condition for z, the arguments in [47] cannot be directly used here. We need to overcome some extra difficulties in the proof.

Let \(T_\lambda :=\{x \in {\mathbb {R}}^N: \; x_1=\lambda \}\), \(\Sigma (\lambda ):=\{x \in B: 0<\lambda<x_1<1\}\) and \(\Sigma '(\lambda )\) denote the reflection of \(\Sigma (\lambda )\) with respect to the plane \(T_\lambda \). Let \(x=(x_1, x_2, \ldots , x_N) \in \Sigma (\lambda )\) and \(x^\lambda =(x_1^\lambda , x_2, \ldots , x_N)\) be the reflection of x with respect to the plane \(T_\lambda \). Then \(x_1^\lambda =2 \lambda -x_1\). Define \(V_\lambda (x):=v(x^\lambda )-v(x)\) and \(Z_\lambda (x):=z(x^\lambda )-z(x)\) for \(x \in \Sigma (\lambda )\). Then \((V_\lambda ,Z_\lambda )\) satisfies the system:

$$\begin{aligned} \left\{ \begin{array}{ll} -\Delta V=Z \;\;\; &{}\quad \text{ in } \Sigma (\lambda ),\\ -\Delta Z=\Big (\frac{N+4}{N-4} \Big ) \xi ^{\frac{8}{N-4}} V \;\;\; &{}\quad \text{ in } \Sigma (\lambda ), \end{array} \right. \end{aligned}$$
(3.18)

where \(\xi \) is between v(x) and \(v(x^\lambda )\).

First we claim that there exist \(t_0>0\) and \(\alpha >0\) depending only on B, such that \(v(x-tn)\) and \(z(x-tn)\) are increasing for \(t \in [0,t_0]\), where \(n \in {\mathbb {R}}^N\) satisfies \(|n|=1\) and \((n, \nu (x)) \ge \alpha \) and \(x \in \partial B\). Indeed, for any \(x_0 \in \partial B\), define

$$\begin{aligned} {\mathcal O}_\epsilon =B \cap \{x \in {\mathbb {R}}^N: \; |x-x_0|<\epsilon \} \end{aligned}$$

and

$$\begin{aligned} S_\epsilon =\partial B \cap \{x \in {\mathbb {R}}^N: \; |x-x_0|<\epsilon \}. \end{aligned}$$

Since \(\frac{\partial v (x_0)}{\partial \nu _{x_0}}<0\) and \(\frac{\partial z (x_0)}{\partial \nu _{x_0}}<0\), we see that there exist \(\epsilon _0>0\) and \(1>\alpha _0>0\) such that for \(0<\epsilon \le \epsilon _0\), \(\alpha _0\le \alpha <1\) and \(x \in S_\epsilon \) with \((\nu _x, \nu _{x_0}) \ge \alpha \),

$$\begin{aligned} \frac{\partial v (x)}{\partial \nu _{x_0}}<0 \;\; \text{ and } \;\; \frac{\partial z (x)}{\partial \nu _{x_0}}<0. \end{aligned}$$
(3.19)

Otherwise, there are sequences \(\{(\epsilon _i, \alpha _i)\}\) with \(\epsilon _i \rightarrow 0\), \(\alpha _i \rightarrow 1\) as \(i \rightarrow \infty \), and \(\{x_i\}\) with \(x_i \in S_{\epsilon _i}\) and \((\nu _{x_i}, \nu _{x_0}) \ge \alpha _i\) such that

$$\begin{aligned} \frac{\partial v (x_i)}{\partial \nu _{x_0}} \ge 0 \;\; \text{ or } \;\; \frac{\partial z (x_i)}{\partial \nu _{x_0}} \ge 0. \end{aligned}$$
(3.20)

Since \(x_i \rightarrow x_0\) and \(\nu _{x_i} \rightarrow \nu _{x_0}\) as \(i \rightarrow \infty \), it follows from (3.20) that

$$\begin{aligned} \frac{\partial v (x_0)}{\partial \nu _{x_0}} \ge 0 \;\; \text{ or } \;\; \frac{\partial z (x_0)}{\partial \nu _{x_0}} \ge 0. \end{aligned}$$
(3.21)

This is a contradiction with the fact

$$\begin{aligned} \frac{\partial v}{\partial \nu }<0 \;\; \text{ and } \;\; \frac{\partial z}{\partial \nu }<0 \;\;\; \text{ on } \partial B. \end{aligned}$$
(3.22)

We next show that there exist \(t_0>0\) and \(\alpha _0>0\) such that for any \(x \in S_{\epsilon _0}\) with \((\nu _{x_0}, \nu _x) \ge \alpha _0\), \(v(x-t\nu _{x_0})\) and \(z(x-t \nu _{x_0})\) are increasing for \(t \in [0,t_0]\). Suppose that \(t_0\) does not exist, then there is a sequence \(\{x^j\} \subseteq {\mathcal O}_{\epsilon _0}\) with \(x^j \rightarrow x_0\) as \(j \rightarrow \infty \) such that \(\frac{\partial v (x^j)}{\partial \nu _{x_0}} \ge 0\) or \(\frac{\partial z (x^j)}{\partial \nu _{x_0}} \ge 0\). Let \(a^j\) be the intersection point on \(S_{\epsilon _0}\) in the positive \(\nu _{x_0}\) direction from \(x^j\), then \(\frac{\partial v (a^j)}{\partial \nu _{x_0}} \le 0\) and \(\frac{\partial z (a^j)}{\partial \nu _{x_0}} \le 0\). Since \(a^j \rightarrow x_0\) as \(j \rightarrow \infty \), we find

$$\begin{aligned} \frac{\partial v (x_0)}{\partial \nu _{x_0}}=0 \;\; \text{ or } \;\; \frac{\partial z (x_0)}{\partial \nu _{x_0}}=0, \end{aligned}$$

which contradicts (3.22). Therefore, there exists \(0<{\tilde{\epsilon }}<\epsilon _0\) such that for any \(\lambda \in (1-{\tilde{\epsilon }},1)\)

$$\begin{aligned} \frac{\partial v}{\partial x_1}<0, \;\;\; \frac{\partial z}{\partial x_1}<0 \;\; \text{ for } x \in \Sigma (\lambda ) \end{aligned}$$

and

$$\begin{aligned} v(x)<v(x^\lambda ), \;\;\; z(x)<z(x^\lambda ) \;\; \text{ for } x \in \Sigma (\lambda ). \end{aligned}$$

Let

$$\begin{aligned} \lambda _0=\inf \{\lambda \ge 0: \; v(x)<v(x^\tau ),\; z(x)<z(x^\tau )\;\; \text{ for } x \in \Sigma (\tau )\hbox { with }\tau \ge \lambda \}. \end{aligned}$$

We will show that \(\lambda _0=0\). On the contrary, we assume that \(0<\lambda _0<1\). Then, we have

$$\begin{aligned} v(x) \le v(x^{\lambda _0}), \;\;\; z(x) \le z(x^{\lambda _0}) \;\; \text{ for } x \in \Sigma (\lambda _0). \end{aligned}$$
(3.23)

We first show

$$\begin{aligned} v(x)<v(x^{\lambda _0}), \;\;\; z(x)<z(x^{ \lambda _0}) \;\; \text{ for } x \in \Sigma (\lambda _0). \end{aligned}$$
(3.24)

Suppose that (3.24) does not hold, then there is \(x_0 \in \Sigma (\lambda _0)\) such that

$$\begin{aligned} (a)\;\;v(x_0)=v(x_0^{\lambda _0}), \;\; \text{ or } \;\;(b)\;\; z(x_0)=z(x_0^{\lambda _0}). \end{aligned}$$
(3.25)

We need to consider two cases here: (i) \(0 \not \in \Sigma '(\lambda _0)\), (ii) \(0 \in \Sigma '(\lambda _0)\).

For the case (i), we first show that (a) of (3.25) cannot hold. On the contrary, since \(V_{\lambda _0} \ge 0\) and \(Z_{\lambda _0} \ge 0\) in \(\Sigma (\lambda _0)\), by the maximum principle, we obtain that \(V_{\lambda _0} \equiv 0\) in \(\Sigma (\lambda _0)\). This contradicts the fact that \(v=0\) on \(\partial B\) and \(v>0\) in B. Therefore, \(V_{\lambda _0}>0\) in \(\Sigma (\lambda _0)\), which implies that (b) of (3.25) holds. Using the maximum principle again, we see that \(Z_{\lambda _0}\equiv 0\) in \(\Sigma (\lambda _0)\). Since \(V_{\lambda _0}>0\) in \(\Sigma (\lambda _0)\), this contradicts the second equation of (3.18). Therefore, (3.24) holds.

For the case (ii), if (3.25) holds, we see that \(x_0^{\lambda _0} \ne 0\) (note that 0 is a non-removable singularity point of v). Let \(x_1 \in \Sigma (\lambda _0)\) and \(x_1^{\lambda _0}=0\). Suppose that (a) of (3.25) holds, by the maximum principle, we find that \(V_{\lambda _0} \equiv 0\) in \(\Sigma (\lambda _0) \backslash \{x_1\}\). This contradicts the fact that \(v=0\) on \(\partial B\) and \(v>0\) in B. Therefore, \(V_{\lambda _0}>0\) in \(\Sigma (\lambda _0)\backslash \{x_1\}\). Suppose that (b) of (3.25) holds, from the maximum principle, we derive that \(Z_{\lambda _0} \equiv 0\) in \(\Sigma (\lambda _0) \backslash \{x_1\}\). This also contradicts the second equation of (3.18). Therefore, (3.24) holds. Since \(V_{\lambda _0}=0\) and \(Z_{\lambda _0}=0\) on \(B \cap T_{\lambda _0}\), by (3.24) and the maximum principle, we find

$$\begin{aligned} \frac{\partial V_{\lambda _0}}{\partial x_1}>0, \;\; \frac{\partial Z_{\lambda _0}}{\partial x_1}>0 \;\; \text{ on } B \cap T_{\lambda _0}. \end{aligned}$$
(3.26)

Since \(\frac{\partial V_{\lambda _0}}{\partial x_1}=-2 \frac{\partial v}{\partial x_1}, \;\; \frac{\partial Z_{\lambda _0}}{\partial x_1}=-2 \frac{\partial z}{\partial x_1}\) on \(B \cap T_{\lambda _0}\), we have

$$\begin{aligned} \frac{\partial v}{\partial x_1}<0, \;\; \frac{\partial z}{\partial x_1}<0 \;\; \text{ on } B \cap T_{\lambda _0}. \end{aligned}$$
(3.27)

We now show that there is a sufficiently small \(\epsilon >0\) such that for \(\lambda \in (\lambda _0-\epsilon , \lambda _0)\),

$$\begin{aligned} v(x)<v(x^\lambda ), \;\;\; z(x)<z(x^\lambda ) \;\; \text{ for } x \in \Sigma (\lambda ) \end{aligned}$$
(3.28)

and

$$\begin{aligned} \frac{\partial v}{\partial x_1}<0, \;\; \frac{\partial z}{\partial x_1}<0 \;\; \text{ on } B \cap T_{\lambda }. \end{aligned}$$
(3.29)

Otherwise, there is an increasing sequence \(\{\lambda _k\}\) with \(\lambda _k \nearrow \lambda _0\) as \(k \rightarrow \infty \), such that for each k there is a point \(x_k \in \Sigma (\lambda _k)\) satisfying

$$\begin{aligned} v (x_k) \ge v(x_k^{\lambda _k}) \;\;\text{ or } \;\;z (x_k) \ge z(x_k^{\lambda _k}). \end{aligned}$$
(3.30)

Up to a subsequence, we may assume that \(\{x_k\}\) converges to a point \(x_* \in {\overline{\Sigma (\lambda _0)}}\) as \(k \rightarrow \infty \). Hence,

$$\begin{aligned} (a)\;v(x_*) \ge v(x_*^{\lambda _0}) \;\; \text{ or } \;\;(b)\; z (x_*) \ge z(x_*^{\lambda _0}). \end{aligned}$$
(3.31)

By (3.24), we see that \(x_* \in \partial \Sigma (\lambda _0)\). We claim that \(x_* \in {\overline{B}} \cap T_{\lambda _0}\). Suppose that this claim is not true, then \(x_* \in \partial B \backslash T_{\lambda _0}\) and \(x_*^{\lambda _0} \in B\), hence \(0=v(x_*)<v(x_*^{\lambda _0})\). Thus (a) of (3.31) cannot occur. Suppose that (b) of (3.31) holds, we find that \(z(x_*)=z(x_*^{\lambda _0})\). Thus, by (3.24) and the strong maximum principle, we find

$$\begin{aligned} \frac{\partial Z_{\lambda _0} (x_*)}{\partial x_1}<0. \end{aligned}$$
(3.32)

Hence there are \(\epsilon _0, \delta >0\) such that \(Z_{\lambda _0}(x-t \gamma )>Z_{\lambda _0}(x)\) for \(x\in S_{\epsilon _0}^*\), \(t \in (0, \delta )\), where \(\gamma =(1,0, \ldots , 0)\), \(S_{\epsilon _0}^*=\partial B\cap \{x\in {\mathbb {R}}^N:|x-x_*|<\epsilon _0\}\). Then we have

$$\begin{aligned} z((x-t \gamma )^{\lambda _0})>z(x-t \gamma ) \;\; \text{ for } x\in S_{\epsilon _0}^*, t \in (0, \delta ). \end{aligned}$$

This contradicts (3.30). Therefore, our claim is true.

For k large, however, the line segment joining \(x_k\) to \(x_k^{\lambda _k}\) is contained in B. Thus, by (3.30) and the mean value theorem, we obtain that there is a point \(y_k\) on this line segment such that

$$\begin{aligned} \frac{\partial v (y_k)}{\partial x_1} \ge 0 \;\; \text{ or } \;\; \frac{\partial z (y_k)}{\partial x_1} \ge 0. \end{aligned}$$
(3.33)

Since \(y_k \rightarrow x_*\) as \(k\rightarrow \infty \), we have

$$\begin{aligned} \frac{\partial v (x_*)}{\partial x_1} \ge 0 \;\; \text{ or } \;\; \frac{\partial z (x_*)}{\partial x_1} \ge 0. \end{aligned}$$
(3.34)

By (3.27), we obtain that \(x_* \in B \cap T_{\lambda _0}\) cannot occur.

We next show that (3.34) does not hold for \(x_* \in \partial B \cap T_{\lambda _0}\). Since \(v=0\) on \(\partial B\), we see that \(\nabla v=\frac{\partial v}{\partial \nu } \nu \) on \(\partial B\). Note that \(\nu _{x_*} \cdot \gamma >0\), we find

$$\begin{aligned} \frac{\partial v (x_*)}{\partial x_1}=\frac{\partial v(x_*)}{\partial \nu _{x_*}} (\nu _{x_*}, \gamma )<0. \end{aligned}$$

Thus the first case of (3.34) is impossible. To deal with the second case in (3.34), we now claim

$$\begin{aligned} \frac{\partial Z_{\lambda _0} (x_*)}{\partial x_1} \ne 0. \end{aligned}$$
(3.35)

Otherwise, by \(Z_{\lambda _0}|_{T_{\lambda _0}} \equiv 0\), we see that \(\nabla Z_{\lambda _0} (x_*)=(0, 0, \ldots ,0)\). On the other hand, by the Hopf “corner” lemma (see Lemma 1 of [45] and Lemma S of [25]), we infer that for any direction \(\mathbf{s} \) at \(x_*\) which enters \(\Sigma _{\lambda _0}\) non-tangentially,

$$\begin{aligned} \frac{\partial Z_{\lambda _0} (x_*)}{\partial \mathbf{s} }>0\;\; \text{ or } \;\;\; \frac{\partial ^2 Z_{\lambda _0} (x_*)}{\partial \mathbf{s} ^2}>0. \end{aligned}$$
(3.36)

It follows from simple calculations that \(\frac{\partial ^2 Z_{\lambda _0} (x_*)}{\partial \mathbf{s} ^2}=0\), then only (3.36)\(_1\) holds. Therefore, \(\nabla Z_{\lambda _0} (x_*) \ne (0, 0, \ldots ,0)\) and our claim (3.35) holds. From (3.26), (3.35) and the continuity of \(\frac{\partial Z_{\lambda _0}}{\partial x_1}\), we obtain

$$\begin{aligned} \frac{\partial Z_{\lambda _0} (x_*)}{\partial x_1}>0, \end{aligned}$$
(3.37)

which implies \(\frac{\partial z(x_*)}{\partial x_1}<0\). This is a contradiction with the second case of (3.34).

Thus, (3.28) and (3.29) hold. But this contradicts the definition of \(\lambda _0\). Hence, \(\lambda _0=0\) and

$$\begin{aligned} v(-x_1, x_2, \ldots , x_N) \ge v(x_1, x_2, \ldots , x_N), \;\;\; z(-x_1, x_2, \ldots , x_N) \ge z(x_1, x_2, \ldots , x_N) \end{aligned}$$

for \(x=(x_1,x_2,\ldots , x_N) \in B\) with \(x_1>0\). By reversing the direction of \(x_1\)-axis, we can also obtain

$$\begin{aligned} v(-x_1, x_2, \ldots , x_N) \le v(x_1, x_2, \ldots , x_N), \;\;\; z(-x_1, x_2, \ldots , x_N) \le z(x_1, x_2, \ldots , x_N) \end{aligned}$$

for \(x=(x_1,x_2,\ldots , x_N) \in B\) with \(x_1>0\). Therefore, v and z are symmetric to the \(x_1\)-axis. Since \(x_1\) can be an arbitrary direction and our equation is invariant under the rotations, we eventually obtain that v and z are radially symmetric about 0. This completes the proof of this lemma. \(\square \)

Lemma 3.2

Let \(N \ge 5\). Then (3.11) does not admit a radially symmetric solution with a non-removable singularity point 0.

Proof

Suppose that (3.11) admits a radial solution \(v ( y):=v(\rho )\), \(\rho =|y|\) and \(v(0)=\infty \). Under the transformations:

$$\begin{aligned} w(t)=|y|^{\frac{N-4}{2}} v(|y|), \;\;\; t=\log |y|, \end{aligned}$$
(3.38)

w(t) satisfies the equation

$$\begin{aligned} w^{(4)}(t)+K_2 w''(t)+K_0 w(t)=w(t)^{\frac{N+4}{N-4}} \;\; \text{ in } (-\infty ,0), \end{aligned}$$
(3.39)

where

$$\begin{aligned} K_2=-\frac{N^2-4N+8}{2}, \;\; K_0=\frac{N^2(N-4)^2}{16}. \end{aligned}$$
(3.40)

By Lemma 2.1 in [30], we see that if \(\lim _{t \rightarrow -\infty } w(t)=\vartheta \), then \(\vartheta \in \{0, K_0^{\frac{N-4}{8}}\}\). In the following, we show that both the cases cannot occur. If \(\lim _{t \rightarrow -\infty } w(t)=0\), it follows from the proof of Lemma 2.5 in [30] that \(w(t)=O(e^{\frac{N-4}{2} t})\) for t near \(-\infty \), which implies

$$\begin{aligned} v(\rho )=O(1) \;\;\; \text{ for } \rho \hbox { near 0}. \end{aligned}$$

This contradicts \(v(0)=\infty \).

We next show that \(\lim _{t \rightarrow -\infty } w(t)=K_0^{\frac{N-4}{8}}\) cannot hold either. Suppose that it holds, by Lemma 2.3 in [30], we find that \(\lim _{t \rightarrow -\infty } w^{(k)} (t)=0\) for all \(k \ge 1\). Let \(\hat{z}(\tau )=r^{\frac{N-4}{2}} u(r)\) where \(\tau =\log r\), \(u(r)=\rho ^{N-4} v(\rho )\), \(r=\rho ^{-1}\), we see that \(\lim _{\tau \rightarrow \infty } \hat{z}(\tau )=K_0^{\frac{N-4}{8}}\) and \(\lim _{\tau \rightarrow \infty } \hat{z}^{(k)} (\tau )=0\) for all \(k \ge 1\). Moreover, by direct calculations, we find that for r sufficiently large,

$$\begin{aligned} u(r)= & {} \Big (K_0^{\frac{N-4}{8}}+o_r (1) \Big ) r^{-\frac{N-4}{2}}, \end{aligned}$$
(3.41)
$$\begin{aligned} u'(r)= & {} \Big (-\frac{N-4}{2} K_0^{\frac{N-4}{8}}+o_r (1) \Big ) r^{-\frac{N-2}{2}}, \end{aligned}$$
(3.42)
$$\begin{aligned} u''(r)= & {} \Big (\frac{(N-4)(N-2)}{4}K_0^{\frac{N-4}{8}}+o_r (1) \Big ) r^{-\frac{N}{2}}, \end{aligned}$$
(3.43)
$$\begin{aligned} u'''(r)= & {} \Big (-\frac{N(N-2)(N-4)}{8} K_0^{\frac{N-4}{8}}+o_r (1) \Big ) r^{-\frac{N+2}{2}}. \end{aligned}$$
(3.44)

Thus, for r sufficiently large, we have

$$\begin{aligned} \Delta u (r)= & {} \Big (-\frac{N (N-4)}{4}K_0^{\frac{N-4}{8}}+o_r (1) \Big ) r^{-\frac{N}{2}}, \end{aligned}$$
(3.45)
$$\begin{aligned} (\Delta u)'(r)= & {} \Big (\frac{N^2 (N-4)}{8}K_0^{\frac{N-4}{8}} +o_r (1) \Big ) r^{-\frac{N+2}{2}}. \end{aligned}$$
(3.46)

On the other hand, by Corollary 4.2, we get

$$\begin{aligned} 0= & {} R^{N-1} \Big [\frac{N-4}{2N} R u^{\frac{2N}{N-4}}(R)-2 R u'(R) (\Delta u)'(R)+R u'(R) (\Delta u)'(R)\\&-\,(N-2) u(R) (\Delta u)'(R)+\frac{R}{2} (\Delta u)^2(R)-\frac{N}{2} (\Delta u)(R) u'(R)+\frac{N}{2} u(R) (\Delta u)'(R)\Big ]\\&+\,u'(1) (\Delta u)'(1). \end{aligned}$$

Using (3.41)–(3.46) and sending \(R \rightarrow \infty \) in the above identity, we have

$$\begin{aligned} -\frac{2}{N} K_0^{\frac{N}{4}}+u'(1) (\Delta u)'(1)=0. \end{aligned}$$

This is a contradiction with \(u'(1) (\Delta u)'(1)<0\). Therefore, \(\lim _{t \rightarrow -\infty } w(t)=K_0^{\frac{N-4}{8}}\) cannot hold.

Thus, \(w'(t)=0\) admits infinitely many roots in \((-\infty , 0)\). Moreover, by (2.10), we see that \(w(t) \le C\) for \(t \in (-\infty ,0)\). As in the proof of (c) of Proposition 3 in [20], we deduce that w is periodic, has a unique local maximum and minimum per period and is symmetric with respect to its local extrema. On the other hand, since \(w(0)=0\) and w(t) is nonnegative and periodic, we see that \(\min _{t \in (-\infty , 0)} w(t)=0\) and there is a sequence \(\{\rho _j\} \subset (0,1)\) such that \(v(\rho _j)=0\). But this is a contradiction with the fact \(\Delta v<0\) in \(B \backslash \{0\}\), which implies that w(t) cannot exist and thus \(v(\rho )\) cannot exist. This completes the proof of this lemma. \(\square \)

Lemmas 3.1 and 3.2 imply that the solution u of problem (P) with \(p=\frac{N+4}{N-4}\) does not exist. This completes the proof of the critical case of Theorem 1.1 and hence the proof of Theorem 1.1 is completed. \(\square \)

Remark 3.3

The assumption (A) in Remark 1.4 holds for radial solutions \(v \in C^4 (B \backslash \{0\}) \cap C^3 ({\overline{B}} \backslash \{0\})\) of (1.7). Indeed, it follows from Remark 1.4 that 0 is a non-removable singularity point of v. As in the proof of Proposition 2.3, we find that \(v^{\frac{N+4}{N-4}} \in L^1 (B)\). Taking advantage of the equation

$$\begin{aligned} (\rho ^{N-1} (\Delta v)' (\rho ))'=\rho ^{N-1} v^{\frac{N+4}{N-4}} \;\; \forall \rho \in (0,1), \end{aligned}$$

we have

$$\begin{aligned} \lim _{\rho \rightarrow 0} \rho ^{N-1} (\Delta v)'(\rho )=0 \end{aligned}$$

and therefore,

$$\begin{aligned} (\Delta v)'(\rho )>0 \;\; \text{ for } \rho \in (0,1). \end{aligned}$$
(3.47)

We can conclude that \((\Delta v)(1)>0\). On the contrary, we assume that \((\Delta v)(1) \le 0\). By (3.47), we see that \((\Delta v)(\rho )<0\) for \(\rho \in (0,1)\), which contradicts \(v'(1)=0\) by using the Hopf’s boundary lemma. We now claim that there is \(R \in (0,1)\) such that \((\Delta v)(\rho )<0\) for \(\rho \in (0,R)\), \((\Delta v)(R)=0\) and \((\Delta v)(\rho )>0\) for \(\rho \in (R,1)\). Suppose that such R does not exist, then, by \((\Delta v)(1)>0\), we obtain that \((\Delta v)(\rho )>0\) for \(\rho \in (0,1)\). It follows from (3.47) that \(\lim _{\rho \rightarrow 0} (\Delta v)(\rho )=\varsigma \in [0, \infty )\). This implies that 0 is a removable singularity point of v, which is a contradiction. Therefore, our claim holds. Thus, the assumption (A) in (1) of Remark 1.4 holds in \(B_R \backslash \{0\}\). Nonexistence of radial solutions v of (1.7) with a non-removable singularity point 0 can be obtained by the similar arguments in the proof Lemma 3.2. Therefore, problem (Q) with \(p=\frac{N+4}{N-4}\) does not admit any radial solution.