1 Introduction

For a fixed natural number p, let \(\mathcal {A}_p\) denote the class of analytic functions f of the form

$$\begin{aligned} f(z)=z^{p}+\sum _{n=p+1}^{\infty }a_{n}z^{n} \end{aligned}$$
(1)

defined in the unit disk \(\mathbb {D}:=\{z: |z|<1\}\). Let \(\mathcal {A}_1=:\mathcal {A}.\) The subclass of \(\mathcal {A}\) consisting of univalent functions is denoted by \(\mathcal {S}\). Finding the bound on coefficients when the function satisfies a certain given geometric condition has been a central point of research in geometric function theory (GFT). In fact, the bound on coefficients also gives rise to many geometric properties. For example, bound on the second coefficient gives growth and distortion theorems for functions in the class \(\mathcal {S}\). The estimate on \(|a_3-a_2^2|\) for the class \(\mathcal {S}\) was obtained by the Fekete–Szegö, and thereafter finding the estimate on the functional \(|a_3-\mu a_2^2|\), for any complex number \(\mu \), is popularly known as the Fekete–Szegö problem.

The Hankel determinants deal with the bound on coefficients and also give many interesting geometric properties, see [3]. For given natural numbers n and q, the Hankel determinant \(H_{q,n}(f)\) for a function \(f \in \mathcal {A}\) is defined by means of the following determinant

with \(a_{1}=1\). It is interesting to note that \(H_{2,1}(f)=a_3-a_2^2\) is the Fekete–Szegö functional. The quantity \(H_{2,2}(f)=a_2a_4-a_3^2\) is called the second Hankel determinant. The Hankel determinant \(H_{q,n}(f)\) for the class of univalent functions was investigated by Pommerenke [14] and Hayman [8]. For more development in this direction one can refer [6, 7, 10, 13,14,15].

If we set \(n=p\) and \(q=3\), then from the definition of the Hankel determinant we have \(H_{3,p}(f)=a_{p+2}(a_{p+1}a_{p+3}-a_{p+2}^2)-a_{p+3}(a_{p+3}-a_{p+1}a_{p+2})+ a_{p+4}(a_{p+2}-a^2_{p+1})\). The upper bound for \(|H_{3,p}(f)|\) is termed as the third Hankel determinant. In order to find the upper bound for the functional \(|H_{3,p}(f)|\) many researchers found the estimates on \(|a_{p+1}a_{p+3}-a_{p+2}^2|\), \( |a_{p+3}-a_{p+1}a_{p+2}|\) and \(|a_{p+2}-a^2_{p+1}|\).

Let us recall that the class of p-valent starlike functions are collection of the functions \(f\in \mathcal {A}_p\) satisfying \(\mathfrak {R}\{zf'(z)/(f(z)) \}>0\), whereas the class of p-valent convex functions is the collection of the functions \(f\in \mathcal {A}_p\) satisfying \(1+\mathfrak {R}\{ zf''(z)/f'(z) \}>0\). Estimate on the functional \(|a_{p+1}a_{p+3}-a_{p+2}^2|\) for the classes of p-valent starlike and p-valent convex functions was obtained by Krishna and Ramreddy [17]. However, for any real number \(\mu \), the sharp estimate on the functional \(|a_{p+2}-\mu a_{p+1}^2|\) for the classes of p-valent starlike and convex functions of order \(\alpha \) was obtained by Hayami and Owa [9]. A non-sharp estimate on the functional \(|a_{p+1}a_{p+2}-a_{p+3}|\) for the functions in the class

$$\begin{aligned} \mathcal {R}_p:=\left\{ f\in \mathcal {A}_p: \mathfrak {R}\left( \frac{f'(z)}{pz^{p-1}}\right) >0\right\} \end{aligned}$$

was obtained by Krishna and Ramreddy [18]. Later they [19] obtained sharp estimate on \(|a_{p+1}a_{p+3}-a_{p+2}^2|\) for a more general class

$$\begin{aligned} \mathcal {R}_p^\alpha :=\left\{ f\in \mathcal {A}_p: \mathfrak {R}\left( \frac{f'(z)}{pz^{p-1}}\right) >\alpha ,\;\; 0\leqslant \alpha <1\right\} . \end{aligned}$$

In 2015, Krishna and Ramreddy [20] considered a more general class

$$\begin{aligned} \mathcal {R}_p(\alpha ):=\left\{ f\in \mathcal {A}_p: \mathfrak {R}\left( (1-\alpha )\frac{f(z)}{z^p}+\alpha \frac{f'(z)}{pz^{p-1}}\right) >0,\;\; 0\leqslant \alpha \leqslant 1\right\} . \end{aligned}$$

They derived sharp estimate on the functional \(|a_{p+1}a_{p+3}-a_{p+2}^2|\). Motivated by their works Amourah et al. [2] obtained estimate on the functional \(|a_{p+1}a_{p+2}-a_{p+3}|\) for functions in the class \(\mathcal {R}_p(\alpha )\). They claimed that their result is sharp. But a careful check reveals that their claim is false (see Sect. 2 for details).

The main motive of this manuscript is to give sharp upper bound for the functional \(|a_{p+1}a_{p+2}-a_{p+3}|\) for functions in the class \(\mathcal {R}_p(\alpha )\). In particular, this result improves (in terms of sharpness) a result of Krishna and Ramreddy [18]. In Sect. 3, we have also derived sharp estimates on the initial coefficients and on the functionals \(|a_{p+1}a_{p+2}-a_{p+3}|\), \(|a_{p+1}a_{p+3}-a_{p+2}^2|\) and a non-sharp bound on third Hankel determinant for functions in the class

$$\begin{aligned} \mathcal {RL}_p(\alpha ):=\left\{ f\in \mathcal {A}_p: (1-\alpha )\frac{f(z)}{z^p}+\alpha \frac{f'(z)}{pz^{p-1}}\prec \sqrt{1+z}\right\} . \end{aligned}$$

Here, the symbol “\(\prec \)” means subordination.

Now we recall some results from the literature of GFT which we need to prove in our main results. Let \({\mathcal P}\) denote the class of Carathéodory [4, 5] functions of the form

$$\begin{aligned} p(z) = 1 + \sum _{n=1}^{\infty }c_n z^n, \quad z\in \mathbb {D}. \end{aligned}$$
(2)

And let \(\overline{\mathbb {D}}:= \{ z\in \mathbb {C}:|z|\leqslant 1 \}\).

The following results shall be used as tools:

Lemma 1

[11, 12, Libera and Zlotkiewicz] If \(p \in {\mathcal P}\) has the form given by (2) with \(c_1 \geqslant 0,\) then

$$\begin{aligned} 2c_{2} = c_1^2 + x\left( 4-c_1^2\right) \end{aligned}$$
(3)

and

$$\begin{aligned} 4c_{3} = c_1^3 + 2c_1\left( 4-c_1^2\right) x - c_1\left( 4-c_1^2\right) x^2 + 2\left( 4-c_1^2\right) \left( 1-|x|^{2}\right) y \end{aligned}$$
(4)

for some x and y in \(\overline{\mathbb {D}}\).

Lemma 2

[16, Lemma 2.1, Ravichandran and Verma] Let \(\beta \), \(\gamma \), \(\delta \) and \({\hat{a}}\) satisfy the inequalities \(0<\beta <1\), \(0<{\hat{a}}<1\) and

$$\begin{aligned}&8{\hat{a}}(1-{\hat{a}})[(\beta \gamma -2\delta )^2+(\beta ({\hat{a}}+\beta )-\gamma )^2]+ \beta (1-\beta )(\gamma -2{\hat{a}}\beta )^2\\&\quad \leqslant 4 {\hat{a}}\beta ^2(1-\beta )^2(1-{\hat{a}}). \end{aligned}$$

If \(p \in {\mathcal P}\) has the form given by (2), then

$$\begin{aligned} |\delta c_1^4+{\hat{a}} c_2^2+2\beta c_1 c_3-(3/2)\gamma c_1^2 c_2-c_4|\leqslant 2. \end{aligned}$$

Lemma 3

[16, Lemma 2.3, Ravichandran and Verma] Let \(p\in {\mathcal P}\). Then, for all \(n, m\in \mathbb {N}\),

$$\begin{aligned} |\mu p_np_m-p_{m+n}|\leqslant \left\{ \begin{array}{ll} 2, &{} 0 \leqslant \mu \leqslant 1; \\ 2|2\mu -1|, &{} \mathrm{{elsewhere.}} \end{array} \right. \end{aligned}$$

If \(0<\mu <1\), then the inequality is sharp for the function

$$\begin{aligned} p(z)=\frac{1+z^{m+n}}{1-z^{m+n}}. \end{aligned}$$

In the other cases, the inequality is sharp for the function

$$\begin{aligned} \hat{p}_0(z)=\frac{1+z}{1-z}. \end{aligned}$$

Lemma 4

[1, Lemma 3, R. M. Ali] Let \(p\in \mathcal {P}\) and \(0\leqslant \beta _0\leqslant 1\) and \(\beta _0(2\beta _0-1)\leqslant \delta _0\leqslant \beta _0,\) then

$$\begin{aligned} |p_3-2\beta _0 p_1p_2+\delta _0 p_1^3|\leqslant 2. \end{aligned}$$

2 The Class \(\mathcal {R}_p(\alpha )\)

Amourah et al. [2], for the function \(f\in \mathcal {R}_p(\alpha )\) with the form given by (1), proved that

$$\begin{aligned} |a_{p+1}a_{p+2}-a_{p+3}| \leqslant \Psi (f,p,\alpha )= {\left\{ \begin{array}{ll} 2, &{}\alpha =0, \\ \frac{2}{3\sqrt{6}}\frac{p(p^2+3p\alpha +6\alpha ^2)^{3/2}}{\alpha (p+\alpha )(p+2\alpha )(p+3\alpha )}, \quad &{}0<\alpha \leqslant 1. \end{array}\right. } \end{aligned}$$
(5)

They claimed that the result is sharp. Now let

$$\begin{aligned} {\tilde{p}}(z) = \frac{1+z^3}{1-z^3}, \quad z\in \mathbb {D} \end{aligned}$$
(6)

and consider the function \({\tilde{f}} \in {\mathcal A}\) defined by

$$\begin{aligned} {\tilde{f}}(z) = \frac{2}{z} \int _{0}^{z} \zeta {\tilde{p}}(\zeta ) \mathrm{d}\zeta = z + \frac{4}{5}z^{4}+\cdots . \end{aligned}$$
(7)

Then, \({\tilde{f}}\) satisfies

$$\begin{aligned} \mathfrak {R}\left\{ (1-\alpha )\frac{{\tilde{f}}(z)}{z^{p}}+\alpha \frac{{\tilde{f}}'(z)}{pz^{p-1}} \right\} = \mathfrak {R}\left\{ {\tilde{p}}(z) \right\} >0, \quad z\in \mathbb {D} \end{aligned}$$

with \(\alpha =1/2\) and \(p=1\). Therefore, the function \({\tilde{f}}\) belongs to the class \(\mathcal {R}_{1}(1/2)\). However, we can get

$$\begin{aligned} \Psi ({\tilde{f}},1,1/2) = 32\sqrt{6}/135 \approx 0.58 < 0.8 = |a_{p+1}a_{p+2}-a_{p+3}|, \end{aligned}$$

for \({\tilde{f}}\) defined by (7). Hence, Inequality (5) is not correct.

The following result gives the correct and sharp version of their result:

Theorem 1

Let \(f\in \mathcal {R}_p(\alpha )\) with the form given by (1). Then, the following inequality holds:

$$\begin{aligned} |a_{p+1}a_{p+2}-a_{p+3}| \leqslant \frac{2p}{p+3\alpha }. \end{aligned}$$
(8)

Proof

Let \(f\in \mathcal {R}_p(\alpha )\). Then, there exists a function \(p\in \mathcal {P}\) with the form given by (2) such that

$$\begin{aligned} (1-\alpha )\frac{f(z)}{z^{p}}+\alpha \frac{f'(z)}{pz^{p-1}}=p(z). \end{aligned}$$
(9)

Comparing the coefficients of similar power terms on both sides of (9), we have

$$\begin{aligned} a_{p+k}=\frac{pc_k}{p+\alpha k}, \quad k \in \mathbb {N}. \end{aligned}$$
(10)

Using (10) and setting

$$\begin{aligned} \mu :=\frac{p(p+3\alpha )}{(p+\alpha )(p+2\alpha )}, \end{aligned}$$

we can write

$$\begin{aligned} a_{p+1}a_{p+2}-a_{p+3}= & {} \frac{p}{(p+\alpha )(p+2\alpha )(p+3\alpha )}\nonumber \\&\quad \left[ p(p+3\alpha )c_1c_2-(p+\alpha )(p+2\alpha )c_3\right] \nonumber \\= & {} \frac{p}{p+3\alpha }\left[ \mu c_1c_2-c_3\right] . \end{aligned}$$
(11)

Since \(0\leqslant \alpha \leqslant 1\) and \(p\in \mathbb {N}\), it follows that \(0<\mu <1\). Noting this point and applying Lemma 3, the result follows at once from (11).

Now, for \(p\in \mathbb {N}\) and \(\alpha \in [0,1]\), consider the function \(g:\mathbb {D} \rightarrow \mathbb {C}\) in \(\mathcal {A}_p \) defined by

$$\begin{aligned} g(z) = {\left\{ \begin{array}{ll} \frac{p}{\alpha } z^{-p((1/\alpha )-1)} \int _{0}^{z} {\tilde{p}}(\zeta ) \zeta ^{ (p/\alpha )-1 } \mathrm{d}\zeta , \quad &{}\text {when} \ \alpha \not =0, \\ z^{p}{\tilde{p}}(z), &{}\text {when} \ \alpha =0, \end{array}\right. } \end{aligned}$$

where \({\tilde{p}}\) is given by (6). Then, \(a_{p+1}=0=a_{p+2}\) and \(a_{p+3}=2p/(p+3\alpha )\). Therefore, we have \( |a_{p+1}a_{p+2}-a_{p+3}|=2p/(p+3\alpha )\). This establishes sharpness of the result and completes the proof. \(\square \)

Remark 1

Recall that for functions in the class \(\mathcal {R}_p(\alpha )\), we have the sharp bounds (see, [2, 20])

$$\begin{aligned}&|a_{p+k}|\leqslant \frac{2p}{p+\alpha k},\;k\in \mathbb {N},\\&|a_{p+2}-a_{p+1}^2|\leqslant \frac{2p}{p+2\alpha } \end{aligned}$$

and

$$\begin{aligned} |a_{p+1}a_{p+3}-a_{p+2}^2|\leqslant \frac{4p^2}{(p+2\alpha )^2}. \end{aligned}$$

Therefore, we have

$$\begin{aligned} |H_{3,p}(f)|\leqslant & {} 4 p^2 \left( \frac{1}{(p+2 \alpha ) (p+4 \alpha )}+\frac{2 p}{(p+2 \alpha )^3}+\frac{1}{(p+3 \alpha )^2}\right) \\= & {} \frac{4 p^2 \left( 68 \alpha ^4+4 p^4+40 \alpha p^3+139 \alpha ^2 p^2+188 \alpha ^3 p\right) }{(p+2 \alpha )^3 (p+3 \alpha )^2 (p+4 \alpha )}. \end{aligned}$$

This estimate is better than that of proved by Amourah et al. [2, Corollary 2.4].

3 The Class \(\mathcal {RL}_p(\alpha )\)

This section deals with bound on coefficient for the functions in the class \(\mathcal {RL}_p(\alpha )\). The sharp bound on the initial coefficients, bounds on the functionals \(|a_{p+1}a_{p+2}-a_{p+3}|\) and \(|a_{p+1}a_{p+3}-a_{p+2}^2|\) for functions in the class \(\mathcal {RL}_p(\alpha )\) are obtained. Further, these estimates are used to derive bound on the third Hankel determinant.

Theorem 2

Let \(f\in \mathcal {RL}_p(\alpha )\) with the form given by (1). Then, the following sharp results hold:

  1. (i)

    \(|a_{p+1}|\leqslant \frac{p}{2(p+\alpha )},\;\; |a_{p+2}|\leqslant \frac{p}{2(p+2\alpha )},\;\;|a_{p+3}|\leqslant \frac{p}{2(p+3\alpha )}\;\;\mathrm {and}\;\;|a_{p+4}|\leqslant \frac{p}{2(p+4\alpha )}\).

  2. (ii)

    \(|a_{p+2}-a_{p+1}^{2}| \leqslant \frac{p}{2(p+2\alpha )},\;\; |a_{p+1}a_{p+3}-a_{p+2}^2|\leqslant \frac{p^2}{4(p+2\alpha )^2} \;\;\mathrm {and}\;\; |a_{p+1}a_{p+2}-a_{p+3}|\leqslant \frac{p}{2(p+3\alpha )}.\)

Proof

Let \(f\in \mathcal {RL}_p(\alpha )\). Then, there exists a function \(p\in \mathcal {P}\) with the form given by (2) such that

$$\begin{aligned} (1-\alpha )\frac{f(z)}{z^{p}}+\alpha \frac{f'(z)}{pz^{p-1}}=\sqrt{\frac{2p(z)}{p(z)+1}}. \end{aligned}$$
(12)

Comparing the coefficients on the both sides of (12), we have

$$\begin{aligned} a_{p+1}= & {} \frac{p}{4(p+\alpha )}c_1,\ \end{aligned}$$
(13)
$$\begin{aligned} a_{p+2}= & {} \frac{p}{32(p+2\alpha )}\left[ 8c_2-5c_1^2\right] , \end{aligned}$$
(14)
$$\begin{aligned} a_{p+3}= & {} \frac{p}{128(p+3\alpha )}\left[ 32c_3-40c_1c_2+13c_1^3\right] \end{aligned}$$
(15)

and

$$\begin{aligned} a_{p+4} = \frac{-p}{2048(p+4\alpha )} \left[ 141c_{1}^{4} - 624c_{1}^{2}c_{2} + 640c_{1}c_{3} + 320c_{2}^{2} -512c_{4} \right] .\quad \end{aligned}$$
(16)

(i) Estimates on \(|a_{p+1}|\) and \(|a_{p+2}|\) can be obtained by using the well-known estimate \(|c_{n}|\leqslant 2\), and for any complex number \(\mu \), the result \(|c_2-\mu c_1^2|\leqslant 2\max \{1; |2\mu -1\}\), respectively. To find the estimate on \(|a_{p+3}|\), we represent \(a_{p+3}\) as follows:

$$\begin{aligned} a_{p+3}= & {} \frac{p}{128(p+3\alpha )}\left[ 32c_3-40c_1c_2+13c_1^3\right] \nonumber \\= & {} \frac{p}{4(p+3\alpha )}\left[ c_3-2\beta _0c_1c_2+\delta _0c_1^3\right] , \end{aligned}$$
(17)

where \(\beta _0=5/8\) and \(\delta _0=13/32.\) With these settings, it can be easily verified that all the conditions of Lemma 4 are satisfied. Now application of Lemma 4 on (17) gives the desired result.

To find the upper bound for \(|a_{p+4}|\), from (16), we have

$$\begin{aligned} \left( \frac{p+4\alpha }{p} \right) |a_{p+4}| = \frac{1}{4} \left| \frac{141}{512}c_{1}^{4} - \frac{39}{32}c_{1}^{2}c_{2} + \frac{5}{4} c_{1}c_{3} + \frac{5}{8}c_{2}^{2} - c_{4} \right| . \end{aligned}$$

Now we use Lemma 2 with

$$\begin{aligned} {\hat{a}}=\beta =\frac{5}{8}, \quad \gamma =\frac{13}{16} \quad \text {and} \quad \delta =\frac{141}{512} \end{aligned}$$

in

$$\begin{aligned} \left( \frac{4(p+4\alpha )}{p} \right) |a_{p+4}| = \left| \frac{141}{512}c_{1}^{4} - \frac{39}{32}c_{1}^{2}c_{2} + \frac{5}{4} c_{1}c_{3} + \frac{5}{8}c_{2}^{2} - c_{4} \right| . \end{aligned}$$
(18)

Then, we see that all the conditions of Lemma 2 are satisfied. Indeed, we have

$$\begin{aligned}&8{\hat{a}}(1-{\hat{a}})[(\beta \gamma -2\delta )^2)\\&\quad +\,(\beta ({\hat{a}}+\beta )-\gamma )^2]+ \beta (1-\beta )(\gamma -2{\hat{a}}\beta )^2\\&\quad -\,4 {\hat{a}}\beta ^2(1-\beta )^2(1-{\hat{a}}) = - \frac{20385}{524288} <0. \end{aligned}$$

Thus,

$$\begin{aligned} \left| \frac{141}{512}c_{1}^{4} - \frac{39}{32}c_{1}^{2}c_{2} + \frac{5}{4} c_{1}c_{3} + \frac{5}{8}c_{2}^{2} - c_{4} \right| \leqslant 2 \end{aligned}$$

and therefore, the result follows at once from (18).

Now, for \(p\in \mathbb {N}\), \(\alpha \in [0,1]\) and \(k=1,2,3,4\), consider the functions \(g_{k}:\mathbb {D}\rightarrow \mathbb {C}\) in \(\mathcal {A}_p\) defined by

$$\begin{aligned} g_{k}(z) = {\left\{ \begin{array}{ll} \frac{p}{\alpha } z^{-p((1/\alpha )-1)} \int _{0}^{z} \zeta ^{ (p/\alpha )-1 } \sqrt{1+\zeta ^{k}} \mathrm{d}\zeta , \quad &{}\text {when} \ \alpha \not =0, \\ z^{p} \sqrt{1+z^k}, &{}\text {when} \ \alpha =0. \end{array}\right. } \end{aligned}$$
(19)

Then, we can easily check that the equalities in (i) hold for \(g_{1}\), \(g_{2}\), \(g_{3}\) and \(g_{4}\in \mathcal {A}_p\), respectively. This shows that the inequalities in (i) are sharp.

(ii) To find the estimate on \(|a_{p+2}-a_{p+1}^{2}|\), we note that

$$\begin{aligned} |a_{p+2}-a_{p+1}^{2}| = \frac{p}{4(p+2\alpha )} \left| c_{2}-\nu c_{1}^{2}\right| , \end{aligned}$$
(20)

where

$$\begin{aligned} \nu = \frac{7p^2+14\alpha p+5\alpha ^2}{8(p+\alpha )^{2}}. \end{aligned}$$

We can easily check \(0< 2\nu -1 <1\) for all \(p \in \mathbb {N}\) and \(\alpha \in [0,1]\). Therefore, by applying the well-known inequality \(\left| c_2- \nu c_1^2\right| \leqslant 2 \max \{1;|2 \nu -1|\}\) in (20), we can obtain the first inequality in (ii). Furthermore, the equality holds for the function \(g_{2} \in {\mathcal A}_{p}\) defined by (19) with \(k=2\).

We now find the estimate on \(|a_{p+1}a_{p+3}-a_{p+2}^2|\). Let us denote

$$\begin{aligned} M:=\frac{p^2}{512(p+\alpha )(p+3\alpha )}\;\;{\text {and}}\;\;N:=\frac{p^2}{1024(p+2\alpha )^2}. \end{aligned}$$

Then, using (13), (14) and (15) we can write

$$\begin{aligned} a_{p+1}a_{p+3}-a_{p+2}^2= & {} M\left( 32c_3-40c_1c_2+13c_1^3\right) c_1-N\left( 8c_2-5c_1^2\right) ^2 \\= & {} (13M-25 N)c_1^4+40(2N-M)c_2c_1^2-64Nc_2^2+32Mc_1c_3. \end{aligned}$$

Now using Lemma 1, and setting \(c=c_1\in [0, 2]\), after a computation we have

$$\begin{aligned} a_{p+1}a_{p+3}-a_{p+2}^2= & {} (M-N)c^4+4(2N-M)c^2(4-c^2)x\nonumber \\&+\,8((2N-M)c^2-8N)(4-c^2)x^2\nonumber \\&+\,16Mc(4-c^2)(1-|x|^2)y \end{aligned}$$
(21)

with x and \(y \in \overline{\mathbb {D}}\). Using the triangle inequality in (21) and using the fact \(|y|\leqslant 1\), we can write

$$\begin{aligned} \left| a_{p+1}a_{p+3}-a_{p+2}^2\right|\leqslant & {} |(M-N)c^4+4(2N-M)c^2(4-c^2)x\nonumber \\&+\,8((2N-M)c^2-8N)(4-c^2)x^2|\nonumber \\&+\,16Mc(4-c^2)(1-|x|^2). \end{aligned}$$
(22)

Now we shall find the maximum of right-hand side of (22), for \(c\in [0, 2]\). We claim that \((2N-M)c^2-8N<0\) for all \(c\in [0, 2]\). Consider

$$\begin{aligned}&(2N-M)c^2-8N \\&\quad = \left( \frac{2p^2}{1024(p+2\alpha )^2}-\frac{p^2}{512(p+\alpha )(p+3\alpha )} \right) c^{2} -\frac{8p^2}{1024(p+2\alpha )^2} \\&\quad = \frac{p^2}{512}\left( \left( \frac{1}{(p+2\alpha )^2}-\frac{1}{(p+\alpha )(p+3\alpha )}\right) c^{2} -\frac{4}{(p+2\alpha )^2}\right) \\&\quad = -\frac{p^2}{512} \left( \frac{\alpha ^2c^2}{(p+\alpha )(p+2\alpha )^2(p+3\alpha )}+\frac{4}{(p+2\alpha )^2}\right) \\&\quad <0. \end{aligned}$$

This establishes our claim.

It should also be noted that \(M>N\). Further computation shows that \(2N\leqslant M\) and \(8N-(2N-M)c^2-2Mc \geqslant 0\) for all \(c\in [0,2]\). Therefore, from (22), we have

$$\begin{aligned} \left| a_{p+1}a_{p+3}-a_{p+2}^2\right|\leqslant & {} (M-N)c^4-4(2N-M)c^2(4-c^2)|x|\\&+\,8(8N-(2N-M)c^2)(4-c^2)|x|^2\\&-\,16Mc(4-c^2)|x|^2+16Mc(4-c^2), \end{aligned}$$

or equivalently

$$\begin{aligned} \left| a_{p+1}a_{p+3}-a_{p+2}^2\right|\leqslant & {} (M-N)c^4-4(2N-M)c^2(4-c^2)|x|\\&+\,\left[ 8(8N-(2N-M)c^2)-16Mc\right] (4-c^2)|x|^2\\&+\,16Mc(4-c^2). \end{aligned}$$

Now using the facts \(|x|\leqslant 1\) and \(8N-(2N-M)c^2-2Mc>0\), we have

$$\begin{aligned} \left| a_{p+1}a_{p+3}-a_{p+2}^2\right|\leqslant & {} (M-N)c^4+4[3(M-2N)c^2+16N](4-c^2) \\= & {} (23N-11M)c^4+16(3M-10N)c^2+256N=:\psi _{2}(c). \end{aligned}$$

Since \(23N-11M \leqslant 0\), it follows from the inequality \(3M-10N<0\) that \(\psi _{2}\) attains its maximum at \(c=0\), whereas for the case \(23N-11M > 0\), the function \(\psi _{2}\) is convex in [0, 2] and therefore we have

$$\begin{aligned} \psi _{2}(c) \leqslant \max \{ \psi _{2}(0), \psi _{2}(2) \} = \psi _{2}(0), \quad c\in [0,2], \end{aligned}$$

since

$$\begin{aligned} \psi _{2}(0)-\psi _{2}(2) = \frac{ p^2(15p^2 +60\alpha p +43\alpha ^2) }{ 64(p+\alpha )(p+2\alpha )^2(p+3\alpha ) } >0, \quad p \in \mathbb {N}, \ \alpha \in [0,1]. \end{aligned}$$

Therefore, we have the following inequality

$$\begin{aligned} \left| a_{p+1}a_{p+3}-a_{p+2}^2\right| \leqslant \frac{p^2}{4(p+2\alpha )^2}. \end{aligned}$$

The function \(g_{2}\in \mathcal {A}_p\) defined by (19) with \(k=2\) reveals that the result is sharp.

Now it remains to find the estimate on \(|a_{p+1}a_{p+2}-a_{p+3}|\). Let us denote

$$\begin{aligned} \tilde{M}:=\frac{p^2}{128(p+\alpha )(p+2\alpha )}\;\;\text {and}\;\; \tilde{N}:=\frac{p}{128(p+3\alpha )}. \end{aligned}$$

From (13), (14) and (15), we have

$$\begin{aligned} \nonumber a_{p+1}a_{p+2}-a_{p+3}= & {} \tilde{M}\left[ 8c_2-5c_1^2\right] c_1- \tilde{N}\left[ 32c_3-40c_1c_2+13c_1^3\right] \\= & {} -32\tilde{N} \left[ c_3-2\beta 'c_1c_2+\delta ' c_1^3\right] , \end{aligned}$$
(23)

where

$$\begin{aligned} \beta '=\frac{\tilde{M}+5 \tilde{N}}{8\tilde{N}}\;\;\text {and}\;\;\delta '=\frac{5 \tilde{M}+13 \tilde{N}}{8\tilde{N}}. \end{aligned}$$

It is easy to see that the conditions \(\beta '(2\beta '-1)\leqslant \delta '\) and \(\delta '\leqslant \beta '\) are equivalent to the conditions \(\tilde{M}^2\leqslant 7 \tilde{N}^2\) and \(\tilde{M}\leqslant 7 \tilde{N}\), respectively. Of course these hold as \(\tilde{M}\leqslant \tilde{N}\) for all \(\alpha \in [0, 1]\) and \(p\in \mathbb {N}\). Now application of Lemma 4 gives the desired estimate.

The function \(g_{3}\in \mathcal {A}_p\) defined by (19) with \(k=3\) gives us that the result is sharp.

Remark 2

As the similar arguments in Remark 1, we can get an upper bound of \(|H_{3,p}(f)|\) for \(f \in \mathcal {RL}_p(\alpha )\) as follows:

$$\begin{aligned} |H_{3,p}(f)| \leqslant \frac{p^2}{8} \left( \frac{p}{(p+2\alpha )^{3}} + \frac{2}{(p+3\alpha )^{2}} + \frac{2}{(p+2\alpha )(p+4\alpha )} \right) . \end{aligned}$$