1 Introduction and preliminaries

Let \({\mathcal {H}}\) denote the family of analytic functions in the open unit disk \({\mathbb {D}}=\{z\in {\mathbb {C}}:|z|<1\}\), and \({\mathcal {A}}\) denote the class of functions \(f \in {\mathcal {H}}\) such that

$$\begin{aligned} f(z) = z + a_2z^2+a_3z^3+\cdots , \quad z \in {\mathbb {D}}. \end{aligned}$$
(1.1)

We denote by \({\mathcal {S}}\) the functions f in \({\mathcal {A}}\) that are univalent in \({\mathbb {D}}\). A function \(f \in {\mathcal {A}}\) is called starlike, if f is univalent in \({\mathbb {D}}\) and \(f(\mathbb {D)}\) is a starlike domain with respect to the origin. Analytically, \(f \in {\mathcal {S}}\) is called starlike, if and only if \(\Re \{zf'(z)/f(z)\}>0\), \(z\in {\mathbb {D}}\). A function \(f\in {\mathcal {S}}\) is called convex, if and only if \(zf'(z)\in {\mathcal {S}}^*\). The class of starlike functions and the class of convex functions are denoted, respectively, by \({\mathcal {S}}^*\) and \({\mathcal {K}}\).

Let \({\mathcal {S}}_*\) and \({\mathcal {K}}_*\), denote the class of functions \(f\in {\mathcal {A}}\) which are stalike and convex of reciprocal orders, respectively. Analytically, \(f \in {\mathcal {S}}\) is called starlike of reciprocal order, if and only if \(\Re \{f(z)/zf'(z)\}>0\), \(z\in {\mathbb {D}}\). Also, a function \(f\in {\mathcal {S}}\) is called convex of reciprocal order, if and only if \(zf'(z)\in {\mathcal {S}}_*\) which analytically is represented by \(\Re \{f'(z)/(zf'(z))'\}>0\). Various authors have studied the classes \({\mathcal {S}}_*\) and \({\mathcal {K}}_*\) and one may see, e.g. the works in [3, 21, 24] and [26].

For \(f\in {\mathcal {A}}\) of the form (1.1), \(\Phi _\lambda (f):=a_3-\lambda a_2^2\) is the classical Fekete-Szegö functional. A classical problem settled by Fekete and Szegö [10] is to find for each \(\lambda \in [0,1]\) the maximum value of \(|\Phi _\lambda (f)|\) over the function \(f\in {\mathcal {S}}\), and they proved that

$$\begin{aligned} \max _{f\in {\mathcal {S}}}\left| \Phi _\lambda (f)\right| =\left\{ \begin{array}{ll} 1+2\,\exp \{-2\lambda /(1-\lambda )\}, &{} \lambda \in [0,1),\\ 1,&{} \lambda =1. \end{array}\right. \end{aligned}$$

The problem of calculating the maximum of \(\left| \Phi _\lambda (f)\right| \) for various subfamilies of \({\mathcal {A}}\), as well as \(\lambda \) being an arbitrary real or complex number, was considered by many authors (see, e.g. [1, 7, 14, 15, 19]).

The Hankel determinant of Taylor coefficients for functions \(f \in {\mathcal {A}}\) of the form (1.1), is denoted by \(H_{q,n}(f)\) and is defined by

$$\begin{aligned} H_{q,n}(f) := \left| \begin{array}{cccc} a_n &{} a_{n+1}&{} \cdots &{}a_{n+q-1}\\ a_{n+1}&{}a_{n+2} &{}\cdots &{}a_{n+q}\\ \vdots &{} \vdots &{}\vdots &{}\vdots \\ a_{n+q-1} &{}a_{n+q} &{}\cdots &{}a_{n+2(q-1)} \end{array} \right| , \end{aligned}$$
(1.2)

where \(a_1=1; \;n, q \in {\mathbb {N}}=\{1, 2, \ldots \}\). Several researchers including Noonan and Thomas [22], Pommerenke [25], Hayman [12], Ehrenborg [9], Noor [23] and many others have studied the Hankel determinant and have given some remarkable results which are useful, for example, in showing the bounded characteristic of a function in \({\mathbb {D}}\).

For \(f\in {\mathcal {A}}\) of the form (1.1), \(H_{2,1}(f):=\Phi _1(f)= a_3-a_2^2\,\) is the Fekete-Szegö functional. Furthermore, the upper bound of the second Hankel determinant \(H_{2,2}(f)\) for various subclasses of \({\mathcal {A}}\) has been studied by many authors (see, e.g. [2, 4, 6, 13, 16]). The third Hankel determinant \(\,H_{3,1}(f)\,\) is defined by

(1.3)

Recently, the author has studied the bounds on \(\,|\mathbf{H}(f)|\,\) for certain classes of analytic functions (see [5, 20]). In the current article, the upper bound of the initial coefficients and the bounds on \(|\mathbf{H}(f)|\) are studied for functions belonging to the classes \(\mathcal {S_*}\) and \(\mathcal {K_*}\) as stated above. In our study we shall need the class \({\mathcal {P}}\) of Carathéodory functions [8] which is defined below.

Let \({\mathcal {P}}\) denote the class of analytic functions in \({\mathbb {D}}\) with \(\Re (p(z))>0\) of the form

$$\begin{aligned} p(z)= 1+ c_1z+c_2z^2+\cdots , \quad z \in {\mathbb {D}}. \end{aligned}$$
(1.4)

It is well known [8] that for the function \(p \in {\mathcal {P}}\) of the form (1.4), \(\,|c_n| \le 2\), for all \(n\ge 1\). This inequality is sharp and the equality holds for the function \(\displaystyle \varphi (z)=(1+z)/(1-z)\).

The power series (1.4) converges in \(\,{\mathbb {D}}\) to a function in \(\,{\mathcal {P}}\), if and only if the Toeplitz determinants

$$\begin{aligned} T_{n}(p) = \left| \begin{array}{ccccc} 2 &{} c_1 &{} c_2 &{} \cdots &{} c_n\\ c_{-1} &{} 2 &{} c_1 &{} \cdots &{} c_{n-1}\\ c_{-2} &{} c_{-1} &{} 2 &{} \cdots &{} c_{n-2}\\ \vdots &{} \vdots &{} \vdots &{} \ddots &{} \vdots \\ c_{-n} &{} c_{-n+1} &{} c_{-n+2} &{} \cdots &{} 2 \end{array} \right| ,\quad n \in {\mathbb {N}} \end{aligned}$$

and \(c_{-n}=\overline{c}_n\), are all nonnegative. The only exception is when p(z) has the form

$$\begin{aligned} p(z)=\sum _{\nu =1}^m \rho _\nu \frac{1+\epsilon _\nu z}{1-\epsilon _\nu z}, \quad m\ge 1, \end{aligned}$$

where \(\rho _\nu >0\), \(|\epsilon _\nu |=1\), and \(\epsilon _k\ne \epsilon _l\) if \(k\ne l\); \(k,\,l=1,2,\ldots ,m\); we have then \(\,T_{n}(p)>0\,\) for \(\,n < m-1\,\) and \(\,T_{n}(p)=0\,\) for \(n \ge m\). This necessary and sufficient condition is due to Carathéodory and Toeplitz and can be found in [11]. In particular, for \(n=2\), we have

$$\begin{aligned} T_{2}(p) = \left| \begin{array}{ccc} 2 &{}c_1 &{}c_2 \\ \overline{c}_1 &{}2 &{}c_1\\ \overline{c}_2 &{}\overline{c}_1 &{}2 \end{array} \right| = 8+ 2\,\Re \{c_1^2\overline{c}_2\}-2|c_2|^2-4|c_1|^2 \ge 0, \end{aligned}$$

which is equivalent to

$$\begin{aligned} 2c_2=c_1^2+x(4-c_1^2) \end{aligned}$$
(1.5)

for some \(\, x\,\) with \(\,|x|\le 1.\) Similarly, if

$$\begin{aligned} T_{3}(p) = \left| \begin{array}{cccc} 2 &{}c_1 &{}c_2 &{}c_3\\ \overline{c}_1 &{}2 &{}c_1 &{}c_2\\ \overline{c}_2 &{}\overline{c}_1 &{}2 &{}c_1 \\ \overline{c}_3 &{}\overline{c}_2 &{}\overline{c}_1 &{}2 \end{array} \right| , \end{aligned}$$

then \(T_{3}(p) \ge 0\) is equivalent to

$$\begin{aligned} |(4c_3-4c_1c_2+c_1^3)(4-c_1^2)+c_1(2c_2-c_1^2)^2| \le 2(4-c_1^2)^2-2|(2c_2-c_1^2)|^2. \end{aligned}$$
(1.6)

Solving (1.6) with the help of (1.5), we get

$$\begin{aligned} 4c_3=c_1^3+2c_1x(4-c_1^2)-c_1x^2(4-c_1^2)+2(4-c_1^2)(1-|x|^2)z, \end{aligned}$$
(1.7)

for some \(\, x \; and \; z\,\) with \(\,|x|\le 1\,\) and \(\,|z|\le 1.\) Furthermore, the following well-known results are being useful to obtain our main results.

Lemma 1.1

[14] If \(p\in {\mathcal {P}}\), then for any complex number \(\nu \),

$$\begin{aligned} |c_2-\nu c_1^2|\le 2\; max\{1,|2\nu -1|\}, \end{aligned}$$

and equality holds for the functions given by

$$\begin{aligned} \psi (z)= \frac{1+z^2}{1-z^2} \;\; \mathrm{and}\;\; \varphi (z)= \frac{1+z}{1-z}. \end{aligned}$$

Lemma 1.2

[18] Let the function p given by (1.4) is in the class \({\mathcal {P}}\). Then for all \(\,n\,\) and \(\,s\,\) \((1\le s<n)\), we have \(\,|c_n-c_sc_{n-s}| \le 2\).

Lemma 1.3

[11] (See also [17, Lemma 3, p. 227]) If \(p\in {\mathcal {P}}\), then the following expressions are all bounded by 2, and are all sharp:

  1. 1.

    \(|c_1^2-c_2|\),

  2. 2.

    \(|c_3-2c_1c_3+c_3|\),

  3. 3.

    \(|c_1^4+2c_1c_3+c_2^2-3c_1^2c_2-c_4|\),

  4. 4.

    \(|c_1^5+3c_1c_2^2+3c_1^2c_3-4c_1^3c_2-2c_1c_4-2c_2c_3+c_5|\)

  5. 5.

    \(|c_1^6+6c_1^2c_2^2+4c_1^3c_3+2c_1c_5+2c_2c_4+c_3^2-c_2^3-5c_1^4c_2-3c_1^2c_4-6c_1c_2c_3-c_6|\).

The following inequalities can also be obtained in the proof of a result in [17, p. 227–228]

  1. a.

    \(|2c_1^2-c_2|\le 6\)

  2. b.

    \(|-6c_1^3+7c_1c_2-2c_3|\le 24\)

  3. c.

    \(|24c_1^4-46c_1^2c_2+22c_1c_3+7c_2^2-6c_4|\le 120\)

  4. d.

    \(|-120c_1^5+96c_4c_1+50c_2c_3+326c_1^3c_2-202c_1^2c_3-127c_1c_2^2-24c_5|\le 720\).

Lemma 1.4

[27, Lemma 2.3, p. 507] Let \(p\in {\mathcal {P}}\). Then for all \(n,m\in {\mathbb {N}}\),

$$\begin{aligned} |\mu c_n c_m-c_{m+n}|\le \left\{ \begin{array}{ll} 2,\, &{} \mu \in [0,1],\\ 2\;|2\mu -1|, &{} elsewhere. \end{array}\right. \end{aligned}$$

If \(0<\mu <1\), then the inequality is sharp for the function \(p(z)=(1+z^{m+n})/(1-z^{m+n})\). In other cases, the inequality is sharp for the function \(p(z)=(1+z)/(1-z)\).

2 Set of coefficient bounds belonging to the class \(\mathcal {S_*}\)

The following is our first result in this section.

Theorem 2.1

Let \(f\in \mathcal {S_*}\) be given by (1.1), then \(|a_n|\le n\) for \(n=2,\,3,\,4\). The result is sharp for the function \(e_1(z)=z(1+z)^{-2}.\)

Proof

Let us consider \(f\in \mathcal {S_*}\). Then by the definition, we have

$$\begin{aligned} f(z) = zf'(z) \,p(z), \end{aligned}$$
(2.1)

where \(p\in {\mathcal {P}}\) is of the form (1.4). Substituting the series expansion of f(z), \(f'(z)\) and p(z) in (2.1) and equating the coefficients, we get

$$\begin{aligned} a_n = \frac{1}{1-n}\left( c_{n-1}+2a_2c_{n-2}+3a_3c_{n-3}+\cdots +(n-1)a_{n-1}c_1\right) , \end{aligned}$$

which in particular gives us

$$\begin{aligned} \begin{array}{l} a_2 = -c_1, \; a_3=\dfrac{1}{2}(2 c_1^2-c_2),\; a_4= \dfrac{1}{6}(7c_1c_2-2c_3-6c_1^3),\\ \text{ and } \\ a_5 = \dfrac{1}{24}(24c_1^4-46c_1^2c_2+20c_1c_3+9c_2^2-6c_4). \end{array} \end{aligned}$$
(2.2)

Bounds for \(|a_2|\) is obvious as \(|c_1|\le 2\). Bounds for \(|a_3|\) and \(|a_4|\) can be directly obtained by using (a) and (b) of Lemma 1.3 . Furthermore, by using (1.5) and (1.7) in (2.2) for some x and z such that \(|x| \le 1\) and \(|z|\le 1\), we obtain

$$\begin{aligned} |a_3|=\dfrac{1}{4}\left| 3c_1^2-x(4-c_1^2)\right| \end{aligned}$$

and

$$\begin{aligned} |a_4|=\dfrac{1}{12}\left| -6c_1^3+(4-c_1^2)\{5c_1x+c_1x^2-2(1-|x|^2)z\}\right| . \end{aligned}$$

To show the sharpness, let us set \(c_1=2\) and \(x=1\) in (1.5) and (1.7), we obtain \(c_2=c_3=2\). Using these values in the above relations, we find that the result is sharp and the extremal function would be \(e_1(z)=z(1+z)^{-2}\). This completes the proof of Theorem 2.1. \(\square \)

Theorem 2.2

Let \(f\in \mathcal {S_*}\) be given by (1.1), then \(|a_5|\le 39/7\).

Proof

If \(f\in \mathcal {S_*}\), then by using the value of \(a_5\) from (2.2), we obtain

$$\begin{aligned} |a_5|= & {} \frac{1}{24}\left| 24c_1^4-46c_1^2c_2+14c_1c_3+9c_2^2-6(c_4-c_1c_3)\right| . \end{aligned}$$

By using (1.5) and (1.7) for some x and z such that \(|x| \le 1\) and \(|z| \le 1\), we get

$$\begin{aligned} |a_5|= & {} \frac{1}{96}\left| 27c_1^4+(4-c_1^2)\{-46c_1^2x-23c_1^2x^2+28c_1(1-|x|^2)z+36x^2\}\right. \\&\left. -24(c_4-c_1c_3)\right| . \end{aligned}$$

If \(p(z)\in {\mathcal {P}}\), then \(p(e^{i\alpha }z)\in {\mathcal {P}}\). We can always select a real \(\alpha \) in \(p(e^{i\alpha }z)\) so that \(c_n e^{i\alpha n}\ge 0\). Hence we may suppose that \(c_n\ge 0 \; (n\in {\mathbb {N}})\). Furthermore, the power series (1.4) converges in \(\,{\mathbb {D}}\) to a function in \({\mathcal {P}}\), if and only if the Toeplitz determinants \(T_{n}(p)\) and \(c_{-n}=\overline{c}_n\) are all nonnegative, i.e. \(c_1\) is real, \(c_1\ge 0\) and \(|c_1|\le 2\). Therefore, letting \(c_1=c\), we may assume without restriction that \(c\in [0,\, 2]\). Hence, applying the triangle inequality with \(\mu =|x|\), and applying Lemma 1.2, we obtain

$$\begin{aligned} |a_5|\le \frac{1}{96}\left[ 27c^4+(4-c^2)\{46c^2\mu +23c^2\mu ^2+28c(1-\mu ^2)+36\mu ^2\}+48\right] := D(c,\mu ). \end{aligned}$$

Now we need to maximize \(D(c,\mu )\) on the region \(\Omega =\{(c,\mu ): 0\le c \le 2 \;\text{ and }\;0 \le \mu \le 1\}\). For this, first we estimate

$$\begin{aligned} \frac{\partial D}{\partial \mu }=\frac{1}{48}\left[ (4-c^2)\{23c^2(1+\mu )+4\mu (9-7c)\}\right] . \end{aligned}$$

For \(0< \mu <1\), and for fixed c with \(0< c < 2\), we observe that \(\frac{\partial D}{\partial \mu } > 0\). Therefore, \(D(c,\mu )\) becomes an increasing function of \(\mu \), and hence it cannot have a maximum value at any point in the interior of the closed region \(\Omega \). Moreover, for a fixed \(c\in [0,2]\), we have

$$\begin{aligned} \max _{\mu \in [0,1]}D(c,\mu )= D(c,1) = \frac{1}{16}(-7c^4+40c^2+32). \end{aligned}$$

Therefore, by the second derivative test we can see that D(c, 1) has maximum value at c, where \(c^2=20/7\).

Furthermore, if we look for the critical points on the boundary of \(\Omega \), we estimate

$$\begin{aligned} \frac{\partial D}{\partial c}= & {} \frac{1}{48}[54c^3+23c\mu (4-c^2)(2+\mu )-c(46c^2\mu +23c^2\mu ^2+36\mu ^2)\\&+14(1-\mu ^2)(4-3c^2)]. \end{aligned}$$

Now we look for the critical point of \(D(c,\mu )\) which must satisfy \(\frac{\partial D}{\partial \mu }=0\) and \(\frac{\partial D}{\partial c}=0\), and one can check easily that the points \((c,\mu )\) satisfying such conditions are not interior point of \(\Omega \). So the maximum cannot attain in the interior of \(\Omega \). Now to see on the boundary, taking the boundary line \(L_1=\{(2, \mu ): 0\le \mu \le 1\}\), we have \(D(2,\mu )=5\) which is a constant. Along \(L_2=\{(0, \mu ): 0\le \mu \le 1\}\), we have \(D(0,\mu )=(1+3\mu ^2)/2\), which gives the critical point (0, 0). Along \(L_3=\{(c, 1): 0\le c \le 2\}\), we have \(D(c,1)=(-7c^4+40c^2+32)/16\), which gives the critical points (0, 1) and \((\sqrt{20/7},1)\). Along \(L_4=\{(c, 0): 0\le c \le 2\}\), we have \(D(c,0)=(27c^4-28c^3+112c+48)/96,\) which gives no critical points in \(\Omega \). Observe that

$$\begin{aligned} D(0, 0)<D(0,1)<D(2,\mu )<D(\sqrt{20/7}, 1). \end{aligned}$$

Hence

$$\begin{aligned} \max _{\Omega }D(c,\mu )=D(\sqrt{20/7}, 1)=39/7. \end{aligned}$$

This completes the proof. \(\square \)

Remark 2.3

For \(f\in \mathcal {S_*}\) of the form (1.1), Arif et al. [3, Corollary 7] obtained that

$$\begin{aligned} |a_2|\le 2 \quad \mathrm{and} \quad |a_n|\le \frac{2}{n-1}\prod _{k=2}^{n-1}\left( \frac{3k-1}{k-1}\right) \quad (n=3,4,5,\ldots ). \end{aligned}$$

Here we observe that, our result obtained in Theorem 2.1 and Theorem 2.2 provides the improvement in the upper bound of the initial coefficients \(a_n,\; n=3,4,5 \).

Theorem 2.4

Let \(f\in \mathcal {S_*}\) be given by (1.1), then

$$\begin{aligned} |a_3-a_2^2|\le 1, \quad |a_2a_3-a_4|\le 2 \quad \mathrm{and }\quad |a_2a_4-a_3^2| \le 1. \end{aligned}$$
(2.3)

Inequalities in (2.3) are sharp for the extremal function given by \(e_1(z)=z(1+z)^{-2}\).

Proof

If \(f\in \mathcal {S_*}\), then the values of \(a_2\), \(a_3\) and \(a_4\) are given in (2.2). Using these values, we obtain

$$\begin{aligned} |a_3-a_2^2| = \left| \frac{-c_2}{2}\right| , \quad |a_2 a_3-a_4| = \frac{1}{3} \left| -2c_1 c_2+c_3\right| \end{aligned}$$

and

$$\begin{aligned} |a_2a_4-a_3^2|= \frac{1}{24}\left| -4c_1^2c_2+8c_1c_3-6c_2^2\right| . \end{aligned}$$

Clearly, it follows that \(|a_3-a_2^2| = \left| c_2/2 \right| \le 1\). Now by using Lemma 1.4, we obtain

$$\begin{aligned} |a_2 a_3-a_4| \le \frac{1}{3} \left| 2c_1 c_2-c_3\right| \le \frac{1}{3}\left[ 2~ |2\cdot 2-1|\right] =2. \end{aligned}$$

Furthermore, by using (1.5) and (1.7) for some x and z such that \(|x| \le 1\) and \(|z| \le 1\), we get

$$\begin{aligned} |a_2a_4-a_3^2|= & {} \dfrac{1}{48}\left| (4-c_1^2)[-2c_1^2x-4c_1^2x^2-3x^2(4-c_1^2)\right. \nonumber \\&\left. +8c_1(1-|x|^2)z]-3c_1^4\right| . \end{aligned}$$
(2.4)

As \(|c_1|\le 2\), letting \(c_1=c\), we may assume without restriction that \(c\in [0, 2]\). Thus, applying the triangle inequality with \(\mu =|x|\), we obtain

$$\begin{aligned} |a_2a_4-a_3^2| \le \dfrac{1}{48}\left[ (4-c^2)\{8c+(c^2-8c+12)\mu ^2+2c^2\mu \}+3c^4\right] :=F_3(c,\mu ). \end{aligned}$$

Next, by differentiating \(F_3\) with respect to \(\mu \), we observe that \(F_3\) is an increasing function of \(\mu \) on [0, 1]. Thus, it attains the maximum value at \(\mu =1\). Again, \(F_3(c,1)=1\), is a constant. Hence

$$\begin{aligned} \max _{\Omega }F_3(c, \mu )=F_3(c, 1)= 1. \end{aligned}$$

To get the sharpness, let us set \(c_1=2\) and \(x=1\) in (1.5) and (1.7), we then obtain \(\,c_2=c_3=2\). Using these values, we find that the results in (2.1) are sharp and the extremal function would be \(\,e_1(z)=z(1+z)^{-2}\). This completes the proof of the theorem. \(\square \)

Theorem 2.5

Let \(f\in \mathcal {S_*}\) be given by (1.1), then

$$\begin{aligned} |\mathbf{H}(f)| \le \frac{116}{7}. \end{aligned}$$

Proof

Using the bounds obtained above in Theorem 2.1–Theorem 2.4 and applying the triangle inequality, we estimate

$$\begin{aligned} |\mathbf{H}(f)| \le |a_3||a_2a_4-a_3^2|+|a_4||a_2a_3-a_4|+|a_5||a_3-a_2^2| \le 3 + 8+\frac{39}{7}= \frac{116}{7}, \end{aligned}$$

and this completes the proof. \(\square \)

3 Set of coefficient bounds belonging to the class \(\mathcal {K_*}\)

Theorem 3.1

Let \(f\in \mathcal {K_*}\) be given by (1.1), then \(|a_n|\le 1\), \(n=2,\,3,\,4\).

Proof

Let \(f\in \mathcal {K_*}\), then by the hypothesis it is clear that \(f(z)\in \mathcal {K_*}\) if and only if \(zf'(z)\in \mathcal {S_*}\). Thus replacing \(\;a_n\;\) by \(\;n a_n\;\) in (2.2), we obtain

$$\begin{aligned} \begin{array}{l} a_2 = -\dfrac{1}{2}c_1, \; a_3=\dfrac{1}{6}(2c_1^2-c_2), \; a_4= \dfrac{1}{24}(7c_1c_2-2c_3-6c_1^3),\\ \text{ and }\\ a_5 = \dfrac{1}{120}(24c_1^4-46c_1^2c_2+20c_1c_3+9c_2^2-6c_4). \end{array} \end{aligned}$$
(3.1)

Bounds for \(|a_2|\) is obvious as \(|c_1|\le 2\). Bounds for \(|a_3|\) and \(|a_4|\) can be directly obtained from results mentioned in a and b of Lemma 1.3. This completes the proof of the theorem. \(\square \)

Theorem 3.2

Let \(f\in \mathcal {K_*}\) be given by (1.1), then \(|a_5|\le 39/35\).

Proof

Let \(f\in {\mathcal {K}}_*\), then using \(a_5\) from (3.1), we can write

$$\begin{aligned} |a_5|= & {} \frac{1}{120}\left| 24c_1^4-46c_1^2c_2+14c_1c_3+9c_2^2-6(c_4-c_1c_3)\right| . \end{aligned}$$

By using the relations (1.5) and (1.7), for some x and z such that \(|x| \le 1\) and \(|z| \le 1\), we estimate

$$\begin{aligned} |a_5|= & {} \frac{1}{480}\left| 27c_1^4+(4-c_1^2)\{-46c_1^2x-23c_1^2x^2+28c_1(1-|x|^2)z+36x^2\}\right. \\&\left. -24(c_4-c_1c_3)\right| . \end{aligned}$$

As \(|c_1|\le 2\), letting \(c_1=c\), we may assume without restriction that \(c\in [0,2]\). Thus applying the triangle inequality and Lemma 1.2 with \(\mu =|x|\), we obtain

$$\begin{aligned} |a_5|\le \frac{1}{480}\left[ 27c^4+(4-c^2)\{46c^2\mu +23c^2\mu ^2+28c(1-\mu ^2)+36\mu ^2\}+48\right] := Z(c,\mu ). \end{aligned}$$

Differentiating \(Z(c,\mu )\) with respect to \(\mu \), we get

$$\begin{aligned} \frac{\partial Z}{\partial \mu }=\frac{1}{480}\left[ (4-c^2)\{23c^2(1+\mu )+4\mu (9-7c)\}\right] >0 \quad \text{ for }\quad (0\le \mu \le 1). \end{aligned}$$

Note that, Z is an increasing function of \(\mu \) on [0, 1]. Thus it attains maximum value at \(\mu =1\). Again, \(Z(c,1) = (-21c^4+120c^2+96)/240\), is an increasing function of c on \([0,\sqrt{20/7}]\). Thus \((\sqrt{20/7}, 1)\) is a critical point of Z.

Again, if we look for the critical points on the boundary of \(\Omega \), as we have done earlier, we get (0, 0), (0, 1) and \((2,\mu ),\, 0\le \mu \le 1\) are the other critical points in \(\Omega \), and for these points we have

$$\begin{aligned} Z(0, 0)<Z(0,1)<Z(2,\mu )<Z(\sqrt{20/7}, 1). \end{aligned}$$

Hence

$$\begin{aligned} \max _{\Omega }Z(c,\mu )=Z(\sqrt{20/7}, 1)=39/35. \end{aligned}$$

This completes the proof of the theorem. \(\square \)

Remark 3.3

For \(f\in \mathcal {K_*}\) of the form (1.1), Arif et al. [3, Corollary 8] obtained that

$$\begin{aligned} |a_2|\le 1 \quad \mathrm{and} \quad |a_n|\le \frac{2}{n(n-1)}\prod _{k=2}^{n-1}\left( \frac{3k-1}{k-1}\right) \quad (n=3,4,5,\ldots ). \end{aligned}$$

We observe here that, our results obtained in Theorem 3.1 and Theorem 3.2 provide the improvement in the upper bound of the initial coefficients \(a_n,\; n=3,4,5 \).

Theorem 3.4

Let \(f\in \mathcal {K_*}\) be given by (1.1), then

$$\begin{aligned} |a_3-a_2^2|\le \frac{1}{3}, \quad |a_2a_3-a_4|\le \frac{4}{3} \quad \mathrm{and }\quad |a_2a_4-a_3^2| \le \frac{1}{8}. \end{aligned}$$
(3.2)

The first inequality of (3.2) is sharp and equality is attended for the function \(\,e_3(z)=z+\frac{1}{3}z^3\).

Proof

If \(f\in \mathcal {K_*}\), then by using the values of \(a_2\), \(a_3\) and \(a_4\) which are given in (3.1), we obtain

$$\begin{aligned} |a_3-a_2^2|= \frac{1}{12}\left| c_1^2-2c_2\right| , \; |a_2a_3-a_4|=\frac{1}{24}\left| 2c_1^3-5c_1c_2+2c_3 \right| , \end{aligned}$$

and

$$\begin{aligned} |a_2a_4-a_3^2|=\frac{1}{144}\left| -5c_1^2c_2+6c_1c_3+2c_1^4-4c_2^2 \right| . \end{aligned}$$

By using Lemma 1.1 we obtain

$$\begin{aligned} |a_3-a_2^2| = \frac{1}{12}\left| c_1^2-2c_2\right| = \frac{1}{6} \left| c_2-\frac{1}{2}c_1^2\right| \le \frac{1}{6} \cdot 2 \max \{1, |2(1/2)-1|\} = \frac{1}{3}. \end{aligned}$$

Now by using Lemma 1.4, we obtain

$$\begin{aligned} |a_2a_3-a_4|=\frac{1}{24}\left| 2c_1^3-5c_1c_2+2c_3 \right|\le & {} \frac{1}{24} \left[ 2|c_1|^3 + 2\left| \frac{5}{2}c_1c_2-c_3\right| \right] \\\le & {} \frac{1}{24} \left[ 2\cdot 8 + 2 \cdot 2|2(5/2)-1| \right] =\frac{4}{3}. \end{aligned}$$

Now, by using the relations (1.5) and (1.7), we obtain

$$\begin{aligned} |a_2a_4-a_3^2|=\frac{1}{288}\left| (4-c_1^2)\{-3c_1^2x-(4-c_1^2)2x^2+6c_1(1-|x|^2)z-3c_1^2x^2 \} \right| . \end{aligned}$$
(3.3)

As \(|c_1|\le 2\), letting \(c_1=c\), we can assume without restriction that \(c\in [0, 2]\). Thus applying the triangle inequality with \(\mu =|x|\), we get

$$\begin{aligned} |a_2a_4-a_3^2|\le \frac{1}{288}\left[ (4-c^2)\{6c+3c^2\mu +(8-6c+c^2)\mu ^2 \}\right] := G_3(c,\mu ). \end{aligned}$$
(3.4)

Furthermore, differentiating \(G_3(c,\mu )\) with respect to \(\mu \), we get

$$\begin{aligned} \frac{\partial G_3}{\partial \mu } = \frac{1}{288}(4-c^2)\{3c^2+(8-6c+c^2)2\mu \} > 0 \quad \text{ for } \quad 0\le \mu \le 1. \end{aligned}$$

Hence \(G_3(c,\mu )\) is an increasing function of \(\mu \) on [0, 1]. Thus, it attains maximum value at \(\mu =1\). Let

$$\begin{aligned} \max _{0\le \mu \le 1}G_3(c,\mu ) = G_3(c,1) = (8+2c^2-c^4)/72 = {\mathcal {G}}_3(c). \end{aligned}$$

Again note that, \({\mathcal {G}}_3(c)\) is an increasing function on [0, 1], so \({\mathcal {G}}_3(c)\) attend maximum value at \(c=1\). Hence \(G_3(c,\mu )\) have maximum value at the point (1, 1), that is

$$\begin{aligned} \max _\Omega G_3(c,\mu )=G_3(1,1)=1/8. \end{aligned}$$

This completes the proof of the theorem. \(\square \)

Theorem 3.5

Let \(f\in \mathcal {K_*}\) be given by (1.1), then

$$\begin{aligned} |\mathbf{H}(f)| \le \frac{1537}{840}. \end{aligned}$$

Proof

Using Theorem 3.1–Theorem 3.4 and applying the triangle inequality, we obtain that

$$\begin{aligned} |\mathbf{H}(f)| \le |a_3||a_2a_4-a_3^2|+|a_4||a_2a_3-a_4|+|a_5||a_3-a_2^2| \le \frac{1}{8}+\frac{4}{3}+\frac{39}{105} = \frac{1537}{840}, \end{aligned}$$

and this completes the proof. \(\square \)