1. Introduction

All groups is finite. We will adhere to the notation of [3, 4]. In particular, \( |G| \) denotes the order of a group \( G \) (or a set \( G \)), while \( \pi(G) \) denotes the set of all prime divisors of \( |G| \). Let \( H_{G} \) be the core of \( H \) in \( G \) when \( H\leq G \) and let \( M<\cdot G \) signify that \( M \) is a maximal subgroup of \( G \). Put

$$ \operatorname{Max}(G,H)=\{M<\!\cdot\,G\mid H\leq M\},\quad\operatorname{Max}_{1}(G,H)=\{M<\!\cdot\,G\mid H<\!\cdot\,M\}. $$

Let \( \operatorname{Irr}(G) \) be the set of all irreducible complex characters of \( G \). An element \( x \) of \( G \) is called nonvanishing if \( \chi(x)\neq 0 \) for all \( \chi\in\operatorname{Irr}(G) \).

Next, we recall some known research that is tightly related to our study.

Firstly, we need to list the following problem by Monakhov [1, Problem 19.54] which motivated our research.

Problem

What are the chief factors of a finite group whose no \( 2 \)-maximal subgroup is \( m \)-maximal for any \( m\geq 3 \)?


Addressing the problem, Meng and Guo [2] considered the properties of the second maximal subgroup of a group and the structure of a \( WSM \)-group under the universe of solvable groups, where the \( WSM \)-group is equal to the group satisfying the condition of the above problem.

On the other hand, Isaacs, Navarro, and Wolf conjectured in [5] that every nonvanishing element of a solvable group \( G \) is contained in the Fitting subgroup \( F(G) \). In [6], Guo, Skiba, and Tang introduced the concept of boundary factors and traces of subgroups in finite groups and investigated the solvability of a group by considering the traces of maximal subgroups.

Continuing to study the Problem and developing the research of Meng and Guo [2], we will investigate the numerical structure of a second maximal subgroup of a group by weakening the condition of solvability. Meanwhile, viewing from the conjecture in [5] and the result in [6], we also consider the relationship between the conjecture and the traces of second maximal subgroups of a group. Here we obtained the following results:

Theorem 1.1

Let \( G \) be a \( WSM \)-group and let \( x \) be a nonvanishing element of \( G \). If each second maximal subgroup of \( G \) has a nilpotent trace, then \( G \) is solvable and \( x\in F(G) \).

Theorem 1.2

Let \( H<\cdot M<\cdot G \) and \( |M:H|=p^{\alpha} \), where \( p\in\pi(G) \). If \( H \) is a \( CAP \)-subgroup of \( G \), then \( |\operatorname{Max}(G,H)\backslash\operatorname{Max}_{1}(G,H)|\leq 1 \).

2. Preliminaries

For the sake of convenience, we start with listing some known results that will be useful in this paper.

Lemma 2.1 [2, Lemma 1]

Let \( G \) be a group and let \( H \) be a subgroup of \( G \). If there exists \( M \), \( X\in\operatorname{Max}(G,H) \) such that \( H \) is maximal in \( M \) and \( H \) is not maximal in \( X \), then \( H_{G}=M_{G} \).

Lemma 2.2 [2, Theorem B]

Let \( G \) be a solvable group and let \( H \) be a weak second maximal subgroup of \( G \). Then there exists at most one \( X\in\operatorname{Max}(G,H) \) such that \( H \) is not maximal in \( X \).

Lemma 2.3

Let \( G \) be a group. If every second maximal subgroup of \( G \) is nilpotent, then \( G \) is either solvable or isomorphic to \( PSL(2,5) \) or \( SL(2,5) \).

Proof

See [7] and [8].

3. The Main Results

In [6], Guo, Skiba, and Tang introduced the concept of boundary factors and traces of subgroups in finite groups. Next, we will study the construction of a nonvanishing element of a group by the nilpotency of the traces of second maximal subgroups of \( G \).

Theorem 3.1

Let \( G \) be a \( WSM \)-group and let \( x \) be a nonvanishing element of \( G \). If each second maximal subgroup of \( G \) has a nilpotent trace, then \( G \) is solvable and \( x\in F(G) \).

Proof

By [2, Theorem A], we only need to prove that \( G \) is solvable.

If there exists a second maximal subgroup \( H \) of \( G \) such that \( H=1 \), then there is a maximal subgroup \( M \) of \( G \) such that \( |M| \) is a prime. By [9, Chapter IV, Theorem 7.4], \( G \) is solvable. Hence every second maximal subgroup of \( G \) is nontrivial. Now we assert that \( G \) is not nonabelian simple. Otherwise, \( G \) is nonabelian simple. Since each second maximal subgroup of \( G \) has a nilpotent trace, every second maximal subgroup of \( G \) is nilpotent. By Lemma 2.3\( G\cong A_{5} \), where \( A_{5} \) is the alternating group of degree \( 5 \). However, \( A_{5} \) is not a \( WSM \)-group by [10].

Further, we may choose a minimal normal subgroup \( L \) of \( G \) and consider the quotient group \( G/L \). If \( L \) is maximal in \( G \), then \( G/L \) is of order \( q \), with \( q \) a prime. If \( L \) is not maximal in \( G \), then \( G/L \) satisfies the hypothesis and \( G/L \) is solvable by induction on \( |G| \). Hence \( G/L \) is solvable for every minimal normal subgroup \( L \) of \( G \). Further, \( L\nleq\Phi(G) \) and \( L \) is the unique minimal normal subgroup of \( G \).

Let \( L_{p} \) be a Sylow \( p \)-subgroup of \( L \) where \( p \) is a largest prime divisor of \( |L| \). Clearly, \( p>3 \). By the Frattini argument, \( G=LN_{G}(L_{p})=LM \), where \( M \) is maximal in \( G \) and \( N_{G}(L_{p})\leq M \). Further, \( L\cap M\neq 1 \) and \( M_{G}=1 \). Hence there is a maximal subgroup \( H \) of \( M \) such that \( L\cap M\leq H \) and \( L\cap H=L\cap M \) is nilpotent by hypothesis. Then \( N_{L}(L_{p})=L\cap N_{G}(L_{p})\leq L\cap M=L\cap H \) is nilpotent and \( N_{L}(L_{p})/C_{L}(L_{p}) \) is a \( p \)-subgroup. Further, \( O^{p}(G)<L \) by [11, Chapter X, Theorem 8.13] and \( L \) is a \( p \)-subgroup of \( G \). Hence, \( G \) is solvable since \( G/L \) is solvable.

In view of [12, Theorem 3.7], we can weaken the condition of solvability in [2, Theorem B] by the following condition and arithmetic description of a second maximal subgroup of a group.

Theorem 3.2

Let \( H<\cdot M<\cdot G \) and \( |M:H|=p^{\alpha} \), where \( p\in\pi(G) \). If \( H \) is a \( CAP \)-subgroup of \( G \), then \( |\operatorname{Max}(G,H)\backslash\operatorname{Max}_{1}(G,H)|\leq 1 \).

Proof

Clearly, we may assume that \( H_{G}=1 \) and \( X_{i}\in\operatorname{Max}(G,H)\backslash\operatorname{Max}_{1}(G,H) \), where \( i \)=1,2. Since \( H<\cdot M \), \( H=M\cap X_{1}=M\cap X_{2} \). Also, \( H_{G}=M_{G}=1 \) by Lemma 2.1.

Since \( G \) is primitive, \( G \) has one of the following structures by [3, Chapter A, Theorem 15.2]:

\( (1) \) \( G=LM \), \( C_{G}(L)=L \) and \( L \) is abelian, where \( L \) is the unique minimal normal subgroup of \( G \);

\( (2) \) \( G=LM \) and \( L \) is nonabelian, where \( L \) is the unique minimal normal subgroup of \( G \);

\( (3) \) \( G \) has exactly two minimal normal subgroups \( L \) and \( L^{*} \) of \( G \), while \( G=LM=L^{*}M \) and \( L\cap M=L^{*}\cap M=1 \). Also, \( C_{G}(L)=L^{*} \), \( C_{G}(L^{*})=L \) and \( L\cong L^{*}\cong LL^{*}\cap M \). Moreover, if \( V<G \) and \( LV=L^{*}V=G \), then \( L\cap V=L^{*}\cap V=1 \).

By Lemma 2.2, we may assume that \( G \) is not solvable. Since \( H \) is a \( CAP \)-subgroup of \( G \), we assert that \( G \) is not simple. Otherwise, \( H=1 \). Then \( M \) is a maximal subgroup of \( G \) of prime order and \( G \) is solvable by [9, Chapter IV, Theorem 7.4]; a contradiction. To prove, we will coincide the following cases:

Case I: \( G \) has the structure \( (1) \) above.

If \( L\nleq X_{i} \) for some \( i\in\{1,2\} \), then \( G=LX_{i} \) and \( L\cap X_{i}=1 \). Further, \( M\cong ML/L \) and \( X_{i}\cong LX_{i}/L \). Since \( HL/L<\cdot ML/L=LX_{i}/L \), \( H<\cdot X_{i} \); a contradiction. So \( L\leq X_{1} \) and \( L\leq X_{2} \). Then \( LM\cap X_{1}=LM\cap X_{2}=LH \) and \( |\operatorname{Max}(G,H)\backslash\operatorname{Max}_{1}(G,H)|\leq 1 \).

Case II: \( G \) has the structure \( (3) \) above.

Since \( G=LM=L^{*}M \), \( L\cap M=L^{*}\cap M=1 \), If there exists some \( X_{i} \) such that \( (X_{i})_{G}=1 \) for some \( i\in\{1,2\} \), then \( G=LX_{i}=L^{*}X_{i} \) and \( L\cap X_{i}=L^{*}\cap X_{i}=1 \). Further, \( HL/L<\cdot ML/L=LX_{i}/L \), \( H<\cdot X_{i} \); a contradiction. So, \( (X_{1})_{G}\neq 1 \) and \( (X_{2})_{G}\neq 1 \). Then we assert that \( L\leq(X_{1})_{G}\cap(X_{2})_{G} \) and \( L^{*}\leq(X_{1})_{G}\cap(X_{2})_{G} \). Otherwise, there exists \( R\in\{L,L^{*}\} \) and \( X_{i} \) for some \( i\in\{1,2\} \) such that \( RX_{i}=G \) and \( R\cap X_{i}=1 \). With the similar discussion of the above, \( H<\cdot X_{i} \); a contradiction. Hence \( LM\cap X_{1}=LM\cap X_{2}=LH \) and \( |\operatorname{Max}(G,H)\backslash\operatorname{Max}_{1}(G,H)|\leq 1 \).

Case III: \( G \) has the structure \( (2) \) above.

Since \( H \) is a \( CAP \)-subgroup of \( G \), \( H\cap L=1 \). Further, we consider the subgroup \( HL \).

If \( HL=G \), then \( M=M\cap G=M\cap HL=H(M\cap L) \) and \( M\cap L \) is a \( p \)-subgroup since \( |M:H|=p^{\alpha} \). Also, \( M\cap L \) is a minimal normal subgroup of \( M \) since \( H\cap L=1 \) and \( H<\cdot M \). Now we assert that \( N_{G}(L\cap M)=M \). Otherwise, \( N_{G}(L\cap M)=G \) and \( L=L\cap M \) by the minimal normality of \( L \). Further, \( HL=G=M \); a contradiction. Hence \( N_{L}(L\cap M)=L\cap N_{G}(L\cap M)=L\cap M \). Clearly, \( L\cap M \) is a Sylow \( p \)-subgroup of \( L \). Thus, \( N_{L}(L\cap M)=L\cap M=C_{L}(L\cap M) \). By the Burnside Theorem, \( L \) is \( p \)-nilpotent. Since \( L\cap M\neq 1 \), \( L \) is a \( p \)-subgroup of \( G \) and \( L=L\cap M\leq M \). Then \( HL=G=M \); a contradiction.

If \( HL<G \), then \( M=M\cap HL=H(M\cap L) \) since \( L\cap M\neq 1 \) and \( H<\cdot M \). Hence \( L\leq M \) and \( LM=G=M \); a contradiction.

The authors proved in [2, Lemma 1] that \( H_{G}=M_{G} \), where \( M\in\operatorname{Max}_{1}(G,H) \) and \( \operatorname{Max}_{1}(G,H) \) is properly included in \( \operatorname{Max}(G,H) \). By dual consideration, we will show the following relationship between \( H \) and \( X \), where \( X\in\operatorname{Max}(G,H)\backslash\operatorname{Max}_{1}(G,H) \):

Theorem 3.3

Let \( G \) be a group and let \( H \) be a subgroup of \( G \). If there exist \( M\in\operatorname{Max}_{1}(G,H) \) and \( X\in\operatorname{Max}(G,H)\backslash\operatorname{Max}_{1}(G,H) \), then either \( H_{G}=X_{G} \) or \( HX_{G}=X \).

Proof

Assume that \( H_{G}\neq X_{G} \). Prove that \( HX_{G}=X \).

If \( H_{G}=1 \), then \( H_{G}=M_{G}=1 \) by Lemma 2.1. Further, \( X_{G}\neq 1 \), \( G \) is primitive and \( G=LM \), where \( L \) is a minimal normal subgroup \( L \) of \( G \) which lies in \( X_{G} \). Since \( H \) is maximal in \( M \), \( H=M\cap X \) and \( LH=X \). Hence \( HX_{G}=X \).

If \( H_{G}\neq 1 \), then \( 1<H_{G}<H \) or \( H_{G}=H \). To proceed the proof, we will consider the following cases:

Case I: \( 1<H_{G}<H \).

We consider the quotient subgroup \( G/H_{G} \). So \( (H/H_{G})_{G/H_{G}}=(X/H_{G})_{G/H_{G}} \) or \( (H/H_{G})(X/H_{G})_{{G/H_{G}}}={X/H_{G}} \) by the induction on \( |\,G| \). Further, \( H_{G}=X_{G} \) or \( HX_{G}=X \). Hence \( HX_{G}=X \) by assumption.

Case II: \( H_{G}=H \).

Since \( H \) is maximal in \( M \); therefore, \( |M:H|=p \) and \( H=M\cap X=M\cap X_{G} \). Then \( G=MX_{G} \) and \( |G:X_{G}|=|M:M\cap X_{G}|=|M:H|=p \). Hence \( X_{G}=X \) and \( HX_{G}=X \).

Corollary 3.4

Under the hypothesis of Theorem 3.3, if \( H \) is subnormal in \( G \), then either \( H_{G}=X_{G} \) or \( X\trianglelefteq G \).

Proof

If \( G \) is simple then \( H_{G}=X_{G}=1 \). Assume now that \( G \) is not simple. Since \( H \) is subnormal in \( G \); therefore, \( \operatorname{Soc}(G)\leq N_{G}(H) \) by [3, Chapter A, Theorem 14.3] and \( N_{G}(H)=M \) or \( N_{G}(H)=G \). If \( N_{G}(H)=G \), then \( H \) is normal in \( G \) and \( H=X_{G} \) or \( X\trianglelefteq G \) by Theorem 3.3. If \( N_{G}(H)=M \), then \( \operatorname{Soc}(G)\leq N_{G}(H)=M \) and \( M_{G}=H_{G}\neq 1 \) by Lemma 2.1. Hence \( 1<H_{G}<H \). Next, we consider the quotient group \( G/H_{G} \). By induction on \( |G| \), we see that \( (H/H_{G})_{G/H_{G}}=(X/H_{G})_{G/H_{G}} \) or \( X/H_{G}\trianglelefteq G/H_{G} \). Then \( H_{G}=X_{G} \) or \( X\trianglelefteq G \).