An extended 3-3-1 model with two scalar triplets and linear seesaw mechanism

Abstract

Low energy linear seesaw mechanism responsible for the generation of the tiny active neutrino masses is implemented in the extended 3-3-1 model with two scalar triplets and right-handed Majorana neutrinos where the gauge symmetry is supplemented by the $A_4$ flavor discrete group and other auxiliary cyclic symmetries, whose spontaneous breaking produces the observed pattern of SM charged fermion masses and fermionic mixing parameters. Our model is consistent with the low energy SM fermion flavor data as well as with the constraints arising from meson oscillations. Some phenomenological aspects, such as the $Z^\prime$ production at proton–proton collider and the lepton flavor violating decay of the SM-like Higgs boson are discussed. The scalar potential of the model is analyzed in detail and the SM-like Higgs boson is identified.

1 Introduction

It is well-known, that there are various experimental and theoretical observations indicating that the Standard Model (SM) must be extended. Among the theories beyond the SM, the models based on the gauge group $SU(3)_C\times SU(3) _L\times U(1) _X$ (called 3-3-1 for short) [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49] have some intriguing features allowing them to explain the number of SM fermion families, the electric charge quantization [50, 51], etc. In the ordinary 3-3-1 models, the Higgs sector contains at least three scalar triplets significantly extending their scalar spectrum. Attempts aimed to reduce the Higgs sector of the 3-3-1 models have been undertaken in the literature. A model with the parameter $\beta = - \frac{1}{\sqrt{3}}$, defined in (3) and characterizing the embedding of the electric charge generator into $SU(3)_{L}$, has been proposed in Refs. [47, 52,53,54,55,56,57]. Due to its restricted scalar sector it is called the economical 3-3-1 model. However, this and other similar versions of the 3-3-1 model with the reduced scalar content failed to reproduce the neutrino oscillation data. In a view of these difficulties a 3-3-1 model with $\beta = \frac{1}{\sqrt{3}}$ and containing just two Higgs triplets has been studied in Ref. [41]. In this model the masses of light active neutrinos and charged fermions are generated via Type-I Seesaw and the Universal Seesaw mechanisms, respectively. However, the fermion mixing was not addressed in Ref. [41].

In the present paper, we propose a multiscalar singlet extension of the 3-3-1 model with two $SU(3)_L$ scalar triplets and three right-handed Majorana neutrinos. The gauge group of the model is extended with the $A_4$ group and some other cyclic symmetries in order to implement the linear seesaw mechanism responsible for the tiny masses of the active neutrinos. A well-known advantage of the linear seesaw mechanism [58,59,60,61,62, 62,63,64,65,66,67] is its testability at the LHC, since it implies sterile neutrinos with TeV-scale masses. Our model also successfully addresses the observed pattern of the SM fermion masses and mixings, as a result of the spontaneous breaking of the above-mentioned discrete group factors, in an analogous way to the Froggatt–Nielsen mechanism [68], which has also been implemented in 3-3-1 models through the breaking of a U(1) global symmetry in Refs. [69,70,71]. We choose $A_4$ as the smallest discrete group having one three-dimensional and three distinct one-dimensional irreducible representations allowing us to naturally accommodate the three families of the SM. The $A_4$ discrete flavor group has received a lot interest by the model building community due to its remarkable ability to elucidate the observed pattern of SM fermion masses and mixing angles [31, 39, 62, 72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112].

Comparing our model with others, we note, in particular, that our $U(1)_X$-charge assignments of the left-handed quark $SU(3)_L$-triplets are different from those in the model of Ref. [41]. Due to this difference we have two exotic down type quarks and one exotic up type quarks, whereas in the model of Ref. [41] there are two exotic up type quarks and one exotic down type quark. In addition, whereas in our model the small masses for the active neutrinos are produced from a linear seesaw mechanism, in the model of Ref. [41] they are generated from a type-I seesaw mechanism. In Ref. [41], the extra fermion lying in the bottom of the lepton triplet is a charged lepton instead of the right-handed neutrino, which is the field of the third component of $SU(3)_L$ leptonic triplet in our model.

Let us also note that our model is more predictive and significantly more economical in its particle content than the 3-3-1 model with $T^{\prime }$ and $S_4$ symmetries proposed in [47, 48]. For instance, whereas the scalar sector of the $T^{\prime }$ flavored 3-3-1 model [47] includes two $SU(3)_L$ scalar triplets and 23 gauge singlet scalar fields, the present model has two $SU(3)_L$ scalar triplets and 16 $SU(3)_L$ singlet scalar fields. As for the scalar sector of the 3-3-1 model with $S_4$ family symmetry [48], it contains 3 $SU(3)_L$ scalar triplets and 32 gauge singlet scalar fields, which is much larger than the number of scalar degrees of freedom of our model. Let us note, that in the proposed model some quarks and scalar fields carry lepton number, which leads to flavor lepton number violating decay modes of the SM-like Higgs boson. In what follows we will study this phenomenological aspect of our model as well as the production of the extra heavy neutral gauge boson $Z'$ and its detection in the dimuon channel at the LHC. However, the emphasis will be made on studying the SM fermion masses and mixings.

The paper is organized as follows. In Sect. 2, we introduce the model setup. Sections 3 and 5 are devoted to the model predictions for the masses and mixings in the quark and lepton sectors, respectively. Section 4 discusses the constraints on the $Z'$ mass arising from meson oscillations. In Sect. 7, the lepton flavor violating (LFV) decays of the charged leptons and the Higgs boson are considered. In Sect. 8, we summarize our results and discuss their further implications. In “Appendix A” we present the discrete group $A_4$ group characters. A detailed description of the Higgs sector of the model is given in “Appendix B.” The analytic formulas for one-loop contributions to the LFV decay amplitudes of the SM-like Higgs boson are collected in “Appendix C.” The couplings of neutral gauge bosons Z and $Z'$ to fermions are listed in “Appendix D.”

2 The model

We propose a 3-3-1 model where the scalar sector is composed of two $SU( 3) _L$ scalar triplets and seven $SU( 3) _L$ scalar singlets and the fermion sector corresponds to one of the 3-3-1 models with three right-handed Majorana neutrinos. In our model the $SU(3)_C\times SU( 3) _L\times U( 1) _X$ gauge symmetry is supplemented with the $A_4\times Z_8\times Z_{14}\times Z_{22}$ discrete group, so that the full symmetry ${\mathcal {G}}$ exhibits the following three-step spontaneous breaking:

$$\begin{aligned}&{\mathcal {G}}=SU(3)_C\times SU(3)_L\times U(1)_X\times A_4\times Z_8\times Z_{14}\times Z_{22} \nonumber \\&\Downarrow \Lambda _{int} \nonumber \\[0.12in]&SU(3)_C\times SU(3)_L\times U(1)_X \nonumber \\[0.12in]&\Downarrow v_\chi \nonumber \\[0.12in]&SU(3)_C\otimes SU( 2) _L\times U( 1) _Y \nonumber \\[0.12in]&\Downarrow v_\eta \nonumber \\[0.12in]&SU(3)_C\otimes U( 1) _Q \end{aligned}$$

(1)

where the different symmetry breaking scales satisfy the following hierarchy

$$\begin{aligned} v_\eta =v=246\text{ GeV }\ll v_\chi \sim {\mathcal {O}}(10)\text{ TeV } . \end{aligned}$$

(2)

In the 3-3-1 model under consideration, the electric charge is defined in terms of the SU(3) generators and the identity by:

$$\begin{aligned} Q=T_3+\beta T_8+X =T_3-\frac{1}{\sqrt{3}}T_8+X, \end{aligned}$$

(3)

where we have chosen $\beta =-\frac{1}{\sqrt{3}}$ (without non-SM electric charges), which implies that bottom component of the lepton $SU(3)_L$-triplet is a neutral field $\nu _R^C$ thus allowing to build the Dirac matrix with the usual field $\nu _L$ in the top component of the lepton triplet. Adding gauge singlet right-handed Majorana neutrinos $N_{iR}$ $(i=1,2,3)$ will allow us to implement a low scale seesaw mechanism, which could be inverse or linear, to generate the masses for the light active neutrinos. These low scale seesaw mechanisms offer attractive explanations for the smallest of neutrino masses, because they can be tested at the LHC via the production and decay of sterile neutrinos. It is worth mentioning that the sterile neutrinos can be produced at the LHC in association with a SM charged lepton and in pairs, via quark–antiquark annihilation mediated by a W and heavy $W^\prime$ and $Z^\prime$ gauge bosons, respectively. In our model the sterile neutrinos have the following two body decay modes: $N^{\pm }_{a}\rightarrow l^{\pm }_iW^{\mp }$ and $N^{\pm }_{a}\rightarrow \nu _iZ$ (where $a,i=1,2,3$), which are suppressed by the small active-sterile neutrino mixing angle. Furthermore, the heavy sterile neutrinos $N^{\pm }_{a}$ can decay via off-shell gauge bosons via the following modes: $N^{\pm }_{a}\rightarrow l^{+}_il^{-}_j\nu _k$, $N^{\pm }_{a}\rightarrow l^{-}_iu_j{\bar{d}}_k$, $N^{\pm }_{a}\rightarrow b{\bar{b}} \nu _k$ (where $a,i,j,k=1,2,3$ are flavor indices). Thus, the heavy sterile neutrinos can be detected at the LHC from the observation of an excess of events with respect to the SM background in a final state composed of a pair of opposite sign charged leptons plus two jets. Studies of inverse seesaw neutrino signatures at colliders as well as the production of heavy neutrinos at the LHC are carried out in [113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129]. A detailed study of the sterile neutrino production at the LHC and the sterile neutrino modes goes beyond the scope of this work and will be done elsewhere.

The cancellation of chiral anomalies implies that the number of triplets equals that of antitriplets, so that quarks are unified in the following $SU(3)_C\times SU(3)_L\times U(1)_X$ left- and right-handed representations [2, 7, 9, 130]:

$$\begin{aligned} Q_{nL}&=\left( D_n \, , -U_n \, , J_n \, \right) ^T_L \sim ( 3,3^*,0) ,\\ Q_{3L}&=\left( U_3 \, , D_3 \, , \, T \right) ^T_L \sim \left( 3,3,\frac{1}{3} \right) ,\quad n=1,2, \\ D_{iR}&\sim \left( 3,1,-\frac{1}{3} \right) ,\quad U_{iR}\sim \left( 3,1,\frac{2}{3} \right) , \nonumber \\ J_{nR}&\sim \left( 3,1,-\frac{1}{3} \right) ,\quad T_R \sim \left( 3,1,\frac{2}{3}\right) ,\quad i=1,2,3 . \end{aligned}$$

Furthermore, the requirement of chiral anomaly cancellation constrains the leptons to the following $SU(3)_C\times SU(3)_L\times U(1)_X$ left- and right-handed representations [2, 7, 130] :

$$\begin{aligned} L_{iL}=\left( \nu _i \, , e_i \, , \nu _i^c \right) ^T _L\sim \left( 1,3,-\frac{1}{3} \right) ,\quad e_{iR}\sim ( 1,1,-1) , \quad i=1,2,3, \end{aligned}$$

(4)

In the present model, the fermion sector is extended by introducing three right-handed Majorana neutrinos, singlets under the 3-3-1 group, so that they have the following $SU(3)_C\times SU(3)_L\times U(1)_X$ assignments:

$$\begin{aligned} N_{iR}\sim ( 1,1,0) ,\quad i=1,2,3. \end{aligned}$$

Note that in Ref. [41], where $\beta =+\frac{1}{\sqrt{3}}$, the third component of lepton triplet is an extra charged leptons.

We assign the scalar fields to the following $SU(3)_C\times SU(3)_L\times U(1)_X$ representations:

$$\begin{aligned} \chi= & {} \begin{pmatrix} \chi _1 ^0 \\ \chi _2^- \\ \frac{1}{\sqrt{2}} (v_\chi +\xi _\chi \pm i\zeta _\chi ) \end{pmatrix} \sim \left( 1,3,-\frac{1}{3} \right) ,\nonumber \\ \eta= & {} \begin{pmatrix} \frac{1}{\sqrt{2}} (v_\eta +\xi _\eta \pm i\zeta _\eta ) \\ \eta _2^- \\ \eta _3 ^0 \end{pmatrix} \sim \left( 1,3,-\frac{1}{3} \right) , \nonumber \\ \sigma\sim & {} ( 1,1,0) ,\nonumber \\ \xi _i\sim & {} ( 1,1,0) ,\quad \quad \zeta _i\sim ( 1,1,0) ,\quad \quad i=1,2,3. \nonumber \\ \rho _i\sim & {} ( 1,1,0) ,\quad \quad \varphi _i\sim ( 1,1,0) ,\quad \quad \phi _i\sim ( 1,1,0) . \end{aligned}$$

(5)

Here $v_{\chi }, v_{\eta }$ are the vev’s setting symmetry breaking scales in (1), (2).

The scalar assignments under the $A_4\times Z_8\times Z_{14}\times Z_{22}$ discrete group are summarized in Table 1.

In our model, this discrete global symmetry group is not only spontaneously broken, it is softly broken as well. Let us note that the gauge singlet scalars of our models are complex, which implies that in order to provide masses for the CP odd parts of these scalars, one has to include $A_{4}\times Z_{8}\times Z_{14}\times Z_{22}$ soft breaking bilinear terms in the scalar potential involving a pair of these scalar singlets. These soft breaking scalar mass terms will also be useful for resolving the domain wall problem, arising from the spontaneous breaking the global discrete symmetries.

Table 1 Scalar assignments under $A_4\times Z_8\times Z_{14}\times Z_{22}$

In “Appendix B” we present more details about the scalar sector of our model.

In what follows we briefly describe the gauge sector of our model. Here, we have 8 electroweak $SU(3)_L$ gauge bosons, $W_{a\mu }$, and a $U(1)_X$ gauge boson, ${\widetilde{B}}_{\mu }$.

From the scalar kinetic term one finds the interactions:

$$\begin{aligned} ( D^\mu H)^\dag D_\mu H \supset \partial ^\mu R_H^\dagger P_\mu I_H - R^\dagger P^\mu \partial _\mu I_H \, {,} \, H= \eta , \chi , \end{aligned}$$

(6)

The covariant derivative is defined as

$$\begin{aligned} D_{\mu }=\partial _{\mu }-iT_{a}W_{a\mu }-ig_{X}T_{9}X{\widetilde{B}}_{\mu }=\partial _{\mu }-i\Pi _{\mu } \end{aligned}$$

(7)

with:

$$\begin{aligned} \Pi _{\mu }\!=\!\frac{g}{2}\left( \! \begin{array}{ccc} W_{3\mu }\!+\!\frac{1}{\sqrt{3}}W_{8\mu }\!+\!t\sqrt{\frac{2}{3}}X{\widetilde{B}}_{\mu } &{} \sqrt{2}W_{\mu }^{+} &{} \sqrt{2}X_{\mu }^{q_{1}} \\ \sqrt{2}W_{\mu }^{-} &{} -W_{3\mu }\!+\!\frac{1}{\sqrt{3}}W_{8\mu }\!+\!t\sqrt{\frac{2 }{3}}X{\widetilde{B}}_{\mu } &{} \sqrt{2}Y_{\mu }^{q_{2}} \\ \sqrt{2}X_{\mu }^{-q_{1}} &{} \sqrt{2}Y_{\mu }^{-q_{2}} &{} -\frac{2}{\sqrt{3}} W_{8\mu }\!+\!t\sqrt{\frac{2}{3}}X{\widetilde{B}}_{\mu } \end{array} \!\right) ,\quad t\!=\!\frac{g_{X}}{g} \end{aligned}$$

where

$$\begin{aligned} W_{\mu }^{\pm }=\frac{1}{\sqrt{2}}\left( W_{1}\mp iW_{2}\right) ,\quad X_{\mu }^{q_{1}}=\frac{1}{\sqrt{2}}\left( W_{4}-iW_{5}\right) ,\quad Y_{\mu }^{q_{2}}=\frac{1}{\sqrt{2}}\left( W_{6}-iW_{7}\right) .\nonumber \\ \end{aligned}$$

(8)

Then, in the gauge sector we have three electrically neutral $q=0$ gauge fields, which combine to form the photon and Z, $Z^{\prime }$-bosons, two fields $W^{\pm }$ with $q=\pm 1$ and $X_{\mu }^{q_{1}}$, $Y_{\mu }^{q_{2}}$ with electrical charges

$$\begin{aligned} q_{1}= & {} \frac{1}{2}+\frac{\sqrt{3}\beta }{2},\quad q_{2}=-\frac{1}{2}+\frac{\sqrt{3}\beta }{2}. \end{aligned}$$

(9)

Physical neutral gauge bosons for $\beta =-\frac{1}{\sqrt{3}}$ are given by:

$$\begin{aligned} A_{\mu }= & {} c_{W}\left( -\sqrt{\frac{1}{3}}t_{W}W_{8\mu }+\frac{\sqrt{ 3-4s_{W}^{2}}}{\sqrt{3}c_{W}}{\widetilde{B}}_{\mu }\right) +s_{W}W_{3\mu },\nonumber \\ Z_{\mu }= & {} c_{W}W_{3\mu }-s_{W}\left( -\sqrt{\frac{1}{3}}t_{W}W_{8\mu }+\frac{ \sqrt{3-4s_{W}^{2}}}{\sqrt{3}c_{W}}{\widetilde{B}}_{\mu }\right) ,\nonumber \\ Z_{\mu }^{\prime }= & {} -\sqrt{\frac{1}{3}}t_{W}{\widetilde{B}}_{\mu }-\frac{\sqrt{ 3-4s_{W}^{2}}}{\sqrt{3}c_{W}}W_{8\mu }\nonumber \\ X_{\mu }^0= & {} \frac{1}{\sqrt{2}}\left( W_{\mu 4}- iW_{\mu 5} \right) ,\quad {\bar{X}}_{\mu }^0= \frac{1}{\sqrt{2}}\left( W_{\mu 4}+iW_{\mu 5} \right) \end{aligned}$$

(10)

where $c_W=\cos \theta _W$, $s_W=\cos \theta _W$ and $t_W=\tan \theta _W$, being $\theta _W$ the weak mixing angle. In addition, for $\beta =-\frac{1}{\sqrt{3}}$, which corresponds to our model, we find the relations:

$$\begin{aligned} {\widetilde{B}}_{\mu }= & {} -\sqrt{\frac{1}{3}}t_{W}Z_{\mu }^{\prime }+\frac{\sqrt{ 3-4s_{W}^{2}}}{\sqrt{3}c_{W}}\left( c_{W}A_{\mu }-s_{W}Z_{\mu }\right) , \nonumber \\ W_{8\mu }= & {} -\sqrt{\frac{1}{3}}t_{W}\left( c_{W}A_{\mu }-s_{W}Z_{\mu }\right) -\frac{\sqrt{3-4s_{W}^{2}}}{\sqrt{3}c_{W}}Z_{\mu }^{\prime }, \end{aligned}$$

(11)

$$\begin{aligned} t= & {} \frac{g_{X}}{g}=\frac{3\sqrt{2}s_{W}}{\sqrt{3-4s_{W}^{2}}}. \end{aligned}$$

(12)

The electrically charged gauge bosons are given by:

$$\begin{aligned} W_\mu ^\pm = \frac{1}{\sqrt{2}} \left( A_{\mu 1} \mp i A_{\mu 2} \right) \, , \quad Y_\mu ^\pm = \frac{1}{\sqrt{2}} \left( A_{\mu 6} \pm i A_{\mu 7} \right) \, \end{aligned}$$

(13)

where $Y^\pm$ and $X^0$ are bilepton gauge bosons. With the above-discussed structure of the scalar sector of the model, the massive gauge bosons acquire the following masses [131]:

$$\begin{aligned} m^2_W= & {} m^2_Zc^2_W =\frac{g^2}{4}v_\eta ^2 \, , \quad M^2_{X^0}=M^2_{{\bar{X}}^0} = \frac{g^2}{4} \left( v^2_\chi +v_\eta ^2\right) \, , \nonumber \\ M^2_Y= & {} \frac{g^2}{4} v_\chi ^2\, \, , \quad M^2_{Z^\prime }\simeq \frac{g^2v_{\chi }^2}{3-t^2_W}, \end{aligned}$$

(14)

where $v_\eta = v =246$ GeV. From (14), we find the mass splitting

$$\begin{aligned} M^2_{X^0} - M^2_Y = m^2_W . \end{aligned}$$

(15)

In Ref. [132] it was shown that the contributions of the bilepton gauge boson $Y^{\pm }, X^{0}$ to the oblique S and T parameters are constrained to be in the ranges $-0.085\lesssim S\lesssim 0.05$, $-0.001\lesssim T\lesssim 0.08$, respectively. In the scenario where the mixing angles between the exotic and the SM quarks are small, which is the case of our model, the exotic quark contributions to these oblique parameters are very subleading since they are suppressed by the square of the small mixing angles. Consequently, the dominant contributions to the oblique S and T parameters are the ones arising from the bilepton gauge bosons $Y^\pm$ and $X^0$. Notice that the aforementioned range of values for the S and T parameters allow one to have a region of the model parameter space where the obtained values for these oblique parameters are inside the experimentally allowed region of Ref. [133] enclosed by the ellipses in the $S-T$ plane.

The fermion assignments under the $A_4\times Z_8\times Z_{14}\times Z_{22}$ discrete group are summarized in Table 2.

Table 2 Fermion assignments under $A_4\times Z_8\times Z_{14}\times Z_{22}$

We assume the following VEV pattern for the $A_4$ triplet SM singlet scalars $\xi$, $\zeta$, $\rho$, $\varphi$ and $\phi$:

$$\begin{aligned} \left\langle \xi \right\rangle= & {} \frac{v_\xi }{\sqrt{3}}( 1,1,1) ,\quad \left\langle \zeta \right\rangle =\frac{v_\zeta }{\sqrt{2}}( 1,0,1) ,\quad \left\langle \rho \right\rangle = \frac{v_\rho }{\sqrt{3}}( 1,1,1) , \nonumber \\ \left\langle \varphi \right\rangle= & {} \frac{v_\varphi }{\sqrt{3}}\left( \cos \alpha +e^{i\psi }\sin \alpha ,\omega \left( \cos \alpha +\omega e^{i\psi }\sin \alpha \right) ,\omega ^2\left( \cos \alpha +\omega ^2e^{i\psi }\sin \alpha \right) \right) , \nonumber \\ \left\langle \phi \right\rangle= & {} \frac{v_\phi }{\sqrt{3}}\left( \cos \alpha -e^{-i\psi }\sin \alpha ,\omega ^2\left( \cos \alpha -\omega ^2e^{-i\psi }\sin \alpha \right) ,\omega \left( \cos \alpha -\omega e^{-i\psi }\sin \alpha \right) \right) , \nonumber \\ \omega= & {} e^{\frac{2\pi i}{3}}, \end{aligned}$$

(16)

which are consistent with the scalar potential minimization equations for a large region of parameter space, as shown in detail in Refs. [39, 134].

With the above particle content, the relevant Yukawa terms for the quark and lepton sectors invariant under the group ${\mathcal {G}}$ are:

$$\begin{aligned} -{\mathcal {L}}_Y^{( q) }= & {} y^{( T) }{\overline{Q}} _{3L}\chi T_R +y_{33}^{( U) }{\overline{Q}}_{3L}\eta U_{3R}\nonumber \\&+y_{22}^{( U) }\varepsilon _{abc}{\overline{Q}}_{2L}^a\eta ^b \chi ^c U_{2R}\frac{( \xi \xi ) _{\mathbf {{\mathbf {1}}}}}{ \Lambda ^3 }+y_{11}^{( U) }\varepsilon _{abc}{\overline{Q}} _{1L}^a \eta ^b\chi ^c U_{1R}\frac{\left( \xi ^3 \zeta \right) _{ \mathbf {{\mathbf {1}}}}}{\Lambda ^{5}} \nonumber \\&+y_1^{( J) }{\overline{Q}}_{1L}\chi ^*J_{1R}+y_2^{( J) }{\overline{Q}}_{2L}\chi ^*J_{2R}\nonumber \\&+y_{33}^{( D) }\varepsilon _{abc}{\overline{Q}} _{3L}^a\left( \eta ^*\right) ^b\left( \chi ^*\right) ^c D_{3R}\frac{\sigma }{\Lambda ^2} +y_{22}^{( D) }{\overline{Q}} _{2L}\eta ^*D_{2R}\frac{\left( \xi ^2\zeta \right) _{\mathbf {\mathbf { 1}}}}{\Lambda ^3 } \nonumber \\&+y_{11}^{( D) }{\overline{Q}}_{1L}\eta ^*D_{1R}\frac{\left( \xi ^4 \zeta \right) _{\mathbf {{\mathbf {1}}}}}{\Lambda ^5}+y_{12}^{( D) }{\overline{Q}}_{1L}\eta ^*D_{2R}\frac{\left( \xi ^2\zeta \right) _{\mathbf {{\mathbf {1}}}}\sigma }{\Lambda ^4}\nonumber \\&+y_{13}^{( D) }{\overline{Q}}_{1L}\eta ^*D_{3R}\frac{\sigma ^6}{\Lambda ^6} +y_{23}^{( D) }{\overline{Q}}_{2L}\eta ^*D_{3R}\frac{\sigma ^5}{\Lambda ^5}+H.c, \end{aligned}$$

(17)

$$\begin{aligned} -{\mathcal {L}}_Y^{( l) }= & {} y_1 ^{( L) }\varepsilon _{abc}\left( {\overline{L}}_L^a\left( \eta ^*\right) ^b\left( \chi ^*\right) ^c\rho \right) _{\mathbf {{\mathbf {1}}}}e_{1R}\frac{\sigma ^6}{{\Lambda ^8}}\nonumber \\&+y_2^{( L) }\varepsilon _{abc}\left( {\overline{L}}_L^a\left( \eta ^*\right) ^b\left( \chi ^*\right) ^c \varphi \right) _{\mathbf {{\mathbf {1}}}}e_{2R}\frac{\sigma ^2}{ \Lambda ^4}+\frac{y_3 ^{( L) }}{\Lambda ^2}\varepsilon _{abc}\left( {\overline{L}}_L^a \left( \eta ^*\right) ^b\left( \chi ^*\right) ^c\rho \right) _{\mathbf {{\mathbf {1}}}}e_{3R}\nonumber \\&+z_1^{(L) }\varepsilon _{abc}\left( {\overline{L}} _L^a \left( \eta ^*\right) ^b\left( \chi ^*\right) ^c \phi \right) _{\mathbf {{\mathbf {1}}}}e_{1R}\frac{\sigma ^6}{{\Lambda ^8}}+\frac{ z_3 ^{( L) }}{\Lambda ^2}\varepsilon _{abc}\left( {\overline{L}} _L^a \left( \eta ^*\right) ^b\left( \chi ^*\right) ^c \phi \right) _{\mathbf {{\mathbf {1}}}}e_{3R} \nonumber \\&+y^{(L)}_\rho \varepsilon _{abc}\varepsilon _{dec}\left( {\overline{L}} _L^a\left( L_L^C\right) ^b\right) _{\mathbf {3a}}\eta ^d\chi ^e \frac{\zeta \sigma ^{11}}{\Lambda ^{13}}\nonumber \\&+y_{1\eta }^{( L) }\left( {\overline{L}}_L\eta N_R\right) _{\mathbf {3s}}\frac{\xi \sigma ^*}{ \Lambda ^2}+y_{2\eta }^{( L) }\left( {\overline{L}}_L\eta N_R \right) _{\mathbf {3a}}\frac{\xi \sigma ^*}{\Lambda ^2}\nonumber \\&+y_\chi ^{( L) }\left( {\overline{L}}_L\chi N_R\right) _{ {\mathbf {1}}^\prime }\frac{\sigma ^*}{\Lambda }+H.c.\ , \end{aligned}$$

(18)

where the dimensionless couplings y, z in Eqs. (17) and (18) are ${\mathcal {O}}(1)$ parameters. In addition to these terms, the symmetries unavoidably allow the following terms:

$$\begin{aligned}&y^{( T3) }{\overline{Q}}_{3L}\chi U_{3R} \frac{(\xi ^{*2}\xi )_1}{\Lambda ^3}, \quad y^{( 3T) }{\overline{Q}}_{3L}\eta T_{R} \frac{(\xi ^*\xi ^2)_1}{\Lambda ^3} \\&y^{(J_2D_3)}{\overline{Q}}_{2L}\chi ^*D_{3R} \frac{(\xi ^2\xi ^*)_1\sigma ^5}{\Lambda ^8}, \quad y^{( D_3J_2) }\varepsilon _{abc}{\overline{Q}} _{3L}^a\left( \eta ^*\right) ^b\left( \chi ^*\right) ^c D_{3R}\frac{(\xi ^{*2}\xi )_{1'}\sigma ^{*4} }{\Lambda ^8}. \end{aligned}$$

These terms will generate very small mixing angles of the third generation SM up and down type quarks with the exotic quarks. Such mixing angles are of the order of $\lambda ^5$ and $\lambda ^{11}$ (being $\lambda =0.225$), for the up and down type quarks, respectively, thus allowing us to safely neglect these strongly suppressed corrections, which will not be considered in our analysis. Furthermore, as it will shown in Sect. 3, the quark assignments under the different group factors of our model will give rise to SM quark mass textures where the CKM quark mixing angles only arise from the down type quark sector. As indicated by the current low energy quark flavor data encoded in the standard parametrization of the quark mixing matrix, the complex phase responsible for CP violation in the quark sector is associated with the quark mixing angle in the 1-3 plane. Thus, the Yukawa coupling $y_{13}^{( D) }$ in Eq. (17) is required to be complex in order to successfully reproduce the experimental values of the quark mixing angles and CP violating phase.

In a generic scenario the Yukawa couplings are complex. However, not all of them are physical. Some phases can be rotated away by the phase rotation of the quark and lepton fields. The conditions for the rotation away of the Yukawa phases in the quark sector by the redefinition of the phases $\alpha _{f}$ of the quark fields are:

$$\begin{aligned} \arg \left( y_{33}^{\left( D\right) }\right) -\alpha _{Q_{3L}}+\alpha _{D_{3R}}= & {} 0,\quad \arg \left( y_{23}^{\left( D\right) }\right) -\alpha _{Q_{2L}}+\alpha _{D_{3R}}=0, \nonumber \\ \arg \left( y_{13}^{\left( D\right) }\right) -\alpha _{Q_{1L}}+\alpha _{D_{3R}}= & {} 0,\quad \arg \left( y_{22}^{\left( D\right) }\right) -\alpha _{Q_{2L}}+\alpha _{D_{2R}}=0, \nonumber \\ \arg \left( y_{12}^{\left( D\right) }\right) -\alpha _{Q_{1L}}+\alpha _{D_{2R}}= & {} 0,\quad \arg \left( y_{11}^{\left( D\right) }\right) -\alpha _{Q_{1L}}+\alpha _{D_{1R}}=0, \nonumber \\ \arg \left( y_{1}^{\left( J\right) }\right) -\alpha _{Q_{1L}}+\alpha _{J_{1R}}= & {} 0,\quad \arg \left( y_{2}^{\left( J\right) }\right) -\alpha _{Q_{2L}}+\alpha _{J_{2R}}=0, \nonumber \\ \arg \left( y_{11}^{\left( U\right) }\right) -\alpha _{Q_{1L}}+\alpha _{U_{1R}}= & {} 0,\quad \arg \left( y_{22}^{\left( U\right) }\right) -\alpha _{Q_{2L}}+\alpha _{U_{2R}}=0, \nonumber \\ \arg \left( y_{33}^{\left( U\right) }\right) -\alpha _{Q_{3L}}+\alpha _{U_{3R}}= & {} 0,\quad \arg \left( y^{\left( T\right) }\right) -\alpha _{Q_{3L}}+\alpha _{T_{R}}=0,\nonumber \\ \end{aligned}$$

(19)

Consequently all the Yukawa phases in the quark sector can be rotated away, unless one considers phases of the scalar fields. Therefore, without considering phase rotation of the scalar fields, all the Yukawa couplings of the quark sector can be set real. Thus, in view of the above, the observed CP violation in the quark sector will arise from complex vacuum expectation values of the gauge singlet scalars charged under the discrete symmetries of the model. Therefore, the spontaneous breaking of the discrete symmetries of our model gives rise to the observed CP violation in the quark sector. This mechanism of generating CP violation in the fermion sector from the spontaneous breaking of the discrete groups is called Geometrical CP violation and has been implemented in other models. A concise review of group theoretical origin of CP violation is provided in Ref. [135]

Next, we explain the reason for introducing the discrete group factors in our model. We introduce the $A_4$ and $Z_{14}$ discrete groups with the aim of reducing the number of model parameters, thus making our model more predictive. In addition, these discrete groups allow us to get predictive and viable textures for the fermion sector capable of successfully explaining the observed pattern of fermion masses and mixing angles, as will be shown in Sects. 3 and 5. The $A_4$ and $Z_{14}$ discrete groups select the allowed entries of the mass matrices for SM quarks.

The $Z_8$ discrete symmetry separates the $A_4$ scalar triplet $\xi$ participating in the charged lepton Yukawa interactions from the remaining $A_4$ scalar triplets. The $Z_{14}$ discrete symmetry separates the $A_4$ scalar triplet $\zeta$ participating in the Dirac neutrino Yukawa interactions from the $A_4$ scalar triplet $\xi$ appearing in some of the neutrino Yukawa interactions involving the right-handed Majorana neutrinos $N_{iR}$ ($i=1,2,3$). Let us note that the different $A_4\times Z_{14}\times Z_{22}$ charge assignments for the quark fields shown in Table 2 give rise to a CKM quark mixing matrix solely emerging from the down type quark sector. The spontaneous breaking of the $Z_{14}\times Z_{22}$ discrete group yields the hierarchical structure of the SM charged fermion mass matrix and quark mixing angles. Furthermore, the $Z_{22}$ symmetry is the smallest cyclic symmetry allowing one to construct a Dirac Yukawa term $\left( {\overline{L}}_L^a\left( L_L^C\right) ^b\right) _{\mathbf {3a}}\eta ^d\chi ^e\frac{\zeta \sigma ^{11}}{\Lambda ^{13}}$ of dimension thirteen from an $\frac{\sigma ^{11}}{\Lambda ^{11}}$ insertion on the $\left( {\overline{L}}_L^a\left( L_L^C\right) ^b\right) _{\mathbf {3a}}\eta ^d\chi ^e\frac{\zeta }{\Lambda ^2}$ operator, necessary for obtaining the required $\lambda ^{19}$ suppression (where $\lambda =0.225$ is one of the Wolfenstein parameters) crucial for natural explanation of the smallness of the Dirac neutrino mass matrix and thus of the light active neutrino masses, as it will be explained in more detail in Sect. 5. Thus, in view of the above, the hierarchy among charged fermion masses and quark mixing angles is caused by the spontaneous breaking of the $A_4\times Z_{14}\times Z_{22}$ discrete group. Consequently, the quark masses are related with the quark mixing angles and we therefore set the VEVs of the scalar fields $\eta$, $\chi$, $\sigma$ , $\xi _j$, $\zeta _j$ ($j=1,2,3$) with respect to the Wolfenstein parameter $\lambda$ and the model cutoff $\Lambda$, as follows:

$$\begin{aligned} v_\eta \sim \lambda ^4\Lambda< v_\zeta \sim \lambda ^3 \Lambda< v_\chi \sim \lambda ^2\Lambda < v_\xi \sim v_\sigma \sim v_\rho \sim v_\varphi \sim v_\phi \sim \lambda \Lambda \,. \end{aligned}$$

(20)

It is worth mentioning, as follows from Eqs. (17) and (18) that the Yukawa interactions have a total of 21 parameters from which 18 are assumed to be real and 3 are taken to be complex. However, not all of these parameters enter in the physical observables of the quark and lepton sectors. Such physical observables are determined by the resulting low energy SM fermion mass matrices which do depend on effective parameters which contain some of the Yukawa couplings as well as the VEVs of the scalar fields of the model. After the assumption shown in Eq. (20) is made and the benchmarks described in Sects. 3 and 4 are considered, the number of effective parameters can be reduced.

Furthermore, the VEV hierarchy $v_{\eta } \ll v_{\chi }\sim v_\zeta \ll v_\xi \sim v_\sigma \sim v_\rho \sim v_\varphi \sim v_\phi$ is followed from the SSB chain of Eq. (1) and it also follows from gauge boson mass expressions: for example, masses of the SM gauge bosons depend on $v_\eta$ while masses of new gauge bosons (X,Y) and $Z'$ depend on $v_\chi$. In addition, the VEV hierarchy $v_\zeta \ll v_\xi \sim v_\sigma \sim v_\rho \sim v_\varphi \sim v_\phi$ can be explained by appropriate relations between the different mass coefficients of the bilinear terms of the scalar potential and the VEVs of such scalar fields. This can be explicitly shown by considering the simplified scenario of two singlet scalar fields $S_1$ and $S_2$, whose VEVs satisfy the hierarchy $v_{S_2} \ll v_{S_1}$. The scalar potential for such singlet fields is:

$$\begin{aligned} V=-\mu _{S_1}^2\left| S_1\right| ^2-\mu _{S_{2}}^{2}\left| S_{2}\right| ^{2}+\lambda _{1}\left| S_{1}\right| ^{4}+\lambda _{2}\left| S_{2}\right| ^{4}+\lambda _{3}\left| S_{1}\right| ^{2}\left| S_{2}\right| ^{2}. \end{aligned}$$

(21)

Its minimization implies:

$$\begin{aligned} \mu _{S_{1}}^{2}=2\lambda _{1}v_{S_{1}}^{2}+\lambda _{3}v_{S_{2}}^{2}, \quad \mu _{S_{2}}^{2}=2\lambda _{2}v_{S_{2}}^{2}+\lambda _{3}v_{S_{1}}^{2}. \end{aligned}$$

(22)

Thus, the VEV hierarchy $v_{S_{2}} \ll v_{S_{1}}$, can be justified by requiring $\mu _{S_{2}}^{2}\simeq 2\mu _{S_{1}}^{2}$ and considering the case where the quartic scalar couplings satisfy $\lambda _i\simeq \lambda$ ($i=1,2,3$). A straightforward but tedious extension of the aforementioned argument will yield to a large set of relationships between the different mass coefficients of the bilinear terms of the scalar potential and the VEVs of the large number of gauge singlet scalar fields of our model that will generate the VEV hierarchy shown in Eq. (20).

It is worth mentioning that there are several operators invariant under the $SU\left( 3\right) _{C}\times SU\left( 3\right) _{L}\times U\left( 1\right) _{X}$ gauge symmetry that can generate flavor and/or baryon number violation. Following [136], we find that these operators are given by:

$$\begin{aligned}&{\overline{L}}_{iL}L_{jL}^{C}L_{kL}\overline{L_{rL}^{C}}, \quad {\overline{L}}_{iL}e_{jR}{\overline{Q}}_{nL}U_{kR},\quad {\overline{Q}}_{nL}Q_{mL}^{C} {\overline{Q}}_{3L}L_{iL}^{C},\quad {\overline{Q}}_{nL}L_{iL}^{C}U_{iR}^{C}{\overline{D}}_{jR}, \nonumber \\&{\overline{L}}_{iL}e_{jR}Q_{3L}{\overline{D}}_{kR}, \quad {\overline{Q}}_{nL}Q_{3L}^{C}{\overline{U}}_{jR}e_{kR}^{C},\quad {\overline{D}}_{iR}U_{jR}^{C} {\overline{U}}_{kR}e_{rR}^{C},\quad {\overline{Q}}_{nL}Q_{3L}^{C}U_{iR}\overline{D_{jR}^{C}}, \nonumber \\&{\overline{L}}_{iL}Q_{3L}^{C}Q_{3L}\overline{L_{jL}^{C} },\quad {\overline{Q}}_{3L}Q_{mL}^{C}Q_{3L} \overline{Q_{sL}^{C}},\quad {\overline{Q}}_{nL}Q_{mL}^{C}Q_{pL}\overline{Q_{sL}^{C}},\quad {\overline{Q}}_{3L}Q_{3L}^{C}Q_{3L}\overline{Q_{3L}^{C}}. \qquad \end{aligned}$$

(23)

where all subindices go from 1 to 3 excepting n, m, s and p, which take the values of 1 and 2. However, all these operators, excepting ${\overline{Q}}_{3L}Q_{3L}^{C}Q_{3L}\overline{Q_{3L}^{C}}$, are forbidden by the $A_{4}\times Z_{8}\times Z_{14}\times Z_{22}$ discrete symmetry. Despite this operator contributes to proton decay, it is phenomenologically innocent, since its contribution is suppressed by the eight power of the very small $\theta _{13}^{\left( q\right) }\sim \lambda ^{4}$ ($\lambda =0.225$) quark mixing angle.

3 Quark masses and mixings

From the quark Yukawa interactions given by Eq. (17) we find the following expressions for the non-vanishing elements of the SM up and down quark mass matrices

$$\begin{aligned}&M_{U11}=y_{11}^{\left( U\right) }\frac{v_{\chi }\left\langle \left( \xi ^{3}\zeta \right) _{{\mathbf {1}}}\right\rangle }{\sqrt{2}\Lambda ^{5}} \frac{v}{\sqrt{2}},\quad M_{U22}=y_{22}^{\left( U\right) }\frac{v_{\chi }\left\langle \left( \xi \xi \right) _{{\mathbf {1}}}\right\rangle }{\sqrt{2}\Lambda ^{3}} \frac{v}{\sqrt{2}},\quad M_{U33}=y_{33}^{\left( U\right) } \frac{v}{\sqrt{2}},\nonumber \\&M_{D11}=y_{11}^{\left( D\right) }\frac{\left\langle \left( \xi ^{4}\zeta \right) _{{\mathbf {1}}}\right\rangle }{\Lambda ^{5}} \frac{v}{\sqrt{2}}, \quad M_{D12}=y_{12}^{\left( D\right) }\frac{v_{\sigma }\left\langle \left( \xi ^{2}\zeta \right) _{{\mathbf {1}}}\right\rangle }{\Lambda ^{4}} \frac{v}{\sqrt{2}},\quad M_{D13}=y_{13}^{\left( D\right) }\frac{v_{\sigma }^{6}}{\Lambda ^{6}} \frac{v}{\sqrt{2}}, \nonumber \\&M_{D22}=y_{22}^{\left( D\right) }\frac{\left\langle \left( \xi ^{2}\zeta \right) _{{\mathbf {1}}}\right\rangle }{\Lambda ^{3}} \frac{v}{\sqrt{2}},\quad M_{D23}=y_{23}^{\left( D\right) }\frac{v_{\sigma }^{5}}{\Lambda ^{5}} \frac{v}{\sqrt{2}}, \quad M_{D33}=-y_{33}^{\left( D\right) }\frac{v_{\chi }v_{\sigma }}{\sqrt{2}\Lambda ^{2}} \frac{v}{\sqrt{2}}.\nonumber \\ \end{aligned}$$

(24)

where $v= 246$ GeV is the scale of electroweak symmetry breaking and $\left\langle ....\right\rangle$ stands for the vacuum expectation value of the product of the singlet scalar fields. For the VEV pattern of our model (20) we find for the SM quark mass matrices:

$$\begin{aligned} M_U=\left( \begin{array}{ccc} a_1 ^{( U) }\lambda ^8 &{} 0 &{} 0 \\ 0 &{} a_2^{( U) }\lambda ^4 &{} 0 \\ 0 &{} 0 &{} a_3 ^{( U) } \end{array} \right) \frac{v}{\sqrt{2}}, M_D=\left( \begin{array}{ccc} a_{11}^{( D) }\lambda ^7 &{} a_{12}^{( D) }\lambda ^6 &{} a_{13}^{( D) }\lambda ^6 \\ 0 &{} a_{22}^{( D) }\lambda ^5 &{} a_{23}^{( D) }\lambda ^5 \\ 0 &{} 0 &{} a_{33}^{( D) }\lambda ^3 \end{array} \right) \frac{v}{\sqrt{2}}, \end{aligned}$$

(25)

where $a_1 ^{( U) }, a_{11}^{( D) }, ...$ are ${\mathcal {O}}(1)$ dimensionless parameters being products of the dimensionless couplings $y^{(K)}$ in Eq. (17).

Note that due to different $A_4\times Z_{14}\times Z_{22}$ charge assignments of the quark fields, the exotic and the SM quarks do not mix with each other. Thus, the exotic quark masses are:

$$\begin{aligned} m_T =y^{( T) }\frac{v_\chi }{\sqrt{2}},\quad m_{J^1 }=y_1 ^{( J) }\frac{v_\chi }{\sqrt{2}}=\frac{ y_1 ^{( J) }}{y^{( T) }}m_T ,\quad m_{J^2}=y_2^{( J) }\frac{v_\chi }{\sqrt{2}}=\frac{ y_2^{( J) }}{y^{( T) }}m_T . \end{aligned}$$

(26)

As seen from Eq. (25), the model has ten physical parameters, allowing one reproduce any value of ten observables: six quark masses, three mixing angles and one Jarlskog CP invariant shown in Table 3. The corresponding values of the model parameters are:

$$\begin{aligned} \begin{array}{c} a_1^{( U) }\simeq 1.085\,,\quad a_2^{( U) }\simeq 1.391\,,\quad a_3 ^{( U) }\simeq 0.994\,, \\ a_{11}^{( D) }\simeq 0.527\,,\quad a_{22}^{( D) }\simeq 0.491\,,\quad a_{33}^{( D) }\simeq 1.438\,, \\ a_{12}^{( D) }\simeq 0.501\,,\quad \left| a_{13}^{( D) }\right| \simeq 0.467\,,\quad \arg (a_{13}^{( D) })\simeq -60.96^{\circ }\,,\quad a_{23}^{( D) }\simeq 1.210\,. \end{array} \end{aligned}$$

(27)

An important feature of the above result is that the absolute values of all a-parameters are of the order of unity. Thus, the symmetries of our model allow us to naturally explain the hierarchy of quark mass spectrum without appreciable tuning of these effective parameters.

Table 3 Model benchmark scenarios S-4, S-3, S-2 with four, three and two free parameters, respectively, as well as experimental values of the quark sector observables from Ref. [137, 138]

Another observation about the set of values given in Eq. (27) is that it shows rather particular pattern: some of them are practically equal between each other. This fact suggests to consider the following simplified benchmark scenarios with a limited number of the free parameters:

$$\begin{aligned} \hbox {S-4 (4 free parameters):}&\quad a_{11}^{(D) }=a_{12}^{(D) }=a_{22}^{(D)},\ \ \ a_1 ^{(U) }=a_{3}^{(U) }=1,\ \ \ a_{23}^{(D) }=a_{33}^{(D) }=a_{2}^{(U) }.\nonumber \\ \hbox {Best-fit values:}&\quad a_{2}^{(U) }\simeq 1.40,\ \ \ a_{11}^{(D) }\simeq 0.53, \ \ \ \left| a_{13}^{(D) }\right| \simeq 0.43,\nonumber \\ {}&\qquad \arg (a_{13}^{( D) })\simeq -60.86^{\circ } \ \end{aligned}$$

(28)

$$\begin{aligned} \hbox {S-3 (3 free parameters):}&\quad a_{11}^{(D) }=a_{12}^{(D) }=a_{22}^{(D) }= \left| a_{13}^{(D) }\right| ,\ \ \ a_1 ^{(U) }=a_{3}^{(U) }=1,\nonumber \\ {}&\qquad a_{23}^{(D) }=a_{33}^{(D) }=a_{2}^{(U) }.\nonumber \\ \hbox {Best-fit values:}&\quad a_{2}^{(U) }\simeq 1.40,\ \ \ a_{11}^{(D) }\simeq 0.45, \ \ \ \arg (a_{13}^{( D) })\simeq -60.9^{\circ }\ \end{aligned}$$

(29)

$$\begin{aligned} \hbox {S-2a (2 free parameters):}&\quad a_{11}^{(D) }=a_{12}^{(D) }=a_{22}^{(D) }= \left| a_{13}^{(D) }\right| ,\ \ \ a_1 ^{(U) }=a_{3}^{(U) }=1,\nonumber \\ {}&\qquad a_{23}^{(D) }=a_{33}^{(D) }=a_{2}^{(U) }, \ \ \ \nonumber \\ \hbox {Best-fit values:}&\quad a_{2}^{(U) }\simeq 1.40,\ \ \ a_{11}^{(D) }\simeq 0.45, \ \ \ \arg (a_{13}^{( D) })=-60^{\circ }. \nonumber \\ \hbox {S-2b (2 free parameters):}&\quad a_{11}^{(D) }=a_{12}^{(D) }=a_{22}^{(D) } =\left| a_{13}^{(D) }\right| ,\ \ \ a_1 ^{(U) }=a_{3}^{(U) }=1,\nonumber \\ {}&\qquad a_{23}^{(D) }=a_{33}^{(D) }=a_{2}^{(U) }, \ \ \ \nonumber \\ \hbox {Best-fit values:}&\quad a_{2}^{(U) }\simeq 1.40,\ \ \ a_{11}^{(D) }\simeq 0.47, \ \ \ \arg (a_{13}^{( D) })=-45^{\circ }. \end{aligned}$$

(30)

As seen from Table 3, all the quark observables are reproduced with a reasonable precision even in the 2-parameter scenarios S-2a and S-2b. This result hints that the model framework allows introduction of certain extra symmetries significantly reducing the number of free parameters. This possibility will be studied elsewhere.

Figure 1 shows the correlation of the quark mixing parameter $\sin \theta ^{(q)}_{13}$ with the Jarlskog invariant. To obtain this figure, the quark sector parameters were randomly generated in a range of values where the CKM parameters and the quark masses are inside the $3\sigma$ experimentally allowed range. Such correlation shows that that the quark mixing parameter $\sin \theta ^{(q)}_{13}$ and the Jarlskog invariant $J_q$ are located in the ranges $0.0033\lesssim \sin \theta ^{(q)}_{13}\lesssim 0.0040$ and $2.7\times 10^{-5}\lesssim J_q\lesssim 3.65\times 10^{-5}$, respectively. We also found in this numerical analysis that the remaining quark mixing parameters are in the following ranges: $0.223\lesssim \sin \theta ^{(q)}_{12}\lesssim 0.226$ and $0.040\lesssim \sin \theta ^{(q)}_{23}\lesssim 0.045$.

Fig. 1

Correlation of the quark mixing parameter $\sin \theta ^{(q)}_{13}$ with the Jarlskog invariant

Finally, the LHC signature of the exotic T, $J_{1}$ and $J_{2}$ quarks in our model is defined by the fact that they will mainly decay into a top quark plus neutral scalar and can be pair produced at the LHC via Drell–Yan and gluon fusion processes mediated by charged gauge bosons and gluons, respectively. Consequently, we consider the observation of an excess of events in the multijet and multilepton final state as the smoking gun of our model at the LHC. A detailed study of the collider phenomenology of the model is beyond the scope of this paper and is left for future studies.

4 Meson oscillations

It is worth mentioning that the non-universal $U(1)_X$ charge assignments for the left-handed quark fields give rise to flavor changing neutral processes (FCNC) mediated by the $Z^{\prime }$ gauge boson. These FCNC interactions contribute to the $K^0-{\bar{K}}^0$, $B^0_d-{\bar{B}}^0_d$ and $B^0_s-{\bar{B}}^0_s$ mass differences. It is worth mentioning that the $D^0-{\bar{D}}^0$ meson oscillations are absent at tree level since the symmetries of our model constrain the up type quark mass matrix to be diagonal. In this section, we discuss the implications of our model in the Flavor Changing Neutral Current (FCNC) interactions in the down type quark sector. The flavor violating $Z^{\prime }$ interactions in the down type quark sector produce meson oscillations. The $K^{0}-{\bar{K}}^{0}$, $B_{d}^{0}- {\bar{B}}_{d}^{0}$ and $B_{s}^{0}-{\bar{B}}_{s}^{0}$ meson mixings are described by the following effective Hamiltonians:

$$\begin{aligned}&{\mathcal {H}}_{eff}^{\left( K^{0}-{\bar{K}}^{0}\right) }\mathcal {=}\frac{4\sqrt{2 }G_{F}c_{W}^{4}m_{Z}^{2}}{\left( 3-4s_{W}^{2}\right) m_{Z^{\prime }}^{2}} \left| \left( V_{DL}^{*}\right) _{32}\left( V_{DL}\right) _{31}\right| ^{2}O^{\left( K^{0}-{\bar{K}}^{0}\right) }, \end{aligned}$$

(31)

$$\begin{aligned}&{\mathcal {H}}_{eff}^{\left( B_{d}^{0}-{\bar{B}}_{d}^{0}\right) }\mathcal {=}\frac{ 4\sqrt{2}G_{F}c_{W}^{4}m_{Z}^{2}}{\left( 3-4s_{W}^{2}\right) m_{Z^{\prime }}^{2}}\left| \left( V_{DL}^{*}\right) _{31}\left( V_{DL}\right) _{33}\right| ^{2}O^{\left( B_{d}^{0}-{\bar{B}}_{d}^{0}\right) }, \end{aligned}$$

(32)

$$\begin{aligned}&{\mathcal {H}}_{eff}^{\left( B_{s}^{0}-{\bar{B}}_{s}^{0}\right) }\mathcal {=}\frac{ 4\sqrt{2}G_{F}c_{W}^{4}m_{Z}^{2}}{\left( 3-4s_{W}^{2}\right) m_{Z^{\prime }}^{2}}\left| \left( V_{DL}^{*}\right) _{32}\left( V_{DL}\right) _{33}\right| ^{2}O^{\left( B_{s}^{0}-{\bar{B}}_{s}^{0}\right) }. \end{aligned}$$

(33)

The $K^{0}-{\bar{K}}^{0}$, $B_{d}^{0}-{\bar{B}}_{d}^{0}$ and $B_{s}^{0}-{\bar{B}} _{s}^{0}$ meson mixings in our model is caused by the tree level $Z^{\prime }$ exchange, thus giving generating the following operators:

$$\begin{aligned} O^{\left( K^{0}-{\bar{K}}^{0}\right) }= & {} \left( {\overline{s}}\gamma _{\mu }P_{L}d\right) \left( {\overline{s}}\gamma ^{\mu }P_{L}d\right) ,\quad O^{\left( B_{d}^{0}-{\bar{B}}_{d}^{0}\right) }=\left( {\overline{d}}\gamma _{\mu }P_{L}b\right) \left( {\overline{d}}\gamma ^{\mu }P_{L}b\right) , \end{aligned}$$

(34)

$$\begin{aligned} O^{\left( B_{s}^{0}-{\bar{B}}_{s}^{0}\right) }= & {} \left( {\overline{s}}\gamma _{\mu }P_{L}b\right) \left( {\overline{s}}\gamma ^{\mu }P_{L}b\right) . \end{aligned}$$

(35)

Furthermore, the following relations have been taken into account:

$$\begin{aligned} {\widetilde{M}}_{f}= & {} \left( M_{f}\right) _{diag}=V_{fL}^{\dagger }M_{f}V_{fR},\quad \quad f_{\left( L,R\right) }=V_{f\left( L,R\right) }{\widetilde{f}}_{\left( L,R\right) }, \nonumber \\ {\overline{f}}_{iL}\left( M_{f}\right) _{ij}f_{jR}= & {} \overline{{\widetilde{f}}} _{kL}\left( V_{fL}^{\dagger }\right) _{ki}\left( M_{f}\right) _{ij}\left( V_{fR}\right) _{jl}{\widetilde{f}}_{lR}\nonumber \\= & {} \overline{{\widetilde{f}}}_{kL}\left( V_{fL}^{\dagger }M_{f}V_{fR}\right) _{kl}{\widetilde{f}}_{lR}=\overline{ {\widetilde{f}}}_{kL}\left( {\widetilde{M}}_{f}\right) _{kl}{\widetilde{f}} _{lR}=m_{fk}\overline{{\widetilde{f}}}_{kL}{\widetilde{f}}_{kR}, \nonumber \\ k= & {} 1,2,3\,. \end{aligned}$$

(36)

Here, ${\widetilde{f}}_{k\left( L,R\right) }$ and $f_{k\left( L,R\right) }$ ($k=1,2,3$) are the SM fermionic fields in the mass and interaction bases, respectively.

It is worth mentioning as shown in detail in “Appendix B,” that our model has the alignment limit for the lightest 126 GeV SM-like Higgs boson given that the remaining scalars are much heavier than the electroweak symmetry breaking scale 246 GeV. Furthermore, our model at low energies, below the scale the scale of breaking of the $SU(3)_C\times SU(3)_L\times U(1)_X$ gauge symmetry, corresponds to a multiscalar singlet extension of the SM. Thus, the light 126 GeV Higgs boson will not induce tree-level FCNC. This phenomenologically dangerous effect can happen in the presence of at least two SM doublet scalars before the electroweak symmetry breaking. To avoid this trouble, one can resort to the Glashow–Weinberg–Paschos theorem [139, 140] stating that there will be no tree-level FCNC coming from the scalar sector, if all right-handed fermions of a given electric charge couple to only one of the doublets.

Besides that, the contributions to FCNC arising from the heavier scalars are strongly suppressed by their large mass scale and the very small mixings of the scalar singlets and the CP even neutral component of $\chi$ with the CP even electrically neutral component of $\eta$ (which is mostly composed of the 126 GeV SM-like Higgs boson). Because of this reason the FCNC interactions in our model mainly arise from the tree-level exchange of the $Z^{\prime }$ gauge boson. This situation is different than the one presented in 3-3-1 models with three scalar triplets like the ones considered in [69,70,71], where two of the three scalar triplets do acquire VEVs at the electroweak symmetry breaking scale thus implying that at low energies below the TeV scale, the theory corresponds to a 2HDM where tree-level neutral scalar contributions to FCNC do exist. This problem was elegantly solved in Refs. [69,70,71] by implementing the Froggatt–Nielsen mechanism in this version of the 3-3-1 model.

On the other hand, the $K-{\bar{K}}$, $B_{d}^{0}-{\bar{B}}_{d}^{0}$ and $B_{s}^{0}-{\bar{B}}_{s}^{0}$ mass splittings are given by:

$$\begin{aligned} \Delta m_{K}= & {} \left( \Delta m_{K}\right) _{SM}+\Delta m_{K}^{\left( NP\right) },\quad \Delta m_{B_{d}}=\left( \Delta m_{B_{d}}\right) _{SM}+\Delta m_{B_{d}}^{\left( NP\right) },\nonumber \\ \Delta m_{B_{s}}= & {} \left( \Delta m_{B_{s}}\right) _{SM}+\Delta m_{B_{s}}^{\left( NP\right) }, \end{aligned}$$

(37)

where $\left( \Delta m_{K}\right) _{SM}$, $\left( \Delta m_{B_{d}}\right) _{SM}$ and $\left( \Delta m_{B_{s}}\right) _{SM}$ are the SM contributions, whereas $\Delta m_{K}^{\left( NP\right) }$, $\Delta m_{B_{d}}^{\left( NP\right) }$ and $\left( \Delta m_{B_{s}}\right) _{SM}$ are new physics contributions.

In our model, the new physics contributions to the meson differences are given by:

$$\begin{aligned} \Delta m_{K}^{\left( NP\right) }= & {} \frac{4\sqrt{2}G_{F}c_{W}^{4}m_{Z}^{2}}{ \left( 3-4s_{W}^{2}\right) m_{Z^{\prime }}^{2}}\left| \left( V_{DL}^{*}\right) _{32}\left( V_{DL}\right) _{31}\right| ^{2}f_{K}^{2}B_{K}\eta _{K}m_{K}, \end{aligned}$$

(38)

$$\begin{aligned} \Delta m_{B_{d}}^{\left( NP\right) }= & {} \frac{4\sqrt{2}G_{F}c_{W}^{4}m_{Z}^{2}}{ \left( 3-4s_{W}^{2}\right) m_{Z^{\prime }}^{2}}\left| \left( V_{DL}^{*}\right) _{31}\left( V_{DL}\right) _{33}\right| ^{2}f_{B_{d}}^{2}B_{B_{d}}\eta _{B_{d}}m_{B_{d}}, \end{aligned}$$

(39)

$$\begin{aligned} \Delta m_{B_{s}}^{\left( NP\right) }= & {} \frac{4\sqrt{2}G_{F}c_{W}^{4}m_{Z}^{2}}{ \left( 3-4s_{W}^{2}\right) m_{Z^{\prime }}^{2}}\left| \left( V_{DL}^{*}\right) _{32}\left( V_{DL}\right) _{33}\right| ^{2}f_{B_{s}}^{2}B_{B_{s}}\eta _{B_{s}}m_{B_{s}}. \end{aligned}$$

(40)

Using the following parameters [141,142,143,144,145,146,147]:

$$\begin{aligned} \Delta m_{K}= & {} \left( 3.484\pm 0.006\right) \times 10^{-12}\,\mathrm{MeV},\quad \left( \Delta m_{K}\right) _{SM}=3.483\times 10^{-12}MeV \\ f_{K}= & {} 160\,\mathrm{MeV},\quad B_{K}=0.85,\quad \eta _{K}=0.57, \quad m_{K}=497.614\,\mathrm{MeV}.\\ \left( \Delta m_{B_{d}}\right) _{\exp }= & {} \left( 3.337\pm 0.033\right) \times 10^{-10}\,\mathrm{MeV},\quad \left( \Delta m_{B_{d}}\right) _{SM}=3.582\times 10^{-10}\,\mathrm{MeV}, \\ f_{B_{d}}= & {} 188\,\mathrm{MeV},\quad B_{B_{d}}=1.26,\quad \eta _{B_{d}}=0.55,\quad m_{B_{d}}=5279.5\,\mathrm{MeV}. \\ \left( \Delta m_{B_{s}}\right) _{\exp }= & {} \left( 104.19\pm 0.8\right) \times 10^{-10}\,\mathrm{MeV},\quad \left( \Delta m_{B_{s}}\right) _{SM}=121.103\times 10^{-10}\,\mathrm{MeV}, \\ f_{B_{s}}= & {} 225\,\mathrm{MeV},\quad B_{B_{s}}=1.26,\quad \eta _{B_{s}}=0.55,\quad m_{B_{s}}=5366.3\,\mathrm{MeV}. \end{aligned}$$

Fig. 2

The $K^{0}-{\bar{K}}^{0}$, $B_{d}^{0}-{\bar{B}}_{d}^{0}$ and $B_{s}^{0}-{\bar{B}}_{s}^{0}$ mass splittings as function of the $Z^{\prime }$ mass

We plot in Fig. 2 the $K^{0}-{\bar{K}}^{0}$, $B_{d}^{0}-{\bar{B}} _{d}^{0}$ and $B_{s}^{0}-{\bar{B}}_{s}^{0}$ mass splittings as function of the $Z^{\prime }$ mass. As seen from Fig. 2, the $K^{0}-{\bar{K}} ^{0}$, $B_{d}^{0}-{\bar{B}}_{d}^{0}$ and $B_{s}^{0}-{\bar{B}}_{s}^{0}$ oscillations caused by the flavor changing neutral interactions reach values close to their experimental upper limits and the constraints arising from these meson oscillations set the $Z^{\prime }$ mass in the range 7 TeV$\lesssim m_{Z^{\prime }}\lesssim 8$ TeV.

5 Lepton masses and mixings

From the charged lepton Yukawa terms, we find the charged lepton mass matrix in the form:

$$\begin{aligned} M_l\!=\! & {} {{ \left( \!\! \begin{array}{ccc} a_1\lambda ^9\!+\!b_1\lambda ^9 \left( \cos (\alpha )\!-\!e^{-i \psi } \sin (\alpha )\right) &{} b_2\lambda ^5 \left( \cos (\alpha )\!+\!e^{i \psi } \sin (\alpha )\right) &{} a_3\lambda ^3\!+\!b_3\lambda ^3 \left( \cos (\alpha )\!-\!e^{-i \psi } \sin (\alpha )\right) \\ a_1\lambda ^9\!+\!b_1\lambda ^9 \omega ^2 \left( \!\cos (\alpha )\!-\!e^{-i \psi } \omega ^2 \sin (\alpha )\!\right) &{} b_2\lambda ^5 \omega \left( \!\cos (\alpha )\!+\!e^{i \psi } \omega \sin (\alpha )\!\right) &{} a_3\lambda ^3\!+\!b_3\lambda ^3 \omega ^2 \left( \!\cos (\alpha )\!-\!e^{-i \psi } \omega ^2 \sin (\alpha )\!\right) \\ a_1\lambda ^9\!+\!b_1\lambda ^9 \omega \left( \!\cos (\alpha )\!-\!e^{-i \psi } \omega \sin (\alpha )\right) &{} b_2\lambda ^9 \omega ^2 \left( \!\cos (\alpha )\!+\!e^{i \psi } \omega ^2 \sin (\alpha )\!\right) &{} a_3\lambda ^3\!+\!b_3\lambda ^3 \omega \left( \!\cos (\alpha )\!-\!e^{-i \psi } \omega \sin (\alpha )\!\right) \\ \end{array}\!\!\right) }}\nonumber \\&\times \frac{v_{\eta }}{\sqrt{2}}, \end{aligned}$$

(41)

where $a_1$, $a_3$, $b_i$ ($i=1,2,3$) are ${\mathcal {O}}(1)$ parameters constructed of the parameters $y^{(L)}_{i}, z^{(L)}$. Note that the charged lepton masses are linked to the scale of the electroweak symmetry breaking through their power dependence on the Wolfenstein parameter $\lambda$, with ${\mathcal {O}}(1)$ coefficients. Furthermore, from the lepton Yukawa terms given in Eq. (18) it follows that our model does not feature flavor changing leptonic neutral Higgs decays at tree level.

For the neutrino sector we find from Eq. (18) the neutrino mass term:

$$\begin{aligned} - 2{\mathcal {L}}_{mass}^{( \nu ) }=\left( \begin{array}{ccc} \overline{\nu _L^{\ C}}&\overline{\nu _R }&\overline{N_R } \end{array} \right) M_\nu \left( \begin{array}{c} \nu _L \\ \nu _R^{\ C} \\ N_R^{\ C} \end{array} \right) +H.c, \end{aligned}$$

(42)

where $\nu _{iR}\equiv ((\nu ^c)_L)^C$ corresponds to the third components of the lepton triplet introduced in Eq. (4). The $A_4$ family symmetry of the model constrains the neutrino mass matrix to be of the form:

$$\begin{aligned} M_\nu =\left( \begin{array}{ccc} 0_{3\times 3} &{} M_1 &{} M_2 \\ M_1 ^T &{} 0_{3\times 3} &{} M_3 \\ M_2^T &{} M_3^T &{} 0_{3\times 3} \end{array} \right) \end{aligned}$$

(43)

with

$$\begin{aligned} M_1= & {} \frac{v_\eta v_\chi v_\zeta }{2\sqrt{2}\Lambda ^2}\left( \frac{v_\sigma }{\Lambda }\right) ^{11}\left( \begin{array}{ccc} 0 &{} \omega ^2 &{} 0 \\ -\omega ^2 &{} 0 &{} 1 \\ 0 &{} -1 &{} 0 \end{array} \right) ,\nonumber \\ M_2= & {} y_{1\eta }^{( L) }\frac{ v_\eta v_\xi }{\sqrt{6}\Lambda }\left( \frac{ v_\sigma }{\Lambda }\right) \left( \begin{array}{ccc} 0 &{} ( 1+x) \omega ^2 &{} ( 1-x) \omega \\ ( 1-x) \omega ^2 &{} 0 &{} 1+x \\ ( 1+x) \omega &{} 1-x &{} 0 \end{array} \right) , \nonumber \\ M_3= & {} y_\chi ^{( L) }\frac{v_\chi }{\sqrt{2}}\left( \frac{ v_\sigma }{\Lambda }\right) \left( \begin{array}{ccc} 1 &{} 0 &{} 0 \\ 0 &{} \omega &{} 0 \\ 0 &{} 0 &{} \omega ^2 \end{array} \right) ,\quad \quad x=\frac{y_{2\eta }^{( L) }}{ y_{1\eta }^{( L) }},\quad \quad \omega =e^{\frac{2\pi i}{3}}. \end{aligned}$$

(44)

The light active masses arise from linear seesaw mechanism and the physical neutrino mass matrices are:

$$\begin{aligned} M_\nu ^{( 1) }= & {} -\left[ M_2M_3 ^{-1}M_1 ^T +M_1 \left( M_3^T \right) ^{-1}M_2^T \right] \,, \end{aligned}$$

(45)

$$\begin{aligned} M_\nu ^{( 2) }= & {} -\frac{1}{2} \left( M_3 +M_3^T \right) + \frac{1}{4}\left[ M_1^T(M_3^*)^{-1}M_2^{*}+M_2^{\dagger }(M_3^{\dagger } )^{-1} M_1\right] \,, \end{aligned}$$

(46)

$$\begin{aligned} M_\nu ^{( 3) }= & {} \frac{1}{2} \left( M_3 +M_3 ^T \right) +\frac{1}{4}\left[ M_1^T(M_3^*)^{-1}M_2^{*}+M_2^{\dagger }(M_3^{\dagger } )^{-1} M_1\right] \, \,, \end{aligned}$$

(47)

where $M_\nu ^{( 1) }$ is the active neutrino mass matrix, whereas $M_\nu ^{( 2) }$ and $M_\nu ^{( 3)}$ are the sterile neutrino mass matrices. Explicitly we have

$$\begin{aligned} M_\nu ^{( 1) }= & {} \frac{y_{1\eta }^{( L) }}{\sqrt{2} y_\chi ^{( L) }}\left( \frac{v_\sigma }{\Lambda }\right) ^{11} \frac{v_\eta v_\zeta v_\xi }{\Lambda ^3 }\left( \begin{array}{ccc} -2(x+1) &{} \omega ^2(x-1) &{} 2\omega x \\ \omega ^2(x-1) &{} -4\omega x &{} x+1 \\ 2\omega x &{} x+1 &{} -2\omega ^2(x-1) \end{array} \right) \frac{v_\eta }{\sqrt{2}} \nonumber \\= & {} \left( \begin{array}{ccc} -2(x+1) &{} \omega ^2(x-1) &{} 2\omega x \\ \omega ^2(x-1) &{} -4\omega x &{} x+1 \\ 2\omega x &{} x+1 &{} -2\omega ^2(x-1) \end{array} \right) m_\nu ,\quad m_\nu =\frac{a_\nu \lambda ^{19}v}{\sqrt{2}}\,. \end{aligned}$$

(48)

The experimental values of charged lepton masses, the neutrino mass squared splittings, the leptonic mixing parameters and Dirac CP violating phase can be reproduced for the normal ordering (NO) of the neutrino mass spectrum with the following values of the model effective parameters:

$$\begin{aligned} a_1\simeq & {} 0.983,\quad a_3\simeq -0.483,\quad b_1\simeq -0.755,\nonumber \\ b_2\simeq & {} -0.597,\quad b_3\simeq -0.199,\quad x\simeq 0.431,\nonumber \\ m_\nu\simeq & {} 16.34\ \text{ meV },\quad \alpha \simeq 122.25^{\circ }\,,\quad \beta \simeq -42.82^{\circ }\,,\nonumber \\ \gamma\simeq & {} -59.36^{\circ }\,,\quad \psi \simeq 98.44^{\circ }\,. \end{aligned}$$

(49)

Using the values of the lepton model effective parameters of Eq. (49), the PMNS leptonic mixing matrix takes the form:

$$\begin{aligned} U_{PMNS}= & {} U^{\dagger }_{l}U_{\nu }\nonumber \\= & {} \left( \begin{array}{ccc} -0.818231-0.0686404 i &{} -0.318382+0.449127 i &{} 0.148954\, +0.0227392 i \\ 0.0515003\, +0.373766 i &{} -0.379958-0.371145 i &{} 0.731222\, +0.202101 i \\ -0.180118-0.388575 i &{} 0.634519\, -0.110392 i &{} 0.564605\, -0.288074 i \\ \end{array} \right) \nonumber \\ \end{aligned}$$

(50)

where:

$$\begin{aligned} U_l= & {} \left( \begin{array}{ccc} -0.625827&{} -0.614417 &{} -0.48045 \\ -0.406057-0.298325 i &{} 0.726662\, +0.202306 i &{} -0.400359+0.129878 i \\ 0.569876\, -0.172341 i &{} -0.112245-0.202306 i &{} -0.59877+0.483205 i \\ \end{array} \right) ,\nonumber \\ U_{\nu }= & {} \left( \begin{array}{ccc} 0.566967 &{} 0.12785 &{} -0.813759 \\ 0.396158\, +0.686167 i &{} -0.177447-0.307348 i &{} 0.248135\, +0.429782 i \\ -0.112675+0.195159 i &{} -0.463062+0.802046 i &{} -0.151256+0.261983 i \\ \end{array} \right) .\nonumber \\ \end{aligned}$$

(51)

As seen from Table 4, the model values are consistent with the experimental ones. Again, akin to the quark sector, the absolute value of the effective dimensionless parameters $a^{(l)}, x$ are of the order of unity. We interpret this fact in a way that the lepton mass hierarchy is explained on account of the model structure, symmetries and field content, without unnatural tuning these effective parameters.

Table 4 Model and experimental values of the light active neutrino masses, leptonic mixing angles and CP violating phase for the scenario of normal (NH) neutrino mass hierarchy

Fig. 3

Correlations between the different lepton sector observables

Figure 3 shows the correlations of the leptonic mixing angles with the leptonic Dirac CP-violating phase as well as the correlations between the leptonic mixing parameters. To obtain these Figures, the lepton sector parameters were randomly generated in a range of values where the neutrino mass squared splittings, leptonic mixing parameters and leptonic Dirac CP violating phase are consistent with the experimental data. These lepton sector observables are inside the $1\sigma$ experimentally allowed range, excepting $\theta ^{(l)}_{23}$ which is inside the $3\sigma$ range. We found the leptonic Dirac CP violating phase in the range $-90^{\circ }\lesssim \delta ^{(l)}_{CP}\lesssim -25^{\circ }$, whereas the leptonic mixing angles are obtained to be in the ranges $31.5^{\circ }\lesssim \theta ^{(l)}_{12}\lesssim 37.5^{\circ }$, $48.0^{\circ }\lesssim \theta ^{(l)}_{23}\lesssim 51.5^{\circ }$ and $8.15^{\circ }\lesssim \theta ^{(l)}_{12}\lesssim 8.9^{\circ }$.

Let us consider the effective Majorana neutrino mass parameter

$$\begin{aligned} m_{\beta \beta }=\left| \sum _j U_{ek}^2m_{\nu _k}\right| , \end{aligned}$$

(52)

where $U_{ej}$ and $m_{\nu _k}$ are the PMNS leptonic mixing matrix elements and the neutrino Majorana masses, respectively. The neutrinoless double beta ($0\nu \beta \beta$) decay amplitude is proportional to $m_{\beta \beta }$.

Figure 4 shows the correlation of the effective Majorana neutrino mass parameter $m_{ee}$ vs the lightest neutrino mass $m_1$.

Fig. 4

Correlations of the effective Majorana neutrino mass parameter $m_{\beta \beta }$ with the lightest neutrino mass $m_1$ and with the leptonic mixing parameters

As can be seen from Fig. 4, our model predicts the values of the effective Majorana neutrino mass parameter in the range 24.5 meV$\lesssim m_{\beta \beta }\lesssim$ 27 meV, which is within the declared reach of the next-generation bolometric CUORE experiment [150] or, more realistically, of the next-to-next-generation ton-scale $0\nu \beta \beta$-decay experiments. The current most stringent experimental upper limit $m_{\beta \beta }\le 160$ meV is set by $T_{1/2}^{0\nu \beta \beta }(^{136}\mathrm {Xe})\ge 1.1\times 10^{26}$ yr at 90% C.L. from the KamLAND-Zen experiment [151].

6 $Z^\prime$ gauge boson production at the LHC

Here, we compute the total cross section for the production of the heavy $Z^\prime$ gauge boson, defined in Eq. (10), at the LHC via Drell–Yan mechanism. We consider the dominant contribution due to the parton distribution functions of the light up, down and strange quarks, so that the total cross section for the production of a $Z^\prime$ via quark antiquark annihilation in proton–proton collisions with center of mass energy $\sqrt{S}$ takes the form:

$$\begin{aligned}&\sigma _{pp\rightarrow Z^\prime }^{\left( DrellYan\right) }( S)\nonumber \\&\quad =\frac{g^2 \pi }{6c_W^2 S}\left\{ \left[ \left( g_{uL}^\prime \right) ^2 +\left( g_{uR}^\prime \right) ^2 \right] \int _{\ln \sqrt{ \frac{m_{Z^\prime }^2 }{S}}}^{-\ln \sqrt{\frac{m_{Z^\prime }^2 }{S}} }f_{p/u}\left( \sqrt{\frac{m_{Z^\prime }^2 }{S}}e^y,\mu ^2 \right) f_{p/{\overline{u}}}\left( \sqrt{\frac{m_{Z^\prime }^2 }{S}}e^{-y},\mu ^2 \right) dy\right. \nonumber \\&\qquad +\left. \left[ \left( g_{dL}^\prime \right) ^2 +\left( g_{dR}^\prime \right) ^2 \right] \int _{\ln \sqrt{\frac{m_{Z^\prime }^2 }{S}}}^{-\ln \sqrt{\frac{m_{Z^\prime }^2 }{S}}}f_{p/d}\left( \sqrt{\frac{m_{Z^\prime }^2 }{S}}e^{y},\mu ^2 \right) f_{p/{\overline{d}}}\left( \sqrt{\frac{ m_{Z^\prime }^2 }{S}}e^{-y},\mu ^2 \right) dy\right. \nonumber \\&\qquad +\left. \left[ \left( g_{dL}^\prime \right) ^2 +\left( g_{dR}^\prime \right) ^2 \right] \int _{\ln \sqrt{\frac{m_{Z^\prime }^2 }{S}}}^{-\ln \sqrt{\frac{m_{Z^\prime }^2 }{S}}}f_{p/s}\left( \sqrt{\frac{m_{Z^\prime }^2 }{S}}e^y,\mu ^2 \right) f_{p/{\overline{s}}}\left( \sqrt{\frac{ m_{Z^\prime }^2 }{S}}e^{-y},\mu ^2 \right) dy\right\} \, , \end{aligned}$$

(53)

where $g_{uL(R)}^\prime$, $g_{dL(R)}^\prime$ are the $Z^\prime$ couplings to left (right)-handed up and down type quarks, respectively. These couplings are given in “Appendix D.” The functions $f_{p/u}\left( x_1,\mu ^2 \right)$ ($f_{p/{\overline{u}}}\left( x_2,\mu ^2 \right)$), $f_{p/d}\left( x_1,\mu ^2 \right)$ ($f_{p/{\overline{d}}}\left( x_2,\mu ^2 \right)$) and $f_{p/s}\left( x_1,\mu ^2 \right)$ ($f_{p/{\overline{s}}}\left( x_2,\mu ^2 \right)$) are the distributions of the light up, down and strange quarks (antiquarks), respectively, in the proton which carry momentum fractions $x_1$ ($x_2$) of the proton.

The factorization scale is taken to be $\mu =m_{Z^\prime }$.

Fig. 5

Total cross section for the $Z^\prime$ production via Drell–Yan mechanism at the LHC for $\sqrt{S}=13$ TeV (left plot) and $\sqrt{S}=28$ TeV (right plot) and as a function of the $Z^\prime$ mass

Figure 5 (left panel) displays the $Z^\prime$ total production cross section at the LHC via the Drell–Yan mechanism for $\sqrt{S}=13$ TeV as a function of the $Z^\prime$ mass $M_{Z'}$ in the range from 7 TeV up to 8 TeV. We consider $M_{Z'}\ge 7$ TeV in order to fulfill the bound arising from the experimental data on K, $B_d$ and $B_s$ meson mixings obtained in Sect. 4.For this region of $Z^\prime$ masses we find that the total production cross section ranges from 0.11 fb up to 0.01 fb. The heavy neutral $Z^\prime$ gauge boson, after being produced, will subsequently decay into the pair of the SM particles, with the dominant decay mode into quark–antiquark pairs as shown in Refs. [9, 152]. The two body decays of the $Z^\prime$ gauge boson in 3-3-1 models have been studied in detail in Ref. [152]. In particular, in Ref. [152] it has been shown that in 3-3-1 models the $Z^\prime$ decays into a lepton pair have branching ratios of the order of $10^{-2}$, which implies that the total LHC cross section for the $pp\rightarrow Z^\prime \rightarrow l^+l^-$ resonant production at $\sqrt{S}=13$ TeV will be of the order of $10^{-3}$ fb for a 7 TeV $Z^\prime$ gauge boson, which is below its corresponding lower experimental limit from the LHC searches [153]. On the other hand, at the proposed energy upgrade of the LHC up to 28 TeV center of mass energy, the total cross section for the Drell–Yan production of a heavy $Z^\prime$ neutral gauge boson gets significantly enhanced reaching values ranging from 82 fb up to 30 fb, as indicated in the right panel of Fig. 5. Consequently, the LHC cross section for the $pp\rightarrow Z^\prime \rightarrow l^{+}l^{-}$ resonant production at $\sqrt{S}=28$ TeV will be of the order of 1 fb for a 7 TeV $Z^\prime$ gauge boson, which is consistent with its corresponding lower experimental limit arising from the LHC searches [153].

7 Lepton flavor violating decays

Let us analyze the implications of our model for the LFV decays of the SM charged leptons and Higgs boson.

Given that the SM charged lepton mass matrix (56) cannot be diagonalized analytically in the practically useful form, in this section, for the sake of simplicity, we restrict ourselves to a simplified benchmark scenario characterized by the relations:

$$\begin{aligned}&z_1^{( L) }=y_1 ^{( L) },\ \ v_\phi =e^{i\gamma }\sin \beta v_{\varphi }, \ \ v_\rho =v_\varphi \cos \beta ,\ \ y_3 ^{( L) }=-e^{-i\gamma }y_4^{( L) }\tan \beta ,\nonumber \\&\quad z_3^{( L) }=e^{-i\gamma }y_4^{( L) }\cot \beta . \end{aligned}$$

(54)

Then, the charged lepton mass matrix takes the form:

$$\begin{aligned} M_l= & {} R_{lL}diag\left( m_e, m_\mu , m_\tau \right) ,\nonumber \\ R_{lL}= & {} \frac{1}{\sqrt{3}}\left( \begin{array}{ccc} 1 &{} 1 &{} 1 \\ 1 &{} \omega &{} \omega ^2 \\ 1 &{} \omega ^2 &{} \omega \end{array} \right) \left( \begin{array}{ccc} 1 &{} 0 &{} 0 \\ 0 &{} \cos \alpha &{} -\sin \alpha e^{-i\psi }\\ 0 &{} \sin \alpha e^{i\psi } &{} \cos \alpha \end{array} \right) \left( \begin{array}{ccc} \cos \beta &{} 0 &{} -\sin \beta e^{-i\gamma } \\ 0 &{} 1 &{} 0 \\ \sin \beta e^{i\gamma } &{} 0 &{} \cos \beta \end{array} \right) , \nonumber \\ \omega= & {} e^{\frac{2\pi i}{3}}, \end{aligned}$$

(55)

where the charged lepton masses are:

$$\begin{aligned} m_e =a_1 ^{( l) }\lambda ^{9}\frac{v}{\sqrt{2}},\quad m_\mu =a_2^{( l) }\lambda ^{5}\frac{v}{\sqrt{2}},\quad m_\tau =a_3 ^{( l) }\lambda ^3 \frac{v}{\sqrt{2}}. \end{aligned}$$

(56)

In “Appendix B” we derived an expression (B13) for the SM Higgs boson, $h^{0}_{1}$, as a linear combination of the scalars present in our model. We combine such relations with the definitions of the charged lepton mass eigenstates and masses:

$$\begin{aligned} {\widetilde{M}}_{f}= & {} \left( M_{f}\right) _{diag}=V_{fL}^{\dagger }M_{f}V_{fR}, \quad \quad f_{\left( L,R\right) }=V_{f\left( L,R\right) } {\widetilde{f}}_{\left( L,R\right) }, \nonumber \\ {\overline{f}}_{iL}\left( M_{f}\right) _{ij}f_{jR}= & {} \overline{{\widetilde{f}}} _{kL}\left( V_{fL}^{\dagger }\right) _{ki}\left( M_{f}\right) _{ij} \left( V_{fR}\right) _{jl}{\widetilde{f}}_{lR}\nonumber \\= & {} \overline{{\widetilde{f}}} _{kL}\left( V_{fL}^{\dagger }M_{f}V_{fR}\right) _{kl}{\widetilde{f}}_{lR}= \overline{{\widetilde{f}}}_{kL}\left( {\widetilde{M}}_{f}\right) _{kl}\widetilde{ f}_{lR}=m_{fk}\overline{{\widetilde{f}}}_{kL}{\widetilde{f}}_{kR}, \nonumber \\ k= & {} 1,2,3 \,. \end{aligned}$$

(57)

where ${\widetilde{f}}_{k\left( L,R\right) }$ and $f_{k\left( L,R\right) }$ ($k=1,2,3$) are the SM fermion mass and interaction eigenstates, respectively.

Then, considering the first three terms in Eq.  (18) we find the $h^0_1ee$ couplings

$$\begin{aligned} -{\mathcal {L}}_{h^0_1 ee}&\subset \left( 1+ \frac{\xi _\eta }{v_\eta } + \frac{\xi _\chi }{v_\chi }\right) \left( m_{e_i} \overline{e_{iL}} e_{iR} + \mathrm {H.c.}\right) \nonumber \\ {}&\quad \rightarrow \frac{g}{2 m_W} \left( c_\alpha +s_{\alpha } t_\theta \right) h^0_1 \left( m_{e_i} \overline{e_{iL}} e_{iR} + \mathrm {H.c.}\right) , \end{aligned}$$

(58)

coinciding in the limit $s_{\alpha } \rightarrow 0$ with the SM ones. As seen from the above formula, there are no lepton flavor violating decays of the SM-like Higgs bosons (LFVHD) $h^0_1\rightarrow e_i^\pm e_j^{\mp }$ with $i\ne j$ at tree level. This is consistent with the latest experimental result, where no signals were found setting the upper bound Br$(h^0_1\rightarrow \tau ^{\mp }\mu ^{\pm },\tau ^{\mp } e^{\pm }) < \mathrm {O}(10^{-3})$ at 95 % confidence level [154, 155]. This feature distinguishes our model from some previous models with discrete symmetry that predicted tree-level LFVHD [156]. However, the SM-like Higgs bosons in our model still couple with the heavy neutrinos through the four last Yukawa terms in Eq. (18). Hence, the LFVHD may arise at one-loop level, as in the models of the standard seesaw, inverse seesaw, and 3-3-1 model with massive neutrinos and inverse seesaw mechanism [157,158,159,160,161,162]. While the standard seesaw model predicts suppressed branching ratios for LFVHD, these branchings can reach interesting values of the order of $10^{-5}$ in the models with inverse seesaw mechanisms. Recent studies predict that the experimental sensitivities for LFVHD can reach values of the order of $10^{-5}$ in the near future [163, 164].

The one-loop diagrams contributing to the LFV decays of $e_i\rightarrow e_j\gamma$ and the SM-like Higgs boson decay $h^0_1\rightarrow e_i e_j$ with $i\ne j$ are exactly the same as those that appear in the seesaw and inverse seesaw versions of the SM. The difference is the neutrino mixing matrix, arising from the linear seesaw mechanism. Hence, it will be interesting to estimate how large the Br$(h^0_1\rightarrow e_ie_j)$ can become under the current bounds of Br$(\mu \rightarrow e\gamma )< 4.2 \times 10^{-13}$ [165]. It is expected that the future experimental sensitivities to the LFV decays will be improved, namely $6\times 10^{-14}$ for Br$(\mu \rightarrow \,e\gamma )$ [166, 167], and about ${\mathcal {O}}(10^{-9})$ for the two decays Br$(\tau \rightarrow \,e\gamma )$ and Br$(\tau \rightarrow \,\mu \gamma )$ [168] (for a recent review see, for instance, Ref. [169]).

We will use the approximate formulas for the Br$(e_i\rightarrow e_j \gamma )$ in 3-3-1 models given in Ref. [170], which were checked to be well-consistent with the results obtained from the exact numerical computation. Other approaches used for discussions of LFV decays of charged leptons in 3-3-1 models were also given previously in the literature [28, 171, 172]. Analytic formulas for calculating the one-loop contributions to LFVHD in the unitary gauge are given in Ref. [32, 161, 162], and were shown to be consistent with previous works [160]. Using these formulas, we only determine couplings between physical states and ignore all Goldstone bosons.

From the definition of the $SU(3)_L\times U(1)_X$ covariant derivative (7) we find its part related with the charged gauge bosons in our model

$$\begin{aligned} \Pi ^{CC}_\mu \equiv \frac{1}{\sqrt{2}} \begin{pmatrix} 0 &{} W^+_\mu &{} 0 \\ W^-_\mu &{} 0 &{} Y^-_\mu \\ 0 &{} Y^+_\mu &{} 0 \end{pmatrix} . \end{aligned}$$

(59)

Hence, the couplings of the SM-like Higgs with the charged gauge bosons are given by:

$$\begin{aligned} {\mathcal {L}}_{h^0_1VV}&\subset \left( D_\mu \eta \right) ^\dagger \left( D^\mu \eta \right) +\left( D_\mu \chi \right) ^\dagger \left( D^\mu \chi \right) , \nonumber \\ {\mathcal {L}}_{h^0_1VV}&=g\, m_W c_{\alpha }h^0_1 W^{+\mu } W^-_\mu + g\,m_Yh^0_1 s_{\alpha }Y^{+\mu } Y^-_\mu . \end{aligned}$$

(60)

The matrix $U_{lL}$ in Eq. (55) will be used to change the basis of the left-handed charged leptons from the flavor basis to the physical one. Specifically, the correspondence between the original basis of the left-handed leptons and the physical one is $\overline{e_L}R_{l L}\leftrightarrow \overline{e_L}$, or $e_L\leftrightarrow R_{l L} e_L$, while the right-handed ones are unchanged. This means that $e_{iL} \rightarrow U_{lL,ij} e_{jL}$ and $\overline{e_{iL}} \rightarrow \overline{e_{jL}} U^\dagger _{lL,ji}$ with $i,j=1,2,3$.

From Eqs. (58) and (60), we note that the couplings SM-like Higgs boson with normal charged leptons and gauge boson $W^\pm$ in the model under consideration and the SM are $(c_{\alpha } +s_{\alpha } t_{\theta })$ and $c_{\alpha }$, respectively. The lower bound $m_{Z'}\ge 4$ TeV gives $v_{\chi }\ge 10$ TeV, which results in small $s_{\alpha }\simeq t_{2\alpha }/2 \sim t_{\theta }\sim v/v_{\chi }\sim {\mathcal {O}}(10^{-2})$, therefore $c_{\alpha }=1+{\mathcal {O}}(10^{-4})$. Similarly for the couplings of SM-like Higgs bosons with the SM quarks and the neutral gauge boson Z, where $\xi _{\eta }$ plays role of the SM Higgs boson after the first breaking step. After the second one, the physical state of the SM-like Higgs boson is $h^0_1 \simeq c_{\alpha } \xi _\eta$ and the relative difference the Z boson with other particle is $c_{\phi }$ with $s_{\phi }\sim v^2/v^2_{\chi }$ given in Eq. (D3). Hence, the largest relative differences between the couplings of the $h^0_1$ predicted by our model and the SM are $c_{\alpha }$ and $c_{\alpha }c_{\phi }$. As a consequence, these couplings of the SM-like Higgs bosons are still in the allowed regions constrained from experiments.

The neutrino mass matrix $M_\nu$ in Eq. (43) is diagonalized via an unitary $9\times 9$ matrix $U_\nu$, namely

$$\begin{aligned} U_\nu ^T M_\nu U_\nu&={\hat{M}}_\nu =\mathrm {diag}(m_{n_1},m_{n_2},..., {\hat{m}}_{n_9})=\mathrm {diag}({\hat{m}}_\nu ,\; {\hat{m}}_N ), \end{aligned}$$

(61)

where ${\hat{m}}_\nu =\mathrm {diag}(m_{n_1},\;m_{n_2},\;m_{n_3})$ and ${\hat{m}} _N =\mathrm {diag}(m_{n_4},\;m_{n_5},...,\;m_{n_9})$ are the masses of active and exotic neutrinos $n_L=(n_{1L},n_{2L},...,n_{9L})$. They are Majorana fermions that satisfy $n_{kR}=n^c_{kL}$ with $k=1,2,...,9$. Relations between the interaction and physical basis for the neutrino fields are: $(\overline{\nu ^C_L} \quad \overline{\nu _R} \quad \overline{N_R})=\overline{n_R}U^T_\nu$ and $(\nu _L \quad \nu ^C_R \quad N^C_R)^T= U_\nu n_L$.

The couplings of charged gauge bosons with leptons are given by

$$\begin{aligned} {\mathcal {L}}_{V^{\pm }\ell \ell }&= i(\overline{L_L}\gamma ^\mu P^{CC}_{ \mu }L_L)_1= \frac{g}{\sqrt{2}}\left( \overline{e_{iL}}\gamma ^\mu \nu _{iL} W^-_\mu +\overline{e_{iL}}\gamma ^\mu (\nu ^c_{i})_L Y^-_\mu +\mathrm {H.c.} \right) . \nonumber \\ \rightarrow {\mathcal {L}}_{V^{\pm }\ell \ell }&=\frac{g}{\sqrt{2}}\left[ (U_{lL})_{ji}(U_\nu )_{ ik} \overline{e_{jL}}\gamma ^\mu n_{kL} W^-_\mu +(U_{lL})_{ji}(U_\nu )_{ (i+3)k} \overline{e_{jL}}\gamma ^\mu n_{kL} Y^-_\mu \! +\!\mathrm {H.c.}\right] , \end{aligned}$$

(62)

where the sums are taken for $i,j=1,2,3$ and $k=1,2,..,9$, and we have used $(\nu ^c_{i})_L=\nu ^C_{iR}$.

Based on Eq. (18), couplings of SM-like Higgs boson with neutrinos are included in the following interactions:

$$\begin{aligned} -{\mathcal {L}}_{h^0_1nn}&\subset \left( 1 +\frac{\xi _\eta }{v_\eta } +\frac{ \xi _\chi }{v_\chi } \right) \left[ \overline{\nu ^C_L}M_1 \nu ^C_R +\mathrm { H.c.}\right] + \left( 1 +\frac{\xi _\eta }{v_\eta } \right) \left[ \overline{ \nu ^C_L}M_2 N^C_R +\mathrm {H.c.}\right] \nonumber \\&\quad +\left( 1 +\frac{\xi _\chi }{v_\chi } \right) \left[ \overline{\nu _R}M_3 N^C_R +\mathrm {H.c.}\right] , \nonumber \\ -{\mathcal {L}}_{h^0_1nn}&= \frac{g}{2 m_W}h^0_1 \left[ \left( c_{\alpha } +s_{\alpha }t_\theta \right) \left( \overline{\nu ^C_L}M_1 \nu ^C_R +\mathrm { H.c.}\right) + c_{\alpha }\left( \overline{\nu ^C_L}M_2 N^C_R +\mathrm {H.c.} \right) \right. \nonumber \\ {}&\quad \left. +s_\alpha t_\theta \left( \overline{\nu ^C_L}M_3 N^C_R +\mathrm { H.c.}\right) \right] \nonumber \\&= \frac{gc_{\alpha }}{2 m_W}h^0_1 \left[ \left( 1+ t_\alpha t_\theta \right) (U_\nu )_{ik} (M_1)_{ij} (U_\nu )_{(j+3)p} + (U_\nu )_{ik} (M_2)_{ij} (U_\nu )_{(j+6)p}\right. \nonumber \\&\quad +\left. t_{\alpha }t_\theta (U_\nu )_{(i+3)k} (M_3)_{ij} (U_\nu )_{(j+6)p}\right] \overline{n_{kR}} n_{pL} +\mathrm {H.c}., \end{aligned}$$

(63)

where the sums are taken for $i,j=1,2,3$ and $k,p=1,2,...,9$. By defining a symmetric coefficient $\lambda _{kp}=\lambda _{pk}$ satisfying

$$\begin{aligned} \lambda _{kp}\equiv & {} \left( 1+ t_{\alpha }t_\theta \right) (U_\nu )_{ik} (M_1)_{ij} (U_\nu )_{(j+3)p} + (U_\nu )_{ik} (M_2)_{ij} (U_\nu )_{(j+6)p} \nonumber \\&+ t_{\alpha }t_\theta (U_\nu )_{(i+3)k} (M_3)_{ij} (U_\nu )_{(j+6)p} + (k \leftrightarrow p), \end{aligned}$$

Eq. (63) can be written in the form

$$\begin{aligned} -{\mathcal {L}}_{h^0_1nn}= \frac{gc_\alpha }{4 m_W}h^0_1 \overline{n_k}\left[ \lambda _{kp} P_L + \lambda ^*_{kp} P_R\right] n_p, \end{aligned}$$

(64)

where $P_{L,R}=(1\mp \gamma _5)/2$ are chiral operators and $n_{p,k}$ are four-component spinors of Majorana neutrinos. This form of the couplings $h \overline{n_k}n_p$ allows us to use the Feynman rules in Ref. [173] for calculating LFVHD at one loop level.

Based on Ref. [170], the branching ratio for the $e_i\rightarrow e_j\gamma$ ($i>j$) decay takes the form:

$$\begin{aligned} \mathrm {Br}(e_i\rightarrow e_j\gamma )&= \frac{12\pi ^2}{G_F^2}|D_{ij}|^2\times \mathrm {Br}(e_i\rightarrow e_j\bar{\nu _j}\nu _i), \end{aligned}$$

(65)

where $G_F=g^2/(4\sqrt{2}m_W^2)$ and $D_{ij}$ is the one-loop contribution due to virtual charged gauge bosons and Majorana neutrinos running in the internal lines of the loops. Such contribution can be written as $D_{ij}=D^{W}_{ij} +D^{Y}_{ij}$, where:

$$\begin{aligned} D^W_{ij}&=-\frac{eg^2}{32\pi ^2m_W^2} \sum _{k=1}^9 \sum _{a,b=1}^3(U_{lL}^*)_{ib} (U^*_\nu )_{bk}(U_{lL})_{ja}(U_\nu )_{ak}F(t_{kW}), \nonumber \\ D^Y_{ij}&= -\frac{eg^2}{32\pi ^2m_Y^2} \sum _{k=1}^9 \sum _{a,b=1}^3 (U_{lL}^*)_{i(b+3)} (U^*_\nu )_{(b+3)k}(U_{lL})_{j(a+3)}(U_\nu )_{(a+3)k}F(t_{kY}), \end{aligned}$$

(66)

where

$$\begin{aligned} t_{kW} \equiv \frac{m^2_{n_k}}{m_W^2},\; t_{kY}\equiv \frac{m^2_{n_k}}{m_Y^2} , \quad F(x) \equiv -\frac{10-43x+78x^2-49x^3 +4x^4 +18x^3\ln (x)}{12(x-1)^4}. \end{aligned}$$

(67)

We note that F(x) was given in Ref. [174]. The above formulas were used in the inverse seesaw 3-3-1 models [162] and were confirmed to be numerically consistent with the previous work of  Ref. [28]. Numerical values of Br$(e_i\rightarrow e_j\bar{\nu _j}\nu _i)$ will be fixed as Br$(\mu \rightarrow e\bar{\nu _e}\nu _\mu )\simeq 100\%$, Br$(\tau \rightarrow e\bar{\nu _e} \nu _\tau )\simeq 17.82\%$, and Br$(\tau \rightarrow \mu \bar{\nu _\mu }\nu _\tau )\simeq 17.39\%$ [138]. At low energy we take $g^2=e^2/s^2_W=4\pi \alpha _{\mathrm {em}}/s^2_W$, where $\alpha _{\mathrm {em}}\simeq 1/137$ and $s^2_W\simeq 0.231$.

For the LFVHD, one loop diagrams for Br$(h^0_1\rightarrow e_i e_j)$ are shown in Fig. 6.

Fig. 6

One-loop diagrams contributing to the SM-like Higgs boson decay $h^0_1\rightarrow e_ie_j$ in the unitary gauge, $V^{\pm }=W^{\pm },Y^{\pm }$

The decay width for the process $h_1^0\rightarrow e_ie_j$ is given by:

$$\begin{aligned} \Gamma (h_1^0\rightarrow e_ie_j)\equiv \Gamma (h_1^0\rightarrow e_i^{-} e_j^{+})+\Gamma (h^0_1\rightarrow e_i^{+} e_j^{-}) = \frac{ m_{h^0_1} }{8\pi }\left( \vert \Delta _{(ij)L}\vert ^2+\vert \Delta _{(ij)R}\vert ^2\right) ,\nonumber \\ \end{aligned}$$

(68)

with the condition $m_{h_1^0}\gg m_{i,j}$ being $m_{i,j}$ the charged lepton masses.

The corresponding branching ratio is

$$\begin{aligned} \mathrm{Br}(h^0_1\rightarrow e_ie_j)= \Gamma (h_1^0\rightarrow e_ie_j)/\Gamma ^{\mathrm {total}}_{h_1^0} \end{aligned}$$

(69)

where $\Gamma ^{\mathrm {total}}_{h_1^0}\simeq 4.1\times 10^{-3}$ GeV [175]. We define the $\Delta _{(ij)L,R}$ functions

$$\begin{aligned} \Delta _{(ij)L,R} =\sum _{i=1}^4 \left( \Delta ^{(i)W}_{(ij)L,R} + \Delta ^{(i)Y}_{(ij)L,R}\right) , \end{aligned}$$

(70)

where analytic forms for the functions in the r.h.s. are shown in “Appendix C” (for detailed calculations, see Refs. [32, 161]). The above formulas were crosschecked using FORM [176, 177].

Numerical input parameters we use for the analysis of the LFV processes correspond to the benchmark point given in Eq. (49), which implies that the corresponding values of the physical observables of the lepton sector are automatically consistent with the neutrino oscillation experimental data. The mixing matrix of the charged lepton sector is fixed as given in Eq. (51). The neutrino mixing matrix $U_\nu$ and neutrino masses can be numerically determined from Eq. (61), by using the numerical parameters given in (49). According to our estimates $M^{(1)}_\nu$ is nearly independent of $v_\chi$. On the other hand the heavy neutrino masses show significant $v_\chi$-dependence, because they get main contributions from $M_3$ given in Eq. (44). Furthermore they are nearly degenerate, which implies, $m_{n_4}\simeq m_{n_5}\simeq ...\simeq \,m_{n_9}\simeq y^{(L)}_\chi \frac{v_\chi }{\sqrt{2}}\left( \frac{v_\sigma }{\Lambda }\right)$ as indicated by Eqs. (46) and (47). Hence, we can see the dependence of the LFV branching ratios on the heavy neutrino masses, which are related to $v_{\chi }$ as shown by Eqs. (46), (47) and (44). Besides the two VEVs $v_\phi$ and $v_\rho$ that were fixed in the discussion of the charged lepton sector, we choose $v_\xi =v_\sigma =v_\phi =\lambda \Lambda$, while the three factors in front of the matrices $M_{1,2,3}$ in Eq. (44) can be written in terms of $y_{1,2}$ as follows

$$\begin{aligned} y_1v_\eta \lambda ^{16}&\equiv \frac{v_\eta v_\chi v_\zeta }{2\sqrt{2}\Lambda ^2}\left( \frac{v_\sigma }{\Lambda }\right) ^{11}, \quad y_{1\eta }^{( L) }\frac{ v_\eta v_\xi }{\sqrt{6}\Lambda }\left( \frac{ v_\sigma }{\Lambda }\right) \!\equiv \! y_2v_\eta \lambda ^2, \quad y_\chi ^{( L) }\frac{v_\chi }{\sqrt{2}}\left( \frac{ v_\sigma }{\Lambda }\right) = \frac{ y_1y_2 v^2_\eta \lambda ^{18}}{m_\nu }, \end{aligned}$$

(71)

where $y_{1,2} \sim {\mathcal {O}}(1)$. In our numerical analysis we fix $\Lambda \simeq 96$ TeV, and the CP-even neutral Higgs mixing parameters are set as follows $s_\alpha =0$, $c_\alpha =1$. In addition, we consider values for the $Z'$ mass satisfying $M_{Z'} >4$ TeV, which correspond to a $SU(3)_L\times U(1)_X$ symmetry breaking scale fulfilling $v_\chi >10$ TeV, as derived from the approximate formula $M^2_{Z'}\simeq g^2c^2_Wv^2_{\chi }/(3-4s^2_W)$ [131]. Numerical results for Br$(e_i\rightarrow e_j\gamma )$ and Br$(h^0_1\rightarrow e_ie_j)$ depending on $y_1$ and $y_2$ are illustrated in Table 5 for $v_\chi =15$ TeV. For $v_\chi$ around this value, all numerical results are the same hence it is unnecessary to discuss them here.

Table 5 Branching ratios for the LFV decays with $v_{\chi }=15$ TeV. The second column presents the numerical values of the heavy neutrino masses

The product $y_1y_2$ is constrained by the perturbative limit of the Yukawa coupling $y_1y_2 \sim y^L_\chi <\sqrt{4\pi }\simeq 3.5$, as follows from Eq. (71). Table 5 shows the numerical values of the Branching ratios for the LFV decays with $v_{\chi }=15$ TeV and different values of the Yukawa couplings $y_1$ and $y_2$ and heavy neutrino masses. Notice that a specific value of $(y_1,y_2)$ in Table 5 will predict a value for the Yukawa coupling $y^L_\chi \simeq \sqrt{2}m_{n_4}/(v_\chi \lambda ) \le 3.5$, leading to $m_{n_4}\le 0.557 v_\chi$. Thus for $v_\chi =15$ TeV we have $m_{n_4}\le 8.35$ TeV.

Based on the numerical results reported in Table 5, we can see that Br$(\mu \rightarrow \,e\gamma )$ can reach values close to its recent experimental bound provided that $y_1$ is small enough. On the other hand, Br$(h^0_1\rightarrow \mu \tau )$ can reach ${\mathcal {O}}(10^{-5})$ values when $y_2$ is large enough, like for example $y_2=4.5$ as shown in Table 5. Furthermore, increasing $y_2$ will result in larger values for Br$(h^0_1\rightarrow \mu \tau )$. We can see that the Br$(h^0_1\rightarrow e_ie_j)$ is enhanced when the heavy neutrino mass $m_{n_4}$ is increased, which is a generic behavior observed in inverse seesaw models [160, 161]. Because the experiment data favors lower bounds of $y_1$, and the perturbative limit of $y^L_\chi$ and $v_\chi$ results in upper bounds of $y_2$, there exist upper bounds, which are order of ${\mathcal {O}}(10^{-5})$ and ${\mathcal {O}}(10^{-6})$ for the Branching ratios of the two decays $h^0_1\rightarrow \mu \tau ,e\tau$ for the numerical values of the free parameters chosen above. The remaining LFV decays $\tau \rightarrow \mu \gamma ,e\gamma$ and $h^0_1\rightarrow e\mu$ have much smaller Branching ratios than the characteristic sensitivities of current experimental searches.

8 Conclusions

We constructed a viable multiscalar singlet extension of the 3-3-1 model with two scalar triplets and three right-handed Majorana neutrinos where the tiny masses for the light active neutrinos are produced by the linear seesaw mechanism. Our model is based on the $A_4$ family symmetry, which is supplemented by other auxiliary symmetries. The observed pattern of the SM charged fermion masses and fermionic mixing parameters originates from the spontaneous breaking of the discrete symmetries of the model and does not require any fine-tuning of the model parameters.

We analyzed the implications of our model in the lepton flavor violating processes. We demonstrated that the branching ratio Br$(\mu \rightarrow \,e\gamma )$ can reach values close to the recent upper experimental bounds, thus constraining the values of Br$(\tau \rightarrow \mu \gamma )$ and Br$(\tau \rightarrow e\gamma )$ to be much smaller than the corresponding experimental sensitivities. On the other hand, the model allows Br$(h^0_1\rightarrow \mu \tau )$ and Br$(h^0_1\rightarrow \,e\tau )$ to reach the values of about ${\mathcal {O}}(10^{-5})$ and ${\mathcal {O}}(10^{-6})$, respectively. Besides that, we have studied the implications of our model in meson oscillations and we have found that our model is consistent with the constraints arising from meson mixings. We also studied the production of the heavy $Z^{\prime }$ gauge boson in proton–proton collisions via the Drell–Yan mechanism. We found that the corresponding total cross section ranges at the LHC from 0.11 fb up to 0.01 fb when the $Z^\prime$ gauge boson mass is varied within $7-8$ TeV interval. The $Z^\prime$ production cross section will be significantly enhanced at the proposed energy upgrade of the LHC with $\sqrt{S}=28$ TeV reaching typical values of $82-30$ fb. From these results we found that the $pp\rightarrow Z^\prime \rightarrow l^+ l^-$ resonant production cross section reach the values of about $10^{-3}$ fb and 1 fb for $M_{Z^\prime }=7$ TeV at the energies $\sqrt{s}=13$ TeV and $\sqrt{s}=28$ TeV, respectively.

The first value of the resonant production cross section is below and the second lies on the verge of the sensitivities of the LHC experiments at the corresponding energies.

Appendices

Appendix A: the product rules for $A_4$

The $A_4$ group has one three-dimensional ${\mathbf {3}}$ and three distinct one-dimensional ${\mathbf {1}}$, ${\mathbf {1}}^\prime$ and ${\mathbf {1}}^{\prime \prime }$ irreducible representations, satisfying the following product rules:

$$\begin{aligned}&{\mathbf {3}}\otimes {\mathbf {3}}={\mathbf {3}}_{s}\oplus {\mathbf {3}} _a\oplus {\mathbf {1}}\oplus {\mathbf {1}}^\prime \oplus {\mathbf {1}}^{\prime \prime }, \nonumber \\&{\mathbf {1}}\otimes {\mathbf {1}}={\mathbf {1}},\quad {\mathbf {1}}^{\prime }\otimes {\mathbf {1}}^{\prime \prime }={\mathbf {1}},\quad {\mathbf {1}} ^\prime \otimes {\mathbf {1}}^\prime ={\mathbf {1}}^{\prime \prime },\quad {\mathbf {1}}^{\prime \prime }\otimes {\mathbf {1}}^{\prime \prime }={\mathbf {1}} ^\prime , \end{aligned}$$

(A1)

Considering $\left( x_1 ,x_2,x_3 \right)$ and $\left( y_1 ,y_2,y_3 \right)$ as the basis vectors for two $A_4$-triplets ${\mathbf {3}}$, the following relations are fulfilled:

$$\begin{aligned}&\left( {\mathbf {3}}\otimes {\mathbf {3}}\right) _{{\mathbf {1}} }=x_1 y_1 +x_2y_2+x_3 y_3 , \nonumber \\&\left( {\mathbf {3}}\otimes {\mathbf {3}}\right) _{{\mathbf {3}}_s}=\left( x_2y_3 +x_3 y_2,x_3 y_1 +x_1 y_3 ,x_1 y_2+x_2y_1 \right) ,\nonumber \\&\quad \left( {\mathbf {3}}\otimes {\mathbf {3}}\right) _{{\mathbf {1}}^\prime }=x_1 y_1 +\omega x_2y_2+\omega ^2x_3 y_3 , \nonumber \\&\left( {\mathbf {3}}\otimes {\mathbf {3}}\right) _{{\mathbf {3}}_a}=\left( x_2y_3 -x_3 y_2,x_3 y_1 -x_1 y_3 ,x_1 y_2-x_2y_1 \right) ,\nonumber \\&\quad \left( {\mathbf {3}}\otimes {\mathbf {3}}\right) _{{\mathbf {1}}^{\prime \prime }}=x_1 y_1 +\omega ^2x_2y_2+\omega x_3 y_3 , \end{aligned}$$

(A2)

where $\omega =e^{i\frac{2\pi }{3}}$. The representation ${\mathbf {1}}$ is trivial, while the non-trivial ${\mathbf {1}}^\prime$ and ${\mathbf {1}} ^{\prime \prime }$ are complex conjugate to each other. Some reviews of discrete symmetries in particle physics are found in Refs. [178,179,180,181]. The discrete symmetry $A_4$ was first implemented to the 3-3-1 models in the Refss [11] and [182].

Appendix B: scalar sector

Here, we present more details about the scalar sector of our model containing the SM Higgs boson.

The scalar potential of the model can be split in the following two parts:

$$\begin{aligned} V_{S}=V^{\mathrm {invariant}}_{S} +V^{\mathrm {soft}}_{S}. \end{aligned}$$

(B1)

The first part $V^{\mathrm {invariant}}_{S}$ is invariant under the $A_4\times Z_8\times Z_{14}\times Z_{22}$ discrete and $SU(3)_C\times SU(3)_L\times U(1)_X$ gauge symmetries,

$$\begin{aligned} V^{\mathrm {invariant}}_{S}&=\mu _\chi ^2 \chi ^\dag \chi +\mu _\eta ^2 \eta ^\dag \eta + \mu _{\sigma }^2 \sigma ^*\sigma +\mu _\xi ^2(\xi ^*\xi )_1 + \mu _\zeta ^2(\zeta ^*\zeta )_1 \nonumber \\&\quad +\mu _{\rho } ^2(\rho ^*\rho )_1 +\mu _\varphi ^2(\varphi ^*\varphi )_1 +\mu _\phi ^2(\phi ^*\phi )_1 + \left[ \mu _{\phi \rho } ^2 (\phi ^*\rho )_1 +\mathrm {H.c.}\right] \nonumber \\&\quad + \lambda _\chi (\chi ^\dag \chi )^2 + \lambda _\eta (\eta ^\dag \eta )^2 + \lambda _{\sigma }(\sigma ^*\sigma )^2 + \sum _{S_i,S_j}\left[ (S_i^*S_i)(S_j^*S_j)\right] _1 \nonumber \\&\quad + \lambda _{\chi \eta } (\chi ^\dag \chi )(\eta ^\dag \eta ) + \lambda ^\prime _{\chi \eta } (\chi ^\dag \eta ) (\eta ^\dag \chi )+ (\sigma ^*\sigma ) \left[ \lambda _{\chi \sigma }(\chi ^\dag \chi )+ \lambda _{\eta \sigma }(\eta ^\dag \eta ) \right] \nonumber \\&\quad + \left\{ \left[ (\rho \rho ) (\phi ^*\phi ^*)\right] _1 + \left[ (\xi \rho ) (\varphi ^*\varphi ^*)\right] _1 + \left[ (\xi \phi ) (\varphi ^*\varphi ^*)\right] _1 + \mathrm {H.c.} \right\} \nonumber \\&\quad + (\sigma ^*\sigma ) \left\{ \sum _{S} \lambda _{S\sigma } (S^*S)_1 +\left[ \lambda _{\phi \rho \sigma } (\phi ^*\rho )_1 +\mathrm {H.c.} \right] \right\} \nonumber \\&\quad + (\chi ^\dagger \chi ) \left\{ \sum _{S} \lambda _{S\chi }(S^*S)_1 +\left[ \lambda _{\phi \rho \chi } (\phi ^*\rho )_1 +\mathrm {H.c.} \right] \right\} \nonumber \\&\quad + (\eta ^\dagger \eta ) \sum _{S} \lambda _{S\eta }(S^*S)_1 +\left[ \lambda _{\phi \rho \eta } (\phi ^*\rho )_1 +\mathrm {H.c.} \right] \nonumber \\&\quad + \sum _{S}\left\{ \left[ (\phi ^*\rho )(S^*S)\right] _1 +\mathrm {H.c.}\right\} , \end{aligned}$$

(B2)

where $S,S_i,S_j=\xi ,\zeta ,\rho ,\varphi ,\phi$ are the scalar fields defined in Eq. (5). The second part $V^{\mathrm {soft}}_{S}$ consists of $A_4\times Z_8\times Z_{14}\times Z_{22}$ soft-breaking terms needed to generate nonzero masses for the CP-odd neutral Higgs bosons as well as to solve the domain wall problem. The complete set of these soft-breaking terms is

$$\begin{aligned} V^{\mathrm {soft}}_{S}&= \mu '^2_{\sigma } \sigma ^2 + f_{\sigma } \sigma ^3 + \sum _{S} \left[ \mu ^2_{1',S} \left( S^2\right) _{1'} + f_S\left( S^2\right) _{1}\sigma \right] +\mathrm {H.c.}, \end{aligned}$$

(B3)

where $S=\xi ,\zeta ,\rho ,\varphi ,\phi$; all parameters $\mu '_{\sigma }$, $f_{\sigma }$, $\mu ^2_{1',S}$, and $f_S$ have the same dimension of mass.

The $A_4$-invariant products of four $A_4$-triplets x, y, z, t can be decomposed as:

$$\begin{aligned} \left[ (xy)(zt)\right] _1&\equiv \lambda ^{xyzt}_1 (xy)_1(zt)_1 + \lambda ^{xyzt}_2 (xy)_{1'}(zt)_{1''} + \lambda ^{xyzt}_3 (xy)_{1''}(zt)_{1'}\nonumber \\&\quad +\lambda ^{xyzt}_4 \left[ (xy)_{3s}(zt)_{3s}\right] _1 \nonumber \\&\quad +\lambda ^{xyzt}_5 \left[ (xy)_{3s}(zt)_{3a}\right] _1 +\lambda ^{xyzt}_6 \left[ (xy)_{3a}(zt)_{3s}\right] _1 +\lambda ^{xyzt}_7 \left[ (xy)_{3a}(zt)_{3a}\right] _1. \end{aligned}$$

(B4)

The products like $\left[ (xz)(yt)\right] _1$, $\left[ (xt)(yz)\right] _1$,... are not included in the scalar potential (B2) because they can always be written as linear combinations of the seven $A_4$-products in the right-hand side of Eq. (B4). This fact can be easily demonstrated, using the rules given in “Appendix A.” Let us note that due to the antisymmetry and symmetry properties of the $3_a$ and $3_s$ triplet components in the products $(\xi \xi )$ and $(\zeta \zeta )$, we obtain $(3_s3_a)_1 +\mathrm {H.c.}=0$. Hence, many terms of this kind does not appear in the scalar potential. Therefore, particular cases are written as

$$\begin{aligned} \left[ (S^*S)^2\right] _1&\equiv \lambda ^S _1(S^*S)_1(S^*S)_1 + \lambda ^S _2(S^*S)_{1^\prime }(S^*S)_{1^{\prime \prime }} +\lambda ^S _4 \left[ (S^*S)_{3_s}(S^*S)_{3_s}\right] _1\nonumber \\&\quad +\lambda ^S _7 \left[ (S^*S)_{3_a}(S^*S)_{3_a} \right] _1 , \nonumber \\ \left[ S_i^*S_iS_j^*S_j\right] _1&\equiv \lambda ^{S_iS_j}_1(S_i^*S_i)_1 (S_j^*S_j)_1 +\left[ \lambda ^{S_iS_j}_2(S_i^*S_i)_{1^\prime }(S_j^*S_j)_{1^{\prime \prime }}+ \mathrm {H.c.} \right] \nonumber \\&\quad +\lambda ^{S_iS_j}_4(S_i^*S_i)_{3_s}(S_j^*S_j)_{3_s} \nonumber \\&\quad +\lambda ^{S_iS_j}_7(S_i^*S_i)_{3_a}(S_j^*S_j)_{3_a}, \quad S_i\ne S_j, \nonumber \\ \left[ (\xi \xi )(\varphi ^*\varphi ^*)\right] _1&= \lambda '^{\xi \varphi }_1 (\xi \xi )_1(\varphi ^*\varphi ^*)_1 +\lambda '^{\xi \varphi }_2 (\xi \xi )_{1'}(\varphi ^*\varphi ^*)_{1''} +\lambda '^{\xi \varphi }_3 (\xi \xi )_{1''}(\varphi ^*\varphi ^*)_{1'}\nonumber \\&\quad +\lambda '^{\xi \varphi }_4 \left[ (\xi \xi )_{3s}(\varphi ^*\varphi ^*)_{3s}\right] _1 , \nonumber \\ \left[ (\rho \rho ) (\phi ^*\phi ^*)\right] _1&=\lambda '^{\rho \phi }_1 (\rho \rho )_1(\phi ^*\phi ^*)_1 +\lambda '^{\rho \phi }_2 (\rho \rho )_{1'}(\phi ^*\phi ^*)_{1''} +\lambda '^{\rho \phi }_3 (\rho \rho )_{1''}(\phi ^*\phi ^*)_{1'}\nonumber \\&\quad +\lambda '^{\rho \phi }_4 \left[ (\rho \rho )_{3s}(\phi ^*\phi ^*)_{3s} \right] _1 , \nonumber \\ \left[ (\xi \varphi ^*)(\rho \phi ^*)\right] _1&= \lambda ^{\xi \varphi \rho \phi ^*}_1 (\xi \varphi ^*)_1(\rho \phi ^*)_1 + \lambda ^{\xi \varphi \rho \phi ^*}_2 (\xi \varphi ^*)_{1'}(\rho \phi ^*)_{1''} \nonumber \\&\quad + \lambda ^{\xi \varphi \rho \phi ^*}_3 (\xi \varphi ^*)_{1''}(\rho \phi ^*)_{1'} +\lambda ^{\xi \varphi \rho \phi ^*}_4 \left[ (\xi \varphi ^*)_{3s}(\rho \phi ^*)_{3s}\right] _1 \nonumber \\&\quad +\lambda ^{\xi \varphi \rho \phi ^*}_5 \left[ (\xi \varphi ^*)_{3s}(\rho \phi ^*)_{3a}\right] _1 +\lambda ^{\xi \varphi \rho \phi ^*}_6 \left[ (\xi \varphi ^*)_{3a}(\rho \phi ^*)_{3s}\right] _1\nonumber \\&\quad +\lambda ^{\xi \varphi \rho \phi ^*}_7 \left[ (\xi \varphi ^*)_{3a}(\rho \phi ^*)_{3a}\right] _1, \nonumber \\ \left[ (\xi \varphi ^*)(\rho ^*\phi ) \right] _1&=\lambda ^{\xi \varphi \rho ^*\phi }_1 (\xi \varphi ^*)_1(\rho ^*\phi )_1 + \lambda ^{\xi \varphi \rho ^*\phi }_2 (\xi \varphi ^*)_{1'}(\rho ^*\phi )_{1''} \nonumber \\&\quad + \lambda ^{\xi \varphi \rho ^*\phi }_3 (\xi \varphi ^*)_{1''}(\rho ^*\phi )_{1'} +\lambda ^{\xi \varphi \rho ^*\phi }_4 \left[ (\xi \varphi ^*)_{3s}(\rho ^*\phi )_{3s}\right] _1 \nonumber \\&\quad +\lambda ^{\xi \varphi \rho ^*\phi }_5 \left[ (\xi \varphi ^*)_{3s}(\rho ^*\phi )_{3a}\right] _1 +\lambda ^{\xi \varphi \rho ^*\phi }_6 \left[ (\xi \varphi ^*)_{3a}(\rho ^*\phi )_{3s}\right] _1\nonumber \\&\quad +\lambda ^{\xi \varphi \rho ^*\phi }_7 \left[ (\xi \varphi ^*)_{3a}(\rho ^*\phi )_{3a}\right] _1. \end{aligned}$$

(B5)

The above scalar potential has a fairly large number of scalar self-interactions.

The VEV’s chosen in Eq. (16) must satisfy all the minimization conditions of the scalar potential (B2), namely

$$\begin{aligned} \left. \frac{\partial \,V_H}{\partial \,S^0}\right| _{S^0=\langle S^0\rangle ,\forall \,S^0}=0. \end{aligned}$$

(B6)

The model contains 20 neutral scalar components, where three of them have zero VEVs. This leads to 20 minimization equations relating the VEVs to the parameters of the scalar potential. We find that two equations for $\chi ^0_1$ and $\rho ^0_3$ are automatically satisfied. The remaining 18 equations allow expressing 18 parameters of the model in terms of the other ones.

In order to generate fermions masses consistent with experiments we introduced in (20) the VEV pattern implying new relations between VEVs. Let us show that this pattern is consistent with the scalar potential (B2). It suffices to consider the simplified case of the scalar potential in the decoupling limit, when the quartic couplings of the scalar $SU(3)_L$-triplets vanish, with the exception of two $SU (3) _L$-triples. We will comment on more general cases later. The minimization conditions for the neutral scalars with real vev’s take in the decoupling limit the form

$$\begin{aligned} S^0=\chi ^0_3\rightarrow \mu _{\chi }^2&= -\frac{\lambda _{\chi \eta } v_{\eta }^2}{2}-\lambda _{\chi } v_{\chi }^2,\nonumber \\ S^0=\eta ^0_1\rightarrow \mu _{\eta }^2&= -\frac{\lambda _{\chi \eta } v_{\chi }^2}{2}-\lambda _{\eta } v_{\eta }^2,\nonumber \\ S^0=\sigma ^0\rightarrow \mu _{\sigma }^2&= - v_{\sigma }^2 \lambda _{\sigma } -2 \mu '^{2}_{\sigma } -\frac{3 f_{\sigma } v_{\sigma }}{\sqrt{2}} -\frac{\sqrt{2} \left( f_{\zeta } v_{\zeta }^2+f_{\xi } v_{\xi }^2+f_{\rho } v_{\rho }^2\right) }{v_{\sigma }} \nonumber \\ {}&\quad -\frac{\sqrt{2} f_{\phi } v_{\phi }^2 e^{i\psi }s_{2\alpha }}{ v_{\sigma }} +\frac{\sqrt{2} f_{\varphi } v_{\varphi }^2 e^{-i\psi }s_{2\alpha }}{ v_{\sigma }} ,\nonumber \\ S^0=\xi _1^0,\xi _2^0,\xi _3^0\rightarrow \mu _{\xi }^2&= -\sqrt{2} f_{\xi } v_{\sigma } -\frac{2}{3} v_{\xi }^2 (3 \lambda _1^{\xi }+4 \lambda _4^{\xi }), \nonumber \\ \mu _{1'\xi }^2&= 0, \nonumber \\ S^0=\zeta _1^0,\zeta _3^0\rightarrow \mu _{\zeta }^2&= -\sqrt{2} f_{\zeta } v_{\sigma } -v_\zeta ^2 (2 \lambda _1^{\zeta }+\lambda _3^{\zeta }+2 \lambda _4^{\zeta }),\nonumber \\ \mu _{1'\zeta }^2&= -\frac{v_{\zeta }^2 (2 w+1) (\lambda _2^{\zeta }-\lambda _3^{\zeta })}{2 (w+2)}, \nonumber \\ S^0=\rho _1^0,\rho _2^0, \rho _3^0\rightarrow \mu _{\rho }^2&= -\frac{2}{3} v_\rho ^2 (3 \lambda _1^{\rho }+4 \lambda _4^{\rho }), \nonumber \\ \mu _{\phi \rho }^{2}&= 0,\; \mu _{1'\rho }^2= 0, \end{aligned}$$

(B7)

where we have used that $\sum _{i=1}^3\langle \phi _i\rangle ^2=v_{\phi }^2 e^{i\psi }s_{2\alpha }$ and $\sum _{i=1}^3\langle \varphi _i\rangle ^2= -v_{\varphi }^2 e^{-i\psi }s_{2\alpha }$.

Next, we consider the $A_4$-triplets $\phi$ and $\varphi$ with complex VEVs given in Eq. (16). With $\mu _{\phi \rho }=0$, we have three different minimization equations for $\phi$ in the following forms:

$$\begin{aligned} S^0=\phi ^0_1\rightarrow 0=&\frac{3 x_1^2 \mu _\phi ^2}{2 v_\phi ^2}+\lambda _1^\phi \left( x_1^2+x_2^2+x_3^2\right) +\lambda _2^\phi \left( 2 x_1^2-x_2^2-x_3^2\right) \nonumber \\&\quad +4 x_1^2 \lambda _4^\phi +\frac{3 \mu _{1'\phi }^2}{v_{\phi }^2} +\frac{3 f_{\phi } v_{\sigma }}{\sqrt{2} v_{\phi }^2},\nonumber \\ S^0=\phi ^0_2\rightarrow 0=&\frac{3 x_2^2 \mu _\phi ^2}{2 v_\phi ^2}+\lambda _1^\phi \left( x_1^2+x_2^2+x_3^2\right) +\lambda _2^\phi \left( -x_1^2+2 x_2^2-x_3^2\right) +4 x_2^2 \lambda _4^\phi +\frac{3w \mu _{1'\phi }^2}{v_{\phi }^2}\nonumber \\&\quad +\frac{3 f_{\phi } v_{\sigma }}{\sqrt{2} v_{\phi }^2},\nonumber \\ S^0=\phi ^0_3\rightarrow 0=&\frac{3 x_3^2 \mu _\phi ^2}{2 v_\phi ^2}+\lambda _1^\phi \left( x_1^2+x_2^2+x_3^2\right) +\lambda _2^\phi \left( -x_1^2-x_2^2+2 x_32\right) +4 x_3^2 \lambda _4^\phi \nonumber \\&\quad +\frac{3w^2 \mu _{1'\phi }^2}{v_{\phi }^2} +\frac{3 f_{\phi } v_{\sigma }}{\sqrt{2} v_{\phi }^2}, \end{aligned}$$

(B8)

where $x_1=c_\alpha +e^{i\psi }s_{\alpha }$, $x_2=w \left( c_\alpha +we^{i\psi }s_\alpha \right)$, and $x_3=w^2 \left( c_\alpha +w^2e^{i\psi }s_\alpha \right)$ that satisfy $x_1+x_2+x_3=0$. Other relations used in our calculation are: $\sum _{i=1}^3 x_i^2= 6c_{\alpha } s_{\alpha } e^{i\psi }$. From the scalar potential minimization equations we find:

$$\begin{aligned} \mu _\phi ^2&= -\frac{2}{3} v_\phi ^2 (3 \lambda _1^\phi +4 \lambda _4^\phi ) -\frac{3 \sqrt{2} f_{\phi } v_{\sigma }}{x_1^2+x_2^2+x_3^2}, \ \mu _{1'\phi }^2=0, \nonumber \\ \lambda _2^\phi&=\lambda _1^\phi + \frac{3 f_{\phi } v_{\sigma }}{\sqrt{2} v_{\phi }^2 \left( x_1^2+x_2^2+x_3^2\right) }. \end{aligned}$$

(B9)

In the same way, we treat the minimization conditions for $\varphi$ and find the following relations

$$\begin{aligned} \mu _\varphi ^2&= -\frac{2}{3} v_\varphi ^2 (3 \lambda _1^\varphi +4 \lambda _4^\varphi ) -\frac{3 \sqrt{2} f_{\varphi } v_{\sigma }}{y_1^2+y_2^2+y_3^2}, \; \mu _{1'\varphi }^2= 0,\nonumber \\ \lambda _2^\varphi&= \lambda _1^\varphi + \frac{3 f_{\varphi } v_{\sigma }}{\sqrt{2} v_{\varphi }^2 \left( y_1^2+y_2^2+y_3^2\right) }, \end{aligned}$$

(B10)

where $y_1=c_{\alpha }-e^{-i\psi } s_{\alpha }$, $y_2=w^2\left( c_{\alpha }-w^2e^{-i\psi } s_{\alpha }\right)$, and $y_3=w\left( c_{\alpha }-we^{-i\psi } s_{\alpha }\right)$.

Thus, we see that the minimization conditions in the decoupling limit do not constrain the vev’s. This conclusion is valid in the general case, when all the quartic coupling return back to the scalar potential. This is trivial because these couplings just introduce new independent parameters, which cannot introduce any constraint on the vev’s.

Let us identify the SM-like Higgs boson with one of the scalars of our model or their linear combination.

Note that the neutral CP-even components of the Higgs bosons always contain only one massless state absorbed by the gauge boson $X^0$. This state is one of the linear combinations of the two real components $R(\chi ^0_1)$ and $R(\eta ^0_3)$, which have zero VEVs. More precisely, the model contains two would-be Goldstone bosons $G_X, G^*_X$, a neutral CP-odd Higgs boson $h_a$, and a mass eigenstate $h^0_3$. Namely, defining

$$\begin{aligned} t_\theta =\tan \theta =\frac{v_\eta }{v_\chi }, \end{aligned}$$

we have the following relations between the original and the mass eigenstates of the neutral Higgs bosons

$$\begin{aligned} \begin{pmatrix} R_{\chi _1} \\ R_{\eta _3} \end{pmatrix}&= \begin{pmatrix} c_\theta &{} s_\theta \\ -s_\theta &{} c_\theta \end{pmatrix} \begin{pmatrix} G_X \\ h^0_3 \end{pmatrix},\; \begin{pmatrix} I_{\chi _1} \\ I_{\eta _3} \end{pmatrix} = \begin{pmatrix} c_\theta &{} s_\theta \\ -s_\theta &{} c_\theta \end{pmatrix} \begin{pmatrix} {\overline{G}}_X \\ h_a \end{pmatrix} , \nonumber \\ m_{G_X}&=m_{{\overline{G}}_X}=0,\quad m^2_{h^0_3}= m^2_{h_a}=\frac{1}{2} \lambda ^{\prime }_{\eta \chi }(v^2_\eta +v^2_\chi ). \end{aligned}$$

(B11)

The $R_{\zeta _2}$ is one mass eigenstate with mass $m^2_{R_{\zeta _2}}=\frac{2 }{9}( -3\lambda ) v_\zeta ^2$.

The remaining CP-even components of the neutral Higgs boson consist of 17 states $\xi _\chi = \sqrt{2}R_{\chi ^0_3}$, $\xi _\eta =\sqrt{2}R_{\eta ^0_1}$, $R_{\sigma }$, $R_{\xi _i}$ ($i=1,2,3$), $R_{\zeta _1}$, and $R_{\zeta _3}$. The squared mass matrix of these states is the $17\times 17$ matrix denoted as ${\mathcal {M}}^2_h$. This matrix has nonzero determinant, which means that all the neutral CP-even Higgs bosons are massive. In addition, $\mathrm {Det}[{\mathcal {M}}^2_h]_{v_\eta =0}=0$ implies that there is at least one Higgs boson with mass at the electroweak scale. That lightest CP even scalar state is identified with the SM-like 126 GeV Higgs boson.

To illustrate that there is one Higgs that can be identified with the 126 GeV SM-like Higgs boson found by LHC, we consider the simplified case when the $SU(3)_L$ triplets $\chi$ and $\eta$ decouple from $S=\sigma$, $\xi _i$, $\zeta _1,\; \zeta _3$, $\rho _i,\; \varphi _i$, and $\phi _i$ so that the corresponding quartic couplings vanish $\lambda _{\eta S}=\lambda _{\chi S}=0$. Then, the matrix ${\mathcal {M}}^2_h$ is split into two block-diagonal $2\times 2$ and $15\times 15$ matrices. The first matrix in the basis $( \xi _\eta ,\;\xi _\chi )$ takes form

$$\begin{aligned} {\mathcal {M}}^2_{h1}= \begin{pmatrix} 2\lambda _\eta v_\eta ^2 &{} \lambda _{\eta \chi } v_\chi v_\eta \\ \lambda _{\eta \chi } v_\chi v_\eta &{} 2 \lambda _\chi v_\chi ^2 \end{pmatrix} . \end{aligned}$$

(B12)

Its mass eigenstates, $h^0_1$ and $h^0_2$, and their masses are

$$\begin{aligned} m^2_{h^0_{1,2}}&= \lambda _\chi v_\chi ^2 + \lambda _\eta v_\eta ^2 \mp \sqrt{\left( \lambda _\chi v_\chi ^2 -\lambda _\eta v_\eta ^2 \right) ^2 +\lambda _{\eta \chi }^2 v_\chi ^2 v_\eta ^2}, \nonumber \\ \begin{pmatrix} \xi _\eta \\ \xi _\chi \end{pmatrix}&= \begin{pmatrix} c_{\alpha } &{} s_{\alpha } \\ -s_{\alpha } &{} c_{\alpha } \end{pmatrix} \begin{pmatrix} h^0_1 \\ h^0_2 \end{pmatrix} ,\quad t_{2\alpha }\equiv \tan (2\alpha )=\frac{\lambda _{\eta \chi }t_\theta }{ \lambda _\chi -\lambda _\eta t_\theta ^2}. \end{aligned}$$

(B13)

These two neutral Higgs bosons are similar in many respects to those discussed in the model [41]. Analogously to this model, in our case in the limit $t_\theta \ll 1$, we find that $m^2_{h^0_1}\simeq \left( 2\lambda _\eta -\frac{\lambda _{\eta \chi }^2}{\lambda _\chi }\right) v^2_\eta$, as should be for the SM Higgs boson, the mass of which is generated on the electroweak scale. Thus, we identify $h^{0}_{1}$ with the SM-like Higgs boson found by the LHC. The simplified case when $t_\theta \ll 1$ is used in our discussion of the LFV Higgs decays in Sec. 7.

The soft breaking terms introduced in the Higgs potential (B3) are enough to generate nonzero masses for all CP-odd Higgs bosons in the model under consideration, even some of them vanish by the minimization conditions of the Higgs potential. Namely, in the limit of $f_S=0$ with all $S=\sigma ,\xi ,\; \zeta , \rho , \; \varphi ,\; \phi$, the total squared mass matrix of the CP-odd neutral components in the basis $S_0=(I_{\sigma },\;I_{\xi _i},\; I_{\zeta _i}, I_{\rho ,_i} \; I_{\phi _i},\; I_{\varphi _i} )$ separates into the six block sub-matrices, including one physical state $I_{\sigma }$ and another five $3\times 3$ matrices.

$$\begin{aligned} m^2_{a_\sigma }&=-4\mu '^{2} _{\sigma },\; \nonumber \\ m^2_{\xi _i}&=\left\{ \frac{1}{3} v_{\xi }^2 (3 \lambda _1^{\xi }+4 \lambda _4^{\xi }),\frac{1}{3} v_{\xi }^2 (-15 \lambda _1^{\xi }+18 \lambda _2^{\xi }+4 \lambda _4^{\xi }),\frac{1}{3} v_{\xi }^2 (-15 \lambda _1^{\xi }+18 \lambda _2^{\xi }+4 \lambda _4^{\xi })\right\} , \nonumber \\ M^2_{I_\zeta }&=\left( \begin{array}{ccc} \frac{-3 \lambda _3^{\zeta } w-4 \lambda _1^{\zeta } (w+2)+\lambda _2^{\zeta } (7 w+8)}{2 (w+2)} &{} 0 &{} 2 \lambda _1^{\zeta }-\lambda _2^{\zeta }-\lambda _3^{\zeta } \\ 0 &{} \frac{3 \lambda _2^{\zeta }+\lambda _3^{\zeta }+8 \lambda _4^{\zeta }+2 \lambda _3^{\zeta } w+4 \lambda _4^{\zeta } w-8 \lambda _1^{\zeta } (w+2)}{2 (w+2)} &{} 0 \\ 2 \lambda _1^{\zeta }-\lambda _2^{\zeta }-\lambda _3^{\zeta } &{} 0 &{} \frac{3 \lambda _3^{\zeta } (w+1)-4 \lambda _1^{\zeta } (w+2)+\lambda _2^{\zeta } (w+5)}{2 (w+2)} \\ \end{array} \right) {v_{\zeta }^2}, \nonumber \\ m^2_{a_{\rho _i}}&= \frac{v_{\rho ^2}}{3}\times \{3 \lambda _1^{\rho }+4 \lambda _4^{\rho },-15 \lambda _1^{\rho }+18 \lambda _2^{\rho }+4 \lambda _4^{\rho },-15 \lambda _1^{\rho }+18 \lambda _2^{\rho }+4 \lambda _4^{\rho }\}, \nonumber \\ \frac{M^2_{I_{\phi }}}{ v^2_{\phi }}&=\mathrm {diag}\left\{ 2 \lambda _4^{\phi } \left( x_1^2+ x_2^2-\frac{4}{3}\right) +\left( 3 x_3^2-2\right) \lambda _1^{\phi },\; 2 \lambda _4^{\phi } \left( x_1^2+ x_3^2 -\frac{4}{3}\right) \right. \nonumber \\&\quad \left. +\left( 3 x_2^2-2\right) \lambda _1^{\phi },\left( 3 x_1^2-2\right) \lambda _1^{\phi }+ 2\lambda _4^{\phi } \left( x_2^2+ x_3^2-\frac{4}{3}\right) \right\} , \nonumber \\ \frac{M^2_{I_{\varphi }}}{v^2_{\varphi }}&=\mathrm {diag}\left\{ 2 \lambda _4^{\varphi } \left( y_1^2+ y_2^2 -\frac{4}{3}\right) +\left( 3 y_3^2-2\right) \lambda _1^{\varphi },2 \lambda _4^{\varphi } \left( 3 y_1^2+3 y_3^2-\frac{4}{3}\right) \right. \nonumber \\&\quad \left. +\left( 3 y_2^2-2\right) \lambda _1^{\varphi },\left( 3 y_1^2-2\right) \lambda _1^{\varphi } +2 \lambda _4^{\varphi } \left( y_2^2+ y_3^2 -\frac{4}{3}\right) \right\} , \end{aligned}$$

(B14)

where $m^2_{a_{S_i}}$ denotes the squared mass eigenstate of the CP-odd Higgs boson corresponding to the original basis $\{I_{S_i}\}$. It can be seen that the CP-odd Higgs boson masses get contributions from the discrete symmetry preserving terms. Other trilinear soft-breaking terms with $f_S\ne 0$ will yield complicated mixings among these Higgs bosons, without affecting the phenomenology of our model, since this scalar sector, being very heavy, is decoupled from the SM fields. Notice that since we are considering a CP conserving scalar potential, the heavy neutral CP odd scalars do not mix with the CP even electrically neutral component of the $SU(3)_L$ scalar triplet $\eta$. On the other hand, the heavy physical scalar states arising from the gauge singlet scalars are mainly decoupled from the 126 GeV SM-like Higgs boson due to the very small mixings between the scalar singlets and the CP even electrically neutral component of $\eta$. Consequently, we are in the decoupling scenario where the coupling strengths of the 126 GeV SM-like Higgs boson with SM particle are very close to the SM expectation. In view of the above, setting $f_S\ne 0$ will not affect the main physics results of this paper. One can also think about introduction of an additional ad hoc symmetry forbidding the trilinear terms in (B3) and thus guarantying $f_{S}=0$. The study of this possibility goes beyond the scope of this paper and is deferred for a future work.

Appendix C: analytic formulas of LFVHD at the one-loop level

One-loop contributions to LFVHD defined in Eq. (70) are written in terms of Passarino–Veltman (PV) functions [183]. In this work, they are denoted as $B^{(i)}_{0,1},\,B^{(12)}_0,\, C_0$ and $C_{1,2}$. In the limit $m_{i,j}\simeq 0$, their analytic formulas were given in Refs. [32, 161, 184]. These functions are used for our numerical analysis. It has been shown numerically that they are in a good agreement with the exact results computed by LoopTools [185] in Ref. [186].

The analytic expressions of $\Delta ^{(i)W}_{L,R}\equiv \Delta ^{(i)W}_{(ij)L,R}$ given in Eq. (70), where i implies the diagram (i) in Fig. 6, are

$$\begin{aligned} \Delta ^{(1)W}_L&= -\frac{g^3c_{\alpha } m_{j}}{64\pi ^2 m_W^3} \sum _{k=1}^{9}\sum _{a,b=1}^3 (U_\nu )_{ak}(U^*_\nu )_{bk} (U_{\ell L})_{ja}(U^*_{\ell L})_{ib} \nonumber \\&\quad \times \left\{ m_{n_k}^2\left( B^{(1)}_1- B^{(1)}_0- B^{(2)}_0\right) -m_i^2 B^{(2)}_1 +\left( 2m_W^2+m^2_{h^0_1}\right) m_{n_k}^2 C_0 \right. \nonumber \\&\quad - \left[ 2m_W^2\left( 2m_W^2+m_{n_k}^2+m_j^2-m_i^2\right) + m_{n_k}^2m_{h^0_1}^2\right] C_1\nonumber \\&\quad \left. + \left[ 2m_W^2\left( m_j^2-m^2_{h^0_1} \right) + m_i^2 m^2_{h^0_1}\right] C_2\frac{}{}\right\} , \nonumber \\ \Delta ^{(1)W}_R&= -\frac{g^3c_{\alpha } m_i}{64\pi ^2 m_W^3} \sum _{k=1}^{9}\sum _{a,b=1}^3 (U_\nu )_{ak}(U^*_\nu )_{bk} (U_{\ell L})_{ja}(U^*_{\ell L})_{ib} \nonumber \\&\quad \times \left\{ -m_{n_k}^2\left( B^{(2)}_1+B^{(1)}_0+ B^{(2)}_0\right) +m_j^2 B^{(1)}_1 +\left( 2m_W^2+m^2_{h^0_1}\right) m_{n_k}^2 C_0 \right. \nonumber \\&\quad - \left[ 2m_W^2\left( m_i^2-m^2_{h}\right) + m_j^2 m^2_{h^0_1}\right] C_1\nonumber \\&\quad \left. + \left[ 2m_W^2\left( 2m_W^2+m_{n_k}^2-m_j^2+m_i^2\right) + m_{n_k}^2m_{h^0_1}^2\right] C_2 \frac{}{}\right\} , \nonumber \\ \Delta ^{(2)W}_L&= -\frac{g^3c_{\alpha } m_j}{64\pi ^2 m_W^3} \sum _{k,p=1}^{9} \sum _{a,b=1}^3 (U_\nu )_{ak}(U^*_\nu )_{bp} (U_{\ell L})_{ja}(U^*_{\ell L})_{ib} \nonumber \\&\quad \times \left\{ \lambda ^{0*}_{kp}m_{n_p}\left[ B^{(12)}_0-m_W^2C_0+\left( 2 m_W^2+m_{n_k}^2-m_j^2\right) C_1\right] \right. \nonumber \\&\quad \left. +\lambda ^{0}_{kp}m_{n_k}\left[ B^{(1)}_1+\left( 2 m_W^2+m_{n_p}^2-m_i^2\right) C_1\right] \right\} , \nonumber \\ \Delta ^{(2)W}_R&=-\frac{g^3c_{\alpha } m_i}{64\pi ^2 m_W^3}\sum _{k,p=1}^{9} \sum _{a,b=1}^3 (U_\nu )_{ak}(U^*_\nu )_{bp} (U_{\ell L})_{ja}(U^*_{\ell L})_{ib} \nonumber \\&\quad \times \left\{ \lambda _{kp}m_{n_k}\left[ B^{(12)}_0-m_W^2C_0-\left( 2 m_W^2+m_{n_p}^2-m_i^2\right) C_2\right] \right. \nonumber \\&\quad -\left. \lambda ^{0*}_{kp}m_{n_p}\left[ B^{(2)}_1+\left( 2 m_W^2+m_{n_k}^2-m_j^2\right) C_2\right] \right\} , \nonumber \\ \Delta ^{(3+4)W}_L&=- \frac{g^3m_jm_i^2(c_{\alpha } +s_\alpha t_\theta )}{64 \pi ^2m^3_W(m_j^2-m_i^2)} \sum _{k=1}^{9}\sum _{a,b=1}^3 (U_\nu )_{ak}(U^*_{ \nu })_{bk} (U_{\ell L})_{ja}(U^*_{\ell L})_{ib} \nonumber \\&\quad \times \left[ 2m_{n_k}^2\left( B^{(1)}_0-B^{(2)}_0\right) - \left( 2 m_W^2 +m_{n_k}^2\right) \left( B^{(1)}_1 +B^{(2)}_1 \right) - m_j^2 B^{(1)}_1 -m_i^2 B^{(1)}_2 \right] , \nonumber \\ \Delta ^{(3+4)W}_R&=\frac{m_j}{m_i}\Delta ^{(3+4)W}_L \nonumber \\ \Delta ^{(1)Y}_L&= -\frac{g^3s_{\alpha } m_{j}}{64\pi ^2 m_Y^3} \sum _{k=1}^{9}\sum _{a,b=1}^3 (U_\nu )_{(a+3)k}(U^*_\nu )_{(b+3)k} (U_{\ell L})_{ja}(U^*_{\ell L})_{ib} \nonumber \\&\quad \times \left\{ m_{n_k}^2\left( B^{(1)}_1- B^{(1)}_0- B^{(2)}_0\right) -m_i^2 B^{(2)}_1 +\left( 2m_Y^2+m^2_{h^0_1}\right) m_{n_k}^2 C_0 \right. \nonumber \\&\quad - \left[ 2m_Y^2\left( 2m_Y^2+m_{n_k}^2+m_j^2-m_i^2\right) + m_{n_k}^2m_{h^0_1}^2\right] C_1\nonumber \\&\quad \left. + \left[ 2m_Y^2\left( m_j^2-m^2_{h^0_1} \right) + m_i^2 m^2_{h^0_1}\right] C_2\frac{}{}\right\} , \nonumber \\ \Delta ^{(1)Y}_R&= -\frac{g^3c_{\alpha } m_i}{64\pi ^2 m_Y^3} \sum _{k=1}^{9}\sum _{a,b=1}^3 (U_\nu )_{(a+3)k}(U^*_\nu )_{(b+3)k} (U_{\ell L})_{ja}(U^*_{\ell L})_{ib} \nonumber \\&\quad \times \left\{ -m_{n_k}^2\left( B^{(2)}_1+B^{(1)}_0+ B^{(2)}_0\right) +m_j^2 B^{(1)}_1 +\left( 2m_Y^2+m^2_{h^0_1}\right) m_{n_k}^2 C_0 \right. \nonumber \\&\quad - \left[ 2m_Y^2\left( m_i^2-m^2_{h}\right) + m_j^2 m^2_{h^0_1}\right] C_1\nonumber \\&\quad \left. + \left[ 2m_Y^2\left( 2m_Y^2+m_{n_k}^2-m_j^2+m_i^2\right) + m_{n_k}^2m_{h^0_1}^2\right] C_2 \frac{}{}\right\} , \nonumber \\ \Delta ^{(2)Y}_L&= -\frac{g^3c_{\alpha } m_j}{64\pi ^2 m_Y^3} \sum _{k,p=1}^{9} \sum _{a,b=1}^3 (U_\nu )_{(a+3)k}(U^*_\nu )_{(b+3)p} (U_{\ell L})_{ja}(U^*_{\ell L})_{ib} \nonumber \\&\quad \times \left\{ \lambda ^{0*}_{kp}m_{n_p}\left[ B^{(12)}_0-m_Y^2C_0+\left( 2 m_Y^2+m_{n_k}^2-m_j^2\right) C_1\right] \right. \nonumber \\&\quad \left. +\lambda ^{0}_{kp}m_{n_k}\left[ B^{(1)}_1+\left( 2 m_Y^2+m_{n_p}^2-m_b^2\right) C_1\right] \right\} , \nonumber \\ \Delta ^{(2)Y}_R&=-\frac{g^3c_\alpha m_i}{64\pi ^2 m_Y^3}\sum _{k,p=1}^{9} \sum _{a,b=1}^3 (U_\nu )_{(a+3)k}(U^*_\nu )_{(b+3)p} (U_{\ell L})_{ja}(U^*_{\ell L})_{ib} \nonumber \\&\quad \times \left\{ \lambda ^{0}_{kp}m_{n_k}\left[ B^{(12)}_0-m_Y^2C_0-\left( 2 m_Y^2+m_{n_p}^2-m_i^2\right) C_2\right] \right. \nonumber \\&\quad -\left. \lambda ^{0*}_{kp}m_{n_p}\left[ B^{(2)}_1+\left( 2 m_Y^2+m_{n_k}^2-m_j^2\right) C_2\right] \right\} , \nonumber \\ \Delta ^{(3+4)Y}_L&=- \frac{g^3m_jm_i^2(c_\alpha +s_\alpha t_\theta )}{64 \pi ^2m^3_Y(m_j^2-m_i^2)} \sum _{k=1}^{9}\sum _{a,b=1}^3 (U_{ \nu })_{(a+3)k}(U^*_\nu )_{(b+3)k} (U_{\ell L})_{ja}(U^*_{\ell L})_{ib} \nonumber \\&\quad \times \left[ 2m_{n_k}^2\left( B^{(1)}_0-B^{(2)}_0\right) - \left( 2 m_Y^2 +m_{n_k}^2\right) \left( B^{(1)}_1 +B^{(2)}_1 \right) - m_j^2 B^{(1)}_1 -m_i^2 B^{(1)}_2 \right] , \nonumber \\ \Delta ^{(3+4)Y}_R&=\frac{m_j}{m_i}\Delta ^{(3+4)Y}_L. \end{aligned}$$

(C1)

Appendix D: couplings of the Z and $Z^{\prime }$ gauge bosons to fermions

The interactions between fermions and neutral gauge bosons are determined as

$$\begin{aligned} {\mathcal {L}}_{\mathrm {ngaugefermion}} = g {\overline{f}} \gamma ^\mu P_\mu ^{NC} f \, , \end{aligned}$$

(D1)

where f denotes all fermions in the model under consideration. Then one gets

• Electromagnetic interaction, as usual: ${\mathcal {L}}_{em} = e {\overline{f}} \gamma ^\mu Q f A_\mu$ .

• Interaction between Z with fermion

  $$\begin{aligned} {\mathcal {L}}_{Z f}&= \frac{g}{c_W}{\overline{f}} \gamma ^\mu \left[ c_\phi \left( T_3 -s^2_W Q\right) - s_\phi \left( \frac{\sqrt{3-4s^2_W}}{\sqrt{3}} T_8 +\frac{ s^2_W }{\sqrt{3-4s^2_W}} X\right) \right] f Z_\mu \nonumber \\&\equiv \frac{g}{c_W}{\overline{f}}_{L,R} \gamma ^\mu g_{L,R} f_{L,R} Z_\mu \, , \end{aligned}$$

  (D2)

  where $\phi$ is the $Z-Z'$ mixing angle given in Ref. [131], $s_\phi \equiv \sin \phi$, $c_\phi \equiv \cos \phi$,

  $$\begin{aligned} \tan \phi \simeq s_\phi \simeq \frac{(1-2s_W^2)\sqrt{3-4s_W^2}}{4 c_W^4} \left( \frac{v_{\eta }^2}{v^2_{\chi }}\right) ,\; M^2_{Z'}\simeq \frac{g^2c_W^2}{4(3-4s_W^2)} \left[ 4v^2_{\chi } +\frac{v^2_{\eta }(1-2s^2_W)^2}{c_W^4} \right] . \end{aligned}$$

  (D3)

  The couplings of the Z gauge boson with fermion are presented in Table 6, ignoring mixing of SM and exotic quarks.

Table 6 Couplings between Z boson and fermions

It can be seen that $s_\phi \rightarrow 0$ when $m_Z^2/M^2_{Z'}\rightarrow 0$, leading to the consequence that $g_L\simeq g_R$ for the exotic quarks $T,J_{1,2}$, as given in Table 6. Note that in the limit $\phi \rightarrow 0$, the couplings of Z to the SM fermions are the same as those of the SM Z boson.

• Interaction between $Z^\prime$ with fermion

  $$\begin{aligned} {\mathcal {L}}_{Z^\prime f}&= \frac{g}{c_W}{\overline{f}} \gamma ^\mu \left[ c_\phi \left( \frac{\sqrt{3-4s^2_W}}{\sqrt{3}} T_8 + \frac{ s^2_W }{\sqrt{3-4s^2_W}} X\right) + s_\phi \left( T_3 -s^2_W Q\right) \right] f Z^\prime _\mu \nonumber \\&\equiv \frac{g}{c_W}{\overline{f}}_{L,R} \gamma ^\mu g^{\prime }_{L,R} f_{L,R} Z^\prime _\mu , \end{aligned}$$

  (D4)

  It is worth noting that couplings of Z and $Z^\prime$ are related to each other by replacing $c_\phi \leftrightarrow s_\phi$. The couplings of the $Z_\prime$ gauge boson with fermion (by replacing $c_\phi \rightarrow s_\phi$ and $s_\phi \rightarrow -c_\phi$) are presented in Table 7 .

Table 7 Couplings between $Z^\prime$ boson and fermions

Note that in both Tables, dealing with neutrino we used $\nu ^c_L \sim \nu _R$.

For practical uses, we present neutral currents in the vector and axial forms as follows

$$\begin{aligned} {\mathcal {L}}_{Z f}= & {} \frac{g}{2c_W}{\overline{f}} \gamma ^\mu ( g_V - \gamma _5 g_A) f Z_\mu \, , \end{aligned}$$

(D5)

$$\begin{aligned} {\mathcal {L}}_{Z^\prime f}= & {} \frac{g}{2c_W}{\overline{f}} \gamma ^\mu ( g^\prime _V - \gamma _5 g^\prime _A) f Z^\prime _\mu \, , \end{aligned}$$

(D6)

where the relation among two kinds of couplings is given by

$$\begin{aligned} g_V = g_L + g_R \, , \quad g_A = g_L - g_R. \end{aligned}$$

(D7)