1 Introduction

A fundamental question regarding the composition of matter at densities several times the nuclear saturation density (\(\rho _0\)) is how it differs from the state that makes up atomic nuclei. It is widely known that several exotic states are potential candidates for matter at ultra-high densities: the creation of hyperons, the onset of Bose–Einstein condensates, and transformation to quark matter. Using subatomic physics theories, one can estimate the conditions under which the new state of matter will emerge. However, this new state of matter has not yet been established. For example, the appearance of hyperons and \(\Delta\) baryons in matter and how they interact with other constituents of matter are still open questions.

In a recent study by Ref. [1], researchers investigated the in-medium interaction of the \(\Lambda\) hyperon using non-relativistic nuclear density functional theory. They determined parameters for the \(\Lambda\) hyperon interaction to match single-\(\Lambda\) hypernuclear data and examined the effect of the density dependence of the symmetry energy in more detail. The study revealed that the density at which \(\Lambda\) hyperons appear strongly depends on the symmetry energy. However, it was found that the density at which hyperons appear is always greater than \(3\rho _0\) (\(\rho _0\): nuclear saturation density), suggesting that hyperons are unlikely to significantly impact the properties of a canonical neutron star with a mass of \(1.4M_\odot\), as the density at the center of such a neutron star is generally less than \(3\rho _0\). The study also considered kaon condensation in both relativistic and non-relativistic models [2, 3]. It was concluded that unless the interaction of the kaon in dense nuclear matter is super-strongly attractive, the formation of kaon condensation is less probable.

Similarly, the matter composition in neutron stars (NSs) can be illustrated as the quarks in the baryons on a microscopic scale. According to the bag model, baryons are spherical bags in which quarks are assumed to be confined. In the bag model, the proton’s charge radius is expected to be around 0.6 to 0.8 fm, corresponding to a volume of 0.9 to 2.1 fm\(^3\). The inverse of the number density (\(\rho\)) gives the volume occupied by a nucleon. At the saturation density, if \(\rho _0 = 0.16\, \textrm{fm}^{-3}\), one nucleon occupies 6.25 fm\(^3\), which is larger than the volume of a nucleon, indicating that nucleons are spatially separated. At densities in the range of three-to-six times \(\rho _0\), the volume per nucleon becomes equal to the volume of a nucleon, causing the nucleons to begin to overlap. As the matter density increases above the onset of overlapping, the nucleons overlap more, making it difficult to define the confinement of a quark in a specific bag. The matter then transforms into the phase of deconfined quarks. A similar condition is expected to occur in neutron stars.

The equation of state (EoS) of dense matter allows for the calculation of the bulk properties of neutron stars, such as mass, radius, and particle distribution in the core, by solving the Tolman–Oppenheimer–Volkoff (TOV) equations. Using nuclear models constrained by nuclear properties, nuclear-matter properties from heavy-ion collision experiments, ab initio calculations, and modern neutron star observation data [4,5,6,7,8,9], the density at the center of a canonical mass of NS is found to be about or less than \(3\rho _0\), and for the heaviest stars (\(\ge 2 M_\odot\)) is around \(6\rho _0\). The core of NSs with masses in the range (1.4–2.0)\(M_\odot\) may likely transform from hadron to quark degrees of freedom.

The main focus of this study is to investigate the uncertainty surrounding the critical density (\(\rho _c\)) at which the transition from the hadronic phase to the deconfined quark matter begins and its impact on the properties of neutron stars. To achieve this goal, we utilize the KIDS (Korea-IBS-Daegu-SKKU) energy-density functional (EDF) for the equation of state of hadronic matter (HM), with the density dependence of the symmetry energy calibrated to the radius of \(1.4M_\odot\) mass NSs in the range \(R_{1.4} = (11.8-12.5)\) km [8]. The KIDS0, KIDS-A, KIDS-B, KIDS-C, and KIDS-D models and the standard Skyrme SLy4 model are employed, each with distinct density dependence of the symmetry energy. The results obtained will provide a range of \(\rho _c\) within the uncertainty of the nuclear-matter EoS.

On the quark matter (QM) side, the proper-time NJL model is used for the EoS of the deconfined quark state. Considering that massive NSs are likely to have a QM phase in the core, we investigate the range of parameters in the NJL model that are compatible with NS masses greater than \(2M_\odot\). This allows us to determine the ranges of the parameters in the QM EoS.

By combining the EoSs of the KIDS model with those of the NJL model, we determine the range of \(\rho _c\) from the condition \(P_\textrm{HM}(\rho _c) = P_{\textrm{QM}}(\rho _c)\) at \(\rho _{\textrm{HM}} = \rho _{\textrm{QM}} = \rho _c\), where \(P_{\textrm{HM}}\), \(P_{\textrm{QM}}\), \(\rho _{\textrm{HM}}\), and \(\rho _{\textrm{QM}}\) represent the pressure in the hadronic phase, pressure in quark phase, density in hadronic phase, and density in quark phase, respectively. The results will provide insights into (i) the effect of the symmetry energy on the change of phase in the NS core, (ii) the consistency of the hybrid models with the NS properties determined from astronomical observations, and (iii) the uncertainty of the EoS in the hadronic and quark phases.

This work is organized as follows. In Sect. 2, we briefly introduce models for both hadronic matter and deconfined quark matter. The KIDS-EDF model is used to describe the hadronic matter. In contrast, the quark matter is described using the flavor SU(2) Nambu–Jona–Lasinio model with the Schwinger proper-time regularization scheme, also known as the NJLPTR model. We present and discuss the numerical results of the current work in Sect. 3, and we summarize the work in Sect. 4.

2 Formalism for the hadron and quark phases

2.1 Hadronic matter from the KIDS functional

This section will describe HM in the KIDS-EDF model [4, 10, 11]. This model is based on the Fermi momentum expansion, relevant for infinite nuclear matter and a neutron star. The energy per particle for homogeneous hadronic matter, expressed in powers of the cubic root of the density, is given by

$$\begin{aligned} {\mathcal {E}}_{{\textrm{HM}}} (\rho ,\delta )= & {} {\mathcal {T}} (\rho ,\delta ) + \sum _{j=0}^{3} \left(\alpha _j + \beta _j \delta ^2\right) \rho ^{1+j/3}. \end{aligned}$$
(1)

The kinetic energy denoted by \({\mathcal {T}} (\rho ,\delta )\) is given as

$$\begin{aligned} {{{\mathcal {T}}}}(\rho , \delta ) = \frac{\hbar ^2}{2M}\frac{3}{5} (3\pi ^2\rho )^{2/3} \left[ \left( \frac{1+\delta }{2}\right) ^{5/3} + \left( \frac{1-\delta }{2}\right) ^{5/3} \right] . \end{aligned}$$
(2)

Here, \(\delta = (\rho _n -\rho _p)/\rho\) and \(\rho = \rho _n +\rho _p\), representing the isospin asymmetry and baryon density, respectively, and \(M\equiv (M_n + M_p)/2\) represents the average nucleon mass in free space. The symmetric and antisymmetric nuclear–matter coefficients are denoted by \(\alpha _j\) and \(\beta _j\).

The symmetry energy \(S(\rho )\) can be straightforwardly determined by taking the second derivative of the energy per particle concerning \(\delta\) at the nuclear saturation density. This yields

$$\begin{aligned} {\mathcal {E}}_{{\textrm{HM}}} (\rho ,\delta )= & {} {\mathcal {E}} (\rho ,0) + S(\rho ) \delta ^2 + {\mathcal {O}} (\delta ^4), \nonumber \\ S (\rho )= & {} \frac{\hbar }{6M} \left( \frac{3\pi ^2}{2}\right) ^{2/3} \rho ^{2/3} + \sum _{j=0}^3 \beta _j \rho ^{(1+j/3)}. \end{aligned}$$
(3)

The pressure for the HM can be calculated from the energy per particle concerning baryon density. It takes the following form:

$$\begin{aligned} P_{{\textrm{HM}}}= & {} \rho ^2 \frac{\partial {\mathcal {E}}_{{\textrm{HM}}}(\rho ,\delta )}{\partial \rho }. \end{aligned}$$
(4)

In this work, the nuclear many-body system is represented by five KIDS-EDF models: KIDS0, KIDS-A, KIDS-B, KIDS-C, and KIDS-D, along with the Skyrme force SLy4 model. These EDF models exhibit different nuclear symmetry energies, ranging from soft to stiff nuclear symmetry energies. The graph in Fig. 1a displays the results for the nuclear symmetry energy of the KIDS0, KIDS-A, KIDS-B, KIDS-C, KIDS-D, and SLy4 models. It clearly illustrates different ranges of symmetry energies, covering soft and stiff symmetry energies that will be used in constructing the EoS for the hybrid model. In Fig. 1a, it can be observed that for HM, the KIDS-A model provides the stiffest symmetry energy, while the SLy4 model offers the softest. Additionally, Fig. 1b shows the pressure of the NS matter as a function of density. The pressures of symmetric matter obtained from the EDF models are consistent with heavy-ion collision (HIC) reaction data at high density [12] as well as chiral perturbation theory (ChPT) at low density [13].

Fig. 1
figure 1

Numerical results of the NS matter from the KIDS EDFs and the SLy4 model: a the symmetry energy, b pressure, and c energy density of HM as functions of \(\rho /\rho _0\), whereas d EoS of HM is given as well

Full size image

The graph in Fig. 1c presents the energy densities \(E_{\textrm{HM}}\) of the NS matter for the hadronic models as a function of density. Additionally, Fig. 1d displays the \(P_{\textrm{HM}}\)-\(E_{\textrm{HM}}\) relationship for the hadronic models. This equation of state relationship is necessary for input into the TOV equation to determine the properties of the NSs. The HM model observed that the \(M_\textrm{NS}/M_{\odot }\) results for the KIDS0 and SLy4 models are quite similar due to both models having soft symmetry energies. In contrast, the KIDS-A and KIDS-B models, which have stiff symmetry energies, result in a larger radius for the NS. Overall, the mass and radius of the NS results for all KIDS models align well with recent observations [14,15,16].

2.2 Pure quark matter from the NJLPTR model

In this section, we discuss pure (nonstrange) quark matter (PQM) within the framework of the flavor SU(2) Nambu–Jona–Lasinio model [17,18,19,20,21,22,23,24,25,26]. This model is constructed using the quark degrees of freedom in the form of four-fermion contact interactions. As a result, it is well suited and powerful for describing quark matters. The general NJL Lagrangian density for two quark flavors can be expressed as

$${\mathcal{L}}_{{NJL}} = \bar{\psi }_{q} \left( {i/\partial - \hat{m}_{q} } \right)\psi _{q} + G_{s} \left[ {(\bar{\psi }_{q} \psi _{q} )^{2} + (\bar{\psi }_{q} \gamma _{5} \vec{\tau }\psi _{q} )^{2} } \right] - G_{\omega } (\bar{\psi }_{q} \gamma _{\mu } \psi _{q} )^{2} - G_{\rho } (\bar{\psi }_{q} \gamma _{\mu } \vec{\tau }\psi _{q} )^{2} .$$
(5)

In this model, the quark field is represented by \(\psi _q = (\psi _u,\psi _d)^T\), \(\vec{\tau }\) denotes the Pauli isospin matrices, and \({\hat{m}}_q = \text {diag}[m_u, m_d]\) is the current (bare) quark mass. Assuming isospin symmetry, we have \(m_u \approx m_d\). The constants \(G_s\), \(G_\omega\), and \(G_\rho\) represent the scalar, isoscalar–vector, and isovector–vector coupling constants, respectively. It is important to note that the NJL model encounters a divergence in the quark propagator, which requires us to apply a regularization scheme to address this issue. In our work, we have applied the Pauli–Villars regularization scheme to mitigate the divergence, as suggested by Schwinger [27]. Using the mean-field approximation (MFA) in the proper-time regularization (PTR) scheme, the effective quark mass can be readily obtained within the standard NJL formalism

$$\begin{aligned} M_q= & {} m_q - 2 G_s \langle {\bar{\psi }}_q \psi _q \rangle = m_q + \frac{3G_s M_q}{\pi ^2} \int _{\Lambda _{\textrm{UV}}^{-2}}^{\infty } d\tau \frac{\exp [-\tau M_q^2]}{\tau ^2} \nonumber \\- & {} \frac{6G_s M_q }{\pi ^2} \left[ {\hat{\mu }}_q\sqrt{{\hat{\mu }}_q^2 -M_q^2} -M_q^2 \log \left( \frac{{\hat{\mu }}_q + \sqrt{{\hat{\mu }}_q^2 -M_q^2}}{M_q}\right) \right] . \end{aligned}$$
(6)

In the equation, the term \(\langle {\bar{\psi }}_q \psi _q\rangle\) represents the chiral condensate for flavor q and is related to the order parameter of the spontaneous breaking of chiral symmetry (SB\(\chi\)S). It is important to note that the effective quark mass in Eq. (6) has an additional correction in quark matter, which arises from the density term in the quark propagator. A similar expression for the effective quark mass can be obtained from \(\partial {\mathcal {V}}_{{\textrm{NJL}}}^{{\textrm{QM}}} (M_q, \mu _q)/\partial M_q\), where \({\mathcal {V}}_{{\textrm{NJL}}}^{{\textrm{QM}}} (M_q, \mu _q)\) will be given in Eq. (10). It is worth noting that the definition of the reduced chemical potential will be provided below.

In the quark matter, the NJL Lagrangian density in Eq. (5) is modified by adding an extra quark density or quark operator chemical potential term. It then has the form

$$\begin{aligned} {\mathcal {L}}_{\textrm{NJL}}\rightarrow & {} {\mathcal {L}}_{\textrm{NJL}} + {\bar{\psi }}_q {\hat{\mu }}_q \gamma ^0 \psi _q, \end{aligned}$$
(7)

where \({\hat{\mu }}_q = \text {diag}[\mu _u, \mu _d]\) is the quark chemical potential matrix. Thus, considering the density effect, the quark propagator for flavor q in momentum space can be written by

$$\begin{aligned} S_q (p)= & {} \frac{1}{\left[ \left( p_0 + {\tilde{\mu }}_q \right) \gamma ^0 - \varvec{p} \cdot \varvec{\gamma } - M_q \right] }, \end{aligned}$$
(8)

where the \({\tilde{\mu }}_q\) is the so-called the reduced quark chemical potential for flavor q and it can be written as

$$\begin{aligned} {\tilde{\mu }}_q= & {} \mu _q - 2 G_v \rho _q^{v}, \end{aligned}$$
(9)

where the second term in the r.h.s. denotes the contribution from the vector interaction, and it reduces the chemical potential effectively, i.e., diminishing the density by the repulsion between quarks. In this work, we use \(\rho _q^{v} \simeq \langle \psi _q^{\dagger } \psi _q \rangle = p_{F_{i}}^3/\pi ^2\) to represent the individual quark number densities of flavor q. Here, \(p_{F_{i}}\) is the Fermi momentum for quark flavor \(i=(u,d)\). For simplicity, we consider the vector coupling \(G_v=G_\omega =G_\rho\) by taking into account vector dominance and the small mass differences between \(\rho\) and \(\omega\) masses [28, 29]. It is important to note that in this work, \(G_v\) will be treated as a free parameter.

In the standard method, the effective potential for the nonstrange QM in the NJL model is given by [30]

$$\begin{aligned} {\mathcal {V}}^{\text {QM}}_{\text {NJL}}(M_q,\mu _q)= & {} 2iN_c \sum _{q=u,d}\int \frac{d^4k}{(2\pi )^4} \log \left[ \frac{k^2-M_q^2 + i \epsilon }{k^2-M_{0}^2 -i \epsilon } \right] + \sum _{q=u,d} \frac{(M_q-m_q)^2}{8G_s} - \sum _{q=u,d}\frac{(M_{0}-m_q)^2}{8 G_s} \\- & {} 2 N_c \sum _{q=u,d} \int \frac{d^3 k}{(2\pi )^3} \Theta \left[ {\tilde{\mu }}_q - E_q (k)\right] \, \left[ {\tilde{\mu }}_q - E_q (k) \right] -\sum _{q=u,d}\frac{V_0^2}{8G_v}. \end{aligned}$$
(10)

The first term exhibits divergence in the given expression and requires regularization using the PTR scheme. The reduced quark chemical potentials for u and d quarks are denoted by \({\tilde{\mu }}_u = {\tilde{\mu }}_d = \mu _u - 2 G_v \rho ^v_u\), as shown in Eq. (9). Here, \(V_0 = 2 G_v \langle \psi _q^\dagger \psi _q \rangle\), \(E^2_u (k) = E^2_d (k) = |\varvec{k}|^2 + M^2_0\), and \(M_{0}\) represent the vector field (potential), quark energy, and the constituent quark mass in free space, respectively. It should be noted that a constant (free space) contribution (\(M_q=M_0\)) has been subtracted from the effective potential in Eq. (10). As a result, the pressure becomes zero in free space. With this effective potential definition and the so-called Gibbs–Duhem relation, the energy density also vanishes in free space [31].

Using the effective potential in Eq. (10), the pressure and energy density for the normal (nonstrange) QM are expressed as

$$\begin{aligned} P_{\textrm{QM}}= & {} - {\mathcal {V}}^{\text {QM}}_{\text {NJL}} (M_q,\mu _q)-{\mathcal {V}}_l (\mu _l), \nonumber \\ E_{\textrm{QM}}= & {} -P_{\textrm{QM}} + \sum _{q=u,d} \mu _q \rho ^v_q, \nonumber \\ \rho _q= & {} - \frac{\partial {\mathcal {V}}^{\text {QM}}_{\text {NJL}} \left( M_q, \mu _q\right) }{\partial \mu _q}. \end{aligned}$$
(11)

The QM EoS calculation involved fitting the NJL parameters to the pion mass \(m_\pi = 0.14\) GeV and the pion weak decay constant \(f_\pi = 0.093\) GeV. This resulted in \(G_{s} = 3.17\) GeV\(^{-2}\) and \(\Lambda _{\textrm{UV}} = 1.0789\) GeV. The constituent quark mass \(M_0\) is 0.20 GeV, and the current quark mass \(m_q = 0.0055\) GeV, similar to the PDG values [32].

The results for the effective quark mass, pressure, and energy density of the quark matter for vector interaction couplings \(G_v = (0.00, 0.25, 0.50) G_s\) are illustrated in Fig. 2 as functions of the nucleon density. Assuming that the quark content of the nucleon is three quarks \(N\sim qqq\) and isospin symmetry is intact, we have the relation between the quark density \(\rho _q\) and the nucleon density \(\rho\) as \(\rho = \rho _q/3\) and the relation for the chemical potential \(\mu = 3 \mu _q\) where \(\mu\) and \(\mu _q\) denote the nucleon and quark chemical potential, respectively. It is observed that the effective quark masses decrease as the density increases, as shown in Fig. 2a. Moreover, the effective quark mass is not significantly influenced by changes in the values of the vector repulsive interaction coupling; the rate of change is relatively low. This result for the effective quark mass is consistent with the findings in Refs. [21, 30].

Fig. 2
figure 2

From the NJLPTR model: a effective quark mass for different values of \(G_v\) as a function of \(\rho /\rho _0\), b pressure of QM as a function of \(\rho /\rho _0\), c energy density of QM as a function of \(\rho /\rho _0\), and d EoS of PQM

Full size image

The results for the pressure of the PQM are depicted in Fig. 2b. The pressure becomes more pronounced with a higher value of \(G_v\). This behavior is a natural and expected outcome, because a higher \(G_v\) value reinforces the repulsive force. Additionally, the results for the E-\(\rho\) relation of the PQM for different values of \(G_v\) are presented in Fig. 2c. A similar trend with the pressure is observed: the energy density increases as the \(G_v\) value and density increase. This relationship can be clearly understood from the equations \(P_{\textrm{QM}} =-{\mathcal {V}}^{\textrm{QM}}_{\textrm{NJL}} - {\mathcal {V}}_l\) and \(E_{\textrm{QM}} = -P_{\textrm{QM}} + \sum _q \mu _q \rho _q^\nu\), as given in Eq. (11). It implies that the increase in pressure is solely due to the increase in energy density and vice versa, as explained in the visualization of Fig. 12 of Ref. [21]. Figure 2d illustrates the EoS P-E relation for different values of \(G_v\). The pressure rises alongside the energy density and \(G_v\). This also indicates that the EoS of the QM becomes stiffer for higher values of \(G_v\).

2.3 Properties of the static hybrid star

Here, we input the obtained equations of state for hadronic and quark matters, with various values of \(G_v\) and symmetry energies, into the TOV equation. This allows us to numerically calculate the properties of a non-rotating neutron star and obtain the mass–radius (M–R) relation, which describes the structure of the hybrid star. The TOV equations can be found in Refs. [33,34,35]

$$\begin{aligned} \frac{dP(r)}{dr}= & {} - \frac{G[E(r) + P(r)] [M(r) + 4\pi r^3 P(r)]}{r[r-2GM(r)]}, \qquad \frac{dM(r)}{dr} = 4\pi r^2 E(r), \end{aligned}$$
(12)

where P(r) and E(r) are the pressure and energy density at radial position r. G and M(r) represent the gravitational constant and the mass within the sphere of radius r, respectively. Using the equations of state of the constructed hybrid stars as input, we can derive the M–R relation. The EoSs of the hybrid stars are illustrated in Fig. 3.

3 Numerical result

Fig. 3
figure 3

Pressure as a function of density for QM with different values of \(G_v\) and for HM in the different EDF models: a KIDS0, b KIDS-A, c KIDS-B, d KIDS-C, e KIDS-D, and f SLy4. The critical density \(\rho _c\) is given by the cross point between \(P_{\textrm{HM}}\) and \(P_{\textrm{QM}}\) with different values of \(G_v\). The thick solid line is P-\(\rho\) relation for HM. The thin solid line is P-\(\rho\) relation for QM with \(G_v = 0.00 \,G_s\), the dashed line is P-\(\rho\) relation for QM with \(G_v = 0.25 \,G_s\), and the dashed line is P-\(\rho\) relation for QM with \(G_v = 0.50 \,G_s\)

Full size image

3.1 Critical density of the hybrid star

In this section, we present the results of the critical densities of the phase transition from hadronic to quark matter for different KIDS models and the Skyrme model. The critical density \(\rho _c\) is determined by finding the point where \(P_{\textrm{HM}}\) and \(P_{\textrm{QM}}\) intersect. It is important to note that \(\rho _c\) is the density at which the phase transition from the hadronic to quark phases occurs when \(P_{\textrm{HM}}(\rho _c) = P_{\textrm{QM}}(\rho _c)\) and \(\rho _{\textrm{HM}} = \rho _{\textrm{QM}}\). In Fig. 3, \(P_{\textrm{HM}}\) is shown for each hadronic model and \(P_{\textrm{QM}}\) with three values of the vector coupling constant \(G_v = 0\), \(0.25\,G_s\), and \(0.5\,G_s\). Figure 3a shows \(P_{\textrm{HM}}\) for the KIDS0 model and \(P_{\textrm{QM}}\) for the NJL model with different values of \(G_v\). The critical density for \(P_{\textrm{HM}}\) and \(P_{\textrm{QM}}\) for \(G_v =0.00 \,G_s\) is around \(\rho _c =3.60 \rho _0\). The \(\rho _c\) increases as the value of \(G_v\) increases. The \(\rho _c\) for the KIDS-A model is shown in Fig. 3b. The KIDS-A model has a stiffer symmetry energy compared to the KIDS0 model. The critical densities for the KIDS-A model and the NJL model with \(G_v = 0.00\, G_s\) and \(G_v=0.25\, G_s\) are smaller than those for the KIDS0 model. However, the critical density for the KIDS-A model and the NJL model with \(G_v = 0.5 \,G_s\) is larger than that obtained for the KIDS0 model. This trend continues for the KIDS-B, KIDS-C, and KIDS-D models as shown in Fig. 3c–e. In contrast, the SLy4 and the NJL models with different values of \(G_v\) have similar critical density values to those for the KIDS0 model and the NJL model with the corresponding values of \(G_v\). This is likely due to KIDS0 and SLy4 models having softer nuclear symmetry energies than other KIDS models, as shown in Fig. 1. For all the hadronic models and the NJL model with different \(G_v\) values, comparing the pressures of HM and QM at a given density, it is smaller in the hadron phase at low densities, and it becomes smaller in the quark phase at high densities.

In comparing the values of \(\rho _c\) for larger values of \(G_v=0.5\,G_s\) among the KIDS-A, KIDS-B, KIDS-C, and KIDS-D models, we observe that the KIDS-A model has the highest value of \(\rho _c\), followed by KIDS-B, KIDS-C, and KIDS-D models in decreasing order. The ordering of \(\rho _c\) can be understood from the stiffness of symmetry energy. A soft symmetry energy (small L and \(K_{\textrm{sym}}\) values, where L and \(K_{\textrm{sym}}\) are the slope and the curvature of symmetry energies, respectively) leads to a small energy required to create a neutron. As a result, the \(\beta\)-equilibrium condition allows easy neutron creation, resulting in a large fraction of neutrons. Pauli blocking strengthens the pressure exerted by the neutron when more neutrons are within the neutron star. For this reason, the pressure of KIDS-D is stiffer than that of KIDS-A. Exact values of \(\rho _c\) are summarized in Table 1. Comparing the results of KIDS-A and KIDS-D models, one can see that, regardless of \(G_v\) value, the transition to the quark phase occurs at low densities with soft symmetry energy. Values of the pressure at \(\rho _c\) (\(P_c\)) are also summarized in the table.

Overall, it indicates that the phase transition is highly sensitive to the nuclear symmetry energy in the hadron phase and the vector repulsion in the quark phase. Their effect on physical observables can be probed by solving the TOV equations and obtaining the mass and radius of neutron stars.

Table 1 Transition point obtained from the P-\(\rho /\rho _0\) relation for the hybrid KIDS-NJLPTR models with different values of \(G_v\)
Full size table

3.2 M-\(\rho\) and M-R relations of the hybrid star

Fig. 4
figure 4

NS mass–density relations of different hadron–quark matter EoS with different values of \(G_v\) for different EDF models: a KIDS0, b KIDS-A, c KIDS-B, d KIDS-C, e KIDS-D, and f SLy4. The thick solid line represents the NS mass for the HM. The thin solid line is the NS mass for the HM+QM with \(G_v = 0.00 \,G_s\), the dashed line is the NS mass for the HM+QM with \(G_v = 0.25 \,G_s\), and the dotted line represents the NS mass for the HM+QM with \(G_v = 0.50 \,G_s\)

Full size image

Figure 4 displays the mass of an NS as a function of the density at the center. For the pure hadronic matter, all six models produce the NS maximum masses (\(M_{\textrm{max}}\)) greater than \(2M_\odot\). The NS maximum mass for the KIDS-D model is the largest, as shown in Fig. 4e, and it decreases in the order of KIDS-C, KIDS-B, and KIDS-A models as in Fig. 4b–d. This result is consistent with the stiffness of EoS in Fig. 1. The result changes dramatically if QM exists in the core of the star.

The thick solid lines denote the NS mass results of the hadronic matter as in Fig. 4a–f. In all the hybrid KIDS-NJL models, the density at which the thin solid lines begin is equal to \(\rho _c\) in Table 1, which means that as the transition to quark matter occurs, the EoS becomes soft, and the matter cannot support strong gravity. As a result, \(M_{\textrm{max}}\) is determined at the density of phase transition. For \(G_v=0\), \(\rho _c\) is not very sensitive to the models, so the NS maximum masses are obtained in a narrow range (1.5\(-\)1.6)\(M_\odot\). These values are substantially low in comparison to \(2M_\odot\), so the result confirms that vector repulsion must be necessarily accounted for in the QM to reproduce the observation of the large mass of NS. The dashed lines correspond to the result of \(G_v=0.25\,G_s\), as shown in Fig. 4a–f. Similar to \(G_v=0\), the density at which a dashed line begins is the same with \(\rho _c\) in Table 1, and \(M_{\textrm{max}}\) is determined at the density where the mass is below \(2M_\odot\) for all EDF models. The result demands that the \(G_v\) value must be higher than \(0.25 \,G_s\).

The results of \(G_v=0.5\,G_s\), which are shown with dotted lines, are now consistent with the \(2M_\odot\) constraint. The density at which the QM curve begins is the highest given by the KIDS-A model, and it then decreases in the order of KIDS-B, KIDS-C, and KIDS-D models. This ordering is understood easily in terms of the stiffness of the EoS of each model. Contrary to the results of \(G_v=0\) and \(0.25\,G_s\), the density at which the dotted curves begin is higher than the \(\rho _c\) values in Table 1. The difference means that even after the QM is formed in the core, its EoS is stiff enough to resist gravitational contraction up to a density at which the NS star reaches the maximum value. However, the interval between \(\rho _c\) and the density at the maximum mass of NS is narrow, so the fraction of QM in the core of the NS is not significant. Summarizing the result, to satisfy the \(2M_\odot\) condition, the \(G_v\) value must be larger than a specific value to obtain the QM EoS stiff. The stiff QM EoS increases \(\rho _c\), and the high \(\rho _c\) value constrains the existence of QM in a limited range.

Fig. 5
figure 5

Models and legends in each panel are the same as Fig. 4 but the results are shown for NS M-R relations as functions of the neutron star radius. Horizontal error bars show the radius of \(1.4M_\odot\) and \(2.08M_\odot\) stars determined in the NICER analysis [36]

Full size image

In Fig. 5, we see the mass–radius (M–R) relation obtained from the TOV equations. The data from GW and NICER provide information on both the mass and radius of NS, allowing for a more stringent constraint on the equation of state of HM and QM than the mass data alone. The radius of an NS with a canonical mass of \(1.4M_\odot\), denoted as \(R_{1.4}\), is crucial. According to the NICER analysis [36], the NS radius is \(R_{1.4} = 12.45 \pm 0.65\) km within a \(1\sigma\) credible interval [36].

On the theoretical side, the density at the center of a \(1.4M_\odot\) NS is found not to exceed \(3 \rho _0\) in general. Exotic states, such as the mixture of hyperons or a transition to QM, typically appear at densities above \(2\rho _0\), with some uncertainty. Therefore, the properties of a \(1.4M_\odot\) NS are crucial for probing (i) the EoS of HM at high densities if there is no transition to QM or hyperons, or (ii) the consistency of the EoS of exotic phases with the data of a \(1.4M_\odot\) NS if these exotic states exist in its interior.

The M–R result indicates that the effect of QM appears at NS masses higher than \(1.5M_\odot\), so the radius of a \(1.4M_\odot\) NS is unaffected by the transition to QM. Hyperons are another source that can soften the EoS of a \(1.4M_\odot\) NS. According to work in progress [37], if the \(\Lambda \Lambda\) interaction determined from the experimental data of double-\(\Lambda\) hypernuclei is included in the EoS of hyperon matter, it makes the EoS stiff. As a consequence, the maximum mass of the NS could be larger than \(2M_\odot\), and the creation of hyperons has a negligible effect on the radius of the \(1.4M_\odot\) star. Therefore, an accurate measurement of \(R_{1.4}\) can provide unique constraints on the EoS of HM at densities up to \(3\rho _0\).

Our results show some differences compared to others, but they are qualitatively compatible overall. In the papers by Refs. [38, 39], they discussed the transition or crossover to quark matter. Both papers described quark matter in terms of the NJL model. However, in the case of the HM, the paper [38] used an RMF model, while the paper [39] employed a chiral effective field theory (\(\chi\)EFT) and Togashi equations of state.

In Ref. [38], the results of the MS-B+vNJL model are comparable to our work. The model assumes three values for the vector repulsion of QM, \(G_v/G_s = 1.5\), 2.0, and 2.5. The results show that the maximum mass of NS is below \(2M_\odot\) when \(G_v=1.5\,G_s\), and it becomes obviously above \(2M_\odot\) when \(G_v= 2.0 \,G_s\). Notably, in our calculation, \(M_{\textrm{max}} \ge 2M_\odot\) is obtained if \(G_v \ge 0.5 \,G_s\), showing a significant difference in the strength of the vector repulsion. The M–R curve of MS-B+vNJL is also very different from what we obtain. Transition to QM occurs at \(\rho = 1.5\rho _0\) or \(\rho =2.0\rho _0\) in the MS-B+vNJL model, and then, the radius becomes smaller than the radius of the HM.

The 1.4 \(M_\odot\) NS radius is about 12.5 km without QM, but it falls in the range 11.7\(-\)12.2 km depending on the \(G_v\) and \(\rho _c\) values. The radii of the maximum mass NSs are located below 11 km, which is much smaller than the NICER estimation \(R_{2.08} = 12.35 \pm 0.75\) km, where \(R_{2.08}\) represents the radius of a \(2.08 M_\odot\) NS [36]. In Ref. [39], the transition from HM to QM is assumed to begin at a density of \(1.5\rho _0\), and its dependence on \(G_v\) is presented. When compared to the \(R_{1.4}\) of the Togashi EoS, which only accounts for nucleons, the radius of a neutron star becomes larger by about 0.2\(-\)0.3 km when quark matter is included in the core. This is contrary to the result of MS-B+vNJL in [38].

The paper shows that the values of \(G_v\) must be larger than \(0.84 \,G_s\) to satisfy the condition \(M_{\textrm{max}} \ge 2.08 M_\odot\). Additionally, it is indicated that the \(G_v\) value in Ref. [39] is much larger than the value obtained in our work to satisfy the \(2M_\odot\) condition. However, recent references in the literature [40, 41] have reported that the strength of \(G_v\) is in the same order as that used in this work to achieve \(2M_\odot\). It is important to note that no strict constraints are currently available for the values of \(G_v\). More precise data are required to constrain \(G_v\).

It is worthwhile to compare our result of the neutron star radius around the NS maximum mass with the NICER result in \(R_{2.08}=11.6 - 13.1\) km. With only hadronic matter, the NS radii at NS maximum mass are in the \(9.8 -11\) km range. The NS radii tend to increase in a stiffer EoS, giving the largest NS radius for the KIDS-C model. However, even the largest NS radius value of the KIDS model is relatively small compared to the lower limit of NICER \(R_{2.08}\). The inconsistency could be understood as a signal for the existence of phases other than the hadrons. Comparing the results of KIDS-A, KIDS-B, KIDS-C, and KIDS-D models, a soft EoS of HM gives larger values of \(\rho _c\), so even if the quark matter is formed in the core, \(R_{\textrm{max}}\) (radius of the maximum mass NS) is too small to be consistent with the NICER \(R_{2.08}\) range. The KIDS-A model clearly shows such behavior. On the other hand, stiff EoSs like the KIDS-C and D models show that the phase transition occurs at a relatively large NS radius, so the NS radius of the maximum mass is shifted to values more prominent than those of pure HM. As a result, the NS with QM in the core can explain the NICER \(R_{2.08}\) data better than the NSs with nucleons only.

The KIDS-A, KIDS-B, KIDS-C, and KIDS-D models aim to meet the constraint \(R_{1.4}=11.8 -12.5\) km, which is more restrictive than the NICER \(R_{1.4}\). The SLy4 and KIDS0 models are designed to match the APR pure neutron matter EoS [42] in the isovector component of the functional. As a result, the M–R behaviors of the SLy4 and KIDS0 models are similar but do not align with NICER \(R_{1.4}\) and \(R_{2.08}\).

The results of this work demonstrate that symmetry energy plays a crucial role in controlling the stiffness of the equation of state of hadronic matter and in affecting the bulk properties of neutron stars with quark matter cores. The KIDS-C and D models, with \(G_v\) values greater than \(0.5\,G_s\), are consistent with NS mass data for \(1.4M_\odot\) and \(2.08 M_\odot\). The values of the nuclear symmetry energy parameters L and \(K_{\textrm{sym}}\) are 58 and \(-91.5\) MeV for the KIDS-C model and 47 and \(-134.5\) MeV for the KIDS-D model, respectively. These values align with the 95% credible range obtained from the KIDS-R14 model set [43], which is \(L=49.8 \pm 10.4\) MeV and \(K_{\textrm{sym}} = -82.4 \pm 67.4\) MeV.

4 Summary

The role of nuclear symmetry energy has been studied in the transition from hadronic matter to the deconfined quark phase in the core of neutron stars. There are two main concerns: first, the critical density for the phase transition depends on the symmetry energy, and second, the repulsive-vector coupling plays a role in the equation of state of quark matter and neutron star properties. The nuclear symmetry energy is determined to be consistent with the data on the canonical mass of neutron stars. The impact of the uncertainty related to the symmetry energy becomes apparent in the mass–radius relationship of neutron stars with masses close to \(2M_\odot\).

The equation of state for quark matter is described in the NJL model. We used the PTR scheme to address the ultraviolet divergence rigorously. It is known that a repulsive-vector coupling is crucial for achieving the maximum mass of the hybrid star consistent with the observation of a \(2M_\odot\) astrophysical object. The dependence on the vector coupling is explored using three different values of the vector coupling constants \(G_v =\) 0.00, 0.25, and 0.50 in units of \(G_s\). The critical density for the phase transition is determined from the condition \(P_{\textrm{HM}}(\rho _c) = P_{\textrm{QM}}(\rho _c)\) at \(\rho _{\textrm{HM}} = \rho _{\textrm{QM}}= \rho _c\).

We have observed that the symmetry energy highly influences the critical density. When comparing the \(\rho _c\) values among the four models (KIDS-A, KIDS-B, KIDS-C, and KIDS-D), we noticed that the value of \(\rho _c\) tends to decrease with softer symmetry energy, meaning \(\rho _c(\textrm{A})> \rho _c(\textrm{B})> \rho _c(\textrm{C}) > \rho _c(\textrm{D})\) regardless of the vector coupling constant. When the vector repulsion is turned off (\(G_v=0\)), the maximum mass of the neutron star is obtained at around \((1.5 - 1.6)M_\odot\). This confirms that repulsive-vector coupling is essential for obtaining consistent results with the observation of \(2M_\odot\). Since the maximum masses of the neutron stars are obtained at \(\rho _c\) for \(G_v=0\), the properties of the 1.4\(M_\odot\) neutron stars are unaffected by the transition to QM. Therefore, the equation of state of a 1.4\(M_\odot\) neutron star could be accurately determined in terms of the hadronic degrees of freedom.

When \(G_v=0.25 \,G_s\), once the quark matter is created in the neutron star core, the NS reaches its maximum mass at around \(1.9M_\odot\), regardless of the model used. To match observations, a stronger repulsion is needed. When \(G_v=0.50 \,G_s\), we found that the maximum NS mass aligns with astrophysical observations, but the NS radius does not match the observed findings. For the KIDS-A and B models, the phase transition does not improve the NS mass–radius relation, falling outside the range determined by the NICER analysis. On the other hand, the KIDS-C and D models agree with the NICER range if quark matter exists in the NS core.

We have demonstrated that the density dependence of the symmetry energy and the vector coupling constant is crucial in the transition from hadronic to quark matter. The symmetry energy and the vector coupling constant influence the critical density. These factors are essential in determining the mass–radius relationship of hybrid stars with masses higher than the canonical mass of neutron stars. Accurately measuring the large masses of neutron stars will provide a unique opportunity to constrain the symmetry energy in hadronic matter and the repulsive-vector coupling in quark matter simultaneously.

To this end, it is worth noting that alternatively, in describing a more realistic quark matter scenario, it would be interesting to consider the quark dynamics in terms of momentum dependence as used in Refs. [44, 45]. The work on this is still in progress, and the result will appear somewhere else in the near future.