1 Introduction

Gauge theories with \(\mathcal{N}=2\) supersymmetry in 4d have been important playground for theoretical physics since the celebrated solution of Seiberg and Witten [1, 2]. They can be geometrically engineered in type IIA string theory [3, 4]. For theories with SU(N) gauge group, compactification manifolds are known to be local toric Calabi–Yau threefolds, and the closed topological string theory on these threefolds encodes essential information about the low energy dynamics of gauge theories. The genus zero topological string amplitude on them determines the prepotential of gauge theories. Higher genus amplitudes further encode gravitational couplings and to a large extent are captured by their holomorphic structure [5]. The topological vertex [6] (see also [7]) and its refinement [8, 9] solve the problem of computing all genus string amplitudes on local toric threefolds.

Topological string theory has been shown to compute contributions coming from inserting codimension two surface operators with certain singularities of the gauge bundle [10, 11]. They are realized by wrapping D4 branes on Lagrangian submanifolds inside the internal space, and extending them along \({\mathbb R}^{2}\subset {\mathbb R}^{4}\). For theories with SU(N) gauge group, we can place the topological branes either on the external or internal legs of the associated toric diagram describing the compactification manifold. The branes provide boundary conditions for holomorphic curves from punctured worldsheet into the target space, and the open topological strings count such maps. The topological vertex in the unrefined case also solves the problem of computing all genus amplitudes for external and internal branes. However, the open amplitude computation turns out to be more subtle when we consider the refined topological string theory, especially when we want to focus on internal branes.

A widely adopted approach in formulating refined open topological amplitudes is based on refining holonomies induced on branes, in addition to the more recent approaches using geometric transition [12] in the refined context [13,14,15]. As it is very common in the refinement, holonomies which are given in terms of Schur functions are replaced by their refined analog, Macdonald functions. This approach looks very plausible and natural, but once the free energy is studied more thoroughly one encounters inconsistencies. This can be observed even in the simplest possible brane configuration: a single internal brane (i.e. one D4 brane) on the resolved conifold. The brane is wrapped on a Lagrangian submanifold on the resolved conifold, and we need to make a choice whether the remaning dimensions of the D4 brane extends either on the plane acted by equivariant parameter \(q=e^{i\varepsilon _{1}}\) or \(t=e^{-i\varepsilon _{2}}\). We call it a q- or t-brane depending which \({\mathbb R}^{2}\) it occupies. After our choice, the brane can not lie on both planes at the same time. Therefore, we expect the amplitude to depend on one of the two equivariant parameters, in addition to the holonomy induced on the brane. However, the existing approach based on naive replacement of the holonomies described above results in an amplitude that depends on both equivariant parameters, a clear indication that the formalism should be modified.

Another important clue can be obtained by placing a stack of branes along one of the refined topological vertex \(C_{\lambda \mu \nu }(t,q)\), say the first leg \(\lambda \). If we choose the holonomy to be expressed in terms of the Schur function, then it is a t-brane. Nevertheless, there is no reason why we should not be able to wrap D4 branes on this Lagrangian submanifold and extend them on the other \({\mathbb R}^{2}\) plane, making it a q-brane. We will address these questions in this short note, and demonstrate that physically consistent results can be obtained by modifying holonomies from Schur functions not only to Macdonald functions but also to dual Macdonald functions and dual Schur functions with respect to the (q, t)-inner product defined by Macdonald [16]. We adopt two different approaches to derive the correct holonomiesFootnote 1: one, which relies on the integral expansion of the open topological string free energy, and another based on refined Chern–Simons theory. All refined holonomies we formulate reduce to Schur functions in the unrefined limit as they should.

As a byproduct, we apply our formalism to reproduce the explicit form of dual elliptic Macdonald functions from toric geometries. In [17], partial and full compactifications of toric webs are introduced, and it is argued that they correspond to elliptic and dual fibrations respectively. We call them horizontal and vertical compactifications, and associate them with position and momentum variables respectively, borrowing terminology from integrable systems. The variable associated with the compactification is elliptically deformed. In this paper, we only focus on placing branes along the horizontal legs. We study both the horizontal compactification in which the brane is an internal brane and the position variable becomes elliptic. In addition, we focus on the vertical compactification when the brane becomes an external brane and the momentum variable has elliptic dependence.

In Sect. 2, we review the open topological string computation for the unrefined case when the brane is placed on an internal leg of the toric diagram. Then we propose how this computation should be refined, and show explicitly that our proposal produces desired form for the free energies. In Sect. 3, we reproduce the refined holonomies using the annuli amplitudes of the refined Chern–Simons theory. In Sect. 4, we use our formalism to reproduce the dual elliptic Macdonald functions. In Sect. 5, we summarize and discuss our results. In Appendix, we collect some relevant information about symmetric functions.

2 Topological Branes

Computing open topological string partition functions on Calabi–Yau threefolds in the presence of D4 branes is an important problem from several perspectives. Geometrically, these topological string amplitudes encode invariants of holomorphic curves with boundaries inside the Calabi–Yau threefold. In terms of type IIA string theory/M-theory, they are counting BPS degeneracies of D2/M2 branes winding holomorphic cycles with boundaries on D4/M5 branes which wrap Lagrangian cycles of the threefold and extend in \({\mathbb R}^{2}\subset {\mathbb R}^{4}\). From the perspective of gauge theory living on \({\mathbb R}^{4}\), they are partition functions of codimension two surface operators.

Historically, this problem has been first addressed and solved in the context of unrefined topological strings. The amplitudes depend on boundary conditions for holomorphic maps, which can be expressed either by conjugacy classes of symmetric group (reflecting how the discs ending on D4-branes are wrapping the \(S^{1}\)) or equivalently in terms of representations of \(U(\infty )\) labelled by arbitrary Young diagrams of any number of rows and columns. On D4 branes, the maps induce a holonomy V for the gauge connection supported by branes. The trace of V in a representation \(\mu \) of the unitary group is given by the Schur function of eigenvalues \((v_1, \ldots , v_n)\) of V,

$$\begin{aligned} \mathrm{tr}_{\mu }V=s_{\mu }(V)=s_{\mu }(v_1, \ldots , v_n). \end{aligned}$$
(2.1)

Two distinct possibilities arise for the location of the D4 branes on a local toric Calabi–Yau threefold: either, they can be placed either along external or internal legs of the associated toric diagram. Accordingly, one talks about external or internal branes. In both cases, a simple prescription is available for the partition function using the topological vertex. The contribution of an edge with an external brane is given by

(2.2)

where \(f_{\mu } (q)= (-1)^{|\mu |} q^{\Arrowvert \mu \Arrowvert ^2/2 - \Arrowvert \mu ^t\Arrowvert ^2/2}\) is the framing factor and p is the amount of framing of the brane. On the other hand, the contribution due to the internal brane is more involved and given by

(2.3)

where \(Q_L\) and \(Q_R\) denote the Kähler parameters associated to the sizes of the left and right discs with respect to the internal brane, respectively, and \(Q=Q_{L}Q_{R}\) is Kähler parameter of the whole \({\mathbb P}^{1}\). \(\mathrm{tr}_{\mu _{L}}V\) is the holonomy induced on the brane from the disc left ending on it, and it describes the boundary conditions for the open amplitudes, and likewise \(V^{-1}\) is due to the disc from the right of the brane.Footnote 2

Above expressions are useful, as they allow us to study various geometric and gauge-theoretic properties of interest. For example, in the presence of external branes the open topological string partition function is conjectured to possess the following integral structure [18, 19],

$$\begin{aligned} Z(V)=\exp \left( \sum _{k=1}^{\infty }\sum _{s,\beta ,\mu }\frac{1}{k}N_{\mu ,\beta ,s}\frac{1}{q^{k/2}-q^{-k/2}}Q_{\beta }^{k}q^{ks}\,\mathrm{tr}_{\mu }V^{k}\right) , \end{aligned}$$
(2.4)

where \(N_{\mu ,\beta ,s}\)’s are a priori undetermined integers and are the degeneracies of BPS states due to D2 branes wrapping a relative homology class \(\beta \in H_{2}(X,{\mathcal L})\) with the Lagrangian submanifold \({\mathcal L}\). They have the 2d space-time spin s and SU(M) representation \(\mu \). One can compute these integers up to any desired order once the open topological string amplitude is computed by any means.

As alluded in the Introduction, the situation is more subtle in the refined case. For unrefined topological string theory, the partition function is insensitive to the choice of \({\mathbb R}^{2}\subset {\mathbb R}^{4}\) that the D4 branes occupy. However, if the space-time is subject to \(\Omega \)-deformation, the planes are distinguished by different rotations. \(\Omega \)-background introduces a complex structure to \({\mathbb R}^{4}\simeq {\mathbb C}^{2}\) and rotates different coordinates by \(\varepsilon _{1}\) and \(\varepsilon _{2}\):

$$\begin{aligned} z_{1}\mapsto e^{i\varepsilon _{1}}z_{1}{:}{=}\,q\,z_{1} ,\qquad z_{2}\mapsto e^{i\varepsilon _{2}}z_{2}{:}{=}\,t^{-1}\,z_{2}. \end{aligned}$$
(2.5)

The refined topological string distinguishes between the two \({\mathbb R}^{2}\) planes. Depending on which plane the D4 brane is extending, we talk about q- or t-branes. Such a distinction is not present in the unrefined topological string. The corresponding open topological string free energy has an integral expansion in terms of \(N_{\mu ,\beta ,s_{L},s_{R}}\), which reads as

$$\begin{aligned} Z=\exp \left( \sum _{k=1}^{\infty }\sum _{s_{L},s_{R},\beta ,\mu }\frac{1}{k}N_{\mu ,\beta ,s_{L},s_{R}}\frac{1}{x^{k/2}-x^{-k/2}}q^{ks_{L}}t^{-ks_{R}} Q_{\beta }^{k}\,\mathrm{tr}_{\mu }V^{k}\right) , \end{aligned}$$
(2.6)

where x is either q or t depending on whether we insert a q- or t-brane. The extra grading can be best understood when we lift to M-theory, as it is the case for closed amplitudes [20, 21]. The BPS particles are charged under the little group of massive particles \(SO(4)\simeq SU(2)_{L}\times SU(2)_{R}\). We measure left \(s_{L}\) and right \(s_{R}\) spins by q and t parameters, respectively.

2.1 External topological branes

We start our discussion with a single stack of external branes on different legs of the refined topological vertex. It is desirable to have a refined version of Eqs. 2.2 and 2.3 to compute refined integer invariants. Such a computation, in addition to the subtlety arising from q versus t-branes, would be also sensitive to whether the brane is located along the preferred or un-preferred leg of the refined topological vertex. We will now demonstrate that the choice of holonomy polynomials \(\mathrm{tr}_{\mu }\,V\) should be “refined” [10, 11, 22]. The need for such a change can easily be seen along the preferred direction: assuming that the holonomy is still given by Schur functions, the integrality of the open free energy is lost:

(2.7)

where problematic terms deviating from the generic structure clearly vanish when \(t=q\). \(\Lambda \) denotes the size of the disc, which could have been absorbed too by rescaling V; we chose not to. This pathology can be cured if we change the holonomy to be given by the Macdonald polynomial \(v^{-|\nu |}\imath P_{\nu ^{t}}(V;t,q)\),

(2.8)

The denominator in the exponential indicates that the deformed holomony gives rise to q-branes. However, nothing prevents us equally wrapping D4 branes on the Lagrangian submanifold along the preferred direction and extend them on the \({\bar{t}}\)-plane. Therefore, the natural question, as it was mentioned in the introduction, is how to obtain the amplitude for a stack of t-branes. The refined topological vertex has a given form, hence, the only freedom we have at our disposal is to change the holonomy. Macdonald defined so-called dual Macdonald polynomials \(Q_{\mu }(V;t,q)\) where the duality is defined with respect to the \(\langle \cdot ,\cdot \rangle _{q,t}\) inner product. If we take the holonomy to be proportional to dual Macdonald function \(Q_{\mu ^{t}}\), we have

(2.9)

Note that the choice of the holonomy also depends on the way we label the refined topological vertex, but once a choice is made then the holonomy is fixed by the integrality condition.

Similarly, we can place a stack of either q- or t-branes along one of the two un-preferred directions. So far in the literature the holonomies along these legs are only taken to be Schur funtions, which are independent of q and t. Up to framing factors, the refined topological vertex depends on one of equivariant parameters if we wrap branes on one of the un-preferred directions. Augmenting with Schur functions as holonomies suggest that along one of un-preferred directions we can only have q-branes and along the other one only t-branes. Again, this picture does not capture the whole physics and therefore is too restrictive. Equipped with our new understanding on the choice of holonomies along the preferred direction, let us determine the holonomies for un-preferred direction. Imagine we want to compute the open amplitude after placing a stack of branes along the first leg of the vertex, \(C_{\lambda \emptyset \emptyset }(t,q)\):

(2.10)

which is the partition function for a t-brane. We can get a q-brane if we chose the holonomy to be proportional to the dual of the Schur function with respect to the (q, t)-inner product; \(S_{\lambda }(V;q,t)\)Footnote 3:

(2.11)

which is clearly a q-brane partition function. As we will elaborate later in Sect. 3.3, there is a nice relationship between the holonomies \(v^{-|\nu |}\imath P_{\nu ^{t}}(V;t,q)\) and \((-1)^{|\nu |} Q_{\nu }(V;q,t)\) along the preferred direction, and the same relationship is true for holonomies \((-v)^{-|\lambda |}s_{\lambda }(V)\) and \((-1)^{|\lambda |}S_{\lambda }(V;q,t)\) along the un-preferred direction.

2.2 Internal topological branes

Let us turn our attention to internal branes, i.e. topological branes placed along internal legs of the toric diagram. For the usual topological string theory, a local prescription is given for the computation of their contributions [6]: one can isolate the internal leg carrying the brane, and compute the associated disc amplitudes locally using,

$$\begin{aligned} \sum _{\mu ,\mu _{L},\mu _{R}}(-Q)^{|\mu |}Q_{L}^{|\mu _{L}|}Q_{R}^{|\mu _{R}|}\,C_{\lambda \sigma (\mu \otimes \mu _{L})}(q)C_{\tau \eta (\mu ^{t}\otimes \mu _{R})}(q)\,\mathrm{tr}_{\mu _{L}}V\,\mathrm{tr}_{\mu _{R}}V^{-1}, \end{aligned}$$
(2.12)

where \(Q=Q_{L}Q_{R}\), and V is the holonomy on the brane from disc ending on it from left, and likewise \(V^{-1}\) is due to discs from the right of the brane. In the above expression, for simplicity we ignore possible framing factors due to line bundles over \({\mathbb P}^{1}\) which are different than \(\mathcal{O}(-1)\oplus \mathcal{O}(-1)\mapsto {\mathbb P}^{1}\), and due to the relative orientation of the interanl brane in the geometry. The representations \(\lambda \), \(\sigma \), \(\tau \) and \(\eta \) allow us to glue this leg with the internal brane to the rest of the geometry.

Although the computation of the open refined topological string partition function includes subtleties related to holonomies, the partition function is expected to algebraically obey similar gluing rules. However, as we pointed out before, we need to choose what type of branes we study, and refine holonomies accordingly. We will seperate the study of q and t-branes in following subsections, and propose refined holonomies based on the requirement that the open free energy has an integral expansion in the refined case as well. Later we will reproduce same holonomies using refined Chern–Simons theory and geometric transition in Sect. 3.

2.2.1 Topological t-branes

In this subsection, we refine the internal brane computation while keeping the algebraic structure the same as in Eq. 2.12. First, we demonstrate that the natural generalization of holonomies from Schur polynomials to Macdonald polynomials leads to a contradiction. Later, we use the integrality requirement for the free energy and predict the correct refinement of holonomies.

Let us first keep holonomies unknown and perform the refined computation explicitly as much as we can. We will assume that the internal brane is placed along the preferred direction of the refined topological vertex,

$$\begin{aligned} Z(V)&=\sum _{\mu ,\mu _{L},\mu _{R}}(-Q)^{|\mu |}Q_{L}^{|\mu _{L}|}Q_{R}^{|\mu _{R}|}C_{\emptyset \emptyset (\mu \otimes \mu _{L})}(t,q)C_{\emptyset \emptyset (\mu ^{t}\otimes \mu _{R})}(q,t)\,\mathrm{tr}_{\mu _{1}}V\, \mathrm{tr}_{\mu _{2}}V^{-1}\nonumber \\&=\sum _{\mu ,\mu _{L},\mu _{R}}(-Q)^{|\mu |}Q_{L}^{|\mu _{L}|}Q_{R}^{|\mu _{R}|}(-v)^{|\mu \otimes \mu _{L}|}P_{\mu \otimes \mu _{L}}(t^{\rho };q,t)(-v)^{-|\mu \otimes \mu _{R}|}\nonumber \\&\quad \times P_{\mu ^{t}\otimes \mu _{R}}(q^{\rho };t,q) \mathrm{tr}_{\mu _{L}}V\, \mathrm{tr}_{\mu _{R}}V^{-1}\nonumber \\&=\sum _{\mu }(-Q)^{|\mu |} \left( \sum _{\mu _{L},\eta _{L}} (-vQ_{L})^{|\mu _{L}|} {\widehat{N}}^{\eta _{L}}_{\mu \mu _{L}}(q,t) P_{\eta _{L}}(t^{\rho };q,t)\,\mathrm{tr}_{\mu _{L}}V \right) \nonumber \\&\quad \,\, \times \left( \sum _{\mu _{R},\eta _{R}} (-v^{-1}Q_{R})^{|\mu _{R}|} {\widehat{N}}^{\eta _{R}}_{\mu ^{t}\mu _{R}}(t,q) P_{\eta _{R}}(q^{\rho };t,q)\,\mathrm{tr}_{\mu _{R}}V^{-1} \right) . \end{aligned}$$
(2.13)

The refined topological vertex with trivial un-preferred and non-trivial preferred direction is proportional to the Macdonald polynomial. Therefore, we use the refined Littlewood–Richardson coefficents \( {\widehat{N}}\) for the tensor product, which are rational functions of q and t and reduce to the usual Littlewood–Richardson coefficients when we take the unrefined limit \(q=t\).

Just as in the external brane example, the crucial point here is the choice of holonomy polynomials \(\mathrm{tr}_{\mu _{L}}V\) and \(\mathrm{tr}_{\mu _{R}}V^{-1}\). Equipped with the previous discussion about the q- and t-branes, we pick correct holonomies. The sums in parentheses are computing the disc amplitudes on each side of the internal branes. If we are studying an internal t-brane, we can pick

$$\begin{aligned} \boxed {\mathrm{tr}_{\mu _{L}}V =v^{-|\mu _{L}|}\imath P_{\mu _{L}^{t}}(V;t,q),} \end{aligned}$$
(2.14)

and we can perform the \(\mu _{L}\)-sum first,

$$\begin{aligned} \sum _{\mu _{L}} (-vQ_{L})^{|\mu _{L}|} {\widehat{N}}^{\eta _{L}}_{\mu \mu _{L}}(q,t)\, v^{-|\mu _{L}|}\imath P_{\mu _{L}^{t}}(V;t,q)=\imath P_{\eta _{L}^{t}/\mu ^{t}}(-Q_{L}\,V;t,q), \end{aligned}$$
(2.15)

and then we peform the remaining \(\eta _{L}\)-sum leading to

(2.16)

which has the correct contribution to the free energy from the left disc, consistent with the results in [18].

Next we need to determine how \(\mathrm{tr}_{\mu _{R}}V^{-1}\) should be refined for a \(\bar{t}\)-brane insertion. One might expect that the holomony is the same as the one for the left disc, possibly up to a \(q\leftrightarrow t\) exchange. However, this choice for the right holonomy fails to produce the correct free energy. Similar to the computation of the left dics amplitude, we first perform the \(\mu _{R}\)-sum for the right disc in Eq. 2.13,

$$\begin{aligned} \sum _{\mu _{R}} (-v^{-1}Q_{R})^{|\mu _{R}|} {\widehat{N}}^{\eta _{R}}_{\mu ^{t}\mu _{R}}(t,q)v^{|\mu _{R}|} \imath P_{\mu _{R}^{t}}(V^{-1};q,t)=\imath P_{\eta _{R}^{t}/\mu }(-Q_{R}V^{-1};q,t), \end{aligned}$$
(2.17)

and then the \(\eta _{R}\)-sum. The contribution from the right disc can again be written as a plethystic exponential,

(2.18)

The right disc contribution appears to be due to the insertion of a q-brane. This leads to a discrepancy between the disc amplitudes coming from different sides of the branes. c.f. Eq. 2.16. Note that our computation is valid for a stack of arbitrary number of branes, and we might as well choose a single brane. In this case, the brane extends only on one \({\mathbb R}^{2}\subset {\mathbb R}^{4}\) in spacetime, not both. Hence, the free energy should depend only on either q or t, not both.

We can avoid such inconsistencies if we allow holonomies to be expressed in terms of other symmetric functions with two parameter deformations. The refined topological vertex is written in a particular basis of symmetric functions [23], which restricts the set of possibilities. In addition to Macdonald polynomials, we can use the dual Macdonald polynomial \(Q_{\mu }(x;q,t)\) as the holonomy without spoiling the integrality of the free energy.Footnote 4 The correct holonomy turns out to be:

$$\begin{aligned} \boxed {\mathrm{tr}_{\mu _{R}}V =(-1)^{|\mu _{R}|}Q_{\mu _{R}}(V^{-1};t,q)} \end{aligned}$$
(2.19)

The \(\mu _{L}\)-sum can be performed explicitly using the definition of the dual skew-Macdonald polynomial,

$$\begin{aligned} \sum _{\mu _{R}} (-v^{-1}Q_{R})^{|\mu _{R}|} {\widehat{N}}^{\eta _{R}}_{\mu ^{t}\mu _{R}}(t,q) (-1)^{|\mu _{R}|}\,Q_{\mu _{R}}(V^{-1};t,q)= Q_{\eta _{R}/\mu ^{t}}(v^{-1}Q_{R}V^{-1};t,q), \end{aligned}$$
(2.20)

and the \(\eta _{R}\)-sum leads to the correct t-brane contribution from the right disc:

(2.21)

Finally, we can do the remaining sum over \(\mu \),

$$\begin{aligned} \sum _{\mu }(-Q)^{|\mu |} P_{\mu }(t^{\rho };q,t)P_{\mu ^{t}}(q^{\rho };t,q)=\prod _{i,j=1}^{\infty }(1-Q\,q^{-\rho _{i}}t^{-\rho _{j}}), \end{aligned}$$
(2.22)

which gives the closed topological string amplitude of the resolved conifold. Therefore, the free energy of an internal t-brane, normalized by the closed amplitude, is given by

(2.23)

The study of integrality properties of the refined open topological string theory for even the simplest possible geometry shows that holonomies need to chosen with some care. The integrality put such severe contraints that we can use it to determine correct holonomies. We show that for internal branes along the preferred direction, the holonomies are not the same type of Macdonald polynomials, and need to be dual with respect to the (q, t)-inner product of Macdonald. We will later discuss this point in more detail.

2.2.2 Topological q-branes

In this subsection, we determine the required refinement of holonomies for internal q-branes. We will be very brief since the computation does not differ much from the one of t-branes in the previous subsection. Equation 2.13 is a general expression assuming the minimal change in the algebraic structure of the internal brane computation from the unrefined case and is valid for either type of branes. Let us again first focus on the “left” sums. We can again borrow our earlier result for the refinement of the external brane and choose the holonomy to be:

$$\begin{aligned} \boxed {\mathrm{tr}_{\mu _{L}}V=(-1)^{|\mu _{L}|}\, Q_{\mu _{L}}(V;q,t)} \end{aligned}$$
(2.24)

After performing all sums for this side of the internal brane, we get

(2.25)

The holomony on the other side can be easily determined to be

$$\begin{aligned} \boxed {\mathrm{tr}_{\mu _{R}}V^{-1}=v^{|\mu _{R}|}\imath P_{\mu _{R}^{t}}(V^{-1};q,t).} \end{aligned}$$
(2.26)

Similarly, all sums on the right side can be performed with this chosen holonomy to get,

(2.27)

The remaining \(\mu \)-sum gives the closed topological string amplitude on the resolved conifold.

Let us also note that all these different holonomies become the usual and same holonomy in the unrefined case!

3 Topological Branes from Refined Chern–Simons Theory

In the previous section, we reviewed that the holonomies of the topological branes need to be modified for refined topological string theory. Moreover, we argued that the general assumption of replacing all Schur polynomials for holonomies with Macdonald polynomials leads to inconsistent free amplitudes for possible brane configurations. In this section, we adopt a different approach and derive the same conclusions from refined Chern–Simons theory. The unrefined topological string amplitudes on local toric Calabi–Yau threefolds were computed using Chern–Simons theory [24, 25] which shortly led to the formulation of the topological vertex [6, 7]. More recently, the refined Chern–Simons theory is constructed [26] and is used to compute the refined topological string amplitudes [23]. In addition to the refined S and T matrices, the refinement of annuli amplitudes are proposed.

The refined Chern–Simons theory is defined as M-theory index in the following background:

$$\begin{aligned} (T^{*}M\times TN\times S^{1})_{q,t}, \end{aligned}$$
(3.1)

where the subscripts denote q and t denote the non-trivial fibration of Taub-NUT space TN over the M-theory circle \(S^{1}\), c.f. Eq. 2.5. Generically, the supersymmetry is broken, but the contangent bundle of M is non-compact and there is an additional \(U(1)_{R}\subset SU(2)_{R}\) that we can use to twist to preserve supersymmetry. If we wrap M5 branes in this background on

$$\begin{aligned} (M\times \mathbb {C}\times S^{1})_{q,t}, \end{aligned}$$
(3.2)

we have a choice to make whether we let the M5 branes extend along the \(z_{1}\) or \(z_{2}\) planes! Therefore, there two distinct refinement of ordinary Chern–Simons theory.

Wilson loops along knots embedded in M are non-local operators of Chern–Simons theory. We compute their contributions by considering the open topological string amplitudes. We can pick the co-normal bundle of the knot that is a Lagrangian sub-manifold \(L_{K}\) in cotangent space \(T^{*}M\). Additional M5 branes can be wrapped on \(L_{K}\), and the theory gets a new sector from M2 branes extend from M to \(L_{K}\). According to this construction, \(L_{K}\) has one real dimensional modulus that allows us to lift it off the base manifold M. Their neighbourhood looks like two Lagrangians each wrapping an \(S^{1}\) in \({\mathbb C}^{*}\). M2 branes wrap the annulus of length \(\Lambda \) connecting these two \(S^{1}\)’s on each stack of M5 branes. Their contribution to the index is captured by the annulus amplitude \({\mathcal O}(\Lambda ;U,V)\) where U and V are the holonomies on M and \(L_{K}\) [18]. For the unrefined topological strings, the annuli partition function is given by

$$\begin{aligned} {\mathcal O}(\Lambda ;U,V)=\det (1-U\otimes V^{-1})^{\pm 1}. \end{aligned}$$
(3.3)

The exponent depends on whether the ground states of strings stretching between the D-branes are fermionic or bosonic. In topological string theory, we can distinguish between branes and anti-branes [6, 27]. Originally in [18], the annulus stretches between two stacks of branes that are wrapped on three cycles which may intersect at most on an \(S^{1}\). This configuration gives rise to bosonic ground states, and the associated amplitude has a \((-1)\) in the exponent. On the other hand, if one of the stacks consists of anti-branes, the ground states turn out to be fermionic; reflected in \((+1)\) in the exponent. This happens when the world volumes of branes are parallel.

In the refined Chern–Simons theory, we can further distinguish between q- and t-branes. The refinement of \({\mathcal O}(\Lambda ;U,V)\) distinguishes all four possibilities. The annuli amplitudes due to two stacks of branes are given by

$$\begin{aligned} \mathcal{O}_{qq}(\Lambda ; U,V)=\prod _{n=0}^{\infty }\frac{\det (1-q^{n}t\Lambda \, U\otimes V^{-1})}{\det (1-q^{n}\Lambda \, U\otimes V^{-1})}, \end{aligned}$$
(3.4)

when q-branes are wrapped on both \(S^{3}\)’s, and take the form

$$\begin{aligned} \mathcal{O}_{qt}(\Lambda ; U,V)=\det (1-v^{-1}\Lambda \,U\otimes V^{-1})^{-1}, \end{aligned}$$
(3.5)

when the branes wrapping different \(S^{3}\)’s are not of the same type. The annuli amplitudes due to brane and anti-brane configurations are given by

$$\begin{aligned} \mathcal{O}_{q\bar{q}}(\Lambda ; U,V)=\prod _{n=0}^{\infty }\frac{\det (1-q^{n}\Lambda \, U\otimes V^{-1})}{\det (1-q^{n}t\Lambda \, U\otimes V^{-1})}, \end{aligned}$$
(3.6)

when both branes and anti-branes are of q-type as the labelling suggests, and

$$\begin{aligned} \mathcal{O}_{q\bar{t}}(\Lambda ; U,V)=\det (1-v^{-1}\Lambda \,U\otimes V^{-1}), \end{aligned}$$
(3.7)

when one of the stacks has q-branes and the other t-branes. We refer the interested reader for the detailed derivation of the refined annuli amplitudes to [23, 26].

Let us also briefly mention that these amplitudes can be expanded in symmetric function with two parameters. For example, the annuli amplitude \(\mathcal{O}_{q\bar{t}}(\Lambda ; U,V)\) can be expanded using the Macdonald functions,

$$\begin{aligned} \mathcal{O}_{q\bar{t}}(\Lambda ; U,V)=\sum _{\mu }(-v^{-1}\Lambda )^{|\mu |}P_{\mu }(U;q,t)P_{\mu ^{t}}(V^{-1};t,q). \end{aligned}$$
(3.8)

This expansion is by no means unique, and in the following section, we will make use of the non-uniqueness of the expansions.

3.1 Topological t-branes from refined Chern–Simons theory

In the previous section, we used the integrality of the open free energy to determine correct holonomies for internal topological branes. This approach was rather mathematical and relied on known identities in the symmetric function theory. As promised, we derive same holonomies using the refined Chern–Simons theory.

We are studying non-compact branes, and need to specify boundary conditions at infinity to define the quantum theory. We employ by now the standard trick of considering branes wrapping on compact Lagrangian cycles. We do so without changing A-model amplitudes. To get additional \(S^{3}\)’s, we allow \(T^{2}\) fibers degenerate over additional loci. In addition to probe branes, we have other \(S^{3}\)’s in the geometry on which we wrap an infinite number of branes and let them go through the geometric transition. In the strict infinite number of branes limit, we take Kähler classes of \({\mathbb P}^{1}\)’s after the transition to infinite as well, leaving only a vertex. We consider the configuration depicted in Fig. 1 to study internal t-branes.

Fig. 1
figure 1

The geometry before the transtion to compute the open amplitude in the presence of a stack of internal t-branes

Full size image

We wrapped q-branes on the left and right \(S^{3}\)’s and t-branes on the middle 3-cycle. We will let the number of q-branes go to infinity to get a resolved conifold with an internal brane insertion. The partition function can be obtained by computing correlators with respect to refined Chern–Simons theories on the left and right 3-cycles. Schematically, we have

$$\begin{aligned} Z(V)=\langle \mathcal{O}_{q\bar{q}}(\Lambda ;U_{R},U_{L}) \mathcal{O}_{qt}(\Lambda _{L};V,U_{L}) \mathcal{O}_{qt}(\Lambda _{R};U_{R},V) \rangle _{SU(N_{L})\otimes SU(N_{R})}, \end{aligned}$$
(3.9)

where we used the notation \(SU(N_{L})\otimes SU(N_{R})\) to emphasize two refined Chern–Simons theories whose large \(N_{L,R}\) limits we are going to take. We have already reviewed refined annuli amplitudes from [23, 26]. These amplitudes can be expanded in various bases of symmetric functions, and we list below the ones of particular importance in our derivation,

$$\begin{aligned} \mathcal{O}_{q\bar{q}}(\Lambda ;U,V)&=\sum _{\mu }\Lambda ^{|\mu |}\imath Q_{\mu }(U;q,t)\,P_{\mu }(V^{-1};q,t), \end{aligned}$$
(3.10)
$$\begin{aligned} \mathcal{O}_{qt}(\Lambda ;U,V)&=\sum _{\mu }(-v^{-1}\Lambda )^{|\mu |}\imath P_{\mu ^{t}}(U;t,q)\,P_{\mu }(V^{-1};q,t)\nonumber \\&=\sum _{\mu }(-v^{-1}\Lambda )^{|\mu |}\imath Q_{\mu ^{t}}(U;q,t)\, Q_{\mu }(V^{-1};t,q). \end{aligned}$$
(3.11)

Using these expansion, it is very easy to show that the partition function takes the following form,

$$\begin{aligned} Z(V)&=\sum _{\begin{array}{c} \mu ,\mu _{L},\mu _{R} \\ \eta _{L},\eta _{R} \end{array}}\Lambda ^{|\mu |}(-v^{-1}\Lambda _{L})^{|\mu _{L}|}(-v^{-1}\Lambda _{R})^{|\mu _{R}|}\widehat{N}_{\mu \mu _{L}}^{\eta _{L}}(q,t)\,\langle P_{\eta _{L}}(U_{L}^{-1};q,t)\rangle _{SU(N_{L})}\nonumber \\&\quad \times \widehat{N}_{\mu ^{t}\mu _{R}}^{\eta ^{t}_{R}}(t,q)\,\langle \imath Q_{\eta _{R}}(U_{R};q,t)\rangle _{SU(N_{R})}\,\imath P_{\mu ^{t}_{L}}(V;t,q)\, Q_{\mu _{R}}(V^{-1};t,q). \end{aligned}$$
(3.12)

To see holonomies we previously found by integrality constraints emerged from refined Chern–Simons annuli amplitudes, we need to compute expectation values, but they have been already computed,

$$\begin{aligned} \langle P_{\eta _{L}}(U_{L}^{-1};q,t)\rangle _{SU(N_{L})}&=P_{\eta _{L}}(t^{\rho };q,t)= (-v)^{-|\eta _{L}|}C_{\emptyset \emptyset \eta _{L}}(t,q), \end{aligned}$$
(3.13)
$$\begin{aligned} \langle \imath Q_{\eta _{R}}(U_{R};q,t)\rangle _{SU(N_{R})}&=(-v)^{-|\eta _{R}|}P_{\eta ^{t}_{R}}(q^{\rho };t,q)=C_{\emptyset \emptyset \eta ^{t}_{R}}(q,t). \end{aligned}$$
(3.14)

Putting everything together reproduces our results for internal \(\bar{t}\) topological brane,

$$\begin{aligned} Z(V)&=\sum _{\mu ,\mu _{L},\mu _{R}}(-v^{-1}\Lambda )^{|\mu |}(v^{-1}\Lambda _{L})^{|\mu _{L}|}(v^{-1}\Lambda _{R})^{|\mu _{R}|}C_{\emptyset \emptyset (\mu \otimes \mu _{L})}(t,q)C_{\emptyset \emptyset (\mu ^{t}\otimes \mu _{R})}(q,t)\nonumber \\&\quad \times v^{-|\mu _{L}|}\imath P_{\mu ^{t}_{L}}(V;t,q)\,(-1)^{|\mu _{R}|}Q_{\mu _{R}}(V^{-1};t,q), \end{aligned}$$
(3.15)

which is the same as we proposed before if we identify

$$\begin{aligned} Q_{L}=v^{-1}\Lambda _{L},\quad Q_{R}=v^{-1}\Lambda _{R},\quad Q=v^{-1}\Lambda . \end{aligned}$$
(3.16)

Let us also comment on this identification. Assume we do not insert any internal branes. After the geometric transition, and letting the \(N_{L,R}\) go to infinity, we end up only with the resolved conifold. For the partition function, we have

$$\begin{aligned} \langle \mathcal{O}_{q\bar{t}}(\Lambda ;U,V)\rangle _{SU(N_{L,R})}&=\sum _{\mu }(-v^{-1}\Lambda )^{|\mu |}P_{\mu ^{t}}(q^{-\rho };t,q)P_{\mu }(t^{-\rho };q,t)\nonumber \\&=\prod _{i,j=1}^{\infty }\left( 1-Q\,q^{-\rho _{i}}t^{-\rho _{i}}\right) \end{aligned}$$
(3.17)

where the first line is from the refined Chern–Simons computation, and the second from the vertex. It is clear that we need \(Q\equiv v^{-1}\Lambda \)!

3.2 Topological q-branes from refined Chern–Simons theory

The derivation of holonomies for q-branes from the refined Chern–Simons theory is analogous to the one for the t-branes, hence we will be brief. Open topological string amplitudes can be obtained by taking the following expectation value:

$$\begin{aligned} Z(V)=\langle \mathcal{O}_{q{\bar{q}}} (\Lambda ;U_{R},U_{L})\mathcal{O}_{qq} (\Lambda _{L};V,U_{L}) \mathcal{O}_{qq} (\Lambda _{R};U_{R},V)\rangle _{SU(N_{L})\otimes SU(N_{R})}. \end{aligned}$$
(3.18)

We will use the following expansions for \( \mathcal{O}_{q{\bar{q}}}\) and \( \mathcal{O}_{qq}\) in terms of Macdonald functions,

$$\begin{aligned} \mathcal{O}_{q{\bar{q}}}(\Lambda ;U,V)&=\sum _{\mu }\Lambda ^{|\mu |}\imath Q_{\mu }(U;q,t)P_{\mu }(V^{-1};q,t), \end{aligned}$$
(3.19)
$$\begin{aligned} \mathcal{O}_{qq}(\Lambda ;U,V)&=\sum _{\mu }\Lambda ^{|\mu |}Q_{\mu }(U;q,t)P_{\mu }(V^{-1};q,t)\end{aligned}$$
(3.20)
$$\begin{aligned}&=\sum _{\mu }\Lambda ^{|\mu |}\imath Q_{\mu ^{t}}(U;q,t) \imath P_{\mu ^{t}}(V^{-1};q,t). \end{aligned}$$
(3.21)

We can again use the expansion in symmetric functions to get,

$$\begin{aligned} Z(V)&=\sum _{\begin{array}{c} \mu ,\mu _{L},\mu _{R} \nonumber \\ \eta _{L},\eta _{R} \end{array}}\Lambda ^{|\mu |}\Lambda _{L}^{|\mu _{L}|}\Lambda _{R}^{|\mu _{R}|}\widehat{N}_{\mu \mu _{L}}^{\eta _{L}}(q,t)\langle P_{\eta _{L}}(U_{L}^{-1};q,t)\rangle _{SU(N_{L})}\widehat{N}_{\mu ^{t}\mu _{R}}^{\eta _{R}^{t}}(t,q)\nonumber \\&\quad \times \langle \imath Q_{\eta _{R}}(U_{R};q,t)\rangle _{SU(N_{R})}\, Q_{\mu _{L}}(V;q,t) \imath P_{\mu _{R}^{t}}(V^{-1};q,t). \end{aligned}$$
(3.22)

We can use the expectation value in Eq. 3.15, and write the partition function in terms of the refined topological vertices:

$$\begin{aligned} Z(V)&=\sum _{\mu ,\mu _{L},\mu _{R}}(-v^{-1}\Lambda )^{|\mu |}(v^{-1}\Lambda _{L})^{|\mu _{L}|}(v^{-1}\Lambda _{R})^{|\mu _{R}|}C_{\emptyset \emptyset (\mu \otimes \mu _{L})}(t,q)C_{\emptyset \emptyset (\mu ^{t}\otimes \mu _{R})}(q,t)\nonumber \\&\quad \times (-1)^{|\mu _{L}|} Q_{\mu _{L}}(V;q,t)\, v^{|\mu _{R}|} \imath P_{\mu _{R}^{t}}(V^{-1};q,t), \end{aligned}$$
(3.23)

agreeing with holonomies we proposed by requiring integrality to the free energy expansion.

3.3 Duality relation and brane changing operator

We would like to point out a nice relation between holonomies associated to the t- and q-branes. Let us first summarize holonomies we found for internal branes:

 

left holonomy

right holonomy

t-brane

\(v^{-|\mu _{L}|}\imath P_{\mu ^{t}_{L}}(V;t,q)\)

\((-1)^{|\mu _{R}|}\, Q_{\mu _{R}}(V^{-1};t,q)\)

q-brane

\((-1)^{|\mu _{L}|}\, Q_{\mu _{L}}(V;q,t) \)

\(v^{|\mu _{R}|} \imath P_{\mu _{R}^{t}}(V^{-1};q,t)\)

One obvious relationship is a diagonal one: the left holonomy of the t-brane becomes the right holonomy of the q-brane after a \(q\leftrightarrow t\) exchange. The same is true for remaining holonomies.

There is yet a second relationship between holonomies of the q- and t-branes that can be understood algebraically. The dual Macdonald function \(Q_{\mu }(V;q,t)\) is defined using the endomorphism \(\omega _{q,t}\) on the ring of symmetric functions. This is just a two parameter generalization of the involution \(\omega \) on Schur functions; \(\omega s_{\lambda }=s_{\lambda ^{t}}\). The action of \(\omega _{q,t}\) on the powersums for \(t\ne 1\) is given by,

$$\begin{aligned} \omega _{q,t}\,p_{n}=(-1)^{n-1}\frac{1-q^{n}}{1-t^{n}}p_{n}. \end{aligned}$$
(3.24)

If we compare two holonomies, it is not hard to find a vertical relationship. Let us define the brane changing operator \(\Omega _{t,q}\):

$$\begin{aligned} \Omega _{t,q}\equiv (-v)^{L_{0}}\imath \, \omega _{t,q}, \end{aligned}$$
(3.25)

where \(L_{0}\) counts the number of boxes labeling the symmetric function. The action of \(\Omega _{t,q}\) can be easily computed on t-brane holonomies:

$$\begin{aligned}&\Omega _{t,q}\,\Big ( v^{-|\mu _{L}|}\imath P_{\mu ^{t}_{L}}(V;t,q)\Big )=\Big ( (-1)^{|\mu _{L}|}\, Q_{\mu _{L}}(V;q,t) \Big ),\nonumber \\&\Omega _{t,q}\,\Big ( (-1)^{|\mu _{R}|}\, Q_{\mu _{R}}(V^{-1};t,q) \Big )=\Big ( v^{|\mu _{R}|}\imath P_{\mu _{R}^{t}}(V^{-1};q,t)\Big ), \end{aligned}$$
(3.26)

which change the holonomies for t-branes into q-branes on each side of the brane. Hence, we reach

$$\begin{aligned} \Omega _{t,q}:\quad {t-brane}\mapsto {q-brane}. \end{aligned}$$
(3.27)

The action of \(\Omega _{t,q}\) on powersums which are isomorphic to oscillator modes is easy to determine as well:

$$\begin{aligned} \Omega _{t,q}\,p_{n}=\frac{t^{n/2}-t^{-n/2}}{q^{n/2}-q^{-n/2}}p_{n}. \end{aligned}$$
(3.28)

Note that the brane changing operator from q-brane to t-brane is just the inverse of the former operator, \(\Omega _{q,t}=\Omega ^{-1}_{t,q}\), similar to \(\omega _{q,t}\). In addition, it become the identity operator when we take the unrefined limit, \(q=t\).

We can see that \(\Omega _{q,t}\) is in fact the brane changing operator from refined Chern–Simons theory as well. Its action on the anti-brane is the same. The amplitude of annuli stretching between q and \(\bar{t}\) branes is given by,

$$\begin{aligned} \mathcal{O}_{q\bar{t}}(\Lambda ;U,V)&=\det \left( 1-v^{-1}\Lambda \,U \otimes V^{-1} \right) \nonumber \\&=\sum _{\mu }(-v^{-1}\Lambda )^{|\mu |}\,P_{\mu }(U;q,t) \,P_{\mu ^{t}}(V^{-1};t,q). \end{aligned}$$
(3.29)

We should consider the action of \(\Omega _{t,q}\) on the stack of probe branes described by \(P_{\mu ^{t}}(V^{-1};t,q)\),

$$\begin{aligned} \Omega _{t,q}\,P_{\mu ^{t}}(V^{-1};t,q)=(-v)^{|\mu |}\imath Q_{\mu }(V^{-1};q,t), \end{aligned}$$
(3.30)

converting the \(\mathcal{O}_{q\bar{t}}(\Lambda ;U,V)\) annulus into \(\mathcal{O}_{q\bar{q}}(\Lambda ;U,V)\):

$$\begin{aligned} \boxed {\Omega _{t,q}\,\mathcal{O}_{q\bar{t}}(\Lambda ;U,V)=\mathcal{O}_{q\bar{q}}(\Lambda ;U,V)} \end{aligned}$$
(3.31)
$$\begin{aligned} \boxed {\Omega _{t,q}\,\mathcal{O}_{qt}(\Lambda ;U,V)=\mathcal{O}_{qq}(\Lambda ;U,V)} \end{aligned}$$
(3.32)

Before we finish this section, let us briefly mention that \(\Omega _{t,q}\) acts the same way on the holonomies along the un-preferred direction:

$$\begin{aligned} \Omega _{t,q}\left( (-v)^{-|\lambda |}s_{\lambda }(V)\right) =\left( (-1)^{|\lambda |}S_{\lambda }(V;q,t)\right) , \end{aligned}$$
(3.33)

changing a t-brane to a q-brane.

4 Application: Double Compactified Toric Calabi–Yau 3-fold

As a principal illustration, we apply our formalism to compute ‘refined open topological amplitudes on the doubly compactified toric Calabi–Yau threefold with Kähler parameters \(Q_{\rho }\) and \(Q_{\tau } = \Lambda x Q_{m}^2\) associated with elliptic curves [17]. We study different limits of the Kähler classes with branes along the horizontal leg of the toric diagram, as depicted in Fig. 2.

Fig. 2
figure 2

The toric diagram assosicated with the M-strings. In addition to the vertical compactification, we also compactified the horizontal direction

Full size image

This geometry generalizes the non-compact case \(Q_{\rho } = Q_{\tau } = 0\), when the partition function can be straightforwardly computed with the help of standard topological vertex methods and has a number of nice properties. One can express the partition function either as a series (in what follows we fix a single brane, \(n = 1\))

$$\begin{aligned} \frac{Z}{Z_0}\Big |_{Q_{\rho }=Q_{\tau }=0}&= 1 + \frac{\sqrt{t}(1-T)}{T(1-q)} \underbrace{\big ( 1 + x Q_m \big )}_{P_1(x)} u \nonumber \\&\quad + \frac{t (1 - T)(1 - q T)}{T^2 (1 - q)(1 - q^2)} \underbrace{\big ( 1 + \frac{(1+q)(1-T)}{(1-qT)} x Q_m + x^2 Q_m^2 \big )}_{P_2(x)} u^2 + \cdots \end{aligned}$$
(4.1)

where \(T = \sqrt{\frac{q}{t}} Q_m^{-1}\), or in closed form

$$\begin{aligned} \dfrac{Z}{Z_0}\Big |_{Q_{\rho }=Q_{\tau }=0} = \dfrac{(t^{\frac{1}{2}} u;q)_{\infty } (t^{\frac{1}{2}} u x Q_m;q)_{\infty }}{(t^{\frac{1}{2}} u / T;q)_{\infty } (t^{\frac{1}{2}} u x Q_m / T;q)_{\infty }} \end{aligned}$$
(4.2)

where \((x;q)_{\infty } = \prod _{n = 0}^{\infty } (1 - q^n x)\). Both representations are interesting. In the series representation, the coefficients \(P_k(x)\) are the celebrated polynomial eigenfunctions of the trigonometric Ruijsenaars–Schneider system—Macdonald polynomials with parameters q and T. In the closed form representation, each of the 4 factors corresponds to a BPS state in the geometry.

In this section, we will give elliptic generalizations of both representations. As it turns out, the first (series) representation generalizes most nicely to the case \(Q_{\rho } \ne 0\), while the second (closed form, BPS) representation admits a nice generalization to the case \(Q_{\tau } \ne 0\). This is an illustration of validity of the the holonomy prescription that we suggest.

In this geometry the edge with the brane has no other parallel edges, hence our local prescription applies and gives for the partition function of the brane

$$\begin{aligned} Z&= \sum _{\{\tau _{i}\},\{\lambda _{i}\}}\sum _{\nu } \sum _{\mu ,\mu _{L},\mu _{R}} (-x)^{|\nu |}(-Q_{m})^{|\lambda _{1}|+|\lambda _{2}|}\nonumber \\&\quad (-Q_{\tau }Q^{-1}_{m})^{|\tau _{1}|+|\tau _{2}|}(-\Lambda )^{|\mu |} \Lambda ^{|\mu _{R}|}C_{\lambda _{1}^{t}\tau _{1}^{t}\nu ^{t}}(q,t) \nonumber \\&\quad \times C_{\lambda _{1}\tau _{1}(\mu \otimes \mu _{L})}(t,q) C_{\lambda _{2}^{t}\tau _{2}^{t}(\mu ^{t}\otimes \mu _{R})}(q,t)C_{\lambda _{2}\tau _{2}\nu }(t,q)\, (-1)^{|\mu _{L}|} \nonumber \\&\quad Q_{\mu _{L}}\big (u_1, \ldots , u_n;q,t\big ) \imath P_{\mu _{R}^{t}}\big (u_1^{-1}, \ldots , u_n^{-1};q,t\big ) \nonumber \\&=\sum _{\nu } \sum _{\mu ,\mu _{L},\mu _{R}} \sum _{\eta _{L},\eta _{R}} (-x)^{|\nu |}(-\Lambda )^{|\mu |} \Lambda ^{|\mu _{R}|}(-1)^{|\mu _{L}|}\nonumber \\&\quad {\widehat{N}}_{\mu \mu _{L}}^{\eta _{L}}(q,t) {\widehat{N}}_{\mu ^{t}\mu _{R}}^{\eta _{R}}(t,q)D_{\nu \eta _{L}}(Q_{\tau },Q_{m}) \nonumber \\&\quad \times D_{\eta _{R}^{t} \nu }(Q_{\tau },Q_{m}) \, Q_{\mu _{L}}\big (u_1, \ldots , u_n;q,t\big ) \imath P_{\mu _{R}^{t}}\big (u_1^{-1}, \ldots , u_n^{-1};q,t\big ). \end{aligned}$$
(4.3)

Some of the Kähler classes in the above expression are rescaled for notational purposes to make the comparison to elliptic Macdonal polynomial easier. The \(\tau _{i}\) and \(\lambda _{i}\) sums can be performed explicitly and we called them \(D_{\nu \mu }(Q_{\tau },Q_{m})\) following [28] where it was first computed and regarded as a building block for M-strings. It takes the following form,

$$\begin{aligned}&D_{\nu \mu }(Q_{\tau },Q_{m})\nonumber \\&\quad \equiv \sum _{\lambda ,\tau }(-Q_{m})^{|\lambda |}(-Q_{\tau }Q_{m}^{-1})^{|\tau |} C_{\lambda ^{t}\tau ^{t}\nu ^{t}}(q,t)C_{\lambda \tau \mu }(t,q) \nonumber \\&\quad =q^{\frac{\Arrowvert \mu \Arrowvert ^{2}}{2}-\frac{\Arrowvert \nu \Arrowvert ^{2}}{2}}t^{-\frac{\Arrowvert \mu ^{t}\Arrowvert ^{2}}{2}+\frac{\Arrowvert \nu ^{t}\Arrowvert ^{2}}{2}}P_{\mu }(t^{-\rho };q,t)P_{\nu ^{t}}(q^{-\rho };t,q)\nonumber \\&\qquad \times \prod _{k=1}^{\infty }\prod _{(i,j)\in \nu }\frac{(1-Q_{\tau }^{k}Q_{m}^{-1}\,q^{-\nu _{i}+j-\frac{1}{2}}t^{-\mu _{j}^{t}+i-\frac{1}{2}})(1-Q_{\tau }^{k-1}Q_{m}\,q^{\nu _{i}-j+\frac{1}{2}}t^{\mu _{j}^{t}-i+\frac{1}{2}})}{(1-Q_{\tau }^{k}\,q^{-\nu _{i}+j}t^{-\nu _{j}^{t}+i-1})(1-Q_{\tau }^{k}\,q^{\nu _{i}-j+1}t^{\nu ^{t}_{j}-i} )}\nonumber \\&\qquad \times \prod _{(i,j)\in \mu }\frac{(1-Q_{\tau }^{k}Q_{m}^{-1}\,q^{\mu _{i}-j+\frac{1}{2}}t^{\nu ^{t}_{j}-i+\frac{1}{2}})(1-Q_{\tau }^{k-1}Q_{m}\,q^{-\mu _{i}+j-\frac{1}{2}}t^{-\nu ^{t}_{j}+i-\frac{1}{2}})}{(1-Q_{\tau }^{k}\,q^{-\mu _{i}+j-1}t^{-\mu ^{t}_{j}+i})(1-Q_{\tau }^{k}\,q^{\mu _{i}-j}t^{\mu _{j}^{t}-i+1})}. \end{aligned}$$
(4.4)

We now proceed to compute Eq. 4.3. When all parameters are generic, we will compute the partition function as a series expansion in the Kahler classes in the Gopakumar–Vafa form, demonstrating that the result has correct pole structure and integrality properties. For some special values of parameters, those series expansions can be summed up in closed form and related to pronounced solutions of elliptic Ruijsenaars–Schneider integrable models: namely, the elliptic Macdonald series and dual elliptic Macdonald polynomials. The two particular cases that we consider correspond to \(Q_{\tau } = 0\) (momentum-elliptic) and \(Q_{\rho } = 0\) (coordinate-elliptic).

4.1 Generic case

As usual for any sufficiently non-trivial Calabi–Yau geometry, if all parameters are generic it is not possible to give a closed form expression for the partition function of topological strings in that background. One of the most convenient forms in which it can be expressed is the Gopakumar–Vafa form also known as the BPS expansion. We first remind how this works for a closed string partition functionFootnote 5

$$\begin{aligned} Z_0= \sum _{\mu ,\nu } (-x)^{|\nu |} \ (-\Lambda )^{|\mu |}D_{\nu \mu } (Q_{\tau },Q_{m})\, D_{\mu \nu } (Q_{\tau },Q_{m}). \end{aligned}$$
(4.5)

In this case, the Gopakumar–Vafa expansion has the form

$$\begin{aligned} Z_0= \exp \left( \sum \limits _{d = 0}^{\infty } \dfrac{ F_0\big (q^d,t^d,x^d,\Lambda ^{d},Q_m^d,Q_{\rho }^d\big ) }{d(q^{d/2} - q^{-d/2})(t^{d/2} - t^{-d/2})}\right) , \end{aligned}$$
(4.6)

where the numerator \(F_0(q,t,Q_{\beta })\) encodes the degeneracies, \(N_{\beta }^{(j_{L},j_{R})}\), of the BPS states with their charges under the little group in 5d coming from wrapping M2 branes on \(\beta \):

$$\begin{aligned} F_0(q,t,Q_{\beta })=\sum _{\beta \in H_{2}(X,\mathbb {Z})}\sum _{j_{L},j_{R}}(-1)^{2j_{L}+2j_{R}}N_{\beta }^{(j_{L},j_{R})}\, \mathrm{tr}_{j_{R}}\left( \frac{q}{t}\right) ^{j_{R,3}}\, \mathrm{tr}_{j_{L}}\left( q\,t\right) ^{j_{L,3}}\, Q_{\beta } \end{aligned}$$
(4.7)

where we used as a collective Kähler class \(Q_{\beta }\) that includes x, \(\Lambda \), \(Q_{m}\), \(Q_{\tau }\) and \(Q_{\rho }\). To the first few orders,

$$\begin{aligned} F_0&= - x + \big ( t^{\frac{1}{2}} q^{-\frac{1}{2}} + q^{\frac{1}{2}} t^{-\frac{1}{2}} \big ) x Q_m - x Q_m^2 + \nonumber \\&\quad + \big ( t^{\frac{1}{2}} q^{-\frac{1}{2}} + q^{\frac{1}{2}} t^{-\frac{1}{2}} \big ) x Q_{\rho } Q_m^{-1} + \big ( t^{\frac{3}{2}} q^{-\frac{3}{2}} + t^{\frac{1}{2}} q^{-\frac{1}{2}} + q^{\frac{1}{2}} t^{-\frac{1}{2}} + q^{\frac{3}{2}} t^{-\frac{3}{2}} \big ) x^2 Q_{\rho } + \nonumber \\&\quad - \Lambda + \big ( t^{\frac{1}{2}} q^{-\frac{1}{2}} + q^{\frac{1}{2}} t^{-\frac{1}{2}} \big ) \Lambda Q_m - 2 \Lambda Q_m x + \cdots . \end{aligned}$$
(4.8)

Note that the result is manifestly symmetric under q, t exchange, as a closed partition function should be. It is also symmetric under \((q,t) \rightarrow (q^{-1},t^{-1})\).

Similarly, for an open string partition function with a (single, \(n=1\)) brane insertion computed with Eq. 4.3 and normalized over the closed part, we find,

$$\begin{aligned} \dfrac{Z}{Z_0} = \exp \left( \sum \limits _{d = 0}^{\infty } \dfrac{ F\big (q^d,t^d,u^d,x^d,\Lambda ^{d},Q_m^d,Q_{\rho }^d\big ) }{d(q^{d/2} - q^{-d/2})} \right) , \end{aligned}$$
(4.9)

where the numerator F, to the first few orders, is given by,

$$\begin{aligned} F&= u + u x Q_m - t^{\frac{1}{2}} q^{-\frac{1}{2}} u Q_m - t^{\frac{1}{2}} q^{-\frac{1}{2}} u x Q_m^2 \nonumber \\&\quad - t^{-\frac{1}{2}} q^{\frac{1}{2}} u Q_m^{-1} Q_{\rho } + u x Q_m^{-1} Q_{\rho } + \big ( 1 + q + t^{-1} \big ) u Q_{\rho } \nonumber \\&\quad - \big ( 2 + 2 t q^{-1} + t + q^{-1} \big ) t^{-\frac{1}{2}} q^{\frac{1}{2}} u Q_{\rho } x + \big ( 1 + t^{-1} q + t q^{-1} \big ) u Q_{\rho } x^2 \nonumber \\&\quad + t^{\frac{1}{2}} q^{-\frac{1}{2}} \Lambda u^{-1} - \Lambda u^{-1} Q_m + t^{\frac{1}{2}} q^{-\frac{1}{2}} \Lambda x Q_m u^{-1} - x \Lambda u^{-1} Q_m^2 \nonumber \\&\quad + (1-t) q^{-\frac{1}{2}} \Lambda Q_m^2 x + \cdots . \end{aligned}$$
(4.10)

Note that the result is not symmetric under q, t exchange, as one can expect for an open partition function: the symmetry is broken by choosing a q-brane vs. a t-brane. This does not mean the multiplicities of BPS states depend on this choice: rather, by choosing a q-brane vs. a t-brane one counts the same BPS states with different weights. In other words, the two partition functions contain identical enumerative information.

4.2 Case \(Q_{\tau } = 0\)

In this subsection, we decompactify the “vertical” elliptic fiber with Kähler class \(Q_{\tau }\), and have only one elliptic fiber in the “horizontal” direction.Footnote 6 In the language of the integrable system literature, this choice constitutes the momentum-elliptic case. In this case for either of the three values of \(Q_m = \sqrt{\frac{q}{t}}, \sqrt{\frac{t}{q}}, \sqrt{q t}\), the following holds for the normalized open amplitude:

$$\begin{aligned} \dfrac{Z}{Z_0} = \sum \limits _{k = 0}^{\infty } \left( \frac{u \sqrt{q}}{T}\right) ^k \; c_k \; \mathcal{P}_{k}(x), \end{aligned}$$
(4.11)

where \(c_k\) are numeric normalization factors,

$$\begin{aligned} c_k = \prod \limits _{i = 0}^{k-1} \prod \limits _{m = 0}^{\infty } \dfrac{(1 - Q_{\rho }^m q^{i} T)(1 - Q_{\rho }^{m+1} q^{-i} T^{-1})}{(1 - Q_{\rho }^m q^{i+1})(1 - Q_{\rho }^{m+1} q^{-i} t^{-1})}, \end{aligned}$$
(4.12)

and \(\mathcal{P}_{k}(x)\) are the dual elliptic Macdonald polynomials,

$$\begin{aligned} \mathcal{P}_{k}(x) = \sum \limits _{m = 0}^{k} \, \left( Q_m x\right) ^m \, \prod \limits _{i = 0}^{m-1} \dfrac{\theta (q^{k-i})\theta (T q^i)}{\theta (q^{k-i}T/q)\theta (q^{i+1})}, \end{aligned}$$
(4.13)

with Macdonald parameters q and \(T = \sqrt{\frac{q}{t}} Q_m^{-1}\). Here

$$\begin{aligned} \theta (z) = \prod \limits _{m = 0}^{\infty } (1 - Q_{\rho }^m z)(1 - Q_{\rho }^{m+1} / z) \end{aligned}$$
(4.14)

is the Jacobi theta function. Note that the result only has a singularity at \(q \rightarrow 1\), but not at \(t \rightarrow 1\), since the choice of holonomies in Eq. 4.3 corresponds to a (in this example single, \(n=1\)) q-brane.

Polynomial \(\mathcal{P}_{k}(x)\) represents the brane partition function with fixed boundary condition corresponding to the k-th symmetric representation of SU(2), and it satisfies

$$\begin{aligned} \left( \dfrac{q Q_m x}{T} \dfrac{\theta ( q^{x \partial _x - k} )}{\theta ( q^{x \partial _x - k + 1} T^{-1} )} - \dfrac{\theta ( q^{x \partial _x} )}{\theta ( q^{x \partial _x - 1} T )} \right) \, \mathcal{P}_{k}(x) = 0, \end{aligned}$$
(4.15)

which is the dual elliptic Ruijsenaars–Schneider equation, as given in [29, Eq. 4.85] and [30, Eq. 4.59].

4.3 Case \(Q_{\rho } = 0\)

In this case, the shortest way of giving the answer is through the BPS expansion,

$$\begin{aligned} \dfrac{Z}{Z_0} = \exp \left( \sum \limits _{d = 0}^{\infty } \dfrac{ F\big (q^d,t^d,u^d,x^d,\Lambda ^{d},Q_m^d,Q_{\tau }^d\big ) }{d(q^{d/2} - q^{-d/2})} \right) , \end{aligned}$$
(4.16)

where the numerator F, to all orders, is given by,

$$\begin{aligned} F = \dfrac{ (1-t) q^{-\frac{1}{2}} \Lambda Q_m^2 x + (1 + x Q_m)\big ( u - t^{\frac{1}{2}} q^{-\frac{1}{2}} u Q_m + t^{\frac{1}{2}} q^{-\frac{1}{2}} \Lambda u^{-1} - Q_m \Lambda u^{-1} \big )}{ 1 - Q_{\tau } }, \end{aligned}$$

where, we remind, \(Q_{\tau } = \Lambda x Q_m^2\) is the total Kähler class of the compact 2-cycle. The geometric progression factor \((1 - Q_{\tau })^{-1} = 1 + Q_{\tau } + Q_{\tau }^2 + \cdots \) implies that infinitely many BPS states that wrap this cycle multiple times are organized in a simple tower of states. This is a characteristic property of the coordinate-elliptic case.

5 Discussion and Outlook

In this paper, we studied the refinement of the holonomies for open topological string amplitudes. The correct holonomies are determined using the integrality properties of the refined open free energies, and are also independently derived with the refined Chern–Simons theory. Our approach demonstrates how one can place q- or t-branes on any leg of the refined topological vertex.

The local prescription for the computation of open topological string amplitudes in the presence of internal branes is well known to be valid in the unrefined case for any toric geometry. However, in the refined case the issue is more subtle. This can be easily observed by studying surface operators in \(\mathcal{N}=2\) SU(2) theory with four flavors. In 4d, the contribution due the surface operator can be calculated by inserting an additional degenerate operator in the Liouville four-point conformal block [31]. If we lift the theory to 5d, \(\mathbb {R}^{4}\times S^{1}\), we can study its q-deformation using the Dotsenko–Fateev Coulomb gas interpretation of the conformal block. From the geometric engineering view point, it is nothing but inserting an internal brane along one of the horizontal line in the toric diagram of the local \(\mathbb {P}^{1}\times \mathbb {P}^{1}\) geometry blown up at four points. We used our refinement of holonomies to reproduce the surface operator contribution and observed a mismatch.

Fig. 3
figure 3

The geometry before the transtion: we place a single brane on the upper \(S^{3}\) and no branes on the lower \(S^{3}\)

Full size image

On the other hand, we used geometric transition to compute the same amplitude and found an agreement with the q-deformed Liouville theory computation. In this approach, we need to wrap a single brane on one of the 3-cycles for a surface operator, and none to the other 3-cycle, depicted in Fig. 3. The failure of the local prescription suggests that annuli can end on the 3-cycle with “no brane” and give contributions. This may look strange since the annuli are not expected to end on a 3-cycle if no-branes are wrapped on it. One possible explanation is that, if the same number branes and anti-branes are wrapped on it, effectively there may not be any gauge theory associated with it but the the annuli still have a place to end. This point is still under investigation and will be reported later [32].

The above observation holds true more generally. Whenever the NS5 brane used in geometric transition intersects a single \(\mathbb {P}^{1}\) in the original geometry, the local prescription that we proposed reproduces the correct result. Conversely, whenever there is more than one intersection, we observe a mismatch. This suggests that further modifications may be necessary in these cases. Because of non-local nature of the “no brane” phenomenon described above, these modifications may ultimately lead to a non-local prescription.