1 Introduction

An important problem in the theory of automorphic forms is to understand periods, and how they are related with L-functions and their special values, as well as with functorial liftings. A prototypical example for this is the connection between symmetric and exterior square L-functions, functorial liftings from classical groups to \(GL_n,\) and certain periods, which dates back at least to [22], and is more fully explicated in [15, 16]. Some more exotic examples are found, for example, in [9, 10], and a general framework which extends beyond classical groups is discussed in [13].

The connection between poles and liftings is well- understood, at least philosophically: one expects that the L function attached to a generic cuspidal representation \(\pi \) and a finite dimensional representation r of the relevant L-group will have a pole at 1 if and only if the stabilizer of a point in general position for the representation r is reductive and \(\pi \) is in the image of the functorial lifting attached to the inclusion of the stabilizer of such a point. For example, in the exterior square representation of \(GL_{2n}(\mathbb {C}),\) the stabilizer of a point in general position is \(Sp_{2n}(\mathbb {C}),\) so one expects that a pole of the exterior square L function indicates cuspidal representations which are lifts from \(SO_{2n+1}.\) The connection with periods is up to now less well understood.

In order to prove the expected relationship between poles and liftings in specific examples, and in order to draw periods into the picture, it is useful, perhaps essential, to have some sort of an analytic handle on both the L-function and the lifting. An analytic handle on the L-function may be provided by an integral representation, either of Langlands–Shahidi type or otherwise (integral representations which are not of Langlands–Shahidi type are often termed “Rankin–Selberg”). An analytic handle on the lifting may be provided by an explicit construction.

Integral representation of L functions and explicit construction of liftings between automorphic forms on different groups are important subjects in their own right as well. For example, integral representations are, as far as we know, the only way to establish analytic properties of L functions in new cases. When an integral representation produces L functions whose analytic properties are already well understood, it nevertheless provides a new insight into the connection with periods, and identities among periods which can be otherwise quite surprising. This is the case in the present paper.

For explicit construction of liftings, there are two main ideas we know of. Each is related to the other and both are related to the theory of Fourier coefficients attached to nilpotent orbits [8, 14].

The first main idea is to use a “small” representation as a kernel function. The prototypical example of this type is the classical theta correspondence [19]. In this type of construction, an automorphic form, which is defined on a large reductive group H is restricted to a pair of commuting reductive subgroups, and integrated against automorphic forms on one member of the pair to produce automorphic forms on the other. In general, there is no reason such a construction should preserve irreducibility, much less be functorial. The right approach seems to be to take automorphic forms on H which only support Fourier coefficients attached to very small nilpotent orbits. For example, a theta function, defined on the group \(\widetilde{Sp}_{4mn}(\mathbb {A})\) only supports Fourier coefficients attached to the minimal nilpotent orbit of this group. Its restriction to \(Sp_{2n}(\mathbb {A}) \times O_{2m}(\mathbb {A})\) provides a kernel for the theta lifting between these groups. Functoriality of this lifting was established in [29]. This method has enjoyed brilliant success, but also has significant limitations. For example, it is not at all clear how the classical theta correspondence could be extended to other groups of type \(C_n\times D_m\): the embedding into \(Sp_{4mn}\) is specific to \(Sp_{2n} \times O_{2m}.\) The results of this paper hint at a possible way around this difficulty.

The second main idea in explicit construction of correspondences is the descent method of Ginzburg, Rallis, and Soudry ([16], see also [20]). This construction treats the Fourier coefficients themselves essentially as global twisted Jacquet modules, mapping representations of larger reductive groups to representations of smaller reductive groups. As before, in general there is no reason for this construction to respect irreducibility, much less be functorial, and a delicate calculus involving Fourier coefficients seems to govern when it is.

In this paper we define and study two new multi-variable Rankin–Selberg integrals, which are defined on the similitude orthogonal groups \(GSO_{12}\) and \(GSO_{18}.\) These integrals are similar to those considered in [3, 6, 7, 11, 12], in that each involves applying a Fourier–Jacobi coefficient to a degenerate Eisenstein series and then pairing the result with a cusp form defined on a suitable reductive subgroup. To be precise, \(GSO_{6n}\) has a standard parabolic subgroup Q whose Levi is isomorphic to \(GL_{2n} \times GSO_{2n}.\) The unipotent radical is a two step nilpotent group and the set of characters of the center may be thought of as the exterior square representation of \(GL_{2n}\) twisted by the similitude factor of \(GSO_{2n}.\) The stabilizer of a character in general position is isomorphic to

$$\begin{aligned} C:= \{ (g_1, g_2) \in GSp_{2n} \times GSO_{2n}: \lambda ( g_1) = \lambda ( g_2^{-1})\}. \end{aligned}$$

Here, \(\lambda \) denotes the similitude factor. The choice of a character in general position as above also determines a projection of the unipotent radical onto a Heisenberg group in \(4n^2+1\) variables, and a compatible embedding of C into \(Sp_{4n^2}.\)

Our Fourier–Jacobi coefficient defines a map from automorphic functions on \(GSO_{6n}(\mathbb {A})\) to automorphic functions on \(C(\mathbb {A}).\) In the case \(n=2\) and 3 we apply this coefficient to a degenerate Eisenstein series on \(GSO_{6n}(\mathbb {A})\) induced from a character of the parabolic subgroup P whose Levi factor is isomorphic to \(GL_3 \times GL_{3n-3} \times GL_1.\) We then pair the result with a pair of cusp forms defined on \(GSp_{2n}(\mathbb {A})\) and \(GSO_{2n}(\mathbb {A})\) respectively. The results suggest an intriguing connection with the theta correspondence for similitude groups.

Indeed, in the case \(n=2,\) the global integral turns out to be Eulerian, and to give an integral representation of

$$\begin{aligned} L^S( s_1, \widetilde{\Pi }\times \tau _1) L^S( s_2, \widetilde{\Pi }\times \tau _2), \end{aligned}$$

where \(\Pi \) is a generic cuspidal automorphic representation of \(GSp_4(\mathbb {A})\) and \(\tau _1, \tau _2\) are two (generic) cuspidal automorphic representations of \(GL_2(\mathbb {A})\) having the same central character, so that \(\tau _1 \otimes \tau _2\) is a (generic) cuspidal automorphic representation of \(GSO_4(\mathbb {A}).\) It follows that the original integral has poles along both the plane \(s_1=1\) and the plane \(s_2=1\) if and only if \(\Pi \) is the weak lift of \(\tau _1 \otimes \tau _2\) corresponding to the embedding

$$\begin{aligned} GSpin_4(\mathbb {C}) = \{ (g_1, g_2)\in GL(2, \mathbb {C})^2 \mid \det g_1 = \det g_2\} \hookrightarrow GSpin_5(\mathbb {C}) = GSp_4(\mathbb {C}). \end{aligned}$$

It is known that the functorial lift corresponding to the embedding

$$\begin{aligned} SO_4(\mathbb {C}) \hookrightarrow SO_5(\mathbb {C}) \end{aligned}$$

is realized via the theta correspondence. Our Eulerian integral suggests that the Fourier–Jacobi coefficient of the iterated residue of our Eisenstein series provides a kernel for the theta correspondence for similitude groups. This is particularly intriguing since the Fourier–Jacobi coefficient construction extends directly to any group of type \(D_{3n},\) whereas there seems to be no hope of extending the theta correspondence to any representations of such groups which do not factor through the orthogonal quotient in any direct way.

The result is also intriguing in that it points to a possible identity relating our Fourier–Jacobi coefficient with a theta series on \(\widetilde{Sp}_{16}(\mathbb {A}).\) We are not aware of any way to see such an identity directly.

The integral corresponding to \(n=3\) provides some more evidence for a connection with the theta correspondence, in that the global integral unfolds to a period of \(GSp_6\) which is known to be nonvanishing precisely on the image of the theta lift from \(GSO_6\) [15].

We now describe the contents of this paper. In Sect. 2 we fix notation and describe a family of global integrals, indexed by positive integers n. In Sect. 3 we unfold the global integral corresponding to the case \(n=2,\) obtaining a global integral involving the Whittaker function of the cusp form involve which, formally, factors as a product of local zeta integrals. These local zeta integrals are studied in Sects. 5, 6, 7, after certain algebraic results required for the unramified case are established in Sect. 4. Once the local zeta integrals have been studied we return to the global setting for Sects. 8 and 9, where we record the global identity relating the original zeta integral and \(L^S( s_1, \widetilde{\Pi }\times \tau _1) L^S( s_2, \widetilde{\Pi }\times \tau _2),\) and deduce a new identity relating poles of these L functions and periods. Finally, in Sect. 10, we briefly describe what happens in the case \(n=3,\) omitting details. We remark that the case \(n=1\) is somewhat degenerate, as the split form of \(GSO_2\) is a torus; our global integral appears to vanish identically in this case.

This work was undertaken while the authors were visiting ICERM for a program on Automorphic Forms, Combinatorial Representation Theory and Multiple Dirichlet Series. They thank ICERM and the organizers, as well as David Ginzburg and the referee.

2 Notation

Write \(J_n\) for the matrix

If g is an \(n\times m\) matrix, write \(^tg\) for the transpose of g and \(_tg\) for the “other transpose,” \(J_m{}\,^tgJ_n.\) Let \(g^* = \,_tg^{-1}.\) Let \(G=GSO_{n}\) denote the identity component of \(GO_{n}:= \{ g \in GL_{n}: gJ_{n} \,^tg \in GL_1\cdot J_{n}\}.\) If n is odd, then \(GO_n\) is the product of \(SO_n\) and the center of \(GL_n.\) If n is even, then \(GSO_n\) is the semidirect product of \(SO_{n}\) and \(\{ {\text {diag}}( \lambda I_{\frac{n}{2}}, I_{\frac{n}{2}}): \lambda \in GL_1\}.\) Here \(I_k\) is the \(k \times k\) identity matrix. The group \(GSO_{n}\) has a rational character \(\lambda : GSO_{n} \rightarrow GL_1,\) called the similitude factor, such that

$$\begin{aligned} gJ_{n} \,^tg = \lambda (g) \cdot J_{n}, \qquad ( g \in GSO_n). \end{aligned}$$

The set of upper triangular (resp. diagonal) elements of \(GSO_{n}\) is a Borel subgroup (resp. split maximal torus) which we denote \(B_{GSO_n}\) (resp. \(T_{GSO_n}\)). A parabolic (resp. Levi) subgroup will be said to be standard if it contains \(B_{GSO_n}\) (resp. \(T_{GSO_n}\)). The unipotent radical of \(B_{GSO_n}\) will be denoted U. We number the simple (relative to \(B_{GSO_{2n}}\)) roots of \(T_{GSO_{2n}}\) in G \(\alpha _1, \ldots , \alpha _n\) so that \(t^{\alpha _i} = t_{ii}/t_{i+1,i+1}\) for \(1 \le i \le n-1,\) and \(t^{\alpha _n} = t_{n-1,n-1}/t_{n+1,n+1}.\) Here, we have used the exponential notation for rational characters, i.e., written \(t^\alpha \) instead of \(\alpha (t)\) for the value of the root \(\alpha \) on the torus element t.

Define \(m_P:GL_{3}\times GL_{3(n-1)}\times GL_{1}\) into \(GSO_{6n}\) by

$$\begin{aligned} m_P (g_{1},g_{2},\lambda )\mapsto diag(\lambda g_{1}, \lambda g_{2},g_{2}^{*}, g_{1}^{*}). \end{aligned}$$
(1)

Denote the image by \(M_P.\) It is a standard Levi subgroup. Let P be the corresponding standard parabolic subgroup. Thus, \(P=M_{P}\ltimes U_{P},\) where \(U_P\) is the unipotent radical. We use (1) to identify \(M_P\) with \(GL_{3}\times GL_{3n-3}\times GL_{1}.\)

Recall that a character of \(F^\times \backslash \mathbb {A}^\times \) (i.e., a character of \(\mathbb {A}^\times \) trivial on \(F^\times \)) is normalized if it is trivial on the positive real numbers (embedded into \(\mathbb {A}^\times \) diagonally at the finite places). An arbitrary quasicharacter of \(F^\times \backslash \mathbb {A}^\times \) may be expressed uniquely as the product of a normalized character and a complex power of the absolute value. If \(\chi = (\chi _{1},\chi _{2},\chi _{3})\) is a triple of normalized characters of \(F^\times \backslash \mathbb {A}^\times \) and \( s = (s_1, s_2 , s_3) \in \mathbb {C}^3,\) write \((\chi ; s)\) for the quasicharacter \(M_P(\mathbb {A}) \rightarrow \mathbb {C}\) by

$$\begin{aligned} (\chi ;s)(g_{1},g_{2},\lambda ):=\chi _1(\det g_{1})|\det g_1|^{s_1}\chi _2(\det g_{2})|\det g_2|^{s_2}\chi _3(\lambda )|\lambda |^{s_3}. \end{aligned}$$
(2)

Then \((\chi ;s)(aI_{6n})= (\chi ;s)(m_P(a^{-1}I_3, a^{-1}I_{3(n-1)}, a^2)) = \chi _1^{-3}\chi _2^{3-3n}\chi _3^2(a)|a|^{2s_3-3s_1+(3-3n)s_2}.\) The pullback of \((\chi ; s)\) to a quasicharacter of \(P(\mathbb {A})\) will also be denoted \((\chi ; s).\)

Consider the family of induced representations \(Ind_{P(\mathbb {A})}^{G(\mathbb {A})}(\chi ;s)\) (non-normalized induction), for fixed \(\chi \) and s varying. Here, we fix a maximal compact subgroup K of \(G(\mathbb {A})\) and consider K-finite vectors. The map \(Ind_{P(\mathbb {A})}^{G(\mathbb {A})}(\chi ;s)\mapsto s\) gives this family the structure of a fiber bundle over \(\mathbb {C}^3.\) By a section we mean a function \(\mathbb {C}^3 \times G(\mathbb {A})\rightarrow \mathbb {C},\) written \((s, g) \mapsto f_{\chi ; s}(g),\) such that \(f_{\chi ;s } \in Ind_{P(\mathbb {A})}^{G(\mathbb {A})} (\chi ; s)\) for each \(s \in \mathbb {C}^3.\) A section \(f_{\chi ; s}\) is flat if the restriction of \(f_{\chi ; s}\) to K is independent of s. Write \({\text {Flat}}(\chi )\) for the space of flat sections.

For such a function \(f_{\chi ;s},\) let \(E(f_{\chi ;s},g)\) be the corresponding Eisenstein series, defined by

$$\begin{aligned} E(f_{\chi ;s},g)=\sum _{\gamma \in P(F)\backslash G(F)}f_{\chi ;s}(\gamma g) \end{aligned}$$

when this sum is convergent and by meromorphic continuation elsewhere. The sum is convergent for \({\text {Re}}(s_1-s_2)\) and \({\text {Re}}(s_2)\) both sufficiently large (Cf. [27], §II.1.5).

Let \(Q=M_{Q}\ltimes U_{Q}\) be the unique standard parabolic subgroup of G, such that \(M_{Q}\cong GL_{2n}\times GSO_{2n}.\) We identify \(M_Q\) with \(GL_{2n}\times GSO_{2n}\) via the isomorphism

$$\begin{aligned} m_Q(g_{1},g_{2}):= diag(\lambda (g_2) g_{1}, g_{2}, g_{1}^{*}), \qquad (g_1 \in GL_{2n}, \; g_2 \in GSO_{2n}). \end{aligned}$$
(3)

The unipotent radical, \(U_Q,\) of Q can be described as

$$\begin{aligned} \left\{ \begin{pmatrix} I_{2n}&{}\quad X&{}\quad Y&{}\quad Z'\\ 0&{}\quad I_n&{}\quad 0&{}\quad -{_t} Y\\ 0&{}\quad 0&{}\quad I_n&{}\quad -{_t} X\\ 0&{}\quad 0&{}\quad 0&{}\quad I_{2n} \end{pmatrix}: Z' + X{_t} Y + Y{_t} X + {_t} Z' = 0 \right\} . \end{aligned}$$

Let \({{}^2 \wedge _{2n}}: = \{ Z \in {\text {Mat}}_{{2n}\times {2n}}: {_t}Z = -Z\}.\) Then we can define a bijection (which is not a homomorphism) \(u_Q: {\text {Mat}}_{{2n}\times n}\times {\text {Mat}}_{{2n}\times n} \times {{}^2 \wedge _{2n}} \rightarrow U_Q\) by

$$\begin{aligned} u_Q(X,Y,Z) = \begin{pmatrix} I_{2n}&{}\quad X&{}\quad Y&{}\quad Z-\frac{1}{2}(X{_t} Y + Y {_t} X)\\ 0&{}\quad I_n&{}\quad 0&{}\quad -{_t} Y\\ 0&{}\quad 0&{}\quad I_n&{}\quad -{_t} X\\ 0&{}\quad 0&{}\quad 0&{}\quad I_{2n} \end{pmatrix}, \quad X, Y \in {\text {Mat}}_{2n\times n}, \quad Z \in {{}^2 \wedge _{2n}}. \end{aligned}$$

Then

$$\begin{aligned} u_Q(X,Y,Z)u_Q(U,V,W) = u_Q(X+U,Y+V, Z+W - \langle X,V\rangle + \langle U,Y\rangle ), \end{aligned}$$

\((X,Y,U,V \in {\text {Mat}}_{n\times {2n}}, Z,W \in {{}^2 \wedge _{2n}}),\) where

$$\begin{aligned} \langle A,B \rangle := A{_t} B- B {_t} A, \qquad (A,B \in {\text {Mat}}_{n\times {2n}}). \end{aligned}$$

It follows that \(u_Q(X,Y,Z)^{-1} = u_Q(-X,-Y,-Z),\) and that if \([x,y]=xyx^{-1}y^{-1}\) denotes the commutator, then \( [ u_Q(X,0,0), \, u_Q(0,Y,0)] = u_Q( 0,0, -X_t Y +Y_t X)=u_Q(0,0,\langle Y,X\rangle ). \) Define \(l(Z) = Tr( Z\cdot diag(I_n,0))=\sum _{i=1}^n Z_{i,i}\). For \(n \in \mathbb {Z}\) define \(\mathcal H_{2n+1}\) to be \(\mathbb {G}_a^n \times \mathbb {G}_a^n \times \mathbb {G}_a\) equipped with the product

$$\begin{aligned} (x_1, y_1, z_1) (x_2, y_2, z_2) = (x_1+x_2, y_1+y_2, z_1+z_2+x_1 {_t} y_2 - y_1 {_t} x_2). \end{aligned}$$

Write r for the map from \({\text {Mat}}_{2n \times n}\) to row vectors corresponding to unwinding the rows: \(r(X) := x_{1,1},\ldots x_{1,n},x_{2,1}, \ldots x_{2n,n},\) and write \(r'\) for the similar map which unwinds the rows and negates the last n. Explicitly:

$$\begin{aligned} r' (Y) = r \begin{pmatrix} Y_1 \\ - Y_2 \end{pmatrix} = (y_{1,1},\ldots y_{1,n},y_{1,1}, \ldots y_{n,n},-y_{n+1,1},\ldots , -y_{2n,n}) \end{aligned}$$

for \(Y = \begin{pmatrix} Y_1 \\ Y_2 \end{pmatrix} \in {\text {Mat}}_{2n\times n}, \; Y_1, Y_2 \in {\text {Mat}}_{n\times n}.\) Then we can define a homomorphism from \(U_{Q}\) to \(\mathcal {H}_{4n^2+1}\), the Heisenberg group with \(4n^2+1\) variables, by

$$\begin{aligned} j(u_Q(X,Y,Z))=(r(X), r'(Y) ,l(Z)). \end{aligned}$$

The stabilizer of l in \(M_{Q}\) is

$$\begin{aligned} C_{Q}:= (GSp_{2n}\times GSO_{2n})^{\circ }=\lbrace (g_{1},g_{2})\in GSp_{2n}\times GSO_{2n}\mid \lambda (g_{1})=\lambda (g_{2})^{-1}\rbrace , \end{aligned}$$

where \(\lambda (g_{i})\) is the similitude of \(g_{i}\). For any subgroup H of \(GSp_{2n}\times GSO_{2n},\) let \(H^\circ := H \cap C_Q.\) The kernel of l is a \(C_Q\)-stable subgroup of \(U_Q,\) and is also equal to the kernel of j. Note that \(\lambda (g_2)\) is also the similitude factor of \((g_1,g_2)\) as an element of \(GSO_{6n},\) and that the center of \(C_Q\) is equal to that of \(GSO_{6n}.\) Define \(T= T_{GSO_n} \cap C_Q, B = B_{GSO_n} \cap C_Q\) and \(N = U \cap C_Q.\) They are a split maximal torus, Borel subgroup, and maximal unipotent subgroup of \(C_Q,\) respectively.

The group of automorphisms of \(\mathcal H_{4n^2+1}\) whose restrictions to the center of \(\mathcal H_{4n^2+1}\) are the identity is isomorphic to \(Sp_{4n^2}.\) Identifying the two groups defines a semidirect product \(Sp_{4n^2} \ltimes \mathcal H_{4n^2+1}.\) Let \(R_Q=C_{Q}\ltimes U_{Q}.\) The homomorphism \(j: U_Q \rightarrow \mathcal H_{4n^2+1}\) extends to a homomorphism \(R_Q \rightarrow Sp_{4n^2} \ltimes \mathcal H_{4n^2+1}.\) Indeed, for each \(c \in C_Q,\) the automorphism of \(U_Q\) defined by conjugation by c preserves the kernel of j,  and therefore induces an automorphism of \(\mathcal H_{4n^2+1}.\) Moreover, this automorphism is identity on the center of \(\mathcal H_{4n^2+1}\) because c fixes l. This induces a homomorphism \(C_Q \rightarrow Sp_{4n^2},\) which we denote by the same symbol j,  and which has the defining property that \(j(cuc^{-1}) = j(c) j(u) j(c)^{-1}\) for all \(c \in C_Q\) and \(u \in U_Q.\) We may then regard the two homomorphisms together as a single homomorphism (still denoted j) from \(R_Q\) to \(Sp_{4n^2} \ltimes \mathcal H_{4n^2+1}.\)

For a positive integer M,  identify the Siegel Levi of \(Sp_{2M}\) with \(GL_M\) via the map \(\begin{pmatrix} g&{} \\ &{}\quad g^* \end{pmatrix} \mapsto g.\) It acts on \(\mathcal H_{2M+1}\) by \(g(x,y,z)g^{-1} = (x g^{-1}, y g , z).\) Note that for \(g_1 \in GSp_{2n}\) and \(g_2 \in GSO_{2n},\) the matrix \(m_Q(g_1, g_2) \in M_Q\) maps into \(GL_{2n^2}\) if and only if it normalizes \(\{ u_Q(X,0,0): X \in {\text {Mat}}_{2n\times n}\},\) i.e., if and only if \(g_2\) is of the form \(\begin{pmatrix} \lambda (g_1^{-1}) g_3 &{}\quad \\ &{}\quad g_3^*\end{pmatrix}\) for \(g_3 \in GL_n.\) Write

$$\begin{aligned} m_Q^1( g_1, g_2) := m_Q\left( g_1, \begin{pmatrix} \lambda (g_1)^{-1} g_2 &{}\quad \\ &{}\quad g_2^* \end{pmatrix}\right) , \qquad ( g_1 \in GSp_{2n}, \; g_2 \in GL_n). \end{aligned}$$
(4)

Then

$$\begin{aligned} m_Q^1( g_1, g_2) u_Q(X,0,0) m_Q^1( g_1^{-1}, g_2^{-1}) = u_Q( g_1 X g_2^{-1},0,0), \qquad ( \forall g_1 \in GSp_{2n},\; g_2 \in GL_n), \end{aligned}$$

and \(j(m_Q^1(g_1, g_2)) \in GL_{2n^2}\subset Sp_{4n^2}\) is the matrix satisfying

$$\begin{aligned} r(X) j(m_Q^1(g_1, g_2)) = r(g_1^{-1} X g_2). \end{aligned}$$

The determinant map \(GL_{2n^2} \rightarrow GL_1\) pulls back to a rational character of this subgroup of \(C_Q\) which we denote by \(\det .\) Thus \(\det (m_Q^1( g_1, g_2)) = \det g_1^{-n}\det g_2^{2n} = \lambda ( g_1)^{-n^2} \det g_2^{2n}\) for \(g_1 \in GSp_{2n}, g_2 \in GL_n.\) On \(T\subset C_Q,\) the rational character \(\det \) coincides with the restriction of the sum of the roots of \(T_{GSO_{6n}}\) in \(\{ u_Q(0,Y,0):Y \in {\text {Mat}}_{2n\times n}\}.\)

Let \(\psi \) be a additive character on \(F\backslash \mathbb {A}\) and \(\psi _{l}(Z):=\psi \circ l\). The group \(\mathcal H_{4n^2+1}(\mathbb {A})\) has a unique (up to isomorphism) unitary representation, \(\omega _\psi ,\) with central character \(\psi ,\) which extends to a projective representation of \(\mathcal H_{4n^2+1}(\mathbb {A}) \rtimes Sp_{4n^2}(\mathbb {A})\) or a genuine representation of \(\mathcal H_{4n^2+1}(\mathbb {A}) \rtimes \widetilde{Sp}_{4n^2}(\mathbb {A}),\) where \(\widetilde{Sp}_{4n^2}(\mathbb {A})\) denotes the metaplectic double cover.

Lemma 2.1

The homomorphism \(j:C_Q(\mathbb {A}) \rightarrow Sp_{4n^2}(\mathbb {A})\) lifts to a homomorphism \(C_Q(\mathbb {A}) \rightarrow \widetilde{Sp}_{4n^2}(\mathbb {A}).\)

Proof

Write \({\text {pr}}\) for the canonical projection \(\widetilde{Sp}_{4n^2}(\mathbb {A})\rightarrow Sp_{4n^2}(\mathbb {A}).\) We must show that the exact sequence

$$\begin{aligned} 1 \rightarrow \{ \pm 1\} \rightarrow {\text {pr}}^{-1}(j(C_Q(\mathbb {A}))) \rightarrow j(C_Q(\mathbb {A})) \rightarrow 1 \end{aligned}$$

splits, i.e., that the cocycle determined by any choice of section is a coboundary. The analogous result for \(Sp_{2n} \times SO_{2n},\) over a local field is proved in [24], corollary 3.3, p. 36, or [28], lemma 4.4, p. 12. The extension to \(C_Q\) follows from section 5.1 of [18]. The global statement then follows from the corresponding local ones. \(\square \)

Thus we obtain a homomorphism \(R_Q(\mathbb {A}) \rightarrow \widetilde{Sp}_{4n^2}(\mathbb {A}) \ltimes \mathcal H_{4n^2+1}(\mathbb {A})\) which we still denote j. Pulling \(\omega _\psi \) back through j produces a representation of \(R_Q(\mathbb {A})\) which we denote \(\omega _{\psi ,l}.\) This representation can be realized on the space of Schwartz functions on \({\text {Mat}}_{{2n}\times n}(\mathbb {A})\) with action by

$$\begin{aligned}{}[\omega _{\psi ,l}(u_Q(0,0,Z)).\phi ] = \psi _l(Z) \phi \qquad [\omega _{\psi ,l}(u_Q(X,0,0)).\phi ](\xi ) = \phi ( \xi +X) \end{aligned}$$
$$\begin{aligned}{}[\omega _{\psi ,l}(u_Q(0,Y,0)).\phi ](\xi ) = \psi _l ( \langle Y, \xi \rangle )\phi (\xi )= \psi _l( Y \,_t \xi -\xi {_t} Y)\phi (\xi ),\end{aligned}$$
(5)
$$\begin{aligned}{}[\omega _{\psi , l}( m_Q^1( g_1, g_2)).\phi ](\xi ) = \gamma _{\psi , \det m_Q^1( g_1, g_2)} |\det m_Q^1( g_1, g_2)|^{\frac{1}{2}} \phi ( g_1^{-1} \xi g_2 ). \end{aligned}$$

(Cf. [16], p. 8). Here \(\gamma _{\psi , a}\) denotes the Weil index. The representation \(\omega _\psi \) has an automorphic realization via theta functions

$$\begin{aligned} \theta (\phi , u \widetilde{g}):= \sum _{\xi \in {\text {Mat}}_{{2n}\times n}(F)} [\omega _\psi ( u \widetilde{g}).\phi ](\xi ), \qquad (u \in \mathcal H_{4n^2+1}(\mathbb {A}),\ \widetilde{g} \in \widetilde{Sp}_{4n^2}(\mathbb {A})). \end{aligned}$$

Here \(\phi \in \mathcal S({\text {Mat}}_{{2n}\times n}(\mathbb {A}))\) (the Schwartz space of \({\text {Mat}}_{{2n}\times n}(\mathbb {A})\)), \(\mathcal H_{4n^2+1}(\mathbb {A})\) is identified with the quotient of \(U_Q\) by the kernel of l,  and \(Sp_{4n^2}\) is identified with the subgroup of its automorphism group consisting of all elements which act trivially on the center.

Lemma 2.2

Consider the Weil representation of \(\mathcal H_{2n+1}(\mathbb {A}) \rtimes \widetilde{Sp}_{2n}(\mathbb {A})\) and its automorphic realization by theta functions. Let V be a subgroup of \(\mathcal H_{2n+1}\) which intersects the center Z trivially. Thus V corresponds to an isotropic subspace of the symplectic space \(\mathcal H_{2n+1}/Z.\) Let \(V^\perp = \{ v' \in \mathcal H_{2n+1}/Z: \langle v', V\rangle = 0\} \supset V,\) and let \(P_V\) be the parabolic subgroup of \(Sp_{2n}\) which preserves the flag \(0 \subset V \subset V^\perp \subset \mathcal H_{2n+1}/Z.\) Note that the Levi quotient of \(P_V\) is canonically isomorphic to \(GL(V) \times Sp( V^\perp /V).\) Thus \(P_V\) has a projection onto the group \(GL_1 \times Sp( V^\perp /V)\) induced by the canonical map onto the Levi quotient and the determinant map \(\det : GL(V) \rightarrow GL_1.\) The function

$$\begin{aligned} \widetilde{g} \mapsto \int _{[V]} \theta (\phi ; v\widetilde{g}) \, dv \end{aligned}$$

is invariant by the \(\mathbb {A}\)-points of the kernel of this map on the left. (Throughout this paper, if H is an algebraic group defined over a global field F,  then \([H]:= H(F) \backslash H(\mathbb {A}).\))

Proof

First assume that V is the span of the last k standard basis vectors for some \(k \le n.\) Then

$$\begin{aligned} \int _{(F\backslash \mathbb {A})^k} \sum _{\xi \in F^n}[\omega _\psi ( (0, \ldots , 0, v , 0)\widetilde{g}). \phi ](\xi ) \, dv = \sum _{ \xi ' \in F^{n-k}} [\omega _\psi ( \widetilde{g}). \phi ](0,\ldots , 0, \xi '), \end{aligned}$$

and invariance follows easily from the explicit formulae for \(\omega _\psi \) given, for example on p. 8 of [16]. The general case follows from this special case, since any isotropic subspace can be mapped to the span of the last k standard basis vectors, for the appropriate value of k,  by using an element of \(Sp_{2n}(F).\) \(\square \)

For \(f_{\chi ;s} \in {\text {Ind}}_{P(\mathbb {A})}^{G(\mathbb {A})} (\chi ;s),\) and \(\phi \in \mathcal S({\text {Mat}}_{{2n}\times n}(\mathbb {A})),\) let

$$\begin{aligned} E^{\theta (\phi )}(f_{\chi ;s},g)=\int _{[U_{Q}]}\ du\ E(f_{\chi ;s},ug)\theta (\phi ,j(ug)), \qquad (g\in {C}_{Q}(\mathbb {A})). \end{aligned}$$
(6)

Recall that \(C_Q\) was identified above with a subgroup of \(GSp_{2n} \times GSO_{2n}.\) If \(g \in C_Q\) then \(g_1\) will denote its \(GSp_{2n}\) component and \(g_2\) will denote its \(GSO_{2n}\) component. Now take two characters \(\omega _1, \omega _2: F^\times \backslash \mathbb {A}^\times \rightarrow \mathbb {C}^\times ,\) and two cuspforms \(\varphi _{1},\) defined on \({GSp}_{2n}(\mathbb {A})\) and \(\varphi _1,\) defined on \(GSO_{2n}(\mathbb {A}),\) such that \(\varphi _i(a\cdot g) = \omega _i(a) \varphi _i(g),\) for \(i =1\) or 2,  \(a \in \mathbb {A}^\times ,\) and \(g \in GSp_{2n}(\mathbb {A})\) or \(GSO_{2n}(\mathbb {A})\) as appropriate. Choose \(\chi _1, \chi _2, \chi _3\) so that \(\chi _1^{-3} \chi _2^{-3} \chi _3^2 \omega _1^{-1} \omega _2\) is trivial, and consider

$$\begin{aligned} I(f_{\chi ;s},\varphi _{1},\varphi _{2}, \phi )=\int _{Z(\mathbb {A})C_{Q}(F)\backslash C_{Q}(\mathbb {A})}E^{\theta (\phi )}(f_{\chi ;s},g)\varphi _{1}(g_{1})\varphi (g_{2})dg. \end{aligned}$$
(7)

To simplify the notation, we may also treat the product \(\varphi _1\varphi _2\) as a single cuspform defined on the group \(C_Q,\) and write \(\varphi (g) = \varphi _1(g_1)\varphi _2(g_2),\) and \(I(f_{\chi ;s}, \varphi , \phi ),\) etc. Note that the integral converges absolutely and uniformly as s varies in a compact set, simply because \(E^{\theta (\phi )}(f_{\chi ;s})\) is of moderate growth, while \(\varphi _1\) and \(\varphi _2\) are of rapid decay.

3 Global integral for \(GSO_{12}\)

In this section we consider a global integral (7) in the case \(n=2.\) Thus \(G=GSO_{12}.\) If \(u_Q(X,Y,Z)\) is an element of \(U_Q,\) we fix individual coordinates as follows:

$$\begin{aligned} X= \begin{pmatrix} x_{1}&{}\quad x_{2} \\ x_{3} &{}\quad x_{4} \\ x_{5} &{}\quad x_{6}\\ x_{7} &{}\quad x_{8} \end{pmatrix}, \qquad Y= \begin{pmatrix} y_{8}&{}\quad y_{7} \\ y_{6} &{} \quad y_{5} \\ y_{4} &{}\quad y_{3}\\ y_{2} &{}\quad y_{1} \end{pmatrix}\qquad Z= \begin{pmatrix} z_1&{}\quad z_2&{}\quad z_3&{}0\\ z_4&{}\quad z_5&{}\quad 0&{}\quad -z_3\\ z_6&{}\quad 0&{}\quad -z_5&{}\quad -z_2\\ 0&{}\quad -z_6&{}\quad -z_4&{}\quad -z_1 \end{pmatrix}. \end{aligned}$$
(8)

Theorem 3.1

For n in the maximal unipotent subgroup N let \(\psi _N(n) = \psi (n_{12}+n_{23}-n_{56}+n_{57}),\) and let

$$\begin{aligned} W_\varphi (g) = \int _{[N]}\varphi (ng) \psi _N(n)\, dn. \end{aligned}$$
(9)

Let \(U_4\) be the codimension one subgroup of N defined by the condition \(n_{23}=n_{56}.\) For \(\phi \in \mathcal S({\text {Mat}}_{4\times 2}(\mathbb {A})),\ g \in R_Q(\mathbb {A}),\) write

$$\begin{aligned} I_0(\phi , g) = \int _{\mathbb {A}^{2}}\, d a\, \, d b\, [\omega _\psi (g)\phi ]\begin{pmatrix} a &{}\quad b \\ 1 &{}\quad 0 \\ 0 &{}\quad 1 \\ 0 &{}\quad 0 \end{pmatrix}\psi (-a). \end{aligned}$$
(10)

Finally, let w be the permutation matrix attached to the permutation

$$\begin{aligned} \left( \begin{array}{l@{\quad }l@{\quad }l@{\quad }l@{\quad }l@{\quad }l@{\quad }l@{\quad }l@{\quad }l@{\quad }l@{\quad }l@{\quad }l} 1&{}2&{}3&{}4&{}5&{}6&{}7&{}8&{}9&{}10&{}11&{}12\\ 7&{}10&{}11&{}12&{}4&{}5&{}8&{}9&{}1&{}2&{}3&{}6 \end{array}\right) , \end{aligned}$$
(11)

and let \(U^w_Q = U_Q \cap w^{-1} Pw.\) Then the global integral (7) is equal to

$$\begin{aligned} \int _{Z(\mathbb {A})U_4(\mathbb {A})\backslash C_Q(\mathbb {A})} W_\varphi (g) \int _{U_Q^w(\mathbb {A}) \backslash U_Q(\mathbb {A})} f_{\chi ;s}(wug)I_0(\phi , ug)\, du \, dg. \end{aligned}$$
(12)

Remark 3.2

The permutation matrix w represents an element of the Weyl group of G relative to \(T_G.\) We also record an expression for w as reduced product of simple reflections. We also introduce some notation for elements of the Weyl group. We write w[i] for the simple reflection attached to the simple root \(\alpha _i,\) and \(w[i_1i_2\ldots i_k]\) for the product \(w[i_1]w[i_2]\ldots w[i_k].\) Then \(w=w[64321465432465434654].\)

Before proceeding to the proof, we need to know the structure of the set \(P\backslash G \slash R_Q.\)

3.1 Description of the double coset space \(P\backslash G \slash R_Q\)

Clearly, the identity map \(G \rightarrow G\) induces a map \({\text {pr}}: P \backslash G / R_Q \rightarrow P \backslash G/ Q.\) Each element of \(P \backslash G/ Q\) contains a unique element of the Weyl group which is of minimal length. Recall that the group of permutation matrices which are contained in G maps isomorphically onto the Weyl group of G. A Weyl element of minimal length in its PQ double coset corresponds to a permutation \(\sigma : \{ 1, \ldots , 12\} \rightarrow \{ 1, \ldots , 12\}\) such that

  • \(\sigma (13-i) = 13-\sigma (i) , \forall i, \)

  • \(\sigma \) is an even permutation.

  • If \( 1\le i<j \le 4, \; 5\le i<j \le 8,\) or \(9 \le i < j \le 12,\) and if \(\{ i,j\} \ne \{ 6,7\},\) then \(\sigma (i) < \sigma (j).\)

  • If \(1 \le i< j \le 3,\;4 \le i< j \le 6,\;7 \le i < j \le 9,\) or \(10 \le i < j \le 12,\) then \(\sigma ^{-1}(i) < \sigma ^{-1}(j).\)

Such a permutation \(\sigma \) is determined by the quadruple

$$\begin{aligned} ( \#(\{ 1,2,3,4\}\cap \sigma ^{-1}(\{ 3i-2, 3i-1, 3i\})))_{i=1}^4. \end{aligned}$$

Deleting any zeros in this tuple gives the ordered partition of 4 corresponding to the standard parabolic subgroup \(P_\sigma :=GL_4 \cap \sigma ^{-1}P \sigma \) (Here we identify the permutation \(\sigma \) with the corresponding permutation matrix, which is in \(GSO_{12},\) and identify \(g \in GL_4\) with \({\text {diag}}(g, I_4, g^*) \in GSO_{12}\)). Now, for any parabolic subgroup \(P_o\) of \(GSO_4,\) we have \(GSO_4 = P_oSO_4.\) It follows that \(g \mapsto \sigma {\text {diag}}(g, I_4, g^*)\) induces a bijection \(P_\sigma \backslash GL_4 / GSp_4 \leftrightarrow {\text {pr}}^{-1}(P \cdot \sigma \cdot Q) \subset P \backslash G/ R_Q.\) Therefore we must study the space \(P' \backslash GL_4 / GSp_4,\) where \(P'\) is an arbitrary parabolic subgroup of \(GL_4.\)

Lemma 3.3

Let S be a subset of the set of simple roots in the root system of type \(A_3.\) Let \(P_S, P'_S\) denote the standard parabolic subgroups \(GL_4,\) and \(SO_6,\) respectively, corresponding to S. Then \(P_S \backslash GL_4/GSp_4\) and \(P'_S \backslash SO_6 / SO_5\) are in canonical bijection.

Proof

This follows from considering the coverings of \(SO_6\) and \(GL_4\) by the group \(GSpin_6\) which are described in [20] and section 2.3 of [1], respectively. The preimage of \(SO_5\) in \(GSpin_6\) is \(GSpin_5 = GSp_4.\) Since the kernels of both projections are contained in the central torus of \(GSpin_6,\) which is contained in any parabolic subgroup of \(GSpin_6\) it follows that both \(P_S \backslash GL_4/GSp_4\) and \(P'_S \backslash SO_6 / SO_5\) are in canonical bijection with \(P_S'' \backslash GSpin_6 / GSpin_5,\) where \(P_S''\) is the parabolic subgroup of \(GSpin_6\) determined by S. \(\square \)

Now, in considering \(SO_6/SO_5,\) we embed \(SO_5\) into \(SO_6\) as the stabilizer of a fixed anisotropic element \(v_0\) of the standard representation of \(SO_6.\) Then \(P_S'' \backslash SO_6/ SO_5\) may be identified with the set of \(P_S'\)-orbits in \(SO_6\cdot v_0.\) For concreteness, take \(SO_6\) to be defined using the quadratic form associated to the matrix \(J_6,\) and take \(v_0 = \, ^t[0,0,1,1,0,0].\) The \(SO_6\) orbit of \(v_0\) is the set of vectors satisfying \(^tv\cdot J_6 \cdot v = {}\,^t\! v_0 \cdot J_6 \cdot v = 2.\) Note that each of the permutation matrices representing a simple reflection attached to an outer node in the Dynkin diagram maps \(v_0\) to \(v_1:=\,^t[0,1,0,0,1,0],\) and that a permutation matrix representing the simple reflection attached to the middle node of the Dynkin diagram maps \(v_1\) to \(v_2:=\,^t[1,0,0,0,0,1].\)

Lemma 3.4

Number the roots of \(SO_6\) so that \(\alpha _2\) is the middle root. (This is not the standard numbering for \(SO_6,\) but it matches the standard numbering for \(GL_4,\) and the numbering inherited as a subgroup of \(GSO_{12}\)). Write V for the standard representation of \(SO_6.\) The decomposition of \(SO_6 \cdot v_0\) into \(P_S'\) orbits is as follows:

S

\(Orbit\, reps\, in\, V\)

\(Double\, coset\, reps\)

\(\emptyset \)

\(v_0\), \(v_1\), \(v_2\)

ew[1], w[2]w[1]

{1}, { 3}, or {1, 3}

\(v_0\), \(v_2\)

ew[1]

{2}

\(v_0\), \(v_1\)

ew[2]w[1]

{1, 2} { 2, 3} or {1, 2, 3}

\(v_0\)

e

Proof

Direct calculation. \(\square \)

Remark 3.5

As elements of \(GSO_{12},\) the double coset representatives are identified with permutations of \(\{ 1,\ldots , 12\}.\) Writing these permutations in cycle notation, we have \(w[1] =(1,2)(11,12),\; w[2]w[1] = (1,3,2)(10,11,12).\) Replacing w[1] by w[3] in any of the representatives above produces a different element of the same double coset.

3.2 Proof of Theorem 3.1

We now apply this description of \(P \backslash G / R_Q,\) to the study of \(I(f_{\chi ;s}, \varphi , \phi ).\) For this section only, let \(w_0\) be the permutation matrix attached to (11), and let w be an arbitrary representative for \(P(F)\backslash G(F)\slash R_Q(F).\)

The global integral (7) is equal to

$$\begin{aligned}&\sum _{w\in P(F)\backslash G(F)\slash R_Q(F)} I_w(f_{\chi ;s}, \varphi , \phi ),\\ \text {where}&I_w(f_{\chi ;s}, \varphi , \phi )= \int _{Z(\mathbb {A})U_{Q}^{w}(F)C_{Q}^{w}(F)\backslash C_{Q}(\mathbb {A})U_{Q}(\mathbb {A})} f_{\chi ;s}(wug)\theta (\phi ,j(ug))\varphi (g)dg, \end{aligned}$$

where \(C_{Q}^{w}=C_{Q}\cap w^{-1}Pw\), and \(U_{Q}^{w}=U_Q \cap w^{-1} P w\).

Proposition 3.6

If w does not lie in the double coset containing \(w_0,\) then \(I_w(f_{\chi ;s}, \varphi _1, \varphi _2)=0.\) Consequently, \(I(f_{\chi ;s}, \varphi _1, \varphi _2) = I_{w_0}(f_{\chi ;s}, \varphi _1, \varphi _2).\)

Proof

Write \(w = \sigma \nu \) where w is a permutation of \(\{ 1, \ldots 12\}\) satisfying the four conditions listed at the beginning of Sect. 3.1, and \(\nu \) is one of the representatives for \(P_\sigma \backslash M_Q/ C_Q\) given in the table in Lemma 3.4. The integral \(I_w( f_{\chi ;s}, \varphi , \phi )\) vanishes if \(\psi _l\) is nontrivial on \(U_Q^w:=U_Q \cap w^{-1} P w,\) or equivalently, if the character \(\nu \cdot \psi _l\) obtained by composing \(\psi _l\) with conjugation by \(\nu \) is nontrivial on \(U_Q \cap \sigma ^{-1} P \sigma .\) For our representatives \(\nu ,\) we have

There are 25 possibilities for \(\sigma .\) However, it’s clear that \(I_w( f_{\chi ;s}, \varphi , \phi )\) vanishes, regardless of \(\nu ,\) if \(\sigma (1) < \sigma (9),\) or if \(\sigma (2) < \sigma (10).\) This eliminates all but seven possibilities for \(\sigma .\) For the remaining seven, the above criterion shows that \(I_w( f_{\chi ;s}, \varphi , \phi )\) vanishes unless \(\nu \) is trivial.

Assume now that \(\psi _l\) is trivial on \(U_Q^w.\) This means that the image of \(U_Q^w\) in the Heisenberg group intersects the center trivially, and maps to an isotropic subspace of the quotient \(\mathcal H_{17}/Z(\mathcal H_{17})\) (which has the structure of a symplectic vector group). Write V for this subspace and \(V^\perp \) for its perp space. Define \(P_V\subset Sp_{16}\) as in Lemma 2.2, and let \(P_V^1\) denote the kernel of the canonical projection \(P_V \rightarrow GL_1 \times Sp( V^\perp /V).\) It follows immediately from Lemma 2.2 and the cuspidality of \(\varphi \) that \(I_w( f_{\chi ;s}, \varphi , \phi )\) vanishes whenever \(P_V^1 \cap C_Q\) contains the unipotent radical of a proper parabolic subgroup of \(C_Q.\) This applies to each of the remaining double coset representatives, except for \(w_0.\) \(\square \)

The following lemma is useful in our calculation.

Lemma 3.7

Let \(f_{1}\), \(f_{2}\) be two continuous functions on \((F\backslash \mathbb {A})^{n}\), and \(\psi \) a nontrivial additive character on \(F\backslash \mathbb {A}\). Then

$$\begin{aligned} \int _{(F\backslash \mathbb {A})^{n}}\, d x\, f_{1}(x)f_{2}(x)=\sum _{\alpha \in F^{n}}\int _{(F \backslash \mathbb {A})^{n}}\, d x\, f_{1}(x)\psi (\alpha \cdot x)\int _{(F \backslash \mathbb {A})^{n}}\, d y\, f_{2}(y)\psi ^{-1}(\alpha \cdot y). \end{aligned}$$
(13)

Moreover, if \(\int _{(F\backslash \mathbb {A})^{n}}\, d x\, f_{1}(x)=0\), then one can replace \(\sum _{\alpha \in F^{n}}\) by \(\sum _{\alpha \in F^{n}-\{0\}}\) in the formula above.

Proof

By Fourier theory on \(F\backslash \mathbb {A}\),

$$\begin{aligned} f_{i}(x)=\sum _{\alpha \in F^{n}} \psi (-\alpha \cdot x)\hat{f}_{i}(\alpha ), \end{aligned}$$

where \(\hat{f}_{i}(\alpha )=\int _{(F\backslash \mathbb {A})^{n}}\, d x\, f_{1}(x)\psi (\alpha x)\) for \(i=1,2\). So the left hand side of (13) is equal to

$$\begin{aligned} \sum _{\alpha ,\beta \in F^{n}}\hat{f}_{1}(\alpha )\hat{f}_{2}(\beta )\int _{(F\backslash \mathbb {A})^{n}}\, d x\, \psi (-(\alpha +\beta )\cdot x). \end{aligned}$$
(14)

The integral on x vanishes when \(\alpha +\beta \ne 0\), and equals 1 if \(\alpha +\beta =0\), so (14) equals

$$\begin{aligned} \sum _{\alpha \in F^{n}}\hat{f}_{1}(\alpha )\hat{f}_{2}(-\alpha ), \end{aligned}$$

which is the right hand side of (13). When \(\int _{(F\backslash \mathbb {A})^{n}}\, d x\, f_{1}(x)=0\), we have \(\hat{f}_{1}(0)=0\), so we can replace \(\sum _{\alpha \in F^{n}}\) by \(\sum _{\alpha \in F^{n}-0}.\) \(\square \)

From now on, let \(w=w[64321465432465434654].\) Then

$$\begin{aligned}&U_{Q}^{w}:=U_Q \cap w^{-1} P w= \left\{ u_{Q}^{w}(y_{7},y_{8})= u_Q\left( 0,\; \begin{pmatrix} y_8&{}\quad y_7 \\ 0&{}\quad 0\\ 0&{}\quad 0\\ 0&{}\quad 0 \end{pmatrix} ,\; 0 \right) : y_{7},y_{8}\in F\right\} , \\&C_{Q}^{w}:=C_Q \cap w^{-1} P w= (P_{1}\times P_{2})^{\circ } \nonumber \end{aligned}$$
(15)

where \(P_{1}\) is the Klingen parabolic subgroup of \(GSp_{4}\) and \(P_{2}\) is the Siegel parabolic subgroup of \(GSO_{4}\). Let \(P_{1}=M_{1}\ltimes U_{1}\) and \(P_{2}=M_{2}\ltimes U_{2}\) be their Levi decompositions. Note that \(f_{\chi ;s}(wug)=f_{\chi ;s}(wg)\) for all \(u\in U_{Q}^{w}.\) So, by Proposition 3.6, \(I(f_{\chi ;s}, \varphi , \phi )\) is equal to

$$\begin{aligned} \int _{Z(\mathbb {A})C_Q^w(F) \backslash C_Q(\mathbb {A})} \varphi (g) \int _{U_Q^w(\mathbb {A})\backslash U(\mathbb {A})} f_{\chi ;s}(wu_2g) \int _{[U_Q^w]} \theta (\phi , j(u_1u_2g))\, du_1\, du_2\, dg. \end{aligned}$$
(16)

But, for \(u=u_{Q}^{w}(y_{7},y_{8})\) (defined in (15)),

$$\begin{aligned}{}[\omega _\psi (j(u))\phi _1](\xi )=\phi _1(\xi )\psi (\xi _{7}y_{7}+\xi _{8}y_{8}), \qquad \xi = \begin{pmatrix} \xi _1 &{}\quad \xi _2 \\ \xi _3&{}\quad \xi _4 \\ \xi _5 &{}\quad \xi _6 \\ \xi _7 &{}\quad \xi _8 \end{pmatrix}, \end{aligned}$$
(17)

for any \(\phi _1 \in \mathcal S({\text {Mat}}_{4 \times 2}(\mathbb {A})).\) It follows that (16) is equal to

$$\begin{aligned} \int _{Z(\mathbb {A})C_Q^w(F) \backslash C_Q(\mathbb {A})} \varphi (g) \int _{U_Q^w(\mathbb {A})\backslash U(\mathbb {A})} f_{\chi ;s}(wug) \theta _0(\phi , j(ug))\, du\, dg, \end{aligned}$$
(18)

where

$$\begin{aligned} \theta _0(\phi , u \widetilde{g}) := \sum _{\xi \in {\text {Mat}}_{3\times 2}(F)} [\omega _\psi ( u \widetilde{g}).\phi ] \begin{pmatrix} \xi \\ 0 \end{pmatrix}, \qquad (u \in \mathcal H_{17}(\mathbb {A}),\ \widetilde{g} \in \widetilde{Sp}_{16}(\mathbb {A})) \end{aligned}$$

Now, \(C_{Q}^{w}=(M_{1}\times M_{2})^{\circ }\ltimes (U_{1}\times U_{2})\), and \(f_{\chi ;s}(wu_{1}u_{2}g)=f_{\chi ;s}(wg)\), for any \(u_{1}\in U_{1}\), \(u_{2}\in U_{2}\), and \(g\in G.\) Moreover, if

$$\begin{aligned} U_{2}(a)= \begin{pmatrix} 1 &{}\quad &{}\quad a &{}\quad \\ &{}\quad 1 &{}\quad &{}\quad -a\\ &{}\quad &{}\quad 1&{}\quad \\ &{}\quad &{}\quad &{}\quad 1 \end{pmatrix}, \end{aligned}$$
(19)

then \([\omega _\psi (U_{2}(a)u_{1}g)\phi ] \begin{pmatrix} \xi _{1}&{}\quad \xi _{2}\\ \xi _{3} &{}\quad \xi _{4} \\ \xi _{5} &{}\quad \xi _{6}\\ 0 &{}\quad 0 \end{pmatrix} =\psi (a(\xi _{3}\xi _{6}-\xi _{4}\xi _{5}))[\omega _\psi (u_{1}g)\phi ] \begin{pmatrix} \xi _{1}&{}\quad \xi _{2}\\ \xi _{3} &{}\quad \xi _{4} \\ \xi _{5} &{}\quad \xi _{6}\\ 0 &{}\quad 0 \end{pmatrix}\). It then follows from the cuspidality of \(\varphi \) that (18) is equal to

$$\begin{aligned} \int _{Z(\mathbb {A})C^w_Q(F) \backslash C_Q(\mathbb {A})} \varphi (g) \int _{U_Q^w(\mathbb {A})\backslash U(\mathbb {A})} f_{\chi ;s}(wug) \theta _1(\phi , j(ug))\, du\, dg, \end{aligned}$$
(20)

where

$$\begin{aligned} \theta _1(\phi , u \widetilde{g}) := \sum _{\xi \in {\text {Mat}}_{3\times 2}(F):\ (\xi _{3}\xi _{6}-\xi _{4}\xi _{5})\ne 0} [\omega _\psi ( u \widetilde{g}).\phi ] \begin{pmatrix} \xi \\ 0 \end{pmatrix}, \qquad (u \in \mathcal H_{17}(\mathbb {A}),\ \widetilde{g} \in \widetilde{Sp}_{16}(\mathbb {A})). \end{aligned}$$

The group \((M_{1}\times M_{2})^{\circ }\) is the set of all

$$\begin{aligned} m(g_3, g_4, t):= diag(t\det g_{3},g_{3},t^{-1};\det g_{3} g_{4},g_{4}^{*};t \det g_{3},\det g_{3}\cdot g_{3}^{*},t^{-1}) \end{aligned}$$
(21)

where \(g_{3}\in GL_{2}\), \(g_{4}\in GL_{2}\) and \(t\in GL_{1}\). Note that the summation over \((\xi _{1},\xi _{2})\) is invariant under the action of \((M_{1}\times M_{2})^{\circ }\). Consider the action of \((M_{1}\times M_{2})^{\circ }\) on \(\lbrace (\xi _{3},\xi _{4},\xi _{5},\xi _{6})\mid \det \begin{pmatrix} \xi _{3} &{}\quad \xi _{4} \\ \xi _{5} &{}\quad \xi _{6} \end{pmatrix} \ne 0\rbrace \). It is not hard to see that it is transitive, and the stabilizer of (1, 0, 0, 1) is \(\lbrace m(t,g_{3},g_{4})\mid g_{4}= g_{3}\cdot \det (g_{3})^{-1}\rbrace \), which is the same as \(\lbrace M_{5}(t,g_{3})=diag(t \det g_{3},g_{3},t^{-1}; g_{3},g_{3}^{*}\det g_{3}; t \det g_{3}, g_{3}^{*}\det g_{3}, t^{-1}): g_3 \in GL_2, \ t \in GL_1\rbrace \). We denote this group by \(M_{5}.\) Let \(\psi _{U_{2}}\) be a character on \(U_{2}\) defined by \(\psi _{U_{2}}(U_{2}(a))=\psi (a)\), then Eq. (20) is equal to

$$\begin{aligned} \int _{Z(\mathbb {A})M_5(F)U_1(F)U_2(\mathbb {A})\backslash C_Q(\mathbb {A})} \varphi ^{(U_2,\psi _{U_2})}(g) \int _{U_Q^w(\mathbb {A}) \backslash U(\mathbb {A})} f_{\chi ;s}(wug) \theta _2(j(ug)) \, du \, dg, \end{aligned}$$
(22)

where

$$\begin{aligned} \theta _2(j(ug)):= \sum _{(\xi _1, \xi _2) \in F^2} [\omega _\psi (j(ug)).\phi ]\begin{pmatrix} \xi _{1}&{}\quad \xi _{2}\\ 1 &{}\quad 0 \\ 0 &{}\quad 1\\ 0 &{}\quad 0 \end{pmatrix}, \end{aligned}$$

and the notation \(\varphi ^{(U_2, \psi _{U_2})}\) is defined as follows. For any unipotent subgroup V of an F-group H,  character \(\vartheta \) of V,  and smooth left V(F)-invariant function \(\Phi \) on \(H(\mathbb {A}),\) we define

$$\begin{aligned} \Phi ^{(V,\vartheta )}(h) : = \int _{[V]}\Phi (vh) \vartheta (v) \, dv. \end{aligned}$$

Now, \(U_{1}\) consists of elements

$$\begin{aligned} U_{1}(a,b,c)= \begin{pmatrix} 1 &{}\quad a &{}\quad b &{}\quad c \\ &{}\quad 1 &{}\quad &{}\quad b \\ &{}\quad &{}\quad 1 &{}\quad -a \\ &{}\quad &{}\quad &{}\quad 1 \end{pmatrix}\in GSp_{4}, \end{aligned}$$
(23)

and for any \(g\in R_{Q}\),

$$\begin{aligned}{}[\omega _{\psi }(U_{1}(0,0,c)g)\phi ] \begin{pmatrix} \xi _{1}&{}\quad \xi _{2}\\ 1 &{}\quad 0 \\ 0 &{}\quad 1\\ 0 &{}\quad 0 \end{pmatrix}=[\omega _{\psi }(g)\phi ] \begin{pmatrix} \xi _{1}&{}\quad \xi _{2}\\ 1 &{}\quad 0 \\ 0 &{}\quad 1\\ 0 &{}\quad 0 \end{pmatrix}. \end{aligned}$$

Factoring the integration over \(U_{1}\) and applying Lemma 3.7 to functions

$$\begin{aligned} (a,b)\mapsto \omega _\psi (U_{1}(a,b,0)g)\phi \begin{pmatrix} \xi _{1}&{}\quad \xi _{2}\\ 1 &{}\quad 0 \\ 0 &{}\quad 1\\ 0 &{}\quad 0 \end{pmatrix} \quad \text {and}\quad (a,b)\mapsto \int _{F\backslash \mathbb {A}}\, d c\, \varphi _{2}(U_{1}(a,b,c)g), \end{aligned}$$

we deduce that (22) is equal to

$$\begin{aligned} \int \limits _{Z(\mathbb {A})M_5(F)U_1(\mathbb {A})U_2(\mathbb {A})\backslash C_Q(\mathbb {A})} \varphi ^{(U_3,\psi _{U_3}^{\alpha ,\beta })}(g) \int \limits _{U_Q^w(\mathbb {A}) \backslash U(\mathbb {A})} f_{\chi ;s}(wug) \theta _2^{(U_1, \psi _{U_1}^{-\alpha ,-\beta })}(j(ug))\, du \, dg, \end{aligned}$$
(24)

where \(\psi _{U_1}^{\alpha ,\beta }(U_1(a,b,c)) = \psi (\alpha a+ \beta b),\) \(U_3=U_1U_2,\) and \(\psi _3^{\alpha ,\beta }=\psi _{U_2} \psi _{U_1}^{\alpha , \beta }.\) The group \(M_{5}(F)\) acts on \(U_1(\mathbb {A})\) and permutes the nontrivial characters \(\psi _{U_1}^{\alpha ,\beta }\) transitively. The stabilizer of \(\psi _{U_1}:= \psi _{U_1}^{1,0}\) is

$$\begin{aligned} M_6:=\left\{ M_6(a_{1},a_{2},a_{4}) \right\} ,\qquad \text { where }\quad M_6(a_{1},a_{2},a_{4})= M_{5}\left( a_{4}^{-1}, \begin{pmatrix} a_{1} &{}\quad a_{2} \\ 0 &{}\quad a_{4} \end{pmatrix}\right) . \end{aligned}$$
(25)

Hence Eq. (24) is equal to

$$\begin{aligned} \int \limits _{Z(\mathbb {A})M_6(F)U_1(\mathbb {A})U_2(\mathbb {A})\backslash C_Q(\mathbb {A})} \varphi ^{(U_3,\psi _{U_3})}(g) \int \limits _{U_Q^w(\mathbb {A}) \backslash U(\mathbb {A})} f_{\chi ;s}(wug) \theta _2^{(U_1, \overline{\psi }_{U_1})}(j(ug))\, du \, dg, \end{aligned}$$
(26)

where \(\psi _{U_3} = \psi _{U_1}\psi _{U_2}.\)

Note that

$$\begin{aligned}{}[\omega _\psi (U_{1}(a,b,0)g)\phi ] \begin{pmatrix} \xi _{1} &{}\quad \xi _{2}\\ 1 &{}\quad 0 \\ 0 &{}\quad 1\\ 0 &{}\quad 0 \end{pmatrix}=[\omega _\psi (g)\phi ] \begin{pmatrix} \xi _{1}+a &{}\quad \xi _{2}+b\\ 1 &{}\quad 0 \\ 0 &{}\quad 1\\ 0 &{}\quad 0 \end{pmatrix}, \end{aligned}$$

and that for \(\xi _{1},\xi _{2}\in F\), \(\psi (\alpha \cdot (a+\xi _{1})+\beta \cdot (b+\xi _{2}))=\psi (\alpha \cdot a+\beta \cdot b)\). We can combine the summation on \((\xi _{1},\xi _{2})\) with the integral over (ab). It follows that \(\theta _2^{(U_1, \overline{\psi }_{U_1})}(g)=I_0(\phi , g),\) defined in (10). Let \(M_{6}= U_{6}T_{6}\) be the Levi decomposition. It is not hard to see that both \(I_0(\phi , g)\) and the function \(g\mapsto f(wg)\) are invariant on the left by \(U_6(\mathbb {A}).\) So, (26) is equal to

$$\begin{aligned} \int \limits _{Z(\mathbb {A})T_6(F)U_1(\mathbb {A})U_2(\mathbb {A})\backslash C_Q(\mathbb {A})} \varphi ^{(U_4,\psi _{U_4})}(g) \int \limits _{U_Q^w(\mathbb {A}) \backslash U(\mathbb {A})} f_{\chi ;s}(wug) \theta _2^{(U_1, \overline{\psi }_{U_1})}(j(ug))\, du \, dg, \end{aligned}$$
(27)

where \(U_4 = U_3U_6,\) and \(\psi _{U_4}\) is the extension of \(\psi _{U_3}\) to a character of \(U_4\) which is trivial on \(U_6.\) Now,

$$\begin{aligned} \varphi ^{(U_4,\psi _{U_4})}(g) = \int _{F \backslash \mathbb {A}} \varphi _1^{(U_1, \psi _1)}\left( \begin{pmatrix} 1&{}\quad &{}\quad &{}\quad \\ &{}\quad 1&{}\quad r&{}\quad \\ &{}\quad &{}\quad 1&{}\quad \\ &{}\quad &{}\quad &{}\quad 1 \end{pmatrix} g_1\right) \varphi _2^{(U_2,\psi _2)}\left( \begin{pmatrix} 1&{}\quad r&{}\quad &{}\quad \\ &{}\quad 1&{}\quad &{}\quad \\ &{}\quad &{}\quad 1&{}\quad -r\\ &{}\quad &{}\quad &{}\quad 1 \end{pmatrix} g_2\right) . \end{aligned}$$

Let \(N_1\) denote the standard maximal unipotent subgroup of \(GSp_4\) and \(N_2\) that of \(GSO_4.\) Let \(\psi _{N_1}^\gamma \) and \(\psi _{N_2}^\gamma \) be the extensions of \(\psi _{U_1}\) and \(\psi _{U_2}\) to characters of \(N_1(\mathbb {A})\) and \(N_2(\mathbb {A})\) respectively, such that

$$\begin{aligned} \psi _{N_1}^ \gamma \begin{pmatrix} 1&{}\quad &{}\quad &{}\quad \\ &{}\quad 1&{}\quad r&{}\quad \\ &{}\quad &{}\quad 1&{}\quad \\ &{}\quad &{}\quad &{}\quad 1\end{pmatrix} =\psi _{N_2}^\gamma \begin{pmatrix} 1&{}\quad r&{}\quad &{}\quad \\ &{}\quad 1&{}\quad &{}\quad \\ &{}\quad &{}\quad 1&{}\quad -r\\ &{}\quad &{}\quad &{}\quad 1 \end{pmatrix} = \psi (\gamma r). \end{aligned}$$

Then it follows from Lemma 3.7 (and the cuspidality of \(\varphi _1, \varphi _2,\)) that

$$\begin{aligned} \varphi ^{(U_4,\psi _{U_4})}(g)= \sum _{\gamma \in F^\times } \varphi _1^{(N_1,\psi _{N_1}^\gamma )}(g_1) \varphi _2^{(N_2, \psi _{N_2}^{-\gamma })}(g_2) = \varphi ^{(N, \psi _N^\gamma )}(g), \end{aligned}$$

where \(N=N_1N_2,\) a maximal unipotent subgroup of \(C_Q,\) and for \(\gamma \in F^\times ,\) \( \psi _N^\gamma = \psi _{N_1}^\gamma \psi _{N_2}^{-\gamma }.\) We plug this in to (27). The group \(T_6\) acts on the characters \(\psi _N^\gamma \) transitively, and the stabilizer of \(\psi _N : = \psi _N^1\) is the center of \(C_Q.\) Since \(\varphi ^{(N, \psi _N)}(g) = W_\varphi (g),\) this completes the Proof of Theorem 3.1.

4 Preparation for the unramified calculation

In this section, we establish some results which describe the structure of the symmetric algebras of some representations of \(Sp_4\times SL_2,\) and \(Sp_4 \times SL_2 \times SL_2,\) which will be used to relate our local zeta integrals to products of Langlands L-functions.

We first consider some representations of \(Sp_4 \times SL_2.\) Let \(\varpi _1\) and \(\varpi _2\) denote the fundamental weights of \(Sp_4\) and \(\varpi \) that of \(SL_2.\) Write \(V_{(n_1, n_2; m)}\) for the irreducible \(Sp_4 \times SL_2\)-module with highest weight \(n_1\varpi _1+n_2\varpi _2 + m\varpi ,\) and let \([n_1, n_2; m]\) denote its trace.

Proposition 4.1

For \(i,j,n_1, n_2\) and m all non-negative integers, let \(\mu _{i,j}(n_1,n_2;m)\) denote the multiplicity of \(V_{(n_1, n_2; m)}\) in \({\text {sym}}^i V_{(1,0;1)}\otimes {\text {sym}}^j V_{(1,0;0)}.\) Then

$$\begin{aligned} \begin{aligned}&\sum _{i,j,n_1,n_2,m=0}^\infty \mu _{i,j}(n_1,n_2; m) t_1^{n_1}t_2^{n_2} t_3^m x^i y^j =\\&\frac{1-t_1t_2t_3 x^3 y^2}{(1-t_1t_3x)(1-x^2)(1-t_2x^2)(1-t_1y)(1-t_3xy)(1-t_2t_3xy)(1-t_1x^2y)(1-t_2x^2y^2)}.\end{aligned} \end{aligned}$$

Proof

We first describe \({\text {sym}}^j V_{(1,0;1)}.\) Write \(V_{(n_1,n_2)}\) for the irreducible \(Sp_4\)-module with highest weight \(n_1\varpi _1+n_2\varpi _2.\) Then we may regard \(V_{(1,0;1)}\) as two copies of \(V_{(1,0)}\) with the standard torus of \(SL_2\) acting on them by eigenvalues, say, \(\eta \) and \(\eta ^{-1}.\) Then, using the well known fact that \({\text {sym}}^kV_{(1,0)}=V_{(j,0)},\) and the decomposition of \(V_{(m,0)}\otimes V_{(n,0)}\) described in [23],

$$\begin{aligned} {\text {Tr}}{\text {sym}}^nV_{(1,0;1)} = \sum _{n_1 =0}^n \eta ^{n-2n_1} \sum _{\ell = 0}^{\min (n_1, n-n_1)} \sum _{j=0}^\ell [n-2\ell ,j] = \sum _{\ell =0}^{\lfloor \frac{n}{2} \rfloor } \sum _{j=0}^\ell [n-2\ell ,j; n-2\ell ], \end{aligned}$$

whence

$$\begin{aligned} \sum _{n=0}^\infty x^n {\text {Tr}}{\text {sym}}^n V_{(1,0;1)} = \sum _{j,k,n=0}^\infty x^{n+2j+2k} [n,j;n] = \frac{1}{1-x^2} \sum _{j,n=0}^\infty [n,j;n] x^{n+2j}. \end{aligned}$$

Using [23] again to compute \(V_{(n,j)}\otimes V_{(m,0)}\) one obtains

$$\begin{aligned}&\sum _{n,m,j=0}^\infty [n,j][m,0] t^n x^{n+2j}y^m \\&\quad = \frac{1}{1-txy} \sum _{\begin{array}{c} {n,j,i_2,k=0}\\ {n+i_2 \ge k} \end{array}}^\infty [n+m+i_2-k_2, j+k_2] t^{n}x^{n+2j+2i_2}y^{m+i_2+k}. \end{aligned}$$

It follows that

$$\begin{aligned} \begin{aligned}&\sum _{i,j,n_1,n_2,m=0}^\infty \mu _{i,j}(n_1,n_2; m) t_1^{n_1}t_2^{n_2} t_3^m x^i y^j\\ {}&\quad = \frac{1}{1-x^2}\frac{1}{1-t_3xy} \sum _{\begin{array}{c} {n_1,n_2,i_2,k,m=0}\\ {n+i_2 \ge k} \end{array}}^\infty t_1^{n_1+m+i_2-k}t_2^{n_2+k} t_3^{n_1}x^{n_1+2n_2+2i_2}y^{m+i_2+k}\\&\quad = \frac{1}{1-x^2}\frac{1}{1-t_3xy}\frac{1}{1-t_2x^2} \frac{1}{1-t_1y} \sum _{\begin{array}{c} {n_1,i_2,k=0}\\ {n+i_2 \ge k} \end{array}}^\infty t_1^{n_1+i_2-k}t_2^{k} t_3^{n_1}x^{n_1+2n_2+2i_2}y^{i_2+k}, \end{aligned} \end{aligned}$$

and the result then follows from the identity

$$\begin{aligned} \sum _{n_1=0}^\infty \sum _{n_2=0}^\infty \sum _{j,k=0}^{n_2} u^{n_2} v^k w^j = \frac{1-u^2 vw}{(1-u)(1-uv)(1-uw)(1-uvw)}. \end{aligned}$$

\(\square \)

Corollary 4.2

For \(n=(n_1, n_2, n_3)\) let \([n]=[n_1,n_2;n_3],\) and, let

$$\begin{aligned} \begin{matrix} a= {}^t\begin{bmatrix} 1&{}\quad 0&{}\quad 1&{}\quad 1&{}\quad 2&{}\quad 2&{}\quad 2 \end{bmatrix}\\ b={}^t \begin{bmatrix} 0&{}\quad 1&{}\quad 1&{}\quad 1&{}\quad 0&{}\quad 1&{}\quad 2 \end{bmatrix} \end{matrix}, \qquad g= {}^t \begin{bmatrix} 1&\quad 1&\quad 0&\quad 0&\quad 0&\quad 1&\quad 0\\ 0&\quad 0&\quad 0&\quad 1&\quad 1&\quad 0&\quad 1\\ 1&\quad 0&\quad 1&\quad 1&\quad 0&\quad 0&\quad 0 \end{bmatrix} \end{aligned}$$

Then \( \sum _{i=0}^\infty x^i {\text {Tr}}{\text {sym}}^i V_{(1,0;1)} \sum _{j=0}^\infty y^j {\text {Tr}}{\text {sym}}^j V_{(1,0;0)}\) equals

$$\begin{aligned} \frac{1}{1-x^2} \left[ \sum _n [n\cdot g] x^{n \cdot a } y^{n \cdot b} - \sum _n [n\cdot g+(1,1,1)] x^{n \cdot a +3} y^{n \cdot b+2}\right] , \end{aligned}$$

where n is summed over row vectors \(n=(n_1, \ldots , n_7)\in \mathbb {Z}_{\ge 0}^7.\)

Our next result describes the decomposition of \({\text {sym}}^i V_{(1,0;1)} \otimes {\text {sym}}^j V_{(1,0;0)} \otimes {\text {sym}}^k V_{(1,0;0)}.\) It is an identity of rational functions in 6 variables. To keep the notation short, we often reflect dependence only on arguments which will vary. Let

$$\begin{aligned} \begin{aligned} d(t_1, t_2)&= (1-t_1t_3x)(1-t_2x^2)(1-t_1y)(1-t_3xy)(1-t_2t_3xy)(1-t_1x^2y)(1-t_2x^2y^2)\\&= \left[ \sum _{n \in \mathbb {Z}_{\ge 0}^7 } t_1^{n\cdot g_1}t_2^{n\cdot g_2}t_3^{n\cdot g_3} x^{n \cdot a} y^{n\cdot b}\right] ^{-1}, \end{aligned} \end{aligned}$$

where \(g_1, g_2\) and \(g_3\) are the three columns of the matrix g in Corollary 4.2. Define rational functions \(\gamma _1, \ldots , \gamma _7\) by

$$\begin{aligned}&\sum _{n_1=0}^{N_1} \sum _{n_2=0}^{N_2} \sum _{k=0}^{n_1+n_2} \!\! u^{n_1} v^{n_2} w^k\\&\quad = \gamma _1(u,v,w) + \gamma _2(u,v,w) u^{N_1} + \gamma _3(u,v,w)v^{N_2} +\gamma _4(u,v,w)u^{N_1}v^{N_2}\\&\quad \quad +\gamma _5(u,v,w)(uw)^{N_1}+\gamma _6(u,v,w) (vw)^{N_2} +\gamma _7(u,v,w)(uw)^{N_1}(vw)^{N_2}, \end{aligned}$$

and let \(c_i =\gamma _i(t_1/z, t_1 z/t_2, t_2z/t_1).\)

Proposition 4.3

Let \(\mu _{i,j,k}(n_1, n_2; m)\) denote the multiplicity of \(V_{(n_1, n_2; m)}\) in \({\text {sym}}^i V_{(1,0;1)} \otimes {\text {sym}}^j V_{(1,0;0)} \otimes {\text {sym}}^k V_{(1,0;0)}.\) Then

$$\begin{aligned} \begin{aligned}&\sum _{i,j,k, n_1,n_2,m=0}^\infty \mu _{i,j,k}(n_1, n_2; m) t_1^{n_1} t_2^{n_2} t_3^m x^i y^j z^k = (1-x^2)^{-1}(1-t_1z)^{-1}\\&\quad \quad \times \frac{c_1 \nu ( t_2z)}{d(z,t_2)} +\frac{c_2 \nu ( t_1t_2)}{d(t_1,t_2)} +\frac{c_3 \nu ( t_1z^2)}{d(z,t_1z)} +\frac{c_4 \nu ( t_1^2z)}{d(t_1,t_1z,)} +\frac{c_5 \nu ( t_2^2z)}{d(t_2 z, t_2)}\\&\qquad +\frac{c_6 \nu ( t_2z^3)}{d(z, t_2z^2)} +\frac{c_7 \nu ( t_2^2z^3)}{d(t_2 z, t_2z^2)}, \end{aligned} \end{aligned}$$

where \(c_1, \ldots , c_7\) and d are as above and let \(\nu (u) = 1-ut_3x^3y^2.\)

Proof

From [23] again, one deduces that

$$\begin{aligned}{}[m_1,m_2]\cdot \sum _{\ell =0}^\infty [\ell ,0]x^\ell = \sum _{\ell =0}^\infty \sum _{i_1=0}^{m_1} \sum _{i_2=0}^{m_2} \sum _{k=0}^{i_1+i_2} [i_1+i_2-k+\ell , m_2-i_2+k] x^{\ell +m_1-i_1+i_2+k}. \end{aligned}$$

Combining with Corollary 4.2 gives

$$\begin{aligned} \begin{aligned}&\sum _{i,j,k,n_1,n_2,m=0}^\infty \mu _{i,j,k}(n_1, n_2; m) t_1^{n_1} t_2^{n_2} t_3^m x^i y^j z^k\\&\quad =\frac{1}{1-x^2}\frac{1}{1-t_1z} \sum _{n\in \mathbb {Z}_{\ge 0}^7} x^{n\cdot a} y^{n\cdot b} t_3^{n\cdot g_3}\left( \sum _{i_1=0}^{n\cdot g_1} \sum _{i_2=0}^{n\cdot g_2} \sum _{k=0}^{i_1+i_2} t_1^{i_1+i_2-k} t_2^{n\cdot g_2-i_2+k} z^{n\cdot g_1 -i_1+i_2+k}\right. \\&\quad \quad \left. - x^3y^2t_3 \sum _{i_1=0}^{n\cdot g_1+1} \sum _{i_2=0}^{n\cdot g_2+1} \sum _{k=0}^{i_1+i_2} t_1^{i_1+i_2-k} t_2^{n\cdot g_2+1-i_2+k} z^{n\cdot g_1+1 -i_1+i_2+k} \right) , \end{aligned} \end{aligned}$$

and and simplifying this rational function gives the result. \(\square \)

Proposition 4.4

Let \(V_{(n_1, n_2; n_3; n_4)}\) denote the irreducible representation of \(Sp_4 \times SL_2 \times SL_2\) such that \(Sp_4\) acts with highest weight \(n_1\varpi _1+n_2\varpi _2,\) the first \(SL_2\) acts with highest weight \(n_3,\) and the second \(SL_2\) acts with highest weight \(n_4.\) For \(n=(n_1, n_2; n_3;n_4),\) let \(\mu _{i,j}(n)\) denote the multiplicity of \(V_n\) in \({\text {sym}}^i V_{(1,0;1;0)} \otimes {\text {sym}}^j V_{(1,0;0;1)}.\) Then

$$\begin{aligned} \sum _{i,j=0}^\infty \sum _{n \in \mathbb {Z}_{\ge 0}^4} \mu _{i,j}(n) t_1^{n_1}t_2^{n_2}t_3^{n_3}t_4^{n_4} x^i y^j = \frac{\nu (x,y,t)}{\delta (x,y,t)}, \end{aligned}$$

where

$$\begin{aligned} \begin{aligned} \nu (x,y,t)&=1 - t_1 t_2 t_3 t_4^2 x^3 y^2 - t_1 t_2 t_3^2 t_4 x^2 y^3 - t_1^2 t_3 t_4 x^3 y^3 - t_1^2 t_2 t_3 t_4 x^3 y^3 - t_1 t_2 t_3^2 t_4 x^4 y^3 \\&\quad - t_1 t_2 t_3 t_4^2 x^3 y^4 - t_1^2 t_2 t_3^2 x^4 y^4 - t_1^2 t_2 t_4^2 x^4 y^4 + 2 t_1^2 t_2 t_3^2 t_4^2 x^4 y^4 - t_2^2 t_3^2 t_4^2 x^4 y^4 \\&\quad + t_1^3 t_2 t_3 t_4^2 x^5 y^4 + t_1 t_2^2 t_3^3 t_4^2 x^5 y^4 + t_1^3 t_2 t_3^2 t_4 x^4 y^5 + t_1 t_2^2 t_3^2 t_4^3 x^4 y^5 + t_1^2 t_2 t_3^3 t_4 x^5 y^5\\&\quad + t_1^2 t_2^2 t_3^3 t_4 x^5 y^5 + t_1^2 t_2 t_3 t_4^3 x^5 y^5 + t_1^2 t_2^2 t_3 t_4^3 x^5 y^5 + t_1^3 t_2 t_3^2 t_4 x^6 y^5 + t_1 t_2^2 t_3^2 t_4^3 x^6 y^5 \\&\quad + t_1^3 t_2 t_3 t_4^2 x^5 y^6 + t_1 t_2^2 t_3^3 t_4^2 x^5 y^6 - t_1^4 t_2 t_3^2 t_4^2 x^6 y^6 + 2 t_1^2 t_2^2 t_3^2 t_4^2 x^6 y^6 - t_1^2 t_2^2 t_3^4 t_4^2 x^6 y^6\\&\quad -t_1^2 t_2^2 t_3^2 t_4^4 x^6 y^6 - t_1^3 t_2^2 t_3^3 t_4^2 x^7 y^6 - t_1^3 t_2^2 t_3^2 t_4^3 x^6 y^7 - t_1^2 t_2^2 t_3^3 t_4^3 x^7 y^7\\&\quad -t_1^2 t_2^3 t_3^3 t_4^3 x^7 y^7 - t_1^3 t_2^2 t_3^2 t_4^3 x^8 y^7 - t_1^3 t_2^2 t_3^3 t_4^2 x^7 y^8 + t_1^4 t_2^3 t_3^4 t_4^4 x^{10} y^{10}, \end{aligned} \end{aligned}$$
$$\begin{aligned} \begin{aligned} \delta (x,y,t)&= (1-t_1 t_3 x) (1-x^2 ) (1-t_2 x^2 ) (1-t_1 t_4 y ) (1-y^2 ) (1-t_2 y^2 )(1-x^2 y^2 ) (1-t_3 t_4 x y ) \\&\quad \times (1-t_2 t_3 t_4 x y ) (1-t_1 t_4 x^2 y ) (1-t_1 t_3 x y^2 ) (1-t_1^2 x^2 y^2 )(1-t_2 t_3^2 x^2 y^2 ) (1-t_2 t_4^2 x^2 y^2) \end{aligned} \end{aligned}$$

Proof

Let p and q be the polynomials such that

$$\begin{aligned} \sum _{i,j,k=0}^\infty \mu _{i,j,k,n_1,n_2,m}(n_1, n_2; m) t_1^{n_1} t_2^{n_2} t_3^m x^i y^j z^k =\frac{p(x,y,z,t)}{q(x,y,z,t)}. \end{aligned}$$

They may be computed explicitly using Proposition 4.3. Set \(t'=(t_1,t_2,t_3),\) and

$$\begin{aligned} f(x,y,t',t_4)=\sum _{i,j=0}^\infty \sum _{n \in \mathbb {Z}_{\ge 0}^4} \mu _{i,j}(n) t_1^{n_1}t_2^{n_2}t_3^{n_3}t_4^{n_4}x^iy^j. \end{aligned}$$

By regarding \(V_{(1,0;0;1)}\) as two copies of \(V_{(1,0;0)}\) with the standard torus of the second \(SL_2\) acting with eigenvalues \(\tau \) and \(\tau ^{-1},\) say, we see that

$$\begin{aligned} \frac{\tau f(x,y,t',\tau ) -\tau ^{-1} f(x,y,t',\tau ^{-1}) }{(\tau - \tau ^{-1})}= \sum _{r=0}^{n_4} \tau ^{n_4-2r}=\frac{p(x,y\tau , y\tau ^{-1}, t')}{q(x,y\tau , y\tau ^{-1}, t')}. \end{aligned}$$

So it suffices to verify that

$$\begin{aligned} \begin{aligned}&p(x,y\tau , y\tau ^{-1}, t')(\tau - \tau ^{-1})\delta (x,y,t',\tau ) \delta (x,y,t',\tau ^{-1}) \\&\quad =q(x,y\tau , y\tau ^{-1}, t')[\tau \nu (x,y,t',\tau )\delta (x,y,t',\tau ^{-1}) \\&\qquad -\tau ^{-1} \nu (x,y,t',\tau ^{-1})\delta (x,y,t',\tau )], \end{aligned} \end{aligned}$$

which is easily done with a computer algebra system. \(\square \)

5 Local zeta integrals I

5.1 Definitions and notation

The next step in the analysis of our global integral is to study the corresponding local zeta integrals. We introduce a “local” notation which will be used throughout Sects. 5 and 6, 7 In these section F is a local field which may be archimedean or nonarchimedean. Abusing notation, we denote the F-points of an F-algebraic group H by H as well. We fix an additive character \(\psi \) of F,  and define a character \(\psi _N:N \rightarrow \mathbb {C}\) by the same formula used in the global setting. Similarly, if we fix a triple \(\chi =(\chi _1, \chi _2, \chi _3)\) of characters of \(F^\times ,\) and \(s\in \mathbb {C}^3,\) then formula (2) now defines a character of \(M_P.\) We write \({\text {Ind}}_{P}^G(\chi ; s)\) for the corresponding (unnormalized) induced representation (K-finite vectors, relative to some fixed maximal compact K). We shall assume that the characters in \(\chi \) are unitary, but not that they are normalized, and define \((\chi ;s)\) for \(s \in \mathbb {C}^2\) by the convention \(s_3 = \frac{3s_1+3s_2}{2}.\) Thus we have a two parameter family of induced representations and let \({\text {Flat}}(\chi )\) denote the space of flat sections.

Let \(\mathcal S( {\text {Mat}}_{4 \times 2})\) be the Bruhat–Schwartz space, which we equip with an action \(\omega _{\psi ,l}\) of \(R_Q\) as in the global setting, and define \(I_0 : \mathcal S({\text {Mat}}_{4\times 2}) \times R_Q \rightarrow \mathbb {C}\) by replacing \(\mathbb {A}\) by F in (10).

Next, take \(\pi \) to be a \(\psi _N\)-generic irreducible admissible representation of \(C_Q\) with \(\psi _N\)-Whittaker model \(\mathcal W_{\psi _N}(\pi ),\) and with central character \(\chi _1^{-3}\chi _2^{-3} \chi _3^2.\)

For \(W \in \mathcal W_{\psi _N}(\pi ), f \in {\text {Flat}}(\chi )\) and \(\phi \in \mathcal S({\text {Mat}}_{4\times 2}),\) define the corresponding local zeta integral to be the local analogue of (12), namely:

$$\begin{aligned} I(W, f, \phi ; s) := \int _{ZU_4\backslash C_Q} W(g) \int _{U^w_Q \backslash U_Q} f_{\chi ;s}(wug)I_0(\phi , ug)\, du \, dg. \end{aligned}$$
(28)

In addition to the above notation, for \(1 \le i,j \le r, i \ne j,\) let \(x_{ij}(r)=I_{12}+rE_{i,j} - rE_{13-j,13-i},\) where \(I_{12}\) is the \(12\times 12\) identity matrix and \(E_{i,j}\) is the matrix with a one at the ij entry and zeros everywhere else, and let

$$\begin{aligned} \Xi _0 := \begin{pmatrix} 0&{}\quad 0 \\ 1&{}\quad 0 \\ 0 &{}\quad 1 \\ 0 &{}\quad 0 \end{pmatrix}, \qquad \Xi _2(a):= \begin{pmatrix} a_1&{}\quad a_2 \\ 0&{}\quad a_4\\ 0&{}\quad 0\\ 0&{}\quad 0 \end{pmatrix}, \qquad (a = (a_1,a_2,a_4) \in F^3). \end{aligned}$$

5.2 Inital computations

In this section we carry out some initial computations with local zeta integrals which will be used in both the proof of convergence in Sect. 7.1 and in the unramified computations carried out in Sect. 6.

The image of the function \(x_{23}\) maps isomorphically onto the one dimensional quotient of \(U_4\backslash N,\) and the function \(g \mapsto f_s^\circ (wg)\) is invariant by the image of \(x_{23}\) on the left. Moreover, \(W(x_{23}(x_4)g) = \overline{\psi }(x_4) W(g), \) while

$$\begin{aligned} \left[ \omega _\psi \left( x_{23}(x_4) ug \right) .\phi \right] (\Xi _0 + \Xi _2(x_1,x_2,0)) =\left[ \omega _\psi \left( ut \right) .\phi \right] (\Xi _0 + \Xi _2(x_1,x_2,x_4)) \end{aligned}$$

Let

$$\begin{aligned} \begin{aligned} I\!I\!I(\phi )&:= \int _{F^3} \phi \begin{pmatrix} r_1&{}\quad r_2 \\ 1&{}\quad r_4\\ 0&{}\quad 1\\ 0&{}\quad 0\end{pmatrix}\overline{\psi }(r_1+r_4) \, dr, \qquad ( \phi \in \mathcal S({\text {Mat}}_{4\times 2})) \\ I\!I(f,\phi , s)&:= \int _{U_Q^w\backslash U_Q} f_{\chi ;s}(wu) I\!I\!I(\omega _{\psi , l}(u).\phi ) \, du,\qquad ( \phi \in \mathcal S({\text {Mat}}_{4\times 2}), f \in {\text {Flat}}(\chi )) \\ I_1( W, f, \phi ; s)&:= \int _{Z\backslash T} W(t) I\!I( R(t). f, \omega _{\psi , l}(t). \phi , s) \ \delta _B^{-1}(t)\, dt,\end{aligned} \end{aligned}$$

where \(\phi \in \mathcal S({\text {Mat}}_{4\times 2}), f \in {\text {Flat}}(\chi ), W \in \mathcal W_{\psi _N}(\pi ),\) and R is right translation. Then expressing Haar measure on \(C_Q\) in terms of Haar measures on TN and K,  and using \(x_{23}\) to parametrize \(U_4\backslash N\) yields

$$\begin{aligned} I(W,f, \phi ; s) = \int _K I_1(R(k).W, R(k).f, \omega _{\psi , l}(k).\phi ; s) \, dk \end{aligned}$$
(29)

where K is the maximal compact.

The integral \(I\!I\!I(\phi )\) is absolutely convergent. Indeed, \(I\!I\!I(\phi ) = \phi _1(\Upsilon _0),\) where \(\phi _1\) is the Schwartz function obtained by taking Fourier transform of \(\phi \) in three of the eight variables, and \(\Upsilon _0\) is a matrix with entries 0 and 1. We study the dependence on \(u \in U_Q^w \backslash U_Q\) and \(t \in Z \backslash T\) using the local analogues of (5). A remark is in order, regarding the Weil index \(\gamma _{\psi , \det m_Q^1( g_1, g_2)}\) which appears in the third formula. In order to reconcile the local and global cases, one should think of this as the ratio \(\gamma _{\psi , \det m_Q^1( g_1, g_2)}/\gamma _{\psi , 1}.\) The denominator can be omitted because the global \( \gamma _{\psi , 1}\) is trivial. In the local setting \(\gamma _{\psi , 1}\) may not be trivial, but \(\gamma _{\psi , a^2} = \gamma _{\psi , 1}\) for any a,  and \(\det m_Q^1( g_1, g_2)\) is always a square, so the ratio is always trivial.

Now, let \(U_0 \subset U_Q\) be the subgroup corresponding to the variables, \(x_1,x_2,x_4, y_3, y_5, y_6,\) \(z_1,z_2,z_3\) and \(z_5.\) That is, the subset in which all other variables equal zero. Let \(U_7 \subset U_Q\) be the subgroup defined by the condition that each variable listed above is 0,  and, in addition, \(y_7=y_8=0.\) Then \(U_0U_7\) maps isomorphically onto the quotient \(U_Q^w\backslash U_Q.\) We parametrize \(U_0\) and \(U_7\) using the coordinates inherited from \(U_Q.\) A direct computation using (5) shows that for \(u_0 \in U_0, u_7 \in U_7,\) and \(t \in T,\) \(I\!I\!I(\omega _\psi ( tu_0 u_7).\phi )\) is equal to

$$\begin{aligned} |t ^{\beta _1 + \beta _2 + \beta _4 }| |\det t|^{\frac{1}{2}} \psi (x_1t^{\beta _1} + x_4t^{\beta _4} - y_3 t^{-\beta _3}+y_6t^{-\beta _6}+z_1+z_5) \phi ' \begin{pmatrix} y_1+t^{\beta _1} &{}\quad y_2 \\ x_3+t^{-\beta _3} &{}\quad y_4+t^{\beta _4} \\ x_5 &{}\quad x_6+t^{-\beta _6}\\ x_7 &{}\quad x_8 \end{pmatrix}, \end{aligned}$$
(30)

where \(\phi '\) is the Schwartz function obtained by taking the Fourier transform of \(\phi \) (relative to \(\psi \)) in \(x_1, x_2\) and \(x_4.\) If we define \(\psi _{U_0,t}(u_0):=\psi (x_1t^{\beta _1} + x_4t^{\beta _4} - y_3 t^{-\beta _3}+y_6t^{-\beta _6}+z_1+z_5),\) and we define \(\delta (t) \in U_7\) and \(\pi _7: U_7 \rightarrow {\text {Mat}}_{4\times 2}\) by

$$\begin{aligned} \delta (t)= u_Q\left( \begin{pmatrix}0&{}\quad 0 \\ t^{-\beta _3} &{}\quad 0 \\ 0 &{}\quad t^{-\beta _6}\\ 0 &{}\quad 0 \end{pmatrix} , \begin{pmatrix} 0&{}\quad 0\\ 0 &{}\quad 0 \\ t^{\beta _4} &{}\quad 0 \\ 0 &{}\quad t^{\beta _1} \end{pmatrix}, 0 \right) \in U_7, \qquad \pi _7(u_7) = \begin{pmatrix} y_1 &{}\quad y_2 \\ x_3 &{}\quad y_4 \\ x_5 &{}\quad x_6\\ x_7 &{}\quad x_8 \end{pmatrix}, \end{aligned}$$

then (30) becomes \(|t ^{\beta _1 + \beta _2 + \beta _4 }| |\det t|^{\frac{1}{2}}\psi _{U_0,t}(u_0) \phi '(\pi _7(\delta (t)^{-1}u_7)).\)

The projection \(\pi _7\) has a two dimensional kernel corresponding to the variables \(z_4\) and \(z_6.\) Let \(U_8\) denote this kernel and choose a subset \(U_8'\) of \(U_7\) which contains \(\delta (t)\) and maps isomorphically onto the quotient. Then we can parametrize \(I\!I( R(t). f, \omega _{\psi , l}(t). \phi , s)\) as a triple integral over \(U_0 \times U_8 \times U_8'.\) The \(U_8'\) integral is convergent because \(\phi '\) is Schwartz, after a change of variables it becomes

$$\begin{aligned} \phi '*_1f_{\chi ;s}(wu_0u_8 \delta (t) ):= \int _{U_8'} f_{\chi ;s}(wu_0u_8 \delta (t) u_8') \phi '(\pi _7(u_8')) \, du_8. \end{aligned}$$

Thus, conjugating t from right to left, and making a change of variables yields \(I_1(W, f, \phi ; s) = I_2( W, \phi *_1f;s),\) where \(I_2( W, f,\phi ;s )\) is defined as

$$\begin{aligned} \int _{Z\backslash T} W(t) \delta _B^{-1}(t) |\det t|^{\frac{1}{2}} | t^{\beta _1+\beta _2+\beta _4}| {\text {Jac}}_1(t) (\chi ;s)(wtw^{-1}) f_{\chi ;s}(wu_0u_8 \delta (t) )\, dt, \end{aligned}$$

with \({\text {Jac}}_1(t)\) being the “Jacobian” of the change of variables \(u_0 \rightarrow tu_0t^{-1}, \ u_7 \rightarrow tu_7 t^{-1}.\) Notice that \(\phi *_1f\) is simply another smooth section of the same family of induced representations, and that if \(f_{\chi ;s}\) and \(\phi \) are both unramified, then \(\phi ' = \phi \) and \(\phi '*_1f_{\chi ;s}= f_{\chi ;s}.\) Thus, we may dispense with the integral over \(u_8'.\)

Next, we dispense with the integral over \(u_8.\) To do this, we use [5] to replace \(f_{\chi ;s}\) by a sum of sections of the form

$$\begin{aligned} \phi _2*_2 f'_{\chi ;s}(g) := \int _{F^2} f_{\chi ;s}(g x_{24}(y_1)x_{34}(y_2)) \phi _1(y_1,y_2) \, dy,\qquad (s\in \mathbb {C}^2, g\in G). \end{aligned}$$

Now, let

conjugating \(u_9(y_1,y_2)\) from right to left shows that

$$\begin{aligned}{}[I\!I_2.f_{\chi ;s}]( \delta (t) u_8 u_9(y_1,y_2)) = \psi (-y_1z_6 - y_2(z_4-t^{\alpha _1})) \cdot [I\!I_2.f_{\chi ;s}]( \delta (t) u_8). \end{aligned}$$

(Recall that \(z_6\) and \(z_4\) are coordinates on \(U_8\)). But then

$$\begin{aligned} \int _{F^2}[I\!I_2.\phi _2*_2f_{\chi ;s}]( \delta (t) u_8) \, dz = \int _{F^2}[I\!I_2.f_{\chi ;s}]( \delta (t) u_8) \widehat{\phi }_2(z_6, z_4-t^{\alpha _1}) \, dz, \end{aligned}$$

which we may write as \([I\!I_2.\widehat{\phi }_2*_3f_{\chi ;s}]( \widetilde{\delta }(t)),\) where \(*_3\) is the action of \(\mathcal S(U_8)\) by convolution, and \(\widetilde{\delta }(t)= \delta (t) x_{29}(t^{\alpha _1}).\) Notice that \(\widehat{\phi }_2*_3f_{\chi ;s}\) is again another smooth section of the same family of induced representations. Note also that if f is spherical then taking \(\phi _2\) to be the characteristic function of \(\mathfrak o^2\) gives \(\widehat{\phi }_2*_3f_{\chi ;s}= f_{\chi ;s}.\)

Thus, we are reduced to the study of the integral

$$\begin{aligned} I_3( W, f; s):= \int _{Z\backslash T} W(t) \delta _B^{-1/2}(t) \nu _s(t) I\!I_2.f_{\chi ; s}( \widetilde{\delta }(t))\, dt, \end{aligned}$$
(31)

where \(\nu _s(t):= \delta _B^{-1/2}(t) |\det t|^{\frac{1}{2}} | t^{\beta _1+\beta _2+\beta _4}| {\text {Jac}}_1(t) (\chi ;s)(wtw^{-1}).\) Write \(w= w_1w_2w_3,\) where \(w_1=w[634],\) \(w_2=w[3236514]\) and \(w_3=w[2356243564].\) Write U for the unipotent radical of our standard Borel of G,  and \(U^-\) for the unipotent radical of the opposite Borel. For \(w \in W,\) let \(U_w = U \cap w^{-1} U^- w.\) Then \(w_3 U_0 w_3^{-1} = U_{w_1w_2}= w_2^{-1}U_{w_1} w_2 U_{w_2}.\) For \(\mathbf{c}:=(c_1, \ldots , c_6) \in F^6,\) define a character \(\psi _{\mathbf{c},0}\) of \(U_0\) in terms of the standard coordinates on \(U_0\) by \( \psi _{\mathbf{c},0}(u_0):= \psi (c_1x_1+c_2x_4+c_3y_6-c_4y_3+c_5z_1+c_6z_5). \) Notice that \(\psi _{U_0,t}\) is obtained by taking \(\mathbf{c}=(t^{\beta _1},t^{\beta _4}, t^{-\beta _3}, t^{-\beta _6},1,1).\) In terms of the entries \(u_{ij}\) we have \( \psi _{\mathbf{c},0}(u) = \psi (c_1u_{15} +c_2 u_{26}+c_3 u_{27} -c_4 u_{38} + c_5 u_{19} + c_6 u_{2,10}), \) Now, \(w_3\) corresponds to the permutation (2, 4, 11, 9)(3, 8, 10, 5). It follows that \(u_0'\mapsto \psi _{\mathbf{c},0}( w_3^{-1} u_0' w_3)\) is the character of \(w_3U_0w_3^{-1}\) given by \(u\mapsto \psi ( c_1u_{13} + c_2 u_{46}+c_3 u_{47}+c_4 u_{35} + c_5 u_{12} + c_6 u_{45}).\) In particular, its restriction to \(w_2^{-1}U_{w_1} w_2\) is trivial. Let \(\psi _{\mathbf{c},2}\) denote the restriction to \(U_{w_2}.\) Then

$$\begin{aligned} \int _{U_0} f_{\chi ;s}(wug) \psi _{\mathbf{c},0}(u) \, du = \int _{U_{w_2}} \int _{U_{w_1}} f_{\chi ;s}(w_1u_1w_2 u_2w_3g) \, du_1\,\psi _{\mathbf{c},2}(u_2)\, du_2, \end{aligned}$$
(32)

and the \(u_1\) integral is a standard intertwining operator, \(M(w_1^{-1}, \chi ; s):{\text {Ind}}_P^G(\chi ;s)\rightarrow \underline{{\text {Ind}}}_{{B_{G}}}^G ((\chi ;s)\delta _{B_G}^{-1/2})^{w_1},\) where \(((\chi ;s)\delta _{B}^{-\frac{1}{2}})^{w_1}(t):= ((\chi ;s)\delta _{B}^{-\frac{1}{2}})(w_1 t w_1^{-1}),\) and \(\underline{{\text {Ind}}}\) denotes normalized induction.

Now let \(w_4=w[32365]\) so that \(w_2=w_4w[14].\) Also, let \(w_3' = w[14]w_3.\) Observe that \(w_4\) is the long element of the Weyl group of a standard Levi subgroup of \(GSO_{12}\) which is isomorphic to \(GL_1 \times GL_3 \times GSO_4.\) For \(c_1, \ldots , c_4 \in F,\) define a character \(\psi _{\mathbf{c},4}\) of \(U_{w_4}\) by \(\psi _{\mathbf{c},4} (u) = \psi ( c_1u_{23}+c_4u_{34}+c_3u_{56}+c_2u_{57}),\) and for \(f_{\chi ;s;w_1}\in \underline{{\text {Ind}}}_{B_G}^G ((\chi ;s)\delta _{B_G}^{-1/2})^{w_1},\) let

$$\begin{aligned} \mathcal J_{\psi _{\mathbf{c},4}}.f_{\chi ;s;w_1} ( g) = \int _{U_{w_4}} f_{\chi ;s,w_1}^\circ (w_4 ug) \psi _{\mathbf{c},4}(u)\, du, \end{aligned}$$

which is a Jacquet integral for this Levi subgroup. Then (32) equals

$$\begin{aligned} \int _{F^2} [\mathcal J_{\psi _{\mathbf{c},4}}\circ M(w_1^{-1}, \chi ;s).f_{\chi ;s}](x_{21}(r_1)x_{54}(r_2)w_3'g)\psi (c_5r_1+c_6r_2) \, dr. \end{aligned}$$

6 Unramified calculation

We keep the notation from Sect. 5.1, and assume further that F is nonarchimedean, with ring of integers \(\mathfrak o\) having unique maximal ideal \(\mathfrak p.\) We fix a generator \(\mathfrak w\) for \(\mathfrak p.\) The absolute value on F is denoted \(|\ |\) and normalized so that \(|\mathfrak w|=q:=\# \mathfrak o/\mathfrak p.\) The corresponding \(\mathfrak p\)-adic valuation is denoted v. Moreover, we assume that \(K=G(\mathfrak o)\), and that the representation \(\pi \) and characters \(\chi _i, \ i=1,2,3\) are unramified, and we let \(W_\pi ^\circ , f^\circ \) and \(\phi ^\circ \) denote the normalized spherical elements of \(\mathcal W_{\psi _N}(\pi ), {\text {Flat}}(\chi )\) and \(\mathcal S({\text {Mat}}_{4\times 2}),\) respectively.

The (finite Galois form of the) L-group of \(GSp_4 \times GSO_4\) is \(GSpin_5(\mathbb {C}) \times GSpin_4(\mathbb {C}).\) Indeed, one may define \(GSpin_{2n+1}\) (resp. \(GSpin_{2n}\)) as the reductive group with root datum dual to that of \(GSp_{2n}\) (resp. \(GSO_{2n}\)). However, both GSpin groups appearing here can be understood more explicitly via “coincidences of low rank.” Indeed, a simple change of \(\mathbb {Z}\)-basis reveals that the root datum of \(GSp_4\) is in fact self dual. Thus \(GSpin_5\) is just \(GSp_4\) in another guise. Note, however, that the isomorphism of \(GSp_4\) with its own dual group does not respect the standard numbering of the simple roots.

Next, we can realize \(GSO_4\) (resp. \(GSpin_4\)) as a quotient (resp. subgroup) of \(GL_2 \times GL_2.\) Indeed, we can realize \(GSO_4\) as the similitude group of the four dimensional quadratic space \(({\text {Mat}}_{2\times 2}, \det ).\) Letting \(GL_2\times GL_2\) act by \((g_1, g_2) \cdot X = g_1 X {}^t \!g_2\) induces a surjection \(GL_2 \times GL_2 \rightarrow GSO_4\) with kernel \(\{ (aI_2, a^{-1}I_2): a \in GL_1\},\) and thence a bijection between representations of \(GSO_4\) and pairs of representations of \(GL_2\) with the same central character. By duality, this induces an isomorphism of \(GSpin_4\) with \(\{ (g_1, g_2) \in GL_2\times GL_2: \det g_1 = \det g_2\}.\) We remark that the induced map \(GSpin_4 \rightarrow SO_4\) is not the restriction of our chosen map \(GL_2 \times GL_2 \rightarrow GSO_4.\)

We regard \(GSp_4 \times GSO_4\) as a subgroup of \(M_Q\) containing \(C_Q\) in the obvious way, and regard its L group as a subgroup of \(GSp_4 \times GL_2\times GL_2.\) We make the identification in such a way that the first \(GL_2\) corresponds to the fifth simple root of \(G=GSO_{12}\) and the second \(GL_2\) corresponds to the sixth simple root of G.

Let \(St_{GSp_4}\) denote the standard representation of \(GSp_4.\) It may also be regarded as the spin representation of \(GSpin_5.\) For this reason, the associated L function is often called the “Spinor L function.” We regard \(St_{GSp_4}\) as a representation of of the L group via projection onto the \(GSp_4(\mathbb {C})\) factor, and let \(St_{GSp_4}^\vee \) denote the dual representation. Let \(St_{GL_2^{(1)}}\) (resp. \(St_{GL_2^{(2)}}\)) denote the representations of the L group obtained by composing the standard representation of \(GL_2\) with projection onto the first (resp. second) \(GL_2(\mathbb {C})\) factor.

Theorem 6.1

Let

$$\begin{aligned} \begin{aligned} N(s, \chi )&= L\left( s_1-s_2, \frac{\chi _1}{\chi _2}\right) L\left( s_1-s_2-1, \frac{\chi _1}{\chi _2}\right) L\left( s_1-s_2-2, \frac{\chi _1}{\chi _2}\right) L\left( s_1+s_2-2, \chi _1\chi _2\right) \\&\quad \times L(s_1+s_2-3, \chi _1\chi _2) L(s_1+s_2-4, \chi _1\chi _2) L(2s_2, \chi _2^2) L(2s_1-6, \chi _1^2). \end{aligned} \end{aligned}$$
(33)

(Local L functions. The corresponding product of global zeta functions is the normalizing factor of the Eisenstein series). Then \(I(W_\pi ^\circ , f^\circ , \phi ^\circ ; s)\) equals

$$\begin{aligned} \frac{L\left( \frac{s_1-s_2}{2}-1, \pi , St_{GSp_4}^\vee \otimes St_{GL_2^{(1)}} \times \frac{\chi _3}{\chi _1 \chi _2^2}\right) L\left( \frac{s_1+s_2}{2}-2, \pi , St_{GSp_4}^\vee \otimes St_{GL_2^{(2)}} \times \frac{\chi _3}{\chi _1 \chi _2}\right) }{N(s,\chi )} \end{aligned}$$

Proof

(Reduction of the general case to the special case of trivial characters) For purposes of this proof, write \(\lambda _H\) for the similitude rational character of H where \(H = GSO_{12}, GSp_4,\) or \(GSO_4.\) Our embedding \((GSp_4 \times GSO_4)^\circ \hookrightarrow GSO_{12}\) is such that \( \lambda _{GSO_{12}}( g, h) = \lambda _{GSp_4}(g)^{-1}= \lambda _{GSO_4}(h), \) and the projection \(p: GL_2 \times GL_2 \rightarrow GSO_4\) is such that \(\lambda _{GSO_4}(p(g_1, g_2)) = \det g_1 \det g_2.\)

Write \(\pi = \Pi \otimes \tau _1\otimes \tau _2\) where \(\Pi \) is an unramified representation of \(GSp_4\) and \(\tau _1\) and \(\tau _2\) are unramified representations of \(GL_2\) with the same central character (so that \(\tau _1 \otimes \tau _2\) is a representation of \(GSO_4\)). Write \(\tau _i = \tau _{i,0}\otimes |\det |^{t_1}\) where \(\tau _{i,0}\) is an unramified representation of \(GL_2\) with trivial central character for \(i=1,2\), and \(t_1\) is a complex number, and write \(\Pi = \Pi _0 \otimes |\lambda _{GSp_4}|^{t_2},\) where \(\Pi _0\) is an unramified representation of \(GSp_4\) with trivial central character and \(t_2\) is a complex number. Then, as representations of \(C_Q=(GSp_4 \times GSO_4)^\circ ,\)

$$\begin{aligned} \pi =\pi _0\otimes |\lambda _{GSO_{12}}|^{t_1-t_2}, \qquad \text { where }\quad \pi _0= \Pi _0 \otimes \tau _{1,0}\otimes \tau _{2,0}. \end{aligned}$$

The operation of twisting \(\Pi _0\) by \(|\lambda _{GSp_4}|^{t_2}\) to obtain \(\Pi \) corresponds, on the L group side, to multiplying the corresponding semisimple conjugacy class in \(GSp_4(\mathbb {C})\) by \(q^{-t_3}I_4.\) Likewise, the operation of twisting \(\tau _{i,0}\) by \(|\det |^{t_1}\) corresponds to multiplying by \(q^{-t_1}\) in \(GL_2(\mathbb {C})\) for \(i=1,2.\) If \(\eta \) is the unramified character \(\eta (a) = |a|^r\), then

$$\begin{aligned} L\left( u, \pi , St^\vee _{GSp_4} \otimes St_{GL_2^{(i)}} \times \eta \right) = L\left( u-t_2+t_1+r, \pi _0, St^\vee _{GSp_4} \otimes St_{GL_2^{(i)}}\right) . \end{aligned}$$

For \(i=1,2,3\) the unramified character \(\chi _i\) is given by \(|\ |^{r_i}\) for some \(r_i \in \mathbb {C}.\) If \(s = (s_1, s_2, \frac{3(s_1+s_2)}{2}) \in \mathbb {C}^3,\) then let \(s' = (s_1+r_1, s_2+r_2, \frac{3(s_1+r_1+s_2+r_2)}{2}),\) and let \(\chi _0\) be the triple consisting of three copies of the trivial character. Then it follows directly from the definitions that \(f^\circ _{\chi ; s} = f^\circ _{\chi _0; s'} \cdot |\lambda _{GSO_{12}}|^{r_3 - \frac{3r_1+3r_2}{2}}.\) The general case now follows from the case \(t_1=t_2=r_1=r_2=r_3=0.\)

Recall that \(I(W, f, \phi ; s)\) is only defined when the triple \(\chi \) and the central character of \(\pi \) are compatible. In the present notation the compatibility condition is that \(-3r_1-3r_2 +2 r_3 + 2( t_1-t_2) = 0.\) But then \(W_\pi \cdot f^\circ _{\chi ; s} =W_{\pi _0}\cdot f^\circ _{\chi _0; s'}.\) The general case now reduces to the special case when \(\chi = \chi _0\) and \(\pi =\pi _0.\) \(\square \)

Remark 6.2

We may now assume that \(\pi = \Pi \otimes \tau _1 \otimes \tau _2,\) where the central characters of \(\Pi , \tau _1\) and \(\tau _2\) are all trivial. Thus, \(\pi \) may be regarded as an unramified representation of \(SO_5 \times PGL_2 \times PGL_2\) and corresponds to a semisimple conjugacy class in \(Sp_4(\mathbb {C}) \times SL_2(\mathbb {C}) \times SL_2(\mathbb {C}).\) Define \(St_{Sp_4}, St_{SL_2^{(1)}}\) and \(St_{SL_2^{(2)}}\) as the restrictions of \(St_{GSp_4}, St_{GL_2^{(1)}}\) and \(St_{GL_2^{(2)}}\), respectively. Note that all three are self dual representations. In particular, \(L(u, \pi , St_{GSp_4}^\vee \otimes St_{GL_2^{(i)}}) = L(u, \pi , St_{Sp_4} \otimes St_{SL_2^{(i)}})\) for all \(u \in \mathbb {C}.\)

Proof

(Proof in the special case of trivial characters) As \(\chi \) is trivial we write \(f^\circ _s\) instead of \(f^\circ _{\chi ; s}.\) Let \(I(s, \pi ):= I(W_\pi ^\circ , f^\circ , \phi ; s).\) As we’ve seen in Sect. 5.2, it is equal to \(I_3(W_\pi ^\circ , f^\circ ; s),\) defined by (31). Let \(\mathbf{I}_1(s; c_1, \ldots , c_6)\) equal

$$\begin{aligned} \int _{U_0} f_{\chi ;s}(wu_0 ) \psi _{\mathbf{c},0}(u_0) \,du_0. \end{aligned}$$

Let \(a=t^{\beta _1}, b=t^{\beta _4}, c=t^{-\beta _6}, d=t^{-\beta _3}.\) Note that \(\beta _1 - \beta _3 = \alpha _1,\) \(\beta _4-\beta _3 = \alpha _5,\) and \(\beta _4-\beta _6= \alpha _2.\) Moreover, the characters \(-\beta _3-\beta _6\) and \(\alpha _6\) are not identical on \(T_G,\) but they have the same restriction to the maximal torus T of \(C_Q.\) It follows that for t in the support of \(W_\pi ^\circ ,\) the quantities |ad|, |bd|, |bc| and |cd| are all \(\le 1.\)

By plugging in the Iwasawa decomposition for \(w_3\widetilde{\delta }(t),\) we find that

$$\begin{aligned}&I\!I_2.f^\circ _s(\widetilde{\delta }(t))=\nonumber \\&\quad {\left\{ \begin{array}{ll} \mathbf{I}_1(s;a,b,c,d,1,1), &{} |a|,|b|,|c|,|d| \le 1,\\ |d|^{-s_1-s_2+4}{} \mathbf{I}_1( s, ad, bd, cd, 1,1,0), &{} |d|> 1, |a|,|b|,|c|\le 1,\\ |c|^{-s_1-s_2+4}{} \mathbf{I}_1(s, a, bc, 1, cd, 1, 0), &{} |c|>1, |a|, |b|,|d| \le 1,\\ |a|^{-s_1+s_2+2}|c|^{-s_1-s_2+4}{} \mathbf{I}_1(s, 1, bc, 1, acd, 0,0), &{}|a|,|c|>1, |b|,|d| \le 1,\\ |a|^{-s_1+s_2+2}{} \mathbf{I}_1(s, 1, b,c,ad, 0,1), &{} |b|,|c|,|d|\le 1, |a|>1,\\ |a|^{-s_1+s_2+2}|b|^{-s_1+s_2+2}{} \mathbf{I}_1( s, 1,1,bc, abd, 0,0), &{} |c|,|d|\le 1, |a|,|b|>1,\\ |b|^{-s_1+s_2+2}{} \mathbf{I}_1(s, a,1,bc,bd,1,0), &{} |d|,|c|,|a| \le 1, |b| > 1. \end{array}\right. }\nonumber \\ \end{aligned}$$
(34)

Now let \(f_{s,w_1}^\circ \) denote the normalized spherical vector in \(\underline{{\text {Ind}}}_{B_G}^G((\chi _0;s)\delta _{B}^{-\frac{1}{2}})^{w_1},\) and let

$$\begin{aligned} Z_1(s):= \frac{\zeta (2s_2-1)}{\zeta (2s_2)} \frac{\zeta (s_1-s_2-1)}{\zeta (s_1-s_2)} \frac{\zeta (s_1+s_2-3)}{\zeta (s_1+s_2-2)} \end{aligned}$$

Then, \(M(w_1^{-1}, \chi ; s).f_s^\circ = Z_1(s) f_{s,w_1}^\circ ,\) by the Gindikin–Karpelevic formula, and hence

$$\begin{aligned} \mathbf{I}_1(s; c_1, \ldots , c_6) = Z_1(s) \int _{F^2} \mathcal J_{\psi _{\mathbf{c},4}} f_{s,w_1}^\circ ( x_{21}(r_1) x_{54}(r_2) ) \psi (c_5r_1+c_6r_2) \,dr. \end{aligned}$$

We remark that \(((\chi _0;s)\delta _{B}^{-\frac{1}{2}})^{w_1}\) maps \({\text {diag}}( \lambda t_1, \ldots , \lambda t_6, t_6^{-1}, \ldots , t_1^{-1})\in T_G\) to

$$\begin{aligned} |t_1|^{s_1-5}|t_2|^{s_1-4}|t_3|^{s_2-2}|t_4|^{-s_2}|t_5|^{s_1-3} |t_6|^{1-s_2}|\lambda |^{s_3-2s_2-\frac{13}{2}}. \end{aligned}$$
(35)

\(\square \)

Lemma 6.3

Assume that each of \(c_5,c_6\) is either zero or a unit. Assume further that if \(c_6 =0\) then at least one of \(c_2, c_3, c_4\) is a unit, and that if \(c_5=0\) then \(c_1\) is a unit, and set \(J_{c_1,c_2, c_3, c_4}= \mathcal J_{\psi _{\mathbf{c},4}} f_{s,w_1}^\circ ( I_{12}).\) Then \(\mathbf{I}_1(s; c_1, \ldots , c_6)/Z_1(s)\) equals

$$\begin{aligned} J_{c_1,c_2, c_3, c_4}-q^{-s_1-s_2+4}J_{ \frac{c_1}{\mathfrak w}, c_2, c_3, c_4} - q^{-2s_1+6}J_{c_1,\frac{c_2}{\mathfrak w}, \frac{c_3}{\mathfrak w}, \frac{c_4}{\mathfrak w}} +q^{-3s_1-s_2+10} J_{\frac{c_1}{\mathfrak w},\frac{c_2}{\mathfrak w}, \frac{c_3}{\mathfrak w}, \frac{c_4}{\mathfrak w}}. \end{aligned}$$

Remark 6.4

Observe that all the sextuples \(c_1, \ldots , c_6\) appearing in (34) satisfy the conditions of Lemma 6.3.

Proof

There exist cocharacters \(h_i: GL_1 \rightarrow T_G,\; (i=1,2)\) such that \(\langle h_1, \alpha _i\rangle = \delta _{i,1}\) and \(\langle h_2, \alpha _i \rangle = \delta _{i,4}\) (Kronecker \(\delta \)). It follows that \( \mathcal J_{\psi _{\mathbf{c},4}} f_{s,w_1}^\circ ( x_{21}(r_1) x_{54}(r_2) )\) depends only on \(v(r_1)\) and \(v(r_2).\) If \(c_5\) is a unit, then

$$\begin{aligned} \int _{v(r_1) = -k} \psi (c_5 r_1) \, dr_1 = {\left\{ \begin{array}{ll} -1, &{} k =1,\\ 0, &{} k >1, \end{array}\right. } \end{aligned}$$

and similarly with \(r_2.\) Since both \(f_{s,w_1}^\circ \) and \(\psi \) are unramified, it follows that \(\mathbf{I}_1(s; c_1, \ldots , c_6)\) equals

$$\begin{aligned}&\mathcal J_{\psi _{\mathbf{c},4}} f_{s,w_1}^\circ (I_{12})- \mathcal J_{\psi _{\mathbf{c},4}} f_{s,w_1}^\circ (\mathfrak w^{-1}))- \mathcal J_{\psi _{\mathbf{c},4}} f_{s,w_1}^\circ (\mathfrak w^{-1}))\\&\quad + \mathcal J_{\psi _{\mathbf{c},4}} f_{s,w_1}^\circ ( x_{21}(\mathfrak w^{-1})x_{54}(\mathfrak w^{-1})). \end{aligned}$$

Plugging in the Iwasawa decomposition of \(x_{21}(\mathfrak w^{-1})\) and/or \(x_{54}(\mathfrak w^{-1})\) gives the result in this case.

Now suppose that \(c_5\) is not a unit. Then it is zero and \(c_1\) is a unit. It follows that

$$\begin{aligned}&\int _{F^2} \mathcal J_{\psi _{\mathbf{c},4}} f_{s,w_1}^\circ ( x_{21}(r_1) x_{54}(r_2) ) \psi (c_5r_1+c_6r_2) \,dr\\&=\int _{F} \mathcal J_{\psi _{\mathbf{c},4}} f_{s,w_1}^\circ ( x_{54}(r_2) ) \psi (c_6r_2) \,dr_2. \end{aligned}$$

Indeed the support of J is contained in \(UT_1K\) where \(K=GSO_{12}(\mathfrak o)\) is the maximal compact subgroup, and \(T_1\) is the set of torus elements t with \(|t^{\alpha _2}|\le 1.\) It follows easily from the Iwasawa decomposition that \(x_{21}(r_1) x_{54}(r_2)\in UT_1K\) if and only if \(r_1 \in \mathfrak o.\) If \(c_6\) is a unit then proceeding as before we obtain

$$\begin{aligned} \int _{F} \mathcal J_{\psi _{\mathbf{c},4}} f_{s,w_1}^\circ ( x_{54}(r_2) ) \psi (c_6r_2) \,dr_2=J_{c_1,c_2, c_3, c_4} - q^{-2s_1+6}J_{c_1,\frac{c_2}{\mathfrak w}, \frac{c_3}{\mathfrak w}, \frac{c_3}{\mathfrak w}}. \end{aligned}$$

On the other hand, when \(c_1\) is a unit then \(J_{ \frac{c_1}{\mathfrak w}, c_2, c_3, c_4} = J_{\frac{c_1}{\mathfrak w},\frac{c_2}{\mathfrak w}, \frac{c_3}{\mathfrak w}, \frac{c_4}{\mathfrak w}}=0,\) and the stated result follows in this case as well. Likewise, if \(c_6\) is zero, then integration over \(r_2\) can be omitted and \(J_{c_1,\frac{c_2}{\mathfrak w}, \frac{c_3}{\mathfrak w}, \frac{c_4}{\mathfrak w}}= J_{\frac{c_1}{\mathfrak w},\frac{c_2}{\mathfrak w}, \frac{c_3}{\mathfrak w}, \frac{c_4}{\mathfrak w}}=0,\) which gives the result in the remaining two cases. \(\square \)

Now consider the subgroup of the torus consisting of all elements of the form

$$\begin{aligned} t={\text {diag}}\left( t_1t_2, t_2, 1, t_1^{-1}, t_3t_4, t_4, \frac{t_2}{t_4}, \frac{t_2}{t_3t_4}, t_1t_2, t_2, 1, t_1^{-1}\right) . \end{aligned}$$
(36)

This subgroup maps isomorphically onto \(Z\backslash T.\) For elements of this torus and with coordinates as in (36) we have

$$\begin{aligned} {\text {Jac}}_1(t) =|t_1^2t_3t_4^2|^{-1}, \qquad {\text {Jac}}_2(t) = \left| \frac{t_1^2 t_2^3}{t_3t_4^3}\right| \qquad \delta _B^{-\frac{1}{2}} =|t_1^2 t_2t_3t_4|^{-1} \qquad |\det t|^{\frac{1}{2}} = |t_2^{-2} t_3^2 t_4^4|. \end{aligned}$$
$$\begin{aligned} (\chi _0;s)( wtw^{-1}) = |t_1|^{s_1-s_2} |t_2|^{-s_1-3s_2+s_3} |t_3t_4^2|^{s_2}. \end{aligned}$$

Set \(x = q^{-( \frac{s_1-3s_2}{2})},\) \( y = q^{-s_2+1},\) and let \(n_i\) be the \(\mathfrak p=v(t_i)\) for \(1 \le i \le 4.\) Then

$$\begin{aligned} \nu _s(t) = x^{2n_1+n_2}y^{2n_1+n_3+2n_4}. \end{aligned}$$
(37)

For \(l=(l_1, l_2, l_3, l_4)\in \mathbb {Z}^4,\) set

$$\begin{aligned} j_1(l)= & {} 1 - x^{2 l_4+2}y^{ 2 l_4+2}-y^{2 l_1+2} - x^{2 l_1+2 l_4+4}y^{ 4 l_1+4 l_4+8}\\&+\, x^{2 l_4+2}y^{2 l_1+4 l_4+6}+x^{2 l_1+2 l_4+4}y^{ 4 l_1+2 l_4+6} \end{aligned}$$
$$\begin{aligned} j_2(l ) = 1 - x^{2 l_2+2}y^{ 4 l_2+4} \qquad j_3(l) = 1 - x^{2 l_3+2}y^{ 2 l_3+2} \end{aligned}$$
$$\begin{aligned} j(l ) = {\left\{ \begin{array}{ll}j_1(l) j_2(l) j_3(l), &{} l_i \ge 0 \forall i, \\ 0, &{} \text { otherwise.}\end{array}\right. } \end{aligned}$$

Then direct computation or the Casselman-Shalika formula shows that

$$\begin{aligned} \mathcal J_{\psi _{\mathbf{c},4}}.f^\circ _{s,w_1}(I_{12}) = \frac{\zeta (s_1+s_2-4)^2}{\zeta (s_1+s_2-3)^2} \frac{\zeta (s_1-s_2-2)^2}{\zeta (s_1-s_2-1)^2} \frac{\zeta (2s_2-2)}{\zeta (2s_2-1)}j(l), \end{aligned}$$

where \(l_i=v(c_i)\) for each i. Hence, if

$$\begin{aligned} i_1(l ) = j(l) - x^{2}y^{ 4} j(l -(1,0,0,0)) - x^{4}y^{ 6} j(l-(0,1,1,1)) + x^{6}y^{ 10} j(l-(1,1,1,1)), \end{aligned}$$

then

$$\begin{aligned} \mathbf{I}_1(s; c_1, \ldots , c_6) =\frac{\zeta (s_1-s_2-2)^2\zeta (s_1+s_2-4)^2\zeta (2s_2-2)i_1(n_1,n_2, n_3,n_4)}{\zeta (s_1-s_2)\zeta (s_1-s_2-1)\zeta (s_1+s_2-2)\zeta (s_1+s_2-3)\zeta (2s_2)}, \end{aligned}$$

for all \(c_1, \ldots , c_6\) satisfying the conditions of Lemma 6.3. Consequently,

$$\begin{aligned} I\!I_2.f_{s}^\circ (\widetilde{\delta }(t)) = \frac{\zeta (s_1-s_2-2)^2\zeta (s_1+s_2-4)^2\zeta (2s_2-2)i(A,B,C,D)}{\zeta (s_1-s_2)\zeta (s_1-s_2-1)\zeta (s_1+s_2-2)\zeta (s_1+s_2-3)\zeta (2s_2)}, \end{aligned}$$

where \(A=v(t^{\beta _{1}}\)), \(B=v(t^{\beta _{4}})\), \(C=v(t^{-\beta _{6}})\), and \(D=v(t^{-\beta _{3}})\) and i is defined piecewise in terms of \(i_1\) according to the seven cases from (34). It is convenient to use an alternate parametrization. Let \(ii(m_1, m_2, m_3, m_4)\) equal

$$\begin{aligned}&i\left( m_1- \frac{-m_2+m_3+m_4}{2}, \frac{m_2+m_3-m_4}{2}, \frac{m_2-m_3+m_4}{2}, \frac{-m_2+m_3+m_4}{2}\right) \nonumber \\&\quad x^{2m_1+m_2}y^{2m_1+m_2+m_4} \end{aligned}$$
(38)

if \(m_2+m_3+m_4\) is even and zero otherwise. Then

$$\begin{aligned} \begin{aligned} \nu _s(t) I\!I_2.f_s^\circ (\widetilde{\delta }(t))&= \frac{\zeta (s_1-s_2-2)^2\zeta (s_1+s_2-4)^2\zeta (2s_2-2)ii(m)}{\zeta (s_1-s_2)\zeta (s_1-s_2-1)\zeta (s_1+s_2-2)\zeta (s_1+s_2-3)\zeta (2s_2)}, \end{aligned} \end{aligned}$$

where now \(m_i=v(t^{\alpha _i})\) for \(i=1,2,3,4.\) Let \([m_1, m_2; m_3; m_4]\) or [m] denote the trace of the irreducible representation of \(^L(C_Q/Z):=Sp_4\times SL_2\times SL_2\) on which \(Sp_4\) acts with highest weight \(m_1\varpi _1+ m_2 \varpi _2,\) the first \(SL_2\) acts with highest weight \(m_3,\) and the second \(SL_2\) acts with highest weight \(m_4.\) Then

$$\begin{aligned} \delta _B^{-\frac{1}{2}}(t)W_\pi (t) = [m_2, m_1; m_3; m_4](\tau _\pi ), \end{aligned}$$

where \(\tau _\pi \) is the semisimple conjugacy class in \(^L(C_Q/Z)\) attached to \(\pi .\) Note the reversal of order between 1 and 2. The reason for this is that when \(GSp_4\) is identified with its own dual group, the standard numberings for the two dual \(GSp_4\)’s are opposite to one another. For example the coroot attached to the short simple root \(\alpha _1\) is the long simple coroot, which makes it the long simple root of the dual group.

Now, let \(Z_2(x,y)= (1-y^2)(1-x^2y^2)^2(1-x^2y^4)^2\). Then j(n) is divisible by \(Z_2\) for any n. Also, for \(\varrho \) the character of a finite dimensional representations of \(^L(C_Q/Z),\) let \( L(u, \varrho ) = \sum _{i=0}^\infty u^k {\text {Tr}}{\text {sym}}^k(\varrho ),\) Then Theorem 6.1 is reduced to the following identity of power series over representation ring of \(^L(C_Q/Z)\):

$$\begin{aligned} \frac{1}{Z_2(x,y)} \sum _{m \in \mathbb {Z}_{\ge 0}^4}ii(m_2, m_1, m_3, m_4) [m]=Z_3(x,y) L(xy, [1,0;1;0])L(xy^2,[1,0;0;1]), \end{aligned}$$
(39)

where \(Z_3(x,y)=(1-x^2y^2)(1-x^2y^4)(1-x^4y^6).\)

Define polynomials \(P_{m}(u,v)\) by

$$\begin{aligned} L(u, [1,0;1;0])L(v,[1,0;0;1]) =\sum _{m \in \mathbb {Z}_{\ge 0}^4} P_m(u,v) [m]. \end{aligned}$$

Then (39) is equivalent to the family of identities of polynomials,

$$\begin{aligned}&(1-y^2)(1-x^2y^2)^3(1-x^2y^4)^3(1-x^4y^6)P_m(xy,xy^2)\\&\quad = ii(m_2, m_1, m_3, m_4) \; (\forall m \in \mathbb {Z}_{\ge 0}^4), \end{aligned}$$

or to the identity of power series over a polynomial ring:

$$\begin{aligned} \begin{aligned} \sum _{m \in \mathbb {Z}_{\ge 0}^4} ii(m_2, m_1, m_3, m_4) t_1^{m_1} t_2^{m_2} t_3^{m_3} t_4^{m_4}&= Z_4(x,y) \sum _{m \in \mathbb {Z}_{\ge 0}^4} P_m(xy, xy^2) t_1^{m_1} t_2^{m_2} t_3^{m_3} t_4^{m_4}\\ {}&= Z_4(x,y) \frac{\nu (x,y,t)}{\delta (x,y,t)}, \end{aligned} \end{aligned}$$
(40)

where \(\nu \) and \(\delta \) are defined as in Proposition 4.4, and \(Z_4=Z_2Z_3.\)

The identity (40) can be proved as follows. Let \(X = (X_1, X_2,X_3,X_4)\) and \(Y= (Y_1, Y_2, Y_3, Y_4)\) be quadruples of indeterminates. Define polynomials,

$$\begin{aligned} {\underline{j}}_1(x,y,X,Y):= & {} 1 - x^2 y^2 X_4^2 Y_4^2 - y^2 Y_1^2 - x^2y^8 X_1^2 X_4^2 Y_1^4 Y_4^4 + x^2 y^6 X_4^2 Y_1^2Y_4^4 \\&+\, x^4 y^6 X_1^2X_4^2 Y_1^4Y_4^2 \end{aligned}$$
$$\begin{aligned} {\underline{j}}_2(x,y,X,Y)= (1 - x^2 y^4 X_2^2 Y_2^4); \qquad {\underline{j}}_3(x,y,X,Y) = (1 - x^2 y^2 X_3^2 Y_3^2); \qquad {\underline{j}} = {{\underline{j}}_{1}}{{\underline{j}}_{2}}{{\underline{j}}_{3}}, \end{aligned}$$

so that for \(k= (k_1, \ldots , k_4) \in \mathbb {Z}_{\ge 0}^4,\) the polynomial j(k) is equal to \({\underline{j}}(x,y,x^k,y^k),\) where \(x^k:=(x^{k_1}, \ldots , x^{k_4})\) and \(y^k:= (y^{k_1}, \ldots , y^{k_4}).\) Likewise, one computes a polynomial \({\underline{i}}_1(x,y,X,Y)\) such that \(i_1(k) = {\underline{i}}_1(x,y,x^k,y^k).\) It can be expressed as a sum of 12 monomials in X and Y,  each with a coefficient which is a polynomial in x and y. Thus \( i_1(k) = \sum _{i=1}^{12} c_i(x,y) \prod _{j=1}^4 (\mu _{i,j}(x,y))^{k_i}, \) for some polynomials \(c_1, \ldots , c_{12}\) and monomials \(\mu _{1,1} \ldots , \mu _{12, 4}\) in x and y. Now,

$$\begin{aligned} \sum _{m \in \mathbb {Z}_{\ge 0}^4} ii(m_2, m_1, m_3, m_4) t_1^{m_1} t_2^{m_2} t_3^{m_3} t_4^{m_4} \end{aligned}$$
$$\begin{aligned} = {\begin{array}{c} \sum _A,B,C,D\in \mathbb {Z}\\ A+D,B+C,B+D,C+D\ge 0 \end{array}} i(A,B,C,D) x^{2A+B+C+2D}y^{2A+B+2C+3D} t_1^{B+C}t_2^{A+D}t_3^{B+D}t_4^{C+D}. \end{aligned}$$

This is a sum of seven subsums corresponding to the seven cases which appear in (34). The simplest of these is

$$\begin{aligned} \begin{aligned}&\sum _{A,B,C,D \ge 0}i_1(A,B,C,D) x^{2A+B+C+2D}y^{2A+B+2C+3D} t_1^{B+C}t_2^{A+D}t_3^{B+D}t_4^{C+D} \\&\quad = \sum _{i=1}^{12} \frac{c_i(x,y)}{(1-\mu _{i,1}(x,y) t_2x^2y^2)(1-\mu _{i,2}(x,y) t_1 t_3xy)(1-\mu _{i,3}(x,y) t_1t_4xy^2)(1-\mu _{i,4}(x,y) t_2t_3t_4x^2y^3). }\end{aligned} \end{aligned}$$

In each of the other six sums one can make a substitution to obtain a similar, fourfold sum of \(A',B',C',D'\) from 0 to infinity. For example, in the second case listed (34), one has the conditions \(A,B,C\ge -D\ge 1.\) Substituting \( A'= A+D,\) \(B' = B+D,\) \(C' = C+D,\) and \(D' = -D-1,\) yields

$$\begin{aligned} \begin{aligned}&\sum _{A',B',C',D'=0}^\infty i_1(A',B',C',0) t_1^{B'+C'+2D'+2}t_2^{A'} t_3^{B'}t_4^{C'} x^{2A'+B'+C'+4D'+4}y^{2A'+B'+2C'+6D'+6} \\ {}&\quad = \sum _{i=1}^{12} \frac{c_i(x,y)x^4y^6t_1^2}{(1-\mu _{i,1}(x,y)t^2x^2y^2)(1-\mu _{i,2}(x,y)t_1t_3xy)(1-\mu _{i,3}(x,y)t_1t_4xy^2)(1-t_1^2x^4y^6)} \end{aligned} \end{aligned}$$

The other five subsums are treated similarly. Totaling up the resulting rational functions and simplifying gives (40), completing the proof of theorem.

7 Local zeta integrals II

In this section we continue our study of the local zeta integral \(I(W, f, \phi ; s)\) at the ramified places.

7.1 Convergence

In this section, we prove the convergence of local zeta integrals

Theorem 7.1

Take \(W \in \mathcal W_{\psi _N}(\pi ), f \in {\text {Flat}}(\chi )\) and \(\phi \in \mathcal S({\text {Mat}}_{4\times 2}).\) Then the local zeta integral \(I(W, f, \phi ; s)\) converges for \({\text {Re}}(s_1-s_2)\) and \({\text {Re}}(s_2)\) both sufficiently large.

Proof

We need to show that convergence of \(I_3( W, f; s)\) defined in (31), for \(W \in W_{\psi _N}(\pi )\) and f a smooth section of the family of induced representations \({\text {Ind}}_P^G(\chi ;s).\) To do this, we simply bound \(f_{\chi ;s}\) by a constant times the spherical section \(f_{{\text {Re}}(s)}^\circ ,\) where \({\text {Re}}(s) \in \mathbb {R}^2\) is the real part of s. Then \(I\!I_2.f_{\chi ;s}(\widetilde{\delta }(t))\) is bounded by a constant multiple of \(M(w_2^{-1}w_1^{-1}, {\text {Re}}(s)).f^\circ _{{\text {Re}}(s)}(w_3\widetilde{\delta }(t)),\) where \(M(w_2^{-1}w_1^{-1},{\text {Re}}(s))\) is a standard intertwining operator. The unramified character \((\chi _0; {\text {Re}}(s))\delta _{B_G}^{-1}\) may be identified with an element \(\varsigma \) of \(X(T_G)\otimes _Z\mathbb {R}.\) The integral defining the standard intertwining operator converges provided the canonical pairing \(\langle \varsigma , \alpha ^\vee \rangle \) is positive for all positive roots \(\alpha \) with \(w_2^{-1} w_1^{-1} \alpha < 0.\) Inspecting this set of roots, one finds it is convergent provided \({\text {Re}}(s_2)>1, {\text {Re}}(s_1-s_2)> 2,\) and \({\text {Re}}(s_1+s_2) > 5.\) Moreover, it converges to a section of the representation induced (via normalized induction) from \(\varsigma ^{w_1w_2}.\)

Next, we need to understand the dependence of \(M(w_1w_2, {\text {Re}}(s)).f^\circ _{{\text {Re}}(s)}(w_3\widetilde{\delta }(t))\) on T. In order to do this, we write \(w_3 \widetilde{\delta }(t) \) as \(\widetilde{\nu }(t) \widetilde{\tau }(t) \widetilde{\kappa }(t)\) where \(\widetilde{\nu }(t) \in U, \ \widetilde{\tau }(t) \in T_G\) and \(\widetilde{\kappa }(t)\) varies in a compact set. It is convenient to do so using the basic algebraic substitution

$$\begin{aligned} \begin{pmatrix} 1 &{}\quad 0 \\ r &{}\quad 1 \end{pmatrix} = \begin{pmatrix} r^{-1} &{}\quad 1 \\ 0 &{}\quad r \end{pmatrix} \begin{pmatrix} 0 &{}\quad -1 \\ 1 &{}\quad r^{-1} \end{pmatrix}, \end{aligned}$$
(41)

which corresponds to the Iwasawa decomposition if F is nonarchimedean, but remains valid in the Archimedean case as well.

Recall that \(\widetilde{\delta }(t)\) is the product of \(x_{37}( -b),\) \(x_{36}( -c)\) and \(x_{25}(-d) x_{48}(-a) x_{29}(ad),\) which all commute with one another. We can partition T into 16 subsets and use the identity (41) to obtain a uniform expression for \(\widetilde{\tau }(t)\) on each subset, and compute \(\varsigma ^{w_1w_2}\delta _{B_G}^{1/2}( \widetilde{\tau }(t))\) in each case, obtaining

$$\begin{aligned}&\left\{ \begin{array}{l@{\quad }r} 1, &{} |b| \le 1,\\ |b|^{-2u_1}, &{} |b|> 1 \end{array}\right\} \times \left\{ \begin{array}{lr} 1, &{} |c|\le 1,\\ |c|^{-2u_2}, &{} |c|> 1 \end{array}\right\} \\&\quad \times \left\{ \begin{array}{l@{\quad }r} 1, &{} |a|, |d| \le 1,\\ |d|^{-2u_2}, &{} |a|\le 1, |d|> 1,\\ |a|^{-2u_1}, &{} |a|> 1, |ad| \le 1,\\ |a|^{-2u_1-u_2}|d|^{-2u_2}, &{} |a|> 1, |ad| >1\end{array} \right\} , \end{aligned}$$

where \(u_1:= {\text {Re}}(\frac{s_1 -s_2-2}{2}),\ u_2:= {\text {Re}}(\frac{s_1+s_2-4}{2}).\) Note that most these contributions are already visible in (34). Moreover, as in (37) we have \( |\nu _s(t)| = |a|^{2u_1} |b|^{u_1} |c|^{u_2} |d|^{u_1+u_2} \)

Next, we consider the quantity \(W(t) \delta _B^{-1/2}(t)\) which appears in the integral (31). Using [4] or [26] in the nonarchimedean case, or [35], and [30] as explicated in [31] and [32] in the archimedean case, we have

$$\begin{aligned} W(t) \delta _B^{-1/2}(t) = \sum _{x \in X_{\pi }} \Phi _x(ad,bc,bd, cd) x(t), \end{aligned}$$
(42)

where \(X_\pi \) is a finite set of finite functions depending on the representation \(\pi ,\) and \(\Phi _x\) is a Bruhat-Schwartz function \(F^4 \rightarrow \mathbb {C}\) for each x.

Thus we obtain a sum of integrals of the form

$$\begin{aligned} \int _D \Phi (ad, bc,bd,cd) x(t) |a|^{k_1u_1+l_1u_2} |b|^{k_2u_1+l_2u_2} |c|^{k_3u_1+l_3u_2} |d|^{k_4u_1+l_4u_2} \, dt, \end{aligned}$$
(43)

where D is one of our 16 subsets, \(k_1, \ldots , k_4\) and \(l_1, \ldots , l_4\) are explicit integers depending only on D\(\Phi \) is a Bruhat-Schwartz function \(F^4 \rightarrow \mathbb {R},\) and x is a real-valued finite function \(Z\backslash T \rightarrow \mathbb {R}.\)

Now, for each of the seven cases which appear in (34), make a change of variables, as in the unramified case so that \(|a|^{k_1u_1+l_1u_2} |b|^{k_2u_1+l_2u_2} |c|^{k_3u_1+l_3u_2} |d|^{k_4u_1+l_4u_2}\) is expressed as powers of the absolute values of the new variables. For example, when \(|d|>1\) and \(|a|,|b|,|c| \le 1,\) we have \(|\nu _s(t)| |d|^{-{\text {Re}}(s_1)-{\text {Re}}(s_2)+4} =|a|^{2u_1} |b|^{u_1} |c|^{u_2} |d|^{u_1-u_2},\) and substitute \(a'=ad, b'=bd, c'=cd\) and \(d'=d^{-1}.\) After the substitution, each exponent is a nontrivial non-negative linear combination of \(u_1\) and \(u_2.\) Also \(|d'|\) is bounded, and we have \(\Phi (a', b'c'(d')^2, b',c'),\) which provides convergence as \(|a'|, |b'|\) or \(|c'| \rightarrow \infty .\) It follows that the integral converges provided \(u_1\) and \(u_2\) are sufficiently large, relative to the finite function x.

As a second example, we consider the case \(|a|,|c|>1, |b|,|d|\le 1.\) In this case, we make the change of variables \(b'=bc, d'=acd, a'=a^{-1}, c'=c^{-1}.\) We obtain the integral

assuming that \(u_1\) and \(u_2\) are positive and sufficiently large (depending on x), the integrals on \(a'\) and \(c'\) are convergent due to the domain of integration, and the integrals on \(b'\) and \(d'\) are convergent from the decay of \(\Phi .\) Indeed, \(\Phi (c'd', b', a'd', a'(b')^2 c'd') \ll |a'b'c'(d')^2|^{-N}\) for any positive integer N because \(\Phi \) is Bruhat-Schwartz, and then \( |a'b'c'(d')^2|^{-N}\le |b'd'|^{-N}\) on the domain D. The other five cases appearing in (34) are handled similarly.

The nine cases which do not appear in (34) are easier. For example suppose that |c| and |d| are both \({>}1\) while |a| and |b| are both \(\le 1.\) Then the exponents of |a| and |b| are the same as in \(\nu _s(t),\) i.e., they are \(2u_1\) and \(u_1\) respectively. This gives convergence of the integrals on a and b when \({\text {Re}}(u_1)\) is sufficiently large (relative to x). The integrals on |c| and |d| converge because of the rapid decay of \(\Phi \) in cd. The other eight cases are treated similarly, completing the proof of the convergence of \(I_1(W, f, \phi ; s).\) Now consider \(I_1(R(k).W, R(k).f, \omega _\psi (k).\phi ; s).\) Each bound used in the analysis of \(I_1\) can be made uniform as k varies in the compact set k. Hence \(I_1(R(k).W, R(k).f, \omega _\psi (k).\phi ; s)\) varies continuously with k so its integral is again absolutely convergent. \(\square \)

7.2 Continuation to a slightly larger domain

In this section, we prove that the local zeta integral \(I(W,f, \phi ;s)\) extends analytically to a domain that includes the point \(s_1=5, s_2=1.\) This point is of particular interest for global reasons. We keep the notation from the previous section. There are two issues. The first is related to the convergence of the integral \(I\!I_2.f_{\chi ;s}.\) As we have seen, this integral is not absolutely convergent at (5, 1). We must show that it extends holomorphically to a domain containing (5, 1). Then we need to prove convergence of the integral over \(Z\backslash T.\) The domain of absolute convergence for this integral depends on the exponents of the representation \(\pi .\) To make this precise, we use terminology and notation from [2], Sect. 3.1.

Proposition 7.2

Suppose that \(\Pi \) satisfies \(H(\theta _4)\) and that \(\tau _1\) and \(\tau _2\) satisfy \(H(\theta _2)\) (as in [2], section 3.1). Then for any \(\varepsilon > 0,\) the local zeta integral \(I(W, f, \phi ; s)\) extends holomorphically to all \(s \in \mathbb {C}^2\) satisfying \({\text {Re}}(s_1-s_2) \ge \max ( 2 \theta _4+2\theta _2+2,3)+\varepsilon , \ {\text {Re}}(s_1+s_2) \ge 5+\varepsilon ,\) \({\text {Re}}(s_2) \ge \frac{1}{2} + \varepsilon ,\) \({\text {Re}}(s_1+2s_2) \ge 2 \theta _2+1.\)

Proof

We first need to extend \(I\!I_2.f_{\chi ;s}\) beyond its domain of absolute convergence. It suffices to do this for flat K-finite sections, even though the convolution sections encountered in Sect. 5.2 are not, in general, flat of K-finite. Indeed, the integral operator \(I\!I_2\) commutes with the convolution operators considered in 5.2. Moreover, these operators are rapidly convergent, and hence preserve holomorphy.

As we have seen in the unramified computation \(I\!I_2\) can be expressed \(I\!I_3 \circ M(w_1^{-1}, \chi ;s),\) where \(I\!I_3\) is an operator defined on \(\underline{{\text {Ind}}}_{B_G}^G((\chi ;s )\delta _{B_G}^{-1/2})^{w_1}\) by the \(u_2\) integral in (32). Then, \(M(w_1^{-1}, \chi ; s)\) is absolutely convergent for \({\text {Re}}(s_2)> \frac{1}{2}, {\text {Re}}(s_1-s_2)> 1, {\text {Re}}(s_1+s_2-3) > 3.\) If we insert absolute values into the integral which defines \(I\!I_3,\) we obtain a standard intertwining operator attached to \(w_2^{-1}.\) We may write is as a composite of rank one intertwining operators attached to \(\{ \alpha > 0: w_2^{-1} \alpha < 0\}.\) The rank one operator attached to \(\alpha \) is absolutely convergent provided that \( \langle \alpha ^\vee , \varsigma ^{w_1}\rangle \) is positive. Running through the eight relevant roots, we find that only one rank one operator diverges at (5, 1). It is attached to the simple root \(\alpha _3\) which satisfies \(\langle \alpha ^\vee , \varsigma ^{w_1}\rangle =2{\text {Re}}(s_2)-2.\)

Thus, we only need to improve our treatment of the integral over a single one-parameter unipotent subgroup. Thus, we consider

$$\begin{aligned} \int _F f^{w_1}_{\chi ;s}(w[3]x_{34}(r) g) \psi (c_4r) \, dr, \end{aligned}$$
(44)

where \(c_4 \in F\) and \(f^{w_1}_{\chi ;s}\) is a section of the family \(\underline{{\text {Ind}}}_{B_G}^G((\chi ; s)\delta _{B_G}^{-1/2})^{w_1}, \ s \in \mathbb {C}^2.\) Notice that (44) may be regarded as a Jacquet integral for the rank-one Levi attached to the simple root \(\alpha _3.\) By [21], this extends to an entire function of s when \(f^{w_1}_{\chi ;s}\) is flat. If we apply it to the output of \(M(w_1^{-1}, \chi ; s),\) then it has no poles other than those of \(M(w_1^{-1}, \chi ; s).\) Now we use again the fact that the asymptotics of a Whittaker function, are controlled by the exponents of the relevant representation. This time we apply it to the induced representation of our rank one Levi. For most values of s,  the exponents are \(((\chi ; s)\delta _{B_G}^{-1/2})^{w_1}\) and \(((\chi ; s)\delta _{B_G}^{-1/2})^{w_1w[3]}\) and the Whittaker function is bounded in absolute value by a linear combination of spherical vectors.

On the line \(s_2=1,\) this may fail: if \(((\chi ; s)\delta _{B_G}^{-1/2})^{w_1}=((\chi ; s)\delta _{B_G}^{-1/2})^{w_1w[3]},\) then an extra log factor appears in the asymptotics of the Whittaker function (cf. [17], 6.8.11, for example). Bounding \(\log |x|\) by \(|x|^{-\varepsilon }\) with \(\varepsilon > 0\) as \(x \rightarrow 0,\) in this case, we again bound the integral (44) by a sum of spherical sections. In fact, the extra \(|x|^\varepsilon \) may be safely ignored, since we obtain convergence for s in an open set and \(\varepsilon >0\) can be taken arbitrarily small. Thus, if \(w_2=w[3]w_2',\) then \(I\!I_2.f_{\chi ;s}\) extends holomorphically to the domain where the standard intertwining operator attached to \(w_2'\) converges on both \(f^\circ _{{\text {Re}}(s), w_1},\) and \(f^\circ _{{\text {Re}}(s), w_1w[3]}.\) Inspecting \(\{ \alpha > 0 : (w_2')^{-1} \alpha < 0 \},\) we see that this means \({\text {Re}}(2s_2-1), {\text {Re}}(s_1-s_2-3)\) and \({\text {Re}}(s_1+s_2-5)\) must all be positive. As a side effect, we find that \(|I\!I_2.f_{\chi ;s}(g)|\) is bounded by a suitable linear combination of \(M(w_2^{-1}w_1^{-1}, {\text {Re}}(s)).f^\circ _{{\text {Re}}(s)}\) and \(M((w_2')^{-1}w_1^{-1}, {\text {Re}}(s)).f^\circ _{ {\text {Re}}(s)}.\)

As before, we obtain a sum of integrals of the form (43) where now the integers \(k_1, \ldots , k_4\) and \(l_1, \ldots , l_4\) depend on the choice of domain D and on a choice of between \(w_2\) and \(w_2'.\)

In order to obtain a precise domain of convergence, we need information about the finite function x. Firstly, since we have taken absolute values and assumed unitary central character, it factors through the map \(t \mapsto (|ad|, |bc|, |bd|, |cd|).\) We may assume that x is given in terms of real powers of the coordinates and non-negative integral powers of their logarithms, since such functions span the space of real-valued finite functions. Since a power of \(\log y\) may be bounded by an arbitrarily small positive (resp. negative) power of y as \(y\rightarrow \infty \) (resp. 0), for purposes of determining the domain of convergence, we may assume that there are no logarithms. Thus we may assume \(x(t) = |ad|^{\rho _1} |bc|^{\rho _2} |bd|^{\rho _3} |cd|^{\rho _4}\) with \(\rho _1,\rho _2,\rho _3,\rho _4 \in \mathbb {R}.\) The quadruples \((\rho _1,\rho _2,\rho _3,\rho _4)\) which appear are governed by the exponents of \(\pi ,\) by [4] or [26] in the nonarchimedean case, and [30, 35] (see also [32]) in the archimedean case. Hence they are bounded in absolute value by \(\max (\theta _2, \theta _4),\) by the definition of \(H(\theta _2)\) and \(H(\theta _4)\) in [2] and the bound on exponents of tempered representations found in [30], Theorem 15.2.2 in the archimedean case, or [34] in the nonarchimedean case, we see that \(|\rho _1|, |\rho _2| \le \theta _4, |\rho _3|, |\rho _4| \le \theta _1.\)

What remains is a careful case-by-case analysis along the same lines as the proof of convergence. For each choice of D,  after a suitable change of variables we have an integral which is convergent provided \(u_1\) and \(u_2\) are sufficiently large, and “sufficiently large” is given explicitly in terms of \(\rho _1, \ldots , \rho _4.\)

For example, the above integral corresponding to the case \(|a|,|c|>1\) and \(|b|,|d|\le 1\) will now feature a Schwartz function integrated against

$$\begin{aligned} |c'd'|^{\rho _1} |b'|^{\rho _2} |a'b' (c')^2d'|^{\rho _3}|a'd'|^{\rho _4} |a'|^{u_1+u_2}|b'|^{u_1}|c'|^{2u_1+2u_2}|d'|^{u_1+u_2}, \end{aligned}$$

and so will converge provided \(u_1+u_2+\rho _3+\rho _4\), \(u_1+\rho _2+\rho _3\), \(2u_1+\rho _1+2\rho _2,\) and \(u_1+u_2+\rho _1+\rho _3+\rho _4\) are all positive. \(\square \)

7.3 Meromorphic continuation and nonvanishing

Write \(U_P^-\) for the unipotent radical of the parabolic opposite P. Notice that \(PU_P^-w\) is a Zariski open subset of \(GSO_{12}.\) We say that \(f \in {\text {Flat}}(\chi )\) is simple if it is supported on \(PK_1\) where \(K_1\) is a compact subset of \(U_P^-w.\)

Proposition 7.3

Suppose that f is simple. Then \(I(W,f, \phi ; s)\) has meromorphic continuation to \(\mathbb {C}^2\) for each \(\phi \in \mathcal S({\text {Mat}}_{4\times 2})\) and each \(W\in \mathcal W_{\psi _N}(\pi )\) Moreover, if \(s_0\) is an element of \(\mathbb {C}^2,\) then there exist Wf and \(\phi \) such that \(I(W, f, \phi ; s_0) \ne 0.\)

Proof

We begin with some formal manipulations which are valid over any local field. The process requires many of the same subgroups which were defined during the Proof of Theorem 3.1, and we freely use notation from that section.

$$\begin{aligned} I(W, f, \phi ;~ s) = \int _{Z U_4\backslash C_Q} \int _{U_Q^w \backslash U_Q} \int _{{\text {Mat}}_{1\times 2}} W(g) f(wug, s) [\omega _\psi (ug).\phi ]\begin{pmatrix} r \\ I_2 \\ 0 \end{pmatrix} \, \overline{\psi }\left( r \cdot \begin{bmatrix} 1\\ 0 \end{bmatrix} \right) \, dr\, du \, dg. \end{aligned}$$

Define \(U_1(a,b,c)\) as in (23). Then

$$\begin{aligned}{}[\omega _\psi (ug).\phi ]\begin{pmatrix} r \\ I_2 \\ 0 \end{pmatrix} = [\omega _\psi (U_1(r_1, r_2, c) ug).\phi ](\Xi _0), \end{aligned}$$

(for any c) where \(\Xi _0:=\begin{pmatrix} 0 \\ I_2 \\ 0\end{pmatrix}\) and \(r = (r_1\ r_2).\) Also \(W(U_1(r_1, r_2, c)g) = \overline{\psi }(r_1) W(g) = \overline{\psi }\left( r \cdot \begin{bmatrix} 1\\ 0 \end{bmatrix} \right) W(g).\) Hence

$$\begin{aligned} I(W, f, \phi ; s) = \int _{Z U_5\backslash C_Q} \int _{U_Q^w \backslash U_Q} W(g) f(wug, s) [\omega _\psi (ug).\phi ](\Xi _0) du \, dg, \end{aligned}$$

where \(U_5\) is the product of \(U_6, U_2\) and the center \(Z(U_1)\) of \(U_1.\) \(\square \)

Recall that \(C_Q^w\) is a standard parabolic subgroup of \(C_Q.\) Let \(U(C_Q^w)^-\) denote the unipotent radical of the opposite parabolic. Then \(C_Q^w \cdot U(C_Q^w)^-\) is a subset of full measure in \(C_Q\) and we can factor the Haar measure on \(C_Q\) as the product of (suitably normalized) left Haar measure on \(C_Q^w\) and Haar measure on \(U(C_Q^w)^-.\) Hence

$$\begin{aligned}&I(W, f, \phi ; s)\\&\quad = \int _{Z U_5\backslash C_Q^w} \int _{U(C_Q^w)^-} \int _{U_Q^w \backslash U_Q} W(gu_1) f(wugu_1, s) [\omega _\psi (ugu_1).\phi ](\Xi _0) du \,du_1\, d_\ell g \end{aligned}$$

Conjugating g across u,  making a change of variables, and making use of the equivariance of f yields

$$\begin{aligned} I(W,f, \phi ; s) = \int _{U(C_Q^w)^-} \int _{U_Q^w \backslash U_Q} J( R(u_1).W, \omega _\psi ( uu_1).\phi ; s) f(wuu_1, s)\, du\, du_1. \end{aligned}$$

where

$$\begin{aligned} J(W, \phi , s) = \int _{Z U_5\backslash C_Q^w}W(g) [\omega _\psi (g).\phi ](\Xi _0) \ {\text {Jac}}_1(g) (\chi ; s)(wgw^{-1})\, d_\ell g. \end{aligned}$$

with \({\text {Jac}}_1(g)\) being the Jacobian of the change of variables in u. Now conjugation by w maps \(U_Q^w \backslash U_Q \times U(C_Q^w)\) isomorphically onto the unipotent radical \(U_P^-\) of the parabolic opposite P. Hence, if \(\Phi \) is any smooth compactly supported function on \(U_Q^w \backslash U_Q \times U(C_Q^w)^-,\) then there is a flat section f such that \(f(wuu_1, s_0 ) = \Phi (u, u_1).\)

We claim that the integral \(J(W, \phi ; s)\) converges provided \({\text {Re}}(s_1-s_2)\) and \({\text {Re}}(s_2)\) are both sufficiently large, and that \(J( R(u_1).W, \omega _\psi ( uu_1).\phi ; s)\) extends meromorphically to \(\mathbb {C}^2\) and is a continuous function of \(uu_1\) away from the poles. Granted this claim, is clear that if the integral of \(J( R(u_1).W, \omega _\psi ( uu_1).\phi ; s_0)\) against the arbitrary test function \( f(wuu_1, s_0)\) is always zero, then \(J( W, \phi ; s_0)\) is zero for all W and \(\phi .\)

Now, \(C_Q^w=(P_1\times P_2)^\circ \) is the intersection of \(C_Q\) with the product of the Klingen parabolic \(P_1\) of \(GSp_4\) and the Siegel parabolic \(P_2\) of \(GSO_4.\) Let \(C'=(P_1\times M_2)^\circ \) denote the subgroup of elements whose \(GSO_4\) component lies in the Levi, and let \(U_5'=C' \cap U_5=U_6Z(U_1).\) Then \(C'\) surjects onto \(ZU_5 \backslash C_Q^w,\) which is thus canonically identified with \(ZU_5' \backslash C'.\) Expressing the measure on \(C_Q^w\) in terms of Haar measures on \(U_1, U_2\) and \((M_1\times M_2)^\circ ,\) and then identifying \(Z(U_1) \backslash U_1\) with \({\text {Mat}}_{2\times 1}(F)\) via the map \(\bar{u}_1(\begin{bmatrix} r_1&\quad r_2\end{bmatrix}) = U_1(r_1, r_2, 0),\) yields the following expression for \(J(W, \phi ; s)\):

$$\begin{aligned} \int \limits _{{\text {Mat}}_{1\times 2}} \int \limits _{(M_1\times M_2)^\circ } W(\bar{u}_1(r) m) [\omega _\psi (\bar{u}_1(r)m).\phi ](\Xi _0) \ {\text {Jac}}_1(m) (\chi ; s)(wmw^{-1})\, \delta _{C_Q^w}^{-1}(m) dm\, dr, \end{aligned}$$
(45)

where \(\delta _{C_Q^w},\) is the modular quasicharacter.

Now, elements of \(C'\) map under j into the Siegel Levi of \(Sp_{16}.\) So that

$$\begin{aligned}{}[\omega _\psi \circ j(c').\phi ](\xi ) = |\det c'|^{\frac{1}{2}} \phi ( \xi \cdot c'), \end{aligned}$$

where \(\cdot \) is the rational right action of \(C'\) on \({\text {Mat}}_{4\times 2}\) by

$$\begin{aligned} \xi \cdot m_Q^1( g_1, g_2) = g_1^{-1} \xi g_2. \end{aligned}$$

[with \(m_Q^1\) as in (4)]. The stabilizer of the matrix \(\Xi _0\) is precisely the group \(M_5\) introduced in the Proof of Theorem 3.1.

In (45), conjugate m across \(\bar{u}_1(r),\) make a change of variables in r,  and let \({\text {Jac}}_2(m)\) denote the Jacobian. Define

$$\begin{aligned} \mu _s(m) = (\chi ;s )(wmw^{-1}) \delta _{C_Q^w}^{-1}(m) |\det m|^{\frac{1}{2}} {\text {Jac}}_1(m) {\text {Jac}}_2(m). \end{aligned}$$

Then replace m by \(m_5 m_5'(g)\) where \(m_5 \in M_5\) and \(m_5'(g) = m(1, I_2, g),\) [with m as in (21)]. Observe that

$$\begin{aligned} \Xi _0 \cdot m_5 m_5'(g) \bar{u}_1(r) = \begin{pmatrix} r \cdot g \\ g \\ 0 \end{pmatrix}. \end{aligned}$$

Hence if \(x(g,r) = m_5'(g) \bar{u}_1(r g^{-1}),\) then

$$\begin{aligned} J(W, \phi ; s)= \int \limits _{{\text {Mat}}_{1\times 2}} \int \limits _{GL_2} J'(R(x(g,r)).W, s) \phi \begin{pmatrix} r \\ g\\ 0 \end{pmatrix} \mu _s( m_5'(g)) |\det g|^{-1} \, dg\, dr, \end{aligned}$$
(46)
$$\begin{aligned} \text { where } \qquad J'(W, s):= \int \limits _{ZU_6 \backslash M_5} W(m_5) \mu _s( m_5) \, dm_5.\end{aligned}$$
(47)

Direct computation shows that \(\mu _s(m_5'(g)) = |\det g|^{s_2} \chi _2(\det g).\)

Write \(M_5=U_6 T_5 K_5,\) where \(T_5 = T \cap M_5\) and \(K_5\) is the maximal compact subgroup of the \(GL_2\) factor. Then

$$\begin{aligned} J'(W, s):= \int _{K_5} \int _{Z \backslash T_5} W(tk) \mu _s(t)\delta _{B_5}^{-1}(t) \end{aligned}$$

where \(\delta _{B_5}\) is the modular quasicharacter of the standard Borel subgroup \(B_5\) of \(M_5.\) Set \(t_6'(a) = {\text {diag}}(a,1,1,a^{-1}, 1,1,1,1,a,1,1,a^{-1}),\) and write \(t \in T_5\) as \(t_6 t_6'(a)\) for \(t_6 \in T_6\) and \(a \in F^\times .\) Then

$$\begin{aligned} J'(W, s) = \int _{K_5} \int _{F^\times } J''( R(t_6'(a)k).W, s) \mu _s(t_6'(a)) \, dt, \end{aligned}$$
$$\begin{aligned} \text { where } J''(W, s) = \int _{Z\backslash T_6} W(t_6) \mu _s(t_6)\delta _{B_5}^{-1}(t_6) \, dt_6. \end{aligned}$$

Observe that \(J'(W,s)\) may be written formally as

$$\begin{aligned} \int _{M_6 \backslash M_5} J''(R(g_1).W, s) \mu _s(g_1) dg_1. \end{aligned}$$

Also, direct computation shows that \(\mu _s(t_6'(a)) = |a|^{s_1-s_2-4} \chi _1(a)/\chi _2(a).\)

For \(\phi _1\) a smooth function of compact support \(F^2 \rightarrow \mathbb {C}\) let

$$\begin{aligned}{}[\phi _1*_1W](g) = \int _{F^2} W(g U_1(r_1,r_2,0)) \phi (r_1,r_2) \, dr. \end{aligned}$$

Observe that

$$\begin{aligned}{}[\phi _1*_1W](M_5(t,g_3)) = W(M_5(t,g_3)) \widehat{\phi }_1( g_3^{-1} \cdot \begin{bmatrix} 1\\ 0 \end{bmatrix} t \det g_3). \end{aligned}$$

Thus, by replacing W by \(\phi _1*_1 W\) (which is justified by [5]) we may introduce what amounts to an arbitrary test function on \(M_6\backslash M_5.\) Hence

$$\begin{aligned} J'(W, s_0 ) = 0 \forall W \iff J''(W, s_0) = 0 \forall s_0. \end{aligned}$$

Similarly, if

$$\begin{aligned}{}[\phi _2*_2 W](g) := \int _{U_6} W(g u_6 ) \phi _2(u_6)\, du_6,\qquad \phi _2 \in C_c^\infty ( U_6), \end{aligned}$$

then \(J''(\phi _2*_2W, s_0) = 0 \forall \phi _2\in C_c^\infty (U_6)\) if and only if W vanishes identically on \(T_6.\) In particular, if \(J''(W,s_0)\) vanishes identically on \(\mathcal W_{\psi _N}(\pi ),\) then \(\mathcal W_{\psi _N}(\pi )\) is trivial– a contradiction.

This completes the formal arguments for Proposition 7.3. It remains to check the convergence and continuity statements made above. These will be proved based on facts about asymptotics of Whittaker functions and the Mellin transform

$$\begin{aligned} Z_{\xi , n}. \Phi (u) = \int _{F^\times } \Phi (x) \xi (x) (\log |x|)^n |x|^u \, d^\times x, \end{aligned}$$
(48)

where \(\Phi \in \mathcal S(F), \ \xi : F^\times \rightarrow \mathbb {C}, \text { a character,}\ n \ge 0 \in \mathbb {Z}, \ u \in \mathbb {C}.\) We recall some properties.

Proposition 7.4

Fix a character \(\xi \) and a non-negative integer n.

  1. (1)

    There is a real number c depending on \(\xi \) such that the integral defining \(Z_{\xi , n}. \Phi \) converges absolutely and uniformly on \(\{ u \in \mathbb {C}: {\text {Re}}(u) \ge c+\varepsilon \}\) for all \(\varepsilon > 0\) and all \(\Phi \in \mathcal S(F).\)

  2. (2)

    There is a discrete subset \(S_\xi \) of \(\mathbb {C}\) such that \(Z_{\xi , n}.\Phi \) extends meromorphically to all of \(\mathbb {C}\) with poles only at points in \(S_\xi .\) Moreover, there is an integer \(o_{\xi , n}\) such that no pole of \(Z_{\xi , n}.\Phi \) has order exceeding \(o_{\xi , n},\) for any \(\Phi .\)

  3. (3)

    If F is archimedean, then \(Z_{\xi , n}.\Phi = Q_\Phi ( q^{u})\) for some rational function Q.

  4. (4)

    We have

    $$\begin{aligned} Z_{\xi , n+1}.\Phi (u) = \frac{d}{du} Z_{\xi , n}.\Phi (u). \end{aligned}$$
    (49)

Proof

If \(n= 0\) then the first three parts are proved in [33]. Convergence for \(n>0\) is straightforward, since \(\log |x|\) grows slower than any positive power of |x| at infinity, and slower than any negative power as \(|x| \rightarrow 0.\) Equation (49) is clear in the domain of convergence, and follows elsewhere by continuation. The first three parts for general n then follow. \(\square \)

Next, we need a version of the expansion (42). Specifically, if we replace W(t) by W(tk) then each \(\Phi _x\) in (42) will be in \(\mathcal S(F^4 \times K)\) (see [31], especially the remark on p. 20).

Let us now consider the convergence issues raised by our formal computations more carefully. Recall that \(I(W, f, \phi ; s)\) was initially expressed as an integral over \(U_Q^w \backslash U_Q \times ZU_5 \backslash C_Q.\) In the course of our arguments, we have expressed it as an iterated integral over

$$\begin{aligned} (U_Q^w \backslash U_Q \times U(C_Q^w)^-)\times (F^2 \times GL_2) \times (F^\times \times K_5) \times Z\backslash T_6. \end{aligned}$$

In order to perform the integration on \(Z\backslash T_6\) we may identify with with \(\{ \bar{t}_6(a) = {\text {diag}}(1,1,a^{-1},a^{-1},1,a^{-1},1,a^{-1},1,1,a^{-1},a^{-1}), a \in F^\times \}.\) Then \(\mu _s( \bar{t}_6(a))= |a|^{\frac{s_1-s_2}{2}-2} \frac{\chi _1^2\chi _2}{\chi _3}(a).\)

Now, the integral \(J''(W,s)\) is the Mellin transform taken along the one dimensional torus we use to parametrize \(Z\backslash T_6.\) Its convergence and analytic continuation follow directly from Proposition 7.4 and (42). Now consider \(J''(R(t_6'(a)k).W, s)\) with \(a \in F^\times , k \in K_5.\) We claim that it is smooth and of rapid decay in a. In the domain of absolute convergence, this is easily seen. For s outside of the domain of convergence, we use (49) to pass to the Mellin transform of a suitable derivative at a point inside the domain of convergence. To get \(J'(W,s)\) we integrate k over the compact set k and take another Mellin transform in the variable a. This of course converges absolutely, and by similar reasoning, we see that \(J'(R(x(g,r).W), s)\) is smooth. Now, set g equal to \(\begin{pmatrix} t_1&{}\quad b \\ 0 &{}\quad t_2 \end{pmatrix} k\) and consider the integral

$$\begin{aligned} \int \limits _{{\text {Mat}}_{1\times 2}} \int \limits _{F^\times } \int \limits _{F^\times } \int \limits _{F} \int \limits _{K_{GL_2}} J'( R(x(g , r)).W, s) \phi \begin{pmatrix} r\\ g\\ 0 \end{pmatrix} \chi _2( t_1 t_2) |t_1|^{s_4} |t_2|^{s_5} \, dk \, db\, dr \, d^\times t_1 \, d^\times t_2, \end{aligned}$$

where \(s_4\) and \(s_5\) are two more complex variables, and \(K_{GL_2}\) is the standard maximal compact subgroup of \(GL_2.\) The integrals on kr and b converge absolutely and uniformly because \(K_{GL_2}\) is compact and \(\phi \) is Schwartz-Bruhat. The integrals on \(t_1\) and \(t_2\) take two more Mellin transforms, yielding a meromorphic function of four complex variables. The restriction to a suitable two-dimensional subspace of \(\mathbb {C}^4\) is \(J(W, \phi , s).\) Moreover, \(J(R(u_1).W, \omega _\psi (uu_1).\phi , s)\) remains continuous in \(u_1 \in U(C_Q^w)^-\) and \(u \in U_Q^w \backslash U_Q,\) which completes the proof.

8 Global identity

We now return to the global situation. Thus F is again a number field with adele ring \(\mathbb {A},\) while and \(\psi _N,\) and \({\text {Flat}}(\chi ),\) are defined as in Sects. 3 and 2, respectively. In addition, let \(\pi = \otimes _v \pi _v\) be an irreducible, globally \(\psi _N\)-generic cuspidal automorphic representation of \(GSp_4(\mathbb {A}) \times GSO_4(\mathbb {A}),\) with normalized central character \(\omega _\pi ,\) and \(\varphi \) be a cusp form from the space of \(\pi ,\) etc.

For r a representation of \(^LG\) define \(L(u, \pi , r\times \eta )\) to be the twisted L function. Thus at an unramified place v the local factor is

$$\begin{aligned} L_v( u, \pi _v, r \times \eta _v) = \det ( I - q^{-u} \eta _v(\mathfrak w_v) r( \tau _{\pi _v}))^{-1}, \end{aligned}$$

where \(\mathfrak w_v\) is a uniformizer, \(q_v\) is the cardinality of the residue class field, \(\tau _{\pi _v}\) is the semisimple conjugacy class attached to \(\pi _v,\) and \(\eta _v\) is the local component of \(\eta \) at v.

Theorem 8.1

Suppose that \(f_\chi =\prod _{v}f_{\chi _v}\in {\text {Flat}}(\chi ), \phi =\prod _v \phi _v\in \mathcal S({\text {Mat}}_{4\times 2}(\mathbb {A}))\) and \(W_{\varphi } = \prod _v W_v\) (the Whittaker function of \(\pi \) as in Theorem 3.1 are factorizable. Let \(I(f_{\chi _v;s}, W_v, \phi _v)\) be the local zeta integral, defined as in (28), and let S be a finite set of places v and all data is unramified for all \(v\notin S.\) Then for \({\text {Re}}(s_1-s_2)\) and \({\text {Re}}(s_2)\) both sufficiently large, the global integral \(I(f_{\chi ;s}, \varphi , \phi ),\) defined as in (7), is equal to

$$\begin{aligned} \frac{L^S\left( \frac{s_1-s_2}{2}-1, \pi , St_{GSp_4}^\vee \otimes St_{GL_2^{(1)}} \times \frac{\chi _3}{\chi _1 \chi _2^2}\right) L^S\left( \frac{s_1+s_2}{2}-2, \pi , St_{GSp_4}^\vee \otimes St_{GL_2^{(2)}} \times \frac{\chi _3}{\chi _1 \chi _2}\right) }{N^S(s,\chi )} \end{aligned}$$

times

$$\begin{aligned} \prod _{v\in S} I(f_{\chi _v}, W_v, \phi _v), \end{aligned}$$

where \(N^S(s, \chi )\) is the product of partial zeta functions corresponding to (33)

Remark 8.2

Let \(\eta _1\) and \(\eta _2\) be any two characters of \(F^\times \backslash \mathbb {A}^\times .\) Fix \(\pi \) and let \(\omega _\pi \) be its central character. Then the system

$$\begin{aligned} \frac{\chi _1^3 \chi _2^3}{\chi _3^2} = \omega _\pi , \qquad \frac{\chi _3}{\chi _1 \chi _2^2} = \eta _1, \qquad \frac{\chi _3}{\chi _1 \chi _2} = \eta _2 \end{aligned}$$

has a unique solution. If \(\eta _1 = \eta _2\) is trivial, then it is given by \(\chi _1 =\chi _3 = \omega _\pi \) and \(\chi _2 \equiv 1.\)

Proof

It follows from Theorem 6.1 the bound obtained in [25] that for any cuspidal representation \(\pi = \otimes _v \pi _v\) of \(GSp_4(\mathbb {A}) \times GSO_4(\mathbb {A})\) the infinite product \(\prod _{v\in S} I( f_{\chi _v}, W_v, \phi _v)\) is convergent for \({\text {Re}}(s_1-s_2)\) and \({\text {Re}}(s_2)\) sufficiently large. It then follows from Theorem 3.1, and the basic results on integration over restricted direct products presented in [33] that

$$\begin{aligned} I(f_{\chi ;s}, \varphi , \phi ) = \prod _v I( f_{\chi _v;s}, W_v, \phi _v), \end{aligned}$$

which, in conjunction with Theorem 6.1 again gives the result. \(\square \)

Corollary 8.3

Let \(\pi _v\) be the local constituent at v of a cuspidal representation \(\pi .\) Then the local zeta integral \(I_v(W_v, f_v, \phi _v; s)\) has meromorphic continuation to \(\mathbb {C}^2\) for any \(W_v, f_v\) and \(\phi _v.\)

Proof

This follows from a globalization argument. We create a section of the global induced representation which is f at one place and simple at every other place. Meromorphic continuation of the global zeta integral follows from that of the Eisenstein series. Having shown meromorphic continuation at every other place in Proposition 7.3, we deduce it at the last place. \(\square \)

9 Application

In this section we give an application relating periods, poles of L functions, and functorial lifting. The connection between L functions and functorial lifting in this case was obtained in [2].

Let \(\Pi \) be a globally generic cuspidal automorphic representation of \(GSp_4,\) and let \(\tau _1\) and \(\tau _2\) be two cuspidal automorphic representations of \(GL_2.\) Assume that \(\Pi ,\) \(\tau _1\) and \(\tau _2\) have the same central character. Then \(\tau _1 \otimes \tau _2\) may be regarded as a representation of \(GSO_4\) via the realization of \(GSO_4\) as a quotient of \(GL_2 \times GL_2\) discussed in Sect. 6, and when \(\Pi \otimes \tau _1 \otimes \tau _2\) is restricted to the group \(C_Q\) (which we identify \(C_Q\) with subgroup of \(GSp_4 \times GSO_4\) as in Sect. 2 its central character is trivial.

Now take \(s_1\) and \(s_2\) to be two complex numbers. Let \(\chi _1 = \chi _2=\chi _3\) be trivial. Consider the space \({\text {Flat}}(\chi )\) of flat sections as in Sect. 2. Its elements are functions \(\mathbb {C}^3 \times G(\mathbb {A}) \rightarrow \mathbb {C},\) but we regard each as a function \(\mathbb {C}^2 \times G(\mathbb {A}) \rightarrow \mathbb {C}\) by pulling it back through the function \((s_1, s_2) \mapsto (s_1, s_2, \frac{3s_1+3s_2}{2}).\) Then Theorem 8.1 relates the global integral (7) with the product of L functions

$$\begin{aligned} L^S\left( \frac{s_1-s_2-2}{2}, \widetilde{\Pi }\times \tau _1\right) L^S\left( \frac{s_1+s_2-4}{2}, \widetilde{\Pi }\times \tau _2\right) . \end{aligned}$$

For \(f \in {\text {Flat}}(\chi ),\) let

$$\begin{aligned} {\underline{r}}(f,g) = {\text {Res}}_{s_1-s_2=4}{\text {Res}}_{s_1+s_2=6} E(f_{\chi ;s},g) \end{aligned}$$

be the iterated residue of the Eisenstein series along the plane \(s_1+s_2=6\) and then the plane \(s_1-s_2=4.\) (It follows from Theorem 8.1 and Proposition 7.3 that this residue is nonzero. It can also be checked directly.) As f varies we obtain a residual automorphic representation which we denote \(\mathcal R.\) Given \({\underline{r}} \in \mathcal R\) and \(\phi \in \mathcal S({\text {Mat}}_{4\times 2}(\mathbb {A})),\) we define the Fourier coefficient \({\underline{ r}}^{\theta (\phi )}\) exactly as in 6. Varying \({\underline{r}}\) and \(\phi \) we obtain a space of smooth, K-finite functions of moderate growth \(Z(\mathbb {A}) C_Q(F)\backslash C_Q(\mathbb {A}) \rightarrow \mathbb {C}.\) We denote this space \(FC(\mathcal R).\) Write \(V_{\Pi }\) for the space of the representation \(\Pi \) and \(V_\tau \) for that of \(\tau .\) Then, define the period \(\mathcal P: V_{\Pi } \times V_\tau \times FC(\mathcal R) \rightarrow \mathbb {C},\) by the formula

$$\begin{aligned} \mathcal P(\varphi _{\Pi }, \varphi _\tau , {\underline{r}}^{\theta (\phi )} ) = \int _{Z\backslash C_Q} {\underline{r}}^{\theta (\phi )}(g) \varphi _\Pi (g_1) \varphi _\tau (g_2) \, dg. \end{aligned}$$

Theorem 9.1

First suppose that \(\tau _1 \ne \tau _2.\) Then the following are equivalent:

  1. (1)

    \(L^S(s, \widetilde{\Pi }\times \tau _1)\) and \(L^S(s, \widetilde{\Pi }\times \tau _2)\) have poles at \(s=1.\)

  2. (2)

    \(\Pi \) is the weak lift of \(\tau _1\times \tau _2\)

  3. (3)

    the period \(\mathcal P\) does not vanish identically on \(V_\Pi \times V_\tau \times FC(\mathcal R).\)

Similarly, if \(\tau _1=\tau _2,\) then the following are equivalent:

  1. (1)

    \(L^S(s, \Pi \times \tau _1)\) has a pole at \(s=1.\)

  2. (2)

    \(\widetilde{\Pi }\) is the weak lift of \(\tau _1\times \tau '\) for some cuspidal representation \(\tau '\) of \(GL_2(\mathbb {A}),\)

  3. (3)

    the period \(\mathcal P\) does not vanish identically on \(V_\Pi \times V_\tau \times FC(\mathcal R).\)

Proof

The relationship between poles and the similitude theta correspondence was established in [2]. What is new here is the period condition, which follows from our earlier results. Indeed, for \(f \in {\text {Flat}}(\chi ), \phi \in \mathcal S({\text {Mat}}_{4\times 2}(\mathbb {A}))\) \(\varphi _\Pi \in V_\Pi \) and \(\varphi _\tau \in V_\tau ,\) the period \(\mathcal P(\varphi _\Pi , \varphi _\tau , R(f)^{\theta (\phi )})\) vanishes if and only if

$$\begin{aligned} {\text {Res}}_{s_1-s_2=4}{\text {Res}}_{s_1+s_2=6}I(f_{\chi ; s} , \varphi _\Pi , \varphi _\tau , \phi ) \ne 0. \end{aligned}$$

By [2], the local components of \(\Pi \) all satisfy H(15 / 34) and the local components of \(\tau \) all satisfy H(1 / 9). Hence each ramified local zeta integral is holomorphic at (5, 1) by Proposition 7.2. Moreover, by Proposition 7.3, each ramified local zeta integral is nonzero at (5, 1) for a suitable choice of data. The result follows. \(\square \)

Remark 9.2

Inspecting the various intertwining operators which appear in the constant term of our Eisenstein series along the Borel, one finds that some have poles along \(s_1-s_2=4\) and \(s_1+s_2=6\) of orders as high as three, as well as simple poles along \(s_1=5\) and \(s_2=4.\) However, it follows from Theorem 8.1 and Proposition 7.2 that the global integral can have at most a simple pole along \(s_1-s_2=4\) and a simple pole along \(s_1+s_2=6.\) It follows that any automorphic forms obtained by considering higher order singularities of the Eisenstein series either do not support our Fourier–Jacobi coefficient or have the property that their Fourier–Jacobi coefficients, regarded as smooth functions of moderate growth on \(C_Q(F)\backslash C_Q(\mathbb {A})\), are orthogonal to cuspforms.

10 A similar integral on \(GSO_{18}\)

In this section we consider the global integral (7) in the case \(n=3.\) Our unfolding does not produce an integral involving the Whittaker functions attached to our cusp forms, but it does reveal another intriguing connection with the theta correspondence.

As before, the space of double cosets \(P \backslash GSO_{18} / R_Q\) is represented by elements of the Weyl group, and

$$\begin{aligned} I(f_{\chi ;s}, \varphi , \phi ) = \sum _{w\in P \backslash GSO_{18} / R_Q} I_w(f_{\chi ;s}, \varphi , \phi ),\qquad \text { where }\quad \end{aligned}$$
$$\begin{aligned} I_w(f_{\chi ;s}, \varphi , \phi ) = \int \limits _{C_Q^{w}(F) \backslash C_Q(\mathbb {A})} \varphi (g) \int \limits _{U_Q^{w}(\mathbb {A}) \backslash U_Q(\mathbb {A}) } f_{\chi ;s}(wu_2 g) \int \limits _{[U_Q^{w}]} \theta (\phi , u_1u_2g) \, du_1 \, du_2\, dg, \end{aligned}$$

which is zero if \(\psi _l \big |_{Z(U_Q) \cap U_Q^w}\) is nontrivial, or if some parabolic subgroup of \(C_Q\) stabilizes the flag \(0 \subset \overline{U}_Q^w \subset (\overline{U}_Q^w)^\perp \) in \(U_Q/Z(U_Q),\) where \(\overline{U}_Q^w\) is the image of \(U_Q^w\) and \((\overline{U}_Q^w)^\perp \) is its perp space relative to the symplectic form defined by composing \(l:Z(U_Q) \rightarrow \mathbb {G}_a\) with the commutator map \(U_Q/Z(U_Q)\rightarrow Z(U_Q).\)

Lemma 10.1

Let \(w_\ell \) denote the longest element of the Weyl group of \(GSO_{18},\) let \(w_1\) be the shortest element of \((W \cap P) \cdot w_\ell \cdot (W \cap Q)\) and let \(w_2 = w_1 \cdot w[32].\) Then \(P w_2 R_Q\) is a Zariski open subset of \(GSO_{18}.\)

Proposition 10.2

\(I_w(f_{\chi ;s}, \varphi , \phi )\) is zero unless w is in the open double coset.

Proposition 10.3

Let \(U_0 \subset C_Q\) be given by

$$\begin{aligned} \left\{ u_0(x,x'):= \left( \begin{pmatrix} 1 &{}\quad x_1 &{}\quad x_2 &{}\quad x_3&{}\quad x_4&{}\quad x_5 \\ &{}\quad 1 &{}\quad 0&{}\quad 0&{}\quad x_6 &{}\quad *\\ &{}&{}\quad 1&{}\quad 0&{}\quad 0 &{}\quad *\\ &{}&{}&{}\quad 1&{}\quad 0&{}\quad *\\ &{}&{}&{}&{}\quad 1&{}\quad *\\ &{}&{}&{}&{}&{}\quad 1 \end{pmatrix}, \right. \right. \\ \left. \left. \begin{pmatrix} 1&{}\quad x_1'&{}\quad x_4' &{}\quad x_7 &{}\quad x_8 &{}\quad *\\ &{}\quad 1 &{}\quad x_6' &{}\quad x_9 &{}\quad *&{}\quad *\\ &{}&{}\quad 1&{}\quad 0&{}\quad -x_9&{}\quad *\\ &{}&{}&{}\quad 1&{}\quad -x_6&{}\quad *\\ &{}&{}&{}&{}\quad 1&{}\quad -x_1\\ &{}&{}&{}&{}&{}\quad 1 \end{pmatrix} \right) \;:\; \begin{matrix} x \in \mathbb {G}_a^9, \\ x' \in \mathbb {G}_a^3 \end{matrix} \right\} , \end{aligned}$$

where entries marked \(*\) are determined by symmetry, and for \(x \in \mathbb {A}^9\) and \(x' \in \mathbb {A}^3,\) let

$$\begin{aligned} \psi _{U_0}(u_0(x,x')) = \psi ( x_1+x_6-x_1'-x_6' +x_9). \end{aligned}$$

Let \(SL_2^{\alpha _3}\) be the copy of \(SL_2\) generated by \(U_{\pm \alpha _{3}},\) and let \(R_0\) be the product of \(U_0\) and \(SL_2^{\alpha _3}.\) Let \(\psi _{R_0}\) be the character of \(R_0\) which restricts to \(\psi _{U_0}\) and to the trivial character of \(SL_2^{\alpha _3}.\) Let

$$\begin{aligned} \varphi ^{(R_0, \psi _{R_0})}(c) = \int \limits _{[R_0]} \varphi (rc) \psi _{R_0}(r) \, dr = \int \limits _{[U_0]} \int \limits _{[SL_2^{\alpha _3}]} \varphi (uhc) \psi _{U_0}(u) \, dh\, du,\qquad (c \in C_Q(\mathbb {A})). \end{aligned}$$

Let \(V_4= \{ u(x,x'): x_i = x_i', i=1,4,6\}\subset U_0,\) and let

$$\begin{aligned} \xi _0 = \begin{pmatrix} 1&{}\quad 0&{}\quad 0\\ 0&{}\quad 1&{}\quad 0 \\ 0&{}\quad 0&{}\quad 0\\ 0&{}\quad 0&{}\quad 0\\ 0&{}\quad 0&{}\quad 1\\ 0&{}\quad 0&{}\quad 0 \end{pmatrix} \in {\text {Mat}}_{6\times 3}(F). \end{aligned}$$

Then

$$\begin{aligned} I_{w_2}(f_{\chi ;s}, \varphi , \phi ) = \int \limits _{Z(\mathbb {A})V_4(\mathbb {A}) \backslash C_Q(\mathbb {A})} \varphi ^{(R_0, \psi _{R_0})}(c) \int \limits _{U^{w_2}(\mathbb {A}) \backslash U(\mathbb {A})} f_{\chi ;s}(w_2 uc) \left[ \omega _\psi ( uc).\phi \right] (\xi _0)\, du \, dc. \end{aligned}$$

Remark 10.4

It was shown in [15] that the period we obtain in the \(GSp_6\) characterizes the image of the theta lift from \(SO_6\) to \(Sp_6.\)

Proof

First, \( U_Q^{w_2}\) is the set of all \(u_Q( 0,Y,0)\) such that rows 2,  5 and 6 of Y are zero. It follows that

$$\begin{aligned} \theta ^{U_Q^{w_2}}( \phi , u_1 u_2 g):= \int \limits _{[U_Q^{w_2}]} \theta ( \phi , u_1 u_2 g) \, du_1 = \sum _{\xi } [\omega _\psi (u_2 g).\phi ]( \xi ), \end{aligned}$$

where the sum is over \(\xi \in {\text {Mat}}_{6 \times 3}(F)\) such that rows 3, 4 and 1 are zero. Next, the identification of \(C_Q\) with a subgroup of \(GSp_6 \times GSO_6\) identifies \(C_Q^{w_2}\) with the subset of elements of the form

$$\begin{aligned}&\left( \begin{pmatrix} t &{}\quad x_1 &{}\quad x_2 &{}\quad x_3&{}\quad x_4&{}\quad x_5 \\ &{}\quad a &{}\quad 0&{}\quad 0&{}\quad b &{}\quad * \\ &{}\quad &{}\quad a'&{}\quad b' &{}\quad 0 &{}\quad *\\ &{}\quad &{}\quad c'&{}\quad d'&{}\quad 0&{}\quad *\\ &{}\quad c&{}\quad &{}\quad &{}\quad d&{}\quad *\\ &{}\quad &{}\quad &{}\quad &{}\quad &{}\quad t\lambda \end{pmatrix}, \begin{pmatrix} g&{}\quad W \\ &{}\quad _tg^{-1}\lambda \end{pmatrix} \right) ,\nonumber \\&\quad \; t \in GL_1, \; \begin{pmatrix} a&{}\quad b\\ c&{}\quad d \end{pmatrix}, \begin{pmatrix} a'&{}\quad b'\\ c'&{}\quad d' \end{pmatrix} \in GL_2, \; g \in GL_3. \end{aligned}$$
(50)

Now we can expand \(\varphi \) along the abelian unipotent subgroup which consists of elements of the form \(u_1(W):=\left( I_6, \;\begin{pmatrix} I_3 &{}\quad W \\ &{}\quad I_3 \end{pmatrix}\right) , \; W \in {{}^2 \wedge _{3}}.\) The constant term is of course zero. The group \(C_Q^{w_2}\) acts transitively on the nontrivial characters. As a representative for the open orbit we select the character \(\psi _{2,1}(u_1(W)) := \psi (W_{2,1}).\) The stabilizer of this representative can be described in terms of the coordinates from (50) as the set of elements of \(C_Q^{w_2}\) such that \(g \in GL_3\) is of the form \(\begin{pmatrix} t_1 &{} u \\ &{} g_1 \end{pmatrix}\) with \(g_1 \in GL_2\) and \(\det g_1 = \lambda .\) Denote this group by \(C_1^{w_2}.\) Now we write the integral as a double integral, with the inner integral being

$$\begin{aligned} \int \limits _{[{{}^2 \wedge _{3}}]}\int \limits _{U_Q^{w_2}(\mathbb {A}) \backslash U_Q(\mathbb {A}) } f_{\chi ;s}( w_2 u_2 u_1(W) g) \theta ^{U_Q^{w_2}}( \phi , u_2 u_1(W) g)\, du_2 \psi ^{-1}_{2,1}(W)\, dW. \end{aligned}$$
(51)

Now,

$$\begin{aligned} u_Q( \xi , 0, 0 ) u_1(W) = u_1(W) u_Q(\xi , 0,0) u_Q( 0,\xi W, -\xi W \, _t\xi ), \qquad ( W \in {{}^2 \wedge _{3}}, \; \xi \in {\text {Mat}}_{6\times 3}). \end{aligned}$$

It follows that (51) is equal to

$$\begin{aligned} \int \limits _{[{{}^2 \wedge _{3}}]}\int \limits _{U_Q^{w_2}(\mathbb {A}) \backslash U_Q(\mathbb {A}) } f_{\chi ;s}( w_2 u_2 g) \sum _{\xi } [\omega _\psi (u_2 g).\phi ](\xi ) \psi _l( \xi W \,_t\xi ) \psi ^{-1}_{2,1}(W)\, du_2 \, dW, \end{aligned}$$

with \(\xi \) summed over \(6\times 3\) matrices such that rows 3, 4 and 6 are zero. Clearly the integral on W picks off the terms such that \(\psi _l( \xi W \,_t\xi ) \psi ^{-1}_{2,1}(W)\) is trivial. Now, direct calculation shows that

$$\begin{aligned} \xi= & {} \begin{pmatrix} \xi _1 &{}\quad \xi _2 &{}\quad \xi _3 \\ \xi _4 &{}\quad \xi _5 &{}\quad \xi _6 \\ 0&{}\quad 0&{}\quad 0\\ 0&{}\quad 0&{}\quad 0 \\ \xi _7&{}\quad \xi _8&{}\quad \xi _9 \\ 0&{}\quad 0&{}\quad 0 \end{pmatrix}, \qquad W = \begin{pmatrix} y_1 &{}\quad y_2 &{}\quad 0 \\ y_3 &{}\quad 0&{}\quad -y_2 \\ 0 &{}\quad -y_3&{}\quad -y_1 \end{pmatrix}, \implies \psi _l ( \xi W \, _t \xi )\\= & {} \psi \left( \det \begin{pmatrix} y_3 &{}\quad -y_1 &{}\quad y_2 \\ \xi _4 &{}\quad \xi _5 &{}\quad \xi _6 \\ \xi _7&{}\quad \xi _8&{}\quad \xi _9 \end{pmatrix} \right) . \end{aligned}$$

So, in the coordinates above, the condition for \(\psi _l( \xi W \,_t\xi ) \psi ^{-1}_{2,1}(W)\) to be trivial is \(\xi _4 = \xi _7 =0\) and \(\det \begin{pmatrix} \xi _5 &{}\quad \xi _6 \\ \xi _8&{}\quad \xi _9\end{pmatrix} = 1.\) Observe that if \(\xi _1\) is also zero, then the function \(g \mapsto [\omega _\psi (g).\phi ](\xi )\) is invariant on the left by \(\bigg \{ \left( I_6, \begin{pmatrix} u&{}\\ &{}\quad _tu^{-1} \end{pmatrix} \right) \in C_Q^{w_2}: u = \begin{pmatrix} 1&{}\quad x&{}\quad y\\ &{}\quad 1&{}\\ &{}&{}\quad 1\end{pmatrix}\in GL_3 \bigg \}.\) Thus, the contribution from such \(\xi \) is trivial by cuspidality. The group \(C_1^{w_2}\) permutes the remaining terms transitively, and the stabilizer of \(\xi _0\) is

$$\begin{aligned} C_2^{w_2}:= & {} \left\{ \left( \begin{pmatrix} t &{}\quad x_1 &{}\quad x_2 &{}\quad x_3&{}\quad x_4&{}\quad x_5 \\ &{}\quad a &{}\quad 0&{}\quad 0&{}\quad b &{}\quad *\\ &{}\quad &{}\quad a'&{}\quad b' &{}\quad 0 &{}\quad *\\ &{}\quad &{}\quad c'&{}\quad d'&{}\quad 0&{}\quad *\\ &{}\quad c&{}\quad &{}\quad &{}\quad d&{}\quad *\\ &{}\quad &{}\quad &{}\quad &{}\quad &{}\quad t\lambda \end{pmatrix}, \; \begin{pmatrix} g&{}\quad W \\ &{}\quad _tg^{-1}\lambda \end{pmatrix} \right) \in C_1^{w_2}\;: \right. \\&\quad \left. g = \begin{pmatrix} t&{}\quad x_1&{}\quad x_4 \\ &{}\quad a&{}\quad b \\ &{}\quad c &{}\quad d \end{pmatrix} \right\} . \end{aligned}$$

Expanding first on \(x_1\) and \(x_4,\) and then on the unipotent radical of the diagonally embedded \(GL_2,\) and using Lemma 3.7 two more times gives the result. \(\square \)