The Gross–Prasad conjecture and local theta correspondence

Abstract

We establish the Fourier–Jacobi case of the local Gross–Prasad conjecture for unitary groups, by using local theta correspondence to relate the Fourier–Jacobi case with the Bessel case established by Beuzart-Plessis. To achieve this, we prove two conjectures of Prasad on the precise description of the local theta correspondence for (almost) equal rank unitary dual pairs in terms of the local Langlands correspondence. The proof uses Arthur’s multiplicity formula and thus is one of the first examples of a concrete application of this “global reciprocity law”.

1 Introduction

In [15, 16, 23, 24], a restriction problem in the representation theory of classical groups was studied and a precise conjecture was formulated for this restriction problem. This so-called Gross–Prasad (GP) conjecture has generated much interest in recent years.

1.1 Restriction problem

In this paper, we shall focus on the restriction problem for unitary groups. Thus, let F be a nonarchimedean local field of characteristic 0 and residue characteristic p, and let E be a quadratic field extension of F. Let $V_{n+1}$ be a Hermitian space of dimension $n+1$ over E and $W_n$ a skew-Hermitian space of dimension n over E. Let $V_n \subset V_{n+1}$ be a nondegenerate subspace of codimension 1, so that we have a natural inclusion of their corresponding unitary groups $\mathrm {U}(V_{n}) \hookrightarrow \mathrm {U}(V_{n+1})$. In particular, if we set

$$\begin{aligned} G_n = \mathrm {U}(V_n) \times \mathrm {U}(V_{n+1}) \quad \text { or} \quad \mathrm {U}(W_n) \times \mathrm {U}(W_n) \end{aligned}$$

and

$$\begin{aligned} H_n = \mathrm {U}(V_n) \quad \text { or} \quad \mathrm {U}(W_n), \end{aligned}$$

then we have a diagonal embedding

$$\begin{aligned} \varDelta : H_n \hookrightarrow G_n. \end{aligned}$$

Let $\pi$ be an irreducible smooth representation of $G_n$. In the Hermitian case, one is interested in determining

$$\begin{aligned} \dim _\mathbb {C}{\text {Hom}}_{\varDelta H_n} ( \pi , \mathbb {C}). \end{aligned}$$

We shall call this the Bessel case (B) of the GP conjecture. In the skew-Hermitian case, the restriction problem requires another piece of data: a Weil representation $\omega _{\psi , {\chi }, W_n}$, where $\psi$ is a nontrivial additive character of F and $\chi$ is a character of $E^{\times }$ whose restriction to $F^{\times }$ is the quadratic character $\omega _{E/F}$ associated to E / F by local class field theory. Then one is interested in determining

$$\begin{aligned} \dim _\mathbb {C}{\text {Hom}}_{\varDelta H_n} ( \pi , \omega _{\psi ,\chi , W_n}). \end{aligned}$$

We shall call this the Fourier–Jacobi case (FJ) of the GP conjecture. To unify notation, we shall let $\nu = \mathbb {C}$ or $\omega _{\psi ,\chi , W_n}$ in the respective cases.

By surprisingly recent results of Aizenbud–Gourevitch–Rallis–Schiffmann [1] and Sun [56], it is known that the above Hom spaces have dimension at most 1. Thus the main issue is to determine when the Hom space is nonzero. In [15], an answer for this issue is formulated in the framework of the local Langlands correspondence, in its enhanced form due to Vogan [58] which takes into account all pure inner forms.

1.2 Local Langlands correspondence

More precisely, a pure inner form of $\mathrm {U}(V_n)$ is simply a group of the form $\mathrm {U}(V_n')$, where $V_n'$ is a Hermitian space of dimension n over E; likewise in the skew-Hermitian case. Thus, a pure inner form of $G_n$ is a group of the form

$$\begin{aligned} G_n' = \mathrm {U}(V_n') \times \mathrm {U}(V'_{n+1}) \quad \text { or} \quad \mathrm {U}(W'_n) \times \mathrm {U}(W''_n). \end{aligned}$$

We say that such a pure inner form is relevant if

$$\begin{aligned} V'_n \subset V'_{n+1} \quad \text { or} \quad W'_n = W''_n, \end{aligned}$$

and

$$\begin{aligned} V_{n+1}'/V_{n}' \cong V_{n+1}/V_{n} \end{aligned}$$

in the Hermitian case. If $G_n'$ is relevant, we set

$$\begin{aligned} H'_n = \mathrm {U}(V'_n) \quad \text { or} \quad \mathrm {U}(W'_n), \end{aligned}$$

so that we have a diagonal embedding

$$\begin{aligned} \varDelta : H'_n \hookrightarrow G'_n. \end{aligned}$$

Now suppose that $\phi$ is an L-parameter for the group $G_n$. Then $\phi$ gives rise to a Vogan L-packet $\varPi _{\phi }$ consisting of certain irreducible smooth representations of $G_n$ and its (not necessarily relevant) pure inner forms $G_n'$. Moreover, after fixing a Whittaker datum for $G_n$, there is a natural bijection

$$\begin{aligned} \varPi _{\phi } \longleftrightarrow {\text {Irr}}(S_{\phi }), \end{aligned}$$

where $S_{\phi }$ is the component group associated to $\phi$. Thus an irreducible smooth representation of $G_n$ is labelled by a pair $(\phi , \eta )$, where $\phi$ is an L-parameter for $G_n$ and $\eta$ is an irreducible character of $S_{\phi }$.

By the recent work of Arthur [2], Mok [44], and Kaletha–Mínguez–Shin–White [33], together with the stabilization of the twisted trace formula established by Waldspurger and Mœglin–Waldspurger [43], the local Langlands correspondence for unitary groups is now unconditional, except that the general case of the weighted fundamental lemma has not been written; the work of Chaudouard–Laumon [8] is limited to the case of split groups.

1.3 Gross–Prasad conjecture

With this short preparation, the GP conjecture can be loosely stated as follows:

Gross–Prasad conjecture

• (i) Given a generic L-parameter $\phi$ for $G_n$, there is a unique representation $\pi (\phi ,\eta )$ in the Vogan L-packet $\varPi _{\phi }$ such that $\pi (\phi ,\eta )$ is a representation of a relevant pure inner form $G_n'$ and such that

  $$\begin{aligned} {\text {Hom}}_{\varDelta H'_n} (\pi (\phi ,\eta ), \nu ) \ne 0. \end{aligned}$$

• (ii) There is a precise recipe for the distinguished character $\eta$ (which we will recall in Sect. 3.2 below).

In a stunning series of papers [61–64], Waldspurger has established the Bessel case of the GP conjecture for special orthogonal groups in the case of tempered L-parameters; the case of general generic L-parameters is then dealt with by Mœglin–Waldspurger [42]. Beuzart-Plessis [4–6] has since extended Waldspurger’s techniques to settle the Bessel case of the GP conjecture for unitary groups in the tempered case.

1.4 Purpose of this paper

The purpose of this paper is to establish the Fourier–Jacobi case of the GP conjecture, as well as two conjectures of  Prasad concerning local theta correspondence in the (almost) equal rank case.

Let us describe the main idea of the proof. For simplicity, we restrict ourselves to the case of tempered L-parameters here. The Bessel and Fourier–Jacobi cases of the GP conjecture are related by the local theta correspondence. More precisely, there is a see-saw diagram

and the associated see-saw identity reads:

$$\begin{aligned} {\text {Hom}}_{\mathrm {U}(W_n)}( \varTheta _{\psi , \chi , V_n, W_n}(\sigma )&\otimes&\omega _{\psi , \chi , V_1, W_n}, \pi )\\\cong & {} {\text {Hom}}_{\mathrm {U}(V_n)}(\varTheta _{\psi , \chi , V_{n+1}, W_n}(\pi ), \sigma ) \end{aligned}$$

for irreducible smooth representations $\pi$ of $\mathrm {U}(W_n)$ and $\sigma$ of $\mathrm {U}(V_n)$. Hence the left-hand side of the see-saw identity concerns the Fourier–Jacobi case (FJ) whereas the right-hand side concerns the Bessel case (B). It is thus apparent that precise knowledge of the local theta correspondence for unitary groups of (almost) equal rank will give the precise relation of (FJ) to (B).

More precisely, one would need to know:

($\varTheta$):

For irreducible tempered representations $\pi$ and $\sigma$, the big theta lifts $\varTheta _{\psi , \chi , V_{n+1}, W_n}(\pi )$ and $\varTheta _{\psi , \chi , V_n, W_n}(\sigma )$ are irreducible (if nonzero).

(P1):

If $\sigma$ has parameter $(\phi , \eta )$ and $\varTheta _{\psi , \chi , V_n, W_n}(\sigma )$ has parameter $(\phi ', \eta ')$, then $(\phi ',\eta ')$ can be precisely described in terms of $(\phi ,\eta )$.

(P2):

Likewise, if $\pi$ has parameter $(\phi , \eta )$ and $\varTheta _{\psi , \chi , V_{n+1}, W_n}(\pi )$ has parameter $(\phi ', \eta ')$, then $(\phi ',\eta ')$ can be precisely described in terms of $(\phi ,\eta )$.

In fact, in [47, 48],  Prasad has formulated precise conjectures regarding (P1) and (P2) for the theta correspondence for $\mathrm {U}(V_n) \times \mathrm {U}(W_n)$ and $\mathrm {U}(V_{n+1}) \times \mathrm {U}(W_n)$ respectively; we shall recall his conjectures precisely in Sect. 4. We shall also denote by (weak P1) the part of the conjecture (P1) concerning only the correspondence of L-parameters $\phi \mapsto \phi '$; likewise we have (weak P2). Then we recall that in our earlier paper [17], we have shown:

Proposition 1.1

The statements $(\varTheta )$, (weak P1) and (weak P2) hold.

Using Proposition 1.1, the first observation of this paper is:

Proposition 1.2

Assume (B) and (P2). Then (FJ) and (P1) follow.

In view of Proposition 1.2 and the work of Beuzart-Plessis [4–6], it remains to show the statement (P2), and our main result is:

Theorem 1.3

The conjecture $(P2 )$, and hence $(FJ )$ and $(P1 )$, holds.

Let us make a few comments about the results:

• In fact, we prove (P1) and (P2) for all (not necessarily tempered nor generic) L-parameters.

• We mention a related result of Mœglin [41] about the local theta correspondence for symplectic-orthogonal dual pairs of arbitrary rank. She considered A-packets for a large class of A-parameters, including all tempered L-parameters, and then determined the analog of the correspondence $(\phi , \eta ) \mapsto (\phi ', \eta ')$ in the sense of Arthur, assuming that the correspondence is known for supercuspidal (and slightly more general) representations.

• It is interesting to note that in Proposition 1.2, the roles of (P1) and (P2) can be switched. In other words, it is also sufficient to prove (P1) in order to prove (FJ). We shall explain in the next subsection why we prefer to prove (P2).

• In [15], both the Bessel (B) and Fourier–Jacobi (FJ) cases of the GP conjecture were formulated for pairs of spaces $V_n \subset V_{n+2k+1}$ or $W_n \subset W_{n+2k}$ for any nonnegative integer k and for any generic L-parameters for $\mathrm {U}(V_n) \times \mathrm {U}(V_{n+2k+1})$ or $\mathrm {U}(W_n) \times \mathrm {U}(W_{n+2k})$. Beuzart-Plessis [4–6] has in fact verified (B) for all tempered L-parameters for $\mathrm {U}(V_n) \times \mathrm {U}(V_{n+2k+1})$. In §9, we check that the argument as in [42] gives (B) for all generic L-parameters for $\mathrm {U}(V_n) \times \mathrm {U}(V_{n+2k+1})$ and then show that Theorem 1.3 continues to hold for all generic L-parameters for $\mathrm {U}(W_n) \times \mathrm {U}(W_n)$.

• On the other hand, it was shown in [15, Theorem 19.1] that the GP conjecture in the case of generic L-parameters for $\mathrm {U}(W_n) \times \mathrm {U}(W_{n+2k})$ (for all $k>0$) follows from that for $\mathrm {U}(W_n) \times \mathrm {U}(W_n)$. Namely, we can deduce from Theorem 1.3 the following:

Corollary 1.4

The Fourier–Jacobi case of the GP conjecture holds for all generic L-parameters for $\mathrm {U}(W_n) \times \mathrm {U}(W_{n+2k})$ for any $k \ge 0$.

1.5 Prasad’s conjectures

Given Proposition 1.1, the main work is to determine how $\eta '$ depends on $(\phi , \eta )$ in (P1) and (P2). In fact, the precise determination of $\eta '$ in (P1) is a very subtle issue, as it depends on certain local roots numbers. In the case of (P2), the dependence of $\eta '$ on $(\phi ,\eta )$ is more simplistic.

The proof of (P2) proceeds by the following steps:

• First, by our results in [17], the nontempered case can be reduced to the tempered case on smaller unitary groups.

• Next, we show that the tempered case can be reduced to the square-integrable case on smaller unitary groups. This is achieved by a nontrivial extension of the techniques in the PhD thesis of the second author [31] and uses the delicate details of the normalization of the intertwining operators involved in the local intertwining relation [2, 33, 44].

• Finally, we show the square-integrable case by a global argument. More precisely, we shall globalize an irreducible square-integrable representation $\pi$ of $\mathrm {U}(W_n)$ to an irreducible cuspidal automorphic representation $\varPi = \otimes _v \varPi _v$ such that

  • $\varPi _v$ is not square-integrable for all places outside the place of interest, so that (P2) is known for $\varPi _v$ outside the place of interest,

  • $\varPi$ has tempered A-parameter whose global component group is equal to the local component group of the L-parameter of $\pi$,

  • $\varPi$ has nonzero global theta lift to a unitary group which globalizes $\mathrm {U}(V_{n+1})$.

  The desired result then follows for the place of interest by applying Arthur’s multiplicity formula for the automorphic discrete spectrum, which can be viewed as a sort of product formula (see (6.3)).

We can now explain why we prefer to prove (P2) rather than (P1). Note that one could attempt to follow the same strategy of proof for the statement (P1). However, in the globalization step above, we need to ensure that $\varPi$ has nonzero global theta lift to a certain unitary group. For the case of (P1), the nonvanishing of this global theta lift is controlled by the nonvanishing of $L(\frac{1}{2}, \varPi )$, and it is well-known that the nonvanishing of this central critical value is a very subtle issue with arithmetic implications. On the other hand, for the statement (P2), the nonvanishing of the global theta lift of $\varPi$ is governed by the nonvanishing of $L(1,\varPi )$. Now it is certainly much easier to ensure the nonvanishing of $L(1, \varPi )$ compared to $L(\frac{1}{2},\varPi )$. For example, if $\varPi$ has tempered A-parameter, then one knows that $L(1, \varPi ) \ne 0$. It is for this reason that we prove (P2) rather than (P1).

1.6 3 Birds and 2 stones

To summarise, in proving our main theorem, we have killed “3 birds” [i.e. (FJ), (P1) and (P2)] with “2 stones” [i.e. (B) and Arthur’s multiplicity formula], though it is probably more accurate to describe the latter as two cannon balls. We stress however that no animals (besides the two authors) have suffered in the preparation of this article.

1.7 Notation

Let F be a nonarchimedean local field of characteristic 0 and residue characteristic p. We fix an algebraic closure $\bar{F}$ of F. Let ${\varGamma = \text { Gal}(\bar{F}/F)}$ be the absolute Galois group of F and $W_F$ the Weil group of F. Let $|\cdot |_F$ be the normalized absolute value on F. We fix a nontrivial additive character $\psi$ of F.

Let E be a quadratic field extension of F and $\omega _{E/F}$ the quadratic character of $F^{\times }$ associated to E / F by local class field theory. Let c denote the nontrivial Galois automorphism of E over F. Let ${\text {Tr}}_{E/F}$ and ${\text {N}}_{E/F}$ be the trace and norm maps from E to F. We choose an element $\delta \in E^{\times }$ such that ${\text {Tr}}_{E/F}(\delta ) = 0$. We write $| \cdot | = |\cdot |_E$ for the normalized absolute value on E. Let $\psi _E$ be the nontrivial additive character of E defined by $\psi _E = \psi \circ {\text {Tr}}_{E/F}$.

If G is a linear algebraic group over F, we identify G with its group of F-rational points G(F). For any totally disconnected locally compact group G, let $\mathbbm {1}_G$ be the trivial representation of G and ${\text {Irr}}(G)$ the set of equivalence classes of irreducible smooth representations of G. For any set X, let $1_X$ be the identity map of X. For any positive integer n, let $1_n$ be the identity matrix in $\mathrm {GL}_n$.

2 Local Langlands correspondence

In this section, we summarize some properties of the local Langlands correspondence for unitary groups.

2.1 Hermitian and skew-Hermitian spaces

Fix $\varepsilon = \pm 1$. Let V be a finite dimensional vector space over E equipped with a nondegenerate $\varepsilon$-Hermitian c-sesquilinear form

$$\begin{aligned} \langle \cdot , \cdot \rangle _V : V \times V \longrightarrow E. \end{aligned}$$

Thus we have

$$\begin{aligned} \langle a v, b w \rangle _V= & {} a b^c \langle v, w \rangle _V,\\ \langle w, v \rangle _V= & {} \varepsilon \cdot \langle v, w \rangle _V^c \end{aligned}$$

for $v, w \in V$ and $a, b \in E$. Put $n = \dim V$ and ${\text {disc}}V = (-1)^{(n-1)n/2} \cdot \det V$, so that

$$\begin{aligned} {\text {disc}}V \in {\left\{ \begin{array}{ll} F^{\times } / \mathrm {N}_{E/F}(E^{\times }) &{}\quad {\text { if } \varepsilon = +1;} \\ \delta ^n \cdot F^{\times } / \mathrm {N}_{E/F}(E^{\times }) &{}\quad {\text { if } \varepsilon = -1.} \end{array}\right. } \end{aligned}$$

We define $\epsilon (V) = \pm 1$ by

$$\begin{aligned} \epsilon (V) = {\left\{ \begin{array}{ll} \omega _{E/F}({\text {disc}}V) &{}\quad {\text { if } \varepsilon = +1;} \\ \omega _{E/F}(\delta ^{-n} \cdot {\text {disc}}V) &{}\quad {\text { if } \varepsilon =-1.} \end{array}\right. } \end{aligned}$$

Given a positive integer n, there are precisely two isometry classes of n-dimensional $\varepsilon$-Hermitian spaces V, which are distinguished from each other by their signs $\epsilon (V)$. Note that $\epsilon (V)$ depends on the choice of $\delta$ if $\varepsilon = -1$ and n is odd. Let $\mathrm {U}(V)$ be the unitary group of V, i.e. the connected reductive linear algebraic group over F defined by

$$\begin{aligned} \mathrm {U}(V) = \{ g \in \mathrm {GL}(V) \, | \, \langle g v, g w \rangle _V = \langle v, w \rangle _V \quad \text { for } v, w \in V\}. \end{aligned}$$

If $n=0$, we interpret $\mathrm {U}(V)$ as the trivial group $\{ 1 \}$.

2.2 L-parameters and component groups

Let $W_E$ be the Weil group of E and $WD _E = W_E \times \mathrm {SL}_2(\mathbb {C})$ the Weil–Deligne group of E. We say that a continuous homomorphism $\phi : WD _E \rightarrow \mathrm {GL}_n(\mathbb {C})$ is a representation of $WD _E$ if

• $\phi$ is semisimple,

• the restriction of $\phi$ to $\mathrm {SL}_2(\mathbb {C})$ is algebraic.

We say that $\phi$ is tempered if the image of $W_E$ is bounded. Let $\phi ^{\vee }$ be the contragredient representation of $\phi$ defined by $\phi ^\vee (w) = {}^t\phi (w)^{-1}$. Fix $s \in W_F \backslash W_E$ and define a representation $\phi ^c$ of $WD _E$ by $\phi ^c(w) = \phi (sws^{-1})$. Then the equivalence class of $\phi ^c$ is independent of the choice of s. We say that $\phi$ is conjugate self-dual if there is a nondegenerate bilinear form $B : \mathbb {C}^n \times \mathbb {C}^n \rightarrow \mathbb {C}$ which satisfies

$$\begin{aligned} B(\phi (w) x, \phi ^c(w) y) = B(x, y) \end{aligned}$$

for all $w \in WD _E$ and $x, y \in \mathbb {C}^n$. Namely, $\phi$ is conjugate self-dual if and only if $\phi ^c$ is equivalent to $\phi ^\vee$. For $b = \pm 1$, we say that $\phi$ is conjugate self-dual with sign b if there is a nondegenerate bilinear form $B : \mathbb {C}^n \times \mathbb {C}^n \rightarrow \mathbb {C}$ which satisfies the above condition and the condition that

$$\begin{aligned} B(y, x) = b \cdot B(x, \phi (s^2) y) \end{aligned}$$

for all $x, y \in \mathbb {C}^n$. Note that the sign b depends not only on $\phi$ but also on B. We also say that $\phi$ is conjugate orthogonal (resp. conjugate symplectic) if it is conjugate self-dual with sign $+1$ (resp. $-1$). If $\phi$ is conjugate self-dual with sign b (with respect to a bilinear form B), then $\det \phi$ is conjugate self-dual with sign $b^n$. By [15, Lemma 3.4], a character $\chi$ of $E^\times$ (or rather the character of $WD _E$ associated to $\chi$ by local class field theory) is conjugate orthogonal (resp. conjugate symplectic) if and only if $\chi |_{F^\times } = \mathbbm {1}_{F^\times }$ (resp. $\chi |_{F^\times } = \omega _{E/F}$).

By [15, Sect. 8], an L-parameter for the unitary group $\mathrm {U}(V)$ is an n-dimensional conjugate self-dual representation $\phi$ of $WD _E$ with sign $(-1)^{n-1}$. We may decompose $\phi$ into a direct sum

$$\begin{aligned} \phi = \bigoplus _i m_i \phi _i \end{aligned}$$

with pairwise inequivalent irreducible representations $\phi _i$ of $WD _E$ and multiplicities $m_i$. We say that $\phi$ is square-integrable if it is multiplicity-free (so that $m_i=1$ for all i) and $\phi _i$ is conjugate self-dual with sign $(-1)^{n-1}$ for all i.

For an L-parameter $\phi$ for $\mathrm {U}(V)$, fix a bilinear form B as above and let ${\text {Aut}}(\phi ,B)$ be the group of elements in $\mathrm {GL}_n(\mathbb {C})$ which centralize the image of $\phi$ and preserve B. Let

$$\begin{aligned} S_{\phi } = {\text {Aut}}(\phi , B) / {\text {Aut}}(\phi , B)^0 \end{aligned}$$

be the component group of $\phi$, where ${\text {Aut}}(\phi , B)^0$ is the identity component of ${\text {Aut}}(\phi , B)$. As shown in [15, Sect. 8], $S_{\phi }$ has an explicit description of the form

$$\begin{aligned} S_{\phi } = \prod _j (\mathbb {Z}/ 2\mathbb {Z}) a_j \end{aligned}$$

with a canonical basis $\{ a_j \}$, where the product ranges over all j such that $\phi _j$ is conjugate self-dual with sign $(-1)^{n-1}$. In particular, $S_{\phi }$ is an elementary abelian 2-group. We shall let $z_\phi$ denote the image of $-1 \in \mathrm {GL}_n(\mathbb {C})$ in $S_\phi$. More explicitly, we have

$$\begin{aligned} z_{\phi } = (m_j a_j) \in \prod _j (\mathbb {Z}/ 2\mathbb {Z}) a_j. \end{aligned}$$

2.3 Local Langlands correspondence

The local Langlands correspondence for general linear groups, which was established by Harris–Taylor [26], Henniart [29], and Scholze [51], is a certain bijection between ${\text {Irr}}(\mathrm {GL}_n(E))$ and equivalence classes of n-dimensional representations of $WD _E$. This bijection satisfies natural properties which determine it uniquely. For example, if $\pi$ is an irreducible smooth representation of $\mathrm {GL}_n(E)$ with central character $\omega _{\pi }$ and $\phi$ is the n-dimensional representation of $WD _E$ associated to $\pi$, then

• $\omega _\pi = \det \phi$,

• $\pi$ is essentially square-integrable if and only if $\phi$ is irreducible,

• $\pi$ is tempered if and only if $\phi$ is tempered.

The local Langlands correspondence (as enhanced by Vogan [58]) for unitary groups says that there is a canonical partition

$$\begin{aligned} {\text {Irr}}(\mathrm {U}(V^+)) \sqcup {\text {Irr}}(\mathrm {U}(V^-)) = \bigsqcup _{\phi } \varPi _{\phi }, \end{aligned}$$

where $V^+$ and $V^-$ are the n-dimensional $\varepsilon$-Hermitian spaces with $\epsilon (V^+) = +1$ and $\epsilon (V^-) = -1$, the disjoint union on the right-hand side runs over all equivalence classes of L-parameters $\phi$ for $\mathrm {U}(V^\pm )$, and $\varPi _{\phi }$ is a finite set of representations known as a Vogan L-packet. We may decompose $\varPi _{\phi }$ as

$$\begin{aligned} \varPi _{\phi } = \varPi _{\phi }^+\sqcup \varPi _{\phi }^{-}, \end{aligned}$$

where for $\epsilon = \pm 1$, $\varPi _{\phi }^{\epsilon }$ consists of the representations of $\mathrm {U}(V^{\epsilon })$ in $\varPi _{\phi }$.

2.4 Whittaker data

To describe the L-packet $\varPi _{\phi }$ more precisely, it is necessary to choose a Whittaker datum, which is a conjugacy class of pairs $(N, \psi _N)$, where

• N is the unipotent radical of a Borel subgroup of the quasi-split unitary group $\mathrm {U}(V^+)$,

• $\psi _N$ is a generic character of N.

Then relative to this datum, there is a canonical bijection

$$\begin{aligned} J_{\psi _N}: \varPi _{\phi } \longleftrightarrow {\text {Irr}}( S_{\phi }). \end{aligned}$$

When n is odd, such a datum is canonical. When n is even, as explained in [15, Sect. 12], it is determined by the choice of an ${\text {N}}_{E/F}(E^{\times })$-orbit of nontrivial additive characters

$$\begin{aligned} {\left\{ \begin{array}{ll} \psi ^E:E/F \rightarrow \mathbb {C}^{\times } &{} {\text { if }\varepsilon = +1;} \\ \psi :F \rightarrow \mathbb {C}^{\times } &{} {\text { if } \varepsilon = -1.} \end{array}\right. } \end{aligned}$$

According to this choice, we write

$$\begin{aligned} {\left\{ \begin{array}{ll} J_{\psi ^E} &{} {\text { if } \varepsilon = +1;} \\ J_{\psi } &{} {\text { if } \varepsilon = -1} \end{array}\right. } \end{aligned}$$

for $J_{\psi _N}$. We formally adopt the same notation when n is odd. Suppose that $\varepsilon = +1$, so that $V^+$ and $V^-$ are Hermitian spaces. Let $W^+ = \delta \cdot V^+$ be the space $V^+$ equipped with the skew-Hermitian form $\delta \cdot \langle \cdot , \cdot \rangle _{V^+}$. Similarly, we define the skew-Hermitian space $W^- = \delta \cdot V^-$. Then for $\epsilon = \pm 1$, $\mathrm {U}(V^\epsilon )$ and $\mathrm {U}(W^\epsilon )$ are physically equal. For a given $\phi$, let $J_{\psi ^E}$ and $J_{\psi }$ be the above bijections for $\mathrm {U}(V^{\pm })$ and $\mathrm {U}(W^{\pm })$ respectively. One has:

• if n is even, then

  $$\begin{aligned} J_{\psi ^E} = J_{\psi } \Longleftrightarrow \psi ^E(x) = \psi \Big ( \tfrac{1}{2} {\text {Tr}}_{E/F}(\delta x)\Big ), \end{aligned}$$

• if n is odd, then $J_{\psi ^E} = J_{\psi }$.

Having fixed the Whittaker datum $(N, \psi _N)$, we shall write $\pi (\phi , \eta )$ or simply $\pi (\eta )$ for the irreducible smooth representation in $\varPi _{\phi }$ corresponding to $\eta \in {\text {Irr}}(S_{\phi })$ under the bijection $J_{\psi _N}$. If $\phi$ is tempered, then for any Whittaker datum $(N, \psi '_N)$, there is a unique $(N,\psi '_N)$-generic representation of $\mathrm {U}(V^+)$ in $\varPi _{\phi }$ by [5, Lemme 7.10.1], and the irreducible characters of $S_{\phi }$ associated to these generic representations under the bijection $J_{\psi _N}$ are described as follows:

• The unique $(N,\psi _N)$-generic representation of $\mathrm {U}(V^+)$ in $\varPi _\phi$ corresponds to the trivial character of $S_{\phi }$.

• When n is even, there are precisely two Whittaker datum. If $(N, \psi _N')$ is not conjugate to $(N, \psi _N)$, then by [32, Sect. 3], the unique $(N,\psi _N')$-generic representation of $\mathrm {U}(V^+)$ in $\varPi _\phi$ corresponds to the character $\eta _-$ of $S_{\phi }$ given by

  $$\begin{aligned} \eta _-(a_j) = (-1)^{\dim \phi _j}. \end{aligned}$$

The character $\eta _-$ has a role even when n is odd. Indeed, if n is odd, we may take $V^- = a \cdot V^+$, i.e. the space $V^+$ equipped with the Hermitian form $a \cdot \langle \cdot , \cdot \rangle _{V^+}$, where $a \in F^{\times } \backslash {\text {N}}_{E/F}(E^{\times })$. Then $\mathrm {U}(V^+)$ and $\mathrm {U}(V^-)$ are physically equal. Under this identification, we have

$$\begin{aligned} \varPi ^+_{\phi } = \varPi ^-_{\phi } \end{aligned}$$

for any $\phi$. Let $\pi = \pi (\phi , \eta )$ be a representation of $\mathrm {U}(V^+)$ in $\varPi _{\phi }$. If we regard $\pi$ as a representation of $\mathrm {U}(V^-)$ via the above identification, then it has associated character $\eta \cdot \eta _-$. In particular, if $\phi$ is tempered, then the unique $(N,\psi _N)$-generic representation of $\mathrm {U}(V^-)$ in $\varPi _\phi$ corresponds to $\eta _-$.

2.5 Properties of the local Langlands correspondence

We highlight some properties of the local Langlands correspondence which are used in this paper:

• $\pi (\phi ,\eta )$ is a representation of $\mathrm {U}(V^{\epsilon })$ if and only if $\eta (z_\phi ) = \epsilon$.

• $\pi (\phi ,\eta )$ is square-integrable if and only if $\phi$ is square-integrable.

• $\pi (\phi ,\eta )$ is tempered if and only if $\phi$ is tempered.

• If $\phi$ is tempered but not square-integrable, then we can write

  $$\begin{aligned} \phi = \phi _1 \oplus \phi _0 \oplus (\phi _1^c)^\vee , \end{aligned}$$

where

• $\phi _1$ is a k-dimensional irreducible representation of $WD _E$ for some positive integer k,

• $\phi _0$ is a tempered L-parameter for $\mathrm {U}(V_0^{\pm })$, where $V_0^{\pm }$ are the $\varepsilon$-Hermitian spaces of dimension $n-2k$ over E.

Note that there is a natural embedding $S_{\phi _0} \hookrightarrow S_\phi$. Let $\eta _0 \in {\text {Irr}}(S_{\phi _0})$ and put $\epsilon = \eta _0(z_{\phi _0})$. We can write

$$\begin{aligned} V^\epsilon = X \oplus V_0^\epsilon \oplus X^*, \end{aligned}$$

where X and $X^*$ are k-dimensional totally isotropic subspaces of $V^\epsilon$ such that $X \oplus X^*$ is nondegenerate and orthogonal to $V_0^\epsilon$. Let P be the maximal parabolic subgroup of $\mathrm {U}(V^\epsilon )$ stabilizing X and M its Levi component stabilizing $X^*$, so that

$$\begin{aligned} M \cong \mathrm {GL}(X) \times \mathrm {U}(V_0^\epsilon ). \end{aligned}$$

Let $\tau$ be the irreducible (unitary) square-integrable representation of $\mathrm {GL}(X)$ associated to $\phi _1$, and let $\pi _0 = \pi (\phi _0, \eta _0)$ be the irreducible tempered representation of $\mathrm {U}(V_0^\epsilon )$ in $\varPi _{\phi _0}$ corresponding to $\eta _0$. Then the induced representation ${\text {Ind}}^{\mathrm {U}(V^\epsilon )}_P(\tau \otimes \pi _0)$ has a decomposition

$$\begin{aligned} {\text {Ind}}^{\mathrm {U}(V^\epsilon )}_P(\tau \otimes \pi _0) = \bigoplus _\eta \pi (\phi , \eta ), \end{aligned}$$

where the sum ranges over all $\eta \in {\text {Irr}}(S_{\phi })$ such that $\eta |_{S_{\phi _0}} = \eta _0$. Moreover, if $\phi _1$ is conjugate self-dual, let

$$\begin{aligned} R(w, \tau \otimes \pi _0) \in {\text {End}}_{\mathrm {U}(V^\epsilon )}({\text {Ind}}^{\mathrm {U}(V^\epsilon )}_P(\tau \otimes \pi _0)) \end{aligned}$$

be the normalized intertwining operator defined in Sect. 7.3 below, where w is the unique nontrivial element in the relative Weyl group for M. Then the restriction of $R(w, \tau \otimes \pi _0)$ to $\pi (\phi , \eta )$ is the scalar multiplication by

$$\begin{aligned} {\left\{ \begin{array}{ll} \epsilon ^k \cdot \eta (a_1) &{}\quad {\text { if } \phi _1 \text { has sign } (-1)^{n-1};} \\ \epsilon ^k &{}\quad {\text { if } \phi _{1} \text { has sign } (-1)^n,} \end{array}\right. } \end{aligned}$$

(2.1)

where $a_1$ corresponds to $\phi _{1}$. These properties follow from the definition of $\eta$, induction in stages [33, Sect. 2.7], and the local intertwining relation [44, Theorem 3.4.3], [33, Theorem 2.6.2]. We also remark that the factor $\epsilon ^k$ arises from the splitting $s' : W_\psi (M,G) \rightarrow \pi _0(N_\psi (M,G))$ defined in [33, Sect. 2.4.1], which can be explicated by using an analog of Lemma 7.2 below for the dual group.

• In general, we can write

  $$\begin{aligned} \phi = \phi _1 \oplus \cdots \oplus \phi _r \oplus \phi _0 \oplus (\phi _r^c)^\vee \oplus \cdots \oplus (\phi _1^c)^\vee , \end{aligned}$$

where

• for $i = 1, \ldots , r$, $\phi _i$ is a $k_i$-dimensional representation of $WD _E$ of the form $\phi _i = \phi _i' \otimes |\cdot |^{e_i}$ for some tempered representation $\phi _i'$ of $WD _E$ and real number $e_i$ such that

  $$\begin{aligned} e_1> \cdots> e_r > 0, \end{aligned}$$

• $\phi _0$ is a tempered L-parameter for $\mathrm {U}(V_0^{\pm })$, where $V_0^{\pm }$ are the $\varepsilon$-Hermitian spaces of dimension $n-2(k_1+\cdots +k_r)$ over E.

Note that the natural map $S_{\phi _0} \rightarrow S_\phi$ is an isomorphism. Let $\eta \in {\text {Irr}}(S_{\phi })$ and put $\epsilon = \eta (z_\phi )$. We can write

$$\begin{aligned} V^\epsilon = X_1 \oplus \cdots \oplus X_r \oplus V_0^\epsilon \oplus X^*_r \oplus \cdots \oplus X^*_1, \end{aligned}$$

where $X_i$ and $X_i^*$ are $k_i$-dimensional totally isotropic subspaces of $V^\epsilon$ such that $X_i \oplus X_i^*$ are nondegenerate, mutually orthogonal, and orthogonal to $V_0^\epsilon$. Let P be the parabolic subgroup of $\mathrm {U}(V^\epsilon )$ stabilizing the flag

$$\begin{aligned} X_1 \subset X_1 \oplus X_2 \subset \dots \subset X_1 \oplus \cdots \oplus X_r \end{aligned}$$

and M its Levi component stabilizing the flag

$$\begin{aligned} X^*_1 \subset X^*_1 \oplus X^*_2 \subset \dots \subset X^*_1 \oplus \cdots \oplus X^*_r, \end{aligned}$$

so that

$$\begin{aligned} M \cong \mathrm {GL}(X_1) \times \cdots \times \mathrm {GL}(X_r) \times \mathrm {U}(V_0^\epsilon ). \end{aligned}$$

Then $\pi (\phi , \eta )$ is the unique irreducible quotient of the standard module

$$\begin{aligned} {\text {Ind}}^{\mathrm {U}(V^\epsilon )}_P(\tau _1 \otimes \cdots \otimes \tau _r \otimes \pi _0), \end{aligned}$$

where for $i = 1, \ldots , r$, $\tau _i$ is the irreducible essentially tempered representation of $\mathrm {GL}(X_i)$ associated to $\phi _i$, and $\pi _0 = \pi (\phi _0, \eta _0)$ is the irreducible tempered representation of $\mathrm {U}(V_0^\epsilon )$ in $\varPi _{\phi _0}$ corresponding to $\eta _0 := \eta |_{S_{\phi _0}} \in {\text {Irr}}(S_{\phi _0})$.

• If $\pi = \pi (\phi , \eta )$, then the contragredient representation $\pi ^\vee$ of $\pi$ has L-parameter $\phi ^{\vee }$ and associated character $\eta _{\pi ^\vee } = \eta \cdot \nu$, where

  $$\begin{aligned} \nu (a_j) = {\left\{ \begin{array}{ll} \omega _{E/F}(-1)^{\dim \phi _j} &{} {\text { if } n \text { is even;}} \\ 1 &{} {\text { if } n \text { is odd.}} \end{array}\right. } \end{aligned}$$

  Note that the component groups $S_{\phi }$ and $S_{\phi ^\vee }$ are canonically identified. In the case of unitary groups, this property follows from a result of Kaletha [32, Sect. 4].

3 Gross–Prasad conjecture

In this section, we explicate the statement of the Gross–Prasad conjecture for unitary groups. In particular, we recall the definition of the distinguished character $\eta$ of the component group.

3.1 Pairs of spaces

For $\epsilon = \pm 1$, let $V_n^\epsilon$ denote the n-dimensional Hermitian space with $\epsilon (V_n^\epsilon ) = \epsilon$ and $W_n^\epsilon$ the n-dimensional skew-Hermitian space with $\epsilon (W_n^\epsilon ) = \epsilon$, so that $W_n^\epsilon = \delta \cdot V_n^\epsilon$. For the Gross–Prasad conjecture, we consider the pair of spaces:

$$\begin{aligned} V_n^+ \subset V_{n+1}^+ \quad \text { or} \quad W_n^+ = W_n^+. \end{aligned}$$

Then the relevant pure inner form (other than itself) is

$$\begin{aligned} V_n^- \subset V_{n+1}^- \quad \text { or} \quad W_n^- = W_n^- \end{aligned}$$

and observe that

$$\begin{aligned} V_{n+1}^{\epsilon }/V_n^{\epsilon } \cong L_{(-1)^n}, \end{aligned}$$

where for $a \in F^{\times }$, $L_a$ denotes the Hermitian line with form $a \cdot {\text {N}}_{E/F}$. We have the groups

$$\begin{aligned} G_n^\epsilon = \mathrm {U}(V_n^\epsilon ) \times \mathrm {U}(V_{n+1}^\epsilon ) \quad \text { or} \quad \mathrm {U}(W_n^\epsilon ) \times \mathrm {U}(W_n^\epsilon ) \end{aligned}$$

and

$$\begin{aligned} H_n^\epsilon = \mathrm {U}(V_n^\epsilon ) \quad \text { or} \quad \mathrm {U}(W_n^\epsilon ), \end{aligned}$$

and the embedding

$$\begin{aligned} \varDelta : H_n^\epsilon \hookrightarrow G_n^\epsilon . \end{aligned}$$

We also have the Langlands–Vogan parametrization (depending on the choice of the Whittaker datum) relative to the fixed pair of spaces. For an L-parameter $\phi = \phi ^{\diamondsuit } \times \phi ^{\heartsuit }$ for $G_n^\pm$, the component group is:

$$\begin{aligned} S_{\phi } = S_{\phi ^{\diamondsuit }} \times S_{\phi ^{\heartsuit }}. \end{aligned}$$

In particular, under the local Langlands correspondence, the representation $\pi (\eta ) \in \varPi _{\phi }$ is a representation of a relevant pure inner form if and only if

$$\begin{aligned} \eta (z_{\phi ^{\diamondsuit }}, z_{\phi ^{\heartsuit }}) = 1, \end{aligned}$$

and $\pi (\eta )$ is a representation of $G_n^{\epsilon }$ if and only if

$$\begin{aligned} \eta (z_{\phi ^{\diamondsuit }}, 1) = \eta (1,z_{\phi ^{\heartsuit }}) = \epsilon . \end{aligned}$$

3.2 The distinguished character $\eta$

We shall now define a distinguished character $\eta \in {\text {Irr}}(S_\phi )$ when $\phi = \phi ^{\diamondsuit } \times \phi ^{\heartsuit }$. Writing

$$\begin{aligned} S_{\phi ^{\diamondsuit }} = \prod _i (\mathbb {Z}/2 \mathbb {Z}) a_i \quad \text { and} \quad S_{\phi ^{\heartsuit }} = \prod _j (\mathbb {Z}/ 2\mathbb {Z}) b_j, \end{aligned}$$

we thus need to specify the signs $\eta (a_i) = \pm 1$ and $\eta (b_j) = \pm 1$. We consider the Bessel and Fourier–Jacobi cases separately.

• Bessel case. We fix a nontrivial character $\psi ^E$ of E / F which determines the local Langlands correspondence for the even unitary group in $G_n^\epsilon = \mathrm {U}(V_n^\epsilon ) \times \mathrm {U}(V_{n+1}^\epsilon )$. We set $\psi ^E_{-2}(x) = \psi ^E(-2 x)$ and define:

  $$\begin{aligned} {\left\{ \begin{array}{ll} \eta ^{\spadesuit }(a_i) = \epsilon \left( \frac{1}{2}, \phi ^{\diamondsuit }_i \otimes \phi ^{\heartsuit }, \psi ^E_{-2}\right) ; \\ \eta ^{\spadesuit }(b_j) = \epsilon \left( \frac{1}{2}, \phi ^{\diamondsuit } \otimes \phi ^{\heartsuit }_{j}, \psi ^E_{-2}\right) . \end{array}\right. } \end{aligned}$$

• Fourier–Jacobi case. In this case, we need to fix a nontrivial character $\psi$ of F and a character $\chi$ of $E^{\times }$ with $\chi |_{F^{\times }} = \omega _{E/F}$ to specify the Weil representation $\nu = \omega _{\psi ,\chi , W_n^\epsilon }$ of $\mathrm {U}(W_n^\epsilon )$. The recipe for the distinguished character $\eta ^{\clubsuit }$ of $S_{\phi }$ depends on the parity of $n = \dim W_n^\epsilon$.

  • If n is odd, recall that $\det W^+_n \in \delta \cdot {\text {N}}_{E/F}(E^{\times })$ and define

    $$\begin{aligned} {\left\{ \begin{array}{ll} \eta ^{\clubsuit }(a_i) = \epsilon \left( \frac{1}{2}, \phi ^{\diamondsuit }_{i} \otimes \phi ^{\heartsuit } \otimes \chi ^{-1}, \psi _2^E\right) ; \\ \eta ^{\clubsuit }(b_j) = \epsilon \left( \frac{1}{2}, \phi ^{\diamondsuit } \otimes \phi ^{\heartsuit }_{j} \otimes \chi ^{-1}, \psi _2^E\right) , \end{array}\right. } \end{aligned}$$

    where

    $$\begin{aligned} \psi ^E_2(x) = \psi ({\text {Tr}}_{E/F}(\delta x)). \end{aligned}$$

  • If n is even, the fixed character $\psi$ is used to fix the local Langlands correspondence for $\mathrm {U}(W_n^\epsilon )$. We set

    $$\begin{aligned} {\left\{ \begin{array}{ll} \eta ^{\clubsuit }(a_i) = \epsilon \left( \frac{1}{2}, \phi ^{\diamondsuit }_i \otimes \phi ^{\heartsuit } \otimes \chi ^{-1}, \psi ^E\right) ; \\ \eta ^{\clubsuit }(b_j) = \epsilon \left( \frac{1}{2}, \phi ^{\diamondsuit } \otimes \phi ^{\heartsuit }_{j} \otimes \chi ^{-1}, \psi ^E\right) , \end{array}\right. } \end{aligned}$$

    where the $\epsilon$-factors are defined using any nontrivial additive character $\psi ^E$ of E / F. (The result is independent of this choice.)

We refer the reader to [15, Sect. 18] for a discussion of the various subtleties in the definition of $\eta ^{\spadesuit }$ or $\eta ^{\clubsuit }$.

3.3 Conjectures (B) and (FJ)

Let us formally state the statements (B)$_n$ and (FJ)$_n$:

(B)$_n$ :

Given a tempered L-parameter $\phi$ for $G_n^\pm = \mathrm {U}(V_n^\pm ) \times \mathrm {U}(V_{n+1}^\pm )$ and a representation $\pi (\eta ) \in \varPi _{\phi }$ of a relevant pure inner form $G_n^\epsilon$,

$$\begin{aligned} {\text {Hom}}_{\varDelta H_n^\epsilon }(\pi (\eta ), \mathbb {C}) \ne 0 \Longleftrightarrow \eta = \eta ^{\spadesuit }. \end{aligned}$$

(FJ)$_n$ :

Given a tempered L-parameter $\phi$ for $G_n^\pm = \mathrm {U}(W_n^\pm ) \times \mathrm {U}(W_n^\pm )$ and a representation $\pi (\eta ) \in \varPi _{\phi }$ of a relevant pure inner form $G_n^\epsilon$,

$$\begin{aligned} {\text {Hom}}_{\varDelta H_n^\epsilon }(\pi (\eta ), \nu ) \ne 0 \Longleftrightarrow \eta = \eta ^{\clubsuit }. \end{aligned}$$

We shall denote by (B) the collection of statements (B)$_n$ for all $n \ge 0$, and by (FJ) the collection of statements (FJ)$_n$ for all $n \ge 0$. We stress that both (B) and (FJ) are considered only for tempered representations in this paper (except in Sect. 9 where we treat the case of generic L-parameters).

4 Local theta correspondence and Prasad’s conjectures

In this section, we explicate the statement of Prasad’s conjectures on the local theta correspondence for unitary groups of (almost) equal rank.

4.1 Weil representations

Let V be a Hermitian space and W a skew-Hermitian space. To consider the theta correspondence for the reductive dual pair $\mathrm {U}(V) \times \mathrm {U}(W)$, one requires certain additional data:

• (i) a nontrivial additive character $\psi$ of F;

• (ii) a pair of characters $\chi _V$ and $\chi _W$ of $E^{\times }$ such that

  $$\begin{aligned} \chi _V |_{F^{\times }} = \omega _{E/F}^{\dim V} \quad \text { and} \quad \chi _W|_{F^{\times }} = \omega _{E/F}^{\dim W}. \end{aligned}$$

  One way to fix such a pair is simply to fix a character $\chi$ of $E^{\times }$ such that $\chi |_{F^{\times }} = \omega _{E/F}$ and then set

  $$\begin{aligned} \chi _V = \chi ^{\dim V} \quad \text { and} \quad \chi _W = \chi ^{\dim W}. \end{aligned}$$

• (iii) a trace zero element $\delta \in E^{\times }$.

To elaborate, the tensor product $V \otimes W$ has a natural symplectic form defined by

$$\begin{aligned} \langle v_1 \otimes w_1, v_2 \otimes w_2 \rangle = {\text {Tr}}_{E/F}(\langle v_1,v_2 \rangle _V \cdot \langle w_1, w_2 \rangle _W). \end{aligned}$$

Then there is a natural map

$$\begin{aligned} \mathrm {U}(V) \times \mathrm {U}(W) \longrightarrow \mathrm {Sp}(V \otimes W). \end{aligned}$$

One has the metaplectic $S^1$-cover $\mathrm {Mp}(V \otimes W)$ of $\mathrm {Sp}(V \otimes W)$, and the character $\psi$ (together with the form $\langle \cdot , \cdot \rangle$ on $V \otimes W$) determines a Weil representation $\omega _{\psi }$ of $\mathrm {Mp}(V \otimes W)$. The data $(\psi , \chi _V,\chi _W, \delta )$ then allows one to specify a splitting of the metaplectic cover over $\mathrm {U}(V) \times \mathrm {U}(W)$, as shown in [25, 37]. In fact, by construction and [25], Lemma A.7], it does not depend on the choice of $\delta$.

Hence, we have a Weil representation $\omega _{\psi , \chi _V, \chi _W, V,W}$ of $\mathrm {U}(V) \times \mathrm {U}(W)$. The Weil representation $\omega _{\psi , \chi _V, \chi _W, V,W}$ depends only on the orbit of $\psi$ under ${\text {N}}_{E/F}(E^{\times })$.

4.2 Local theta correspondence

Given an irreducible smooth representation $\pi$ of $\mathrm {U}(W)$, the maximal $\pi$-isotypic quotient of $\omega _{\psi , \chi _V, \chi _W, V,W}$ is of the form

$$\begin{aligned} \varTheta _{\psi , \chi _V, \chi _W, V,W}(\pi ) \boxtimes \pi \end{aligned}$$

for some smooth representation $\varTheta _{\psi , \chi _V, \chi _W, V,W}(\pi )$ of $\mathrm {U}(V)$ of finite length. By the Howe duality, which was proved by Waldspurger [59] for $p \ne 2$ and by the first author and Takeda [20, 21] for any p (so that the assumption $p \ne 2$ can be removed from the results of [17] stated below), the maximal semisimple quotient $\theta _{\psi , \chi _V, \chi _W, V,W}(\pi )$ of $\varTheta _{\psi , \chi _V, \chi _W, V,W}(\pi )$ is either zero or irreducible. If $\chi _V$ and $\chi _W$ are clear from the context, we simply write $\varTheta _{\psi , V,W}(\pi ) = \varTheta _{\psi , \chi _V, \chi _W, V,W}(\pi )$ and $\theta _{\psi , V,W}(\pi ) = \theta _{\psi , \chi _V, \chi _W, V,W}(\pi )$.

In this paper, we consider the theta correspondence for $\mathrm {U}(V) \times \mathrm {U}(W)$ with

$$\begin{aligned} |\dim V - \dim W| \le 1. \end{aligned}$$

We will state two conjectures of  Prasad which describe the local theta correspondence in terms of the local Langlands correspondence.

4.3 Equal rank case

We first consider the case $\dim V = \dim W = n$. We shall consider the theta correspondence for $\mathrm {U}(V_n^\epsilon ) \times \mathrm {U}(W_n^{\epsilon '})$. The following summarises some results of [17]:

Theorem 4.1

Let $\phi$ be an L-parameter for $\mathrm {U}(W_n^\pm )$. Then we have:

• (i) For any fixed $\pi \in \varPi _{\phi }^{\epsilon '}$, exactly one of $\varTheta _{\psi , V_n^+,W_n^{\epsilon '}}(\pi )$ or $\varTheta _{\psi , V_n^-,W_n^{\epsilon '}}(\pi )$ is nonzero.

• (ii) $\varTheta _{\psi , V_n^\epsilon , W_n^{\epsilon '}}(\pi ) \ne 0$ if and only if

  $$\begin{aligned} \epsilon \left( \tfrac{1}{2}, \phi \otimes \chi _V^{-1}, \psi ^E_2\right) = \epsilon \cdot \epsilon ', \end{aligned}$$

  where

  $$\begin{aligned} \psi ^E_2(x) = \psi ({\text {Tr}}_{E/F}(\delta x)). \end{aligned}$$

• (iii) If $\varTheta _{\psi , V_n^{\epsilon },W_n^{\epsilon '}}(\pi )$ is nonzero, then $\theta _{\psi , V_n^{\epsilon },W_n^{\epsilon '}}(\pi )$ has L-parameter

  $$\begin{aligned} \theta (\phi ) = \phi \otimes \chi _V^{-1} \chi _W. \end{aligned}$$

• (iv) The theta correspondence $\pi \mapsto \theta _{\psi , V_n^{\epsilon }, W_n^{\epsilon '}}(\pi )$ gives a bijection

  $$\begin{aligned} \varPi _{\phi } \longleftrightarrow \varPi _{\theta (\phi )}. \end{aligned}$$

• (v) If $\phi$ is tempered and $\varTheta _{\psi , V_n^{\epsilon }, W_n^{\epsilon '}}(\pi )$ is nonzero, then $\varTheta _{\psi , V_n^{\epsilon }, W_n^{\epsilon '}}(\pi )$ is irreducible.

4.4 Conjecture (P1)

After the above theorem, the remaining question is to specify the bijection of Vogan L-packets given in (iv). We shall do this using the bijections

$$\begin{aligned} J_{\psi }: \varPi _{\phi } \longleftrightarrow {\text {Irr}}(S_{\phi }) \quad \text { and} \quad J_{\psi ^E}: \varPi _{\theta (\phi )} \longleftrightarrow {\text {Irr}}(S_{\theta (\phi )}), \end{aligned}$$

where

$$\begin{aligned} \psi ^E(x) = \psi \Big (\tfrac{1}{2} {\text {Tr}}_{E/F}(\delta x)\Big ). \end{aligned}$$

(4.1)

Note that the bijections $J_{\psi }$ and $J_{\psi ^E}$ are independent of $\psi$ and $\psi ^E$ when n is odd, but when n is even, they do depend on these additive characters and it is crucial for $\psi$ and $\psi ^E$ to be related as in (4.1) for what follows to hold.

Having fixed the bijections $J_{\psi }$ and $J_{\psi ^E}$, we need to describe the bijection

$$\begin{aligned} {\text {Irr}}(S_{\phi })&\longleftrightarrow {\text {Irr}}(S_{\theta (\phi )}) \\ \eta&\longleftrightarrow \theta (\eta ) \end{aligned}$$

induced by the theta correspondence. Note that the component groups $S_{\phi }$ and $S_{\theta (\phi )}$ are canonically identified, since $\theta (\phi )$ is simply a twist of $\phi$ by a conjugate orthogonal character.

Now the first conjecture of Prasad states the following.

(P1)$_n$ :

Let $\phi$ be an L-parameter for $\mathrm {U}(W_n^\pm )$ and let $\eta \in {\text {Irr}}(S_\phi )$. Suppose that

$$\begin{aligned} S_{\phi } = S_{\theta (\phi )} = \prod _i (\mathbb {Z}/2\mathbb {Z}) a_i. \end{aligned}$$

Then, relative to $J_{\psi }$ and $J_{\psi ^E}$ as above,

$$\begin{aligned} \theta (\eta )(a_i) / \eta (a_i) = \epsilon \left( \tfrac{1}{2}, \phi _i \otimes \chi _V^{-1}, \psi ^E_2\right) , \end{aligned}$$

where

$$\begin{aligned} \psi ^E_2(x) = \psi ({\text {Tr}}_{E/F}(\delta x)). \end{aligned}$$

We shall denote by (P1) the collection of all statements (P1)$_n$ for all $n \ge 0$. Note that we consider (P1) for all L-parameters, and not just tempered ones. However, we note:

Proposition 4.2

Suppose that $({P1})_k$ holds for all tempered L-parameters for all $k < n$. Then $({P1})_k$ holds for all nontempered L-parameters for all $k \le n$.

Proof

This follows from the analog of [19, Theorem 8.1(iii)] for unitary groups. $\square$

Moreover, the following is a corollary of Theorem 4.1(ii):

Corollary 4.3

The statement $({P1})_n$ holds if $\phi$ is irreducible.

4.5 Almost equal rank case

Now we consider the case $\dim V = n+1$ and $\dim W = n$. We shall consider the theta correspondence for $\mathrm {U}(V_{n+1}^\epsilon ) \times \mathrm {U}(W_n^{\epsilon '})$. The following summarises some results of [17]:

Theorem 4.4

Let $\phi$ be an L-parameter for $\mathrm {U}(W_n^{\pm })$. Then we have:

• (i) Suppose that $\phi$ does not contain $\chi _V$.

  • (a) For any $\pi \in \varPi _{\phi }^{\epsilon '}$, $\varTheta _{\psi , V_{n+1}^{\epsilon }, W_n^{\epsilon '}}(\pi )$ is nonzero and $\theta _{\psi , V_{n+1}^{\epsilon }, W_n^{\epsilon '}}(\pi )$ has L-parameter

    $$\begin{aligned} \theta (\phi ) = (\phi \otimes \chi _V^{-1} \chi _W) \oplus \chi _W. \end{aligned}$$

  • (b) For each $\epsilon = \pm 1$, the theta correspondence $\pi \mapsto \theta _{\psi , V_{n+1}^{\epsilon }, W_n^{\epsilon '}}(\pi )$ gives a bijection

    $$\begin{aligned} \varPi _{\phi } \longleftrightarrow \varPi _{\theta (\phi )}^{\epsilon }. \end{aligned}$$

• (ii) Suppose that $\phi$ contains $\chi _V$.

  • (a) For any fixed $\pi \in \varPi _{\phi }^{\epsilon '}$, exactly one of $\varTheta _{\psi , V_{n+1}^+, W_n^{\epsilon '}}(\pi )$ or $\varTheta _{\psi , V_{n+1}^-, W_n^{\epsilon '}}(\pi )$ is nonzero.

  • (b) If $\varTheta _{\psi , V_{n+1}^{\epsilon }, W_n^{\epsilon '}}(\pi )$ is nonzero, then $\theta _{\psi , V_{n+1}^{\epsilon }, W_n^{\epsilon '}}(\pi )$ has L-parameter

    $$\begin{aligned} \theta (\phi ) = (\phi \otimes \chi _V^{-1} \chi _W) \oplus \chi _W. \end{aligned}$$

  • (c) The theta correspondence $\pi \mapsto \theta _{\psi , V_{n+1}^{\epsilon }, W_n^{\epsilon '}}(\pi )$ gives a bijection

    $$\begin{aligned} \varPi _{\phi } \longleftrightarrow \varPi _{\theta (\phi )}. \end{aligned}$$

• (iii) If $\phi$ is tempered and $\varTheta _{\psi , V_{n+1}^{\epsilon }, W_n^{\epsilon '}}(\pi )$ is nonzero, then $\varTheta _{\psi , V_{n+1}^{\epsilon }, W_n^{\epsilon '}}(\pi )$ is irreducible.

4.6 Conjecture (P2)

After the above theorem, it remains to specify the bijections given in (i)(b) and (ii)(c). As in the case of (P1), we shall do this using the bijections

$$\begin{aligned} J_{\psi }: \varPi _{\phi } \longleftrightarrow {\text {Irr}}(S_{\phi }) \quad \text { and} \quad J_{\psi ^E}: \varPi _{\theta (\phi )} \longleftrightarrow {\text {Irr}}(S_{\theta (\phi )}), \end{aligned}$$

where

$$\begin{aligned} \psi ^E(x) = \psi \Big (\tfrac{1}{2} {\text {Tr}}_{E/F}(\delta x)\Big ). \end{aligned}$$

Note that $J_{\psi }$ is independent of $\psi$ when n is odd, whereas $J_{\psi ^E}$ is independent of $\psi ^E$ when n is even.

Observe that:

• If $\phi$ does not contain $\chi _V$, then

  $$\begin{aligned} S_{\theta (\phi )} = S_{\phi } \times (\mathbb {Z}/2 \mathbb {Z}) a_0, \end{aligned}$$

  where the extra copy of $\mathbb {Z}/2\mathbb {Z}$ arises from the summand $\chi _W$ in $\theta (\phi )$. Thus, for each $\epsilon$, one has a canonical bijection

  $$\begin{aligned} {\text {Irr}}(S_{\phi })&\longleftrightarrow {\text {Irr}}^{\epsilon }(S_{\theta (\phi )}) \\ \eta&\longleftrightarrow \theta (\eta ) \end{aligned}$$

  induced by the theta correspondence, where ${\text {Irr}}^{\epsilon }(S_{\theta (\phi )})$ is the set of irreducible characters $\eta '$ of $S_{\theta (\phi )}$ such that $\eta '(z_{\theta (\phi )}) = \epsilon$.

• On the other hand, if $\phi$ contains $\chi _V$, then $\phi \otimes \chi _V^{-1} \chi _W$ contains $\chi _W$, so that

  $$\begin{aligned} S_{\theta (\phi )} = S_{\phi }. \end{aligned}$$

  Thus, one has a canonical bijection

  $$\begin{aligned} {\text {Irr}}(S_{\phi })&\longleftrightarrow {\text {Irr}}(S_{\theta (\phi )}) \\ \eta&\longleftrightarrow \theta (\eta ) \end{aligned}$$

  induced by the theta correspondence.

Now we can state the second conjecture of Prasad.

(P2)$_n$ :

Let $\phi$ be an L-parameter for $\mathrm {U}(W_n^\pm )$ and let $\eta \in {\text {Irr}}(S_\phi )$. Fix the bijections $J_{\psi }$ and $J_{\psi ^E}$ as above.

• If $\phi$ does not contain $\chi _V$, then $\theta (\eta )$ is the unique irreducible character in ${\text {Irr}}^{\epsilon }(S_{\theta (\phi )})$ such that

  $$\begin{aligned} \theta (\eta )|_{S_{\phi }} =\eta . \end{aligned}$$

• On the other hand, if $\phi$ contains $\chi _V$, then

  $$\begin{aligned} \theta (\eta ) = \eta . \end{aligned}$$

We shall denote by (P2) the collection of all the statements (P2)$_n$ for all $n \ge 0$. Note that we consider (P2) for all L-parameters, and not just tempered ones. However, we note:

Proposition 4.5

Suppose that $(P2 )_k$ holds for all tempered L-parameters for all $k < n$. Then $(P2 )_k$ holds for all nontempered L-parameters for all $k \le n$.

Proof

This follows from [17, Proposition C.4(ii)]. $\square$

5 (B) $+$ (P2) $\Longrightarrow$ (FJ) $+$ (P1)

In this section, we shall show that Conjectures (FJ) and (P1) follow from Conjectures (B) and (P2), together with Theorems 4.1 and 4.4.

Suppose that we are given tempered L-parameters $\phi ^{\diamondsuit }$ and $\phi ^{\heartsuit }$ for $\mathrm {U}(W_n^\pm )$. Let

$$\begin{aligned} \pi ^{\diamondsuit } = \pi (\eta ^{\diamondsuit }) \in \varPi _{\phi ^{\diamondsuit }}^{\epsilon '} \quad \text { and} \quad \pi ^{\heartsuit } = \pi (\eta ^{\heartsuit }) \in \varPi _{\phi ^{\heartsuit }}^{\epsilon '} \end{aligned}$$

be representations such that

$$\begin{aligned} {\text {Hom}}_{\mathrm {U}(W_n^{\epsilon '})}(\pi ^{\diamondsuit } \otimes \pi ^{\heartsuit }, \omega _{\psi , \chi , W_n^{\epsilon '}}) \ne 0. \end{aligned}$$

We first show that

$$\begin{aligned} \eta ^{\diamondsuit } \otimes \eta ^{\heartsuit } = \eta ^{\clubsuit }. \end{aligned}$$

Since the representations involved are unitary (as $\phi ^{\diamondsuit }$ and $\phi ^{\heartsuit }$ are tempered),

$$\begin{aligned} {\text {Hom}}_{\mathrm {U}(W_n^{\epsilon '})}(\pi ^{\diamondsuit } \otimes \pi ^{\heartsuit }, \omega _{\psi , \chi , W_n^{\epsilon '}}) \ne 0 \end{aligned}$$

if and only if

$$\begin{aligned} {\text {Hom}}_{\mathrm {U}(W_n^{\epsilon '})}( (\pi ^{\diamondsuit })^{\vee } \otimes \omega _{\psi , \chi , W_n^{\epsilon '}}, \pi ^{\heartsuit }) \ne 0. \end{aligned}$$

5.1 See-Saw

Now we consider the see-saw diagram (for an $\epsilon$ to be determined soon):

We shall consider the local theta correspondence for the above see-saw diagram. For this, we need to specify precisely the data used in setting up the theta correspondence. More precisely, for the dual pair $\mathrm {U}(V_{n+1}^\epsilon ) \times \mathrm {U}(W_n^{\epsilon '})$, we shall use the characters

$$\begin{aligned} \chi _{V_{n+1}^\epsilon } = \chi ^{n+(-1)^n} \quad \text { and} \quad \chi _{W_n^{\epsilon '}} = \chi ^n, \end{aligned}$$

and for the dual pair $\mathrm {U}(V_n^\epsilon ) \times \mathrm {U}(W_n^{\epsilon '})$, we use

$$\begin{aligned} \chi _{V_n^\epsilon } = \chi _{W_n^{\epsilon '}} = \chi ^n. \end{aligned}$$

Then for the dual pair $\mathrm {U}(L_{(-1)^n}) \times \mathrm {U}(W_n^{\epsilon '})$, we have no choice but to use

$$\begin{aligned} \chi _{L_{(-1)^n}} = \chi ^{(-1)^n} \quad \text { and} \quad \chi _{W_n^{\epsilon '}} = \chi ^n. \end{aligned}$$

In particular, the restriction of $\omega _{\psi , \chi _{L_{(-1)^n}}, \chi _{W_n^{\epsilon '}}, L_{(-1)^n}, W_n^{\epsilon '}}$ to $\mathrm {U}(W_n^{\epsilon '})$ is equal to

$$\begin{aligned} {\left\{ \begin{array}{ll} \omega _{\psi , \chi , W_n^{\epsilon '}} &{} {\text { if } n \text { is even;}} \\ \omega _{\psi , \chi , W_n^{\epsilon '}}^{\vee } &{} {\text { if } n \text { is odd.}} \end{array}\right. } \end{aligned}$$

In any case, having fixed these normalizations, we shall suppress them from the notation for simplicity.

Because of the above differences for even and odd n, it will now be convenient to treat the even and odd cases separately.

5.2 Even case

Assume first that n is even. By Theorem 4.1, we may choose $\sigma \in {\text {Irr}}(\mathrm {U}(V_n^{\epsilon }))$ such that

$$\begin{aligned} \varTheta _{\psi , V_n^{\epsilon }, W_n^{\epsilon '}}(\sigma ) = (\pi ^{\diamondsuit })^{\vee }. \end{aligned}$$

This uniquely determines $\epsilon$. Moreover, by Theorem 4.1, we know that $\sigma$ has L-parameter

$$\begin{aligned} \phi _{\sigma } = (\phi ^{\diamondsuit })^{\vee }, \end{aligned}$$

since the L-parameter of $(\pi ^{\diamondsuit })^{\vee }$ is $(\phi ^{\diamondsuit })^{\vee }$.

Taking the representation $\pi ^{\heartsuit }$ on $\mathrm {U}(W_n^{\epsilon '})$ and the representation $\sigma$ on $\mathrm {U}(V_n^{\epsilon })$, the resulting see-saw identity reads:

$$\begin{aligned} 0\ne & {} {\text {Hom}}_{\mathrm {U}(W_n^{\epsilon '})}( (\pi ^{\diamondsuit })^{\vee } \otimes \omega _{\psi , \chi , W_n^{\epsilon '}}, \pi ^{\heartsuit })\\= & {} {\text {Hom}}_{\mathrm {U}(V_n^{\epsilon })}( \varTheta _{\psi , V_{n+1}^{\epsilon }, W_n^{\epsilon '}}(\pi ^{\heartsuit }), \sigma ). \end{aligned}$$

By Theorem 4.4,

$$\begin{aligned} \tau := \varTheta _{\psi , V_{n+1}^{\epsilon }, W_n^{\epsilon '}}(\pi ^{\heartsuit }) \end{aligned}$$

has L-parameter

$$\begin{aligned} \phi _{\tau } = (\phi ^{\heartsuit } \otimes \chi ^{-1}) \oplus \chi ^n. \end{aligned}$$

Recall that we have used the character $\psi$ to fix the local Langlands correspondence for $\mathrm {U}(W_n^{\epsilon '})$. The component group $S_{\phi ^{\heartsuit }}$ is of the form

$$\begin{aligned} S_{\phi ^{\heartsuit }} = \prod _j (\mathbb {Z}/ 2\mathbb {Z}) b_j \end{aligned}$$

and there is a natural embedding $S_{\phi ^{\heartsuit }} \hookrightarrow S_{\phi _\tau }$. Now, by (P2), the representation $\tau$ has associated character $\eta _{\tau } \in {\text {Irr}}(S_{\phi _{\tau }})$ which satisfies:

$$\begin{aligned} \eta _{\tau } = \eta ^{\heartsuit } \quad \text { on} \quad S_{\phi ^{\heartsuit }}. \end{aligned}$$

On the other hand, by (B), one knows exactly what $\eta _{\tau }$ is. Namely, (B) gives:

$$\begin{aligned} \eta _{\tau }(b_j)&= \epsilon \left( \tfrac{1}{2}, \phi _{\sigma }^{\vee } \otimes \phi _j^{\heartsuit } \otimes \chi ^{-1}, \psi ^E\right) \\&= \epsilon \left( \tfrac{1}{2}, \phi ^{\diamondsuit } \otimes \phi _j^{\heartsuit } \otimes \chi ^{-1}, \psi ^E\right) \\&= \eta ^{\clubsuit }(b_j), \end{aligned}$$

where $\psi ^E$ is any nontrivial character of E / F. Thus, we deduce that

$$\begin{aligned} \eta ^{\heartsuit } = \eta ^{\clubsuit } \quad \text { on} \quad S_{\phi ^{\heartsuit }}. \end{aligned}$$

Now of course we could reverse the role of $\pi ^{\diamondsuit }$ and $\pi ^{\heartsuit }$ in the above argument. Then we conclude that

$$\begin{aligned} \eta ^{\diamondsuit } \otimes \eta ^{\heartsuit } = \eta ^{\clubsuit } \end{aligned}$$

as desired.

5.3 Odd case

Now suppose that n is odd. Then we use the character

$$\begin{aligned} \psi ^E(x) = \psi \Big (\tfrac{1}{2} {\text {Tr}}_{E/F}(\delta x)\Big ) \end{aligned}$$

of E / F to specify the local Langlands correspondence for $\mathrm {U}(V_{n+1}^{\epsilon })$. By Theorem 4.1, we may choose $\sigma \in {\text {Irr}}(\mathrm {U}(V_n^{\epsilon }))$ such that

$$\begin{aligned} \varTheta _{\psi , V_n^{\epsilon }, W_n^{\epsilon '}}(\sigma ) = \pi ^{\diamondsuit }. \end{aligned}$$

This uniquely determines $\epsilon$. Moreover, by Theorem 4.1, we know that $\sigma$ has L-parameter

$$\begin{aligned} \phi _{\sigma } = \phi ^{\diamondsuit }. \end{aligned}$$

Taking the representation $(\pi ^{\heartsuit })^{\vee }$ on $\mathrm {U}(W_n^{\epsilon '})$ and the representation $\sigma$ on $\mathrm {U}(V_n^{\epsilon })$, the resulting see-saw identity reads:

$$\begin{aligned} 0\ne & {} {\text {Hom}}_{\mathrm {U}(W_n^{\epsilon '})}( \pi ^{\diamondsuit } \otimes \omega _{\psi , \chi , W_n^{\epsilon '}}^{\vee }, (\pi ^{\heartsuit })^{\vee })\\= & {} {\text {Hom}}_{\mathrm {U}(V_n^{\epsilon })}( \varTheta _{\psi , V_{n+1}^{\epsilon }, W_n^{\epsilon '}}((\pi ^{\heartsuit })^{\vee }), \sigma ). \end{aligned}$$

By Theorem 4.4,

$$\begin{aligned} \tau := \varTheta _{\psi , V_{n+1}^{\epsilon }, W_n^{\epsilon '}}((\pi ^{\heartsuit })^{\vee }) \end{aligned}$$

has L-parameter

$$\begin{aligned} \phi _{\tau } = ((\phi ^{\heartsuit })^{\vee } \otimes \chi ) \oplus \chi ^n. \end{aligned}$$

Now by (P2), the representation $\tau$ has associated character $\eta _{\tau } \in {\text {Irr}}(S_{\phi _{\tau }})$ satisfying:

$$\begin{aligned} \eta _{\tau } = \eta ^{\heartsuit } \quad \text { on} \quad S_{\phi ^{\heartsuit }} = S_{(\phi ^{\heartsuit })^{\vee } \otimes \chi }. \end{aligned}$$

On the other hand, by (B), we know that

$$\begin{aligned} \eta _{\tau }(b_j)&= \epsilon \left( \tfrac{1}{2}, \phi _\sigma ^\vee \otimes (\phi _j^{\heartsuit })^{\vee } \otimes \chi , \psi ^E_{-2}\right) \\&= \epsilon \left( \tfrac{1}{2}, \phi ^{\diamondsuit } \otimes \phi ^{\heartsuit }_j \otimes \chi ^{-1}, \psi ^E_2\right) \\&= \eta ^{\clubsuit }(b_j). \end{aligned}$$

Hence, we conclude that

$$\begin{aligned} \eta ^{\heartsuit } = \eta ^{\clubsuit } \quad \text { on} \quad S_{\phi ^{\heartsuit }}. \end{aligned}$$

Reversing the role of $\pi ^{\diamondsuit }$ and $\pi ^{\heartsuit }$ in the above argument, we conclude that

$$\begin{aligned} \eta ^{\diamondsuit } \otimes \eta ^{\heartsuit } = \eta ^{\clubsuit } \end{aligned}$$

as desired.

5.4 Proof of (FJ)

At this point, we have shown that if

$$\begin{aligned} {\text {Hom}}_{\mathrm {U}(W_n^{\epsilon '})}(\pi ^{\diamondsuit } \otimes \pi ^{\heartsuit }, \omega _{\psi , \chi , W_n^{\epsilon '}}) \ne 0, \end{aligned}$$

then $\eta ^{\diamondsuit } \otimes \eta ^{\heartsuit }$ is equal to the distinguished character $\eta ^{\clubsuit }$. To complete the proof of (FJ), it remains to show that the above Hom space is nonzero for some $\epsilon '$ and pair of representations $(\pi ^{\diamondsuit }, \pi ^{\heartsuit }) \in \varPi ^{\epsilon '}_{\phi ^{\diamondsuit }} \times \varPi ^{\epsilon '}_{\phi ^{\heartsuit }}$. This will follow from the above see-saw diagram, Theorems 4.1 and 4.4. Let us illustrate this in the case when n is even; the case when n is odd is similar.

Consider the tempered L-parameters $\phi := (\phi ^{\heartsuit } \otimes \chi ^{-1}) \oplus \chi ^n$ for $\mathrm {U}(V_{n+1}^{\pm })$ and $\phi ' := (\phi ^{\diamondsuit })^{\vee }$ for $\mathrm {U}(V_n^{\pm })$. By (B), there is a pair of representations

$$\begin{aligned} (\tau , \tau ') \in \varPi ^{\epsilon }_{\phi } \times \varPi ^{\epsilon }_{\phi '} \end{aligned}$$

such that

$$\begin{aligned} {\text {Hom}}_{\mathrm {U}(V_n^{\epsilon })}( \tau , \tau ') \ne 0. \end{aligned}$$

By Theorem 4.4, we can find a unique $\pi ^{\heartsuit } \in \varPi ^{\epsilon '}_{\phi ^{\heartsuit }}$ (which determines $\epsilon '$) such that

$$\begin{aligned} \tau = \varTheta _{\psi , V^{\epsilon }_{n+1}, W^{\epsilon '}_n}( \pi ^{\heartsuit } ). \end{aligned}$$

Now the see-saw identity gives

$$\begin{aligned} 0\ne & {} {\text {Hom}}_{\mathrm {U}(V_n^{\epsilon })}( \varTheta _{\psi , V^{\epsilon }_{n+1}, W^{\epsilon '}_n}( \pi ^{\heartsuit }), \tau ')\\= & {} {\text {Hom}}_{\mathrm {U}(W_n^{\epsilon '})} (\varTheta _{\psi , V_n^{\epsilon }, W_n^{\epsilon '}}(\tau ') \otimes \omega _{\psi , \chi , W_n^{\epsilon '}}, \pi ^{\heartsuit }). \end{aligned}$$

In particular,

$$\begin{aligned} \pi ^{\diamondsuit } := \varTheta _{\psi , V_n^{\epsilon }, W_n^{\epsilon '}}(\tau ')^{\vee } \ne 0 \end{aligned}$$

and by Theorem 4.1, it has L-parameter $(\phi ')^\vee = \phi ^{\diamondsuit }$. Thus we see that for some $(\pi ^{\diamondsuit }, \pi ^{\heartsuit }) \in \varPi ^{\epsilon '}_{\phi ^{\diamondsuit }} \times \varPi ^{\epsilon '}_{\phi ^{\heartsuit }}$, we have

$$\begin{aligned} {\text {Hom}}_{\mathrm {U}(W^{\epsilon '}_n)} ( (\pi ^{\diamondsuit })^{\vee } \otimes \omega _{\psi ,\chi , W_n^{\epsilon '}} , \pi ^{\heartsuit }) \ne 0 \end{aligned}$$

as desired. This completes the proof of (FJ).

5.5 Proof of (P1)

Now we come to the proof of (P1). In particular, we consider the theta correspondence for $\mathrm {U}(V_n^\epsilon ) \times \mathrm {U}(W_n^{\epsilon '})$ relative to the Weil representation $\omega _{\psi , \chi _V, \chi _W, V_n^{\epsilon }, W_n^{\epsilon '}}$. Given an L-parameter $\phi$ for $\mathrm {U}(W_n^{\pm })$, we would like to explicate the bijection

$$\begin{aligned} \theta : {\text {Irr}}(S_{\phi }) \longleftrightarrow {\text {Irr}}(S_{\theta (\phi )}) \end{aligned}$$

furnished by Theorem 4.1, with $\theta (\phi ) = \phi \otimes \chi _V^{-1} \chi _W$. Here, recall that

$$\begin{aligned} S_{\phi } = S_{\theta (\phi )} = \prod _i (\mathbb {Z}/ 2 \mathbb {Z}) a_i. \end{aligned}$$

Since we now have (B), (FJ) and (P2) at our disposal, we shall be able to determine $\theta$ using the see-saw diagram.

More precisely, we start with a tempered L-parameter $\phi$ and consider an irreducible tempered representation $\pi = \pi (\eta ) \in \varPi _{\phi }^{\epsilon '}$. One knows by Theorem 4.1 that $\varTheta _{\psi , V^{\epsilon }_n, W_n^{\epsilon '}}(\pi ) \in \varPi _{\theta (\phi )}^{\epsilon }$ is a nonzero irreducible tempered representation of $\mathrm {U}(V_n^\epsilon )$ for a unique $\epsilon$. By the analog of [19, Lemma 12.5] for unitary groups, one can find an irreducible tempered representation $\sigma$ of $\mathrm {U}(V_{n-1}^{\epsilon })$ such that

$$\begin{aligned} {\text {Hom}}_{\mathrm {U}(V_{n-1}^{\epsilon })}(\varTheta _{\psi , V^{\epsilon }_n, W_n^{\epsilon '}}(\pi ), \sigma ) \ne 0. \end{aligned}$$

By (B), one has

$$\begin{aligned} \theta (\eta )(a_i) = \epsilon \Big (\tfrac{1}{2}, \phi _{\sigma }^{\vee } \otimes \phi _i \otimes \chi _V^{-1} \chi _W, \psi ^E_{-2}\Big ), \end{aligned}$$

where $\phi _\sigma$ is the L-parameter of $\sigma$.

On the other hand, one has the see-saw diagram

We consider the theta correspondence for $\mathrm {U}(L_{(-1)^{n-1}}) \times \mathrm {U}(W_n^{\epsilon '})$ relative to the pair of characters $(\chi ^{(-1)^{n-1}}, \chi _W)$, so that the theta correspondence for $\mathrm {U}(V^{\epsilon }_{n-1}) \times \mathrm {U}(W_n^{\epsilon '})$ is with respect to the pair $(\chi _V \chi ^{(-1)^n}, \chi _W)$. We shall suppress these pairs of characters from the notation in the following. By Theorem 4.4, the representation

$$\begin{aligned} \tau := \varTheta _{\psi , V^{\epsilon }_{n-1}, W_n^{\epsilon '}}(\sigma ) \ne 0 \end{aligned}$$

is irreducible and tempered. Moreover, $\tau$ has L-parameter

$$\begin{aligned} \phi _{\tau } = (\phi _{\sigma } \otimes \chi _V \chi _W^{-1} \chi ^{(-1)^n} ) \oplus \chi _V \chi ^{(-1)^n}. \end{aligned}$$

(5.1)

It will now be convenient to consider the even and odd cases separately.

5.6 Even case

Assume first that n is even. By the see-saw identity, one has

$$\begin{aligned} 0\ne & {} {\text {Hom}}_{\mathrm {U}(W_n^{\epsilon '})}( \varTheta _{\psi , V^{\epsilon }_{n-1}, W_n^{\epsilon '}}(\sigma ) \otimes \omega _{\psi , \chi , W_n^{\epsilon '}}^{\vee }, \pi )\\= & {} {\text {Hom}}_{\mathrm {U}(W_n^{\epsilon '})} (\tau \otimes \pi ^{\vee }, \omega _{\psi ,\chi , W_n^{\epsilon '}}). \end{aligned}$$

It follows by (FJ) that

$$\begin{aligned} \eta (a_i) \cdot \omega _{E/F}(-1)^{\dim \phi _i} = \eta _{\pi ^{\vee }}(a_i) = \epsilon \left( \tfrac{1}{2}, \phi _{\tau } \otimes \phi _i^{\vee } \otimes \chi ^{-1}, \psi _2^E\right) , \end{aligned}$$

where the local root number appearing here is independent of the choice of the additive character of E / F used since $\dim \phi _\tau = n$ is even. Hence, by (5.1), one has

$$\begin{aligned} \eta (a_i)= & {} \epsilon \left( \tfrac{1}{2}, \phi _{\sigma } \otimes \phi _i^{\vee } \otimes \chi _V \chi _W^{-1}, \psi _2^E\right) \\&\times \epsilon \left( \tfrac{1}{2}, \phi _i^{\vee }\otimes \chi _V, \psi _2^E\right) \cdot \omega _{E/F}(-1)^{\dim \phi _i}. \end{aligned}$$

Noting that $\phi _i$ is conjugate symplectic, we may compute:

$$\begin{aligned} \theta (\eta )(a_i) / \eta (a_i)&= \epsilon \left( \tfrac{1}{2}, \phi _i^{\vee } \otimes \chi _V, \psi _2^E\right) \cdot \omega _{E/F}(-1)^{\dim \phi _i} \\&= \epsilon \left( \tfrac{1}{2}, \phi _i^{\vee } \otimes \chi _V, \psi _{-2}^E\right) \\&= \epsilon \left( \tfrac{1}{2}, \phi _i \otimes \chi _V^{-1}, \psi _2^E\right) \\ \end{aligned}$$

as desired.

5.7 Odd case

Now suppose that n is odd. By the see-saw identity, one has

$$\begin{aligned} {\text {Hom}}_{\mathrm {U}(W_n^{\epsilon '})}(\tau \otimes \omega _{\psi ,\chi , W_n^{\epsilon '}}, \pi ) \ne 0, \end{aligned}$$

so that

$$\begin{aligned} {\text {Hom}}_{\mathrm {U}(W_n^{\epsilon '})}(\tau ^{\vee } \otimes \pi , \omega _{\psi , \chi , W_n^{\epsilon '}}) \ne 0. \end{aligned}$$

By (FJ), one has

$$\begin{aligned} \eta (a_i)&= \epsilon \left( \tfrac{1}{2}, \phi _{\tau }^{\vee } \otimes \phi _i \otimes \chi ^{-1}, \psi ^E_2\right) \\&= \epsilon \left( \tfrac{1}{2}, \phi _{\sigma }^{\vee } \otimes \phi _i \otimes \chi _V^{-1} \chi _W, \psi ^E_2\right) \cdot \epsilon \left( \tfrac{1}{2}, \phi _i \otimes \chi _V^{-1}, \psi ^E_2\right) , \end{aligned}$$

where the second equality follows from (5.1). On the other hand, we have seen that

$$\begin{aligned} \theta (\eta )(a_i)&= \epsilon \left( \tfrac{1}{2}, \phi _{\sigma }^{\vee } \otimes \phi _i \otimes \chi _V^{-1} \chi _W, \psi ^E_{-2}\right) \\&= \epsilon \left( \tfrac{1}{2}, \phi _{\sigma }^{\vee } \otimes \phi _i \otimes \chi _V^{-1} \chi _W , \psi ^E_2\right) , \end{aligned}$$

where the second equality follows because $\dim \phi _{\sigma }^{\vee } = n-1$ is even. Hence, we conclude that

$$\begin{aligned} \theta (\eta )(a_i) / \eta (a_i) = \epsilon \left( \tfrac{1}{2}, \phi _i \otimes \chi _V^{-1}, \psi ^E_2\right) \end{aligned}$$

as desired.

We have thus shown Conjecture (P1) for tempered L-parameters. For nontempered L-parameters, (P1) follows from the tempered case by Proposition 4.2.

To summarise, we have shown the following proposition:

Proposition 5.1

Assume that $(B )_k$ and $(P2 )_k$ hold for all tempered L-parameters for all $k \le n$. Then $(FJ )_k$ and $(P1 )_k$ also hold for all tempered L-parameters for all $k \le n$.

5.8 (B) $+$ (P1) $\Longrightarrow$ (FJ) $+$ (P2)

Instead of assuming (B) and (P2) as we have done above, one may assume (B) and (P1). Using the same arguments as above, together with Theorems 4.1 and 4.4, one can then deduce (FJ) and (P2). We state this formally as a proposition and leave the details of the proof to the reader.

Proposition 5.2

Assume that $(B )_k$ and $(P1 )_k$ hold for all tempered L-parameters for all $k \le n$. Then $(FJ )_k$ and $(P2 )_k$ also hold for all tempered L-parameters for all $k \le n$.

6 Proof of (P2)

After the previous section, and in view of the results of Beuzart-Plessis [4–6] (who proves (B)), it remains to prove (P2)$_n$. We shall prove (P2)$_n$ by using induction on n.

6.1 The base cases

For (P2)$_0$, there is nothing to prove. By [16, 25] and [5], we know that (B)$_1$ and (P1)$_1$ hold. Hence it follows by Proposition 5.2 that (P2)$_1$ holds.

For (P2)$_2$, the nontempered case follows from the tempered case by Proposition 4.5. To show (P2)$_2$ for tempered L-parameters, it follows by Proposition 5.2 that it suffices to show (P1)$_2$ for tempered L-parameters. Now (P1)$_2$ was shown in [16, Theorem 11.2] by a global argument, appealing to the analog of (P1)$_2$ at archimedean places. However, we can also give a purely local proof here.

Suppose that $\phi$ is a tempered L-parameter for $\mathrm {U}(W_2^{\pm })$ and we are considering the theta correspondence for $\mathrm {U}(V_2^{\epsilon }) \times \mathrm {U}(W_2^{\epsilon '})$ with respect to a pair of characters $(\chi _V, \chi _W)$. If $\phi$ is irreducible, then Corollary 4.3 guarantees that (P1)$_2$ holds. Hence we shall assume that $\phi = \phi _1 \oplus \phi _2$ with 1-dimensional characters $\phi _i$. If $\phi _1$ or $\phi _2$ is not conjugate symplectic, then $S_\phi$ is trivial and (P1)$_2$ follows from Theorem 4.1. Thus, we shall further assume that both $\phi _1$ and $\phi _2$ are conjugate symplectic, so that

$$\begin{aligned} S_{\phi } = {\left\{ \begin{array}{ll} (\mathbb {Z}/2\mathbb {Z})a_1 \times (\mathbb {Z}/2\mathbb {Z}) a_2 &{} {\text { if } \phi _1 \ne \phi _2;} \\ ((\mathbb {Z}/2\mathbb {Z}) a_1 \times (\mathbb {Z}/2\mathbb {Z}) a_2 ) / \varDelta \mathbb {Z}/2\mathbb {Z}&{} {\text { if } \phi _1 = \phi _2.} \end{array}\right. } \end{aligned}$$

To unify notation in the two cases, we shall regard ${\text {Irr}}(S_{\phi })$ as a subset of the irreducible characters of $(\mathbb {Z}/2\mathbb {Z}) a_1 \times (\mathbb {Z}/2\mathbb {Z}) a_2$ even when $\phi _1 = \phi _2$.

Let $\pi = \pi (\eta ) \in \varPi _{\phi }^{\epsilon '}$. By Theorem 4.1, we know that the theta lift of $\pi$ to $\mathrm {U}(V_2^{\epsilon })$ is nonzero for a uniquely determined $\epsilon$ given by

$$\begin{aligned} \epsilon = \epsilon \left( \tfrac{1}{2}, \phi \otimes \chi _V^{-1}, \psi ^E_2\right) \cdot \epsilon ', \end{aligned}$$

and has L-parameter

$$\begin{aligned} \theta (\phi ) = \phi \otimes \chi _V^{-1} \chi _W. \end{aligned}$$

Set

$$\begin{aligned} \sigma = \varTheta _{\psi , V_2^{\epsilon }, W_2^{\epsilon '}}(\pi ) \in \varPi _{\theta (\phi )}^{\epsilon } \end{aligned}$$

and let $\theta (\eta ) \in {\text {Irr}}(S_{\theta (\phi )})$ be the irreducible character associated to $\sigma$. Then we need to compute $\theta (\eta )(a_i) / \eta (a_i)$.

Consider the decomposition

$$\begin{aligned} V_2^{\epsilon } = V_1^{\epsilon } \oplus L_{-1}, \end{aligned}$$

and choose a character $\mu \in {\text {Irr}}(\mathrm {U}(V_1^{\epsilon }))$ such that

$$\begin{aligned} {\text {Hom}}_{\mathrm {U}(V_1^{\epsilon })}(\sigma , \mu ) \ne 0. \end{aligned}$$

Then by (B)$_1$, one sees that

$$\begin{aligned} \begin{aligned} \theta (\eta )(a_i)&= \epsilon \left( \tfrac{1}{2}, \mu _E^{-1} \phi _i \chi _V^{-1} \chi _W, \psi ^E_{-2}\right) \\&= \epsilon \left( \tfrac{1}{2}, \mu _E^{-1} \phi _i \chi _V^{-1} \chi _W, \psi ^E_2\right) \cdot \omega _{E/F}(-1), \end{aligned} \end{aligned}$$

(6.1)

where $\mu _E$ is the character of $E^{\times }$ given by $\mu _E(x) = \mu (x/x^c)$.

On the other hand, consider the see-saw diagram

For a conjugate symplectic character $\chi$ of $E^{\times }$, we consider the theta correspondences for

$$\begin{aligned} \mathrm {U}(V_1^{\epsilon } )\times \mathrm {U}(W^{\epsilon '}_2) \quad \text { with respect to} \quad (\chi _V \chi , \chi _W) \end{aligned}$$

and

$$\begin{aligned} \mathrm {U}(L_{-1}) \times \mathrm {U}(W_2^{\epsilon '}) \quad \text { with respect to} \quad (\chi ^{-1}, \chi _W). \end{aligned}$$

Set

$$\begin{aligned} \tau := \varTheta _{\psi , \chi _V \chi , \chi _W, V_1^{\epsilon }, W_2^{\epsilon '}}(\mu ) \quad \text { on} \quad \mathrm {U}(W_2^{\epsilon '}). \end{aligned}$$

Then Theorem 4.4 implies that $\tau$ has L-parameter

$$\begin{aligned} \phi _{\tau } = \mu _E \chi _V \chi _W^{-1} \chi \oplus \chi _V \chi . \end{aligned}$$

Now the see-saw identity then gives

$$\begin{aligned} 0 \ne {\text {Hom}}_{\mathrm {U}(V_1^{\epsilon })}(\sigma , \mu ) = {\text {Hom}}_{\mathrm {U}(W_2^{\epsilon '})}( \tau \otimes \omega _{\psi , \chi , W_2^{\epsilon '}}^{\vee }, \pi ). \end{aligned}$$

Since we do not know (FJ)$_2$ at this point, this nonvanishing does not give us the desired information about $\eta$. However, we note that

$$\begin{aligned} {\text {Hom}}_{\mathrm {U}(W_2^{\epsilon '})}( \tau \otimes \omega _{\psi , \chi , W_2^{\epsilon '}}^{\vee }, \pi ) = {\text {Hom}}_{\mathrm {U}(W_2^{\epsilon '})}(\pi ^{\vee } \otimes \omega _{\psi , \chi , W_2^{\epsilon '}}^{\vee }, \tau ^{\vee }). \end{aligned}$$

This allows one to exchange the roles of $\pi$ and $\tau$ in a variant of the above see-saw diagram.

More precisely, since $\phi = \phi _1 \oplus \phi _2$ with conjugate symplectic characters $\phi _i$, it follows by (P2)$_1$ (which we have shown) that the L-packet $\varPi _{\phi ^{\vee }}$ can be constructed via theta lifts from $\mathrm {U}(V_1^{\pm })$. Namely, if we start with the L-parameter

$$\begin{aligned} \phi ' := \phi _1^{-1} \phi _2 \chi _W \quad \text { for} \quad \mathrm {U}(V_1^{\pm }) \end{aligned}$$

and consider the theta correspondence for $\mathrm {U}(V_1^{\epsilon ''}) \times \mathrm {U}(W_2^{\epsilon '})$ with respect to the pair $(\phi _2^{-1}, \chi _W)$, then the theta lifts of $\varPi _{\phi '}$ give the L-packet $\varPi _{\phi ^{\vee }}$. In particular, we see that

$$\begin{aligned} \pi ^{\vee } = \varTheta _{\psi , \phi _2^{-1}, \chi _W, V_1^{\epsilon ''}, W_2^{\epsilon '}}(\mu ') \end{aligned}$$

for a unique $\mu ' \in \varPi _{\phi '}^{\epsilon ''}$ (which determines $\epsilon ''$). Indeed, (P2)$_1$ says that

$$\begin{aligned} \epsilon '' = \eta _{\pi ^{\vee }}(a_1) = \eta (a_1) \cdot \omega _{E/F}(-1). \end{aligned}$$

(6.2)

Thus, we may consider the see-saw diagram

and the theta correspondences for

$$\begin{aligned} \mathrm {U}(V_1^{\epsilon ''}) \times \mathrm {U}(W_2^{\epsilon '}) \quad \text { with respect to} \quad (\phi _2^{-1}, \chi _W) \end{aligned}$$

and

$$\begin{aligned} \mathrm {U}(L_{-1}) \times \mathrm {U}(W_2^{\epsilon '}) \quad \text { with respect to} \quad (\chi ^{-1}, \chi _W), \end{aligned}$$

so that the theta correspondence for

$$\begin{aligned} \mathrm {U}(V_2^{\epsilon ''}) \times \mathrm {U}(W_2^{\epsilon '}) \end{aligned}$$

is with respect to $(\phi _2^{-1} \chi ^{-1}, \chi _W)$. The see-saw identity then reads:

$$\begin{aligned} 0\ne & {} {\text {Hom}}_{\mathrm {U}(W_2^{\epsilon '})}( \pi ^{\vee } \otimes \omega _{\psi , \chi , W_2^{\epsilon '}}^{\vee }, \tau ^{\vee })\\= & {} {\text {Hom}}_{\mathrm {U}(V_1^{\epsilon ''})} (\varTheta _{\psi , \phi _2^{-1} \chi ^{-1}, \chi _W, V_2^{\epsilon ''}, W_2^{\epsilon '}}(\tau ^{\vee }), \mu '). \end{aligned}$$

In particular, $\varTheta _{\psi , \phi _2^{-1} \chi ^{-1}, \chi _W, V_2^{\epsilon ''}, W_2^{\epsilon '}}(\tau ^{\vee }) \ne 0$ on $\mathrm {U}(V_2^{\epsilon ''})$. By Theorem 4.1(ii), one deduces that

$$\begin{aligned} \epsilon '' \cdot \epsilon '&= \epsilon \left( \tfrac{1}{2}, \phi _{\tau }^\vee \otimes \phi _2 \chi , \psi ^E_2\right) \\&= \epsilon \left( \tfrac{1}{2}, \mu _E^{-1} \phi _2 \chi _V^{-1} \chi _W, \psi ^E_2\right) \cdot \epsilon \left( \tfrac{1}{2}, \phi _2 \chi _V^{-1} , \psi ^E_2\right) . \end{aligned}$$

By (6.1) and (6.2), and noting that $\epsilon ' = \eta (a_1) \cdot \eta (a_2)$, we see that

$$\begin{aligned} \eta (a_2) = \theta (\eta )(a_2) \cdot \epsilon \left( \tfrac{1}{2}, \phi _2 \chi _V^{-1}, \psi ^E_2\right) \end{aligned}$$

as desired. It then follows by Theorem 4.1(ii) that

$$\begin{aligned} \eta (a_1) = \theta (\eta )(a_1) \cdot \epsilon \left( \tfrac{1}{2}, \phi _1 \chi _V^{-1} , \psi ^E_2\right) \end{aligned}$$

as well.

Thus, we have demonstrated (P1)$_2$, and hence (P2)$_2$.

6.2 Inductive step

Now we assume that $n \ge 3$ and (P2)$_k$ holds for all $k < n$. Proposition 4.5 implies that (P2)$_n$ holds for all nontempered L-parameters. We are thus reduced to the case of tempered L-parameters. Then we have the following theorem whose proof will be given in the next two sections:

Theorem 6.1

If $(P2 )_k$ holds for all tempered L-parameters for all $k < n$, then $(P2 )_n$ holds for all tempered but non-square-integrable L-parameters.

The proof of this theorem is an elaborate extension of the techniques developed in the PhD thesis of the second author [31]. Assuming this theorem for the moment, we are thus reduced to the case of square-integrable L-parameters.

6.3 Square-integrable case

We now consider (P2)$_n$ for a square-integrable L-parameter

$$\begin{aligned} \phi = \phi _1 \oplus \cdots \oplus \phi _r \end{aligned}$$

for $\mathrm {U}(W_n^\pm )$. Thus $\phi$ is multiplicity-free and each $\phi _i$ is an $n_i$-dimensional irreducible conjugate self-dual representation of $WD _E$ with sign $(-1)^{n-1}$. Recall that the component group $S_{\phi }$ is of the form

$$\begin{aligned} S_{\phi } = \prod _{i=1}^r (\mathbb {Z}/ 2\mathbb {Z}) a_i. \end{aligned}$$

We shall first assume that $r > 1$. Then either $r \ge 3$ or else $r =2$ in which case we may assume that $n_1 = \dim \phi _1 \ge 2$.

Let $\pi = \pi (\eta ) \in \varPi _{\phi }^{\epsilon '}$ be an irreducible square-integrable representation of $\mathrm {U}(W_n^{\epsilon '})$ with associated character $\eta \in {\text {Irr}}(S_{\phi })$. We consider the theta correspondence for $\mathrm {U}(V_{n+1}^\epsilon ) \times \mathrm {U}(W_n^{\epsilon '})$ with respect to the data $(\psi , \chi _V, \chi _W)$, and suppose that

$$\begin{aligned} \pi ' := \varTheta _{\psi , V_{n+1}^{\epsilon }, W_n^{\epsilon '}}(\pi ) \ne 0. \end{aligned}$$

Then by Theorem 4.4, $\pi ' = \pi '(\eta ') \in \varPi _{\theta (\phi )}^\epsilon$ is an irreducible tempered representation of $\mathrm {U}(V_{n+1}^\epsilon )$ with associated character $\eta ' \in {\text {Irr}}(S_{\theta (\phi )})$. We want to determine $\eta '$ in terms of $\eta$. Indeed, recall that there is a natural embedding

$$\begin{aligned} S_{\phi } \hookrightarrow S_{\theta (\phi )} \end{aligned}$$

and we need to show that $\eta '(a_i) = \eta (a_i)$. We shall do so by a global argument.

6.4 Globalization

Let us begin the process of globalization which is the most delicate part of the argument. Choose a number field $\mathbb {F}$ and a quadratic field extension $\mathbb {E}$ of $\mathbb {F}$ such that

• $\mathbb {F}$ is totally complex;

• $\mathbb {E}_{v_0} / \mathbb {F}_{v_0} = E/F$ for a finite place $v_0$ of $\mathbb {F}$;

• there is a fixed finite place w of $\mathbb {F}$ which is split in $\mathbb {E}$.

Fix:

• a nontrivial additive character $\varPsi$ of ${\mathbb {A}} / F$ such that $\varPsi _{v_0} = \psi$ (in its ${\text {N}}_{E/F}(E^{\times })$-orbit);

• a conjugate symplectic Hecke character $\chi$ of $\mathbb {A}_{\mathbb {E}}^{\times }$;

• a trace zero element $\delta \in \mathbb {E}^{\times }$ so that the signs of the skew-Hermitian spaces $W_n^{\pm }$ at the place $v_0$ are defined using $\delta$.

Let S be a sufficiently large finite set of inert finite places of $\mathbb {F}$, not containing $v_0$, such that for all $v \notin S \cup \{ v_0 \}$, either v is split in $\mathbb {E}$ or else $\mathbb {E}_v/\mathbb {F}_v$, $\varPsi _v$ and $\chi _v$ are all unramified. Moreover, S can be made arbitrarily large.

If is an isobaric sum of irreducible cuspidal automorphic representations of $\mathrm {GL}_{n_i}(\mathbb {A}_{\mathbb {E}})$, we say that $\varSigma$ is a tempered A-parameter for $\mathrm {U}(\mathbb {W}_n)$, where $\mathbb {W}_n$ is an n-dimensional skew-Hermitian space over $\mathbb {E}$, if

• $\sum _{i=1}^r n_i = n$,

• $\varSigma _i \ne \varSigma _j$ if $i \ne j$,

• the (twisted) Asai L-function $L(s, \varSigma _i, \mathrm {As}^{(-1)^{n-1}})$ has a pole at $s=1$ for all i.

We shall globalize the L-parameter $\phi$ to a tempered A-parameter $\varSigma$ as follows.

• (i) At $v_0$, consider the given irreducible representation $\phi _i$ of $WD _E$. Since $\phi _i$ is conjugate self-dual with sign $(-1)^{n-1}$, it may not be an L-parameter for $\mathrm {U}(W_{n_i}^{\pm })$. Instead, the representation

  $$\begin{aligned} \phi _{i,v_0}' := \phi _i \otimes \chi _{v_0}^{n_i-n} \end{aligned}$$

  is conjugate self-dual with sign $(-1)^{n_i-1}$, and thus defines an L-parameter for $\mathrm {U}(W_{n_i}^{\pm })$.

• (ii) At $v \in S$, choose a representation $\phi _{i,v}$ of $WD _E$ which is the multiplicity-free sum of 1-dimensional conjugate self-dual characters with sign $(-1)^{n-1}$. As above, $\phi _{i,v}$ is conjugate self-dual with sign $(-1)^{n-1}$ and thus may not be an L-parameter for $\mathrm {U}(W_{n_i,v}^{\pm })$, where $W_{n_i,v}^{\pm }$ are the $n_i$-dimensional skew-Hermitian spaces over $E_v$. We set

  $$\begin{aligned} \phi _{i,v}' := \phi _{i,v} \otimes \chi _v^{n_i-n}, \end{aligned}$$

  so that $\phi _{i,v}'$ is an L-parameter for $\mathrm {U}(W_{n_i,v}^\pm )$. The local component group $S_{\phi '_{i,v}}$ of $\phi '_{i,v}$ is of the form

  $$\begin{aligned} S_{\phi '_{i,v}} = (\mathbb {Z}/ 2\mathbb {Z})^{n_i} \end{aligned}$$

  and the Vogan L-packet $\varPi _{\phi '_{i,v}}$ consists of $2^{n_i}$ irreducible square-integrable representations of $\mathrm {U}(W_{n_i,v}^{\pm })$.

• (iii) We require in addition that, for all $v \in S$,

  $$\begin{aligned} \phi _v := \phi _{1,v} \oplus \cdots \oplus \phi _{r,v} \end{aligned}$$

  is not multiplicity-free, i.e. $\phi _v$ is not a square-integrable L-parameter for $\mathrm {U}(W_{n,v}^{\pm })$. To achieve this, we pick a character $\mu _v$ contained in $\phi _{1,v}$ and then ensure that $\mu _v$ is also contained in $\phi _{i_v,v}$ for some $i_v \ge 2$. It is here that we use the assumption that $r > 1$. Moreover, we may ensure that

  $$\begin{aligned} i_v \ne i_{v'} \end{aligned}$$

  for some distinct $v, v' \in S$ if $r > 2$.

• (iv) For each $v \in S$, there is a natural map

  $$\begin{aligned} (\mathbb {Z}/ 2 \mathbb {Z})^r = \prod _{i=1}^r (\mathbb {Z}/ 2\mathbb {Z}) a_i \longrightarrow S_{\phi _v} \end{aligned}$$

  which sends $a_i$ to the image of the element $-1_{\phi _{i,v}}$ in $S_{\phi _v}$. In view of (iii), for $\# S$ large enough (indeed, for $\# S \ge 2$), the induced diagonal map

  $$\begin{aligned} (\mathbb {Z}/ 2\mathbb {Z})^r \longrightarrow \prod _{v \in S} S_{\phi _v} \end{aligned}$$

  is injective.

• (v) Now for each $i=1, \ldots , r$, we have a collection of square-integrable L-parameters $\phi '_{i,v}$ for $v \in S \cup \{ v_0 \}$. For each $v \in S \cup \{ v_0 \}$, pick an irreducible square-integrable representation $\pi _v \in \varPi _{\phi '_{i,v}}^+$. Let $\mathbb {W}_{n_i}^+$ be the $n_i$-dimensional skew-Hermitian space over $\mathbb {E}$ whose localization at each inert v is $W_{n_i,v}^+$, where we have used the trace zero element $\delta \in \mathbb {E}^{\times }$ to define the sign of a skew-Hermitian space over $E_v$. Then by a result of Shin [55, Theorem 5.13] (proved using the trace formula), one can find an irreducible cuspidal automorphic representation $\varPi '_i$ of $\mathrm {U}(\mathbb {W}_{n_i}^+)(\mathbb {A})$ such that

  • $\varPi '_{i,v} = \pi _v$ for all $v \in S\cup \{ v_0 \}$;

  • $\varPi '_{i,v}$ is unramified for all inert $v \notin S \cup \{ v_0 \}$;

  • $\varPi '_{i,w}$ is an irreducible supercuspidal representation of $\mathrm {U}(\mathbb {W}_{n_i,w}^+) \cong \mathrm {GL}_{n_i}(\mathbb {F}_w)$.

• (vi) By results of Mok [44], the representation $\varPi '_i$ has tempered A-parameter $\varSigma '_i$, which is an irreducible cuspidal automorphic representation of $\mathrm {GL}_{n_i}(\mathbb {A}_{\mathbb {E}})$ such that $L(s, \varSigma _i', \mathrm {As}^{(-1)^{n_i-1}})$ has a pole at $s=1$. The cuspidality of $\varSigma '_i$ is a consequence of the fact that $\varPi '_{i,w}$ is supercuspidal at the split place w. If we set

  $$\begin{aligned} \varSigma _i = \varSigma _i' \otimes \chi ^{n-n_i}, \end{aligned}$$

  then $\varSigma _i$ is an irreducible cuspidal automorphic representation of $\mathrm {GL}_{n_i}(\mathbb {A}_{\mathbb {E}})$ such that $L(s, \varSigma _i, \mathrm {As}^{(-1)^{n-1}})$ has a pole at $s=1$. In particular, setting

  

  we see that $\varSigma$ is a tempered A-parameter for $\mathrm {U}(\mathbb {W}_n)$, where $\mathbb {W}_n$ is an n-dimensional skew-Hermitian space over $\mathbb {E}$.

6.5 Properties of $\varSigma$

We have completed the construction of a global tempered A-parameter $\varSigma$. Let us examine some crucial properties of $\varSigma$.

• (Local components) It follows by construction that the local components of the A-parameter $\varSigma$ are given as follows:

  • at the place $v_0$, $\varSigma _{v_0}$ has L-parameter $\phi$;

  • at all places $v \in S$, $\varSigma _v$ has L-parameter $\phi _v$;

  • at all inert places $v \notin S \cup \{ v_0 \}$, $\varSigma _v$ is unramified.

  In particular, we have found a globalization $\varSigma$ of the given local L-parameter $\phi$ so that at all inert places $v \ne v_0$ of $\mathbb {F}$, $\varSigma _v$ defines a non-square-integrable L-parameter for $\mathrm {U}(W_{n,v}^{\pm })$.

• (Whittaker data) We shall use the additive character $\varPsi = \otimes _v \varPsi _v$ to fix the Whittaker datum at each place v. Together with the fixed trace zero element $\delta \in \mathbb {E}^{\times }$, we have thus fixed the local Langlands correspondence for $\mathrm {U}(W_{n,v}^{\pm })$ for each v.

• (Component groups) The global component group $S_{\varSigma }$ of the A-parameter $\varSigma$ admits a natural map $S_{\varSigma } \rightarrow S_{\varSigma _v}$ for each place v. For $v = v_0$, this natural map is an isomorphism, so that we have a canonical identification:

  $$\begin{aligned} S_{\varSigma } = S_{\varSigma _{v_0}} = \prod _{i=1}^r (\mathbb {Z}/2 \mathbb {Z}) a_i. \end{aligned}$$

  On the other hand, in view of (iv) above, we see that the diagonal map

  $$\begin{aligned} S_{\varSigma } \longrightarrow \prod _{v \ne v_0} S_{\varSigma _v} \end{aligned}$$

  is injective. Thus, given any $\eta \in {\text {Irr}}(S_{\phi }) = {\text {Irr}}(S_{\varSigma _{v_0}})$, one can find $\eta _v \in {\text {Irr}}(S_{\varSigma _v})$ for $v \ne v_0$ so that

  $$\begin{aligned} \Bigl ( \eta \otimes \Bigl ( \bigotimes _{v \ne v_0} \eta _v \Bigr ) \Bigr ) \circ \varDelta = \mathbbm {1}_{S_{\varSigma }}, \end{aligned}$$

  where

  $$\begin{aligned} \varDelta : S_{\varSigma } \longrightarrow \prod _v S_{\varSigma _v} \end{aligned}$$

  is the diagonal map.

• (Arthur’s multiplicity formula) Consider the global A-packet associated to $\varSigma$. For any collection $\eta _v \in {\text {Irr}}(S_{\varSigma _v})$ of irreducible characters with associated representations $\pi (\eta _v)$ of local unitary groups $\mathrm {U}(W_{n,v}^{\epsilon _v'})$, consider the representation

  $$\begin{aligned} \varPi : = \bigotimes _v \pi (\eta _v) \end{aligned}$$

  of the adelic unitary group $\prod '_v \mathrm {U}(W_{n,v}^{\epsilon _v'})$. Arthur’s multiplicity formula [33, Theorem 1.7.1] then states that the following are equivalent:

  • the adelic unitary group $\prod '_v \mathrm {U}(W_{n,v}^{\epsilon _v'})$ is equal to $\mathrm {U}(\mathbb {W}_n)(\mathbb {A})$ for a skew-Hermitian space $\mathbb {W}_n$ over $\mathbb {E}$ and $\varPi$ occurs in the automorphic discrete spectrum

    $$\begin{aligned} L^2_{\mathrm {disc}}(\mathrm {U}(\mathbb {W}_n)(\mathbb {F}) \backslash \mathrm {U}(\mathbb {W}_n)(\mathbb {A})); \end{aligned}$$

  • the character $(\otimes _v \eta _v) \circ \varDelta$ of $S_{\varSigma }$ is trivial.

By the above discussion combined with a result of Wallach [65], [9, Proposition 4.10], we may find an n-dimensional skew-Hermitian space $\mathbb {W}_n$ over $\mathbb {E}$ and an irreducible cuspidal automorphic representation $\varPi$ of $\mathrm {U}(\mathbb {W}_n)(\mathbb {A})$ in the global A-packet associated to $\varSigma$ such that $\varPi _{v_0} = \pi (\eta )$. For each v, we shall write the local component $\varPi _v$ as $\pi (\eta _v)$.

6.6 Global theta correspondence

Now we shall construct a Hermitian space $\mathbb {V}_{n+1}$ of dimension $n+1$ over $\mathbb {E}$, and consider the global theta correspondence for $\mathrm {U}(\mathbb {V}_{n+1}) \times \mathrm {U}(\mathbb {W}_n)$. To define such a global theta correspondence, we shall use the fixed additive character $\varPsi$ of $\mathbb {A}/ \mathbb {F}$, and we also need to fix a pair of Hecke characters $\chi _{\mathbb {V}}$ and $\chi _{\mathbb {W}}$ of $\mathbb {A}_{\mathbb {E}}^{\times }$ such that

$$\begin{aligned} \chi _{\mathbb {V}}|_{\mathbb {A}^{\times }} = \omega _{\mathbb {E}/ \mathbb {F}}^{n+1} \quad \text { and} \quad \chi _{\mathbb {W}}|_{\mathbb {A}^{\times }} = \omega _{\mathbb {E}/ \mathbb {F}}^n, \end{aligned}$$

where $\omega _{\mathbb {E}/ \mathbb {F}}$ is the quadratic Hecke character of $\mathbb {A}^{\times }$ associated to $\mathbb {E}/\mathbb {F}$ by global class field theory. We pick these so that, in addition:

• (a) at the place $v_0$, we have

  $$\begin{aligned} \chi _{\mathbb {V},v_0} = \chi _V \quad \text { and} \quad \chi _{\mathbb {W},v_0} = \chi _W; \end{aligned}$$

• (b) at some place $v_1 \in S$, $\chi _{\mathbb {V},v_1}$ is not contained in the L-parameter associated to $\varSigma _{v_1}$.

Indeed, since $\mathbb {E}^{\times } / \mathbb {F}^{\times } \cong {\text {Ker}}({\text {N}}_{\mathbb {E}/\mathbb {F}})$ is anisotropic, for given conjugate orthogonal characters $\mu _i$ of $\mathbb {E}_{v_i}^{\times }$, there is a conjugate orthogonal Hecke character $\mu$ of $\mathbb {A}_{\mathbb {E}}^{\times }$ such that $\mu _{v_i} = \mu _i$ for $i=0,1$. Thus, we can achieve (a) and (b) by replacing $\chi _{\mathbb {V}}$ and $\chi _{\mathbb {W}}$ by their twists by conjugate orthogonal Hecke characters of $\mathbb {A}_\mathbb {E}^{\times }$ if necessary. The condition (b) guarantees that at the place $v_1$, the representation $\varPi _{v_1}$ has nonzero local theta lift to both $\mathrm {U}(V_{n+1,v_1}^+)$ and $\mathrm {U}(V_{n+1, v_1}^-)$ by Theorem 4.4(i)(a). Moreover, the conservation relation (proved by Sun–Zhu [57]) implies that the theta lifts of $\varPi _{v_1}$ to $\mathrm {U}(V_{n-1,v_1}^+)$ and $\mathrm {U}(V_{n-1, v_1}^-)$ are both zero.

Now we note:

Lemma 6.2

There is a Hermitian space $\mathbb {V}_{n+1}$ of dimension $n+1$ over $\mathbb {E}$ such that:

• at the place $v_0$, $\mathbb {V}_{n+1,v_0}$ is equal to the given Hermitian space $V_{n+1}^\epsilon$;

• for all places v, the representation $\varPi _v$ has nonzero local theta lift to $\mathrm {U}(\mathbb {V}_{n+1,v})$ with respect to the theta lift defined by the data $(\varPsi _v, \chi _{\mathbb {V}_v}, \chi _{\mathbb {W}_v})$.

Proof

For all $v \ne v_0, v_1$, we may pick $\mathbb {V}_{n+1, v}$ so that the local theta lift of $\varPi _v$ to $\mathrm {U}(\mathbb {V}_{n+1, v})$ is nonzero, and then complete these to a coherent collection of Hermitian spaces by picking $V_{n+1}^\epsilon$ at $v_0$ and the uniquely determined Hermitian space at $v_1$. $\square$

6.7 Completion of the proof

Consider the global theta lift $\varPi ' := \varTheta _{\varPsi , \mathbb {V}_{n+1}, \mathbb {W}_n}(\varPi )$ to $\mathrm {U}(\mathbb {V}_{n+1})(\mathbb {A})$. The condition (b) above ensures that $\varPi '$ is cuspidal. To show that $\varPi '$ is nonzero, we consider the standard L-function $L(s, \varPi )$ of $\varPi$ defined using the doubling zeta integral of Piatetski-Shapiro–Rallis [40, 46]. Observe that the partial L-function $L^{S \cup \{v_0\}}(s, \varPi )$ agrees with the partial standard L-function $L^{S \cup \{v_0\}}(s, \varSigma )$ of $\varSigma$, so that

$$\begin{aligned} L^{S \cup \{v_0\}}(1, \varPi ) = L^{S \cup \{v_0\}}(1, \varSigma ) = \prod _{i=1}^r L^{S \cup \{v_0\}}(1, \varSigma _i) \ne 0 \end{aligned}$$

since $\varSigma _i$ is unitary and cuspidal. By [40, Proposition 5], the local standard L-factor $L(s, \varPi _v)$ at $v \in S \cup \{ v_0\}$ is holomorphic and nonzero at $s=1$ since $\varPi _v$ is tempered. Hence

$$\begin{aligned} L(1, \varPi ) \ne 0 \end{aligned}$$

and it follows by [18, Theorem 1.4] that $\varPi '$ is nonzero. Thus $\varPi '$ is an irreducible cuspidal automorphic representation of $\mathrm {U}(\mathbb {V}_{n+1})(\mathbb {A})$ such that $\varPi '_{v_0} = \pi '(\eta ')$.

Recall that we have fixed the local Langlands correspondence for $\mathrm {U}(\mathbb {W}_{n,v})$ for each v using the Whittaker datum determined by the additive character $\varPsi _v$ together with the trace zero element $\delta$. To fix the local Langlands correspondence for $\mathrm {U}(\mathbb {V}_{n+1,v})$ for each v, we shall use the Whittaker datum determined by the additive character $\varPsi ^{\mathbb {E}_v}_v = \varPsi _v\Big (\frac{1}{2} {\text {Tr}}_{\mathbb {E}_v/\mathbb {F}_v}(\delta \, \cdot \,)\Big )$. Then we may write

$$\begin{aligned} \varPi = \bigotimes _v \pi (\eta _v) \quad \text { and} \quad \varPi ' = \bigotimes _v \pi '(\eta '_v) \end{aligned}$$

with associated irreducible characters $\eta _v$ and $\eta '_v$ of the local component groups.

Recall that $\varPi$ has tempered A-parameter. By Theorem 4.4, $\varPi '$ also has tempered A-parameter. Hence, applying Arthur’s multiplicity formula [33, Theorem 1.7.1] to $\varPi$ and $\varPi '$, we see that

$$\begin{aligned} \prod _v \eta _v(a_{i,v}) = 1 \quad \text { and} \quad \prod _v \eta _v'(a_{i,v}) = 1 \end{aligned}$$

(6.3)

for all i, where $a_{i,v}$ is the image of $a_i$ in $S_{\varSigma _v}$. However, for all places $v \ne v_0$, either v is split, or else the L-parameter of $\varPi _v$ is not square-integrable. Thus, for all inert $v \ne v_0$, one knows that (P2)$_n$ holds. In particular,

$$\begin{aligned} \eta '_v(a_{i,v}) = \eta _v(a_{i,v}) \end{aligned}$$

for all $v \ne v_0$. Thus, we conclude that at the place $v_0$, we have

$$\begin{aligned} \eta ' (a_i) = \eta (a_i) \end{aligned}$$

as desired.

We have thus completed the proof of (P2)$_n$ when $r >1$, i.e. when $\phi$ is reducible. To deal with the case when $\phi$ is irreducible, with $r = 1$, we can again appeal to a variation of the global argument as above. Namely, in the globalization step above, we may now take the L-parameter $\phi _v$ for $v \in S$ to be square-integrable L-parameters which are reducible. Then the rest of the argument is the same, using the fact that we have shown (P2)$_n$ for every place $v \ne v_0$. This completes the proof of (P2)$_n$.

7 Preparations for the proof of Theorem 6.1

To finish the proof of (P2), it now remains to prove Theorem 6.1. For this, we need to introduce more notation. Fix $\varepsilon = \pm 1$. In this and next sections, we shall let V and W be an $\varepsilon$-Hermitian space and a $(-\varepsilon )$-Hermitian space respectively. Put

$$\begin{aligned} m = \dim V \quad \text { and} \quad n = \dim W. \end{aligned}$$

7.1 Parabolic subgroups

Let r be the Witt index of V and $V_\mathrm {an}$ an anisotropic kernel of V. Choose a basis $\{ v_i, v_i^* \, | \, i = 1, \ldots , r \}$ of the orthogonal complement of $V_\mathrm {an}$ such that

$$\begin{aligned} \langle v_i, v_j \rangle _V = \langle v^*_i, v^*_j \rangle _V = 0, \quad \langle v_i, v^*_j \rangle _V = \delta _{i,j} \end{aligned}$$

for $1 \le i, j \le r$. Let k be a positive integer with $k \le r$ and set

$$\begin{aligned} X = E v_1 \oplus \cdots \oplus E v_k, \quad X^* = E v^*_1 \oplus \cdots \oplus E v^*_k. \end{aligned}$$

Let $V_0$ be the orthogonal complement of $X \oplus X^*$ in V, so that $V_0$ is an $\varepsilon$-Hermitian space of dimension $m_0 = m -2k$ over E. We shall write an element in the unitary group $\mathrm {U}(V)$ as a block matrix relative to the decomposition $V = X \oplus V_0 \oplus X^*$. Let $P = M_P U_P$ be the maximal parabolic subgroup of $\mathrm {U}(V)$ stabilizing X, where $M_P$ is the Levi component of P stabilizing $X^*$ and $U_P$ is the unipotent radical of P. We have

$$\begin{aligned} M_P&= \{ m_P(a) \cdot h_0 \, | \, a \in \mathrm {GL}(X), \, h_0 \in \mathrm {U}(V_0)\}, \\ U_P&= \{ u_P(b) \cdot u_P(c) \, | \, b \in {\text {Hom}}(V_0, X), \, c \in {\text {Herm}}(X^*,X) \}, \end{aligned}$$

where

$$\begin{aligned} m_P(a)&= \begin{pmatrix} a &{} &{} \\ &{} 1_{V_0} &{} \\ &{} &{} (a^*)^{-1} \end{pmatrix}, \\ u_P(b)&= \begin{pmatrix} 1_X &{} b &{} - \tfrac{1}{2} b b^* \\ &{} 1_{V_0} &{} -b^* \\ &{} &{} 1_{X^*} \end{pmatrix}, \\ u_P(c)&= \begin{pmatrix} 1_X &{} &{} c \\ &{} 1_{V_0} &{} \\ &{} &{} 1_{X^*} \end{pmatrix}, \end{aligned}$$

and

$$\begin{aligned} {\text {Herm}}(X^*, X) = \{ c \in {\text {Hom}}(X^*,X) \, | \, c^* = -c \}. \end{aligned}$$

Here, the elements $a^* \in \mathrm {GL}(X^*)$, $b^* \in {\text {Hom}}(X^*, V_0)$, and $c^* \in {\text {Hom}}(X^*, X)$ are defined by requiring that

$$\begin{aligned} \langle a x, x' \rangle _V \!= & {} \! \langle x, a^* x' \rangle _V,\\ \langle b v, x' \rangle _V= & {} \langle v, b^* x' \rangle _V, \\ \langle c x', x'' \rangle _V= & {} \langle x', c^* x'' \rangle _V \end{aligned}$$

for $x \in X$, $x', x'' \in X^*$, and $v \in V_0$. In particular, $M_P \cong \mathrm {GL}(X) \times \mathrm {U}(V_0)$ and

$$\begin{aligned} 1 \longrightarrow {\text {Herm}}(X^*,X) \longrightarrow U_P \longrightarrow {\text {Hom}}(V_0, X) \longrightarrow 1. \end{aligned}$$

Put

$$\begin{aligned} \rho _P = \frac{m_0+k}{2}, \quad w_P = \begin{pmatrix} &{} &{} -I_X \\ &{} 1_{V_0} &{} \\ -\varepsilon I_X^{-1} &{} &{} \end{pmatrix}, \end{aligned}$$

where $I_X \in {\text {Isom}}(X^*, X)$ is defined by $I_X v_i^* = v_i$ for $1 \le i \le k$.

Similarly, let $r'$ be the Witt index of W and choose a basis $\{ w_i, w_i^* \, | \, i = 1, \ldots , r' \}$ of the orthogonal complement of an anisotropic kernel of W such that

$$\begin{aligned} \langle w_i, w_j \rangle _W = \langle w^*_i, w^*_j \rangle _W = 0, \quad \langle w_i, w^*_j \rangle _W = \delta _{i,j} \end{aligned}$$

for $1 \le i, j \le r'$. We assume that $k \le r'$ and set

$$\begin{aligned} Y = E w_1 \oplus \cdots \oplus E w_k, \quad Y^* = E w^*_1 \oplus \cdots \oplus E w^*_k. \end{aligned}$$

Let $W_0$ be the orthogonal complement of $Y \oplus Y^*$ in W, so that $W_0$ is a $(- \varepsilon )$-Hermitian space of dimension $n_0 = n -2k$ over E. Let $Q = M_Q U_Q$ be the maximal parabolic subgroup of $\mathrm {U}(W)$ stabilizing Y, where $M_Q$ is the Levi component of Q stabilizing $Y^*$ and $U_Q$ is the unipotent radical of Q. Then $M_Q \cong \mathrm {GL}(Y) \times \mathrm {U}(W_0)$ and

$$\begin{aligned} 1 \longrightarrow {\text {Herm}}(Y^*,Y) \longrightarrow U_Q \longrightarrow {\text {Hom}}(W_0, Y) \longrightarrow 1, \end{aligned}$$

where

$$\begin{aligned} {\text {Herm}}(Y^*,Y)= \{c \in {\text {Hom}}(Y^*, Y)\,|\,c^*=-c\}. \end{aligned}$$

For $a \in \mathrm {GL}(Y)$, $b \in {\text {Hom}}(W_0, Y)$, and $c \in {\text {Herm}}(Y^*,Y)$, we define elements $m_Q(a) \in M_Q$ and $u_Q(b), u_Q(c) \in U_Q$ as above. Put

$$\begin{aligned} \rho _Q = \frac{n_0+k}{2}, \quad w_Q = \begin{pmatrix} &{} &{} -I_Y \\ &{} 1_{W_0} &{} \\ \varepsilon I_Y^{-1} &{} &{} \end{pmatrix}, \end{aligned}$$

where $I_Y \in {\text {Isom}}(Y^*, Y)$ is defined by $I_Y w_i^* = w_i$ for $1 \le i \le k$.

7.2 Haar measures

We need to choose Haar measures on various groups. In particular, we shall define Haar measures on $U_P$ and $U_Q$ in the following.

Recall the symplectic form

$$\begin{aligned} \langle \cdot , \cdot \rangle = {\text {Tr}}_{E/F}(\langle \cdot , \cdot \rangle _V \otimes \langle \cdot , \cdot \rangle _W) \end{aligned}$$

on $V \otimes W$ over F. We consider the following spaces and pairings:

• $(x, y) \mapsto \psi (\langle x, I_Y^{-1} y \rangle )$ for $x, y \in V \otimes Y$;

• $(x, y) \mapsto \psi (\langle x, I_Y y \rangle )$ for $x, y \in V_0 \otimes Y^*$;

• $(x, y) \mapsto \psi (\langle I_X^{-1} x, y \rangle )$ for $x, y \in X \otimes W_0$;

• $(x, y) \mapsto \psi (\langle I_X x, y \rangle )$ for $x, y \in X^* \otimes W_0$;

• $(x, y) \mapsto \psi (\langle I_X^{-1} x, I_Y y \rangle )$ for $x, y \in X \otimes Y^*$;

• $(x, y) \mapsto \psi (\langle I_X x, I_Y^{-1} y \rangle )$ for $x, y \in X^* \otimes Y$;

• $(x, y) \mapsto \psi (\langle I_X x, I_Y y \rangle )$ for $x, y \in X^* \otimes Y^*$.

On these spaces, we take the self-dual Haar measures with respect to these pairings. Put

$$\begin{aligned} e^{**} = v^*_1 \otimes w^*_1 + \cdots + v_k^* \otimes w^*_k \in X^* \otimes Y^*. \end{aligned}$$

• We transfer the Haar measure on $V_0 \otimes Y^*$ to ${\text {Hom}}(X^*, V_0)$ via the isomorphism $x \mapsto x e^{**}$ for $x \in {\text {Hom}}(X^*, V_0)$.

• We transfer the Haar measure on ${\text {Hom}}(X^*,V_0)$ to ${\text {Hom}}(V_0, X)$ via the isomorphism $x \mapsto x^*$ for $x \in {\text {Hom}}(V_0, X)$.

• Similarly, we define the Haar measure on ${\text {Hom}}(W_0, Y)$.

Furthermore:

• We transfer the Haar measure on $X \otimes Y^*$ to ${\text {Hom}}(X^*, X)$ via the isomorphism $x \mapsto x e^{**}$ for $x \in {\text {Hom}}(X^*, X)$. This Haar measure on ${\text {Hom}}(X^*, X)$ is self-dual with respect to the pairing $(x, y) \mapsto \psi (\langle I_X^{-1} x e^{**}, I_Y y e^{**} \rangle )$.

• We take the Haar measure $|2|_F^{-k^2/2} \, dx$ on ${\text {Herm}}(X^*,X)$, where dx is the self-dual Haar measure on ${\text {Herm}}(X^*,X)$ with respect to the pairing $(x, y) \mapsto \psi (\langle I_X^{-1} x e^{**}, I_Y y e^{**} \rangle )$.

• Similarly, we define the Haar measure on ${\text {Herm}}(Y^*, Y)$.

Then:

• We take the Haar measure $du = db \, dc$ on $U_{P}$ for $u = u_{P}(b) u_{P}(c)$ with $b \in {\text {Hom}}(V_0,X)$ and $c \in {\text {Herm}}(X^*,X)$.

• Similarly, we define the Haar measure on $U_Q$.

We note the following Fourier inversion formula:

Lemma 7.1

For $\varphi \in \mathscr {S}(X \otimes Y^*)$, we have

$$\begin{aligned}&\int _{{\text {Herm}}(Y^*,Y)} \left( \int _{{\text {Hom}}(X^*,X)} \varphi (x e^{**}) \psi ( \langle x e^{**}, c e^{**} \rangle ) dx \right) dc\\&\qquad \qquad \qquad \qquad \qquad \qquad \quad \qquad \qquad \qquad \qquad \qquad \quad = \int _{{\text {Herm}}(X^*,X)} \varphi (c e^{**}) \, d c. \end{aligned}$$

Proof

We consider the nondegenerate symmetric bilinear form $(x, y) \mapsto \langle I_X^{-1} x, I_Y y \rangle$ on $X \otimes Y^*$ over F, and the subspaces

$$\begin{aligned} {\text {Herm}}(X^*, X) e^{**} \quad \text { and} \quad I_X I_Y^{-1} {\text {Herm}}(Y^*, Y) e^{**} \end{aligned}$$

of $X \otimes Y^* = {\text {Hom}}(X^*, X) e^{**}$. For $x \in {\text {Hom}}(X^*, X)$ and $y \in {\text {Herm}}(Y^*, Y)$, we have

$$\begin{aligned} \langle I_X^{-1} x e^{**}, I_Y I_X I_Y^{-1} y e^{**} \rangle&= \langle I_X^{-1} x e^{**}, I_X y e^{**} \rangle \\&= \langle I_X^* I_X^{-1} x e^{**}, y e^{**} \rangle \\&= \varepsilon \cdot \langle x e^{**}, y e^{**} \rangle \end{aligned}$$

since $I_X^* = \varepsilon I_X$. For $x \in {\text {Herm}}(X^*, X)$ and $y \in {\text {Herm}}(Y^*, Y)$, noting that $x^* = -x$, $y^* = -y$, and x commutes with y, we have

$$\begin{aligned} \langle x e^{**}, y e^{**} \rangle&= \langle y^* e^{**}, x^* e^{**} \rangle \\&= \langle y e^{**}, x e^{**} \rangle \\&= - \langle x e^{**}, y e^{**} \rangle , \end{aligned}$$

so that

$$\begin{aligned} \langle x e^{**}, y e^{**} \rangle = 0. \end{aligned}$$

Since ${\text {Hom}}(X^*, X) e^{**}$ is nondegenerate with respect to the above bilinear form, we see that $X \otimes Y^*$ decomposes as the orthogonal direct sum

$$\begin{aligned} X \otimes Y^* = {\text {Herm}}(X^*, X) e^{**} \oplus I_X I_Y^{-1} {\text {Herm}}(Y^*, Y) e^{**}. \end{aligned}$$

These yield the desired Fourier inversion formula. $\square$

7.3 Normalized intertwining operators

In this subsection, we define the normalized intertwining operator which is used to describe the local Langlands correspondence.

Let $\tau$ be an irreducible (unitary) square-integrable representation of $\mathrm {GL}(X)$ on a space $\mathscr {V}_{\tau }$ with central character $\omega _{\tau }$. For any $s \in \mathbb {C}$, we realize the representation $\tau _s := \tau \otimes |\det |^s$ on $\mathscr {V}_{\tau }$ by setting $\tau _s(a) v := |\det a|^s \tau (a) v$ for $a \in \mathrm {GL}(X)$ and $v \in \mathscr {V}_{\tau }$. Let $\sigma _0$ be an irreducible tempered representation of $\mathrm {U}(V_0)$ on a space $\mathscr {V}_{\sigma _0}$. We consider the induced representation

$$\begin{aligned} {\text {Ind}}^{\mathrm {U}(V)}_P(\tau _s \otimes \sigma _0) \end{aligned}$$

of $\mathrm {U}(V)$, which is realized on the space of smooth functions $\varPhi _s : \mathrm {U}(V) \rightarrow \mathscr {V}_\tau \otimes \mathscr {V}_{\sigma _0}$ such that

$$\begin{aligned} \varPhi _s(u m_P(a) h_0 h) = |\det a|^{s+\rho _P} \tau (a) \sigma _0(h_0) \varPhi _s(h) \end{aligned}$$

for all $u \in U_P$, $a \in \mathrm {GL}(X)$, $h_0 \in \mathrm {U}(V_0)$, and $h \in \mathrm {U}(V)$. Let $A_P$ be the split component of the center of $M_P$ and $W(M_P) = {\text {Norm}}_{\mathrm {U}(V)}(A_P)/M_P$ the relative Weyl group for $M_P$. Noting that $W(M_P) \cong \mathbb {Z}/2 \mathbb {Z}$, we denote by w the nontrivial element in $W(M_P)$. For any representative $\tilde{w} \in \mathrm {U}(V)$ of w, we define an unnormalized intertwining operator

$$\begin{aligned} \mathcal {M}(\tilde{w}, \tau _s \otimes \sigma _0) : {\text {Ind}}^{\mathrm {U}(V)}_P(\tau _s \otimes \sigma _0) \longrightarrow {\text {Ind}}^{\mathrm {U}(V)}_P(w(\tau _s \otimes \sigma _0)) \end{aligned}$$

by (the meromorphic continuation of) the integral

$$\begin{aligned} \mathcal {M}(\tilde{w}, \tau _s \otimes \sigma _0) \varPhi _s(h) = \int _{U_P} \varPhi _s(\tilde{w}^{-1} u h) \, du, \end{aligned}$$

where $w(\tau _s \otimes \sigma _0)$ is the representation of $M_P$ on $\mathscr {V}_{\tau } \otimes \mathscr {V}_{\sigma _0}$ given by

$$\begin{aligned} (w(\tau _s \otimes \sigma _0))(m) = (\tau _s \otimes \sigma _0)(\tilde{w}^{-1} m \tilde{w}) \end{aligned}$$

for $m \in M_P.$

Now, following [2, 33, 44], we shall normalize the intertwining operator $\mathcal {M}(\tilde{w}, \tau _s \otimes \sigma _0)$, depending on the choice of the Whittaker datum. Having fixed the additive character $\psi$ and the trace zero element $\delta$, we define the sign $\epsilon (V)$ and use the Whittaker datum relative to

$$\begin{aligned} {\left\{ \begin{array}{ll} \psi ^E = \psi \Big (\frac{1}{2} {\text {Tr}}_{E/F}(\delta \, \cdot \,)\Big ) &{} {\text { if } \varepsilon = +1;} \\ \psi &{} {\text { if } \varepsilon = -1.} \end{array}\right. } \end{aligned}$$

The definition of the normalized intertwining operator is very subtle because one has to choose the following data appropriately:

• a representative $\tilde{w}$;

• a normalizing factor $r(w, \tau _s \otimes \sigma _0)$;

• an intertwining isomorphism $\mathcal {A}_w$.

Following the procedure of [39, Sect. 2.1], [2, Sect. 2.3], [44, Sect. 3.3], [33, Sect. 2.3], we take the representative $\tilde{w} \in \mathrm {U}(V)$ of w defined by

$$\begin{aligned} \tilde{w} = w_P \cdot m_P((-1)^{m'} \cdot \kappa _V \cdot J) \cdot (- 1_{V_0})^k, \end{aligned}$$

where $w_{P}$ is as in Sect. 7.1, $m' = {[}\frac{m}{2}{]}$,

$$\begin{aligned} \kappa _V = {\left\{ \begin{array}{ll} -\delta &{}\quad {\text { if } m \text { is even and } \varepsilon = +1;} \\ 1 &{}\quad {{\text { if } m \text { is even and } \varepsilon = -1;}} \\ -1 &{}\quad {\text { if } m \text { is odd and } \varepsilon = +1;} \\ -\delta &{} \quad {\text { if } m \text { is odd and } \varepsilon = -1,} \end{array}\right. } \end{aligned}$$

and

Here, we have identified $\mathrm {GL}(X)$ with $\mathrm {GL}_k(E)$ using the basis $\{ v_1, \ldots , v_k \}$. This element ${\tilde{w}}$ arises as follows.

First assume that $\epsilon (V) = +1$. In particular, $\mathrm {U}(V)$ is quasi-split. We have $V_\mathrm {an}= \{ 0 \}$ if m is even and $V_\mathrm {an}= E v_\mathrm {an}$ for some $v_\mathrm {an}\in V_\mathrm {an}$ such that

$$\begin{aligned} \langle v_\mathrm {an}, v_\mathrm {an}\rangle _V = {\left\{ \begin{array}{ll} 1 &{} \quad {\text { if } \varepsilon = +1;} \\ \delta &{} \quad {\text { if } \varepsilon = -1} \end{array}\right. } \end{aligned}$$

if m is odd. Via the decomposition

$$\begin{aligned} V = E v_1 \oplus \dots \oplus E v_r \oplus V_\mathrm {an}\oplus E v_r^* \oplus \dots \oplus E v_1^*, \end{aligned}$$

we regard $\mathrm {U}(V)$ as a subgroup of $\mathrm {GL}_m(E)$, which induces an isomorphism $\mathrm {U}(V)(\bar{F}) \cong \mathrm {GL}_m(\bar{F})$. Let ${{\varvec{spl}}}= (B,T,\{ X_i \})$ be the F-splitting of $\mathrm {U}(V)$ consisting of the Borel subgroup B stabilizing the flag

$$\begin{aligned} E v_1 \subset E v_1 \oplus E v_2 \subset \dots \subset E v_1 \oplus \dots \oplus E v_r, \end{aligned}$$

the maximal torus T of diagonal matrices, and the set $\{ X_i \, | \, i = 1, \ldots , m-1 \}$ of simple root vectors given as follows:

• $X_i = E_{i,i+1}$ for $1 \le i \le r-1$;

• $X_i = -E_{i,i+1}$ for $m-r+1 \le i \le m-1$;

• if m is even, then

  $$\begin{aligned} X_r = {\left\{ \begin{array}{ll} \delta ^{-1} \cdot E_{r,r+1} &{} {\text { if } \varepsilon = +1,} \\ E_{r,r+1} &{} {\text { if } \varepsilon = -1;} \end{array}\right. } \end{aligned}$$

• if m is odd, then $X_r = E_{r,r+1}$ and

  $$\begin{aligned} X_{r+1} = {\left\{ \begin{array}{ll} -E_{r+1,r+2} &{} {\text { if } \varepsilon = +1,} \\ \delta ^{-1} \cdot E_{r+1,r+2} &{} {\text { if } \varepsilon = -1.} \end{array}\right. } \end{aligned}$$

Here, $E_{i,j} \in {\text {Lie}} \mathrm {U}(V)(\bar{F}) \cong \mathrm {M}_m(\bar{F})$ is the matrix with one at the (i, j)th entry and zero elsewhere. Then ${{\varvec{spl}}}$ and $\psi$ give rise to the above Whittaker datum, whose restriction to $M_P$ is preserved by the representative $\tilde{w}^{\mathrm {LS}}$ of w defined in [39, Sect. 2.1], [2, Sect. 2.3], [44, Sect. 3.3] with respect to ${{\varvec{spl}}}$.

Lemma 7.2

We have $\tilde{w}^{\mathrm {LS}} = \tilde{w}$.

Proof

First, we review the case of $\mathrm {SL}_2$. We take an F-splitting of $\mathrm {SL}_2$ consisting of the Borel subgroup of upper triangular matrices, the maximal torus of diagonal matrices, and a simple root vector

$$\begin{aligned} X = \begin{pmatrix} 0 &{}\quad a \\ 0 &{}\quad 0 \end{pmatrix}. \end{aligned}$$

Let $\{ H, X, Y \}$ be the $\mathfrak {sl}_2$-triple containing X, so that

$$\begin{aligned} Y = \begin{pmatrix} 0 &{}\quad 0 \\ a^{-1} &{}\quad 0 \end{pmatrix}. \end{aligned}$$

If s is the simple reflection with respect to X, then the representative of s defined in [39, Sect. 2.1] is

$$\begin{aligned} \exp (X) \exp (-Y) \exp (X) = \begin{pmatrix} &{} a \\ -a^{-1} &{} \end{pmatrix}. \end{aligned}$$

Now we compute $\tilde{w}^\mathrm {LS}$. Let $\iota _i : \mathrm {GL}(E v_i \oplus E v_{i+1}) \hookrightarrow \mathrm {GL}(X)$ and $\iota '_j : \mathrm {U}(E v_j \oplus E v_j^*) \hookrightarrow \mathrm {U}(V)$ be the natural embeddings. Let $s_i$ be the simple reflection with respect to $X_i$ and $\tilde{s}_i$ the representative of $s_i$ as above. Put $w_i = s_i s_{m-i}$ and $\tilde{w}_i = \tilde{s}_i \tilde{s}_{m-i}$ for $1 \le i \le r-1$, and

$$\begin{aligned} w_r = {\left\{ \begin{array}{ll} s_r &{} {\text { if } m \text { is even,}} \\ s_r s_{r+1} s_r &{} {\text { if } m \text { is odd}} \end{array}\right. } \quad \text { and} \quad \tilde{w}_r = {\left\{ \begin{array}{ll} \tilde{s}_r &{} {\text { if } m \text { is even,}} \\ \tilde{s}_r \tilde{s}_{r+1} \tilde{s}_r &{} {\text { if } m \text { is odd.}} \end{array}\right. } \end{aligned}$$

More explicitly, we have

$$\begin{aligned} \tilde{w}_i = m_P \! \left( \iota _i \! \begin{pmatrix} &{} 1 \\ -1 &{} \end{pmatrix} \right) \end{aligned}$$

for $1 \le i \le r-1$ and

$$\begin{aligned} \tilde{w}_r = \iota '_r \! \left( \begin{pmatrix} &{} 1 \\ \varepsilon &{} \end{pmatrix} \begin{pmatrix} \kappa _V &{} \\ &{} (\kappa _V^c)^{-1} \end{pmatrix} \right) \cdot (-1_{V_{\mathrm {an}}}). \end{aligned}$$

Put

$$\begin{aligned} x_i&= w_{k-1} \cdots w_{i+1} w_i,&y_j&= w_j w_{j+1} \cdots w_{r-1} w_r w_{r-1} \cdots w_{j+1} w_j, \\ \tilde{x}_i&= \tilde{w}_{k-1} \cdots \tilde{w}_{i+1} \tilde{w}_i,&\tilde{y}_j&= \tilde{w}_j \tilde{w}_{j+1} \cdots \tilde{w}_{r-1} \tilde{w}_r \tilde{w}_{r-1} \cdots \tilde{w}_{j+1} \tilde{w}_j \end{aligned}$$

for $1 \le i \le k-1$ and $1 \le j \le k$. Let $w_T$ be the representative of w in the Weyl group for T which preserves the set of roots of T in $B \cap M_P$. Then $w_T$ has a reduced expression

$$\begin{aligned} w_T = y_k x_1 y_k x_2 \cdots y_k x_{k-1} y_k \end{aligned}$$

and hence $\tilde{w}^\mathrm {LS}$ is defined by

$$\begin{aligned} \tilde{w}^\mathrm {LS}= \tilde{y}_k \tilde{x}_1 \tilde{y}_k \tilde{x}_2 \cdots \tilde{y}_k \tilde{x}_{k-1} \tilde{y}_k. \end{aligned}$$

If we put $\tilde{x}_i' = \tilde{w}_{k-1}^{-1} \cdots \tilde{w}_{i+1}^{-1} \tilde{w}_i^{-1}$, then we have $\tilde{y}_k \tilde{x}_i = \tilde{x}'_i \tilde{y}_i$, so that

$$\begin{aligned} \tilde{w}^\mathrm {LS}= \tilde{x}'_1 \tilde{y}_1 \tilde{x}'_2 \tilde{y}_2 \cdots \tilde{x}'_{k-1} \tilde{y}_{k-1} \tilde{y}_k. \end{aligned}$$

On the other hand, we have

$$\begin{aligned} \tilde{x}'_i = m_P \! \begin{pmatrix} 1_{i-1} &{} &{} \\ &{} &{} -1_{k-i} \\ &{} 1 &{} \end{pmatrix} \end{aligned}$$

and

$$\begin{aligned} \tilde{y}_j = \iota '_j \! \left( \begin{pmatrix} &{} 1 \\ \varepsilon &{} \end{pmatrix} \begin{pmatrix} \kappa _V &{} \\ &{} (\kappa _V^c)^{-1} \end{pmatrix} \right) \cdot m_P \! \begin{pmatrix} 1_{j-1} &{} &{} \\ &{} (-1)^{r-j} &{} \\ &{} &{} - 1_{k-j} \end{pmatrix} \cdot (-1_{V_0}). \end{aligned}$$

In particular, $\tilde{x}_i'$ commutes with $\tilde{y}_j$ if $i > j$, so that

$$\begin{aligned} \tilde{w}^\mathrm {LS}= \tilde{x}'_1 \tilde{x}'_2 \cdots \tilde{x}'_{k-1} \tilde{y}_1 \tilde{y}_2 \cdots \tilde{y}_{k-1} \tilde{y}_k. \end{aligned}$$

Since $\tilde{x}'_1 \cdots \tilde{x}'_{k-1} = m_P(J)$ and

$$\begin{aligned} \tilde{y}_1 \cdots \tilde{y}_k&= \prod _{j=1}^k \iota '_j \! \left( \begin{pmatrix} &{} 1 \\ \varepsilon &{} \end{pmatrix} \begin{pmatrix} \kappa _V &{} \\ &{} (\kappa _V^c)^{-1} \end{pmatrix} \right) \cdot m_{P}((-1)^{r-1} \cdot 1_k) \cdot (-1_{V_0})^k \\&= \prod _{j=1}^k \iota '_j \! \begin{pmatrix} &{} 1 \\ \varepsilon &{} \end{pmatrix} \cdot m_{P}((-1)^{r-1} \cdot \kappa _V \cdot 1_k) \cdot (-1_{V_0})^k, \end{aligned}$$

the assertion follows. $\square$

Next, we consider the case $\epsilon (V) = -1$. Let $V^+$ be the m-dimensional $\varepsilon$-Hermitian space with $\epsilon (V^+) = +1$. We may assume that $V^+ = X \oplus V_0^+ \oplus X^*$ for some $m_0$-dimensional $\varepsilon$-Hermitian space $V_0^+$ with $\epsilon (V_0^+) = +1$. Let $P^+$ be the maximal parabolic subgroup of $\mathrm {U}(V^+)$ stabilizing X and $M_{P^+}$ its Levi component stabilizing $X^*$, so that $M_{P^+} \cong \mathrm {GL}(X) \times \mathrm {U}(V_0^+)$. Fix an isomorphism $V_0^+ \otimes _F \bar{F} \cong V_0 \otimes _F \bar{F}$ as $\varepsilon$-Hermitian spaces over $E \otimes _F \bar{F}$ and extend it to an isomorphism $V^+ \otimes _F \bar{F} \cong V \otimes _F \bar{F}$ whose restriction to $(X \otimes _F \bar{F}) \oplus (X^*\otimes _F \bar{F})$ is the identity map. This induces a pure inner twist $(\xi , z)$, i.e. $\xi : \mathrm {U}(V^+) \rightarrow \mathrm {U}(V)$ is an inner twist and $z \in Z^1(\varGamma , \mathrm {U}(V^+))$ is a 1-cocyle such that $\xi ^{-1} \circ \sigma \circ \xi \circ \sigma ^{-1} = \mathrm {Ad}(z(\sigma ))$ for all $\sigma \in \varGamma$. Then $P^+ = \xi ^{-1}(P)$ and $\xi$ induces an inner twist $\xi : M_{P^+} \rightarrow M_P$ whose restriction to $\mathrm {GL}(X)$ is the identity map. Moreover, z satisfies the assumption in [33, Sect. 2.4.1]. Let $w^+$ be the nontrivial element in the relative Weyl group for $M_{P^+}$ and $\tilde{w}^+ \in \mathrm {U}(V^+)$ the representative of $w^+$ as above. Then the representative of w defined in [33, Sect. 2.3] is $\xi (\tilde{w}^+)$, which is equal to $\tilde{w}$.

We use the normalizing factor $r(w, \tau _s \otimes \sigma _0)$ defined as follows. Let $\lambda (E/F, \psi )$ be the Langlands $\lambda$-factor (see [14, Sect. 5]) and put

$$\begin{aligned} \lambda (w, \psi ) = {\left\{ \begin{array}{ll} \lambda (E/F, \psi )^{(k-1)k/2} &{} {\text { if } m \text { is even;}} \\ \lambda (E/F, \psi )^{(k+1)k/2} &{} {\text { if } m \text { is odd.}} \end{array}\right. } \end{aligned}$$

Let $\phi _\tau$ and $\phi _0$ be the L-parameters of $\tau$ and $\sigma _0$ respectively. Let $\mathrm {As}^+$ be the Asai representation of the L-group of ${\text {Res}}_{E/F} \mathrm {GL}_k$ and $\mathrm {As}^- = \mathrm {As}^+ \otimes \omega _{E/F}$ its twist (see [15, Sect. 7]). If we set

$$\begin{aligned} r(w, \tau _s \otimes \sigma _0) = \lambda (w, \psi ) \cdot \gamma (s, \phi _\tau \otimes \phi _0^\vee , \psi _E)^{-1} \cdot \gamma (2s, \mathrm {As}^{(-1)^m} \circ \phi _\tau , \psi )^{-1}, \end{aligned}$$

then by [33, Lemmas 2.2.3 and 2.3.1], the normalized intertwining operator

$$\begin{aligned} \mathcal {R}(w, \tau _s \otimes \sigma _0) := |\kappa _V|^{k \rho _P} \cdot r(w,\tau _s \otimes \sigma _0)^{-1} \cdot \mathcal {M}(\tilde{w}, \tau _s \otimes \sigma _0) \end{aligned}$$

is holomorphic at $s=0$ and satisfies

$$\begin{aligned} \mathcal {R}(w, w(\tau _s \otimes \sigma _0)) \circ \mathcal {R}(w, \tau _s \otimes \sigma _0) = 1. \end{aligned}$$

Here, the factor $|\kappa _V|^{k \rho _P}$ arises because the Haar measure on $U_P$ defined in [33, Sect. 2.2] with respect to ${{\varvec{spl}}}$ is equal to $|\kappa _V|^{k \rho _P} \, du$.

Now assume that $w(\tau \otimes \sigma _0) \cong \tau \otimes \sigma _0$, which is equivalent to $(\tau ^c)^\vee \cong \tau$. We may take the unique isomorphism

$$\begin{aligned} \mathcal {A}_w : \mathscr {V}_{\tau } \otimes \mathscr {V}_{\sigma _0} \longrightarrow \mathscr {V}_{\tau } \otimes \mathscr {V}_{\sigma _0} \end{aligned}$$

such that:

• $\mathcal {A}_w \circ (w(\tau \otimes \sigma _0))(m) = (\tau \otimes \sigma _0)(m) \circ \mathcal {A}_w$ for all $m \in M_P$;

• $\mathcal {A}_w = \mathcal {A}'_w \otimes 1_{\mathscr {V}_{\sigma _0}}$ with an isomorphism $\mathcal {A}'_w : \mathscr {V}_\tau \rightarrow \mathscr {V}_\tau$ such that $\varLambda \circ \mathcal {A}'_w = \varLambda$. Here, $\varLambda : \mathscr {V}_\tau \rightarrow \mathbb {C}$ is the unique (up to a scalar) Whittaker functional with respect to the Whittaker datum $(N_k, \psi _{N_k})$, where $N_k$ is the group of unipotent upper triangular matrices in $\mathrm {GL}_k(E)$ and $\psi _{N_k}$ is the generic character of $N_k$ given by $\psi _{N_k}(x) = \psi _E(x_{1,2} + \cdots + x_{k-1,k})$.

Note that $\mathcal {A}_w^2 = 1_{\mathscr {V}_{\tau } \otimes \mathscr {V}_{\sigma _0}}$. We define a self-intertwining operator

$$\begin{aligned} R(w, \tau \otimes \sigma _0) : {\text {Ind}}^{\mathrm {U}(V)}_P(\tau \otimes \sigma _0) \longrightarrow {\text {Ind}}^{\mathrm {U}(V)}_P(\tau \otimes \sigma _0) \end{aligned}$$

by

$$\begin{aligned} R(w,\tau \otimes \sigma _0) \varPhi (h) = \mathcal {A}_w(\mathcal {R}(w,\tau \otimes \sigma _0) \varPhi (h)). \end{aligned}$$

By construction,

$$\begin{aligned} R(w,\tau \otimes \sigma _0)^2 = 1. \end{aligned}$$

7.4 Weil representations

In this subsection, we recall some explicit formulas for the Weil representations.

Let $\mathbb {W}$ be a finite dimensional vector space over F equipped with a nondegenerate symplectic form $\langle \cdot , \cdot \rangle _\mathbb {W}: \mathbb {W}\times \mathbb {W}\rightarrow F$. Let $\mathscr {H}(\mathbb {W}) = \mathbb {W}\oplus F$ be the associated Heisenberg group, i.e. the multiplication law is given by

$$\begin{aligned} (w, t) \cdot (w', t') = \left( w+w', t+t' + \frac{1}{2} \langle w, w' \rangle _\mathbb {W}\right) \end{aligned}$$

for $w, w' \in \mathbb {W}$ and $t, t' \in F$. Fix maximal totally isotropic subspaces $\mathbb {X}$ and $\mathbb {X}^*$ of $\mathbb {W}$ such that $\mathbb {W}= \mathbb {X}\oplus \mathbb {X}^*$. Let $\rho$ be the Heisenberg representation of $\mathscr {H}(\mathbb {W})$ on $\mathscr {S}(\mathbb {X}^*)$ with central character $\psi$. Namely,

$$\begin{aligned} \rho ((x+x', t)) \varphi (x'_0) = \psi \Big (t + \langle x'_0, x \rangle _\mathbb {W}+ \tfrac{1}{2} \langle x', x \rangle _\mathbb {W}\Big ) \varphi (x'_0+x') \end{aligned}$$

for $\varphi \in \mathscr {S}(\mathbb {X}^*)$, $x \in \mathbb {X}$, $x', x'_0 \in \mathbb {X}^*$, and $t \in F$.

In Sect. 4.1, we have introduced the Weil representations for unitary groups. To define these representations, we have fixed the additive character $\psi$ and the pair of characters $(\chi _V, \chi _W)$. For simplicity, we write:

• $\omega$ for the Weil representation $\omega _{\psi , \chi _V, \chi _W, V, W}$ of $\mathrm {U}(V) \times \mathrm {U}(W)$ on a space $\mathscr {S}$;

• $\omega _0$ for the Weil representation $\omega _{\psi , \chi _V, \chi _W, V, W_0}$ of $\mathrm {U}(V) \times \mathrm {U}(W_0)$ on a space $\mathscr {S}_0$;

• $\omega _{00}$ for the Weil representation $\omega _{\psi , \chi _V, \chi _W, V_0, W_0}$ of $\mathrm {U}(V_0) \times \mathrm {U}(W_0)$ on a space $\mathscr {S}_{00}$.

We take a mixed model

$$\begin{aligned} \mathscr {S}= \mathscr {S}(V \otimes Y^*) \otimes \mathscr {S}_0 \end{aligned}$$

of $\omega$, where we regard $\mathscr {S}$ as a space of functions on $V \otimes Y^*$ with values in $\mathscr {S}_0$. Similarly, we take a mixed model

$$\begin{aligned} \mathscr {S}_0 = \mathscr {S}(X^* \otimes W_0) \otimes \mathscr {S}_{00} \end{aligned}$$

of $\omega _0$, where we regard $\mathscr {S}_0$ as a space of functions on $X^* \otimes W_0$ with values in $\mathscr {S}_{00}$. Also, we write:

• $\rho _0$ for the Heisenberg representation of $\mathscr {H}(V \otimes W_0)$ on $\mathscr {S}_0$ with central character $\psi$;

• $\rho _{00}$ for the Heisenberg representation of $\mathscr {H}(V_0 \otimes W_0)$ on $\mathscr {S}_{00}$ with central character $\psi$.

Using [37, Theorem 3.1], we can derive the following formulas for the Weil representations $\omega$ and $\omega _0$. Put $\varDelta = \delta ^2 \in F^{\times }$. As in [49, Appendix], let $\gamma _F(\psi )$ be the Weil index of the character $x \mapsto \psi (x^2)$ of second degree and set

$$\begin{aligned} \gamma _F(a,\psi ) = \frac{\gamma _F(\psi _a)}{\gamma _F(\psi )} \end{aligned}$$

for $a \in F^{\times }$, where $\psi _a(x) = \psi (ax)$. Note that $\gamma _F(\varDelta , \psi ) = \lambda (E/F, \psi )^{-1}$. For $\varphi \in \mathscr {S}$ and $x \in V \otimes Y^*$, we have

where

Also, for $\varphi _0 \in \mathscr {S}_0$ and $x \in X^* \otimes W_0$, we have

where

$$\begin{aligned} \gamma _W \!=\! {\left\{ \begin{array}{ll} \chi _W(\delta )^{-1} \cdot \omega _{E/F}(\delta ^{-n} \cdot \det W) \cdot \gamma _F(-\varDelta , \psi )^n \cdot \gamma _F(-1,\psi )^{-n} &{} {\text { if } \varepsilon \!=\! +1;} \\ \omega _{E/F}(\det W) \cdot \gamma _F(\!-\!\varDelta , \psi )^n \cdot \gamma _F(-1,\psi )^{-n} &{} {\text { if } \varepsilon = -1,} \end{array}\right. } \end{aligned}$$

and

$$\begin{aligned} (\rho _0((y+y',0)) \varphi _0)(x)&= \psi \Big (\langle x, y \rangle + \tfrac{1}{2}\langle y', y \rangle \Big ) \varphi _0(x + y'),&y&\in X \otimes W_0, \\&y'&\in X^* \otimes W_0, \\ (\rho _0((y_0,0)) \varphi _0)(x),&= \rho _{00}((y_0,0)) \varphi _0(x),&y_0&\in V_0 \otimes W_0. \end{aligned}$$

7.5 Zeta integrals of Godement–Jacquet

In this subsection, we review the theory of local factors for $\mathrm {GL}_k$ developed by Godement–Jacquet [22].

Let $\tau$ be an irreducible smooth representation of $\mathrm {GL}_k(E)$ on a space $\mathscr {V}_{\tau }$ with central character $\omega _{\tau }$. For any character $\chi$ of $E^{\times }$, we realize the representation $\tau \chi := \tau \otimes (\chi \circ \det )$ on $\mathscr {V}_{\tau }$ by setting $(\tau \chi )(a) v := \chi (\det a) \tau (a) v$ for $a \in \mathrm {GL}_k(E)$ and $v \in \mathscr {V}_{\tau }$. Put $\tau _s := \tau |\cdot |^s$ for $s \in \mathbb {C}$. Let $\tau ^c$ be the representation of $\mathrm {GL}_k(E)$ on $\mathscr {V}_{\tau }$ defined by $\tau ^c(a) = \tau (a^c)$. We write

$$\begin{aligned} L(s, \tau ) = L(s, \phi _\tau ) \quad \text { and} \quad \epsilon (s, \tau , \psi _E) = \epsilon (s, \phi _\tau , \psi _E) \end{aligned}$$

for the standard L-factor and $\epsilon$-factor of $\tau$, where $\phi _\tau$ is the k-dimensional representation of $WD _E$ associated to $\tau$ and $\psi _E$ is the nontrivial additive character of E defined by $\psi _E = \psi \circ {\text {Tr}}_{E/F}$. Then the standard $\gamma$-factor of $\tau$ is defined by

$$\begin{aligned} \gamma (s, \tau , \psi _E) = \epsilon (s, \tau , \psi _E) \cdot \frac{L(1-s, \tau ^\vee )}{L(s, \tau )}, \end{aligned}$$

where $\tau ^\vee$ is the contragredient representation of $\tau$.

For $s \in \mathbb {C}$, $\phi \in \mathscr {S}(\mathrm {M}_k(E))$, and a matrix coefficient f of $\tau$, put

$$\begin{aligned} Z(s, \phi , f) = \int _{\mathrm {GL}_k(E)} \phi (a) f(a) |\det a|^s \, da, \end{aligned}$$

where we have fixed a Haar measure da on $\mathrm {GL}_k(E)$. This integral is absolutely convergent for ${\text {Re}}(s) \gg 0$ and admits a meromorphic continuation to $\mathbb {C}$. Moreover,

$$\begin{aligned} \frac{Z\Big (s+\frac{k-1}{2}, \phi , f\Big )}{L(s,\tau )} \end{aligned}$$

is an entire function of s. If $\tau$ is square-integrable, then $Z(s, \phi , f)$ is absolutely convergent for ${\text {Re}}(s) > \frac{k-1}{2}$ by [22, Proposition 1.3].

Let $\hat{\phi } \in \mathscr {S}(\mathrm {M}_k(E))$ be the Fourier transform of $\phi$ defined by

$$\begin{aligned} \hat{\phi }(x) = \int _{\mathrm {M}_k(E)} \phi (y) \psi _E({\text {Tr}}(xy)) \, dy, \end{aligned}$$

where dy is the self-dual Haar measure on ${\mathrm {M}_k(E)}$ with respect to the pairing $(x, y) \mapsto \psi _E({\text {Tr}}(xy))$. Let $\check{f}$ be the matrix coefficient of $\tau ^\vee$ given by $\check{f}(a) = f(a^{-1})$. Then the local functional equation asserts that

$$\begin{aligned} Z\left( -s+\tfrac{k+1}{2}, \hat{\phi }, \check{f}\right) = \gamma (s, \tau , \psi _E) \cdot Z\Big (s+\tfrac{k-1}{2}, \phi , f\Big ). \end{aligned}$$

8 Proof of Theorem 6.1

Now we can begin the proof of Theorem 6.1. This will be proved by an explicit construction of an equivariant map which realizes the theta correspondence. Recall from Sect. 7 that we have fixed $\varepsilon =\pm 1$, an m-dimensional $\varepsilon$-Hermitian space $V=X\oplus V_{0}\oplus X^{*}$, and an n-dimensional ($-\varepsilon$)-Hermitian space $W=Y\oplus W_{0}\oplus Y^{*}$.

8.1 Construction of equivariant maps

Recall that we have identified $\mathrm {GL}(X)$ with $\mathrm {GL}_k(E)$ using the basis $\{ v_1, \ldots , v_k \}$. Similarly, we identify $\mathrm {GL}(Y)$ with $\mathrm {GL}_k(E)$ using the basis $\{ w_1, \ldots , w_k \}$. Thus we can define an isomorphism $i:\mathrm {GL}(Y) \rightarrow \mathrm {GL}(X)$ via these identifications. Put

$$\begin{aligned} e&= v_1 \otimes w_1^* + \cdots + v_k \otimes w_k^* \in X \otimes Y^*, \\ e^*&= v^*_1 \otimes w_1 + \cdots + v_k^* \otimes w_k \in X^* \otimes Y. \end{aligned}$$

Then $i(a)^c e = a^* e$ and $(i(a)^c)^* e^* = a e^*$ for $a \in \mathrm {GL}(Y)$.

For $\varphi \in \mathscr {S}= \mathscr {S}(V \otimes Y^*) \otimes \mathscr {S}_0$, we define functions $\mathfrak {f}(\varphi )$, $\hat{\mathfrak {f}}(\varphi )$ on $\mathrm {U}(W) \times \mathrm {U}(V)$ with values in $\mathscr {S}_0$ by

$$\begin{aligned} \mathfrak {f}(\varphi )(gh)&= (\omega (gh)\varphi ) \! \begin{pmatrix} e \\ 0 \\ 0 \end{pmatrix}, \\ \hat{\mathfrak {f}}(\varphi )(gh)&= \int _{X \otimes Y^*} (\omega (gh)\varphi ) \! \begin{pmatrix} x \\ 0 \\ 0 \end{pmatrix} \! \psi (\varepsilon \langle x, e^* \rangle ) \, dx \end{aligned}$$

for $g \in \mathrm {U}(W)$ and $h \in \mathrm {U}(V)$. Here, we write an element in $V \otimes Y^*$ as a block matrix

$$\begin{aligned} \begin{pmatrix} y_1 \\ y_2 \\ y_3 \end{pmatrix} \end{aligned}$$

with $y_1 \in X \otimes Y^*$, $y_2 \in V_0 \otimes Y^*$, and $y_3 \in X^* \otimes Y^*$. We also define functions $f(\varphi )$, $\hat{f}(\varphi )$ on $\mathrm {U}(W) \times \mathrm {U}(V)$ with values in $\mathscr {S}_{00}$ by

$$\begin{aligned} f(\varphi )(gh)&= {\text {ev}}(\mathfrak {f}(\varphi )(gh)), \\ \hat{f}(\varphi )(gh)&= {\text {ev}}(\hat{\mathfrak {f}}(\varphi )(gh)), \end{aligned}$$

where ${\text {ev}}: \mathscr {S}_0 = \mathscr {S}(X^* \otimes W_0) \otimes \mathscr {S}_{00} \rightarrow \mathscr {S}_{00}$ is the evaluation at $0 \in X^* \otimes W_0$. If $f = f(\varphi )$ or $\hat{f}(\varphi )$, then

Let $\tau$ be an irreducible (unitary) square-integrable representation of $\mathrm {GL}_k(E)$ on a space $\mathscr {V}_{\tau }$. We may regard $\tau$ as a representation of $\mathrm {GL}(X)$ or $\mathrm {GL}(Y)$ via the above identifications. Let $\pi _0$ and $\sigma _0$ be irreducible tempered representations of $\mathrm {U}(W_0)$ and $\mathrm {U}(V_0)$ on spaces $\mathscr {V}_{\pi _0}$ and $\mathscr {V}_{\sigma _0}$ respectively. Fix nonzero invariant nondegenerate bilinear forms $\langle \cdot , \cdot \rangle$ on $\mathscr {V}_{\tau } \times \mathscr {V}_{\tau ^\vee }$, $\mathscr {V}_{\pi _0} \times \mathscr {V}_{\pi _0^\vee }$, and $\mathscr {V}_{\sigma _0} \times \mathscr {V}_{\sigma _0^\vee }$. Let

$$\begin{aligned} \langle \cdot , \cdot \rangle : (\mathscr {V}_{\tau } \otimes \mathscr {V}_{\sigma _0^\vee }) \times \mathscr {V}_{\tau ^\vee } \longrightarrow \mathscr {V}_{\sigma _0^\vee } \end{aligned}$$

be the induced map.

Now assume that

$$\begin{aligned} \sigma _0 =\varTheta _{\psi , V_0, W_0}(\pi _0). \end{aligned}$$

We fix a nonzero $\mathrm {U}(V_0) \times \mathrm {U}(W_0)$-equivariant map

$$\begin{aligned} \mathcal {T}_{00}: \omega _{00} \otimes \sigma _0^\vee \longrightarrow \pi _0. \end{aligned}$$

For $\varphi \in \mathscr {S}$, $\varPhi _s \in {\text {Ind}}^{\mathrm {U}(V)}_P(\tau ^c_s \chi _W^c \otimes \sigma _0^\vee )$, $g \in \mathrm {U}(W)$, $\check{v} \in \mathscr {V}_{\tau ^\vee }$, and $\check{v}_0 \in \mathscr {V}_{\pi _0^\vee }$, put

$$\begin{aligned}&\langle \mathcal {T}_s(\varphi \otimes \varPhi _s)(g), \check{v} \otimes \check{v}_0 \rangle \\&\qquad = L\Big (s-s_0 + \tfrac{1}{2}, \tau \Big )^{-1} \\&\qquad \qquad \times \int _{U_P \mathrm {U}(V_0) \backslash \mathrm {U}(V)} \langle \mathcal {T}_{00}(\hat{f}(\varphi )(gh) \otimes \langle \varPhi _s(h), \check{v} \rangle ), \check{v}_0 \rangle \, dh, \end{aligned}$$

where we have fixed Haar measures on $\mathrm {U}(V)$ and $\mathrm {U}(V_0)$, and set

$$\begin{aligned} s_0 = \frac{m-n}{2} = \frac{m_0-n_0}{2}. \end{aligned}$$

Note that $\langle \varPhi _s(h), \check{v} \rangle \in \mathscr {V}_{\sigma _0^\vee }$.

Lemma 8.1

The integral $\langle \mathcal {T}_s(\varphi \otimes \varPhi _{s})(g), \check{v} \otimes \check{v}_0 \rangle$ is absolutely convergent for ${\text {Re}}(s) > s_0 - \frac{1}{2}$ and admits a holomorphic continuation to $\mathbb {C}$.

Proof

We may assume that $\varphi = \varphi ' \otimes \varphi _0$ and $\varPhi _s(1) = v \otimes v_0$, where $\varphi ' \in \mathscr {S}(V \otimes Y^*)$, $\varphi _0 \in \mathscr {S}_0$, $v \in \mathscr {V}_{\tau }$, and $v_0 \in \mathscr {V}_{\sigma _0^\vee }$. By the Iwasawa decomposition, it suffices to consider the integral

$$\begin{aligned} \int _{\mathrm {GL}(X)} \langle \mathcal {T}_{00}(\hat{f}(\varphi )(m_P(a)) \otimes \langle \varPhi _s(m_P(a)), \check{v} \rangle ), \check{v}_0 \rangle |\det a|^{-2 \rho _P} \, da. \end{aligned}$$

(8.1)

Put

$$\begin{aligned} \phi (y) = \int _{X \otimes Y^*} \varphi ' \! \begin{pmatrix} x \\ 0 \\ 0 \end{pmatrix} \! \psi (\varepsilon \langle x, y \rangle ) \, dx \end{aligned}$$

for $y \in X^* \otimes Y$. Then we have

$$\begin{aligned} \hat{f}(\varphi )(m_P(a)) = \chi _W(\det a) |\det a|^{k + n_0/2} \phi (a^* e^*) \cdot {\text {ev}}(\varphi _0) \end{aligned}$$

for $a \in \mathrm {GL}(X)$. Hence we have

$$\begin{aligned} (8.1)&= \langle \mathcal {T}_{00}({\text {ev}}(\varphi _0) \otimes v_0), \check{v}_0 \rangle \\&\quad \times \int _{\mathrm {GL}(X)} \phi (a^* e^*) \langle \tau (a^c) v, \check{v} \rangle |\det a|^{s-s_0+k/2} \, da. \end{aligned}$$

This completes the proof, in view of Sect. 7.5. $\square$

Thus we obtain a $\mathrm {U}(V) \times \mathrm {U}(W)$-equivariant map

$$\begin{aligned} \mathcal {T}_s:\omega \otimes {\text {Ind}}^{\mathrm {U}(V)}_P(\tau ^c_s \chi _W^c \otimes \sigma _0^\vee ) \longrightarrow {\text {Ind}}^{\mathrm {U}(W)}_Q(\tau _s \chi _V \otimes \pi _0). \end{aligned}$$

Lemma 8.2

If ${\text {Re}}(s) < s_0 + \frac{1}{2}$, then we have

$$\begin{aligned}&\langle \mathcal {T}_s(\varphi \otimes \varPhi _s)(g), \check{v} \otimes \check{v}_0 \rangle \\&\qquad = L\Big (s-s_0+\tfrac{1}{2}, \tau \Big )^{-1} \cdot \gamma \Big (s-s_0+\tfrac{1}{2}, \tau , \psi _E\Big )^{-1} \\&\qquad \quad \times \int _{U_P \mathrm {U}(V_0) \backslash \mathrm {U}(V)} \langle \mathcal {T}_{00}(f(\varphi )(gh) \otimes \langle \varPhi _s(h), \check{v} \rangle ), \check{v}_0 \rangle \, dh. \end{aligned}$$

Proof

We may assume that $\varphi = \varphi ' \otimes \varphi _0$ and $\varPhi _s(1) = v \otimes v_0$, where $\varphi ' \in \mathscr {S}(V \otimes Y^*)$, $\varphi _0 \in \mathscr {S}_0$, $v \in \mathscr {V}_{\tau }$, and $v_0 \in \mathscr {V}_{\sigma _0^\vee }$. Put $f(a) = \langle \tau (a) v, \check{v} \rangle$ for $a \in \mathrm {GL}(X)$. Let $\phi \in \mathscr {S}(X^* \otimes Y)$ be as in the proof of Lemma 8.1. We define its Fourier transform $\hat{\phi } \in \mathscr {S}(X \otimes Y^*)$ by

$$\begin{aligned} \hat{\phi }(x) = \int _{X^* \otimes Y} \phi (y) \psi (-\varepsilon \langle x, y \rangle ) \, dy. \end{aligned}$$

By the Fourier inversion formula, we have

$$\begin{aligned} \hat{\phi }(x) = \varphi ' \! \begin{pmatrix} x \\ 0 \\ 0 \end{pmatrix}. \end{aligned}$$

Hence we have

$$\begin{aligned} f(\varphi )(m_P(a)) = \chi _W(\det a) |\det a|^{n_0/2} \hat{\phi }(a^{-1} e) \cdot {\text {ev}}(\varphi _0) \end{aligned}$$

for $a \in \mathrm {GL}(X)$. If $s_0-\frac{1}{2}< {\text {Re}}(s) < s_0+\frac{1}{2}$, then by the local functional equation of the zeta integrals of Godement–Jacquet (see Sect. 7.5), we have

$$\begin{aligned}&\int _{\mathrm {GL}(X)} \langle \mathcal {T}_{00}(\hat{f}(\varphi )(m_P(a)) \otimes \langle \varPhi _s(m_P(a)), \check{v} \rangle ), \check{v}_0 \rangle |\det a|^{-2 \rho _P} \, da \\&\quad = \langle \mathcal {T}_{00}({\text {ev}}(\varphi _0) \otimes v_0), \check{v}_0 \rangle \cdot \int _{\mathrm {GL}(X)} \phi (a^* e^*) f(a^c) |\det a|^{s-s_0+k/2} \, da \\&= \langle \mathcal {T}_{00}({\text {ev}}(\varphi _0) \otimes v_0), \check{v}_0 \rangle \\&\quad \times \gamma \Big (s-s_0+\tfrac{1}{2}, \tau , \psi _E\Big )^{-1} \cdot \int _{\mathrm {GL}(X)} \hat{\phi }(ae) \check{f}(a^c) |\det a|^{-s+s_0+k/2} \, da \\&= \langle \mathcal {T}_{00}({\text {ev}}(\varphi _0) \otimes v_0), \check{v}_0 \rangle \\&\quad \times \gamma \Big (s-s_0+\tfrac{1}{2}, \tau , \psi _E\Big )^{-1} \cdot \int _{\mathrm {GL}(X)} \hat{\phi }(a^{-1} e) f(a^c) |\det a|^{s-s_0-k/2} \, da \\&= \gamma \Big (s-s_0+\tfrac{1}{2}, \tau , \psi _E\Big )^{-1} \\&\quad \times \int _{\mathrm {GL}(X)} \langle \mathcal {T}_{00}(f(\varphi )(m_P(a)) \otimes \langle \varPhi _s(m_P(a)), \check{v} \rangle ), \check{v}_0 \rangle |\det a|^{-2\rho _P} \, da. \end{aligned}$$

This completes the proof. $\square$

Lemma 8.3

Assume that $m \ge n$. Let $\varPhi \in {\text {Ind}}^{\mathrm {U}(V)}_P(\tau ^c \chi _W^c \otimes \sigma _0^\vee )$. If $\varPhi \ne 0$, then there exists $\varphi \in \mathscr {S}$ such that

$$\begin{aligned} \mathcal {T}_0(\varphi \otimes \varPhi ) \ne 0. \end{aligned}$$

Proof

Fix a special maximal compact subgroup K of $\mathrm {U}(V)$. We extend $\varPhi$ to a holomorphic section $\varPhi _s$ of ${\text {Ind}}^{\mathrm {U}(V)}_P(\tau _s^c \chi _W^c \otimes \sigma _0^\vee )$ so that $\varPhi _s|_K$ is independent of s. We have

$$\begin{aligned} L\Big (s-s_0+\tfrac{1}{2}, \tau \Big )^{-1} \cdot \gamma \Big (s-s_0+\tfrac{1}{2}, \tau , \psi _E\Big )^{-1} = L\Big (-s+s_0+\tfrac{1}{2}, \tau ^\vee \Big )^{-1} \end{aligned}$$

up to an invertible function. Since $\tau$ is square-integrable and $s_0 \ge 0$, the right-hand side is holomorphic and nonzero at $s=0$. By Lemma 8.2, it suffices to show that there exist $\varphi \in \mathscr {S}$, $\check{v} \in \mathscr {V}_{\tau ^\vee }$, and $\check{v}_0 \in \mathscr {V}_{\pi _0^\vee }$ such that

$$\begin{aligned} \int _{U_P \mathrm {U}(V_0) \backslash \mathrm {U}(V)} \langle \mathcal {T}_{00}(f(\varphi )(h) \otimes \langle \varPhi _s(h), \check{v} \rangle ), \check{v}_0 \rangle \, dh \end{aligned}$$

(8.2)

is nonzero and independent of s for ${\text {Re}}(s) \ll 0$.

Let $\varphi = \varphi ' \otimes \varphi _0$, where $\varphi ' \in \mathscr {S}(V \otimes Y^*)$ and $\varphi _0 \in \mathscr {S}_0$. Then we have

$$\begin{aligned} (8.2) = \int _{U_P \mathrm {U}(V_0) \backslash \mathrm {U}(V)} \varphi '(h^{-1} x_0) \varPsi _s(h) \, dh, \end{aligned}$$

where

$$\begin{aligned} x_0 = \begin{pmatrix} e \\ 0 \\ 0 \end{pmatrix}, \quad \varPsi _s(h) = \langle \mathcal {T}_{00}({\text {ev}}(\omega _0(h) \varphi _0) \otimes \langle \varPhi _s(h), \check{v} \rangle ), \check{v}_0 \rangle . \end{aligned}$$

We can choose $\varphi _0$, $\check{v}$, and $\check{v}_0$ so that $\varPsi _s|_K$ is nonzero and independent of s. Since $h \mapsto h^{-1} x_0$ induces a homeomorphism

$$\begin{aligned} U_P \mathrm {U}(V_0) \backslash \mathrm {U}(V) \overset{\sim }{\longrightarrow } \mathrm {U}(V) x_0 \end{aligned}$$

and $\mathrm {U}(V) x_0$ is locally closed in $V \otimes Y^*$, there exists $\varphi '$ such that

$$\begin{aligned} {\text {supp}}\varphi ' \cap \mathrm {U}(V) x_0 = K x_0 \end{aligned}$$

and such that $\varphi '(k^{-1} x_0) = \overline{\varPsi _s(k)}$ for all $k \in K$. Hence we have

$$\begin{aligned} (8.2)= & {} \int _{U_P \mathrm {U}(V_0) \backslash U_P \mathrm {U}(V_0) K} \varphi '(h^{-1} x_0) \varPsi _s(h) \, dh\\= & {} \int _{(U_P \mathrm {U}(V_0) \cap K) \backslash K} |\varPsi _s(k)|^2 \, dk \ne 0. \end{aligned}$$

Since $\varPsi _s|_K$ is independent of s, so is this integral. This completes the proof. $\square$

8.2 Compatibilities with intertwining operators

Now we shall prove a key property of the equivariant map we have constructed.

Let $w \in W(M_P)$ and $w' \in W(M_Q)$ be the nontrivial elements in the relative Weyl groups. As in Sect. 7.3, we take the representatives $\tilde{w} \in \mathrm {U}(V)$ of w and $\tilde{w}' \in \mathrm {U}(W)$ of $w'$ defined by

$$\begin{aligned} \tilde{w}&= w_P \cdot m_P((-1)^{m'} \cdot \kappa _V \cdot J) \cdot (- 1_{V_0})^k, \\ \tilde{w}'&= w_Q \cdot m_Q((-1)^{n'} \cdot \kappa _W \cdot J) \cdot (- 1_{W_0})^k, \end{aligned}$$

where $m' = [\frac{m}{2}]$ and $n' = [\frac{n}{2}]$. Having fixed $\tau$, $\pi _0$, and $\sigma _0$, we shall write

$$\begin{aligned} \mathcal {M}(\tilde{w},s)&= \mathcal {M}(\tilde{w},\tau ^c_s \chi _W^c \otimes \sigma _0^\vee ), \\ \mathcal {M}(\tilde{w}',s)&= \mathcal {M}(\tilde{w}',\tau _s \chi _V \otimes \pi _0) \end{aligned}$$

for the unnormalized intertwining operators, which are defined by the integrals

$$\begin{aligned} \mathcal {M}(\tilde{w},s) \varPhi _s(h)&= \int _{U_P} \varPhi _s(\tilde{w}^{-1} u h) \, du, \\ \mathcal {M}(\tilde{w}',s) \varPsi _s(g)&= \int _{U_Q} \varPsi _s(\tilde{w}'^{-1} u g) \, du \end{aligned}$$

for $\varPhi _s \in {\text {Ind}}^{\mathrm {U}(V)}_P(\tau ^c_s \chi _W^c \otimes \sigma _0^\vee )$ and $\varPsi _s \in {\text {Ind}}^{\mathrm {U}(W)}_Q(\tau _s \chi _V \otimes \pi _0)$. By the Howe duality, the diagram

commutes up to a scalar. The following proposition determines this constant of proportionality explicitly.

Proposition 8.4

For $\varphi \in \mathscr {S}$ and $\varPhi _s \in {\text {Ind}}^{\mathrm {U}(V)}_P(\tau ^c_s \chi _W^c \otimes \sigma _0^\vee )$, we have

$$\begin{aligned}&\mathcal {M}(\tilde{w}', s) \mathcal {T}_s(\varphi \otimes \varPhi _s) \\&= \left[ \gamma _V^{-1} \cdot \gamma _W \cdot \chi _V((-1)^{n'} \cdot \varepsilon \cdot \kappa _W^{-1}) \cdot \chi _W((-1)^{m'-1} \cdot \kappa _V^{-1}) \cdot (\chi _V^{-n} \chi _W^m)(\delta ) \right] ^k \\&\quad \times \omega _{\tau }((-1)^{m'+n'-1} \cdot \kappa _V^c \kappa _W^{-1}) \cdot |\kappa _V|^{k(s+\rho _P)} \cdot |\kappa _W|^{-k(s+\rho _Q)} \\&\quad \times L\Big (s\!-\!s_0\!+\!\tfrac{1}{2}, \tau \Big )^{-1} \cdot L\Big (-s-s_0\!+\!\tfrac{1}{2}, (\tau ^c)^\vee \Big ) \cdot \gamma \Big (-s-s_0\!+\!\tfrac{1}{2}, (\tau ^c)^\vee , \psi _E\Big ) \\&\quad \times \mathcal {T}_{-s}(\varphi \otimes \mathcal {M}(\tilde{w}, s) \varPhi _s). \end{aligned}$$

Proof

We may assume that ${\text {Re}}(s) \gg 0$. Let $\check{v} \in \mathscr {V}_{\tau ^\vee }$ and $\check{v}_0 \in \mathscr {V}_{\pi _0^\vee }$. Noting that $\det J = 1$, we have by definition

$$\begin{aligned}&\langle \mathcal {M}(\tilde{w}', s) \mathcal {T}_s(\varphi \otimes \varPhi _s)(g), \check{v} \otimes \check{v}_0 \rangle \\&\quad = \omega _\tau ((-1)^{n'} \cdot \kappa _W^{-1}) \cdot \chi _V((-1)^{n'} \cdot \kappa _W^{-1})^k \cdot |\kappa _W|^{-k(s+\rho _Q)} \cdot \omega _{\pi _0}(-1)^k \\&\qquad \times \langle \mathcal {M}(w_Q, s) \mathcal {T}_s(\varphi \otimes \varPhi _s)(g), \tau ^\vee (J) \check{v} \otimes \check{v}_0 \rangle \end{aligned}$$

and

$$\begin{aligned}&\langle \mathcal {T}_{-s}(\varphi \otimes \mathcal {M}(\tilde{w}, s) \varPhi _s)(g), \check{v} \otimes \check{v}_0 \rangle \\&\quad = \omega _\tau ((-1)^{m'} \cdot (\kappa _V^c)^{-1}) \cdot \chi _W((-1)^{m'} \cdot \kappa _V)^k \cdot |\kappa _V|^{-k(s+\rho _P)} \cdot \omega _{\sigma _0}(-1)^k \\&\qquad \times \langle \mathcal {T}_{-s}(\varphi \otimes \mathcal {M}(w_P, s) \varPhi _s)(g), \tau ^\vee (J) \check{v} \otimes \check{v}_0 \rangle , \end{aligned}$$

where $\omega _{\pi _0}$ and $\omega _{\sigma _0}$ are the central characters of $\pi _0$ and $\sigma _0$ respectively. Since $\sigma _0 =\varTheta _{\psi , \chi _V, \chi _W, V_0, W_0}(\pi _0)$, we know that

$$\begin{aligned} \omega _{\sigma _0} = \nu \cdot \omega _{\pi _0}, \end{aligned}$$

where $\nu$ is the character of ${\text {Ker}}({\text {N}}_{E/F})$ defined by

$$\begin{aligned} \nu (x/x^c) = (\chi _V^{-n_0} \chi _W^{m_0})(x) \end{aligned}$$

for $x \in E^{\times }$. In particular, we have

$$\begin{aligned} \omega _{\pi _0}(-1) \cdot \omega _{\sigma _0}(-1)= & {} (\chi _V^{-n_0} \chi _W^{m_0})(\delta )\\= & {} (\chi _V^{-n} \chi _W^{m})(\delta ) \cdot \chi _V(-1)^k \cdot \chi _W(-1)^k. \end{aligned}$$

Thus it suffices to show that

$$\begin{aligned}&L\Big (s-s_0+\tfrac{1}{2}, \tau \Big ) \cdot \mathcal {M}(w_Q, s) \mathcal {T}_s(\varphi \otimes \varPhi _s) \\&= (\chi _V(-\varepsilon ) \cdot \gamma _V^{-1} \cdot \gamma _W)^k \cdot \omega _{\tau }(-1) \\&\quad \times L\Big (-s-s_0+\tfrac{1}{2}, (\tau ^c)^\vee \Big ) \cdot \gamma \Big (-s-s_0 +\tfrac{1}{2}, (\tau ^c)^\vee , \psi _E\Big )\\&\quad \times \mathcal {T}_{-s}(\varphi \otimes \mathcal {M}(w_P, s) \varPhi _s). \end{aligned}$$

We have

$$\begin{aligned}&L\Big (s - s_0 + \tfrac{1}{2}, \tau \Big ) \cdot \langle \mathcal {M}(w_Q, s) \mathcal {T}_s(\varphi \otimes \varPhi _s)(g), \check{v} \otimes \check{v}_0 \rangle \\&= L\Big (s - s_0 + \tfrac{1}{2}, \tau \Big ) \cdot \int _{U_Q} \langle \mathcal {T}_s(\varphi \otimes \varPhi _s)(w_Q^{-1} u g), \check{v} \otimes \check{v}_0 \rangle \, du \\&= \int _{U_Q} \int _{U_P \mathrm {U}(V_0) \backslash \mathrm {U}(V)} \langle \mathcal {T}_{00} (\hat{f}(\varphi )(w_Q^{-1} u g h) \otimes \langle \varPhi _s(h), \check{v} \rangle ), \check{v}_0 \rangle \, dh \, du \\&= \int _{U_P \mathrm {U}(V_0) \backslash \mathrm {U}(V)} \int _{U_Q} \langle \mathcal {T}_{00} (\hat{f}(\varphi )(w_Q^{-1} u g h) \otimes \langle \varPhi _s(h), \check{v} \rangle ), \check{v}_0 \rangle \, du \, dh. \end{aligned}$$

In Lemma 8.6(i) below, we shall show that these integrals are absolutely convergent, so that this manipulation is justified. By Lemma 8.2, we have

$$\begin{aligned}&L\Big (-s-s_0+\tfrac{1}{2}, (\tau ^c)^\vee \Big ) \cdot \gamma \Big (-s-s_{0}+\tfrac{1}{2}, (\tau ^c)^\vee , \psi _E\Big )\\&\quad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \times \langle \mathcal {T}_{-s}(\varphi \otimes \mathcal {M}(w_P, s) \varPhi _s)(g), \check{v} \otimes \check{v}_0 \rangle \\&= \int _{U_P \mathrm {U}(V_0) \backslash \mathrm {U}(V)} \langle \mathcal {T}_{00} (f(\varphi )(gh) \otimes \langle \mathcal {M}(w_P, s) \varPhi _s(h), \check{v} \rangle ), \check{v}_0 \rangle \, dh \\&= \int _{U_P \mathrm {U}(V_0) \backslash \mathrm {U}(V)} \int _{U_P} \langle \mathcal {T}_{00} (f(\varphi )(gh) \otimes \langle \varPhi _s(w_P^{-1} u h), \check{v} \rangle ), \check{v}_0 \rangle \, du \, dh \\&= \int _{\mathrm {U}(V_0) \backslash \mathrm {U}(V)} \langle \mathcal {T}_{00} (f(\varphi )(gh) \otimes \langle \varPhi _s(w_P^{-1} h), \check{v} \rangle ), \check{v}_0 \rangle \, dh \\&= \int _{\mathrm {U}(V_0) \backslash \mathrm {U}(V)} \langle \mathcal {T}_{00} (f(\varphi )(g w_P h) \otimes \langle \varPhi _s(h), \check{v} \rangle ), \check{v}_0 \rangle \, dh \\&= \int _{U_P \mathrm {U}(V_0) \backslash \mathrm {U}(V)} \int _{U_P} \langle \mathcal {T}_{00} (f(\varphi )(g w_P u h) \otimes \langle \varPhi _s(h), \check{v} \rangle ), \check{v}_0 \rangle \, du \, dh \\&= \int _{U_P \mathrm {U}(V_0) \backslash \mathrm {U}(V)} \int _{U_P}\\&\quad \times \langle \mathcal {T}_{00} (f(\varphi )(g w_P u m_P(-1_X) h) \otimes \langle \varPhi _s(m_P(-1_X) h), \check{v} \rangle ), \check{v}_0 \rangle \, du \, dh \\&= \omega _{\tau }(-1) \cdot \chi _W(-1)^k \\&\qquad \times \int _{U_P \mathrm {U}(V_0) \backslash \mathrm {U}(V)} \int _{U_P} \\&\qquad \qquad \qquad \qquad \quad \times \langle \mathcal {T}_{00} (f(\varphi )(g w_P u m_P(-1_X) h) \otimes \langle \varPhi _s(h), \check{v} \rangle ), \check{v}_0 \rangle \, du \, dh. \end{aligned}$$

In Lemma 8.6(ii) below, we shall show that these integrals are absolutely convergent, so that this manipulation is justified. Thus it remains to show that

$$\begin{aligned} \begin{aligned}&\chi _V(-\varepsilon )^k \cdot \gamma _{V}^k \cdot \int _{U_Q} \hat{f}(\varphi )(w_Q^{-1} u)\, du \\&\quad = \chi _W(-1)^k \cdot \gamma _{W}^k \cdot \int _{U_P} f(\varphi )(w_P u m_P(-1_X)) \, du. \end{aligned} \end{aligned}$$

(8.3)

We may assume that $\varphi = \varphi ' \otimes \varphi _0$, where $\varphi ' \in \mathscr {S}(V \otimes Y^*)$ and $\varphi _0 \in \mathscr {S}_0$. We have $w_Q^{-1} = m_Q(-\varepsilon 1_Y) \cdot w_Q$ and

Hence, noting that $I_X^{-1} e = I_Y^{-1} e^* = e^{**}$, we have

$$\begin{aligned}&\chi _V(-\varepsilon )^k \cdot \gamma _V^k \cdot \int _{{\text {Herm}}(Y^*,Y)} \hat{\mathfrak {f}}(\varphi )(w_Q^{-1} u_Q(c)) \, dc \\&\quad = \int _{{\text {Herm}}(Y^*,Y)} \! \left( \! \int _{X \otimes Y^*} \int _{V_0 \otimes Y^*} \varphi \! \begin{pmatrix} y_1 \\ y_2 \\ e^{**} \end{pmatrix} \! \psi \Big (\langle c y_1, e^{**} \rangle + \tfrac{1}{2} \langle cy_2, y_2 \rangle \Big ) \, dy_2 \, dy_1 \! \right) dc. \end{aligned}$$

We change the variables

$$\begin{aligned} y_1&= x_1 e^{**} \in X \otimes Y^*,&x_1&\in {\text {Hom}}(X^*,X), \\ y_2&= x_2 e^{**} \in V_0 \otimes Y^*,&x_2&\in {\text {Hom}}(X^*,V_0). \end{aligned}$$

Then the inner integral is equal to

By Lemma 7.1, the integral over $c \in {\text {Herm}}(Y^*,Y)$ of this integral is equal to

$$\begin{aligned} \int _{{\text {Herm}}(X^*,X)} \int _{{\text {Hom}}(X^*,V_0)} \varphi \! \begin{pmatrix} \Big (c - \tfrac{1}{2} x_2^* x_2\Big ) e^{**} \\ x_2 e^{**} \\ e^{**} \end{pmatrix} dx_2 \, dc. \end{aligned}$$

Hence the left-hand side of (8.3) is equal to

$$\begin{aligned}&\chi _V(-\varepsilon )^k \cdot \gamma _V^k \cdot \int _{{\text {Hom}}(W_0,Y)} \int _{{\text {Herm}}(Y^*,Y)} \hat{f}(\varphi )(w_Q^{-1} u_Q(c) u_Q(b)) \, dc \, db \\&= \int _{{\text {Hom}}(W_0,Y)} \int _{{\text {Herm}}(X^*,X)} \int _{{\text {Hom}}(X^*,V_0)}\\&\quad \qquad \qquad \qquad \qquad \qquad \qquad \times {\text {ev}}\! \Big ( \! (\omega (u_Q(b)) \varphi ) \! \begin{pmatrix} \Big (c - \tfrac{1}{2} x_2^* x_2\Big ) e^{**} \\ x_2 e^{**} \\ e^{**} \end{pmatrix} \! \Big )\ dx_2 \, dc \, db \\&= \int _{{\text {Hom}}(W_0,Y)} \int _{{\text {Herm}}(X^*,X)} \int _{{\text {Hom}}(X^*,V_0)} \varphi ' \! \begin{pmatrix} \Big (c - \tfrac{1}{2} x_2^* x_2\Big ) e^{**} \\ x_2 e^{**} \\ e^{**} \end{pmatrix} \\&\qquad \times \psi \Big ( \tfrac{1}{2} \langle b^* e^{**}, b^* (c - \tfrac{1}{2} x_2^* x_2) e^{**} \rangle \Big ) \rho _{00}((b^* x_2 e^{**}, 0)) \varphi _0(b^* e^{**}) \, dx_2 \, dc \, db \\&= \int _{{\text {Hom}}(W_0,Y)} \int _{{\text {Herm}}(X^*,X)} \int _{{\text {Hom}}(X^*,V_0)} \varphi ' \! \begin{pmatrix} \Big (c - \tfrac{1}{2} x_2^* x_2\Big ) e^{**} \\ x_2 e^{**} \\ e^{**} \end{pmatrix} \\&\quad \qquad \qquad \quad \times \psi \Big (- \tfrac{1}{2} \langle c b^* e^{**}, b^* e^{**} \rangle \Big ) \rho _{00}((x_2 b^* e^{**}, 0)) \varphi _0(b^* e^{**}) \, dx_2 \, dc \, db. \end{aligned}$$

Note that $\langle b^* e^{**}, b^* x_2^* x_2 e^{**} \rangle = \langle b b^* e^{**}, x_2^* x_2 e^{**} \rangle = 0$.

On the other hand, the right-hand side of (8.3) is equal to the product of $\chi _W(-1)^k \cdot \gamma _{W}^k$ and

$$\begin{aligned}&\int _{{\text {Hom}}(V_0, X)} \int _{{\text {Herm}}(X^*,X)} f(\varphi )(w_P u_P(c') u_P(b') m_P(-1_X)) \, dc' \, db' \\&\quad = \int _{{\text {Hom}}(V_0, X)} \int _{{\text {Herm}}(X^*,X)} \varphi ' \! \begin{pmatrix} -\Big (c' + \frac{1}{2}b' b'^*\Big ) e^{**} \\ -b'^* e^{**} \\ e^{**} \end{pmatrix} \\&\qquad \qquad \qquad \qquad \qquad \qquad \times {\text {ev}}(\omega _0(w_P u_P(c') u_P(b') m_P(-1_X)) \varphi _0) \, dc' \, db'. \end{aligned}$$

We have

$$\begin{aligned}&{\text {ev}}(\omega _0(w_P u_P(c') u_P(b') m_P(-1_X)) \varphi _0) \\&\quad = \gamma _{W}^{-k} \cdot \int _{X^* \otimes W_0} (\omega _0(u_P(c') u_P(b') m_P(-1_X)) \varphi _0)(y) \, dy \\&\quad = \gamma _{W}^{-k} \cdot \int _{X^* \otimes W_0} \psi \Big (\tfrac{1}{2} \langle c'y, y \rangle \Big ) (\omega _0(u_P(b') m_P(-1_X)) \varphi _0)(y) \, dy \\&\quad = \gamma _{W}^{-k} \cdot \int _{X^* \otimes W_0} \psi \Big (\tfrac{1}{2} \langle c'y, y \rangle \Big ) \rho _{00}((b'^*y,0)) (\omega _0(m_P(-1_X)) \varphi _0)(y) \, dy \\&\quad = \chi _W(-1)^k \cdot \gamma _{W}^{-k} \cdot \int _{X^* \otimes W_0} \psi \Big (\tfrac{1}{2} \langle c'y, y \rangle \Big ) \rho _{00}((b'^*y,0)) \varphi _0(-y) \, dy. \end{aligned}$$

Changing the variables

$$\begin{aligned} b'&= -x_2^* \in {\text {Hom}}(V_0, X),&x_2&\in {\text {Hom}}(X^*, V_0), \\ c'&= -c \in {\text {Herm}}(X^*,X),&c&\in {\text {Herm}}(X^*,X), \\ y&= - b^* e^{**} \in X^* \otimes W_0,&b&\in {\text {Hom}}(W_0, Y), \end{aligned}$$

we see that the equality (8.3) holds. This completes the proof. $\square$

Let $\phi _{\tau }$, $\phi _0$, and $\phi _0'$ be the L-parameters of $\tau$, $\pi _0$, and $\sigma _0$ respectively. As a consequence of Proposition 8.4, we deduce:

Corollary 8.5

For $\varphi \in \mathscr {S}$ and $\varPhi _s \in {\text {Ind}}^{\mathrm {U}(V)}_P(\tau ^c_s \chi _W^c \otimes \sigma _0^\vee )$, we have

$$\begin{aligned}&\mathcal {R}(w', \tau _s \chi _V \otimes \pi _0) \mathcal {T}_s(\varphi \otimes \varPhi _s)\\&\qquad \quad =\alpha \cdot \beta (s) \cdot \mathcal {T}_{-s}(\varphi \otimes \mathcal {R}(w, \tau _s^c \chi _W^c \otimes \sigma _0^\vee ) \varPhi _s), \end{aligned}$$

where

$$\begin{aligned} \alpha&= \left[ \gamma _V^{-1} \cdot \gamma _W \cdot \chi _V((-1)^{n'} \cdot \varepsilon \cdot \kappa _W^{-1}) \cdot \chi _W((-1)^{m'-1} \cdot \kappa _V^{-1}) \cdot (\chi _V^{-n} \chi _W^m)(\delta )\right] ^k \\&\quad \times \omega _{\tau }((-1)^{m'+n'-1} \cdot \kappa _V^c \kappa _W^{-1}) \cdot \lambda (w, \psi ) \cdot \lambda (w',\psi )^{-1} \end{aligned}$$

and

$$\begin{aligned} \beta (s)&= L\Big (s-s_0+\tfrac{1}{2}, \phi _\tau \Big )^{-1}\cdot L\Big (-s-s_{0}+\tfrac{1}{2}, (\phi _\tau ^c)^\vee \Big )\\&\quad \times \gamma \Big (-s-s_0+\tfrac{1}{2}, (\phi ^c_\tau )^\vee , \psi _E\Big ) \cdot |\kappa _V \kappa _W^{-1}|^{ks} \\&\quad \times \gamma (s, \phi _{\tau }^c \otimes \phi _0' \otimes \chi _W^c, \psi _E)^{-1} \cdot \gamma (s, \phi _{\tau } \otimes \phi _0^\vee \otimes \chi _V, \psi _E). \end{aligned}$$

Proof

The corollary immediately follows from Proposition 8.4 and the following facts:

• $\gamma (s, \mathrm {As}^+ \circ \phi _{\tau ^c}, \psi ) = \gamma (s, \mathrm {As}^+ \circ \phi _\tau , \psi )$;

• for any conjugate self-dual character $\chi$ of $E^{\times }$,

  $$\begin{aligned} \gamma (s, \mathrm {As}^+ \circ \phi _{\tau \chi }, \psi ) = {\left\{ \begin{array}{ll} \gamma (s, \mathrm {As}^+ \circ \phi _\tau , \psi ) &{} {\text { if } \chi |_{F^{\times }} = \mathbbm {1}_{F^{\times }};} \\ \gamma (s, \mathrm {As}^- \circ \phi _\tau , \psi ) &{} {\text { if } \chi |_{F^{\times }} = \omega _{E/F}.} \end{array}\right. } \end{aligned}$$

$\square$

8.3 Convergence of integrals

To finish the proof of Proposition 8.4, it remains to show the following convergence of the integrals.

Lemma 8.6

Let $\varphi \in \mathscr {S}$, $\varPhi _s \in {\text {Ind}}^{\mathrm {U}(V)}_P(\tau ^c_s \chi _W^c \otimes \sigma _0^\vee )$, $\check{v} \in \mathscr {V}_{\tau ^\vee }$, and $\check{v}_0 \in \mathscr {V}_{\pi _0^\vee }$. Assume that ${\text {Re}}(s) \gg 0$.

• (i) The integral

  $$\begin{aligned} \int _{U_Q} \int _{U_P \mathrm {U}(V_0) \backslash \mathrm {U}(V)} \langle \mathcal {T}_{00} (\hat{f}(\varphi )(w_Q^{-1} u h) \otimes \langle \varPhi _s(h), \check{v} \rangle ), \check{v}_0 \rangle \, dh \, du \end{aligned}$$

  (8.4)

  is absolutely convergent.

• (ii) The integral

  $$\begin{aligned} \int _{U_P \mathrm {U}(V_0) \backslash \mathrm {U}(V)} \int _{U_P} \langle \mathcal {T}_{00} (f(\varphi )(h) \otimes \langle \varPhi _s(w_P^{-1} u h), \check{v} \rangle ), \check{v}_0 \rangle \, du \, dh \end{aligned}$$

  (8.5)

  is absolutely convergent.

Proof

Put $t = {\text {Re}}(s) \gg 0$. Fix a special maximal compact subgroup K of $\mathrm {U}(V)$. We may assume that

• $\varphi = \varphi ' \otimes \varphi _0$ for some $\varphi ' \in \mathscr {S}(V \otimes Y^*)$ and $\varphi _0 \in \mathscr {S}_0$;

• $\varPhi _s|_K$ is independent of s;

• $\varPhi _s$ is $K_0$-fixed for some open compact subgroup $K_0$ of K;

• ${\text {supp}}\varPhi _s = P k_0 K_0$ for some $k_0 \in K$;

• $\varPhi _s (k_0)$ is a pure tensor in $\mathscr {V}_{\tau } \otimes \mathscr {V}_{\sigma _0^\vee }$.

In particular, there exist maps $v:K \rightarrow \mathscr {V}_{\tau }$ and $v_0 : K \rightarrow \mathscr {V}_{\sigma _0^\vee }$ such that

$$\begin{aligned} \varPhi _s(k) = v(k) \otimes v_0(k) \end{aligned}$$

for all $k \in K$.

Recall that $\tau$, $\pi _0$, and $\sigma _0$ are tempered and hence unitarizable. We can choose invariant Hilbert space norms $\Vert \cdot \Vert$ on $\mathscr {V}_{\tau }$ and $\mathscr {V}_{\tau ^\vee }$ so that

$$\begin{aligned} | \langle v, \check{v} \rangle | \le \Vert v \Vert \Vert \check{v} \Vert \end{aligned}$$

for all $v \in \mathscr {V}_{\tau }$ and $\check{v} \in \mathscr {V}_{\tau ^\vee }$. Similarly, we choose invariant Hilbert space norms on $\mathscr {V}_{\pi _0}$, $\mathscr {V}_{\sigma _0}$, and so on. We may regard $\mathcal {T}_{00}$ as a $\mathrm {U}(V_0) \times \mathrm {U}(W_0)$-equivariant map $\mathcal {T}_{00}: \mathscr {S}_{00} \rightarrow \mathscr {V}_{\sigma _0} \otimes \mathscr {V}_{\pi _0}$, i.e.

$$\begin{aligned} \langle \mathcal {T}_{00}(\varphi _{00}), v_0 \otimes \check{v}_0 \rangle = \langle \mathcal {T}_{00}(\varphi _{00} \otimes v_0), \check{v}_0 \rangle \end{aligned}$$

for $\varphi _{00} \in \mathscr {S}_{00}$, $v_0 \in \mathscr {V}_{\sigma _0^\vee }$, and $\check{v}_0 \in \mathscr {V}_{\pi _0^\vee }$. Then we have

$$\begin{aligned} \Vert \mathcal {T}_{00}(\omega _{00}(g_0h_0)\varphi _{00}) \Vert = \Vert \mathcal {T}_{00}(\varphi _{00}) \Vert \end{aligned}$$

for $g_0 \in \mathrm {U}(W_0)$ and $h_0 \in \mathrm {U}(V_0)$, and

$$\begin{aligned} | \langle \mathcal {T}_{00}(\varphi _{00} \otimes v_0), \check{v}_0 \rangle | \le \Vert \mathcal {T}_{00}(\varphi _{00}) \Vert \Vert v_0 \Vert \Vert \check{v}_0 \Vert . \end{aligned}$$

Fix $\check{v} \in \mathscr {V}_{\tau ^\vee }$ and $\check{v}_0 \in \mathscr {V}_{\pi _0^\vee }$, and put

$$\begin{aligned} C = \Vert \check{v} \Vert \Vert \check{v}_0 \Vert \max _{k \in K} \Vert v(k) \Vert \Vert v_0(k) \Vert . \end{aligned}$$

Let $\varPsi _t$ be the K-fixed element in ${\text {Ind}}^{\mathrm {U}(V)}_P(|\det |^t \otimes \mathbbm {1}_{\mathrm {U}(V_0)})$ such that $\varPsi _t(1) = 1$. Let $\ell$ denote the representation of $\mathrm {U}(V)$ on $\mathscr {S}(V \otimes Y^*)$ defined by $(\ell (h) \varphi ')(x) = \varphi '(h^{-1} x)$. Recall that ${\text {ev}}: \mathscr {S}_0 \rightarrow \mathscr {S}_{00}$ is the evaluation at 0.

First, we prove the absolute convergence of (8.4). We have

$$\begin{aligned} \hat{f}(\varphi )(h) = \phi (\ell (h) \varphi ')(e^*) \cdot {\text {ev}}(\omega _0(h) \varphi _0), \end{aligned}$$

where $\phi :\mathscr {S}(V \otimes Y^*) \rightarrow \mathscr {S}(X^* \otimes Y)$ is defined by

$$\begin{aligned} \phi (\varphi ')(y) = \int _{X \otimes Y^*} \varphi ' \! \begin{pmatrix} x \\ 0 \\ 0 \end{pmatrix} \! \psi (\varepsilon \langle x, y \rangle ) \, dx. \end{aligned}$$

Put

$$\begin{aligned} \hat{\xi }_s(g,h)&= \langle \mathcal {T}_{00}(\hat{f}(\varphi )(gh) \otimes \langle \varPhi _s(h), \check{v} \rangle ), \check{v}_0 \rangle \\&= \chi _W^c(\det a) |\det a|^{s+\rho _P} \langle \tau ^c(a) v(k), \check{v} \rangle \\&\quad \quad \times \langle \mathcal {T}_{00}(\hat{f}(\varphi )(gh) \otimes \sigma _0^\vee (h_0) v_0(k)), \check{v}_0 \rangle \end{aligned}$$

for $g \in \mathrm {U}(W)$, $h = u m_P(a) h_0 k \in \mathrm {U}(V)$, $u \in U_P$, $a \in \mathrm {GL}(X)$, $h_0 \in \mathrm {U}(V_0)$, and $k \in K$. Then we have

$$\begin{aligned} |\hat{\xi }_s(g,h)|&\le |\det a|^{t+\rho _P} \Vert v(k) \Vert \Vert \check{v}\Vert \cdot \Vert \mathcal {T}_{00}(\hat{f}(\varphi )(gh)) \Vert \Vert v_0(k) \Vert \Vert \check{v}_0 \Vert \\&\le C \cdot \varPsi _t(h) \cdot \Vert \mathcal {T}_{00}(\hat{f}(\varphi )(gh)) \Vert \end{aligned}$$

and

$$\begin{aligned} \Vert \mathcal {T}_{00}(\hat{f}(\varphi )(h)) \Vert&= \Vert \mathcal {T}_{00}(\hat{f}(\varphi )(m_P(a) k)) \Vert \\&= |\det a|^{k+n_0/2} |\phi (\ell (k) \varphi ')(a^* e^*)| \cdot \Vert \mathcal {T}_{00}({\text {ev}}(\omega _0(k) \varphi _0)) \Vert . \end{aligned}$$

Hence we have

$$\begin{aligned}&\int _{U_P \mathrm {U}(V_0) \backslash \mathrm {U}(V)} |\hat{\xi }_s(g,h)| \, dh \\&\quad \le C \cdot \int _{U_P \mathrm {U}(V_0) \backslash \mathrm {U}(V)} \varPsi _t(h) \Vert \mathcal {T}_{00}(\hat{f}(\varphi )(gh)) \Vert \, dh \\&\quad = C \cdot \int _{\mathrm {GL}(X)} \int _K |\det a|^{t-\rho _P} \Vert \mathcal {T}_{00}(\hat{f}(\varphi )(g m_P(a)k)) \Vert \, dk \, da \\&\quad < \infty \end{aligned}$$

since the last integral is the zeta integral of Godement–Jacquet associated to the trivial representation of $\mathrm {GL}(X)$. Put

$$\begin{aligned} \hat{\varXi }_t(g) = C \cdot \int _{U_P \mathrm {U}(V_0) \backslash \mathrm {U}(V)} \varPsi _t(h) \Vert \mathcal {T}_{00}(\hat{f}(\varphi )(gh)) \Vert \, dh. \end{aligned}$$

Then we have

$$\begin{aligned} \hat{\varXi }_t(u m_Q(a) g_0 g) = |\det a|^{t+\rho _Q} \hat{\varXi }_t(g) \end{aligned}$$

for $u \in U_Q$, $a \in \mathrm {GL}(Y)$, $g_0 \in \mathrm {U}(W_0)$, and $g \in \mathrm {U}(W)$, i.e. $\hat{\varXi }_t \in {\text {Ind}}^{\mathrm {U}(W)}_Q(|\det |^t \otimes \mathbbm {1}_{\mathrm {U}(W_0)})$. Hence we have

$$\begin{aligned} \int _{U_Q} \int _{U_P \mathrm {U}(V_0) \backslash \mathrm {U}(V)} |\hat{\xi }_s(w_Q^{-1} u, h)| \, dh \, du \le \int _{U_Q} \hat{\varXi }_t(w_Q^{-1} u) \, du < \infty . \end{aligned}$$

Next, we prove the absolute convergence of (8.5). We have

$$\begin{aligned} f(\varphi )(h) = \hat{\phi }(\ell (h) \varphi ')(e) \cdot {\text {ev}}(\omega _0(h) \varphi _0), \end{aligned}$$

where $\hat{\phi }:\mathscr {S}(V \otimes Y^*) \rightarrow \mathscr {S}(X \otimes Y^*)$ is defined by

$$\begin{aligned} \hat{\phi }(\varphi ')(x) = \varphi ' \! \begin{pmatrix} x \\ 0 \\ 0 \end{pmatrix}. \end{aligned}$$

Put

$$\begin{aligned} \xi _s(h,h')&= \langle \mathcal {T}_{00} (f(\varphi )(h') \otimes \langle \varPhi _s(h), \check{v} \rangle ), \check{v}_0 \rangle \\&= \chi _W^c(\det a) |\det a|^{s+\rho _P} \langle \tau ^c(a) v(k), \check{v} \rangle \\&\qquad \times \langle \mathcal {T}_{00}(f(\varphi )(h') \otimes \sigma _0^\vee (h_0) v_0(k)), \check{v}_0 \rangle \end{aligned}$$

for $h = u m_P(a) h_0 k, h' \in \mathrm {U}(V)$, $u \in U_P$, $a \in \mathrm {GL}(X)$, $h_0 \in \mathrm {U}(V_0)$, and $k \in K$. Then we have

$$\begin{aligned} |\xi _s(h,h')|&\le |\det a|^{t+\rho _P} \Vert v(k) \Vert \Vert \check{v} \Vert \cdot \Vert \mathcal {T}_{00}(f(\varphi )(h')) \Vert \Vert v_0(k) \Vert \Vert \check{v}_0 \Vert \\&\le C \cdot \varPsi _t(h) \cdot \Vert \mathcal {T}_{00}(f(\varphi )(h')) \Vert . \end{aligned}$$

Hence we have

$$\begin{aligned} \int _{U_P} |\xi _s(w_P^{-1} u h, h')| \, du \le C \cdot \Vert \mathcal {T}_{00}(f(\varphi )(h')) \Vert \cdot \int _{U_P} \varPsi _t(w_P^{-1} u h) \, du < \infty . \end{aligned}$$

Put

$$\begin{aligned} \varXi _t(h) = C \cdot \Vert \mathcal {T}_{00}(f(\varphi )(h)) \Vert \cdot \mathcal {M}(w_P,t) \varPsi _t(h), \end{aligned}$$

where

$$\begin{aligned} \mathcal {M}(w_P,t) \varPsi _t(h) = \int _{U_P} \varPsi _t(w_P^{-1} u h) \, du. \end{aligned}$$

Then we have

$$\begin{aligned} \varXi _t(u m_P(a) h_0h)&= C \cdot |\det a|^{-t+\rho _P+n_0/2} |\hat{\phi }(\ell (h) \varphi ')(a^{-1} e)| \\&\quad \times \Vert \mathcal {T}_{00}({\text {ev}}(\omega _0(h) \varphi _0))\Vert \cdot \mathcal {M}(w_P,t) \varPsi _t(h) \end{aligned}$$

for $u \in U_P$, $a \in \mathrm {GL}(X)$, $h_0 \in \mathrm {U}(V_0)$, and $h \in \mathrm {U}(V)$. Hence, putting

$$\begin{aligned} C' = C \cdot \max _{k \in K} \Vert \mathcal {T}_{00}({\text {ev}}(\omega _0(k) \varphi _0)) \Vert \cdot \mathcal {M}(w_P,t) \varPsi _t(1), \end{aligned}$$

we have

$$\begin{aligned}&\int _{U_P \mathrm {U}(V_0) \backslash \mathrm {U}(V)} \int _{U_P} |\xi _s(w_P^{-1} u h, h)| \, du \, dh \\&\quad \le \int _{U_P \mathrm {U}(V_0) \backslash \mathrm {U}(V)} \varXi _t(h) \, dh \\&\quad \le C' \cdot \int _{\mathrm {GL}(X)} \int _K |\det a|^{-t-\rho _P + n_0/2} |\hat{\phi }(\ell (k) \varphi ')(a^{-1} e)| \, dk \, da \\&\quad < \infty \end{aligned}$$

since the last integral is the zeta integral of Godement–Jacquet associated to the trivial representation of $\mathrm {GL}(X)$. $\square$

8.4 Completion of the proof

Now assume that $\varepsilon = +1$ and $m = n+1$. Let $\phi$ be a tempered but non-square-integrable L-parameter for $\mathrm {U}(W_n^\pm )$. Since $\phi$ is not square-integrable, we can write

$$\begin{aligned} \phi = (\phi _\tau \otimes \chi _V) \oplus \phi _0 \oplus ((\phi _\tau \otimes \chi _V)^c)^\vee \end{aligned}$$

for some irreducible (unitary) square-integrable representation $\tau$ of $\mathrm {GL}_k(E)$ and tempered L-parameter $\phi _0$ for $\mathrm {U}(W_{n_0}^\pm )$, where k is a positive integer and $n_0 = n-2k$. Fix $\epsilon ' = \pm 1$, and set $W = W_n^{\epsilon '}$ and $W_0 = W_{n_0}^{\epsilon '}$. Let $\pi = \pi (\eta ) \in \varPi _{\phi }$ be an irreducible tempered representation of $\mathrm {U}(W)$ with associated character $\eta \in {\text {Irr}}(S_{\phi })$. Then $\pi$ is an irreducible constituent of ${\text {Ind}}^{\mathrm {U}(W)}_Q(\tau \chi _V \otimes \pi _0)$ for some irreducible tempered representation $\pi _0 = \pi _0(\eta _0) \in \varPi _{\phi _0}$ of $\mathrm {U}(W_0)$ with associated character $\eta _0 \in {\text {Irr}}(S_{\phi _0})$ such that

$$\begin{aligned} \eta |_{S_{\phi _0}} = \eta _0. \end{aligned}$$

Fix $\epsilon = \pm 1$, and set $V = V_{n+1}^{\epsilon }$ and $V_0 = V_{n_0+1}^{\epsilon }$. Suppose that $\sigma := \varTheta _{\psi , V, W}(\pi ) \ne 0$. By the argument as in [19, pp. 1674–1676], we see that $\sigma _0 := \varTheta _{\psi ,V_0, W_0}(\pi _0) \ne 0$ and $\sigma$ is an irreducible constituent of ${\text {Ind}}^{\mathrm {U}(V)}_P(\tau \chi _W \otimes \sigma _0)$. This implies that $\sigma ^\vee$ is an irreducible constituent of ${\text {Ind}}^{\mathrm {U}(V)}_P(\tau ^c \chi _W^c \otimes \sigma _0^\vee )$. By Theorem 4.4, $\sigma = \sigma (\eta ') \in \varPi _{\phi '}$ and $\sigma _0 = \sigma _0(\eta _0') \in \varPi _{\phi _0'}$ are irreducible tempered representations of $\mathrm {U}(V)$ and $\mathrm {U}(V_0)$ respectively, with L-parameters

$$\begin{aligned} \phi ' = (\phi \otimes \chi _V^{-1} \chi _W) \oplus \chi _W \quad \text { and} \quad \phi '_0 = (\phi _0 \otimes \chi _V^{-1} \chi _W) \oplus \chi _W, \end{aligned}$$

and associated characters $\eta ' \in {\text {Irr}}(S_{\phi '})$ and $\eta _0' \in {\text {Irr}}(S_{\phi _0'})$ such that

$$\begin{aligned} \eta '|_{S_{\phi _0'}} = \eta _0'. \end{aligned}$$

We need to show that $\eta '|_{S_{\phi }} = \eta$.

Consider a commutative diagram

of natural embeddings. Since $n_0 < n$, we know that (P2)$_{n_0}$ holds by assumption, so that

$$\begin{aligned} \eta _0'|_{S_{\phi _0}} = \eta _0. \end{aligned}$$

Hence, we conclude that

$$\begin{aligned} \eta '|_{S_{\phi _0}} = (\eta '|_{S_{\phi _0'}})|_{S_{\phi _0}} = \eta '_0|_{S_{\phi _0}} = \eta _0 = \eta |_{S_{\phi _0}}. \end{aligned}$$

In particular, if $S_{\phi _0} = S_{\phi }$, then $\eta '|_{S_\phi } = \eta$ as desired.

Finally, we assume that $S_{\phi _0} \ne S_{\phi }$, which is the case if and only if $\phi _\tau$ is conjugate orthogonal and $\phi _\tau \otimes \chi _V$ is not contained in $\phi _0$. Then the component group $S_{\phi }$ is of the form

$$\begin{aligned} S_{\phi } = S_{\phi _0} \times (\mathbb {Z}/2 \mathbb {Z}) a_1, \end{aligned}$$

where the extra copy of $\mathbb {Z}/ 2 \mathbb {Z}$ arises from the summand $\phi _\tau \otimes \chi _V$ in $\phi$. Since we already know that $\eta '|_{S_{\phi _0}} = \eta |_{S_{\phi _0}}$, it suffices to show that $\eta '(a_1) = \eta (a_1)$. To see this, we recall the $\mathrm {U}(V) \times \mathrm {U}(W)$-equivariant map

$$\begin{aligned} \mathcal {T}_0 : \omega \otimes {\text {Ind}}^{\mathrm {U}(V)}_P(\tau ^c \chi _W^c \otimes \sigma _0^\vee ) \longrightarrow {\text {Ind}}^{\mathrm {U}(W)}_Q(\tau \chi _V \otimes \pi _0). \end{aligned}$$

Since $\mathcal {T}_0(\varphi \otimes \varPhi ) \in \pi$ for $\varphi \in \mathscr {S}$ and $\varPhi \in \sigma ^\vee$, it follows by (2.1), Lemma 8.3, and Corollary 8.5 that

$$\begin{aligned} \epsilon (W)^k \cdot \eta (a_1) = \alpha \cdot \beta (0) \cdot \epsilon (V)^k \cdot \eta _{\sigma ^\vee }(a_1), \end{aligned}$$

where $\alpha$ and $\beta (s)$ are as in Corollary 8.5, and $\eta _{\sigma ^\vee } \in {\text {Irr}}(S_{(\phi ')^\vee })$ is the irreducible character associated to $\sigma ^\vee$. But we know that

$$\begin{aligned} \eta _{\sigma ^\vee }(a_1) = \eta '(a_1) \times {\left\{ \begin{array}{ll} 1 &{} {\text { if } n \text { is even;}} \\ \omega _{E/F}(-1)^k &{} {\text { if } n \text { is odd.}} \end{array}\right. } \end{aligned}$$

Thus it remains to show that

$$\begin{aligned} \epsilon (V)^k \cdot \epsilon (W)^k \cdot \alpha \cdot \beta (0) = {\left\{ \begin{array}{ll} 1 &{} {\text { if } n \text { is even;}} \\ \omega _{E/F}(-1)^k &{} {\text { if } n \text { is odd.}} \end{array}\right. } \end{aligned}$$

First, we compute $\epsilon (V)^k \cdot \epsilon (W)^k \cdot \alpha$ when n is even. In this case, we see that $\gamma _V = \epsilon (V) \cdot \lambda (E/F, \psi )$ and $\gamma _W = \epsilon (W) \cdot \chi _W(\delta )^{-1}$. Hence $\epsilon (V)^k \cdot \epsilon (W)^k \cdot \alpha$ is equal to

$$\begin{aligned}&\left[ \lambda (E/F, \psi )^{-1} \cdot \chi _W(\delta )^{-1} \cdot \chi _V(-1)^{n'} \cdot \chi _W(-1)^{n'} \cdot (\chi _V^{-n} \chi _W^{n+1})(\delta )\right] ^k \\&\quad \times \omega _{\tau }(-1)^{2n'} \cdot \lambda (E/F, \psi )^{(k+1)k/2} \cdot \lambda (E/F, \psi )^{-(k-1)k/2} \\&= 1. \end{aligned}$$

Next, we compute $\epsilon (V)^k \cdot \epsilon (W)^k \cdot \alpha$ when n is odd. In this case, we see that $\gamma _V = \epsilon (V)$ and $\gamma _W = \epsilon (W) \cdot \chi _W(\delta )^{-1} \cdot \lambda (E/F, \psi )$. Hence $\epsilon (V)^k \cdot \epsilon (W)^k \cdot \alpha$ is equal to

$$\begin{aligned}&\left[ \chi _W(\delta )^{-1} \cdot \lambda (E/F, \psi ) \cdot \chi _V((-1)^{n'-1} \cdot \delta ^{-1}) \cdot \chi _W((-1)^{n'-1} \cdot \delta ^{-1})\cdot (\chi _V^{-n} \chi _W^{n+1})(\delta )\right] ^k \\&\quad \times \omega _{\tau }(-1)^{2n'-1} \cdot \lambda (E/F, \psi )^{(k-1)k/2} \cdot \lambda (E/F, \psi )^{-(k+1)k/2} \\&= \omega _{E/F}(-1)^k \cdot \omega _\tau (-1) \\&= \omega _{E/F}(-1)^k, \end{aligned}$$

where the last equality follows because $\omega _\tau |_{F^{\times }} = \mathbbm {1}_{F^{\times }}$.

Finally, we compute $\beta (0)$. Noting that $s_0 = \frac{1}{2}$, $(\phi _\tau ^c)^\vee = \phi _\tau$, and $\phi '_0 = (\phi _0 \otimes \chi _V^{-1} \chi _W) \oplus \chi _W$, we see that

$$\begin{aligned} \beta (s)&= L(s, \phi _\tau )^{-1} \cdot L(-s, \phi _\tau ) \cdot \gamma (-s, \phi _\tau , \psi _E) \cdot \gamma (s, \phi _{\tau }, \psi _E)^{-1} \\&= \frac{\epsilon (-s, \phi _\tau , \psi _E)}{\epsilon (s, \phi _\tau , \psi _E)} \cdot \frac{L(1+s, \phi _\tau ^\vee )}{L(1-s, \phi _\tau ^\vee )}. \end{aligned}$$

Since $\tau$ is square-integrable, $L(s, \phi _\tau ^\vee )$ is holomorphic and nonzero at $s=1$, and hence

$$\begin{aligned} \beta (0) = 1. \end{aligned}$$

Thus, we have shown the desired formula for $\epsilon (V)^k \cdot \epsilon (W)^k \cdot \alpha \cdot \beta (0)$ and completed the proof of Theorem 6.1.

Remark 8.7

Using Theorem 4.1 (instead of Theorem 4.4) and the above argument, one can also prove the analog of Theorem 6.1 for (P1). Indeed, this can be reduced to the computation of $\epsilon (V)^k \cdot \epsilon (W)^k \cdot \alpha \cdot \beta (0)$ when $\epsilon = +1$, $m=n$, and $\phi _\tau$ is conjugate symplectic, in which case one sees that

$$\begin{aligned} \epsilon (V)^k \cdot \epsilon (W)^k \cdot \alpha = {\left\{ \begin{array}{ll} \omega _{\tau }(\delta ) \cdot \omega _{E/F}(-1)^k &{} {\text { if } n\, \text{ is } \text{ even; }} \\ \omega _{\tau }(\delta ) &{} {\text { if } n \text { is odd,}} \end{array}\right. } \end{aligned}$$

and

$$\begin{aligned} \beta (0) = \epsilon \bigg (\tfrac{1}{2}, \phi _\tau , \psi _E\bigg ) \end{aligned}$$

as desired.

9 Generic case

So far, we have verified the Fourier–Jacobi case (FJ) of the Gross–Prasad conjecture for tempered L-parameters for $\mathrm {U}(W_n) \times \mathrm {U}(W_n)$. As in the proof of [15, Theorem 19.1], this implies (FJ) for tempered L-parameters for $\mathrm {U}(W_n) \times \mathrm {U}(W_{n+2k})$ with $k>0$. In this section, we extend (FJ) to the case of generic L-parameters.

9.1 Generic L-parameters

Let V be an n-dimensional $\varepsilon$-Hermitian space. Recall that an L-parameter $\phi$ for $\mathrm {U}(V)$ is generic if, by definition, its associated L-packet $\varPi _{\phi }$ contains generic representations (i.e. those which possess some Whittaker models). In Proposition B.1 below, we shall show that $\phi$ is generic if and only if its adjoint L-factor $L(s, \mathrm {Ad}\circ \phi ) = L(s, \mathrm {As}^{(-1)^n} \circ \phi )$ is holomorphic at $s=1$.

Let $\phi$ be an L-parameter for $\mathrm {U}(V)$, so that we may write

$$\begin{aligned} \phi = \rho \oplus \phi _0 \oplus (\rho ^c)^{\vee } \quad \text { with} \quad \rho = \bigoplus _{i=1}^r \rho _i |\cdot |^{s_i}, \end{aligned}$$

where

• $\rho _i$ is a $k_i$-dimensional tempered representation of $WD _E$,

• $s_i$ is a real number such that $s_1> \cdots> s_r > 0$,

• $\phi _0$ is a tempered L-parameter for $\mathrm {U}(V_0)$, where $V_0$ is the $\varepsilon$-Hermitian space of dimension $n-2(k_1+\cdots +k_r)$ such that $\epsilon (V_0) = \epsilon (V)$.

As mentioned in Sect. 2.5, by the construction of the local Langlands correspondence, the representations in the Vogan L-packet $\varPi _{\phi }$ are given by the unique irreducible quotient of the standard module

$$\begin{aligned} {\text {Ind}}\biggl ( \biggl ( \bigotimes _{i=1}^r \tau _i |\cdot |^{s_i} \biggr ) \otimes \pi _0 \biggr ) \end{aligned}$$

(9.1)

for $\pi _0 \in \varPi _{\phi _0}$, where ${\text {Ind}}$ is the appropriate parabolic induction and $\tau _i$ is the irreducible tempered representation of $\mathrm {GL}_{k_i}(E)$ associated to $\rho _i$. If $\phi$ is generic, then we have the following result of Heiermann [27], which extends a result of Mœglin–Waldspurger [42, Corollaire 2.14] for special orthogonal groups and symplectic groups.

Proposition 9.1

Let $\phi$ be a generic L-parameter for $\mathrm {U}(V)$. Then the standard modules as in (9.1) are all irreducible, so that the L-packet $\varPi _{\phi }$ consists of standard modules.

9.2 Local theta correspondence

Proposition 9.1 has consequences for the local theta correspondence. Let V be an m-dimensional Hermitian space and W an n-dimensional skew-Hermitian space. Consider the theta correspondence for $\mathrm {U}(V) \times \mathrm {U}(W)$ relative to a pair of characters $(\chi _V, \chi _W)$. Let $\phi$ be an L-parameter for $\mathrm {U}(W)$ and $\pi$ a representation of $\mathrm {U}(W)$ in $\varPi _\phi$. If $m = n$, then by Theorem 4.1, we have $\theta _{\psi ,V,W}(\pi ) \in \varPi _{\theta (\phi )}$ (if nonzero) with

$$\begin{aligned} \theta (\phi ) = \phi \otimes \chi _V^{-1} \chi _W, \end{aligned}$$

so that $L(s, \mathrm {Ad}\circ \theta (\phi )) = L(s, \mathrm {Ad}\circ \phi )$. Thus $\theta (\phi )$ is generic if and only if $\phi$ is. On the other hand, if $m=n+1$, then by Theorem 4.4, we have $\theta _{\psi ,V,W}(\pi ) \in \varPi _{\theta (\phi )}$ (if nonzero) with

$$\begin{aligned} \theta (\phi ) = (\phi \otimes \chi _V^{-1} \chi _W) \oplus \chi _W. \end{aligned}$$

In this case, it is possible that $\theta (\phi )$ is nongeneric even if $\phi$ is. More precisely, since

$$\begin{aligned} L(s, \mathrm {Ad}\circ \theta (\phi )) = L(s, \mathrm {Ad}\circ \phi ) \cdot L(s, \phi \otimes \chi _V^{-1}) \cdot L(s, \omega _{E/F}), \end{aligned}$$

$\theta (\phi )$ is generic if and only if $\phi$ is generic and does not contain $\chi _V |\cdot |^{\pm \frac{k+1}{2}} \boxtimes \mathrm {Sym}^{k-1}$ for any positive integer k, where $\mathrm {Sym}^{k-1}$ is the unique k-dimensional irreducible representation of $\mathrm {SL}_2(\mathbb {C})$. Hence we see that for all but finitely many choices of $\chi _V$ (depending on $\phi$), $\theta (\phi )$ is generic if $\phi$ is.

Proposition 9.2

Let $\phi$ be an L-parameter for $\mathrm {U}(W)$ and $\pi$ a representation of $\mathrm {U}(W)$ in $\varPi _\phi$. Then we have:

• (i) Assume that $m=n$. If $\phi$ is generic (so that $\theta (\phi )$ is also generic), then

  $$\begin{aligned} \varTheta _{\psi ,V,W}(\pi ) = \theta _{\psi ,V,W}(\pi ). \end{aligned}$$

• (ii) Assume that $m=n+1$. If $\phi$ is generic and does not contain $\chi _V |\cdot |^{\pm \frac{k+1}{2}} \boxtimes \mathrm {Sym}^{k-1}$ for any positive integer k (so that $\theta (\phi )$ is also generic), then

  $$\begin{aligned} \varTheta _{\psi ,V,W}(\pi ) = \theta _{\psi ,V,W}(\pi ). \end{aligned}$$

Proof

We shall give the proof of (ii) since the proof of (i) is similar. We may assume that $\varTheta _{\psi ,V,W}(\pi ) \ne 0$. If $\phi$ is tempered, then $\varTheta _{\psi ,V,W}(\pi )$ is irreducible and tempered by [17, Proposition C.4(i)]. In general, by Proposition 9.1, $\pi$ is a standard module of the form

$$\begin{aligned} {\text {Ind}}\biggl ( \biggl (\bigotimes _{i=1}^r \tau _i |\cdot |^{s_i} \biggr ) \otimes \pi _0 \biggr ) \end{aligned}$$

as in (9.1). Then by [17, Proposition C.4(ii)], $\varTheta _{\psi ,V,W}(\pi )$ is a quotient of the standard module

$$\begin{aligned} {\text {Ind}}\biggl ( \biggl ( \bigotimes _{i=1}^r \tau _i \chi _V^{-1} \chi _W |\cdot |^{s_i} \biggr ) \otimes \varTheta _{\psi ,V_0,W_0}(\pi _0) \biggr ). \end{aligned}$$

Since $\theta (\phi )$ is generic as well, Proposition 9.1 implies that this standard module is irreducible, so that $\varTheta _{\psi ,V,W}(\pi )$ is irreducible. $\square$

9.3 (B) for generic L-parameters

For special orthogonal groups, Mœglin–Waldspurger [42] extended the Bessel case (B) of the Gross–Prasad conjecture from tempered L-parameters to generic L-parameters. We carry out the analogous extension for unitary groups.

Proposition 9.3

The statement (B) holds for all generic L-parameters for $\mathrm {U}(V_n) \times \mathrm {U}(V_{n+2k+1})$.

To prove Proposition 9.3, we adapt the proof of Mœglin–Waldspurger [42] to the case of unitary groups. For any (not necessarily irreducible) smooth representations $\pi$ and $\pi '$ of $\mathrm {U}(V_n)$ and $\mathrm {U}(V_{n+2k+1})$ respectively, we write $m(\pi , \pi ')$ or $m(\pi ', \pi )$ for

$$\begin{aligned} \dim _\mathbb {C}{\text {Hom}}_H(\pi \otimes \pi ', \nu ) \end{aligned}$$

with the subgroup H of $\mathrm {U}(V_n) \times \mathrm {U}(V_{n+2k+1})$ and the character $\nu$ of H as in [15, Sect. 12]. Then as explained in [42, Sect. 3], Proposition 9.3 follows from (B) for all tempered L-parameters (which was proved by Beuzart-Plessis [4–6]), together with Proposition 9.1 and the following proposition:

Proposition 9.4

Let $\pi = {\text {Ind}}((\bigotimes _{i=1}^r \tau _i |\cdot |^{s_i}) \otimes \pi _0)$ be a smooth representation of $\mathrm {U}(V_n)$, where

• $\tau _i$ is an irreducible tempered representation of $\mathrm {GL}_{k_i}(E)$,

• $s_i$ is a real number such that $s_1 \ge \cdots \ge s_r \ge 0$,

• $\pi _0$ is an irreducible tempered representation of $\mathrm {U}(V_{n-2(k_1+\dots +k_r)})$.

Likewise, let $\pi ' = {\text {Ind}}((\bigotimes _{j=1}^{r'} \tau '_j |\cdot |^{s'_j}) \otimes \pi _0')$ be a smooth representation of $\mathrm {U}(V_{n+2k+1})$ with analogous data $\tau _j'$, $k_j'$, $s_j'$, $\pi _0'$. Then we have

$$\begin{aligned} m(\pi , \pi ') = m(\pi _0, \pi '_0). \end{aligned}$$

Proof

Since the proof is similar to that of [42, Proposition 1.3], we shall only give a sketch of the proof. First, we prove that $m(\pi , \pi ') \le m(\pi _0, \pi '_0)$.

• (i) Let $\sigma = {\text {Ind}}(\tau _0 |\cdot |^{s_0} \otimes \sigma _0)$ be a smooth representation of $\mathrm {U}(V_{n+1})$, where

  • $\tau _0$ is an irreducible (unitary) square-integrable representation of $\mathrm {GL}_{k_0}(E)$,

  • $s_0$ is a real number,

  • $\sigma _0$ is a smooth representation of $\mathrm {U}(V_{n-2k_0+1})$ of finite length.

  Assume that $s_0 \ge s_1$ (which is interpreted as $s_0 \ge 0$ when $r=0$). Then as in [42, Lemme 1.4], we have

  $$\begin{aligned} m(\pi , \sigma ) \le m(\pi , \sigma _0). \end{aligned}$$

• (ii) Let $\sigma$ be as in (i). Assume that

  • $\tau _0$ is supercuspidal;

  • if a representation $\tau _\sharp \otimes \pi _\sharp$ with

    • an irreducible smooth representation $\tau _\sharp$ of a general linear group;

    • an irreducible smooth representation $\pi _\sharp$ of a general linear group or a unitary group

    intervenes in a Jacquet module of $\tau _i^\vee$, $\tau _i^c$, or $\pi _0^\vee$ as a subquotient, then $\tau _0 |\cdot |^s$ does not intervene in the supercuspidal support of $\tau _\sharp$ for any $s \in \mathbb {R}$.

  Then by [15, Theorem 15.1] (see also [42, Lemme 1.5]), we have

  $$\begin{aligned} m(\pi , \sigma ) = m(\pi , \sigma _0). \end{aligned}$$

• (iii) To prove $m(\pi , \pi ') \le m(\pi _0, \pi '_0)$ in general, we may assume that $\tau _i$, $\tau _j'$ are square-integrable for all i, j. As in [42, Sect. 1.6], we argue by induction on

  $$\begin{aligned} l := \sum _{\begin{array}{c} 1 \le i \le r \\ s_i \ne 0 \end{array}} k_i + \sum _{\begin{array}{c} 1 \le j \le r' \\ s'_j \ne 0 \end{array}} k'_j. \end{aligned}$$

  If $l=0$, then it follows by [6, Sects. 14–15] combined with (ii) that $m(\pi , \pi ') = m(\pi _0, \pi '_0)$. Suppose that $l \ne 0$.

  • (a) If $k=0$ and $s_1' \ge s_1$ (in particular $r'\ge 1$), then by (i), we have $m(\pi , \pi ') \le m(\pi , \pi '')$, where $\pi '' = {\text {Ind}}((\bigotimes _{j=2}^{r'} \tau '_j |\cdot |^{s'_j}) \otimes \pi _0')$. By induction hypothesis, we have $m(\pi , \pi '') \le m(\pi _0, \pi _0')$.

  • (b) If $s_1 \ge s_1'$ (in particular $r \ge 1$), then we can reduce to (a) by using (ii).

  • (c) If $s_1' \ge s_1$ (in particular $r' \ge 1$), then we can reduce to (b) by using (ii).

  This proves the assertion (see [42, Sect. 1.6] for details).

Next, we prove that $m(\pi , \pi ') \ge m(\pi _0, \pi '_0)$. By (ii), we may assume that $k=0$. If $m(\pi _0, \pi _0') = 0$, then there is nothing to prove. If $m(\pi _0, \pi _0') \ne 0$, then by [1], [15, Corollary 15.3], it suffices to show that

$$\begin{aligned} m(\pi , \pi ') \ge 1. \end{aligned}$$

Put

$$\begin{aligned} \pi _z = {\text {Ind}}\bigg (\bigg (\bigotimes _{i=1}^r \tau _i |\cdot |^{z_i}\bigg ) \otimes \pi _0\bigg ) \end{aligned}$$

and

$$\begin{aligned} \pi '_{z'} = {\text {Ind}}\bigg (\bigg (\bigotimes _{j=1}^{r'} \tau '_j |\cdot |^{z'_j}\bigg ) \otimes \pi _0'\bigg ) \end{aligned}$$

for $z = (z_1, \dots , z_r) \in \mathbb {C}^r$ and $z' = (z'_1, \dots , z'_{r'}) \in \mathbb {C}^{r'}$. As in [42, Lemme 1.7], we can define a $\varDelta (\mathrm {U}(V_n) \times \mathrm {U}(V_n))$-equivariant map

$$\begin{aligned} \mathcal {L}_{z,z'} : \pi _z \otimes (\pi _z)^\vee \otimes \pi '_{z'} \otimes (\pi '_{z'})^\vee \longrightarrow \mathbb {C}\end{aligned}$$

by (meromorphic continuation of) an integral of matrix coefficients, which is absolutely convergent for $(z, z')$ near $(\sqrt{-1} \mathbb {R})^r \times (\sqrt{-1} \mathbb {R})^{r'}$. Since $m(\pi _0, \pi _0') \ne 0$, it follows by [6, Théorème 14.3.1, Proposition 15.2.1, Proposition 15.3.1] that the map $(z, z') \mapsto \mathcal {L}_{z, z'}$ is not identically zero. In particular, the leading term of $\mathcal {L}_{z,z'}$ at $z = (s_1, \dots , s_r)$ and $z' = (s_1', \dots , s'_{r'})$ is nonzero and hence $m(\pi , \pi ') \ge 1$ (see [42, Sect. 1.8] for details). This completes the proof. $\square$

9.4 (FJ) for generic L-parameters

In view of Propositions 9.2 and 9.3, one may repeat the see-saw argument in Sect. 5 for generic L-parameters, using (P1) and (P2) (which were shown for all L-parameters) to prove:

Proposition 9.5

The statement (FJ) holds for all generic L-parameters for $\mathrm {U}(W_n) \times \mathrm {U}(W_n)$.

Here, in repeating the see-saw argument, one may choose a character $\chi _V$ so that the condition of Proposition 9.2(ii) holds. Finally, Proposition 9.5 together with [15, Theorem 19.1] implies:

Corollary 9.6

The statement (FJ) holds for all generic L-parameters for $\mathrm {U}(W_n) \times \mathrm {U}(W_{n+2k})$.

Appendices

Appendix A: Addendum to [17]

In this appendix, we elaborate on some results of [17, Appendix C] which are used in the proof of Theorem 4.4. In particular,

• we fill in some missing details in the proof of [17, Proposition C.1(ii)] and streamline its proof by exploiting the recently established Howe duality conjecture [20, 21];

• we extend some results of Muić [45, Lemma 4.2 and Theorem 5.1(i)] (used in the proof of [17, Proposition C.1(ii)]), which were written only for symplectic-orthogonal dual pairs, to cover all dual pairs considered in [17], streamlining some of his proofs in the process.

1.1 A.1 The issues

Let us be more precise. We freely use the notation of [17, Sect. C.1].

Let $\pi$ be an irreducible square-integrable representation of G(W) such that

$$\begin{aligned} \sigma _0 := \varTheta _{\tilde{V},W,\varvec{\chi },\psi }(\pi ) \ne 0. \end{aligned}$$

By the bullet point on [17, p. 645], together with the Howe duality, $\sigma _0$ is irreducible and square-integrable. Then we showed that

• (i) any irreducible subquotient of $\varTheta _{V,W,\varvec{\chi },\psi }(\pi )$ is tempered in the first bullet point on [17, p. 646];

• (ii) $\sigma := \theta _{V,W,\varvec{\chi },\psi }(\pi )$ is an irreducible constituent of $I^{H(V)}_{Q(Y_1)}(\chi _W \otimes \sigma _0)$ in the the second bullet point on [17, p. 646],

and claimed that

• (iii) any irreducible subquotient of $\varTheta _{V,W,\varvec{\chi },\psi }(\pi )$ is not square-integrable in the third bullet point on [17, p. 646];

• (iv) any irreducible subquotient of $\varTheta _{V,W,\varvec{\chi },\psi }(\pi )$ is a subrepresentation of $I^{H(V)}_{Q(Y_1)}(\chi _W \otimes \sigma _0')$ for some irreducible smooth representation $\sigma _0'$ of $H(\tilde{V})$ in the fourth bullet point on [17, p. 646].

However, in the third and fourth bullet points on [17, p. 646], we have used results of Muić [45, Lemma 4.2 and Theorem 5.1(i)], which were written only for symplectic-orthogonal dual pairs. Moreover, we have not given the proof of (iv): we have simply asserted that it is true as if it is obvious (which it is not). Thus, we need to give the details of the proof of (iii) and (iv), as well as that of the results of Muić for all dual pairs considered in [17].

1.2 A.2 Proof of (iii)

First, we address (iii). Our original argument in [17] used [45, Lemma 4.2 and Theorem 5.1(i)], which we state and prove in Lemma A.1 and Corollary A.5 below. Here, we give a more streamlined argument using the recently established Howe duality conjecture [20, 21].

Let $\sigma '$ be an irreducible subquotient of $\varTheta _{V,W,\varvec{\chi },\psi }(\pi )$. Suppose that $\sigma '$ is square-integrable. Since $\varTheta _{V,W,\varvec{\chi },\psi }(\pi )$ is of finite length and tempered by (i), it follows by [60, Corollaire III.7.2] that $\sigma '$ is in fact a quotient of $\varTheta _{V,W,\varvec{\chi },\psi }(\pi )$. Hence we must have $\sigma ' \cong \sigma$ by the Howe duality. But $\sigma$ is not square-integrable by (ii), which is a contradiction. This completes the proof of (iii).

1.3 A.3 Proof of [45, Lemma 4.2]

For the proof of (iv), we will need the following result of Muić [45, Lemma 4.2].

Lemma A.1

(Muić) Let $G(W) \times H(V)$ be an arbitrary reductive dual pair as in [17, Sect. 3]. Let $\pi$ be an irreducible smooth representation of G(W). Then all irreducible subquotients of $\varTheta _{V,W,\varvec{\chi },\psi }(\pi )$ have the same supercuspidal support.

Proof

We may assume that $\varTheta _{V,W,\varvec{\chi },\psi }(\pi ) \ne 0$. Since $\varTheta _{V,W,\varvec{\chi },\psi }(\pi )$ is of finite length, it follows by the theory of the Bernstein center [3] that

$$\begin{aligned} \varTheta _{V,W,\varvec{\chi },\psi }(\pi ) = \sigma _1 \oplus \dots \oplus \sigma _r \end{aligned}$$

for some smooth representations $\sigma _i$ of H(V) of finite length such that

• for each i, all irreducible subquotients of $\sigma _i$ have the same supercuspidal support, say, ${\text {supp}}\sigma _i$;

• if $i \ne j$, then ${\text {supp}}\sigma _i \ne {\text {supp}}\sigma _j$.

Of course, if we were willing to appeal to the Howe duality, then it would follow immediately that $r=1$, so that the lemma is proved. However, we may appeal to an older result of Kudla. Namely, Kudla’s supercuspidal support theorem [36] (see also [17, Proposition 5.2] and the references therein) says that the supercuspidal support of $\theta _{V,W,\varvec{\chi },\psi }(\pi )$ is determined by that of $\pi$. Hence we must have $r=1$. $\square$

1.4 A.4 Plancherel measures

To prove (iv), we will also need the following property of Plancherel measures. We freely use the convention of [17, Appendix B].

Lemma A.2

Let G(W) be an arbitrary classical group as in [17, Sect. 2]. Let $\pi$ be an irreducible tempered representation of G(W) such that

$$\begin{aligned} \pi \subset I^{G(W)}_P(\tau _1 \otimes \dots \otimes \tau _r \otimes \pi _0), \end{aligned}$$

where P is a parabolic subgroup of G(W) with Levi component $\mathrm {GL}_{k_1}(E) \times \dots \times \mathrm {GL}_{k_r}(E) \times G(W_0)$, $\tau _i$ is an irreducible (unitary) square-integrable representation of $\mathrm {GL}_{k_i}(E)$, and $\pi _0$ is an irreducible square-integrable representation of $G(W_0)$. Let $\tau$ be an irreducible (unitary) square-integrable representation of $\mathrm {GL}_k(E)$ and put

$$\begin{aligned} \mathcal {I}(\tau ) = \{ i \, | \, \tau _i \cong \tau \}. \end{aligned}$$

Then we have

$$\begin{aligned} \mathop {\mathrm{ord}}\limits _{s=0} \mu (\tau _s \otimes \pi ) = 2 \cdot \# \mathcal {I}(\tau ) + 2 \cdot \# \mathcal {I}((\tau ^c)^\vee ) + \mathop {\mathrm{ord}}\limits _{s=0} \mu (\tau _s \otimes \pi _0). \end{aligned}$$

Moreover, we have

$$\begin{aligned} \mathop {\mathrm{ord}}\limits _{s=0} \mu (\tau _s \otimes \pi _0) = {\left\{ \begin{array}{ll} 0\text { or } 2 &{}\quad {\text { if } (\tau ^c)^\vee \cong \tau ;} \\ 0 &{} \quad {\text { if } (\tau ^c)^\vee \ncong \tau .} \end{array}\right. } \end{aligned}$$

Proof

By the multiplicativity of Plancherel measures (see [17, Sect. B.5]), we have

$$\begin{aligned} \mu (\tau _s \otimes \pi ) = \left( \prod _{i=1}^r \mu (\tau _s \otimes \tau _i) \cdot \mu (\tau _s \otimes (\tau _i^c)^\vee ) \right) \cdot \mu (\tau _s \otimes \pi _0). \end{aligned}$$

For any irreducible (unitary) square-integrable representation $\tau '$ of $\mathrm {GL}_{k'}(E)$, we have

$$\begin{aligned} \mu (\tau _s \otimes \tau ') = \gamma (s, \tau \times (\tau ')^\vee , \psi _E) \cdot \gamma (-s, \tau ^\vee \times \tau ', \bar{\psi }_E) \end{aligned}$$

and hence

$$\begin{aligned} \mathop {\mathrm{ord}}\limits _{s=0} \mu (\tau _s \otimes \tau ') = {\left\{ \begin{array}{ll} 2 &{} {\text { if } \tau \cong \tau ';} \\ 0 &{} {\text { if } \tau \ncong \tau ',} \end{array}\right. } \end{aligned}$$

which reflects the triviality of R-groups for general linear groups. This proves the first assertion. The second assertion follows from [60, Corollaire IV.1.2] if $(\tau ^c)^\vee \cong \tau$ and [60, Proposition IV.2.2] if $(\tau ^c)^\vee \ncong \tau$. $\square$

1.5 A.5 Proof of (iv)

Now we prove (iv). Let $\sigma '$ be an irreducible subquotient of $\varTheta _{V,W,\varvec{\chi },\psi }(\pi )$. By (i) and (iii), we have

$$\begin{aligned} \sigma ' \subset I^{H(V)}_Q(\tau _1 \otimes \dots \otimes \tau _r \otimes \sigma _0') \end{aligned}$$

for some $r \ge 1$ and irreducible square-integrable representations $\tau _i$ and $\sigma _0'$ of $\mathrm {GL}_{k_i}(E)$ and $H(V_0)$ respectively, where Q is a parabolic subgroup of H(V) with Levi component $\mathrm {GL}_{k_1}(E) \times \dots \times \mathrm {GL}_{k_r}(E) \times H(V_0)$. We need to show that $\tau _i = \chi _W$ for some i.

By Lemma A.1 and the multiplicativity of Plancherel measures, we have

$$\begin{aligned} \mu ((\chi _W)_s \otimes \sigma ') = \mu ((\chi _W)_s \otimes \sigma ). \end{aligned}$$

By (ii) and Lemma A.2, the right-hand side has a zero at $s=0$ of order at least 4. Hence, by Lemma A.2again, we must have $\tau _i = \chi _W$ for some i. This completes the proof of (iv).

Remark A.1

In the proof of (iii) and (iv), we have used some results of Waldspurger [60], which were written only for connected reductive linear algebraic groups. However, it is straightforward to extend them to the cases of (disconnected) orthogonal groups and (nonlinear) metaplectic groups.

1.6 A.6 Proof of [45, Theorem 5.1(i)]

As we noted above, we have used [45, Theorem 5.1(i)] besides [45, Lemma 4.2] in our original argument in [17]. Although it is not necessary for the proof of (iii) and (iv) (because of the use of the Howe duality), we shall give a proof here. In fact, we prove the following more general result by refining the argument in the proof of (iv).

Lemma A.4

Let G(W) be an arbitrary classical group as in [17, Sect. 2]. Let $\pi$ be an irreducible tempered representation of G(W) such that

$$\begin{aligned} \pi \subset I^{G(W)}_P(\tau _1 \otimes \dots \otimes \tau _r \otimes \pi _0), \end{aligned}$$

where P is a parabolic subgroup of G(W) with Levi component $\mathrm {GL}_{k_1}(E) \times \dots \times \mathrm {GL}_{k_r}(E) \times G(W_0)$, $\tau _i$ is an irreducible (unitary) square-integrable representation of $\mathrm {GL}_{k_i}(E)$, and $\pi _0$ is an irreducible square-integrable representation of $G(W_0)$. Likewise, let $\pi '$ be an irreducible tempered representation of G(W) such that

$$\begin{aligned} \pi ' \subset I^{G(W)}_{P'}(\tau '_1 \otimes \dots \otimes \tau '_{r'} \otimes \pi _0') \end{aligned}$$

with analogous data $P'$, $r'$, $\tau '_i$, $\pi '_0$. Assume that

$$\begin{aligned} \mu (\tau _s \otimes \pi ) = \mu (\tau _s \otimes \pi ') \end{aligned}$$

for all irreducible (unitary) square-integrable representations $\tau$ of $\mathrm {GL}_k(E)$ for all $k \ge 1$. Then we have $r=r'$ and

$$\begin{aligned} \{ \tau _1, \dots , \tau _r, (\tau _1^c)^\vee , \dots , (\tau _r^c)^\vee \} = \{ \tau _1', \dots , \tau _r', ((\tau _1')^c)^\vee , \dots , ((\tau _r')^c)^\vee \} \end{aligned}$$

as multi-sets. Moreover, we have

$$\begin{aligned} \mu (\tau _s \otimes \pi _0) = \mu (\tau _s \otimes \pi _0') \end{aligned}$$

for all irreducible (unitary) square-integrable representations $\tau$ of $\mathrm {GL}_k(E)$ for all $k \ge 1$.

Proof

Note that the second assertion is an immediate consequence of the first assertion and the multiplicativity of Plancherel measures. To prove the first assertion, it suffices to show that

$$\begin{aligned} \# \mathcal {I}(\tau ) + \# \mathcal {I}((\tau ^c)^\vee ) = \# \mathcal {I}'(\tau ) + \# \mathcal {I}'((\tau ^c)^\vee ) \end{aligned}$$

(A.1)

for any irreducible (unitary) square-integrable representation $\tau$ of $\mathrm {GL}_k(E)$, where $\mathcal {I}(\tau ) = \{ i \, | \, \tau _i \cong \tau \}$ and $\mathcal {I}'(\tau ) = \{ i \, | \, \tau _i' \cong \tau \}$. If $(\tau ^c)^\vee \cong \tau$, then by Lemma A.2, we have

$$\begin{aligned} 4 \cdot \# \mathcal {I}(\tau ) + \alpha = 4 \cdot \# \mathcal {I}'(\tau ) + \alpha ' \end{aligned}$$

for some $0 \le \alpha , \alpha ' \le 2$. This forces $\# \mathcal {I}(\tau ) = \# \mathcal {I}'(\tau )$, so that (A.1) holds. If $(\tau ^c)^\vee \ncong \tau$, then (A.1) is a direct consequence of Lemma A.2. This completes the proof. $\square$

The following corollary (which is [45, Theorem 5.1(i)]) is now immediate:

Corollary A.5

(Muić) Suppose that $\pi$ and $\pi '$ are irreducible tempered representations of G(W) which have the same supercuspidal support. If $\pi$ is square-integrable, then so is $\pi '$.

Proof

If $\pi$ and $\pi '$ have the same supercuspidal support, then the multiplicativity of Plancherel measures implies that

$$\begin{aligned} \mu (\tau _s \otimes \pi ) = \mu (\tau _s \otimes \pi ') \end{aligned}$$

for all irreducible (unitary) square-integrable representations $\tau$ of $\mathrm {GL}_k(E)$ for all $k \ge 1$. The assertion then follows from Lemma A.4. $\square$

1.7 A.7 Some variant

Finally, admitting the local Langlands correspondence, we shall state a variant of Lemma A.4 in terms of L-parameters.

Let G(W) be an arbitrary classical group as in [17, Sect. 2]. To each irreducible tempered representation $\pi$ of G(W), the local Langlands correspondence assigns an L-parameter $\phi$, which we regard as a semisimple representation of $WD _E$ as described in [15, Sect. 8]. Moreover, for any irreducible tempered representation $\tau$ of $\mathrm {GL}_k(E)$ with associated L-parameter $\phi _\tau$, Langlands’ conjecture on Plancherel measures [38, Appendix II] says that

$$\begin{aligned} \begin{aligned} \mu (\tau _s \otimes \pi )&= \gamma (s, \phi _\tau \otimes \phi ^\vee , \psi _E) \cdot \gamma (-s, \phi _\tau ^\vee \otimes \phi , \bar{\psi }_E)\\&\quad \times \gamma (2s, R \circ \phi _\tau , \psi ) \cdot \gamma (-2s, R \circ \phi _\tau ^\vee , \bar{\psi }), \end{aligned} \end{aligned}$$

(A.2)

where

$$\begin{aligned} R = {\left\{ \begin{array}{ll} \mathrm {Sym}^2 &{} {\text { if } G(W) \text { is odd orthogonal or metaplectic;}} \\ \wedge ^2 &{} {\text { if } G(W) \text { is even orthogonal or symplectic;}} \\ \mathrm {As}^+ &{} {\text { if } G(W) \text { is even unitary;}} \\ \mathrm {As}^- &{} {\text { if } G(W) \text { is odd unitary.}} \end{array}\right. } \end{aligned}$$

In fact, (A.2) immediately follows from [2, Proposition 2.3.1], [44, Proposition 3.3.1], [33, Lemma 2.2.3] (together with induction in stages) for classical groups considered there. (See also §7.3 in the case of unitary groups.) In other words, recalling the definitions of of Plancherel measures and normalized intertwining operators, we see that (A.2) is a consequence of a property of normalized intertwining operators. Also, in the case of metaplectic groups, (A.2) follows from the case of odd orthogonal groups combined with [19, Proposition 10.1].

Lemma A.6

Let $\pi$ and $\pi '$ be irreducible tempered representations of G(W) with associated L-parameters $\phi$ and $\phi '$ respectively. Assume that

$$\begin{aligned} \mu (\tau _s \otimes \pi ) = \mu (\tau _s \otimes \pi ') \end{aligned}$$

for all irreducible (unitary) square-integrable representations $\tau$ of $\mathrm {GL}_k(E)$ for all $k \ge 1$. Then we have

$$\begin{aligned} \phi = \phi '. \end{aligned}$$

Proof

For any irreducible (unitary) square-integrable representation $\tau$ of $\mathrm {GL}_k(E)$ with associated L-parameter $\phi _\tau$, we have

$$\begin{aligned}&\gamma (s, \phi _\tau \otimes \phi ^\vee , \psi _E) \cdot \gamma (-s, \phi _\tau ^\vee \otimes \phi , \bar{\psi }_E)\\&\quad = \gamma (s, \phi _\tau \otimes (\phi ')^\vee , \psi _E) \cdot \gamma (-s, \phi _\tau ^\vee \otimes \phi ', \bar{\psi }_E) \end{aligned}$$

by assumption and (A.2). Comparing the orders of zero at $s=0$, we see that the multiplicities of $\phi _\tau$ in $\phi$ and $\phi '$ are equal (see also [19, Lemma 12.3]). This completes the proof. $\square$

1.8 A.8 Erratum to [17]

On this occasion, we also correct some typos in [17].

• Lemma C.2: ${\text {Isom}}(Y_a', X_a)$ should be read as the set of invertible conjugate linear maps from $Y_a'$ to $X_a$.

• Bottom of p. 650: $\mathrm {Asai}$ should be read as $\mathrm {As}^+$ (resp. $\mathrm {As}^-$) if $G(W^\bullet )$ is even unitary (resp. odd unitary).

Appendix B: Generic L-packets and adjoint L-factors

In this appendix, we prove a conjecture of Gross–Prasad and Rallis [23, Conjecture 2.6] under a certain working hypothesis.

1.1 B.1 Notation

Let G be a connected reductive algebraic group defined and quasi-split over F. Fix a Borel subgroup B of G over F and a maximal torus T in B over F. Let N be the unipotent radical of B, so that $B = T N$. If P is a parabolic subgroup of G over F, we say that P is standard (relative to B) if $P \supset B$. If P is a standard parabolic subgroup of G over F, then we have a Levi decomposition $P = MU$, where M is the unique Levi component of P such that $M \supset T$ and U is the unipotent radical of P. We call M a standard Levi subgroup of G. Let $W^M = {\text {Norm}}_M(T)/T$ be the Weyl group of M and $w^M_0$ the longest element in $W^M$. Put

$$\begin{aligned} \mathfrak {a}_M^* = {\text {Rat}}(M) \otimes _\mathbb {Z}\mathbb {R}, \quad \mathfrak {a}_M = {\text {Hom}}_\mathbb {Z}({\text {Rat}}(M), \mathbb {R}), \end{aligned}$$

where ${\text {Rat}}(M)$ is the group of algebraic characters of M defined over F. We write $\langle \cdot , \cdot \rangle : \mathfrak {a}_M^* \times \mathfrak {a}_M \rightarrow \mathbb {R}$ for the natural pairing. Let $\mathfrak {a}^*_{M,\mathbb {C}} = \mathfrak {a}_M^* \otimes _\mathbb {R}\mathbb {C}$ be the complexification of $\mathfrak {a}_M^*$. Let $A_M$ be the split component of the center of M and $\varSigma (P)$ the set of reduced roots of $A_M$ in P. We may regard $\varSigma (P)$ as a subset of $\mathfrak {a}_M^* \cong {\text {Rat}}(A_M) \otimes _\mathbb {Z}\mathbb {R}$. For $\alpha \in \varSigma (P)$, let $\alpha ^\vee \in \mathfrak {a}_M$ denote its corresponding coroot. Put

$$\begin{aligned} (\mathfrak {a}_M^*)^+ = \{ \lambda \in \mathfrak {a}_M^{*} \, | \, \langle \lambda , \alpha ^\vee \rangle > 0 \quad \text { for all} \ \alpha \in \varSigma (P) \}. \end{aligned}$$

We define a homomorphism $H_M : M \rightarrow \mathfrak {a}_M$ by requiring that

$$\begin{aligned} |\chi (m)|_F = q^{-\langle \chi , H_M(m) \rangle } \end{aligned}$$

for all $\chi \in {\text {Rat}}(M)$ and $m \in M$, where q is the cardinality of the residue field of F.

Let $\pi$ be an irreducible smooth representation of M. For $\lambda \in \mathfrak {a}^*_{M,\mathbb {C}}$, we define a representation $\pi _\lambda$ of M by $\pi _\lambda (m) = q^{- \langle \lambda , H_M(m) \rangle } \pi (m)$. We write

$$\begin{aligned} I^G_P(\pi _\lambda ) := {\text {Ind}}^G_P(\pi _\lambda ) \end{aligned}$$

for the induced representation of G. If $\pi$ is tempered and ${\text {Re}}(\lambda ) \in (\mathfrak {a}^*_M)^+$, then $I^G_P(\pi _\lambda )$ has a unique irreducible quotient $J^G_P(\pi _\lambda )$.

Let $\widehat{M}$ be the dual group of M and ${}^L M = \widehat{M} \rtimes W_F$ the L-group of M. Let $Z(\widehat{M})$ be the center of $\widehat{M}$. We write $\iota _M : {}^L M \hookrightarrow {}^L G$ for the natural embedding. If $\phi : WD _F \rightarrow {}^L M$ is an L-parameter, we say that $\phi$ is tempered if the projection of $\phi (W_F)$ to $\widehat{M}$ is bounded. For $\lambda \in \mathfrak {a}^*_{M,\mathbb {C}}$, we define an L-parameter $\phi _\lambda : WD _F \rightarrow {}^L M$ by $\phi _\lambda = a_\lambda \cdot \phi$, where $a_\lambda \in Z^1(W_F, Z(\widehat{M}))$ is a 1-cocycle which determines the character $m \mapsto q^{- \langle \lambda , H_M(m) \rangle }$ of M.

1.2 B.2 Hypothesis

In this appendix, we admit the local Langlands correspondence for any standard Levi subgroup M of G:

$$\begin{aligned} {\text {Irr}}(M) = \bigsqcup _{\phi } \varPi _{\phi }, \end{aligned}$$

where the disjoint union on the right-hand side runs over all equivalence classes of L-parameters $\phi$ for M and $\varPi _{\phi }$ is a finite set of representations of M, the so-called L-packet. More precisely, we will use the following properties of the local Langlands correspondence:

• (i) $\pi \in \varPi _\phi$ is tempered if and only if $\phi$ is tempered.

• (ii) $\varPi _{\phi _\lambda } = \{ \pi _\lambda \, | \, \pi \in \varPi _\phi \}$ for $\lambda \in \mathfrak {a}_{M,\mathbb {C}}^*$.

• (iii) If $\phi$ is an L-parameter for G, then replacing $\phi$ by its $\widehat{G}$-conjugate if necessary, we can write

  $$\begin{aligned} \phi = \iota _M \circ (\phi _M)_{\lambda _0}, \end{aligned}$$

  where

  • M is a standard Levi subgroup of G,

  • $\phi _M$ is a tempered L-parameter for M,

  • $\lambda _0 \in (\mathfrak {a}_M^*)^+$.

  Then we have

  $$\begin{aligned} \varPi _{\phi } = \{ J^G_P(\pi _{\lambda _0}) \, | \, \pi \in \varPi _{\phi _M} \}, \end{aligned}$$

  where P is the standard parabolic subgroup of G with Levi component M. Note that $\pi \in \varPi _{\phi _M}$ is tempered by (i) and $\pi _{\lambda _0}$ has L-parameter $(\phi _M)_{\lambda _0}$ by (ii).

• (iv) If $\phi$ is a tempered L-parameter for M, then for any generic character $\psi _{N_M}$ of $N_M := N \cap M$, $\varPi _\phi$ contains a $(N_M, \psi _{N_M})$-generic representation $\pi$ of M (see [53, Conjecture 9.4]). Moreover, we have

  $$\begin{aligned} \gamma ^{\mathrm {Sh}}(s, \pi _\lambda , r_M, \psi ) = \gamma (s, r_M \circ \phi _\lambda , \psi ), \end{aligned}$$

  where the left-hand side is Shahidi’s $\gamma$-factor [53] and $r_M$ is the adjoint representation of ${}^L M$ on ${\text {Lie}}({}^L U)$. In fact, we only need the equality up to an invertible function.

The above hypothesis is known to hold for general linear groups by [26, 29, 51] and for classical groups by [2, 44].

1.3 B.3 A conjecture of Gross–Prasad and Rallis

If $\phi$ is an L-parameter for G, we say that $\phi$ is generic if its associated L-packet $\varPi _\phi$ contains a $(N, \psi _N)$-generic representation of G for some generic character $\psi _N$ of N.

Proposition B.1

Let $\phi$ be an L-parameter for G. Then, under the hypothesis in Sect. B.2, $\phi$ is generic if and only if $L(s, \mathrm {Ad}\circ \phi )$ is holomorphic at $s=1$. Here, $\mathrm {Ad}$ is the adjoint representation of ${}^L G$ on its Lie algebra ${\text {Lie}}({}^L G)$.

1.4 B.4 Proof of Proposition B.1

Fix an L-parameter $\phi$ for G and write $\phi = \iota _M \circ (\phi _M)_{\lambda _0}$ as in (iii). Then by (iii), $\phi$ is generic if and only if $J^G_P(\pi _{\lambda _0})$ is $(N, \psi _N)$-generic for some $\pi \in \varPi _{\phi _M}$ and some generic character $\psi _N$ of N, in which case $\pi$ is necessarily $(N_M, \psi _N|_{N_M})$-generic by a result of Rodier [50], [7, Corollary1.7]. Here, we have also used the fact that for any element w in $W^G$, there exists a representative $\tilde{w}$ of w (depending on $\psi _N$) such that $\psi _N$ is compatible with $\tilde{w}$ (see [54, Sect. 2], [12, Sect. 1.2]). Now we invoke the following result of Heiermann–Muić [28, Proposition 1.3].

Lemma B.2

Let $\psi _N$ be a generic character of N and $\pi$ an irreducible tempered $(N_M, \psi _N|_{N_M})$-generic representation of M. Then $J^G_P(\pi _{\lambda _0})$ is $(N, \psi _N)$-generic if and only if $\gamma ^{\mathrm {Sh}}(0, \pi _\lambda , r_M, \psi )$ is holomorphic at $\lambda =\lambda _0$.

Proof

Since the assertion in [28, Proposition 1.3] is slightly different, we include a proof for the convenience of the reader. We realize the representation $I^G_P(\pi _\lambda )$ by using the unique (up to a scalar) Whittaker functional on $\pi$ with respect to $(N_M, \psi _N|_{N_M})$. Then we can define a Whittaker functional

$$\begin{aligned} \varLambda (\pi _\lambda ) : I^G_P(\pi _\lambda ) \longrightarrow \mathbb {C}\end{aligned}$$

with respect to $(N,\psi _N)$ by (holomorphic continuation of) the Jacquet integral (see [52, Proposition 3.1]). By [50], [7, Corollary 1.7], $\varLambda (\pi _\lambda )$ is a basis of ${\text {Hom}}_N(I^G_P(\pi _\lambda ), \psi _N)$ for all $\lambda \in \mathfrak {a}_{M,\mathbb {C}}^*$. Put $w = w^G_0 w^M_0$ and choose its representative $\tilde{w}$ so that $\psi _N$ is compatible with $\tilde{w}$. As in Sect. 7.3, we can define an unnormalized intertwining operator

$$\begin{aligned} \mathcal {M}(\tilde{w}, \pi _\lambda ) : I^G_P(\pi _\lambda ) \longrightarrow I^G_{w(P)}(w(\pi _\lambda )) \end{aligned}$$

by (meromorphic continuation of) an integral which is absolutely convergent for ${\text {Re}}(\lambda ) \in (\mathfrak {a}^*_M)^+$ (see [60, Proposition IV.2.1]), where w(P) is the standard parabolic subgroup of G with Levi component $w M w^{-1}$. Then we have

$$\begin{aligned} \varLambda (\pi _\lambda ) = C(\tilde{w}, \pi _\lambda ) \cdot \varLambda (w(\pi _\lambda )) \circ \mathcal {M}(\tilde{w}, \pi _\lambda ) \end{aligned}$$

(B.1)

for some meromorphic function $C(\tilde{w}, \pi _\lambda )$, the so-called local coefficient. Here, $C(\tilde{w}, \pi _\lambda )$ depends on the choice of Haar measures in the definitions of $\varLambda (\pi _\lambda )$, $\varLambda (w(\pi _\lambda ))$, $\mathcal {M}(\tilde{w}, \pi _\lambda )$, but we ignore the normalization of Haar measures since it does not affect the proof. Since $J^G_P(\pi _{\lambda _0})$ is isomorphic to the image of $\mathcal {M}(\tilde{w}, \pi _{\lambda _0})$ and the functor ${\text {Hom}}_N(\, \cdot \, , \psi _N)$ is exact, $J^G_P(\pi _{\lambda _0})$ is $(N, \psi _N)$-generic if and only if the restriction of $\varLambda (w(\pi _{\lambda _0}))$ to the image of $\mathcal {M}(\tilde{w}, \pi _{\lambda _0})$ is nonzero. By (B.1), this condition is equivalent to the holomorphy of $C(\tilde{w}, \pi _\lambda )$ at $\lambda = \lambda _0$. On the other hand, by the definition of Shahidi’s $\gamma$-factor, we have

$$\begin{aligned} C(\tilde{w}, \pi _\lambda ) = \gamma ^{\mathrm {Sh}}(0, \pi _\lambda , r_M, \psi ) \end{aligned}$$

up to an invertible function. (Note that the convention in [53] is different from ours: the homomorphism $H_M$ is normalized so that $|\chi (m)|_F = q^{\langle \chi , H_M(m) \rangle }$ in [53]. This is why we have $\gamma ^{\mathrm {Sh}}(0, \pi _\lambda , r_M, \psi )$ on the right-hand side rather than $\gamma ^{\mathrm {Sh}}(0, \pi _\lambda , r_M^\vee , \bar{\psi })$.) This completes the proof. $\square$

Now it follows by Lemma B.2 combined with (iv) that $\phi$ is generic if and only if

$$\begin{aligned} \frac{L(1, r_M^\vee \circ (\phi _M)_\lambda )}{L(0, r_M \circ (\phi _M)_\lambda )} \end{aligned}$$

(B.2)

is holomorphic at $\lambda = \lambda _0$. We consider the analytic property of (B.2). For $\alpha \in \varSigma (P)$, let $A_\alpha$ be the identity component of ${\text {Ker}}(\alpha )$, $M_\alpha$ the centralizer of $A_\alpha$ in G, and $U_\alpha$ the root subgroup associated to $\alpha$. Then $M_\alpha$ is a Levi subgroup of G (but not necessarily a Levi component of a standard parabolic subgroup of G) and $M U_\alpha$ is a maximal parabolic subgroup of $M_\alpha$. We may regard $\mathfrak {a}_{M_\alpha }$ as a subspace of $\mathfrak {a}_M$. Put

$$\begin{aligned} (\mathfrak {a}_{M}^{M_\alpha })^* = \{ \lambda \in \mathfrak {a}_M^* \, | \, \langle \lambda , H \rangle = 0 \quad \text { for all} \ H \in \mathfrak {a}_{M_\alpha } \}. \end{aligned}$$

For $\lambda \in \mathfrak {a}_{M,\mathbb {C}}^*$, let $\lambda ^{M_\alpha }$ denote its orthogonal projection to $(\mathfrak {a}_M^{M_\alpha })^* \otimes _\mathbb {R}\mathbb {C}$. We can write

$$\begin{aligned} \lambda ^{M_\alpha } = s_\alpha (\lambda ) \cdot \varpi _\alpha \end{aligned}$$

for some $s_\alpha (\lambda ) \in \mathbb {C}$, where $\varpi _\alpha \in (\mathfrak {a}_{M}^{M_\alpha })^*$ is the unique element such that $\langle \varpi _\alpha , \alpha ^\vee \rangle = 1$. Then we have

$$\begin{aligned} (\mathrm{B}.2) = \prod _{\alpha \in \varSigma (P)} \frac{L(1 - s_\alpha (\lambda ), r_\alpha ^\vee \circ \phi _M)}{L(s_\alpha (\lambda ), r_\alpha \circ \phi _M)}, \end{aligned}$$

where $r_\alpha$ is the adjoint representation of ${}^L M$ on ${\text {Lie}}({}^L U_\alpha )$. Note that $L(s, r_\alpha \circ \phi _M)$ is holomorphic and nonzero for ${\text {Re}}(s) > 0$ since $\phi _M$ is tempered. Since $\lambda _0 \in (\alpha ^*_M)^+, s_{\alpha }(\lambda _0)$ is a positive real number for all $\alpha \in \varSigma (P)$. Hence (B.2) is holomorphic at $\lambda = \lambda _0$ if and only if

$$\begin{aligned} \prod _{\alpha \in \varSigma (P)} L(1 - s_\alpha (\lambda ) - s_{\alpha }(\lambda _{0}), r_\alpha ^\vee \circ \phi _M) \end{aligned}$$

is holomorphic at $\lambda = 0$. Since the L-factors have no zeros, this condition is equivalent to the holomorphy of $L(s - s_{\alpha }(\lambda _{0}), r_\alpha ^\vee \circ \phi _M)$ at $s=1$ for all $\alpha \in \varSigma (P)$, which in turn is equivalent to the holomorphy of

$$\begin{aligned} L(s, r_M^\vee \circ (\phi _M)_{\lambda _{0}}) = \prod _{\alpha \in \varSigma (P)} L(s - s_{\alpha }(\lambda _{0}), r_\alpha ^\vee \circ \phi _M) \end{aligned}$$

at $s=1$. Thus, we have shown that $\phi$ is generic if and only if $L(s, r_M^\vee \circ (\phi _M)_{\lambda _0})$ is holomorphic at $s=1$.

On the other hand, we have

$$\begin{aligned} L(s, \mathrm {Ad}\circ \phi ) = L(s, r_M \circ (\phi _M)_{\lambda _0}) \cdot L(s, \mathrm {Ad}_M \circ (\phi _M)_{\lambda _0}) \cdot L(s, r_M^\vee \circ (\phi _M)_{\lambda _0}), \end{aligned}$$

where $\mathrm {Ad}_M$ is the adjoint representation of ${}^L M$ on ${\text {Lie}}({}^L M)$. Since $\phi _M$ is tempered and $s_{\alpha }(\lambda _{0}) > 0$ for all $\alpha \in \varSigma (P)$,

$$\begin{aligned} L(s, r_M \circ (\phi _M)_{\lambda _0}) = \prod _{\alpha \in \varSigma (P)} L(s + s_{\alpha }(\lambda _{0}), r_\alpha \circ \phi _M) \end{aligned}$$

and $L(s, \mathrm {Ad}_M \circ (\phi _M)_{\lambda _0}) = L(s, \mathrm {Ad}_M \circ \phi _M)$ are holomorphic and nonzero for ${\text {Re}}(s) > 0$. Hence $L(s, \mathrm {Ad}\circ \phi )$ is holomorphic at $s=1$ if and only if $L(s, r_M^\vee \circ (\phi _M)_{\lambda _0})$ is holomorphic at $s=1$. This completes the proof of Proposition B.1.

Remark B.3

If G is a classical group, then one has the following variant of Proposition B.1 which does not rely on the local Langlands correspondence. Fix a generic character $\psi _N$ of N. If $\pi$ is an irreducible $(N,\psi _N)$-generic representation of G, let $\varPi$ be its functorial lift to the general linear group established in [10, 11, 13, 34, 35] (see [11, Definition 7.1] for the precise definition in the case when G is split over F). Put

$$\begin{aligned} L^{\mathrm {Sh}}(s, \pi , {\mathrm {Ad}}) := L^{\mathrm {Sh}}(s, \varPi , R), \end{aligned}$$

where the right-hand side is Shahidi’s L-factor [53] and

$$\begin{aligned} R = {\left\{ \begin{array}{ll} \mathrm {Sym}^2 &{} {\text { if } G \text { is odd special orthogonal;}} \\ \wedge ^2 &{} {\text { if } G \text { is even special orthogonal or symplectic;}} \\ \mathrm {As}^+ &{} {\text { if } G \text { is even unitary;}} \\ \mathrm {As}^- &{} {\text { if } G \text { is odd unitary.}} \end{array}\right. } \end{aligned}$$

If $\pi$ is tempered, then so is $\varPi$ (see [11, Proposition 7.4] when G is split over F and [35, Proposition 8.6] when G is even unitary) and hence $L^{\mathrm {Sh}}(s, \pi , {\mathrm {Ad}})$ is holomorphic and nonzero for ${\text {Re}}(s)>0$ (see [53, Proposition 7.2]). If we admit the local Langlands correspondence, then by [30], we have $L^{\mathrm {Sh}}(s, \pi , {\mathrm {Ad}}) = L(s, {\mathrm {Ad}} \circ \phi )$, where $\phi$ is the L-parameter of $\pi$.

Now let P be a standard parabolic subgroup of G with Levi component M and $\pi$ an irreducible tempered $(N_M, \psi _N|_{N_M})$-generic representation of M. For any $\lambda \in \mathfrak {a}_{M,\mathbb {C}}^*$, one has the L-factor $L^{\mathrm {Sh}}(s, I^G_P(\pi _\lambda ), {\mathrm {Ad}})$ as above since the set of $\lambda$ such that $I^G_P(\pi _\lambda )$ is irreducible and $(N,\psi _N)$-generic is Zariski dense in $\mathfrak {a}_{M,\mathbb {C}}^*$. Then by the above argument (together with the multiplicativity), one can show that for $\lambda _0 \in (\mathfrak {a}_M^*)^+$, $J^G_P(\pi _{\lambda _0})$ is $(N, \psi _N)$-generic if and only if $L^{\mathrm {Sh}}(s, I^G_P(\pi _{\lambda _0}), {\mathrm {Ad}})$ is holomorphic at $s=1$.