Linear periods for unitary representations

Abstract

Let F be a local non-Archimedean field of characteristic zero with a finite residue field. Based on Tadić’s classification of the unitary dual of ${\mathrm {GL}}_{2n}(F)$, we classify irreducible unitary representations of ${\mathrm {GL}}_{2n}(F)$ that have nonzero linear periods, in terms of Speh representations that have nonzero periods. We also give a necessary and sufficient condition for the existence of a nonzero linear period for a Speh representation.

1 Introduction

1.1 Main results

Let F be a local non-Archimedean field of characteristic zero with a finite residue field. Denote the group $G_n = {\mathrm {GL}}_{n}(F)$. Let p and q be two nonnegative integers with $p + q = n$, we denote by $H = H_{p,q}$ the subgroup of $G_n$ of matrices of the form:

$$\begin{aligned} \begin{pmatrix} g_1 &{} 0 \\ 0 &{} g_2 \end{pmatrix} \quad \text {with }g_1 \in G_p,\ g_2 \in G_q. \end{aligned}$$

Let $\pi$ be a smooth representation of $G_n$ on a complex vector space V and a character of H, denote by the space of linear forms l on V such that for all $v \in V$ and $h \in H$. Smooth representations $\pi$ of $G_n$ with are called -distinguished, or simply H-distinguished if is the trivial character ${\textbf {1}}$ of H.

Elements of $\mathrm {Hom}_H(\pi ,{\textbf {1}})$ are called (local) linear periods of $\pi$. Linear periods have been studied by many authors. The uniqueness of linear periods was proved by Jacquet and Rallis in [11]; the uniqueness of twisted linear periods, with respect to almost all characters of H and in the case $p = q$, was proved by Chen and Sun in [3]. It thus remains an interesting question of characterizing irreducible representations that have nonzero linear periods. It is known that a tempered representation of ${\mathrm {GL}}_{2n}(F)$ has nonzero linear periods with respect to $H_{n,n}$ if and only if it is a functorial transfer of a generic tempered representation of ${\mathrm {SO}}_{2n+1}(F)$, see [13, 20, 22]. Another closely related characterization of the existence of nonzero linear periods for an essentially square-integrable representation is through poles of the local exterior square L-functions associated with the representation, see [20] and references therein. A recent preprint by Sécherre [29] studied supercuspidal representations with nonzero linear periods from the point of view of type theory. However, all of these characterizations are for generic representations. Motivated by the recent work of Gan–Gross–Prasad [9] on branching laws in the non-tempered case, we are led to consider in this work the existence of nonzero linear periods for irreducible unitary representations.

Our main results are as follows. We refer the reader to Sect. 2 for unexplained notation in the following two theorems.

Theorem 1.1

Let $\text {Sp}(\delta ,k)$ be a Speh representation of $G_{2n}$, where $\delta$ is a square-integrable representation of $G_d$ with $d > 1$, and k is a positive integer ($2n = dk$). Then $\text {Sp}(\delta ,k)$ is $H_{n,n}$-distinguished if and only if d is even and $\delta$ is $H_{d/2,d/2}$-distinguished.

Theorem 1.2

An irreducible unitary representation $\pi$ of $G_{2n}$ is $H_{n,n}$-distinguished if and only if it is self-dual and its Arthur part $\pi _{Ar }$ is of the form

$$\begin{aligned} (\sigma _1 \times \sigma _1^{\vee }) \times \cdots \times (\sigma _r \times \sigma _r^{\vee }) \times \sigma _{r+1} \times \cdots \times \sigma _s. \end{aligned}$$

where each $\sigma _i$ is a Speh representation for $i = 1, \cdots , s$, and each representation $\sigma _j$ is $H_{m_j,m_j}$-distinguished for some positive integer $m_j$, $j = r+1 , \ldots ,s$.

Distinction problem for unitary representation has already been considered by Matringe for local Galois periods in [21] and by Offen and Sayag for local Symplectic periods in [27, 28]. We remark that the special case of Theorem 1.2 for representations of Arthur type (see Theorem 7.3) is similar to [22, Theorem 3.13] about local linear periods for generic representations and the main result in [21] about local Galois periods for unitary representations. A global analogue of our result is to find the $H_{n,n}$-distinguished representation in the automorphic dual of $G_{2n}$, which we will pursue in future works. We also refer the reader to [7, 11] for the role of local linear periods and their global analogues in the study of standard L-functions.

1.2 Remarks on the method of the proof

Most of our work deals with distinction of parabolically induced representations of $G_n$. The main tool to study distinction of induced representations is the geometric lemma of Bernstein-Zelevinsky [1], which relates distinction of an induced representation to distinction of some Jacquet module of the inducing data. It was shown by Tadić in [30] that every irreducible unitary representation is isomorphic to the parabolic induction of Speh representations or their twists. The observation is that Jacquet modules of Speh representations have convenient combinatorial descriptions similar to those of Jacquet modules of essentially square-integrable representations [16]. As hinted by the geometric lemma, to classify $H_{n,n}$-distinguished irreducible unitary representations, it is necessary to consider $H_{p,q}$-distinction with respect to a particular family of characters in (2.1), not only of Speh representations, but also of a larger class of representations, ladder representations. The class of ladder representations was introduced by Lapid and Mínguez in [17], and has many remarkable properties which make them an ideal testing ground for distinction of non-generic representations and some other questions in the representation theory of general linear groups, see for example [6, 10, 18, 24]. The most complicated part of the paper, Sect. 6, is devoted to the study of distinction of ladder representations. Our treatment is largely combinatorial based on detailed analysis by the geometric lemma. We refer the reader to [19, 22] for a similar approach to the classification of distinguished generic representations in Galois symmetric space and our setting respectively.

We next outline the proof of Theorem 1.1. For the ‘if’ part, the existence of non-zero linear periods for the standard module of a Speh representation $\text {Sp}(\varDelta ,k)$ is guaranteed by the work of Blanc and Delorme [2] when $\varDelta$ is $H_{d/2,d/2}$-distinguished. Thus it suffices to show that the maximal proper subrepresentation of the standard module associated with $\text {Sp}(\varDelta ,k)$ is not $H_{n,n}$-distinguished. The explicit structure of this maximal proper subrepresentation is well known by the work of Tadić [32] (see also [17]). For the ‘only if’ part of Theorem 1.1, however, we cannot expect to get any information on the distinguishedness of $\varDelta$ from that of the standard module of $\text {Sp}(\varDelta ,k)$ when k is an even, as in this case, the standard module of $\text {Sp}(\varDelta ,k)$ is $H_{n,n}$-distinguished for any self-dual $\varDelta$ by the work of Blanc and Delorme [2]. We instead use the idea of ‘restricting to the mirabolic subgroup’, and relate linear periods on $\text {Sp}(\varDelta ,k)$ with those on its highest shifted derivative, which is exactly $\text {Sp}(\varDelta ,k-1)$. The ‘only if’ part is then proved by induction on k. We remark that the idea of exploiting the theory of derivatives in distinction problems has already appeared many times in the literature, see for example [4, 15, 21, 22].

The paper is organized as follows. In Sect. 2 we introduce notations and some preliminaries on the representation theory of general linear groups. In Sect. 3 we present some general facts on $(H_{p,q},\mu _a)$-distinguished representations, where $\mu _a$ is the character in (2.1). In this section, we recall a result of Gan which is crucial for our combinatorial study of twisted linear periods. In Sect. 4 we give a detailed analysis of the parabolic orbits of the symmetric space involved and in Sect. 5 we draw some consequences of the geometric lemma. Section 6 is devoted to the study of distinction of ladder representations. We then complete the classification in Sect. 7.

2 Preliminaries

Throughout the paper let F be a local non-Archimedean field of characteristic zero with a finite residue field.

For any $n \in {\mathbb {Z}}_{\geqslant 0}$, let $G_n = {\mathrm {GL}}_n(F)$ and let ${\mathscr {R}}(G_n)$ be the category of smooth complex representations of $G_n$ of finite length. Denote by $Irr (G_n)$ the set of equivalence classes of irreducible objects of ${\mathscr {R}}(G_n)$ and by ${\mathscr {C}}(G_n)$ the subset consisting of supercuspidal representations. (By convention we define $G_0$ as the trivial group and $Irr (G_0)$ consists of the trivial representation of $G_0$.) Let $Irr$ and ${\mathscr {C}}$ be the disjoint union of $Irr (G_n)$ and ${\mathscr {C}}(G_n)$,$n \geqslant 0$, respectively. For a representation $\pi \in {\mathscr {R}}(G_n)$, we call n the degree of $\pi$.

Let $\mathfrak {R}_n$ be the Grothendieck group of ${\mathscr {R}}(G_n)$ and $\mathfrak {R}= \oplus _{n \geqslant 0} \mathfrak {R}_n$. The canonical map from the objects of ${\mathscr {R}}(G_n)$ to $\mathfrak {R}_n$ will be denoted by $\pi \mapsto [\pi ]$.

Denote by $\nu$ the character $\nu (g) = |\det g |$ on any $G_n$. (The n will be implicit and hopefully clear from the context.) For any $\pi \in {\mathscr {R}}(G_n)$ and $a \in {\mathbb {R}}$, denote by $\nu ^a \pi$ the representation obtained from $\pi$ by twisting it by the character $\nu ^a$, and denote by $\pi ^{\vee }$ the contragredient of $\pi$. The sets $Irr$ and ${\mathscr {C}}$ are invariant under taking contragredient. For a character $\chi$ of $F^{\times }$, define the real part $\Re (\chi )$ of $\chi$ to be the real number a such that $| \chi (z) |_{{\mathbb {C}}} = |z|^a$, $z \in F^{\times }$, where $|\cdot |_{{\mathbb {C}}}$ is the absolute value on ${\mathbb {C}}$. For a subgroup Q of $G_n$, denote by $\delta _Q$ the modular character of Q.

For two nonnegative integers p and q with $p + q = n$, we denote by $w_{p,q}$ the matrix

$$\begin{aligned} w_{p,q} = \begin{pmatrix} 0 &{} I_q \\ I_p &{} 0 \end{pmatrix}. \end{aligned}$$

Let $H_{p,q}$ be the subgroup of $G_n$ as in the introduction. For $a \in {\mathbb {R}}$, define the character $\mu _a$ of $H_{p,q}$ by

$$\begin{aligned} \mu _a \left( \begin{pmatrix} g_1 &{} \\ &{} g_2 \end{pmatrix} \right) = \nu ^a(g_1)\nu ^{-a}(g_2),\quad g_1 \in G_p,\ g_2 \in G_q. \end{aligned}$$

(2.1)

(By convention we allow the case where p or q is zero.)

2.1 Jacquet modules of induced representations

The standard parabolic subgroups of $G_n$ are in bijection with compositions $(n_1, \ldots ,n_t)$ of n. The corresponding standard Levi subgroup is the group of block diagonal invertible matrices with block sizes $n_1, \ldots , n_t$. It is isomorphic to $G_{n_1} \times \cdots \times G_{n_t}$.

Let $P = M < imes U$ be a standard parabolic subgroup of $G_n$ and $\sigma$ a smooth, complex representation of M. We denote by $\mathrm {Ind}_P^{G_n}(\sigma )$ its normalized parabolic induction; for any standard Levi subgroup $L \subset M$, we denote by $r_{L,M}(\sigma )$ the normalized Jacquet module (see [1, Sect. 2.3]).

If $\rho _1, \ldots , \rho _t$ are representations of $G_{n_1}, \ldots , G_{n_t}$ respectively, we denote by

$$\begin{aligned} \rho _1 \times \cdots \times \rho _t \end{aligned}$$

the representation $\mathrm {Ind}_P^{G_n} \sigma$ where $\sigma$ is the representation $\rho _1 \otimes \cdots \otimes \rho _t$ of M, where M is the standard Levi subgroup of the parabolic subgroup P corresponding to $(n_1,\ldots ,n_t)$.

Next we briefly review the Jacquet module of a product of representations of finite length [33, Sect. 1.6] (or more precisely, its composition factors). Let $\alpha = (n_1,\ldots , n_t)$ and $\beta = (m_1,\ldots , m_s)$ be two compositions of n. For every $i \in \{ 1, \ldots , t\}$, let $\rho _i \in \mathscr {R}(G_{n_i})$. Denote by $\text {Mat}^{\alpha , \beta }$ the set of $t \times s$ matrices $B = (b_{i,j})$ with nonnegative integer entries such that

$$\begin{aligned} \sum _{j=1}^s b_{i,j} = n_i,\quad i \in \{ 1,\ldots , t\},\quad \sum _{i=1}^t b_{i,j} = m_j, \quad j \in \{1,\ldots ,s \}. \end{aligned}$$

Fix $B\in \text {Mat}^{\alpha ,\beta }$. For any $i \in \{1,\ldots ,t \}$, $\alpha _i = (b_{i,1},\ldots ,b_{i,s})$ is a composition of $n_i$ and we write the compostion factors of $r_{\alpha _i} (\rho _i)$ as

$$\begin{aligned} \sigma _i^{k} = \sigma ^k_{i,1} \otimes \cdots \otimes \sigma ^k_{i,s}, \quad \sigma ^k_{i,j} \in \text {Irr}(G_{b_{i,j}}), \quad k \in \{ 1, \ldots , l_i\}, \end{aligned}$$

where $l_i$ is the length of $r_{\alpha _i}(\rho _i)$. For any $j \in \{ 1, \ldots , s\}$ and a sequence $\underline{k} = (k_1,\ldots ,k_r)$ of integers such that $1 \leqslant k_i \leqslant l_i$, define

$$\begin{aligned} \Sigma _j^{B,\underline{k}} = \sigma ^{k_1}_{1,j} \times \cdots \times \sigma ^k_{t,j} \in {\mathscr {R}}(G_{m_j}). \end{aligned}$$

Then we have

$$\begin{aligned}{}[r_{\beta }(\rho _1 \times \cdots \times \rho _t] = \sum _{B \in \text {Mat}^{\alpha ,\beta },\underline{k}} [ \Sigma _1^{b,\underline{k}} \otimes \cdots \otimes \Sigma _s^{B,\underline{k}}]. \end{aligned}$$

2.2 Langlands classification

By a segment of cuspidal representations we mean a set

$$\begin{aligned}{}[a,b]_{\rho } = \{\nu ^a \rho , \nu ^{a+1}\rho , \ldots , \nu ^b\rho \}, \end{aligned}$$

where $\rho \in {\mathscr {C}}$ and $a, b \in {\mathbb {R}}$, $b- a \in {\mathbb {Z}}_{\geqslant 0}$. The representation $\nu ^a\rho \times \nu ^{a+1}\rho \times \cdots \times \nu ^b\rho$ has a unique irreducible quotient, which is an essentially square-integrable representaton and is denoted by $\varDelta ([a,b]_{\rho })$. The map $[a,b]_{\rho } \mapsto \varDelta ([a,b]_{\rho })$ gives a bijection between the set of segments of cuspidal representations and the subset of essentially square-integrable representations in $Irr$. (In what follows, for simplicity of notation, we shall use $\varDelta$ to denote either a segment of cuspidal representations or the essentially square-integrable representations corresponding to it; we hope this will not cause any confusion.) We use the convention that $\varDelta ([a,b]_{\rho }) = 0$ if $b < a-1$ and $\varDelta ([a,a-1]_{\rho }) = 1$, the trivial represntation of $G_0$.

We denote the extremities of $\varDelta = \varDelta ([a,b]_{\rho })$ by ${\textbf {b}}(\varDelta ) = \nu ^a\rho \in {\mathscr {C}}$ and ${\textbf {e}}(\varDelta ) = \nu ^b\rho \in {\mathscr {C}}$ respectively. We also write $l(\varDelta ) = b -a + 1$ for the length of $\varDelta$.

For $\rho \in {\mathscr {C}}$, we denote by ${\mathbb {Z}}\rho$ the set $\{\nu ^a \rho \ |\ a \in {\mathbb {Z}}\}$ and call it the cuspidal line of $\rho$. We then transport the order and additive structure of ${\mathbb {Z}}$ to the cuspidal line ${\mathbb {Z}}\rho$. Thus we shall sometimes write $\nu ^a\rho + b = \nu ^{a+b} \rho$ and $\nu ^a \rho \leqslant \nu ^b \rho$ if $a \leqslant b$, where a, b are integers. By the contragredient of ${\mathbb {Z}}\rho$ we mean the cuspidal line ${\mathbb {Z}}\rho ^{\vee }$.

Let $\varDelta$ and $\varDelta '$ be two segments. We say that $\varDelta$ and $\varDelta '$ are linked if $\varDelta \cup \varDelta '$ forms a segment but neither $\varDelta \subset \varDelta '$ nor $\varDelta ' \subset \varDelta$. If $\varDelta$ and $\varDelta '$ are linked and ${\textbf {b}}(\varDelta ) = {\textbf {b}}(\varDelta ')\nu ^j$ with $j < 0$, then we say that $\varDelta$ precedes $\varDelta '$ and write $\varDelta \prec \varDelta '$.

A multisegment is a multiset (that is, set with multiplicities) of segments. Denote by ${\mathscr {O}}$ the set of multisegements. For $\rho \in {\mathscr {C}}$, let ${\mathscr {O}}_{\rho }$ denote the multisegements such that all of its segements are contained in the cuspidal line ${\mathbb {Z}}\rho$. An order $\mathfrak {m}= \{\varDelta _1,\ldots , \varDelta _t \} \in {\mathscr {O}}$ on a multisegments $\mathfrak {m}$ is of standard form if $\varDelta _i \nprec \varDelta _j$ for all $i < j$. Every $\mathfrak {m}\in {\mathscr {O}}$ admits at least one standard order.

Let $\mathfrak {m}= \{\varDelta _1, \ldots , \varDelta _t \} \in {\mathscr {O}}$ be ordered in standard form. The representation

$$\begin{aligned} \lambda (\mathfrak {m}) = \varDelta _1 \times \cdots \times \varDelta _t \end{aligned}$$

is independent of the choice of order of standard form. It has a unique irreducible quotient that we denote by $L (\mathfrak {m})$. The Langlands classification says that the map $\mathfrak {m}\mapsto L (\mathfrak {m})$ is a bijection between ${\mathscr {O}}$ and $Irr$.

2.3 Unitary dual of $G_n$

We briefly recall the classification of the unitary dual of $G_n$ by Tadić [30, Theorem D]. Let $\text {Irr}^u$ be the subset of unitarizable representations in $\text {Irr}$, and ${\mathscr {D}}^u$ the subset of all square-integrable classes in $Irr ^u$. Let k be a positive integer, and let $\delta \in {\mathscr {D}}^u$. The repersentation

$$\begin{aligned} \nu ^{(k-1)/2}\delta \times \nu ^{(k-3)/2 } \delta \times \cdots \times \nu ^{-(k-1)/2}\delta \end{aligned}$$

has a unique irreducible unitarizable quotient $\text {Sp}(\delta ,k)$, called a Speh representation.

Suppose $0< \alpha < 1/2$. The representation $\nu ^{\alpha } \text {Sp}(\delta ,k) \times \nu ^{-\alpha } \text {Sp}(\delta ,k)$ is irreducible and unitarizable; we denote it by $\text {Sp}(\delta ,k)[\alpha ,-\alpha ].$

Let B be the set of all

$$\begin{aligned} \text {Sp}(\delta ,k), \ \text {Sp}(\delta ,k)[\alpha ,-\alpha ], \end{aligned}$$

where $\delta \in \text {D}^u$, k is a positive integer and $0< \alpha < 1/2$. By [30, Theorem D], an irreducible representation $\pi$ is unitarizable if and only if it is of the form

$$\begin{aligned} \pi _1 \times \cdots \times \pi _t, \quad \pi _i \in B, \ i=1,\ldots , t. \end{aligned}$$

Moreover, this expresssion is unique up to permutation. We call it a Tadić decomposition of $\pi$.

By an irreducible representation of Arthur type, we mean an irreducible unitary representation whose Tadić decomposition does not involve any $\text {Sp}(\delta ,k)[\alpha ,-\alpha ]$. For $\pi \in \text {Irr}^u$, we then have a decomposition $\pi = \pi _{Ar } \times \pi _c$, where $\pi _{Ar }$ is a representation of Arthur type and is called the Arthur part of $\pi$.

3 Preliminaries on $(H_{p,q},\mu _a)$-distinguished representations

3.1 Basic facts

Lemma 3.1

• (1) Let $\pi$ be a smooth representation of $G_n$. If $\pi$ is $(H_{p,q},\mu _a)$-distinguished for two nonnegative integers p, q with $p + q = n$ and $a \in {\mathbb {R}}$, then $\pi$ is also $(H_{q,p},\mu _{-a})$-distinguished;

• (2) Let $\pi _1,\ldots , \pi _t \in Irr (G_n)$. If $\pi _1 \times \cdots \times \pi _t$ is $(H_{p,q},\mu _a)$-distinguished for two nonnegative integers p, q with $p + q = n$ and $a \in {{\mathbb {R}}}$, then $\pi _t^{\vee } \times \cdots \times \pi _1^{\vee }$ is $(H_{p,q},\mu _{-a})$-distinguished.

Proof

The statement (1) follows from the fact that $\pi \cong \pi ^{w_{q,p}}$. Let $\iota$ denote the involution of transpose inversion. Then (2) follows from the fact that $\pi \circ \iota \cong \pi ^{\vee }$ for any irreducible representation $\pi$ and the fact that

$$\begin{aligned} (\pi _1 \times \cdots \times \pi _t) \circ \iota \cong (\pi _t \circ \iota ) \times \cdots \times (\pi _1\circ \iota ) \end{aligned}$$

$\square$

For representations of dimension one, we have the following simple lemma, whose proof we omit.

Lemma 3.2

Let be a character of $G_n$. Assume that is $(H_{p,q},\mu _a)$-distinguished for nonnegative integers p, q with $p + q = n$ and $a \in {{\mathbb {R}}}$. If $q = 0$ (resp. $p = 0$), then is the character $\nu ^a$ (resp. $\nu ^{-a}$) of $G_n$; If p, $q > 0$, then $a = 0$ and , the trivial character of $G_n$.

For untwisted linear periods, we have the following fundamental result due to Jacquet and Rallis [11].

Lemma 3.3

Let p, q be two positive integers with $p + q = n$. If $\pi \in Irr (G_n)$, then $\dim \mathrm {Hom}_{H_{p,q}}(\pi ,{\textbf {1}}) \leqslant 1$. Furthermore, if $\dim \mathrm {Hom}_{H_{p,q}} (\pi , {\textbf {1}}) = 1$, then $\pi \cong \pi ^{\vee }$.

Remark 3.4

In this work we will not need multiplicity one results about (twisted) linear periods. However, the self-dualness property of distinguished representations is important for our applications of the geometric lemma. For example, one key ingredient is Proposition 3.9 which asserts self-duality for distinguished essentially square-integrable representations. In the case $p = q$, twisted linear periods have been studied by Chen and Sun in [3]. Their result shows that, for all but finitely many a, $\dim \mathrm {Hom}_{H_{p,p}}(\pi ,\mu _a) \leqslant 1$ for all $\pi \in Irr (G_{2p})$. Due to the author’s limited knowledge, one cannot deduce self-duality for distingsuished representations as in the untwisted case. For generic representations, however, one can deduce self-duality from a result of Gan as shown in the next subsection.

3.2 Relations with Shalika periods

The Shalika subgroup of $G_{2n}$ is defined to be

$$\begin{aligned} S_{2n} = \left\{ \left. \begin{pmatrix} a &{} b \\ 0 &{} a \end{pmatrix}\ \right| a \in G_n,\ b \in M_n \right\} = G_n < imes N_{n,n}, \end{aligned}$$

where $M_n$ indicates the set of $n \times n$ matrices with entries in F. Define a character $\psi _{S_{2n}}$ on $S_{2n}$ by

$$\begin{aligned} \psi _{S_{2n}} \left( \begin{pmatrix} a &{} \\ &{} a \end{pmatrix} \begin{pmatrix} 1 &{} b \\ &{} 1 \end{pmatrix} \right) = \psi _F({\mathrm {Tr}}(b)), \end{aligned}$$

(3.1)

where $\psi _F$ is a non-trivial character of F. For a smooth representation $\pi$ of $G_{2n}$, an element in $\mathrm {Hom}_{S_{2n}} (\pi ,\psi _{S_{2n}})$ is called a local Shalika period of $\pi$.

In the untwisted case, the relation between linear periods and Shalika periods is well known (see [14] for their equivalence in the case of supercuspidal representations; see also a discussion for relatively square-integrable representations in [20, Sect.  5]). Using a theta correspondence approach, Gan proved the following result that relates generalized linear periods and generalized Shalika periods on $G_n$.

Proposition 3.5

Let $\pi$ be an irreducible generic representation of $G_{2n}$ and $\sigma$ an irreducible representation of $G_n$. One has

$$\begin{aligned} \mathrm {Hom}_{S_{2n}} (\pi ,\sigma \boxtimes \psi _{S_{2n}}) \cong \mathrm {Hom}_{H_{n,n}} (\pi , \sigma \boxtimes {{\mathbb {C}}}), \end{aligned}$$

(3.2)

where $\sigma \boxtimes \psi _{S_{2n}}$ is viewed as a representation of $S_{2n} = G_n < imes N_{n,n}$.

Proof

This is a consequence of Theorem 3.1 and Theorem 4.1 of [8]. $\square$

In fact, in Theorem 3.1 of [8], Gan obtained a statement that relates the generalized linear period of an irreducible representation to the generalized Shalika period of the big theta lift of its contragradient. We refer interested readers to the original paper of Gan for more details. What is pertinent to this work is the following simple corollary that relates as well twisted linear periods in our context to Shalika periods.

Corollary 3.6

Let $\pi$ be a generic representation of $G_{2n}$. The followings are equivalent:

• (1) $\pi$ is $(H_{n,n},\mu _a)$-distinguished for some $a \in {{\mathbb {R}}}$;

• (2) $\pi$ is $(H_{n,n},\mu _a)$-distinguished for all $a \in {{\mathbb {R}}}$;

• (3) $\pi$ is $(S_{2n},\psi _{S_{2n}})$-distinguished.

In particular, if one of these equivalent conditions holds, then $\pi$ is self-dual.

Proof

As $\pi$ is generic, its twist $\nu ^a \pi$, for $a \in {\mathbb {R}}$, is also generic. So

$$\begin{aligned} \mathrm {Hom}_{H_{n,n}}(\pi ,\mu _a)&= \mathrm {Hom}_{H_{n,n}}( \pi , \nu ^a \boxtimes \nu ^{-a}) \cong \mathrm {Hom}_{H_{n,n}} (\nu ^{a}\pi , \nu ^{2a}\boxtimes {\mathbb {C}}) \\&\cong \mathrm {Hom}_{S_{2n}} (\nu ^{a}\pi , \nu ^{2a} \boxtimes \psi _{S_{2n}}) \cong \mathrm {Hom}_{S_{2n}} (\pi , \psi _{S_{2n}}). \end{aligned}$$

$\square$

3.3 The theory of Bernstein–Zelevinsky derivatives

Let $P_n \subset G_n$ be the mirabolic subgroup of $G_n$ consisting of matrices with the last row $(0,0,\ldots ,0,1)$. We refer the reader to [1, 3.2] for the definition of the following functors

$$\begin{aligned}&\varPsi ^-: Alg \,P_n \rightarrow Alg \, G_{n-1}, \qquad \varPsi ^+ : Alg \, G_{n-1} \rightarrow Alg \,P_n, \\&\varPhi ^-: Alg \, P_n \rightarrow Alg \, P_{n-1}, \qquad \varPhi ^+ : Alg \, P_{n-1} \rightarrow Alg \,P_n. \end{aligned}$$

Define $\pi ^{(k)} = \varPsi ^-(\varPhi ^-)^{k-1}\left( \pi |_{P_n} \right)$ to be the kth derivative of a representation $\pi$ of $G_n$.

The following proposition can be proved by the same argument as those in [15, Proposition 1] (see also [20, Proposition 3.1], where the linear subgroups $H_{p,q}$ take different forms.)

Proposition 3.7

If $\sigma$ is a representation of $P_{n-1}$ and is a character of $H_{p,q}$, then

as complex vector spaces, where is the character of $H_{q-1,p}$ defined by . In particular, for all $a \in {{\mathbb {R}}}$, one has

$$\begin{aligned} \mathrm {Hom}_{P_n \cap H_{p,q}} (\varPhi ^+ \sigma , \mu _a) \cong \mathrm {Hom}_{P_{n-1} \cap H_{q-1,p}} (\sigma , \mu _{-a-1/2}). \end{aligned}$$

(3.3)

As a corollary, we have the following result due to Matringe [20, Theorem 3.1].

Corollary 3.8

Let $\varDelta$ be an essentially square-integrable representation of $G_n$. Let p, q be two positive integers with $p + q =n$, and a character of $H_{p,q}$. Assume that $\pi$ is -distinguished. Then $p=q$.

Another application of Proposition 3.7 will generalize Corollary 3.8 to essentially Speh representations in Corollary 6.14 of Sect. 6.3.

As a direct consequence of Corollarys 3.8 and 3.6, we have:

Proposition 3.9

Let $\varDelta$ be an essentially square-integrable representation of $G_n$. If $\varDelta$ is $(H_{p,q},\mu _a)$-distinguished for two positive integers p, q with $p + q = n$ and some $a \in {\mathbb {R}}$, then $p = q$ and $\varDelta$ is $H_{p,p}$-distingusihed (hence self-dual).

4 Symmetric spaces and parabolic orbits

The main tool we use to classify distinguished unitary representations is the geometric lemma of Bernstein and Zelevinsky [1, Theorem 5.2]. Applying it requires a detailed analysis of the double coset space $P \backslash G_n / H_{p,q}$, where P is a parabolic subgroup of $G_n$. As $H_{p,q}$ is a symmetric subgroup of $G_n$, we follow the framework given by Offen in [26].

4.1 General notations

Let $G = G_{n}$, $H = H_{p,q}$ be the subgroup of $G_n$ as in the introduction. Let

$$\begin{aligned} \varepsilon = \varepsilon _{p,q} = \left( \begin{matrix} I_p &{} \\ &{} -I_q \end{matrix} \right) , \end{aligned}$$

and $\theta = \theta _{p,q}$ be the involution on $G_n$ defined by $\theta (g) = \varepsilon g \varepsilon ^{-1}$. The symmetric space associated to $(G,\theta )$ is

$$\begin{aligned} X = \{ g \in G \ |\ \theta (g) = g^{-1} \}, \end{aligned}$$

equipped with the G-action $g \cdot x = g x \theta (g)^{-1}$. The map $g \mapsto g \cdot e$ gives a bijection of the coset space G/H onto the orbit $G \cdot e \subset X$, and thus a bijection of the double coset space $P \backslash G / H$ onto the P-orbits in $G \cdot e$, where P denotes a parabolic subgroup of G. For any $g \in G$, denote by $[g]_G$ the conjugacy class of g in G. Note that the map $g \mapsto g \varepsilon$ gives a bijection of $G \cdot e$ onto $[\varepsilon ]_G$ and that the G-action on $G \cdot e$ is transformed to the conjugation action of G on $[\varepsilon ]_G$.

For any subgroup Q of G and $x \in X$, let $Q_x = \{ g \in Q \ |\ g \cdot x = x \}$ be the stabilizer of x in Q. Note that $Q_x$ is just the centralizer of $x \varepsilon$ in Q.

4.2 Twisted involutions in Weyl groups

A first coarse classification of the double cosets in $P \backslash G / H$ is given by certain Weyl group elements. Let W be the Weyl group of G. Let

$$\begin{aligned} W[2] = \{w \in W\ |\ w \theta (w) = e \} = \{w \in W \ |\ w^2 = e \} \end{aligned}$$

be the set of twisted involutions in W. For two standard Levi subgroups M and $M'$ of G, let be the set of all $w \in W$ that are left $W_M$-reduced and right $W_{M'}$-reduced.

Given a standard parabolic subgroup $P = M < imes U$, define a map

(4.1)

by the relation

$$\begin{aligned} PxP = P \iota _M(P \cdot x) P. \end{aligned}$$

(4.2)

For $x \in X$, let

$$\begin{aligned} w = \iota _M(P \cdot x) \text {\quad and \quad } L = M(w) = M \cap wMw^{-1}. \end{aligned}$$

Then L is a standard Levi subgroup of M satisfying $L = w L w^{-1}$.

4.3 Admissible orbits

It is noted in [26] that, to apply the geometric lemma in particular cases, it is necessary to first understand the admissible orbits. Recall that $x \in X$ (or a P-orbit $P \cdot x$ in X) is said to be M-admissible if $M = w M w^{-1}$ where $w = \iota _M(P \cdot x)$. We now describe the relevant data for M-admissible P-orbits in $G \cdot e$.

By [26, Corollary 6.2], M-admissible P-orbits in $G \cdot e$ is in bijection with M-orbits in $G \cdot e \cap N_G(M)$, or equivalently M-conjugacy classes in $[\varepsilon ]_G \cap N_G(M)$.

Fix a composition $\bar{n}=(n_1,\ldots ,n_t)$ of n. Let $P = M < imes U$ be the standard parabolic subgroup of $G_n$ associated to $\bar{n}$. Denote by $\mathfrak {S}_t^{(\bar{n})}$ the set of permutations $\tau$ on the set $\{ 1, 2, \ldots ,t\}$ such that $n_i = n_{\tau (i)}$ for all $i \in \{1, \ldots ,t \}$. To each $\tau$ in $\mathfrak {S}_t^{(\bar{n})}$, we associate a block matrix $w_{\tau }$ which has $I_{n_i}$ on its $(\tau (i),i)$-block for each i and has 0 elsewhere. Then the map

$$\begin{aligned} \tau \mapsto w_{\tau }M \end{aligned}$$

defines an isomorphism of groups from $\mathfrak {S}_t^{(\bar{n})}$ to $N_G(M) / M$. Write an element of M as $\text {diag}\{A_1,\ldots ,A_t\}$. Note that an element $w_{\tau }\text {diag}\{A_1,\ldots ,A_t \}$ of $N_G(M)$ has order 2 if and only if

$$\begin{aligned} \tau ^2 = 1 \quad \text {and} \quad A_iA_{\tau (i)} = I_{n_i}~\ \text {for all } i \in \{ 1,\ldots ,t \}. \end{aligned}$$

One sees that the M-conjugacy classes in $[\varepsilon ]_{G} \cap N_G(M)$ are parameterized by the set of pairs $(\mathfrak {c}_{\tau },\tau )$ where $\tau \in \mathfrak {S}_t^{(\bar{n})}$, $\tau ^2 = 1$, and $\mathfrak {c}_{\tau }$ is a set of the form

$$\begin{aligned} \{(n_{k,+}, n_{k,-}) \ |\ \text {for all} k \text {such that} \tau (k) = k \} \end{aligned}$$

such that

$$\begin{aligned} {\left\{ \begin{array}{ll} n_k = n_{k,+} + n_{k,-},\ n_{k,+},n_{k,-} \geqslant 0;\\ \sum _{k,\tau (k)= k} n_{k,+} + \sum _{(i,\tau (i)),i< \tau (i)} n_i = p;\\ \sum _{k,\tau (k)= k} n_{k,-} + \sum _{(i,\tau (i)),i < \tau (i)} n_i = q. \end{array}\right. } \end{aligned}$$

(4.3)

Denote by $\mathcal {I}_{p,q}^{\sharp }(\bar{n})$ the set of all such pairs.

For the M-admissible P-orbit $\mathcal {O}$ corresponding to $(\mathfrak {c}_{\tau },\tau )$ in $\mathcal {I}_{p,q}^{\sharp }(\bar{n})$, we can choose a natural orbit representative $x = x_{(\mathfrak {c}_{\tau },\tau )} \in \mathcal {O} \cap N_G(M)$ as follows: The matrix $x \varepsilon$ has $I_{n_i}$ on its $(\tau (i),i)$-block when $\tau (i) \ne i$, $\text {diag}(I_{n_{i,+}}, -I_{n_{i,-}})$ on its (i, i)-block when $\tau (i) = i$, and 0 elsewhere. One sees easily that $M_x$ consists of elements $\text {diag}\{A_1,\ldots ,A_t \}$ such that

$$\begin{aligned} {\left\{ \begin{array}{ll} A_i = A_{\tau (i)}, &{} \tau (i) \ne i; \\ A_i I_{n_{i,+},n_{i,-}} = I_{n_{i,+},n_{i,-}} A_i &{} \tau (i) = i. \end{array}\right. } \end{aligned}$$

(4.4)

Here and in what follows, we denote by $I_{n_1,n_2}$ for the diagonal matrix $\text {diag}\{I_{n_1}, -I_{n_2} \}$. Thus, when $\tau (i) = i$, we may further write $A_i$ as $\text {diag}\{A_{i,+},A_{i,-} \}$. One also has $P_x = M_x < imes U_x$.

The following computation of modular characters is indispensable for applications of the geometric lemma, see [26, Theorem 4.2]. We omit the proof here as it is obtained by a routine calculation.

Lemma 4.1

Let $x \in G\cdot e \cap N_G(M)$ be the representative as above of the M-admissible P-orbit corresponding to $(\mathfrak {c},\tau ) \in \mathcal {I}_{p,q}^{\sharp }(\bar{n})$. Then, for $m = diag \{A_1,\ldots ,A_t\} \in M_x$, we have

$$\begin{aligned} \delta _{P_x}\delta _P^{-1/2}(m) = \prod _{{\mathop {\tau (i) = i, \tau (j) = j}\limits ^{i<j}}}&\nu (A_{i,+})^{(n_{j,+} - n_{j,-})/2} \nu (A_{i,-})^{(n_{j,-} - n_{j,+})/2} \nu (A_{j,+})^{(n_{i,-} - n_{i,+})/2} \\&\cdot \nu (A_{j,-})^{(n_{i,+} - n_{i,-})/2} \prod _{{\mathop {\tau (i) > \tau (j)}\limits ^{i < j}}} \nu (A_i)^{-n_j/2} \nu (A_j)^{n_i/2}. \nonumber \end{aligned}$$

(4.5)

4.4 General orbits

For our purposes, we consider only P-orbits in $G \cdot e \subset X$ where P is a maximal parabolic subgroup. Let $P = P_{k,n-k}$ be the standard parabolic subgroup associated to $(k,n-k)$ with M its Levi subgroup. We follow the geometric method as in [22]. The case where $|p-q|\leqslant 1$ can be essentially covered by the results there. We remark however that the symmetric subgroup H there takes a different form and the treatment here is independent.

Let V be a n-dimensional F-vector space with a basis $\{e_1,\ldots , e_{n}\}$. Let $V_+$ (resp. $V_-$) be the subspace of V of dimension p (resp. q) which is generated by $\{e_1,\ldots ,e_p \}$ (resp. $\{ e_{p+1},\ldots , e_n \}$). The coset space G/P can be identified with the set of subspaces of V of dimension k. For such a subspace W, set

$$\begin{aligned} r_W = \dim _F (W \cap V_+),\quad s_W = \dim _F (W \cap V_-). \end{aligned}$$

Lemma 4.2

Let $W_1$ and $W_2$ be two subspaces of V of dimension k. Then they are in the same H-orbit if and only if $r_{W_1} = r_{W_2}$ and $s_{W_1} = s_{W_2}$. For a pair of nonnegative integers (r, s), there is a subspace W of V such that $r = r_W$ and $s = s_W$ if and only if

$$\begin{aligned} {\left\{ \begin{array}{ll} r + s \leqslant k, \\ k - s \leqslant p, \quad k -r \leqslant q. \end{array}\right. } \end{aligned}$$

(4.6)

Denote by $\mathcal {I}^{k}_{p,q}$ the set of pairs of nonnegative integers (r, s) that satisfying (4.6). Then, by Lemma 4.2, the double cosets in $H \backslash G / P$ can be parameterized by $\mathcal {I}^k_{p,q}$. For $(r,s) \in \mathcal {I}^{k}_{p,q}$, call $d = k - r -s$ the defect of (r, s).

We first seek a complete set of representatives of $P \backslash G / H$. We split the discussions into two cases.

Case $k \geqslant p$. Let $W_{(r,s)}$ be the subspace of V generated by

$$\begin{aligned} \{ e_1,\ldots ,e_r;e_{r+1}+e_{q+r+1},\ldots , e_{k-s}+e_{q+k-s};e_{q+k-s+1},\ldots e_n;e_{p+1},\ldots ,e_k \}. \end{aligned}$$

Then $\dim _F W_{r,s} = k$, $\dim _F (W_{(r,s)} \cap V_+) = r$ and $\dim _F (W_{(r,s)} \cap V_-) = s$. Let $\tilde{\eta }_{(r,s)}^{-1}$ be the block matrix

$$\begin{aligned} \begin{pmatrix} C_1 &{} C_2 \\ C_3 &{} C_4 \end{pmatrix} \end{aligned}$$

where $C_1$ and $C_4$ are matrices of size $p \times p$ and $q \times q$ respectively, and

$$\begin{aligned}&C_1 = \begin{pmatrix} I_{k-s} &{} \\ &{} 0 \end{pmatrix},\quad C_4 = \begin{pmatrix} I_{k-s+q-p} &{} \\ &{} 0 \end{pmatrix} \\&C_2 = \begin{pmatrix} 0 &{} 0 \\ 0 &{} I_{s+p-k} \end{pmatrix},\quad C_3 = \begin{pmatrix} 0 &{} 0 \\ 0 &{} I_{p-r} \end{pmatrix}. \end{aligned}$$

Then $\{\tilde{\eta }_{(r,s)}^{-1} \}$ is a complete set of representatives of the double coset space $H \backslash G / P$. Taking inverse, we thus get a complete set of representatives $\{ \tilde{\eta }_{(r,s)}\}$ of $P \backslash G / H$.

Case $k \leqslant p$. Let $W_{(r,s)}$ be the subspace of V of dimension k generated by

$$\begin{aligned} \{e_1,e_2,\ldots , e_r;e_{r+1}+e_{n-k+r+1},\ldots , e_{k-s}+e_{n-s}; e_{n-s+1},\ldots ,e_n \}. \end{aligned}$$

Then $\dim _F W_{r,s} = k$, $\dim _F (W_{r,s} \cap V_+) = r$ and $\dim _F (W_{(r,s)} \cap V_-) = s$. Let $\tilde{\eta }_{(r,s)}^{-1}$ be the block matrix

$$\begin{aligned} \begin{pmatrix} D_1 &{} D_2 \\ D_3 &{} D_4 \end{pmatrix} \end{aligned}$$

where $D_1$ and $D_4$ are matrices of size $k \times k$ and $(n-k) \times (n-k)$ respectively, and

$$\begin{aligned}&D_1 = \begin{pmatrix} I_{k-s} &{} \\ &{} 0 \end{pmatrix},\quad D_4 = \begin{pmatrix} I_{n-k-s} &{} \\ &{} 0 \end{pmatrix} \\&D_2 = \begin{pmatrix} 0 &{} 0 \\ 0 &{} I_{s} \end{pmatrix},\quad D_3 = \begin{pmatrix} 0 &{} 0 \\ 0 &{} I_{k-r} \end{pmatrix}. \end{aligned}$$

Then $\{\tilde{\eta }_{(r,s)}^{-1} \}$ is a complete set of representatives of the double coset space $H \backslash G / P$. Taking inverse, we thus get a complete set of representatives $\{ \tilde{\eta }_{(r,s)}\}$ of $P \backslash G / H$.

We then describe the relevant data for these general P-orbits in $G \cdot e$. For $(r,s) \in \mathcal {I}^k_{p,q}$, let $\tilde{x}_{(r,s)} = \tilde{\eta }_{(r,s)} \theta (\tilde{\eta }_{(r,s)})^{-1} \in G \cdot e$. Thus $\{ \tilde{x}_{(r,s)} \}$ is a complete set of representatives of P-orbits in $G\cdot e$. Write $w_{(r,s)} = \iota _{M} (P \cdot \tilde{x}_{(r,s)})$. Recall that $w_{(r,s)}$ is left and right $W_{M}$-reduced. In either case, we have that

$$\begin{aligned} w_{(r,s)} = \left( \begin{matrix} I_{k-d} &{} &{} &{} \\ &{} &{} I_d &{} \\ &{} I_d &{} &{} \\ &{} &{} &{} I_{n-k-d} \end{matrix}\right) . \end{aligned}$$

(4.7)

Thus $L = L_{(r,s)} = M \cap w_{(r,s)} M w_{(r,s)}^{-1}$ is the standard Levi subgroup associated to the composition $(k-d,d,d,n-k-d)$ of n. Denote by Q the standard parabolic subgroup of $G_n$ with Levi subgroup L. We can choose, in either case, an orbit representative $x_{(r,s)} \in P \cdot \tilde{x}_{(r,s)} \cap L w_{(r,s)}$ such that

$$\begin{aligned} x_{(r,s)} \varepsilon = \left( \begin{matrix} I_{r,s} &{} &{} &{} \\ &{} &{} I_d &{} \\ &{} I_d &{} &{} \\ &{} &{} &{} I_{p+s-k, q+r-k} \end{matrix} \right) . \end{aligned}$$

(4.8)

So the group $L_{x_{(r,s)}}$ consists of elements $\text {diag}\{A_{1,+},A_{1,-},A_2,A_3,A_{4,+},A_{4,-} \}$ such that

$$\begin{aligned} {\left\{ \begin{array}{ll} A_{1,+} \in G_r, \ A_{1,-} \in G_s,\quad A_{4,+} \in G_{p+s-k},\ A_{4,-} \in G_{q+r-k}; \\ A_2 = A_3 \in G_d. \end{array}\right. } \end{aligned}$$

We can also choose $\eta _{(r,s)} \in G_n$ such that $\eta _{(r,s)} \theta (\eta _{(r,s)})^{-1} = x_{(r,s)}$ and that

$$\begin{aligned} \eta _{(r,s)}^{-1} \left( {\begin{matrix} A_{1,+} &{} &{} &{} &{} &{} \\ &{} A_{1,-}&{} &{} &{} &{} \\ &{} &{} A_2&{} &{} &{} \\ &{} &{} &{} A_3 &{} &{} \\ &{} &{} &{} &{} A_{4,+}&{} &{} \\ &{} &{} &{} &{} &{}A_{4,-} \end{matrix}} \right) \eta _{(r,s)} = \left( {\begin{matrix} A_{1,+} &{} &{} &{} &{} &{} \\ &{} A_2&{} &{} &{} &{} \\ &{} &{} A_{4,+}&{} &{} &{} \\ &{} &{} &{} A_{4,-} &{} &{} \\ &{} &{} &{} &{} A_3&{} &{} \\ &{} &{} &{} &{} &{}A_{1,-} \end{matrix}} \right) \in H_{p,q} \end{aligned}$$

(4.9)

The modular characters for general orbtis that are relavent to us are computed as follows.

Lemma 4.3

For $(r,s) \in \mathcal {I}^k_{p,q}$, let $x = x_{(r,s)}$, $\eta = \eta _{(r,s)}$, L and Q as given above. For $a \in {\mathbb {R}}$, let $\mu _a$ be the character of $H = H_{p,q}$ defined in (2.1).

For

$$\begin{aligned} m = diag \{A_{1,+},A_{1,-},A_2,A_3,A_{4,+},A_{4,-} \} \in L_x, \end{aligned}$$

then

$$\begin{aligned} \delta _{Q_x}\delta _Q^{-1/2} (m)&= \nu (A_{1,+})^{(p-q+s-r)/2} \nu (A_{1,-})^{(q-p+r-s)/2} \nu (A_{4,+})^{(s-r)/2} \nu (A_{4,-})^{(r-s)/2}, \\ \mu _a^{\eta ^{-1}} (m)&= \nu ( A_{1,+} )^{a} \nu ( A_{1,-})^{-a} \nu ( A_{4,+})^{a} \nu ( A_{4,-})^{-a}. \nonumber \end{aligned}$$

(4.10)

Proof

Note that $x_{(r,s)}$ is the natural representative for an L-admissible Q-orbit in $G\cdot e$ chosen in Sect. 4.3. Then (4.10) follows directly from Lemma 4.1. $\square$

5 Consequences of the geometric Lemma

5.1 The geometric lemma

We first recall the formulation of the geometric lemma of Bernstein and Zelevinsky in [26, Theorem 4.2], and we refer the reader to loc.cit for unexplained notation.

Proposition 5.1

Let $P = M < imes U$ be a standard parabolic subgroup of G. Let $\sigma$ be a representation of M, and a character of H. If the representation $\mathrm {Ind}_P^G (\sigma )$ is -distinguished, then there exist a P-orbit $\mathcal {O}$ in $P \backslash (G \cdot e)$ and $\eta \in G$ satisfying $x = \eta \cdot e \in \mathcal {O} \cap Lw$ (where $w = \iota _M(P \cdot x)$ and $L = M(w)$) such that the Jacquet module $r_{L,M}(\sigma )$ is -distinguished. Here $Q = L < imes V$ is the standard parabolic subgroup of G with Levi subgroup L.

We retain the notation of Sect. 4. As a consequence of the orbit analysis there, we formulate the following corollary.

Corollary 5.2

Let $\sigma _1$ resp. $\sigma _2$ be a representation of $G_k$ resp. $G_{n-k}$. If the representation $\sigma _1 \times \sigma _2$ is $(H_{p,q},\mu _a)$-distinguished for some p, $q \geqslant 0$, $p + q =n$ and $a \in {\mathbb {R}}$, then there exists a pair $(r,s) \in \mathcal {I}^k_{p,q}$ with defect $d = k - r -s$ such that the representation $r_{(k-d,d)} \sigma _1 \otimes r_{(d,n-k-d)}\sigma _2$ of L is $(L_x, \delta _{Q_x}\delta _Q^{-1/2}\mu _a^{\eta ^{-1}})$-distinguished, where L is the standard Levi subgroup of $G_n$ associated to $(k-d,d,d,n-k-d)$, Q is the standard parabolic subgroup with L its Levi part, $x = x_{(r,s)}$ is given in (4.8) and $\eta = \eta _{(r,s)} \in G_n$ such that $x = \eta \cdot e$ and (4.9) holds.

Often in practice there is a filtration of the Jacquet module of the inducing data whose successive factors are pure tensor representations. The following lemma is a direct consequence of Lemma 4.3.

Lemma 5.3

Notation being as above. Let $\rho = \rho _1 \otimes \rho _2 \otimes \rho _3 \otimes \rho _4$ be a pure tensor representation of L. Then $\rho$ is $(L_{x},\delta _{Q_x}\delta _Q^{-1/2}\mu _a^{\eta ^{-1}})$-distinguished if and only if

$$\begin{aligned} {\left\{ \begin{array}{ll} \rho _2 \cong \rho _3^{\vee }, \\ \rho _1 \text { is }(H_{r,s},\mu _{a+(p - q + s - r)/2}) \text {-distinguished}, \\ \rho _4 \text { is }(H_{p+s-k,q+r-k}, \mu _{a+(s-r)/2}) \text { distinguished}. \end{array}\right. } \end{aligned}$$

(5.1)

Remark 5.4

Our proof of classification has an inductive structure. This necessary conditions (5.1) is the reason why we study $(H_{p,q},\mu _a)$-distinction from the beginning, although our main concern is about $H_{p,p}$-distinction.

Remark 5.5

The subscripts in the pair $(H_{p,q},\mu _a)$ play a subtle role in this work as, for example, seen from Proposition 6.16. We do not have a conceptual explanation for this now. The following observation might be helpful when applying this lemma. For a pair $(H_{p,q},\mu _a)$, set $S^+ (p,q,a) = p - q + 2a$ and $S^-(p,q,a) = p - q - 2a$. When passing from distingusihed $\sigma _1 \times \sigma _2$ to distingsuihed $\rho _1$ and $\rho _4$, the invariants $S^+$ and $S^-$ for the subgroup pairs are preserved respectively.

To handle the duality relation in (5.1), we have the following

Lemma 5.6

Let $\mathfrak {m}_1,\ldots ,\mathfrak {m}_r$ and $\mathfrak {n}_1,\ldots ,\mathfrak {n}_s$ be multisegments. If

$$\begin{aligned} L (\mathfrak {m}_1) \times \cdots \times L (\mathfrak {m}_r) \cong L (\mathfrak {n}_1) \times \cdots \times L (\mathfrak {n}_s), \end{aligned}$$

then $\mathfrak {m}_1 + \cdots + \mathfrak {m}_r = \mathfrak {n}_1 + \cdots + \mathfrak {n}_s$.

Proof

It is known that $L (\mathfrak {m}_1 + \cdots + \mathfrak {m}_r)$ is a subquotient of $L (\mathfrak {m}_1) \times \cdots \times L (\mathfrak {m}_r)$. By our condition, it is then a subquotient of $\lambda (\mathfrak {n}_1 + \cdots + \mathfrak {n}_s)$. Reversing the roles of $\mathfrak {m}_i$’s and $\mathfrak {n}_j$’s, the required equality follows from [33, Theorem 7.1] (see also [31, Theorem 5.3]). $\square$

As seen from above, the geometric lemma provides us necessary conditions for distinction of induced representations.We now present a sufficient condition that is due to Matringe [22, Proposition 3.8].

Lemma 5.7

Let $n_1 = 2 m_1$ and $n_2 = 2 m_2$ be even integers, let $a \in {\mathbb {R}}$. Assume that $\pi _1$ is $(H_{m_1,m_1},\mu _a )$-distinguished and $\pi _2$ is $(H_{m_2,m_2} , \mu _a)$-distinguished. Then $\pi _1 \times \pi _2$ is $(H_{m_1+m_2,m_1+m_2},\mu _a)$-distinguished.

5.2 Distinction of products of essentially square-integrable representations

We now apply Corollary 5.2 to products of essentially square-integrable representations.

Proposition 5.8

Let $\pi = \varDelta _1 \times \cdots \times \varDelta _t$ be a representation of $G_n$, where $\varDelta _i = \varDelta ([a_i,b_i]_{\rho _i})$ is an essentially square-integrable representation of $G_{n_i}$, $i = 1, \ldots ,t$. (Here we assume all $a_i$, $b_i$ are integers.) Suppose that $\pi$ is $(H_{p,q},\mu _a)$-distinguished with p, q two nonnegative integers, $p + q =n$, and $a \in {\mathbb {R}}$. Then there exist an integer $c_t$ satisfying $a_t - 1 \leqslant c_t \leqslant b_t$ such that one of the following cases must hold:

Case A1. One has $a_t = c_t < b_t$. The representation $\varDelta ([a_t,c_t]_{\rho _t}) = {\textbf {b}}(\varDelta _t)$ is either the character $\nu ^{a+(q-p+1)/2}$ or the character $\nu ^{-a + (p-q + 1)/2}$ of $G_1$; and there exists $i \in \{ 1, 2, \ldots , t-1\}$, and an integer $c_{i}$, $a_{i} \leqslant c_{i} \leqslant b_i$, such that

• (i) one has $\varDelta ([a_t+1,b_t]_{\rho _t})^{\vee } \cong \varDelta ([a_i,c_i]_{\rho _i})$;

• (ii) the representation

  $$\begin{aligned} \varDelta _1 \times \cdots \times \varDelta ([c_i + 1,b_i]_{\rho _i}) \times \cdots \times \varDelta _{t-1} \end{aligned}$$

  is $(H_{p-n_t, 1 + q -n_t},\mu _{a + 1/2})$ or $(H_{ 1 +p-n_t, q -n_t},\mu _{a - 1/2})$-distinguished, depending on ${\textbf {b}}(\varDelta _t)$.

Case A2. One has $a_t \leqslant c_t < b_t$. The representation $\varDelta ([a_t,c_t]_{\rho _t})$, with its degree $n_t'$ an even integer, is $H_{n_t'/2,n_t'/2}$-distinguished; and there exists $i \in \{1, 2,\ldots , t-1\}$ and an integer $c_{i}$, $a_{i} \leqslant c_{i} \leqslant b_i$, such that

• (i) one has $\varDelta ([c_t+1,b_t]_{\rho _t})^{\vee } \cong \varDelta ([a_i,c_i]_{\rho _i})$;

• (ii) the representation

  $$\begin{aligned} \varDelta _1 \times \cdots \times \varDelta ([c_i + 1,b_i]_{\rho _i}) \times \cdots \times \varDelta _{t-1} \end{aligned}$$

  is $(H_{p', q'},\mu _{a})$ -distinguished with $p' = p-n_t + n_t'/2$ and $q' = q - n_t + n_t'/2$.

Case B1. One has $c_t = b_t$. The representation $\varDelta ([a_t,c_t]_{\rho _t}) = \varDelta _t$ is either the character $\nu ^{a+(q-p+1)/2}$ or the character $\nu ^{-a + (p-q + 1)/2}$ of $G_1$; and the representation

$$\begin{aligned} \varDelta _1 \times \cdots \times \varDelta _{t-1} \end{aligned}$$

is $(H_{p-1, q},\mu _{a + 1/2})$ or $(H_{ p, q - 1},\mu _{a - 1/2})$-distinguished, depending on $\varDelta _t$.

Case B2. One has $c_t = b_t$. The representation $\varDelta _t$ is $H_{n_t/2,n_t/2}$-distinguished, where $n_t$ is even; and the representation

$$\begin{aligned} \varDelta _1 \times \cdots \times \varDelta _{t-1} \end{aligned}$$

is $(H_{p-n_t/2, q-n_t/2},\mu _a)$-distinguished.

Case C. One has $c_t = a_t - 1$. There exists $i \in \{1,2,\ldots , t-1 \}$ and an integer $c_{i}$, $a_{i} \leqslant c_{i} \leqslant b_i$, such that

• (i) one has $\varDelta _t^{\vee } \cong \varDelta ([a_i,c_i]_{\rho _i})$;

• (ii) the representation

  $$\begin{aligned} \varDelta _1 \times \cdots \times \varDelta ([c_i + 1,b_i]_{\rho _i}) \times \cdots \times \varDelta _{t-1} \end{aligned}$$

  is $(H_{p-n_t, q -n_t},\mu _a)$-distinguished.

Proof

Write $\sigma _1 = \varDelta _1 \times \cdots \times \varDelta _{t-1}$ and $\sigma _2 = \varDelta _t$, and $k = n - n_t$. By Corollary 5.2, in its notation, there exists $(r,s) \in \mathcal {I}^k_{p,q}$ with defect $d = k - r - s$ such that the representation $r_{(k-d,d)} \sigma _1 \otimes r_{(d,n-k-d)}\sigma _2$ of L is $(L_x, \delta _{Q_x}\delta _Q^{-1/2}\mu _a^{\eta })$-distinguished. By [33, 9.5], the Jacquet module $r_{(d,n-k-d)} \sigma _2$ of $\sigma _2$ is either zero or of the form $\varDelta ([c_t + 1, b_t]_{\rho _t}) \otimes \varDelta ([a_t,c_t]_{\rho _t})$ for certain integer $c_t$ with $a_t - 1 \leqslant c_t \leqslant b_t$. By [33, 1.2, 1.6], there exists a filtration $0 \subset V_1 \subset \cdots \subset V= r_{(k-d,d)} \sigma _1$ such that each successive factor is equivalent to a representation of the form

$$\begin{aligned}&\varDelta ([c_1 + 1,b_1]_{\rho _1}) \times \cdots \times \varDelta ([c_{t-1} + 1,b_{t-1}]_{\rho _{t-1}})\\&\quad \otimes \varDelta ([a_1,c_1]_{\rho _1}) \times \cdots \times \varDelta ([a_{t-1},c_{t-1}]_{\rho _{t-1}}), \end{aligned}$$

for certain integers $c_i$ such that $a_i - 1 \leqslant c_i \leqslant b_i$, $i = 1, \ldots ,t-1$. Therefore, there exists integers $c_i$, $i =1 ,2,\ldots ,t$, such that the pure tensor representation

$$\begin{aligned} \prod _{i=1}^{t-1} \varDelta ([c_i +1, b_i]_{\rho _i}) \otimes \prod _{i=1}^{t-1} \varDelta ([a_i,c_i]_{\rho _i}) \otimes \varDelta ([c_t + 1, b_t]_{\rho _t}) \otimes \varDelta ([a_t,c_t]_{\rho _t}) \end{aligned}$$

is $(L_x,\delta _{Q_x}\delta _Q^{-1/2}\mu _a^{\eta ^{-1}})$-distinguished. By Lemma 5.3, we have

$$\begin{aligned} \varDelta ([c_t +1, b_t]_{\rho _t})^{\vee } \cong \prod _{i=1}^{t-1} \varDelta ([a_i,c_i]_{\rho _i}). \end{aligned}$$

By Lemma 5.6, $c_i = a_i - 1$ for all but one i between 1 and $t-1$. So, for this i, we have

$$\begin{aligned} \varDelta ([c_t +1, b_t]_{\rho _t})^{\vee } \cong \varDelta ([a_i , c_i]_{\rho _i}). \end{aligned}$$

(5.2)

Lemma 5.3 also implies that

$$\begin{aligned} \varDelta ([a_t,c_t]_{\rho _t}) \text { is} (H_{p+s-k,q+r-k}, \mu _{a + (s-r)/2})\text {-distinguished,} \end{aligned}$$

(5.3)

and that

$$\begin{aligned} \varDelta _1 \times \cdots \times \varDelta ([c_i + 1,b_i]_{\rho _i}) \times \cdots \times \varDelta _{t-1} \nonumber \\ \text {is} (H_{r,s}, \mu _{a + (p-q + s - r) /2})\text {-distinguished.} \end{aligned}$$

(5.4)

When $a_t \leqslant c_t < b_t$, we have two subcases. If $c_t = a_t$ and the degree of $\rho _t$ equals to 1, it follows from (5.3) that $(p+s-k, q+r-k) = (1,0)$ or (0, 1). By (5.4), (5.2) and simple calculations, we then have Case A1; Otherwise, the representation $\varDelta ([a_t,c_t]_{\rho _t})$ is not one dimensional. Thus, in (5.3) we have $p+s-k >0$ and $q+r-k >0$. By Proposition 3.9, we get that $\varDelta ([a_t,c_t]_{\rho _t})$ is $H_{n_t'/2,n_t'/2}$-distinguished with $n_t'$ its degree. The rest statements of Case A2 follow from simple calculations. Thus we have Case A2.

When $c_t = b_t$, we have two subcases. If $\varDelta _t$ is a character of $G_1$, then by similar arguments as in Case A1, we have Case B1. Otherwise, by similar arguments as in Case A2, we have Case B2. In these two cases, we have $d = 0$ and $c_i = a_i - 1$ by our convention.

When $c_t = a_t - 1$, by (5.3), we have $p+s-k = q + r -k = 0$. The statements of Case C follow from (5.4), (5.2) and simple calculations. So we are done. $\square$

Corollary 5.9

Let $\pi = \varDelta _1 \times \cdots \times \varDelta _t$ be as above. If $\pi$ is $(H_{p,q},\mu _a)$-distinguished with p, q and a as above, then either the representation $\varDelta _t$ is the character $\nu ^{a+(q-p+1)/2}$ or the character $\nu ^{-a + (p-q + 1)/2}$ of $G_1$, or there is $i \in \{ 1, 2 , \ldots , t\}$ such that ${\textbf {e}}(\varDelta _t)^{\vee } \cong {\textbf {b}}(\varDelta _i)$.

Proof

Note that in all cases other than Case B1, we have a duality relation. $\square$

Considering the duality relation between extremities of segments, a generalization of Corollary 5.9 is given later in Proposition 6.8.

6 Distinction of ladder representations

6.1 Notations and basic facts

The class of ladder representations was first introduced by Lapid and Mínguez in [17], and was further studied by Lapid and his collaborators in [16] and [18]. We start by reviewing some basic facts of these representations.

6.1.1 Definitions

Let $\rho \in {\mathscr {C}}$. By a ladder we mean a set $\{\varDelta _1,\ldots , \varDelta _t \} \in {\mathscr {O}}_{\rho }$ such that

$$\begin{aligned} {\textbf {b}}(\varDelta _1)> \cdots> {\textbf {b}}(\varDelta _t) \quad \text {and}\quad {\textbf {e}}(\varDelta _1)> \cdots > {\textbf {e}}(\varDelta _t). \end{aligned}$$

(6.1)

A representation $\pi \in Irr$ is called a ladder representation if $\pi = L (\mathfrak {m})$ where $\mathfrak {m} \in {\mathscr {O}}_{\rho }$ is a ladder. Whenever we say that $\mathfrak {m} = \{\varDelta _1,\ldots ,\varDelta _t \} \in \mathscr {O}_{\rho }$ is a ladder, we implicitly assume that $\mathfrak {m}$ is already ordered as in (6.1). We denote by $\mathfrak {m}^{\vee } \in {\mathscr {O}}_{\rho ^{\vee }}$ the ladder $\{\varDelta _t^{\vee },\ldots ,\varDelta _1^{\vee }\}$.

Lemma 6.1

Let $\mathfrak {m} \in {\mathscr {O}}_{\rho }$ be a ladder. One has $L (\mathfrak {m})^{\vee } = L (\mathfrak {m}^{\vee })$.

Proof

See [30, Proposition 5.6] $\square$

We introduce some more notation. For a ladder $\mathfrak {m} = \{ \varDelta _1,\ldots , \varDelta _t \} \in {\mathscr {O}}_{\rho }$ ordered as in (6.1), set $\pi = L (\mathfrak {m})$. We shall denote ${\textbf {b}}(\varDelta _1)$ by ${\textbf {b}}(\pi )$, called the beginning of the ladder representaion $\pi$; denote ${\textbf {e}}(\varDelta _t)$ by ${\textbf {e}}(\pi )$, called the end of $\pi$. We shall denote the number t of segments in $\mathfrak {m}$ by ${\textbf {ht}}(\pi )$, called the height of $\pi$.

We say that $\pi$ is a decreasing (resp. increasing) ladder representation if

$$\begin{aligned} l(\varDelta _1) \geqslant \cdots \geqslant l(\varDelta _t) \quad (resp. l(\varDelta _1) \leqslant \cdots \leqslant l(\varDelta _t) ). \end{aligned}$$

We say that $\pi$ is a left aligned (resp. right aligned) representation if ${\textbf {b}}(\varDelta _i) = {\textbf {b}}(\varDelta _{i+1}) + 1$ (resp. ${\textbf {e}}(\varDelta _i) = {\textbf {e}}(\varDelta _{i+1}) + 1$ ), $i= 1,\ldots , t-1$. Note that left aligned repreesentations are decreasing ladder representations and right aligned representations are increasing ladder representations.

A ladder representation is called an essentially Speh representation if it is both left aligned and right aligned. Note that essentially Speh representations are just the usual Speh representations up to twist by a non-unitary character. Let $\varDelta$ be an essentially square-integrable representation of $G_d$ and k a positive integer. Then $\mathfrak {m}_1 = \{\nu ^{(k-1)/2}\varDelta , \nu ^{(k-3)/2}\varDelta , \ldots , \nu ^{(1-k)/2}\varDelta \}$ is a ladder, and the ladder representation $L (\mathfrak {m}_1)$ is an essentially Speh representation, which we denote by $\text {Sp}(\varDelta ,k)$. All essentially Speh representations can be obtained in this manner.

Let $\pi = L (\mathfrak {m})$ as above. Let us further write $\varDelta _i = \varDelta ([a_i,b_i]_{\rho })$. (The $a_i$’s are integers by our convention.) By a division of $\pi$ as two ladder representations $\pi '$ and $\pi ''$, denoted by $\pi = \pi ' \sqcup \pi ''$, we mean that there exist integers $c_i$ with $a_i - 1 \leqslant c_i \leqslant b_i$, $i = 1, \ldots , t$, such that

$$\begin{aligned} c_1> c_2> \cdots > c_t \end{aligned}$$

and that

$$\begin{aligned} \pi '&= L (\varDelta ([a_1,c_1]_{\rho }), \ldots , \varDelta ([a_t,c_t]_{\rho })), \\ \pi ''&= L (\varDelta ([c_1 + 1, b_1]_{\rho }), \ldots , \varDelta ([c_t + 1, b_t]_{\rho })). \end{aligned}$$

Note that if $\pi$ is an essentially Speh representation and $\pi = \pi ' \sqcup \pi ''$ with neither $\pi '$ nor $\pi ''$ the trivial representation of $G_0$, then we have ${\textbf {b}}(\pi ) = {\textbf {b}}(\pi ')$ and ${\textbf {e}}(\pi ) = {\textbf {e}}(\pi '')$.

6.1.2 Standard module

One useful property of ladder representations is that the relation between them and their standard modules is explicit. Let $\mathfrak {m}= \{\varDelta _1,\ldots , \varDelta _t \} \in {\mathscr {O}}_{\rho }$ be a ladder with $\varDelta _i = \varDelta ([a_i, b_i])_{\rho }$. Set

$$\begin{aligned} \mathcal {K}_i = \varDelta _1 \times \cdots \times \varDelta _{i-1} \times \varDelta ([a_{i+1}, b_i]_{\rho }) \times \varDelta ([a_i,b_{i+1}]_{\rho }) \times \varDelta _{i+1} \times \cdots \times \varDelta _t, \end{aligned}$$

for $i = 1, \ldots , t-1$. (By our convention, $\mathcal {K}_i = 0$ if $a_i > b_{i+1} + 1$). By [17, Theorem 1] we have

Proposition 6.2

With the above notation let $\mathcal {K}$ be the kernel of the projection $\lambda (\mathfrak {m}) \rightarrow L (\mathfrak {m})$. Then $\mathcal {K} = \sum _{i=1}^{t-1} \mathcal {K}_i$.

6.1.3 Jacquet modules

The Jacquet modules of ladder representations were computed in [16, Corollary 2.2], where it is shown that the Jacquet module of a ladder representation is semisimple, multiplicity free, and that its irreducible constituents are themselves tensor products of ladder representations. For us, we need only the Jacquet modules with respect to maximal parabolic subgroups. We record the result in [16] here. Let $P = M < imes U$ be the standard parabolic subgroup of $G_n$ associated to $(k,n-k)$.

Proposition 6.3

Let $\mathfrak {m}= \{\varDelta _1,\ldots , \varDelta _t \} \in {\mathscr {O}}_{\rho }$ be a ladder with $\varDelta _i = [a_i, b_i]_{\rho }$, and $\pi = L (\mathfrak {m})$. Then

$$\begin{aligned} r_{M,G} (\pi ) = \mathop {{\mathop {\sum }\nolimits '}}\limits _{\pi = \pi _1 \sqcup \pi _2} \pi _2 \otimes \pi _1, \end{aligned}$$

where the summation takes over all divisions of $\pi$ as two ladder representations $\pi _1$ and $\pi _2$ such that the degree of $\pi _1$ is $n - k$ and that the degree of $\pi _2$ is k.

6.1.4 Bernstein–Zelevinsky derivatives

The full derivative of a ladder representation was computed in [17, Theorem 14], where it is shown that the semisimplification of all of the derivatives of a ladder representation consists of ladder representations of smaller groups. In particular, the derivatives of a left aligned representation take simple forms, which we recall here.

Lemma 6.4

Let $\rho \in {\mathscr {C}} (G_d)$, and $\mathfrak {m}= \{\varDelta _1,\ldots , \varDelta _t \} \in {\mathscr {O}}_{\rho }$ be a ladder with $\varDelta _i = \varDelta ([a_i, b_i]_{\rho })$. Suppose that $\pi = L (\mathfrak {m})$ is a left aligned representation. If k is not divided by d, then $\pi ^{(k)} = 0$. If $k = rd$, then

$$\begin{aligned} \pi ^{(k)} = L (\varDelta ([a_1 + r,b_1]_{\rho }),\varDelta _2, \ldots ,\varDelta _t). \end{aligned}$$

6.2 Distinction of products of essentially Speh representations

In this subsection we apply Corollary 5.2 to products of essentially Speh representations.

Instead of Lemma 5.6, we will use the following lemma to handle the duality relation in consequences of the geometric lemma.

Lemma 6.5

Let $\sigma$ and $\pi _i$ be left aligned representations of $G_n$ and $G_{n_i}$, $i = 1, \ldots , k$. If $\sigma \cong \pi _1 \times \cdots \times \pi _k$, then $k = 1$.

Proof

By Lemma 6.4, the derivatives of left aligned representations are either 0 or irreducible representations. Our assertion then follows form the description of the derivatives of a product of representations in [1, Corollary 4.6] $\square$

In view of Lemma 6.5 and the description of Jacquet modules of a ladder representation in Proposition 6.3, we formulate the following proposition, whose proof is very similar to that of Proposition 5.8 and is omitted here.

Proposition 6.6

Let $\pi = \pi _1 \times \cdots \times \pi _t$ be a representation of $G_n$, where $\pi _i$ is an essentiallly Speh representation of $G_{n_i}$, $i = 1 , \ldots , t$. Assume that $\pi$ is $(H_{p,q},\mu _a)$-distinguished with p, q two nonnegative integers, $p + q =n$ and $a \in {\mathbb {R}}$. Then there exist a division of $\pi _t$ as two ladder representations $\pi _t'$ and $\pi _t''$, $\pi _t = \pi _t' \sqcup \pi _t''$, with degrees $n_t'$ and $n_t''$ respectively, such that one of the following cases must hold:

Case A. The representation $\pi _t'$ is neither $\pi _t$ nor the trivial representation of $G_0$. There exists $i_0$, $1 \leqslant i_0 \leqslant t-1$, and a division of $\pi _{i_0}$ as two ladder representations $\pi _{i_0}'$ and $\pi _{i_0}''$, $\pi _{i_0} = \pi _{i_0}' \sqcup \pi _{i_0}''$, such that

• (i) $\pi _t'$ is $(H_{r,s},\mu _{a+(r-s+q-p)/2})$-distinguished, for two nonnegative integers $r, s \geqslant 0$, $r + s = n_t'$;

• (ii) $\pi _t''^{\vee } \cong \pi _{i_0}'$;

• (iii) the representation

  $$\begin{aligned} \pi _1 \times \cdots \times \pi _{i_0 -1} \times \pi _{i_0}'' \times \pi _{i_0 +1} \times \cdots \times \pi _{t-1} \end{aligned}$$

  (6.2)

  is $(H_{r',s'},\mu _{a+(s'-r'+ p-q)/2})$-distinguished, for two nonnegative integers $r'$, $s' \geqslant 0$, $r' + s' = n - n_t - n_t''$.

Case B. One has $\pi _t' = \pi _t$ is $(H_{r,s},\mu _{a+(r-s+q-p)/2})$-distinguished, for two nonnegative integers r, $s \geqslant 0$, $r + s = n_t$, and the representation

$$\begin{aligned} \pi _1 \times \cdots \times \pi _{t-1} \end{aligned}$$

(6.3)

is $(H_{r',s'},\mu _{a+(s'-r'+ p-q)/2})$-distinguished, for two nonnegative integers $r',s' \geqslant 0$, $r' + s' = n - n_t$.

Case C. The representation $\pi _t'$ is the trivial representation of $G_0$, so $\pi _t'' = \pi _t$. There exists $i_0$, $1 \leqslant i_0 \leqslant t-1$, and a division of $\pi _{i_0}$ as two ladder representations $\pi _{i_0}'$ and $\pi _{i_0}''$, $\pi _{i_0} = \pi _{i_0}' \sqcup \pi _{i_0}''$, such that

• (i) $\pi _t^{\vee } \cong \pi _{i_0}'$;

• (ii) the representation

  $$\begin{aligned} \pi _1 \times \cdots \times \pi _{i_0 -1} \times \pi _{i_0}'' \times \pi _{i_0 +1} \times \cdots \times \pi _{t-1} \end{aligned}$$

  (6.4)

  is $(H_{r,s},\mu _{a+(s-r+ p-q)/2})$-distinguished, for two nonnegative integers $r,s \geqslant 0$, $r + s = n - 2 n_t$.

Remark 6.7

It is easy to see that Lemma 6.5 fails if one removes the condition that $\sigma$ is left aligned. This is the reason why we restrict ourselves to products of essentially Speh representations here. Proposition 6.6 makes an inductive proof of the classification result possible as, in many cases, the representations (6.2), (6.3) and (6.4) are still products of essentially Speh representations.

Nevertheless, we have the following proposition for products of ladder representations that is very useful in later arguments.

Proposition 6.8

Let $\varPi = \pi _1 \times \cdots \times \pi _t$ and $\varPi ' = \pi '_1 \times \cdots \times \pi '_s$ be two products of ladder representations. If $\varPi \times \varPi '$ is $(H_{p,q},\mu _a)$-distinguished for two nonnegative integers p, q, $p + q =n$ and $a \in {\mathbb {R}}$, then there are two possibilities here:

• (1) $\varPi$ is $(H_{p_1,q_1},\mu _{a_1})$-distinguished and $\varPi '$ is $(H_{p_2,q_2},\mu _{a_2})$-distinguished for some $p_i$, $q_i$ and $a_i$, $i = 1, 2$. Here the subscripts $(p_i,q_i,a_i)$, $i = 1,2$, satisfy

  $$\begin{aligned} {\left\{ \begin{array}{ll} p_1 + p_2 = p \\ q_1 + q_2 = q \end{array}\right. } and \, {\left\{ \begin{array}{ll} p_1 - q_1 + 2 a_1 = p - q + 2 a \\ p_2 - q_2 - 2 a_2 = p - q - 2a. \end{array}\right. } \end{aligned}$$

• (2) There exist $i \in \{1,\ldots ,t\}$ and $j \in \{ 1, \ldots ,s\}$ such that ${\textbf {e}}(\pi '_j)^{\vee } \cong {\textbf {b}}(\pi _i)$.

Proof

This follows from similar arguments of Proposition 5.8 and the following simple implication of Lemma 5.6 when applied to ladder representations. $\square$

Lemma 6.9

Let $\mathfrak {m}_1,\ldots ,\mathfrak {m}_r$ and $\mathfrak {n}_1,\ldots ,\mathfrak {n}_s$ be ladders. If

$$\begin{aligned} L (\mathfrak {m}_1) \times \cdots \times L (\mathfrak {m}_r) \cong L (\mathfrak {n}_1) \times \cdots \times L (\mathfrak {n}_s), \end{aligned}$$

then there exist $i \in \{1,\ldots ,r\}$ and $j \in \{ 1, \ldots ,s\}$ such that ${\textbf {b}}(L (\mathfrak {m}_i)) \cong {\textbf {b}}(L (\mathfrak {n}_j))$.

Proof

By Lemma 5.6, one has $\mathfrak {m}_1+ \cdots + \mathfrak {m}_r = \mathfrak {n}_1 + \cdots + \mathfrak {n}_s$. Write $\mathfrak {m}_i = \{\varDelta _{i,1},\ldots ,\varDelta _{i,k_i} \}$ and $\mathfrak {n}_j = \{ \varDelta '_{j,1},\ldots , \varDelta '_{j,l_j}\}$ for these i’s and j’s. Let $\varDelta$ be one segement in $\sum _i \mathfrak {m}_i$ such that ${\textbf {b}}(\varDelta )$ is maximal, which means that, if for some $\varDelta _0 \in \sum _i \mathfrak {m}_i$ with ${\textbf {b}}(\varDelta _0)$ lying in the same cuspidal line with ${\textbf {b}}(\varDelta )$, then ${\textbf {b}}(\varDelta _0) \leqslant {\textbf {b}}(\varDelta )$. As these $\mathfrak {m}_i$’s are ladders, one has $\varDelta \in \{\varDelta _{1,1},\ldots ,\varDelta _{r,1} \}$. Also, one has $\varDelta \in \{\varDelta '_{1,1}, \ldots , \varDelta '_{s,1} \}$. So the lemma follows. $\square$

It will turns out that the ordering of representations in a product is important for the geometric lemma approach to distinction problems. The commutativity of a product of two ladder representations was studied by Lapid and Mínguez in [18]. Here we present a special case of their results that is sufficient for our purpose.

Lemma 6.10

Let $\rho \in {\mathscr {C}}$. Let $\mathfrak {m}_1, \mathfrak {m}_2 \in \mathscr {O}_{\rho }$ be two ladders, with $\mathfrak {m}_1 = \{\varDelta _{1,1},\ldots , \varDelta _{1,t_1} \}$ and $\mathfrak {m}_2 = \{\varDelta _{2,1},\ldots , \varDelta _{2,t_2} \}$. Suppose that $L (\mathfrak {m}_1)$ is an essentially Speh representation and $L (\mathfrak {m}_2)$ is a right aligned representation. If ${\textbf {e}}(\varDelta _{1,t_1}) = {\textbf {e}}(\varDelta _{2,t_2})$ and $t_2 \leqslant t_1$, or ${\textbf {e}}(\varDelta _{1,t_1}) = {\textbf {e}}(\varDelta _{2,t_2})$ and ${\textbf {b}}(\varDelta _{1,t_1}) \leqslant {\textbf {b}}(\varDelta _{2,t_2})$, then $L (\mathfrak {m}_1) \times L (\mathfrak {m}_2)$ is irreducible and $L (\mathfrak {m}_1) \times L (\mathfrak {m}_2) = L (\mathfrak {m}_2) \times L (\mathfrak {m}_1)$.

Proof

Note that the results in [18] are expressed in terms of Zelevinsky classification. By the combinatorial description of Zelevinsky involution by Moeglin-Waldspurger [25] (see also [17, Sect.  3.2]), we can rewrite the conditions in the lemma in terms of the Zelevinsky involution $\mathfrak {m}_1^t$ and $\mathfrak {m}_2^t$ of $\mathfrak {m}_1$ and $\mathfrak {m}_2$. The assertion then follows from Proposition 6.20 and Lemma 6.21 in [18]. $\square$

6.3 Distinction of essentially Speh representations

From now on, we shall perform some detailed analysis using our consequences of the geometric lemma.

Proposition 6.11

Let $\pi$ be an essentially Speh representation of $G_n$. If $\pi$ is $(H_{p,q},\mu _a)$-distinguished for two positive integers p, q with $p + q = n$ and some $a \in {\mathbb {R}}$, then $\pi$ is self-dual.

Proof

Write $\pi = L (\mathfrak {m})$ with $\mathfrak {m}= \{\varDelta _1,\ldots ,\varDelta _t \}$ a ladder. The case where $\pi$ is one dimensional is obvious. So we assume that $\pi$, hence $\varDelta _t$, is not one dimensional. By assumption, $\varDelta _1 \times \cdots \times \varDelta _t$ is $(H_{p,q},\mu _a)$-distinguished. By Corollary 5.9, there exists i, $1 \leqslant i \leqslant t$, such that

$$\begin{aligned} {\textbf {b}}(\varDelta _i) \cong {\textbf {e}}(\varDelta _t)^{\vee }. \end{aligned}$$

(6.5)

We claim that $i = 1$. If so, by Lemma 6.1, we see that $\pi$ is self-dual. In fact, if otherwise $i > 1$, we apply Proposition 6.8 to $\pi _1 \times \pi _2$, where $\pi _1 = \varDelta _1 \times \cdots \times \varDelta _{i-1}$ and $\pi _2 = \varDelta _i \times \cdots \times \varDelta _t$. We get either that

$$\begin{aligned} {\textbf {b}}(\varDelta _j) \cong {\textbf {e}}(\varDelta _k)^{\vee } \end{aligned}$$

(6.6)

for some j, $1 \leqslant j \leqslant i-1$ and some k, $i \leqslant k \leqslant t$, or that $\pi _1$ is $(H_{p_1,q_1},\mu _{a_1})$-distinguished with some $p_1,q_1$ and $a_1$, which implies, using Corollary 5.9 again, that

$$\begin{aligned} {\textbf {b}}(\varDelta _l) \cong {\textbf {e}}(\varDelta _{i-1})^{\vee } \end{aligned}$$

(6.7)

for some $l, 1\leqslant l \leqslant i-1$. But we see easily that both (6.6) and (6.7) contradict with (6.5). $\square$

Corollary 6.12

Let $\pi$ be an essentially Speh representation of $G_n$. If the representation $\pi$ is $(H_{p,q},\mu _a)$-distinguished for two positive integers p, q with $p + q = n$ and some $a \in {\mathbb {R}}$, $a \ne 0$, then $p = q$.

Proof

This follows from Proposition 6.11 and consideration of the central character of $\pi$. $\square$

Now we are in a position to prove one direction of Theorem 1.1 (what we actualy prove is slightly more). The arguments involve an application of the theory of Bernstein-Zelevinsky derivatives.

Proposition 6.13

Let $\pi = \text {Sp}(\varDelta ,l)$ be an essentially Speh representation of $G_n$, where $\varDelta$ is an essentially square-integrable representaion of $G_d$, $d > 1$, and l is a positive integer. Assume that $\pi$ is $H_{p,q}$-distinguished or $(H_{p,q},\mu _{-1/2})$ for two positive integers p, q, $p + q = n$. Then the degree d of $\varDelta$ is even, and $\varDelta$ is $H_{d/2,d/2}$-distinguished; also one has $p = q$.

Proof

We prove this by induction on l. The case $l = 1$ follows from Proposition 3.9. Suppose that $\pi$ is $(H_{p,q},\mu _a)$-distinguished with $a = 0$ or $-1/2$. By Proposition 6.11 we know that $\pi$ is self-dual, hence $\varDelta$ is also self-dual. Note that $\pi$ is irreducible. By Lemma 3.1, we may assume that $p \geqslant q$. By the assumption on $\pi$, we have

$$\begin{aligned} \mathrm {Hom}_{P_n \cap H_{p,q}} (\pi |_{P_n}, \mu _a) \ne 0, \end{aligned}$$

where $a = 0$ or $-1/2$. By [1, Sect.  3.5], the restriction $\pi |_{P_n}$ of $\pi$ to $P_n$ has a filtration which has composition factors $(\varPhi ^+)^{i-1}\varPsi ^+ (\pi ^{(i)})$, $i = 1,\ldots , h$, where $\pi ^{(h)}$ is the highest derivative of $\pi$. We first analyze linear functionals on these factor sapces using the theory of Bernstein-Zelevinsky derivatives.

• (1) When $i = 2k$ is even. If $q > k$ and $p > k-1$, by applying (3.3) repeatly, we have

  $$\begin{aligned} \mathrm {Hom}_{P_n \cap H_{p,q}} ((\varPhi ^+)^{i-1}\varPsi ^+ (\pi ^{(i)}), \mu _a)&\cong \mathrm {Hom}_{P_{n-i+1} \cap H_{q-k,p-k+1}}(\varPsi ^+(\pi ^{(i)}),\mu _{-a-1/2}) \nonumber \\&\cong \mathrm {Hom}_{ H_{q-k,p-k}} (\nu ^{1/2}\pi ^{(i)},\mu _{-a-1/2}). \end{aligned}$$

  (6.8)

  Otherwise, there exists $i_0 \geqslant 0$ such that

  $$\begin{aligned} \mathrm {Hom}_{P_n \cap H_{p,q}} ((\varPhi ^+)^{i-1}\varPsi ^+ (\pi ^{(i)}), \mu _a)&\cong \mathrm {Hom}_{P_{n-i+i_0+1} }((\varPhi ^+)^{i_0}\varPsi ^+(\pi ^{(i)}),\mu _{a'}), \end{aligned}$$

  (6.9)

  where $a' = a$ or $-a - 1/2$ depending on $i_0$ odd or even.

• (2) When $i = 2k + 1$ is odd. If $q > k$ and $p > k$, by applying (3.3) repeatly, we have

  $$\begin{aligned} \mathrm {Hom}_{P_n \cap H_{p,q}} ((\varPhi ^+)^{i-1}\varPsi ^+ (\pi ^{(i)}), \mu _a)&\cong \mathrm {Hom}_{P_{n-i+1} \cap H_{p-k,q-k}}(\varPsi ^+(\pi ^{(i)}),\mu _{a})\nonumber \\&\cong \mathrm {Hom}_{H_{p-k,q-k-1}} (\nu ^{1/2}\pi ^{(i)},\mu _{a}). \end{aligned}$$

  (6.10)

  Otherwise, there exists $i_0 \geqslant 0$ such that

  $$\begin{aligned} \mathrm {Hom}_{P_n \cap H_{p,q}} ((\varPhi ^+)^{i-1}\varPsi ^+ (\pi ^{(i)}), \mu _a)&\cong \mathrm {Hom}_{P_{n-i+i_0+1} }((\varPhi ^+)^{i_0}\varPsi ^+(\pi ^{(i)}),\mu _{a'}), \end{aligned}$$

  (6.11)

  where $a' = a$ or $-a - 1/2$ depending on $i_0$ even or odd.

We claim that the factor spaces corresponding to non-highest derivatives contribute nothing, that is, we have

$$\begin{aligned} \mathrm {Hom}_{P_n \cap H_{p,q}} ( (\varPhi ^+)^{i - 1} \varPsi ^+ (\pi ^{(i)}), \mu _a) = 0,\quad \text {for all } 1 \leqslant i < h. \end{aligned}$$

(6.12)

We shall discuss separately according to i is even or odd, $a = 0$ or $-1/2$. Note first that, by Lemma 6.4, when $1 \leqslant i < h$, the i-th derivative $\pi ^{(i)}$ is either 0 or a ladder representation of the form

$$\begin{aligned} L (\varDelta _1 \times \nu ^{(l-3)/2}\varDelta \times \cdots \times \nu ^{(1-l)/2}\varDelta ), \end{aligned}$$

(6.13)

where $\varDelta _1$ is a subsegment of $\nu ^{(l-1)/2}\varDelta$ obtained by discarding the first few terms. In particular, $\pi ^{(i)}$ is either 0 or an irreducible representation. Thus, if we are in the case where (6.9) or (6.11) holds, then

$$\begin{aligned} \mathrm {Hom}_{P_n \cap H_{p,q}} ((\varPhi ^+)^{i-1}\varPsi ^+ (\pi ^{(i)}), \mu _a)&\cong \mathrm {Hom}_{P_{n-i+i_0+1} }((\varPhi ^+)^{i_0}\varPsi ^+(\pi ^{(i)}),\mu _{a'}) \\&= 0, \end{aligned}$$

as the representation $(\varPhi ^+)^{i_0}\varPsi ^+(\pi ^{(i)})$ is either 0 or an irreducible representation of $P_{n-i+i_0+1}$ that is not one dimensional by [1, 3.3 Remarks].

Now we deal with the case where (6.8) or (6.10) holds. Note that, from (6.13), $\nu ^{1/2}\pi ^{(i)}$ either is 0 or can be realized as the unique irreducible quotient of a representation of the form $\nu ^{1/2}\varDelta _1 \times \text {Sp}(\varDelta ,l-1)$ with $\varDelta _1$ as above. We discuss as follows.

Case (1) where $a = 0$ and $i = 2k$ is even. By (6.8), it suffices to show that

$$\begin{aligned} \mathrm {Hom}_{ H_{q-k,p-k}}( \nu ^{1/2}\varDelta _1 \times \text {Sp}(\varDelta ,l-1), \mu _{-1/2}) = 0. \end{aligned}$$

(6.14)

Assume, on the contrary, that $\nu ^{1/2}\varDelta _1 \times \text {Sp}(\varDelta ,l-1)$ is $(H_{q-k,p-k},\mu _{-1/2})$-distinguished. As $\varDelta$ is self-dual, ${\textbf {e}}(\text {Sp}(\varDelta ,l-1))^{\vee } = {\textbf {b}}(\text {Sp}(\varDelta ,l-1)) \ne {\textbf {b}}(\nu ^{1/2}\varDelta _1)$. So, by Proposition 6.8,

$$\begin{aligned} \nu ^{1/2}\varDelta _1 \text { is} (H_{r,s},\mu _{(s - p- r + q-1)/2})\text { -distinguished } \end{aligned}$$

and

$$\begin{aligned} \text {Sp}(\varDelta ,l-1) \text {is} (H_{q - k -r, p - k -s},\mu _{(s - r -1)/2})\text {-distinguished} \end{aligned}$$

for some nonnegative integers r and s. If the degree of $\nu ^{1/2}\varDelta _1$ is greater than 1, then $\nu ^{1/2}\varDelta _1$ is self-dual by Proposition 3.9. This is absurd because the central character of $\nu ^{1/2}\varDelta _1$ has positive real part; If the degree of $\nu ^{1/2}\varDelta _1$ is 1, then $(r,s) = (1,0)$ or (0, 1). If $r =1$ and $s = 0$, then $\text {Sp}(\varDelta , l -1)$ is $(H_{q-k-1,p-k},\mu _{-1})$-distinguished. Thus we have $p = q - 1$ by Corollary 6.12. This is absurd as we have assumed that $p \geqslant q$; If $r = 0$ and $s = 1$, then $\nu ^{1/2}\varDelta _1$ is the character $\nu ^{(p-q)/2}$ of $G_1$ and $\text {Sp}(\varDelta ,l-1)$ is $(H_{q-k,p-k-1},{\textbf {1}})$-distinguished. So, by induction hypothesis, we have $p - 1 = q$. This implies that ${\textbf {e}}(\nu ^{(l-1)/2}\varDelta ) = {\textbf {e}}(\varDelta _1) = {\textbf {1}}$, the trivial character of $G_1$. This is impossible as $\varDelta$ is self-dual and its degree d is greater than 1.

Case (2) where $a = 0$ and $i = 2k+1$ is odd. In this case we see easily that

$$\begin{aligned} \mathrm {Hom}_{H_{p-k,q-k-1}} (\nu ^{1/2}\pi ^{(i)},{\textbf {1}}) = 0, \end{aligned}$$

(6.15)

as the central character of $\nu ^{1/2}\pi ^{(i)}$ has positive real part when $i < h$.

The arguments for the remaining two cases where $a = -1/2$, i is even or odd are similar to those of the above two cases and are omitted here. So we have proved (6.12).

By Lemma 6.4, we know that the highest derivative of $\pi$ is $\pi ^{(d)}$ and $\nu ^{1/2}\pi ^{(d)} = \text {Sp}(\varDelta ,l-1)$. Now we have

$$\begin{aligned} \mathrm {Hom}_{P_n \cap H_{p,q}} ( (\varPhi ^+)^{d-1} \varPsi ^+ (\pi ^{(d)}), \mu _a) \ne 0, \end{aligned}$$

(6.16)

where $a = 0$ or $-1/2$. We analyze the left hand side of (6.16) as above. The cases (6.9) and (6.11) cannot happen by the same arguments as above. The case (6.10) cannot happen by induction hypothesis and the fact that $p \geqslant q$. So the only possible case is when (6.8) holds, that is, d is even and

$$\begin{aligned} \mathrm {Hom}_{P_n \cap H_{p,q}} ((\varPhi ^+)^{i-1}\varPsi ^+ (\pi ^{(d)}), \mu _a) \cong \mathrm {Hom}_{ H_{q-k,p-k}} (\text {Sp}(\varDelta ,l-1),\mu _{-a-1/2}). \end{aligned}$$

Note that when $a = 0$ or $-1/2$, $-a-1/2 = -1/2$ or 0. Thus we are done by induction hypothesis. $\square$

We have the following generalization of Corollary 3.8 to essentially Speh representations.

Corollary 6.14

Let $\pi$ be an essentially Speh representation of $G_n$ that is not one dimensional. If $\pi$ is $(H_{p,q},\mu _a)$-distinguished for two positive integers p, q with $p + q = n$ and $a \in {\mathbb {R}}$, then we have $p = q$.

Proof

The case $a \ne 0$ is Corollary 6.12. The case $a = 0$ follows from Proposition 6.13. $\square$

Remark 6.15

We postpone the proof of the other direction of Theorem 1.1 in Sect. 7.1.

6.4 Distinguished left aligned representations

The results of this subsection are used only in Sect. 7.2 where we classify distinguished representations that are products of Speh representations. The analysis in this subsection is quite involved; the readers can skip it for the fisrt reading.

The purpose of this subsection is to show the following

Proposition 6.16

Let $\pi$ be a left aligned (resp. right aligned) representation of $G_n$. If $\pi$ is $(H_{p,q},\mu _{(p-q)/2})$ (resp. $(H_{p,q},\mu _{(q-p)/2})$)-distinguished for two nonnegative integers p, q, $p + q =n$, then $\pi$ is an essentially Speh representation.

We need the following technical lemmas. When the supercuspidal representations in the support of the left aligned representation have degree greater than 1, we can prove slightly more.

Lemma 6.17

Let $\rho \in {\mathscr {C}}(G_d)$, $d > 1$, and $\mathfrak {m} = \{\varDelta _1, \ldots , \varDelta _t \} \in {\mathscr {O}}_{\rho }$ be a ladder. Assume that $\pi = L (\mathfrak {m})$ is a decreasing or an increasing ladder representation of $G_n$. If $\pi$ is $(H_{p,q},\mu _a)$-distinguished for two positive integers p, q, $p + q =n$ and some $a \in {\mathbb {R}}$, then all the $l(\varDelta _i)$’s are the same. Moreover, $\pi$ is self-dual.

Proof

Note that $\pi$ is irreducible. By Lemma 3.1, passing to contragradient if necessary, we may assume that $l(\varDelta _1) \leqslant l(\varDelta _2) \leqslant \cdots \leqslant l(\varDelta _t)$.By our assumption, the representation $\varDelta _1 \times \cdots \times \varDelta _t$ is $(H_{p,q},\mu _a)$-distinguished. We now appeal to Proposition 5.8. Write $\varDelta _i = \varDelta ([a_i,b_i]_{\rho })$, $i = 1, 2, \ldots ,t$. Note that by our assumption that $d > 1$, Case A1 and Case B1 cannot happen.

Case A2. In this case We have $a_t \leqslant c_t < b_t$, and $\varDelta ( [a_t,c_t]_{\rho })$ is self-dual. Thus we have $\nu ^{a_t} \rho \cong \nu ^{-c_t} \rho ^{\vee }$, and consequently $(a_t + c_t)d + 2 \Re (w_{\rho }) = 0$. We also have $\varDelta ( [c_t+1,b_t]_{\rho } )^{\vee } \cong \varDelta ([a_i,c_i]_{\rho })$ for some $i < t$ and $c_i \geqslant a_i$. Thus we get $\nu ^{a_i}\rho \cong \nu ^{-b_t}\rho ^{\vee }$, and then $(a_i + b_t)d + 2 \Re (w_{\rho }) = 0$. But this is absurd because $a_i > a_t$ and $b_t > c_t$.

Case B2. In this case we have $c_t = b_t$, and $\varDelta ( [a_t,b_t]_{\rho })$ is self-dual. Thus we have $\nu ^{a_t} \rho \cong \nu ^{-b_t} \rho ^{\vee }$, and consequently $(a_t + b_t)d + 2 \Re (w_{\rho }) = 0$. We also have $\varDelta _1 \times \cdots \times \varDelta _{t-1}$ is $(H_{p',q'},\mu _{a'})$-distinguished for some $p'$, $q'$ and $a'$. If $t = 1$, there is nothing to be proved. If $t > 1$, by Corollay 5.9, we get that $(\nu ^{b_{t-1}}\rho )^{\vee } \cong \nu ^{a_i}\rho$ for some $1 \leqslant i \leqslant t-1$. Thus we get $(a_i + b_{t-1})d + 2 \Re (w_{\rho }) = 0$. This is absurd because $a_i > a_t$ and $b_{t-1} > b_t$.

So the only possible case is Case C. We then have $\varDelta ([a_t,b_t])^{\vee } \cong \varDelta ([a_i,c_i] )$ for $i < t$ and certain $a_i \leqslant c_i \leqslant b_i$. Note that, by our assumption, we have $l(\varDelta _i) \leqslant l(\varDelta _t)$. Thus we have $l(\varDelta _i) = l(\varDelta _t)$. We claim that $i = 1$. If so, all $l(\varDelta _i)$’s will be the same by our assumption. Indeed, if $i > 1$, consider the $(H_{p,q},\mu _a)$-distinguished representaion

$$\begin{aligned} (\varDelta _1 \times \cdots \times \varDelta _{i-1}) \times (\varDelta _i \times \cdots \times \varDelta _t). \end{aligned}$$

By Propostion 6.8, either we have ${\textbf {e}}(\varDelta _{i-1})^{\vee } \cong {\textbf {b}}(\varDelta _a)$ with $1 \leqslant a \leqslant t-1$, or we have ${\textbf {e}}(\varDelta _c)^{\vee } \cong {\textbf {b}}(\varDelta _b)$ with $1 \leqslant b \leqslant t-1$ and $i \leqslant c \leqslant t$. We then get a contradiction as in Case A2 or B2. The assertion on the self-dualness property follows from a repeated analysis as above. $\square$

If we drop the assumption that $d > 1$, the argument becomes complicated by the possible occurrence of Case A1 or Case B1 when applying Proposition 5.8. We have the following result on the shape of right aligned representations when it is distinguished.

Lemma 6.18

Let $\rho$ be a character of $G_1$, and $\mathfrak {m} \in \mathscr {O}_{\rho }$ be a ladder. Assume that $\pi = L (\mathfrak {m})$ is a right aligned representation of $G_n$. If $\pi$ is $(H_{p,q},\mu _a)$-distinguished for two nonnegative integers p, q, $p + q =n$ and some $a \in {\mathbb {R}}$, then either

• (1) we have

  $$\begin{aligned} \mathfrak {m}= \{ \varDelta _1,\ldots ,\varDelta _{i_1}, \varDelta _{i_1 + 1}, \ldots , \varDelta _{i_1 + i_2}, \varDelta _{i_1 + i_2 +1}, \ldots ,\varDelta _{i_1 + i_2 +i_3} \} \end{aligned}$$

  (6.17)

  with $i_1$, $i_2$ and $i_3 \geqslant 0$, such that $l(\varDelta _k) = 1$ when $1\leqslant k \leqslant i_1$, $l(\varDelta _{i_1 + k}) = l > 1$ when $1 \leqslant k \leqslant i_2$, $l(\varDelta _{i_1 + i_2 + k})) = l + 1$ when $1 \leqslant k\leqslant i_2$, and that ${\textbf {e}}(\varDelta _{i_1 + i_2 + i_3})^{\vee } \cong {\textbf {b}}(\varDelta _{i_1 + 1})$ (See Fig. 1 for an example), or

• (2) we have

  $$\begin{aligned} \mathfrak {m}= \{\varDelta _1,\ldots , \varDelta _{i_1}, \varDelta _{i_1 + 1}, \ldots ,\varDelta _{i_1 + i_2} \} \end{aligned}$$

  (6.18)

  with $i_1$ and $i_2 >0$, such that $l(\varDelta _k) = 1$ when $1 \leqslant k \leqslant i_1$, $l(\varDelta _{i_1 + k}) = 2$ when $1 \leqslant k \leqslant i_2$, and that ${\textbf {e}}(\varDelta _{i_1 + i_2})^{\vee } \cong {\textbf {b}}(\varDelta _1)$ (See Figure 2 for an example).

Proof

Write $\mathfrak {m}= \{\varDelta _1,\ldots ,\varDelta _t \}$. If $l(\varDelta _t) =1$, then $\pi$ is a one dimensional representation and $\mathfrak {m}$ is of the form (6.17) with $i_2 = i_3 = 0$. If $l(\varDelta _t) = l(\varDelta _1) =2$, then $\pi$ is an essentially Speh representation. It follows from Proposition 6.11 that $\mathfrak {m}$ is of the form (6.17) with $i_1 = i_3 = 0$. If $l(\varDelta _t) = 2$ and $l(\varDelta _1) = 1$, then $\pi$ can be realized as the unique irreducible quotient of $\pi _1 \times \pi _2$, where $\pi _1$ is a one dimensional representation and $\pi _2$ is an essentially Speh representation of length 2. Thus $\pi _1 \times \pi _2$ is $(H_{p,q},\mu _a)$-distinguished. By Proposition 6.6, $\mathfrak {m}$ is either of the form (6.18) (Case A), or of the form (6.17) with $i_3 = 0$, $i_1 > 0$, $i_2 > 0$ and $l = 2$ (Case B and Proposition 6.11). Note that here Case C is impossible by our assumption on $\pi _1$ and $\pi _2$. If $l(\varDelta _t) > 2$, then we apply Proposition 5.8 to the product $\varDelta _1 \times \cdots \times \varDelta _t$ and discuss case by case. Note first that Case A2 cannot happen by similar arguments as those in Lemma 6.17; Case B1 cannot happen by our assumption on $\varDelta _t$. In the remaining cases, it follows from Corollary 5.9, Proposition 6.8 and arguments similar to those in Lemma 6.17 that $\mathfrak {m}$ is of the form (6.17). $\square$

Fig. 1

An example of a ladder of the form (6.17) with $i_1 = i_2 = 1$ and $i_3 = 2$

Fig. 2

An example of a ladder of the form (6.18) with $i_1 = 2$ and $i_2 = 2$

The following lemma is a simple consequence of Lemma 3.2.

Lemma 6.19

Let $\pi$ be a one dimensional representation of $G_n$. If $\pi$ is $(H_{p,q},\mu _{(q-p)/2})$-distinguished with $p + q = n$, then $\pi$ is either the trivial character ${\textbf {1}}$ of $G_n$ or the character $\nu ^{-n/2}$ of $G_n$. In particular, ${\textbf {b}}(\pi )$ is either $\nu ^{(n-1)/2}$ or $\nu ^{-1/2}$ of $G_1$.

As shown in Lemma 6.18, there are two possibilities for the shape of distinguished right aligned representations. Now we remove one possibility if we impose some restriction on the subscripts (p, q, a).

Lemma 6.20

Keep the notation as in Lemma 6.18, let $\pi = L (\mathfrak {m})$ with $\mathfrak {m}$ of the form (6.18). Then $\pi$ cannot be $(H_{p,q},\mu _{(q-p)/2})$-distinguished.

Proof

We assume on the contrary that $\pi$ is $(H_{p,q},\mu _{(q-p)/2})$-distinguished. By part (1) of Lemma 3.1, we may assume that $p \leqslant q$. Note that $\pi$ can be realized as the unique quotient of $\pi _1 \times \pi _2$, where $\pi _1$ is a one dimensional representation, $\pi _2$ is an essentially Speh representation of length 2, and ${\textbf {e}}(\pi _2)^{\vee } \cong {\textbf {b}}(\pi _1)$. Thus, $\pi _1 \times \pi _2$ is $(H_{p,q},\mu _{(q-p)/2})$-disintuished. By Proposition 6.6, there exist divisions of $\pi _1$ and $\pi _2$, $\pi _1 = \pi _1' \sqcup \pi _1''$ and $\pi _2 = \pi _2' \sqcup \pi _2''$ respectively, such that, among other things, $\pi _1''$ is $(H_{r,s},\mu _{(s-r)/2})$-distinguished for two nonnegative integers r and s. Note that $\pi _1'$ is not the trivial representation of $G_0$ by our assumption on $\pi$ and Proposition 6.11. We shall discuss further according to the values of r and s.

• (1) If exactly one of r and s is 0, then $\pi _1''$ is the character $\nu ^{-n_1''/2}$ of $G_{n_1''}$. Thus ${\textbf {b}}(\pi _1'') = \nu ^{-1/2}$, ${\textbf {e}}(\pi _1') = \nu ^{1/2}$. By Proposition 6.6, we also have $\pi _1'^{\vee } \cong \pi _2''$. Thus ${\textbf {b}}(\pi _2'') = {\textbf {e}}(\pi _1')^{\vee } = \nu ^{-1/2} = {\textbf {b}}(\pi _1'')$, which is absurd.

• (2) If $r > 0$ and $s > 0$, then $\pi _1''$ is the character ${\textbf {1}}$ of $G_{2r}$, that is, ${\textbf {b}}(\pi _1'') = \nu ^{r - 1/2}$ and ${\textbf {e}}(\pi _1'') = \nu ^{-r + 1/2}$. So, ${\textbf {e}}(\pi _1') = {\textbf {b}}(\pi _1'') + 1 = \nu ^{r+1/2}$. By Proposition 6.6, we have $\pi _1'^{\vee } \cong \pi _2''$. So ${\textbf {b}}(\pi _2'') = \nu ^{-r-1/2}$. By our assumption on the shape of $\mathfrak {m}$, this implies that $\pi _2'$ is also a one dimensional representation which, by Proposition 6.6, is $(H_{r',s'},\mu _{(r' -s')/2+q - p})$-distinguished for certain nonnegative integers $r'$ and $s'$ and that ${\textbf {b}}(\pi _2') = {\textbf {b}}(\pi _2'') - 1 = \nu ^{-r-3/2}$. One of $r'$ and $s'$ has to be 0. Recall that we have assumed that $p \leqslant q$. We then see easily that $\pi _2'$ is the character $\nu ^{p-q + n' /2}$ of ${\mathrm {GL}}_{n'}$. Note that we have an equality of central characters, $\omega _{\pi _2'} = \omega _{\pi }$. This implies that

  $$\begin{aligned} n'(p-q + n' /2) = - (p-q)^2 / 2. \end{aligned}$$

  So, $n' = q - p$ and ${\textbf {b}}(\pi _2') = \nu ^{-1/2}$. This is absurd as we have shown that ${\textbf {b}}(\pi _2') = \nu ^{-r - 3/2}$ with r a positive integer.

• (3) If $r = s =0$, we have two subcases according to whether or not $\pi _2'$ is a one dimensional representaiton. If it is, we get a contradiction by exactly the same arguments as in (2) with r being repalced by 0. If it is not, it follows from the duality relations in Lemma 6.18, applied to the contragradient of $\pi _2'$, that ${\textbf {b}}(\pi _2')^{\vee }$ is either ${\textbf {b}}(\pi _2'') + 1$ or ${\textbf {e}}(\pi _2'') - 1$. But, note that ${\textbf {b}}(\pi _2') = {\textbf {e}}(\pi _1') - 2$. It follows from the relation $\pi _1'^{\vee } \cong \pi _2''$ that ${\textbf {b}}(\pi _2') ^{\vee } = {\textbf {b}}(\pi _2'') + 2$. This is absurd as $\pi _2''$ is one dimensional and ${\textbf {b}}(\pi _2'') \geqslant {\textbf {e}}(\pi _2'')$.

$\square$

Proof of Proposition 6.16

By part (2) of Lemma 3.1, we only need to prove the statement for left aligned representations. We may further assume that $p \leqslant q$ by part (1) of Lemma 3.1. By Lemma 6.17, we may write $\pi = L (\mathfrak {m})$ with $\mathfrak {m} \in {\mathscr {O}}_{\rho }$ a ladder and $\rho$ a character of $G_1$. By considering the contragredient $\pi ^{\vee } = L (\mathfrak {m}^{\vee })$, we see from Lemma 6.20 that $\mathfrak {m}^{\vee }$ is of the form (6.17). So, we may write

$$\begin{aligned} \mathfrak {m}= \{ \varDelta _1,\ldots ,\varDelta _{i_1}, \varDelta _{i_1 + 1}, \ldots , \varDelta _{i_1 + i_2}, \varDelta _{i_1 + i_2 +1}, \ldots ,\varDelta _{i_1 + i_2 +i_3} \} \end{aligned}$$

with $i_1$, $i_2$ and $i_3 \geqslant 0$, such that $l(\varDelta _k) = l_1 > 2$ when $1\leqslant k \leqslant i_1$, $l(\varDelta _{i_1 + k}) = l_1 -1$ when $1 \leqslant k \leqslant i_2$, $l(\varDelta _{i_1 + i_2 + k})) = 1$ when $1 \leqslant k\leqslant i_3$, and that ${\textbf {e}}(\varDelta _{i_1 + i_2 })^{\vee } \cong {\textbf {b}}(\varDelta _1)$.

We may as well assume that $i_1$ and $i_2$ are not all zero. Our first step is to show that $i_3 = 0$. If not so, we realize $\pi$, in the obvious way, as the unique irreducible quotient of $\pi _1 \times \pi _2 \times \pi _3$ with $\pi _i$ an essentially Speh representation for each i, such that $\pi _3$ is a character of $G_{n_3}$, $n_3 > 0$, and that at least one of $\pi _1$ and $\pi _2$ is not the trivial representation of $G_0$. By our assumption on $\pi$, the representation $\pi _1 \times \pi _2 \times \pi _3$ is $(H_{p,q},\mu _{(p-q)/2})$-distinguished. Note that as $i_3 > 0$, ${\textbf {e}}(\pi _3)$ is not dual to ${\textbf {b}}(\pi _1)$ or ${\textbf {b}}(\pi _2)$. So, by Proposition 6.8, $\pi _3$ is $(H_{r,s},\mu _{(r-s)/2})$-distinguished with respect to two nonnegative integers r and s. As $\pi _3$ is one dimensional, $\pi _3$ is either the trivial represntation ${\textbf {1}}$ of $G_{n_3}$ or the character $\nu ^{n_3/2}$ of $G_{n_3}$. In particular, ${\textbf {b}}(\varDelta _{i_1 + i_2 + 1}) = \nu ^{(n_3 - 1)/2}$ or $\nu ^{n_3 - 1/2}$. But this will contradict with the fact that ${\textbf {e}}(\varDelta _{i_1 + i_2 })^{\vee } \cong {\textbf {b}}(\varDelta _1)$.

Our next step is to show that $i_1 = 0$ or $i_2 = 0$. Assume on the contrary that $i_1 > 0$ and $i_2 > 0$. By our assumption on $\pi$, the representation $\pi ^{\vee } = L (\mathfrak {m}^{\vee })$ is $(H_{p,q},\mu _{(q-p)/2})$-distinguished. Thus, the representation

$$\begin{aligned} \varDelta _{i_1 + i_2}^{\vee } \times \cdots \times \varDelta _{i_1 + 1}^{\vee } \times \varDelta _{i_1}^{\vee } \times \cdots \times \varDelta _1^{\vee } \end{aligned}$$

is $(H_{p,q},\mu _{(q-p)/2})$-distinguished. By Proposition 5.8, we deduce that ${\textbf {b}}(\varDelta _1^{\vee }) \cong {\textbf {e}}(\varDelta _1)^{\vee }$ is the character $\nu ^{q-p + 1/2}$ or $\nu ^{p-q + 1/2}$ of $G_1$. (This is the consequence of Case A1; Case A2 and Case B2 are eliminated by arguments similar to those in Lemma 6.17; Case B1 and Case C are eliminated by our assumptions.) It follows easily from the condition ${\textbf {b}}(\varDelta _1) \cong {\textbf {e}}(\varDelta _{i_1 + i_2})^{\vee }$ and the assumption $p \leqslant q$ that ${\textbf {e}}(\varDelta _1)^{\vee } = \nu ^{p -q + 1/2}$. Hence we have ${\textbf {e}}(\varDelta _1) = \nu ^{q - p - 1/2}$.

We show that $i_1 = q - p$ and ${\textbf {e}}(\varDelta _{i_1}) = \nu ^{1/2}$ by consideration on the central character of $\pi$. In fact, on the one hand, we see from the assumption on $\mathfrak {m}$ and the fact ${\textbf {e}}(\varDelta _1) = \nu ^{ q - p -1/2}$ that the central character $w_{\pi }$ of $\pi$ is $\nu ^a$ where $a =(q-p)i_1 - i_1^2/2$; on the othe hand, as $\pi$ is $(H_{p,q},\mu _{(p-q)/2})$-distinguished, we have $w_{\pi } = \nu ^{a'}$ where $a' = (q-p)^2 /2$. Thus the assertion follows. Also, from the fact that ${\textbf {e}}(\varDelta _{i_1 + i_2 })^{\vee } \cong {\textbf {b}}(\varDelta _1)$, we get that ${\textbf {b}}(\varDelta _1) = \nu ^{i_2 + 1/2}$. Thus, $l(\varDelta _1) = q - p -i_2 >2$, in particular $i_1 > i_2$.

Now, as in the first step, we have that $\pi _1 \times \pi _2$ is $(H_{p,q},\mu _{(p-q)/2})$-distinguished, where $\pi _1 = L (\varDelta _1,\ldots ,\varDelta _{i_1})$ and $\pi _2 = L (\varDelta _{i_1 + 1}, \ldots , \varDelta _{i_1 + i_2})$. We appeal to Proposition 6.6, and claim that Case A and Case B cannot happen. In fact, if Case A or Case B happens, there will be a division of $\pi _2$ as $\pi _2'$ and $\pi _2''$, where $\pi _2'$ is not the trivial representation of $G_0$, such that $\pi _2'$ is $(H_{r,s},\mu _{(r-s)/2})$-distinguished for two nonnegative integers r and s. In particular, the central character $w_{\pi _2'}$ of $\pi _2'$ has nonnegative real part. But this will contradict with the fact that ${\textbf {e}}(\varDelta _{i_1 + 1}) = \nu ^{-3/2}$. So, there exists a division of $\pi _1$ as two ladder representations $\pi _1'$ and $\pi _1''$ such that $\pi _2 \cong \pi _1'^{\vee }$ and that $\pi _1''$ is $(H_{p-n_2,q-n_2},\mu _{(p-q)/2})$-distinguished. Note that $\pi _1''$ is a right aligned representation, and is not a one dimensional representation due to the fact that $i_1 > i_2$. By Lemma 6.18, we then get a contradiction as we can check easily that the ladder $\mathfrak {m}_1''$ of $\pi _1''$ is not of the form (6.17) or (6.18). $\square$

7 Distinction in the unitary dual

7.1 The case of Speh representations

We now classify distinguished Speh representations in terms of distinguished discrete series. In fact, we will do it for essentially Speh representations.

Theorem 7.1

Let $n = 2m$, and $\text {Sp}(\varDelta ,k)$ be an essentially Speh representation of $G_n$, where $\varDelta$ is an essentially square-integrable representation of $G_d$ with $d > 1$, and k is a positive integer. Then $\text {Sp}(\varDelta ,k)$ is $H_{m,m}$-distinguished if and only if d is even and $\varDelta$ is $H_{d/2,d/2}$-distinguished.

Proof

One direction has been proved in Proposition 6.13. We now assume that d is even and that $\varDelta$ is $H_{d/2,d/2}$-distinguished. By [26, Proposition 7.2], which is based on the work of Blanc and Delorme [2], the representation

$$\begin{aligned} \nu ^{(k-1)/2}\varDelta \times \nu ^{(k-3)/2} \varDelta \times \cdots \times \nu ^{(1-k)/2}\varDelta \end{aligned}$$

(7.1)

is $H_{m,m}$-distinguished. (The distinguishedness of $\varDelta$ is unnecessary when k is even).We have the following exact sequence of representations of $G_n$,

$$\begin{aligned} 0 \rightarrow \mathcal {K} \rightarrow \nu ^{(k-1)/2}\varDelta \times \nu ^{(k-3)/2} \varDelta \times \cdots \times \nu ^{(1-k)/2}\varDelta \rightarrow \text {Sp}(\varDelta ,k) \rightarrow 0, \end{aligned}$$

(7.2)

where the kernel $\mathcal {K} = \sum _{i=1}^{k-1} \mathcal {K}_i$ is given explicitly in Proposition 6.2. To show that $\text {Sp}(\varDelta ,k)$ is $H_{m,m}$-distinguished, it suffices to show that each $\mathcal {K}_i$ is not $H_{m,m}$-distinguished. Write the representation (7.1) as $\varDelta ([a_1,b_1]_{\rho }) \times \cdots \times \varDelta ([a_k,b_k]_{\rho })$, here the cuspidal representation $\rho$ is taken to be self-dual and thus $a_i$ and $b_i$, $i = 1, 2,\ldots ,k$ need not be integers. So we have

$$\begin{aligned}&a_{i+1} = a_i - 1, \quad b_{i+1} = b_i - 1,\ i = 1, \ldots , k-1\\&a_i + b_{k+1-i} = 0,\ i = 1, \ldots , k. \end{aligned}$$

We further omit the subscript $\rho$ in the sequel. Recall that, by Proposition 6.2,

$$\begin{aligned} \mathcal {K}_i = \varDelta ([a_1,b_1]) \times \cdots \times \varDelta ([a_{i+1},b_i]) \times \varDelta ([a_i,b_{i+1}]) \times \cdots \times \varDelta ([a_k,b_k]). \end{aligned}$$

If $i+1 \leqslant (k+1)/2$ and $\mathcal {K}_i$ is $H_{m,m}$-distinguished, by applying Proposition 5.8 repeatly, we get that $\varDelta ([a_{i+1},b_i]) \times \varDelta ([a_i,b_{i+1}]) \times \cdots \times \varDelta ([a_{k+1-i},b_{k+1-i}])$ is $H_{m',m'}$-distinguished for certain $m'$. (In each step, only Case C is possible.) When we apply Proposition 5.8 once again, still, only Case C is possible. But this is absurd as $l(\varDelta ([a_i,b_{i+1}])) < l(\varDelta )$. Similar arguments can show that $\mathcal {K}_i$ is not $H_{m,m}$-distinguished if $i \geqslant (k+1)/2$.

The remaining case is when k is even and $i = k/2$. In what follows, to save notation, we sometimes write H-distinguished for $H_{m',m'}$-distinguished when there is no need to address $m'$. If $\mathcal {K}_i$ is $H_{m,m}$-distinguished, by applying Proposition 5.8 repeatly, we get that $\varDelta ([a_{i+1},b_i]) \times \varDelta ([a_i,b_{i+1}])$ is H-distinguished. This in turn implies that both $\varDelta ([a_{i+1},b_i])$ and $\varDelta ([a_i,b_{i+1}])$ are H-distinguished by Proposition 6.8. Let us write $\varDelta = St (\rho ,l)$. Then by our assumption on i, we have $\varDelta ([a_i,b_{i+1}]) = St (\rho ,l-1)$ and $\varDelta ([a_{i+1},b_i]) = St (\rho ,l+1)$. By [20, Theorem 6.1], we can conclude that $St (\rho ,l)$ is H-distinguished if and only if $St (\rho ,l-1)$ (or $St (\rho ,l+1)$) is not H-distinguished. Actually, as $\rho$ is self-dual, the L-function $L(s,\phi (\rho )\otimes \phi (\rho ))$ has a simple pole at $s = 0$, where $\phi (\rho )$ is the Langlands parameter of $\rho$. By the factorization

$$\begin{aligned} L(s,\phi (\rho )\otimes \phi (\rho )) = L(s,\Lambda ^2 \circ \phi (\rho )) \cdot L(s,Sym ^2 \circ \phi (\rho )), \end{aligned}$$

we know that exactly one of the symmetric or exterior square L-factors of $\rho$ has a pole at $s=0$. The above conclusion then follows from [20, Theorem 6.1] where distinction of $St (\rho ,l)$ is related to the pole of symmetric or exterior square L-facotrs of $\rho$ according to l is even or odd. Thus by our assumption that $\varDelta$ is $H_{d/2,d/2}$-distingusihed, we get that $\mathcal {K}_i$ is not $H_{m,m}$-distinguished. So we are done. $\square$

7.2 The general case

We start with an auxiliary result, which is needed in one step of the proof of Theorem 7.3.

Lemma 7.2

Let $\pi = \pi _1 \times \cdots \times \pi _t$ be an irreducible unitary representation of $G_{2m}$ with each $\pi _i$ a Speh representation. Let h be a positive integer. Assume that, for all of those $\pi _i$ such that $\mathrm {supp}(\pi _i)$ is contained in the cuspidal line ${\mathbb {Z}}\nu ^{-1/2}$, we have ${\textbf {b}}(\pi _i) \leqslant \nu ^{h-1/2}$. If the representation $\pi \times \nu ^{-h/2}$ is $(H_{m,m+h},\mu _{h/2})$-distinguished, where $\nu ^{-h/2}$ is viewed as a representation of $G_h$, then $\pi$ is $H_{m,m}$-distinguished.

Proof

A crucial fact, on which we rely, is that $\pi$ is a commutative product of Speh representations. Our first step is to show that we can reduce the proposition to the case that for all i,

$$\begin{aligned} \text { the support } \mathrm {supp}(\pi _i) \text {is contained in} {\mathbb {Z}}\nu ^{-1/2} \text {and} {\textbf {b}}(\pi _i) = \nu ^{h-1/2}. \end{aligned}$$

(7.3)

Indeed, write $\varPi = \pi _1 \times \cdots \times \pi _r$ and $\varPi ' = \pi _{r+1} \times \cdots \times \pi _t \times \nu ^{-h/2}$ where, $\pi _j$’s, $r+1 \leqslant j \leqslant t$, are all the representations in the Tadić decomposition of $\pi$ that satisfy (7.3). So $\varPi \times \varPi '$ is $(H_{m,m+h},\mu _{h/2})$-distinguished. By Proposition 6.8, $\varPi$ is $(H_{m_1,m_1 + h_1},\mu _{h_1/2})$-distinguished and $\varPi '$ is $(H_{m-m_1,m-m_1+h-h_1},\mu _{(h+h_1)/2})$-distinguished for certain integers $m_1$ and $h_1$. Note that the central character of $\varPi$ has real part 0. We have $h_1 = 0$. So, $\varPi$ is $H_{m_1,m_1}$-distinguished and $\varPi '$ is $(H_{m-m_1,m-m_1+h},\mu _{h/2})$-distinguished. The reduction then follows from Lemma 5.7.

We thus assume that $\pi = \pi _1 \times \cdots \times \pi _t$ with ${\textbf {b}}(\pi _i) = \nu ^{h-1/2}$ for all i. Moreover, we arrange the ordering of $\pi _i$’s such that ${\textbf {ht}}(\pi _1) \geqslant \cdots \geqslant {\textbf {ht}}(\pi _t)$. We prove the lemma by induction on t.

As the representation $\pi \times \nu ^{-h/2}$ is $(H_{m,m+h},\mu _{h/2})$-distinguished, by Proposition 6.6, there exist two representations $\sigma '$ and $\sigma ''$ of dimension one, $\nu ^{-h/2} = \sigma ' \sqcup \sigma ''$, such that, among other things, $\sigma '$ is $(H_{a,b},\mu _{h+(a-b)/2})$-distinguished for two nonnegative integers a and b.

• (1). If $\sigma '$ is not the trivial representation of $G_0$, that is, a and b are not all zero, we have three cases. If $a > 0$ and $b > 0$, then by Lemma 3.2, $\sigma '$ must be the trivial representation ${\textbf {1}}$ of $G_{a+b}$. This is absurd as we have ${\textbf {b}}(\sigma ') = \nu ^{ - 1/2}$; If $a > 0$ and $b = 0$, then $\sigma '$ is the character $\nu ^{h+a/2}$ of $G_a$. Thus ${\textbf {b}}(\sigma ') = \nu ^{h+a -1/2}$ which is absurd; If $a =0$ and $b>0$, we see easily that $a = 0$ and $b= h$, that is, $\sigma '$ is the character $\nu ^{-h/2}$ of $G_h$. So, it follows from Case B of Proposition 6.6 that $\pi$ is $H_{m,m}$-distinguished.

• (2). If $\sigma '$ is the trivial representaion of $G_0$, then we are in Case C of Proposition 6.6.Hence there exists i, $1 \leqslant i \leqslant t$ and a division of $\pi _i$ as two ladder representations $\pi _i'$ and $\pi _i''$, $\pi _i = \pi _i' \sqcup \pi _i''$, such that $\pi _i'$ is the character $\nu ^{h/2}$ of $G_h$ and the representation

  $$\begin{aligned} \pi _1 \times \cdots \times \pi _{i-1} \times \pi _i'' \times \pi _{i+1} \times \cdots \times \pi _t \end{aligned}$$

  (7.4)

  is $(H_{m-h,m},\mu _{h/2})$-distinguished. We have two subcases. If $\pi _i$ is one dimensional, then $\pi _i$ must be the trivial representaion ${\textbf {1}}$ of $G_{2h}$ as ${\textbf {b}}(\pi _i) = \nu ^{h-1/2}$. Thus $\pi _i''$ is the character $\nu ^{-h/2}$ of $G_h$. By Lemma 6.10, $\nu ^{-h/2} \times \pi _j = \pi _j \times \nu ^{-h/2}$ for $j = 1,\ldots ,t$. So we move $\pi _i''$ to the end of the product (7.4) and get by induction hypothesis that

  $$\begin{aligned} \pi _1 \times \cdots \times \pi _{i-1} \times \pi _{i+1} \times \cdots \times \pi _k \end{aligned}$$

  is $H_{m-h,m-h}$-distinguished. Hence $\pi$ is $H_{m,m}$-distingusihed by Lemma 5.7. If otherwise $\pi _i$ is not one dimensional, we can also move $\pi _i''$ to the beginning of the product (7.4) by Lemma 6.10 and our ordering of $\pi _i$’s. By part (2) of Lemma 3.1,

  $$\begin{aligned} \pi _1 \times \cdots \times \pi _{i-1} \times \pi _{i+1} \times \cdots \times \pi _t \times (\pi _i'')^{\vee } \end{aligned}$$

  (7.5)

  is $(H_{m,m-h},\mu _{-h/2})$-distinguished. This is impossible by Proposition 6.8, and then we are done. Indeed, firstly, we can check easily that $\pi _i''$, hence its contragradient $(\pi _i'')^{\vee }$, cannot be $(H_{p_1,q_1},\mu _{a_1})$-distinguished for any $(p_1,q_1,a_1)$ by Lemma 6.18. Secondly, note that ${\textbf {e}}((\pi _i'')^{\vee }) = \nu ^{-h-1/2}$ is not dual to ${\textbf {b}}(\pi _i) = \nu ^{h-1/2}$ for all i.

$\square$

Theorem 7.3

Let $\pi$ be an irreducible unitary representation of $G_{2m}$ of Arthur type. Then $\pi$ is $H_{m,m}$-distinguished if and only if $\pi$ is of the form

$$\begin{aligned} (\sigma _1 \times \sigma _1^{\vee }) \times \cdots \times (\sigma _r \times \sigma _r^{\vee }) \times \sigma _{r+1} \times \cdots \times \sigma _s. \end{aligned}$$

(7.6)

where each $\sigma _i$ is a Speh representation for $i = 1, \ldots , r$, and each representation $\sigma _j$ is $H_{m_j,m_j}$-distinguished for some positive integer $m_j$, $j = r+1 , \ldots ,s$.

Proof

By the work of Blanc and Delorme [2], we know that $\sigma _j \times \sigma _j^{\vee }$ is $H_{m_j,m_j}$-distinguished with $m_j$ the degree of $\sigma _j$, $j = 1,\ldots ,r$. One direction then follows from Lemma 5.7. Write $\pi = \pi _1 \times \cdots \times \pi _t$ to be the Tadić decomposition of $\pi$. We prove the other direction by induction on t. The case $t = 1$ is obvious. In general, as $\pi$ is a commutative product, we order these $\pi _i$ in the following way: We first group these $\pi _i$ by cuspidal supports. Namely, representations with cuspidal supports contained in the union of one cuspidal line and its contragredient are put in the same group. The ordering of the groups can be arbitrary. For representations within the same group, if their cuspidal supports are contained in one cuspidal line, we arrange the ordering such that when $i < j$, we have either ${\textbf {b}}(\pi _i) < {\textbf {b}}(\pi _j)$, or ${\textbf {b}}(\pi _i) = {\textbf {b}}(\pi _j)$ and ${\textbf {ht}}(\pi _i) \leqslant {\textbf {ht}}(\pi _j)$; if their cuspidal supports are contained in two different cuspidal lines, we arrange the ordering such that when $i < j$, we have ${\textbf {ht}}(\pi _i) \leqslant {\textbf {ht}}(\pi _j)$.

By our assumption, $\pi$ is $H_{m,m}$-distinguished. We apply Propositon 6.6 and discuss case by case.

Case A. There exists a division of $\pi _t$, $\pi _t = \pi _t' \sqcup \pi _t''$, where $\pi _t'$ is neither $\pi _t$ nor the trivial representation of $G_0$, such that, among other things, $\pi _t'$ is $(H_{r,s},\mu _{(r-s)/2})$-distinguished for two nonnegative integers r and s. We have two subcases.

• (1) The representation $\pi _t'$ is not one dimensional. By Proposition 6.16, we know $\pi _t'$ is an essentially Speh representation. So, by Corollary 6.14, we have $r = s$. That is, $\pi _t'$ is $H_{r,r}$-distinguished. In particular, $\pi _t'$ is self-dual, and hence $\pi _t$ is self-dual. This further shows that $\pi _t'$ is a Speh representation. By Proposition 6.6, there exists i, $1 \leqslant i \leqslant t-1$, and a division of $\pi _i$, $\pi _i = \pi _i' \sqcup \pi _i''$ such that $(\pi _t'')^{\vee } \cong \pi _i'$ and that

  $$\begin{aligned} \pi _1 \times \cdots \times \pi _{i-1} \times \pi _i'' \times \pi _{i+1} \times \cdots \times \pi _{t-1} \end{aligned}$$

  (7.7)

  is $H_{m',m'}$-distinguished for some positive integer $m'$. Thus we have ${\textbf {b}}(\pi _t) = {\textbf {b}}(\pi _i)$. By our assumption on the ordering of representations, we have ${\textbf {ht}}(\pi _i) \leqslant {\textbf {ht}}(\pi _t)$. As $\pi _t'$ is a self-dual Speh representation that does not equal to $\pi _t$, we have ${\textbf {ht}}(\pi _t'') = {\textbf {ht}}(\pi _t)$. As ${\textbf {ht}}(\pi _i') \leqslant {\textbf {ht}}(\pi _i)$, we have ${\textbf {ht}}(\pi _t) = {\textbf {ht}}(\pi _i)$ due to the fact that $(\pi _t'')^{\vee } \cong \pi _i'$. Thus we have $\pi _i \cong \pi _t$ and $\pi _i'' \cong \pi _t'$. Recall that $\pi _t'$ is a $H_{r,r}$-distinguished Speh representation. So, by induction hypothesis, the representation (7.7) is of the form (7.6). After removing $\pi _i''$ in the product, we still get a representation of the form (7.6). Therefore, by adding $\pi _t \times \pi _i$, we get that $\pi$ is of the form (7.6).

• (2) The representation $\pi _t'$ is one dimensional. If $r > 0$ and $s > 0$, then $\pi _t'$ is the trivial representation ${\textbf {1}}$ of $G_{2r}$ by Lemma 3.2. Note that, in this case, $\pi _t$ is not a one dimensional representaion. Then by the same arguments as in Case A (1), we are done in this case. If one of r, s is 0, then $\pi _t'$ is the character $\nu ^{h/2}$ of $G_h$, $h = \max \{r,s\}$. Thus we have ${\textbf {b}}(\pi _t) = {\textbf {b}}(\pi _t') = \nu ^{h-1/2}$. In particular, $\pi _t$ is self-dual. By Proposition 6.6, there exists i, $1 \leqslant i \leqslant t-1$, and a division of $\pi _i$, $\pi _i = \pi _i' \sqcup \pi _i''$ such that $(\pi _t'')^{\vee } \cong \pi _i'$ and that

  $$\begin{aligned} \pi _1 \times \cdots \times \pi _{i-1} \times \pi _i'' \times \pi _{i+1} \times \cdots \times \pi _{t-1} \end{aligned}$$

  (7.8)

  is $(H_{m-n_t,m-n_t +h},\mu _{h/2})$-distinguished with $n_t$ the degree of $\pi _t$. Thus we have ${\textbf {b}}(\pi _i) = {\textbf {b}}(\pi _t) = \nu ^{h - 1/2}$, and $\pi _i$ is also self-dual. By our assumption on the ordering of representations, we have ${\textbf {ht}}(\pi _i) = {\textbf {ht}}(\pi _t)$, and hence $\pi _i \cong \pi _t$. Thus, the representation $\pi _i''$ is the character $\nu ^{-h/2}$ of $G_h$. By Lemma 6.10, the representation (7.8) is isomorphic to the representation

  $$\begin{aligned} \pi _1 \times \cdots \times \pi _{i-1} \times \pi _{i+1} \times \cdots \times \pi _{t-1} \times \nu ^{-h/2}. \end{aligned}$$

  By Lemma 7.2, the representation $\pi _1 \times \cdots \times \pi _{i-1} \times \pi _{i+1} \times \cdots \times \pi _{t-1}$ is $H_{m-n_t,m-n_t}$-distinguished, and hence is of the form (7.6) by induction hypothesis. Therefore, by adding $\pi _i \times \pi _t$, we get that $\pi$ is of the form (7.6).

Case B. In this case the representation $\pi _t$ is $(H_{r,s},\mu _{(r-s)/2})$-distinguished for two nonnegative integers r and s, and the representation

$$\begin{aligned} \pi _1 \times \cdots \times \pi _{t-1} \end{aligned}$$

is $(H_{m - r,m-s},\mu _{(r-s)/2})$-distinguished. As $\pi _t$ is a Speh representation, by consideration of its central character, we have $r = s$. Therefore, by induction hypothesis we are done.

Case C. There exists i, $1 \leqslant i \leqslant t-1$, and a division of $\pi _i$, $\pi _i = \pi _i' \sqcup \pi _i''$, such that $(\pi _t)^{\vee } \cong \pi _i'$ and that the representation

$$\begin{aligned} \pi _1 \times \cdots \times \pi _{i-1} \times \pi _i'' \times \pi _{i+1} \times \cdots \times \pi _{t-1} \end{aligned}$$

is $H_{m-n_t,m-n_t}$-distinguished. By our assumption on the ordering of representations, we have $\pi _i \cong (\pi _t)^{\vee }$. Thus $\pi _i''$ is the trivial representation of $G_0$. By induction hypothesis, the representation $\pi _1 \times \cdots \times \pi _{i-1} \times \pi _{i+1} \times \cdots \times \pi _{t-1}$ is of the form (7.6). Therefore, by adding $\pi _t \times (\pi _t)^{\vee }$, the representation $\pi$ is of the form (7.6). $\square$

To classify distinguished representations in the entire unitary dual, it remains to consider distinction of complementary series representations. Recall that a complementary series representation is an irreducible unitary representation of the form $\nu ^{\alpha }\text {Sp}(\delta ,k) \times \nu ^{-\alpha }\text {Sp}(\delta ,k)$ with $0< \alpha < 1/2$, and is denoted by $\text {Sp}(\delta ,k)[\alpha ,-\alpha ]$. By the work of Blanc and Delorme [2], one sees that $\text {Sp}(\delta ,k)[\alpha ,-\alpha ]$ is $H_{m,m}$-distinguished if and only if it is self-dual, where m is the degree of $\text {Sp}(\delta ,k)$. To apply the geometric lemma, we first note the following lemma.

Lemma 7.4

Let $\rho$ be a unitary supercuspidal representation of $G_d$ and c a fixed integer. Let $\pi$ be a ladder representation of $G_n$ with cuspidal supports contained in the cuspidal line ${\mathbb {Z}}\nu ^{\alpha +c/2}\rho$ or ${\mathbb {Z}}\nu ^{-\alpha +c/2}\rho$ with $0< \alpha < 1/2$, then $\pi$ cannot be self-dual. If, moreover, $\pi$ is left aligned, then $\pi$ cannot be $(H_{p,q},\mu _{(p-q)/2})$-distinguished for certain nonnegative integers p, q with $p+q=n$.

Proof

As $0< \alpha < 1/2$, the cuspidal line ${\mathbb {Z}}\nu ^{\alpha +c/2}\rho$ (or ${\mathbb {Z}}\nu ^{-\alpha +c/2}\rho$) is not self-dual. Thus $\pi$ cannot be self-dual by Lemma 6.1. For the second statement, if $\pi$ is one dimensional, then by Lemma 6.19, the cuspidal supports of $\pi$ is contained in ${\mathbb {Z}}\nu ^0$ or ${\mathbb {Z}}\nu ^{-1/2}$. This contradicts with our assumption; if $\pi$ is not one dimensional, then by Proposition 6.16 and Corollary 6.14, one sees $\pi$ is self-dual. This is absurd as shown by the first statement. $\square$

Theorem 7.5

An irreducible unitary representation $\pi$ of $G_{2n}$ is $H_{n,n}$-distinguished if and only if it is self-dual and its Arthur part $\pi _{Ar }$ is of the form (7.6).

Proof

To simplify notation, we will say a representaion H-distinguished for $H_{m,m}$-distinguished when there is no need to address m. Write $\pi = \pi _{Ar } \times \pi _c$. If $\pi$ is self-dual, by uniqueness of Tadić decomposition, we have $\pi _c$ is also self-dual. As $\pi _c$ is a commutative product of complementary series representations, we have $\pi _c$ is H-distinguished. The ‘if’ part then follows from Lemma 5.7. For the ‘only if’ part, write $\pi$ as a product of essentially Speh representations

$$\begin{aligned} \pi _1 \times \cdots \times \pi _t \times \nu ^{\alpha _1}\text {Sp}(\delta _1,k_1) \times \nu ^{-\alpha _1}\text {Sp}(\delta _1,k_1) \times \cdots \times \nu ^{\alpha _r}\text {Sp}(\delta _r,k_r) \times \nu ^{-\alpha _r}\text {Sp}(\delta _r,k_r) \end{aligned}$$

(7.9)

such that $k_1 \leqslant k_2 \leqslant \cdots \leqslant k_r$, and that $\pi _i$ is a Speh representation for $i = 1, \ldots ,t$. Now we appeal to Proposition 6.6. By Lemma 7.4, only Case C can happen. Note that we have $k_1 \leqslant \cdots \leqslant k_r$ and $0< \alpha _i < 1/2$, $i = 1 ,\ldots ,r$. By simple arguments we can show that each time after applying Proposition 6.6, we can delete two non-unitary essentially Speh representations in the product (7.9), and the new representation is H-distinguished. Thus by a repeated use of Proposition 6.6, we get $\pi _{Ar } = \pi _1 \times \cdots \times \pi _t$ is H-distinguished. The ‘only if’ part then follows from Theorem 7.3. $\square$