Abstract
Let F be a local non-Archimedean field of characteristic zero with a finite residue field. Based on Tadić’s classification of the unitary dual of \({\mathrm {GL}}_{2n}(F)\), we classify irreducible unitary representations of \({\mathrm {GL}}_{2n}(F)\) that have nonzero linear periods, in terms of Speh representations that have nonzero periods. We also give a necessary and sufficient condition for the existence of a nonzero linear period for a Speh representation.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
1.1 Main results
Let F be a local non-Archimedean field of characteristic zero with a finite residue field. Denote the group \(G_n = {\mathrm {GL}}_{n}(F)\). Let p and q be two nonnegative integers with \(p + q = n\), we denote by \(H = H_{p,q}\) the subgroup of \(G_n\) of matrices of the form:
Let \(\pi \) be a smooth representation of \(G_n\) on a complex vector space V and a character of H, denote by the space of linear forms l on V such that for all \(v \in V\) and \(h \in H\). Smooth representations \(\pi \) of \(G_n\) with are called -distinguished, or simply H-distinguished if is the trivial character \({\textbf {1}}\) of H.
Elements of \(\mathrm {Hom}_H(\pi ,{\textbf {1}})\) are called (local) linear periods of \(\pi \). Linear periods have been studied by many authors. The uniqueness of linear periods was proved by Jacquet and Rallis in [11]; the uniqueness of twisted linear periods, with respect to almost all characters of H and in the case \(p = q\), was proved by Chen and Sun in [3]. It thus remains an interesting question of characterizing irreducible representations that have nonzero linear periods. It is known that a tempered representation of \({\mathrm {GL}}_{2n}(F)\) has nonzero linear periods with respect to \(H_{n,n}\) if and only if it is a functorial transfer of a generic tempered representation of \({\mathrm {SO}}_{2n+1}(F)\), see [13, 20, 22]. Another closely related characterization of the existence of nonzero linear periods for an essentially square-integrable representation is through poles of the local exterior square L-functions associated with the representation, see [20] and references therein. A recent preprint by Sécherre [29] studied supercuspidal representations with nonzero linear periods from the point of view of type theory. However, all of these characterizations are for generic representations. Motivated by the recent work of Gan–Gross–Prasad [9] on branching laws in the non-tempered case, we are led to consider in this work the existence of nonzero linear periods for irreducible unitary representations.
Our main results are as follows. We refer the reader to Sect. 2 for unexplained notation in the following two theorems.
Theorem 1.1
Let \(\text {Sp}(\delta ,k)\) be a Speh representation of \(G_{2n}\), where \(\delta \) is a square-integrable representation of \(G_d\) with \(d > 1\), and k is a positive integer (\(2n = dk\)). Then \(\text {Sp}(\delta ,k)\) is \(H_{n,n}\)-distinguished if and only if d is even and \(\delta \) is \(H_{d/2,d/2}\)-distinguished.
Theorem 1.2
An irreducible unitary representation \(\pi \) of \(G_{2n}\) is \(H_{n,n}\)-distinguished if and only if it is self-dual and its Arthur part \(\pi _{Ar }\) is of the form
where each \(\sigma _i\) is a Speh representation for \(i = 1, \cdots , s\), and each representation \(\sigma _j\) is \(H_{m_j,m_j}\)-distinguished for some positive integer \(m_j\), \(j = r+1 , \ldots ,s\).
Distinction problem for unitary representation has already been considered by Matringe for local Galois periods in [21] and by Offen and Sayag for local Symplectic periods in [27, 28]. We remark that the special case of Theorem 1.2 for representations of Arthur type (see Theorem 7.3) is similar to [22, Theorem 3.13] about local linear periods for generic representations and the main result in [21] about local Galois periods for unitary representations. A global analogue of our result is to find the \(H_{n,n}\)-distinguished representation in the automorphic dual of \(G_{2n}\), which we will pursue in future works. We also refer the reader to [7, 11] for the role of local linear periods and their global analogues in the study of standard L-functions.
1.2 Remarks on the method of the proof
Most of our work deals with distinction of parabolically induced representations of \(G_n\). The main tool to study distinction of induced representations is the geometric lemma of Bernstein-Zelevinsky [1], which relates distinction of an induced representation to distinction of some Jacquet module of the inducing data. It was shown by Tadić in [30] that every irreducible unitary representation is isomorphic to the parabolic induction of Speh representations or their twists. The observation is that Jacquet modules of Speh representations have convenient combinatorial descriptions similar to those of Jacquet modules of essentially square-integrable representations [16]. As hinted by the geometric lemma, to classify \(H_{n,n}\)-distinguished irreducible unitary representations, it is necessary to consider \(H_{p,q}\)-distinction with respect to a particular family of characters in (2.1), not only of Speh representations, but also of a larger class of representations, ladder representations. The class of ladder representations was introduced by Lapid and Mínguez in [17], and has many remarkable properties which make them an ideal testing ground for distinction of non-generic representations and some other questions in the representation theory of general linear groups, see for example [6, 10, 18, 24]. The most complicated part of the paper, Sect. 6, is devoted to the study of distinction of ladder representations. Our treatment is largely combinatorial based on detailed analysis by the geometric lemma. We refer the reader to [19, 22] for a similar approach to the classification of distinguished generic representations in Galois symmetric space and our setting respectively.
We next outline the proof of Theorem 1.1. For the ‘if’ part, the existence of non-zero linear periods for the standard module of a Speh representation \(\text {Sp}(\varDelta ,k)\) is guaranteed by the work of Blanc and Delorme [2] when \(\varDelta \) is \(H_{d/2,d/2}\)-distinguished. Thus it suffices to show that the maximal proper subrepresentation of the standard module associated with \(\text {Sp}(\varDelta ,k)\) is not \(H_{n,n}\)-distinguished. The explicit structure of this maximal proper subrepresentation is well known by the work of Tadić [32] (see also [17]). For the ‘only if’ part of Theorem 1.1, however, we cannot expect to get any information on the distinguishedness of \(\varDelta \) from that of the standard module of \(\text {Sp}(\varDelta ,k)\) when k is an even, as in this case, the standard module of \(\text {Sp}(\varDelta ,k)\) is \(H_{n,n}\)-distinguished for any self-dual \(\varDelta \) by the work of Blanc and Delorme [2]. We instead use the idea of ‘restricting to the mirabolic subgroup’, and relate linear periods on \(\text {Sp}(\varDelta ,k)\) with those on its highest shifted derivative, which is exactly \(\text {Sp}(\varDelta ,k-1)\). The ‘only if’ part is then proved by induction on k. We remark that the idea of exploiting the theory of derivatives in distinction problems has already appeared many times in the literature, see for example [4, 15, 21, 22].
The paper is organized as follows. In Sect. 2 we introduce notations and some preliminaries on the representation theory of general linear groups. In Sect. 3 we present some general facts on \((H_{p,q},\mu _a)\)-distinguished representations, where \(\mu _a\) is the character in (2.1). In this section, we recall a result of Gan which is crucial for our combinatorial study of twisted linear periods. In Sect. 4 we give a detailed analysis of the parabolic orbits of the symmetric space involved and in Sect. 5 we draw some consequences of the geometric lemma. Section 6 is devoted to the study of distinction of ladder representations. We then complete the classification in Sect. 7.
2 Preliminaries
Throughout the paper let F be a local non-Archimedean field of characteristic zero with a finite residue field.
For any \( n \in {\mathbb {Z}}_{\geqslant 0}\), let \(G_n = {\mathrm {GL}}_n(F)\) and let \({\mathscr {R}}(G_n)\) be the category of smooth complex representations of \(G_n\) of finite length. Denote by \(Irr (G_n)\) the set of equivalence classes of irreducible objects of \({\mathscr {R}}(G_n)\) and by \({\mathscr {C}}(G_n)\) the subset consisting of supercuspidal representations. (By convention we define \(G_0\) as the trivial group and \(Irr (G_0)\) consists of the trivial representation of \(G_0\).) Let \(Irr \) and \({\mathscr {C}}\) be the disjoint union of \(Irr (G_n)\) and \({\mathscr {C}}(G_n)\),\(n \geqslant 0\), respectively. For a representation \(\pi \in {\mathscr {R}}(G_n)\), we call n the degree of \(\pi \).
Let \(\mathfrak {R}_n\) be the Grothendieck group of \({\mathscr {R}}(G_n)\) and \(\mathfrak {R}= \oplus _{n \geqslant 0} \mathfrak {R}_n\). The canonical map from the objects of \({\mathscr {R}}(G_n)\) to \(\mathfrak {R}_n\) will be denoted by \(\pi \mapsto [\pi ]\).
Denote by \(\nu \) the character \(\nu (g) = |\det g |\) on any \(G_n\). (The n will be implicit and hopefully clear from the context.) For any \(\pi \in {\mathscr {R}}(G_n)\) and \(a \in {\mathbb {R}}\), denote by \(\nu ^a \pi \) the representation obtained from \(\pi \) by twisting it by the character \(\nu ^a\), and denote by \(\pi ^{\vee }\) the contragredient of \(\pi \). The sets \(Irr \) and \({\mathscr {C}}\) are invariant under taking contragredient. For a character \(\chi \) of \(F^{\times }\), define the real part \(\Re (\chi )\) of \(\chi \) to be the real number a such that \( | \chi (z) |_{{\mathbb {C}}} = |z|^a\), \(z \in F^{\times }\), where \(|\cdot |_{{\mathbb {C}}}\) is the absolute value on \({\mathbb {C}}\). For a subgroup Q of \(G_n\), denote by \(\delta _Q\) the modular character of Q.
For two nonnegative integers p and q with \(p + q = n\), we denote by \(w_{p,q}\) the matrix
Let \(H_{p,q}\) be the subgroup of \(G_n\) as in the introduction. For \(a \in {\mathbb {R}}\), define the character \(\mu _a\) of \(H_{p,q}\) by
(By convention we allow the case where p or q is zero.)
2.1 Jacquet modules of induced representations
The standard parabolic subgroups of \(G_n\) are in bijection with compositions \( (n_1, \ldots ,n_t)\) of n. The corresponding standard Levi subgroup is the group of block diagonal invertible matrices with block sizes \(n_1, \ldots , n_t\). It is isomorphic to \(G_{n_1} \times \cdots \times G_{n_t}\).
Let \(P = M < imes U\) be a standard parabolic subgroup of \(G_n\) and \(\sigma \) a smooth, complex representation of M. We denote by \(\mathrm {Ind}_P^{G_n}(\sigma )\) its normalized parabolic induction; for any standard Levi subgroup \(L \subset M\), we denote by \(r_{L,M}(\sigma )\) the normalized Jacquet module (see [1, Sect. 2.3]).
If \(\rho _1, \ldots , \rho _t\) are representations of \(G_{n_1}, \ldots , G_{n_t}\) respectively, we denote by
the representation \(\mathrm {Ind}_P^{G_n} \sigma \) where \(\sigma \) is the representation \(\rho _1 \otimes \cdots \otimes \rho _t\) of M, where M is the standard Levi subgroup of the parabolic subgroup P corresponding to \((n_1,\ldots ,n_t)\).
Next we briefly review the Jacquet module of a product of representations of finite length [33, Sect. 1.6] (or more precisely, its composition factors). Let \(\alpha = (n_1,\ldots , n_t)\) and \(\beta = (m_1,\ldots , m_s)\) be two compositions of n. For every \(i \in \{ 1, \ldots , t\}\), let \(\rho _i \in \mathscr {R}(G_{n_i})\). Denote by \(\text {Mat}^{\alpha , \beta }\) the set of \(t \times s\) matrices \(B = (b_{i,j})\) with nonnegative integer entries such that
Fix \(B\in \text {Mat}^{\alpha ,\beta }\). For any \(i \in \{1,\ldots ,t \}\), \(\alpha _i = (b_{i,1},\ldots ,b_{i,s})\) is a composition of \(n_i\) and we write the compostion factors of \(r_{\alpha _i} (\rho _i)\) as
where \(l_i\) is the length of \(r_{\alpha _i}(\rho _i)\). For any \( j \in \{ 1, \ldots , s\}\) and a sequence \(\underline{k} = (k_1,\ldots ,k_r)\) of integers such that \(1 \leqslant k_i \leqslant l_i\), define
Then we have
2.2 Langlands classification
By a segment of cuspidal representations we mean a set
where \(\rho \in {\mathscr {C}}\) and \(a, b \in {\mathbb {R}}\), \(b- a \in {\mathbb {Z}}_{\geqslant 0}\). The representation \(\nu ^a\rho \times \nu ^{a+1}\rho \times \cdots \times \nu ^b\rho \) has a unique irreducible quotient, which is an essentially square-integrable representaton and is denoted by \(\varDelta ([a,b]_{\rho })\). The map \([a,b]_{\rho } \mapsto \varDelta ([a,b]_{\rho })\) gives a bijection between the set of segments of cuspidal representations and the subset of essentially square-integrable representations in \(Irr \). (In what follows, for simplicity of notation, we shall use \(\varDelta \) to denote either a segment of cuspidal representations or the essentially square-integrable representations corresponding to it; we hope this will not cause any confusion.) We use the convention that \(\varDelta ([a,b]_{\rho }) = 0\) if \(b < a-1\) and \(\varDelta ([a,a-1]_{\rho }) = 1\), the trivial represntation of \(G_0\).
We denote the extremities of \(\varDelta = \varDelta ([a,b]_{\rho })\) by \({\textbf {b}}(\varDelta ) = \nu ^a\rho \in {\mathscr {C}}\) and \({\textbf {e}}(\varDelta ) = \nu ^b\rho \in {\mathscr {C}}\) respectively. We also write \(l(\varDelta ) = b -a + 1\) for the length of \(\varDelta \).
For \(\rho \in {\mathscr {C}}\), we denote by \({\mathbb {Z}}\rho \) the set \(\{\nu ^a \rho \ |\ a \in {\mathbb {Z}}\}\) and call it the cuspidal line of \(\rho \). We then transport the order and additive structure of \({\mathbb {Z}}\) to the cuspidal line \({\mathbb {Z}}\rho \). Thus we shall sometimes write \(\nu ^a\rho + b = \nu ^{a+b} \rho \) and \(\nu ^a \rho \leqslant \nu ^b \rho \) if \(a \leqslant b\), where a, b are integers. By the contragredient of \({\mathbb {Z}}\rho \) we mean the cuspidal line \({\mathbb {Z}}\rho ^{\vee }\).
Let \(\varDelta \) and \(\varDelta '\) be two segments. We say that \(\varDelta \) and \(\varDelta '\) are linked if \(\varDelta \cup \varDelta '\) forms a segment but neither \(\varDelta \subset \varDelta '\) nor \(\varDelta ' \subset \varDelta \). If \(\varDelta \) and \(\varDelta '\) are linked and \({\textbf {b}}(\varDelta ) = {\textbf {b}}(\varDelta ')\nu ^j\) with \(j < 0\), then we say that \(\varDelta \) precedes \(\varDelta '\) and write \(\varDelta \prec \varDelta '\).
A multisegment is a multiset (that is, set with multiplicities) of segments. Denote by \({\mathscr {O}}\) the set of multisegements. For \(\rho \in {\mathscr {C}}\), let \({\mathscr {O}}_{\rho }\) denote the multisegements such that all of its segements are contained in the cuspidal line \({\mathbb {Z}}\rho \). An order \(\mathfrak {m}= \{\varDelta _1,\ldots , \varDelta _t \} \in {\mathscr {O}}\) on a multisegments \(\mathfrak {m}\) is of standard form if \(\varDelta _i \nprec \varDelta _j\) for all \(i < j\). Every \(\mathfrak {m}\in {\mathscr {O}}\) admits at least one standard order.
Let \(\mathfrak {m}= \{\varDelta _1, \ldots , \varDelta _t \} \in {\mathscr {O}}\) be ordered in standard form. The representation
is independent of the choice of order of standard form. It has a unique irreducible quotient that we denote by \(L (\mathfrak {m})\). The Langlands classification says that the map \(\mathfrak {m}\mapsto L (\mathfrak {m})\) is a bijection between \({\mathscr {O}}\) and \(Irr \).
2.3 Unitary dual of \(G_n\)
We briefly recall the classification of the unitary dual of \(G_n\) by Tadić [30, Theorem D]. Let \(\text {Irr}^u\) be the subset of unitarizable representations in \(\text {Irr}\), and \({\mathscr {D}}^u\) the subset of all square-integrable classes in \(Irr ^u\). Let k be a positive integer, and let \(\delta \in {\mathscr {D}}^u\). The repersentation
has a unique irreducible unitarizable quotient \(\text {Sp}(\delta ,k)\), called a Speh representation.
Suppose \( 0< \alpha < 1/2\). The representation \(\nu ^{\alpha } \text {Sp}(\delta ,k) \times \nu ^{-\alpha } \text {Sp}(\delta ,k)\) is irreducible and unitarizable; we denote it by \(\text {Sp}(\delta ,k)[\alpha ,-\alpha ].\)
Let B be the set of all
where \(\delta \in \text {D}^u\), k is a positive integer and \(0< \alpha < 1/2\). By [30, Theorem D], an irreducible representation \(\pi \) is unitarizable if and only if it is of the form
Moreover, this expresssion is unique up to permutation. We call it a Tadić decomposition of \(\pi \).
By an irreducible representation of Arthur type, we mean an irreducible unitary representation whose Tadić decomposition does not involve any \(\text {Sp}(\delta ,k)[\alpha ,-\alpha ]\). For \(\pi \in \text {Irr}^u\), we then have a decomposition \(\pi = \pi _{Ar } \times \pi _c\), where \(\pi _{Ar }\) is a representation of Arthur type and is called the Arthur part of \(\pi \).
3 Preliminaries on \((H_{p,q},\mu _a)\)-distinguished representations
3.1 Basic facts
Lemma 3.1
-
(1)
Let \(\pi \) be a smooth representation of \(G_n\). If \(\pi \) is \((H_{p,q},\mu _a)\)-distinguished for two nonnegative integers p, q with \(p + q = n\) and \(a \in {\mathbb {R}}\), then \(\pi \) is also \((H_{q,p},\mu _{-a})\)-distinguished;
-
(2)
Let \(\pi _1,\ldots , \pi _t \in Irr (G_n)\). If \(\pi _1 \times \cdots \times \pi _t\) is \((H_{p,q},\mu _a)\)-distinguished for two nonnegative integers p, q with \(p + q = n\) and \(a \in {{\mathbb {R}}}\), then \(\pi _t^{\vee } \times \cdots \times \pi _1^{\vee }\) is \((H_{p,q},\mu _{-a})\)-distinguished.
Proof
The statement (1) follows from the fact that \(\pi \cong \pi ^{w_{q,p}}\). Let \(\iota \) denote the involution of transpose inversion. Then (2) follows from the fact that \(\pi \circ \iota \cong \pi ^{\vee }\) for any irreducible representation \(\pi \) and the fact that
\(\square \)
For representations of dimension one, we have the following simple lemma, whose proof we omit.
Lemma 3.2
Let be a character of \(G_n\). Assume that is \((H_{p,q},\mu _a)\)-distinguished for nonnegative integers p, q with \(p + q = n\) and \(a \in {{\mathbb {R}}}\). If \(q = 0\) (resp. \(p = 0\)), then is the character \(\nu ^a\) (resp. \(\nu ^{-a}\)) of \(G_n\); If p, \(q > 0\), then \(a = 0 \) and , the trivial character of \(G_n\).
For untwisted linear periods, we have the following fundamental result due to Jacquet and Rallis [11].
Lemma 3.3
Let p, q be two positive integers with \(p + q = n\). If \(\pi \in Irr (G_n)\), then \(\dim \mathrm {Hom}_{H_{p,q}}(\pi ,{\textbf {1}}) \leqslant 1\). Furthermore, if \(\dim \mathrm {Hom}_{H_{p,q}} (\pi , {\textbf {1}}) = 1\), then \(\pi \cong \pi ^{\vee }\).
Remark 3.4
In this work we will not need multiplicity one results about (twisted) linear periods. However, the self-dualness property of distinguished representations is important for our applications of the geometric lemma. For example, one key ingredient is Proposition 3.9 which asserts self-duality for distinguished essentially square-integrable representations. In the case \(p = q\), twisted linear periods have been studied by Chen and Sun in [3]. Their result shows that, for all but finitely many a, \(\dim \mathrm {Hom}_{H_{p,p}}(\pi ,\mu _a) \leqslant 1\) for all \(\pi \in Irr (G_{2p})\). Due to the author’s limited knowledge, one cannot deduce self-duality for distingsuished representations as in the untwisted case. For generic representations, however, one can deduce self-duality from a result of Gan as shown in the next subsection.
3.2 Relations with Shalika periods
The Shalika subgroup of \(G_{2n}\) is defined to be
where \(M_n\) indicates the set of \(n \times n\) matrices with entries in F. Define a character \(\psi _{S_{2n}}\) on \(S_{2n}\) by
where \(\psi _F\) is a non-trivial character of F. For a smooth representation \(\pi \) of \(G_{2n}\), an element in \(\mathrm {Hom}_{S_{2n}} (\pi ,\psi _{S_{2n}})\) is called a local Shalika period of \(\pi \).
In the untwisted case, the relation between linear periods and Shalika periods is well known (see [14] for their equivalence in the case of supercuspidal representations; see also a discussion for relatively square-integrable representations in [20, Sect. 5]). Using a theta correspondence approach, Gan proved the following result that relates generalized linear periods and generalized Shalika periods on \(G_n\).
Proposition 3.5
Let \(\pi \) be an irreducible generic representation of \(G_{2n}\) and \(\sigma \) an irreducible representation of \(G_n\). One has
where \(\sigma \boxtimes \psi _{S_{2n}}\) is viewed as a representation of \(S_{2n} = G_n < imes N_{n,n}\).
Proof
This is a consequence of Theorem 3.1 and Theorem 4.1 of [8]. \(\square \)
In fact, in Theorem 3.1 of [8], Gan obtained a statement that relates the generalized linear period of an irreducible representation to the generalized Shalika period of the big theta lift of its contragradient. We refer interested readers to the original paper of Gan for more details. What is pertinent to this work is the following simple corollary that relates as well twisted linear periods in our context to Shalika periods.
Corollary 3.6
Let \(\pi \) be a generic representation of \(G_{2n}\). The followings are equivalent:
-
(1)
\(\pi \) is \((H_{n,n},\mu _a)\)-distinguished for some \(a \in {{\mathbb {R}}}\);
-
(2)
\(\pi \) is \((H_{n,n},\mu _a)\)-distinguished for all \(a \in {{\mathbb {R}}}\);
-
(3)
\(\pi \) is \((S_{2n},\psi _{S_{2n}})\)-distinguished.
In particular, if one of these equivalent conditions holds, then \(\pi \) is self-dual.
Proof
As \(\pi \) is generic, its twist \(\nu ^a \pi \), for \(a \in {\mathbb {R}}\), is also generic. So
\(\square \)
3.3 The theory of Bernstein–Zelevinsky derivatives
Let \(P_n \subset G_n\) be the mirabolic subgroup of \(G_n\) consisting of matrices with the last row \((0,0,\ldots ,0,1)\). We refer the reader to [1, 3.2] for the definition of the following functors
Define \(\pi ^{(k)} = \varPsi ^-(\varPhi ^-)^{k-1}\left( \pi |_{P_n} \right) \) to be the kth derivative of a representation \(\pi \) of \(G_n\).
The following proposition can be proved by the same argument as those in [15, Proposition 1] (see also [20, Proposition 3.1], where the linear subgroups \(H_{p,q}\) take different forms.)
Proposition 3.7
If \(\sigma \) is a representation of \(P_{n-1}\) and is a character of \(H_{p,q}\), then
as complex vector spaces, where is the character of \(H_{q-1,p}\) defined by . In particular, for all \(a \in {{\mathbb {R}}}\), one has
As a corollary, we have the following result due to Matringe [20, Theorem 3.1].
Corollary 3.8
Let \(\varDelta \) be an essentially square-integrable representation of \(G_n\). Let p, q be two positive integers with \(p + q =n\), and a character of \(H_{p,q}\). Assume that \(\pi \) is -distinguished. Then \(p=q\).
Another application of Proposition 3.7 will generalize Corollary 3.8 to essentially Speh representations in Corollary 6.14 of Sect. 6.3.
As a direct consequence of Corollarys 3.8 and 3.6, we have:
Proposition 3.9
Let \(\varDelta \) be an essentially square-integrable representation of \(G_n\). If \(\varDelta \) is \((H_{p,q},\mu _a)\)-distinguished for two positive integers p, q with \(p + q = n\) and some \(a \in {\mathbb {R}}\), then \(p = q\) and \(\varDelta \) is \(H_{p,p}\)-distingusihed (hence self-dual).
4 Symmetric spaces and parabolic orbits
The main tool we use to classify distinguished unitary representations is the geometric lemma of Bernstein and Zelevinsky [1, Theorem 5.2]. Applying it requires a detailed analysis of the double coset space \(P \backslash G_n / H_{p,q}\), where P is a parabolic subgroup of \(G_n\). As \(H_{p,q}\) is a symmetric subgroup of \(G_n\), we follow the framework given by Offen in [26].
4.1 General notations
Let \(G = G_{n}\), \(H = H_{p,q}\) be the subgroup of \(G_n\) as in the introduction. Let
and \(\theta = \theta _{p,q}\) be the involution on \(G_n\) defined by \(\theta (g) = \varepsilon g \varepsilon ^{-1}\). The symmetric space associated to \((G,\theta )\) is
equipped with the G-action \(g \cdot x = g x \theta (g)^{-1}\). The map \(g \mapsto g \cdot e\) gives a bijection of the coset space G/H onto the orbit \(G \cdot e \subset X\), and thus a bijection of the double coset space \(P \backslash G / H\) onto the P-orbits in \(G \cdot e\), where P denotes a parabolic subgroup of G. For any \(g \in G\), denote by \([g]_G\) the conjugacy class of g in G. Note that the map \(g \mapsto g \varepsilon \) gives a bijection of \(G \cdot e \) onto \([\varepsilon ]_G\) and that the G-action on \(G \cdot e\) is transformed to the conjugation action of G on \([\varepsilon ]_G\).
For any subgroup Q of G and \(x \in X\), let \(Q_x = \{ g \in Q \ |\ g \cdot x = x \}\) be the stabilizer of x in Q. Note that \(Q_x\) is just the centralizer of \(x \varepsilon \) in Q.
4.2 Twisted involutions in Weyl groups
A first coarse classification of the double cosets in \(P \backslash G / H\) is given by certain Weyl group elements. Let W be the Weyl group of G. Let
be the set of twisted involutions in W. For two standard Levi subgroups M and \(M'\) of G, let be the set of all \(w \in W\) that are left \(W_M\)-reduced and right \(W_{M'}\)-reduced.
Given a standard parabolic subgroup \(P = M < imes U\), define a map
by the relation
For \(x \in X\), let
Then L is a standard Levi subgroup of M satisfying \(L = w L w^{-1}\).
4.3 Admissible orbits
It is noted in [26] that, to apply the geometric lemma in particular cases, it is necessary to first understand the admissible orbits. Recall that \(x \in X\) (or a P-orbit \(P \cdot x\) in X) is said to be M-admissible if \( M = w M w^{-1}\) where \(w = \iota _M(P \cdot x)\). We now describe the relevant data for M-admissible P-orbits in \(G \cdot e\).
By [26, Corollary 6.2], M-admissible P-orbits in \(G \cdot e\) is in bijection with M-orbits in \(G \cdot e \cap N_G(M)\), or equivalently M-conjugacy classes in \([\varepsilon ]_G \cap N_G(M)\).
Fix a composition \(\bar{n}=(n_1,\ldots ,n_t)\) of n. Let \(P = M < imes U\) be the standard parabolic subgroup of \(G_n\) associated to \(\bar{n}\). Denote by \(\mathfrak {S}_t^{(\bar{n})}\) the set of permutations \(\tau \) on the set \(\{ 1, 2, \ldots ,t\}\) such that \(n_i = n_{\tau (i)}\) for all \(i \in \{1, \ldots ,t \}\). To each \(\tau \) in \(\mathfrak {S}_t^{(\bar{n})}\), we associate a block matrix \(w_{\tau }\) which has \(I_{n_i}\) on its \((\tau (i),i)\)-block for each i and has 0 elsewhere. Then the map
defines an isomorphism of groups from \(\mathfrak {S}_t^{(\bar{n})}\) to \(N_G(M) / M\). Write an element of M as \(\text {diag}\{A_1,\ldots ,A_t\}\). Note that an element \(w_{\tau }\text {diag}\{A_1,\ldots ,A_t \}\) of \(N_G(M)\) has order 2 if and only if
One sees that the M-conjugacy classes in \([\varepsilon ]_{G} \cap N_G(M)\) are parameterized by the set of pairs \((\mathfrak {c}_{\tau },\tau )\) where \(\tau \in \mathfrak {S}_t^{(\bar{n})}\), \(\tau ^2 = 1\), and \(\mathfrak {c}_{\tau }\) is a set of the form
such that
Denote by \(\mathcal {I}_{p,q}^{\sharp }(\bar{n})\) the set of all such pairs.
For the M-admissible P-orbit \(\mathcal {O}\) corresponding to \((\mathfrak {c}_{\tau },\tau )\) in \(\mathcal {I}_{p,q}^{\sharp }(\bar{n})\), we can choose a natural orbit representative \( x = x_{(\mathfrak {c}_{\tau },\tau )} \in \mathcal {O} \cap N_G(M)\) as follows: The matrix \(x \varepsilon \) has \(I_{n_i}\) on its \((\tau (i),i)\)-block when \(\tau (i) \ne i\), \(\text {diag}(I_{n_{i,+}}, -I_{n_{i,-}})\) on its (i, i)-block when \(\tau (i) = i\), and 0 elsewhere. One sees easily that \(M_x\) consists of elements \(\text {diag}\{A_1,\ldots ,A_t \}\) such that
Here and in what follows, we denote by \(I_{n_1,n_2}\) for the diagonal matrix \(\text {diag}\{I_{n_1}, -I_{n_2} \}\). Thus, when \(\tau (i) = i\), we may further write \(A_i\) as \(\text {diag}\{A_{i,+},A_{i,-} \}\). One also has \(P_x = M_x < imes U_x\).
The following computation of modular characters is indispensable for applications of the geometric lemma, see [26, Theorem 4.2]. We omit the proof here as it is obtained by a routine calculation.
Lemma 4.1
Let \(x \in G\cdot e \cap N_G(M)\) be the representative as above of the M-admissible P-orbit corresponding to \((\mathfrak {c},\tau ) \in \mathcal {I}_{p,q}^{\sharp }(\bar{n})\). Then, for \(m = diag \{A_1,\ldots ,A_t\} \in M_x\), we have
4.4 General orbits
For our purposes, we consider only P-orbits in \(G \cdot e \subset X\) where P is a maximal parabolic subgroup. Let \(P = P_{k,n-k}\) be the standard parabolic subgroup associated to \((k,n-k)\) with M its Levi subgroup. We follow the geometric method as in [22]. The case where \(|p-q|\leqslant 1\) can be essentially covered by the results there. We remark however that the symmetric subgroup H there takes a different form and the treatment here is independent.
Let V be a n-dimensional F-vector space with a basis \(\{e_1,\ldots , e_{n}\}\). Let \(V_+\) (resp. \(V_-\)) be the subspace of V of dimension p (resp. q) which is generated by \(\{e_1,\ldots ,e_p \}\) (resp. \(\{ e_{p+1},\ldots , e_n \}\)). The coset space G/P can be identified with the set of subspaces of V of dimension k. For such a subspace W, set
Lemma 4.2
Let \(W_1\) and \(W_2\) be two subspaces of V of dimension k. Then they are in the same H-orbit if and only if \(r_{W_1} = r_{W_2}\) and \(s_{W_1} = s_{W_2}\). For a pair of nonnegative integers (r, s), there is a subspace W of V such that \(r = r_W\) and \(s = s_W\) if and only if
Denote by \(\mathcal {I}^{k}_{p,q}\) the set of pairs of nonnegative integers (r, s) that satisfying (4.6). Then, by Lemma 4.2, the double cosets in \(H \backslash G / P\) can be parameterized by \(\mathcal {I}^k_{p,q}\). For \((r,s) \in \mathcal {I}^{k}_{p,q}\), call \(d = k - r -s\) the defect of (r, s).
We first seek a complete set of representatives of \(P \backslash G / H\). We split the discussions into two cases.
Case \(k \geqslant p\). Let \(W_{(r,s)}\) be the subspace of V generated by
Then \(\dim _F W_{r,s} = k\), \(\dim _F (W_{(r,s)} \cap V_+) = r\) and \(\dim _F (W_{(r,s)} \cap V_-) = s\). Let \(\tilde{\eta }_{(r,s)}^{-1}\) be the block matrix
where \(C_1\) and \(C_4\) are matrices of size \(p \times p\) and \(q \times q\) respectively, and
Then \(\{\tilde{\eta }_{(r,s)}^{-1} \}\) is a complete set of representatives of the double coset space \(H \backslash G / P\). Taking inverse, we thus get a complete set of representatives \(\{ \tilde{\eta }_{(r,s)}\}\) of \(P \backslash G / H\).
Case \(k \leqslant p\). Let \(W_{(r,s)}\) be the subspace of V of dimension k generated by
Then \(\dim _F W_{r,s} = k\), \(\dim _F (W_{r,s} \cap V_+) = r\) and \(\dim _F (W_{(r,s)} \cap V_-) = s\). Let \(\tilde{\eta }_{(r,s)}^{-1}\) be the block matrix
where \(D_1\) and \(D_4\) are matrices of size \(k \times k\) and \((n-k) \times (n-k)\) respectively, and
Then \(\{\tilde{\eta }_{(r,s)}^{-1} \}\) is a complete set of representatives of the double coset space \(H \backslash G / P\). Taking inverse, we thus get a complete set of representatives \(\{ \tilde{\eta }_{(r,s)}\}\) of \(P \backslash G / H\).
We then describe the relevant data for these general P-orbits in \(G \cdot e\). For \((r,s) \in \mathcal {I}^k_{p,q}\), let \(\tilde{x}_{(r,s)} = \tilde{\eta }_{(r,s)} \theta (\tilde{\eta }_{(r,s)})^{-1} \in G \cdot e \). Thus \(\{ \tilde{x}_{(r,s)} \}\) is a complete set of representatives of P-orbits in \(G\cdot e\). Write \(w_{(r,s)} = \iota _{M} (P \cdot \tilde{x}_{(r,s)})\). Recall that \(w_{(r,s)}\) is left and right \(W_{M}\)-reduced. In either case, we have that
Thus \(L = L_{(r,s)} = M \cap w_{(r,s)} M w_{(r,s)}^{-1}\) is the standard Levi subgroup associated to the composition \((k-d,d,d,n-k-d)\) of n. Denote by Q the standard parabolic subgroup of \(G_n\) with Levi subgroup L. We can choose, in either case, an orbit representative \(x_{(r,s)} \in P \cdot \tilde{x}_{(r,s)} \cap L w_{(r,s)}\) such that
So the group \(L_{x_{(r,s)}}\) consists of elements \(\text {diag}\{A_{1,+},A_{1,-},A_2,A_3,A_{4,+},A_{4,-} \}\) such that
We can also choose \(\eta _{(r,s)} \in G_n\) such that \(\eta _{(r,s)} \theta (\eta _{(r,s)})^{-1} = x_{(r,s)}\) and that
The modular characters for general orbtis that are relavent to us are computed as follows.
Lemma 4.3
For \((r,s) \in \mathcal {I}^k_{p,q}\), let \(x = x_{(r,s)}\), \(\eta = \eta _{(r,s)}\), L and Q as given above. For \(a \in {\mathbb {R}}\), let \(\mu _a\) be the character of \(H = H_{p,q}\) defined in (2.1).
For
then
Proof
Note that \(x_{(r,s)}\) is the natural representative for an L-admissible Q-orbit in \(G\cdot e\) chosen in Sect. 4.3. Then (4.10) follows directly from Lemma 4.1. \(\square \)
5 Consequences of the geometric Lemma
5.1 The geometric lemma
We first recall the formulation of the geometric lemma of Bernstein and Zelevinsky in [26, Theorem 4.2], and we refer the reader to loc.cit for unexplained notation.
Proposition 5.1
Let \(P = M < imes U\) be a standard parabolic subgroup of G. Let \(\sigma \) be a representation of M, and a character of H. If the representation \(\mathrm {Ind}_P^G (\sigma )\) is -distinguished, then there exist a P-orbit \(\mathcal {O}\) in \(P \backslash (G \cdot e)\) and \(\eta \in G\) satisfying \(x = \eta \cdot e \in \mathcal {O} \cap Lw\) (where \(w = \iota _M(P \cdot x)\) and \(L = M(w)\)) such that the Jacquet module \(r_{L,M}(\sigma )\) is -distinguished. Here \(Q = L < imes V\) is the standard parabolic subgroup of G with Levi subgroup L.
We retain the notation of Sect. 4. As a consequence of the orbit analysis there, we formulate the following corollary.
Corollary 5.2
Let \(\sigma _1\) resp. \(\sigma _2\) be a representation of \(G_k\) resp. \(G_{n-k}\). If the representation \(\sigma _1 \times \sigma _2\) is \((H_{p,q},\mu _a)\)-distinguished for some p, \(q \geqslant 0\), \(p + q =n\) and \(a \in {\mathbb {R}}\), then there exists a pair \((r,s) \in \mathcal {I}^k_{p,q}\) with defect \(d = k - r -s\) such that the representation \(r_{(k-d,d)} \sigma _1 \otimes r_{(d,n-k-d)}\sigma _2\) of L is \((L_x, \delta _{Q_x}\delta _Q^{-1/2}\mu _a^{\eta ^{-1}})\)-distinguished, where L is the standard Levi subgroup of \(G_n\) associated to \((k-d,d,d,n-k-d)\), Q is the standard parabolic subgroup with L its Levi part, \(x = x_{(r,s)}\) is given in (4.8) and \(\eta = \eta _{(r,s)} \in G_n\) such that \(x = \eta \cdot e\) and (4.9) holds.
Often in practice there is a filtration of the Jacquet module of the inducing data whose successive factors are pure tensor representations. The following lemma is a direct consequence of Lemma 4.3.
Lemma 5.3
Notation being as above. Let \(\rho = \rho _1 \otimes \rho _2 \otimes \rho _3 \otimes \rho _4\) be a pure tensor representation of L. Then \(\rho \) is \((L_{x},\delta _{Q_x}\delta _Q^{-1/2}\mu _a^{\eta ^{-1}})\)-distinguished if and only if
Remark 5.4
Our proof of classification has an inductive structure. This necessary conditions (5.1) is the reason why we study \((H_{p,q},\mu _a)\)-distinction from the beginning, although our main concern is about \(H_{p,p}\)-distinction.
Remark 5.5
The subscripts in the pair \((H_{p,q},\mu _a)\) play a subtle role in this work as, for example, seen from Proposition 6.16. We do not have a conceptual explanation for this now. The following observation might be helpful when applying this lemma. For a pair \((H_{p,q},\mu _a)\), set \(S^+ (p,q,a) = p - q + 2a\) and \(S^-(p,q,a) = p - q - 2a\). When passing from distingusihed \(\sigma _1 \times \sigma _2\) to distingsuihed \(\rho _1\) and \(\rho _4\), the invariants \(S^+\) and \(S^-\) for the subgroup pairs are preserved respectively.
To handle the duality relation in (5.1), we have the following
Lemma 5.6
Let \(\mathfrak {m}_1,\ldots ,\mathfrak {m}_r\) and \(\mathfrak {n}_1,\ldots ,\mathfrak {n}_s\) be multisegments. If
then \(\mathfrak {m}_1 + \cdots + \mathfrak {m}_r = \mathfrak {n}_1 + \cdots + \mathfrak {n}_s\).
Proof
It is known that \(L (\mathfrak {m}_1 + \cdots + \mathfrak {m}_r)\) is a subquotient of \(L (\mathfrak {m}_1) \times \cdots \times L (\mathfrak {m}_r)\). By our condition, it is then a subquotient of \(\lambda (\mathfrak {n}_1 + \cdots + \mathfrak {n}_s)\). Reversing the roles of \(\mathfrak {m}_i\)’s and \(\mathfrak {n}_j\)’s, the required equality follows from [33, Theorem 7.1] (see also [31, Theorem 5.3]). \(\square \)
As seen from above, the geometric lemma provides us necessary conditions for distinction of induced representations.We now present a sufficient condition that is due to Matringe [22, Proposition 3.8].
Lemma 5.7
Let \(n_1 = 2 m_1\) and \(n_2 = 2 m_2\) be even integers, let \(a \in {\mathbb {R}}\). Assume that \(\pi _1\) is \((H_{m_1,m_1},\mu _a )\)-distinguished and \(\pi _2\) is \((H_{m_2,m_2} , \mu _a)\)-distinguished. Then \(\pi _1 \times \pi _2\) is \((H_{m_1+m_2,m_1+m_2},\mu _a)\)-distinguished.
5.2 Distinction of products of essentially square-integrable representations
We now apply Corollary 5.2 to products of essentially square-integrable representations.
Proposition 5.8
Let \(\pi = \varDelta _1 \times \cdots \times \varDelta _t\) be a representation of \(G_n\), where \(\varDelta _i = \varDelta ([a_i,b_i]_{\rho _i})\) is an essentially square-integrable representation of \(G_{n_i}\), \(i = 1, \ldots ,t\). (Here we assume all \(a_i\), \(b_i\) are integers.) Suppose that \(\pi \) is \((H_{p,q},\mu _a)\)-distinguished with p, q two nonnegative integers, \(p + q =n\), and \(a \in {\mathbb {R}}\). Then there exist an integer \(c_t\) satisfying \(a_t - 1 \leqslant c_t \leqslant b_t\) such that one of the following cases must hold:
Case A1. One has \(a_t = c_t < b_t\). The representation \(\varDelta ([a_t,c_t]_{\rho _t}) = {\textbf {b}}(\varDelta _t) \) is either the character \(\nu ^{a+(q-p+1)/2}\) or the character \(\nu ^{-a + (p-q + 1)/2}\) of \(G_1\); and there exists \(i \in \{ 1, 2, \ldots , t-1\}\), and an integer \(c_{i}\), \(a_{i} \leqslant c_{i} \leqslant b_i\), such that
-
(i)
one has \(\varDelta ([a_t+1,b_t]_{\rho _t})^{\vee } \cong \varDelta ([a_i,c_i]_{\rho _i})\);
-
(ii)
the representation
$$\begin{aligned} \varDelta _1 \times \cdots \times \varDelta ([c_i + 1,b_i]_{\rho _i}) \times \cdots \times \varDelta _{t-1} \end{aligned}$$is \((H_{p-n_t, 1 + q -n_t},\mu _{a + 1/2})\) or \((H_{ 1 +p-n_t, q -n_t},\mu _{a - 1/2})\)-distinguished, depending on \({\textbf {b}}(\varDelta _t)\).
Case A2. One has \(a_t \leqslant c_t < b_t\). The representation \(\varDelta ([a_t,c_t]_{\rho _t})\), with its degree \(n_t'\) an even integer, is \(H_{n_t'/2,n_t'/2}\)-distinguished; and there exists \(i \in \{1, 2,\ldots , t-1\}\) and an integer \(c_{i}\), \(a_{i} \leqslant c_{i} \leqslant b_i\), such that
-
(i)
one has \(\varDelta ([c_t+1,b_t]_{\rho _t})^{\vee } \cong \varDelta ([a_i,c_i]_{\rho _i})\);
-
(ii)
the representation
$$\begin{aligned} \varDelta _1 \times \cdots \times \varDelta ([c_i + 1,b_i]_{\rho _i}) \times \cdots \times \varDelta _{t-1} \end{aligned}$$is \((H_{p', q'},\mu _{a})\) -distinguished with \(p' = p-n_t + n_t'/2\) and \(q' = q - n_t + n_t'/2\).
Case B1. One has \(c_t = b_t\). The representation \(\varDelta ([a_t,c_t]_{\rho _t}) = \varDelta _t\) is either the character \(\nu ^{a+(q-p+1)/2}\) or the character \(\nu ^{-a + (p-q + 1)/2}\) of \(G_1\); and the representation
is \((H_{p-1, q},\mu _{a + 1/2})\) or \((H_{ p, q - 1},\mu _{a - 1/2})\)-distinguished, depending on \(\varDelta _t\).
Case B2. One has \(c_t = b_t\). The representation \(\varDelta _t\) is \(H_{n_t/2,n_t/2}\)-distinguished, where \(n_t\) is even; and the representation
is \((H_{p-n_t/2, q-n_t/2},\mu _a)\)-distinguished.
Case C. One has \(c_t = a_t - 1\). There exists \(i \in \{1,2,\ldots , t-1 \}\) and an integer \(c_{i}\), \(a_{i} \leqslant c_{i} \leqslant b_i\), such that
-
(i)
one has \(\varDelta _t^{\vee } \cong \varDelta ([a_i,c_i]_{\rho _i})\);
-
(ii)
the representation
$$\begin{aligned} \varDelta _1 \times \cdots \times \varDelta ([c_i + 1,b_i]_{\rho _i}) \times \cdots \times \varDelta _{t-1} \end{aligned}$$is \((H_{p-n_t, q -n_t},\mu _a)\)-distinguished.
Proof
Write \(\sigma _1 = \varDelta _1 \times \cdots \times \varDelta _{t-1}\) and \(\sigma _2 = \varDelta _t\), and \(k = n - n_t\). By Corollary 5.2, in its notation, there exists \((r,s) \in \mathcal {I}^k_{p,q}\) with defect \(d = k - r - s\) such that the representation \(r_{(k-d,d)} \sigma _1 \otimes r_{(d,n-k-d)}\sigma _2\) of L is \((L_x, \delta _{Q_x}\delta _Q^{-1/2}\mu _a^{\eta })\)-distinguished. By [33, 9.5], the Jacquet module \(r_{(d,n-k-d)} \sigma _2\) of \(\sigma _2\) is either zero or of the form \(\varDelta ([c_t + 1, b_t]_{\rho _t}) \otimes \varDelta ([a_t,c_t]_{\rho _t})\) for certain integer \(c_t\) with \( a_t - 1 \leqslant c_t \leqslant b_t\). By [33, 1.2, 1.6], there exists a filtration \(0 \subset V_1 \subset \cdots \subset V= r_{(k-d,d)} \sigma _1\) such that each successive factor is equivalent to a representation of the form
for certain integers \(c_i\) such that \(a_i - 1 \leqslant c_i \leqslant b_i\), \(i = 1, \ldots ,t-1\). Therefore, there exists integers \(c_i\), \(i =1 ,2,\ldots ,t\), such that the pure tensor representation
is \((L_x,\delta _{Q_x}\delta _Q^{-1/2}\mu _a^{\eta ^{-1}})\)-distinguished. By Lemma 5.3, we have
By Lemma 5.6, \(c_i = a_i - 1\) for all but one i between 1 and \(t-1\). So, for this i, we have
Lemma 5.3 also implies that
and that
When \(a_t \leqslant c_t < b_t\), we have two subcases. If \(c_t = a_t\) and the degree of \(\rho _t\) equals to 1, it follows from (5.3) that \((p+s-k, q+r-k) = (1,0)\) or (0, 1). By (5.4), (5.2) and simple calculations, we then have Case A1; Otherwise, the representation \(\varDelta ([a_t,c_t]_{\rho _t})\) is not one dimensional. Thus, in (5.3) we have \(p+s-k >0\) and \(q+r-k >0\). By Proposition 3.9, we get that \(\varDelta ([a_t,c_t]_{\rho _t})\) is \(H_{n_t'/2,n_t'/2}\)-distinguished with \(n_t'\) its degree. The rest statements of Case A2 follow from simple calculations. Thus we have Case A2.
When \(c_t = b_t\), we have two subcases. If \(\varDelta _t\) is a character of \(G_1\), then by similar arguments as in Case A1, we have Case B1. Otherwise, by similar arguments as in Case A2, we have Case B2. In these two cases, we have \(d = 0\) and \(c_i = a_i - 1\) by our convention.
When \(c_t = a_t - 1\), by (5.3), we have \(p+s-k = q + r -k = 0\). The statements of Case C follow from (5.4), (5.2) and simple calculations. So we are done. \(\square \)
Corollary 5.9
Let \(\pi = \varDelta _1 \times \cdots \times \varDelta _t\) be as above. If \(\pi \) is \((H_{p,q},\mu _a)\)-distinguished with p, q and a as above, then either the representation \(\varDelta _t\) is the character \(\nu ^{a+(q-p+1)/2}\) or the character \(\nu ^{-a + (p-q + 1)/2}\) of \(G_1\), or there is \(i \in \{ 1, 2 , \ldots , t\}\) such that \({\textbf {e}}(\varDelta _t)^{\vee } \cong {\textbf {b}}(\varDelta _i)\).
Proof
Note that in all cases other than Case B1, we have a duality relation. \(\square \)
Considering the duality relation between extremities of segments, a generalization of Corollary 5.9 is given later in Proposition 6.8.
6 Distinction of ladder representations
6.1 Notations and basic facts
The class of ladder representations was first introduced by Lapid and Mínguez in [17], and was further studied by Lapid and his collaborators in [16] and [18]. We start by reviewing some basic facts of these representations.
6.1.1 Definitions
Let \(\rho \in {\mathscr {C}}\). By a ladder we mean a set \(\{\varDelta _1,\ldots , \varDelta _t \} \in {\mathscr {O}}_{\rho }\) such that
A representation \(\pi \in Irr \) is called a ladder representation if \(\pi = L (\mathfrak {m})\) where \(\mathfrak {m} \in {\mathscr {O}}_{\rho }\) is a ladder. Whenever we say that \(\mathfrak {m} = \{\varDelta _1,\ldots ,\varDelta _t \} \in \mathscr {O}_{\rho }\) is a ladder, we implicitly assume that \(\mathfrak {m}\) is already ordered as in (6.1). We denote by \(\mathfrak {m}^{\vee } \in {\mathscr {O}}_{\rho ^{\vee }}\) the ladder \(\{\varDelta _t^{\vee },\ldots ,\varDelta _1^{\vee }\}\).
Lemma 6.1
Let \(\mathfrak {m} \in {\mathscr {O}}_{\rho }\) be a ladder. One has \(L (\mathfrak {m})^{\vee } = L (\mathfrak {m}^{\vee })\).
Proof
See [30, Proposition 5.6] \(\square \)
We introduce some more notation. For a ladder \(\mathfrak {m} = \{ \varDelta _1,\ldots , \varDelta _t \} \in {\mathscr {O}}_{\rho }\) ordered as in (6.1), set \(\pi = L (\mathfrak {m})\). We shall denote \({\textbf {b}}(\varDelta _1)\) by \({\textbf {b}}(\pi )\), called the beginning of the ladder representaion \(\pi \); denote \({\textbf {e}}(\varDelta _t)\) by \({\textbf {e}}(\pi )\), called the end of \(\pi \). We shall denote the number t of segments in \(\mathfrak {m}\) by \({\textbf {ht}}(\pi )\), called the height of \(\pi \).
We say that \(\pi \) is a decreasing (resp. increasing) ladder representation if
We say that \(\pi \) is a left aligned (resp. right aligned) representation if \({\textbf {b}}(\varDelta _i) = {\textbf {b}}(\varDelta _{i+1}) + 1\) (resp. \({\textbf {e}}(\varDelta _i) = {\textbf {e}}(\varDelta _{i+1}) + 1\) ), \(i= 1,\ldots , t-1\). Note that left aligned repreesentations are decreasing ladder representations and right aligned representations are increasing ladder representations.
A ladder representation is called an essentially Speh representation if it is both left aligned and right aligned. Note that essentially Speh representations are just the usual Speh representations up to twist by a non-unitary character. Let \(\varDelta \) be an essentially square-integrable representation of \(G_d\) and k a positive integer. Then \(\mathfrak {m}_1 = \{\nu ^{(k-1)/2}\varDelta , \nu ^{(k-3)/2}\varDelta , \ldots , \nu ^{(1-k)/2}\varDelta \}\) is a ladder, and the ladder representation \(L (\mathfrak {m}_1)\) is an essentially Speh representation, which we denote by \(\text {Sp}(\varDelta ,k)\). All essentially Speh representations can be obtained in this manner.
Let \(\pi = L (\mathfrak {m})\) as above. Let us further write \(\varDelta _i = \varDelta ([a_i,b_i]_{\rho })\). (The \(a_i\)’s are integers by our convention.) By a division of \(\pi \) as two ladder representations \(\pi '\) and \(\pi ''\), denoted by \(\pi = \pi ' \sqcup \pi ''\), we mean that there exist integers \(c_i\) with \(a_i - 1 \leqslant c_i \leqslant b_i\), \(i = 1, \ldots , t\), such that
and that
Note that if \(\pi \) is an essentially Speh representation and \(\pi = \pi ' \sqcup \pi ''\) with neither \(\pi '\) nor \(\pi ''\) the trivial representation of \(G_0\), then we have \({\textbf {b}}(\pi ) = {\textbf {b}}(\pi ')\) and \({\textbf {e}}(\pi ) = {\textbf {e}}(\pi '')\).
6.1.2 Standard module
One useful property of ladder representations is that the relation between them and their standard modules is explicit. Let \(\mathfrak {m}= \{\varDelta _1,\ldots , \varDelta _t \} \in {\mathscr {O}}_{\rho }\) be a ladder with \(\varDelta _i = \varDelta ([a_i, b_i])_{\rho }\). Set
for \(i = 1, \ldots , t-1\). (By our convention, \(\mathcal {K}_i = 0\) if \(a_i > b_{i+1} + 1\)). By [17, Theorem 1] we have
Proposition 6.2
With the above notation let \(\mathcal {K}\) be the kernel of the projection \(\lambda (\mathfrak {m}) \rightarrow L (\mathfrak {m})\). Then \(\mathcal {K} = \sum _{i=1}^{t-1} \mathcal {K}_i\).
6.1.3 Jacquet modules
The Jacquet modules of ladder representations were computed in [16, Corollary 2.2], where it is shown that the Jacquet module of a ladder representation is semisimple, multiplicity free, and that its irreducible constituents are themselves tensor products of ladder representations. For us, we need only the Jacquet modules with respect to maximal parabolic subgroups. We record the result in [16] here. Let \(P = M < imes U\) be the standard parabolic subgroup of \(G_n\) associated to \((k,n-k)\).
Proposition 6.3
Let \(\mathfrak {m}= \{\varDelta _1,\ldots , \varDelta _t \} \in {\mathscr {O}}_{\rho }\) be a ladder with \(\varDelta _i = [a_i, b_i]_{\rho }\), and \(\pi = L (\mathfrak {m})\). Then
where the summation takes over all divisions of \(\pi \) as two ladder representations \(\pi _1\) and \(\pi _2\) such that the degree of \(\pi _1\) is \( n - k\) and that the degree of \(\pi _2\) is k.
6.1.4 Bernstein–Zelevinsky derivatives
The full derivative of a ladder representation was computed in [17, Theorem 14], where it is shown that the semisimplification of all of the derivatives of a ladder representation consists of ladder representations of smaller groups. In particular, the derivatives of a left aligned representation take simple forms, which we recall here.
Lemma 6.4
Let \(\rho \in {\mathscr {C}} (G_d)\), and \(\mathfrak {m}= \{\varDelta _1,\ldots , \varDelta _t \} \in {\mathscr {O}}_{\rho }\) be a ladder with \(\varDelta _i = \varDelta ([a_i, b_i]_{\rho })\). Suppose that \(\pi = L (\mathfrak {m})\) is a left aligned representation. If k is not divided by d, then \(\pi ^{(k)} = 0\). If \(k = rd\), then
6.2 Distinction of products of essentially Speh representations
In this subsection we apply Corollary 5.2 to products of essentially Speh representations.
Instead of Lemma 5.6, we will use the following lemma to handle the duality relation in consequences of the geometric lemma.
Lemma 6.5
Let \(\sigma \) and \(\pi _i\) be left aligned representations of \(G_n\) and \(G_{n_i}\), \(i = 1, \ldots , k\). If \(\sigma \cong \pi _1 \times \cdots \times \pi _k\), then \(k = 1\).
Proof
By Lemma 6.4, the derivatives of left aligned representations are either 0 or irreducible representations. Our assertion then follows form the description of the derivatives of a product of representations in [1, Corollary 4.6] \(\square \)
In view of Lemma 6.5 and the description of Jacquet modules of a ladder representation in Proposition 6.3, we formulate the following proposition, whose proof is very similar to that of Proposition 5.8 and is omitted here.
Proposition 6.6
Let \(\pi = \pi _1 \times \cdots \times \pi _t \) be a representation of \(G_n\), where \(\pi _i\) is an essentiallly Speh representation of \(G_{n_i}\), \(i = 1 , \ldots , t\). Assume that \(\pi \) is \((H_{p,q},\mu _a)\)-distinguished with p, q two nonnegative integers, \(p + q =n\) and \(a \in {\mathbb {R}}\). Then there exist a division of \(\pi _t\) as two ladder representations \(\pi _t'\) and \(\pi _t''\), \(\pi _t = \pi _t' \sqcup \pi _t''\), with degrees \(n_t'\) and \(n_t''\) respectively, such that one of the following cases must hold:
Case A. The representation \(\pi _t'\) is neither \(\pi _t\) nor the trivial representation of \(G_0\). There exists \(i_0\), \(1 \leqslant i_0 \leqslant t-1\), and a division of \(\pi _{i_0}\) as two ladder representations \(\pi _{i_0}'\) and \(\pi _{i_0}''\), \(\pi _{i_0} = \pi _{i_0}' \sqcup \pi _{i_0}''\), such that
-
(i)
\(\pi _t'\) is \((H_{r,s},\mu _{a+(r-s+q-p)/2})\)-distinguished, for two nonnegative integers \(r, s \geqslant 0\), \( r + s = n_t'\);
-
(ii)
\(\pi _t''^{\vee } \cong \pi _{i_0}'\);
-
(iii)
the representation
$$\begin{aligned} \pi _1 \times \cdots \times \pi _{i_0 -1} \times \pi _{i_0}'' \times \pi _{i_0 +1} \times \cdots \times \pi _{t-1} \end{aligned}$$(6.2)is \((H_{r',s'},\mu _{a+(s'-r'+ p-q)/2})\)-distinguished, for two nonnegative integers \(r'\), \(s' \geqslant 0\), \(r' + s' = n - n_t - n_t''\).
Case B. One has \(\pi _t' = \pi _t\) is \((H_{r,s},\mu _{a+(r-s+q-p)/2})\)-distinguished, for two nonnegative integers r, \(s \geqslant 0\), \( r + s = n_t\), and the representation
is \((H_{r',s'},\mu _{a+(s'-r'+ p-q)/2})\)-distinguished, for two nonnegative integers \(r',s' \geqslant 0\), \(r' + s' = n - n_t\).
Case C. The representation \(\pi _t'\) is the trivial representation of \(G_0\), so \(\pi _t'' = \pi _t\). There exists \(i_0\), \(1 \leqslant i_0 \leqslant t-1\), and a division of \(\pi _{i_0}\) as two ladder representations \(\pi _{i_0}'\) and \(\pi _{i_0}''\), \(\pi _{i_0} = \pi _{i_0}' \sqcup \pi _{i_0}''\), such that
-
(i)
\(\pi _t^{\vee } \cong \pi _{i_0}'\);
-
(ii)
the representation
$$\begin{aligned} \pi _1 \times \cdots \times \pi _{i_0 -1} \times \pi _{i_0}'' \times \pi _{i_0 +1} \times \cdots \times \pi _{t-1} \end{aligned}$$(6.4)is \((H_{r,s},\mu _{a+(s-r+ p-q)/2})\)-distinguished, for two nonnegative integers \(r,s \geqslant 0\), \(r + s = n - 2 n_t \).
Remark 6.7
It is easy to see that Lemma 6.5 fails if one removes the condition that \(\sigma \) is left aligned. This is the reason why we restrict ourselves to products of essentially Speh representations here. Proposition 6.6 makes an inductive proof of the classification result possible as, in many cases, the representations (6.2), (6.3) and (6.4) are still products of essentially Speh representations.
Nevertheless, we have the following proposition for products of ladder representations that is very useful in later arguments.
Proposition 6.8
Let \(\varPi = \pi _1 \times \cdots \times \pi _t\) and \(\varPi ' = \pi '_1 \times \cdots \times \pi '_s\) be two products of ladder representations. If \(\varPi \times \varPi '\) is \((H_{p,q},\mu _a)\)-distinguished for two nonnegative integers p, q, \(p + q =n \) and \(a \in {\mathbb {R}}\), then there are two possibilities here:
-
(1)
\(\varPi \) is \((H_{p_1,q_1},\mu _{a_1})\)-distinguished and \(\varPi '\) is \((H_{p_2,q_2},\mu _{a_2})\)-distinguished for some \(p_i\), \(q_i\) and \(a_i\), \(i = 1, 2\). Here the subscripts \((p_i,q_i,a_i)\), \(i = 1,2\), satisfy
$$\begin{aligned} {\left\{ \begin{array}{ll} p_1 + p_2 = p \\ q_1 + q_2 = q \end{array}\right. } and \, {\left\{ \begin{array}{ll} p_1 - q_1 + 2 a_1 = p - q + 2 a \\ p_2 - q_2 - 2 a_2 = p - q - 2a. \end{array}\right. } \end{aligned}$$ -
(2)
There exist \(i \in \{1,\ldots ,t\}\) and \(j \in \{ 1, \ldots ,s\}\) such that \({\textbf {e}}(\pi '_j)^{\vee } \cong {\textbf {b}}(\pi _i)\).
Proof
This follows from similar arguments of Proposition 5.8 and the following simple implication of Lemma 5.6 when applied to ladder representations. \(\square \)
Lemma 6.9
Let \(\mathfrak {m}_1,\ldots ,\mathfrak {m}_r\) and \(\mathfrak {n}_1,\ldots ,\mathfrak {n}_s\) be ladders. If
then there exist \(i \in \{1,\ldots ,r\}\) and \(j \in \{ 1, \ldots ,s\}\) such that \({\textbf {b}}(L (\mathfrak {m}_i)) \cong {\textbf {b}}(L (\mathfrak {n}_j))\).
Proof
By Lemma 5.6, one has \(\mathfrak {m}_1+ \cdots + \mathfrak {m}_r = \mathfrak {n}_1 + \cdots + \mathfrak {n}_s\). Write \(\mathfrak {m}_i = \{\varDelta _{i,1},\ldots ,\varDelta _{i,k_i} \}\) and \(\mathfrak {n}_j = \{ \varDelta '_{j,1},\ldots , \varDelta '_{j,l_j}\}\) for these i’s and j’s. Let \(\varDelta \) be one segement in \(\sum _i \mathfrak {m}_i\) such that \({\textbf {b}}(\varDelta )\) is maximal, which means that, if for some \(\varDelta _0 \in \sum _i \mathfrak {m}_i\) with \({\textbf {b}}(\varDelta _0)\) lying in the same cuspidal line with \({\textbf {b}}(\varDelta )\), then \({\textbf {b}}(\varDelta _0) \leqslant {\textbf {b}}(\varDelta )\). As these \(\mathfrak {m}_i\)’s are ladders, one has \(\varDelta \in \{\varDelta _{1,1},\ldots ,\varDelta _{r,1} \}\). Also, one has \(\varDelta \in \{\varDelta '_{1,1}, \ldots , \varDelta '_{s,1} \}\). So the lemma follows. \(\square \)
It will turns out that the ordering of representations in a product is important for the geometric lemma approach to distinction problems. The commutativity of a product of two ladder representations was studied by Lapid and Mínguez in [18]. Here we present a special case of their results that is sufficient for our purpose.
Lemma 6.10
Let \(\rho \in {\mathscr {C}}\). Let \(\mathfrak {m}_1, \mathfrak {m}_2 \in \mathscr {O}_{\rho }\) be two ladders, with \(\mathfrak {m}_1 = \{\varDelta _{1,1},\ldots , \varDelta _{1,t_1} \}\) and \(\mathfrak {m}_2 = \{\varDelta _{2,1},\ldots , \varDelta _{2,t_2} \}\). Suppose that \(L (\mathfrak {m}_1)\) is an essentially Speh representation and \(L (\mathfrak {m}_2)\) is a right aligned representation. If \({\textbf {e}}(\varDelta _{1,t_1}) = {\textbf {e}}(\varDelta _{2,t_2})\) and \(t_2 \leqslant t_1\), or \({\textbf {e}}(\varDelta _{1,t_1}) = {\textbf {e}}(\varDelta _{2,t_2})\) and \({\textbf {b}}(\varDelta _{1,t_1}) \leqslant {\textbf {b}}(\varDelta _{2,t_2})\), then \(L (\mathfrak {m}_1) \times L (\mathfrak {m}_2)\) is irreducible and \(L (\mathfrak {m}_1) \times L (\mathfrak {m}_2) = L (\mathfrak {m}_2) \times L (\mathfrak {m}_1)\).
Proof
Note that the results in [18] are expressed in terms of Zelevinsky classification. By the combinatorial description of Zelevinsky involution by Moeglin-Waldspurger [25] (see also [17, Sect. 3.2]), we can rewrite the conditions in the lemma in terms of the Zelevinsky involution \(\mathfrak {m}_1^t\) and \(\mathfrak {m}_2^t\) of \(\mathfrak {m}_1\) and \(\mathfrak {m}_2\). The assertion then follows from Proposition 6.20 and Lemma 6.21 in [18]. \(\square \)
6.3 Distinction of essentially Speh representations
From now on, we shall perform some detailed analysis using our consequences of the geometric lemma.
Proposition 6.11
Let \(\pi \) be an essentially Speh representation of \(G_n\). If \(\pi \) is \((H_{p,q},\mu _a)\)-distinguished for two positive integers p, q with \(p + q = n\) and some \(a \in {\mathbb {R}}\), then \(\pi \) is self-dual.
Proof
Write \(\pi = L (\mathfrak {m})\) with \(\mathfrak {m}= \{\varDelta _1,\ldots ,\varDelta _t \}\) a ladder. The case where \(\pi \) is one dimensional is obvious. So we assume that \(\pi \), hence \(\varDelta _t\), is not one dimensional. By assumption, \(\varDelta _1 \times \cdots \times \varDelta _t\) is \((H_{p,q},\mu _a)\)-distinguished. By Corollary 5.9, there exists i, \(1 \leqslant i \leqslant t\), such that
We claim that \(i = 1\). If so, by Lemma 6.1, we see that \(\pi \) is self-dual. In fact, if otherwise \(i > 1\), we apply Proposition 6.8 to \(\pi _1 \times \pi _2\), where \(\pi _1 = \varDelta _1 \times \cdots \times \varDelta _{i-1}\) and \(\pi _2 = \varDelta _i \times \cdots \times \varDelta _t\). We get either that
for some j, \( 1 \leqslant j \leqslant i-1\) and some k, \( i \leqslant k \leqslant t\), or that \(\pi _1\) is \((H_{p_1,q_1},\mu _{a_1})\)-distinguished with some \(p_1,q_1\) and \(a_1\), which implies, using Corollary 5.9 again, that
for some \(l, 1\leqslant l \leqslant i-1\). But we see easily that both (6.6) and (6.7) contradict with (6.5). \(\square \)
Corollary 6.12
Let \(\pi \) be an essentially Speh representation of \(G_n\). If the representation \(\pi \) is \((H_{p,q},\mu _a)\)-distinguished for two positive integers p, q with \(p + q = n\) and some \(a \in {\mathbb {R}}\), \(a \ne 0\), then \(p = q\).
Proof
This follows from Proposition 6.11 and consideration of the central character of \(\pi \). \(\square \)
Now we are in a position to prove one direction of Theorem 1.1 (what we actualy prove is slightly more). The arguments involve an application of the theory of Bernstein-Zelevinsky derivatives.
Proposition 6.13
Let \(\pi = \text {Sp}(\varDelta ,l)\) be an essentially Speh representation of \(G_n\), where \(\varDelta \) is an essentially square-integrable representaion of \(G_d\), \(d > 1\), and l is a positive integer. Assume that \(\pi \) is \(H_{p,q}\)-distinguished or \((H_{p,q},\mu _{-1/2})\) for two positive integers p, q, \(p + q = n\). Then the degree d of \(\varDelta \) is even, and \(\varDelta \) is \(H_{d/2,d/2}\)-distinguished; also one has \(p = q\).
Proof
We prove this by induction on l. The case \(l = 1\) follows from Proposition 3.9. Suppose that \(\pi \) is \((H_{p,q},\mu _a)\)-distinguished with \(a = 0\) or \(-1/2\). By Proposition 6.11 we know that \(\pi \) is self-dual, hence \(\varDelta \) is also self-dual. Note that \(\pi \) is irreducible. By Lemma 3.1, we may assume that \(p \geqslant q\). By the assumption on \(\pi \), we have
where \(a = 0\) or \(-1/2\). By [1, Sect. 3.5], the restriction \(\pi |_{P_n}\) of \(\pi \) to \(P_n\) has a filtration which has composition factors \((\varPhi ^+)^{i-1}\varPsi ^+ (\pi ^{(i)})\), \(i = 1,\ldots , h\), where \(\pi ^{(h)}\) is the highest derivative of \(\pi \). We first analyze linear functionals on these factor sapces using the theory of Bernstein-Zelevinsky derivatives.
-
(1)
When \(i = 2k\) is even. If \( q > k\) and \(p > k-1\), by applying (3.3) repeatly, we have
$$\begin{aligned} \mathrm {Hom}_{P_n \cap H_{p,q}} ((\varPhi ^+)^{i-1}\varPsi ^+ (\pi ^{(i)}), \mu _a)&\cong \mathrm {Hom}_{P_{n-i+1} \cap H_{q-k,p-k+1}}(\varPsi ^+(\pi ^{(i)}),\mu _{-a-1/2}) \nonumber \\&\cong \mathrm {Hom}_{ H_{q-k,p-k}} (\nu ^{1/2}\pi ^{(i)},\mu _{-a-1/2}). \end{aligned}$$(6.8)Otherwise, there exists \(i_0 \geqslant 0\) such that
$$\begin{aligned} \mathrm {Hom}_{P_n \cap H_{p,q}} ((\varPhi ^+)^{i-1}\varPsi ^+ (\pi ^{(i)}), \mu _a)&\cong \mathrm {Hom}_{P_{n-i+i_0+1} }((\varPhi ^+)^{i_0}\varPsi ^+(\pi ^{(i)}),\mu _{a'}), \end{aligned}$$(6.9)where \(a' = a\) or \(-a - 1/2\) depending on \(i_0\) odd or even.
-
(2)
When \(i = 2k + 1\) is odd. If \( q > k\) and \(p > k\), by applying (3.3) repeatly, we have
$$\begin{aligned} \mathrm {Hom}_{P_n \cap H_{p,q}} ((\varPhi ^+)^{i-1}\varPsi ^+ (\pi ^{(i)}), \mu _a)&\cong \mathrm {Hom}_{P_{n-i+1} \cap H_{p-k,q-k}}(\varPsi ^+(\pi ^{(i)}),\mu _{a})\nonumber \\&\cong \mathrm {Hom}_{H_{p-k,q-k-1}} (\nu ^{1/2}\pi ^{(i)},\mu _{a}). \end{aligned}$$(6.10)Otherwise, there exists \(i_0 \geqslant 0\) such that
$$\begin{aligned} \mathrm {Hom}_{P_n \cap H_{p,q}} ((\varPhi ^+)^{i-1}\varPsi ^+ (\pi ^{(i)}), \mu _a)&\cong \mathrm {Hom}_{P_{n-i+i_0+1} }((\varPhi ^+)^{i_0}\varPsi ^+(\pi ^{(i)}),\mu _{a'}), \end{aligned}$$(6.11)where \(a' = a\) or \(-a - 1/2\) depending on \(i_0\) even or odd.
We claim that the factor spaces corresponding to non-highest derivatives contribute nothing, that is, we have
We shall discuss separately according to i is even or odd, \(a = 0\) or \(-1/2\). Note first that, by Lemma 6.4, when \(1 \leqslant i < h\), the i-th derivative \(\pi ^{(i)}\) is either 0 or a ladder representation of the form
where \(\varDelta _1\) is a subsegment of \(\nu ^{(l-1)/2}\varDelta \) obtained by discarding the first few terms. In particular, \(\pi ^{(i)}\) is either 0 or an irreducible representation. Thus, if we are in the case where (6.9) or (6.11) holds, then
as the representation \((\varPhi ^+)^{i_0}\varPsi ^+(\pi ^{(i)})\) is either 0 or an irreducible representation of \(P_{n-i+i_0+1}\) that is not one dimensional by [1, 3.3 Remarks].
Now we deal with the case where (6.8) or (6.10) holds. Note that, from (6.13), \(\nu ^{1/2}\pi ^{(i)}\) either is 0 or can be realized as the unique irreducible quotient of a representation of the form \(\nu ^{1/2}\varDelta _1 \times \text {Sp}(\varDelta ,l-1)\) with \(\varDelta _1\) as above. We discuss as follows.
Case (1) where \(a = 0\) and \(i = 2k\) is even. By (6.8), it suffices to show that
Assume, on the contrary, that \(\nu ^{1/2}\varDelta _1 \times \text {Sp}(\varDelta ,l-1)\) is \((H_{q-k,p-k},\mu _{-1/2})\)-distinguished. As \(\varDelta \) is self-dual, \({\textbf {e}}(\text {Sp}(\varDelta ,l-1))^{\vee } = {\textbf {b}}(\text {Sp}(\varDelta ,l-1)) \ne {\textbf {b}}(\nu ^{1/2}\varDelta _1)\). So, by Proposition 6.8,
and
for some nonnegative integers r and s. If the degree of \(\nu ^{1/2}\varDelta _1\) is greater than 1, then \(\nu ^{1/2}\varDelta _1\) is self-dual by Proposition 3.9. This is absurd because the central character of \(\nu ^{1/2}\varDelta _1\) has positive real part; If the degree of \(\nu ^{1/2}\varDelta _1\) is 1, then \((r,s) = (1,0)\) or (0, 1). If \(r =1\) and \(s = 0\), then \(\text {Sp}(\varDelta , l -1)\) is \((H_{q-k-1,p-k},\mu _{-1})\)-distinguished. Thus we have \( p = q - 1\) by Corollary 6.12. This is absurd as we have assumed that \(p \geqslant q\); If \(r = 0\) and \( s = 1\), then \(\nu ^{1/2}\varDelta _1\) is the character \(\nu ^{(p-q)/2}\) of \(G_1\) and \(\text {Sp}(\varDelta ,l-1)\) is \((H_{q-k,p-k-1},{\textbf {1}})\)-distinguished. So, by induction hypothesis, we have \(p - 1 = q\). This implies that \({\textbf {e}}(\nu ^{(l-1)/2}\varDelta ) = {\textbf {e}}(\varDelta _1) = {\textbf {1}}\), the trivial character of \(G_1\). This is impossible as \(\varDelta \) is self-dual and its degree d is greater than 1.
Case (2) where \(a = 0\) and \(i = 2k+1\) is odd. In this case we see easily that
as the central character of \(\nu ^{1/2}\pi ^{(i)}\) has positive real part when \(i < h\).
The arguments for the remaining two cases where \(a = -1/2\), i is even or odd are similar to those of the above two cases and are omitted here. So we have proved (6.12).
By Lemma 6.4, we know that the highest derivative of \(\pi \) is \(\pi ^{(d)}\) and \(\nu ^{1/2}\pi ^{(d)} = \text {Sp}(\varDelta ,l-1)\). Now we have
where \(a = 0\) or \(-1/2\). We analyze the left hand side of (6.16) as above. The cases (6.9) and (6.11) cannot happen by the same arguments as above. The case (6.10) cannot happen by induction hypothesis and the fact that \(p \geqslant q\). So the only possible case is when (6.8) holds, that is, d is even and
Note that when \(a = 0\) or \(-1/2\), \(-a-1/2 = -1/2\) or 0. Thus we are done by induction hypothesis. \(\square \)
We have the following generalization of Corollary 3.8 to essentially Speh representations.
Corollary 6.14
Let \(\pi \) be an essentially Speh representation of \(G_n\) that is not one dimensional. If \(\pi \) is \((H_{p,q},\mu _a)\)-distinguished for two positive integers p, q with \(p + q = n\) and \(a \in {\mathbb {R}}\), then we have \(p = q\).
Proof
The case \(a \ne 0\) is Corollary 6.12. The case \(a = 0\) follows from Proposition 6.13. \(\square \)
Remark 6.15
We postpone the proof of the other direction of Theorem 1.1 in Sect. 7.1.
6.4 Distinguished left aligned representations
The results of this subsection are used only in Sect. 7.2 where we classify distinguished representations that are products of Speh representations. The analysis in this subsection is quite involved; the readers can skip it for the fisrt reading.
The purpose of this subsection is to show the following
Proposition 6.16
Let \(\pi \) be a left aligned (resp. right aligned) representation of \(G_n\). If \(\pi \) is \((H_{p,q},\mu _{(p-q)/2})\) (resp. \((H_{p,q},\mu _{(q-p)/2})\))-distinguished for two nonnegative integers p, q, \(p + q =n\), then \(\pi \) is an essentially Speh representation.
We need the following technical lemmas. When the supercuspidal representations in the support of the left aligned representation have degree greater than 1, we can prove slightly more.
Lemma 6.17
Let \(\rho \in {\mathscr {C}}(G_d)\), \(d > 1\), and \(\mathfrak {m} = \{\varDelta _1, \ldots , \varDelta _t \} \in {\mathscr {O}}_{\rho }\) be a ladder. Assume that \(\pi = L (\mathfrak {m})\) is a decreasing or an increasing ladder representation of \(G_n\). If \(\pi \) is \((H_{p,q},\mu _a)\)-distinguished for two positive integers p, q, \(p + q =n\) and some \(a \in {\mathbb {R}}\), then all the \(l(\varDelta _i)\)’s are the same. Moreover, \(\pi \) is self-dual.
Proof
Note that \(\pi \) is irreducible. By Lemma 3.1, passing to contragradient if necessary, we may assume that \(l(\varDelta _1) \leqslant l(\varDelta _2) \leqslant \cdots \leqslant l(\varDelta _t)\).By our assumption, the representation \(\varDelta _1 \times \cdots \times \varDelta _t\) is \((H_{p,q},\mu _a)\)-distinguished. We now appeal to Proposition 5.8. Write \(\varDelta _i = \varDelta ([a_i,b_i]_{\rho })\), \(i = 1, 2, \ldots ,t\). Note that by our assumption that \(d > 1\), Case A1 and Case B1 cannot happen.
Case A2. In this case We have \(a_t \leqslant c_t < b_t\), and \(\varDelta ( [a_t,c_t]_{\rho })\) is self-dual. Thus we have \(\nu ^{a_t} \rho \cong \nu ^{-c_t} \rho ^{\vee }\), and consequently \((a_t + c_t)d + 2 \Re (w_{\rho }) = 0\). We also have \(\varDelta ( [c_t+1,b_t]_{\rho } )^{\vee } \cong \varDelta ([a_i,c_i]_{\rho })\) for some \(i < t\) and \(c_i \geqslant a_i\). Thus we get \(\nu ^{a_i}\rho \cong \nu ^{-b_t}\rho ^{\vee }\), and then \((a_i + b_t)d + 2 \Re (w_{\rho }) = 0\). But this is absurd because \(a_i > a_t\) and \(b_t > c_t\).
Case B2. In this case we have \(c_t = b_t\), and \(\varDelta ( [a_t,b_t]_{\rho })\) is self-dual. Thus we have \(\nu ^{a_t} \rho \cong \nu ^{-b_t} \rho ^{\vee }\), and consequently \((a_t + b_t)d + 2 \Re (w_{\rho }) = 0\). We also have \(\varDelta _1 \times \cdots \times \varDelta _{t-1}\) is \((H_{p',q'},\mu _{a'})\)-distinguished for some \(p'\), \(q'\) and \(a'\). If \(t = 1\), there is nothing to be proved. If \(t > 1\), by Corollay 5.9, we get that \((\nu ^{b_{t-1}}\rho )^{\vee } \cong \nu ^{a_i}\rho \) for some \(1 \leqslant i \leqslant t-1\). Thus we get \((a_i + b_{t-1})d + 2 \Re (w_{\rho }) = 0\). This is absurd because \(a_i > a_t \) and \(b_{t-1} > b_t\).
So the only possible case is Case C. We then have \(\varDelta ([a_t,b_t])^{\vee } \cong \varDelta ([a_i,c_i] )\) for \(i < t\) and certain \( a_i \leqslant c_i \leqslant b_i\). Note that, by our assumption, we have \(l(\varDelta _i) \leqslant l(\varDelta _t)\). Thus we have \(l(\varDelta _i) = l(\varDelta _t)\). We claim that \(i = 1\). If so, all \(l(\varDelta _i)\)’s will be the same by our assumption. Indeed, if \(i > 1\), consider the \((H_{p,q},\mu _a)\)-distinguished representaion
By Propostion 6.8, either we have \({\textbf {e}}(\varDelta _{i-1})^{\vee } \cong {\textbf {b}}(\varDelta _a)\) with \(1 \leqslant a \leqslant t-1\), or we have \({\textbf {e}}(\varDelta _c)^{\vee } \cong {\textbf {b}}(\varDelta _b)\) with \(1 \leqslant b \leqslant t-1\) and \(i \leqslant c \leqslant t\). We then get a contradiction as in Case A2 or B2. The assertion on the self-dualness property follows from a repeated analysis as above. \(\square \)
If we drop the assumption that \(d > 1\), the argument becomes complicated by the possible occurrence of Case A1 or Case B1 when applying Proposition 5.8. We have the following result on the shape of right aligned representations when it is distinguished.
Lemma 6.18
Let \(\rho \) be a character of \(G_1\), and \(\mathfrak {m} \in \mathscr {O}_{\rho }\) be a ladder. Assume that \(\pi = L (\mathfrak {m})\) is a right aligned representation of \(G_n\). If \(\pi \) is \((H_{p,q},\mu _a)\)-distinguished for two nonnegative integers p, q, \(p + q =n\) and some \(a \in {\mathbb {R}}\), then either
-
(1)
we have
$$\begin{aligned} \mathfrak {m}= \{ \varDelta _1,\ldots ,\varDelta _{i_1}, \varDelta _{i_1 + 1}, \ldots , \varDelta _{i_1 + i_2}, \varDelta _{i_1 + i_2 +1}, \ldots ,\varDelta _{i_1 + i_2 +i_3} \} \end{aligned}$$(6.17)with \(i_1\), \(i_2\) and \(i_3 \geqslant 0\), such that \(l(\varDelta _k) = 1\) when \(1\leqslant k \leqslant i_1\), \(l(\varDelta _{i_1 + k}) = l > 1\) when \(1 \leqslant k \leqslant i_2\), \(l(\varDelta _{i_1 + i_2 + k})) = l + 1\) when \(1 \leqslant k\leqslant i_2\), and that \({\textbf {e}}(\varDelta _{i_1 + i_2 + i_3})^{\vee } \cong {\textbf {b}}(\varDelta _{i_1 + 1})\) (See Fig. 1 for an example), or
-
(2)
we have
$$\begin{aligned} \mathfrak {m}= \{\varDelta _1,\ldots , \varDelta _{i_1}, \varDelta _{i_1 + 1}, \ldots ,\varDelta _{i_1 + i_2} \} \end{aligned}$$(6.18)with \(i_1\) and \(i_2 >0\), such that \(l(\varDelta _k) = 1\) when \(1 \leqslant k \leqslant i_1\), \(l(\varDelta _{i_1 + k}) = 2\) when \(1 \leqslant k \leqslant i_2\), and that \({\textbf {e}}(\varDelta _{i_1 + i_2})^{\vee } \cong {\textbf {b}}(\varDelta _1)\) (See Figure 2 for an example).
Proof
Write \(\mathfrak {m}= \{\varDelta _1,\ldots ,\varDelta _t \}\). If \(l(\varDelta _t) =1\), then \(\pi \) is a one dimensional representation and \(\mathfrak {m}\) is of the form (6.17) with \(i_2 = i_3 = 0\). If \(l(\varDelta _t) = l(\varDelta _1) =2 \), then \(\pi \) is an essentially Speh representation. It follows from Proposition 6.11 that \(\mathfrak {m}\) is of the form (6.17) with \(i_1 = i_3 = 0\). If \(l(\varDelta _t) = 2\) and \(l(\varDelta _1) = 1\), then \(\pi \) can be realized as the unique irreducible quotient of \(\pi _1 \times \pi _2\), where \(\pi _1\) is a one dimensional representation and \(\pi _2\) is an essentially Speh representation of length 2. Thus \(\pi _1 \times \pi _2\) is \((H_{p,q},\mu _a)\)-distinguished. By Proposition 6.6, \(\mathfrak {m}\) is either of the form (6.18) (Case A), or of the form (6.17) with \(i_3 = 0\), \(i_1 > 0\), \(i_2 > 0\) and \(l = 2\) (Case B and Proposition 6.11). Note that here Case C is impossible by our assumption on \(\pi _1\) and \(\pi _2\). If \(l(\varDelta _t) > 2\), then we apply Proposition 5.8 to the product \(\varDelta _1 \times \cdots \times \varDelta _t\) and discuss case by case. Note first that Case A2 cannot happen by similar arguments as those in Lemma 6.17; Case B1 cannot happen by our assumption on \(\varDelta _t\). In the remaining cases, it follows from Corollary 5.9, Proposition 6.8 and arguments similar to those in Lemma 6.17 that \(\mathfrak {m}\) is of the form (6.17). \(\square \)
The following lemma is a simple consequence of Lemma 3.2.
Lemma 6.19
Let \(\pi \) be a one dimensional representation of \(G_n\). If \(\pi \) is \((H_{p,q},\mu _{(q-p)/2})\)-distinguished with \(p + q = n\), then \(\pi \) is either the trivial character \({\textbf {1}}\) of \(G_n\) or the character \(\nu ^{-n/2}\) of \(G_n\). In particular, \({\textbf {b}}(\pi )\) is either \(\nu ^{(n-1)/2}\) or \(\nu ^{-1/2}\) of \(G_1\).
As shown in Lemma 6.18, there are two possibilities for the shape of distinguished right aligned representations. Now we remove one possibility if we impose some restriction on the subscripts (p, q, a).
Lemma 6.20
Keep the notation as in Lemma 6.18, let \(\pi = L (\mathfrak {m})\) with \(\mathfrak {m}\) of the form (6.18). Then \(\pi \) cannot be \((H_{p,q},\mu _{(q-p)/2})\)-distinguished.
Proof
We assume on the contrary that \(\pi \) is \((H_{p,q},\mu _{(q-p)/2})\)-distinguished. By part (1) of Lemma 3.1, we may assume that \(p \leqslant q\). Note that \(\pi \) can be realized as the unique quotient of \(\pi _1 \times \pi _2\), where \(\pi _1\) is a one dimensional representation, \(\pi _2\) is an essentially Speh representation of length 2, and \({\textbf {e}}(\pi _2)^{\vee } \cong {\textbf {b}}(\pi _1)\). Thus, \(\pi _1 \times \pi _2\) is \((H_{p,q},\mu _{(q-p)/2})\)-disintuished. By Proposition 6.6, there exist divisions of \(\pi _1\) and \(\pi _2\), \(\pi _1 = \pi _1' \sqcup \pi _1''\) and \(\pi _2 = \pi _2' \sqcup \pi _2''\) respectively, such that, among other things, \(\pi _1''\) is \((H_{r,s},\mu _{(s-r)/2})\)-distinguished for two nonnegative integers r and s. Note that \(\pi _1'\) is not the trivial representation of \(G_0\) by our assumption on \(\pi \) and Proposition 6.11. We shall discuss further according to the values of r and s.
-
(1)
If exactly one of r and s is 0, then \(\pi _1''\) is the character \(\nu ^{-n_1''/2}\) of \(G_{n_1''}\). Thus \({\textbf {b}}(\pi _1'') = \nu ^{-1/2}\), \({\textbf {e}}(\pi _1') = \nu ^{1/2}\). By Proposition 6.6, we also have \(\pi _1'^{\vee } \cong \pi _2''\). Thus \({\textbf {b}}(\pi _2'') = {\textbf {e}}(\pi _1')^{\vee } = \nu ^{-1/2} = {\textbf {b}}(\pi _1'')\), which is absurd.
-
(2)
If \(r > 0\) and \(s > 0\), then \(\pi _1''\) is the character \({\textbf {1}}\) of \(G_{2r}\), that is, \({\textbf {b}}(\pi _1'') = \nu ^{r - 1/2}\) and \({\textbf {e}}(\pi _1'') = \nu ^{-r + 1/2}\). So, \({\textbf {e}}(\pi _1') = {\textbf {b}}(\pi _1'') + 1 = \nu ^{r+1/2}\). By Proposition 6.6, we have \(\pi _1'^{\vee } \cong \pi _2''\). So \({\textbf {b}}(\pi _2'') = \nu ^{-r-1/2}\). By our assumption on the shape of \(\mathfrak {m}\), this implies that \(\pi _2'\) is also a one dimensional representation which, by Proposition 6.6, is \((H_{r',s'},\mu _{(r' -s')/2+q - p})\)-distinguished for certain nonnegative integers \(r'\) and \(s'\) and that \({\textbf {b}}(\pi _2') = {\textbf {b}}(\pi _2'') - 1 = \nu ^{-r-3/2}\). One of \(r'\) and \(s'\) has to be 0. Recall that we have assumed that \(p \leqslant q\). We then see easily that \(\pi _2'\) is the character \(\nu ^{p-q + n' /2}\) of \({\mathrm {GL}}_{n'}\). Note that we have an equality of central characters, \(\omega _{\pi _2'} = \omega _{\pi }\). This implies that
$$\begin{aligned} n'(p-q + n' /2) = - (p-q)^2 / 2. \end{aligned}$$So, \(n' = q - p\) and \({\textbf {b}}(\pi _2') = \nu ^{-1/2}\). This is absurd as we have shown that \({\textbf {b}}(\pi _2') = \nu ^{-r - 3/2}\) with r a positive integer.
-
(3)
If \(r = s =0\), we have two subcases according to whether or not \(\pi _2'\) is a one dimensional representaiton. If it is, we get a contradiction by exactly the same arguments as in (2) with r being repalced by 0. If it is not, it follows from the duality relations in Lemma 6.18, applied to the contragradient of \(\pi _2'\), that \({\textbf {b}}(\pi _2')^{\vee }\) is either \({\textbf {b}}(\pi _2'') + 1\) or \({\textbf {e}}(\pi _2'') - 1\). But, note that \({\textbf {b}}(\pi _2') = {\textbf {e}}(\pi _1') - 2\). It follows from the relation \(\pi _1'^{\vee } \cong \pi _2''\) that \({\textbf {b}}(\pi _2') ^{\vee } = {\textbf {b}}(\pi _2'') + 2\). This is absurd as \(\pi _2''\) is one dimensional and \({\textbf {b}}(\pi _2'') \geqslant {\textbf {e}}(\pi _2'')\).
\(\square \)
Proof of Proposition 6.16
By part (2) of Lemma 3.1, we only need to prove the statement for left aligned representations. We may further assume that \(p \leqslant q\) by part (1) of Lemma 3.1. By Lemma 6.17, we may write \(\pi = L (\mathfrak {m})\) with \(\mathfrak {m} \in {\mathscr {O}}_{\rho }\) a ladder and \(\rho \) a character of \(G_1\). By considering the contragredient \(\pi ^{\vee } = L (\mathfrak {m}^{\vee })\), we see from Lemma 6.20 that \(\mathfrak {m}^{\vee }\) is of the form (6.17). So, we may write
with \(i_1\), \(i_2\) and \(i_3 \geqslant 0\), such that \(l(\varDelta _k) = l_1 > 2\) when \(1\leqslant k \leqslant i_1\), \(l(\varDelta _{i_1 + k}) = l_1 -1\) when \(1 \leqslant k \leqslant i_2\), \(l(\varDelta _{i_1 + i_2 + k})) = 1 \) when \(1 \leqslant k\leqslant i_3\), and that \({\textbf {e}}(\varDelta _{i_1 + i_2 })^{\vee } \cong {\textbf {b}}(\varDelta _1)\).
We may as well assume that \(i_1\) and \(i_2\) are not all zero. Our first step is to show that \(i_3 = 0\). If not so, we realize \(\pi \), in the obvious way, as the unique irreducible quotient of \(\pi _1 \times \pi _2 \times \pi _3\) with \(\pi _i\) an essentially Speh representation for each i, such that \(\pi _3\) is a character of \(G_{n_3}\), \(n_3 > 0\), and that at least one of \(\pi _1\) and \(\pi _2\) is not the trivial representation of \(G_0\). By our assumption on \(\pi \), the representation \(\pi _1 \times \pi _2 \times \pi _3\) is \((H_{p,q},\mu _{(p-q)/2})\)-distinguished. Note that as \(i_3 > 0\), \({\textbf {e}}(\pi _3)\) is not dual to \({\textbf {b}}(\pi _1)\) or \({\textbf {b}}(\pi _2)\). So, by Proposition 6.8, \(\pi _3\) is \((H_{r,s},\mu _{(r-s)/2})\)-distinguished with respect to two nonnegative integers r and s. As \(\pi _3\) is one dimensional, \(\pi _3\) is either the trivial represntation \({\textbf {1}}\) of \(G_{n_3}\) or the character \(\nu ^{n_3/2}\) of \(G_{n_3}\). In particular, \({\textbf {b}}(\varDelta _{i_1 + i_2 + 1}) = \nu ^{(n_3 - 1)/2}\) or \(\nu ^{n_3 - 1/2}\). But this will contradict with the fact that \({\textbf {e}}(\varDelta _{i_1 + i_2 })^{\vee } \cong {\textbf {b}}(\varDelta _1)\).
Our next step is to show that \(i_1 = 0\) or \(i_2 = 0\). Assume on the contrary that \(i_1 > 0\) and \(i_2 > 0\). By our assumption on \(\pi \), the representation \(\pi ^{\vee } = L (\mathfrak {m}^{\vee })\) is \((H_{p,q},\mu _{(q-p)/2})\)-distinguished. Thus, the representation
is \((H_{p,q},\mu _{(q-p)/2})\)-distinguished. By Proposition 5.8, we deduce that \({\textbf {b}}(\varDelta _1^{\vee }) \cong {\textbf {e}}(\varDelta _1)^{\vee }\) is the character \(\nu ^{q-p + 1/2}\) or \(\nu ^{p-q + 1/2}\) of \(G_1\). (This is the consequence of Case A1; Case A2 and Case B2 are eliminated by arguments similar to those in Lemma 6.17; Case B1 and Case C are eliminated by our assumptions.) It follows easily from the condition \({\textbf {b}}(\varDelta _1) \cong {\textbf {e}}(\varDelta _{i_1 + i_2})^{\vee }\) and the assumption \(p \leqslant q\) that \({\textbf {e}}(\varDelta _1)^{\vee } = \nu ^{p -q + 1/2}\). Hence we have \({\textbf {e}}(\varDelta _1) = \nu ^{q - p - 1/2}\).
We show that \(i_1 = q - p\) and \({\textbf {e}}(\varDelta _{i_1}) = \nu ^{1/2}\) by consideration on the central character of \(\pi \). In fact, on the one hand, we see from the assumption on \(\mathfrak {m}\) and the fact \({\textbf {e}}(\varDelta _1) = \nu ^{ q - p -1/2}\) that the central character \(w_{\pi }\) of \(\pi \) is \(\nu ^a\) where \(a =(q-p)i_1 - i_1^2/2\); on the othe hand, as \(\pi \) is \((H_{p,q},\mu _{(p-q)/2})\)-distinguished, we have \(w_{\pi } = \nu ^{a'}\) where \(a' = (q-p)^2 /2 \). Thus the assertion follows. Also, from the fact that \({\textbf {e}}(\varDelta _{i_1 + i_2 })^{\vee } \cong {\textbf {b}}(\varDelta _1)\), we get that \({\textbf {b}}(\varDelta _1) = \nu ^{i_2 + 1/2}\). Thus, \(l(\varDelta _1) = q - p -i_2 >2\), in particular \(i_1 > i_2\).
Now, as in the first step, we have that \(\pi _1 \times \pi _2\) is \((H_{p,q},\mu _{(p-q)/2})\)-distinguished, where \(\pi _1 = L (\varDelta _1,\ldots ,\varDelta _{i_1})\) and \(\pi _2 = L (\varDelta _{i_1 + 1}, \ldots , \varDelta _{i_1 + i_2})\). We appeal to Proposition 6.6, and claim that Case A and Case B cannot happen. In fact, if Case A or Case B happens, there will be a division of \(\pi _2\) as \(\pi _2'\) and \(\pi _2''\), where \(\pi _2'\) is not the trivial representation of \(G_0\), such that \(\pi _2'\) is \((H_{r,s},\mu _{(r-s)/2})\)-distinguished for two nonnegative integers r and s. In particular, the central character \(w_{\pi _2'}\) of \(\pi _2'\) has nonnegative real part. But this will contradict with the fact that \({\textbf {e}}(\varDelta _{i_1 + 1}) = \nu ^{-3/2}\). So, there exists a division of \(\pi _1\) as two ladder representations \(\pi _1'\) and \(\pi _1''\) such that \(\pi _2 \cong \pi _1'^{\vee }\) and that \(\pi _1''\) is \((H_{p-n_2,q-n_2},\mu _{(p-q)/2})\)-distinguished. Note that \(\pi _1''\) is a right aligned representation, and is not a one dimensional representation due to the fact that \(i_1 > i_2\). By Lemma 6.18, we then get a contradiction as we can check easily that the ladder \(\mathfrak {m}_1''\) of \(\pi _1''\) is not of the form (6.17) or (6.18). \(\square \)
7 Distinction in the unitary dual
7.1 The case of Speh representations
We now classify distinguished Speh representations in terms of distinguished discrete series. In fact, we will do it for essentially Speh representations.
Theorem 7.1
Let \(n = 2m\), and \(\text {Sp}(\varDelta ,k)\) be an essentially Speh representation of \(G_n\), where \(\varDelta \) is an essentially square-integrable representation of \(G_d\) with \(d > 1\), and k is a positive integer. Then \(\text {Sp}(\varDelta ,k)\) is \(H_{m,m}\)-distinguished if and only if d is even and \(\varDelta \) is \(H_{d/2,d/2}\)-distinguished.
Proof
One direction has been proved in Proposition 6.13. We now assume that d is even and that \(\varDelta \) is \(H_{d/2,d/2}\)-distinguished. By [26, Proposition 7.2], which is based on the work of Blanc and Delorme [2], the representation
is \(H_{m,m}\)-distinguished. (The distinguishedness of \(\varDelta \) is unnecessary when k is even).We have the following exact sequence of representations of \(G_n\),
where the kernel \(\mathcal {K} = \sum _{i=1}^{k-1} \mathcal {K}_i\) is given explicitly in Proposition 6.2. To show that \(\text {Sp}(\varDelta ,k)\) is \(H_{m,m}\)-distinguished, it suffices to show that each \(\mathcal {K}_i\) is not \(H_{m,m}\)-distinguished. Write the representation (7.1) as \(\varDelta ([a_1,b_1]_{\rho }) \times \cdots \times \varDelta ([a_k,b_k]_{\rho })\), here the cuspidal representation \(\rho \) is taken to be self-dual and thus \(a_i\) and \(b_i\), \(i = 1, 2,\ldots ,k\) need not be integers. So we have
We further omit the subscript \(\rho \) in the sequel. Recall that, by Proposition 6.2,
If \(i+1 \leqslant (k+1)/2\) and \(\mathcal {K}_i\) is \(H_{m,m}\)-distinguished, by applying Proposition 5.8 repeatly, we get that \(\varDelta ([a_{i+1},b_i]) \times \varDelta ([a_i,b_{i+1}]) \times \cdots \times \varDelta ([a_{k+1-i},b_{k+1-i}])\) is \(H_{m',m'}\)-distinguished for certain \(m'\). (In each step, only Case C is possible.) When we apply Proposition 5.8 once again, still, only Case C is possible. But this is absurd as \(l(\varDelta ([a_i,b_{i+1}])) < l(\varDelta )\). Similar arguments can show that \(\mathcal {K}_i\) is not \(H_{m,m}\)-distinguished if \(i \geqslant (k+1)/2\).
The remaining case is when k is even and \(i = k/2\). In what follows, to save notation, we sometimes write H-distinguished for \(H_{m',m'}\)-distinguished when there is no need to address \(m'\). If \(\mathcal {K}_i\) is \(H_{m,m}\)-distinguished, by applying Proposition 5.8 repeatly, we get that \(\varDelta ([a_{i+1},b_i]) \times \varDelta ([a_i,b_{i+1}])\) is H-distinguished. This in turn implies that both \(\varDelta ([a_{i+1},b_i])\) and \(\varDelta ([a_i,b_{i+1}])\) are H-distinguished by Proposition 6.8. Let us write \(\varDelta = St (\rho ,l)\). Then by our assumption on i, we have \(\varDelta ([a_i,b_{i+1}]) = St (\rho ,l-1)\) and \(\varDelta ([a_{i+1},b_i]) = St (\rho ,l+1)\). By [20, Theorem 6.1], we can conclude that \(St (\rho ,l)\) is H-distinguished if and only if \(St (\rho ,l-1)\) (or \(St (\rho ,l+1)\)) is not H-distinguished. Actually, as \(\rho \) is self-dual, the L-function \(L(s,\phi (\rho )\otimes \phi (\rho ))\) has a simple pole at \(s = 0\), where \(\phi (\rho )\) is the Langlands parameter of \(\rho \). By the factorization
we know that exactly one of the symmetric or exterior square L-factors of \(\rho \) has a pole at \(s=0\). The above conclusion then follows from [20, Theorem 6.1] where distinction of \(St (\rho ,l)\) is related to the pole of symmetric or exterior square L-facotrs of \(\rho \) according to l is even or odd. Thus by our assumption that \(\varDelta \) is \(H_{d/2,d/2}\)-distingusihed, we get that \(\mathcal {K}_i\) is not \(H_{m,m}\)-distinguished. So we are done. \(\square \)
7.2 The general case
We start with an auxiliary result, which is needed in one step of the proof of Theorem 7.3.
Lemma 7.2
Let \(\pi = \pi _1 \times \cdots \times \pi _t\) be an irreducible unitary representation of \(G_{2m}\) with each \(\pi _i\) a Speh representation. Let h be a positive integer. Assume that, for all of those \(\pi _i\) such that \(\mathrm {supp}(\pi _i)\) is contained in the cuspidal line \({\mathbb {Z}}\nu ^{-1/2}\), we have \({\textbf {b}}(\pi _i) \leqslant \nu ^{h-1/2}\). If the representation \(\pi \times \nu ^{-h/2}\) is \((H_{m,m+h},\mu _{h/2})\)-distinguished, where \(\nu ^{-h/2}\) is viewed as a representation of \(G_h\), then \(\pi \) is \(H_{m,m}\)-distinguished.
Proof
A crucial fact, on which we rely, is that \(\pi \) is a commutative product of Speh representations. Our first step is to show that we can reduce the proposition to the case that for all i,
Indeed, write \(\varPi = \pi _1 \times \cdots \times \pi _r\) and \(\varPi ' = \pi _{r+1} \times \cdots \times \pi _t \times \nu ^{-h/2}\) where, \(\pi _j\)’s, \(r+1 \leqslant j \leqslant t\), are all the representations in the Tadić decomposition of \(\pi \) that satisfy (7.3). So \(\varPi \times \varPi '\) is \((H_{m,m+h},\mu _{h/2})\)-distinguished. By Proposition 6.8, \(\varPi \) is \((H_{m_1,m_1 + h_1},\mu _{h_1/2})\)-distinguished and \(\varPi '\) is \((H_{m-m_1,m-m_1+h-h_1},\mu _{(h+h_1)/2})\)-distinguished for certain integers \(m_1\) and \(h_1\). Note that the central character of \(\varPi \) has real part 0. We have \(h_1 = 0\). So, \(\varPi \) is \(H_{m_1,m_1}\)-distinguished and \(\varPi '\) is \((H_{m-m_1,m-m_1+h},\mu _{h/2})\)-distinguished. The reduction then follows from Lemma 5.7.
We thus assume that \(\pi = \pi _1 \times \cdots \times \pi _t\) with \({\textbf {b}}(\pi _i) = \nu ^{h-1/2}\) for all i. Moreover, we arrange the ordering of \(\pi _i\)’s such that \({\textbf {ht}}(\pi _1) \geqslant \cdots \geqslant {\textbf {ht}}(\pi _t)\). We prove the lemma by induction on t.
As the representation \(\pi \times \nu ^{-h/2}\) is \((H_{m,m+h},\mu _{h/2})\)-distinguished, by Proposition 6.6, there exist two representations \(\sigma '\) and \(\sigma ''\) of dimension one, \(\nu ^{-h/2} = \sigma ' \sqcup \sigma ''\), such that, among other things, \(\sigma '\) is \((H_{a,b},\mu _{h+(a-b)/2})\)-distinguished for two nonnegative integers a and b.
-
(1).
If \(\sigma '\) is not the trivial representation of \(G_0\), that is, a and b are not all zero, we have three cases. If \(a > 0\) and \(b > 0\), then by Lemma 3.2, \(\sigma '\) must be the trivial representation \({\textbf {1}}\) of \(G_{a+b}\). This is absurd as we have \({\textbf {b}}(\sigma ') = \nu ^{ - 1/2}\); If \(a > 0\) and \(b = 0\), then \(\sigma '\) is the character \(\nu ^{h+a/2}\) of \(G_a\). Thus \({\textbf {b}}(\sigma ') = \nu ^{h+a -1/2}\) which is absurd; If \(a =0\) and \(b>0\), we see easily that \(a = 0\) and \(b= h\), that is, \(\sigma '\) is the character \(\nu ^{-h/2}\) of \(G_h\). So, it follows from Case B of Proposition 6.6 that \(\pi \) is \(H_{m,m}\)-distinguished.
-
(2).
If \(\sigma '\) is the trivial representaion of \(G_0\), then we are in Case C of Proposition 6.6.Hence there exists i, \(1 \leqslant i \leqslant t\) and a division of \(\pi _i\) as two ladder representations \(\pi _i'\) and \(\pi _i''\), \(\pi _i = \pi _i' \sqcup \pi _i''\), such that \(\pi _i'\) is the character \(\nu ^{h/2}\) of \(G_h\) and the representation
$$\begin{aligned} \pi _1 \times \cdots \times \pi _{i-1} \times \pi _i'' \times \pi _{i+1} \times \cdots \times \pi _t \end{aligned}$$(7.4)is \((H_{m-h,m},\mu _{h/2})\)-distinguished. We have two subcases. If \(\pi _i\) is one dimensional, then \(\pi _i\) must be the trivial representaion \({\textbf {1}}\) of \(G_{2h}\) as \({\textbf {b}}(\pi _i) = \nu ^{h-1/2}\). Thus \(\pi _i''\) is the character \(\nu ^{-h/2}\) of \(G_h\). By Lemma 6.10, \(\nu ^{-h/2} \times \pi _j = \pi _j \times \nu ^{-h/2}\) for \(j = 1,\ldots ,t\). So we move \(\pi _i''\) to the end of the product (7.4) and get by induction hypothesis that
$$\begin{aligned} \pi _1 \times \cdots \times \pi _{i-1} \times \pi _{i+1} \times \cdots \times \pi _k \end{aligned}$$is \(H_{m-h,m-h}\)-distinguished. Hence \(\pi \) is \(H_{m,m}\)-distingusihed by Lemma 5.7. If otherwise \(\pi _i\) is not one dimensional, we can also move \(\pi _i''\) to the beginning of the product (7.4) by Lemma 6.10 and our ordering of \(\pi _i\)’s. By part (2) of Lemma 3.1,
$$\begin{aligned} \pi _1 \times \cdots \times \pi _{i-1} \times \pi _{i+1} \times \cdots \times \pi _t \times (\pi _i'')^{\vee } \end{aligned}$$(7.5)is \((H_{m,m-h},\mu _{-h/2})\)-distinguished. This is impossible by Proposition 6.8, and then we are done. Indeed, firstly, we can check easily that \(\pi _i''\), hence its contragradient \((\pi _i'')^{\vee }\), cannot be \((H_{p_1,q_1},\mu _{a_1})\)-distinguished for any \((p_1,q_1,a_1)\) by Lemma 6.18. Secondly, note that \({\textbf {e}}((\pi _i'')^{\vee }) = \nu ^{-h-1/2} \) is not dual to \({\textbf {b}}(\pi _i) = \nu ^{h-1/2}\) for all i.
\(\square \)
Theorem 7.3
Let \(\pi \) be an irreducible unitary representation of \(G_{2m}\) of Arthur type. Then \(\pi \) is \(H_{m,m}\)-distinguished if and only if \(\pi \) is of the form
where each \(\sigma _i\) is a Speh representation for \(i = 1, \ldots , r\), and each representation \(\sigma _j\) is \(H_{m_j,m_j}\)-distinguished for some positive integer \(m_j\), \(j = r+1 , \ldots ,s\).
Proof
By the work of Blanc and Delorme [2], we know that \(\sigma _j \times \sigma _j^{\vee }\) is \(H_{m_j,m_j}\)-distinguished with \(m_j\) the degree of \(\sigma _j\), \(j = 1,\ldots ,r\). One direction then follows from Lemma 5.7. Write \(\pi = \pi _1 \times \cdots \times \pi _t\) to be the Tadić decomposition of \(\pi \). We prove the other direction by induction on t. The case \(t = 1\) is obvious. In general, as \(\pi \) is a commutative product, we order these \(\pi _i\) in the following way: We first group these \(\pi _i\) by cuspidal supports. Namely, representations with cuspidal supports contained in the union of one cuspidal line and its contragredient are put in the same group. The ordering of the groups can be arbitrary. For representations within the same group, if their cuspidal supports are contained in one cuspidal line, we arrange the ordering such that when \(i < j\), we have either \({\textbf {b}}(\pi _i) < {\textbf {b}}(\pi _j)\), or \({\textbf {b}}(\pi _i) = {\textbf {b}}(\pi _j)\) and \({\textbf {ht}}(\pi _i) \leqslant {\textbf {ht}}(\pi _j)\); if their cuspidal supports are contained in two different cuspidal lines, we arrange the ordering such that when \(i < j\), we have \({\textbf {ht}}(\pi _i) \leqslant {\textbf {ht}}(\pi _j)\).
By our assumption, \(\pi \) is \(H_{m,m}\)-distinguished. We apply Propositon 6.6 and discuss case by case.
Case A. There exists a division of \(\pi _t\), \(\pi _t = \pi _t' \sqcup \pi _t''\), where \(\pi _t'\) is neither \(\pi _t\) nor the trivial representation of \(G_0\), such that, among other things, \(\pi _t'\) is \((H_{r,s},\mu _{(r-s)/2})\)-distinguished for two nonnegative integers r and s. We have two subcases.
-
(1)
The representation \(\pi _t'\) is not one dimensional. By Proposition 6.16, we know \(\pi _t'\) is an essentially Speh representation. So, by Corollary 6.14, we have \(r = s\). That is, \(\pi _t'\) is \(H_{r,r}\)-distinguished. In particular, \(\pi _t'\) is self-dual, and hence \(\pi _t\) is self-dual. This further shows that \(\pi _t'\) is a Speh representation. By Proposition 6.6, there exists i, \(1 \leqslant i \leqslant t-1\), and a division of \(\pi _i\), \(\pi _i = \pi _i' \sqcup \pi _i''\) such that \((\pi _t'')^{\vee } \cong \pi _i'\) and that
$$\begin{aligned} \pi _1 \times \cdots \times \pi _{i-1} \times \pi _i'' \times \pi _{i+1} \times \cdots \times \pi _{t-1} \end{aligned}$$(7.7)is \(H_{m',m'}\)-distinguished for some positive integer \(m'\). Thus we have \({\textbf {b}}(\pi _t) = {\textbf {b}}(\pi _i)\). By our assumption on the ordering of representations, we have \({\textbf {ht}}(\pi _i) \leqslant {\textbf {ht}}(\pi _t)\). As \(\pi _t'\) is a self-dual Speh representation that does not equal to \(\pi _t\), we have \({\textbf {ht}}(\pi _t'') = {\textbf {ht}}(\pi _t)\). As \({\textbf {ht}}(\pi _i') \leqslant {\textbf {ht}}(\pi _i)\), we have \({\textbf {ht}}(\pi _t) = {\textbf {ht}}(\pi _i)\) due to the fact that \((\pi _t'')^{\vee } \cong \pi _i'\). Thus we have \(\pi _i \cong \pi _t\) and \(\pi _i'' \cong \pi _t'\). Recall that \(\pi _t'\) is a \(H_{r,r}\)-distinguished Speh representation. So, by induction hypothesis, the representation (7.7) is of the form (7.6). After removing \(\pi _i''\) in the product, we still get a representation of the form (7.6). Therefore, by adding \(\pi _t \times \pi _i\), we get that \(\pi \) is of the form (7.6).
-
(2)
The representation \(\pi _t'\) is one dimensional. If \(r > 0\) and \(s > 0\), then \(\pi _t'\) is the trivial representation \({\textbf {1}}\) of \(G_{2r}\) by Lemma 3.2. Note that, in this case, \(\pi _t\) is not a one dimensional representaion. Then by the same arguments as in Case A (1), we are done in this case. If one of r, s is 0, then \(\pi _t'\) is the character \(\nu ^{h/2}\) of \(G_h\), \(h = \max \{r,s\}\). Thus we have \({\textbf {b}}(\pi _t) = {\textbf {b}}(\pi _t') = \nu ^{h-1/2}\). In particular, \(\pi _t\) is self-dual. By Proposition 6.6, there exists i, \(1 \leqslant i \leqslant t-1\), and a division of \(\pi _i\), \(\pi _i = \pi _i' \sqcup \pi _i''\) such that \((\pi _t'')^{\vee } \cong \pi _i'\) and that
$$\begin{aligned} \pi _1 \times \cdots \times \pi _{i-1} \times \pi _i'' \times \pi _{i+1} \times \cdots \times \pi _{t-1} \end{aligned}$$(7.8)is \((H_{m-n_t,m-n_t +h},\mu _{h/2})\)-distinguished with \(n_t\) the degree of \(\pi _t\). Thus we have \({\textbf {b}}(\pi _i) = {\textbf {b}}(\pi _t) = \nu ^{h - 1/2}\), and \(\pi _i\) is also self-dual. By our assumption on the ordering of representations, we have \({\textbf {ht}}(\pi _i) = {\textbf {ht}}(\pi _t)\), and hence \(\pi _i \cong \pi _t\). Thus, the representation \(\pi _i''\) is the character \(\nu ^{-h/2}\) of \(G_h\). By Lemma 6.10, the representation (7.8) is isomorphic to the representation
$$\begin{aligned} \pi _1 \times \cdots \times \pi _{i-1} \times \pi _{i+1} \times \cdots \times \pi _{t-1} \times \nu ^{-h/2}. \end{aligned}$$By Lemma 7.2, the representation \(\pi _1 \times \cdots \times \pi _{i-1} \times \pi _{i+1} \times \cdots \times \pi _{t-1}\) is \(H_{m-n_t,m-n_t}\)-distinguished, and hence is of the form (7.6) by induction hypothesis. Therefore, by adding \(\pi _i \times \pi _t\), we get that \(\pi \) is of the form (7.6).
Case B. In this case the representation \(\pi _t\) is \((H_{r,s},\mu _{(r-s)/2})\)-distinguished for two nonnegative integers r and s, and the representation
is \((H_{m - r,m-s},\mu _{(r-s)/2})\)-distinguished. As \(\pi _t\) is a Speh representation, by consideration of its central character, we have \(r = s\). Therefore, by induction hypothesis we are done.
Case C. There exists i, \(1 \leqslant i \leqslant t-1\), and a division of \(\pi _i\), \(\pi _i = \pi _i' \sqcup \pi _i''\), such that \((\pi _t)^{\vee } \cong \pi _i'\) and that the representation
is \(H_{m-n_t,m-n_t}\)-distinguished. By our assumption on the ordering of representations, we have \(\pi _i \cong (\pi _t)^{\vee }\). Thus \(\pi _i''\) is the trivial representation of \(G_0\). By induction hypothesis, the representation \(\pi _1 \times \cdots \times \pi _{i-1} \times \pi _{i+1} \times \cdots \times \pi _{t-1}\) is of the form (7.6). Therefore, by adding \(\pi _t \times (\pi _t)^{\vee }\), the representation \(\pi \) is of the form (7.6). \(\square \)
To classify distinguished representations in the entire unitary dual, it remains to consider distinction of complementary series representations. Recall that a complementary series representation is an irreducible unitary representation of the form \(\nu ^{\alpha }\text {Sp}(\delta ,k) \times \nu ^{-\alpha }\text {Sp}(\delta ,k)\) with \( 0< \alpha < 1/2\), and is denoted by \(\text {Sp}(\delta ,k)[\alpha ,-\alpha ]\). By the work of Blanc and Delorme [2], one sees that \(\text {Sp}(\delta ,k)[\alpha ,-\alpha ]\) is \(H_{m,m}\)-distinguished if and only if it is self-dual, where m is the degree of \(\text {Sp}(\delta ,k)\). To apply the geometric lemma, we first note the following lemma.
Lemma 7.4
Let \(\rho \) be a unitary supercuspidal representation of \(G_d\) and c a fixed integer. Let \(\pi \) be a ladder representation of \(G_n\) with cuspidal supports contained in the cuspidal line \({\mathbb {Z}}\nu ^{\alpha +c/2}\rho \) or \({\mathbb {Z}}\nu ^{-\alpha +c/2}\rho \) with \(0< \alpha < 1/2\), then \(\pi \) cannot be self-dual. If, moreover, \(\pi \) is left aligned, then \(\pi \) cannot be \((H_{p,q},\mu _{(p-q)/2})\)-distinguished for certain nonnegative integers p, q with \(p+q=n\).
Proof
As \(0< \alpha < 1/2\), the cuspidal line \({\mathbb {Z}}\nu ^{\alpha +c/2}\rho \) (or \({\mathbb {Z}}\nu ^{-\alpha +c/2}\rho \)) is not self-dual. Thus \(\pi \) cannot be self-dual by Lemma 6.1. For the second statement, if \(\pi \) is one dimensional, then by Lemma 6.19, the cuspidal supports of \(\pi \) is contained in \({\mathbb {Z}}\nu ^0\) or \({\mathbb {Z}}\nu ^{-1/2}\). This contradicts with our assumption; if \(\pi \) is not one dimensional, then by Proposition 6.16 and Corollary 6.14, one sees \(\pi \) is self-dual. This is absurd as shown by the first statement. \(\square \)
Theorem 7.5
An irreducible unitary representation \(\pi \) of \(G_{2n}\) is \(H_{n,n}\)-distinguished if and only if it is self-dual and its Arthur part \(\pi _{Ar }\) is of the form (7.6).
Proof
To simplify notation, we will say a representaion H-distinguished for \(H_{m,m}\)-distinguished when there is no need to address m. Write \(\pi = \pi _{Ar } \times \pi _c\). If \(\pi \) is self-dual, by uniqueness of Tadić decomposition, we have \(\pi _c\) is also self-dual. As \(\pi _c\) is a commutative product of complementary series representations, we have \(\pi _c\) is H-distinguished. The ‘if’ part then follows from Lemma 5.7. For the ‘only if’ part, write \(\pi \) as a product of essentially Speh representations
such that \(k_1 \leqslant k_2 \leqslant \cdots \leqslant k_r\), and that \(\pi _i\) is a Speh representation for \(i = 1, \ldots ,t\). Now we appeal to Proposition 6.6. By Lemma 7.4, only Case C can happen. Note that we have \(k_1 \leqslant \cdots \leqslant k_r\) and \(0< \alpha _i < 1/2\), \(i = 1 ,\ldots ,r\). By simple arguments we can show that each time after applying Proposition 6.6, we can delete two non-unitary essentially Speh representations in the product (7.9), and the new representation is H-distinguished. Thus by a repeated use of Proposition 6.6, we get \(\pi _{Ar } = \pi _1 \times \cdots \times \pi _t\) is H-distinguished. The ‘only if’ part then follows from Theorem 7.3. \(\square \)
Code Availability
Not applicable.
Materials Availability
Not applicable.
References
Bernstein, I.N., Zelevinsky, A.V.: Induced representations of reductive \({\mathfrak{p} }\)-adic groups. I. Annales Scientifiques de l’École Normale Supérieure 10, 441–472 (1977)
Blanc, P., Delorme, P.: Vecteurs distributions \(H\)-invariants de représentations induites, pour un espace symétrique réductif \(p\)-adique \(G/H\). Annales de l’Institut Fourier. 58, 213–261 (2008)
Chen, F., Sun, B.: Uniqueness of twisted linear periods and twisted Shalika periods. Sci. China Math. 63, 1–22 (2020)
Cogdell, J. W., Piatetski-Shapiro, I. I.: Derivatives and \(L\)-functions for \({\rm GL}(n)\). In: Cogdell, J., Kim, J.L., Zhu, C.B. (eds.) Representation theory, number theory, and invariant theory, pp. 115–173 Progress in Mathematics, vol. 323. Birkhäuser, Basel (2017)
Fang, Y., Sun, B., Xue, H.: Godement–Jacquet \(L\)-functions and full theta lifts. Mathematische Zeitschrift. 289, 593–604 (2018)
Feigon, B., Lapid, E., Offen, O.: On representations distinguished by unitary groups. Publications Mathématiques de l’IHÉS 115(1), 185–323 (2012)
Friedberg, S., Jacquet, H.: Linear periods. J. Reine Angew. Math. 443, 91–140 (1993)
Gan, W. T.: Periods and theta correspondence. In: Aizenbud, A., Gourevitch, D., Kazhdan, D., Lapid, E.M. (eds.) Representations of Reductive Groups, pp. 113–132 Proceedings of Symposia in Pure Mathematics, volume 101. American Mathematical Society (2019). https://doi.org/10.1090/pspum/101
Gan, W.T., Gross, B.H., Prasad, D.: Branching laws for classical groups: the non-tempered case. Compos. Math. 156(11), 2298–2367 (2020). https://doi.org/10.1112/s0010437x20007496
Gurevich, M.: On a local conjecture of Jacquet, ladder representations and standard modules. Mathematische Zeitschrift 281(3–4), 1111–1127 (2015)
Jacquet, H., Rallis, S.: Uniqueness of linear periods. Compos. Math. 102(1), 65–123 (1996)
Jacquet, H., Piatetskii-Shapiro, I.I., Shalika, J.: Rankin-selberg convolutions. Am. J. Math. 105(2), 367–464 (1983)
Jiang, D., Soudry, D.: Generic representations and local langlands reciprocity law for \(p\)-adic \({\rm SO}_{2n+1}\). In: Hida, H., Ramakrishnan, D., Shahidi, F. (eds.) Contributions to automorphic forms, geometry, and number theory, pp. 457–519. The Johns Hopkins University Press, Baltimore and London (2004)
Jiang, D., Nien, C., Qin, Y.: Local Shalika models and functoriality. Manusc. Math. 127(2), 187–217 (2008)
Kable, A.C.: Asai \(L\)-functions and Jacquet’s conjecture. Am. J. Math. 126(4), 789–820 (2004)
Kret, A., Lapid, E.: Jacquet modules of ladder representations. Comptes Rendus Math. 350(21–22), 937–940 (2012)
Lapid, E., Mínguez, A.: On a determinantal formula of Tadić. Am. J. Math. 136(1), 111–142 (2014)
Lapid, E., Mínguez, A.: On parabolic induction on inner forms of the general linear group over a non-archimedean local field. Selecta Math. 22(4), 2347–2400 (2016)
Matringe, N.: Distinguished generic representations of \({\rm GL} (n)\) over \(p\)-adic fields. Int. Math. Res. Not. 2011(1), 74–95 (2010)
Matringe, N.: Linear and Shalika local periods for the mirabolic group, and some consequences. J. Number Theory 138, 1–19 (2014)
Matringe, N.: Unitary representations of \({\rm GL}(n, K)\) distinguished by a Galois involution for a \(p\)-adic field \(K\). Pacific J. Math. 271(2), 445–460 (2014)
Matringe, N.: On the local Bump–Friedberg \(L\)-function. J. Reine Angew. Math. 2015(709), 119–170 (2015)
Mínguez, A.: Correspondance de Howe explicite: paires duales de type II. Annales Scientifiques de l’École Normale Supérieure. 41, 717–741 (2008)
Mitra, A., Offen, O., Sayag, E.: Klyachko models for ladder representations. Docum. Math. 22, 611–657 (2017)
Moeglin, C., Waldspurger, J.L.: Sur l’involution de Zelevinski. J. Reine Angew. Math. 1986(372), 136–177 (1986)
Offen, O.: On parabolic induction associated with a \(p\)-adic symmetric space. J. Number Theory 170, 211–227 (2017)
Offen, O., Sayag, E.: On unitary representations of \({\rm GL} _{2n}\) distinguished by the symplectic group. J. Number Theory 125(2), 344–355 (2007)
Offen, O., Sayag, E.: Uniqueness and disjointness of Klyachko models. J. Funct. Anal. 254(11), 2846–2865 (2008)
Sécherre, V.: Représentations cuspidales de \({\rm GL}(r,D)\) distinguées par une involution intérieure. https://arxiv.org/pdf/2005.05615.pdf
Tadić, M.: Classification of unitary representations in irreducible representations of general linear group (non-archimedean case). Annales Scientifiques de l’Ecole Normale Supérieure. 19, 335–382 (1986)
Tadić, M.: Induced representations of \({\rm GL}(n, A)\) for p-adic division algebras \(A\). J. Reine Angew. Math. 405, 48–77 (1990)
Tadić, M.: On characters of irreducible unitary representations of general linear groups. Abhandlungen aus dem Mathematischen Seminar der Universität Hamburg 65(1), 341–363 (1995)
Zelevinsky, A.V.: Induced representations of reductive p-adic groups. II. on irreducible representations of \({\rm GL}(n)\). Annales Scientifiques de l’École Normale Supérieure. 13(2), 165–210 (1980)
Acknowledgements
The author thanks Wee Teck Gan for his kindness of sharing his paper [8] with us. He would also like to thank Dipendra Prasad for helpful comments and suggestions to improve the paper. The author thanks the referee for many remarks and suggestions and for pointing out an error in an earlier draft of the paper. The author was supported by the National Natural Science Foundation of China (no.12001191).
Funding
This work was supported in part by the National Natural Science Foundation of China (no. 12001191).
Author information
Authors and Affiliations
Contributions
Not applicable.
Corresponding author
Ethics declarations
Conflicts of interest
The author declares that there is no conflict of interests regarding the publication of this article.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Yang, C. Linear periods for unitary representations. Math. Z. 302, 2253–2284 (2022). https://doi.org/10.1007/s00209-022-03136-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00209-022-03136-y