On a local conjecture of Jacquet, ladder representations and standard modules

Abstract

Let E / F be a quadratic extension of p-adic fields. We prove that every smooth irreducible ladder representation of the group $GL_n(E)$ which is contragredient to its own Galois conjugate, possesses the expected distinction properties relative to the subgroup $GL_n(F)$. This affirms a conjecture attributed to Jacquet for a large class of representations. Along the way, we prove a reformulation of the conjecture which concerns standard modules in place of irreducible representations.

1 Introduction

Let E / F be a quadratic extension of p-adic fields. Denote the group $G_n= GL_n(E)$. Let $g\mapsto g^\tau$ be the Galois involution on $G_n$ relative to the extension E / F. Denote $H_n = G_n^\tau = GL_n(F)$. We study the $H_n$-invariant functionals on admissible representations of $G_n$. In particular, we are interested to know whether a non-zero such functional exists for a given representation. In this case we will say that the representation is distinguished.

Let $\eta$ denote the quadratic character of $F^\times$ related to the extension E / F. We will call an admissible $G_n$-representation $\eta$-distinguished, if a non-zero $H_n,\eta (\det )$-equivariant functional exists on it.

It is long known ([5]) that a distinguished irreducible smooth representation $\pi$ of $G_n$ must satisfy $\pi ^\vee \cong \pi ^\tau$, where the left-hand side of the equation is the contragredient representation, while the right-hand side is the twist induced on representations by the involution $\tau$. The prediction, that certain variations of the converse implication should hold, are often referred to as Jacquet’s conjecture. Let us formulate it as a general principle.

An irreducible smooth representation of $G_n$ which satisfies ¹ $\pi ^\vee \cong \pi ^\tau$, should be either distinguished or $\eta$-distinguished.

In general this formulation is evidently false. As a counter-example, one can choose an $\eta$-distinguished irreducible supercuspidal representation $\rho$ of $G_2$, and look on the representation $1\times \rho$ of $G_3$, where 1 is the trivial representation of $E^\times$ and the multiplication is in the sense of parabolic induction.

Yet, for significant large families of irreducible representations the principle above was indeed shown to hold. All discrete series are such, as proved by Kable in [9]. Matringe’s results in [14] implied the same for certain other unitarizable representations, including the so-called Speh representations.

In this work, we extend the validity of Jacquet’s conjecture to the class of ladder representations, which was introduced by Lapid and Mínguez in [10]. This wide family of irreducible representations of $G_n$ includes discrete series and Speh representations as special cases.

Theorem A

(Theorems 4.3 and 4.6) A ladder representation of $G_n$ which satisfies $\pi ^\vee \cong \pi ^\tau$ is either distinguished or $\eta$-distinguished. Moreover, a proper ladder representation of $G_n$ cannot be both distinguished and $\eta$-distinguished.

The second statement of the theorem is again an expected property which was previously established for discrete series and Speh representations.

The precise statements of Theorems 4.3 and 4.6 go further by fully characterizing distinction and $\eta$-distinction of ladder representations from the combinatorial properties of the defining data of the representation.

For the proof of Theorem A, we turn to the class of reducible admissible representations of $G_n$ called standard modules. These are representations constructed by parabolically inducing a tempered representation $\kappa$ of a Levi subgroup $M< G_n$ twisted by an unramified character $\alpha$ of M chosen from a certain cone. A standard module is uniquely defined by the triple $(M,\kappa ,\alpha )$. Recall that the Langlands classification describes each irreducible smooth representation $\pi$ of $G_n$ as a unique irreducible quotient of a standard module $\varSigma (\pi )$.

Thus, studying invariant functionals on an irreducible representation can be done by constructing such functionals on the corresponding standard module, and then determining whether they factor through the irreducible quotient. Such methods were explored in [4] for studying distinction relative to unitary subgroups (in place of our $H_n$).

Note, that an irreducible smooth representation $\pi$ is generic, if and only if, the standard module $\varSigma (\pi )$ is irreducible (equivalently, $\pi \cong \varSigma (\pi )$). We propose and prove the following reformulation of Jacquet’s conjecture for all smooth irreducible representations, which coincides with the original formulation on generic representations.

Theorem B

(Corollary 3.5 and Lemma 3.10) Suppose that $\pi$ is a smooth irreducible representation of $G_n$, whose standard module $\varSigma (\pi )$ is distinguished. Then, $\pi ^\vee \cong \pi ^\tau$ holds.

Conversely, let $\pi$ be a smooth irreducible distinguished representation of $G_n$ of pure type, i.e. the supercuspidal support of $\pi$ is contained in the set $\{\rho \otimes |\det |_E^n\,:\,n\in {\mathbb {Z}}\}$ for some supercuspidal representation $\rho$. If $\pi ^\vee \cong \pi ^\tau$ holds, then the standard module $\varSigma (\pi )$ is either distinguished or $\eta$-distinguished.

This result is then further extended in Theorem 3.11 to a description of the same situation without the assumption on pure type. It again can be seen to coincide with a known statement ([13, Theorem 5.2]) when dealing with generic representations.

The feature which makes ladder representations approachable to our discussion is Theorem 1 of [10]. It states that when an irreducible $\pi$ is a ladder representation, the kernel of the quotient map $\varSigma (\pi )\rightarrow \pi$ can itself be described in terms of standard modules. Thus, in order to claim that an invariant functional on $\varSigma (\pi )$ factors through the quotient, it is enough to know it must vanish on the standard modules which generate the above kernel. This is the method by which we manage to deduce Theorem A from Theorem B.

The paper is organized as follows. Section 2 brings into our setting some known tools for studying distinction problems. The foremost tool is the geometric lemma of Bernstein–Zelevinsky which is long-known to serve as the Mackey theory of admissible representations. It allows us to study invariant functionals on induced representations through distinction properties of the inducing data. On top of that, our main tool for producing functionals on induced representations is the Blanc–Delorme theory developed in [3]. Their method, adapted to our needs in Proposition 2.3, can construct a desired functional by taking a continuation of an analytic family of integrals.

Section 3 deals with the proof of both implications of Theorem B. We present each standard module as a multiplication, in the sense of parabolic induction, of essentially square-integrable representations. We then can state arguments of a combinatorial nature, such as Lemma 3.3, about the structure of the space of invariant functionals on a standard module. The first implication of Theorem B, which is also the crucial step for the proof of Theorem A, follows from this key lemma. We also deduce a multiplicity one theorem (Proposition 3.6) for $H_n$-invariant functionals on a large class of (possibly reducible) standard modules.

The second implication is shown by obtaining the existence of invariant functionals from the Blanc–Delorme theory.

Finally, Sect. 4 deals with ladder representations. We show the deduction of Theorem A from Theorem B as described above. Theorem 4.2 further resolves between distinction and $\eta$-distinction of a proper ladder representation, in terms of the inducing data of its standard module.

For the last part of Theorem A (Theorem 4.6), the additional tool of Gelfand–Kazhdan derivatives needs to be introduced. The idea, which traces back to [7, 9], is that distinction of a given smooth irreducible representation implies the distinction of at least one of its derivatives. Using this in combination with the derivative data of ladder representations, which was obtain in [10], allows us to contradict the possibility of distinction of certain representations.

2 Distinction of induced representations

We will write representations of a locally compact totally disconnected group G as $(\pi ,V)$, or simply as $\pi$, where V is a complex vector space and $\pi :G\rightarrow GL(V)$ is a homomorphism. The representation $(\pi ,V)$ is called (smooth) admissible if the stabilizer of each vector in V is an open subgroup of G and for every compact open subgroup $K<G$, the space of vectors in V invariant under K is of finite dimension.

Given an admissible representation $(\pi ,V)$ of $G_n$ and a character $\alpha$ of $H_n$, we say that a functional $\ell$ on V is $(H_n,\alpha )$-equivariant if $\ell (\pi (h)v) = \alpha (h)\ell (v)$ for all $h\in H_n$, $v\in V$. If such $\pi$ (not necessarily irreducible) has a non-zero $(H_n,\alpha )$-equivariant functional on it, we will say that $\pi$ is $\alpha$-distinguished. Note, that n will sometimes be implicit in our notation. We will say that $\pi$ is distinguished, if it is 1-distinguished.

For an admissible representation $\pi$ of $G_n$, we denote by $\pi ^\vee$ its contragredient representation. Also, denote by $\pi ^\tau$ its Galois twist, that is, $\pi ^\tau (g)=\pi (g^\tau )$, for all $g\in G_n$.

Given a character $\beta$ of $G_n$, we will often write $\beta \pi$ for the tensor product representation $\beta \otimes \pi$ of $G_n$.

2.1 Consequences of the geometric lemma

Any standard Levi subgroup of $G_n$ is of the form $M=M_{(m_1,\cdots , m_t)}=G_{m_1}\times \cdots \times G_{m_t} < G_n$ $(\sum _{i=1}^t m_i= n)$, with the obvious diagonal embedding. If $\sigma _1,\ldots ,\sigma _t$ are admissible representations of $G_{m_1},\ldots , G_{m_t}$, respectively, we denote by $\sigma _1\times \cdots \times \sigma _t$ the representation of $G_n$ constructed by normalized parabolic induction. In other words, the M-representation $\sigma :=\sigma _1\otimes \cdots \otimes \sigma _t$ is naturally lifted to P, where $M\subseteq P\subseteq G_n$ is the standard parabolic subgroup corresponding to M. Then,

$$\begin{aligned} \sigma _1\times \cdots \times \sigma _t = ind_P^{G_n} (\delta _P^{1/2}\sigma ), \end{aligned}$$

where $\delta _P$ is the modular character of P and ind denotes the (non-normalized) induction functor of smooth representations from the subgroup $P<G_n$. For an admissible representation $\pi$ of $G_n$, we denote by $r_{M,G_n}(\pi )$ the representation of M which is the normalized Jacquet module of $\pi$. See, for example, [2, Section 2.3] for the definitions of induction of smooth representations and the Jacquet module.

First, we would like to deal with distinction properties of representations of $G_n$ that are parabolically induced from a standard Levi subgroup. Let us fix one such subgroup $M<G_n$ and its corresponding parabolic subgroup P for the rest of this section. We will need a convenient description of the double cosets space $P{\setminus } G_n/H_n$.

Let W denote the Weyl group of $G_n$ realized as $N_{G_n}(T)/T$, where T is the diagonal maximal torus of $G_n$. Set $W^M =\{w\in W\,:\, wMw^{-1}=M\}$, and its normal subgroup $W_M = N_M(T)/T$, which is the Weyl group of M. There is a natural mapping $p_M: W^M\rightarrow S_t$, whose kernel is $W_M$. Here $S_t$ is the permutation group on $\{1,\ldots ,t\}$. It will sometimes be convenient to have the notation $\mathfrak {J}_M = \{1,\ldots ,t\}$ and refer to the image of $p_M$ as permutations on $\mathfrak {J}_M$.

Let $W[M]\subset W$ be the set of representatives of minimal length for the double cosets space $W_M{\setminus } W/W_M$. Let $W_2[M]\subset W[M]$ denote the subset of involutions inside it. By [8, Proposition 20] there is a bijection between $W_2[M]$ and $P{\setminus } G_n/H_n$. Explicitly, each $P-H_n$ double coset has a representative $\eta$ for which $\xi =\eta ^\tau \eta ^{-1}$ belongs to the normalizer $N_{G_n}(T)$. The representative can be chosen so that the projection of $\xi$ to W will fall inside $W_2[M]$ (see also [8, Lemma 19]). The resulting involutive permutation is uniquely defined by the double coset. See also [13, Section 3] for an equivalent description of the double cosets in different terms.

It follows easily from Hilbert’s Theorem 90 that for some $d\in T$, $d^\tau \xi d^{-1}= (d\eta )^\tau (d\eta )^{-1}$ is in fact a permutation matrix (consists only of 1 and 0 entries). Thus, for each $w\in W_2[M]$, we fix a representative $\eta _w$ of the associated $P-H_n$ double coset, for which $\eta _w^\tau \eta _w^{-1}$ is a permutation matrix (given by w).

An element $w\in W_2[M]$ and the double coset associated with it will be called M-admissible if $w\in W^M$. Applying $p_M$, we see that the M-admissible double cosets are in natural correspondence with involutive permutations $\epsilon$ on $\mathfrak {J}_{M}$, for which $m_{\epsilon (i)}=m_i$ holds for all $i\in \mathfrak {J}_M$. The rest of the double cosets can still be described in similar terms, but by descending to a smaller Levi subgroup. Namely, for $w\in W[M]$, we need to observe the subgroup $M(w):= M\cap wMw^{-1}$ which must be a standard Levi subgroup for $G_n$. Note, that each $w\in W_2[M]\subseteq W_2[M(w)]$ is M(w)-admissible.

Looking at any inclusion $L\subseteq M=M_{(m_1,\ldots ,m_t)}$ of standard Levi subgroups, we can describe L as $M_\gamma$ with $\gamma = (l_{11},\ldots ,l_{1s_1},\ldots ,l_{t1},\ldots ,l_{ts_t})$ and $\sum _{j=1}^{s_i} l_{ij} = m_i$, for all i. With this in mind, we give another natural enumeration of the blocks of L as pairs

$$\begin{aligned} \mathfrak {J}_{L,M} = \{(i,j)\,:\, i=1,\ldots , t,\quad j=1,\ldots ,s_i\} \end{aligned}$$

with the lexicographical ordering. Naturally, the ordered set $\mathfrak {J}_{L,M}$ is identified with $\mathfrak {J}_L$ by sending (i, j) to $\sum _{k=1}^{i-1}s_k+j$.

Given $w\in W_2[M]$, we have a description of $\epsilon _w:=p_{M(w)}(w)$ as an involution on $\mathfrak {J}_{M(w)}\cong \mathfrak {J}_{M(w), M}$. It must satisfy the rule that for each $1\le i\le t$ and $1\le j<k\le s_i$, if $\epsilon _w(i,j)=(i'_1,j'),\,\epsilon _w(i,k)=(i'_2,k')$, then $i'_1<i'_2$. In fact, going over all standard Levi subgroups $L\subseteq M$ and all involutions $\epsilon$ of $\mathfrak {J}_{L,M}$, which satisfy the above condition and for which $l_{\epsilon (i,j)}=l_{i,j}$ holds for all $(i,j)\in \mathfrak {J}_{L,M}$, would give a full description of $W_2[M]$.

For each $w\in W_2[M]$, we define the subgroup $M^w:= M\cap \eta _w H_n\eta _w^{-1}$, and similarly for $P^w$. Note, that,

$$\begin{aligned} M^w = M(w)^w = \left\{ (g_i)_{i\in \mathfrak {J}_{M(w)}}\in M(w)\,:\, g_{\epsilon _w(i)} = g^\tau _i\right\} . \end{aligned}$$

Suppose that $\sigma$ is an admissible representation of M. Let $(\pi ,V)$ be the representation parabolically induced from $\sigma$ to $G_n$. For $w\in W_2[M]$, let us define the $H_n$-representation ${\mathcal {V}}_w(\pi ):= ind^{H_n}_{\eta _w^{-1} P\eta _w\cap H_n} \left( \delta _P^{1/2}\sigma |_{P^w}\right) ^{\eta _w}$, where $(\cdot )^{\eta _w}$ denotes the conjugation functor that transfers a $P\cap \eta _w H_n\eta _w^{-1}$-representation into a $\eta _w^{-1} P\eta _w \cap H_n$-representation. Mackey theory (proved in the Geometric Lemma of [2]) gives a filtration of $\pi$ by $H_n$-sub-representations $\{0\}=V_0\subset V_1\subset \cdots \subset V_k=V$, in such a way that each subquotient $V_i/V_{i-1},\,i=1,\ldots ,k$ is isomorphic to ${\mathcal {V}}_{w_i}(\pi )$ for some enumeration $(w_i)$ of $W_2[M]$.

The above geometric decomposition allows us to study the invariant functionals on $\pi$ in terms of distinction properties of certain Jacquet modules of $\sigma$.

Lemma 2.1

Let $w\in W_2[M]$ be an involution, and suppose that the Jacquet module $r_{M(w),M}(\sigma )$ is a pure tensor representation, that is,

$$\begin{aligned} r_{M(w),M}(\sigma ) = \otimes _{i\in \mathfrak {J}_{M(w)}} \sigma _i. \end{aligned}$$

Let $\mathfrak {F}\subset \mathfrak {J}_{M(w)}$ be a choice of representatives for the orbits of $\epsilon _w$ on $\mathfrak {J}_{M(w)}$ (i.e. one index out of $\{i,\epsilon _w(i)\}$ belongs to $\mathfrak {F}$, for all $i\in \mathfrak {J}_{M(w)}$). Then,

$$\begin{aligned}&Hom_{H_n}({\mathcal {V}}_w(\pi ),{\mathbb {C}})\\&\quad \cong \left( \otimes _{i\in \mathfrak {F}\,:\,\epsilon _w(i)=i} Hom_{H_{m_i}}(\sigma _i,{\mathbb {C}})\right) \otimes \left( \otimes _{i\in \mathfrak {F}\,:\,\epsilon _w(i)\ne i} Hom_{G_{m_i}}(\sigma _i^\tau ,\sigma ^\vee _{\epsilon _w(i)})\right) . \end{aligned}$$

Proof

By [4, Lemma 6.4]² we deduce that

$$\begin{aligned} Hom_{H_n}({\mathcal {V}}_w(\pi ),{\mathbb {C}})\cong Hom_{M^w}(r_{M(w),M}(\sigma ),{\mathbb {C}}). \end{aligned}$$

Now, from the description of $M^w$ it follows that,

$$\begin{aligned}&Hom_{M^w}(r_{M(w),M}(\sigma ),{\mathbb {C}})\cong \left( \otimes _{i\in \mathfrak {F}\;:\;\epsilon _w(i)=i} Hom_{H_{m_i}}(\sigma _i,{\mathbb {C}})\right) \\&\qquad \otimes \left( \otimes _{i\in \mathfrak {F}\;:\;\epsilon _w(i)\ne i} Hom_{\left\{ (g^\tau ,g)\in G_{m_i}\times G_{m_{\epsilon _w(i)}}\right\} }(\sigma _i\otimes \sigma _{\epsilon _w(i)},{\mathbb {C}}) \right) . \end{aligned}$$

The second term of the tensor product is evidently built from spaces of invariant pairings between $\sigma _i^\tau$ and $\sigma _{\epsilon _w(i)}$. $\square$

Finally we will need the following simple fact about induction and distinguished representations.

Lemma 2.2

([6, Proposition 26] or [15, Lemma 6.4]) Suppose that $M=M_{(m_1,\ldots ,m_t)}< G_n$ is a standard Levi subgroup. Let $\sigma _i$ be a distinguished admissible representation of $G_{m_i}$, for all $1\le i\le t$. Then, $\pi = \sigma _1 \times \cdots \times \sigma _t$ is distinguished.

2.2 Blanc–Delorme theory

For the standard Levi subgroup $M=M_{(m_1,\ldots ,m_t)}< G_n$, let us define the complex algebraic variety X of unramified characters of M. That is, X consists of characters of the form $\nu ^\lambda :=\nu ^{\lambda _1}\otimes \cdots \otimes \nu ^{\lambda _t}$, where $\nu$ is the character (of any $G_{m_i}$) given by the formula $\nu (g) = |\det (g)|_E$, and $\lambda =(\lambda _1,\ldots ,\lambda _t)\in {\mathbb {C}}^t$. The natural covering map ${\mathbb {C}}^t\rightarrow X$ equips X with a structure of an affine algebraic variety isomorphic to $\left( {\mathbb {C}}^\times \right) ^t$. As complex functions, the regular functions on X composed with the covering map are polynomials in the variables $q^{\pm \lambda _1},\ldots , q^{\pm \lambda _t}$, where q is the size of the residual field of E.

Let $\sigma$ be an admissible representation of M, and $(\pi ,V)$ the representation of $G_n$ parabolically induced from $\sigma$ as before. It is possible to see $\pi$ as one element of a family of representations parametrized by unramified characters in X. Namely, for all $\chi \in X$, the representation $\pi _\chi := ind^{G_n}_P (\chi \sigma )$ can be realized on the same space V (see [4, 1.3] for the precise construction), making $\pi _\chi (g)$ an analytic family of operators, for each $g\in G_n$. We will omit the description of such a realization, since it will not be of relevance here.

Now, suppose that $m_i=m_{t+1-i}$, for all $1\le i\le t$. In this setting, we would like to exploit the theory developed in [3] to produce a non-zero $H_n$-invariant functional on $\pi$, under suitable conditions.

Let $\beta \in W_2[M]$ be the M-admissible element that is given by $\epsilon _\beta (i) = t+1-i$, for all $i\in \mathfrak {J}_M=\{1,\ldots ,t\}$. Note, that $\eta _\beta H_n\eta _\beta ^{-1}$ is the fixed point subgroup of the involution $\theta =\theta _\beta$ on $G_n$ given by $\theta (g) = \xi _\beta ^{-1}g^\tau \xi _\beta$, where $\xi _\beta \in G_n$ is the permutation matrix corresponding to $\beta$. The subgroup $\theta (P)$ is then the opposite parabolic to P relative to the $\theta$-stable maximal torus T (the fixed diagonal torus).

Noting the action of $\theta$ on the characters of $G_n$, we define $X_\theta \subset X$ as the connected component of the identity character inside the affine variety of $\theta$-anti-invariants of X. Explicitly,

$$\begin{aligned} X_\theta = \left\{ \nu ^{\lambda _1}\otimes \cdots \otimes \nu ^{\lambda _t}\in X\,:\, \forall i,\,\lambda _i = -\lambda _{t+1-i}\right\} \cong \left( {\mathbb {C}}^\times \right) ^{\lfloor t/2 \rfloor }. \end{aligned}$$

Suppose now that $\ell$ is a non-zero $H_n$-invariant functional on ${\mathcal {V}}_{\beta }(\pi )$. Recalling [4, Lemma 6.4] as before, this gives the existence of a non-zero functional $\widetilde{\ell }$ on the space of $\sigma$, which is invariant under the action of $M^\beta = M \cap G_n^\theta$. Now, when such $\widetilde{\ell }$ is put in the setting of [3, Theorem 2.8], we conclude the following statement. There is a regular function r on $X_\theta$, such that for each $\chi \in X_\theta$ with $r(\chi )\ne 0$ there is a functional $0\ne J(\ell ,\chi )\in V^*$ which is invariant under the action of $\pi _\chi |_{G_n^\theta }$. Moreover, for every $\phi \in V$, the function $\chi \mapsto r(\chi )J(\ell ,\chi )(\phi )$ can be prolonged to a regular function on $X_\theta$.

This family of functionals can be used to construct a single non-zero functional on the original representation $\pi$. We summarize it in the following statement.

Proposition 2.3

Let $M=M_{(m_1,\ldots ,m_t)}< G_n$ be a standard Levi subgroup, with $m_i=m_{t+1-i}$, for all $1\le i\le t$. Let $\pi$ be a representation of $G_n$ parabolically induced from M. Let $\beta \in W_2[M]$ be the M-admissible element that is given by $\epsilon _\beta (i) = t+1-i$, for all $i\in \mathfrak {J}_M=\{1,\ldots ,t\}$.

If ${\mathcal {V}}_\beta (\pi )$ has a non-zero $H_n$-invariant functional, then $\pi$ is distinguished.

Proof

The analytic continuation of $L(\chi ):=r(\chi )J(\ell ,\chi )$ to $\chi =1$ (the trivial character) defines a functional on V invariant under $\pi (\eta _\beta H_n\eta _\beta ^{-1})$. Fix an (affine) curve $Y\subset X_\theta$ such that $1\in Y$, and $r|_Y$ is a non-zero function. It follows that $S =\{L(\chi )(\phi )|_Y\,:\;\phi \in V\}$ is a collection of non-zero regular functions on Y. Let $p(\chi )$ be a regular function on Y whose order of vanishing at 1 equals to the minimum of these orders for S. Then, $\overline{L}= p(\chi )^{-1}L(\chi )|_{\chi =1}$ gives a non-zero functional. Hence, $0\ne \overline{L}\circ \pi (\eta _\beta )\in V^*$ is $H_n$-invariant. $\square$

3 Distinction of standard modules

3.1 Notations

Denote by $\varPi _{G_n}$ the set of isomorphism classes of irreducible admissible representations of $G_n$. Given integers $a\le b$, such that $b-a+1$ divides n, and a supercuspidal $\rho \in \varPi _{G_{n/(b-a+1)}}$, there is a unique irreducible quotient representation of $\nu ^a\rho \times \nu ^{a+1}\rho \times \cdots \times \nu ^b\rho$. We denote this representation by $\varDelta (\rho ,a,b)$, and call it a segment. These are exactly the essentially square-integrable representations in $\varPi _{G_n}$. We say that a segment $\varDelta _1\in \varPi _{G_{n_1}}$ precedes another segment $\varDelta _2\in \varPi _{G_{n_2}}$ if $\varDelta _1\cong \varDelta (\rho ,a_1, b_1)$ and $\varDelta _2\cong \varDelta (\rho ,a_2,b_2)$ for some supercuspidal $\rho$ and integers with $a_1<a_2\le b_1+1$ and $b_1<b_2$. A representation that is induced from two segments $\varDelta _1\times \varDelta _2$ is irreducible, if and only if, none of the segments precedes the other ([18, 9.7]). Also, we have $\varDelta _1\times \varDelta _2\cong \varDelta _2\times \varDelta _1$ when it is irreducible.

Denote by $c_\pi$ the central character of $\pi \in \varPi _{G_n}$. For all $g\in E^\times$, $|c_\pi (g)| = |g|_E^r$ for some $r\in {\mathbb {R}}$. We will call r the real exponent of $\pi$ and denote it by $\mathfrak {R}(\pi )=r$. Clearly, $\mathfrak {R}(\nu \pi ) = n +\mathfrak {R}(\pi )$. Also, if $\pi$ is a subquotient of $\pi _1\times \pi _2$, then $\mathfrak {R}(\pi )=\mathfrak {R}(\pi _1)+\mathfrak {R}(\pi _2)$. Together with the fact that $\nu ^k\varDelta (\rho ,a,b)\cong \varDelta (\rho ,a+k,b+k)$ for any integer k, it is easy to see that if a segment $\varDelta _1$ precedes $\varDelta _2$, then $\mathfrak {R}(\varDelta _1)<\mathfrak {R}(\varDelta _2)$.

The normalized Jacquet module of segments has a clear description ([18, 9.5]). Suppose that $\varDelta =\varDelta (\rho ,a,b)\in \varPi _{G_n}$ with $\rho \in \varPi _{G_d}$. In case d divides all $m_i$’s, we have $r_{M_{(m_1,\ldots ,m_t)},G_n}(\varDelta ) = \varDelta (\rho ,a_1,b_1)\otimes \cdots \otimes \varDelta (\rho ,a_t,b_t)$, where $b_1=b$, $d(b_i-a_i+1)=m_i$, and $b_{i+1} =a_i-1$, for all i. Otherwise, the Jacquet module is the zero representation.

A representation of $G_n$ is called a standard module if it is parabolically induced from a representation $\kappa \nu ^\lambda$ of a standard Levi subgroup $M=M_{(m_1,\ldots ,m_t)}< G_n$, where $\kappa$ is an irreducible tempered representation, and $\lambda =(\lambda _1,\ldots ,\lambda _t)\in {\mathbb {R}}^t$, with $\lambda _1>\lambda _2>\cdots >\lambda _t$. Denote by $\mathfrak {S}_{G_n}$ the set of isomorphism classes of standard modules of $G_n$. Each element of it can be described by the triple $(M,\kappa ,\lambda )$ known as the Langlands data. The Langlands classification for $G_n$ (proved in [16]) can be formulated as a bijection $\varSigma :\varPi _{G_n}\rightarrow \mathfrak {S}_{G_n}$ (or between $\varPi _{G_n}$ and triples of Langlands data), which satisfies the property that each $\pi \in \varPi _{G_n}$ is the unique irreducible quotient of $\varSigma (\pi )$. It is known that $\varSigma (\pi )=\pi$, if and only if, $\pi$ is generic. We will slightly abuse notation by using $\varSigma$ as a notation for a given element in $\mathfrak {S}_{G_n}$ as well.

In fact, our treatment of standard modules will not be focused on the above definition, but rather on the following well-known description of a standard module in terms of segments.

Proposition 3.1

Every $\varSigma \in \mathfrak {S}_{G_n}$ can be realized as an induced representation of the form $\varDelta _1\times \cdots \times \varDelta _t$, where each $\varDelta _i$ is a segment, such that $\varDelta _i$ does not precede $\varDelta _j$ whenever $i<j$.

Proof

Suppose that $\varSigma$ is parabolically induced from $\kappa _1\nu ^{\lambda _1}\otimes \cdots \otimes \kappa _t\nu ^{\lambda _t}$, as in the definition. It is known that a tempered representation $\kappa _i\in \varPi _{G_{m_i}}$ can be realized as an induced representation of the form $\varDelta _{i,1}\times \cdots \times \varDelta _{i,r_i}$, where $\{\varDelta _{i,j}\}_j$ is a uniquely defined multiset³ of segments, all of which have real exponent 0. Bearing in mind that $\nu ^{\lambda _i}\varDelta (\rho ,a,b)\cong \varDelta (\nu ^{\lambda _i}\rho ,a,b)$ and the transitivity property of parabolic induction, we see that $\varSigma$ can be realized as $\varDelta _1\times \cdots \times \varDelta _t$, where each $\varDelta _i$ is a segment, such that $\mathfrak {R}(\varDelta _1)<\cdots <\mathfrak {R}(\varDelta _t)$. $\square$

We will call an induced representation as described in the above proposition a rangée module. Each rangée module is a realization of a standard module. Two rangée modules are isomorphic, if and only if, they differ by a permutation on their defining segments. Moreover, it is true that each multiset of segments can be reordered in such a way that their multiplication would give a rangée module. Thus, $\mathfrak {S}_{G_n}$ is in bijection with multisets of segments.

If ${\mathcal {S}}$ is a rangée module realization of the standard module $\varSigma \in \mathfrak {S}_{G_n}$, we will write $\overline{{\mathcal {S}}}=\varSigma$. Yet, it will be useful to make a distinction between an element $\varSigma \in \mathfrak {S}_{G_n}$ and any of its concrete realizations ${\mathcal {S}}$ as a representation induced from a specified ordering of segments.

For a supercuspidal $\rho \in \varPi _{G_k}$, consider the collection $[\rho ]= \{\nu ^l\rho \,:\,l\in {\mathbb {Z}}\}$. Given a rangée module ${\mathcal {S}} = \varDelta _1\times \cdots \times \varDelta _t$, we set ${\mathcal {S}}_{[\rho ]} = \varDelta _{i_1}\times \cdots \times \varDelta _{i_s}$, where $1\le i_1< \cdots < i_s\le t$ are the indices for which $\varDelta _{i_j}\cong \varDelta (\rho , a,b)$ for some a, b. Let us set ${\mathcal {S}}_{[\rho ]}=1$ if there are no such indices. In these terms we always have a canonical (up to permutation) decomposition ${\mathcal {S}} \cong {\mathcal {S}}_{[\rho _1]}\times \cdots \times {\mathcal {S}}_{[\rho _l]}$ for some supercuspidals $\rho _1,\ldots ,\rho _l$, such that $[\rho _i]$ and $[\rho _j]$ are disjoint for distinct i, j. Clearly, if $\overline{{\mathcal {S}}}= \varSigma$, then $\varSigma _{[\rho ]}:= \overline{{\mathcal {S}}_{[\rho ]}}$ is well-defined.

We will say that a rangée module ${\mathcal {S}}$ is right-ordered, if ${\mathcal {S}} = {\mathcal {S}}_{[\rho _1]}\times \cdots \times {\mathcal {S}}_{[\rho _l]}$ and for each i, ${\mathcal {S}}_{[\rho _i]} = \varDelta (\rho _i, a^i_1,b^i_1)\times \cdots \times \varDelta (\rho _i, a^i_{t_i}, b^i_{t_i})$ with $b^i_1\ge b^i_2\ge \cdots \ge b^i_{t_i}$, for all i. It is easily seen that each $\varSigma \in \mathfrak {S}_{G_n}$ has a (possibly non-unique) right-ordered rangée module realization.

For $\pi \in \varPi _{G_n}$ we set $\pi _{[\rho _1]},\ldots ,\pi _{[\rho _l]}$ to be the irreducible representations of the corresponding groups, such that $\varSigma (\pi _{[\rho _i]})= \varSigma _{[\rho _i]}$, for all $1\le i\le l$. This provides a decomposition of the form $\pi \cong \pi _{[\rho _1]}\times \cdots \times \pi _{[\rho _l]}$. The elements of the decomposition are sometimes called the pure components of $\pi$. When $\pi =\pi _{[\rho ]}$, we will say that $\pi$ is of pure type $[\rho ]$. Similar notation will also be used for standard modules.

3.2 Jacquet’s conjecture—first implication

Let us recall the results of [5, Propositions 11, 12] which state that a representation $\pi \in \varPi _{G_n}$ has at most one non-zero $H_n$-invariant functional up to a scalar. If indeed $\pi$ is distinguished, then we must have $\pi ^\tau \cong \pi ^\vee$. As mentioned, the converse claim is not always true. Yet, let us investigate the standard module $\varSigma (\pi )$ of $\pi \in \varPi _{G_n}$ which satisfies $\pi ^\tau \cong \pi ^\vee$.

It is easy to see that segments satisfy $\varDelta (\rho , a,b)^\tau \cong \varDelta (\rho ^\tau , a,b)$. It is also known ([18, 9.4]) that $\varDelta (\rho ,a,b)^\vee \cong \varDelta (\rho ^\vee , -b,-a)$. It then follows that a segment $\varDelta _1$ precedes another segment $\varDelta _2$, if and only if, $\varDelta _1^\tau$ precedes $\varDelta _2^\tau$, if and only if, $\varDelta _2^\vee$ precedes $\varDelta _1^\vee$. Thus, for a rangée module ${\mathcal {S}} = \varDelta _1\times \cdots \times \varDelta _t$, both the representations ${\mathcal {S}}^\tau \cong \varDelta _1^\tau \times \cdots \times \varDelta _t^\tau$ and $\varDelta _t^\vee \times \cdots \times \varDelta _1^\vee$ will be rangée modules. If $\overline{{\mathcal {S}}}=\varSigma$, let us denote by $\varSigma ^\tau$ the isomorphism class of the former, and by $\varSigma ^*$ the class of the latter. Let us remark though, that ${\mathcal {S}}^\vee$ is generally not a standard module, hence, $\varSigma ^*$ must not be confused with the contragredient representation to $\varSigma$.

Proposition 3.2

Any $\pi \in \varPi _{G_n}$ satisfies $\varSigma (\pi ^\tau ) = \varSigma (\pi )^\tau$ and $\varSigma (\pi ^\vee )= \varSigma (\pi )^*$. In particular, $\pi ^\tau \cong \pi ^\vee$ holds, if and only if, $\varSigma (\pi )^*= \varSigma (\pi )^\tau$.

Proof

Since the Galois automorphism is obviously an exact functor, $\pi ^\tau$ is the irreducible quotient of $\varSigma (\pi )^\tau$ and the first equality must hold. The second equality is proved, for example, in [17, Proposition 5.6]. $\square$

Our first mission is to show that distinction of $\varSigma (\pi )$, for $\pi \in \varPi _{G_n}$, already imposes the condition $\pi ^\tau \cong \pi ^\vee$. For that we will need the following key lemma.

Lemma 3.3

Suppose ${\mathcal {S}}$ is a right-ordered rangée module realization of a standard module of $G_n$ that is induced from segments on a Levi subgroup M. Then, for every non-M-admissible element $w\in W_2[M]$, the space of $H_n$-invariant functionals on the representation ${\mathcal {V}}_w({\mathcal {S}})$ is zero.

Proof

Assume the contrary, that is, $w\in W_2[M]$ is a non-admissible element such that ${\mathcal {V}}_w({\mathcal {S}})$ has a non-zero $H_n$-invariant functional $\ell$.

Let us write $\delta =\otimes _{i\in \mathfrak {J}_M} \delta _i$ for the M-representation from which ${\mathcal {S}}$ is induced (each $\delta _i$ is a segment). We can also write

$$\begin{aligned} r_{M(w),M}(\delta ) = \otimes _{(i,j)\in \mathfrak {J}_{M(w),M}} \delta _{i,j}, \end{aligned}$$

where $\delta _{i,1}\otimes \cdots \otimes \delta _{i,s_i}$ is the Jacquet module of $\delta _i$ as a representation of the corresponding Levi subgroup of $G_{m_i}$. From the formula for Jacquet modules of segments we know that each $\delta _{i,j}$ must be a segment or the zero representation. In particular, if $\delta _{i,j}$ is distinguished, then it must satisfy $\delta _{i,j}^\vee \cong \delta _{i,j}^\tau$. Combining the last fact, Lemma 2.1, and the existence of $\ell$, we conclude that the $\delta _{i,j}$’s must all be non-zero representations and that $\delta _{\epsilon _w(i,j)}^\vee \cong \delta _{i,j}^\tau$ holds for all $(i,j)\in \mathfrak {J}_{M(w),M}$.

Since w is non-M-admissible, M(w) is strictly contained in M, which means there exists $i_0\in \mathfrak {J}_M$ with $s_{i_0}>1$. Let us assume $i_0$ is the minimal such index. Then, $\delta _{i_0} = \varDelta (\rho ,a,b)$ for some supercuspidal $\rho$ and integers $a<b$, $\delta _{i_0,1} = \varDelta (\rho , d+1,b)$ and $\delta _{i_0,2} = \varDelta (\rho ,c,d)$ for some integers $a\le c\le d< b$. Suppose that $\epsilon _w(i_0,1) = (i_1,j_1)$ and $\epsilon _w(i_0,2) = (i_2, j_2)$. We know that $\delta _{i_1,j_1}\cong \left( \delta _{i_0,1}^\tau \right) ^\vee \cong \varDelta \left( \left( \rho ^\tau \right) ^\vee , -b,-d-1\right)$, and similarly $\delta _{i_2,j_2}\cong \varDelta (\left( \rho ^\tau \right) ^\vee , -d,-c)$. We also know that $i_1<i_ 2$. Recalling that ${\mathcal {S}}$ was right-ordered, this must mean that $\delta _{i_1}\cong \varDelta (\left( \rho ^\tau \right) ^\vee , a', b')$ for some $a'\le -b$ and $-c\le b'$. Now, since $-d-1<-c$, we deduce that $j_1>1$ and that $\delta _{i_1,1}\cong \varDelta (\left( \rho ^\tau \right) ^\vee , e,b')$ for some $-d\le e\le b'$. But, that means $\epsilon _w(i_1,1)=(i_3,j_3)$ for some $i_3<i_0$. From minimality of $i_0$, we must have $j_3=1$ and $\delta _{i_3} = \delta _{i_3,j_3} \cong \left( \delta _{i_1,1}^\tau \right) ^\vee \cong \varDelta (\rho ,-b',-e)$. Finally, notice that $-e< b$, which is a contradiction to ${\mathcal {S}}$ being right-ordered. $\square$

Let ${\mathcal {S}}=\varDelta _1\times \cdots \times \varDelta _t$ be a right-ordered rangée module. Let $\ell$ be a non-zero $H_n$-invariant functional on ${\mathcal {S}}$. When filtering the representation space of ${\mathcal {S}}$ as an induced representation from segments on a Levi subgroup M, $\ell$ must induce a non-zero $H_n$-invariant functional on ${\mathcal {V}}_w({\mathcal {S}})$, for some $w\in W_2[M]$. By the above lemma, w must be M-admissible, hence, $\epsilon _w$ is a permutation on $\mathfrak {J}_M$. Recalling Lemma 2.1, we see that $\varDelta ^\tau _{\epsilon _w(i)}\cong \varDelta _i^\vee$ for all $i\in \mathfrak {J}_M$ with $\epsilon _w(i)\ne i$ and that $\varDelta _i$ is distinguished for $i\in \mathfrak {J}_M$ such that $\epsilon _w(i)=i$. This analysis has the following corollaries.

Proposition 3.4

If ${\mathcal {S}}= \varDelta _1\times \cdots \varDelta _t$ is any rangée module realization of a distinguished standard module of $G_n$, then $\varDelta ^\tau _{\epsilon (i)}\cong \varDelta _i^\vee$ for some involution $\epsilon$ on $\{1,\ldots ,t\}$. In particular, $\overline{{\mathcal {S}}}^*=\overline{{\mathcal {S}}}^\tau$.

Moreover, when $\epsilon (i)=i$, the segment $\varDelta _i$ is distinguished.

Proof

There is a permutation $\alpha$ of $\{1,\ldots ,t\}$ such that ${\mathcal {S}}_\alpha = \varDelta _{\alpha (1)}\times \cdots \times \varDelta _{\alpha (t)}$ is a right-ordered rangée module, and $\overline{{\mathcal {S}}_\alpha }=\overline{{\mathcal {S}}}$. Recalling again that a distinguished segment $\varDelta _i\in \varPi _{G_{m_i}}$ must satisfy $\varDelta _i^\tau \cong \varDelta _i^\vee$, we see that $\epsilon := \alpha ^{-1}\epsilon _w\alpha$ would fill the requirements of the statement.

The existence of such $\epsilon$ also shows the multisets of segments defining both $\overline{{\mathcal {S}}}^*$ and $\overline{{\mathcal {S}}}^\tau$ are the same. $\square$

Remark 3.1

The above can be seen as a generalization of the main theorem of [13], where the case of a generic irreducible representation was handled, that is, when $\overline{{\mathcal {S}}} = \varSigma (\pi )=\pi \in \varPi _{G_n}$.

Corollary 3.5

If $\varSigma (\pi )$ is distinguished for $\pi \in \varPi _{G_n}$, then $\pi ^\vee \cong \pi ^\tau$.

Proposition 3.6

Suppose that ${\mathcal {S}}=\varDelta _1\times \cdots \times \varDelta _t$ is a rangée module realization of a standard module of $G_n$, such that $\varDelta _i\not \cong \varDelta _j$ for all distinct i, j. Then, $\dim Hom_{H_n}({\mathcal {S}}, {\mathbb {C}})\le 1$.

Proof

As before, we can assume ${\mathcal {S}}$ is right-ordered and parabolically induced from segments on a standard Levi subgroup $M<G_n$. Suppose that ${\mathcal {S}}$ is distinguished. We have seen that as a consequence ${\mathcal {V}}_w({\mathcal {S}})$ is distinguished for some M-admissible $w\in W_2[M]$. That forces $\varDelta ^\tau _{\epsilon _w(i)}\cong \varDelta _i^\vee$ for all i, as in the proof of Proposition 3.4. Now, if ${\mathcal {V}}_{w'}({\mathcal {S}})$ had been distinguished for some other M-admissible $w\ne w'\in W_2[M]$, it would have also imposed the condition $\varDelta ^\tau _{\epsilon _{w'}(i)}\cong \varDelta _i^\vee$. But, that condition cannot hold for two different involutions because of our assumption. Hence, ${\mathcal {V}}_w({\mathcal {S}})$ is the only distinguished geometric subquotient of ${\mathcal {S}}$. Yet, since segments are irreducible, by the general multiplicity-one theorem $\dim Hom_{H_{m_i}}(\varDelta _i,{\mathbb {C}})\le 1$ (where $\varDelta _i\in \varPi _{G_{m_i}}$). Together with Lemma 2.1, it implies that $\dim Hom_{H_n}({\mathcal {V}}_w({\mathcal {S}}),{\mathbb {C}})\le 1$. The validity of the statement follows easily. $\square$

3.3 Jacquet’s conjecture—converse implication

We treat the converse problem, that is, what can be said about the distinction properties of a general standard module $\varSigma$ which satisfies $\varSigma ^*=\varSigma ^\tau$.

Let $\eta$ be the quadratic character of $F^\times$ associated to the extension E / F. Let $\chi$ be any extension of $\eta$ to $E^\times$. We also denote by $\eta$ and $\chi$ the corresponding characters of $H_m$ and $G_m$, for any m, obtained by composition with the determinant maps $G_m\rightarrow G_1$, $H_m\rightarrow H_1$. Since $\chi$ is trivial on the group of norms (for the extension E / F) of $E^\times$, we have $\chi ^\tau = \chi ^{-1}$.

The representation $\chi \pi$ is distinguished, if and only, $\pi$ is $\eta$-distinguished. Note, that $\chi (\pi _1\times \pi _2)\cong (\chi \pi _1)\times (\chi \pi _2)$, and that $\chi \varDelta (\rho ,a,b)\cong \varDelta (\chi \rho ,a,b)$ for all segments. In particular, $\mathfrak {S}_{G_n}$ is closed under tensoring with $\chi$. In this context we note the following obvious corollary of Lemma 2.2.

Corollary 3.7

Suppose that $M=M_{(m_1,\ldots ,m_t)}< G_n$ is a standard Levi subgroup. Let $\sigma _i$ be a $\eta$-distinguished admissible representation of $G_{m_i}$, for all $1\le i\le t$. Then, $\pi = \sigma _1 \times \cdots \times \sigma _t$ is $\eta$-distinguished.

Let us recall what is known about distinction of segments. We will recast the accumulated results of [9, 12] and [1] on the issue into a unified notation.

Note, that $\mathfrak {R}((\pi ^\tau )^\vee )=-\mathfrak {R}(\pi )$ for all $\pi \in \varPi _{G_n}$. Now, when $\rho \in \varPi _{G_n}$ is a supercuspidal representation satisfying $[(\rho ^\tau )^\vee ] = [\rho ]$, we have $\mathfrak {R}(\rho )= nr + \mathfrak {R}((\rho ^\tau )^\vee )= nr - \mathfrak {R}(\rho )$ for some integer r. Thus, $\frac{\mathfrak {R}(\rho )}{n}$ must be half-integer.

Proposition 3.8

Let $\rho \in \varPi _{G_n}$ be a supercuspidal representation satisfying $(\rho ^\tau )^\vee = \rho$. Then, there is a bit $\gamma (\rho )\in \{0,1\}$, such that $\rho$ is $\eta ^{\gamma (\rho )}$-distinguished.

Proof

It follows from the assumption that $\mathfrak {R}(\rho )=0$. As a supercuspidal representation with a unitary central character, $\rho$ is square-integrable. The statement then follows essentially from the main theorem of [9]. We only remark why the requirement on the central character in Kable’s result is superfluous with our formulation. Indeed, the proof of [9, Theorem 7] shows that the local Asai L-function of one of the representations $\rho$ and $\chi \rho$ has a pole at 0. Applying [9, Theorem 4] is then enough to finish the argument. $\square$

Proposition 3.9

Let $\rho \in \varPi _{G_n}$ be a supercuspidal representation, and $\varDelta = \varDelta (\rho ,a,b)$ a segment. Then,

• 1. The identity $\varDelta \cong (\varDelta ^\tau )^\vee$ holds, if and only if, $[(\rho ^\tau )^\vee ] = [\rho ]$ and $\mathfrak {R}(\varDelta )=0$.

• 2. Suppose that $[(\rho ^\tau )^\vee ] = [\rho ]$ holds. Then, there is an invariant bit $\gamma (\rho )= \gamma \left( [\rho ]\right) \in \{0,1\}$ extending the previous definition of $\gamma$ for unitarizable supercuspidals, such that if $\varDelta \cong (\varDelta ^\tau )^\vee$ holds, then the segment $\varDelta$ is $\eta ^{\gamma (\rho )}$-distinguished, and is not $\eta ^{\gamma (\rho )+1}$-distinguished.

Proof

• 1. If $\varDelta \cong (\varDelta ^\tau )^\vee$ holds, then $[(\rho ^\tau )^\vee ] = [\rho ]$ and $\mathfrak {R}(\varDelta )=0$ are immediate. Conversely, suppose that $[(\rho ^\tau )^\vee ] = [\rho ]$ holds. Note, that

  $$\begin{aligned} \mathfrak {R}(\varDelta )= \frac{b-a+1}{2}\left( na+nb+2\mathfrak {R}(\rho )\right) \end{aligned}$$

  as a sum of an arithmetic progression. Thus, when $\mathfrak {R}(\varDelta )=0$, we have $a+b=-2\mathfrak {R}(\rho )/n$. Since $\rho$ is the only element of $[\rho ]$ with real exponent $\mathfrak {R}(\rho )$, we must have $\nu ^{2\mathfrak {R}(\rho )/n}(\rho ^\tau )^\vee \cong \rho$. From this, $\varDelta ((\rho ^\tau )^\vee ,-b,-a)\cong \varDelta (\rho ,a,b)$ easily follows.

• 2. We denote $\rho _0 = \nu ^{-\mathfrak {R}(\rho )/n}\rho$ and $\gamma _0=\gamma (\rho _0)$, as defined in Proposition 3.8. When $\varDelta \cong (\varDelta ^\tau )^\vee$ holds, [12, Corollary 4.2] states that $\varDelta$ is $\eta ^{\gamma _0+a-b}$-distinguished. Yet, we have observed above that in this the parity of $a-b$ is the same as $2\mathfrak {R}(\rho )/n$. Thus,

  $$\begin{aligned} \gamma (\rho ):= \gamma _0 + 2\mathfrak {R}(\rho )/n \;(\mod 2) \end{aligned}$$

  will satisfy the condition in the statement. This definition of $\gamma$ is also easily seen to be an invariant of $[\rho ]$. Finally, the fact that $\varDelta$ cannot be both distinguished and $\eta$-distinguished is proved in [1]. $\square$

Remark 3.2

We clearly have $\gamma (\chi \rho ) =1-\gamma (\rho )$.

Also, if $[(\rho ^\tau )^\vee ] = [\rho ]$, note that $((\nu ^{1/2}\rho )^\tau )^\vee = \nu ^{-1/2}(\rho ^\tau )^\vee \in [\nu ^{1/2}\rho ]$. Thus, $\gamma (\nu ^{1/2}\rho )$ is well-defined. Going through the definition of $\gamma$ in the above proof, one can deduce that $\gamma (\nu ^{1/2}\rho ) = 1- \gamma (\rho )$.

The knowledge of the distinction properties of segments can be combined with the Blanc–Delorme method for producing functionals, and thus learning about the distinction of standard modules. Let us start with the treatment of a standard module of pure type.

Lemma 3.10

Suppose that $\varSigma = \varSigma _{[\rho ]}= \left( \varSigma ^\tau \right) ^*\in \mathfrak {S}_{G_n}$ for a supercuspidal $\rho$. Then $\varSigma$ is $\eta ^{\gamma (\rho )}$-distinguished.

If the real exponents of all segments from which $\varSigma$ is constructed are non-zero, then $\varSigma$ is both distinguished and $\eta$-distinguished.

Proof

Let us write the multiset of segments which define $\varSigma$ as $\{\varDelta _i\}_{i\in I}$. Let us partition this multiset as $I = I_{-}\cup I_0 \cup I_+$ according to whether $\mathfrak {R}(\varDelta _i)$ is negative, zero or positive. Define ${\mathcal {S}}_0 = \times _{i\in I_0} \varDelta _i$, with an arbitrary order of multiplication. Recall again, that if a segment $\varDelta '$ precedes $\varDelta ^{''}$, we must have $\mathfrak {R}(\varDelta ')<\mathfrak {R}(\varDelta ^{''})$. Since $\mathfrak {R}(\varDelta _i)=0$ for all $i\in I_0$, ${\mathcal {S}}_0$ is a rangée module.

Clearly, the assumption on $\varSigma$ compels $[(\rho ^\tau )^\vee ] = [\rho ]$ to hold. So, by Proposition 3.9, all $\{\varDelta _i\}_{i\in I_0}$ are $\eta ^{\gamma (\rho )}$-distinguished. By passing from $\varSigma$ to $\chi ^{\gamma (\rho )} \varSigma$, we may assume they are in fact all distinguished while retaining the condition $\varSigma ^\tau = \varSigma ^*$. Thus, by Lemma 2.2, ${\mathcal {S}}_0$ is distinguished. Clearly when ${\mathcal {S}}_0=1$, what follows applies to both $\varSigma$ and $\chi \varSigma$.

Let ${\mathcal {S}}_+ = \varDelta ^+_1\times \cdots \times \varDelta ^+_t$ be a rangée module constructed from the segments $\{\varDelta _i\}_{i\in I_+}$. By our assumption $\{\varDelta _i\}_{i\in I}$ is closed under the operation $\varDelta \mapsto (\varDelta ^\tau )^\vee$. Since this operation negates the real exponent of a segment, it actually induces a bijection $w:I_+\rightarrow I_-$ such that $\varDelta _{w(i)}\cong (\varDelta _i^\tau )^\vee$, for all $1\le i\le t$. If $\varDelta ^+_j = \varDelta _i$, let us define $\varDelta ^-_j:= \varDelta _{w(i)}$. Then, ${\mathcal {S}}_{-} = \varDelta ^{-}_t \times \cdots \times \varDelta ^-_1$ is a rangée module.

Moreover, arguing as before about the real exponent of preceding segments, we see that

$$\begin{aligned} \varDelta ^+_1\times \cdots \times \varDelta ^+_t \times \times _{i\in I_0} \varDelta _i \times \varDelta ^{-}_t \times \cdots \times \varDelta ^{-}_{1} \end{aligned}$$

is a rangée module realization of $\varSigma$. In particular, it obviously means that $\varSigma$ can be realized as the induced representation $\pi :=\varDelta ^+_1\times \cdots \times \varDelta ^+_t \times {\mathcal {S}}_0 \times \varDelta ^{-}_t \times \cdots \times \varDelta ^-_1$. Since $\varDelta ^+_i\cong ((\varDelta ^-_i)^\tau )^\vee$ for all $1\le i\le t$ and ${\mathcal {S}}_0$ is distinguished, by Lemma 2.1 ${\mathcal {V}}_\beta (\pi )$ is distinguished, where $\beta$ is as described in Proposition 2.3. So, by that proposition $\pi$ must be distinguished as well. $\square$

In general, we can now formulate the analogue of the Jacquet’s conjecture on the level of standard modules.

Theorem 3.11

Let $\pi \in \varPi _{G_n}$ be such that $\pi ^\vee \cong \pi ^\tau$. Then, there is a decomposition $\pi \cong \pi _1\times \pi _2\times \pi _3$, where $\pi _i\in \varPi _{G_{n_i}}$ for $i=1,2,3$, $\varSigma (\pi ) = \varSigma (\pi _1)\times \varSigma (\pi _2)\times \varSigma (\pi _3)$, $\varSigma (\pi _1)$ is both distinguished and $\eta$-distinguished, $\varSigma (\pi _2)$ is distinguished but not $\eta$-distinguished, and $\varSigma (\pi _3)$ is not distinguished but $\eta$-distinguished. Each of $\pi _1, \pi _2, \pi _3$ may be missing from the decomposition.

Proof

Let $\varSigma (\pi ) = \varSigma (\pi _{[\rho _1]})\times \cdots \times \varSigma (\pi _{[\rho _t]})$ be the canonical decomposition to standard modules of the pure components of $\pi$. By Proposition 3.2, $\varSigma (\pi )^*= \varSigma (\pi )^\tau$. Hence, there is an involution w on $\{1,\ldots ,t\}$, such that $\left[ \rho _{w(i)}^\tau \right] = [\rho _i^\vee ]$, and $\varSigma (\pi _{[\rho _{w(i)}]})^*= \varSigma (\pi _{[\rho _i]})^\tau$. Since a change in the order of the pure components is of no effect, we can assume that there is $0\le r<t$ such that $w(2i)=2i-1$ for all $1\le i\le r$, and $w(i)=i$ for all $2r<i\le t$.

Fix $1\le i\le r$. Let $\varDelta _1\times \cdots \times \varDelta _k$ be a realization of $\varSigma (\pi _{[\rho _{2i}]})$. Then,

$$\begin{aligned} {\mathcal {S}} = (\varDelta _k^\tau )^\vee \times \cdots \times (\varDelta _1^\tau )^\vee \times \varDelta _1\times \cdots \times \varDelta _k \end{aligned}$$

must be a realization of $\varSigma (\pi _{[\rho _{2i-1}]})\times \varSigma (\pi _{[\rho _{2i}]})$. Yet, by Proposition 2.3 and Lemma 2.1, ${\mathcal {S}}$ is distinguished. Furthermore, the fact that $\left( \left( \chi \varDelta \right) ^\tau \right) ^\vee \cong \chi \left( \varDelta ^\tau \right) ^\vee$ shows that $\chi {\mathcal {S}}$ is distinguished by the same argument.

By invoking Lemma 2.2, we see that $\varSigma '= \left( \varSigma (\pi _{[\rho _1]})\times \varSigma (\pi _{[\rho _2]})\right) \times \cdots \times \left( \varSigma (\pi _{[\rho _{2r-1}]})\times \varSigma (\pi _{[\rho _{2r}]})\right)$ is both distinguished and $\eta$-distinguished.

Finally, we have $\varSigma (\pi )= \varSigma '\times \varSigma (\pi _{[\rho _{2r+1}]})\times \cdots \times \varSigma (\pi _{[\rho _t]})$, where each $2r<i\le t$ satisfies $\varSigma (\pi _{[\rho _{i}]})^*= \varSigma (\pi _{[\rho _i]})^\tau$. Thus, after a proper rearrangement of the indices, by Lemma 3.10 there will be $2r+1\le s_1\le s_2\le t$, such that $\varSigma (\pi _{[\rho _{i}]})$ is both distinguished and $\eta$-distinguished for all $2r+1\le i\le s_1$, distinguished for all $s_1\le i\le s_2$, and $\eta$-distinguished for all $s_1\le i\le t$. Put

$$\begin{aligned} \pi _1'=\times _{i=1}^{s_1}\pi _{[\rho _i]} \quad \pi _2'=\times _{i=s_1+1}^{s_2}\pi _{[\rho _i]}\quad \pi _3'=\times _{i=s_2+1}^{t}\pi _{[\rho _i]}. \end{aligned}$$

The statement will then be partially satisfied by Lemma 2.2 and Corollary 3.7, when setting $\pi _i=\pi _i'$. For the complete statement, we should switch to $\pi _1:=\pi _1'\times \pi _2'$ and omit $\pi _2$ from the decomposition, in case $\varSigma (\pi _2')$ happens to be $\eta$-distinguished as well. A similar switch can be done when $\pi _3'$ happens to be distinguished. $\square$

Reading the above theorem, it may be tempting to conjecture that the irreducible representations $\pi _i$, $i=1,2,3$, should satisfy the same distinction properties as their respective standard modules. Yet, we will see that the class of ladder representations serves as a source of examples for $\pi \in \varPi _{G_n}$ of pure type, whose standard module is both distinguished and $\eta$-distinguished, while $\pi$ itself satisfies only one “kind” of distinction.

4 Distinction of ladder representations

We will say that a rangée module ${\mathcal {S}}= \varDelta _1\times \cdots \times \varDelta _t$ is of proper ladder type if $\varDelta _i$ precedes $\varDelta _{i-1}$, for all $1< i\le t$. Note, that for such ${\mathcal {S}}$, the standard module $\overline{{\mathcal {S}}}\in \mathfrak {S}_{G_n}$ has a unique rangée module realization. Hence, we can safely say that $\overline{{\mathcal {S}}}$ is of proper ladder type. A representation $\pi \in \varPi _{G_n}$ is called a proper ladder representation if $\varSigma (\pi )$ is of proper ladder type. In particular, proper ladder representations are of pure type.

In [10], a ladder representation was defined as $\pi \in \varPi _{G_n}$, for which $\varSigma (\pi )$ can be realized as $\varDelta (\rho ,a_1,b_1)\times \cdots \times \varDelta (\rho ,a_t,b_t)$, with $a_1>a_2>\cdots >a_t$ and $b_1>b_2>\cdots >b_t$.

The following straightforward lemma shows that ladder representations are easily decomposed into proper ladder representations.

Lemma 4.1

Let $\pi$ be a ladder representation. There are unique proper ladder representations $\pi _1,\ldots ,\pi _k$ such that

$$\begin{aligned} \varSigma (\pi ) = \varSigma (\pi _1)\times \cdots \times \varSigma (\pi _k) \end{aligned}$$

holds, and for which the defining segments of $\varSigma (\pi _i)$ do not precede those of $\varSigma (\pi _j)$, for distinct i, j. Moreover, $\pi \cong \pi _1\times \cdots \times \pi _k$.

Proof

Suppose that $\pi$ is realized as $\varDelta (\rho ,a_1,b_1)\times \cdots \times \varDelta (\rho ,a_t,b_t)$. We write

$$\begin{aligned} i_1 = \min \{1\le i \le t\;:\; b_{i+1}< a_i-1\}\quad (b_{t+1}=-\infty ), \end{aligned}$$

and denote $\varSigma _1 = \varDelta (\rho ,a_1,b_1)\times \cdots \times \varDelta (\rho ,a_{i_1},b_{i_1})$. Next, we write

$$\begin{aligned} i_2 = \min \{i_1< i \le t\;:\; b_{i+1}< a_i-1\}, \end{aligned}$$

and $\varSigma _2 = \varDelta (\rho ,a_{i_1+1},b_{i_1+1})\times \cdots \times \varDelta (\rho ,a_{i_2},b_{i_2})$. We continue like so until $i_k=t$.

The irreducible representations $\pi _1,\ldots ,\pi _k$ for which $\varSigma (\pi _i)=\varSigma _i$ clearly satisfy the first conditions of the statement. The equality $\pi \cong \pi _1\times \cdots \times \pi _k$ then follows from [18, Proposition 8.5]. $\square$

Suppose that $\varDelta = \varDelta (\rho ,a,b)$ is a segment that precedes $\varDelta '=\varDelta (\rho ,a',b')$. Then, $(\varDelta \cup \varDelta ')\times (\varDelta \cap \varDelta ')$ is a sub-representation of $\varDelta '\times \varDelta$, where $\varDelta '\cup \varDelta = \varDelta (\rho , a,b')$ and $\varDelta '\cap \varDelta = \varDelta (\rho , a',b)$. Moreover, if $\varSigma (\pi ) = \varDelta '\times \varDelta$, then $\pi$ is given as $\varSigma (\pi )/\left( (\varDelta \cup \varDelta ')\times (\varDelta \cap \varDelta ')\right)$. Such description of the maximal sub-representation of a standard module was generalized in [10] for standard modules of ladder representations.

Suppose that $\pi \in \varPi _{G_n}$ is a proper ladder representation, with ${\mathcal {S}}=\varDelta _1\times \cdots \times \varDelta _t$ realizing $\varSigma (\pi )$. For all $i=1,\ldots ,t-1$, we define

$$\begin{aligned} {\mathcal {S}}_i = \varDelta _1\times \cdots \times \varDelta _{i-1} \times (\varDelta _{i+1}\cup \varDelta _i)\times (\varDelta _{i+1} \cap \varDelta _i)\times \varDelta _{i+2}\times \cdots \times \varDelta _t. \end{aligned}$$

It is easy to check that the ${\mathcal {S}}_i$’s are all standard modules, which by exactness of parabolic induction can be embedded as sub-representations of ${\mathcal {S}}$. Let us denote $\varSigma _i =\overline{{\mathcal {S}}_i}$, and consider $\varSigma _1,\ldots , \varSigma _{t-1}\subset \varSigma (\pi )$ as sub-representations. The main theorem of [10] states that $\pi \cong \varSigma (\pi )/\left( \varSigma _1+\cdots +\varSigma _{t-1}\right)$. We would like to use this description to obtain invariant functionals on ladder representations.

Theorem 4.2

Let $\pi = \pi _{[\rho ]}\in \varPi _{G_n}$ be a proper ladder representation, with $\varSigma (\pi )= \varDelta _1\times \cdots \times \varDelta _t$. Then $\pi ^\tau \cong \pi ^\vee$ holds, if and only if, $\pi$ is $\eta ^{\gamma (\rho )+t+1}$-distinguished.

Proof

As mentioned before, when $\pi$ is distinguished, $\pi ^\tau \cong \pi ^\vee$ follows from [5, Proposition 12]. In the same manner, when $\chi \pi$ is distinguished, $\chi ^\tau \pi ^\tau \cong \chi ^{-1}\pi ^\vee$ holds, but $\chi ^\tau =\chi ^{-1}$.

Conversely, suppose that $\pi ^\tau \cong \pi ^\vee$ holds.

By Proposition 3.2 we have $\varSigma (\pi )^*=\varSigma (\pi )^\tau$. Hence, $(\varDelta _t^\tau )^\vee \times \cdots \times (\varDelta _1^\tau )^\vee \cong \varDelta _1\times \cdots \varDelta _t$. Since there is only one rangée module realization of a standard module of proper ladder type, we must have $\varDelta _{t+1-i} \cong \left( \varDelta _i^\tau \right) ^\vee$. In particular, $\mathfrak {R}(\varDelta _i)=-\mathfrak {R}(\varDelta _{t+1-i})$. The ladder condition also imposes $\mathfrak {R}(\varDelta _1)>\mathfrak {R}(\varDelta _2)>\cdots > \mathfrak {R}(\varDelta _t)$. Thus, if t is even, $\mathfrak {R}(\varDelta _i)\ne 0$ for all i. Lemma 3.10 then indicates that $\varSigma (\pi )$ is $\eta ^{\gamma (\rho )+t+1}$-distinguished. The same lemma also gives the same conclusion when t is odd $(\eta ^{\gamma (\rho )+t+1}=\eta ^{\gamma (\rho )})$.

We exhibited a non-zero $(H_n,\eta ^{\gamma (\rho )+t+1})$-equivariant functional on $\varSigma (\pi )$. Now, we would like to show that it factors through the map $\varSigma (\pi )\mapsto \pi$. In other words, we like to show the functional vanishes on each $\varSigma _i,\,i=1,\ldots , t-1$. It is enough to show that these standard modules are not $\eta ^{\gamma (\rho )+t+1}$-distinguished.

Assume the contrary, that is, $\varSigma '=\chi ^{\gamma (\rho )+t+1}\varSigma _{i_0}$ is distinguished, for some $i_0$. Let $\varDelta '_1\times \cdots \times \varDelta '_t$ be a realization of $\varSigma '$, where $\varDelta '_j= \chi ^{\gamma (\rho )+t+1}\varDelta _j$ for $1\le j<i_0$ or $i_0+1<j\le t$, and $\varDelta '_{i_0}= \chi ^{\gamma (\rho )+t+1}\left( \varDelta _{i_0+1}\cup \varDelta _{i_0}\right)$, $\varDelta '_{i_0+1}= \chi ^{\gamma (\rho )+t+1}\left( \varDelta _{i_0+1}\cap \varDelta _{i_0}\right)$. By Proposition 3.4 there is an involution $\epsilon$ on $\{1,\ldots ,t\}$, such that $\left( \varDelta '^\tau _{j}\right) ^\vee \cong \varDelta '_{\epsilon (j)}$.

Suppose that $t+1-i_0\not \in \{i_0,i_0+1\}$. Then,

$$\begin{aligned} \varDelta '_{\epsilon ( t+1-i_0)}\cong \chi ^{\gamma (\rho )+t+1} \left( \varDelta _{t+1-i_0}^\tau \right) ^\vee \cong \chi ^{\gamma (\rho )+t+1}\varDelta _{i_0}. \end{aligned}$$

Yet, since $\varSigma (\pi )$ is of ladder type, it can be easily seen that none of $\varDelta '_1,\ldots ,\varDelta '_t$ can be isomorphic to $\chi ^{\gamma (\rho )+t+1}\varDelta _{i_0}$. Thus, we must have $t+1-i_0\in \{i_0,i_0+1\}$. The same argument also shows that $t+1-(i_0+1)\in \{i_0,i_0+1\}$. In other words, t must be even, and $i_0=t/2$.

Now, by repeating a similar argument we can see that if $\epsilon (i_0)\not \in \{i_0,i_0+1\}$, then $\varDelta '_{i_0}\cong \chi ^{\gamma (\rho )+t+1} \varDelta _{t+1-\epsilon (i_0)}$. That would have meant $\varDelta _{i_0+1}\cup \varDelta _{i_0} \cong \varDelta _{t+1-\epsilon (i_0)}$, which is again a contradiction, because $\varSigma (\pi )$ is of ladder type. So, $\epsilon (i_0)\in \{i_0,i_0+1\}$. Note, that $\left( \varDelta '^\tau _{i_0}\right) ^\vee \cong \varDelta '_{i_0+1}$ cannot hold, because those are segments of different lengths. Hence, $\epsilon (i_0)=i_0$, and by Proposition 3.4 it means $\varDelta _{i_0+1}\cup \varDelta _{i_0}$ is $\eta ^{\gamma (\rho )+t+1}$-distinguished. Recalling that t is even, this gives a contradiction to Proposition 3.9. $\square$

Theorem 4.3

Let $\pi =\pi _{[\rho ]}\in \varPi _{G_n}$ be a ladder representation satisfying

$$\begin{aligned} \pi ^\tau \cong \pi ^\vee . \end{aligned}$$

Suppose that $\pi \cong \pi _1\times \cdots \times \pi _k$, with each $\pi _i$ being a proper ladder representation, as in Lemma 4.1. Suppose further that $\varSigma (\pi ) = \varDelta _1\times \cdots \times \varDelta _t$.

Then, in case k is even, $\pi$ is both distinguished and $\eta$-distinguished.

In case k is odd, $\pi$ is $\eta ^{\gamma (\rho )+t+1}$-distinguished.

Proof

Recall that $\varSigma (\pi ) = \varSigma (\pi _1)\times \cdots \times \varSigma (\pi _k)$ and by Proposition 3.2,

$$\begin{aligned} \varSigma (\pi _k)^*\times \cdots \times \varSigma (\pi _1)^*= \varSigma (\pi )^*= \varSigma (\pi )^\tau = \varSigma (\pi _1)^\tau \times \cdots \times \varSigma (\pi _k)^\tau . \end{aligned}$$

The above equality shows two decompositions of a standard module of a ladder representation into standard modules of proper ladder type, each of which ordered by decreasing real exponent of their irreducible quotient. Thus, from uniqueness we have $\pi ^\vee _i\cong \pi ^\tau _{k+1-i}$, for all $1\le i\le k$.

In case k is even, we can use Proposition 2.3 to produce the desired functionals on $\pi$, and on $\chi \pi$, by applying the same argument as in the end of the proof of Lemma 3.10.

In case k is odd, this argument would still be valid if we knew that $\pi _{(k+1)/2}\cong (\pi _{(k+1)/2}^\tau )^\vee$ is $\eta ^{\gamma (\rho )+t+1}$-distinguished. Indeed, by Theorem 4.2, $\pi _{(k+1)/2}$ is $\eta ^{\gamma (\rho )+t_0+1}$-distinguished, where $t_0$ is the number of segments in the construction of $\varSigma (\pi _{(k+1)/2})$. Note, that for any $i\ne (k+1)/2$, $\varSigma (\pi _i)$ and $\varSigma (\pi _{k+1-i})$ are distinct standard modules constructed from an equal number of segments. Thus, $t-t_0$ is an even number, and $\eta ^{\gamma (\rho )+t+1}=\eta ^{\gamma (\rho )+t_0+1}$. $\square$

Finally, we want to complete the above result by showing that a ladder representation that is induced from an odd number of proper ladder representations, cannot be both distinguished and $\eta$-distinguished.

Lemma 4.4

Suppose that ${\mathcal {S}}={\mathcal {S}}_{[\rho ]}$ is a rangée module of pure type induced from t segments. If t is odd, then ${\mathcal {S}}$ is not $\eta ^{\gamma (\rho )+1}$-distinguished.

Proof

Assume that ${\mathcal {S}}':= \chi ^{\gamma (\rho )+1} {\mathcal {S}} = \varDelta _1\times \cdots \times \varDelta _t$ is distinguished. Let $\epsilon$ be the involution on $\{1,\ldots ,t\}$ supplied by Proposition 3.4. Since t is odd, there is a fixed point $1\le r\le t$ of $\epsilon$. Hence, by Proposition 3.4, $\varDelta _r$ is distinguished. However, $\varDelta _r\cong \chi ^{\gamma (\rho )+1} \varDelta (\rho ,a,b)$ for some a, b. This contradicts Proposition 3.9. $\square$

For the case of a proper ladder representation for which $\varSigma (\pi )$ is constructed from an even number of segments, the situation is more complicated. The standard module $\varSigma (\pi )$ will, in fact, be both distinguished and $\eta$-distinguished. Yet, we must show that not all of these functionals can factor through the quotient $\pi$.

For that cause, we will apply the theory of Gelfand-Kazhdan derivatives for representations of $G_n$. Let us recall the mirabolic subgroup $P_n< G_n$ consisting of matrices whose bottom row is $(0\cdots 0\;1)$. For an admissible representation $\sigma$ of $G_{k-1}$, denote by $\varPsi ^+(\sigma )$ the representation of $P_k$ obtained by composing the natural homomorphism $P_k\rightarrow G_{k-1}$ on $\nu ^{1/2}\sigma$. Also, there is a canonical functor $\varPhi ^+$ taking representations of $P_k$ to representations of $P_{k+1}$, whose definition we will refrain from writing here. Recall, that given $\pi \in \varPi _{G_n}$, there are well-defined finite-length (possibly zero) representations $\pi ^{(k)}$ of $G_{n-k}$ for $k=1,\ldots ,n$,⁴ called the derivatives of $\pi$. There is a filtration of $\pi$ as a $P_n$-representation, such that its subquotients are isomorphic to $(\varPhi ^+)^{k-1}\circ \varPsi ^+(\pi ^{(k)})$, $k=1,\ldots ,n$.

The theory of derivatives is useful for our analysis of distinction because of the following known equality ([1, Lemma 2.4]) for a finite-length admissible representation $\sigma$ of $G_{n-k}$ $(1\le k\le n-1)$:

$$\begin{aligned} \dim Hom_{P_n\cap H_n}((\varPhi ^+)^{k-1}\circ \varPsi ^+(\sigma ),{\mathbb {C}}) = \dim Hom_{H_{n-k}}(\nu ^{1/2}\sigma ,{\mathbb {C}}). \end{aligned}$$

Lemma 4.5

Let $\pi \in \varPi _{G_n}$ be a non-generic distinguished representation. Then $\nu ^{1/2}\sigma$ must be distinguished for at least one irreducible subquotient $\sigma$ of one of the derivatives of $\pi$.

Proof

When $\pi$ is non-generic, $\pi ^{(n)}$ is zero (one may take it as a definition). The non-zero $H_n$-invariant functional on $\pi$ induces a non-zero $P_n\cap H_n$-invariant functional on at least one of the $P_n$-subquotients $(\varPhi ^+)^{k-1}\circ \varPsi ^+(\pi ^{(k)})$ of $\pi$, for $1\le k\le n-1$. The rest follows from the above equality. $\square$

The so-called full derivative of a ladder representation was computed in [10, Theorem 14]. In other words, the semisimplification of all of the derivatives of a ladder $\pi \in \varPi _{G_n}$ is known, and in fact consists of ladder representations of smaller groups. Let us repeat it here. When $\varSigma (\pi )$ is of ladder type and is given by $\varDelta (\rho ,a_1,b_1)\times \cdots \times \varDelta (\rho ,a_t,b_t)$, the set of all irreducible representations appearing as subquotients of the derivatives of $\pi$ is

$$\begin{aligned} {\mathcal {D}}(\pi )=\left\{ \sigma \in \bigcup _{k<n}\varPi _{G_{k}}\,:\, \varSigma (\sigma )\cong \varDelta (\rho ,a'_1,b_1)\times \cdots \times \varDelta (\rho ,a'_t,b_t),\,\forall i\; a_i\!\le \! a'_i<a_{i-1} \right\} \!, \end{aligned}$$

where $a_0$ is taken as $\infty$ and $\varDelta (\rho , a,b)=1$ if $a>b$.

Theorem 4.6

Let $\pi \in \varPi _{G_n}$ be a ladder representation with a decomposition $\pi \cong \pi _1\times \cdots \times \pi _k$ into proper ladder representations, as in Lemma 4.1.

If k is odd, $\pi$ cannot be both distinguished and $\eta$-distinguished.

Proof

Suppose that $\pi =\pi _{[\rho ]}\in \varPi _{G_n}$ and that $\varSigma (\pi )=\varDelta _1\times \cdots \times \varDelta _t$. Recalling Theorem 4.2, we should prove that $\pi$ is not $\eta ^{\gamma (\rho )+t}$-distinguished. If it was, then pulling back the non-zero functional would make $\varSigma (\pi )$ $\eta ^{\gamma (\rho )+t}$-distinguished as well. For an odd t this cannot happen by Lemma 4.4. Thus, we will assume t is even.

We write $k_0= (k+1)/2$. Let us denote by $i_0$ and $t_0$ the indices for which

$$\begin{aligned} \varSigma (\pi _{k_0}) = \varDelta _{i_0}\times \cdots \times \varDelta _{i_0+t_0-1}\,. \end{aligned}$$

Assume $\pi ' = \chi ^{\gamma (\rho )} \pi$ is distinguished. Then $(\pi ')^\vee \cong (\pi ')^\tau$ holds. When writing $\pi '_i = \chi ^{\gamma (\rho )} \pi _i$, we clearly have $\pi '\cong \pi '_1\times \cdots \times \pi '_k$ as the decomposition of Lemma 4.1. The argument in the proof of Theorem 4.3 shows that $(\pi '_{k_0})^\vee \cong (\pi '_{k_0})^\tau$ holds in such case. The same proof also shows that $t_0$ must be even as well.

In particular, $t_0>1$, which means that $\varDelta _{i_0+1}$ precedes $\varDelta _{i_0}$. Hence, $\varSigma (\pi ')$ is not irreducible and $\pi '$ is non-generic. By Lemma 4.5, there is a representation $\sigma \in {\mathcal {D}}(\pi ')$, for which $\nu ^{1/2} \sigma$ is distinguished. Hence, its standard module $\varSigma (\nu ^{1/2} \sigma )$ is distinguished as well.

It can be easily seen from the description of the full derivative of a ladder representation that $\varSigma (\sigma ) = \varSigma (\sigma _1)\times \cdots \times \varSigma (\sigma _k)$, for some $\sigma _i\in {\mathcal {D}}(\pi '_i)$. Up to some possibly empty $\sigma _i$, this is the decomposition of Lemma 4.1.

From the same description, we see that $\varSigma (\sigma _{k_0})$ is constructed of either $t_0$ or $t_0-1$ segments.

Let us first handle the case that $\varSigma (\nu ^{1/2} \sigma _{k_0})=\nu ^{1/2}\varSigma ( \sigma _{k_0}) = \varDelta '_{i_0}\times \cdots \times \varDelta '_{i_0+t_0-1}$ (where all $\varDelta '_i$ are non-empty segments). Arguing the same as for $\pi '$, the distinction of $\nu ^{1/2} \sigma$ forces $(\nu ^{1/2} \sigma _{k_0})^\vee \cong (\nu ^{1/2} \sigma _{k_0})^\tau$. Now, by the proof of Theorem 4.2, this means that $\mathfrak {R}(\varDelta '_{2i_0 + t_0 -1-i})=-\mathfrak {R}(\varDelta '_i)$ for all $i_0\le i\le i_0+t_0-1$. In particular, $\sum _{i=i_0}^{i_0+t_0-1} \mathfrak {R}(\varDelta '_i)=0$. From the distinction of $\pi '$ we can again argue the same to deduce $\left( \sum _{i=i_0}^{i_0+t_0-1} \mathfrak {R}(\varDelta _i)=\;\right) \sum _{i=i_0}^{i_0+t_0-1} \mathfrak {R}(\chi ^{\gamma (\rho )}\varDelta _i)=0$. Yet, by the description of ${\mathcal {D}}(\pi ')$, $\mathfrak {R}(\varDelta '_i) \ge 1/2 + \mathfrak {R}(\chi ^{\gamma (\rho )}\varDelta _i)$, for all $i_0\le i\le t_0+i_0-1$. Hence, a contradiction.

Otherwise, $\varSigma (\nu ^{1/2} \sigma _{k_0})$ has a rangée module realization induced from $t_0-1$ segments. Suppose that $\varSigma =\varSigma (\nu ^{1/2} \sigma )$ has a rangée module realization induced from s segments. Again, referring to the same situation as in the proof of Theorem 4.3, we see that $s-(t_0-1)$ is even. Hence, s is odd. Keeping track of the tensoring operations we see that $\varSigma$ is of pure type $[\nu ^{1/2}\chi ^{\gamma (\rho )} \rho ]$. Since $\varSigma$ is distinguished, by Lemma 4.4 we must have $\gamma (\nu ^{1/2}\chi ^{\gamma (\rho )} \rho )=0$. The remark after Proposition 3.9 shows this is impossible, because obviously $\gamma (\chi ^{\gamma (\rho )} \rho )=0$ holds. $\square$