Positivity Properties for Spherical Functions of Maximal Young Subgroups

Abstract

Let $S_k \times S_{n-k}$ be a maximal Young subgroup of the symmetric group $S_n$. We introduce a basis ${{\mathcal {B}}}_{n,k}$ for the coset space $S_n/S_k \times S_{n-k}$ that is naturally parametrized by the set of standard Young tableaux with n boxes, at most two rows, and at most k boxes in the second row. The basis ${{\mathcal {B}}}_{n,k}$ has positivity properties that resemble those of a root system, and there is a composition series of the coset space in which each term is spanned by the basis elements that it contains. We prove that the spherical functions of the associated Gelfand pair are nonnegative linear combinations of the ${{\mathcal {B}}}_{n,k}$.

1 Introduction

A pair (G, K) of finite groups is called a Gelfand pair if $K \le G$ and the permutation representation of G on the set of left cosets $X = G/K$ of K in G is multiplicity-free as a ${{\mathbb {C}}}G$-module. One source of examples of Gelfand pairs arises from the action of a group G on a finite metric space (X, d). Such an action is said to be distance transitive if for all $(x_1, y_1), (x_2, y_2) \in X \times X$, we have $d(x_1, y_1) = d(x_2, y_2)$ if and only if there exists a $g \in G$ satisfying $g(x_1) = x_2$ and $g(y_1) = y_2$; this condition implies that G acts transitively as a group of isometries of X. If K is the stabilizer of $x_0 \in X$ under a distance transitive action of G, then (G, K) is a Gelfand pair [2, Lemma 4.3.4, Example 4.3.7]. Furthermore, in this case, the number of orbits of K on X (i.e., the rank of G acting on X as a permutation group as in [8, Definition 8.2.4]) is equal to the number of irreducible direct summands of the permutation module on the cosets G/K [2, Corollary 4.4.3 (iii)].

The action of a Weyl group on the weights of a minuscule representation of a simple Lie algebra satisfies the conditions of the previous paragraph with respect to Euclidean distance by [8, Theorem 8.2.22 (ii)], so each minuscule representation of a simple Lie algebra gives rise to a Gelfand pair. In this paper, we concentrate on the special case where the Lie algebra has type $A_{n-1}$, which means that the Gelfand pair (G, K) is given by $(S_n, S_k \times S_{n-k})$ for some $0< k < n$. We will assume that $n \ge 2$ throughout, and without loss of generality that $k \le n/2$. In this case, each left coset of $X = G/K$ can be naturally identified with a squarefree monomial of degree k in the commuting indeterminates $x_1, x_2, \ldots , x_n$, where the action of G is the natural action on subscripts, and where the identity coset K is identified with the monomial $x_1x_2 \cdots x_k$. If we denote the set of ${{\mathbb {C}}}$-valued functions on X by L(X), then L(X) decomposes as a ${{\mathbb {C}}}G$-module into a direct sum of pairwise nonisomorphic irreducible representations

$$\begin{aligned} L(X) \cong V_0 \oplus V_1 \oplus \cdots \oplus V_k. \end{aligned}$$

We will identify the vector space L(X) with the linear span, $V_{n,k}$, of the squarefree monomials of degree k in the commuting indeterminates $\{x_1, x_2, \ldots , x_n\}$. We will refer to both these versions of the coset basis as the monomial basis, and denote it by ${{\mathcal {M}}}_{n,k}$.

It follows from Frobenius reciprocity that each of the $V_j$ has a 1-dimensional K-invariant submodule. For each $0 \le j \le k$, the jth spherical function ${\Phi (n,k,j)}\in L(X)$ is defined to be the element of this 1-dimensional submodule that is normalized so that ${\Phi (n,k,j)}$ sends the identity coset $K=x_1x_2\cdots x_k$ to 1. The value of the spherical function ${\Phi (n,k,j)}$ on a coset gK turns out to be a function of the distance d between gK and K in the natural metric on X. These spherical functions are known explicitly [2, Theorem 6.1.10] and are sometimes called dual Hahn polynomials. They have applications to random walks and the Bernoulli–Laplace diffusion model [3, §3]. We will not use the metric in this paper, and instead view the spherical functions ${\Phi (n,k,j)}$ as homogeneous polynomials of degree k. Because the irreducible representations of $S_n$ over ${{\mathbb {C}}}$ are defined over ${{\mathbb {Q}}}$, we will work over the field ${{\mathbb {Q}}}$ unless stated otherwise, but scalars can be extended if necessary.

If B is a basis for an F-vector space V with $F \le {{\mathbb {R}}}$, we say that an element $v = \sum _{b \in B} \lambda _b b$ is B-positive with coefficients $\{\lambda _b\}_{b \in B}$ if we have $\lambda _b \ge 0$ for all $b \in B$. The spherical functions ${\Phi (n,k,j)}$ are generally not ${{\mathcal {M}}}_{n,k}$-positive elements of $V_{n,k}$, but in this paper, we will introduce a basis ${{\mathcal {B}}}_{n,k}$ for $V_{n,k}$ with respect to which the spherical functions are ${{\mathcal {B}}}_{n,k}$-positive. To illustrate this, consider the case $n=4$ and $k=1$, where we have ${{\mathcal {M}}}_{4,1}= \{x_1, x_2, x_3, x_4\}$ and

$$\begin{aligned} {{\mathcal {B}}}_{4,1} = \{x_1 - x_2, x_2 - x_3, x_3 - x_4, x_3 + x_4\}. \end{aligned}$$

If we identify the $x_i$ with an orthonormal basis of ${{\mathbb {R}}}^n$, the latter basis corresponds to the basis of simple roots of type $D_4$, as in [10, §2.10]. In this case, $\Phi (4,1,0)$ is both ${{\mathcal {M}}}_{4,1}$-positive and ${{\mathcal {B}}}_{4,1}$-positive:

$$\begin{aligned} \Phi (4,1,0)= & {} x_1 + x_2 + x_3 + x_4\\= & {} (x_1 - x_2) + 2(x_2 - x_3) + (x_3 - x_4) + 2(x_3 + x_4), \end{aligned}$$

whereas $\Phi (4,1,1)$ is not ${{\mathcal {M}}}_{4,1}$-positive, but is ${{\mathcal {B}}}_{4,1}$-positive:

$$\begin{aligned} \Phi (4,1,1) = x_1 - \frac{1}{3} x_2 - \frac{1}{3} x_3 - \frac{1}{3} x_4 = (x_1 - x_2) + \frac{2}{3} (x_2 - x_3) + \frac{1}{3} (x_3 - x_4). \end{aligned}$$

The purpose of this paper is to study ${{\mathcal {B}}}_{n,k}$-positivity for arbitrary $n \ge 2$ and $0 < k \le n/2$. We replace the root system of type $D_n$ by a generalization called k-roots. Using a suitable total order, we can construct a canonical basis ${{\mathcal {B}}}_{n,k}$ of k-roots analogous to the simple roots in the case $k=1$. The basis ${{\mathcal {B}}}_{n,k}$ is naturally parametrized by the set of lattice words in the alphabet $\{1, 2\}$ that have length n and at most k occurrences of 2, or equivalently (see Remark 3.2) by the set of standard Young tableaux with n boxes that have at most two rows and at most k boxes in the second row. The basis ${{\mathcal {B}}}_{n,k}$ may be constructed in other ways, for example by using Kazhdan–Lusztig theory (see Remark 3.13), but the k-root approach has the advantage that it is easy to deal with computationally.

The results of this paper are largely self-contained, although the key result Proposition 2.7 is implicit in recent work of the author and Xu [9] on the case $k=2$ in a much more general setting. In [9], a k-root is defined to be a symmetrized tensor product of k mutually orthogonal roots in the sense of Lie theory. The cases we study in this paper correspond to performing this construction with a root system of type D, where convenient Euclidean coordinates are available. We therefore usually dispense with the root system point of view, and instead think of k-roots as polynomials in these Euclidean coordinates. It should be noted that the polynomials corresponding to certain pairs of orthogonal roots, such as $(x_1 - x_2)(x_1 + x_2)$, do not appear in the construction because they are not linear combinations of squarefree monomials, and such polynomials are not counted as k-roots for the purposes of this paper.

We develop the combinatorial tools needed to define and study the canonical basis ${{\mathcal {B}}}_{n,k}$ in Sects. 2 and 3. Although the initial definition of ${{\mathcal {B}}}_{n,k}$ in Definition 2.6 may seem ad hoc, we will prove in Theorem 3.10 that ${{\mathcal {B}}}_{n,k}$ has a simple characterization as the set of positive k-roots that are minimal in the sense of being indecomposable into sums of other positive k-roots. Sect. 4 explores some applications of k-roots to representation theory. Theorem 4.5 gives a closed formula for the spherical functions in terms of k-roots; the formula does not involve the metric on the coset space G/K. We use this formula to prove that the main result of this paper, which is that the spherical functions are ${{\mathcal {B}}}_{n,k}$-positive. Theorem 4.8 gives a sufficient condition for a monomial basis element to be ${{\mathcal {B}}}_{n,k}$-positive.

2 k-Roots

Let $n \ge 2$ and $0 < k \le n/2$ be integers, and let $V_{n,k}$ be the ${{\mathbb {Q}}}$-vector space of dimension $\left( {\begin{array}{c}n\\ k\end{array}}\right)$ with basis consisting of the set ${{\mathcal {M}}}_{n,k}$ of all squarefree monomials of degree k in the commuting indeterminates $\{x_1, x_2, \ldots , x_n\}$.

The set ${{\mathcal {M}}}_{n,k}$ can be ordered lexicographically, as follows. Let $I = \{i_1, i_2, \ldots , i_k\}$ and $J = \{j_1, j_2, \ldots , j_k\}$ be two distinct subsets of $\{1, 2, \ldots , n\}$ of size k, and let $x_I = x_{i_1} x_{i_2} \cdots x_{i_k}$ and $x_J = x_{j_1} x_{j_2} \cdots x_{j_k}$ be the corresponding squarefree monomials. If t is the smallest element of the symmetric difference $I \ \Delta \ J$, then we say $x_I \prec x_J$ if $t \in I$, and $x_J \prec x_I$ if $t \in J$. (For example, we have $x_1 x_2 x_5 \prec x_1 x_3 x_4$, with $t=2$.)

We order the ${{\mathbb {Q}}}$-vector space $V_{n, k}$ by saying that a nonzero vector $v \in V_{n, k}$ satisfies $v > 0$ (respectively, $v < 0$) if the lexicographically minimal monomial appearing in v has a positive (respectively, negative) coefficient. We then say that $v_1 < v_2$ if $v_2 - v_1 > 0$. This makes $V_{n, k}$ into a totally ordered vector space. (Note that we have $x_1 x_2 x_5 > x_1 x_3 x_4$ in this case.)

Definition 2.1

Let ${{\mathcal {C}}}_{n,k}$ be the subset of $V_{n,k}$ consisting of all elements of the form

$$\begin{aligned} \prod _{r = 1}^k (\pm x_{i_{2r-1}} \pm x_{i_{2r}}), \end{aligned}$$

where the signs are chosen independently and where the set $\{i_1, i_2, \ldots , i_{2k}\}$ is a set of 2k distinct indices from the set $\{1, 2, \ldots , n\}$. An element of ${{\mathcal {C}}}_{n,k}$ is called a k-root. A k-root is called positive if it is positive in the ordering on $V_{n,k}$, and negative otherwise.

Remark 2.2

The definitions imply that if $\lambda$ is a scalar and $\alpha$ is a k-root, then $\lambda \alpha$ is also a k-root if and only if $\lambda = \pm 1$. Note that for each k-root $\alpha$, the factors $(\pm x_i \pm x_j)$ are well-defined up to order and multiplication by nonzero scalars, because they are the irreducible factors of the k-root in the unique factorization domain ${{\mathbb {Q}}}[x_1, x_2, \ldots , x_n]$.

Lemma 2.3

A k-root is positive if and only if it can be written in the form

$$\begin{aligned} \prod _{r = 1}^k (x_{i_{2r-1}} \pm x_{i_{2r}}), \end{aligned}$$

where the indices satisfy $i_{2r-1} < i_{2r}$ for all i.

Proof

Let $\alpha$ be a k-root. We can reorder each factor of $\alpha$ to be of the form $(\pm x_i \pm x_j)$ where $i < j$, and commute the negative signs on $x_i$ to the front to obtain

$$\begin{aligned} \alpha = \pm \prod _{r = 1}^k (x_{i_{2r-1}} \pm x_{i_{2r}}), \end{aligned}$$

where $i_{2r-1} < i_{2r}$ for all i. The lexicographically minimal monomial appearing in $\alpha$ is

$$\begin{aligned} x_{i_1} x_{i_3} \cdots x_{i_{2k-1}}, \end{aligned}$$

which occurs with coefficient 1 if $\alpha$ is in the form given in the statement, and with coefficient $-1$ if $-\alpha$ is in the form given in the statement. The conclusion now follows. $\square$

Definition 2.4

We say that a positive k-root is in normal form if it is in the form given by Lemma 2.3. If $\alpha$ is a positive k-root in normal form, then we call a factor of $\alpha$ antisymmetric if it is of the form $(x_i - x_j)$ for $i < j$, and symmetric if it is of the form $(x_i + x_j)$. If $x_i$ does not appear in the factorization of $\alpha$, then we say that i is an unused index of $\alpha$.

Note that the normal form of a k-root is unique up to reordering the factors.

In the next result, $\chi ^{(n-i,i)}$ refers to the character of the irreducible ${{\mathbb {C}}}S_n$-module corresponding to the two-part partition $(n-i, i)$ of n, as in [6, §4]. Part (ii) is well known.

Lemma 2.5

• (i) The set ${{\mathcal {C}}}_{n,k}$ is a spanning set for $V_{n,k}$ over ${{\mathbb {Q}}}$.

• (ii) The character of $V_{n,k}$ as a ${{\mathbb {C}}}S_n$-module is $\sum _{i = 0}^k \chi ^{(n-i, i)}$.

Proof

Let $x_I = x_{i_1}x_{i_2} \cdots x_{i_k}$ be an arbitrary monomial basis element. Because $k \le n/2$, we can choose indices $j_1, j_2, \ldots , j_k$ so that the set $\{i_1, i_2, \ldots , i_k, j_1, j_2, \ldots , j_k\}$ has cardinality 2k. By substituting $(x_{i_r}+x_{j_r})+(x_{i_r}-x_{j_r})$ for $2x_{i_r}$, the monomial $2^k x_I$ can be expressed as a sum of $2^k$ distinct k-roots, and (i) follows.

Since $V_{n,k}$ is induced from the trivial module of $S_k \times S_{n-k}$, its character corresponds to the product of the Schur functions $s_{(n-k)}$ and $s_{(k)}$. The Pieri rule shows that we have

$$\begin{aligned} s_{(n-k)} s_{(k)} = \sum _{i = 0}^k s_{(n-i, i)}, \end{aligned}$$

which proves (ii). $\square$

Definition 2.6

Let $\alpha \in {{\mathcal {C}}}_{n,k}$ be a positive k-root. We say that $\alpha$ has a defect if the normal form of $\alpha$ has any of the following features, where in each case we have $i< j< r < s$:

• (i) two factors of the form $(x_i \pm x_r)$ and $(x_j \pm x_s)$;

• (ii) a factor of the form $(x_i \pm x_s)$ and a symmetric factor of the form $(x_j + x_r)$;

• (iii) a factor of the form $(x_i \pm x_r)$, and an unused index j;

• (iv) a symmetric factor of the form $(x_i + x_j)$, and an unused index r.

In case (i), we say that $\alpha$ has a crossing; in case (ii), we say that $(x_j + x_r)$ is a nested symmetric factor; in case (iii), we say that j is a nested unused index; and in case (iv), we say that $(x_i + x_j)$ is an obstructed symmetric factor.

We define ${{\mathcal {B}}}_{n,k}$ to be the set of positive k-roots with no defects.

The next result is implicit in [9, §5], and will be very useful in the sequel.

Proposition 2.7

Let $\alpha \in {{\mathcal {C}}}_{n,k}$ be a positive k-root, written in normal form, and suppose that $\alpha$ has a defect. Then $\alpha$ can be written as a sum of positive k-roots that are strictly lower than $\alpha$ in the total order on $V_{n,k}$, by replacing the factor(s) involved in the defect according to the following rules, where $i< j< r < s$ in all cases:

$$\begin{aligned} (x_i - x_r)(x_j - x_s)&\quad \longrightarrow \quad (x_i - x_j)(x_r- x_s) + (x_i - x_s)(x_j - x_r); \end{aligned}$$

(2.1)

$$\begin{aligned} (x_i + x_r)(x_j - x_s)&\quad \longrightarrow \quad (x_i + x_j)(x_r- x_s) + (x_i + x_s)(x_j - x_r); \end{aligned}$$

(2.2)

$$\begin{aligned} (x_i - x_r)(x_j + x_s)&\quad \longrightarrow \quad (x_i - x_j)(x_r+ x_s) + (x_i + x_s)(x_j - x_r); \end{aligned}$$

(2.3)

$$\begin{aligned} (x_i + x_r)(x_j + x_s)&\quad \longrightarrow \quad (x_i + x_j)(x_r+ x_s) + (x_i - x_s)(x_j - x_r); \end{aligned}$$

(2.4)

$$\begin{aligned} (x_i - x_s)(x_j + x_r)&\quad \longrightarrow \quad (x_i - x_j)(x_r+ x_s) \nonumber \\&\qquad \qquad + (x_i + x_j)(x_r- x_s) + (x_i + x_s)(x_j - x_r); \end{aligned}$$

(2.5)

$$\begin{aligned} (x_i + x_s)(x_j + x_r)&\quad \longrightarrow \quad (x_i - x_j)(x_r- x_s) \nonumber \\&\qquad \qquad + (x_i + x_j)(x_r+ x_s) + (x_i - x_s)(x_j - x_r); \end{aligned}$$

(2.6)

$$\begin{aligned} (x_i - x_r)&\quad \longrightarrow \quad (x_i - x_j) + (x_j - x_r); \end{aligned}$$

(2.7)

$$\begin{aligned} (x_i + x_r)&\quad \longrightarrow \quad (x_i - x_j) + (x_j + x_r); \end{aligned}$$

(2.8)

$$\begin{aligned} (x_i + x_j)&\quad \longrightarrow \quad (x_i - x_j) + (x_j - x_r) + (x_j + x_r). \end{aligned}$$

(2.9)

The index j in relations 2.7 and 2.8 and the index r in relation 2.9 are assumed to be unused indices of $\alpha$, and they may be chosen arbitrarily subject to the constraint that $i< j < r$.

Proof

If $\alpha$ has a defect, then the normal form factorization of $\alpha$ must contain the factor(s) on the left-hand side of one of these nine identities, as follows. If $\alpha$ has a crossing, then one of the relations 2.1, 2.2, 2.3, or 2.4 is applicable. If $\alpha$ has a nested symmetric factor, then one of the relations 2.5 or 2.6 is applicable. If $\alpha$ has a nested unused index, then one of the relations 2.7 or 2.8 is applicable. Finally, if $\alpha$ has an obstructed symmetric factor, then relation 2.9 is applicable.

A routine verification shows that in each of the nine cases in the statement, the polynomial on the left-hand side is equal to the polynomial on the right-hand side. By inspection, all of the factors appearing are of the correct type to appear in a normal form factorization. It follows that making the substitutions indicated will express the normal form of the positive k-root $\alpha$ as a sum of other positive k-roots that are also in normal form.

It remains to show that if we have $\alpha = \sum _{p=1}^m \beta _p$ as above, where the $\beta _p$ are positive k-roots and $m > 1$, then we have $\beta _p < \alpha$ for all p. This follows from the definition of the total order on $V_{n,k}$, because we have

$$\begin{aligned} \alpha - \beta _p = \sum _{q: 1 \le q \le m, q \ne p} \beta _q. \end{aligned}$$

The right-hand side is positive because it is a nontrivial sum of positive elements of $V_{n,k}$, which means that $\alpha - \beta _p > 0$. By definition, this means that we have $\beta _p < \alpha$, as required. $\square$

Remark 2.8

It may be convenient to visualize the relations in Proposition 2.7 as (singular) skein relations, in which the factors of the form $(x_i - x_j)$ (respectively, $(x_i + x_j)$) are represented by an undecorated (respectively, decorated) arc from i to j. Figure 1 shows the pictorial version of relation 2.6, and Remark 3.13 gives some more details on the relationship between k-roots and diagram algebras.

Fig. 1

Relation 2.6 interpreted as a skein relation

Lemma 2.9

• (i) Every positive k-root in ${{\mathcal {C}}}_{n,k}$ can be written as a linear combination of elements of ${{\mathcal {B}}}_{n,k}$ with nonnegative integer coefficients.

• (ii) The set ${{\mathcal {B}}}_{n,k}$ is a spanning set for $V_{n,k}$ over ${{\mathbb {Q}}}$, and we have $|{{\mathcal {B}}}_{n,k}| \ge \left( {\begin{array}{c}n\\ k\end{array}}\right)$.

Proof

To prove (i), suppose that $\alpha$ is a positive k-root. If we have $\alpha \in {{\mathcal {B}}}_{n,k}$, then we are done, so suppose that $\alpha$ has a defect. We then apply Proposition 2.7 repeatedly to $\alpha$, applying the nine types of reduction in any order. This process must eventually terminate because the poset ${{\mathcal {C}}}_{n,k}$ is finite, and it will result in an expression for $\alpha$ as a positive integral linear combination of elements of ${{\mathcal {B}}}_{n,k}$.

The first assertion of (ii) follows from (i) and Lemma 2.5, and the second assertion holds because $V_{n,k}$ has dimension $\left( {\begin{array}{c}n\\ k\end{array}}\right) .$ $\square$

Remark 2.10

If $1 \le i < j \le n$ are integers, then the signed transposition $\overline{(i, j)}$ of the 2n symbols $\{\pm x_1, \pm x_2, \ldots , \pm x_n\}$ is the permutation that sends $\pm x_i$ to $\mp x_j$, and fixes $\pm x_r$ for $r \ne i, j$. For each k-root $\alpha$, there is a natural way to assign a transposition (i, j) to each antisymmetric factor $(x_i - x_j)$ and a signed transposition $\overline{(i, j)}$ to each symmetric factor $(x_i + x_j)$ to give a set $\{t_1, t_2, \ldots , t_k\}$ of distinct, mutually commuting signed and unsigned transpositions with the property that $t_i(\alpha ) = -\alpha$ for all i. Up to multiplying by nonzero scalars, the vector $\alpha \in V_{n,k}$ can be characterized as the unique common eigenvector of the $\{t_1, t_2, \ldots , t_k\}$ with common eigenvalue $-1$.

3 The Canonical Basis

The main result of this section is Theorem 3.10, which proves that the set ${{\mathcal {B}}}_{n,k}$ of k-roots without defects is a ${{\mathbb {Q}}}$-basis for $V_{n,k}$, and that ${{\mathcal {B}}}_{n,k}$ can naturally be parametrized by a certain set of lattice words. Recall that a lattice word is a sequence $a_1 a_2 \cdots a_n$ of positive integers with the property that for each positive integer i, each initial segment of the sequence contains at least as many occurrences of i as of $i+1$.

Definition 3.1

Let $\alpha \in {{\mathcal {B}}}_{n,k}$ be a positive k-root with no defects. We define the label, $\lambda (\alpha )$, of $\alpha$ to be the word of length n in the alphabet $\{1, 2\}$ with the property that $\lambda (\alpha )_j = 2$ if and only if the normal form of $\alpha$ has a factor of the form $(x_i - x_j)$ for some $i < j$. We define ${\Lambda _{n,k}}$ to be the set of lattice words of length n that have entries in the set $\{1, 2\}$, and at most k occurrences of 2.

Remark 3.2

The set ${\Lambda _{n,k}}$ is in canonical bijection with the set of all standard Young tableaux with n boxes having at most two rows and at most k boxes in the second row. The positions of the occurrences of 2 in the lattice word correspond to the labels of the boxes in the second row of the tableau.

Because each negative term $-x_j$ in the normal form is paired with a distinct term $x_i$ with $i < j$, the following result follows immediately.

Lemma 3.3

If $\alpha \in {{\mathcal {B}}}_{n,k}$ is a positive k-root with no defects, then we have $\lambda (\alpha ) \in {\Lambda _{n,k}}$. $\square$

Definition 3.4

Let $\alpha \in {{\mathcal {C}}}_{n,k}$ be a positive k-root. We define the height, $h(\alpha )$, of $\alpha$ to be the number of symmetric factors appearing in the normal form of $\alpha$. We define ${{\mathcal {B}}}_{n,k,h}$ to be the subset of ${{\mathcal {B}}}_{n,k}$ consisting of k-roots of height h.

Note that if $\alpha \in {{\mathcal {C}}}_{n,k}$ then we always have $0 \le h(\alpha ) \le k$.

Lemma 3.5

• (i) There is a function $f: {{\mathcal {B}}}_{n,k,h} \rightarrow {{\mathcal {B}}}_{n,k-h,0}$, where $f(\alpha )$ is defined to be the homogeneous polynomial of degree $k-h$ obtained by removing the $h = h(\alpha )$ symmetric factors from the normal form of $\alpha$.

• (ii) The label $\lambda (\alpha )$ has $k-h(\alpha )$ occurrences of 2, and satisfies $\lambda (\alpha ) = \lambda (f(\alpha ))$.

• (iii) The function f is injective.

Proof

It follows from the definition of normal form that removing h factors from the normal form of a positive k-root will give the normal form of a positive $(k-h)$-root. It remains to show that the resulting $(k-h)$-root has no defects.

Because $f(\alpha )$ has no symmetric factors by construction, it can have no defects of types (ii) or (iv) in Definition 2.6. It is also immediate that the removal of factors cannot create new crossings, which means that $f(\alpha )$ has no defects of type (i) in Definition 2.6. The only way $f(\alpha )$ can have a defect is if we are in the situation of Definition 2.6 (iii), and because $f(\alpha )$ has no symmetric factors, we must be in the more specific situation of relation 2.7 of Proposition 2.7.

We may now assume that $f(\alpha )$ has a factor of the form $(x_i - x_r)$ and an unused index j with $i< j < r$. Because $\alpha$ has no defects, the index j must be involved in a symmetric factor $(x_j + x_m)$ of $\alpha$, for some $m \ne i, j, r$. We cannot have $m < i$ or $m > r$ because the factors $(x_i - x_r)$ and $(x_j + x_m)$ of $\alpha$ would create a crossing, and we cannot have $i< m < r$ because the factors $(x_i - x_r)$ and $(x_j + x_m)$ of $\alpha$ would create a nested symmetric factor. This completes the proof of (i).

Part (ii) follows from (i) and Definition 3.1.

To prove (iii), we need to show that the h symmetric factors of $\alpha$ are uniquely determined by h and $f(\alpha )$. Because $\alpha$ has no obstructed symmetric factors, it must be the case that the largest 2h unused indices of $f(\alpha )$ are precisely the indices of the symmetric factors of $\alpha$. Let us denote these indices by $\{i_1, i_2, \ldots , i_{2\,h}\}$, where $i_1< i_2< \cdots < i_{2\,h}$. Because $\alpha$ has no crossings and no nested symmetric factors, the symmetric factors of $\alpha$ must be

$$\begin{aligned} (x_{i_1} + x_{i_2}), (x_{i_3} + x_{i_4}), \ldots , (x_{i_{2h-1}} + x_{i_{2h}}). \end{aligned}$$

This completes the proof of (iii). $\square$

Example 3.6

Consider the case $n = 12$, $k = 5$, $h = 2$. Let $\alpha$ be the element of ${{\mathcal {B}}}_{12,5,2}$ given by

$$\begin{aligned} \alpha = (x_2 - x_3)(x_5 + x_{10})(x_6 - x_9)(x_7 - x_8)(x_{11} + x_{12}), \end{aligned}$$

so that $f(\alpha )$ is the element of ${{\mathcal {B}}}_{12,3,0}$ given by

$$\begin{aligned} f(\alpha ) = (x_2 - x_3)(x_6 - x_9)(x_7 - x_8). \end{aligned}$$

Both $\alpha$ and $f(\alpha )$ have the label 112111122111, where the 2 s appear in positions 3, 8, and 9. The largest $2h (= 4)$ indices not appearing in $f(\alpha )$ are $\{5, 10, 11, 12\}$, from which it follows that the symmetric factors appearing in $\alpha$ are $(x_5 + x_{10})$ and $(x_{11} + x_{12})$.

Lemma 3.7

Let $\alpha \in {{\mathcal {B}}}_{n,k,0}$ be a k-root of height 0 that has no defects, and let $p(\alpha )$ be the normal form of $\alpha$. For each index j satisfying $\lambda (\alpha )_j = 2$, define g(j) to be the unique index for which $(x_{g(j)} - x_j)$ is a factor in $p(\alpha )$. Then $i = g(j)$ is the largest index $i < j$ such that both $\lambda (\alpha )_i = 1$ and $p(\alpha )$ contains no factor of the form $(x_i - x_m)$ for any $m < j$.

Remark 3.8

If one replaces the 1 s in $\lambda (\alpha )$ by open parentheses and the 2 s by close parentheses, then the map g in the statement locates the open parenthesis that matches a given close parenthesis. Lemma 3.3 shows that it is always possible to find a match.

Proof of Lemma 3.7

We know that there is at least one index $i < j$ such that both $\lambda (\alpha )_i = 1$ and $p(\alpha )$ contains no factor of the form $(x_i - x_m)$ for any $m < j$, because g(j) itself satisfies these conditions. Suppose for a contradiction that there exists such an i for which $g(j)< i < j$.

If i is an unused index in $\alpha$, then it is a nested unused index relative to the factor $(x_{g(j)} - x_j)$, which contradicts the assumption that $\alpha \in {{\mathcal {B}}}_{n,k,0} \subseteq {{\mathcal {B}}}_{n,k}$. Because $\lambda (\alpha )_i = 1$, the only other possibility is for $x_i$ to be involved in a factor of the form $(x_i - x_m)$ with $g(j)< i< j < m$. In this case, the pair $(x_{g(j)} - x_j)$, $(x_i - x_m)$ forms a crossing, which also contradicts the assumption $\alpha \in {{\mathcal {B}}}_{n,k,0}$, completing the proof. $\square$

Lemma 3.9

Maintain the above notation.

• (i) For each $0 \le h \le k$, the restriction of $\lambda$ to ${{\mathcal {B}}}_{n,k,h}$ is injective.

• (ii) The labelling function $\lambda : {{\mathcal {B}}}_{n,k}\rightarrow {\Lambda _{n,k}}$ is injective.

Proof

Lemma 3.7 shows that if $h = 0$ then we can use induction on j to reconstruct $\alpha$ from $\lambda (\alpha )$. This proves (i) in the case $h = 0$. The general result of (i) now follows by combining the result for $h = 0$ with Lemma 3.5.

For (ii), observe that the set ${{\mathcal {B}}}_{n,k}$ is the disjoint union of the sets ${{\mathcal {B}}}_{n,k,h}$ for $0 \le h \le k$. If $\alpha \in {{\mathcal {B}}}_{n,k,h}$, then the number of occurrences of 2 in $\lambda (\alpha )$ is $k - h$. It follows that the images of the sets ${{\mathcal {B}}}_{n,k,h}$ for $0 \le h \le k$ are pairwise disjoint, which completes the proof. $\square$

Theorem 3.10

Let $n \le 2$ and $0 \le k \le n/2$ be integers, and let ${{\mathcal {B}}}_{n,k}$ be the set of positive k-roots with no defects.

• (i) The labelling function $\lambda : {{\mathcal {B}}}_{n,k}\rightarrow {\Lambda _{n,k}}$ is a bijection.

• (ii) The set ${{\mathcal {B}}}_{n,k}$ is a basis for $V_{n,k}$ over ${{\mathbb {Q}}}$.

• (iii) Every positive k-root is ${{\mathcal {B}}}_{n,k}$-positive, with integer coefficients.

• (iv) The elements ${{\mathcal {B}}}_{n,k}$ are the only positive k-roots that cannot be written as positive linear combinations of other positive k-roots.

Proof

Let T(n, j) be the number of lattice words of length n in the alphabet $\{1, 2\}$ where there are precisely j occurrences of 2. It is known (see Sequence A008315 of [12]) that

$$\begin{aligned} |T(n, j)| = \left( {\begin{array}{c}n\\ j\end{array}}\right) - \left( {\begin{array}{c}n\\ j-1\end{array}}\right) , \end{aligned}$$

where we interpret $\left( {\begin{array}{c}n\\ -1\end{array}}\right)$ to be zero.

Lemma 3.5 now implies that we have $|{{\mathcal {B}}}_{n,k,h}| \le T(n, k-h)$, and summing over h gives

$$\begin{aligned} |{{\mathcal {B}}}_{n,k}|= & {} \sum _{h = 0}^k |{{\mathcal {B}}}_{n,k,h}| \le \sum _{h=0}^k T(n, k-h) = \sum _{h=0}^k T(n, h) \\= & {} \sum _{h=0}^k \left[ \left( {\begin{array}{c}n\\ h\end{array}}\right) - \left( {\begin{array}{c}n\\ h-1\end{array}}\right) \right] = \left( {\begin{array}{c}n\\ k\end{array}}\right) . \end{aligned}$$

Lemma 2.9 (ii) now implies that the inequality of the last paragraph is an equality, and the injective maps of Lemma 3.9 are bijective, proving (i).

Part (ii) follows because we have $|{{\mathcal {B}}}_{n,k}| = \left( {\begin{array}{c}n\\ k\end{array}}\right) = \dim (V_{n,k})$, and part (iii) follows by combining (ii) with Lemma 2.9 (i).

If $\alpha$ is a positive k-root that is not an element of ${{\mathcal {B}}}_{n,k}$, then $\alpha$ has a defect, and it can be written as a positive integral linear combination of other positive k-roots by Proposition 2.7. On the other hand, if $\alpha \in {{\mathcal {B}}}_{n,k}$ and $\alpha$ is a positive linear combination of other positive k-roots, it follows from (iii) that each of these positive k-roots must be a scalar multiple of $\alpha$. By Remark 2.2, each of the positive k-roots must equal $\alpha$, which is a contradiction and proves (iv). $\square$

From now on, we will call ${{\mathcal {B}}}_{n,k}$ the canonical basis of $V_{n,k}$.

Remark 3.11

Parts (iii) and (iv) of Theorem 3.10 are familiar in the context of root systems. They show that the canonical basis may be characterized purely in terms of the vector space ordering on $V_{n,k}$, without relying on the concept of defects at all.

Example 3.12

An example of the basis ${{\mathcal {B}}}_{n,k}$ that does not come from a root system is the case $n=4$, $k=2$. There are 12 positive and 12 negative 2-roots, and the elements of ${{\mathcal {B}}}_{4,2}$ and their labels are as follows.

Canonical basis element	Label
$(x_1 + x_2)(x_3 + x_4)$	1111
$(x_1 + x_2)(x_3 - x_4)$	1112
$(x_1 + x_4)(x_2 - x_3)$	1121
$(x_1 - x_2)(x_3 + x_4)$	1211
$(x_1 - x_4)(x_2 - x_3)$	1122
$(x_1 - x_2)(x_3 - x_4)$	1212

The six positive roots that are not basis elements come from the left hand sides of relations 2.1–2.6 of Proposition 2.7, taking $i=1$, $j=2$, $r=3$ and $s=4$.

Remark 3.13

There are other constructions of the basis ${{\mathcal {B}}}_{n,k}$. One of these comes from the Kazhdan–Lusztig basis $\{C_w\}$ from [11], specifically, the basis of the module arising from the left cell containing the permutation $(1, 2)(3, 4)\cdots (k-1, k)$ in type $D_n$ with $q=1$. One can also construct ${{\mathcal {B}}}_{n,k}$ from a basis for the generalized Temperley–Lieb algebra of type $D_n$, after specializing q to 1 and twisting by sign. The latter basis may be defined in terms of monomials as in [4, §6.2], or in terms of diagrams as in [7], and the definition of “defect” in this paper is closely related to the diagrammatic rules in [7]. When $k=n/2$, it is necessary for all these constructions to take the union of two cells: the one just described, and its image under the automorphism that sends $x_n$ to $-x_n$ and fixes $x_i$ for $i < n$. One can find module isomorphisms between these various constructions using the characterization of Remark 2.10.

The k-root approach has a significant advantage over these other constructions, which is that Proposition 2.7 makes it easy (a) to work out the effect of applying an arbitrary (signed) permutation w to a basis element $\alpha$ and then (b) to express the result as a linear combination of basis elements.

4 Main Results

In Sect. 4, we explore some applications of k-roots and the basis ${{\mathcal {B}}}_{n,k}$ in representation theory. We first show how ${{\mathcal {B}}}_{n,k}$ naturally gives rise to a composition series of $V_{n,k}$ as an $S_n$-module. We refer the reader to Fulton and Harris [6] for background information on the character theory of the symmetric groups.

Definition 4.1

Let $V_{n,k,t}$ be the ${{\mathbb {Q}}}$-linear span of all k-roots in ${{\mathcal {B}}}_{n,k}$ that have height at most t; that is,

$$\begin{aligned} V_{n,k,t}:= {\textrm{Span}}\left( \bigsqcup _{h=0}^t {{\mathcal {B}}}_{n,k,h} \right) . \end{aligned}$$

Proposition 4.2

• (i) The subspaces $V_{n,k,t}$ of Definition 4.1 are ${{\mathbb {Q}}}S_n$-submodules of $V_{n,k}$.

• (ii) The chain

  $$\begin{aligned} V_{n,k,-1}:= 0< V_{n,k,0}< V_{n,k,1}< \cdots < V_{n,k,k} = V_{n,k}\end{aligned}$$

  is a composition series of $V_{n,k}$ as a ${{\mathbb {Q}}}S_n$-module.

Proof

Let $\alpha \in {{\mathcal {B}}}_{n,k}$ be a canonical basis element of height h, and let $w \in S_n$ be a permutation. One of $\pm w(\alpha )$ is a positive k-root of height h. A routine case-by-case check shows that the reduction rules in Proposition 2.7 all express a positive k-root as a linear combination of positive k-roots of the same, or lower heights. It follows that $w(\alpha )$ is a linear combination of canonical basis elements of height at most h, and this proves part (i).

Note that for each j satisfying $0 \le j \le k$, there exists an element of ${\Lambda _{n,k}}$ with precisely j occurrences of 2; for example, the word $1^{n-j}2^j$. Each such word corresponds via Theorem 3.10 (i) to an element of ${{\mathcal {B}}}_{n,k}$ of height $k-j$, so there exist basis elements of all possible heights h in the range $0 \le h \le k$. It follows that each step in the chain in (ii) corresponds to a strict submodule, and thus that the series has $k+1$ nontrivial quotients $V_{n,k,h}/V_{n,k,h-1}$.

Lemma 2.5 (ii) implies that $V_{n,k}$ is the direct sum of $k+1$ irreducible $S_n$-submodules. Since this is the same as the number of nontrivial quotients in the series of (ii), it follows both that $V_{n,k}$ is a direct sum of $k+1$ irreducible submodules over ${{\mathbb {Q}}}$, and that the series in (ii) is a composition series. $\square$

The next result is useful for determining when a positive k-root stays positive after a permutation acts on it.

Lemma 4.3

Let $\alpha \in {{\mathcal {C}}}_{n,k}$ be a positive k-root, and let $w \in S_n$ be a permutation. If $w(\alpha )$ is negative, then the normal form of $\alpha$ must contain a factor $(x_i - x_j)$ for which $w(i) > w(j)$.

Proof

If $w(\alpha )$ is negative, but there is no factor in $\alpha$ of the form $(x_i - x_j)$ satisfying $w(i) > w(j)$, then each factor in the normal form of $\alpha$ is sent by w to another factor in normal form. It follows from Lemma 2.3 that $w(\alpha )$ is positive, which is a contradiction. $\square$

Although we know that the irreducible components of $V_{n,k}$ have characters $\chi ^{(n-i, i)}$, we will need to be able to match these to the composition factors of the series in Proposition 4.2. The next result helps with this.

Lemma 4.4

Let A be a subset of $\{1, 2, \ldots , n\}$ of cardinality a, and let $S_A$ be the full symmetric group on A considered as a subgroup of $S_n$ of order a!. Define

$$\begin{aligned} x_A:= \sum _{w \in S_A} w. \end{aligned}$$

• (i) If $V_i$ is an irreducible ${{\mathbb {C}}}S_n$-module with character $\chi ^{(n-i, i)}$ for some $i \le n/2$, then we have $x_A.V_i \ne 0$ if and only if $a \le n-i$.

• (ii) If $1 \le j \le k$, then the k-root

  $$\begin{aligned} \beta _j = \prod _{i = 1}^j (x_i - x_{k+i}) \prod _{i=j+1}^k (x_i + x_{k+i}) \end{aligned}$$

  lies in $V_{n,k,k-j}$.

• (iii) If $V_{n,k,t}$ is as in Proposition 4.2, then we have $x_A. V_{n,k,t} \ne 0$ if and only if $a \le n-k+t$.

• (iv) The composition factor $V_{n,k,t}/V_{n,k,t-1}$ has character $\chi ^{(n-k+t,k-t)}$, and $V_{n,k,t}$ has character

  $$\begin{aligned} \sum _{i = k-t}^k \chi ^{(n-i, i)}. \end{aligned}$$

Proof

If we define $B = \{1, 2, \ldots , a\}$, then we have $x_A = g x_B g^{-1}$ for some $g \in S_n$, which implies that $x_A$ and $x_B$ annihilate the same modules. It is therefore enough to consider the case where $A = \{1, 2, \ldots , a\}$.

The condition that $x_A.V_i \ne 0$ is equivalent to the condition that $V_i$, when regarded as an $S_A$-module, contains a copy of the trivial representation, that is, that

$$\begin{aligned} \langle 1, V_i \downarrow ^{S_n}_{S_A} \rangle \ne 0 \end{aligned}$$

in the usual inner product on characters. By Frobenius reciprocity, this is equivalent to

$$\begin{aligned} \langle 1 \uparrow _{S_A}^{S_n}, V_i \rangle \ne 0. \end{aligned}$$

Since $S_A$ is a Young subgroup of $S_n$ of type $S_a \times S_1 \times \cdots \times S_1$, where there are $n-a$ copies of $S_1$, it follows that the character of $1 \uparrow _{S_A}^{S_n}$ corresponds to the product of Schur functions

$$\begin{aligned} s_{(a)} s_{(1)} \cdots s_{(1)}, \end{aligned}$$

where again there are $n-a$ copies of $s_{(1)}$. This corresponds to adding $n-a$ boxes, one at a time, to the partition (a). This will result in at least one copy of $s_{(n-i, i)}$ if and only if we have $n - a \ge i$; otherwise, there are not enough single boxes to fill the second row. This proves (i).

Let the k-root $\beta _j$ be as in the statement of (ii). Observe that $\beta _j$ is a k-root of height $k-j$, and it is in the same $S_n$-orbit as any element of ${{\mathcal {B}}}_{n,k,k-j}$, for example, the element of ${{\mathcal {B}}}_{n,k}$ whose label is $1^{n-j}2^j$. Since $\beta _j$ is in the same $S_n$-orbit as an element of $V_{n,k,k-j}$, it follows that $\beta _j \in V_{n,k,k-j}$, proving (ii).

To prove (iii), note that any basis element $\alpha$ of $V_{n,k,t}$ has at least $k-t$ antisymmetric factors. If we have $a > n-(k-t)$, then it is inevitable that at least one of the antisymmetric factors of $\alpha$ is of the form $(x_i - x_j)$ where both of i and j lie in A. It follows that $\alpha$ is annihilated by $1 + w$, where w is the transposition (i, j). By summing over a set of left coset representatives of $\langle w \rangle$ in $S_A$, we can factorize $x_A$ as $x'(1+w)$, from which it follows that $x_A$ annihilates $\alpha$. Since $\alpha$ was arbitrary, we deduce that $x_A$ annihilates $V_{n,k,t}$ if $a > n-k+t$.

It remains to show that if $a \le n-k+t$, then $x_A$ does not annihilate $V_{n,k,t}$. Let $\beta = \beta _{k-t}$ be the k-root defined in (ii). By construction, no antisymmetric factor of $\beta$ has both endpoints in the set A. Lemma 4.3 now implies that any $w \in S_A$ has the property that $w(\beta )$ is a positive k-root. It follows that $x_A(\beta )$ is a nontrivial sum of positive k-roots, and Theorem 3.10 (iii) shows that $x_A(\beta )$ is a nontrivial sum of canonical basis elements. In particular, $x_A$ does not annihilate $\beta$, and therefore $x_A$ does not annihilate $V_{n,k,t}$, proving (iii).

If we set $A = \{1, 2, \ldots , n-k+t\}$ and $B = \{1, 2, \ldots , n-k+t+1\}$, then (iii) implies that $x_A$ annihilates $V_{n,k,t-1}$, but not $V_{n,k,t}$, and that $x_B$ annihilates $V_{n,k,t}$. The character of $V_{n,k,t}/V_{n,k,t-1}$ is therefore the character of the form $\chi ^{(n-i, i)}$ that is annihilated by $x_B$ but not by $x_A$. By (i), we find the solution is to take $i = k-t$, which proves the first assertion of (iv). The second assertion of (iv) follows by summing over all the composition factors of $V_{n,k,t}$.

$\square$

Recall (for example, from sections 2 and 3 of [3] or the proof of [2, Corollary 4.6.4 (ii)]) that the spherical functions ${\Phi (n,k,j)}$ are characterized by the following properties:

• (i) ${\Phi (n,k,j)}$ lies in the irreducible summand of L(X) with character $\chi ^{(n-j, j)}$;

• (ii) ${\Phi (n,k,j)}$ is fixed pointwise by the subgroup $K = S_k \times S_{n-k}$;

• (iii) ${\Phi (n,k,j)}$ takes the value 1 at the identity coset; in other words, the coefficient of $x_1 x_2 \cdots x_k$ in ${\Phi (n,k,j)}$ is 1.

We are now ready to give the construction of these spherical functions in terms of k-roots.

Theorem 4.5

Let $n \ge 2$ and $0 \le k \le n/2$, and $0 \le j \le k$ be integers. Let $A = \{1, 2, \ldots , k\}$ and let $B = \{j+1, j+2, \ldots , n\}$.

• (i) As a homogeneous polynomial in $x_1, x_2, \ldots , x_n$, the jth spherical function ${\Phi (n,k,j)}$ of the Gelfand pair $(S_n, S_k \times S_{n-k})$ is given by

  $$\begin{aligned} \frac{(n-2k)!}{k!(n-k)!2^{k-j}(k-j)!} \left( \sum _{v \in S_A} v \right) \left( \sum _{w \in S_B} w \right) \cdot \beta _j, \end{aligned}$$

  where $\beta _j$ is the k-root defined by

  $$\begin{aligned} \beta _j = \prod _{i = 1}^j (x_i - x_{k+i}) \prod _{i=j+1}^k (x_i + x_{k+i}). \end{aligned}$$

• (ii) The function ${\Phi (n,k,j)}$ is ${{\mathcal {B}}}_{n,k}$-positive, and its coefficients are nonnegative integer multiples of 1/N, where N is the integer

  $$\begin{aligned} (k)_j(n-k)_k = \frac{k!(n-k)!}{(k-j)!(n-2k)!}. \end{aligned}$$

Proof

Let ${\Psi (n,k,j)}$ be the polynomial given in the formula; we will show that ${\Psi (n,k,j)}$ is equal to the jth spherical function, ${\Phi (n,k,j)}$. Any permutation in $S_A$ will fix ${\Psi (n,k,j)}$, because we have already symmetrized over $S_A$. Any permutation of $\{k+1, k+2, \ldots , n\}$ will commute with each element of $S_A$, and be equal to an element of $S_B$, so these too will fix ${\Psi (n,k,j)}$. It follows that ${\Psi (n,k,j)}$ is fixed by the subgroup $K = S_k \times S_{n-k}$.

We next prove that ${\Psi (n,k,j)}$ lies in the unique irreducible submodule $V_j$ of $V_{n,k}$ with character $\chi ^{(n-j, j)}$. The k-root $\beta _j$ lies in $V_{n,k,k-j}$ by Lemma 4.4 (ii), and $V_{n,k,k-j}$ has character $\sum _{i = j}^k \chi ^{(n-i, i)}$ by Lemma 4.4 (iv). It is enough to show that $\sum _{w \in S_B} w \cdot \beta _j$ lies in $V_j$. Since B has cardinality $n-j$, it follows from Lemma 4.4 (i) that $\sum _{w \in S_B} w$ will annihilate every submodule whose character is in the set

$$\begin{aligned} \{\chi ^{(n-i, i)}: j \le i \le k\} \end{aligned}$$

except the one with character $\chi ^{(n-j, j)}$. It follows that $\sum _{w \in S_B} w \cdot \beta _j$ lies in $V_j$, as required.

To complete the proof of (i), it remains to show that $x_1 x_2 \ldots x_k$ appears in ${\Psi (n,k,j)}$ with coefficient 1. Observe that the polynomial $\prod _{i=j+1}^k (x_i + x_{k+i})$ is stabilized by a subgroup $U \le S_B$, generated by transpositions $(i, k+i)$ that fix each factor, together with permutations of the $k-j$ factors. It follows that U has order $|U|=2^{k-j}(k-j)!$. By summing over a set of left coset representatives, $X_B$, for the left cosets $S_B/U$ of U in $S_B$, we can obtain an expression for ${\Psi (n,k,j)}$ that is equivalent to the one in the statement but has fewer terms, as follows:

$$\begin{aligned} {\Psi (n,k,j)}= \frac{(n-2k)!}{k!(n-k)!} \left( \sum _{v \in S_A} v \right) \left( \sum _{w \in X_B} w \right) \cdot \beta _j. \end{aligned}$$

Note that every antisymmetric factor of $\beta _j$ contains precisely one index from the set $\{1, 2, \ldots , j\}$, and these indices are fixed pointwise by every element in $S_B$. Lemma 4.3 shows that the k-roots $\{w \cdot \beta _j: w \in X_B\}$ are all positive, and because we are summing over cosets of the stabilizer, each element of $X_B$ gives a different positive k-root $w \cdot \beta _j$. It is convenient to separate the k-roots $w \cdot \beta _j$ into three mutually exclusive types.

• Type 1: positive k-roots containing at least one antisymmetric factor $(x_p - x_q)$ where $1 \le p < q \le k$;

• Type 2: positive k-roots that are not of type 1, but that contain least one symmetric factor $(x_p + x_q)$ where either $1 \le p < q \le k$ or $k+1 \le p < q \le n$;

• Type 3: positive k-roots where each factor contains precisely one $x_p$ where $1 \le p \le k$.

The k-roots of type 1 are annihilated by elements of the form $1 + w$ where w is the transposition (p, q). As in the proof of Lemma 4.4 (iii), it follows that k-roots of type 1 are annihilated by $\sum _{v \in S_A} v$, and thus that they make no net contribution to the sum. These terms may be ignored from now on.

The k-roots of types 2 and 3 contain no antisymmetric factors $(x_p - x_q)$ with $1 \le p < q \le k$. If $\alpha$ is a k-root of type 2 or 3 and $v \in S_A$, it follows by Lemma 4.3 that $v \cdot \alpha$ is also a positive k-root.

If $\alpha$ has type 2, then the k-root $v \cdot \alpha$ will have a factor $(x_{v(p)} + x_{v(q)})$ with two indices in the range $1 \le v(p), v(q) \le k$ or in the range $k +1 \le v(p), v(q) \le n$, and this means that $x_1 x_2 \cdots x_k$ appears in $v \cdot \alpha$ with a coefficient zero.

A k-root $\alpha$ is of type 3 if and only if it has the form

$$\begin{aligned} \alpha = \prod _{i = 1}^j (x_i - x_{\iota (i)}) \prod _{i=j+1}^k (x_i + x_{\iota (i)}) \end{aligned}$$

for some injective function $\iota : \{1, 2, \ldots , k\} \rightarrow \{k+1, k+2, \ldots , n\}$. The monomial $x_1 x_2 \cdots x_k$ appears in each such k-root with coefficient 1, and the number of k-roots of type 3 is the same as the number of functions $\iota$, which is $(n-k)_k = (n-k)!/(n-2k)!$. The action of a permutation $v \in S_A$ leaves invariant the coefficient of $x_1 x_2 \cdots x_k$, which implies that the coefficient of $x_1 x_2 \cdots x_k$ in $\left( \sum _{v \in S_A} v \right) \left( \sum _{w \in X_B} w \right) \cdot \beta _j$ is $k!(n-k)!/(n-2k)!$, proving (i).

The above argument has also shown that $\beta '_j:= \left( \sum _{v \in S_A} v \right) \left( \sum _{w \in X_B} w \right) \cdot \beta _j$ is a sum of positive k-roots: the terms of type 1 all cancel, and the terms of types 2 and 3 lead to sums of positive k-roots. Theorem 3.10 (iii) implies that $\beta '_j$ is a linear combination of canonical basis elements with nonnegative integer coefficients. Now let $C = A \cap B$, so that $|C|=k-j$, and let $X_A$ be a set of left coset representatives of $S_C$ in $S_A$. The left $S_B$-invariance of $\sum _{w \in X_B} w \cdot \beta _j$ then implies that

$$\begin{aligned} \beta '_j = \left( \sum _{v \in X_A} v \right) \left( \sum _{u \in S_C} u \left( \sum _{w \in X_B} w.\beta _j \right) \right) = (k-j)! \left( \sum _{v \in X_A} v \right) \left( \sum _{w \in X_B} w \right) \cdot \beta _j. \end{aligned}$$

It follows that the coefficients of the canonical basis elements in $\beta '_j$ are all integer multiples of $(k-j)!$, and dividing by the factor of $k!(n-k)!/(n-2k)!$ from the previous paragraph then proves (ii). $\square$

Remark 4.6

The bound on the denominator given in Theorem 4.5 (ii) is sharp in some cases, such as the case $n=4$ and $k=j=1$ in which the denominator in the theorem is the best possible denominator of 1/3.

Finally, we consider the problem of expressing elements of ${{\mathcal {M}}}_{n,k}$ as linear combinations of the canonical basis ${{\mathcal {B}}}_{n,k}$. The basis ${{\mathcal {B}}}_{n,k}$ only has one element that is a positive linear combination of the natural basis of monomials, namely the basis element whose label is $1^n$. However, it often happens that a square-free monomial can be written as a positive linear combination of the ${{\mathcal {B}}}_{n,k}$. The following definition is helpful for understanding this.

Definition 4.7

Let $x_I = x_{i_1} x_{i_2} \cdots x_{i_k}$ be a squarefree monomial of degree k in the indeterminates $x_1, \ldots x_n$. Define the label, $\mu (x_I)$ of $x_I$ to be the sequence of length n in the alphabet $\{1, 2\}$ with the property that $\mu (x_I)_j = 2$ if and only if $x_j$ appears in $x_I$. We say that $\mu (x_I)$ is a reverse lattice word if every terminal segment of $\mu (x_I)$ contains at least as many 1 s as 2 s.

Theorem 4.8

Let $n \ge 2$ and $0 \le k \le n/2$ be integers, and let $x_I = x_{i_1} x_{i_2} \cdots x_{i_k}$ be a squarefree monomial of degree k in the indeterminates $x_1, \ldots x_n$. If the label $\mu (x_I)$ is a reverse lattice word, then $x_I$ is ${{\mathcal {B}}}_{n,k}$-positive, with coefficients that are nonnegative integer multiples of $1/2^k$.

Proof

Suppose that $x_I$ satisfies the hypotheses in the statement. It is enough to prove that $2^k x_I$ is ${{\mathcal {B}}}_{n,k}$-positive with integer coefficients.

The hypothesis that $\mu (x_I)$ is a reverse lattice word means that there is an injective function $f: \{i_1, i_2, \ldots , i_k\} \rightarrow \{1, 2, \ldots , n\}$ with the properties that for all $1 \le r \le k$, both (a) $f(i_r) > i_r$ and (b) $\mu (x_I)$ has a 1 at position $f(i_r)$. By making the substitutions

$$\begin{aligned} 2x_{i_r} \rightarrow (x_{i_r} - x_{f(i_r)}) + (x_{i_r} + x_{f(i_r)}), \end{aligned}$$

we can express $2^k x_I = (2x_{i_1})(2x_{i_2}) \cdots (2x_{i_k})$ as an sum of $2^k$ positive k-roots in normal form. The result now follows from Theorem 3.10 (iii). $\square$

Remark 4.9

The number of reverse lattice words with k occurrences of 2 is $\left( {\begin{array}{c}n\\ k\end{array}}\right) - \left( {\begin{array}{c}n\\ k-1\end{array}}\right)$. It therefore follows that all but at most $\left( {\begin{array}{c}n\\ k-1\end{array}}\right)$ elements of the monomial basis are ${{\mathcal {B}}}_{n,k}$-positive. If k is small compared to n, then the hypotheses of Theorem 4.8 will usually be satisfied, but if k is close to n/2, the hypotheses will rarely be satisfied. The monomial $x_1 x_2 \cdots x_k$, whose label is $2^k 1^{n-k}$, will always satisfy the hypotheses.

It would be interesting to know whether the necessary condition in the theorem is also sufficient. This is the case when $k=1$, where $x_n$ is the only monomial that is not ${{\mathcal {B}}}_{n,1}$-positive.

5 Concluding Remarks

5.1 Other Gelfand Pairs

A natural question is whether the results of this paper have analogues for other Gelfand pairs. This happens, for example, in the case of the Gelfand pair $(S_{2n}, S_n \wr ({{\mathbb {Z}}}/2{{\mathbb {Z}}}))$, which corresponds to the action of $S_{2n}$ on the size n subsets of $\{1, 2, \ldots , 2n\}$, where each subset is identified with its complement. In this case, the coset space of the Gelfand pair has a canonical basis induced by the elements of ${{\mathcal {B}}}_{2n,n}$ that have an even number of asymmetric factors.

An example of a Gelfand pair that has a simpler treatment than the one in this paper is $(({{\mathbb {Z}}}/2{{\mathbb {Z}}}) \wr S_n, S_n)$. In this case, the $2^n$ cosets correspond to the weights of the spin representation of a simple Lie algebra of type $B_n$ [8, Proposition 6.4.5]. There is a well-known action of the Weyl group $W(B_n) \cong ({{\mathbb {Z}}}/2{{\mathbb {Z}}}) \wr S_n$ on 2n symbols $\{ 1, {\overline{1}}, 2, {\overline{2}}, \ldots , n, {\overline{n}}\}$ [8, Example 1.4.5]. This induces an action by signed permutations on the span of the $2^n$ linearly independent polynomials in the 2n commuting indeterminates $x_1, x_{{\overline{1}}}, \ldots , x_n, x_{{\overline{n}}}$ of the form

$$\begin{aligned} (x_1 \pm x_{{\overline{1}}})(x_2 \pm x_{{\overline{2}}}) \cdots (x_n \pm x_{{\overline{n}}}), \end{aligned}$$

where the signs are chosen independently. This is a basis for the permutation module on the cosets that is compatible with the direct sum decomposition into $W(B_n)$-irreducibles, and the corresponding spherical functions are given by taking the average of each $S_n$-orbit of basis elements.

The results of this paper can also be thought of in terms of averaging operators. It follows from Theorem 4.5 that when the spherical functions of $(S_n, S_k \times S_{n-k})$ are written as linear combinations of canonical basis elements, the denominators of the coefficients divide the order of K, namely $k!(n-k)!$. Using this, one can replace the expression in Theorem 4.5 (i) by an averaging operator over K acting on a sum with far fewer terms. This suggests that there may be continuous versions of these results in which the averaging operator is replaced by a suitable integral.

5.2 Sign-Coherence

Because the set ${{\mathcal {C}}}_{n,k}$ of k-roots is permuted by the action of any permutation w, it follows from Theorem 3.10 (iii) that the matrix $\rho (w)$ representing w with respect to ${{\mathcal {B}}}_{n,k}$ is an integer-valued column sign-coherent matrix. The property of column sign-coherence comes from the theory of cluster algebras ([1, Definition 2.2 (i)], [5, Definition 6.12], [9, §5]), and means that any two nonzero entries in the same column of $\rho (w)$ have the same sign. Each simple $S_n$-module $V_{n,k,t}/V_{n,k,t-1}$ inherits a basis from ${{\mathcal {B}}}_{n,k}$ that also has the sign-coherence property. This sign-coherence property is remarkable because it fails easily for irreducible $S_n$-modules corresponding to partitions with more than two rows; for example, the irreducible module for $S_4$ with character $\chi ^{(2,1,1)}$ contains a counterexample. The monomial basis for $V_{n,k}$ and the basis mentioned in Sect. 5.1 both have the sign-coherence property, but in the trivial sense that the matrices representing group elements have only one nonzero entry per column.

5.3 Differential Operators

Some of the results of this paper say something about the differential operators $d: V_{n,k} \rightarrow V_{n,k-1}$ given by $d = \sum _{i=1}^n \partial /\partial x_i$. It follows from the definitions that d sends positive k-roots to linear combinations of positive k-roots with positive even integer coefficients. Theorem 3.10 (iii) then implies that the entries of the matrix of d relative to ${{\mathcal {B}}}_{n,k}$ and ${{\mathcal {B}}}_{n,k-1}$ are positive even integers. The submodules $V_{n,k,t}$ of Definition 4.1 can be simply characterized as the kernels of the composite operators $d^{t+1}$, as in [2, Theorem 6.1.6 (v)].

5.4 Categorification

The appearance of ${{\mathcal {B}}}_{n,k}$-positivity in various contexts in this paper raises the question of whether the positive integers and rational numbers that arise have combinatorial interpretations. A related question is whether k-roots can be categorified, and the connection with Kazhdan–Lusztig bases mentioned in Remark 3.13 is an additional hint that this may be possible. It seems likely that the reduction rules in Proposition 2.7 would play an important role in any such categorification.