Hyperinvariant subspaces for sets of polynomially compact operators

Abstract

We prove the existence of a non-trivial hyperinvariant subspace for several sets of polynomially compact operators. The main results of the paper are: (i) a non-trivial norm closed algebra $\mathcal {A}\subseteq \mathcal {B}(\mathscr {X})$ which consists of polynomially compact quasinilpotent operators has a non-trivial hyperinvariant subspace; (ii) if there exists a non-zero compact operator in the norm closure of the algebra generated by an operator band $\mathcal {S}$, then $\mathcal {S}$ has a non-trivial hyperinvariant subspace.

1 Introduction

Let $\mathscr {X}$ be a complex Banach space. Denote by $\mathcal {B}(\mathscr {X})$ the algebra of all bounded linear operators on $\mathscr {X}$. A closed subspace $\mathscr {M}\subseteq \mathscr {X}$ is said to be invariant for an operator $T\in \mathcal {B}(\mathscr {X})$ if $T\mathscr {M}\subseteq \mathscr {M}$. Let $\mathcal {S}\subseteq \mathcal {B}(\mathscr {X})$ be a non-empty set of operators. Then, $\mathscr {M}$ is an invariant subspace of $\mathcal {S}$ if it is invariant for every operator in $\mathcal {S}$. If $\mathscr {M}$ is invariant for every operator in $\mathcal {S}$ and for every operator in the commutant $\mathcal {S}'=\{ T\in \mathcal {B}(\mathscr {X});\; TS=ST\,\text {for every}\, S\in \mathcal {S}\}$, then it is a hyperinvariant subspace of $\mathcal {S}$. Of course, the trivial subspaces $\{ 0\}$ and $\mathscr {X}$ are (hyper)invariant for any set of operators. We are interested in the existence of non-trivial invariant and hyperinvariant subspaces. The problem of existence of invariant and hyperinvariant subspaces for a given operator or a non-empty set of operators is an extensively studied topic in operator theory. The problem is solved in the finite-dimensional setting by Burnside’s theorem (see [14, Theorem 1.2.2]). In the context of infinite-dimensional Banach spaces, the problem is open for reflexive Banach spaces, in particular, for the infinite-dimensional separable Hilbert space. However, there are some Banach spaces for which we know either that every operator has a non-trivial invariant subspace or that there exist operators without it. For instance, Argyros and Haydon [1] have proved the existence of an infinite-dimensional Banach spaces $\mathscr {X}$ such that every operator in $\mathcal {B}(\mathscr {X})$ is of the form $\lambda I+K$, where I is the identity operator and K is compact. It follows, by the celebrated von Neumann-Aronszajn-Smith theorem [2] and Lomonosov’s theorem [12], that any operator in $\mathcal {B}(\mathscr {X})$ has a non-trivial invariant subspace and any non-scalar operator in $\mathcal {B}(\mathscr {X})$ has a non-trivial hyperinvariant subspace. On the other hand, several examples of Banach spaces (including $\ell _1$) with operators without a non-trivial invariant subspace are known (see [3, 15] for the first examples and [6] for a general approach to Read’s type constructions of operators without non-trivial invariant closed subspaces).

With the von Neumann–Aronszajn–Smith theorem and Lomonosov’s theorem in mind, it is not a surprise that suitable compactness conditions imply existence of non-trivial invariant and hyperinvariant subspaces for different classes of operators and sets of operators. For instance, Shulman [16, Theorem 2] proved that an algebra of operators whose radical contains a non-zero compact operator has a non-trivial hyperinvariant subspace. Turovskii [17, Corollary 5] extended this result to semigroups of quasinilpotent operators. Another type of results are those related to triangularizability of a set of operators. Recall that a non-empty set $\mathcal {S}\subseteq \mathcal {B}(\mathscr {X})$ is triangularizable if there exists a chain $\mathscr {C}$ which is maximal as a chain of subspaces of $\mathscr {X}$ and every subspace in $\mathscr {C}$ is invariant for all operators in $\mathcal {S}$. Every commutative set of compact operators is triangularizable (see [14, Theorem 7.2.1]). Konvalinka [10, Corollary 2.6] has extended this result by showing that a commuting family of polynomially compact operators is triangularizable. Another result in this direction, obtained by the second author [9], says that for a norm closed subalgebra $\mathcal {A}\subseteq \mathcal {B}(\mathscr {X})$ of power compact operators the following assertions are equivalent: (a) $\mathcal {A}$ is triangularizable; (b) the Jacobson radical $\mathcal {R}(\mathcal {A})$ consists precisely of quasinilpotent operators in $\mathcal {A}$; (c) the quotient algebra $\mathcal {A}/\mathcal {R}(\mathcal {A})$ is commutative.

The aim of this paper is to consider the problem of existence of a non-trivial hyperinvariant subspace for sets of polynomially compact operators. For instance, we prove (Theorem 3.3) that a non-trivial norm closed algebra $\mathcal {A}\subseteq \mathcal {B}(\mathscr {X})$ which consists of polynomially compact quasinilpotent operators has a non-trivial hyperinvariant subspace. Another result (Theorem 4.3) which we mention here is related to operator bands, that is, to semigroups of idempotent operators. It says that an operator band $\mathcal {S}$ has a non-trivial hyperinvariant subspace if there exists a non-zero compact operator in the norm closure of the algebra generated by $\mathcal {S}$.

2 Preliminaries

2.1 Notation

Let $\mathscr {X}$ be a non-trivial complex Banach space. Since the results proved in this paper are either trivial or well known when $\mathscr {X}$ is finite-dimensional, we always assume that $\dim (\mathscr {X})=\infty$. Let $\mathcal {B}(\mathscr {X})$ denote the Banach algebra of all bounded linear operators on $\mathscr {X}$ and let $\mathcal {K}(\mathscr {X})\subseteq \mathcal {B}(\mathscr {X})$ be the ideal of compact operators. The identity operator is denoted by I and an operator is said to be scalar if it is a scalar multiple of I. The norm closure of a set $\mathcal {S}\subseteq \mathcal {B}(\mathscr {X})$ is denoted by $\overline{\mathcal {S}}$.

For two operators $S_1, S_2\in \mathcal {B}(\mathscr {X})$, we denote their commutator $S_1S_2-S_2S_1$ by $[S_1,S_2]$. A non-empty set $\mathcal {S}\subseteq \mathcal {B}(\mathscr {X})$ is commutative if any two operators from $\mathcal {S}$ commute, that is, $[S_1,S_2]=0$ for all $S_1,S_2\in \mathcal {S}$. Similarly, $\mathcal {S}$ is essentially commutative if $[S_1,S_2]\in \mathcal {K}(\mathscr {X})$. For an arbitrary non-empty subset $\mathcal {S}\subseteq \mathcal {B}(\mathscr {X})$ the commutant of $\mathcal {S}$ is $\mathcal {S}'=\{ T\in \mathcal {B}(\mathscr {X});\; [T,S]=0,\, \forall \, S\in \mathcal {S}\}$. It is clear that $\mathcal {S}'$ is a closed subalgebra of $\mathcal {B}(\mathscr {X})$. If $\mathcal {S}$ is commutative, then $\mathcal {S}\subseteq \mathcal {S}'$.

2.2 Invariant subspaces

A non-empty subset $\mathscr {M}\subseteq \mathscr {X}$ is a subspace if it is a closed linear manifold. It is said that a subspace $\mathscr {M}$ of $\mathscr {X}$ is invariant for the operator T if $T\mathscr {M}\subseteq \mathscr {M}$. An invariant subspace $\mathscr {M}$ is non-trivial if $\{ 0\}\ne \mathcal {M}\ne \mathscr {X}$. A non-empty set $\mathcal {S}\subseteq \mathcal {B}(\mathscr {X})$ is reducible if there exists a non-trivial subspace $\mathscr {M}\subseteq \mathscr {X}$ which is invariant for every $T\in \mathcal {S}$. If there exists a chain $\mathscr {C}$ which is maximal as a chain of subspaces of $\mathscr {X}$ and every subspace in $\mathscr {C}$ is invariant for all operators in $\mathcal {S}$, then $\mathcal {S}$ is said to be triangularizable.

If a subspace $\mathscr {M}$ is invariant for every operator T in a set $\mathcal {S}$ and in its commutant $\mathcal {S}'$, then $\mathscr {M}$ is said to be a hyperinvariant subspace for $\mathcal {S}$.

Next theorem (see Assertion in the end of [12]) is one of the deepest results in the theory of invariant subspaces.

Theorem 2.1

(Lomonosov) Every non-scalar operator which commutes with a non-zero compact operator has a non-trivial hyperinvariant subspace.

The proof of Theorem 2.1 relies on the following useful lemma.

Lemma 2.2

(Lomonosov’s Lemma) Let $\mathcal {A}\subseteq \mathcal {B}(\mathscr {X})$ be an algebra. If $\mathcal {A}$ is not reducible, then for every non-zero compact operator $K\in \mathcal {B}(\mathscr {X})$ there exist an operator $A\in \mathcal {A}$ and a non-zero vector $x\in \mathscr {X}$ such that $AKx=x$.

2.3 Spectral radius

The spectrum of an operator $T\in \mathcal {B}(\mathscr {X})$ is denoted by $\sigma (T)$ and the spectral radius of T is $\rho (T)=\max \{ |z|;\; z\in \sigma (T)\}$. By the spectral radius formula (Gelfand’s formula), $\rho (T)=\lim \limits _{n\rightarrow \infty } \Vert T^n\Vert ^{\frac{1}{n}}$. An operator $T\in \mathcal {B}(\mathscr {X})$ is quasinilpotent if $\rho (T)=0$. A compact quasinilpotent operator is called a Volterra operator.

Let $\mathcal {F}$ be a non-empty set of operators in $\mathcal {B}(\mathscr {X})$. For each $n\in \mathbb {N}$, let $\mathcal {F}^{(n)}=\{T_1\cdots T_n;\; T_1,\ldots , T_n\in \mathcal {F}\}$. By $\Vert \mathcal {F}\Vert =\sup \{\Vert T\Vert ;\; T\in \mathcal {F}\}$ we denote the joint norm of $\mathcal {F}$ and by $\rho (\mathcal {F})$, we denote the joint spectral radius of $\mathcal {F}$ defined as

$$\begin{aligned} \rho (\mathcal {F})=\limsup _{n\rightarrow \infty } \Vert \mathcal {F}^{(n)}\Vert ^{\frac{1}{n}}. \end{aligned}$$

A subalgebra $\mathcal {A}\subseteq \mathcal {B}(\mathscr {X})$ is said to be finitely quasinilpotent if $\rho (\mathcal {F})=0$ for every finite subset $\mathcal {F}$ of $\mathcal {A}$. By [16, Theorem 1], every subalgebra $\mathcal {A}\subseteq \mathcal {B}(\mathscr {X})$ of Volterra operators is finitely quasinilpotent.

2.4 Polynomially compact operators

An operator $T\in \mathcal {B}(\mathscr {X})$ is polynomially compact if there exists a non-zero complex polynomial p such that p(T) is a compact operator. In particular, algebraic operators (nilpotents, idempotents, etc.) are polynomially compact. Hence, T is polynomially compact if and only if $\pi (T)$, where $\pi :\mathcal {B}(\mathscr {X})\rightarrow \mathcal {B}(\mathscr {X})/\mathcal {K}(\mathscr {X})$ is the quotient projection, is an algebraic element in the Calkin algebra $\mathcal {B}(\mathscr {X})/\mathcal {K}(\mathscr {X})$. If $T^n$ is compact for some $n\in \mathbb {N}$, then T is said to be power compact. For a polynomially compact operator T, there exists a unique monic polynomial $m_T$ of the smallest degree such that $m_T(T)$ is a compact operator. The polynomial $m_T$ is called the minimal polynomial of T. The following is the structure theorem for polynomially compact operators proved by Gilfeather [4, Theorem 1].

Theorem 2.3

Let $T\in \mathcal {B}(\mathscr {X})$ be a polynomially compact operator with minimal polynomial $m_T(z)=(z-\lambda _1)^{n_1}\cdots (z-\lambda _k)^{n_k}$. Then there exist invariant subspaces $\mathscr {X}_1,\ldots , \mathscr {X}_k$ for T such that $\mathscr {X}=\mathscr {X}_1 \oplus \cdots \oplus \mathscr {X}_k$ and $T=T_1\oplus \cdots \oplus T_k$, where $T_i$ is the restriction of T to $\mathscr {X}_i$. The operators $(T_j-\lambda _j I_j)^{n_j}$ are all compact.

The spectrum of T consists of countably many points with $\{ \lambda _1,\ldots ,\lambda _k\}$ as the only possible limit points and such that all but possibly $\{ \lambda _1,\ldots ,\lambda _k\}$ are eigenvalues with finite-dimensional generalized eigenspaces. Each point $\lambda _j$ $(j=1, \ldots ,k)$ is either the limit of eigenvalues of T or else $\mathscr {X}_j$ is infinite-dimensional and $T_j-\lambda _j I_j$ is a quasinilpotent operator on $\mathscr {X}_j$.

Corollary 2.4

A quasinilpotent operator $T\in \mathcal {B}(\mathscr {X})$ is polynomially compact if and only if it is power compact.

Proof

It is clear that every power compact operator is polynomially compact. On the other hand, if T is a polynomially compact and quasinilpotent, then $\sigma (T)=\{0\}$ and therefore $m_T(z)=z^n$ for a positive integer n, by Theorem 2.3, that is, T is power compact. $\square$

2.5 Algebras and ideals

For a non-empty set of operators $\mathcal {S}\subseteq \mathcal {B}(\mathscr {X})$, let $\mathcal {A}(\mathcal {S})$ be the subalgebra of $\mathcal {B}(\mathscr {X})$ generated by $\mathcal {S}$ and let $\mathcal {A}_1(\mathcal {S})$ be the subalgebra of $\mathcal {B}(\mathscr {X})$ generated by $\mathcal {S}$ and I. By $\mathcal {H}(\mathcal {S})$ we denote the algebra which is generated by $\mathcal {S}$ and its commutant $\mathcal {S}'$. We will call it the hyperalgebra of $\mathcal {S}$. If $\mathcal {S}$ is a semigroup, then an operator $T\in \mathcal {B}(\mathscr {X})$ is in $\mathcal {H}(\mathcal {S})$ if and only if there exist $n\in \mathbb {N}$ and operators $S_1, \ldots , S_n\in \mathcal {S}$ and $T_0, T_1, \ldots , T_n\in \mathcal {S}'$ such that

$$\begin{aligned} T=T_0+S_1 T_1+\cdots + S_n T_n=T_0+T_1 S_1+\cdots + T_n S_n . \end{aligned}$$

Here, we used the fact that $I\in \mathcal {S}'$. Since $\mathcal {S}\subseteq \mathcal {H}(\mathcal {S})$ and $\mathcal {A}(\mathcal {S})$ is the smallest algebra which contains $\mathcal {S}$, we have $\mathcal {A}(\mathcal {S})\subseteq \mathcal {H}(\mathcal {S})$. On the other hand, it is obvious that $\mathcal {S}'=\mathcal {A}(\mathcal {S})'$ and therefore $\mathcal {A}(\mathcal {S})'\subseteq \mathcal {H}(\mathcal {S})$. We conclude that the hyperalgebra of $\mathcal {S}$ is generated by $\mathcal {A}(\mathcal {S})$ and $\mathcal {A}(\mathcal {S})'$, that is, $\mathcal {H}(\mathcal {S})=\mathcal {H}\bigl (\mathcal {A}(\mathcal {S})\bigr )$.

If $\mathcal {M}$ and $\mathcal {N}$ are non-empty subsets of $\mathcal {B}(\mathscr {X})$, then let $\mathcal {M}+\mathcal {N}=\{ M+N;\; M\in \mathcal {M}, N\in \mathcal {N}\}$ and let $\mathcal {M}\mathcal {N}$ be the set of all finite sums $M_1 N_1+\cdots +M_k N_k$, where $M_i\in \mathcal {M}$ and $N_i\in \mathcal {N}$ for each $i=1, \ldots , k$. Hence, if $\mathcal {A}$ is a subalgebra of $\mathcal {B}(\mathscr {X})$, then its hyperalgebra is $\mathcal {H}(\mathcal {A})=\mathcal {A}'+\mathcal {A}\mathcal {A}'=\mathcal {A}'+\mathcal {A}'\mathcal {A}$.

If $\mathcal {J}$ is an ideal in an algebra $\mathcal {A}\subseteq \mathcal {B}(\mathscr {X})$, then we write $\mathcal {J}\triangleleft \mathcal {A}$. We will denote by $\mathcal {J}_{_{\mathcal {H}}}$ the ideal in $\mathcal {H}(\mathcal {A})$ generated by $\mathcal {J}$.

Lemma 2.5

Let $\mathcal {A}\subseteq \mathcal {B}(\mathscr {X})$ be an algebra. If $\mathcal {J}\triangleleft \mathcal {A}$, then $\mathcal {J}_{_{\mathcal {H}}}=\mathcal {A}'\mathcal {J}=\mathcal {J}\mathcal {A}'$.

Proof

From equalities $\mathcal {J}_{_{\mathcal {H}}}=\mathcal {J}+\mathcal {H}(\mathcal {A})\mathcal {J}+\mathcal {J}\mathcal {H}(\mathcal {A})+\mathcal {H}(\mathcal {A})\mathcal {J}\mathcal {H}(\mathcal {A})$ and $\mathcal {H}(\mathcal {A})=\mathcal {A}'+\mathcal {A}'\mathcal {A}$, we conclude

$$\begin{aligned} \mathcal {J}_{_{\mathcal {H}}}&=\mathcal {J}+\mathcal {H}(\mathcal {A})\mathcal {J}+\mathcal {J}\mathcal {H}(\mathcal {A})+\mathcal {H}(\mathcal {A})\mathcal {J}\mathcal {H}(\mathcal {A})\\&=\mathcal {J}+\mathcal (\mathcal {A}'+\mathcal {A}'\mathcal {A})\mathcal {J}+\mathcal {J}\mathcal (\mathcal {A}'+\mathcal {A}'\mathcal {A})+\mathcal (\mathcal {A}'+\mathcal {A}'\mathcal {A})\mathcal {J}\mathcal (\mathcal {A}'+\mathcal {A}'\mathcal {A})\\&\subseteq \mathcal {J}+\mathcal {A}'\mathcal {J}\subseteq \mathcal {A}' \mathcal {J}\end{aligned}$$

as $\mathcal {A}'$ contains the identity operator. On the other hand, since we also have $\mathcal {A}'\mathcal {J}\subseteq \mathcal {J}_{_{\mathcal {H}}}$, we obtain the equality $\mathcal {J}_{_{\mathcal {H}}}=\mathcal {A}'\mathcal {J}$. $\square$

An algebra $\mathcal {A}$ over an arbitrary field is said to be a nil-algebra if every element of $\mathcal {A}$ is nilpotent. A nil-algebra $\mathcal {A}$ is of bounded nil-index if there exists a positive integer n such that $x^n=0$ for each $x\in \mathcal {A}$. If there exists $n\in \mathbb {N}$ such that $a_1\cdots a_n=0$ for all $a_1,\ldots ,a_n\in \mathcal {A}$, then $\mathcal {A}$ is said to be a nilpotent algebra. By the celebrated Nagata-Higman theorem (see [8, 13]), every nil-algebra of bounded nil-index is nilpotent.

Lemma 2.6

Let $\mathcal {A}\subseteq \mathcal {B}(\mathscr {X})$ be an algebra. An ideal $\mathcal {J}\triangleleft \mathcal {A}$ is nilpotent if and only if $\mathcal {J}_{_{\mathcal {H}}}\triangleleft \mathcal {H}(\mathcal {A})$ is nilpotent. The nilpotency indices of $\mathcal {J}$ and $\mathcal {J}_{_{\mathcal {H}}}$ are equal.

Proof

Since $\mathcal {A}'$ and $\mathcal {J}$ commute, an easy induction shows that for each $n\in \mathbb {N}$ we have $(\mathcal {A}'\mathcal {J})^n=\mathcal {A}' \mathcal {J}^n$. Hence, if $\mathcal {J}^n=\{0\}$, then $(\mathcal {A}'\mathcal {J})^n=\{0\}$, as well. If $(\mathcal {A}'\mathcal {J})^n=\{0\}$, then $\mathcal {J}^n \subseteq \mathcal {A}'\mathcal {J}^n=(\mathcal {A}'\mathcal {J})^n=\{0\}$ yields that $\mathcal {J}^n=\{0\}$. $\square$

If $\mathcal {J}$ is a nil-ideal of bounded nil-index, then $\mathcal {J}$ is nilpotent by the Nagata-Higman theorem. This immediately implies that $\mathcal {J}_{_{\mathcal {H}}}$ is nilpotent. In particular, if $\mathcal {A}$ is a nilpotent algebra, then $\mathcal {A}\mathcal {A}'$ is a nilpotent ideal in the hyperalgebra $\mathcal {H}(\mathcal {A})$.

3 Hyperinvariant subspaces of algebras of polynomially compact operators

The simplest polynomially compact operators which are not necessary compact are algebraic operators, in particular nilpotent operators. Hadwin et al. [7, Corollary 4.2] proved that a norm closed algebra of nilpotent operators on the separable infinite-dimensional complex Hilbert space is triangularizable. The following proposition shows that a norm closed algebra of nilpotent operators on an arbitrary complex Banach space has a non-trivial hyperinvariant subspace.

Proposition 3.1

If a subalgebra $\{ 0\}\ne \mathcal {A}\subseteq \mathcal {B}(\mathscr {X})$ consists of nilpotent operators, then its hyperalgebra $\mathcal {H}(\mathcal {A})$ is reducible in either of the following cases.

• (a) The algebra $\mathcal {A}$ is nilpotent.

• (b) The algebra $\mathcal {A}$ is norm closed.

Proof

(a) There exists $n\in \mathbb {N}$ such that an arbitrary product of at least n operators from $\mathcal {A}$ is the zero operator. Let $n_0$ be the smallest positive integer with this property. Since $\mathcal {A}\ne \{ 0\}$ we have $n_0>1.$ There exist operators $A_1,\ldots ,A_{n_0-1}\in \mathcal {A}$ such that $A_0:=A_1,\ldots ,A_{n_0-1}\ne 0$. Note that $A_0T=TA_0=0$ for every operator $T\in \mathcal {A}$, that is, $\mathcal {A}\subseteq (A_0)'$, where $(A_0)'$ is the commutant of $A_0$. It is clear that $\mathcal {A}'\subseteq (A_0)'$. Hence, $\mathcal {H}(\mathcal {A})\subseteq (A_0)'$. Since $A_0\ne 0$ the kernel $\ker (A_0)$ is a non-trivial subspace of $\mathscr {X}$ and it is hyperinvariant for $A_0$. It follows that $\ker (A_0)$ is a non-trivial hyperinvariant subspace for $\mathcal {A}$, that is, $\mathcal {H}(\mathcal {A})$ is reducible.

(b) Since $\mathcal {A}$ is a closed subalgebra of $\mathcal {B}(\mathscr {X})$, it is a Banach algebra. Therefore, $\mathcal {A}$ is a Banach algebra which is also a nil-algebra, so that by Grabiner’s theorem [5], $\mathcal {A}$ is a nilpotent algebra. Now, we apply (a). $\square$

Hadwin et al. have constructed a semi-simple algebra of nilpotent operators on a separable Hilbert space such that for each pair (x, y) of vectors, where $x\ne 0$, there exists an operator $A\in \mathcal {A}$ such that $Ax=y$ (see [7, Section 4]). Hence, Proposition 3.1(b) (and consequently Theorem 3.3 below) does not hold, in general, for non-closed algebras. However, as the following example shows there are simple special cases when a not necessarily closed algebra generated by a set of nilpotents has a non-trivial hyperinvariant subspace.

Example

The idea for this example is from [7, Theorem 4.3] where quadratic nilpotent operators are considered. Choose and fix $\lambda \in \mathbb {C}$. Let $\mathcal {S}\subseteq \mathcal {B}(\mathscr {X})$ be a non-empty set of operators such that $A^2=\lambda A$ for every $A\in \mathcal {S}$. If $\mathcal {S}$ is a linear manifold and contains a non-scalar operator, then $\mathcal {A}(\mathcal {S})$ has a non-trivial hyperinvariant subspace. To see this, choose a non-scalar operator $A \in \mathcal {S}$. Then its kernel $\ker (A)$ is a non-trivial hyperinvariant subspace for A. Since $\mathcal {A}(\mathcal {S})'\subseteq (A)'$, we see that $\ker (A)$ is invariant for every operator from $\mathcal {A}(\mathcal {S})'$. Let $B\in \mathcal {S}$ be arbitrary. It follows from $(A+B)^2=\lambda (A+B)$ that $AB=-BA$. Hence, $ABx=-BAx=0$ for every $x\in \ker (A)$, that is, $\ker (A)$ is invariant for every $B\in \mathcal {S}$. We conclude that $\ker (A)$ is invariant for every operator from the hyperalgebra $\mathcal {H}(\mathcal {S})$.

Proposition 3.2

Let $\mathcal {A}\subseteq \mathcal {B}(\mathscr {X})$ be a non-trivial subalgebra. If there exists a non-trivial nilpotent ideal $\mathcal {J}\triangleleft \mathcal {A}$, then $\mathcal {A}$ has a non-trivial hyperinvariant subspace.

Proof

Let $\mathcal {J}$ be a non-trivial nilpotent ideal in $\mathcal {A}(\mathcal {S})$. Then, by Lemma 2.6, the ideal $\mathcal {J}_{_{\mathcal {H}}}$ which is generated by $\mathcal {J}$ in the hyperalgebra $\mathcal {H}(\mathcal {A})$ is nilpotent, as well. By Proposition 3.1(a) the ideal $\mathcal {J}_{_{\mathcal {H}}}$ has a non-trivial hyperinvariant subspace, in particular, it is reducible. By [14, Lemma 7.4.6], $\mathcal {H}(\mathcal {A})$ is reducible, as well. $\square$

The following two theorems show that an algebra $\mathcal {A}$ of polynomially compact operators has a non-trivial hyperinvariant subspace if $\mathcal {A}$ satisfies some additional condition. For instance, as we already mentioned, the key assumption in Theorem 3.3 is that the involved algebra is norm closed.

Theorem 3.3

Let $\{ 0\}\ne \mathcal {A}\subseteq \mathcal {B}(\mathscr {X})$ be a norm closed algebra. If every operator in $\mathcal {A}$ is quasinilpotent and polynomially compact, then $\mathcal {A}$ has a non-trivial hyperinvariant subspace.

Proof

Note first that each operator in $\mathcal {A}$ is power compact, by Corollary 2.4. If each operator in $\mathcal {A}$ is nilpotent, then $\mathcal {H}(\mathcal {A})$ is reducible, by Proposition 3.1(b). Assume therefore that there exists an operator $T\in \mathcal {A}$ which is not nilpotent. Since T is power compact there exists $m\in \mathbb {N}$ such that $K=T^m\ne 0$ is compact. Let $\mathcal {J}$ be the two-sided ideal in $\mathcal {A}_1$ generated by K. It is clear that $K\in \mathcal {J}\subseteq \mathcal {A}$. Hence, $\mathcal {J}$ is an algebra of Volterra operators. By [16, Theorem 1], $\mathcal {J}$ is finitely quasinilpotent. Let $A=B_0+\sum _{i=1}^{n} A_iB_i$, where $A_i\in \mathcal {A}$ and $B_i\in \mathcal {A}'$, be an arbitrary operator in $\mathcal {H}(\mathcal {A})$ and let $\mathcal {M}=\{K,A_1K,\ldots ,A_nK\}$. Since $\mathcal {M}$ is a finite subset of $\mathcal {J}$ it is a quasinilpotent set, that is, $\rho (\mathcal {M})=0$. Let $\mathcal {N}=\{B_0, B_1,\ldots ,B_n\}\subseteq \mathcal {A}'$. Since $B_0, B_1,\ldots ,B_n$ commute with operators from $\mathcal {M}$ we have $\rho (AK)\le (n+1)\rho (\mathcal {M}) \rho (\mathcal {N})=0$, by [16, Lemma 1]. This shows that the operator AK is quasinilpotent for each $A\in \mathcal {H}(\mathcal {A})$. It follows, by Lemma 2.2, that the hyperalgebra $\mathcal {H}(\mathcal {A})$ is reducible.

Theorem 3.4

Let $\mathcal {S}\subseteq \mathcal {B}(\mathscr {X})$ be an essentially commutative set of polynomially compact operators which contains at least one non-scalar operator. If $\mathcal {S}$ is triangularizable, then $\mathcal {A}(\mathcal {S})$ has a non-trivial hyperinvariant subspace.

Proof

It is obvious that every subspace of $\mathscr {X}$ which is invariant for $\mathcal {S}$ is invariant for $\mathcal {A}(\mathcal {S})$, as well. Hence, $\mathcal {A}(\mathcal {S})$ is triangularizable. Since $\mathcal {S}$ consists of polynomially compact operators, the same holds for the algebra $\mathcal {A}(\mathcal {S})$, by [10, Theorem 1.5]. Denote by $\pi \bigl (\mathcal {A}(\mathcal {S})\bigr )$ and $\pi (\mathcal {S})$ the image of $\mathcal {A}(\mathcal {S})$ and $\mathcal {S}$, respectively, in the Calkin algebra $\mathcal {B}(\mathscr {X})/\mathcal {K}(\mathscr {X})$. Since $\pi \bigl (\mathcal {A}(\mathcal {S})\bigr )$ is the subalgebra of the Calkin algebra generated by the commutative set $\pi (\mathcal {S})$ we see that $\pi \bigl (\mathcal {A}(\mathcal {S})\bigr )$ is commutative, as well. It follows that $\mathcal {A}(\mathcal {S})$ is an essentially commutative subalgebra of $\mathcal {B}(\mathscr {X})$. By [9, Theorem 3.5], for an algebra of essentially commuting polynomially compact operators triangularizability of $\mathcal {A}(\mathcal {S})$ is equivalent to the fact that each commutator [S, T], where $S, T\in \mathcal {A}(\mathcal {S})$, is a quasinilpotent operator.

Suppose that there exist operators $S, T\in \mathcal {A}(\mathcal {S})$ such that the commutator $K:=[S,T]$ is non-zero. Let $\mathcal {J}$ be the ideal in $\mathcal {A}(\mathcal {S})$ generated by K. Of course, $\mathcal {J}$ consists of compact operators and, by Ringrose’s Theorem (see [14, Theorem 7.2.3]), every operator in $\mathcal {J}$ is quasinilpotent. Hence, $\mathcal {J}$ is an algebra of Volterra operators. Let $\mathcal {J}_{_{\mathcal {H}}}$ be the ideal in the hyperalgebra $\mathcal {H}(\mathcal {S})$ generated by $\mathcal {J}$. It is clear that $\mathcal {J}_{_{\mathcal {H}}}\subseteq \mathcal {K}(\mathscr {X})$. We claim that $\mathcal {J}_{_{\mathcal {H}}}$ consists of quasinilpotent operators, as well. Choose an arbitrary operator $A\in \mathcal {J}_{_{\mathcal {H}}}$. By Lemma 2.5, there exist operators $J_1,\ldots ,J_n\in \mathcal {J}$ and operators $B_1,\ldots ,B_n\in \mathcal {A}(\mathcal {S})'$ such that $A=\sum _{i=1}^{n} J_i B_i$. By [16, Lemma 1], $\rho (A)\le n \rho (\{J_1,\ldots ,J_n\}) \rho (\{B_1,\ldots , B_n\})$. Since each finite subset of an algebra of Volterra operators is quasinilpotent, by [16, Theorem 1], we have $\rho (\{J_1,\ldots ,J_n\})=0$ and consequently $\rho (A)=0$. We have proved that $\mathcal {J}_{_{\mathcal {H}}}$ is a non-trivial ideal of Volterra operators. By [16, Theorem 2], the hyperalgebra $\mathcal {H}(\mathcal {S})$ is reducible.

Suppose now that $\mathcal {A}(\mathcal {S})$ is commutative. Hence, $\mathcal {A}(\mathcal {S})\subseteq \mathcal {A}(\mathcal {S})'$. Let T be any non-scalar polynomially compact operator in $\mathcal {A}(\mathcal {S})$ and let $m_T$ be its minimal polynomial. Hence, $m_T(T)$ is a compact operator. If m(T) is a non-zero compact operator, then it has a non-trivial hyperinvariant subspace $\mathscr {Y}$, by Lomonosov’s Theorem. Since $\mathcal {A}(\mathcal {S})\subseteq \mathcal {A}(\mathcal {S})'\subseteq (m_T(T))'$ subspace $\mathscr {Y}$ is invariant for every operator in $\mathcal {A}(\mathcal {S})$ and $\mathcal {A}(\mathcal {S})'$. Thus, $\mathcal {H}(\mathcal {S})$ is reducible. On the other hand, if $m_T(T)=0$, then T is a non-scalar algebraic operator. Hence, for every $\lambda \in \sigma (T)$, the kernel $\ker (T-\lambda I)$ is a non-trivial hyperinvariant subspace for T, and consequently, for $\mathcal {A}(\mathcal {S})$ as $\mathcal {A}(\mathcal {S})\subseteq \mathcal {A}(\mathcal {S})'\subseteq (T)'$. $\square$

4 Hyperinvariant subspaces for operator bands

An operator band on a Banach space $\mathscr {X}$ is a (multiplicative) semigroup $\mathcal {S}\subseteq \mathcal {B}(\mathscr {X})$ of idempotents, that is, $S^2=S$ for each $S\in \mathcal {S}$. The linear span of an operator band $\mathcal {S}$ is the algebra $\mathcal {A}(\mathcal {S})$ called a band algebra. We will denote by $\mathcal {N}(\mathcal {S})$, the set of all nilpotent operators in the band algebra $\mathcal {A}(\mathcal {S})$. By [11, Theorem 5.2], $\mathcal {N}(\mathcal {S})$ is the linear span of $[\mathcal {S},\mathcal {S}]$, that is, $\mathcal {N}(\mathcal {S})=\{ \sum _{i=1}^{n}[A_i,B_i];\; n\in \mathbb {N},\, A_i,B_i\in \mathcal {A}(\mathcal {S})\}$. By [11, Corollary 5.5], the set $\mathcal {N}(\mathcal {S})$ of nilpotent operators in $\mathcal {A}(\mathcal {S})$ coincides with the Jacobson radical ${\mathcal {R}}\bigl (\mathcal {A}(\mathcal {S})\bigr )$ of $\mathcal {A}(\mathcal {S})$.

Proposition 4.1

Let $\{ 0\}\ne \mathcal {S}\subseteq \mathcal {B}(\mathscr {X})$ be an operator band. If $\mathcal {N}\subseteq \mathcal {N}(\mathcal {S})$ is a non-empty set, then the ideal $\mathcal {J}_{_{\mathcal {H}}}$ in the hyperalgebra $\mathcal {H}(\mathcal {S})$ generated by $\mathcal {N}$ is a nil-ideal.

Proof

Denote by $\mathcal {J}$ the ideal in $\mathcal {A}(\mathcal {S})$ generated by $\mathcal {N}$. We have already mentioned that ${\mathcal {R}}\bigl (\mathcal {A}(\mathcal {S})\bigr )=\mathcal {N}(\mathcal {S})$. Since the radical is an ideal, we have that $\mathcal {J}\subseteq {\mathcal {R}}\bigl (\mathcal {A}(\mathcal {S})\bigr )$. Hence, $\mathcal {J}$ is a nil-ideal in $\mathcal {A}(\mathcal {S})$.

It is clear that $\mathcal {N}$ and ideal $\mathcal {J}$ generate the same ideal $\mathcal {J}_{_{\mathcal {H}}}$ in the hyperalgebra $\mathcal {H}(\mathcal {S})$. Let $A\in \mathcal {J}_{_{\mathcal {H}}}$ be arbitrary. By Lemma 2.5, there exist $n\in \mathbb {N}$, $J_i\in \mathcal {J}$ and $B_i\in \mathcal {A}(\mathcal {S})'$ such that $A=\sum _{i=1}^n J_i B_i$. Let $\mathcal {F}$ be a finite subset of $\mathcal {S}$ such that $J_1,\ldots ,J_n$ are contained in the algebra $\mathcal {A}(\mathcal {F})$ generated by $\mathcal {F}$. Note that such finite sets exist, because each $J_i$ is of the form $p_i(S_{1}^{(i)}, \ldots , S_{k_i}^{(i)})$, where $p_i$ is a polynomial of $k_i$ non-commuting variables and $S_{1}^{(i)}, \ldots , S_{k_i}^{(i)}$ $(i=1, \ldots , n)$ are idempotents from $\mathcal {S}$. Denote by $\mathcal {S}_0$ the operator band generated by $\mathcal {F}$. Thus, $\mathcal {A}(\mathcal {F})\subseteq \mathcal {A}(\mathcal {S}_0)$. Since $\mathcal {F}$ is finite, the operator band $\mathcal {S}_0$ is finite, as well, by the Green-Rees theorem (see [14, Theorem 9.3.11]). Now, we apply [14, Theorem 9.3.15] which says that there exists a finite chain $\{0\}=\mathscr {X}_0\subseteq \mathscr {X}_1\subseteq \cdots \subseteq \mathscr {X}_{m}=\mathscr {X}$ of invariant subspaces for $\mathcal {S}_0$ such that for each $E\in \mathcal {S}_0$ the operator induced by E on $\mathscr {X}_i/\mathscr {X}_{i-1}$ is either zero or the identity operator. This implies that operator induced by $J_i$ on $\mathscr {X}_j/\mathscr {X}_{j-1}$ is a scalar multiple of the identity operator. Since each $J_i$ is nilpotent, every operator $J_i$ induces the zero operator on $\mathscr {X}_j/\mathscr {X}_{j-1}$. Hence, for each i and each j, we have $J_i(\mathscr {X}_j)\subseteq \mathscr {X}_{j-1}$. This implies that an arbitrary product of length at least m with letters from $\{J_1,\ldots ,J_n\}$ is zero. Now it is obvious that $A^{m}=\left( \sum _{i=1}^n J_i B_i\right) ^m=0$ as $J_i$ and $B_j$ commute. $\square$

Proposition 4.2

Let $\{ 0\}\ne \mathcal {S}\subseteq \mathcal {B}(\mathscr {X})$ be an operator band. If $K\in \overline{\mathcal {A}(\mathcal {S})}$ is a compact operator, then for each operator $A\in \overline{\mathcal {H}(\mathcal {S})}$ the commutator [K, A] is in the Jacobson radical of $\overline{\mathcal {H}(\mathcal {S})}$, that is, $[K,\overline{\mathcal {H}(\mathcal {S})}]\subseteq {\mathcal {R}}\bigl (\overline{\mathcal {H}(\mathcal {S})}\bigr )$.

Proof

Let $K\in \overline{\mathcal {A}(\mathcal {S})}$ be a compact operator and let $A\in \overline{\mathcal {H}(\mathcal {S})}$ be an arbitrary operator. To prove that the commutator [K, A] is in ${\mathcal {R}}\bigl (\overline{\mathcal {H}(\mathcal {S})}\bigr )$, we need to show that [K, A]C is quasinilpotent for each $C\in \overline{\mathcal {H}(\mathcal {S})}$. Since [K, A]C is compact, the continuity of the spectral radius at compact operators yields that it suffices to prove that [K, A]C is quasinilpotent for all $A, C\in \mathcal {H}(\mathcal {S})$.

Let $A,C\in \mathcal {H}(\mathcal {S})$ be arbitrary. Since K belongs to the closure of $\mathcal {A}(\mathcal {S})$, there is a sequence $(K_n)_{n\in \mathbb {N}}\subseteq \mathcal {A}(\mathcal {S})$ which converges to K. We claim that for each $n\in \mathbb {N}$ the commutator $[K_n,A]$ is nilpotent and belongs to the ideal $\mathcal {N}(\mathcal {S})_{_{\mathcal {H}}}$ in $\mathcal {H}(\mathcal {S})$ generated by the Jacobson radical $\mathcal {N}(\mathcal {S})$ of the band algebra $\mathcal {A}(\mathcal {S})$. Since $A\in \mathcal {H}(\mathcal {S})$, there exist operators $A_1,\ldots , A_k\in \mathcal {A}(\mathcal {S})$ and $B_0,\ldots ,B_k\in \mathcal {A}(\mathcal {S})'$ such that $A=B_0+\sum _{i=1}^kA_iB_i$. It follows that $[K_n,A]=\sum _{i=1}^k[K_n,A_i]B_i.$ Since every commutator $[K_n,A_i]$ is in $\mathcal {N}(\mathcal {S})$ and $\mathcal {N}(\mathcal {S})\subseteq \mathcal {N}(\mathcal {S})_{_{\mathcal {H}}}$, it follows $[K_n,A_i] \in \mathcal {N}(\mathcal {S})_{_{\mathcal {H}}}$. Since $\mathcal {N}(\mathcal {S})_{_{\mathcal {H}}}$ is an ideal in the hyperalgebra $\mathcal {H}(\mathcal {S})$ we have $[K_n,A]C\in \mathcal {N}(\mathcal {S})_{_{\mathcal {H}}}$. By Proposition 4.1, the operator $[K_n,A]C$ is nilpotent. To finish the proof, we apply the fact that the spectral radius is continuous at compact operators and that the compact operator [K, A]C is the limit of the sequence $([K_n,A]C)_{n\in \mathbb {N}}$ of nilpotent operators. $\square$

Theorem 4.3

Let $\mathcal {S}\subseteq \mathcal {B}(\mathscr {X})$ be an operator band. If there exists a non-zero compact operator in $\overline{\mathcal {A}(\mathcal {S})}$, then $\mathcal {S}$ has a non-trivial hyperinvariant subspace.

In this proof we added the closures (\overline) eight times around \mathcal {H}(\mathcal {S})

Let K be a non-zero compact operator in $\overline{\mathcal {A}(\mathcal {S})}$. Then, by Proposition 4.2, [K, A] is a compact operator contained in the radical ${\mathcal {R}}\bigl (\overline{\mathcal {H}(\mathcal {S})}\bigr )$ for each $A\in \overline{\mathcal {H}(\mathcal {S})}$. If $[K,A]=0$ for every $A\in \overline{\mathcal {H}(\mathcal {S})}$, then $\overline{\mathcal {H}(\mathcal {S})}\subseteq (K)'$ and, therefore, every non-trivial hyperinvariant subspace of K is a non-trivial hyperinvariant subspace for $\mathcal {A}(\mathcal {S})$. Hence, we may assume that for some $A\in \overline{\mathcal {H}(\mathcal {S})}$ the commutator [K, A] is a non-zero operator. Let $\mathcal {J}$ be the ideal in $\overline{\mathcal {H}(\mathcal {S})}$ generated by [K, A]. Of course, each operator in $\mathcal {J}$ is compact. Since, by Proposition 4.2, [K, A] is in the Jacobson ideal of ${\mathcal {R}}\bigl (\overline{\mathcal {H}(\mathcal {S})}\bigr )$ we have $\mathcal {J}=\overline{\mathcal {H}(\mathcal {S})} [K,A] \overline{\mathcal {H}(\mathcal {S})}\subseteq \overline{\mathcal {H}(\mathcal {S})} {\mathcal {R}}\bigl (\overline{\mathcal {H}(\mathcal {S})}\bigr ) \overline{\mathcal {H}(\mathcal {S})}\subseteq {\mathcal {R}}\bigl (\overline{\mathcal {H}(\mathcal {S})}\bigr )$. It follows that operators in $\mathcal {J}$ are quasinilpotent. Thus, $\mathcal {J}$ is a Volterra ideal in $\overline{\mathcal {H}(\mathcal {S})}$. By [16, Theorem 2], $\mathcal {J}$ is reducible. Now we apply [14, Lemma 7.4.6] and conclude that $\overline{\mathcal {H}(\mathcal {S})}$ is reducible. $\square$

Corollary 4.4

Every essentially commuting non-scalar operator band $\mathcal {S}\subseteq \mathcal {B}(\mathscr {X})$ has a non-trivial hyperinvariant subspace.

Proof

If $\mathcal {S}$ is commutative, then $\mathcal {S}\subseteq \mathcal {S}'$. In this case the kernel of any non-scalar operator from $\mathcal {S}$ is invariant for each $S\in \mathcal {S}'$. If $\mathcal {S}$ is not commutative, then there exist idempotents $E,F\in \mathcal {S}$ with a non-zero compact commutator $EF-FE\in \mathcal {A}(\mathcal {S})$. The assertion follows, by Theorem 4.3. $\square$

Since every essentially commuting band of operators has an invariant subspace, an application of the Triangularization lemma (see [14, Lemma 7.1.11]) immediately implies the following result.

Corollary 4.5

Every essentially commuting band of operators on a Banach space is triangularizable.