Higgs bundles for the non-compact dual of the special orthogonal group

Abstract

Higgs bundles over a closed orientable surface can be defined for any real reductive Lie group $G$. In this paper we examine the case $G=\mathrm {SO}^*(2n)$. We describe a rigidity phenomenon encountered in the case of maximal Toledo invariant. Using this and Morse theory in the moduli space of Higgs bundles, we show that the moduli space is connected in this maximal Toledo case. The Morse theory also allows us to show connectedness when the Toledo invariant is zero. The correspondence between Higgs bundles and surface group representations thus allows us to count the connected components with zero and maximal Toledo invariant in the moduli space of representations of the fundamental group of the surface in $\mathrm {SO}^*(2n)$.

1 Introduction

Higgs bundles over a Riemann surface are intrinsically holomorphic objects. Their moduli spaces can nevertheless be identified with representation varieties for the fundamental group of the surface even if the target group for the representations, or equivalently the group defining the Higgs bundles, is a real reductive Lie group. If the group, say $G$, is of Hermitian type, i.e. if the homogeneous space $G/H$ (where $H$ is a maximal compact subgroup) is a Hermitian symmetric space, then the associated $G$-Higgs bundles have especially rich structure. The real connected semisimple classical groups with this property are $\mathrm {SU}(p,q), \mathrm {Sp}(2n,\mathbb {R})$, $\mathrm {SO}(2,n)$, and $\mathrm {SO}^*(2n)$. In this paper we examine in detail the case of $G=\mathrm {SO}^*(2n)$. In particular, we give proofs of the results that were announced in [6].

The theory of $G$-Higgs bundles with $G$ a real Lie group goes back to Hitchin’s seminal papers [23, 24] in which split real forms were considered. Since then, the $G$-Higgs bundles for many other real forms have been examined. Among the real groups of Hermitian type, $\mathrm {Sp}(2n,\mathbb {R})$ is special because it is also a split real form and therefore a particular case of the situation studied by Hitchin. Higgs bundles for the groups $\mathrm {Sp}(2n,\mathbb {R})$ and $\mathrm {SU}(p,q)$ have been studied (in chronological order) in [5–7, 14, 17, 18, 20, 37] and also, most recently and from a different point of view in [25, 33]. In the paper [6] we announced results on $\mathrm {SO}(2,n)$ and $\mathrm {SO}^*(2n)$; the recent preprint [26] addresses some aspects of the $\mathrm {SO}^*(2n)$ case from a different point of view. Higgs bundles for all the groups of Hermitian symmetric type, including the two exceptional cases found among the real forms of $E_6$ and $E_7$, have also been studied in [32], where the first steps towards a unified treatment were taken. The group $\mathrm {SO}(2,n)$ falls into the more general (but not in general Hermitian symmetric) case of $\mathrm {SO}(p,q)$-Higgs bundles, which were studied in [1, 2]. On the other side of the correspondence between Higgs bundles and surface group representations, the groups of Hermitian type have been extensively studied, notably recently in [3, 9, 21].

In some ways the work described in this paper is one more in a series of case-by-case analyses of $G$-Higgs bundles for different $G$. Adding to its interest, however, is the fact that the analysis of $\mathrm {SO}^*(2n)$-Higgs bundles unavoidably involves other reductive groups. Any discussion of $\mathrm {SO}^*(2n)$-Higgs bundles is thus a showcase for several types of $G$-Higgs bundles.

The most direct way that other groups enter the picture is through the structure of polystable $\mathrm {SO}^*(2n)$-Higgs bundles. In general (see Theorem 3.25) such Higgs bundles decompose as a sum of $G$-Higgs bundles where $G$ can be one of a number of different groups, including $\mathrm {SO}^*(2m)$ for $m<n$, but also $\mathrm {U}^*(m), \mathrm {U}(p,q)$, and $\mathrm {U}(m)$ for suitable values of $m,p,q$. At the level of Lie theory, these are the groups which appear as factors in Levi subgroups of $\mathrm {SO}(2n,\mathbb {C})$ intersected with $\mathrm {SO}^*(2n)$. Note that this list of groups includes both compact and non-compact real forms. In the latter case the corresponding symmetric space may be Hermitian or not.

The group $\mathrm {U}^*(m)$ appears in a second way that depends on a key feature of $G$-Higgs bundles for non-compact real forms of Hermitian type. In these cases a discrete invariant known as the Toledo invariant can be defined. The invariant has several interpretations (see [6, 9–12, 26, 29, 33]) but all lead to a bound that generalizes the Milnor inequality on the Euler class of flat $\mathrm {SL}(2,\mathbb {R})$-bundles. The $G$-Higgs bundles with maximal Toledo invariant all have special properties but these fall into two categories, depending on whether the Hermitian symmetric space is of tube type or not. In the tube cases, a correspondence emerges between polystable $G$-Higgs bundles with maximal Toledo invariant and objects called $K^2$-twisted $G'$-Higgs bundles, where $G'$ is a new reductive group. We call this $G'$ $K^2$-twisted $G'$-Higgs bundle the Cayley partner to the original $G$-Higgs bundle. In the non-tube cases, the maximal $G$-Higgs bundles do not have Cayley partners but decompose into two parts, one of which has a Cayley partner and the other of which corresponds to a compact group. This imposes constraints which we refer to as ‘rigidity’ on the moduli spaces. For $G=\mathrm {SO}^*(2n)$ we see both types of phenomena, depending on whether $n$ is even or odd. In the odd case, the group is not of tube type and we see rigidity (see Sect. 4.2). For $n=2m$, the group is of tube type and the Cayley partner to $\mathrm {SO}^*(2n)$ is the group $\mathrm {U}^*(n)$.

There is one more group that enters the discussion, namely $\mathrm {Sp}(2n,\mathbb {R})$. While the nature of the relation between $\mathrm {SO}^*(2n)$-Higgs bundles and $\mathrm {Sp}(2n,\mathbb {R})$-Higgs bundles is more subtle than in the case of the groups which appear in Levi subgroups, the comparison between the two cases is instructive and unavoidable. In both cases the maximal compact subgroups are isomorphic to $\mathrm {U}(n)$, and the complexified isotropy representations are

$$\begin{aligned} {\left\{ \begin{array}{ll} \Lambda ^2(\mathbb {C}^n)\oplus \Lambda ^2((\mathbb {C}^n)^*)\ &{} \mathrm {for}\quad \mathrm {SO}^*(2n)\\ {{\mathrm{Sym}}}^2(\mathbb {C}^n)\oplus {{\mathrm{Sym}}}^2((\mathbb {C}^n)^*) &{} \mathrm {for}\quad \mathrm {Sp}(2n,\mathbb {R}) \end{array}\right. } \end{aligned}$$

(1.1)

These structural similarities between $\mathrm {SO}^*(2n)$ and $\mathrm {Sp}(2n,\mathbb {R})$ carry over to the theory of Higgs bundles. In both cases a $G$-Higgs bundle over a Riemann surface is defined by triple $(V,\beta ,\gamma )$ where $V$ is a rank $n$ holomorphic bundle, and $\beta$ and $\gamma$ are homomorphisms

$$\begin{aligned} \beta : V^*\longrightarrow V \otimes K \quad \text{ and }\quad \gamma : V\longrightarrow V^* \otimes K. \end{aligned}$$

The difference between the cases $G=\mathrm {SO}^*(2n)$ and $G=\mathrm {Sp}(2n,\mathbb {R})$ is that in the former case the maps $\beta$ and $\gamma$ are skew-symmetric, while in the latter case the maps are symmetric. However in both cases, the quadruple $(V,V^*,\beta ,\gamma )$ defines a $\mathrm {SU}(n,n)$-Higgs bundle.¹ (see Sect. 8.2.1, where $\mathrm {U}(n,n)$-Higgs bundles $(V,W,\beta ,\gamma )$ are defined. One has here the extra condition $\det W={(\det V)}^{-1}$ since the group is $\mathrm {SU}(n,n)$). Indeed both types of Higgs bundles appear in the moduli space of $\mathrm {SU}(n,n)$-Higgs bundles as fixed points of involutions, namely

$$\begin{aligned} (V,W,\beta ,\gamma )\mapsto (W^*, V^*,\pm \beta ^t,\pm \gamma ^t). \end{aligned}$$

The similarities between the two cases mean that many of the details worked out in [14] for $\mathrm {Sp}(2n,\mathbb {R})$-Higgs bundles require only minor modification in order to be applied to $\mathrm {SO}^*(2n)$-Higgs bundles. However, we believe that the presentation in this paper naturally benefits from a more systematic understanding of the theory. Our main results show that the outcomes in the two cases are significantly different in at least two respects. First, the parity of $n$ plays a role if $G=\mathrm {SO}^*(2n)$ (but not if $G=\mathrm {Sp}(2n,\mathbb {R})$), and second the moduli space of Higgs bundles with maximal $\deg (V)$ has just one connected component if $G=\mathrm {SO}^*(2n)$ but has several connected components distinguished by ‘hidden’ topological invariants revealed through the Cayley correspondence in the case $G=\mathrm {Sp}(2n,\mathbb {R})$.

We now describe the contents of the paper in a bit more detail. Let $X$ be a Riemann surface of genus $g\ge 2$. After some general definitions in Sect. 2, in Sect. 3 we describe the main features of the groups $\mathrm {SO}^*(2n)$ and $\mathrm {SO}^*(2n)$-Higgs bundles. We give structure results for stable and polystable objects. As in the case $G=\mathrm {Sp}(2n,\mathbb {R})$, the moduli space of polystable $\mathrm {SO}^*(2n)$-Higgs bundles, denoted by $\mathcal {M}(\mathrm {SO}^*(2n))$, is not connected. The Toledo invariant, which in the case of $\mathrm {SO}^*(2n)$-Higgs bundles corresponds to the the degree of the bundle $V$, separates the moduli space into components $\mathcal {M}_d$ (where $d=\deg V$). In Sect. 3.7 we establish the bounds on this invariant, namely

$$\begin{aligned} 0\le |d|\le \left\lfloor \frac{n}{2}\right\rfloor (2g-2). \end{aligned}$$

(1.2)

In Sect. 4 we study the case $d=\lfloor \frac{n}{2}\rfloor (2g-2)$ (the case $d=-\lfloor \frac{n}{2}\rfloor (2g-2)$ is analogous). The special feature in this maximal situation is that the component

$$\begin{aligned} \gamma :V\longrightarrow V^*\otimes K \end{aligned}$$

of the Higgs field has maximal rank. Since $\gamma$ is skew-symmetric, this means that it defines a symplectic structure on either $V\otimes K^{-1/2}$ (if $n$ is even) or on a rank $n-1$ quotient of this (if $n$ is odd). This leads to the Cayley correspondence we describe in Sect. 4.1 and to the rigidity result in Sect. 4.2.

The moduli spaces of Higgs bundles come equipped with a natural function that can be used in a Morse-theoretic way to detect topological properties. First described by Hitchin [23], this function measures the $L^2$-norm of the Higgs field. For each $d$, the function provides a proper map from $\mathcal {M}_d$ to $\mathbb {R}$ and thus attains its minimum on each connected component. In Sect. 5 we examine the minima and show that they are precisely the polystable Higgs bundles in which $\beta =0$ or $\gamma =0$ (depending on the sign of $d$). This reduces the problem of the connectivity of the components to one of the connectivity of the locus of minima. Unfortunately for most values of $d$ this is itself a difficult problem. The only exceptions are the cases where $d=0$ or where $|d|$ has its maximum value. In Sect. 5 we also examine these exceptional cases and show the following.

Theorem 1.1

For $d=0$ or $|d|$ maximal, the components $\mathcal {M}_d(\mathrm {SO}^*(2n))$ of the moduli space of polystable $\mathrm {SO}^*(2n)$-Higgs bundles are connected.

In Sect. 6 we invoke the non-abelian Hodge theory correspondence between the moduli space of $\mathrm {SO}^*(2n)$-Higgs bundles over $X$ and the moduli space of representations of the fundamental group of $X$ in $\mathrm {SO}^*(2n)$ to count the number of connected components of the latter in the zero and maximal Toledo invariant cases, and to give a rigidity result for maximal representations when $n$ is odd.

In Sect. 7 we examine some special features of $\mathrm {SO}^*(2n)$-Higgs bundles and their moduli spaces in the low rank cases, i.e. for $n=1,2,3$. These features are mostly reflections of special low rank isomorphisms between Lie groups, but they yield interesting relations between Higgs bundle moduli spaces.

Finally, in the “Appendix” we summarize salient features of $G$-Higgs bundles for the groups other than $\mathrm {SO}^*(2n)$ which come up in the discussion of the case $G=\mathrm {SO}^*(2n)$.

We conclude this introduction by pointing out that a number of works which appeared after the first version of this paper was posted on the arXiv exploit an interesting complementary approach to $G$-Higgs bundles via the Hitchin fibration. These include [25, 26, 30, 33]. The comparison between the two approaches is instructive and deserves further investigation.

2 $G$-Higgs bundles

The original notion of a $G$-Higgs bundle when $G$ is a real reductive Lie group can be traced back to [23, 24]. For the convenience of the reader we summarize the basic definitions and constructions. These have appeared at various levels of explicitness in several places including [6, 14, 19].

2.1 Moduli space of $G$-Higgs bundles

Let $G$ be a real reductive Lie group. By this we mean² that we are given the data $(G,H,\theta ,B)$, where $H \subset G$ is a maximal compact subgroup, $\theta :\mathfrak {g}\rightarrow \mathfrak {g}$ is a Cartan involution and $B$ is a non-degenerate bilinear form on $\mathfrak {g}$, which is ${{\mathrm{Ad}}}(G)$-invariant and $\theta$-invariant. The data $(G,H,\theta ,B)$ has to satisfy in addition that

• (1) the Lie algebra $\mathfrak {g}$ of $G$ is reductive,

• (2) $\theta$ gives a decomposition (the Cartan decomposition)

  $$\begin{aligned} \mathfrak {g} = \mathfrak {h}\oplus \mathfrak {m}\end{aligned}$$

  into its $\pm 1$-eigenspaces, where $\mathfrak {h}$ is the Lie algebras of $H$,

• (3) $\mathfrak {h}$ and $\mathfrak {m}$ are orthogonal under $B$, and $B$ is positive definite on $\mathfrak {m}$ and negative definite on $\mathfrak {h}$,

• (4) multiplication as a map from $H\times \exp \mathfrak {m}$ into $G$ is an onto diffeomorphism.

We will refer sometimes to the data $(G,H,\theta ,B)$ as the Cartan data.

The group $H$ acts linearly on $\mathfrak {m}$ through the adjoint representation of $G$. Complexifying, we get the isotropy representation $\iota :H^{\mathbb {C}}\rightarrow \mathrm {GL}(\mathfrak {m}^{\mathbb {C}}).$

Definition 2.1

A $G$ -Higgs bundle on $X$ is a pair $(E,\varphi )$, where $E$ is a holomorphic $H^{\mathbb {C}}$-principal bundle over $X$ and $\varphi$ is a holomorphic section of $E(\mathfrak {m}^{\mathbb {C}})\otimes K$, where $E(\mathfrak {m}^{\mathbb {C}})= E \times _{H^{\mathbb {C}}}\mathfrak {m}^{\mathbb {C}}$ is the $\mathfrak {m}^{\mathbb {C}}$-bundle associated to $E$ via the isotropy representation and $K$ is the canonical bundle of $X$. The section $\varphi$ is called the Higgs field. Two $G$-Higgs bundles $(E,\varphi )$ and $(E',\varphi ')$ are isomorphic if there is an isomorphism $f:E \xrightarrow {\simeq } E'$ such that $\varphi = f^*\varphi '$ where $f^*$ is the obvious induced map.

More generally, replacing $K$ by an arbitrary line bundle on $X$ in the preceding definition, we obtain the notion of a $L$ -twisted $G$ -Higgs pair on $X$.

Just as for vector bundles, there are notions of stability, semistability and polystability for $G$-Higgs bundles (and more generally for $L$-twisted Higgs pairs). In this paper we consider only the particular cases we need (cf. Sect. 3.2) and refer the reader to [15] for the general definitions.

Henceforth, we shall assume that $G$ is connected. Then the topological classification of $H^\mathbb {C}$-bundles $E$ on $X$ is given by a characteristic class

$$\begin{aligned} c(E) \in \pi _1(H^\mathbb {C})=\pi _1(H)=\pi _1(G). \end{aligned}$$

Definition 2.2

For a fixed $d \in \pi _1(G)$, the moduli space of polystable $G\!\!$ -Higgs bundles $\mathcal {M}_d(G)$ is the set of isomorphism classes of polystable $G$-Higgs bundles $(E,\varphi )$ such that $c(E)=d$.

The moduli space $\mathcal {M}_d(G)$ has the structure of a complex analytic variety. This can be seen by the standard slice method (see, e.g., Kobayashi [28]). Moreover, it is a consequence of the general constructions of Schmitt [34, 35] that for all the groups which appear in this paper the moduli space $\mathcal {M}_d(G)$ is actually algebraic.

2.2 The Hitchin equation

In general, i.e. for any real reductive group $G$, the Hitchin equations for a $G$-Higgs bundle, say $(E,\varphi )$, can be regarded as conditions for a reduction of the structure group of $E$. Recall that $E$ is a principal holomorphic $H^{\mathbb {C}}$-bundle, where $H^{\mathbb {C}}$ is the complexification of $H$ (a maximal compact subgroup of $G$). A reduction of structure group to $H$ defines a principal $H$-bundle, $E_H$, such that $E=E_H\times _HH^{\mathbb {C}}$. Then, together with the holomorphic structure on $E$, the reduction to $E_H$ defines a unique connection (the Chern connection) on $E$. We denote the curvature of this connection by $F_h$. Assume now that $G$ is a real form of its complexification $G^{\mathbb {C}}$, and let $\tau :\mathfrak {g}^{\mathbb {C}}\longrightarrow \mathfrak {g}^{\mathbb {C}}$ denote the involution which defines the compact real form of $G^{\mathbb {C}}$. The relation between $\tau$, the involution which defines the real form $G$, and the Cartan involution on $\mathfrak {g}$, ensures that the combination $[\mathfrak {m}^{\mathbb {C}},\tau (\mathfrak {m}^{\mathbb {C}})]$ takes values in $\mathfrak {h}$. Using the reduction $E(\mathfrak {g}^{\mathbb {C}})=E_H\times _H\mathfrak {g}^{\mathbb {C}}$ we can extend $\tau$ to a bundle map $\tau _h: E(\mathfrak {g}^{\mathbb {C}})\longrightarrow E(\mathfrak {g}^{\mathbb {C}})$. Combined with conjugation on the canonical bundle $K$ this defines a bundle map (also denoted by $\tau _h$) on $E(\mathfrak {g}^{\mathbb {C}})\otimes K$. Applying this map to the Higgs field $\varphi$ allows us to form a $\mathfrak {h}$-valued (1,1)-form $[\varphi ,\tau (\varphi )]$.

Definition 2.3

If $G$ is semisimple the $G$ -Hitchin equation for a reduction of structure group to $H$ of a $G$-Higgs bundle $(E,\varphi )$ is

$$\begin{aligned} F_h-[\varphi ,\tau _h(\varphi )]= 0 \end{aligned}$$

(2.1)

where $F_h$ and $\tau _h$ are as above.

The following result can be found in [15] (see also [8]).

Theorem 2.4

(Theorem 3.21 in [15]) Let $(E,\varphi )$ be a $G$-Higgs bundle. The bundle $E$ admits a reduction of structure group from $H^{\mathbb {C}}$ to $H$ satisfying the Hitchin equation for a $G$-Higgs bundle if and only if $(E,\varphi )$ is polystable.

2.3 Deformation theory of $G$-Higgs bundles

In this section we recall some standard facts about the deformation theory of $G$-Higgs bundles (see [14] and [15] for more detail). We also take care of issues that arise considering general reductive groups, rather than just semisimple ones. In particular we introduce a reduced deformation complex which is relevant in analyzing smoothness of the moduli space.

Definition 2.5

Let $(E,\varphi )$ be a $G$-Higgs bundle. Let $d\iota :\mathfrak {h}^{\mathbb {C}} \rightarrow {{\mathrm{End}}}(\mathfrak {m}^{\mathbb {C}})$ be the derivative at the identity of the complexified isotropy representation $\iota = {{\mathrm{Ad}}}_{|H^{\mathbb {C}}}:H^{\mathbb {C}} \rightarrow \mathrm{Aut }(\mathfrak {m}^{\mathbb {C}})$. The deformation complex of $(E,\varphi )$ is the following complex of sheaves:

$$\begin{aligned} C^{\bullet }(E,\varphi ):E(\mathfrak {h}^\mathbb {C}) \xrightarrow {d\iota (\varphi )} E(\mathfrak {m}^\mathbb {C})\otimes K. \end{aligned}$$

(2.2)

Proposition 2.6

The space of infinitesimal deformations of a $G$-Higgs bundle $(E,\varphi )$ is naturally isomorphic to the hypercohomology group $\mathbb {H}^1(C^{\bullet }(E,\varphi ))$. The Lie algebra of $\mathrm{Aut }(E,\varphi )$, denoted by $\mathrm{aut }(E,\varphi )$, can be identified with $\mathbb {H}^0(C^{\bullet }(E,\varphi ))$.

Next we introduce two concepts which are important for understanding smoothness of the moduli space (cf. Proposition 2.14 below).

Definition 2.7

A $G$-Higgs bundle $(E,\varphi )$ is called simple if $\mathrm {Aut}(E,\varphi )=Z(H^{\mathbb {C}})\cap \ker (\iota )$ where $Z(H^{\mathbb {C}})$ denotes the center. A $G$-Higgs bundle $(E,\varphi )$ is said to be infinitesimally simple if the infinitesimal automorphism space $\mathrm{aut }(E,\varphi )$ is isomorphic to $H^0(X,E(\ker d\iota \cap Z(\mathfrak {h}^{\mathbb {C}}))$ where $Z(\mathfrak {h}^{\mathbb {C}})$ denotes the Lie algebra of $Z(H^{\mathbb {C}})$.

Thus a $G$-Higgs bundle is (infinitesimally) simple if its (infinitesimal) automorphism group is as small as possible.

Remark 2.8

It is clear that a simple $G$-Higgs bundle is infinitesimally simple. If $G$ is complex then $\iota$ is the adjoint representation and $(E,\varphi )$ is simple (resp. infinitesimally simple) if $\mathrm{Aut }(E,\varphi )=Z(G)$ (resp. $\mathrm{aut }(E,\varphi )=Z(\mathfrak {h}^{\mathbb {C}})$).

Example 2.9

View a $\mathrm {GL}(n,\mathbb {C})$-Higgs bundle a Higgs vector bundle $(E,\Phi )$ with $\Phi \in H^0(X,{{\mathrm{End}}}(E)\otimes K)$. Then $(E,\Phi )$ is simple if its automorphism group is $\mathrm{Aut }(E,\Phi )=\mathbb {C}^*$ and infinitesimally simple if its infinitesimal automorphism space ${{\mathrm{End}}}(E,\Phi ) = \mathbb {C}$. In this case the two notions coincide, but this is not the case for all groups. Indeed, this phenomenon already occurs for principal bundles without a Higgs field: as an example, let $L$ be a line bundle of degree zero such that $L^2 \ne \mathcal {O}$. Then the $\mathrm {SO}(2,\mathbb {C})$-bundle $(V,Q)=\left( L \oplus L^{-1},\left( \begin{matrix} 0 &{} 1\\ 1 &{} 0 \end{matrix}\right) \right)$ has $\mathrm{Aut }(V,Q) = \{\pm 1\}$ and $\mathrm{aut }(V,Q) = 0$ so it is infinitesimally simple but not simple.

In order to study smoothness of the moduli space in the general case of reductive groups (i.e. for non-semisimple $G$), we introduce a reduced deformation complex.

Lemma 2.10

([27, p. 388]) Let $\mathfrak {z}$ be the center of $\mathfrak {g}$ and $\mathfrak {z}^{\mathbb {C}}$ be the center of $\mathfrak {g}^{\mathbb {C}}$. There are decompositions $\mathfrak {z}= (\mathfrak {h}\cap \mathfrak {z}) \oplus (\mathfrak {m}\cap \mathfrak {z})$ and $\mathfrak {z}^{\mathbb {C}} = (\mathfrak {h}^{\mathbb {C}} \cap \mathfrak {z}^{\mathbb {C}}) \oplus (\mathfrak {m}^{\mathbb {C}} \cap \mathfrak {z}^{\mathbb {C}}).$

In view of this Lemma, we can decompose as $H$-modules

$$\begin{aligned} \mathfrak {h}= (\mathfrak {h}\cap \mathfrak {z}) \oplus \mathfrak {h}_0, \qquad \mathfrak {m}= (\mathfrak {m}\cap \mathfrak {z}) \oplus \mathfrak {m}_0, \end{aligned}$$

where we have defined

$$\begin{aligned} \mathfrak {h}_0 = \mathfrak {h}/ (\mathfrak {h}\cap \mathfrak {z}), \qquad \mathfrak {m}_0 = \mathfrak {m}/ (\mathfrak {m}\cap \mathfrak {z}). \end{aligned}$$

Analogously we define $\mathfrak {h}_0^{\mathbb {C}}$ and $\mathfrak {m}_0^{\mathbb {C}}$ and we have similar decompositions of $\mathfrak {h}^{\mathbb {C}}$ and $\mathfrak {m}^{\mathbb {C}}$. Note also that

$$\begin{aligned}{}[\mathfrak {m}^{\mathbb {C}},\mathfrak {h}^{\mathbb {C}} \cap \mathfrak {z}^{\mathbb {C}}] = 0, \qquad [\mathfrak {m}^{\mathbb {C}}_0,\mathfrak {h}^{\mathbb {C}}_0] \subset \mathfrak {m}^{\mathbb {C}}_0. \end{aligned}$$

We can thus define the following reduced complex.

Definition 2.11

Let $(E,\varphi )$ be a $G$-Higgs bundle. The reduced deformation complex of $(E,\varphi )$ is the following complex of sheaves:

$$\begin{aligned} C^{\bullet }_0(E,\varphi ):E(\mathfrak {h}_0^\mathbb {C}) \xrightarrow {{{\mathrm{ad}}}(\varphi )} E(\mathfrak {m}_0^\mathbb {C})\otimes K. \end{aligned}$$

(2.3)

Remark 2.12

If $G$ is semisimple the reduced deformation complex (2.3) coincides with the non-reduced complex (2.2). If $G$ is a complex reductive group, then the reduced complex $C^{\bullet }_0(E,\varphi )$ can be identified with the (non-reduced) deformation complex for the $PG$-Higgs bundle associated to $(E,\varphi )$, where $PG=G/Z(G)$.

Let $(E,\varphi )$ be a $G$-Higgs bundle and assume that $G$ is a real form of a complex reductive group $G_{\mathbb {C}}$. Let

$$\begin{aligned} \tilde{E} = E \times _{H^{\mathbb {C}}}G^{\mathbb {C}} \end{aligned}$$

be the principal $G^{\mathbb {C}}$-bundle associated by extension of structure group. Note that

$$\begin{aligned} \tilde{E}(\mathfrak {g}^{\mathbb {C}}) = E(\mathfrak {g}^{\mathbb {C}}) = E(\mathfrak {h}^{\mathbb {C}}) \oplus E(\mathfrak {m}^{\mathbb {C}}). \end{aligned}$$

Hence we can let $\tilde{\varphi }$ be the image of $\varphi$ under the inclusion

$$\begin{aligned} H^0(X,E(\mathfrak {m}^{\mathbb {C}})\otimes K) \hookrightarrow H^0(X,\tilde{E}(\mathfrak {g}^{\mathbb {C}})\otimes K). \end{aligned}$$

Definition 2.13

The $G^{\mathbb {C}}$-Higgs bundle $(\tilde{E},\tilde{\varphi })$ is called the $G^{\mathbb {C}}$ -Higgs bundle associated to the $G$ -Higgs bundle $(E,\varphi )$.

Proposition 2.14

Let $(E,\varphi )$ be a $G$-Higgs bundle.

• (1) If $(E, \varphi )$ is stable and $\varphi \ne 0$ then it is infinitesimally simple.

• (2) If $(E,\varphi )$ is stable and simple and $\mathbb {H}^2(C^{\bullet }_0(E,\varphi ))=0$ then $(E,\varphi )$ represents a smooth point in the moduli space.

• (3) If $G$ is complex and $(E,\varphi )$ is stable and simple then $(E,\varphi )$ represents a smooth point in the moduli space.

• (4) Let $(\tilde{E},\tilde{\varphi })$ be the $G^{\mathbb {C}}$-Higgs bundle associated to $(E,\varphi )$. If $(E,\varphi )$ is stable then $(\tilde{E},\tilde{\varphi })$ is polystable. If $(E,\varphi )$ is stable, simple and stable as a $G^\mathbb {C}$-Higgs bundle then it represents a smooth point in the moduli space.

Proof

(1) See [15, Proposition 3.11].

(2) If $(E,\varphi )$ is simple, there are no singularities coming from automorphisms of the pair. Therefore the obstruction to smoothness lies in $\mathbb {H}^2(C^{\bullet }(E,\varphi ))$. Analyzing the Kuranishi model (as done in Kobayashi [28] in the case of vector bundles on higher dimensional manifolds, cf. also Friedman–Morgan [13, p. 301]), one sees that the image of the Kuranishi map in fact lies in the hypercohomolgy of the reduced deformation complex, i.e., in $\mathbb {H}^2(C^{\bullet }_0(E,\varphi ))=0$. The point is that the Kuranishi map is given by the quadratic part of the holomorphicity condition

$$\begin{aligned} 0 = \bar{\partial }_{A+\dot{A}}(\varphi +\dot{\varphi }) = \bar{\partial }_A\varphi + \bar{\partial }_A\dot{\varphi } + [\dot{A},\varphi ] + [\dot{A},\dot{\varphi }], \end{aligned}$$

which lies in $\Omega ^{0,1}E(\mathfrak {m}_0^\mathbb {C})$. This leads to the result. (An alternative method of proof would be to go through the proof of Theorem 3.1 of [4] and see that the vanishing of $\mathbb {H}^2(C^{\bullet }_0(E,\varphi ))=0$ is really what is required in this case.)

(3) By stability we have the vanishing $\mathbb {H}^0(C^{\bullet }_0(E,\varphi ))=0$ and Serre duality of complexes implies $\mathbb {H}^2(C^{\bullet }_0(E,\varphi ))=0$. The result now follows by (2).

(4) Stability of $(\tilde{E},\tilde{\varphi })$ implies that it is infinitesimally simple, i.e., $\mathbb {H}^0(C^{\bullet }(\tilde{E},\tilde{\varphi })) = Z(\mathfrak {g}^{\mathbb {C}})$, where

$$\begin{aligned} C^{\bullet }(\tilde{E},\tilde{\varphi }):\tilde{E}(\mathfrak {g}^{\mathbb {C}}) \xrightarrow {{{\mathrm{ad}}}(\tilde{\varphi })}\tilde{E}(\mathfrak {g}^{\mathbb {C}})\otimes K. \end{aligned}$$

It follows that $\mathbb {H}^0(C^{\bullet }_0(\tilde{E},\tilde{\varphi })) = 0.$ Moreover,

$$\begin{aligned} C^{\bullet }_0(\tilde{E},\tilde{\varphi }) = C^{\bullet }_0(E,\varphi ) \oplus C^{\bullet }_0(E,\varphi )^*\otimes K \end{aligned}$$

and hence, by Serre duality of complexes, we obtain the vanishing $\mathbb {H}^2(C^{\bullet }_0(E,\varphi ))=0.$ Again the result is now a consequence of (2). $\square$

3 $\mathrm {SO}^*(2n)$-Higgs bundles

3.1 Preliminaries: the group $\mathrm {SO}^*(2n)$

In this section we collect together some basic facts about the group $\mathrm {SO}^*(2n)$ (see [22] for more details). We concentrate on the features that are needed to describe $\mathrm {SO}^*(2n)$-Higgs bundles and to understand their relation to $G$-Higgs bundles for related groups such as $\mathrm {SL}(2n,\mathbb {C})$ and $\mathrm {SU}(n,n)$. The group $\mathrm {SO}^*(2n)$ may be defined as the the set of matrices $g\in \mathrm {SL}(2n,\mathbb {C})$ satisfying

$$\begin{aligned} g^tJ_n\bar{g}=J_n\quad \mathrm {and}\quad g^tg=I_{2n},\quad \mathrm {where}\quad J_n=\left( \begin{matrix} 0&{}I_n\\ -I_n&{}0 \end{matrix}\right) . \end{aligned}$$

(3.1)

It is thus a subgroup of $\mathrm {SO}(2n,\mathbb {C})$ which leaves invariant a skew-Hermitian form. The group is connected, semisimple, and a non-compact real form of $\mathrm {SO}(2n,\mathbb {C})$. The maximal compact subgroups are isomorphic to $\mathrm {U}(n)$. The choice $\Theta (g)=J_ngJ^{-1}_n$ of Cartan involution on $\mathrm {SO}^*(2n)$ gives the Cartan decomposition

$$\begin{aligned} \mathfrak {so}^*(2n)=\mathfrak {u}(n)+\mathfrak {m}\end{aligned}$$

with

$$\begin{aligned} \mathfrak {u}(n)&= \left\{ \left( \begin{matrix} X_1&{}\quad X_2\\ -X_2&{}\quad X_1 \end{matrix}\right) \ | \ X_1,X_2\in {{\mathrm{Mat}}}_{n,n}(\mathbb {R}), X_1^t+X_1=0, X_2^t-X_2=0\right\} ,\nonumber \\ \mathfrak {m}&= \left\{ i\left( \begin{matrix} Y_1&{}\quad Y_2\\ Y_2&{}\quad -Y_1 \end{matrix}\right) \ | \ Y_1,Y_2\in {{\mathrm{Mat}}}_{n,n}(\mathbb {R}), Y_1^t+Y_1=0, Y_2^t+Y_2=0\right\} . \end{aligned}$$

(3.2)

Remark 3.1

It follows immediately from (3.2) that

$$\begin{aligned} \mathfrak {u}(n)+i\mathfrak {m}=\left\{ \left( \begin{matrix} A&{}\quad B\\ -B^t&{}\quad D \end{matrix}\right) \ |\ A,B,D\in {{\mathrm{Mat}}}_{n,n}(\mathbb {C}),\ A+A^t=D+D^t=0\right\} , \end{aligned}$$

which can be identified with the Lie algebra of $\mathrm {SO}(2n)$. This shows that the real form $\mathrm {SO}^*(2n)$ is the non-compact dual to the compact real form $\mathrm {SO}(2n)\subset \mathrm {SO}(2n,\mathbb {C})$.

The complexification of the Cartan decomposition is

$$\begin{aligned} \mathfrak {so}^*(2n)\otimes \mathbb {C}=\mathfrak {gl}(n,\mathbb {C})+\mathfrak {m}^{\mathbb {C}}, \end{aligned}$$

(3.3)

where

$$\begin{aligned} \mathfrak {gl}(n,\mathbb {C})&= \left\{ \left( \begin{array}{l@{\quad }l} \frac{Z-Z^t}{2}&{}-\frac{Z+Z^t}{2i}\\ \frac{Z+Z^t}{2i}&{}\frac{Z-Z^t}{2} \end{array}\right) \ |\ Z \in {{\mathrm{Mat}}}_{n,n}(\mathbb {C})\right\} ,\nonumber \\ \mathfrak {m}^{\mathbb {C}}&= \left\{ \left( \begin{array}{l@{\quad }l} Y_1&{}Y_2\\ Y_2&{}-Y_1 \end{array}\right) \ | \ Y_1,Y_2\in {{\mathrm{Mat}}}_{n,n}(\mathbb {C}), Y_1^t+Y_1=0, Y_2^t+Y_2=0\right\} . \end{aligned}$$

(3.4)

It follows that if $T$ is the complex automorphism of $\mathbb {C}^{2n}$ defined by

$$\begin{aligned} T=\left( \begin{matrix} I&{}iI\\ I&{}-iI \end{matrix}\right) , \end{aligned}$$

(3.5)

then

$$\begin{aligned} T\mathfrak {gl}(n,\mathbb {C})T^{-1}&= \left\{ \left( \begin{matrix} Z&{}\quad 0\\ 0&{}\quad -Z^t \end{matrix}\right) \ | \ Z \in {{\mathrm{Mat}}}_{n,n}(\mathbb {C})\right\} ,\nonumber \\ T\mathfrak {m}^{\mathbb {C}}T^{-1}&= \left\{ \left( \begin{matrix} 0&{}\quad \beta \\ \gamma &{}\quad 0 \end{matrix}\right) \ | \ \beta ,\gamma \in {{\mathrm{Mat}}}_{n,n}(\mathbb {C}),\ \beta ^t+\beta =0,\ \gamma ^t+\gamma =0\right\} . \end{aligned}$$

(3.6)

This reflects the following fact.

Proposition 3.2

With $T$ defined in (3.5),

$$\begin{aligned} T\mathrm {SO}^*(2n) T^{-1}\subset \mathrm {SU}(n,n), \end{aligned}$$

(3.7)

where $\mathrm {SU}(n,n)\subset \mathrm {SL}(2n,\mathbb {C})$ is the subgroup defined by

$$\begin{aligned} \mathrm {SU}(n,n)=\{ A\in \mathrm {SL}(2n,\mathbb {C})\ | \ \bar{A}^tI_{n,n}A=I_{n,n},\ \mathrm {\det }(A)=1\} \end{aligned}$$

(3.8)

with $I_{n,n}=\left( \begin{matrix} I_n&{}\quad 0\\ 0&{}\quad -I_n \end{matrix}\right)$.

Proof

Using $\bar{T}^tI_{n,n}T=2iJ$ it follows that if $g\in \mathrm {SO}^*(2n)$ then $A=TgT^{-1}$ satisfies $\bar{A}^tI_{n,n}A=I_{n,n}$. Also $\det (A)=\det (g)=1$. $\square$

Remark 3.3

If $g\in \mathrm {SO}^*(2n)$, i.e. $g$ satisfies (3.1), and $A=TgT^{-1}$ then a simple calculation shows that $A^tI_{n,n}JA=I_{n,n}J$. Combined with Proposition 3.2 we can thus identify $\mathrm {SO}^*(2n)\subset \mathrm {SU}(n,n)\subset \mathrm {SL}(2n,\mathbb {C})$ as the subgroup defined by the relation $A^tI_{n,n}JA=I_{n,n}J$. This is the definition given in [27].

3.2 $\mathrm {SO}^*(2n)$-Higgs bundles and stability

When $H^\mathbb {C}$ is a classical group we prefer to work with the vector bundle $V$ associated to the standard representation rather than the $H^\mathbb {C}$-principal bundle. We take this point of view for $\mathrm {SO}^*(2n)$-Higgs bundles, for which $H^{\mathbb {C}}=\mathrm {GL}(n,\mathbb {C})$ and $V$ is a rank $n$ vector bundle. In view of (3.6), Definition 2.1 then becomes the following.

Definition 3.4

A $\mathrm {SO}^*(2n)$ -Higgs bundle over $X$ is a pair $(V,\varphi )$ in which $V$ is a rank $n$ holomorphic vector bundle over $X$, and the Higgs field $\varphi =(\beta ,\gamma )$ has components $\beta \in H^0(X,\Lambda ^2V\otimes K)$ and $\gamma \in H^0(X,\Lambda ^2V^*\otimes K)$. We will sometimes write $\varphi =\beta +\gamma$, where the sum is interpreted as being in ${{\mathrm{End}}}(V \oplus V^*)\otimes K$, viewing $\beta$ and $\gamma$ as skew-symmetric maps $\beta :V^*\rightarrow V\otimes K$ and $\gamma :V\rightarrow V^*\otimes L$. We will also sometimes use the notation $(V,\varphi )=(V,\beta ,\gamma )$.

In order to state the (semi,poly)stability condition for a $\mathrm {SO}^*(2n)$-Higgs bundle we need to introduce some notation.

Let $V \rightarrow X$ be a holomorphic vector bundle. Then there is an isomorphism $V \otimes V \simeq \Lambda ^2 V \oplus S^2 V$. Let $U$ and $W$ be subbundles of $V$. We define $U \otimes _{A} W$ to be the sheaf theoretic kernel of the projection $V \otimes V \rightarrow S^2V$ restricted to $U \otimes V$:

$$\begin{aligned} 0\rightarrow U \otimes _A W \rightarrow U \otimes W \rightarrow S^2 V. \end{aligned}$$

Since $U \otimes W$ is locally free and $X$ is a curve, $U \otimes _A W$ can be viewed as a subbundle of $\Lambda ^2 V$. We define $U^{\perp } \subset V^*$ to be the kernel of the restriction map $V^* \rightarrow U^*$, i.e.

$$\begin{aligned} 0 \rightarrow U^{\perp } \rightarrow V^* \rightarrow U^*\rightarrow 0. \end{aligned}$$

Definition 3.5

Let $k$ be an integer satisfying $k\ge 1$. We define a filtration of $V$ of length $k-1$ to be any strictly increasing filtration by holomorphic subbundles

$$\begin{aligned} {\fancyscript{V}}=(0 \subsetneq V_1\subsetneq V_2\subsetneq \dots \subsetneq V_k=V). \end{aligned}$$

Let $\lambda =(\lambda _1<\lambda _2 <\dots <\lambda _k)$ be a strictly increasing sequence of $k$ real numbers. Define the subbundle

$$\begin{aligned} N({\fancyscript{V}},\lambda )= \sum _{\lambda _i+\lambda _j\le 0} K\otimes V_i\otimes _A V_j \oplus \sum _{\lambda _i+\lambda _j\ge 0} K\otimes V_{i-1}^{\perp }\otimes _A V_{j-1}^{\perp } \subset K\otimes (\Lambda ^2V\oplus \Lambda ^2V^*). \end{aligned}$$

(3.9)

Define also

$$\begin{aligned} d({\fancyscript{V}},\lambda )= \lambda _k\deg V_k +\sum _{j=1}^{k-1}(\lambda _j-\lambda _{j+1})\deg V_j. \end{aligned}$$

(3.10)

We say that the pair $({\fancyscript{V}},\lambda )$ is trivial if the length of ${\fancyscript{V}}$ is $0$ and $\lambda _1=0$. We say that the pair $({\fancyscript{V}},\lambda )$ is $\varphi$-invariant if $\varphi =\beta +\gamma \in H^0(X,N({\fancyscript{V}},\lambda ))$.

The general results of [15] allow us to express stability, semistability and polystability for $\mathrm {SO}^*(2n)$-Higgs bundles in terms of filtrations, as follows.

Definition 3.6

The Higgs bundle $(V,\varphi )$ is semistable if for any integer $k\ge 1$, any filtration ${\fancyscript{V}}$ of length $k-1$ of $V$ and any strictly increasing sequence $\lambda$ of $k$ real numbers such that $({\fancyscript{V}},\lambda )$ is $\varphi$-invariant we have

$$\begin{aligned} d({\fancyscript{V}},\lambda )\ge 0. \end{aligned}$$

(3.11)

The Higgs bundle $(V,\varphi )$ is stable if under the same conditions as above with the additional condition that $({\fancyscript{V}},\lambda )$ be non-trivial we have the strict inequality

$$\begin{aligned} d({\fancyscript{V}},\lambda )> 0. \end{aligned}$$

(3.12)

The Higgs bundle $(V,\varphi )$ is polystable if it is semistable and for any integer $k\ge 1$, any filtration ${\fancyscript{V}}$ of length $k-1$ of $V$ and any strictly increasing sequence $\lambda$ of $k$ real numbers such that $({\fancyscript{V}},\varphi )$ is $\varphi$-invariant and $d({\fancyscript{V}},\lambda )=0$ there is an isomorphism of holomorphic bundles

$$\begin{aligned} V\simeq V_1\oplus V_2/V_1\oplus \cdots \oplus V_k/V_{k-1} \end{aligned}$$

with respect to which

$$\begin{aligned} \beta \in H^0\left( X,\bigoplus _{\lambda _i+\lambda _j=0}K\otimes V_i/V_{i-1}\otimes _A V_j/V_{j-1}\right) \end{aligned}$$

and

$$\begin{aligned} \gamma \in H^0\left( X,\bigoplus _{\lambda _i+\lambda _j=0}K\otimes (V_i/V_{i-1})^*\otimes _A (V_j/V_{j-1})^*\right) . \end{aligned}$$

We follow the convention that a direct sum of vector bundles over an empty indexing set is the zero vector bundle.

Remark 3.7

In general the notion of (semi,poly)stability depend on a real parameter related to the fact that the center of the maximal compact subgroup of $\mathrm {SO}^*(2n)$ is isomorphic to $\mathrm {U}(1)$ (see [14]). However, since our main interest is in relation to representations of the fundamental group, we have the value of this parameter to be zero.

Following the same arguments given in [15] for the group $\mathrm {Sp}(2n,\mathbb {R})$, the stability conditions for $\mathrm {SO}^*(2n)$-Higgs bundles can be simplified. Before we give the simplified conditions are given in Proposition 3.13 we need some preliminaries.

Definition 3.8

Let $(V,\varphi )$ be a $\mathrm {SO}^*(2n)$-Higgs bundle with $\varphi =(\beta ,\gamma )$. A filtration of subbundles

$$\begin{aligned} 0\subset V_1 \subset V_2 \subset V \end{aligned}$$

such that

$$\begin{aligned} \beta \in H^0(X,K\otimes (\Lambda ^2V_2 + V_1\otimes _{A} V)), \quad \gamma \in H^0(X,K\otimes (\Lambda ^2V_1^{\perp } + V_2^{\perp }\otimes _A V^*)), \end{aligned}$$

(3.13)

is called a $\varphi$ -invariant two-step filtration.

Remark 3.9

It is important to note that the summands in the bundles $\Lambda ^2V_2 + V_1\otimes _{A}V$ and $\Lambda ^2V_1^{\perp }+V_2^{\perp }\otimes _A V^*$ intersect non-trivially, so they do not form a direct sum.

Remark 3.10

We allow equality between the terms of the filtration in order to avoid having to consider separately filtrations that are length one or zero. For example the filtration $0\subset V_1\subset V$ is included as the two-step filtration in which $V_1=V_2$.

It is sometimes convenient to reformulate the $\varphi$-invariance condition using the following lemma, which is easily proved.

Lemma 3.11

Let $(V,\varphi )$ be a $\mathrm {SO}^*(2n)$-Higgs bundle with $\varphi =(\beta ,\gamma )$. A two-step filtration $0 \subset V_1 \subset V_2 \subset V$ is $\varphi$-invariant if and only if the following conditions are satisfied:

$$\begin{aligned} \beta (V_2^{\perp })&\subset V_1\otimes K,&\quad \gamma (V_2) \subset V_1^{\perp }\otimes K,\\ \beta (V_1^{\perp })&\subset V_2\otimes K,&\quad \gamma (V_1) \subset V_2^{\perp }\otimes K. \end{aligned}$$

There is yet another useful interpretation of the $\varphi$-invariance of a two step filtration that will be used later. To explain this, let $\Omega _\gamma :V\times V\rightarrow K$ be the $K$-twisted skew-symmetric bilinear pairing defined by $\gamma$ as

$$\begin{aligned} \Omega _{\gamma }(u, v):=(\gamma (v))(u), \quad \text{ for }\quad u,v\in V, \end{aligned}$$

and denote, for a subbundle $V'\subset V$,

$$\begin{aligned} V'^{\perp _\gamma }:=\{v\in V| \Omega _\gamma (u,v)=0\quad \text{ for } \text{ every }\quad u\in V'\}. \end{aligned}$$

The following lemma is immediate.

Lemma 3.12

For any filtration $0\subset V_1\subset V_2\subset V$, we have that $\gamma (V_1)\subset K\otimes V_2^\perp$ is equivalent to $V_1\subset V_2^{\perp _\gamma }$. This is equivalent to $V_2\subset V_1^{\perp _\gamma }$ which, in turn, is equivalent to $\gamma (V_2)\subset K\otimes V_1^\perp$. Similar statements apply to $\beta$.

The following simplified version of the stability conditions follows in the same way as the analogous results for $\mathrm {Sp}(2n,\mathbb {R})$-Higgs bundles (see [15]).

Proposition 3.13

A $\mathrm {SO}^*(2n)$-Higgs bundle $(V,\varphi )$ with $\varphi =(\beta ,\gamma )$ is semistable if and only for every $\varphi$-invariant two-step filtration $0 \subset V_1 \subset V_2 \subset V$ we have that

$$\begin{aligned} \deg (V) - \deg (V_1) - \deg (V_2)\ge 0. \end{aligned}$$

(3.14)

A $\mathrm {SO}^*(2n)$-Higgs bundle $(V,\varphi )$ with $\varphi =(\beta ,\gamma )$ is stable if and only if for every $\varphi$-invariant two-step filtration $0 \subset V_1 \subset V_2 \subset V$ except the filtration $0 =V_1 \subset V_2 = V$ we have that

$$\begin{aligned} \deg (V)-\deg (V_1) - \deg (V_2) > 0. \end{aligned}$$

(3.15)

A $\mathrm {SO}^*(2n)$-Higgs bundle $(V,\varphi )$ with $\varphi =(\beta ,\gamma )$ is polystable if is semistable and for any $\varphi$-invariant filtration $0 \subset V_1 \subset V_2 \subset V$, distinct from the filtration $0=V_1\subset V_2=V$ such that

$$\begin{aligned} \deg (V)-\deg (V_1) - \deg (V_2)=0, \end{aligned}$$

there exists an isomorphism of holomorphic vector bundles

$$\begin{aligned} V\simeq V_1\oplus V_2/V_1\oplus V/V_2 \end{aligned}$$

with respect to which we have:

• (a) $V_2\simeq V_1\oplus V_2/V_1$,

• (b) $\beta \in H^0(X,K\otimes (\Lambda ^2(V_2/V_1)\oplus V_1\otimes _A(V/V_2))$,

• (c) $\gamma \in H^0(X,K\otimes (\Lambda ^2(V_2/V_1)^*\oplus V_1^*\otimes _A (V/V_2)^*)$.

Remark 3.14

If $\beta =\gamma =0$ then the semistability condition is equivalent to the requirements that $\deg V =0$ and $V$ is semistable.

3.3 The $\mathrm {SO}^*(2n)$-Hitchin equations

Using the vector bundle picture, in which a $\mathrm {SO}^*(2n)$-Higgs bundle is specified by data $(V,\beta ,\gamma )$, we now make explicit the Hitchin equations in this case. A reduction of structure group to $H=\mathrm {U}(n)$ corresponds to a choice of Hermitian metric $h$ on the holomorphic bundle $V$. The Hitchin equations now become

$$\begin{aligned} F^h_V+\beta \beta ^*+\gamma ^*\gamma = 0. \end{aligned}$$

(3.16)

Here we denote the curvature for the Chern connection on $V$ by $F_V^h$ and the adjoints are with respect to the hermitian metric $h$ (combined with complex conjugation $dz\mapsto d\bar{z}$ on the form component).

We refer to Eq. (3.16) as the $\mathrm {SO}^*(2n)$ -Hitchin equation. Theorem 2.4 thus becomes the following.

Theorem 3.15

Let $(V,\beta ,\gamma )$ be a $\mathrm {SO}^*(2n)$-Higgs bundle. The bundle $V$ admits a metric satisfying the $\mathrm {SO}^*(2n)$-Hitchin equation (3.16) if and only if $(V,\beta ,\gamma )$ is polystable.

3.4 The moduli spaces

The topological invariant attached to a $\mathrm {SO}^*(2n,\mathbb {R})$-Higgs bundle $(V,\beta ,\gamma )$ is an element in the fundamental group of $\mathrm {U}(n)$ (see Sect. 2.1). Since $\pi _1(U(n))\simeq \mathbb {Z}$, this is an integer. This integer coincides with the degree of $V$. Under the correspondence between Higgs bundles and surface group representations (see Sect. 6), this integer corresponds to the Toledo invariant of a representation.³ Following Definition 2.2 we let $\mathcal {M}_d(\mathrm {SO}^*(2n))$ denote the moduli space of polystable $\mathrm {SO}^*(2n)$ -Higgs bundles $(V,\beta ,\gamma )$ with $\deg (V) = d$. For brevity we shall sometimes write simply $\mathcal {M}_d$ for this moduli space. We have the following result (cf. [15, Theorem 3.4 and Proposition 3.19]).

Proposition 3.16

Assume $n\ge 2$. The moduli space $\mathcal {M}_d$ of $\mathrm {SO}^*(2n)$-Higgs bundles over a compact Riemann surface $X$ of genus $g\ge 2$ is a complex algebraic variety of expected $n(2n-1)(g-1)$ (where $g$ is the genus of $X$). The dimension is exactly $n(2n-1)(g-1)$ if the stable locus is nonempty.

The reason for excluding $n=1$ in the preceding proposition is that $\mathrm {SO}^*(1)\simeq \mathrm {SO}(2)$ which is not semisimple. In this case the dimension of the moduli space is $g$ (cf. Sect. 7.1).

One has the following easily proven duality result.

Proposition 3.17

The map $(V,\beta ,\gamma )\mapsto (V^*,\gamma ,\beta )$ gives an isomorphism $\mathcal {M}_d\simeq \mathcal {M}_{-d}$.

3.5 Structure of stable $\mathrm {SO}^*(2n)$-Higgs bundles

The kernel of the isotropy representation

$$\begin{aligned} \iota :\mathrm {GL}(n,\mathbb {C}) \rightarrow \mathrm{Aut }(\Lambda ^2(\mathbb {C}^n) \oplus \Lambda ^2 (\mathbb {C}^n)^*) \end{aligned}$$

for $\mathrm {SO}^*(2n)$ is formed by the central subgroup $\{\pm I \} \subset \mathrm {GL}(n,\mathbb {C})$. Moreover the infinitesimal isotropy representation has injective differential: $\ker (d\iota ) =0$. Thus Definition 2.7 specializes to the following.

Definition 3.18

A $\mathrm {SO}^*(2n)$-Higgs bundle $(V,\beta ,\gamma )$ is simple if $\mathrm{Aut }(V,\beta ,\gamma ) = \{\pm I \}$ and it is infinitesimally simple if $\mathrm{aut }(V,\beta ,\gamma )=0$.

Contrary to the cases of vector bundles and $\mathrm {U}(p,q)$-Higgs bundles, stability of an $\mathrm {SO}^*(2n)$-Higgs bundle does not imply that it is simple. However, we have the following.

Theorem 3.19

Let $(V,\varphi )$ be a stable $\mathrm {SO}^*(2n)$-Higgs bundle. If $(V,\varphi )$ is not simple, then one of the following alternatives occurs:

• (1) The bundle $V$ is a stable vector bundle of degree zero and $\varphi =0$. In this case $\mathrm{Aut }(V,\varphi )\simeq \mathbb {C}^*$.

• (2) There is a nontrivial decomposition, unique up to reordering,

  $$\begin{aligned} (V,\varphi ) = \Bigl (\bigoplus _{i=1}^{k} V_i,\sum _{i=1}^{k} \varphi _i\Bigr ) \end{aligned}$$

  with $\varphi _i = \beta _i + \gamma _i \in H^0(X,K \otimes (\Lambda ^2V_i \oplus \Lambda ^2V_i^*))$, such that each $(V_i,\varphi _i)$ is a stable and simple $\mathrm {SO}^*(n_i)$-Higgs bundle. Furthermore, each $\varphi _i \ne 0$ and $(V_i,\varphi _i) \not \simeq (V_j,\varphi _j)$ for $i \ne j$. The automorphism group of $(V,\varphi )$ is

  $$\begin{aligned} \mathrm{Aut }(V,\varphi ) \simeq \mathrm{Aut }(V_1,\varphi _1) \times \dots \times \mathrm{Aut }(V_k,\varphi _k) \simeq (\mathbb {Z}/2)^k. \end{aligned}$$

Proof

The proof is precisely the same as for the corresponding result for $\mathrm {Sp}(2n,\mathbb {R})$-Higgs bundles (Theorem 3.17 in [14]). $\square$

In view of Theorem 3.19 we can shift our attention to $\mathrm {SO}^*(2n)$-Higgs bundles which are stable and simple. Unlike in the case of $G$-Higgs bundles for complex reductive $G$, the combination of stability and simplicity is not necessarily sufficient to guarantee smoothness in the moduli space. Our analysis involves the relation between $\mathrm {SO}^*(2n)$-Higgs bundles and $G$-Higgs bundles for various other⁴ groups $G$. We begin by noting that a $\mathrm {SO}^*(2n)$-Higgs bundle can be viewed as a Higgs bundle for the larger complex groups $\mathrm {SO}(2n,\mathbb {C})$ and $\mathrm {SL}(2n,\mathbb {C})$.

Theorem 3.20

Let $(V,\varphi )$ be a $\mathrm {SO}^*(2n)$-Higgs bundle with $\varphi =(\beta ,\gamma )$. Let $(E,\Phi )$ be the $\mathrm {SL}(2n,\mathbb {C})$-Higgs bundle given by

$$\begin{aligned} E = V \oplus V^*,\quad \Phi = \left( \begin{matrix} 0 &{}\quad \beta \\ \gamma &{}\quad 0 \end{matrix}\right) \end{aligned}$$

and let $((E,Q),\Phi )$ be the $\mathrm {SO}(2n,\mathbb {C})$-Higgs bundle given by $E$ and $\Phi$ as above and with $Q$ defined by

$$\begin{aligned} Q\bigl ((v,\xi ),(w,\zeta )\bigr ) = \xi (w) + \zeta (v),\quad \text {for} \quad v,w\in V \quad \hbox {and}\quad \xi ,\zeta \in V^*. \end{aligned}$$

Then

• (1) The following are equivalent:

  • (a) $(E,\Phi )$ is semistable (resp. polystable).

  • (b) $((E,Q),\Phi )$ is semistable (resp. polystable).

  • (c) $(V,\varphi )$is semistable (resp. polystable).

• (2) If $(E,\Phi )$ is stable then $((E,Q),\Phi )$ is stable.

• (3) If $((E,Q),\Phi )$is stable then $(V,\varphi )$ is stable.

• (4) If $(V,\varphi )$ is stable and simple then

  • (a) $(E,\Phi )$ is stable unless there is an isomorphism $f:V\xrightarrow {\simeq } V^*$ such that $\beta f=f^{-1}\gamma$;

  • (b) $((E,Q),\Phi )$ is stable unless there is an isomorphism $f:V\xrightarrow {\simeq } V^*$ which is skew-symmetric and with $\beta f=f^{-1}\gamma$.

Proof

The equivalences in (1) can be proved in exactly the same way as done for $\mathrm {Sp}(2n,\mathbb {R})$-Higgs bundles in [14] (see Theorems 3.26 and 3.27). Although the equivalence analogous to the equivalence between (a) and (b) is not explicitly stated in [14] in the case of semistability, it is implicit in the proof of the equivalence analogous to the equivalence between (a) and (c).

The implication in (2) follows directly from the stability conditions.

For the implication in (3) note that a $\varphi$-invariant two-step filtration $0 \subset V_1 \subset V_2 \subset V$ gives rise to an isotropic subbundle $V_1 \oplus V_2^{\perp }$ of $(E,Q)$ which, by Lemma 3.11, is $\Phi$-invariant. These are exactly the subbundles which enter the stability condition for $\mathrm {SO}(2,\mathbb {C})$-Higgs bundles (see Proposition 8.2). Note that $V_1 \oplus V_2^{\perp } \subset E$ is non-zero and proper if and only if the filtration $0 \subset V_1 \subset V_2 \subset V$ is distinct from the filtration $0 =V_1 \subset V_2 = V$. Moreover,

$$\begin{aligned} \deg (V_1 \oplus V_2^{\perp }) = \deg (V_1)+\deg (V_2)-\deg (V), \end{aligned}$$

so the stability conditions coincide.

The statements in (4) can be proved in the same way as the analogous result for $\mathrm {Sp}(2n,\mathbb {R})$-Higgs bundles (see Theorem 3.27 in [14]). $\square$

Remark 3.21

If $\deg V\ne 0$, then it follows from (3) of Theorem 3.20 that $(E,\Phi )$ (and hence $((E,Q),\Phi )$) is stable if $(V,\varphi )$ is stable and simple. Similarly, if the rank $n$ is odd, then $((E,Q),\Phi )$ is stable if $(V,\varphi )$ is stable and simple. On the other hand, in the situation described in Theorem 3.19(2), the $\mathrm {SO}(2n,\mathbb {C})$-Higgs bundle $((E,Q),\Phi )$ is not stable, because $V_i \oplus V_i^*\subset E=V \oplus V^*$ is an isotropic $\phi$-invariant subbundle of degree $0$).

Proposition 3.22

Let $(V,\varphi )$ be a $\mathrm {SO}^*(2n)$-Higgs bundle which is stable and simple and assume that there is no skewsymmetric isomorphism $f:V \xrightarrow {\simeq } V^*$ intertwining $\beta$ and $\gamma$ (i.e. such that $\gamma =(f\otimes 1_K)\circ \beta \circ f$). Then $(V,\varphi )$ represents a smooth point of the moduli space of polystable $\mathrm {SO}^*(2n)$-Higgs bundles. In particular, if $d=\deg V$ is not zero or $n$ is odd, then all stable and simple $\mathrm {SO}^*(2n)$-Higgs bundles represent smooth points of the moduli space $\mathcal {M}_d$.

Proof

By (3b) of Theorem 3.20 the $\mathrm {SO}(2n,\mathbb {C})$-Higgs bundle corresponding to $(V,\varphi )$ is stable and hence by (4) in Proposition 2.14 it represents a smooth point in $\mathcal {M}_d$. $\square$

It remains to analyze the case in which $(V,\varphi )$ is stable and simple but admits a skewsymmetric isomorphism $f:V \xrightarrow {\simeq } V^*$ intertwining $\beta$ and $\gamma$. By (3b) of Theorem 3.20 this is equivalent to the associated $\mathrm {SO}(2n,\mathbb {C})$-Higgs bundle being non-stable. Furthermore $d=\deg V=0$ and $n$ is even.

Proposition 3.23

Let $(V,\varphi )$ be a $\mathrm {SO}^*(2n)$-Higgs bundle with $\phi =(\beta ,\gamma )$ which admits a skewsymmetric isomorphism $f:V \xrightarrow {\simeq } V^*$ such that $\beta f = f^{-1}\gamma$. Then with $\psi : = \beta f$, the data $((V,f),\psi )$ defines a $\mathrm {U}^*(n)$-Higgs bundle (as defined in Sect. 8.2.2).

Let $(V,\varphi )$ be stable. Then $((V,f),\psi )$ is stable. Assume moreover that $(V,\varphi )$ is simple. Then $((V,f),\psi )$ is stable and simple and the corresponding $\mathrm {GL}(n,\mathbb {C})$-Higgs bundle $(V,\psi )$ is stable. Hence $((V,f),\psi )$ represents a smooth point in the moduli space of $\mathrm {U}^*(n)$-Higgs bundles.

Proof

The fact that $((V,f),\psi )$ defines a $\mathrm {U}^*(n)$-Higgs bundles follows directly from the definition given in Sect. 8.2.2. The argument to prove the stability result is similar to the one given in the proof of Theorem 3.22 in [14]. The statement about simplicity follows directly from the fact that for both $\mathrm {SO}^*(2n)$- and $\mathrm {U}^*(n)$-Higgs bundles simplicity means that the only automorphisms are $\pm$ Identity. $\square$

Notation

We shall, somewhat imprecisely, say that a $\mathrm {SO}^*(2n)$-Higgs bundle of the form described in Proposition 3.23 is a $\mathrm {U}^*(n)$ -Higgs bundle.

3.6 Structure of polystable $\mathrm {SO}^*(2n)$-Higgs bundles

A general structure theorem for polystable $G$-Higgs bundles was given in [15], where it is shown that any strictly polystable $G$-Higgs bundle admits a reduction to a stable $G'$-Higgs bundle for a uniquely determined reductive subgroup $G' \subset G$. Here we give an elementary argument in the case $G=\mathrm {SO}^*(2n)$, identifying explicitly this reduction, without recourse to Lie theory. Our result is the following.

Proposition 3.24

A $\mathrm {SO}^*(2n)$-Higgs bundle $(V,\varphi )$ with $\varphi =(\beta ,\gamma )$ is polystable if and only if there are decompositions

$$\begin{aligned} V&= V_1 \oplus \dots \oplus V_k, \\ \varphi&= \varphi _1 + \dots +\varphi _k, \end{aligned}$$

such that each $(V_i,\varphi _i)$ is a $\mathrm {SO}^*(2n_i)$-Higgs bundle i.e. $\varphi _i=(\beta _i,\gamma _i)$ with $\beta _i \in H^0(X,\Lambda ^2V_i \otimes K)$ and $\gamma _i \in H^0(X,\Lambda ^2V^*_i \otimes K)$, and is of one of the following mutually exclusive types:

• (1) a stable $\mathrm {SO}^*(2n_i)$-Higgs bundle with $\varphi \ne 0$;

• (2) $V_i = \tilde{V}_i \oplus \tilde{W}^*_i$, with respect to this decomposition $\beta _i= \left( \begin{matrix} 0 &{}\quad \tilde{\beta }_i \\ -\tilde{\beta }_i^t &{}\quad 0 \end{matrix}\right)$ and $\gamma _i= \left( \begin{matrix} 0 &{}\quad -\tilde{\gamma }_i^t \\ \tilde{\gamma }_i&{}\quad 0 \end{matrix}\right)$ where $\tilde{\beta }_i \in H^0(X,{{\mathrm{Hom}}}(\tilde{W}_i,\tilde{V}_i)\otimes K)$ and $\tilde{\gamma }_i \in H^0(X,{{\mathrm{Hom}}}(\tilde{V}_i,\tilde{W}_i)\otimes K)$, and $(\tilde{V}_i,\tilde{W}_i,\tilde{\beta }_i,\tilde{\gamma }_i)$ is a stable $\mathrm {U}(p_i,q_i)$-Higgs bundle in which $p_iq_i\ne 0$, $\deg \tilde{V}_i +\deg \tilde{W}_i = 0$ and at least one of $\tilde{\beta }_i, \tilde{\gamma }_i$ is non-zero.

• (3) $\varphi _i=0$ and $V_i$ is a degree zero stable vector bundle.

Proof

Suppose $(V,\beta ,\gamma )$ is polystable. If it is stable then the result is trivially true (with $k=1$). Suppose that $(V,\beta ,\gamma )$ is not stable. Then by Definition 3.6 we can find a non trivial filtration (i.e. with $l \ge 2$) ${\fancyscript{V}}=(0\subsetneq V'_1\subsetneq V'_2\subsetneq \dots \subsetneq V'_l=V)$ and a sequence of weights $\lambda =(\lambda _1<\lambda _2<\dots <\lambda _l)$ such that

• $\varphi \in H^0(X,N({\fancyscript{V}},\lambda ))$

• $d({\fancyscript{V}},\lambda )=0$

• there is a splitting of vector bundles

  $$\begin{aligned} V\simeq V'_1\oplus V'_2/V'_1\oplus \dots \oplus V'_l/V'_{l-1} \end{aligned}$$

  with respect to which

  $$\begin{aligned} \beta \in H^0\left( X,\bigoplus _{\lambda _i+\lambda _j=0}K\otimes V'_i/V'_{i-1}\otimes _A V'_j/V'_{j-1}\right) \end{aligned}$$

  and

  $$\begin{aligned} \gamma \in H^0\left( X,\bigoplus _{\lambda _i+\lambda _j=0}K\otimes (V'_i/V'_{i-1})^*\otimes _A (V'_j/V'_{j-1})^*\right) . \end{aligned}$$

We can write the set of weights as a disjoint union

$$\begin{aligned} \{\lambda _1,\dots ,\lambda _l\} = I_1 \cup I_2 \cup I_3, \end{aligned}$$

where each of the sets, if non-empty, can be written as follows:

$$\begin{aligned} I_1&= \{0\},\\ I_2&= \{\mu _1,-\mu _1,\dots ,\mu _{r},-\mu _{r}\},\\ I_3&= \{\eta _1,\dots ,\eta _{s}\}, \end{aligned}$$

where $\mu _i >0$ and $\eta _i \ne 0$ for all $i$, and $|\eta _i| \ne |\eta _j|$ for $i \ne j$. In other words, $I_2$ contains pairs of non-zero weights $\pm \mu _i$ and $I_3$ contains non-zero weights that cannot be paired. Note that $I_2 \cup I_3 \ne \emptyset$ since at least one weight is non-zero.

We can now rewrite the splitting of $V$ as

$$\begin{aligned} V \simeq U_0 \oplus (U_{-\mu _1} \oplus U_{\mu _1}) \oplus \dots \oplus (U_{-\mu _r} \oplus U_{\mu _r}) \oplus U_{\eta _1}\oplus \dots \oplus U_{\eta _s}, \end{aligned}$$

(3.17)

where $U_{\nu } = V'_i / V'_{i-1}$ if $\nu =\lambda _i$ for some $i=1,\dots ,l$ and zero otherwise.

If $I_1$ is not empty, let $\beta _{0}$ be the component of $\beta$ in $H^0(X,K\otimes U_0\otimes _A U_0)$ and similarly define $\gamma _0$. If both $\beta _0=0$ and $\gamma _0=0$ then the vector bundle $U_{0}$ is a $\mathrm {U}(n_0)$-Higgs bundle. Otherwise, $(U_0,\beta _{0},\gamma _{0})$ defines an $\mathrm {SO}^*(2n_{0})$-Higgs bundle, where $n_{0} = {{\mathrm{rk}}}(U_0)$.

For each positive element $\mu _i \in I_2$, let $\tilde{\beta }_{i}$ be the component of $\beta$ in $H^0(X,K\otimes U_{\mu _i}\otimes _A U_{-\mu _i})$ and similarly define $\tilde{\gamma }_i$. If both $\tilde{\beta }_i=0$ and $\tilde{\gamma }_i=0$ then the vector bundles $U_{\mu _i}$ and $U_{-\mu _i}$ are $\mathrm {U}(p_i)$- and $\mathrm {U}(q_i)$-Higgs bundles respectively, where $p_i={{\mathrm{rk}}}(U_{\mu _i})$ and $q_i={{\mathrm{rk}}}(U_{-\mu _i})$. Otherwise,

$$\begin{aligned} (\tilde{V}_i,\tilde{W}_i,\tilde{\beta }_i,\tilde{\gamma }_i)) =(U_{\mu _i} \oplus U_{-\mu _i}^*,\tilde{\beta }_i,\tilde{\gamma }_i) \end{aligned}$$

defines a $\mathrm {U}(p_i,q_i)$-Higgs bundle, where $p_i={{\mathrm{rk}}}(U_{\mu _i})$ and $q_i={{\mathrm{rk}}}(U_{-\mu _i})$. In order to see that $\deg \tilde{V}_i +\deg \tilde{W}_i = 0$, we note that we can write the decomposition (3.17) as

$$\begin{aligned} V = U_{-\mu _i} \oplus V' \oplus U_{\mu _i}, \end{aligned}$$

where we have pulled out $U_{-\mu _i}$ and $U_{\mu _i}$ and we denote the rest by $V'$. Now consider the induced filtration ${\fancyscript{V}}'$ of $V$ with the weights $\lambda '=(-1<0<1)$. Clearly $\varphi \in H^0(X,N({\fancyscript{V}}',\lambda '))$. Hence semistability implies that

$$\begin{aligned} d({\fancyscript{V}}',\lambda ') = \deg (U_{\mu _i}) - \deg (U_{-\mu _i}) \ge 0. \end{aligned}$$

Similarly, considering the filtration induced by $V = U_{\mu _i} \oplus V' \oplus U_{-\mu _i}$ with weights $(-1<0<1)$ we obtain $\deg (U_{-\mu _i}) - \deg (U_{\mu _i}) \ge 0$, and hence we conclude that

$$\begin{aligned} \deg \tilde{V}_i +\deg \tilde{W}_i = \deg (U_{\mu _i}) - \deg (U_{-\mu _i}) = 0 \end{aligned}$$

Finally, for each $\eta _i \in I_3$, the vector bundle $U_{\eta _i}$ is a $\mathrm {U}(n_i)$-Higgs bundle and we see that $\deg (U_{\eta _i})=0$ by a similar argument, using the decomposition $V = U_{\eta _i} \oplus V'$.

Altogether, this leads to a decomposition with summands of the type in the statement of the Proposition. Now we show that each summand is polystable as a $G$-Higgs bundle, where $G$ is the appropriate group, i.e, $G=\mathrm {SO}^*(2n_0)$, $G=\mathrm {U}(p_i,q_i)$ or $G=\mathrm {U}(n_i)$. By Proposition 8.5, it follows that the $\mathrm {U}(p_i,q_i)$ and $\mathrm {U}(n_i)$ summands are direct sums of stable ones. Suppose one of the $\mathrm {SO}^*(2n_0)$ summands is not polystable. Then there is a filtration and weight system violating polystability of this summand. This filtration and weight system can be extended by adding the remaining summands in $V$ to each term and by taking the same weights. The resulting filtration and weight system violates polystability for the original $\mathrm {SO}^*(2n)$-Higgs bundle $(V,\varphi )$. Moreover, $n_0 < n$ because $I_2 \cup I_3 \ne \emptyset$. Hence we can iterate the procedure until all summands are stable.

Finally, we show that the three types are mutually exclusive. The conditions on $\varphi$ clearly make (1) and (3) mutually exclusive. Suppose that $(V_i,\beta _i,\gamma _i)$ is of type (2). Since it is stable, it must have $\varphi _i\ne 0$ and hence cannot be of type (3). Suppose that $(V_i,\beta _i,\gamma _i)$ is also stable as a $\mathrm {SO}^*(2n)$-Higgs bundle. Then it is infinitesimally simple and thus $\mathrm{aut }(V_i,\beta _i,\gamma _i)=0$. But if $(V_i,\beta _i,\gamma _i)$ is of type (2) then $\mathbb {C}^*\subset \mathrm{aut }(V_i,\beta _i,\gamma _i)$. Thus cases (1) and (2) are mutually exclusive. $\square$

Notation

We shall write $(V,\varphi ) = (V,\varphi _1) \oplus \dots \oplus (V,\varphi _k)$ for a $\mathrm {SO}^*(2n)$-Higgs bundle of the kind described in Proposition 3.24. Moreover, somewhat imprecisely, we shall say that a $\mathrm {SO}^*(2n)$-Higgs bundle of the form described in (2) of Proposition 3.24 is a $\mathrm {U}(p,q)$ -Higgs bundle (here $n=p+q$).

By Theorem 3.19 and Propositions 3.22 and 3.23, case (1) in Proposition 3.24 divides further into two cases. The resulting refinement, given in the next theorem, will be essential for proving our connectedness results in Sect. 5.

Theorem 3.25

A $\mathrm {SO}^*(2n)$-Higgs bundles $(V,\varphi =\beta +\gamma )$ is polystable if and only if there is a decomposition $(V,\varphi ) = (V_1,\varphi _1) \oplus \dots \oplus (V_k,\varphi _k)$ such that each $(V_i,\varphi _i)$ is a $\mathrm {SO}^*(2n_i)$-Higgs bundle of one of the following mutually exclusive types:

• (1) $(V_i,\varphi _i)$ is a stable and simple $\mathrm {SO}^*(2n_i)$-Higgs bundle with $\varphi _i\ne 0$ which is stable as an $\mathrm {SO}(2n_i,\mathbb {C})$-Higgs bundle;

• (2) $(V_i,\varphi _i)$ is a stable and simple $\mathrm {SO}^*(2n_i)$-Higgs bundle with $\varphi _i\ne 0$ which admits a skewsymmetric isomorphism as in Proposition 3.23 and thus defines a stable $\mathrm {U}^*(n_i)$-Higgs bundle;

• (3) $(V_i,\varphi _i)$ is as described in (2) of Proposition 3.24) and thus defines a stable $\mathrm {U}(p_i,q_i)$-Higgs bundle where $p_iq_i\ne 0$, $\deg \tilde{V}_i +\deg \tilde{W}_i = 0$ and $\varphi _i\ne 0$;

• (4) $\varphi _i=0$ and $V_i$ defines a degree zero stable vector bundle.

3.7 Bounds on $d=\deg (V)$.

In this section we give an inequality which bounds the number of non-empty moduli spaces $\mathcal {M}_d=\mathcal {M}_d(\mathrm {SO}^*(2n))$. The inequality corresponds to the Milnor-Wood inequality for surface group representations into $\mathrm {SO}^*(2n)$ (see Sect. 6).

Proposition 3.26

Let $(V,\beta ,\gamma )$ be a semistable $\mathrm {SO}^*(2n)$-Higgs bundle. Then

$$\begin{aligned} {{\mathrm{rank}}}(\beta )(1-g)\le \deg (V)\le {{\mathrm{rank}}}(\gamma )(g-1). \end{aligned}$$

(3.18)

In particular,

$$\begin{aligned} |\deg (V)| \le n(g-1) \end{aligned}$$

(3.19)

where $\deg (V)=n(g-1)$ if and only if $\gamma$ is an isomorphism, and $\deg (V)=-n(g-1)$ if and only if $\beta$ is an isomorphism.

Proof

This is proved by first using the equivalence between the semistability of $(V,\beta ,\gamma )$ and the $\mathrm {SL}(2n,\mathbb {C})$-Higgs bundle $(W,\Phi )$ associated to it (see (1) in Theorem 3.20), and then applying the semistability numerical criterion to special Higgs subbundles defined by the kernel and image of $\Phi$ (see Section 3.4 in [5], and also [18]). $\square$

Notice that since $\beta$ and $\gamma$ are skew-symmetric, they cannot be isomorphisms if $n$ is odd. If $n=2m+1$ then $2m$ is the upper bound on ${{\mathrm{rank}}}(\beta )$ and ${{\mathrm{rank}}}(\gamma )$. Denote by $\left\lfloor \frac{n}{2}\right\rfloor$ the integer part of $\frac{n}{2}$. As a corollary of Proposition 3.26, we obtain the following.

Proposition 3.27

The moduli space $\mathcal {M}_d$ is empty unless

$$\begin{aligned} |d| \le \left\lfloor \frac{n}{2}\right\rfloor (2g-2). \end{aligned}$$

(3.20)

In view of this result, we say that $d = \deg (V)$ is maximal when equality holds in (3.20).

4 The case of maximal $d$

4.1 Cayley correspondence for $n=2m$

In this section we will assume that $n=2m$ is even and we will describe the $\mathrm {SO}^*(2n)$ moduli space for the extreme value $|d| = 2m(g-1)$. In fact, for the rest of this section we shall assume that $d = 2m(g-1)$. This involves no loss of generality, since, by Proposition 3.17 there is an isomorphism between the moduli spaces for $d$ and $-d$. The main result is Theorem 4.3, which we refer to as the Cayley correspondence.

Let $(V,\beta ,\gamma )$ be a $\mathrm {SO}^*(4m)$-Higgs bundle such that $\gamma \in H^0(X,K\otimes \Lambda ^2V^*)$ is an isomorphism. Let $L_0=K^{-1/2}$ be a fixed square root of $K^{-1}$, and define $W:=V\otimes L_0$. Then $\omega :=\gamma \otimes I_{L_0}: W \rightarrow W^*$ is a skew-symmetric isomorphism defining a non-degenerate symplectic $\Omega$ on $W$, in other words, $(W,\Omega )$ is a $\mathrm {Sp}(2m,\mathbb {C})$-holomorphic bundle. The $K^2$-twisted endomorphism $\psi :W\rightarrow W \otimes K^2$ defined by $\psi : = \beta \otimes I_{L_0^{-1}}\circ (\gamma \otimes I_{L_0})$ is $\Omega$-skewsymmetric and hence $(W,\Omega ,\psi )$ defines a $K^2$-twisted $\mathrm {U}^*(2m)$-Higgs pair (in the sense of Sect. 8.2.2, suitably modified to incorporate a twisting by an arbitrary line bundle), from which we can recover the original $\mathrm {SO}^*(4m)$-Higgs bundle.

Definition 4.1

With $(V,\beta ,\gamma )$ and $(W,\Omega ,\psi )$ as above, we say that $(W,\Omega ,\psi )$ is the Cayley partner to $(V,\beta ,\gamma )$.

Theorem 4.2

Let $(V,\beta ,\gamma )$ be a $\mathrm {SO}^*(4m)$-Higgs bundle with $d=2m(g-1)$ such that $\gamma$ is an isomorphism. Let $(W,\Omega ,\psi )$ be the corresponding $K^2$-twisted $\mathrm {U}^*(n)$-Higgs pair. Then $(V,\beta ,\gamma )$ is semistable (resp. stable, polystable) if and only if $(W,\Omega ,\psi )$ is semistable (resp. stable, polystable).

Proof

The proof is similar to that of Theorem 4.2 in [14], so we will just sketch the main arguments. We will used the simplified stability notions given in Propositions 3.13 and 8.7. We first show that if $(V,\beta ,\gamma )$ is semistable then the corresponding $\mathrm {U}^*(2m)$-Higgs pair is semistable. Suppose otherwise, then there exists an isotropic $\psi$-invariant subbundle $W'\subset W$ such that $\deg W'>0$. Let $V_1:=W'\otimes L_0^{-1}$ and let $V_2= V_1^{\perp _\gamma }$ (see Lemma 3.12 for the definition of $\perp _\gamma$). We can check that the filtration $0 \subset V_1 \subset V_2 \subset V$ is $\varphi$-invariant and $\deg (V) - \deg (V_1) - \deg (V_2) < 0$, contradicting the semistability of $(V,\beta ,\gamma )$.

To prove the converse, i.e., that $(V,\beta ,\gamma )$ is semistable if the corresponding $\mathrm {U}^*(2m)$-Higgs pair is semistable, suppose that there is a $\varphi$-invariant filtration $0\subset V_1\subset V_2\subset V$ such that $\deg (V) - \deg (V_1) - \deg (V_2) < 0$. From this filtration we cannot immediately obtain a destabilizing isotropic subbundle of the $\mathrm {U}^*(2m)$-Higgs pair, but we can construct an appropriate filtration giving the destabilizing subobject of the $\mathrm {U}^*(2m)$-Higgs pair. To do this, we first observe that the $\varphi$-invariance condition for $\gamma$ (second condition in (3.13)) is equivalent, by Lemma 3.12, to $V_2\subset V_1^{\perp _{\gamma }}$. We define two new filtrations as follows:

$$\begin{aligned} (0\subset V_1'\subset V_2'\subset V):=(0\subset V_1\subset V_1^{\perp _{\gamma }}\subset V) \end{aligned}$$

(we indeed have $V_1\subset V_1^{\perp _{\gamma }}$ because $V_1\subset V_2$ and $V_2\subset V_1^{\perp _{\gamma }}$) and

$$\begin{aligned} (0\subset V_1''\subset V_2''\subset V):=(0\subset V_2\cap V_2^{\perp _{\gamma }} \subset V_2+V_2^{\perp _{\gamma }}\subset V). \end{aligned}$$

One can check (see Theorem 4.2 in [14]) that these two filtrations are $\varphi$-invariant and that one of the two inequalities

$$\begin{aligned} \deg V-\deg V_1-\deg V_1^{\perp _{\gamma }}<0,\qquad \qquad \deg V-\deg (V_2\cap V_2^{\perp _{\gamma }})-\deg (V_2+V_2^{\perp _{\gamma }})<0 \end{aligned}$$

holds. These two filtrations give $\psi$-invariant isotropic subbundles $W':=V_1'\otimes L_0$ and $W'':=V_1''\otimes L_0$ such that either $\deg W'>0$ or $\deg W''>0$, contradicting the semistability of $(W,\Omega ,\psi )$.

The proof of the statement for stability is basically the same, observing that the trivial filtration $0=V_1\subset V_2=V$ corresponds to the trivial subbundle $0\subset W$. The proof of the equivalence of the polystability conditions follows word by word the argument for $\mathrm {Sp}(2n,\mathbb {R})$ given in Theorem 4.2 in [14]. $\square$

Theorem 4.3

Let $\mathcal {M}_{\max }(\mathrm {SO}^*(4m))$ be the moduli space of polystable $\mathrm {SO}^*(4m)$-Higgs bundles with $d=2m(g-1)$ and let $\mathcal {M}_{K^2}(\mathrm {U}^*(2m))$ be the moduli space of polystable $K^2$-twisted $\mathrm {U}^*(2m)$-Higgs pairs. The map $(V,\beta ,\gamma )\mapsto (W,\Omega ,\psi )$ defines an isomorphism of complex algebraic varieties

$$\begin{aligned} \mathcal {M}_{{\mathrm {max}}}(\mathrm {SO}^*(4m))\simeq \mathcal {M}_{K^2}(\mathrm {U}^*(2m)). \end{aligned}$$

Proof

Let $(V,\beta ,\gamma )$ be a semistable $\mathrm {SO}^*(4m)$-Higgs bundle with $d=2m(g-1)$. By Proposition 3.26, $\gamma$ is an isomorphism and hence the map $(V,\beta ,\gamma )\mapsto (W,\Omega ,\psi )$ is well defined. The result follows now from Theorem 4.2 and the existence of local universal families (see [35]). $\square$

Remark 4.4

Note that a maximal $\mathrm {SO}^*(2n)$-Higgs bundle $(V,\beta ,\gamma )$ has $\beta =0$ if and only if the Cayley partner $(W,\Omega ,\psi )$ has $\psi =0$. Thus, in particular, Theorem 4.2 implies that a maximal $\mathrm {SO}^*(2n)$-Higgs bundle of the form $(V,0,\gamma )$ is polystable if and only if the corresponding $\mathrm {Sp}(n,\mathbb {C})$-bundle $(W,\Omega )$ is polystable. Hence, the isomorphism of Theorem 4.3 restricts to an isomorphism between the subspace of $\mathrm {SO}^*(2n)$-Higgs bundles with $\beta =0$ in $\mathcal {M}_{\max }(\mathrm {SO}^*(2n))$ and the moduli space of polystable $\mathrm {Sp}(n,\mathbb {C})$-bundles (note that there is only one topological class of such bundles, since $\mathrm {Sp}(n,\mathbb {C})$ is simply connected.) This will be important in the proof of Theorem 5.2.

4.2 Rigidity for $n=2m+1$

In this section we consider the case in which $n=2m+1$ and describe the $\mathrm {SO}^*(2n)$ moduli space for the extreme value $|d| = 2m(g-1)$. As in Sect. 4.1, we assume without loss of generality that $d$ is positive. The main result is the following theorem.⁵

Theorem 4.5

Let $\mathcal {M}_{\max } (\mathrm {SO}^*(4m+2))$ be the moduli space of polystable $\mathrm {SO}^*(2n)$-Higgs bundles with $n=2m+1$ and $d=2m(g-1)$. If $m>0$ and $g\ge 2$ then the stable locus of $\mathcal {M}_{\max } (\mathrm {SO}^*(4m+2))$ is empty and

$$\begin{aligned} \mathcal {M}_{\max } (\mathrm {SO}^*(4m+2))\simeq \mathcal {M}_{\max }(\mathrm {SO}^*(4m))\times {{\mathrm{Jac}}}(X), \end{aligned}$$

where ${{\mathrm{Jac}}}(X)$ is the Jacobian of $X$.

Proof

Let $(V,\beta ,\gamma )$ be a polystable $\mathrm {SO}^*(2n)$-Higgs bundle with $n=2m+1$. The map $\gamma :V\longrightarrow V^*\otimes K$ defines kernel and image sheaves:

$$\begin{aligned} 0\longrightarrow \ker (\gamma )\longrightarrow V\longrightarrow {{\mathrm{im}}}(\gamma )\longrightarrow 0. \end{aligned}$$

(4.1)

The kernel $\ker (\gamma )$ is a subbundle of $V$, while ${{\mathrm{im}}}(\gamma )$ is in general a subsheaf of $V^*\otimes K$. Let $W_{\gamma }$ denote the saturation of ${{\mathrm{im}}}(\gamma )\otimes K^{-1}\subset V^*$, so that we have

$$\begin{aligned} 0\longrightarrow {{\mathrm{im}}}(\gamma )\otimes K^{-1}\longrightarrow W_{\gamma }\longrightarrow T\longrightarrow 0, \end{aligned}$$

(4.2)

where $T$ is a torsion sheaf.

Let $\ker (\gamma )^{\perp }$ denote the annihilator of $\ker (\gamma )$, i.e. let it be defined by

$$\begin{aligned} 0\longrightarrow \ker (\gamma )^{\perp }\longrightarrow V^*\longrightarrow \ker (\gamma )^*\longrightarrow 0 \end{aligned}$$

(4.3)

The skew-symmetry of $\gamma$ implies the following:

$$\begin{aligned}&\ker (\gamma )^{\perp }=W_{\gamma }, \end{aligned}$$

(4.4)

$$\begin{aligned}&{{\mathrm{rank}}}(\gamma )\le 2m \end{aligned}$$

(4.5)

Combining (4.4) with (4.3), we get

$$\begin{aligned} \deg (\ker (\gamma ))-\deg (W_{\gamma })= d \end{aligned}$$

(4.6)

In addition, we get linear relations from (4.1) and (4.2), namely

$$\begin{aligned} \deg (\ker (\gamma ))+\deg ({{\mathrm{im}}}(\gamma ))=d \end{aligned}$$

(4.7)

and

$$\begin{aligned} \deg ({{\mathrm{im}}}(\gamma ))-\deg (W_{\gamma })=l(2g-2)-t \end{aligned}$$

(4.8)

where $t=\deg (T)$ and $l={{\mathrm{rank}}}(\gamma )$. The system (4.6), (4.7), (4.8) can be solved, giving in particular

$$\begin{aligned} \deg (\ker (\gamma ))=d+\deg (W_{\gamma })=d-l(g-1)+\frac{t}{2}. \end{aligned}$$

(4.9)

Consider now the subobject $V\oplus W_{\gamma }\subset V\oplus V^*$. This clearly satisfies

• (1) $W_{\gamma }^{\perp }\subset V$,

• (2) $\beta (W_{\gamma })\subset V\otimes K$,

• (3) $\gamma (V)\subset W_{\gamma }\otimes K$.

Thus, setting $V_1=W_{\gamma }^{\perp }$ and $V_2=V$, we get a filtration which is $\varphi$-invariant, i.e. satisfies condition (3.13) in Definition 3.8. The semistability condition thus yields the inequality $\deg (W_{\gamma }^{\perp })\le 0$ or, equivalently,

$$\begin{aligned} d+\deg (W_{\gamma })\le 0. \end{aligned}$$

(4.10)

Combined with (4.9) this gives

$$\begin{aligned} d-l(g-1)+\frac{t}{2}\le 0. \end{aligned}$$

(4.11)

It follows immediately from (4.11) and (4.9)—and the non-negativity of $t$—that if $d=2m(g-1)=l(g-1)$ then $T=0$, i.e. ${{\mathrm{im}}}(\gamma )\otimes K^{-1}$ is a subbundle of $V^*$, and $\deg (\ker (\gamma ))=0$.

By Theorem 2.4 the $\mathrm {SO}^*(2n)$-Higgs bundle $(V,\beta ,\gamma )$ is polystable if and only if $V$ admits a Hermitian metric $h$ satisfying the $\mathrm {SO}^*(2n)$-Hitchin equations. As described in Sect. 3.3, these equations take the form

$$\begin{aligned} F_V+\beta \beta ^*+\gamma ^*\gamma =0 \end{aligned}$$

(4.12)

where $F_V$ is the curvature of the metric connection determined by $h$, and the adjoints $\beta ^*$ and $\gamma ^*$ are with respect to $h$. Fix a local frame for $V$ and take the dual frame for $V^*$. With respect to these frames, $\beta$ and $\gamma$ are represented by a skew-symmetric matrices. If the frame for $V$ is compatible with the smooth decomposition $V=\ker (\gamma )\oplus V_{\perp }$, where $V_{\perp }$ denotes the complement to $\ker (\gamma )$, then the matrices have the form

$$\begin{aligned} \gamma =\left( \begin{matrix} 0&{}\quad 0\\ 0&{}\quad \gamma \end{matrix}\right) \ ,\ \beta =\left( \begin{matrix} \beta 1&{}\quad \beta _2\\ -\beta _2&{}\quad \beta _3 \end{matrix}\right) \end{aligned}$$

(4.13)

with respect to the decompositions $V=\ker (\gamma )\oplus V_{\perp }$ and $V^*=(\ker (\gamma ))^*\oplus (V_{\perp })^*$.

The metric connection decomposes as

$$\begin{aligned} D_{V}=\left( \begin{matrix} D_{\ker }&{}\quad A\\ -\bar{A}^T&{}\quad D_{\perp } \end{matrix}\right) \end{aligned}$$

(4.14)

where $A\in \Omega ^{0,1}({{\mathrm{Hom}}}(V_{\perp },\ker (\gamma )))$ is the second fundamental form for the embedding of the subbundle $\ker (\gamma )\subset V$. The corresponding decomposition of the curvature is

$$\begin{aligned} F_{V}=\left( \begin{matrix} F_{\ker }-A\wedge \bar{A}^T&{}\quad *\\ *&{}\quad F_{V_{\perp }}-\bar{A}^T\wedge A \end{matrix}\right) \ . \end{aligned}$$

(4.15)

Applying $i\Lambda \mathrm{Tr }$ to Eq. (4.12), and using (4.13) thus yields

$$\begin{aligned}&\displaystyle \deg (\ker (\gamma ))+\Pi +||\beta _1||^2+||\beta _2||^2=0\end{aligned}$$

(4.16)

$$\begin{aligned}&\displaystyle \deg (V_{\perp })-\Pi +||\beta _{2}||^2+||\beta _3||^2-||\gamma ||^2=0 \end{aligned}$$

(4.17)

where $\Pi =-i\Lambda \mathrm{Tr }(A\wedge \bar{A}^T)$. Notice that, since the second fundamental form is of type $(0,1)$, we get that

$$\begin{aligned} \Pi \ge 0\ . \end{aligned}$$

(4.18)

But if $d=2m(g-1)$ and ${{\mathrm{rank}}}(\gamma )=2m$ then $\deg (\ker (\gamma ))=0$. It thus follows from (4.16) that $\Pi =0$ and also that $\beta _1=\beta _2=0$. This immediately implies that the $\mathrm {SO}^*(2n)$-Higgs bundle $(V,\beta ,\gamma )$ decomposes as a sum

$$\begin{aligned} (V,\beta ,\gamma )=(\ker (\gamma ),0,0)\oplus (V_{\perp },\beta _3,\gamma ). \end{aligned}$$

(4.19)

Notice that with $V_1=0$ and $V_2= V_{\perp }$ we get a $\varphi$-invariant two-step filtration (see definition 3.8) with

$$\begin{aligned} \deg (V)-\deg (V_1)-\deg (V_2) = 0 \end{aligned}$$

(4.20)

By Proposition 3.13 $(V,\beta ,\gamma )$ is thus not stable. Moreover, $\ker (\gamma )$ is a holomorphic line bundle, while $(V_{\perp },\beta _3,\gamma )$ is a $\mathrm {SO}^*(4m)$-Higgs bundle. The data thus define a Higgs bundle with structure group

$$\begin{aligned} \mathrm {SO}^*(4m) \times \mathrm {SO}(2) = \mathrm {SO}^*(4m) \times \mathrm {U}(1)\ . \end{aligned}$$

This completes the proof of Theorem 4.5. $\square$

Remark 4.6

It follows from Theorem 4.5 that $\mathcal {M}_{\max } (\mathrm {SO}^*(4m+2))$ has dimension $2m(2m-1)(g-1)+g$. Comparing with the expected dimension given in Proposition 3.16 we see that $\dim (\mathcal {M}_{\max } (\mathrm {SO}^*(4m+2)))$ is smaller than expected if $g\ge 2$ and $m>0$. This explains why we refer to Theorem 4.5 as a rigidity result.

5 Connected components of the moduli space

5.1 The Hitchin functional and connected components of the moduli space

The method we shall use for studying the topology of the moduli space goes back to Hitchin [23]. In the following, we very briefly outline the general aspects of this approach, applied to the count of connected components (more details can be found in, for instance, [5, 6, 14, 24]). We then apply this programme (in Theorem 5.2 below) to show that $\mathcal {M}_d$ is connected for $d=0$ and the maximal value of $|d|$ (where $\mathcal {M}_d=\mathcal {M}_d(\mathrm {SO}^*(2n)$, as in Sect. 3.4).

The method rests on the gauge theoretic interpretation of the moduli space (provided by Theorem 2.4) as the moduli space of solutions to the Hitchin equations (2.1). Given defining data for a $\mathrm {SO}^*(2n)$-Higgs bundle, namely $(V,\beta ,\gamma )$, the solution to the equations is a Hermitian metric on the vector bundle $V$.

Thus it makes sense to define the Hitchin function

$$\begin{aligned} f:\mathcal {M}_d&\rightarrow \mathbb {R}\nonumber \\ (V,\beta , \gamma )&\mapsto ||\beta ||^2 + ||\gamma ||^2 \end{aligned}$$

(5.1)

where the $L^2$-norms of $\beta$ and $\gamma$ are computed using the metric which satisfies the Hitchin equation. The function $f$ is proper and therefore attains a minimum on each connected component of $\mathcal {M}_d$. Hence, if the subspace of local minima of $f$ restricted to $\mathcal {M}_{d}$ can be shown to be connected, then it will follow that $\mathcal {M}_{d}$ itself is connected.

Theorem 5.1

Let $(V,\beta ,\gamma )$ be a poly-stable $\mathrm {SO}^*(2n)$-Higgs bundle.

• (1) If $d > 0$, then $(V,\beta ,\gamma )$ represents a local minimum on $\mathcal {M}_d$ if and only if $\beta =0$.

• (2) If $d < 0$, then $(V,\beta ,\gamma )$ represents a local minimum on $\mathcal {M}_d$ if and only if $\gamma =0$.

• (3) If $d = 0$, then $(V,\beta ,\gamma )$ represents a local minimum on $\mathcal {M}_d$ if and only if $\beta =0$ and $\gamma =0$.

Before giving the proof of this result (at the end of Sect. 5.2 below), we apply it to prove our main theorem on the connectedness of $\mathcal {M}_{0}$ and $\mathcal {M}_{\max }$.

Theorem 5.2

The moduli space $\mathcal {M}_d$ is non-empty⁶ and connected if $d=0$ or $|d|=\left\lfloor \frac{n}{2}\right\rfloor (2g-2)$.

Proof

Consider first the case $d=0$. From (3) of Theorem 5.1 it is immediate that the subspace of local minima of the Hitchin function on $\mathcal {M}_0$ consists of polystable $\mathrm {SO}^*(n)$-Higgs bundles $(V,\beta ,\gamma )$ with $\beta =\gamma =0$. Furthermore, we conclude from Theorem 3.25 that such an $\mathrm {SO}^*(2n)$-Higgs bundle is polystable if and only if $V$ is a polystable vector bundle. Therefore, the subspace of local minima of the Hitchin function on $\mathcal {M}_0$ can be identified with the moduli space of polystable vector bundles of degree zero, which is known to be connected. This completes the proof of the case $d=0$.

Next we turn to the case $|d|=\lfloor \frac{n}{2}\rfloor (2g-2)$, i.e., the proof of connectedness of $\mathcal {M}_{\max }$. By Proposition 3.17 we may assume, without loss of generality, that $d$ is positive. From (1) of Theorem 5.1, we have that the subspace of local minima of the Hitchin function on $\mathcal {M}_{\max }$ can be identified with the subspace of $(V,\beta ,\gamma )$ with $\beta =0$. Suppose now that $n$ is even. Then, using Remark 4.4, we have that this subspace is isomorphic to the moduli space of polystable $\mathrm {Sp}(n,\mathbb {C})$-bundles. This space is connected by Ramanathan [31, Proposition 4.2] and hence $\mathcal {M}_{\max }$ is connected when $n$ is even. The connectedness of $\mathcal {M}_{\max }$ for odd $n$ now follows from the rigidity result of Theorem 4.5 and the connectedness of $\mathcal {M}_{\max }$ for even $n$.

Finally, non-emptiness of the moduli spaces follows from the non-emptiness of the subspaces of local minima of the Hitchin functional, which in turn follows from the identifications given in the course of the present proof. $\square$

5.2 Minima of the Hitchin functional

The purpose of this section is to prove Theorem 5.1. For this we need to show various preliminary results and, using these, we give the proof of the Theorem at the end of the section.

The following result is completely analogous to [5, Proposition 4.5].

Proposition 5.3

The absolute minimum of the Hitchin functional restricted to $\mathcal {M}_d$ is $|d|$. This minimal value is attained at a point represented by $(V,\beta ,\gamma )$ (with $\deg (V)=d$) if and only if $\beta =0$ (if $d\ge 0$) or $\gamma =0$ (if $d\le 0$).

Proof

Using the Hitchin equation and Chern–Weil theory we get that

$$\begin{aligned} d+||\beta ||^2 - ||\gamma ||^2=0 \end{aligned}$$

(5.2)

and hence the Hitchin function can be expressed as

$$\begin{aligned} f(V,\beta ,\gamma )={\left\{ \begin{array}{ll}d+2||\beta ||^2\\ -d+ 2||\gamma ||^2 \end{array}\right. } \end{aligned}$$

(5.3)

The result follows immediately from (5.3). $\square$

Of course not all local minima are necessarily absolute minima. We thus need to examine more closely the structure of the local minima.

On the smooth locus of $\mathcal {M}_d$, the Hitchin functional $f$ arises as the moment map of the $S^1$-action given by multiplication of the Higgs field $\varphi$ by complex numbers of modulus one. Considering the moduli space from the algebraic or holomorphic point of view, this action extends to the $\mathbb {C}^*$-action given by $(V,\phi ) \mapsto (V,w \phi )$ for $w \in \mathbb {C}^*$. The moment map interpretation shows that, on the smooth locus of $\mathcal {M}_d(\mathrm {SO}^*(2n))$, the critical points of $f$ are exactly the fixed points of the $\mathbb {C}^*$-action. On the full moduli space, the fixed point locus of the $\mathbb {C}^*$-action coincides with the locus of Hodge bundles (this can be easily seen by arguments like the ones used in [23, 24, 36]), which are defined as follows.

Definition 5.4

A $\mathrm {SO}^*(2n)$-Higgs bundle $(V,\beta ,\gamma )$ is called a Hodge bundle if

• there is a decomposition of $V$ into holomorphic subbundles

  $$\begin{aligned} V= \bigoplus _{i}F_{i} \end{aligned}$$

  (5.4)

  and, with respect to this decomposition,

• $\beta : F^*_{-i}\longrightarrow F_{i+1}\otimes K$, and $\gamma : F_{i}\longrightarrow F^*_{-i+1}\otimes K$

Here $F^*_i\subset V^*$ is the dual of $F_i$.

The weight of $F_{i}$ is $i$ and the weight of $F^*_{i}$ is $-i$.

Thus, in view of (4) of Proposition 2.14, we have the following characterization of the critical points of $f$.

Proposition 5.5

A simple $\mathrm {SO}^*(2n)$-Higgs bundle, which is stable as an $\mathrm {SO}(2n,\mathbb {C})$-Higgs bundle, represents a critical point of $f$ if and only if it is a Hodge bundle.

The deformation complex (2.2) for a $\mathrm {SO}^*(2n)$-Higgs bundle $(E,\varphi )$ is

$$\begin{aligned} C^\bullet (V,\varphi ):{{\mathrm{End}}}(V)&\xrightarrow {{{\mathrm{ad}}}(\varphi )} \Lambda ^2 V \otimes K \oplus \Lambda ^2V^* \otimes K\ .\nonumber \\ \psi&\mapsto (-\beta \psi ^t-\psi \beta ,\gamma \psi +\psi ^t\gamma ). \end{aligned}$$

(5.5)

If $(V,\varphi )$ is a Hodge bundle, then the decomposition (5.4) of $V$ induces corresponding weight decompositions

$$\begin{aligned} {{\mathrm{End}}}(V) = \bigoplus U_k^+\quad \text {and}\quad \Lambda ^2 V\oplus \Lambda ^2 V^* = \bigoplus U_k^- \end{aligned}$$

where

$$\begin{aligned} U^+_k=\bigoplus _{j-i=k}F^*_i\otimes F_j,\quad \text {and}\quad U^-_k=\bigoplus _{i+j=k}F_i\otimes _A F_j \oplus \bigoplus _{i+j=-k}F^*_i\otimes _A F^*_j. \end{aligned}$$

(5.6)

Moreover, since the Higgs field $\phi$ has weight one, the deformation complex (5.5) decomposes accordingly as

$$\begin{aligned} C^{\bullet }(V,\phi ) = \bigoplus _k C^{\bullet }_k(V,\phi ), \end{aligned}$$

where we let $C^{\bullet }_k(V,\phi ):U^+_k \xrightarrow {{{\mathrm{ad}}}(\phi )} U^-_{k+1} \otimes K.$ If we write $C^{\bullet }_{-}(V,\phi ) = \bigoplus _{k>0}C^{\bullet }_k(V,\phi )$ we then have the corresponding positive weight subspace

$$\begin{aligned} \mathbb {H}^1(C^{\bullet }_{-}(V,\phi )) \subset \mathbb {H}^1(C^{\bullet }(V,\phi )) \end{aligned}$$

of the infinitesimal deformation space. When $(V,\phi )$ represents a smooth point of the moduli space, the hypercohomology $\mathbb {H}^1(C^{\bullet }_{-}(V,\phi ))$ is the negative eigenvalue subspace of the Hessian of $f$ and so $(V,\phi )$ is a local minimum of $f$ if and only if $\mathbb {H}^1(C^{\bullet }_{-}(V,\phi )) = 0$.

The key result we need for identifying the minima of $f$ on the smooth locus of the moduli space is the following ([6, Corollary 5.8]).

Proposition 5.6

Assume that $(V,\phi )$ is a $\mathrm {SO}^*(2n)$-Higgs bundle which is stable as a $\mathrm {SO}(2n,\mathbb {C})$-Higgs bundle. Then $(V,\phi )$ represents a local minimum of $f$ in $\mathcal {M}_d$ if and only if it is a Hodge bundle and

$$\begin{aligned} {{\mathrm{ad}}}(\phi ):U^+_k \longrightarrow U^-_{k+1}\otimes K \end{aligned}$$

is an isomorphism for all $k>0$.

Using this result, we can prove the following lemma.

Lemma 5.7

Let $(V,\beta ,\gamma )$ be a simple $\mathrm {SO}^*(2n)$-Higgs bundle which is stable as a $\mathrm {SO}(2n,\mathbb {C})$-Higgs bundle and assume that $(V,\beta ,\gamma )$ represents a local minimum of $f$ on $\mathcal {M}_d$. Then, if $d = \deg (V) \ge 0$ the vanishing $\beta =0$ holds and, if $d = \deg (V) \le 0$ the vanishing $\gamma =0$ holds.

Proof

Let $(V,\beta ,\gamma )=(V,\phi )$ be a minimum. Then Proposition 5.5 implies that $(V,\beta ,\gamma )$ is a Hodge bundle. Moreover, arguing as in [14, Section 6], we see that $(V,\beta ,\gamma )$ being simple implies the following: there is a decomposition of $V$ into $2p+1$ non-zero holomorphic subbundles (for some $p \in \frac{1}{2}\mathbb {Z}$), which is either of the form:

$$\begin{aligned} \begin{aligned}&V=F_{-p+\frac{1}{2}}\oplus F_{-p+2+\frac{1}{2}}\oplus \cdots \oplus F_{p-2+\frac{1}{2}}\oplus F_{p+\frac{1}{2}}, \\&\beta :F^*_{p-2j+\frac{1}{2}}\longrightarrow F_{-p+2j+\frac{1}{2}}\otimes K,\quad \text {for} \quad 0\le j\le p, \text {and}\\&\gamma :F_{-p+2j+\frac{1}{2}}\longrightarrow F^*_{p-2(j+1)+\frac{1}{2}}\otimes K,\quad \text {for} \quad 0\le j\le p. \end{aligned} \end{aligned}$$

(5.7)

or of the form

$$\begin{aligned}&V=F_{-p-\frac{1}{2}}\oplus F_{-p+2-\frac{1}{2}}\oplus \cdots \oplus F_{p-2-\frac{1}{2}}\oplus F_{p-\frac{1}{2}},\nonumber \\&\beta :F^*_{p-2j-\frac{1}{2}}\longrightarrow F_{-p+2j-\frac{1}{2}}\otimes K,\quad \text {for}\quad 0\le j\le p, \text {and}\nonumber \\&\gamma :F_{-p+2j-\frac{1}{2}}\longrightarrow F^*_{p-2(j+1)-\frac{1}{2}}\otimes K,\quad \text {for} \quad 0\le j\le p. \end{aligned}$$

(5.8)

Let $k_0$ be the largest index such that $U^+_{k_0}\ne 0$. Since otherwise there is nothing to prove, we may assume that $k_0 >0$. For definiteness, assume that the decomposition of $V$ is of the form (5.7)—a similar argument applies when $V$ is of the form (5.8). Using (5.6), we see that $k_0=2p$ and thus (by Proposition 5.6) we have an isomorphism

$$\begin{aligned} {{\mathrm{ad}}}(\varphi ):F^*_{-p+\frac{1}{2}}\otimes F_{p+\frac{1}{2}}\longrightarrow \Lambda ^2 F_{p+\frac{1}{2}}\otimes K. \end{aligned}$$

(5.9)

In this case, since $\gamma =0$ on $F_{p+\frac{1}{2}}$, the map ${{\mathrm{ad}}}(\phi )$ is given explicitly by

$$\begin{aligned} x\mapsto \phi \circ x-x\circ \phi = -x\circ \beta , \end{aligned}$$

where

$$\begin{aligned} \beta :F_{p+\frac{1}{2}}^* \rightarrow F_{-p+\frac{1}{2}} \end{aligned}$$

(5.10)

for a local section $x:F^*_{-p+\frac{1}{2}}\longrightarrow F_{p+\frac{1}{2}}$. Denote the ranks of $F_{p+\frac{1}{2}}$ and $F_{-p+\frac{1}{2}}$ by $a$ and $b$ respectively. Then (5.9) implies that $ab=\frac{a(a-1)}{2}$ and hence that

$$\begin{aligned} a=2b+1>b. \end{aligned}$$

(5.11)

But then the map $\beta$ in (5.10) must have a non-trivial kernel and, therefore, the map

$$\begin{aligned} -x\circ \beta : F^*_{p+\frac{1}{2}}\longrightarrow F_{-p+\frac{1}{2}}\longrightarrow F_{p+\frac{1}{2}} \end{aligned}$$

vanishes on $\ker (\beta )$ for any local section $x$. Now, (5.11) implies that

$$\begin{aligned} a={{\mathrm{rk}}}(F_{p+\frac{1}{2}}) \ge 2. \end{aligned}$$

Hence there are non-zero antisymmetric local sections $y$ of $\Lambda ^2 F_{p+\frac{1}{2}}\otimes K$ which do not vanish on the kernel of $\beta$. This is in contradiction with the existence of the isomorphism (5.9). $\square$

In order to show that certain singular points of the moduli space are not minima, we need the following lemma (cf. Hitchin [24, §8]).

Lemma 5.8

Let $(V,\varphi )$ be a polystable $\mathrm {SO}^*(2n)$-Higgs bundle which is a Hodge bundle. Suppose there is a family $(V_t,\varphi _t)$ of polystable $\mathrm {SO}^*(2n)$-Higgs bundles, parametrized by $t$ in the open unit disk $D \subset \mathbb {C}$, such that $(V_0,\varphi _0) = (V,\varphi )$ and the corresponding infinitesimal deformation is a non-zero element of $\mathbb {H}^1(C^\bullet _-(V,\varphi ))$. Then $(V,\varphi )$ is not a local minimum of $f$ on $\mathcal {M}_d$.

Using this criterion and Theorem 3.25, we can now extend the result of Lemma 5.7 to cover all polystable $\mathrm {SO}^*(2n)$-Higgs bundles.

Lemma 5.9

Let $(V,\beta ,\gamma )$ be a polystable $\mathrm {SO}^*(2n)$-Higgs bundle and assume that $(V,\beta ,\gamma )$ represents a local minimum of $f$ on $\mathcal {M}_d$. Then, if $d = \deg (V) \ge 0$ the vanishing $\beta =0$ holds and, if $d = \deg (V) \le 0$ the vanishing $\gamma =0$ holds.

Proof

Let $(V,\varphi ) = (V_1,\varphi _1) \oplus \dots \oplus (V_k,\varphi _k)$ be the decomposition given in Theorem 3.25. As observed by Hitchin [24], the Hitchin function (5.1) is additive in the sense that

$$\begin{aligned} f(V,\varphi ) =\sum _{i=1}^k f (V_i,\varphi _i). \end{aligned}$$

It follows that each summand $(V_i,\varphi _i)$ represents a local minimum for the Hitchin functional on its own moduli space.

If a summand $(V_i,\varphi _i)$ is of type (1) in Theorem 3.25, then Lemma 5.7 shows that $\beta _i=0$ or $\gamma _i=0$. Similarly, if a summand $(V_i,\varphi _i)$ is of type (3), then it is shown in [5, Theorem 4.6] that $\beta _i=0$ or $\gamma _i=0$. With regard to summands of type (2), it is shown in [16, Proposition 4.6] that a stable $\mathrm {U}^*(n_i)$-Higgs bundle $(V_i,\varphi _i)$ representing a local minimum on the corresponding moduli space has $\varphi _i=0$. Finally we note that the summands $(V_i,\varphi _i)$ of type (4) have $\varphi _i=0$.

Thus each of the summands $(V_i,\varphi _i)$ of type (1) or (3) has either $\beta _i=0$ or $\gamma _i=0$ and each of the summands of type (2) or (4) has $\varphi _i=0$.

To complete the proof, assume that there are summands $(V',\beta ',\gamma ')$ and $(V'',\beta '',\gamma '')$ with $\beta '=0$, $\gamma '\ne 0$, $\beta ''\ne 0$ and $\gamma ''=0$, and that each of these summands is either of type (1) or of type (3). If we can construct a family $(V_t,\varphi _t)$ of polystable $\mathrm {SO}^*(2n)$-Higgs bundles such that

$$\begin{aligned} (V_0,\varphi _0)=(V',\beta '+\gamma ') \oplus (V'',\beta ''+\gamma '') \end{aligned}$$

and satisfying the hypothesis of Lemma 5.8, this proposition guarantees that $(V',\beta '+\gamma ') \oplus (V'',\beta ''+\gamma '')$ is not a minimum (on its own moduli space) and hence $(V,\varphi )$ cannot be a minimum. In the analogous case of $\mathrm {Sp}(2n,\mathbb {R})$-Higgs bundles, such a family is constructed in Lemmas 7.2 and 7.3 of [14]. Inspection of the proofs of these two lemmas shows that they are not sensitive to the symmetry properties of $\beta$ and $\gamma$ and so go through unchanged in the present case of $\mathrm {SO}^*(2n)$-Higgs bundles. This completes the proof. $\square$

Finally we are in a position to prove Theorem 5.1.

Proof of Theorem 5.1

The “if” part is immediate from Proposition 5.3. In the case $|d|= \lfloor \frac{n}{2}\rfloor (2g-2)$, the “only if” part follows from Lemma 5.9. In the case $d=0$ the result follows from the observation that if one of the Higgs fields $\beta$ and $\gamma$ vanishes, then polystability of $(V,\beta ,\gamma )$ forces the other Higgs field to vanish. $\square$

6 Representations of $\pi _1(X)$ in $\mathrm {SO}^*(2n)$

Let $X$ be a compact Riemann surface of genus $g$ and let

$$\begin{aligned} \pi _{1}(X) = \left\langle a_{1},b_{1}, \ldots , a_{g},b_{g} \;\;|\;\;\prod _{i=1}^{g}[a_{i},b_{i}] = 1 \right\rangle \end{aligned}$$

be its fundamental group. By a representation of $\pi _1(X)$ in $\mathrm {SO}^*(2n)$ we mean a homomorphism $\rho :\pi _1(X) \rightarrow \mathrm {SO}^*(2n)$. The set of all such homomorphisms,

$$\begin{aligned} {{\mathrm{Hom}}}(\pi _1(X),\mathrm {SO}^*(2n)), \end{aligned}$$

can be naturally identified with the subset of $\mathrm {SO}^*(2n)^{2g}$ consisting of $2g$-tuples

$$\begin{aligned} (A_{1},B_{1},\ldots ,A_{g},B_{g}) \end{aligned}$$

satisfying the algebraic equation $\prod _{i=1}^{g}[A_{i},B_{i}] = 1$. This shows that ${{\mathrm{Hom}}}(\pi _1(X),\mathrm {SO}^*(2n))$ is a real algebraic variety.

The group $\mathrm {SO}^*(2n)$ acts on ${{\mathrm{Hom}}}(\pi _1(X),\mathrm {SO}^*(2n))$ by conjugation:

$$\begin{aligned} (g \cdot \rho )(\gamma ) = g \rho (\gamma ) g^{-1} \end{aligned}$$

for $g \in \mathrm {SO}^*(2n)$, $\rho \in {{\mathrm{Hom}}}(\pi _1(X),\mathrm {SO}^*(2n))$ and $\gamma \in \pi _1(X)$. Recall that a representation is reductive if its composition with the adjoint representation is semisimple. If we restrict the action to the subspace ${{\mathrm{Hom}}}^{\mathrm {red}}(\pi _1(X),\mathrm {SO}^*(2n))$ consisting of reductive representations, the orbit space is Hausdorff. By a reductive representation we mean one for which the Zariski closure of the image of $\pi _1(X)$ in $\mathrm {SO}^*(2n)$ is a reductive group. Define the moduli space of representations of $\pi _1(X)$ in $\mathrm {SO}^*(2n)$ to be the orbit space

$$\begin{aligned} \mathcal {R} = {{\mathrm{Hom}}}^{\mathrm {red}}(\pi _1(X),\mathrm {SO}^*(2n)) / \mathrm {SO}^*(2n). \end{aligned}$$

Since $\mathrm {U}(n)\subset \mathrm {SO}^*(2n)$ is a maximal compact subgroup, we have

$$\begin{aligned} \pi _1(\mathrm {SO}^*(2n))\simeq \pi _1(\mathrm {U}(n))\simeq \mathbb {Z}, \end{aligned}$$

and there is a topological invariant attached to a representation $\rho \in \mathcal {R}$ given by an element $d=d(\rho )\in \mathbb {Z}$. This integer is called the Toledo invariant and coincides with the first Chern class of a reduction to a $\mathrm {U}(n)$-bundle of the flat $\mathrm {SO}^*(2n)$-bundle associated to $\rho$.

Fixing the invariant $d\in \mathbb {Z}$ we consider,

$$\begin{aligned} \mathcal {R}_d:=\{\rho \in \mathcal {R}\;\;\;\text{ such } \text{ that } \;\;\; d(\rho )=d\}. \end{aligned}$$

Proposition 6.1

The transformation $\rho \mapsto {(\rho ^t)}^{-1}$ in $\mathcal {R}$ induces an isomorphism of the moduli spaces $\mathcal {R}_d$ and $\mathcal {R}_{-d}$.

As shown by Domic–Toledo [11], the Toledo invariant $d$ of a representation satisfies the Milnor–Wood type inequality:

Proposition 6.2

The moduli space $\mathcal {R}_d$ is empty unless

$$\begin{aligned} |d| \le \left\lfloor \frac{n}{2}\right\rfloor (2g-2). \end{aligned}$$

As a special case of of the non-abelian Hodge theory correspondence (see [15, Theorem 3.32]) we have the following.

Proposition 6.3

The moduli spaces $\mathcal {R}_d$ and $\mathcal {M}_d$ are homeomorphic.

From Proposition 6.3 and Theorem 5.2 we have the main result of this paper regarding the connectedness properties of $\mathcal {R}$ given by the following.

Theorem 6.4

The moduli space $\mathcal {R}_d$ is non-empty and connected if $d=0$ or $|d|=\lfloor \frac{n}{2}\rfloor (2g-2)$.

From Proposition 6.3 and Theorem 6.4 we also have the following rigidity result for maximal representations.

Theorem 6.5

Let $\mathcal {R}_{\max }(\mathrm {SO}^*(4m+2))$ be the moduli space of maximal representations in $\mathrm {SO}^*(2n)$ with $n=2m+1$ and $d=2m(g-1)$. If $m>0$ and $g\ge 2$ then the locus of irreducible representations of $\mathcal {R}_{\max } (\mathrm {SO}^*(4m+2))$ is empty and

$$\begin{aligned} \mathcal {R}_{\max } (\mathrm {SO}^*(4m+2))\simeq \mathcal {R}_{\max }(\mathrm {SO}^*(4m))\times {{\mathrm{Hom}}}(\pi _1(X),\mathrm {U}(1)). \end{aligned}$$

7 Low rank cases

In this section we exploit well known Lie-theoretic isomorphisms to examine $\mathrm {SO}^*(2n)$-Higgs bundles for low values of $n$.

7.1 The case $n=1$

The group $\mathrm {SO}^*(2)$ is isomorphic to $\mathrm {SO}(2)$ and hence, in particular, it is compact. A $\mathrm {SO}^*(2)$-Higgs bundle is thus simply a bundle (with zero Higgs field). Identifying the maximal compact subgroup (in this case the group itself) with $\mathrm {U}(1)$, we see that a $\mathrm {SO}^*(2)$-Higgs bundle consists of a $\mathrm {GL}(1,\mathbb {C})$-bundle, or equivalently, a holomorphic line bundle. Using the usual identification $\mathrm {GL}(1,\mathbb {C})\simeq \mathrm {SO}(2,\mathbb {C})$, we see that the associated $\mathrm {SO}(2,\mathbb {C})$-Higgs bundle is equivalent to the vector bundle $L\oplus L^{-1}$ with the standard off-diagonal quadratic form.

Proposition 7.1

As a $\mathrm {SO}^*(2)$-Higgs bundle, a line bundle $L$ is semistable if and only if $\deg (L)=0$. Moreover, semistability implies stability for $\mathrm {SO}^*(2)$-Higgs bundles.

Proof

We apply Proposition 3.13. The only two-step filtrations are:

$$\begin{aligned} 0 \subset 0&\subset 0 \subset L\\ 0 \subset 0&\subset L \subset L\\ 0 \subset L&\subset L \subset L \end{aligned}$$

All are $\varphi$-invariant since the Higgs field is zero. Applying (3.14) to these filtrations in turn yields $\deg (L)\le 0$, $0\le 0$, and $\deg (L)\ge 0$. The first result follows from this. The second result is a consequence of the fact that there are no $\varphi$-invariant two-step filtrations in which at least one of the subbundles is proper. $\square$

Remark 7.2

Since $L$ and $L^{-1}$ are isotropic subbundles of $L\oplus L^{-1}$, it follows that $L\oplus L^{-1}$ is semistable as a $\mathrm {SO}(2,\mathbb {C})$-bundle if and only if $\deg (L)=0$. This gives an alternative proof for Proposition 7.1.

It follows that the moduli space of $\mathcal {M}_d(\mathrm {SO}^*(2))$ is non-empty only for $d=0$, in which case we can identify

$$\begin{aligned} \mathcal {M}_0(\mathrm {SO}^*(2))\simeq {{\mathrm{Jac}}}^0(X) \end{aligned}$$

where ${{\mathrm{Jac}}}^0(X)$ denotes the Jacobian of degree zero line bundles over $X$.

Remark 7.3

It may look paradoxical that we do not obtain the whole moduli space of line bundles of arbitrary degree over $X$. This is because, as indicated in Remark 3.7, we are fixing the parameter of stability to be zero. In order to obtain the other components of the moduli space we have to consider stability for other integral values of the parameter.

7.2 The case $n=2$

In this section we examine the $\mathrm {SO}^*(2n)$-Higgs bundles $(V,\beta ,\gamma )$ in which ${{\mathrm{rank}}}(V)=2$. The low rank and the isomorphism

$$\begin{aligned} \mathfrak {so}^*(4)\simeq \mathfrak {su}(2)\oplus \mathfrak {sl}(2,\mathbb {R}) \end{aligned}$$

(7.1)

lead us to descriptions that are more explicit than in the general case.

7.2.1 Stability conditions

If ${{\mathrm{rank}}}(V)=2$ there are no two-step filtrations $0\subset V_1\subset V_2\subset V$ in which all the inclusions are strict. The two-step filtrations with at least one non-zero proper subbundle are thus of one of the following types:

• (1) $V_1=0$ and $V_2=L$ where $L$ is a line subbundle, or

• (2) $V_2=V$ and $V_1=L$ where $L$ is a line subbundle, or

• (3) $V_1=V_2=L$ where $L$ is a line subbundle.

The corresponding conditions in Lemma 3.11 for such two-step filtration to be $\varphi$-invariant are:

• (1) $\beta (L^{\perp })=0$ if $V_1=0$ and $V_2=L$,

• (2) $\gamma (L)=0$ if $V_1=L$ and $V_2=V$, and

• (3) $\beta (L^{\perp })\subset L\otimes K$ and $\gamma (L)\subset L^{\perp }\otimes K$ if $V_1=V_2=L$.

Remark 7.4

In case (1) the condition $\beta (L^{\perp })=0$ implies that $\beta :V^*\rightarrow V\otimes K$ has rank less than two. The skew symmetry of $\beta$ thus forces $\beta =0$. Similarly, in case (2), $\gamma (L)=0$ implies that $\gamma =0$. In case (3), the skew symmetry of $\beta$ and $\gamma$ ensure that the conditions $\beta (L^{\perp })\subset L\otimes K$ and $\gamma (L)\subset L^{\perp }\otimes K$ apply for all line subbundles $L\subset V$.

The stability condition for $\mathrm {SO}^*(4)$-Higgs bundles thus reduces to the following.

Proposition 7.5

A $\mathrm {SO}^*(4)$-Higgs bundle $(V,\beta ,\gamma )$ with $\deg (V)>0$ is (semi)stable if and only if $V$ is (semi)stable as a bundle and $\gamma \ne 0$.

A $\mathrm {SO}^*(4)$-Higgs bundle $(V,\beta ,\gamma )$ with $\deg (V)<0$ is (semi)stable if and only if $V$ is (semi)stable as a bundle and $\beta \ne 0$.

A $\mathrm {SO}^*(4)$-Higgs bundle $(V,\beta ,\gamma )$ with $\deg (V)=0$ is (semi)stable if and only if $V$ is (semi)stable as a bundle.

Proof

Suppose that $(V,\beta ,\gamma )$ is a (semi)stable $\mathrm {SO}^*(4)$-Higgs bundle with $\deg (V)=d$. By (3.18), if $d>0$ then $\gamma$ cannot be zero and if $d<0$ then $\beta$ cannot be zero. If $d=0$ then ( see Remark 3.14) there is no restriction on $\beta$ or $\gamma$. Any line subbundle $L\subset V$ defines a $\varphi$-invariant two-step filtration in which $V_1=V_2=L$. Applying Proposition (3.13) we see that if $(V,\beta ,\gamma )$ is semistable then $\deg (L)\le \deg (V)/2$, and the inequality is strict if $(V,\beta ,\gamma )$ is stable. This proves the ‘only if’ direction.

To prove the converse it remains to check that the inequalities (3.14) and (3.15) are satisfied by $\varphi$-invariant two-step filtrations of the form (a) $V_1=0$, $V_2=L$ or (b) $V_1=L$, $V_2=V$. By Remark 7.4, the first case occurs only if $\beta =0$ and hence, by (3.18), $\deg (V)\ge 0$. Thus in this case

$$\begin{aligned} \deg (L)\le \deg (V)/2\implies \deg (L)\le \deg (V). \end{aligned}$$

Similarly, the second case occurs only if $\gamma =0$ and hence $\deg (V)\le 0$. Thus

$$\begin{aligned} \deg (L)\le \deg (V)/2\implies \deg (L)\le 0. \end{aligned}$$

The requisite inequalities thus follow from the (semi)stability of $V$. $\square$

From Proposition 3.24 we have the following.

Proposition 7.6

A $\mathrm {SO}^*(4)$-Higgs bundle $(V,\beta ,\gamma )$ is polystable if and only if

• (1) it is stable with $\varphi \ne 0$, or

• (2) $V$ decomposes as a sum of two line bundles of degree zero and $\beta =\gamma =0$, or

• (3) $V=L_1\oplus L^*_2$ with $\deg (L_1)=-\deg (L_2)$ and with respect to this decomposition $\beta =\left( \begin{matrix} 0&{}\quad \tilde{\beta }\\ -\tilde{\beta }&{}\quad 0 \end{matrix}\right)$ and $\gamma =\left( \begin{matrix} 0&{}\quad \tilde{\gamma }\\ -\tilde{\gamma }&{}\quad 0 \end{matrix}\right)$.

Corollary 7.7

Let $M_d(2)$ denote the moduli space of rank 2, degree $d$ semistable bundles and let $M^s_d(2)\subset M_d(2)$ be the stable locus. There is a map

$$\begin{aligned} \mathcal {M}_d(\mathrm {SO}^*(4))&\longrightarrow M_d(2)\\ [V,\beta ,\gamma ]&\mapsto [V]\nonumber \end{aligned}$$

(7.2)

• (1) If $d>0$ then the image of the map is the locus of bundles for which $h^0(\det (V)^{-1}\otimes K)$ is greater than zero. The fiber over $[V]\in M^s_d(2)$ can be identified with $\mathcal {O}_{\mathbb {P}^s}(1)^{\oplus r}$ where $r=h^0(\det (V)\otimes K)$ and $s=h^0(\det (V)^{-1}\otimes K)$.

• (2) If $d<0$ then the image is the locus of bundles for which $h^0(\det (V)\otimes K)$ is greater than zero. The fiber over $[V]\in M^s_d(2)$ can be identified with $\mathcal {O}_{\mathbb {P}^r}(1)^{\oplus s}$ where $r=h^0(\det (V)\otimes K)$ and $s=h^0(\det (V)^{-1}\otimes K)$.

• (3) If $d=0$ then the map is surjective.

Proof

Everything is immediate from Propositions 7.5 and 7.6 except for the description of the fibers.

Suppose that $d>0$ and consider the fiber over a point in ${M}_d(2)$ represented by the bundle $V$. The $\mathrm {SO}^*(4)$-Higgs bundles $(V,\beta ,\gamma )$ are semistable for all $(\beta ,\gamma )\in H^0(X,\det (V)\otimes K)\oplus (H^0(X,\det (V)^{-1}\otimes K)-\{0\}$. However, since the points in $\mathcal {M}_d(\mathrm {SO}^*(4))$ are isomorphism classes of objects, we need to consider when two objects, say $(V,\beta ,\gamma )$ and $(V,\beta ',\gamma ')$, are isomorphic as $\mathrm {SO}^*(4)$-Higgs bundles. By definition the object are isomorphic if there exists a bundle automorphism $f:V\rightarrow V$ such that $f^*(\beta ')=\beta$ and $f^*(\gamma ')=\gamma$. But if $V$ is stable, then the only automorphisms are multiples of the identity, say $f=tI$, and the induced map on $\beta$ and $\gamma$ is

$$\begin{aligned} f^*(\beta )=t^2\beta \ ,\ f^*(\gamma )=t^{-2}\gamma \end{aligned}$$

(7.3)

The fiber over $[V]\in M_d(2)$ is thus given by $(H^0(X,\det (V)\otimes K)\oplus (H^0(X,\det (V)^{-1}\otimes K)-\{0\}))/\mathbb {C}^*$ where the $\mathbb {C}^*$-action is given by $t(\beta ,\gamma )=(t^2\beta ,t^{-2}\gamma )$. The results follows from this.

The description of the fibers in the $d<0$ case is similar. $\square$

Remark 7.8

• (1) Brill-Noether theory shows that in fact the map is surjective for all $d<(g-1)$.

• (2) If $\deg (V)$ is odd then $M_d(2)=M^s_d(2)$, so all fibers are direct sums of copies of the degree one line bundle over a suitable projective space. Note, though, that the number of summands and the dimension of the projective space need not be constant.

• (3) In the case $d=0$, the fiber over a point $[V]\in M_d(2)$ is the quotient

  $$\begin{aligned} (H^0(X,\det (V)\otimes K)\oplus H^0(X,\det (V)^{-1}\otimes K))/\mathbb {C}^*\ . \end{aligned}$$

7.2.2 Simplicity and smoothness in $\mathcal {M}_d(\mathrm {SO}^*(4))$

Applying Theorem 3.19 to the case of $\mathrm {SO}^*(4)$-Higgs bundles yields:

Theorem 7.9

Let $(V,\varphi )$ be a stable $\mathrm {SO}^*(4)$-Higgs bundle. If $(V,\varphi )$ is not simple, then $V$ is a stable vector bundle of degree zero and $\varphi =0$. In this case $\mathrm{Aut }(V,\varphi )\simeq \mathbb {C}^*$.

Proof

Theorem 3.19 says that there are two alternatives for stable $\mathrm {SO}^*(2n)$-Higgs bundle which are not simple and we wish to show that alternative (1) occurs when $n=2$. To exlude alternative (2) we note that it requires $(V,\varphi )$ to decompose into two $\mathrm {SO}^*(2)$-Higgs bundles with non-zero Higgs fields. This is impossible since the Higgs field necessarily vanishes in a $\mathrm {SO}^*(2)$-Higgs bundle. $\square$

By Proposition 3.22 a stable and simple $\mathrm {SO}^*(4)$-Higgs bundle $(V,\beta ,\gamma )$ represents a smooth point in $\mathcal {M}_d(\mathrm {SO}^*(4))$ (where $d=\deg (V)$) unless $d=0$ and there is a skewsymmetric isomorphism $f:V \xrightarrow {\simeq } V^*$ intertwining $\beta$ and $\gamma$. By Lemma 7.11 such an isomorphism can exist only if $\det (V)=\mathcal {O}$. We thus get:

Proposition 7.10

• (1) If $d$ is odd then $\mathcal {M}_d(\mathrm {SO}^*(4))$ is smooth.

• (2) If $d$ is even and $d\ne 0$ then $\mathcal {M}_d(\mathrm {SO}^*(4))$ is smooth except possibly at points represented by $\mathrm {SO}^*(4)$-Higgs bundles $(V,\beta ,\gamma )$ of the form:

  $$\begin{aligned} V&= L_1\oplus L^*_2, \quad \text {with }\deg (L_1)=-\deg (L_2)\text { and, with respect to this decomposition,}\nonumber \\ \beta&= \left( \begin{matrix} 0&{}\quad \tilde{\beta }\\ -\tilde{\beta }&{}\quad 0 \end{matrix}\right) \text { and } \gamma =\left( \begin{matrix} 0&{}\quad \tilde{\gamma }\\ -\tilde{\gamma }&{}\quad 0 \end{matrix}\right) \end{aligned}$$

  (7.4)

• (3) If $d=0$ then $\mathcal {M}_d(\mathrm {SO}^*(4))$ is smooth except possibly at points represented by $\mathrm {SO}^*(4)$-Higgs bundles $(V,\beta ,\gamma )$ such that

  • (a) $\beta =\gamma =0$, or

  • (b) $(V,\beta ,\gamma )$ is of the form (7.4), or

  • (c) $\det (V)=\mathcal {O}$ and $f\beta =f^{-1}\gamma$ where $f:V\xrightarrow {\simeq } V^*$ is a skew-symmetric isomorphism.

Proof

(1) If $d$ is odd then all semistable and polystable Higgs bundles are stable, simple and do not admit a skew-symmetric isomorphism intertwining the components of the Higgs field.

(2) If $d$ is even and $d\ne 0$ then all stable Higgs bundles are simple and do not admit a skew-symmetric isomorphism intertwining the components of the Higgs field. The non-smooth points can occur only at points represented by polystable Higgs bundles.

(3) The cases (a)–(c) correspond to polystable Higgs bundles (cases (a) and (b)), stable but not simple Higgs bundles (case (a)), or stable and simple bundles which admit a skew-symmetric isomorphism intertwining the components of the Higgs field (case (c)). $\square$

7.2.3 The even degree case

Notice that if $V$ is a rank 2 bundle, then $\Lambda ^2(V)=\det (V)$. Furthermore if $\deg (V)$ is even then $V$ can be decomposed as

$$\begin{aligned} V=U\otimes L\ ,\ \mathrm {with}\ {\left\{ \begin{array}{ll} \det (U) \simeq \mathcal {O}\\ L^2=\det (V) \end{array}\right. }\!\!\!\!. \end{aligned}$$

(7.5)

Lemma 7.11

If $U$ is a rank 2 holomorphic bundle then the following are equivalent:

• (1) $\det (U)\simeq \mathcal {O}$,

• (2) the structure group of $U$ reduces to $\mathrm {SL}(2,\mathbb {C})$,

• (3) $U^*\simeq U$, with the isomorphism defined by a symplectic form $\Omega \in H^0(X,\Lambda ^2U^*)$.

Proof

The equivalence of (1) and (2) is straightforward. The equivalence of (2) and (3) follows from the fact that $\mathrm {SL}(2,\mathbb {C})\simeq \mathrm {Sp}(2,\mathbb {C})$. $\square$

Lemma 7.12

Let $(V,\beta ,\gamma )$ be a $\mathrm {SO}^*(4)$-Higgs bundle with $\deg (V)$ even. Let $V=U\otimes L$ as in (7.5), and let $\Omega \in H^0(X,\Lambda ^2U^*)$ be the symplectic form on $U$ given by (3) of Lemma 7.11, with induced symplectic form $\Omega ^*\in H^0(X,\Lambda ^2U)$ on $U^*$. Then we can write

$$\begin{aligned} \beta&= \Omega \otimes \tilde{\beta }\quad \mathrm {where}\quad \tilde{\beta }\in H^0(X,L^2\otimes K),\nonumber \\ \gamma&= \Omega ^*\otimes \tilde{\gamma }\quad \mathrm {where}\quad \tilde{\gamma }\in H^0(X,L^{-2}\otimes K). \end{aligned}$$

(7.6)

Proof

Immediate from $\Lambda ^2 V \simeq \Lambda ^2U \otimes L^2\simeq L^2$ and the existence of the nowhere vanishing section $\Omega$ of $\Lambda ^2U$. $\square$

Applying Definition 2.1 to the case $G=\mathrm {SL}(2,\mathbb {R})$, a $\mathrm {SL}(2,\mathbb {R})$-Higgs bundle can be described as a triple $(L,\beta ,\gamma )$ where $L$ is a line bundle and $\beta \in H^0(X,L^-2K), \gamma \in H^0(X,L^2K)$. We denote by $\mathcal {M}_l(\mathrm {SL}(2,\mathbb {R}))$ the component of the moduli space of polystable $\mathrm {SL}(2,\mathbb {R})$-Higgs bundles in which $\deg (L)=l$.

The following result shows that $\mathrm {SO}^*(4)$-Higgs bundles of even degree are intimately related to $\mathrm {SL}(2,\mathbb {R})$-Higgs bundles.

Proposition 7.13

Let $(V,\beta ,\gamma )$ be a $\mathrm {SO}^*(4)$-Higgs bundle with $deg(V)$ even. Pick $L$ such that $L^2= \det (V)$ and define $U=V\otimes L^{-1}$. Then

• (1) $U$ is a $\mathrm {SL}(2,\mathbb {C})$-bundle and

• (2) $(L,\tilde{\beta },\tilde{\gamma })$ defines a $\mathrm {SL}(2,\mathbb {R})$-Higgs bundle

where $\tilde{\beta },\tilde{\gamma }$ are as in Lemma 7.12. The $\mathrm {SO}^*(4)$-Higgs bundle $(V,\beta ,\gamma )$ is (semi)stable if and only if $U$ is (semi)stable as a bundle and $(L,\tilde{\beta },\tilde{\gamma })$ is (semi)stable as a $\mathrm {SL}(2,\mathbb {R})$-Higgs bundle.

Proof

Properties (1) and (2) follow from Lemmas and 7.12, and the fact that a triple $(L,\tilde{\beta },\tilde{\gamma })$ (as in Lemma 7.12) defines a $\mathrm {SL}(2,\mathbb {R})$-Higgs bundle. The statement about (semi)stability follows from Proposition 7.5 and the fact that (semi)stability for a $\mathrm {SL}(2,\mathbb {R})$-Higgs bundle $(L,\tilde{\beta },\tilde{\gamma })$ with $\deg (L)\ge 0$ is equivalent to the condition that $\tilde{\gamma }\ne 0$ (if $\deg (L)>0$). $\square$

Remark 7.14

The isomorphism (7.1) is the infinitesimal version of a 2:1 homomorphism

$$\begin{aligned} \eta : \mathrm {SU}(2)\times \mathrm {SL}(2,\mathbb {R})\longrightarrow \mathrm {SO}^*(4)\ . \end{aligned}$$

(7.7)

Proposition 7.13 shows that if $\deg (V)$ is even then the structure group of the $\mathrm {SO}^*(4)$-Higgs bundle lifts via $\eta$ to $\mathrm {SU}(2)\times \mathrm {SL}(2,\mathbb {R})$. If $\deg (V)$ is odd, then the structure group does not lift. The obstruction to the lift can be viewed as an element of $H^2(X,\mathbb {Z}/2)$. In fact, the homomorphism $\eta$ is induced by the homomorphism $\mathrm {Spin}(4,\mathbb {C})\longrightarrow \mathrm {SO}(4,\mathbb {C})$. To see this, recall that

$$\begin{aligned} \mathrm {Spin}(4,\mathbb {C}) \simeq \mathrm {Spin}(3,\mathbb {C}) \times \mathrm {Spin}(3,\mathbb {C}) \simeq \mathrm {SL}(2,\mathbb {C})\times \mathrm {SL}(2,\mathbb {C}). \end{aligned}$$

Under this homomorphism, the real form $\mathrm {SU}(2)\times \mathrm {SL}(2,\mathbb {R})$ of $\mathrm {SL}(2,\mathbb {C})\times \mathrm {SL}(2,\mathbb {C})$ maps to $\mathrm {SO}^*(4)$.

7.2.4 The Cayley partner

Applying Proposition 3.26 with $n=2$, we see that

$$\begin{aligned} |\deg (V)|\le 2g-2 \end{aligned}$$

and that $\gamma$ is an isomorphism if (and only if) $\deg (V)=2g-2$. As in Proposition 7.13 we write $V=U\otimes L$ with $\det (U)=\mathcal {O}$ and $L^2=\det (V)$. In particular, if $\deg (V)=2g-2$ then $\deg (L^{-2}\otimes K)=0$. Moreover, since $\gamma$ is an isomorphism, it follows that $\tilde{\gamma }$ is a non-zero section of $L^{-2}\otimes K$ and thus $L^2=K$. Proposition 7.13 thus becomes the following.

Proposition 7.15

Let $(V,\beta ,\gamma )$ be a $\mathrm {SO}^*(4)$-Higgs bundle with $\deg (V)=2g-2$. Pick $L$ such that $L^2=K$ and define $U=V\otimes L^{-1}$. Then

• (1) $U$ is a $\mathrm {SL}(2,\mathbb {C})$-bundle and

• (2) $(L,\tilde{\beta },\tilde{\gamma })$ defines a $\mathrm {SL}(2,\mathbb {R})$-Higgs bundle where $\tilde{\gamma }$ is a non-zero section in $H^0(X,\mathcal {O})$, and $\tilde{\beta }\in H^0(X,K^2)$. In particular, $(L,\tilde{\beta },\tilde{\gamma })$ defines a Higgs bundle in a Teichmüller component of $\mathcal {M}_{g-1}(\mathrm {SL}(2,\mathbb {R}))$.

Moreover, the polystability of $(V,\beta ,\gamma )$ is equivalent to the polystability of $U$.

Remark 7.16

With $\Omega$ as Lemma 7.12, the data $(U,\Omega ; \tilde{\beta })$ as in Proposition 7.15 defines a $K^2$-twisted $\mathrm {U}^*(2)$-Higgs bundle. Indeed if $(V,\Omega ; \varphi )$ is a $L$-twisted $U^*(2)$-Higgs bundle then we can assume that $\Omega =J$ with respect to suitable local frames. Since, by definition of a $\mathrm {U}^*(2n)$-Higgs bundle, $\varphi ^t\Omega =-\Omega \varphi$, we get that $\varphi =\tilde{\varphi }I$ with respect to the same frames. It follows that locally $\varphi =\tilde{\varphi }I$, where $\tilde{\varphi }\in H^0(X,L)$ (see “Appendix” and [16] for details on $\mathrm {U}^*(2n)$-Higgs bundles). The polystability of the $(U,\tilde{\beta })$ as a $K^2$-twisted $\mathrm {U}^*(2)$-Higgs bundle is equivalent to the polystability of $U$.

Remark 7.17

The ambiguity in the decomposition $V=U\otimes L$ corresponds, in this case, to the choice of a square root of $K$. This is the same choice as the one which distinguishes the Teichmüller component of $\mathcal {M}_{g-1}(\mathrm {SL}(2,\mathbb {R}))$.

Combining Propositions 7.13 and (7.5) gives rise to a $2^{2g}:1$ map

$$\begin{aligned}&T: M_0(2)\times \mathcal {M}_l(\mathrm {SL}(2,\mathbb {R}))&\longrightarrow \mathcal {M}_{2l}(\mathrm {SO}^*(4))\nonumber \\&\quad ([U],[L,\tilde{\beta },\tilde{\gamma }])&\mapsto [U\otimes L, \beta ,\gamma ] \end{aligned}$$

(7.8)

where $M_0(2)$ denotes the moduli space of polystable rank 2 bundles with trivial determinant. This is the Higgs bundle manifestation of isomorphism (7.1).

Proposition 7.18

For each $0\le l\le g-1$ the moduli space $\mathcal {M}_{2l}(\mathrm {SO}^*(4))$ is connected.

Proof

Under the map $T$, the $2^{2g}$ Teichmüller components in $\mathcal {M}_{g-1}(\mathrm {SL}(2,\mathbb {R}))$ are all identified in the component $\mathcal {M}_{2g-2}(\mathrm {SO}^*(4))$. For $0 \le l < g-1$ the moduli spaces $\mathcal {M}_{l}(\mathrm {SL}(2,\mathbb {R}))$ are connected. $\square$

7.3 The case $n=3$

The Lie algebra of $\mathrm {SO}^*(6)$ is isomorphic to $\mathfrak {su}(1,3)$, the Lie algebra of $\mathrm {SU}(1,3)$. The groups differ because they have different centers, with $Z(\mathrm {SO}^*(6))\simeq \mathbb {Z}/2$ and $Z(\mathrm {SU}(1,3))\simeq \mathbb {Z}/4$. Both groups are finite covers of $\mathrm {PU}(1,3)$, the adjoint form of the Lie algebra. The relationships among the groups $\mathrm {SO}^*(6),\mathrm {SU}(1,3)$, and $\mathrm {PU}(1,3)$ leads to relations among the corresponding Higgs bundles for the groups (see Proposition 7.29). As in the case of $\mathrm {SO}^*(4)$, the relation can be explained in terms of the spin group. Namely, the $2:1$ homomorphism $\mathrm {Spin}(6,\mathbb {C}) \longrightarrow \mathrm {SO}(6,\mathbb {C})$ restricts to a $2:1$ homomorphism $\mathrm {Spin}^*(6) \longrightarrow \mathrm {SO}^*(6)$. But under the isomorphism $\mathrm {Spin}(6,\mathbb {C})\simeq \mathrm {SL}(4,\mathbb {C})$, one has the isomorphism of the corresponding real forms $\mathrm {Spin}^*(6)$ and $\mathrm {SU}(1,3)$.

The key to understanding the relation between the Higgs bundles is the isomorphism

$$\begin{aligned} \Lambda ^k(\mathbf {V}^*)\otimes \Lambda ^n(\mathbf {V})\longrightarrow \Lambda ^{n-k}(\mathbf {V}). \end{aligned}$$

where $\mathbf {V}$ is a vector space of dimension $n>k$, and the map is defined by the interior product. This extends to exterior powers of vector bundles of rank $n$. In particular, if $n=3$ and $k=2$ we get $\Lambda ^2 V^*\otimes \det (V)\simeq V$ or equivalently

$$\begin{aligned} \Lambda ^2 V^*\simeq \det (V)^*\otimes V\simeq {{\mathrm{Hom}}}(\det (V),V). \end{aligned}$$

(7.9)

Hence sections $\gamma \in H^0(X,\Lambda ^2V^*\otimes K)$ and $\beta \in H^0(X,\Lambda ^2V\otimes K)$ define holomorphic bundle maps $\tilde{\gamma }:\det (V)\rightarrow V\otimes K$ and $\tilde{\beta }:V\rightarrow \det (V)\otimes K$ by

$$\begin{aligned} \tilde{\gamma }(\omega )&= \iota _{\gamma }(\omega ),\nonumber \\ \tilde{\beta }(v)&= \beta \wedge v, \end{aligned}$$

(7.10)

where $\iota _{\gamma }$ denotes interior product.

Proposition 7.19

A $\mathrm {SO}^*(6)$-Higgs bundle defines a $\mathrm {U}(1,3)$-Higgs bundle via the map

$$\begin{aligned} (V,{\beta },{\gamma })\mapsto (\det (V),V, \tilde{\beta },\tilde{\gamma }) \end{aligned}$$

(7.11)

where $\tilde{\beta }$ and $\tilde{\gamma }$ are related to $\beta$ and $\gamma$ as in (7.10).

Proof

This follows immediately from the definitions. In general, a $\mathrm {U}(p,q)$-Higgs bundle is defined by a tuple $(V,W,\beta ,\gamma )$ where $V$ and $W$ are bundles of rank $p$ and $q$ respectively, and $\beta ,\gamma$ are maps $\beta :V\rightarrow W\otimes K$ and $\gamma :W\rightarrow V\otimes K$ (see [5] and Sect. 8.2.1 for more details). $\square$

Remark 7.20

We refer the reader to [5] and Sect. 8.2.1 for more details but note here the following key features:

• (1) The tuple $(V,W,\beta ,\gamma )$ represents a $\mathrm {SU}(p,q)$-Higgs bundle if it satisfies the determinant condition $\det (V\oplus W)=\mathcal {O}$. In particular, $\mathrm {SU}(1,3)$-Higgs bundles are represented by tuples $(L,W,\tilde{\beta },\tilde{\gamma })$ with $L$ a line bundle, $W$ a rank three bundle, $\tilde{\beta }:W\rightarrow L \otimes K$ and $\tilde{\gamma }:L\rightarrow W \otimes K$ and such that $\det (L\oplus W)$ is trivial.

• (2) While a $\mathrm {PU}(p,q)$-Higgs bundle is defined by a principal $\mathrm {P}(\mathrm {U}(p)\times \mathrm {U}(q))$-bundle together with an appropriate Higgs field, the structure group of the bundle can always be lifted to $\mathrm {U}(p)\times \mathrm {U}(q)$. Together with the Higgs field, the principal $\mathrm {U}(p)\times \mathrm {U}(q)$-bundle defines a $\mathrm {U}(p,q)$-Higgs bundle. The lifts are defined up to a twisting by a line bundle.

• (3) The notion of polystability and the corresponding Hitchin equations for $\mathrm {U}(p,q)$-Higgs bundles are described in Sect. 8.2.1 and in [5]. The notions for $\mathrm {SU}(p,q)$ and $\mathrm {PU}(p,q)$ are similar.

• (4)

  • (a) The components of the moduli space of polystable $\mathrm {U}(p,q)$-Higgs bundles are labeled by the integer pair $(a,b)$ where $a=\deg (V)$ and $b=\deg (W)$. We will denote these components by $\mathcal {M}_{a,b}(\mathrm {U}(p,q))$.

  • (b) For a $\mathrm {PU}(p,q)$-Higgs bundle, the components of the moduli spaces are labeled by the combination $\tau =2\frac{aq-bp}{p+q}$, where $(V,W,\beta ,\gamma )$ represents a $\mathrm {U}(p,q)$-Higgs bundles obtained by lifting the structure group. This combination, known as the Toledo invariant, is independent of the lifts to $\mathrm {U}(p,q)$. We will denote the components with Toledo invariant $\tau$ by $\mathcal {M}_{\tau }(\mathrm {PU}(p,q))$.

  • (c) For $\mathrm {SU}(p,q)$-Higgs bundles, for which $\deg (V)=-\deg (W)$, the components of the moduli space can be labeled by the single integer $a=\deg (V)$. We will denote these components by $\mathcal {M}_{a}(\mathrm {SU}(p,q))$.

Proposition 7.21

Let $(V,{\beta },{\gamma })$ and $(\det (V),V, \tilde{\beta },\tilde{\gamma })$ be a $\mathrm {SO}^*(6)$-Higgs bundle and corresponding $\mathrm {U}(1,3)$-Higgs bundle, as in (7.11). Then the following are equivalent:

• (A) The bundle $V$ admits a metric, say $H$, satisfying the $\mathrm {SO}^*(6)$-Hitchin equation on $(V, \beta , \gamma )$, namely (see (3.16))

  $$\begin{aligned} F^H_{V}+\beta \beta ^{*_H}+\gamma ^{*_H}\gamma = 0. \end{aligned}$$

  (7.12)

• (B) The bundles $V$ and $\det (V)$ admit metrics, say $K$ and $k$, satisfying the $\mathrm {U}(1,3)$-Hitchin equation on $(\det (V), V, \tilde{\beta }, \tilde{\gamma })$, namely (see [5])

  $$\begin{aligned} F^K_{V}+\tilde{\beta }^{*_{K,k}}\tilde{\beta }+\tilde{\gamma } \tilde{\gamma }^{*_{K,k}}&= -\sqrt{-1} \mu \mathbf {I}_{V}\omega ,\nonumber \\ F^k_{\det (V)}+\tilde{\beta }\tilde{\beta }^{*_{K,k}}+\tilde{\gamma }^{*_{K,k}} \tilde{\gamma }&= -\sqrt{-1}\mu \omega . \end{aligned}$$

  (7.13)

In these equations

• the first terms denote the curvature of the Chern connection with respect to the indicated metrics,

• the adjoints in (7.12) are with respect to $H$ and the metric it induces on $V^*$,

• the adjoints in (7.13) are with respect to $K$ and $k$

• $\mu =\dfrac{\sqrt{-1}\int _X\mathrm{Tr }(F^H_V)}{2\mathrm {Vol}(X)}=\dfrac{\pi \deg (V)}{\mathrm {Vol}(X)}$,

• $\mathbf {I}_V$ is the identity map on $V$, and

• $\omega$ denotes the Kähler form of the metric on the Riemann surface $X$.

The proof of Proposition 7.21 uses the following technical Lemma.

Lemma 7.22

Let $(\det (V),V, \tilde{\beta },\tilde{\gamma })$ be a $\mathrm {U}(1,3)$-Higgs bundle, as in (7.11). Let $H$ and $h$ be any metrics on $V$ and $\det (V)$ respectively. Let $K$ be a metric on $V$ which is related to $H$ by a conformal factor $e^u$, i.e. $K(\phi ,\psi )=e^uH(\phi ,\psi )$ for any sections $\phi$ and $\psi$ of $V$. Similarly let $k$ be a metric on $\det (V)$ which is related to $h$ by the same conformal factor $e^u$. Then (in the notation of Proposition 7.21, and denoting by $*_{H,h}$ adjoints with respect to $H$ and $h$)

• (1) $\tilde{\gamma }^{*_{K,k}}=\tilde{\gamma }^{*_{H,h}}$,

• (2) $\tilde{\beta }^{*_{K,k}}=\tilde{\beta }^{*_{H,h}}$,

• (3) $F^K_V=F^H_V-\sqrt{-1}\Delta (u)\omega \mathbf {I}_V$, and

• (4) $F^{k}_{\det (V)}=F^{h}_{\det (V)}-\sqrt{-1}\Delta (u)\omega$.

where in (3) and (4) $\omega$ denotes the Kähler form on $X$.

Proof

Let $a$ be a point in the fiber of $V$ over a point $x\in X$ and let $b$ be a point in the fiber over $x$ of $\det (V)\otimes \bar{K}$. Then

$$\begin{aligned} h(b,\tilde{\gamma }^{*_{K,k}}(a))&=\,e^{-u(x)}k(b,\tilde{\gamma }^{*_{K,k}}(a))\\&=\,e^{-u(x)}K(\tilde{\gamma }(b),a)\\&=\,e^{-u(x)}e^{u(x)}H(\tilde{\gamma }(b),a)=h(b,\tilde{\gamma }^{*_{H,h}}(a)). \end{aligned}$$

This proves (1). The proof of (2) is similar. The proof of (3) and (4) follows directly from the definition of the Chern connection. Indeed, if metrics $H_1$ and $H_2$ on a holomorphic bundle $E$ are related by $H_1=H_2s$ where $s$ is a (positive definite) automorphism of $E$, then the curvatures of the Chern connections are related by

$$\begin{aligned} F_{H_1}=F_{H_2}+\bar{\partial }_E(s^{-1}D'_{H_1}(s)) \end{aligned}$$

(7.14)

where $\bar{\partial }_E$ and $D'_{H_1}$ are the antiholomorphic and holomorphic parts of the Chern connection for $H_1$. If $s=e^u\mathbf {I}$ then the second term reduces to $-\sqrt{-1}\Delta (u)\omega$. $\square$

We now prove Proposition 7.21.

Proof of Proposition 7.21

Fix a local frame for $V$ and use the dual frame for $V^*$. Also, fix a local complex coordinate on the base. Then $\gamma$, as a map from $V$ to $V^*\otimes K$ is given locally by a matrix of holomorphic 1-forms, which we write as

$$\begin{aligned} \gamma =\begin{bmatrix} 0&\quad \gamma _1&\quad \gamma _2\\ -\gamma _1&\quad 0&\quad \gamma _3\\ -\gamma _2&\quad -\gamma _3&\quad 0 \end{bmatrix}dz\ . \end{aligned}$$

(7.15)

Using the induced frame for $\det (V)$, the map $\tilde{\gamma }$ is then given by

$$\begin{aligned} \tilde{\gamma }=\begin{bmatrix} \gamma _3\\ -\gamma _2\\ \gamma _1 \end{bmatrix}dz\ . \end{aligned}$$

(7.16)

Similarly, if $\beta$ as a map from $V^*$ to $V\otimes K$ is given locally by a matrix of holomorphic 1-forms of the form

$$\begin{aligned} \beta =\begin{bmatrix} 0&\quad \beta _1&\quad \beta _2\\ -\beta _1&\quad 0&\quad \beta _3\\ -\beta _2&\quad -\beta _3&\quad 0 \end{bmatrix}dz\ . \end{aligned}$$

(7.17)

then the map $\tilde{\beta }$ is then given by

$$\begin{aligned} \tilde{\beta }=\begin{bmatrix} \beta _3&\quad -\beta _2&\quad \beta _1 \end{bmatrix}dz\ . \end{aligned}$$

(7.18)

Given a metric, say $H$, on $V$, we can pick the local frame to be unitary with respect to $h$. Then locally

$$\begin{aligned} \gamma ^{*_H}=\begin{bmatrix} 0&\quad -\bar{\gamma }_1&\quad -\bar{\gamma }_2\\ \bar{\gamma }_1&\quad 0&\quad -\bar{\gamma }_3\\ \bar{\gamma }_2&\quad \bar{\gamma }_3&\quad 0 \end{bmatrix} d\bar{z}\ . \end{aligned}$$

(7.19)

The metric $H$ induces a metric on $\det (V)$, which we denote by $h$. With respect to the metrics $H$ on $V$ and $h$ on $\det (V)$, the adjoint of $\tilde{\gamma }$ is given locally by

$$\begin{aligned} \tilde{\gamma }^{*_{H,h}}=\begin{bmatrix} \bar{\gamma }_3&-\bar{\gamma }_2&\bar{\gamma }_1 \end{bmatrix}d\tilde{z}\ . \end{aligned}$$

(7.20)

Using the metrics $H$ and $h$, and taking into account that the entries in the matrix are 1-forms, we get that

$$\begin{aligned} \gamma ^{*_H}\gamma&= \tilde{\gamma }\tilde{\gamma }^{*_{H,h}} +\tilde{\gamma }^{*_{H,h}}\tilde{\gamma }\mathbf {I}_V,\nonumber \\ \beta \beta ^{*_H}&= \tilde{\beta }^{*_{H,h}}\tilde{\beta }+ \tilde{\beta }\tilde{\beta }^{*_{H,h}}\mathbf {I}_V, \end{aligned}$$

(7.21)

and also

$$\begin{aligned} \mathrm{Tr }(\tilde{\gamma }\tilde{\gamma }^{*_{H,h}})&= -\tilde{\gamma }^{*_{H,h}} \tilde{\gamma },\nonumber \\ \mathrm{Tr }(\tilde{\beta }^{*_{H,h}}\tilde{\beta })&= -\tilde{\beta } \tilde{\beta }^{*_{H,h}}. \end{aligned}$$

(7.22)

Suppose that $V$ admits a metric which satisfies the $\mathrm {SO}^*(6)$-Hitchin equations for $(V,\beta ,\gamma )$, namely equation (7.12). Because of (7.21) this is equivalent to

$$\begin{aligned} F^H_V+\tilde{\beta }^{*_{H,h}}\tilde{\beta }+\tilde{\gamma } \tilde{\gamma }^{*_{H,h}}=-(\tilde{\gamma }^{*_{H,h}} \tilde{\gamma }+\tilde{\beta }\tilde{\beta }^{*_{H,h}})\mathbf {I}_V. \end{aligned}$$

(7.23)

Taking the trace of this, and using (7.22), we also get

$$\begin{aligned} \mathrm{Tr }(F^h_V)+\tilde{\gamma }^{*_{H,h}}\tilde{\gamma }+\tilde{\beta } \tilde{\beta }^{*_{H,h}}=-(\tilde{\gamma }^{*_{H,h}}\tilde{\gamma } +\tilde{\beta }\tilde{\beta }^{*_{H,h}}). \end{aligned}$$

(7.24)

We can write the $(1,1)$-form $\tilde{\gamma }^{*_{H,h}}\tilde{\gamma }+\tilde{\beta } \tilde{\beta }^{*_{H,h}}$ as

$$\begin{aligned} \tilde{\gamma }^{*_{H,h}}\tilde{\gamma }+\tilde{\beta } \tilde{\beta }^{*_{H,h}}=\sqrt{-1}t\omega =-\left( \sum _{i=1}^3| \tilde{\gamma }_i|^2-\sum _{i=1}^3|\tilde{\beta }_i|^2\right) dz\wedge d\bar{z} \end{aligned}$$

(7.25)

where the last expression is in local coordinates. Notice that by (7.24) we get

$$\begin{aligned} -2\sqrt{-1}\int _Xt\omega =\int _X\mathrm{Tr }(F^h_V)=-2\pi \sqrt{-1}\deg (V). \end{aligned}$$

(7.26)

Since $\mathrm{Tr }(F^H_V)=F^{h}_{\det (V)}$, Eqs. (7.23) and (7.24) can thus be written as

$$\begin{aligned} F^H_V+\tilde{\beta }^{*_{H,h}}\tilde{\beta }+\tilde{\gamma } \tilde{\gamma }^{*_{H,h}}&= -\sqrt{-1}t\omega \mathbf {I}_V\nonumber \\ F^{h}_{\det (V)}+\tilde{\gamma }^{*_{H,h}}\tilde{\gamma }+ \tilde{\beta }\tilde{\beta }^{*_{H,h}}&= -\sqrt{-1}t\omega \end{aligned}$$

(7.27)

where

$$\begin{aligned} \frac{\int t\omega }{{{\mathrm{Vol}}}(X)}=\frac{\pi \deg (V)}{{{\mathrm{Vol}}}(X)}=\mu . \end{aligned}$$

(7.28)

Equations (7.27) differ from the required $\mathrm {U}(1,3)$-Hitchin equations only in that the right hand side is not constant, but instead involves a function whose average value is the required constant. Lemma 7.22 allows us to remove this discrepancy by rescaling the metrics on $V$ and $\det (V)$. Indeed if we pick a function $u$ such that it satisfies the condition

$$\begin{aligned} \Delta (u)=t-\mu \end{aligned}$$

and define metrics $K=He^u$ on $V$ and $k=he^u$ on $\det (V)$ then

$$\begin{aligned} F^K_V+\tilde{\beta }^{*_{K,k}}\tilde{\beta }+\tilde{\gamma } \tilde{\gamma }^{*_{K,k}}&=-\sqrt{-1}\mu \omega \mathbf {I}_V\nonumber \\ F^{k}_{\det (V)}+\tilde{\gamma }^{*_{K,k}}\tilde{\gamma }+ \tilde{\beta }\tilde{\beta }^{*_{K,k}}&=-\sqrt{-1}\mu \omega \end{aligned}$$

as required.

Conversely, suppose that $V$ and $\det (V)$ admit metrics $K$ and $k$ which satisfy the $\mathrm {U}(1,3)$-Hitchin equations on $(\det (V), V, \tilde{\beta },\tilde{\gamma })$, namely (7.13). In general $k$ will differ from the metric induced by $K$ on $\det (V)$. Denoting the latter by $\det (K)$, we can write

$$\begin{aligned} k=\det (K)e^u \end{aligned}$$

(7.29)

where $u$ is a smooth function on $X$. Now define new metrics on $V$ and $\det (V)$ which are related to $K$ and $k$ by the conformal factor $e^{u/2}$, i.e. set

$$\begin{aligned} H=Ke^{u/2}\quad \text {and}\quad h=ke^{u/2}. \end{aligned}$$

(7.30)

Notice that $\det (H)=\det (K)e^{3u/2}=h$, where $\det (H)$ denotes the metric induced by $H$ on $\det (V)$. Moreover, since both metrics are modified by the same conformal factor, the adjoints $\tilde{\beta }^*$ and $\tilde{\gamma }^*$ are unaffected (see Lemma 7.22). By parts (3) and (4) of Lemma 7.22 and the fact that $K$ and $k$ satisfy the $\mathrm {U}(1,3)$-Hitchin equations, we thus get

$$\begin{aligned} F^H_V+\tilde{\beta }^{*_{H,h}}\tilde{\beta }+\tilde{\gamma } \tilde{\gamma }^{*_{H,h}}&=-\sqrt{-1}(\mu -\frac{\Delta (u)}{2}) \omega \mathbf {I}_V=-\sqrt{-1}t\omega \mathbf {I}_V\\ F^{h}_{\det (V)}+\tilde{\gamma }^{*_{H,h}}\tilde{\gamma }+\tilde{\beta } \tilde{\beta }^{*_{H,h}}&=-\sqrt{-1}(\mu -\frac{\Delta (u)}{2}) \omega =-\sqrt{-1}t\omega \end{aligned}$$

where $t= \mu -\frac{\Delta (u)}{2}$ and $h=\det (H)$. Exactly as above (see Eqs. (7.23)–(7.28)) we find that these two equations combine to yield

$$\begin{aligned} F^H_{V}+\beta \beta ^{*_H}+\gamma ^{*_H}\gamma = 0 \end{aligned}$$

as required. $\square$

Corollary 7.23

Let $(V, {\beta },{\gamma })$ and $(\det (V),V, \tilde{\beta },\tilde{\gamma })$ be as in (7.11). Then $(V, {\beta },{\gamma })$ defines a polystable $\mathrm {SO}^*(6)$-Higgs bundle if and only if $(\det (V),V, \tilde{\beta },\tilde{\gamma })$ defines a polystable $\mathrm {U}(1,3)$-Higgs bundle. Moreover, the map (7.11) defines an embedding

$$\begin{aligned} \mathcal {M}_d(\mathrm {SO}^*(6))\hookrightarrow \mathcal {M}_{d,d}(\mathrm {U}(1,3)) \end{aligned}$$

(7.31)

where $\mathcal {M}_{d,d}(\mathrm {U}(1,3))$ denotes the component in the moduli space of polystable $\mathrm {U}(1,3)$-Higgs bundles in which the bundles both have degree $d$.

Proof

The first part follows immediately from Proposition 7.21 because of the Hitchin-Kobayashi correspondence for $G$-Higgs bundles, i.e. Theorem 2.4. The map defined by (7.11) is clearly injective, with image given by the subvariety in which the $\mathrm {U}(1,3)$-Higgs bundles are defined by tuples $(L,V,\beta ,\gamma )$ in which $L=\det (V)$. $\square$

Remark 7.24

By Proposition 3.16 the dimension of $\mathcal {M}_d(\mathrm {SO}^*(6))$ is $15(g-1)$, while the dimension of $\mathcal {M}_{d,d}(\mathrm {U}(1,3))$ is $16(g-1)+1$ (see [5]). The image of the embedding given by (7.31) thus has codimension $g$ in $\mathcal {M}_{d,d}(\mathrm {U}(1,3))$.

Proposition 7.25

Let $(\det (V),V, \tilde{\beta },\tilde{\gamma })$ be a $\mathrm {U}(1,3)$-Higgs bundle in which $\deg (V)$ is even. Pick $L$ such that $L^2=\det (V)$ and define maps

$$\begin{aligned} \tilde{\beta }_L&= \tilde{\beta }\otimes 1_{L}:V\otimes L^{-1}\rightarrow L \otimes K\nonumber \\ \tilde{\gamma }_L&= \tilde{\gamma }\otimes 1_{L}:L\rightarrow V\otimes L^{-1}\otimes K \end{aligned}$$

(7.32)

where $1_{L}:L^{-1}\rightarrow L^{-1}$ is the identity map. Then $(L,V\otimes L^{-1},\tilde{\beta }_L,\tilde{\gamma }_L)$ defines an $\mathrm {SU}(1,3)$-Higgs bundle and, with the same notation as in Proposition 7.21, the following are equivalent:

• (A) The bundles $V$ and $\det (V)$ admit metrics, say $H$ and $h$, satisfying

  $$\begin{aligned} F^H_{V}+\tilde{\beta }^{*_{H,h}}\tilde{\beta }+\tilde{\gamma } \tilde{\gamma }^{*_{H,h}}&= -\sqrt{-1} \mu \mathbf {I}_{V}\omega \nonumber \\ F^h_{\det (V)}+\tilde{\beta }\tilde{\beta }^{*_{H,h}}+ \tilde{\gamma }^{*_{H,h}}\tilde{\gamma }&= -\sqrt{-1}\mu \omega \ . \end{aligned}$$

  (7.33)

• (B) The bundles $V\otimes L^{-1}$ and $L$ admit metrics, say $K$ and $k$, satisfying

  $$\begin{aligned} F^K_{V\otimes L^{-1}}+(\tilde{\beta }_L)^{*_{K,k}}(\tilde{\beta }_L)+ (\tilde{\gamma }_L)(\tilde{\gamma }_L)^{*_{K,k}}&= 0\nonumber \\ F^k_{L}+(\tilde{\beta }_L)(\tilde{\beta }_L^{*_{K,k}})+ (\tilde{\gamma }_L)^{*_{K,k}}(\tilde{\gamma }_L)&= 0. \end{aligned}$$

  (7.34)

Proof

Since $L^{2}=\det (V)$ it follows that

$$\begin{aligned} \det (L\oplus V\otimes L^{-1})=\det (V)\otimes L^{-2}=\mathcal {O}\ . \end{aligned}$$

(7.35)

and hence $(L,V\otimes L^{-1},\tilde{\beta }_L,\tilde{\gamma }_L)$ defines a $\mathrm {SU}(1,3)$-Higgs bundle.

Let $h_0$ be the Hermitian-Einstein metric on $L^{-1}$, so that the curvature of the corresponding Chern connection satisfies $F^{h_0}_L=\sqrt{-1}\deg (L)\omega$. Given metrics $H$ and $h$ which satisfy (A), define $K=H\otimes h_0$ on $V\otimes L^{-1}$ and $k=h\otimes h_0$ on $L=\det (V)\otimes L^{-1}$. Conversely, given metrics $K$ and $k$ which satisfy (B), define $H=K\otimes h^{-1}_0$ on $V=V\otimes L^{-1}\otimes L$ and $h=k\otimes h^{-1}_0$ on $\det (V)=L^{2}$. $\square$

Remark 7.26

The equations (7.34) are not exactly the $\mathrm {SU}(1,3)$-Hitchin equations. If $(L,W,b,c)$ is any $\mathrm {SU}(1,3)$-Higgs bundle, the Hitchin equations for metrics $k$ and $K$ on $L$ and $W$ respectively are equivalent to the condition

$$\begin{aligned} \begin{bmatrix} F^K_{W}+b^{*_{K,k}}b+cc^{*_{K,k}}&\quad 0\\ 0&\quad F^k_{L}+bb^{*_{K,k}}+c^{*_{K,k}}c \end{bmatrix}_0=0 \end{aligned}$$

(7.36)

where $[A]_0$ denotes the trace free part of the matrix $[A]$. The pair (7.34) (for the $\mathrm {SU}(1,3)$-Higgs bundle $(\det (V),V, \tilde{\beta },\tilde{\gamma })$) is equivalent to (7.36) together with the extra condition $\mathrm{Tr }(F^K_{V\otimes L^{-1}})+F^k_L=0$. In fact this condition can always be achieved by a simultaneous conformal transformation of the metrics $K$ and $k$, as in (7.30). As explained above, such conformal transformations affect only the curvature terms in the equation but do not change the trace-free parts of those terms.

Remark 7.27

By defining $V=W\otimes L$, any $\mathrm {SU}(1,3)$-Higgs bundle $(L,W,\beta ,\gamma )$ can be represented by a tuple $(L, V\otimes L^{-1},\tilde{\beta }_L,\tilde{\gamma }_L)$, where the Higgs fields are maps $\tilde{\beta }_L: V\otimes L^{-1}\rightarrow L\otimes K$ and $\tilde{\gamma }_L: L\rightarrow V\otimes L^{-1}\otimes K$. Notice that

• $L^2=\det (V)$, and hence

• $\tilde{\beta }_L: V\otimes L^{-1}\rightarrow L\otimes K$ defines $\beta \in H^0(X,V^*\det (V)\otimes K)\simeq H^0(X,\Lambda ^2V\otimes K)$,

• $\tilde{\gamma }_L:L\rightarrow V\otimes L^{-1}\otimes K$ defines $\gamma \in H^0(X,V\otimes \det (V)^*\otimes K)\simeq H^0(X,\Lambda ^2V^*\otimes K)$.

Corollary 7.28

With notation as in Remark 7.27, the map

$$\begin{aligned} (L,V\otimes L^{-1},\tilde{\beta }_L,\tilde{\gamma }_L)\mapsto (L^2,V,{\beta },{\gamma }) \end{aligned}$$

(7.37)

defines a map

$$\begin{aligned} \mathcal {M}_l(\mathrm {SU}(1,3))\rightarrow \mathcal {M}_{2l,2l}(\mathrm {U}(1,3))\ , \end{aligned}$$

(7.38)

and the map

$$\begin{aligned} (L,V\otimes L^{-1},\tilde{\beta }_L,\tilde{\gamma }_L)\mapsto (V,{\beta },{\gamma }) \end{aligned}$$

(7.39)

defines a $2^{2g}:1$ surjective map

$$\begin{aligned} \mathcal {M}_l(\mathrm {SU}(1,3))\rightarrow \mathcal {M}_{2l}(\mathrm {SO}^*(6)). \end{aligned}$$

(7.40)

Here $l=\deg (L)$, $\tau$ denotes the Toledo invariant, and the notation for the moduli spaces is as in (4) of Remark 7.20.

Proof

The tuple $(L^2,V,{\beta },{\gamma })$ clearly defines a $\mathrm {U}(1,3)$-Higgs bundle with $\deg (L^2)=\deg (V)=2l$, while remark 7.27 shows that $(V,\beta ,\gamma )$ defines a $\mathrm {SO}^*(6)$-Higgs bundle. In order to show that the given maps induces maps between the indicated moduli spaces we need to show that the maps preserve polystability. We do this by invoking the Hitchin–Kobayashi correspondences for $\mathrm {SU}(1,3)$-, $\mathrm {U}(1,3)$-, and $\mathrm {SO}^*(6)$-Higgs bundles, i.e. we show that the map preserves the conditions for existence of solutions to the Hitchin equations for the Higgs bundles. Moreover, by Proposition 7.25 together with remark 7.26, $(L, V\otimes L^{-1},\tilde{\beta }_L,\tilde{\gamma }_L)$ admits a solution to the $\mathrm {SU}(1,3)$-Hitchin equations if and only if $(L^2=\det (V), V,\beta ,\gamma )$ admits a solution to the $\mathrm {U}(1,3)$-Hitchin equations; and by Proposition 7.21, $(\det (V), V,\beta ,\gamma )$ admits a solution to the $\mathrm {U}(1,3)$-Hitchin equations if and only if $(V,\beta ,\gamma )$ admits a solution to the $\mathrm {SO}^*(6)$-Hitchin equations.

Finally, take any point in $\mathcal {M}_{2l}(\mathrm {SO}^*(6))$, represented say by $(V,{\beta },{\gamma })$. For any $L$ such that $L^2=\det (V)$, the $\mathrm {SU}(1,3)$-Higgs bundles $(L, V\otimes L^{-1},\tilde{\beta }_L,\tilde{\gamma }_L)$ is in the pre-image of $(V,{\beta },{\gamma })$ under the map. This shows that the map is surjective. The multiplicity comes from choices of square roots of $\det (V)$. $\square$

In addition to the maps (7.31), (7.38), and (7.40), we have the surjective map (see [5])

$$\begin{aligned} \mathcal {M}_{l,b}(\mathrm {U}(1,3))&\rightarrow \mathcal {M}_{\tau }(\mathrm {PU}(1,3)),\nonumber \\ (L,W,\beta ,\gamma )&\mapsto (\mathbb {P}(L\oplus W),\beta ,\gamma ), \end{aligned}$$

(7.41)

where $l=\deg (L), b=\deg (W),$ and $\tau = (3l-b)/2$. Conversely, any $\mathrm {PU}(1,3)$-Higgs bundle in $\mathcal {M}_{\tau }(\mathrm {PU}(1,3))$ is in the image of such a map, where the degrees $(l,b)$ are determined only up to the $\mathbb {Z}$-action $(l,b)\mapsto (l+k, b+3k)$. This corresponds to twisting $L\oplus W$ by a line bundle of degree $k$.

These maps lead to the following relations among Higgs bundles for the groups $\mathrm {SO}^*(6), \mathrm {SU}(1,3)$, and $\mathrm {PU}(1,3)$.

Proposition 7.29

• (1) The composition of maps (7.41) and (7.31) defines a surjective map

  $$\begin{aligned} \mathcal {M}_{d}(\mathrm {SO}^*(6))\mapsto \mathcal {M}_{d}(\mathrm {PU}(1,3))\ . \end{aligned}$$

  (7.42)

  Moreover a $\mathrm {PU}(1,3)$-Higgs bundle in $\mathcal {M}_{\tau }(\mathrm {PU}(1,3))$ is in the image of such a map if and only if $\tau$ is an integer.

• (2) The composition of maps (7.41) and (7.38) defines a surjective map

  $$\begin{aligned} \mathcal {M}_{d}(\mathrm {SU}(1,3))\mapsto \mathcal {M}_{2d}(\mathrm {PU}(1,3))\ . \end{aligned}$$

  (7.43)

  Moreover a $\mathrm {PU}(1,3)$-Higgs bundle in $\mathcal {M}_{\tau }(\mathrm {PU}(1,3))$ is in the image of such a map if and only if $\tau$ is an even integer.

• (3) A $\mathrm {SO}^*(6)$-Higgs bundle in $\mathcal {M}_d(\mathrm {SO}^*(6))$ lies in the image of a map of the form (7.40) if and only if $d$ is an even integer.

Proof

(1) The map to $\mathcal {M}_{\tau =d}(\mathrm {PU}(1,3))$ is surjective since $\mathrm {PU}(1,3)$-Higgs bundles with $\tau =d$ lift to $\mathrm {U}(1,3)$-Higgs bundles of the form $(L,W,\beta ,\gamma )$ with $3\deg (L)-\deg (W)=2d$ (see (7.41)). After twisting with a line bundle if necessary, we can assume that $\deg (L)=\deg (W)=d$. Furthermore, we can assume that $L=\det (W)$ since if not, then twisting by a square root of $\det (V)\otimes L^{-1}$ will make it so. The assertion that $\tau$ must be an even integer is clear from the definitions of the maps (7.41) and (7.38).

(2) As in (1), any $\mathrm {PU}(1,3)$-Higgs bundles with $\tau =4d$ lift to $\mathrm {U}(1,3)$-Higgs bundles of the form $(\det (W),W,\beta ,\gamma )$. Such a Higgs bundle is in the image of (7.38) if and only if $\deg (\det (W))$ is even. This condition is satisfied precisely when $\deg (W)=2d$.

(3) This follows from the fact that the map is defined by (7.39) in which $\det (V)=L^2$ and hence $\deg (V)=2\deg (L)$. $\square$

Expressed in terms of the corresponding surface group representations, Proposition 7.29 gives conditions under which reductive surface group representations into $\mathrm {PU}(1,3), \mathrm {SO}^*(6)$ or $\mathrm {SU}(1,3)$ lift from one group to another.

Proposition 7.30

• (1) A reductive surface group representation into $\mathrm {PU}(1,3)$ lifts to a representation into $\mathrm {SO}^*(6)$ if and only if the Toledo invariant of the associated $\mathrm {PU}(1,3)$-Higgs bundle is an integer.

• (2) A reductive surface group representation into $\mathrm {PU}(1,3)$ lifts to a representation into $\mathrm {SU}(1,3)$ if and only if the Toledo invariant of the associated $\mathrm {PU}(1,3)$-Higgs bundle is an even integer.

• (3) A reductive surface group representation into $\mathrm {SO}^*(6)$ lifts to a representation into $\mathrm {SU}(1,3)$ if and only if the Toledo invariant of the associated $\mathrm {SO}^*(6)$-Higgs bundle is an even integer.

7.3.1 Maximal components

By Proposition 3.27, the moduli spaces $\mathcal {M}_{d}(\mathrm {SO}^*(6))$ are non-empty for $|d|\le 2g-2$. The maximal components are thus those with $|d|=2g-2$ (and these are connected by Theorem 5.2). We discuss here only the case $d=2g-2$, but the case $d=-(2g-2)$ is analogous.

By Theorem 4.5, the moduli spaces $\mathcal {M}_{2g-2} (\mathrm {SO}^*(6))$ exhibit a rigidity which leads to the factorization

$$\begin{aligned} \mathcal {M}_{2g-2} (\mathrm {SO}^*(6))\simeq \mathcal {M}_{2g-2}(\mathrm {SO}^*(4))\times \mathrm {Jac}(X) \end{aligned}$$

(7.44)

given by

$$\begin{aligned}&\displaystyle (V,\beta ,\gamma )=(V_{\perp },\beta ,\gamma )\oplus \ker (\gamma ).\end{aligned}$$

(7.45)

$$\begin{aligned}&\displaystyle T_4: M_0(2)\times \mathcal {M}_{g-1}(\mathrm {SL}(2,\mathbb {R}))\longrightarrow \mathcal {M}_{2g-2}(\mathrm {SO}^*(4)) \end{aligned}$$

(7.46)

given by

$$\begin{aligned} (U, (K^{1/2},\beta ,1_{K^{1/2}}))\mapsto (U\otimes K^{1/2},\Omega \otimes \beta ,\Omega ^*\otimes 1_{K^{1/2}})\ , \end{aligned}$$

(7.47)

where $\beta \in H^0(X,K^2)$, $1_{K^{1/2}}$ denotes the identity map on $K^{1/2}$, and $\Omega : U^*\simeq U$ is as in Lemma 7.11.

We thus get a $2^{2g}$-fold covering of $\mathcal {M}_{2g-2} (\mathrm {SO}^*(6))$

$$\begin{aligned} T_6: M_0(2)\times \mathcal {M}_{g-1}(\mathrm {SL}(2,\mathbb {R}))\times \mathrm {Jac}(X) \longrightarrow \mathcal {M}_{2g-2}(\mathrm {SO}^*(6)) . \end{aligned}$$

(7.48)

Remark 7.31

A choice of $K^{1/2}$ defines a section for the map $T_4$—and hence for $T_6$—and picks out a Teichmüller component of $\mathcal {M}_{g-1}(\mathrm {SL}(2,\mathbb {R}))$.

We get a different description of the maximal components if we exploit the embedding of $\mathcal {M}_{2g-2}(\mathrm {SO}^*(6))$ in $\mathcal {M}_{(2g-2,2g-2)}(\mathrm {U}(1,3))$ given by Corollary 7.23:

$$\begin{aligned} (V,{\beta },{\gamma })\mapsto (\det (V),V, \tilde{\beta },\tilde{\gamma }). \end{aligned}$$

As shown in [6], the component $\mathcal {M}_{(2g-2,2g-2)}(\mathrm {U}(1,3))$ has maximal Toledo invariant for $\mathrm {U}(1,3)$-Higgs bundles and, moreover, this moduli space itself exhibits a rigidity. Indeed (see Theorem 3.32 in [6]) the component $\mathcal {M}_{(2g-2,2g-2)}(\mathrm {U}(1,3))$ factors as

$$\begin{aligned} \mathcal {M}_{(2g-2,2g-2)} (\mathrm {U}(1,3))\simeq \mathcal {M}_{(2g-2,0)}(\mathrm {U}(1,1))\times M_{2g-2}(2), \end{aligned}$$

(7.49)

where $\mathcal {M}_{(2g-2,0)}$ denotes the moduli space of $\mathrm {U}(1,1)$-Higgs bundles $(L,M,\beta ,\gamma )$ with $\deg (L)=2g-2$ and $\deg (M)=0$, and $M_{d}(2)$ denotes the moduli space of polystable rank 2 bundles of degree $d$. The factorization is given by

$$\begin{aligned} (L,W,\beta ,\gamma )=(L,L\otimes K^{-1},\beta ,1_L)\oplus Q \end{aligned}$$

(7.50)

where $W=L\otimes K^{-1}\oplus Q$. Notice that $L=\det (W)$ if and only if $\det (Q)=K$. In that case, for any choice of $K^{-1/2}$ the determinant of $Q\otimes K^{1/2}$ is trivial and we can write $Q=U\otimes K^{1/2}$ with $\det (U)=\mathcal {O}$.

The image of the embedding of $\mathcal {M}_{2g-2}(\mathrm {SO}^*(6))$ in $\mathcal {M}_{(2g-2,2g-2)}(\mathrm {U}(1,3))$ is thus characterized by the condition that $Q=U\otimes K^{1/2}$ with $\det (U)=\mathcal {O}$ in (7.50). We define

$$\begin{aligned} M_K(2)=\{Q\in M_{2g-2}(2)\ |\ \det (Q)=K\}. \end{aligned}$$

(7.51)

The Toledo invariant is maximal for $\mathcal {M}_{(2g-2,0)}(\mathrm {U}(1,1))$ and hence, by Proposition 3.30 in [6] we can identify $\mathcal {M}_{(2g-2,0)}(\mathrm {U}(1,1))$ with the moduli space of degree zero, $K^2$-twisted $\mathbb {C}^*$-Higgs bundles⁷, i.e.

$$\begin{aligned} \mathcal {M}_{(2g-2,0)}(\mathrm {U}(1,1))&\xrightarrow {\simeq } \mathrm {Jac}(X)\times H^0(X,K^2),\nonumber \\ (L,M,\beta ,\gamma )&\mapsto (L,\beta \circ \gamma ). \end{aligned}$$

(7.52)

Putting together (7.51), (7.52) and (7.49) we thus get an identification of the image of $\mathcal {M}_{2g-2}(\mathrm {SO}^*(6))$ in $\mathcal {M}_{(2g-2,2g-2)}(\mathrm {U}(1,3))$ as

$$\begin{aligned} \mathcal {M}_{2g-2}(\mathrm {SO}^*(6))\simeq \mathrm {Jac}(X)\times H^0(X,K^2)\times M_K(2). \end{aligned}$$

(7.53)

Comparing (7.48) and (7.53) we see that the two descriptions match up via the map

$$\begin{aligned} (U, (K^{1/2},\beta , 1), L_0)\longrightarrow (L_0,\beta , Q=U\otimes K^{1/2}). \end{aligned}$$

The fibers of this map are the $2^{2g}$ points of order 2 in $\mathrm {Jac}(X)$.

We note finally that the dimension of $\mathcal {M}_{\pm (2g-2)}(\mathrm {SO}^*(6))$ can be computed from the isomorphism (7.44). We find $\dim (\mathcal {M}_{\pm (2g-2)}(\mathrm {SO}^*(6))=7g-6$ whereas the expected dimension is $15(g-1)$.

Appendix: \(G\)-Higgs bundles for other groups

We collect here some basic results about $G$-Higgs bundles for groups other than $\mathrm {SO}^*(2n)$ which play a role in our analysis of $\mathrm {SO}^*(2n)$-Higgs bundles. The groups include three complex reductive groups ($\mathrm {GL}(n,\mathbb {C})$, $\mathrm {SL}(n,\mathbb {C})$ and $\mathrm {SO}(n,\mathbb {C})$) and two non-compact real forms ($\mathrm {U}(p,q)$ and $\mathrm {U}^*(2n)$).

In all cases the basic definitions of stability properties follow from the general definition formulated for $G$-Higgs bundles in [15].

1.1 The groups $\mathrm {GL}(n,\mathbb {C}), \mathrm {SL}(n,\mathbb {C})$ and $\mathrm {SO}(n,\mathbb {C})$

We begin by recalling how the notion of $G$-Higgs bundle specializes when $G$ is a complex group. In this case, the complexified isotropy representation is just the adjoint representation of $G$ on $\mathfrak {g}$. Thus, a $G$-Higgs bundle for a complex group $G$ is a pair $(E,\varphi )$, where $E \rightarrow X$ is a holomorphic principal $G$-bundle and $\varphi \in H^0(X,{{\mathrm{Ad}}}E\otimes K)$; here ${{\mathrm{Ad}}}E = E\times _{{{\mathrm{Ad}}}}\mathfrak {g}$ is the adjoint bundle of $E$. We shall use this observation for all three groups considered in this section.

Consider first the case of $G=\mathrm {GL}(n,\mathbb {C})$. A $\mathrm {GL}(n,\mathbb {C})$-Higgs bundle may be viewed as a pair consisting of a rank $n$ holomorphic vector bundle $E$ over $X$ and a holomorphic section

$$\begin{aligned} \Phi \in H^0(X,K \otimes {{\mathrm{End}}}E). \end{aligned}$$

We refer the reader to [15] for the general statement of the stability conditions for $\mathrm {GL}(n,\mathbb {C})$-Higgs bundles. The notions of (semi-,poly-)stability in this case are equivalent to the original notions given by Hitchin in [23] (see [15]). Denote by $\mu (E) = \deg (E) / {{\mathrm{rk}}}(E)$ the slope of $E$.

Proposition 8.1

A $\mathrm {GL}(n,\mathbb {C})$-Higgs bundle $(E,\Phi )$ is semistable if and only if for any subbundle $E'\subset E$ such that $\Phi (E')\subset E' \otimes K$ we have $\mu (E')\le \mu (E)$. Furthermore, $(E,\Phi )$ is stable if for any nonzero and strict subbundle $E'\subset E$ such that $\Phi (E')\subset E' \otimes K$ we have $\mu (E')<\mu (E)$. Finally, $(E,\Phi )$ is polystable if it is semistable and for each subbundle $E'\subset E$ such that $\Phi (E')\subset E' \otimes K$ and $\mu (E')=\mu (E)$ there is another subbundle $E''\subset E$ satisfying $\Phi (E'')\subset E'' \otimes K$ and $E=E'\oplus E''$. As a consequence $(E,\Phi )=\oplus (E_i,\Phi _i)$ where $(E_i,\Phi _i)$ is a stable $\mathrm {GL}(n_i,\mathbb {C})$-Higgs bundle with $\mu (E_i)=\mu (E)$.

The group $\mathrm {SL}(n,\mathbb {C})$ is the subgroup of $\mathrm {GL}(n,\mathbb {C})$ defined by the usual condition on the determinant. A $\mathrm {SL}(n,\mathbb {C})$-Higgs bundle may thus be viewed as a $\mathrm {GL}(n,\mathbb {C})$-Higgs bundle $(E,\Phi )$ with the extra conditions that $E$ is endowed with a trivialization $\det E\simeq \mathcal {O}$ and $\Phi \in H^0(X,K\otimes {{\mathrm{End}}}_0 E)$ where ${{\mathrm{End}}}_0 E$ denotes the bundle of traceless endomorphisms of $E$. The (semi-,poly-)stability condition is the same as the one for $\mathrm {GL}(n,\mathbb {C})$-Higgs bundles given in Proposition 8.1.

Finally we consider the case $G=\mathrm {SO}(n,\mathbb {C})$. A principal $\mathrm {SO}(n,\mathbb {C})$-bundle on $X$ corresponds to a rank $n$ holomorphic orthogonal vector bundle $(E,Q)$, where $E$ is a rank $n$ vector bundle and $Q$ is a holomorphic section of $S^2E^*$ whose restriction to each fibre of $E$ is non degenerate. The adjoint bundle can be identified with $\Lambda ^2_QE \subset {{\mathrm{End}}}(E)$, the subbundle of ${{\mathrm{End}}}(E)$ consisting of endomorphisms which are skew-symmetric with respect to $Q$. A $\mathrm {SO}(n,\mathbb {C})$-Higgs bundle is thus a pair consisting of a rank $n$ holomorphic orthogonal vector bundle $(E,Q)$ over $X$ and a section

$$\begin{aligned} \Phi \in H^0(X,\Lambda ^2_Q E\otimes K). \end{aligned}$$

The general notions of (semi-,poly-)stability specialize in the case of $\mathrm {SO}(n,\mathbb {C})$-Higgs bundles to the following (see [1, 2]).

Proposition 8.2

A $\mathrm {SO}(n,\mathbb {C})$-Higgs bundle $((E,Q),\Phi )$ is semistable if and only if for any isotropic subbundle $E'\subset E$ such that $\Phi (E')\subset K\otimes E'$ we have $\deg E'\le 0$. Furthermore, $((E,Q),\Phi )$ is stable if for any nonzero and strict isotropic subbundle $0\ne E'\subset E$ such that $\Phi (E')\subset K\otimes E'$ we have $\deg E'<0$. Finally, $((E,Q),\Phi )$ is polystable if it is semistable and for any nonzero and strict isotropic subbundle $E'\subset E$ such that $\Phi (E')\subset K\otimes E'$ and $\deg E'=0$ there is a coisotropic subbundle $E''\subset E$ such that $\Phi (E'')\subset K\otimes E''$ and $E=E'\oplus E''$.

Remark 8.3

Recall that if $(E,Q)$ is an orthogonal vector bundle, a subbundle $E'\subset E$ is said to be isotropic if the restriction of $Q$ to $E'$ is identically zero, and coisotropic if $E'^{\perp _Q}$ is isotropic.

Remark 8.4

For complex groups $G$, Definition 2.7 implies that a $G$-Higgs bundle $(E,\varphi )$ is simple if $\mathrm{Aut }(E,\varphi )=Z(H^{\mathbb {C}})$. For $G=\mathrm {GL}(n,\mathbb {C})$ or $\mathrm {SL}(n,\mathbb {C})$ it is well known that stability implies simplicity. This is not so for $\mathrm {SO}(n,\mathbb {C})$-Higgs bundles. For instance it is possible for a stable $\mathrm {SO}(n,\mathbb {C})$-Higgs bundle to decompose as sum of stable $\mathrm {SO}(n_i,\mathbb {C})$-Higgs bundles (with $\Sigma n_i=n$). In all cases though, the Higgs bundles which are stable and simple represent smooth points in their moduli spaces (see Proposition 2.14).

1.2 The groups $\mathrm {U}(p,q)$ and $\mathrm {U}^*(2n)$

1.2.1 $\mathrm {U}(p,q)$-Higgs bundles

The maximal compact subgroups of $\mathrm {U}(p,q)$ are isomorphic to $H=\mathrm {U}(p)\times \mathrm {U}(q)$ and hence $H^{\mathbb {C}}=\mathrm {GL}(p,\mathbb {C})\times \mathrm {GL}(q,\mathbb {C})$. The complexified isotropy representation space is $\mathfrak {m}^{\mathbb {C}}={{\mathrm{Hom}}}(\mathbb {C}^q,\mathbb {C}^p)\oplus {{\mathrm{Hom}}}(\mathbb {C}^p,\mathbb {C}^q)$. A $\mathrm {U}(p,q)$-Higgs bundle may thus be described by the data $(V, W,\varphi =\beta +\gamma )$, where $V$ and $W$ are vector bundles of rank $p$ and $q$, respectively, $\beta \in H^0(X,{{\mathrm{Hom}}}(W,V)\otimes K)$ and $\gamma \in H^0(X,{{\mathrm{Hom}}}(V,W)\otimes K)$.

The following proposition gives the simplified stability conditions for $\mathrm {U}(p,q)$-Higgs bundles. It can be proved using arguments similar to the ones for other real groups (cf. Sect. 3.2 and [15, Section 4]).

Proposition 8.5

A $\mathrm {U}(p,q)$-Higgs bundle $(V,W,\varphi =\beta +\gamma )$ is semistable if

$$\begin{aligned} \mu (V'\oplus W') \le \mu (V\oplus W), \end{aligned}$$

is satisfied for all $\varphi$-invariant pairs of subbundles $V'\subset V$ and $W'\subset W$, i.e. for pairs such that

$$\begin{aligned} \beta&:W'\longrightarrow V'\otimes K\nonumber \\ \gamma&:V'\longrightarrow W'\otimes K. \end{aligned}$$

A $\mathrm {U}(p,q)$-Higgs bundle $(V,W,\varphi )$ is stable if the slope inequality is strict whenever $V'\oplus W'$ is a proper non-zero $\varphi$-invariant subbundle of $V\oplus W$.

A $\mathrm {U}(p,q)$-Higgs bundle $(V,W,\varphi )$ is polystable if it is semistable and for any $\varphi$-invariant pair of subbundles $V'\subset V$ and $W'\subset W$ satisfying $\mu (V'\oplus W')=\mu (V\oplus W)$ there is another $\varphi$-invariant pair of subbundles $V''\subset V$ and $W''\subset W$ such that $V=V'\oplus V''\ \mathrm {and}\ W=W'\oplus W''$. As a consequence there is a decomposition

$$\begin{aligned} (V,W,\beta ,\gamma )=\bigoplus (V_i,W_i,\beta _i,\gamma _i), \end{aligned}$$

where $V=\bigoplus V_i$, $W=\bigoplus W_i, \beta =\Sigma \beta _i ,\gamma =\Sigma \gamma _i$ and $(V_i,W_i,\beta _i,\gamma _i)$ is a stable $\mathrm {U}(p_i,q_i)$-Higgs bundle with $\mu (V_i\oplus W_i)=\mu (V\oplus W)$.

Remark 8.6

In the case $q=0$, the group is $\mathrm {U}(p)$ and hence $\varphi = 0$. Thus a $\mathrm {U}(p)$-Higgs bundle is an ordinary vector bundle. Proposition 8.5 shows that in this case the $\mathrm {U}(p,q)$-Higgs bundles stability condition coincides with the usual one for vector bundles.

1.2.2 $\mathrm {U}^*(2n)$-Higgs bundles

The group $\mathrm {U}^*(2n)$ is a non-compact real form of $\mathrm {GL}(2n,\mathbb {C})$ consisting of matrices $M$ verifying that $\bar{M}J_n=J_nM$ where $J_n=\left( \begin{matrix} 0 &{} I_n \\ -I_n &{} 0 \end{matrix}\right) .$ A maximal compact subgroup of $\mathrm {U}^*(2n)$ is the compact symplectic group $\mathrm {Sp}(2n)$ (or, equivalently, the group of $n\times n$ quaternionic unitary matrices), whose complexification is $\mathrm {Sp}(2n,\mathbb {C})$, the complex symplectic group. The group $\mathrm {U}^*(2n)$ is the non-compact dual of $\mathrm {U}(2n)$, in the sense that the non-compact symmetric space $\mathrm {U}^*(2n)/\mathrm {Sp}(2n)$ is the dual of the compact symmetric space $\mathrm {U}(2n)/\mathrm {Sp}(2n)$ in Cartan’s classification of symmetric spaces (cf. [22]).

The corresponding Cartan decomposition of the complex Lie algebra is

$$\begin{aligned} \mathfrak {gl}(2n,\mathbb {C})=\mathfrak {sp}(2n,\mathbb {C})\oplus \mathfrak {m}^{\mathbb {C}}, \end{aligned}$$

where $\mathfrak {m}^{\mathbb {C}}=\{A\in \mathfrak {gl}(2n,\mathbb {C})\;|\;A^tJ_n=J_nA\}$. Hence a $\mathrm {U}^*(2n)$-Higgs bundle over $X$ is a pair $(E,\varphi )$, where $E$ is a holomorphic $\mathrm {Sp}(2n,\mathbb {C})$-principal bundle and the Higgs field $\varphi$ is a global holomorphic section of $E\times _{\mathrm {Sp}(2n,\mathbb {C})}\mathfrak {m}^{\mathbb {C}}\otimes K$.

Given a symplectic vector bundle $(W,\Omega )$, denote by $S^{2}_{\Omega }W$ the bundle of endomorphisms $\xi$ of $W$ which are symmetric with respect to $\Omega$ i.e. such that $\Omega (\xi \,\cdot ,\cdot )=\Omega (\cdot ,\xi \,\cdot )$. In terms of vector bundles, we have that a $\mathrm {U}^*(2n)$-Higgs bundle over $X$ is a triple $(W,\Omega ,\varphi )$, where $W$ is a holomorphic vector bundle of rank $2n$, $\Omega \in H^0(X,\Lambda ^2W^*)$ is a symplectic form on $W$, and the Higgs field $\varphi \in H^0(X,S_{\Omega }^2 W\otimes K)$ is a $K$-twisted endomorphism $W\rightarrow W \otimes K$, symmetric with respect to $\Omega$.

Given the symplectic form $\Omega$, we have the usual skew-symmetric isomorphism

$$\begin{aligned} \omega :W\mathop {\longrightarrow }\limits ^{\simeq }W^* \end{aligned}$$

given by

$$\begin{aligned} \omega (v)=\Omega (v,-). \end{aligned}$$

The map $f\mapsto f\omega ^{-1}$ defines an isomorphism between $S^2_\Omega W$ and $\Lambda ^2W$. Hence we can think of a $\mathrm {U}^*(2n)$-Higgs bundle as a triple $(W,\Omega ,\varphi )$ with $\varphi \in H^0(X,S_{\Omega }^2W\otimes K)$ or as a triple $(W,\Omega ,\tilde{\varphi })$ with $\tilde{\varphi }\in H^0(X,\Lambda ^2W\otimes K)$ given by

$$\begin{aligned} \tilde{\varphi }=\varphi \omega ^{-1}. \end{aligned}$$

(8.1)

The general (semi-,poly-)stability conditions for $\mathrm {U}^*(2n)$-Higgs bundles are studied in [16], where simplified conditions (similarly to the case of other groups) are given. We have the following ([16, Proposition 3.6]).

Proposition 8.7

A $\mathrm {U}^*(2n)$-Higgs bundle $(W,\Omega ,\varphi )$ semistable if and only if $\deg W'\le 0$ for any isotropic and $\varphi$-invariant subbundle $W'\subset W$.

A $\mathrm {U}^*(2n)$-Higgs bundle $(W,\Omega ,\varphi )$ is stable if and only if it is semistable and $\deg W'<0$ for any isotropic and $\varphi$-invariant strict subbundle $0\ne W' \subset W$.

The $\mathrm {U}^*(2n)$-Higgs bundle $(W,\Omega ,\varphi )$ is polystable if and only if it is semistable and, for any isotropic (respectively coisotropic) and $\varphi$-invariant strict subbundle $0\ne W'\subset W$ such that $\deg W'=0$, there is another coisotropic (respectively isotropic) and $\varphi$-invariant subbundle $0\ne W''\subset W$ such that $W\simeq W'\oplus W''$.