The Oort Conjecture for Shimura Curves of Small Unitary Rank

Abstract

We prove that a Shimura curve in the Siegel modular variety is not generically contained in the open Torelli locus as long as the rank of unitary part in its canonical Higgs bundle satisfies a numerical upper bound. As an application we show that the Coleman–Oort conjecture holds for Shimura curves associated with partial corestriction upon a suitable choice of parameters, which generalizes a construction due to Mumford.

1 Introduction

In this paper we study the Oort conjecture for some Shimura curves. We prefer the following equivalent formulation rather than the original one, also known as the Coleman–Oort conjecture:

Conjecture 1.1

Let $\mathcal {T}_g^\circ$ be the open Torelli locus in the Siegel modular variety $\mathcal {A}_g$. Then for g sufficiently large, the intersection of $\mathcal {T}_g^\circ$ with any Shimura subvariety $M\subsetneq \mathcal {A}_g$ of strictly positive dimension is NOT Zariski open in M.

Here $\mathcal {T}_g^\circ$ is the scheme-theoretic image of the Torelli morphism $\mathcal {M}_g\rightarrow \mathcal {A}_g$, where $\mathcal {A}_g$ is the Siegel modular variety with suitably chosen level structure so that corresponding moduli functor is representable, and the similar constraint on level structure is understood for $\mathcal {M}_g$.

The André–Oort conjecture holds for $\mathcal {A}_g$, regardless of the level structures, cf. [17], and it implies the equivalence of Conjecture 1.1 with the original conjecture of Coleman claiming the finiteness of CM points in $\mathcal {T}_g^\circ$ for g sufficiently large. In the particular case of dimension one, the Oort conjecture predicts that for g sufficiently large, $\mathcal {T}_g^\circ$ meets any Shimura curve in at most finitely many points.

Previous works, cf. [3, 4], etc., have proved the conjecture for certain Shimura subvarieties whose canonical Higgs bundles contain large unitary subbundles, and the main technique is motivated from surface fibration. Roughly speaking, if a Shimura subvariety M of dimension $>0$ is contained generically in $\mathcal {T}_g^\circ$, then one finds a curve C of generic position lying in $M\cap \mathcal {T}_g^\circ$, such that

• the inclusion $C\subset \mathcal {T}_g^\circ$ lifts C into a curve in $\mathcal {M}_g$ which, after suitable compactification and normalization, supports a semi-stable surface fibration ${\overline{f}}:{\overline{S}}\rightarrow {\overline{C}}$, and inequality of Xiao’s type bounds the maximal slope in the Hodge bundle ${\overline{f}}_*\omega _{{\overline{S}}/{\overline{C}}}$ in terms of the degree of ${\overline{f}}_*\omega _{{\overline{S}}/{\overline{C}}}$, which leads to an upper bound on the rank of unitary part in the Hodge bundle;

• on the other hand, the Hodge bundle above is induced from the Hodge bundle on C due to the modular interpretation of $C\hookrightarrow M\hookrightarrow \mathcal {A}_g$, and a fine description of the symplectic representation defining $M\hookrightarrow \mathcal {A}_g$ leads to an explicit lower bound of the unitary part in the Hodge bundle.

Combining these two ingredients one reaches the generic exclusion of Shimura curves when the unitary part in the canonical Higgs bundle is large.

In this paper we are interested in the case of Shimura curves whose canonical Higgs bundles only contain a small portion of unitary subbundles:

Theorem 1.2

Let $C\subseteq \mathcal {A}_g$ be any Shimura curve whose associated logarithmic Higgs bundle $(E_{\overline{C}},\theta _{\overline{C}})$ decomposes as

$$\begin{aligned} (E_{\overline{C}},\theta _{\overline{C}})=(A_{\overline{C}},\theta _{\overline{C}}|_{A_{\overline{C}}}) \oplus (F_{\overline{C}},0), \end{aligned}$$

where $A_{\overline{C}}^{1,0}$ is ample and $F_{\overline{C}}$ is the maximal unitary flat subbundle. Assume that ${\mathrm{rank\,}}F_{\overline{C}}^{1,0}\le \frac{2g-22}{7}$ (equivalently, ${\mathrm{rank\,}}A_{\overline{C}}^{1,0}>\frac{5g+22}{7}$). Then C is not contained generically in the Torelli locus $\mathcal {T}_g$ of curves of genus g.

Mumford has considered embeddings of Shimura curves into $\mathcal {A}_g$ using symplectic representations defined by corestriction of quaternion algebras, which is different from the construction using restriction of scalars. In this paper we consider an interpolation between restriction and corestriction, called “partial corestriction,” and the unitary portion in the Higgs bundles on Shimura curves embedded in this way could be small upon suitable choice of parameters, terminology and details for which are given in Sect. 4:

Corollary 1.3

Let $C\hookrightarrow \mathcal {A}_g$ be a Shimura curve defined in the following way:

• (i) either C is associated with a quaternion F-algebra over a totally real field F, or C is associated with an Hermitian form $h:E^2\times E^2\rightarrow E$ for some CM field E of totally real part F, and the embedding $C\hookrightarrow \mathcal {A}_g$ is associated with the partial corestriction of index t;

• (ii) or C is associated with a quaternion division E-algebra for some CM field E of totally real part F and some Hermitian pairing $A\times A\rightarrow A$, and $C\hookrightarrow \mathcal {A}_g$ is associated with the partial corestriction of index t.

Here t is a positive integer not exceeding the degree $d=[F:\mathbb {Q}]$. Then C is NOT contained generically in $\mathcal {T}_g^\circ$ as long as $\frac{t}{d}>\frac{5}{7}+\frac{22}{7g}$, where $g=2^t\left( {\begin{array}{c}d\\ t\end{array}}\right)$ in case (i) and $g=4^t\left( {\begin{array}{c}d\\ t\end{array}}\right)$ in case (ii).

The notion of partial corestriction is defined in Sect. 4 as an interpolation between the usual notions of restriction and corestriction of semi-simple algebras, and t is a positive integer not exceeding d.

The material is organized as follows. Section 2 recalls preliminaries on Shimura curves and Higgs bundles, including a description of forms of $\mathbf {SL}_{2,F}$ that could define Shimura curves. Section 3 contains the proof of the main theorem on the generic exclusion of Shimura curves from $\mathcal {T}_g^\circ$ with small unitary part in the canonical Higgs bundle. Section 4 discusses the notion of partial corestriction, the related Hermitian forms giving rise to symplectic representations, and ends with an elementary computation for Corollary 1.3.

1.1 Notations

We write $\mathbb {S}$ for Deligne’s torus $\mathrm {Res}_{\mathbb {C}/\mathbb {R}}\mathbb {G}_\mathrm {m}$. If $\sigma :k\rightarrow K$ is a homomorphism of rings and $\mathbf {H}$ is a k-scheme, then we write $\mathbf {H}(K,\sigma )$ for the set of K-valued points of $\mathbf {H}$ with respect to the structure of k-algebra given by $\sigma$; this is often the case when we need to distinct the k-structures on $\mathbf {H}(K)$ involving different embeddings of fields $k\hookrightarrow K$.

2 Preliminaries on Shimura Curves and Higgs Bundles

2.1 Shimura Curves and Quaternion Algebras

We refer to [3] for our convention on notions such as Shimura (sub)data and Shimura (sub)varieties. In particular, the Siegel modular variety $\mathcal {A}_g:=\Gamma \backslash \mathcal {H}_g^+$ is the connected Shimura variety associated with the connected Shimura datum $(\mathrm {GSp}_{2g},\mathcal {H}_g;\mathcal {H}_g^+)$, where $\mathcal {H}_g^+$ is the Siegel upper half space of genus g, and we choose $\Gamma$ to be a torsion-free congruence subgroup in $\mathrm {Sp}_{2g}(\mathbb {Z})$, so that the smooth quasi-projective variety $\mathcal {A}_g$ represents the corresponding moduli problem (with level-$\Gamma$ structure).

By Shimura curves, we mean connected Shimura varieties of dimension one. Such a curve is defined by a connected Shimura datum $(\mathbf {G},X;X^+)$, where $X^+$ is a one-dimensional Hermitian symmetric domain, namely the Poincaré upper half plane $\mathcal {H}^+$.

This already forces $\mathbf {G}^\mathrm {der}$ to be a $\mathbb {Q}$-simple $\mathbb {Q}$-group, and according to [5] it has to be of the form $\mathrm {Res}_{F/\mathbb {Q}}\mathbf {H}$ for some F-group $\mathbf {H}$ which remains simple after the base change $F\hookrightarrow {\bar{F}}$. Here F is a totally number field, and ${\bar{F}}$ is a fixed separable closure of F. Since $X^+$ is the Poincaré upper half plane, the F-group $\mathbf {H}$ has to be a simple F-group of type $A_1$, i.e., it is an F-form of either $\mathbf {SL}_{2,F}$ or $\mathbf {PGL}_{2,F}$. Moreover, among the real embeddings $\{\tau \}$ of $F\hookrightarrow \mathbb {R}$, there is exactly one embedding giving rise to a non-compact Lie group $\mathbf {H}(\mathbb {R},\tau )$ isomorphic to $\mathbf {SL}_2(\mathbb {R})$ or $\mathbf {PGL}_2(\mathbb {R})$, and the other embeddings $\tau '$ lead to compact Lie groups $\mathbf {H}(\mathbb {R},\tau ')$.

One is mainly interested in Shimura curves C inside a Siegel modular variety $\mathcal {A}_g$ defined by some inclusion of the form $(\mathbf {G},X;X^+)\hookrightarrow (\mathrm {GSp}_{2g},\mathcal {H}_g;\mathcal {H}_g^+)$, and the modular interpretation of the inclusion $C\hookrightarrow \mathcal {A}_g$ gives the canonical $\mathbb {Q}$-VHS of weight 1 on C, whose associated Higgs bundle $\mathcal {E}_C$ plays an essential role in our work. Various properties of the Higgs bundles are read from the algebraic representation $\mathbf {G}\hookrightarrow \mathrm {GSp}_{2g}$. If the F-group $\mathbf {H}$ above were an F-form of $\mathbf {PGL}_{2,F}$, then the algebraic representation $\mathbf {G}\hookrightarrow \mathrm {GSp}_{2g}$ would not produce $\mathbb {Q}$-VHS of odd weights. Hence $\mathbf {H}$ has to be an F-form of $\mathbf {SL}_{2,F}$.

The following classification of forms of $\mathbf {SL}_{2,F}$ is found in [16], divided into the inner and outer cases. For simplicity we use the following convention of notation:

• (i) If B is a finite-dimensional unital k-algebra (not necessarily commutative), k being a fixed base field, we write $\mathbb {G}_\mathrm {m}^{B/k}$ for the linear k-group sending a k-algebra R to $(B\otimes _kR)^\times$, and sometimes we write $\mathbb {G}_\mathrm {m}^B$ if k is clear from the context. If $k'\subset k$ is a subfield with $[k:k']<\infty$, then we have $\mathbb {G}_\mathrm {m}^{B/k'}\simeq \mathrm {Res}_{k/k'}\mathbb {G}_\mathrm {m}^{B/k}$.

• (ii) If B is a central simple k-algebra of dimension $m^2$, then $\mathbb {G}_\mathrm {m}^{B/k}$ is a k-form of $\mathbf {GL}_{m,k}$, endowed with the reduced norm $\mathrm {Nm}_{B/k}\mathbb {G}^B_\mathrm {m}\rightarrow \mathbb {G}_{\mathrm {m},k}$ which is a k-form of the determinant map $\det :\mathbf {GL}_m\rightarrow \mathbb {G}_\mathrm {m}$, and we denote its kernel by ${\mathbb {U}}^{B/k}$, which is a k-form of $\mathbf {SL}_m$.

We also write $\mathbb {H}$ for Hamilton’s quaternion division $\mathbb {R}$-algebra, associated with which we have $\mathbf {SU}_2\simeq {\mathbb {U}}^{\mathbb {H}/\mathbb {R}}$.

Case (1):

The inner case of the classification involves a central simple F-algebra A, and we have $\mathbf {H}\simeq {\mathbb {U}}_\mathrm {m}^{A/F}$. Note that A splits over F, i.e., $A\simeq \mathrm {Mat}_2(F)$, if and only if $\mathbf {H}$ splits over F, i.e., $\mathbf {H}\simeq \mathbf {SL}_{2,F}$.

Case (2):

The outer case involves an Hermitian form, and we recall the more general description for outer forms of $\mathbf {SL}_{mn,F}$: there exists some quadratic extension E of F, a central simple E-algebra D of E-dimension $n^2$ which is a skew field, endowed with an involution of second kind (i.e., restricting to the F-conjugate on E), and an Hermitian pairing $H:D^{\oplus m}\times D^{\oplus m}\rightarrow D$ of Hermitian matrix $\Phi$ under the natural D-basis of $D^{\oplus m}$, such that the following group functor $\mathbf {U}_\Phi$ is an F-form of $\mathbf {GL}_{mn,F}$: an F-algebra R is sent to

$$\begin{aligned} \left\{ g\in \mathrm {Mat}_m(D):g^*\Phi g=\Phi ,\ g\ \mathrm {invertible}\right\} \end{aligned}$$

and its derived part is an F-form of $\mathbf {SL}_{mn,F}$. The constraint $mn=2$ thus leads to:

• (2-1) either $n=1$ and $m=2$: namely H is an Hermitian form $E^2\times E^2\rightarrow E$, $(v,w)\mapsto \bar{v}^t\Phi w$ for some Hermitian matrix $\Phi =\bar{\Phi }^t$;

• (2-2) or $n=2$ and $m=1$: namely D is a quaternion division E-algebra and $H:D\times D\rightarrow D$ is of the form $(a,b)\mapsto a^*\delta b$ for some $\delta =\delta ^*$ in D.

Note that in (2-2), D is of dimension 4 over E, and the composition $h=\mathrm {tr}_{D/E}\circ H$ of H with the reduced trace of D over E is an Hermitian form $D\times D\rightarrow E$, and the outer form in this case is an F-subgroup of the unitary F-group $\mathbf {U}_h$.

In our case of interest for Shimura curves, we have a $\mathbb {Q}$-group $\mathbf {G}$ with $\mathbf {G}^\mathrm {der}=\mathrm {Res}_{F/\mathbb {Q}}\mathbf {H}$ for F a totally real field of degree d, such that $\mathbf {G}^\mathrm {der}(\mathbb {R})^+$ defines a connected Hermitian symmetric domain of dimension 1, namely the Poincaré upper half plane. Write $\tau _1,\ldots ,\tau _d$ for the real embeddings of F, we have $\mathbf {G}^\mathrm {der}(\mathbb {R})\simeq \prod _{i=1,\ldots ,d}\mathbf {H}(\mathbb {R},\tau _i)$, where $\mathbf {H}(\mathbb {R},\tau _i)$ stands for the $\mathbb {R}$-points of $\mathbf {H}$ with respect to the F-structure $\tau _i:F\hookrightarrow \mathbb {R}$ on $\mathbb {R}$, and we may rearrange the subscripts so that

• $\mathbf {H}(\mathbb {R},\tau _1)=\mathbf {SL}_2(\mathbb {R})$;

• $\mathbf {H}(\mathbb {R},\tau _i)=\mathbf {SU}_2(\mathbb {R})$ for $i=2,\ldots ,d$.

Thus for the F-forms described above for $\mathbf {H}$, we have:

• (1) in the inner case, A is an quaternion F-algebra such that $A\otimes _{F,\tau _1}\mathbb {R}\simeq \mathrm {Mat}_2(\mathbb {R})$ and $A\otimes _{F,\tau _i}\mathbb {R}\simeq \mathbb {H}$ for $i=2,\ldots ,d$;

• (2) in the outer case:

  • (2-1) either $\mathbf {H}$ is associated with an Hermitian form $h:E^2\times E^2\rightarrow E$ which is indefinite (i.e., of signature (1, 1)) along $\tau _1$, giving rise to a factor $\mathbf {SU}(1,1)\simeq \mathbf {SL}_{2,\mathbb {R}}$, and definite along $\tau _2,\ldots ,\tau _d$ giving rise to the compact factor $\mathbf {SU}_2(\mathbb {R})$; note that K has to be purely imaginary over F in this case, and thus K is a CM field of real part F;

  • (2-2) or $\mathbf {H}$ is associated with an Hermitian form $H:A\times A\rightarrow A$ with A a quaternion division E-algebra whose signatures follow the same pattern as above: becoming $\mathbf {SU}(1,1)$ along $\tau _1$ and $\mathbf {SU}_2(\mathbb {R})$ along $\tau _2,\ldots ,\tau _d$, and E is a CM field.

For the construction of Shimura data $(\mathbf {G},X;X^+)$, we follow the construction in [9] so that $\mathbf {G}$ only differ from $\mathrm {Res}_{F/\mathbb {Q}}\mathbf {H}$ by a central $\mathbb {Q}$-torus. For example, in the outer case $H:D\times D\rightarrow D$, we may compose H with the reduced trace $D\rightarrow E$ and get an Hermitian form $h:D\times D\rightarrow E$ whose imaginary part is a symplectic F-form $D\times D\rightarrow F$. The F-group of unitary similitude $\mathbf {H}'$ of H differs from $\mathbf {H}$ by a central F-torus $\mathbb {G}_{\mathrm {m}F}$. Taking trace again from F to $\mathbb {Q}$ gives a symplectic $\mathbb {Q}$-form on D (viewing as a $\mathbb {Q}$-vector space), and we may take $\mathbf {G}$ to be the $\mathbb {Q}$-subgroup of $\mathrm {Res}_{F/\mathbb {Q}}\mathbf {H}'$ which only differs from $\mathrm {Res}_{F/\mathbb {Q}}\mathbf {H}$ by the central $\mathbb {Q}$-torus $\mathbb {G}_{\mathrm {m}\mathbb {Q}}$ in $\mathrm {Res}_{F/\mathbb {Q}}\mathbb {G}_{\mathrm {m}F}$. This is often used in the construction of Shimura subdata of $(\mathrm {GSp}_{2g},\mathcal {H}_g;\mathcal {H}_g^+)$, cf. [9].

Finally, it should be mentioned that quaternion algebras and Shimura curves from Case (1) can be reduced to Case (2-1): for the application we have in mind, the field F in Case (1) is a totally real number field, and by choosing E a CM number field of totally real part F such that $A\otimes _FE\simeq \mathrm {Mat}_2(E)$, we obtain an involution of second kind on $\mathrm {Mat}_2(E)$ which is the transposed conjugate on coordinates with fixed part isomorphic to A, and ${\mathbb {U}}^{A/F}$ can be identified with the special unitary F-group $\mathbf {SU}_h$ of the standard Hermitian form $E^2\times E^2\rightarrow E, (u,v)\mapsto \bar{u}^tv$. It even suffices to take E to be $F\otimes _\mathbb {Q}K$ with K some imaginary quadratic number field, similar to the construction used in [2], which realizes Shimura curves in Case (1) as a Shimura curve of PEL-type in Case (2).

2.2 Decomposition of Higgs Bundles

Let $(V,\psi )$ be a symplectic $\mathbb {Q}$-space giving rise to a Shimura datum $(\mathrm {GSp}_V,\mathcal {H}_V;\mathcal {H}_V^+)$ and the Siegel modular variety $\mathcal {A}_V=\Gamma \backslash \mathcal {H}_V^+$ for suitable torsion-free congruence subgroup $\Gamma$ in $\mathrm {Sp}_V(\mathbb {Q})$, and we may assume that $\Gamma$ stabilizes a $\mathbb {Z}$-structure $V_\mathbb {Z}$ for V. For $M\hookrightarrow \mathcal {A}_V$ a Shimura subvariety defined by some subdatum $(\mathbf {G},X;X^+)$, the modular interpretation of $\mathcal {A}_V$ gives a universal abelian M-scheme $f:A\rightarrow M$ and a $\mathbb {Q}$-PVHS on M, whose underlying local system in $\mathbb {Q}$-vector spaces $\mathbb {V}_M=Rf_*\mathbb {Q}_A$ is determined by the representation of fundamental group $\pi _1(M)\rightarrow \mathbf {GL}_V(\mathbb {Q})$, which in turn is determined by the algebraic representation $\mathbf {G}^\mathrm {der}\rightarrow \mathrm {Sp}_V$. The Hodge filtration of ${\mathcal {V}}_M:=\mathbb {V}_M\otimes _{\mathbb {Q}_M}\mathcal {O}_M$ gives

$$\begin{aligned} 0\rightarrow R^0f_*\Omega _{M/A}\rightarrow {\mathcal {V}}_M\rightarrow R^1f_*\mathcal {O}_A\rightarrow 0, \end{aligned}$$

and we have the canonical Higgs bundle $\mathcal {E}_M=\mathcal {E}_M^{0,1}\oplus \mathcal {E}_M^{1,0}$ with $\mathcal {E}_{M}^{0,1}=R^1f_*\mathcal {O}_A$ and $\mathcal {E}_M^{1,0}=R^0f_*\Omega ^1_{A/M}$. More generally, the graded quotient of the Hodge filtration $F^\cdot {\mathcal {V}}$ for any PVHS ${\mathcal {V}}$ on M is a Higgs bundle on M, and for smoothly compactified Shimura varieties (by joining boundary divisors using toroidal compactification) we have a similar notion of Higgs bundles with logarithmic poles.

The theory of Simpson correspondence implies that, upon suitable choice of smooth compactification, there is an equivalence of categories between finite-dimensional $\mathbb {C}$-linear representations of $\pi _1(M)$ and logarithmic Higgs bundles on M. In particular, Higgs subbundles of $\mathcal {E}_M$ (or rather, its logarithmic version over smooth compactification) associated with sub-$\mathbb {R}$-PVHS of $({\mathcal {V}}_M,\mathbb {V}_M\otimes _{\mathbb {Q}_M}\mathbb {R}_M)$ corresponds to $\mathbb {R}$-linear subrepresentations of $\pi _1(M)\rightarrow \mathbf {GL}_\mathbb {R}(V_\mathbb {R})$, which are in turn characterized by algebraic subrepresentations of $\mathbf {G}^\mathrm {der}_\mathbb {R}\hookrightarrow \mathrm {Sp}_{V_\mathbb {R},\mathbb {R}}$. Such a Higgs subbundle is unitary if and only if the corresponding $\mathbb {R}$-subrepresentation factors through a compact linear $\mathbb {R}$-group.

3 Generic Exclusion of Shimura Curves

In this section we prove Theorem 1.2 by contradiction. The strategy is along a similar way as that of [12], where the special case with trivial unitary part has been considered. Assume that such a Shimura curve C is contained generically in the Torelli locus $\mathcal {T}_g$. Since suitable level structures are pre-attached in our setting, one may represent C by a semi-stable family $f:\,\overline{S} \rightarrow \overline{B}$ of curves of genus g as in [12, § 3]. The contradiction is deduced by studying the slope inequality of such a semi-stable family together with the logarithmic Miyaoka–Yau inequality.

3.1 Setups

Given such a Shimura curve C contained generically in the Torelli locus $\mathcal {T}_g$, one obtains as in [12, § 3] a semi-stable family $f:\,\overline{S} \rightarrow \overline{B}$ of curves of genus g representing C by taking suitable level structure into account. The natural map

$$\begin{aligned} \bar{f}^*A^{1,0}_{\overline{B}} \hookrightarrow \bar{f}^{*}\bar{f}_{*}\omega _{\overline{S}/\overline{B}} \longrightarrow \omega _{\overline{S}/\overline{B}} \end{aligned}$$

induces a rational map $\overline{\Phi }_{A}:\,\overline{S} \dashrightarrow \mathbb {P}_{\overline{B}}(A^{1,0}_{\overline{B}})$ over $\overline{B}$. By resolution of possible singularities on the image and a suitable sequence of blowing-ups $\sigma :\,\widetilde{S} \rightarrow \overline{S}$ (which does not affect the general fiber $\overline{F}$), the above rational map becomes a morphism $\widetilde{\Phi }:\, \widetilde{S} \rightarrow \widetilde{Y}$.

By contracting vertical exceptional curves, we may assume that $\tilde{h}$ is relatively minimal. Let $M\in {\mathrm{Pic\,}}(\widetilde{S})$ be the moving part of the pull-back of the tautological line bundle H on $\mathbb {P}_{\overline{B}}(A^{1,0}_{\overline{B}})$. Denote by $\Gamma$ the image of the general fiber $\overline{F}$, and $\gamma =g(\Gamma )$. Then

$$\begin{aligned} h^0(\overline{F}, M|_{\overline{F}}) \ge h^0(\Gamma , H|_{\Gamma }) \ge r:={\mathrm{rank\,}}A^{1,0}_{\overline{B}}={\mathrm{rank\,}}A^{1,0}_{\overline{C}}. \end{aligned}$$

(3.1)

Lemma 3.1

If $r>\frac{5g+22}{7}$, then $\deg \widetilde{\Phi }\le 2$. Moreover, if $\deg \widetilde{\Phi } = 2$, then $\tilde{h}$ is locally trivial with

$$\begin{aligned} \gamma <\frac{2g-22}{7}, \end{aligned}$$

(3.2)

where $\gamma$ is the genus of a general fiber of $\tilde{h}$.

Proof

Let $\Phi _0:\,\overline{F} \rightarrow \Gamma \subseteq \mathbb {P}^{r-1}$ be the restricted morphism on the general fiber. Then it is clear that $\deg (\widetilde{\Phi })=\deg (\Phi _0)$. By construction,

$$\begin{aligned} \begin{aligned} 2g-2\ge \deg (M|_{\overline{F}})&\,=\deg (\Phi _0)\cdot \deg (H|_{\Gamma })\\&\,\ge \deg (\Phi _0)\cdot \big (h^0(\Gamma ,H|_{\Gamma })-1\big )\\&\,> \deg (\Phi _0)\cdot \Big (\frac{5g+22}{7}-1\Big ). \end{aligned} \end{aligned}$$

Hence $\deg (\widetilde{\Phi })=\deg (\Phi _0)\le 2$ as required. Moreover, if $\deg \widetilde{\Phi } = 2$ and $\tilde{h}$ is locally trivial, then the Hodge bundle $\tilde{h}_*\omega _{\widetilde{Y}/\overline{B}}$ is flat of rank $\gamma$, and hence (3.2) follows since the pull-back of $\tilde{h}_*\omega _{\widetilde{Y}/\overline{B}}$ under $\widetilde{\Phi }^*$ is a direct summand of $\bar{f}_*\omega _{\overline{S}/\overline{B}}$. Therefore, it remains to show that $\tilde{h}$ is locally trivial if $\deg \widetilde{\Phi } = 2$.

The decomposition

$$\begin{aligned} \bar{f}_*\omega _{\overline{S}/ \overline{B}} = A^{1,0}_{\overline{B}} \oplus F^{1,0}_{\overline{B}} \end{aligned}$$

(3.3)

corresponds to a decomposition on $V:=H^0(\overline{F},\omega _{\overline{F}})$:

$$\begin{aligned} V=V_{A}\oplus V_{F}. \end{aligned}$$

The map $\Phi _0$ is exactly the map defined by the linear subsystem $\Lambda _A \subseteq |\omega _{\overline{F}}|$ corresponding to $V_A$. If $\deg (\Phi _0)=2$, it induces an involution $\tau$ on $\overline{F}$. It is clear that the subsheaf $A^{1,0}_{\overline{B}}$, which is the ample part, is invariant under the induced action of $\tau$ on $\bar{f}_*\omega _{\overline{S}/\overline{B}}$. Hence $V_A$ is also invariant under the induced action of $\tau$ on $H^0(\overline{F},\omega _{\overline{F}})$, i.e., $\tau ^*(\omega )\in V_A$ for any $\omega \in V_A$.

We claim that the induced action of $\tau$ on $V_A$ is the multiplication by $(-\,1)$. We prove the claim by contradiction. Since $V_A$ is invariant under the induced action of $\tau$, it admits a basis consisting of eigenvectors of $\tau$. Let $\{\omega _1,\ldots ,\omega _r\}$ be such a basis of $V_A$, and $D_i={\mathrm{div}}\,(\omega _i)$. Let $D_0$ be the fixed part of $\Lambda _A$. Then there exists a divisor $\Delta _i$ on $\Gamma$ for each $1\le i\le r$ such that

$$\begin{aligned} D_i=D_0+\Phi _0^*(\Delta _i). \end{aligned}$$

Since $\tau$ is an involution, without loss of generality we may assume that $\tau ^*\omega _1=\omega _1$ if the claim does not hold. It follows that $\omega _1=\Phi _0^*(\omega _1')$ for some $\omega _1'\in H^0(\Gamma ,\omega _{\Gamma })$. Equivalently,

$$\begin{aligned} D_1=R+\Phi _0^*\left( D_1'\right) , \end{aligned}$$

where $D_1'={\mathrm{div}}\,(\omega _1')$ and R is the ramification divisor of $\Phi _0$. Therefore,

$$\begin{aligned} D_0+\Phi _0^*(\Delta _1)=R+\Phi _0^*\left( D_1'\right) . \end{aligned}$$

Taking any point $p\in R$ and $q=\Phi _0(p)$, let $a\ge 0$ be the multiplicity of p in $D_0$, and b and c be the multiplicities of q in $\Delta _1$ and $D_1'$ respectively. Then the above equality implies that $a+2b=1+2c$. It follows that $a\ge 1$. Hence $D_0\ge R$; equivalently, $V_A \subseteq \Phi _0^*H^0(\Gamma ,\omega _{\Gamma })$. In particular, $r=\dim V_A \le \gamma$, which is a contradiction.

Coming back to the proof, the above claim implies that the induced action of $\tau$ on $A^{1,0}_{\overline{B}}$ is also the multiplication by $(-\,1)$. Note that $\tilde{h}_*\omega _{\widetilde{Y}/\overline{B}}$ can be naturally viewed as a subsheaf of $\bar{f}_*\omega _{\overline{S}/ \overline{B}}$ and that $\tau$ acts trivially on $\tilde{h}_*\omega _{\widetilde{Y}/\overline{B}}$. Hence $\tilde{h}_*\omega _{\widetilde{Y}/\overline{B}} \subseteq F^{1,0}_{\overline{B}}$. In particular, $\deg (\tilde{h}_*\omega _{\widetilde{Y}/\overline{B}})=0$, and hence $\tilde{h}$ is locally trivial as required. $\square$

Thus the proof of Theorem 1.2 is divided into two cases according to the value of $\deg \widetilde{\Phi }$.

3.2 The Case when $\deg (\widetilde{\Phi })=1$

Proof of Theorem 1.2

when $\deg (\widetilde{\Phi })=1$. We mimic the proof as in [12]. It suffices to prove the following strict Arakelov inequality for the semi-stable fibration $\bar{f}:\,\overline{S} \rightarrow \overline{B}$ representing the Shimura curve C generically in $\mathcal {T}_g$.

$$\begin{aligned} \deg \bar{f} _*\omega _{\overline{S}/\overline{B}}<\frac{r}{2}\cdot \left( \deg \Omega ^1_{\overline{B}}(\log \Delta _{nc})-|\Lambda |\right) , \qquad \text {where}~r={\mathrm{rank\,}}A^{1,0}_{\overline{C}}, \end{aligned}$$

(3.4)

where $\Upsilon _{nc} \rightarrow \Delta _{nc}$ is the singular locus of $\bar{f}$ with non-compact Jacobian, and $\Lambda \subseteq B$ is the ramification divisor of the double cover $j_B:\,B \rightarrow C$ as in [12, § 3].

According to [12, Theorem 4.2] together with Theorem 3.2 below, one obtains

$$\begin{aligned} \deg \bar{f} _*\omega _{\overline{S}/\overline{B}} \le \frac{4r(g-1)}{3g+7r-12}\cdot \left( \deg \Omega ^1_{\overline{B}}(\log \Delta _{nc})-|\Lambda |\right) +\frac{4r}{3g+7r-12}\cdot |\Lambda |. \end{aligned}$$

Note that $\omega _{\overline{S}/\overline{B}}^2\le 12 \deg \bar{f} _*\omega _{\overline{S}/\overline{B}}$ by Noether’s equality. Hence from (3.5) it follows that

$$\begin{aligned} |\Lambda |\le \frac{17r-3g+12}{4r(g-2)}\cdot \deg \bar{f} _*\omega _{\overline{S}/\overline{B}}. \end{aligned}$$

Therefore,

$$\begin{aligned} \begin{aligned} \deg \bar{f} _*\omega _{\overline{S}/\overline{B}} \le&\frac{4r(g-1)(g-2)}{(7g-31)r+3(g-1)(g-4)}\cdot \left( \deg \Omega ^1_{\overline{B}}(\log \Delta _{nc})-|\Lambda |\right) ,\\ =&\left( \frac{r}{2}-\frac{r}{2}\cdot \frac{(7g-31)r-(g-1)(5g-4)}{(7g-31)r+3(g-1)(g-4)}\right) \\&\cdot \left( \deg \Omega ^1_{\overline{B}}(\log \Delta _{nc})-|\Lambda |\right) ,\\ <&\frac{r}{2}\cdot \left( \deg \Omega ^1_{\overline{B}}(\log \Delta _{nc})-|\Lambda |\right) , \qquad \text {since }r>\frac{5g+22}{7}. \end{aligned} \end{aligned}$$

This proves (3.4). $\square$

To finish the proof, it remains to prove the following slope inequality.

Theorem 3.2

Let $\bar{f}:\, \overline{S} \rightarrow \overline{B}$ be the family of semi-stable genus-g curves representing a Shimura curve $C\Subset \mathcal {T}_{g}$, and $\widetilde{\Phi }:\, \widetilde{S} \rightarrow \widetilde{Y}$ be the morphism induced by $A_{\overline{B}}^{1,0}$ as above. Assume that $\deg \widetilde{\Phi }=1$. Then

$$\begin{aligned} \begin{aligned} \omega _{\overline{S}/\overline{B}}^2 ~\ge&~ \frac{7r+3g-12}{2r}\deg \bar{f}_*\omega _{\overline{S}/\overline{B}} +2(g-2)\cdot |\Lambda |\\&+\sum _{p\in \Delta _{ct}\,\cap \,\Lambda } 2\big (l_h(F_p)+l_1(F_p)-1\big ) +\sum _{p\in \Delta _{ct} \setminus \Lambda }\big (3l_h(F_p)+2l_1(F_p)-3\big ). \end{aligned} \end{aligned}$$

(3.5)

Here $\Lambda \subseteq B$ is the ramification divisor of the double cover $j_B:\,B \rightarrow C$ as in [12, § 3], $\Upsilon _{ct}\rightarrow \Delta _{ct}$ are the singular fibers with compact Jacobians, $l_i(F_p)$ is the number of components of geometric genus equal to i in $F_p$, and $l_h(F_p)=\sum _{i\ge 2}l_i(F_p)$.

Proof

The proof is quit similar to [12, Theorem 5.2]. It is based on analyzing the following natural multiplication

$$\begin{aligned} \varrho :\,S^2\left( \bar{f}_*\omega _{\overline{S}/\overline{B}}\right) \longrightarrow \bar{f}_*\big (\omega _{\overline{S}/\overline{B}}^{\otimes 2}\big ), \end{aligned}$$

(3.6)

where $S^2\left( \bar{f}_*\omega _{\overline{S}/\overline{B}}\right)$ is the symmetric power of $\bar{f}_*\omega _{\overline{S}/\overline{B}}$.

As $\deg \widetilde{\Phi }=1$, $\bar{f}$ is non-hyperelliptic. Hence the morphism $\varrho$ in (3.6) is generically surjective by Noether’s theorem (cf. [1, § III.2]). Let $\mathcal {I}$ be the image of $\varrho$. Then one gets an exact sequence as below:

$$\begin{aligned} 0 \longrightarrow \mathcal {I} \longrightarrow \bar{f}_*\big (\omega _{\overline{S}/\overline{B}}^{\otimes 2}\big ) \longrightarrow \mathcal {S}\longrightarrow 0, \end{aligned}$$

where $\mathcal {S}$ is the cokernel of $\varrho$, which is a torsion sheaf. So

$$\begin{aligned} \deg \bar{f}_*\big (\omega _{\overline{S}/\overline{B}}^{\otimes 2}\big )=\deg \mathcal {I} +\deg \mathcal {S}. \end{aligned}$$

Hence it suffices to prove

$$\begin{aligned} \deg (\mathcal {I}) \ge \frac{9r+3g-12}{2r}\deg \bar{f}_*\omega _{\overline{S}/\overline{B}}. \end{aligned}$$

(3.7)

Let

$$\begin{aligned} \varrho _1:~S^2A^{1,0}_{\overline{B}} \hookrightarrow S^2\left( \bar{f}_*\omega _{\overline{S}/\overline{B}}\right) \longrightarrow \mathcal {I}, \end{aligned}$$

and

$$\begin{aligned} \varrho _2:~A^{1,0}_{\overline{B}}\otimes \bar{f}_*\omega _{\overline{S}/\overline{B}} \longrightarrow S^2\left( \bar{f}_*\omega _{\overline{S}/\overline{B}}\right) \longrightarrow \mathcal {I}, \end{aligned}$$

be the induced maps. Denote by $\tilde{\mu }_1=\frac{2\deg \bar{f}_*\omega _{\overline{S}/\overline{B}}}{r}$ and $\tilde{\mu }_2=\frac{\deg \bar{f}_*\omega _{\overline{S}/\overline{B}}}{r}$. Then $\mu _f(\mathrm{Im}(\varrho _1))\ge \tilde{\mu }_1$ and $\mu _f(\mathrm{Im}(\varrho _2))\ge \tilde{\mu }_2$, where

$$\begin{aligned} \mu _f(\mathcal {E})=\max \{\deg \mathcal {F}~|~\mathcal {E} \otimes \mathcal {F}^{\vee } ~\text {is semi-positive}\},~ \forall ~\text {locally free sheaf }\mathcal {E}. \end{aligned}$$

Since the map $\Phi _0$, as well as $\widetilde{\Phi }$, is birational, one has

$$\begin{aligned} \left\{ \begin{aligned} {\mathrm{rank\,}}(\mathrm{Im}(\varrho _1))&\,\ge 3r-3, \qquad \text {by the Clifford plus theorem, cf. } [1,\S \,\mathrm{III.2}],\\ {\mathrm{rank\,}}(\mathrm{Im}(\varrho _2))&\,\ge g+\deg (M|_{\overline{F}})+r-1-s, \quad \text {by }[14, {\mathrm{Lemma}}\, 3.10],\\&\,\ge g+\frac{g+3s-4}{2}+r-1-s, ~ ~\text {by Castelnuovo's bound, cf. }[1,\S \,\mathrm{III.2}],\\&\,\ge \frac{3g+3r-6}{2},\qquad \text {where}~s:=h^0\big (\overline{F},M|_{\overline{F}}\big )\ge r. \end{aligned}\right. \end{aligned}$$

Hence (3.7) follows from the next proposition. $\square$

The next proposition was stated for $f_*\big (\omega _{X/B}^{\otimes 2}\big )$ in [13, Proposition 2.5]. But we note that the proof is still valid if we replace $f_*\big (\omega _{X/B}^{\otimes 2}\big )$ by the image $\mathcal {I}$.

Proposition 3.3

Let $\tilde{\mu }_1>\cdots >\tilde{\mu }_k\ge 0$ (resp. $0<\tilde{r}_1<\cdots <\tilde{r}_k\le 3g-3$) be any decreasing (resp. increasing) sequence of rational (resp. integer) numbers. Assume that there exists a subsheaf $\mathcal {F}_i \subseteq \mathcal {I}$ such that $\mu _f(\mathcal {F}_i)\ge \tilde{\mu }_i$ and ${\mathrm{rank\,}}\mathcal {F}_i \ge \tilde{r}_i$ for each i. Then

$$\begin{aligned} \deg (\mathcal {I}) \ge \sum _{i=1}^k\tilde{r}_i(\tilde{\mu }_i-\tilde{\mu }_{i+1}), \qquad \text {where}~\tilde{\mu }_{k+1}=0. \end{aligned}$$

3.3 The Case when $\deg (\widetilde{\Phi })=2$

In this case, we have to consider the slope of semi-stable double cover fibrations. We first recall some facts about the double cover fibrations from [14].

One starts from a relatively minimal fibration $\tilde{h}:\,\widetilde{Y}\rightarrow \overline{B}$ of genus $\gamma >0$ and a reduced divisor $\widetilde{R}\in {\mathrm{Pic\,}}(\widetilde{Y})$ with $\widetilde{R}\cdot \widetilde{\Gamma }=2g+2-4\gamma$ and $\mathcal {O}_{\widetilde{Y}}(\widetilde{R}) \equiv \widetilde{L}^{\otimes 2}$ for some line bundle $\widetilde{L}$, where $\widetilde{\Gamma }$ is a general fiber of $\tilde{h}$. From these data one constructs a double cover $\pi _0:\,S_0\rightarrow Y_0=\widetilde{Y}$. By the canonical resolution, one gets a smooth fibered surface $\tilde{f}:\,\widetilde{S} \rightarrow \overline{B}$, and by contracting further $(-\,1)$-curves contained in the fibers one obtains a relatively minimal fibration $\bar{f}:\,\overline{X} \rightarrow \overline{B}$ of genus g. We call $\bar{f}$ a double cover fibration of type $(g,\gamma )$ (Fig. 1).

Fig. 1

Canonical resolution

Here $\psi _i$s are successive blowing-ups resolving the singularities of $\widetilde{R}$, and $\pi _{i}:\,S_i \rightarrow Y_i$ is the double cover determined by $\mathcal {O}_{Y_i}(R_i) \equiv L_i^{\otimes 2}$ with

$$\begin{aligned} R_i=\psi _i^*(R_{i-1})-2[m_{i-1}/2]\, \mathcal {E}_i,\qquad L_i=\psi _i^*(L_{i-1})\otimes \mathcal {O}_{Y_i}\left( \mathcal {E}_i^{-[m_{i-1}/2]}\right) , \end{aligned}$$

where $\mathcal {E}_i$ is the exceptional divisor of $\psi _i$, $m_{i-1}$ is the multiplicity of the singular point $y_{i-1}$ in $R_{i-1}$ (also called the multiplicity of the blowing-up $\psi _i$), [ ] stands for the integral part, $R_0=\widetilde{R}$ and $L_0=\widetilde{L}$. A singularity $y_j \in R_{j}\subseteq Y_{j}$ is said to be infinitely near to $y_{i}\in R_{i}\subseteq Y_{i}$ ($j>i$), if $\psi _{i+1}\circ \cdots \circ \psi _j(y_j)=y_{i}\,.$

We remark that the order of these blowing-ups contained in $\psi =\psi _1\circ \cdots \circ \psi _t$ is not unique. If $y_{i-1}$ is a singular point of $R_{i-1}$ of odd multiplicity $2k+1$ ($k\ge 1$) and there is a unique singular point y of $R_i$ on the exceptional curve $\mathcal {E}_i$ of multiplicity $2k+2$, then we always assume that $\psi _{i+1}: Y_{i+1} \rightarrow Y_{i}$ is a blowing-up at $y_i=y$. We call such a pair $(y_{i-1},y_{i})$ a singularity of R of type $(2k+1 \rightarrow 2k+1)$, and $y_{i-1}$ (resp. $y_i$) the first (resp. second) component.

Definition 3.4

([14, Definition 4.1]) For any singular fiber F of f and $j\ge 2$, we define

• if j is odd, $s_j(F)$ equals the number of $(j\rightarrow j)$ type singularities of R over the image f(F);

• if j is even, $s_j(F)$ equals the number of singularities of multiplicity j or $j+1$ of R over the image f(F), neither belonging to the second component of type $(j-1 \rightarrow j-1)$ singularities nor to the first component of type $(j+1 \rightarrow j+1)$ singularities.

Let $h_i:\,Y_i\rightarrow \overline{B}$ be the induced fibration, $\omega _{h_i}=\omega _{Y_i}\otimes h_i^*\omega _{\overline{B}}^{-1}$ and $R_t'= R_t \setminus V_t$, where $V_t$ is the union of vertical isolated $(-\,2)$-curves in $R_t$. Here a curve $C\subseteq R_t$ is called to be isolated in $R_t$, if there is no other curve $C'\subseteq R_t$ such that $C\cap C' \ne \emptyset$. We define

$$\begin{aligned} \begin{aligned} s_2&:=\left( \omega _{h_t}+R_t'\right) \cdot R_t'+2\sum _{F ~\text {is singular}} s_2(F),\\ s_j&:=\sum _{F ~\text {is singular}} s_j(F),\qquad \qquad \forall ~ j\ge 3. \end{aligned} \end{aligned}$$

Note that the contraction $\psi$ is unique since $\gamma >0$ (although the order of these blowing-ups contained in $\psi$ is not unique). Hence the invariants $s_j$s are well defined.

Theorem 3.5

([14, Theorem 4.3]) Let $\bar{f}$ be a double cover fibration of type $(g, \gamma )$. Then

$$\begin{aligned} \begin{aligned} (2g+1-3\gamma )\omega _{\overline{S}/\overline{B}}^2~=&~\,x\cdot \frac{\omega _{\widetilde{Y}/\overline{B}}^2}{\gamma -1}+yT+zs_2+\sum _{k\ge 1} a_ks_{2k+1}+\sum _{k\ge 2}b_ks_{2k},\\ (2g+1-3\gamma )\deg \bar{f}_*\omega _{\overline{S}/\overline{B}}~=&~\, \bar{x}\cdot \frac{\omega _{\widetilde{Y}/\overline{B}}^2}{\gamma -1}+2(2g+1-3\gamma )\deg \tilde{h}_*\omega _{\widetilde{Y}/\overline{B}}+\bar{y}T\\&\,+\bar{z}s_2\!-\!\frac{2g+1\!-\!3\gamma }{4}\cdot n_2 \!+\!\sum _{k\ge 1} \bar{a}_ks_{2k+1}+\sum _{k\ge 2}\bar{b}_ks_{2k}, \end{aligned} \end{aligned}$$

where we set $\frac{\omega _{\widetilde{Y}/\overline{B}}^2}{\gamma -1}=0$ if $\gamma =1$,  $n_2$ the number of vertical isolated $(-\,2)$-curves of $\widetilde{R}$, and

$$\begin{aligned} x= & {} \frac{(3g+1-4\gamma )(g-1)}{2},\quad y=\frac{3}{2},\quad ~z=g-1;\\ \bar{x}= & {} \frac{(g+1-2\gamma )^2}{8},\quad \bar{y}=\frac{1}{8},\quad \bar{z}=\frac{g-\gamma }{4}.\\ a_k= & {} 12\bar{a}_k-(2g+1-3\gamma ),\quad b_k=12\bar{b}_k-2(2g+1-3\gamma ),\\ \bar{a}_k= & {} k\big (g-1+(k-1)(\gamma -1)\big ),\quad \bar{b}_k=\frac{k\big (g-1+(k-2)(\gamma -1)\big )}{2},\\ T= & {} -\frac{\big ((g+1-2\gamma )\omega _{\widetilde{Y}/\overline{B}}-(\gamma -1)R\big )^2}{\gamma -1}-2(\gamma -1)n_2\ge 0. \end{aligned}$$

In the case when the fibration $\tilde{h}$ is locally trivial, it is clear that

$$\begin{aligned} n_2=\omega _{\widetilde{Y}/\overline{B}}^2=\deg \tilde{h}_*\omega _{\widetilde{Y}/\overline{B}}=0. \end{aligned}$$

(3.8)

Moreover, similar to [10], one proves that

Lemma 3.6

Let $\bar{f}:\,\overline{S} \rightarrow \overline{B}$ be a double cover fibration as above. If $\tilde{h}$ is locally trivial and $\bar{f}$ is semi-stable, then

$$\begin{aligned} \delta _0=s_2+\sum _{k\ge 2} 2s_{2k},\qquad \delta _i=s_{2i+1}+s_{2(g-i)+1},~\text {if }i>0. \end{aligned}$$

(3.9)

Here $\delta _i$ is the number of nodes of type i contained in the singular fibers of $\bar{f}$, and a node p in a singular fiber $\overline{F}$ of $\bar{f}$ is called of type 0 (resp. i with $0<i\le g/2$) if the partial normalization of $\overline{F}$ at p is connected (resp. consists of two connected components of arithmetic genera i and $g-i$).

Proposition 3.7

Let $\bar{f}:\,\overline{S} \rightarrow \overline{B}$ be a double cover fibration as above. Assume that $\tilde{h}$ is locally trivial and $\bar{f}$ is semi-stable.

• (1) Let $\delta _h=\sum \limits _{i\ge 2}\delta _i$. Then it holds

  $$\begin{aligned} \omega _{\overline{S}/\overline{B}}^2\ge \frac{4(g-1)}{g-\gamma }\deg \bar{f} _*\omega _{\overline{S}/\overline{B}}+3\delta _1+7\delta _h. \end{aligned}$$

  (3.10)

• (2) If $\delta _0=0$, then

  $$\begin{aligned} \omega _{\overline{S}/\overline{B}}^2\ge \frac{3(2g+3\gamma -5)}{g-1}\deg \bar{f} _*\omega _{\overline{S}/\overline{B}}+2\delta _1+5\delta _h. \end{aligned}$$

  (3.11)

• (3) If $\gamma <q_{\bar{f}}$, then

  $$\begin{aligned} \omega _{\overline{S}/\overline{B}}^2\ge \frac{1}{3}\Big (\frac{8(g-1)}{g-\gamma }+\frac{6g+4\gamma -10}{g-1}\Big )\deg \bar{f} _*\omega _{\overline{S}/\overline{B}}+2\delta _1+\frac{14}{3}\delta _h. \end{aligned}$$

  (3.12)

Proof

The first two inequalities follow directly from Theorem 3.5 together with (3.8) and (3.9). For the third one, one first notes that when $\gamma <q_{\bar{f}}$, the double cover fibration $\bar{f}$ is irregular, and hence by [14, Theorem 4.10],

$$\begin{aligned} \omega _{\overline{S}/\overline{B}}^2\ge \frac{6g+4\gamma -10}{g-1}\deg \bar{f} _*\omega _{\overline{S}/\overline{B}}. \end{aligned}$$

Combining this with (3.10), one proves (3.12). $\square$

We can now finish the proof of Theorem 1.2.

Proof of Theorem 1.2

when $\deg (\widetilde{\Phi })=2$. Let $f:\,\overline{S} \rightarrow \overline{B}$ be the semi-stable family of curves of genus g representing the Shimura curve $C\Subset \mathcal {T}_g$ as above, and assume that $\deg (\widetilde{\Phi })=2$, where $\widetilde{\Phi }$ is the map induced by $A_{\overline{B}}^{1,0}$ in Sect. 3.1. By Lemma 3.1, $\bar{f}$ is a double cover fibration and the quotient fibration $\tilde{h}$ is locally trivial. Note that ${\mathrm{rank\,}}A_{\overline{C}}^{1,0}\le g$. Hence the assumption ${\mathrm{rank\,}}A_{\overline{C}}^{1,0}>\frac{5g+22}{7}$ implies in particular that $g\ge 12$. Thus we may assume that $\gamma >0$ by [12], where $\gamma$ is the genus of a general fiber of $\tilde{h}$. We claim that

Claim 3.8

The family $\bar{f}:\, \overline{S} \rightarrow \overline{B}$ contains no hyperelliptic fiber with compact Jacobian, equivalently, it holds $\Lambda =\emptyset$, where $\Lambda \subseteq B$ is the ramification divisor of the double cover $j_B:\,B \rightarrow C$ as in ([12], § 3).

Proof of Claim 3.8

We prove by contradiction. Assume there exists a hyperelliptic fiber $\overline{F}_0$ with compact Jacobian. Let $\tau$ and $\iota$ be the two involutions on $\overline{F}_0$, such that $\overline{F}_0/\langle \tau \rangle$ is of arithmetic genus $\gamma$ and $\overline{F}_0/\langle \iota \rangle$ is of arithmetic genus zero.

First it is easy to see that $\overline{F}_0$ is not smooth; in fact, if $\overline{F}_0$ is smooth, it admits two different double covers to $\Gamma$ and $\mathbb {P}^1$ respectively, and hence $g\le 2\gamma +1$ by the Castelnuovo–Severi inequality (cf. [8, Exercise V.1.9]), a contradiction to (3.2).

We now assume that $\overline{F}_0$ is singular, and let $\overline{F}_0=\sum C_i$. Since $\overline{F}_0$ has a compact Jacobian, $\overline{F}_0$ is a tree of smooth curves. We divide the proof into two cases according to whether there exists a component $C_i$ of positive genus such that $C_i$ is invariant under $\tau$ with $g\big (C_i/\langle \tau \rangle \big )=0$.

If there is no component of positive genus invariant under $\tau$ with $g\big (C_i/\langle \tau \rangle \big )=0$, then $\overline{F}_0$ contains at most two components whose genera are positive since the quotient $\overline{F}_0/\langle \tau \rangle$ contains only one component whose genus is positive (its genus is $\gamma$). If there is only one such a component, then again $g\le 2\gamma +1$ by the Castelnuovo–Severi inequality, which gives a contradiction; if there are two such components, then $\tau$ exchanges them and both of them are of genus $\gamma$, and hence $g=2\gamma$, which again contradicts (3.2).

We assume now that there is one component, saying $C_1$, of positive genus invariant under $\tau$ with $g\big (C_1/\langle \tau \rangle \big )=0$. We first claim that $g(C_1)=1$; indeed, since the quotient $C_1/\langle \tau \rangle$ is also of genus zero, it follows that $C_1$ admits two different double to $\mathbb {P}^1$, which implies $g(C_1)=1$. As two different involutions on $C_1$, $\tau$ and $\iota$ have no common fixed points. According to the proof of [12, Lemma 5.7], every point in $\overline{(\overline{F}_0\setminus C_1)} \cap C_1$ is a fixed point of $\iota$. It follows that there is no component except $C_1$ invariant under $\tau$. Thus, besides $C_1$, $\overline{F}_0$ consists of exactly two other components, saying $C_2$ and $C_3$, of positive genus, which are the pre-image of the component of positive genus in $\overline{F}_0/\langle \tau \rangle$. Therefore, $g(C_2)=g(C_3)=\gamma$, and $g=2\gamma +1$. It again contradicts (3.2). $\square$

Coming back to the proof of Theorem 1.2. We consider first the case when the Shimura curve C is compact, i.e., the family $\bar{f}:\, \overline{S} \rightarrow \overline{B}$ has no singular fiber with non-compact Jacobian, or equivalently $\delta _0=0$. By (3.11) together with [12, Theorem 4.1], one obtains that

$$\begin{aligned} \deg \bar{f} _*\omega _{\overline{S}/\overline{B}}\le \frac{2(g-1)^2}{3(2g+3\gamma -5)} \deg \Omega ^1_{\overline{B}} <\frac{r}{2} \deg \Omega ^1_{\overline{B}}. \end{aligned}$$

The last inequality follows from the assumption that $r={\mathrm{rank\,}}A_{\overline{C}}^{1,0}>\frac{5g+22}{7}$. This is a contradiction to [12, Corollary 3.6], since $\Lambda =\emptyset$ by Claim 3.8.

In the rest part of the proof, we assume that C is not compact. Hence the family $\bar{f}:\, \overline{S} \rightarrow \overline{B}$ admits singular fibers with non-compact Jacobians, and hence we may assume that the flat factor $F_{\overline{C}}^{1,0}$ is trivial up to a suitable base change, cf. [18, § 4]. Under this assumption, we claim that

Claim 3.9

$$\begin{aligned} \gamma \ge \frac{g-r-1}{2}. \end{aligned}$$

(3.13)

Proof of Claim 3.9

Since $F_{\overline{C}}^{1,0}$ is trivial, $F_{\overline{B}}^{1,0}$ is also trivial. By [6], this is equivalent to saying that the relative irregularity $q_{\bar{f}}=g-r$.

Assume on the contrary that $\gamma < \frac{g-r-1}{2}$. Then $q(\widetilde{S})-q(\widetilde{Y})\ge q_{\bar{f}}-\gamma >\gamma +1$. Thus by [14, Lemma 4.8], the image $J_0(\widetilde{S})\subseteq \mathrm{Alb}_0(\widetilde{S})$ is a curve of genus at least $q_{\bar{f}}-\gamma$, where $J_0:\,\widetilde{S} \rightarrow \mathrm{Alb}_0(\widetilde{S})$ is the relative Albanese map with respect to the double cover $\widetilde{\Phi }$ as defined in [14, § 4.2]. On the other hand, one knows that any fiber of $\bar{f}$ over $\Delta _{nc}$ is of geometric genus equal to $q_{\bar{f}}$ [11, Corollary 1.7]. Therefore one sees that the restricted map $J_0\big |_{F_0}:\,F_0 \rightarrow J_0(\widetilde{S})\subseteq \mathrm{Alb}_0(\widetilde{S})$ is of degree one. This implies that $\widetilde{S}$ is birational to $\overline{B} \times J_0(\widetilde{S})$, which is a contradiction. $\square$

By (3.11) together with [12, Theorem 4.1] and (3.13), one gets

$$\begin{aligned} \deg \bar{f} _*\omega _{\overline{S}/\overline{B}}<\frac{6(g-1)}{~\frac{8(g-1)}{g-\gamma }+\frac{6g+4\gamma -10}{g-1}~} \deg \Omega ^1_{\overline{B}}(\log \Delta _{nc}) <\frac{r}{2} \deg \Omega ^1_{\overline{B}}(\log \Delta _{nc}). \end{aligned}$$

Since $\Lambda =\emptyset$ by Claim 3.8, this is again a contradiction to [12, Corollary 3.6]. $\square$

Remark 3.10

When the unitary part satisfies ${\mathrm{rank\,}}F_{\overline{C}}^{1,0}\le \lceil (g+1)/2\rceil$, we refer to [7] for some results on the restriction of a possible Shimura curve generically in the Torelli locus.

4 Partial Corestriction and Associated Symplectic Representations

In this section we discuss a variant from the construction in [15] which produced symplectic representations from corestriction of central simple algebras.

4.1 Partial Corestriction

For a finite separable extension of fields $F\supset L$ and A a central simple F-algebra, we have the notion of restriction and corestriction:

• the restriction of scalars for A along $L\hookrightarrow F$ is the semi-simple L-algebra $\mathrm {Res}_{F/L}A$, which splits into $A_{{\bar{L}}}^{{\mathrm {Emb}}_L(F)}$ after the base change $L\hookrightarrow {\bar{L}}$, and we may equally write

  $$\begin{aligned} \mathrm {Res}_{F/L}A\otimes _L{\bar{L}}=\bigoplus _{\sigma \in {\mathrm {Emb}}_L(F)}\sigma ^*A \end{aligned}$$

• the corestriction for A along $L\hookrightarrow F$ is a central simple L-algebra D, uniquely characterized by

  $$\begin{aligned} D_{\bar{L}}\simeq \bigotimes _{\sigma \in {\mathrm {Emb}}_L(F)}\sigma ^*A \end{aligned}$$

  up to isomorphism.

Here ${\bar{L}}$ is a fixed separable closure of L and ${\mathrm {Emb}}_L(F)$ is the set of L-embeddings of F into ${\bar{L}}$, with $\sigma ^*A=A\otimes _{F,\sigma }{\bar{L}}$.

For the restriction we have an evident diagonal homomorphism $A\rightarrow \mathrm {Res}_{F/L}A\otimes _LF$ of F-algebras by the adjunction between restriction and tensor product, and for the corestriction we still have a multiplicative map $A\rightarrow \mathrm {Cor}_{F/L}A=D$, which is the multiplicative diagonal map $a\mapsto \otimes _{\sigma \in {\mathrm {Emb}}_L(F)}\sigma ^*(a)$ viewed in $A_{\bar{L}}\rightarrow D_{\bar{L}}$. Both lead to homomorphisms of linear L-groups: the restriction gives $\mathbb {G}_\mathrm {m}^{A/F}\rightarrow \mathbb {G}_\mathrm {m}^{A/L}\otimes _LF$, and the corestriction gives $\mathbb {G}_\mathrm {m}^{A/L}\rightarrow \mathbb {G}_\mathrm {m}^{D/L}$.

We would like to consider the following construction as an interpolation between restriction and corestriction: for F / L a finite separable extension of fields of degree r and $t\in \{1,\ldots ,r\}$, together with A a central simple F-algebra, we define the tth partial corestriction of A along $L\hookrightarrow F$ to be the semi-simple L-algebra D(t) with

$$\begin{aligned} D(t)\otimes _L{\bar{L}}=\bigoplus _{T}\bigotimes _{\sigma \in T}\sigma ^*A, \end{aligned}$$

where the summation $\bigoplus _T$ is taken over all subsets T in ${\mathrm {Emb}}_L(F)$ of cardinality t, and $\sigma ^*A=A\otimes _{F,\sigma }{\bar{L}}$. It is clear that D(t) is unique up to L-isomorphism, with $D(1)\simeq \mathrm {Res}_{F/L}A$ and $D(t)\simeq \mathrm {Cor}_{F/L}A$ as the extremal examples.

For t fixed as above and each $T\subset {\mathrm {Emb}}_L(F)$ of cardinality t, we have a multiplicative map

$$\begin{aligned} A \rightarrow \bigotimes _{\sigma \in T}\sigma ^*A, \ a\mapsto \otimes \sigma ^*(a). \end{aligned}$$

They sum up to a multiplicative map

$$\begin{aligned} (A\otimes _L{\bar{L}})^\times \rightarrow \prod _{T}\left( \bigotimes _{\sigma \in T}\sigma ^*A\right) ^\times \end{aligned}$$

and it sheafifies into a homomorphism of linear L-group $\mathbb {G}_\mathrm {m}^{A/L}\rightarrow \mathbb {G}_\mathrm {m}^{D(t)/L}$.

We may also use a single ${\mathrm {Gal}}({\bar{L}}/L)$-orbit in ${\mathrm {Emb}}_L(F)$ instead of summing over all subsets of given cardinality. For example, let $\Lambda$ be a ${\mathrm {Gal}}({\bar{L}}/L)$-orbit in ${\mathrm {Emb}}_L(F)$, in the sense that $\Lambda =\{g(T_0):g\in {\mathrm {Gal}}({\bar{L}}/L)\}$ for some $T_0\subset {\mathrm {Emb}}_L(F)$ non-empty, and we define $D(\Lambda )$ to be the semi-simple L-algebra characterized as:

$$\begin{aligned} D(\Lambda )\otimes _L{\bar{L}}=\bigoplus _{T\in \Lambda }\bigotimes _{\sigma \in T}\sigma ^*A, \end{aligned}$$

which is unique up to L-isomorphism. The homomorphism of linear L-group $\mathbb {G}_\mathrm {m}^{A/L}\rightarrow \mathbb {G}_\mathrm {m}^{D(\Lambda )/L}$ is constructed in a parallel way.

4.2 Construction of Representations via Hermitian Forms

In this section we focus on the case of symplectic representations defining Shimura curves in $\mathcal {A}_g$ which are associated with partial corestrictions. We fix $\iota :L\hookrightarrow F$ a finite separable extension of totally real number fields. Write $\overline{\mathbb {Q}}$ for the algebraic closure of $\mathbb {Q}$ in $\mathbb {C}$, and we may identify ${\mathrm {Emb}}(F)=\mathrm {Hom}_\mathbb {Q}(F,\overline{\mathbb {Q}})$ the set of embeddings of F into $\overline{\mathbb {Q}}$ with $\mathrm {Hom}_\mathbb {Q}(F,\mathbb {C})$ and $\mathrm {Hom}_{\mathbb {Q}}(F,\mathbb {R})$, together with a transitive action of ${\mathrm {Gal}}(\overline{\mathbb {Q}}/\mathbb {Q})$. Similarly, fix $L\hookrightarrow \mathbb {C}$ with ${\bar{L}}$ the separable closure of L in $\mathbb {C}$, we may identity ${\mathrm {Emb}}_L(F)$ the set of L-embeddings of F into ${\bar{L}}$ with $\mathrm {Hom}_L(F,\mathbb {C})$ and $\mathrm {Hom}_L(F,\mathbb {R})$, and the evident transitive action of ${\mathrm {Gal}}({\bar{L}}/L)$ on ${\mathrm {Emb}}_L(F)$ passes on. We are given an F-form $\mathbf {H}$ of $\mathbf {SL}_{2,F}$, and we write $\{\sigma _1,\ldots ,\sigma _s\}$ for the real embeddings of L, $\{\tau _{i,1},\ldots ,\tau _{i,r}\}$ for the real embeddings of F extending $\sigma _i$, such that $\mathbf {H}(\mathbb {R},\tau _{1,1})\simeq \mathbf {SL}_2(\mathbb {R})$ and $\mathbf {H}(\mathbb {R},\tau _{i,j})=\mathbf {SU}_2(\mathbb {R})$ for $(i,j)\ne (1,1)$. Also write $\mathbf {J}=\mathrm {Res}_{F/L}\mathbf {H}$ with $\mathbf {J}(\mathbb {R},\sigma _i)=\prod _{j=1,\ldots ,r}\mathbf {H}(\mathbb {R},\tau _{i,j})$ for $i=1,\ldots ,s$.

We proceed to the construction of symplectic representations associated with partial corestrictions of quaternion algebras defining Shimura curves.

Case (1) + (2-1):

Following the discussion in Sect. 2.1, Cases (1) and (2-1) are treated together. We are given a CM field of the form $E=F\otimes _LK$ of real part F, with K a CM field of real part L, and $h:V\times V \rightarrow E$ an Hermitian form on $V=E^2$, of signature (1, 1) along $\tau _{1,1}$, definite along the other real embeddings of F, and $\mathbf {H}=\mathbf {SU}_h$. For $T\subset {\mathrm {Emb}}_L(F)$ non-empty, we have an Hermitian form $h_T:V_T\times V_T\rightarrow {\bar{L}}\otimes _LK$ with respect to ${\bar{L}}\hookrightarrow {\bar{L}}\otimes _LK$, where $V_T$ is the ${\bar{L}}$-linear tensor product of $V\otimes _{F,\tau }{\bar{L}}$ over $\tau \in T$, and $\mathbf {J}({\bar{L}})$ preserves $h_T$, with its action on $V_T$ through the projection $\mathbf {J}({\bar{L}})\simeq \prod _{\tau \in {\mathrm {Emb}}_L(F)}\mathbf {H}({\bar{L}},\tau )\rightarrow \prod _{\tau \in T}\mathbf {H}({\bar{L}},\tau )$. Taking orthogonal direct sum over the ${\mathrm {Gal}}({\bar{L}}/L)$-orbit $\Lambda$ of T in ${\mathrm {Emb}}_L(F)$, we obtain an Hermitian space $h_\Lambda :V_\Lambda \times V_\Lambda \rightarrow {\bar{L}}\otimes _LK$ with $V_\Lambda =\bigoplus _{T\in \Lambda }\bigotimes _{\tau \in T}\tau ^*V$ on which $\mathbf {J}({\bar{L}})$ acts by automorphisms, and the ${\mathrm {Gal}}({\bar{L}}/L)$-invariance descends it into an Hermitian space with respect to K / L, which we still denote as $h_\Lambda :V_\Lambda \times V_\Lambda \rightarrow K$. Again $\mathbf {J}$ preserves $h_\Lambda$ and a further scalar restriction from L to $\mathbb {Q}$ gives $\mathrm {Res}_{F/\mathbb {Q}}\mathbf {H}=\mathrm {Res}_{L/\mathbb {Q}}\mathbf {J}\hookrightarrow \mathrm {Res}_{L/\mathbb {Q}}\mathbf {SU}_{h_\Lambda }$. In particular, $\mathbf {J}$ preserves the imaginary part of $h_\Lambda$ which is a symplectic L-form, the $L/\mathbb {Q}$-trace of which is a symplectic $\mathbb {Q}$-form $\psi$ on M the $\mathbb {Q}$-vector space underlying $V_\Lambda$ and is preserved by $\mathrm {Res}_{F/\mathbb {Q}}\mathbf {H}$.

Note that $M=\mathrm {Res}_{K/\mathbb {Q}}V_\Lambda$ is of $\mathbb {Q}$-dimension $2\cdot 2^{t}s\cdot \#\Lambda$, where $s=[L:\mathbb {Q}]$ and t is the common cardinality of $T\in \Lambda$. Let $\mathbb {C}^\times$ act on $V\otimes _{F,\tau _{i,j}}\mathbb {R}$ through the similitude by the norm $\mathbb {C}^\times \rightarrow \mathbb {R}^\times$ if $(i,j)\ne (1,1)$, and act on $V\otimes _{F,\tau _{1,1}}\mathbb {R}$ preserving the Hermitian form up to $\mathbb {C}^\times \rightarrow \mathbb {R}^\times$, then we obtain a homomorphism $\mathbb {S}\rightarrow \mathbf {G}$ where $\mathbf {G}$ is the $\mathbb {Q}$-subgroup of $\mathrm {GSp}_M$ extending $\mathrm {Res}_{F/\mathbb {Q}}\mathbf {H}$ by a central $\mathbb {Q}$-torus $\mathbb {G}_\mathrm {m}$. This gives rise to an inclusion of Shimura datum $(\mathbf {G},X;X^+)\hookrightarrow (\mathrm {GSp}_M,\mathcal {H}_M;\mathcal {H}_M^+)$ and a Shimura curve C in $\mathcal {A}_M$. Note that the symplectic $\mathbb {R}$-representation of $\mathbf {G}^\mathrm {der}(\mathbb {R})$ on $M\otimes _\mathbb {Q}\mathbb {R}$ admits a decomposition $M\otimes _\mathbb {Q}\mathbb {R}\simeq \bigoplus _{i=1,\ldots ,s}V\otimes _{L,\sigma _i}\mathbb {R}$, with $\mathbf {G}^\mathrm {der}(\mathbb {R})$ acting on $V\otimes _{L,\sigma _i}\mathbb {R}$ through $\mathbf {J}(\mathbb {R},\sigma _i)$, which is non-compact for $i=1$ and compact for $i=2,\ldots ,s$. Hence $\bigoplus _{i\ne 1}V_\Lambda \otimes _{L,\sigma _i}\mathbb {R}$ only contribute to the unitary part in the canonical Higgs bundle on C.

The remaining $\mathbb {R}$-subrepresentation $V_\Lambda \otimes _{L,\sigma _1}\mathbb {R}=\bigoplus _{T\in \Lambda }V_T\otimes _{L,\sigma _1}\mathbb {R}$ is isomorphic to $\bigoplus _{T\in \Lambda }(\mathbb {C}^2)^{\otimes T}$, with $\mathbf {G}^\mathrm {der}(\mathbb {R})$ acting on $(\mathbb {C}^2)^{\otimes T}$ via the projection through the product of those $\mathbf {H}(\mathbb {R},\tau _{1,j})$ corresponding to $\tau \in T$. Upon the natural identification of $\{\tau _{1,1},\ldots ,\tau _{1,r}\}$ with ${\mathrm {Emb}}_L(F)$, we see that the summand $(\mathbb {C}^2)^{\otimes T}$ contributes to the unitary part in the canonical Higgs bundle if and only if $\tau _{1,1}$ does not appear in T; when it appears $(\mathbb {C}^2)^{\otimes T}$ is an Hermitian space of signature $(2^{t-1},2^{t-1})$ ($t=\#T$).

It is in general difficult to compute the unitary rank for an arbitrary ${\mathrm {Gal}}({\bar{L}}/L)$-orbit $\Lambda$ as above. We may still treat the simpler case using the representation V(t): t is a fixed integer in [1, r] ($r=[F:L]$), and V(t) is an Hermitian space for K / L with $V(t)\otimes _L{\bar{L}}=\bigoplus _{\#T=t}\bigotimes _{\sigma \in T}V\otimes _{L,\sigma }{\bar{L}}$ using orthogonal direct sum of Hermitian spaces constructed as above. Again write M for the $\mathbb {Q}$-vector space underlying V(t), which is of $\mathbb {Q}$-dimension $2\cdot 2^{t}\left( {\begin{array}{c}r\\ t\end{array}}\right) [L:\mathbb {Q}]$, we see the contribution to the unitary part of the Higgs bundle associated with M are from:

• $V(t)\otimes _{L,\sigma _i}\mathbb {R}$ with $i=2,\ldots ,s$;

• those $(\mathbb {C}^2)^{\otimes T}$ with $T\subset {\mathrm {Emb}}_L(F)$ of cardinality t in which the embedding corresponding to $\tau _{1,1}$ does not appear; each of these tensor product is of $\mathbb {C}$-dimension $2^t$ on which $\mathbf {G}^\mathrm {der}(\mathbb {R})$ acts through a compact group, and there are $\left( {\begin{array}{c}r-1\\ t\end{array}}\right)$ such summands.

Those $(\mathbb {C}^2)^{\otimes T}$ with $\tau _{1,1}$ appearing in $T\subset {\mathrm {Emb}}_L(F)$ of cardinality t do not contribute to the unitary part: $\mathbf {G}^\mathrm {der}(\mathbb {R})$ preserves an Hermitian form of signature $(2^{t-1},2^{t-1})$ on such an summand, and there are $\left( {\begin{array}{c}r-1\\ t-1\end{array}}\right)$ such summands.

To summarize, the unitary part in the Higgs bundle associated with $M=\mathrm {Res}_{K/\mathbb {Q}}V(t)$ in this case is of rank $\frac{{\mathrm{rank\,}}M}{s}(s-1+\frac{\left( {\begin{array}{c}r-1\\ t\end{array}}\right) }{\left( {\begin{array}{c}r\\ t\end{array}}\right) })=\frac{{\mathrm{rank\,}}M}{s}(s-\frac{t}{r})={\mathrm{rank\,}}M(1-\frac{t}{d})$ with $d=rs=[F:\mathbb {Q}]$.

Case (2-2):

In this case we have a CM field E of totally real part F, a quaternion division E-algebra A carrying an involution of second kind which extends the F-conjugation on E, and an Hermitian pairing $H:A\times A\rightarrow A$. A further composition with the reduced trace gives an Hermitian form $h:A\times A\rightarrow E$, which is preserved by $\mathbf {H}$ the outer form of $\mathbf {SL}_{2,F}$ as we have seen in Sect. 2. Along the real embedding $\tau _{1,1}$ we have $\tau _{1,1}^*A=A\otimes _{F,\tau _{1,1}}\mathbb {R}\simeq \mathbb {C}^4$, on which $\mathbf {H}(\mathbb {R},\tau _{1,1})\simeq \mathbf {SU}(1,1)$ has a faithful action preserving $\tau _{1,1}^*h=h\otimes _{F,\tau _{1,1}}\mathbb {R}$: this forces $\tau _{1,1}^*A\simeq (\mathbb {C}^2)^{\oplus 2}$ as a direct sum of two copies of the standard representation of $\mathbf {SU}(1,1)$ on $\mathbb {C}^2$, and $\tau _{1,1}^*h$ has to be an Hermitian form of signature (2, 2). The other real embeddings only lead to definite Hermitian spaces preserved by compact Lie groups $\mathbf {H}(\mathbb {R},\tau _{i,j})$ ($(i,j)\ne (1,1)$).

Given a finite extension of fields $K\hookrightarrow E$ and $\Lambda$ a ${\mathrm {Gal}}({\bar{K}}/K)$-orbit of some non-empty subset $T_0$ in ${\mathrm {Emb}}_K(E)$ we have the partial corestriction $D(\Lambda )$. In order to have natural Hermitian spaces on suitable modules over $D(\Lambda )$, we assume for simplicity that K is also a CM field, and the extension $K\hookrightarrow E$ is extended from an extension of totally real fields $L\hookrightarrow F$ with L the real part of K and $E\simeq F\otimes _LK$, and we identify ${\mathrm {Emb}}_K(E)$ with ${\mathrm {Emb}}_L(F)$. Thus the ${\mathrm {Gal}}({\bar{K}}/K)$-orbit $\Lambda$ above can be identified as a ${\mathrm {Gal}}({\bar{L}}/L)$-orbit in ${\mathrm {Emb}}_L(F)$, which is again denoted as $\Lambda$.

The semi-simple K-algebra $D(\Lambda )$ is characterized by the isomorphism

$$\begin{aligned} D(\Lambda )\otimes _K{\bar{K}}\simeq \bigoplus _{T\in \Lambda }\bigotimes _{\tau \in T}\tau ^*A. \end{aligned}$$

Write V for the E-module underlying A, and we also have the following $D(\Lambda )$-module $V(\Lambda )$ again characterized as $V(\Lambda )\otimes _K{\bar{K}}=\bigoplus _{T\in \Lambda }\bigotimes _{\tau \in T}\tau ^*V$ which is just the K-vector space underlying $D(\Lambda )$. The Hermitian structure $h:V\times V\rightarrow E$ passes to an Hermitian form $h_\Lambda$ on $V(\Lambda )$ similar to Cases (1)+(2-1) using orthogonal direct sums of Hermitian structures on the summands, and it is preserved by $\mathbf {J}=\mathrm {Res}_{F/L}\mathbf {H}$. Taking a further scalar restriction we obtain an action of $\mathrm {Res}_{F/\mathbb {Q}}\mathbf {H}$ on $M=\mathrm {Res}_{K/\mathbb {Q}}V(\Lambda )$ preserving the symplectic $\mathbb {Q}$-structure induced from the imaginary part of $h_\Lambda$. Arguments parallel to the previously established case produce a Shimura datum $(\mathbf {G},X;X^+)\hookrightarrow (\mathrm {GSp}_M,\mathcal {H}_M;\mathcal {H}_M^+)$ defining a Shimura curve C.

The computation of unitary rank in the canonical Higgs bundle associated with $\mathbf {G}\hookrightarrow \mathrm {GSp}_M$ on C is similar:

• the embeddings $\sigma _2,\ldots ,\sigma _d$ correspond to summands $V(\Lambda )\otimes _{E,\sigma _i}\mathbb {C}$ on which $\mathbf {G}^\mathrm {der}(\mathbb {R})$ acts through a compact quotient, which only contribute to the unitary Higgs bundle;

• inside $V(\Lambda )\otimes _{E,\sigma _1}\mathbb {C}$, the summands $\bigotimes _{\tau _{1,j}\in T}\tau _{1,j}^*V$ contributes to the unitary part if and only if $\tau _{1,1}\in T$; here we identify $\{\tau _{1,j}:j=1,\ldots ,[E:K]\}$ with ${\mathrm {Emb}}_K(E)$, similar to Case (1)+(2-1).

The case D(t) remains computable: we fix t an integer in [1, r] ($r=[E:K]$), and we have the semi-simple K-algebra $D(t)=[\bigoplus _{T}\bigotimes _{\tau \in T}\tau ^*A]^{{\mathrm {Gal}}({\bar{K}}/K)}$ with T running through subsets of ${\mathrm {Emb}}_L(K)$ of cardinality t, and the K-vector space V(t) underlying D(t) carries an Hermitian form h(t), obtained as orthogonal direct sums $\bigoplus h_\Lambda$ taken over ${\mathrm {Gal}}({\bar{K}}/K)$-orbits considered as above. Write M for the $\mathbb {Q}$-vector space underlying V(t), it carries a symplectic $\mathbb {Q}$-form preserved by $\mathrm {Res}_{F/\mathbb {Q}}\mathbf {H}$, induced from the imaginary part of h(t), and we obtain a Shimura subdatum $(\mathbf {G},X;X^+)\hookrightarrow (\mathrm {GSp}_{M},\mathcal {H}_M;\mathcal {H}_M^+)$ defining a Shimura curve C. In this case, the unitary part of the canonical Higgs bundle associated with $\mathbf {G}\rightarrow \mathrm {GSp}_M$ is computed similarly: its rank equals ${\mathrm{rank\,}}(M)(1-\frac{t}{d})$ with $d=rs=[F:\mathbb {Q}]$. The $\mathbb {Q}$-dimension of M is clearly $2\cdot 4^{t}s\left( {\begin{array}{c}r\\ t\end{array}}\right)$.

4.3 End of the Proof

So far the rank 2g of M is $2\cdot 2^{t}\left( {\begin{array}{c}r\\ t\end{array}}\right) s$ in Case (1)+(2-1) and $2\cdot 4^{t}\left( {\begin{array}{c}r\\ t\end{array}}\right) s$ in Case (2-2), and the ample part $A_{\overline{C}}^{1,0}$ is of rank $g\frac{t}{d}\le \frac{g}{[L:\mathbb {Q}]}$. Theorem 1.2 affirms the generic exclusion of such a Shimura curve from $\mathcal {T}_g^\circ$ as soon as $g\frac{t}{d}>\frac{5g+22}{7}$. Note that we are only interested in the Coleman–Oort conjecture for $g\ge 7$, and Theorem 1.2 would not be applicable for $L\ne \mathbb {Q}$. Hence we assume $L=\mathbb {Q}$ and Corollary 1.3 is clear for $\frac{t}{d}>\frac{5}{7}+\frac{22}{7g}$.

Remark 4.1

The symplectic representation V(t) is in general reducible: given $T\subset {\mathrm {Emb}}_L(F)$ we have at most $r=[F:L]$ Galois conjugates of T inside ${\mathrm {Emb}}_L(F)$, while V(t) is a direct sum over $\left( {\begin{array}{c}r\\ t\end{array}}\right)$ such subsets. We have restricted to this case only for the simplicity of computation; the case of a general ${\mathrm {Gal}}({\bar{L}}/L)$-orbit of a given subset in ${\mathrm {Emb}}_L(F)$ remains unclear for the moment.