1 Introduction

In many applications of elliptic modular functions to number theory, the Dedekind eta function plays a central role. It is defined in the upper half-plane \(\mathbf{H}=\{ z \in \mathbf{C}: \text{ Im }(z)>0 \}\) by \(\eta (z):=q^{\frac{1}{24}} \prod _{n=1}^{\infty } (1-q^n)\) with \(q=e^{2 \pi i z}\). The Dedekind eta function is closely related to the partition function. A partition of a positive integer n is any non-increasing sequence of positive integers whose sum is n. Let p(n) denote the number of partitions of n. The partition function p(n) has the well-known generating function \(\sum _{n=0}^{\infty } p(n) q^n=\prod _{n=1}^{\infty } (1-q^n)^{-1} =q^{\frac{1}{24}}/\eta (z)\).

In his ground-breaking works [20–22], Ramanujan found the following famous Ramanujan partition congruences: \(p(5n+4) \equiv 0\) (mod 5) and \(p(7n+5) \equiv 0\) (mod 7). His proofs are quite ingenious:

$$\begin{aligned} q^{\frac{19}{24}} \sum _{n=0}^{\infty } p(5n+4) q^n =5 \frac{\eta (5z)^5}{\eta (z)^6}, \quad q^{\frac{17}{24}} \sum _{n=0}^{\infty } p(7n+5) q^n =7 \frac{\eta (7z)^3}{\eta (z)^4}+49 \frac{\eta (7z)^7}{\eta (z)^8}.\nonumber \\ \end{aligned}$$
(1.1)

In [29], Zuckerman found an identity in the spirit of (1.1) (see also [4, 26]):

$$\begin{aligned} q^{\frac{11}{24}} \sum _{n=0}^{\infty } p(13n+6) q^n= & {} 11 \frac{\eta (13z)}{\eta (z)^2}+36 \cdot 13 \frac{\eta (13z)^3}{\eta (z)^4}+38 \cdot 13^2 \frac{\eta (13z)^5}{\eta (z)^6} +\nonumber \\&+\,20 \cdot 13^3 \frac{\eta (13z)^7}{\eta (z)^8} +6 \cdot 13^4 \frac{\eta (13z)^9}{\eta (z)^{10}}+13^5 \frac{\eta (13z)^{11}}{\eta (z)^{12}}\nonumber \\&+\,13^5 \frac{\eta (13z)^{13}}{\eta (z)^{14}}. \end{aligned}$$
(1.2)

Rademacher (see [19]) pointed out that (1.1) can be rewritten as

$$\begin{aligned} \sum \limits _{\lambda =0}^{4} \eta (5z) \eta \left( \frac{z+24 \lambda }{5}\right) ^{-1}= & {} 5^2 \left( \frac{\eta (5z)}{\eta (z)}\right) ^6,\nonumber \\ \sum \limits _{\lambda =0}^{6} \eta (7z) \eta \left( \frac{z+24 \lambda }{7}\right) ^{-1}= & {} 7^2 \left( \frac{\eta (7z)}{\eta (z)}\right) ^4+7^3 \left( \frac{\eta (7z)}{\eta (z)}\right) ^8. \end{aligned}$$
(1.3)

In fact, (1.2) can also be rewritten as

$$\begin{aligned} \sum _{\lambda =0}^{12} \eta (13z) \eta \left( \frac{z+24 \lambda }{13}\right) ^{-1}= & {} 11 \cdot 13 \left( \frac{\eta (13z)}{\eta (z)}\right) ^2+\,36 \cdot 13^2 \left( \frac{\eta (13z)}{\eta (z)}\right) ^4 \nonumber \\&+\,38 \cdot 13^3 \left( \frac{\eta (13z)}{\eta (z)}\right) ^6 +20 \cdot 13^4 \left( \frac{\eta (13z)}{\eta (z)}\right) ^8 +\nonumber \\&\,+\,6 \cdot 13^5 \left( \frac{\eta (13z)}{\eta (z)}\right) ^{10} +13^6 \left( \frac{\eta (13z)}{\eta (z)}\right) ^{12} \nonumber \\&+\,13^6 \left( \frac{\eta (13z)}{\eta (z)}\right) ^{14}. \end{aligned}$$
(1.4)

Let \(X=X_0(p)\) be the compactification of \(\mathbf{H}/\Gamma _0(p)\), where \(\Gamma _0(p)\) is the level-p Hecke modular group and p is prime. The complex function field of X consists of the modular functions f(z) for \(\Gamma _0(p)\) which are meromorphic on the extended upper half-plane. A function f lies in the rational function field \(\mathbf{Q}(X)\) if and only if the Fourier coefficients in its expansion at \(\infty \): \(f(z)=\sum a_n q^n\) are all rational numbers. The field \(\mathbf{Q}(X)\) is known to be generated over \(\mathbf{Q}\) by the classical j-functions

$$\begin{aligned} \left\{ \begin{array}{ll} j &{}=j(z)=q^{-1}+744+196884 q+\cdots ,\\ j_p &{}=j \left( \frac{-1}{pz}\right) =j(pz)=q^{-p}+744+\cdots . \end{array}\right. \end{aligned}$$

A further element in the function field \(\mathbf{Q}(X)=\mathbf{Q}(j, j_p)\) is the modular unit \(u=\Delta (z)/\Delta (pz)\) with divisor \((p-1) \{ (0)-(\infty ) \}\), where \(\Delta (z)\) is the discriminant. If \(m=\text{ gcd }(p-1, 12)\), then an mth root of u lies in \(\mathbf{Q}(X)\). This function has the Fourier expansion

$$\begin{aligned} t=\root m \of {u}=q^{(1-p)/m} \prod _{n \ge 1} \left( \frac{1-q^n}{1-q^{np}}\right) ^{24/m}=\left( \frac{\eta (z)}{\eta (pz)}\right) ^{24/m}. \end{aligned}$$

When \(p-1\) divides 12, so \(m=p-1\), the function t is a Hauptmodul for the curve X which has genus zero (see [11]). It is well known that the genus of the modular curve X for prime p is zero if and only if \(p=2, 3, 5, 7, 13\). In his paper [13], Klein studied the modular equations of orders 2, 3, 5, 7, 13 with degrees 3, 4, 6, 8, 14, respectively. They are uniformized, i.e., parametrized, by the so-called Hauptmoduln (principal moduli). A Hauptmodul is a function \(J_{\Gamma }\) that is a modular function for some subgroups of \(\Gamma (1)=PSL(2, \mathbf{Z})\), with any other modular function expressible as a rational function of it. In this case, \(\Gamma =\Gamma _0(p)\). Hence, (1.3) and (1.4) are intimately related to the Hauptmoduln for modular curves \(X_0(5)\), \(X_0(7)\), and \(X_0(13)\).

The modular curves \(X_0(5)\) and \(X_0(7)\) were studied by Klein in his pioneering work (see [12–14, 16, 17]) and by Ramanujan (see [24–26]). It is well known that the celebrated Rogers–Ramanujan identities

$$\begin{aligned} G(q):= & {} \sum \limits _{n=0}^{\infty } \frac{q^{n^2}}{(1-q) \cdots (1-q^n)} =\prod \limits _{n=0}^{\infty } \frac{1}{(1-q^{5n+1})(1-q^{5n+4})},\\ H(q):= & {} \sum \limits _{n=0}^{\infty } \frac{q^{n^2+n}}{(1-q) \cdots (1-q^n)} =\prod \limits _{n=0}^{\infty } \frac{1}{(1-q^{5n+2})(1-q^{5n+3})} \end{aligned}$$

are intimately associated with the Rogers–Ramanujan continued fraction

$$\begin{aligned} R(q):=\frac{q^{\frac{1}{5}}}{1+\frac{q}{1+\frac{q^2}{1+\frac{q^3}{1+\cdots }}}}, \end{aligned}$$

namely, they satisfy that \(R(q)=q^{\frac{1}{5}} \frac{H(q)}{G(q)}\). In [23], Ramanujan found an algebraic relation between G(q) and H(q): \(G^{11}(q) H(q)-q^2 G(q) H^{11}(q)=1+11 q G^6(q) H^6(q)\), which is equivalent to one of the most important formulas for R(q) (see also [1]):

$$\begin{aligned} \frac{1}{R^5(q)}-11-R^5(q)=\left( \frac{\eta (z)}{\eta (5z)}\right) ^6. \end{aligned}$$
(1.5)

In his celebrated work on elliptic modular functions (see [16, p. 640], [17, p. 73], and [7]), Klein showed that \(R(q)=\frac{a(z)}{b(z)}\), where \(a(z)=e^{-\frac{3 \pi i}{10}} \theta \left[ \begin{array}{l} \frac{3}{5}\\ 1 \end{array}\right] (0, 5z)\) and \(b(z)=e^{-\frac{\pi i}{10}} \theta \left[ \begin{array}{c} \frac{1}{5}\\ 1 \end{array}\right] (0, 5z)\) are theta constants of order 5. In fact, \(a(z)=q^{\frac{1}{60}} \eta (5z)/G(q)\) and \(b(z)=q^{-\frac{11}{60}} \eta (5z)/H(q)\). Hence, (1.5) is equivalent to the following formula:

$$\begin{aligned} \left( \frac{\eta (z)}{\eta (5z)}\right) ^6=-\frac{f(a(z), b(z))}{a(z)^6 b(z)^6}, \end{aligned}$$
(1.6)

where \(f(z_1, z_2)=z_1 z_2 (z_1^{10}+11 z_1^5 z_2^5-z_2^{10})\) is an invariant of degree 12 associated to the icosahedron, i.e., it is invariant under the action of the simple group \(PSL(2, 5)=\Gamma (1)/\Gamma (5)\), where \(\Gamma (5)\) is the principal modular group of level 5. Also \(z_1^2 z_2^2\) is invariant under the action of the image of a Borel subgroup of PSL(2, 5), i.e., a maximal subgroup of order 10 of PSL(2, 5), which is simply \(\Gamma _0(5)/\Gamma (5)\) (see [12] for more details). We call (1.6) the invariant decomposition formula for the icosahedron.

In his work on elliptic modular functions (see [16, p. 746]), Klein also obtained the invariant decomposition formula for the simple group \(PSL(2, 7)=\Gamma (1)/\Gamma (7)\) of order 168:

$$\begin{aligned} \left( \frac{\eta (z)}{\eta (7z)}\right) ^4=\frac{\Phi _6(a(z), b(z), c(z))}{a(z)^2 b(z)^2 c(z)^2}, \end{aligned}$$
(1.7)

where \(a(z)=-e^{-\frac{5 \pi i}{14}} \theta \left[ \begin{array}{l} \frac{5}{7}\\ 1 \end{array}\right] (0, 7z)\), \(b(z)=e^{-\frac{3 \pi i}{14}} \theta \left[ \begin{array}{l} \frac{3}{7}\\ 1 \end{array}\right] (0, 7z)\), and \(c(z)=e^{-\frac{\pi i}{14}} \theta \left[ \begin{array}{l} \frac{1}{7}\\ 1 \end{array}\right] (0, 7z)\) are theta constants of order 7. Here \(\Phi _6(x, y, z)=xy^5+yz^5+zx^5-5 x^2 y^2 z^2\) is an invariant of degree 6 associated to PSL(2, 7), and xyz is invariant under the action of the image of a Borel subgroup of PSL(2, 7), i.e., a maximal subgroup of order 21 of PSL(2, 7), which is simply \(\Gamma _0(7)/\Gamma (7)\) (see [14] for more details). Independently, Ramanujan also gave the same formula (see [25, p. 300], [3, p. 174], [8, 18] for more details).

For the modular curve \(X_0(13)\), in his monograph (see [17, p. 73]), Klein showed that \(\left( \eta (z)/\eta (13 z)\right) ^2\) is a Hauptmodul for \(\Gamma _0(13)\). However, neither Klein nor Ramanujan could obtain the invariant decomposition formula for \(PSL(2, 13)=\Gamma (1)/\Gamma (13)\) in the spirit of (1.6) and (1.7) (see the end of Sect. 4 for more details). The following facts about the modular subgroups of level 13 should be noted. One has that \(\Gamma (13)<\Gamma _0(13)<\Gamma (1)\), the respective subgroup indices being \(78=6 \cdot 13\) and \(14=13+1\). The quotient \(\Gamma (1)/\Gamma (13)\) is of order \(1092=78 \cdot 14\), and \(\Gamma _0(13)/\Gamma (13)\) is a subgroup of order 78, which is isomorphic to a semidirect product of \(\mathbf{Z}_{13}\) by \(\mathbf{Z}_6\). The respective quotients of \(\mathbf{H}\) by these three groups (compactified) are the modular curves X(13), \(X_0(13)\), X(1), and the coverings \(X(13) \rightarrow X_0(13) \rightarrow X(1)\) are, respectively, 78-sheeted and 14-sheeted. The curve X(13) is of genus 50, despite \(X_0(13)\) like X(1) being of genus zero.

In the present paper, we establish the invariant theory for PSL(2, 13). Combining with theta constants of order 13, we obtain an invariant decomposition formula for PSL(2, 13) in the spirit of (1.6) and (1.7). Let

$$\begin{aligned} \left\{ \begin{array}{ll} a_1(z) &{}:=e^{-\frac{11 \pi i}{26}} \theta \left[ \begin{array}{l} \frac{11}{13}\\ 1 \end{array}\right] (0, 13z) =q^{\frac{121}{104}} \sum \limits _{n \in \mathbf{Z}} (-1)^n q^{\frac{1}{2}(13n^2+11n)},\\ a_2(z) &{}:=e^{-\frac{7 \pi i}{26}} \theta \left[ \begin{array}{l} \frac{7}{13}\\ 1 \end{array}\right] (0, 13z) =q^{\frac{49}{104}} \sum \limits _{n \in \mathbf{Z}} (-1)^n q^{\frac{1}{2}(13n^2+7n)},\\ a_3(z) &{}:=e^{-\frac{5 \pi i}{26}} \theta \left[ \begin{array}{l} \frac{5}{13}\\ 1 \end{array}\right] (0, 13z) =q^{\frac{25}{104}} \sum \limits _{n \in \mathbf{Z}} (-1)^n q^{\frac{1}{2}(13n^2+5n)},\\ a_4(z) &{}:=-e^{-\frac{3 \pi i}{26}} \theta \left[ \begin{array}{l} \frac{3}{13}\\ 1 \end{array}\right] (0, 13z) =-q^{\frac{9}{104}} \sum \limits _{n \in \mathbf{Z}} (-1)^n q^{\frac{1}{2}(13n^2+3n)},\\ a_5(z) &{}:=e^{-\frac{9 \pi i}{26}} \theta \left[ \begin{array}{l} \frac{9}{13}\\ 1 \end{array}\right] (0, 13z) =q^{\frac{81}{104}} \sum \limits _{n \in \mathbf{Z}} (-1)^n q^{\frac{1}{2}(13n^2+9n)},\\ a_6(z) &{}:=e^{-\frac{\pi i}{26}} \theta \left[ \begin{array}{l} \frac{1}{13}\\ 1 \end{array}\right] (0, 13z) =q^{\frac{1}{104}} \sum \limits _{n \in \mathbf{Z}} (-1)^n q^{\frac{1}{2}(13n^2+n)} \end{array}\right. \end{aligned}$$
(1.8)

be theta constants of order 13. Set

$$\begin{aligned} \Phi _{12}(z_1, z_2, z_3, z_4, z_5, z_6) :=-\frac{1}{26}(7 \cdot 13^2 \mathbf{G}_0^2+\mathbf{G}_1 \mathbf{G}_{12} +\mathbf{G}_2 \mathbf{G}_{11}+\cdots +\mathbf{G}_6 \mathbf{G}_7), \end{aligned}$$
(1.9)

where

$$\begin{aligned} \left\{ \begin{array}{ll} \mathbf{G}_0 =&{}\mathbf{D}_0^2+\mathbf{D}_{\infty }^2,\\ \mathbf{G}_1 =&{}-\mathbf{D}_7^2+2 \mathbf{D}_0 \mathbf{D}_1+10 \mathbf{D}_{\infty } \mathbf{D}_1 +2 \mathbf{D}_2 \mathbf{D}_{12}+\\ &{}-2 \mathbf{D}_3 \mathbf{D}_{11}-4 \mathbf{D}_4 \mathbf{D}_{10}-2 \mathbf{D}_9 \mathbf{D}_5,\\ \mathbf{G}_2 =&{}-2 \mathbf{D}_1^2-4 \mathbf{D}_0 \mathbf{D}_2+6 \mathbf{D}_{\infty } \mathbf{D}_2 -2 \mathbf{D}_4 \mathbf{D}_{11}+\\ &{}+2 \mathbf{D}_5 \mathbf{D}_{10}-2 \mathbf{D}_6 \mathbf{D}_9-2 \mathbf{D}_7 \mathbf{D}_8,\\ \mathbf{G}_3 =&{}-\mathbf{D}_8^2+2 \mathbf{D}_0 \mathbf{D}_3+10 \mathbf{D}_{\infty } \mathbf{D}_3 +2 \mathbf{D}_6 \mathbf{D}_{10}+\\ &{}-2 \mathbf{D}_9 \mathbf{D}_7-4 \mathbf{D}_{12} \mathbf{D}_4-2 \mathbf{D}_1 \mathbf{D}_2,\\ \mathbf{G}_4 =&{}-\mathbf{D}_2^2+10 \mathbf{D}_0 \mathbf{D}_4-2 \mathbf{D}_{\infty } \mathbf{D}_4 +2 \mathbf{D}_5 \mathbf{D}_{12}+\\ &{}-2 \mathbf{D}_9 \mathbf{D}_8-4 \mathbf{D}_1 \mathbf{D}_3-2 \mathbf{D}_{10} \mathbf{D}_7,\\ \mathbf{G}_5 =&{}-2 \mathbf{D}_9^2-4 \mathbf{D}_0 \mathbf{D}_5+6 \mathbf{D}_{\infty } \mathbf{D}_5 -2 \mathbf{D}_{10} \mathbf{D}_8+\\ &{}+2 \mathbf{D}_6 \mathbf{D}_{12}-2 \mathbf{D}_2 \mathbf{D}_3-2 \mathbf{D}_{11} \mathbf{D}_7,\\ \mathbf{G}_6 =&{}-2 \mathbf{D}_3^2-4 \mathbf{D}_0 \mathbf{D}_6+6 \mathbf{D}_{\infty } \mathbf{D}_6 -2 \mathbf{D}_{12} \mathbf{D}_7+\\ &{}+2 \mathbf{D}_2 \mathbf{D}_4-2 \mathbf{D}_5 \mathbf{D}_1-2 \mathbf{D}_8 \mathbf{D}_{11},\\ \mathbf{G}_7 =&{}-2 \mathbf{D}_{10}^2+6 \mathbf{D}_0 \mathbf{D}_7+4 \mathbf{D}_{\infty } \mathbf{D}_7 -2 \mathbf{D}_1 \mathbf{D}_6+\\ &{}-2 \mathbf{D}_2 \mathbf{D}_5-2 \mathbf{D}_8 \mathbf{D}_{12}-2 \mathbf{D}_9 \mathbf{D}_{11},\\ \mathbf{G}_8 =&{}-2 \mathbf{D}_4^2+6 \mathbf{D}_0 \mathbf{D}_8+4 \mathbf{D}_{\infty } \mathbf{D}_8 -2 \mathbf{D}_3 \mathbf{D}_5+\\ &{}-2 \mathbf{D}_6 \mathbf{D}_2-2 \mathbf{D}_{11} \mathbf{D}_{10}-2 \mathbf{D}_1 \mathbf{D}_7,\\ \mathbf{G}_9 =&{}-\mathbf{D}_{11}^2+2 \mathbf{D}_0 \mathbf{D}_9+10 \mathbf{D}_{\infty } \mathbf{D}_9 +2 \mathbf{D}_5 \mathbf{D}_4+\\ &{}-2 \mathbf{D}_1 \mathbf{D}_8-4 \mathbf{D}_{10} \mathbf{D}_{12}-2 \mathbf{D}_3 \mathbf{D}_6, \end{array} \right. \end{aligned}$$
$$\begin{aligned} \left\{ \begin{array}{ll} \mathbf{G}_{10} =&{}-\mathbf{D}_5^2+10 \mathbf{D}_0 \mathbf{D}_{10}-2 \mathbf{D}_{\infty } \mathbf{D}_{10} +2 \mathbf{D}_6 \mathbf{D}_4+\\ &{}-2 \mathbf{D}_3 \mathbf{D}_7-4 \mathbf{D}_9 \mathbf{D}_1-2 \mathbf{D}_{12} \mathbf{D}_{11},\\ \mathbf{G}_{11} =&{}-2 \mathbf{D}_{12}^2+6 \mathbf{D}_0 \mathbf{D}_{11}+4 \mathbf{D}_{\infty } \mathbf{D}_{11} -2 \mathbf{D}_9 \mathbf{D}_2+\\ &{}-2 \mathbf{D}_5 \mathbf{D}_6-2 \mathbf{D}_7 \mathbf{D}_4-2 \mathbf{D}_3 \mathbf{D}_8,\\ \mathbf{G}_{12} =&{}-\mathbf{D}_6^2+10 \mathbf{D}_0 \mathbf{D}_{12}-2 \mathbf{D}_{\infty } \mathbf{D}_{12} +2 \mathbf{D}_2 \mathbf{D}_{10}+\\ &{}-2 \mathbf{D}_1 \mathbf{D}_{11}-4 \mathbf{D}_3 \mathbf{D}_9-2 \mathbf{D}_4 \mathbf{D}_8 \end{array} \right. \end{aligned}$$
(1.10)

are the senary sextic forms (sextic forms in six variables). Here,

$$\begin{aligned} \left\{ \begin{array}{ll} \mathbf{D}_0 &{}=z_1 z_2 z_3,\\ \mathbf{D}_1 &{}=2 z_2 z_3^2+z_2^2 z_6-z_4^2 z_5+z_1 z_5 z_6,\\ \mathbf{D}_2 &{}=-z_6^3+z_2^2 z_4-2 z_2 z_5^2+z_1 z_4 z_5+3 z_3 z_5 z_6,\\ \mathbf{D}_3 &{}=2 z_1 z_2^2+z_1^2 z_5-z_4 z_6^2+z_3 z_4 z_5,\\ \mathbf{D}_4 &{}=-z_2^2 z_3+z_1 z_6^2-2 z_4^2 z_6-z_1 z_3 z_5,\\ \mathbf{D}_5 &{}=-z_4^3+z_3^2 z_5-2 z_3 z_6^2+z_2 z_5 z_6+3 z_1 z_4 z_6,\\ \mathbf{D}_6 &{}=-z_5^3+z_1^2 z_6-2 z_1 z_4^2+z_3 z_4 z_6+3 z_2 z_4 z_5,\\ \mathbf{D}_7 &{}=-z_2^3+z_3 z_4^2-z_1 z_3 z_6-3 z_1 z_2 z_5+2 z_1^2 z_4,\\ \mathbf{D}_8 &{}=-z_1^3+z_2 z_6^2-z_2 z_3 z_5-3 z_1 z_3 z_4+2 z_3^2 z_6,\\ \mathbf{D}_9 &{}=2 z_1^2 z_3+z_3^2 z_4-z_5^2 z_6+z_2 z_4 z_6,\\ \mathbf{D}_{10} &{}=-z_1 z_3^2+z_2 z_4^2-2 z_4 z_5^2-z_1 z_2 z_6,\\ \mathbf{D}_{11} &{}=-z_3^3+z_1 z_5^2-z_1 z_2 z_4-3 z_2 z_3 z_6+2 z_2^2 z_5,\\ \mathbf{D}_{12} &{}=-z_1^2 z_2+z_3 z_5^2-2 z_5 z_6^2-z_2 z_3 z_4,\\ \mathbf{D}_{\infty }&{}=z_4 z_5 z_6 \end{array}\right. \end{aligned}$$
(1.11)

are the senary cubic forms (cubic forms in six variables). Our main result is as follows:

Theorem 1.1

(Main Theorem 1) The invariant decomposition formula for the simple group PSL(2, 13) of order 1092 is given as follows:

$$\begin{aligned} \left[ \frac{\eta ^2(z)}{\eta ^2(13z)}\right] ^5 =\frac{\Phi _{12}(a_1(z), a_2(z), a_3(z), a_4(z), a_5(z), a_6(z))}{(a_1(z) a_2(z) a_3(z) a_4(z) a_5(z) a_6(z))^2}, \end{aligned}$$
(1.12)

where \(\Phi _{12}(z_1, z_2, z_3, z_4, z_5, z_6)\) is an invariant of degree 12 associated to PSL(2, 13), and \((z_1 z_2 z_3 z_4 z_5 z_6)^2\) is invariant under the action of the image of a Borel subgroup of PSL(2, 13), i.e., a maximal subgroup of order 78 of PSL(2, 13), which can be viewed as \(\Gamma _0(13)/\Gamma (13)\).

It should be pointed out that (1.12) is a formula not for the Hauptmodul of \(\Gamma _0(13)\), but for its fifth power. The appearance of the fifth power is due to the denominator \((a_1(z) a_2(z) a_3(z) a_4(z) a_5(z) a_6(z))^2\) having a factor \(\eta (13 z)^{10}\). In [13], Klein obtained the modular equation of degree 14 for \(\Gamma _0(13)\) (see [5, p. 267] for the modular relation for \(\Gamma _0(13)\)). Combining with Theorem 1.1, we give a new expression of the classical j-function in terms of theta constants of order 13.

Corollary 1.2

The following formulas hold:

$$\begin{aligned} j(z)=\frac{(\tau ^2+5\tau +13)(\tau ^4+247 \tau ^3+3380 \tau ^2+15379 \tau +28561)^3}{\tau ^{13}} \end{aligned}$$
(1.13)

and

$$\begin{aligned} j(13z)=\frac{(\tau ^2+5 \tau +13)(\tau ^4+7 \tau ^3+20 \tau ^2+19 \tau +1)^3}{\tau }, \end{aligned}$$
(1.14)

where the Hauptmodul

$$\begin{aligned} \tau =\left( \frac{\eta (z)}{\eta (13z)}\right) ^2 =\root 5 \of {\frac{\Phi _{12}(a_1(z), a_2(z), a_3(z), a_4(z), a_5(z), a_6(z))}{(a_1(z) a_2(z) a_3(z) a_4(z) a_5(z) a_6(z))^2}}. \end{aligned}$$
(1.15)

Note that in the right-hand side of modular equations (1.13) and (1.14), we have the following factorizations over \(\mathbf{Q}(\sqrt{13})\):

$$\begin{aligned}&\tau ^4+247 \tau ^3+3380 \tau ^2+15379 \tau +28561\\&\quad =\left( \tau ^2+\frac{247+65 \sqrt{13}}{2} \tau +\frac{1859+507 \sqrt{13}}{2}\right) \\&\qquad \times \,\left( \tau ^2+\frac{247-65 \sqrt{13}}{2} \tau +\frac{1859-507 \sqrt{13}}{2}\right) ,\\&\tau ^4+7 \tau ^3+20 \tau ^2+19 \tau +1\\&\quad =\left( \tau ^2+\frac{7+\sqrt{13}}{2} \tau +\frac{11+3 \sqrt{13}}{2}\right) \left( \tau ^2+\frac{7-\sqrt{13}}{2} \tau +\frac{11-3 \sqrt{13}}{2}\right) . \end{aligned}$$

P. Deligne (Letter to the author, July 29, 2014. Private communication) gave a modular interpretation of why such factorizations exist. More generally, P. Deligne (Letter to the author, August 7, 2014. Private communication) showed that for \(p=3, 5, 7, 13\), the corresponding modular equations of degrees 4, 6, 8, 14 have two relatively prime conjugate factors over some real quadratic fields. All of these factorizations have nice geometric interpretations.

In his notebooks (see [24, p. 326], [25, p. 244]), Ramanujan (see also [9] or [2, pp. 373–375]) obtained the following modular equation of degree 13:

$$\begin{aligned} 1+G_1(z) G_5(z)+G_2(z) G_3(z)+G_4(z) G_6(z)=-\frac{\eta ^2(z/13)}{\eta ^2(z)} \end{aligned}$$
(1.16)

with

$$\begin{aligned} G_1(z)G_5(z)G_2(z)G_3(z)G_4(z)G_6(z)=-1, \end{aligned}$$
(1.17)

where \(G_m(z):=G_{m, p}(z):=(-1)^m q^{m(3m-p)/(2p^2)} \frac{f(-q^{2m/p}, -q^{1-2m/p})}{f(-q^{m/p}, -q^{1-m/p})}\). Here, m is a positive integer, \(p=13\), \(q=e^{2 \pi i z}\), and Ramanujan’s general theta function f(a, b) is given by \(f(a, b):=\sum _{n=-\infty }^{\infty } a^{n(n+1)/2} b^{n(n-1)/2}\), \(|ab|<1\). We call (1.16) the standard modular equation of degree 13. In contrast with it, we find the following invariant decomposition formula (exotic modular equation) which has the same form as (1.16), but with different kinds of modular parametrizations. Let us define the following senary quadratic forms (quadratic forms in six variables):

$$\begin{aligned} \left\{ \begin{array}{ll} \mathbf{A}_0 &{}=z_1 z_4+z_2 z_5+z_3 z_6,\\ \mathbf{A}_1 &{}=z_1^2-2 z_3 z_4,\\ \mathbf{A}_2 &{}=-z_5^2-2 z_2 z_4,\\ \mathbf{A}_3 &{}=z_2^2-2 z_1 z_5,\\ \mathbf{A}_4 &{}=z_3^2-2 z_2 z_6,\\ \mathbf{A}_5 &{}=-z_4^2-2 z_1 z_6,\\ \mathbf{A}_6 &{}=-z_6^2-2 z_3 z_5. \end{array}\right. \end{aligned}$$
(1.18)

Theorem 1.3

(Main Theorem 2) The following invariant decomposition formula (exotic modular equation) holds : 

$$\begin{aligned} \frac{\Psi _2(a_1(z), a_2(z), a_3(z), a_4(z), a_5(z), a_6(z))}{\mathbf{A}_0(a_1(z), a_2(z), a_3(z), a_4(z), a_5(z), a_6(z))^2}=0, \end{aligned}$$
(1.19)

where the quadric

$$\begin{aligned} \Psi _2(z_1, z_2, z_3, z_4, z_5, z_6):= \mathbf{A}_0^2+\mathbf{A}_1 \mathbf{A}_5+\mathbf{A}_2 \mathbf{A}_3+\mathbf{A}_4 \mathbf{A}_6 \end{aligned}$$
(1.20)

is an invariant associated to PSL(2, 13) and \(\mathbf{A}_0^2\) is invariant under the action of the image of a Borel subgroup of PSL(2, 13), i.e., a maximal subgroup of order 78 of PSL(2, 13), which can be viewed as \(\Gamma _0(13)/\Gamma (13)\).

Note that (1.19) has the same form as (1.16). However, in contrast with (1.17), it can be proved that (see Sect. 3, (3.8))

$$\begin{aligned} \prod _{j=1}^{6} \frac{\mathbf{A}_j(a_1(z), a_2(z), a_3(z), a_4(z), a_5(z), a_6(z))}{\mathbf{A}_0(a_1(z), a_2(z), a_3(z), a_4(z), a_5(z), a_6(z))} \ne -1. \end{aligned}$$
(1.21)

Moreover, the invariant quadric (1.20) is closely related to the exceptional Lie group \(G_2\) (see [27] for more details). As an application, we obtain the following quartic four-fold \(\Phi _4(z_1, z_2, z_3, z_4, z_5, z_6)=0\), where

$$\begin{aligned} \begin{array}{ll} \Phi _4:=&{}(z_3 z_4^3+z_1 z_5^3+z_2 z_6^3)-(z_6 z_1^3+z_4 z_2^3+z_5 z_3^3)\\ &{}+\,3(z_1 z_2 z_4 z_5+z_2 z_3 z_5 z_6+z_3 z_1 z_6 z_4), \end{array} \end{aligned}$$
(1.22)

which is just the quadric (1.20) up to a constant. It is a higher dimensional counterpart of the Klein quartic curve (see [14]) and the Klein cubic three-fold (see [15]). Its significance comes from the following:

Corollary 1.4

The coordinates \((a_1(z), a_2(z), a_3(z), a_4(z), a_5(z), a_6(z))\) map X(13) into the quartic four-fold \(\Phi _4(z_1, z_2, z_3, z_4, z_5, z_6)=0\) in \(\mathbf{C} \mathbf{P}^5\).

We remark that in our preprint [28], the full invariant theory of PSL(2, 13), i.e., the determination of all polynomial invariants (not merely \(\Phi _4\), \(\Phi _{12}\)) is worked out to give the exotic structure associated with the equation of the \(E_8\) singularity.

This paper consists of four sections. In Sect. 2, we give a six-dimensional representation of PSL(2, 13) defined over \(\mathbf{Q}(e^{\frac{2 \pi i}{13}})\) and the transformation formulas for theta constants associated with \(\Gamma (13)\). In Sect. 3, we give a seven-dimensional representation of PSL(2, 13) which is deduced from our six-dimensional representation. As an application, we obtain the exotic modular equation of degree 13. Thus, we give the proof of Theorem 1.3. In Sect. 4, we give a 14-dimensional representation of PSL(2, 13) which is also deduced from our six-dimensional representation. By this representation, we find the exotic modular equation of degree 14. Using it, we give the proof of Theorem 1.1.

2 Six-dimensional representations of PSL(2, 13) and transformation formulas for theta constants

In this section, we will study the six-dimensional representation of the simple group PSL(2, 13) of order 1092, which acts on the five-dimensional projective space \(\mathbf{P}^5=\{ (z_1, z_2, z_3, z_4, z_5, z_6): z_i \in \mathbf{C} (i=1, 2, 3, 4, 5, 6) \}\).

Let \(\zeta =\exp (2 \pi i/13)\), \(\theta _1=\zeta +\zeta ^3+\zeta ^9\), \(\theta _2=\zeta ^2+\zeta ^6+\zeta ^5\), \(\theta _3=\zeta ^4+\zeta ^{12}+\zeta ^{10}\), and \(\theta _4=\zeta ^8+\zeta ^{11}+\zeta ^7\). We find that

$$\begin{aligned} \left\{ \begin{array}{ll} \theta _1+\theta _2+\theta _3+\theta _4=-1,\\ \theta _1 \theta _2+\theta _1 \theta _3+\theta _1 \theta _4+\theta _2 \theta _3+\theta _2 \theta _4+\theta _3 \theta _4=2,\\ \theta _1 \theta _2 \theta _3+\theta _1 \theta _2 \theta _4+\theta _1 \theta _3 \theta _4+\theta _2 \theta _3 \theta _4=4,\\ \theta _1 \theta _2 \theta _3 \theta _4=3. \end{array}\right. \end{aligned}$$

Hence, \(\theta _1\), \(\theta _2\), \(\theta _3\), and \(\theta _4\) satisfy the quartic equation \(z^4+z^3+2 z^2-4z+3=0\), which can be decomposed as two quadratic equations

$$\begin{aligned} \left( z^2+\frac{1+\sqrt{13}}{2} z+\frac{5+\sqrt{13}}{2}\right) \left( z^2+\frac{1-\sqrt{13}}{2} z+\frac{5-\sqrt{13}}{2}\right) =0 \end{aligned}$$

over the real quadratic field \(\mathbf{Q}(\sqrt{13})\). Therefore, the four roots are given as follows:

$$\begin{aligned} \left\{ \begin{array}{ll} \theta _1=\frac{1}{4} \left( -1+\sqrt{13}+\sqrt{-26+6 \sqrt{13}}\right) ,\\ \theta _2=\frac{1}{4} \left( -1-\sqrt{13}+\sqrt{-26-6 \sqrt{13}}\right) ,\\ \theta _3=\frac{1}{4} \left( -1+\sqrt{13}-\sqrt{-26+6 \sqrt{13}}\right) ,\\ \theta _4=\frac{1}{4} \left( -1-\sqrt{13}-\sqrt{-26-6 \sqrt{13}}\right) . \end{array}\right. \end{aligned}$$

Moreover, we find that

$$\begin{aligned} \left\{ \begin{array}{ll} \theta _1+\theta _3+\theta _2+\theta _4 &{}=-1,\\ \theta _1+\theta _3-\theta _2-\theta _4 &{}=\sqrt{13},\\ \theta _1-\theta _3-\theta _2+\theta _4 &{}=-\sqrt{-13+2 \sqrt{13}},\\ \theta _1-\theta _3+\theta _2-\theta _4 &{}=\sqrt{-13-2 \sqrt{13}}. \end{array} \right. \end{aligned}$$

Let \(S=-\frac{1}{\sqrt{13}} \left( \begin{array}{cc} -M &{} N\\ N &{} M \end{array}\right) \) and \(T=\text{ diag }(\zeta ^7, \zeta ^{11}, \zeta ^8, \zeta ^6, \zeta ^2, \zeta ^5)\), where

$$\begin{aligned} M=\left( \begin{array}{lll} \zeta -\zeta ^{12} &{} \zeta ^3-\zeta ^{10} &{} \zeta ^9-\zeta ^4\\ \zeta ^3-\zeta ^{10} &{} \zeta ^9-\zeta ^4 &{} \zeta -\zeta ^{12}\\ \zeta ^9-\zeta ^4 &{} \zeta -\zeta ^{12} &{} \zeta ^3-\zeta ^{10} \end{array}\right) , \end{aligned}$$
$$\begin{aligned} N=\left( \begin{array}{lll} \zeta ^5-\zeta ^8 &{} \zeta ^2-\zeta ^{11} &{} \zeta ^6-\zeta ^7\\ \zeta ^2-\zeta ^{11} &{} \zeta ^6-\zeta ^7 &{} \zeta ^5-\zeta ^8\\ \zeta ^6-\zeta ^7 &{} \zeta ^5-\zeta ^8 &{} \zeta ^2-\zeta ^{11} \end{array}\right) . \end{aligned}$$

Then \(MN=NM=-\sqrt{13} I\), \(M^2+N^2=-13 I\) and \(S^2=I\).

Theorem 2.1

Let \(G=\langle S, T \rangle \). Then \(G \cong PSL(2, 13)\).

The proof is elementary: Magma will confirm it immediately.

Recall that the theta functions with characteristic \(\left[ \begin{array}{l} \epsilon \\ \epsilon ^{\prime } \end{array} \right] \in \mathbf{R}^2\) are defined by the following series which converges uniformly and absolutely on compact subsets of \(\mathbf{C} \times \mathbf{H}\) (see [10, p. 73]):

$$\begin{aligned} \theta \left[ \begin{array}{l} \epsilon \\ \epsilon ^{\prime } \end{array} \right] (z, \tau ) =\sum _{n \in \mathbf{Z}} \exp \left\{ 2 \pi i \left[ \frac{1}{2} \left( n+\frac{\epsilon }{2}\right) ^2 \tau +\left( n+\frac{\epsilon }{2}\right) \left( z+\frac{\epsilon ^{\prime }}{2}\right) \right] \right\} . \end{aligned}$$

We introduce the modified theta constants (see [10], p. 215) \(\varphi _l(\tau ):=\theta [\chi _l](0, k \tau )\), where the characteristic \(\chi _l=\left[ \begin{array}{c} \frac{2l+1}{k}\\ 1 \end{array}\right] \), \(l=0, \ldots , \frac{k-3}{2}\), for odd k and \(\chi _l=\left[ \begin{array}{c} \frac{2l}{k}\\ 0 \end{array}\right] \), \(l=0, \cdots , \frac{k}{2}\), for even k. We have the following:

Theorem 2.2

(See [10, p. 236]) For each odd integer \(k \ge 5\), the map \(\Phi : \tau \mapsto (\varphi _0(\tau ), \varphi _1(\tau ), \ldots , \varphi _{\frac{k-5}{2}}(\tau ), \varphi _{\frac{k-3}{2}}(\tau ))\) from \(\mathbf{H} \cup \mathbf{Q} \cup \{ \infty \}\) to \(\mathbf{C}^{\frac{k-1}{2}}\) defines a holomorphic mapping from \(\overline{\mathbf{H}/\Gamma (k)}\) into \(\mathbf{C} \mathbf{P}^{\frac{k-3}{2}}\).

In our case, the map \(\Phi : \tau \mapsto (\varphi _0(\tau ), \varphi _1(\tau ), \varphi _2(\tau ), \varphi _3(\tau ), \varphi _4(\tau ), \varphi _5(\tau ))\) gives a holomorphic mapping from the modular curve \(X(13)=\overline{\mathbf{H}/\Gamma (13)}\) into \(\mathbf{C} \mathbf{P}^5\), which corresponds to our six-dimensional representation, i.e., up to the constants, \(z_1, \ldots , z_6\) are just modular forms \(\varphi _0(\tau ), \ldots , \varphi _5(\tau )\).

Let \(a_i(z)\) \((1 \le i \le 6)\) be given as in (1.8). We use the standard notation \((a)_{\infty }:=(a; q):=\prod _{k=0}^{\infty } (1-aq^k)\) for \(a \in \mathbf{C}^{\times }\) and \(|q|<1\) so that \(\eta (z)=q^{1/24}(q; q)\). By the Jacobi triple product identity, we have that

$$\begin{aligned} \theta \left[ \begin{array}{l} \frac{l}{k}\\ 1 \end{array} \right] (0, kz) =\exp \left( \frac{\pi i l}{2k}\right) q^{\frac{l^2}{8k}} (q^{\frac{k-l}{2}}; q^k)(q^{\frac{k+l}{2}}; q^k)(q^k; q^k), \end{aligned}$$

where k is odd and \(l=1, 3, 5, \ldots , k-2\). Hence,

$$\begin{aligned} \left\{ \begin{array}{ll} a_1(z) &{}=q^{\frac{121}{104}} (q; q^{13})(q^{12}; q^{13})(q^{13}; q^{13}),\\ a_2(z) &{}=q^{\frac{49}{104}} (q^3; q^{13})(q^{10}; q^{13})(q^{13}; q^{13}),\\ a_3(z) &{}=q^{\frac{25}{104}} (q^9; q^{13})(q^4; q^{13})(q^{13}; q^{13}),\\ a_4(z) &{}=-q^{\frac{9}{104}} (q^5; q^{13})(q^8; q^{13})(q^{13}; q^{13}),\\ a_5(z) &{}=q^{\frac{81}{104}} (q^2; q^{13})(q^{11}; q^{13})(q^{13}; q^{13}),\\ a_6(z) &{}=q^{\frac{1}{104}} (q^6; q^{13})(q^7; q^{13})(q^{13}; q^{13}). \end{array} \right. \end{aligned}$$
(2.1)

It is known that Ramanujan’s general theta functions are given as follows:

$$\begin{aligned} f(a, b):=\sum _{n=-\infty }^{\infty } a^{n(n+1)/2} b^{n(n-1)/2}, \end{aligned}$$
$$\begin{aligned} f(-q):=f(-q, -q^2)=\sum _{n=-\infty }^{\infty } (-1)^n q^{n(3n-1)/2}=(q; q)_{\infty }. \end{aligned}$$

In his notebooks (see [2, p. 372], [24, p. 326], and [25, p. 244]), Ramanujan obtained his modular equations of degree 13.

Theorem 2.3

Define

$$\begin{aligned} \mu _1=\frac{f(-q^4, -q^9)}{q^{7/13} f(-q^2, -q^{11})}, \mu _2=\frac{f(-q^6, -q^7)}{q^{6/13} f(-q^3, -q^{10})}, \mu _3=\frac{f(-q^2, -q^{11})}{q^{5/13} f(-q, -q^{12})}, \end{aligned}$$
$$\begin{aligned} \mu _4=\frac{f(-q^5, -q^8)}{q^{2/13} f(-q^4, -q^9)}, \mu _5=\frac{q^{5/13} f(-q^3, -q^{10})}{f(-q^5, -q^8)}, \mu _6=\frac{q^{15/13} f(-q, -q^{12})}{f(-q^6, -q^7)}. \end{aligned}$$

Then

$$\begin{aligned} 1+\frac{f^2(-q)}{q f^2(-q^{13})}=\mu _1 \mu _2-\mu _3 \mu _5-\mu _4 \mu _6, \end{aligned}$$
(2.2)
$$\begin{aligned} -4-\frac{f^2(-q)}{q f^2(-q^{13})}=\frac{1}{\mu _1 \mu _2}-\frac{1}{\mu _3 \mu _5}-\frac{1}{\mu _4 \mu _6}, \end{aligned}$$
(2.3)
$$\begin{aligned} 3+\frac{f^2(-q)}{q f^2(-q^{13})}=\mu _2 \mu _3 \mu _4-\mu _1 \mu _5 \mu _6, \end{aligned}$$
(2.4)

where \(\mu _1 \mu _2 \mu _3 \mu _4 \mu _5 \mu _6=1\).

Let \(G_m(z):=G_{m, p}(z):=(-1)^m q^{m(3m-p)/(2p^2)} \frac{f(-q^{2m/p}, -q^{1-2m/p})}{f(-q^{m/p}, -q^{1-m/p})}\) where m is a positive integer, \(p=13\), and \(q=e^{2 \pi i z}\). Then the above three formulas (2.2), (2.3), and (2.4) are equivalent to the following (see [9] or [2, pp. 373–375])

$$\begin{aligned} 1+G_1(z) G_5(z)+G_2(z) G_3(z)+G_4(z) G_6(z)=-\frac{\eta ^2(z/13)}{\eta ^2(z)}, \end{aligned}$$
(2.5)
$$\begin{aligned} \frac{1}{G_1(z) G_5(z)}+\frac{1}{G_2(z) G_3(z)}+\frac{1}{G_4(z) G_6(z)} =4+\frac{\eta ^2(z/13)}{\eta ^2(z)}, \end{aligned}$$
(2.6)
$$\begin{aligned} G_1(z) G_3(z) G_4(z)-\frac{1}{G_1(z) G_3(z) G_4(z)} =3+\frac{\eta ^2(z/13)}{\eta ^2(z)}, \end{aligned}$$
(2.7)

where \(G_1(z) G_2(z) G_3(z) G_4(z) G_5(z) G_6(z)=-1\). Moreover, there is the following formula (see [9] or [2, pp. 375–376]): for \(t=q^{1/13}\),

$$\begin{aligned} \frac{1}{(t^2)_{\infty } (t^3)_{\infty } (t^{10})_{\infty } (t^{11})_{\infty }} +\frac{t}{(t^4)_{\infty } (t^6)_{\infty } (t^7)_{\infty } (t^9)_{\infty }} =\frac{1}{(t)_{\infty } (t^5)_{\infty } (t^8)_{\infty } (t^{12})_{\infty }},\nonumber \\ \end{aligned}$$
(2.8)

which is equivalent to, for \(p=13\), \(G_1^{-1}(z) G_5^{-1}(z)+G_4(z) G_6(z)=1\).

It is known that (see [9]) \(G(m; z)=(-1)^m F(2m/p; z)/F(m/p; z)\), where

$$\begin{aligned} F(u; z)= & {} -i \sum _{k=-\infty }^{\infty } (-1)^k q^{(k+u+1/2)^2/2}\\= & {} -i q^{(u+1/2)^2/2} \prod _{m=1}^{\infty } (1-q^{m+u})(1-q^{m-1-u}) (1-q^m) \end{aligned}$$

satisfies that \(F(u+1; z)=-F(u; z)\) and \(F(-u; z)=-F(u; z)\). We have that

$$\begin{aligned} G_1(z) G_5(z)=\frac{F(\frac{2}{13}; z) F(\frac{3}{13}; z)}{F(\frac{1}{13}; z) F(\frac{5}{13}; z)} =\frac{(t^3)_{\infty } (t^{10})_{\infty } (t^2)_{\infty } (t^{11})_{\infty }}{(t)_{\infty } (t^{12})_{\infty } (t^5)_{\infty }(t^8)_{\infty }}, \end{aligned}$$

where \((x)_{\infty }=\prod _{m=0}^{\infty } (1-xq^m)\) and \(t=q^{\frac{1}{13}}\). Similarly,

$$\begin{aligned} G_2(z) G_3(z)=-\frac{(t^9)_{\infty } (t^4)_{\infty } (t^6)_{\infty } (t^7)_{\infty }}{t (t^3)_{\infty } (t^{10})_{\infty } (t^2)_{\infty } (t^{11})_{\infty }}, \end{aligned}$$
$$\begin{aligned} G_4(z) G_6(z)=\frac{t (t)_{\infty } (t^{12})_{\infty } (t^5)_{\infty } (t^8)_{\infty }}{(t^9)_{\infty } (t^4)_{\infty } (t^6)_{\infty }(t^7)_{\infty }}. \end{aligned}$$

Note that \((t^k)_{\infty }=(t^k; t^{13})\) for \(1 \le k \le 13\). The above formula (2.5) is equivalent to

$$\begin{aligned}&t (t^3; t^{13})^2 (t^{10}; t^{13})^2 (t^2; t^{13})^2 (t^{11}; t^{13})^2 (t^9; t^{13}) (t^4; t^{13}) (t^6; t^{13}) (t^7; t^{13})\\&\quad -\,(t^9; t^{13})^2 (t^4; t^{13})^2 (t^6; t^{13})^2 (t^7; t^{13})^2 (t; t^{13}) (t^{12}; t^{13}) (t^5; t^{13}) (t^8; t^{13})\\&\quad +\,t^2 (t; t^{13})^2 (t^{12}; t^{13})^2 (t^5; t^{13})^2 (t^8; t^{13})^2 (t^3; t^{13}) (t^{10}; t^{13}) (t^2; t^{13}) (t^{11}; t^{13})\\&=[-1-\eta ^2(z/13)/\eta ^2(z)] t (t; t^{13}) (t^2; t^{13}) \cdots (t^{12}; t^{13}), \quad t=q^{1/13}=e^{\frac{2 \pi i z}{13}}. \end{aligned}$$

On the other hand,

$$\begin{aligned} a_2(z)^2 a_5(z)^2 a_3(z) a_6(z)= & {} q^{\frac{11}{4}} (q^3; q^{13})^2 (q^{10}; q^{13})^2 (q^2; q^{13})^2 (q^{11}; q^{13})^2\\&\times \, (q^9; q^{13}) (q^4; q^{13}) (q^6; q^{13}) (q^7; q^{13}) (q^{13}; q^{13})^6, \end{aligned}$$
$$\begin{aligned} a_3(z)^2 a_6(z)^2 a_1(z) a_4(z)= & {} -q^{\frac{7}{4}} (q^9; q^{13})^2 (q^4; q^{13})^2 (q^6; q^{13})^2 (q^7; q^{13})^2\\&\times \,(q; q^{13}) (q^{12}; q^{13}) (q^5; q^{13}) (q^8; q^{13}) (q^{13}; q^{13})^6, \end{aligned}$$
$$\begin{aligned} a_1(z)^2 a_4(z)^2 a_2(z) a_5(z)= & {} q^{\frac{15}{4}} (q; q^{13})^2 (q^{12}; q^{13})^2 (q^5; q^{13})^2 (q^8; q^{13})^2\\&\times \, (q^3; q^{13}) (q^{10}; q^{13}) (q^2; q^{13}) (q^{11}; q^{13}) (q^{13}; q^{13})^6. \end{aligned}$$

This implies that

$$\begin{aligned}&a_1(z)^2 a_4(z)^2 a_2(z) a_5(z)+a_2(z)^2 a_5(z)^2 a_3(z) a_6(z) +a_3(z)^2 a_6(z)^2 a_1(z) a_4(z)\nonumber \\&\,\,=[-1-\eta ^2(z)/\eta ^2(13 z)] \eta (z) \eta (13 z)^5. \end{aligned}$$
(2.9)

Similarly, the other three formulas (2.6), (2.7), and (2.8) are equivalent to the following:

$$\begin{aligned}&a_1(z)^2 a_4(z)^2 a_3(z) a_6(z)+a_2(z)^2 a_5(z)^2 a_1(z) a_4(z) +a_3(z)^2 a_6(z)^2 a_2(z) a_5(z)\nonumber \\&\,\,=[4+\eta ^2(z)/\eta ^2(13 z)] \eta (z) \eta (13 z)^5, \end{aligned}$$
(2.10)
$$\begin{aligned} a_4(z)^2 a_5(z)^2 a_6(z)^2-a_1(z)^2 a_2(z)^2 a_3(z)^2 =[3+\eta ^2(z)/\eta ^2(13 z)] \eta (z) \eta (13 z)^5,\nonumber \\ \end{aligned}$$
(2.11)

and

$$\begin{aligned} \frac{1}{a_1(z) a_4(z)}+\frac{1}{a_2(z) a_5(z)}+\frac{1}{a_3(z) a_6(z)}=0, \end{aligned}$$
(2.12)

where \(a_1(z) a_2(z) a_3(z) a_4(z) a_5(z) a_6(z)=-\eta (z) \eta (13 z)^5\).

Let \(\mathbf{A}(z)=(a_1(z), a_2(z), a_3(z), a_4(z), a_5(z), a_6(z))^{T}\). The significance of our six-dimensional representation of PSL(2, 13) comes from the following:

Proposition 2.4

If \(z \in \mathbf{H}\), then the following relations hold:

$$\begin{aligned} \mathbf{A}(z+1)=e^{-\frac{3 \pi i}{4}} T \mathbf{A}(z), \quad \mathbf{A}\left( -\frac{1}{z}\right) =e^{\frac{\pi i}{4}} \sqrt{z} S \mathbf{A}(z), \end{aligned}$$
(2.13)

where \(T=\text{ diag }(\zeta ^7, \zeta ^{11}, \zeta ^8, \zeta ^6, \zeta ^2, \zeta ^5)\),

$$\begin{aligned} S=\frac{-1}{\sqrt{13}} \left( \begin{array}{cccccc} \zeta ^{12}-\zeta &{} \zeta ^{10}-\zeta ^3 &{} \zeta ^4-\zeta ^9 &{} \zeta ^5-\zeta ^8 &{} \zeta ^2-\zeta ^{11} &{} \zeta ^6-\zeta ^7\\ \zeta ^{10}-\zeta ^3 &{} \zeta ^4-\zeta ^9 &{} \zeta ^{12}-\zeta &{} \zeta ^2-\zeta ^{11} &{} \zeta ^6-\zeta ^7 &{} \zeta ^5-\zeta ^8\\ \zeta ^4-\zeta ^9 &{} \zeta ^{12}-\zeta &{} \zeta ^{10}-\zeta ^3 &{} \zeta ^6-\zeta ^7 &{} \zeta ^5-\zeta ^8 &{} \zeta ^2-\zeta ^{11}\\ \zeta ^5-\zeta ^8 &{} \zeta ^2-\zeta ^{11} &{} \zeta ^6-\zeta ^7 &{} \zeta -\zeta ^{12} &{} \zeta ^3-\zeta ^{10} &{} \zeta ^9-\zeta ^4\\ \zeta ^2-\zeta ^{11} &{} \zeta ^6-\zeta ^7 &{} \zeta ^5-\zeta ^8 &{} \zeta ^3-\zeta ^{10} &{} \zeta ^9-\zeta ^4 &{} \zeta -\zeta ^{12}\\ \zeta ^6-\zeta ^7 &{} \zeta ^5-\zeta ^8 &{} \zeta ^2-\zeta ^{11} &{} \zeta ^9-\zeta ^4 &{} \zeta -\zeta ^{12} &{} \zeta ^3-\zeta ^{10} \end{array}\right) , \end{aligned}$$

and \(0<\text{ arg } \sqrt{z} \le \pi /2\).

The proof is an application of the following transformation formulas for theta constants (see [10, pp. 216–217]) to the special case \(k=13\):

$$\begin{aligned} \theta \left[ \begin{array}{l} \frac{2l+1}{k}\\ 1 \end{array}\right] (0, k(z+1))=\exp \left( \frac{\pi i}{4k}\right) \exp \left( \frac{\pi i}{k}(l^2+l) \right) \theta \left[ \begin{array}{l} \frac{2l+1}{k}\\ 1 \end{array}\right] (0, kz) \end{aligned}$$

and

$$\begin{aligned} \theta \left[ \begin{array}{l} \frac{2l+1}{k}\\ 1 \end{array} \right] (0, k(-1/z))= & {} \frac{\sqrt{z}}{\sqrt{ik}} \exp \left( \frac{\pi i}{2k}\right) \sum \limits _{j=0}^{\frac{k-3}{2}} \left[ \exp \left( 2 \pi i \frac{l(j+1)}{k}\right) \right. \\&\left. +\, \exp \left( -\frac{\pi i}{k}\right) \exp \left( -2 \pi i \frac{(l+1)j}{k}\right) \right] \theta \left[ \begin{array}{l} \frac{2j+1}{k}\\ 1 \end{array}\right] (0, kz), \end{aligned}$$

where k is odd and \(l=0, 1, \ldots , \frac{k-3}{2}\).

3 Seven-dimensional representations of PSL(2, 13), exotic modular equation and geometry of modular curve X(13)

We will construct a seven-dimensional representation of PSL(2, 13) which is deduced from our six-dimensional representation. Let us study the action of \(S T^{\nu }\) on the five-dimensional projective space \(\mathbf{P}^5=\{(z_1, z_2, z_3, z_4, z_5, z_6)\}\), where \(\nu =0, 1, \ldots , 12\). Put

$$\begin{aligned} \alpha =\zeta +\zeta ^{12}-\zeta ^5-\zeta ^8, \quad \beta =\zeta ^3+\zeta ^{10}-\zeta ^2-\zeta ^{11}, \quad \gamma =\zeta ^9+\zeta ^4-\zeta ^6-\zeta ^7. \end{aligned}$$

We find that

$$\begin{aligned}&13 ST^{\nu }(z_1) \cdot ST^{\nu }(z_4)\\&\,\,=\beta z_1 z_4+\gamma z_2 z_5+\alpha z_3 z_6+\\&\quad +\,\gamma \zeta ^{\nu } z_1^2+\alpha \zeta ^{9 \nu } z_2^2+\beta \zeta ^{3 \nu } z_3^2 -\gamma \zeta ^{12 \nu } z_4^2-\alpha \zeta ^{4 \nu } z_5^2-\beta \zeta ^{10 \nu } z_6^2+\\&\quad +\,(\alpha -\beta ) \zeta ^{5 \nu } z_1 z_2+(\beta -\gamma ) \zeta ^{6 \nu } z_2 z_3 +(\gamma -\alpha ) \zeta ^{2 \nu } z_1 z_3+\\&\quad +\,(\beta -\alpha ) \zeta ^{8 \nu } z_4 z_5+(\gamma -\beta ) \zeta ^{7 \nu } z_5 z_6 +(\alpha -\gamma ) \zeta ^{11 \nu } z_4 z_6+\\&\quad -\,(\alpha +\beta ) \zeta ^{\nu } z_3 z_4-(\beta +\gamma ) \zeta ^{9 \nu } z_1 z_5 -(\gamma +\alpha ) \zeta ^{3 \nu } z_2 z_6+\\&\quad -\,(\alpha +\beta ) \zeta ^{12 \nu } z_1 z_6-(\beta +\gamma ) \zeta ^{4 \nu } z_2 z_4 -(\gamma +\alpha ) \zeta ^{10 \nu } z_3 z_5. \end{aligned}$$
$$\begin{aligned}&13 ST^{\nu }(z_2) \cdot ST^{\nu }(z_5)\\&\,\,\,=\gamma z_1 z_4+\alpha z_2 z_5+\beta z_3 z_6+\\&\quad +\,\alpha \zeta ^{\nu } z_1^2+\beta \zeta ^{9 \nu } z_2^2+\gamma \zeta ^{3 \nu } z_3^2 -\alpha \zeta ^{12 \nu } z_4^2-\beta \zeta ^{4 \nu } z_5^2-\gamma \zeta ^{10 \nu } z_6^2+\\&\quad +\,(\beta -\gamma ) \zeta ^{5 \nu } z_1 z_2+(\gamma -\alpha ) \zeta ^{6 \nu } z_2 z_3 +(\alpha -\beta ) \zeta ^{2 \nu } z_1 z_3+\\&\quad +\,(\gamma -\beta ) \zeta ^{8 \nu } z_4 z_5+(\alpha -\gamma ) \zeta ^{7 \nu } z_5 z_6 +(\beta -\alpha ) \zeta ^{11 \nu } z_4 z_6+\\&\quad -\,(\beta +\gamma ) \zeta ^{\nu } z_3 z_4-(\gamma +\alpha ) \zeta ^{9 \nu } z_1 z_5 -(\alpha +\beta ) \zeta ^{3 \nu } z_2 z_6+\\&\quad -\,(\beta +\gamma ) \zeta ^{12 \nu } z_1 z_6-(\gamma +\alpha ) \zeta ^{4 \nu } z_2 z_4 -(\alpha +\beta ) \zeta ^{10 \nu } z_3 z_5. \end{aligned}$$
$$\begin{aligned}&13 ST^{\nu }(z_3) \cdot ST^{\nu }(z_6)\\&\,\,\,=\alpha z_1 z_4+\beta z_2 z_5+\gamma z_3 z_6+\\&\quad +\,\beta \zeta ^{\nu } z_1^2+\gamma \zeta ^{9 \nu } z_2^2+\alpha \zeta ^{3 \nu } z_3^2 -\beta \zeta ^{12 \nu } z_4^2-\gamma \zeta ^{4 \nu } z_5^2-\alpha \zeta ^{10 \nu } z_6^2+\\&\quad +\,(\gamma -\alpha ) \zeta ^{5 \nu } z_1 z_2+(\alpha -\beta ) \zeta ^{6 \nu } z_2 z_3 +(\beta -\gamma ) \zeta ^{2 \nu } z_1 z_3+\\&\quad +\,(\alpha -\gamma ) \zeta ^{8 \nu } z_4 z_5+(\beta -\alpha ) \zeta ^{7 \nu } z_5 z_6 +(\gamma -\beta ) \zeta ^{11 \nu } z_4 z_6+\\&\quad -\,(\gamma +\alpha ) \zeta ^{\nu } z_3 z_4-(\alpha +\beta ) \zeta ^{9 \nu } z_1 z_5 -(\beta +\gamma ) \zeta ^{3 \nu } z_2 z_6+\\&\quad -\,(\gamma +\alpha ) \zeta ^{12 \nu } z_1 z_6-(\alpha +\beta ) \zeta ^{4 \nu } z_2 z_4 -(\beta +\gamma ) \zeta ^{10 \nu } z_3 z_5. \end{aligned}$$

Note that \(\alpha +\beta +\gamma =\sqrt{13}\), we find that

$$\begin{aligned}&\sqrt{13} \left[ ST^{\nu }(z_1) \cdot ST^{\nu }(z_4)+ST^{\nu }(z_2) \cdot ST^{\nu }(z_5) +ST^{\nu }(z_3) \cdot ST^{\nu }(z_6)\right] \\&\quad =(z_1 z_4+z_2 z_5+z_3 z_6)+(\zeta ^{\nu } z_1^2+\zeta ^{9 \nu } z_2^2+\zeta ^{3 \nu } z_3^2)+\\&\qquad -\,(\zeta ^{12 \nu } z_4^2+\zeta ^{4 \nu } z_5^2+\zeta ^{10 \nu } z_6^2)-2(\zeta ^{\nu } z_3 z_4+ \zeta ^{9 \nu } z_1 z_5+\zeta ^{3 \nu } z_2 z_6)\\&\qquad -\,2(\zeta ^{12 \nu } z_1 z_6+\zeta ^{4 \nu } z_2 z_4+\zeta ^{10 \nu } z_3 z_5). \end{aligned}$$

Let

$$\begin{aligned} \varphi _{\infty }(z_1, z_2, z_3, z_4, z_5, z_6)=\sqrt{13} (z_1 z_4+z_2 z_5+z_3 z_6) \end{aligned}$$
(3.1)

and

$$\begin{aligned} \varphi _{\nu }(z_1, z_2, z_3, z_4, z_5, z_6)=\varphi _{\infty }(ST^{\nu }(z_1, z_2, z_3, z_4, z_5, z_6)) \end{aligned}$$
(3.2)

for \(\nu =0, 1, \ldots , 12\). Then

$$\begin{aligned} \varphi _{\nu }= & {} (z_1 z_4+z_2 z_5+z_3 z_6)+\zeta ^{\nu } (z_1^2-2 z_3 z_4)+\zeta ^{4 \nu } (-z_5^2-2 z_2 z_4)\nonumber \\&+\,\zeta ^{9 \nu } (z_2^2-2 z_1 z_5)+\zeta ^{3 \nu } (z_3^2-2 z_2 z_6)+\zeta ^{12 \nu } (-z_4^2-2 z_1 z_6)\nonumber \\&+\,\zeta ^{10 \nu } (-z_6^2-2 z_3 z_5). \end{aligned}$$
(3.3)

This leads us to define the senary quadratic forms (quadratic forms in six variables) \(\mathbf{A}_0, \ldots , \mathbf{A}_6\) given in (1.18). Hence,

$$\begin{aligned} \sqrt{13} ST^{\nu }(\mathbf{A}_0)= & {} \mathbf{A}_0+\zeta ^{\nu } \mathbf{A}_1+\zeta ^{4 \nu } \mathbf{A}_2+ \zeta ^{9 \nu } \mathbf{A}_3+\zeta ^{3 \nu } \mathbf{A}_4+\zeta ^{12 \nu } \mathbf{A}_5 \nonumber \\&+\,\zeta ^{10 \nu } \mathbf{A}_6. \end{aligned}$$
(3.4)

Let \(H:=Q^5 P^2 \cdot P^2 Q^6 P^8 \cdot Q^5 P^2 \cdot P^3 Q\), where \(P=S T^{-1} S\) and \(Q=S T^3\). Then

$$\begin{aligned} H=\left( \begin{array}{l@{\quad }l@{\quad }l@{\quad }l@{\quad }l@{\quad }l} 0\,\, &{} 0\,\, &{} 0\,\, &{} 0\,\, &{} 0\,\, &{} 1\\ 0\,\, &{} 0\,\, &{} 0\,\, &{} 1\,\, &{} 0\,\, &{} 0\\ 0\,\, &{} 0\,\, &{} 0\,\, &{} 0\,\, &{} 1\,\, &{} 0\\ 0\,\, &{} 0\,\, &{} -1\,\, &{} 0\,\, &{} 0\,\, &{} 0\\ -1\,\, &{} 0\,\, &{} 0\,\, &{} 0\,\, &{} 0\,\, &{} 0\\ 0\,\, &{} -1\,\, &{} 0\,\, &{} 0\,\, &{} 0\,\, &{} 0 \end{array} \right) . \end{aligned}$$
(3.5)

Note that \(H^6=-I\). In the projective coordinates, this means that \(H^6=1\). We have that \(H^{-1} T H=-T^4\). Thus, \(\langle H, T \rangle \) is isomorphic to the semidirect product of \(\mathbf{Z}_{13}\) by \(\mathbf{Z}_6\). Hence, it is a maximal subgroup of order 78 of G with index 14 (see [6]). It should be pointed out that this complicated expression for H is chosen because it represents an element of \(\Gamma (1)\) which is, in fact, an element of \(\Gamma _0(13)\). In consequence, the group \(\Gamma _0(13)/\Gamma (13)\) is generated. We find that \(\varphi _{\infty }^2\) is invariant under the action of the maximal subgroup \(\langle H, T \rangle \). Note that

$$\begin{aligned} \varphi _{\infty }=\sqrt{13} \mathbf{A}_0, \quad \varphi _{\nu }=\mathbf{A}_0+\zeta ^{\nu } \mathbf{A}_1+\zeta ^{4 \nu } \mathbf{A}_2+ \zeta ^{9 \nu } \mathbf{A}_3+\zeta ^{3 \nu } \mathbf{A}_4+\zeta ^{12 \nu } \mathbf{A}_5+\zeta ^{10 \nu } \mathbf{A}_6 \end{aligned}$$

for \(\nu =0, 1, \ldots , 12\). Let \(w=\varphi ^2\), \(w_{\infty }=\varphi _{\infty }^2\), and \(w_{\nu }=\varphi _{\nu }^2\). Then \(w_{\infty }\), \(w_{\nu }\) for \(\nu =0, \ldots , 12\) form an algebraic equation of degree 14, which is just the Jacobian equation of degree 14, whose roots are these \(w_{\nu }\) and \(w_{\infty }\): \(w^{14}+a_1 w^{13}+\cdots + a_{13} w+a_{14}=0\). In particular, the coefficients

$$\begin{aligned} -a_1=w_{\infty }+\sum _{\nu =0}^{12} w_{\nu }= 26 (\mathbf{A}_0^2+\mathbf{A}_1 \mathbf{A}_5+\mathbf{A}_2 \mathbf{A}_3+\mathbf{A}_4 \mathbf{A}_6). \end{aligned}$$
(3.6)

This leads to an invariant quadric \(\Psi _2:=\mathbf{A}_0^2+\mathbf{A}_1 \mathbf{A}_5+\mathbf{A}_2 \mathbf{A}_3+\mathbf{A}_4 \mathbf{A}_6 =2 \Phi _4(z_1, z_2, z_3, z_4, z_5, z_6)\), where

$$\begin{aligned} \begin{array}{ll} \Phi _4:=&{}(z_3 z_4^3+z_1 z_5^3+z_2 z_6^3)-(z_6 z_1^3+z_4 z_2^3+z_5 z_3^3)\\ &{}+\,3(z_1 z_2 z_4 z_5+z_2 z_3 z_5 z_6+z_3 z_1 z_6 z_4), \end{array} \end{aligned}$$
(3.7)

Hence, the variety \(\Psi _2=0\) is a quartic four-fold, which is invariant under the action of the simple group G.

Recall that the principal congruence subgroup of level 13 is the normal subgroup \(\Gamma (13)\) of \(\Gamma =PSL(2, \mathbf{Z})\) defined by the exact sequence \(1 \rightarrow \Gamma (13) \rightarrow \Gamma (1) \mathop {\rightarrow }\limits ^{f} G \rightarrow 1\), where \(f(\gamma ) \equiv \gamma \) (mod 13) for \(\gamma \in \Gamma =\Gamma (1)\). Then there is a representation \(\rho : \Gamma \rightarrow PGL(6, \mathbf{C})\) with kernel \(\Gamma (13)\) and leaving \(\Phi _4\) invariant. It is defined as follows: if \(t=\left( \begin{array}{ll} 1 &{} 1\\ 0 &{} 1 \end{array} \right) \) and \(s=\left( \begin{array}{ll} 0 &{} -1\\ 1 &{} 0 \end{array} \right) \), then \(\rho (t)=T\) and \(\rho (s)=S\). To see that such a representation exists, note that \(\Gamma \) is defined by the presentation \(\langle s, t; s^2=(st)^3=1 \rangle \) satisfied by s and t and we have proved that S and T satisfy these relations. Moreover, we have proved that G is defined by the presentation \(\langle S, T; S^2=T^{13}=(ST)^3=1, (Q^3 P^4)^3=1 \rangle .\) Let \(p=s t^{-1} s\) and \(q=s t^3\). Then

$$\begin{aligned} h:=q^5 p^2 \cdot p^2 q^6 p^8 \cdot q^5 p^2 \cdot p^3 q =\left( \begin{array}{ll} 4, 428, 249 &{}\quad -10,547,030\\ -11, 594, 791 &{} \quad 27,616,019 \end{array} \right) \end{aligned}$$

satisfies that \(\rho (h)=H\). The off-diagonal elements of the matrix h, which corresponds to H, are congruent to 0 mod 13. The connection to \(\Gamma _0(13)\) should be obvious.

Theorem 3.1

There is a relation between the invariant quartic four-fold \(\Phi _4(z_1, \ldots , z_6)=0\) and theta constants of order 13  :  \(\Phi _4(a_1(z), \ldots , a_6(z))=0\).

Proof

Let \(y_i(z)=\eta ^3(z) a_i(z)\) \((1 \le i \le 6)\) and \(Y(z):=(y_1(z), \ldots , y_6(z))^{T}\). Then \(Y(z)=\eta ^3(z) \mathbf{A}(z)\). Recall that \(\eta (z)\) satisfies the following transformation formulas \(\eta (z+1)=e^{\frac{\pi i}{12}} \eta (z)\) and \(\eta \left( -\frac{1}{z}\right) =e^{-\frac{\pi i}{4}} \sqrt{z} \eta (z)\). By Proposition 2.4, we have that \(Y(z+1)=e^{-\frac{\pi i}{2}} \rho (t) Y(z)\) and \(Y\left( -\frac{1}{z}\right) =e^{-\frac{\pi i}{2}} z^2 \rho (s) Y(z)\). Define \(j(\gamma , z):=cz+d\) if \(z \in \mathbf{H}\) and \(\gamma =\left( \begin{array}{ll} a &{} b\\ c &{} d \end{array} \right) \in \Gamma (1)\). Hence, \(Y(\gamma (z))=v(\gamma ) j(\gamma , z)^2 \rho (\gamma ) Y(z)\) for \(\gamma \in \Gamma (1)\), where \(v(\gamma )=\pm 1\) or \(\pm i\). Since \(\Gamma (13)=\text{ ker }\) \(\rho \), we have that \(Y(\gamma (z))=v(\gamma ) j(\gamma , z)^2 Y(z)\) for \(\gamma \in \Gamma (13)\). This means that the functions \(y_1(z), \ldots , y_6(z)\) are modular forms of weight 2 for \(\Gamma (13)\) with the same multiplier \(v(\gamma )=\pm 1 \) or \(\pm i\). Thus,

$$\begin{aligned} \Phi _4(Y(\gamma (z)))=v(\gamma )^4 j(\gamma , z)^8 \Phi _4(Y(z)) =j(\gamma , z)^8 \Phi _4(Y(z)) \quad \text{ for } \gamma \in \Gamma (13). \end{aligned}$$

Moreover, for \(\gamma \in \Gamma (1)\),

$$\begin{aligned} \Phi _4(Y(\gamma (z)))= & {} \Phi _4(v(\gamma ) j(\gamma , z)^2 \rho (\gamma ) Y(z))\\= & {} v(\gamma )^4 j(\gamma , z)^8 \Phi _4(\rho (\gamma ) Y(z))=j(\gamma , z)^8 \Phi _4(\rho (\gamma ) Y(z)). \end{aligned}$$

Note that \(\rho (\gamma ) \in \langle \rho (s), \rho (t) \rangle =G\) and \(\Phi _4\) is a G-invariant polynomial, we have that

$$\begin{aligned} \Phi _4(Y(\gamma (z)))=j(\gamma , z)^8 \Phi _4(Y(z)), \quad \text{ for } \gamma \in \Gamma (1). \end{aligned}$$

This implies that \(\Phi _4(Y(z))\) is a modular form of weight 8 for the full modular group \(\Gamma (1)\). Moreover, we will show that it is a cusp form. A straightforward calculation gives that

$$\begin{aligned} \Phi _4(a_1(z), \ldots , a_6(z))=q^{\frac{1}{2}} \sum _{n=0}^{\infty } a_n q^n, \quad \text{ where } a_n \in \mathbf{Z}. \end{aligned}$$

On the other hand, \(\eta (z)^{12}=q^{\frac{1}{2}} \prod _{n=1}^{\infty } (1-q^n)^{12}\). We have that

$$\begin{aligned} \Phi _4(y_1(z), \ldots , y_6(z))=q \sum _{n=0}^{\infty } a_n q^n \prod _{n=1}^{\infty } (1-q^n)^{12} \end{aligned}$$

is a cusp form of weight 8 for the full modular group \(\Gamma =PSL(2, \mathbf{Z})\), but the only such form is zero. This completes the proof of Theorem 3.1. \(\square \)

Corollary 3.2

The following invariant decomposition formula (exotic modular equation) holds :  \(\Psi _2(a_1(z), \ldots , a_6(z))/\mathbf{A}_0(a_1(z), \ldots , a_6(z))^2=0\), where the quadric \(\Psi _2\) is an invariant associated to PSL(2, 13) and \(\mathbf{A}_0^2\) is invariant under the action of the image of a Borel subgroup of PSL(2, 13), i.e., a maximal subgroup of order 78 of PSL(2, 13), which can be viewed as \(\Gamma _0(13)/\Gamma (13)\).

Proof

This comes from Theorem 3.1 by noting that (3.6) and (3.7). Thus, we complete the proof of Theorem 1.3.

Let \(\mathbf{A}_j(a(z)):=\mathbf{A}_j(a_1(z), \ldots , a_6(z))\) for \(i=0, 1, \ldots , 6\). We will show that

$$\begin{aligned} \prod _{j=1}^{6} \mathbf{A}_j(a(z))\ne -\mathbf{A}_0(a(z))^6. \end{aligned}$$
(3.8)

We have that \(\mathbf{A}_0(a(z))=q^{\frac{1}{4}} (1+O(q))\), \(\mathbf{A}_1(a(z))=q^{\frac{34}{104}} (2+O(q))\), \(\mathbf{A}_2(a(z))=q^{\frac{58}{104}} (2+O(q))\), \(\mathbf{A}_3(a(z))=q^{\frac{98}{104}} (1+O(q))\), \(\mathbf{A}_4(a(z))=q^{\frac{50}{104}} (-1+O(q))\), \(\mathbf{A}_5(a(z))=q^{\frac{18}{104}} (-1+O(q))\), and \(\mathbf{A}_6(a(z))=q^{\frac{2}{104}} (-1+O(q))\). Thus,

$$\begin{aligned} \prod _{j=1}^{6} \mathbf{A}_j(a(z))=q^{\frac{5}{2}} (-4+O(q)). \end{aligned}$$

On the other hand, \(-\mathbf{A}_0(a(z))^6=q^{\frac{3}{2}} (-1+O(q))\). This gives the proof of (3.8). \(\square \)

Corollary 3.3

The coordinates \((a_1(z), a_2(z), a_3(z), a_4(z), a_5(z), a_6(z))\) map X(13) into the quartic four-fold \(\Phi _4(z_1, z_2, z_3, z_4, z_5, z_6)=0\) in \(\mathbf{C} \mathbf{P}^5\).

Proof

This comes from Theorems 2.2 and 3.1. \(\square \)

4 Fourteen-dimensional representations of PSL(2, 13) and invariant decomposition formula

We will construct a 14-dimensional representation of PSL(2, 13) which is deduced from our six-dimensional representation. It should be emphasized that our 14-dimensional representation is not a Weil representation. In contrast with this, both our six-dimensional and seven-dimensional representations of G are Weil representations, i.e., \(\frac{p-1}{2}\)-dimensional and \(\frac{p+1}{2}\)-dimensional representations of PSL(2, p), respectively. In fact, what Klein used in his papers [12–15] are all Weil representations. Hence, our method is completely different from Klein’s method.

To construct our 14-dimensional representation which is generated under the action of PSL by a specific vector in \(\text{ Sym }^3\)(six-dimensional representation), we begin with a cubic polynomial \(z_1 z_2 z_3\) and study the action of \(ST^{\nu }\) (\(\nu \) mod 13) on it. We have that

$$\begin{aligned}&-13 \sqrt{13} ST^{\nu }(z_1) \cdot ST^{\nu }(z_2) \cdot ST^{\nu }(z_3)\\&\quad =-\sqrt{\frac{-13-3 \sqrt{13}}{2}} (\zeta ^{8 \nu } z_1^3+\zeta ^{7 \nu } z_2^3+\zeta ^{11 \nu }z_3^3)+\\&\qquad -\,\sqrt{\frac{-13+3 \sqrt{13}}{2}} (\zeta ^{5 \nu } z_4^3+\zeta ^{6 \nu } z_5^3+\zeta ^{2 \nu } z_6^3)+\\&\qquad -\,\sqrt{-13+2 \sqrt{13}} (\zeta ^{12 \nu } z_1^2 z_2+\zeta ^{4 \nu } z_2^2 z_3+\zeta ^{10 \nu } z_3^2 z_1)+\\&\qquad -\,\sqrt{-13-2 \sqrt{13}} (\zeta ^{\nu } z_4^2 z_5+\zeta ^{9 \nu } z_5^2 z_6+\zeta ^{3 \nu } z_6^2 z_4)+\\&\qquad +\,2 \sqrt{-13-2 \sqrt{13}} (\zeta ^{3 \nu } z_1 z_2^2+\zeta ^{\nu } z_2 z_3^2+\zeta ^{9 \nu } z_3 z_1^2)+\\&\qquad -\,2 \sqrt{-13+2 \sqrt{13}} (\zeta ^{10 \nu } z_4 z_5^2+\zeta ^{12 \nu } z_5 z_6^2+\zeta ^{4 \nu } z_6 z_4^2)+\\&\qquad +\,2 \sqrt{\frac{-13-3 \sqrt{13}}{2}} (\zeta ^{7 \nu } z_1^2 z_4+\zeta ^{11 \nu } z_2^2 z_5+\zeta ^{8 \nu } z_3^2 z_6)+\\&\qquad -\,2 \sqrt{\frac{-13+3 \sqrt{13}}{2}} (\zeta ^{6 \nu } z_1 z_4^2+\zeta ^{2 \nu } z_2 z_5^2+\zeta ^{5 \nu } z_3 z_6^2)+\\&\qquad +\,\sqrt{-13-2 \sqrt{13}} (\zeta ^{3 \nu } z_1^2 z_5+\zeta ^{\nu } z_2^2 z_6+\zeta ^{9 \nu } z_3^2 z_4)+\\ \end{aligned}$$
$$\begin{aligned}&\,\,\quad \qquad +\,\sqrt{-13+2 \sqrt{13}} (\zeta ^{10 \nu } z_2 z_4^2+\zeta ^{12 \nu } z_3 z_5^2+\zeta ^{4 \nu } z_1 z_6^2)+\\&\,\,\quad \qquad +\,\sqrt{\frac{-13+3 \sqrt{13}}{2}} (\zeta ^{6 \nu } z_1^2 z_6+\zeta ^{2 \nu } z_2^2 z_4+\zeta ^{5 \nu } z_3^2 z_5)+\\&\,\,\quad \qquad +\,\sqrt{\frac{-13-3 \sqrt{13}}{2}} (\zeta ^{7 \nu } z_3 z_4^2+\zeta ^{11 \nu } z_1 z_5^2+\zeta ^{8 \nu } z_2 z_6^2)+\\&\,\,\quad \qquad +\,[2(\theta _1-\theta _3)-3(\theta _2-\theta _4)] z_1 z_2 z_3+\\&\qquad +\,[2(\theta _4-\theta _2)-3(\theta _1-\theta _3)] z_4 z_5 z_6+\\&\qquad -\,\sqrt{\frac{-13-3 \sqrt{13}}{2}} (\zeta ^{11 \nu } z_1 z_2 z_4+\zeta ^{8 \nu } z_2 z_3 z_5+\zeta ^{7 \nu } z_1 z_3 z_6)+\\&\qquad +\,\sqrt{\frac{-13+3 \sqrt{13}}{2}} (\zeta ^{2 \nu } z_1 z_4 z_5+\zeta ^{5 \nu } z_2 z_5 z_6+\zeta ^{6 \nu } z_3 z_4 z_6)+\\&\qquad -\,3 \sqrt{\frac{-13-3 \sqrt{13}}{2}} (\zeta ^{7 \nu } z_1 z_2 z_5+\zeta ^{11 \nu } z_2 z_3 z_6+\zeta ^{8 \nu } z_1 z_3 z_4)+\\&\qquad +\,3 \sqrt{\frac{-13+3 \sqrt{13}}{2}} (\zeta ^{6 \nu } z_2 z_4 z_5+\zeta ^{2 \nu } z_3 z_5 z_6+\zeta ^{5 \nu } z_1 z_4 z_6)+\\&\qquad -\,\sqrt{-13+2 \sqrt{13}} (\zeta ^{10 \nu } z_1 z_2 z_6+\zeta ^{4 \nu } z_1 z_3 z_5+\zeta ^{12 \nu } z_2 z_3 z_4)+\\&\qquad +\,\sqrt{-13-2 \sqrt{13}} (\zeta ^{3 \nu } z_3 z_4 z_5+\zeta ^{9 \nu } z_2 z_4 z_6+\zeta ^{\nu } z_1 z_5 z_6). \end{aligned}$$

This leads us to define the senary cubic forms (cubic forms in six variables) \(\mathbf{D}_0, \ldots , \mathbf{D}_{12}, \mathbf{D}_{\infty }\) given in (1.11). Let \(r_0=2(\theta _1-\theta _3)-3(\theta _2-\theta _4)\), \(r_{\infty }=2(\theta _4-\theta _2)-3(\theta _1-\theta _3)\), \(r_1=\sqrt{-13-2 \sqrt{13}}\), \(r_2=\sqrt{\frac{-13+3 \sqrt{13}}{2}}\), \(r_3=\sqrt{-13+2 \sqrt{13}}\), and \(r_4=\sqrt{\frac{-13-3 \sqrt{13}}{2}}\). We have that

$$\begin{aligned} -13 \sqrt{13} ST^{\nu }(\mathbf{D}_0)= & {} r_0 \mathbf{D}_0+r_1 \zeta ^{\nu } \mathbf{D}_1+r_2 \zeta ^{2 \nu } \mathbf{D}_2+ r_1 \zeta ^{3 \nu } \mathbf{D}_3+r_3 \zeta ^{4 \nu } \mathbf{D}_4+\\&+\,r_2 \zeta ^{5 \nu } \mathbf{D}_5+r_2 \zeta ^{6 \nu } \mathbf{D}_6+r_4 \zeta ^{7 \nu } \mathbf{D}_7+r_4 \zeta ^{8 \nu } \mathbf{D}_8+r_1 \zeta ^{9 \nu } \mathbf{D}_9+\\&+\,r_3 \zeta ^{10 \nu } \mathbf{D}_{10}+r_4 \zeta ^{11 \nu } \mathbf{D}_{11}+ r_3 \zeta ^{12 \nu } \mathbf{D}_{12}+r_{\infty } \mathbf{D}_{\infty }. \end{aligned}$$
$$\begin{aligned} -13 \sqrt{13} ST^{\nu }(\mathbf{D}_{\infty })= & {} r_{\infty } \mathbf{D}_0-r_3 \zeta ^{\nu } \mathbf{D}_1-r_4 \zeta ^{2 \nu } \mathbf{D}_2 -r_3 \zeta ^{3 \nu } \mathbf{D}_3+r_1 \zeta ^{4 \nu } \mathbf{D}_4+\\&-\,r_4 \zeta ^{5 \nu } \mathbf{D}_5-r_4 \zeta ^{6 \nu } \mathbf{D}_6+r_2 \zeta ^{7 \nu } \mathbf{D}_7+r_2 \zeta ^{8 \nu } \mathbf{D}_8-r_3 \zeta ^{9 \nu } \mathbf{D}_9+\\&+\,r_1 \zeta ^{10 \nu } \mathbf{D}_{10}+r_2 \zeta ^{11 \nu } \mathbf{D}_{11}+r_1 \zeta ^{12 \nu } \mathbf{D}_{12}-r_0 \mathbf{D}_{\infty }. \end{aligned}$$

Let

$$\begin{aligned} \delta _{\infty }(z_1, z_2, z_3, z_4, z_5, z_6)=13^2 (\mathbf{D}_0^2+\mathbf{D}_{\infty }^2) \end{aligned}$$
(4.1)

and

$$\begin{aligned} \delta _{\nu }(z_1, z_2, z_3, z_4, z_5, z_6)=\delta _{\infty }(ST^{\nu }(z_1, z_2, z_3, z_4, z_5, z_6)) \end{aligned}$$
(4.2)

for \(\nu =0, 1, \ldots , 12\). Then

$$\begin{aligned} \delta _{\nu }=13^2 ST^{\nu }(\mathbf{G}_0)=-13 \mathbf{G}_0+\zeta ^{\nu } \mathbf{G}_1 +\zeta ^{2 \nu } \mathbf{G}_2+\cdots +\zeta ^{12 \nu } \mathbf{G}_{12}, \end{aligned}$$
(4.3)

where the senary sextic forms (i.e., sextic forms in six variables) \(\mathbf{G}_0, \ldots , \mathbf{G}_{12}\) are given in (1.10). We have that \(\mathbf{G}_0\) is invariant under the action of \(\langle H, T \rangle \), a maximal subgroup of order 78 of G with index 14. Note that \(\delta _{\infty }\), \(\delta _{\nu }\) for \(\nu =0, \ldots , 12\) form an algebraic equation of degree 14. However, we have that \(\delta _{\infty }+\sum _{\nu =0}^{12} \delta _{\nu }=0\). Hence, it is not the Jacobian equation of degree 14. We call it exotic modular equation of degree 14. We have that

$$\begin{aligned} \delta _{\infty }^2+\sum _{\nu =0}^{12} \delta _{\nu }^2 =26 (7 \cdot 13^2 \mathbf{G}_0^2+\mathbf{G}_1 \mathbf{G}_{12}+\mathbf{G}_ 2 \mathbf{G}_{11}+\cdots +\mathbf{G}_6 \mathbf{G}_7). \end{aligned}$$
(4.4)

This leads us to define \(\Phi _{12}\) as in (1.9).

Theorem 4.1

The invariant decomposition formula for the simple group PSL(2, 13) of order 1092 is given as follows:

$$\begin{aligned} \left[ \frac{\eta ^2(z)}{\eta ^2(13z)}\right] ^5 =\frac{\Phi _{12}(a_1(z), a_2(z), a_3(z), a_4(z), a_5(z), a_6(z))}{(a_1(z) a_2(z) a_3(z) a_4(z) a_5(z) a_6(z))^2}, \end{aligned}$$
(4.5)

where \(\Phi _{12}(z_1, z_2, z_3, z_4, z_5, z_6)\) is an invariant of degree 12 associated to PSL(2, 13), and \((z_1 z_2 z_3 z_4 z_5 z_6)^2\) is invariant under the action of the image of a Borel subgroup of PSL(2, 13), i.e., a maximal subgroup of order 78 of PSL(2, 13), which can be viewed as \(\Gamma _0(13)/\Gamma (13)\).

Proof

Let \(x_i(z)=\eta (z) a_i(z)\) \((1 \le i \le 6)\) and \(X(z)=(x_1(z), \ldots , x_6(z))^{T}\). Then \(X(z)=\eta (z) \mathbf{A}(z)\). Recall that \(\eta (z)\) satisfies the following transformation formulas \(\eta (z+1)=e^{\frac{\pi i}{12}} \eta (z)\) and \(\eta \left( -\frac{1}{z}\right) =e^{-\frac{\pi i}{4}} \sqrt{z} \eta (z)\). By Proposition 2.4, we have that \(X(z+1)=e^{-\frac{2 \pi i}{3}} \rho (t) X(z)\) and \(X\left( -\frac{1}{z}\right) =z \rho (s) X(z)\). Hence, \(X(\gamma (z))=u(\gamma ) j(\gamma , z) \rho (\gamma ) X(z)\) for \(\gamma \in \Gamma (1)\), where \(u(\gamma )=1, \omega \) or \(\omega ^2\) with \(\omega =e^{\frac{2 \pi i}{3}}\). Since \(\Gamma (13)=\text{ ker }\) \(\rho \), we have that \(X(\gamma (z))=u(\gamma ) j(\gamma , z) X(z)\) for \(\gamma \in \Gamma (13)\). This means that the functions \(x_1(z)\), \(\ldots \), \(x_6(z)\) are modular forms of weight 1 for \(\Gamma (13)\) with the same multiplier \(u(\gamma )=1, \omega \) or \(\omega ^2\). Thus,

$$\begin{aligned} \Phi _{12}(X(\gamma (z)))=u(\gamma )^{12} j(\gamma , z)^{12} \Phi _{12}(X(z)) =j(\gamma , z)^{12} \Phi _{12}(X(z)) \end{aligned}$$

for \(\gamma \in \Gamma (13)\). Moreover, for \(\gamma \in \Gamma (1)\),

$$\begin{aligned} \Phi _{12}(X(\gamma (z)))= & {} \Phi _{12}(u(\gamma ) j(\gamma , z) \rho (\gamma ) X(z))\\= & {} u(\gamma )^{12} j(\gamma , z)^{12} \Phi _{12}(\rho (\gamma ) X(z))=j(\gamma , z)^{12} \Phi _{12}(\rho (\gamma ) X(z)). \end{aligned}$$

Note that \(\rho (\gamma ) \in \langle \rho (s), \rho (t) \rangle =G\) and \(\Phi _{12}\) is a G-invariant polynomial, we have that

$$\begin{aligned} \Phi _{12}(X(\gamma (z)))=j(\gamma , z)^{12} \Phi _{12}(X(z)), \quad \gamma \in \Gamma (1). \end{aligned}$$

This implies that \(\Phi _{12}(X(z))\) is a modular form of weight 12 for the full modular group \(\Gamma (1)\). Moreover, we will show that it is a cusp form. We have that

$$\begin{aligned} \left\{ \begin{array}{ll} \mathbf{D}_0 &{}=q^{\frac{15}{8}} (1+O(q)),\\ \mathbf{D}_{\infty } &{}=q^{\frac{7}{8}} (-1+O(q)),\\ \mathbf{D}_1 &{}=q^{\frac{99}{104}} (2+O(q)),\\ \mathbf{D}_2 &{}=q^{\frac{3}{104}} (-1+O(q)),\\ \mathbf{D}_3 &{}=q^{\frac{11}{104}} (1+O(q)),\\ \mathbf{D}_4 &{}=q^{\frac{19}{104}} (-2+O(q)),\\ \mathbf{D}_5 &{}=q^{\frac{27}{104}} (-1+O(q)), \end{array}\right. \quad \quad \left\{ \begin{array}{ll} \mathbf{D}_6 &{}=q^{\frac{35}{104}} (-1+O(q)),\\ \mathbf{D}_7 &{}=q^{\frac{43}{104}} (1+O(q)),\\ \mathbf{D}_8 &{}=q^{\frac{51}{104}} (3+O(q)),\\ \mathbf{D}_9 &{}=q^{\frac{59}{104}} (-2+O(q)),\\ \mathbf{D}_{10} &{}=q^{\frac{67}{104}} (1+O(q)),\\ \mathbf{D}_{11} &{}=q^{\frac{75}{104}} (-4+O(q)),\\ \mathbf{D}_{12} &{}=q^{\frac{83}{104}} (-1+O(q)). \end{array}\right. \end{aligned}$$

Hence,

$$\begin{aligned} \left\{ \begin{array}{ll} \mathbf{G}_0 &{}=q^{\frac{7}{4}} (1+O(q)),\\ \mathbf{G}_1 &{}=q^{\frac{86}{104}} (13+O(q)),\\ \mathbf{G}_2 &{}=q^{\frac{94}{104}} (-22+O(q)),\\ \mathbf{G}_3 &{}=q^{\frac{102}{104}} (-21+O(q)),\\ \mathbf{G}_4 &{}=q^{\frac{6}{104}} (-1+O(q)),\\ \mathbf{G}_5 &{}=q^{\frac{14}{104}} (2+O(q)),\\ \mathbf{G}_6 &{}=q^{\frac{22}{104}} (2+O(q)), \end{array} \right. \quad \quad \left\{ \begin{array}{ll} \mathbf{G}_7 &{}=q^{\frac{30}{104}} (-2+O(q)),\\ \mathbf{G}_8 &{}=q^{\frac{38}{104}} (-8+O(q)),\\ \mathbf{G}_9 &{}=q^{\frac{46}{104}} (6+O(q)),\\ \mathbf{G}_{10} &{}=q^{\frac{54}{104}} (1+O(q)),\\ \mathbf{G}_{11} &{}=q^{\frac{62}{104}} (-8+O(q)),\\ \mathbf{G}_{12} &{}=q^{\frac{70}{104}} (17+O(q)). \end{array} \right. \end{aligned}$$

Therefore,

$$\begin{aligned}&7 \cdot 13^2 \mathbf{G}_0^2+\mathbf{G}_1 \mathbf{G}_{12}+\mathbf{G}_2 \mathbf{G}_{11}+\cdots +\mathbf{G}_6 \mathbf{G}_7\\&\quad =7 \cdot 13^2 q^{\frac{7}{2}} (1+O(q))+q^{\frac{3}{2}} (13 \cdot 17+O(q))+q^{\frac{3}{2}} (22 \cdot 8+O(q))+\\&\qquad +\,q^{\frac{3}{2}} (-21+O(q))+q^{\frac{1}{2}}(-6+O(q)) +q^{\frac{1}{2}}(-16+O(q))+\\&\qquad +\,q^{\frac{1}{2}}(-4+O(q))\\&\quad =q^{\frac{1}{2}}(-26+O(q)). \end{aligned}$$

We have that

$$\begin{aligned} \Phi _{12}(x_1(z), \ldots , x_6(z))=\eta (z)^{12} q^{\frac{1}{2}} (1+O(q))=q(1+O(q)) \end{aligned}$$

is a cusp form of weight 12 for the full modular group \(\Gamma (1)\). Because every \(\Gamma (1)\) cusp form of weight 12 is a multiple of \(\Delta (z)\), checking the \(q^1\) coefficient, we find that \(\Phi _{12}(x_1(z), \ldots , x_6(z))=\Delta (z)\). On the other hand, we have that \(x_1(z) \ldots x_6(z)=-\eta (z)^7 \eta (13z)^5\). This completes the proof of Theorem 4.1. \(\square \)

Note that (2.9), (2.10), and (2.11) are equivalent to the following:

$$\begin{aligned}&a_1(z)^2 a_4(z)^2 a_2(z) a_5(z)+a_2(z)^2 a_5(z)^2 a_3(z) a_6(z)+ \nonumber \\&\quad +\,a_3(z)^2 a_6(z)^2 a_1(z) a_4(z)-a_1(z) a_4(z) a_2(z) a_5(z) a_3(z) a_6(z)\nonumber \\&=-\eta (z)^3 \eta (13z)^3, \end{aligned}$$
(4.6)
$$\begin{aligned}&a_1(z)^2 a_4(z)^2 a_3(z) a_6(z)+a_2(z)^2 a_5(z)^2 a_1(z) a_4(z)+\nonumber \\&\quad +a_3(z)^2 a_6(z)^2 a_2(z) a_5(z)+4 a_1(z) a_4(z) a_2(z) a_5(z) a_3(z) a_6(z)\nonumber \\&=\eta (z)^3 \eta (13z)^3, \end{aligned}$$
(4.7)
$$\begin{aligned}&a_4(z)^2 a_5(z)^2 a_6(z)^2-a_1(z)^2 a_2(z)^2 a_3(z)^2+\nonumber \\&\quad +\,3 a_1(z) a_4(z) a_2(z) a_5(z) a_3(z) a_6(z)=\eta (z)^3 \eta (13z)^3. \end{aligned}$$
(4.8)

The corresponding polynomials are given by

$$\begin{aligned} f_6(z_1, z_2, z_3, z_4, z_5, z_6):=z_1^2 z_4^2 z_2 z_5+z_2^2 z_5^2 z_3 z_6+z_3^2 z_6^2 z_1 z_4-z_1 z_2 z_3 z_4 z_5 z_6, \end{aligned}$$
$$\begin{aligned} g_6(z_1, z_2, z_3, z_4, z_5, z_6):=z_1^2 z_4^2 z_3 z_6+z_2^2 z_5^2 z_1 z_4+z_3^2 z_6^2 z_2 z_5+4 z_1 z_2 z_3 z_4 z_5 z_6, \end{aligned}$$
$$\begin{aligned} h_6(z_1, z_2, z_3, z_4, z_5, z_6):=z_4^2 z_5^2 z_6^2-z_1^2 z_2^2 z_3^2 +3 z_1 z_2 z_3 z_4 z_5 z_6. \end{aligned}$$

Similarly to the above argument in the proof of Theorem 4.1, we have that

$$\begin{aligned} f_6(X(\gamma (z)))= & {} f_6(u(\gamma ) j(\gamma , z) \rho (\gamma ) X(z)) =u(\gamma )^6 j(\gamma , z)^6 f_6(\rho (\gamma ) X(z))\\= & {} j(\gamma , z)^6 f_6(\rho (\gamma ) X(z)) \quad \text{ for } \gamma \in \Gamma (1). \end{aligned}$$

If \(f_6\) is a G-invariant polynomial, we have that

$$\begin{aligned} f_6(X(\gamma (z)))=j(\gamma , z)^6 f_6(X(z)) \quad \text{ for } \gamma \in \Gamma (1).\end{aligned}$$

This implies that \(f_6(X(z))\) is a modular form of weight 6 for the full modular group \(\Gamma (1)\). On the other hand, by (4.6), we have that

$$\begin{aligned} f_6(X(z))=-\eta (z)^9 \eta (13 z)^3=-q^2 \prod _{n=1}^{\infty } (1-q^n)^9 \prod _{n=1}^{\infty } (1-q^{13n})^3. \end{aligned}$$

This shows that \(f_6(X(z))\) is a cusp form of weight 6 for the full modular group \(\Gamma (1)\), but the only such form is zero. This leads to a contradiction! Therefore, \(f_6\) is not a G-invariant polynomial. Similarly, we can prove that \(g_6\) and \(h_6\) are not G-invariant polynomials.