On some determinants involving the tangent function

Abstract

Let p be an odd prime and let $a,b\in {\mathbb {Z}}$ with $p\not \mid ab$. In this paper,we mainly evaluate

$$\begin{aligned} T_p^{(\delta )}(a,b,x):=\det \left[ x+\tan \pi \frac{aj^2+bk^2}{p}\right] _{\delta \leqslant j,k\leqslant (p-1)/2}\ \ (\delta =0,1). \end{aligned}$$

For example, in the case $p\equiv 3\ ({\textrm{mod}}\ 4)$, we show that $T_p^{(1)}(a,b,0)=0$ and

$$\begin{aligned} T_p^{(0)}(a,b,x)={\left\{ \begin{array}{ll} 2^{(p-1)/2}p^{(p+1)/4}&{}\text {if}\ (\frac{ab}{p})=1, \\ p^{(p+1)/4}&{}\text {if}\ (\frac{ab}{p})=-1,\end{array}\right. } \end{aligned}$$

where $(\frac{\cdot }{p})$ is the Legendre symbol. When $(\frac{-ab}{p})=-1$, we also evaluate the determinant $\det [x+\cot \pi \frac{aj^2+bk^2}{p}]_{1\leqslant j,k\leqslant (p-1)/2}.$ In addition, we pose several conjectures one of which states that for any prime $p\equiv 3\ ({\textrm{mod}}\ 4)$, there is an integer $x_p\equiv 1\ ({\textrm{mod}}\ p)$ such that

$$\begin{aligned}\det \left[ \sec 2\pi \frac{(j-k)^2}{p}\right] _{0\leqslant j,k\leqslant p-1}=-p^{(p+3)/2}x_p^2.\end{aligned}$$

1 Introduction

Let p be an odd prime. It is well known that the numbers

$$\begin{aligned} 0^2,\ 1^2,\ \ldots ,\ \left( \frac{p-1}{2}\right) ^2 \end{aligned}$$

are pairwise incongruent modulo p. In [10], the author investigated the determinants

$$\begin{aligned} S(d,p)=\det \left[ \left( \frac{j^2+dk^2}{p}\right) \right] _{1\leqslant j,k\leqslant (p-1)/2} \end{aligned}$$

and

$$\begin{aligned} T(d,p)=\det \left[ \left( \frac{j^2+dk^2}{p}\right) \right] _{0\leqslant j,k\leqslant (p-1)/2}, \end{aligned}$$

where d is an integer not divisible by p, and $(\frac{\cdot }{p})$ is the Legendre symbol. In particular, Sun [10] showed that if $(\frac{d}{p})=1$ then

$$\begin{aligned} \left( \frac{-S(d,p)}{p}\right) =1\ \ \text {and}\ \ T(d,p)=\frac{p-1}{2}S(d,p). \end{aligned}$$

Inspired by the determinants S(d, p) and T(d, p) with $d\in {\mathbb {Z}}$ and $p\not \mid d$, and noting that the tangent function $\tan x$ has period $\pi$, for $a,b\in {\mathbb {Z}}$, we introduce

$$\begin{aligned} T_p^{(0)}(a,b,x):=\det \left[ x+\tan \pi \frac{aj^2+bk^2}{p}\right] _{0\leqslant j,k\leqslant (p-1)/2} \end{aligned}$$

(1.1)

and

$$\begin{aligned} T_p^{(1)}(a,b,x):=\det \left[ x+\tan \pi \frac{aj^2+bk^2}{p}\right] _{1\leqslant j,k\leqslant (p-1)/2}, \end{aligned}$$

(1.2)

and denote $T_p^{(0)}(a,b,0)$ and $T_p^{(1)}(a,b,0)$ by $T_p^{(0)}(a,b)$ and $T_p^{(1)}(a,b)$, respectively. To study the novel determinants $T_p^{(0)}(a,b,x)$ and $T_p^{(1)}(a,b,x)$, we first find their values by numerical experiments via Mathematica, and then seek for detailed proofs via related known results involving roots of unity.

Now we present our main results.

Theorem 1.1

Let $p > 3$ be a prime, and let $a,b\in {\mathbb {Z}}$ with $p\not \mid ab$.

• (i) Assume that $p\equiv 1\ ({\textrm{mod}}\ 4)$. Then

  $$\begin{aligned} T_p^{(0)}(a,b,-x)=-T_p^{(0)}(a,b,x), \end{aligned}$$

  (1.3)

  and in particular $T_p^{(0)}(a,b)=0.$ If $(\frac{ab}{p})=1$ and $b\equiv ac^2\ ({\textrm{mod}}\ p)$ with $c\in {\mathbb {Z}}$, then

  $$\begin{aligned} T_p^{(1)}(a,b,x)=\left( \frac{2c}{p}\right) p^{(p-3)/4}\varepsilon _p^{(\frac{a}{p})(2-(\frac{2}{p}))h(p)}, \end{aligned}$$

  (1.4)

  where $\varepsilon _p$ and h(p) are the fundamental unit and the class number of the real quadratic field ${\mathbb {Q}}(\sqrt{p})$, respectively. When $(\frac{ab}{p})=-1$, we have

  $$\begin{aligned} T_p^{(1)}(a,b,x)=T_p^{(1)}(a,b)=\pm 2^{(p-1)/2}p^{(p-3)/4}. \end{aligned}$$

  (1.5)

• (ii) Suppose that $p\equiv 3\ ({\textrm{mod}}\ 4)$. Then

  $$\begin{aligned} T_p^{(1)}(a,b,-x)=-T_p^{(1)}(a,b,x), \end{aligned}$$

  (1.6)

  and in particular $T_p^{(1)}(a,b)=0$. Also,

  $$\begin{aligned} T_p^{(0)}(a,b,x)={\left\{ \begin{array}{ll} 2^{(p-1)/2}p^{(p+1)/4}&{}\text {if}\ (\frac{ab}{p})=1, \\ p^{(p+1)/4}&{}\text {if}\ (\frac{ab}{p})=-1.\end{array}\right. } \end{aligned}$$

  (1.7)

Remark 1.1

When p is a prime with $p\equiv 1\ ({\textrm{mod}}\ 4)$, and a and b are integers with $(\frac{ab}{p})=-1$, we are unable to determine the sign in (1.5). For any prime $p\equiv 3\ ({\textrm{mod}}\ 4)$ and integers a and b with $p\not \mid ab$, the identity (1.7) looks surprising and interesting. We believe that Theorem 1.1 has certain potential applications.

Theorem 1.2

Let $n>1$ be an odd integer, and let a and b be integers with $\gcd (ab,n)=1$. Then

$$\begin{aligned} \det \left[ x+\tan \pi \frac{aj+bk}{n}\right] _{0\leqslant j,k\leqslant n-1}+\det \left[ -x+\tan \pi \frac{aj+bk}{n}\right] _{0\leqslant j,k\leqslant n-1}=0 \end{aligned}$$

(1.8)

and

$$\begin{aligned} \det \left[ x+\tan \pi \frac{aj+bk}{n}\right] _{1\leqslant j,k\leqslant n-1}=\left( \frac{-ab}{n}\right) n^{n-2}, \end{aligned}$$

(1.9)

where $(\frac{\cdot }{n})$ is the Jacobi symbol.

For the cotangent function, we establish the following two theorems.

Theorem 1.3

Let $p>3$ be a prime, and let $a,b\in {\mathbb {Z}}$ with $(\frac{-ab}{p})=-1$. Then

$$\begin{aligned} \begin{aligned}&\ \det \left[ x+\cot \pi \frac{aj^2+bk^2}{p}\right] _{1\leqslant j,k\leqslant (p-1)/2} \\=&\ {\left\{ \begin{array}{ll} T_p^{(1)}(a,b)/(-p)^{(p-1)/4}=\pm 2^{(p-1)/2}/\sqrt{p}&{}\text {if}\ p\equiv 1\ ({\textrm{mod}}\ 4), \\ (-1)^{(h(-p)+1)/2}(\frac{a}{p})2^{(p-1)/2}/\sqrt{p}&{}\text {if}\ p\equiv 3\ ({\textrm{mod}}\ 4), \end{array}\right. }\end{aligned} \end{aligned}$$

(1.10)

where $h(-p)$ is the class number of the imaginary quadratic field ${\mathbb {Q}}(\sqrt{p}\,\textrm{i})$ with $\textrm{i}=\sqrt{-1}$.

Remark 1.2

It is known that $2\not \mid h(-p)$ for each prime $p\equiv 3\ ({\textrm{mod}}\ 4)$. In 1961, Mordell [8] even proved that for any prime $p>3$ with $p\equiv 3\ (\mathrm{{mod}\ }\ 4)$, we have

$$\begin{aligned} \frac{p-1}{2}!\equiv (-1)^{(h(-p)+1)/2}\ ({\textrm{mod}}\ p). \end{aligned}$$

Theorem 1.4

For any odd prime p, we have

$$\begin{aligned} D_p:=\det \left[ \cot \pi \frac{jk}{p}\right] _{1\leqslant j,k\leqslant (p-1)/2}\in {\left\{ \begin{array}{ll}{\mathbb {Q}}&{}\text {if}\ p\equiv 1\ ({\textrm{mod}}\ 4),\\ \sqrt{p}\ {\mathbb {Q}}&{}\text {if}\ p\equiv 3\ ({\textrm{mod}}\ 4). \end{array}\right. } \end{aligned}$$

We are going to provide several lemmas in the next section and then prove Theorem 1.1 in Sect. 3. Theorems 1.2–1.4 will be shown in Sect. 4. In Sect. 5, we pose some conjectures on determinants involving the trigonometric functions.

2 Some lemmas

Lemma 2.1

Let $A=[a_{jk}]_{0\leqslant j,k\leqslant n}$ be a matrix over a field. Then

$$\begin{aligned} \det [x+a_{jk}]_{0\leqslant j,k\leqslant n}=\det A+x\det B, \end{aligned}$$

(2.1)

where $B=[b_{jk}]_{1\leqslant j,k\leqslant n}$ with $b_{jk}=a_{jk}-a_{j0}-a_{0k}+a_{00}$.

Proof

As $(x+a_{jk})-(x+a_{0k})=a_{jk}-a_{0k}$ for all $0<j\leqslant n$ and $0\leqslant k\leqslant n$, we have

$$\begin{aligned} \det [x+a_{jk}]_{0\leqslant j,k\leqslant n}=&\begin{vmatrix} x+a_{00}&x+a_{01}&x+a_{02}&\ldots&x+a_{0n}\\a_{10}-a_{00}&a_{11}-a_{01}&a_{12}-a_{02}&\ldots&a_{1n}-a_{0n} \\\vdots&\vdots&\vdots&\ddots&\vdots \\a_{n0}-a_{00}&a_{n1}-a_{01}&a_{n2}-a_{02}&\ldots&a_{nn}-a_{0n}\end{vmatrix} \\=&\begin{vmatrix} x&x&x&\ldots&x\\a_{10}-a_{00}&a_{11}-a_{01}&a_{12}-a_{02}&\ldots&a_{1n}-a_{0n} \\\vdots&\vdots&\vdots&\ddots&\vdots \\a_{n0}-a_{00}&a_{n1}-a_{01}&a_{n2}-a_{02}&\ldots&a_{nn}-a_{0n}\end{vmatrix} \\ {}&+\begin{vmatrix} a_{00}&a_{01}&a_{02}&\ldots&a_{0n}\\a_{10}-a_{00}&a_{11}-a_{01}&a_{12}-a_{02}&\ldots&a_{1n}-a_{0n} \\\vdots&\vdots&\vdots&\ddots&\vdots \\a_{n0}-a_{00}&a_{n1}-a_{01}&a_{n2}-a_{02}&\ldots&a_{nn}-a_{0n} \end{vmatrix}, \end{aligned}$$

and hence $\det [x+a_{jk}]_{0\leqslant j,k\leqslant n}-\det A$ coincides with

$$\begin{aligned} \begin{vmatrix} x&0&\ldots&0\\a_{10}-a_{00}&a_{11}-a_{01}-(a_{10}-a_{00})&\ldots&a_{1n}-a_{0n}-(a_{10}-a_{00}) \\\vdots&\vdots&\ddots&\vdots \\a_{n0}-a_{00}&a_{n1}-a_{01}-(a_{n0}-a_{00})&\ldots&a_{nn}-a_{0n}-(a_{n0}-a_{00})\end{vmatrix}=x\det B. \end{aligned}$$

This concludes the proof of (2.1). $\square$

Corollary 2.1

Let m and n be positive integers with $2\not \mid n$. Let $f:{\mathbb {Z}}\rightarrow {\mathbb {R}}$ be an odd function, where ${\mathbb {R}}$ is the field of real numbers. Then, for any integer d, the determinant

$$\begin{aligned} \det \left[ x+f((j+d)^m-(k+d)^m)\right] _{0\leqslant j,k\leqslant n} \end{aligned}$$

does not depend on x.

Proof

Let

$$\begin{aligned} a_{jk}=f((j+d)^m-(k+d)^m)\quad \ \text {for}\ j,k=0,\ldots ,n. \end{aligned}$$

For $1\leqslant j,k\leqslant n$ set $b_{jk}=a_{jk}-a_{j0}-a_{0k}+a_{00}$. As f is an odd function, we have

$$\begin{aligned} b_{jk}&=f((j+d)^m-(k+d)^m)-f((j+d)^m-d^m)-f(d^m-(k+d)^m) \\ {}&=-f((k+d)^m-(j+d)^m)+f((k+d)^m-d^m)+f(d^m-(j+d)^m)=-b_{kj}. \end{aligned}$$

Thus

$$\begin{aligned} \det [b_{jk}]_{1\leqslant j,k\leqslant n}=(-1)^{n}\det [b_{kj}]_{1\leqslant j,k\leqslant n}=-\det [b_{jk}]_{1\leqslant j,k\leqslant n} \end{aligned}$$

and hence $\det [b_{jk}]_{1\leqslant j,k\leqslant n}=0$. Applying Lemma 2.1, we immediately get the desired result. $\square$

The following lemma is Frobenius’ extension (cf. [3] and [9, (8)]) of Cauchy’s determinant identity (cf. [7, (5.5)]).

Lemma 2.2

We have

$$\begin{aligned} \det \bigg [z+\frac{1}{x_j+y_k}\bigg ]_{0\leqslant j,k\leqslant n}=\frac{\prod _{0\leqslant j<k\leqslant n}(x_k-x_j)(y_k-y_j)}{\prod _{j=0}^n\prod _{k=0}^n(x_j+y_k)}\bigg (1+z\sum _{k=0}^n(x_k+y_k)\bigg ). \end{aligned}$$

(2.2)

Proof

We present here an induction proof of (2.2) by using Cauchy’s determinant identity which is the special case $z=0$ of (2.2).

In the case $n=0$, both sides of (2.2) coincide with $z+1/(x_0+y_0)$.

Now, let n be a positive integer, and suppose that

$$\begin{aligned} \det \bigg [z+\frac{1}{x_j+y_k}\bigg ]_{1\leqslant j,k\leqslant n}=\frac{\prod _{1\leqslant j<k\leqslant n}(x_k-x_j)(y_k-y_j)}{\prod _{j=1}^n\prod _{k=1}^n(x_j+y_k)}\bigg (1+z\sum _{k=1}^n(x_k+y_k)\bigg ). \end{aligned}$$

(2.3)

By Lemma 2.1 and (2.2) in the case $z=0$,

$$\begin{aligned} \det \bigg [z+\frac{1}{x_j+y_k}\bigg ]_{0\leqslant j,k\leqslant n}=\frac{\prod _{0\leqslant j<k\leqslant n}(x_k-x_j)(y_k-y_j)}{\prod _{j=0}^n\prod _{k=0}^n(x_j+y_k)}+z\det [b_{jk}]_{1\leqslant j,k\leqslant n}, \end{aligned}$$

(2.4)

where

$$\begin{aligned} b_{jk}= & {} \frac{1}{x_j+y_k}-\frac{1}{x_j+y_0}-\frac{1}{x_0+y_k}+\frac{1}{x_0+y_0}\\ {}= & {} \frac{(x_j-x_0)(y_k-y_0)(x_j+y_k+x_0+y_0)}{(x_0+y_0)(x_j+y_0)(x_0+y_k)(x_j+y_k)}. \end{aligned}$$

With the aid of (2.3), we have

$$\begin{aligned}\det [b_{jk}]_{1\leqslant j,k\leqslant n}&=\prod _{j=1}^n\frac{x_j-x_0}{x_j+y_0} \times \prod _{k=1}^n\frac{y_k-y_0}{y_k+x_0}\times \det \bigg [\frac{1}{x_0+y_0}+\frac{1}{x_j+y_k}\bigg ]_{1\leqslant j,k\leqslant n} \\&=\prod _{k=1}^n\frac{(x_k-x_0)(y_k-y_0)}{(x_k+y_0)(y_k+x_0)}\\&\quad \times \frac{\prod _{1\leqslant j<k\leqslant n}(x_k-x_j)(y_k-y_j)}{\prod _{j=1}^n\prod _{k=1}^n(x_j+y_k)}\left( 1+\frac{\sum _{k=1}^n(x_k+y_k)}{x_0+y_0}\right) \\&=\frac{\prod _{0\leqslant j<k\leqslant n}(x_k-x_j)(y_k-y_j)}{\prod _{j=0}^n\prod _{k=0}^n(x_j+y_k)}\sum _{k=0}^n(x_k+y_k). \end{aligned}$$

Combining this with (2.4), we obtain the desired (2.2). This concludes the proof. $\square$

An analogue of Lemma 2.2 for Pfaffians can be found in Okada’s paper [9].

Lemma 2.3

[Huang and Pan [4]] Let $n>1$ be an odd integer, and let c be any integer relatively prime to n. For each $j=1,\ldots ,(n-1)/2$, let $\rho _c(j)$ be the unique $r\in \{1,\ldots ,(n-1)/2\}$ with cj congruent to r or $-r$ modulo n. For the permutation $\rho _c$ on $\{1,\ldots ,(n-1)/2\}$, its sign is given by

$$\begin{aligned} \textrm{sign}(\rho _c)=\left( \frac{c}{n}\right) ^{(n+1)/2}. \end{aligned}$$

(2.5)

Lemma 2.4

[Sun [11]] Let $p > 3$ be a prime. Let $\zeta =e^{2\pi \textrm{i}/p}$ and $a\in {\mathbb {Z}}$ with $p\not \mid a$.

• (i) If $p\equiv 1\ ({\textrm{mod}}\ 4)$, then

  $$\begin{aligned} \prod _{1\leqslant j<k\leqslant (p-1)/2}(\zeta ^{aj^2}+\zeta ^{ak^2}) =\pm \varepsilon _p^{(\frac{a}{p})h(p)((\frac{2}{p})-1)/2} \end{aligned}$$

  (2.6)

  and

  $$\begin{aligned} \prod _{1\leqslant j<k\leqslant (p-1)/2}(\zeta ^{aj^2}-\zeta ^{ak^2})^2 =(-1)^{(p-1)/4}p^{(p-3)/4}\varepsilon _p^{(\frac{a}{p})h(p)}. \end{aligned}$$

  (2.7)

• (ii) Suppose that $p\equiv 3\ ({\textrm{mod}}\ 4)$. Then

  $$\begin{aligned} \prod _{1\leqslant j<k\leqslant (p-1)/2}(\zeta ^{aj^2}+\zeta ^{ak^2})=1, \end{aligned}$$

  (2.8)

  and

  $$\begin{aligned}{} & {} \prod _{1\leqslant j<k\leqslant (p-1)/2}(\zeta ^{aj^2}-\zeta ^{ak^2})\nonumber \\ {}{} & {} \quad = {\left\{ \begin{array}{ll}(-p)^{(p-3)/8}&{}\text {if}\ p\equiv 3\ ({\textrm{mod}}\ 8), \\ (-1)^{(p+1)/8+(h(-p)-1)/2}(\frac{a}{p})p^{(p-3)/8}\textrm{i}&{}\text {if}\ p\equiv 7\ ({\textrm{mod}}\ 8). \end{array}\right. } \end{aligned}$$

  (2.9)

  Also,

  $$\begin{aligned} \prod _{k=1}^{(p-1)/2}(1-\zeta ^{ak^2})=(-1)^{(h(-p)+1)/2}\left( \frac{a}{p}\right) \sqrt{p}\,\textrm{i}. \end{aligned}$$

  (2.10)

Lemma 2.5

Let $p > 3$ be a prime, and let $a,b\in {\mathbb {Z}}$ with $(\frac{-ab}{p})=-1$. Then

$$\begin{aligned}{} & {} \prod _{j=1}^{(p-1)/2}\prod _{k=1}^{(p-1)/2}\left( 1-e^{2\pi \textrm{i}(aj^2+bk^2)/p}\right) \nonumber \\{} & {} \quad = p^{(p-1)/4}\times {\left\{ \begin{array}{ll}1&{}\text {if}\ p\equiv 1\ ({\textrm{mod}}\ 4), \\ (-1)^{(h(-p)-1)/2}(\frac{a}{p})\textrm{i}&{}\text {if}\ p\equiv 3\ ({\textrm{mod}}\ 4). \end{array}\right. }\end{aligned}$$

(2.11)

Proof

For $m\in {\mathbb {Z}}$ set

$$\begin{aligned} r(m):&=\left| \left\{ (j,k):\ 1\leqslant j,k\leqslant \frac{p-1}{2}\ \text {and}\ aj^2+bk^2\equiv m\ ({\textrm{mod}}\ p)\right\} \right| \\ {}&=\left| \left\{ 1\leqslant x\leqslant p-1:\ \left( \frac{x}{p}\right) =1\ \text {and}\ \left( \frac{m-ax}{p}\right) =\left( \frac{b}{p}\right) \right\} \right| . \end{aligned}$$

Note that $r(0)=0$ since $(\frac{-ab}{p})\not =1$.

Let $m\in \{1,\ldots ,p-1\}$. Then

$$\begin{aligned} r(m)=\ {}&\sum _{\begin{array}{c} 0<x<p\\ p\not \mid ax-m \end{array}}\frac{(\frac{x}{p})+1}{2}\cdot \frac{(\frac{b(m-ax)}{p})+1}{2} \\ =\ {}&\frac{1}{4}\sum _{x=1}^{p-1}\left( \left( \frac{bx(m-ax)}{p}\right) +\left( \frac{x}{p}\right) +\left( \frac{b(m-ax)}{p}\right) +1\right) -\frac{(\frac{am}{p})+1}{4} \\ =\ {}&\frac{1}{4}\sum _{x=0}^{p-1}\left( \frac{-abx^2+bmx}{p}\right) +\frac{1}{4}\sum _{x=0}^{p-1}\left( \frac{x}{p}\right) +\frac{1}{4}\sum _{x=0}^{p-1}\left( \frac{-abx+bm}{p}\right) \\&- \frac{1}{4}\left( \frac{bm}{p}\right) +\frac{p-1}{4}-\frac{(\frac{am}{p})+1}{4}. \end{aligned}$$

It is well known that for any $a_0,a_1,a_2\in {\mathbb {Z}}$ with $p\not \mid a_0$ or $p\not \mid a_1$,we have

$$\begin{aligned} \sum _{x=0}^{p-1}\left( \frac{a_0x^2+a_1x+a_2}{p}\right) ={\left\{ \begin{array}{ll}-(\frac{a_0}{p})&{}\text {if}\ p\not \mid a_1^2-4a_0a_2, \\ (p-1)(\frac{a_0}{p})&{}\text {if}\ p\mid a_1^2-4a_0a_2.\end{array}\right. } \end{aligned}$$

(2.12)

(See, e.g., [1, p. 58].) Therefore

$$\begin{aligned} r(m)=-\frac{1}{4}\left( \frac{-ab}{p}\right) +\frac{p-1}{4}-\frac{(\frac{am}{p})+(\frac{bm}{p})+1}{4}=\frac{p-1}{4}-\frac{1-(\frac{-1}{p})}{4}\left( \frac{am}{p}\right) . \end{aligned}$$

In view of the above,

$$\begin{aligned}&\ \prod _{j=1}^{(p-1)/2}\prod _{k=1}^{(p-1)/2}\left( 1-e^{2\pi \textrm{i}(aj^2+bk^2)/p}\right) \\=&\ \prod _{m=1}^{p-1}(1-e^{2\pi \textrm{i}m/p})^{r(m)} =\frac{\prod _{m=1}^{p-1}(1-e^{2\pi \textrm{i}m/p})^{(p-1+(\frac{a}{p})(1-(\frac{-1}{p})))/4}}{\prod _{\begin{array}{c} 0<m<p\\ (\frac{m}{p})=1 \end{array}}(1-e^{2\pi \textrm{i}m/p})^{(\frac{a}{p})(1-(\frac{-1}{p}))/2}}. \end{aligned}$$

Clearly,

$$\begin{aligned} \prod _{m=1}^{p-1}(1-e^{2\pi \textrm{i}m/p})=\lim _{x\rightarrow 1}\frac{x^p-1}{x-1}=p. \end{aligned}$$

As (2.10) holds for $p\equiv 3\ ({\textrm{mod}}\ 4)$, we have

$$\begin{aligned} \prod _{\begin{array}{c} 0<m<p\\ (\frac{m}{p})=1 \end{array}}(1-e^{2\pi \textrm{i}m/p})^{(1-(\frac{-1}{p}))/2} ={\left\{ \begin{array}{ll}1&{}\text {if}\ p\equiv 1\ ({\textrm{mod}}\ 4),\\ (-1)^{(h(-p)+1)/2}\sqrt{p}\,\textrm{i}&{}\text {if}\ p\equiv 3\ ({\textrm{mod}}\ 4).\end{array}\right. } \end{aligned}$$

Thus the desired (2.11) follows. $\square$

3 Proof of Theorem 1.1

For convenience, we set $n=(p-1)/2$ and $\zeta =e^{2\pi \textrm{i}/p}$. Since $p > 3$ and

$$\begin{aligned} \sum _{k=0}^nk^2=\frac{n(n+1)(2n+1)}{6}=\frac{p^2-1}{24}p\equiv 0\ ({\textrm{mod}}\ p), \end{aligned}$$

we have

$$\begin{aligned} \prod _{k=0}^n\zeta ^{k^2}=1. \end{aligned}$$

(3.1)

As

$$\begin{aligned} \tan x=\frac{2\sin x}{2\cos x}=\frac{(e^{\textrm{i}x}-e^{-\textrm{i}x})/\textrm{i}}{e^{\textrm{i}x}+e^{-\textrm{i}x}} =\frac{-\textrm{i}(e^{2\textrm{i}x}-1)}{e^{2\textrm{i}x}+1}=-\textrm{i}+\frac{2\textrm{i}}{e^{2\textrm{i}x}+1}, \end{aligned}$$

we also have

$$\begin{aligned} \textrm{i}+\tan \pi \frac{ aj^2+bk^2}{p}=\frac{2\textrm{i}}{\zeta ^{aj^2+bk^2}+1}\quad \text {for all}\ j,k=0,\ldots ,n. \end{aligned}$$

(3.2)

For each $\delta \in \{0,1\}$ and integer $d\not \equiv 0\ ({\textrm{mod}}\ p)$, we claim that

$$\begin{aligned} T_p^{(\delta )}(a,\pm ad^2,x)=\left( \frac{d}{p}\right) ^{n+1}T_p^{(\delta )}(a,\pm a,x). \end{aligned}$$

(3.3)

We now explain this. For $k=1,\ldots ,n$ let $\rho _d(k)$ be the unique $r\in \{1,\ldots ,n\}$ with dk congruent to r or $-r$ modulo p. In view of Lemma 2.3,

$$\begin{aligned} T_p^{(1)}(a,\pm ad^2,x)&=\sum _{\tau \in S_n}\textrm{sign}(\tau )\prod _{j=1}^n\left( x+\tan \pi \frac{aj^2\pm a(d\tau (j))^2}{p}\right) \\ {}&=\textrm{sign}(\rho _d)\sum _{\tau \in S_n}\textrm{sign}(\rho _d\tau )\prod _{j=1}^n\left( x+\tan \pi \frac{aj^2\pm a\rho _d(\tau (j))^2}{p}\right) \\ {}&=\left( \frac{d}{p}\right) ^{n+1}T_p^{(1)}(a,\pm a,x). \end{aligned}$$

If we extend the function $\rho _d$ by defining $\rho _d(0)=0$, then the new $\rho _d$ is a permutation of $\{0,1,\ldots ,n\}$ and its sign is the same as the old one. So, (3.3) also holds for $\delta =0$.

Proof of the First Part of Theorem 1.1

As $p\equiv 1\ ({\textrm{mod}}\ 4)$, we have $n=(p-1)/2\equiv 0\ ({\textrm{mod}}\ 2)$. For $q=n!$, we have $q^2\equiv -1\ ({\textrm{mod}}\ p)$ by Wilson’s theorem, hence

$$\begin{aligned}&-T_p^{(0)}(a,b,x)=\det \left[ -x-\tan \pi \frac{aj^2+bk^2}{p}\right] _{0\leqslant j,k\leqslant n} \\&\quad =\det \left[ -x+\tan \pi \frac{a(qj)^2+b(qk)^2}{p}\right] _{0\leqslant j,k\leqslant n}\\&\quad =T_p^{(0)}(a,b,-x) \end{aligned}$$

and thus $\det T_p^{(0)}(a,b)=0$.

Case 1. $(\frac{ab}{p})=1$.

In this case, $b\equiv ac^2\ ({\textrm{mod}}\ p)$ for some integer $c\not \equiv 0\ ({\textrm{mod}}\ p)$. Note that $b\equiv -a(qc)^2\ ({\textrm{mod}}\ p)$ and hence

$$\begin{aligned} T_p^{(1)}(a,b,x)=\left( \frac{2c}{p}\right) T_p^{(1)}(a,-a,x) \end{aligned}$$

by (3.3) and the equality $(\frac{q}{p})=(\frac{2}{p})$ (cf. [10, Lemma 2.3]).

By Corollary 2.1,

$$\begin{aligned}&\det \left[ x+\tan \pi \frac{aj^2-ak^2}{p}\right] _{1\leqslant j,k\leqslant n} =\det \left[ x+\tan \pi \frac{a(j+1)^2-a(k+1)^2}{p}\right] _{0\leqslant j,k\leqslant n-1} \end{aligned}$$

does not depend on x. So, with the aid of (3.2), we get

$$\begin{aligned} T_p^{(1)}(a,-a,x)&=\det \left[ \textrm{i}+\tan \pi \frac{aj^2-ak^2}{p}\right] _{1\leqslant j,k\leqslant n} \\ {}&=\det \left[ \frac{2\textrm{i}}{e^{2\pi \textrm{i}a(j^2-k^2)/p}+1}\right] _{1\leqslant j,k\leqslant n} \\ {}&=\prod _{k=1}^n(2\textrm{i}\zeta ^{ak^2})\times \det \left[ \frac{1}{\zeta ^{aj^2}+\zeta ^{ak^2}}\right] _{1\leqslant j,k\leqslant n}. \end{aligned}$$

(Recall that $\zeta =e^{2\pi \textrm{i}/p}$.) In light of Lemma 2.2,

$$\begin{aligned} \det \left[ \frac{1}{\zeta ^{aj^2}+\zeta ^{ak^2}}\right] _{1\leqslant j,k\leqslant n}=\frac{\prod _{1\leqslant j<k\leqslant n}(\zeta ^{ak^2}-\zeta ^{aj^2})^2}{\prod _{k=1}^n(\zeta ^{ak^2}+\zeta ^{ak^2})\times \prod _{1\leqslant j<k\leqslant n}(\zeta ^{aj^2}+\zeta ^{ak^2})^2}. \end{aligned}$$

Therefore,

$$\begin{aligned} T_p^{(1)}(a,-a,x)&=\textrm{i}^{n}\prod _{1\leqslant j<k\leqslant n}\left( \frac{\zeta ^{ak^2}-\zeta ^{aj^2}}{\zeta ^{ak^2}+\zeta ^{aj^2}}\right) ^2 \\ {}&=(-1)^{(p-1)/4}\frac{\prod _{1\leqslant j<k\leqslant n}(\zeta ^{ak^2}-\zeta ^{aj^2})^2}{\prod _{1\leqslant j<k\leqslant n}(\zeta ^{aj^2}+\zeta ^{ak^2})^2} =p^{(p-3)/4}\varepsilon _p^{(\frac{a}{p})(2-(\frac{2}{p}))h(p)} \end{aligned}$$

with the aid of Lemma 2.4(i).

Case 2. $(\frac{ab}{p})=-1$.

Recall that $T_p^{(1)}(a,b)=\det [c_{jk}]_{1\leqslant j,k\leqslant n}$ with $c_{jk}=\tan \pi (aj^2+bk^2)/p$. By Lemma 2.1,

$$\begin{aligned} T_p^{(1)}(a,b,x)=\det [x+c_{jk}]_{1\leqslant j,k\leqslant n}=T_p^{(1)}(a,b)+x\det [d_{jk}]_{1<j,k\leqslant n}, \end{aligned}$$

(3.4)

where $d_{jk}=c_{jk}-c_{j1}-c_{1k}+c_{11}$. In light of (3.2) and (3.4),

$$\begin{aligned} \det \left[ \frac{2\textrm{i}}{\zeta ^{aj^2+bk^2}+1}\right] _{1\leqslant j,k\leqslant n}=\det [\textrm{i}+c_{jk}]_{1\leqslant j,k\leqslant n} =T_p^{(1)}(a,b)+\textrm{i}\det [d_{jk}]_{1<j,k\leqslant n}, \end{aligned}$$

and hence (1.5) is implied by

$$\begin{aligned} D_p(a,b):=\det \left[ \frac{2\textrm{i}}{\zeta ^{aj^2+bk^2}+1}\right] _{1\leqslant j,k\leqslant n}=\pm 2^{(p-1)/2}p^{(p-3)/4}. \end{aligned}$$

(3.5)

(Note that both $T_p^{(1)}(a,b)$ and $\det [d_{jk}]_{1<j,k\leqslant n}$ are real numbers.)

In view of Lemma 2.2 and (3.1),

$$\begin{aligned} D_p(a,b)&=\prod _{k=1}^n\left( \frac{2\textrm{i}}{\zeta ^{bk^2}}\right) \times \det \left[ \frac{1}{\zeta ^{aj^2}+\zeta ^{-bk^2}}\right] _{1\leqslant j,k\leqslant n} \\&=\frac{(2\textrm{i})^{n}}{\prod _{k=1}^n\zeta ^{bk^2}}\times \frac{\prod _{1\leqslant j<k\leqslant n}(\zeta ^{ak^2}-\zeta ^{aj^2})(\zeta ^{-bk^2}-\zeta ^{-bj^2})}{\prod _{j=1}^n\prod _{k=1}^n(\zeta ^{aj^2}+\zeta ^{-bk^2})} \\&=(-1)^{(p-1)/4}2^{(p-1)/2}\frac{\prod _{1\leqslant j<k\leqslant n}(\zeta ^ {ak^2}-\zeta ^{aj^2})(\zeta ^{-bk^2}-\zeta ^{-bj^2})}{\prod _{j=1}^n\prod _{k=1}^n(\zeta ^{aj^2+bk^2}+1)}. \end{aligned}$$

Note that

$$\begin{aligned} \prod _{j=1}^n\prod _{k=1}^n(\zeta ^{aj^2+bk^2}+1) =\prod _{j=1}^n\prod _{k=1}^n\frac{1-\zeta ^{2aj^2+2bk^2}}{1-\zeta ^{aj^2+bk^2}}=1 \end{aligned}$$

by Lemma 2.5. So

$$\begin{aligned} D_p(a,b)=(-1)^{(p-1)/4}2^{(p-1)/2}\prod _{1\leqslant j<k\leqslant n}(\zeta ^{ak^2}-\zeta ^{aj^2})(\zeta ^{-bk^2}-\zeta ^{-bj^2}). \end{aligned}$$

(3.6)

Observe that

$$\begin{aligned} \prod _{1\leqslant j<k\leqslant n}(\zeta ^{ak^2}-\zeta ^{aj^2})^2(\zeta ^{-bk^2}-\zeta ^{-bj^2})^2=p^{(p-3)/2}\varepsilon _p^{((\frac{a}{p})+(\frac{-b}{p}))h(p)}=p^{(p-3)/2} \end{aligned}$$

by (2.7). Therefore (3.5) holds and hence so does (1.5).

In view of the above, we have completed the proof of part (i) of Theorem 1.1. $\square$

Proof of the Second Part of Theorem 1.1

As $p\equiv 3\ ({\textrm{mod}}\ 4)$, we have $n=(p-1)/2\equiv 1\ ({\textrm{mod}}\ 2).$

Case 1. $(\frac{ab}{p})=-1$.

In this case, $b\equiv -ad^2\ ({\textrm{mod}}\ p)$ for some integer $d\not \equiv 0\ ({\textrm{mod}}\ p)$, and hence by (3.3), we have

$$\begin{aligned} T_p^{(0)}(a,b,x)=T_p^{(0)}(a,-a,x)\ \ \text {and}\ \ T_p^{(1)}(a,b,x)=T_p^{(1)}(a,-a,x). \end{aligned}$$

As

$$\begin{aligned} T_p^{(1)}(a,-a,-x)= & {} \det \left[ -x+\tan \pi \frac{ak^2-aj^2}{p}\right] _{1\leqslant j,k\leqslant n}\\= & {} (-1)^nT_p^{(1)}(a,-a,x)=-T_p^{(1)}(a,-a,x), \end{aligned}$$

we get $T_p^{(1)}(a,b,-x)=-T_p^{(1)}(a,b,x)$, and in particular $T_p^{(1)}(a,b)=0$.

To obtain the equality $T_p^{(0)}(a,b,x)=p^{(p+1)/4}$, we now determine $T_p^{(0)}(a,-a,x)$ which equals $T_p^{(0)}(a,b,x)$. In view of Corollary 2.1 and (3.2), we have

$$\begin{aligned} T_p^{(0)}(a,-a,x)&=\det \left[ \textrm{i}+\tan \pi \frac{aj^2-ak^2}{p}\right] _{0\leqslant j,k\leqslant n} \\ {}&=\det \left[ \frac{2\textrm{i}}{\zeta ^{a(j^2-k^2)}+1}\right] _{0\leqslant j,k\leqslant n} \\ {}&=\prod _{k=0}^n(2\textrm{i}\zeta ^{ak^2})\times \det \left[ \frac{1}{\zeta ^{aj^2}+\zeta ^{ak^2}}\right] _{0\leqslant j,k\leqslant n}. \end{aligned}$$

By Lemma 2.2,

$$\begin{aligned} \det \left[ \frac{1}{\zeta ^{aj^2}+\zeta ^{ak^2}}\right] _{0\leqslant j,k\leqslant n}=\frac{\prod _{0\leqslant j<k\leqslant n}(\zeta ^{ak^2}-\zeta ^{aj^2})^2}{\prod _{k=0}^n(\zeta ^{ak^2}+\zeta ^{ak^2})\times \prod _{0\leqslant j<k\leqslant n}(\zeta ^{aj^2}+\zeta ^{ak^2})^2}. \end{aligned}$$

Therefore,

$$\begin{aligned} T_p^{(0)}(a,-a,x)&=\textrm{i}^{n+1}\prod _{0\leqslant j<k\leqslant n}\left( \frac{\zeta ^{ak^2}-\zeta ^{aj^2}}{\zeta ^{ak^2}+\zeta ^{aj^2}}\right) ^2 \\ {}&=(-1)^{(p+1)/4}\prod _{k=1}^n\left( \frac{\zeta ^{ak^2}-1}{\zeta ^{ak^2}+1}\right) ^2\times \frac{\prod _{1\leqslant j<k\leqslant n}(\zeta ^{ak^2}-\zeta ^{aj^2})^2}{\prod _{1\leqslant j<k\leqslant n}(\zeta ^{aj^2}+\zeta ^{ak^2})^2}. \end{aligned}$$

By Lemma 2.4(ii),

$$\begin{aligned} \prod _{k=1}^n(\zeta ^{ak^2}-1)^2=-p\ \ \text {and}\ \ \prod _{k=1}^n(\zeta ^{ak^2}+1)^2=\prod _{k=1}^n\frac{(\zeta ^{2ak^2}-1)^2}{(\zeta ^{ak^2}-1)^2}=\frac{-p}{-p}=1, \end{aligned}$$

and

$$\begin{aligned} \prod _{1\leqslant j<k\leqslant n}(\zeta ^{ak^2}-\zeta ^{aj^2})^2=(-p)^{(p-3)/4}\ \text {and}\ \prod _{1\leqslant j<k\leqslant n}(\zeta ^{ak^2}+\zeta ^{aj^2})^2=1. \end{aligned}$$

Therefore

$$\begin{aligned} T_p^{(0)}(a,-a,x)=(-1)^{(p+1)/4}(-p)(-p)^{(p-3)/4}=p^{(p+1)/4} \end{aligned}$$

as desired.

Case 2. $(\frac{ab}{p})=1$.

In this case, $b\equiv ac^2\ ({\textrm{mod}}\ p)$ for some $c\in {\mathbb {Z}}$ with $p\not \mid c$, and hence by (3.3), we have $T_p^{(0)}(a,b,x)=T_p^{(0)}(a,a,x)$ and $T_p^{(1)}(a,b,x)=T_p^{(1)}(a,a,x)$ since $n+1$ is even.

Clearly $T_p^{(0)}(a,a)=\det [a_{jk}]_{0\leqslant j,k\leqslant n}$ with $a_{jk}=\tan \pi (aj^2+ak^2)/p$. By Lemma 2.1,

$$\begin{aligned} T_p^{(0)}(a,a,x)=\det [x+a_{jk}]_{0\leqslant j,k\leqslant n}=T_p^{(0)}(a,a)+x\det [b_{jk}]_{1\leqslant j,k\leqslant n} \end{aligned}$$

(3.7)

where

$$\begin{aligned} b_{jk}:=a_{jk}-a_{j0}-a_{0k}+a_{00}=\tan \pi \frac{aj^2+ak^2}{p} -\tan \pi \frac{aj^2}{p}-\tan \pi \frac{ak^2}{p}. \end{aligned}$$

Using the well known identity

$$\begin{aligned} \tan (x_1+x_2)=\frac{\tan x_1+\tan x_2}{1-\tan x_1\tan x_2}, \end{aligned}$$

we obtain

$$\begin{aligned} b_{jk}=\tan \pi \frac{aj^2}{p}\times \tan \pi \frac{ak^2}{p}\times \tan \pi \frac{aj^2+ak^2}{p} \end{aligned}$$

and hence

$$\begin{aligned} \det [b_{jk}]_{1\leqslant j,k\leqslant n}=T_p^{(1)}(a,a)\prod _{j=1}^n\tan ^2\pi \frac{aj^2}{p}. \end{aligned}$$

(3.8)

In view of (3.2), (3.7) and (3.8),

$$\begin{aligned} \det \left[ \frac{2\textrm{i}}{\zeta ^{a(j^2+k^2)}+1}\right] _{0\leqslant j,k\leqslant n}&=\det [\textrm{i}+a_{jk}]_{0\leqslant j,k\leqslant n}\\ {}&=T_p^{(0)}(a,a)+\textrm{i}T_p^{(1)}(a,a)\prod _{j=1}^n\tan ^2\pi \frac{aj^2}{p}. \end{aligned}$$

Thus

$$\begin{aligned} T_p^{(0)}(a,a)=2^{(p-1)/2}p^{(p+1)/4},\ T_p^{(1)}(a,a)=0\ \text {and}\ \det [b_{jk}]_{1\leqslant j,k\leqslant n}=0 \end{aligned}$$

(3.9)

if and only if

$$\begin{aligned} \det \left[ \frac{2\textrm{i}}{\zeta ^{a(j^2+k^2)}+1}\right] _{0\leqslant j,k\leqslant n}=2^{(p-1)/2}p^{(p+1)/4}. \end{aligned}$$

(3.10)

With the aid of Lemma 2.2,

$$\begin{aligned} \det \left[ \frac{2\textrm{i}}{\zeta ^{a(j^2+k^2)}+1}\right] _{0\leqslant j,k\leqslant n}&=\prod _{k=0}^n\frac{2\textrm{i}}{\zeta ^{ak^2}}\times \det \left[ \frac{1}{\zeta ^{aj^2}+\zeta ^{-ak^2}}\right] _{0\leqslant j,k\leqslant n} \\ {}&=\frac{(2\textrm{i})^{n+1}}{\prod _{k=0}^n\zeta ^{ak^2}}\times \frac{\prod _{0\leqslant j<k\leqslant n}(\zeta ^{ak^2}-\zeta ^{aj^2})(\zeta ^{-ak^2}-\zeta ^{-aj^2})}{\prod _{j=0}^n\prod _{k=0}^n(\zeta ^{aj^2}+\zeta ^{-ak^2})}. \end{aligned}$$

This, together with (3.1), yields

$$\begin{aligned} \det \left[ \frac{2\textrm{i}}{\zeta ^{a(j^2+k^2)}+1}\right] _{0\leqslant j,k\leqslant n} (-1)^{(p+1)/4}2^{(p+1)/2} \frac{\prod _{0\leqslant j<k\leqslant n}(\zeta ^{ak^2}-\zeta ^{aj^2})(\zeta ^{-ak^2}-\zeta ^{-aj^2})}{\prod _{j=0}^n\prod _{k=0}^n(\zeta ^{a(j^2+k^2)}+1)}. \end{aligned}$$

(3.11)

By Lemma 2.4(ii),

$$\begin{aligned}&\ \prod _{0\leqslant j<k\leqslant n}(\zeta ^{ak^2}-\zeta ^{aj^2})(\zeta ^{-ak^2}-\zeta ^{-aj^2}) \\=&\ \prod _{k=1}^n(\zeta ^{ak^2}-1)(\zeta ^{-ak^2}-1)\times \prod _{1\leqslant j<k\leqslant n}(\zeta ^{aj^2}-\zeta ^{ak^2})(\zeta ^{-aj^2}-\zeta ^{-ak^2}) \\=&\ p\times p^{(p-3)/4}=p^{(p+1)/4}. \end{aligned}$$

In view of Lemmas 2.4(ii) and 2.5,

$$\begin{aligned} \prod _{j=0}^n\prod _{k=0}^n(\zeta ^{a(j^2+k^2)}+1)&=(\zeta ^0+1)\prod _{j=1}^n\left( \frac{1-\zeta ^{2aj^2}}{1-\zeta ^{aj^2}}\right) ^2 \times \prod _{j=1}^n\prod _{k=1}^n\frac{1-\zeta ^{2a(j^2+k^2)}}{1-\zeta ^{a(j^2+k^2)}} \\ {}&=2\left( \frac{2}{p}\right) ^2\left( \frac{2}{p}\right) =2(-1)^{(p+1)/4}. \end{aligned}$$

Combining these with (3.11), we get (3.10) and hence (3.9) holds. In view of (3.7) and (3.9), we finally obtain that

$$\begin{aligned} T_p^{(0)}(a,b,x)=T_p^{(0)}(a,a,x)=2^{(p-1)/2}p^{(p+1)/4}. \end{aligned}$$

By the above, we have finished the proof of part (ii) of Theorem 1.1. $\square$

4 Proofs of Theorems 1.2–1.4

The following lemma is Frobenius’ extension (cf. [2]) of the Zolotarev lemma [14].

Lemma 4.1

Let n be a positive odd integer, and let $a\in {\mathbb {Z}}$ be relatively prime to n. For $j=0,\ldots ,n-1$, let $\lambda _a(j)$ be the least nonnegative residue of aj modulo n. Then the permutation $\lambda _a$ of $\{0,\ldots ,n-1\}$ has the sign $\textrm{sign}(\lambda _a)=(\frac{a}{n})$.

We also need another lemma.

Lemma 4.2

Let $n>1$ be an odd number and let $a\in {\mathbb {Z}}$ with $\gcd (a,n)=1$. Then

$$\begin{aligned} \prod _{1\leqslant j<k\leqslant n-1}\left( e^{2\pi \textrm{i}ak/n}-e^{2\pi \textrm{i}aj/n}\right) ^2=(-1)^{(n-1)/2}n^{n-2}. \end{aligned}$$

(4.1)

Proof

Let $\zeta =e^{2\pi \textrm{i}a/n}$. Clearly,

$$\begin{aligned} \prod _{r=1}^{n-1}(1-\zeta ^r)=\lim _{x\rightarrow 1}\frac{x^n-1}{x-1}=n \end{aligned}$$

(4.2)

and hence

$$\begin{aligned}(-1)^{\left( {\begin{array}{c}n-1\\ 2\end{array}}\right) }\prod _{1\leqslant j<k\leqslant n-1}(\zeta ^k-\zeta ^j)^2&=\prod _{j=1}^{n-1}\prod _{\begin{array}{c} k=1\\ k\not =j \end{array}}^{n-1}(\zeta ^j-\zeta ^{k}) =\prod _{j=1}^{n-1}\prod _{\begin{array}{c} k=1\\ k\not =j \end{array}}^{n-1}\zeta ^j(1-\zeta ^{k-j}) \\&=\prod _{j=1}^{n-1}\bigg (\frac{(\zeta ^j)^{n-2}}{1-\zeta ^{-j}}\prod _{\begin{array}{c} k=0\\ k\not =j \end{array}}^{n-1}(1-\zeta ^{k-j})\bigg ) \\ {}&=\frac{\zeta ^{(n-1)\sum _{j=0}^{n-1}j}}{\prod _{j=1}^{n-1}(\zeta ^j-1)}\prod _{j=1}^{n-1}\prod _{r=1}^{n-1}(1-\zeta ^r) =n^{n-2}. \end{aligned}$$

So (4.1) holds. $\square$

Proof of Theorem 1.2

In view of Lemma 4.1, for each $\delta =0,1$, we have

$$\begin{aligned}&\det \left[ x+\tan \pi \frac{aj+bk}{n}\right] _{\delta \leqslant j,k\leqslant n-1}\\&\quad =\left( \frac{a}{n}\right) \det \left[ x+\tan \pi \frac{j+bk}{n}\right] _{\delta \leqslant j,k\leqslant n-1} =\left( \frac{-ab}{n}\right) D_n^{(\delta )}(x), \end{aligned}$$

where

$$\begin{aligned} D_n^{(\delta )}(x):=\det \left[ x+\tan \pi \frac{j-k}{n}\right] _{\delta \leqslant j,k\leqslant n-1}. \end{aligned}$$

Since

$$\begin{aligned} D_n^{(0)}(-x)&=\det \left[ -x+\tan \pi \frac{k-j}{n}\right] _{0\leqslant j,k\leqslant n-1}=\det \left[ -x-\tan \pi \frac{j-k}{n}\right] _{0\leqslant j,k\leqslant n-1} \\ {}&=(-1)^n\det \left[ x+\tan \pi \frac{j-k}{n}\right] _{0\leqslant j,k\leqslant n-1}=-D_n^{(0)}(x), \end{aligned}$$

we have

$$\begin{aligned}{} & {} \det \left[ -x+\tan \pi \frac{aj+bk}{n}\right] _{0\leqslant j,k\leqslant n-1}\\{} & {} \quad =-\left( \frac{-ab}{n}\right) D_n^{(0)}(x) =-\det \left[ x+\tan \pi \frac{aj+bk}{n}\right] _{0\leqslant j,k\leqslant n-1} \end{aligned}$$

and hence (1.8) holds.

Now it remains to show that $D_n^{(1)}(x)=n^{n-2}$. Write $\zeta =e^{2\pi \textrm{i}/n}$. Similar to (3.2), we have

$$\begin{aligned} \textrm{i}+\tan \pi \frac{j-k}{n}=\frac{2\textrm{i}}{\zeta ^{j-k}+1}\quad \ \text {for all}\ j,k=1,\ldots ,n-1. \end{aligned}$$

Thus

$$\begin{aligned} D_n^{(1)}(\textrm{i})=\det \left[ \frac{2\textrm{i}}{\zeta ^{j-k}+1}\right] _{1\leqslant j,k\leqslant n-1} =\prod _{k=1}^{n-1}(2\textrm{i}\zeta ^k)\times \det \left[ \frac{1}{\zeta ^j+\zeta ^k}\right] _{1\leqslant j,k\leqslant n-1}. \end{aligned}$$

By Lemma 2.2,

$$\begin{aligned} \det \left[ \frac{1}{\zeta ^j+\zeta ^k}\right] _{1\leqslant j,k\leqslant n-1}&=\frac{\prod _{1\leqslant j<k\leqslant n-1}(\zeta ^k-\zeta ^j)^2}{\prod _{j=1}^{n-1}\prod _{k=1}^{n-1}(\zeta ^j+\zeta ^k)} \\ {}&=\frac{\prod _{1\leqslant j<k\leqslant n-1}(\zeta ^k-\zeta ^j)^2}{\prod _{k=1}^{n-1}(2\zeta ^k)\times \prod _{1\leqslant j<k\leqslant n-1}(\zeta ^k+\zeta ^j)^2}. \end{aligned}$$

Therefore

$$\begin{aligned} D_n^{(1)}(\textrm{i})=\textrm{i}^{n-1}\prod _{1\leqslant j<k\leqslant n-1}\frac{(\zeta ^k-\zeta ^j)^4}{(\zeta ^{2k}-\zeta ^{2j})^2}=(-1)^{(n-1)/2}\prod _{1\leqslant j<k\leqslant n-1}\frac{(\zeta ^k-\zeta ^j)^4}{(\zeta ^{2k}-\zeta ^{2j})^2}. \end{aligned}$$

Combining this with Lemma 4.2, we immediately get $D_n^{(1)}(\textrm{i})=(n^{n-2})^2/n^{n-2}=n^{n-2}$. By Lemma 2.1,

$$\begin{aligned} D_n^{(1)}(x)=D_n^{(1)}(0)+rx \end{aligned}$$

for certain real number r. As $D_n^{(1)}(\textrm{i})=n^{n-2}$, we have $D_n^{(1)}(0)=n^{n-2}$ and $r=0$. Thus $D_n^{(1)}(x)=D_n^{(1)}(0)=n^{n-2}$ as desired.

The proof of Theorem 1.2 is now complete. $\square$

Proof of Theorem 1.3

For any nonzero real number $x\not \in \pi {\mathbb {Z}}$, we obviously have

$$\begin{aligned} \cot x=\frac{\cos x}{\sin x}=\frac{(e^{\textrm{i}x}+e^{-\textrm{i}x})/2}{(e^{\textrm{i}x}-e^{-\textrm{i}x})/(2\textrm{i})}=\textrm{i}+\frac{2\textrm{i}}{e^{2\textrm{i}x}-1}. \end{aligned}$$

Thus

$$\begin{aligned} -\textrm{i}+\cot \pi \frac{aj^2+bk^2}{p}=\frac{2\textrm{i}}{\zeta ^{aj^2+bk^2}-1}\quad \text {for all}\ j,k=1,\ldots ,n, \end{aligned}$$

where $n=(p-1)/2$ and $\zeta =e^{2\pi \textrm{i}/p}$. Let

$$\begin{aligned} C(x)=\det \left[ x+\cot \pi \frac{aj^2+bk^2}{p}\right] _{1\leqslant j,k\leqslant n}. \end{aligned}$$

Then

$$\begin{aligned}C(-\textrm{i})=\det \left[ \frac{2\textrm{i}}{\zeta ^{aj^2+bk^2}-1}\right] _{1\leqslant j,k\leqslant n}=\prod _{k=1}^n(2\textrm{i}\zeta ^{-bk^2})\times \det \left[ \frac{1}{\zeta ^{aj^2}-\zeta ^{-bk^2}}\right] _{1\leqslant j,k\leqslant n}. \end{aligned}$$

By Lemma 2.2,

$$\begin{aligned}\det \left[ \frac{1}{\zeta ^{aj^2}-\zeta ^{-bk^2}}\right] _{1\leqslant j,k\leqslant n}&=\frac{\prod _{1\leqslant j<k\leqslant n}(\zeta ^{ak^2}-\zeta ^{aj^2})(-\zeta ^{-bk^2}-(-\zeta ^{-bj^2}))}{\prod _{j=1}^n\prod _{k=1}^n(\zeta ^{aj^2}-\zeta ^{-bk^2})} \\ {}&=(-1)^{\left( {\begin{array}{c}n\\ 2\end{array}}\right) }\frac{\prod _{1\leqslant j<k\leqslant n}(\zeta ^{ak^2}-\zeta ^{aj^2})(\zeta ^{-bk^2}-\zeta ^{-bj^2})}{(\prod _{k=1}^n\zeta ^{-bk^2})^n\prod _{j=1}^n\prod _{k=1}^n(\zeta ^{aj^2+bk^2}-1)}. \end{aligned}$$

Note that $\prod _{k=1}^n\zeta ^{k^2}=1$ by (3.1). So

$$\begin{aligned} C(-\textrm{i})=(2\textrm{i})^n \frac{(-1)^{\left( {\begin{array}{c}n\\ 2\end{array}}\right) }\prod _{1\leqslant j<k\leqslant n}(\zeta ^{ak^2}-\zeta ^{aj^2})(\zeta ^{-bk^2}-\zeta ^{-bj^2})}{(-1)^n\prod _{j=1}^n\prod _{k=1}^n(1-\zeta ^{aj^2+bk^2})}. \end{aligned}$$

(4.3)

Case 1. $p\equiv 1\ ({\textrm{mod}}\ 4)$, i.e., $2\mid n$.

In this case, $(\frac{ab}{p})=(\frac{-ab}{p})=-1$. By Lemma 2.5,

$$\begin{aligned} \prod _{j=1}^n\prod _{k=1}^n(1-\zeta ^{aj^2+bk^2})=p^{(p-1)/4}. \end{aligned}$$

Combining this with (4.3) and (3.6), we get

$$\begin{aligned}C(-\textrm{i})=\frac{2^{(p-1)/2}}{p^{(p-1)/4}} \prod _{1\leqslant j<k\leqslant n}(\zeta ^{ak^2}-\zeta ^{aj^2})(\zeta ^{-bk^2}-\zeta ^{-bj^2}) =\frac{D_p(a,b)}{(-p)^{(p-1)/4}}, \end{aligned}$$

where $D_p(a,b)$ is defined as in (3.5). Thus

$$\begin{aligned} C(-\textrm{i})=\frac{D_p(a,b)}{(-p)^{(p-1)/4}}=\frac{T_p^{(1)}(a,b)}{(-p)^{(p-1)/4}} =\pm \frac{2^{(p-1)/2}}{\sqrt{p}} \end{aligned}$$

in view of (3.2) and (1.5). By Lemma 2.1, $C(x)=C(0)+rx$ for certain real number r. Since $C(-\textrm{i})$ is real, we have $r=0$ and hence

$$\begin{aligned} C(x)=C(-\textrm{i})=\frac{T_p^{(1)}(a,b)}{(-p)^{(p-1)/4}}=\pm \frac{2^{(p-1)/2}}{\sqrt{p}}. \end{aligned}$$

Case 2. $p\equiv 3\ ({\textrm{mod}}\ 4)$, i.e., $2\not \mid n$.

In light of (2.9),

$$\begin{aligned} \prod _{1\leqslant j<k\leqslant n}(\zeta ^{ak^2}-\zeta ^{aj^2})(\zeta ^{-bk^2}-\zeta ^{-bj^2})=p^{(p-3)/4}. \end{aligned}$$

Combining this with (2.11) and (4.3), we obtain

$$\begin{aligned} C(-\textrm{i})&=(2\textrm{i})^n(-1)^{\left( {\begin{array}{c}n\\ 2\end{array}}\right) }\frac{p^{(p-3)/4}}{(-1)^n(-1)^{(h(-p)-1)/2}(\frac{a}{p})p^{(p-1)/4}\textrm{i}}\\&=\frac{2^n\textrm{i}(\textrm{i}^2)^{(n-1)/2}(-1)^{(n-1)/2}}{(-1)^{(h(-p)+1)/2}(\frac{a}{p})\sqrt{p}\,\textrm{i}} \end{aligned}$$

and hence

$$\begin{aligned} C(-\textrm{i})=(-1)^{(h(-p)+1)/2}\left( \frac{a}{p}\right) \frac{2^{(p-1)/2}}{\sqrt{p}} \end{aligned}$$

is a real number. Combining this with Lemma 2.1, we get that

$$\begin{aligned} C(x)=C(-\textrm{i})=(-1)^{(h(-p)+1)/2}\left( \frac{a}{p}\right) \frac{2^{(p-1)/2}}{\sqrt{p}}. \end{aligned}$$

In view of the above, we have completed the proof of Theorem 1.3. $\square$

The following lemma is a well known result on quadratic Gauss sums (cf. [6, pp. 70-76]).

Lemma 4.3

Let p be an odd prime. Then, for any integer $a\not \equiv 0\ ({\textrm{mod}}\ p)$, we have

$$\begin{aligned} \sum _{x=0}^{p-1}e^{2\pi \textrm{i}ax^2/p}=\left( \frac{a}{p}\right) \sum _{t=0}^{p-1}\left( \frac{t}{p}\right) e^{2\pi \textrm{i}t/p} =\left( \frac{a}{p}\right) \sqrt{(-1)^{(p-1)/2}p}. \end{aligned}$$

Let p be an odd prime, and let $\zeta =e^{2\pi \textrm{i}/p}$. For $a,b\in {\mathbb {Z}}$ with $p\not \mid ab$, Lemmas 2.2 and 4.3 are helpful to evaluate $\det [z+1/(\zeta ^{aj^2}+\zeta ^{bk^2})]_{1\leqslant j,k\leqslant (p-1)/2}$. However, we actually only need the case $z=0$ in our previous proofs of Theorems 1.1–1.3.

Proof of Theorem 1.4

The Galois group $\text {Gal}({\mathbb {Q}}(e^{2\pi \textrm{i}/p})/{\mathbb {Q}})$ consists of those ${\mathbb {Q}}$-auto- morphisms $\sigma _a$ $(1\le a\le p-1)$ with $\sigma _a(e^{2\pi \textrm{i}/p})=e^{2\pi \textrm{i}a/p}$. For any integer $x\not \equiv 0\ ({\textrm{mod}}\ p)$, we have

$$\begin{aligned} \cot \pi \frac{x}{p}=\textrm{i}\frac{e^{\pi \textrm{i}x/p}+e^{-\pi \textrm{i}x/p}}{e^{\pi \textrm{i}x/p}-e^{-\pi \textrm{i}x/p}}=\textrm{i}\frac{e^{2\pi \textrm{i}x/p}+1}{e^{2\pi \textrm{i}x/p}-1}. \end{aligned}$$

It follows that

$$\begin{aligned} \frac{D_p}{\textrm{i}^{(p-1)/2}}=\det \left[ \frac{e^{2\pi \textrm{i}jk/p}+1}{e^{2\pi \textrm{i}jk/p}-1}\right] _{1\leqslant j,k\leqslant (p-1)/2}. \end{aligned}$$

Let $a\in \{1,\ldots ,p-1\}$. By the last equality,

$$\begin{aligned} \sigma _a\left( \frac{D_p}{\textrm{i}^{(p-1)/2}}\right)&=\sigma _a\bigg (\sum _{\tau \in S_{(p-1)/2}}\textrm{sign}(\tau )\prod _{j=1}^{(p-1)/2}\frac{e^{2\pi \textrm{i}j\tau (j)/p}+1}{e^{2\pi \textrm{i}j\tau (j)/p}-1}\bigg ) \\ {}&= \sum _{\tau \in S_{(p-1)/2}}\textrm{sign}(\tau )\prod _{j=1}^{(p-1)/2}\frac{e^{2\pi \textrm{i}aj\tau (j)/p}+1}{e^{2\pi \textrm{i}aj\tau (j)/p}-1} \\ {}&=\det \left[ \frac{e^{2\pi \textrm{i}ajk/p}+1}{e^{2\pi \textrm{i}ajk/p}-1}\right] _{1\leqslant j,k\leqslant (p-1)/2}\\&=\frac{1}{\textrm{i}^{(p-1)/2}}\det \left[ \cot \pi \frac{ajk}{p}\right] _{1\le j,k\le (p-1)/2}. \end{aligned}$$

By Gauss’ Lemma (see, e.g., [6, p. 52]),

$$\begin{aligned} \left( \frac{a}{p}\right) =(-1)^{|\{1\le j\le (p-1)/2:\ \{aj/p\}>1/2\}|}, \end{aligned}$$

where $\{x\}$ denotes the fractional part of a real number x. Therefore,

$$\begin{aligned} \sigma _a\left( \frac{D_p}{\textrm{i}^{(p-1)/2}}\right) =&\frac{(\frac{a}{p})}{\textrm{i}^{(p-1)/2}}\det \left[ \cot \pi \frac{\rho _a(j)k}{p}\right] _{1\le j,k\le (p-1)/2}, \end{aligned}$$

where $\rho _a(j)$ is the unique $r\in \{1,\ldots ,(p-1)/2\}$ with $aj\equiv \pm r\ ({\textrm{mod}}\ p)$. Combining this with Lemma 2.3, we deduce that

$$\begin{aligned} \sigma _a\left( \frac{D_p}{\textrm{i}^{(p-1)/2}}\right)&=\frac{(\frac{a}{p})}{\textrm{i}^{(p-1)/2}}\left( \frac{a}{p}\right) ^{(p+1)/2}\det \left[ \cot \pi \frac{jk}{p}\right] _{1\le j,k\le (p-1)/2} \\ {}&=\left( \frac{a}{p}\right) ^{(p-1)/2}\frac{D_p}{\textrm{i}^{(p-1)/2}}. \end{aligned}$$

If $p\equiv 1\ ({\textrm{mod}}\ 4)$, then $\textrm{i}^{(p-1)/2}=(-1)^{(p-1)/4}\in {\mathbb {Q}}$ and hence $\sigma _a(D_p)=D_p$. When $p\equiv 3\ ({\textrm{mod}}\ 4)$, by Lemma 4.3 we have $\sqrt{-p}\in {\mathbb {Q}}(e^{2\pi \textrm{i}/p})$ and

$$\begin{aligned} \sigma _a(\sqrt{-p})=\sigma _a\bigg (\sum _{x=0}^{p-1}e^{2\pi \textrm{i}x^2/p}\bigg ) =\sum _{x=0}^{p-1}e^{2\pi \textrm{i}ax^2/p}=\left( \frac{a}{p}\right) \sqrt{-p}, \end{aligned}$$

therefore

$$\begin{aligned} \sigma _a\left( (-1)^{(p+1)/4}\frac{D_p}{\sqrt{p}}\right)&=\sigma _a\left( \frac{D_p}{\textrm{i}(-1)^{(p-3)/4}\sqrt{-p}}\right) =\frac{\sigma _a(D_p/\textrm{i}^{(p-1)/2})}{\sigma _a(\sqrt{-p})} \\ {}&=\frac{(\frac{a}{p})D_p/\textrm{i}^{(p-1)/2}}{(\frac{a}{p})\sqrt{-p}}=(-1)^{(p+1)/4}\frac{D_p}{\sqrt{p}} \end{aligned}$$

and hence $\sigma _a(D_p/\sqrt{p})=D_p/\sqrt{p}$.

By the above, if $p\equiv 1\ ({\textrm{mod}}\ 4)$, then $\sigma (D_p)=D_p$ for all $\sigma \in \text {Gal}((e^{2\pi \textrm{i}/p})/{\mathbb {Q}})$, and hence $D_p\in {\mathbb {Q}}$ by Galois theory. Similarly, when $p\equiv 3\ ({\textrm{mod}}\ 4)$,we have $D_p/\sqrt{p}\in {\mathbb {Q}}$.

The proof of Theorem 1.4 is now complete. $\square$

5 Some open conjectures

Conjecture 5.1

Let p be any odd prime. Then

$$\begin{aligned} \left( \frac{-2}{p}\right) \frac{\det \left[ \cot \pi {jk}/p\right] _{1\leqslant j,k\leqslant (p-1)/2}}{2^{(p-3)/2}p^{(p-5)/4}}\in \{1,2,3,\ldots \}, \end{aligned}$$

(5.1)

and this number is divisible by $h(-p)$ if $p\equiv 3\ ({\textrm{mod}}\ 4)$.

Remark 5.1

By Theorem 1.4, for any odd prime p, we have

$$\begin{aligned} \det \left[ \cot \pi \frac{jk}{p}\right] _{1\leqslant j,k\leqslant (p-1)/2}\in p^{(p-1)/4}{\mathbb {Q}}. \end{aligned}$$

Conjecture 5.2

Let n be a positive integer.

• (i) The number

  $$\begin{aligned} s_n:=(2n+1)^{-n/2}\det \left[ \tan \pi \frac{jk}{2n+1}\right] _{1\leqslant j,k\leqslant n} \end{aligned}$$

  (5.2)

  is always an integer.

• (ii) We have

  $$\begin{aligned} \det \left[ \tan ^2\pi \frac{jk}{2n+1}\right] _{1\leqslant j,k\leqslant n}\in (2n+1)^{(n+1)/2}4^{n-1}{\mathbb {Z}}. \end{aligned}$$

  (5.3)

Remark 5.2

Via Mathematica, we find that

$$\begin{aligned} s_1=1,\ s_2=-2,\ s_3=s_4=4,\ s_5=48,\ s_6=-160, \\ s_7=32,\ s_8=2176,\ s_9=6912,\ s_{10}=0,\ s_{11}=273408. \end{aligned}$$

Let $t_n$ denote the nth term of the sequence [5, A277445], which is the determinant of a matrix $T(n)=[t_{jk}]_{1\leqslant j,k\leqslant n}$ with entries among $0,\pm 1$ such that

$$\begin{aligned} 2\sum _{k=1}^nt_{jk}\sin \frac{\pi k}{2n+1}=\tan \frac{\pi j}{2n+1}\quad \ \text {for all}\ j=1,\ldots ,n. \end{aligned}$$

We guess that $s_n=-t_n$ if $n\equiv 3\ ({\textrm{mod}}\ 4)$, and $s_n=t_n$ otherwise.

Conjecture 5.3

For any odd integer $n>1$, we have

$$\begin{aligned} \det \left[ \tan ^2\pi \frac{j+k}{n}\right] _{1\leqslant j,k\leqslant n-1}\in n^{n-2}{\mathbb {Z}}. \end{aligned}$$

(5.4)

Remark 5.3

We are able to prove that $\det [\tan ^2\pi \frac{j-k}{n}]_{1\leqslant j,k\leqslant n-1}\in {\mathbb {Z}}$ for any odd integer $n>1$.

Conjecture 5.4

Let $p\equiv 3\ ({\textrm{mod}}\ 4)$ be a prime, and let $a,b\in {\mathbb {Z}}$ with $p\not \mid ab$. Then

$$\begin{aligned} \det \left[ \tan ^2\pi \frac{aj^2+bk^2}{p}\right] _{1\leqslant j,k\leqslant (p-1)/2}\in p^{(p-3)/4}{\mathbb {Z}}\end{aligned}$$

(5.5)

and

$$\begin{aligned} \det \left[ \tan ^2\pi \frac{aj^2+bk^2}{p}\right] _{0\leqslant j,k\leqslant (p-1)/2}\in p^{(p+1)/4}{\mathbb {Z}}. \end{aligned}$$

(5.6)

If $(\frac{ab}{p})=1$, then

$$\begin{aligned} \det \left[ \cot ^2\pi \frac{aj^2+bk^2}{p}\right] _{1\leqslant j,k\leqslant (p-1)/2}\in \frac{2^{p-3}}{p}{\mathbb {Z}}. \end{aligned}$$

(5.7)

Let $p\equiv 1\ ({\textrm{mod}}\ 4)$ be a prime, and let $a,b\in {\mathbb {Z}}$ with $p\not \mid ab$. Choose $q\in {\mathbb {Z}}$ with $q^2\equiv -1\ ({\textrm{mod}}\ p)$. Then

$$\begin{aligned}&\ \det \left[ \left( \frac{a(qj)^2+b(qk)^2}{p}\right) \tan \pi \frac{a(qj)^2+b(qk)^2}{p}\right] _{0\leqslant j,k\leqslant (p-1)/2} \\=&\ (-1)^{(p+1)/2}\det \left[ \left( \frac{aj^2+bk^2}{p}\right) \tan \pi \frac{aj^2+bk^2}{p}\right] _{0\leqslant j,k\leqslant (p-1)/2} \end{aligned}$$

and hence

$$\begin{aligned} \det \left[ \left( \frac{aj^2+bk^2}{p}\right) \tan \pi \frac{aj^2+bk^2}{p}\right] _{0\leqslant j,k\leqslant (p-1)/2}=0. \end{aligned}$$

(5.8)

Conjecture 5.5

Let $p\equiv 3\ ({\textrm{mod}}\ 4)$ be a prime and let $a,b\in {\mathbb {Z}}$ with $p\not \mid ab$. Then

$$\begin{aligned} \det \left[ \left( \frac{aj^2+bk^2}{p}\right) \tan \pi \frac{aj^2+bk^2}{p}\right] _{0\leqslant j,k\leqslant (p-1)/2}\in p{\mathbb {Z}}. \end{aligned}$$

(5.9)

If $(\frac{ab}{p})=1$, then

$$\begin{aligned} \sqrt{p}\det \left[ \left( \frac{aj^2+bk^2}{p}\right) \cot \pi \frac{aj^2+bk^2}{p}\right] _{1\leqslant j,k\leqslant (p-1)/2}\in {\mathbb {Z}}. \end{aligned}$$

(5.10)

Remark 5.4

For any prime $p\equiv 3\ ({\textrm{mod}}\ 4)$, set

$$\begin{aligned} a_p^{\pm }:=\frac{1}{p}\det \left[ \left( \frac{j^2\pm k^2}{p}\right) \tan \pi \frac{j^2\pm k^2}{p}\right] _{0\leqslant j,k\leqslant (p-1)/2}. \end{aligned}$$

Via Mathematica,we find that

$$\begin{aligned} a_3^+=a_3^-=-1,\ a_7^+=60,\ a_7^-=3,\ a_{11}^+=2^6\times 3^3,\ a_{11}^-=-373, \\a_{19}^+=2^{12}\times 3\times 5^2\times 7\times 11\times 17 \ \text {and}\ a_{19}^-=-5\times 7\times 89\times 3803. \end{aligned}$$

Conjecture 5.6

Let p be an odd prime.

• (i) Define

  $$\begin{aligned} S(p):=\det \left[ \sec 2\pi \frac{jk}{p}\right] _{0\leqslant j,k\leqslant (p-1)/2}. \end{aligned}$$

  If $p\equiv 1\ ({\textrm{mod}}\ 4)$, then $S(p)=0$. When $p\equiv 3\ ({\textrm{mod}}\ 4)$, the number

  $$\begin{aligned} \frac{S(p)}{2^{(p-3)/2}(-p)^{(p+1)/4}} \end{aligned}$$

  is a positive odd integer.

• (ii) We have

  $$\begin{aligned} c_p:=\frac{1}{2^{(p-1)/2}p^{(p-5)/4}}\det \left[ \csc 2\pi \frac{jk}{p}\right] _{1\leqslant j,k\leqslant (p-1)/2}\in {\mathbb {Z}}. \end{aligned}$$

  Moreover, $c_p=0$ if $p\equiv 7\ ({\textrm{mod}}\ 8)$.

Remark 5.5

By the way,we prove Theorem 1.4, and we can show that $S(p)/p^{(p+1)/4}\in {\mathbb {Q}}$ and $c_p\in {\mathbb {Q}}$ for any odd prime p. In 2019,the author [12] conjectured that

$$\begin{aligned} \frac{1}{2n}\det \left[ \cos \pi \frac{jk}{n}\right] _{0\leqslant j,k\leqslant n}=\det \left[ \cos \pi \frac{jk}{n}\right] _{1\leqslant j,k\leqslant n}=(-1)^{\lfloor \frac{n+1}{2}\rfloor }\frac{n^{(n-1)/2}}{2^{(n-1)/2}} \end{aligned}$$

(5.11)

for every positive integer n, this was later confirmed by Petrov (cf. the answer in [12]).

Conjecture 5.7

For any prime $p\equiv 3\ ({\textrm{mod}}\ 4)$, there is an integer $x_p\equiv 1\ ({\textrm{mod}}\ p)$ such that

$$\begin{aligned} \det \left[ \sec 2\pi \frac{(j-k)^2}{p}\right] _{0\leqslant j,k\leqslant p-1}=-p^{(p+3)/2}x_p^2. \end{aligned}$$

(5.12)

Remark 5.6

For $p=3,7,11$, we may take $x_p=1$ in (5.12). For each prime $p\equiv 3\ ({\textrm{mod}}\ 4)$, the author [13] conjectured in 2021 that

$$\begin{aligned} \det \left[ \sin 2\pi \frac{(j-k)^2}{p}\right] _{1\leqslant j,k\leqslant p-1}=-\frac{p^{(p-1)/2}}{2^{p-1}}, \end{aligned}$$

which was later confirmed by Kalmynin (cf. the answer in [13]).