A Hypergeometric Function Transform and Matrix-Valued Orthogonal Polynomials

Abstract

The spectral decomposition for an explicit second-order differential operator T is determined. The spectrum consists of a continuous part with multiplicity two, a continuous part with multiplicity one, and a finite discrete part with multiplicity one. The spectral analysis gives rise to a generalized Fourier transform with an explicit hypergeometric function as a kernel. Using Jacobi polynomials, the operator T can also be realized as a five-diagonal operator, leading to orthogonality relations for 2×2-matrix-valued polynomials. These matrix-valued polynomials can be considered as matrix-valued generalizations of Wilson polynomials.

1 Introduction

It is well known that a three-term recurrence relation

$$\lambda p_n(\lambda ) = a_n p_{n+1}(\lambda ) + b_n p_{n}(\lambda ) + a_{n-1} p_{n-1}(\lambda ), \quad n =0,1,2,\ldots, $$

with a ₋₁=0, can be solved using orthogonal polynomials. A generalization of this is obtained by Durán and Van Assche [5], who show that a 2N+1-term recurrence relation can be solved using N×N-matrix-valued orthogonal polynomials. Motivated by this result and previous work by Ismail and the second author [12, 13], a method is presented by Ismail and the authors [8] to obtain orthogonality relations for 2×2-matrix-valued orthogonal polynomials from an operator T on a Hilbert space $\mathcal{H}$ of functions. The operator T must satisfy the following conditions:

• 1. T is self-adjoint;

• 2. there exists a weighted Hilbert space $L^{2}(\mathcal{V})$ and a unitary operator $U:\mathcal{H} \to L^{2}(\mathcal{V})$ so that UT=MU, where M is the multiplication operator on $L^{2}(\mathcal{V})$;

• 3. there exists an orthonormal basis $\{f_{n}\} _{n=0}^{\infty}$ of $\mathcal{H}$, and there exist sequences $(a_{n})_{n=0}^{\infty}$, $(b_{n})_{n=0}^{\infty}$, $(c_{n})_{n=0}^{\infty}$ of numbers with a _n >0 and $c_{n} \in \mathbb{R}$ for all $n \in \mathbb{N}$, such that

  $$T f_n = a_n f_{n+2} + b_n f_{n+1} + c_n f_n + \overline{b_{n-1}} f_{n-1} + a_{n-2}f_{n-2}, $$

  where we assume a ₋₁=a ₋₂=b ₋₁=0.

In [8], two explicit examples are worked out, where the operator T is, besides a five-term operator, also realized as the second-order q-difference operator corresponding to well-known q-hypergeometric orthogonal polynomials. Thus, the unitary operator U is the integral transform with the corresponding orthogonal polynomials as a kernel. This leads to complicated, but explicit, orthogonality relations for certain matrix-valued polynomials defined by an explicit matrix three-term recurrence relation. We note that the explicit weight function differs structurally from the usually considered weight functions for matrix-valued orthogonal polynomials consisting of a matrix-deformation of a classical weight.

In this paper, we apply the method from [8] with the second-order differential operator T=T ^(α,β;κ) defined by

$$\begin{aligned} T =& \bigl(1-x^2\bigr)^2 \frac{d^2}{dx^2} + \bigl(1-x^2\bigr)\bigl[\beta -\alpha -(\alpha +\beta +4)x\bigr]\frac{d}{dx} \\ &{}+ \frac{1}{4}\bigl[\kappa ^2-(\alpha +\beta +3)^2\bigr] \bigl(1-x^2\bigr). \end{aligned}$$

(1.1)

Here x∈(−1,1), α,β>−1, and $\kappa \in \mathbb{R}_{\geq0} \cup i\mathbb{R}_{>0}$. The differential operator T is closely related to the second-order differential operator to which the Jacobi polynomials are eigenfunctions. It should be noted that T raises the degree of a polynomial by 2, so there are no polynomial eigenfunctions. We will show that the differential operator T, considered as an unbounded operator on a weighted L ²-space, satisfies conditions (i)–(iii) given above. An interesting problem here is that T does not correspond to orthogonal polynomials or to a known unitary integral transform such as the Jacobi function transform [15].

The unitary operator U needed in condition (ii) is given by an explicit integral transform $\mathcal{F}$ which is obtained from spectral analysis of T. The spectrum of T consists of a continuous part with multiplicity two, a continuous part with multiplicity one, and a (possibly empty) finite discrete part of multiplicity one. As a result, the integral transform $\mathcal{F}$ has a hypergeometric kernel which is partly $\mathbb{C}^{2}$-valued and partly $\mathbb{C}$-valued. There are several (but not very many) hypergeometric integral transforms with $\mathbb{C}^{2}$-valued kernels available in the literature, see, e.g., [9, 17], [14, Exercise (4.4.11)], see also [7] for an example with basic hypergeometric functions. To the best of our knowledge, all known examples can be considered as nonpolynomial extensions of hypergeometric orthogonal polynomials, in the sense that the corresponding kernels are eigenfunctions of a differential/difference operator that also has orthogonal polynomials as eigenfunctions. For example, Neretin’s $\mathbb{C}^{2}$-valued ₂ F ₁-integral transform [17] generalizes the Jacobi polynomials. The integral transform $\mathcal{F}$ we consider in this paper, however, does not seem to generalize a family of orthogonal polynomials, although in a special case it can be considered as a nonpolynomial extension of two different one-parameter families of Jacobi polynomials. Furthermore, other hypergeometric integral transforms and hypergeometric orthogonal polynomials correspond to a bispectral problem, see, e.g., [10], which can always be related directly to contiguous relations for hypergeometric functions. From the explicit expressions as hypergeometric functions for the kernel of $\mathcal{F}$, it is unclear whether $\mathcal{F}$ is also related to a bispectral problem.

In a special case, the 2×2-matrix-valued orthogonal polynomials we obtain can be diagonalized. In this case, the orthogonality relations correspond to orthogonality relations for two subfamilies of Wilson polynomials [19], and the matrix-three-term recurrence relation corresponds to the three-term recurrence relations for the two subfamilies. Moreover, the differential operator arises as an extension of [12, §3] from a three-term to five-term recurrence operator for such a differential operator as described in [8]. In [12, §3], a subfamily of the Wilson polynomials play the role of the matrix-valued polynomials of this paper. This is why we consider our matrix-valued polynomials as generalizations of (subfamilies of) Wilson polynomials. It should be stressed that it is completely unclear at this point whether the matrix-valued polynomials have other properties that are fundamental for the Wilson polynomials. For example, the Wilson polynomials are eigenfunctions of a second-order difference operator, but a similar result for the matrix-valued polynomials in this paper is not (yet) known.

The organization of this paper is as follows. In Sect. 2, we introduce the integral transform $\mathcal{F}$ and show that the differential operator T (1.1) satisfies conditions (i) and (ii). The proofs for this section are given separately in Sect. 4, where the spectral analysis of T is carried out, which can be quite technical at certain points. In Sect. 3, we realize T as a five-diagonal operator on a basis consisting of Jacobi polynomials, so that condition (iii) is also satisfied. The corresponding five-term recurrence relation is equivalent to a matrix three-term recurrence relation that defines 2×2-matrix-valued orthogonal polynomials P _n for which the orthogonality relations are determined. We also consider briefly the special case α=β, in which case the integral transform $\mathcal{F}$ reduces to two Jacobi function transforms and the orthogonality relations for P _n correspond to the orthogonality relations for certain Wilson polynomials. Finally, in Sect. 4, eigenfunctions of T are given, which are needed for the spectral decomposition of T. The spectral decomposition leads to a proof of the unitarity of the integral transform $\mathcal{F}$, and to an explicit formula for its inverse.

Notation

We write $\mathbb{N}$ for the set of nonnegative integers. We use standard notation for hypergeometric functions, as in, e.g., [2, 11]. For products of Γ-functions and of shifted factorials, we use the shorthand notation

$$\begin{aligned} \varGamma (a_1,a_2,\ldots,a_n) & = \varGamma (a_1)\varGamma (a_2)\cdots \varGamma (a_n), \\ (a_1,a_2,\ldots, a_n)_k & = (a_1)_k (a_2)_k \cdots (a_n)_k. \end{aligned} $$

2 Spectral Analysis and a Hypergeometric Function Transform

In this section, we describe the spectral analysis of the operator T defined by (1.1). The spectral decomposition is given by an integral transform with certain hypergeometric ₂ F ₁-functions as a kernel which is interesting in its own right. The proofs for this section are postponed until Sect. 4.

Let α,β>−1 be fixed, and let w ^(α,β) be the Jacobi weight function on [−1,1] given by

$$ w^{(\alpha ,\beta )}(x) = 2^{-\alpha -\beta -1} \frac{\varGamma (\alpha +\beta +2)}{\varGamma (\alpha +1,\beta +1) }(1-x)^\alpha (1+x)^\beta . $$

(2.1)

The corresponding inner product is denoted by 〈⋅,⋅〉,

$$\langle f,g \rangle= \int_{-1}^1 f(x) \overline{g(x)} w^{(\alpha ,\beta )}(x)\,dx. $$

The weight is normalized such that 〈1,1〉=1. We denote by $\mathcal{H}=\mathcal{H}^{(\alpha ,\beta )}$ the corresponding weighted L ²-space; $\mathcal{H}= L^{2}((-1,1),w^{(\alpha ,\beta )}(x)\,dx)$. To stress the dependence on the parameters α and β, we will sometimes denote the inner product in $\mathcal {H}^{(\alpha ,\beta )}$ by 〈⋅,⋅〉 _α,β . Let us remark that the substitution x↦−x sends T ^(α,β;κ) to T ^(β,α;κ), and $\mathcal{H}^{(\alpha ,\beta )}$ to $\mathcal{H}^{(\beta ,\alpha )}$. So without loss of generality we may assume β≥α, which we do from here on.

We consider T as an unbounded operator on $\mathcal{H}$. The domain $\mathcal{D}_{0}$ for T is described in Sect. 4.2, where the following result is proved.

Proposition 2.1

The operator $(T,\mathcal{D}_{0})$ has a unique self-adjoint extension.

We denote the extension of T again by T. The spectral analysis of T will be described by the integral transform $\mathcal{F}$ mapping functions in $\mathcal{H}$ (under suitable conditions) to functions in the Hilbert space $L^{2}(\mathcal{V})$. We first introduce the latter space.

Let $\varOmega _{1}, \varOmega _{2} \subset \mathbb{R}$ be given by

$$ \varOmega _1= \bigl({-}(\beta +1)^2,-( \alpha +1)^2 \bigr)\quad\text{and} \quad \varOmega _2= \bigl({-} \infty,-(\beta +1)^2 \bigr). $$

(2.2)

We set

$$ \begin{array}{l@{\quad}l} \delta _\lambda = i \sqrt{ -\lambda -( \alpha +1)^2},& \lambda \in \varOmega _1 \cup \varOmega _2, \\ \eta_\lambda = i \sqrt{ -\lambda -(\beta +1)^2},& \lambda \in \varOmega _2, \\ \delta (\lambda ) = \sqrt{\lambda +(\alpha +1)^2 },& \lambda \in \mathbb{C}\setminus( \varOmega _1 \cup \varOmega _2 ), \\ \eta(\lambda ) = \sqrt{\lambda +(\beta +1)^2}, & \lambda \in \mathbb{C}\setminus \varOmega _2. \end{array} $$

(2.3)

Here $\sqrt{\cdot}$ denotes the principal branch of the square root. For $n \in \mathbb{N}$, we define $\lambda _{n} \in \mathbb{C}$ as the solution of

$$ \delta (\lambda )+\eta(\lambda )=\sqrt{ \lambda + (\alpha +1)^2 } + \sqrt{\lambda +(\beta +1)^2} = -2n-1 + \kappa . $$

(2.4)

We define the finite set Ω _d by

$$ \varOmega _d = \biggl\{ \lambda _n \,\big|\, n \in \mathbb{N}\text{ and } n\leq\frac{1}{2}(\kappa -1)\biggr\}, $$

(2.5)

i.e., Ω _d consists of the real solutions of (2.4). Note that Ω _d =∅ if κ<1 or $\kappa \in i\mathbb{R}_{>0}$. The number λ _n ∈Ω _d has the explicit expression

$$\begin{aligned} \lambda _n &= \biggl( -n+\frac{1}{2}(\kappa -1) + \frac{ (\alpha -\beta )(\alpha +\beta +2)}{-4n-2+2\kappa } \biggr)^2 - (\alpha +1)^2 \\ & = \biggl( -n+\frac{1}{2}(\kappa -1) - \frac{ (\alpha -\beta )(\alpha +\beta +2)}{-4n-2+2\kappa } \biggr)^2 - ( \beta +1)^2. \end{aligned} $$

We will denote by σ the set Ω ₂∪Ω ₁∪Ω _d . Theorem 2.2 will show that σ is the spectrum of T.

Next we introduce the weight functions that we need to define $L^{2}(\mathcal{V})$. First we define

$$ c(x;y) = \frac{\varGamma (1+y,-x)}{\varGamma (\frac{1}{2}(1+y-x+\kappa ),\frac{1}{2}(1+y-x-\kappa ))}. $$

(2.6)

With this function, we define for λ∈Ω ₁,

$$ v(\lambda ) = \frac{1}{ c (\delta _\lambda ;\eta(\lambda ) )c (-\delta _\lambda ;\eta(\lambda ) ) }. $$

(2.7)

For λ∈Ω ₂, we define the matrix-valued weight function V(λ) by

$$ V(\lambda )= \begin{pmatrix} 1 & v_{12}(\lambda ) \\ v_{21}(\lambda ) & 1 \end{pmatrix} , $$

(2.8)

with

$$ v_{21}(\lambda ) = \frac{ c(\eta_\lambda ;\delta_\lambda ) }{c(-\eta _\lambda ;\delta_\lambda )} =\frac{ \varGamma (-\eta_\lambda ,\frac{1}{2}(1+\delta _\lambda +\eta_\lambda +\kappa ),\frac{1}{2}(1+\delta _\lambda +\eta_\lambda -\kappa ) ) }{\varGamma (\eta_\lambda ,\frac{1}{2}(1+\delta _\lambda -\eta_\lambda +\kappa ),\frac{1}{2}(1+\delta _\lambda - \eta_\lambda -\kappa ) )}, $$

(2.9)

and $v_{12}(\lambda ) = \overline{v_{21}(\lambda )}$. Finally, for λ _n ∈Ω _d , we set

$$\begin{aligned} N_{\lambda _n} =& \underset {\lambda =\lambda _n}{\mathrm {Res}} \biggl(\frac{c (\eta(\lambda );\delta (\lambda ) )}{\eta(\lambda )c (-\eta(\lambda );\delta (\lambda ) )} \biggr) \\ =& \frac{4\delta (\lambda _n)}{-2n-1+\kappa } \\ &{}\times \frac{(-1)^n \varGamma (-\eta(\lambda _n),\kappa -n ) }{n!\,\varGamma (\eta(\lambda _n), \frac{1}{2}(1+\delta (\lambda _n)-\eta(\lambda _n)+\kappa ), \frac{1}{2}(1+\delta (\lambda _n)-\eta(\lambda _n)-\kappa ) ) }. \end{aligned}$$

(2.10)

Note here that $\delta (\lambda _{n})-\eta(\lambda _{n}) = \frac{(\alpha -\beta )(\alpha +\beta +2)}{-2n-1+\kappa }$.

Now we are ready to define the Hilbert space $L^{2}(\mathcal{V})$. It consists of functions that are $\mathbb{C}^{2}$-valued on Ω ₂ and $\mathbb{C}$-valued on Ω ₁∪Ω _d . The inner product on $L^{2}(\mathcal{V})$ is given by

$$\begin{aligned} \langle f,g \rangle_{\mathcal{V}} &= \frac{1}{2\pi D} \int _{\varOmega _2} g(\lambda )^* V(\lambda ) f(\lambda ) \frac{d\lambda }{-i\eta_\lambda } \\ & \quad{}+ \frac{1}{2\pi D} \int_{\varOmega _1} f(\lambda ) \overline{g( \lambda )} v(\lambda ) \frac{d\lambda }{-i\delta _\lambda } + \frac{1}{D} \sum _{\lambda \in \varOmega _d} f(\lambda ) \overline{g(\lambda )} N_{\lambda }, \end{aligned} $$

where

$$ D=\dfrac{4\varGamma (\alpha +\beta +2)}{\varGamma (\alpha +1,\beta +1) }. $$

(2.11)

Next we introduce the integral transform $\mathcal{F}$. For λ∈Ω ₁ and x∈(−1,1), we define

$$\begin{aligned} \varphi_\lambda (x) =& \biggl( \frac{1-x}{2} \biggr)^{-\frac{1}{2}(\alpha - \delta _\lambda +1)} \biggl(\frac{1+x}{2} \biggr)^{-\frac{1}{2}(\beta - \eta(\lambda )+1)} \\ &{}\times{} _{2}F_{1} \left ( \genfrac {.}{.}{0pt}{}{ \frac{1}{2}\bigl(1 +\delta _\lambda + \eta(\lambda )- \kappa \bigr), \frac{1}{2}\bigl(1+ \delta _\lambda + \eta(\lambda )+\kappa \bigr)}{1+ \eta( \lambda )};\frac{1+x}{2} \right ). \end{aligned}$$

(2.12)

By Euler’s transformation, see, e.g., [2, (2.2.7)], we can replace δ _λ by −δ _λ in (2.12). Furthermore, we define for λ∈Ω ₂ and x∈(−1,1),

$$\begin{aligned} \varphi^\pm_\lambda (x) =& \biggl(\frac{1-x}{2} \biggr)^{-\frac{1}{2}(\alpha - \delta _\lambda +1)} \biggl(\frac {1+x}{2} \biggr)^{-\frac{1}{2}(\beta \mp\eta_\lambda +1)} \\ &{}\times{} _{2}F_{1} \left ( \genfrac {.}{.}{0pt}{}{ \frac{1}{2}(1 + \delta _\lambda \pm \eta_\lambda -\kappa ), \frac{1}{2}(1 + \delta _\lambda \pm \eta_\lambda +\kappa )}{1\pm\eta_\lambda };\frac{1+x}{2} \right ). \end{aligned}$$

(2.13)

Observe that $\overline{\varphi^{+}_{\lambda }(x)}=\varphi^{-}_{\lambda }(x)$, again by Euler’s transformation. Finally, for λ _n ∈Ω _d , we define

$$\begin{aligned} \varphi_{\lambda _n}(x) =& \biggl(\frac{1-x}{2} \biggr)^{-\frac{1}{2}(\alpha -\delta (\lambda _n)+1)} \biggl(\frac{1+x}{2} \biggr )^{-\frac{1}{2}(\beta -\eta(\lambda _n)+1)} \\ &{}\times {}_{2}F_{1} \left ( \genfrac {.}{.}{0pt}{}{-n,\kappa-n}{1+\eta(\lambda _n)};\frac{1+x}{2} \right ). \end{aligned}$$

(2.14)

Now, let $\mathcal{F}$ be the integral transform defined by

$$ (\mathcal{F}f) (\lambda ) = \begin{cases} \displaystyle\int_{-1}^1 f(x) \begin{pmatrix} \varphi^+_\lambda (x)\\ \varphi^-_\lambda (x) \end{pmatrix} w^{(\alpha ,\beta )}(x) \,dx,& \lambda \in \varOmega _2,\\ \displaystyle\int_{-1}^1 f(x) \varphi_\lambda (x) w^{(\alpha ,\beta )}(x) \,dx,& \lambda \in \varOmega _1,\\ \displaystyle\int_{-1}^1 f(x) \varphi_{\lambda _n}(x) w^{(\alpha ,\beta )}(x) \, dx,& \lambda =\lambda _n \in \varOmega _d, \end{cases} $$

(2.15)

for all $f \in\mathcal{H}$ such that the integrals converge. The following result says that $\mathcal{F}$ is the required unitary operator U from the introduction.

Theorem 2.2

The integral transform $\mathcal{F}$ extends uniquely to a unitary operator $\mathcal{F}:\mathcal{H} \to L^{2}(\mathcal{V})$ such that $\mathcal{F} T = M \mathcal{F}$, where $M:L^{2}(\mathcal{V}) \to L^{2}(\mathcal{V})$ is the unbounded multiplication operator given by (Mg)(λ)=λg(λ) for almost all λ∈σ.

Remark 2.3

In case α=β, the spectral decomposition of T can be described using the Jacobi function transform [15]. To see this, we apply the change of variable x=tanh(t). Then the second-order differential operator T defined by (1.1) turns into

$$\widehat{T} = \frac{d^2}{dt^2} + \bigl[\beta -\alpha -(\alpha +\beta +2)\tanh(t)\bigr] \frac{d}{dt} + \frac{\kappa ^2-(\alpha +\beta +3)^2}{4\cosh^2(t)}. $$

For α=β, let f _λ be a solution of the eigenvalue equation $\widehat{T} f_{\lambda }= \lambda f_{\lambda }$. Now define $F_{\lambda }^{\pm}(t) = \cosh^{\frac{1}{2}(2\alpha +3\pm \kappa )}(t) f_{\lambda }(t)$. Then F _λ satisfies

$$\frac{d^2 F_\lambda ^\pm}{dt^2} + (1\pm \kappa ) \tanh(t) \frac{ dF_\lambda ^\pm}{dt} = \biggl(\lambda + ( \alpha -1)^2 - \frac{1}{4}(1\pm \kappa )^2 \biggr) F_\lambda ^\pm. $$

Using the differential equation for Jacobi functions, see [15, (1.1)], we now see that the spectral decomposition of T can be given using the Jacobi function transforms corresponding to the Jacobi functions $\phi_{\delta _{\lambda }}^{(-\frac{1}{2},\frac{1}{2}\kappa )}$ and $\phi_{\delta _{\lambda }}^{(-\frac{1}{2},-\frac{1}{2}\kappa )}$.

We have an explicit inverse of the integral transform $\mathcal{F}$. Define for x∈(−1,1) the integral transform $\mathcal{G}$ by

$$\begin{aligned} (\mathcal{G}f) (x) &=\frac{1}{2\pi D} \int_{\varOmega _2} \bigl(\, \varphi_\lambda ^+(x) \ \varphi_\lambda ^-(x)\, \bigr) V( \lambda ) f(\lambda ) \frac{d\lambda }{-i\eta_\lambda } \\ & \quad{}+ \frac{1}{2\pi D} \int_{\varOmega _1} f(\lambda ) \varphi_\lambda (x)v(\lambda ) \frac{d\lambda }{-i\delta _\lambda } + \frac{1}{D} \sum _{\lambda \in \varOmega _d} f(\lambda ) \varphi_{\lambda }(x) N_{\lambda } \end{aligned} $$

for all functions $f \in L^{2}(\mathcal{V})$ for which the above integrals converge.

Theorem 2.4

The integral transform $\mathcal{G}$ extends uniquely to an operator $\mathcal{G}: L^{2}(\mathcal{V}) \to\mathcal{H}$ such that $\mathcal{G} = \mathcal {F}^{-1}$.

Theorem 2.2 and 2.4 are proved in Sect. 4. The following orthogonality relations are a result of Theorem 2.2 by considering the discrete spectrum of T, see Corollary 4.17 for the proof.

Corollary 2.5

Let κ≥1. Then the following orthogonality relations hold:

$$\begin{aligned} & \int_{-1}^1{} _{2}F_{1} \left ( \genfrac {.}{.}{0pt}{}{-m, \kappa-m}{1+\eta(\lambda _m)};\frac{1+x}{2} \right ) {}_{2}F_{1} \left ( \genfrac {.}{.}{0pt}{}{-n, \kappa-n}{1+\eta(\lambda _n)};\frac{1+x}{2} \right ) \\ & \qquad{}\times(1-x)^{\frac{1}{2}(\delta (\lambda _m)+\delta (\lambda _n)-2)}(1+x)^{\frac{1}{2}(\eta(\lambda _m)+\eta(\lambda _n)-2)} \,dx \\ &\quad = \delta _{mn} \frac{2^{\kappa -n-m}}{N_{\lambda _n}} \end{aligned} $$

for all $n,m \in \mathbb{N}$ such that $n,m \leq\frac{1}{2}(\kappa -1)$.

3 Matrix-Valued Orthogonal Polynomials

In this section, we show that the differential operator T can be realized as a five-diagonal operator with respect to an orthonormal basis for $\mathcal{H}$. Using the spectral decomposition for T, this leads to orthogonality relations for 2×2-matrix-valued orthogonal polynomials.

3.1 The Five-Diagonal Operator

The Jacobi polynomials are defined by

$$P_n^{(\alpha ,\beta )}(x) = \frac{(\alpha +1)_n}{n!} {}_{2}F_{1} \left ( \genfrac {.}{.}{0pt}{}{-n,n+\alpha + \beta +1}{\alpha +1};\frac{1-x}{2} \right ). $$

For α,β>−1, they form an orthogonal basis for $\mathcal{H}$;

The Jacobi polynomials are eigenfunctions of the Jacobi differential operator

$$\begin{aligned} L^{(\alpha ,\beta )} =& \bigl(1-x^2\bigr) \frac{d^2}{dx^2} + \bigl[\beta - \alpha -(\alpha +\beta +2)x\bigr]\frac{d}{dx}, \\ L^{(\alpha ,\beta )} P_n^{(\alpha ,\beta )} =& -n(n+\alpha +\beta +1)P_n^{(\alpha ,\beta )}. \end{aligned}$$

We define r(x)=1−x ². Then for x∈(−1,1), the polynomial r can be written as

$$ r(x) = K \frac{w^{(\alpha +1,\beta +1)}(x) }{w^{(\alpha ,\beta )}(x)}, \quad K= \frac{4(\alpha +1)(\beta +1)}{ (\alpha +\beta +2)(\alpha +\beta +3)}. $$

(3.1)

The differential operator T=T ^(α,β;κ) defined by (1.1) is related to L ^(α,β) by

$$ T^{(\alpha ,\beta ;\kappa )} =M(r) \bigl( L^{(\alpha +1,\beta +1)} +\rho\bigr), \quad\rho=\frac{1}{4} \bigl(\kappa ^2-(\alpha +\beta +3)^2 \bigr), $$

(3.2)

where M(r) denotes multiplication by r.

In [8, Sect. 3.1], it is shown that an operator of the form (3.2) acts as a five-term operator on a suitable basis of $\mathcal{H}$. In this case, the basis consists of Jacobi polynomials. We define $\phi_{n} = P_{n}^{(\alpha ,\beta )}/(h_{n}^{(\alpha ,\beta )})^{1/2}$, $n \in \mathbb{N}$. Then $\{\phi_{n}\}_{n \in \mathbb{N}}$ is an orthonormal basis for $\mathcal{H}^{(\alpha ,\beta )}$. We also define $\varPhi_{n} = P_{n}^{(\alpha +1,\beta +1)}/(h_{n}^{(\alpha +1,\beta +1)})^{1/2}$, $n \in \mathbb{N}$. Then $\{\varPhi_{n}\}_{n \in \mathbb{N}}$ is an orthonormal basis for $\mathcal{H}^{(\alpha +1,\beta +1)}$. In order to write T explicitly as a five-diagonal operator on the basis $\{\phi_{n}\}_{n \in \mathbb{N}}$, we need a connection formula between $\{\phi_{n}\}_{n \in \mathbb{N}}$ and $\{ \varPhi_{n}\}_{n \in \mathbb{N}}$.

Lemma 3.1

The following connection formula holds:

$$\phi_n = \alpha _n \varPhi_n + \beta _n \varPhi_{n-1}+ \gamma _n \varPhi_{n-2}, $$

where

$$\begin{aligned} \alpha _n & = \frac{2}{\sqrt{K}} \frac{1}{2n+\alpha +\beta +2}\\ &\quad {}\times \sqrt{ \frac{ (\alpha +n+1)(\beta +n+1)(n+\alpha +\beta +1)(n+\alpha +\beta +2) }{ (\alpha +\beta +2n+1)(\alpha +\beta +2n+3)}}, \\ \beta _n &= (-1)^{n} \frac{2}{\sqrt{K}} \frac{(\beta -\alpha ) \sqrt{n(n+\alpha +\beta +1)}}{(\alpha +\beta +2n)(\alpha +\beta +2n+2)}, \\ \gamma _n & = - \frac{2}{\sqrt{K}} \frac{1}{2n+\alpha +\beta } \sqrt{ \frac{n(n-1)(\alpha +n)(\beta +n)}{(\alpha +\beta +2n-1)(\alpha +\beta +2n+1)} }. \end{aligned} $$

Proof

There exists an expansion $\phi_{n} = \sum_{k = 0}^{n} a_{n,k} \varPhi_{k}$, where

$$a_{n,k} = \langle\phi_n ,\varPhi_k \rangle_{\alpha +1,\beta +1} = K^{-1} \langle\phi_n,r \varPhi_k \rangle_{\alpha ,\beta }. $$

Since r has degree 2, it follows from the orthogonality relations for ϕ _n that a _n,k =0 for 0≤k≤n−3.

We compute the three remaining coefficients. The value of a _n,n follows from comparing leading coefficients; $a_{n,n} = \frac{\mathrm {lc}(\phi_{n})}{\mathrm {lc}(\varPhi_{n})}$. We have

$$\mathrm {lc}(\phi_n) = 2^{-n} (n+\alpha +\beta +1)_n \sqrt{ \frac{2n+\alpha +\beta +1}{\alpha +\beta +1} \frac{(\alpha +\beta +1)_{n} }{n! (\alpha +1,\beta +1)_n}} $$

and lc(Φ _n ) is obtained by replacing (α,β) by (α+1,β+1), which leads to the result for α _n =a _n,n .

For a polynomial p of degree n, let k(p) denote the coefficient of (1−x)ⁿ⁻¹. Then

$$a_{n,n-1} = \frac{ k(\phi_n) - a_{n,n} k(\varPhi_n) }{\mathrm {lc}(\varPhi_{n-1})}. $$

We have

$$k(\phi_n) = (-1)^{n+1} \mathrm {lc}(\phi_n) \frac{2 n(\alpha +n)}{\alpha +\beta +2n}, $$

and k(Φ _n ) is obtained by replacing (α,β) by (α+1,β+1), which gives the expression for β _n =a _n,n−1.

Finally, the expression for γ _n =a _n,n−2 follows from

$$a_{n,n-2} = K^{-1}\frac{\mathrm {lc}(r\varPhi_{n-2})}{ \mathrm {lc}(\phi_n)} =-K^{-1} \frac{\mathrm {lc}(\varPhi_{n-2})}{ \mathrm {lc}(\phi_n)}. $$

 □

We now use [8, Lemma 3.1] to write the differential operator T as a five-diagonal operator.

Proposition 3.2

The operator T defined by (1.1) acts as a five-diagonal operator on the basis $\{\phi_{n}\}_{n \in \mathbb{N}}$ of $\mathcal{H}$ by

$$ T \phi_n = a_n \phi_{n+2} + b_n\phi_{n+1} +c_n \phi_n + b_{n-1} \phi_{n-1} + a_{n-2} \phi_{n-2}, $$

(3.3)

with coefficients given by

$$\begin{aligned} a_n =& K \alpha _n \gamma _{n+2} (\varLambda _n + \rho), \qquad b_n=K \alpha _n \beta _{n+1}( \varLambda _n+\rho)+ K \beta _n \gamma _{n+1} ( \varLambda _{n+1}+\rho), \\ c_n =& K \alpha _n^2 (\varLambda _n+ \rho) + K \beta _n^2(\varLambda _{n-1}+\rho) + K \gamma _n^2 (\varLambda _{n-2}+\rho), \end{aligned}$$

where Λ _n =−n(n+α+β+3), K, ρ are given by (3.1), (3.2), and α _n ,β _n ,γ _n are as in Lemma 3.1.

One easily verifies that a ₋₁=a ₋₂=b ₋₁=0. Furthermore, we have the factorization

$$\varLambda _n+\rho= - \biggl(n+\frac{1}{2}(\alpha +\beta +3+\kappa ) \biggr) \biggl(n+ \frac{1}{2}(\alpha +\beta +3-\kappa ) \biggr). $$

3.2 Matrix-Valued Orthogonal Polynomials

From Theorem 2.2 and Proposition 3.2, it follows that the functions $\mathcal{F}\phi_{n}$, $n \in \mathbb{N}$, satisfy the five-term recurrence relation

$$\begin{aligned} \lambda (\mathcal{F} \phi_n) ( \lambda ) =& a_n (\mathcal{F}\phi_{n+2}) (\lambda ) + b_n ( \mathcal{F}\phi_{n+1}) (\lambda ) +c_n (\mathcal{F} \phi_n) (\lambda ) \\ &{} + b_{n-1} (\mathcal{F}\phi_{n-1}) ( \lambda ) + a_{n-2} (\mathcal{F}\phi_{n-2}) (\lambda ), \end{aligned}$$

(3.4)

for almost all λ∈σ. Furthermore, the set $\{\mathcal{F}\phi _{n}\}_{n \in \mathbb{N}}$ is an orthonormal basis for $L^{2}(\mathcal{V})$. We can determine an explicit expression for $\mathcal{F}\phi_{n}$ in terms of hypergeometric functions.

Lemma 3.3

For $n \in \mathbb{N}$, let F _n (δ,η)=F _n (δ,η;α,β,κ) denote the series

$$\begin{aligned} F_n(\delta ,\eta) =& D_n \sum_{l=0}^n \frac{ (-n,n+\alpha +\beta +1,\frac{1}{2}(\alpha +\delta +1))_l }{ l! (\alpha +1,\frac{1}{2}(\alpha +\beta +\eta+\delta +2))_l} \\ &\hphantom{D_n \sum_{l=0}^n} {} \times{} _{3}F_{2} \left ( \genfrac {.}{.}{0pt}{}{ \frac{1}{2}(1+\delta +\eta+\kappa ),\frac{1}{2}(1+\delta +\eta-\kappa ),\frac{1}{2}( \beta +\eta+1)}{1+\eta,\frac{1}{2}(\alpha +\beta +\eta+\delta +2+2l)};1 \right ), \end{aligned}$$

with

$$\begin{aligned} D_n =& \frac{1}{2} \sqrt{ \frac{ 2n + \alpha +\beta +1 }{\alpha +\beta +1} \frac{ (\alpha +\beta +1, \alpha +1)_n }{n! (\beta +1)_n } }\\ &{}\times \frac{ \varGamma (\alpha +\beta +2,\frac{1}{2}(\alpha +\delta +1),\frac{1}{2}(\beta +\eta+1))}{\varGamma (\alpha +1,\beta +1,\frac{1}{2}(\alpha +\beta +\eta+\delta +2))}. \end{aligned}$$

Then, for λ∈σ,

$$(\mathcal{F}\phi_n) (\lambda ) = \begin{cases} \begin{pmatrix} F_n(\delta _\lambda , \eta_\lambda ) \\ F_n(\delta _\lambda ,-\eta_\lambda ) \end{pmatrix} , & \lambda \in \varOmega _2,\\ F_n(\delta _\lambda ,\eta(\lambda )), & \lambda \in \varOmega _1,\\ F_n(\delta (\lambda ),\eta(\lambda )), & \lambda \in \varOmega _d. \end{cases} $$

The above ₃ F ₂-series converges absolutely if ℜ(α−δ+1+2l)>0, which is the case if λ∈Ω ₁∪Ω ₂. For λ∈Ω _d , the ₃ F ₂-series terminates.

Proof

We compute

$$\begin{aligned} I_n =& \int_{-1}^1 \biggl(\frac{1-x}{2} \biggr)^{-\frac{1}{2}(\alpha -\delta +1)} \biggl(\frac{1+x}{2} \biggr)^{-\frac{1}{2}(\beta -\eta+1)}\\ & \hphantom{\int_{-1}^1} {}\times {} _{2}F_{1} \left ( \genfrac {.}{.}{0pt}{}{\frac{1}{2}(1+\delta +\eta+\kappa ), \frac{1}{2}(1+\delta +\eta-\kappa )}{1+\eta};\frac{1+x}{2} \right ) \\ & \hphantom{\int_{-1}^1} {}\times{} _{2}F_{1} \left ( \genfrac {.}{.}{0pt}{}{-n,n+\alpha +\beta +1}{\alpha +1};\frac{1-x}{2} \right ) w^{(\alpha ,\beta )}(x)\,dx. \end{aligned}$$

Interchanging the order of summation and integration, and using the Beta-integral, we obtain

$$\begin{aligned} I_n =& C_n \sum _{l=0}^n \sum_{m=0}^\infty \frac{ (-n,n+\alpha +\beta +1)_l }{2^l l! (\alpha +1)_l } \frac{ (\frac{1}{2}(1+\delta +\eta+\kappa ),\frac{1}{2}(1+\delta +\eta-\kappa ))_m }{2^m m! (1+\eta)_m } \\ &{}\times\int_{-1}^1 (1-x)^{\frac{1}{2}(\alpha +\delta -1)+l} (1+x)^{\frac{1}{2}(\beta +\eta-1)+m} \,dx \\ =& C_n' \sum_{l=0}^n \sum_{m=0}^\infty\frac{ (-n,n+\alpha +\beta +1,\frac{1}{2}(\alpha +\delta +1)_l }{ l! (\alpha +1)_l } \\ &{}\times\frac{ (\frac{1}{2}(1+\delta +\eta+\kappa ),\frac{1}{2}(1+\delta +\eta-\kappa ),\frac{1}{2}(\beta +\eta+1))_m }{ m! (1+\eta)_m \, (\frac{1}{2}(\alpha +\beta +\eta+\delta +2))_{l+m}}, \end{aligned}$$

where

$$C_n' = \frac{1}{2} \frac{ \varGamma (\alpha +\beta +2,\frac{1}{2}(\alpha +\delta +1),\frac{1}{2}(\beta +\eta +1))}{\varGamma (\alpha +1,\beta +1,\frac{1}{2}(\alpha +\beta +\eta+\delta +2))}. $$

Now the result follows from the explicit expressions for ϕ _n and the integral transform $\mathcal{F}$. □

From Lemma 3.3, it follows that

$$F_0(\delta ,\eta) = D_0 {}_{3}F_{2} \left ( \genfrac {.}{.}{0pt}{}{\frac{1}{2}(1+\delta +\eta+ \kappa ),\frac{1}{2}(1+\delta +\eta-\kappa ),\frac{1}{2}(\beta +\eta+1)}{1+\eta, \frac{1}{2}(\alpha +\beta +\eta+ \delta +2)};1 \right ), $$

and

$$\begin{aligned} F_1(\delta ,\eta) =& D_1 \biggl[ {}_{3}F_{2} \left ( \genfrac {.}{.}{0pt}{}{\frac{1}{2}(1+\delta + \eta+\kappa ),\frac{1}{2}(1+\delta +\eta-\kappa ),\frac{1}{2}(\beta +\eta+1)}{1+\eta, \frac{1}{2}(\alpha +\beta + \eta+\delta +2)};1 \right ) \\ &{} - \frac{(\alpha +\beta +2)(\alpha +\delta +1)}{(\alpha +1)(\alpha +\beta +\eta+\delta +2)}\\ &{}\times {}_{3}F_{2} \left ( \genfrac {.}{.}{0pt}{}{\frac{1}{2}(1+\delta +\eta+\kappa ),\frac{1}{2}(1+\delta +\eta- \kappa ),\frac{1}{2}(\beta +\eta+1)}{1+\eta, \frac{1}{2}(\alpha +\beta +\eta+\delta +4)};1 \right ) \biggr]. \end{aligned}$$

These two functions and the five-term recurrence relation from (3.4) completely determine the functions $\mathcal{F}\phi_{n}$.

We define 2×2-matrix-valued orthogonal polynomials P _n , $n \in \mathbb{N}$, by the three-term recurrence relations

$$\begin{aligned} &\lambda P_n(\lambda ) = A_n P_{n+1}(\lambda ) + B_n P_{n}(\lambda ) + A_{n-1}^* P_{n-1}(\lambda ), \quad \lambda \in \sigma , \\ &\quad A_n = \begin{pmatrix} a_{2n} & 0 \\ b_{2n+1} & a_{2n+1} \end{pmatrix} , \quad B_n = \begin{pmatrix} c_{2n} & b_{2n} \\ b_{2n} & c_{2n+1} \end{pmatrix}, \end{aligned}$$

(3.5)

and the initial conditions P ₋₁(λ)=0 and P ₀(λ)=I. The matrix elements a _n , b _n , and c _n are given in Proposition 3.2. From the five-term recurrence relation (3.4), we obtain, for $m \in \mathbb{N}$,

$$ \begin{array}{l@{\quad}l} \begin{pmatrix} \mathcal{F}\phi_{2m}(\lambda ) \\ \mathcal{F}\phi_{2m+1}(\lambda ) \end{pmatrix} = P_m(\lambda ) \begin{pmatrix} \mathcal{F}\phi_0(\lambda )\\ \mathcal{F}\phi_1(\lambda ) \end{pmatrix} & \text{if}\ \lambda \in \varOmega _1 \cup \varOmega _d, \\ \begin{pmatrix} \mathcal{F}\phi_{2m}(\lambda )^t \\ \mathcal{F}\phi _{2m+1}(\lambda )^t \end{pmatrix} = P_m(\lambda ) \begin{pmatrix} \mathcal{F}\phi_0(\lambda )^t\\ \mathcal{F}\phi_1(\lambda )^t \end{pmatrix} & \text{if}\ \lambda \in \varOmega _2. \end{array} $$

(3.6)

The orthogonality relations for $\mathcal{F} \phi_{n}$, $n \in \mathbb{N}$, can now be reformulated as orthogonality relations for the matrix-valued polynomials P _n , see [8, Theorem 2.1].

Theorem 3.4

The 2×2-matrix-valued orthogonal polynomials P _n , $n \in \mathbb{N}$, defined by (3.5) satisfy the orthogonality relations

$$\begin{aligned} \delta _{mn}I&= \frac{1}{2\pi D} \int _{\varOmega _2} P_m(\lambda ) W_2(\lambda ) P_n(\lambda )^* \frac{ d\lambda }{-i\eta_\lambda } \\ &\quad {}+ \frac{1}{2\pi D} \int_{\varOmega _1} P_m(\lambda ) W_1(\lambda ) P_n(\lambda )^* v(\lambda ) \frac{ d\lambda }{-i\delta _\lambda }\\ &\quad {} + \frac{1}{D} \sum_{\lambda \in \varOmega _d} P_m(\lambda ) W_1(\lambda ) P_n(\lambda )^* N_{\lambda }, \end{aligned} $$

with

$$\begin{aligned} W_1(\lambda ) =& \begin{pmatrix} |(\mathcal{F} \phi_0)(\lambda )|^2 & (\mathcal{F} \phi_0)(\lambda )\overline{(\mathcal{F} \phi_1)(\lambda )} \\ \overline{(\mathcal{F} \phi_0)(\lambda )}(\mathcal{F} \phi_1)(\lambda ) & |(\mathcal{F} \phi_1)(\lambda )|^2 \end{pmatrix} , \\ W_2(\lambda ) =& \begin{pmatrix} \langle(\mathcal{F} \phi_0)(\lambda ), (\mathcal{F} \phi_0)(\lambda ) \rangle _{V(\lambda )} & \langle(\mathcal{F} \phi_0)(\lambda ), (\mathcal{F} \phi_1)(\lambda ) \rangle_{V(\lambda )} \\ \langle(\mathcal{F} \phi_1)(\lambda ), (\mathcal{F} \phi_0)(\lambda ) \rangle _{V(\lambda )} & \langle(\mathcal{F} \phi_1)(\lambda ), (\mathcal{F} \phi_1)(\lambda ) \rangle_{V(\lambda )} \end{pmatrix} , \end{aligned}$$

where 〈x,y〉 _V(λ)=x ^∗ V(λ)y. Here $\varOmega _{1},\varOmega _{2},\varOmega _{d} \subset \mathbb{R}$ are as in (2.2) and (2.5), the functions δ _λ and η _λ are as defined in (2.3), the weight functions v,V and N are defined by respectively (2.7), (2.8), and (2.10), and the constant D is given in (2.11).

Remark 3.5

In [8, Proposition 3.6], a q-analog of Theorem 3.4 is considered. The functions ϕ _n in this case are the little q-Jacobi polynomials, and the integral transform $\mathcal{F}$ is simply the integral transform corresponding to the continuous dual q-Hahn polynomials. It would be very interesting to see if similar results can be obtained for other q-analogs of the Jacobi polynomials, such as big q-Jacobi polynomials [1], Askey–Wilson polynomials [3], and Ruijsenaars’ R-function [18].

3.3 The Special Case α=β

We assume α=β, and for convenience we also assume Ω _d =∅. In this case, Ω ₁=∅, and δ _λ =η _λ for all λ∈Ω ₂. The spectral decomposition of T can now be obtained in a different way.

The coefficient b _n in the five-diagonal expression for T vanishes, so T reduces to a tridiagonal operator or Jacobi operator. Explicitly,

$$T\phi_n = a_n \phi_{n+2} + c_n \phi_n + a_{n-2} \phi_{n-2}, $$

with

$$\begin{aligned} a_n &= \frac{(n+\frac{1}{2}(\alpha +\beta +3+\kappa ))(n+\frac{1}{2}(\alpha +\beta +3-\kappa ))}{2n+2\alpha +3}\\ &\quad {}\times \sqrt{ \frac{(n+2)(n+1)(n+2\alpha +1)(n+2\alpha +2)}{(2n+2\alpha +1)(2n+2\alpha +5)}}, \\ c_n &= - \frac{(n+2\alpha +1)(n+2\alpha +2)(n+\frac{1}{2}(\alpha +\beta +3+\kappa ))(n+\frac{1}{2}(\alpha +\beta +3-\kappa ))}{(2n+2\alpha +1)(2n+2\alpha +3)} \\ & \quad{}- \frac{n(n-1)(n+\frac{1}{2}(\alpha +\beta -1+\kappa ))(n+\frac{1}{2}(\alpha +\beta -1-\kappa ))}{(2n+2\alpha -1)(2n+2\alpha +1)}. \end{aligned} $$

The spectral decomposition can be described with the help of the orthonormal Wilson polynomials [2, 19], which are defined by

$$\begin{aligned} W_n\bigl(x^2;a,b,c,d\bigr) & = \sqrt{ \frac{(a+b,a+c,a+d)_n (a+b+c+d)_{2n} }{n!(b+c,b+d,c+d,n+a+b+c+d-1)_n }} \\ & \quad{}\times{} _{4}F_{3} \left ( \genfrac {.}{.}{0pt}{}{-n,n+a+b+c+d-1,a+ix,a-ix}{a+b,a+c,a+d};1 \right ). \end{aligned} $$

If a,b,c,d>0, these polynomials are orthonormal with respect to an absolutely continuous measure on (0,∞).

For $m \in \mathbb{N}$, we define

$$\begin{aligned} W_m^\mathrm{e}(x) &= W_m \bigl((2x)^2;\tfrac{1}{2}(\alpha +1),\tfrac{1}{2}(\alpha +1),\tfrac{1}{4}(1+\kappa ), \tfrac{1}{4}(1-\kappa ) \bigr), \\ W_m^\mathrm{o}(x) &= W_m \bigl((2x)^2; \tfrac{1}{2}(\alpha +1),\tfrac{1}{2}(\alpha +1),\tfrac{1}{4}(3+\kappa ),\tfrac{1}{4}(3-\kappa ) \bigr). \end{aligned} $$

Then we obtain from the three-term recurrence relation for the Wilson polynomials,

$$\begin{aligned} - \bigl((\alpha +1)^2+x^2 \bigr) W_m^\mathrm{e}(x)& = a_{2m} W_{m+1}^\mathrm{e}(x) + c_{2m}W_m^\mathrm{e}(x) + a_{2m-2} W_{m-1}^\mathrm{e}(x), \\ - \bigl((\alpha +1)^2+x^2 \bigr) W_m^\mathrm{o}(x)& = a_{2m+1} W_{m+1}^\mathrm{o}(x) + c_{2m+1}W_m^\mathrm{o}(x) + a_{2m-1} W_{m-1}^\mathrm{o}(x). \end{aligned} $$

Let μ ^e and μ ^o denote the orthogonality measures for the Wilson polynomials $W_{m}^{\mathrm{e}}$ and $W_{m}^{\mathrm{o}}$. The unitary operator $U:\mathcal{H} \to L^{2}(\mu^{\mathrm{e}})\oplus L^{2}(\mu ^{\mathrm{o}})$, given by

$$ U \phi_n = \begin{cases} W_{m}^\mathrm{e}& \text{if}\ n=2m,\\ W_m^\mathrm{o} & \text{if}\ n=2m+1, \end{cases} $$

(3.7)

satisfies UT=MU, where M is multiplication by −((α+1)²+x ²). So T indeed has continuous spectrum (−∞,−(α+1)²)=Ω ₂, with multiplicity 2.

The Hilbert space $\mathcal{H}^{(\alpha ,\alpha )}$ can also be split up in a natural way. From (1.1), we see that T (with α=β) leaves invariant the subspaces of even/odd functions, so we can split $\mathcal{H}$ accordingly into $\mathcal{H}^{\mathrm{e}}$ and $\mathcal{H}^{\mathrm{o}}$. The Jacobi (Gegenbauer) polynomials ϕ _2m (x) are even polynomials; hence they form an orthonormal basis for $\mathcal{H}^{\mathrm{e}}$, and by a quadratic transformation they can be transformed into multiples of $P_{m}^{(\alpha ,-\frac{1}{2})}(2x^{2}-1)$. Similarly, the odd polynomials ϕ _2m+1(x) form an orthonormal basis for $\mathcal{H}^{\mathrm{o}}$, and they can transformed into multiples of $xP_{m}^{(\alpha ,\frac{1}{2})}(2x^{2}-1)$. Obviously there are similar transformations for the Jacobi polynomials Φ _2m and Φ _2m+1. Now the operator T restricted to $\mathcal{H}^{\mathrm{e}}$ or $\mathcal{H}^{\mathrm{o}}$ can be treated as in [12, Sect. 3]. The unitary operator U is given in each case by a Jacobi function transform, see Remark 2.3, so that we obtain a special case of Koornwinder’s formula [16] stating that Jacobi polynomials are mapped to Wilson polynomials by the Jacobi function transform. In this light, (3.6) can be considered as a matrix-analog of Koornwinder’s formula.

Remark 3.6

There exists an extension of Koornwinder’s formula on the level of Wilson polynomials [6, Theorem 6.7]: Wilson polynomials are mapped to Wilson polynomials by the Wilson function transform. It would be interesting to see if there also exists a matrix-analog of this formula.

4 Proofs for Sect. 2

In this section, we perform the spectral analysis of the second-order differential operator T defined by (1.1), considered as an unbounded operator on the Hilbert space $\mathcal{H}$ defined in the beginning of Sect. 2.

4.1 Eigenfunctions

The eigenvalue equation Tf _λ =λf _λ is a second order differential equation with regular singular points at 1,−1 and ∞, so it has hypergeometric (i.e., ₂ F ₁) solutions. We first determine these solutions.

We define for $\lambda \in \mathbb{C}\setminus (\varOmega _{1}\cup \varOmega _{2} )$, see (2.2), the functions

$$\begin{aligned} \phi^-_\lambda (x) & = \biggl(\frac{1-x}{2} \biggr)^{-\frac{1}{2}(\alpha +\delta (\lambda )+1)} \biggl(\frac{1+x}{2} \biggr)^{-\frac{1}{2}(\beta +\eta(\lambda )+1)} \\ & \quad{}\times{} _{2}F_{1} \left ( \genfrac {.}{.}{0pt}{}{ \frac{1}{2}\bigl(1-\delta (\lambda )-\eta(\lambda )-\kappa \bigr ), \frac{1}{2}\bigl(1-\delta (\lambda )-\eta(\lambda )+\kappa \bigr)}{1-\eta(\lambda )};\frac{1+x}{2} \right ), \\ \phi^+_\lambda (x) & = \biggl(\frac{1-x}{2} \biggr)^{-\frac{1}{2}(\alpha +\delta (\lambda )+1)} \biggl(\frac{1+x}{2} \biggr)^{-\frac{1}{2}(\beta -\eta(\lambda )+1)} \\ & \quad{}\times{} _{2}F_{1} \left ( \genfrac {.}{.}{0pt}{}{ \frac{1}{2}\bigl(1-\delta (\lambda )+\eta(\lambda )-\kappa \bigr ), \frac{1}{2}\bigl(1-\delta (\lambda )+\eta(\lambda )+\kappa \bigr)}{1+\eta(\lambda )};\frac {1+x}{2} \right ), \end{aligned} $$

and

$$\begin{aligned} \psi^-_\lambda (x) =& \biggl(\frac{1-x}{2} \biggr)^{-\frac{1}{2}(\alpha +\delta (\lambda )+1)} \biggl(\frac{1+x}{2} \biggr)^{-\frac{1}{2}(\beta +\eta(\lambda )+1)} \\ &{}\times{} _{2}F_{1} \left ( \genfrac {.}{.}{0pt}{}{ \frac{1}{2}\bigl(1-\delta (\lambda )-\eta(\lambda )-\kappa \bigr ), \frac{1}{2}\bigl(1-\delta (\lambda )-\eta(\lambda )+\kappa \bigr)}{1-\delta (\lambda )};\frac{1-x}{2} \right ), \\ \psi^+_\lambda (x) =& \biggl(\frac{1-x}{2} \biggr)^{-\frac{1}{2}(\alpha -\delta (\lambda )+1)} \biggl(\frac{1+x}{2} \biggr)^{-\frac{1}{2}(\beta +\eta(\lambda )+1)} \\ &{}\times{} _{2}F_{1} \left ( \genfrac {.}{.}{0pt}{}{ \frac{1}{2}\bigl(1+\delta (\lambda )-\eta(\lambda )-\kappa \bigr ), \frac{1}{2}\bigl(1+\delta (\lambda )-\eta(\lambda )+\kappa \bigr)}{1+\delta (\lambda )};\frac {1-x}{2} \right ), \end{aligned}$$

where δ and η are defined by (2.3). From Euler’s transformation for ₂ F ₁-series, it follows that $\phi^{\pm}_{\lambda }$ is invariant under δ(λ)↦−δ(λ); similarly, $\psi^{\pm}_{\lambda }$ is invariant under η(λ)↦−η(λ). Note that $\psi_{\lambda }^{\pm}(x)$ is obtained from $\phi_{\lambda }^{\pm}(x)$ by the substitution (α,β,x)↦(β,α,−x), and vice versa. Note also that the ₂ F ₁-series in $\phi^{\pm}_{\lambda }$ is summable at x=1 if ℜ(δ(λ))>0, and that the ₂ F ₁-series in $\psi^{\pm}_{\lambda }$ is summable at x=−1 if ℜ(η(λ))>0.

Remark 4.1

From here on we will just write δ and η, instead of δ(λ) and η(λ).

Proposition 4.2

The functions $\phi_{\lambda }^{\pm}$, $\psi_{\lambda }^{\pm}$ are solutions of the eigenvalue equation Tf=λf.

Proof

Suppose f is a solution of the eigenvalue equation Tf=λf. A calculation shows that if $f(x) = (1-x)^{-\frac{1}{2}(\alpha +\delta +1)}(1+x)^{-\frac{1}{2}(\beta +\eta+1)}\phi(x)$, then ϕ satisfies

$$\begin{aligned} &\bigl(1-x^2\bigr) \phi''(x)+\bigl[\delta - \eta+(\delta +\eta-2)x\bigr]\phi'(x)\\ &\quad {} -\frac{1}{4} (1-\eta-\delta -\kappa ) (1-\eta -\delta +\kappa ) \phi(x)=0. \end{aligned}$$

Now set $t=\frac{1}{2}(1+x)$. Then

$$\begin{aligned} &t(1-t)\frac{d^2\phi}{dt^2} + \biggl[(1-\eta)- \biggl(1+\frac{1}{2}(1-\delta -\eta-\kappa )+ \frac{1}{2}(1-\delta -\eta+\kappa ) \biggr)t \biggr]\frac{d\phi}{dt} \\ &\quad {}- \frac{1}{4} (1-\eta-\delta -\kappa ) (1-\eta-\delta +\kappa ) \phi= 0. \end{aligned}$$

This is the hypergeometric differential equation (see, e.g., [2, Chap. 2]) with coefficients

$$a=\frac{1}{2}(1-\delta -\eta-\kappa ), \qquad b=\frac{1}{2}(1-\delta -\eta+ \kappa ), \qquad c = 1-\eta. $$

The ₂ F ₁-functions in $\phi^{-}_{\lambda }, \psi^{-}_{\lambda }$ are well-known solutions of this differential equation, so $\phi^{-}_{\lambda }, \psi^{-}_{\lambda }$ are solutions of the eigenvalue equation. The proof for $\phi^{+}_{\lambda }$ and $\psi^{+}_{\lambda }$ is similar. □

For later reference, we need connection formulas for $\phi_{\lambda }^{\pm}$ and $\psi_{\lambda }^{\pm}$.

Proposition 4.3

For c defined by (2.6),

$$\begin{aligned} \psi^\pm_\lambda (x) & = c(\eta;\pm \delta ) \phi^+_\lambda (x) + c(-\eta;\pm \delta )\phi^-_\lambda (x), \end{aligned}$$

(4.1)

$$\begin{aligned} \phi^\pm_\lambda (x) & = c(\delta ;\pm\eta) \psi^+_\lambda (x) + c(-\delta ;\pm\eta)\psi^-_\lambda (x). \end{aligned}$$

(4.2)

Proof

This follows from a three-term transformation for ₂ F ₁-functions, see, e.g., [2, (2.3.11)]. □

The following identities for the c-function turn out to be useful.

Lemma 4.4

The c-function defined by (2.6) satisfies:

(i):

$c(x;y) = -\frac{y}{x} c(-y;-x)$,

(ii):

$c(x;y)c(-x;-y) - c(x;-y)c(-x;y)=-\frac{y}{x}$.

Remark 4.5

Using the reflection equation for the Γ-function, Lemma 4.4(ii) is equivalent to the trigonometric identity

$$\begin{aligned} \sin(\pi x)\sin(\pi y) =& \sin\biggl(\frac{\pi}{2}(y-x+\kappa )\biggr)\sin \biggl(\frac{\pi}{2}(y-x-\kappa )\biggr)\\ &{} - \sin\biggl(\frac{\pi}{2}(-y-x+ \kappa )\biggr)\sin\biggl(\frac{\pi}{2}(-y-x-\kappa )\biggr), \end{aligned}$$

and can also be proved in this way.

Proof

The first identity follows from Γ(z+1)=zΓ(z).

For the second identity, we note that Proposition 4.3 implies

$$ \begin{pmatrix} c(\delta ;\eta) & c(-\delta ;\eta) \\ c(\delta ;-\eta) & c(-\delta ;-\eta) \end{pmatrix} = \begin{pmatrix} c(\eta;\delta ) & c(-\eta;\delta ) \\ c(\eta;-\delta ) & c(-\eta;-\delta ) \end{pmatrix} ^{-1}, $$

which in turn implies

$$c(\delta ;\eta) = \frac{ c(-\eta;-\delta ) }{c(\eta;\delta )c(-\eta;-\delta ) - c(\eta ;-\delta )c(-\eta;\delta )}. $$

Applying the first identity to the numerator then gives

$$c(\eta;\delta )c(-\eta;-\delta ) - c(\eta;-\delta )c(-\eta;\delta )=-\frac{\delta }{\eta}, $$

which proves the second identity. □

We also need the behavior of the eigenfunctions near the endpoints −1 and 1.

Lemma 4.6

For x↓−1, we have

$$\phi^\pm_\lambda (x) = \biggl(\frac{1+x}{2} \biggr)^{-\frac{1}{2}(\beta \mp\eta+ 1)} \bigl(1+\mathcal{O}(1+x) \bigr). $$

For x↑1, we have

$$\psi^\pm_\lambda (x) = \biggl(\frac{1-x}{2} \biggr)^{-\frac{1}{2}(\alpha \mp \delta + 1)} \bigl(1+\mathcal{O}(1-x) \bigr). $$

Proof

This is straightforward from the explicit expressions as ₂ F ₁-series. □

Remark 4.7

Observe that the function $|\phi^{\pm}_{\lambda }|^{2} w^{(\alpha ,\beta )}$ is in L ¹(−1,0) if and only if ±ℜ(η)>0. Furthermore, $|\psi^{\pm}_{\lambda }|^{2} w^{(\alpha ,\beta )}$ is in L ¹(0,1) if and only if ±ℜ(δ)>0.

4.2 Spectral Analysis

We determine the spectrum and the spectral decomposition of T.

For functions f,g that are differentiable at a point x∈(−1,1), we define

$$[f,g](x)= p(x)W(f,g) (x), $$

where

$$p(x) = C (1-x)^{\alpha +2}(1+x)^{\beta +2} , \quad C=2^{-\alpha -\beta -1} \frac{\varGamma (\alpha +\beta +2)}{\varGamma (\alpha +1,\beta +1) }, $$

and W(f,g) denotes the Wronskian

$$W(f,g) (x)=f'(x)g(x)-f(x)g'(x). $$

For 1≤a<b≤1 we denote by $\mathcal{D}(a,b)$ the subspace of L ²((a,b),w ^(α,β)(x) dx) consisting of functions f such that:

• f is continuously differentiable on (a,b),

• f′ is absolutely continuous on (a,b),

• Tf∈L ²((a,b),w ^(α,β)(x) dx).

Note that $\mathcal{D}(a,b)$ is dense in L ²((a,b),w ^(α,β)(x) dx).

Lemma 4.8

Let 1≤a<b≤1 and $f,g \in\mathcal{D}(a,b)$. Then

$$\int_{a}^b \bigl( (Tf) (x) \overline{g(x)} - f(x) \overline{ (Tg) (x) } \bigr) w^{(\alpha ,\beta )}(x)\, dx =[f,\bar{g}](b)-[f,\bar{g}](a). $$

Proof

We write the differential operator T as

$$T = (1-x)^{-\alpha }(1+x)^{-\beta } \frac{d}{dx} \biggl((1-x)^{\alpha +2}(1+x)^{\beta +2} \frac{d}{dx} \biggr) + \rho \bigl(1-x^2\bigr). $$

Then it follows that

$$\begin{aligned} &\int_{a}^b \bigl( (Tf) (x) \overline{g(x)} - f(x) \overline{ (Tg) (x) } \bigr) w^{(\alpha ,\beta )}(x)\, dx \\ &\quad =\int_{a}^b \biggl[\overline{g(x)} \frac{d}{dx} \bigl(p(x) f'(x) \bigr) - f(x)\overline{ \frac{d}{dx} \bigl(p(x) g'(x) \bigr)} \biggr]\,dx. \end{aligned} $$

Using integration by parts, this is equal to

$$\bigl[p(x) f'(x)\overline{g(x)} - p(x)f(x)\overline{g'(x)} \bigr]_{a}^b, $$

which gives the result. □

Let $\mathcal{D}_{0} \subset\mathcal{H}$ consist of the functions in $\mathcal{D}(-1,1)$ with support on a compact interval in (−1,1).

Proposition 4.9

The densely defined operator $(T,\mathcal{D}_{0})$ is symmetric.

Proof

Clearly, we have $\lim_{x \downarrow-1}[f,\bar{g}](x) = \lim_{x\uparrow 1}[f,\bar{g}](x)=0$ for $f,g \in\mathcal{D}_{0}$. Then the result follows from Lemma 4.8. □

The function x↦[f,g](x), x∈(−1,1), is constant if f and g are solutions of the eigenvalue equations Ty=λy. In the following lemma, we determine the value of the constant in the case of the eigenfunctions $\psi_{\lambda }^{\pm}$ en $\phi_{\lambda }^{\pm}$.

Lemma 4.10

For $\lambda \in \mathbb{C}\setminus(-\infty,-(\alpha +1)^{2})$,

$$\bigl[\phi^-_\lambda ,\phi^+_{\lambda }\bigr] = -\eta D\quad \mbox{\textit{and}} \quad\bigl[\psi^+_{\lambda },\phi^+_{\lambda }\bigr]= -\eta D\, c(-\eta;\delta ), $$

where $D=2^{\alpha +\beta +3}C=\frac{4\varGamma (\alpha +\beta +2)}{\varGamma (\alpha +1,\beta +1) }$ (see also (2.11)).

Note that $[\psi_{\lambda }^{-},\psi_{\lambda }^{+}]$ and $[\psi_{\lambda }^{-},\phi_{\lambda }^{-}]$ can be obtained from Lemma 4.10 using (α,β,x)↦(β,α,−x) and (δ,η)↦(−δ,−η), respectively.

Proof

We have

$$\bigl[\phi^-_\lambda ,\phi^+_{\lambda }\bigr] = C \lim _{x\downarrow-1} (1-x)^{\alpha +2} (1+x)^{\beta +2} \biggl( \frac{d\phi^-_\lambda }{dx}(x) \phi^+_{\lambda }(x) - \phi^-_\lambda (x) \frac{d\phi^+_{\lambda }}{dx}(x) \biggr). $$

Using

$$\frac{d\phi^\pm_\lambda }{dx}(x) = -\frac{1}{4}(\beta \mp\eta+1) \biggl(\frac {1+x}{2} \biggr)^{-\frac{1}{2}(\beta \mp\eta+1)-1} \bigl(1+\mathcal{O}(1+x) \bigr), \quad x \downarrow-1, $$

we find

$$\bigl[\phi^-_\lambda ,\phi^+_{\lambda }\bigr] = -\eta D. $$

Now from the connection formula (4.1), we obtain

$$\bigl[\psi^+_\lambda ,\phi^+_{\lambda }\bigr]=c(-\eta;\delta ) \bigl[ \phi^-_\lambda ,\phi^+_{\lambda }\bigr] = -\eta D\, c(-\eta;\delta ). $$

 □

Let us mention that from the explicit formula (2.6) for c(−η;δ) and Lemma 4.10, it follows that $[\psi^{+}_{\lambda },\phi^{+}_{\lambda }]=0$ if and only if $\lambda \in \mathbb{C}\setminus(-\infty,-(\alpha +1)^{2})$ is a solution of

$$ \frac{1}{2} \bigl(1+\delta (\lambda )+\eta(\lambda )\pm \kappa \bigr) = -n, $$

(4.3)

for some $n\in \mathbb{N}$, or equivalently,

$$ \sqrt{ \lambda + (\alpha +1)^2 } + \sqrt{\lambda +( \beta +1)^2} = -2n-1 \mp \kappa . $$

(4.4)

Proposition 4.11

The symmetric operator $(T,\mathcal{D}_{0})$ has a unique self-adjoint extension.

We denote the self-adjoint extension again by T.

Proof

By [4, Theorem XIII.2.10], the adjoint of $(T,\mathcal{D}_{0})$ is $(T,\mathcal{D}(-1,1))$, so the deficiency spaces of $(T,\mathcal{D}_{0})$ consist of the solutions of the differential equations Tf=±if that are in $\mathcal{H}$, [4, Corollary XIII.2.11]. Let $f\in\mathcal{H}$ be a solution of Tf=if. Then f must be a linear combination of $\phi_{i}^{+}$ and $\phi_{i}^{-}$, since these are linearly independent solutions of this eigenvalue equation by Lemma 4.10. Note that ℜ(η(i))>0, so by Remark 4.7, $\phi_{i}^{-}$ is not L ² near −1, which implies that f is a multiple of $\phi_{i}^{+}$. In the same way, it follows that f is a multiple of $\psi^{+}_{i}$. But $\phi_{i}^{+}$ and $\psi_{i}^{+}$ are linearly independent by Lemma 4.10; hence f=0. In the same way, it follows that $f \in\mathcal{H}$ satisfying Tf=−if is the zero function. So T has deficiency indices (0,0), which implies it has a unique self-adjoint extension. □

Assume $\lambda \in \mathbb{C}\setminus \mathbb{R}$. In this case, ℜ(η),ℜ(δ)>0, so $\phi_{\lambda }^{+} \in L^{2} ((-1,0),w^{(\alpha ,\beta )}(x)\,dx )$ and $\psi_{\lambda }^{+} \in L^{2} ((0,1),w^{(\alpha ,\beta )}(x)\,dx )$. We define the Green kernel by

$$K_\lambda (x,y) = \begin{cases} \displaystyle\frac{ \phi^+_\lambda (x) \psi^+_{\lambda }(y)}{[\psi^+_\lambda ,\phi ^+_{\lambda }]}, & x < y,\\ \displaystyle\frac{ \phi^+_\lambda (y) \psi^+_{\lambda }(x)}{[\psi^+_\lambda ,\phi ^+_{\lambda }]}, & x > y. \end{cases} $$

Then $K(\cdot,y) \in\mathcal{H}$ for any y∈(−1,1). The Green kernel is useful for describing the resolvent operator R _λ =(T−λ)⁻¹.

Lemma 4.12

For $\lambda \in \mathbb{C}\setminus \mathbb{R}$, the resolvent R _λ is given by

$$R_{\lambda } f = \bigl(y \mapsto\bigl\langle f, \overline{K_\lambda ( \cdot,y)}\bigr\rangle\bigr), \quad f \in\mathcal{D}_0. $$

Proof

First note that if $f \in\mathcal{D}_{0}$,

$$\lim_{x \downarrow-1}\bigl[\phi^+_\lambda ,f\bigr](x) = 0 = \lim _{x \uparrow1} \bigl[\psi^+_\lambda ,f\bigr](x). $$

Now from Lemma 4.8 we obtain

$$\begin{aligned} R_\lambda (T-\lambda ) f(y) &= \frac{\psi_{\lambda }^+(y)}{[\psi_\lambda ^+,\phi_{\lambda }^+]} \int _{-1}^y \bigl((T-\lambda )f \bigr) (x) \phi_\lambda ^+(x) w^{(\alpha ,\beta )}(x)\, dx \\ & \quad{}+ \frac{\phi_{\lambda }^+(y)}{[\psi_\lambda ^+,\phi_{\lambda }^+]}\int_{y}^1 \bigl((T-\lambda )f \bigr) (x) \psi_{\lambda }^+(x) w^{(\alpha ,\beta )}(x)\, dx \\ & = \frac{1}{[\psi_\lambda ^+,\phi_{\lambda }^+]} \bigl(\psi_{\lambda }^+(y)\bigl[f, \phi_{\lambda }^+\bigr](y) - \phi_\lambda ^+(y)\bigl[f, \psi_\lambda ^+\bigr](y) \bigr), \end{aligned} $$

since $(T-\lambda )\phi_{\lambda }^{+}=0$ and $(T-\lambda )\psi_{\lambda }^{+}=0$. Writing out the terms between brackets, we obtain

$$\begin{aligned} &R_\lambda (T-\lambda ) f(y)\\ &\quad = \frac{1}{[\psi_\lambda ^+,\phi_{\lambda }^+]} \biggl(-f(y)p(y) \psi_{\lambda }^+(y) \frac{d\phi_\lambda ^+}{dx}(y) + f(y)p(y) \phi_\lambda ^+(y) \frac{d\psi_{\lambda }^+}{dx}(y) \biggr)\\ &\quad = f(y). \end{aligned}$$

 □

Now we can determine the spectral measure E of T by

$$ \bigl\langle E(a,b)f,g \bigr\rangle= \lim _{\varepsilon ' \downarrow0} \lim_{\varepsilon \downarrow0} \frac{1}{2\pi i} \int _{a+\varepsilon '}^{b-\varepsilon '} \bigl( \langle R_{\lambda +i\varepsilon }f,g \rangle- \langle R_{\lambda -i\varepsilon }f,g\rangle\bigr) \,d\lambda ,\quad f,g \in \mathcal{D}_0, $$

(4.5)

see [4, Theorem XII.2.10]. We write

$$\begin{aligned} \langle R_{\lambda }f,g\rangle =& \iint_{(x,y) \in \varDelta} \bigl(f(x)\overline{g(y)} + f(y)\overline{g(x)} \bigr) \frac{ \phi_\lambda ^+(x) \psi_{\lambda }^+(y) }{[\psi_\lambda ^+,\phi_{\lambda }^+]} \\ &{}\times w^{(\alpha ,\beta )}(x) w^{(\alpha ,\beta )}(y)\, d(x,y), \end{aligned}$$

(4.6)

where $\varDelta= \{ (x,y) \in \mathbb{R}^{2} \mid-1<x<1,\ x<y<1 \}$.

Let $\lambda \in \mathbb{R}$. To compute the spectral measure, we have to consider the limit

$$ \lim_{\varepsilon \downarrow0} \biggl(\frac{\phi_{\lambda -i\varepsilon }^+(x)\psi_{\lambda -i\varepsilon }^+(y) }{\eta(\lambda -i\varepsilon ) c(-\eta(\lambda -i\varepsilon );\delta (\lambda -i\varepsilon ))}- \frac{ \phi_{\lambda +i\varepsilon }^+(x) \psi_{\lambda +i\varepsilon }^+(y) }{\eta(\lambda +i\varepsilon ) c(-\eta(\lambda +i\varepsilon );\delta (\lambda +i\varepsilon ))} \biggr). $$

(4.7)

Note that

$$\lim_{\varepsilon \downarrow0} \eta(\lambda +i\varepsilon ) = \begin{cases} \displaystyle{\lim_{\varepsilon \downarrow0} \eta(\lambda -i\varepsilon ) \in \mathbb{R}_{\geq 0}} & \text{if}\ \lambda + (\beta +1)^2 \geq0,\\ \displaystyle{\lim_{\varepsilon \downarrow0} \overline{\eta(\lambda -i\varepsilon )} \in i\mathbb{R}_{> 0}} & \text{if}\ \lambda + (\beta +1)^2 <0, \end{cases} $$

and

$$\lim_{\varepsilon \downarrow0} \delta (\lambda +i\varepsilon ) = \begin{cases} \displaystyle{\lim_{\varepsilon \downarrow0} \delta (\lambda -i\varepsilon ) \in \mathbb{R}_{\geq0}} & \text{if}\ \lambda + (\alpha +1)^2 \geq0,\\ \displaystyle{\lim_{\varepsilon \downarrow0} \overline{\delta (\lambda -i\varepsilon )} \in i\mathbb{R}_{> 0}} & \text{if}\ \lambda + (\alpha +1)^2 <0. \end{cases} $$

We see that we have to distinguish several cases.

4.3 The Continuous Spectrum

We define an integral transform $\mathcal{F}_{c}^{(2)}$ mapping $f \in \mathcal{D}_{0}$ to a $\mathbb{C}^{2}$-valued function on Ω ₂=(−∞,−(β+1)²) by

$$\bigl(\mathcal{F}_c^{(2)} f\bigr) (\lambda ) = \int _{-1}^1 f(x) \begin{pmatrix} \varphi^+_\lambda (x)\\ \varphi^-_\lambda (x) \end{pmatrix} w^{(\alpha ,\beta )}(x) \,dx, \quad \lambda \in \varOmega _2,\ f \in\mathcal{D}_0, $$

where the functions $\varphi_{\lambda }^{\pm}$ are defined by (2.13).

Proposition 4.13

Let a,b∈Ω ₂ with a<b. Then

$$\bigl\langle E(a,b) f, g\bigr\rangle= \frac{1}{2\pi D} \int _a^b \bigl(\mathcal{F}_c^{(2)} g\bigr) (\lambda )^* \begin{pmatrix} 1 & v_{12}(\lambda ) \\ v_{21}(\lambda ) & 1 \end{pmatrix} \bigl(\mathcal{F}_c^{(2)} f\bigr) (\lambda ) \frac{ d\lambda }{-i\eta_\lambda }, $$

where (recall from (2.9))

$$v_{21}(\lambda ) = \frac{ c(\eta_\lambda ;\delta_\lambda ) }{c(-\eta_\lambda ;\delta_\lambda )}, $$

and $v_{12}(\lambda ) = \overline{v_{21}(\lambda )}$. Here δ _λ and η _λ are defined by (2.3).

Proof

First observe that lim _ε↓0 δ(λ+iε)=δ _λ and $\lim_{\varepsilon \downarrow0} \delta (\lambda -i\varepsilon ) = \overline{\delta _{\lambda }}=-\delta _{\lambda }$. Similarly, lim _ε↓0 η(λ+iε)=η _λ and $\lim _{\varepsilon \downarrow0} \eta(\lambda -i\varepsilon ) = \overline{\eta_{\lambda }}=-\eta_{\lambda }$. This gives us

$$\lim_{\varepsilon \downarrow0} \phi_{\lambda +i\varepsilon }^+ = \varphi_\lambda ^+, \qquad\lim_{\varepsilon \downarrow0} \phi_{\lambda -i\varepsilon }^+ = \varphi_\lambda ^-. $$

Now the limit in (4.7) is equal to

$$I_\lambda (x,y)=\lim_{\varepsilon \downarrow0} \biggl( \frac{ \varphi^+_\lambda (x) \psi^+_{\lambda +i\varepsilon }(y)}{-\eta_\lambda c(-\eta_\lambda ;\delta _\lambda )} +\frac{\varphi^-_\lambda (x) \psi^+_{\lambda -i\varepsilon }(y)}{-\eta_\lambda c(\eta_\lambda ;-\delta _\lambda )} \biggr). $$

Using the connection formula (4.1), this becomes

$$\begin{aligned} I_\lambda (x,y) =& -\frac{1}{\eta_\lambda } \biggl(\frac{ c(\eta_\lambda ;\delta_\lambda ) }{c(-\eta_\lambda ;\delta_\lambda ) }\varphi_\lambda ^+(x) \varphi_\lambda ^+(y) + \varphi_\lambda ^+(x)\varphi_\lambda ^-(y) + \varphi_\lambda ^-(x)\varphi_\lambda ^+(y)\\ &{} + \frac{ c(-\eta_\lambda ;-\delta_\lambda ) }{c(\eta_\lambda ;-\delta_\lambda ) } \varphi_\lambda ^-(x)\varphi_\lambda ^-(y) \biggr), \end{aligned}$$

which is manifestly symmetric in x and y. Using $\overline{\varphi _{\lambda }^{+}(x)}=\varphi_{\lambda }^{-}(x)$, we see that

$$I_\lambda (x,y) = -\frac{1}{\eta_\lambda } \begin{pmatrix} \varphi_\lambda ^+(y) \\ \varphi_\lambda ^-(y) \end{pmatrix} ^* \begin{pmatrix} 1 & v_{12}(\lambda ) \\ v_{21}(\lambda ) & 1 \end{pmatrix} \begin{pmatrix} \varphi_\lambda ^+(x) \\ \varphi_\lambda ^-(x) \end{pmatrix}. $$

Now we can symmetrize the double integral in (4.6) again, and then the result follows from (4.5). □

Next we define an integral transform $\mathcal{F}_{c}^{(1)}$ mapping $\mathcal{D}_{0}$ to complex-valued functions on Ω ₁=(−(β+1)²,−(α+1)²) by

$$\bigl(\mathcal{F}_c^{(1)}f\bigr) (\lambda ) = \int _{-1}^1 f(x) \varphi_\lambda (x) w^{(\alpha ,\beta )}(x)\,dx,\quad \lambda \in \varOmega _1,\ f \in \mathcal{D}_0, $$

where φ _λ (x) is defined by (2.12).

Proposition 4.14

Let a,b∈Ω ₁ with a<b. Then

$$\bigl\langle E(a,b)f,g\bigr\rangle= \frac{1}{2\pi D} \int _a^b \bigl(\mathcal{F}_c^{(1)}f \bigr) (\lambda ) \overline{\bigl(\mathcal{F}_c^{(1)}g\bigr) ( \lambda )}v(\lambda ) \frac{ d\lambda }{-i\delta _\lambda }, $$

where (recall from (2.7))

$$v(\lambda ) = \frac{1}{ c (\delta _\lambda ;\eta(\lambda ) )c (-\delta _\lambda ;\eta(\lambda ) ) }. $$

Proof

In this case,

$$\lim_{\varepsilon \downarrow0} \delta (\lambda +i\varepsilon ) =\delta _\lambda = -\lim _{\varepsilon \downarrow0} \delta (\lambda -i\varepsilon ) \quad\text{and}\quad\lim _{\varepsilon \downarrow0}\eta(\lambda +i\varepsilon ) = \eta(\lambda ) = \lim_{\varepsilon \downarrow0} \eta(\lambda -i\varepsilon ). $$

Consequently, using Euler’s transformation for ₂ F ₁-functions,

$$\lim_{\varepsilon \downarrow0} \phi_{\lambda +i\varepsilon }^+ = \varphi_\lambda = \lim_{\varepsilon \downarrow0} \phi_{\lambda -i\varepsilon }^+, $$

and

$$\lim_{\varepsilon \downarrow0} \psi_{\lambda - i\varepsilon }^+(x) = \lim _{\varepsilon \downarrow0} \psi_{\lambda +i\varepsilon }^+ (x), $$

so that the limit (4.7) is equal to

$$I_\lambda (x,y) =\lim_{\varepsilon \downarrow0} \biggl( \frac{ \varphi_\lambda (x) \psi_{\lambda +i\varepsilon }^-(y) }{-\delta _\lambda c (\delta _\lambda ;\eta (\lambda ) ) }+ \frac{ \varphi_\lambda (x) \psi_{\lambda +i\varepsilon }^+(y)}{-\delta _\lambda c (-\delta _\lambda ;\eta(\lambda ) )} \biggr), $$

where we have used Lemma 4.4(i). Using the connection formula (4.2), we obtain

$$\begin{aligned} I_\lambda (x,y) & = \lim_{\varepsilon \downarrow0} \frac{ \varphi_\lambda (x)[c (-\delta _\lambda ;\eta(\lambda ) ) \psi_{\lambda +i\varepsilon }^-(y) +c (\delta _\lambda ;\eta(\lambda ) )\psi_{\lambda +i\varepsilon }^+(y)] }{-\delta _\lambda c (\delta _\lambda ;\eta (\lambda ) )c (-\delta _\lambda ;\eta(\lambda ) ) } \\ & = \frac{ \varphi_\lambda (x)\varphi_\lambda (y) }{-\delta _\lambda c (\delta _\lambda ;\eta(\lambda ) )c (-\delta _\lambda ;\eta(\lambda ) ) }. \end{aligned} $$

The result follows from (4.5) and (4.6) after symmetrizing the double integral. □

Since the spectrum is a closed set, the points −(α+1)² and −(β+1)² must belong to the spectrum.

Proposition 4.15

The points −(α+1)² and −(β+1)² belong to the continuous spectrum of T.

Proof

This follows from the fact that none of the eigenfunctions is in $\mathcal {H}$ for these values of λ, see Remark 4.7. □

4.4 The Discrete Spectrum

Recall from (2.5) the finite set

$$\varOmega _d=\biggl\{ \lambda _n \,\big|\, n \in \mathbb{N}\text{ and } n\leq \frac{1}{2}(\kappa -1)\biggr\}, $$

where λ _n is defined by (2.4). For κ≥1, i.e., if Ω _d is nonempty, we define the integral transform $\mathcal{F}_{d}$ on $\mathcal{H}$ by

$$(\mathcal{F}_d f) (\lambda ) = \langle f, \varphi_{\lambda } \rangle, \quad \lambda \in \varOmega _d,\ f \in\mathcal{H}, $$

where $\varphi_{\lambda _{n}}$ is defined by (2.14). Note that $\varphi_{\lambda _{n}}=\phi_{\lambda _{n}}^{+}$.

Proposition 4.16

Let −(α+1)²<a<b. If Ω _d ∩(a,b) consists of exactly one number λ _n , then

$$\bigl\langle E(a,b)f,g \bigr\rangle= (\mathcal{F}_df) ( \lambda _n) \overline{(\mathcal{F}_dg) (\lambda _n)} \frac{ N_{\lambda _n}}{D},\quad f,g\in\mathcal{H}, $$

where (recall from (2.10))

$$N_{\lambda _n} = \underset {\lambda =\lambda _n}{\mathrm {Res}} \biggl(\frac{c (\eta(\lambda ),\delta (\lambda ) )}{\eta (\lambda )c (-\eta(\lambda );\delta (\lambda ) )} \biggr). $$

Furthermore, if Ω _d ∩(a,b)=∅, then

$$\bigl\langle E(a,b)f,g\bigr\rangle=0, \quad f,g \in\mathcal{H}. $$

Proof

Assume Ω _d ∩(a,b)={λ _n }. By (4.5) and (4.6), we have

$$\begin{aligned} \bigl\langle E(a,b)f,g \bigr\rangle&= D^{-1} \iint_{(x,y) \in\varDelta} \bigl(f(x)\overline{g(y)} + f(y) \overline{g(x)} \bigr) w^{(\alpha ,\beta )}(x) w^{(\alpha ,\beta )}(y) \\ &\quad {} \times\biggl[\frac{1}{2\pi i}\int_{\mathcal{C}} \frac{ \phi_{\lambda }^+(x) \psi_{\lambda }^+(y) }{-\eta(\lambda ) c (-\eta(\lambda );\delta (\lambda ) )} \,d\lambda \biggr] \, d(x,y), \end{aligned} $$

where $\mathcal{C}$ is a small clockwise oriented rectifiable closed curve encircling λ _n exactly once. The integral over the curve $\mathcal{C}$ is equal to

$$\phi^+_{\lambda _n}(x) \psi_{\lambda _n}^+(y) \underset {\lambda =\lambda _n}{\mathrm {Res}} \biggl(\frac{1}{\eta(\lambda ) c (-\eta(\lambda );\delta (\lambda ) )} \biggr). $$

By (4.3), we have c(−η(λ);δ(λ))=0 if and only if λ=λ _n , $n \in \mathbb{Z}$, where λ _n is defined by (2.4). So in this case, (4.1) becomes $\psi_{\lambda _{n}}^{+} = c(\eta(\lambda _{n});\delta (\lambda _{n}))\phi_{\lambda _{n}}^{+}$, from which we see that the integrand is symmetric in x and y. Symmetrizing the double integral gives the result. □

Corollary 4.17

Suppose Ω _d is nonempty. Then the following orthogonality relations hold:

$$\langle\varphi_{\lambda _m},\varphi_{\lambda _n} \rangle= \frac{ D}{N_{\lambda _n}} \delta _{mn}, \quad \lambda _m, \lambda _n \in \varOmega _d. $$

Proof

Let λ _n ,λ _m ∈Ω _d , and set $f = \varphi_{\lambda _{m}}$ and $g = \varphi_{\lambda _{n}}$. From Proposition 4.16, it follows that $\langle\varphi_{\lambda _{m}}, \varphi_{\lambda _{n}} \rangle=0$ if λ _n ≠λ _m . Furthermore, if λ _n =λ _m , then

$$\langle\varphi_{\lambda _n}, \varphi_{\lambda _n} \rangle= \langle \varphi_{\lambda _n}, \varphi_{\lambda _n} \rangle\langle \varphi_{\lambda _n}, \varphi_{\lambda _n} \rangle\frac{N_{\lambda _n}}{D}, $$

from which the result follows. □

We have now completely determined the spectrum of T.

Theorem 4.18

The self-adjoint closure of the densely defined operator $(T,\mathcal{D}_{0})$ has continuous spectrum (−∞,−(α+1)²] and (possibly empty) discrete spectrum Ω _d . The sets Ω ₂ and Ω ₁ inside the continuous spectrum have multiplicity two and one, respectively.

4.5 The Integral Transform

We define an integral transform $\mathcal{F}$ on $\mathcal{D}_{0}\subset \mathcal{H}$ by

$$ \mathcal{F}f = \mathcal{F}_c^{(2)}f+ \mathcal{F}_c^{(1)}f + \mathcal{F}_df, \quad f \in\mathcal{D}_0. $$

(4.8)

For $f \in\mathcal{D}_{0}$, this coincides with the integral transform defined in (2.15).

Proposition 4.19

$\mathcal{F}$ extends uniquely to an isometry $\mathcal{F}: \mathcal{H} \to L^{2}(\mathcal{V})$.

Proof

For $f,g\in\mathcal{D}_{0}$, we have

$$\langle f,g \rangle=\langle\mathcal{F}f,\mathcal{F}g \rangle_{\mathcal{V}} $$

by Propositions 4.13, 4.14, and 4.16, so $\mathcal{F}:\mathcal{D}_{0} \to L^{2}(\mathcal{V})$ is an isometry. By continuity of $\mathcal{F}$ and density of $\mathcal{D}_{0}$ in $\mathcal {H}$, it extends uniquely to an isometry $\mathcal{H} \to L^{2}(\mathcal{V})$. □

Our next goal is to show that $\mathcal{F}:\mathcal{H}\to L^{2}(\mathcal{V})$ is surjective and determine the inverse. For convenience, we assume that T has no discrete spectrum.

For 0<a<1, we define

$$\langle f,g \rangle_{a} = \int_{-a}^{a} f(x) g(x) w^{(\alpha ,\beta )}(x)\,dx $$

for all functions f,g for which the integral converges. Note that for $f,g \in\mathcal{H}$, the limit a↑1 gives the inner product $\langle f,\bar{g}\rangle$. Suppose now that f _λ is a solution of the eigenvalue equation Tf _λ =λf _λ . Then by Lemma 4.8,

$$\langle f_{\lambda }, f_{\lambda '} \rangle_{a} = \frac{ [f_{\lambda },f_{\lambda '}](a) - [f_{\lambda },f_{\lambda '}](-a) }{\lambda -\lambda '}, \quad \lambda ,\lambda ' \in \mathbb{R}, \ \lambda \neq \lambda '. $$

We will use this expression with $f_{\lambda }=\varphi_{\lambda }^{\pm}$, and we want to let a↑1. We need to consider several cases.

4.5.1 Case 1: λ,λ′∈Ω ₂

From Lemma 4.6, we find for x↓−1, cf. the proof of Lemma 4.10,

$$\begin{aligned} \bigl[\varphi_{\lambda }^+,\varphi_{\lambda '}^- \bigr](x)&=\frac{D}{2}(\eta_{\lambda }+\eta_{\lambda '}) \biggl( \frac{ 1+x }{2} \biggr)^{\frac{1}{2}(\eta_{\lambda }-\eta_{\lambda '})} \bigl (1+\mathcal{O}(1+x) \bigr), \\ \bigl[\varphi_{\lambda }^+,\varphi_{\lambda '}^+\bigr](x)&= \frac{D}{2}(\eta_{\lambda }-\eta_{\lambda '}) \biggl( \frac{ 1+x }{2} \biggr)^{\frac{1}{2}(\eta_{\lambda }+\eta_{\lambda '})} \bigl (1+\mathcal{O}(1+x) \bigr). \end{aligned} $$

The behavior at x=−1 of $[\varphi_{\lambda }^{-},\varphi_{\lambda '}^{+}](x)$ and $[\varphi_{\lambda }^{-},\varphi_{\lambda '}^{-}](x)$ follows from $\varphi_{\lambda }^{+}(x) = \overline{\varphi_{\lambda }^{-}(x)}$. For x↑1, we use the expansion from (4.2) (recall that $\varphi_{\lambda }^{\pm}= \lim_{\varepsilon \downarrow0} \phi _{\lambda +i\varepsilon }^{\pm}$) and Lemma 4.6 to find

$$\begin{aligned} \bigl[\varphi_{\lambda }^+,\varphi_{\lambda '}^-\bigr](x) =& \frac{D}{2} \sum_{\epsilon,\epsilon' \in\{+,-\}} c(\epsilon \delta _\lambda ;\eta_\lambda ) c\bigl(-\epsilon' \delta _{\lambda '};-\eta_{\lambda '}\bigr) \bigl(\epsilon \delta _\lambda + \epsilon'\delta _{\lambda '}\bigr)\\ &{}\times \biggl(\frac{ 1-x}{2} \biggr)^{\frac{1}{2}(\epsilon \delta _\lambda - \epsilon' \delta _{\lambda '})} \bigl(1+ \mathcal{O}(1-x) \bigr), \end{aligned}$$

and

$$\begin{aligned} \bigl[\varphi_{\lambda }^+,\varphi_{\lambda '}^+\bigr](x) =& \frac{D}{2} \sum_{\epsilon,\epsilon' \in\{+,-\}} c(\epsilon \delta _\lambda ;\eta_\lambda ) c\bigl(-\epsilon' \delta _{\lambda '};\eta_{\lambda '}\bigr) \bigl(\epsilon \delta _\lambda + \epsilon'\delta _{\lambda '}\bigr)\\ &{}\times \biggl(\frac{ 1-x}{2} \biggr)^{\frac{1}{2}(\epsilon \delta _\lambda - \epsilon' \delta _{\lambda '})} \bigl(1+ \mathcal{O}(1-x) \bigr). \end{aligned}$$

We will need the following behavior of the c-functions.

Lemma 4.20

The c-function defined by (2.6) satisfies

$$\begin{aligned} c(\delta _\lambda ;\eta_\lambda ) &= \begin{cases} \mathcal{O}(e^{-\pi\sqrt{-\lambda }}), & \lambda \to-\infty,\\ \mathcal{O}(1), & \lambda \uparrow-(\beta +1)^2, \end{cases} \\ c(-\delta _\lambda ;\eta_\lambda ) &= \begin{cases} \mathcal{O}(1), & \lambda \to-\infty,\\ \mathcal{O}(1), & \lambda \uparrow-(\beta +1)^2. \end{cases} \end{aligned} $$

Proof

This follows from the definition (2.6) of the c-function and well-known asymptotic properties of the Γ-function, see, e.g., [2, Sect. 1.4]. □

Proposition 4.21

Let f∈C(Ω ₂) satisfy

$$f(\lambda ) = \begin{cases} \mathcal{O}(|\lambda |^{-1-\varepsilon }), & \lambda \to-\infty,\\ \mathcal{O}(1), & \lambda \uparrow-(\beta +1)^2, \end{cases} $$

for some ε>0, and let λ′∈Ω ₂. Then

$$\lim_{a \uparrow1} \int_{\varOmega _2} f(\lambda ) \bigl\langle \varphi_\lambda ^+,\varphi_{\lambda '}^- \bigr\rangle_a\, d\lambda =-2\pi i D \,\delta _{\lambda '}c(-\delta _{\lambda '};\eta_{\lambda '}) c(\delta _{\lambda '};-\eta_{\lambda '}) f\bigl(\lambda '\bigr). $$

Proof

Note that, for λ≠λ′,

$$ \frac{\eta_\lambda +\eta_{\lambda '}}{\lambda -\lambda '} = \frac {1}{\eta_\lambda -\eta_{\lambda '}}, \qquad\frac{\eta_\lambda -\eta_{\lambda '}}{\lambda -\lambda '} = \frac{1}{\eta_\lambda +\eta _{\lambda '}}, $$

(4.9)

and similar expressions are valid for δ _λ . Now use δ _λ =i|δ _λ | and η _λ =i|η _λ |, and write $N=-\frac{1}{2}\ln(\frac {1-a}{2})$. Then

$$\begin{aligned} &\lim_{a \uparrow1} \int_{\varOmega _2} f( \lambda ) \bigl\langle\varphi_\lambda ^+,\varphi_{\lambda '}^- \bigr \rangle_a \,d\lambda \\ &\quad = \lim_{a \uparrow1} \int _{\varOmega _2} f(\lambda ) \frac{[\varphi_{\lambda }^+,\varphi_{\lambda '}^-](a) - [\varphi _{\lambda }^+,\varphi_{\lambda '}^-](-a)}{\lambda -\lambda '} \,d\lambda \\ &\quad = \frac{D}{2}\lim_{N \to\infty} \biggl(\int _{\varOmega _2} f(\lambda ) \sum_{\epsilon\in\{+,-\}} \biggl( \xi^\epsilon_-(\lambda ) \frac{\cos N(|\delta _{\lambda }|+\epsilon|\delta _{\lambda '}|)}{\delta _{\lambda }+\epsilon \delta _{\lambda '}}\\ &\qquad{} + i\xi_+^\epsilon(\lambda ) \frac{\sin N(|\delta _{\lambda }|+\epsilon|\delta _{\lambda '}|)}{\delta _{\lambda }+\epsilon \delta _{\lambda '}} \biggr)\, d\lambda , \\ & \qquad{}- \int_{\varOmega _2} f(\lambda )\frac{ \cos N(|\eta_\lambda |-|\eta_{\lambda '}|) }{\eta_\lambda - \eta_{\lambda '}} \,d\lambda - i\int_{\varOmega _2} f(\lambda )\frac{ \sin N(|\eta_\lambda |-|\eta_{\lambda '}|) }{\eta_\lambda - \eta_{\lambda '}} \,d\lambda \biggr), \end{aligned} $$

where

$$\xi_\pm^\epsilon(\lambda )= c(\delta _\lambda ; \eta_\lambda ) c(\epsilon \delta _{\lambda '};-\eta_{\lambda '}) \pm c(-\delta _\lambda ;\eta_\lambda ) c(-\epsilon \delta _{\lambda '};- \eta_{\lambda '}). $$

The terms with $\xi_{\pm}^{+}$ vanish by the Riemann–Lebesgue lemma, which follows from Lemma 4.20 and the assumptions on f.

Claim

$$\begin{aligned} &\lim_{N \to\infty} \biggl(\int_{\varOmega _2} f(\lambda ) \xi_-^-(\lambda ) \frac{ \cos N(|\delta _\lambda |-|\delta _{\lambda '}|) }{\delta _\lambda - \delta _{\lambda '}} \,d\lambda - \int_{\varOmega _2} f(\lambda ) \frac{ \cos N(|\eta_\lambda |-|\eta_{\lambda '}|) }{\eta_\lambda - \eta_{\lambda '}} \,d\lambda \biggr)\\ &\quad =0. \end{aligned}$$

Proof of Claim

Using (4.9) and $\cos \theta _{1}-\cos \theta _{2} = -2\sin(\frac{\theta _{1}+\theta _{2}}{2}) \sin(\frac{\theta _{1}-\theta _{2}}{2})$, we obtain

$$\begin{aligned} &\xi_-^-(\lambda ) \frac{ \cos N(|\delta _\lambda |-|\delta _{\lambda '}|) }{\delta _\lambda - \delta _{\lambda '}} - \frac{ \cos N(|\eta_\lambda |-|\eta_{\lambda '}|) }{\eta _\lambda - \eta_{\lambda '}}\\ &\quad = \frac{\xi_-^-(\lambda )(\delta _\lambda +\delta _{\lambda '})-(\eta_\lambda +\eta_{\lambda '})}{\lambda -\lambda '}\cos N\bigl(|\delta _\lambda |-|\delta _{\lambda '}|\bigr) \\ &\qquad{}+ \frac{2 \sin\frac{N}{2}(|\delta _\lambda |+|\eta_\lambda |-|\delta _{\lambda '}|-|\eta_{\lambda '}|) \sin\frac{N}{2}(|\delta _\lambda |-|\eta_\lambda |+|\delta _{\lambda '}|-|\eta_{\lambda '}|) }{\eta_\lambda -\eta_{\lambda '}} . \end{aligned} $$

We multiply the right-hand side of the above identity by f(λ) and integrate over λ. Since the function $\lambda \mapsto\frac{\xi_{-}^{-}(\lambda )(\delta _{\lambda }+\delta _{\lambda '})-(\eta _{\lambda }+\eta_{\lambda '})}{\lambda -\lambda '}$ has a removable singularity by Lemma 4.4(ii), it follows from the Riemann–Lebesgue lemma that the first term vanishes as N→∞. For the second term, we may use

$$\biggl \vert \frac{\sin\frac{N}{2}(|\delta _\lambda |+|\eta_\lambda |-|\delta _{\lambda '}|-|\eta_{\lambda '}|) }{\eta_\lambda -\eta_{\lambda '}}\biggr \vert \leq B $$

for λ in a neighborhood of λ′ and for some B>0. Then we see that we can apply the Riemann–Lebesgue lemma again, which proves the claim. □

To finish the proof of the proposition, we use

$$ \lim_{N \to\infty} \frac{1}{\pi} \int_A^B g(x) \frac{ \sin N(x-y)}{x-y}\, dx = g(y) $$

(4.10)

if g∈L ¹(A,B) is continuous. Then

$$\begin{aligned} \lim_{a \uparrow1} \int_{\varOmega _2} f( \lambda ) \bigl\langle\varphi_\lambda ^+,\varphi_{\lambda '}^- \bigr \rangle_a\, d\lambda &= \frac{iD}{2}\lim_{N \to\infty} \int_{\varOmega _2} f(\lambda ) \xi_+^-(\lambda ) \frac{\sin N(|\delta _\lambda |-|\delta _{\lambda '}|)}{\delta _\lambda -\delta _{\lambda '}} \,d\lambda \\ &\quad{}- \frac{iD}{2} \lim_{N \to\infty} \int _{\varOmega _2} f(\lambda )\frac{ \sin N(|\eta_\lambda |-|\eta_{\lambda '}|) }{\eta_\lambda - \eta _{\lambda '}} \,d\lambda \\ &=- \pi i D \bigl( \delta _{\lambda '} \xi_+^-\bigl(\lambda '\bigr) + \eta_{\lambda '} \bigr) f\bigl(\lambda '\bigr), \end{aligned} $$

provided $\xi_{+}^{-}\,f$ and f are continuous functions in L ¹(Ω ₂), which is indeed the case. Here we used the substitutions x=|δ _λ | and x=|η _λ | before applying (4.10); note that $\frac{dx}{d\lambda }=-\frac{1}{2x}$ in both cases. Finally, applying Lemma 4.4(ii) with (x,y)=(δ _λ ,η _λ ), the last expression becomes

$$-2\pi i D \,\delta _{\lambda '}c(-\delta _{\lambda '};\eta_{\lambda '}) c( \delta _{\lambda '};-\eta_{\lambda '}) f\bigl(\lambda '\bigr), $$

which finishes the proof. □

The following result is proved in the same way as Proposition 4.21.

Proposition 4.22

Let f∈C(Ω ₂) satisfy the same conditions as in Proposition 4.21, and let λ′∈Ω ₂. Then

$$\lim_{a \uparrow1} \int_{\varOmega _2} f(\lambda ) \bigl\langle \varphi_\lambda ^+,\varphi_{\lambda '}^+ \bigr\rangle_a\, d\lambda =-2\pi i D \,\delta _{\lambda '}c(\delta _{\lambda '};\eta_{\lambda '}) c(-\delta _{\lambda '};\eta_{\lambda '}) f\bigl(\lambda '\bigr). $$

By combining Propositions 4.21 and 4.22, we obtain the following result.

Proposition 4.23

Let f ₁ and f ₂ satisfy the conditions from Proposition 4.21. Then

$$\mathcal{F}_c^{(2)} \biggl[ \frac{1}{2\pi D} \int _{\varOmega _2} \begin{pmatrix} \varphi_\lambda ^+(x) \\ \varphi_\lambda ^-(x) \end{pmatrix} ^* \begin{pmatrix} f_1(\lambda ) \\ f_2(\lambda ) \end{pmatrix} \, d\lambda \biggr]\bigl( \lambda '\bigr) =-i\delta _{\lambda '} A\bigl(\lambda '\bigr) \begin{pmatrix} f_1(\lambda ') \\ f_2(\lambda ') \end{pmatrix}, $$

where

$$A\bigl(\lambda '\bigr) = \begin{pmatrix} c(-\delta _{\lambda '};\eta_{\lambda '}) c(\delta _{\lambda '};-\eta_{\lambda '}) & c(\delta _{\lambda '};\eta_{\lambda '}) c(-\delta _{\lambda '};\eta_{\lambda '}) \\ c(\delta _{\lambda '};-\eta_{\lambda '}) c(-\delta _{\lambda '};-\eta_{\lambda '})& c(-\delta _{\lambda '};\eta_{\lambda '}) c(\delta _{\lambda '};-\eta_{\lambda '}) \end{pmatrix} . $$

Proof

Let f ₁ and f ₂ satisfy the conditions of Proposition 4.21. Then from this proposition and from applying Fubini’s theorem, we obtain

$$\begin{aligned} & -i\delta _{\lambda '}c(\delta _{\lambda '};\eta_{\lambda '}) c(-\delta _{\lambda '};-\eta_{\lambda '}) f_2\bigl( \lambda '\bigr)\\ &\quad = \frac{1}{2\pi D} \lim_{a \uparrow1} \int _{\varOmega _2} f_2(\lambda ) \int_{-a}^a \varphi_\lambda ^+(x) \varphi_{\lambda '}^-(x) w^{(\alpha ,\beta )}(x)\,dx \,d\lambda \\ &\quad = \int_{-1}^1 \biggl[ \frac{1}{2\pi D} \int _{\varOmega _2} f_2(\lambda ) \overline{ \varphi_\lambda ^-(x)} \, d\lambda \biggr] \varphi_{\lambda '}^-(x)w^{(\alpha ,\beta )}(x)\,dx. \end{aligned} $$

From Propositions 4.21 and 4.22, we find three similar identities, leading to

$$\begin{aligned} &\int_{-1}^1 \biggl[ \frac{1}{2\pi D} \int _{\varOmega _2} \begin{pmatrix} \varphi_\lambda ^+(x) \\ \varphi_\lambda ^-(x) \end{pmatrix} ^* \begin{pmatrix} f_1(\lambda ) \\ f_2(\lambda ) \end{pmatrix} \, d\lambda \biggr] \begin{pmatrix} \varphi_{\lambda '}^+(x) \\ \varphi_{\lambda '}^-(x) \end{pmatrix} w^{(\alpha ,\beta )}(x)\,dx\\ &\quad =-i\delta _{\lambda '} A\bigl(\lambda '\bigr) \begin{pmatrix} f_1(\lambda ') \\ f_2(\lambda ') \end{pmatrix} , \end{aligned}$$

which is the desired result. □

We need the inverse of the matrix A(λ) from Proposition 4.23.

Lemma 4.24

For λ∈Ω ₂, A(λ)⁻¹=V(λ), with V(λ) defined by (2.8).

Proof

We have

$$\begin{aligned} \det A(\lambda )& = c(\delta _\lambda ;-\eta_\lambda ) c(- \delta _\lambda ;\eta_\lambda ) \bigl( c(\delta _\lambda ;- \eta_\lambda ) c(-\delta _\lambda ;\eta_\lambda ) -c( \delta _\lambda ;\eta_\lambda )c(-\delta _\lambda ;- \eta_\lambda ) \bigr) \\ & = \frac{\eta_\lambda }{\delta _\lambda } c(\delta _\lambda ;-\eta_\lambda ) c(- \delta _\lambda ;\eta_\lambda ) \end{aligned} $$

by Lemma 4.4(ii). Now it is straightforward to compute the inverse of A. The result then follows from the definition of V(λ) and Lemma 4.4(i). □

Let $C_{0}(\varOmega _{2};\mathbb{C}^{2})$ denote the set of continuous $\mathbb{C}^{2}$-valued functions $g= \big( \begin{array}{c} \scriptstyle g_{1}\\[-3pt] \scriptstyle g_{2} \end{array} \big)$ on Ω ₂ satisfying

$$g_j(\lambda ) = \begin{cases} \mathcal{O}(|\lambda |^{-\frac{1}{2}-\varepsilon }), & \lambda \to-\infty,\\ \mathcal{O}(|\lambda +(\beta +1)^2|^{\frac{1}{2}}), & \lambda \uparrow-(\beta +1)^2, \end{cases} \quad j=1,2, $$

for some ε>0. For $g \in C_{0}(\varOmega _{2};\mathbb{C}^{2})$, we define the function $\mathcal{G}_{c}^{(2)}g$ by

$$\bigl(\mathcal{G}_c^{(2)}g\bigr) (x) = \frac{1}{2\pi D} \int_{\varOmega _2} \begin{pmatrix} \varphi_{\lambda }^+(x) \\ \varphi_\lambda ^-(x) \end{pmatrix} ^* V(\lambda ) g(\lambda ) \frac{d\lambda }{-i\eta_\lambda }, \quad x \in(-1,1). $$

Proposition 4.25

Let $g \in C_{0}(\varOmega _{2};\mathbb{C}^{2})$ and λ∈Ω ₂. Then $(\mathcal {F}_{c}^{(2)} \mathcal{G}_{c}^{(2)} g)(\lambda ) = g(\lambda )$.

Proof

Let $g \in C_{0}(\varOmega _{2};\mathbb{C}^{2})$, and define the $\mathbb{C}^{2}$-valued function f by $f(\lambda )=\big( \begin{array}{c} \scriptstyle f_{1}(\lambda )\\[-3pt] \scriptstyle f_{2}(\lambda ) \end{array} \big) = \frac{1}{-i\delta _{\lambda }} A(\lambda )^{-1}g(\lambda )$. Since

$$v_{21}(\lambda )= \begin{cases} \mathcal{O}(e^{-\pi\sqrt{-\lambda }}), & \lambda \to-\infty,\\ \mathcal{O}(1), & \lambda \uparrow-(\beta +1)^2, \end{cases} $$

by Lemma 4.20, the functions f ₁ and f ₂ satisfy the conditions from Proposition 4.21. Now Proposition 4.23 shows that

$$\mathcal{F}^{(2)}_c \biggl[ \frac{1}{2\pi D} \int _{\varOmega _2} \begin{pmatrix} \varphi_\lambda ^+(x) \\ \varphi_\lambda ^-(x) \end{pmatrix} ^* \frac{1}{-i\delta _\lambda }A(\lambda )^{-1} g( \lambda ) \, d\lambda \biggr] \bigl(\lambda '\bigr) = g\bigl(\lambda ' \bigr). $$

From Lemma 4.24, we see that the term inside square brackets is exactly $(\mathcal{G}_{c}^{(2)}g)(x)$. □

4.5.2 Case 2: λ,λ′∈Ω ₁

In this case,

$$\lim_{x \downarrow-1}[\varphi_\lambda ,\varphi_{\lambda '}](x) = 0, $$

and for x↑1, we have

$$\begin{aligned}{} [\varphi_\lambda ,\varphi_{\lambda '}](x) =& \frac{D}{2} \sum_{\epsilon,\epsilon' \in\{+,-\}} c\bigl(\epsilon \delta _\lambda ;\eta(\lambda )\bigr) c\bigl(-\epsilon' \delta _{\lambda '};\eta\bigl(\lambda '\bigr)\bigr) \bigl(\epsilon \delta _\lambda + \epsilon'\delta _{\lambda '}\bigr)\\ &{}\times \biggl( \frac{ 1-x}{2} \biggr)^{\frac{1}{2}(\epsilon \delta _\lambda - \epsilon' \delta _{\lambda '})} \bigl(1+ \mathcal{O}(1-x) \bigr). \end{aligned}$$

We have the following behavior of the c-functions.

Lemma 4.26

The c-function defined by (2.6) satisfies

$$c\bigl(\pm \delta _\lambda ;\eta(\lambda )\bigr) = \begin{cases} \mathcal{O}(1), & \lambda \downarrow-(\beta +1)^2,\\ \mathcal{O}(|\lambda +(\alpha +1)^2|^{-\frac{1}{2}}), & \lambda \uparrow-(\alpha +1)^2. \end{cases} $$

In the same way as in Proposition 4.21, this leads to the following result.

Proposition 4.27

Let f be a continuous function satisfying

$$f(\lambda ) = \begin{cases} \mathcal{O}(1), & \lambda \downarrow-(\beta +1)^2,\\ \mathcal{O}(|\lambda +(\alpha +1)^2|^{-\frac{1}{2}+\varepsilon }), & \lambda \uparrow-(\alpha +1)^2, \end{cases} $$

for some ε>0, and let λ′∈Ω ₁. Then

$$\lim_{a \uparrow1} \int_{\varOmega _1} f(\lambda ) \langle \varphi_\lambda ,\varphi_{\lambda '} \rangle_a\, d\lambda =- \frac{2\pi i D \,\delta _{\lambda '} }{v(\lambda ')} f\bigl(\lambda '\bigr), $$

where (recall from (2.7)) v(λ′)=(c(δ _λ′;η(λ′))c(−δ _λ′;η(λ′)))⁻¹.

Note that

$$v(\lambda ) = \begin{cases} \mathcal{O}(1), & \lambda \downarrow-(\beta +1)^2,\\ \mathcal{O}(|\lambda +(\alpha +1)^2|), & \lambda \uparrow-(\alpha +1)^2, \end{cases} $$

by Lemma 4.26. Let C ₀(Ω ₁) denote the set of continuous functions g on Ω ₁ satisfying

$$g(\lambda ) = \begin{cases} \mathcal{O}(1), & \lambda \downarrow-(\beta +1)^2,\\ \mathcal{O}(|\lambda +(\alpha +1)^2|^{\varepsilon }), & \lambda \uparrow-(\alpha +1)^2, \end{cases} $$

for some ε>0. We define an integral transform $\mathcal {G}_{c}^{(1)}$ on C ₀(Ω ₁) by

$$\bigl(\mathcal{G}_c^{(1)}g\bigr) (x) = \frac{1}{2\pi D} \int_{\varOmega _1} g(\lambda ) \varphi_\lambda (x) W^{(1)}( \lambda ) \frac{d\lambda }{-i \delta _\lambda }, \quad x \in(-1,1),\ g \in C_0( \varOmega _1). $$

Now similarly to Proposition 4.25, it follows from Proposition 4.27 that $\mathcal {F}_{c}^{(1)}$ is a left-inverse of $\mathcal{G}_{c}^{(1)}$.

Proposition 4.28

For g∈C ₀(Ω ₁) and λ∈Ω ₁, we have $(\mathcal {F}_{c}^{(1)} \mathcal{G}_{c}^{(1)} g)(\lambda ) = g(\lambda )$.

4.6 The Integral Transform $\mathcal{G}$

We define $\mathcal{G}$ on $C_{0}(\varOmega _{1}) \cup C_{0}(\varOmega _{2};\mathbb{C}^{2})$ by $\mathcal{G} = \mathcal{G}_{c}^{(1)} \oplus\mathcal{G}_{c}^{(2)}$. We will show that $\mathcal{F}$ is a left-inverse of $\mathcal{G}$. We need the following result.

Proposition 4.29

(i):

Let λ∈Ω ₁ and g∈C ₀(Ω ₁). Then $(\mathcal {F}_{c}^{(2)} \mathcal{G}_{c}^{(1)} g)(\lambda ) = ( \begin{array}{c} \scriptstyle 0 \\[-3pt] \scriptstyle0 \end{array} )$.

(ii):

Let λ∈Ω ₂ and $g \in C_{0}(\varOmega _{2};\mathbb{C}^{2})$. Then $(\mathcal {F}_{c}^{(1)} \mathcal{G}_{c}^{(2)} g)(\lambda ) = 0$.

Proof

Let λ∈Ω ₂ and λ′∈Ω ₁. Then

$$\lim_{x \downarrow-1}\bigl[\varphi_\lambda ^{\pm}, \varphi_{\lambda '}\bigr](x) = 0, $$

and for x↑1,

$$\begin{aligned} \bigl[\varphi_\lambda ^\pm, \varphi_{\lambda '}\bigr](x)& = \frac{D}{2} \sum_{\epsilon,\epsilon' \in\{+,-\}} c(\epsilon \delta _\lambda ;\pm\eta_\lambda ) c\bigl(-\epsilon' \delta _{\lambda '};\eta\bigl(\lambda '\bigr)\bigr) \bigl(\epsilon \delta _\lambda + \epsilon'\delta _{\lambda '}\bigr)\\ &\quad {}\times \biggl( \frac{ 1-x}{2} \biggr)^{\frac{1}{2}(\epsilon \delta _\lambda - \epsilon' \delta _{\lambda '})} \bigl(1+ \mathcal{O}(1-x) \bigr). \end{aligned} $$

Similarly to the proof of Proposition 4.21, it follows by application of the Riemann–Lebesgue lemma that

$$\lim_{a \uparrow1} \int_{\varOmega _2} f(\lambda ) \bigl\langle \varphi_\lambda ^\pm,\varphi_{\lambda '} \bigr \rangle_a \,d\lambda = \lim_{a \uparrow1} \int _{\varOmega _2} f(\lambda ) \frac{[\varphi_{\lambda }^\pm,\varphi_{\lambda '}](a) - [\varphi _{\lambda }^\pm,\varphi_{\lambda '}](-a)}{\lambda -\lambda '} \,d\lambda =0 $$

for suitable functions f. As in Proposition 4.23, we obtain from this $(\mathcal{F}_{c}^{(2)} \mathcal{G}_{c}^{(1)} g)(\lambda ) = \big( \begin{array}{c} \scriptstyle 0 \\[-3pt] \scriptstyle0 \end{array} \big)$.

In the same way, it follows from

$$\lim_{a \uparrow1} \int_{\varOmega _1} f(\lambda ) \bigl\langle \varphi_{\lambda },\varphi_{\lambda '}^\pm, \bigr \rangle_a \,d\lambda = \lim_{a \uparrow1} \int _{\varOmega _1} f(\lambda ) \frac{[\varphi_{\lambda } ,\varphi_{\lambda '}^\pm](a) - [\varphi _{\lambda },\varphi_{\lambda '}^\pm](-a)}{\lambda -\lambda '} \,d\lambda =0, $$

for suitable functions f, that $(\mathcal{F}_{c}^{(1)} \mathcal {G}_{c}^{(2)} g)(\lambda ) = 0$. □

Combining Propositions 4.25, 4.28, and 4.29 shows that $(\mathcal{F}\circ\mathcal{G})g=g$ for $g \in C_{0}(\varOmega _{1}) \cup C_{0}(\varOmega _{2};\mathbb{C}^{2})$.

Proposition 4.30

The integral transform $\mathcal{G}$ extends uniquely to an operator $\mathcal{G}:L^{2}(\mathcal{V}) \to\mathcal{H}$ such that $\mathcal{G} = \mathcal {F}^{-1}$.

Proof

Let $g \in C_{0}(\varOmega _{1}) \cup C_{0}(\varOmega _{2};\mathbb{C}^{2})$. Then

$$\langle g,g \rangle_{\mathcal{V}} = \bigl\langle(\mathcal{F} \circ \mathcal{G})g, (\mathcal{F} \circ\mathcal{G})g \bigr\rangle_{\mathcal{V}} = \langle\mathcal{G}g, \mathcal{G}g \rangle $$

by Proposition 4.19. Since $C_{0}(\varOmega _{1}) \cup C_{0}(\varOmega _{2};\mathbb{C}^{2})$ is dense in $\mathcal{H}$, $\mathcal{G}$ extends by continuity uniquely to an operator $\mathcal {G}:L^{2}(\mathcal{V}) \to\mathcal{H}$, and $\mathcal{F} \circ\mathcal{G}$ extends to the identity operator on $\mathcal{H}$; hence $\mathcal{G} = \mathcal{F}^{-1}$. □

Remark 4.31

In case the discrete spectrum Ω _d is nonempty, the inverse of $\mathcal{F}$ is the extension of the operator $\mathcal{G} = \mathcal {G}^{(1)}_{c} \oplus\mathcal{G}^{(2)}_{c} \oplus\mathcal{G}_{d}$, with

$$(\mathcal{G}_d g) (x) = \frac{1}{D} \sum _{\lambda \in \varOmega _d} g(\lambda ) \varphi_\lambda (x) N_\lambda , \quad x \in(-1,1), $$

for any function $g:\varOmega _{d} \to \mathbb{C}$. The proof in this case is the same as in the case of the empty discrete spectrum.