Abstract
In the regression model \(Y=b(X)+\sigma(X)\varepsilon\), where \(X\) has a density \(f\), this paper deals with an oracle inequality for an estimator of \(bf\), involving a kernel in the sense of Lerasle et al. [13], selected via the PCO method. In addition to the bandwidth selection for kernel-based estimators already studied in Lacour et al. [12] and Comte and Marie [3], the dimension selection for anisotropic projection estimators of \(f\) and \(bf\) is covered.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 INTRODUCTION
Consider \(n\in\mathbb{N}^{*}\) independent \(\mathbb{R}^{d}\times\mathbb{R}\)-valued (\(d\in\mathbb{N}^{*}\)) random variables \((X_{1},Y_{1}),\dots,(X_{n},Y_{n})\), having the same probability distribution assumed to be absolutely continuous with respect to Lebesgue’s measure, and
where \(\ell:\mathbb{R}\rightarrow\mathbb{R}\) is a Borel function and \(K\) is a symmetric continuous map from \(\mathbb{R}^{d}\times\mathbb{R}^{d}\) into \(\mathbb{R}\). This is an estimator of the function \(s:\mathbb{R}^{d}\rightarrow\mathbb{R}\) defined by
where \(f\) is a density of \(X_{1}\). For \(\ell=1\), \(\widehat{s}_{K,\ell}(n;.)\) coincides with the estimator of \(f\) studied in Lerasle et al. [13], covering Parzen-Rosenblatt’s and projection estimators already deeply studied in the literature (see Parzen [16], Rosenblatt [17], Tsybakov [18], etc.), but for \(\ell\not=1\), it covers estimators involved in nonparametric regression. Assume that for every \(i\in\{1,\dots,n\}\),
where \(\varepsilon_{i}\) is a centered random variable of variance \(1\), independent of \(X_{i}\), and \(b,\sigma:\mathbb{R}^{d}\rightarrow\mathbb{R}\) are Borel functions.
-
If \(\ell=\textrm{Id}_{\mathbb{R}}\), \(k\) is a symmetric kernel and
$$K(x^{\prime},x)=\prod_{q=1}^{d}\frac{1}{h_{q}}k\left(\frac{x_{q}^{\prime}-x_{q}}{h_{q}}\right)\textrm{\ with\ }h_{1},\dots,h_{d}>0$$(2)for every \(x,x^{\prime}\in\mathbb{R}^{d}\), then \(\widehat{s}_{K,\ell}(n;.)\) is the numerator of the well-known Nadaraya-Watson estimator of the regression function \(b\) (see Nadaraya [15] and Watson [20]). Precisely, \(\widehat{s}_{K,\ell}(n;.)\) is an estimator of \(s=bf\) because \(\varepsilon_{1}\) is independent to \(X_{1}\) and \(\mathbb{E}(\varepsilon_{1})=0\). If \(\ell\not=\textrm{Id}_{\mathbb{R}}\), then \(\widehat{s}_{K,\ell}(n;.)\) is the numerator of the estimator studied in Einmahl and Mason [7, 8].
-
If \(\ell=\textrm{Id}_{\mathbb{R}}\), \(\mathcal{B}_{m_{q}}=\{\varphi_{1}^{m_{q}},\dots,\varphi_{m_{q}}^{m_{q}}\}\) (\(m_{q}\in\mathbb{N}^{*}\) and \(q\in\{1,\dots,d\}\)) is an orthonormal family of \(\mathbb{L}^{2}(\mathbb{R})\) and
$$K(x^{\prime},x)=\prod_{q=1}^{d}\sum_{j=1}^{m_{q}}\varphi_{j}^{m_{q}}(x_{q})\varphi_{j}^{m_{q}}(x_{q}^{\prime})$$(3)for every \(x,x^{\prime}\in\mathbb{R}^{d}\), then \(\widehat{s}_{K,\ell}(n;.)\) is the projection estimator on \(\mathcal{S}=\textrm{span}(\mathcal{B}_{m_{1}}\otimes\dots\otimes\mathcal{B}_{m_{d}})\) of \(s=bf\).
Now, assume that \(b=0\) in Model (1): for every \(i\in\{1,\dots,n\}\),
If \(\ell(x)=x^{2}\) for every \(x\in\mathbb{R}\), then \(\widehat{s}_{K,\ell}(n;.)\) is an estimator of \(s=\sigma^{2}f\).
These ten last years, several data-driven procedures have been proposed in order to select the bandwidth of Parzen–Rosenblatt’s estimator (\(\ell=1\) and \(K\) defined by (2)). First, Goldenshluger–Lepski’s method, introduced in [10], which reaches the adequate bias-variance compromise, but is not completely satisfactory on the numerical side (see Comte and Rebafka [5]). More recently, in [12], Lacour, Massart and Rivoirard proposed the PCO (Penalized Comparison to Overfitting) method and proved an oracle inequality for the associated adaptive Parzen-Rosenblatt’s estimator by using a concentration inequality for the U-statistics due to Houdré and Reynaud-Bouret [11]. Together with Varet, they established the numerical efficiency of the PCO method in Varet et al. [19]. Still in the density estimation framework, the PCO method has been extended to bandwidths selection for the recursive Wolverton-Wagner estimator in Comte and Marie [3].
Comte and Marie [4] deal with an oracle inequality and numerical experiments for an adaptive Nadaraya-Watson’s estimator with a numerator and a denominator having distinct bandwidths, both selected via the PCO method. Since the output variable in a regression model has no reason to be bounded, there were significant additional difficulties, bypassed in [4], to establish an oracle inequality for the numerator’s adaptive estimator. Via similar arguments, the present article deals with an oracle inequality for \(\widehat{s}_{\widehat{K},\ell}(n;.)\), where \(\widehat{K}\) is selected via the PCO method in the spirit of Lerasle et al. [13]. As in Comte and Marie [4], one can deduce an oracle inequality for the adaptive quotient estimator \(\widehat{s}_{\widehat{K},\ell}(n;.)/\widehat{s}_{\widehat{L},1}(n;.)\) of \(\mathbb{E}(\ell(Y_{1})|X_{1}=\cdot)\), where \(\widehat{K}\) and \(\widehat{L}\) are both selected via the PCO method.
In addition to the bandwidth selection for kernel-based estimators already studied in [12, 4], the present paper covers the dimension selection for projection estimators of \(f\), \(bf\) when \(Y_{1},\dots,Y_{n}\) are defined by Model (1) with \(\ell=\textrm{Id}_{\mathbb{R}}\), and \(\sigma^{2}f\) when \(Y_{1},\dots,Y_{n}\) are defined by Model (4) with \(\ell(x)=x^{2}\) for every \(x\in\mathbb{R}\). For projection estimators, when \(d=1\), the usual model selection method (see Comte [2, Chapter 2, Section 5]) seems hard to beat. However, when \(d>1\) and \(K\) is defined by (3), \(m_{1},\dots,m_{d}\) are selected via a Goldenshluger–Lepski type method (see Chagny [1]), which has the same numerical weakness than the Goldenshluger–Lepski method for bandwidth selection when \(K\) is defined by (2). So, for the dimension selection for anisotropic projection estimators, the PCO method is interesting.
In Section 2, some examples of kernels sets are provided and a risk bound on \(\widehat{s}_{K,\ell}(n;.)\) is established. Section 3 deals with an oracle inequality for \(\widehat{s}_{\widehat{K},\ell}(n;.)\), where \(\widehat{K}\) is selected via the PCO method.
2 RISK BOUND
Throughout the paper, \(s\in\mathbb{L}^{2}(\mathbb{R}^{d})\). Let \(\mathcal{K}_{n}\) be a set of symmetric continuous maps from \(\mathbb{R}^{d}\times\mathbb{R}^{d}\) into \(\mathbb{R}\), of cardinality less or equal than \(n\), fulfilling the following assumption.
Assumption 2.1. There exists a deterministic constant \(\mathfrak{m}_{\mathcal{K},\ell}>0\), not depending on \(n\), such that
(1) for every \(K\in\mathcal{K}_{n}\),
(2) for every \(K\in\mathcal{K}_{n}\),
with
(3) for every \(K,K^{\prime}\in\mathcal{K}_{n}\),
with
(4) for every \(K\in\mathcal{K}_{n}\) and \(\psi\in\mathbb{L}^{2}(\mathbb{R}^{d})\),
The elements of \(\mathcal{K}_{n}\) are called kernels. Let us provide two natural examples of kernels sets.
Proposition 2.2. Consider
where \(k\) is a symmetric kernel (in the usual sense), \(h_{\textrm{min}}\in[n^{-1/d},1]\) and \(\mathcal{H}(h_{\textrm{min}})\) is a finite subset of \([h_{\textrm{min}},1]\). The kernels set \(\mathcal{K}_{k}(h_{\textrm{min}})\) fulfills Assumption 2.1 and, for any \(K\in\mathcal{K}_{k}(h_{\textrm{min}})\) (i.e., defined by (2) with \(h_{1},\dots,h_{d}\in\mathcal{H}(h_{\textrm{min}})\)),
Proposition 2.3. Consider
where \(m_{\textrm{max}}^{d}\in\{1,\dots,n\}\) and, for every \(m\in\{1,\dots,n\}\), \(\mathcal{B}_{m}=\{\varphi_{1}^{m},\dots,\varphi_{m}^{m}\}\) is an orthonormal family of \(\mathbb{L}^{2}(\mathbb{R})\) such that
with \(\mathfrak{m}_{\mathcal{B}}>0\) not depending on \(m\) and \(n\), and such that one of the two following conditions is satisfied:
or
The kernels set \(\mathcal{K}_{\mathcal{B}_{1},\dots,\mathcal{B}_{n}}(m_{\textrm{max}})\) fulfills Assumption 2.1 and, for any \(K\in\mathcal{K}_{\mathcal{B}_{1},\dots,\mathcal{B}_{n}}(m_{\textrm{max}})\) (i.e., defined by (3) with \(m_{1},\dots,m_{n}\in\{1,\dots,m_{\textrm{max}}\}\)),
Remark 2.4. For the sake of simplicity, the present paper focuses on \(\mathcal{K}_{\mathcal{B}_{1},\dots,\mathcal{B}_{n}}(m_{\textrm{max}})\), but Proposition 2.3 is still true for the weighted projection kernels set
where \(w_{1},\dots,w_{n}\in[0,1]\).
Remark 2.5. Note that Condition (5) is close, but more restrictive than Condition (19) of Lerasle et al. [13], Proposition 3.2, which is that the spaces \({\textrm{span}}(\mathcal{B}_{m})\), \(m\in\mathbb{N}\) are nested. See Massart [14], Subsection 7.5.2 for examples of nested spaces. Our condition (5) is fulfilled by the trigonometric basis, Hermite’s basis or Laguerre’s basis.
Note also that in the same proposition of Lerasle et al. [13], Condition (20) coincides with our condition (6). The regular histograms basis satisfies condition (6). Indeed, by taking \(\varphi_{j}^{m}=\psi_{j}^{m}:=\sqrt{m}\mathbf{1}_{[(j-1)/m,j/m[}\) for every \(m\in\{1,\dots,n\}\) and \(j\in\{1,\dots,m\}\),
for every \(m_{1},\dots,m_{d}\in\{1,\dots,n\}\) and \(x\in\mathbb{R}^{d}\).
The following proposition shows that Legendre’s basis also fulfills condition (6).
Proposition 2.6. For every \(m\in\{1,\dots,n\}\) and \(j\in\{1,\dots,m\}\), let \(\xi_{j}^{m}\) be the function defined on \([-1,1]\) by
where
is the \(j\)th Legendre’s polynomial. If \(f\in C^{2d}([0,1]^{d})\) and \(\mathcal{B}_{m}=\{\xi_{1}^{m},\dots,\xi_{m}^{m}\}\) for every \(m\in\{1,\cdots,n\}\), then \(\mathcal{K}_{\mathcal{B}_{1},\dots,\mathcal{B}_{m}}(m_{\textrm{max}})\) fulfills condition (6).
The following proposition provides a suitable control of the variance of \(\widehat{s}_{K,\ell}(n;.)\).
Proposition 2.7. Under Assumptions 2.1.(1)–2.1.(3), if \(s\in\mathbb{L}^{2}(\mathbb{R}^{d})\) and if there exists \(\alpha>0\) such that \(\mathbb{E}(\exp(\alpha|\ell(Y_{1})|))<\infty\), then there exists a deterministic constant \(\mathfrak{c}_{2.7}>0\), not depending on \(n\), such that for every \(\theta\in]0,1[\),
Finally, let us state the main result of this section.
Theorem 2.8. Under Assumption 2.1, if \(s\in\mathbb{L}^{2}(\mathbb{R}^{d})\) and if there exists \(\alpha>0\) such that \(\mathbb{E}(\exp(\alpha|\ell(Y_{1})|))<\infty\), then there exist deterministic constants \(\mathfrak{c}_{2.8},\overline{\mathfrak{c}}_{2.8}>0\), not depending on \(n\), such that for every \(\theta\in]0,1[\),
and
Remark 2.9. Note that the first inequality in Theorem 2.8 gives a risk bound on the estimator \(\widehat{s}_{K,\ell}(n;.)\):
for every \(\theta\in]0,1[\). The second inequality is useful in order to establish a risk bound on the adaptive estimator defined in the next section (see Theorem 3.2).
Remark 2.10. In Proposition 2.7 and Theorem 2.8, the exponential moment condition may appear too strong. Nevertheless, this is de facto satisfied when
This last condition is satisfied in the density estimation framework because \(\ell=1\), but even in the nonparametric regression framework, where \(\ell\) is not bounded, when \(Y_{1},\dots,Y_{n}\) have a compactly supported distribution. Moreover, note that under condition (7), the risk bounds of Theorem 2.8 can be stated in deviation, without additional steps in the proof. Precisely, under Assumption 2.1 and condition (7), if \(s\in\mathbb{L}^{2}(\mathbb{R}^{d})\), then there exists a deterministic constant \(\mathfrak{c}_{L}>0\), depending on \(L=\sup_{z\in{\textrm{supp}}(\mathbb{P}_{Y_{1}})}|\ell(z)|\) but not on \(n\), such that for every \(\vartheta\in]0,1[\) and \(\lambda>0\),
with probability larger than \(1-9.4|\mathcal{K}_{n}|e^{-\lambda}\).
When condition (7) doesn’t hold true, one can replace the exponential moment condition of Proposition 2.7 and Theorem 2.8 by a \(q\)th order moment condition on \(\ell(Y_{1})\) (\(q\in\mathbb{N}^{*}\)), but with a damaging effect on the rate of convergence of \(\widehat{s}_{K,\ell}(n;.)\). For instance, at Remark B.5, it is established that under a \((12-4\varepsilon)/\beta\)th moment condition (\(\varepsilon\in]0,1[\) and \(0<\beta<\varepsilon/2\)), the rate of convergence is of order \(O(1/n^{1-\varepsilon})\) (instead of \(1/n\)) in Lemma B.2. This holds true for the three technical lemmas of Appendix B.1, and then for Proposition 2.7 and Theorem 2.8.
3 KERNEL SELECTION
This section deals with a risk bound on the adaptive estimator \(\widehat{s}_{\widehat{K},\ell}(n;.)\), where
\(K_{0}\) is an overfitting proposal for \(K\) in the sense that
and
Example. On the one hand, for any \(K\in\mathcal{K}_{k}(h_{\textrm{min}})\) (i.e., defined by (2) with \(h_{1},\dots,h_{d}\in\mathcal{H}(h_{\textrm{min}})\)),
Then, for \(\mathcal{K}_{n}=\mathcal{K}_{k}(h_{\textrm{min}})\),
On the other hand, for any \(K\in\mathcal{K}_{\mathcal{B}_{1},\dots,\mathcal{B}_{n}}(m_{\textrm{max}})\) (i.e., defined by (3) with \(m_{1},\dots,m_{n}\in\{1,\dots,m_{\textrm{max}}\}\)),
Then, for \(\mathcal{K}_{n}=\mathcal{K}_{\mathcal{B}_{1},\dots,\mathcal{B}_{n}}(m_{\textrm{max}})\), at least for the usual bases mentioned at Remark 2.5,
In the sequel, in addition to Assumption 2.1, the kernels set \(\mathcal{K}_{n}\) fulfills the following assumption.
Assumption 3.1. There exists a deterministic constant \(\overline{\mathfrak{m}}_{\mathcal{K},\ell}>0\), not depending on \(n\), such that
The following theorem provides an oracle inequality for the adaptive estimator \(\widehat{s}_{\widehat{K},\ell}(n;.)\).
Theorem 3.2. Under Assumptions 2.1 and 3.1, if \(s\in\mathbb{L}^{2}(\mathbb{R}^{d})\) and if there exists \(\alpha>0\) such that \(\mathbb{E}(\exp(\alpha|\ell(Y_{1})|))<\infty\), then there exists a deterministic constant \(\mathfrak{c}_{3.2}>0\), not depending on \(n\), such that for every \(\vartheta\in]0,1[\),
Remark 3.3. As mentioned in Comte and Marie [4, p. 6], when \(\mathcal{K}_{n}=\mathcal{K}_{k}(h_{\textrm{min}})\), if \(s\) belongs to a Nikol’skii ball and \(h_{\textrm{min}}=1/n\), then Theorem 3.2 says that the PCO estimator has a performance of same order than \(O_{n}:=\min_{K\in\mathcal{K}_{n}}\mathbb{E}(||\widehat{s}_{K,\ell}(n;.)-s||_{2}^{2})\) up to a factor \(1+\vartheta\). When \(\mathcal{K}_{n}=\mathcal{K}_{\mathcal{B}_{1},\dots,\mathcal{B}_{n}}(m_{\textrm{max}})\), it depends on the bases \(\mathcal{B}_{1},\dots,\mathcal{B}_{n}\). For instance, with the same ideas than in Comte and Marie [4], thanks to DeVore and Lorentz [6, Theorem 2.3 p. 205], if \(s\) belongs to a Sobolev space and \(m_{\textrm{max}}=n\), then our Theorem 3.2 also says that the PCO estimator has a performance of same order than \(O_{n}\).
Notation. For any \(B\in\mathcal{B}(\mathbb{R}^{d})\), \(||.||_{2,f,B}\) is the norm on \(\mathbb{L}^{2}(B,f(x)\lambda_{d}(dx))\) defined by
The following corollary provides an oracle inequality for \(\widehat{s}_{\widehat{K},\ell}(n;.)/\widehat{s}_{\widehat{L},1}(n;.)\), where \(\widehat{K}\) and \(\widehat{L}\) are both selected via the PCO method.
Corollary 3.4. Let \((\beta_{j})_{j\in\mathbb{N}}\) be a decreasing sequence of elements of \(]0,\infty[\) such that \(\lim_{\infty}\beta_{j}=0\) and, for every \(j\in\mathbb{N}\), consider
Under Assumptions 2.1 and 3.1 for \(\ell\) and \(1\), if \(s,f\in\mathbb{L}^{2}(\mathbb{R}^{d})\) and if there exists \(\alpha>0\) such that \(\mathbb{E}(\exp(\alpha|\ell(Y_{1})|))<\infty\), then there exists a deterministic constant \(\mathfrak{c}_{3.2}>0\), not depending on \(n\), such that for every \(\vartheta\in]0,1[\),
where
and
The proof of Corollary 3.4 is the same than the proof of Comte and Marie [4], Corollary 4.3.
Finally, let us discuss about Assumption 3.1. This assumption is difficult to check in practice, then let us provide a sufficient condition.
Assumption 3.5. The function \(s\) is bounded and
doesn’t depend on \(n\).
Under Assumption 3.5, \(\mathcal{K}_{n}\) fulfills Assumption 3.1. Indeed,
Note that in the nonparametric regression framework (see Model (1)), to assume \(s\) bounded means that \(bf\) is bounded. For instance, this condition is fulfilled by the linear regression models with Gaussian inputs.
Let us provide two examples of kernels sets fulfilling Assumption 3.5, the sufficient condition for Assumption 3.1:
-
Consider \(K\in\mathcal{K}_{k}(h_{\textrm{min}})\). Then, there exist \(h_{1},\dots,h_{d}\in\mathcal{H}(h_{\textrm{min}})\) such that
$$K(x^{\prime},x)=\prod_{q=1}^{d}\frac{1}{h_{q}}k\left(\frac{x_{q}^{\prime}-x_{q}}{h_{q}}\right);\quad\forall x,x^{\prime}\in\mathbb{R}^{d}.$$Clearly, \(||K(x^{\prime},.)||_{1}=||k||_{1}^{d}\) for every \(x^{\prime}\in\mathbb{R}^{d}\). So, for \(\mathcal{K}_{n}=\mathcal{K}_{k}(h_{\textrm{min}})\), \(\mathfrak{m}_{\mathcal{K}}\leqslant||k||_{1}^{2d}\).
-
For \(\mathcal{K}_{n}=\mathcal{K}_{\mathcal{B}_{1},\dots,\mathcal{B}_{n}}(m_{\textrm{max}})\), the condition on \(\mathfrak{m}_{\mathcal{K}}\) seems harder to check in general. Let us show that it is satisfied for the regular histograms basis defined in Section 2. For every \(m_{1},\dots,m_{d}\in\{1,\dots,n\}\),
$$\left|\left|\prod_{q=1}^{d}\sum_{j=1}^{m_{q}}\psi_{j}^{m_{q}}(x_{q}^{\prime})\psi_{j}^{m_{q}}(.)\right|\right|_{1}\leqslant\prod_{q=1}^{d}\left(m_{q}\sum_{j=1}^{m_{q}}\mathbf{1}_{[(j-1)/m_{q},j/m_{q}[}(x_{q}^{\prime})\int\limits_{(j-1)/m_{q}}^{j/m_{q}}dx\right)\leqslant 1.$$
Now, let us show that even if it doesn’t fulfill Assumption 3.5, the trigonometric basis fulfills Assumption 3.1.
Proposition 3.6. Consider \(\chi_{1}:=\mathbf{1}_{[0,1]}\) and, for every \(j\in\mathbb{N}^{*}\), the functions \(\chi_{2j}\) and \(\chi_{2j+1}\) defined on \(\mathbb{R}\) by
If \(s\in C^{2}(\mathbb{R}^{d})\) and \(\mathcal{B}_{m}=\{\chi_{1},\dots,\chi_{m}\}\) for every \(m\in\{1,\dots,n\}\), then \(\mathcal{K}_{\mathcal{B}_{1},\dots,\mathcal{B}_{n}}(m_{\textrm{max}})\) fulfills Assumption 3.1.
REFERENCES
G. Chagny, ‘‘Warped Bases for Conditional Density Estimation,’’ Math. Methods Statist. 22, 253–282 (2013).
F. Comte, ‘‘Estimation non-paramétrique,’’ Spartacus IDH (2014).
F. Comte and N. Marie, ‘‘Bandwidth selection for the Wolverton–Wagner estimator,’’ Journal of Statistical Planning and Inference 207, 198–214 (2020).
F. Comte and N. Marie, ‘‘On a Nadaraya–Watson estimator with two bandwidths,’’ Submitted (2020).
F. Comte and T. Rebafka, ‘‘Nonparametric weighted estimators for Biased data,’’ Journal of Statistical Planning and Inference 174, 104–128 (2016).
R. A. DeVore and G. G. Lorentz, Constructive Approximation (Springer-Verlag, 1993).
U. Einmahl and D. M. Mason, ‘‘An empirical process approach to the uniform consistency of kernel-type function estimators,’’ Journal of Theoretical Probability 13, 1–37 (2000).
U. Einmahl and D. M. Mason, ‘‘Uniform in Bandwidth Consistency of Kernel-Type Function Estimators,’’ Annals of Statistics 33, 1380–1403 (2005).
E. Giné and R. Nickl, Mathematical Foundations of Infinite-Dimensional Statistical Models (Cambridge University Press, 2015).
A. Goldenshluger and O. Lepski, ‘‘Bandwidth selection in kernel density estimation: Oracle inequalities and adaptive minimax optimality,’’ The Annals of Statistics 39, 1608–1632 (2011).
C. Houdré and P. Reynaud-Bouret, Exponential Inequalities, with Constants, for U-statistics of Order Two. Stochastic Inequalities and Applications, vol. 56 of Progr. Proba., Birkhauser, 2003, pp. 55–69.
C. Lacour, P. Massart, and V. Rivoirard, ‘‘Estimator Selection: a New Method with Applications to Kernel Density Estimation,’’ Sankhya A 79 (2), 298–335 (2017).
M. Lerasle, N.M. Magalhaes, and P. Reynaud-Bouret. Optimal Kernel Selection for Density Estimation. High Dimensional Probabilities VII: The Cargese Volume, vol. 71 of Prog. Proba., Birkhauser, 2016, pp. 435–460.
P. Massart, Concentration Inequalities and Model Selection. Lecture Notes in Mathematics 1896 (Springer, 2007).
E.A. Nadaraya, ‘‘On a regression estimate,’’ (Russian) Verojatnost. i Primenen. 9, 157–159 (1964).
E. Parzen, ‘‘On the estimation of a probability density function and the mode,’’ The Annals of Mathematical Statistics 33, 1065–1076 (1962).
M. Rosenblatt, ‘‘Remarks on some nonparametric estimates of a density function,’’ The Annals of Mathematical Statistics 27, 832–837 (1956).
A. Tsybakov, Introduction to Nonparametric Estimation (Springer, 2009).
S. Varet, C. Lacour, P. Massart, and V. Rivoirard, Numerical Performance of Penalized Comparison to Overfitting for Multivariate Density Estimation. Preprint, 2020.
G. S. Watson, ‘‘Smooth regression analysis,’’ Sankhya A 26, 359–372 (1964).
Author information
Authors and Affiliations
Corresponding authors
APPENDIX A.
A. DETAILS ON KERNELS SETS: PROOFS OF PROPOSITIONS 2.2, 2.3, 2.6, AND 3.6
A.1. Proof of Proposition 2.2
Consider \(K,K^{\prime}\in\mathcal{K}_{k}(h_{\textrm{min}})\). Then, there exist \(h,h^{\prime}\in\mathcal{H}(h_{\textrm{min}})^{d}\) such that
for every \(x,x^{\prime}\in\mathbb{R}^{d}\), where
(1) For every \(x^{\prime}\in\mathbb{R}^{d}\), since \(nh_{\textrm{min}}^{d}\geqslant 1\),
(2) Since \(s_{K,\ell}=K\ast s\) and by Young’s inequality, \(||s_{K,\ell}||_{2}^{2}\leqslant||k||_{1}^{2d}||s||_{2}^{2}\).
(3) On the one hand, thanks to Eq. (A.1),
On the other hand, for every \(x,x^{\prime}\in\mathbb{R}^{d}\),
Then,
(4) For every \(\psi\in\mathbb{L}^{2}(\mathbb{R}^{d})\),
A.2. Proof of Proposition 2.3
Consider \(K,K^{\prime}\in\mathcal{K}_{\mathcal{B}_{1},\dots,\mathcal{B}_{n}}(m_{\textrm{max}})\). Then, there exist \(m,m^{\prime}\in\{1,\dots,m_{\textrm{max}}\}^{d}\) such that
for every \(x,x^{\prime}\in\mathbb{R}^{d}\).
(1) For every \(x^{\prime}\in\mathbb{R}^{d}\), since \(m_{\textrm{max}}^{d}\leqslant n\),
(2) Since
by Pythagoras theorem, \(||s_{K,\ell}||_{2}^{2}\leqslant||s||_{2}^{2}\).
(3) First of all, thanks to Eq. (A.2),
On the one hand, under condition (5) on \(\mathcal{B}_{1},\dots,\mathcal{B}_{n}\), for any \(j\in\{1,\dots,m\}\), \(\varphi_{j}^{m}\) doesn’t depend on \(m\), so it can be denoted by \(\varphi_{j}\), and then
On the other hand, under condition (6) on \(\mathcal{B}_{1},\dots,\mathcal{B}_{n}\), since \(X_{1}\) and \((X_{2},Y_{2})\) are independent, and since \(K(x,x)\geqslant 0\) for every \(x\in\mathbb{R}^{d}\),
(4) For every \(\psi\in\mathbb{L}^{2}(\mathbb{R}^{d})\),
A.3. Proof of Proposition 2.6
For the sake of readability, assume that \(d=1\). Consider \(m\in\{1,\dots,m_{\textrm{max}}\}\). Since each Legendre’s polynomial is uniformly bounded by \(1\),
Moreover, since \(Q_{j}\) is a solution to Legendre’s differential equation for any \(j\in\{1,\dots,m\}\), thanks to the integration by parts formula,
Then,
with \(\mathfrak{c}_{1}=\max\{2||f^{\prime}||_{\infty},||f^{\prime\prime}||_{\infty}\}\). So,
where \(\zeta\) is Riemann’s zeta function. Thus, Legendre’s basis satisfies condition (6).
A.4. Proof of Proposition 3.6
The proof of Proposition 3.6 relies on the following technical lemma.
Lemma A.1. For every \(x\in[0,2\pi]\) and \(p,q\in\mathbb{N}^{*}\) such that \(q>p\),
See Subsubsection A.4.1. for a proof.
For the sake of readability, assume that \(d=1\). Consider \(K,K^{\prime}\in\mathcal{K}_{\mathcal{B}_{1},\dots,\mathcal{B}_{n}}(m_{\textrm{max}})\). Then, there exist \(m,m^{\prime}\in\{1,\dots,m_{\textrm{max}}\}\) such that
First, there exist \(\mathfrak{m}_{1}(m,m^{\prime})\in\{0,\dots,n\}\) and \(\mathfrak{c}_{1}>0\), not depending on \(n\), \(K\) and \(K^{\prime}\), such that for any \(x^{\prime}\in[0,1]\),
Moreover, for any \(j\in\{2,\dots,\mathfrak{m}_{1}(m,m^{\prime})\}\),
where \(\alpha_{j}(x^{\prime}):=\sin(2\pi jx^{\prime})\) and
Then, there exists a deterministic constant \(\mathfrak{c}_{2}>0\), not depending on \(n\), \(K\), \(K^{\prime}\), and \(x^{\prime}\), such that
Let us show that each term of the right-hand side of inequality (A.3) is uniformly bounded in \(x^{\prime}\), \(m\), and \(m^{\prime}\). On the one hand,
On the other hand, for every \(x\in]0,\pi[\) such that \([\pi/x]+1\leqslant\mathfrak{m}_{1}(m,m^{\prime})\) (without loss of generality), by Lemma A.1,
Since \(x\mapsto\sin(x)\) is continuous, odd and \(2\pi\)-periodic, inequality (A.4) holds true for every \(x\in\mathbb{R}\). So,
Therefore,
A.4.1. Proof of Lemma A.1. For any \(x\in[0,2\pi]\) and \(q\in\mathbb{N}^{*}\), consider
On the one hand,
Then,
On the other hand,
Then,
and, for any \(p\in\mathbb{N}^{*}\) such that \(q>p\),
Therefore,
In conclusion,
B. PROOFS OF RISK BOUNDS
B.1. Preliminary Results
This subsection provides three lemmas used several times in the sequel.
Lemma B.1. Consider
Under Assumptions 2.1.(1)–2.1.(3), if \(s\in\mathbb{L}^{2}(\mathbb{R}^{d})\) and if there exists \(\alpha>0\) such that \(\mathbb{E}(\exp(\alpha|\ell(Y_{1})|))<\infty\), then there exists a deterministic constant \(\mathfrak{c}_{B.1}>0\), not depending on \(n\), such that for every \(\theta\in]0,1[\),
Lemma B.2. Consider
Under Assumptions 2.1.(1), 2.1.(2), if \(s\in\mathbb{L}^{2}(\mathbb{R}^{d})\) and if there exists \(\alpha>0\) such that \(\mathbb{E}(\exp(\alpha|\ell(Y_{1})|))<\infty\), then there exists a deterministic constant \(\mathfrak{c}_{B.2}>0\), not depending on \(n\), such that for every \(\theta\in]0,1[\),
Lemma B.3. Consider
Under Assumptions 2.1.(1), 2.1.(2), 2.1.(4), if \(s\in\mathbb{L}^{2}(\mathbb{R}^{d})\) and if there exists \(\alpha>0\) such that \(\mathbb{E}(\exp(\alpha|\ell(Y_{1})|))<\infty\), then there exists a deterministic constant \(\mathfrak{c}_{B.3}>0\), not depending on \(n\), such that for every \(\theta\in]0,1[\),
B.1.1. Proof of Lemma B.1. The proof of Lemma B.1 relies on the following concentration inequality for U-statistics, proved in dimension \(1\) in Houdré and Reynaud-Bouret [11] first, and then extended to the infinite-dimensional framework by Giné and Nickl [9].
Lemma B.4. Let \(\xi_{1},\dots,\xi_{n}\) be i.i.d. random variables on a Polish space \(\Xi\) equipped with its Borel \(\sigma\)-algebra. Let \(f_{i,j}\), \(1\leqslant i\not=j\leqslant n\), be some bounded and symmetric measurable maps from \(\Xi^{2}\) into \(\mathbb{R}\) such that, for every \(i\not=j\),
Consider the totally degenerate second order U-statistic
There exists a universal constant \(\mathfrak{m}>0\) such that for every \(\lambda>0\),
where
with
See Giné and Nickl [9], Theorem 3.4.8 for a proof.
Consider \(\mathfrak{m}(n):=8\log(n)/\alpha\). For any \(K,K^{\prime}\in\mathcal{K}_{n}\),
where
with, for every \((x^{\prime},y),(x^{\prime\prime},y^{\prime})\in E=\mathbb{R}^{d}\times\mathbb{R}\),
and, for every \(k\in\mathcal{K}_{n}\),
On the one hand, since \(\mathbb{E}(g_{K,K^{\prime},\ell}^{1}(n;x^{\prime},y,X_{1},Y_{1}))=0\) for every \((x^{\prime},y)\in E\), by Lemma B.4, there exists a universal constant \(\mathfrak{m}\geqslant 1\) such that for any \(\lambda>0\), with probability larger than \(1-5.4e^{-\lambda}\),
where the constants \(\mathfrak{a}_{K,K^{\prime},\ell}(n)\), \(\mathfrak{b}_{K,K^{\prime},\ell}(n)\), \(\mathfrak{c}_{K,K^{\prime},\ell}(n)\), and \(\mathfrak{d}_{K,K^{\prime},\ell}(n)\) are defined and controlled later. First, note that
where
and
for every \(k,k^{\prime}\in\mathcal{K}_{n}\) and \((x^{\prime},y),(x^{\prime\prime},y^{\prime})\in E\). Let us now control \(\mathfrak{a}_{K,K^{\prime},\ell}(n)\), \(\mathfrak{b}_{K,K^{\prime},\ell}(n)\), \(\mathfrak{c}_{K,K^{\prime},\ell}(n)\), and \(\mathfrak{d}_{K,K^{\prime},\ell}(n)\).
-
The constant \(\mathfrak{a}_{K,K^{\prime},\ell}(n)\). Consider
$$\mathfrak{a}_{K,K^{\prime},\ell}(n):=\sup_{(x^{\prime},y),(x^{\prime\prime},y^{\prime})\in E}|g_{K,K^{\prime},\ell}^{1}(n;x^{\prime},y,x^{\prime\prime},y^{\prime})|.$$By (B.3), Cauchy–Schwarz’s inequality and Assumption 2.1.(1),
$$\mathfrak{a}_{K,K^{\prime},\ell}(n)\leqslant 4\sup_{(x^{\prime},y),(x^{\prime\prime},y^{\prime})\in E}|\langle K(x^{\prime},.)\ell(y)\mathbf{1}_{|\ell(y)|\leqslant\mathfrak{m}(n)},K^{\prime}(x^{\prime\prime},.)\ell(y^{\prime})\mathbf{1}_{|\ell(y^{\prime})|\leqslant\mathfrak{m}(n)}\rangle_{2}|$$$${}\leqslant 4\mathfrak{m}(n)^{2}\left(\sup_{x^{\prime}\in\mathbb{R}^{d}}||K(x^{\prime},.)||_{2}\right)\left(\sup_{x^{\prime\prime}\in\mathbb{R}^{d}}||K^{\prime}(x^{\prime\prime},.)||_{2}\right)\leqslant 4\mathfrak{m}_{\mathcal{K},\ell}\mathfrak{m}(n)^{2}n.$$So,
$$\frac{1}{n^{2}}\mathfrak{a}_{K,K^{\prime},\ell}(n)\lambda^{2}\leqslant\frac{4}{n}\mathfrak{m}_{\mathcal{K},\ell}\mathfrak{m}(n)^{2}\lambda^{2}.$$ -
The constant \(\mathfrak{b}_{K,K^{\prime},\ell}(n)\). Consider
$$\mathfrak{b}_{K,K^{\prime},\ell}(n)^{2}:=n\sup_{(x^{\prime},y)\in E}\mathbb{E}(g_{K,K^{\prime},\ell}^{1}(n;x^{\prime},y,X_{1},Y_{1})^{2}).$$By (B.3), Jensen’s inequality, Cauchy–Schwarz’s inequality and Assumption 2.1.(1),
$$\mathfrak{b}_{K,K^{\prime},\ell}(n)^{2}\leqslant 16n\sup_{(x^{\prime},y)\in E}\mathbb{E}(\langle K(x^{\prime},.)\ell(y)\mathbf{1}_{|\ell(y)|\leqslant\mathfrak{m}(n)},K^{\prime}(X_{1},.)\ell(Y_{1})\mathbf{1}_{|\ell(Y_{1})|\leqslant\mathfrak{m}(n)}\rangle_{2}^{2})$$$${}\leqslant 16n\mathfrak{m}(n)^{2}\sup_{x^{\prime}\in\mathbb{R}^{d}}||K(x^{\prime},.)||_{2}^{2}\mathbb{E}(||K^{\prime}(X_{1},.)\ell(Y_{1})\mathbf{1}_{|\ell(Y_{1})|\leqslant\mathfrak{m}(n)}||_{2}^{2})\leqslant 16\mathfrak{m}_{\mathcal{K},\ell}n^{2}\mathfrak{m}(n)^{2}\overline{s}_{K^{\prime},\ell}.$$So, for any \(\theta\in]0,1[\),
$$\frac{1}{n^{2}}\mathfrak{b}_{K,K^{\prime},\ell}(n)\lambda^{3/2}\leqslant 2\left(\frac{3\mathfrak{m}}{\theta}\right)^{1/2}\frac{2}{n^{1/2}}\mathfrak{m}_{\mathcal{K},\ell}^{1/2}\mathfrak{m}(n)\lambda^{3/2}\times\left(\frac{\theta}{3\mathfrak{m}}\right)^{1/2}\frac{1}{n^{1/2}}\overline{s}_{K^{\prime},\ell}^{1/2}$$$${}\leqslant\frac{\theta}{3\mathfrak{m}n}\overline{s}_{K^{\prime},\ell}+\frac{12\mathfrak{m}\lambda^{3}}{\theta n}\mathfrak{m}_{\mathcal{K},\ell}\mathfrak{m}(n)^{2}.$$ -
The constant \(\mathfrak{c}_{K,K^{\prime},\ell}(n)\). Consider
$$\mathfrak{c}_{K,K^{\prime},\ell}(n)^{2}:=n^{2}\mathbb{E}(g_{K,K^{\prime},\ell}^{1}(n;X_{1},Y_{1},X_{2},Y_{2})^{2}).$$By (B.3), Jensen’s inequality and Assumption 2.1.(3),
$$\mathfrak{c}_{K,K^{\prime},\ell}(n)^{2}\leqslant 16n^{2}\mathbb{E}(\langle K(X_{1},.)\ell(Y_{1})\mathbf{1}_{|\ell(Y_{1})|\leqslant\mathfrak{m}(n)},K^{\prime}(X_{2},.)\ell(Y_{2})\mathbf{1}_{|\ell(Y_{2})|\leqslant\mathfrak{m}(n)}\rangle_{2}^{2})$$$${}\leqslant 16n^{2}\mathfrak{m}(n)^{2}\mathbb{E}(\langle K(X_{1},.),K^{\prime}(X_{2},.)\ell(Y_{2})\rangle_{2}^{2})\leqslant 16\mathfrak{m}_{\mathcal{K},\ell}n^{2}\mathfrak{m}(n)^{2}\overline{s}_{K^{\prime},\ell}.$$So,
$$\frac{1}{n^{2}}\mathfrak{c}_{K,K^{\prime},\ell}(n)\lambda^{1/2}\leqslant\frac{\theta}{3\mathfrak{m}n}\overline{s}_{K^{\prime},\ell}+\frac{12\mathfrak{m}\lambda}{\theta n}\mathfrak{m}_{\mathcal{K},\ell}\mathfrak{m}(n)^{2}.$$ -
The constant \(\mathfrak{d}_{K,K^{\prime},\ell}(n)\). Consider
$$\mathfrak{d}_{K,K^{\prime},\ell}(n):=\sup_{(a,b)\in\mathcal{A}}\mathbb{E}\left[\sum_{i<j}a_{i}(X_{i},Y_{i})b_{j}(X_{j},Y_{j})g_{K,K^{\prime},\ell}^{1}(n;X_{i},Y_{i},X_{j},Y_{j})\right],$$where
$$\mathcal{A}:=\left\{(a,b):\sum_{i=1}^{n-1}\mathbb{E}(a_{i}(X_{i},Y_{i})^{2})\leqslant 1\quad\textrm{and}\quad\sum_{j=2}^{n}\mathbb{E}(b_{j}(X_{j},Y_{j})^{2})\leqslant 1\right\}.$$By (B.3), Jensen’s inequality, Cauchy-Schwarz’s inequality and Assumption 2.1.(3),
$$\mathfrak{d}_{K,K^{\prime},\ell}(n)\leqslant 4\sup_{(a,b)\in\mathcal{A}}\mathbb{E}\left[\sum_{i=1}^{n-1}\sum_{j=i+1}^{n}|a_{i}(X_{i},Y_{i})b_{j}(X_{j},Y_{j})\varphi_{K,K^{\prime},\ell}(n;X_{i},Y_{i},X_{j},Y_{j})|\right]$$$${}\leqslant 4n\mathfrak{m}(n)\mathbb{E}(\langle K(X_{1},.),K^{\prime}(X_{2},.)\ell(Y_{2})\rangle_{2}^{2})^{1/2}\leqslant 4\mathfrak{m}_{\mathcal{K},\ell}^{1/2}n\mathfrak{m}(n)\overline{s}_{K^{\prime},\ell}^{1/2}.$$So,
$$\frac{1}{n^{2}}\mathfrak{d}_{K,K^{\prime},\ell}(n)\lambda\leqslant\frac{\theta}{3\mathfrak{m}n}\overline{s}_{K^{\prime},\ell}+\frac{12\mathfrak{m}\lambda^{2}}{\theta n}\mathfrak{m}_{\mathcal{K},\ell}\mathfrak{m}(n)^{2}.$$
Then, since \(\mathfrak{m}\geqslant 1\) and \(\lambda>0\), with probability larger than \(1-5.4e^{-\lambda}\),
So, with probability larger than \(1-5.4|\mathcal{K}_{n}|^{2}e^{-\lambda}\),
For every \(t\in\mathbb{R}_{+}\), consider
Then, for any \(T>0\),
Moreover,
So, by taking
and since \(|\mathcal{K}_{n}|\leqslant n\),
On the other hand, by Assumption 2.1.(1), Cauchy–Schwarz’s inequality and Markov’s inequality,
with
So,
and, symmetrically,
By Assumption 2.1.(1), Cauchy–Schwarz’s inequality and Markov’s inequality,
with
So,
Therefore,
B.1.2. Proof of Lemma B.2. First, the two following results are used several times in the sequel:
and
Consider \(\mathfrak{m}(n):=2\log(n)/\alpha\) and
where
with, for every \((x^{\prime},y)\in E\),
and
On the one hand, by Bernstein’s inequality, for any \(\lambda>0\), with probability larger than \(1-2e^{-\lambda}\),
where
Moreover,
by inequality (B.4), and
by inequality (B.4) and equality (B.5). Then, for any \(\theta\in]0,1[\),
with probability larger than \(1-2e^{-\lambda}\). So, with probability larger than \(1-2|\mathcal{K}_{n}|e^{-\lambda}\),
For every \(t\in\mathbb{R}_{+}\), consider
Then, for any \(T>0\),
Moreover,
So, by taking
and since \(|\mathcal{K}_{n}|\leqslant n\),
On the other hand, by inequality (B.4) and Markov’s inequality,
with
Therefore,
and, by equality (B.5), the definition of \(v_{K,\ell}(n)\) and Assumption 2.1.(2),
Remark B.5. As mentioned in Remark 2.10, replacing the exponential moment condition by the weaker \(q\)-th moment condition with \(q=(12-4\varepsilon)/\beta\), \(\varepsilon\in]0,1[\) and \(0<\beta<\varepsilon/2\), allows to get a rate of convergence of order \(1/n^{1-\varepsilon}\). Indeed, by inequality (B.6), with \(\mathfrak{m}(n)=n^{\beta}\) and
and by letting \(\alpha=1+2\beta-\varepsilon\), there exist \(n_{\varepsilon,\alpha}\in\mathbb{N}^{*}\) and \(\mathfrak{c}_{\varepsilon,\alpha}>0\) not depending on \(n\), such that for any \(n\geqslant n_{\varepsilon,\alpha}\),
Furthermore, by Markov’s inequality,
So, as previously, there exists a deterministic constant \(\mathfrak{c}_{2}>0\) such that
and then
B.1.3. Proof of Lemma B.3. Consider \(\mathfrak{m}(n)=12\log(n)/\alpha\). For any \(K,K^{\prime}\in\mathcal{K}_{n}\),
where
with, for every \((x^{\prime},y)\in E\),
and
On the one hand, by Bernstein’s inequality, for any \(\lambda>0\), with probability larger than \(1-2e^{-\lambda}\),
where
Moreover,
by Assumption 2.1.(1) and
by Assumption 2.1.(4). Then, since \(\lambda>0\), for any \(\theta\in]0,1[\),
with probability larger than \(1-2e^{-\lambda}\). So, with probability larger than \(1-2|\mathcal{K}_{n}|^{2}e^{-\lambda}\),
For every \(t\in\mathbb{R}_{+}\), consider
Then, for any \(T>0\),
Moreover,
So, by taking
and since \(|\mathcal{K}_{n}|\leqslant n\),
On the other hand, by Assumption 2.1.(2), 2.1.(4), Cauchy–Schwarz’s inequality and Markov’s inequality,
with
Therefore,
with \(\mathfrak{c}_{4}=(2^{3}+2\mathfrak{c}_{1})\mathfrak{c}_{2}+\mathfrak{c}_{3}\).
B.2. Proof of Proposition 2.7
For any \(K\in\mathcal{K}_{n}\),
with \(U_{K,\ell}(n)=U_{K,K,\ell}(n)\) and \(V_{K,\ell}(n)=V_{K,K,\ell}(n)\). Then, by Lemmas B.1 and B.2,
with \(\mathfrak{c}_{2.7}=\mathfrak{c}_{B.1}+\mathfrak{c}_{B.2}\).
B.3. Proof of Theorem 2.8
On the one hand, for every \(K\in\mathcal{K}_{n}\),
can be written
where \(W_{K,\ell}(n):=W_{K,K,\ell}(n)\) (see (14)). Then, by Proposition 2.7 and Lemma B.3,
with \(\mathfrak{c}_{2.8}=\mathfrak{c}_{2.7}+\mathfrak{c}_{B.3}\). On the other hand, for any \(K\in\mathcal{K}_{n}\),
Then,
where
By equalities (B.5) and (B.7),
with \(U_{K,\ell}(n)=U_{K,K,\ell}(n)\) (see (B.1)). By Lemmas B.1 and B.2, there exists a deterministic constant \(\mathfrak{c}_{1}>0\), not depending \(n\) and \(\theta\), such that
By Lemma B.3,
Therefore,
with \(\overline{\mathfrak{c}}_{2.8}=\mathfrak{c}_{B.3}+\mathfrak{c}_{1}\).
B.4. Proof of Theorem 3.2
The proof of Theorem 3.2 is dissected in three steps.
Step 1. This first step is devoted to provide a suitable decomposition of
First,
From (8), it follows that for any \(K\in\mathcal{K}_{n}\),
where
Let’s complete the decomposition of \(||\widehat{s}_{\widehat{K},\ell}(n;\cdot)-s||_{2}^{2}\) by writing
where
Step 2. In this step, we give controls of the quantities
-
By Lemma B.1, for any \(\theta\in]0,1[\),
$$\mathbb{E}(|\psi_{1,n}(K)|)\leqslant\frac{\theta}{n}\overline{s}_{K,\ell}+\mathfrak{c}_{B.1}\frac{\log(n)^{5}}{\theta n}$$and
$$\mathbb{E}(|\psi_{1,n}(\widehat{K})|)\leqslant\frac{\theta}{n}\mathbb{E}(\overline{s}_{\widehat{K},\ell})+\mathfrak{c}_{B.1}\frac{\log(n)^{5}}{\theta n}.$$ -
On the one hand, for any \(K,K^{\prime}\in\mathcal{K}_{n}\), consider
$$\Psi_{2,n}(K,K^{\prime}):=\dfrac{1}{n}\displaystyle\sum_{i=1}^{n}\ell(Y_{i})\langle K(X_{i},.),s_{K^{\prime},\ell}\rangle_{2}.$$Then, by Assumption 3.1,
$$\mathbb{E}\left(\sup_{K,K^{\prime}\in\mathcal{K}_{n}}|\Psi_{2,n}(K,K^{\prime})|\right)\leqslant\mathbb{E}(\ell(Y_{1})^{2})^{1/2}\mathbb{E}\left(\sup_{K,K^{\prime}\in\mathcal{K}_{n}}\langle K(X_{1},.),s_{K^{\prime},\ell}\rangle_{2}^{2}\right)^{1/2}$$$${}\leqslant\overline{\mathfrak{m}}_{\mathcal{K},\ell}^{1/2}\mathbb{E}(\ell(Y_{1})^{2})^{1/2}.$$On the other hand, by Assumption 2.1.(2),
$$|\langle s_{K,\ell},s_{K_{0},\ell}\rangle_{2}|\leqslant\mathfrak{m}_{\mathcal{K},\ell}.$$Then, there exists a deterministic constant \(\mathfrak{c}_{1}>0\), not depending on \(n\) and \(K\), such that
$$\mathbb{E}(|\psi_{2,n}(K)|)\leqslant\frac{\mathfrak{c}_{1}}{n}\quad\textrm{and}\quad\mathbb{E}(|\psi_{2,n}(\widehat{K})|)\leqslant\frac{\mathfrak{c}_{1}}{n}.$$ -
By Lemma B.3,
$$\mathbb{E}(|\psi_{3,n}(K)|)\leqslant\dfrac{\theta}{4}(||s_{K,\ell}-s||_{2}^{2}+||s_{K_{0},\ell}-s||_{2}^{2})+8\mathfrak{c}_{B.3}\frac{\log(n)^{4}}{\theta n}$$$${}+\left(\dfrac{\theta}{2}\right)^{1/2}||s_{K,\ell}-s||_{2}\times\left(\dfrac{2}{\theta}\right)^{1/2}||s_{K_{0},\ell}-s||_{2}$$$${}\leqslant\dfrac{\theta}{2}||s_{K,\ell}-s||_{2}^{2}+\left(\dfrac{\theta}{4}+\dfrac{1}{\theta}\right)||s_{K_{0},\ell}-s||_{2}^{2}+8\mathfrak{c}_{B.3}\frac{\log(n)^{4}}{\theta n}$$and
$$\mathbb{E}(|\psi_{3,n}(\widehat{K})|)\leqslant\frac{\theta}{2}\mathbb{E}(||s_{\widehat{K},\ell}-s||_{2}^{2})+\left(\dfrac{\theta}{4}+\dfrac{1}{\theta}\right)||s_{K_{0},\ell}-s||_{2}^{2}+8\mathfrak{c}_{B.3}\frac{\log(n)^{4}}{\theta n}.$$
Step 3. By the previous step, there exists a deterministic constant \(\mathfrak{c}_{2}>0\), not depending on \(n\), \(\theta\), \(K\), and \(K_{0}\), such that
and
Then, by Theorem 2.8,
and
By decomposition (B.8), there exist two deterministic constants \(\mathfrak{c}_{3},\mathfrak{c}_{4}>0\), not depending on \(n\), \(\theta\), \(K\), and \(K_{0}\), such that
This concludes the proof.
ACKNOWLEDGMENTS
The authors want also to thank Fabienne Comte for her careful reading and advices.
FUNDING
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement no. 811017.
About this article
Cite this article
Halconruy, H., Marie, N. Kernel Selection in Nonparametric Regression. Math. Meth. Stat. 29, 32–56 (2020). https://doi.org/10.3103/S1066530720010044
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.3103/S1066530720010044