Kick-One-Out-Based Variable Selection Method Using Ridge-Type $C_{p}$ Criterion in High-Dimensional Multi-response Linear Regression Models

Abstract

In this paper, the kick-one-out method using a ridge-type $C_{p}$ criterion is proposed for variable selection in multi-response linear regression models. Sufficient conditions for the consistency of this method are obtained under a high-dimensional asymptotic framework such that the number of explanatory variables and response variables, k and p, may go to infinity with the sample size n, and p may exceed n but k is less than n. It is expected that the method satisfying these sufficient conditions has a high probability of selecting the true model, even when $p>n$.

Appendices

Appendix 1: Proof of Theorem 1

First, we consider the case of $\ell \notin j_{*}$. Note that $(\boldsymbol{P}_{\omega }-\boldsymbol{P}_{\omega _{\ell }})\boldsymbol{X}_{j_{*}}=\boldsymbol{O}_{n,k_{j_{*}}}$ holds. Let $\boldsymbol{W}=\boldsymbol{\mathcal {E}}'(\boldsymbol{I}_{n}-\boldsymbol{P}_{\omega })\boldsymbol{\mathcal {E}}$ and $\boldsymbol{V}_{\omega ,\omega _{\ell }}=\boldsymbol{\mathcal {E}}'(\boldsymbol{P}_{\omega }-\boldsymbol{P}_{\omega _{\ell }})\boldsymbol{\mathcal {E}}$. Then, the upper bound of $\mathcal {D}_{\ell }$ can be written as

$$\begin{aligned} \mathcal {D}_{\ell }&= \mathrm{{tr}}(\boldsymbol{V}_{\omega ,\omega _{\ell }}\boldsymbol{S}^{-1}_{\lambda })-p\alpha \le \lambda (n-k)\frac{\mathrm{{tr}}(\boldsymbol{V}_{\omega ,\omega _{\ell }})}{\mathrm{{tr}}(\boldsymbol{W})}-p\alpha . \end{aligned}$$

(10)

Let $E_{1}$ be the event defined by $E_{1}=\{(n-k)^{-1}\mathrm{{tr}}(\boldsymbol{W}) \ge \tau \mathrm{{tr}}(\boldsymbol{\varSigma }_{*})\}$. Using (10) and the event $E_{1}$, we have

$$\begin{aligned} P\left( \cup _{\ell \notin j_{*}}\{\mathcal {D}_{\ell }\ge 0\}\right) \le \sum _{\ell \in j^{c}_{*}}P\left( \mathrm{{tr}}(\boldsymbol{V}_{\omega ,\omega _{\ell }}) \ge \lambda ^{-1}\tau p\alpha \mathrm{{tr}}(\boldsymbol{\varSigma }_{*})\right) +P(E^{c}_{1}). \end{aligned}$$

(11)

Applying (i) and (iii) of Lemma 1 in [9] to (11), the following equation can be derived:

$$\begin{aligned}&P\left( \cup _{\ell \notin j_{*}}\{\mathcal {D}_{\ell }\ge 0\}\right) \nonumber \\&\le O\left( k\xi ^{2}\mathrm{{tr}}(\boldsymbol{\varSigma }_{*})^{-2}(\lambda ^{-1}\tau p\alpha -1)^{-2}\right) +O\left( \xi ^{2}\mathrm{{tr}}(\boldsymbol{\varSigma }_{*})^{-2}n^{-1}(\tau -1)^{-2}\right) . \end{aligned}$$

(12)

Next, we consider the case of $\ell \in j_{*}$. Since $(\boldsymbol{I}_{n}-\boldsymbol{P}_{\omega })\boldsymbol{X}_{j_{*}}=\boldsymbol{O}_{n,k_{j_{*}}}$ holds, notice that

$$\begin{aligned} \mathrm{{tr}}\{\boldsymbol{Y}'(\boldsymbol{P}_{\omega }-\boldsymbol{P}_{\omega _{\ell }})\boldsymbol{Y}\}=\mathrm{{tr}}(\boldsymbol{V}_{\omega ,\omega _{\ell }}) + 2\mathrm{{tr}}(\boldsymbol{U}_{\omega _{\ell }}) + np\delta ^{2}_{\ell }, \end{aligned}$$

where $\delta ^{2}_{\ell }=\mathrm{{tr}}(\boldsymbol{\varDelta }_{\ell })$ and $\boldsymbol{U}_{\omega _{\ell }}=\boldsymbol{\varTheta }'_{j_{*}}\boldsymbol{X}'_{j_{*}}(\boldsymbol{I}_{n}-\boldsymbol{P}_{\omega _{\ell }})\boldsymbol{\mathcal {E}}$. Using this notation, the lower bound of $\mathcal {D}_{\ell }$ can be written as

$$\begin{aligned} \mathcal {D}_{\ell } \ge (1+\lambda ^{-1})^{-1}(n-k)\mathrm{{tr}}(\boldsymbol{W})^{-1}\left\{ \mathrm{{tr}}(\boldsymbol{V}_{\omega ,\omega _{\ell }}) + 2\mathrm{{tr}}(\boldsymbol{U}_{\omega _{\ell }}) + np\delta ^{2}_{\ell }\right\} -p\alpha . \end{aligned}$$

(13)

Let $E_{2}$ and $E_{3,\ell }$ be the events defined by $E_{2}=\{(n-k)^{-1}\mathrm{{tr}}(\boldsymbol{W}) \le 3\mathrm{{tr}}(\boldsymbol{\varSigma }_{*})/2\}$ and $E_{3,\ell }=\{\mathrm{{tr}}(\boldsymbol{U}_{\omega _{\ell }})\ge -np\delta ^{2}_{\ell }/4\}$. Using this notation and (13), we have

$$\begin{aligned}&P\left( \cup _{\ell \in j_{*}}\{\mathcal {D}_{\ell }\le 0\}\right) \nonumber \\&\le \sum _{\ell \in j_{*}}P\left( \mathrm{{tr}}(\boldsymbol{V}_{\omega ,\omega _{\ell }}) + np\min _{\ell \in j_{*}}\delta ^{2}_{\ell }/2 \le 3(1+\lambda ^{-1})p\alpha \mathrm{{tr}}(\boldsymbol{\varSigma }_{*})/2\right) \nonumber \\&\quad +P(E^{c}_{2})+\sum _{\ell \in j_{*}}P(E^{c}_{3,\ell })\nonumber \\&\le \sum _{\ell \in j_{*}}P\left( \mathrm{{tr}}(\boldsymbol{V}_{\omega ,\omega _{\ell }})-\mathrm{{tr}}(\boldsymbol{\varSigma }_{*})+ npc/2 \le \mathrm{{tr}}(\boldsymbol{\varSigma }_{*})\left\{ 3(1+\lambda ^{-1})p\alpha /2-1\right\} \right) \nonumber \\&\quad +P(E^{c}_{2})+\sum _{\ell \in j_{*}}P(E^{c}_{3,\ell }). \end{aligned}$$

(14)

Applying (i), (ii), and (iv) of Lemma 1 in [9] to (14), the following equation can be derived:

$$\begin{aligned} P\left( \cup _{\ell \in j_{*}}\{\mathcal {D}_{\ell }\le 0\}\right)&\le O \left( k_{j_{*}}\xi ^{2}n^{-2}p^{-2}\right) + O\left( \xi ^{2}\mathrm{{tr}}(\boldsymbol{\varSigma }_{*})^{-2}n^{-1}\right) \nonumber \\&\quad + O\left( k_{j_{*}}E[||\boldsymbol{\varepsilon }||^{4}]n^{-2}p^{-2}\right) + O\left( k_{j_{*}}\lambda _{\mathrm{{max}}}(\boldsymbol{\varSigma }_{*})^{2}n^{-2}p^{-2}\right) . \end{aligned}$$

(15)

Therefore, (12) and (15) complete the proof of Theorem 1.   $\square$

Appendix 2: Proof of Theorem 2

To prove Theorem 2, we need the following lemma concerning the 8-th moment of $\boldsymbol{\varepsilon }$ (the proof is given in Appendix 3):

Lemma 1

Let $\boldsymbol{A}$ be an $n \times n$ symmetric idempotent matrix satisfying $\mathrm{{rank}}(\boldsymbol{A})=m<n$. Then, $E[\mathrm{{tr}}(\boldsymbol{\mathcal {E}}'\boldsymbol{A}\boldsymbol{\mathcal {E}})^{4}]\le \phi m^{3}E[||\boldsymbol{\varepsilon }||^{8}]$ holds, where $\phi$ is a constant not depending on n, p, and m.

Using Lemma 1, for $\ell \notin j_{*}$ we have

$$\begin{aligned} P\left( \mathrm{{tr}}(\boldsymbol{V}_{\omega ,\omega _{\ell }}) \ge \lambda ^{-1}\tau p\alpha \mathrm{{tr}}(\boldsymbol{\varSigma }_{*})\right)&\le E[\mathrm{{tr}}(\boldsymbol{V}_{\omega ,\omega _{\ell }})^4]/\{\lambda ^{-1}\tau p\alpha \mathrm{{tr}}(\boldsymbol{\varSigma }_{*})\}^4\nonumber \\&\le \phi E[||\boldsymbol{\varepsilon }||^{8}]/\{\lambda ^{-1}\tau p\alpha \mathrm{{tr}}(\boldsymbol{\varSigma }_{*})\}^4\nonumber \\&=O\left( \lambda ^4 p^{-4}\alpha ^{-4}\right) . \end{aligned}$$

(16)

Notice that $\kappa _{4}\le E[||\boldsymbol{\mathcal {\varepsilon }}||^{4}] \le E[||\boldsymbol{\mathcal {\varepsilon }}||^{8}]^{1/2}$ holds. Therefore, (11), (12), (15), and (16) complete the proof of Theorem 2.   $\square$

Appendix 3: Proof of Lemma 1

Denote the i-th row vector of $\boldsymbol{\mathcal {E}}$ as $\boldsymbol{\varepsilon }_{i}$. The expectation $E[\mathrm{{tr}}(\boldsymbol{\mathcal {E}}'\boldsymbol{A}\boldsymbol{\mathcal {E}})^{4}]$ can be expressed as follows:

$$\begin{aligned}&E[\mathrm{{tr}}(\boldsymbol{\mathcal {E}}'\boldsymbol{A}\boldsymbol{\mathcal {E}})^{4}]\\&=\sum ^{n}_{i=1}\{(\boldsymbol{A})_{ii}\}^{4}E[||\boldsymbol{\varepsilon }_{i}||^{8}]+\phi _{1} \sum ^{n}_{i \ne j}\{(\boldsymbol{A})_{ii}\}^{3}(\boldsymbol{A})_{jj}E[||\boldsymbol{\varepsilon }_{i}||^{6}||\boldsymbol{\varepsilon }_{j}||^{2}]\nonumber \\&\quad + \phi _{2} \sum ^{n}_{i \ne j}\{(\boldsymbol{A})_{ii}\}^{2}\{(\boldsymbol{A})_{ij}\}^{2}E[||\boldsymbol{\varepsilon }_{i}||^{4}(\boldsymbol{\varepsilon }'_{i}\boldsymbol{\varepsilon }_{j})^{2}]\nonumber \\&\quad +\phi _{3} \sum ^{n}_{i \ne j}\{(\boldsymbol{A})_{ii}\}^{2}\{(\boldsymbol{A})_{jj}\}^{2}E[||\boldsymbol{\varepsilon }_{i}||^{4}||\boldsymbol{\varepsilon }_{j}||^{4}] + \phi _{4} \sum ^{n}_{i \ne j}\{(\boldsymbol{A})_{ij}\}^{4}E[(\boldsymbol{\varepsilon }'_{i}\boldsymbol{\varepsilon }_{j})^{4}]\nonumber \\&\quad +\phi _{5} \sum ^{n}_{i \ne j}(\boldsymbol{A})_{ii}(\boldsymbol{A})_{jj}\{(\boldsymbol{A})_{ij}\}^{2}E[||\boldsymbol{\varepsilon }_{i}||^{2}||\boldsymbol{\varepsilon }_{j}||^{2}(\boldsymbol{\varepsilon }'_{i}\boldsymbol{\varepsilon }_{j})^{2}]\nonumber \\&\quad +\phi _{6} \sum ^{n}_{i \ne j \ne k}(\boldsymbol{A})_{ii}(\boldsymbol{A})_{jj}(\boldsymbol{A})_{kk}(\boldsymbol{A})_{ij}E[||\boldsymbol{\varepsilon }_{i}||^{2}||\boldsymbol{\varepsilon }_{j}||^{2}||\boldsymbol{\varepsilon }_{k}||^{2}(\boldsymbol{\varepsilon }'_{i}\boldsymbol{\varepsilon }_{j})]\nonumber \\&\quad +\phi _{7} \sum ^{n}_{i \ne j \ne k}(\boldsymbol{A})_{ii}\{(\boldsymbol{A})_{jk}\}^{3}E[||\boldsymbol{\varepsilon }_{i}||^{2}(\boldsymbol{\varepsilon }'_{j}\boldsymbol{\varepsilon }_{k})^{3}]\nonumber \\&\quad +\phi _{8} \sum ^{n}_{i \ne j \ne k}(\boldsymbol{A})_{ii}(\boldsymbol{A})_{jj}(\boldsymbol{A})_{ik}(\boldsymbol{A})_{jk}E[||\boldsymbol{\varepsilon }_{i}||^{2}||\boldsymbol{\varepsilon }_{j}||^{2}(\boldsymbol{\varepsilon }'_{i}\boldsymbol{\varepsilon }_{k})(\boldsymbol{\varepsilon }'_{j}\boldsymbol{\varepsilon }_{k})]\nonumber \\&\quad +\phi _{9} \sum ^{n}_{i \ne j \ne k}\{(\boldsymbol{A})_{ij}\}^{2}(\boldsymbol{A})_{jk}(\boldsymbol{A})_{ki}E[(\boldsymbol{\varepsilon }'_{i}\boldsymbol{\varepsilon }_{j})^{2}(\boldsymbol{\varepsilon }'_{j}\boldsymbol{\varepsilon }_{k})(\boldsymbol{\varepsilon }'_{k}\boldsymbol{\varepsilon }_{i})]\nonumber \\&\quad +\phi _{10} \sum ^{n}_{i \ne j \ne k}(\boldsymbol{A})_{ii}(\boldsymbol{A})_{ij}\{(\boldsymbol{A})_{jk}\}^{2}E[||\boldsymbol{\varepsilon }_{i}||^{2}(\boldsymbol{\varepsilon }'_{i}\boldsymbol{\varepsilon }_{j})(\boldsymbol{\varepsilon }'_{j}\boldsymbol{\varepsilon }_{k})^{2}], \end{aligned}$$

where the summation $\sum ^{n}_{i \ne j \ne k}$ is defined by $\sum ^{n}_{i \ne j}\sum ^{n}_{k:k\ne i,j}$ and $\phi _{1},\ldots ,\phi _{10}$ are natural numbers not depending on n, p, and m. Since $\boldsymbol{A}$ is positive semi-definite and is also symmetric idempotent, $0 \le (\boldsymbol{A})_{ii}\le 1$, $\{(\boldsymbol{A})_{ij}\}^{2} \le (\boldsymbol{A})_{ii}(\boldsymbol{A})_{jj}$ (e.g., [2], Fact 8.9.9), $\boldsymbol{1}'_{n}\boldsymbol{D}_{\boldsymbol{A}}\boldsymbol{A}\boldsymbol{D}_{\boldsymbol{A}}\boldsymbol{1}_{n}\le \mathrm{{tr}}(\boldsymbol{A})$ and $\boldsymbol{1}'_{n}\boldsymbol{D}^{2}_{\boldsymbol{A}}\boldsymbol{A}\boldsymbol{D}_{\boldsymbol{A}}\boldsymbol{1}_{n}\le \mathrm{{tr}}(\boldsymbol{A})$ hold. Recalling $\boldsymbol{D}_{\boldsymbol{A}}=\mathrm{{diag}}\{(\boldsymbol{A})_{11},\ldots ,(\boldsymbol{A})_{nn}\}$ and using these facts, we have

$$\begin{aligned}&\sum ^{n}_{i=1}\{(\boldsymbol{A})_{ii}\}^{4} \le m, \ \sum ^{n}_{i \ne j}\{(\boldsymbol{A})_{ii}\}^{3}(\boldsymbol{A})_{jj}\le m^{2}, \ \sum ^{n}_{i \ne j}\{(\boldsymbol{A})_{ii}\}^{2}\{(\boldsymbol{A})_{ij}\}^{2}\le m^{2},\\&\sum ^{n}_{i \ne j}\{(\boldsymbol{A})_{ii}\}^{2}\{(\boldsymbol{A})_{jj}\}^{2}\le m^{2}, \ \sum ^{n}_{i \ne j}\{(\boldsymbol{A})_{ij}\}^{4}\le m^{2}, \ \sum ^{n}_{i \ne j}(\boldsymbol{A})_{ii}(\boldsymbol{A})_{jj}\{(\boldsymbol{A})_{ij}\}^{2}\le m^{2},\\&\sum ^{n}_{i \ne j \ne k}(\boldsymbol{A})_{ii}(\boldsymbol{A})_{jj}(\boldsymbol{A})_{kk}(\boldsymbol{A})_{ij}\le m(m+4), \ \sum ^{n}_{i \ne j \ne k}(\boldsymbol{A})_{ii}\{(\boldsymbol{A})_{jk}\}^{3}\le m(m^{2}+m+1),\\&\sum ^{n}_{i \ne j \ne k}(\boldsymbol{A})_{ii}(\boldsymbol{A})_{jj}(\boldsymbol{A})_{ik}(\boldsymbol{A})_{jk}\le 5m,\ \sum ^{n}_{i \ne j \ne k}\{(\boldsymbol{A})_{ij}\}^{2}(\boldsymbol{A})_{jk}(\boldsymbol{A})_{ki}\le m(3m+2),\\&\sum ^{n}_{i \ne j \ne k}(\boldsymbol{A})_{ii}(\boldsymbol{A})_{ij}\{(\boldsymbol{A})_{jk}\}^{2}\le m(m+6). \end{aligned}$$

Moreover, for any $a,b,c,d,e,f,g,h \in \mathbb {N}$, $E[(\boldsymbol{\varepsilon }'_{a}\boldsymbol{\varepsilon }_{b})(\boldsymbol{\varepsilon }'_{c}\boldsymbol{\varepsilon }_{d})(\boldsymbol{\varepsilon }'_{e}\boldsymbol{\varepsilon }_{f})(\boldsymbol{\varepsilon }'_{g}\boldsymbol{\varepsilon }_{h})] \le E[||\boldsymbol{\varepsilon }_{a}||^{8}]$ holds. Therefore, the proof of Lemma 1 is completed.   $\square$