On Asymptotic Power of the New Test for Equality of Two Distributions

Abstract

The paper introduces a new test for equality of two distributions in a class of models. We proved analytically and by stochastic simulation that the test possesses high efficiency. For the case of normal and Cauchy distributions that differ only by shift the asymptotic power of the test appears to be approximately the same as for the Wilcoxon-Mann-Whitney, the Kolmogorov-Smirnov and the Anderson-Darling tests. But if the distributions differ by scale parameters the power of the new test is considerably better.

1 Formulation of the Problem

Let us consider the classical problem of testing hypothesis on the equality of two distributions

$$\begin{aligned} H_0\,:\,F_1 = F_2 \end{aligned}$$

(1)

against the alternative

$$\begin{aligned} H_1\,:\,F_1 \not = F_2 \end{aligned}$$

(2)

in the case of two independent samples $X=(X_{1},\ldots , X_{n})$ and $Y=(Y_{1},\ldots , Y_{m})$ with the distributions functions $F_1$ and $F_2$ respectively.

It is well known (see e.g. [1]) that in the case when both distributions differ only by the means and are normal the classical Student test has a few optimal properties. If the distributions are not normal but still differs only by means a widely popular Wilcoxon-Mann-Whitney (WMW) U-statistic is often used instead. However, it can be shown that if two normal populations differ only in variances, the power of WMW test is very low. If distributions are arbitrary there are some universal techniques such as tests by Kolmogorov-Smirnov and Cramer-von Mises (see [2]) and the Anderson-Darling test (see [3]) that can be applied but in many cases these tests can be not powerful.

Recently, Zech and Aslan [4] suggested the test based on U-statistics with the logarithmic kernel and provided its numerical justification for one and many dimensional cases in comparison with a few alternative techniques. However, to the best authors knowledge there are no analytical results about its asymptotic power. Here we introduce a similar but different test and provide a few analytical results on its power.

2 The New Test and Its Statistical Motivation

Assume that the distribution functions $F_1$ and $F_2$ belongs to the class of distribution functions of random variables $\xi$, such that

$$\begin{aligned} E [\ln ^2 (1+ \xi ^2) ] < \infty . \end{aligned}$$

(3)

Many distributions and, in particular, the Cauchy distribution have this property.

Among all distributions with given left hand side of (3) the Cauchy’s one has the maximum entropy.

Consider the following test

$$\begin{aligned} \varPhi _{A}=-\frac{1}{n^2}\sum _{1\le i<j\le n} g(X_i-X_j), \varPhi _{B}=-\frac{1}{m^2}\sum _{1\le i<j\le m} g(Y_i-Y_j), \end{aligned}$$

(4)

$$\begin{aligned} \varPhi _{AB}=-\frac{1}{nm}\sum _{i=1}^n\sum _{j=1}^m g(X_i-Y_j), \varPhi _{nm}=\varPhi _{AB}-\varPhi _{A}- \varPhi _{B}, \end{aligned}$$

(5)

where

$$ g(u)= -\ln (1+|u|^2), $$

g(x) is under a constant term precision the logarithm of the density of the standard Cauchy distribution. (Note that Zech and Aslan (2005) took $g(u)= -\ln (|u|)$).

We would like to have a test that is appropriate for the case where the basic distribution belongs to a rather general class of distributions and the alternative distribution differs only by shift and scale transformations.

In particular, we consider the class of distributions satisfying (3), but the approach can be generalized for other classes of distributions.

Consider the class of distributions given by the property (3). Note that if the parameters are known the test based on likelihood ratio is the most powerful among tests with given parameters.

The test suggested above can be considered as an approximation of logarithm of this ratio for the Cauchy distribution. We suppose that it will be very efficient for all distributions with property (3).

3 The Analytical Study of Asymptotic Power

Let us consider the case of two distributions having the property (3) and, in particular, the two that differ only by a shift. To simplify notations assume that $m=n$. The case $m\ne n$ is similar. Now the criterion (4)–(5) assumes the form

$$\begin{aligned} T_n=\varPhi _{nn}= \frac{1}{n^2}\sum _{i,j=1}^n \ln (1 + (X_i - Y_j)^2)-\frac{1}{n^2}\sum _{1\le i<j\le n} \ln (1 + (X_i - X_j)^2) \end{aligned}$$

(6)

$$\begin{aligned} - \frac{1}{n^2}\sum _{1\le i<j\le n} \ln (1 + (Y_i - Y_j)^2). \end{aligned}$$

(7)

Denote by C(u, v) the Cauchy distribution with the density function

$$ v/(\pi (v^2 + (x-u)^2)). $$

Let f(x) denotes the density of $F_1$. Denote

$$\begin{aligned} J_h =\int _R -g(x-y-|h|/\sqrt{n})f(x)f(y)dxdy, \end{aligned}$$

where $g(u)= - \ln (1+|u|^2)$.

By expending the function $g(u)= \ln (1+|u|^2)$ into the Taylor series we obtain that for arbitrary density function f(x) there exists the finite limit

$$\begin{aligned} J^*(h)= lim_{n\rightarrow \infty } n(J_h - J_0) \end{aligned}$$

(8)

and it is equal to

$$ (1/2)h^2 \int _R g''_\theta (x-y-\theta )f(x)f(y)dxdy|_{\theta =0}. $$

(Note that the differentiation under integral is justified since the derivative $g ''_\theta (x-y-\theta )|_{\theta =0}$ is less than 2.) That is

$$ J^*(h)=h^2 \int _R \frac{1-(x-y)^2}{(1+(x-y)^2)^2} f(x)f(y)dxdy. $$

Denote

$$ \bar{b} =\sqrt{J^*(h)/h^2}. $$

The basic analytical result of the present paper is the following

Theorem 1

Consider the problem of testing hypothesis on the equality of two distributions (1)–(2) where both functions have the property (3). Then

• (i) under the condition $n \rightarrow \infty$ the distribution function of $nT_n$ converges under $H_0$ to that of the random variable

  $$\begin{aligned} (aL)^2, \end{aligned}$$

  (9)

  where L has the normal distribution with zero expectation and variance equal to 1, $a>0$ is some number.

• (ii) Let $F_1(x)= F(x),F_2=F(x+\theta ),$ where F is an arbitrary distribution function that is symmetric around a point and possess property (3), $\theta =h/\sqrt{n},h$ is an arbitrary given number. Then the distribution function of $nT_n$ converges under $H_1$ to that of the random variable

  $$ (aL + b)^2, $$

  where $b=0$ for the case of $H_0$ and $b=\bar{b} h$ for $H_1$. In this case the power of the criterion $T_n$ with significance $\alpha$ is asymptotically equal to that is given by the formula

  $$ Pr\{L\ge z_{1-\alpha /2}-\bar{b}h/a\} + Pr\{L\le - z_{1-\alpha /2}-\bar{b}h/a\}, $$

  where $z_{1-\alpha /2}$ is such that

  $$ Pr \{L\ge z_{1-\alpha /2}\}= \alpha /2. $$

If $F_1=C(\nu ,1), F_2=C(\nu + \theta ,1)$ then $b= h/3$.

Note that the analytical presentation for the coefficient a is a difficult problem that is not solved up to now. However this coefficient can be easily found by stochastic simulation. In the case of Cauchy distribution we found a heuristic formula $3a^2= J_0$, that means $a =\sqrt{(2/3)\ln 3}$. This formula provide a very exact approximation for empirical power (see Tables 1, 2 and 3 in the next section).

Thus in the case of Cauchy distributions with scale parameter equal to 1 the power of the criterion $T_n$ with significance $\alpha$ is approximately equal to

$$ Pr\{L\ge z_{1-\alpha /2}-(1/\sqrt{6\ln 3})h\} + Pr\{L\le - z_{1-\alpha /2}-(1/\sqrt{6\ ln 3})h\}. $$

The proof of the theorem is given in the Appendix.

4 Simulation Results

In this section we present numerical results of the efficiency of new criterion in comparison with a few alternative criteria.

At the Tables 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 and 12 results for cases n = 100, 500, 1000 and different values of h with $\alpha =0.05$ are given for normal and Cauchy distributions that differ either by shift or by scale parameters. The critical values were calculated in two ways: by simulation of the initial distribution and by random permutations (we used 800 random permutation in all cases). It worth to be noted that the results are very similar. Since the permutation technique is more universal, it can be recommended for practical applications.

Note that in all these cases when the distributions differ only in the shift parameters the power of $T_n$ and that of the Wilcoxon-Mann-Whitney, the Kolmogorov-Smirnov and the Anderson-Darling tests were approximately equal to each other. It can be pointed out also that if the variances are not standard but are known we should simply make the corresponding normalisation. But for the cases where the distributions differ in scale parameters the Wilcoxon-Mann-Whitney is not appropriate at all and the power of the Kolmogorov-Smirnov and the Anderson-Darling tests is considerably lower.

Table 1. Cauchy distribution, $X\sim C(0,1)$, $Y\sim C(h/\sqrt{n},1)$, $n=100$

Table 2. Cauchy distribution, $X\sim C(0,1)$, $Y\sim C(h/\sqrt{n},1)$, $n=500$

Table 3. Cauchy distribution, $X\sim C(0,1)$, $Y\sim C(h/\sqrt{n},1)$, $n=1000$

Table 4. Cauchy distribution, $X\sim C(0,1)$, $Y\sim C(0, 1 + h/\sqrt{n})$, $n=100$

Table 5. Cauchy distribution, $X\sim C(0,1)$, $Y\sim C(0, 1 + h/\sqrt{n})$, $n=500$

Table 6. Cauchy distribution, $X\sim C(0,1)$, $Y\sim C(0, 1 + h/\sqrt{n})$, $n=1000$

Table 7. Normal distribution, $X\sim N(0,1)$, $Y\sim N(h/\sqrt{n},1)$, $n=100$

Table 8. Normal distribution, $X\sim N(0,1)$, $Y\sim N(h/\sqrt{n},1)$, $n=500$

Table 9. Normal distribution, $X\sim N(0,1)$, $Y\sim N(h/\sqrt{n},1)$, $n=1000$

Table 10. Normal distribution, $X\sim N(0,1)$, $Y\sim N(0, 1 + h/\sqrt{n})$, $n=100$

Table 11. Normal distribution, $X\sim N(0,1)$, $Y\sim N(0, 1 + h/\sqrt{n})$, $n=500$

Table 12. Normal distribution, $X\sim N(0,1)$, $Y\sim N(0, 1 + h/\sqrt{n})$, $n=1000$

5 Conclusion

In this paper we suggested a new test for equality of two distributions. In a wide class of distributions it was proved that the limiting distribution is the square of a Normal distribution. It allows to find asymptotic power analytically for the case of distributions that differ only by shift up to unknown parameter that can be found by stochastic simulation. The high efficiency of the test was confirmed by stochastic simulations.

6 Appendix

Proof of Theorem 1. Let us consider the test (4)–(5) with the function $g(u)=-u^2$ that is the logarithm of the density of the standard Normal distribution.

Lemma 1

For $g(x)= x^2$ the following identity holds

$$ \varPhi _{nn}= (\bar{x} - \bar{y})^2 $$

where

$$ \bar{x} = (\sum _{i=1}^n X_i)/n, \bar{y} = (\sum _{i=1}^n Y_i)/n. $$

Denote

$$ Z=(X,Y)= (X_1,\dots , X_n,Y_1,\dots , Y_n), V(Z)=\frac{1}{2}\sum _{i=1}^{2n}\sum _{j=1}^{2n} (Z_i-Z_j)^2. $$

The proof follows from the known formula [see e.g. [5], p. 296]

$$\begin{aligned} \frac{1}{n(n-1)}\sum _{1\le i<j\le n} (X_i-X_j)^2=\frac{1}{(n-1)} \sum _{i=1}^n (X_i - \bar{x})^2 \end{aligned}$$

(10)

and the obvious identity

$$\begin{aligned} \sum _{i=1}^{2n}\sum _{j=1}^{2n} (Z_i-Z_j)^2=\sum _{i,j=1}^n(X_i-X_j)^2+ \sum _{i,j=1}^n (Y_i-Y_j)^2+2\sum _{i=1}^n\sum _{j=1}^n (X_i-Y_j)^2, \end{aligned}$$

(11)

by direct but non trivial calculations.

Really, let us use the standard notation

$$ S_x^2=\frac{1}{(n-1)} \sum _{i=1}^n (X_i - \bar{x})^2 $$

And $S_y^2$ and $S_z^2$ will be understood in the similar way. Denote

$$ S_{xy}=\frac{1}{n^2}\sum _{i=1}^n\sum _{j=1}^n (X_i-Y_j)^2. $$

Note that due to formula (10) for X replaced by Z

$$\begin{aligned} V(Z)=2n[\sum _{i=1}^n(X_i- (\bar{x} + \bar{y})/2)^2 +\sum _{j=1}^n(Y_i- (\bar{x} + \bar{y})/2)^2] = 2n(n-1) (S_x^2 + S_y^2) +n^2(\bar{x}- \bar{y})^2. \end{aligned}$$

(12)

From (10) and (11) we obtain

$$\begin{aligned} n^2S_{xy}=V(Z)-n(n-1)(S_x^2+S_y^2). \end{aligned}$$

(13)

Therefore

$$ S_{xy} =\frac{1}{n}(n-1)(S_x^2+S_y^2)+ (\bar{x}-\bar{y})^2, $$

and we obtain

$$ \varPhi _{nn}= S_{xy} - \frac{1}{n}(n-1)(S_x^2+S_y^2)=(\bar{x} - \bar{y})^2. $$

Thus Lemma 1 is proved. It follows from this lemma, that the criterion $\varPhi _{nn}$ in this case is equivalent to the criterion $(\bar{x} - \bar{y})^2.$

Let us turn to the proof of the theorem.

Assume that either $H_0$ or $H_1$ holds. Then due to the law of large numbers for $U-$statistics [5] each of the sums

$$ \varPhi _{AB}=\frac{1}{n^2}\sum _{i,j=1}^n \ln (1 + (X_i - Y_j)^2), $$

$$ \varPhi _{A}+ \varPhi _{B}=\frac{1}{n^2}\sum _{1\le i<j\le n} \ln (1 + (X_i - X_j)^2)+ \frac{1}{n^2}\sum _{1\le i<j\le n} \ln (1 + (Y_i - Y_j)^2) $$

tends to $J_0$.

Moreover,

$$ \varPhi _{AB}= J_0 + o(n^2), $$

$$ \varPhi _{A}+ \varPhi _{B}=J_0(1-\frac{1}{n}) + o(n^2). $$

Note that

$$ nT_n=n[\varPhi _{AB}-J_0] -n[\varPhi _{A}-\frac{1}{2}J_0]-n[\varPhi _{B}- \frac{1}{2}J_0]. $$

Let us apply the limit theorem for U-statistics (see Theorem 7.1 [5]) to each of the three terms in brackets. We obtain that $nT_n$ tends to a random variable with a finite variance. Note that the conditions of the limit theorem are fulfilled for distributions $F_1$ with the property (3).

Note that $0 \le \ln (1+x^2) \le x^2$. By this reason $\varPhi _{AB}$ is between 0 and $S_{xy}$. Due to theorem about the mean it is equal to $c_nS_{xy}$, $0<c_n<1$ and $c_n$ tends to a constant c with $n \rightarrow \infty$. In a similar way, $\varPhi _{A}+ \varPhi _{B}= c_{1n}(\frac{n}{n-1}(S_x^2+S_y)^2)$ and $c_{1n}$ tends to $c_1$ while $c_1=c$.

Let C be an arbitrary positive number,

$$ \tilde{X}=(\tilde{X_{1}},\ldots ,\tilde{X_{n}}),\,\,\, \tilde{Y}=(\tilde{Y_{1}},\ldots , \tilde{Y_{n}}), $$

where $\tilde{X_{i}}=X_{i}$, if $|X_{i}| \le C$ and $\tilde{X_i}=C$ if $X_{i}>0$, $\tilde{X_i}=-C$ if $X_{i}<0$ otherwise. And $\tilde{Y_{i}}$ are determined similarly.

Consider the function

$$\begin{aligned} n\{\frac{1}{n^2}\sum _{i,j=1}^n \ln (1 + (\tilde{X_{i}} - \tilde{Y_j})^2 -\frac{1}{n^2}\sum _{i<j} \ln (1 + (\tilde{X_i} - \tilde{X_j})^2) - \end{aligned}$$

(14)

$$\begin{aligned} \frac{1}{n^2}\sum _{i<j} \ln (1 + (\tilde{Y_i} - \tilde{Y_j})^2)\}. \end{aligned}$$

(15)

Due to the presentations for $\varPhi _{AB}$, $\varPhi _{A}$ and $\varPhi _{B}$ derived above it can be checked that there exists a value $t_n$ that depends on $\tilde{X}$ and $\tilde{Y}$ and numbers $B_n$ such that it is equal to

$$\begin{aligned} t(\sum _{i=1}^n \tilde{X_{i}}/\sqrt{n}-\sum _{i=1}^n \tilde{Y_{i}}/\sqrt{n})^2 + B_n, \end{aligned}$$

(16)

and $B_n$ is o(1).

Consider expression (14)–(15). Note that for distributions $F_1$ and $F_2$ satisfying (3) with $\tilde{X}_i$ and $\tilde{Y}_i$ replaced by $X_i$ and $Y_i$, respectively, its variance is bounded from above due to that $nT_n$ tends to a random variable with a finite variance. Therefore the expression (14)–(15) tends with $n\rightarrow \infty$ to a random variable with a finite variance for arbitrary C. Passing to the limit with $n\rightarrow \infty$ we obtain due to the central limit theorem that (16) has the limit distribution of the form (9), where L has the standard normal distribution. Since C is arbitrary we obtain that the limiting distribution has the required form.

For determining b in the part (ii) of the theorem we now can use the equality

$$\begin{aligned} (aL+b)^2=\lim _{n \rightarrow \infty } nT_n, \end{aligned}$$

(17)

that follows from the equality between (14)-(15) and (16). If $H_0$ take place we obviously have $b=0$. In the case when $H_1$ take place $EnT_n$ is asymptotically equivalent to

$$ (n(J_h - J_0))^2+En\hat{T_n} $$

where $\hat{T_n}$ received from $T_n$ by replacing $Y_i$ by $Y_i - b/\sqrt{n},$ $i=1,\dots ,n$ and we obtain by passing to the limit with $n\rightarrow \infty$ that

$$ b= \bar{b}h, \bar{b} =\sqrt{J^*(h)/h^2}. $$

And the asymptotic behaviour of the power announced in (ii) follows from the asymptotic normality of $\sqrt{nT_n}$. In order to calculate $\bar{b}$ in the case when $F_1$ is the standard Cachy distribution the following result is crucial.

Lemma 2

If X and Y are independent random variables with the distribution C(0, 1), then

$$ E \ln (1+ (X- Y)^2)= \ln 9,\,\,\,\, E \ln (1+ (X- Y - \theta )^2)-\ln 9 = ln(1+ \theta ^2/9). $$

In order to prove this Lemma we need the following integrals

$$ \int _{R} \frac{\ln (1+(x-y)^2)}{\pi (1+y^2)} dy = \ln (4+x^2), $$

$$ \quad \int _{R} \frac{\ln (4+x^2)}{\pi (1+x^2)} dx = \ln 9, $$

([6] 4.296.2 and 4.295.7.)

$$ \int _{R} \frac{\ln (4 +(x +\theta )^2 )}{\pi (x^2 +1)} dx = \ln (9+\theta ^2), $$

[see [7], formula (2.6.14.19)]. Using these integrals we obtain

$$ E \ln (1+ (X- Y - \theta )^2)-\ln 9 = 2\int _{R} \int _{R} \frac{\ln (1+(x-y-\theta )^2)}{\pi ^2(1+x^2)(1+y^2)} dx dy -\ln 9 $$

$$ = \int _{R}\frac{\ln (4 +(y+\theta )^2)}{\pi (1+y^2)} dy- \ln 9= \ln (9+\theta ^2) -\ln 9 = \ln (1+ \theta ^2/9). $$

Submitting here $\theta =0$ we obtain both formulas of the Lemma. Note that $\theta ^2=nh^2$ and

$$ \lim _{n\rightarrow \infty } n \ln (1+ \theta ^2/9)= (1/9)h^2. $$

Therefore we obtain $\bar{b} = 1/3$ that completes the proof of the theorem.