Generalizations of Entropy and Information Measures

Abstract

This paper presents and discusses two generalized forms of the Shannon entropy, as well as a generalized information measure. These measures are applied on a exponential-power generalization of the usual Normal distribution, emerged from a generalized form of the Fisher’s entropy type information measure, essential to Cryptology. Information divergences between these random variables are also discussed. Moreover, a complexity measure, related to the generalized Shannon entropy, is also presented, extending the known SDL complexity measure.

1 Introduction

Since the time of Clausius, 1865, Entropy plays an important role in linking physical experimentation and statistical analysis. It was later in 1922, when Fisher developed in [9] the Experiment Design Theory, another link between Statistics with Chemistry, as well as in other fields. For the principle of maximum entropy, the normal distribution is essential and eventually it is related with the energy and the variance involved.

The pioneering work by Shannon [28] related Entropy with Information Theory and gave a new perspective to the study of information systems and of Cryptography, see [1, 14] among others. Shannon entropy (or entropy) measures the average uncertainty of a random variable (r.v.). In Information Theory, it is the minimum number of bits required, on the average, to describe the value x of the r.v. X. In Cryptography, entropy gives the ultimately achievable error-free compression in terms of the average codeword length symbol per source. There are two different roles of entropy measures: (a) positive results can be obtained in the form of security proofs for (unconditionally secure) cryptographic systems, and (b) lower bounds on the required key sizes are negative results, in some scenarios, and follow from entropy-based arguments. See also [14].

Recall that the relative entropy or discrimination or information divergence between two r.v., say X and Y, measures the increase, or decrease, of information, about an experiment, when the probability Pr(X) (associated with the knowledge of the experiment) is changed to Pr(Y ). Relative entropy is the underlying idea of the Authentication Theory which provides a level of assurance to the receiver of a message originating from a legitimate sender.

A central concept of Cryptography is that of information measure or information, as cryptographic scenarios can be modelled with information-theoretic methods. There are several kinds of information measures which all quantify the uncertainty of an outcome of a random experiment, and, in principle, information is a measure of the reduction of uncertainty.

Fisher’s entropy type information measure is a fundamental one, see [5]. Poincaré and Sobolev Inequalities play an important role in the foundation of the generalized Fisher’s entropy type information measure. Both classes of inequalities offer a number of bounds for a number of physical applications. The Gaussian kernel or the error function (which produce the normal distribution) usually has two parameters, the mean and the variance. For the Gaussian kernel an extra parameter was then introduced in [15], and therefore a generalized form of the Normal distribution was obtained. Specifically, the generalized Gaussian is obtained as an extremal for the Logarithm Sobolev Inequality (LSI), see [4, 30], and is referred here as the γ-order Normal Distribution, or $\mathcal{N}_{\gamma }$. In addition, the Poincaré Inequality (PI), offers also the “best” constant for the Gaussian measure, and therefore is of interest to see how Poincaré and Sobolev inequalities are acting on the Normal distribution.

In this paper we introduce and discuss two generalized forms of entropy and their behavior over the generalized Normal distribution. Moreover, the specific entropy measures as collision and the mean-entropy are discussed. A complexity measure for an r.v. is also evaluated and studied.

2 Information Measures and Generalizations

Let X be a multivariate r.v. with parameter vector $\theta = (\theta _{1},\theta _{2},\ldots,\theta _{p}) \in \mathbb{R}^{p}$ and p.d.f. $f_{X} = f_{X}(x;\;\theta )$, $x \in \mathbb{R}^{p}$. The parametric type Fisher’s Information Matrix $\mathrm{I}_{\mathrm{F}}(X;\;\theta )$ (also denoted as $\mathrm{I}_{\theta }(X)$) defined as the covariance of $\nabla _{\theta }\log f_{X}(X;\;\theta )$ (where $\nabla _{\theta }$ is the gradient with respect to the parameters $\theta _{i}$, $i = 1,2,\ldots,p$) is a parametric type information measure, expressed also as

$$\displaystyle{\begin{array}{llll} &\mathrm{I}_{\theta }(X)&\!\!\! =\ \ &\mathop{\mathrm{Cov}}\nolimits \left (\nabla _{\theta }\log f_{X}(X;\;\theta )\right ) =\mathrm{ E}_{\theta }\left [\nabla _{\theta }\log f_{X} \cdot (\nabla _{\theta }\log f_{X})^{\mathrm{T}}\right ] \\ & &\!\!\! =\ \ &\mathrm{E}_{\theta }\left [\|\nabla _{\theta }\log f_{X}\|^{2}\right ],\end{array} }$$

where $\|\cdot \|$ is the usual $\mathcal{L}^{2}(\mathbb{R}^{p})$ norm, while $\mathrm{E}_{\theta }[\cdot ]$ denotes the expected value operator applied to random variables, with respect to parameter $\theta$.

Recall that the Fisher’s entropy type information measure I_F(X), or J(X), of an r.v. X with p.d.f. f on $\mathbb{R}^{p}$, is defined as the covariance of r.v. $\nabla \log f(X)$, i.e. $\mathrm{J}(X):=\mathrm{ E}[\|\nabla \log f(X)\|^{2}]$, with E[⋅ ] now denotes the usual expected value operator of a random variable with respect to the its p.d.f. Hence, J(X) can be written as

$$\displaystyle\begin{array}{rcl} \mathrm{J}(X)& \ =\ \ & \int \limits _{\mathbb{R}^{p}}f(x)\|\nabla \log f(x)\|^{2}dx =\int \limits _{ \mathbb{R}^{p}}f(x)^{-1}\|\nabla f(x)\|^{2}dx \\ & \ =\ & \int \limits _{\mathbb{R}^{p}}\nabla f(x) \cdot \nabla \log f(x)dx = 4\int \limits _{\mathbb{R}^{p}}\left \|\nabla \sqrt{f(x)}\right \|^{2}dx.{}\end{array}$$

(1)

Generally, the family of the entropy type information measures I(X), of a p-variate r.v. X with p.d.f. f, are defined through the score function of X, i.e.

$$\displaystyle{U(X):=\| \nabla \log f(X)\|,}$$

as

$$\displaystyle{\mathrm{I}(X):=\mathrm{ I}(X;\;g,h):= g\left (\mathrm{E}[h(U(X))]\right ),}$$

where g and h being real-valued functions. For example, when g = i.d. and h(X) = X ² we obtain the entropy type Fisher’s information measure of X as in (1), i.e.

$$\displaystyle{ \mathrm{I_{F}}(X) =\mathrm{ E}[\|\nabla \log f(X)\|^{2}]. }$$

(2)

Besides I_F, other entropy type information measures as the Vajda’s, Mathai’s, and Boeke’s information measures, denoted with I_V, I_M, and I_B, respectively, are defined as:

$$\displaystyle{ \begin{array}{lll} \mathrm{I_{F}}(X):=\mathrm{ I}(X), &\text{with}\;\;g:= \text{id.} &\text{and}\;\;h(U):= U^{2}, \\ \mathrm{I_{V}}(X):=\mathrm{ I}(X), &\text{with}\;\;g:= \text{id.} &\text{and}\;\;h(U):= U^{\lambda },\;\lambda \geq 1, \\ \mathrm{I_{M}}(X):=\mathrm{ I}(X),&\text{with}\;\;g(X):= X^{1/\lambda } &\text{and}\;\;h(U):= U^{\lambda },\;\lambda \geq 1, \\ \mathrm{I_{B}}(X):=\mathrm{ I}(X), &\text{with}\;\;g(X):= X^{\lambda -1} & \text{and}\;\;h(U):= U^{ \frac{\lambda }{\lambda -1} },\;\lambda \in \mathbb{R}_{+}\setminus 1. \end{array} }$$

The notion of information “distance” or divergence of a p-variate r.v. X over a p-variate r.v. Y is given by

$$\displaystyle{\mathrm{D}(X,Y ) =\mathrm{ D}(X,Y;\;g,h):= g\left (\;\int \limits _{\mathbb{R}^{p}}h(f_{X},f_{Y })\right ),}$$

where f _X and f _Y are the probability density functions (p.d.f) of X and Y, respectively. Some known divergences, such as the Kullback–Leibler D_KL, the Vajda’s D_V, the Kagan D_K, the Csiszar D_C, the Matusita D_M, as well as the Rényi’s D_R divergence, see also [8], are defined as follows:

$$\displaystyle{ \begin{array}{lll} \mathrm{D_{KL}}(X,Y ):=\mathrm{ D}(X,Y ),&\text{with}\;\;g:= \text{id.} &\text{and}\;\;h(f_{X},f_{Y }):= f_{X}\log (f_{X}/f_{Y }), \\ \mathrm{D_{V}}(X,Y ):=\mathrm{ D}(X,Y ), &\text{with}\;\;g:= \text{id.} &\text{and}\;\;h(f_{X},f_{Y }):= f_{X}\left \vert 1 - (f_{Y }/f_{X})\right \vert ^{\lambda },\;\lambda \geq 1, \\ \mathrm{D_{K}}(X,Y ):=\mathrm{ D}(X,Y ), &\text{with}\;\;g:= \text{id.} &\text{and}\;\;h(f_{X},f_{Y }):= f_{X}\left \vert 1 - (f_{Y }/f_{X})\right \vert ^{2}, \\ \mathrm{D_{C}}(X,Y ):=\mathrm{ D}(X,Y ), &\text{with}\;\;g:= \text{id.} &\text{and}\;\;h(f_{X},f_{Y }):= f_{Y }\phi (f_{X}/f_{Y }),\;\phi \;\text{convex}, \\ \mathrm{D_{M}}(X,Y ):=\mathrm{ D}(X,Y ), &\text{with}\;\;g(A):= \sqrt{A}&\text{and}\;\;h(f_{X},f_{Y }):= \left (\sqrt{f_{X}} -\sqrt{f_{Y}}\right )^{2}, \\ \mathrm{D_{R}}(X,Y ):=\mathrm{ D}(X,Y ), &\text{with}\;\;g(A):= \tfrac{\log A} {1-\lambda } &\text{and}\;\;h(f_{X},f_{Y }):= f_{X}^{\lambda }f_{ Y }^{1-\lambda },\;\lambda \in \mathbb{R}_{ +}\setminus 1. \end{array} }$$

Consider now the Vajda’s parametric type measure of information $\mathrm{I}_{\mathrm{V}}(X;\;\theta,\alpha )$, which is in fact a generalization of $\mathrm{I}_{\mathrm{F}}(X;\;\theta )$, defined as, [8, 33],

$$\displaystyle{ \mathrm{I}_{V }(X;\;\theta,\alpha ):=\mathrm{ E}_{\theta }[\|\nabla _{\theta }\log f(X)\|^{\alpha }],\quad \alpha \geq 1. }$$

(3)

Similarly, the Vajda’s entropy type information measure J _α (X) generalizes Fisher’s entropy type information J(X), defined as

$$\displaystyle{ \mathrm{J}_{\alpha }(X):=\mathrm{ E}[\|\nabla \log f(X)\|^{\alpha }],\quad \alpha \geq 1, }$$

(4)

see [15]. We shall refer to J _α (X) as the generalized Fisher’s entropy type information measure or α-GFI. The second-GFI is reduced to the usual J, i.e. J₂(X) = J(X). Equivalently, from the definition of the α-GFI above we can obtain

$$\displaystyle\begin{array}{rcl} \mathrm{J}_{\alpha }(X)& \ =& \int \limits _{\mathbb{R}^{p}}\|\nabla \log f(x)\|^{\alpha }f(x)dx =\int \limits _{\mathbb{R}^{p}}\|\nabla f(x)\|^{\alpha }f^{1-\alpha }(x)dx \\ & \ =& \alpha ^{\alpha }\int \limits _{\mathbb{R}^{p}}\|\nabla f^{1/\alpha }(x)\|^{\alpha }dx. {}\end{array}$$

(5)

The Blachman–Stam inequality [2, 3, 31] still holds through the α-GFI measure J _α , see [15] for a complete proof. Indeed:

Theorem 1.

For two given p-variate and independent random variables X and Y, it holds

$$\displaystyle{ \mathrm{J}_{\alpha }\left (\lambda ^{1/\alpha }X + (1-\lambda )^{1/\alpha }Y \right ) \leq \lambda \mathrm{ J}_{\alpha }(X) + (1-\lambda )\mathrm{J}_{\alpha }(Y ),\quad \lambda \in (0,1). }$$

(6)

The equality holds when X and Y are normally distributed with the same covariance matrix.

As far as the superadditivity of J _α is concerned, the following Theorem it can be stated, see [19] for a complete proof.

Theorem 2.

Let an orthogonal decomposition $\mathbb{R}^{p} = \mathbb{R}^{r} \oplus \mathbb{R}^{s}$ , $p = s + t$ , with the corresponding marginal densities of a p.d.f. f on $\mathbb{R}^{p}$ being f ₁ on $\mathbb{R}^{s}$ and f ₂ on $\mathbb{R}^{t}$ , i.e.

$$\displaystyle{ f_{1}(x) =\int \limits _{\mathbb{R}^{s}}f(x,y)d^{s}y,\quad f_{ 2}(y) =\int \limits _{\mathbb{R}^{t}}f(x,y)d^{t}x, }$$

(7)

Then, for r.v. X, X ₁ and X ₂ following f, f ₁ and f ₂ , it holds

$$\displaystyle{ \mathrm{J}_{\alpha }(X) \geq \mathrm{ J}_{\alpha }(X_{1}) +\mathrm{ J}_{\alpha }(X_{2}), }$$

(8)

with the equality holding when f(x,y) = f ₁ (x)f ₂ (y) almost everywhere.

The Shannon entropy H(X) of a continuous r.v. X with p.d.f.f is defined as, [5],

$$\displaystyle{ \mathrm{H}(X):=\mathrm{ E}[\log f(X)] =\int \limits _{\mathbb{R}^{p}}f(x)\log f(x)dx, }$$

(9)

(we drop the usual minus sign) and its corresponding entropy power N(X) is defined as

$$\displaystyle{ \mathrm{N}(X):=\nu e^{\frac{2} {p}\mathrm{H}(X)}, }$$

(10)

with $\nu:= (2\pi e)^{-1}$. The generalized entropy power N _α (X), introduced in [15], is of the form

$$\displaystyle{ \mathrm{N}_{\alpha }(X):=\nu _{\alpha }e^{ \frac{\alpha }{ p}\mathrm{H}(X)}, }$$

(11)

with normalizing factor ν _α given by the appropriate generalization of ν, namely

$$\displaystyle{ \nu _{\alpha }:= \left (\tfrac{\alpha -1} {\alpha e} \right )^{\alpha -1}\pi ^{-\frac{\alpha }{2} }\left [ \frac{\Gamma \left (\tfrac{p} {2} + 1\right )} {\Gamma \left (p\frac{\alpha -1} {\alpha } + 1\right )}\right ]^{ \frac{\alpha }{ p} },\quad \alpha \in \mathbb{R}\setminus [0,1]. }$$

(12)

For the parameter case of α = 2, (11) is reduced to the known entropy power N(X), i.e. N₂(X) = N(X) and ν ₂ = ν.

The known information inequality J(X)N(X) ≥ p still holds under the generalized entropy type Fisher’s information, as J _α (X)N _α (X) ≥ p, α > 1, see [15]. As a result the Cramér–Rao inequality, $\mathrm{J}(X)\mathop{\mathrm{Var}}\nolimits (X) \geq p$, can be extended to

$$\displaystyle{ \left [\tfrac{2\pi e} {p} \mathop{ \mathrm{Var}}\nolimits (X)\right ]^{1/2}\left [ \tfrac{\nu _{\alpha }} {p}\mathrm{J}_{\alpha }(X)\right ]^{1/\alpha } \geq 1,\quad \alpha> 1, }$$

(13)

see [15]. Under the normality parameter α = 2, (13) is reduced to the usual Cramér–Rao inequality.

Furthermore, the classical entropy inequality

$$\displaystyle{ \mathop{\mathrm{Var}}\nolimits (X) \geq p\mathrm{N}(X) = \tfrac{p} {2\pi e}e^{\frac{2} {p}\mathrm{H}(X)}, }$$

(14)

can be extended, adopting our extension above, to the general form

$$\displaystyle{ \mathop{\mathrm{Var}}\nolimits (X) \geq p(2\pi e)^{\frac{\alpha -4} {\alpha } }\nu _{\alpha }^{2/\alpha }\mathrm{N}_{\alpha }^{2/\alpha }(X),\quad \alpha> 1. }$$

(15)

Under the “normal” parameter value α = 2, the inequality (15) is reduced to the usual entropy inequality as in (14).

Through the generalized entropy power N _α a generalized form of the usual Shannon entropy can be produced. Indeed, consider the Shannon entropy of which the corresponding entropy power is N _α (instead of the usual N), i.e.

$$\displaystyle{ \mathrm{N}_{\alpha }(X) =\nu \exp \{ \tfrac{2} {p}\mathrm{H}_{\alpha }(X)\},\quad \alpha \in \mathbb{R}\setminus [0,\,1]. }$$

(16)

We shall refer to the quantity H _α as the generalized Shannon entropy, or α-Shannon entropy, see for details [17]. Therefore, from (11) a linear relation between the generalized Shannon entropy H _α (X) and the usual Shannon entropy H(X) is obtained, i.e.

$$\displaystyle{ \mathrm{H}_{\alpha }(X) = \tfrac{p} {2}\log \tfrac{\nu _{\alpha }} {\nu } + \tfrac{\alpha } {2}\mathrm{H}(X),\quad \alpha \in \mathbb{R}\setminus [0,\,1]. }$$

(17)

Essentially, (17) represents a linear transformation of H(X) which depends on the parameter α and the dimension $p \in \mathbb{N}$. It is also clear that the generalized Shannon entropy with α = 2 is the usual Shannon entropy, i.e. H₂ = H.

3 Entropy, Information, and the Generalized Gaussian

For a p-variate random vector X the following known Proposition bounds the Shannon entropy using only the covariance matrix of X.

Proposition 1.

Let the random vector X have zero mean and covariance matrix $\Sigma$ . Then

$$\displaystyle{\mathrm{H}(X) \leq \tfrac{1} {2}\log \left \{(2\pi e)^{p}\vert \det \Sigma \vert \right \},}$$

with equality holding if and only if $X \sim \mathcal{N}(0,\Sigma )$ .

This Proposition is crucial and denotes that the entropy for the Normal distribution is depending, eventually, only on the variance–covariance matrix, while equality holds when X is following the (multivariate) normal distribution, a result widely applied in engineering problems and information systems.

A construction of an exponential-power generalization of the usual Normal distribution can be obtained as an extremal of (an Euclidean) LSI. Following [15], the Gross Logarithm Inequality with respect to the Gaussian weight, [13], is of the form

$$\displaystyle{ \int \limits _{\mathbb{R}^{p}}\|g\|^{2}\log \|g\|^{2}dm \leq \tfrac{1} {\pi } \int \limits _{\mathbb{R}^{p}}\|\nabla g\|^{2}dm, }$$

(18)

where $\|g\|_{2}:=\int _{\mathbb{R}^{p}}\|g(x)\|^{2}dx = 1$ is the norm in $\mathcal{L}^{2}(\mathbb{R}^{p},dm)$ with $dm:=\exp \{ -\pi \vert x\vert ^{2}\}dx$. Inequality (18) is equivalent to the (Euclidean) LSI,

$$\displaystyle{ \int \limits _{\mathbb{R}^{p}}\|u\|^{2}\log \|u\|^{2}dx \leq \tfrac{p} {2}\log \left \{ \tfrac{2} {\pi pe}\int \limits _{\mathbb{R}^{p}}\|\nabla u\|^{2}dx\right \}, }$$

(19)

for any function $u \in \mathcal{W}^{1,2}(\mathbb{R}^{p})$ with $\|u\|_{2} = 1$, see [15] for details. This inequality is optimal, in the sense that

$$\displaystyle{ \tfrac{2} {\pi pe} =\inf \left \{ \frac{\int \limits _{\mathbb{R}^{p}}\|\nabla u\|^{2}dx} {\exp \left ( \tfrac{2} {n}\int \limits _{\mathbb{R}^{p}}\|u\|^{2}\log \|u\|^{2}dx\right )}: \quad u \in \mathcal{W}^{1,2}(\mathbb{R}^{p}),\quad \|u\|_{ 2} = 1\right \},}$$

see [34]. Extremals for (19) are precisely the Gaussians

$$\displaystyle{u(x) = (\pi \sigma /2)^{-p/4}\exp \left \{-\left \vert \tfrac{x-\mu } {\sigma } \right \vert ^{2}\right \},}$$

with $\sigma> 0$ and $\mu \in \mathbb{R}^{p}$, see [3, 4] for details.

Now, consider the extension of Del Pino and Dolbeault in [6] for the LSI as in (19). For any $u \in \mathcal{W}^{1,2}(\mathbb{R}^{p})$ with $\|u\|_{\gamma } = 1$, the γ-LSI holds, i.e.

$$\displaystyle{ \int \limits _{\mathbb{R}^{p}}\|u\|^{\gamma }\log \|u\|dx \leq \tfrac{p} {\gamma ^{2}} \log \left \{K_{\gamma }\int \limits _{\mathbb{R}^{p}}\|\nabla u\|^{\gamma }dx\right \}, }$$

(20)

with the optimal constant K _γ being equal to

$$\displaystyle{ K_{\gamma } = \tfrac{\gamma } {p}(\tfrac{\gamma -1} {e} )^{\gamma -1}\pi ^{-\gamma /2}\left [ \frac{\Gamma (\frac{p} {2} + 1)} {\Gamma (p\frac{\gamma -1} {\gamma } + 1)}\right ]^{\gamma /p}, }$$

(21)

where $\Gamma (\cdot )$ is the usual gamma function.

Inequality (20) is optimal and the equality holds when u(x): = f _X (x), $x \in \mathbb{R}^{p}$ where X is an r.v. following the multivariate distribution with p.d.f. f _X defined as

$$\displaystyle{ f_{X}(x) = f_{X}(x;\;\mu,\Sigma,\gamma ):= C_{\gamma }^{p}(\Sigma )\exp \left \{-\tfrac{\gamma -1} {\gamma } Q_{\theta }(x)^{ \frac{\gamma }{ 2(\gamma -1)} }\right \},\quad x \in \mathbb{R}^{p}, }$$

(22)

with normalizing factor

$$\displaystyle{ C_{\gamma }^{p}(\Sigma ):= \frac{(\tfrac{\gamma -1} {\gamma } )^{p\frac{\gamma -1} {\gamma } }} {\pi ^{p/2}\sqrt{\vert \det \Sigma \vert }}\left [ \frac{\Gamma (\frac{p} {2} + 1)} {\Gamma (p\frac{\gamma -1} {\gamma } + 1)}\right ] =\max f_{X}, }$$

(23)

and p-quadratic form $Q_{\theta }(x):= (x-\mu )\Sigma ^{-1}(x-\mu )^{\mathrm{T}}$, $x \in \mathbb{R}^{p}$ where $\theta:= (\mu,\Sigma ) \in \mathbb{R}^{p\times (p\times p)}$. The function $\phi (\gamma ) = f_{X_{\gamma }}(x)^{1/\gamma }$ with $\Sigma = (\sigma ^{2}/\alpha )^{2(\gamma -1)/\gamma }\mathbb{I}_{p}$ corresponds to the extremal function for the LSI due to [6]. The essential result is that the defined p.d.f f _X works as an extremal function to a generalized form of the Logarithmic Sobolev Inequality.

We shall write $X_{\gamma } \sim \mathcal{N}_{\gamma }^{p}(\mu,\Sigma )$ where $\mathcal{N}_{\gamma }^{p}(\mu,\Sigma )$ is an exponential-power generalization of the usual p-variate Normal distribution $\mathcal{N}^{p}(\mu,\Sigma )$ with location parameter vector $\mu \in \mathbb{R}^{1\times p}$ and positive definite scale matrix $\Sigma \in \mathbb{R}^{p\times p}$, involving a new shape parameter $\gamma \in \mathbb{R}\setminus [0,1]$. These distributions shall be referred to as the γ-order Normal distributions. It can be easily seen that the parameter vector μ is, indeed, the mean vector of the $\mathcal{N}_{\gamma }^{p}$ distribution, i.e. μ = E[X _γ ] for all parameters $\gamma \in \mathbb{R}\setminus [0,1]$, see [20]. Notice also that for γ = 2 the second-ordered Normal $\mathcal{N}_{2}^{p}(\mu,\Sigma )$ is reduced to the usual multivariate Normal $\mathcal{N}^{p}(\mu,\Sigma )$, i.e. $\mathcal{N}_{2}^{p}(\mu,\Sigma ) = \mathcal{N}^{p}(\mu,\Sigma )$. One of the merits of the γ-order Normal distribution defined above belongs to the symmetric Kotz type distributions family, [21], as $\mathcal{N}_{\gamma }^{p}(\mu,\Sigma ) = \mathcal{K}_{m,r,s}(\mu,\Sigma )$ with m: = 1, $r:= (\gamma -1)/\gamma$ and $s:=\gamma /(2\gamma - 2)$.

It is worth noting that the introduced univariate γ-order Normal $\mathcal{N}_{\gamma }(\mu,\sigma ^{2}):= \mathcal{N}_{\gamma }^{1}(\mu,\sigma ^{2})$ coincides with the existent generalized normal distribution introduced in [23], with density function

$$\displaystyle{f(x) = f(x;\;\mu,a,b):= \frac{b} {2a\Gamma (1/b)}\exp \left \{-\left \vert \tfrac{x-\mu } {a} \right \vert ^{b}\right \},\quad x \in \mathbb{R},}$$

where $a =\sigma [\gamma /(\gamma -1)]^{(\gamma -1)/\gamma }$ and $b =\gamma /(\gamma -1)$, while the multivariate case of the γ-order Normal $\mathcal{N}_{\gamma }^{p}(\mu,\Sigma )$ coincides with the existent multivariate power exponential distribution $\mathcal{P}\mathcal{E}^{p}(\mu,\Sigma ',b)$, as introduced in [10], where $\Sigma ' = 2^{2(\gamma -1)/\gamma }\Sigma$ and $b:= \frac{1} {2}\gamma /(\gamma -1)$. See also [11, 22]. These existent generalizations are technically obtained (involving an extra power parameter b) and there are not resulting from a strong mathematical background, as the Logarithmic Sobolev Inequalities offer. Moreover, they cannot provide application to the generalized Fisher Information or entropy power, etc. as their form does not really contribute to technical proofs we have already provided, see [15, 18, 20].

Denote with $\mathbb{E}_{\theta }$ the area of the p-ellipsoid $Q_{\theta }(x) \leq 1$, $x \in \mathbb{R}^{p}$. The family of $\mathcal{N}_{\gamma }^{p}(\mu,\Sigma )$, i.e. the family of the elliptically contoured γ-order Normals, provides a smooth bridging between the multivariate (and elliptically countered) Uniform, Normal and Laplace r.v. U, Z and L, i.e. between $U \sim \mathcal{U}^{p}(\mu,\Sigma )$, $Z \sim \mathcal{N}^{p}(\mu,\Sigma )$ and Laplace $L \sim \mathcal{L}^{p}(\mu,\Sigma )$ r.v. as well as the multivariate degenerate Dirac distributed r.v. $D \sim \mathcal{D}^{p}(\mu )$ (with pole at the point μ), with density functions

$$\displaystyle\begin{array}{rcl} f_{U}(x) = f_{U}(x;\;\mu,\Sigma )&:=& \left \{\begin{array}{ll} \dfrac{\Gamma (\frac{p} {2} + 1)} {\pi ^{p/2}\sqrt{\left \vert \det \Sigma \right \vert }},&x \in \mathbb{E}_{\theta }, \\ 0, &x\notin \mathbb{E}_{\theta }, \end{array} \right.{}\end{array}$$

(24)

$$\displaystyle\begin{array}{rcl} f_{Z}(x) = f_{Z}(x;\;\mu,\Sigma )&:=& \frac{1} {(2\pi )^{p/2}\sqrt{\left \vert \det \Sigma \right \vert }}\exp \left \{-\tfrac{1} {2}Q_{\theta }(x)\right \},\quad x \in \mathbb{R}^{p},{}\end{array}$$

(25)

$$\displaystyle\begin{array}{rcl} f_{L}(x) = f_{L}(x;\;\mu,\Sigma )&:=& \frac{\Gamma (\tfrac{p} {2} + 1)} {p!\pi ^{p/2}\sqrt{\left \vert \det \Sigma \right \vert }}\exp \left \{-\sqrt{Q_{\theta }(x)}\right \},\quad x \in \mathbb{R}^{p},{}\end{array}$$

(26)

$$\displaystyle\begin{array}{rcl} f_{D}(x) = f_{D}(x;\;\mu )&:=& \left \{\begin{array}{ll} + \infty,&x =\mu,\\ 0, &x \in \mathbb{R}^{p}\setminus \mu, \end{array} \right.{}\end{array}$$

(27)

respectively, see [20]. That is, the $\mathcal{N}_{\gamma }^{p}$ family of distributions generalizes not only the usual Normal but also two other significant distributions, as the Uniform and Laplace ones. The above discussion is summarized in the following Theorem, [20].

Theorem 3.

The elliptically contoured p-variate γ-order Normal distribution $\mathcal{N}_{\gamma }^{p}(\mu,\Sigma )$ for order values of $\gamma = 0,1,2,\pm \infty$ coincides with

$$\displaystyle{ \mathcal{N}_{\gamma }^{p}(\mu,\Sigma ) = \left \{\begin{array}{ll} \mathcal{D}^{p}(\mu ), &\ \ \text{for}\ \ \gamma = 0\ \ \text{and}\ \ p = 1,2, \\ 0, &\ \ \text{for}\ \ \gamma = 0\ \ \text{and}\ \ p \geq 3, \\ \mathcal{U}^{p}(\mu,\Sigma ), &\ \ \text{for}\ \ \gamma = 1, \\ \mathcal{N}^{p}(\mu,\Sigma ),&\ \ \text{for}\ \ \gamma = 2, \\ \mathcal{L}^{p}(\mu,\Sigma ), &\ \ \text{for}\ \ \gamma = \pm \infty. \end{array} \right. }$$

(28)

Remark 1.

Considering the above Theorem, the definition values of the shape parameter γ of $\mathcal{N}_{\gamma }^{p}$ distributions can be extended to include the limiting extra values of $\gamma = 0,1,\pm \infty$, respectively, i.e. γ can now be considered as a real number outside the open interval (0, 1). Particularly, when $X_{\gamma } \sim \mathcal{N}_{\gamma }^{p}(\mu,\Sigma )$, $\gamma \in \mathbb{R}\setminus (0,1) \cup \{\pm \infty \}$, the r.v. X ₀, $X_{1} \sim \mathcal{U}^{p}(\mu,\Sigma )$ and $X_{\pm \infty }\sim \mathcal{L}^{p}(\mu,\Sigma )$ can be defined as

$$\displaystyle{ X_{0}:=\mathop{\lim }\limits_{\gamma \rightarrow 0^{-}}X_{\gamma },\quad X_{ 1}:=\mathop{\lim }\limits_{\gamma \rightarrow 1^{+}}X_{\gamma },\quad X_{ \pm \infty }:=\mathop{\lim }\limits_{\gamma \rightarrow \pm \infty }X_{\gamma }. }$$

(29)

Eventually, the Uniform, Normal, Laplace and also the degenerate distribution $\mathcal{N}_{0}^{p}$ (like the Dirac one for dimensions p = 1, 2) can be considered as members of the “extended” $\mathcal{N}_{\gamma }^{p}$, $\gamma \in \mathbb{R}\setminus (0,\,1) \cup \{\pm \infty \}$, family of generalized Normal distributions.

Notice also that $\mathcal{N}_{1}^{1}(\mu,\sigma )$ coincides with the known (continuous) Uniform distribution $\mathcal{U}(\mu -\sigma,\mu +\sigma )$. Specifically, for every Uniform distribution expressed with the usual notation $\mathcal{U}(a,b)$, it holds that $\mathcal{U}(a,b) = \mathcal{N}_{1}^{1}(\frac{a+b} {2}, \frac{b-a} {2} ) = \mathcal{U}^{1}(\mu,\sigma )$. Also $\mathcal{N}_{2}(\mu,\sigma ^{2}) = \mathcal{N}(\mu,\sigma ^{2})$, $\mathcal{N}_{\pm \infty }(\mu,\sigma ^{2}) = \mathcal{L}(\mu,\sigma )$ and finally $\mathcal{N}_{0}(\mu,\sigma ) = \mathcal{D}(\mu )$. Therefore the following holds.

Corollary 1.

For order values $\gamma = 0,1,2,\pm \infty$ , the univariate γ-ordered Normal distributions $\mathcal{N}_{\gamma }^{1}(\mu,\sigma ^{2})$ coincides with the usual (univariate) degenerate Dirac $\mathcal{D}(\mu )$ , Uniform $\mathcal{U}(\mu -\sigma,\mu +\sigma )$ , Normal $\mathcal{N}(\mu,\sigma ^{2})$ , and Laplace $\mathcal{L}(\mu,\sigma )$ distributions, respectively.

Recall now the cumulative distribution function (c.d.f.) $\Phi _{Z}(z)$ of the standardized normally distributed $Z \sim \mathcal{N}(0,1)$, i.e.

$$\displaystyle{ \Phi _{Z}(z) = \tfrac{1} {2} + \tfrac{1} {2}\mathop{ \mathrm{erf}}\nolimits (\tfrac{z} {2}),\quad z \in \mathbb{R}, }$$

(30)

with $\mathop{\mathrm{erf}}\nolimits (\cdot )$ being the usual error function. For the c.d.f. of the $\mathcal{N}_{\gamma }$ family of distributions the generalized error function $\mathop{\mathrm{Erf}}\nolimits _{\gamma /(\gamma -1)}(\cdot )$ or the upper (or complementary) incomplete gamma function $\Gamma (\cdot,\cdot )$ is involved, [12]. Indeed, the following holds, [19].

Theorem 4.

Let X be a γ-order normally distributed r.v., i.e. $X \sim \mathcal{N}_{\gamma }(\mu,\sigma ^{2})$ with p.d.f. f _γ . If F _X is the c.d.f. of X and $\Phi _{Z}$ the c.d.f. of the standardized $Z = \frac{1} {\sigma } (X-\mu ) \sim \mathcal{N}_{\gamma }(0,1)$ , then

$$\displaystyle\begin{array}{rcl} F_{X}(x)& =& \Phi _{Z}(\tfrac{x-\mu } {\sigma } ) = \tfrac{1} {2} + \frac{\sqrt{\pi }} {2\Gamma (\frac{\gamma -1} {\gamma } )\Gamma ( \frac{\gamma } {\gamma -1})}\mathop{\mathrm{Erf}}\nolimits _{ \frac{\gamma }{\gamma -1} }\left \{(\tfrac{\gamma -1} {\gamma } )^{\frac{\gamma -1} {\gamma } } \tfrac{x-\mu } {\sigma }\right \}{}\end{array}$$

(31)

$$\displaystyle\begin{array}{rcl} & =& 1 - \frac{1} {2\Gamma (\frac{\gamma -1} {\gamma } )}\Gamma \left (\tfrac{\gamma -1} {\gamma }, \tfrac{\gamma -1} {\gamma } \right (\tfrac{x-\mu } {\sigma } \left )^{ \frac{\gamma }{\gamma -1} }\right ),\quad x \in \mathbb{R}.{}\end{array}$$

(32)

3.1 Shannon Entropy and Generalization

Applying the Shannon entropy on a γ-order normally distributed random variable we state and prove the following.

Theorem 5.

The Shannon entropy of a random variable $X \sim \mathcal{N}_{\gamma }^{p}(\mu,\Sigma )$ , with p.d.f. f _X , is of the form

$$\displaystyle{ \mathrm{H}(X) = p\tfrac{\gamma -1} {\gamma } -\log C_{\gamma }^{p}(\Sigma ) = p\tfrac{\gamma -1} {\gamma } -\log \max f_{X}. }$$

(33)

Proof.

From (22) and the definition (9) we obtain that the Shannon entropy of X is

$$\displaystyle{\mathrm{H}(X) = -\log C_{\gamma }^{p}(\Sigma ) + C_{\gamma }^{p}(\Sigma )\tfrac{\gamma -1} {\gamma } \int \limits _{\mathbb{R}^{p}}Q_{\theta }(x)^{ \frac{\gamma }{ 2(\gamma -1)} }\exp \left \{-\tfrac{\gamma -1} {\gamma } Q_{\theta }(x)^{ \frac{\gamma }{ 2(\gamma -1)} }\right \}dx.}$$

Applying the linear transformation $z:= (x-\mu )^{\mathrm{T}}\Sigma ^{-1/2}$ with $dx = d(x-\mu ) = \sqrt{\left \vert \det \Sigma \right \vert }dz$, the H(X _γ ) above is reduced to

$$\displaystyle{\mathrm{H}(X) = -\log C_{\gamma }^{p}(\Sigma ) + C_{\gamma }^{p}(\mathbb{I}_{ p})\tfrac{\gamma -1} {\gamma } \int \limits _{\mathbb{R}^{p}}\|z\|^{ \frac{\gamma }{\gamma -1} }\exp \left \{-\tfrac{\gamma -1} {\gamma } \|z\|^{ \frac{\gamma }{\gamma -1} }\right \}dz,}$$

where $\mathbb{I}_{p}$ denotes the p × p identity matrix. Switching to hyperspherical coordinates, we get

$$\displaystyle{\mathrm{H}(X) = -\log C_{\gamma }^{p}(\Sigma ) + C_{\gamma }^{p}(\mathbb{I}_{ p})\tfrac{\gamma -1} {\gamma } \omega _{p-1}\int \limits _{\mathbb{R}_{+}}\rho ^{ \frac{\gamma } {\gamma -1} }\exp \left \{-\tfrac{\gamma -1} {\gamma } \rho ^{ \frac{\gamma }{\gamma -1} }\right \}\rho ^{p-1}d\rho,}$$

where $\omega _{p-1}:= 2\pi ^{p/2}/\Gamma \left (\frac{p} {2}\right )$ is the volume of the (p − 1)-sphere. Applying the variable change $du:= d(\frac{\gamma -1} {\gamma } \rho ^{\gamma /(\gamma -1)}) =\rho ^{1/(\gamma -1)}d\rho$ we obtain successively

$$\displaystyle\begin{array}{rcl} \mathrm{H}(X)& =& -\log C_{\gamma }^{p}(\Sigma ) + C_{\gamma }^{p}(\mathbb{I}_{ p})\omega _{p-1}\int \limits _{\mathbb{R}_{+}}ue^{-u}\rho ^{\frac{(p-1)(\gamma -1)-1} {\gamma -1} }du {}\\ & =& \log C_{\gamma }^{p}(\Sigma ) - C_{\gamma }^{p}(\mathbb{I}_{ p})\omega _{p-1}\int \limits _{\mathbb{R}_{+}}ue^{-u}\left (\rho ^{ \frac{\gamma }{\gamma -1} }\right )^{\frac{(p-1)(\gamma -1)-1} {\gamma } }du {}\\ & =& -\log C_{\gamma }^{p}(\Sigma ) + C_{\gamma }^{p}(\mathbb{I}_{ p})\omega _{p-1}( \tfrac{\gamma } {\gamma -1})^{p\frac{\gamma -1} {\gamma } -1}\int \limits _{\mathbb{R}_{ +}}u^{p\frac{\gamma -1} {\gamma } }e^{-u}du {}\\ & =& -\log C_{\gamma }^{p}(\Sigma ) + p\tfrac{\gamma -1} {\gamma } \Gamma (p\tfrac{\gamma -1} {\gamma } )C_{\gamma }^{p}(\mathbb{I}_{ p})\omega _{p-1}. {}\\ \end{array}$$

Finally, by substitution of the volume ω _p−1 and the normalizing factor $C_{\gamma }^{p}(\Sigma )$ and $C_{\gamma }^{p}(\mathbb{I}_{p})$, as in (23), relation (33) is obtained. ⊓ ⊔

We state and prove the following Theorem which provides the results for the Shannon entropy of the elliptically contoured family of the $\mathcal{N}_{\gamma }$ distributions.

Theorem 6.

The Shannon entropy for the multivariate and elliptically countered Uniform, Normal, and Laplace distributed X (for $\gamma = 1,2,\pm \infty$ , respectively), with p.d.f. f _X , as well as for the degenerate $\mathcal{N}_{0}$ distribution, is given by

$$\displaystyle{ \mathrm{H}(X) = \left \{\begin{array}{lcll} -\log \max f_{X} & =&\log \dfrac{\pi ^{p/2}\sqrt{\vert \det \Sigma \vert }} {\Gamma \left (\frac{p} {2} + 1\right )}, &\;\text{for}\;\;X \sim \mathcal{U}^{p}(\mu,\Sigma ), \\ \frac{p} {2} -\log \max f_{X}& =&\log \sqrt{(2\pi e)^{p } \vert \det \Sigma \vert }, &\;\text{for}\;\;X \sim \mathcal{N}^{p}(\mu,\Sigma ), \\ p -\log \max f_{X} & =&\log \dfrac{p!e\pi ^{p/2}\sqrt{\vert \det \Sigma \vert }} {\Gamma \left (\frac{p} {2} + 1\right )},&\;\text{for}\;\;X \sim \mathcal{L}^{p}(\mu,\Sigma ), \\ + \infty, & & &\text{for}\;\;X \sim \mathcal{N}_{0}^{p}(\mu,\Sigma ). \end{array} \right. }$$

(34)

Proof.

Applying Theorem 3 into (33) we obtain the first three branches of (34) for γ = 1 (in limit), γ = 2 (normality), and $\gamma = \pm \infty$ (in limit), respectively. Consider now the limiting case of γ = 0. We can write (33) in the form

$$\displaystyle{\mathrm{H}(X) =\log {\biggl \{ \frac{\pi ^{p/2}\sqrt{\vert \det \Sigma \vert }} {\Gamma (\frac{p} {2} + 1)} \cdot \frac{\Gamma (pg + 1)} {(\frac{g} {e})^{pg}} \biggr \}},}$$

where $g:= \frac{\gamma -1} {\gamma }$. We then have,

$$\displaystyle{ \mathop{\lim }\limits_{\gamma \rightarrow 0^{-}}\mathrm{H}(X) =\log {\biggl \{ \frac{\pi ^{p/2}\sqrt{\vert \det \Sigma \vert }} {\Gamma (\frac{p} {2} + 1)}\mathop{\lim }\limits_{\;k = p[g] \rightarrow \infty \;}\frac{p^{k}k!} {(\frac{k} {e})^{k}}\biggr \}}, }$$

(35)

and using the Stirling’s asymptotic formula $k! \approx \sqrt{2\pi k}(\frac{k} {e})^{k}$ as $k \rightarrow \infty$, (35) finally implies

$$\displaystyle{\mathop{\mathop{\lim }}\limits_{\gamma \rightarrow 0^{-}}\mathrm{H}(X) =\log {\biggl \{ \sqrt{2\pi \vert \det \Sigma \vert } \frac{\pi ^{p/2}} {\Gamma (\frac{p} {2} + 1)}\mathop{\;\mathop{\lim }\;}\limits_{k \rightarrow \infty }p^{k}\sqrt{k}\biggr \}} = +\infty,}$$

which proves the Theorem. ⊓ ⊔

Example 1.

For the univariate case p = 1, we are reduced to

$$\displaystyle{ \mathrm{H}(X) = \left \{\begin{array}{lcllcl} -\log \max f_{X} & =&\log 2\sigma, &\;\text{for}\;\;X \sim \mathcal{N}_{1}(\mu,\sigma ) & =&\mathcal{U}(\mu -\sigma,\mu +\sigma ), \\ \frac{1} {2} -\log \max f_{X}& =&\log \sqrt{2\pi e}\sigma, &\;\text{for}\;\;X \sim \mathcal{N}_{2}(\mu,\sigma ^{2}) & =&\mathcal{N}(\mu,\sigma ^{2}), \\ 1 -\log \max f_{X} & =&1 +\log 2\sigma,&\;\text{for}\;\;X \sim \mathcal{N}_{\pm \infty }(\mu,\sigma )& =&\mathcal{L}(\mu,\sigma ), \\ + \infty, & & &\;\text{for}\;\;X \sim \mathcal{N}_{0}(\mu,\sigma ) & =&\mathcal{D}(\mu ). \end{array} \right. }$$

Figure 1 below illustrates the univariate case of Theorem 5. The Shannon entropy H(X _γ ), of an r.v. $X_{\gamma } \sim \mathcal{N}_{\gamma }(\mu,\sigma ^{2})$ is presented as a bivariate function of $\sigma \in (0,\,3]$ and $\gamma \in [-10,\,0) \cup [1,\,10]$, which forms the appeared surface (for arbitrary $\mu \in \mathbb{R}$). The Shannon entropy values of Uniform (γ = 1) and Normal (γ = 2) distributions are denoted (as curves), recall Example 1. Moreover, the entropy values of the r.v. $X_{\pm 10} \sim \mathcal{N}_{\pm 10}(\mu,\sigma ^{2})$, which approximates the Shannon entropy of Laplace distributed r.v. $X_{\pm \infty }\sim \mathcal{L}(\mu,\sigma )$, as well as the entropy of the r.v. $X_{-0.01} \sim \mathcal{N}_{-0.01}(\mu,\sigma ^{2})$ which approaches the degenerated Dirac r.v. $X_{0} \sim \mathcal{D}(\mu )$, are also depicted. One can also notice the logarithmic increase of H(X _γ ) as $\sigma$ increases (for every fixed γ value), which holds due to the form of (33).

Fig. 1

Graph of all H(X _γ ), $X_{\gamma } \sim \mathcal{N}_{\gamma }(\mu,\sigma ^{2})$ across the $\sigma (> 0)$-semi-axis and γ-axis

Due to the above proved Theorems, for the generalized Shannon entropy we obtain the following results.

Proposition 2.

The α-Shannon entropy H_α of the multivariate $X \sim \mathcal{N}_{\gamma }(\mu,\Sigma )$ is given by

$$\displaystyle\begin{array}{rcl} & & \mathrm{H}_{\alpha }(X) = \tfrac{2\gamma -\alpha } {2\gamma } p + \tfrac{p} {2}\log \left \{2\pi (\tfrac{\alpha -1} {\alpha } )^{\alpha -1}( \tfrac{\gamma } {\gamma -1})^{\alpha \frac{\gamma -1} {\gamma } }\left [\frac{\Gamma (p\frac{\gamma -1} {\gamma } + 1)} {\Gamma (p\frac{\alpha -1} {\alpha } + 1)}\right ]^{ \frac{\alpha }{ p} }\vert \det \Sigma \vert ^{ \frac{\alpha }{2p} }\right \}. \\ & & {}\end{array}$$

(36)

Moreover, in case of α = γ, we have

$$\displaystyle{ \mathrm{H}_{\gamma }(X) = \tfrac{p} {2}\log \left \{2\pi e\vert \det \Sigma \vert ^{ \frac{\gamma }{ 2p} }\right \}. }$$

(37)

Proof.

Substituting (12) and (33) into (16) we obtain

$$\displaystyle\begin{array}{rcl} \mathrm{H}_{\alpha }(X)& =& \tfrac{p} {2}\log \left \{2\pi ^{\frac{2-\alpha } {2} }e^{\frac{2\gamma -\alpha } {\gamma } }(\tfrac{\alpha -1} {\alpha } )^{\alpha -1}\right \} + {}\\ & =& \tfrac{\alpha }{2}\log {\biggl \{\pi ^{p/2}( \tfrac{\gamma } {\gamma -1})^{p\frac{\gamma -1} {\gamma } } \frac{\Gamma (p\frac{\gamma -1} {\gamma } + 1)} {\Gamma (p\frac{\alpha -1} {\alpha } + 1)}\sqrt{\vert \det \Sigma \vert }\biggr \}}, {}\\ \end{array}$$

and after some algebra we derive (36).

In case of α = γ we have $\mathrm{H}_{\gamma }(X) = \frac{p} {2}\log \{2\pi e\vert \det \Sigma \vert ^{\gamma /(2p)}\}$, i.e. (37) holds. ⊓ ⊔

Proposition 3.

For a random variable X following the multivariate Uniform, Normal, and Laplace distributions ( $\gamma = 1,2,\pm \infty$ , respectively), it is

$$\displaystyle{ \mathrm{H}_{\alpha }(X) = \left \{\begin{array}{ll} \tfrac{2-\alpha } {2} p + h(\Sigma ), &\;\text{for}\;\;X \sim \mathcal{U}^{p}(\mu,\Sigma ), \\ p + \tfrac{\alpha } {2}\log \left \{(2/e)^{p/2}\Gamma (\frac{p} {2} + 1)\right \} + h(\Sigma ),&\;\text{for}\;\;X \sim \mathcal{N}^{p}(\mu,\Sigma ), \\ p + \tfrac{p} {2}\log p! + h(\Sigma ), &\;\text{for}\;\;X \sim \mathcal{L}^{p}(\mu,\Sigma ), \end{array} \right. }$$

(38)

where

$$\displaystyle{ h(\Sigma ):= \tfrac{\alpha } {2}\log \left \{(2\pi )^{p/\alpha }(\tfrac{\alpha -1} {\alpha } )^{p\frac{\alpha -1} {\alpha } }[\Gamma (p\tfrac{\alpha -1} {\alpha } + 1)]^{-1}\sqrt{\vert \det \Sigma \vert }\right \}, }$$

(39)

while for the limiting degenerate case of $X \sim \mathcal{N}_{0}^{p}(\mu,\Sigma )$ we obtain

$$\displaystyle{ \mathrm{H}_{\alpha }(X) = \left \{\begin{array}{ll} (\mathop{\mathrm{sgn}}\nolimits \alpha )(+\infty ),&\;\text{for}\;\;\alpha \neq 0, \\ p\log \sqrt{2\pi e}, &\;\text{for}\;\;\alpha = 0. \end{array} \right. }$$

(40)

Proof.

Recall (29) and let X _γ : = X. The α-Shannon entropy of r.v. X _γ , with $\gamma = 1,\pm \infty$, can be considered as

$$\displaystyle{\mathrm{H}_{\alpha }(X_{1}):=\mathop{\lim }\limits_{\gamma \rightarrow 1^{+}}\mathrm{H}_{\alpha }(X_{\gamma }),\;\;\text{and}\;\;\mathrm{H}_{\alpha }(X_{ \pm \infty }):=\mathop{\lim }\limits_{\gamma \rightarrow \pm \infty }\mathrm{H}_{\alpha }(X_{\gamma }).}$$

Hence, for order values γ = 1 (in limit), γ = 2 and $\gamma = \pm \infty$ (in limit), we derive (38).

Consider now the limiting case of γ = 0. We can write (36) in the form

$$\displaystyle\begin{array}{rcl} \mathrm{H}_{\alpha }(X_{\gamma })& =& \tfrac{p} {2}(2 -\alpha +\gamma g) + \tfrac{p} {2}\log \left \{2\pi (\tfrac{g-1} {g} )^{\alpha -1}g^{-g\alpha }\left [\frac{\Gamma (pg + 1)\sqrt{\vert \det \Sigma \vert }} {\Gamma (p\frac{\alpha -1} {\alpha } + 1)} \right ]^{ \frac{\alpha }{ p} }\right \} {}\\ & =& \log \left \{(2\pi )^{p/2}(\tfrac{\alpha -1} {\alpha } )^{p\frac{\alpha -1} {2} }\left [ \frac{\Gamma (pg + 1)} {(\frac{g} {e})^{pg}\Gamma (p\frac{\alpha -1} {\alpha } + 1)}\right ]^{ \frac{\alpha }{ 2} }\vert \det \Sigma \vert ^{\alpha }\right \}, {}\\ \end{array}$$

where $g:= \frac{\gamma -1} {\gamma }$. We then have,

$$\displaystyle{ \mathrm{H}_{\alpha }(X_{0}):=\mathop{\lim }\limits_{\gamma \rightarrow 0^{-}}\mathrm{H}_{\alpha }(X_{\gamma }) =\log \left \{(2\pi )^{p/2}(\tfrac{\alpha -1} {\alpha } )^{p\frac{\alpha -1} {2} }\left [\mathop{\lim }\limits_{\;k:= p[g] \rightarrow \infty \;}\frac{p^{k}k!} {(\frac{k} {e})^{k}}\right ]^{ \frac{\alpha }{ 2} }\vert \det \Sigma \vert ^{\alpha }\right \}. }$$

Using the Stirling’s asymptotic formula (similar as in Theorem 6), the above relation for α ≠ 0 implies

$$\displaystyle{\mathrm{H}_{\alpha }(X_{0}) =\log \left \{(2\pi )^{p/2}(\tfrac{\alpha -1} {\alpha } )^{p\frac{\alpha -1} {2} }\vert \det \Sigma \vert ^{\alpha }\left (\mathop{\lim }\limits_{\;k \rightarrow \infty \;}p^{k}\sqrt{k}\right )^{ \frac{\alpha }{2} }\right \} = (\mathop{\mathrm{sgn}}\nolimits \alpha )(+\infty ),}$$

where $\mathop{\mathrm{sgn}}\nolimits \alpha$ is the sign of parameter α, and hence the first branch of (40) holds. For the limiting case of $\gamma =\alpha = 0$, (37) implies the second branch of (40). ⊓ ⊔

Notice that despite the rather complicated form of the H _α (X) when α ≠ γ in Proposition 2, the γ-Shannon entropy of a γ-order normally distributed X _γ has a very compact expression, see (37), while in (36) varies both the shape parameter γ (to decide the distributions “fat tails” or not) and the parameter α (of the Shannon entropy) vary.

Recall now the known relation of the Shannon entropy of a normally distributed random variable $Z \sim \mathcal{N}(\mu,\Sigma )$, i.e. $\mathrm{H}(Z) = \frac{1} {2}\log \{(2\pi e)^{p}\vert \det \Sigma \vert \}$. Therefore, H _γ (X _γ ), where $X_{\gamma } \sim \mathcal{N}_{\gamma }(\mu,\Sigma )$ generalizes H(Z), or equivalently H₂(X ₂), preserving the simple formulation for every γ, as parameter γ affects only the scale matrix $\Sigma$.

Another interesting fact about H _γ (X _γ ) is that, $\mathrm{H}_{0}(X_{0}) = \frac{p} {2}\log \{2\pi e\}$ or $\mathrm{H}_{0}(X_{0}) = -\frac{p} {4}\log \nu$, recall Corollary (40) and (25). According to (40) the Shannon entropy diverges to $+\infty$ for the degenerated distribution $\mathcal{N}_{0}$. However, the 0-Shannon entropy H₀ (in limit), for an r.v. following $\mathcal{N}_{0}$, converges to $\log \sqrt{2\pi e} = -\frac{1} {2}\log \nu \approx 1.4189$, which is the same value as the Shannon entropy of the standardized normally distributed $Z \sim \mathcal{N}(0,1)$. Thus, the generalized Shannon entropy, introduced already, can “handle” the degenerated $\mathcal{N}_{0}$ distribution in a more “coherent” way than the usual Shannon entropy (i.e., not diverging to infinity).

We can mention also that (36) expresses the generalized α-Shannon entropy of the multivariate Uniform, Normal, and Laplace distributions relative to each other. For example (recall Corollary 3), the difference of these entropies between Uniform and Laplace is independent of the same scale matrix $\Sigma$, i.e. $\mathrm{H}_{\alpha }(X_{\pm \infty }) -\mathrm{ H}_{\alpha }(X_{1}) = \frac{p} {2}(\alpha +\log p!)$, while for the usual Shannon entropy, $\mathrm{H}(X_{\pm \infty }) -\mathrm{ H}(X_{1}) = p + \frac{p} {2}\log p!$, i.e. their Shannon entropies differ by a dimension-depending constant. The difference ratio is then

$$\displaystyle{\frac{\mathrm{H}_{\alpha }(X_{\pm \infty }) -\mathrm{ H}_{\alpha }(X_{1})} {\mathrm{H}(X_{\pm \infty }) -\mathrm{ H}(X_{1})} = \frac{\log (p!e^{\alpha })} {\log (p!e^{2})}.}$$

3.2 Generalized Entropy Power

So far we have developed a generalized form for the Shannon entropy. We shall now discuss and provide general results about the generalized entropy power. The typical cases are presented in (46). Notice that, as N _α and Hα are related, some of the proofs are consequences of this relation, see Proposition 4. The following holds for different α and γ parameters.

Proposition 4.

The generalized entropy power N_α (X) of the multivariate $X_{\gamma } \sim \mathcal{N}_{\gamma }^{p}(\mu,\Sigma )$ is given, for all defined parameters $\alpha,\gamma \in \mathbb{R}\setminus [0,\,1]$ , by

$$\displaystyle{ \mathrm{N}_{\alpha }(X_{\gamma }) = \left (\tfrac{\alpha -1} {e\alpha } \right )^{\alpha -1}( \tfrac{e\gamma } {\gamma -1})^{\alpha \frac{\gamma -1} {\gamma } }\left [\frac{\Gamma (p\frac{\gamma -1} {\gamma } + 1)} {\Gamma \left (p\frac{\alpha -1} {\alpha } + 1\right )}\right ]^{\alpha /p}\vert \det \Sigma \vert ^{ \frac{\alpha }{2p} }. }$$

(41)

Moreover, in case of $\alpha =\gamma \in \mathbb{R}\setminus [0,\,1]$ ,

$$\displaystyle{ \mathrm{N}_{\gamma }(X_{\gamma }) = \vert \det \Sigma \vert ^{ \frac{\gamma }{ 2p} }. }$$

(42)

Proof.

Substituting (33) into (11), we obtain (41) and (42). ⊓⊔

Corollary 2.

For the usual entropy power of the γ-order normally distributed r.v. $X_{\gamma } \sim \mathcal{N}_{\gamma }^{p}(\mu,\Sigma )$ , we have that

$$\displaystyle{ \mathrm{N}(X_{\gamma }) = \tfrac{1} {2e}( \tfrac{e\gamma } {\gamma -1})^{2\frac{\gamma -1} {\gamma } }\left [\frac{\Gamma (p\frac{\gamma -1} {\gamma } + 1)} {\Gamma \left (\frac{p} {2} + 1\right )} \right ]^{2/p}\vert \det \Sigma \vert ^{1/p},\quad \gamma \in \mathbb{R}\setminus [0,\,1]. }$$

(43)

For the multivariate Uniform, Normal, and Laplace distributions ( $\gamma = 1,2,\pm \infty$ , respectively), as well as for the degenerate case of γ = 0, it is

$$\displaystyle{ \mathrm{N}(X) = \left \{\begin{array}{lr} \dfrac{\vert \det \Sigma \vert ^{1/p}} {2e\Gamma \left (\frac{p} {2} + 1\right )^{2/p}}, & \;\text{for}\;\;X \sim \mathcal{U}^{p}(\mu,\Sigma ), \\ \root{p}\of{\vert \det \Sigma \vert }, & \;\text{for}\;\;X \sim \mathcal{N}^{p}(\mu,\Sigma ), \\ 2^{\frac{2-p} {p} }e\left [\dfrac{(p - 1)!\sqrt{\vert \det \Sigma \vert }} {\Gamma (p/2)} \right ]^{2/p},& \;\text{for}\;\;X \sim \mathcal{L}^{p}(\mu,\Sigma ), \\ + \infty, &\;\text{for}\;\;X \sim \mathcal{N}_{0}^{p}(\mu,\Sigma ). \end{array} \right. }$$

(44)

Proof.

For the normality parameter α = 2, (43) is obtained from (41).

Recall (29) and let X _γ : = X. The usual entropy power of r.v. X _γ , with $\gamma = 1,\pm \infty$, can be considered as

$$\displaystyle{\mathrm{N}(X_{1}):=\mathop{\lim }\limits_{\gamma \rightarrow 1^{+}}\mathrm{N}(X_{\gamma })\;\;\text{and}\;\;\mathrm{N}(X_{ \pm \infty }):=\mathop{\lim }\limits_{\gamma \rightarrow \pm \infty }\mathrm{N}(X_{\gamma }).}$$

Hence, for order values γ = 1 (in limit), γ = 2 and $\gamma = \pm \infty$ (in limit), we derive the first three branches of (44).

Consider now the limiting case of γ = 0. We can write (43) in the form

$$\displaystyle{ \mathrm{N}(X_{\gamma }) = \frac{\vert \det \Sigma \vert ^{1/p}} {2e\Gamma \left (\frac{p} {2} + 1\right )^{2/p}}(\tfrac{e} {g})^{2g}\Gamma (pg + 1)^{2/p}, }$$

where $g:= \frac{\gamma -1} {\gamma }$. We then have

$$\displaystyle{ \mathrm{N}(X_{0}):=\mathop{\lim }\limits_{\gamma \rightarrow 0^{-}}\mathrm{N}(X_{\gamma }) = \frac{\vert \det \Sigma \vert ^{1/p}} {2e\Gamma \left (\frac{p} {2} + 1\right )^{2/p}}\mathop{\lim }\limits_{\;\;k:= p[g] \rightarrow \infty \;}(\tfrac{ep} {k} )^{2k/p}(k!)^{2/p}. }$$

Using the Stirling’s asymptotic formula (similar as in Theorem 6), the above relation implies

$$\displaystyle{\mathrm{N}(X_{0}) = \frac{(2\pi \vert \det \Sigma \vert )^{1/p}} {2e\Gamma \left (\frac{p} {2} + 1\right )^{2/p}}\mathop{\lim }\limits_{\;k \rightarrow \infty \;}p^{2k/p}k^{1/p} = +\infty,}$$

and hence the last branch of (44) holds. ⊓ ⊔

Example 2.

For the univariate p = 1, (43) implies

$$\displaystyle{ \mathrm{N}(X_{\gamma }) = \tfrac{2} {\pi e}( \tfrac{e\gamma } {\gamma -1})^{2\frac{\gamma -1} {\gamma } }\Gamma (\tfrac{\gamma -1} {\gamma } + 1)^{2}\sigma ^{2}, }$$

(45)

and thus we derive from (44) that

$$\displaystyle{ \mathrm{N}(X) = \left \{\begin{array}{lr} \frac{b-a} {\pi e}, &\;\text{for}\;\;X \sim \mathcal{U}(a,b), \\ \sigma ^{2}, & \;\text{for}\;\;X \sim \mathcal{N}(\mu,\sigma ^{2}), \\ \tfrac{2e\sigma } {\pi }, & \;\text{for}\;\;X \sim \mathcal{L}(\mu,\sigma ), \\ + \infty,& \;\text{for}\;\;X \sim \mathcal{D}(\mu ). \end{array} \right. }$$

(46)

Corollary 3.

For the generalized entropy power of the multivariate Uniform, Normal, and Laplace distributions ( $\gamma = 1,2,\pm \infty$ , respectively), it is

$$\displaystyle{ \mathrm{N}_{\alpha }(X) = \left \{\begin{array}{lr} \left (\tfrac{\alpha -1} {e\alpha } \right )^{\alpha -1} \dfrac{\vert \det \Sigma \vert ^{ \frac{\alpha }{ 2p} }} {\Gamma \left (p\frac{\alpha -1} {\alpha } + 1\right )^{\alpha /p}}, & \;\text{for}\;\;X \sim \mathcal{U}^{p}(\mu,\Sigma ), \\ \left (\tfrac{\alpha -1} {e\alpha } \right )^{\alpha -1}(2e)^{\alpha /2}\left [ \dfrac{\Gamma (\frac{p} {2} + 1)} {\Gamma \left (p\frac{\alpha -1} {\alpha } + 1\right )}\right ]^{\alpha /p}\vert \det \Sigma \vert ^{ \frac{\alpha }{ 2p} },&\;\text{for}\;\;X \sim \mathcal{N}^{p}(\mu,\Sigma ), \\ e\left (\tfrac{\alpha -1} {\alpha } \right )^{\alpha -1}\left [ \dfrac{p!} {\Gamma \left (p\frac{\alpha -1} {\alpha } + 1\right )}\right ]^{\alpha /p}\vert \det \Sigma \vert ^{ \frac{\alpha }{ 2p} }, & \;\text{for}\;\;X \sim \mathcal{L}^{p}(\mu,\Sigma ), \end{array} \right. }$$

(47)

while for the degenerate case of $X \sim \mathcal{N}_{0}^{p}(\mu,\Sigma )$ we have

$$\displaystyle{ \mathrm{N}_{\alpha }(X) = \left \{\begin{array}{lr} + \infty,&\;\text{for}\;\;\alpha> 1,\\ 0, &\;\text{for} \;\;\alpha <0. \end{array} \right. }$$

(48)

Proof.

Recall (29) and let $X_{\gamma } \sim \mathcal{N}_{\gamma }(\mu,\Sigma )$. The generalized entropy power of r.v. X _γ , with $\gamma = 1,\pm \infty$, can be considered as

$$\displaystyle{\mathrm{N}_{\alpha }(X_{1}):=\mathop{\lim }\limits_{\gamma \rightarrow 1^{+}}\mathrm{N}_{\alpha }(X_{\gamma }),\;\;\text{and}\;\;\mathrm{N}_{\alpha }(X_{ \pm \infty }):=\mathop{\lim }\limits_{\gamma \rightarrow \pm \infty }\mathrm{N}_{\alpha }(X_{\gamma }),}$$

and hence, for order values γ = 1 (in limit), γ = 2 and $\gamma = \pm \infty$ (in limit), we derive (47).

Consider now the limiting case of γ = 0. We can write (41) in the form

$$\displaystyle{ \mathrm{N}_{\alpha }(X_{\gamma }) = \left (\tfrac{\alpha -1} {e\alpha } \right )^{\alpha -1}(\tfrac{e} {g})^{g\alpha }\left [ \frac{\Gamma (pg + 1)} {\Gamma \left (p\frac{\alpha -1} {\alpha } + 1\right )}\right ]^{\alpha /p}\vert \det \Sigma \vert ^{ \frac{\alpha }{2p} }, }$$

where $g:= \frac{\gamma -1} {\gamma }$. We then have

$$\displaystyle{ \mathrm{N}_{\alpha }(X_{0}):=\mathop{\lim }\limits_{\gamma \rightarrow 0^{-}}\mathrm{N}_{\alpha }(X_{\gamma }) = \frac{\left (\tfrac{\alpha -1} {e\alpha } \right )^{\alpha -1}\vert \det \Sigma \vert ^{ \frac{\alpha }{2p} }} {\Gamma \left (p\frac{\alpha -1} {\alpha } + 1\right )^{\alpha /p}}\mathop{\lim }\limits_{\;\;k:= p[g] \rightarrow \infty \;}(\tfrac{ep} {k} )^{\alpha k/p}(k!)^{\alpha /p}. }$$

Using the Stirling’s asymptotic formula, the above relation implies

$$\displaystyle{ \mathrm{N}_{\alpha }(X_{0}) = \frac{\left (\tfrac{\alpha -1} {e\alpha } \right )^{\alpha -1}\vert \det \Sigma \vert ^{ \frac{\alpha }{2p} }} {\Gamma \left (p\frac{\alpha -1} {\alpha } + 1\right )^{\alpha /p}}\mathop{\lim }\limits_{\;k \rightarrow \infty \;}(2\pi p^{2}k)^{\alpha k/(2p)} = \left \{\begin{array}{lr} + \infty,&\;\;\text{for}\;\;\alpha> 1, \\ 0, &\;\;\text{for}\;\;\alpha <0, \end{array} \right. }$$

and hence (48) holds. ⊓ ⊔

For the special cases $\alpha = 0,1,\pm \infty$ of the parameter $\alpha \in \mathbb{R}\setminus [0,\,1]$ of the generalized entropy power, the following holds.

Proposition 5.

The generalized entropy power N_α (X), for the limiting values $\alpha = 0,1,\pm \infty$ , of the multivariate r.v. $X_{\gamma } \sim \mathcal{N}_{\gamma }^{p}(\mu,\Sigma )$ , for all shape parameter values $\gamma \in \mathbb{R}\setminus [0,\,1]$ , is given by

$$\displaystyle\begin{array}{rcl} \mathrm{N}_{0}(X_{\gamma })& =& p,{}\end{array}$$

(49)

$$\displaystyle\begin{array}{rcl} \mathrm{N}_{1}(X_{\gamma })& =& ( \tfrac{e\gamma } {\gamma -1})^{\frac{\gamma -1} {\gamma } }\Gamma (p\tfrac{\gamma -1} {\gamma } + 1)^{\frac{1} {p} }\vert \det \Sigma \vert ^{\frac{1} {2p} },{}\end{array}$$

(50)

$$\displaystyle\begin{array}{rcl} \mathrm{N}_{+\infty }(X_{\gamma })& =& \left \{\begin{array}{lr} + \infty,&\;\;\text{for}\;\;\vert \det \Sigma \vert> S_{\gamma }^{2}, \\ 0, &\;\;\text{for}\;\;\vert \det \Sigma \vert <S_{\gamma }^{2}, \end{array} \right.{}\end{array}$$

(51)

$$\displaystyle\begin{array}{rcl} \mathrm{N}_{-\infty }(X_{\gamma })& =& \left \{\begin{array}{lr} + \infty,& \;\;\text{for}\;\;\vert \det \Sigma \vert <S_{\gamma }^{2}, \\ 0, &\;\;\text{for}\;\;\vert \det \Sigma \vert> S_{\gamma }^{2}, \end{array} \right.{}\end{array}$$

(52)

where

$$\displaystyle{ S_{\gamma }:= \dfrac{e^{p}p!(\tfrac{\gamma -1} {e\gamma } )^{p\frac{\gamma -1} {\gamma } }} {\Gamma (p\frac{\gamma -1} {\gamma } + 1)}. }$$

(53)

Proof.

For the limiting value α = 0, we can consider

$$\displaystyle{ \mathrm{N}_{0}(X_{\gamma }):=\mathop{\lim }\limits_{\alpha \rightarrow 0^{-}}\mathrm{N}_{\alpha }(X_{\gamma }), }$$

which can be written, through (41), into the form

$$\displaystyle{\mathrm{N}_{0}(X_{\gamma }) =\mathop{\lim }\limits_{\beta \rightarrow +\infty }( \tfrac{\beta }{e})^{ \frac{\beta }{ 1-\beta }}\left [\frac{\Gamma (p\frac{\gamma -1} {\gamma } + 1)} {\Gamma (p\beta + 1)} \right ]^{ \frac{1} {p(1-\beta )} }\vert \det \Sigma \vert ^{ \frac{1} {2p(1-\beta )} },}$$

where $\beta:= \frac{\alpha -1} {\alpha }$, or

$$\displaystyle{\mathrm{N}_{0}(X_{\gamma }) =\mathop{\lim }\limits_{ k:= p[\beta ] \rightarrow \infty }( \tfrac{k} {pe})^{ \frac{k} {p-k} }\left [\frac{\Gamma (p\frac{\gamma -1} {\gamma } + 1)} {k!} \right ]^{ \frac{1} {p-k} }\vert \det \Sigma \vert ^{ \frac{1} {2(p-k)} }.}$$

Applying the Stirling’s asymptotic formula for k! , the above relation implies

$$\displaystyle{\mathrm{N}_{0}(X_{\gamma }) =\mathop{\lim }\limits_{ k \rightarrow \infty }\left [\frac{\Gamma (p\frac{\gamma -1} {\gamma } + 1)\sqrt{\vert \det \Sigma \vert }} {p^{k}\sqrt{2\pi k}} \right ]^{ \frac{1} {p-k} } =\mathop{\lim }\limits_{\;\; k \rightarrow \infty \;}p^{ \frac{k} {k-p} }k^{ \frac{1} {2(k-p)} } = p \cdot 1 = p,}$$

and therefore, (49) holds due to the fact that

$$\displaystyle{ \mathop{\lim }\limits_{k \rightarrow \infty }k^{ \frac{1} {2(k-p)} } =\exp \left \{\tfrac{1} {2}\mathop{\lim }\limits_{\;\;k \rightarrow \infty \;} \frac{\log k} {k - p}\right \} = e^{0} = 1. }$$

(54)

For the limiting value α = 1, we can consider

$$\displaystyle{ \mathrm{N}_{1}(X_{\gamma }):=\mathop{\lim }\limits_{\alpha \rightarrow 1^{+}}\mathrm{N}_{\alpha }(X_{\gamma }), }$$

and thus (50) hold, through (41).

For the limiting value $\alpha = \pm \infty$, we can consider

$$\displaystyle{ \mathrm{N}_{\pm \infty }(X_{\gamma }):=\mathop{\lim }\limits_{\alpha \rightarrow \pm \infty }\mathrm{N}_{\alpha }(X_{\gamma }) =\mathop{\lim }\limits_{\alpha \rightarrow \pm \infty }\left [ \frac{A(\Sigma )} {\Gamma (p\frac{\alpha -1} {\alpha } + 1)}\right ]^{\alpha /p}, }$$

(55)

where

$$\displaystyle{ A(\Sigma ):= e^{-p}( \tfrac{e\gamma } {\gamma -1})^{p\frac{\gamma -1} {\gamma } }\Gamma (p\tfrac{\gamma -1} {\gamma } + 1)\sqrt{\vert \det \Sigma \vert }, }$$

(56)

due to (41) and the fact that $\lim _{\alpha \rightarrow \pm \infty }[(\alpha -1)/\alpha ]^{\alpha -1} = e^{-1}$. Moreover, $\Gamma (p\frac{\alpha -1} {\alpha } + 1) \rightarrow p!$ as $\alpha \rightarrow \pm \infty$, and thus, from (55) we obtain (51) and (52). ⊓ ⊔

Corollary 2 presents the usual entropy power $\mathrm{N}(X) =\mathrm{ N}_{\alpha =2}(X)$ when X follows a Uniform, Normal, Laplace, or a degenerated ($\mathcal{N}_{0}$) random variable. The following Proposition investigates the limiting cases of $\mathrm{N}_{\alpha =0,1,\pm \infty }(X)$, as it provides results for applications working with “extreme-tailed” distributions. Notice the essential use of the quantity S ₂, as in (65), for the determinant of the distributions’ scale matrix $\Sigma$, that alters the behavior of the extreme case of $\mathrm{N}_{\pm \infty }$.

Proposition 6.

For the multivariate Uniform, Normal, and Laplace distributions, i.e. $\mathcal{N}_{\gamma =1,2,\pm \infty }$ , as well as for the degenerate $\mathcal{N}_{\gamma =0}$ , the “limiting values” of the generalized entropy power $\mathrm{N}_{\alpha =0,1,\pm \infty }$ , are given by

$$\displaystyle{ \mathrm{N}_{0}(X) = \left \{\begin{array}{lr} p,& \;\text{for}\;\;X \sim \mathcal{U}^{p}(\mu,\Sigma ), \\ p,& \;\text{for}\;\;X \sim \mathcal{N}^{p}(\mu,\Sigma ), \\ p,& \;\text{for}\;\;X \sim \mathcal{L}^{p}(\mu,\Sigma ), \\ 1,&\;\text{for}\;\;X \sim \mathcal{N}_{0}^{p}(\mu,\Sigma ), \end{array} \right. }$$

(57)

$$\displaystyle{ \mathrm{N}_{1}(X) = \left \{\begin{array}{lr} \vert \det \Sigma \vert ^{\frac{1} {2p} }, & \;\text{for}\;\;X \sim \mathcal{U}^{p}(\mu,\Sigma ), \\ \sqrt{2e}\Gamma (\frac{p} {2} + 1)^{1/p}\vert \det \Sigma \vert ^{\frac{1} {2p} },& \;\text{for}\;\;X \sim \mathcal{N}^{p}(\mu,\Sigma ), \\ e(p!)^{1/p}\vert \det \Sigma \vert ^{\frac{1} {2p} }, & \;\text{for}\;\;X \sim \mathcal{L}^{p}(\mu,\Sigma ), \\ + \infty, &\;\text{for}\;\;X \sim \mathcal{N}_{0}^{p}(\mu,\Sigma ), \end{array} \right. }$$

(58)

$$\displaystyle\begin{array}{rcl} \mathrm{N}_{+\infty }(X)& =& \left \{\begin{array}{lr} + \infty,&\;\;\text{for}\;\;\vert \det \Sigma \vert> (e^{p}p!)^{2}, \\ 0, &\;\;\text{for}\;\;\vert \det \Sigma \vert <(e^{p}p!)^{2}, \end{array} \right.\;\text{and}\;\;X \sim \mathcal{U}^{p}(\mu,\Sigma ),{}\end{array}$$

(59)

$$\displaystyle\begin{array}{rcl} \mathrm{N}_{-\infty }(X)& =& \left \{\begin{array}{lr} + \infty,&\;\;\text{for}\;\;\vert \det \Sigma \vert <(e^{p}p!)^{2}, \\ 0, &\;\;\text{for}\;\;\vert \det \Sigma \vert> (e^{p}p!)^{2}, \end{array} \right.\;\text{and}\;\;X \sim \mathcal{U}^{p}(\mu,\Sigma ),{}\end{array}$$

(60)

$$\displaystyle\begin{array}{rcl} \mathrm{N}_{+\infty }(X)& =& \left \{\begin{array}{lr} + \infty,&\;\;\text{for}\;\;\vert \det \Sigma \vert> S_{2}^{2}, \\ 0, &\;\;\text{for}\;\;\vert \det \Sigma \vert <S_{2}^{2}, \end{array} \right.\;\text{and}\;\;X \sim \mathcal{N}^{p}(\mu,\Sigma ),{}\end{array}$$

(61)

$$\displaystyle\begin{array}{rcl} \mathrm{N}_{-\infty }(X)& =& \left \{\begin{array}{lr} + \infty,&\;\;\text{for}\;\;\vert \det \Sigma \vert <S_{2}^{2}, \\ 0, & \;\;\text{for}\;\;\vert \det \Sigma \vert> S_{2}, \end{array} \right.\;\text{and}\;\;X \sim \mathcal{N}^{p}(\mu,\Sigma ),{}\end{array}$$

(62)

$$\displaystyle\begin{array}{rcl} \mathrm{N}_{+\infty }(X)& =& \left \{\begin{array}{lr} + \infty,&\;\;\text{for}\;\;\vert \det \Sigma \vert> 1,\\ 0, &\;\;\text{for} \;\;\vert \det \Sigma \vert <1, \end{array} \right.\;\text{and}\;\;X \sim \mathcal{L}^{p}(\mu,\Sigma ),{}\end{array}$$

(63)

$$\displaystyle\begin{array}{rcl} \mathrm{N}_{-\infty }(X)& =& \left \{\begin{array}{lr} + \infty,&\;\;\text{for}\;\;\vert \det \Sigma \vert <1,\\ 0, &\;\;\text{for} \;\;\vert \det \Sigma \vert> 1, \end{array} \right.\;\text{and}\;\;X \sim \mathcal{L}^{p}(\mu,\Sigma ),{}\end{array}$$

(64)

where

$$\displaystyle{ S_{2}:= \dfrac{e^{p}p!} {(2e)^{p/2}\Gamma (\frac{p} {2} + 1)}. }$$

(65)

Proof.

For the limiting value α = 1, the first three branches of (58) holds, through (47). Moreover, for the degenerate case of $\mathcal{N}_{\gamma =0}$, we consider $\mathrm{N}_{1}(X_{0}):=\lim _{\gamma \rightarrow 0^{-}}\mathrm{N}_{1}(X_{\gamma })$, with $X_{\gamma } \sim \mathcal{N}_{\gamma }(\mu,\Sigma )$, i.e.

$$\displaystyle{ \mathrm{N}_{1}(X_{0}) =\mathop{\lim }\limits_{ g \rightarrow +\infty }(\tfrac{e} {g})^{g}\Gamma (pg + 1)^{\frac{1} {p} }\vert \det \Sigma \vert ^{\frac{1} {2p} }, }$$

where $g:= \frac{\gamma -1} {\gamma }$. Then,

$$\displaystyle{\mathrm{N}_{1}(X_{0}) =\mathop{\lim }\limits_{ k:= p[g] \rightarrow \infty }(\tfrac{ep} {k} )^{k/p}(k!)^{\frac{1} {p} }\vert \det \Sigma \vert ^{\frac{1} {2p} }.}$$

Applying the Stirling’s asymptotic formula of k! , the above relation implies

$$\displaystyle{\mathrm{N}_{1}(X_{0}) =\mathop{\lim }\limits_{ k \rightarrow \infty }p^{k/p}(2\pi k\vert \det \Sigma \vert )^{ \frac{1} {2p} } = +\infty,}$$

and thus (58) holds.

For the limiting value $\alpha = \pm \infty$ and $X \sim \mathcal{N}_{1}(\mu,\Sigma ) = \mathcal{U}^{p}(\mu,\Sigma )$, we consider $\mathrm{N}_{\pm \infty }(X):=\lim _{\alpha \rightarrow \pm \infty }\mathrm{N}_{\alpha }(X)$, i.e.

$$\displaystyle{ \mathrm{N}_{\pm \infty }(X) =\mathop{\lim }\limits_{\alpha \rightarrow \pm \infty }\left [ \frac{\sqrt{\vert \det \Sigma \vert }} {e^{p}\Gamma (p\frac{\alpha -1} {\alpha } + 1)}\right ]^{\alpha /p}, }$$

(66)

from (47). Moreover, $\Gamma (p\frac{\alpha -1} {\alpha } + 1) \rightarrow p!$ as $\alpha \rightarrow \pm \infty$, and thus, from (66), we obtain (59) and (60)

For the limiting value $\alpha = \pm \infty$ and $X \sim \mathcal{N}_{2}(\mu,\Sigma ) = \mathcal{N}^{p}(\mu,\Sigma )$, relations (61) and (62) hold due to (51) and (52), where S ₂ as in (53) with γ = 2.

For the limiting value $\alpha = \pm \infty$ and $X \sim \mathcal{N}_{\pm \infty }(\mu,\Sigma ) = \mathcal{L}^{p}(\mu,\Sigma )$, relations (63) and (64) hold due to (42) with $\gamma \rightarrow \pm \infty$.

For the limiting value α = 0 and $X \sim \mathcal{N}_{1}(\mu,\Sigma ) = \mathcal{U}^{p}(\mu,\Sigma )$ we consider $\mathrm{N}_{0}(X):=\lim _{\alpha \rightarrow 0^{-}}\mathrm{N}_{\alpha }(X)$ which can be written, through the first branch of (47), into the form

$$\displaystyle{\mathrm{N}_{0}(X) =\mathop{\lim }\limits_{\beta \rightarrow +\infty \;\;} \frac{( \tfrac{\beta }{e})^{ \frac{\beta }{ 1-\beta }}} {\Gamma (p\beta + 1)^{ \frac{1} {p(1-\beta )} }} \vert \det \Sigma \vert ^{ \frac{1} {2p(1-\beta )} },}$$

where $\beta:= \frac{\alpha -1} {\alpha }$, or

$$\displaystyle{\mathrm{N}_{0}(X) =\mathop{\lim }\limits_{ k:= p[\beta ] \rightarrow \infty }( \tfrac{k} {pe})^{ \frac{k} {p-k} }(k!)^{ \frac{1} {k-p} }\vert \det \Sigma \vert ^{ \frac{1} {2(p-k)} }.}$$

Applying the Stirling’s asymptotic formula for k! , the above relation implies

$$\displaystyle{\mathrm{N}_{0}(X) =\mathop{\lim }\limits_{ k \rightarrow \infty }\left ( \frac{\sqrt{\vert \det \Sigma \vert }} {p^{k}\sqrt{2\pi k}}\right )^{ \frac{1} {p-k} } =\mathop{\lim }\limits_{\;\; k \rightarrow \infty \;}p^{ \frac{k} {k-p} }k^{ \frac{1} {2(k-p)} } = p \cdot 1 = p,}$$

and therefore the first branch of (57) holds due to (54).

For the limiting value α = 0 and $X \sim \mathcal{N}_{2}(\mu,\Sigma ) = \mathcal{N}^{p}(\mu,\Sigma )$, the second branch of (57) holds due to (49).

For the limiting value α = 0 and $X \sim \mathcal{N}_{\pm \infty }(\mu,\Sigma ) = \mathcal{L}^{p}(\mu,\Sigma )$ we consider $\mathrm{N}_{0}(X):=\lim _{\alpha \rightarrow 0^{-}}\mathrm{N}_{\alpha }(X)$ which can be written, through the last branch of (47), into the form

$$\displaystyle{\mathrm{N}_{0}(X) =\mathop{\lim }\limits_{\beta \rightarrow +\infty \;\;}( \tfrac{\beta }{e})^{ \frac{\beta }{ 1-\beta }}\left [ \frac{p!} {\Gamma (p\beta + 1)}\right ]^{ \frac{1} {p(1-\beta )} }\vert \det \Sigma \vert ^{ \frac{1} {2p(1-\beta )} },}$$

or

$$\displaystyle{\mathrm{N}_{0}(X) =\mathop{\lim }\limits_{ k:= p[\beta ] \rightarrow \infty }( \tfrac{k} {pe})^{ \frac{k} {p-k} }(\tfrac{p!} {k!})^{ \frac{1} {p-k} }\vert \det \Sigma \vert ^{ \frac{1} {2(p-k)} }.}$$

Applying, again, the Stirling’s asymptotic formula for k! , the above relation implies

$$\displaystyle{\mathrm{N}_{0}(X) =\mathop{\lim }\limits_{ k \rightarrow \infty }\left (\frac{p!\sqrt{\vert \det \Sigma \vert }} {p^{k}\sqrt{2\pi k}} \right )^{ \frac{1} {p-k} } =\mathop{\lim }\limits_{\;\; k \rightarrow \infty \;}p^{ \frac{k} {k-p} }k^{ \frac{1} {2(k-p)} } = p \cdot 1 = p,}$$

and therefore the third branch of (57) holds due to (54).

For the limiting value α = 0 and $X \sim \mathcal{N}_{0}(\mu,\Sigma )$, the last branch of (57) holds due to (42) with γ → 0⁻. ⊓⊔

Recall Proposition 5 where $\gamma \in \mathbb{R}\setminus [0,\,1]$. For the limiting extra values of γ = 1 (Uniform case), $\gamma = \pm \infty$ (Laplace case), and γ = 0 (degenerate case), the results (50)–(52) still hold in limit, see (58) and from (59) to (64). Therefore, the relations (50)–(52) hold for all shape parameters γ taking values over its “extended” domain, i.e. $\gamma \in \mathbb{R}\setminus (0,\,1) \cup \{\pm \infty \}$. However, from (49) and (57), it holds that

$$\displaystyle{ \mathrm{N}_{0}(X_{\gamma }) = \left \{\begin{array}{ll} p,&\;\text{for}\;\;\gamma \in \mathbb{R}\setminus [0,\,1) \cup \{\pm \infty \},\\ 1, &\;\text{for} \;\;\gamma = 0. \end{array} \right. }$$

(67)

while for the univariate case, the generalized entropy power N₀, as in (67), is always unity for all the members of the “extended” γ-order Normal distribution’s family, i.e. N₀(X _γ ) = 1 with $\gamma \in \mathbb{R}\setminus (0,\,1) \cup \{\pm \infty \}$.

Figure 2 presents the generalized entropy power N _α (X _γ ) as a function of its parameter $\alpha \in \mathbb{R}\setminus [0,\,1]$ for various $X_{\gamma } \sim \mathcal{N}_{\gamma }(0,1)$ random variables, with the special cases of Uniform (γ = 1), Normal (γ = 2), and Laplace ($\gamma = \pm \infty$) r.v. being denoted. Figure 3 depicts the cases of $X_{\gamma } \sim \mathcal{N}_{\gamma }(0,\sigma ^{2})$ with $\sigma <1$ (left sub-Figure) and $\sigma> 1$ (right sub-Figure).

Fig. 2

Graphs of N _α (X _γ ) along α, for various γ values, where $X_{\gamma } \sim \mathcal{N}_{\gamma }(0,1)$

Fig. 3

Graphs of N _α (X _γ ) along α, for various γ values, where $X_{\gamma } \sim \mathcal{N}_{\gamma }(0,0.8)$ (left-side) and $X_{\gamma } \sim \mathcal{N}_{\gamma }(0,1.5)$ (right-side)

3.3 Rényi Entropy

We discuss now the Rényi entropy, another significant entropy measure which also generalizes Shannon entropy, and can be best introduced through the concept of generalized random variables. These variables extend the usual notion of a random experiment that cannot always be observed. See for details the Rényi’s original work in [25] and [26].

See [5]. For a p-variate continuous random variable, with p.d.f. f _X , the Rényi entropy R _α (X) is defined, through the α-norm $\|\cdot \|_{\alpha }$ on $\mathcal{L}^{\alpha }(\mathbb{R}^{p})$, by

$$\displaystyle{ \mathrm{R}_{\alpha }(X):= - \tfrac{\alpha }{\alpha -1}\log \|f_{X}\|_{\alpha } = \tfrac{1} {1-\alpha }\log \int \limits _{\mathbb{R}^{p}}\vert f_{X}(x)\vert ^{\alpha }dx, }$$

(68)

with $\alpha \in \mathbb{R}_{+}^{{\ast}}\setminus 1$, i.e. 0 < α ≠ 1. For the limiting case of α = 1 the Rényi entropy converges to the usual Shannon entropy H(X) as in (9). Notice that we use the minus sign for R _α to be in accordance with the definition of (9), where we reject the usual minus sign of the Shannon entropy definition.

Considering now an r.v. from the $\mathcal{N}_{\gamma }^{p}$ family of the generalized Normal distributions, the following Theorem provides a general result to calculate the Rényi entropy for different α and γ parameters.

Theorem 7.

For the elliptically contoured γ-order normally distributed r.v. $X_{\gamma } \sim \mathcal{N}_{\gamma }^{p}(\mu,\Sigma )$ , with p.d.f. $f_{X_{\gamma }}$ , the Rényi R_α entropy of X _γ is given by

$$\displaystyle{ \mathrm{R}_{\alpha }(X_{\gamma }) = p \tfrac{\gamma -1} {\gamma (\alpha -1)}\log \alpha -\log C_{\gamma }^{p}(\Sigma ) = p \tfrac{\gamma -1} {\gamma (\alpha -1)}\log \alpha -\log \max f_{X_{\gamma }}, }$$

(69)

for all the defined parameters $\alpha \in \mathbb{R}_{+}^{{\ast}}\setminus \{1\}$ and $\gamma \in \mathbb{R}\setminus [0,\,1]$ .

Proof.

Consider the p.d.f. $f_{X_{\gamma }}$ as in (22). From the definition (68) it is

$$\displaystyle{\mathrm{R}_{\alpha }(X_{\gamma }) = \tfrac{\alpha } {1-\alpha }\log C_{\gamma }^{p}(\Sigma ) + \tfrac{1} {1-\alpha }\log \int \limits _{\mathbb{R}^{p}}\exp \left \{-\tfrac{\alpha (\gamma -1)} {\gamma } \left [(x-\mu )\Sigma ^{-1}(x-\mu )^{\mathrm{T}}\right ]^{ \frac{\gamma }{2(\gamma -1)} }\right \}dx.}$$

Applying the linear transformation $z = (x-\mu )\Sigma ^{-1/2}$ with $dx = d(x-\mu ) = \sqrt{\left \vert \det \Sigma \right \vert }dz$, the R _α above is reduced to

$$\displaystyle{\mathrm{R}_{\alpha }(X_{\gamma }) = \tfrac{\alpha } {1-\alpha }\log C_{\gamma }^{p}(\Sigma ) + \tfrac{1} {1-\alpha }\log \int \limits _{\mathbb{R}^{p}}\exp \left \{-\tfrac{\alpha (\gamma -1)} {\gamma } \|z\|^{ \frac{\gamma }{\gamma -1} }\right \}dz.}$$

Switching to hyperspherical coordinates, we get

$$\displaystyle{\mathrm{R}_{\alpha }(X_{\gamma }) = \tfrac{\alpha } {1-\alpha }\log \left \{C_{\gamma }^{p}(\Sigma )\omega _{ p-1}^{1/\alpha }\right \} + \tfrac{1} {1-\alpha }\log \int \limits _{\mathbb{R}_{+}}\exp \left \{-\tfrac{\alpha (\gamma -1)} {\gamma } \rho ^{ \frac{\gamma }{\gamma -1} }\right \}\rho ^{p-1}d\rho,}$$

where $\omega _{p-1} = 2\pi ^{p/2}/\Gamma \left (\frac{p} {2}\right )$ is the volume of the (p − 1)-sphere. Assuming $du:= d(\frac{\gamma -1} {\gamma } \rho ^{\gamma /(\gamma -1)}) =\rho ^{1/(\gamma -1)}d\rho$ we obtain successively

$$\displaystyle{ \begin{array}{rcl} \mathrm{R}_{\alpha }(X_{\gamma })& =& \tfrac{\alpha } {1-\alpha }\log M(\Sigma ) + \tfrac{1} {1-\alpha }\log \int \limits _{\mathbb{R}_{+}}e^{-\alpha u}\rho ^{\frac{(p-1)(\gamma -1)-1} {\gamma -1} }du \\ & =& \tfrac{\alpha } {\alpha -1}\log M(\Sigma ) + \tfrac{1} {1-\alpha }\log \int \limits _{\mathbb{R}_{+}}e^{-\alpha u}\left (\rho ^{ \frac{\gamma }{\gamma -1} }\right )^{\frac{(p-1)(\gamma -1)-1} {\gamma } }du \\ & =& \tfrac{\alpha } {1-\alpha }\log M(\Sigma ) + \tfrac{1} {1-\alpha }\log ( \tfrac{\gamma } {\gamma -1})^{p\frac{\gamma -1} {\gamma } -1} + \tfrac{1} {1-\alpha }\log \int \limits _{\mathbb{R}_{ +}}e^{-\alpha u}u^{p\frac{\gamma -1} {\gamma } -1}du \\ & =& \tfrac{\alpha } {1-\alpha }\log M(\Sigma ) + \tfrac{1} {1-\alpha }\log ( \tfrac{\gamma } {\gamma -1})^{p\frac{\gamma -1} {\gamma } -1} - p\tfrac{\gamma -1} {\gamma } \cdot \tfrac{\log \alpha } {1-\alpha } + \tfrac{1} {1-\alpha }\log \Gamma (p\tfrac{\gamma -1} {\gamma }), \end{array} }$$

where $M(\Sigma ):= C_{\gamma }^{p}(\Sigma )\omega _{p-1}^{1/\alpha }$. Finally, by substitution of the volume ω _p−1 we obtain, through the normalizing factor $C_{\gamma }^{p}(\Sigma )$ as in (23),

$$\displaystyle{\mathrm{R}_{\alpha }(X_{\gamma }) = - \tfrac{\alpha }{\alpha -1}\log C_{\gamma }^{p}(\Sigma ) + \tfrac{1} {\alpha -1}\log C_{\gamma }^{p}(\Sigma ) + p\tfrac{\gamma -1} {\gamma } \cdot \tfrac{\log \alpha } {\alpha -1},}$$

and thus (69) holds true. □ 

For the limiting parameter values $\alpha = 0,1,+\infty$ we obtain a number of results for other well-known measures of entropy, applicable to Cryptography, as the Hartley entropy, the Shannon entropy, and min-entropy, respectively, while for α = 2 the collision entropy is obtained. Therefore, from Theorem 7, we have the following.

Corollary 4.

For the special cases of $\alpha = 0,1,2,+\infty$ , the Rényi entropy of the elliptically contoured r.v. $X_{\gamma } \sim \mathcal{N}_{\gamma }(\mu,\Sigma )$ is reduced to

$$\displaystyle{ \mathrm{R}_{\alpha }(X_{\gamma }) = \left \{\!\begin{array}{llr} + \infty, &\;\;\text{for}\;\;\alpha = 0, & (\text{Hartley entropy}) \\ p\tfrac{\gamma -1} {\gamma } -\log \max f_{X_{\gamma }}, &\;\;\text{for}\;\;\alpha = 1, &(\text{Shannon entropy}) \\ p\frac{\gamma -1} {\gamma } \log 2 -\log \max f_{X_{\gamma }},&\;\;\text{for}\;\;\alpha = 2, & (\text{collision entropy}) \\ -\log \max f_{X_{\gamma }}, &\;\;\text{for}\;\;\alpha = +\infty,& (\text{min-entropy}) \end{array} \right. }$$

(70)

where $\max f_{X_{\gamma }} = C_{\gamma }^{p}(\Sigma )$ .

The Rényi entropy R _α (X _γ ), as in (69), is a decreasing function of parameter $\alpha \in \mathbb{R}_{+}^{{\ast}}\setminus \{1\}$, and hence

$$\displaystyle{\mathrm{R}_{+\infty }(X_{\gamma }) <\mathrm{ R}_{2}(X_{\gamma }) <\mathrm{ R}_{1}(X_{\gamma }) <\mathrm{ R}_{0}(X_{\gamma }),\quad \gamma \in \mathbb{R}\setminus [0,\,1],}$$

while

$$\displaystyle{\mathop{\min }\limits_{0 <\alpha \neq 1}\{\mathrm{R}_{\alpha }(X_{\gamma })\} =\mathrm{ R}_{+\infty }(X_{\gamma }) = -\log \max f_{X_{\gamma }} = -\log C_{\gamma }^{p}(\Sigma ).}$$

Corollary 5.

The Rényi entropy R_α of the multivariate and elliptically contoured Uniform random variable $X \sim \mathcal{U}(\mu,\Sigma )$ is α-invariant, as R_α (X) equals to the logarithm of the volume $\omega (\mathbb{E}_{\theta })$ of the (p − 1)-ellipsoid $\mathbb{E}_{\theta }:\; Q_{\theta }(x) = 1$ , $x \in \mathbb{R}^{p}$ , in which the p.d.f. of the elliptically contoured Uniform r.v. X is actually defined, i.e.

$$\displaystyle{ \mathrm{R}_{\alpha }(X) =\log \omega (\mathbb{E}_{\theta }) =\log \dfrac{\pi ^{p/2}\vert \det \Sigma \vert ^{-1/2}} {\Gamma (\frac{p} {2} + 1)},\quad \alpha \in \mathbb{R}_{+}^{{\ast}}\setminus \{1\}, }$$

(71)

while for the univariate case of $X \sim \mathcal{U}(a,b)$ it is reduced to

$$\displaystyle{ \mathrm{R}_{\alpha }(X) =\log (b - a),\quad \alpha \in \mathbb{R}_{+}^{{\ast}}\setminus \{1\}. }$$

Proof.

Recall (29) and let $X_{\gamma } \sim \mathcal{N}_{\gamma }(\mu,\Sigma )$. Then, the Rényi entropy of the uniformly r.v. X can be considered as $\mathrm{R}_{\alpha }(X):=\lim _{\gamma \rightarrow 1^{+}}\mathrm{R}_{\alpha }(X_{\gamma })$ and therefore, from (69), we obtain (71). ⊓⊔

Notice, from the above Corollary 5, that the Hartley, Shannon, collision, and min-entropy of a multivariate uniformly distributed r.v. coincide with $\log \omega (\mathbb{E}_{\theta })$.

Corollary 6.

For the multivariate Laplace random variable $X \sim \mathcal{L}(\mu,\Sigma )$ , the Rényi entropy is given by

$$\displaystyle{ \mathrm{R}_{\alpha }(X) = p \tfrac{\log \alpha }{\alpha -1} + L(S), }$$

(72)

and the Hartley, Shannon, collision, and the min-entropy are then given by

$$\displaystyle{ \mathrm{R}_{\alpha }(X) = \left \{\!\begin{array}{llr} + \infty, &\;\;\text{for}\;\;\alpha = 0, & (\text{Hartley entropy}) \\ p + L(\Sigma ), &\;\;\text{for}\;\;\alpha = 1, &(\text{Shannon entropy}) \\ p\log 2 + L(\Sigma ),&\;\;\text{for}\;\;\alpha = 2, & (\text{collision entropy}) \\ L(\Sigma ), &\;\;\text{for}\;\;\alpha = +\infty,& (\text{min-entropy}) \end{array} \right. }$$

(73)

where $L(\Sigma ):=\log \{ p!\pi ^{p/2}\vert \det \Sigma \vert ^{1/2}\Gamma (\frac{p} {2} + 1)^{-1}\}$ .

Proof.

Recall (29) and let $X_{\gamma } \sim \mathcal{N}_{\gamma }(\mu,\Sigma )$. Then, the Rényi entropy of the Laplace r.v. X can be considered as $\mathrm{R}_{\alpha }(X):=\lim _{\gamma \rightarrow \pm \infty }\mathrm{R}_{\alpha }(X_{\gamma })$ and therefore, from (69), we obtain (72), while through (70), relation (73) is finally derived. ⊓⊔

Relations (71) and (72) below provide a general compact form of Rényi entropy R _α (for the Uniform and Laplace r.v.) and can be compared with the α-Shannon entropy H _α (for the such r.v.), as in (38).

3.4 Generalized Fisher’s Entropy Type Information

As far as the generalized Fisher’s entropy type information measure J _α (X _γ ) is concerned, for the multivariate and spherically contoured r.v. $X_{\gamma } \sim \mathcal{N}_{\gamma }^{p}(\mu,\sigma ^{2}\mathbb{I}_{p})$, it holds, [18],

$$\displaystyle{ \mathrm{J}_{\alpha }(X_{\gamma }) = ( \tfrac{\gamma }{\gamma -1})^{\frac{\alpha }{\gamma } }\frac{\Gamma \left (\frac{\alpha +p(\gamma -1)} {\gamma } \right )} {\sigma ^{\alpha }\Gamma \left (p\frac{\gamma -1} {\gamma } \right )}. }$$

(74)

More general, the following holds [19].

Theorem 8.

The generalized Fisher’s entropy type information J_α of a γ-order normally distributed r.v. $X_{\gamma } \sim \mathcal{N}_{\gamma }^{p}(\mu,\Sigma )$ , where $\Sigma$ is a definite positive real matrix consisted of orthogonal vectors (matrix columns) with the same norm, is given by

$$\displaystyle{ \mathrm{J}_{\alpha }(X_{\gamma }) = \frac{( \frac{\gamma }{\gamma -1})^{\frac{\alpha }{\gamma } }\Gamma \left (\frac{\alpha +p(\gamma -1)} {\gamma } \right )} {\vert \det \Sigma \vert ^{ \frac{\alpha }{ 2p} }\Gamma (p\frac{\gamma -1} {\gamma } )}. }$$

(75)

Therefore, for the spherically contoured case, (74) holds indeed, through Theorem 8.

Corollary 7.

The generalized Fisher’s information J_α of a spherically contoured r.v. $X_{\gamma } \sim \mathcal{N}_{\gamma }^{p}(\mu,\sigma ^{2}\mathbb{I}_{p})$ , with $\alpha /\gamma \in \mathbb{N}^{{\ast}}$ , is reduced to

$$\displaystyle{ \mathrm{\mathbf{J}}_{\alpha }(X_{\gamma }) =\sigma ^{-\alpha }(\gamma -1)^{-\alpha \gamma }\prod \limits _{ k=1}^{\alpha /\gamma }\{\alpha - p + (p - k)\gamma \},\quad \alpha,\gamma> 1. }$$

Proof.

From (74) and the gamma function additive identity, i.e. $\Gamma (x + 1) = x\Gamma (x)$, $x \in \mathbb{R}_{+}^{{\ast}}$, relation (7) holds

3.5 Kullback–Leibler Divergence

As far as the information “discrimination” or “distance” is concerned between two $\mathcal{N}_{\gamma }$ r.v., the Kullback–Leibler (K–L) measure of information divergence (also known as relative entropy) is evaluated. Recall the K–L divergence D_KL(X, Y ) defined in Sect. 1. Specifically, for two multivariate γ-order normally distributed r.v. with the same mean and shape, i.e. $X_{i} \in \mathcal{N}_{\gamma }(\mu _{i},\sigma _{i}^{2}\mathbb{I}_{p})$, i = 1, 2, with μ ₁ = μ ₂, the K–L divergence of X ₁ over X ₂ is given by, [16],

$$\displaystyle{ \mathrm{D_{KL}}(X_{1},X_{2}) = p\log \tfrac{\sigma _{2}} {\sigma _{1}} - p(\tfrac{\gamma -1} {\gamma } )\left [1 - (\tfrac{\sigma _{1}} {\sigma _{2}} )^{ \frac{\gamma }{\gamma -1} }\right ], }$$

(76)

while for μ ₁ ≠ μ ₂ and γ = 2,

$$\displaystyle{\mathrm{D_{KL}}(X_{1},X_{2}) = \tfrac{p} {2}\left [\left (\log \tfrac{\sigma _{2}^{2}} {\sigma _{1}^{2}} \right ) - 1 + \tfrac{\sigma _{1}^{2}} {\sigma _{2}^{2}} + \tfrac{\|\mu _{1}-\mu _{0}\|^{2}} {p\sigma _{2}^{2}} \right ].}$$

Moreover, from (76), the K–L divergence between two uniformly distributed r.v. $U_{1},U_{2} \in \mathcal{U}^{p}(\mu,\sigma _{i}^{2}\mathbb{I}_{p})$, i = 1, 2, is given by,

$$\displaystyle{\mathrm{D_{KL}}(U_{1},U_{2}) =\mathop{\lim }\limits_{\gamma \rightarrow 1^{+}}\mathrm{D_{ KL}}(X_{1},X_{2}) = \left \{\!\begin{array}{lr} p\log \tfrac{\sigma _{2}} {\sigma _{1}}, &\sigma _{1}>\sigma _{2}, \\ + \infty,&\sigma _{1} <\sigma _{2}, \end{array} \right.}$$

while the K–L divergence between two Laplace distributed r.v. $L_{1},L_{2} \in \mathcal{L}^{p}(\mu,\sigma _{i}^{2}\mathbb{I}_{p})$, i = 1, 2, is given by

$$\displaystyle{\mathrm{D_{KL}}(L_{1},L_{2}) =\mathop{\lim }\limits_{\gamma \rightarrow + \pm \infty }\mathrm{D_{KL}}(X_{1},X_{2}) = p\left (\log \tfrac{\sigma _{2}} {\sigma _{1}} - 1 + \tfrac{\sigma _{1}} {\sigma _{2}} \right ).}$$

We have already discussed all the well-known entropy type measures and new generalized results have been obtained. We now approach the notion of complexity from a new generalized point of view, as discussed below.

4 Complexity and the Generalized Gaussian

The entropy of a continuous system is defined over a random variable X as the expected value of the information content, say I(X), of X, i.e. H(X): = E[I(X)]. For the usual Shannon entropy (or differential entropy) case, the information content $I(X) =\log f_{X}$ is adopted, where f _X is the p.d.f. of the r.v. X.

In principle, the entropy can be considered as a measure of the “disorder” of a system. However in applied sciences, the normalized Shannon entropy $\mathrm{H}^{{\ast}} =\mathrm{ H}/\max \mathrm{H}$ is usually considered as a measure of “disorder” because H^∗ is independent of all various states that the system can adopt, [24]. Respectively, the quantity $\varOmega = 1 -\mathrm{ H}^{{\ast}}$ is considered as a measure of “order”. For the estimation of “disorder,” information measures play a fundamental role on describing the inner-state or the complexity of a system, see [27] among others. We believe that concepts are useful in Cryptography.

A quantitative measure of complexity with the simplest possible expression is considered to be the “order–disorder” product K^ω, h given by

$$\displaystyle{ \mathrm{K}^{\omega,h} =\varOmega ^{\omega }\mathrm{H}^{{\ast}h} =\mathrm{ H}^{{\ast}h}(1 -\mathrm{ H}^{{\ast}})^{\omega } =\varOmega ^{\omega }(1-\varOmega )^{h},\quad \omega,h \in \mathbb{R}_{ +}. }$$

(77)

This is usually called as simple complexity with “order” power ω and “disorder” power h.

The above measure K^ω, h, ω, h ≥ 1, satisfies the three basic rules of complexity measures. Specifically, we distinguish the following cases:

Rule 1. :

Vanishing “order” power, ω = 0. Then K^ω, h = (H^∗)^h, i.e. K^0, h is an increasing function of the system’s “disorder” H.

Rule 2. :

Non-vanishing “order” and “disorder” powers, ω, h > 0. Then for “absolute-ordered” or “absolute-disordered” systems the complexity vanishes. Moreover, it adopts a maximum value (with respect to H^∗) for an intermediate state $\mathrm{H}^{{\ast}} = h/(\omega +h)$ or $\Omega =\omega /(\omega +h)$, with $\max _{\mathrm{H}^{{\ast}}}\{\mathrm{K}^{\omega,h}\} = h^{h}\omega ^{\omega }(\omega +h)^{\omega +h}$. In other words the “absolute-complexity” systems are such that their “order” and “disorder” are “balanced,” hence $\mathrm{H}^{{\ast}} = h/(\omega +h)$.

Rule 3. :

Vanishing “disorder” power, h = 0. Then K^ω, h = Ω ^ω, i.e. K^ω, 0 is an increasing function of the system’s “order” Ω.

The Shiner–Davison–Landsberg (SDL) measure of complexity K_SDL is an important measure in bio-sciences that satisfies the second rule as it is defined by, [29],

$$\displaystyle{ \mathrm{K_{SDL}} = 4\mathrm{K}^{1,1} = 4\mathrm{H}^{{\ast}}(1 -\mathrm{ H}^{{\ast}}) = 4\varOmega (1-\varOmega ). }$$

(78)

It is important to mention that all the systems with the same degree of “disorder” have the same degree of SDL complexity. Moreover, SDL complexity vanishes for all systems in an equilibrium state and therefore it cannot distinguish between systems with major structural and organizing differences, see also [7, 27].

Now, consider the evaluation of the SDL complexity in a system where its various states are described by a wide range of distributions, such as the univariate γ-ordered Normal distributions. In such a case we may consider the normalized Shannon entropy $\mathrm{H}^{{\ast}}(X_{\gamma }):=\mathrm{ H}(X_{\gamma })/\mathrm{H}(Z)$ where $X_{\gamma } \sim \mathcal{N}_{\gamma }(\mu,\sigma ^{2})$ as in (22), and we let $Z \sim \mathcal{N}(\mu,\sigma _{Z}^{2})$ with $\sigma _{Z}^{2} =\mathop{ \mathrm{Var}}\nolimits Z$. That is, we adopt for the maximum entropy, with respect to $X_{\gamma } \sim \mathcal{N}_{\gamma }$, the Shannon entropy of a normally distributed Z with its variance $\sigma _{Z}^{2}$ being equal to the variance of X _γ . This is due to the fact that the Normal distribution (included also into the $\mathcal{N}_{\gamma }(\mu,\sigma ^{2})$ family for γ = 2) provides the maximum entropy of every distribution (here $\mathcal{N}_{\gamma }$) for equally given variances, i.e. $\sigma _{Z}^{2} =\mathop{ \mathrm{Var}}\nolimits Z =\mathop{ \mathrm{Var}}\nolimits X_{\gamma }$. Hence, $\max _{\gamma }\{\mathrm{H}(X_{\gamma })\} =\mathrm{ H}(X_{2}) =\mathrm{ H}(Z)$.

The use of the above normalized Shannon entropy defines a complexity measure that “characterizes” the family of the γ-ordered Normal distributions as it is obtained in the following Theorem, [17].

Theorem 9.

The SDL complexity of a random variable $X_{\gamma } \sim \mathcal{N}_{\gamma }(\mu,\sigma ^{2})$ is given by

$$\displaystyle{ \mathrm{K_{SDL}}(X_{\gamma }) = 8 \frac{\log \left \{2\sigma ( \frac{\gamma e} {\gamma -1})^{\frac{\gamma -1} {\gamma } }\Gamma (\frac{\gamma -1} {\gamma } + 1)\right \}} {\log ^{2}\left \{2\pi e\sigma ^{2}( \tfrac{\gamma }{\gamma -1})^{2\frac{\gamma -1} {\gamma } } \frac{\Gamma (3\frac{\gamma -1} {\gamma } )} {\Gamma (\frac{\gamma -1} {\gamma } )} \right \}}\log \left \{ \tfrac{\pi } {2}e^{\frac{2-\gamma } {\gamma } }( \tfrac{\gamma }{\gamma -1})^{2}\frac{\Gamma (3\frac{\gamma -1} {\gamma } )} {\Gamma ^{3}(\frac{\gamma -1} {\gamma } )}\right \}, }$$

which vanishes (giving the “absolute-order” or “absolute-disorder” state of a system) for: (a) the normally distributed r.v. X ₂ , and (b) for scale parameters

$$\displaystyle{ \sigma = \tfrac{1} {2}(\tfrac{\gamma -1} {\gamma e} )^{\frac{\gamma -1} {\gamma } }[\Gamma (\tfrac{\gamma -1} {\gamma } + 1)]^{-1}. }$$

Figure 4 illustrates the behavior of the SDL complexity K_SDL(X _γ ) with $X_{\gamma } \sim \mathcal{N}_{\gamma }(\mu,\sigma ^{2})$ for various scale parameters $\sigma ^{2}$. Notice that, for $\sigma ^{2}> 1$, depicted in sub-figure (a), the negative-ordered Normals close to 0, i.e. close to the degenerate Dirac distribution (recall Theorem 3), provide the “absolute-complexity” state, i.e. K_SDL(X _γ ) = 1, of a system, in which their different states described from the γ-ordered Normal distributions. The sub-figure (a) is obtained for $\sigma ^{2} \geq 1$, while (b) for $\sigma ^{2} <1$. Notice, in sub-figure (b), that among all the positive-ordered random variables $X_{\gamma } \sim \mathcal{N}_{\gamma \geq 0}(\mu,\sigma ^{2})$ with $\sigma ^{2} <1$, the uniformly distributed ones γ = 1 provide the maximum (but not the absolute) 2-SDL complexity measure.

Fig. 4

Graphs of the SDL complexity KSDL(X _γ ) along γ, with $X_{\gamma } \sim \mathcal{N}_{\gamma }(\mu,\sigma ^{2})$, for various $\sigma ^{2}$ values. (a) corresponds to $\sigma \geq 1$ while (b) to $\sigma ^{2} <1$ values

5 Discussion

In this paper we have provided a concise presentation of a class of generalized Fisher’s entropy type information measures, as well as entropy measures, that extend the usual Shannon entropy, such as the α-Shannon entropy and the Rényi entropy. A number of results were stated and proved, and the well-known results were just special cases. These extensions were based on an extra parameter. In the generalized Normal distribution the extra shape parameter γ adjusts fat, or not, tails, while the extra parameter α of the generalized Fisher’s entropy type information, or of the generalized entropy, adjusts “optimistic” information measures to better levels. Under this line of thought we approached other entropy type measures as special cases. We believe that these generalizations need further investigation using real data in Cryptography and in other fields. Therefore, these measures were applied on γ-order normally distributed random variables (an exponential-power generalization of the usual Normal distribution) and discussed. A study on a certain form of complexity is also discussed for such random variables.