Abstract
We develop new discrete uncertainty principles in terms of numerical sparsity, which is a continuous proxy for the 0-norm. Unlike traditional sparsity, the continuity of numerical sparsity naturally accommodates functions which are nearly sparse. After studying these principles and the functions that achieve exact or near equality in them, we identify certain consequences in a number of sparse signal processing applications.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
Uncertainty principles have maintained a significant role in both science and engineering for most of the past century. In 1927, the concept was introduced by Werner Heisenberg in the context of quantum mechanics [24], in which a particle’s position and momentum are represented by wavefunctions \(f,g\in L^2(\mathbb {R})\), and g happens to be the Fourier transform of f. Measuring the position or momentum of a particle amounts to drawing a random variable whose probability density function is a normalized version of \(|f|^2\) or \(|g|^2\), respectively. Heisenberg’s uncertainty principle postulates a fundamental limit on the precision with which one can measure both position and momentum; in particular, the variance of the position measurement is small only if the momentum measurement exhibits large variance. From a mathematical perspective, this physical principle can be viewed as an instance of a much broader meta-theorem in harmonic analysis:
A nonzero function and its Fourier transform cannot be simultaneously localized.
Heisenberg’s uncertainty principle provides a lower bound on the product of the variances of the probability density functions corresponding to f and \(\hat{f}\). In the time since, various methods have emerged for quantifying localization. For example, instead of variance, one might consider entropy [6], the size of the density’s support [2], or how rapidly it decays [23]. Furthermore, the tradeoff in localization need not be represented by a product—as we will see, it is sometimes more telling to consider a sum.
Beyond physics, the impossibility of simultaneous localization has had significant consequences in signal processing. For example, when working with the short-time Fourier transform, one is forced to choose between temporal and frequency resolution. More recently, the emergence of digital signal processing has prompted the investigation of uncertainty principles underlying the discrete Fourier transform, notably by Donoho and Stark [17], Tao [42], and Tropp [45]. Associated with this line of work is the uniform uncertainty principle of Candès and Tao [12], which played a key role in the development of compressed sensing. The present paper continues this investigation of discrete uncertainty principles with an eye on applications in sparse signal processing.
1.1 Background and Overview
For any finite abelian group G, let \(\ell (G)\) denote the set of functions \(x:G\rightarrow \mathbb {C}\), and \(\widehat{G}\subseteq \ell (G)\) the group of characters over G. Then taking inner products with these characters and normalizing leads to the (unitary) Fourier transform \(F:\ell (G)\rightarrow \ell (\widehat{G})\), namely
The reader who is unfamiliar with Fourier analysis over finite abelian groups is invited to learn more in [43]. In the case where \(G=\mathbb {Z}/n\mathbb {Z}\) (which we denote by \(\mathbb {Z}_n\) in the sequel), the above definition coincides with the familiar discrete Fourier transform after one identifies characters with their frequencies. The following theorem provides two uncertainty principles in terms of the so-called 0-norm \(\Vert \cdot \Vert _0\), defined to be number of nonzero entries in the argument.
Theorem 1
([17, Theorem 1], [42, Theorem 1.1]) Let G be a finite abelian group, and let \(F:\ell (G)\rightarrow \ell (\widehat{G})\) denote the corresponding Fourier transform. Then
Furthermore, if |G| is prime, then
Proof
Sketch For (1), apply the fact that the \(\ell _1/\ell _\infty \)-induced norm of F is given by \(\Vert F\Vert _{1\rightarrow \infty }=1/\sqrt{|G|}\), along with Cauchy–Schwarz and Parseval’s identity:
where the last step bounds a sum in terms of its largest summand. Rearranging gives the result.
For (2), suppose otherwise that there exists \(x\ne 0\) which violates the claimed inequality. Denote \(\mathcal {J}={\text {supp}}(x)\) and pick some \(\mathcal {I}\subseteq \widehat{G}{\setminus }{\text {supp}}(Fx)\) with \(|\mathcal {I}|=|\mathcal {J}|\). Then \(0=(Fx)_\mathcal {I}=F_{\mathcal {I}\mathcal {J}}x_\mathcal {J}\). Since the submatrix \(F_{\mathcal {I}\mathcal {J}}\) is necessarily invertible by a theorem of Chebotarëv [39], we conclude that \(x_\mathcal {J}=0\), a contradiction. \(\square \)
We note that the additive uncertainty principle above is much stronger than its multiplicative counterpart. Indeed, with the help of the arithmetic mean–geometric mean inequality, (1) immediately implies
which is sharp when \(G=\mathbb {Z}_n\) and n is a perfect square (simply take x to be a Dirac comb, specifically, the indicator function \(1_K\) of the subgroup K of size \(\sqrt{n}\)). More generally, if n is not prime, then \(n=ab\) with integers \(a,b\in [2,n/2]\), and so \(a+b\le n/2+2<n+1\); as such, taking x to be an indicator function of the subgroup of size a (whose Fourier transform necessarily has 0-norm b) will violate (2). Overall, the hypothesis that |G| is prime cannot be weakened. Still, something can be said if one slightly strengthens the hypothesis on x. For example, Theorem A in [44] gives that for every \(S\subseteq G\),
for almost every \(x\in \ell (G)\) supported on S. This suggests that extreme functions like the Dirac comb are atypical, i.e., (3) is “barely sharp”.
One could analogously argue that, in some sense, (2) is “barely true” when |G| is prime. For an illustration, Fig. 1 depicts a discrete version of the Gaussian function, which is constructed by first periodizing the function \(f(t)=e^{-n\pi t^2}\) over the real line in order to have unit period, and then sampling this periodized function at multiples of 1 / n. As we verify in Sect. 3.2, the resulting function \(x\in \ell (\mathbb {Z}_n)\) satisfies \(Fx=x\), analogous to the fact that a Gaussian function in \(L^2(\mathbb {R})\) with the proper width is fixed by the Fourier transform. Given its resemblance to the fast-decaying Gaussian function over \(\mathbb {R}\), it comes as no surprise that many entries of this function are nearly zero. In the depicted case where \(n=211\) (which is prime), only 99 entries of this function manage to be larger than machine precision, and so from a numerical perspective, this function appears to contradict Theorem 1: \(99+99=198<212=211+1\).
To help resolve this discrepancy, we consider a numerical version of traditional sparsity which is aptly named numerical sparsity:
See Fig. 2 for an illustration. This ratio appeared as early as 1978 in the context of geophysics [20]. More recently, it has been used as a proxy for sparsity in various signal processing applications [14, 25, 31, 36, 40]. The numerical rank of a matrix is analogously defined as the square ratio of the nuclear and Frobenius norms, and has been used, for example, in Alon’s work on extremal combinatorics [1]. We note that numerical sparsity is invariant under nonzero scaling, much like traditional sparsity. In addition, one bounds the other:
To see this, apply Cauchy–Schwarz to get
where |x| denotes the entrywise absolute value of x. Rearranging then gives (4). For this paper, the most useful feature of numerical sparsity is its continuity, as this will prevent near-counterexamples like the one depicted in Fig. 1. What follows is our main result, which leverages numerical sparsity to provide uncertainty principles that are analogous to those in Theorem 1:
Theorem 2
(Main resultFootnote 1) Let U be an \(n\times n\) unitary matrix. Then
where \(\Vert \cdot \Vert _{1\rightarrow \infty }\) denotes the induced matrix norm. Furthermore, there exists a universal constant \(c>0\) such that if U is drawn uniformly from the unitary group \({\text {U}}(n)\), then with probability \(1-e^{-\Omega (n)}\),
Perhaps the most glaring difference between Theorems 1 and 2 is our replacement of the Fourier transform with an arbitrary unitary matrix. Such generalizations have appeared in the quantum physics literature (for example, see [29]), as well as in the sparse signal processing literature [15, 16, 21, 40, 44, 45]. Our multiplicative uncertainty principle still applies when \(U=F\), in which case \(\Vert U\Vert _{1\rightarrow \infty }=1/\sqrt{n}\). Considering (4), the uncertainty principle in this case immediately implies the analogous principle in Theorem 1. Furthermore, the proof is rather straightforward: Apply Hölder’s inequality to get
By contrast, the proof of our additive uncertainty principle is not straightforward, and it does not hold if we replace U with F. Indeed, as we show in Sect. 3.2, the discrete Gaussian function depicted in Fig. 1 has numerical sparsity \(O(\sqrt{n})\), thereby violating (6); recall that the same function is a near-counterexample of the analogous principle in Theorem 1. Interestingly, our uncertainty principle establishes that the Fourier transform is rare in that the vast majority of unitary matrices offer much more uncertainty in the worst case. This naturally leads to the following question:
Problem 3
For each n, what is the largest \(c=c(n)\) for which there exists a unitary matrix U that satisfies \({\text {ns}}(x)+{\text {ns}}(Ux)\ge cn\) for every \(x\in \mathbb {C}^n{\setminus }\{0\}\)?
Letting \(x=e_1\) gives \({\text {ns}}(x)+{\text {ns}}(Ux)\le 1+\Vert Ux\Vert _0\le n+1\), and so \(c(n)\le 1+o(1)\); a bit more work produces a strict inequality \(c(n)<1+1/n\) for \(n\ge 4\). Also, our proof of the uncertainty principle implies \(\liminf _{n\rightarrow \infty }c(n)\ge 1/540{,}000\).
1.2 Outline
The primary focus of this paper is Theorem 2. Having already proved the multiplicative uncertainty principle in (7), it remains to prove the additive counterpart, which we do in the following section. Next, Sect. 3 considers functions which achieve either exact or near equality in (5) when U is the discrete Fourier transform. Surprisingly, exact equality occurs in (5) precisely when it occurs in (1). We also show that the discrete Gaussian depicted in Fig. 1 achieves near equality in (5). We conclude in Sect. 4 by studying a few applications, specifically, sparse signal demixing, compressed sensing with partial Fourier operators, and the fast detection of sparse signals.
2 Proof of Additive Uncertainty Principle
In this section, we prove the additive uncertainty principle in Theorem 2. The following provides a more explicit statement of the principle we prove:
Theorem 4
Draw U uniformly from the unitary group \({\text {U}}(n)\). Then with probability \(\ge 1-8e^{-n/4096}\),
For the record, we did not attempt to optimize the constants. Our proof of this theorem makes use of several ideas from the compressed sensing literature:
Definition 5
Take any \(m\times n\) matrix \(\Phi =[\varphi _1\cdots \varphi _n]\).
-
(a)
We say \(\Phi \) exhibits \((k,\theta )\)-restricted orthogonality if
$$\begin{aligned} |\langle \Phi x,\Phi y\rangle |\le \theta \Vert x\Vert _2\Vert y\Vert _2 \end{aligned}$$for every \(x,y\in \mathbb {C}^n\) with \(\Vert x\Vert _0,\Vert y\Vert _0\le k\) and disjoint support.
-
(b)
We say \(\Phi \) satisfies the \((k,\delta )\)-restricted isometry property if
$$\begin{aligned} (1-\delta )\Vert x\Vert _2^2\le \Vert \Phi x\Vert _2^2\le (1+\delta )\Vert x\Vert _2^2 \end{aligned}$$for every \(x\in \mathbb {C}^n\) with \(\Vert x\Vert _0\le k\).
-
(c)
We say \(\Phi \) satisfies the (k, c)-width property if
$$\begin{aligned} \Vert x\Vert _2\le \frac{c}{\sqrt{k}}\Vert x\Vert _1 \end{aligned}$$for every x in the nullspace of \(\Phi \).
The restricted isometry property is a now-standard sufficient condition for uniformly stable and robust reconstruction from compressed sensing measurements (for example, see [11]). As the following statement reveals, restricted orthogonality implies the restricted isometry property:
Lemma 6
[4, Lemma 11] If a matrix satisfies \((k,\theta )\)-restricted orthogonality and its columns have unit norm, then it also satisfies the \((k,\delta )\)-restricted isometry property with \(\delta =2\theta \).
To prove Theorem 4, we will actually make use of the width property, which was introduced by Kashin and Temlyakov [27] to characterize uniformly stable \(\ell _1\) reconstruction for compressed sensing. Luckily, the restricted isometry property implies the width property:
Lemma 7
([9, Theorem 11], cf. [27]) If a matrix satisfies the \((k,\delta )\)-restricted isometry property for some positive integer k and \(\delta <1/3\), then it also satisfies the (k, 3)-width property.
What follows is a stepping-stone result that we will use to prove Theorem 4, but it is also of independent interest:
Theorem 8
Draw U uniformly from the unitary group \({\text {U}}(n)\). Then [I U] satisfies the \((k,\delta )\)-restricted isometry property with probability \(\ge 1-8e^{-\delta ^2n/256}\) provided \(\delta <1\) and
This is perhaps not surprising, considering various choices of structured random matrices are known to form restricted isometries with high probability [3, 8, 12, 28, 34, 35, 37]. To prove Theorem 8, we show that the structured matrix enjoys restricted orthogonality with high probability, and then appeal to Lemma 6. Before proving this result, we first motivate it by proving the desired uncertainty principle:
Proof of Theorem 4
Take \(k=\lfloor n/60{,}000\rfloor \) and \(\delta =1/4\). We will show \({\text {ns}}(x)+{\text {ns}}(Ux)\ge k/9\) for every nonzero \(x\in \mathbb {C}^n\). If \(k=0\), the result is immediate, and so \(n\ge 60{,}000\) without loss of generality. In this regime, we have \(k\in [n/120{,}000{,}n/60{,}000]\), and so
Theorem 8 and Lemma 7 then give that [I U] satisfies the (k, 3)-width property with probability \(\ge 1-8e^{-n/4096}\). Observe that \(z=[Ux;-x]\) resides in the nullspace of [I U] regardless of \(x\in \mathbb {C}^n\). In the case where x (and therefore z) is nonzero, the width property and the arithmetic mean–geometric mean inequality together give
\(\square \)
Proof of Theorem 8
Take \([I~U]=[\varphi _1\cdots \varphi _{2n}]\), and let k be the largest integer satisfying (8). We will demonstrate that [I U] satisfies the \((k,\delta )\)-restricted isometry property, which will then imply the \((k',\delta )\)-restricted isometry property for all \(k'<k+1\), and therefore all k satisfying (8). To this end, define the random quantities
We first claim that \(\theta ^\star (U)\le \theta (U)\). To see this, for any x, y satisfying the constraints in \(\theta ^\star (U)\), decompose \(x=x_1+x_2\) so that \(x_1\) and \(x_2\) are supported in [n] and \([n]^c\), respectively, and similarly \(y=y_1+y_2\). For notational convenience, let S denote the set of all 4-tuples (a, b, c, d) of k-sparse vectors in \(\mathbb {C}^{2n}\) such that a and b are disjointly supported in [n], while c and d are disjointly supported in \([n]^c\). Then \((x_1,y_1,x_2,y_2)\in S\). Since \({\text {supp}}(x)\) and \({\text {supp}}(y)\) are disjoint, and since I and U each have orthogonal columns, we have
As such, the triangle inequality gives
where the last step follows from squaring and applying the arithmetic mean–geometric mean inequality:
At this point, we seek to bound the probability that \(\theta (U)\) is large. First, we observe an equivalent expression:
To estimate the desired probability, we will pass to an \(\epsilon \)-net \(\mathcal {N}_\epsilon \) of k-sparse vectors with unit 2-norm. A standard volume-comparison argument gives that the unit sphere in \(\mathbb {R}^{m}\) enjoys an \(\epsilon \)-net of size \(\le (1+2/\epsilon )^m\) (see [47, Lemma 5.2]). As such, for each choice of k coordinates, we can cover the corresponding copy of the unit sphere in \(\mathbb {C}^k=\mathbb {R}^{2k}\) with \(\le (1+2/\epsilon )^{2k}\) points, and unioning these produces an \(\epsilon \)-net of size
To apply this \(\epsilon \)-net, we note that \(\Vert x-x'\Vert _2,\Vert y-y'\Vert _2\le \epsilon \) and \(\Vert x'\Vert _2=\Vert y'\Vert _2=1\) together imply
where the last step assumes \(\epsilon \le 1\). As such, the union bound gives
where the last step uses the fact that the distribution of U is invariant under left- and right-multiplication by any deterministic unitary matrix (e.g., unitary matrices that send \(e_1\) to x and y to \(e_1\), respectively). It remains to prove tail bounds on \(U_{11}:=\langle e_1,Ue_1\rangle \). First, we apply the union bound to get
where the last step uses the fact that \({\text {Re}}(U_{11})\) has even distribution. Next, we observe that \({\text {Re}}(U_{11})\) has the same distribution as \(g/\sqrt{h}\), where g has standard normal distribution and h has chi-squared distribution with 2n degrees of freedom. Indeed, this can be seen from one method of constructing the matrix U: Start with an \(n\times n\) matrix G with iid \(N(0,1)+iN(0,1)\) complex Gaussian entries and apply Gram–Schmidt to the columns; the first column of U is then the first column of G divided by its norm \(\sqrt{h}\). Let \(s>0\) be arbitrary (to be selected later). Then \(g/\sqrt{h}>u/\sqrt{2}\) implies that either \(g>\sqrt{s}u/\sqrt{2}\) or \(h<s\). As such, the union bound implies
For the first term, Proposition 7.5 in [19] gives
For the second term, Lemma 1 in [30] gives \({\text {Pr}}(h<2n-\sqrt{8nx})\le e^{-x}\) for any \(x>0\). Picking \(x=(2n-s)^2/(8n)\) then gives
We use the estimate \(\left( {\begin{array}{c}n\\ k\end{array}}\right) \le (en/k)^k\) when combining (9)–(13) to get
Notice \(n/k\ge (256/\delta ^2)\log (en/k)\ge 256\) implies that taking \(\epsilon =\sqrt{(k/n)\log (en/k)}\) gives
which can be rearranged to get
As such, we also pick \(s=n\) and \(t=\sqrt{(64k/n)\log (en/k)}\) to get
Since we chose k to be the largest integer satisfying (8), we therefore have \(\theta (U)\le \sqrt{(64k/n)\log (n/k)}\) with probability \(\ge 1-8e^{-\delta ^2n/256}\). Lemma 6 then gives the result. \(\square \)
3 Low Uncertainty with the Discrete Fourier Transform
In this section, we study functions which achieve either exact or near equality in our multiplicative uncertainty principle (6) in the case where the unitary matrix U is the discrete Fourier transform.
3.1 Exact Equality in the Multiplicative Uncertainty Principle
We seek to understand when equality is achieved in (6) in the special case of the discrete Fourier transform. For reference, the analogous result for (1) is already known:
Theorem 9
[17, Theorem 13] Suppose \(x\in \ell (\mathbb {Z}_n)\) satisfies \(\Vert x\Vert _0\Vert Fx\Vert _0=n\). Then x has the form \(x=cT^aM^b1_K\), where \(c\in \mathbb {C}\), K is a subgroup of \(\mathbb {Z}_n\), and \(T,M:\ell (\mathbb {Z}_n)\rightarrow \ell (\mathbb {Z}_n)\) are translation and modulation operators defined by
Here, i denotes the imaginary unit \(\sqrt{-1}\).
In words, equality is achieved in (1) by indicator functions of subgroups, namely, the so-called Dirac combs (as well as their scalar multiples, translations, modulations). We seek an analogous characterization for our uncertainty principle (6). Surprisingly, the characterization is identical:
Theorem 10
Suppose \(x\in \ell (\mathbb {Z}_n)\). Then \({\text {ns}}(x){\text {ns}}(Fx)=n\) if and only if \(\Vert x\Vert _0\Vert Fx\Vert _0=n\).
Proof
(\(\Leftarrow \)) This follows directly from (4), along with Theorems 1 and 2.
(\(\Rightarrow \)) It suffices to show that \({\text {ns}}(x)=\Vert x\Vert _0\) and \({\text {ns}}(Fx)=\Vert Fx\Vert _0\). Note that both F and \(F^{-1}\) are unitary operators and \(\Vert F\Vert _{1\rightarrow \infty }^2=\Vert F^{-1}\Vert _{1\rightarrow \infty }^2=1/n\). By assumption, taking \(y:=Fx\) then gives
We will use the fact that x and y each achieve equality in the first part of Theorem 2 with \(U=F\) and \(U=F^{-1}\), respectively. Notice from the proof (7) that equality occurs only if x and y satisfy equality in Hölder’s inequality, that is,
To achieve the first equality in (14),
This implies that \(|x[j]|=\max _k|x[k]|\) for every j with \(x[j]\ne 0\). Similarly, in order for the second equality in (14) to hold, \(|y[j]|=\max _k|y[k]|\) for every j with \(y[j]\ne 0\). As such, \(|x|=a1_A\) and \(|y|=b1_B\) for some \(a,b>0\) and \(A,B\subseteq \mathbb {Z}_n\). Then
and similarly, \({\text {ns}}(y)=\Vert y\Vert _0\). \(\square \)
3.2 Near Equality in the Multiplicative Uncertainty Principle
Having established that equality in the new multiplicative uncertainty principle (5) is equivalent to equality in the analogous principle (1), we wish to separate these principles by focusing on near equality. For example, in the case where n is prime, \(\mathbb {Z}_n\) has no nontrivial proper subgroups, and so by Theorem 9, equality is only possible with identity basis elements and complex exponentials. On the other hand, we expect the new principle to accommodate nearly sparse vectors, and so we appeal to the discrete Gaussian depicted in Fig. 1:
Theorem 11
Define \(x\in \ell (\mathbb {Z}_n)\) by
Then \(Fx=x\) and \({\text {ns}}(x){\text {ns}}(Fx)\le (2+o(1))n\).
In words, the discrete Gaussian achieves near equality in the uncertainty principle (5). Moreover, numerical evidence suggests that \({\text {ns}}(x){\text {ns}}(Fx)=(2+o(1))n\), i.e., the 2 is optimal for the discrete Gaussian. Note that this does not depend on whether n is prime or a perfect square. Recall that a function \(f\in C^{\infty }(\mathbb {R})\) is Schwarz if \(\sup _{x\in \mathbb {R}}|x^{\alpha } f^{(\beta )}(x)|<\infty \) for every pair of nonnegative integers \(\alpha \) and \(\beta \). We use this to quickly prove a well-known lemma that will help us prove Theorem 11:
Lemma 12
Suppose \(f\in C^{\infty }(\mathbb {R})\) is Schwarz and construct a discrete function \(x\in \ell (\mathbb {Z}_n)\) by periodizing and sampling f as follows:
Then the discrete Fourier transform of x is determined by \(\hat{f}(\xi ):=\int _{-\infty }^\infty f(t)e^{-2\pi i \xi t}dt\):
Proof
Since f is Schwarz, we may apply the Poisson summation formula:
Next, the geometric sum formula gives
The result then follows from a change of variables. \(\square \)
Proof of Theorem 11
It is straightforward to verify that the function \(f(t)=e^{-n\pi t^2}\) is Schwarz. Note that defining x according to (16) then produces (15). Considering \(\hat{f}(\xi )=n^{-1/2}e^{-\pi \xi ^2/n}\), one may use Lemma 12 to quickly verify that \(Fx=x\). To prove Theorem 11, it then suffices to show that \({\text {ns}}(x)\le (\sqrt{2}+o(1))\sqrt{n}\). We accomplish this by bounding \(\Vert x\Vert _2\) and \(\Vert x\Vert _1\) separately.
To bound \(\Vert x\Vert _2\), we first expand a square to get
Since all of the terms in the sum are nonnegative, we may infer a lower bound by discarding the terms for which \(j''\ne j'\). This yields the following:
where the last inequality follows from an integral comparison. Next, we bound \(\Vert x\Vert _1\) using a similar integral comparison:
Overall, we have
\(\square \)
4 Applications
Having studied the new uncertainty principles in Theorem 2, we now take some time to identify certain consequences in various sparse signal processing applications. In particular, we report consequences in sparse signal demixing, in compressed sensing with partial Fourier operators, and in the fast detection of sparse signals.
4.1 Sparse Signal Demixing
Suppose a signal x is sparse in the Fourier domain and corrupted by noise \(\epsilon \) which is sparse in the time domain (such as speckle). The goal of demixing is to recover the original signal x given the corrupted signal \(z=x+\epsilon \); see [32] for a survey of various related demixing problems. Provided Fx and \(\epsilon \) are sufficiently sparse, it is known that this recovery can be accomplished by solving
where, if successful, the solution \(v^{\star }\) is the column vector obtained by concatenating Fx and \(\epsilon \); see [38] for an early appearance of this sort of approach. To some extent, we know how sparse Fx and \(\epsilon \) must be for this \(\ell _1\) recovery method to succeed. Coherence-based guarantees in [15, 16, 21] show that it suffices for \(v^{\star }\) to be k-sparse with \(k=O(\sqrt{n})\), while restricted isometry–based guarantees [5, 11] allow for \(k=O(n)\) if [I F] is replaced with a random matrix. This disparity is known as the square-root bottleneck [46]. In particular, does [I F] perform similarly to a random matrix, or is the coherence-based sufficient condition on k also necessary?
In the case where n is a perfect square, it is well known that the coherence-based sufficient condition is also necessary. Indeed, let K denote the subgroup of \(\mathbb {Z}_n\) of size \(\sqrt{n}\) and suppose \(x=1_K\) and \(\epsilon =-1_K\). Then \([Fx;\epsilon ]\) is \(2\sqrt{n}\)-sparse, and yet \(z=0\), thereby forcing \(v^\star =0\). On the other hand, if n is prime, then the additive uncertainty principle of Theorem 1 implies that every member of the nullspace of [I F] has at least \(n+1\) nonzero entries, and so \(v^\star \ne 0\) in this setting. Still, considering Fig. 1, one might expect a problem from a stability perspective. In this section, we use numerical sparsity to show that \(\Phi =[I~F]\) cannot break the square-root bottleneck, even if n is prime. To do this, we will make use of the following theorem:
Theorem 13
(see [9, 27]) Denote \(\Delta (y):={\text {argmin}}\Vert x\Vert _1\) subject to \(\Phi x=y\). Then
if and only if \(\Phi \) satisfies the (k, c)-width property. Furthermore, \(C\asymp c\) in both directions of the equivalence.
Take x as defined in (15). Then \([x;-x]\) lies in the nullspace of [I F] and
where the last step follows from the proof of Theorem 11. As such, [I F] satisfies the (k, c)-width property for some c independent of n only if \(k=O(\sqrt{n})\). Furthermore, Theorem 13 implies that stable demixing by \(\ell _1\) reconstruction requires \(k=O(\sqrt{n})\), thereby proving the necessity of the square-root bottleneck in this case.
It is worth mentioning that the restricted isometry property is a sufficient condition for (18) (see [11], for example), and so by Theorem 8, one can break the square-root bottleneck by replacing the F in [I F] with a random unitary matrix. This gives a uniform demixing guarantee which is similar to those provided by McCoy and Tropp [33], though the convex program they consider differs from (17).
4.2 Compressed Sensing with Partial Fourier Operators
Consider the random \(m\times n\) matrix obtained by drawing rows uniformly with replacement from the \(n\times n\) discrete Fourier transform matrix. If \(m=\Omega _\delta (k{\text {polylog}} n)\), then the resulting partial Fourier operator satisfies the restricted isometry property, and this fact has been dubbed the uniform uncertainty principle [12]. A fundamental problem in compressed sensing is determining the smallest number m of random rows necessary. To summarize the progress to date, Candès and Tao [12] first found that \(m=\Omega _\delta (k\log ^6n)\) rows suffice, then Rudelson and Vershynin [37] proved \(m=\Omega _\delta (k\log ^4n)\), and recently, Bourgain [8] achieved \(m=\Omega _\delta (k\log ^3n)\); Nelson, Price and Wootters [34] also achieved \(m=\Omega _\delta (k\log ^3n)\), but using a slightly different measurement matrix. In this subsection, we provide a lower bound: in particular, \(m=\Omega _\delta (k\log n)\) is necessary whenever k divides n. Our proof combines ideas from the multiplicative uncertainty principle and the classical problem of coupon collecting.
The coupon collector’s problem asks how long it takes to collect all k coupons in an urn if you repeatedly draw one coupon at a time randomly with replacement. It is a worthwhile exercise to prove that the expected number of trials scales like \(k\log k\). We will require even more information about the distribution of the random number of trials:
Theorem 14
(see [13, 18]) Let \(T_k\) denote the random number of trials it takes to collect k different coupons, where in each trial, a coupon is drawn uniformly from the k coupons with replacement.
-
(a)
For each \(a\in \mathbb {R}\),
$$\begin{aligned} \lim _{k\rightarrow \infty }{\text {Pr}}\Big (T_k\le k\log k+ak\Big ) =e^{-e^{-(a+\gamma )}}, \end{aligned}$$where \(\gamma \approx 0.5772\) denotes the Euler–Mascheroni constant.
-
(b)
There exists \(c>0\) such that for each k,
$$\begin{aligned} \sup _{a\in \mathbb {R}}\bigg |{\text {Pr}}\Big (T_k\le k\log k+ak\Big )-e^{-e^{-(a+\gamma )}}\bigg | \le \frac{c\log k}{k}. \end{aligned}$$
Lemma 15
Suppose k divides n, and draw m iid rows uniformly from the \(n\times n\) discrete Fourier transform matrix to form a random \(m\times n\) matrix \(\Phi \). If \(m< k\log k\), then the nullspace of \(\Phi \) contains a k-sparse vector with probability \(\ge 0.4-c(\log k)/k\), where c is the constant from Theorem 14(b).
Proof
Let K denote the subgroup of \(\mathbb {Z}_n\) of size k, and let \(1_K\) denote its indicator function. We claim that some modulation of \(1_K\) resides in the nullspace of \(\Phi \) with the probability reported in the lemma statement. Let H denote the subgroup of \(\mathbb {Z}_n\) of size n / k. Then the Fourier transform of each modulation of \(1_K\) is supported on some coset of H. Letting M denote the random row indices that are drawn uniformly from \(\mathbb {Z}_n\), a modulation of \(1_K\) resides in the nullspace of \(\Phi \) precisely when M fails to intersect the corresponding coset of H. As there are k cosets, each with probability 1 / k, this amounts to a coupon-collecting problem (explicitly, each “coupon” is a coset, and we “collect” the cosets that M intersects). The result then follows immediately from Theorem 14(b):
\(\square \)
Presumably, one may remove the divisibility hypothesis in Lemma 15 at the price of weakening the conclusion. We suspect that the new conclusion would declare the existence of a vector x of numerical sparsity k such that \(\Vert \Phi x\Vert _2\ll \Vert x\Vert _2\). If so, then \(\Phi \) fails to satisfy the so-called robust width property, which is necessary and sufficient for stable and robust reconstruction by \(\ell _1\) minimization [9]. For the sake of simplicity, we decided not to approach this, but we suspect that modulations of the discrete Gaussian would adequately fill the role of the current proof’s modulated indicator functions.
What follows is the main result of this subsection:
Theorem 16
Let k be sufficiently large, suppose k divides n, and draw m iid rows uniformly from the \(n\times n\) discrete Fourier transform matrix to form a random \(m\times n\) matrix \(\Phi \). Take \(\delta <1/3\). Then \(\Phi \) satisfies the \((k,\delta )\)-restricted isometry property with probability \(\ge 2/3\) only if
where \(C(\delta )\) is some constant depending only on \(\delta \).
Proof
In the event that \(\Phi \) satisfies \((k,\delta )\)-RIP, we know that no k-sparse vector lies in the nullspace of \(\Phi \). Therefore, Lemma 15 implies
since otherwise \(\Phi \) fails to be \((k,\delta )\)-RIP with probability \(\ge 0.4-c(\log k)/k>1/3\), where the last step uses the fact that k is sufficiently large. Next, we leverage standard techniques from compressed sensing: \((k,\delta )\)-RIP implies (18) with \(C=C_1(\delta )\) (see [10, Theorem 3.3]), which in turn implies
by Theorem 11.7 in [19]. Since \(\Phi \) is \((k,\delta )\)-RIP with positive probability, we know there exists an \(m\times n\) matrix which is \((k,\delta )\)-RIP, and so m must satisfy (20). Combining with (19) then gives
The result then follows from applying the bound \(\max \{a,b\}\ge (a+b)/2\) and then taking \(C(\delta ):=(1/2)\min \{1,C_2(\delta )\}\). \(\square \)
We note that the necessity of \(k\log n\) random measurements contrasts with the proportional-growth asymptotic adopted in [7] to study the restricted isometry property of Gaussian matrices. Indeed, it is common in compressed sensing to consider phase transitions in which k, m and n are taken to infinity with fixed ratios k / m and m / n. However, since random partial Fourier operators fail to be restricted isometries unless \(m=\Omega _\delta (k\log n)\), such a proportional-growth asymptotic fails to capture the so-called strong phase transition of these operators [7].
The proof of Theorem 16 relies on the fact that the measurements are drawn at random. By contrast, it is known that every \(m\times n\) partial Hadamard operator fails to satisfy \((k,\delta )\)-RIP unless \(m=\Omega _\delta (k\log n)\) [22, 41]. We leave the corresponding deterministic result in the Fourier case for future work.
4.3 Fast Detection of Sparse Signals
The previous subsection established fundamental limits on the number of Fourier measurements necessary to perform compressed sensing with a uniform guarantee. However, for some applications, signal reconstruction is unnecessary. In this subsection, we consider one such application, namely sparse signal detection, in which the goal is to test the following hypotheses:
Here, we assume we know the 2-norm of the sparse vector we intend to detect, and we set it to be \(\sqrt{n/k}\) without loss of generality (this choice of scaling will help us interpret our results later). We will assume the data is accessed according to the following query–response model:
Definition 17
(Query–response model) If the ith query is \(j_i\in \mathbb {Z}_n\), then the ith response is \((Fx)[j_i]+\epsilon _i\), where the \(\epsilon _i\)’s are iid complex random variables with some distribution such that
The coefficient of variation v of \(|\epsilon _i|\) is defined as
Note that for any scalar \(c\ne 0\), the mean and variance of \(|c\epsilon _i|\) are \(|c|\alpha \) and \(|c|^2{\text {Var}}|\epsilon _i|\), respectively. As such, v is scale invariant and is simply a quantification of the “shape” of the distribution of \(|\epsilon _i|\). We will evaluate the responses to our queries with an \(\ell _1\) detector, defined below.
Definition 18
(\(\ell _1\) detector) Fix a threshold \(\tau \). Given responses \(\{y_i\}_{i=1}^m\) from the query–response model, if
then reject \(H_0\).
The following is the main result of this section:
Theorem 19
Suppose \(\alpha \le 1/(8k)\). Randomly draw m indices uniformly from \(\mathbb {Z}_n\) with replacement, input them into the query–response model and apply the \(\ell _1\) detector with threshold \(\tau =2m\alpha \) to the responses. Then
and
provided \(m\ge (8k+2v^2)/p\), where v is the coefficient of variation defined in (21).
In words, the probability that the \(\ell _1\) detector delivers a false positive is at most p, as is the probability that it delivers a false negative. These error probabilities can be estimated better given more information about the distribution of the random noise, and presumably, the threshold \(\tau \) can be modified to decrease one error probability at the price of increasing the other. Notice that we only use O(k) samples in the Fourier domain to detect a k-sparse signal. Since the sampled indices are random, it will take \(O(\log n)\) bits to communicate each query, leading to a total computational burden of \(O(k\log n)\) operations. This contrasts with the state-of-the-art sparse fast Fourier transform algorithms which require \(\Omega (k\log (n/k))\) samples and take \(O(k{\text {polylog}}n)\) time (see [26] and references therein). We suspect k-sparse signals cannot be detected with substantially fewer samples (in the Fourier domain or any domain).
We also note that the acceptable noise magnitude \(\alpha =O(1/k)\) is optimal in some sense. To see this, consider the case where k divides n and x is a properly scaled indicator function of the subgroup of size k. Then Fx is the indicator function of the subgroup of size n / k. (Thanks to our choice of scaling, each nonzero entry in the Fourier domain has unit magnitude.) Since a proportion of 1 / k entries is nonzero in the Fourier domain, we can expect to require O(k) random samples in order to observe a nonzero entry, and the \(\ell _1\) detector will not distinguish the entry from accumulated noise unless \(\alpha =O(1/k)\).
Before proving Theorem 19, we first prove a couple of lemmas. We start by estimating the probability of a false positive:
Lemma 20
Take \(\epsilon _1,\ldots ,\epsilon _m\) to be iid complex random variables with \(\mathbb {E}|\epsilon _i|=\alpha \) and \(\mathbb {E}|\epsilon _i|^2=\beta ^2\). Then
provided \(m\ge v^2/p\), where v is the coefficient of variation of \(|\epsilon _i|\) defined in (21).
Proof
Denoting \(X:=\sum _{i=1}^m|\epsilon _i|\), we have \(\mathbb {E}X=m\alpha \) and \({\text {Var}}X=m(\beta ^2-\alpha ^2)\). Chebyshev’s inequality then gives
Finally, we take \(t=m\alpha \) to get
\(\square \)
Next, we leverage the multiplicative uncertainty principle in Theorem 2 to estimate moments of noiseless responses:
Lemma 21
Suppose \(\Vert x\Vert _0\le k\) and \(\Vert x\Vert _2^2=n/k\). Draw j uniformly from \(\mathbb {Z}_n\) and define \(Y:=|(Fx)[j]|\). Then
Proof
Recall that \({\text {ns}}(x)\le \Vert x\Vert _0\le k\). With this, Theorem 2 gives
We rearrange and apply the definition of numerical sparsity to get
where the second to last equality is due to Parseval’s identity. Thus, \(\Vert Fx\Vert _1\ge n/k\). Finally,
and
\(\square \)
Proof of Theorem 19
Lemma 20 gives (22), and so it remains to prove (23). Denoting \(Y_i:=|(Fx)[j_i]|\), we know that \(|y_i|\ge Y_i-|\epsilon _i|\), and so
For notational convenience, put \(Z:=\sum _{i=1}^mY_i-\sum _{i=1}^m|\epsilon _i|\). We condition on the size of the noise and apply Lemma 20 with the fact that \(m\ge v^2/(p/2)\) to bound (24):
Now we seek to bound the second term of (25). Taking \(X=\sum _{i=1}^mY_i\), Lemma 21 gives \(\mathbb {E}X\ge m/k\) and \({\text {Var}}X=m{\text {Var}}Y_i\le m\mathbb {E}Y_i^2=m/k\). As such, applying Chebyshev’s inequality gives
Recalling that \(\alpha \le 1/(8k)\), we take \(t=m/(2k)\) to get
where the last step uses the fact that \(m\ge 8k/p\). Combining (24), (25), and (26) gives the result. \(\square \)
Notes
Recall that \(f(n)=O(g(n))\) if there exists \(C,n_0>0\) such that \(f(n)\le Cg(n)\) for all \(n>n_0\). We write \(f(n)=O_\delta (g(n))\) if the constant C is a function of \(\delta \). Also, \(f(n)=\Omega (g(n))\) if \(g(n)=O(f(n))\), and \(f(n)=o(g(n))\) if \(f(n)/g(n)\rightarrow 0\) as \(n\rightarrow \infty \).
References
Alon, N.: Problems and results in extremal combinatorics. Discret. Math. 273, 31–53 (2003)
Amrein, W.O., Berthier, A.M.: On support properties of \(L^p\)-functions and their Fourier transforms. J. Funct. Anal. 24, 258–267 (1977)
Bandeira, A.S., Fickus, M., Mixon, D.G., Moreira, J.: Derandomizing restricted isometries via the Legendre symbol. arXiv:1406.4089
Bandeira, A.S., Fickus, M., Mixon, D.G., Wong, P.: The road to deterministic matrices with the restricted isometry property. J. Fourier Anal. Appl. 19, 1123–1149 (2013)
Baraniuk, R., Davenport, M., DeVore, R., Wakin, M.: A simple proof of the restricted isometry property for random matrices. Constr. Approx. 28, 253–263 (2008)
Beckner, W.: Inequalities in Fourier analysis. Ann. Math. 102, 159–182 (1975)
Blanchard, J.D., Cartis, C., Tanner, J.: Compressed sensing: How sharp is the restricted isometry property? SIAM Rev. 53, 105–125 (2011)
Bourgain, J.: An improved estimate in the restricted isometry problem. Lect. Notes Math. 2116, 65–70 (2014)
Cahill, J., Mixon, D.G.: Robust width: A characterization of uniformly stable and robust compressed sensing. arXiv:1408.4409
Cai, T.T., Zhang, A.: Sharp RIP bound for sparse signal and low-rank matrix recovery. Appl. Comput. Harmon. Anal. 35, 74–93 (2013)
Candès, E.J.: The restricted isometry property and its implications for compressed sensing. C. R. Acad. Sci. Paris Ser. I 346, 589–592 (2008)
Candès, E.J., Tao, T.: Near-optimal signal recovery from random projections: Universal encoding strategies? IEEE Trans. Inf. Theory 52, 5406–5425 (2006)
Csörgő, S.: A rate of convergence for coupon collectors. Acta Sci. Math. (Szeged) 57, 337–351 (1993)
Demanet, L., Hand, P.: Scaling law for recovering the sparsest element in a subspace. arXiv:1310.1654
Donoho, D.L., Elad, M.: Optimally sparse representation in general (nonorthogonal) dictionaries via \(\ell ^1\) minimization. Proc. Natl. Acad. Sci. USA 100, 2197–2202 (2003)
Donoho, D.L., Huo, X.: Uncertainty principles and ideal atomic decomposition. IEEE Trans. Inf. Theory 47, 2845–2862 (2001)
Donoho, D.L., Stark, P.B.: Uncertainty principles and signal recovery. SIAM J. Appl. Math. 49, 906–931 (1989)
Erdős, P., Rényi, A.: On a classical problem of probability theory. Magy. Tud Akad. Mat. Kutató Int. Közl. 6, 215–220 (1961)
Foucart, A., Rauhut, H.: A Mathematical Introduction to Compressive Sensing. Applied and Numerical Harmonic Analysis. Birkhäuser, Basel (2013)
Gray, W.C.: Variable norm deconvolution, Tech. Rep. SEP-14, Stanford University (1978)
Gribonval, R., Nielsen, M.: Highly sparse representations from dictionaries are unique and independent of the sparseness measure. Appl. Comput. Harmon. Anal. 22, 335–355 (2007)
Guédon, O., Mendelson, S., Pajor, A., Tomczak-Jaegermann, N.: Majorizing measures and proportional subsets of bounded orthonormal systems. Rev. Mat. Iberoam. 24, 1075–1095 (2008)
Hardy, G.H.: A theorem concerning Fourier transforms. J. Lond. Math. Soc. 8, 227–231 (1933)
Heisenberg, W.: Über den anschaulichen Inhalt der quantentheoretischen Kinematik und Mechanik. Z. Phys. (in German) 43, 172–198 (1927)
Hurley, N., Rickard, S.: Comparing measures of sparsity. IEEE Trans. Inf. Theory 55, 4723–4741 (2009)
Indyk P, Kapralov M.: Sample-optimal Fourier sampling in any constant dimension. In: foundations of computer science (FOCS), 2014 IEEE 55th Annual Symposium on 2014, pp. 514–523
Kashin, B.S., Temlyakov, V.N.: A remark on compressed sensing. Math. Notes 82, 748–755 (2007)
Krahmer, F., Mendelson, S., Rauhut, H.: Suprema of chaos processes and the restricted isometry property. Commun. Pure Appl. Math. 67, 1877–1904 (2014)
Kraus, K.: Complementary observables and uncertainty relations. Phys. Rev. D 35, 3070–3075 (1987)
Laurent, B., Massart, P.: Adaptive estimation of a quadratic functional by model selection. Ann. Stat. 28, 1302–1338 (2000)
Lopes, M.E.: Estimating unknown sparsity in compressed sensing. JMLR W&CP 28, 217–225 (2013)
McCoy, M.B., Cevher, V., Dinh, Q.T., Asaei, A., Baldassarre, L.: Convexity in source separation: models, geometry, and algorithms. Signal Proc. Mag. 31, 87–95 (2014)
McCoy, M.B., Tropp, J.A.: Sharp recovery bounds for convex demixing, with applications. Found. Comput. Math. 14, 503–567 (2014)
Nelson, J., Price, E., Wootters, M.: New constructions of RIP matrices with fast multiplication and fewer rows. In: SODA, pp. 1515–1528 (2014)
Rauhut, H.: Compressive sensing and structured random matrices. Theor. Found. Numer. Methods Sparse Recover. 9, 1–92 (2010)
Repetti, A., Pham, M.Q., Duval, L., Chouzenoux, É., Pesquet, J.-C.: Euclid in a taxicab: sparse blind deconvolution with smoothed \(\ell _1/\ell _2\) regularization. IEEE Signal Process. Lett. 22, 539–543 (2014)
Rudelson, M., Vershynin, R.: On sparse reconstruction from Fourier and Gaussian measurements. Comm. Pure Appl. Math. 61, 1025–1045 (2008)
Santosa, F., Symes, W.W.: Linear inversion of band-limited reflection seismograms. SIAM J. Sci. Stat. Comput. 7, 1307–1330 (1986)
Stevenhagen, P., Lenstra, H.W.: Chebotarëv and his density theorem. Math. Intell. 18, 26–37 (1996)
Studer, C.: Recovery of signals with low density. arXiv:1507.02821
Talagrand, M.: Selecting a proportion of characters. Israel J. Math. 108, 173–191 (1998)
Tao, T.: An uncertainty principle for cyclic groups of prime order. Math. Res. Lett. 12, 121–127 (2005)
Terras, A.: Fourier Analysis on Finite Groups and Applications. Cambridge University Press, Cambridge (1999)
Tropp, J.A.: The sparsity gap: uncertainty principles proportional to dimension. In: CISS, pp. 1–6 (2010)
Tropp, J.A.: On the linear independence of spikes and sines. J. Fourier Anal. Appl. 14, 838–858 (2008)
Tropp, J.A.: On the conditioning of random subdictionaries. Appl. Comput. Harmon. Anal. 25, 1–24 (2008)
Vershynin, R.: Introduction to the non-asymptotic analysis of random matrices. Theory and Applications, Cambridge University Press, Compressed Sensing (2012)
Acknowledgements
The authors thank Laurent Duval, Joel Tropp, and the anonymous referees for multiple suggestions that significantly improved the presentation of our results and our discussion of the relevant literature. ASB was supported by AFOSR Grant No. FA9550-12-1-0317. DGM was supported by an AFOSR Young Investigator Research Program award, NSF Grant No. DMS-1321779, and AFOSR Grant No. F4FGA05076J002. The views expressed in this article are those of the authors and do not reflect the official policy or position of the United States Air Force, Department of Defense, or the U.S. Government.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Roman Vershynin.
Rights and permissions
About this article
Cite this article
Bandeira, A.S., Lewis, M.E. & Mixon, D.G. Discrete Uncertainty Principles and Sparse Signal Processing. J Fourier Anal Appl 24, 935–956 (2018). https://doi.org/10.1007/s00041-017-9550-x
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00041-017-9550-x