Improved Constructions for Non-adaptive Threshold Group Testing

Abstract

The basic goal in combinatorial group testing is to identify a set of up to d defective items within a large population of size n≫d using a pooling strategy. Namely, the items can be grouped together in pools, and a single measurement would reveal whether there are one or more defectives in the pool. The threshold model is a generalization of this idea where a measurement returns positive if the number of defectives in the pool reaches a fixed threshold u>0, negative if this number is no more than a fixed lower threshold ℓ<u, and may behave arbitrarily otherwise. We study non-adaptive threshold group testing (in a possibly noisy setting) and show that, for this problem, O(d ^g+2(logd)log(n/d)) measurements (where g:=u−ℓ−1 and u is any fixed constant) suffice to identify the defectives, and also present almost matching lower bounds. This significantly improves the previously known (non-constructive) upper bound O(d ^u+1log(n/d)). Moreover, we obtain a framework for explicit construction of measurement schemes using lossless condensers. The number of measurements resulting from this scheme is ideally bounded by O(d ^g+3(logd)logn). Using state-of-the-art constructions of lossless condensers, however, we obtain explicit testing schemes with O(d ^g+3(logd)quasipoly(logn)) and O(d ^g+3+β poly(logn)) measurements, for arbitrary constant β>0.

1 Introduction

Combinatorial group testing is a classical problem that deals with identification of sparse Boolean vectors using disjunctive queries. Suppose that among a large set of n items it is suspected that, for some sparsity parameter d≪n, up to d items might be “defective”. In technical terms, defective items are known as positives and the rest are called negatives. In a pooling strategy, the items may be arbitrarily grouped in pools, and a single “measurement” reveals whether there is one or more positives within the chosen pool. The basic goal in group testing to design the pools in such a way that the set of positives can be identified from a number of measurements that is substantially less than n.

Since its introduction in 1940’s [16], group testing and its variations have been extensively studied and have found surprisingly many applications in seemingly unrelated areas. In particular, we mention applications in molecular biology and DNA library screening (cf. [3, 24, 31, 33, 38, 44, 45] and the references therein), multi-access communication [43], data compression [28], pattern matching [12], streaming algorithms [13], software testing [2], compressed sensing [14], and secure key distribution [6], among others. We refer the reader to [17, 18] for an extensive review of the major results in this area.

Formally, in classical group testing one aims to learn an unknown Boolean vector (x ₁,…,x _n )∈{0,1}ⁿ which is known to be d-sparse (that is, contains at most d non-zero entries) using a set of m measurements, where each measurement is defined by a subset of the coordinates $\mathcal{I} \subseteq[n]$ and outputs the logical “or” $\bigvee_{i \in\mathcal{I}} x_{i}$. The goal is then to design the measurements in such a way that all d sparse vectors become uniquely identifiable using as few number of measurements as possible.

A natural generalization of classical group testing (that we call threshold testing), introduced by Damaschke [15], considers the case where the measurement outcomes are determined by a threshold predicate instead of the logical or. Namely, this model is characterized by two integer parameters ℓ,u such that 0≥ℓ<u (that are considered fixed constants), and each measurement outputs positive if the number of positives within the corresponding pool is at least u. On the other hand, if the number of positives is less than or equal to¹ ℓ, the test returns negative, and otherwise the outcome can be arbitrary (that is, either 0 or 1 in any arbitrary way). In this view, classical group testing corresponds to the special case where ℓ=0 and u=1. In addition to being of theoretical interest, the threshold model is interesting for applications, in particular in biology, where the measurements have reduced or unpredictable sensitivity or may depend on various factors that must be simultaneously present in the sample to result in a positive outcome.

The difference g:=u−ℓ−1 is known as the gap parameter. As shown by Damaschke [15], in threshold group testing identification of the set of positives is only possible when the number of positives is at least u. Moreover, regardless of the number of measurements, in general the set of positives can be only approximately identified within up to g false positives and g false negatives (thus, unique identification can only be guaranteed when ℓ=u−1). Additionally, Damaschke constructed a scheme for identification of the positives in the threshold model. For the gap-free case where g=0, the number of measurements in this scheme is O(dlogn), which is nearly optimal (within constant factors). However, when g>0, the number of measurements becomes O(dn ^b+d ^u), for an arbitrary constant b>0, if up to g+(u−1)/b misclassifications are allowed.

A drawback of the scheme presented by Damaschke is that the measurements are adaptive; i.e., the group chosen by each measurement can depend on the outcomes of the previous ones. For numerous applications (in particular, in molecular biology), adaptive measurements are infeasible and must be avoided. In a non-adaptive setting, all measurements must be specified before their outcomes are revealed. This makes it convenient to think of the measurements in a matrix form. Specifically, a non-adaptive measurement matrix is an m×n Boolean matrix whose ith row is the characteristic vector of the set of items participating in the ith pool, and the goal would be to design a suitable measurement matrix.

More recently, non-adaptive threshold testing has been considered by Chen and Fu [5]. They observe that a generalization of the standard notion of disjunct matrices, the latter being extensively used in the literature of classical group testing, is suitable for the threshold model. Throughout this work, we refer to this generalized notion as strongly disjunct matrices and to the standard notion as classical disjunct matrices. Using strongly disjunct matrices, they show that O(ed ^u+1log(n/d)) non-adaptive measurements suffices to identify the set of positives (within g false positives/negatives) even if up to e erroneous measurements are allowed in the model. This number of measurements almost matches (up to constant factors) the known lower bounds on the number of rows of strongly disjunct matrices. However, the dependence on the sparsity parameter is d ^u+1, which might be prohibitive for an interesting range of parameters, when the thresholds are not too small (e.g., ℓ+1=u=10) and the sparsity parameter is rather large (e.g., d=n ^1/10).

In this work, we consider the non-adaptive threshold model in a possibly noisy setting, where a number of measurement outcomes (specified by an error parameter e≥0) may be incorrect. Our first observation is that, a new variation of classical disjunct matrices (that is in general strictly weaker than strongly disjunct matrices) suffices for the purpose of threshold group testing. Using a randomness-efficient probabilistic construction (that requires poly(d,logn) bits of randomness), we construct generalized disjunct matrices with O(d ^g+2(logd)log(n/d)) rows. Thus, we bring the exponent of d in the asymptotic number of measurements from u+1 (that is optimal for strongly disjunct matrices) down to g+2, which is independent of the actual choice of the thresholds and only depends on the gap between them. We also show that this tradeoff is essentially optimal for our notion of disjunct matrices. In the gap-free case, we furthermore show that this tradeoff is in fact the best to hope for (up to a logd term) for any threshold testing design, and thus our notion of disjunct matrices is indeed optimal (Corollary 13). For the positive-gap case, we show that the dependence d ^g+2, up to poly-logarithmic factors, is necessary for any threshold testing design, and thus our notion obtains the correct exponent (Corollary 28).

We proceed to define a new auxiliary object, namely the notion of regular matrices, that turns out to be the key combinatorial object in our explicit constructions. Intuitively, given a gap g≥0, a suitable regular matrix M ₁ can be used to take any measurement matrix M ₂ designed for the threshold model with lower threshold ℓ=0 and higher threshold u=g+1 and “lift” it up to matrix that works for any arbitrary lower threshold ℓ′>0 and the same gap g. Therefore, for instance, in order to address the gap-free model, it would suffice to have a non-adaptive scheme for the classical group testing model with ℓ+1=u=1. This transformation is accomplished using a simple product that increases the height of the original matrix M ₂ by a multiplicative factor equal to the height of the regular matrix M ₁, while preserving the “low-threshold” distinguishing properties of the original matrix M ₂.

Next, we introduce a framework for construction of regular matrices using strong lossless condensers that are fundamental objects in derandomization theory, and more generally, theoretical computer science. We show that, by using an optimal condenser, it is possible to construct regular matrices with only O(d(logd)logn) rows. This almost matches the upper bound achieved by a probabilistic construction that we also present in this work. To this date, no explicit construction of such optimal lossless condensers is known (though probabilistic constructions are easy to obtain). However, using state of the art in explicit condensers [4, 27], we will obtain two explicit constructions of regular matrices with incomparable parameters. Namely, one with O(d(logd)quasipoly(logn)) rows and another with O(d ^1+β poly(logn)), where β>0 is any arbitrary constant and the exponent of the term poly(logn) depends on the choice of β. By combining regular matrices with strongly disjunct ones (designed for the lowered thresholds ℓ′=0 and u′=g+1), we obtain our threshold testing schemes. The bounds obtained by our final schemes are summarized in Table 1. When the lower threshold ℓ is not too small, our explicit constructions (rows M8 and M9 of Table 1) significantly improve what was previously known to be achievable even using non-constructive proofs.

Table 1 Summary of the parameters achieved by various constructions of threshold disjunct matrices. The noise parameter p∈[0,1) is arbitrary, and thresholds ℓ,u=ℓ+g+1 are fixed constants. “Exp” and “Rnd” respectively indicate explicit and randomized constructions. “KS” refers to the construction of strongly disjunct matrices based on Kautz-Singleton superimposed codes [29], as described later in Sect. 5 (the bounds in rows M1–M5 are obtained by strongly disjunct matrices)

The rest of the paper is organized as follows. In Sect. 1.1 we introduce preliminary notions and fix some notation. In Sect. 2 we formalize the notion of threshold testing designs. Moreover, we review the notion of strongly disjunct matrices and introduces our weaker notion of threshold disjunct matrices (for the gap-free case g=0), in addition to the notion of regular matrices and its properties. We will also prove lower bounds on the number of rows of such matrices. In Sect. 3 we obtain matching probabilistic upper bounds on the number of rows using the probabilistic method. Furthermore, we develop our construction of regular matrices from lossless condensers, and instantiate the parameters in Sect. 3.1. This in particular leads to our explicit threshold testing schemes. In Sect. 4 we extend all our results to the case with nonzero gap. In Sect. 5, we obtain explicit constructions of strongly disjunct matrices from error-correcting codes, by extending the classical technique initiated by Kautz and Singleton. Finally, in Sect. 6 we discuss the future directions.

1.1 Preliminaries

For a matrix M, we denote by M[i,j] the entry of M at the ith row and the jth column. Similarly, we denote the ith entry of a vector v by v(i). The support a vector x∈{0,1}ⁿ, denoted by supp(x), is a subset of [n]:={1,…,n} such that i∈supp(x) if and only if x(i)=1. The Hamming weight of x, denoted by wgt(x) is defined as |supp(x)|. The Hamming distance between vectors x,x′∈{0,1}ⁿ is denoted by dist(x,x′).

For an m×n Boolean matrix M and S⊆[n], we denote by M| _S the m×|S| submatrix of M formed by restricting M to the columns picked by S. Moreover, for a vector x∈{0,1}ⁿ, we use M[x] _ℓ,u to denote the set of all possible outcomes of measuring x in the threshold model with lower and upper thresholds ℓ and u and using the measurement matrix M. Formally, for any y∈M[x] _ℓ,u we have y(i)=1 if |supp(M(i))∩supp(x)|≥u, and y(i)=0 if |supp(M(i))∩supp(x)|≤ℓ, where here M(i) indicates the ith row of M. In the gap-free case, the measurement outcome is uniquely defined (since there is no ambiguity in the measurement process), and thus the set M[x] _ℓ,u only contains a single element that we denote by M[x] _u .

The min-entropy of a distribution $\mathcal {X}$ with finite support Ω is given by

$$H_\infty(\mathcal {X}) := \min_{x \in\varOmega}\{-\log \mathcal {X}(x)\}, $$

where $\mathcal {X}(x)$ is the probability that $\mathcal {X}$ assigns to the outcome x and logarithm is taken to base 2. A flat distribution is one that is uniform on its support. For such a distribution $\mathcal {X}$, we have $H_{\infty}(\mathcal {X}) = \log(|\mathsf {supp}(\mathcal {X})|)$. The statistical distance between two distributions $\mathcal {X}$ and $\mathcal {Y}$ defined on the same finite space Ω is given by $\frac{1}{2} \sum_{s \in\varOmega} |\mathcal {X}(s) - \mathcal {Y}(s)|$, which is half the ℓ ₁ distance of the two distributions when regarded as vectors of probabilities over Ω. Two distributions $\mathcal {X}$ and $\mathcal {Y}$ are said to be ϵ-close if their statistical distance is at most ϵ. We will use the shorthand $\mathcal {U}_{n}$ for the uniform distribution on {0,1}ⁿ, and $X \sim \mathcal {X}$ for a random variable X drawn from a distribution $\mathcal {X}$.

The main technical tool that we use in our explicit constructions is the notion of lossless condensers, defined below.

Definition 1

A function $f\colon \{0,1\}^{{\tilde {n}}}\times \{0,1\}^{t}\to \{0,1\}^{{\tilde {\ell }}}$ is a strong lossless condenser for entropy k and with error ϵ (in short, (k,ϵ)-condenser) if for every distribution $\mathcal {X}$ on $\{0,1\}^{{\tilde {n}}}$ with min-entropy at least k, random variable $X \sim \mathcal {X}$ and a seed $Y \sim \mathcal {U}_{t}$, the distribution of (Y,f(X,Y)) is ϵ-close to some distribution $(\mathcal {U}_{t}, \mathcal {Z})$ with min-entropy at least t+k. A condenser is explicit if it is polynomial-time computable.

We will use the following “almost-injectivity” property of lossless condensers in our proofs.

Proposition 2

Let $\mathcal {X}$ be a flat distribution with min-entropy logK over a finite sample space Ω and f:Ω→Γ be a mapping to a finite set Γ. If $f(\mathcal {X})$ is ϵ-close to having min-entropy logK, then there is a set T⊆Γ of size at least (1−4ϵ)K such that

$$(\forall y \in T)\quad f(x)=y \land f(x') = y\quad \Rightarrow \quad x = x'. $$

Proof

Suppose that $\mathcal {X}$ is uniformly supported on a set S⊆Ω of size K. For each y∈Γ, define n _y :=|{x∈Ω:f(x)=y}|. Denote by μ the distribution $f(\mathcal {X})$ over Γ and by μ′ a distribution on Γ with min-entropy logK that is ϵ-close to μ, which is guaranteed to exist by the assumption. Define T:={y∈Γ:n _y =1}, and similarly, T′:={y∈Γ:n _y ≥2}. Observe that for each y∈Γ we have μ(y)=n _i /K, and also supp(μ)=T∪T′. Thus,

$$ |T| + \sum_{y \in T'} n_y = K. $$

(1)

The fact that μ and μ′ are ϵ-close implies that

$$\sum_{y \in T'} | \mu(y) - \mu'(y) | \leq2 \epsilon \quad\Rightarrow\quad\sum_{y \in T'} (n_y - 1) \leq2 \epsilon K. $$

In particular, this means that |T′|≤2ϵK (since by the choice of T′, for each y∈T′ we have n _y ≥2). Furthermore,

$$\sum_{y \in T'} (n_y - 1) \leq2 \epsilon K \quad\Rightarrow\quad\sum_{y \in T'} n_y \leq2 \epsilon K + |T'| \leq4\epsilon K. $$

This combined with (1) gives

$$|T| = K - \sum_{y \in T'} n_y \geq(1-4\epsilon ) K, $$

as desired. □

2 Variations of Disjunct Matrices

The combinatorial structure used by Chen and Fu in their non-adaptive scheme is the following generalization of the standard notion of disjunct matrices that we refer to as strongly disjunct matrices throughout this work.

Definition 3

A matrix (with at least d+u columns) is said to be strongly (d,e;u)-disjunct if for every choice of d+u columns $C_{1},\ldots, C_{u}, C'_{1}, \ldots, C'_{d}$, all distinct, we have

$$\bigg|\bigcap_{i=1}^u \mathsf {supp}(C_i) \setminus\bigcup_{i=1}^{d} \mathsf {supp}(C'_i) \bigg| > e. $$

Observe that, strongly (d,e;u)-disjunct matrices are, in particular, strongly (d′,e′;u′)-disjunct for any d′≤d, e′≤e, and u′≤u. Moreover, classical (d,e)-disjunct matrices that are extensively used in group testing literature (see [17, Chap. 7]) are equivalent to strongly (d,e;1)-disjunct matrices.

To make the main ideas more transparent, until Sect. 4 we will focus on the gap-free case where ℓ=u−1. The extension to nonzero gaps is straightforward and will be discussed in Sect. 4. Moreover, often we will implicitly assume that the Hamming weight of the Boolean vector that is to be identified is at least u (since otherwise, we know from the work of Damaschke [15] that confusions cannot be avoided), and will take the thresholds ℓ,u to be fixed constants.

The notion of strongly disjunct matrices, in its general form, has been studied in the literature under different names and equivalent formulations, e.g., superimposed (u,d)-designs/codes, superimposed distance codes, and (u,d) cover-free families (see [6, 7, 20, 23, 30, 39, 40] and the references therein). An important motivation for the study of this notion is the following hidden hypergraph-learning problem (cf. [17, Chap. 12]), itself being motivated by the so-called complex model in computational biology [6]: Suppose that G is a u-hypergraph; that is, a hypergraph where each edge is a set of u vertices. on a vertex set V of size n, and denote by $\mathcal{V}(G)$ the set of vertices induced by the hyper-edge set of G; i.e., $v \in \mathcal{V}(G)$ if and only if G has a hyper-edge incident to v. Then assuming that $|\mathcal{V}(G)| \leq d$ for a sparsity parameter d, the aim is to learn G using as few (non-adaptive) queries of the following type as possible: Each query specifies a set Q⊆V, and its corresponding answer is a Boolean value which is 1 if and only if G has a hyperedge contained in Q. It is known that [6, 25], in the hypergraph-learning problem, any suitable grouping strategy defines a strongly disjunct matrix (whose rows are characteristic vectors of individual queries Q), and conversely, any strongly disjunct matrix can be used as the incidence matrix of the set of queries. The parameter e determines “noise tolerance” of the measurement scheme. Namely, a strongly (d,e;u)-disjunct matrix can uniquely distinguish between d-sparse hypergraphs even in presence of up to ⌊e/2⌋ erroneous query outcomes.

For gap-free threshold group testing, the successful strategy needed for distinguishing between d-sparse Boolean vectors can trivially be captured by the following definition.

Definition 4

Let n≥d≥u>0 and e≥0 be integer parameters. A Boolean matrix M with n columns is said to be a (d,e;u)-threshold design if for every d-sparse x,x′∈{0,1}ⁿ of Hamming weight u or more such that x≠x′, we have dist(M[x] _u ,M[x′] _u )>e.

The key observation made by Chen and Fu [5] is that threshold group testing corresponds to the special case of the hypergraph learning problem where the hidden graph G is known to be a u-clique.² In this case, the unknown Boolean vector in the corresponding threshold testing problem would be the characteristic vector of $\mathcal{V}(G)$. It follows that strongly disjunct matrices are threshold designs as defined in Definition 4 Specifically,

Theorem 5

[5]

Let M be a Boolean matrix with n columns that is strongly (d,e;u)-disjunct. Then, M is a (d,e;u)-threshold design.

Nonconstructively, a probabilistic argument akin to the standard argument for the case of classical disjunct matrices (see [17, Chap. 7]) can be used to show that strongly (d,e;u)-disjunct matrices exist with m=O(d ^u+1(log(n/d))/(1−p)²) rows and error tolerance e=Ω(pdlog(n/d)/(1−p)²), for any noise parameter p∈[0,1). On the negative side, however, several concrete lower bounds are known for the number of rows of such matrices [23, 39, 40]. In asymptotic terms, these results show that one must have m=Ω(d ^u+1log _d n+ed ^u), and thus, the probabilistic upper bound is essentially optimal.

For the underlying strongly disjunct matrix, Chen and Fu [5] use a greedy construction [7] that achieves, for any e≥0, O((e+1)d ^u+1log(n/d)) rows, but may take exponential time in the size of the resulting matrix. Nevertheless, as observed by several researchers [6, 23, 25, 30], a classical explicit construction of combinatorial designs due to Kautz and Singleton [29] can be extended to construct strongly disjunct matrices. This concatenation-based construction transforms any error-correcting code having large distance into a disjunct matrix. While the original construction uses Reed-Solomon codes and achieves nice bounds, it is possible to use other families of codes. In particular, as recently shown by Porat and Rothschild [34], codes on the Gilbert-Varshamov bound (cf. [32]) result in nearly optimal disjunct matrices. Moreover, for a suitable range of parameters, they give a deterministic construction of such codes that runs in polynomial time in the size of the resulting disjunct matrix (albeit exponential in the dimension of the code)³. We will elaborate on details of this class of constructions in Sect. 5, and will additionally consider a family of algebraic-geometric codes and Hermitian codes which give incomparable bounds, as summarized in Table 1 (rows M2–M5).

2.1 Threshold Disjunct and Regular Matrices

Even though, as discussed above, the general notion of strongly (d,e;u)-disjunct matrices is sufficient for threshold group testing with upper threshold u, in this section we show that a new, weaker, notion of disjunct matrices defined below (which, as we show later, turns out to be strictly weaker when u>1), would also suffice. We also define an auxiliary notion of regular matrices.

Definition 6

A Boolean matrix M with n columns is called (d,e;u)-regular if for every subset of columns S⊆[n] (called the critical set) and every Z⊆[n] (called the zero set) such that u≤|S|≤d, |Z|≤|S|, S∩Z=∅, there are more than e rows of M at which M| _S has weight exactly u and (at the same rows) M| _Z has weight zero. Any such row is said to u-satisfy S and Z. If, in addition, for every distinguished column i∈S, more than e rows of M both u-satisfy S and Z and have a 1 at the ith column, the matrix is called threshold (d,e;u)-disjunct (and the corresponding “good” rows are said to u-satisfy i, S, and Z).

To distinguish between the above variant of disjunct matrices and strongly disjunct matrices or classical disjunct matrices, we will refer to our variant as threshold disjunct matrices throughout the paper.

It is easy to verify that (assuming 2d≤n) the classical notion of (2d−1,e)-disjunct matrices is equivalent to strongly (2d−1,e;1)-disjunct and threshold (d,e;1)-disjunct. Moreover, any threshold (d,e;u)-disjunct matrix is (d,e;u)-regular, (d−1,e;u−1)-regular, and classical (d,e)-disjunct (but the reverse implications do not in general hold). Therefore, the known lower bound of m=Ω(d ²log _d n+ed) that applies for (d,e)-disjunct matrices holds for threshold (d,e;u)-disjunct matrices as well (see Theorem 10). Below we show that our notion of disjunct matrices suffices for threshold designs.

Lemma 7

Let M be an m×n Boolean matrix that is threshold (d,e;u)-disjunct. Then for every distinct d-sparse vectors x,x′∈{0,1}ⁿ such that supp(x)⊈supp(x′), wgt(x)≥|supp(x′)∖supp(x)| and wgt(x)≥u, we have

$$ |\mathsf {supp}(M[x]_u) \setminus \mathsf {supp}(M[x']_u)| > e. $$

(2)

Moreover, M is a (d,e;u)-threshold design. Conversely, if M satisfies (2) for every choice of x and x′ as above, it must be threshold (⌊d/2⌋,e;u)-disjunct.

Proof

First, suppose that M is threshold (d,e;u)-disjunct, and let y:=M[x] _u and y′:=M[x′] _u . Take any i∈supp(x)∖supp(x′), and let S:=supp(x) and Z:=supp(x′)∖supp(x). Note that |S|≤d and by assumption, we have |Z|≤|S|. Now, Definition 6 implies that there is a set E of more than e rows of M that u-satisfy i as the distinguished column, S as the critical set and Z as the zero set. Thus for every j∈E, the jth row of M restricted to the columns chosen by supp(x) must have weight exactly u, while its weight on supp(x′) is less than u. Therefore, y(j)=1 and y′(j)=0 for more than e choices of j.

The claim that M is a (d,e;u)-threshold design follows from the above argument combined with the observation that at least one of the two possible orderings of any two distinct d-sparse vectors, at least one having weight u or more, satisfies the conditions required by the lemma.

For the converse, consider any choice of a distinguished column i∈[n], a critical set S⊆[n] containing i (such that |S|≥u), and a zero set Z⊆[n] where |Z|≤|S|. Define d-sparse Boolean vectors x,x′∈{0,1}ⁿ so that supp(x):=S and supp(x′):=Z∪(S∖{i}). Let y:=M[x] _u and y′:=M[x′] _u and E:=supp(y)∖supp(y′). By assumption we know that |E|>e. Take any j∈E. Since y(j)=1 and y′(j)=0, we get that the jth row of M restricted to the columns picked by Z∪(S∖{i}) must have weight at most u−1, whereas it must have weight at least u when restricted to S. As the sets {i},S∖{i}, and Z are disjoint, this can hold only if M[j,i]=1, and moreover, the jth row of M restricted to the columns picked by S (resp., Z) has weight exactly u (resp., zero). Hence, this row (as well as all the rows of M picked by E) must u-satisfy i,S, and Z, confirming that M is threshold (⌊d/2⌋,e;u)-disjunct. □

We point out that Lemma 7 proves a matching converse, suggesting that the notion of threshold disjunct matrices might be “close” to a characterization of threshold designs (Definition 4), up to a constant factor in the sparsity parameter. However, this does not imply a precise characterization since the assumptions of Lemma 7 consider a particular ordering on the sparse vectors x and x′, which must be consistent with the ordering in (2). However, as we show in Sect. 2.3, threshold designs (Definition 4) and threshold disjunct matrices (Definition 6) satisfy the same asymptotic lower bounds on the number of rows, which nearly matches the upper bounds that we prove by probabilistic arguments (Lemma 14), assuming that the threshold parameter is an absolute constant. Thus, quantitatively, our notion of threshold disjunct matrices essentially provides an optimal way of constructing threshold group testing designs.

2.2 Direct Product of Matrices

We will use regular matrices as intermediate building blocks in our constructions of disjunct matrices to follow. The connection with disjunct matrices is made apparent through a direct product of matrices defined in Construction 1. Intuitively, using this product, regular matrices can be used to transform any measurement matrix suitable for the standard group testing model to one with comparable properties in the threshold model. The following lemma formalizes the idea.

Construction 1

Direct product of measurement matrices

Lemma 8

Let M ₁ and M ₂ be Boolean matrices with n columns, such that M ₁ is (d−1,e ₁;u−1)-regular. Let M:=M ₁⊙M ₂, and suppose that for d-sparse Boolean vectors x,x′∈{0,1}ⁿ such that wgt(x)≥wgt(x′), we have

$$| \mathsf {supp}(M_2[x]_1) \setminus \mathsf {supp}(M_2[x']_1)| \geq e_2. $$

Then, |supp(M[x] _u )∖supp(M[x′] _u )|≥(e ₁+1)e ₂.

Proof

First we consider the case where u>1. Let $y := M_{2}[x]_{1} \in \{0,1\}^{m_{2}}$, $y' := M_{2}[x']_{1} \in \{0,1\}^{m_{2}}$, where m ₂ is the number of rows of M ₂, and let E:=supp(y)∖supp(y′). By assumption, |E|≥e ₂. Fix any i∈E so that y(i)=1 and y′(i)=0. Therefore, the ith row of M ₂ must have all zeros at positions corresponding to supp(x′) and there is a j∈supp(x)∖supp(x′) such that M ₂[i,j]=1. Define S:=supp(x)∖{j}, Z:=supp(x′)∖supp(x), z:=M[x] _u and z′:=M[x′] _u .

As wgt(x)≥wgt(x′), we know that |Z|≤|S|+1. The extreme case |Z|=|S|+1 only happens when x and x′ have disjoint supports, in which case one can remove an arbitrary element of Z to ensure that |Z|≤|S| and the following argument (considering the assumption u>1) still goes through.

By the definition of regularity, there is a set E ₁ consisting of at least e ₁+1 rows of M ₁ that (u−1)-satisfy the critical set S and the zero set Z. Pick any k∈E ₁, and observe that z must have a 1 at position (k,i). This is because the row of M indexed by (k,i) has a 1 at the jth position (since the kth row of M ₂ does), and at least u−1 more 1’s at positions corresponding to supp(x)∖{j} (due to regularity of M ₁). On the other hand, note that the kth row of M ₁ has at most u−1 ones at positions corresponding to supp(x′) (because supp(x′)⊆S∪Z), and the ith row of M ₂ has all zeros at those positions (because y′(i)=0). This means that the row of M indexed by (k,i) (which is the bit-wise or of the kth row of M ₁ and the ith row of M ₂) must have less than u ones at positions corresponding to supp(x′), and thus, z′ must be 0 at position (k,i). Therefore, z and z′ differ at position (k,i).

Since there are at least e ₂ choices for i, and for each choice of i, at least e ₁+1 choices for k, we conclude that in at least (e ₁+1)e ₂ positions, z has a one while z′ has a zero.

The argument for u=1 is similar, in which case it suffices to take S:=supp(x) and Z:=supp(x′)∖supp(x). □

As a corollary it follows that, when M ₁ is a (d−1,e ₁;u−1)-regular and M ₂ is a classical (d,e ₂)-disjunct matrix, the product M:=M ₁⊙M ₂ will distinguish between any two distinct d-sparse vectors (of weight at least u) in at least (e ₁+1)(e ₂+1) positions of the measurement outcomes. This combined with Lemma 7 would imply that M is, in particular, threshold (⌊d/2⌋,(e ₁+1)(e ₂+1)−1;u)-disjunct. However, using a direct argument similar to the above lemma it is possible to obtain a slightly better result, given by Lemma 9.

Lemma 9

Suppose that M ₁ is a (d,e ₁;u−1)-regular and M ₂ is a classical (2d,e ₂)-disjunct matrix. Then M ₁⊙M ₂ is a threshold (d,(e ₁+1)(e ₂+1)−1;u)-disjunct matrix.

As a particular example of where Lemma 8 can be used, we remark that the measurement matrices constructed in [9] that are not necessarily disjunct but allow approximation of sparse vectors in highly noisy settings of the standard group testing model (as well as those used in adaptive two-stage schemes; cf. [8] and the references therein), can be combined with regular matrices to offer the same qualities in the threshold model. In the same way, numerous existing results in group testing can be ported to the threshold model by using Lemma 8.

2.3 Lower Bounds

In this section, we show that the known asymptotic lower bounds on the number of rows of classical disjunct matrices apply to threshold designs (Definition 4) and our notion of threshold disjunct matrices (6) as well. It is immediate from the definitions that, assuming 2d≤n, a threshold (d,e;u)-disjunct matrix is in particular a classical (d,e)-disjunct matrix. Thus the latter lower bound is straightforward.

Theorem 10

For every integer d>0 there is an n ₀>0 such that the following holds. For any n≥n ₀, let M be an m×n threshold (d,e;u)-disjunct matrix. Then,

$$m = \varOmega( d^2 \log_d n + de ). $$

Proof

Immediate from the known bounds on the number of rows of classical disjunct matrices (e.g., Theorem 2.19 of [39]). □

Now, in order to show that any (d,e;u)-threshold design must satisfy essentially the same lower bound as in Theorem 10, we first observe the following combinatorial property of such matrices.

Lemma 11

Let M be a (d+1,e;ℓ+1)-threshold design. Then it satisfies the following property:

“For every S⊆[n] such that |S|=d and every i∈[n]∖S, there are more than e rows of M at which the ith column of M contains a 1 and moreover in those rows, M| _S has weight exactly ℓ.”

Proof

This is a special case of Lemma 26 that will be proved later (it suffices to set u=ℓ+1 and g=1 in Lemma 26). □

Theorem 12

For every integer d>0 there is an n ₀>0 such that the following holds. For any n≥n ₀, let M be an m×n Boolean matrix that satisfies the property quoted in Lemma 11. Then,

$$m = \varOmega\bigg( \bigg(\frac{d}{\ell+1}\bigg)^2 \log_d n + \frac {de}{(\ell+1)^2} \bigg). $$

Proof

We reduce the matrix to a classical disjunct matrix, and use the existing lower bounds. Let d′:=⌊d/(ℓ+1)⌋ and e′:=e/(ℓ+1). We define the following notation: For a set S⊆[n] and i∈[n]∖S, a vector v∈{0,1}ⁿ is said to satisfy (i,S) if v(i)=1 and v(j)=0 for all j∈S.

For each i∈[n], we create a set T(i)⊆[n] according to the following greedy algorithm:

• 1. Initialize T(i) with the empty set.

• 2. Let S⊆[n]∖(T(i)∪{i}) be any set of size at most d′ such that the number of rows of M that satisfy (i,S) is at most e′. If there is no such S, terminate.

• 3. Set T(i):=T(i)∪S, and go to step 2.

First, we argue that the above algorithm always terminates after looping at most ℓ times. Suppose for the sake of contradiction that the algorithm loops more and let S ₁,…,S _ℓ+1 be the disjoint sets S obtained in the first ℓ+1 iterations of the loop. Let M′ be the matrix obtained from M by removing all the rows where the ith column has a zero, and define T′:=S ₁∪⋯∪S _ℓ+1.

By the way the algorithm chooses the sets S _j , we know for each S _j that all but at most e′ rows of $M'|_{S_{j}}$ have nonzero weights. Therefore, all but at most e′(ℓ+1)=e rows of M′| _T have weights at least ℓ+1 (i.e., at least one nonzero entry for the range of each S _j ).

On the other hand, since |S _j |≤d′ for all j, we have |T′|≤d′(ℓ+1)≤d. So, the property of Lemma 11 implies that there are more than e rows of M′ where M′| _T′ has weight exactly ℓ. This is a contradiction. Therefore, we conclude, for every i, that |T(i)|≤ℓd′<d.

Now, define an undirected graph G=(V,E) where V:=[n] and {i,j}∈E iff either j∈T(i) or i∈T(j). We know from the upper bound on the size of every T(i) that the maximum degree of this graph is less than 2d. Therefore, the graph has an independent set V′⊆V of size at least n/(2d). Let M″:=M| _V′, with columns indexed by the elements of V′.

Now, consider any i∈V′ and any set S⊆V′∖i where |S|=d′. Since V′ is an independent set of G, we know that T(i)∩V′=∅. Since the greedy algorithm, given input i, has terminated at step 2, we know that there are more than e′ rows of M″ that satisfy (i,S) (otherwise the algorithm would add S to T(i) and loop another time). Since this holds for every choice of (i,S), we conclude that the matrix M″ must be a classical (d′,e′)-disjunct matrix.

Let n′ be the number of columns of M″, so we know that n′≥n/(2d). Now it suffices to apply the known asymptotic lower bounds for the number of rows of classical disjunct matrices [19, 37, 39] on M″. In particular, Theorem 2.19 of [39] implies that, for some absolute constant c>0, and whenever n is sufficiently large for the given parameter d,

which implies the claimed bound assuming n is large enough. □

Corollary 13

For every integer d>0 there is an n ₀>0 such that the following holds. For any n≥n ₀, let M be an m×n Boolean matrix that is a (d,e;u)-threshold design, for some constant u>0. Then,

$$m = \varOmega_u(d^2 \log_d n + de). $$

Proof

Immediate from Lemma 11 and Theorem 12. □

3 Constructions

In this section, we obtain several construction of regular and disjunct matrices. Our first construction, described in Construction 2, is a randomness-efficient probabilistic construction that can be analyzed using standard techniques from the probabilistic method. The bounds obtained by this construction are given in Lemma 14 below. The amount of random bits required by this construction is polynomially bounded in d and logn, which is significantly smaller than it would be had we picked the entries of M fully independently.

Construction 2

Probabilistic construction of regular and disjunct matrices

Lemma 14

For every p∈[0,1) and integer parameter u>0, Construction  2 with m′=O _u (dlog(n/d)/(1−p)²) (resp., m′=O _u (d ²log(n/d)/(1−p)²)) outputs a (d,Ω _u (pm′);u)-regular (resp., threshold (d,Ω _u (pm′/d);u)-disjunct) matrix with probability 1−o(1).

Proof

We show the claim for regular matrices, the proof for disjunct matrices is similar. Consider any particular choice of a critical set S⊆[n] and a zero set Z⊆[n] such that u≤|S|≤d and |Z|≤|S|. Choose an integer i so that 2ⁱ⁻¹ u≤|S|≤2ⁱ u, and take any j∈[m′]. Denote the (i,j)th row of M by the random variable w∈{0,1}ⁿ, and by q the “success” probability that w| _S has weight exactly u and w| _Z is all zeros. For an integer r>0, we will use the shorthand 1^r (resp., 0^r) for the all-ones (resp., all-zeros) vector of length r. We have

(3)

where $\mathrm{(a)}$ and $\mathrm{(b)}$ use the fact that the entries of w are (u+1)-wise independent, and $\mathrm{(b)}$ uses an additional union bound. Here the lower bound c>0 is a constant that only depends on u. Now, let e:=m′pq. using Chernoff bounds, and independence of the rows, the probability that there are at most e rows (among (i,1),…,(i,m′)) whose restrictions to S and Z have weights u and 0, respectively, becomes upper bounded by

$$\exp( -(m'q-e)^2/(2m'q) ) = \exp( -(1-p)^2 m' q/2 ) \leq\exp( -(1-p)^2 m' c/2 ). $$

Now take a union bound on all the choices of S and Z to conclude that the probability that the resulting matrix is not (d,e;u)-regular is at most

$$\left(\sum_{s=u}^{d} \binom{n}{s} \sum_{z=0}^{s} \binom {n-s}{z}\right) \exp( -(1-p)^2 m' c/2 ), $$

which can be made o(1) by choosing m′=O _u (dlog(n/d)/(1−p)²).

The proof of the claim for disjunct matrices follows along the same lines, except that we additionally need the vector w to be 1 at the position corresponding to the distinguished column i. Under this additional requirement, the lower bound on q would become Ω _u (1/d), and this only increases the number of rows by a factor O _u (d). □

A significant part of this work is a construction of regular matrices using strong lossless condensers. Details of the construction are described in Construction 4 that assumes a family of lossless condensers with different entropy requirements,⁴ and in turn, uses Construction 3 as a building block. The theorem below analyzes the obtained parameters without specifying any particular choice for the underlying family of condensers.

Construction 3

A building block for construction of regular matrices

Theorem 15

The m×n matrix M output by Construction  4 is (d,pγ2^t;u)-regular, where $\gamma= \max\{1, \varOmega_{u}(d \cdot\min\{2^{k(i)-{\tilde {\ell }}(i)} \colon i=0,\ldots,r \}) \}$.

Construction 4

Regular matrices from strong lossless condensers

Proof

As a first step, we verify the upper bound on the number of measurements m. Each matrix M _i has $m_{i} = 2^{t+k(i)} O_{u}(2^{u({\tilde {\ell }}(i)-k(i))})$ rows, and $M'_{i}$ has m _i r _i rows, where r _i =2^r−i. Therefore, the number of rows of M is

$$\sum_{i=0}^r r_i m_i = \sum_{i=0}^r 2^{t+\log u' + r+1} m_i = 2^t d \sum_{i=0}^r O_u(2^{u({\tilde {\ell }}(i)-k(i))}). $$

Let $S, Z \subseteq \{0,1\}^{{\tilde {n}}}$ respectively denote any choice of a critical set and zero set of size at most d, where |Z|≤|S|, and choose an integer i≥0 so that 2ⁱ⁻¹ u′≤|S|≤2ⁱ u′. Arbitrarily grow the two sets S and Z to possibly larger, and disjoint, sets S′⊇S and Z′⊇Z such that |S′|=|Z′|=2ⁱ u′ (for simplicity we have assumed that d≤n/2). Our goal is to show that there are “many” rows of the matrix M _i (in Construction 4) that u-satisfy S and Z.

Let k:=k(i)=logu′+i+1, ${\tilde {\ell }}:= {\tilde {\ell }}(i)$, and denote by G ₁,G ₂,G ₃ the bipartite graphs used by the instantiation of Construction 3 that outputs M _i . Thus we need to show that “many” right vertices of G ₃ are each connected to exactly u of the vertices in S and none of those in Z.

Consider the uniform distribution $\mathcal {X}$ on the set S′∪Z′, which has min-entropy logu′+i+1. By an averaging argument, since the condenser f _i is strong, for more than a p fraction of the choices of the seed y∈{0,1}^t (call them good seeds), the distribution $\mathcal {Z}_{y} := f_{i}(\mathcal {X}, y)$ is ϵ/(1−p)-close (in particular, (1/32)-close) to a distribution with min-entropy logu′+i+1.

Fix any good seed y∈{0,1}^t. Let $G=(\{0,1\}^{{\tilde {n}}}, \{0,1\}^{{\tilde {\ell }}}, E)$ denote a bipartite graph representation of f _i , where each left vertex $x \in \{0,1\}^{{\tilde {n}}}$ is connected to f _i (x,y) on the right. Denote by Γ _y (S′∪Z′) the right vertices of G corresponding to the neighborhood of the set of left vertices picked by S′∪Z′. Note that $\varGamma_{y}(S' \cup Z') = \mathsf {supp}(\mathcal {Z}_{y})$. Using Proposition 2, we see that since $\mathcal {Z}_{y}$ is (1/32)-close to having min-entropy log(|S′∪Z′|), there are at least (7/8)|S′∪Z′| vertices in Γ(S′∪Z′) that are each connected to exactly one left vertex in S′∪Z′. Since |S|≥|S′∪Z′|/4, this implies that at least |S′∪Z′|/8 vertices in Γ(S′∪Z′) (call them $\varGamma'_{y}$) are connected to exactly one left vertex in S and no other vertex in S′∪Z′. In particular we get that $|\varGamma'_{y}| \geq2^{k-3}$.

Now, in G ₁, let T _y be the set of left vertices corresponding to $\varGamma'_{y}$ (regarding the left vertices of G ₁ in one-to-one correspondence with the right vertices of G). The number of edges going out of T _y in G ₁ is d _ℓ |T _y |≥u2^k. Therefore, as the number of the right vertices of G ₁ is 2^k, there must be at least one right vertex that is connected to at least u vertices in T _y . Moreover, a counting argument shows that the number of right vertices connected to u or more vertices in T _y is at least $2^{k-{\tilde {\ell }}} 2^{k}/(10u)$.

Observe that in construction of G ₂ from G ₁, any right vertex of G ₁ is replicated $\binom{d_{r}}{u}$ times, one for each u-subset of its neighbors. Therefore, for a right vertex of G ₁ that is connected to at least u left vertices in T _y , one or more of its copies in G ₂ must be connected to exactly u vertex in T _y (among the left vertices of G ₂) and no other vertex (since the right degree of G ₂ is equal to u).

Define $\gamma' := \max\{1, 2^{k-{\tilde {\ell }}} 2^{k}/(10u)\}$. From the previous argument we know that, looking at T _y as a set of left vertices of G ₂, there are at least γ′ right vertices on the neighborhood of T _y in G ₂ that are connected to exactly u of the vertices in T _y and none of the left vertices outside T _y . Letting v _y be any such vertex, this implies that the vertex (y,v _y )∈V ₃ on the right part of G ₃ is connected to exactly u of the vertices in S, and none of the vertices in Z. Since the argument holds for every good seed y, the number of such vertices is at least the number of good seeds, which is more than pγ′2^t. Since the rows of the matrix m _i are repeated r _i =2^r−i times in M, we conclude that M has at least pγ′2^t+r−i≥pγ2^t rows that u-satisfy S and Z, and the claim follows. □

3.1 Instantiations

We now instantiate the result obtained in Theorem 15 by various choices of the family of lossless condensers. The crucial factors that influence the number of measurements are the seed length and the output length of the condenser.

Non-constructively, it can be shown that strong (k,ϵ) lossless condensers with input length ${\tilde {n}}$, seed length $t= \log {\tilde {n}}+ \log(1/\epsilon ) + O(1)$, and output length ${\tilde {\ell }}= k+\log(1/\epsilon )+ O(1)$ exist, and moreover, almost matching lower bounds are known [4]. In fact, the optimal parameters can be achieved by a random function with overwhelming probability. In this work, we consider two important explicit constructions of lossless condensers. Namely, one based on “zig-zag products” due to Capalbo et al. [4] and another, coding theoretic, construction due to Guruswami et al. [27].

Theorem 16

[4]

For every $k\leq {\tilde {n}}\in \mathbb {N}$, ϵ>0 there is an explicit lossless (k,ϵ) condenser with seed length $O(\log^{3} ({\tilde {n}}/\epsilon ))$ and output length k+log(1/ϵ)+O(1).

Theorem 17

[27]

For all constants α∈(0,1) and every $k\leq {\tilde {n}}\in \mathbb {N}$, ϵ>0 there is an explicit strong lossless (k,ϵ) condenser with seed length $t=(1+1/\alpha) \log({\tilde {n}}k/\epsilon ) + O(1)$ and output length ${\tilde {\ell }}=t+(1+\alpha)k$.

As a result, we use Theorem 15 with the above condensers to obtain the following.

Theorem 18

Let u>0 be fixed, and p∈[0,1) be a real parameter. Then for integer parameters d,n∈ℕ where u≤d≤n,

• 1. Using an optimal lossless condenser in Construction  4 results in an m ₁×n matrix M ₁ that is (d,e ₁;u)-regular, where m ₁=O(d(logn)(logd)/(1−p)^u+1) and e ₁=Ω(pdlogn).

• 2. Using the lossless condenser of Theorem 16 in Construction  4 results in an m ₂×n matrix M ₂ that is (d,e ₂;u)-regular, where m ₂=O(T ₂ d(logd)/(1−p)^u) for some T ₂=exp(O(log³((logn)/(1−p))))=quasipoly(logn), and e ₂=Ω(pdT ₂(1−p)).

• 3. Let β>0 be any fixed constant. Then Construction  4 can be instantiated using the lossless condenser of Theorem 17 so that we obtain an m ₃×n matrix M ₃ that is (d,e ₃;u)-regular, where $m_{3} = O(T_{3}^{1+u} d^{1+\beta} (\log d))$ for T ₃:=((logn)(logd)/(1−p))^1+u/β=poly(logn,logd), and e ₃=Ω(pmax{T ₃,d ^1−β/u}).

Proof

First we show the claim for M ₁. In this case, we take each f _i in Construction 4 to be an optimal lossless condenser satisfying the (non-constructive) bounds obtained in⁵ [4]. Thus we have that $2^{t}= O({\tilde {n}}/\epsilon )=O(\log n/\epsilon )$, and for every i=0,…,r, we have $2^{{\tilde {\ell }}(i)-k(i)} = O(1/\epsilon )$, where ϵ=O(1−p). Now we apply Theorem 15 to obtain the desired bounds (and in particular, γ=Ω(ϵd)).

Similarly, for the construction of M ₂ we set up each f _i with the explicit construction of condensers in Theorem 16 for min-entropy k(i). In this case, the maximum required seed length is $t = O(\log^{3}({\tilde {n}}/\epsilon ))$, and we let T ₂:=2^t=exp(O(log³((logn)/(1−p)))). Moreover, for every i=0,…,r, we have $2^{{\tilde {\ell }}(i)-k(i)} = O(1/\epsilon )$. Plugging these parameters in Theorem 15 gives γ=Ω(ϵd) and the bounds on m ₂ and e ₂ follow.

Finally, for M ₃ we use Theorem 17 with α:=β/u. Thus the maximum seed length becomes $t = (1+u/\beta) \log({\tilde {n}}(\log d)/(1-p)) + O(1)$, and for every i=0,…,r, we have ${\tilde {\ell }}(i)-k(i) = O(t+\beta(\log d)/u)$. Clearly, T ₃=Θ(2^t), and thus (using Theorem 15) the number of measurements becomes m ₃=T ^1+u d ^1+β(logd). Moreover, we get γ=max{1,Ω(d ^1−β/u/T)}, which gives e ₃=Ω(pTγ)=pmax{T,d ^1−β/u}, as claimed. □

By combining this result with Lemma 9 using any explicit construction of classical disjunct matrices, we obtain threshold (d,e;u)-disjunct matrices that can be used in the threshold model with any fixed threshold, sparsity d, and error tolerance ⌊e/2⌋. In particular, using the coding-theoretic explicit construction of nearly optimal classical disjunct matrices from codes on the Gilbert-Varshamov bound [34] (Theorem 30 in the appendix), we obtain threshold (d,e;u)-disjunct matrices with m=O(m′d ²(logn)/(1−p)²) rows and error tolerance e=Ω(e′pd(logn)/(1−p)), where m′ and e′ are respectively the number of rows and error tolerance of any of the regular matrices obtained in Theorem 18. We note that in all cases, the final dependence on the sparsity parameter d is, roughly, O(d ³) which has an exponent independent of the threshold u. Rows M7–M9 of Table 1 summarize the obtained parameters for the general case (with arbitrary gaps). We see that, when d is not negligibly small (e.g., d=n ^1/10), the bounds obtained by our explicit constructions are significantly better than those offered by strongly disjunct matrices.

4 The Case with Positive Gaps

In preceding sections we have focused on the case where g=0. However, in this section we observe that all the techniques that we have developed in this work can be extended to the positive-gap case in a straightforward way. The main observations are listed below. Recall from [15] that in the positive-gap case, we can only hope to distinguish between distinct d-sparse vectors x and x′ where at least one has support size u or more and either |supp(x)∖supp(x′)|>g or |supp(x′)∖supp(x)|>g. We will call any pair of such vectors distinguishable. Moreover, we naturally extend the Definition 4 of threshold designs to the positive-gap case as follows.

Definition 19

(Definition 4, generalized)

Let n≥d≥u>0 and g∈[0,u), and e≥0 be integer parameters, and define ℓ:=u−g−1. A Boolean matrix M with n columns is said to be a (d,e;u,g)-threshold design if for every d-sparse x,x′∈{0,1}ⁿ of Hamming weight u or more such that |supp(x)∖supp(x′)|>g, every y∈M[x] _ℓ,u and every y′∈M[x′] _ℓ,u , we have dist(y,y′)>e.

4.1 Generalized Threshold Disjunct Matrices

For the positive-gap case, Definition 6 of threshold disjunct matrices can be adapted to allow more than one distinguished column in disjunct matrices. In particular, in general we may require the matrix M to have more than e rows that u-satisfy every choice of a critical set S, a zero set Z, and any set of g+1 designated columns I⊆S (at which all entries of the corresponding rows must be 1). Denote this generalized notion by threshold (d,e;u,g)-disjunct matrices. It is straightforward to extend the arguments of Lemma 7 to show that the generalized notion of threshold (d,e;u,g)-disjunct matrices suffices to capture non-adaptive threshold group testing with upper threshold u and gap g. More precisely, the generalized definitions of threshold disjunct and regular matrices are as follows.

Definition 20

(Definition 6, generalized)

Let n,d,e,u,g be non-negative integers where g<u≤d≤n. A Boolean matrix M with n columns is called threshold (d,e;u,g)-disjunct if for every subset of columns S⊆[n] (called the critical set), every Z⊆[n] (called the zero set) such that u≤|S|≤d, |Z|≤|S|, S∩Z=∅, and every set I⊆S of g+1 distinguished columns (|I|=g+1), there are more than e rows of M that u-satisfy S and Z and moreover, M| _I has all ones at those columns. Moreover, M is called (d,e;u,g)-regular if for every choice of the critical and zero sets S,Z⊆[n] with |Z|≤|S|+g, there is a set of more than e rows of M that (u−g)-satisfy S and Z.

Note the slight difference between the notion of regular matrices above compared to Definition 6, namely, that the zero set Z can now be slightly larger than the critical set S (by at most u), and that the matrix is now required to (u−g)-satisfy (as opposed to u-satisfy) every choice of S and Z. The two notions coincide for g=0. In general, the difference between the two notions of regular matrices is negligible as long as the parameter g remains small. In particular, it is straightforward to verify that all our results about the construction of regular matrices in the gap-free case (Constructions 2 and 4) as well as the obtained bounds (Lemma 14, Theorems 15 and 18) hold for the generalized notion of regular matrices with only a slight effect on the hidden terms that only depend on the threshold parameter u. We will see, however, that the generalized notion of threshold disjunct matrices is stronger than Definition 6 and the extra requirements may substantially affect the bounds (but not the construction techniques).

Below we show that the generalized notion of threshold disjunct matrices suffices for construction of threshold designs for the positive-gap case.

Lemma 21

(Lemma 7, generalized)

Let M be an m×n Boolean matrix that is threshold (d,e;u,g)-disjunct, and define ℓ:=u−g−1. Then for every distinguishable d-sparse vectors x,x′∈{0,1}ⁿ, each having support size u or more and such that |supp(x)∖supp(x′)|>g and wgt(x)≥|supp(x′)∖supp(x)|, the following holds. Let y∈M[x] _ℓ,u and y′∈M[x′] _ℓ,u . Then,

$$ |\mathsf {supp}(y) \setminus \mathsf {supp}(y')| > e. $$

(4)

Moreover, M is a (d,e;u,g)-threshold design. Conversely, if M satisfies (4) for every choice of x, x′, y, y′ as above, it must be threshold (⌊d/2⌋,e;u,g)-disjunct (assuming n>d+g).

Proof

First, suppose that M is threshold (d,e;u,g)-disjunct, and let y∈M[x] _ℓ,u and y′∈M[x′] _ℓ,u be arbitrarily chosen. Take any I⊆supp(x)∖supp(x′) of size g+1, and let S:=supp(x) and Z:=supp(x′)∖supp(x). Note that |S|≤d and by assumption, we have |Z|≤|S|. Now, Definition 20 implies that there is a set E of more than e rows of M that u-satisfy I as the set of distinguished columns, S as the critical set and Z as the zero set. Thus for every j∈E, the jth row of M restricted to the columns chosen by supp(x) must have weight exactly u, while its weight on supp(x′) is at most u−g−1=ℓ. Therefore, y(j)=1 and y′(j)=0 for more than e choices of j.

The claim that M is a (d,e;u,g)-threshold design follows from the above argument combined with the observation that, given any two d-sparse distinguishable vectors, having Hamming weight u or more, at least one of their two possible orderings satisfies the conditions required by the lemma.

For the converse, consider any choice of a set of distinguished columns I⊆[n] with |I|=g+1, a critical set S⊆[n] containing I (such that |S|≥u), and a zero set Z⊆[n] where |Z|≤|S|. Define d-sparse Boolean vectors x,x′∈{0,1}ⁿ so that supp(x):=S and supp(x′):=Z∪(S∖I). We note that wgt(x)=|supp(x)|≥u and also, without loss of generality, wgt(x′)≥u (if the latter is not the case, we can simply enlarge Z by arbitrarily adding up to g+1 elements outside the support of S to it and observe that is suffices to show the claim for the larger Z).

Let y:=M[x] _ℓ+1 and y′:=M[x′] _u , and observe that y,y′∈M[x] _ℓ,u . Moreover, let E:=supp(y)∖supp(y′). By assumption we know that |E|>e. Take any j∈E. Since y(j)=1 and y′(j)=0, we get that the jth row of M restricted to the columns picked by Z∪(S∖I) must have weight at most ℓ=u−(g+1), whereas it must have weight at least u when restricted to S. As the sets I,S∖I, and Z are disjoint and |I|=g+1, this can hold only if the jth row of M restricted to the columns picked by S, Z, and I has weights exactly u, 0, and g+1, respectively. Hence, this row (as well as all the rows of M picked by E) must u-satisfy I,S, and Z, confirming that M is threshold (⌊d/2⌋,e;u,g)-disjunct. □

4.2 Strongly Disjunct Versus Threshold Disjunct Matrices

The following proposition directly follows from the definitions, and relates strongly disjunct matrices to generalized threshold disjunct matrices.

Proposition 22

Let n,d,e,u,g be non-negative integers where g<u≤d≤n−(d+g+1). Suppose that M and M′ are binary m×n matrices, where M is threshold (d,e;u,g)-disjunct and M′ is strongly (2d,e;u)-disjunct. Then, M is strongly (d,e;g+1)-disjunct and M′ is threshold (d,e;u,g)-disjunct.

Proof

First, we verify the conditions of Definition 3 for M. Consider any pair of disjoint sets I,Z⊆[n] where |I|=g+1 and |Z|≤d. Let S⊆[n] be any set of size d containing I and disjoint from Z. Note that |Z|≤|S|. From Definition 20 (with the critical set S, zero set Z, and distinguished set I), there is a set of more than e rows of M at which M| _Z is all zeros and M| _I is all ones. In other words, denoting the ith column of M by C _i , we have that

$$\bigg|\bigcap_{i \in I} \mathsf {supp}(C_i) \setminus\bigcup_{i \in Z} \mathsf {supp}(C_i) \bigg| > e, $$

as required by Definition 3.

Now consider the matrix M′ and any choice of a I,S,Z as in Definition 20. Let J⊆S be any subset of S of size u that contains I, and S′:=Z∪(S∖J). Note that |S′|≤|S|+|Z|≤2d. Now from Definition 3 of strongly disjunct matrices, we know that

$$\bigg|\bigcap_{i \in J} \mathsf {supp}(C_i) \setminus\bigcup_{i \in S'} \mathsf {supp}(C_i) \bigg| > e. $$

In other words, there is a set of more than e rows of M′ at which M′| _I is all ones, M′| _S has weight exactly u, and M′| _Z is all zeros, as required by Definition 20. □

The special case u=g+1 in the above proposition is particularly interesting. A chain of reductions between strongly disjunct and threshold disjunct matrices in this case implied by the above result is schematically shown below.

$$\begin{array}{c} \text{threshold-$(2d,e;g+1,g)$-disjunct} \\ \downarrow\\ \text{strongly $(2d,e;g+1)$-disjunct} \\ \downarrow\\ \text{threshold-$(d,e;g+1,g)$-disjunct} \\ \downarrow\\ \text{strongly $(d,e;g+1)$-disjunct} \end{array} $$

Therefore, when the upper threshold u is more than the gap parameter g by one (equivalently, when the lower threshold ℓ is zero), the two notions of threshold disjunct matrices and strongly disjunct matrices become equivalent up to a multiplicative factor in the sparsity parameter d. As discussed in Sect. 2, almost matching lower bounds and upper bounds on the number of rows m achievable by a strongly (d,e;g+1)-disjunct matrix are known. Asymptotically, the number of rows must always satisfy m=Ω(d ^g+2log _d n+ed ^g+1) and moreover, a probabilistic construction achieves m=O(d ^g+2log(n/d)) and e=Ω(dlog(n/d)) with probability 1−o(1) (see Table 1). As a result, the upper and lower bounds on the number of rows of strongly disjunct and threshold disjunct matrices become equivalent up multiplicative constants when the lower threshold ℓ is zero.

Proposition 22 asserts that the notion of strongly disjunct matrices is in general stronger than threshold disjunct matrices. As we will see below, the former becomes strictly stronger when ℓ>0. As the lower threshold ℓ becomes larger, the discrepancy between the number of rows achievable by threshold disjunct matrices and strongly disjunct matrices becomes more significant (see Table 1).

4.3 Probabilistic Upper Bounds

As pointed out after Definition 20, the generalized definition of regular matrices may affect the bounds obtained by our probabilistic and explicit constructions (Constructions 2 and 4) only by hidden factors depending on u (essentially without any change in the proofs). For the case of generalized disjunct matrices, however, the bounds may substantially change depending on the gap parameter g.

Below we generalize Lemma 14 for the case of threshold disjunct matrices and show that Construction 2 results in a threshold (d,Ω _u (pdlog(n/d)/(1−p)²);u,g)-disjunct matrix (with probability 1−o(1)) if the number of measurements is increased by a factor O(d ^g). More precisely, we can now show the following lemma.

Lemma 23

(Lemma 14, generalized)

For every p∈[0,1) and integer parameters u>g≥0, Construction  2 with m′=O _u (d ^g+2log(n/d)/(1−p)²) outputs a threshold (d,Ω _u (pm′/d ^g+1);u,g)-disjunct matrix with probability 1−o(1).

Proof

The proof essentially follows along the same lines as the proof of Lemma 14. The difference, compared to the case g=0 covered by Lemma 14, is that we have a set I of distinguished columns I⊆[n] in Definition 20 where |I|=g+1 and the random vector w in the proof of Lemma 14 must have ones at all positions picked by I. With this requirement, the lower bound on the success probability q in (3) becomes c=Ω _u (1/d ^g+1). The rest of the proof remains unchanged except for the new lower bound on c, which makes the error tolerance parameter e in the proof lower bounded by Ω(pm′/d ^g+1), while increasing the parameter m′ to a quantity upper bounded by O _u (d ^g+2log(n/d)/(1−p)²). □

4.4 The Direct Product

Lemma 9 can be extended to positive gaps as follows.

Lemma 24

(Lemma 9, generalized)

Suppose that M ₁ is a (d,e ₁;u,g+1)-regular and M ₂ is a strongly (2d,e ₂;g+1)-disjunct matrix. Then M ₁⊙M ₂ is a threshold (d,(e ₁+1)(e ₂+1)−1;u,g)-disjunct matrix.

Proof

Let ℓ:=u−g−1 and M:=M ₁⊙M ₂. Towards verifying that M satisfies the requirements of Definition 20, consider a set I⊆[n] of distinguished columns of M, where n is the number of columns of the matrices and |I|=g+1, in addition to critical and zero sets S,Z⊆[n] as in Definition 20 satisfying |Z|≤|S|. Index the rows of M naturally by the elements of [m ₁]×[m ₂], where m ₁ and m ₂ are the number of rows of M ₁ and M ₂, respectively, and the (i,j)th row of M is the bitwise disjunction of the ith row of M ₁ and the jth row of M ₂.

Let S′:=S∖I and Z′:=Z∪(S∖I). Observe that |Z|≤|S|≤|S′|+g+1=|S|+|I| and |Z′|≤2d. From Definition 20, there is a set E ₁⊆[m ₁] of size more than e ₁ such that M ₁| _S′ has weight exactly ℓ=u−|I|, and M ₂| _Z is all zeros. Moreover, there is a set E ₂⊆[m ₂] of size more than e ₂ at which M ₂| _I has all ones and M ₂| _Z′ has all zeros. This means that, at all rows corresponding to E ₁×E ₂, the product matrix M has weight exactly ℓ+|I|=u at positions corresponding to S and all zeros at positions corresponding to Z. Therefore, M indeed u-satisfies any choice of the sets I, S, Z at more than (e ₁+1)(e ₂+1)−1 rows. □

Consequently, using the coding-theoretic construction of strongly disjunct matrices described in Sect. 5, our explicit constructions of threshold (d,e;u)-disjunct matrices obtained in Sect. 3 can be extended to the positive gap model at the cost of a factor O(d ^g) increase in the number of measurements. The results from combining the above lemma with various constructions of regular and strongly disjunct matrices are summarized in Table 1.

4.5 Lower Bounds

We now extend the lower bounds proved in Sect. 2.3 to the positive-gap case, and show that the optimal exponent of d in the number of measurements is g+1.

The lower bound on the number of rows of threshold disjunct matrices is an immediate consequence of Proposition 22.

Theorem 25

For every integer d>0 there is an n ₀>0 such that the following holds. For any n≥n ₀, let M be an m×n threshold (d,e;u,g)-disjunct matrix. Then, for some absolute constant c>0,

Proof

Immediate from combining Proposition 22 and Theorem 2.19 of [39] that proves a lower bound on the number of rows of strongly disjunct matrices. The asymptotic simplification is straightforward. □

In order to lower bound the number of measurements in a threshold design, we first generalize Lemma 11 as follows.

Lemma 26

Let M be a (d+g,e;u,g−1)-threshold design, and ℓ:=u−(g−1)−1=u−g. be the lower threshold Then M satisfies the following property:

“For every S⊆[n] such that |S|=d and every T⊆[n]∖S such that |T|=g, there are more than e rows of M at which M| _T consists of all ones and moreover in those rows, M| _S has weight exactly ℓ.”

Proof

Let D:=d+g be the sparsity parameter in the threshold model that M is designed for. In order to verify the claimed property for a given choice of S and T, consider the D-sparse vectors x,x′∈{0,1}ⁿ such that supp(x)=S∪T and supp(x′)=S. Let y:=M[x] _ℓ+1 and y′:=M[x′] _u , and observe that y,y′∈M[x] _ℓ,u . Also, since x′ is point-wise less than or equal to x, or in symbols x′⪯x, by monotonicity it also follows that y′⪯y.

Thus, we know from the assumption that there are more than e positions at which y and y′ are different. Let j be any such position. In particular, we know that y(j)=1 and y′(j)=0. Therefore, by Definition 19 and the way that the threshold model is defined, the submatrix M| _supp(x′) must have weight at most ℓ at the jth row and M| _supp(x) must have weight at least u at the jth row. Since the support of x is chosen to be larger than the support of x′ at exactly g positions, and g=u−ℓ, the only possibility is to have M| _supp(x) (that is, M| _S∪T ) with weight exactly u at the jth row and M| _{supp(x)∖supp(x′)} (that is, M| _T ) having all ones at the jth row. In turn, this implies that M| _supp(x′) (that is, M| _S ) must have weight exactly ℓ at that row.

This concludes proof, since the argument holds for every possible choice of the distinguishing entry j. □

The following theorem is the analogous version of Theorem 12 for the positive-gap case.

Theorem 27

For every integer d>0 there is an n ₀>0 such that the following holds. For any n≥n ₀, let M be an m×n Boolean matrix that satisfies the property quoted in Lemma 26. Then,

In particular, when ℓ,g are absolute constants, we have

$$m = \varOmega_{\ell,g}\biggl( \frac{1}{\log d+\log(e+2)} \cdot\bigg( \frac{d^{g+1}}{\log d} + \frac{d^g e}{\log n} \bigg) + d^2 \log_d n + de \biggr). $$

Proof

First, observe that the property quoted in Lemma 26 is stronger than the property quote in Lemma 11. Thus, the lower bound of Theorem 12 holds; namely, we have

$$m = \varOmega\biggl( \bigg(\frac{d}{\ell+1}\bigg)^2 \log_d n + \frac {de}{(\ell+1)^2}\biggr). $$

Therefore, it suffices to show that m=Ω(m′/logm′). Given the matrix M, we will use a “random resampling method” to create a strongly disjunct matrix out of M, and will then use the known lower bounds related to strongly disjunct matrices.

Given a vector v∈{0,1}ⁿ, a resampling of v is a random vector v′∈{0,1}ⁿ defined in the following way: For ever i∈[n], if v(i)=0, then we set v′(i)=0. Otherwise, we independently set v′(i)=1 with probability 1/ℓ and zero with the remaining probability.

Let t>0 be an integer parameter to be determined later. Let M ₁,…,M _t be random m×n Boolean matrices, such that each M _i is obtained from M by independently resampling each row of the matrix. Also, define the mt×n Boolean matrix M′ by stacking M ₁,…,M _t on top of one another. We will argue that, if t is chosen sufficiently large, there is a nonzero probability that M′ becomes a strongly disjunct matrix. Thus, there is a fixing of the resampling randomness that indeed makes M′ strongly disjunct, which then allows us to obtain the desired lower bound.

Consider any triple (j,T,W) where j∈[m], T,W⊆[n] such that |T|=g, |W|=ℓ, T∩W=∅, and moreover, the jth row of M has all-ones at the columns picked by T and W. We say that the triple survives in M _i when on the jth row of M _i , the columns picked by T all contain ones and those picked by W all contain zeros. The probability of this happening is exactly

$$p = (1 - 1/\ell)^\ell\cdot1/\ell^g \geq c/\ell^g, $$

for some absolute constant c>0. The probability that the triple does not survive in any of M ₁,…,M _t is

$$(1-p)^t \leq(1-c/\ell^g)^t \leq C^{t/\ell^g}, $$

for some absolute constant C∈(0,1). Combined with a union bound on all choices of (j,T,W), we deduce that the probability that some triple does not survive in any of M ₁,…,M _t is at most

$$m n^{g+\ell} C^{t/\ell^g} = 2^{\log m + (g+\ell) \log n - (t/\ell ^g) \log(1/C)}, $$

which is strictly less than 1 for some large enough choice of t, namely, for t≥t ₀ such that

$$t_0 = O(\ell^g((g+\ell) \log n + \log m)). $$

Now, pick t:=t ₀ and fix the resampling randomness so that all triples (j,T,W) survive. We claim that the matrix M′ is strongly (d,e;g)-disjunct.

In order to verify the disjunctness property, consider any choice of sets S,T⊆[n] such that |S|=d and |T|=g. Let J be the set of rows of M where M| _T has all-ones and M| _S has Hamming weight ℓ. By the property quoted in Lemma 26, we know that |J|>e.

For any j∈J, we know that the jth row of M| _S is supported on some set W⊆[n] of size ℓ. We know, on the other hand, that the triplet (j,T,W) survives in some M _i . Clearly, by the way we defined the survival property, this implies that jth row of M _i (and thus, the corresponding row in M′) contains all-ones at columns picked by T and all-zeros at columns picked by S. Since this argument holds for any choice of S, T, and j, we conclude that M′ is strongly (d,e;g)-disjunct.

The number of rows of M′ is mt. We can now apply the known lower bounds on the number of rows of strongly disjunct matrices in order to lower bound the number of rows of M′. In particular, Theorem 2.19 of [39] implies that, for some absolute constant c′>0, and whenever n is sufficiently large for the given parameter d,

Now we substitute the chosen value of t in the above bound to obtain

Now if n is sufficiently large for the given d, we can ensure that the conditions of Proposition 31 in the appendix are satisfied, and the above bound implies that

$$m = \varOmega( m'/\log m' ),\quad \text{where } m' := \frac{(d/\ell g)^{g+1} \log_d n + (d/\ell g)^g e}{(g+\ell)\log n}, $$

as claimed. The simplification when ℓ and g are absolute constants is straightforward. □

Theorem 27 combined with Lemma 26 implies the desired lower bound on the number of measurements of a threshold design. The following corollary summarizes the simplified bounds for the case e=0.

Corollary 28

For every integer d>0 there is an n ₀>0 such that the following holds. For any n≥n ₀, let M be an m×n Boolean matrix that is a (d,0;u,g)-threshold design, for constants u>g≥0. Then,

$$m = \varOmega_u \bigg( \frac{d^{g+2}}{\log^2 d} + \frac{d^2 \log n}{\log d} \bigg). $$

5 Strongly Disjunct Matrices from Codes

A well known coding-theoretic construction of combinatorial designs, and classical disjunct matrices is due to Kautz and Singleton [29], which was further refined in several subsequent works (such as [21, 22]).

In this section we describe a construction of strongly disjunct matrices (as in Definition 3) which is a straightforward extension of the original construction of Kautz and Singleton. Construction 5 explains the idea, which is analyzed in Lemma 29 below. In this section we use standard tools from the theory of error-correcting codes.⁶ The interested reader is referred the standard texts in coding theory (e.g., the books by MacWilliams and Sloane [32], van Lint [42], and Roth [36]) for background.

Construction 5

Extension of Kautz-Singleton’s method [29]

Lemma 29

Construction  5 outputs a strongly (d,e;u)-disjunct matrix for every $d < ({\tilde {n}}- e)/(({\tilde {n}}-{\tilde {d}})u)$.

Proof

Let C:={c ₁,…,c _u }⊆[n] and $C' := \{ c'_{1}, \ldots, c'_{d} \} \subseteq[n]$ be disjoint subsets of column indices. We wish to show that, for more than e rows of M, the entries at positions picked by C are all-ones while those picked by C′ are all-zeros. For each j∈[n], denote the jth column of M′ by M′(j), and let M′(C):={M′(c _j ):j∈[u]}, and $M'(C') := \{ M'(c'_{j})\colon j \in[d] \}$.

From the minimum distance of $\mathcal {C}$, we know that every two distinct columns of M′ agree in at most ${\tilde {n}}- {\tilde {d}}$ positions. By a union bound, for each i∈[d], the number of positions where $M'(c'_{i})$ agrees with one or more of the codewords in M′(C) is at most $u({\tilde {n}}- {\tilde {d}})$, and the number of positions where some vector in M′(C′) agrees with one or more of those in M′(C) is at most $du({\tilde {n}}- {\tilde {d}})$.

By assumption, we have ${\tilde {n}}- du({\tilde {n}}- {\tilde {d}}) > e$, and thus, for a set $E \subseteq[{\tilde {n}}]$ of size greater than e, at positions picked by E none of the codewords in M′(C′) agree with any of the codewords in M′(C).

Now let w∈[q]ⁿ be any of the rows of M′ picked by E, and consider the q ^u×n Boolean matrix W formed by applying the mapping φ(⋅) on each entry of w. We know that $\{ w(c_{j})\colon j \in[u]\} \cap\{ w(c'_{j})\colon j \in[d]\} = \emptyset$. Thus we observe that the particular row of W indexed by (w(c ₁),…,w(c _u )) (and in fact, any of its permutations) must have all-ones at positions picked by C and all-zeros at those picked by C′. As any such row is a distinct row of M, it follows that M is strongly (d,e;u)-disjunct. □

Here we mention a few specific instantiations of the above construction. Namely, we will first consider the family of Reed-Solomon codes, that are also used in the original work of Kautz and Singleton [29], and then move on to the family of algebraic geometric (AG) codes on the Tsfasman-Vlăduţ-Zink (TVZ) bound, Hermitian codes, and finally, codes on the Gilbert-Varshamov (GV) bound.

5.1 Reed-Solomon Codes

Let p∈[0,1) be an arbitrary “noise” parameter. If we take $\mathcal {C}$ to be an $[{\tilde {n}}, k, {\tilde {d}}]_{{\tilde {n}}}$ Reed-Solomon code over an alphabet of size ${\tilde {n}}$ (which we assume to be a prime power), where ${\tilde {d}}= {\tilde {n}}-k+1$, we get a strongly disjunct (d,e;u)-matrix with m=O(dulogn/(1−p))^u+1 rows and $e = p {\tilde {n}}= \varOmega(p d u (\log n)/(1-p))$.

5.2 Algebraic Geometric Codes on the TVZ Bound

Another interesting family for the code $\mathcal {C}$ is the family of algebraic geometric codes that attain the Tsfasman-Vlăduţ-Zink bound (cf. [26, 41]). This family is defined over any alphabet size q≥49 that is a square prime power, and achieves a minimum distance ${\tilde {d}}\geq {\tilde {n}}-k-{\tilde {n}}/(\sqrt{q}-1)$. Let e:=pn, for a noise parameter p∈[0,1). By Lemma 29, the underlying code $\mathcal {C}$ needs to have minimum distance at least ${\tilde {n}}(1-(1-p)/(du))$. Thus in order to be able to use the above-mentioned family of AG codes, we need to have q≫(du/(1−p))²=:q ₀. Let us take an appropriate q∈[2q ₀,8q ₀], and following Lemma 29, ${\tilde {n}}- {\tilde {d}}= \lceil {\tilde {n}}(1-p)/(du) \rceil$. Thus, the dimension of $\mathcal {C}$ becomes at least

$$k\geq {\tilde {n}}- {\tilde {d}}- \frac{{\tilde {n}}}{\sqrt{q}-1} = \varOmega\left( \frac{{\tilde {n}}(1-p)}{du} \right) = \varOmega({\tilde {n}}/ \sqrt{q_0}), $$

and subsequently⁷ we get that $\log n = k\log q \geq k= \varOmega({\tilde {n}}/ \sqrt{q_{0}})$. Now, noting that $m = q^{u} {\tilde {n}}$, we conclude that

$$m = q^u {\tilde {n}}= O(q_0^{u+1/2} \log n) = O\left(\frac{du}{1-p} \right)^{2u+1} \log n, $$

and e=Ω(pdu(logn)/(1−p)).

We see that the dependence of the number of measurements on the sparsity parameter d is worse for AG codes than Reed-Solomon codes by a factor d ^u, but the construction from AG codes benefits from a linear dependence on logn, compared to log^u+1 n for Reed-Solomon codes. Thus, AG codes become more favorable only when the sparsity is substantially low; namely, when d≪logn.

5.3 Hermitian Codes

A particularly nice family of AG codes arises from the Hermitian function field. Let q′ be a prime power and q:=q′². Then the Hermitian function field over $\mathbb {F}_{q}$ is a finite extension of the rational function field $\mathbb {F}_{q}(x)$, denoted by $\mathbb {F}_{q}(x,y)$, where we have y ^q′+y=x ^q′+1. The structure of this function field is relatively well understood and the family of Goppa codes defined over the rational points of the Hermitian function field is known as Hermitian codes. This family is recently used by Ben-Aroya and Ta-Shma [1] for construction of small-bias sets. Below we quote some parameters of Hermitian codes from their work.

The number of rational points of the Hermitian function field is equal to q′³+1, which includes a common pole Q _∞ of x and y. The genus of the function field is $\tilde {g}= q'(q'-1)/2$. For some integer parameter r, we take G:=rQ _∞ as the divisor defining the Riemann-Roch space $\mathcal {L}(G)$ of the code $\mathcal {C}$, and the set of rational points except Q _∞ as the evaluation points of the code. Thus the length of $\mathcal {C}$ becomes ${\tilde {n}}= {q'}^{3}$. Moreover, the minimum distance of the code is ${\tilde {d}}= n-\deg(G) = n-r$. When $r \geq 2\tilde {g}- 1$, the dimension of the code is given by the Riemann-Roch theorem, which is equal to $r-\tilde {g}+1$. For the low-degree regime where $r < 2\tilde {g}-1$, the dimension k of the code is the size of the Wirestrauss semigroup of G, which turns out to be the set W={(i,j)∈ℕ²:j≤q′−1∧iq′+j(q′+1)≤r}.

Now, given parameters d,p of the disjunct matrix, define ρ:=(1−p)/((d+1)u), take the alphabet size q as a square prime power, and set r:=ρq ^3/2. First we consider the case where $r < 2\tilde {g}- 1 = 2q - 2\sqrt{q} - 1$. In this case, the dimension of the Hermitian code becomes k=|W|=Ω(r ²/q)=Ω(ρ ² q ²). The distance ${\tilde {d}}$ of the code satisfies ${\tilde {d}}= {\tilde {n}}- r \geq {\tilde {n}}(1-\rho)$ and thus, for $e := p {\tilde {n}}$, conditions of Lemma 29 are satisfied. The number of the rows of the resulting measurement matrix becomes m=q ^u+3/2, and we have n=q ^k. Therefore,

and in order to ensure that $r < 2\tilde {g}-1$, we need to have $du/(1-p) \gg \sqrt{\log n}$. On the other hand, when $du/(1-p) \ll\sqrt{\log n}$, we are in the high-degree regime, in which case the dimension of the code becomes $k = r - \tilde {g}+1 = \varOmega(r) = \varOmega(\rho q^{3/2})$, and we will thus have

$$q = O((\log n / \rho)^{2/3}) \quad\Rightarrow\quad m = O\left(\bigg(\frac{d \log n}{1-p}\bigg)^{1+2u/3}\right) $$

Altogether, we conclude that Construction 5 with Hermitian codes results in a strongly (d,e;u)-disjunct matrix with

$$m = O\left(\bigg( \frac{d \sqrt{\log n}}{1-p} + \bigg(\frac{d \log n}{1-p}\bigg)^{2/3} \bigg)^{u+3/2}\right) $$

rows, where $e = p \cdot\varOmega( d (\log n)/(1-p) + (d \sqrt {\log n} / (1-p))^{3/2} )$. Compared to the Reed-Solomon codes, the number of measurements has a slightly worse dependence on d, but a much better dependence on n. Compared to AG codes on the TVZ bound, the dependence on d is better while the dependence on n is inferior.

5.4 Codes on the Gilbert-Varshamov Bound

A q-ary $({\tilde {n}},k,{\tilde {d}})$-code (of sufficiently large length) is said to be on the Gilbert-Varshamov bound if it satisfies $k\geq {\tilde {n}}(1-h_{q}({\tilde {d}}/{\tilde {n}}))$, where h _q (⋅) is the q-ary entropy function defined as

$$h_q(x) := x \log_q(q-1) - x \log_q(x) - (1-x) \log_q(1-x). $$

It is well known that a random linear code achieves the bound with overwhelming probability (cf. [32]). Now we apply Lemma 29 on a code on the GV bound, and calculate the resulting parameters. Let ρ:=(1−p)/(4du), choose any alphabet size q∈[1/ρ,2/ρ], and let $\mathcal {C}$ be any q-ary code of length ${\tilde {n}}$ on the GV bound, with minimum distance ${\tilde {d}}\geq {\tilde {n}}(1-2/q)$. By the Taylor expansion of the function h _q (x) around x=1−1/q, we see that the dimension of $\mathcal {C}$ asymptotically behaves as $k= \varTheta({\tilde {n}}/ (q \log q))$. Thus, the number of columns of the resulting measurement matrix becomes $n = q^{k}= 2^{\varOmega({\tilde {n}}/ q)}$. Moreover, the number m of its rows becomes

$$m = q^u {\tilde {n}}= O(q^{u+1} \log n) = O((d/(1-p))^{u+1} \log n), $$

and the matrix becomes strongly (d,e;u)-disjunct for $e = p {\tilde {n}}= \varOmega(p d (\log n)/ (1-p))$.

We remark that for the range of parameters that we are interested in, Porat and Rothschild [34] have obtained a deterministic construction of linear codes on the GV bound that runs in time poly(q ^k) (and thus, polynomial in the size of the resulting measurement matrix).

Their construction is based on a derandomization of the probabilistic argument for random linear codes using the method of conditional expectations, and as such, can be considered weakly explicit (in the sense that, the entire measurement matrix can be computed in polynomial time in its length; whereas for a fully explicit construction one must ideally be able to deterministically compute any single entry of the measurement matrix in time poly(d,logn), which is not the case for this construction). Altogether, we obtain the following result.

Theorem 30

There is an algorithm that, given integer parameters d≤n and u>0 and real parameter p∈[0,1) outputs an m×n binary matrix which is strongly (d,e;u)-disjunct. The parameters m and e satisfy the bounds m=O((d/(1−p))^u+1logn) and e=Ω(pd(logn)/(1−p)). Moreover, the running time of the algorithm is polynomial in mn.

Using a standard probabilistic argument it is easy to see that a random m×n matrix, where each entry is an independent Bernoulli random variable with probability 1/d of being 1, is with overwhelming probability strongly (d,e;u)-disjunct for e=Ω(pdlog(n/d)/(1−p)²) and m=O(d ^u+1(log(n/d))/(1−p)²) (the proof is very similar to the proof of Lemma 14). Thus we see that, for a fixed p, Construction 5 when using codes on the GV bound almost matches these parameters. Moreover, the explicit construction based on Reed-Solomon codes possesses the “right” dependence on the sparsity d, while AG codes on the TVZ bound have a matching dependence on the vector length n with random measurement matrices, and finally, the trade-off offered by the construction based on Hermitian codes lies in between the one for Reed-Solomon codes and AG codes.

6 Concluding Remarks

In this work we have introduced the combinatorial notion of regular binary matrices, that is used as an intermediate tool towards obtaining threshold testing designs.

Even though our construction, assuming an optimal lossless condenser, matches the probabilistic upper bound for regular matrices, the number of measurements in the resulting threshold testing scheme (obtained from the simple direct product in Construction 1) becomes larger than the probabilistic upper bound by a factor of Ω(dlogn). Thus, an outstanding question is directly constructing threshold disjunct matrices that match the probabilistic upper bound. Despite this, the notion of regular matrices may be of independent interest, and an interesting question is to obtain (nontrivial) concrete lower bounds on the number of rows of such matrices in terms of the parameters n,d,e,u (and the gap parameter g in the generalized definition of Sect. 4).

Moreover, in this work we have assumed the upper threshold u to be a fixed constant, allowing the constants hidden in asymptotic notions to have a poor dependence on u. An outstanding question is whether the number of measurements can be reasonably controlled when the upper threshold u and possibly the gap parameter g become large; e.g., g,u=Ω(d).

The lower bound proved in Corollary 28 on the number of rows of threshold designs shows an exponent g+2 for the sparsity parameter, which matches the upper bounds obtained using the probabilistic method. We conjecture that this bound can be improved to Ω _u (d ^g+2log _d n) and more generally when e>2, to Ω _u (d ^g+2log _d n+d ^g+1 e). In other words, for fixed thresholds, we suspect that the asymptotic bounds for (d,e;u,g)-threshold designs and strongly (d,e;g+1)-disjunct matrices should nearly be the same.

Another interesting problem is decoding. While our constructions can combinatorially guarantee identification of sparse vectors, for applications it is important to have an efficient reconstruction algorithm as well. Contrary to the case of strongly disjunct matrices that allow a straightforward decoding procedure (cf. [5]), it is not clear whether in general our notion of disjunct matrices allow efficient decoding, and thus it becomes important to look for constructions that are equipped with efficient reconstruction algorithms.

Finally, for clarity of the exposition, in this work we have only focused on asymptotic trade-offs, and it would be nice to obtain good, finite length, estimates on the obtained bounds that are useful for applications.

Appendix: A Technical Lemma

The following simple proposition is used in the proof of Theorem 27.

Proposition 31

Suppose for some values of a>0, b,m≥2, and c≥2b/a, we have $m \geq c \cdot \frac{a}{b+\log m}$, where the logarithm is to base 2. Then,

$$m \geq \frac{(ac/b)}{\log(ac/b)}. $$

Proof

We can write

$$m \geq c \cdot \frac{a}{b+\log m} \geq \frac{ca}{b \log m} \quad\Rightarrow\quad m \log m \geq ac/b, $$

where the second inequality is from the assumption that b,m≥2.

Since mlogm is an increasing and convex function of m, we know that m≥m ₀, where m ₀ is the solution to the equation m ₀logm ₀=ac/b. Thus it suffices to lower bound m ₀.

Since ac/b≥2 by assumption, it follows that m ₀≤m ₀logm ₀=ac/b, and thus,

$$m_0 \log(ac/b) \geq m_0 \log m_0 = ac/b \quad\Rightarrow\quad m_0 \geq \frac{(ac/b)}{\log(ac/b)}, $$

and the claim follows. □