1 Introduction

A secret sharing scheme allows a dealer to distribute shares of a secret among a set of parties \(P = \{p_1, \dots , p_n\}\) such that any authorized subset \(A \subseteq P\) can reconstruct the secret, yet any unauthorized subset learns nothing about it. The family \(\mathcal {A}\subseteq 2^{\{p_1,\dots ,p_n\}}\) of all authorized subsets is called the access structure. Secret sharing was introduced independently by Shamir  [35] and Blakley  [11], who presented constructions for threshold access structures that contains all subsets with a cardinality larger than some threshold t. The first construction for general (monotone) access structures was presented by Ito, Saito, and Nishizeki  [21].

The main measure of efficiency for secret sharing schemes is the share size. For threshold access structures it is known that Shamir’s secret sharing, which has a share size of \(\varTheta (\lg n)\), is optimal up to additive constants  [13]. This stands in stark contrast to the smallest share sizes we can achieve for general monotone access structures. The construction of Ito, Saito, and Nishizeki has a share size of and 29 years later the best known upper bound on the share size, due to Applebaum et al.  [3] is still \(2^{0.892n}\). A widely believed conjecture suggests that these upper bounds are, up to constants, the best ones one can hope for. More concretely, Beimel conjectured:

Conjecture 1

( [5, 6]). There exists an \(\epsilon > 0\) such that for every integer n there exists an access structure with n parties for which every secret sharing scheme distributes shares of length exponential in the number of parties, that is, \(2^{\epsilon n}\).

Proving this conjecture is a major open problem in the research area of secret sharing schemes. Karnin, Greene, and Hellman  [23] initiated a line of works  [12, 14, 17, 18] that proved different lower bounds on the share size using tools from information theory. The best of those lower bounds is due to Csirmaz  [17, 18], who uses Shannon information inequalities to prove that there exists an explicit access structure that requires shares of size \(\varOmega (n/\lg n)\). Csirmaz himself and subsequent works  [9, 30] indicate that it is unlikely that one can prove a super-polynomial lower bound on the share size using such information inequalities.

A different line of works focuses on linear secret sharing schemes, where the shared secret is a linear combination of the shares. Many of the existing schemes, e.g.  [35], are linear and applications like multiparty computation  [10, 16, 34] crucially rely on this property. Karchmer and Wigderson  [22] introduce monotone span programs and show that these are closely related to linear secret sharing schemes. Through the lens of monotone span programs, a series of works obtained increasingly stronger lower bounds. Karchmer and Wigderson prove the first super-linear lower bound on the share size. Babai, Gál, and Wigderson  [4] prove the first super-polynomial lower bound. Finally, Pitassi and Robere  [33] prove an exponential lower bound, however, the gap between the constants in the exponent of the lower and upper bound remain far apart.

Several works consider different flavors of the original secret sharing notion. Beimel and Franklin  [7] consider a relaxed security notion of weak privacy, which only requires that any unauthorized subset can not exclude any secret value with certainty. The unauthorized subset can, however, conclude that some secret is more probable than another one. The authors show that this notion is strictly weaker than the original notion of secret sharing by constructing schemes with share sizes that are impossible for secret sharing schemes with perfect privacy. Among other results, the authors construct a weakly-private secret sharing scheme for the threshold access structures, where the share size is independent of n. The authors conclude that any sensible lower bound proof has to make use of the privacy requirement of secret sharing schemes. Applebaum et al.  [1, 2] consider the efficiency of secret sharing schemes for large secrets. The authors show that, for a certain class of access structures, one can construct secret sharing schemes, where the share size does not grow with an increasing number n of parties. Their approach requires the secrets to be exponentially large in n.

A different line of works, which is closely related to secret sharing, deals with the conditional disclosure of secrets (CDS) problem  [20]. In this setting, n parties have a common secret s, some common randomness r and separate inputs \(x_i\). The goal of the parties is to each send a single message to a referee, who should learn the secret s iff the inputs \(x_i\) satisfy some publicly known predicate F, i.e. if \(F(x_1, \dots , x_n) = 1\). Beimel et al.  [8] show that any predicate F can be realized with communication complexity and subsequently Liu, Vaikuntanathan and Wee  [29] improve this upper bound to . Gay, Kerenidis, and Wee  [19] prove a lower bound for CDS schemes, which, very roughly speaking, shows that the communication complexity of any CDS is at least as large as the one-way communication complexity of the underlying predicate F. The techniques in this work are similar to some of the techniques in their work and our lower bound proof can be seen as a non-trivial generalization of their initial proof strategy to the case of secret sharing schemes. In contrast to their linear lower bound, our lower bound here is exponential.

Apart from being an interesting primitive on its own, CDS is also the main building block underlying the secret sharing scheme of Liu and Vaikuntanathan  [28] described above. All of the CDS schemes mentioned above run in time exponential in n for certain predicates F.

Despite all progress that was made towards understanding the complexity of secret sharing, a lower bound on the share size of secret sharing schemes for general access structures remained out of reach.

1.1 Our Contribution

In this work we make some progress towards proving Beimel’s conjecture. Informally, we show that either the total share size or the computational effort for reconstructing the secret has to be exponential in n. A bit more formally, let us consider a secret sharing scheme \(\varSigma \) for some access structure \(\mathcal {A}\) that takes a 1-bit secret as input and outputs n shares, which are at most k bits long in total. Let \(\mathcal {F}\) be some family of reconstruction functions. We require that for any authorized subset of parties \(A \subseteq \mathcal {A}\), there exists at least one function in \(\mathcal {F}\) that these parties can use to reconstruct the correct secret with probability at least 3/4. For any \(A \notin \mathcal {A}\), we require that all functions in \(\mathcal {F}\) reconstruct the correct secret with probability at most 1/4. These correctness and privacy requirements are very weak. Neither do we require perfect correctness, nor do we require privacy against an unauthorized set of parties that may use some function outside of \(\mathcal {F}\) to reconstruct the secret. Proving a lower bound for such a secret sharing scheme makes our result only stronger, since any lower bound we can prove here also applies to any secret sharing scheme with better correctness and privacy guarantees. In this work we prove:

Theorem 1

(Informal). For any secret sharing scheme \(\varSigma \) for general access structures, with domain of secrets \(\{0,1\}\) and total share length k, then there exists an access structure \(\mathcal {A}\) such that

$$ \lg (|\mathcal {F}|) \cdot k = \varOmega (2^n/\sqrt{n}). $$

Our result does not prove Beimel’s conjecture, but it tells us that any secret sharing scheme for 1-bit secrets for general access structures, which has a reconstruction function whose description is sub-exponentially large in n, must have a share size that is exponential in n. In particular, this holds even if the secret sharing itself runs in time exponential in n.

To get a better feeling of what \(\mathcal {F}\) is, one can, for example, imagine it to be the set of all functions from \(\{0,1\}^k \rightarrow \{0,1\}\) that are computable by a constant fan-in boolean circuit of some size \(t(k) \ge k\). Any one circuit can compute exactly one function, there are a constant amount of different gates types, and for any gate with constant fan-in, there are choices for the input wires. It follows that there are at most \(t(k)^{O(t(k))}\) different reconstruction functions in \(\mathcal {F}\). Now, if for example \(t(k) \le k^c\) for a constant \(c \ge 1\) (decoding by a circuit of size polynomial in the secret share length), then our theorem says that there exists an access structure \(\mathcal {A}\) for which the share length k must be exponential in n. On the other hand, if k is for example polynomial in n, then our theorem tells us that there exists some access structure \(\mathcal {A}\) which requires an exponentially large reconstruction circuit.

We prove Theorem 1 via a counting argument, meaning that we do not explicitly provide an access structure \(\mathcal {A}\) that is affected by the lower bound. The high-level idea of our proof is as follows. Assume that there exists some secret sharing scheme \(\varSigma _\mathcal {A}\) for every access structure \(\mathcal {A}\) with the desired correctness and privacy properties and a total share size of \(k^\mathcal {A}\le k\). In the first step, we construct a family D that contains all access structures \(\mathcal {A}\) of a certain type and we show that the size of this family is \(2^{\varOmega (2^n/\sqrt{n})}\). By the pingeon-hole principle, we know that the description of any \(\mathcal {A}\in \mathcal {D}\) is at least \(\lg \vert \mathcal {D}\vert = \varOmega (2^n/\sqrt{n})\) bits long. On the other hand, we show that for any \(\mathcal {A}\in \mathcal {D}\) one can use \(\varSigma _\mathcal {A}\) to construct a -bit long lossless encoding from which \(\mathcal {A}\) can be uniquely recovered. Combining the two observations directly yields the theorem stated above. The main challenge in realizing this proof idea lies in the construction of an appropriate encoding (and decoding) algorithm with the desired efficiency. Our encoding algorithm proceeds in two steps. First, we exploit the correctness and privacy properties of our secret sharing scheme to construct a randomized lossless encoding algorithm that works well for \(99\%\) of the sets A in any given \(\mathcal {A}\) and encodes them into bits. A careful analysis reveals that we can simply write out the remaining \(1\%\) of \(A \in \mathcal {A}\) as part of the encoding and still obtain a lower bound on \(\lg \vert \mathcal {F}\vert \cdot k\).

Proving lower bounds via such encoding arguments has been done quite extensively in the area of data structure lower bounds, see e.g.  [15, 24, 25, 31, 32, 36] and was also used recently to prove optimality of the Johnson-Lindenstrauss lemma in dimensionality reduction  [26] and to prove optimality of ORAMs without balls-in-bins assumptions  [27].

Remark. At first sight it may seem that the reconstruction function must take the access structure as input to be able to reconstruct. If this was the case, then our lower bound would be meaningless, since we can construct exponentially large access structures. This, however, is not the case. Consider the following trivial secret sharing scheme for some bitstring x among parties \(p_1, \dots , p_n\) for any access structure \(\mathcal {A}\). For each authorized set \(A = \{p_{i_1}, \dots , p_{i_m}\} \in \mathcal {A}\), we pick uniformly random \(s_{i_1}, \dots , s_{i_m}\) such that \(x = s_{i_1} \oplus \dots \oplus s_{i_m}\) and give \((A, s_{i_j})\) to \(p_{i_j}\). If a set A wants to reconstruct a secret, they check whether they have a share corresponding to A and xor their corresponding shares together if this is the case. If they do not have a share corresponding to A, they conclude that they are not an authorized set. Note that reconstruction function only takes the shares as input and not the access structure itself.

2 Formal Model and Result

In this section, we formally define secret sharing schemes and the precise conditions under which our lower bounds holds. Except for the security requirements, we define a secret sharing scheme precisely as in  [6].

Definition 1

Let \(\{p_1,\dots ,p_n\}\) be a set of parties. A collection \(\mathcal {A}\subseteq 2^{\{p_1,\dots ,p_n\}}\) is monotone if \(B \in \mathcal {A}\) and \(B \subseteq C\) imply \(C \in \mathcal {A}\). An access structure is a monotone collection \(\mathcal {A}\subseteq 2^{\{p_1,\dots ,p_n\}}\) of non-empty subsets of \(\{p_1,\dots ,p_n\}\). Sets in \(\mathcal {A}\) are called authorized, and sets not in \(\mathcal {A}\) are called unauthorized.

Definition 2

Let \(\{p_1,\dots ,p_n\}\) be a set of parties. A distribution scheme \(\varSigma = (\varPi , \mu )\) with domain of secrets \(\{0,1\}\) is a pair, where \(\mu \) is a probability distribution on some finite set R called the set of random strings and \(\varPi \) is a mapping from \(\{0,1\} \times R\) to a set of n-tuples \(\{0,1\}^{k_1} \times \cdots \times \{0,1\}^{k_n}\), where \(\{0,1\}^{k_j}\) is called the domain of shares of \(p_j\). A dealer distributes a secret \(b \in \{0,1\}\) according to \(\varSigma \) by first sampling a random string \(r \in R\) according to \(\mu \), computing a vector of shares \(\varPi (b,r) = (s_1,\dots ,s_n)\) and privately communicating each share \(s_j\) to party \(p_j\). For a set \(A \subseteq \{p_1,\dots ,p_n\}\), we denote \(\varPi (b,r)_A\) as the restriction of \(\varPi (b,r)\) to its A-entries.

When designing secret sharing schemes, one would typically consider larger domains of secrets than just a single bit as in Definition 2. In this paper we are proving a lower bound, so focusing on the simplest possible setting of a secret consisting of a single bit only makes our lower bound stronger and the proof simpler. The lower bound we prove in this paper holds for secret sharing schemes that are computationally more efficient when authorized parties reconstruct the secret than when unauthorized parties attempt to. We define this formally in the following:

Definition 3

Let \(\{p_1,\dots ,p_n\}\) be a set of parties, let \(\mathcal {A}\subseteq 2^{\{p_1,\dots ,p_n\}}\) be an access structure and \(\varSigma = (\varPi , \mu )\) a distribution scheme with domain of secrets \(\{0,1\}\) and domain of shares \(\{0,1\}^{k_1} \times \cdots \times \{0,1\}^{k_n}\). Let \(\mathcal {F}\) be a family of functions from \(\cup _{i=1}^\infty \left( \{0,1\}^i \rightarrow \{0,1\}\right) \) and let \(\mathcal {U}\) be the uniform distribution on \(\{0,1\}\). We say that \((\mathcal {F}, \mathcal {A}, \varSigma )\) is an efficient secret sharing scheme if it satisfies the following two conditions:

  • For any \(A \in \mathcal {A}\), there exists a function \(f_A \in \left( \mathcal {F}\cap \left( \{0,1\}^{\sum _{j \in A} k_j} \rightarrow \{0,1\} \right) \right) \) such that

  • For any \(A \notin \mathcal {A}\), it holds for all functions \(f \in \left( \mathcal {F}\cap \left( \{0,1\}^{\sum _{j \in A} k_j} \rightarrow \{0,1\} \right) \right) \) that

For intuition on Definition 3, consider as an example instantiating \(\mathcal {F}\) to be the set that contains for each i, the set of all functions from \(\{0,1\}^i \rightarrow \{0,1\}\) that are computable by a constant fan-in boolean circuit of size \(t(i) \le i^c\) for a constant \(c>1\), i.e. \(\mathcal {F}\) contains functions computable by polynomially sized circuits. With this choice of \(\mathcal {F}\), consider an access structure \(\mathcal {A}\). A distribution scheme \(\varSigma \) gives an efficient secret sharing scheme \((\mathcal {F},\mathcal {A},\varSigma )\) precisely if any authorized set of parties \(A \in \mathcal {A}\) can recover the secret using some constant fan-in boolean circuit with size polynomial in the share length, whereas no unauthorized set of parties can recover the secret using any constant fan-in boolean circuit with size polynomial in the share length. We can thus think of \(\mathcal {F}\) as defining the computational resources with which authorized parties can recover the secret, but unauthorized parties cannot. For ease of notation, define \(\mathcal {F}_{\le k}\) as

$$ \mathcal {F}_{\le k} := \mathcal {F}\cap \left( \cup _{i=1}^{k} \left( \{0,1\}^i \rightarrow \{0,1\}\right) \right) $$

and define

$$ \mathcal {F}_{=k} := \mathcal {F}\cap \left( \{0,1\}^k \rightarrow \{0,1\}\right) . $$

in the remainder of the paper.

Discussion 1. When designing secret sharing schemes, one would typically insist that authorized parties can reconstruct the secret with probability . Similarly, one would insist that unauthorized parties cannot reconstruct the secret except with probability . Since we are proving a lower bound, using the constants 3/4 and 1/4 in Definition 3 only makes our results stronger.

Discussion 2. One could consider allowing randomization in the algorithms used for reconstructing the secret, both for the authorized and unauthorized parties. That is, a natural extension of Definition 3 would say that there exists a distribution \(\gamma _A\) over functions in

$$\left( \mathcal {F}\cap \left( \{0,1\}^{\sum _{j \in A} k_j} \rightarrow \{0,1\} \right) \right) $$

such that \(\Pr _{b \sim \mathcal {U}, r \sim \mu , f_A \sim \gamma _A} [ \cdots \). We remark that the definition would be equivalent to Definition 3 since one can always fix the randomness in \(f_A\) to achieve the same guarantees (equivalent to one direction of Yao’s minimax principle).

Discussion 3. Our definition may seem superficially similar to the definition of weakly-private secret sharing schemes by Beimel and Franklin  [7]. Their definition states that any unauthorized set cannot exclude any potential secret with probability 1. It does, however, allow the adversary to guess the secret correctly with a probability that is arbitrarily close to 1. In contrast to their definition, ours is strictly stronger, since it requires a sharp upper bound on the probability that an unqualified set of parties guesses the correct secret.

We are ready to present our main theorem in its full generality:

Theorem 2

Let \(\mathcal {F}\) be a family of functions from \(\cup _{i=1}^\infty \left( \{0,1\}^i \rightarrow \{0,1\}\right) \) and let \(\{p_1,\dots ,p_n\}\) be a set of parties. There exists an access structure \(\mathcal {A}\subseteq 2^{\{p_1,\dots ,p_n\}}\) such that any efficient secret sharing scheme \((\mathcal {F}, \mathcal {A},\varSigma )\) with domain of secrets \(\{0,1\}\) and domain of shares \(\{0,1\}^{k_1} \times \cdots \times \{0,1\}^{k_n}\) with \(k = \sum _j k_j\), satisfies

$$ \lg (|\mathcal {F}_{\le k}|) \cdot k = \varOmega (2^n/\sqrt{n}). $$

To appreciate Theorem 2, consider instantiating \(\mathcal {F}\) to be the set that contains for each i, the set of all functions from \(\{0,1\}^i \rightarrow \{0,1\}\) that are computable by a constant fan-in boolean circuit of size t(i) (with \(t(i) \ge i\)). A simple counting argument shows that \(|\mathcal {F}_{\le k}| \le t(k)^{O(t(k))}\) (A circuit computes only one function and there are \(t(k)^{O(1)}\) choices for the input wires to each gate, there are O(1) choices for the function computed by each gate, and there are t(k) gates). Theorem 2 thus gives us that there must exist an access structure \(\mathcal {A}\) such that any efficient secret sharing scheme \((\mathcal {F},\mathcal {A},\varSigma )\) with domain of shares \(\{0,1\}^{k_1} \times \cdots \{0,1\}^{k_n}\) with \(k = \sum _j k_j\) must satisfy \(t(k) \lg (t(k)) k = \varOmega (2^n/n^{1/2})\). If we plug in polynomially sized constant fan-in boolean circuits, i.e. \(t(i) \le i^c\) for a constant \(c \ge 1\), this gives us that \(k^{c+2} = \varOmega (2^n/n^{1/2}) \Rightarrow k = 2^{\varOmega (n)}\), i.e. any secret sharing scheme for \(\mathcal {A}\) must have shares with exponential length if decoding can be done by constant fan-in boolean circuits with size polynomial in the share length. Moreover, the lower bound holds even if we only require that unauthorized parties cannot reconstruct the secret using a polynomially sized constant fan-in boolean circuit (polynomial in the length of the shares). Notice that since this is a lower bound, it only makes the result stronger than if we e.g. required that a computationally unbounded set of parties cannot reconstruct the secret. We can also deduce from Theorem 2 that the size of the decoding circuit must be exponential in n, regardless of the share length.

Another interesting instantiation of Theorem 2 is to let \(\mathcal {F}\) consist of all functions computable by a Turing machine with at most \(10^6\) states and alphabet \(\{0,1\}\) (or some other constant number of states). Then \(|\mathcal {F}\cap (\{0,1\}^i \rightarrow \{0,1\})| = O(1)\) and the lower bound says that there exists an access structure \(\mathcal {A}\) for which any efficient secret sharing scheme \((\mathcal {F},\mathcal {A},\varSigma )\) must satisfy \(k^2 = \varOmega (2^n/\sqrt{n}) \Rightarrow k = 2^{\varOmega (n)}\), i.e. shares must have exponential length if the secret can be reconstructed by authorized parties using a Turing machine with at most \(10^6\) states and binary alphabet. The lower bound holds as long as we require that unauthorized parties cannot recover the secret using a Turing machine with at most \(10^6\) states and alphabet \(\{0,1\}\).

An even more exotic instantiation of Theorem 2 follows by letting \(\mathcal {F}\) contain, for every i, the set of functions from \(\{0,1\}^i \rightarrow \{0,1\}\) that are computable by a C-program with up to t ASCII characters. A counting argument shows that \(|\mathcal {F}_{\le k}| \le k 2^{O(t)}\) (there are \(2^{O(t)}\) sequences of t ASCII characters, and any program computes at most one function from \(\{0,1\}^i \rightarrow \{0,1\}\)) and we conclude that it must be the case that there exists an access structure \(\mathcal {A}\) such that any efficient secret sharing \((\mathcal {F},\mathcal {A},\varSigma \)) must have \((t + \lg k) \cdot k = \varOmega (2^n/\sqrt{n})\). This means that either the length of the C-program has to grow exponentially with the number of parties n, or the length of the shares has to grow exponentially with n. Thus if we insist on short shares, then the C-programs for reconstructing the secret have to be extremely non-uniform, and if we insist on reconstructing secrets using C-programs of any constant length t independent of n, then the shares must have exponential length. This lower bound holds as long as we require that unauthorized parties cannot recover the secret via a C-program of length t or less.

Finally, if one insist that authorized parties can efficiently reconstruct the secret via a C-program of length at most t ASCII characters, then the previous lower bound is strengthened. That is, we can now let \(\mathcal {F}\) contain, for every i, the set of functions from \(\{0,1\}^i \rightarrow \{0,1\}\) that are computable by a C-program with up to t ASCII characters that terminates in at most h steps. If we insist that authorized parties can reconstruct the secret by running such a C-program, then the lower bound \((t + \lg k) \cdot k = \varOmega (2^n/\sqrt{n})\) holds even if we only require that unauthorized parties cannot reconstruct the secret via a C-program of length t and running time at most h steps.

3 Lower Bound Proof

To prove Theorem 2, let \(\{p_1,\dots ,p_n\}\) be a set of parties and let \(\mathcal {F}\) be a family of functions from \(\cup _{i=1}^\infty \left( \{0,1\}^i \rightarrow \{0,1\}\right) \). Assume that there is a parameter k such that it holds for all access structures \(\mathcal {A}\subseteq 2^{\{p_1,\dots ,p_n\}}\), that there exists an efficient secret sharing scheme \((\mathcal {F}, \mathcal {A},(\varPi _\mathcal {A}, \mu _\mathcal {A}))\) with domain of secrets \(\{0,1\}\) and domain of shares \(\{0,1\}^{k^\mathcal {A}_1} \times \cdots \times \{0,1\}^{k^\mathcal {A}_n}\) with \(\sum _j k^\mathcal {A}_j = k^\mathcal {A}\le k\).

We will prove a lower bound on \(\lg (|\mathcal {F}_{\le k}|) \cdot k\) via a counting argument. The high level intuition is that two distinct access structures \(\mathcal {A}_1\) and \(\mathcal {A}_2\) must be different either in terms of the shares they use, or in terms of the procedures used for reconstructing the secrets. Since there are overwhelmingly many distinct access structures, this gives a lower bound on either the share length (a lower bound on k), or on the descriptional size of the procedures used for reconstructing secrets (a lower bound on \(\lg (|\mathcal {F}_{\le k}|)\)).

More formally, let \(\mathcal {D}\) be the family containing all access structures \(\mathcal {A}\subseteq 2^{\{p_1,\dots ,p_n\}}\) such that \(\mathcal {A}\) contains no sets A of cardinality less than \(\lfloor n/2\rfloor \) and \(\mathcal {A}\) contains all sets A of cardinality more than \(\lfloor n/2 \rfloor \). We claim that \(|\mathcal {D}| = 2^{\left( {\begin{array}{c}n\\ \lfloor n/2\rfloor \end{array}}\right) } = 2^{\varOmega (2^n/\sqrt{n})}\). To see this, observe that \(\mathcal {A}\) is monotone for any choice of subsets with cardinality \(\lfloor n/2\rfloor \) that we might include in it. Since there are \(\left( {\begin{array}{c}n\\ \lfloor n/2 \rfloor \end{array}}\right) \) subsets of cardinality \(\lfloor n/2\rfloor \), we conclude that there are \(2^{\left( {\begin{array}{c}n\\ \lfloor n/2 \rfloor \end{array}}\right) }\) ways of choosing which subsets to include in \(\mathcal {A}\).

We will show that we can encode any \(\mathcal {A}\in \mathcal {D}\) into

$$ \lambda = O(\lg (|\mathcal {F}_{\le k}|) \cdot k) + 0.1 \cdot \left( {\begin{array}{c}n\\ \lfloor n/2\rfloor \end{array}}\right) $$

bits and still uniquely recover \(\mathcal {A}\) from the encoding alone. The encoding procedure thus defines an injective mapping from \(\mathcal {D}\) to \(\{0,1\}^\lambda \). By the pigeon-hole principle, this implies that

$$\begin{aligned} \lambda\ge & {} \lg |\mathcal {D}| \Rightarrow \\ O(\lg (|\mathcal {F}_{\le k}|) \cdot k)\ge & {} 0.9 \cdot \left( {\begin{array}{c}n\\ \lfloor n/2\rfloor \end{array}}\right) \Rightarrow \\ \lg (|\mathcal {F}_{\le k}|) \cdot k= & {} \varOmega (2^n/\sqrt{n}). \end{aligned}$$

We are now ready to describe our encoding and decoding procedures.

Encoding. Let \(\mathcal {A}\in \mathcal {D}\). Our procedure for uniquely encoding \(\mathcal {A}\) is as follows:

  1. 1.

    For \(i=1,\dots ,T\) for a parameter T to be fixed, consider sampling \(b_i \sim \mathcal {U}\) as a uniform random bit, and sample \(r_i \sim \mu _\mathcal {A}\). Let \(A \subseteq \{p_1,\dots ,p_n\}\) be an arbitrary set of cardinality \(\lfloor n/2\rfloor \) and define \(k^\mathcal {A}_A = \sum _{j \in A} k^\mathcal {A}_j\). By Definition 3, it holds that:

    • If \(A \in \mathcal {A}\), then there exists a function \(f_A \in \mathcal {F}_{= k_A^\mathcal {A}}\) such that

    • If \(A \notin \mathcal {A}\), then for all functions \(f \in \mathcal {F}_{= k_A^\mathcal {A}}\), it holds that

    We use this observation as follows: We set \(T = c \lg |\mathcal {F}_{\le k}|\) for a sufficiently large constant \(c > 1\). If \(A \notin \mathcal {A}\), then since \(|\mathcal {F}_{=k_A^\mathcal {A}}| \le |\mathcal {F}_{\le k}|\), we can use a Chernoff bound and a union bound over all \(f \in \mathcal {F}_{=k_A^\mathcal {A}}\) to conclude that with probability at least 99/100, it holds simultaneously for all \(f \in \mathcal {F}_{=k_A^\mathcal {A}}\) that

    $$ \left| | \{ i : f(\varPi _\mathcal {A}(b_i,r_i)_A) = b_i \}| - |\{ i : f(\varPi _\mathcal {A}(b_i,r_i)_A) \ne b_i \}| \right| < T/3. $$

    At the same time, if \(A \in \mathcal {A}\) and we have \(T = c\lg |\mathcal {F}_{\le k}|\), then with overwhelming probability, we will have that there exists at least one function \(f \in \mathcal {F}_{=k_A^\mathcal {A}}\) such that

    $$ \left| | \{ i : f(\varPi _\mathcal {A}(b_i,r_i)_A) = b_i \}| - |\{ i : f(\varPi _\mathcal {A}(b_i,r_i)_A) \ne b_i \}| \right| > T/3. $$

    Thus intuitively, the variables \(b_1,\dots ,b_T\) and \(r_1,\dots ,r_T\) reveal whether A is in \(\mathcal {A}\) or not, i.e. they carry information about \(\mathcal {A}\). We exploit this as follows: Let \(\chi _A\) be the random variable taking the value 1 if the test

    $$\begin{aligned}&\exists f \in \mathcal {F}_{=k_A^\mathcal {A}} : \\&\left| | \{ i : f(\varPi _\mathcal {A}(b_i,r_i)_A) = b_i \}| - |\{ i : f(\varPi _\mathcal {A}(b_i,r_i)_A) \ne b_i \}|\right| > T/3? \end{aligned}$$

    correctly predicts whether \(A \in \mathcal {A}\). Then \(\Pr [\chi _A = 1] \ge 99/100\). Let \(\mathcal {S}\) be the family of all subsets of \(\{p_1,\dots ,p_n\}\) that have cardinality \(\lfloor n/2\rfloor \). It follows by linearity of expectation that \(\mathbb {E}[\sum _{A \in \mathcal {S}} \chi _A] \ge 99|\mathcal {S}|/100\). This means that there must exist a choice values \(\hat{b}_1,\dots ,\hat{b}_T\) and \(\hat{r}_1,\dots ,\hat{r}_T\) such that the test \(\exists f \in \mathcal {F}_{= k_A^\mathcal {A}} : | | \{ i : f(\varPi _\mathcal {A}(\hat{b}_i,\hat{r}_i)_A) = \hat{b}_i \}| - |\{ i : f(\varPi _\mathcal {A}(\hat{b}_i,\hat{r}_i)_A) \ne \hat{b}_i \}|| > T/3?\) correctly predicts whether \(A \in \mathcal {A}\) for at least \(99|\mathcal {S}|/100\) sets \(A \in \mathcal {S}\). Fix such values.

  2. 2.

    Write down \(\lg k\) bits specifying \(k^\mathcal {A}\), followed by k bits specifying \(k_1^\mathcal {A},\dots ,k_n^\mathcal {A}\) (this can be done by writing a length k bit string, where positions \(\sum _{i=1}^j k_i^\mathcal {A}\) are set to 1 for all \(j=1,\dots ,n\)). Then write down the bits \(\hat{b}_1, \cdots , \hat{b}_T\) and \(\varPi _\mathcal {A}(\hat{b}_1,\hat{r}_1)), \cdots ,\varPi _\mathcal {A}(\hat{b}_T,\hat{r}_T))\) for a total of at most \(\lg k + k + T(1 + k)\) bits.

  3. 3.

    Let \(\bar{\mathcal {S}}\) be the subset of sets from \(\mathcal {S}\) where the prediction is incorrect. Encode \(\bar{\mathcal {S}}\) as a subset of \(\mathcal {S}\) using \(\lg \left( {\begin{array}{c}n\\ \lfloor n/2\rfloor \end{array}}\right) \le n\) bits to specify \(|\bar{\mathcal {S}}|\) and \(\lg \left( {\begin{array}{c}|\mathcal {S}|\\ |\bar{\mathcal {S}}|\end{array}}\right) \le |\bar{\mathcal {S}}|\lg (e |\mathcal {S}|/|\bar{\mathcal {S}}|) \le (|\mathcal {S}|/100) \lg (100 e) < 0.1 \cdot \left( {\begin{array}{c}n\\ \lfloor n/2\rfloor \end{array}}\right) \) bits to specify the subset.

Next we argue how to recover \(\mathcal {A}\) from the above encoding:

Decoding.

  1. 1.

    Read the first \(\lg k + k\) bits to recover \(k^\mathcal {A}\) and \(k_1^\mathcal {A},\dots ,k_n^\mathcal {A}\). Then use the following \(T(k+1)\) bits to recover \(\hat{b}_1,\dots ,\hat{b}_T\) and \(\varPi _\mathcal {A}(\hat{b}_1,\hat{r}_1), \dots , \varPi _\mathcal {A}(\hat{b}_T,\hat{r}_T)\).

  2. 2.

    For each \(A \in \mathcal {S}\), iterate over all \(f \in \mathcal {F}_{= k_A^\mathcal {A}}\) and compute the value

    $$ \varDelta _f := \left| | \{ i : f(\varPi _\mathcal {A}(\hat{b}_i,\hat{r}_i)_A) = \hat{b}_i \}| - |\{ i : f(\varPi _\mathcal {A}(\hat{b}_i,\hat{r}_i)_A) \ne \hat{b}_i \}|\right| . $$

    Observe that the decoder can extract \(\varPi _\mathcal {A}(\hat{b}_i,\hat{r}_i)_A\) from \(\varPi _\mathcal {A}(\hat{b}_i,\hat{r}_i)\) since the decoder knows \(k_1^\mathcal {A},\dots ,k_n^\mathcal {A}\). Thus the decoder can indeed compute \(\varDelta _f\). If there is at least one f with \(\varDelta _f \ge T/3\), we initially predict that \(A \in \mathcal {A}\) and otherwise, we predict that \(A \notin \mathcal {A}\). These predictions are correct, except for \(A \in \bar{\mathcal {S}}\).

  3. 3.

    Finally we read the last part of the encoding to determine which sets A that were predicted incorrectly in step 2. Together with the correct predictions from step 2., this recovers \(\mathcal {A}\).

Analysis. Finally we derive the lower bound. We have just argued that we can give a unique encoding of each \(\mathcal {A}\in \mathcal {D}\), hence the length of the encoding must be at least \(\lg |\mathcal {D}| = \left( {\begin{array}{c}n\\ \lfloor n/2\rfloor \end{array}}\right) \) bits. But the above encoding uses at most:

$$ \lg k + k + T(1+k) + n + 0.1 \cdot \left( {\begin{array}{c}n\\ \lfloor n/2\rfloor \end{array}}\right) $$

bits. Thus we must have

$$\begin{aligned} \lg k + k + T(1+k) + n + 0.1 \cdot \left( {\begin{array}{c}n\\ \lfloor n/2\rfloor \end{array}}\right)\ge & {} \left( {\begin{array}{c}n\\ \lfloor n/2\rfloor \end{array}}\right) \Rightarrow \\ Tk = \varOmega \left( \left( {\begin{array}{c}n\\ \lfloor n/2\rfloor \end{array}}\right) \right)= & {} \varOmega (2^n/\sqrt{n}). \end{aligned}$$

But \(T = c \lg |\mathcal {F}_{\le k}|\) and we conclude:

$$ \lg |\mathcal {F}_{\le k}| \cdot k = \varOmega (2^n/\sqrt{n}). $$