Abstract
Selectivity estimation of a boolean query based on frequent itemsets can be solved by describing the problem by a linear program. However, the number of variables in the equations is exponential, rendering the approach tractable only for small-dimensional cases. One natural approach would be to project the data to the variables occurring in the query. This can, however, change the outcome of the linear program.
We introduce the concept of safe sets: projecting the data to a safe set does not change the outcome of the linear program. We characterise safe sets using graph theoretic concepts and give an algorithm for finding minimal safe sets containing given attributes. We describe a heuristic algorithm for finding almost-safe sets given a size restriction, and show empirically that these sets outperform the trivial projection.
We also show a connection between safe sets and Markov Random Fields and use it to furtherreduce the number of variables in the linear program, given some regularity assumptions on the frequent itemsets.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Agrawal, R., Imielinski, T., Swami, A.N.: Mining association rules between sets of items in large databases. In: Buneman, P., Jajodia, S. (eds.) Proceedings of the 1993 ACM SIGMOD International Conference on Management of Data, pp. 207–216. Washington, DC, 26–28 (1993)
Agrawal, R., Mannila, H., Srikant, R., Toivonen, H., Verkamo, A.I.: Fast discovery of association rules. In: Fayyad, U.M., Piatetsky-Shapiro, G., Smyth, P., Uthurusamy, R. (eds.) Advances in Knowledge Discovery and Data Mining, pp. 307–328. AAAI Press/The MIT Press (1996)
Bykowski, A., Seppänen, J.K., Hollmén, J.: Model-independent bounding of the supports of Boolean formulae in binary data. In: Lanzi, P.L., Meo, R. (eds.) Database Support for Data Mining Applications: Discovering Knowledge with Inductive Queries, LNCS 2682, pp. 234–249. Springer Verlag (2004)
Calders, T.: Computational complexity of itemset frequency satisfiability. In: Proceedings of the 23nd ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database System (2004)
Calders, T., Goethals, B.: Mining all non-derivable frequent itemsets. In: Proceedings of the 6th European Conference on Principles and Practice of Knowledge Discovery in Databases. (2002)
Chow, C.K., Liu, C.N.: Approximating discrete probability distributions with dependence trees. IEEE Transactions on Information Theory 14(3), 462–467 (May 1968)
Darroch, J., Ratchli, D.: Generalized iterative scaling for log-linear models. The Annals of Mathematical Statistics 43(5), 1470–1480 (1972)
Forselius, M.: Neogene of the old world database of fossil mammals (NOW). University of Helsinki, http://www.helsinki.fi/science/now/ (2005)
Georgakopoulos, G., Kavvadias, D., Papadimitriou, C.H.: Probabilistic satisfiability. Journal of Complexity 4(1), 1–11 (March 1988)
Goethals, B., Zaki, M.J. (eds.): FIMI ‘03, Frequent itemset mining implementations. In: Proceedings of the ICDM 2003 Workshop on Frequent Itemset Mining Implementations, 19 December 2003, Melbourne, Florida, USA, vol. 90 of CEUR Workshop Proceedings (2003)
Hailperin, T.: Best possible inequalities for the probability of a logical function of events. The American Mathematical Monthly 72(4), 343–359 (Apr. 1965)
Jiroušek, R., Přeušil, S.: On the effective implementation of the iterative proportional fitting procedure. Computational Statistics and Data Analysis 19, 177–189 (1995)
Jordan, M.I. (ed.): Learning in graphical models. MIT Press (1999)
Lukasiewicz, T.: Efficient global probabilistic deduction from taxonomic and probabilistic knowledge-bases over conjunctive events. In: Proceedings of the Sixth International Conference on Information and Knowledge Management, pp. 75–82 (1997)
Lukasiewicz, T.: Probabilistic logic programming with conditional constraints. ACM Transactions on Computational Logic (TOCL) 2(3), 289–339 (July 2001)
Mannila, H., Toivonen, H.: Multiple uses of frequent sets and condensed representations (extended abstract). In: Knowledge Discovery and Data Mining, pp. 189–194 (1996)
Matias, Y., Vitter, J.S., Wang, M.: Wavelet-based histograms for selectivity estimation. In Proceedings of ACM SIGMOD International Conference on Management of Data, pp. 448–459 (1998)
Muralikrishna, M., DeWitt, D.: Equi-depth histograms for estimating selectivity factors for multi-dimensional queries. In Proceedings of ACM SIGMOD International Conference on Management of Data, pp. 28–36 (1988)
Nilsson, N.: Probabilistic logic. Artificial Intelligence, 28(1), 71–87 (1986)
Papadimitriou, C., Steiglitz, K.: Combinatorial Optimization Algorithms and Complexity. Dover, 2nd edn. (1998)
Pavlov, D., Mannila, H., Smyth, P.: Beyond independence: Probabilistic models for query approximation on binary transaction data. IEEE Transactions on Knowledge and Data Engineering 15(6), 1409–1421 (2003)
Pavlov, D., Smyth, P.: Probabilistic query models for transaction data. In: Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 164–173 (2001)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Tatti, N. Safe projections of binary data sets. Acta Informatica 42, 617–638 (2006). https://doi.org/10.1007/s00236-006-0009-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00236-006-0009-9