Abstract
Today, the detection of anomalous records is a highly valuable application in the analysis of current huge datasets. In this paper we propose a new algorithm that, with the help of a human expert, efficiently explores a dataset with the goal of detecting relevant anomalous records. Under this scheme the computer selectively asks the expert for data labeling, looking for relevant semantic feedback in order to improve its knowledge about what characterizes a relevant anomaly. Our rationale is that while computers can process huge amounts of low level data, an expert has high level semantic knowledge to efficiently lead the search. We build upon our previous work based on Bayesian networks that provides an initial set of potential anomalies. In this paper, we augment this approach with an active learning scheme based on the clustering properties of Dirichlet distributions. We test the performance of our algorithm using synthetic and real datasets. Our results indicate that, under noisy data and anomalies presenting regular patterns, our approach significantly reduces the rate of false positives, while decreasing the time to reach the relevant anomalies.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Aamodt, A., Plaza, E.: Case-based reasoning: Foundational issues, methodological variations, and system approaches. Artificial Intelligence Communications 7(1), 39–59 (1994)
Asuncion, A., Newman, D.J.: UCI machine learning repository (2007), http://www.ics.uci.edu/~mlearn/MLRepository.html
Blackwell, D., MacQueen, J.: Ferguson distribution via polya urn schemes. The Annals of Statistics 1(2), 353–355 (1973)
Cansado, A., Soto, A.: Unsupervised anomaly detection in large databases using bayesian networks. Applied Artificial Intelligence 22(4), 309–330 (2008)
Ferguson, T.: A bayesian analysis of some nonparametric problems. The Annals of Statistics 1(2), 209–230 (1973)
Hodge, V., Austin, J.: A survey of outlier detection methodologies. Artificial Intelligence Review 22(2), 85–126 (2004)
Jackson, P.: Introduction to Expert Systems. Addison-Wesley, Reading (1998)
Kou, Y., Lu, C., Sirwongwattana, S., Huang, Y.: Survey of fraud detection techniques. In: Proc. of the IEEE Int. Conf. on Networking, Sensing and Control, pp. 749–754 (2004)
Lewis, D., Gale, W.: A sequential algorithm for training text classifiers. In: Proc. of 17th Int. Conf. ACM SIGIR, pp. 3–12 (1994)
Neapolitan, R.: Learning Bayesian Networks. Prentice-Hall, Englewood Cliffs (2004)
Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, San Francisco (1988)
Pelleg, D., Moore, A.: Active learning for anomaly and rare-category detection. In: Proc. of the 18th Conf. on Advances in Neural Information Processing Systems, NIPS (2004)
Roy, N., McCallum, A.: Toward optimal active learning through sampling estimation of error reduction. In: Proc. of 18th Int. Conf. on Machine Learning, ICML, pp. 441–448 (2001)
Seung, S., Opper, M., Sompolinski, H.: Query by committee. In: Proc. of 5th Annual ACM Workshop on Computational Learning Theory, pp. 287–294 (1992)
Soto, A., Zavala, F., Araneda, A.: An accelerated algorithm for density estimation in large databases using Gaussian mixtures. Cybernetics and Systems 38(2), 123–139 (2007)
Tong, S., Koller, D.: Active learning for parameter estimation in bayesian networks. In: Proc. of the 13th Conf. on Advances in Neural Information Processing Systems, NIPS, pp. 647–653 (2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Pichara, K., Soto, A., Araneda, A. (2008). Detection of Anomalies in Large Datasets Using an Active Learning Scheme Based on Dirichlet Distributions. In: Geffner, H., Prada, R., Machado Alexandre, I., David, N. (eds) Advances in Artificial Intelligence – IBERAMIA 2008. IBERAMIA 2008. Lecture Notes in Computer Science(), vol 5290. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-88309-8_17
Download citation
DOI: https://doi.org/10.1007/978-3-540-88309-8_17
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-88308-1
Online ISBN: 978-3-540-88309-8
eBook Packages: Computer ScienceComputer Science (R0)