Abstract
Streaming data also known as unbounded data are increasingly common in the big data era. Modern business collects massive amount of data that are never ending. Hence, there is an increasing need for continuous applications that can process streaming data from massive data ingestion pipelines. However, the majority of streaming system in existence remains relatively immature compare to batch process system, for it can be of great challenge for developers to overcome many obstacles, including: reliability, correctness guarantees, and handling out of order data. Fortunately, the latest Spark Structured Streaming provides the ability to tackle these obstacles. In this paper, we design an architecture that is build on top of Spark Structured Streaming to implement a big data processing system that can provide fast, scalable, fault-tolerant ability to process massive unbounded data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
J. Ekanayake, S. Pallickara, and G. Fox, “MapReduce for data intensive scientific analyses,” in Proceedings of the IEEE International Conference on eScience (eScience ’08), 2008, pp. 277–284.
A. Matsunaga, M. Tsugawa, and J. Fortes, “CloudBLAST: Combining MapReduce and virtualization on distributed resources for bioinformatics applications,” in Proceedings of the IEEE International Conference on eScience (eScience ’08), 2008, pp. 222–229.
Apache. Apache Hadoop. http://hadoop.apache.org, 2012.
Apache. Apache Storm. http://storm.apache.org, 2013.
Apache. Apache Flink. http://flink.apache.org/, 2014.
Apache. Apache Samza. http://samza.apache.org, 2014.
Google. Dataflow SDK. GoogleCloudPlatform/DataflowJavaSDK, 2015.
Google. Google Cloud Dataflow. https://cloud.google.com/dataflow/, 2015.
Apache. Apache Spark. http://spark.apache.org, 2013.
R. S. Barga et al. Consistent Streaming Through Time: A Vision for Event Stream Processing. In Proc. of the Third Biennial Conf. on Innovative Data Systems Research (CIDR), pages 363–374, 2007.
Botan et al. SECRET: A Model for Analysis of the Execution Semantics of Stream Processing Systems. Proc. VLDB Endow., 3(1–2):232–243, Sept. 2010.
Apache. StructuredStreaming, http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html.
J. E. Gonzalez, R. S. Xin, A. Dave, D. Crankshaw, M. J. Franklin, and I. Stoica, “GraphX: Graph processing in a distributed dataflow framework,” in Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14), 2014.
M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. Franklin, S. Shenker, and I. Stoica, “Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing,” in Proceedings of the USENIX Conference on Networked Systems Design and Implementation (NSDI ’12). USENIX Association, 2012, pp. 2–2.
M. Zaharia et al. Discretized Streams: Fault-Tolerant Streaming Computation at Scale. In Proc. of the 24th ACM Symp. on Operating Systems Principles, 2013.
Credit card transaction data set. https://www.kaggle.com/dalpozz/creditcardfraud, 2013.
Acknowledgements
Here and now, we would like to extend my sincere thanks to all those who have helped me make this paper possible and better. Thanks to the colleges and leaders who have taught me over the past years of work and study. We also wish to organize the impressive and tiresome efforts of everyone who make contribution and help to bring this paper to life.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Xiao, C., Zhang, S., Zeng, Q., Cao, X. (2019). Real-Time and Distributed Anomalies Detection Architecture and Implementation with Structured Streaming. In: Patnaik, S., Jain, V. (eds) Recent Developments in Intelligent Computing, Communication and Devices. Advances in Intelligent Systems and Computing, vol 752. Springer, Singapore. https://doi.org/10.1007/978-981-10-8944-2_112
Download citation
DOI: https://doi.org/10.1007/978-981-10-8944-2_112
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-10-8943-5
Online ISBN: 978-981-10-8944-2
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)