Abstract
Today, the issue of data processing has become a topic of vital importance for society and at the business level. Data has been essential for decision-making and business opportunities for companies. Data processing is not a trivial task since large amounts of data are generated in different formats, which makes it difficult to process, store and visualize them. This chapter presents an architecture for real-time data stream processing on Apache Flink. Sensors connected to a prototype of a weather station built for this purpose were used to generate the data. To transmit the data generated by the weather station prototype to Apache Flink, the Apache Kafka streaming tool was used. Finally, the Elasticsearch and Kibana tools were used for data processing and visualization. The tests carried out to verify the operation of all the components of the proposed architecture were satisfactory.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Rodríguez, G.: Temperature streaming with Arduino + Big data tools. Hackster. https://www.hackster.io/Gersaibot/temperature-streaming-with-arduino-big-data-tools-eb22fc. Accessed 2 Feb 2022
Barba, C.J.: Seguimiento de datos en tiempo real con Apache Kafka en Raspberry Pi; Caso práctico: monitoreo ambiental de la actividad del invernadero Espe–Iasa (2018)
Miralles, J.: Publish your arduino data to the cloud. Hackster. https://www.hackster.io/jaume_miralles/publish-your-arduino-data-to-the-cloud-9dfaa2. Accessed 3 Feb 2020
Joyanes, L.: Big data—Análisis de grandes volúmenes de datos en organizaciones. Alfaomega, México (2013)
Pazos-Rangel, R.A., Florencia-Juarez, R., Paredes-Valverde, M.A., Rivera, G. (Eds.).: Handbook of Research on Natural Language Processing and Smart Service Systems. IGI Global, (2021). https://doi.org/10.4018/978-1-7998-4730-4
Bolívar, A., García, V., Florencia, R., Alejo, R., Rivera, G., Sánchez-Solís, J.P.: A preliminary study of smote on imbalanced big datasets when dealing with sparse and dense high dimensionality. In: Pattern Recognition: 14th Mexican Conference, MCPR 2022, Ciudad Juárez, Mexico, June 22–25, 2022, Proceedings, pp. 46–55. Springer International Publishing, Cham (2022). https://doi.org/10.1007/978-3-031-07750-0_5
Microsoft. Microsoft Learn. https://learn.microsoft.com/es-es/azure/data-explorer/ingest-data-overview. Accessed 20 Dec 2022
Amazon Web Services.: AWS. https://aws.amazon.com/es/streaming-data/. Accessed 20 Dec 2022
Amazon Web Services. https://aws.amazon.com/es/nosql/. Accessed 20 Dec 2022
Narkhede, N., Shapira, G., Palino T.: Kafka: The definitive guide. O'Reilly, Sebastopol (2017)
Apache Kafka.: https://kafka.apache.org/intro. Accessed 14 Mar 2023
Cloud Kafka.: https://www.cloudkarafka.com/. Accessed 14 Mar 2023
Hueske, F., Kalavri, V.: Stream processing with apache flink. O'Reilly Media, Inc., (2019)
Apache Flink. https://flink.apache.org/. Accessed 14 Mar 2023
Elasticsearch. s.f.: https://www.elastic.co/es/what-is/elasticsearch. Accessed 13 Sep 2020
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Herrera Castro, J.A., López Najera, A., López Orozco, F., Ponce Rodríguez, B.A. (2023). Acquisition, Processing and Visualization of Meteorological Data in Real-Time Using Apache Flink. In: Rivera, G., Cruz-Reyes, L., Dorronsoro, B., Rosete, A. (eds) Data Analytics and Computational Intelligence: Novel Models, Algorithms and Applications. Studies in Big Data, vol 132. Springer, Cham. https://doi.org/10.1007/978-3-031-38325-0_4
Download citation
DOI: https://doi.org/10.1007/978-3-031-38325-0_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-38324-3
Online ISBN: 978-3-031-38325-0
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)