Introduction

Urban landscapes create a specific microclimate with direct impact on air/surface temperature patterns, human comfort conditions (Bajšanski et al. 2015), pluvial/river floods, and other climatic hazards. Nowadays, more than half of the global population lives in urban areas, and it is expected that two-thirds of world’s population will live in cities by 2050 (UN 2014). The population in urban areas is directly affected by urban climate (Savić et al. 2018a), which is modified, compared to natural surroundings, and has mostly negative consequences during extreme weather events. Climate can be modified in large, medium-sized, and small cities and these modifications are based on surface indicators, as well as on the indicators of cities’ metabolism, such as artificial surface and surface roughness, intensity of traffic, energy consumption, and population density. The phenomena of the urban heat island (UHI) and the surface urban heat island (SUHI), which represent urban/rural air/surface temperature differences, higher intensity of heat waves in urbanized areas (mostly densely built-up), and outdoor human thermal discomfort, are results of urban climate and they affect population health, the ecosystem, and local communities (Wilhelmi and Hayden 2010; Cetin 2015; Cetin et al. 2018a, b; Savić et al. 2018b). Furthermore, urban areas are marked by pluvial floods, mostly during summer period, after heavy and intensive rains that directly affect traffic and the movement of the urban population. It is possible to observe differences in air humidity, which varies in urban areas (depending on the current synoptic situation) (Milošević et al. 2018), and very complex wind directions due to the orientation of streets and buildings (Savić et al. 2018a).

The constant demand for better living environments drives the need for a thorough understanding of urban landscapes and ecosystems. Urban climate factors affect people to feel thermally comfortable or discomfortable because the planning policy and urbanization management are suboptimal, even though studies show that there are a range of bioclimatic comfort zones which population perceives as comfortable (Cetin 2015). The ability to shape the landscape in a way that ensures the greatest benefit for the urban population, local communities, and stakeholders requires detailed knowledge of the actual urban climate conditions.

To meet this challenge, the construction of a long-term database with high-density data is mandatory, and the tools for this purpose in urban climate research are urban meteorological networks (UMNs). Most commonly, UMNs have a short-term field campaign role, i.e., they are set up on a temporary basis to collect data over short periods, such as weeks, months, or sometimes years (Grimmond 2006). During the period, UMNs have semi-permanent or permanent roles with near-time communication capabilities in order to provide long-term measurement data. Unfortunately, the permanent function of UMNs is still a serious challenge mainly due to the rapid development of technology, funding issues, and objective difficulties. Over the past two decades, an increasing number of such systems have been established all over the world for the study of the atmospheric processes occurring in urban areas (Muller et al. 2013a).

The first city-scale UMNs with the purpose of monitoring the UHI phenomenon, urban air quality, or other hazardous events (e.g., heat waves or cold waves) were deployed in the USA, Tokyo, and Finland (Muller et al. 2013a). The Oklahoma City Micronet (OKCNET) was set up in 2008 with 4 Mesonet and 36 Micronet stations mounted on traffic signals (a height of 10 m) (Basara et al. 2010; Hu et al. 2016). The Metropolitan Environmental Temperature and Rainfall Observation System (METROS) with 120 climate stations was created across the Central Tokyo area (2187 km2) and was mostly deployed at schools (Mikami et al. 2003). In Finland, 300 sensors were deployed by the Finnish Meteorological Institute (FMI) on an area of 150 km2, including the city of Helsinki, as well as peri-urban and rural areas (Dabberdt et al. 2005; Poutiainen et al. 2006; Koskinen et al. 2011). The Birmingham Urban Climate Laboratory (BULC) is one of the first high-density urban monitoring networks with 30 weather stations and 250 air temperature measurement sites across the city (Chapman et al. 2012; Chapman et al. 2015; Warren et al. 2016). In Turku (Finland), 58 Hobo H8 Pro temperature loggers were installed within the Turku Urban Climate Research Project (TURCLIM) and set up to record temperature at 30-min intervals (Suomi and Käyhkö 2012). A total of 24 stations with air temperature and relative humidity sensors were deployed across the urban area of Szeged (Hungary) in order to monitor the UHI pattern and intra- and inter-urban differences (Lelovics et al. 2014; Skarbit et al. 2017). The Metropolitan Station System in Olomouc (MESSO) in the Czech Republic includes up to 24 sites with weather stations or individual sensors (air temperature and humidity) for UHI analysis and temperature/humidity/thermal comfort differences in various local climate zones (LCZs) (Lehnert et al. 2015). The high-density street-level air temperature observation network (HiSAN) that comprises 102 automatic weather stations was created in Tainin City along the coast of urban and suburban areas. The HISAN is used for the analysis of the intensity of UHIs and the characteristics of their temporal variations (Chen et al. 2018; Lin et al. 2019). Within the scope of the MOCCA project, six automatic weather stations have been installed in Ghent (Belgium) for the UHI and thermal comfort analysis. The stations will be in place for 3 years (http://www.observatory.ugent.be/index_eng.html). Over the past several years, there has been a new approach to UMNs in the investigation of urban thermal climates, using crowdsourcing techniques and obtaining freely available citizen weather station data (Bell et al. 2013) from several hundred stations in London (Chapman et al. 2017) and Berlin (Meier et al. 2017; Fenner et al. 2017).

A successful UMN deployment depends on solving several challenges, including reliable and cost-effective data distribution, a thorough knowledge of the behavior of the entire system, and a strong core infrastructure. Various communication technologies are used for data transfer, from wired Ethernet (IEEE 802.3), modem-based DSL/ADSL lines, or radio transmission, such as wireless networks (IEEE 802.11), which are usually preferred over GSM/GRPS due to lower data transmission costs (Muller et al. 2013b). The Oklahoma City Micronet (OKCNET) (Basara et al. 2010; Schroeder et al. 2010; Basara et al. 2011) and the Birmingham UMN (e.g., BUCL) (Young et al. 2012; Chapman et al. 2012, 2015) are examples of systems that use wireless Ethernet. However, using GSM/GRPS technologies allows greater area coverage.

Air temperature, relative humidity, direct solar radiation, wind speed, and direction are the fundamental values that must be measured in urban areas and high-resolution measurements are often required for numerous applications (Chapman et al. 2012; Lelovics et al. 2014; Šećerov et al. 2015). Monitoring microenvironment conditions and their variables can improve human living conditions and reveal opportunities for lowering energy consumption. To obtain long-term and effective measurement data within the Novi Sad urban area, the Novi Sad Urban Network (NSUNET) system was created between 2014 and 2015 with 27 remote stations (equipped with air temperature and relative humidity sensors) and one automatic weather station (AWS) used solely for solar radiation and wind measurements. The system was developed within the framework of an EU-funded project under the Interreg-IPA CBC Hungary-Serbia program (project title: Evaluation and public display of URBAN PATterns of Human thermal conditions—URBAN-PATH; project ID: HUSRB/1203/122/166) (Unger et al. 2014).

This paper presents the detailed architecture and implementation of the NSUNET system, an urban monitoring network built solely using open-source technologies. The proposed instrumentation and communication structure can be implemented for high-density networks that measure various natural or social elements in any geographical location with Internet coverage. Furthermore, the construction of a cost-effective UMN is one of the challenges that this paper aims to overcome. In order to identify all advantages and disadvantages of the NSUNET, the communication infrastructure is compared with similar UMNs in other cities.

Geographical features of the research area

Novi Sad is the second largest city in Serbia, with approximately 330,000 inhabitants living in the built-up area of 102 km2 (as of 2017). The urban area is located in the northern part of Serbia (45° 15′ N, 19° 50′ E), on the Carpathian plain between 80 and 130 m a.s.l. (Fig. 1). The Danube River passes through the southern and eastern edges of the urban area (260–680 m width), and the narrow Danube-Tisza-Danube Canal passes through the northern section of the city (Fig. 2). The northern slopes of the Fruška Gora Mountain are located south of the Novi Sad urban area (Bajšanski et al. 2015; Unger et al. 2011).

Fig. 1
figure 1

Novi Sad urban area and its location in Serbia and Europe

Full size image
Fig. 2
figure 2

Spatial patterns of the local climate zones (LCZs) and the Novi Sad Urban Network (NSUNET) station sites in the built-up and land cover areas

Full size image

The Novi Sad region has a Cfb climate (temperate climate, fully humid, and warm summers, with at least four Tmon ≥ + 10 °C), according to the Köppen-Geiger climate classification (Kottek et al. 2006). The mean monthly air temperature ranges from − 0.4 °C in January to 21.7 °C in July. The mean annual precipitation is 598 mm (based on data registered from 1949 to 2013).

Methodology and results of the monitoring network system

The major task in data acquisition systems is to ensure a low percentage of data loss. As stations are placed across the urban area, different locations introduce various problems, such as human vandalism, inadequate power supply to the stations, and varying conditions for data transport (Muller et al. 2013b). Solving these problems requires a systematic approach to the system topology design. To make the UMN system reliable and easy to monitor, it must be divided into autonomous modules that are mutually independent.

The NSUNET system is divided into three major segments: (a) a remote segment with remote stations that collect and send the measurement data; (b) a core segment that stores and processes primary and secondary data received from stations; and (c) further data redistribution, which is covered by a third-party segment.

Spatial distribution of the NSUNET stations

The location for each station was determined based on the local climate zone (LCZ) classification system guidelines (Stewart and Oke 2012), GIS tools for the calculation methodology (Lelovics et al. 2014), and guidance for representative meteorological observations at urban sites (Aguilar et al. 2003; Oke 2006; Muller et al. 2013b). According to the urbanization pattern of Novi Sad and the local knowledge of researchers, the NSUNET consists of 25 stations positioned at 7 different LCZ classes that represent built-up areas. Furthermore, 2 stations are located in the land cover classes (LCZs A and D) and represent climate conditions in non-urbanized areas (Unger et al. 2015; Šećerov et al. 2015). Finally, AWS is placed on the top of the Faculty of Sciences building (at the University of Novi Sad campus) and it has been used for solar radiation and wind measurements (Fig. 2).

Remote segment

The NSUNET stations are equipped with ChipCap 2 sensors placed in radiation protection screens; their dimensions are 200 × 240 mm. Fully calibrated temperature and humidity sensors are developed by the General Electric Measurement & Control Company. The accuracy of the air temperature sensor is ± 0.3 °C, and that of the relative humidity sensor is ± 2% (20–80% RH). Each station contains a central processor, an erasable programmable read-only memory (EPROM) chip for data storage, a GPRS/EDGE/3G modem, power supply, and a backup battery installed inside the console. The sensors and the equipment were installed at least 4 m above the ground (with a deviation of ± 0.2 m) on the arms (50 cm long) fixed to selected lampposts (Fig. 3a). Two stations in the land cover areas were installed at 2 m above the ground (Šećerov et al. 2015). The AWS is a PanSense V2 model developed by Davis Instruments and is installed approximately 35 m above the ground on the roof of the Faculty of Sciences building (at the University of Novi Sad campus) (Fig. 3b).

Fig. 3
figure 3

Station (5-2) with air temperature and relative humidity sensors mounted on a lamppost (a) and the AWS installed on the roof of the Faculty of Sciences building (University of Novi Sad campus) (b)

Full size image

Stations A-1 and PMF-1 have a continuous power supply, whereas other stations have power supply only during evening and night hours when city lights are on. During the day, i.e., when the city lights are off, stations operate on an installed battery supply (Šećerov et al. 2015). Generally, the time available for battery recharge is proportional to the time between sunset and sunrise, which changes with the seasons of the year. To overcome any unexpected issues from city power distribution (e.g., unintended stoppages in the lamppost power supply), the installed batteries must have a long-life expectancy and a minimal recharge time. Each station uses Power-Sonic corporation PS-1242 12 V 4.5AH batteries and can operate without power supply for up to 15 days, during which period data acquisition and early warning feature are fully operational.

Stations are activated each minute to perform data measurements and check power supply input values, battery status, and other technical details that provide valuable data about system reliability. The number of required measurements that define one session is represented by a special variable, session length. How often will station send data to the core segment is controlled using the transmission period. All data are stored in EPROM memory.

The air temperature (T) (Eq. 1) and relative humidity (RH) (Eq. 2) where session length is 10 are calculated as:

$$ \frac{1}{10}\sum \limits_{i=1}^{10}{T}_i $$
(1)
$$ \frac{1}{10}\sum \limits_{i=1}^{10}R{H}_i $$
(2)

Furthermore, to preserve battery power in the NSUNET system, the interval between two sessions could be prolonged. By setting a higher value for the transmission period, battery consumption can be decreased, which is desirable at locations where frequent problems occur with power supply distribution. The overall data acquisition reliability is thereby increased. Stations use the coordinated universal time (UTC) as the system time, which is synchronized with the core segment’s servers (installed at the Faculty of Sciences). During transmission, the latest measurements are sent first, followed by the data that failed to be sent in previous sessions (if such errors occurred). This method provides city emergency services with the most up-to-date information about urban weather conditions. When the session length is reached, the stations collect additional data on their progress, temporary and critical failures (if any), and store them together with the measurement data using the same data structure. Since the NSUNET system uses the GSM network for data transmission, one additional notification mechanism is implemented. The stations use SMS notification to alert in case of critical system failure, thereby increasing the system’s overall reliability. Within each transmission, the information about errors detected during station work is stored and sent using 16-bit status value, which is explained in detail further in the text.

The stations’ built-in algorithm is designed to preserve data for a long period of time when the network service is not available. The storage system in the station’s internal memory is organized in a circular first in/first out (FIFO) (Han and Stevens 2009) round-robin buffer, which can store 65,471 files (approximately 450 days). Although the system cannot produce early warnings and information to urban citizens until network errors are resolved, it can still collect data for future analysis.

Core segment

The data sent from the remote segment are received by the input module’s FTP server. All received files are processed by the parsing module. The measurement data are collected together with the information about the station’s operation and uploaded to the database server. The current implementation supports MySQL. If any anomaly in the data file is detected during the parsing process, such events are logged and sent to the notification system, which has the task to alert of any NSUNET errors. The analysis of this type of events helps detect problems in the stations’ firmware code and increase the system’s overall reliability.

However, the parsing module does not analyze the integrity of primary and secondary data values, such as temperature, humidity, or the status value. Its task is only to obtain these values, check data for inconsistency, and ingest them into the database server. The parsing module is the “bottleneck” of the entire system. To increase the speed of the process, an adjustable buffer is used to store the collected data. When the buffer is full or no more data are left to parse, the ingestion into the database is performed.

Parsed (primary, secondary, and statistical) data are classified and stored in three different databases. This classification allows easier control of further data redistribution.

The integrity check module is implemented to perform a validity test on the data stored in the database. Currently, this process supports detection for possible “cross-table” insert errors. It sometimes happens that stations fail to send data to the core segment, causing the lack of data in the time series (Chiaradia et al. 2015). These scenarios are related to GSM modem firmware issues. Unsent files with the measurement data remain in the station’s internal memory for a long period of time and the integrity check module “missing data” test logs such errors. The results of these tests provide information about the usability of the acquired data for scientific studies.

The initial deployment time for every station can be set separately. Based on this value, the time when the station should send the data is calculated. The databases are periodically checked for such time stamps. If there is no data for the specified time period, a statistical counter is incremented. If the station manages to send the missing data in the future, the statistical counter is modified. This provides a clear picture of the suitability and reliability of data for scientific studies.

The current NSUNET implementation does not support the retransmission forced feature, which allows a station to retransmit on demand all stored data from its internal memory.

The analyzing module is currently used only for some basic statistical calculations but it may have additional functions in the future. Currently, all calculations are performed using MySQL procedures.

According to the session length, each station generates a new file every 10 min, and it is sent to the core segment. The NSUNET system receives 4032 data files every day from the stations. Working with such a large number of files stored in the same location can cause problems with further data redistribution. This problem is solved by implementing the archiving module.

Based on the measurement time threshold value, which is currently set to 3 days, the archiving module detects, prepares a list for compression, and checks for the number of currently detected and the overall number of expected files for each station.

Further data redistribution and redundancy solution

The NSUNET system supports two different methods for further data redistribution through the distribution module: MySQL server data replication and direct access to the server data. Applications built for public display purposes, such as Android, can use the direct data access method. Moreover, the distribution module is a link between the NSUNET system and the city services that can include the acquired data for their own analyses; this shows that the database is available for direct public use by local communities or stakeholders (Fig. 4).

Fig. 4
figure 4

Official GIS portal of the City of Novi Sad with the data from the NSUNET stations intended for possible use by the local government, public and private companies, and the local population

Full size image

The core segment input module is placed on two servers making it less prone to the network errors. The stations’ built-in algorithm supports sending data to two different servers (primary and secondary).

Visual monitoring and data access tool

In order to ensure that everyday work with the collected data files is feasible, the NSUNET Portal is developed so as to support two different functions. The main function allows an easy access and work with the primary and secondary data. The secondary function offers the visual monitoring of the entire system’s operation.

The NSUNET Portal highlights any data anomalies or extreme data amplitudes. Additionally, if a station’s sensors stop working, the station sends an alert by changing its status value. If the sensors malfunction and start to measure too high or too low values, this event is easily spotted through the NSUNET Portal.

The current implementation supports four different final result outputs. For preview purposes, the “screen” output can be selected. Selecting all stored values for a long period of time might produce enormous amounts of output data, which is where the remaining three output types are useful. Using one of the available delimiters, e.g., character “,” data are stored in a comma-separated value file. Possible output destinations are “file,” “download,” and “download_compressed.” The size of the selected data for approximately 1 year using a compressed comma-separated value file is 12 MB, and the time required to generate such a file is approximately 30 s. When selecting the final output destination, the “time_res” option can be used to decrease the data resolution. The main purpose of the NSUNET Portal is scientific data analysis.

The NSUNET Portal’s visual monitoring alerts if the data are older than the defined thresholds using a different color scheme, produces integrity check module calculations, and decodes the status value. The example shows that the station with ID 6-2 has a critical error due to a long period of time without input power (Fig. 5). In addition, SMS notification is sent to the technicians to alert them of the problem details.

Fig. 5
figure 5

Detection of errors in the stations’ operation using the SV monitor

Full size image

Data type and structure

The stations collect two different data types: climate (primary) data, i.e., climatological measurements detected by the stations’ sensors, and debug (secondary) data, i.e., diagnostic data about the stations’ operation. Data sent to the core segment are stored in a regular text file. Text files as the final data output make the NSUNET system different from other solutions. Data acquisition requires only a FTP server implementation running inside the input module. Files with measurements can be analyzed using any text processor.

Placing data in a plain, human-readable text file renders the NSUNET system independent of the need for sophisticated software solutions (Table 1). Data can be analyzed at any moment. This approach can be implemented as part of solutions such as open-source building science sensors (OSBSS)—a low-cost Arduino-based platform for long-term indoor environmental data collection (Ali et al. 2016).

Table 1 File naming convention and structure (text and hexadecimal values), e.g., for file: 10-1_150708_011000.txt
Full size table

The naming convention ensures that information such as station ID and measurement time (year, month, minutes) is displayed in file names and makes it easy to determine which station sent data and when. Seconds are not stored, but the last two trailing zeros in the time representation are left for future implementation.

Each data value stored in a file is defined as a constant. For example, the session length constant is stored in a file as SL:10.

Each constant occupies one line in the file. The line begins with the characters that define the constant name (two/three characters in capital letters) followed by the delimiter “:” (ASCII code 0x3A). The remaining characters until the UNIX end-of-line (line feed, ASCII code 0x0A, and carriage return, ASCII code 0xD) are constant values. Characters such as space (ASCII code 0x20) and tab (ASCII code 0x9) are not allowed. If a constant value is non-numerical, letters are represented using small letters.

The NSUNET system currently recognizes 28 constants that represent primary and secondary data. The data structure used in the NSUNET system increases the performance of the parsing process. The beginning of each line starts with a constant name that can be two or three characters long, the end-of-line is an end-of-constant value, and the delimiter between them is the character “:”. Any exception from this syntax means that an error exists in the station code, and it is easily detected by the core segment’s parsing process.

One special constant that requires careful explanation is the status value, which represents the debug data type using a 16-bit binary number. If no problems are detected during the station’s work, the status value decimal representation is zero (SV:0) that occupies only 4 bytes in transmitted data.

If a station is experiencing problems, the number of characters that define the status value can grow to its maximum value of 65,535.

The size of each file that is sent from a station is approximately 250 bytes, due to which the transmission costs in the NSUNET system are low.

Remote configuration

The initial setup for each station is performed using a universal asynchronous receiver/transmitter (UART) through the RC-232 connection to the computer. As the stations are distributed throughout the city area and mounted on lampposts at a height of 4 m, each station can also be configured remotely using configuration files. Time synchronization is performed in a similar manner. Each station has its own two types of configuration files stored in the core segment, i.e., set and get. After the station sends measurement results, it downloads its own set config file and checks for differences in the configuration. If differences are detected, the station applies the received configuration and uploads the get config file to the server as the confirmation of the applied changes.

Combining the config period, the session length, and the transmission period, the amount of data sent from the remote to the core segment can be tuned to lower data transfer costs charged by the Internet provider. The amount of data transferred by each station based on the Internet provider’s statistics is 7 MB per month.

Solving the time synchronization problem turns out to be a significant obstacle. Stations are unaware of time changes. Although the initial time can be set manually, more sophisticated and automated methods should be used. Standards such as the network time protocol could make the station code more complex and more prone to errors. Based on the concept used in the remote configuration of the stations, one special file type is defined as the time config. The core segment’s server generates the file “time.conf” on a 1-min basis, and stations collect these files during data transmission. The time config (TC) constant is used in time synchronization, and its value represents the number of seconds from the start of the Unix epoch (January 1, 1970, the Unix time stamp) in UTC.

The proposed solution offers time synchronization without the need for complex code. The drawback is that stations become aware of the time change only during transmission to the core segment. Keeping in mind the nature of the NSUNET’s climate value measurements, this is an acceptable solution.

Discussion and data evaluation

The guiding idea in developing the NSUNET system design was to make it alive and functional as long as possible (Carlos-Mancilla et al. 2016), adaptable, independent of proprietary software solutions with a strong ability to monitor, analyze, and notify different aspects of errors. The station firmware is specifically designed to meet these demands. The NSUNET stations are organized according to a star topology with only one hop distance from the core segment (Reina et al. 2013), as opposed to the mesh topology implemented in the solutions such as the Trout Lake watershed monitoring network (Watras et al. 2014). Depending on the transport protocol used in transmission between the remote and core segments, different communication devices can be used. The most reliable solution for transmission between the two segments would be wired communication, but due to the costs of such an infrastructure, the focus is placed on radio technologies, such as wireless Ethernet and the GSM mobile network. Transmission costs could be reduced by using technologies such as IEEE 802.15.4 and ZigBee protocol stack (Kulkarni and Zambare 2018; Gondi et al. 2014). Wireless Ethernet is usually free, which is not the case with GSM, even though mobile Internet offers better area coverage. The Oklahoma City Micronet (Basara et al. 2010; Schroeder et al. 2010; Basara et al. 2011) and Birmingham BUCL (Chapman et al. 2012, 2015) use wireless Ethernet in data transmission, as opposed to the NSUNET, which uses GSM to transfer data between the two segments.

Radio technologies are affected by signal strength problems resulting in sporadic data retransmissions. The priority question is not how to prevent problems from happening but rather how to address them properly. A good approach is to ensure that the amount of data sent by stations is as small as possible but with sufficient information necessary for real-time system analysis.

Delivery methods

Two possible data delivery types are commonly used: best-effort and reliable delivery. Best-effort offers a greater speed, smaller packs, with decreased reliability as a drawback. Reliable delivery requires additional data to be acknowledged with each transmission, but offers data retransmission, flow, and congestion control, which can be a great advantage in scenarios with a weak signal. In contrast to solutions such as the server-based Aginova software, which uses UDP (Young et al. 2012; Chapman et al. 2015), the NSUNET system aims to achieve the maximum reliability, and that is why the TCP protocol with error checking and congestion-controlled stream delivery is used.

To solve weak signal delivery problems, a finite number of different states are introduced into the station algorithm. Each station keeps a record of what has been sent and what should be sent. If transmission has not been successful, data are retransmitted during the following sessions. Combined together, this ensures that everyone who benefits from the NSUNET system (i.e., emergency services, urban citizens, government facilities, university researchers, and other stakeholders who use the provided data for different purposes) can rely on the validity of the received data. A strong alerting mechanism is implemented in both segments with the minimum possible amount of data used in transmission. The status value offers information about station conditions using only a few extra bytes in data transmission. The measurement data are human readable from the moment the measurements are performed.

In contrast to the Birmingham BUCL, which transfers data using UDP (on the order of 2 kB) (Chapman et al. 2015), the NSUNET successfully stores all data inside of a 250-byte file (approximately 0.24 kB), thereby meeting the requirement for small data transfer sizes. Based on the GMS Internet provider’s statistics, the NSUNET needs only 7 MB per month for each station, which makes data costs low and excludes the need for wireless Ethernet data transfer. Small amounts of transferred data increase the system’s response time when providing early alert warning notifications.

The entire core segment is based on open-source technologies and this allows anyone with sufficient knowledge to modify and upgrade the NSUNET system. Using the proposed data structure to store measurement and diagnostic data, together with a built-in adaptable algorithm, combined with the core segment’s real-time system analysis, makes the NSUNET a good candidate for UMN solution.

The NSUNET transfers the entire body of data between two segments using the FTP protocol, as opposed to transmission limited to selected data types (Chiaradia et al. 2015). Discussions have addressed whether to use FTP or HTTP in data transmission. Although both protocols support authentication and privacy using transport layer security, FTP has been the protocol of choice, mainly because it is designed primary for file transfer.

Performances and further development

Most of the NSUNET system’s automated tasks are performed by the NSUNET-SYS_TOOL, which is based on the Bourne-Again shell (BASH) used in many Unices (Unix operating systems) (Šećerov et al. 2015). The reason behind this choice was less complexity equals more reliable system (e.g., less code, therefore less potential errors in it) (Tashtoush et al. 2014). The NSUNET-SYS_TOOL is used in the initial system configuration. Modules such as parsing, integrity check, analyzing, and archiving, together with the notification system, are integrated into the NSUNET-SYS_TOOL.

A disadvantage of the entire system design is the parsing of data already stored on the core segment’s hard drives. Implementing the parsing module inside the FTP server would significantly increase the system performance. Such an algorithm is out of scope for a BASH script and would require a more sophisticated C language Linux daemon.

The guiding idea behind the development of the NSUNET Portal has been to make it highly adaptive to changes. It is completely written in PHP. A single configuration file created by the NSUNET-SYS_TOOL is the only requirement for the NSUNET Portal to generate proper page output. As a large amount of transferred data use “screen” as the final result output, the NSUNET Portal uses HTTP header compression.

Based on thorough testing, the entire NSUNET system, which is currently operating as a city-scale network (Muller et al. 2013a), with 28 measurement sites, could be expanded into a (local-scale or micro-scale) network with significantly higher station site density.

Detailed analysis of the NSUNET system

In this paper, different segments of the NSUNET system are analyzed in detail in order to demonstrate that the implemented design is able to ensure reliable early warning of extremely high or low temperatures for the citizens of Novi Sad. The Android NSUNET-Weather application is designed to alert city emergency services of such occurrences. The goal is to evaluate the effectiveness of the entire system during the periods of the year marked by extreme weather conditions. The following are the results of an analysis that started on 19 December 2016 and lasted until 28 March 2017. Since the AWS (PMF-1) station measured different data, it was excluded from the analysis. The stations (6-2, 6-5, 10-1, and D-1) experiencing technical issues during the monitoring period were excluded in order to obtain as accurate data on the overall performance of the system as possible.

Using data from the NSUNET’s system log files, 2000 sessions were extracted and used in this analysis. Since its deployment, the system was configured to send measurements once per hour. For the period of 3600 s, the expected number of received measurements was 6 (system record measurements each minute; the average values were calculated every 600 s). The delta session length (deltasl) for consecutive connections from the remote segment was calculated, which in ideal conditions using the current system configuration would be equal to 3600 s. Keeping in mind all possible latencies during data transmission from the stations to the servers, delta session extreme date (deltased) variable was calculated as the number of occurrences when deltasl values were ± 5% of its ideal value.

This analysis places focus on each station’s session length (seslen), together with the number of transmitted bytes: FTP protocol command messages data (ftpstb) and the overall amount of data transmitted through the Internet (overallstb). The transmission of station configuration files (90 bytes) together with time.conf file (14 bytes) revealed only slight deviations. The average size of measurement files was 260 bytes.

Based on the analysis of the deltasl frequency values for each station, six groups were defined, each representing a range of values (Table 2). The seventh group was defined as the control group (CG) with the most frequent number of deltasl occurrences. The sum value of all seven groups was equal to the number of cases, which was 2000 sessions.

Table 2 Seven groups defining deltasl frequency values
Full size table

Most of the consecutive sessions were started within the expected period from 3000 to 3999 s (Table 3). In the whole system, prolonged periods between two consecutive sessions were detected in group G3. An analysis of log files confirmed that most of these sessions experienced communication delays, which resulted in FTP timeouts and session retransmissions.

Table 3 deltasl frequencies per each station based on 2000 sessions
Full size table

Some extreme periods between two consecutive sessions were detected at stations 9-1 and 9-2 and they were explained by the low GSM coverage in the area where the two stations were located. The most intriguing were the stations that had frequencies in groups G1 and G2. Keeping in mind that a station had to send data at the configured intervals (3600 s) because otherwise it would have just stored the measurement data, waiting for the next defined measurement time, the only possible conclusion was that the station firmware contained a bug. The stations that experienced this issue were 5-3 and A-1.

The focus was further placed on the minimum and maximum values of two consecutive sessions (Table 4). The observations showed that each station had a large range between the minimum and maximum deltasl values, due to which standard deviation was used to analyze the transmission pace for each station.

Table 4 Minimum, maximum, and standard deviation for deltasl values for each station in the remote segment (with or without THE)
Full size table

The first thing to be noticed were the standard deviation values for stations 9-1 and 9-2. Although the city area where the two stations were placed had a moderate GSM coverage (in general, in Novi Sad, the GSM coverage is the best in the city center), the results indicated that these two stations had no more than a few high deltasl values with only one high and one extreme value per station. Apart from those few examples of anomalies, the deltasl values recorded at stations 9-1 and 9-2 were the closest to the expected session length values in the entire system. Stations 5-3 and A-1 had the most unpredicted data transmissions in the entire system. Station A-1 was located in a forest, between Kać (a settlement near Novi Sad) and Novi Sad, while station 5-3 was placed in Banatić, a highly urban part of Novi Sad. As extremely small intervals between two consecutive sessions were observed, it was concluded that these two stations had issues in the firmware code. The same conclusion was made in case of station 6-1. The total number of system deviations from the configured 3600-s intervals between two consecutive sessions for the period of 100 days was 764 (Table 5).

Table 5 Deviations from the configured 3600-s intervals between two consecutive sessions for each station
Full size table

A global analysis of the system revealed that most deltasl were concentrated around the configured value of 3600 s. The overall number of bytes needed for communication between the remote station and the server was in general constant, as expected. A small amount of transmitted data (around 3 kB) confirmed that the system was able to transfer the measurement data, synchronize in a timely manner, and configure files reliably, while keeping the Internet provider data costs at a low rate. The disturbing fact was that the average session length to transfer such a small amount of data was around 3 min. A close inspection of the stations’ source code revealed the use of the delay function as a solution to hardware problems.

The core segment produced a more balanced pace. The time used by the NSUNET-SYS_TOOL to process data (parst) was in general less than 1 min. In extreme cases, when the remote segment had large deltasl values, the parst had an extreme value of 932, but this happened only once. The frequencies from 65 to 160 s were registered no more than 31 times with an average value of 1.84.

In general, most stations managed to send data at regular intervals, proving that the system could provide reliable early warning to emergency services. Long sessions followed by a FTP server timeout message resulted in zero-sized files (files created upon a successful station FTP connection that failed to transmit the entire measurement content). These measurements were received in future sessions but were detected by the system. Over the 100-day period and in 12,300 NSUNET-SYS_TOOL occurrences, 2223 zero-sized files were detected.

The results of the core segment analysis confirmed that there were slight variations in the number of processed files (filesp) during the monitoring period and that they coincided with the periods when the NSUNET-SYS_TOOL was running. The FTP timeouts between the remote and core segments resulting in zero-sized files were generally few. Only a small number of sessions lasted longer than average and had extremely high values. Communication issues between two segments such as interruptions in the Internet traffic between the GSM and University networks caused such errors.

Statistical analysis of the NSUNET measurement data

The NSUNET provided air temperature and relative humidity differences between the built-up and land cover areas, as well as the differences among various built-up areas. Hence, the main purpose of the NSUNET system was to monitor the climate values, mostly during the extreme events (i.e., heat waves or cold waves), as well as the outdoor human thermal comfort conditions. This type of data is useful for public utility companies for heating distribution, healthcare, and urban development planning that are already receiving data using the NSUNET’s third-party segment.

In this study, the analysis is focused on data from the summer of 2015, as one of the hottest summers in the past several decades, when the magnitude of warming was comparable to previous hot summers in Europe, e.g., in 2003 and 2010 (Dong et al. 2016). Figures 6 and 7 show air temperature differences among the LCZs based on the data from the entire summer period and from two most intensive 9- and 12-day heat wave periods, respectively.

Fig. 6
figure 6

Boxplot with 1-h air temperature measurements in LCZ classes in Novi Sad between 1 June and 31 August 2015

Full size image
Fig. 7
figure 7

Boxplot with 10-min nocturnal air temperature measurements (from 9 PM to 5 AM in CET) in LCZ classes in Novi Sad from two intensive heat waves. (a) July 17th to 25th. (b) August 4th to 15th

Full size image

The station measurements from different LCZs show that the highest temperatures were registered in the most built-up and most populated urban areas of Novi Sad (LCZs 2, 3, and 5). Toward the suburban neighborhoods and outskirts, the temperatures were lower, going below 24 °C (LCZ 9). The lowest air temperatures were above the land cover areas, LCZs A and D, with 22.8 and 23.4 °C respectively (Fig. 6).

The nocturnal (from 21 PM to 5 AM in CET) 10-min air temperature measurements during the heat wave periods (Fig. 6) showed higher differences between the built-up and land cover areas. Furthermore, the highest average temperatures were registered in LCZ 2 and the lowest in LCZs A and D.

During both intensive heat waves in July and August, the highest average air temperature difference (5 °C) was observed between a densely built-up area (LCZ 2) and the land cover areas (LCZs A and D). The main differences between the built-up and land cover concerned the maximum and minimum temperature values (Fig. 7a, b). The intra-urban comparisons showed that the highest temperatures were mostly registered in urbanized areas (LCZs 2, 5, and 8), whereas in the suburban zones and outskirts (LCZs 3, 6, 9, and 10), the temperatures were lower. The temperatures registered in the land cover areas were the lowest (Fig. 7a, b).

The outdoor human thermal comfort conditions in Novi Sad were assessed based on the NSUNET data from a heat wave period that lasted from 4 to 15 August 2015. Over a day (24 h), the largest differences were registered between the most built-up area of the city (LCZ 2) and a dense forest (LCZ A) outside the city, with values reaching 5.1 °C. The thermal comfort differences inside the city were lower, reaching the values between 0.5 and 3.3 °C. The largest differences between urban and natural LCZs occur during nighttime period with values up to 7.6 °C (Fig.  8 ). Quite the contrary, the differences registered during daytime were more than twice smaller. In general, it can be concluded that thermal comfort conditions are improving as the proportion of urban surface decreases and the proportion of green areas increases, especially during nighttime in heat wave periods.

Fig. 8
figure 8

Differences in outdoor human thermal comfort conditions between urban LCZs and natural LCZ A outside Novi Sad expressed using the Physiological Equivalent Temperature index (°C) on: 24 h, daytime, and nighttime level during the heat wave period (from 4 to 15 August 2015)

Full size image

Based on the presented data and graphs, the NSUNET provides adequate measurements and the database for further intra-urban and inter-urban air temperature and thermal comfort comparisons, which is very important in urban climate research.

Conclusions

The Novi Sad Urban Network (NSUNET) for monitoring urban climate issues has been developed as a result of the international European Union project URBAN-PATH (project grant: HUSRB/1203/122/166). The network consists of 28 stations located in an urban area of 102 km2. The NSUNET’s long time series monitoring seeks to deliver measurements for intra-urban and inter-urban comparisons. The temporal resolution allows the exploration of both diurnal and seasonal peculiarities. The acquired knowledge is expected to contribute to the effectiveness of sustainable development and climate-conscious urban planning strategies, the mitigation of the impacts of global climate change, and the maintenance of the health status of the urban population. The results of the analysis of the data collected through the NSUNET system show that the thermal load (air temperature and outdoor human thermal indices) is higher in densely urbanized areas than in suburban and land cover areas.

The system’s design offers to urban citizens early warning of extreme weather conditions. It also provides the most reliable information about the current weather conditions in the city’s urban area. Measuring long time series needed for scientific studies and providing current information useful for everyday life make the NSUNET system a solution suitable for the needs of both public and science institutions.

Recent research related to the UMN generally offers little or no information on the structures used for sending data between stations and servers. Furthermore, the algorithms stored inside the entire systems that are used for system error notification, retransmission, and permanent data loss problems are not appropriately covered in the literature. The lack of relevant information degrades the development of urban monitoring systems in other cities.

The maintenance of the UMNs involves significant costs over the long period of time required to build a database that can be used in scientific studies. Often, the implemented solutions cannot be further developed without additional funds, and this situation leaves the developed systems in an unchanged state, resulting in degradation over time.

This paper provides detailed information required to develop a reliable system for urban meteorological studies together with reliable early warnings about current urban weather. The proposed data structure offers a reliable solution for data transfer between stations and servers. The migration from the currently used GSM to WIFI data transmission would require minor changes to the station hardware. The measurement data are stored in a plain text format and can undergo different system modifications.

The NSUNET system is designed with minimal points of failure. Redundant and backup solutions are implemented on both segments. Each station can store approximately data collected over 1 year, which enables a great improvement in the reliability of long-time urban climate change studies.

Servers are placed in two separate locations to overcome network-related problems. Using different configuration files, the amount of data sent during NSUNET transmissions can be finely tuned for different types of networks. For example, in environments with low signal coverage, the system can be set to send smaller amounts of data over longer periods of time. The cost-effective data transfer (around 7 MB per month per station), together with an open data structure, distinguishes NSUNET system from other similar solutions.

Although the system requirements have been to monitor the urban climate, the NSUNET system design is not limited only to this purpose. The proposed data structure makes the NSUNET suitable for monitoring different environmental or social conditions in an urban area.