Keywords

1 Introduction

Sensors are an important part of data producers, especially in the field of spatial data. There are large number of sensor data available on the Web in variety of formats encoding. Wide-ranging services and solutions for sensor data publishing for different type of sensors are usually very complex and complicated for non-expert users. The authors address this problem by designing tools and processes for effortless and straightforward utilization of existing sensors available on the Web. This work has been done in the scope of the Open Sensors Network pilot as part of the SDI4Apps projectFootnote 1.

Reusing existing sensors operated by third-party providers is currently problematic. The problems range from initial search of services through metadata description of sensors to filtering of found sensor candidates. It could be understood as searching for suitable repairman company in yellow pages. In the same way, it would be useful to find some “yellow pages” for searching appropriate sensors and sensor data producers on the Web. A good starting point for reusing sensors is to find the rate between usability and complexity of sensor services and providing them straightforwardly. The SensLog application described in following paper can be given as an example of sensor data management for small and medium sensor networks [1,2,3,4]. SensLog serves a data provider as a tool for collecting, processing and publishing of sensor data on the Web. SensLog allows to receive measurements from sensor networks (from sensors, gateways or by intermediate daemons) through defined web services. Received data are processed and stored in a data storage. Stored data can be published for data consumers by different web services in various formats and metadata of registered sensors can be exported to semantic sensor catalogues together with the endpoint description [5, 6].

2 SensLog

SensLog is a software component for sensor data management on the Web. SensLog receives sensor data by requests over HTTP protocol and stores them in a database. SensLog publishes raw and/or processed sensor data using several interfaces. One of the communication interfaces is proprietary-defined but extensive RESTful API [9] publishing data mostly in the JSON format. Another interface is based on the core methods of the OGC Sensor Observation ServiceFootnote 2 version 1.0.0 standard [8]. SensLog has also a simple graphical user interface (GUI) with several functions to visualize data. However, the application is intended to be used in combination with other standalone clients.

The SensLog application was designed as a modular solution from the very beginning. New modules were developed and tested during contemporary projects including SDI4Apps, FOODIEFootnote 3 and OTNFootnote 4). A module for collecting volunteered geographic information (VGI) was designed and developed with the emphasis on character and properties of such type of data. The expansion of precision farming utilization has affected the design and development of another SensLog module for telemetry of agricultural machinery. Both new modules are based on the SensLog core functions, methods and modules providing extensions to the current interfaces. The server-side part of SensLog is written in the Java programming language. The database part is written in the PL/SQL language and the Web GUI uses the JavaScript jQuery library.

2.1 The SensLog Data Model

The SensLog data model is based on the OGC Observations & Measurements 1.0 standard [7]. The model was further extended based on requirements provided by the SDI4Apps, FOODIE and OTN projects. The SensLog data model is used for storage of raw sensor data and results of data analyses. A diagram of the last version of the SensLog data model is shown in Fig. 1. The core tables of the model include units, sensors, phenomenons, observations and units_positions. The object unit represents a physical device that has got one or more connected sensors which are producing observations. A unit can be equipped with a GNSS chip to periodically determine its own position (e.g. in case of mobile devices monitoring), or the position of the unit can be defined by another method (e.g. in case of static in-situ monitoring).

Fig. 1.
figure 1

Diagram of the SensLog data model

Full size image

The SensLog data model was recently improved especially for potentially large tables. The observations and units_positions tables are the largest ones. Their volume is growing in coherence with continuous data monitoring. These tables were supplemented with a table partitioning mechanism (shown as series of tables in Fig. 1). The core SensLog model was extended by tables defined by the new VGI and telemetry modules. The consistency of stored data is ensured not only by the referential integrity but also by the system of triggers and stored functions. The consistency is ensured especially in the case that extension tables are stored in separate database schemas. In this way, child tables of partitioned observations and unit positions allow better querying, backing up and maintenance. The SensLog data model was implemented in RDBMS PostgreSQL 9.x+ with spatial extension PostGIS 2.x+.

2.2 SensLog VGI Module

The importance of VGI is growing in present days and current projects brought requirements on incorporating this type of data into SensLog. The character of VGI does not allow to store such type of data directly to the present SensLog data model without loss of information. Therefore, a new module for VGI was designed and implemented. This VGI module contains new services for receiving and publishing data over the Web. The SensLog data model was extended with new tables with emphasis on variability of VGI. A VGI observation is characterized by several mandatory attributes and can be enriched with additional attributes. It’s not necessary to define the final number and names of additional attributes before starting collecting VGI. The datatype of an additional attribute is only limited to data types which values can be stored in the JSON format. A VGI observation can include a list of multimedia files that are also stored in the data model. A schema of tables of the VGI module is shown in Fig. 2. A common type of VGI is a point of interest (POI). The concept of POI is similar to the general observation concept. The implemented scenario defined by the SDI4Apps project considers a POI as a spatially and temporally localized observation with several attributes and an array of connected multimedia files. Each POI is classified into one of the predefined categories and is sorted as a user-defined dataset.

Fig. 2.
figure 2

The data model of the SensLog VGI module

Full size image

The tables added by the VGI module include:

  • observations_vgi – this is the main table that stores VGI observations with all attributes except for multimedia content,

  • observations_vgi_media – this table stores multimedia files connected to VGI observations,

  • observations_vgi_category – this table stores categories of VGI observations, the table uses a partitioning mechanism to sort categories,

  • vgi_datasets – this table stores user-defined datasets with dataset metadata.

The VGI observation has got relations to some core tables of the SensLog model. The information about the device that produced the VGI observation is stored in the units table. The information about the user is stored in the system_users table. The main VGI geometry is a type of point and it is stored in the units_positions table. The geometry of VGI represents the point where the VGI observation was localized. An additional geometry of different type can be stored as an additional attribute. The categories of POIs can be stored in tables inherited from the master table observations_vgi_category. This enables to store different kinds of categories in a hierarchical order.

2.3 SensLog RDF Export of VGI

In order to publish data according to the Open Linked Data best practices and with a self-describing data structure, the Virtuoso graph model engine is used. The data are stored in quads, which consist of a graph name describing a dataset and a triple which maps an attribute value (object) to POI (subject). The mapping is done by using a property which is in the best case scenario defined in some public ontology thus making it easier to integrate data with other systems. The data can be queried and inserted into the quad-store through the SPARQL [11] endpoint at http://data.plan4all.eu/sparql or uploaded and parsed on the server via RDF [10] files if the amount in one batch exceeds 100 megabytes. An insert operation for one POI can be seen in the following example:

figure a

The data visualization is performed through a web-based map client based on the HSLayers-NG library (Fig. 3). Map layers are dynamically generated for each graph (dataset) and category (http://gis.zcu.cz/SPOI/Ontology#bus_stop in the above mentioned example in Fig. 3) which can be found in a particular dataset. Displaying of the triples is based on principle that each subject in a triple is displayed as a new vector (point) feature in the map. Likewise each property and object is converted to the feature’s attribute and value in GIS terms. Styling of the features such as choosing an icon and color is based on attribute (category) values and is application specific.

Fig. 3.
figure 3

Visualization of POIs in the HSLayers-NG client

Full size image

2.4 SensLog Interfaces

The publication of stored data is provided by the system of web services. The system of web services allows publication of stored data in different levels of detail and in various export formats. The modularity of SensLog allows to have a different interface for each module with specific web services according to the character of the following application. The diagram in Fig. 4 shows the structure of SensLog modules along with the corresponding interfaces and formats provided by each module.

Fig. 4.
figure 4

Diagram of the structure of the SensLog modules

Full size image

On the one side, there is the feeder module for inserting new observations and positions through methods of the HTTP protocol. The feeder module is independent from other modules which enables having an allocated database connection for continuous insertion. Only authenticated clients are allowed to insert new objects.

On the other side, there is provider module that contains the system of web services via methods of the HTTP protocol to publish stored data. The provider module uses mainly the RESTful interface publishing data in the JSON format in a proprietary-defined form. The provider module contains also a standardized interface based on the core methods of OGC SOS 1.0.0. The proprietary interface contains a wider range of services with a variety of data formats.

The VGI module has also the RESTful interface defined with an emphasis on character of the VGI observation object. A VGI observation can be inserted encoded in the JSON format or as a multipart form of data. The insertion of a new VGI observation is variable due to the possibility of pushing VGI observations from mobile applications. There is an option to store a plain VGI observation with mandatory attributes and to add multimedia contents afterwards (e.g. while having a better Internet connection). The stored VGI observations can be published encoded in the JSON format in a proprietary form or in the GeoJSONFootnote 5 format as the Feature or FeatureCollection object. The connected multimedia files are addressed in a form of a direct URL. An exporting web service according to the requirements from the SDI4Apps project was designed. This exporting service publishes stored VGI observations encoded in the RDF format to be further processed in the Virtuoso system.

The telemetry module uses mainly the previously described feeder module for inserting observations and positions. The telemetry module defines only standalone web services for publishing analysis results on utilization of machinery and farm fields. The data are provided in the JSON format in a proprietary form or in case of any listings, data are provided in the textual CSV format.

The import of sensor data to SensLog is provided by the services of the feeder module in most cases. The pattern of services of the feeder module is straightforward without the necessity of any specially object mapping. An ordinary scenario is to insert data to SensLog when the data are observed. In exceptional cases, especially when the data connection is lost for longer time and observations are restored from flash memory of gateway, data can be pushed into the SensLog storage by a script in a batch. In case of very long time series that need to be imported, it is possible to import data directly to the database model by PL/SQL scripts.

3 SensLog Applications

The modularity of SensLog allows its use in a variety of applications. Currently, the authors are supporting three types of sensors as pilots including the monitoring of the level and quality of ground water, utilization of a wide spectrum of sensors in precision farming and utilization of SensLog as a gateway for collecting VGI directly by users.

3.1 Groundwater Monitoring

The water quality is dependent on several aspects including the crop type, irrigation method, soil types, groundwater levels, soil and water chemistry, nutrient loads, limits on chemicals, salt tolerance of crops, leaching of salts and management of drainage water. There are several barriers to the reuse of wastewater in agriculture. The key barrier is that many stakeholders do not view wastewater as a resource, even if adequately treated. The stakeholders see the energy costs for treating wastewater to an adequate standard as being prohibitively expensive. One of the aims of the project pilot is to utilize recent innovations to turn wastewater reuse into a profitable, socially beneficiary and environmentally safe solution. A number of phenomena are monitored in this pilot including chemical and physical properties of the surface and underground water.

The observations are produced by sensors deployed in shallow wells and they are sent by gateway stations on the ground in defined intervals. The observations are processed and user-defined analyses are run. There is a system of alerts that is triggered in case of reaching some defined thresholds. The services of the provider module are publishing not only measured data but also results of analyses and alerts.

3.2 Agrometeorological Monitoring

Agriculture requires collecting, storing, sharing and analyzing large quantities of spatially referenced data. Such type of data, to be used effectively, must be transferred between different hardware, software and organizations. These data flows present a hurdle to uptake of precision agriculture. The use of multitude of data models, formats, interfaces and reference systems is resulting in incompatibilities. The management of huge amounts of data is a challenge. Spatio-temporal data is increasingly collected by remote or in-situ sensors rather than by field campaigns. Wireless communications have several benefits, but also pose challenges to the data exchange reliability and power supply. Sensor calibration and deployment as well as the maintenance of sensors need resources and technical skills and increase the costs of data acquisition. The increasing of both the data amount and the data quality awareness issue highlighted importance.

An important part of using sensors in agriculture is represented by tracking of agriculture machinery to provide analyses oriented not only on utilization of monitoring of particular machinery but also on intervention monitoring of a particular field. The combination of machinery data together with field information provides a higher level of economic evaluation. The screenshot in Fig. 5 shows a client application visualizing results of analyses provided by the SensLog telemetry module.

Fig. 5.
figure 5

Client application for visualization of analysis results

Full size image

3.3 VGI and Citizen Observatories

A human as a sensor, also referred to as VGI monitoring, represents collecting measurements and observations by users by portable devices. It is a typical way of collecting spatially referenced thematic data in the domain of Earth observations. The characteristics of such type of observations include high variability of attributes of not only primitive data types and an uncertainty in data accuracy. An example of collaboration between pilots is shown by collecting of POIs. The Smart Points of InterestFootnote 6 (SPOI) [5] data set was developed in the frame of the SDI4Apps project providing information as linked data [6]. SensLog became a segment in the chain of collecting and updating user-defined POIs. SensLog provides services to push new POIs collected by users in the database on the one side and it provides export services to pull sets of POIs to final storage based on the Virtuoso serverFootnote 7 on the other side. Screenshots of client mobile application for collecting POIs is shown in Fig. 6.

Fig. 6.
figure 6

Mobile application for collecting POIs. Main screen on the right and attributes of the selected POI on the left side.

Full size image

4 Conclusions

SensLog is an application for sensor data management and it is used operationally in a number of solutions and pilots. SensLog provides a system of web services for both data producers and data consumers. A stress was put on finding a good ratio between the usability and generality of these web services during the design and the development phases of the solution. SensLog is an effective way how to combine the OGC standards [7, 8] with a light API for development of subsequent applications. The SensLog data model was designed to store observations produced by variety of sensor types. Extensibility of SensLog by additional modules was confirmed during pilot applications in the current projects. Additional modules for agricultural machinery telemetry and for collecting POIs by smart devices were developed.

The authors plan to add an additional module that will be oriented on devices from the Internet of things domain. One of the main tasks will be to implement the OMAFootnote 8 NGSIFootnote 9 specification which is a primary interface for Generic Enablers in FIWARE.