Keywords

1 Introduction

There are numerous earthquakes [1] around the world every year. Although the strength of most of them is insignificant (thus posing no threat to human society), those rare but powerful earthquakes are known for their devastating consequences and cannot be ignored. The number of victims from one such seismic event can number in the hundreds of thousands of people. To avoid these drastic consequences, researchers from different fields of science set a common goal to find an effective way to predict the earthquakes. Despite their considerable efforts, it is still impossible to predict with absolute accuracy, the date and time when a seismic phenomenon will begin.

The seriousness of this problem lies not only in the fact that the inaccurate detection of seismic hazard can lead to numerous human casualties and material losses, but also that actions taken due to a false alarm may unnecessarily damage the economy or create skepticism regarding warnings for individuals in the future.

A significant part of earthquake prediction methods is based on information about earthquakes that happened before. In this case, the more data, the higher the probability of an accurate prediction. Such methods involve the processing of a massive flow of information. It is advisable to analyze not only the events that have already occurred but also process the reports obtained as a result of monitoring the earth’s surface in real time [2,3,4]. New methods of monitoring are being developed to track the so-called earthquake precursors. Such methods of forecasting include the monitoring of changes in seismic wave velocities, the monitoring of groundwater levels, or radon and hydrogen content in rocks. In addition, studies aimed at finding the correlation of earthquakes with the motion of the earth’s crust and a change in the concentration of ions in the ionosphere have been carried out, as well as the monitoring of changes in the behavior of biological objects (animals) as the seismic event approaches [5,6,7].

However, despite multiple types of research and the development of various systems of seismological monitoring using all-pervasive sensor networks, as well as artificial intelligence technologies and neural networks [8, 9], this issue remains open.

As it has been mentioned above, there is a theory that the behavior of animals can be a kind of indicator of natural disasters. When examining the results of numerous observations and experiments, researchers were able to determine that some animals are capable of predicting earthquakes more than a day in advance. A few of them might even be aware several weeks before the occurrence of a natural disaster (in particular, species of fish).

Considering the behavior of animals (as it relates to earthquakes) it is necessary to pay attention the sensitivity of the ‘head compass,’ that is, its ability to capture electromagnetic oscillations in response to the approach of an earthquake in different species of animals.

Thus, the mountain goats of the Sierra Nevada range descend from the highland pastures to the plains several days before the shocks. Other examples include lizards crawling out of their cracks into open spaces or foxes and wolves leaving the forests.

If we compare the response of the ‘seismic sensor’ to stimulation by electromagnetic or magnetic fields in humans and animals, we can see that there are differences between them, mainly due to the sensitivity under exogenous effects. In animals, the sensitivity threshold of the ‘sensor’ is much lower than that of a human.

It can be assumed that with a higher organization of the organism (human), the realization of its functions in ontogeny is provided not only by morphofunctional specialization and integration of regulatory systems but also due to the high sensitivity threshold of the ‘seismic sensor.’

Many species of living organisms react to the upcoming seismic event with a genetically determined ‘behavioral repertoire’, despite the fact that quite often the dominant behavioral responses are unpredictable in view of the complex nature of their behavior. Therefore the problem of observing and deciphering the unusual behavior of animals remains relevant.

Many factors, even including ordinary ones (for example reaction to another animal), may cause non-standard behavior in animals. How then to distinguish the reaction of animals to local stimuli from a reaction to an impending earthquake? To answer this question, the project ‘IoT Based Earthquake Prediction Technology.’ was initated. The main goal of the project is to create a technological basis for monitoring a large number of animals simultaneously. Widespread data collection is a key aspect of the suggested approach and is supposed to reveal the presence of a systemic cause in ‘not normal’ behavior of animals, minimizing the impact on the result of commonplace or accidental causes.

This article is a description of the first stage of research and development conducted by the authors aimed at creating a client-server software and hardware system that allows to analyze automatically the behavior of biological objects. The task of the system is to monitor a large number of animals and perform online analysis of their behavior on the server. Among the parameters to be monitored are the motor activity, the level of noise created, the pulse, and other factors.

The information obtained as a result of monitoring of biological objects can be used in conjunction with existing methods for predicting earthquakes providing a more accurate result.

The article is organized in the following way: Sect. 2 gives a brief survey on the technologies of animal’s behavior monitoring. Section 3 considers an architecture (software) of the system and integration with existing systems. Section 4 describes a subsystem for monitoring of the activity of animals using wearable inertial sensors. Section 5 describes a subsystem for birds and fishes monitoring based on the use of computer vision (OpenCV). In conclusion, the results of the work are analyzed, with ideas for future research being considered.

2 Related Works

Much development has been suggested to study the behavior of animals. Some of it involves the use of video cameras and algorithms for analyzing the images and recognizing various behavior patterns [10,11,12]. The main difference between these works is the type of animals on which the authors are focused, which gives specificity to the algorithms being developed. It is worth highlighting the work of Kriti Bhargava and others, which suggests using fog computation technology to perform all necessary calculations, pattern recognition, and behavior analysis of living organisms, among other studies. [13].

The use of various miniature devices (sensors placed on the body of the animal) has become a more economical solution. It is assumed that such sensors can measure parameters, the analysis of the changes of which can help to determine deviations from the normal behavior of animals. In [14], a device that is a wireless accelerometer measuring 3.2 × 2.5 cm, weighing 10.2 grams and a battery capacity of 150 mA, and an algorithm for recognizing behavior has been developed. Such an accelerometer turned out to be small enough to be used to monitor the activity of laboratory rats and was attached to them as a homemade vest. In the course of the experiment, the researchers obtained three models of the behavior of rats: Eat, Stand, and Groom. The authors note that grooming is of particular interest, because as a rule, rats cease to care for themselves when they are sick. Thus, with the help of the results of the work done, it is possible to monitor not only the behavior but also the health of the animal.

In addition, using the wireless sensor network (WSN), it is possible to track the behavior of not just one, but whole groups of animals and their social behavior. Handcock and others in [15] suggested putting on collars with built-in GPS-modules on cows for obtaining images from the satellite, providing the trajectory of the herd movement, the interaction between individuals, and formation of local groups.

In [16], like in [14], it is supposed to use wireless devices, which include an accelerometer and a gyroscope. A prototype of such a device is presented in the article, as well as an algorithm for calculating the level of activity of an animal. In accordance with this algorithm, the activity level is determined by the sum of the activity levels for all three axes. In turn, the activity along the axes is calculated as the average value of the deviations of the accelerometer values from the average value on this axis for the last 60 values.

Chandrakant and others put forward a new architecture for an earthquake prediction system. It is assumed that a system analyzing the unusual behavior of animals will be integrated with all existing systems, such as electromagnetic signals detectors, accelerometers detector for volcanism, air ionization detectors and enhanced infrared radiation detection images by satellite, a system analyzing the unusual behavior of animals will be integrated. In addition, a new algorithm is proposed, according to which danger can be determined and an alarm signal is given. The results of the experiment graphically represent the level of activity of dogs, cats, ducks, and hens, as well as readings of accelerometers detector for volcanism, air ionization detectors, and enhanced infrared radiation detection.

In [17, 18], a network architecture based on ZigBee is developed using a neural network for recognition of the patterns of behavior of animals derived from data from sensors. The neural network [18] is also described, and the simulation of energy consumption is carried out.

As can be seen from the analysis of previous works, the researchers have achieved considerable results in monitoring the behavior of biological objects, but not significant enough to be realized in the prediction of earthquakes. In this article, it is proposed to supplement and improve existing achievements through the use of technologies and methods of Big Data, which implies the ability to monitor a vast number of objects. This will allow avoiding erroneous conclusions when analyzing the causes of deviant behavior.

3 The Architecture of the System

The use of living organisms as earthquake sensors should complement the existing developments in the field of seismic activity prediction and the emerging developments. To achieve this goal, it is necessary to design the application architecture in a modular way (Fig. 1). Such architecture would allow easy integration of various elements using standard interfaces.

Fig. 1.
figure 1

The architecture of the earthquake prediction system

Full size image

Data collection is carried out with use of IoT technologies. The data collection aspects are discussed in Sects. 4 and 5. For data collection, both existing and emerging IoT devices and technologies can be used. In order for applications to work successfully, gateways require both: converting data between protocols of data link and the network level, and providing compatibility at higher levels. This provides interoperability of devices using various protocols. Devices providing conversion at the semantic level (the application layer according to OSI model) are called semantic gateways (SG). They allow transformation between messaging protocols such as CoAP, MQTT, XMPP, HTTP v.1.1, HTTP/2, these protocols are widely used in IoT. The Gateway as Service Architecture for IoT Interoperability is considered in [19]. The alternative approach to already known principles suggested in [20], it is the employment of the SIP protocol as a container for M2M data. Using an SG in the system provides full connectivity with various types of IoT devices (animals monitoring devices).

The next element is the processing and storage unit. This device collects, analyzes and stores the data obtained from monitoring devices and provides access to the data for users (animal owners) and the Earthquake Prediction Server. At this stage of analysis, the non-typical and non-standard behavior of the monitored objects is revealed. The data is used by the Earthquake Prediction Server to compare with data from existing geological and environmental monitoring systems, presented in the form of a time series, to identify the correlation between geophysical factors and animal behavior. The time series trend selection is the first step in the analysis. Splitting the signal into “trend” and “remainder” is based on the assumption that smooth, low-frequency changes are due to the influence of some deterministic the reasons, while the relatively high-frequency oscillations reflect a random component of the process. The presence of a correlation between the identified trends in geo-monitoring and non-standard behavior of animals may be evidence of a pending earthquake.

Another function of the processing and storage unit is an analysis of data regarding animal health (a fitness tracker for animals) and providing the results of this analysis to animal owners in the form of a mobile application or website. This function is necessary to create a strong motivation for the owners of animals to use the proposed system. Such an approach may help to involve a more significant number of animals to the data collection process.

4 The Subsystem for Monitoring of the Activity of Animals Using Wearable Inertial Sensors

As follows from Sect. 2, one of the characteristic reactions of the animal to external stimuli is motor activity. To monitor motor activity, it is necessary to register movements in the timescale. This can be done both with the help of global positioning (GPS and others) and with the help of autonomous inertial positioning systems. Such systems are based on the use of accelerometers and gyroscopes. From the data obtained, it is possible to calculate (within an acceptable margin of error) the distance traveled by the object and even recognize specific movements.

The development of a subsystem for monitoring animal activity with the help of wearable inertial sensors is necessary for the collection and analysis of data on the motor activity of animals that live near humans (such as domestic dogs and cats), as well as farm animals like horses, cows, or sheep.

4.1 Data Collection

To carry out the experiment on data collection, a prototype of the data acquisition device was developed - a smart collar (Fig. 2). This device contains an accelerometer, a gyroscope, a massive battery pack and a NodeMCU microcontroller, which is an MQTT client. The server on which the data is processed was also an MQTT client and received it through the MQTT broker. The data received was subjected to low-frequency filtering, in order to separate the gravitational component of the signal.

Fig. 2.
figure 2

A dog with a prototype of the device for data collection during the experiment

Full size image

During the experiment, activity data came to the server in CSV files containing a timestamp in the UNIX format and acceleration on three axes.

In connection with the fact that in future it is supposed to apply the methods of Machine Learning, the data was collected in three stages. The first stage, ‘Walking,’ implied walking down the street with the owner. The second stage, ‘Playing,’ was an artificially initiated (by the owner) intense action. The third stage, ‘Rest’, - the collection of data was carried out in conditions that excluded the presence of artificial factors that might stimulate activity. Each record has an ‘Activity’ parameter, which takes into account the stage at which data were collected (Table 1). This parameter is necessary for further training the neural network.

Table 1. The form for presenting activity data
Full size table

In total, about 262,420 lines with records of activity were collected in the resulting database. Figure 3 shows the visualization of activity data characterizing various types of activity. As seen in the graph, the dataset is quite different for various activity types.

Fig. 3.
figure 3

Visualization of activity data

Full size image

4.2 Recognizing Activity Using a Recurrent Neural Network

To improve the quality of the animal activity analysis, it is necessary to exclude periods of activity caused artificially (walks, play) from the sample. To solve this problem, artificial neural networks have been used. The development and training of the neural network were carried out with the help of Python.

A recurrent neural network with Long Short-Term Memory (LSTM) was chosen as a neural network architecture. The LSTM network is universal in the sense that, with a sufficient number of network elements, it can perform any calculation that a conventional computer can do, which requires an appropriate matrix of weights that can be considered as a program.

The LSTM network receives a fixed length data entry, so the data is divided into 200-line segments or 10 s. Activity tags are converted to the unitary code. Data is divided into training and practice sets in the ratio of 8:2.

The network model contains 2 fully connected layers and 2 LSTM layers, each of which contains 64 hidden nodes.

The hyperparameters of learning:

  • Optimizer: Adam

  • Number of epochs: 50

  • Number of samples per iteration: 1024

  • Learning speed: 0.0025

During training, the parameter ‘accuracy’ was monitored - the function inverse to the error function in recognition of activity and the parameter ‘loss’ - cross entropy determining a slightly near-predictable distribution to the true one.

It is expressed as follows (Fig. 4):

Fig. 4.
figure 4

Machine learning process

Full size image

As a result of learning, the neural network recognized the type of activity, with an accuracy of 97%. The codes developed as a result of the program and the source code are available for public access [21].

The considered method can be used for recognition of activity data, periods that are caused by artificial factors. Excluding these periods from the collected data, provides a data set that can be used to identify non-standard behavior.

4.3 Calculation of Motor Activity

To determine the deviations of the animal’s behavior from the norm, the parameter ‘level of motor activity’ has been used in this project. For the case with a wearable inertial system, this parameter was calculated as the sum of the activity on all the three axes of the accelerometer. The activity on the axes is calculated as the average value of the deviation of the acceleration from the average value on this axis in the past minute.

$$ A = \frac{1}{n}\sum\nolimits_{i = 1}^{n} {x_{i} - m(X) \vee } + \frac{1}{n}\sum\nolimits_{i = 1}^{n} {y_{i} - m(Y) \vee } + \frac{1}{n}\sum\nolimits_{i = 1}^{n} {z_{i} - m(Z) \vee } $$
(1)

Where n is the number of measurements made per minute, m is the average value of the accelerometer readings on the X, Y, and Z axes.

The values ​​obtained are compared with the average activity at the same time in the previous days, as well as the average activity for the previous day, week, month, and year.

5 The Monitoring Subsystem Based on Use of Computer Vision Technologies

It is challenging to apply inertial methods of monitoring activity to some animal species. These animals include aquarium fish and animals in cages (birds, mice, rats, etc.). For these, it is necessary to develop a subsystem that allows monitoring activity without the use of wearable sensors. Taking into account that the area of ​​movement of a biological object is limited to certain boundaries (an aquarium or a cage), computer vision systems can be used to solve the task of monitoring activity. In this project, the library of OpenCV computer vision algorithms has been used.

The OpenCV-based monitoring subsystem in this project consists of two modules:

  1. 1.

    The module for detecting and recording the trajectory of the movement of a biological object.

  2. 2.

    The module for storing and analyzing the trajectories of objects.

The module for detecting and recording the trajectory of a biological object is a device and software providing the ability to identify objects using the camera, and also record the trajectory of the motion of detected objects.

The Java programming language has been used to create a subsystem. The object is detected based on its color. To determine the color of the object, the HSV color model (Hue, Saturation, Value) has been used. Thus, by setting the boundaries of each parameter of the given color model, the sought object is detected.

Two cameras (Fig. 5) are used to record the trajectory of an object, which allows determining the trajectory in three-dimensional space. The trajectory is a collection of points (x, y, z), where a point of time is tied to each point. These trajectories are accumulated on the local computer center (in the experiment it has been implemented on Raspberry Pi 3) and sent to the server at the frequency of 1 time per minute. Each device that monitors a biological object has its own unique identifier and binding to a specific region.

Fig. 5.
figure 5

Data collection schemes

Full size image

The module for storing and analyzing trajectories of the objects has been implemented on the server. This module contains a database that stores a set of motion trajectories (Table 2).

Table 2. The format of the activity data stored in the database on the server
Full size table

In addition to the database, the module implements the functions of receiving and analyzing the obtained trajectories. It is possible to view the trajectory of movements from each device for a set period. The software of the module for detecting and recording the trajectory of a biological object developed during the experiment has been uploaded on GitLab [22].

6 Conclusion

The results presented in this article are the first stage of the project ‘IoT Based Earthquake Prediction Technology.’ A similar approach to data collection can be applied to other types of animals (like cats, horses, or cows). After the data collection and analysis system is done, another important task needs to be solved - replicating the system so that as many objects as possible can participate in the monitoring. Mass data collection is a key condition for the success of the project, according to the authors. To involve pet owners in the project, it is planned to implement a subsystem that will allow the owner to analyze the animal’s behavior online, identify animal health problems promptly, and monitor the achievement of the required level of activity.

It should be noted that some of the proposed approaches, for example, recognition of the type of activity in dogs, can give inaccurate results in connection with the diversity of animal breeds and their essential differences. Perhaps, in this case, it will not be possible to exclude the activity caused by artificial factors from the data set. The calculation of the level of motor activity, in this case, will be performed on all the collected data.

One of the additional ways to use the developed system is to study the animals themselves. It can be assumed that the data collected allows us to study new aspects of animal physiology; for example, the influence of climatic conditions, magnetic storms, seasonal factors, noise, etc. on their behavior.