1 Introduction

Recently, the various technologically advanced ways of fire detection have been introduced, as most of the time, the fire breakout has been found to cause severe economic damage as well as loss of lives. Fire is easily spreadable to its surrounding carrying strong heat and stopping the people from escape. When the fire breakout spreads to the surrounding it can become more destructive resulting in higher casualties and death. Fire accounts for huge economic losses and death. However, early fire detection is more effective to astound the economic loss and to save human lives. The fire detection at the very initial stage is necessary to escape the fireplace and to drench the fire source properly. The most suitable way to detect fire at an early stage and to avoid losses is the fire alarm system installation. The fire alarm system is equipped with various interconnected devices which works together to detect fire and give fire alerts to the surrounding people via connected video and audio appliances regarding fire breakout and emergency exit. The various detection devices such as heat, smoke and gas detectors initially detect their respective happenings followed by auto-alarm activation. The bells, mountable sounders and horns are fitted in the alarm system to give a proper alert.

Many researchers carried out work on early fire detection using WSN (wireless sensor network) [2, 21, 28, 30] and video-based fire detection [3, 5, 11, 19]. Most of the fire alarm systems use WSN (wireless sensor network) technology. Presently, sensor-based fire detector typically based on the wireless sensors such as heat detectors, smoke detectors, gas detectors and visual sensors to detect the surrounding environment. Each detector is designed to have some limits and if the detected sensed cross that limits then it makes an alert and sends this information to the cloud. These methods present high accuracy but also have some limitation which allows the researchers to come up with a more effective method. WSN based strategies require an extensive amount of smoke, heat or gas for detection and it is only conceivable if the flame is high and heat actually reaches to the sensor, but fire ought to be detected at very early stage.

Similarly, there are many methods exists that use images data for fire or smoke detection. These images datasets are taken from cameras installed for surveillance of the environment. But these methods also have some drawbacks. For example, in images some object which looks like fire and smoke i.e. bright light, smog, or fog. It is very difficult for these methods to classify these types of objects as no fire and it generates a false alarm. The methods that use a convolutional neural network for detection process takes time to compute the feature maps in the whole image and it is impractical for early fire detection, therefore, this problem comes with a demand of algorithm with high accuracy and with potential to minimize false alarms.

Fire carries various features such as light, heat, smoke and many others which are detectable by different sensors. These days a large amount of fire data exist on the cloud in the form of smoke and heat sensors data as well as fire images data (smoke images, fire images). We can use these datasets in fire detection. This is the era of artificial intelligence and by using machine learning methods we can predict the fire as well classify it. To provide an efficient solution for the above-mentioned problem, we proposed a strategy by using three machine learning methods. Our first proposed model is a hybrid neural network model which is made up of Adaboost and many MLP neural networks. The main aim of using hybrid Adaboost-MLP model is to predict fire proficiently. This model utilized distinctive sensors information like smoke, heat, and gas for training. We used four different training algorithm to train our Adaboost-MLP model. These training algorithms are (i)gradient descent with adaptive learning and backpropagation (ii)gradient descent with momentum and adaptive learning rate backpropagation (iii)conjugate gradient descent with backpropagation and fletcher reeves (iv) broaden Fletcher Goldfarb Shannon. The primary idea of using these training algorithms was to evaluate their accuracy against various learning algorithms with different hyperparameters. The second model is Adaboost-LBP model which is used for the assistance of our third model that is a convolutional neural network. Adaboost-LBP model for generating the ROIs where emergencies exist and then these ROIs are checked by our CNN model. The key contribution of this work is manifolded as following.

  • Fire prevention with high efficiency using machine learning algorithms.

  • In this work, we proposed an Adaboost with MLP algorithm to make forecasting of fire using temperature and gas in the environment.

  • We compare our final predicted results against different learning algorithms.

  • We used Adaboost-LBP model as the assistance of the CNN model. This model generates ROIs in the image where emergencies present.

  • We used a convolutional neural network for the classification of fire in the images.

The rest of the paper is organized as follows. The related work is presented in Section 2. Section 3 discusses the proposed work and the detailed design of the fire prevention system. Experimental details are discussed in Section 4. The section 5 describes the final results of our proposed model. Finally, the article is concluded in Section 6.

2 Related work

Research on fire detection based on wireless sensor network and videos are growing exponentially. Plenty of work is already done on this problem. Fire sensors have been widely recognized and used for early fire detection recently [2, 31, 34, 36]. Bhattacharjee S and co-workers [2] presented work for fire prevention in the mines. They deployed the gas sensor and devised a WSN. In the work proposed in [36], they used temperature sensors to evaluate the fire intensity in the forest. For their network node, they utilize CC2430 chip in hardware design. In the same way, Yu L et al. [34] also schemed a structure for fire detection in the forest. In their work, they utilized clustering schemes and communication protocols in the model for forest fire detection. They accomplished the stimulation for endorsement and assessment of their work. Similarly, W. Tan et al. [31] reported the forest fire detection. Multi-sensor and remote IP cameras were used to evade false cautions. Their framework additionally associated with the web by means of gateways for transferring the information to the cloud. The work based on ZigBee remote sensor network was also presented in China for forest fire detection [36]. The visual sensor-based or video-based fire detection also has rich research. A work for effective detection proposed in [26] which claims the fire detection using visual sensors. They generated the lookup table to investigate the fire flames and they are confirming these regions using the temporal variations. This work is totally heuristic features based and decreasing the probability of getting the same results by changing the inputs. Thinking about the heuristic highlights of [26], the authors in [6] introduced a rule-based fire detection technique utilizing the dynamic investigation of fire alongside RGB/HSI color space. Their technique examined the development of pixels with the cluttered properties of fire for detection. Though, it neglects to distinguish moving sites and fire as it depends on the frame-to-frame distinction. In [12], the researcher presented a fire detection technique based on correlation of a normal picture with its color’s information for tunnels. The technique is appropriate just for static fire circumstances because of its utilization of numerous informal parameters. Another work proposed by Liu et al. [20] studied three models, which contains spatial, spectral and temporal. This work is for detecting the fire regions in the image But, their work is an assumption acknowledging capricious patterns of fire in the image. The work proposed in [37] is based on a color-based model that is YCbCr and conceived a new algorithm which rules-based algorithm. Their algorithm worked for the splitting the chrominance and luminance components, which force them to the classification of flame based on rule-based pixel.

From these study we came to know that current fire caution sensors (i.e. infrared, particle, and optical sensors) require nearness to the heat, fire, radiation or smoke for activation, henceforth such sensors are not thought to be a great contestant for critical situations. These strategies have great precision, but such techniques experience a few problems, for example, the unpredictability of the scenes under inspection because of people and items looks like the fire in surrounding, the abnormality of lighting (night, day, fake, shadows, light reflections, and glinting), and the low nature of the caught pictures, their lower differentiation, and lower signals transmission. Considering the previously mentioned fire identification strategies, it can be scrutinized that some of them are excessively immature; their execution time is quick, but it cannot facilitate the early fire detection because such techniques can work only when the fire happens. On the other hand, a few techniques have accomplished great fire location exactness’s, yet their execution time is too high, consequently, they can’t be connected in genuine situations particularly in basic territories where a minor deferral can prompt an enormous debacle. Subsequently, for more precise and early fire detection, we require a vigorous component that can foresee fire amid utilizing conditions changing.

3 Proposed work

Fires are one of the manmade disasters threatened human life and economic damages all over the world. To avoid this categories of fire disasters and its damage, some point-type thermal, smoke detectors, heat detectors, and many vision-based fire detection methods are widely used. But such detectors need to be in proximity of the fire and easy to fail or damaged in a bad environment. To overcome these problems, we proposed a machine learning based fire prevention method. Figure 1 is an overview of our proposed work. In our proposed model, we have two layers, the first one is the data assortment layer and the second layer is the processing unit. In the data collection layer, we are gathering the fire data into a local server. The data is of two types i.e. fire sensors data and the data in the form of images and videos. In the data processing layer, we used three machine learning methods that are Adaboost-MLP model with four different learning algorithms, Adaboost-LBP model, and deep Convolutional Neural Network. Adaboost-MLP model is used to predict the fire environment. For this model, we used sensors data for the purpose of training and testing. If the prediction is fire, then remaining both models are triggered to confirm the fire. We used Adaboost-LBP model for the assistance of our CNN model in the detection process. The purpose of using Adaboost-LBP model is to generate ROIs in the image to make the detection process faster. To train the Adaboost-LBP model we used datasets available on the cloud as well as the data from social media. CNN model is trained by.

Fig. 1
figure 1

Overview of Proposed Scheme

Full size image

ROIs data that is generated by Adaboost-LBP model as well as video fire datasets already exist on the cloud. The detail description of each model is described following subparts.

3.1 Adaboost-MLP model

AdaBoost is the abbreviation of Adaptive Boosting and it is a strong machine learning algorithm. It is designed by a Scientist named as Freund [9]. Commonly, it is used for the boosting of many weak neural networks. It combines the output of weak learners into a weighted sum (provided by Adaboost), and then it classified as the final output. It was devised for classification of features and regression, but after its high efficiency regarding classification, it has been used in many image processing applications [10]. The AdaBoost algorithm is much sensitive for noisy data and outliers.

MLP is abbreviated as multilayer perceptron, basically, it is very simple and feed-forward artificial neural network [27]. It has great efficiency for nonlinear matching and has high generalization abilities, because of that, it has been used in many signal processing works. A simple MLP model can be of three nodes, i.e. one input, one output, and one hidden node. Except for the input layer, every neuron (each node is a neuron) uses nonlinear activation function. Usually, for training, the backpropagation algorithm is used. Our proposed Adaboost MLP model is shown in Fig. 2 which contain these following processes.

  • First, we collect data and pre-processed it to make it noise free.

  • There is an Adaboost algorithm to make better predictions with multiple MLP neural networks.

  • Many MLP neural networks with four different learning algorithms (as weak predictors) for multi predictions. These four different learning algorithms are gradient descent with adaptive learning and backpropagation, gradient descent with momentum and adaptive learning backpropagation, conjugate gradient descent with backpropagation and fletcher reeves and Broyden Fletcher Goldfarb Shannon.

  • Final all the forecasted results from MLP neural networks are summarized using weight provided by AdaBoost.

Fig. 2
figure 2

Overview Hybrid Adaboost MLP model

Full size image

3.2 Adaboost-LBP model

LBP abbreviated as local binary patterns is a visual type descriptor used for classification in computer vision. Firstly, it was devised in 1994 [24]. Afterward, it has been found as a very influential feature for text data classification [25]. A work which is the comparison of different enhancements in the original form of LBP was proposed in 2015 [29]. The procedure of LBP feature vector creation is as follows. (i)First, the examined window is divided into cells e.g. 16 × 16. (ii) From either side, every pixel in the cell compares with its neighbor’s. (iii) The pixel values set as 0 if the focal pixel value is greater than its neighbor value otherwise 1. This generates an 8-digit binary number (convertible to decimal for future usability). (iv) Throughout the cell, calculate the histogram for the frequency of every member number on its generation. (Likewise, most of the combination of pixels are minimum, and maximum than of the center). The histogram can be visualized as a vector combined with features of 256-dimensional. (v)The histogram can be normalized for the verity of use cases. Normalization of the histogram for every cell result in the form of feature vector for the whole frame.

After getting the alert from Adaboost MLP model our fire detection module is triggered. The videos and the pictures from the surveillance cameras are classified by this module. We used the CNN model to detect emergencies from the video frames. But, the speed of CNN is enormously very slow to calculate the entire frame. As, the fire needs very quick detection, we used the Adaboost LBP network to calculate emergencies in the pictures. This model generates ROIs in rectangular form.

As the natural phenomenon of fire and smoke, the particles are continuously moving in the various direction and changing its position, due to this variation, ROI must be updated in a timely manner. Algorithm 1 explain each point in detail of ROI formation. The object visibility resemblance that of smoke or flame can be identified through the input received. This information was gathered from the Adaboost-LBP model which contains: the object weight Wi, Object height Hi, object central coordinates Ci(x,y), the four coordinates (RC1(x, y), RC2(x, y), RC3(x, y), RC4(x, y)) are the rectangle with each coordinate described by ROI.

figure e

3.3 Convolutional neural networks

Convolutional Neural Networks [17] is a biologically motivated deep neural network. Deep CNNs have very efficient and powerful applications in the field of computer vision like renewal of images, classification [1, 4, 14, 33], localization, image segmentation [18, 22] and object detection [15, 38]. The fundamental concept behind the CNN is dividing the problem into different subparts, repeatedly, until unless the final solution. The complex method of pre-processing (which is very common in other machine learning algorithms) can be avoided by training the model from raw pixel value to classifier. A simple CNN model is multi-layer feedforward network consist repeated layers of convolutions and subsampling. The final layers which are for high reasoning classification of the CNNs. The detail description of each layer is as follow.

3.3.1 Convolution layers

Convolutional layers apply a convolution process on the image (input) and the results are passed to the next layer. Every single point in convolutional layer is a receptive field which is set of units of previous layers. Basic visual features in local receptive field like corners, endpoints and oriented edges can be derived by neurons. This layer has various feature maps that have ability to pull of multiple features. In each feature map, all the units have the same weights and bias. Because of that reason the detected features are same for all possible locations of input. Generally, the form of a convolution layer is given by the expression [32].

$$ {X}_j^l=f\left[{\sum}_{i\varepsilon {M}_j}{x}_i^{l-1}\ast {k}_{i j}^l+{b}_i^l\right] $$
(1)

where \( {X}_j^l\ \mathrm{i} \)s the output as jth feature map of a particular convolutional layer, Mjis a collection of input maps, k is the kernel size that determines the how much our model convol the picture and b is a bias.

3.3.2 Subsampling layers

The subsampling or pooling layer execute a subsampling and local averaging, for lowering the complexity resolution of the feature map. It also removes the sensitivity of the output. The form of a Subsampling layer is [32].

$$ {X}_j^l=f\left[{\beta}_{j^{down}}^l\left({x}_j^{l-1}\right)+{b}_i^l\right] $$
(2)

where (down) is a function called sub-sampling. Usually, sub-sampling (down) function provides n-by-n block in the input image that’s computes the final output and will give n-times smaller and normalized output. Where β is the multiplicative bias and b is the additive bias. Our proposed CNN model is as follows.

3.3.3 Proposed CNN model

The used CNN model had a similar design to AlexNet [16] but with little changes according to our problem. We changed output neurons to two. Our model has five convolutional layers along with five max-pooling layers and two fully connected layers having 4096 each. Furthermore, we change the order max-pooling and normalization layers which is with first and second convolutional layer and with fifth convolutional layers. We implemented the SoftMax classifier for high reasoning classification in final layers. Leaky Relu used as an activation unit which is based on following equation.

$$ f(x)=\Big\{{\displaystyle \begin{array}{cc}x& if\;x>0\\ {} ax& x\le 0\end{array}} $$
(3)

Our model framework is shown in Fig. 3. Our CNN model takes an input image and resized it 256 × 256 × 3. After that convolution operation applied on a first convolutional layer which applied a filter on the input image with 11 × 11 × 3 size 96 kernels with a stride of 4 pixels. Then, the output of that layer is passed through the pooling process where complexity and dimensionality reduced. After that, the second convolutional layer came in where 256 kernels of size 5 × 5 × 64 with a stride of 2 applied and output is passed to pooling layer. On 3rd convolutional layer, again convolutional operation applied having 384 kernels of size 3 × 3 × 256 but no pooling process. After that, rest of the two convolutional layers are used which applies a filter with kernel 384 and 256 respectively with a stride of 1. After 5th convolutional layer again pooling layer which uses 3 × 3 filters. Finally, two fully-connected layers came in for final classification having 4096 neurons each. Moreover, the output layer made of only 2 neurons which classify the final output into fire or no fire images.

Fig. 3
figure 3

Proposed CNN Model

Full size image

4 Experimentation

We implemented our models in python and the training and testing process of these models is made on a CPU i.e. Intel® Core™ i5-3570 CPU @ 3.40GHz 3.80GHz having 64-bit windows operating system, 64 GB RAM and GTX 970 graphics card. For the training and testing of Adaboost MLP model we used dataset taken from [13]. We used 700 different data samples to train and test the model. Training and testing process is illustrated in Fig. 4. Where we can see we divided data samples into two parts that is 1-600 data samples for training and 601-700 for testing purpose. To train different week learners that is MLP in our case we used four different learning algorithms. For them initial learning rate was 0.1 but after some it decreased to 0.01.

Fig. 4
figure 4

Training process of Adaboost MLP model

Full size image

The purpose of Adaboost LBP model is to detect the fire emergencies like smoke, fire in the image and to generate ROI in rectangular of that emergencies. And these ROIs are further test by our CNN model for better classification of fire or not fire. To train both of our model (Adaboost-LBP and CNN) we used different datasets taken from Foggia and Chino [7, 8] and other datasets [23]. To train and test Adaboost LBP model we used 35,845 images used which contains smoke and fire images. We divided these images set into three parts i.e. for 24,205 images for training, 4346 for validating the model and remaining 7303 for testing the model. We set 0.01 learning rate.

To train and test our CNN model we used Foggia video fire data set [8], Chino smoke dataset [7] and other datasets [23, 35]. We train our CNN model using 30% of data and remaining 70% data is used for validating and testing. Total 64,170 images are used in which 19,251 images are used for training 1543 images for validation and remaining 43,376 are used for testing. These images first passed through Adaboost LBP model to generate ROIs and these ROI are used for training and testing of that model. Training done in 80,000 iterations and it used 128 batch size. Initially the learning rate was 0.01 but we used step decay learning process so, every after 1000 iterations it decreases by a factor 0.5. Later, after 40,000 iteration our model learning rate is fixed that is 0.001. We also set momentum value of 0.9. After training we test our model on fire videos and smoke images. We calculated accuracy, precision and recall using following equations. Where TP and Tn are true positive and true negative respectively and Qp and Qn are positive and negative samples respectively.

$$ \mathrm{a}=\frac{T^p+{T}^n}{T^p+{T}^n+{F}^p+{F}^n} $$
(4)
$$ \mathrm{p}=\frac{T^p}{T^p+{F}^p} $$
(5)
$$ \mathrm{r}=\frac{T^p}{T^p+{F}^n} $$
(6)

5 Results and discussion

The main idea of our Adaboost MLP model is to forecast fire. As, we trained our model with sensor data. After training, to validate our model we made testing of our model using 100 temperature data samples. As, our model consist of many MLP models with four different learning algorithms and then using Adaboost algorithm we boosted their results. Figure 5 (a), (b), (c) and (d) is showing the predicted temperature results of four learning algorithms that is gradient decent with adaptive learning rate and back propagation, gradient decent and momentum with adaptive learning rate and back propagation, conjugate gradient decent and back propagation with fletcher reeves and broyden fletcher goldfarb and shanno. The results are showing MLP model after boosting have better prediction results.

Fig. 5
figure 5

Testing result of Adaboost MLP model against Temperature

Full size image

As, to train and test our both Adaboost-LBP and CNN model we used two different datasets. These datasets include fire and smoke images as well as no smoke and no fire images. During the training and testing of both of our model we calculate the training and testing accuracies. Figure 6 (a) and (b) is showing the combined training and testing accuracy of our models. We compute it against two different data sets. We also compared our model accuracies with CNN model that use whole picture for testing. The results show that accuracy of our models is better than traditional CNN model. Our model accuracy is almost 97.8%.

Fig. 6
figure 6

Training and Testing accuracy of combined Adaboost-LBP and CNN against two datasets

Full size image

We tested our Adaboost LBP and CNN model using a dataset of fire videos and images. It contains of 7303 images with different resolutions. We divided these datasets in two categories i.e. 5248 images containing fire and the remaining images are of without fire. These videos and images of fire contains emergency conditions with difference fire occurrences, as industrial fire, fire on buildings, cars, forest and many other places. The remaining images include the emergency circumstances fire that is not visible and the circumstances where the images with fire-like regions, such as fog objects, red and yellow objects, sunsets etc. The reason behind the selection of the datasets that we used for training and testing of our models includes huge number of fire and no fire videos and images. These images are quit challenging’s for the classification. All the color-based methods create more false alarm on these datasets because many objects are present in these images that looks like fire and smoke. Similarly, motion-based method also generates false alarm because in many of the images the mountains exists with smog, cloud and fog. Figure 8 shows some sample images of these datasets. For example, in image (a) there are clouds which look like smoke, in image (b) sharp light looks like fire, in (c) again white light looks like smoke and image (d) clearly it is seen clearly like smoke but none of them is fire not smoke. Since we trained our model using labled data in which we divided dataset into fire and no fire images to make it labled. First, we tested these images using our Adaboost LBP model. Our model detects the fire object and then RoI captured algorithm captured that specific region. The histograms of all feature vectors are illustrated in Fig. 7 (a) (b). We can see in the histograms of feature maps of two different images (fire image, no fire image) is different. As, testing dataset is quit challenging because existence of some objects that look like fire and smoke. Because of this reason our model creates RoIs of these objects also and cause false alarming. To reduce this false alarming, we tested these RoIs using our CNN model to make better classification.

Fig. 7
figure 7

Adaboost-LBP model Histogram of Feature vectors

Full size image
Fig. 8
figure 8

Sample images that contain objects which look like fire or smoke

Full size image
Fig. 9
figure 9

Performance evaluation of our CNN model Against ROIs generated by Adaboost-LBP model

Full size image
Fig. 10
figure 10

Fire and smoke detection Using CNN model in feature map and original image

Full size image
Fig. 11
figure 11

Precision, accuracy and recall of CNN model against smoke and fire datasets

Full size image

The accuracy of CNN model against these ROIs test is very high and efficient. Some tested images are illustrated in Fig. 9. Figure 10 is the illustration of fire classification in images and in terms of feature maps. As, CNN model mainly has two parts, feature selection part and then fully connected layers i.e. classification part. After all the convolution and pooling processes, we got the feature map as shown second image of Fig. 10. In the image we can see white dots are the features of fire and the darker parts are of the smoke. After getting the features maps, these features are classified in the second part of our CNN model where we use SoftMax classifier. In Fig. 10, 3rd and 4th image is the result of classification. In images we can see red and green areas where red area is the fire detected features and green area is the smoke features. We also evaluated our CNN model results based on some parameter that is precision, recall and accuracy using above eqs. 4,5 and 6. Graph in Fig. 11 is the illustration of precision, recall and accuracy. High results for these parameters show our model very efficient and accurate.

6 Conclusion

The fire and smoke need an urgent detection to avoid many damages in terms of human lives and economically. Existing methods like fire detection using WSN and visual base fire detection have some drawbacks. But this problem needs a method which can prevent that manmade disaster in very efficient manner, without any delay and less false alarm rate. In this context, we proposed a method based on powerful machine learning algorithms. We used Adaboost-MLP model for fire forecasting using sensors data. We also used Adaboost-LBP model along with convolutional neural network for fire detection using images which are taken from surveillance cameras. The purpose of Adaboost-LBP model is to find emergencies from the image and to generate ROI of that detected object. The false alarm rate of that model is high, then, to reduce false alarm we tested these ROIs in our CNN model. The final result of our proposed model is efficient as compared to traditional methods. The accuracy of our proposed model is more than 99%. It can be more accurate after more training.