1 Introduction

As on 24th April 2021 the 7 days average of Covid-19 new cases are 8,16,354 & 12,457 deaths [1]. The current Covid-19 pandemic situation has created an intimidating challenge to all countries. Highly populated country like India is struggling a lot to handle this pandemic situation. Person who is infected with the COVID-19 virus will be having respiratory problems when he is exposed, the person nearby can get infected as when a person who is infected talks with a nearby person the droplets from the mouth carrying the virus can exist in the air for about 8–15 min [2]. This will become even worst then an infected person talks loud, cough or sneeze, because more droplets can come out of mouth carrying the virus. It is important to restrict the spread of Covid-19 virus spread by maintaining the social distancing and wearing mask in the public places [3].

Government of India has enforced many limitations since March 2020 in order to prevent the spread of virus in public entertainment spots such as theaters, hotels, malls, marriage halls and in temples, such as only 50% occupancy is allowed in theaters, shopping malls and in recreation clubs, only 100 people are allowed in marriage functions and public transports can be operated with limited passengers. There are various systems available to monitor the temperature of a person without contact such as over head Infrared Temperature sensors, which has a microcontroller with IR temperature sensor, connected with a OLED display displaying the temperature or there are systems to monitor the temperature using thermal scanners which are not economical for implementing in public places. Implementing high cost cameras for monitoring the occupancy and for monitoring public wearing mask is also not an economical method, or it will involve some person to operate those equipments.

There are few other general problems in theater environment between the producers and theater owners on number of occupancy and in collections status if there is a system to monitor the number of occupancy and if it can be monitored from the remote places, these disputes can be fixed. Similar problems exist in other places like smart parking system through which we can monitor the number of parking slots free, passenger counting in public transport, also for automating the air conditioner for increasing the required BTU (British Thermal Units) in party hall based on the occupancy (Generally Air-conditioned capacity (Ton) is measured in BTU).

IoT is a system of objects, people or machines with some unique identifiers which are capable of sharing the data with the help of network facility. This helps to collect the data in a smarter and faster way without any human intervention [4]. Since there are nearly 32 billion devices are connected to internet in today's world, the life without internet seems unimaginable. IoT has already created an impact in various sectors including logistics, industries, agriculture and many more, the automation becomes an easy task without any investment of human resource [5].

The main objectives of this Invention are:

  1. 1.

    Developing a hardware system which incorporates a Raspberry Pi with two Passive Infrared Sensors to monitor the number of occupancy.

  2. 2.

    Raspberry Pi Camera is incorporated to monitor the public without mask.

  3. 3.

    Ultrasonic sensor is incorporated to start measuring the temperature when a person is in a particular distance to the set up.

  4. 4.

    Temperature Screening is done through MLX90614 Infrared temperature connected with the Raspberry pi.

  5. 5.

    To incorporate a fuzzy logic based on the previous results and provide access to enter in to the premises.

  6. 6.

    Upload the all these details in the cloud through Wifi module in Raspberry pi, so that authorities can monitor the status from a remote place through which occupancies can be monitored.

The rest of the paper is organized as Literature review in Sect. 2, proposed system in Sect. 3, results and discussion in Sect. 4 and conclusion in Sect. 5 which is followed by references.

2 Literature Review

A face mask detection technique is used to sort out whether the person is wearing the mask or not. This technique is very much similar to object detection from a scene. For this purpose, deep learning techniques are largely used [6, 7] in medical field. An outstanding works have been carried over in the field of object detection in the recent past using various deep learning techniques [8]. Furthermore, many smart cities across the world have numerous IoT sensors using which numerous data are collected which can be used to perform various tasks across the city. The rapid growth of the COVID-19 can be drastically reduced at public places in the smart city network by detecting the usage of facial mask by the people. In the recent past many systems have been introduced for reducing the spread of COVID-19.Primarily Blue Dot method [9] was used which predicted that this COVID-19 as pandemic and it would spread from Wuhan to nearby counties. Health Map facility [9] used AI and big data resources to identify that the people with initial symptoms of cough is affected by COVID-19. Garcia et al. [10] proposed a detailed research work on proper usage of facial mask to prevent the spread of COVID-19 through coughing droplets and sneezing.

Allam et al. [11] proposed a framework which is capable of performing the effective data sharing during the COVID-19 pandemic outbreak. Here the data are collected from various sources like sensors and trackers placed at various parts of the cities. In order to facilitate the transport facility during the COVID-19, Gupta et al. [12] proposed a model which is capable of enforcing the social distancing during transportation by means of sensors which are deployed at various places in smart cities to monitor the movement of the objects and an effective data sharing platform. Won Sonn et al. [13] proposed a system which effectively reduces the spread of COVID-19 virus in South Korea by monitoring the COVID-19 patient movements using purchase history, cell phone usage and cell phone location of the affected patients.

Qin et al. [14] proposed a framework for detecting the people with the mask by using four stages of operations namely image pre-processing, cropping of face region, performing super-resolution operation and mask predicting process. Ranjan et al. [15] proposed anprecise face mask detection and verification mechanism where deep learning and CNN techniques are used for verification. Zou et al. [16] proposed a face mask detection system were they use using a improved convolutional neural network for detecting the face and YCrCb elliptic skin colour model for detecting the face mask the experimental result shows that the proposed system detects the face mask with more accuracy. Abbasi et al. [17] presented a new set of data and proposed two methods for in the first model object detection method is incorporated for classifying the mask wearing and the other model is YOLO face detector spot face for classifying the masked and unmasked with CNN classifier, with this proposed method they achieve an accuracy of 99.5% with new dataset. Ayyappa et al. [18] presented a MobileNetV2 architecture is utilized using python programming. Here the data sets were trained using Keras and Tensorflow, then the trained model is applied in webcam based live video stream and for available images. Both these situations the accuracy of the proposed system is substantially very high. Baluprithviraj et al. [19] designed a Raspberry pi with a pi camera to detect whether the visitor is wearing a mask or not. If the visitor is wearing a mask the door automatically opens. This system is incorporated with a mobile app which identifies the person who enters without a face mask.

Singh et al. [20] compared two different state of the art algorithms for detecting the face mask which are R-CNN and YOLOv3. Both these algorithms classifies the person wearing mask and without mask and draw a bounding box which in red or green colour representing with mask and without mask. The author also compared both the algorithms performance based on accuracy, time, and precision rate, experimental results shows that the YOLOv3 is best suited for real time application. Costanzo and Flores [21] proposed a non contact temperature monitoring system were the capacitive humidity sensor is integrated which an able to detect the temperature of an object accurately and fastly. Experimental results shows that the proposed system is best suited for covid19 fever with high accuracy.

One of the most vital symptoms of COVID-19 [22] is found to be fever. The measurement of fever in the people has become a challenging task due to the transmissible effect of the disease. The challenging task is to find the body temperature of the people in a quick manner without any contact with the people with the symptoms. With advancement in technology, the wearable sensors [23, 24] are introduced to monitor the human body temperature of the patients. Thermal imaging cameras are introduced in to the market which is used to detect the temperature of the humans using thermography technique [25]. Kaikai Zheng et al. [26] developed a system which detects the facial temperature of the humans with the presence and absence of facial cosmetics using infrared cameras. Human body temperature is a virtuous indicator of human health. Even a small change in temperature in few degrees can be used as an important symptom in identifying the illness among the people [27]. A temperature screening system [28] was invented which is capable of identifying temperature of multiple humans at a time with the help of thermal imaging camera and facial recognition technique. A RFID chip less tag tattoo was designed by Silveira et al. [29], which is capable of continuously monitoring the human temperature between 36 and 40 ℃. A device for monitoring the human temperature using thermal sensor camera was developed by Dongxu He et al. [30] which is capable of measuring the human temperature accurately when it is maintained in the environment whose temperature should be between 45 and 80 ℃. Priyamvadaa [31] developed a temperature monitoring scheme which monitors the human temperature and transfers them to the health officials effectively with high degree of accuracy and reliability during COVID-19 pandemic outbreak using Message Queuing Telemetry Transport [MQTT] protocol.

Bidirectional visitor counter Malik et al. [32] is also an important task in this proposed system, Mohana Prasad et al. [33] proposed an Arduino based bidirectional visitor counter for industrial application, here they have used IR sensor for detecting the visitor activity, but the range of the IR sensor will be very low. Crisnapati et al. [34] implemented the same in Node MCU development board for uploading the visitor details in to the cloud for IoT applications. Therib et al. [35] also proposed uploading the visitor count details in the cloud but here they have used two NodeMCU one for detecting the visitor entering into the premises and one for detecting the visitor exiting the premises which is detected using Ultrasonic sensor, both the details will be uploaded to the thinkspeak cloud and can be monitored through the web application [36].

General observations from the existing works are very less work was done in the proposed area of screening the visitors during the pandemic situations [37]. Complex algorithms were used for detecting the face mask and which consumes more power when compared with the portable OpenCV with MobileNetV2.Usage of Infrared sensor and Ultrasonic sensor for counting the number of visitors can be less accurate, in the proposed system we implement ultrasonic sensor to detect the person and Passive Infrared sensor for counting the number of visitors which will be more accurate than the existing systems [38]. By using Arduino or NodeMCU the performance of the system will be degraded because of the execution time. Usage of Raspberry pi or NVIDIA Jetson Nano can improve the performance of the system [39].

In our proposed system we use OpenCV and MobileNetV2 based face mask detection, MLX90614 for monitoring the temperature of the visitor, and finally to monitor the premises occupancy we use a Passive Infrared Sensor to count the occupancy. The outputs of all the three parameters are given to the fuzzy control system a fuzzy based decision system decide the visitors to permit or to forbid then to entering the premises during the pandemic situation [40].

3 Proposed System

To monitor the entertainment and other recreation spots occupancy count and to screen the public with high temperature and without mask, an IoT based system is developed and those parameters were uploaded to the cloud so that authorities can monitor those details from a remote place. The hardware module consists of Raspberry Pi 3B+, PIR Sensor, Ultrasonic Sensor, Pi Camera, MLX90614 temperature sensor, Resistive HDMI Display and motors for opening and closing the flap gate (Fig. 1).

Fig. 1
figure 1

Seven days average of Covid-19 cases. Sources: Wikipedia

Full size image

In this proposed system monitors three different parameters the first important parameters is the number of occupancies inside the premises for which we use a pair of Passive Infrared Sensor (HC-SR501), the second parameter is checking for wearing of mask for which we incorporate a PI camera and the third parameter is temperature screening for which we use a long distance temperature sensor MLX90614, the block diagram of the propose system is shown in Fig. 2.

Fig. 2
figure 2

Block diagram of the proposed system

Full size image

3.1 Occupancy Monitoring

We perform a bidirectional visitor monitoring system using two Passive Infrared Sensor (HC-SR501) PIR 1& PIR 2 connected to the Raspberry pi processor one for counting the visitor entry and one to count the visitor exit and the processor will consolidate it and finds the number of occupancy inside the premises. Each PIR sensor will be consisting of three terminals which are VCC & GND which the power source and ground, and OUT through which the detected signal will be transmitted to the Raspberry pi. When PIR 1 senses five visitor movements the count will be incremented by 5. If PIR 2 senses two visitor movements now the count will be decremented by 2, now the occupancy inside the premises is calculated as 3.

figure d

3.2 Temperature Scanning

Body temperature is considered as an important aspect and symptom of COVID 19. In our proposed system the temperature is also considered for permitting the visitor in to the premises, to perform this task we incorporate a MLX90614 infrared long distance temperature sensor. This sensor is capable of measuring the object temperature ranging from 69 to 382 ℃. This sensor works on the principle of Stefan-Boltzmann Law, i.e. every living and non living object produces IR energy, which will be directly proportional to the temperature of the person or an object. So this sensor determines the temperature based on the amount of IR energy emitted from the human body. This module is factory calibrated and can be used in the microcontroller unit like a plug and play device. This module consist of two unit one is sensing unit and the other is the processing unit the sending unit takes care of sensing the temperature and processing unit converts the sensed values to its digital form and it will be taken to the processor through Inter Integrated circuit (I2C) Protocol.

3.3 Face Mask Detection

For facemask detection operation we divide the entire module into two phases, which are training phase and testing phase. Initially the datasets are loaded into the system and the system is trained for detecting the face mask with tensor flow. To make the system robust training is done with different variants of face images with and without mask this process can avoid the false positive ration and makes the system reliable. For this task system is provided with 1500 image data of which 800 are with mask and 700 are without mask image data (Fig. 3).

Fig. 3
figure 3

Sample dataset with and without mask

Full size image

Once the images are trained the phase two which is the testing phase or deployment phase starts. Now the Pi camera initiates and starts streaming the video to the system. We use OpenCV deep learning system for detecting the faces in the stream of video. If the face is identified the next step is to identify the Region of Interest, which is done using OpenCV and NymPy Slicing through which the nose and eyes are identified using facial landmarks. Region of Interest (RoI) for detecting the facemask, this ROI is taken as an input to the classifier, for which we use a MobileNetV2 Classifier for performing this task. MobileNetV2 is best suited for mobile devices and it is a subclass of Convolutional Neural Networks, the architecture of the classifier is shown in Fig. 4. MobileNetV2 classifier has preliminary convolutional layer with 32 layers of filter which is followed by 19 layers of residual bottle neck layer.

Fig. 4
figure 4

MobileNetV2 architecture

Full size image

Here we got two blocks which are (S1) Stride of 1 and (S2) Stride of 2, were S1 act as residual block and S2 act as downsizing each layer consist of 3 layers. The first layer is a convolutional layer in 1 × 1 with ReLU6, the second layer is a 3 × 3 depth wise convolutional layer with ReLU6 and the last layer is a linear convolutional layer (Fig. 5).

Fig. 5
figure 5

Overall flow diagram of the proposed system

Full size image

3.4 Working Steps

Step 1 Raspberry Pi is initiated and it starts receiving inputs from all the sensors.

Step 2 Ultrasonic sensor (HC-SR04) detects visitors trying to enter the premises by measuring the distance between the setup and the person.

Step 3 If the system dint detect any visitor, step 2 will be performed iteratively until it detects a visitor.

Step 4 if the HC-SR04 detects a visitor, it starts counting the visitor and initiates the other sensors.

Step 5 When the visitor is detected and added in the count, MLX90614 temperature sensor monitors the visitor body temperature.

Step 6 Then by using machine learning and a raspberry pi camera the system check whether the visitor has worn a mask.

Step 7 Results of step 4, step 5 and step 6 will be provided as a input to the fuzzy logic system to take the decision to open the flap gate or not.

Step 8 If Count Reached the maximum permissible occupancy limit, Decision will be flap gate closed.

Step 9 If the body temperature of the of the visitor is higher than the permissible threshold, Decision will be Flap Gate Closed.

Step 10 If the visitor is not wearing mask, warning will be given and iterates step 6 for 5 times in every 1 s, Decision will be Flap gate Closed until the system identifies that the visitor is wearing a mask.

The fuzzy rule is designed in such a way that Occupancy limit and the body temperature of the visitor are given a highest priority. If this condition is not satisfied the overall decision will be flap gate closed. Mask is given the next highest priority but if the system identifies the visitor without mask or not wearing the mask properly, it warns the visitor to wear a mask if he wears a mask in the next attempt the decision will be flap gate open (Fig. 6).

Fig. 6
figure 6

Fuzzy presumption scheme

Full size image

3.5 Mathematical Modelling

The face mask classification result, temperature input from the sensor and the occupancy details from the passive infrared sensor is given as an input to the fuzzy logic control (Fig. 7). So we have three inputs and 9 different logics and it takes decisions based on the hidden layers (Table 1).

Fig. 7
figure 7

Membership function for mask screening

Full size image
Table 1 The fuzzy rule table
Full size table

3.5.1 Membership Function for Mask Screening

If the accuracy of face mask detection is greater than 95% the decision on face mask detection will go to its next phase of whether the visitor is wearing the mask (Fig. 8). A warning will be provided to wear the mask or to wear the mask properly. Until then the Flap gate will be kept closed.

$$ Mask\;Accuracy x = \left\{ {\begin{array}{*{20}c} {1,} & {x \ge 95\% , ROI\; Green \, \left( {Wearing\;Mask} \right)} \\ {0,} & {x < 95\% , ROI\;{{Re}}d \, \left( {Not \;Wearing\;Mask} \right)} \\ \end{array} } \right.. $$
(1)
Fig. 8
figure 8

Membership function for temperature screening

Full size image

3.5.2 Membership Function for Temperature Screening

For a normal human if the body temperature is above 37 ℃ it is considered as fever, in this pandemic situation raise in temperature is also considered as one of the symptoms of COVID-19, so for a visitor with more than 37 ℃, flap gate will not open (Fig. 9).

$$ Body \, Temperature \left( {\text{y}} \right) = \left\{ {\begin{array}{*{20}l} {1, \, {\text{y}} < 37\,^\circ {\text{C}}, \left( {Permited} \right)} & {} \\ {0, \, {\text{y}} > 38\,^\circ {\text{C}}, \left( {Not \; Permited} \right)} & {} \\ \end{array} } \right. $$
(2)
Fig. 9
figure 9

Membership function for occupancy restriction

Full size image

3.5.3 Membership Function for Occupancy Restriction

The occupancy limit is set to 150 in our case, if the PIR state is high, counter is set to 1 and for each high state of the PIR the counter Count is incremented by 1. If the count reaches 150 flap gate will not open further as the maximum count is set as 150.

$$ Occupancy \, count \left( z \right) = f\left( x \right) = \left\{ {\begin{array}{*{20}l} {1, \, z \le 150 \left( {Permited} \right)} & {} \\ {0, \, z > 150 \left( {\left( {Not\; Permited} \right)} \right)} & {} \\ \end{array} } \right.. $$
(3)

The Fuzzification output has the crisp value to the particular fuzzy function which is computed through the membership function and the feature rank values. The ranked membership function is computed in Eq. (4).

$$ \delta_{R} \left( A \right) = \delta \left( A \right) \times \mu \left( A \right) $$
(4)

where \(\delta_{R} \left( A \right)\) is the membership value for the entire fuzzy set A, \(\delta \left( A \right)\) is the specific value to A and \(\mu \left( A \right)\) is the rank of the occupancy restriction. The normalization value for the membership values for every fuzzy set is used to generate the value within 0.0–1.0. The normalized value for the fuzzy set A is demonstrated as \(\delta_{N} \left( A \right)\) and it is computed in Eq. (5).

$$ \delta_{N} \left( A \right) = \frac{{\delta_{R} \left( A \right)}}{{\mathop {\max }\limits_{\forall \omega \in A} \delta_{R} \left( \omega \right)}} $$
(5)

where \(\delta_{R} \left( \omega \right)\) is the ranked membership value for the subset \(\omega\) of the whole fuzzy set A. The normalization output for the fuzzy rule has the linguistic terms for producing the output through the fuzzy rules. The min–max interference rule is demonstrated in Eq. (6).

$$ \delta_{Y} = \max \left[ {\min (\delta_{{X_{11} }} , \delta_{{X_{12} }} ), \ldots , \min \left( {\delta_{{X_{N1} }} , \delta_{{X_{N2} }} } \right)} \right]. $$
(6)

Normally, the interference output within the fuzzy set has the crisp values which are normally needed for implementing the real-time applications. Moreover, the fuzzy rules has the output values that must be completed the Defuzzification process; it is the mapping within the non-fuzzy values. The Defuzzification involves the membership functions of mask screening, temperature screening and the occupancy restriction as the implementation of Covid 19 screening process.

4 Results and Discussion

The proposed work is implemented using a Python programming in Raspberry Pi Processor environment. MLX90614 temperature sensor used for measuring the visitor body temperature, similarly Passive Infrared Sensor (PIR) is used to count the number of visitors entering into the premises. We use a Pi Camera to detect whether the patient is wearing a face mask Fig. 10 shows the circuit diagram of the proposed system.

Fig. 10
figure 10

Circuit diagram of proposed system

Full size image

Figures 11 and 12 shows the steps involved in detecting the visitor facemask.

Fig. 11
figure 11

Face mask detection (no mask)

Full size image
Fig. 12
figure 12

Face mask detection (wearing mask)

Full size image

Step 1 The video is streamed through the web cam.

Step 2 Using OpenCV Detect the face which is the ROI from the streamed video.

Step 3 Now the ROI is applied to the pre trained network MobileNetV2.

Step 4 Now the MobileNetV2 network gives the prediction with labelled webcam image.

Step 5 If the mask is detected the ROI will be shown in Green colour Box, If mask not detected the ROI will be shown in Red colour box with Label and percentage of accuracy.

The comparison of the proposed algorithm for face mask detection is compared with the other state of the art algorithms the results shows all the algorithms were perform well for face mask detection, of that the YoloV4 and the proposed MobileNetV2 performs well for raspberry pi based face mask detection application.

The output of the temperature sensor scanning the ambient temperature and visitor body temperature running on the raspberry pi window is shown Figs. 13 ,14.

Fig. 13
figure 13

Performance comparison of proposed algorithm with other state of the art algorithms

Full size image
Fig. 14
figure 14

Body temperature scanning

Full size image

Figure 15 shows the google spreadsheet details of Temperature, occupancy and mask screening details are uploaded in the cloud, for which the following steps were performed.

Fig. 15
figure 15

Logging parameters to google spreadsheet

Full size image

Steps to upload data in google spreadsheet

Step 1: Create a new google spreadsheet and rename the sheet, and create a blank sheet

Step 2: For fetching the exact spreadsheet, we need a parameters of the created spreadsheet such as spreadsheet ID

Step 3: Next step is to configure the google sheet API, through google spreadsheet developer’s account, here enable the google sheet API

Step 4: Download the client configuration, install the google client library using the following code

Step 5: Copy python code for log the data in the google spreadsheet

The data uploaded in the google spreadsheet is shown in Figs. 15,  16 shows the thinkspeak visualization of the uploaded temperature details of the visitors in all the spots where the system is installed, so that the authorities can monitor those data using the URL. Figure 17 shows the occupancy details in a local LCD display attached to the processor, at the same time all these details will be displayed in the monitor kept at the place where the system is installed.

Fig. 16
figure 16

Visitor temperature uploaded in thinkspeak cloud

Full size image
Fig. 17
figure 17

Visitor count in LCD display

Full size image

5 Conclusion

In this work we propose an IoT Based Smart Assist System to Monitor Entertainment Spots Occupancy and COVID 19 Screening during the Pandemic. This system was developed to combat the COVID 19 spreading in public places like theatres, marriage halls, cafeterias and other similar entertainment and recreation spots. The proposed system is divided in to three separate modules, were the first module is restricting the number of occupancy using Passive Infrared sensor, the second module is to screen the visitors with high temperature using a MLX90614 infrared long distance temperature sensor, and the third module is to check whether the visitor is wearing a mask for which we use a pi camera connected which transmit the live video to raspberry pi and using MobileNetV2 classifier the system identifies whether the visitor is wearing a mask. All these outputs were given to the Fuzzy logic controller as an input; the FL controller decides to open or to close the flap gate. All the data is uploaded to the cloud and which can be viewed and monitored through the google spreadsheet. Thus the proposed system is very economical such that it can be installed in public places to reduce the COVID 19 public spreading.