Keywords

1 Introduction

Parking space management is one of the biggest problems faced in urban areas in both developed as well as developing countries. The number of cars running on the road is increasing with the car demand day by day, but the availability of parking areas does not meet the requirements. Along with this, the improper management of available parking spaces makes this problem worse.

There are many existing solutions to deal with this problem, such as barriers and ground sensors placed on every parking space, but they are expensive to use as they include installation cost, maintenance cost, and hardware cost and mostly not feasible in every scenario. Our approach is advantageous over the existing solution, which can monitor several parking spaces simultaneously reducing the cost [1].

The video to monitor parking areas is not new; see for the instance [2]. Many techniques proposed are not generalized for multiple types of parking lots. However, authors are proposing solutions that can be used in multiple scenarios like open parking space, off-street parking spaces, etc. Our solution not only converses the problem of parking but aims to solve almost every problem likely to occur in parking areas.

Our solution has three major parts in it, which builds up a complete system used in real time in parking spaces. Since authors are using live video streams, the capture can not only be used for parking space management but also the anomaly activities (like accidents, fire, burglary, etc.) which are captured in the camera. It will make the system splattered for parking occupancy detection, Parking space prediction, and anomaly action detection creating a fully fledged application that can be used in real-life scenarios.

1.1 Parking Occupancy Detection

In parking space detection, the use of CNN and LSTM models is producing quite accurate results even with the presence of disturbances created by the partial occlusions, obstacles, shadows, different light conditions, etc. trained and tested upon the CNRPark dataset [1, 2]. The CNN model will produce output free or occupied based on a trained dataset. The one with no car in the picture will be labelled as ‘free’ and the one with a car in a slot will be labelled as ‘occupied’. In the parking lot, each camera is assumed to be still in its position and hence the slots in the frames that are extracted will be fed to the model and the expected result will be attained.

1.2 Car Parking Availability Prediction Service Using LSTM

There are times when one can need information about the available parking space in a particular time period. It can be quite challenging to predict available parking space at a particular time before the head because parking space depends on many factors like place, time, weather, day, occasions, holidays, etc.

Since it is a time series-based problem, this can be solved using the Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) model. LSTM is a type of RNN which is capable of learning long-time-based data [3, 4].

Since the factors under consideration will be different for each parking area, there will be a need for a unique LSTM model to be trained for each parking space. The dataset to be used to train these models will be collected over a period of time for each parking space using our real-time parking space detection system. Once good time-series data is observed, it will be ready to be trained and used [15].

1.3 Additional Anomaly Detection Using Deep Learning

Surveillance cameras are available at almost all public places, for example, shopping complex parking, banks, streets, which can be used in both parking space management and public safety as well. Authors can use computer vision capabilities to take small steps towards managing all these anomalous activities which may occur in parking spaces. Anomalous events, for example, violence or accident or any theft in a parking area can be detected by developing algorithms using deep learning.

To tackle anomalous events like accidents and violence in parking areas, authors have developed a Multiple Instance Learning (MIL) solution by holding only weakly labelled training videos. MIL is a type of supervised learning where instead of using every labelled instance, the author feeds a set of labelled instances called bags to a learner. This will help a deep learning model to learn anomaly scores for video segments [4, 5].

2 Related Work

The existing work on smart parking systems consists of combinations of hardware and software collaborations. The solutions that have been proposed till now to be addressed as smart parking systems are mainly focused on three types: Internet of Things (IoT)-based, networking protocol-based and software-based. The IoT provides a wide range of solutions to parking-related problems [16]. It involves usage of different types of sensors as they are the most important component of IoT-based smart parking systems as they play a significant role in collecting data. Types of sensors used in these systems are mainly Camera, Ultrasonic, Infrared, Cellular sensors, Radar and many others [17]. The usage of sensors and other IoT technology is based on factors such as ease of installation, usage of sensors per slot, detection autonomy, etc. These factors play a role in sensor selection strategy. Sensors used have the ability to detect the occupancy status of the slot in which they are installed. Sometimes, usage of sensors results in a complex infrastructure that involves structured cabling, electrical connections, aerial hindrances, etc. This makes the system ‘not easy to maintain and increases maintenance cost’. Apart from above-listed sensors, there is RFID; it is an invasive sensor, as a card which is given to a user or installed on a vehicle, so whenever the user parks the car in a slot, it is determined if it is free or not. Use of radar sensors is another way to deal with the problem; these sensors use an algorithm to calculate the presence of any object in parking space and analyse how big it is and how it is positioned. This is then further analysed using software combinations.

Networking protocols are used along with sensors to pass information from sensors to smart parking systems so as to prevent the overhead of retrieving data from each sensor installed. Networking in smart parking has given better results, reliable throughput, less energy consumption and minimum delay. But still as the whole system relies on sensors and any fault in these will create a hindrance in the entire system. As per selected research papers, 40% of sensor-based solution did not specify usage of sensor networking in proposed solutions; among the rest, 85% have proposed usage of wireless implementation using Wi-Fi, Bluetooth and other wireless IoT protocols for communication, and remaining are using wired technologies such as Ethernet, USB, serial communications in user networks. Most of the solutions preferred wireless solutions because of reduced cost of deploying architecture [28, 29]. Improvements in networking protocols are also proposed for better performance [27].

Software solutions are information management systems mainly used for processing the accumulated data from sensors. To achieve the objective of smart parking and management systems, solutions are integrated with software technologies in order to provide intuitive tools which are easy to use and affordable for users [20]. Software solutions implemented provide features such as information management, analysis and prediction, and e-parking services. Some authors proposed a processed system in which driver requests and real-time information of the driver is collected, and slots are allocated through Variable Message Signs (VMS) or Internet [30]. A similar interface is provided to users, which allows to book parking space in some systems [31]. In both, the system directly communicates with sensors and processes data in real time. Many systems are handling states of parking slots: free or occupied. Some monitor and handle scenarios like illegal parking manners, unpaid parking and also monitor parking limits are exceeded or not [32].

Some solutions shows real-time parking information to the users and handle queries; payment information is also saved [33, 34]. Schema is also being used to store user location, payment details as shown in [35]. Many emphasise on cloud storage of data collected [23, 25].

Prediction using the collected information through sensors is another approach. This allows pre-booking of slots in nearby locations by taking other factors such as free space, distance, traffic flow [30, 36].

E-parking allows users to decide manual or automatic allocation of parking slots [37]. Once information is collected, it is being presented to the user through user side applications such as mobile or web interfaces.

3 Proposed Method

In this project, authors have proposed a way to reduce the congestion caused by the improper use of parking spaces. A real-time indication of occupancy of parking spaces by building a deep learning-based framework will be a robust, inexpensive solution to the problem. Here, authors are assuming parking spaces are preinstalled with surveillance cameras, from which the author will be taking live footage and feeding them to the frame extraction system followed by the slot extraction system. The extracted slots will be classified as free or occupied spaces, and this result will be live updated on the hosted server. The server is connected to mobile and web applications. The results of the CNN model will be collected over a long period and fed to the LSTM model which will give predictions for space availability in the parking area, considering the factors of day, time, weather conditions.

Initially, the model may or may not provide very accurate results, but when a large dataset is collected over a long period with large variations in considered factors, surely there will be an output leading to good results and dependency.

Along with the CNN model, the live stream frames are also fed into our anomaly detection model, to detect any anomaly that has occurred in the parking space. Some common anomalies that occur in parking space are Theft, Accident, Explosion, and Fights. Our model will be able to detect all these anomalies so that the admin can be notified about these activities in real time and he can take serious action accordingly.

3.1 Parking Occupancy Detection

Dataset used for training and validating the data:

The authors have used the CNRPark datasets for training and validating the Convolutional Neural Network (CNN) model. CNRPark+EXT is a dataset used in our smart parking system with roughly 150,000 images labelled; CNR-EXT, which is composed of images collected over the period of November 2015 to February 2016 where multiple weather conditions are covered by nine different cameras with different perspectives. This subset captures different situations based on light conditions and also includes ‘partial occlusion patterns’ due to obstacles such as trees, lampposts, etc. [8]. From this huge dataset, the authors used 70% of data in training and 30% for validating our results (Fig. 1).

Fig. 1
figure 1

Different scenarios for space detection

Full size image

Assessment of the Proposed CNNs

This chapter is targeted to find the occupancy of parking spaces using the frames obtained from the surveillance camera feed. The camera-based system is considered a more cost-effective approach as compared to other solutions implemented. The proposed Sequential Convolutional Neural Network Architecture consists of three convolutional layers. All the patches are shuffled and resized to a fixed size of 224 × 224 pixels. Hence, each image is classified independently from each other. In order to limit the problem of over fitting, authors split the dataset into 80% and 20% for training the model to classify whether the slot is empty or occupied and for testing, respectively. From the training dataset, 15% is used for validation. This allowed us to make the model robust for the proposed solution to possible changes that may occur in the real-time situation [9, 10].

The authors trained the model with randomly cropped images to 224 × 224 × 3, while the testing is done with resizing them to 224 × 224 × 3 centrally. The hyper parameter ‘relu’ (rectified linear unit) is used as activation for all the layers and ‘softmax’ is used as activation in the output layer.

The first convolutional layer consists of 1,50,528 neurons (i.e. input shape of 224 × 224 × 3) with strides of 4 × 4 and kernel size 11 × 11 followed by the Max pooling and strides of size 2 × 2. The second convolutional layer consists of strides of size 1 × 1 with 20 filters and kernel size 11 × 11 followed by Max pooling and strides of 2 × 2.

The third convolutional layer consists of 30 filters and strides 1 × 1 and kernel size 3 × 3 followed by Max pooling and strides of 2 × 2. Hence, giving a total 46,951 of total trainable parameters (Table 1).

Table 1 Assessment of the proposed CNNs
Full size table

This model reaches an accuracy of 97.89. The accuracy may get affected by the change in factors like the height of the camera, clarity of images, angle of view, distance from parking space, etc.

3.2 LSTM Model for Parking Space Prediction

The proposed solution involves the use of Long Short Term Memory (LSTM), which is a specific type of Recurrent Neural Network (RNN) architecture. LSTM is able to learn long time series-based data more accurately by understanding the factors like time and date of visitors in a particular parking area. Other factors can be taken into consideration such as weather, occasion, day, holidays, etc. for better results [7].

3.2.1 Dataset

The data for this model will be generated over time as our system will be used. The collected data will be parking lot occupancy on a particular day which can be further characterized based on date, month, week, occasion, and weather [6].

3.2.2 Model Structure

A typical LSTM network consists of different memory blocks called cells (the rectangles in Fig. 2). Hidden states and cell states are the two states that are being transferred to the next cells.

Fig. 2
figure 2

Model structure

Full size image

Three types of gates are used in memory blocks:

(1) Forget Gate, (2) Input Gate, (3) Output gate [8]: These gates help in regulations and changes in the memory blocks responsible for remembering.

3.3 Deep Learning Model for Anomaly Detection

As authors are making a manless parking management system, authors need to improve the security also. For improving security, authors are implementing an anomaly detection system that will create an alert if anomalies like violence, theft, accident, etc. occur in the area covered by surveillance cameras. The frames from the video will be segmented and classified into two groups positive (for anomalous) and negative (for normal). Then, will make two instance bags, using this model will be trained and used.

3.3.1 Dataset

Authors briefly review the anomaly detection datasets which include long untrimmed surveillance videos that cover 13 real-world anomalies such as Accident, Stealing, Fighting, Explosion, and many more.

3.3.2 Model

The above dataset is used to identify the abnormal activities in the parking areas. To recognize the activities, it identifies the task in two outputs based on fourfold cross-validation (Fig. 3).

Fig. 3
figure 3

Deep learning model for anomaly detection

Full size image

It constructs a 4096-D feature vector by averaging C3D [21] features from each 16-frames clip followed by an L2-normalization. The feature vector is used as an input for the nearest neighbour classifier. The Tube Convolutional Neural Network (TCNN) [11, 12] introduces the tube of interest (TOL) pooling layer to replace the fifth 3d-max-pooling layer in the C3D pipeline. The TOL pooling layer aggregates features from all the clips and outputs one feature vector for an entire video [13].

3.4 User End Application

The final objective of our smart parking system is to provide a convenient and intuitive system for our users so to interact with our system effortlessly with the technology affordable to them for use.

Here, the author wants to provide a tool, as easy to operate as it could be and provide means to find the most convenient parking slot at the given time and location in the least time possible. The mobile technologies are growing day by day, reaching to every person, providing them an access to the new world of possibilities. Given the rise of this technology, the best solution was to build an Android mobile application accompanied with a web application which can provide our customer a platform through which they can easily either check or book the available slots for the location they are interested in. On top of the native Android language, that is, Java, authors have used Firebase to accompany us on the journey to be fast and easily accessible. Firebase helps to notify the updates in real time to the user. Authors have also integrated Google Maps in our application to allow users to search the location of the parking space directly from the Maps UI [14].

Our application starts with the Login screen where the user is asked to Login to our platform or continue as the guest user. Then the user is greeted with the welcome screen where he/she can search a location either by typing it in the search box or adding a marker on the Map which can be opened from the Map Icon besides the search button. Given the appropriate input from the user, authors display the details of the location, that is, Open Time, Crowd Prediction for that particular day and number of slots available in that parking space. Then the user has an option to book one or multiple slots for a specific time, also chosen by the user. As the user progresses, they will be sent to the payment gateway as they proceed to pay in the advance. After the payment is successful, they will be sent a confirmation through SMS and email and also can view their booking status through the Booking History menu. In the Booking History menu, the user can view their booking status, details and can edit the time or date of their booking. They will also be able cancel the request for a refund.

As the user interacts with the Android mobile application , the information about the user’s booking history will get synchronised with the Firebase through Firebase API available for Android mobile application. As the user information gets synchronized with the Firebase DB, he/she can view their information about the booking status and history as a web application also (Fig. 4).

Fig. 4
figure 4

Demonstration of user interface

Full size image

4 Results and Conclusion

A centralized and efficient image-based system is designed and developed for the management of parking spaces using deep learning models. The proposed smart system is equipped with cameras that are usually installed in the parking areas which exempts the additional cost of installation and makes the system dynamic and easy to install. The overall performance of the CNN model used for occupancy of parking space in different weather conditions (sunny, cloudy, rainy) is reported as approximately 97.5 ± 0.25. The three performance measures: Accuracy, Recall and Precision are used for the classifiers for the various weather conditions, and they lie very similar in the range of 97.34–97.75%, 97.53–97.71%, and 97.27–97.67%, respectively (Fig. 5).

Fig. 5
figure 5

Achieved accuracy corresponding to different weather images on which model is trained

Full size image

The proposed image-based system is suitable for real-time applications as it processes all the parking spaces in a particular frame in approximately 2 s. The system will also predict the peak time of the traffic so that users can check the availability of parking slots as per their convenience and book them in advance accordingly. Since the whole system is automated and does not need any human interference, therefore, it required a smart surveillance system for security of the parking area, which covers 13 real-world anomalies such as Accident, Stealing, Fighting, Explosion, and many more. Thus, authors have successfully implemented a Smart Parking Management System.