Keywords

1 Introduction

Health is a primary matter of concern. Food that we eat at home is mostly kept in the fridge. When we shop for groceries, we do not know exactly what is there in the refrigerator or the exact quantity of a particular food item. This problem exists for all people who use refrigerators. If we do not know exactly what is there in the fridge, we will not be able to buy the proper quantity or the essential items of groceries or other edibles. The refrigerators that we use in our daily life are not smart enough to understand our needs and convey the same to us. Their only role is to keep the food fresh and healthy. However, with the upcoming advanced generation, with digital technology, every device is getting smarter day by day. A device like a refrigerator can also be a smart device that makes the user’s life easier by prompting about the items the user likes more and informing him for necessary procurement if they are missing.

Smart Refri: Smart Refrigerator for tracking human usage and prompting based on behavioral consumption.

The objectives of Smart Refri are as follows:

  • To recognize the food items kept in the fridge.

  • To observe the pattern followed by the user while taking out and storing food items in the refrigerator.

  • To learn the behavioral patterns exhibited by the user while using the system.

  • To prompt the user, if items discovered from his consumption behavior are found either in less quantity or missing.

2 Existing Systems

2.1 Smart Refrigerator –a Next-Generation Refrigerator Connected to IoT

The design of the proposed system of Smart Refrigerator [1] is based on the core concept of product identification utilizing the RFID technology. The researchers of this paper have considered a use case that in the nearby future, all or most products bought from the store will have a tracking RFID tag along with information stored in a global level database maintained by all or most manufacturers, that will eventually serve two purposes: Firstly, the manufacturers will be able to easily track their products from the assembly line, transportation route and finally on the store shelf, which will provide them with more information about assembly and transportation costs along with information about product visibility in the store itself. Secondly, the consumer will be able to easily access the above information, in time. The set of well-defined functionalities on which the concept of the proposed system is built are-

  1. A.

    Identifying new products

  2. B.

    Identifying removed products

  3. C.

    Inventory and shopping list

  4. D.

    Triggered alerts

  5. E.

    Information output

However, the proposed system does not provide a solution for the identification of products stored in the refrigerator that is not contained in RFID-equipped packaging like fruits and vegetables.

2.2 IoT Based Smart Refrigerator System

The proposed design is to implement a smart refrigerator system [2], which is easy to use and economical for the user. The system is capable of notifying its owner about the activities going on inside the fridge via the android app that is developed to be used as a GUI for the user. Through the application, the user will be able to see the condition of the food items kept inside the refrigerator. An alert notification is sent to the user’s mobile when the weight of the items is below the set threshold value. This helps the user to replenish any food item in the proper time. The proposed system is designed by following the steps mentioned below:

  • The initial value (quantity) of a particular food item kept in the fridge is determined. This step is followed for all food items kept in the smart refrigerator.

  • The food items are picked from the inventory tray of the fridge.

  • The status of food items is compared (i.e. present value with threshold value).

  • If the load value is lesser than the threshold value then a notification is sent to the user on the mobile phone.

The proposed smart refrigerator is designed for efficiently managing items stored in it and therefore enables a healthy lifestyle. However, it does not act on analyzing the behavioral patterns of the user and thus helps to minimize food waste.

2.3 Smart Refrigerator Based on Internet of Things (IoT)—An Approach to Efficient Food Management

The proposed concept of the smart refrigerator [3] is designed to include a variety of functionalities that help in both improved food management and also to keep track of the balanced diet for the consumer. The primary issue is to allow consumers to allow the distribution of the food to the neighbors that need the same grocery items before the expiration date has been reached. The application of this project will eventually allow improved food management and hence reduce food wastage. The consumer can input a pre-determined minimum quantity for a food item and when the amount falls below that threshold value, this will trigger an automated connection to the central database to find whether the same item is available for selling by another consumer. The contact to the consumer can be made through a text message and the quantity at the price available for sale can be specified. Since the distribution is done within the same neighborhood, the cost of transport will be reduced and simultaneously it will be a much efficient way for food management instead of letting it expire on the fridge shelf without consumption. In case, the potential buyer is not interested in the purchase, an online option to purchase from a fresh grocery store or a farmer’s market can be made available. The user can also have an option to discard the purchase altogether.

3 Proposed System

With the tremendous improvement in technology, all devices are connected to the internet, which forms the internet of things. The sensors are used to collect the data and send them to a host where it is intended to be processed over the internet. Improvement in technologies, make our day-to-day life simpler. The technologies implemented using IOT in electrical appliances at homes make it smarter. One of those technologies is a smart refrigerator which is used to store food items. Refrigerators are used to prevent the spoilage of food and keep it fresh. It reduces illness and makes our lifestyle healthier in the modern world.

The proposed IOT based device uses Raspberry Pi as the central server to make the fridge smarter. It will be able to detect the different items kept inside the refrigerator and also measure the quantity of the items present inside it. The device will learn the pattern in which the items from the refrigerator are being used in day-to-day life. A Mobile application will be developed to monitor the quantity of items in the refrigerator and notify the user about the items inside the fridge, also about the quantity and if there is any shortage.

When the refrigerator door is opened, the camera clicks a picture (Say img1). Figure 1 will become the input to the TensorFlow Object detection Classifier which will detect the objects (food items) in it. After the door is closed, the camera will click another picture (Say img2). Again the TensorFlow Object detection Classifier will detect the objects (food items) in it. After the objects are detected from both the images then a comparison will be made among the objects in both the images. The objects which will not be found in the Fig. 2 will be considered to be consumed. The objects which are consumed will be inserted into the database. Then Self Organizing Maps(SOM) will read from the database, learn the weights (quantity) of the objects in it and thus it will create a consumption behavior. The consumption behavior will then be added to the Behavioral Database. The alert system created in the Android app will read from the Behavioral Database about the objects that are less than the threshold value and thus it will alert the user about it.

Fig. 1
figure 1

The proposed model

Full size image
Fig. 2
figure 2

Block Diagram for the proposed system

Full size image

TensorFlow Object Detection Classifier. TensorFlow is an open-source deep learning framework created by Google Brain. For training the classifier, we have used TensorFlow’s Object Detection API [4]. An image classifier is trained so that it can detect objects in an image and then classify the set of data into different classes or categories.

Self Organizing Maps (SOM). The Self-Organizing Map is one of the most popular neural network models and it belongs to the category of competitive learning networks. The SOM is based on unsupervised learning, which means that little needs to be known about the characteristics of the input data and that no human intervention is needed during the learning.

Behavior Database. The information produced as a result of actions, typically commercial behavior using a range of devices connected to the Internet, such as a PC, tablet, or smartphone is referred as a Behaviour Database. It is therefore not static. Behavioral data is valuable because it provides information above and beyond what static data can provide. The collection of all behavioral data of users is stored in the Behavioral Database.

Alert System. The Alert System is an Android application accessible to the user via a smartphone. A notification sent by the application to the user’s device alerts him/her regarding the current circumstances in real-time.

4 Results and Discussion

The goal of the proposed system is to mitigate the problem of food wastage and provide an efficient solution to handle the same. It will allow the enhancement of IoT technologies in new ways to realize the vision of sustainable IoT applications that in return will enable things and objects to become more reliable, resilient, more autonomous and smarter [5].

The proposed system will result in a smart refrigerator system that will be able to help the user to look through the items that are present inside the refrigerator via an android application, which will also prompt the user if any of the items are required to be replenished. The system also learns the pattern in which the items are being consumed by the user.

This system comprises an object detection image classifier that detects the items kept inside the fridge, an android application that will connect the user with the refrigerator to look for items that are kept inside it, and it also includes a behavior learning system that will learn the user’s behavior towards their daily usage of the items kept inside the fridge. An alert system is also included inside the android application which will prompt the user about the items that are to be refilled and also about their consumption behavior of the items (Fig. 3).

Fig. 3
figure 3

Output for object detection

Full size image

The object detection algorithm used to achieve the above outputs is Faster R-CNN. The steps followed by a Faster R-CNN algorithm [6] to detect objects in an image are summarized below:

  • An input image is taken and passed to the ConvNet which returns feature maps for the image

  • Region Proposal Network (RPN) is applied on these feature maps and object proposals are obtained

  • ROI pooling layer is then applied to bring down all the proposals to the same size

  • Finally, these proposals are passed to a fully connected layer in order to classify any predict the bounding boxes for the image

ConvNet. A Convolutional Neural Network (ConvNet/CNN) is a Deep Learning algorithm that can take in an input image to assign importance (learnable weights and biases) to various aspects/objects in the image and then be able to differentiate one image from the other. Compared to other classification algorithms, the pre-processing required in a ConvNet is much lower. With enough training, ConvNets have the ability to learn the filters/characteristics while in primitive methods these filters are hand-engineered. Without losing features that are critical for getting a good prediction, the role of the ConvNet is to reduce the images into a form that is easier to process.

Feature Maps. The convolutional layer is the main building block of CNN which utilizes the concept of convolution. Convolution is a mathematical operation to merge two sets of information and in our case, the convolution is applied on the input data using a convolution filter to produce a feature map. An image is represented as a 3D matrix with dimensions of height, width and depth in reality, where depth corresponds to color channels (RGB). A convolution filter has a specific height and width, like 3 × 3 or 5 × 5 and by design it covers the entire depth of its input so it also needs to be 3D.

We perform multiple convolutions on a single input, using a different filter for each that results in a distinct feature map. Stacking all these feature maps together becomes the final output of the convolution layer.

Region Proposal Network (RPN). The purpose of the Region Proposal Network is to propose multiple objects that are identifiable within a particular image. Using the features that were computed by the CNN, it finds up to a predefined number of regions (bounding boxes), which may contain objects.

ROI pooling layer. Region of Interest (ROI) pooling is used for utilizing a single feature map for all the proposals that are generated by RPN in a single pass. It solves the problem of fixed image size requirements for object detection networks. The RoI Pooling layer is a type of max-pooling, where the pool size is dependent on the size of the input. It ensures that the output is always of the same size. This layer is used because the fully-connected layer always expects the same input size, but input regions to the fully-connected layer may have different sizes.

Fully Connected Layer. Fully connected layers are an essential component of Convolutional Neural Networks (CNNs). It has been proven to be very successful in recognizing and classifying images for computer vision. The CNN process begins with convolution followed by pooling, breaking down the image into features, and then analyzing them independently. The result of this process is fed into a fully connected neural network structure that drives the final classification decision.

The final decision of the object detection with a proper bounding box and labeling is done with the help of a functional matrix. It is a single dimension array, containing the scores with respect to the classes of objects (Fig. 4).

Fig. 4
figure 4

Real time object detection

Full size image

However, the above mentioned approach could not always deliver accurate results. The Faster R-CNN algorithm failed to detect multiple food items in a single image while providing accurate results in detecting single food items within an image. Therefore, a new approach was adapted to obtain better and accurate results for object detection. The object detection algorithm used in this approach is called YOLOv3. It is the third object detection algorithm in YOLO (You Only Look Once) family with improved accuracy and is more capable of detecting small objects.

YOLOv3. The YOLOv3 algorithm works by applying a single neural network to the full image. This network divides the image into regions and then predicts bounding boxes and probabilities for each region. These bounding boxes are weighted by the predicted probabilities meaning that each object detected in an image is bounded within a box with its corresponding probability in percentage. The YOLOv3 model has several advantages over classifier-based systems. Its predictions are informed by the global context in the image as it looks at the whole image at test time. Unlike systems like R-CNN which require thousands for a single image, YOLOv3 also makes predictions with a single network evaluation. This makes it extremely fast, faster than R-CNN (1000x) and Fast R-CNN (100x) [7].

5 Conclusion

A Smart Refrigerator system has been proposed for detecting the items kept inside the refrigerator, which learns the behavioral consumption of a single user and notifies the user if there is any shortage of food items. Also through this smart refrigerator, people can save their time in searching for the items inside the refrigerator. We are confident that such type of smart refrigerator will be an important component in the future smart home. The concept of a smart refrigerator is far more reaching than notifying the user about the contents of the refrigerator. A smart refrigerator is cost-effective and user-friendly. Thus, this system saves manual effort and time in predicting future needs. In the future, this system may be expanded to detect the freshness of the food items and also suggest the dishes that can be made from the available food items in the fridge considering the freshness of that item. This system may also be made to display the content inside the fridge from a remote place.