HandFruitNet: A Deep Learning Based Model for Fruits Classification from Hand

Abstract

Detecting a hand and classifying fruits from an image is a significant research area in the field of computer vision. For many applications, detecting a hand and classifying fruits together from a single system is also significant. This type of system is also effective for shop owner to detect suspicious people, as well as for the disabled people to easily pick different fruits from fruits shop without any other help. In this regard, in this paper, a system is proposed where hand detection and fruit classification are performed together when the fruit is in a person’s hand. For that, initially, a hand is detected by using two methods. The first method is developed based on the hand features extraction, i.e., contour feature of the hand. Another method is developed based on deep learning, i.e., YOLOv3. After that, a convolutional neural network model based on deep learning is developed to classify the fruits. Finally, both models are combined to detect hands and classify the fruits from a single system. For training the model, 11 K and Fruits_360 datasets are used for hand detection and fruits classification, respectively. Beyond that, two different own datasets are created for hand detection and fruit classification. These datasets are utilized during testing the models. Among the models, the feature extraction and deep learning based hand detection model show 95.17% and 99.90% accuracy, whereas deep learning based fruits classification model shows 99.99% accuracy.