Shape-based Fruit Recognition and Classification

Abstract

Classification of fruits is traditionally done using manual resources due to which the time and economic involvements increase adversely with number of fruit types and items per class. In recent times computer based automated techniques have been used to alleviate this problem to a certain extent. These techniques utilize image analysis and pattern recognition methodologies to automatically classify fruits based on their visual features like color, texture, and shape. However, challenges of such techniques include the fact that fruit appearances differ due to natural environments, geographical locations, stages of growth, size, orientations and imaging equipments. In this paper, a shape based fruit recognition approach has been proposed which is independent of many of these factors. It involves a pre-processing step to normalize a fruit image with respect to variations in translation, rotation, scaling and utilizes features which do not change due to varying distances, growth stages and surface appearances of fruits. The proposed method has been applied to 210 images of 7 fruit classes. The overall recognition accuracy ranges from 88–95%.

1 Introduction

Agriculture and cultivation of fruits and vegetables are major revenue earners in many countries of the world. These, however, need expert resources and knowledge for planting, harvesting, sorting & packing [2]. In recent times, auto-harvesting robots are being used to reduce the manual effort to a large extent as well as the time involvement. Image analysis and pattern recognition techniques are used to identify fruits and vegetables using their visual characteristics. However, variation in viewpoints, distance, and illumination pose are the major challenges for an imaging system for reliable recognition. Additionally handling inter-class similarities and intra-class differences are required for robust classification systems. This research work studies recognition and classification of fruits for auto harvesting and attempts to address some of these challenges.

Extant literature consists of a number of automated fruit recognition approaches based on visual features like color [1, 3,4,5,6,7,8, 10,11,12], texture [5,6,7,8,9,10,11,12], shape [2,3,4,5,6, 8, 10, 11], and size [3, 10, 11]. But, these include a number of deficiencies and shortcomings. Shape-based approaches have translation, rotation, and scaling dependency, the surface features (color and texture) may be affected by surface defects, fruit size changes over different growth stages and so on. To overcome these challenges a shape-based recognition and classification method has been proposed, which is independent of translation, rotation, uniform scaling, surface features, and growth stage of fruit.

The rest of this paper is organized as follows. Section 2 surveys the related previous works. Section 3 contains the explanation of the proposed approach. Experimentation and corresponding results are reported in Sect. 4 followed by the analysis of results in Sect. 5. Finally, Sect. 6 concludes the work with future scopes.

2 Previous Work

There are a number of works related to fruit recognition and classification based on visual features captured from images. A color histogram based fruit recognition method has been proposed to recognize multiclass fruits [1]. In RGB color space, histogram of each color channel has been quantized to 16 bins. In total, it generates 48 features, but using chi-square method best 25 features have been selected. Experimentation results show that system performs well with k-NN classifier for 32 subclasses of fruit. A classification system has been developed to classify 3 sub-types of Mango fruit [2] with object contour model, region based descriptor, boundary based descriptor, and Bayes classifier. A new method for fruit recognition has been proposed combining color (Mean of RGB Color Values), shape (Roundness), and size (Area, Perimeter) features [3] to improve classification accuracy. The system archives up to 90% accuracy with Nearest Neighbor Classifier. Apple, strawberry, and orange classification process have been proposed using shape, color, Scale Invariant Feature Transform (SIFT) and Random Forest classifier [4]. The system has not been tested with the fruit type; whose color and shape both are nearly similar. Another fruit classification approach, based on combined texture (Hausdorff dimension), color (Mean of Hue values), and shape (Roundness) features using Neural Network Classifier, has been proposed [5]. The proposed approach has been experimented on 150 images of 6 fruit class. An automatic classification system has been proposed [6] for five classes of fruit. The authors combined color (Mean, Variance), shape (Area, Perimeter, and Eccentricity) and texture (Statistical features from GLCM) features and achieved best classification result with SVM classifier. A multiclass fruit detection method has been proposed using a fusion of Global Color Histogram, HOG, LBP and Gabor LBP features for multiple color channel of multiple color space [7]. Experimentation has been done with 5 classes of fruit. Due to the larger dimension of the feature vector, the detection speed is not acceptable in comparison to real-time image recognition. Based on a combination of color (HSV histogram), texture (LBP), shape (Roundness) features [8] and Back Propagation Neural Network classifier an automatic fruit type recognition method has been proposed. The method is not stable with different camera angle and lighting. Some researcher from GLA University proposed a texture feature and SVM based fruit and vegetable classification method [9]. A single histogram feature has been considered with the combination of sum and difference intensity of neighboring pixel for 3 color channels. This achieved a good accuracy but, intensity depends on various conditions like illumination. In paper [10], authors combined color (Mean of RGB Values), shape (Roundness), size (Area, Perimeter) and texture (Entropy) features, to achieve a most improved accuracy of fruit recognition. The proposed method archives 95% overall accuracy for classification of 5 fruit class with k-NN classifier. The auto-identification method of sub-type of a Date fruit has been proposed to reduce manual effort of sorting in the Middle East [11]. The authors have experimented with 7 sub-types of Date. In total, color, shape, size and texture related fifteen features are used for recognition purpose. Neural Network Classifier shows satisfactory result than other classifiers. Each type of feature has its own limitations. Most of the authors considered combined approach of color, texture shape and size features to overcome those limitations. In this paper, a complete shape based approach has been proposed to overcome some limitations.

3 Proposed Approach

Challenges of an automated fruit recognition system include handling variations of visual features due to translation, rotation, and scaling of captured images, as well as changes pertaining to different growth stages of the fruit. A shape-based approach has been proposed here to deal with these challenges. Figure 1 depicts the flow diagram of the proposed method.

Fig. 1.

Process flow diagram of complete proposed system

3.1 Pre-processing

An RGB color image of a fruit, Fig, 2(a), is converted to a grayscale image using the Eq. (1).

Fig. 2.

(a) RGB Color Image (b) Grayscale Image (c) Binary Image (d) Background noises removed (e) Rotated Image (f) Inner holes filled (Color figure online)

$$ GrayImage = 0.2989 \times R + 0.5870 \times G + 0.1140 \times B $$

(1)

The grayscale image, Fig. 2(b), is subsequently binarized using an appropriate threshold [13], Fig. 2(c). Background noise is removed using a threshold based on the actual number of pixels in the region, Fig. 2(d). The angle of orientation of the major axis of the fruit with respect to the horizontal is computed from the image and it is then rotated by this angle to align the major axis along with the horizontal, Fig. 2(e). This step is necessary to normalize the object of interest against arbitrary orientation factors which might affect the shape features, discussed in the next section. Any hole inside the fruit region which might have resulted from the binarization operation, is filled up by the morphological flood-fill operation, Fig. 2(f).

3.2 Feature Extraction

To classify the shape following shape based features are computed from the pre-processed image. These are Area ($A$), and Perimeter ($P$) of the fruit region, Major Axis Length ($W$), Minor Axis Length ($H$), and Distance between the Foci ($F$) of an equivalent ellipse, Width ($W_{b}$), Height ($H_{b}$), and Area ($A_{b}$) of Minimum Bounding Box, Area ($A_{c}$), and Perimeter ($P_{c}$) of Smallest Convex Polygon. All the parameters are measured in terms of pixels. Width measured in horizontal direction and height measured in vertical direction. In Fig. 3 parameters have been visualized for a cucumber sample. White colored region in this image shows the actual fruit Area, and fruit Perimeter is calculated from the border of the white region. Approximated ellipse from the fruit region is represented with blue. Major and Minor axis are marked with cyan horizontal and magenta vertical lines passing through the centroid of the ellipse. Centroid of the ellipse is marked with a red cross. Two small red circles show the position of Foci on the Major Axis. Minimum Bounding Box is represented by a red rectangle while a green polygon represents the Smallest Convex Polygon.

Fig. 3.

Shape parameters for a cucumber sample (Color figure online)

These feature values are both dependent on the orientation and scale of the image. The variation due to orientation is normalized by the pre-processing step, as discussed in the previous section. To make the features independent of scaling factors, eight extended descriptors are derived from the basic shape descriptors as shown below.

$$ sf_{1} = F/W $$

(2)

$$ sf_{2} = H/W $$

(3)

$$ sf_{3} = 1 - \left( {H_{b} /W_{b} } \right) $$

(4)

$$ sf_{4} = P/\left( {H + W} \right) $$

(5)

$$ sf_{5} = A/P^{2} $$

(6)

$$ sf_{6} = A/A_{b} $$

(7)

$$ sf_{7} = A/A_{c} $$

(8)

$$ sf_{8} = P_{c} /P $$

(9)

It is to be noted that all extended features are ratios which make them invariant to uniform scaling. Final feature vector ($SF$) contains those 8 extended shape descriptors.

$$ SF = \left\{ {sf_{1} , sf_{2} , sf_{3} ,sf_{4} , sf_{5} , sf_{6} , sf_{7} , sf_{8} } \right\} $$

(10)

3.3 Classification

A fruit class $i$ contains a set of $n$ number images. Each class is specified by its feature values $T_{i}$ obtained at the time of training process. A test image $S$ is said to belong from a specific class if the matching probability of its feature value is maximum for a specific training class. As such there is no standard mathematical model which can be used to classify from sample observed data.

Naïve Bayes classifier is one of the superior classification models which work on the basis of Bayes theorem of independent assumptions between predictors using Eq. (11). It assumes that the effect of the value of a predictor $\left( X \right)$ on a given class $\left( c \right)$ is not dependent on the values of other predictors.

$$ P\left( {c|x} \right) = P\left( {x|c} \right).P\left( c \right)/P\left( x \right) $$

(11)

In Eq. (11), $P\left( {c|x} \right)$ is the posterior probability, $P\left( {x|c} \right)$ is the likelihood, $P\left( c \right)$ is the class prior probability, and $P\left( x \right)$ is the predictor prior probability. This classifier is well known for sophisticated classification and handling large dataset.

Back Propagation Neural Network is an effective classifier known for its non-linear solution ability. It consists of multiple layers of neurons which can be varied according to the complexity of the problem and is thus also known as Multi-Layered Perception (MLP). During the learning process, the difference between actual output and desired output is calculated and back-propagated to the previous layer(s). To scale the accuracy, the connection weights are adjusted using a transfer function.

K-Nearest Neighbor (k-NN) is a simple and efficient classification algorithm which assigns the class of sample data based on a distance metric and the number of training samples within a specified distance in feature space. There are few types of distance metric e.g. Euclidean, cityblock, hamming etc. Here k is the number of nearest neighbor for the prediction. Larger k value gives smoother boundary but decreases efficiency.

4 Experiments and Results

To test the effectiveness of the proposed system, experimentations are performed on 210 digital images belonging to 7 fruit classes viz. Apple, Banana, Cucumber, Lemon, Mango, Strawberry, and Tomato. Most of the images are captured using a smartphone camera and some are downloaded from websites [14]. Each class contains 30 images. Half of the images are used for training and rest for testing. Samples of the training and testing dataset are shown in Figs. 4 and 5 respectively.

Fig. 4.

Training samples: (a) Apple (b) Banana (c) Cucumber (d) Lemon (e) Mango (f) Strawberry (g) Tomato

Fig. 5.

Testing samples: (a) Apple (b) Banana (c) Cucumber (d) Lemon (e) Mango (f) Strawberry (g) Tomato

To introduce a large amount of variation the fruit images has been captured by varying camera to fruit distance, fruit orientation, and fruit size. Figure 6 represents the variations of some of the most discriminative shape features on 15 samples for each of the 7 fruit class training images. The horizontal axis represents the index of training image and the vertical axis represents corresponding feature value for each fruit class normalized within the range 0 to 1. Each fruit type is marked with a different color as given in the legend.

Fig. 6.

Variation of features on the training dataset using legend shown

Table 1 shows the overall classification accuracy with the 8 features and their combinations using Naïve Bayes classifier on the dataset of 7 fruit class. The first eight rows pertain to the individual features, row 9 to the best two features, row 10 to the best four features, row 11 to the best six features and row 12 to all the features. It is observed that the top four features are sufficient for providing an accuracy of 95.24%.

Table 1. Classification results of feature(s)

Table 2 lists the best classification accuracies of proposed method using Naïve Bayes Classifier, Neural Network Classifier, and k-NN Classifier on this dataset.

Table 2. Classification results of proposed method using different classifier

Figures 7 and 8 shows the classification plot on test dataset using Naïve Bayes classifier and KNN classifier respectively. The horizontal axis represents the index of test images and the vertical axis represents the class labels.

Fig. 7.

Classification plot using Naïve Bayes Classifier

Fig. 8.

Classification plot using KNN Classifier

Figure 9 shows the classification plot on test dataset using Neural Network Classifier. The horizontal axis represents the test file index and the vertical axis represents the classification output of corresponding class as mentioned in the legend.

Fig. 9.

Classification plot using neural network classifier

5 Analysis

Table 2 in the previous section depicts that the proposed method provides an overall classification accuracy ranging from 88.57% to 95.24%. It is also observed that the overall accuracy is better with the Naïve Bayes Classifier than the other classifiers used for experimentations. The reason behind this is that the Naïve Bayes Classifier assumes prediction effect of a feature value on a fruit class is independent of the values of other features. Table 3 depicts the classification accuracy using previous approaches [1,2,3] on current dataset.

Table 3. Classification results of previous approaches on this dataset

Figure 10 depicts the classification accuracy of proposed approach vis-à-vis the previous approaches [1,2,3] for all the 7 classes of fruit. Performance is not satisfactory with the color-based approach in [1] for the fruit pair Banana- Lemon and Strawberry-Tomato because the color of the two fruit member of each pair is nearly similar to each other. The shape-based features used by in approach [2] are effective for classification of small number of classes and where there is a large amount of shape variation among the classes. In the current work, the number of classes is large and the shape-based descriptors, used by the authors [2], are closely similar for the classes like lemon, strawberry, and tomato. Approach [3] is a mixed approach of color, shape, and size based features. The shape and size based descriptors, used by the authors, enormously diverges due to geometric transformation as well as different growth stages of a fruit. To establish the robustness of the proposed method this work introduces sufficient amount variation of shape, size and surface color for a particular class. This is the reason behind the very poor classification performance of previous approach [3]. Figure 11 Shows how the overall classification accuracy has been improved than the previous works.

Fig. 10.

Comparison of recognition accuracies with different approaches

Fig. 11.

Improvement of overall accuracy

6 Conclusions and Future Scopes

This work proposes a shape-based fruit class recognition and classification using a number of classifiers. It achieves 95.24% overall classification accuracy over seven classes of fruits varying in shape, color, size and orientation. The major contributions of this paper involves proposing schemes which are: (1) invariant to translation, rotation, and uniform scaling, (2) not dependent on surface based features (3) invariant to the different growth stage of fruit.

Fruit Recognition and classification is a very demanding research topic for the automation in the agricultural field. Recognition of fruit class in the 3D environment can be researched in future.