1 Introduction

Half of the world population consume rice as their daily diet. According to the World Bank, the projected demand for rice consumption will increase by 51% by the year 2025. The production of rice is affected by several factors, including mineral deficiency. The minerals required for maintaining standard health of rice crop are Nitrogen (N), Potassium (K), Phosphorus (P), Boron(B), Zinc (Zn), Sodium (S), Copper (Cu) and Magnesium (Mg) etc., (Sanyal et al. 2007). Nitrogen (N) is one of the essential minerals for growth and development of rice plant. To escalate the productivity, the utilisation of Nitrogen (N) fertilizer in cropland is increased since the last six decades (Mulvaney et al. 2009). Nitrogen (N) deficiency affects rice growth and development, producing yellow leaves, dwarf plants and lower grain yields (Borrell et al. 1998, 2001; Lian et al. 2005). Sometimes, the farmers apply excess nitrogen with an expectation of more production, but it causes stem rot disease. Simultaneously, the application of balanced mineral develops immunity among the rice plant for getting affected by diseases. So, it is essential to predict the nitrogen requirement in cropland for standard growth and development of rice plant. The agriculturist mainly determines the plant nutrition status using information about the leaf (Shi et al. 2009; Chen et al. 2013, 2014). When rice exhibits nutrition deficiency, the leaf sheath will also present specific symptoms (Chen et al. 2014). Therefore, this study is carried out based on images of rice leaves for diagnosis of nitrogen deficiency of rice plant.

Western Odisha is a territory in the western part of Odisha, India and well known for rice production, especially Sambalpur and Bargarh districts (known as the rice bowl of Odisha). In this region varieties of rice cultivars are cultivated in two farming season a year. The Kharif season (July–October) depends on monsoon and Rabi season (October to March) depend on the water supply of Hirakud dam. It has been reported every year; the paddy fields are damage due to various diseases and pest attack. According to the report of (Sahu et al. 2005), the western Odisha, especially Sambalpur and Bargarh district, have laterite, mixed red and yellow–red soil. In Sambalpur district, the soil is laterite soil, which contains very less nitrogen. Again, the mixed red and yellow soil are present in both Sambalpur and Bargarh district in which the upland soil has low nitrogen and phosphorus, and low land has low nitrogen and medium phosphorus. Fortunately, my university is situated in Sambalpur district. “PLAGRON” describes plants with a nitrogen deficiency are also more susceptible to problems like diseases and insects. Again, excess application of Nitrogen (N) causes stem rot disease. Therefore, it is a necessity to know the actual level of Nitrogen deficiency. In addition, no significant research has been reported for prediction of Nitrogen deficiency of rice crop in an early stage based on computational intelligence. This situation motivates us to carry forward the research for prediction of Nitrogen deficiency in rice crop.

The leaf colour chart (LCC) is developed by the International Rice Research Institute (IRRI) which is used to determine the Nitrogen (N) fertilizer needs of rice crops. LCC has five green swaps, with colours ranging from yellow-green to dark green. It determines the greenness of the rice leaf, which indicates its N content (Yang et al. 2003).

In this research, the images of rice leaves are captured with matching the swap of Leaf Colour Chart (LCC). The images are captured using a smartphone camera of 13 Megapixel in daylight as per the uses instruction of LCC. The LCC developed by ICAR-National Rice Research Institute (ICAR-NRRI), Cuttack, India has five swaps. The Swap-5 indicates sufficient nitrogen and Swap-1 indicates highly deficient nitrogen. In this research, four number of swaps are considered from Swap-4 to Swap-1, indicating slight N deficiency to high N deficiency. We have ignored the Swap-5 samples as it indicates sufficiency of Nitrogen (N) and our study concentrate on determining the deficiency of Nitrogen (N). Thus, the present prediction of Nitrogen (N) deficiency is a four-class classification problem. In this classification task, CNN + SVM is used as classifier. The LCC and matching samples of rice leaf are shown in Fig. 1. The four-level of Nitrogen (N) deficient leaf samples of rice crop are illustrated in Fig. 2.

Fig. 1
figure 1

a Leaf colour chart developed by ICAR-NRRI, Cuttack (Courtesy ICAR-NRRI, Cuttack), b matching sample of Rice leaf with LCC

Full size image
Fig. 2
figure 2

Illustration of four nitrogen (N) deficient leaf samples of rice crop

Full size image

In recent years, with the application of computer vision and machine learning, there has been an incredible advancement on disease diagnosis of crops such as detection, identification and quantification of various diseases. In most of the cases, SVM (Semary et al. 2015; Tian et al. 2010), K-Nearest neighbours (KNN) (Prasad et al. 2016; Zhang et al. 2015) and Discriminant analysis (Wang et al. 2014) are used for disease identification purposes. Many researchers reported automated rice disease diagnosis methods based on digital image processing technique (Barbedo 2013), SVM (Jian and Wei 2010), pattern recognition (Phadikar et al. 2008) and computer vision (Asfarian et al. 2014). The research is not only for rice disease classification but also for other crops such as wheat (Khairnar and Dagade 2014), maize (Zhang and Yang 2014), cotton (Shicha et al. 2007), tomato (Chai et al. 2013) etc. Although, the machine learning techniques have made the great accomplishment on image identification, till it has some limitations: restricted data handing capability, the requirement of segmentation and feature extraction (Chen et al. 2019a, b). The diseased region segmentation is not always an easy task for all agricultural images (Lu et al. 2017). Therefore, the traditional machine learning techniques face difficulty for classification of agricultural diseased images with adequate results. With the advancement of machine learning techniques, deep learning methods are capable enough to solve and model big data problems. The deep learning methods can be applied in agricultural diseased image classification without the need for pre-required processes such as segmentation and feature extraction.

In past few years, the CNN is applied in various fields such as object detection (Ren et al. 2017; Girshick et al. 2014; Girshick 2015; Zitnick and Dollar 2014; Uijlings et al. 2013), image classification (Deng et al. 2009; Krizhersky et al. 2012; Simonyan and Zisserman 2014; Donahue et al. 2013) and video classification (Karpathy et al. 2014). In the last couple of year, many research has been conducted for the diagnosis of plant diseases based on CNN (Kwasaki et al. 2015; Zhang et al. 2016; Zhao and Peng 2012; Lu et al. 2017). In addition, recently some researcher reported advance filter techniques (Chen et al. 2019a, b), hybrid feature selection method (Moslehi and Haeri 2019) and powerful classification models such as Voronoi diagram-based classifier (VDBC) and neuro-fuzzy (NF) classifier (Misra and Laskar 2019). In recent deep learning approach has been adopted in many fields such as detection of cloth quality (Ge et al. 2019), Leveraging Class Hierarchy in Fashion Classification (Cho et al. 2019), identification of mango cultivars (Borianne et al. 2019).

As far as our investigation, no research has been published for prediction of Nitrogen (N) deficiency of rice crop based on deep learning networks.

The objective of developing CNN based classification model provides a user-friendly system to farmers for prediction of N deficiency in rice crop. So far very few Web portals are accessible for prediction of N deficiency such as Leaf Coder and at Leaf+. Again, CNN is highly expected to be automated feature learning from the raw inputs systematically. Therefore, it is needful to develop CNN based model for prediction of N deficiency for rice crop.

In this paper, we present a novel CNN- based model for the prediction of Nitrogen (N) deficiency in rice crop. A total of 5790 number of healthy rice leaf are collected and pre-processed. The image collection process strictly follows the uses instruction of LCC. The images of fully expanded, young, healthy middle and/or tip part of rice leaf are captured. The images are captured either in the morning (8 AM–10 AM) or, in the afternoon (2 PM–4 PM) in normal daylight under the shade of the body. All samples are collected from the farming site of Sambalpur and Bargarh districts of Odisha, India. At the time of capturing images of rice leaf, two strategies are strictly followed. First, the rice leaf and swap of LCC must match properly. Second, behind the rice leaf, a white paper is placed to avoid unessential background. Finally, the raw images are processed by subtracting the white background. The processed images are examined through six leading CNN-based models such as AlexNet, GoogleNet, VGG-16, VGG-19, ResNet-18 and ResNet-50. Again, to improve the diagnostic results, SVM is included for better classification in all six architectures. The performance of classification models is measured in terms of accuracy, sensitivity, specificity, FPR and F1 Score.

The main contribution of this article is as follows:

  • A dataset is established, contains 5790 number of nitrogen-deficient rice leaf images in four levels.

  • The six leading learning architectures such as ResNet-18, ResNet-50, GoogleNet, AlexNet, VGG-16 and VGG-19 with SVM are used for classification of four levels of Nitrogen deficiency and performance of each classifier are evaluated.

  • The statistical analysis is conducted to choose the best classifier with consideration of 100 number of independent simulation results.

  • Also, the CNN based classifiers are compared with other traditional image classification methods such as bag-of-feature, colour feature + SVM, HOG + SVM, LBP + SVM, GLCM + SVM.

The remaining of this paper is organized as follow. Section 2 describes the CNN-based model for prediction of rice Nitrogen (N) deficiency. The experimental results with statistical analysis are reported in Sect. 3. Finally, Sect. 4 concludes the paper by discussing the future scopes.

2 CNN-based model for prediction of rice nitrogen deficiency

Deep learning is a class of machine learning algorithms that use multiple layers that contain nonlinear processing units (Schmidhuber 2015; Bengio et al. 2015). Each layer uses the output from the previous layer input. Convolutional neural networks (CNN) are classified as a deep learning algorithm. The CNN consists of convolutional layer, Rectified Linear Unit (ReLU) layer, pooling layer and fully connected (FC) layer.

For this research, we consider six leading pre-trained CNN model such as AlexNet, GoogleNet, VGG-16, VGG-19, ResNet-18 and ResNet-50. The pre-trained CNN model is not used for original classification task but, it is re-purpose for classification of Nitrogen (N) deficient rice leaf images. For this purpose, we prepare training, validation and testing image dataset. The test data consists of 400 images with 100 images of each class. The trained and validation set consists of 70% and 30% of remaining image data. The selection of training and validation data is random. The input image size is 227 × 227 × 3. To resize and conversion from gray to RGB, augmentation is used. The other benefit of augmentation is to provide additional data for network training. The CNN is multilayer structure network, and each layer produces a response. The layers extract the basic image feature and pass to the next layer. The feature use in ResNet-18 & ResNet-50 is ‘fc1000’, VGGNet-16 & VGGNet-19 is ‘fc6’ and GoogleNet is ‘pool5-drop_7×7_s1’. The activation is in GPU with a minibatch size of 32 and GPU memory have space enough to fit image dataset. The activation output is in the form of the column to fit in linear SVM training. To train the SVM, the function ‘fit class error-correcting output codes (fitcecoc)’ is used. This function returns full trained multiclass error-correcting output of the model. The function ‘fitcecoc’ uses K(K−1)/2, binary SVM model, using One-Vs-All coding design. Here, K is a unique class label. So, the pre-trained architectures are modifying as AlexNet + SVM, GoogleNet + SVM, VGGNet-16 + SVM, VGGNet-19_SVM, ResNet-18 + SVM and ResNet-50 + SVM.Finally, we have evaluated the performance of all the six classifiers. The confusion matrix is obtained by comparing the measured labels and predicted labels. From the confusion matrix, the performance measure parameter such as accuracy, sensitivity, specificity, FPR and F1 score is calculated. Here, the accuracy, sensitivity and specificity are not the sufficient parameters for performance evaluation of classifiers as the dataset is an imbalanced one. So, FPR and F1 score is taken into account for performance evaluation of classification models.

The image data are processed in all six classification models individually for classification of N deficiency level into four categories. The performance of classification models is measured in terms of accuracy, sensitivity, specificity, FPR and F1 score. To choose the best classification model, the statistical analysis is conducted.

The CNN models are created by MATLAB 2019a and statistical analysis is done using SPSS tool. Figure 3 describes the CNN model, which is built in pre-trained architecture with SVM.

Fig. 3
figure 3

The architecture of CNN with SVM for prediction N deficient levels of rice crop

Full size image

The deep features of pre-trained CNNs are extracted and used to train the SVM classifier. The deep features are extracted from fully connected (FC) layer of CNN networks. The certain layer of each CNNs models are considered for extraction of deep features, which are AlexNet, GoogleNet, ResNet-18, ResNet-50, VGG-16 and VGG-19 are fc6, pool5-drop_7×7_s1, fc1000, fc1000, fc6 and fc6 respectively. In addition, feature vectors were obtained from these layers of the AlexNet, GoogleNet, ResNet-18, ResNet-50, VGG-16 and VGG-19 networks: 4096, 1000, 1000, 1000, 4096 and 4096 respectively. The obtained deep features are employed in the classification phase by using the pre-eminent classifier SVM for classification of four levels of Nitrogen deficiency. Again, to test the new query sample, the same method is adopted as training to obtain the features. The SVM match the features of testing samples with training and predict the Nitrogen deficiency levels.

3 Results and observations

The processed image data are examined in six classification model i.e. AlexNet+SVM, GoogleNet+SVM, VGGNet-16+SVM, VGGNet-19+SVM, ResNet-18+SVM and ResNet-50+SVM. The performance of the classification models is evaluated in terms of accuracy, sensitivity, specificity, FPR and F1 score. The collection of image samples according to swap category is illustrated in Table 1.

Table 1 Image samples according to Swap of LCC
Full size table

The accuracy score of all six classifier results in a range between 84 and 100%. Therefore, to evaluate the performance of classifier statistically, WILCOXON signed-rank test is carried out. The descriptive statistics of the performance measures (%) are presented in Table 2.

Table 2 Descriptive Statistics of the performance measures (%)
Full size table

The descriptive statistics show that the performance measures: accuracy, sensitivity, specificity and F1 score of ResNet-50+SVM is highest as compared to other classifiers. The FPR of ResNet-50+SVM is lowest compared to other classifiers. Again, for more clarity, we have applied the Wilcoxon signed-rank test. The Wilcoxon signed-rank test is illustrated in Table 3.

Table 3 Individual hypothesis test results of performance measures indicating the superiority (+), inferiority (−) or equivalency (≈) of ResNet-50+SVM with respect to other five classifiers
Full size table

Table 3 shows superiority, inferiority and equivalency of other classifiers with respect to ResNet-50+SVM by using + , −, ≈ respectively on 100 independent simulations. It can be observed that the ResNet-50+SVM classification method provides statistically better performance than the other five classification methods. Hence, Resnet-50+SVM result better classification for prediction of Nitrogen deficiency with accuracy, sensitivity, specificity, FPR and F1 score are 99.84%, 99.84%, 99.94%, 0.051% and 99.84% respectively. The confusion matrix of this classification model is given in Table 4.

Table 4 Confusion Matrix of Resnet50+SVM
Full size table

The performance score for each class of ResNet-50+SVM is illustrated in Table 5.

Table 5 The performance score for each class of ResNet50 model plus SVM
Full size table

In the end, 400 number of image samples with 100 number of each four class is used for testing purposes using ResNet-50+SVM and all are classified correctly.

In addition, the comparison of CNN classification models with other tradition image classification method is carried out. In image processing and machine learning approach, mostly bag-of-feature, colour feature with SVM HOG with SVM, GLCM with SVM and LBP with SVM are applied for image classification. The accuracy score of those approaches is given in Table 6.

Table 6 Accuracy (%) score of traditional image classification methods
Full size table

Figure 4 illustrates the accuracy score of all executed methods and models. As shown in Fig. 5, ResNet-50+SVM achieved the highest accuracy score, i.e. 99.84% and GoogleNet + SVM achieved the second-highest accuracy score, i.e. 99.4%. Among the traditional methods, the highest accuracy score of 91.6% was produced by GLCM+SVM.36.

Fig. 4
figure 4

Accuracy score of all executed methods and models

Full size image

Again bag-of-feature produced 80.53% of accuracy. Hence, conventional methods were found to have inferior levels of performance compared to the CNN models.

In this study, the performance of six CNN based classification models are evaluated, and found ResNet-50+SVM is statistically superior. Further, a comparative study of all CNN based classification models and traditional image classification methods is carried out. The results show that CNN based classification models achieved better performance compared to traditional methods. As per the literature, very less research paper has been published for prediction of mineral deficiency in rice crop, especially Nitrogen (N). The identification of Nitrogen (N), Phosphorous (P) and Potassium (K) deficiency in rice based on static scanning and hierarchical Identification method are reported (Chen et al. 2014). In this method, the hyperspectral technique is adapted for collection of samples in four growth stages. Hyperspectral imaging devices are costly and have less availability. This method identifies the deficiency of minerals (N, P, K) but not the levels of deficiency. And reported the highest accuracy of 90.77%. The diagnosis of mineral levels of rice based on colour texture analysis is described (Sanyal et al. 2007). Here, also deficiency of minerals is identified but not the levels. They claim 88.57% of accuracy but no clear evidence about the dataset. Among the minerals, Nitrogen (N) status of rice is closely related to photosynthetic rate and biomass production, and it is a sensitive indicator of changes in the crop. Nitrogen diagnosis based on dynamic characteristics of rice leaf image (Sun et al. 2018) is described. Here, the Nitrogen (N) status is identified with the application of Nitrogen supplement in 3 days and 6 days interval. The remarkable changes in leaf etiolation area and degree of etiolation are taken into account with the application of Nitrogen (N) supplements. The samples are collected using scanning technique and reported optimal accuracy (training accuracy 84.1%, validation accuracy 72.7%). Hence, it is clear that no research has been conducted using on-field images and adapted conventional camera for collection of samples. Further, the proposed methods are not cross-checked with any scientific mechanisms. In this study, the experimentation is carried using on-field images, cross-checked by LCC and found 99.84% of accuracy. Besides, the proposed method is trained with a large dataset based on the CNN network and helpful for developing a smart mobile device application that can identify the Nitrogen (N) deficiency status.

4 Conclusion and future work

In this study, we present a method for N deficiency prediction of rice crop based on convolutional neural network (CNN). Because till now no dataset is available for this particular research, our first contribution to establish dataset, matching with LCC swap. We hope that this dataset will be useful for other researchers in this area. In addition, we have conducted comparative experimentation of six modified pre-trained deep learning networks and analyse the results. The “Wilcoxon signed-rank test” indicates that the ResNet-50+SVM is the best classification method for prediction of N deficiency of rice crop compare to other classification methods such as AlexNet+SVM, GoogleNet+SVM, ResNet-18+SVM, VGGNet-16+SVM and VGGNet-19+SVM. The future work may include construction of classification model for prediction of nitrogen deficiency with increasing data set and performance. This work can be extended by developing a smart mobile device application that can identify the Nitrogen (N) status of rice crop, which could be of great benefit to users with little to no knowledge of the rice plants that they are cultivating. Although, the Nitrogen status prediction as per the LCC having five swaps is useful but, the exactness of Nitrogen status can be improved with a higher number of swaps. As per “California rice news” an updated LCC is designed having eight number of swaps but, not available for public purpose till date. Hence, this research may be upgraded with updated LCC with its availability.