An adaptive plant leaf mobile informatics using RSSC

Abstract

An automated plant biometric system is now an important step in preserving nature’s biodiversity. This paper presents a novel Relative Sub-image Sparse Coefficient (RSSC) algorithm for mobile devices (MDs) representing plant leaves into a mathematically compact vector for its classification. The RSSC feature vector includes local Statistical Entropy Texture (SET) information inter-related to all the sub-images within a leaf. RSSC space is merged with Gray Level Co-occurrence Matrix (GLCM) feature to refine the outputs using best-Nearest Neighbor (best-NN), designed for MDs. The experiments were performed on three different types of leaf datasets: (i) Flavia, (ii) ICL and (iii) Diseased leaf datasets. The results proves our method more accurate and better compared to other existing plant identification systems. The proposed approach is also tolerant under shape distortion caused while capturing. The mobile machine learning system for leaf image informatics is deployed on Android devices which helps botanists, agriculturists and medical biologists to recognize ubiquitously the herbs and plant species anywhere-anytime.

1 Introduction

The growth of mobiles and ubiquitous technologies are serving in many application domains allowing information accessibility anywhere-anytime. The users of smartphones / phablets / tablets are increasing day-by-day and more than the population, as predicted by Cisco [6]. In mobile agriculture informatics, m-agriculture includes agricultural statistics, monitoring, commerce and botanical research which can aid farmers and botanists in their daily routines. As we know, the manual experimenting and monitoring the biodiversity is a tedious job involving human experts and is a time consuming practice - for both experts and non-experts. Therefore, an automated plant biometric system heightens such traditional identification process.

A human vision system (HVS) recognizes the multitude of an object in a real and digital world even with diversified changes in the viewpoints. Whereas, the artificial or computer vision (CV) fails to analyze them as accurately and timely as HVS. Thus, leaving several challenges for CV researchers including leveraging human intelligence, object recognition, invariant to orientation and occlusion.

With increasing global warming, it’s important to recognize and preserve the rare and the new hybrid plant species for better nature understanding by just looking at their leaves, fruits, flowers, stems, and roots. However, leaf information is sufficiently rich and is available throughout the year for species identification based on its shape, color, and/or texture [9]. In the past few years, various machine learning algorithms were proposed and used in field of species identification [9] such as LeafSnap [19], LeafView [24] and ApLeaf [43]. In general, an automated plant leaf image informatics involves: (i) leaf acquisition, (ii) pre-processing, (iii) feature extraction and (iv) leaf learning/matching, as listed in Fig. 1. In this paper, we are mainly focusing on novel feature extraction from plant leaves.

Fig. 1

The basic block diagram for plant informatics system

As we know, a colored leaf image $I_{leaf}^{3}$, with resolution (x×y), is a 3D matrix having 0 to 255 intensity values; representing three different colors: red, green and blue. Similarly, a sub-image is a 3D matrix selected using some pattern or relation from $I^{3}_{leaf}$, as shown in Fig. 2. These sub-images may be called as relative sub-images if they share some common information. In an ongoing research AgroMobile [30] and Agriculture-as-a-Service (AaaS) [34], proposes a cloud based mobile assistant for farmers (or for any non-experts) to monitor their crops in real-time by just using their MDs as their third eye. AaaS framework may also be used by the experts like botanists in their research to find the new species and explore the huge biodiversity of 4,20,000 species (reported by Govaerts [14]) within few seconds.

Fig. 2

3D colored leaf image, $I_{leaf}^{3}$ and its 3D sub-image matrices, $I_{leaf_{i}}^{3}$ of resolution $(x^{\prime }\times y^{\prime }\times 3)$

In this paper, three basic contributions are highlighted. Firstly, a novel local statistical entropy-based texture (SET) leaf matrix for mobile plant leaf informatics is proposed. Secondly, the plant leaves are uniquely represented using the proposed derivational relative sub-image sparse coefficient (RSSC) from SET matrix for classification of plant species, efficiently and robustly. It makes feature space more rich and compact, suitable for MDs and has high accuracy rate compared to state-of-the-art (see Section 5). Lastly, we have also designed an Android based mobile client-server architecture for plant leaf analysis accessing information related to it, anytime-anywhere.

Rest of the paper is organized as follows: Section 2 presents a brief literature background on the existing systems, Section 3 proposes a new RSSC sparse transform for plant image informatics, Section 4 shows the architecture for Mobile Vision (MV) system to retrieve plant species, and in Section 5 the experimental results and discussions are covered. Lastly, the paper concludes with few directions on future scope in Section 6.

2 Background and existing systems

In past, several machine learning methods have been proposed for identification of plant species using their fruits, flowers or leaves. Majority of these methods are plant leaf features based such as leaf shape, leaf margin, leaf vein and/or leaf texture [9]. These classification are briefly described in Table 1.

Table 1 Summary of state-of-the-art of plant leaf identification

Warren [38], in 1997, proposed an automated system to recognize plant genus Chrysanthemum with 30 species. In this, the three basic mathematical descriptors such as shape, color and size of leaf, flower and petal of a species were used for identification. White [24, 39, 40] used morphological descriptor for shape information extraction from a complete leaf image. They designed a MV Tablet-PC-based an electronic field guide to identify plant leaves. Since LeafSnap [19] is designed for iOS, Zhao et al. [43], in 2015, proposed ApLeaf – an Android based species identification.

On the other side, Mobile Computing (MC) can aid anytime-anywhere accessing agricultural related information such as production, monitoring, banking policy, m-agriculture commerce, and so on [30, 34]. As a third eye to farmers, Mobile Cloud Computing (MC2) automatically monitors the crop field by remote experts. MC2 also manages the energy consumption of MD by supporting an off-line accessibility of plant and pathological information. Kim et al. [18] proposed a self-growing agriculture knowledge using cloud computing (CC) services assisting farmers to make smart decisions.

In plant identification, Bai et al. [4] used graph transductive learning approach on Swedish leaf dataset and achieved an accuracy of 93.8 % in 2010. Whereas, in 2011-12, ImageCLEFs plant identification task [12, 13] provided a testbed for plant leaf based automated plant species identification. Further, in 2013, Mouine et al. [25] proposed a multi-scale triangular shape representation method for plant species recognition. This triangle side lengths (TSL) gives an accuracy of 95.73 % on Swedish leaf dataset. While in the latest literature, Wang et al. [37] proposed MARCH description for mobile retrieval of leaf images.

There must be a generic low-cost automated plant leaf informatics system in every ones pocket which is easy to operate by anyone-anywhere-anytime. Hence, this paper aims to increase the reliability and accuracy of classification by reducing the complexity such that it is suitable for MDs. This proposed MV system, aims to various groups working with and for agricultural sectors such as botanists, agronomists, gardeners, foresters, land managers and farmers.

3 Proposed methodology

To automate the process of plant leaf identification in real-time, accurately and efficiently, we proposed SET texture based RSSC algorithm. The system is designed as a combination of several modules, viz image acquisition followed by extraction of RSSC features with a detailed discussion on the algorithm properties, matching and computational complexity in further sections. The complete flow graph of the approach is shown in Fig. 3 for an overview.

Fig. 3

Flow graph of proposed leaf image informatics

3.1 Image acquisition and pre-processing

In pre-processing three basic steps are involved: (i) transformation of device dependent RGB color space to device independent L*a*b* color space, (ii) noise removal and (iii) extraction of region of interest (ROI) with minimal computational power and memory consumption.

While capturing leaf image MD uses a device dependent RGB sensors and therefore for same leaf samples the color may vary from one device to another device. Hence, a device independent color space transform such as RGB to CIE L*a*b* color space is used [32], (1). In all cases, the color space volume is same.

$$ I_{leaf}^{3} (x,y) \overset{XYZ}{\longrightarrow} I_{leaf_{XYZ}}^{3} (x,y) \overset{{L*a*b*}}{\longrightarrow} I_{leaf_{L*a*b*}}^{3} (x,y) $$

(1)

Then an average filter is applied to remove the unwanted non-uniform pixel distribution of size (5×5) by using (2).

$$ I_{leaf_{L*a*b*}}^{3} (x,y) \overset{5 \times 5}{\longrightarrow} I_{leaf_{\eta}}^{3} (x,y) $$

(2)

Now, in order to have translation and scale invariant leaf features, $I_{leaf_{\eta }}^{3}$ is cropped and rescaled to a standard resolution (w×h), by using (3) and (4). It helps to distinctively identify and focus on ROI [32].

$$ I_{leaf_{\eta}}^{3} (x,y) \overset{crop}{\longrightarrow} I_{leaf_{cropped}}^{3} (x^{\prime},y^{\prime}) $$

(3)

$$ I_{leaf_{cropped}}^{3} (x^{\prime},y^{\prime}) \overset{rescaled,(w\times h)}{\longrightarrow} I_{leaf}^{3} (w,h) $$

(4)

where, (x ^′×y ^′) is the translation image size. For example, as in Fig. 4a, the leaf sample of resolution (1600×1200) is cropped and scaled to (w×h)=(460×460) and Fig. 4b, to focus on ROI.

Fig. 4

(a) The input leaf of resolution (1600×1200), and (b) cropped and rescaled (460×460) leaf image (ROI)

3.2 Relative sub-image sparse coefficient (RSSC)

RSSC features are extracted from a series of 3D sub-image matrices in such a way that each sub-image within the image has a relation with the other adjacent sub-images. The colored leaf, $I_{leaf}^{3} (w,h)$ is divided into (s ₁×s ₂) grid matrix each of size ((w/s ₁ + c)×(h/s ₂ + c)), lets say it as (x ^″×y ^″). Here, s _i denotes the grid size and c is the number of common overlapping pixels between the two sub-images. Note: c is a whole number and s _i is a natural number, i.e. $c \in \mathbb {R}$ and $s_{i} \in \mathbb {N}$.

s _i and c highly influence the computation cost because if they are increased the grid matrix tends to behave like $I_{leaf}^{3}$ and therefore increases the computation cost. Here, in this paper, we have considered a square grid matrix, that is s ₁ = s ₂ = s to study the behavioral analysis of c in species classification.

On the other hand, the criteria for selecting s _i depends on the ratio of w and h to s _i which must be a natural number $\mathbb {N}$. Depending upon s _i , in total for a colored image, there will be (s ²×3) sub-images and in case of gray image (L*) its only s ², (5).

$$\begin{array}{@{}rcl@{}} I_{leaf}^{3} (w,h) &=& \{I_{leaf_{L*}}^{2} (w,h) + I_{leaf_{a*}}^{2} (w,h) + I_{leaf_{b*}}^{2} (w,h)\} \\ &=& \{\{I_{leaf_{L*_{1}}}^{2} (x^{\prime\prime},y^{\prime\prime}),{\ldots} I_{leaf_{L*_{s^{2}}}}^{2} (x^{\prime\prime},y^{\prime\prime} )\} \\ &&~~~~ + \{I_{leaf_{a*_{1}}}^{2} (x^{\prime\prime},y^{\prime\prime}),{\ldots} I_{leaf_{a*_{s^{2}}}}^{2} (x^{\prime\prime},y^{\prime\prime})\} \\ &&~~~~ + \{I_{leaf_{b*_{1}}}^{2} (x^{\prime\prime},y^{\prime\prime}),{\ldots} I_{leaf_{b*_{s^{2}}}}^{2} (x^{\prime\prime},y^{\prime\prime})\} \} \end{array} $$

(5)

To clearly visualize and represent the local leaf patterns in $I_{leaf}^{3} (w,h)$, each and every sub-image, $I_{leaf_{k_{l}}}^{2} (x^{\prime \prime },y^{\prime \prime })$, denoted as nodes in Fig. 5, are inter-related to adjacent nodes by minimum of c ² pixels where k={L∗,a∗,b∗} and l={1,…s ²}. If c is non-zero, x ^″ = y ^″ and s=5 then the connected weighted graph is as shown in Fig. 5.

Fig. 5

Connected weighted graph for $I_{leaf_{k_{l}}}^{2} (x^{\prime \prime },y^{\prime \prime })$. Double rounded nodes are the corner sub-images in $I_{leaf}^{3} (w,h)$. For example, (1,5,21,25) for s=5

$I_{leaf_{l}}^{3}$ is then further processed to extract the local statistical entropy texture (SET) information of every sub-image. Entropy is a key feature of a matrix used to uniquely identify it. Whereas SET is a quantitative measure of randomness of intensities within the sub-image locally. So it is the suitable parameter to measure irregular leaf patterns with minimum energy consumption and is computed using (6).

$$\vspace*{12pt} epy_{k_{l}}={\sum}_{L=0}^{L-1} (-p_{k_{l}} (L)*\log_{2} p_{k_{l}} (L)) ~~~~ |~~~ \forall ~~~ l=\{1, {\ldots} s^{2}\} $$

(6)

where, $p_{k_{l}} (L)$ is the probability of l ^th sub-image in k ^th layer, defined by (7).

$$ p_{k_{l}} (L)=\frac{ngray_{k_{l}} (L)}{n} $$

(7)

Here, $ngray_{k_{l}} (L)$ is the number of pixels in $I_{leaf_{k_{l}}}^{2}$ with maximum intensity L for total number n=(x ^″×y ^″) pixels. Note that, the sum of $p_{k_{l}} (i)$ is always a unity, where 0<i<L and $ngray_{k_{l}}$ is calculated using (8).

$$ ngray_{k_{l}} (L) = \left\{ \begin{array}{rl} 1; & if {\sum}_{i}^{x^{\prime\prime}} {\sum}_{j}^{y^{\prime\prime}} (I_{leaf_{k_{l}}}^{2} (i,j)=L) \\ 0; & otherwise \end{array} \right. $$

(8)

The SET, $epy_{k_{l}}$ defines the information present in l ^th sub-image of k ^th layer by analyzing its gray level distribution. The higher the entropy, the lower the information, see Fig. 6b.

Fig. 6

$I_{leaf}^{3} (w,h)$: (a) segmented to 5² sub-image, and (b) SET matrix of $I_{leaf}^{3} (w, h)$ for L∗, a∗, and b∗

A series of these local SET characterizes the leaf and independently used to represent them. But in majority of times it happens that the other leaf species have few SET elements similar and so may misclassify. Therefore, to separate such SETs, a derivative relative sub-image sparse coefficients are computed. A simple relative approach given by (9), relates $epy_{k_{i}}$ with $epy_{k_{j}}$, such that the min-max ranges between −1 to +1.

$$ RSSC_{k_{\alpha}}= |\frac{(epy_{k_{i}}-epy_{k_{j}})}{(epy_{k_{i}}+epy_{k_{j}})} ~~|~~ ~~|~~ \forall ~(k~ \& ~l) ~but~i\neq j $$

(9)

where, α is the total number of RSSC features in a single color space, i,j∈l. Note that i is paired with every other j and i≠j. The only condition is that the denominators must not be zero, i.e. $|~epy_{k_{i}}+epy_{k_{j}} ~|\neq 0$, in any situation. The total $RSSC_{k_{\alpha }}$ features of a leaf for a single color layer is $C_{2}^{s^{2}}$, a combination formula, which is again multiplied by 3 for colored space. The majority of adjacent sub-images may have similar pixel intensity and the entropy texture possibly be same so (9) results in many zeros. This vector resembles a sparse vector and thus, the coefficients are named as sparse coefficients.

RSSC presents a local block-by-block SET feature vector representation of an image. The SET graph comparison of colored layers with gray leaf is shown in Fig. 7. It is seen that even grayscale vector is sufficient to represent leaf in feature space. But along with this RSSC, to refine the classification, there must be a global texture information too and so GLCM is fused to complete the vector.

Fig. 7

SET matrix distribution of $I_{leaf}^{3}$ projected in 1D space. (Layer 1 = L∗, Layer 2 = a∗ and Layer 3 = b∗)

3.3 Gray level co-occurrence matrix (GLCM)

In this subsection, $I_{leaf}^{3} (w,h)$ is processed to extract global GLCM texture features [15]. GLCM is a well-known second order derivative texture matrix [15]. It considers the reference pixel and the neighbor pixel relationship which is a user defined entity. GLCM offset is the reference window length used to find relationship between the pixels within that window. As we know GLCM is not rotation invariant and so in this paper a circular GLCM [5] is used.

The circular GLCM is applied on $I_{leaf_{gray}}^{2} (w,h)$ to get a global leaf texture, $GLCM_{leaf}^{2} (i,j)$ by using (10).

$$ GLCM_{leaf}^{2} (i,j) (\delta x, \delta y) = \!\!{\sum}_{x=1}^{w} {\sum}_{y=1}^{h} \left\{\begin{array}{cccccccc} 1, & I_{leaf_{gray}}^{2} (x,y)=i ~\&~ I_{leaf_{gray}}^{2} (x+ \delta x, y+ \delta y)= j \\ 0, & \text{otherwise} \end{array} \right. $$

(10)

where, 0≤i,j≤(L−1), and (δ x,δ y) is the offset, i.e., {[0 δ],[−δ δ],[−δ 0],[−δ −δ]}, as shown in Fig. 8. In L*a*b* color space, the L* component shows the intensity of an image and independently it acts as a gray image. Therefore, $I_{leaf_{gray}}^{2}$ is the L* component of L*a*b* space (11).

$$ I_{leaf}^{3} (w,h) \overset{L* ~ component}{\longrightarrow} I_{leaf_{gray}}^{2} (w,h) $$

(11)

Fig. 8

GLCM matrix calculation using four different directions

The four basic statistical properties of $GLCM_{leaf}^{2}$ matrix: energy or uniformity E, contrast C o n, homogeneity H, and correlation C r n, are calculated using Equations, as shown in Table 2.

Table 2 GLCM parameters used in this paper

In Table 2, (μ _i ,μ _j ) and (σ _i ,σ _j ) are the means and the standard deviations of the marginal probability of $I_{leaf_{gray}}^{2} (w,h)$, respectively. This is calculated for all four angles (0^∘,45^∘,90^∘,135^∘). Therefore, GLCM feature vector is a set of all 16 features computed at four different orientations, i.e., G L C M={E _a ,C o n _a ,H _a ,C r n _a } such that a={0^∘,45^∘,90^∘,135^∘}.

RSSC and GLCM are fused to form a single vector, F _{l e a f} ={G L C M + R S S C}. Note that, the GLCM features are normalized to a range [−1+1] before F _{l e a f} . Thus, the feature length of F _{l e a f} is $((4\times 4)+(C_{2}^{s^{2}} \times 3)) = (16 + (C_{2}^{s^{2}} \times 3)) = N$, which is a high dimensional feature vector and requires N dimension feature space to represent it. In such a high dimension space, feature vector faces the curse of dimensionality where it has a high chance to mis-classify a leaf with the other species. This vector, $F_{leaf} \in \mathbb {R}^{N}$ (12) is a type of sparse vector and so it may be possible to reduce the feature space dimension by reducing the zeros by using Multi Linear Discriminant (MLD).

$$ F_{leaf}=\left[\begin{array}{ccccccc} x_{11} & {\ldots} & x_{m1} \\ 0 & {\ldots} & x_{m2} \\ {\ldots} & {\ldots} & {\ldots} \\ x_{1i} & {\ldots} & 0 \\ {\ldots} & {\ldots} & {\ldots} \\ x_{1N} & {\ldots} & x_{mN} \end{array}\right] $$

(12)

Here, m is the total number of samples from each C class, having N attributes with zeros and non-zeros.

3.4 Multi linear discriminant (MLD)

The MLD is the commonly used dimension reduction method in pattern recognition [10]. It is a generalized form of Fishers LD for multiple classes which is used to extract the linear combination of features such that it clearly separates the objects. Unlike principal component analysis (PCA), MLD explicitly models the difference between the classes to clearly separate them in feature space. Basically, PCA seek directions that are efficient for presentation of the classes while MLD seeks for efficient discriminations of classes. MLD maximizes the ratio of between-class variance $\delta _{between}^{2}$ to within-class variance $\delta _{within}^{2}$ so that the adequate class separability is obtained. The separation between the classes is

$$ S= \max\frac{\delta_{between}^{\mathbf{C}}}{\delta_{within}^{\mathbf{C}}} $$

(13)

where, $\delta _{between}^{\mathbf {C}}$ and $\delta _{within}^{\mathbf {C}}$ are defined by (14) and (15), respectively.

$$ \delta_{between}^{\mathbf{C}} = \frac{1}{m^{\prime}} {\sum}_{i=1:\mathbf{C}} (\mu_{i}^{\prime}-\mu^{\prime}) (\mu_{i}^{\prime}-\mu^{\prime})^{T} $$

(14)

$$ \delta_{within}^{\mathbf{C}}={\sum}_{i=1:\mathbf{C}} \delta_{i}^{\mathbf{C}} $$

(15)

Here, m ^′ is the number of samples per class, $\mu _{i}^{\prime }$ is the mean of i ^th leaf sample, μ ^′ is mean of complete leaf data and $\delta _{i}^{\mathbf {C}}$ is the class dependent scatter variance matrix, given by (16):

$$ \delta_{i}^{\mathbf{C}} = {\sum}_{i=1:\mathbf{C}} {\sum}_{F_{leaf_{m^{\prime}}} \in i } (F_{leaf_{m^{\prime}}}-\mu_{i}^{\prime}) (F_{leaf_{m^{\prime}}}-\mu_{i}^{\prime})^{T} $$

(16)

Since F _{l e a f} is a sparse matrix, the computation cost required to compute MLD basis vector to represent leaf will be reduced. The Fishers basis vector is calculated using eigenvalue (λ) and eigenvector (ω), that is

$$ \omega = max | \frac{\omega^{T} \delta_{between}^{\mathbf{C}} \omega}{\omega^{T} \delta_{within}^{\mathbf{C}} \omega } | $$

(17)

The high dimensional RSSC feature vector can be now projected to a low dimensional Fishers hyperplain by using (18):

$$ [F_{MLD}]^{m \times (\mathbf{C}-1)} = [F_{leaf} ]^{m \times N} \omega^{N \times (\mathbf{C}-1)} $$

(18)

That is, the F _{M L D} will be minimized to a vector with optimal hyperplane having minimum zeros, similar to (19).

$$ F_{MLD} = \left[\begin{array}{ccccccc} x_{11} & {\ldots} & x_{1i} \\ x_{1j} & {\ldots} & x_{1N^{\prime}} \\ x_{m1} & {\ldots} & x_{mi} \\ x_{mj} & {\ldots} & 0 \end{array}\right] ~|~ N^{\prime}= (\mathbf{C}-1) $$

(19)

The maximum length of Fishers hyperspace to separate plant species is (C−1). This is because the maximum rank of $\delta _{between}^{\mathbf {C}}$ is (C−1), which can be increased by its recursive application.

Now after dimension reduction, F _{M L D} is used for classification via a well-known k-Nearest Neighbor (k-NN) classifier. Since k-NN is a simple, powerful, accurate and efficient method; easily implemented on a low processing device like mobile phone. The value k is set to one an so named best-NN. The experimental results and discussion are presented in detail in Section 5 after discussing the proposed architecture for mobile device.

4 Proposed architecture for mobile device

Today, the smart ubiquitous device possesses all required sensors for the proposed system: mobile machine learning for plant leaf image informatics. Mobile phones with camera, processor and Internet connection (not always compulsory) are sufficient for proposed species identification. Android is today’s major market capturing operating system (OS) and is open source. Therefore, we targeted to deploy the proposed system on Android.

Due to MD constraints, the majority of high computing processes are offloaded to a high-end processing unit - the Server. To offload the tasks, unfortunately a regular communication connectivity is required and in countries like India where communication channels are very poor and overloaded, the system may not be so feasible. Thus, to have a real-time efficient system, the algorithms must directly execute on device without depending up on a remote Server. Therefore, in our implementation, both the problems are solved by introducing an intelligent agent to adaptively decide whether to compute-on device or to offload the job.

The complete architecture of the proposed mobile plant leaf informatics system is shown in Fig. 9. As discussed previously (Section 3.1), MD is used to capture leaf image and pre-process, extract features, then classify and augment results on the screen. Here, in this architecture, there are three scenarios: (1) execute everything on MD, if it is capable, (2) computer pre-processing and SET extraction on MD and rest is processed on the remote Server, and (3) perform pre-processing on MD and transmit rest to the Server for further processing, as labeled in Fig. 9. There is an Agent module in pre-processing phase, designed to record the current health status of MD and decide the transmission. The decision is made by estimating the processing capacity, battery life and communication cost of mobile device at that instant [32].If the bandwidth is very less and requires more energy and time to transmit compared to computing on MD, then only the RSSC feature vector is transmitted for classification on the server.

Fig. 9

Architecture for proposed mobile system: 1. Completely on mobile device; 2. Pre-processing on mobile and rest on server; and 3. Feature extraction on mobile and classification on Server

On the other hand, whenever MD is online, an update module automatically updates the online database storage. This helps in getting better performance time-to-time and explore new plant species even if there is no connectivity. The description of various modules used in Fig. 9 are given below and is briefed in Table 3.

• 1. Pre-Processing: As the very first module of the proposed mobile client-server architecture, it captures and transforms image to device independent color space (as in Section 3.1). The L*a*b* color image is denoised and transformed to translation and scale invariant space (Section 3.1). All these tasks are carried on mobile itself.

  Table 3 Module description of the proposed mobile architecture

• 2. Agent: Then the Agent sitting on mobile device estimates the computation cost and transmission cost, decides whether further high-level computations can be carried on on-board or offloaded to the server [32]. For the estimation, Agent computes following parameters: battery strength, network strength, bandwidth, image size, and computation power of MD. This task is also performed on MD.

• 3. SET Matrix: This module of mobile client -server architecture computes SET matrix of pre-processed images for further RSSC features (Section 3.2). Depending upon the transmission cost it can be performed either on-board or on the server. If the bandwidth is small then mobile computes the SET values and transmits on the SET matrix else the pre-processed image is transmitted, as shown in Fig. 9.

• 4. RSSC Feature: The SET matrix is derived to RSSC for classification as discussed in Section 3.2. This task can also be performed on both the sides: client or server depending upon Agent. Since RSSC is local vector it is fussed with the global GLCM leaf features to increase the efficiency.

• 5. MLD & Classification: The RSSC vector is further reduced to MLD feature space and then classified using best-NN either on-board using online dataset or on server using offline dataset (Section 3.4).

• 6. Result Display: The result module simply transforms the information into the readable form such as species name and augmented on mobile screens. It helps user in understand nature more continently anywhere-anytime. This module is performed on client device and so is a light-weighted thread.

• 7. Update: The Update module updates the parameters of mobile-based plant leaf informatic system whenever it is connected to Internet so that the result is up to the users satisfaction. It also uploads and downloads the online dataset on local mobile machines from server. It fetches a connection with server and updates the dataset with the key instances. This task is performed on client side.

• 8. Online & Offline Database: In this phase, the online database, that is, the subset of the original plant leaf database is stored on MD to perform species identification in offline mode. Whereas, the offline database is the actual complete leaf database that performs the complete match and validates the results before any decision is made. The offline database is updated by the research term day-to-day and is also updated with exceptions received from users.

5 Experimental results and discussion

In this segment of paper, the experimental results and the performances are shown on three different types of datasets: two are publically available leaf datasets and the third is a diseased leaf dataset. The first dataset is ICL (Intelligent Computing Laboratory) Leaf dataset [17], second is Flavia Leave dataset [11, 41] and the third is a diseased leaf dataset collected from Indian Institute of Technology (IIT) Roorkee, India and Forest Research Institute (FIR) Dehradun, India campuses [32, 33], as seen in Fig. 10. The ICL leaf dataset includes 220 different plant species (Fig. 10a) whereas Flavia leaf dataset contains around 1905 leaf samples for 32 different plant species (Fig. 10b). The diseased leaf dataset [33] is a small dataset with four different species having five different diseases and are around 297 leaf samples, Fig. 10c.

Fig. 10

Sample images from: (a) ICL dataset [17] (b) Flavia dataset [11] and (c) Diseased leaf dataset [33] one per species used in this leaf dataset

5.1 Implementation

The proposed system is implemented using Android SDK and OpenCV on Eclipse Windows environment. The system is deployed on an original Micromax mobile device of configuration: 1GHz processor, 512MB internal RAM, and Android 4.1.x or above. We used different evaluation approach to compare the proposed method with the existing systems; discussed in Section 5.2.

5.2 Parameters

In this paper, for experiment, the parameters selected are: w = h=460, a square matrix, to acquire at least a standard of VGA quality image, the size of grid is five (s=5), bigger blocks to reduce the computational cost and memory requirement. The third and the most important parameter is c=10, i.e. at least 100 pixels are common with their adjacent sub-images, (x ^″=100 = y ^″) and therefore, the length of vector will be 916. Whereas in case of gray image, it will be 316, which is reduced further by using MLD.

5.3 Experiments

With these parameters, F _{M L D} is computed for a plant leaf to recognize it using best-NN. The accuracy of the proposed RSSC algorithm is better than other existing methods discussed in this paper and is fast and efficient. The state-of-the-art comparison of all different datasets are shown in Table 4. Here, the accuracy of algorithms like Riemannian Elastic Metric (REM), probability neural network (PNN), RSC, CT, Multi-scale Convexity Concavity (MCC), Triangle-Area Representation (TAR), Inner Distance (IDSC), Multiscale-ARCH-height (MARCH), FT, GT, Elliptic Fourier transform (EFT), curvature scale scope corner detection (CSSCD) method and Triangle Oriented Angle (TOA) are compared with RSSC.

Table 4 Accuracy comparison of various plant leaf recognition methods

From Table 4, the maximum accuracy achieved with Flavia dataset is 99.36 % and 92.95 % for ICL leaf dataset using our proposed RSSC approach. The accuracy rate tested with our diseased leaf dataset is 100 %. Even a single dimension feature space clearly separates the diseased leaves, Fig. 11a. This is due to the small size of dataset.

Fig. 11

Accuracy measure with varying MLD feature length. (a) Diseased leaf dataset, (b) Flavia leaf dataset, and (c) ICL leaf dataset

Further, experiments are carried out to find the minimum feature dimension of F _{M L D} required to represent a leaf image properly, Fig. 11. This approach again optimizes the classification computation cost and increases the accuracy rate, avoiding the curse of dimensionality. Figure 11a, shows an accuracy rate of diseased leaf dataset with varying F _{M L D} length followed by Flavia and ICL leaf datasets, Fig. 11b and c, respectively. The comparison is also made with other different local statistical textures: energy and sum, (20) and (21), respectively. It is seen that, in Fig. 11a, even single dimension is sufficient to separate the species. But in Flavia dataset, the vector of length 20 successfully represents the leaf for mobile vision identification and in case of ICL leaf dataset, its 31.

$$ egy_{k_{l}} = {\sum}_{i,j=0}^{L-1} (I_{leaf_{k_{l}}}^{2} (i,j))^{2} $$

(20)

$$ sum_{k_{l}} = {\sum}_{i,j=0}^{L-1} I_{leaf_{k_{l}}}^{2} (i,j)) $$

(21)

In shape recognition, Mouine et al. [25], proposed TOA and triangle side length and angle (TSLA) to represent the object shape uniquely, but it is a complex approach, see Table 6. Unlike TOA, RSSC is a simple and less complex to identify species/shape. Figure 12 shows the RSSC comparison with different color dimension local SET features for all three different datasets using different classifiers. The bar graph compares five different types of classifiers: Bayes - Nave Bayes, Functional - Sequential Minimal Optimization (SMO), Lazy - k-NN, Meta-classifier, and Tree - Random Forest and found that best-NN outperforms in all the cases. Whereas, Nave Bayes also results equally good compared to best-NN but due to MD limitations best-NN is chosen. It is clear that the local entropy texture of $I_{leaf}^{3}$ results better using RSSC representation. The computation cost is even reduced in case of grayscale image with marginal accuracy compensation.

Fig. 12

Accuracy bar graph for L*a*b* and gray image. (a) ICL dataset, (b)Flavia dataset and (c) Homemade dataset

Another comparison graph, Fig. 13, shows an error rate change with varying Flavia training dataset size for different vector length (N). The graph shows a relation between grayscale image, i.e. L* component of L*a*b* color space with L*a*b* color image. The graph compares entropy texture with energy and sum texture at 15, 25, 50 and 80 percent of training sets, $F_{MLD_{train}}$. Similarly, Fig. 14 shows the error rate change for ICL leaf dataset with feature length: 1, 5, 10, 20, 31 and 100.

Fig. 13

Flavia error graph for L*a*b* (square) and gray (triangle) image. (a–e) Length of F _{M L D} 1, 5, 10, 20 and 31 respectively

Fig. 14

ICL error graph for L*a*b* (square) and gray (triangle) image. (a–g) Length of F _{M L D} 1, 5, 10, 20, 31 and 100 respectively

The time complexity of computing F _{M L D} , in both L*a*b* and grayscale, is compared in Table 5. It is clear that the cost involved in RSSC texture feature calculated of grayscale leaf is almost three times smaller than L*a*b* image and thus highly suitable for MD computing.

Table 5 Feature extraction time complexity for L*a*b* and grayscale images

5.4 Performance on mobile

The research is also carried on optimizing the different parameters (w,h,s, and c) in terms of accuracy and complexity, see Table 6. The third column in Table 6 gives the execution time for SET-based RSSC feature extraction (in seconds) for Flavia leaf dataset. Here, the L* (grayscale) image is used for comparison. The snapshots of leaf image informatics is shown in Fig. 15 where user captures the plant leaf and partially processed on the device and rest of the computations are pushed to the computing server.

Table 6 Parameter comparison of RSSC feature of L* image

Fig. 15

Snapshots of proposed leaf informatics system: (a) Leaf samples in a plain background, (b) captured via MD, (c) partially processed and (d) offloads the computation to remote computing server

The detailed misclassification of each class of a dataset can easily be shown by using a confusion matrix, representing the confusing classes. A confusion matrix or the error (matching) matrix is the visualization of performance of any algorithm. A confusion matrix, $\mathfrak {C}_{matrix}$ of Flavia dataset is shown in Table 7. In $\mathfrak {C}_{matrix}$, the values in each column represents the prediction class and each row represents the actual class. Note, for the readability purpose, the scientific names of species in $\mathfrak {C}_{matrix}$ are represented by their serial numbers as used in [41]. The accuracy rate A, in this paper, is a ratio of total number of predictions that were correct to the total number of observations and is defined by (22).

$$ \mathsf{A}=\frac{total_{cor_{prdt}}}{total_{cor_{prdt}}+total_{incor_{prdt}}} $$

(22)

where, $total_{cor_{prdt}}$ and $total_{incor_{prdt}}$ are the count of correct and incorrect prediction in $\mathfrak {C}_{matrix}$, respectively.

Table 7 Confusion matrix, $\mathfrak {C}_{matrix}$ for Flavia leaf dataset

Wang and Gao [35], proposed and RST-mirror invariant hierarchical string cuts (HSC) algorithm for shape retrieval with a maximum accuracy of 89.40 %. The accuracy achieved by this approach is around 86.03 % for ICL dataset which is very less compared to our proposed algorithm, see Table 4.

5.5 Computational complexity

The final parameter to measure any algorithm is its computational complexity. Since the proposed model is executed on MD, the cost must be minimum. The computational complexity of the proposed system is a sum of two models: the cost involved in RSSC feature extraction and the cost involved in best-NN classification.

Suppose, the size of a normalized input leaf image is (w×h), divided into s ² sub-images, each of (x ^″×y ^″) then the complexity required to extract RSSC feature is O(s ² x ^″ y ^″) which can be rewritten as O(w h). If (w h)=N, then the complexity will be O(w h) = O(N). At the classification end, the time complexity required for best-NN is O(F _{M L D} m C). If (m×C)=M, that is, if the dataset size is M and F _{M L D} =20 then complexity reduces to O(M). Therefore, the total time complexity involved for our proposed system is (O(N) + O(M)) and if M is reduced to 25 % for MD the complexity is directly affected, without effecting accuracy much (Figs. 12 – 13). In Table 8, a complexity comparison of different algorithms used in leaf shape identification is compared.

Table 8 Comparison of computational complexity

The proposed leaf informatics system helps in identification of species and further provide information for plant leaf disease diagnosis [14, 33, 34]. Species identification reduces the search database for disease recognition and enhance the system performance.

6 Conclusion and future work

This paper presents a novel RSSC algorithm for mobile vision based plant leaf informatic system. The RSSC entropy texture feature describes uniquely the texture of a plant leaf captured via MD. F _{l e a f} is a combination of four orientation independent GLCM features and local SET information. Best-NN classifier designed for MD is used for species classification. To validate the proposed system, the experiments were carried out on three different leaf datasets: ICL, Flavia and Diseased leaf datasets. The algorithm is compared with various other methods and techniques in state-of-the-art and come up with a decision that the proposed algorithm is more robust and efficient in terms of accuracy and computational complexity.

The proposed system is designed for isolated leaf and so leaf segmentation in a complex background in a low vision is still an open challenge for future. Secondly, for real-time feasibility and scope, mobile augmented reality in agriculture can be a new direction. And thirdly, the RSSC can be tested on other texture datasets to study its robustness.