Introduction

According to the FAO (FAOSTAT 2012), there are 140 citrus-producing countries. Current annual worldwide citrus production is estimated at over 105 million tons, with more than half of this being oranges, which makes this fruit the number one crop in the world. There are two clearly differentiated markets in the citrus sector: the fresh citrus fruits market and the processed citrus products market, mainly orange juice. About a third of citrus fruit production goes for processing and more than 80 % of this is for the production of orange juice.

Growers want to grow better fruit free of pests and diseases at a lower cost, which can be achieved in many ways such as by introducing good mechanisation practices or precision agriculture techniques. On the other hand, consumers demand healthy fruit at a good price, free of defects and diseases, and with a pleasant taste (Campbell et al. 2004), which can be guaranteed with adequate monitoring in the field during growing and postharvest quality inspection. Quality standards of citrus for fresh consumption are mainly based on the absence of bruises and decay, as well as their having an adequate shape, colour and size (Codex Alimentarius, 2011). Maturity of citrus fruit is defined by parameters specified for each species concerning minimum juice content, minimum total soluble solids content (sugar contents), sugar-to-acid ratio and colouring. Size is determined by the maximum diameter of the equatorial section of the fruit.

Major damage to citrus is caused in the field by atmospheric phenomena or insects, during harvesting due to bad practices or postharvest due to infection penetrating through injuries or to the evolution of previous diseases. However, they can be easily handled in the packinghouse due to their thick peel, which facilitates some postharvest processing tasks. One of the first properties to be measured is the external colour. The skin colour of fruit oranges and mandarins at commercial stage is normally orange and uniform. However, early varieties of citrus in many countries usually meet legal maturity standards before the peel attains the characteristic varietal colour and therefore require degreening. Occasionally, later-maturing varieties may similarly require degreening. Early-season citrus fruit may reach an acceptable degree of internal maturity and sugar-to-acid ratio before the peel attains a marketable yellow or orange colour. The ethylene used in citrus degreening rooms will degrade the green-pigmented chloroplasts present in the peel. The automatic inspection of early fruit has to cope with the possibility of finding fruits with different levels of greens on their surface.

One standard way to measure colour is by means of the citrus colour index (CCI), defined as \( CCI=\frac{1000\times a}{L\times b} \) (Jiménez-Cuesta et al. 1981) with L, a, and b being the HunterLab colour space coordinates. A negative CCI value means dark green/green, values around zero mean fruit turning from green to orange, while high positive values mean mature fruit. This index is used to determine the harvesting date, but it also plays an important role in establishing the duration of the degreening treatment.

As is the case with other commodities, citrus fruits can present bruises or diseases caused in the field or during handling and processing in the packinghouse. However, not all of these defects are of the same economic importance. Some kinds of damage do not evolve, but others do, especially those related with fungal or bacterial infections. It is extremely important to detect them in the earlier stages; otherwise, they can spread infections to other fruits or they may be invisible during inspection but then appear during transport to the destination marketplace, thus causing the buyer to reject them. Types of damage that do not evolve or that only cause appearance-related problems include mechanical damage during growth or at harvest, wind scars, hail damage, oleocellosis, incorrect cultural practices (fertilisation, irrigation or pest control) that produce physiological damage such as phytotoxicity, damage caused by pests such as citrus thrips (Scirtothrips citri) or scale insects such as California red scale (Aonidiella aurantii, Mask), purple scale (Lepidosaphes beckii, Newman) or Glover scale (Lepidosaphes gloveri, Packard), which are small armoured insects that attach themselves to the fruit and are difficult to remove. Severe infestations of these pests cause damage similar to scars and lower the value of the fruit. When the fruit has reached its typical mature colour, they can be detected because the contrast between the sound peel and the defect is high, but in earlier stages of maturity, when the peel is still green or is turning to orange/yellow, they are more difficult to see. On the other hand, the most important damage that evolves and produces economic losses in the postharvest period is caused by fungi. In particular, Penicillium digitatum (green mould) and Penicillium italicum (blue mould) are the most deleterious fungi causing fruit decay, since a small set of infected fruits can contaminate the whole batch, especially during storage or transport, and they affect several cultivars around the world (Holmes and Eckert 1999; Palou 2014). Other important fungi causing the loss of all the fruit are Alternaria citri, which causes internal decay, and anthracnose caused by Colletotrichum gloeosporioides, which produces rind collapses. Citrus canker is a bacterial disease affecting all types of citrus that causes premature leaf and fruit drop. The main symptoms are lesions on the leaves, stems and fruit of citrus trees. While not harmful to humans, the disease affects the health and marketability of infected fruit.

The main drawback that appears when developing automated systems to control the quality of citrus fruits is precisely the high number of diverse types of defects, blemishes or diseases presenting very different external symptoms that cause economic losses to varying extents. Systems based only on colour information have severe limitations, since the colour of the defect in one fruit can be similar to the sound skin of other fruit, while the colour of the sound skins of both fruits are very different, as can be seen in the detailed views in Fig. 1. Hence, despite the large number of recent advances in computer vision, in-line inspection equipment and multispectral (MIS) or hyperspectral (HIS) imaging systems for inspecting fruits and vegetables (Cubero et al. 2011; Lorente et al. 2012), the challenge is still large, especially taking into account the fact that new and dangerous pests and diseases are still appearing. This work reviews the recent developments and applications to create new fast, automated and reliable inspection systems based on computer vision to monitor the production of citrus fruits from the trees to the packing lines.

Fig. 1
figure 1

Comparison of the colour of the sound skin of two oranges and the colour of a defective skin. The sound skins of the two oranges present different colours (top) while their defective and sound skins have very similar colours and textures (bottom)

Full size image

Inspection in the Field

Robotics for Citrus Harvesting

The first applications of machine vision in the field for citrus were installed in robots for harvesting purposes. Since a robotic system for citrus harvesting was developed in the mid-1980s (Harrell et al. 1985), more efforts have been made to improve robotic citrus harvesting. Moltó et al. (1992) utilised the differences in the reflectance spectrum of the fruit and the leaves to locate the fruits on the trees. In the same work, they studied the reflectance of the light on the fruit surface due to the spherical shape of the fruits when they are illuminated using a flash. The particular pattern of the reflectance was used to locate the fruits. In the early 2000s, Burks et al. (2003) overviewed various aspects of successful robotic harvesting for citrus. They reported that field conditions, plant population and spacing, and plant shape and size were the most important factors for mechanical harvesting in the horticultural aspect. In the engineering aspect, factors to be considered were fruit detection and removal methods, and physical properties of fruit such as detachment force, damage potential, pressure resistance and rind penetration force. Design of an end-effector and robotic manipulation were also crucial to decrease fruit damage during harvesting and increase the efficiency of the harvesting operation. To identify necessary force requirements, Flood et al. (2006) conducted puncture studies of oranges using a universal testing machine and reported that the contact area was directly related to puncture force, which was a function of a punch diameter and the fruit’s radius of curvature. They identified a maximum value for an end-effector grasping force, which should not exceed the bursting or puncture limits. As part of the development of a robotic harvesting system, Subramanian et al. (2006) developed an autonomous guidance system for citrus grove navigation using machine vision and laser radar. The system was tested on straight and curved paths and achieved successful guidance within 2.5–2.8 cm while travelling at 3.1 m/s. Hannan et al. (2009) also developed a machine vision algorithm to identify oranges for robotic harvesting. The red chromaticity coefficient was efficient in segmenting oranges under varying illumination conditions. They achieved 90 % detection accuracy with a 4 % false-positive rate. More recently, Mehta and Burks (2014) developed a vision-based fruit depth estimation and robotic harvesting system using a computationally efficient method and in-depth visual servo control formulation.

Inspection of Fruit on the Trees

To inspect the crops in the field using computer vision is especially complex due to the changing environment, which requires the development of robust methods and algorithms capable of adapting to different natural conditions. Moreover, the equipment normally has to be carried by agricultural vehicles or, more recently, cameras are mounted on unmanned aerial vehicles (UAV). Among the problems that arise in citrus crops, a new dangerous pest known as citrus Huanglongbing (HLB) has recently appeared, this being one of the most destructive diseases in citrus worldwide. Currently, there is no treatment for this disease and efforts are being paid to the early detection of the symptoms in the trees using machine vision. Li et al. (2012b) acquired airborne spectral images of citrus groves in 2007 and 2010, developed various HLB detection algorithms and reported correct detection accuracies of 29–95 % depending on ground truth accuracies, image spatial resolution and methods used. Further, Li et al. (2014) developed a novel algorithm, called extended spectral angle mapping or ESAM, compared its performance with other commonly used methods and reported a better detection accuracy of 86 % with ESAM. Garcia-Ruiz et al. (2013) compared images taken from an aircraft with those acquired by a UAV to detect HLB, and found that UAV-based images produced better detection accuracies with lower rates of false negatives. Li et al. (2015) investigated the feasibility of utilising multispectral images captured by the WorldView-2 satellite with a 2-m spatial resolution to detect HLB-infected canopies, and compared their performance with aerial images. They reported that the satellite image produced the highest accuracy, although the selection of the spectral library would be the most important factor in achieving accurate detection.

Other studies were conducted to develop handheld detection devices. Pourreza et al. (2015a) developed a handheld HLB detection system using starch accumulation in HLB symptomatic leaves, and achieved accuracies of 95.5 and 98.5 % in lab and field tests, respectively. Furthermore, Pourreza et al. (2015b) reported that the vision sensor system developed was able to separate HLB infection from zinc-deficient leaves, which looked similar to HLB symptoms, and that the separability of infected leaves was higher after the leaves were ground than in unground leaves. In a separate study related to HLB, Choi et al. (2015) developed a machine vision system to detect fruit that had been dropped on the ground due to HLB, which caused tremendous premature fruit drop before harvesting. The system developed achieved 83–88 % detection accuracies depending on the different methods used. A geo-referenced map of dropped fruit was created to help further identify causes of fruit drop.

On the other hand, one of the principal aims of detecting fruit on citrus trees by computer vision is yield prediction, the main problem being the similarity of the colour of the immature fruit and leaves. Various methods have been developed to detect immature green and mature citrus fruit for yield mapping purposes. In the early 2000s, Annamalai and Lee (2003) developed an image processing algorithm to detect citrus fruits in an image using hue and saturation thresholds of citrus fruit, leaves and background classes. They reported an R 2 of 0.76 between the number of fruit obtained by the machine vision algorithm and the number counted manually. The total processing time for an image was 283 ms with a 750-MHz Pentium processor, excluding image acquisition. Further, Annamalai and Lee (2004) investigated spectral signatures of immature green citrus fruit and leaves for the purpose of developing spectral-based fruit identification and an early yield mapping system. Diffuse reflectance of fruit and leaf samples was measured in the range of 400–2500 nm, and two important wavelengths, 815 and 1190 nm, were identified as shown in Fig. 2. A ratio of these two wavelengths was used to distinguish immature green fruit from leaves.

Fig. 2
figure 2

Reflectance spectra of immature green citrus and green leaves (adapted from Annamalai and Lee 2004)

Full size image

Employing an outdoor HIS, Ye et al. (2008) utilised the two-band vegetation index (TBVI) to develop yield prediction models, canopy size and both of them together. The model with both TBVI and canopy size produced a better yield prediction accuracy of 76 %. In contrast, Okamoto and Lee (2009) developed ground-based detection algorithms to identify green immature citrus for three different varieties using the range of 369–1042 nm. A linear discriminant analysis for pixels was used to identify fruit objects, and spatial image processing steps were used to detect green citrus. They reported detection accuracies of 70–85 % depending on citrus varieties and 80–89 % accuracy for the fruit in the foreground. For a combined set of three varieties, a 75.8 % success rate was reported for the validation set images. Young leaves were the main obstacle for correction identification, since they were spectrally very similar to green citrus.

An alternative to HIS was presented by Bulanon et al. (2009) who used a combination of thermal and visible imaging to detect immature fruits in citrus orchards. They adopted two image fusion methods, Laplacian pyramid transform and fuzzy logic, discovering that image fusion performed better than any single technology.

Since HIS or thermal systems are usually expensive, attempts were made to utilise a typical consumer-grade digital camera to detect immature green citrus fruit. Kurtulmus et al. (2011) developed a machine vision algorithm to distinguish immature green citrus fruit from other objects in natural outdoor colour digital images using colour, circular Gabor texture and a novel ‘eigenfruit’ method, a 75.3 % rate of correct identification of immature green citrus being obtained for a validation set. Example results are shown in Fig. 3.

Fig. 3
figure 3

Immature green citrus recognition results: a results with the algorithm developed, b detection of occluded fruit with a false positive and c detection for a smaller partially occluded fruit (adapted from Kurtulmus et al. 2011)

Full size image

In Bansal et al. (2013), a percent leakage of fast Fourier transform (FFT) was used to distinguish fruit from other objects in natural outdoor colour digital images, and 82 % of fruit was correctly identified from a set of 60 validation images. They suggested that the FFT leakage technique would be useful in detecting other green immature fruit. Sengupta and Lee (2014) also developed a method for identifying immature green citrus from digital colour images, and reported a detection accuracy of 80.4 % in a validation set. A Hough circle detection and texture classification by a support vector machine (SVM) were used to find all potential citrus fruits. False positives were then removed using Canny edge detection and keypoints obtained by a scale invariant feature transform (SIFT). They reported that varying illumination, partial occlusion in outdoor images and detecting immature green citrus in the presence of green leaves were major problems and that an additional method would be needed to remove more false positives. Li et al. (2016) implemented a template matching method using a fast normalised cross correlation (FNCC) to identify immature green citrus in natural outdoor colour images, and reported an accuracy of 84 %.

Inspection of Fruit in the Field Using Mobile Platforms

Electronic sorters for quality and safety inspection of fruits based on machine vision have been developed to be used mostly in packinghouses, these being controlled environments with powerful processing units and illumination systems, thus enabling the development of complex image analysis algorithms capable of inspecting the products at a very high speed, but also with high needs in terms of electrical energy. However, the demands of markets and consumers as well as social concerns about good practices, sustainability and traceability all make it necessary to guarantee the quality of agricultural products from the earliest stages of crop production. The inspection and early detection of low-quality fruits for the fresh fruit market in the field as they are harvested would make it possible to prevent them from reaching the packinghouse, thus saving in further transport or treatments of this bad fruit or preventing the possibility of spreading diseases to other healthy fruit during transportation or storage. The information about the quality of all the production related with the exact location of harvesting of each individual fruit can be the basis on which to create maps of the crop associated with the quality or other important parameters of the fruits. Moreover, the pre-sorting of the fruit in the field makes it possible to anticipate the management of the fruit when it comes into the packinghouse. Kohno et al. (2011) developed a computer vision system in combination with an NIR spectrometer that, when mounted on a mobile platform, allowed both the external (colour and diameter) and the internal (sugar content and acidity) qualities of citrus fruits to be assessed during harvesting. However, the large amount of time needed by the system to inspect a single fruit (between 12 and 20 s) made it impractical for a commercial situation. Cubero et al. (2014a) presented a more efficient computer vision system to inspect citrus fruits externally at harvest time mounted on an agricultural vehicle. The system had a smart camera capable of capturing the images and optimised the algorithms so as to be able to perform the image processing on the slow microprocessor of the camera. An illumination system based on pulsed LEDs ensured low energy consumption. The system was capable of inspecting and sorting the fruit into three categories using the standard citrus colour index (Vidal et al. 2013) and size properties at a rate of 8 fruits/s.

These systems of fruit inspection in the field are even more necessary when the fruit is harvested using mechanical methods, such as canopy shakers (Torregrosa et al. 2014), because these devices are not selective and harvest all the fruit regardless of its condition (Chinchuluun et al. 2009). For these cases, it is also important to measure the amount of fruit harvested and obtain statistics about some quality like size or estimated weight. Shin et al. (2012a, b) developed a computer vision system that inspected the fruit on a conveyor belt while it was transferred from a catch harvester. The system could count the number of harvested fruits and the size distribution but in the case of need cannot sort the fruit individually.

Table 1 summarises the different works carried out for the application of computer vision in the citrus inspection in the field ordered by different topics chronologically.

Table 1 Chronological achievements in citrus crop monitoring using computer vision
Full size table

Postharvest Inspection of Citrus Fruits Using Visible Imaging

After arriving at the packinghouse after harvesting, a number of postharvest handling practices are applied to the fruit, especially when it is destined for consumption as fresh fruit. The first inspection is usually performed on a roller elevator that transports the fruit to the washing machine. At this point, foreign materials as well as rotten, cut and very-bad-quality fruits are removed. Fruits are then washed and fungicide may also be applied if needed. Later, different processes that include rinsing, waxing and being temporarily held in cold storage are applied. However, these operations may be modified under certain conditions, for instance when the fruit is harvested while the skin is still green (but internally it is fully ripe) and a process of degreening is therefore needed. The degreening process depends on the colour at harvest and thus a sorting by colour is required (Ladaniya, 2010). Figure 4 shows oranges of different colours ranging from green to orange.

Fig. 4
figure 4

Colour classification of citrus fruits

Full size image

Later, when the fruit is going to be shipped to the market, the whole production is inspected and sorted into commercial categories to ensure that only fruit accomplishing the minimum requirements of quality reaches the consumer. Moreover, to maximise profits, they have to be classified in commercial categories according to several parameters that have a certain influence on the price the consumer is willing to pay for the product such as size or colour. In the past and in less-automated facilities, external inspection was carried out visually by trained workers and calibration was performed using a drop-roll sizer to sort the fruit, usually into increasing sizes, according to the equatorial diameter. Currently, to deal with the inspection of the huge amount of fruit handled by packinghouses, they rely on automated inspection systems.

Principles of Electronic Sorters

Modern citrus packinghouses are equipped with electronic sorters that can capture and analyse images of fruits at a very high speed, by measuring external properties such as colour, size or the presence of damage or defects (Fig. 5). These sorters use machine vision, whose application in the postharvest processing of fruits and vegetables has increased considerably in recent years (Blasco et al. 2016). Nevertheless, automated inspection of agricultural produce shows certain particularities and problems that are not present in industrial pieces due to their biological nature. While manufactured products often present similar colours, shapes, sizes and other external features, fruit such as citrus fruits may show very different characteristics from one item to another. Moreover, it is essential that the presence of stem ends, leaves, dirt or any extraneous material be identified and not confused with true skin defects, which is particularly complex in the case of round fruits because all the surface has to be analysed. On the other hand, markets demand very fast image processing and, for this reason, a trade-off between speed and accuracy must be found. Examples of databases of images of citrus fruits showing different defects can be found at http:\www.cofilab.com\downloads.

Fig. 5
figure 5

Commercial electronic sorter for citrus fruits working in a packinghouse

Full size image

In all electronic fruit sorters that make use of machine vision, the performance depends not only on the algorithms developed to extract information from the samples but also on the proper design and maintenance of key elements such as the illuminating system or the camera. Hence, it is important to take this into account to design algorithms that are robust against failures or defects of these elements (Ottavian et al. 2013, 2014). Illumination is one of the conditioning factors that most seriously affect performance. Indeed one of the main problems that arise when illuminating citrus fruit is that bright spots appear on the peel due to the very reflective surface of the fruit, especially after the waxing and polishing operations. These bright spots can affect the estimation of the colour of the fruit and sometimes mask defects or blemishes that are important to detect. The problem grows because of the spherical shape of the fruit. A common way to avoid specular reflections when illuminating any object is to use a 45°/0° geometry, where the sample is illuminated at 45° off the normal to the sample and the reflected light is detected at 0° (perpendicular to the sample). This method works well for flat objects, but in the case of spherical samples like citrus fruit, most of the surface is curved and hence not perpendicular to the camera, thus causing glares at particular angles between the surface and the camera. For these cases, it is recommendable to use very diffuse light or polarising filters (Morgan and Stockford, 2003). However, in some exceptional cases, a particular reflection of the light is needed such as, for instance, to measure the texture or roughness of the surface, which was investigated by Jafary et al. (2014) as indirect estimation of the thickness of the peel. This work could lead to detect the puffiness in mandarins which is characterised by separation of the peel from the pulp on the tree or in storage, causing the rejection of the fruit. This disorder is hardly detectable by typical image processing algorithms focused on estimating colour or detecting skin damages and needs the development of particular algorithms based on texture. Unfortunately, this work did not include such aim but can serve as further reference.

An additional problem comes from the fact that the top of the fruit will always appear brighter than the edges, with a gradient from the top to the edges that depends on the curvature of the fruit. Two approaches have been envisaged to correct this effect. On the one hand, the amount of light reflected by the fruit in each area can be corrected. Gómez-Sanchis et al. (2008) created an elevation model from the circumference of the oranges, assuming them to be perfectly spherical with a Lambertian surface. For the case of mandarins, which are more flattened, they applied an adjustment to the total height depending on the cultivar. In this way, the reflection of each pixel was corrected according to the value given by the model, thereby achieving a similar reflection for the pixels at the top and near the edge. A similar strategy was followed by Li et al. (2013) under the premise that the illumination component of the oranges is generally characterised by smooth spatial variations, while the reflectance component tends to vary abruptly. On the other hand, the edges can be removed simply from the analysis using morphological operations, as did Niphadkar et al. (2013b). Probably this last approach is the lowest from the accuracy of the analysis point of view but probably the fastest, and it is important to take into account that industry demands fast methods capable of inspecting tons of fruit per hour; therefore, a commitment between computing speed and accuracy is needed.

Colour cameras are still the most widely used in the quality inspection of fruits because the external properties of the peel are the ones that are taken into consideration most in the quality standards. Traditionally, quality inspection has been carried out in the packinghouses visually by trained workers and, hence, methods based on colour cameras trying to mimic their capabilities have been largely developed. A drawback of cameras is that they only capture the partial view of the fruit that faces the camera, missing the information on the other side. To inspect as much of the surface as possible, it is necessary to use several cameras with different angles and/or to rotate the fruit while passing it under the camera to acquire images with different views of the fruit. Two imaging techniques are normally used to achieve this, and both are based on acquiring several images of the fruit while rotating it under the camera and hence showing different parts of the surface. The difference is the way in which the different images captured of the same fruit are processed. The first type of system processes the images separately; each image is processed in the moment it is captured, and finally, when the fruit is completely inspected, a decision is made based on the results of the partial images. This was the strategy followed by Vidal et al. (2013) and Cubero et al. (2014a). The second type extracts the central strip from every image captured. When the last image is captured, all these strips are joined into a single image to create a map of the fruit’s surface. The main advantage of this method is that only one image is processed, thus saving time. The main disadvantage is that the shape features of the fruit are missed, and hence, size cannot be estimated. This strategy was followed by Aleixos et al. (2002). Both methods are currently implemented in modern electronic citrus sorters based on computer vision.

Estimation of External Properties of the Fruit

The sound peel of commercial oranges and mandarins is expected to be orange in colour, and defects and stains normally (but not always) have different colours. In the case of postharvest inspection of citrus fruits based on computer vision, the main goals are to estimate the size and colour and detect the presence of defects to sort the fruit on the quality determined by these properties. The first step in the image processing is the image segmentation that is the process of dividing an image into meaningful structures (Basavaprasad and Ravi 2014). The goal of the image segmentation could be, for instance, to separate the fruit from other fruits in the image or from the background and to divide each fruit into different areas: stem, defects, discolourations, etc. By segmenting the fruit from the background, the size can be estimated. In most packinghouses, fruit is graded by size with the aim of maximising profits and offering the consumer packages with homogeneous products. Size is usually estimated as the area or diameter of the circle-shaped projection of the fruit in the images. However, Omid et al. (2010) estimated the volume of the citrus using two cameras and computing the volume by dividing the fruit into a series of discs but the advantage of this method with respect to the size estimation using 2D projections is not demonstrated since volume is not a required property of the industry. The estimation of the size by a computer vision system can be altered by the presence of long stems that may confuse the algorithms when measuring the diameter or the length of the fruit. Cubero et al. (2014b) developed a robust method to detect stalks in different fruits, including oranges and mandarins, which led to more exact estimations.

Segmentation methods based only on colour information have been largely investigated (Iqbal et al. 2016), but most of them require a previous training step because the large variability in colours makes it difficult to find absolute systems capable of dealing with all this variability. A typical supervised method is to define particular classes like background, stem, sound peel, different types of defects, etc., and to assign them to particular colours. The application FoodColorInspector, which has been developed especially for the segmentation of colour images and can be downloaded for free from (http://www.cofilab.com/downloads), allows this to be performed by selecting pixels belonging to the different objects in the image (training set) and assigning them to different classes, while trying to cover all the variability of the colours and properties of the samples. The red, green and blue (RGB) coordinates of the selected pixels (X) and the class they belong to (Y) are used as inputs to create a model based on a Bayesian approach of the discriminant analysis in order to later classify any unseen pixel as belonging to one of the predefined classes. Once the model has been created, it is possible to pre-classify all possible combinations of RGB and hence any possible colour in the image and store this classification in a table to be used during in-line real-time applications. However, the potential of supervised segmentation methods based only on colour information is limited because, due to the large variability that can be found in the fruits, such systems need frequent re-training. To solve this, Blasco et al. (2007a) exploited the contrast between sound peel and defects by means of an unsupervised segmentation method based on region growing, which, on the other hand, consumed a large amount of processing time. Another technique was used by López-García et al. (2010), who employed a multivariate image analysis approach consisting in introducing textural information about each pixel and its 3 × 3 neighbourhood and later using all the variables in a model based on principal component analysis (PCA) (Jackson, 1991). Several segmentation techniques applied on citrus fruit images were compared by Vijayarekha (2012a) to detect defects such as iterative intensity enhancement, contrast stretching, comparison against a reference colour of sound peel and Euclidean distance between the colour of different regions of interest. They succeeded with all methods but using clearly differentiated defects. The fastest method was the one based on the Euclidean distance among colours, but it still took more than 1 s to segment the image, which makes the method incapable of being implemented on an in-line quality inspection system. The work was expanded to identify the defects after segmentation (Vijayarekha, 2012b). For each defect found, several features were analysed using linear discriminant analysis (LDA) and artificial neural networks (ANN), the results showing that LDA performed better in distinguishing between pitting, splitting and stem end. However, the methods were tested on a reduced number of samples with very evident defects easy to discriminate.

A serious problem arises when the appearance of other features, like the stem, is similar to a stain in the peel. Blasco et al. (2007b) tested different colour spaces to discriminate among 11 types of defects in the citrus peel and the stem. Using the hue, saturation, intensity (HSI) colour space, success ranged from 43 % in detecting scale infestation to 100 % in the case of chilling and stem-end injuries. Later, they introduced morphological information about the defects into the analysis, thereby increasing the identification of all defects to a global average rate of success of 85 % (Blasco et al. 2009). Sometimes it is important to differentiate those defects that only affect the appearance of the fruit from the ones that cause fruit decay and can spread to other fruits, hence causing great losses. Kim et al. (2009) introduced textural features in colour images to distinguish between some serious damage like canker and other cosmetic defects like wind scar, among others, thus achieving a rate of good classification of more than 96 %. López et al. (2011) used colour and texture features extracted in the RGB and HSI colour spaces to discriminate among seven common defects of citrus fruits. Best results were achieved in the HSI colour space with success rates ranging from 63 % in the case of scale infestation to 100 % in the case of stem-end breakdown. A different strategy was followed by Li et al. (2013), who employed RGB image ratios to discriminate the stem from different defects in oranges, achieving a good score on classification of defects in images including and excluding the stem. However, the reduction in the prices and the popularisation of MIS and HIS have increased research on identifying some particularly dangerous defects using this non-standard computer vision technology.

Inspection of Defects of Special Importance Using Non-standard Computer Vision

The use of non-standard computer vision systems allows the creation of systems that are specific to detect certain important defects. The importance of these defects is due mostly to the economic losses they cause, the ease with which they can spread to other fruits in postharvest handling or because of the impossibility of detecting them using standard colour vision while they can often be observed using special equipment like magnetic resonance imaging (Hernández-Sánchez et al. 2006; Barreiro et al. 2008) or X-ray (van Dael et al. 2016), or at certain specific wavelengths of the spectrum (Gómez-Sanchis et al. 2014). The studies conducted by Gaffney (1973) and Magwaza et al. (2012) determined some visible and near-infrared wavelengths at which more contrast between the peel and major defects could be found. Recent advances in technology have made new equipment for acquiring hyperspectral images accessible to researchers that combines the advantages of imaging and spectroscopy, resulting in many successful applications in the food industry for quality and safety evaluation and inspection (Moltó et al. 2010; Lorente et al. 2012). HIS are characterised by offering the possibility of acquiring a large number of images at different consecutive wavebands. Basically, research on citrus fruits applying MIS or HIS has been conducted to detect two different types of diseases: decay lesions caused by fungi, mainly by P. digitatum (green mould), and citrus canker, but these technologies have been used for other types of defects of certain importance for the citrus industry.

Many works are focused on the selection of the best wavelengths for detecting these major diseases. However, there is not an agreement on them. Actually, it does not really matter what particular wavelengths are selected for better detection of one or another defect because in most cases they are different and depend the most on the methods used for feature selection than on the defects themselves. Therefore, particular wavelengths found by the different works are not cited in this review.

Detection of Decay Lesions

Early detection of fungal infections still remains one of the major issues in the postharvest processing of citrus fruits because of the great economic losses that this disease causes to the industry around the world (Zhu et al. 2013). These fungi reproduce very rapidly and their spores can be quickly disseminated to sound fruits (Palou et al. 2011). The symptoms these lesions cause in the fruit are difficult for workers to detect because the appearance of the infected skin is similar to that of the sound skin. In consequence, there is a need to develop new inspection methods to detect the infected fruit in the packinghouses as soon as possible.

Because of the difficulties involved in detecting visually decayed fruit, the technique adopted in packing lines is the use of fluorescence induced by ultraviolet (UV) light. Fluorescence is produced when a molecule is excited with high-energy (short wavelength) light, and the subsequent instantaneous relaxation emits lower-energy (longer wavelength) light. The use of such UV light allows certain types of external damage that are not visible using visible light to be observed. In the application to detect decay lesions in citrus, fruit is illuminated with UV sources (around 380 nm) that induce visible fluorescence (around 550 nm). However, this procedure has a high risk of human error and it is potentially harmful for operators (Lopes et al. 2010). Therefore, various machine vision technologies are being incorporated to automatically detect decay in citrus fruit, such as the use of UV-induced fluorescence or the introduction of HIS equipment. Blanc et al. (2010) patented a commercial sorter for decay detection in citrus fruits using UV illumination (Maf Industries 2016). Ogawa et al. (2011) presented a system to detect decay lesions in citrus using UV images. However, Momin et al. (2012) demonstrated that different cultivars have different excitation wavelengths to produce UV-induced fluorescence in the rotten areas and therefore this information should be taken into account in any work using fluorescence to detect fruit decay in postharvest. For this reason, it is of special importance to detect the compounds involved in the fluorescence process as it was studied by Kondo et al. (2009) and Momin et al. (2013a). Another problem is related to other defects that also present some degree of fluorescence and hence can confuse a potential automated system (Blasco et al. 2007b; Slaughter et al. 2008; Obenland et al. 2010). A way to differentiate among defects presenting fluorescence is by determining the pattern in which the fluorescence is emitted (Momin et al. 2013b), but the problem arises when a system has to detect different types of defects, that is, decay lesions that present fluorescence and mild defects that only affect the appearance of the fruit. On the other hand, this technique presents some problems, such as the fact that visible lighting could hide the effect of fluorescence and therefore this inspection has to be performed in a separate chamber or using the innovative technique presented by Kurita et al. (2009), which alternatively switched on UV and white pulsed LED in a very short period of time, thus allowing the inspection of almost the same scene with both types of illumination, and hence allowed both a fluorescent and a colour image to be captured. Figure 6 shows a hyperspectral image in the range 510–990 nm of an orange with a decay lesion. Figure 7 shows a hyperspectral image of the same fruit acquired using UV light to produce the fluorescence in the damaged area.

Fig. 6
figure 6

Hyperspectral image in the range 520–990 nm of an orange with a decay lesion with a resolution of 10 nm. The disease is difficult to detect in both the visible and NIR bands

Full size image
Fig. 7
figure 7

Hyperspectral image in the range 510–660 nm of an orange with a decay lesion (the same as in Fig. 6) with a resolution of 10 nm, illuminated with UV light

Full size image

HIS systems are also currently being investigated with the aim of detecting decay (Gómez-Sanchis et al., 2014). Gómez et al. (2007) and Gómez-Sanchis et al. (2008, 2012) developed machine learning techniques to distinguish fruit with decay lesions from sound fruit and fruit with other less-severe external defects. A common characteristic of these systems is their capability of capturing a huge amount of information which, on the other hand, is also a drawback because part of this information is redundant and unnecessary (Lorente et al. 2013a). Some works have been focused on reducing the amount of information which is also important due the high price of the necessary equipment. These techniques transform the data in the high-dimensional space into a lower-dimensional space that preserves the observed properties of the data. The spectral features are used as inputs of classification algorithms in order to increase the performance of the classifiers developed to discriminate between sound and decaying skin (Gómez-Sanchis et al. 2013; Lorente et al. 2013b).

Recently, laser light backscattering imaging (LBI) has also been studied as an alternative machine vision technique for detecting fruit quality (Adebayo et al. 2016). When a light beam interacts with a biological system like a fruit, reflectance, absorption and transmittance of such light take place (Birth, 1976). Particularly, light reflectance (scattering) appears with three different geometries: regular reflectance, which occurs when photons are reflected at the same angle as the incident light; external diffuse reflectance, which occurs at 45° to the incident light; and diffuse reflectance. In the latter case, light interacts with the internal components of the fruit and is then scattered backward to the exterior tissue surface, thus carrying information about the morphology and structures of the tissue in addition to the absorption properties. A system based on this technology was tested by Lorente et al. (2013c) using five lasers emitting light in different NIR wavelengths to capture backscattering images of sound and damaged skin areas of the fruit. Backscattering images had radial symmetry with respect to the light incident point, and their intensity decreases with increasing distance from the incident point. Hence, the images were reduced to one-dimensional profiles after radial averaging. The radial intensity of the backscattering profiles was calculated by obtaining the average value of all pixels within each circular ring with one pixel size. This work used the Gaussian-Lorentzian (GL) cross-product distribution function to describe backscattering profiles from backscattering images and was later enhanced in Lorente et al. (2015) to include a physical approach consisting in extracting optical properties of fruits by Farrell’s diffusion theory (Farrell et al. 1992). Despite the fact that these works have some interest because they try to develop new approaches to detect decay lesions avoiding the problems related with UV illumination, the practical implementation of LBI still needs research because the laser has to point exactly to the damage in order to be able to detect it.

Detection of Citrus Canker

Citrus canker is another severe disease affecting citrus that causes a significant socioeconomic impact in Florida, and has implications for national and international trade. It is characterised by erumpent lesions on fruit, foliage and young stems of most cultivars of citrus, and infected fruit are less valuable or entirely unmarketable (Gottwald et al. 2002); hence, automated methods to detect such a serious disease have been studied. All efforts have been focused on the development of spectral-based computer vision systems. Balasundaram et al. (2009) investigated reflectance properties of citrus canker among other types of defects and sound skin in the range 400 to 2500 nm. In the VIS/NIR range (400 to 1100 nm), they found up to seven wavelengths that could serve as the foundation for further developments based on computer vision.

Qin et al. (2008) used a portable imaging spectrograph to acquire hyperspectral images of red grapefruits affected by canker and other defects in the range 400 to 900 nm. The spectra of the different defects obtained from the images were analysed using PCA and showed a good rate of detection of the most severe diseases. This work was enhanced by Qin et al. (2009) by introducing spectral information divergence as a classification method to increase the performance up to 97.6 % success in canker lesion detection. Based on these works, Zhao et al. (2010) introduced the effect of the harvesting time and its influence in the detection of the damage. In Qin et al. (2011), the authors exploited the bands selected using PCA and correlation analysis to obtain a system capable of detecting the canker using ratios of two bands, which allows for the possibility of creating a fast detection system. Therefore, a system to detect canker lesions in-line was developed by Qin et al. (2012) using real-time algorithms. Moreover, PCA and band ratios were used by Li et al. (2010, 2012a) to select important bands in the detection of this disease among other common defects. However, to assess the practical application of such systems, it is important to know some limitations, such as the minimum detectable size limit for canker lesions (Niphadkar et al. 2013a) or the influence of the distortion and reflectance problems caused by the sphericity of the citrus fruits (Niphadkar et al. 2013b).

Detection of Other Skin Defects

HIS has also been used to detect other defects that commonly appear on the skin of the citrus fruits in packinghouses. Blasco et al. (2007b) discriminated among 11 types of external defects of oranges and mandarins using a multispectral system composed of a colour, a NIR and a UV camera. Using the different techniques separately, the results were lower than when combining them, since different defects could be detected better using some of the cameras than with the others. Accordingly, only damage by anthracnose and sooty mould were detected in NIR, only the stem-end injury was detected in UV, while thrips and wind scarring and decay lesions were the only defects that could be detected using UV-induced fluorescence. Li et al. (2011) used a HIS to distinguish among different types of defects, discrimination higher than 90 % being achieved for all defects using ratios between bands selected through PCA. A dangerous defect that causes the fruit to be rejected is the citrus black spot, as stated by Bulanon et al. (2013), who utilised hyperspectral imaging to determine four wavelengths capable of distinguishing citrus black spot among four other defects and good fruit with an overall accuracy of 92 % but a black spot discrimination rate of 100 %. The wavelengths were selected using band ratios and differences while the classification was performed by means of ANN and Bayesian linear analysis. With the same dataset, Kim et al. (2014) increased the accuracy in the overall discrimination around 98 % using spectral angle mapper (SAM) and spectral information divergence (SID). A major problem in which HIS can help arises when the goal is to discriminate between similar defects or when the fruit is still immature and presents green and orange colours that make inspection difficult. Dong et al. (2014) used PCA to select a reduced set of wavelengths to detect citrus thrips in green citrus fruits.

ICT in the Citrus Inspection

The application of information and communications technology (ICT) in agriculture is becoming increasingly more frequent. Very recently, smartphones are being introduced into agriculture to perform several tasks, most of them related to calculators or virtual advisors for different operations of crop handling (Pongnumkul et al. 2015). These applications are starting to use the capabilities of the sensors integrated within these smart devices, like the built-in camera. Gong et al. (2013) developed a citrus yield estimation algorithm utilising a modified 8-connected chain code. The algorithm was incorporated into an app for Android mobile devices obtaining a correct recognition accuracy of 90 %. However, the processing time was about 3 s for one image with a 1.4-GHz CPU of the handheld device and these results cannot be generalised because the segmentation algorithm is based on thresholds and this technique is not robust under natural conditions. On the other hand, yield estimation in citrus has already been investigated using colour cameras, thermal cameras or hyperspectral equipment with different success due to fruit overlapping or fruit hidden by leaves; it is therefore doubtful that it could be achieved using a smartphone with such accuracy under real conditions.

Another application available in Google Play® using computer vision for citrus is the ICC-Calc for citrus fruits (https://play.google.com/store/apps/details?id=org.ivia.icc_calc). This app estimates the citrus colour index (CCI) of citrus fruits in the field or in the packinghouse, which makes it a tool that can be used to decide the maturity of the fruits. The colour index is shown live on the screen by placing the camera in front of the fruit and allowing the phone’s torch to be switched on to make it more robust under different lighting conditions. In addition, it shows the colour coordinates of the fruit in several colour spaces. The image processing is carried out in real time using the OpenCV libraries for Android (http://www.opencv.org).

Table 2 summarises the different works carried out for the application of computer vision in the citrus inspection in postharvest ordered by different topics chronologically.

Table 2 Chronological achievements in the postharvest handling of citrus fruits using computer vision
Full size table

Conclusions

Citrus fruits are the number one fruit crop in international trade in terms of value, which has led to a lot of effort being made to automate different inspection operations along the production chain. Machine vision was proved to be very useful and practical from automatic fruit inspection indoors and outdoors to yield mapping with proven performance and accuracies. These applications are well illustrated in this study. Also, problems dealing with citrus fruit are well described along with their solutions.

Many studies have been conducted for on-tree fruit identification using different imaging platforms under natural outdoor illumination conditions. Various detection algorithms have been developed with acceptable detection accuracies; varying illumination and fruit occlusion are, however, still major obstacles hindering the development of a real-time in-field fruit detection system. For the robotic citrus harvesting system to be successful, there are various factors to be considered, such as population, spacing, shape and size of the plant, physical properties of the fruit and the removal method.

In postharvest, machines are only capable of sorting fruits by size and weight, but consumers appreciate more quality attributes, like a good appearance and the absence of defects. Furthermore, producers need innovative and non-destructive solutions to detect dangerous diseases like citrus decay that can be a focus of future infections causing undesired losses of quality.

Research and development on machine vision for citrus fruit inspection have been focused until current on the measurement of external properties with great success. Today’s commercial machines based on colour images are currently able to obtain good accuracy in the quality control process with high processing speed. This process maximises the profit of the producer allowing packing the fruit sorted by size, colour or appearance which makes the fruits more attractive to the consumer.

The next step is the possibility of identifying different types of defects or dangerous diseases so as to be able to sort the fruits according to commercial strategies and remove undesired fruit from the line or delivering the fruit to appropriated markets. This objective is starting to be solved in some commercial electronic sorters aimed at the early detection of decay using UV, but more effort is required to improve their precision. The possibility of using non-standard computer vision systems outside the visible spectra, like hyperspectral imaging, has offered new methods to explore other spectral ranges and reaches this objective. Nevertheless, the in-line assessment of internal quality in citrus is still a major challenge for researchers.

Several advances have been recently achieved using non-standard computer vision systems like hyperspectral imaging for the detection of special types of dangerous defects (i.e., decay, citrus canker, black spot, etc.), but different individual wavelengths are reported for the different types of defects and, moreover, even for the same type of defect. The selection of particular wavelengths seems to depend mostly on the particular statistical method used, and hence, at the moment, few conclusions can be extracted on this, or maybe that many different wavelengths can be used for the same purpose. Therefore, to develop a system capable of detecting all of them at the same time, which is what the industry would demand, is still a challenge that needs more research.