Keywords

1 Introduction

Food standards are evolving both to ensure the sustainability of agriculture and to address consumer concerns. The reputation of producers, and consequently their position in the market, is based on the quality of the product, which makes quality controls essential. The market and consumer exigencies, as well as increasing social concerns about good practices, including environmental, economic and social sustainability and traceability, require guarantees of optimal quality from the earliest stages of the crop to postharvest storage and treatments.

Optical devices and sensors have been introduced in the industry as non-destructive techniques for inspecting fruit [1]. Such technological advancements have been used for various purposes, ranging from the automatic sorting of products into categories to the control of processes which are difficult to observe, for instance, because of their long duration [2]. At this point it is important to note that the quality of biological products is not easy to assess, as individuals of the same category may differ greatly from one to another in terms of colour, shape or size. Furthermore, because they are living products, their physiochemical properties evolve over time. Their inherent variability sometimes introduces a certain amount of subjectivity into quality control, thus increasing the difficulty involved in developing automated inspection systems. Addressing these challenges often requires research in advanced and multidisciplinary technologies, and sometimes the use of expensive equipment.

Machine vision inspection is aimed at ensuring the quality of each product and the correct classification (including rejection) of those individual items based on quality standards. Automation is aimed at reducing production, processing and handling costs, but also at delivering the produce to the appropriate markets, thus optimising the overall profit. Furthermore, the excellence of a commodity is often achieved by ensuring correct and regular sizes, suitable colouring, and absence of external damage, optimal organoleptic properties and the absence of harmful residues. However, despite the great amount of research devoted to machine vision-based inspection systems [2, 3], the introduction of this technology in the industry is still relatively scarce because of its relatively high cost, the complexity of the equipment needed and the particular requirements for each implementation.

Current computer-based applications for the inspection of fruit and vegetables are described in the following sections. Most of these systems use visible (VIS) information to inspect the external quality of the produce using conventional techniques. However, recent advances include hyperspectral imaging to assess chemical composition, inspection of the internal quality of the produce or the detection of invisible damage, almost always for the real-time implementation of automated, in-line inspection and quality control systems.

2 Machine Vision Systems Based on Visible Information

The success of computer vision-based systems for the external inspection of fruit or vegetables depends largely on the quality and resolution of the acquired images, which is closely related to the cameras employed and the illumination of the scene. Frequently, such systems measure and compare colours, and for this reason illumination with a good colour rendering index is required. This index is a quantitative measure of the degree to which a test illuminant renders colours similar to their appearance under a reference illuminant [4]. The illumination must be uniform and avoid specular reflection, which produces bright spots that can mask certain blemishes. This is especially important for the estimation of two of the main external properties associated with the quality of the fruit by consumers, namely the colour and the presence of external defects.

2.1 Measurement of Colour

Colour is one of the most important attributes of many agricultural products that can be expressed using several standard colour spaces, the most used in image processing in agriculture being RGB (Red, Green, Blue), HSI (Hue, Saturation, Intensity) and CIELAB. The first is the native colour space of computers and digital devices and the others try to imitate human perception better. Figure 1 show some examples of apples and the distribution of their colour in different colour spaces obtained using the program FoodColorInspector (available at http://www.cofilab.com).

Fig. 1
figure 1

Apples of Royal Gala and Golden cultivars with different colours and the distribution of the colours in the RGB, HSI and CIELAB colour spaces

Full size image

As fruit ripens, chlorophyll degrades and new pigments such as anthocyanins or carotenoids start to be synthesised, resulting in the fruit turning from green to a wide variety of colours, mainly ranging from red to blue [5]. Hence, the consumer normally associates colour with the stage of maturity or ripeness of fruit and thus it plays an important role in the purchase decision. For this reason, colour has mainly been studied as an indicator of maturity. However, the presence of discoloration or stains on the skin can make it difficult to measure the average colour of the whole fruit, leading to inaccurate results. For this reason, Mohammadi et al. [6] developed an algorithm to classify persimmon into three maturity stages. In their work, black stains on persimmon were segmented and removed from the analysis. Then they used colour bands such as R and G from the RGB colour space, b* from the CIELAB colour space, S from the HSI colour space and grey levels and linear (LDA) and quadratic discriminant analysis (QDA) to assess fruit maturity stages with a 90% rate of success. Another approach is reported by Taghadomi-Saberi et al. [7], who used the CIELAB colour space to study the evolution of the ripening of sweet cherries. The colour coordinates were measured using a chroma meter and a CCD camera-based device that employed an artificial neural network (ANN) classifier. They achieved a coefficient of determination R 2 = 0.99 between both measurements. They also observed that L* and b* values decreased during the ripening of the cherries, whereas a* values first increased and then decreased. Baltazar et al. [8] sorted tomatoes using a colorimeter and a firmness sensor. The ratio a*/b* and the L* coordinate of the CIELAB colour space were used to study the changes in colour associated with storage time. El-Bendary et al. [9] used HSV colour coordinates and the first three colour moments (mean, standard deviation and skewness) to sort tomatoes. They employed Principal Components Analysis (PCA) for feature extraction and Support Vector Machines (SVM) and LDA for classification. The performance of the system was evaluated by means of the area under the curve of the receiver-operating characteristic (ROC curve) [10].

The colour of fruit is often expressed using indexes. Guzmán et al. [11] set up a maturity index for olives to determine the optimal harvest time. The index was based on colour segmentation of the olives using the k-nearest neighbour (KNN) algorithm, and calculating the percentage of the area of the olives that belonged to one of four predefined classes (bright-green, greenish-yellow, reddish-brown or black). From these data, olives were assigned a maturity index ranging from 0 to 4. The citrus colour index (CCI) is an industrial standard index to estimate the maturity of oranges and mandarins based on Hunter Lab coordinates. Vidal et al. [12] developed a computer vision system for on-line estimation of this index at a rate of eight fruits per second. The algorithm converted RGB coordinates into Hunter Lab coordinates and calculated the CCI of each fruit individually. Four images from each fruit were acquired as they rotated under the camera, the CCI being assigned an average of the four images.

Apart from colour, other external properties of fruit can be related to maturity and quality. The advantage of image analysis is that it allows several of them to be estimated simultaneously from the same image. For instance, Surya Prabha and Satheesh Kumar [13] developed an image analysis system to assess colour intensity and different geometric features (area, perimeter, major axis length and minor axis length) of bananas.

Furthermore, colour has often been combined with other information for better assessment of ripeness. For example, Vélez-Rivera et al. [14] classified Manila mangoes into four stages of ripeness. They built a PCA-based model that included colour information (CIELAB and HSB colour coordinates), soluble solid contents, total acidity, firmness and a ripening index based on these physical properties.

2.2 Detection of External Defects

Most consumers associate fruit and vegetable quality with good appearance, that is colour, shape and total absence of external defects. Deformations and presence of skin damage or diseases are the most influential factors affecting price. However, skin damage and diseases are more difficult to assess or detect than colour, shape or size because of the wide diversity of potential defects that can be found in packing houses [15]. Moreover, particular types of defects may present diverse colourations in the same piece or batch, or even coincide with the colour of the sound skin of other fruit of the same commodity [16]. This can be seen in Fig. 2, which shows images of different types of external defects in oranges cv. ‘Navel’ and the images after a segmentation process.

Fig. 2
figure 2

External defects in oranges cv. ‘Navel’ and the images after a segmentation process

Full size image

Many authors report inspection systems based on colour information alone. Al-Rahbi et al. [17] classified dates into three categories (no-crack, low crack level, high crack level) depending on the extension of the damaged surface found by image analysis. They used the R coordinate of the RGB colour space and the H and V coordinates of the HSV space after selecting the most discriminant ones using LDA. They achieved more than 70% accuracy and more than 80% when the problem was reduced to only two classes (sound and cracked dates).

In some machine vision applications, the calyx or the stem-end can be confused with some skin defects. The colour of the stem/calyx region often differs from the typical colour of the skin of the fruit and it is therefore identified as a defect. For this reason, authors report exploiting other information sources such as morphological or multispectral parameters. Blasco et al. [16] developed a system to detect these elements and 11 different types of defects in oranges developing a region-growing algorithm. A region of the image was considered to be a defect when its colour diverged from the colour of the largest region of homogenous colour, which was assumed to be the sound skin. The system could detect the defects in 94% of cases, with only 4% of false detections, the stem being distinguished from the defects in 100% of cases. Later, Blasco et al. [18] identified these types of defects, achieving 70% correct identification using colour information alone and 76% on adding multispectral information. These results increased to 86% when morphological information about the regions was included in the decision algorithm [19]. However, correct identification of the stem was achieved in only 66% of cases. Li et al. [20] described a system to discriminate seven types of common defects in oranges with a 99% rate of success by employing colour and morphological information. However, they could not discriminate between these types of defects. Rokunuzzaman and Jayasuriya [21] also used morphological information to differentiate skin defects and calyxes for the automatic inspection of tomatoes at a rate of 180 fruits per minute. They used colour thresholding to detect blossom end rot and a shape factor to discriminate between cracks and calyxes with 87% success. Xu et al. [22] reported a complete machine vision system to grade kiwifruits based on their appearance, including the presence of skin defects. The system was able to grade them by size, shape and defective surface at 1.2 kg/min, with a success rate of 89%, 91% and 94%, respectively.

3 Use of Hyperspectral Imaging for Qualitative Assessment of Fruit and Vegetables

Systems based on the visible spectrum have been described in the previous paragraphs. Such systems have been designed with the intention of emulating the human eye. Nowadays they are relatively low-priced and fast. However, current technology offers the possibility of going far beyond the capabilities of the human eye. For instance, some damage or the presence of contaminant agents can often be observed in particular regions outside the visible spectrum, or their detection may be enhanced at certain specific wavelengths. Initial approaches found in the literature took advantage of the acceptable sensitivity to near-infrared (NIR) of most charge-coupled device (CCD) sensors in many cameras, and combined this information with colour information. The first successful applications were aimed at detecting visible and invisible blemishes [23].

Multispectral systems have been considered to be the first based on non-standard optical devices for simultaneously processing the same scene in different spectral regions. Originally, they consisted of a series of interferometric filters coupled to a wheel placed between a monochrome camera and the scene [24]. These systems had several drawbacks, such as their low acquisition speed and the small number of images that could be acquired. Recent technological progress has allowed faster simultaneous acquisition of images, thereby opening up the possibility of developing new applications in fields such as remote sensing [25] and pest detection [26, 27] among others. Moreover, the price of the equipment has gradually become more affordable, thus enabling the use of related technologies in many applications to agriculture [28].

Although capturing images in stationary applications can be accomplished by swapping narrow band pass filters in front of the camera lens, a more sophisticated and versatile solution is offered by the use of electronically tuneable filters, capable of acquiring a large number of images at different consecutive wavelengths, thus making it possible to develop new inspection systems based on hyperspectral images [29]. Hyperspectral imaging systems allow spatial and reflectance information to be acquired at the same time, which can be decisive for certain applications [3]. Such systems acquire a huge amount of information, but this is also their major drawback because some of this information is redundant or unnecessary [30, 31]. For this reason, much work using this technology has focused on reducing the amount of redundant information by projecting the high-dimensional data space into a lower-dimensional space and trying to preserve most of the meaningful information [10, 32]. In parallel, spectroscopy has been used to assess certain properties of food and even to detect some pathogens [33, 34]. Figure 3 shows a hyperspectral imaging of an orange with a windscar defect captured in the range 430–1,050 nm with a resolution of 10 nm.

Fig. 3
figure 3

Hyperspectral image of an orange cv. ‘Navelate’ with an external defect captured in the range 430–1,050 nm

Full size image

Electronically tuneable filters are devices whose spectral transmission can be electronically controlled by applying a voltage or acoustic signals. Three main technologies are currently available for acquiring hyperspectral images for fruit analysis: image spectrophotometers [35], Acousto-Optical Tunable Filters (AOTF) [36–38] and Liquid Crystal Tunable Filters (LCTF) [39]. AOTF consist of a crystal in which selected wavelengths of light are separated from a broadband source using acoustic waves at specific radio frequencies. Alternative compression and relaxation of the crystal lattice generates density changes that produce refractive index variations which act as a transmission diffraction grating. Unlike a classical diffraction grating, AOTF only diffract one specific wavelength of light, so they act more as a filter than a diffraction grating. LCTF use a stack of successively thicker, polariser birefringent liquid crystal plates which can generate a tuneable retardation of light transmission. Switching speed is limited by the relaxation time of the crystal and is of the order of 50 ms. Spectral resolution of LCTF is typically of the order of several nanometres.

Image spectrophotometers acquire spectral data by scanning the scene line by line, making use of the relative movement of the objects in the scene with respect to the instrument. These sensors usually offer an excellent spectral resolution, but require precise synchronisation of the image acquisition with the movement of the sample or the instrument [40]. This is probably the most extended configuration of AOTF-based systems, offering good tuning times (around 50 ms) and accurate frequency selectivity. However, they have a limited field of view [41]. In general, LCTF-based devices are usually more compact than those based on AOTF and provide a wider field of view. Nonetheless, their major drawback is related to their greater requirements in terms of time needed for tuning [42].

Even if the use of this equipment is expanding, one has still to take into account important requirements for them to work properly, such as adequate spatial and spectral distribution of the lighting, correct focusing of the scene across all the different wavelengths or spectral regions and proper spatial matching of the images, or at least of the objects in the scene, when using changeable filters or when the object of interest is moving during the acquisition. Furthermore, the sensitivity of the different components of the acquisition sensors is not uniform across the spectrum, and this should be taken into account in the design. Suitable calibrations and adjustments are always necessary to keep the results of the image analysis independent of undesired phenomena [43].

Proper lighting is also crucial when acquiring hyperspectral images. Unwanted bright spots must be prevented when providing high-quality, homogenous scene illumination. Light sources have different characteristic spectral emissions. Daylight-type fluorescent tubes rarely go beyond 700 nm and should be discarded in work that uses near-infrared (NIR). Incandescent lamps generate strong NIR emissions but produce directional light which is difficult to diffuse. In addition, it is important to take into account the shape of the object to be analysed to avoid specular reflections. This is particularly important for spherical or quasi spherical shapes. On the other hand, cameras should be sensitive to the specific spectral region used in each application. Standard CCD cameras are almost insensitive to wavelengths from 900 nm and beyond, which can make it impossible to take full advantage of the possibilities of tuneable filters. Cameras based on InGaAs sensors with a stabilised temperature are sensitive beyond 1,000 nm, thus enabling an optimal use of NIR information. Lenses are also critical and must be properly selected for each particular application. Optical paths through the lens change depending on the frequency of the transmitted light (refraction indexes depend on wavelength). This makes the focus planes vary considerably between bands that are separated by relatively far distances in the spectrum (e.g. between some visible and some infrared bands), resulting in scenes that are focused in some bands being out of focus in others.

3.1 Automatic Assessment of Bio-Chemical Properties of Fruit and Vegetables

Assessment of the ripening stage of many fruits still relies on trained people’s experience or on destructive measurements of certain physicochemical properties. These approaches are inefficient and incompatible with large-scale production and trading. Machine vision systems based on hyperspectral imaging offer new tools to assess the concentration of some chemical compounds or properties related to maturity. This is the case of the work reported by Schmilovitch et al. [44], who presented a method for the non-destructive measurement of total soluble solids (TSS), ascorbic acid, chlorophyll and carotenoid contents in three bell pepper cultivars using hyperspectral images obtained using AOTF cameras. By means of partial least squares (PLS) regression models developed throughout the grooving session and specific to each variety, they managed to estimate the distribution of such internal components in whole peppers. Rajkumar et al. [45] analysed hyperspectral images of bananas at different ripening stages, and stored at different temperatures, to develop calibration models for the prediction of some quality parameters (moisture content, TSS and firmness). Munera et al. [46] used hyperspectral images to predict astringency in persimmons and to build the astringency distribution maps.

Hyperspectral image processing can also be used to obtain information about the presence of biochemical substances related to certain damage or physiological disorders of some agricultural produce. For instance, Gaston et al. [47] investigated the potential of visible and near-infrared (VIS/NIR) (445–945 nm) hyperspectral imaging for the prediction of polyphenol oxidase (PPO) enzyme activity, which produces browning on mushroom caps and is the major cause of their quality loss, accounting for a reduction in their market value. Yang et al. [48] studied the anthocyanin contents of the pericarp of lychees, because it is related to postharvest browning. They processed hyperspectral images of fruits in the range 308–1,105 nm by removing differences in light intensity between different areas of the samples, extracting the average spectra from the regions of interest (ROI) and selecting two sets of optimal wavelengths using successive projection and stepwise regression algorithms. Finally, they built models for mapping the anthocyanin distribution in the samples.

Long-distance transoceanic shipment of fruit requires delivery of high quality, consistent fresh fruit in the country of origin so as to meet the quality standards upon arrival at the destination. Hua et al. [49] investigated the potential of hyperspectral imaging to study how the mechanical properties of blueberries are related to their organoleptic quality, storability, transportability, resistance to mechanical damage and susceptibility to spoilage during postharvest and marketing handling. They tried to link spectral data to mechanical properties obtained from texture profiles and puncture analysis. Similarly, Leiva-Valenzuela et al. [50, 51] acquired reflectance and transmittance hyperspectral images of blueberries in the range 400–1,000 nm to build calibration models to predict TSS and firmness, and to assess the effect of fruit orientation on the durability of the fruit during transportation.

Consumers are willing to pay higher prices for fruit with health-stimulating properties such as bioactive compounds or antioxidant ingredients, for example lycopene and phenolic compounds. Liu et al. [52] reported an application of multispectral imaging for predicting the contents of such compounds in tomatoes, and compared the performance of different prediction models based on PLS, least squares support vector machines (LS-SVM) and back-propagation neural networks (BPNN).

Similarly, anthocyanins are phenolic components of red wine grapes which have a great influence on the quality of wine. Nogales-Bueno et al. [53] developed a non-destructive method, based on hyperspectral images, for the assessment of the important parameters that determine the technological and phenolic maturity of white and red grapes (pH, total acidity, sugar concentration and total phenols). Later, Nogales-Bueno et al. [54] used a similar approach to estimate maturity and sugar content and investigated the possibility of using anthocyanin profiles, colour image analysis and near-infrared hyperspectral imaging tools to distinguish between the varieties Tempranillo, Graciano, Garnacha and Mazuelo. Chen et al. [55] developed a model to estimate the anthocyanin contents of wine grape skins using NIR hyperspectral imaging. They used this information to assess the phenolic maturation stage of grapes after veraison, with the final goal of predicting the best harvest time. Baiano et al. [56] also used hyperspectral imaging to predict the physicochemical indices and sensory characteristics of table grapes. Furthermore, Leiva-Valenzuela et al. [50] studied the potential of VIS/NIR spectroscopy and hyperspectral imaging to estimate the internal or external constituents of potato tubers, which are important to the processing industries.

3.2 Detection of Skin Defects and Diseases

Machine vision systems based on hyperspectral imaging open up the possibility of automatically detecting early stages of fruit damage invisible to the human eye because they can provide information from outside the visible spectrum. For instance, Lü et al. [57] developed a VIS/NIR hyperspectral imaging system covering the spectral region 408–1,117 nm for the automatic detection of bruising caused by excessive mechanical loading and stress of kiwifruits during harvest, transport, handling and storage. Such bruises are very difficult to detect in the first hours after they have been produced. For this purpose they selected particular wavelengths to develop algorithms to differentiate between bruised and sound tissues. Likewise, Baranowski et al. [58] worked on the early detection of bruises in apples, using VIS/NIR and short-wave infrared (SWIR) wavelength ranges. Similarly, Vélez-Rivera et al. [59] reported on the feasibility of an automatic system for early detection of mechanical damage in ‘Manila’ mangoes using specific spectral bands. Lee et al. [60] investigated an extended range of NIR to detect bruises on pears.

The presence of a few fruits infested by a pest or affected by fungus in a shipment can render the entire consignment unmarketable. Furthermore, many other kinds of skin damage must be detected during postharvest quality control because they can be the starting point of fungal infestations. For instance, tomato cracking is one of the main causes of produce rejection by retailers because it creates a path for the potential entrance of pathogens. Cho et al. [61] investigated the feasibility of an inspection system based on hyperspectral fluorescence and determined optimal wavebands to distinguish between defective areas and sound surfaces to detect cuticle cracks. Analogously, Yu et al. [62] investigated the potential of hyperspectral imaging for crack detection in fresh jujubes. They identified some wavelengths to develop image processing algorithms for locating the cracks, but reported that the best option was to use ratios of bands instead.

Pests themselves are also a major cause of fruit rejection in the market. As an example, the Mediterranean fruit fly causes damage to many different fruits and significant economic losses for growers, processers and exporters, and is impossible to detect using colour information alone [18]. Haffa et al. [63] proposed a system for detecting infested mangoes using greyscale images generated from absorbance levels at particular NIR wavelengths. Another approach was taken by Wang et al. [64] who identified effective wavelengths for maximum discrimination of jujube fruits affected by damage caused by insects. They created a discriminant function to identify the stem-end/calyx-end, the sound cheek and insect damage, and finally a method to distinguish damaged fruits from those free of infestation. This approach has also recently been investigated by Gómez-Sanchis et al. [65, 66] with the aim of detecting decay lesions in citrus fruits.

Physiological disorders and decay are a consequence of postharvest processes which also generate important economic losses. Simko et al. [67] developed several indices to estimate decay and freezing injuries of different cultivars of lettuces, based on ratios of particular wavelengths obtained from the spectral analysis in the range 380–1,000 nm.

Fruit and vegetable diseases, often caused by bacteria, are also a major source of trouble for fruit and vegetable exporters. For instance, citrus canker is a severe disease of citrus fruit causing enormous socioeconomic loses for those countries affected. Qin et al. [68] used spectral information divergence estimated from NIR images as a method for detecting affected fruit, and Zhao et al. [69] introduced the effect of the harvesting time and its influence on the detection of the damage. Later, Qin et al. [70] exploited particular bands obtained from the PCA technique and used ratios between bands to create a fast detection system. Subsequently, Qin et al. [71] reported a real-time system to detect such canker lesions. A different approach was taken by Wang et al. [72] who compared the spectral reflectance of onions affected by sour skin (a bacterial disease) in the spectral region of 950–1,650 nm and determined optimal bands for identifying infected onions.

3.3 Detection of Pathogens and Contaminants

Foodborne illnesses are also of major concern for consumers and, hence, interest in methods and technologies for detecting contaminated food and preventing the presence of pathogens causing such illnesses has grown significantly, both in the agri-food industries and in regulatory agencies. Hyperspectral imaging offers a vast potential for detecting contaminants and pathogens in food. For instance, Lee et al. [60] determined two significant wavelengths and developed multispectral imaging algorithms to detect faecal contamination on leafy greens (spinach and lettuces) in an automated system for in-line inspection at the processing plants. At the same time, Everard et al. [73] used ultraviolet (UV)-induced fluorescence, violet-induced fluorescence, VIS/NIR reflectance and hyperspectral image processing, in combination with multivariate statistical analysis, for the detection of faecal contamination on spinach leaves.

Tomato hornworm is one of the several types of large caterpillars that attack tomatoes in the United States and whose faecal matter is closely related to the presence of Escherichia coli and Salmonella. Yang et al. [74] studied the development of a multispectral imaging algorithm to detect such contamination on the surface of mature red tomatoes. In similar work, Yang et al. [75] developed a simple multispectral algorithm to detect faecal contamination on the surface of apples. Aspergillus flavus generates toxins on dates and logically causes food safety concerns which greatly depreciate the value of the product. Teena et al. [76] studied the presence of lesions caused by this fungus using NIR hyperspectral imaging.

4 Specific Problems To Be Solved for Real-Time, Automatic Quality Monitoring

To satisfy market demands, producers must inspect the quality of each piece of fruit or vegetable before shipping. This task is traditionally carried out by workers situated on one or both sides of a conveyor belt. They visually inspect the produce and remove those pieces not meeting the quality standards. Pieces are transported slowly enough to allow the workers to inspect all of them and even manipulate them to ensure the inspection of most of their surface. The quality of the product is not always fully guaranteed because workers have different tolerance criteria and, at the same time, their criteria may vary during the day, as inspection requires concentration and is a tiring, repetitive activity. For all these reasons, this operation is normally time-consuming, subjective and expensive. The alternative is the use of electronic sorters based on machine vision.

In most in-line applications, products travel rapidly under the machine vision system, often carried by a conveyor belt or on top of rollers as shown in Fig. 4. In such cases, the camera has to be able to acquire images at a very high rate when freezing the scene, and computer hardware and software has to be set to cope with very fast image processing. For example, Al-Mallahi et al. [77] developed an automatic machine vision system for sorting potatoes using UV-induced fluorescence capable of discriminating potatoes from undesired material. They processed one image every 94 ms with a success rate of 98%. ElMasry et al. [78] developed an automatic system to sort potatoes by size and shape. Roundness and length features as well as four other parameters calculated from Fourier analysis of the polar signature of the potato boundary were found to be effective in describing potato size and shape. The system achieved a high level of accuracy in estimating the shape and size of potatoes travelling at 1 m/s. Nevertheless, real-time inspection encompasses a number of problems which need to be solved. Some are related to the need to avoid blurred images and freezing the movement of the pieces when acquiring the images, whereas others are related to the exact positioning of the piece. Both require very precise synchronisation between the camera shot and the movement of the conveyor. Furthermore, adequate intensity and uniform illumination are always necessary. Other problems to be overcome are associated with processing speed constraints, which require considerable effort in algorithm optimisation and sometimes the use of dedicated hardware. Moreover, the need to inspect the whole surface of the pieces requires the use of specific mechanisms or the use of multiple cameras situated to acquire several points of view.

Fig. 4
figure 4

Machine for the automatic inspection of fruit in-line using computer vision

Full size image

The electronic shutters of the cameras allow proper synchronisation between the movement of the pieces and the image acquisition and a short exposure time, both required for a correct freezing effect. The fast development of powerful Light Emitting Diodes (LED) has allowed the development of very uniform illumination systems with multiple light sources. Furthermore, it has opened up the possibility of generating accurate light pulses instead of continuously illuminating the scene. Pulsed illumination helps to avoid the blurring effect of the movement of the objects in the images and to save energy, which may be crucial for some applications, especially those related to machines working outdoors. The use of fast shutters and strips of pulsed LEDs was reported by Vidal et al. [12] to take different views of oranges moving at 0.4 m/s on a conveyor. Kohno et al. [79] used this type of illumination on a mobile platform to inspect citrus fruits during harvesting, but they needed 12–20 s to process a single fruit because an NIR spectrometer was also used to estimate the sugar content of fruits. However, Cubero et al. [80] used this type of illumination and settings for real-time inspection of oranges on a citrus harvesting machine. The system was able to work at a rate of eight fruit per second, and captured four images of each fruit to make a decision. They reported a 0.99 coefficient of determination (R 2 = 0.99) for size prediction and R 2 = 0.92 for colour.

To achieve real-time operation, image segmentation and processing must be carried out extremely quickly. The work of Aleixos et al. [23], who developed a camera capable of acquiring multispectral images (VIS/NIR) from the same scene, is an example of the use of specific hardware and algorithm optimisation to reduce processing time. They used parallel image processing algorithms run on two digital signal processors (DSP) to process 10 citrus fruits per second and sort them by colour, size and presence of defects, using nonlinear discriminant procedures to segment the images and to sort the pieces. However, the processing speed of current hardware allows complex algorithms to be implemented in relatively low-cost devices. Commercial cameras equipped with microprocessors can be used to create smart equipment for in-line processing, as reported by Cubero et al. [80], who achieved real-time colour image processing for citrus sorting by implementing optimised algorithms in a camera with standard computing capability.

Several approaches have been made to solve the problem of inspecting most of the surface of the pieces. Many inspection machines rotate the pieces when a series of images are captured. Leemans and Destain [81] used a roller conveyor to capture images of apples. They adjusted the rotational speed of the rollers in such a way that a spherical object with a diameter of 72 mm made one complete rotation in exactly four images. Images were acquired at a rate of 11 per second. A hierarchical grading method based on the analysis of 16 external properties including colour, shape, texture and position was used to classify apples. Bennedsen et al. [82] developed a system to capture six different views of each apple as the fruit was transported on a conveyor. Other reported solutions include the use of different cameras to capture different views of the fruit, as depicted by Xiao-bo et al. [83], who employed three colour cameras to inspect rotating apples and classify them into two categories depending on the presence of defects. A cheaper alternative is the use of mirrors instead of cameras, as described by Reese et al. [84], who used parabolic mirrors to show parts of the fruits that were hidden from the camera.

Soft or very small processed fruits are more difficult to handle and hence to inspect, and so may require particular solutions. For instance, Blasco et al. [85] developed a prototype to grade fragile mandarin segments and to separate marketable segments from undesired material such as small pieces of peel, broken segments or segments with seeds. Segments travelled on narrow conveyor belts under two cameras which acquired images every 48 ms. Their vision system was able to process 20 images per second, but mechanical limitations related to the difficulty in handling the segments reduced the operational speed to four images per second. Semitransparent conveyor belts were employed to illuminate the scene from the back, enhance the silhouette of the segments and detect the seeds easily. A similar machine was reported by Blasco et al. [86] for real-time inspection of pomegranate arils using front illumination and opaque blue conveyor belts. In this case, colour parameters were used to detect rotten or immature arils and to grade arils into uniform colour batches, which are more attractive to the consumer.

5 Conclusions

This chapter summarises the current state of the art in computer vision-based fruit and vegetable inspection. The final aim of the technologies described here is the implementation of machines capable of automatically inspecting the quality of these products, removing those not reaching an adequate level of quality and ensuring an objective sorting in categories that make them more attractive for the consumer and optimise their value. To enhance current sorting systems giving high added value to the products, the use of physicochemical and morphological information other than simple visual appearance is becoming more and more relevant.

Machine vision-based systems are always under constant evolution thanks to the development of new types of cameras and imaging devices. UV and NIR acquisition systems are becoming easily available. Hyperspectral systems have demonstrated their ability to capture information invisible to the human eye, such as the presence of internal defects or the measurement of chemical compounds. However, despite all these technological advancements, a compromise between their increase in performance (image acquisition rate, resolution) and costs must be found in the forthcoming years.