Keywords

8.1 The Situation in Horticulture

In the production and harvest of fruit and vegetables (horticultural crops), large investment costs and expected high productivity of the land provide strong incentives for implementing technology-supported production methods because the application of more technology possibly results in more precise, optimized management of production. The horticultural product is highly perishable, of high value and has heterogeneous quality . This heterogeneity of horticultural produce often requires the handling of individual plants (Schouten et al. 2007).

With increasing demands regarding the yield , sustainable use of land and water resources, and requests for good crop quality production methods have been changing. Conventional citrus orchards with fixed tree spacing have been replaced by systems with greater tree density supported by water-efficient drip irrigation. Intensive production systems using adapted cultivars and pruning techniques show densities up to 3200 trees per ha, which is in the range that we find in apple production with small spindle tree forms enabling planting density of 2600–3700 trees per ha. In stone fruit , such as mango (Mangifera indica), sweet cherry and olive production similar trends can be seen after rootstocks became available that reduce vegetative growth of the cultivar. In addition, sophisticated tree training has been practised for over 30 years (Fig. 8.1). In olive production, for example, the system is called hedgerow or super high density allowing 600–1250 trees per ha. Mechanization of spraying, foliar application of fertilizers and thinning agents, pruning, and harvesting fruit (Fig. 8.2) has been introduced because machinery ensure the safety of application and labour costs have also become a crucial factor in fruit and vegetable production.

Fig. 8.1
figure 1

Tree forms in olive (Olea europaea L.) (a), sweet cherry (Prunus avium L.) as slender spindle (b), and apple (Malus x domestica Borkh) in Y-shaped trellis system (c). Photographs courtesy of: Zude / Zude / Penzel

Full size image
Fig. 8.2
figure 2

Machine pruning in citrus (a) and apple (b) production, and harvester prototype (c). Photographs courtesy of: Zude / Betz / Karkee

Full size image

Applications are usually carried out uniformly over the orchard blocks because information on the individual plant has either not been available or utilized. A huge step forward in precision horticulture was achieved when the variable-rate application of chemical inputs was introduced, allowing treatment to be based on the requirement of the crop. In the first approach, recognition of plant biomass, tree height, gaps due to missing plants as this occurs regularly in perennial orchards represent the crop’s variability and, therefore, the ‘variable’, i.e. adaptive component of the precise production. Efforts are also being made to integrate variable-rate application legally into the pest management of tree fruit production. In Europe according to common agricultural policy (CAP), the amount of foliage needs to be known instead of a simple area-dependant calculation. The actual plant information on the foliage has been used as the basis for calculating the expenditure since 2020. Companies are now marketing variable-rate equipment and software so that innovation is entering horticultural practises.

Furthermore, horticulturists seek new or renewed tree training such as planar, cordon and y-shaped trellis systems that enhance light interception, which is closely related to the yield and fruit quality that the plants can achieve. Coincidently, such light-effective orchards (Fig. 8.1) might also favour the application of robots for harvesting and other tasks. Mechanical harvesting of table fruit is a benchmark (Fig. 8.2c), which still needs to be achieved because harvest costs amount to >50 % of all production costs in many countries.

The advantage of an intensive production system is to grow fruit more economically and in a way that allows variable-rate application in plant protection, foliar application of fertilizers, flower and fruit thinning, and harvest. The drawback is the increased susceptibility of the system to unfavourable conditions because the soil’s capacity to act as a buffer is diminishing. Furthermore, the tree is tied in the production system possibly reducing the tree ’s adaptation capacity. Taking into account the enhanced investment costs of intensive production systems, the introduction of sensors for in situ analysis of specific traits of the plant or even the plant’s physiological responses appears increasingly reasonable with feedback from the plant to reduce errors in the management of production and harvest.

In situ data acquisition can now be applied to reflect the plant status in digital format. At present, we see close cooperation between industry and research partners to develop digital twins of orchards and vegetable greenhouse production. The digital twins represent a model tree or single fruit in digital format that can be used to follow or simulate a process with an underlying physiological model. Here tree or fruit sensors provide information, preferably in real time, on the plant or fruit to define the digital twin, y. The environmental and production system variables form the X matrix to simulate the development of tree or fruit . The new information might provide valuable knowledge for sustainable fruit and vegetable production because, for the first time, agronomic models based on sensor data can be applied easily with cloud computing. This approach can possibly lead to new services and business models based on selling information rather than sensor devices.

From a fruit grower’s perspective ‘farming with sensors is so much easier’ because knowledge of the crop in real-time assists decision-making for precise management. To meet this goal, sensors should collect data in situ during production and postharvest. Information and communication technology tools, such as satellites, drones, autonomous platforms, wireless networks and data management techniques are available for all scales to support data acquisition by remote and proximal sensors directly. On the other hand, the translation of sensor data into information on the crop and knowledge of the process is still challenging.

8.2 Biotic and Abiotic Plant Stresses

8.2.1 Background

Pests and diseases can cause major economic loss in horticulture, and managing pests and diseases accounts for a significant cost of production. Consequently, one of the major components of precision horticulture is the detection of pests and disease at an early stage. Detection of anomalies at an early stage, in most cases, could allow the growers to manage the stress cost-effectively and prevent the adverse effects of that stress on yield and profit. Plant stresses can be divided into biotic and abiotic stresses. Abiotic stress is defined as the stress caused by non-living factors on living organisms under a specific environment (Husain et al. 2017). Abiotic stress can be caused by high irradiation, salinity, heat, frost, drought, flooding or undesirable conditions such as nutritional availability, nutrient imbalance or contamination by toxic chemical compounds. When the environmental conditions change and are beyond the normal range, abiotic stress can occur and adversely affect the performance of the plants. Abiotic stress is one of the major harmful factors affecting the growth and productivity of crops worldwide (Gao et al. 2007).

Biotic stress occurs as a result of plant interactions with other living organisms, such as arthropods, bacteria, viruses, fungi, parasites, insects, weeds, and cultivated or native plants (Gupta and Senthil-Kumar 2017; Shabala 2017; Atkinson et al. 2015). Herbivores can be harmful to the plants causing mechanical damage and competition for growth factors, resulting in reduced growth rate. On the other hand, plant pathogens can cause different types of diseases. Once an attack is perceived, plant metabolism must balance the demands for resources to support defence mechanisms against the requirements for cellular maintenance, growth and reproduction (Berger et al. 2007). Hence defence processes can be costly in terms of plant growth and health. Plants, herbivores and their natural enemies have coexisted for at least 100 million years, evolving a variety of beneficial and deleterious interactions (Karimzadeh and Wright 2008).

Currently, stress detection mainly relies on manual scouting, which is labour intensive and costly. Furthermore, human scouts rely on their senses, which are often not sufficiently quantitative and often fail to detect early stress . For example, certain plant stresses can be detected easily in the near-infrared range, but are not detectable by the human eye, which can only see in the visible bands. Much effort has been applied by engineers and scientists to develop sensor systems for detecting plant stress . Sensors for detecting plant stress in the field need to be cost-effective, easy to use, non-invasive, real-time or near real-time, and robust. They can be very valuable if they can detect the stress at an asymptomatic stage. The majority of the sensors that have these capabilities are based on spectroscopy or imaging techniques in the electromagnetic wavelength range from UV to far-infrared (Table 8.1). Other emerging techniques are volatile profiling-based and polymerase chain reaction-based sensors. These are new and growing areas of research, and more commercially available sensors that use these principles are anticipated. The advantage of these types of sensors is that they can be very accurate, and crop and disease-specific. The material in this chapter describes the optical techniques, which are currently used for biotic and abiotic stress detection. Sensors can be operated at the production site with tractors, autonomous platforms or robots, cranes, unmanned aerial vehicles (UAV), wireless sensor networks and manual readings (Fig. 8.3). Earlier approaches used cranes and sensors mounted on tractors (Zude-Sasse et al. 2016), whereas now trees can be monitored on all scales (Figs. 8.3b and 8.4).

Table 8.1 Examples of techniques reported in the literature for biotic stress detection in different crops
Full size table
Fig. 8.3
figure 3

Sensor platforms supporting optical sensors: (a) an agricultural vehicle and (b) unmanned aerial vehicle. Photographs courtesy of: Ehsani / Zude

Full size image
Fig. 8.4
figure 4

(a) Contour yield map of a citrus orchard , (b) contour map of tree volume (in cubic metres), (c) map of apparent soil electrical conductivity. Photographs courtesy of: Ehsani

Full size image

8.2.2 Remote Sensing for Stress Detection

One of the significant applications of remote sensing in agriculture has been stress and disease detection. Numerous different types of platforms can be used for remote sensing in precision agriculture (PA), such as manned aircraft (Hunt and Daughty 2018; Kamal et al. 2020), satellite (Hegarty-Craver et al. 2020; Zhou et al. 2017), and UAVs (Xiang and Tian 2011). Satellite-based remote sensing technologies are used more often for general stress detection over a large area because they have the capacity for continuously monitoring the earth’s surface. However, the measurement uncertainty is increased due to mixed pixels capturing the plant rows and the soil or ground cover in between the rows.

Recent advancements in UAV technologies resulted in the production of more affordable UAVs; lower costs and ease of use made them the platform of choice for stress detection when a high-resolution image is needed. Different types of sensor such as thermal cameras, multispectral cameras, NIR cameras, and regular digital cameras have been used on-board UAVs to collect data or monitor plant health (Martinelli et al., 2015). Computer vision and image processing have also been applied to remotely sensed images to make decisions in agricultural applications. These decisions could be carried out after collecting and processing the data or on-board while the UAV is flying (Maes and Steppe 2019).

Extensive reviews on the application of remote sensing for the detection of plant diseases were published by, e.g., Jackson (Jackson 1986), and West and co-workers (West et al. 2003). The topic was also covered in a review of video remote sensing systems for agricultural assessment (Abd El-Ghany et al. 2020). Remote sensing has been a useful tool in detecting foliar and soil-borne plant diseases (Oerke 2020). Assessments of plant disease infestations are typically carried out visually by workers. However, individual differences in perception of colour and light, as well as lack of concentration or fatigue, can significantly reduce the accuracy of visual disease assessments. Here as well as in instrumental remote sensing, other factors, such as cloud cover causing varying levels of light on different parts of the plant canopy, also limit accuracy of visual and instrumental disease assessments.

Instrumental remote sensing can be applied to plant disease detection because of pathogen-induced colour changes from chlorosis or necrosis. Furthermore, water-stressed crop canopies consistently have higher temperatures than non-stressed crop canopies because of reduced transpiration; thermal IF emissivity may be measured to detect these types of changes (Martinelli et al. 2015). Remote sensing may provide earlier detection of plant stress , even prior to the appearance of visual symptoms (Matese et al. 2018). Utilizing remote sensing instruments capable of detecting non-visible wavelengths could allow these reduced photosynthesis rates to be detected earlier than by visible disease symptoms (Reddy 2018; James et al. 2020; Vitrack-Tamam et al. 2020). A wide variety of remotely sensed data can be collected, and these data can help plan and schedule irrigation, water management (Gonzalez-Dugo et al. 2020), detect insect infestation problems and weed infiltration, and determine plant stress by carrying out small scale photogrammetric surveys using RGB and/or 4-band multispectral imaging and LiDAR laser scanning.

Plant health is detectable by airborne sensors because of the plant’s reflectivity and absorption of electromagnetic radiation. The pigmentation of the plants controls this reflectivity and absorption, creating incident radiation depending on the plant size, orientation, and colour. Plant pigment heavily relies on the amount of chlorophyll (Saglam et al. 2019), which intensely absorbs radiation within the visible spectrum. When a plant is stressed, chlorophyll production declines, increasing the reflectance of wavelengths in the visible spectrum, including those in the red bands.

Plant health and heterogeneity can be quantitatively and qualitatively measured by means of calculating a series of remotely sensed vegetation indices (VIs) (Zare et al. 2020). This is done through a series of image band calculations (Rasmussen et al. 2016). The most common calculation performed is related to crop status such as leaf area index (LAI), canopy cover, biomass, vegetation wellbeing, or ‘greenness’. Most estimations are based on the chlorophyll content, which is frequently calculated by the normalized difference vegetation index (NDVI), and is calculated using the sensor ’s red band (Rouse et al. 1973), which registers the absorption of red wavelengths due to chlorophyll content (Rouse 1973; c.f. Walsh et al. 2020). The NIR channel can be useful in determining plant stress due to higher reflection in plants containing more chlorophyll and vice versa. Meggio et al. (2010) used hyperspectral imagery via manned aerial vehicles to calculate a series of VI for detecting iron chlorosis in a vineyard . It was concluded that this type of imagery is useful in determining plants with iron chlorosis, which is a huge problem in vineyards and citrus plantations. Most sensing principles of remotes sensing for stress detection have been focused on optical techniques so far, but in recent years more complex approaches are emerging using multiple sensors and sensor fusion techniques. For example, it is common to see more cost-effective sensors that combine multispectral images and thermal images. Multi-sensor remote sensing could potentially enhance the possibility of early stress detection and has a lot of potential for large scale adaptation by growers.

8.2.3 Visible and NIR Bands

Visible and NIR bands are ideal for developing sensors for disease detection because the detectors used in this part of the electromagnetic spectrum are relatively inexpensive (Table 8.1). Sankaran and Ehsani (2013) used a portable spectrophotometer (SVC HR-1024, SpectraVista, USA) in the 350–2500 nm range for detection and classification of citrus greening or Huanglongbing (HLB) and citrus canker from healthy trees. They used quadratic discriminant analysis and a K-nearest neighbour classifier; both techniques provided large detection accuracy. Similar studies were performed with different classification techniques. Sankaran and co-workers (Sankaran et al. 2012) used a hand-held spectrophotometer to collect data from asymptomatic, symptomatic, freeze-damaged and healthy plants. Linear discriminant analysis, quadratic discriminant analysis, Naïve-Bayes and bagged decision trees were used as classifiers with 77, 92, 84 and 99 % accuracy, respectively. All classifiers were able to discriminate symptomatic-infected leaves from freeze-damaged leaves, but some asymptomatic leaves were incorrectly detected as healthy. Low-cost, rugged disease-specific sensors can be built that use only a few selected bands rather than continuous spectral data in the whole range of the visible and NIR bands. Active multispectral sensors using high-density illuminators at a specific narrow band for detecting disease and stress might be even better suited to field application. These bands are usually tailored for a specific diseases rather than for general stress detection. They use two bands in the red and NIR range to calculate the NDVI based on changes in chlorophylls (Olsen et al. 1969; Zude 2003) and flavonoids (Moran and Moran 1998) that provide data on the plant and its stress response. Previous experiments indicated that these approaches can distinguish between adaptation and irreversible damage to plants. Consequently, reflectance measurements based on remote sensing may be used to estimate integrated stress responses of the canopies (e.g., Peñuelas and Filella 1998). Mishra et al. (Mishra et al. 2011) used a four-band active optical sensor for detecting Huanglongbing (HLB) disease in citrus, two bands of which were in the visible region (570 and 670 nm), whereas the other two were in the NIR region (750 and 870 nm). They used K-nearest neighbour, support vector machines (Rançon et al. 2019) and decision trees for classification and achieved 96, 97 and 95.5 % accuracy, respectively. However, it can be assumed that distinguishing diseases by means of chlorophyll, water and microstructural changes is almost impossible because they are almost universal in all fruit and leaf responses. An additional trait, but still unspecific, is the intensity difference of reflected light between 450 and 570 nm, which is caused by changes in the xanthophyll cycle. The xanthophyll cycle describes variation in the actual composition of carotenoids. Changes in the related carotenoids (zeaxanthin, violaxanthin and antheraxanthin) appeared measurable with reflectance spectra and were recorded in remote- or proximal-sensing with the leaf under various stress conditions (Peñuelas and Filella 1998). They have been little studied on fruit because reproductive plant organs tend to accumulate carotenoids during seasonal development to enhance attraction to vectors. However, wavelength-specific indices or multivariate methods for processing the spectra of harvest products or leaves might provide relevant data for detecting anomalies.

8.2.4 Mid-Infrared Bands

Mid-infrared (MIR) spectra range from 2500–50,000 nm. The MIR spectroscopy usually requires sample preparation, which is a disadvantage compared to NIR spectroscopy. However, MIR has some advantages over NIR spectroscopy. One of the major advantages of MIR spectroscopy is that many chemicals have a unique peak or signature in this region which makes spectral interpretation easy compared to the NIR region where the spectra comprise overlapping overtones of absorption bands of many interfering functional groups of molecules. The application of MIR for measuring the nitrogen content in impregnated tomato leaf has been indicated. Sankaran et al. (2010) used a portable MIR spectrometer (InfraSpec VFAIR, Spectro Scientific, USA) in the range of 5.15–10.72 μm to detect healthy leaves as well as those infected with HLB and canker. The leaves were ground into a fine powder and were placed on the crystal window of a MIR spectrometer. The scan time was 1 minute. Quadratic discriminant analysis and K-nearest neighbour classifiers were used, and resulted in large accuracies of 98 ± 0.9 % and 99 ± 0.9 %, respectively.

8.2.5 Fluorescence Spectroscopy

Fluorescence spectroscopy is an optical sensing technique that takes advantages of the re-emission of light from a sample. In this process, excitation light is absorbed by the material of interest and then emits this light at longer wavelengths. There are commercial sensors that use this technique. A hand-held multiparameter optical sensor (Multiplex_3, Force A, France) was used to detect HLB disease in leaves of two different sweet orange cultivars, Hamlin and Valencia. Four excitation wavelengths are employed in the instrument: UV, blue, green, and red (c.f. Zude 2009). For each excitation wavelength , yellow, red and far-red fluorescence can be measured. The classifiers used were Naïve-Bayes and bagged decision tree with accuracies of 85 % and more than 94 %, respectively. The bagged decision tree classifier performed better than Naïve-Bayes; however, it needed more time for the computation process, at least 10 times more than the Naïve-Bayes classifier (Sankaran et al. 2012).

8.2.6 Laser-Induced Breakdown Spectroscopy

Laser-induced breakdown spectroscopy (LIBS) is destructive, but it is an in situ spectroscopic technique that can be used for qualitative and quantitative analysis of elemental composition in solids, liquids and gases. This technique uses a high-power laser to generate plasma around the target and emissions from the atoms in the plasma are analyzed by a detector. The advantages of this technique include little to no sample preparation, analysis is done on very small samples and there is simultaneous measurement of multiple elements. The technique can also be rapid and cost-effective. The LIBS has been used in several applications, many of which include elemental analysis in different media (Yamamoto et al. 1996; Xu et al. 1997; Hussain and Gondal 2008). In addition to these studies, LIBS has also been used to measure nutrients in plant materials (Santos Jr. et al. 2012). Trevizan et al. (2009) used LIBS for analyzing microelements such as B, Cu, Fe, Mn and Zn in plant materials. They compared the laser-induced breakdown spectrometer results with those of conventional acid digestion. The detection limit of the LIBS method using their analysis protocol was 2.2 mg kg−1 for B, 3.0 mg kg−1 for Cu, 3.6 mg kg−1 for Fe, 1.8 mg kg−1 for Mn and 1.2 mg kg−1 for Zn. Similarly, Yao et al. (2010) used the LIBS technique to identify nutrients in orange leaves. The authors found that the spectral peaks indicate the specific elements and that their intensity is proportional to the elemental concentration. Thus, they concluded that the nutrient status can be evaluated from the spectral characteristics of the orange leaves using a LIBS system. Despite being able to demonstrate the potential of LIBS for nutrient analysis in citrus leaves, the study did not evaluate different citrus anomalies with the LIBS system. Sankaran et al. (2015) studied the variation in LIBS spectra obtained from healthy citrus leaves and leaves with anomalies such as diseases (HLB, canker) and nutrient deficiencies (Zn, Fe, Mg and Mn). Pattern recognition algorithms, support vector machine (SVM) and quadratic discriminant analysis (QDA) were applied successfully to classify the healthy leaves from leaves with anomalies.

8.2.7 Thermal Bands

The water status of fruit and vegetable plants as well as the detection of fruit have been analyzed frequently with thermal imaging. Thermal data can be obtained from satellites, airplanes and drones or by proximal sensing over the entire thermal range or with filters at a specific wavelength . The crop water stress index (CWSI) (Jones and Corlett 1992) and adapted indices have been employed for water stress detection. The CWSI is a surface temperature-based index between 1 and 0, with 1 representing the temperature of non-transpiring dry leaves and 0 representing fully transpiring wet leaves. Thermal imaging of canopies has been carried out by cranes, drones, terrestrial robots and tractors (c.f. González-Dugo et al. 2013). The CWSI has been used to guide irrigation protocols, for example in olives in arid climates (Ben-Gal et al. 2009) and in peach orchards considering the fruit development stages in the semi-arid environment (Bellvert et al. 2016). Interpretation of the data differs considerably in semi-arid conditions from that in arid sub-tropics in relation to emissivity itself and even more considering the thresholds indicating water stress (Jones and Corlett 1992). In differently irrigated apple trees under a hail net, CWSI values ranged between 0.08 and 0.55 with values >0.3 defined as stressed trees under the given conditions (Nagy 2015).

8.2.8 Ranging Sensors

Ranging sensors, particularly those based on light detection and ranging (LiDAR) and the ultrasonic sensors, gained importance in fruit tree management for variable-rate application in plant protection and foliar fertilizer application. These types of sensors are mainly utilized to estimate the geometric (height, width, volume) and structural (leaf area, leaf density, stem diameter, etc.) characteristics of fruit trees. The initial applications of ranging sensors in horticulture have been deployed with ultrasonic sensors (McConnell et al. 1984; Giles et al. 1987). The sensors were arranged at different heights along a vertical pole, facing the side of the tree row. Each ultrasonic unit measured its distance to the canopy as the system moved along the row at constant speed and aimed to estimate height, width and tree-row volume (TRV). This system was used to control variable-rate application in real-time, resulting in spray volume savings up to 52 % in apples and 28 % in peach (Giles et al. 1987) and 30 %–37 % in citrus (Moltó et al. 2000). In a similar study, the spray application based on the actual tree width measured by the ultrasonic sensor reduced spray deposits by 70, 28 and 39 % in olive , pear and apple orchards, respectively (Solanelles et al. 2006). Moreover, Zaman et al. (2005) created a real-time variable application of nitrogen fertilizer in citrus, considering the TRV and nitrogen content of leaves; it decreased the costs by 40 %. However, in comparison with LiDAR sensors, the ultrasonic sensors had a lower resolution and poorer accuracies because of sound divergence or distance within the sensors and slower sampling rate (Tumbo et al. 2002).

The implementation of LiDAR sensors enabled the detection of geometric and structural properties of fruit trees with enhanced accuracy, but the sensor devices were more costly. The LiDAR sensors followed the arrangement of the ultrasonic sensor and were mounted on a vehicle and moved along the tree rows to generate 3D point clouds. In earlier studies, the structural properties of apple and citrus trees have been described by simplified geometric equations (Walklate et al. 2002; Lee and Ehsani 2009; Tsoulias et al. 2019a). Polo et al. (2009), after analysing the TRV in 3D, obtained a strong correlation and high coefficients of determination with leaf area in pear trees (R2 = 0.85), apple trees (R2 = 0.81) and a vineyard (R2 = 0.80). In a similar experiment, a logarithmic equation was used to describe the relation between TRV and leaf area density (R2 = 0.89) in pear and apple trees, and vineyards denoting the reciprocal relation between both properties (Sanz et al. 2018). The LAI, which is one of the most widely used indices to characterize grapevine vigour, was well estimated with TAI (tree area index) from LiDAR scanning with R2 = 0.91 (Polo et al. 2009).

Escolà et al. (2017) developed tools for point cloud data analyzes from the LiDAR-based system to extract further geometric and structural information. Meanwhile the estimation of the leaf area (Fig. 8.5a) is state of the art in research (Tsoulias et al. 2019b). Fruit detection and fruit size estimation are currently approached based on geometric and full waveform (intensity) information derived from LiDAR systems (Tsoulias et al. 2020). Particularly LiDAR-based laser scanning seems to be promising for in situ fruit size analysis due to high density 3D point clouds (Fig. 8.5b), which are not affected by varying lighting conditions making segmentations easier. Irrigation treatments showed a positive effect on canopy growth in grapevines measured by LiDAR (Chakraborty et al. 2019). Spatial dependence between soil electrical conductivity and leaf area has been observed in apple production with a terrestrial 2D LiDAR-based laser scanner. The spatial information of LiDAR-estimated leaf area was implemented in a water balance model, revealing variation in water needs in an apple orchard (Tsoulias et al. 2019a) providing a good example of how plant sensor data can be integrated in existing agronomic models, aimed at more precise management.

Fig. 8.5
figure 5

3D point cloud of apple trees (green) and ground (purple) obtained by means of a light detection and ranging (LiDAR) –based system (a); apple fruit segmented from the tree 3D point cloud 120 days after full bloom, fruit size is 69.7 mm (b). Photo courtesy of: Tsoulias

Full size image

8.3 Proximal fruit Sensing

8.3.1 Background

Proximal sensing in fruit production can be approached with gas exchange analysis, which provides information on the growth efficiency, but this has not been automated for monitoring the crop throughout the season. Dendrometers can be employed for measuring growth and shrinkage versus swelling of stem, branch or fruit . Dendrometers can be placed in orchards and data can be continuously recorded with a wireless network (Vougioukas 2013). For real-time analysis of produce after harvest and for grading and monitoring during processing, machine vision systems became commercially available in 1980. Research groups in cooperation with industry developed new sorting lines using spectroscopic methods that became commercially available e.g. 2002 by Greefa, Netherlands. Meanwhile it is possible to classify fruit and vegetables according to their soluble solids content (Walsh et al., 2020), various internal defects (e.g. internal browning, glassiness, stone cracking, bruising, bitter pit), storage reserve level (soluble sugar, starch or oil) and pigment contents (Merzlyak et al., 2003), but attempts have been made to analyze the fruit flesh for firmness only (Lu et al. 2020; Zude-Sasse et al. 2019). Desktop modules and portable instruments for individual product testing became available at the same time based on the same technology, in 2001 by Fantec, Japan, 2002 by CP, Germany and 2004 by Integrated Spectronics, Australia. With the new sensors, the quality of fruit can be assessed in situ at the production site and subsequently followed postharvest. It is precisely this repeated analysis along the supply chain that is essential to optimize the processes for variable-rate application in production and at postharvest.

8.3.2 Maturity

Horticultural maturity or commercial maturity is the stage of development when the produce develops several attributes and characteristics that make them desirable to consumers. Physiological maturity refers to the stage that fruits or vegetables reach the characteristic, fully ripe eating sensation. For certain crops, maturity can be defined chronologically considering the amount of heat energy an organism accumulates over a period of time. Based on the principle that plant organs grow in proportion with ambient temperature, a certain amount of heat energy is required over a period of time for crops to reach maturity. This amount is expressed as growing degree-days (GDD) and is frequently used in practise as the optimal harvest time indicator (Holmes and Robertson 1959). Researchers have experimented with models to estimate crop maturity using meteorological data. Jenni et al. (1998) used heat unit formulae to forecast cantaloupe melon (Cucumis melo var. cantalupensis) yield . The relation between heat energy and time that iceberg lettuce needs to reach maturity has been modelled by Wurr et al. (1988). In fruit trees many models exist based on the GDD.

The stage of maturity at the time of picking influences the storage life and quality of the crop (Rahman et al. 2016). If harvesting is done too early, fruits often fail to ripen on reaching the market, whereas on the other hand harvesting too late will result in fruits softening too early in the supply chain. Consumers demand high quality food and are likely to refuse to buy unripe or overripe fruit . Harvesting time is one of the most important variables that affects final economic benefit. However, deciding on the optimal harvest time is a major problem because specific indices that determine maturity have been developed for only a few fruits and vegetables, while for the vast majority such indices do not exist.

Several studies demonstrate a connection between vegetation indices and fruit maturity, suggesting fruit sensing is a reliable method for maturity assessment. Reduced vegetation growth and shading can lead to changes in sugar content in grapes, suggesting that NDVI or other vegetation indices would be a good tool for quality estimation (Bonilla et al. 2015). Guthrie and Walsh (1997) used temporal satellite data to determine the optimum harvest date for pineapple fields and mango orchards. Early remote sensing applications to identify the optimal date of harvesting of vegetable crops were also reported by Nageswara Rao et al. (2004), who used aerial imagery to quantify colour changes of the visible spectrum related to several horticultural vegetables and fruit ripening processes in the canopy. Mango maturity was analyzed using spectral data recorded with a ground-based vehicle (Wendel et al. 2018). However, as experienced recently in PA applications, the chlorophyll-based NDVI can be influenced by several factors that may limit the robustness of the harvest date model; for example, in citrus production, physiological iron deficiency can cause yellowing of leaves. Furthermore, large within-field variation of fruit maturity was described for most fruit and vegetables, which can indicate the need for selective harvesting in some high value climacteric crops needing to be maintained at high quality in long-term storage. Consequently, the analysis of maturity and quality is requested at the individual fruit level. Mobile applications have been investigated already to provide decision support systems for fruit maturity monitoring (King 2017).

8.3.3 Visible and NIR Bands

With reflectance readings in the visible to infrared wavelength range, changes in pigment profiles, overtones of a molecule’s vibration and rotation, temperature and emission coefficients can be measured. Most commonly silicon-based detectors in the 400–1100 nm range are applied because they are inexpensive and allow easy handling in outdoor conditions. With a light source and a detector, attenuation of the radiation by the fruit tissue can be recorded. Colour data are related to the red (700.0 nm), green (546.1 nm) and blue (435.8 nm) bands, which can be transferred in colour spaces such as the L*a*b* space described in the CIE standards. However, to approach the pigment contents in the visible and functional groups such as –OH in the SWNIR range (700–1100 nm), an enhanced spectral resolution is required. The coinciding absorption of red pigments and chlorophylls can be addressed now with a spectral resolution of 10 nm provided that calibration and interpolation are carried out. For example, the NDVI calculated on two wavelengths or, with enhanced sensitivity, the red edge (nm) calculated by means of the second and third derivative of the full spectrum considering the wavelength at zero in the range 660–690 nm (Zude-Sasse et al. 2002) (Fig. 8.6) can be employed to calculate the chlorophyll at its absorption peak even if visually masked by red pigments. It is even possible to detect the specific chlorophylls out of the total pool of chlorophylls (Seifert et al. 2015). Consequently, the possible perturbation by means of coinciding absorption spectra from the complex matrix of fruit tissue can be better addressed with high spectral resolution.

The wavelength of the molecule under question can be approached by the relevant spectrum of the molecule in solvent, but the binding conditions may change the actual absorption peak in vivo. For chlorophyll, the peak of dissolved chlorophyll, an absorption, appears at 660 nm, while in vivo the peak changes during fruit development, ranging between 682 nm and 686 nm (Seifert et al. 2015). This discrepancy is usually disregarded, but it provides considerable potential for more indicative plant sensing. In summary, changes in the chlorophylls can be detected non-destructively in situ assuming rather isotropic pip fruit such as apples .

In more anisotropic fruit , the changes in scattering coefficients of the sample can perturbate the measurement (Cubeddu et al. 2001; Seifert et al. 2015). Here, fluorescence analysis might be a better option to detect chlorophyll even if the measurement appears less robust than visible spectroscopy (Kuckenberg et al. 2008; Baluja et al. 2012).

An interesting development is that radiometric information can be obtained with ranging sensors that have little information on surface colour. Such data might provide a reasonable approach to measure fruit location and size in the tree (Wang et al. 2017; Tsoulias et al. 2020) (Fig. 8.6).

Fig. 8.6
figure 6

Spectrum of unripe (solid line) and ripe (dotted line) apple measured in optical geometry of diffuse reflection with a halogen lamp, integrating sphere, silicon-based and InGAs detectors. Photograph courtesy of: Zude

Full size image
Fig. 8.7
figure 7

Typical steps involved in data processing of plant stress detector sensors with examples of methodology and necessary reporting of results

Full size image
Fig. 8.8
figure 8

Workflow to develop a calibration model and achieve yield prediction mapping by integrating reference or frequently named ground truth data, critical interfering or beneficial stages, and final validation

Full size image
Fig. 8.9
figure 9

Current situation of the steps of data acquisition by sensors (measuring), deriving plant information with physiological model (deriving information) and agronomic modelling of the production process (application) supported by information and communication technology, when approaching precision horticulture

Full size image

8.4 Data Analyses

Sensors usually provide raw data that cannot be used directly for data analysis. Figure 8.7 shows the typical steps involved in processing data obtained from optical sensors. It includes baseline correction and background removal, dimension reduction and feature extraction, and then estimation by numerical analysis or a classification algorithm. In the estimation of certain plant properties, the adjusted coefficient of determination (R2adj) in calibration and cross-validation, e.g. using the leave-one-out method, needs to be reported. Obviously better than a cross-validation on the same data set coming from the same population of samples, an independent test-set validation on samples of different origin should be carried out. This will result in robust calibrations. In addition to the coefficient of determination, the root mean square error (RMSE) of calibration, cross-validation, and test-set validation are requested (Walsh et al. 2020). Recently, machine learning algorithms have been used extensively for classification for grading fruit inline, and subsequently for spatial or tree-wise information of the planting system. Classification data measured in situ in the planting system provides a 3D reconstruction for detection and monitoring (Zha et al. 2020). The results usually report both false positives and false negatives. In most cases it is desirable to have the minimum number of false negatives because they mean that there are stressed trees that were not detected which could lead to more problems in the future. The work flow can also be visualized considering the ground truth data (Fig. 8.8).

8.5 Yield Estimation

8.5.1 Background

Crop yield is the most important information for crop management in PA. Yield monitors are commercially available for many crops and are used increasingly, while the estimation of yield before harvest is implemented far less in practise. Yield monitor data can be obtained after harvest, whereas some problems such as nutrient deficiencies, water stress or disease occurrence that affect yield , should be managed during the growing season (Usha and Singh 2013). These problems can be solved with continuous data sources that can provide valuable information on crop health and stress , nutrient requirements and, ultimately, yield estimates. The need for accurate yield predictions has become evident since precision viticulture was first adopted, because vineyards have considerable variability in both yield and quality (Bramley 2005). Yield estimation can help the growers to optimize the timing of harvest operations, as well as storage and shipping of their products. However, the difficulty of sampling and lack of efficient methods are obstacles that greatly limit the growth and development of the sector. The traditional method used by wine growers for predicting yield is based on the weight of bunches (Wolpert and Villas 1992), an inefficient and time-consuming operation, which also fails to provide accurate estimates (Clingeleffer et al. 2001).

8.5.2 Weather Data

Quality and yield estimation of various crops is also a function of weather data (Frioni et al. 2017). An early assessment of yield reduction could help to avert a disastrous situation and help in strategic planning to meet market demands. Lobell et al. (2006) developed yield estimation models for 12 crops, including horticultural crops, cultivated in California using data from proximal meteorological stations as inputs. The models showed high accuracy in cross-validation testing for a period of 23 years (1980–2003). Laxmi and Kumar (2011) developed a neural network-based yield forecasting model for various crops that used several meteorological data sources as inputs. Similarly, yield forecasting models based on applications of artificial intelligence with open access to weather data have been developed for various other horticultural crops recently (Kartika et al. 2016). McKeown et al. (2005) found that yield of cool season vegetable crops was inversely proportional to the number of hot days with temperatures >30 °C. In the same year, Koller and Upadhyaya (2005) predicted tomato yield with a more complex model based on soil, crop and environmental variables. While there was no clear correlation with yield , similar yield patterns were observed in the maps. Solar radiation and temperature data were applied by Higashide (2009) for estimating the yield of greenhouse tomato at different growth stages. The results showed a strong correlation between fruit yield and solar radiation. Radiation, in particular the light interception, has been identified as the driving force for the yield in apple production leading to the development of cordon and V-shaped trellis systems (Fig. 8.1).

Analysis of trellis tension continuously measures the tension in the horizontal (cordon) support wire of the trellis and is indirectly related to plant data (Tarara et al., 2004). As a field-based method, it allows the collection of temporal information on the growing crop, but also provides information on changes during the growing stages and in different growing seasons (Blom and Tarara, 2009).

8.5.3 Remote Sensing

Remote sensing applications can provide data throughout the season, and also present an alternative when yield monitor data are not available (Li et al., 2010). Yang and Liu (2008) evaluated aerial photogrammetry and field reflectance data for estimating the physical properties of cabbage, showing that both air-borne images and reflectance spectra can be used to extract reliable information on cabbage plant growth and yield . Applying machine vision to detect and count flowers was a crucial step in the estimation of yield in apple (Aggelopoulou et al., 2011) and mango (Koirala et al., 2020). Smart et al. (1990) described the relation between canopy management and yield in vineyards. Vegetation indices are robust and feasible predictors of vegetative growth (Bonilla et al., 2015) and are strongly related to yield and quality of grapes (Fiorillo et al., 2012) provided that no unfavourable event takes place affecting the generative growth. High correlation can be found between proximal NDVI measurement, yield , and fruit quality in apple (Aggelopoulou et al. 2010; Liakos et al. 2017) and pear orchards (Vatsanidou et al. 2017), if the optimum (Penzel et al. 2020) leaf:fruit ratio is adjusted in crop management. The variable-rate thinning (Penzel et al. 2021) would be an economically interesting approach for including plant sensor data in agronomic models that can be used for precise crop load management.

Much research has been done on the application of satellite data. Nageswara Rao et al. (2004) estimated yield of potato fields as well as other crops with an accuracy exceeding 90 % with satellite imagery-derived NVDI data. O’Connell and Goodwin (2005) used canopy coverage data derived from high resolution aerial imagery of a peach orchard and estimated yield . Shrivastava and Gebelein (2007) found a significant correlation between citrus yields with remotely sensed Landsat images of canopy coverage. Anastasiou (2018) examined several vegetation indices derived from both proximal and satellite sensing to predict yield and quality in table grapes. Although satellite data that can be converted into vegetation indices show a strong correlation to yield , the number of factors affecting the values of indices is believed to be too large to be considered a reliable stand-alone data source by many researchers. Best et al. (2005) combined soil electrical conductivity and NDVI values in multifactorial spatial regression models to generate the yield map of a vineyard . In the same study, selective harvesting of grapes was introduced as a method to improve grape quality , a method that is well-known in tree fruit production. Tagarakis and Käthner also combined soil variables (elevation, electrical conductivity, soil texture) and plant data to delineate management zones based on the correlations with yield and quality in vineyards and plum orchards, respectively (Tagarakis et al. 2013, Käthner et al. 2017). In plum orchards, the ECa was combined with thermal imaging (Käthner et al., 2017).

Considering that the thermodynamic properties of fruit and vegetables differ from those of the surrounding objects, yield forecasts can also be made with thermal imagery data, as demonstrated by Stajnko et al. (2004) and Bulanon et al. (2008). The latter, as well as Wachs et al. (2009), combined thermal and RGB images and managed to increase the accuracy of estimates. Grossetete et al. (2012) introduced an easy-to-use application that allows producers to create yield estimation maps using only their cell phones. Serrano et al. (2005) introduced a handheld system equipped with a camera and a GPS receiver that could make estimates based on image data from the fields. Mango fruits were detected based on colour images, and particularly testing colour spaces, with a high coefficient of determination (Payne et al. 2013). Various data processing techniques were tested on green citrus fruits (Maldonado and Barbosa 2016). In vineyards, on-the-go yield estimation based on photogrammetry has been advanced (Millan et al. 2018) by the reconstruction of fruit bunches (Herrero-Huerta et al. 2015) by structure from motion 3D point clouds.

The spatio-temporal monitoring of structural and geometric plant properties has become more reliable in outdoor applications with the implementation of ranging sensor technology. However, their relation with yield is complex and needs to be investigated further. The use of ultrasonic sensors, again revealed a correlation between yield and canopy volume similar to the vegetation indices studied earlier (Mann et al. 2011). Whitney et al. (2002) and Zaman et al. (2006) used ultrasonic sensors to create vegetation maps and accurate yield estimates in citrus groves. Zaman et al. (2005) and Schumann et al. (2006) found high coefficients of determination (R2 = 0.80 and 0.64, respectively) between canopy volume and fruit yield in commercial citrus plots.

The spatial dependence of height and volume with yield was investigated in three different citrus groves (Colaço et al. 2019). The canopy geometry showed consistent spatial coregionalization with historical yield in two of the three groves, yet the delineation of management zones based on the canopy volume has been proposed. A similar relation has also been identified between canopy volume and yield (R2 = 0.77) in almond trees (Underwood et al. 2016). However, this is not always the case because the fruit load can vary considerably in trees with similar canopy volume, for example because of pollination problems, pruning measures, frost events and alternate bearing (Uribeetxebarria et al. 2019; Penzel et al. 2020). Yield prediction has been attempted from canopy volume and trunk circumference clustering to minimize sampling effort in mango trees (Anderson et al., 2019). However, the frequently reported low coefficient of determination between trunk circumference and yield (R2 = 0.17) suggests that the leaf area represented in canopy volume or vegetation indices provides better predictions than trunk measurements in pruned and well managed production. Furthermore, the tree structure is a useful indicator for identifying the fruiting potential of orchard crops.

Tagarakis et al. (2018), found that the pruning weight is strongly related with yield , suggesting the potential of LiDAR-based systems to assess the spatial variation in vine vigour to regulate site-specific application of inputs (irrigation, fertilization) and to adapt vineyard management operations (i.e. pruning) to improve vine balance and grape quality . Despite the use of LiDAR and photogrammetric data collection methods, it has been suggested that in dwarf apple trees the yield prediction results appear highly variable and difficult to predict simply from structural assessment (Murray et al. 2020). Fruit detection has been reviewed recently (Fu et al. 2020). Stein et al. (2016) developed a multi-view approach with a LiDAR mask to detect, track, count and locate fruit within a mango orchard . Gené-Mola et al. (2019), detected and measured apples in the trees with different LiDAR systems using the ranging and radiometric (full wave form) information. The systems revealed a < 5 % error compared to ground truth, while the authors note that denser canopies could prevent a view of all fruit on the tree (Gené-Mola et al. 2019; Tsoulias et al. 2020). A similar drawback was pointed out in photogrammetry earlier and attempts have been undertaken to estimate weighting factors to link the number of fruit in the visible first layer with the crop load of the entire tree .

Holistic models such as the one proposed by Martínez-Casanovas et al. (2012) or Bonilla et al. (2015) can integrate spectral reflectance data and various other field variables and historical patterns, which might increase the accuracy of predictions in the horticultural sector.

8.6 Harvesting

Selective fruit picking is a financially viable strategy considering the need to maintain quality along the supply chain. However, the harvesting of fruit and vegetables is a time-consuming operation. Mechanization of the process has been desired for decades, but several aspects such as the strong susceptibility of fresh produce to mechanical stress and the inability of robots to separate mature from immature fruits create very challenging conditions. Van Henten et al. (2003) developed an autonomous harvesting robot for cucumbers that could assess maturity in real time and harvest only the cucumbers that were mature enough based on two RGB images. The images were also used to calculate the exact position of the cucumbers and to harvest with considerable accuracy. A robotic system was reported that could instantly detect and assess the maturity of strawberries in real time while harvesting (Hayashi et al., 2005). The robot determined the maturity of the fruit based on ripening index data extracted from RGB images. The group also created a similar system for tomato harvesting, which demonstrated high accuracy of 85 % yield success. A sweet pepper-harvesting robot achieved success rates between 26 and 33 % and a cycle time of 94 seconds for a full harvesting operation in a greenhouse (Bac et al., 2017). A 76.5 % success rate, validated on 68 fruit , was reported recently (Lehnert et al. 2020). Si et al. (2015) developed a robot with a stereo camera system to recognize, locate, and harvest mature apples in tree canopies. The robot successfully recognized over 89.5 % of the crop, while the errors were less than 20 mm when the measuring distances were between 400 and 1500 mm. An apple harvesting robot used a spoon-shaped end-effector to grip and an electric blade to cut the stem of each fruit , reporting an average harvesting time of 15.4 seconds per apple (Zhao et al. 2011). Silwal et al. (2017) designed and evaluated an apple robotic harvester for a V-shaped trellis orchard system with an overall success rate of 84 % and an average picking time of 6.0 seconds per fruit . Still, 13 % of the fruit was missed because the end-effector finger made unplanned contact with obstacles during grasping. Even cherry harvesting has been approached (Amatya et al. 2016). However, the cooperation between engineering and research is ongoing (Fig. 8.3c) and it seems that we are rather close to obtaining a harvest robot. The harvest accounts for roughly 50 % of the entire production costs in horticulture and, due to the high costs and decreased availability of workers, more developments are needed.

8.7 Conclusions for the Chapter

At present, proximal and remote sensors can be used in precision horticulture. However, the implementation of sensor data in agronomic models to control processes directly would be the next logical step. Here we speak about digital twins representing the crop information obtained by employing the crop data in physiological models. Using the digital twins for simulating the process can support the optimization of this process.

However, there is a further need to develop cost-effective sensors that can detect stress of the whole plant and harvest products at an early stage. Currently, optical-based sensors are the most widely used tools for stress detection. In general, optical sensors based on emission, reflectance, and fluorescence are feasible for detecting anomalies in orchards, vegetables production, and berry fields (pre-screening) because they can detect stressed plants and are very cost effective. Some of these sensors can also be used to detect a specific stressor. However, most methods need a re-calibration for a given site or given variety with appropriate reference data that need to be produced by extension services or technical service providers. Another business model would be that the manufacturer remains the owner of the sensor system and sells the service. This would result in easy to access information for the fruit grower as well as big data sets that are interesting for further deep learning analysis. Recent developments in information and communication technologies provide some incentive for implementation in, e.g., yield prediction and harvest. Mobile applications on the smart phone for precise horticulture have been developed based on fruit sensor data (King 2017) to inform stakeholders in real time, but the economic advantage needs to be calculated.

The use of the newly available plant data in agronomic models such as precise yield prediction, variable rate thinning, and precise harvest, represent the challenge for the next step of automation (Fig. 8.9).