Introduction

Irregular flowering and fruiting is a common trait in some important horticultural plants such as apple, pear, plum, olive and citrus (Monselise and Goldschmidt 1982). Undesired climatic factors can result in deflowering or defruiting in the wider fruit growing areas but the year-to-year crop load variation of single trees within the orchard can be induced by technological factors (inappropriate pruning, plant protection or growth regulator management) or tree physiological status (Tromp 2000; Dennis 2000). The annual variation of the bloom density of individual trees in an apple orchard is considerable and can cause large variation of orchard yield (Bukovac et al. 2010; Aggelopoulou et al. 2010). Low fruit quality and the prevention of flower bud formation in the following year can be expected with heavy flower/fruit setting apple trees (Monselise and Goldschmidt 1982; Dennis 2000). In practice, apple-thinning agent spraying, as a measure to reduce fruit set of over-heavily loaded (flowering) trees, is used uniformly in all orchards, without regard to actual flower/fruit set of single trees. Unnecessary use of spray/thinning agents and too strong fruit set reduction can possibly occur if light flower/fruit setting trees are included in the thinning practice in the plantation. Site-specific spraying on an individual tree basis could be used in irregular flower/fruit setting orchards and differential crop load management practice may be optimal for maximizing yield quality and quantity (Manfrini et al. 2009).

However, the physiological response of apple trees to external factors (technological measures) is always specific to individual tree physiological characteristics. Tree-specific management is a necessity for a growth regulator spraying strategy in the orchard; non-flowering or low flowering apple trees should be excluded from thinning management programs while high flower/fruit setting trees should be sprayed with a higher dose of thinning agents than moderate bearing trees. If fruit set and the location of the individual tree in the orchard can be determined, an adjusted growth regulation strategy later in the season can be also managed according to the needs of the individual trees (e.g. spraying of growth retardants or fruit retention agents). Thus, a site-specific plant protection strategy may be adapted to crop load specification of each tree in the orchard.

Sensing of the orchard architecture has been studied extensively to plan horticultural site-specific management practices (Giles et al. 1989; Schumann and Zaman 2004; Jeon et al. 2011). Most researchers have tried to adapt the application rate of plant protection products to single vine or fruit tree canopy dimensions in the plantation (Doruchowski and Holownicki 2000; Gil et al. 2007; Llorens et al. 2010; Jejčič et al. 2011; Hočevar et al. 2010). While machine vision in agricultural systems is yet to reach its full potential, many applications have been developed for various tasks in the fields, orchards, and greenhouses. Visual detection of flowering has been undertaken in a few common crops, particularly those with prominent and color-distinguishable flowers. Vina et al. (2004) demonstrated the use of a spectral index computed from several visible light bands to detect the emergence of tassels in maize. Pimstein et al. (2009) developed a method to detect heads in wheat (Triticium aestivum L.). Image segmentation was used by Thorp and Dierig (2011) to monitor flowering of lesquerella plants. Also for lesquerella, Adamsen et al. (2000) used a digital camera and developed an automatic image segmentation algorithm to estimate flower counts; they managed to estimate flower counts with a coefficient of determination (R 2) better than 0.8. Efforts to estimate flower density on apple trees have also been undertaken to estimate yield; Aggelopoulou et al. (2011) analyzed images of blooming apple trees. Due to multiple unpredictable factors such as weather conditions during the long period between bloom and harvest, yield was not predicted well and the results were also related to a group of 10 trees and not to individual trees. Using conventional sensing methods, Thorp et al. (2011) measured the spectral reflectance of lesquerella canopies with a field spectro-radiometer and estimated flower counts. However, imaging methods offer an advantage over more traditional remote sensing approaches with spectrometers. Imaging methods allow a combined spatial and spectral approach for analyzing crop scenes, whereas spectrometer measurements are point-based.

The aim of the experiment was to automate estimation of number of flower clusters (FC) on an individual tree basis in a modern, mature apple orchard. The estimation consisted of image acquisition by still and industrial color cameras and offline image analysis with HSL thresholding and parameter optimization. Estimated number of FC was compared to manually counted ground truth data.

Materials and methods

Site description and plant material

The research was carried out in an experimental orchard of the Agricultural Institute of Slovenia, located in central Slovenia (latitude 46°10′N, longitude 14°41′E). Mature, eleven-year-old ‘Golden Delicious’ apple trees (Malus × domestica Borkh.) grafted on dwarf M.9 rootstocks were grown in a single row system (3.2 m between rows and 0.8 m on the row, planting density 3 900 trees/ha). Trees were trained as slender spindle (2.5–3.0 m high, 0.7–1.0 m width) and were chosen from one row oriented N–S. 136 non-selected individual trees from the single row were used for analysis. Analysis was performed at full bloom time. Flowering of clusters at full bloom time on ‘Golden Delicious’ are similar to other apple cultivars with 4–6 individual flowers in the cluster open at the same time thus one flowering cluster appeared as a flowering unit. Number of FC were manually counted a day before image acquisition on each of 136 trees. A wide variability in flowering density of the trees was found: 17 trees bore <25 FC, 5 trees 25–49 FC, 10 trees 50–99 FC, and 104 trees >100 FC. Such variability of flowering abundance between the trees is commonly found in many modern, high density apple orchards (Stopar 2010), while some of the trees bore very low number of FC. It should be mentioned that a lot of trees extended their branches into the crown of the neighboring tree.

Each of 136 trees was marked by a wide plastic tag where the exact number of manually counted FC was written. The tags were used to recognize individual trees during image analysis. During measurement, wind velocity was always 0 m/s and humidity was from 85 % to 100 %.

Image acquisition

Blooming trees in the row were photographed by four treatments. The first two treatments were performed during the day (cloudy weather) with a still color camera and an industrial color camera. The third and fourth treatments were performed with the same equipment, but during the night of the same day. Orchard and experimental setup for measurements using an industrial color camera are shown in Fig. 1. No background screen was used behind measured rows to prevent interference with other rows in the background. The reason for omitting the background screen was to increase applicability of the measurement method: the second tractor would be required in orchards with hail protecting nets to tow the background screen.

Fig. 1
figure 1

Experimental setup for image acquisition using an industrial color camera. The image acquisition computer was mounted on the left of the tractor protection cab. The camera was located in front of the tractor and protected by a grey enclosure. A power generator for illumination lamps was attached to the rear

Full size image

Still color camera

The still color camera used was a Nikon D80 (Tokyo, Japan) with 2 592 × 3 872 pixel resolution. Lens used was a Tamron XR 17–50 mm, F/2.8, focus settings were set to automatic. Aperture and shutter selection during the day was undertaken with auto settings. During the night, illumination for the still color camera was provided by a flash unit Nikon SB22, mounted on the camera. The illumination was aperture-corrected such that the image was not too bright; an under-exposure of one aperture stop was used. A similar approach to image acquisition was used for counting of apples by Linker et al. (2012). Photos of each tree were taken from a photographic stand at 2 m distance, 1.7 m height and by using a lens focal length of 17 mm. The photographer was moving along the row and took pictures of each tree such that the tree was in the middle of the image.

Industrial color camera

The industrial color camera was a firewire FLEA2 (Pointgrey Research, Richmond, BC, Canada). The resolution of the camera was 1 024 × 768 pixel, 24 bit RGB. Acquisition settings were set to automatic, except that shutter speed was set to 1 ms. Frame rate was set to 2 Hz. The lens used was a 6 mm megapixel C-mount lens from Vivitar. The aperture was set to F4 during the day and to F1.4 during the night.

Illumination during night treatments was provided by three 500 W halogen lamps operating on a 230 V power supply from a power generator. Images were recorded using Labview software with image acquisition and vision toolkits from National Instruments (Austin, USA).

The industrial color camera was mounted on the tractor at 1.7 m height, attached by a heavy metal support. Orientation of the camera was set using a camera ball head to horizontal. Distance of the sensor from the tree was 1.9 m. The camera was located in front of the tractor and was protected by an enclosure. During the experiment, the tractor transversed along the row at a speed of 1 m/s.

The acquisition computer was mounted on a tractor and acquisition was turned on and off using a wireless connection from another computer.

Analysis

The image analysis procedure is schematically shown in Fig. 2. All image processing was performed offline. The analysis procedure consisted of two parts, image analysis and iterative selection of parameters (Figs. 3, 4).

Fig. 2
figure 2

Image processing algorithm schematics. Optimization processes are in square frames, while processes which did not require optimization are in round frames. Counting of objects rejects too large and too small objects

Full size image
Fig. 3
figure 3

Example of image processing taken during the day with still color camera. a Original image, b image after HSL thresholding, c image after thresholding and rejection of objects outside the tree frame (green) and rejection of too large and too small objects. In this particular case, figures show a tree with 172 FC, while image analysis has identified 117 objects (Color figure online)

Full size image
Fig. 4
figure 4

Example of image processing taken at night with still color camera. a Original image, b image after HSL thresholding, c image after thresholding and rejection of objects outside the tree frame (green) and rejection of too large and too small objects. In this particular case, figures show a tree with 172 FC, while image analysis has identified 172 objects (Color figure online)

Full size image

Images were first loaded from disk to memory. A color image may be encoded in memory as either an RGB or HSL image while, for further analysis, HSL representation was selected. HSL representation is based on hue, saturation and luminance. HSL representation is preferred in comparison to RGB in conditions of changing illumination (Thorp and Dierig 2011). The HSL color space provides more accurate and robust color information for use with color processing functions. The mapping to the HSL image is however much dependent on illumination, in sunny, cloudy or artificial illumination conditions: FC and rest of the tree are both represented by different values of HLS depending on illumination conditions.

The following equations were used to transform images from RGB representation to HSL representation:

$$ L = 0.299R + 0.587G + 0.114B $$
(1)
$$ V_{2} = \sqrt 3 \cdot \left( {G - B} \right) $$
(2)
$$ V_{1} = 2R - G - B $$
(3)
$$ H = \frac{{256 \cdot \tan^{ - 1} \left( {\frac{{v_{2} }}{{v_{1} }}} \right)}}{2\pi } $$
(4)
$$ S = 255\left( {1 - \frac{{3 \cdot \hbox{min} \left( {R,G,B} \right)}}{R + G + B}} \right) $$
(5)

Here R, G and B represent components of red, green and blue value in RGB images and H, S and L represent hue, saturation and luminance components in the HSL images.

For analysis, positions of trees in the images were determined. In images, recorded using the still color camera, trees were positioned in the middle of the image during recording. In the case of images, recorded using the free running industrial color camera, individual trees were not located in the middle of corresponding images, while also several successive images showed the same tree in different locations. Therefore, individual images were manually picked from among the recorded ones, showing an individual tree as nearly as possible in the middle of the image. Then the position of the tree was manually marked in the image. Image analysis was later performed in a rectangular frame around this manually marked position. Frame size was constant and corresponded to the inter-tree distance in a row. The frame is shown in Figs. 3 and 4.

In the second step, HSL images were thresholded to a binary image. A a similar thresholding procedure to that used by Thorp and Dierig (2011) for flower counting in lesquerella was used. The HSL thresholding procedure was such that for each pixel in the input image test was performed with three ranges (hue, saturation and luminance) to determine whether the corresponding pixel from the HSL image under test belonged to the value specified. If a pixel from the input HSL image did not have corresponding pixel values specified in all three ranges, the corresponding pixel in the output binary image was set to 0, otherwise it was set to 1.

During HSL thresholding, the most important criterion is a low saturation (nearly white color). In the thresholded images, objects were counted, while it was assumed, that each FC corresponded to a group of neighboring pixels, classified as objects. However, there were cases where each FC was represented by two or more objects, and there were also cases when FC was not detected, possibly due to occlusion, illumination, reflectance or shadow problems, challenges common to image analysis in severely unconstrained environments such as agricultural applications (Kapach et al. 2012).

After HSL thresholding, objects were rejected that were too large or too small. The parameters used for this were minimum cluster size (pixel) and maximum cluster size (pixel). Among the too large objects rejected were most notably the plastic tags, where the exact numbers of manually counted FC were shown. Too small objects were most often representation of noise.

The thresholding levels from HSL images to a binary image were selected such that the number of FC detected in all images corresponded most closely to the actual counted FC. The algorithm for selecting levels to achieve the highest possible agreement between measured and counted FC was estimated using the coefficient of determination (R 2 ). The highest R 2 represents the best agreement between measured and actual counted FC and is therefore a measure of fit among both data sets. The following equation for R 2 was used

$$ R^{2} = \left( {\frac{{\mathop \sum \nolimits_{i = 1}^{n} \left( {x_{i} - \bar{x}} \right) \cdot \left( {y_{i} - \bar{y}} \right)}}{{\sqrt {\mathop \sum \nolimits_{i = 1}^{n} \left( {x_{i} - \bar{x}} \right)^{2} \cdot \mathop \sum \nolimits_{i = 1}^{n} \left( {y_{i} - \bar{y}} \right)^{2} } }}} \right)^{2} $$
(6)

Here x i is the number of actual counted FC on the tree i and y i are the predicted number of FC on the tree. \( \bar{x} \) and \( \bar{y} \) are average numbers of actual and counted FC on all trees.

Several other image processing algorithms such as morphological filters or image enhancement tools were used, among them erosion and dilation. However, they did not improve analysis accuracy. They were therefore not used.

HSL settings for variable parameters are shown in Table 1.

Table 1 Settings for variable parameters for image analysis
Full size table

Different settings for HSL parameters for the still color and industrial color cameras were required because of different color mapping of each camera (industrial camera produced images of visibly different colors). Settings of HSL parameters are also dependent on the illumination and vary among different treatments.

Results estimation and statistics

Statistical estimation was performed on individual trees, while every tree was represented by 4 images of the four image acquisitions. Three hypothetical management practices (thinning programs) with separate thresholds were selected to run according to the physiological needs of an individual tree, i.e. based on flowering density of each tree as follows:

  1. Threshold 1:

    First threshold was selected at <25 FC per tree. On these low bearing individual trees, the fruitlet thinning practice would result in yield quality and quantity reduction (Stopar 2010). This threshold is aimed at low bearing individual trees.

  2. Threshold 2:

    Second threshold was selected at <50 FC per tree. On these low to medium bearing individual trees, spraying the thinning agents may also result in yield quality and quantity reduction but to a lesser extent than on trees on which threshold 1 may also be applied. This threshold is aimed at medium bearing individual trees.

  3. Threshold 3:

    <100 FC per tree was used as full blooming threshold, because high fruit set is expected. Tree bearing FC >100 are the most frequent individuals in this kind of mature high density apple orchard and thinning them is the necessary management practice.

All thresholds in this experiment were fixed according to multi-year experience of growers, as weather conditions during fruit set are not known during the spraying period. However, if there is rainy or cold weather during the fruit set period, weather conditions may influence the yield and decrease correlation between yield and flower cluster.

The estimated number of FC per tree was used for the hypothetical decision on selective management practice based on the criterion of the above three flowering thresholds. The estimated number of FC per tree was also evaluated against manually counted number of FC per tree. Based on decisions about estimated versus manually counted cases, two types of wrong hypothetical management practice on each tree can be identified:

  1. (i)

    the tree is sprayed but, according to manually counted FC number, spraying is not required (false positive),

  2. (ii)

    the tree is not sprayed but, according to manually counted FC number, spraying is required (false negative).

In the following, both false positive and false negative cases were treated as wrong decisions. Also, results are represented in a confusion matrix of estimated number of FC in four classes of flowering abundance. For example, if the manually counted number of FC per tree is 153, and the estimated number of FC per tree is 68, the tree would be correctly included in the hypothetical management practice when threshold >25 FC or >50 FC is used. The same would be counted as wrong management practice when threshold >100 FC is used, and this tree would be excluded from spraying due to incorrect FC number estimation.

Coefficients of determination (R 2) were analyzed by Statgraphics Centurion XVI program (Manugistics Inc., Rockville, MD, USA).

Results and discussion

Results of measurements and analysis are shown in Table 2 and in Figs. 5 and 6. Figures show comparison of manually counted and estimated number of FC for four treatments.

Table 2 Confusion matrix of estimated number of FC in four classes of flowering abundance with the number of wrong decisions when three hypothetical FC-dependent management practices were considered
Full size table
Fig. 5
figure 5

Manually counted and estimated number of FC, day measurements

Full size image
Fig. 6
figure 6

Manually counted and estimated number of FC, night measurements

Full size image

Management practice was estimated for all four treatments and showed comparable performance as shown in confusion matrix in Table 2. Among all four treatments, best performance was achieved with the still camera during the day at >25 or >50 FC criterion (thresholds 1 and 2) with 8.8 % false decisions, and with the still camera during the night at >25, >50 and >100 FC threshold with 9.5, 5.1 and 9.5 % false decisions. The reason for the poor performance of the still camera used through the day with >100 FC threshold (wrong decisions 23.5 %) is unknown, but may be due to a) the superior image resolution and quality of the still camera in comparison to the industrial camera during the daytime or b) due to light reflection on leaves. Superior image resolution may lead to increased detection of FC in the background rows during daytime which were included in analysis. Illumination during a sunny day introduces additional sources of uncertainty to image acquisition and processing. Cuticle wax composition on apple leaf surface may reflect in the direct sunshine and such leaves appear in the image as white spots with similar HSL characteristics to a flower cluster. To avoid this, daily measurement was performed in mostly cloudy weather; however irrespective of this, a high value of percentage of wrong decisions in Table 2 (still camera, daytime) may still be caused by such reflections. The industrial camera showed 10.3–14.0 % wrong decisions during the day at all three thresholds and, at night, 11.8–15.4 % wrong decisions.

Results from Table 2 show that the coefficients of determination (R 2) were higher than 0.50 for three treatments. In these cases, more than half of the variability was explained when correlations between real and estimated FC number were calculated in the case of the industrial camera (day and night) and the still camera (night).

When analyzing the raw data from 136 trees, it was found that both still and industrial color cameras during the day or night overestimated the manual counted FC number on low bearing trees (<100 FC, Table 2), as seen also in Figs. 5 and 6. Large over-estimation (up to 2–4 fold) was found frequently on trees with FC number <40 FC. The reason could be the high density apple orchard planting system where individual mature trees overlap into the neighboring crown with some branches. The effect of wrong FC estimation on a selected tree is even more emphasized if the neighboring apple tree interferes with branches blooming on one-year-old wood where a high number of FC is present. Stajnko et al. (2009) attained 89 % detection accuracy by a multi-stage image analysis algorithm on presumably more complicated (green) apple fruit counting in the crown, but the trees were small, had light crop load, and did not overlap. Another important reason for disagreement between number of manually counted and estimated FC is that, in this experiment, a background screen was not used behind the measured row. Such a screen has been used by many authors for improvement of object detection in orchards. The background screen behind a measured row eliminates interference with other rows in the background. This reason is however limited only to image acquisition during the day. During the night, illumination is only powerful enough for measuring the front row while rows in the background are poorly illuminated and interfering objects are not detected. Another reason for inaccurate FC number estimation might also be non-perpendicular position from which images were acquired. In that case, some FC may protrude into the frame from both neighboring trees on the left and right.

From Figs 5 and 6, it can be found, that in all cases the number of FC was underestimated on trees, which bear more than 200 FC. For this, two most important reasons can be determined, the first is screening by other parts of trees, while the second is that these trees may extend their branches out of the measurement frame (as seen in Figs. 3, 4).

As a result of variability of flowering abundance, Figs. 5 and 6 show that many trees bore less than 15 FC. For these trees during night measurements using both industrial color camera and still color camera, estimated number of FC was overestimated. For these trees during day measurements, estimated number of FC was also overestimated and in addition variability of estimation of FC was high.

Without regard to the percent of successful decisions between different types of image acquisitions, the only possibility to perform automated management practice, based on number of detected FC, is to use an industrial camera for image acquisition. Moreover, by using an industrial camera, only a few percent worse performance was achieved as compared to time consuming still camera photography.

In the case of this experiment, 24 % of trees bore less than 100 FC per tree. On some trees, only a few or even no FC were manually counted. Taking into consideration that more than 100 FC on the tree would mostly satisfy the criterion for fruit thinning as the most important growth regulator practice in the apple orchard, 24 % of trees will be managed incorrectly by a common overall spraying program. Further, far-reaching negative consequences on the next year flower set will be expected. In many apple orchards with alternate bearing behavior, more than 50 % of insufficiently flowering trees can be found in the off-years (Monselise and Goldschmidt 1982). The proposed tree-specific management practice gives 10.3 % of incorrectly sprayed trees (based on an industrial camera used in the day), and such practice is an important improvement over common overall spraying with 24 % of incorrectly sprayed trees. The orchard used in this experiment has 3 900 trees/ha and within is among the most dense apple orchards in Europe. Nowadays, most common orchards have about 2 500–3 500 trees/ha and trees overlap less.

Conclusions

Alternate bearing trees in apple orchards may benefit from a tree-specific management practice, especially when spraying growth regulators. An automated tree FC measurement system is required to design a flowering abundance-adapted spraying system operating on an individual tree basis. In this experiment, estimation of FC number of individual trees from a very high density apple orchard was undertaken. From images acquired by a tractor-mounted industrial camera, FC number of individual trees was estimated with almost the same success rate as by using slow still photography. Overlapping of branches from neighboring trees and influence of tree rows in the background were the main reason for estimation uncertainty. By using still and industrial color cameras, around 10 % wrong decisions were made when using a spraying threshold of 100 FC per tree as the on/off criterion for tree-specific management practice. Improved FC detection and better tree-specific management practice can be expected on more common apple orchards with a density of 2 500–3 500 trees/ha. Two or three years of tree specific management practice would decrease alternate bearing habit of treated trees leading to better bearing properties of orchards and increase of yields.

The most important limitation of the method is their dependence on illumination during estimation of number of FC. In the present study, the image processing parameters have been tuned according to the results obtained with respect to manual counting; obviously this procedure could not be used in real operation. The robustness of color thresholding parameters is low and the method performs better at night than with variable daylight: for real field use, methods that are not based on color thresholding should be developed.