Keywords

1 Introduction

Plant phenotyping is the process of completely assessing the basic and complex characteristics of the plant. The plants complex characteristics include its growth development and yield [1]. Some of the basic characteristics observed when phenotyping plants, specifically rice crops, are its height and tiller count. Plant height is related to the productivity and growth rate of a plant. Plants tend to grow to a certain height in each of its growth state [2]. However, plants drop in growth rate when plants have diseases or lack in water, which results to lower yield rates. In rice crops, more tillers would generally mean more yield [3]. Tillers are the grain-bearing part of rice crops and it possesses the leaves [4]. Leaf count indicates a plant’s age and as it grows, its leaves will develop and grow in size [5]. Plants growing in good conditions develop leaves in a faster rate than those leaves growing in non-conducive environment for the plant.

1.1 Rice Crop Tillers & Panicle

This research focuses on phenotyping of rice crops. The two key part of a rice crop that is useful to this research are its tiller and panicle shown in Fig. 1.

Fig. 1
figure 1

Tiller of a rice crop [6]

Full size image

Tillers in rice crops appear as soon the rice crop is selfsupporting [7]. The first tiller usually emerges when the seedling has five leaves. This signifies the start of the tillering stage. From the main stem, another stem would develop. This new stem would be the tiller. Once these tillers are fully developed, they would develop flowers. These flowers are more commonly called as panicles. Tillers are used to measure the height of the rice crop. The height of the tiller with the longest leaf would be the height of the plant.

Panicles are the flowers of the rice. They are placed at the end of the tillers. The seeds or grains produced in the fertilization of the rice plant grow in the panicles of the rice crops. Panicles are one of the identifying quality of a fully developed tiller.

According to Rousseau et al. [8], phenotyping of these traits requires properly tuned rating scales and well trained raters which adds to the cost and time. Furthermore, it is influenced by the subjectivity of raters making it lack in accuracy, reproducibility and traceability [8]. Fatigue of experienced staff can also be one cause of the degradation of accuracy and efficiency [9]. This is the current situation in the International Rice Research Institute (IRRI).

Automated image processing of the phenotyping of plants offers an effective substitute to manual visual assessment [8]. It also allows a regulated image analysis that can be reproduced because of the absence of subjectivity of manual visual assessment and it enables a high throughput because of the automation of the process. Moreover, calibrated protocols and data storage offer beneficial tools to trace or compare results [8].

1.2 Automated Plant Phenotyping

There are a number of researches and systems that has been done that uses image processing to automately measure height of plants and count its tillers. They used different tools to measure the height of an object and number of tillers of plants while applying image processing techniques.

Sritarapipat et al. [2] used marker bars and a formula where the values are taken from the processed image to get the height of the rice crops. They used band selection, filtering, and thresholding to remove noises affected by the rain, wind, and outdoor light, in the images. They took images of rice paddies in a very low plain called Tha Cheen River, a part of a rice field in Suphanburi, Thailand. These images were taken by two Single-Lens Reflex (SLR) cameras controlled by the field servers control unit. However, there are a lot of images being taken and storing these images in high resolution will take up too much storage space. Therefore, they took images in lower resolution at 10:30 a.m. daily to resolve the issues with regards to storage space and data transfer rate. Local Thai Meteorological Department staff recommended that images be taken in the morning as clouds and precipitation has less effect than other times of the day.

According to Sritarapipat et al. [2], the results were compared to the data of manually-measured rice crops. They were able to conclude that excess green had better results than red band. However, they also state that results are dependent on the marker used. Results might also change if there are plants present in the field image that are similar to rice crops, like wheat. The authors wrote that it will be identified as a rice crop. The system is limited by having only one marker bar and it should be in the same place always. Having multiple marker bars in one field image and bars varying in distances was not tested.

A system created by Ikiz [10] uses image processing techniques, like skeletonisation, to measure fiber length. The system would be depended on these factors: sample preparation, lighting technique, resolution, preprocessing algorithm, and processing algorithm. There were two levels of sample preparation that Ikiz [10] did, the one with fiber crossovers and one without. There were also two types of lighting, frontlighting and backlighting to create negative images since the only concern is to know the pixels of the fiber and the background. For preprocessing, both images that were assumed to contain single fiber and images that were assumed to have random crossovers, outline, thinning, and adding algorithms were applied.

The system that Ikiz [10] created achieved a 0.65 mm confidence level, higher than the 0.5 mm that is required. However, when running low resolution images through the adding algorithm, it sometimes results to negative bias. Added noises in the image, like dust or glare, causes points to disconnect resulting to shorter measurements.

Yang et al. [9] created the H-SMART system that performs x-ray computed tomography (CT) to automatically count the number of tillers in rice plant. Pots of plants pass through the x-ray system through a conveyor belt and stops at a rotation platform. There is a distance of 1122 mm between the focal spot and the center of rotation. All computations are done in 20 s and image processing in 5 s. Filtered Blackprojection (FBP) algorithm was used to create an image of the rice culms. Image processing methods, such as median filtering for image denoising, threshold operator for image segmentation, and some morphological operators are used to identify tillers in the FBP image. The researchers used three batches of rice, 50 pots each, from the tillering, heading, and flowering stages were taken and used to test the accuracy of the system. The system was able to have a 95% confidence in accuracy.

The systems, although well-advanced, are still noticeably young and can still be improved. Systems for measuring height are more common to researchers and many have tried different approaches and techniques but there are still a few things that can be improved. The use of marker bars and rulers, done by Sritarapipat et al. [2] and Lee et al. [11], with image processing techniques seems to be two of the most effective approaches. This is the same with the systems that count tillers, although there are fewer studies regarding this.

1.3 Importance of the Research

The problems presented by manual visual phenotyping, such as the additional cost and time due to the requirement of properly tuned rating scales and well trained raters which may be subjective and experiencing fatigue can cause the lack in accuracy, reproducibility and traceability [8] as well as the efficiency, led to the automation of phenotyping through image processing. It also speeds up the process of phenotyping which is a big factor for researchers. Rice crops, in particular, would benefit in the automation because of the growing population of the world, the demand for food increases [2].

Currently, in the Philippines, they are still using manual visual phenotyping and are experiencing the problems stated. The researchers proposes an image processing system, named Seight, that automates the measuring of height and the counting of tillers of a rice crop because although it’s a fairly young field and many previous studies used image processing techniques for their systems, there are new and unused image processing techniques that can be used. Also, previous studies do not consider noises present in the image and its effect to the result. An appropriate infrastructure setup would be designed for the automated phenotyping. Seight would only focus in height measurement and tiller count.

2 Methodology

2.1 System Design

To develop the system, the researchers came up with a system design with 6 modules: Data Capturing Module, Data Management Module, Preprocessing Module, Height Module, and Tiller Module shown in Fig. 2.

Fig. 2
figure 2

System architecture

Full size image
  1. (1)

    Data Modules: The Data Capturing Module is where images of C4 rice crops gathered for this research. These are currently cultivated in the International Rice Research Institute (IRRI) in screenhouse. In Fig. 3, it shows the setup used to capture the images. The distance of the camera from the board was also adjusted to 104.5 cm and the height to 53 cm. The angle of the camera was lowered to 5° and a colored card, which will be used as a marker, was placed at the top of the board. The camera used was a Canon 60D with an image size of 1296 × 3110 pixels. The Data Management Module handles the plant images from the Data Capturing Module. The Data Management Module includes the database of the system and the settings to change the schedule of image capture of the plants. The user interface for this module is in a form of a website.

    Fig. 3
    figure 3

    Camera setup

    Full size image
  2. (2)

    Preprocessing Module: The Preprocessing Module prepares the raw images for analysis. Background removal, filtering and edge detection are the techniques that would be used for preprocessing. Background removal and filtering fall under noise reduction. These techniques would, respectively, lessen the noise and unnecessary details in the image. Edge detection is a subfunction of plant segmentation. Plant segmentation would segment the plants present in the image from each other. This allows the system to focus more on assessing a single plant which in turn increases in the efficiency of the assessment.

  3. (3)

    Tiller Module: The Tiller Module determines the tiller of the plant. It is in this module that the tillers are counted. Edge detection and getting of the Region of Interest (ROI) were applied to the preprocessed image before counting the tillers.

  4. (4)

    Height Module: The Height Module is responsible for determining the height of the plant in the image. Skeletonization and tracing were applied to the preprocessed image to get the height.

3 Seight

Four experiments were conducted using different setups and different combinations of algorithms.

3.1 Experiment 1

The first experiment was done as a part of the concept formulation of the methodology cycle. In this experiment, the design of the board and the specifications for the camera setup was formulated. This implementation of the designs formulated was not done inside the screenhouse of IRRI. The lemongrass and citronella plant were used as substitutes to the C4 rice plant.

Three boards, plain white background, white background with black stripes, and black background with white stripes, were used as part of the experiment for the board design. The board that is most appropriate for the study is the board with the plain white background. The striped board design was deemed unnecessary due to the new information gathered about height measurement. Initially, height was to be measured by taking the highest part of the plant and computing the height from that to a parallel point in the plant base row. However, since the method changed upon confirmation, the striped design became unnecessary.

The board’s measurement were 76.2 cm by 101.6 cm. It is elevated from the ground at 20.32 cm, which is the height of the pot. The camera used for this experiment is Nikon Coolpix L21 digital camera that has an image resolution of 640 × 480 pixels. It was placed perpendicularly to the boards center with a distance of 200–300 cm. It was elevated 50–100 cm from the flat ground using a tripod.

3.2 Experiment 2

In the second experiment, the camera and board setup from the first experiment was followed in this experiment. A meter stick was placed on the left side of the board to aid in height measurement. The images for this experiment were taken using Nikon D40 with an image resolution of 2000 × 3008 pixels. This was done in the IRRI screenhouse.

The raw image of the plant was processed to isolate the rice plant from the background through thresholding in the HSV color space. Using the OpenCV function inRange() a minimum HSV color value and a maximum HSV color value was taken to determine which pixels to get in the image. The minimum HSV values were 0, 45 and 10 for hue, saturation and value, respectively. The maximum HSV values were 135, 255, 100.

To get the height measurement of the plant, its form is identified using the skeletonization technique, Zhang-Suen’s algorithm. The system gets the highest and the lowest pixels in the image by iterating the image starting from the top left corner until the bottom right corner, moving to the right downwards. It takes the first white pixel it sees as the highest pixel. The same process was done to get the lowest pixel but the system starts at the bottom-right of the image to the topleft, moving left, upwards.

The Euclidean distance of the highest and lowest pixels would be identified. The Euclidean distance was converted from pixels to centimeters. The conversion from pixel to centimeters was based on the meter stick placed within the image. The pixels in a centimeter with reference to the meter stick are counted. This is then assigned as the conversion factor which is used to divide the height in pixels. The height of the plant was the result of the conversion.

The tiller count of the plant was computed by getting the edges of the plant in the image by applying Canny edge detection algorithm [12]. From the edge detected image and the preprocessed image, two regions of interest was created to focus on the lower part of the image. This region contains the section of the rice crop where the tillers are most visible. With the coordinates 0, 0 in the image is at the topmost-left, the starting coordinates of the region of interest is at 160, 515 with a width and height of 106 pixels.

The x variable was fixed at the middle row of pixels in the image and only the middle row of the regions of interest was tested. The tillers are identified by iterating through the middle row of pixels. Two regions of interest was used to count the tillers; the edge-detected image region of interest and the preprocessed image region of interest. When the system detected an edge on the edge-detected image, it looks at the neighboring pixel of the region of interest in the preprocessed image. If the neighboring pixel is not black, the tiller count was incremented.

3.3 Experiment 3

In the third experiment, the board and camera setup was modified to accommodate the actual setup of the plants inside IRRI’s screenhouse. A separator was added in the board design to separate the two rice plants that was captured in the image. The new camera setups new height from the ground to the lens of the camera is 21.59 cm, the distance from the plant to the stand of the camera is 36.5 cm with an 8° vertical angle and a 0° degree horizontal angle. The camera used in this experiment was D-Link DCS-932L, an IP camera, with a 640 × 480 pixels resolution.

Due to the change in the quality of images, there was a need to add color balancing to the preprocessing module. The balancing of the images were done through equalization of the images histogram. The thresholding technique was also modified. In this experiment, there were two color spaces observed, HSV and YCrCb. Combinations of the equalization and thresholding in this color spaces were assessed to see what yielded the better result. The combination that yielded the better result was HSV for the equalization and YCrCb on thresholding. Table 1 show the values used for equalization and thresholding for this experiment.

Table 1 Table of values used for equalization and thresholding
Full size table

The skeletonization algorithm used for this experiment was still the Zhang-Suen algorithm. The method of getting the height was getting the highest point in the image and the base as the lowest point. It uses converted Euclidean distance as the height. The reference used to get the conversion factor is the frame of the separation board which has a width of 2.54 cm (1 inch). The pixels within that area is counted and used to divide the computed distance resulting to the height measurement of the plant in centimeters.

The algorithm for counting the tiller still remains the same. However, in this experiment, all rows of pixels were considered for the counting. The mode of the tiller count of the rows of pixels were considered by the system as the tiller count. The region of interest was changed to accommodate the change in resolution of the images. With the coordinates 0,0 in the image is at the topmost-left, the starting coordinates of the region of interest would be at 23, 380 with a width of 170 pixels and a height of 75 pixels.

3.4 Experiment 4

In the fourth experiment, the board and camera setup from the third experiment was followed in this experiment. However, additional side panels were added as walls to make the background of the plants similar on both sides. Figure 4 shows the schematics of the added design. As shown in the said figure, the side panels are 38.1 cm wide and 101.6 cm long.

Fig. 4
figure 4

Third board setup

Full size image

The distance of the camera from the board was also adjusted to 104.5 cm and the height to 53 cm. The angle of the camera was lowered to 5° and a colored card, which will be used as a marker, was placed at the top of the board. Each image contains two pots of rice crops against the white board. These were taken using a Canon 60D with an image size of 1296 × 3110 pixels. A total of 28 images of rice plants in pots were taken.

Since the raw images taken include the background behind the marker board, the images were cropped. A quarter of the image was removed from each side. A tenth portion was removed from the bottom of the image to cut out the pot. Then, the resulting image was split in half to separate the two plants in the image. The images of individual plants were then normalized using the OpenCV function normalize(). Afterwards, the brightness were adjusted by +5 and the contrast was increased by 70% to make the plant more perceivable in the image. The plant was segmented from its background using HSV segmentation. The minimum HSV color values were 15, 35 and 40. The maximum color values were 70, 250, 240.

The height still used Zhang-Suen algorithm to skeletonize the image. After getting the skeleton, the system looked for the highest white-valued pixel. From there, it traced the tiller to the base. The neighboring pixels of the current pixel is checked to get the closest white pixel. The pixel which is directly below the current pixel is always checked first. The algorithm was also designed to address disconnecting lines in the image due to the skeletonization process done in the image. This is done by increasing the range the algorithm looks at. It can increase from checking 1 pixel up to checking 1% of the images height.

Euclidean distance was still used to get the height in pixels. It would be converted to centimeters using a conversion factor. A marker was used to identify the conversion factor to be used for converting the pixels to centimeters. A red marker with 4.7625 cm (1.875 inches) height and width was used. This was a quarter of a 3.75 inched note paper. This makes the marker have a height and width of 1.875 inches and was then converted into centimeters. It is placed on the top portion of the marker board. It would be detected in the image through HSV segmentation. Canny edge detection would be applied to the image from the segmentation. The largest contour was identified as the marker. The system gets the height of the contour. The height of the contour became the conversion factor.

The tiller module still used Canny edge detection algorithm and the algorithm used in counting the tillers in the previous experiment. In this experiment, the computation of the region of interest became dynamic. The starting y-value of the region of interest is the bottom one-eighth of the image, assuming that the tillers are in the bottom portion of it. The starting x-value of the region of interest is the middle column of the image subtracted with one-sixth of the width of the image. The width of the region of interest is one-third of the images width, while the height of the region of interest is one-eighth of the images height.

4 Results

For the fourth experiment, HSV was used for both preprocessing and segmentation of the raw images.

For the tiller count, it yielded a 76.14% average percentage error. It is because the bottom part of the images contains a significant amount of noise. A lot of unnecessary edges are detected, resulting in a higher tiller count than the actual count. Also, in the new set of images gathered, there were a lot of low hanging leaves, adding to the noise. That is why in Fig. 5, there are more instances where system counted more than the view count.view count. The view count is the number of tillers counted manually base from the visible tillers in the image.

Fig. 5
figure 5

Tiller scatter plot

Full size image

For the height measurement, there are three major metrics that were used to analyze the results. The first one is the tracing length error which allows the evaluation of the tracing algorithm in the system separately from the conversion algorithm. It is computed by comparing the length of the traced tiller of the tracing algorithm in the system using raw images and the length of the manually traced tiller using an image which was generated by manually tracing the tiller with an image-editing software which resulted in a binary image with one traced tiller of one pixel width. The length of the manually traced tiller is considered as the measured value and the length of the system traced tiller is considered as the computed value when computing for the relative percentage error. The tracing relative percentage error is 22.5% with a standard deviation of 26 pixels that was rounded down from 26.15 pixels. Those with low trace length error have longer lengths than the measurements with the manually traced images while those with high error have short lengths.

In Fig. 6, the histogram shows the distribution of the relative percentage error, the standard deviation is a measure how spread out the numbers are. There is also a high frequency of relative percentage error that is from 0 to 25%. Some of the plants have high percentage errors because the separation board was detected as a tiller. There are also big gaps in the skeleton that stopped the tracing algorithm. The small branches in the skeleton could be improved by adding a backtracking method in the tracing algorithm.

Fig. 6
figure 6

Tracing length relative percentage error distribution

Full size image

The next metric is the conversion error which is used to evaluate the conversion algorithm independently from the tracing algorithm. The length of the manually traced tiller is converted to centimeters using the conversion algorithm and it is compared to the actual length of the plant in centimeters. The actual length of the plant was taken by the IRRI researchers by using a meter stick to measure the length of the tallest tiller in the image. The manually traced tiller length was used because it is the ideal tracing of the tiller and with it the results will only be influenced by the conversion algorithm or an error in the actual physical measurement of the plant. The mean of the relative percentage error of the conversion is 230.73% with a standard deviation of 314.60 cm.

There are some are very high errors that reaches almost up to 1250% error. It is because it detects the noise around instead of the marker. Furthermore, as seen in Fig. 7, those that resulted in a low conversion error has a conversion factor of ranging from 30 to 35 of pixels per centimeter based from the marker and as the conversion factor decreases, the conversion error increases.

Fig. 7
figure 7

Conversion factor and conversion error

Full size image

The final metric for the height measurement is the overall relative percentage error. It is computed by comparing the actual physically measured length of the plant and the system traced and converted length using the raw images. The mean of the overall relative percentage error is 238.11% with a standard deviation of 345.93 cm. The distribution of the overall error is positively skewed as seen in Fig. 8. Those that resulted in a very low conversion factor had very high conversion errors which in turn affected the overall percentage error.

Fig. 8
figure 8

Overall relative percentage error distribution

Full size image

5 Summary & Conclusion

Plant phenotyping is important in studying the growth and yield of the crops. The automation of plant phenotyping is essential in making it produce high throughputs and outputting results that are more accurate and reproducible. Seight is a system developed to automatically phenotype C4 rice crops. C4 rice is a rice that is being enhanced to increase yield. The system measures the height and counts the tiller of a rice crop through different image processing techniques. The plant is segmented from the image with HSV segmentation and thresholding. The height was measured by getting the highest pixel from the skeletonized image, identifying points in between until it reaches the base of the plant. It then gets the sum of the distances between points and converted from pixel to cm. The tillers were counted by setting an ROI in the image and all pixels within the ROI is checked. The tiller count is the mode of all the counted tillers for every row.

The second experiment has an average percentage error of the tiller count is 34.02 and 17.25% for the height measurement. For the third experiment, the combination with the lowest average percentage error of 33.47% was the HSV for preprocessing with YCrCb colorspace. On the other hand, the fourth experiment was able to get an average percentage error of 76.14% for the tiller count and 238.11% for the height measurement.

In conclusion, although the system is still lacking in accuracy, it was able to automatically count the number of tillers and measure the height of the plant, which is the main objective of the system. The fourth experiment was the best in terms of algorithms used for both the height measurement and tiller count.