Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Observations of the plant Mimosa pudica under natural/wild conditions reveal that the plant opens its leaves prior to sun rise and closes its leaves prior to sun set, i.e. anticipating the shift in illuminative conditions. These responses are described as being related to the plants circadian rhythm, which is a periodic, self-sustaining and temperature compensated biological process [2, 7, 9,10,11].

We wanted to investigate in controlled conditions how this behaviour is related to the day length, to the dark/light ratio and to phase shifts in the entraining dark/light cycle. Our overall goal in the project is to describe the observed behaviour in mathematical terms using non-linear differential equations, and relating these to mathematical models of circadian rhythms. By investigating the influence of different light/dark cycles and constant illumination we seek to get valuable insights into the regulatory mechanisms behind circadian rhythms. The work presented in this paper, however, is related to the image processing tool used to analyse the time lapse images of Mimosa pudica in order to get a quantitative measure of leaf movements, and is based on the master’s thesis of the first author [1].

Mimosa pudica originates from the tropical parts of America. It is a perennial plant (can live several years) and it is usually between 15–45 cm tall. It is also known as the sensitive plant as it closes its leaves upon touching. The young plants will usually consist of two to four compound leaves which again contain between 7 and 10 pairs of leaflets arranged in a bi-pinnate (feather like) pattern, see Fig. 1(a). These leaflets are observed to be in the color range of dark greenish to yellowish with an oblong shape. One of the favorable properties of Mimosa pudica from an experimental viewpoint is its rapid response to a stimulus. These responses originate from the plant’s different pulvini which can be described as motor organs that cause movement by increasing or decreasing the turgor pressure within the pulvini cells [14]. The location of the three different pulvini are shown in Fig. 1, i.e. (i) the primary pulvini at the joint of the petiole and stem, (ii) the secondary pulvini located in the joints between the petiole and rachis, and (iii) the tertiary pulvini found at the base of each leaflet.

Fig. 1.
figure 1

Mimosa pudica stem and leaf description [13]. The drawings correspond to light (panel a) and dark (panel b) conditions. We have added colored lines to indicate the typical movements caused by the different pulvini, red lines correspond to primary pulvini, green to secondary pulvini and blue to tertiary pulvini. (Color figure online)

Full size image

The different pulvini have also been proven to respond differently [4], where, as an example, the primary and tertiary pulvini responds to touch or vibration, while the secondary pulvini do not. The most apparent difference between the leaves at light and dark conditions is caused by the tertiary pulvini which cause the folding/unfolding of each leaflet. Our aim is therefore to capture the motion caused by these cells.

2 Experimental Setup

Each experiment is run for several weeks where the light/dark stimulus is either a combination of several different periodic patterns, or is kept constant for several days. The plant movement will in this period be recorded as a 5 min interval time-lapse series by a Logitech C615 full HD 1080 p web-camera. The camera is installed vertically above the plant in a sealed container with light emitting diode (LED) strips mounted on each of the four walls, see Fig. 2(a).

Fig. 2.
figure 2

(Panel a): The experimental setup where the plant is placed at the bottom of a container with white walls and white lid resulting in good light distribution conditions. There are 4 LED strips to provide light, and one single LED strip for dark conditions to allow monitoring of leaf movements. Each LED strip consists of 9 LEDs with a total power consumption of 0.7 W. (Panel b): Picture taken by the Logitech camera mounted above the plant. The leaf is resting on the surfare of a plastic board, and the petiole is loosely strapped to avoid the vertical movement caused by primary pulvini.

Full size image

The reason for using LED as a light source is its low heat dissipation, resulting in neglectable temperature increase. In order to adjust the amount of light at dark condition, a single LED strip can be moved back and forth inside a 12 mm gray PVC pipe. Usually dark conditions are made up of 3–4 singel LEDs.

As shown in Fig. 1, the primary and secondary pulvini create motions in the petiole and rachis, respectively. For this reason it is therefore more or less impossible to isolate the folding/unfolding motions caused by the tertiary pulvini cells using a single camera mounted above the plant. However, by strapping a single leaf on a horizontally mounted plastic board enables us to isolate motions caused by the secondary and tertiary pulvini cells, see Fig. 2(b). The light/dark conditions used in this paper is 8 h of darkness and 16 h of light, giving a circadian rhythm of 24 h.

3 Image Processing and Motion Estimation

3.1 Color Model

As the images are captured under both light and dark conditions where leaf movements cause changes in local shading and where the leaves have varying color and shape, the choice of the color model will certainly affect the image processing. We therefore evaluated and compared the individual components of each of the following three color models with respect to the effect of segmenting the leaves from the background (individual components in parenthesis):

  1. (i)

    RGB (red, green and blue), standard color representation.

  2. (ii)

    HSV (hue, saturation and value), decoupling intensity and color information.

  3. (iii)

    CIELAB (L \(^*\), a \(^*\), b \(^*\), which corresponds to lightness, green-red colors, and blue-yellow colors, respectively), a uniform color opponent space.

Furthermore, as a part of the evaluation we also investigated the effect of using a colored background. From the initial results with a white background, we observed that most of the hue component of the plant was gathered in the yellow/green area. Hence, by using a blue colored background, the HSV and CIELAB color models gave a large distance between the background and foreground colors. Therefore, the evaluation is based on using a blue colored background, and the best individual component of each color model was

  1. (i)

    the blue component from RGB

  2. (ii)

    the hue component from HSV, and

  3. (iii)

    the b* component from CIELAB.

A comparison of these three representations are shown Fig. 3, where two pictures (one at dark and one at light conditions) are used as test pictures. As we see, the blue RGB component is most affected by changes in the illumination and is therefore discarded. On the other hand, both hue from HSV and b* from CIELAB showed promising results, and there is no reason for choosing one over the other. However, based on the more intuitive representation of the hue component, the HSV color model was chosen.

Fig. 3.
figure 3

The dark regime (top row) and the light regime (bottom row) used in the color space comparison (Color figure online)

Full size image

3.2 Image Segmentation

One of the challenges with image segmentation is the lack of a universal method [8], and the method must therefore be chosen based on the problem to be solved. Based on the observation that the blue and green colors are well distinguished using the HSV color model, we found that an intensity-based segmentation method localizing a threshold value from the hue component histogram was suitable. The hue component of the image pixels \(\in [0^\circ \ 360^\circ ]\) is assigned into 100 histogram bins, and the almost uniform blueish background will be seen as a narrow symmetric uni-modal distribution in the histogram. This suggests that a suitable intensity threshold can be found by implementing a triangle based method [15]. This method is based on locating (i) the first non-zero bin and (ii) the largest bin by iterating through the histogram data. Their location is then used to parameter fit a straight line between the two bins, and the most suitable pixel intensity threshold is assumed to be the histogram bin with the largest distance to this line [15]. Compared to the manually segmented pictures, the triangle threshold segmentation performs pretty well as shown in Fig. 4. Some background pixels are though included in the foreground where the spacing between leaflets is small. An objective evaluation of the segmentation can be done using the Dice similarity coefficient as a measure, where the quotient of similarity (QS) is found as

$$\begin{aligned} QS = \frac{2 |X\cap Y| }{|X| + |Y|}. \end{aligned}$$
(1)

The Dice similarity coefficient is a metric that describes the similarity of two images, X and Y, in the range [0 1], where the coefficients 0 and 1 is the points of non- and full spatial overlap respectively [12]. The quotient of similarity (QS) is found by first multiplying the intersecting samples of binary value 1 in image X and Y by two, before the product is divided by the total amount of samples with binary value of 1  in both images. The calculation of the Dice similarity coefficients is executed with the DiceScore function developed by Dr. Hanno Scharr [12]. Compared to the ground truth image, the image at dark conditions in Fig. 4 has \(QS=0.9556\), whereas the image at light condition has \(QS=0.9366\).

Fig. 4.
figure 4

Comparing manually segmentet images against the triangle threshold method. (Color figure online)

Full size image

3.3 Motion Estimation

The idea of estimating the movement of the leaves from the time-lapse series has its root in a recent published article [5] where two frame motion estimation is used to locate vertical movements of plants. They have used their own implementation based on the method by Lucas and Kanade [6]. The algorithm that we have chosen for the estimation of motion is the method of Farnebäck [3]. This method has the benefit of being able to estimate motion at coarser scale due to its pyramid layer implementation, and thus making it more suitable to estimate the larger displacements associated with e.g. leaf contraction. It also has the benefit of being tensor based algorithm, which is ideal for Matlab implementation in terms of running speeds. We used the implementation found in the Mathworks Matlab 2015b Computer Vision System Toolbox. This algorithm computes a dense optical flow field, i.e. for each pixel in the image a 2D velocity vector is estimated.

The method is based on approximating an image neighborhood by a quadratic polynomial. The velocity vector is modeled as a displacement of the neighborhood and it is assumed that the displacement field is slowly varying in the spatial coordinates of the image frame. The result is that the optical flow field is smeared (in the spatial image coordinates). The intensity (pixel value) on the leaves has a low variation. Therefore it is difficult to estimate reliable optical flow vectors for some parts of the leaves where the intensity is nearly constant. An example of the optical flow is shown in Fig. 5.

Fig. 5.
figure 5

Optical flow field, blue arrows. The border of the two compound leaves are shown in red. (Color figure online)

Full size image

4 Results

Our purpose is to analyse plant behavior as a function of time when a plant is exposed to different light-dark regimes, and where the overall goal is to identify properties that will support the development of a dynamic mathematical model of the plant. An example of such a property is the total leaf motion represented by the pixel count performed on the segmented images, and a typical result is shown in Fig. 6. The curve represents the total white pixel count as a function of an entrained circadian light rhythm of 8 h of darkness and 16 h of light during a 7 days experiment. The plant was prior to the experiment kept in constant light conditions for 10 days, and the result displays a number of interesting observations. First of all, the plant uses three days to be entrained by the new light/dark cycle. This can e.g. be seen at the end of each light period where the leaf first opens up more (indicated with three red arrows), and from the fourth day displays the anticipatory behaviour of initiating leaf closure prior to the light-to-dark transition (indicated with green arrows).

Secondly, we saw from inspection of the time lapse movie that the leaflets behaved differently hours before the light-to-dark transition compared to the hours before the dark-to-light transition. The behavior of the leaflets consisted of rapid motions, similar to shivering, which is recorded in the pixel count as noise, see e.g. in the range from frame number 1100 to 1200. To capture and quantify this behavior, we estimated this type of motion by optical flow. However, the complete optical flow for each frame is difficult to use, and in order to find a more simple description we computed the distribution of velocity magnitudes and velocity directions. As a measure we used the mean flow magnitude and the main flow direction of motion computed for each image frame. The main direction is computed as the maximum of the histogram for the directions. If the directions are too diverse, i.e. several peaks in the histogram, no value is computed.

Fig. 6.
figure 6

Amount of white pixels in the leaf during a 7 days experiment as a function of frame number. Time distance between each frame is 5 min. The dark period starts at 12 p.m. and lasts for 8 h, whereas the light period starts at 8 a.m. and lasts 16 h. The dark/light period is therefore 24 h. (Color figure online)

Full size image

As the leaf used in the experiment consists of two compound leaves (see Fig. 5), the motion estimation will therefore result in averaged values based on often opposite individual values, and is thereby of limited relevance. Hence, in order to capture the individual motion of each compound leaf, we separated the leaf by introducing the stem indicated in red in Fig. 7.

Fig. 7.
figure 7

Separation of the leaf into the two compound leaves. In the situation when the compound leaves interfere, the result is summed and divided by two. The two upper images correspond to the frames 880 and 881 prior to the light-to-dark transition shown in Fig. 6. The two lower images correspond to the frames 892 and 893 coinciding with the same transition. (Color figure online)

Full size image
Fig. 8.
figure 8

Detailed results for the frames 850 to 1050 in Fig. 6. The black curve represents one of the two compound leaves and the red curve the other. There is a light-to-dark transition at frame number 892 and a dark-to-light transition at frame number 989, both marked by a vertical line. Frame 880 is indicated to relate the result to the two upper images of Fig. 7. (Panel a): White pixel count. (Panel b): Optical flow magnitude. (Panel c): Optical flow direction. (Color figure online)

Full size image

These images correspond to the frames 880, 881, 892 and 893 from Fig. 6, and represents the situation prior to and right at the light-to-dark transition. For the two lower images, the contraction of the compound leaves results in a situation where the leaf is considered as one, and the motion estimation for the two parts are calculated from the entire leaf.

An example of (a) white pixel count, (b) mean optical flow and (c) flow direction is shown in Fig. 8, which corresponds to the frames 850 to 1050 from Fig. 6 and where the light-to-dark transition occurring at frame 892 is indicated.

In order to relate the upper two images of Fig. 7 to the results in Fig. 8, we have indicated frame 880 and we see that there are individual results (one black and one red curve) for the white pixel count, the mean optical flow and the flow direction. We also see that around the light-to-dark transition at frame 892, the calculated values are similar for the black and red curves (corresponds to the leaf being considered as one in the lower two images of Fig. 7).

An interesting result is that the shivering we manually observed in the time lapse movie prior to the light-to-dark transition is found in the flow magnitude and direction in Fig. 8 (panels b) and (c). We see that the optical flow has large variations between frames around the periode of light-to-dark transition, though a similar variation is not found at the dark-to-light transition.

To summarize, our work was originally inspired by [5] where cameras were mounted alongside Arabidopsis plants to register the vertical motion of the entire plant. This vertical motion was again related to the circadian rhythm of the plants. In our work, we have placed the camera above the plant, focusing on a single leaf in the horizontal plane. We apply (i) pixel count to estimate the circadian rhythm and (ii) motion estimation to investigate motion details for each compound leaf. Thus, as shown in Fig. 8, these two measures reveal additional details regarding leaf motion. We believe that these details are important for the development of a mathematical model of the circadian rhythm of the plant.

5 Conclusion

We have in this paper shown how images of Mimosa pudica obtained through experiments in a controlled environment can be used in an image processing routine in order to reveal behavioral information with respect to different light/dark regimes. Using a blue background and the HSV color model, the images are segmented using the triangle threshold method. These segmented images are further used to calculate the pixel count which gives an overall estimate of the leaf movement and also an estimate of the plants anticipatory behavior with respect to open or to close leaves prior to light changes. In order to capture details in the motion of the compound leaves, the leaf images are separated into two parts using an artificial stem. Based on the magnitude and direction of the optical flow, the results indicate that there are behavioral information hidden in the individual movement of each compound leaf. An example of such information is the frequency content of the leaflet motion.