Skip to main content

Research Article

Optical Flow-Based Analysis of the Relationships between Leaf Wilting and Stem Diameter Variations in Tomato Plants

Kazumasa Wakamori1 and Hiroshi Mineno1,2

1Graduate School of Integrated Science and Technology, Shizuoka University, 3-5-1 Johoku, Naka-ku, Hamamatsu, Shizuoka 432-8011, Japan
2JST, PRESTO, 4-1-8 Honcho, Kawaguchi, Saitama 332-0012, Japan
Correspondence should be addressed to Hiroshi Mineno; pj.ca.akouzihs.fni@onenim

Abstract

The estimation of water stress is critical for the reliable production of high-quality fruits cultivated using the tacit knowledge of expert farmers. Multimodal deep neural network has achieved success in the estimation of stem diameter variations as a water stress index, calculated from leaf wilting and environmental data. However, these studies have not addressed the specific role of leaf wilting in the estimation. Revealing the role of leaf wilting not only ensures the reliability of the estimation model but also provides an opportunity for improving the estimation method. In this paper, we investigated the relationships between leaf wilting and stem diameter variations without resorting to black-box approaches such as deep neural network. To clarify the role of leaf wilting, this study uses cross-correlation analysis to analyze the time lag correlation between leaf wilting, quantified by optical flow, and stem diameter variations as a water stress index. The analysis showed that leaf wilting had a significant time lag correlation with short-term stem diameter variations, which were water stress responses in plants. As the results were consistent with known plant water transport mechanisms, it was suggested that leaf wilting quantified by optical flow can explain short-term stem diameter variations.

1. Introduction

Recently, several studies have successfully demonstrated that the use of information technology to monitor key indicators could replace the visual decision-making of expert farmers and prevent the loss of sophisticated cultivation techniques [13]. In these studies, data mining, image processing, and machine learning technologies were applied to plant images and environmental data in order to determine and extract indicator variables for such decision-making. Defining an indicator of water stress in cultivation is very important for producing fruits with high sugar content and improving the market value of such products. In Japan, water stress cultivation is commonly used to produce fruits with high sugar content. Such water stress cultivation is currently supported by the tacit knowledge of expert farmers, and a decision-making tool on irrigation control to replace the visual judgement of the expert farmers would be very beneficial. The fruit sugar content ratio can be increased by restricting irrigation and decreasing the amount of water flow into the fruit during stress cultivation, resulting in high sugar content, although the fruit sizes are smaller [4]. However, when plants are exposed to excessive water stress, they may die if they reach permanent wilting point. Therefore, we always need to keep plants under optimal water stress conditions and develop a reasonable method for measuring the water stress condition of plants for optimal decision-making about irrigation.

Several measurement methods have been proposed in previous studies for plant water stress management. Conventionally, water stress changes according to the water potential of a plant. In particular, it depends on the water content of the leaves and the stem. Thus, water stress can be accurately measured based on the water potential of a plant [5]; however, this method cannot be applied to real-time irrigation control systems because it requires the destruction of the plants being measured and hence is unsuitable for time series measurements. Several methods aimed at estimating water stress in real-time irrigation settings have been proposed, which involve estimating various plant-stress responses such as leaf vibration [6] and stem diameter variations [716]. The leaf vibration approach measures water stress using an ultrasonic speaker and microphone, while the stem diameter approaches involve employing contact-type displacement sensors. These approaches can be applied to real-time irrigation control systems because water stress is measured with nondestructive sensors in real time. Some characteristics of stem diameter variations in tomato plants have been previously investigated, and it has been noted that the stem diameter has diurnal variations, which are affected by environmental data. In particular, stem diameter decreases with exposure to high solar radiation in the daytime and increases with decreased exposure to solar radiation in the evening [79, 12, 15]. In addition, the maximum daily stem shrinkage of stem diameter is related to the vapor pressure deficit [17].

Other water stress estimation methods have been proposed utilizing plant images and environmental data measured without contacting plants [1826]. Crucially, the contactless measurement allows new farmers to use the water stress estimation method without previous experience of taking such measurements. Contactless approaches make it possible to replace costly sensors such as displacement sensors and ultrasonic speakers that require expert knowledge, with more affordable sensors such as cheap cameras and environmental sensors. By using time series posture images with environmental data, optical flow can be applied to obtain water stress estimates in a densely planted environment—such as with tomato cultivation—to detect the movement of wilting leaves in a complex background [25, 26]. In these approaches, stem diameter variations are estimated as a water stress index using a multimodal deep neural network, which is a machine learning method that uses multimodal data as the input [27]. Plant posture images and environmental data that are used as input data for neural network studies have revealed that the use of multimodal data is effective for estimating stem diameter variations [25, 26]. As previously mentioned, environmental data are known to have a strong influence on stem diameter diurnal variations, and therefore, it is necessary to measure these variations. On the other hand, because these approaches were based on the deep neural network modeling tending to be a black-box process, the role of leaf wilting quantified by optical flow is yet to be clarified and validated. Clarifying what leaf wilting represents can help improve the existing machine learning-based estimation methods for stem diameter variations.

The objective of the current study was to clarify the role of leaf wilting in the estimation of stem diameter variations. Specifically, we aimed to analyze (1) the characteristics of stem diameter variations, which reflect the water stress of plants, (2) the relationships between leaf wilting and stem diameter variations, and (3) the effects of environmental data on the relationships between leaf wilting and stem diameter variations. By linking the results from our analysis to plant water transport mechanisms, we investigated the relationships between leaf wilting quantified by optical flow and stem diameter variations. Finally, we discuss how to improve the performance of machine learning-based estimation methods for water stress using leaf wilting and environmental data.

2. Materials and Methods

2.1. Data Collection Environment

We collected plant images, stem diameter, and environmental data from three pinched tomato (Solanum lycopersicum L. “Frutica”) plants (labelled target plant 1, target plant 2, and target plant 3) in a dense tomato cultivation from a commercial greenhouse in Fukuroi, Japan (Figure 1). The plants were sown on January 21, 2018, and planted on February 20, 2018; the plants were pinched on April 2, 2018, and crop harvesting began on April 25, 2018. We collected data from April 6, 2018, to April 24, 2018, between pinching and harvesting. For the data collection, we installed an RGB camera (GoPro HERO5 Session, GoPro Inc.), a laser displacement sensor (HL-T1010A, Panasonic Corp.), and a wireless scattered light sensor node [28] for each tomato plant as illustrated in Figure 1(c); the data were collected every minute. The plant image data were collected from 9:00 to 17:00, which was an ideal time range when natural light was sufficiently bright, and other sensor data were collected for the entire day. The RGB cameras were installed at a location where the target tomato was situated at the center of the captured posture images. Laser displacement sensors were located between the 9th and 10th nodes of the tomato plants, and they were connected to a data logger (midi LOGGER GL840, GRAPHTEC Corp.). Wireless scattered light sensor nodes were located above the target plant where the sensor node collected all the environmental data. Each scattered light sensor node had a temperature and humidity sensor (SHT-25, Sensirion) that measured the temperature, relative humidity, and vapor pressure deficit (VPD) and silicon photodiodes (S1133-14, Hamamatsu Photonics K.K.) that measured scattered light. The sensor node was shaped like a cube covered with vinyl chloride plates except for one surface, for sunlight protection; the two sensors were attached to the inside of the box. Since the measured scattered light inside the box was not affected by the shadows of steel pipes in the greenhouse, the value always represented the overall brightness around the target plant [29]. In addition, scattered light data were converted from lux to photosynthetic photon flux density (PPFD) ( ) by dividing the value by a conversion factor of 54 [30].

Figure 1

Alt text
Data collection environment. Overhead view of the greenhouse (a), overhead view of a cultivation line (b), layout of measuring equipment for one targeted tomato plant (c).

2.2. Cultivation Method and Growth Environment

In the present study, we designed a new growing system for tomato plants with high density planting and low-truss cultivation to collect time series cultivation data. This system used a rockwool culture (Yasaihana-pod, Nippon Rockwool Corp.) for each tomato plant, and the plant density was 4 plants/m2. Since the rockwool culture was shaped like a small cube ( ), the roots of the plant were restricted in the cube. The system irrigated the culture with a nutrient solution (electrical conductivity 2.2 dS/m, pH 6.7). The irrigation control was based on their measured stem diameter and an indexed “relative stem diameter calculated from the most recent irrigation (RSR).” The from which the RSR value in current time ( ) is calculated was as follows: where means the current time and means the elapsed time since the last irrigation; thus, RSR is a value calculated by dividing the current stem diameter by the maximum stem diameter achieved at the most recent irrigation event. We set a threshold for RSR, and when the RSR fell below the threshold, an irrigation system irrigated the rockwool cube. After irrigation, the stem diameter gradually increased by absorbing water and the RSR value was set to 1.0000. When the stem diameter decreased because of water stress, the RSR fell below 1.0000. Therefore, when the threshold was set to less than 1.0000, the irrigation control caused water stress to the plants. The present study required a sufficient irrigation interval to cause water stress to the plants because the irrigation impact on the water status was not instantaneous. The RSR-based irrigation control helped adjust the optimal interval based on plant water stress using the threshold . We defined the threshold based on the opinions of experienced farmers in this experiment. was 0.9950 from April 6, 2018, to April 7, 2018, and 0.9975 from the April 8, 2018, to April 24, 2018. The automatic irrigation control worked from sunrise to 19:00 each day. Sunrise was defined as when the PPFD reached and each irrigation event was 15 ml of water per plant.

2.3. Data Preprocessing

Plant stress responses including leaf wilting and stem diameter were affected by diurnal environmental variations [15]. However, diurnal variations were not controlled by irrigation, which was able to control only short-term variations in stress responses. To analyze the relationships between leaf wilting and stem diameter variations considering irrigation, this study compared only short-term variations of leaf wilting and stem diameter. Thus, this analysis focused on the short-term variations of leaf wilting and stem diameter and excluded diurnal variations. To extract information about short-term leaf wilting and stem diameter variations, we used optical flow of plant images and temporal variations in stem diameter. Optical flow is an image processing method that calculates pixel-level velocity vectors, which are composed of angle and magnitude, between two images. In other words, optical flow expresses the relative motion pattern of objects in the two images. In this study, we quantified relative leaf wilting using the optical flow from the time series plant posture images. Figure 2 shows the workflow of image processing for the quantification of relative leaf wilting. The image processing quantified relative leaf wilting ( ) at time and . Firstly, an optical flow algorithm detected the wilting motion of leaves. In this study, we used DeepFlow [31], which is one optical flow algorithm that detects the relative motion of nongrid objects, such as wilting at high density and robustly from a complex background. Secondly, an excess green (ExG) index is calculated for masking the detected optical flow because optical flow detection involves the objects in motion such as clouds from the sky in the background. We extracted the plant area in the plant images with a color filter (threshold of 0.0) based on the ExG index to generate masked optical flow. Thirdly, we converted the masked optical flows to a histogram called the histograms of oriented optical flow (HOOF) [32]. HOOF shows each optical flow vector according to its angle and is weighted on its magnitude. We have a hypothesis that downward angles indicate leaf wilting, whereas upward angles depicted a recovery of wilting in optical flow from the time series plant images. Thus, we generated an elementary histogram based on the upward bin ( ) and downward bin ( ) to enable the analysis program to reveal the features that related to leaf wilting. Finally, we calculated a scalar value ( ) that represented relative wilting by subtracting from .

Figure 2

Alt text
Processing flow of leaf wilting quantification from time series plant images. The parameter indicates sampling periods used in calculating the optical flow. Note that we set for DeepFlow to sufficiently detect leaf wilting. Time series plant images were resized to pixels in each image.

For time series stem diameters ( ), we calculated temporal differences in stem diameter ( ) as follows: where indicates short-term variations in the stem diameter and the relative value between time and as with . By calculating the temporal differences, we can remove the diurnal variations of stem diameter and avoid spurious correlations caused by diurnal variations in the data analysis. Therefore, we can obtain an estimation of the relationships between leaf wilting and short-term stem diameter variations.

In this analysis, we used , which is half of the minimum irrigation interval (10 min), for calculating the optical flow of time series plant images and temporal differences in stem diameter. Choosing the interval involved a trade-off between removing diurnal variations in stem diameter and leaf wilting quantification. A shorter could remove diurnal variations in stem diameter clearly; however, it would not be able to detect slight leaf wilting by optical flow. On the other hand, a longer could detect the slight leaf wilting, but the diurnal variations remained in . Moreover, this study set the interval to be less than the minimum irrigation interval. If was above the minimum irrigation interval, two or more irrigations could be included in the interval , in which case, we could not have analyzed the short-term relationships affected by the irrigation. In addition, and were calculated at 1 min intervals of a moving window.

2.4. Data Analysis

We analyzed the relationship between and using cross-correlation analysis that could evaluate the correlation and time lag. It was assumed that and have temporal dependencies caused by water transport in a plant, and thus, the cross-correlation analysis can find temporal dependence. In this analysis, we slid along the time series, and at every lag position , we calculated the correlation coefficient with . The correlation was expressed by the cross-correlation coefficient where is the expected value of for all possible values of . and are the average values, and and are the standard deviations of and , respectively. We used the “numpy.correlate” function in Python 2.7 to calculate , and we divided the return values by to calculate the expected value. In this study, we calculated the cross-correlation coefficient for each day separately, because the cross-correlation coefficient was not applicable to time series data with missing information such as the lack of image data at night. Moreover, as the greenhouse environmental conditions differed with daily weather, it was possible to examine whether the relationships between leaf wilting and stem diameter variations were dependent on cross-correlation coefficient calculations of environmental data for each day. In the cross-correlation coefficient, the sign of means time lag at the highest correlation coefficient and represents the direction of the temporal dependence relationships between the two time series data. A positive means that the stem diameter variations preceded leaf wilting, while a negative indicates the opposite temporal relationship.

Then, we analyzed the relationships between the maximum cross-correlation coefficient and the mean of environmental data to clarify the environmental condition in which leaf wilting had the closest relationships to the stem diameter variations. The mean of the environmental data is an arithmetic mean and was calculated from data collected from 9:00 to 19:00; the time range was based on the time of plant image collection. Linear, second-order polynomial, piecewise, log, and exponential regressions were used for regression analysis between the maximum cross-correlation coefficient and environmental data. The piecewise regression split the explanatory variables and made two regression lines. The threshold for splitting the regression line was fitted as a parameter in the regression method. The regressions were fitted using the “scipy.optimize.curve_fit” function in Python 2.7. The score was applied to select the most probable regression model.

3. Results

3.1. Diurnal Variations and Short-Term Variations in Stem Diameter

To examine the characteristics of stem diameter variations, the stem diameters of tomato plants were measured over time using laser displacement sensors. Figure 3 shows the stem diameter variations on a sunny day and a cloudy day. The stem diameter on a sunny day exhibited larger diurnal variation than that on a cloudy day. On a sunny day, the stem diameter decreased after sunrise and increased after sunset, and the diurnal variation was related to scattered light intensity. We confirmed that the characteristic diurnal variations were in agreement with the results of previous studies [79, 12, 15]. However, stem diameter included not only diurnal variations but also short-term variations (red dotted line area in Figures 3(a) and 3(b)). Because short-term variations are repeated changes based on irrigation and the variations are notable on a sunny day, it was assumed that short-term variations in stem diameter were related to water uptake. To analyze irrigation-based relationships between leaf wilting and stem diameter, the short-term variations of leaf wilting and stem diameter were compared.

Figure 3

Alt text
Typical variations in stem diameter. Diurnal variations and short-term variations in stem diameter and irrigation on a sunny day (a) and on a cloudy day (b). PPFD and vapor pressure deficits in the two days (c, d). All data were collected from target plant 2, and daily averages of the PPFD were maximum and minimum on April 19, 2018, and April 24, 2018, respectively.

3.2. Relationships between Relative Leaf Wilting and Short-Term Stem Diameter Variations

We subsequently analyzed the relationship between and using cross-correlation analyses. Figure 4(a) shows and data from a day where leaf wilting was detected, and Figures 4(b) and 4(c) show the same data from days where leaf wilting was not detected. was time series data, where diurnal variations were removed by adjusting for temporal differences to the stem diameter; the units are in mm. was time series data calculated from the optical flow of the time series plant images in the same intervals as . indicates an average value per pixel of the optical flow vectors, and the units are in pixels. Both and represent the amount of relative displacement in a 5 min interval.

Figure 4

Alt text
Examples of relative leaf wilting ( ) and temporal differences in stem diameter ( ). An example on a day which leaf wilting was detected (a), examples on days which leaf wilting was not detected (b, c) in target plant 2.

Figures 5(a)–5(r) show cross-correlation graphs of and . The -axis of the figures indicates a delay of in the cross-correlation analysis and the unit of the time lag is minutes. A time lag of 0 means no time lag, less than 0 means delayed for negative temporal direction, and greater than 0 means delayed for positive temporal direction. We confirmed significant positive correlations between and from these graphs as shown in Figures 5(a)–5(d) and 5(f)–5(q). The results show that and have a positive correlation with a time delay. was the maximum cross-correlation coefficients confirmed at a time lag of 0 or more during this time; thus, it is suggested that had a slight time delay compared to .

Figure 5

Alt text
Cross-correlation coefficients. Cross-correlation graphs between and from 9:00 to 17:00 for each day (a–r). The red dotted line shows and the two black dotted lines represent the 99% confidence interval. A graph on April 10, 2018, and target plant 3’s lines on (a), (c), (d), (j), and (k) are not shown because of missing plant images on those days. A scatter plot represents the relationships between the maximum cross-correlation coefficients and the time lags (s).

Figure 4(a) shows a time series plot of and on a sunny day when and were recorded. Both and increased from the irrigation time points and then both responses decreased. In addition, it appears that is delayed when compared to . On the other 2 days, the cross-correlation graphs do not change periodically (Figures 5(e) and 5(r)), because leaf wilting did not occur, and remained nearly unchanged as shown in Figures 4(b) and 4(c). Moreover, the weather on the two days was mostly cloudy.

Figure 5(s) presents a scatter plot of and , time lags of . When is high, the time lag converges at 2 min; the average of is 1.96 min when (the maximum of is 0.849 on April 22, 2018). Thus, this indicates that leaf wilting or recovery occurs approximately 2 min after the stem diameter changes in this environment. Leaf wilting occurs noticeably on sunny days because of water stress.

3.3. Relationship between Maximum Cross-Correlation Coefficient and Environmental Data

We finally analyzed the relationship between environmental data and the maximum cross-correlation coefficient of and or its time lag to confirm that the correlation strength was affected by the greenhouse environmental conditions. The maximum cross-correlation coefficient between the two stress responses represented the strength of the linear relationship in terms of the time delay between the two time series data. Figures 6(a) and 6(b) show a scatter plot of the maximum cross-correlation coefficients and mean environmental data. The figures show that the maximum cross-correlation coefficient is related to the mean PPFD and mean VPD suggesting that the maximum cross-correlation between the and coefficients was affected by environmental data. The scores with the mean PPFD and VPD were 0.4298 and 0.4291, respectively. The maximum cross-correlation coefficient increased logarithmically with increase in the mean PPFD as shown in Figure 6(a) ( ), and the relationship with mean VPD was best fit with a piecewise linear function as shown in Figure 6(b) ( ). In addition, the time lag between and converged at about 2 min with an increasing mean PPFD or mean VPD as shown in Figures 6(c) and 6(d). When the mean PPFD exceeded , the average time lag was 1.96 min between and and the maximum and minimum time lags were ±1 min from 2 min. In the case of VPD exceeding 2 kPa, the average time lag was 2.00 min and the maximum and minimum time lags were ±1 min from 2 min, likewise the mean PPFD.

Figure 6

Alt text
Relationships between cross-correlation and environmental data. Relationships between the maximum cross-correlation coefficient and the mean PPFD (a) and mean VPD (b). The relationships between the time lags and mean PPFD (c) and mean VPD (d).

4. Discussion

We studied the relationships between leaf wilting quantified by optical flow and stem diameter variations with the objective of clarifying the role of leaf wilting in the estimation of stem diameter variations. We analyzed the relationships between relative leaf wilting time and short-term stem diameter variations using cross-correlation analyses. Image processing algorithms, including optical flow, aided calculation of relative leaf wilting values, and temporal differences in stem diameter variations were measured. The results showed that (1) the short-term variations were correlated with leaf wilting and the two data had a time lag and (2) the strength of the correlation between leaf wilting and short-term stem diameter variations was affected by environmental data. Finally, the observed results were supported by plant water transport literature.

4.1. Water Transport-Based Interpretation of Results

Water transportation in a plant is caused by transpiration during the day [33]. When osmotic pressure, which is involved in the water potential, is decreased and negative pressure is exerted in the leaves by transpiration, the water potential gradient transports water from the roots and stems to the leaves. If water is available from the irrigated soil at this time, the root will have a positive pressure and water can continue to be transported to the leaves. If water is not available from the soil, the water potential is reduced, the stem diameter is reduced, and the plant becomes depleted of water and the leaves wilt. The water potentials of stem and leaves successively change through water osmosis from the root. The above sequence might indicate that the stem is water stressed, which manifests as stem diameter variations, and precedes the leaf water potential change, which appears as wilting. In this study, we investigated the time lag correlation between the two plant responses (Figure 5), and we found a significant correlation. In the scatter plot of time lag and maximum correlation coefficient (Figure 5(s)), the average time lag was 1.96 min, when the maximum correlation coefficient was 0.6. Thus, the short-term stem diameter variations preceded leaf wilting by approximately 2 min, and this result supported the hypothesis that the relationships between the water potential variations in the stems and leaves correlated with each other in the short-term and the relationship has a time lag. Therefore, it was concluded that the time required for the transport of water from the stems to the leaves was 2 min according to this analysis.

When the transpiration rate was correlated with the relative leaf wilting and short-term stem diameter variations, the strength of their relationship was affected by environmental data, which affected the transpiration rate. We analyzed the relationship between mean of environmental data (PPFD and VPD) and the maximum cross-correlation coefficient. The maximum cross-correlation coefficient indicated the strength of the relationships between relative leaf wilting and the short-term stem diameter variations (Figures 6(a) and 6(b)). These analyses showed that the maximum correlation coefficient was related to these environmental data. In particular, the maximum correlation increased with increasing PPFD, indicating solar radiation or increasing VPD of 0–4 kPa. The characteristics were similar to relationships between these environmental data and sap flow or transpiration [34, 35]. The maximum correlations with the VPD did not increase from 4 kPa, and therefore, it is conjectured that stomatal closure caused by high VPDs inhibited transpiration and leaf wilting. Therefore, the results suggested that the strength of the relationships between relative leaf wilting and stem diameter variations may be affected by environmental data. In the same way, environmental data affected the time lag of the relationship. It appeared that the time lag converged when the mean PPFD and VPD values increased (Figures 6(c) and 6(d)). In addition, the average time lag was around 2 min in conditions where mean and mean in this environment. These results suggested that stem diameter variations preceded leaf wilting by approximately 2 min in environmental conditions where leaf transpiration is facilitated.

Furthermore, the analysis of these results showed that relative leaf wilting, quantified by optical flow, and short-term variations of stem diameter had time lag correlations with environmental conditions where transpiration was constantly promoted. These results could be explained by the water transport mechanism. Thus, it is concluded that the optical flow-based leaf wilting can explain short-term variations of stem diameter caused by transpiration during the day.

4.2. Improvement of Stem Diameter Variation Estimation by Multimodal Machine Learning

Since our results suggested that the correlation between and occurred only under conditions when the transpiration rate was high, only using optical flow-based monitoring of water stress may not be useful for irrigation under other environmental conditions. Alternatively, modeling based on multimodal machine learning can estimate stem diameter variations irrespective of whether the environment facilitates transpiration. The multimodal machine learning method can use leaf wilting data and environmental data dynamically to estimate stem diameter variations which are then used as a water stress index. Thus, when leaf wilting occurred, the model estimated stem diameter from both leaf wilting and environmental data, and when leaf wilting did not occur, the model estimated only using the environmental data. The modeling by multimodal machine learning achieved the stem diameter-based irrigation in a nondestructive manner from leaf wilting and environmental data.

In this study, we quantified the relationships between relative leaf wilting and short-term stem diameter variations using optical flow and cross-correlation analyses. Moreover, we noted how the strength of the relationships between these parameters was associated with environmental data. These results verified the effectiveness of using time series plant posture images and environmental data as input data in existing estimation methods of stem diameter variations [25, 26]. Several studies have reported that environmental data affect the diurnal variation of the stem diameter [15]; thus, the existing methods can estimate such diurnal variation using environmental data. In addition, the existing methods could estimate this short-term variation by superposing features of leaf wilting extracted from plant images and environmental data which were related to short-term stem diameter variations. Therefore, we assumed that existing methods could estimate stem diameter variations because they used two types of input data that characterize the diurnal and short-term variations in stem diameter. Moreover, the analysis provided an approach that improved stem diameter estimation of plant images and environmental data using a deep neural network. Recurrent neural network layers, such as long short-term memory (LSTM) [36] and gated recurrent unit (GRU) [37], were able to extract time series leaf wilting features after considering time lags with short-term stem diameter variations, as with the existing method [25]. However, different temporal features should be extracted from plant posture images and environmental data in deep neural networks, because the two-type input data have different temporal relationships with stem diameter variations. Environmental data and leaf wilting explained diurnal and short-term variations of stem diameter, respectively. Thus, a neural network with one general LSTM layer, as in the existing method [25], seemed inadequate to estimate stem diameter variations. We expected that recurrent neural networks that can extract multiple levels of temporal features such as stacked LSTM [38], tensorized LSTM [39], and nested LSTM [40] would all be suitable for estimating stem diameter variations, and we should investigate the improvement in deep neural networks for water stress estimation in future work.

4.3. Limitations and Future Work

Our results had four limitations. First, other environmental conditions may not cause the relationships between leaf wilting and stem diameter variations that we described here. In the analysis, we assumed equilibrium conditions of transpiration, which were based on the fact that environmental conditions did not change within intervals of . Therefore, the time lag correlation was not observed in the transpiration disequilibrium conditions such as on a cloudy day. In transpiration disequilibrium conditions, the stem diameter variations can affect leaf wilting status based on changes in the driving force of transpiration. Future studies are needed to develop an analysis method that can elucidate the relationship between the stem diameter and leaf wilting under various environmental conditions including transpiration disequilibrium.

Second, we analyzed tomato plants after patching; the transpiration rate and maximum water potential are different in each type of plant, and they are also affected by its growth stage. Thus, the quantified results including the correlation coefficient, time lag, and the regression equation applied to tomato plant may not be generalizable to other plants. On the other hand, many studies have reported that stem diameter variations are expressed as a result of water stress in various types of plants [7, 9, 11, 13, 14], and these plants generally exhibited leaf wilting caused by an imbalance between the water uptake rate and transpiration rate. Thus, it is expected that the relationships between leaf wilting and variations of the stem diameter could be elucidated for various types of plants by applying our analysis methods.

Third, we analyzed only the relationship between relative leaf wilting and relative stem diameter in response to water stress in this study. There are many biological factors related to plant water stress or transpiration, such as soil water potential, stomatal conductance, and CO2 levels, and it is unclear if there are relationships between our results and these factors. An analysis of the relationships of stem diameter variations with these biological factors may reduce the residual from the results of the maximum cross-correlation coefficient and PPFD or VPD (Figures 6(a) and 6(b)). However, it was a limitation of this study, and in the future, we should collect data related to these biological indices and conduct an exhaustive analysis. In addition, the results were based on only one set of cultivation conditions and the environmental data were not independent; thus, these thresholds of PPFD and VPD might be dependent on these cultivation conditions. In a future study, we should analyze data from cultivations under various stresses to evaluate the generality of these environmental thresholds.

Finally, a plant experiences both leaf rolling and leaf wilting with limited irrigation [41]. The optical flow-based method cannot detect and quantify leaf rolling; however, it is necessary to do so to develop a more robust irrigation control mechanism based on the total leaf condition. Leaf rolling leads to a change in its three-dimensional (3D) shape, and it is possible that this can be detected with a depth camera that can measure both an RGB image and depth data in 3D. We would like to examine the applicability of the depth camera for leaf condition quantification in future studies.

Conflicts of Interest

The authors declare that there is no conflict of interest regarding the publication of this article.

Authors’ Contributions

K.W. and H.M. conceived this work. K.W. collected and analyzed the data and wrote the manuscript. H.M. supervised and coordinated this work. All authors reviewed the manuscript.

Acknowledgments

We greatly appreciate Mr. Makoto Miyachi (Happy Quality Co., Ltd., Japan) and Mr. Daigo Tamai (Sun Farm Nakayama Co., Inc., Japan) for providing the facilities for data collection. This work was supported by the Japan Science and Technology Agency (JST) PRESTO (grant number JPMJPR15O5).

Supplementary Materials

Supplementary Fig. 1: box-and-whisker plot of daily environmental data in the growth environment. Each box-and-whisker includes environmental data from 9 am to 7 pm, as shown in Figure 5. The maximum and minimum of whisker represent the maximum and minimum of each environmental data, respectively. (Supplementary Materials)

References

  1. A. Singh, B. Ganapathysubramanian, A. K. Singh, and S. Sarkar, “Machine learning for high-throughput stress phenotyping in plants,” Trends in Plant Science, vol. 21, no. 2, pp. 110–124, 2016. View at Publisher · View at Scopus · View at Google Scholar
  2. Y. Jiang, C. Li, J. S. Robertson, S. Sun, R. Xu, and A. H. Paterson, “GPhenoVision: a ground mobile system with multi-modal imaging for field- based high throughput phenotyping of cotton,” Scientific Reports, vol. 8, no. 1, p. 1213, 2018. View at Publisher · View at Scopus · View at Google Scholar
  3. S. Ghosal, D. Blystone, A. K. Singh, B. Ganapathysubramanian, A. Singh, and S. Sarkar, “An explainable deep machine vision framework for plant stress phenotyping,” Proceedings of the National Academy of Sciences of the United States of America, vol. 115, no. 18, pp. 4613–4618, 2018. View at Publisher · View at Scopus · View at Google Scholar
  4. C. Patanè and S. L. Cosentino, “Effects of soil water deficit on yield and quality of processing tomato under a Mediterranean climate,” Agricultural Water Management, vol. 97, no. 1, pp. 131–138, 2010. View at Publisher · View at Scopus · View at Google Scholar
  5. J. S. Boyer, “Leaf water potentials measured with a pressure chamber,” Plant Physiology, vol. 42, no. 1, pp. 133–137, 1967. View at Publisher · View at Google Scholar
  6. M. Sano, Y. Nakagawa, T. Sugimoto et al., “Estimation of water stress of plant by vibration measurement of leaf using acoustic radiation force,” Acoustical Science and Technology, vol. 36, no. 3, pp. 248–253, 2015. View at Publisher · View at Scopus · View at Google Scholar
  7. B. Klepper, V. D. Browning, and H. M. Taylor, “Stem diameter in relation to plant water status,” Plant Physiology, vol. 48, no. 6, pp. 683–685, 1971. View at Publisher · View at Google Scholar
  8. K. Fujita, M. Okada, K. Lei et al., “Effect of P-deficiency on photoassimilate partitioning and rhythmic changes in fruit and stem diameter of tomato (Lycopersicon esculentum) during fruit growth,” Journal of Experimental Botany, vol. 54, no. 392, pp. 2519–2528, 2003. View at Publisher · View at Scopus · View at Google Scholar
  9. Y. Ohashi, N. Nakayama, H. Saneoka, and K. Fujita, “Effects of drought stress on photosynthetic gas exchange, chlorophyll fluorescence and stem diameter of soybean plants,” Biologia Plantarum, vol. 50, no. 1, pp. 138–141, 2006. View at Publisher · View at Scopus · View at Google Scholar
  10. M. Gallardo, R. B. Thompson, L. C. Valdez, and M. D. Fernández, “Use of stem diameter variations to detect plant water stress in tomato,” Irrigation Science, vol. 24, no. 4, pp. 241–255, 2006. View at Publisher · View at Scopus · View at Google Scholar
  11. M. F. Ortuño, Y. García-Orellana, W. Conejero, F. Pérez-Sarmiento, and A. Torrecillas, “Assessment of maximum daily trunk shrinkage signal intensity threshold values for deficit irrigation in lemon trees,” Agricultural Water Management, vol. 96, no. 1, pp. 80–86, 2009. View at Publisher · View at Scopus · View at Google Scholar
  12. T. De Swaef and K. Steppe, “Linking stem diameter variations to sap flow, turgor and water potential in tomato,” Functional Plant Biology, vol. 37, no. 5, pp. 429–438, 2010. View at Publisher · View at Scopus · View at Google Scholar
  13. E. Badal, I. Buesa, D. Guerra, L. Bonet, P. Ferrer, and D. S. Intrigliolo, “Maximum diurnal trunk shrinkage is a sensitive indicator of plant water, stress in Diospyros kaki (Persimmon) trees,” Agricultural Water Management, vol. 98, no. 1, pp. 143–147, 2010. View at Publisher · View at Scopus · View at Google Scholar
  14. N. Livellara, F. Saavedra, and E. Salgado, “Plant based indicators for irrigation scheduling in young cherry trees,” Agricultural Water Management, vol. 98, no. 4, pp. 684–690, 2011. View at Publisher · View at Scopus · View at Google Scholar
  15. Z. Meng, A. Duan, D. Chen et al., “Suitable indicators using stem diameter variation-derived indices to monitor the water status of greenhouse tomato plants,” PLoS One, vol. 12, no. 2, article e0171423, 2017. View at Publisher · View at Scopus · View at Google Scholar
  16. X. Wang, Z. Meng, X. Chang, Z. Deng, Y. Li, and M. Lv, “Determination of a suitable indicator of tomato water content based on stem diameter variation,” Scientia Horticulturae, vol. 215, pp. 142–148, 2017. View at Publisher · View at Scopus · View at Google Scholar
  17. D. Zhang, Q. Du, Z. Zhang, X. Jiao, X. Song, and J. Li, “Vapour pressure deficit control in relation to water transport and water productivity in greenhouse tomato production during summer,” Scientific Reports, vol. 7, no. 1, article 43461, 2017. View at Publisher · View at Scopus · View at Google Scholar
  18. J. A. Sánchez-Molina, F. Rodríguez, J. L. Guzmán, and J. A. Ramírez-Arias, “Water content virtual sensor for tomatoes in coconut coir substrate for irrigation control design,” Agricultural Water Management, vol. 151, pp. 114–125, 2015. View at Publisher · View at Scopus · View at Google Scholar
  19. P. Foucher, P. Revollon, B. Vigouroux, and G. Chassériaux, “Morphological image analysis for the detection of water stress in potted forsythia,” Biosystems Engineering, vol. 89, no. 2, pp. 131–138, 2004. View at Publisher · View at Scopus · View at Google Scholar
  20. H. Takayama and K. Nishina, “Early detection of water stress in tomato plants based on projected plant area,” Environmental Control in Biology, vol. 45, no. 4, pp. 241–249, 2007. View at Publisher · View at Scopus · View at Google Scholar
  21. Y. Kim, D. M. Glenn, J. Park, H. K. Ngugi, and B. L. Lehman, “Hyperspectral image analysis for water stress detection of apple trees,” Computers and Electronics in Agriculture, vol. 77, no. 2, pp. 155–160, 2011. View at Publisher · View at Scopus · View at Google Scholar
  22. D. Story and M. Kacira, “Design and implementation of a computer vision-guided greenhouse crop diagnostics system,” Machine Vision and Applications, vol. 26, no. 4, pp. 495–506, 2015. View at Publisher · View at Scopus · View at Google Scholar
  23. X. Cai, Y. Sun, Y. Zhao et al., “Smart detection of leaf wilting by 3D image processing and 2D Fourier transform,” Computers and Electronics in Agriculture, vol. 90, pp. 68–75, 2013. View at Publisher · View at Scopus · View at Google Scholar
  24. D. Guo, J. Juan, L. Chang, J. Zhang, and D. Huang, “Discrimination of plant root zone water status in greenhouse production based on phenotyping and machine learning techniques,” Scientific Reports, vol. 7, no. 1, p. 8303, 2017. View at Publisher · View at Scopus · View at Google Scholar
  25. S. Shibata, Y. Kaneda, and H. Mineno, “Motion-specialized deep convolutional descriptor for plant water stress estimation,” in Engineering Applications of Neural Networks. EANN 2017. Communications in Computer and Information Science, vol 744, G. Boracchi, L. Iliadis, C. Jayne, and A. Likas, Eds., pp. 3–14, Springer, Cham, 2017. View at Publisher · View at Scopus · View at Google Scholar
  26. Y. Kaneda, S. Shibata, and H. Mineno, “Multi-modal sliding window-based support vector regression for predicting plant water stress,” Knowledge-Based Systems, vol. 134, pp. 135–148, 2017. View at Publisher · View at Scopus · View at Google Scholar
  27. J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng, “Multimodal deep learning,” in Proceedings of the 28th International Conference on Machine Learning, pp. 689–696, Bellevue, WA, USA, 2011.
  28. H. Ibayashi, Y. Kaneda, J. Imahara, N. Oishi, M. Kuroda, and H. Mineno, “A reliable wireless control system for tomato hydroponics,” Sensors, vol. 16, no. 5, p. 644, 2016. View at Publisher · View at Scopus · View at Google Scholar
  29. N. Ohishi, “Non-disruptive evaluation of leaf area index using diffused light sensor for tomato cultivation,” Shokubutsu Kankyo Kogaku, vol. 28, no. 3, pp. 125–132, 2016. View at Publisher · View at Google Scholar
  30. R. W. Thimijan and R. D. Heins, “Photometric, radiometric, and quantum light units of measure: a review of procedures for interconversion,” HortScience, vol. 18, pp. 818–822, 1983. View at Google Scholar
  31. P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid, “DeepFlow: large displacement optical flow with deep matching,” in The IEEE International Conference on Computer Vision (ICCV), pp. 2818–2826, Sydney, Australia, 2013.
  32. R. Chaudhry, A. Ravichandran, G. Hager, and R. Vidal, “Histograms of oriented optical flow and Binet-Cauchy kernels on nonlinear dynamical systems for the recognition of human actions,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1932–1939, FL, USA, June 2009. View at Publisher · View at Scopus · View at Google Scholar
  33. W. Fricke, “Water transport and energy,” Plant, Cell & Environment, vol. 40, no. 6, pp. 977–994, 2017. View at Publisher · View at Scopus · View at Google Scholar
  34. R. Qiu, T. Du, S. Kang, R. Chen, and L. Wu, “Influence of water and nitrogen stress on stem sap flow of tomato grown in a solar greenhouse,” Journal of the American Society for Horticultural Science, vol. 140, no. 2, pp. 111–119, 2015. View at Publisher · View at Google Scholar
  35. C. Chanseetis, Y. Shinohara, T. Maruo, M. Takagaki, and M. Hohjo, “An estimation of tomato transpiration for effective fertigation management system using intergrated solar radiation and vapor pressure deficit,” Environmental Control in Biology, vol. 43, no. 2, pp. 105–112, 2005. View at Publisher · View at Scopus · View at Google Scholar
  36. S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997. View at Publisher · View at Scopus · View at Google Scholar
  37. J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” 2014, http://arxiv.org/abs/1412.3555. View at Google Scholar
  38. A. Graves, A. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks,” in 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6645–6649, Vancouver, Canada, 2013.
  39. Z. He, S. Gao, L. Xiao, D. Liu, H. He, and D. Barber, “Wider and deeper, cheaper and faster: Tensorized lstms for sequence learning,” in 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA, 2017.
  40. J. R. A. Moniz and D. Krueger, “Nested LSTMs,” pp. 530–544, 2017, http://arxiv.org/abs/1801.10308. View at Google Scholar
  41. A. Kadioglu and R. Terzi, “A dehydration avoidance mechanism: leaf rolling,” The Botanical Review, vol. 73, no. 4, pp. 290–302, 2007. View at Publisher · View at Scopus · View at Google Scholar

Copyright © 2019 Kazumasa Wakamori and Hiroshi Mineno. Exclusive Licensee Nanjing Agricultural University. Distributed under a Creative Commons Attribution License (CC BY 4.0).

  • Views 340
  • Citations 0
Altmetric Attention Score
Find out more