Keywords

3.1 Introduction

3.1.1 Precision Agriculture Technology in Agricultural Production

Cotton (Gossypium hirsutum L.) is an important cash crop that provides approximately 35% of the total fibers for textile industry, including clothing and fine paper [1, 2]. Cotton seeds are also critical sources for feed and oil industries because of their rich oil (18 - 24%) and protein (20–40%) [1, 2]. Cotton plays an important role in human daily life and livestock industry, and is grown in 17 states and as a major crop in 14 states of the United States of America [3]. There is a great need to increase cotton production due to the increasing global population of more than nine billion in 2050. However, there are adverse conditions for improving agricultural production, such as decreasing arable land due to urbanization, declining soil quality (e.g., soil erosion, salinization, and nutrient reduction) due to long-term cultivation, and insufficient freshwater for crop irrigation [4]. It is time to improve cotton production using emerging technologies to reduce the inputs of natural resources and impacts on environments.

Cotton production is determined by the combined effects of genotype, environment, and management (G × E × M) [5]. To understand the G × E × M interaction, acquiring high-resolution data of crop, water, and other environments is needed using emerging remote and proximal sensing technologies. Studies have shown that timely crop monitoring and accurate yield estimation are important in making optimal decisions for field management, increasing cotton production, and reducing the negative impacts on environments. For example, the information of cotton plants and the environment obtained from different growing stages can be used to quantify the environmental impacts and nutrient deficiencies [6] on cotton development and yield. Precision agriculture (PA) technology has been used as a promising field management strategy to improve crop production and management efficiency based on site-specific information. The key factors for the success of PA include fast and accurate data acquisition of crops, soil, and environments in a large field. It also needs timely data processing and analysis pipelines to translate sensor data to executive data for field management decisions [7,8,9].

Sensor systems are the key components for the implementation of PA for agricultural production. Various sensors have been used to quantify crop development and environmental variations [10, 11], which provide necessary information to develop decision-making tools for field management. Some examples include in-field weather stations that continuously record environmental conditions, e.g., air temperature, humidity, solar radiation, and precipitation. Proximal sensors (e.g., soil apparent electrical conductivity (ECa) sensor and soil moisture sensor) are used to estimate soil texture, organic matter content [12], and soil moisture content at different depths. Infrared thermometers (IRTs) are used to measure crop canopy temperature to determine cotton water stress and schedule irrigation [13]. In addition, remote sensing systems based on satellite, airplane, unmanned aerial vehicles (UAVs), and ground vehicles are equipped with different sensors (e.g., optical sensors) for crop monitoring. The above site-specific information has been used to improve the efficiency of field management and crop production.

3.1.2 UAV-Based Remote Sensing (RS) for Crop Monitoring

Unmanned aerial vehicle (UAV)-based RS is currently widely used for crop monitoring due to the advantages of flexibility in sensor selection and data collection time when compared to satellite-based and ground-based sensing platforms. The UAV-based RS systems are usually equipped with multiple cameras and a global navigation satellite system (GNSS) to collect georeferenced imagery data of crops and soil in a high-throughput manner [14]. These systems are widely used to scout crops and collect site-specific information to make accurate decisions of crop management [15, 16]. Typical cameras used in UAV-based RS systems include visible red–green–blue (RGB) cameras, multispectral cameras, hyperspectral cameras, and infrared (IR) thermal cameras [14, 17]. Imagery data are processed using advanced machine learning (ML) to quantify crop characteristics in architecture, physiology, and chemical compositions [18], which are associated with plant health conditions and responses to the variation of environment and management.

The visible RGB cameras are the most ready-to-use sensors that are less expensive and in higher resolution comparing to multispectral and thermal cameras. The RGB cameras consist of three optical sensors with spectral bands of red (typical 550–650 nm), green (typical 470–600 nm), and blue (typical 420–530 nm) to produce digital images [19]. They have been used to acquire color information of different plant organs (e.g., leaf, stem, flower, and cotton boll) that can be used to characterize plants, quantify their health conditions and responses to biotic and abiotic stresses. The RGB images are also used to build point cloud data of plants to extract three-dimensional (3D) information, such as plant height and 3D architecture. As shown in Table 3.1, RGB images are used to calculate different image features that are used to quantify cotton development and yield. For example, plant height (PH), canopy cover (CC), greenness described as a* and triangular greenness index (TGI) are used for the assessment of cotton stand count [14, 20]. Moreover, the cotton fiber index (CFI) derived from RGB images has been used as a useful index for cotton yield estimation [14, 21].

Table 3.1 Useful image features for cotton monitoring with UAV-based RS systems
Full size table

Despite the advantages of using RGB cameras, they are still limited by only being able to acquire information of plants in the spectral range of 400 to 700 nm wavelength. Research has found that some spectral bands in near-infrared range (800–2,500 nm) are more sensitive to the variations of plants due to biotic and abiotic stresses. Multispectral and hyperspectral cameras have been used in many studies to acquire high spatial resolution spectral information from the plants. Multispectral cameras usually consist of optical sensors with less than ten independent spectral bands, and hyperspectral cameras consist of up to 300 spectral bands. The image features extracted from these spectral cameras include spectral information of each band and their combination, which have shown great potential to quantify plant heath conditions and estimate yield. Some examples (as summarized in Table 3.1) show that different vegetation indices (VIs) based on the combination of multiple spectral bands are able to monitor plant health status and estimate cotton yield [14, 33, 40].

In addition, thermal cameras have been used to quantify the energy of infrared (IR) radiation of an object with a temperature above absolute zero (-273 °C). Infrared thermal cameras capture the long-wave IR radiation (7.5–14 μm) emitted from crops and convert such radiation to electrical signals [41]. Plant temperature measurement has been primarily used to study plant water relations (e.g., stomatal conductance) since a major determinant of leaf temperature is the rate of evaporation or transpiration from the leaf [41]. Canopy temperature is highly correlated with canopy water stress and is used to capture plant responses to biotic and abiotic stresses [38]. Canopy temperature and the calculated crop water stress index (CWSI, Table 3.1) are found to be related to leaf water content [42,43,44] and show great potential for yield prediction [14].

3.1.3 UAV Imagery Data Processing Pipeline

Generally, the UAV imagery data collected are in the format of interval snapshots (still images) or videos. A large number of images may be collected for a large field. Figure 3.1 illustrates one of the typical data processing pipelines for UAV images, including image pre-processing, image processing, and analysis. The pre-processing steps consist of generating geo-referenced orthomosaic images and digital surface models (DSMs) [16] from the collected hundreds and thousands of sequential images or videos. The processes are usually conducted using customized or commercial UAV image stitching software such as Agisoft PhotoScan (Agisoft LLC, St. Petersburg, Russia) and Pix4D (Pix4D S.A., Prilly, Switzerland). Geo-referenced orthomosaic images and DSMs are further processed using advanced image processing techniques to remove background, segment plants, and extract image features. The image processing can be performed using commercial or open-source software such as Matlab (The MathWorks, Inc., Natick, MA, USA) and OpenCV (https://opencv.org/). The final step, image analysis, uses advanced ML models to translate the original images or extracted image features to useful information that is important for crop management. Some example applications include seedling assessment and stand count, water stress identification, and yield estimation. Some widely used commercial and open-source software for this step include Matlab, RStudio (RStudio, Boston, MA, USA) and PyTorch (https://pytorch.org/).

Fig. 3.1
figure 1

General steps for UAV imagery data collection, processing, and analysis

Full size image

3.2 UAV Systems in Cotton Production

3.2.1 Field Management for Cotton Production

Cotton production includes several key growth stages, as shown in Fig. 3.2, including seed emergence, seedling, squaring, blooming (flowering), open boll, and harvest, that need different management strategies. The life span of cotton is about 150 – 180 days in the United States [3]. Germination and emergence usually take 5–15 days depending on the weather condition, soil temperature and moisture [45,46,47]. Some of the major factors that influence cotton growth and yield include planting rate, planting depth, row spacing, and seed placement [48,49,50]. Early stage field management in crop emergence and seedling establishment includes emergence and stand count assessment [50, 51] and weed control [3]. Pests, such as spider mites, aphids, whitefly and lygus bugs, are the primary targets to be managed at the stage after the first square [50, 52]. In addition, irrigation management is also an important practice in cotton production during square to bloom growth stages to improve yield [3, 13, 53,54,55]. Lastly, harvesting at the right time can potentially reduce crop yield loss.

Fig. 3.2
figure 2

Cotton development and related field managements. DAP: days after planting

Full size image

Based on the general field management for cotton production mentioned perviously, the following sections will discuss the applications of UAV-based RS systems in making decisions on these field management. The sections are arranged based on different growth stages: (1) cotton emergence evaluations, (2) full-season cotton growth and health monitoring, and (3) yield estimation. The contents consist of the progress and advances of UAV applications in cotton production and the potentials and challenges of adopting UAVs in large-scale cotton production. Figure 3.3 summarizes the general approaches (sensors and image features used) and associated applications of using UAV-based RS systems in cotton production.

Fig. 3.3
figure 3

UAV systems for cotton monitoring from emergence, development to harvest. RGB: red–green–blue; MS: multispectral; HS: hyperspectral; VIs: vegetation indices

Full size image

3.2.2 Cotton Emergence Assessment

Crop emergence is an important agronomic factor for field management in early stages. Cotton emergence is usually assessed based on plant population, stand count, uniformity, seedling size, and etc. [50, 51]. Accurate and timely assessment of cotton stand count and seedling size helps farmers to make important management decisions, such as replanting, to reduce the yield loss due to missing plants [52]. Meanwhile, accurate cotton emergence information can be used to quantify the impact of soil and environments on crop emergence [56, 57], which may help farmers make optimal decisions on seed placement and planting. Conventionally, cotton emergence is assessed through visual observation (manual counts) in a small number of sampling sites [58], which is time-consuming, labor-intensive, and not suited to cover a large production field. Hence, UAV-based imaging technology has been used to assess some cotton emergence parameters such as stand count, canopy size, seedling number, and uniformity based on plant spacing as indicated in Table 3.2.

Table 3.2 Studies related to cotton emergence assessment (stand count, canopy size, and seedling number) using UAV imagery
Full size table

The primary challenge of assessing the cotton emergence using UAV imaging systems is the small seedling size at the early stage. For instance, their diameters were 2.4 and 3.5 cm on 11 and 17 DAP, respectively [61]. Hence, sufficient image resolution or GSD is needed and high-resolution RGB cameras have been widely used compared to other cameras (Table 3.1). With a low flight height of 10 to 20 m, images taken from the RGB cameras can achieve a better GSD ranging from 0.25 to 0.9 cm pixel−1. However, using RGB cameras give another challenge on image segmentation to remove the background and detect the small seedlings. The large portion of soil background in the images caused the color distortion of seedlings due to the strong reflectance of soil [62] and the color contrast between crops and soil was weak [17]. Previous research indicated that RGB images were potentially affected by sunlight conditions and suggested that using multispectral images with near-infrared (NIR) spectral bands could be more efficient for crop seedling segmentation [63]. For example, Feng et al. [22] assessed cotton emergence using a UAV-based hyperspectral imager for quantifying cotton plant density and uniformity. However, the accuracy for plant density could only achieve 0.61, which was lower than that of the studies using RGB cameras. The advantage of using hyperspectral imagers is the higher spectral resolution, which can be useful for other stress studies during the growing season, but they are limited by their higher cost and a large amount of data with proper data processing and analysis needed [64] as compared to simple RGB cameras.

Deep learning (DL) models are widely used to directly locate and detect each cotton seedlings [60, 61] and extract information such as average stand count and canopy size [22] from the UAV images. The DL models used included Resnet18, YOLOv3, MobileNet, and CenterNet and achieved the highest R2 of 0.98 [22, 60, 61]. One example of the DL model used by Feng, Zhou, et al. [14] is illustrated in Fig. 3.4, where the DL model was able to extract “hidden” information (a subtle difference) using multiple convolution and pooling layers to distinguish seedlings and background information.

Fig. 3.4
figure 4

The Resnet18 deep learning model used for cotton stand count and seedling canopy size estimation in the study of [22]

Full size image

The commonly used image processing pipeline (Fig. 3.1) requires stitching collected images using commercial software. The image stitching process may take a long time when thousands of images are collected from high-resolution cameras and at low flight height in field-scale studies. This issue may restrict their usage for PA applications to conduct timely management practices at the right time with accurate data. To improve the efficiency, Feng, Zhou [22] proposed a novel pipeline, as illustrated in Fig. 3.5, to process and analyze each UAV image directly to avoid image stitching procedure. The pipeline can process one image (20 M pixels) in about 2 s to extract the information of emergence, which provides an alternative method to assess cotton emergence in near real-time. It is possible to implement the image processing pipeline to an edge computing system that is integrated with a UAV system for real-time and on-site data processing and decision making. To conclude, DL and real-time image processing will enhance the UAV-based imaging systems in the cotton emergence assessment in commercial farms in the future.

Fig. 3.5
figure 5

A framework (pipeline) for processing single frame of UAV images in a near real-time manner. The pipeline was validated in the study of cotton emergence evaluation [22]

Full size image

3.2.3 Cotton Growth Monitoring Using UAV-Based RS

Crop growth and production are complex and determined by many factors, including crop genotypes (varieties), environments (e.g., weather, soil, microclimate, and location), and agronomic management strategies [65]. During the cotton growing season, several field management practices, including irrigation and chemicals (fertilization, pesticides, and herbicides) applications, need to be optimized to achieve optimal production. High-resolution site-specific crop information is needed to determine crop stresses for variable rate applications.

Irrigation is one of the most important management practices in cotton production since cotton’s growth and yield are sensitive to water deficit [66, 67]. The optimal irrigation schedule can be made according to the cotton water demand to replace the water loss due to evapotranspiration (ET), determined by cotton varieties, weather conditions, soil texture, and irrigation treatment [68, 69]. Infrared thermal cameras mounted on UAV had been widely used to detect cotton water stress by first estimating the canopy temperature and computing stress indices such as CWSI in Table 3.1 [44, 70,71,72]. Stomatal conductance has been used as one of the ground truth data to describe crop water stress response. Studies have shown a moderate to high correlation between stomatal conductance and CWSI computed using canopy temperature from UAV-based thermal cameras with r = -0.48 [71] and R2 = 0.66 to 0.91 [44, 70]. One of the challenges in using thermal imagery from thermal cameras is the background (i.e., soil and residue) removal since the pixel numbers from the thermal images are based on temperature values and not color information. Hence, some other image segmentation methods have been used, such as edge detection algorithms (Canny, Prewitt, and Roberts methods) [44]. Some thermal cameras have visible cameras attached, which could be used for the image segmentation based on visible images and creating a mask and co-registered with the thermal images to remove background [73]. These procedures are required to ensure that only crop canopy temperature is used in the crop water stress indices to improve the estimation accuracy.

Besides thermal cameras, some studies also demonstrated the usage of VIs computed from RGB and multispectral images in determining the cotton water stress and plant water content [44, 70, 74]. For instance, moderate to high correlations (significant r = 0.3 to 0.9 regardless of negative or positive correlation; R2 = 0.6 to 0.9) were found between different VIs from multispectral cameras with plant water contents at different parts (leaves, petioles, buds and bolls, stalks) [74]. When relating the different VIs to stomatal conductance, mixed results were found with low to high R2 (for example, NDVI ranged from 0.01 to 0.89) and their R2s were mostly lower as compared to thermal images [44, 70]. However, RGB and multispectral cameras are frequently used for other purposes such as weed and pest detection and mapping for UAV-based variable-rate spraying during the growing seasons as delineated in the following.

Using UAV-based RS systems is a promising way for weed, pest, nutrient and disease control in commercial cotton fields [3, 50, 52, 75, 76] and harvest aids [77, 78]. The key to efficient UAV-based variable-rate spraying is to identify the right regions of the cotton field and the right volume and time of spraying [78,79,80,81]. Optimal crop managment decisions are usually made according to cotton plant growth status that can be quantified by VIs [82, 83], canopy coverage [80], boll opening rate [77,78,79, 84], and pressure of weed [17, 85, 86], disease [87,88,89,90,91] and pest [81] of cotton fields. Advanced image processing methods and ML models (such as support vector machine, k-means classification, maximum likelihood, and random forest) are still the key for weed and disease identification and management decision making [81, 85, 87, 88]. Considering the complexity in field background of the images, DL methods are also promising image processing tools for cotton leaves disease and stress identification [91, 92].

Overall, UAV-based imagery from RGB, multispectral, and thermal cameras has demonstrated their great potential for in-season cotton field management. Some applicaitons include irrigation scheduling through plant water content and stress detection, variable rate chemical applications through mapping weeds and pests, and UAV variable-rate spraying. Most of these studies only used a single factor when developing the estimation models, such as crop response to irrigation [93, 94] or fertilizer [95, 96]. Few studies have integrated the interaction effects of environment and management due to the lack of long-term data and efficient tools for developing reliable models [97]. With the advance in the UAV-based RS, high-resolution imagery data can quickly quantify characteristics of crops and soil, which brings challenges in efficiently processing and analyzing the big data of different variety, resolution, and data structures. Hence, future studies will be focused on more integration of different UAV-based cameras with more advanced ML and DL modellings to explore the relationships between soil properties, weather conditions, and cotton growth variation [98].

3.2.4 Cotton Yield Estimation

Accurate cotton yield estimation could help farmers make better decisions on management, such as harvest, transportation, and storage [99]. Cotton yield estimation can also provide needed information to understand the interaction effect of environment and management on crop development and yield, which is important in developing precision management strategies for cotton production [14]. The UAV-based imagery has been used to predict cotton yield using different image features, including plant height, canopy cover, canopy temperature, VIs (NDVI and GNDVI), and open cotton bolls (Table 3.1) [14, 21, 100,101,102,103]. The UAV imagery data are typically collected on the critical stages, such as flowering and/or boll opening [14]. Research showed that the combination of the image features could estimate yield with R2 > 0.80 [14].

Several studies used high-resolution RGB images to detect and count open cotton bolls [101, 102]. Yeom et al. (2018) used image processing techniques (e.g. Otsu automatic thresholding) to extract the cotton bolls from the UAV images collected in the boll opening stage, achieving an accuracy of R2 close to 0.6. Xu et al. (2020) used two fully convolutional DL networks to extract the cotton bolls, obtaining a better accuracy of R2 = 0.8.

Current studies in yield estimation based on UAV imagery usually used data within a single year, i.e., using data from the same year for both model training and validation [104,105,106,107]. Future studies should focus on developing models for predicting crop yield of the coming years using historical data, which may have higher values for farmers to make proper decisions in advance to maximize their profit. In recent years, ML and DL techniques have been used to predict crop yield of future years using historical data of environment, management, and crop production [108,109,110,111]. Integration of the soil, weather, and crop data with different spatial and temporal resolutions utilizing advanced data analytic methods has the potential to improve the performance in quantifying field-scale crop growth and yield in a future year. One such example is a study conducted by Feng et al. (2021) that used an improved recurrent neural network (RNN) [112] model known as Gated Recurrent Units (GRU) [113] to analyze sets of sequential data, including soil, weather, and time-series imagery, for predicting cotton yield. As shown in Fig. 3.6, the architecture of the GRU comprises a reset gate and an update gate to control how much information is needed to be forgotten and memorized through the sequence.

Fig. 3.6
figure 6

Illustration of the Gated Recurrent Unit (GRU). a GRU continuously accepts inputs from a sequence. To easily understand the loop operation in the GRU, the GRU was drawn with the unfolded way. The ‘t’ represented each time step. b The architecture of the GRU

Full size image

The architecture of the yield prediction model based on the GRU is shown in Fig. 3.7, which included four layers, i.e., soil features (S_CNN) and weather features (W_CNN) pre-processed using CNN filters, GRU layers for NDVI prediction (‘GRU’ in Fig. 3.7), and the fully connected layers for yield prediction (‘FCL_2’ in Fig. 3.7). The input parameters of the GRU network included eleven soil features processed by the S_CNN and weather data from May 1 to October 29, 2019, processed by the W_CNN. The initial hidden-input vector (‘Init’ in Fig. 3.7) of the GRU was set as zeros and the corresponding output was assumed to be the NDVI in July (the GRU output of its first loop). The hidden outputs of the July GRU were passed to the next loop of the GRU unit (i.e., the GRU in August). The processed weather data from July 31 to September 3 were also passed to the August GRU, which had an output assumed to be the NDVI in August. The NDVI in July was used to replace the ‘Init’ to input to the August GRU. Similar procedures then defined the GRU processing units for September NDVI and for yield. The GRU unit was a three-month loop processing unit that each loop received the weather data from current month and the NDVI images from the previous month. The GRU was used to predict the NDVI spatial distribution of the current month based on the weather of the current month and the NDVI of the previous month.

Fig. 3.7
figure 7

The architecture of the GRU network. SL is sequence length, which was set to 1 in this study. BZ is the batch size for the training procedure. FCL means fully connected layer. IF is image feature (i.e., NDVI). All the GRUs highlighted with yellow color were the same loop processing unit in the network and had the same parameters. All FCL_1 units highlighted with green color were using the same parameters

Full size image

The study indicates that the yield prediction model based on DL is able to integrate data of soil, weather, imagery, and yield that have different resolutions and data structures. The yield prediction model shows an improved performance compared to the models without the full integration of related environmental factors, which could explain 68–84% of yield measured with the yield monitor (ground truth data), with the prediction errors of MAE = 247 kg ha−1 (8.9%) - 345 kg ha−1 (12.3%) at different years. In summary, the study indicates the potential of predicting the cotton yield of a future year based on soil texture, weather conditions, and UAV imagery. Future research could investigate more leading methods in integrating multiple source data of multiple years to estimate field-scale yield in the following years.

3.3 Summary

This chapter introduced the UAV applications in cotton production, from stand count, growth monitoring to yield prediction. UAV-based RS systems, equipped with RGB, multispectral, hyperspectral and thermal cameras, provide a low-cost and efficient tool for monitoring cotton growth at different stages. The collected information will then be used to improve crop management, such as seedling replanting, weed and disease control, and irrigation. Future research would focus on integrating multiple source data (i.e., environmental, ground, and UAV data) to develop near real-time tools using advanced ML and DL modellings for more effective and accurate cotton emergence, growth, and yield estimation for commercial applications.