Crop Scouting and Surrounding Awareness for Specialty Crops

Abstract

Until the beginning of the twentieth century, farmers typically managed small family farms where decisions used to be made according to the experience accumulated over long working days in their fields. The size of these fields allowed for a precise management of the land. However, with the advent of mechanization, the land managed by every farmer increased significantly and decisions began to be made through sampling, which often led to unrealistic generalizations. With the adoption of precision agriculture, the acquisition of massive data provided a way to systematically scout crops and enhance decision-making at management level. This chapter describes the main features desired for autonomous scouting machines. In particular, it focuses on the mechanical design and dynamic performance of agricultural robots, their reliability and safety needs, and the agile control of actuators. Also, the chapter presents a discussion on how usability and user acceptance would be impacted by the careful development of human-machine interaction through effective graphic user interfaces and ergonomic joysticks. As the principal task of scouting machines is the perception of the surrounding environment that requires the optimal combination of sensors, this chapter also explains the basic principles of nonvisual perception through ultrasonic devices and laser rangefinders, as well as vision systems and the information they retrieve as digital images. Specifically, it covers the implementation of noninvasive crop sensing devices to assess canopy properties, plant stress, and vegetative indices while a vehicle is moving in the field. Finally, it discusses how to deal with crop data through two-dimensional maps, by applying geospatial techniques for data smoothing and clustering, to reach the ideal situation in which a map delineates a few treatment zones directly applicable to the equipment commonly available in the farm. A use case is presented for vineyard scouting with ground robots for water status assessment.

1 Introduction to Crop Scouting

At the beginning of the nineteenth century, farmers managed limited pieces of land from which one family could survive. Decisions were made according to the knowledge accumulated after spending long days in small fields. The size of the fields allowed for a very precise management of the land. With the advent of mechanization, the land that every farmer could manage grew exponentially, and decisions had to be made through sampling, which often led to unrealistic generalizations. With the introduction of precision agriculture, massive data acquisition provides a way to systematically scouting crops as a tool for enhancing decision-making at management level. The successful application of precision agriculture principles to wine production, for example, leads to an optimized management of the vineyard, especially in everything related to grape harvesting. It is possible to make great wine by dumb luck or recipe, but not consistently; only by measuring key parameters can you make the best possible wine year after year (Cox 1999), using the variability inside the vineyard in your favor. However, methodical crop scouting to support key decisions on objective and precise field data is currently a dream for many producers due to the following main reasons:

• Monitoring cost : it is very expensive to pay operators for acquiring field data. Due to these high costs, it can only be done once per year at most, which impedes updating information and assesssing the evolution of crops along the entire growing season. The nitrogen content of the leaves, for example, continuously varies as fertilizer or water is applied.

• Low sampling rate : since it is not feasible to sample data every meter, measurements are realized once every 400 m² (20 m × 20 m) for advanced management of high-value specialty crops, and usually at a much less resolution for the rest, if data is sampled at all. Hence, data sampling rate is so low that conclusions are often biased. For example, the weekly assessment of grape ripening status in 6 weeks prior to harvesting is either biased (low-sampled) or unaffordable.

• Weight of current handheld devices: data sampling for hours with a handheld device that may easily weigh several kg becomes a torture for the operators, who in addition have typically to walk under the sun in the summer.

• Costs of service providers: although service providers offer data maps, airborne information tends to be of low resolution, and if several measurements are needed to check the evolution of the plants along the season, say every week or 2 weeks, the cost per season may become dissuasive.

The use of noninvasive monitoring devices for agriculture started with remote sensing, where optical sensors to perceive the vegetative activity of plants provided images that by recording the light reflected from the canopy, typically in the red and near-infrared spectra, allowed the calculation of certain vegetative indices such as the NDVI (normalized difference vegetative index). The acquisition of data from satellites, airplanes, or unmanned aerial vehicles presents a number of drawbacks that have discouraged their use in many practical applications in commercial orchards. Among them, it can be mentioned the need of a clear atmosphere for image taking, the low resolution when the aircrafts fly far away from the field, the unresolved legal problems in many countries caused by UAVs that require special licenses, the need for paying every image – something growers are not happy to do – and in general the lack of temporal flexibility to get vital information from the field that end users would like to have access to as soon and as many times as fields need. Aerial images have been convenient to assess vegetation differences – e.g., due to water stress or chlorophyll activity – over large areas of cropland, but result quite inappropriate for the precision needed in modern specialty crops, especially when the canopy is stretched along vertical structures. Due to these disadvantages, the use of proximal sensing has been growing in the last decade and is becoming the de facto management system of the future, through which precision agriculture may be successfully implemented in orchards.

There exist several parameters that have awaken interest among specialty crop producers, such as the nutritional content in the leaves or nitrogen uptake, water status and stress, maturity rate for fruits, disease and pest attack spatial distribution, and the estimation of yield at different periods of time before harvest. Yield estimation, in particular, has been claimed by growers as key piece of information for the optimal management of orchards, groves, and vineyards, as it can significantly fluctuate among seasons and also spatially within the same field and season. Although there are no commercial solutions yet, several approaches have produced promising results. Nuske et al. (2011) applied machine vision techniques to estimate yield in vineyards. The system uses a sideways-facing color camera of resolution 3264 × 2448 (pixels) with halogen lamp lighting to detect and count the number of berries per vine at 0.8 Hz, based on shape and texture, with a satisfactory detection rate, even for green grapes over green leaves. To reduce the presence of false positives (mistaken berries that were mostly isolated), as grape berries naturally grow in clusters, berries that do not have at least five other berries within their immediate neighborhood were eliminated. An average error of 10 % and a correlation coefficient of determination of 0.74 were reported when predicting the actual harvest weight with this experimental setting. As important as with vineyards is crop yield estimation for apple orchard management, mainly due to the fact that current manual sampling-based yield estimation is time-consuming, labor-intensive, and quite inaccurate. Optimizing fruit thinning and harvesting efforts as well as packing house strategies can reduce operation costs. To face this challenge, Wang et al. (2013) deployed a computer vision system that uses a stereoscopic camera with 28 cm baseline between two lenses of 11 mm focal length and artificial lighting (ring flashes) for yield estimation (number of apples) during the night. The system works well with both red (Red Delicious) and green (Granny Smith) apples in a tall spindle planting structure in such a way that apples are approximately 2 m from the stereo rig. The implemented algorithm uses the color space HSV (hue-saturation-value) to discriminate the apples from the background, even green apples from foliage, given the robustness of this model to decouple the light intensity from the color information in the image. For the red apple rows that received conventional fruit thinning, the computer was very close to ground truth, with errors around 3 %. However, apples were undercounted by 41 % for the trees without fruit thinning. Similarly, undercounts for the green apples, which have thicker foliage and more occlusions, reached 30 %. In order to account for occlusions, a finer calibration was added to the estimation model, lowering errors below 2 %. At the end, the orchard manager received a 2D yield map indicating the spatial distribution of red apples for the upcoming harvest.

It is no random coincidence the fact that both yield prediction cases exposed in Nuske et al. (2011) and Wang et al. (2013) together with the use case example for water stress monitoring of Sect. X require supporting structures that enhance canopy and fruit exposure. When tractors coexisted with animals to farm orchards, it soon became apparent that old orchards had been outlined for horses, and therefore, many tractors did not fit between rows, encountering serious problems to maneuver at the end of the rows in narrow headlands. As tractors eventually replaced animals, orchards were delineated with wider row spacing and larger headlands. This is an excellent example of how farming practices changed to better adapt to technology; it made no sense to design tractors with the size of horses to fit well in the traditional orchards. Several decades have passed and we are again facing a similar dilemma. Orchard automation, scouting, and site-specific approaches are not fully efficient with the way many fields are set today. Fruit exposure highly depends on the way trees are shaped. A critical barrier to yield prediction, and robotic harvesting of fruits in trees, has been the high percentage of fruits occluded by branches and leaves, which puts fruits out of reach of end-effectors commanded by computerized vision systems. However, new production structures are being incorporated for high-value specialty crops, some of them already implemented to facilitate mechanized operations. Such is the case of vineyards for wine production. The traditional goblet or head-trained vines are steadily being replaced by more modern vertical shoot position (VSP) trellis, where most of the bunches are openly exposed in either side of the vine. The percentage of fruit exposure may be further enhanced by field operations such as canopy defoliation. Figure 5.1 illustrates the differences between these two production systems for wine grapes; Fig. 5.1a shows a traditional goblet vine, whereas Fig. 5.1b represents the trellis system used for VSP architectures. As evidenced in the images, VSP is far more convenient for automation and data acquisition from a moving vehicle.

Fig. 5.1

Vineyard architectures: (a) head-trained vines or goblet (France); (b) vertical shoot position or VSP (Portugal)

Although wine-producing vineyards were pioneers in the adoption of vertical and flat supporting structures for specialty crops, other crops have been gradually introducing trellises to convert spherical canopies into vegetation walls. This kind of high-exposure architectures is favorable to automation because they are simple, narrow, accessible, and productive (SNAP). Figure 5.2a depicts the case for apples, Fig. 5.2b provides a production alternative for cherries, Fig. 5.2c shows a vertical structure for almonds, and Fig. 5.2d brings the innovative high-density plantations proposed for olive trees. Unlike orchards, row crops and short plants are easier to automate and monitor, as agricultural equipment is not tightly constrained inside rigid structures and GPS signal is less prone to be blocked.

Fig. 5.2

Vertical supporting systems for specialty crops: (a) apples (USA); (b) cherries (USA); (c) almonds (Spain); (d) olive trees being harvested with a modified grape harvester. (Spain, courtesy of prof. Antonio Torregrosa)

Given that proximal sensing requires keeping a limited distance between the measuring sensor and the target, typically between 1 and 3 meters, a ground vehicle is necessary to carry the data acquisition equipment and scan the field in a reasonable time while assuring a sampling rate with statistical significance. This excludes manual sampling and therefore favors automation and the introduction of intelligent off-road equipment in the field. One possibility is represented by conventional equipment – tractors, harvesters, and sprayers – endowed with the necessary suite of sensors and computers to automatically save crop data. Another option is to deploy a scouting robot to autonomously register large amounts of data, as described in detail in Sect. 5.2. This alternative has been gaining momentum over the last years, as a consequence of the rapid expansion of robotics and big data applications. The amount of data retrieved from commercial orchards is not massive and pervasive, generally speaking, and as a result, we are not into big data yet, but having access to a high-rate data-collecting machine will likely lead the way to it. A great number of financial publications coincide in forecasting a rapid growth of the service robots market, with a significant role of agricultural applications for the next two decades. Figure 5.3 shows the expected growth of the US agricultural robots market by application for 2015 (USD million) (Verified Market Intelligence 2018). Although milking robots and dairy management will still dominate the deployment of robotic solutions in the next decade, the introduction of digital farming through advanced management of soil and crops with precision agriculture techniques, in combination with a stronger presence of automation and intelligent systems in field farming, will result in the emergence of a growing group of first adopters who will manage their land at the same technological level reached by other industrial sectors. This transformation for the twenty-first-century farmer from laborer to digital age manager may be instrumental to attract young generations to agricultural production, counterweighting the risks of leaving such a high responsibility as feeding the world to an ever-aging population of farmers.

Fig. 5.3

US agricultural robots market by application (Verified Market Intelligence 2018)

2 Scouting Robots: Architecture and Design

There are two practical approaches to endow agricultural machinery with intelligence and automation: (a) modify conventional equipment by introducing digital technology and automatic controls and (b) design a new vehicle that integrates a mechatronics design from scratch. At the dawn of digital technology, the former type was mostly represented by large vehicles powered by diesel engines, while the latter consisted of small- or medium-sized platforms run by electrical drives and traditional lead batteries. At present, however, solutions are not so clearly divided, and large equipment manufacturers are already offering big tractors with a good dose of electrification, whereas innovative farm robots may incorporate small combustion engines to fulfill high-demand field tasks, such as blast spraying or mowing, in platforms propelled by lithium batteries, which have increased efficiency significantly thanks to research conducted by the automotive industry. The convenient size of a farm vehicle endowed with certain degree of automation is still an unresolved question. Productivity and liability are antagonist concepts that need to be balanced until finding a reasonable and cost-effective trade-off. Large machines such as combine harvesters are very efficient because they can harvest big amounts of product per day, but at the same time, they are hard to automate because of their size, weight, and power; if a navigation or control problem arises, it is very difficult to stop a machine of such magnitude. Small robots, or moderate-sized autonomous assistance platforms, on the contrary, entail lighter risks when operating in the field, but, unfortunately, are limited in their capacity to produce work because of small operating widths and low power autonomy. Better batteries and safer architectures will probably lead to larger off-road vehicles propelled by electricity on one hand, and farm machinery more modest in size but with higher levels of automation on the other.

When conventional agricultural equipment is modified to adopt digital farming techniques, it retains a highly effective mechanical design inherited from many years of continuous evolution. However, when a new farm robot is developed, there is a risk of overlooking its mechanical design in favor of software, sensors, or electronics. A vehicle built to operate in off-road conditions must include a strong chassis, a responsive suspension, and an effective steering mechanism; without them, its behavior will likely become inconsistent and unpredictable with time. Figure 5.4a depicts the computer-based design of the entire chassis for a vineyard robot, Fig. 5.4b provides a close-up evaluation of highly stressed and highly strained joints in its chassis (wheel motor carriers), and Fig. 5.4c depicts the four-wheel independent suspension constructed for the designed robot. We should evaluate the dynamic performance of agricultural robots as systematically as any conventional off-road vehicle, taking into consideration such crucial parameters as slippage, weight transfer, and traction. The size and type of the wheels, or the potential need of ballast, may improve field performance significantly.

Fig. 5.4

Mechanical design in agricultural robots: (a) 3D model for robot chassis (courtesy of Andrés Cuenca); (b) high stress joints in motor carriers; (c) independent suspension in actual robot

The dynamic performance of a farming robot, which mostly moves on off-road terrains, is crucial for the safety of the vehicle and the quality of the data collected on the fly. Its dimensions and weight distribution over the wheel axles are important design parameters, but the critical point for reaching a stable behavior is the optimal design of suspension and steering. The prototype of Fig. 5.4, for instance, was equipped with four custom-made coil springs with an elastic constant of 15,675 N·m⁻¹, with the purpose of improving the results of a previous prototype featuring commercial suspension springs with an elastic constant of 27,210 N·m⁻¹. In order to compare the dynamic performance of both prototypes (Saiz-Rubio & Rovira-Más 2016), a three-axis accelerometer was mounted on the front right wheel of each prototype to record vertical accelerations at a frequency of 16 Hz. The robots were set to overcome two obstacles at approximately 1 km·h⁻¹: first, a round curb of approximately 4 cm in height and 8 cm wide causing a sudden drop and, second, an office footrest forming a wedge of 45 cm in length and 16 cm in height (slope angle of 21°). Figure 5.5 shows the outcome of this comparison. As expected, the softer suspension of the second prototype (Fig. 5.4c) absorbs better the vertical accelerations provoked by the obstacles, especially with the curb.

Fig. 5.5

Suspension test for two prototypes of agricultural robots

The study of the most favorable suspension system for each particular robotic platform needs to be complemented by its corresponding design of the steering mechanism. When robots are endowed with autonomous navigation, this stage in the design process becomes fundamental. An advantageous steering system must provide crisp and accurate corrections when the robots advances in straight guidance between orchard or vineyard rows, but at the same time must allow sharp turns at the headlands to change rows with the minimum slippage. Figure 5.6 gives an overview of the key design components that form the steering mechanisms of scouting robots. In this particular case, as the robot is four-wheel-drive two-wheel-steer, the electric motors powering the front wheels must turn with them as the robot steers, which complicates the outline of the steering linkages. The model of Fig. 5.6a shows an example of a tie rod assembly that facilitates wheel turns with electric motors attached to each wheel. The actual rotation of the wheels is accomplished by displacing a sliding bar actuated by the pinion-rack coupling of Fig. 5.6b. In general, a significant change in the suspension system implies a reformulation of the steering linkage design. The steering system for the prototype of Fig. 5.4 was also improved by introducing slight caster and camber angles in the front wheels. A caster angle induces an inclination angle at maximum turns that facilitates the negotiation of sharp turns in a similar way automobiles and tractors steer (Fig. 5.6c). The light positive camber angle helps the robot self-center, which is key for automatic steering along the rows.

Fig. 5.6

Steering design in scouting robots: (a) tie-rod assembly; (b) pinion-rack actuation; (c) caster angle

The benefits of optimizing the steering system of a scouting robot must necessarily translate to the actual field, where the modified robot will have to outperform the navigation aptitudes shown by the robot before the improvements. A comparison similar to that of Fig. 5.5, but focused on automated guidance, is plotted in Fig. 5.7. The robot of Figs. 5.4 and 5.6c was automatically guided in Spain, whereas a previous prototype with a simpler steering mechanism was tested in France. Both vineyards had a row spacing of 2 m, and the forward speed oscillated between 1 km·h⁻¹ and 1.8 km·h⁻¹. The plot graphs the Ackerman angle registered by the robot’s central computer and clearly reveals a superior behavior for the enhanced steering system (blue line), which behaved more stable than the old prototype (red line), that additionally showed some asymmetry in the actuation of the wheels. Experience taught that slight deviations in the construction of the tie rod assembly had significant consequences in the navigation performance of autonomous vehicles. But the opposite was true too; a committed design of the suspension and steering systems as a whole resulted in gratifying outcomes.

Fig. 5.7

Steering performance comparison between two robotic prototypes automatically guided

The deployment of automated systems, even in the case of low-level automation, always involves solving important technical difficulties. Out of all of them, the biggest barrier to commercialization has typically been reliability and safety. There is a long way between a working prototype and a product ready for the market. As a result, it takes years for a solution to hit the market and become accessible to average end users. A safe solution must be reliable in both hardware and software. The environmental conditions found in agriculture – low temperatures in the winter, high humidity, strong sun radiation, application of chemicals, and potential knocks – are usually a threat to delicate off-the-shelf electronics, so a first step in making scouting machines stronger is by protecting sensors, electronic cards, and wires from weather aggressions, in addition to using components with an IP rate of 65 or above. Once hardware is protected and secure, communication among sensors, actuators, and computers or processors has to be granted. Furthermore, this communication must be permanent and take place at the necessary frequency to assure a stable and predictable performance. Even though each sensor requires a particular interface, we can enunciate some general rules. Serial data are usually transmitted through universal serial ports (USB) or RS-232. When both protocols are available, the latter is preferable for agricultural applications as it is stronger and can be tightly screwed, providing a sturdy link between sensors and computers. The fact that computers permanently number serial ports is a guarantee of stability before device confusion. Typical sensors that use RS-232 communication are GPS receivers and lidar rangefinders. The increasing popularity of USB ports for video streaming has led to their dominance in imaging sensors over the traditional I2C and firewire (IEEE 1394). However, caution is necessary when using USB-connected devices; first, the connector itself has been designed for desktop computers and, as a result, is weak and easy to unhook under the vibration of off-road equipment; and second, the power supplied through USB ports is limited to 5 V up to 5 A, which many times requires the addition of extra power for power-avid devices.

The transmission between sensors and computers – or processors – must be set at a conservative sample rate, in such a way that if a few measurements are missed, information is still available in a timely manner. However, the control of actuators, especially in navigation and safeguarding, has to be as reliable as possible. For such cases, the commands sent by the processing units must arrive integral and at the right pace. This necessity immediately discards any kind of wireless communication between the central computer and the control of motors and electrohydraulic valves. The most accepted protocol in industry for serial data transmission is the controller area network (CAN) bus, developed by Bosch in the 1980s. It allows controllers, sensors, and actuators be connected on a common serial bus that behaves like a “data highway.” Based upon the CAN bus, the manufacturers of agricultural equipment are developing the ISO bus, a protocol to connect farm implements and tractors in a general, safe, and efficient way, with only one control console for all the implements regardless of the brand. This standard is still in progress but many manufacturers already offer it.

There has been a considerable gap between technology development and user adoption with regard to digital farming, and a significant contribution to it comes from the lack of harmony between solutions offered and user needs. Some solutions tend to be very complex and not in tune with current demands made by growers. The average user is not an information technology (IT) expert with an engineering degree; rather, we expect to find practical workers in search of consistent solutions. Intelligent systems must be straightforward and intuitive, easy to use, and easy to explain. In this context, only indispensable sensors and functions should be used to keep complexity at reasonable levels. Adding extra sensors, even if they are low cost, will increase the failure rate of any system. To ease usability, human-machine interaction must be clear and easy, with well-defined graphic user interfaces (GUI) and ergonomic joysticks for manual operations. For autonomous operations, a well-designed network of emergency stops is mandatory to stop the vehicle in case of unexpected behavior. Figure 5.8a shows the graphic user interface of the robot utilized in the use case of Sect. 5, and Fig. 5.8b illustrates its manual control with an ergonomic joystick. Overall, the skills required to operate these scouting vehicles should be similar to using a cell phone and not much more advanced.

Fig. 5.8

User-centered design in agricultural robots: (a) intuitive GUI; (b) manual control with ergonomic joystick

3 Surrounding Awareness for Scouting and Data Collection

The principal task of any scouting machine is the perception of its surroundings, and that will only be possible with a combination of sensors holding the capacity of measuring physical properties of what is enclosed in the machine’s vicinity. Not only the sensors must have the capacity of perceiving these physical properties in a quantitative or numerical format, but they also must facilitate their recording in a data-logging device for their further processing. Data may be processed in real time for immediate actuation, but the majority of scouting applications require onboard data storage. The development of sensors is in continuous growth, and we will see new devices hitting the market in the following years, but the implementation of some sensors over the last decade has resulted particularly attractive and effective for the complex scenario of agricultural environments, where open off-road terrains that require working outdoors pose challenging situations to the advanced manager willing to apply the principles of precision agriculture and digital farming. This section reviews some of the most popular sensors onboard intelligent vehicles for crop scouting applications, providing a general overview of their workings to show their potential, but without getting into technical details.

3.1 Visible Light Machine Vision

Machine vision is the computer version of the farmer’s sight; the eyes are represented by cameras and the brain by computers. The output of vision systems are digital images. A monocular camera yields digital images of a scene at a given rate (frames per second or fps), and a stereoscopic camera produces a depth image as a result of comparing two images taken simultaneously by two equal cameras mounted on a rigid rig. In this section, we will deal with cameras that perceive the environment in the visible range, which means that the cameras register the same scene seen by humans, filtering out other electromagnetic sources such as ultraviolet or infrared. A digital image consists of little squares called pixels (picture elements), each of which carries information on its level of intensity. If the image is in black and white (technically called monochrome image), the intensity level is in reality a gray level between a minimum value (0) and a maximum value (i_max). The number of gray levels depends on the number of bits (binary digits) in which the image has been coded. Most of the images used in agriculture are 8 bits, which means that the image can distinguish 256 gray levels (2⁸) and the maximum value is 255 representing pure white. In practical terms, our eyes cannot distinguish so many levels, and 8 bits are many times more than enough. When digital images reproduce a scene in color, pixels carry information of intensity levels for three channels red, green, and blue, leading to RGB images, whose processing is more intricate than monochrome images and therefore falls outside the scope of this chapter.

Monocular cameras constitute the basic component of typical vision systems and retrieve powerful and useful information from plants, trees, and other objects of interest in agricultural fields. In addition to selecting the camera manufacturer and image type (monochrome or color), users must also choose important technical parameters such as the lens focal distance, the size of the sensor, and optical filters when there is a need to block determined spectral ranges (colors). Although the detailed description of these parameters would take too long for an overview like this, the focal distance (f) is related to the scope of scene that fits into the image, and as a result, the smaller value of f, the wider angle of view covered by the camera, and therefore a bigger portion of the targeted scene will be captured in the resulting image. Once images are taken, we have completed the first step of the vision process – image acquisition – but the images still have to be properly interpreted to be meaningful, the image processing stage, which involves the delicate step or extracting the useful information from the image, something that must be efficiently carried out by computer algorithms tailored for each given application. Figure 5.9a shows a monocular monochrome camera used to capture zenithal scenes from a vineyard, and Fig. 5.9b reproduces the results of a color-based segmentation algorithm to extract oranges from a citrus grove, application that uses the HSV color space in similar fashion to the apple detection algorithm of Wang et al. (2013).

Fig. 5.9

Monocular vision for scouting: (a) digital camera; (b) color image processed to detect oranges

Even though digital images reproduce scenes with great detail, this representation is flat, that is, in two dimensions, whereas the real scene is actually in three dimensions. What dimension is left then? The depth, or distance between the camera and the scene. In the image shown in Fig. 5.9b, we can locate a given orange with precision in the horizontal and vertical axis, but we cannot know how far it is. This information would be essential, for example, if we had to program a robotic arm to retrieve the orange, as it could be out of reach. Stereo cameras allow the acquisition of two (or more) images in a certain relative position to which the principles of stereoscopy can be applied. These principles mimic how human vision works, as the images captured by our eyes in the retinas are slightly offset – as are our eyes – and this offset, known as disparity, is what allows the brain estimate the depth. In agriculture, stereovision is used for assisting in autonomous navigation (Rovira-Más et al. 2015) and for generating three-dimensional (3D) maps of crops (Rovira-Más et al. 2008), which allows the estimation of plant growth and canopy vigor. Figure 5.10a shows a compact stereo camera mounted on a rice harvester, and Figs. 5.10b, c and d, depict a commercial vineyard monitored with a stereo camera mounted at the front of a tractor, its depth image (c) where darker pixels indicate farther distances, and its 3D virtual representation (frontal view d) that depicts the canopy at both sides of the vehicle and the row spacing ahead showing that is free for navigation. Notice that stereo matching is a pixel-by-pixel operation that requires texture to identify the same objects in both images of the stereo pair. Therefore, image areas without enough texture (dark shadows under the canopy or uniformly lit patches of soil) produce (white) pixels with no information in the disparity image (c), eventually creating voids in the 3D representation of the scene. These voids are easily identifiable within the vineyard canopy of Fig. 5.10d.

Fig. 5.10

Stereo vision: (a) binocular camera; (b) real scene; (c) depth or disparity image; (d) 3D frontal representation

3.2 Biosensing for Crop Monitoring

With the development of non-invasive crop sensing tools that can operate on the fly, i. e., while a vehicle is moving in the field, the availability of crop information has grown significantly. The accessibility to massive field data is key to apply the principles of precision agriculture, a modern approach to grant economical and environmental sustainability in farms. Machine vision with cameras sensitive to non-visible spectral bands, infrared radiometers, and other optical devices can produce amounts of data that growers can fit into decision support models to make better choices. Among the crop parameters that have risen more interest in agriculture, it is worth mentioning the canopy volume, the temperature of the leaves measured with infrared radiometers (Sect. 5.5) or, alternatively, with thermographic cameras yielding the CWSI (crop water stress index), and plant vigor assessed through the calculation of the NDVI (normalized difference vegetation index) or the NBI (nitrogen balance index). The use of vegetative indices, such as NDVI, NBI, and CWSI, provides objective and quantitative methods to evaluate the status of crops at any given moment of the productive cycle and, therefore, allows farmers reach a deeper understanding of how crops evolve with time. The fact that many intelligent farm vehicles include a GPS receiver facilitates the acquisition of crop information in map format, which is convenient to compare the spatial variability of key parameters and the generation of historical data series. At present, there exist several off-the-shelf devices to calculate the NDVI in real time and apply a variable rate of fertilizer according to the nutritional status of plants. The systematic record of plant physical properties at high resolution is known as plant phenotyping and is becoming instrumental to understand the relationship between the genetic makeup of a plant and its complex traits such as growth or water stress resistance. Figure 5.11a shows a multispectral camera that allows the calculation of NDVI and NBI, and Fig. 5.11b depicts an infrared radiometer that can estimate the canopy temperature of specialty crops.

Fig. 5.11

Sensors for crop monitoring: (a) multispectral camera; (b) infrared radiometer

3.3 Nonvisual Range Perception

Ultrasonic rangefinders are devices that use sound to measure distances, and for that reason, they are also known as sonar sensors. The principle that lays behind sonars is the fact that the speed of sound is known (343 m·s⁻¹ at 20 °C), and measuring the time that the wave needs to hit an obstacle and return – the echo – allows the estimation of that distance. The speed of sound in the air depends on the ambient temperature, but this relationship is known and easy to apply. However, the use of sonar sensors in agricultural fields is subjected to some difficulties that can be detrimental for the accuracy and consistency of measurements. The sound wave sent by the sensor has to hit an object and return to the sensor receiver that must capture it to measure the time used by the wave in its round trip. The physical properties of the target objects, especially their reflective nature, are essential to obtain reliable results. Sound waves do not behave as linear beams; instead, they propagate in irregular cones that expand in coverage with distance. As a result, when the target object is uneven (as canopies) and the cone is not narrow, the measurements may become too noisy. An important design feature to consider is the distance between adjacent ultrasonic sensors, as echo interference is a source of unstable behavior. Overall, sonar rangefinders are helpful sensors to estimate short distances, say up to 2 m, cost-efficiently, and when accuracy and reliability are not essential, for instance, when detecting the distance of a vigor-estimating sensor to the canopy for fine-tuning measurements. Figure 5.12 shows a robust ultrasonic sensor to estimate the lateral distance of a robot to the canopy of grape vines (a) and a lower cost sonar to issue warnings when the robot moves backward and objects interfere in its trajectory (b).

Fig. 5.12

Ultrasonic sensors for estimating distances in agricultural vehicles

Lidar stands for light detection and ranging and is an optical device that provides distances to objects. Although different light sources can be used to estimate ranges, most of the lidars use laser pulses because the beam density and coherency make them very accurate. However, laser beams are very narrow and one static emitter cannot cover the amplitude of zones typically needed. As a result, off-the-shelf rangefinders come in two working modes: (1) a rotating head with an emitter that sweeps an area that can reach 270° and (2) a stationary head with several emitters equally spaced to cover an area, usually smaller than that spanned by rotating heads. Lidars are not affected by sunlight unless it hits the emitter directly and work excellently at night when most vision systems provide poor perceptive capabilities. Machine vision and lidar are technologies that complement well each other to provide local perception to intelligent vehicles. Figure 5.13 shows a perception unit comprising a stereovision camera, two sonar sensors on the sides for lateral perception, and a frontal lidar rangefinder with a motionless head consisting of 11 emitters that cover an angle of 88°, with beams angularly spaced by 8.8°.

Fig. 5.13

Lidar rangefinder (black component) with 11 beams covering a horizontal field of view of 88°

4 Crop Monitoring and Mapping

Regardless of the level of intelligence embedded in a farm machine, field work has to be carried out in the most efficient way. In fact, automation and control is introduced to enhance performance of machinery and therefore increase efficiency, safety, and comfort for the operators. In a general formulation of crop scouting, a vehicle operating in the field can be ground-based or airborne. In either case, it will have to cover an area in a given time, and this is actually a relevant parameter when comparing different options to do a given task, i.e., the number of ha·h⁻¹ that the machine can cover. Let us put the example of a sprayer equipped with a multispectral camera to monitor a vineyard NDVI with a row spacing of 2 m. The sprayer is supposed to pass along every row, with an average speed of 5 km·h⁻¹. The time invested in headland turns for this example has been estimated in 20 % of the total operation. At that pace, the sprayer advances 1.4 m every second, and considering the regular row spacing of 2 m, the area covered is 2.8 m²·s⁻¹, or 10,080 m² per hour, which can be approximated to 1 ha·h⁻¹. Taking into account that a working day is equivalent to 8 h, and 20 % of the time is lost in headland maneuvers, the final efficiency will be approximately 6.4 ha·day⁻¹. This is the working efficiency we need to compare among alternative solutions.

At present, there is a raising interest in electrical drives and actuators. The automotive industry has pushed the development of high-performance batteries for hybrid or totally electric vehicles. Electric actuators facilitate vehicle control and provide a clean alternative to fossil fuels. This technological move is obviously affecting agricultural equipment, and major manufacturers are already developing innovative vehicles with an ever-growing presence of electrical components. However, performing demanding tasks in off-road conditions requires a considerable amount of energy, and it is critical to estimate the autonomy of electrically powered vehicles. The efficiency of the example mentioned above will hardly be accomplished if the sprayer cannot run for at least 8 h per day. The pervasive advent of unmanned aerial vehicles for data acquisition is greatly dependent on battery performance and payload. The use of renewable sources of energy, while recommendable, must always be compatible with the actual needs of power and workload encountered in the field to satisfy a minimum degree of efficiency.

Crops can be monitored by simple visual inspection or just by reading a sensor output in a display, but the large amount of data logged by most of current sensors, even if running at low frequencies, makes this approach inefficient. Some applications require a real-time processing of sensor data, for example, variable rate fertilizing with an NDVI-based nitrogen assessment, but the majority of monitoring applications require saving data for their later processing. The storage of data in numerical tables and oversized matrices is suitable for engineering work, but field managers and crop growers find it more intuitive when field data is conveyed in a map format. Therefore, the preferred method to deal with crop data is through two-dimensional maps. The fact that GPS receivers are ubiquitous and easily accessible favors the coupling of crop information with global positioning. However, the standard geodetic coordinates coded in NMEA (National Marine Electronics Association) strings that all GNSS receivers get are not convenient for the precision and detail needed in proximal sensing applications, where the position of individual trees is often necessary. In addition, working in degrees, minutes, and seconds is not intuitive for the daily planning of field managers, who would rather use meters instead of degrees. The local tangent plane (LTP or NED) offers several advantages that make it the ideal coordinate frame for proximal sensing monitoring. It uses Cartesian coordinates north and east, which are by themselves intuitive for growers to orientate in the field, and also features a local origin chosen by the user in every field. This allows both, the use of Euclidean geometry for the calculation of distances and areas, and having convenient magnitudes for the coordinates, which will normally be expressed in meters. The flat coordinates NED (North-East-Down) provide global references for scouting vehicles and therefore for each crop measurement carried out from them by simply translating each sensor location to the GPS antenna’s location.

Choosing an advantageous coordinate system for representing the spatial distribution of field data is a necessary step, but the way crop information is displayed in the user map is crucial. Accurate maps of difficult interpretation are useless for the average field manager. Three features may ruin the usability of monitoring maps: (1) the measurement scale has so many intervals or classes that practical actuation is not easy to figure out; (2) measurements are not coinciding in space, and therefore multiple variables cannot be correlated; and (3) the parameter displayed in the map has no practical meaning or application for the daily operations of the farm, either because variability is too high or the parameter itself is more scientific than operational. The first two drawbacks are counterweighted by discretizing both the space and the measurements. A grid approach that maintains global references in LTP coordinates offers good guarantees for map usability (Rovira-Más 2012) and facilitates the comparison of data through the years. This approach lets users decide the cell size – map resolution – and the number of classes that are manageable for each specific situation, typically represented in a friendly color scale. The temperature maps of Fig. 5.19 are examples of grid representations in the LTP frame. Notice that a scouting robot recording a temperature measurement every second will gather several data points for each cell of 4 m² area, and without computing the average measurement of each cell, comparison and correlation between zones is not practical.

The third drawback mentioned above requires a different sort of solution. A grid map that directly represents raw data acquired in the field will naturally present a lot of variability, mainly with cells of reduced area. It may be the case of having a crop map with the right spatial resolution and with a reasonable interval for the studied parameter, but still yielding a large dispersion among cell values. For such cases, and if no real-time decisions are to be made, geospatial techniques for data smoothing and clustering become very practical to make crop maps operative. When field data is scarce, interpolation has been the typical technique used, but many times it has led to artificial mapping that has little to do with field reality. When massive data is available, however, smoothing and clustering are the most appropriate ways to treat data. In any case, geostatistics in precision agriculture must avoid the common plague of overfitting and overtransforming data (Schueller 2010). The ideal situation will be when a crop map has a few treatment zones for a parameter directly applicable with the equipment available in the farm. To reduce the number of treatment zones, cells that fall within a “close” vicinity must be classified as equal dose. The problem is how to determine the radius – known as the range – around any given cell above which influence from other cells is negligible, following the accepted principle of Tobler stating that everything is related to everything else, but nearby objects are more related than distant objects (Tobler 1970). Clustering techniques are usually optimal when data follow a Gaussian distribution. The estimation of the maximum distance of influence can be found through the plotting of semivariograms, which consists of computing statistical variances (Y axis) for growing distances (X axis) until the variance remains close to a steady state called the sill. The geometrical locus of the variances is then fit to an equation whose intersection with the sill establishes the maximum range, the distance considered for clustering and smoothing data. Figure 5.14a shows the distribution of nitrogen (0 is poor; 1 is maximum) in leaves of a vineyard where 1062 measurements were made noninvasively (Saiz-Rubio et al. 2017). Note the closeness of the actual distribution of nitrogen values with a perfect Gaussian distribution. Figure 5.14b plots its corresponding semivariogram where the best fit was exponential and the maximum range found was 30 m.

Fig. 5.14

Normal distribution of foliar nitrogen (0–1) in a vineyard (a) and semivariogram (b)

Once the range has been set, a raw map plotting data directly retrieved from crop sensors can be further processed to indicate site-specific treatment zones. A multitude of algorithms may be applied, taking into account the range as the radius of influence. A simple way of smoothing data would be by estimating the median in the surrounding cells considering two concentric rings, that is, by finding the median of each cell and its surrounding 24 cells (5 × 5 window) (Saiz-Rubio et al. 2017). Figure 5.15a shows a grid map of nitrogen in leaves directly plotted with sensor data, and Fig. 5.15b plots the result of applying the 25-cell smoothing filter based on the median. The filtered map facilitates decision-making based on zoning and allows a rational use of variable-rate equipment.

Fig. 5.15

Original grid-based nitrogen map (a) and results after applying smoothing techniques (b) (Data courtesy of Mª Paz Diago)

5 Use Case: Vineyard Scouting with Ground Robots for Water Status Assessment

The Alto Douro Wine Region is a UNESCO World Heritage site centered on the Douro (or Duero) river, which is sheltered by mountains from the coastal influence of the Atlantic Ocean. This region is home of the most important wines elaborated in Portugal and includes both table wine – typically known as Douro wines – and the world-famous Porto wine. Producing wine in the Alto Douro is hard due to steep terrain and high temperatures, and even though wine makers are used to these conditions, and grape vines, most of which indigenous local varieties, are acclimated to the region, the steadily rising temperatures of the last years have posed serious concerns among growers, who need to keep the yield and reputation of their products before an uncertain availability of water for irrigation. An excess of water stress in vines may lead to important economic losses and long-standing damage to renowned Porto brands. Figure 5.16a shows the effects of water stress (1 August 2018) in the canopy of Touriga Nacional vines, where plant coverage is too poor. A traditional method in the region to palliate the effect of extreme sun radiation has been to spray the vine canopy with lime to lower leaf temperature. Figure 5.16b depicts the deposition of lime on the leaves.

Fig. 5.16

Water stress in grape vines: (a) scanty canopy; (b) lime treatment to decrease leaf temperature

The natural response of a stressed vine is to limit transpiration by closing the leaf stomata, with the purpose of preserving plant water as much as possible when soil water is not available. The predominant method so far to assess the degree of water stress in a plant tissue has been by estimating the water potential of leaves with a Scholander pressure chamber. These pressure bombs are field portable and easy to operate. However, while the measurements yielded are quite reliable, the actual operation is time-consuming and physically demanding. It requires transporting a nitrogen tank along the field to reach the sampled points of measurement. Single leaves must be introduced into the chamber, and for every test, an equilibrium must be reached: as the pressure increases, at some point the liquid contents of the sample will be forced out of the xylem and will be visible at the cut end of the stem or petiole. In order to see the little drop coming out of the stem, a magnifying lens is typically used. In addition to the time needed for a single sample, measurements must be carried out either at midday or before dawn, which are not the best time for being in the field. All these inconveniencies have resulted in very few growers, even in large wineries, assessing water stress with the proper instrumentation, and therefore, there is an urgent claim to develop a new methodology using non-invasive technologies that allow getting more data, more often and less cumbersomely. Figure 5.17 shows the use of the Scholander pressure chamber in a commercial vineyard both at midday (center, right) and before dawn (left).

Fig. 5.17

Sampling and measurement of plant water stress in a vineyard with the Scholander pressure chamber (Courtesy of Symington Family Estates)

The effects of closing stomata result in an increase of leaf temperature; therefore, by tracking the thermal dynamics of vine canopies, we can define objective indicators of how stressed plants are. Unfortunately, plant biology involves complex, not fully understood physiological phenomena, and additional parameters must be taken into account to enrich plant monitoring. For example, the sun-exposed side of the rows will evolve differently from the shady side, and the vegetative vigor of each single plant will result in different resistance to stress. The EU-funded research project VineScout (2017-2020) aimed at understanding and estimating water status in vineyards by the massive measurement of canopy temperature and vine vigor. To do so, the robot of Fig. 5.18 was equipped with an infrared radiometer to measure temperature and a multispectral camera (Fig. 5.11a) to track vegetation indices. The onboard GPS receiver assigned a submeter precise location for each point sensed in the field.

Fig. 5.18

Robot VineScout to assess vine water stress: (a) day mapping; (b) night mapping

The output of a field monitoring session comes in the format of a map, where a text file stores the parameters measured in the field and their corresponding GPS location. One of the maps generated by the robot of Fig. 5.18 to assess water stress is the canopy temperature, which was measured with an infrared radiometer (Apogee Instruments, Logan, UT, USA) placed in the right side of the robot (Fig. 5.11b) facing the canopy central section at an approximate separation of 50 cm. This map was displayed in real time in the robot’s console (Fig. 5.8a) and also saved in a text file for its further processing. Figure 5.19 shows various canopy temperature maps recorded in the Portuguese vineyard of Fig. 5.16 in the summer, where the winery Symington Family Estates established an experimental irrigation scheduling for research. Figure 5.19a is an aerial image of the studied field that highlights differences in vine vigor caused by uneven soil and terrain. Figure 5.19b plots the trajectory of the robot during one of the mapping missions conducted in the vineyard in 2018, when some of the highest temperatures of the summer were registered in the afternoon. The robot only mapped one side of the canopy and scanned every two rows. The temperature map in grid format recorded over the afternoon is shown in Fig. 5.19c. In contrast, Fig. 5.19d represents the temperature map for the same field the day before at dawn. The map of Fig. 5.19c covers 12 rows and sums up a total of 16,529 points (including the headland turns where no valid measurements were made) acquired in a time frame of 67 min. After removing the headland section where the temperature was not registered, the total number of points was 15,469. The map of Fig. 5.19d comprises 15 rows and 23,012 data points, which were reduced to 18,945 after removing the headlands.

Fig. 5.19

Crop monitoring maps: (a) aerial NDVI map of monitored field (Courtesy of Symington F. E.); (b) robot trajectory for building the grid map of afternoon temperature; (c) grid map of afternoon temperature; (d) grid map of dawn temperature

According to Fuchs (1990), radiation, air temperature, humidity, and wind speed modify leaf temperature and may mask indications of water stress. Furthermore, the position, inclination, and orientation of leaves within the canopy also produce considerable variation of leaf temperature. The intricacies behind the knowledge of how much stress a vine is actually supporting in a given period of time lead to complex models and a multiplicity of influencing parameters. However, the majority of field data used so far has been taken remotely and occasionally. The opportunities brought by the automatic monitoring of crops at submeter distances, producing massive amounts of data at the grower request, open a new paradigm for understanding crop growth and fruit production. The temperature maps of Fig. 5.19 were taken in August and involve 34,414 measurements grabbed from dawn to evening. When comparing these figures with manual sampling, the resolution of robot-generated maps reaches significantly higher orders of magnitude. Maps 5.19c and d involve an area of 4800 m² (≈ 0.5 ha), considering an approximate row length of 100 m, a row spacing of 2 m, and a mapping frequency of one pass every two rows. The resolution achieved is 3.2 measurements per m². The same rationale can be applied to the map of Fig. 5.19d, yielding a similar sampling resolution. The data represented in the maps are the straight measurements carried out by the infrared radiometer, and no filtering has been introduced yet. However, some patterns and preliminary conclusions may be extracted from these results. The distribution of temperatures graphed in Fig. 5.19c reveals that many locations in the 15 rows sampled reached temperatures above 40 °C, with an uneven distribution of temperatures that resembles the S shape of Fig. 5.19a (dark blue in Fig. 5.19a). The map recorded at dawn, from 5:27 am to 6:55 am, produced a more homogeneous result, and more data is required for a deeper understanding of thermal dynamics along the day. Fortunately, the specific measurements recorded by the robot at every location are available for the user, and can be easily plotted for the entire battery of tests. Figure 5.20 shows the precise temperatures acquired for the maps of Fig. 5.19c and d.

Fig. 5.20

Specific temperatures acquired by the robot in the maps of Fig. 5.19c and d

Every row in the field of Fig. 5.19 has an approximate length of 100 m and presents a basin-like lower section toward the center of the row, where plants are more vigorous. The extremes of the rows (headlands) are higher and plant canopies significantly thinner. The robot velocity oscillated between 1.2 km·h⁻¹ and 1.7 km·h⁻¹ according to the motion being upslope or downslope. Overall, each run took between 4 and 5 min. Figure 5.20 confirms that there is much higher dispersion for the temperatures measured during the afternoon, but also reveals a pattern in the chilly temperatures at dawn; at the central part of each row, where vines have more foliage and the terrain profile forms a mild basin, registered temperatures were typically several degrees lower. As time progressed over the morning test, the average temperature slowly increased, and after 1 h, the differences between headlands and row centers practically disappeared. The thermal behavior of the vineyard over time and location was transferred to the vineyard manager, who will add this information to other field data for a better decision-making in the management of this plot.

6 Summary and Concluding Thoughts

Feeding an ever-growing population is becoming a big challenge for the agricultural engineers of the future. Fortunately, disruptive technologies tend to appear to face up such challenging scenarios, and digital farming has brought powerful tools to do so, such as precision agriculture and robotics. Making the right managerial decisions is key for the sustainable production of specialty crops, and to make good decisions, reliable data is essential. This chapter has explained the basic conditions for crop scouting and the principles for designing monitoring vehicles, paying especial attention to the workings of fundamental sensors, and how to display their information in legible maps. However, although results are positive and technology is more accessible every day, systematic field monitoring is still scarce. Many high-interest crop properties such as maturity and yield have no commercial device to monitor noninvasively from moving vehicles. In some cases, measuring sensors are available, but their practical implementation or the data that they generate are too cumbersome for regular applications at user level. Nevertheless, digital farming has come to stay and these difficulties will eventually be overcome with time. Agriculture is not a source of big data yet, but it is a question of time.