Emerging Directions of Precision Agriculture and Agricultural Robotics

Abstract

In this chapter, we aim to shed some light on the next steps in the evolution of Precision Agriculture (PA) and Agricultural Robotics Systems (ARS), and the technological factors that will drive this evolution.

8.1 Introduction and Definitions

In this chapter, we aim to shed some light on the next steps in the evolution of Precision Agriculture (PA) and Agricultural Robotics Systems (ARS), and the technological factors that will drive this evolution. To that end, we summarize a variety of research projects that are at the frontiers of Precision Agriculture and Agricultural Robotics Systems that integrate these two areas.

Precision Agriculture, as stated and discussed earlier in this book, is a field in agriculture concentrating on selective decision making and planning based on the processing of detailed farm-timely information, knowledge and thoughtful expertise. Underpinning Precision Agriculture is the need to improve aspects of the future farm, such as crop profitability and affordability, farm productivity and long-term sustainability, and environmental benefit. Precision agriculture is designed to follow these aims by reducing, through technological means, the required amount of fertilizers and other chemicals, irrigation, fuel, manual work, and lease and crop insurance payments (e.g. Mulla 2013).

In complex systems and systems-of-systems, intelligent control techniques and systems are necessary for dynamic, real-time interpretation and guidance of the environment and the objects operating in it (Nof 2009). Many PA related projects have been undertaken that use the potential of technologies and concepts, such as Cloud computing, Internet of Things (IoT), Internet of Services (IoS), Cyber Physical System (CPS), robotic simulators with realistic motion simulations, cyber augmented collaborative control and Human–Robot Collaboration. Some of these emerging technologies are described in this chapter.

8.1.1 Why Is Precision Collaboration Essential in Precision Agriculture?

The concept of Precision Collaboration (Bechar et al. 2015) is the underlying aspect in all emerging trends in Precision Agriculture. Why? Because many, often highly dispersed and distributed agents and resources are integrated to enable and accomplish the goals of PA. The details of Collaborative Control Theory and Precision Collaboration will be expounded in Sects. 8.4 and 8.5. Two key aspects of Precision Collaboration are:

• 1. When networks and systems of systems scale up, and the probability of inefficiencies, gaps of responsibility, errors and conflicts increase, precise interaction becomes crucial. Therefore, it is worth implementing Precision Collaboration methods and tools.

• 2. Augmentation by sensors and collaborative control theory (CCT) enable and enhance smart and precise coordination and collaboration beyond communication and processing, and as contributors to collaboration support systems, has been found in recent research and surveys to be an important and valuable emerging area.

A few definitions are included below because they are used often in this chapter:

• 1. Cloud computing: An information technology paradigm that enables ubiquitous access to shared pools of configurable system resources and higher-level services that can be provisioned with minimal management effort, usually over the Internet.

• 2. Internet of things (IoT): A system of interrelated computing devices, mechanical or digital machines, objects and people that are provided with unique identifiers and the ability to interact and transfer data over a network without requiring human-to-human or human-to-computer interaction.

• 3. Internet of services (IoS): A technology that provides the network infrastructure to support a service-oriented ecosystem. A fundamental characteristic of the IoS is that services combine and integrate collaboratively the functionalities of other services. (Van der Mei et al. 2018).

• 4. Cyber physical systems (CPS): CPSs are commonly defined as the systems that offer collaborative integration of computation, networking and physical processes (Khaitan and McCalley 2015). The US National Science Foundation states “In cyber-physical systems, physical and software components are deeply intertwined, each operating on different spatial and temporal scales, exhibiting multiple and distinct behavioral modalities, and interacting with each other in a myriad of ways that change with context.”

• 5. Service oriented architecture (SOA): A service-oriented architecture is a collection of services that communicate with each other. The communication can involve either simple data transfers or could involve dynamic coordination and collaboration among two or more services that combine temporarily for required purposes and timely execution.

• 6. e-Work: e-Work is a collection of collaborative, computer-supported and communication-enabled e-Activities, e-Operations, e-Functions and e-Support systems that enables other e-Systems and e-Activities (Nof 2003). The c-Work is a more advanced e-Work, augmented for smart collaboration by cyber-physical models and techniques. The Cc-Work is the currently emerging Cyber-Collaborative Work, enabled by cyber-augmented (e.g. wearables, augmented and virtual reality) human–robot–machine work processes and systems (Nof 2019).

• 7. e-Service: e-Service is the provision of services over electronic networks such as Internet, intranets or extranets without its scope being limited to service organizations, but rather encompassing all enterprises, even those that manufacture goods and which require the development and implementation of sound service practices over electronic networks (Nof et al. 2015). The c-Service is a more advanced e-Service, where cyber-augmented collaboration is enabled.

8.2 Cloud Computing and Physical Internet/IOT, IOS and CPS for ARS c-Work and c-Service for Precision Agriculture

Precision Agriculture is an innovative effort that combines agricultural with digital and data science technologies that increasingly include cyber technologies, in the context of what is defined as ARS: Agricultural Robotic Systems. Innovations that involve various implementations based on cloud computing and Internet of Things/Services (IoT/S) into Precision Agriculture are expected to emerge in the future, given the rapid advancement and benefits of these technologies.

Cloud computing based applications of Agriculture IoT Sensor Monitoring Network were reviewed by Mekala and Viswanathan (2017a, b). A simple IoT model for an agricultural problem is presented in this chapter. The problem addressed was that farmers in India lack sufficient knowledge of soil characteristics and environmental information because the number of testing laboratories available in the country is limited. Internet of things based agriculture was proposed as a solution to this problem. The four layered IoT architecture can be applied to Precision Agriculture.

Layered architectures are commonly applied for the design and standardization of complex systems. Similar to layered architectures in industry and services, the first (top) layer in agriculture-related applications serves as a user interface layer. With this layer farmers can make decisions regarding crop protection and optimizing food production outputs and food security. The second layer involves data compilation, classification, processing, monitoring, and decision analysis. The third layer involves network management which would include communication technologies, such as Gateway, RFID, GSM, Wifi, 3G, UMTS, Bluetooth Low Energy, Zigbee, and so on. The fourth layer is the information collection layer that contains all physical instruments, sensors, cameras, and so on. This study also compares and contrasts various available hardware technologies and their use in an agricultural IoT setup. According to this survey, challenges for implementing IoT in agriculture include design of Service-oriented Architecture (SOA), Decision Support Systems (DSS) capabilities, efficient data mining and analytics, and IoT maintenance costs. The study addresses challenges and provides an IoT agricultural framework. Light Fidelity (Li-Fi) technology was introduced and evaluated for fixed area structure topology. The cloud computing framework was used to facilitate remotely controlled processes to perform spraying, weeding, bird and animal scaring, vigilance, moisture sensing, and so on. The methodology included smart warehouse management, which includes temperature and humidity maintenance, and theft detection. It also included intelligent decision making based on accurate real-time field data for smart irrigation with smart control.

Wang et al. (2014) also explored the architecture of the Internet of Things in agriculture with heterogeneous sensor data and proposed a data management system involving cloud computing to enable an IoT in agriculture (Fig. 8.1). Their design is based on a two-tier storage structure of a distributed database with large scalability, named HBase. Their work also proposes a management mechanism for heterogeneous sensor data for IoT in agriculture based on cloud computing. It consists of a data unification module, abnormal data processing module and a two-layer architecture to store data and access data. A cloud computing-based framework for agriculture information integration was also created by Duan (2012). In this research, a methodology and system for the integration of agricultural information and sharing a platform based on cloud computing were developed.

Fig. 8.1

Topological structure of IoT in agriculture

The data management problem of large size remote sensing images in soil moisture mapping for Precision Agriculture was addressed by Zhou et al. (2016). This methodology implements a Remote Sensing Observation Sharing method based on cloud computing (ROSCC) to enhance storage of remote sensing images and to achieve large-scale soil moisture mapping in Precision Agriculture (Fig. 8.2).

Fig. 8.2

Architecture of the ROSCC methodology

A system that combines wireless sensor networks (WSNs) and cloud computing into an integrated architecture for agricultural environmental applications was designed by Kassim and Harun (2017).

Cyber-Physical Systems (CPS) need to adapt to the changing physical world and expand their capabilities dynamically (Pradilla and Palau 2016). They designed a three-tier architecture that integrates: cloud computing, fog computing, and networks of sensors and actuators. The implementation involves the use of micro virtual machines and sensor observation, combining the isolation of virtual machines with standardized storage and information-exchange under a Sensor Web Enablement framework. The proposed architecture is coupled with the Internet of things (IoT) and applicable to Precision Agriculture.

The issue of information security and privacy in agriculture cloud information systems was addressed by Tan et al. (2014). Most encryption schemes cannot support encryption based on ciphertext. Therefore, it is difficult to build up the corporate and individual information security and privacy-securing in the information system based on a cloud computing platform. To enable information security and privacy of the cloud computing infrastructure that would be practical for an Agriculture Information System (AIS), the researchers have created an innovative encryption method for an agriculture intelligent information system (AIIS) based on a cloud computing platform. It is based on matrix theory and supports a series of cipher–text–operations? essential to create a secure communication protocol between users, owner and cloud server. This methodology can perform crypto-function at a moderate speed and can be used for securing corporate–individual privacy with regard to AIISs.

Further research will be needed to assure security of a cyber-augmented precision agricultural system and to prevent malicious disruptions and remote intervention in their safe and smooth operations.

8.3 Simulating an Agricultural Robotic System for Precision Agriculture Tasks

As more robotic systems are being developed and implemented in the agricultural domain, it would be cost effective to simulate such systems in the development phase. Recently there have been a few research projects on simulating a robotic system for human–robot collaboration. A computational simulation environment named ‘Simulation Environment for Precision Agriculture Tasks using Robot Fleets’ (SEARFS) was developed (Emmi et al. 2013) to study and evaluate the execution of agricultural tasks that can be performed by an autonomous fleet of robots. The environment is based on a mobile robot simulation tool that enables the performance, cooperation and interaction of a set of autonomous robots to be analysed while simulating the execution of specific actions on a three-dimensional (3-D) crop field. The SEARFS computational simulation environment is capable of simulating new technological advances such as GPS, GIS, automatic control, in-field and remote sensing, and mobile computing, which will enable the evaluation of new algorithms derived from PA techniques. This environment was designed as an open source computer application. The SEARFS environment consists of four levels of configurations, where the lower levels depend on the configuration of the higher levels (Fig. 8.3).

Fig. 8.3

SEARFS environment configuration levels

A general method for the development of customized robot simulation and control system software with a robot operating system (ROS) was also developed by Wang et al. (2016). The simulation designed in this research involves: a) a 3-D visualization model, created in URDF (unified robot description format) and viewed in Rviz to achieve motion planning with the MoveIt! software package, b) machine vision provided by a camera driver package in ROS to enable the use of tools for image processing, and 3-D point cloud analysis to reconstruct the environment to achieve accurate target locations and c) communication protocols provided by ROS for serial, Modbus support of the communication system development. A tomato harvesting scenario was simulated using this methodology to demonstrate its features and effectiveness.

8.4 Cyber Physical Systems and ARS

To overcome difficult problems such as the variability in agricultural produce and continuously changing conditions, development of intelligent systems is necessary to perform tasks successfully in such environments. Information acquisition systems, including sensors, fusion algorithms and data analysis need to be improved and adjusted to the dynamic and uncertain conditions of unstructured agricultural environments (Bechar 2010).

The trend in digital transformation has offered considerable opportunity for more efficient production using Cyber Physical Systems (CPS), which will enable new concepts for future farming systems (Herlitzius 2017). The rapid development of information and communication technologies is driving the evolution of mobile machines and devices into cyber-physical systems with few limitations with regard to communication.

A Precision Agriculture architecture (Fig. 8.4) was developed by Nie et al. (2014) based on CPS technology that comprises three control layers, i.e. the physical, network and decision layers.

Fig. 8.4

Architecture of Precision Agriculture CPS nodes

A CPS oriented framework and workflow for agricultural greenhouse stress management, called MDR–CPS, was designed by Guo et al. (2018). It has been designed to focus on monitoring, detecting and responding to various types of stress. The system combines sensors, robots, humans and agricultural greenhouses as an integrated CPS, aimed at monitoring, detecting and responding to abnormal situations and conditions. The purpose is to provide an innovative solution that combines wireless sensor networks, agricultural robots and humans applying collaborative control theory (CCT) to detect and respond selectively to stresses as early as possible. The agricultural MDR–CPS framework is depicted in Fig. 8.5.

Fig. 8.5

(Courtesy of Guo et al. 2018)

Agricultural MDR–CPS framework

Sensor nodes are used in greenhouses to provide information on environmental properties that influence the healthy development of the agricultural crops. An agricultural cloud model platform is used in the field based on several server clusters (Guo et al. 2018; Zamora-Izquierdo et al. 2019).

8.5 Cyber-Augmented Collaborative Control of ARS

8.5.1 Collaborative Control Theory (CCT)

Collaborative control theory has been developed by researchers at the PRISM center at Purdue University and elsewhere (Nof 2007; Seok et al. 2012; Barbosa et al. 2014; Hernandez 2014; Nof et al. 2015; Yilmaz et al. 2017; Moghaddam and Nof 2017; Reyes Levalle 2018; Zhong and Nof 2020) to optimize distributed, decentralized and multi-agent based e-Work and s-Service. Collaboration is known to be essential for effective design and control of e-Work and e-Service. It enables all involved entities, human and artificial, in decentralized e-Systems to share their resources, information and responsibilities, such that mutual benefits are obtained (Figs. 8.6, 8.7 and 8.8).

Fig. 8.6

(Source Nof et al. 2015)

Coordination vs Cooperation vs Collaboration in terms of interaction level

Fig. 8.7

(Source Nof 2007)

The different components of cyber enhanced processes

Fig. 8.8

(Source Seok et al. 2012)

Collaborative mechanisms of CCT and DSS (decision support system) for sustainability planning and control

Future precision agricultural systems will comprise multiple distributed and autonomous agents, therefore, the efficiency and effectiveness of the CPS would depend upon how well its constituent agents can collaborate.

Figure 8.9 illustrates the precision requirements for collaborative support features as evaluated by Bechar et al. (2015).

Fig. 8.9

(Source Bechar et al. 2015)

Collaboration support augmented by laser: Features and their precision requirements

Automated processes in an uncertain and unstructured environment (such as agriculture) are challenged by changing peripheral requirements (Zhong et al. 2015). Addition of extra flexibility to the existing equipment to handle a larger range of tasks is a desirable solution, which can be offered, for example by Reconfigurable End-Effectors (REEs). An REE system has an adjustable structure to facilitate the adaptation of the end-effectors to various objects, therefore it is an intermediate solution between flexible and dedicated end-effectors (Zhong et al. 2015). Use of multiple end effectors enables the robot to adapt directly to multiple agricultural functions as and when required. For effective REE operations, the asynchronous cooperation requirement planning (ACRP) framework was created to facilitate the design and control of REE. The ACRP provides a dynamic solution, extending from the planning facet of collaborative control theory (CCT) for designing (offline) and controlling (online) multi-agent collaborations. The ACRP methodology is illustrated in Fig. 8.10.

Fig. 8.10

(Courtesy of Zhong et al. 2015)

Framework of Asynchronous Cooperation Requirement Planning (ACRP)

The framework is illustrated with a case study of vegetable harvesting by multi-arm automated systems (Zhong et al. 2015). In harvesting processes, the grasp quality is one of the most important factors for production quality, therefore research on effective design and control of reconfigurable end effectors is highly relevant.

In emerging and future agricultural robotic systems, we can expect heterogeneity in multi-robot teams. To handle the varieties and variations of tasks observed in unstructured agricultural environments, multiple configurations of robots or heterogeneous robots, would need to be designed and included in the system. In such collaborative systems consisting of heterogeneous robots, ineffective task assignments can result in weak? collaboration and thus poor efficiency. Zhang et al. (2015) define the collaborative task assignment problem and develop a fuzzy collaborative intelligence-based algorithm to optimize the assignment plans as a solution to the challenging requirement of collaboration in heterogeneous multi-robot systems. This research introduces the concepts in collaboration type, the collaboration matrix and assignment matrix, and introduces an algorithm for adaptive fuzzy collaborative task assignment that is based on fuzzy set theory. Experimental results show and validate a shorter completion time, less energy consumption and a statistically significant larger loading accuracy. The methodology and algorithm were simulated in a general setting, and the methodology can be adapted directly to agricultural robotic systems. Below is a brief description of the types of collaboration in a multi-robot and a human–robot system (Zhang et al. 2015) (Table 8.1).

Table 8.1 Collaboration among robots working together (following Nof 1999)

 

Research by Zhang et al. (2015) solves a heterogeneous multi-robot collaboration problem where stochastic and consecutive tasks are assigned to single or multiple robots in a dynamic changing environment.

The remainder of this section describes several recent research projects involving cyber enhanced and/or cyber augmented collaboration.

A. Methods for simultaneous orchard and harvesting robot design

Robotic manipulators can perform a variety of agricultural tasks, many of them with precision. However, despite decades of research, few agricultural robots have been commercialized. One of the reasons for the lack of agricultural robots on the market today is their high cost and lack of precision enabling functions, which makes them unprofitable for farmers.

Bloch et al. (2015, 2017, 2018) from the Agricultural Research Organization, Rishon LeZion, Israel prepared robotic systems that are optimal for specific tasks. In the optimization process, the robot’s performance is maximized while allowing it to perform the task. To achieve a reliable result, the actual field task must be described and modelled with sufficient precision. However, the complex and unstructured environment of agricultural tasks complicates the task description as well as the robot-design process.

The main goal was to characterize and analyse the environment of a given orchard and the required agricultural tasks, to understand their combined influence and interaction with the optimal design of a task-based robot for that orchard. This analysis allows the task description to be simplified by characteristics of the environment during simultaneous design of the robot and its environment.

The main results of the research are as follows. For the task-based robot optimization, we created a library with approximately 20 plant models. Software for evaluating the robot’s performance effectiveness (optimization of cost function) was written and used for the optimal robot design. Based on the model library and software, robots were designed with optimal kinematics for a number of agricultural tasks and environments. During robot optimization, the level of complexity of the environment included yet did not enable the proposed software to solve the optimization problem in an acceptable time. In addition, a methodology for optimal robot location was developed.

To solve the robot-optimization problem for picking fruit in complex environments, a method was developed for characterizing the agricultural environment by fruit clustering and reaching cones. The method systematically reduces the complexity of the environments, thereby decreasing the number of calculations and providing a near-optimal solution. The method was approved and successfully applied to complex environments, solving the optimization problem in hours, rather than after weeks of calculations. The expected precision of the achieved solutions was 10% in the case examined.

A preliminary design for the robot working environment was prepared. Research findings include an environment that was fitted maximally to the robotic operation and that optimized one of the variables defining the structure of the environment.

In general, a set of tools and methodology was developed for analysis and design of the agricultural environment, together with optimal robot design. This methodology is novel in robot design, in particular in agriculture. It helps to improve the robot performance while designing low-cost robots affordable for farmers. The methods developed in this research are applied to apple and nectarine harvesting, although they can be used for robotic harvesting of any type of fruits, for other agricultural tasks, or in any area where the robot-environment design is used or is applicable.

B. Development of a selective autonomous sprayer for greenhouses

The essential process of pest control and chemical application of nutrients is one of the most important processes in any agricultural production. Nevertheless, the application requires human resources; it is a time-consuming task and exposes the operators to the danger of contamination with hazardous chemicals. Integrating autonomous robots and machinery for agricultural tasks involving expensive labour, and that are monotonous and hazardous has accelerated recently. An autonomous robot is an alternative in many cases. This research focuses on the development of a navigation procedure for an autonomous sprayer in a greenhouse growing sweet peppers.

C. A robotic sonar system for specific yield assessment and plant status evaluation

Specific yield assessment is essential for precision farming and agriculture in general. It is an important tool in agriculture for forecasting crop revenues, planning the budget and store capacity, labour management and compensation calculations (Fermont and Benson 2011). In several crops, such as fruit trees, fruit thinning is done based on yield estimation.

Subsidized crop insurance has become the most important single support policy in agriculture in both the USA and Israel. The program is immense in the USA, currently insuring over $120 billion in agricultural values and costing its taxpayers approximately $10 billion each year (Glauber 2013; Goodwin and Smith 2013). In Israel, for example, the aggregate premium payments from government subsidies for crop and disaster insurance programmes amount to over $25 million annually (source: Israeli Agriculture ministry budget).

An accurate and site-specific yield assessment technique that will decrease the assessment cost and increase its accuracy has the potential to reduce production costs, increase yield and profitability and save billions of dollars in tax subsidies:

The present techniques, however, for yield assessment are labour intensive and tend to be expensive. Moreover, the process is inaccurate because it is carried out manually by workers in the field and is based on crop sampling in small quantities, which in addition loses information on the variation. There is a tradeoff between the amount of time invested in sampling the crop and the accuracy given the inhomogeneous nature of crop distribution. To meet this challenge, various modern sensing technologies, such as thermal imaging (Stajnko et al. 2004), depth cameras (Andújar et al. 2016) and optical methods (Wachs et al. 2010) have been suggested for developing an automated system to detect crop biomass and for yield estimation (Lee et al. 2010). Recent vision-based studies in the context of specialty crops include one by Moonrinta et al. (2010) who developed a vision-based pineapple mapping algorithm with a detection success rate of 80%. Another vision-based yield estimation study by Nuske et al. (2011) detected 50–70% of the visible grapefruit and predicted the amount of crop mass with an error of 9%.

An ultrasonic sensing system was developed and the resulting classification features that would ultimately be used for a yield estimation robotic system were analysed (Mizrach et al. 2003; Mizrach 2008; Finkelstein et al. 2017). An algorithm was also developed to predict fruit mass per plant based on the ultrasonic echo return from a plant. The ultrasonic sensor system was tested in laboratory and greenhouse (with peppers) environments and on single pepper plants, single leaves and fruit. The results showed the potential of ultrasonic sensors for such a robot in classifying plants and greenhouse infrastructures such as walls. It showed the robot’s ability to detect hidden plant rows and fruits as well as estimating the fruit mass in single plants. The system developed can detect and map crop rows without a direct line of sight using a matched filter and normalizing the acoustic energy by distance.

8.6 Human-Robot Collaborative System for ARS in PA Tasks

An overview and a framework for Precision Collaboration are shown below (Fig. 8.11). As mentioned in earlier sections, when networks and systems of systems scale up, and the probability of inefficiencies, errors and conflicts increases, the precision of interactions becomes crucial.

Fig. 8.11

Precision Collaboration support framework (H- Human; C- Computer; R- Robot)

Unstructured environments such as agriculture are characterized by rapid changes in time and space (Bechar and Edan 2003). Fully automated systems do not perform well in such environments where they become cumbersome, complicated and expensive to develop and operate. Therefore, optimal output of a Precision Agriculture robotic system would depend on the effectiveness of collaboration between human agents and cyber controlled agents.

Cheein et al. (2015) reported a study that included guidelines for designing a human–robot interaction strategy for harvesting tasks that could be used for other agricultural tasks. The four design cores of a service unit are: mapping, navigation, sensing and action. This research addressed the problem of a decline in availability of human labour in agriculture in Chile and Argentina.

The research also discusses the current constrains related to precision farming and associated with flexible automation of farms in Argentina and Chile. The constraints include environmental constraints such as the variation in yield, the field, soil, crop, anomalous factors and management. For the latter this includes tillage practice and seeding rate, crop rotation, fertilizer and pesticide application and irrigation pattern; these are facts that the service unit must know during its incursion into the workplace to avoid interference with the manual labour.

With regard to collaboration in a networked telerobotic environment, Nof and the PRISM center at Purdue University have developed a mechanism and tool  called HUB-CI, a hub for Collaborative Intelligence (Fig. 8.12). Collaborative intelligence is a concept and a potential measure of performance of an e-System. It is a combination of communication intelligence, cumulative intelligence, cooperative intelligence and collective intelligence (Zhong et al. 2013). The HUB is an online portal that enables users to create and share research materials and computational tools. It can deliver all resources and simulations by a standard web browser and use high performance Grid computing resources. The majority of HUBs allow collaboration on virtual materials and simulations, but there has been no tool for users to perform physical collaboration (Zhong 2012). The HUB together with cloud computing allows software and data to be shared directly by groups of users, and provides knowledge and analytical tools that can be applied in Precision Agriculture systems (Nair et al. 2019; Sreeram and Nof 2021). The HUBs enable better, faster, smarter collaboration among decentralized, asynchronous decision-makers. Furthermore, it is considered a major enabler of precision in manufacturing, logistics and agriculture (Fig. 8.12).

Fig. 8.12

(Source Zhong 2012; Devadasan et al. 2013)

The HUB-CI concept and architecture, enabling precision in operations planning and control

The HUB-CI has been applied and tested in knowledge-based service planning (Zhong et al. 2013) and also to collaboration between telerobots and human agents in manufacturing (Zhong et al. 2013). Current research is being undertaken to apply HUB-CI in a telerobotic agricultural cyber physical system. Below are some projects in which Human–robot collaboration was applied to enhance output and productivity of the agricultural robot system:

A. Human–Robot collaborative system for selective tree pruning

Orchard pruning is a labour-intensive task that involves more than 25% of the labour costs. The main objectives of this task are to increase exposure to sunlight, control the tree shape and remove unwanted branches. In most orchards this task is done once a year and up to 20% of the branches are removed selectively.

A human–robot collaborative system for selective tree pruning has been developed (Bechar et al. 2014). The system consists of a Motoman manipulator, a colour camera, a single beam laser distance sensor, a human machine interface (HMI) and a cutting tool based on a circular saw developed for this task. The cutting tool, camera and laser sensor are mounted on the manipulator’s end-effector, and aligned parallel to one another (Fig. 8.13).

Fig. 8.13

(Source Agricultural Research Organization Israel)

Cutting tool design for tree pruning

Experiments were established to examine the performance of the system under different conditions, human–robot collaboration methods and different trajectory types (Bechar et al. 2014). A cutting tool was designed for pruning branches with a diameter of up to 26 mm at a 45° cutting angle. The saw diameter was determined to be 115 mm with a standard shaft diameter of 41 mm. An interface to connect the cutting tool to the robot’s end effector was designed to minimize the total dimensions of the tool and to increase e robot dexterity. An average cycle time of 9.2 s was achieved when the human operator and robot perform simultaneously. The results also revealed that the average time required to determine the location and orientation of the cut was 2.51 s.

B. Robot for automatic melon collection

Melon and watermelon harvesting require intensive manual labour. Machines with automatic robotic arms may replace personnel, especially in a simple routine that requires considerable physical effort. In this project a human is involved but in a different way. Based on preliminary tests it was found that about 80% of the workers’s time is invested in transferring the picked melons from the bed and only 20% in locating and disconnecting the ripe melons from the plant. Therefore, the task is conducted in two steps. In the first, the human passes in the field, detects the ripe melons, marks their locations and disconnects them from the plants with pruning shears. In the second steps, the robotic system passes and collects only the melons that were marked and harvested. A robotic arm system has been developed (Fig. 8.14) that can collect the melons automatically knowing their coordinates, while moving through the collection area. An electro-mechanical robotic arm system has been assembled that consists of a wheeled frame, cylindrical rails with end limit switches, stepper motors with encoder for X- and Y-axis arm movement, a pneumatically operated robotic arm system for additional Y- and Z-axis movements, vacuum operated gripper, motor controllers and a PLC.

Fig. 8.14

(Source Agricultural Research Organization, Israel)

A close up of the melon picking robot and the robotic arm for melon picking (circled)

A human machine interface has been developed to enable operator intervention. A melon ‘picking-up’ simulator program has been created, capable of demonstrating the process of collecting melons by the robotic arm. For experimental applications, the melon collecting path optimization algorithm was used. The system was tested and succeeded in reaching up to seven target points in sequence with an accuracy of 84% (with a target reaching error of 7–10 mm, collection time 7–8 melons min⁻¹, at a distance of up to 4000 mm, with arm velocity of up to 800 mm s⁻¹ and acceleration of up to 50 m per s²).

C. Multi-sensor fault tolerant learning algorithm in an agricultural robotic system

Ajidarma 2017; Ajidarma and Nof 2021 aimed to develop a new fault tolerant interface design based on the collaborative control theory (CCT) principles of best matching (BM), error prevention and conflict resolution (EPCR) for an agricultural robotic system. They developed a fault tolerant learning algorithm to process the data of moving sensors in an agricultural robotic system. The sensor data and actual state of the object were modelled as a function of error and rate of conflict. Two learning algorithms, adaptive learning algorithm (ALA) and cumulative learning algorithm (CLA) were developed and tested. This method involves collaboration with a human operator and an adaptive learning mechanism to minimize measurement and detection errors. It is an excellent example of the concept of Precision Collaboration. This research addressed the problem of having an interface with fault tolerant sensor data processing in a collaborative agricultural robotic system where multiple sensors are mounted on a mobile robot, and a human operator performs supervisory functions.

D. Human–robot collaborative system for early detection of crop diseases

Traditional agricultural management practices assume that fields growing crops have homogeneous properties (Oerke et al. 2011). In contrast, modern, Precision Agriculture integrates different technologies, such as: sensors, information and management systems for adapting agricultural practices to variation within the field (McBratney et al. 2005; Dong et al. 2013). Monitoring is a major component in Precision Agriculture and of precision crop protection (Gebbers and Adamchuk 2010; Schellberg et al. 2008).

Crop yield is affected by different stresses, e.g. pests, diseases, weeds, environmental conditions, nutrition or water deficiencies, which can impair production. Oerke and Dehne (2004) indicated that the impact of diseases, insects and weeds represents a potential annual loss of 40% of world food production. The occurrence of diseases depends on environmental factors and they often have a sporadic spatial distribution, therefore sensing techniques can be useful in identifying primary disease foci and distribution (Franke and Menz 2007; Franke et al. 2009). Sankaran et al. (2010) and Lee et al. (2010) suggested that detection and quantification of diseases with visible and infrared spectroscopy would be feasible. If a symptom or a disease can be detected by the naked eye, a sensor should be able to record the stress symptoms (Nutter et al. 1990; Stafford 2000).

Currently, disease detection and monitoring in greenhouses are conducted manually by an expert inspector and are limited because of the availability of human resources, sparse sampling and large monitoring costs. Sampling intensity and resolution are low with about 20 arbitrarily locations sampled per hectare in a fixed pattern (the same locations are revisited) and each plot is monitored every 7–10 days. The plants are inspected for symptoms by an inspector crossing the greenhouse rows on foot. Thus, the inspector walks about 20 km per day covering about 8 hectares, and a designated inspector is required for every 80 hectares. The limitations of the current inspection methods can lead to late detection and inability to contain a disease. As a precaution, repeated, large doses of pesticide are often applied even when symptoms are far below thresholds that require pesticide application. Moreover, pesticides are typically applied uniformly throughout the greenhouse while disease distribution is typically centred in distinct locations, resulting in additional pesticides use, increased material cost and adverse environmental effects.

In greenhouses, a current challenge is the early detection of stresses (potentially leading to diseases) and other crop risks to prevent the spread of uncontrolled disease and hence improve productivity. Often detection is too late even though there is enough knowledge on how to address the specific stress in plants. Different biotic and abiotic stresses affect the expected potential crop yield. These stresses and other factors that limit yields must be detected as early as possible such that appropriate and precise counter measures may be applied. In the absence of an affordable and effective monitoring mechanism or system, the decisions taken by farmers could be wrong and might result in over- or under-application of pesticides, nutrients and water.

Robotic systems in greenhouses enable early detection and improved control of plant diseases. They are expected to increase yield, improve quality, reduce pesticide application, increase sustainability and reduce costs. Symptoms vary for each disease and crop, and each plant might suffer from multiple threats, thus, dedicated integrated disease detection systems and algorithms are required.

Automation of disease detection can alleviate current difficulties and lead to improvement in yield together with considerable reduction in pesticide use (Franke and Menz 2007; Franke et al. 2009; Bock et al. 2010). In addition to reduced production costs, this will also lead to reduced exposure to pesticides of farm workers and inspectors, and increased sustainability (Hillnhuetter and Mahlein 2008). Plant diseases can affect various optical foliage characteristics, therefore disease detection can be based on different spectral ranges (Lee et al. 2010). Image processing of foliage light reflection has been applied to many different diseases and cultivars (for reviews see: Barbedo and Garcia 2013; Pujari et al. 2015; Patil and Kumar 2011; Lee et al. 2010). Methods based on fluorescence (Wetterich et al. 2016) or thermography (Oerke et al. 2011) can also be used for disease detection and have been extensively studied, but they are less relevant for a robotic detection system operating in the field because of cost, payload weight or required setup. Mobile robotic manipulators with various sensing capabilities offer an automated solution suitable for disease detection in greenhouses. There has been, however, little comprehensive research on the development of such integrated robotic disease detection systems for greenhouses, probably because the primary challenge of developing robust disease detection algorithms is still an open research question. Aerial platforms (Gennaro et al. 2012) and ground mobile robotic platforms with fixed sensor configurations (Harper and McKerrow 2001; Moshou et al. 2011; Pilli et al. 2014) for disease detection have been tested for open field crops. Yet, in greenhouses both solutions have inherent shortcomings. The maneuverability and flight duration of aerial systems within greenhouses is limited, and navigation and location cannot rely on GPS sensors because the structure can cause unpredictable errors, therefore they lose their main outdoor advantage. In greenhouses, sensory position and adaptation of orientation can greatly improve detection, especially early detection where symptoms are typically centered on distinct locations. For fixed sensor configuration, position and orientation adaptation are not possible. Moreover, in fixed configuration systems, the requirement for multiple disease detection can lead to a requirement for multiple detection positions and orientations, which tend to increase system complexity and cost and hinder maneuverability.

To address this issue, a robotic disease detection system for greenhouse pepper plants was developed based on the concept of a mobile robotic manipulator (Schor et al. 2015; Schor et al. 2016), which provides the required maneuverability and flexibility (Fig. 8.15). Prior to the above, no major system had been developed for disease detection for specialty crops in greenhouses that involved a mobile robotic manipulator.

Fig. 8.15

(Source Agricultural Research Organization, Israel)

The apparatus for disease detection for pepper plants

The robotic disease detection system was developed holistically, i.e. system architecture, operation cycle and detection algorithms for multiple threats to a pepper crop were developed in an integrated manner. Eizicovits et al. (2016) showed that early integration and testing of perceived requirements can lead to improved system design and operation in environments with taxing needs (e.g. the agricultural environment).

The detection system comprises a mechanical structure, sensor suite, motion planning (Fig. 8.16) and disease detection algorithms. Visual spectrum imagery is used for motion planning and disease detection for fast, non-destructive and cost-effective operation. An algorithm based on principal component analysis (PCA) was developed for powdery mildew, and three algorithms were developed for tomato spotted wilt virus)TSWV(disease detection, one based on PCA and two on the coefficient of variation (CV). Principal component analysis is a statistical tool used to reduce the dimensionality of data and demonstrate patterns in a dataset. The CV is a statistical measure of dispersion, calculated as the ratio of the standard deviation to the mean.

Fig. 8.16

(Source Agricultural Research Organization, Israel)

Example of motion planning for the robotic arm in disease detection in plants

The algorithms were tested using images of healthy and infected plants taken from a greenhouse. For RGB-based detection of TSWV, PCA-based classification with leaf veins removed achieved the greatest classification accuracy (90%), and the accuracy of CV methods was also high (85%, 87%). For powdery mildew (PM), the accuracy of pixel-level classification was high (95.2%) while that of leaf condition classification was low (64.3%) because leaf images came from the top of the leaf and disease symptoms start appearing on the bottom. The NIR-R-G-based detection produced inferior results for both diseases. The components of the system were integrated, and preliminary integration tests were done in a laboratory environment to verify that all system components would work together. The integrated system operated successfully for 110 consecutive minutes with an average cycle time of 26.7 s for end-effector velocity of 15 mm s⁻¹ and PCA-based detection algorithms. Future research will examine improvement of disease detection, aiming to achieve greater accuracy together with earlier detection, e.g. by facilitating PM examination on the bottom of the leaf or by integration of the two CV-based methods. For complete integration tests and field performance studies, a dynamic detection process (i.e. with a moving conveyor) will be implemented and tested.

Results are encouraging because the cycle time attained was slower than the calculated required baseline (Schor et al. 2017). However, the laboratory environment comprising a conveyor belt, stationary sensor system and black background for simplifying plant identification and background removal procedures makes the disease detection task easier and faster. Conducting a disease detection task in an unstructured environment such as a greenhouse will require more sophisticated algorithms for motion control, path planning and image processing because of a more complex environment that includes obstacles, background noises, illumination etc., thus cycle time may be extended.

A subsequent multidisciplinary project was undertaken by researchers at the Agricultural Research Organization (ARO) in Israel, PRSIM center – Purdue University, USA and the University of Maryland, USA. This research was funded by BARD,¹ the US–Israel binational agricultural research and development fund (Bechar et al. 2020). It combines the following three disciplines to solve the problem of consistent early detection:

• (1) Smart agricultural robots

• (2) Human–robot collaboration (based on the HUB-CI and CCT described in Sects. 8.4 and 8.5)

• (3) Early stress detection and classification using multispectral imaging and image classification and creation of a stress map

The robotic platform (cart) was modified at ARO to improve the control and autonomous navigation, and to suit the disease detection task better in a greenhouse. The platform is equipped with a UR5 manipulator, a sensory system comprising two depth cameras to create 3-D and 2-D maps of the greenhouse, the Kinect V2 and RealSense 435 and an RGB 1080p camera. A real-time environment mapping application was developed and modified with the robot sensors while it moves in the environment and generates a 3-D model of it. A 3-D mapping experiment was conducted in the laboratory and in a pepper greenhouse at ARO (Fig. 8.17).

Fig. 8.17

Three-D mapping of a pepper greenhouse (a) and the robotic platform (b)

For the ‘human-in-the-loop’ tasks of the agricultural robot system, a HUB-CI (hub for collaborative intelligence) system was developed by the PRISM team at Purdue and the ARO team. The objective: To enable effective and timely integration, and resulting collaboration tasks, by optimized exchange and leveraging of signals and information gathered in real-time from distributed components. The outcome of the HUB-CI is collaborative intelligence from the ARS networked components, thus enabling precision tasks (Nair et al. 2019). The following algorithms and protocols were developed by Dusadeerungsikul and Nof (2019): (a) algorithm to determine what image or case must be reviewed by remote human users, (b) adaptive search: use knowledge-based information, (c) routing algorithm: create a tour for a mobile robot, (d) detection-routing protocol: mechanism for remote disease detection algorithm to communicate with the routing algorithm, (e) manual control protocol: mechanism and constraints for manual control of the robot and (f) human-in-the-loop protocol: mechanism for human operator to communicate with the search and routing algorithm. The HUB-CI system has been designed as a virtual platform to integrate signals, data and control logic from several participating agents (cyber and human agents). It enables the cyber-collaborative protocols to make local control decisions based on global, real-time information. An initial prototype of HUB-CI was developed and tested in the experiments. Unique features designed with the HUB-CI system include (Nair et al. 2019): (i) planned collaboration between diverse users (farmer, engineer, pathology expert, etc.) of the agricultural robotic system in a HUB-CI environment, (ii) collaborative semi-automated and manual control (remote and local) of agricultural robot, (iii) learning-based filtering algorithm for spectral images taken off plants, (iv) collaborative decision making regarding the greenhouse system based on intelligent information sharing, (v) scheduling and task administration of all cyber and human agents in the agricultural robotic system (ARS) and (vi) adaptive search and routing algorithms: use resource (time) to perform monitoring and inspection tasks. Three experiments were conducted to examine the collaborative control of the system. In all experiments, the robot was controlled from Purdue University. Two-way collaboration frames were developed: (1) an ad-hoc connection using TeamViewer in which researchers at Purdue controlled the robot’s computer directly and (2) through a server using dropbox. In all experiments, collaboration with direct commands from Purdue to ARO was tested.

The hyperspectral imaging analysis can be divided into two research steps. First, a classification algorithm needs to be developed based on full spectral information of healthy and diseased spots. Second, some key hyperspectral bands need to be selected specifically for real-time in-field detection. The decrease in number of spectral bands should not affect classification accuracy. The University of Maryland research group developed a new method of hyperspectral analysis named ‘outlier removal auxiliary classifier generative adversarial nets (OR-AC-GAN)’ (Wang et al. 2019). The model uses full spectral information (395–1005 nm) to integrate the tasks of background removal, pixel-level spectral analysis and image-level plant classification. The model starts from generative adversarial nets (GAN) to learning the data distribution of different spectral classes. It can augment the training dataset online according to the data distribution and effectively remove the side effects of data outliers and imbalance on the dataset. This model can classify the one-dimensional spectral signal into different classes. Images were taken at ARO using a Specim hyperspectral camera with a high-resolution, high-speed image acquisition device (NI PCIe-1427) installed on an i7-4770 CPU PC. The computer was equipped with the Specim data recording application for hyperspectral images (HSI): Lumo Scanner. In the experiment for 54 independent test images of the TSWV disease database constructed by ARO, the model can reach 96.25% prediction accuracy (92.59% sensitivity, 100% specificity) before visible symptoms appear (as early as 5 days after disease inoculation) (Wang et al. 2019). In contrast, human experts can tell the difference of diseased and healthy plants 15 days after disease inoculation. For pixel-level classification accuracy, the prediction of false positives in healthy plants was as small as 1.47%. The OR-AC-GAN is an all-in-one model meeting the first requirement of hyperspectral data analysis. The experiment proved that the augmented data, a ‘by-product’ of OR-AC-GAN can markedly improve the performance of existing band selection algorithms (Wang et al. 2019).

8.7 Bio Inspired Robots for ARS in Precision Agriculture

Recent research has created bio-inspired robots for various agricultural applications. The fundamental motivation behind the development of bio-inspired multi-robot teams is that living organisms can successfully cope and provide good solutions to almost all robot-related problems (Tsourveloudis 2014). Navigation, material handling, sensors and machine learning are only some of the research areas that have benefited from examining and adopting methods, techniques or mimicking behaviours proved sustainable and successful for animals and humans (Tsourveloudis 2014). This section describes several bio-inspired robots that have been built for agriculturally related tasks.

Climbot (Guan et al. 2016,) is a biomimetic biped-climbing robot for potential applications in agriculture (like climbing and grasping), forestry and the building industry. Built with a modular approach, the robot consists of five joint modules connected in series and two special grippers mounted at the ends, with the scalability of changing degrees-of-freedom (DoFs). With this configuration, Climbot not only has superior mobility on multiple climbing media such as poles and trusses, but can also grasp and manipulate objects. It was inspired by observing the climbing patterns of animals such as caterpillars, chimpanzees, monkeys and sloths. Climbot may climb in several modes. The study proposed three basic climbing gaits, which are the inchworm gait, the swinging-around gait and the flipping-over gait. Autonomous climbing will be highly relevant for augmenting manual work in unstructured environments.

Guanjun et al. (2017) proposed a bio-soft robot inspired by the elephant trunk and octopus which has applications to robotic agricultural harvesting. A basic static model for axial elongation was established for the fundamental analysis of the bio-soft robot module’s features, such as iso-force, isobaric and isometric characteristics.

A plant-inspired robot, named Plantoid, with sensorized robotic roots was developed by Sadeghi et al. (2016). It is the first robot prototype inspired by plants and, in particular, by the movements, sensing capabilities and behaviour of their roots. Plantoid, integrates artificial roots able to respond to environmental conditions and stimuli, performing bending movements and obstacle avoidance response. Each robotic root integrates three soft spring-based actuators that imitate the different bending capability of plant roots through variable elongation of the actuators, obtained by the direct assembly of helical springs on the shafts of DC gear-motors. Each robotic root apex embeds a matrix of commercial gravity and temperature sensors and innovative sensors for touch and humidity, customized for the specific robotic root application. The combination of sensors and a root-inspired behaviour algorithm allowed the robotic roots to move and follow external stimuli in air.

8.8 Machine Learning Applications in Agricultural CPS

An important feature of intelligence in Precision Agriculture is the ability to learn automatically from historical data and experiences (generally called ‘machine learning’). Various learning methods and algorithms have been implemented in cyber physical systems, which facilitate continuous improvements, adaptations and learning from mistakes, as well as from success. Common applications of machine learning in cyber physical systems include, for example, fault detection (Sargolzaei et al. 2016), system security (Junejo and Goh 2016), pattern recognition or detection (Spezzano and Vinci 2015), predictive maintenance (Wu et al. 2018) and adaptive scheduling (Linard and Bueno 2016).

In agricultural CPS, machine learning research (Airlanga and Liu. 2019) has addressed several Precision Agriculture topics: image classification for plant recognition, plant disease detection using hyperspectral imaging (Moghadam et al. 2017; Wang et al. 2019), smart irrigation management (Goap et al. 2018), data mining and knowledge extraction (Schuster et al. 2011; Dimitriadis, and Goumopoulos 2008), detection and prediction of biotic stresses in plants (Behmann et al. 2015; Wani and Ashtankar 2017), crop yield evaluation (Finkelstein et al. 2015, 2017), predicting environmental factors (Taki et al. 2018; Pandey et al. 2019) and automatic plant phenotyping (Yahata et al. 2017).

Future research could explore predictive maintenance, pattern detection, enhanced collaboration among agents (human or non-human agents) and system security, as related to agriculture.

8.9 Summary

Exciting capabilities and opportunities are emerging in the application of robotics in Precision Agriculture. The main areas described and illustrated in this chapter, as well as in previous chapters for robotics in different Precision Agriculture tasks include: Precision Collaboration and collaborative control (collaborative robotics), cyber physical systems, human–robot collaborative system, cloud computing, multi-robots and robot fleets, bio inspired robots, and integration of machine learning.

A summary of the dimensions of Precision Collaboration in six Precision Agriculture case studies described in this chapter is shown below (Table 8.2).

Table 8.2 Dimensions of precision collaboration in precision agriculture case studies

Emerging trends and future developments are planned and anticipated in all the above areas. Particular advantages can be expected by cyber-augmentation for further smart automation and autonomy (autonomation), including cyber-augmented Precision Collaboration of stakeholder farmers and human–robot agents of Precision Agriculture.

A summary of the research challenges of Precision Collaboration in different Precision Agriculture tasks is given in Table 8.3.

Table 8.3 Summary of research challenges in Precision Collaboration and Precision Agriculture