Abstract
Robotic harvesters that use visual servoing must choose the best direction from which to approach the fruit to minimize occlusion and avoid obstacles that might interfere with the detection along the approach. This work proposes different approach strategies, compares them in terms of cycle times, and presents a failure analysis methodology of the different approach strategies. The different approach strategies are: in-field assessment by human observers, evaluation based on an overview image using advanced algorithms or remote human observers, or attempting multiple approach directions until the fruit is successfully reached. In the latter approach, each attempt costs time, which is a major bottleneck in bringing harvesting robots into the market. Alternatively, a single approach strategy that only attempts one direction can be applied if the best approach direction is known a-priori. The different approach strategies were evaluated for a case study of sweet pepper harvesting in laboratorial and greenhouse conditions. The first experiment, conducted in a commercial greenhouse, revealed that the fruit approach cycle time increased 8% and 116% for reachable and unreachable fruits respectively when the multiple approach strategy was applied, compared to the single approach strategy. The second experiment measured human observers’ ability to provide insights to approach directions based on overview images taken in both greenhouse and laboratorial conditions. Results revealed that human observers are accurate in detecting unapproachable directions while they tend to miss approachable directions. By detecting fruits that are unreachable (via automatic algorithms or human operators), harvesting cycle times can be significantly shortened leading to improved commercial feasibility of harvesting robots.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Bac, C.W.: Improving Obstacle Awareness for Robotic Harvesting of Sweet-Pepper. PhD thesis, Wageningen University (2015)
Bac, C.W., Hemming, J., van Tuijl, B.A.J., Barth, R., Wais, E., van Henten, E.J.: Performance evaluation of a harvesting robot for sweet pepper. J. Field Rob. 34(6), 1123–1139 (2017). https://doi.org/10.1002/rob.21709
Bac, C.W., Hemming, J., Van Henten, E.J.: Stem localization of sweet-pepper plants using the support wire as a visual cue. Comput. Electron. Agric. 105, 111–120 (2014). https://doi.org/10.1016/j.compag.2014.04.011
Bac, C.W., Henten, E.J., Hemming, J., Edan, Y.: Harvesting robots for high-value crops: state-of-the-art review and challenges ahead. J. Field Rob. 31(6), 888–911 (2014)
Bac, C.W., Roorda, T., Reshef, R., Berman, S., Hemming, J., van Henten, E.J.: Analysis of a motion planning problem for sweet-pepper harvesting in a dense obstacle environment. Biosyst. Eng. 146, 85–97 (2016)
Barth, R., Hemming, J., van Henten, E.J.: Design of an eye-in-hand sensing and servo control framework for harvesting robotics in dense vegetation. Biosyst. Eng. 146, 71–84 (2016). https://doi.org/10.1016/j.biosystemseng.2015.12.001
Bechar, A., Edan, Y.: Human-robot collaboration for improved target recognition of agricultural robots. Ind. Robot Int. J. 30(5), 432–436 (2003)
Berenstein, R., Edan, Y.: Human-Robot Cooperative precision spraying: Collaboration levels and optimization function. IFAC Proceedings 45(22), 799–804 (2012)
Bontsema, J., Hemming, J., Pekkeriet, E., Saeys, W., Edan, Y., Shapiro, A., Hočevar, M., Oberti, R., Armada, M., Ulbrich, H., et al.: CROPS: Clever robots for crops. Eng. Technol. Ref. 1(1), 11 (2015). https://doi.org/10.1049/etr.2015.0015
Chang, W.C.: Precise positioning of binocular eye-to-hand robotic manipulators. J. Intell. Robot. Syst. 49(3), 219–236 (2007)
Comba, L., Gay, P., Piccarolo, P., Ricauda Aimonino, D.: Robotics and automation for crop management : trends and perspective. In: International Conference Ragusa SHWA2010, pp. 471–478 (2010)
Edan, Y., Engel, B.A., Miles, G.E.: Intelligent control system simulation of an agricultural robot. J. Intell. Robot. Syst. 8(2), 267–284 (1993)
Edan, Y., Flash, T., Peiper, U.M., Shmulevich, I., Sarig, Y.: Near-minimum-time task planning for fruit-picking robots. IEEE Trans. Robot. Autom. 7(1), 48–56 (1991)
Eizicovits, D., Berman, S.: Efficient sensory-grounded grasp pose quality mapping for gripper design and online grasp planning. Robot. Auton. Syst. 62(8), 1208–1219 (2014). https://doi.org/10.1016/j.robot.2014.03.011
Eizicovits, D., van Tuijl, B., Berman, S., Edan, Y.: Integration of perception capabilities in gripper design using graspability maps. Biosyst. Eng. 146, 98–113 (2016). https://doi.org/10.1016/j.biosystemseng.2015.12.016. http://www.sciencedirect.com/science/article/pii/S1537511015001956
Fleiss, J.L., Cohen, J.: The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educ. Psychol. Meas. 33(3), 613–619 (1973)
Font, D., Pallejà, T., Tresanchez, M., Teixidó, M., Martinez, D., Moreno, J., Palacín, J.: Counting red grapes in vineyards by detecting specular spherical reflection peaks in RGB images obtained at night with artificial illumination. Comput. Electron. Agric. 108, 105–111 (2014). https://doi.org/10.1016/j.compag.2014.07.006. http://www.sciencedirect.com/science/article/pii/S0168169914001902
Gongal, A., Amatya, S., Karkee, M., Zhang, Q., Lewis, K.: Sensors and systems for fruit detection and localization: a review. Comput. Electron. Agric. 116, 8–19 (2015)
Han, K.S., Kim, S.C., Lee, Y.B., Kim, S.C., Im, D.H., Choi, H.K., Hwang, H.: Strawberry harvesting robot for bench-type cultivation. J. Biosyst. Eng. 37(1), 65–74 (2012)
Harel, B., Kurtser, P., Van Herck, L., Parmet, Y., Edan, Y.: Sweet pepper maturity evaluation via multiple viewpoints color analyses. In: Agen (2016)
Hellström, T., Ringdahl, O.: A software framework for agricultural and forestry robots. Ind. Robot Int. J. 40(1), 20–26 (2013). https://doi.org/10.1108/01439911311294228. http://www.emeraldinsight.com/journals.htm?issn=0143-991X&volume=40&issue=1&articleid=17072869&show=html
Hemming, J., Ruizendaal, J., Hofstee, J.W., van Henten, E.J.: Fruit detectability analysis for different camera positions in sweet-pepper. Sensors 14(4), 6032–6044 (2014)
Hosmer, D.W. Jr, Lemeshow, S., Sturdivant, R.X.: Applied Logistic Regression, vol. 398. Wiley, Hoboken (2013)
Kapach, K., Barnea, E., Mairon, R., Edan, Y., Ben-Shahar, O.: Computer vision for fruit harvesting robots–state of the art and challenges ahead. Int. J. Comput. Vision Robot. 3(1-2), 4–34 (2012)
Kitamura, S., Oka, K.: Recognition and cutting system of sweet pepper for picking robot in greenhouse horticulture. In: IEEE International Conference Mechatronics and Automation, 2005, vol. 4, pp. 1807–1812. https://doi.org/10.1109/ICMA.2005.1626834 (2005)
Kurtser, P., Arad, B., Ben Shahar, O., van Bree, M., Moonen, J., van Tujil, B., Edan, Y.: Robotic data acquisition of sweet pepper images for research and development. In: Israeli Conference on Robotics (2016)
Landis, J.R., Koch, G.G.: The Measurement of Observer Agreement for Categorical Data. Biometrics pp. 159–174 (1977)
Lass, L.W., Prather, T.S.: Detecting the locations of brazilian pepper trees in the everglades with a hyperspectral sensor. Weed Technol. 18(2), 437–442 (2004)
Lehnert, C., Sa, I., McCool, C., Upcroft, B., Perez, T.: Sweet pepper pose detection and grasping for automated crop harvesting. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 2428–2434. IEEE (2016)
Mann, M.P., Zion, B., Shmulevich, I., Rubinstein, D., Linker, R.: Combinatorial optimization and performance analysis of a multi-arm cartesian robotic fruit harvester-extensions of graph coloring. J. Intell. Robot. Syst. 82(3-4), 399–411 (2016)
McCool, C., Sa, I., Dayoub, F., Lehnert, C., Perez, T., Upcroft, B.: Visual detection of occluded crop: for automated harvesting. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 2506–2512 (2016)
Ringdahl, O., Kurtser, P., Barth, R., Edan, Y.: Operational flow of an autonomous sweetpepper harvesting robot. In: The 5th Israeli Conference on Robotics 2016, 13-14 April 2016. Air Force Conference Center Hertzilya, Israel (2016)
Ringdahl, O., Kurtser, P., Edan, Y.: Strategies for selecting best approach direction for a sweet-pepper harvesting robot. In: Towards Autonomous Robotic Systems : 18th annual conference, pp. 516–525. Springer, Guildford (2017)
Rosenbaum, D.A., Cohen, R.G., Meulenbroek, R.G.J., Vaughan, J.: Plans for grasping objects. In: Motor Control and Learning, pp. 9–25. Kluwer Academic Publishers, Boston. http://springerlink.bibliotecabuap.elogim.com/10.1007/0-387-28287-4_2 (2006)
Sa, I., Ge, Z., Dayoub, F., Upcroft, B., Perez, T., McCool, C.: Deepfruits: a fruit detection system using deep neural networks. Sensors 16(8), 1222 (2016)
Sakia, R.M.: The Box-Cox transformation technique: a review. Journal of the Royal Statistical Society. Series D (The Statistician) 41(2), 169–178 (1992). http://www.jstor.org/stable/2348250
Suh, H.K., Hofstee, J.W., van Henten, E.J.: Improved vegetation segmentation with ground shadow removal using an HDR camera. Precis. Agric. 19(2), 218–237 (2018). https://doi.org/10.1007/s11119-017-9511-z
Tadesse, T., Hewett, E.W., Nichols, M.A., Fisher, K.J.: Changes in physicochemical attributes of sweet pepper cv. Domino during fruit growth and development. Sci. Hortic. 93(2), 91–103 (2002). https://doi.org/10.1016/S0304-4238(01)00317-X. http://www.sciencedirect.com/science/article/pii/S030442380100317X
Tang, J.L., Chen, X.Q., Miao, R.H., Wang, D.: Weed detection using image processing under different illumination for site-specific areas spraying. Comput. Electron. Agric. 122, 103–111 (2016). https://doi.org/10.1016/j.compag.2015.12.016. http://www.sciencedirect.com/science/article/pii/S0168169915003981
Urrea, C., Muṅoz, J.: Path tracking of mobile robot in crops. J. Intell. Robot. Syst. 80(2), 193–205 (2015)
Vitzrabin, E., Edan, Y.: Adaptive thresholding with fusion using a RGBD sensor for red sweet-pepper detection. Biosyst. Eng. 146, 45–56 (2016)
Yoshida, J., Okuyama, S., Suzuki, H.: Fruit Harvesting Apparatus with Television Camera and Monitor. US Patent 4,519,193 (1985)
Zemmour, E., Kurtser, P., Edan, Y.: Dynamic thresholding algorithm for robotic apple detection. In: 2017 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), pp. 240–246 (2017)
Acknowledgements
This research was supported by the European Commission (SWEEPER GA no. 644313) and by Ben-Gurion University of the Negev through the Helmsley Charitable Trust, the Agricultural, Biological and Cognitive Robotics Initiative, the Marcus Endowment Fund and the Rabbi W. Gunther Plaut Chair in Manufacturing Engineering.
Author information
Authors and Affiliations
Corresponding authors
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Ringdahl, O., Kurtser, P. & Edan, Y. Evaluation of approach strategies for harvesting robots: Case study of sweet pepper harvesting. J Intell Robot Syst 95, 149–164 (2019). https://doi.org/10.1007/s10846-018-0892-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10846-018-0892-7