Skip to main content
SpringerLink
Log in

Real-Life Agricultural Data Retrieval for Large-Scale Annotation Flow Optimization

  • Chapter
  • First Online:
Computer Vision and Machine Learning in Agriculture, Volume 2

Part of the book series: Algorithms for Intelligent Systems ((AIS))

  • 582 Accesses

Abstract

This chapter investigates the performance of deep image retrieval approaches for semantic clustering of agricultural images in two use cases: crop identification and crop emergence. In this domain, the classical public benchmarks are seldom applicable, as the data at hand is usually heavily imbalanced, and the variety of different appearances of objects of interest is significant. Hence, expert human annotators are required to manually annotate such images for training. Since annotation is time-consuming and labor-intensive, efficient retrieval techniques that prioritize annotation with good retrieval performance, even on unseen classes, are required. Approaches enabling classification networks to have such a flexible scope, including image retrieval followed by a clustering approach, to enable recognition of new classes exist but have not been extensively explored on agricultural data. More advanced architectures such as transformers have also not been applied to this data before. This chapter presents a solution to speed up annotation time by providing annotators semantically similar images to their target image. An image retrieval task is conducted to map crop images to a single feature vector. In case of crop emergence, first, a pre-trained object detection model is applied to detect the crop plants and then map each bounding box to a feature vector using an image retrieval model. On the crop identification use case and within an unbalanced test set of 25 K images and 22 classes, the largest class scored a retrieval accuracy of 97.66%, while an almost 55 times smaller class scored 99.65% on the same metric. The crop emergence identification use case model achieved 98.83% accuracy on a new class against 98% on a 7\(\times \) larger class from the training scope, within a test set of almost 3 K images and 13 classes.

Hiba was doing her internship at BASF while working on this book chapter.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 179.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Benos L, Tagarakis AC, Dolias G, Berruto R, Kateris D, Bochtis D (2021) Machine learning in agriculture: a comprehensive updated review. Sensors 21(11):3758

    Google Scholar 

  2. Cai E, Baireddy S, Yang C, Crawford M, Delp EJ (2020) Deep transfer learning for plant center localization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp 62–63

    Google Scholar 

  3. Kamilaris A, Prenafeta-Boldú FX (2018) Deep learning in agriculture: a survey. Comput Electron Agric 147:70–90

    Google Scholar 

  4. Hafiz AM, Parah SA, Bhat RUA (2021) Attention mechanisms and deep learning for machine vision: a survey of the state of the art

    Google Scholar 

  5. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M et al (2015) ImageNet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252

    Article  MathSciNet  Google Scholar 

  6. Van Horn G, Mac Aodha O, Song Y, Cui Y, Sun C, Shepard A, Adam H, Perona P, Belongie S (2018) The iNaturalist species classification and detection dataset. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8769–8778

    Google Scholar 

  7. Lin T-Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Lawrence Zitnick C (2014) Microsoft coco: common objects in context. In: European conference on computer vision. Springer, pp 740–755

    Google Scholar 

  8. Nilsback M-E, Zisserman A (2006) A visual vocabulary for flower classification. In: 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06), vol 2. IEEE, pp 1447–1454

    Google Scholar 

  9. Kumar N, Belhumeur PN, Biswas A, Jacobs DW, John Kress W, Lopez IC, Soares JVB (2012) Leafsnap: a computer vision system for automatic plant species identification. In: European conference on computer vision. Springer, pp 502–516

    Google Scholar 

  10. Zheng Y-Y, Kong J-L, Jin X-B, Wang X-Y, Su T-L, Zuo M (2019) CropDeep: the crop vision dataset for deep-learning-based classification and detection in precision agriculture. Sensors 19(5):1058

    Article  Google Scholar 

  11. Ling H, Gao J, Kar A, Chen W, Fidler S (2019) Fast interactive object annotation with curve-GCN. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5257–5266

    Google Scholar 

  12. Lin Z, Zhang Z, Chen L-Z, Cheng M-M, Lu S-P (2020) Interactive image segmentation with first click attention. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 13339–13348

    Google Scholar 

  13. Barbosa JZ, Prior SA, Pedreira GQ, Motta ACV, Poggere GC, Goularte GD (2020) Global trends in apps for agriculture. Multi-Sci J 3(1):16–20. ISSN 2359-6902

    Google Scholar 

  14. Shankar P, Werner N, Selinger S, Janssen O (2020) Artificial intelligence driven crop protection optimization for sustainable agriculture. In: 2020 IEEE/ITU international conference on artificial intelligence for good (AI4G). IEEE, pp 1–6

    Google Scholar 

  15. Birner R, Daum T, Pray C (2021) Who drives the digital revolution in agriculture? A review of supply-side trends, players and challenges. Appl Econ Perspect Policy

    Google Scholar 

  16. Gordo A, Almazan J, Revaud J, Larlus D (2017) End-to-end learning of deep visual representations for image retrieval. Int J Comput Vis 124(2):237–254

    Article  MathSciNet  Google Scholar 

  17. Latif A, Rasheed A, Sajid U, Ahmed J, Ali N, Ratyal NI, Zafar B, Dar SH, Sajid M, Khalil T (2019) Content-based image retrieval and feature extraction: a comprehensive review. Math Probl Eng 2019

    Google Scholar 

  18. Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380

    Google Scholar 

  19. Chen W, Liu Y, Wang W, Bakker E, Georgiou T, Fieguth P, Liu L, Lew MS (2021) Deep image retrieval: a survey. arXiv preprint arXiv:2101.11282

  20. Schroff F, Kalenichenko D, Philbin J (2015) FaceNet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 815–823

    Google Scholar 

  21. Zheng L, Shen L, Tian L, Wang S, Wang J, Tian Q (2015) Scalable person re-identification: a benchmark. In: Proceedings of the IEEE international conference on computer vision, pp 1116–1124

    Google Scholar 

  22. Liu Z, Luo P, Qiu S, Wang X, Tang X (2016) DeepFashion: powering robust clothes recognition and retrieval with rich annotations. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1096–1104

    Google Scholar 

  23. Qayyum A, Anwar SM, Awais M, Majid M (2017) Medical image retrieval using deep convolutional neural network. Neurocomputing 266:8–20

    Google Scholar 

  24. Tong X-Y, Xia G-S, Fan H, Zhong Y, Datcu M, Zhang L (2019) Exploiting deep features for remote sensing image retrieval: a systematic investigation. IEEE Trans Big Data 6(3):507–521

    Article  Google Scholar 

  25. Cheng H, Yang W, Tang R, Mao J, Luo Q, Li C, Wang A (2015) Distributed indexes design to accelerate similarity based images retrieval in airport video monitoring systems. In: 2015 12th international conference on fuzzy systems and knowledge discovery (FSKD). IEEE, pp 1908–1912

    Google Scholar 

  26. Liu Y, Huang Y, Gao Z (2014) Feature extraction and similarity measure for crime scene investigation image retrieval. J Xi’an Univ Posts Telecommun 19(6):11–16

    Google Scholar 

  27. Bakhshipour A, Jafari A (2018) Evaluation of support vector machine and artificial neural networks in weed detection using shape features. Comput Electron Agric 145:153–160

    Article  Google Scholar 

  28. Tóth BP, Tóth MJ, Papp D, Szücs G (2016) Deep learning and SVM classification for plant recognition in content-based large scale image retrieval. In: CLEF (working notes), pp 569–578

    Google Scholar 

  29. Chen X, You J, Tang H, Wang B, Gao Y (2021) Fine-grained plant leaf image retrieval using local angle co-occurrence histograms. In: 2021 IEEE international conference on image processing (ICIP), pp 1599–1603

    Google Scholar 

  30. Babenko A, Lempitsky V (2015) Aggregating deep convolutional features for image retrieval. arXiv preprint arXiv:1510.07493

  31. Trong VH, Yu G-H, Vu DT, Lee J-H, Toan NH, Kim J-Y (2020) A study on weeds retrieval based on deep neural network classification model. J Korean Inst Inf Technol 18(8):19–30

    Google Scholar 

  32. Yang Z, Yue J, Li Z, Zhu L (2018) Vegetable image retrieval with fine-tuning VGG model and image hash. IFAC-PapersOnLine 51(17):280–285

    Article  Google Scholar 

  33. Loddo A, Loddo M, Di Ruberto C (2021) A novel deep learning based approach for seed image classification and retrieval. Comput Electron Agric 187:106269

    Article  Google Scholar 

  34. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:1097–1105

    Google Scholar 

  35. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  36. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

    Google Scholar 

  37. Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2017) Inception-v4, inception-ResNet and the impact of residual connections on learning. In: Thirty-first AAAI conference on artificial intelligence

    Google Scholar 

  38. Yin H, Gu YH, Park C-J, Park J-H, Yoo SJ (2020) Transfer learning-based search model for hot pepper diseases and pests. Agriculture 10(10):439

    Google Scholar 

  39. Gao Z-Y, Xie H-X, Li J-F, Liu S-L (2018) Spatial-structure Siamese network for plant identification. Int J Pattern Recogn Artif Intell 32(11):1850035

    Article  Google Scholar 

  40. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008

    Google Scholar 

  41. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J, Houlsby N (2021) An image is worth 16x16 words: transformers for image recognition at scale. In: International conference on learning representations (ICLR)

    Google Scholar 

  42. Zhu X, Su W, Lu L, Li B, Wang X, Dai J (2020) Deformable DETR: deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159

  43. Ye L, Rochan M, Liu Z, Wang Y (2019) Cross-modal self-attention network for referring image segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10502–10511

    Google Scholar 

  44. Khan S, Naseer M, Hayat M, Zamir SW, Khan FS, Shah M (2021) Transformers in vision: a survey. arXiv preprint arXiv:2101.01169

  45. Wu H, Xiao B, Codella N, Liu M, Dai X, Yuan L, Zhang L (2021) CvT: introducing convolutions to vision transformers. arXiv preprint arXiv:2103.15808

  46. Sakai T (2007) Alternatives to bpref. In: Proceedings of the 30th annual international ACM SIGIR conference on research and development in information retrieval, pp 71–78

    Google Scholar 

  47. Hubert L, Arabie P (1985) Comparing partitions. J Classif 2:193–218

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. MINES NANCY, Campus Artem, 92 rue du Sergent Blandan, 54042, Nancy, France

    Hiba Najjar

  2. BASF Digital Farming GmbH, Im Zollhafen. 24, 50678, Cologne, Germany

    Priyamvada Shankar, Jonatan Aponte & Marek Schikora

Authors
  1. Hiba Najjar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Priyamvada Shankar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jonatan Aponte
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marek Schikora
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Priyamvada Shankar .

Editor information

Editors and Affiliations

  1. Department of Computer Science and Engineering, Jahangirnagar University, Dhaka, Bangladesh

    Mohammad Shorif Uddin

  2. Department of Mathematics, South Asian University, New Delhi, India

    Jagdish Chand Bansal

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Najjar, H., Shankar, P., Aponte, J., Schikora, M. (2022). Real-Life Agricultural Data Retrieval for Large-Scale Annotation Flow Optimization. In: Uddin, M.S., Bansal, J.C. (eds) Computer Vision and Machine Learning in Agriculture, Volume 2. Algorithms for Intelligent Systems. Springer, Singapore. https://doi.org/10.1007/978-981-16-9991-7_4

Download citation

Publish with us

Policies and ethics

Search

Navigation