Abstract
Sketch-based Image Retrieval (SBIR) is an approach where natural images are retrieved according to the given input sketch query. SBIR has many applications, for example, searching for a product given the sketch pattern in digital catalogs, searching for missing people given their prominent features from a digital people photo repository etc. The main challenge involved in implementing such a system is the absence of semantic information in the sketch query. In this work, we propose a combination of image prepossessing and deep learning-based methods to tackle this issue. A binary image highlighting the edges in the natural image is obtained using Canny-Edge detection algorithm. The deep features were extracted by an ImageNet based CNN model. Cosine similarity and Euclidean distance measures are adopted to generate the rank list of candidate natural images. Relevance feedback using Rocchio’s method is used to adapt the query of sketch images and feature weights according to relevant images and non-relevant images. During the experimental evaluation, the proposed approach achieved a Mean average precision (MAP) of 71.84%, promising performance in retrieving relevant images for the input query sketch images.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bui, T., Ribeiro, L.S.F., Ponti, M., Collomosse, J.: Generalisation and sharing in triplet convnets for sketch based visual search. arXiv:1611.05301 (2016)
Bui, T., Ribeiro, L., Ponti, M., Collomosse, J.: Compact descriptors for sketch-based image retrieval using a triplet loss convolutional neural network. Comput. Vis. Image Underst. 164, 27–37 (2017)
Chollet, F., et al.: Keras (2015). https://github.com/fchollet/keras
Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), vol. 1, pp. 886–893 (2005). https://doi.org/10.1109/CVPR.2005.177
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR09 (2009)
Devis, N., Pattara, N.J., Shoni, S., Mathew, S., Kumar, V.A.: Sketch based image retrieval using transfer learning. In: 2019 3rd International conference on Electronics, Communication and Aerospace Technology (ICECA), pp. 642–646 (2019)
Dorst, M.: Distinctive image features from scale-invariant keypoints abstract by Matthijs Dorst based on the paper by (2011)
Dutta, T., Singh, A., Biswas, S.: StyleGuide: zero-shot sketch-based image retrieval using style-guided image generation. IEEE Trans. Multimedia 1 (2020). https://doi.org/10.1109/TMM.2020.3017918
Eitz, M., Hays, J., Alexa, M.: How do humans sketch objects? ACM Trans. Graph. (Proc. SIGGRAPH) 31(4), 44:1–44:10 (2012)
Eitz, M., Hildebrand, K., Boubekeur, T., Alexa, M.: A descriptor for large scale image retrieval based on sketched feature lines. Association for Computing Machinery, New York (2009)
Wang, F., Kang, L., Li, Y.: Sketch-based 3D shape retrieval using convolutional neural networks. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1875–1883 (2015). https://doi.org/10.1109/CVPR.2015.7298797
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition (2015)
Hu, R., Collomosse, J.: A performance evaluation of gradient field hog descriptor for sketch based image retrieval. Comput. Vis. Image Underst. 117(7), 790–806 (2013)
Hunter, J.D.: Matplotlib: a 2D graphics environment. Comput. Sci. Eng. 9(3), 90–95 (2007). https://doi.org/10.1109/MCSE.2007.55
Jiang, T., Xia, G., Lu, Q.: Sketch-based aerial image retrieval. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 3690–3694 (2017). https://doi.org/10.1109/ICIP.2017.8296971
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems, NIPS 2012, vol. 1, pp. 1097–1105. Curran Associates Inc., Red Hook (2012)
Li, Y., Li, W.: A survey of sketch-based image retrieval. Mach. Vis. Appl. 29(7), 1083–1100 (2018). https://doi.org/10.1007/s00138-018-0953-8
Matsui, Y., Ito, K., Aramaki, Y., et al.: Sketch-based manga retrieval using Manga109 dataset. Multimedia Tools Appl. 76(20), 21811–21838 (2016). https://doi.org/10.1007/s11042-016-4020-z
Portenier, T., Hu, Q., Favaro, P., Zwicker, M.: SmartSketcher: sketchbased image retrieval with dynamic semantic re-ranking. In: Proceedings of the Symposium Sketch Based Interfaces Model (2017)
Qi, Q., Huo, Q., Wang, J., Sun, H., Cao, Y., Liao, J.: Personalized sketch-based image retrieval by convolutional neural network and deep transfer learning. IEEE Access 7, 16537–16549 (2019)
Qi, Y., et al.: Making better use of edges via perceptual grouping. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1856–1865 (2015). https://doi.org/10.1109/CVPR.2015.7298795
Qi, Y., Song, Y., Zhang, H., Liu, J.: Sketch-based image retrieval via Siamese convolutional neural network. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 2460–2464 (2016). https://doi.org/10.1109/ICIP.2016.7532801
Rocchio, J., Salton, G.: Information search optimization and interactive retrieval techniques. In: Proceedings of the Fall Joint Computer Conference, Part I, 30 November–1 December 1965, pp. 293–305 (1965)
Sangkloy, P., Burnell, N., Ham, C., Hays, J.: The sketchy database. ACM Trans. Graph. (TOG) 35, 1–12 (2016)
Schifanella, R., Redi, M., Aiello, L.M.: An image is worth more than a thousand favorites: surfacing the hidden beauty of Flickr pictures. In: ICWSM 2015: Proceedings of the 9th AAAI International Conference on Weblogs and Social Media. AAAI (2015)
Shechtman, E., Irani, M.: Matching local self-similarities across images and videos. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007). https://doi.org/10.1109/CVPR.2007.383198
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2015)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Kumar, N., Ahmed, R., B. Honnakasturi, V., Sowmya Kamath, S., Mayya, V. (2022). Sketch-Based Image Retrieval Using Convolutional Neural Networks Based on Feature Adaptation and Relevance Feedback. In: Mandal, J.K., De, D. (eds) Advanced Techniques for IoT Applications. EAIT 2021. Lecture Notes in Networks and Systems, vol 292. Springer, Singapore. https://doi.org/10.1007/978-981-16-4435-1_12
Download citation
DOI: https://doi.org/10.1007/978-981-16-4435-1_12
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-16-4434-4
Online ISBN: 978-981-16-4435-1
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)