Abstract
Explosive growth of multimedia content leads to massive amount of images which are uploaded every day in the cyber world, medical imaging repository, and other areas. Retrieval of image of interest from internet or huge repository of image data set is still challenging and an open problem. Thus, content based image retrieval (CBIR) systems are developed. Success of CBIR system mainly depends on the image features which used for indexing and similarity measurement. CBIR system developed in deep learning framework has recently demonstrated promising results. In this paper, we present a modified-VGG16 (M-VGG16), deep convolution neural network (DCNN), for image feature extraction. These features are used for image indexing and retrieval in the CBIR system. In M-VGG16, we added \(1\times 1\) and \(3\times 3\) convolution kernels into input layer, followed by depth concatenation of the output of both convolution kernels. We apply principal component analysis (PCA) on the features obtained from M-VGG16 for getting robust features with reduced dimension, this leads to the compact image indexing and better retrieval performance. The performance M-VGG16 and M-VGG16 with PCA (M-VGG16 + PCA) is evaluated on benchmark datasets Corel-1k, Corel-10k, Coil-10k, and KADID-10k. Our proposed M-VGG16 + PCA model shows better results in terms of performance metric: precision, recall, and F1-score, when compared to state-of-art model AlexNet , VGG16, and other model proposed using DCNN and hand-crafted feature together. Moreover, in M-VGG16 + PCA the dimension of feature vector is 88% lesser than the M-VGG16 model.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Alzu’bi A, Amira A, Ramzan N (2015) Semantic content-based image retrieval: a comprehensive study. J Vis Commun Image Represent 32:20–54
Gurrin C, Smeaton AF, Doherty AR (2014) Lifelogging: personal big data. Found Trends Inf Retrieval 8(1):1–125
Smeulders AW, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380
Gudivada VN, Raghavan VV (1995) Design and evaluation of algorithms for image retrieval by spatial similarity. ACM Trans Inf Syst 13(2):115–144
Gu Y, Panda B, Haque K.A (2001) Design and analysis of data structures for querying image databases. In: Proceedings of the 2001 ACM symposium on applied computing, pp 236–241
Niblack CW, Barber R, Equitz W, Flickner M.D, Glasman E.H, Petkovic D, Yanker P, Faloutsos C, Taubin G (1993) Qbic project: querying images by content, using color, texture, and shape. In: Storage and retrieval for image and video databases, vol 1908, pp 173–187
Smith JR, Chang S-F (1997) Visualseek: a fully automated content-based image query system. In: Proceedings of the fourth ACM international conference on multimedia, pp 87–98
Wang JZ, Li J, Wiederhold G (2001) Simplicity: semantics-sensitive integrated matching for picture libraries. IEEE Trans Pattern Anal Mach Intell 23(9):947–963
Sivic J, Zisserman A (2003) Video google: a text retrieval approach to object matching in videos. In: IEEE international conference on computer vision, vol 3, pp 1470–1470
Zhu L, Jin H, Zheng R, Feng X (2014) Weighting scheme for image retrieval based on bag-of-visual-words. IET Image Process 8(9):509–518
Elsayad I, Martinet J, Urruty T, Djeraba C (2010) A new spatial weighting scheme for bag-of-visual-words. In: 2010 International workshop on content based multimedia indexing (CBMI), pp 1–6
Huang J, Kumar S.R, Mitra M, Zhu W.-J, Zabih R (1997) Image indexing using color correlograms. In: Proceedings of IEEE computer society conference on computer vision and pattern recognition, pp 762–768
Singha M, Hemachandran K (2012) Content based image retrieval using color and texture. Signal Image Process 3(1):39
Duanmu X (2010) Image retrieval using color moment invariant. In: 2010 Seventh international conference on information technology: new generations. IEEE, pp 200–203
Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987
Heikkilä M, Pietikäinen M, Schmid C (2006) Description of interest regions with center-symmetric local binary patterns. In: Computer vision, graphics and image processing, pp 58–69
Robert MH, Shanmugam K (1973) It shak dinstein “texture features for image classification”. IEEE Trans Syst Man Cybernet SMC-3:610–621
Gómez W, Pereira WCA, Infantosi AFC (2012) Analysis of co-occurrence texture statistics as a function of gray-level quantization for classifying breast ultrasound. IEEE Trans Med Imaging 31(10):1889–1899
Bronstein AM, Bronstein MM, Guibas LJ, Ovsjanikov M (2011) Shape google: geometric words and expressions for invariant shape retrieval. ACM Trans Graph 30(1):1–20
Wang X-Y, Yu Y-J, Yang H-Y (2011) An effective image retrieval scheme using color, texture and shape features. Comput Stand Interfaces 33(1):59–68
Zhou XS, Huang TS (2000) Cbir: from low-level features to high-level semantics. In: Image and video communications and processing 2000, vol 3974, pp 426–431
Liu W, Wang Z, Liu X, Zeng N, Liu Y, Alsaadi FE (2017) A survey of deep neural network architectures and their applications. Neurocomputing 234:11–26
Guo Y, Liu Y, Oerlemans A, Lao S, Wu S, Lew MS (2016) Deep learning for visual understanding: a review. Neurocomputing 187:27–48
Hinton G, Deng L, Yu D, Dahl GE, Mohamed A-R, Jaitly N, Senior A, Vanhoucke V, Nguyen P, Sainath TN et al (2012) Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process Mag 29(6):82–97
Babenko A, Slesarev A, Chigorin A, Lempitsky V (2014) Neural codes for image retrieval. In: European conference on computer vision, pp 584–599
Wan J, Wang D, Hoi S.C.H, Wu P, Zhu J, Zhang Y, Li J (2014) Deep learning for content-based image retrieval: a comprehensive study. In: Proceedings of the 22nd ACM international conference on multimedia, pp 157–166
Gong Y, Wang L, Guo R, Lazebnik S (2014) Multi-scale orderless pooling of deep convolutional activation features. In: European conference on computer vision, pp 392–407
Flickner M, Sawhney H, Niblack W, Ashley J, Huang Q, Dom B, Gorkani M, Hafner J, Lee D, Petkovic D et al (1995) Query by image and video content: the qbic system. Computer 28(9):23–32
Pentland A, Picard RW, Sclaroff S (1996) Photobook: content-based manipulation of image databases. Int J Comput Vis 18(3):233–254
Giveki D, Soltanshahi MA, Montazer GA (2017) A new image feature descriptor for content based image retrieval using scale invariant feature transform and local derivative pattern. Optik 131:242–254
Naghashi V (2018) Co-occurrence of adjacent sparse local ternary patterns: a feature descriptor for texture and face image retrieval. Optik 157:877–889
Tolias G, Sicre R, Jégou H (2015) Particular object retrieval with integral max-pooling of cnn activations. arXiv preprint arXiv:1511.05879
Tzelepi M, Tefas A (2016) Relevance feedback in deep convolutional neural networks for content based image retrieval. In: Proceedings of the 9th hellenic conference on artificial intelligence, pp 1–7
Saritha RR, Paul V, Kumar PG (2019) Content based image retrieval using deep learning process. Clust Comput 22(2):4187–4200
Maji S, Bose S (2021) Cbir using features derived by deep learning. ACM/IMS Trans Data Sci 2(3):1–24
Mustafic F, Prazina I, Ljubovic V (2019) A new method for improving content-based image retrieval using deep learning. In: 2019 XXVII international conference on information, communication and automation technologies (ICAT), pp 1–4
Liu P, Guo J-M, Wu C-Y, Cai D (2017) Fusion of deep learning and compressed domain features for content-based image retrieval. IEEE Trans Image Process 26(12):5706–5717
Cai Y, Li Y, Qiu C, Ma J, Gao X (2019) Medical image retrieval based on convolutional neural network and supervised hashing. IEEE Access 7:51877–51885
Özaydın U, Georgiou T, Lew M (2019) A comparison of CNN and classic features for image retrieval. In: 2019 International conference on content-based multimedia indexing (CBMI). IEEE, pp 1–4
Tzelepi M, Tefas A (2018) Deep convolutional learning for content based image retrieval. Neurocomputing 275:2467–2478
Shakarami A, Tarrah H (2020) An efficient image descriptor for image classification and cbir. Optik 214:164833
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:1097–1105
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp 248–255
Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L (2015) ImageNet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252
Martinez AM, Kak AC (2001) Pca versus lda. IEEE Trans Pattern Anal Mach Intell 23(2):228–233
Cichocki A, Amari S (2002) Adaptive blind signal and image processing: learning algorithms and applications. Wiley
Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 3431–3440
Zhang Z (2016) Derivation of backpropagation in convolutional neural network (CNN). University of Tennessee, Knoxville
Wang J (2021) Modeling objects, concepts, aesthetics and emotions in big visual data
Pavithra L, Sharmila TS (2018) An efficient framework for image retrieval using color, texture and edge features. Comput Electr Eng 70:580–593
Kanaparthi SK, Raju U, Shanmukhi P, Aneesha GK, Rahman MEU (2020) Image retrieval by integrating global correlation of color and intensity histograms with local texture features. Multimed Tools Appl 79(47):34875–34911
Verma M, Raman B, Murala S (2015) Local extrema co-occurrence pattern for color and texture image retrieval. Neurocomputing 165:255–269
Singh VP, Srivastava R (2018) Improved image retrieval using fast colour-texture features with varying weighted similarity measure and random forests. Multimed Tools Appl 77(11):14435–14460
Xie G, Huang Z, Guo B, Zheng Y, Yan Y (2020) Image retrieval based on the combination of region and orientation correlation descriptors. J Sens 6:66
Bhunia AK, Bhattacharyya A, Banerjee P, Roy PP, Murala S (2020) A novel feature descriptor for image retrieval by combining modified color histogram and diagonally symmetric co-occurrence texture pattern. Pattern Anal Appl 23(2):703–723
Alsmadi MK (2017) An efficient similarity measure for content based image retrieval using memetic algorithm. Egypt J Basic Appl Sci 4(2):112–122
Ashraf R, Ahmed M, Ahmad U, Habib MA, Jabbar S, Naseer K (2020) Mdcbir-mf: multimedia data for content-based image retrieval by using multiple features. Multimed Tools Appl 79(13):8553–8579
Bu H-H, Kim N-C, Kim S-H (2021) Content-based image retrieval using a combination of texture and color features. Hum Centric Comput Inf Sci 11:66
Ghodratnama S, Moghaddam HA (2021) Content-based image retrieval using feature weighting and c-means clustering in a multi-label classification framework. Pattern Anal Appl 24(1):1–10
Kayhan N, Fekri-Ershad S (2021) Content based image retrieval based on weighted fusion of texture and color features derived from modified local binary patterns and local neighborhood difference patterns. Multimed Tools Appl 66:1–28
Zeiler Matthew D, Rob F (2013) Visualizing and understanding convolutional networks. CoRR. arXiv:abs/1311.2901
Chu K, Liu G-H (2020) Image retrieval based on a multi-integration features model. Math Probl Eng 6:66
Nene SA, Nayar SK, Murase H et al (1996) Columbia object image library (coil-100)
Joseph A, Rex ES, Christopher S, Jose J (2021) Content-based image retrieval using hybrid k-means moth flame optimization algorithm. Arab J Geosci 14(8):1–14
Lin H, Hosu V, Saupe D (2019) Kadid-10k: A large-scale artificially distorted IQA database. In: 2019 Tenth international conference on quality of multimedia experience (QoMEX), pp 1–3
Lin H, Hosu V, Saupe D (2020) Deepfl-iqa: weak supervision for deep IQA feature learning. arXiv preprint arXiv:2001.08113
Acknowledgements
This work is supported by Science and Engineering Research Board (SERB) of the Department of Science and Technology (DST), Government of India, under Mathematical Research Impact Centric Support (MATRICS) scheme. The Grant No. MTR/2018/001103.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Kumar, S., Singh, M.K. & Mishra, M. Efficient Deep Feature Based Semantic Image Retrieval. Neural Process Lett 55, 2225–2248 (2023). https://doi.org/10.1007/s11063-022-11079-y
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11063-022-11079-y