Abstract
Salient object detection is used as a pre-process in many computer vision tasks (such as salient object segmentation, video salient object detection, etc.). When performing salient object detection, depth information can provide clues to the location of target objects, so effective fusion of RGB and depth feature information is important. In this paper, we propose a new feature information aggregation approach, weighted group integration (WGI), to effectively integrate RGB and depth feature information. We use a dual-branch structure to slice the input RGB image and depth map separately and then merge the results separately by concatenation. As grouped features may lose global information about the target object, we also make use of the idea of residual learning, taking the features captured by the original fusion method as supplementary information to ensure both accuracy and completeness of the fused information. Experiments on five datasets show that our model performs better than typical existing approaches for four evaluation metrics.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Marchesotti, L.; Cifarelli, C.; Csurka, G. A framework for visual saliency detection with applications to image thumbnailing. In: Proceedings of the IEEE 12th International Conference on Computer Vision, 2232–2239, 2009.
Ding, Y.; Xiao, J.; Yu. J. Importance filtering for image retargeting. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, 89–96, 2011.
Goferman, S.; Zelnik-Manor, L.; Tal, A. Context-aware saliency detection. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 34, No. 10, 1915–1926, 2012.
Wang, W.; Shen, J. Deep cropping via attention box prediction and aesthetics assessment. In: Proceedings of the IEEE International Conference on Computer Vision, 2205–2213, 2017.
Guo, C. L.; Zhang, L. M. A novel multiresolution spatiotemporal saliency detection model and its applications in image and video compression. IEEE Transactions on Image Processing Vol. 19, No. 1, 185–198, 2010.
Rutishauser, U.; Walther, D.; Koch, C.; Perona, P. Is bottom-up attention useful for object recognition? In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, II, 2004.
Han, J.; Ngan, K. N.; Li, M. J.; Zhang, H. J. Unsupervised extraction of visual attention objects in color images. IEEE Transactions on Circuits and Systems for Video Technology Vol. 16, No. 1, 141–145, 2006.
Liu, Z.; Shi, R.; Shen, L. Q.; Xue, Y. Z.; Ngan, K. N., Zhang, Z. Y. Unsupervised salient object segmentation based on kernel density estimation and two-phase graph cut. IEEE Transactions on Multimedia Vol. 14, No. 4, 1275–1289, 2012.
Jerripothula, K. R.; Cai, J. F.; Yuan, J. S. Image co-segmentation via saliency co-fusion. IEEE Transactions on Multimedia Vol. 18, No. 9, 1896–1909, 2016.
Ye, L. W.; Liu, Z.; Li, L. N.; Shen, L. Q.; Bai, C.; Wang, Y. Salient object segmentation via effective integration of saliency and objectness. IEEE Transactions on Multimedia Vol. 19, No. 8, 1742–1756, 2017.
Borji, A.; Frintrop, S.; Sihite, D. N.; Itti, L. Adaptive object tracking by learning background context. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 23–30, 2012.
Hong, S.; You, T.; Kwak, S.; Han, B. Online tracking by learning discriminative saliency map with convolutional neural network. In: Proceedings of the International Conference on Machine Learning, 597–606, 2015.
Lee, H.; Kim, D. Salient region-based online object tracking. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision, 1170–1177, 2018.
Gao, Y.; Wang, M.; Tao, D. C.; Ji, R. R.; Dai, Q. H. 3-D object retrieval and recognition with hypergraph analysis. IEEE Transactions on Image Processing Vol. 21, No. 9, 4290–4303, 2012.
He, J.; Feng, J.; Liu, X.; Cheng, T.; Lin, T.; Chung, H.; Chang, S. Mobile product search with bag of hash bits and boundary reranking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3005–3012, 2012.
Hou, Q.; Cheng, M.-M.; Hu, X.; Borji, A.; Tu, Z.; Torr, P. H. Deeply supervised salient object detection with short connections. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3203–3212, 2017.
Wang, W.; Zhao, S.; Shen, J.; Hoi, S. C.; Borji, A. Salient object detection with pyramid attention and salient edges. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1448–1457, 2019.
Fan, D.; Wang, W.; Cheng, M.; Shen, J. Shifting more attention to video salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 8554–8564, 2019.
Bi, H.; Lu, D.; Li, N.; Yang, L.; Guan, H. Multi-level model for video saliency detection. In: Proceedings of the IEEE International Conference on Image Processing, 4654–4658, 2019.
Fu, H. Z.; Cao, X. C.; Tu, Z. W. Cluster-based co-saliency detection. IEEE Transactions on Image Processing Vol. 22, No. 10, 3766–3778, 2013.
Zhang, K.; Li, T.; Shen, S.; Liu, B.; Chen, J.; Liu, Q. Adaptive graph convolutional network with attention graph clustering for co-saliency detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9050–9059, 2020.
Wang, A. Z.; Wang, M. H.; Li, X. Y.; Mi, Z. T.; Zhou, H. A two-stage Bayesian integration framework for salient object detection on light field. Neural Processing Letters Vol. 46, No. 3, 1083–1094, 2017.
Zhang, M.; Ji, W.; Piao, Y. R.; Li, J. J.; Zhang, Y.; Xu, S.; Lu, H. LFNet: Light field fusion network for salient object detection. IEEE Transactions on Image Processing Vol. 29, 6276–6287, 2020.
Zhang, J.; Wang, X. Light field salient object detection via hybrid priors. In: Proceedings of the International Conference on Multimedia Modeling, 361–372, 2020.
Zhao, J.; Cao, Y.; Fan, D.; Cheng, M.; Li, X.; Zhang, L. Contrast prior and fluid pyramid integration for RGBD salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3927–3936, 2019.
Zhang, J.; Fan, D. P.; Dai, Y.; Anwar, S.; Saleh, F. S.; Zhang, T.; Barnes, N. UC-Net: Uncertainty inspired RGB-D saliency detection via conditional variational autoencoders. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8582–8591, 2020.
Fu, K.; Fan, D. P.; Ji, G. P.; Zhao, Q. JL-DCF: Joint learning and densely-cooperative fusion framework for RGB-D salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3049–3059, 2020.
Peng, H. W.; Li, B.; Xiong, W. H.; Hu, W. M.; Ji, R. R. RGBD salient object detection: A benchmark and algorithms. In: Computer Vision — ECCV 2014. Lecture Notes in Computer Science, Vol. 8691. Fleet, D.; Pajdla, T.; Schiele, B.; Tuytelaars, T. Eds. Springer Cham, 92–109, 2014.
Song, H. K.; Liu, Z.; Du, H.; Sun, G. L.; Le Meur, O.; Ren, T. W. Depth-aware salient object detection and segmentation via multiscale discriminative saliency fusion and bootstrap learning. IEEE Transactions on Image Processing Vol. 26, No. 9, 4204–4216, 2017.
Qu, L. Q.; He, S. F.; Zhang, J. W.; Tian, J. D.; Tang, Y. D.; Yang, Q. X. RGBD salient object detection via deep fusion. IEEE Transactions on Image Processing Vol. 26, No. 5, 2274–2285, 2017.
Liu, Z. Y.; Shi, S.; Duan, Q. T.; Zhang, W.; Zhao, P. Salient object detection for RGB-D image by single stream recurrent convolution neural network. Neurocomputing Vol. 363, 46–57, 2019.
Feng, D.; Barnes, N.; You, S.; Mccarthy, C. Local background enclosure for RGB-D salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2343–2350, 2016.
Shigematsu, R.; Feng, D.; You, S.; Barnes, N. Learning RGB-D salient object detection using background enclosure, depth contrast, and top-down features. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, 2749–2757, 2017.
Piao, Y.; Ji, W.; Li, J.; Zhang, M.; Lu, H. Depth-induced multi-scale recurrent attention network for saliency detection. In: Proceedings of the IEEE International Conference on Computer Vision, 7254–7263, 2019.
Chen, H.; Li, Y. F.; Su, D. Multi-modal fusion network with multi-scale multi-path and cross-modal interactions for RGB-D salient object detection. Pattern Recognition Vol. 86, 376–385, 2019.
Desingh, K.; Madhava Krishna, K.; Rajan, D.; Jawahar, C. V. Depth really matters: Improving visual salient region detection with depth. In: Proceedings of the British Machine Vision Conference, 98.1–98.11, 2013.
Fan, X.; Liu, Z.; Sun, G. Salient region detection for stereoscopic images. In: Proceedings of the 19th International Conference on Digital Signal Processing, 454–458, 2014.
Cheng, Y. P.; Fu, H. Z.; Wei, X. X.; Xiao, J. J.; Cao, X. C. Depth enhanced saliency detection method. In: Proceedings of the International Conference on Internet Multimedia Computing and Service, 23–27, 2014.
Cong, R. M.; Lei, J. J.; Zhang, C. Q.; Huang, Q. M.; Cao, X. C.; Hou, C. P. Saliency detection for stereoscopic images based on depth confidence analysis and multiple cues fusion. IEEE Signal Processing Letters Vol. 23, No. 6, 819–823, 2016.
Cong, R.; Lei, J.; Fu, H.; Hou, J.; Huang, Q.; Kwong, S. Going from RGB to RGBD saliency: A depth-guided transformation model. IEEE Transactions on Cybernetics Vol. 50, No. 8, 3627–3639, 2019.
Ju, R.; Ge, L.; Geng, W.; Ren, T.; Wu, G. Depth saliency based on anisotropic center-surround difference. In: Proceedings of the IEEE International Conference on Image Processing, 1115–1119, 2014.
Quo, J.; Ren, T.; Bei, J. Salient object detection for RGB-D image via saliency evolution. In: Proceedings of the IEEE International Conference on Multimedia and Expo, 1–6, 2016.
Wang, A. Z.; Wang, M. H. RGB-D salient object detection via minimum barrier distance transform and saliency fusion. IEEE Signal Processing Letters Vol. 24, No. 5, 663–667, 2017.
Liang, F. F.; Duan, L. J.; Ma, W.; Qiao, Y. H.; Cai, Z.; Qing, L. Y. Stereoscopic saliency model using contrast and depth-guided-background prior. Neurocomputing Vol. 275, 2227–2238, 2018.
Niu, Y.; Geng, Y.; Li, X.; Liu, F. Leveraging stereopsis for saliency analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 454–461, 2012.
Chen, H.; Li, Y. F. Progressively complementarity-aware fusion network for RGB-D salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3051–3060, 2018.
Han, J. W.; Chen, H.; Liu, N.; Yan, C. G.; Li, X. L. CNNs-based RGB-D saliency detection via cross-view transfer and multiview fusion. IEEE Transactions on Cybernetics Vol. 48, No. 11, 3171–3183, 2018.
Huang, P.; Shen, C.; Hsiao, H. RGBD salient object detection using spatially coherent deep learning framework. In: Proceedings of the IEEE 23rd International Conference on Digital Signal Processing, 1–5, 2018.
Zhu, C.; Cai, X.; Huang, K.; Li, T. H.; Li, G. PDNet: Prior-model guided depth-enhanced network for salient object detection. In: Proceedings of the IEEE International Conference on Multimedia and Expo, 199–204, 2019.
Wang, N. N.; Gong, X. J. Adaptive fusion for RGB-D salient object detection. IEEE Access Vol. 7, 55277–55284, 2019.
Zhou, T.; Fan, D.-P.; Cheng, M.-M.; Shen, J.; Shao, L. RGB-D salient object detection: A survey. arXiv preprint arXiv:2008.00230, 2020.
Zhang, J.; Fan, D.-P.; Dai, Y.; Anwar, S.; Saleh, F.; Aliakbarian, S.; Barnes, N. Uncertainty inspired RGB-D saliency detection. arXiv preprint arXiv:2009.03075, 2020.
Zhang, M.; Zhang, Y.; Piao, Y. R.; Hu, B. Q.; Lu, H. C. Feature reintegration over differential treatment: A top-down and adaptive fusion network for RGB-D salient object detection. In: Proceedings of the 28th ACM International Conference on Multimedia, 4107–4115, 2020.
Zhang, J.; Wang, M.; Lin, L.; Yang, X.; Gao, J.; Rui, Y. Saliency detection on light field. ACM Transactions on Multimedia Computing, Communications, and Applications Vol. 13, No. 3, 1–22, 2017.
Chen, H.; Li, Y. F. Three-stream attention-aware network for RGB-D salient object detection. IEEE Transactions on Image Processing Vol. 28, No. 6, 2825–2835, 2019.
Piao, Y.; Rong, Z.; Zhang, M.; Ren, W.; H. Lu. A2dele: Adaptive and attentive depth distiller for efficient RGB-D salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 9060–9069, 2020.
Li, C. Y.; Cong, R. M.; Kwong, S.; Hou, J. H.; Fu, H. Z.; Zhu, G. P.; Zhang, D. W.; Huang, Q. M. ASIF-net: Attention steered interweave fusion network for RGB-D salient object detection. IEEE Transactions on Cybernetics Vol. 51, No. 1, 88–100, 2021.
Fan, D. P.; Lin, Z.; Zhang, Z.; Zhu, M. L.; Cheng, M. M. Rethinking RGB-D salient object detection: Models, data sets, and large-scale benchmarks. IEEE Transactions on Neural Networks and Learning Systems doi: https://doi.org/10.1109/TNNLS.2020.2996406, 2020.
Borji, A.; Sihite, D. N.; Itti, L. Salient object detection: A benchmark. In: Computer Vision — ECCV 2012. Lecture Notes in Computer Science, Vol. 7573. Fitzgibbon, A.; Lazebnik, S.; Perona, P.; Sato, Y.; Schmid, C. Eds. Springer Berlin Heidelberg, 414–429, 2012.
Achanta, R.; Hemami, S. S.; Estrada, F. J.; Susstrunk, S. Frequency-tuned salient region detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1597–1604, 2009.
Fan, D.; Cheng, M.; Liu, Y.; Li, T.; Borji, A. Structure-measure: A new way to evaluate foreground maps. In: Proceedings of the IEEE International Conference on Computer Vision, 4558–4567, 2017.
Fan, D.; Gong, C.; Cao, Y.; Ren, B.; Cheng, M.; Borji, A. Enhanced-alignment measure for binary foreground map evaluation. arXiv preprint arXiv:1805.10421, 2018.
Acknowledgements
This work was supported by the NEPU Natural Science Foundation under Grants Nos. 2017PY ZL-05, 2018QNL-51, JY_CX_CX06 2018, JY_CX_JG06 2018, and JY_CX_14_2020.
Author information
Authors and Affiliations
Corresponding author
Additional information
Yanliang Ge received his bachelor degree in communications engineering in 2002 from Northeast Petroleum University, Daqing, China. He received his master degree in 2008 from Northeast Petroleum University in oil and gas information and control engineering. Currently he is an associate professor in the School of Electrical Information Engineering in Northeast Petroleum University. His main research interests concern digital watermarking, signal processing, and digital video processing.
Cong Zhang is pursuing her master degree at Northeast Petroleum University. Her current research interests include camouflaged object detection, RGB-D salient object detection, and deep learning.
Kang Wang is pursuing his master degree at Northeast Petroleum University. His current research interests include co-saliency detection, camouflaged object detection, RGB-D salient object detection, and deep learning.
Ziqi Liu is pursuing her master degree at Northeast Petroleum University. Her current research interests include RGB-D salient object detection, camouflaged object detection, RGB salient object detection, and deep learning.
Hongbo Bi received his bachelor and master degrees in communications engineering from Northeast Petroleum University in 2001 and 2004, respectively. He received his Ph.D. degree in 2013 from Beijing University of Posts and Telecommunications and worked as a postdoctoral fellow in Harbin Engineering University in 2014–2017. He also worked as a visiting scholar in the University of Waterloo, Canada in 2014–2015. Currently, he is an associate professor in the School of Electrical Information Engineering in Northeast Petroleum University. His main research interests focus on salient object detection, camouflaged object detection, compressive sensing, deep learning, digital watermarking, and signal processing.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Ge, Y., Zhang, C., Wang, K. et al. WGI-Net: A weighted group integration network for RGB-D salient object detection. Comp. Visual Media 7, 115–125 (2021). https://doi.org/10.1007/s41095-020-0200-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-020-0200-x