Abstract
Single image super-resolution has attracted increasing attention and has a wide range of applications in satellite imaging, medical imaging, computer vision, security surveillance imaging, remote sensing, objection detection, and recognition. Recently, deep learning techniques have emerged and blossomed, producing “the state-of-the-art” in many domains. Due to their capability in feature extraction and mapping, it is very helpful to predict high-frequency details lost in low-resolution images. In this paper, we give an overview of recent advances in deep learning-based models and methods that have been applied to single image super-resolution tasks. We also summarize, compare and discuss various models from the past and present for comprehensive understanding and finally provide open problems and possible directions for future research.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
D. Glasner, S. Bagon, M. Irani. Super-resolution from a single image. In Proceedings of the 12th International Conference on Computer Vision, IEEE, Kyoto, Japan, pp. 349–356, 2009. DOI: https://doi.org/10.1109/ICCV.2009.5459271.
J. B. Huang, A. Singh, N. Ahuja. Single image super-resolution from transformed self-exemplars. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, pp. 5197–5206, 2015. DOI: https://doi.org/10.1109/CVPR.2015.7299156.
W. T. Freeman, E. C. Pasztor, O. T. Carmichael. Learning low-level vision. International Journal of Computer Vision, vol. 40, no. 1, pp. 25–47, 2000. DOI: https://doi.org/10.1023/A:1026501619075.
W. T. Freeman, T. R. Jones, E. C. Pasztor. Example-based super-resolution. IEEE Computer Graphics and Applications, vol. 22, no. 2, pp. 56–65, 2002. DOI: https://doi.org/10.1109/38.988747.
H. Chang, D. Y. Yeung, Y. M. Xiong. Super-resolution through neighbor embedding. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, USA, 2004. DOI: https://doi.org/10.1109/CVPR.2004.1315043.
C. Y. Yang, M. H. Yang. Fast direct super-resolution by simple functions. In Proceedings of IEEE International Conference on Computer Vision, Sydney, Australia, pp. 561–568, 2013. DOI: https://doi.org/10.1109/ICCV.2013.75.
R. Timofte, V. De Smet, L. Van Gool. Anchored neighborhood regression for fast example-based super-resolution. In Proceedings of IEEE International Conference on Computer Vision, Sydney, Australia, pp. 1920–1927, 2013. DOI: https://doi.org/10.1109/ICCV.2013.241.
R. Timofte, V. De Smet, L. Van Gool. A+: Adjusted anchored neighborhood regression for fast super-resolution. In Proceedings of the 12th Asian Conference on Computer Vision, Springer, Singapore, pp. 111–126, 2015. DOI: https://doi.org/10.1007/978-3-319-16817-3_8.
S. Schulter, C. Leistner, H. Bischof. Fast and accurate image upscaling with super-resolution forests. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, pp. 3791–3799, 2015. DOI: https://doi.org/10.1109/CVPR.2015.7299003.
E. Pérez-Pellitero, J. Salvador, J. Ruiz-Hidalgo, B. Rosenhahn. PSyCo: Manifold span reduction for super resolution. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 1837–1845, 2016. DOI: https://doi.org/10.1109/CVPR.2016.203.
J. C. Yang, J. Wright, T. S. Huang, Y. Ma. Image super-resolution via sparse representation. IEEE Transactions on Image Processing, vol. 19, no. 11, pp. 2861–2873, 2010. DOI: https://doi.org/10.1109/TIP.2010.2050625.
J. C. Yang, Z. W. Wang, Z. Lin, S. Cohen, T. Huang. Coupled dictionary training for image super-resolution. IEEE Transactions on Image Processing, vol. 21, no. 8, pp. 3467–3478, 2012. DOI: https://doi.org/10.1109/TIP.2012.2192127.
T. Peleg, M. Elad. A statistical prediction model based on sparse representations for single image super-resolution. IEEE Transactions on Image Processing, vol. 23, no. 6, pp. 2569–2582, 2014. DOI: https://doi.org/10.1109/TIP.2014.2305844.
S. L. Wang, L. Zhang, Y. Liang, Q. Pan. Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Providence, USA, pp. 2216–2223, 2012. DOI: https://doi.org/10.1109/CV-PR.2012.6247930.
L. He, H. R. Qi, R. Zaretzki. Beta process joint dictionary learning for coupled feature spaces with application to single image super-resolution. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Portland, USA, pp. 345–352, 2013. DOI: https://doi.org/10.1109/CVPR.2013.51.
C. Dong, C. C. Loy, K. M. He, X. O. Tang. Learning a deep convolutional network for image super-resolution. In Proceedings of the 13th European Conference on Computer Vision, Springer, Zurich, Switzerland, pp. 184–199, 2014. DOI: https://doi.org/10.1007/978-3-319-10593-2_13.
C. Dong, C. C. Loy, K. M. He, X. O. Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 2, pp. 295–307, 2016. DOI: https://doi.org/10.1109/TPAMI.2015.2439281.
J. Kim, J. Kwon Lee, K. Mu Lee. Accurate image super-resolution using very deep convolutional networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 1646–1654, 2016. DOI: https://doi.org/10.1109/CVPR.2016.182.
J. Kim, J. Kwon Lee, K. Mu Lee. Deeply-recursive convolutional network for image super-resolution. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 1637–1645, 2016. DOI: https://doi.org/10.1109/CVPR.2016.181.
Y. Tai, J. Yang, X. M. Liu. Image super-resolution via deep recursive residual network. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, vol. 1, pp. 2790–2798, 2017. DOI: https://doi.org/10.1109/CVPR.2017.298.
X. J. Mao, C. H. Shen, Y. B. Yang. Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections, [Online], Available: https://arxiv.org/abs/1606.08921, May, 2018.
J. Yamanaka, S. Kuwashima, T. Kurita. Fast and accurate image super resolution by deep CNN with skip connection and network in network. In Proceedings of the 24th International Conference on Neural Information Processing, Springer, Guangzhou, China, 2017. DOI: https://doi.org/10.1007/978-3-319-70096-0_23.
T. Tong, G. Li, X. J. Liu, Q. Q. Gao. Image super-resolution using dense skip connections. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Venice, Italy, pp. 4809–4817, 2017. DOI: https://doi.org/10.1109/ICCV.2017.514.
Y. L. Zhang, K. P. Li, K. Li, L. C. Wang, B. N. Zhong, Y. Fu. Image super-resolution using very deep residual channel attention networks. In Proceedings of the 15th European Conference on Computer Vision, Springer, Munich, Germany, pp. 286–301, 2018. DOI: https://doi.org/10.1007/978-3-030-01234-2_18.
Z. S. Zhong, T. C. Shen, Y. B. Yang, Z. C. Lin, C. Zhang. Joint sub-bands learning with clique structures for wavelet domain super-resolution. In Proceedings of the 32nd Conference on Neural Information Processing Systems, Curran Associates, Inc., Montréal, Canada, pp. 165–175, 2018.
Y. L. Zhang, Y. P. Tian, Y. Kong, B. N. Zhong, Y. Fu. Residual dense network for image super-resolution. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 2472–2481, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00262.
J. H. Yu, Y. C. Fan, J. C. Yang, N. Xu, Z. W. Wang, X. C. Wang, T. Huang. Wide Activation for Efficient and Accurate Image Super-resolution, [Online], Available: https://arxiv.org/abs/1808.08718v1, April 8, 2019.
N. Ahn, B. Kang, K. A. Sohn. Fast, accurate, and lightweight super-resolution with cascading residual network. In Proceedings of the 15th European Conference on Computer Vision, Springer, Munich, Germany, pp. 252–268, 2018. DOI: https://doi.org/10.1007/978-3-030-01249-6_16.
Z. Hui, X. M. Wang, X. B. Gao. Fast and accurate single image super-resolution via information distillation network. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 723–731, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00082.
W. S. Lai, J. B. Huang, N. Ahuja, M. H. Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, vol. 2, pp. 5835–5843, 2017. DOI: https://doi.org/10.1109/CVPR.2017.618.
R. S. Asamwar, K. M. Bhurchandi, A. S. Gandhi. Interpolation of images using discrete wavelet transform to simulate image resizing as in human vision. International Journal of Automation and Computing, vol. 7, no. 1, pp. 9–16, 2010. DOI: https://doi.org/10.1007/s11633-010-0009-7.
B. Lim, S. Son, H. Kim, S. Nah, K. M. Lee. Enhanced deep residual networks for single image super-resolution. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops, IEEE, Honolulu, USA, vol. 1, pp. 1132–1140, 2017. DOI: https://doi.org/10.1109/CVPRW.2017.151.
Y. F. Wang, F. Perazzi, B. McWilliams, A. Sorkine-Hornung, O. Sorkine-Hornung, C. Schroers. A fully progressive approach to single-image super-resolution. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, IEEE, Salt Lake City, USA, 2018. DOI: https://doi.org/10.1109/CVPRW.2018.00131.
M. Haris, G. Shakhnarovich, N. Ukita. Deep back-projection networks for super-resolution. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 1664–1673, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00179.
K. Zhang, W. M. Zuo, L. Zhang. Learning a single convolutional super-resolution network for multiple degradations. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 3262–3271, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00344.
A. Shocher, N. Cohen, M. Irani. Zero-shot super-resolution using deep internal learning. In Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 3118–3126, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00329.
Q. L. Liao, T. Poggio. Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex, [Online], Available: https://arxiv.org/abs/1604.03640, July 10, 2018.
W. Han, S. Y. Chang, D. Liu, M. Yu, M. Witbrock, T. S. Huang. Image super-resolution via dual-state recurrent networks. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 1654–1663, 2018. DOI: https://doi.org/10.1109/CV-PR.2018.00178.
Y. Tai, J. Yang, X. M. Liu, C. Y. Xu. MemNet: A persistent memory network for image restoration. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Venice, Italy, pp. 4539–4547, 2017. DOI: https://doi.org/10.1109/ICCV.2017.486.
X. L. Wang, R. Girshick, A. Gupta, K. M. He. Non-local neural networks. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 7794–7803, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00813.
D. Liu, B. H. Wen, Y. C. Fan, C. C. Loy, T. S. Huang. Non-local recurrent network for image restoration. In Proceedings of the 32nd Conference on Neural Information Processing Systems, Curran Associates, Inc., Montréal, Canada, pp. 1680–1689, 2018.
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems, MIT Press, Montreal, Canada, pp. 2672–2680, 2014.
C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. P. Aitken, A. Tejani, J. Totz, Z. H. Wang, W. Z. Shi. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, vol. 2, pp. 105–114, 2017. DOI: https://doi.org/10.1109/CVPR.2017.19.
M. S. Sajjadi, B. Schölkopf, M. Hirsch. EnhanceNet: Single image super-resolution through automated texture synthesis. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Venice, Italy, pp. 4501–4510, 2017. DOI: https://doi.org/10.1109/ICCV.2017.481.
J. Johnson, A. Alahi, F. F. Li. Perceptual losses for real-time style transfer and super-resolution. In Proceedings of the 14th European Conference on Computer Vision, Springer, Amsterdam, The Netherlands, pp. 694–711, 2016. DOI: https://doi.org/10.1007/978-3-319-46475-6_43.
S. J. Park, H. Son, S. Cho, K. S. Hong, S. Lee. SRFeat: Single image super-resolution with feature discrimination. In Proceedings of the 15th European Conference on Computer Vision, Springer, Munich, Germany, pp. 439–455, 2018. DOI: https://doi.org/10.1007/978-3-030-01270-0_27.
M. Bevilacqua, A. Roumy, C. Guillemot, M. L. Alberi-Morel. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In Proceedings of British Machine Vision Conference, BMVA Press, Surrey, UK, 2012.
R. Zeyde, M. Elad, M. Protter. On single image scale-up using sparse-representations. In Proceedings of the 7th International Conference on Curves and Surfaces, Springer, Avignon, France, pp. 711–730, 2010. DOI: https://doi.org/10.1007/978-3-642-27413-8_47.
D. Martin, C. Fowlkes, D. Tal, J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings the 8th IEEE International Conference on Computer Vision, IEEE, Vancouver, Canada, 2001. DOI: https://doi.org/10.1109/ICCV.2001.937655.
V. K. Ha, J. C. Ren, X. Y. Xu, S. Zhao, G. Xie, V. M. Vargas. Deep learning based single image super-resolution: A survey. In Proceedings of the 9th International Conference on Brain Inspired Cognitive Systems, Springer, Xi’an, China, pp. 106–119, 2018. DOI: https://doi.org/10.1007/978-3-030-00563-4_11.
T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, X. Chen. Improved techniques for training GANs. In Proceedings of the 30th Conference on Neural Information Processing Systems, Curran Associates, Inc., Barcelona, Spain, pp. 2234–2242, 2016.
M. Arjovsky, L. Bottou. Towards Principled Methods for Training Generative Adversarial Networks, [Online], Available: https://arxiv.org/abs/1701.04862, April 8, 2018.
L. Metz, B. Poole, D. Pfau, J. Sohl-Dickstein. Unrolled Generative Adversarial Networks, [Online], Available: https://arxiv.org/abs/1611.02163, June 10–20, 2018.
X. T. Wang, K. Yu, C. Dong, C. C. Loy. Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform, [Online], Available: https://arxiv.org/abs/1804.02815, October, 2018.
Acknowledgements
The authors would like acknowledge the support from the Shanxi Hundred People Plan of China and colleagues from the Image Processing Group in Strathclyde University (UK), Anhui University (China) and Taibah Valley (Taibah University, Saudi Arabia) respectively, for their valuable suggestions.
Author information
Authors and Affiliations
Corresponding author
Additional information
Recommended by Associate Editor Bin Luo
Viet Khanh Ha received the B. Eng. degrees in electrical and electronics from Le Quy Don University, Viet Nam in 2008, the M. Eng. degree in electrical and electronics from Wollongong University, Australia in 2012. He is currently a Ph. D. degree candidate at the University of Strathclyde, UK.
His research interests include image super resolution using deep learning.
Jin-Chang Ren received the B. Eng. degree in computer software in 1992, the M. Eng. degree in image processing in 1997, the Ph. D. degree in computer vision in 2000, all from the North-western Polytechnical University (NWPU), China. He was also awarded a Ph. D. in electronic imaging and media communication from Bradford University, UK in 2009. Currently, he is with Centre for Signal and Image Processing (CeSIP), University of Strathclyde, UK. He has published over 150 peer reviewed journals and conferences papers. He acts as an associate editor for two international journals including Multidimensional Systems and Signal Processing and International Journal of Pattern Recognition and Artificial Intelligence.
His research interests focus mainly on visual computing and multi-media signal processing, especially on semantic content extraction for video analysis and understanding more recently hyperspectral imaging.
Xin-Ying Xu received the B. Sc. and Ph. D. degrees from the Taiyuan University of Technology, China, in 2002 and 2009, respectively. He is currently a professor with the College of Information Engineering, Taiyuan University of Technology, China. He has published more than 30 academic papers. He is a member of the Chinese Computer Society, and has been a visiting scholar in Department of Computer Science, San Jose State University, USA.
His research interests include computational intelligence, data mining, wireless networking, image processing, and fault diagnosis.
Sophia Zhao received the B. Sc. degree in education from Henan University, China in 1999, and several qualifications from Shipley College, UK during 2003–2005. Currently, she is a research assistant with the Department of Electronic and Electrical Engineering, University of Strathclyde, UK.
Her research interests include image/signal analysis, machine learning and optimisation.
Gang Xie received the B. S. degree in control theory and the Ph. D. degree in circuits and systems from the Taiyuan University of Technology, China, in 1994 and 2006, respectively. He has been a professor and vice principle of Taiyuan University of Science and Technology, China. He has published over 80 research papers.
His research interests include rough sets, intelligent computing, image processing, automation and big data analysis.
Valentin Masero received the B. Eng. degree in computer science and business administration from University of Extremadura (UEX), Spain, and another B. Eng. degree in computer engineering specialized in software development and artificial intelligence from University of Granada, Spain. He received the Ph. D. degree in computer engineering from UEX, Spain. Now he is an associate professor at UEX.
His research interests include image processing, machine learning, artificial intelligence, computer graphics, computer programming, software development, computer applications in industrial engineering, computer applications in agricultural engineering and computer applications in healthcare.
Amir Hussain received the B. Eng. and Ph. D. degrees from the University of Strathclyde in Glasgow, UK, in 1992 and 1997, respectively. Following postdoctoral and academic positions at the Universities of West of Scotland (1996–1998), Dundee (1998–2000) and Stirling (2000-2018), respectively, he joined Edinburgh Napier University (UK) in 2018, as founding director of the Cognitive Big Data and Cybersecurity (CogBiD) Research Laboratory, managing over 25 academic and research staffs. He has been appointed to invited visiting professorships at several Universities and Research and Innovation Centres, including at Anhui University (China) and Taibah Valley (Taibah University, Saudi Arabia). He has (co)authored three international patents, around 400 publications, including over a dozen books and 150 journal papers. He has led major multi-disciplinary research projects, funded by national and European research councils, local and international charities and industry, and supervised more than 35 Ph. D. students. He is founding Editor-in-Chief of (Springer Nature’s) Cognitive Computation journal and BMC Big Data Analytics journal. He has been appointed Associate Editor of several other world-leading journals including, IEEE Transactions on Neural Networks and Learning Systems, (Elsevier’s) Information Fusion journal, IEEE Transactions on Emerging Topics in Computational Intelligence, and IEEE Computational Intelligence Magazine. Amongst other distinguished roles, he is General Chair for IEEE WCCI 2020 (the world’s largest and top IEEE technical event in computational intelligence, comprising IJCNN, FUZZ-IEEE and IEEE CEC), Vice-Chair of Emergent Technologies Technical Committee of the IEEE Computational Intelligence Society, and chapter Chair of the IEEE UK & Ireland, Industry Applications Society Chapter.
His research interests include developing cognitive data science and AI technologies, to engineer the smart and secure systems of tomorrow.
Rights and permissions
About this article
Cite this article
Ha, V.K., Ren, JC., Xu, XY. et al. Deep Learning Based Single Image Super-resolution: A Survey. Int. J. Autom. Comput. 16, 413–426 (2019). https://doi.org/10.1007/s11633-019-1183-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11633-019-1183-x